[Bug 21443] New: CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Summary: CryptSignHash() fails when CryptCreateHash() ALG_ID is
                    set to CALG_SSL3_SHAMD5
           Product: Wine
           Version: 1.1.36
          Platform: x86-64
               URL: http://www.secureneterm.com
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]

Created an attachment (id=25823)
 --> (http://bugs.winehq.org/attachment.cgi?id=25823)
Problem discussion on how to duplicate.

CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

This can be duplicated using the latest version of OpenSSL as outlined in the
attachment.  Under Wine, the test outlined in the attachment is:

CONNECTED(00000040)
depth=0 CN = InterSoft International Inc.
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = InterSoft International Inc.
verify return:1
43:error:8907006F:lib(137):CAPI_RSA_SIGN:error signing
hash:.\engines\e_capi.c:844:Error code= 0x80090008
43:error:14099004:SSL routines:SSL3_SEND_CLIENT_VERIFY:RSA
lib:.\ssl\s3_clnt.c:2707:

Note that this error does not happen under Microsoft Windows.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|-unknown                    |advapi32

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Juan Lang  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|advapi32                    |rsaenh

--- Comment #1 from Juan Lang   2010-01-21 13:18:40 ---
This bug is likely to be in rsaenh, as advapi32 just forwards the call to the
provider.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #2 from Kenneth Robinette   2010-01-30 09:32:29 ---
(In reply to comment #1)
> This bug is likely to be in rsaenh, as advapi32 just forwards the call to the
> provider.

Yes, the bug is in rsaenh.c in the RSAENH_CPHashData() function at line 3959:

if (!get_algid_info(hProv, pCryptHash->aiAlgid) || pCryptHash->aiAlgid ==
CALG_SSL3_SHAMD5)
{
    SetLastError(NTE_BAD_ALGID);
    return FALSE;
}

This check forces an error, but why?  There is no comment or documentation on
why the Wine implementation does not support CALG_SSL3_SHAMD5.

Again, it can be confirmed that the Microsoft CryptSighHash function supports
this.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #3 from Juan Lang   2010-03-11 15:21:26 ---
That check was present in the initial commit of rsaenh:
http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;hb=64dc...

As you say, no comment or test to justify it.

What happens if you remove the check from rsaenh.c?

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #4 from Juan Lang   2010-03-11 15:50:26 ---
Actually, I'm not sure that the line you highlighted is in error.  It needs
tests to determine the correct behavior.  As Microsoft states in MSDN at
"Creating a CALG_SSL3_SHAMD5 Hash" [1],

3. Get a handle to a hash object by calling CryptCreateHash with
CALG_SSL3_SHAMD5 passed in the Algid parameter.
4. Set the hash value with a call to CryptSetHashParam. The concatenated hash
values are passed as a BYTE* in the pbData parameter, and the HP_HASHVAL value
must be passed in the dwParam parameter. Calling CryptHashData using the handle
returned by CryptCreateHash in step 3 will fail.

This is indeed what the OpenSSL source does:  it calls CryptCreateHash,
CryptSetHashParam, and CryptSignHash, but not CryptHashData.  Internally,
Wine's CPSignHash doesn't call CPHashData, either.

I think the more likely source of the problem is in build_hash_signature:
http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;h=665bc...

aOIDDescriptor has implementations for CALG_MD2, CALG_MD4, CALG_MD5, and
CALG_SHA, but not for CALG_SSL3_SHAMD5.  Likewise, encrypt_block_impl doesn't
have an implementation for CALG_SSL3_SHAMD5.

[1] Assuming MSDN doesn't go breaking its links again,
http://msdn.microsoft.com/en-us/library/aa379865(VS.85).aspx

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #5 from Kenneth Robinette   2010-03-11 19:47:32 ---
(In reply to comment #4)
> Actually, I'm not sure that the line you highlighted is in error.  It needs
> tests to determine the correct behavior.  As Microsoft states in MSDN at
> "Creating a CALG_SSL3_SHAMD5 Hash" [1],
> - Get a handle to a hash object by calling CryptCreateHash with
>
> CALG_SSL3_SHAMD5 passed in the Algid parameter.
> 4. Set the hash value with a call to CryptSetHashParam. The concatenated hash
> values are passed as a BYTE* in the pbData parameter, and the HP_HASHVAL value
> must be passed in the dwParam parameter. Calling CryptHashData using the handle
> returned by CryptCreateHash in step 3 will fail.
>
> This is indeed what the OpenSSL source does:  it calls CryptCreateHash,
> CryptSetHashParam, and CryptSignHash, but not CryptHashData.  Internally,
> Wine's CPSignHash doesn't call CPHashData, either.
>
> I think the more likely source of the problem is in build_hash_signature:
> http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;h=665bc...
>
> aOIDDescriptor has implementations for CALG_MD2, CALG_MD4, CALG_MD5, and
> CALG_SHA, but not for CALG_SSL3_SHAMD5.  Likewise, encrypt_block_impl doesn't
> have an implementation for CALG_SSL3_SHAMD5.
>
> [1] Assuming MSDN doesn't go breaking its links again,
> http://msdn.microsoft.com/en-us/library/aa379865(VS.85).aspx

Yea, I went down the wrong path trying to trace the source of the problem.  The
code I have that is failing does not call CryptHashData().  However, you can
duplicate the problem using the OpenSSL test case I documented in my
attachment.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #6 from Kenneth Robinette   2010-05-20 11:31:18 ---
(In reply to comment #5)
> (In reply to comment #4)
> > Actually, I'm not sure that the line you highlighted is in error.  It needs
> > tests to determine the correct behavior.  As Microsoft states in MSDN at
> > "Creating a CALG_SSL3_SHAMD5 Hash" [1],
> > - Get a handle to a hash object by calling CryptCreateHash with
> >
> > CALG_SSL3_SHAMD5 passed in the Algid parameter.
> > 4. Set the hash value with a call to CryptSetHashParam. The concatenated hash
> > values are passed as a BYTE* in the pbData parameter, and the HP_HASHVAL value
> > must be passed in the dwParam parameter. Calling CryptHashData using the handle
> > returned by CryptCreateHash in step 3 will fail.
> >
> > This is indeed what the OpenSSL source does:  it calls CryptCreateHash,
> > CryptSetHashParam, and CryptSignHash, but not CryptHashData.  Internally,
> > Wine's CPSignHash doesn't call CPHashData, either.
> >
> > I think the more likely source of the problem is in build_hash_signature:
> > http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;h=665bc...
> >
> > aOIDDescriptor has implementations for CALG_MD2, CALG_MD4, CALG_MD5, and
> > CALG_SHA, but not for CALG_SSL3_SHAMD5.  Likewise, encrypt_block_impl doesn't
> > have an implementation for CALG_SSL3_SHAMD5.
> >
> > [1] Assuming MSDN doesn't go breaking its links again,
> > http://msdn.microsoft.com/en-us/library/aa379865(VS.85).aspx
>
> Yea, I went down the wrong path trying to trace the source of the problem.  The
> code I have that is failing does not call CryptHashData().  However, you can
> duplicate the problem using the OpenSSL test case I documented in my
> attachment.

Futher testing confirms that the problem does happen in the
build_hash_signature() called from RSAENH_CPSignHash().  The following is a
partial trace showing trace data displayed from within the RSAENH_CPSighHash
function.

trace:crypt:RSAENH_CPSignHash (hProv=00000002, hHash=00000004,
dwKeySpec=00000001, sDescription=(null), dwFlags=00000000,
pbSignature=0x12ec478, pdwSigLen=0x33e844)
trace:crypt:RSAENH_CPGetUserKey (hProv=00000002, dwKeySpec=00000001,
phUserKey=0x33e764)
trace:crypt:RSAENH_CPGetHashParam (hProv=00000002, hHash=00000004,
dwParam=00000001, pbData=0x33e758, pdwDataLen=0x33e75c, dwFlags=00000000)
trace:crypt:RSAENH_CPGetHashParam (hProv=00000002, hHash=00000004,
dwParam=00000002, pbData=0x33e6f0, pdwDataLen=0x33e75c, dwFlags=00000000)
trace:crypt:RSAENH_CPDestroyKey (hProv=00000002, hKey=00000006)

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #7 from Juan Lang   2010-05-20 11:57:48 ---
Created an attachment (id=28127)
 --> (http://bugs.winehq.org/attachment.cgi?id=28127)
Patch: dummy implementation

Out of curiosity, how does this work for you?  I haven't tested it at all, it's
just a quick check.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

--- Comment #8 from Kenneth Robinette   2010-05-20 13:33:40 ---
(In reply to comment #7)
> Created an attachment (id=28127)

--> (http://bugs.winehq.org/attachment.cgi?id=28127) [details]
> Patch: dummy implementation
> Out of curiosity, how does this work for you?  I haven't tested it at all, it's
> just a quick check.

That fixed the problem!

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Juan Lang  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--- Comment #9 from Juan Lang   2010-05-20 14:20:59 ---
(In reply to comment #8)
> That fixed the problem!

Thanks!  I'll try to write some tests so this can go in.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Juan Lang  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |FIXED

--- Comment #10 from Juan Lang   2010-05-21 12:18:37 ---
Fixed by commit 62d806601bfa992f2e36ff4d208ea9781218bcd6.

[Bug 21443] CryptSignHash() fails when CryptCreateHash() ALG_ID is set to CALG_SSL3_SHAMD5

http://bugs.winehq.org/show_bug.cgi?id=21443

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #11 from Alexandre Julliard   2010-05-21 14:39:37 ---
Closing bugs fixed in 1.2-rc1.