http://bugs.winehq.org/show_bug.cgi?id=21443
--- Comment #6 from Kenneth Robinette support@securenetterm.com 2010-05-20 11:31:18 --- (In reply to comment #5)
(In reply to comment #4)
Actually, I'm not sure that the line you highlighted is in error. It needs tests to determine the correct behavior. As Microsoft states in MSDN at "Creating a CALG_SSL3_SHAMD5 Hash" [1],
- Get a handle to a hash object by calling CryptCreateHash with
CALG_SSL3_SHAMD5 passed in the Algid parameter. 4. Set the hash value with a call to CryptSetHashParam. The concatenated hash values are passed as a BYTE* in the pbData parameter, and the HP_HASHVAL value must be passed in the dwParam parameter. Calling CryptHashData using the handle returned by CryptCreateHash in step 3 will fail.
This is indeed what the OpenSSL source does: it calls CryptCreateHash, CryptSetHashParam, and CryptSignHash, but not CryptHashData. Internally, Wine's CPSignHash doesn't call CPHashData, either.
I think the more likely source of the problem is in build_hash_signature: http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;h=665bc...
aOIDDescriptor has implementations for CALG_MD2, CALG_MD4, CALG_MD5, and CALG_SHA, but not for CALG_SSL3_SHAMD5. Likewise, encrypt_block_impl doesn't have an implementation for CALG_SSL3_SHAMD5.
[1] Assuming MSDN doesn't go breaking its links again, http://msdn.microsoft.com/en-us/library/aa379865(VS.85).aspx
Yea, I went down the wrong path trying to trace the source of the problem. The code I have that is failing does not call CryptHashData(). However, you can duplicate the problem using the OpenSSL test case I documented in my attachment.
Futher testing confirms that the problem does happen in the build_hash_signature() called from RSAENH_CPSignHash(). The following is a partial trace showing trace data displayed from within the RSAENH_CPSighHash function.
trace:crypt:RSAENH_CPSignHash (hProv=00000002, hHash=00000004, dwKeySpec=00000001, sDescription=(null), dwFlags=00000000, pbSignature=0x12ec478, pdwSigLen=0x33e844) trace:crypt:RSAENH_CPGetUserKey (hProv=00000002, dwKeySpec=00000001, phUserKey=0x33e764) trace:crypt:RSAENH_CPGetHashParam (hProv=00000002, hHash=00000004, dwParam=00000001, pbData=0x33e758, pdwDataLen=0x33e75c, dwFlags=00000000) trace:crypt:RSAENH_CPGetHashParam (hProv=00000002, hHash=00000004, dwParam=00000002, pbData=0x33e6f0, pdwDataLen=0x33e75c, dwFlags=00000000) trace:crypt:RSAENH_CPDestroyKey (hProv=00000002, hKey=00000006)