http://bugs.winehq.org/show_bug.cgi?id=21443
--- Comment #5 from Kenneth Robinette support@securenetterm.com 2010-03-11 19:47:32 --- (In reply to comment #4)
Actually, I'm not sure that the line you highlighted is in error. It needs tests to determine the correct behavior. As Microsoft states in MSDN at "Creating a CALG_SSL3_SHAMD5 Hash" [1],
- Get a handle to a hash object by calling CryptCreateHash with
CALG_SSL3_SHAMD5 passed in the Algid parameter. 4. Set the hash value with a call to CryptSetHashParam. The concatenated hash values are passed as a BYTE* in the pbData parameter, and the HP_HASHVAL value must be passed in the dwParam parameter. Calling CryptHashData using the handle returned by CryptCreateHash in step 3 will fail.
This is indeed what the OpenSSL source does: it calls CryptCreateHash, CryptSetHashParam, and CryptSignHash, but not CryptHashData. Internally, Wine's CPSignHash doesn't call CPHashData, either.
I think the more likely source of the problem is in build_hash_signature: http://source.winehq.org/git/wine.git/?a=blob;f=dlls/rsaenh/rsaenh.c;h=665bc...
aOIDDescriptor has implementations for CALG_MD2, CALG_MD4, CALG_MD5, and CALG_SHA, but not for CALG_SSL3_SHAMD5. Likewise, encrypt_block_impl doesn't have an implementation for CALG_SSL3_SHAMD5.
[1] Assuming MSDN doesn't go breaking its links again, http://msdn.microsoft.com/en-us/library/aa379865(VS.85).aspx
Yea, I went down the wrong path trying to trace the source of the problem. The code I have that is failing does not call CryptHashData(). However, you can duplicate the problem using the OpenSSL test case I documented in my attachment.