http://bugs.winehq.org/show_bug.cgi?id=3254
--- Comment #77 from Dylan Smith dylan.ah.smith@gmail.com 2009-09-02 19:00:52 --- Created an attachment (id=23392) --> (http://bugs.winehq.org/attachment.cgi?id=23392) secur32: Handle incomplete schannel records on decryption.
I pulled myself away from the richedit controls for a bit to track this down.
I found that the problem was with DencryptMessage being called with a data buffer that contains incomplete records. This will happen when a lot of data is received at a time. The schannel code in wine was handling this by letting the partial record be pulled by libgnutls, so no decrypted data will be returned for the partial record, but will be included in the data returned from the following call to DecryptMessage. This seems alright so far, but the amount of data that can be returned for a call to DecryptMessage is limited to the size of the buffer holding the encrypted data. Once the buffer was filled on a call to DecryptMessage, it would ignore the rest of the encrypted data without notifying the caller in any way. So the next DecryptMessage will try to read the next record header, but will read from the wrong place, causing errors that cause the disconnection.
The patch I submitted deals with this by not letting the pull function give libgnutls a partial record for DecryptMessage, and then notifying the caller about the unprocessed data by changing an empty buffer to a buffer fo type SECBUFFER_EXTRA with a size field that gives the size of the unprocessed data. This is documented on MSDN at http://msdn.microsoft.com/en-us/library/aa375412(VS.85).aspx
Currently the test suites for schannel don't cover EncryptMessage or DecryptMessage. These tests should probably be implemented in order to make sure this is implemented properly, but my patch at least seems to make Google Talk happy. Let me know if you still experience connection problems with this patch applied.