https://bugs.winehq.org/show_bug.cgi?id=54320
Bug ID: 54320 Summary: BCryptDecrypt Symmetric Key Flags Behavior Product: Wine Version: 7.2 Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: minor Priority: P2 Component: bcrypt Assignee: wine-bugs@winehq.org Reporter: jutras.marc@gmail.com Distribution: ---
In the BCryptDecrypt function, the behavior of the function is to check if the `key_is_symmetric`, then if that is the case confirm whether the flags are implemented or not. If the flags are not implemented a FIXME is printed and the function returns an error.
In the software Solid Edge 2023, this behavior is preventing the software from running correctly. Solid Edge appears to have a bug where it is passing an invalid key to the `handle` parameter (a bad pointer?) which results in the key type being recognized as symmetric (likely because `key->alg_id` of zero (after a deref to empty memory) is 3DES which is symmetric).
The call to BCryptDecrypt has the flag `BCRYPT_PAD_PKCS1` set which, combined with the key being recognized as symmetric, causes the function to return an error status. My assumption is that Windows is not checking the flags and is returning garbage as the decryption. This allows the program to continue running (ignorance is bliss).
I have not run the debugger to confirm that `key->alg_id` is zero. I have also not confirmed this function's behavior on Windows.
I have confirmed that removing the `return STATUS_NOT_IMPLEMENTED` fixes the issue in Solid Edge 2023.
The proposal would be to change the check from `flags & ~BCRYPT_BLOCK_PADDING` to `flags == BCRYPT_BLOCK_PADDING`. This allows the STATUS_NOT_IMPLEMENTED to be returned in the correct scenario. All other flag values are errors, which it appears Windows ignores.
Tested in Wine 7.2-2 Lutris. There were additional problems which prevented running the software under my system's Wine 7.22.