[Bug 54320] New: BCryptDecrypt Symmetric Key Flags Behavior

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54320

Bug ID: 54320
           Summary: BCryptDecrypt Symmetric Key Flags Behavior
           Product: Wine
           Version: 7.2
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: minor
          Priority: P2
         Component: bcrypt
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

In the BCryptDecrypt function, the behavior of the function is to check if the
`key_is_symmetric`, then if that is the case confirm whether the flags are
implemented or not. If the flags are not implemented a FIXME is printed and the
function returns an error.

In the software Solid Edge 2023, this behavior is preventing the software from
running correctly. Solid Edge appears to have a bug where it is passing an
invalid key to the `handle` parameter (a bad pointer?) which results in the key
type being recognized as symmetric (likely because `key->alg_id` of zero (after
a deref to empty memory) is 3DES which is symmetric).

The call to BCryptDecrypt has the flag `BCRYPT_PAD_PKCS1` set which, combined
with the key being recognized as symmetric, causes the function to return an
error status. My assumption is that Windows is not checking the flags and is
returning garbage as the decryption. This allows the program to continue
running (ignorance is bliss).

I have not run the debugger to confirm that `key->alg_id` is zero. I have also
not confirmed this function's behavior on Windows.

I have confirmed that removing the `return STATUS_NOT_IMPLEMENTED` fixes the
issue in Solid Edge 2023.

The proposal would be to change the check from `flags & ~BCRYPT_BLOCK_PADDING`
to `flags == BCRYPT_BLOCK_PADDING`. This allows the STATUS_NOT_IMPLEMENTED to
be returned in the correct scenario. All other flag values are errors, which it
appears Windows ignores.

Tested in Wine 7.2-2 Lutris. There were additional problems which prevented
running the software under my system's Wine 7.22.

[Bug 54320] BCryptDecrypt Symmetric Key Flags Behavior

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54320

--- Comment #1 from Hans Leidekker  ---
Thanks. Would you be able to add a test case for this in
dlls/bcrypt/tests/bcrypt.c?