[Bug 47480] New: Rhinoceros 6 (.NET 4.x app) crashes in rsaenh during local license file validation

https://bugs.winehq.org/show_bug.cgi?id=47480

Bug ID: 47480
           Summary: Rhinoceros 6 (.NET 4.x app) crashes in rsaenh during
                    local license file validation
           Product: Wine
           Version: 4.12
          Hardware: x86-64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: rsaenh
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Hello folks,

as it says.

Prerequisites:

* .NET Framework 4.5.x ('winetricks -q dotnet452')

The app installer downloads and installs MS VC++ 2005, 2010, 2013, 2017 runtime
redistributables on its own.

--- snip ---
$ pwd
/home/focht/.wine/drive_c/Program Files/Rhino 6/System

$ WINEDEBUG=+seh,+loaddll,+process,+relay,+crypt wine ./Rhino.exe >>log.txt
2>&1
...
00fd:Call
advapi32.CryptImportKey(02499d30,103ab638,0000002c,00000000,00000001,0024a648)
ret=0c7855dc
00fd:trace:crypt:CryptImportKey (0x2499d30, 0x103ab638, 44, 0x0, 00000001,
0x24a648)
00fd:Call
rsaenh.CPImportKey(00000005,103ab638,0000002c,00000000,00000001,02499cf0)
ret=7f515c928426
00fd:trace:crypt:RSAENH_CPImportKey (hProv=00000005, pbData=00000000103AB638,
dwDataLen=44, hPubKey=00000000, dwFlags=00000001, phKey=0000000002499CF0)
00fd:trace:crypt:import_key blob type: 8
00fd:Call KERNEL32.IsBadStringPtrA(6652bb20,ffffffffffffffff) ret=6650c54b
00fd:Ret  KERNEL32.IsBadStringPtrA() retval=00000000 ret=6650c54b
00fd:trace:crypt:new_key alg = "AES-256", dwKeyLen = 256
00fd:Call ntdll.RtlAllocateHeap(00010000,00000000,00000408) ret=665033c4
00fd:Ret  ntdll.RtlAllocateHeap() retval=024e5020 ret=665033c4
00fd:Call msvcrt.memcpy(024e5360,103ab644,00000020) ret=665130b7
00fd:Ret  msvcrt.memcpy() retval=024e5360 ret=665130b7
00fd:Ret  rsaenh.CPImportKey() retval=00000001 ret=7f515c928426 
...
00fd:Call KERNEL32.CreateFileW(101bb42c
L"C:\ProgramData\McNeel\Rhinoceros\6.0\License
Manager\Licenses\55500d41-3a41-4474-99b3-684032a4f4df.lic",ffffffff80000000,00000001,00000000,00000003,00100000,00000000)
ret=0a2d47b4
00fd:Ret  KERNEL32.CreateFileW() retval=00000308 ret=0a2d47b4
...
00fd:Call advapi32.CryptContextAddRef(02499d30,00000000,00000000) ret=0c78b52c
00fd:trace:crypt:CryptContextAddRef (0x2499d30, (nil), 00000000)
00fd:Ret  advapi32.CryptContextAddRef() retval=00000001 ret=0c78b52c
...
00fd:Call advapi32.CryptDuplicateKey(02499ce0,00000000,00000000,0024a5c8)
ret=0c78c54f
00fd:trace:crypt:CryptDuplicateKey (0x2499ce0, (nil), 00000000, 0x24a5c8)
00fd:Call rsaenh.CPDuplicateKey(00000005,00000006,00000000,00000000,024e4910)
ret=7f515c926238
00fd:trace:crypt:RSAENH_CPDuplicateKey (hUID=00000005, hKey=00000006,
pdwReserved=0000000000000000, dwFlags=00000000, phKey=00000000024E4910)
00fd:Call ntdll.RtlAllocateHeap(00010000,00000000,00000408) ret=665033c4
00fd:Ret  ntdll.RtlAllocateHeap() retval=024ca230 ret=665033c4
00fd:Call ntdll.RtlAllocateHeap(00010000,00000000,00000000) ret=6650d79d
00fd:Ret  ntdll.RtlAllocateHeap() retval=024e4980 ret=6650d79d
00fd:Call msvcrt.memcpy(024e4980,00000000,00000000) ret=6650d7cf
00fd:Ret  msvcrt.memcpy() retval=024e4980 ret=6650d7cf
00fd:Call ntdll.RtlAllocateHeap(00010000,00000000,00000000) ret=6650d7f8
00fd:Ret  ntdll.RtlAllocateHeap() retval=0239bca0 ret=6650d7f8
00fd:Call msvcrt.memcpy(0239bca0,00000000,00000000) ret=6650d826
00fd:Ret  msvcrt.memcpy() retval=0239bca0 ret=6650d826
00fd:Ret  rsaenh.CPDuplicateKey() retval=00000001 ret=7f515c926238
00fd:Ret  advapi32.CryptDuplicateKey() retval=00000001 ret=0c78c54f
00fd:Call advapi32.CryptContextAddRef(02499d30,00000000,00000000) ret=0c784f09
00fd:trace:crypt:CryptContextAddRef (0x2499d30, (nil), 00000000)
00fd:Ret  advapi32.CryptContextAddRef() retval=00000001 ret=0c784f09
00fd:Call KERNEL32.GetLastError() ret=02b2a82b
00fd:Ret  KERNEL32.GetLastError() retval=00000000 ret=02b2a82b 
...
00fd:Call advapi32.CryptSetKeyParam(024e4900,00000004,101c2030,00000000)
ret=0c78cbf6
00fd:trace:crypt:CryptSetKeyParam (0x24e4900, 4, 0x101c2030, 00000000)
00fd:Call rsaenh.CPSetKeyParam(00000005,00000004,00000004,101c2030,00000000)
ret=7f515c928b9d
00fd:trace:crypt:RSAENH_CPSetKeyParam (hProv=00000005, hKey=00000004,
dwParam=00000004, pbData=00000000101C2030, dwFlags=00000000)
00fd:Ret  rsaenh.CPSetKeyParam() retval=00000001 ret=7f515c928b9d
00fd:Ret  advapi32.CryptSetKeyParam() retval=00000001 ret=0c78cbf6
00fd:Call KERNEL32.GetLastError() ret=02b2a82b
00fd:Ret  KERNEL32.GetLastError() retval=00000000 ret=02b2a82b
00fd:Call advapi32.CryptSetKeyParam(024e4900,00000001,101c1fe0,00000000)
ret=0c78cbf6
00fd:trace:crypt:CryptSetKeyParam (0x24e4900, 1, 0x101c1fe0, 00000000)
00fd:Call rsaenh.CPSetKeyParam(00000005,00000004,00000001,101c1fe0,00000000)
ret=7f515c928b9d
00fd:trace:crypt:RSAENH_CPSetKeyParam (hProv=00000005, hKey=00000004,
dwParam=00000001, pbData=00000000101C1FE0, dwFlags=00000000)
00fd:Call msvcrt.memcpy(024ca5b0,101c1fe0,00000010) ret=6650db94
00fd:Ret  msvcrt.memcpy() retval=024ca5b0 ret=6650db94
00fd:Ret  rsaenh.CPSetKeyParam() retval=00000001 ret=7f515c928b9d
00fd:Ret  advapi32.CryptSetKeyParam() retval=00000001 ret=0c78cbf6
00fd:Call KERNEL32.GetLastError() ret=02b2a82b
00fd:Ret  KERNEL32.GetLastError() retval=00000000 ret=02b2a82b 
...
00fd:Call
advapi32.CryptDecrypt(024e4900,00000000,00000000,00000000,101c2100,0024a638)
ret=0c78de54
00fd:trace:crypt:CryptDecrypt (0x24e4900, 0x0, 0, 00000000, 0x101c2100,
0x24a638)
00fd:Call
rsaenh.CPDecrypt(00000005,00000004,00000000,00000000,00000000,101c2100,0024a638)
ret=7f515c92594c
00fd:trace:crypt:RSAENH_CPDecrypt (hProv=00000005, hKey=00000004,
hHash=00000000, Final=0, dwFlags=00000000, pbData=00000000101C2100,
pdwDataLen=000000000024A638)
00fd:Call msvcrt.memcpy(101c2100,0024a240,00000010) ret=66513f82
00fd:Ret  msvcrt.memcpy() retval=101c2100 ret=66513f82
00fd:Call msvcrt.memcpy(101c2110,0024a240,00000010) ret=66513f82
00fd:Ret  msvcrt.memcpy() retval=101c2110 ret=66513f82
00fd:Call msvcrt.memcpy(101c2120,0024a240,00000010) ret=66513f82 
...
00fd:Call msvcrt.memcpy(101c2820,0024a240,00000010) ret=66513f82
00fd:Ret  msvcrt.memcpy() retval=101c2820 ret=66513f82
00fd:Ret  rsaenh.CPDecrypt() retval=00000001 ret=7f515c92594c
00fd:Ret  advapi32.CryptDecrypt() retval=00000001 ret=0c78de54
...
00fd:Call
advapi32.CryptDecrypt(024e4900,00000000,00000000,00000000,101c2ff0,0024a6f8)
ret=0c78de54
00fd:trace:crypt:CryptDecrypt (0x24e4900, 0x0, 0, 00000000, 0x101c2ff0,
0x24a6f8)
00fd:Call
rsaenh.CPDecrypt(00000005,00000004,00000000,00000000,00000000,101c2ff0,0024a6f8)
ret=7f515c92594c
00fd:trace:crypt:RSAENH_CPDecrypt (hProv=00000005, hKey=00000004,
hHash=00000000, Final=0, dwFlags=00000000, pbData=00000000101C2FF0,
pdwDataLen=000000000024A6F8)
00fd:Call msvcrt.memcpy(101c2ff0,0024a300,00000010) ret=66513f82
00fd:Ret  msvcrt.memcpy() retval=101c2ff0 ret=66513f82
00fd:Ret  rsaenh.CPDecrypt() retval=00000001 ret=7f515c92594c
00fd:Ret  advapi32.CryptDecrypt() retval=00000001 ret=0c78de54
00fd:Call KERNEL32.GetLastError() ret=02b2a82b
...
00fd:Call
advapi32.CryptDecrypt(024e4900,00000000,00000001,00000000,101c3068,0024a680)
ret=0c78de54
00fd:trace:crypt:CryptDecrypt (0x24e4900, 0x0, 1, 00000000, 0x101c3068,
0x24a680)
00fd:Call
rsaenh.CPDecrypt(00000005,00000004,00000000,00000001,00000000,101c3068,0024a680)
ret=7f515c92594c
00fd:trace:crypt:RSAENH_CPDecrypt (hProv=00000005, hKey=00000004,
hHash=00000000, Final=1, dwFlags=00000000, pbData=00000000101C3068,
pdwDataLen=000000000024A680)
00fd:trace:seh:NtRaiseException code=c0000005 flags=0 addr=0x66513fc9
ip=66513fc9 tid=00fd
00fd:trace:seh:NtRaiseException  info[0]=0000000000000000
00fd:trace:seh:NtRaiseException  info[1]=00000001101c3067
00fd:trace:seh:NtRaiseException  rax=0000000000000000 rbx=0000000000000005
rcx=00000000ffffffff rdx=00000000ffffffff
00fd:trace:seh:NtRaiseException  rsi=0000000000000000 rdi=00000000101c3068
rbp=00000000024ca230 rsp=000000000024a1f0
00fd:trace:seh:NtRaiseException   r8=0000000073620457  r9=000000000024a268
r10=00000000024ca230 r11=0000000000000000
00fd:trace:seh:NtRaiseException  r12=0000000000000000 r13=000000000024a680
r14=0000000000000000 r15=0000000000000001
00fd:trace:seh:call_vectored_handlers calling handler at 0x2c729d8
code=c0000005 flags=0
00fd:Call KERNEL32.GetLastError() ret=02c72a0c
00fd:Ret  KERNEL32.GetLastError() retval=00000000 ret=02c72a0c
00fd:trace:seh:call_vectored_handlers handler at 0x2c729d8 returned 0
...
--- snip ---

Debugger:

--- snip ---
...

Wine-dbg>si
0x0000000066513fbe RSAENH_CPDecrypt+0x28e
[Z:\home\focht\projects\wine\mainline-src\dlls\rsaenh\rsaenh.c:2696] in rsaenh:
leal    0xffffffffffffffff(%rax),%edx
2696                if (pbData[*pdwDataLen-1] &&

Wine-dbg>si
0x0000000066513fc1 RSAENH_CPDecrypt+0x291
[Z:\home\focht\projects\wine\mainline-src\dlls\rsaenh\rsaenh.c:2696] in rsaenh:
movq    0x0000000000000078(%rsp),%r10
2696                if (pbData[*pdwDataLen-1] &&

Wine-dbg>si
0x0000000066513fc6 RSAENH_CPDecrypt+0x296
[Z:\home\focht\projects\wine\mainline-src\dlls\rsaenh\rsaenh.c:2696] in rsaenh:
movq    %rdx,%rcx
2696                if (pbData[*pdwDataLen-1] &&

Wine-dbg>si
0x0000000066513fc9 RSAENH_CPDecrypt+0x299
[Z:\home\focht\projects\wine\mainline-src\dlls\rsaenh\rsaenh.c:2696] in rsaenh:
movzbl    (%rdi,%rdx,1),%r9d
2696                if (pbData[*pdwDataLen-1] &&

Wine-dbg>info reg
Register dump:
 rip:0000000066513fc9 rsp:000000000022a2f0 rbp:00000000024c5650 eflags:00000202
(   - --  I   - - - )
 rax:0000000000000000 rbx:0000000000000005 rcx:00000000ffffffff
rdx:00000000ffffffff
 rsi:0000000000000000 rdi:00000000101cbe58  r8:0000000073620457 
r9:000000000022a368 r10:00000000024c5650
 r11:0000000066513d30 r12:0000000000000000 r13:000000000022a680
r14:0000000000000000 r15:0000000000000001
--- snip ---

https://source.winehq.org/git/wine.git/blob/18fbc3a3cf880f35ab497c0e2026fe85...

--- snip ---
2623 BOOL WINAPI RSAENH_CPDecrypt(HCRYPTPROV hProv, HCRYPTKEY hKey, HCRYPTHASH
hHash, BOOL Final, 
2624                              DWORD dwFlags, BYTE *pbData, DWORD
*pdwDataLen)
2625 {
2626     CRYPTKEY *pCryptKey;
2627     BYTE *in, out[RSAENH_MAX_BLOCK_SIZE], o[RSAENH_MAX_BLOCK_SIZE];
2628     DWORD i, j, k;
2629     DWORD dwMax;
2630 
2631     TRACE("(hProv=%08lx, hKey=%08lx, hHash=%08lx, Final=%d, dwFlags=%08x,
pbData=%p, "
2632           "pdwDataLen=%p)\n", hProv, hKey, hHash, Final, dwFlags, pbData,
pdwDataLen);
...
2661     dwMax=*pdwDataLen;
2662 
2663     if (GET_ALG_TYPE(pCryptKey->aiAlgid) == ALG_TYPE_BLOCK) {
2664         for (i=0, in=pbData; i<*pdwDataLen; i+=pCryptKey->dwBlockLen,
in+=pCryptKey->dwBlockLen) {
2665             switch (pCryptKey->dwMode) {
2666                 case CRYPT_MODE_ECB:
2667                     encrypt_block_impl(pCryptKey->aiAlgid, 0,
&pCryptKey->context, in, out, 
2668                                        RSAENH_DECRYPT);
2669                     break;
2670                 
2671                 case CRYPT_MODE_CBC:
2672                     encrypt_block_impl(pCryptKey->aiAlgid, 0,
&pCryptKey->context, in, out, 
2673                                        RSAENH_DECRYPT);
2674                     for (j=0; j<pCryptKey->dwBlockLen; j++) out[j] ^=
pCryptKey->abChainVector[j];
2675                     memcpy(pCryptKey->abChainVector, in,
pCryptKey->dwBlockLen);
2676                     break;
2677 
2678                 case CRYPT_MODE_CFB:
2679                     for (j=0; j<pCryptKey->dwBlockLen; j++) {
2680                         encrypt_block_impl(pCryptKey->aiAlgid, 0,
&pCryptKey->context, 
2681                                            pCryptKey->abChainVector, o,
RSAENH_ENCRYPT);
2682                         out[j] = in[j] ^ o[0];
2683                         for (k=0; k<pCryptKey->dwBlockLen-1; k++) 
2684                             pCryptKey->abChainVector[k] =
pCryptKey->abChainVector[k+1];
2685                         pCryptKey->abChainVector[k] = in[j];
2686                     }
2687                     break;
2688                     
2689                 default:
2690                     SetLastError(NTE_BAD_ALGID);
2691                     return FALSE;
2692             }
2693             memcpy(in, out, pCryptKey->dwBlockLen);
2694         }
2695         if (Final) {
2696             if (pbData[*pdwDataLen-1] &&
2697              pbData[*pdwDataLen-1] <= pCryptKey->dwBlockLen &&
2698              pbData[*pdwDataLen-1] <= *pdwDataLen) {
2699                 BOOL padOkay = TRUE;
2700 
2701                 /* check that every bad byte has the same value */
2702                 for (i = 1; padOkay && i < pbData[*pdwDataLen-1]; i++)
2703                     if (pbData[*pdwDataLen - i - 1] != pbData[*pdwDataLen
- 1])
2704                         padOkay = FALSE;
2705                 if (padOkay)
2706                     *pdwDataLen -= pbData[*pdwDataLen-1];
2707                 else {
2708                     SetLastError(NTE_BAD_DATA);
2709                     setup_key(pCryptKey);
2710                     return FALSE;
2711                 }
2712             }
2713             else {
2714                 SetLastError(NTE_BAD_DATA);
2715                 setup_key(pCryptKey);
2716                 return FALSE;
2717             }
2718         }
...
2734     
2735     if (Final) setup_key(pCryptKey);
...
--- some ---

Line 2696, the caller which seems some jitted .NET code passed zero *pdwDataLen
for whatever reason.

Microsoft docs:

https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryp...

--- quote ---
...
pbData

A pointer to a buffer that contains the data to be decrypted. After the
decryption has been performed, the plaintext is placed back into this same
buffer.

The number of encrypted bytes in this buffer is specified by pdwDataLen.

pdwDataLen

A pointer to a DWORD value that indicates the length of the pbData buffer.
Before calling this function, the calling application sets the DWORD value to
the number of bytes to be decrypted. Upon return, the DWORD value contains the
number of bytes of the decrypted plaintext.

When a block cipher is used, this data length must be a multiple of the block
size unless this is the final section of data to be decrypted and the Final
parameter is TRUE.
--- quote ---

I've added a check for zero *pdwDataLen in line 2695 to skip that block and it
made the app successfully start. I don't know if this is just undocumented
behaviour or an application bug.

$ sha1sum rhino_en-us_6.15.19164.21011.exe 
85e34606c761ec7229cbe2c6ba7efeae7e778a97  rhino_en-us_6.15.19164.21011.exe

$ du -sh rhino_en-us_6.15.19164.21011.exe 
263M    rhino_en-us_6.15.19164.21011.exe

$ wine --version
wine-4.12.1

Regards