[Bug 47480] Multiple .NET 4.x games and applications crash in rsaenh (Rhinoceros 6, Project Reality Launcher 5.5.x)(CryptEncrypt / CryptDecrypt must support state reset)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=47480

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Rhinoceros 6 (.NET 4.x app) |Multiple .NET 4.x games and
                   |crashes in rsaenh during    |applications crash in
                   |local license file          |rsaenh (Rhinoceros 6,
                   |validation                  |Project Reality Launcher
                   |                            |5.5.x)(CryptEncrypt /
                   |                            |CryptDecrypt must support
                   |                            |state reset)

--- Comment #2 from Anastasius Focht  ---
Hello Lorenzo,

--- quote ---
Project Reality Launcher 5.5.0 is also affected from this bug. (prequisiste
winetricks dotnet40)
--- quote ---

indeed, refining summary to reflect this.

Managed backtrace of 'PRLauncher' process:

--- snip ---
Unhandled Exception: System.Reflection.TargetInvocationException: Exception has
been thrown by the target of an invocation. --->
System.Reflection.TargetInvocationException: Exception has been thrown by the
target of an invocation. ---> System.AccessViolationException: Attempted to
read or write protected memory. This is often an indication that other memory
is corrupt.
   at
System.Security.Cryptography.CapiNative.UnsafeNativeMethods.CryptDecrypt(SafeCapiKeyHandle
hKey, SafeCapiHashHandle hHash, Boolean Final, Int32 dwFlags, IntPtr pbData,
Int32& pdwDataLen)
   at System.Security.Cryptography.CapiSymmetricAlgorithm.Reset()
   at
System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[]
inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.Read(Byte[] buffer, Int32
offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadToEnd()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method,
Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes
methodAttributes, RuntimeType typeOwner)
   at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method,
Object target, Object[] arguments, Signature sig, MethodAttributes
methodAttributes, RuntimeType typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean
skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
...
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method,
Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes
methodAttributes, RuntimeType typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean
skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
...
   at PRLauncher.WPF.ViewModel.MainWindowViewModel.<.ctor>b__121_18()
...
   at -.f.c()
wine: Unhandled exception 0xe0434352 in thread 2d at address 000000007104F93D
(thread 002d), starting debugger...
--- snip ---

--- snip ---
Threads:
process  tid      prio (all id:s are in hex)
...
0000002c (D) C:\Program Files (x86)\Project Reality\Project Reality
BF2\mods\pr\bin\PRLauncher.exe
    00000042    0
...
    0000002e    0
    0000002d    0 <==
--- snip --

Corresponding .NET reference source:

'TransformFinalBlock' ->
https://github.com/microsoft/referencesource/blob/master/System.Core/System/...

which calls 'Reset' method ->
https://github.com/microsoft/referencesource/blob/master/System.Core/System/...

--- snip ---
        /// <summary>
        ///     Reset the state of the algorithm so that it can begin
processing a new message
        /// </summary>
        [SecuritySafeCritical]
        private void Reset() {
            Contract.Requires(m_key != null);
            Contract.Ensures(m_depadBuffer == null);

//
            // CryptEncrypt / CryptDecrypt must be called with the Final
parameter set to true so that
            // their internal state is reset. Since we do all padding by hand,
this isn't done by
            // TransformFinalBlock so is done on an empty buffer here.
            //

byte[] buffer = new byte[OutputBlockSize];
            int resetSize = 0;
            unsafe {
                fixed (byte* pBuffer = buffer) {
                    if (m_encryptionMode == EncryptionMode.Encrypt) {
...
                    }
                    else {
                        if
(!LocalAppContextSwitches.AesCryptoServiceProviderDontCorrectlyResetDecryptor)
{
                            resetSize = buffer.Length;
                        }
                        CapiNative.UnsafeNativeMethods.CryptDecrypt(m_key,

SafeCapiHashHandle.InvalidHandle,
                                                                    true,
                                                                    0,
                                                                    new
IntPtr(pBuffer),
                                                                    ref
resetSize);
                    }
                }
            }

// Also erase the depadding buffer so we don't cross data from the
previous message into this one
            if (m_depadBuffer != null) {
                Array.Clear(m_depadBuffer, 0, m_depadBuffer.Length);
                m_depadBuffer = null;
            }
}
--- snip ---

The function description highlights the use-case:

--- quote ---
Reset the state of the algorithm so that it can begin processing a new message
--- quote ---

Another tidbit from the sources, indicating a workaround:

'LocalAppContextSwitches.AesCryptoServiceProviderDontCorrectlyResetDecryptor'

Download:

http://files.realitymod.com/downloadassistant/PRBF2-Download-Assistant.exe

$ sha1sum prbf2_1.5.0.0_full.iso 
b097e6cef3fa063a08fddd945ac74b389f864806  prbf2_1.5.0.0_full.iso

$ du -sh prbf2_1.5.0.0_full.iso 
8.3G    prbf2_1.5.0.0_full.iso

$ wine --version
wine-5.0-rc3

Regards