[Bug 50849] Elgato Stream Deck 4.9.3 (.NET 4.5 app) installer: invocation of managed custom action 'DeleteInvalidPathRegistryKey' returns incorrect result with Wine-Mono

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=50849

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]
            Summary|Streamdeck MSI installation |Elgato Stream Deck 4.9.3
                   |fails                       |(.NET 4.5 app) installer:
                   |                            |invocation of managed
                   |                            |custom action
                   |                            |'DeleteInvalidPathRegistryK
                   |                            |ey' returns incorrect
                   |                            |result with Wine-Mono
                URL|                            |https://web.archive.org/web
                   |                            |/20210323091854/https://edg
                   |                            |e.elgato.com/egc/windows/sd
                   |                            |/Stream_Deck_4.9.3.13222.ms
                   |                            |i
     Ever confirmed|0                           |1
          Component|-unknown                    |mscoree
           Keywords|                            |download, Installer
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from Anastasius Focht  ---
Hello folks,

confirming. It's a .NET 4.5 app which requires 'Windows 10' setting.

Using native .NET Framework 4.5 ('winetricks --force -q dotnet45') works around
- only to run into the next bug.

--- snip ---
$ WINE_MONO_VERBOSE=1 WINEDEBUG=+seh,+loaddll,+mscoree,+relay,+msi wine msiexec
-i Stream_Deck_4.9.3.13222.msi
...
0024:trace:msi:ACTION_CustomAction Handling custom action
L"DeleteInvalidPathRegistryKey" (1 L"StreamDeckCustomAction.CA.dll"
L"DeleteRegistryEntryIfNeeded") 
...
0108:Call KERNEL32.CreateProcessW(0150e690
L"C:\windows\system32\rundll32.exe",0150eab0 L"rundll32.exe
"C:\users\focht\Temp\msi30b8.tmp",zzzzInvokeManagedCustomActionOutOfProc
SfxCA_13906298 1
StreamDeckCustomAction!StreamDeckCustomAction.CustomActions.DeleteRegistryEntryIfNeeded",00000000,00000000,00000000,00000000,00000000,00000000,0150e550,0150e4b8)
ret=01523037 
...
011c:trace:mscoree:DllMain (0000000001390000, 1, 0000000000000000)
011c:trace:mscoree:GetRequestedRuntimeInfo
(L"C:\users\focht\Temp\msi96a0.tmp-\Microsoft.Deployment.WindowsInstaller.dll",
(null), L"C:\users\focht\Temp\msi96a0.tmp-\CustomAction.config",
0x00000000, 0x00000000, 000000000022F040, 0x00000104, 0000000000000000,
000000000022F018, 0x00000014, 0000000000000000)
011c:trace:mscoree:CreateConfigStream
(L"C:\users\focht\Temp\msi96a0.tmp-\CustomAction.config",
000000000022EA00)
...
Method int
Microsoft.Deployment.WindowsInstaller.CustomActionProxy:InvokeCustomAction64
(int,string,long) emitted at 000000000c880520 to 000000000c880579 (code length
89) [CustomAction]
Method (wrapper runtime-invoke) object
<Module>:runtime_invoke_int_int_object_long (object,intptr,intptr,intptr)
emitted at 000000000c8805a0 to 000000000c880780 (code length 480)
[CustomAction]
Method int
Microsoft.Deployment.WindowsInstaller.CustomActionProxy:InvokeCustomAction
(int,string,intptr) emitted at 000000000c880790 to 000000000c881021 (code
length 2193) [CustomAction]
Method System.Delegate
System.Runtime.InteropServices.Marshal:GetDelegateForFunctionPointer
(intptr,System.Type) emitted at 000000000c8810e0 to 000000000c8812c3 (code
length 483) [CustomAction]
Method (wrapper managed-to-native) System.Delegate
System.Runtime.InteropServices.Marshal:GetDelegateForFunctionPointerInternal
(intptr,System.Type) emitted at 000000000c8812e0 to 000000000c8813ed (code
length 269) [CustomAction]
Method (wrapper managed-to-native) void object:wrapper_native_0000000000ae3134
(Microsoft.Deployment.WindowsInstaller.RemoteMsiFunctionId,intptr,intptr&)
emitted at 000000000c881400 to 000000000c88154d (code length 333)
[CustomAction]
Method void
Microsoft.Deployment.WindowsInstaller.RemotableNativeMethods:set_RemotingDelegate
(Microsoft.Deployment.WindowsInstaller.MsiRemoteInvoke) emitted at
000000000c881570 to 000000000c881672 (code length 258) [CustomAction]
...
Method Microsoft.Deployment.WindowsInstaller.ActionResult
StreamDeckCustomAction.CustomActions:DeleteRegistryEntryIfNeeded
(Microsoft.Deployment.WindowsInstaller.Session) emitted at 000000000cb6ad10 to
000000000cb6aefd (code length 493) [CustomAction] 
...
Method Microsoft.Deployment.WindowsInstaller.ActionResult
StreamDeckCustomAction.CustomActions:DeleteRegistryEntryIfNeeded
(Microsoft.Deployment.WindowsInstaller.Session) emitted at 000000000cb6ad10 to
000000000cb6aefd (code length 493) [CustomAction]
Method (wrapper runtime-invoke) object
<Module>:runtime_invoke_ActionResult_object (object,intptr,intptr,intptr)
emitted at 000000000cb6af30 to 000000000cb6b0f0 (code length 448)
[CustomAction]
Method (wrapper remoting-invoke-with-check) Microsoft.Win32.RegistryKey
Microsoft.Win32.RegistryKey:OpenSubKey (string,bool) emitted at
000000000cb6b100 to 000000000cb6b1b9 (code length 185) [CustomAction]
Method Microsoft.Win32.RegistryKey Microsoft.Win32.RegistryKey:OpenSubKey
(string,bool) emitted at 000000000cb6b1f0 to 000000000cb6b291 (code length 161)
[CustomAction]
Method void Microsoft.Win32.RegistryKey:ValidateKeyName (string) emitted at
000000000cb6b2d0 to 000000000cb6b40d (code length 317) [CustomAction]
...
Method int Microsoft.Win32.RegistryKey:GetRegistryKeyAccess (bool) emitted at
000000000cb6c5e0 to 000000000cb6c601 (code length 33) [CustomAction]
Method (wrapper managed-to-native) int Interop/Advapi32:RegOpenKeyEx
(Microsoft.Win32.SafeHandles.SafeRegistryHandle,string,int,int,Microsoft.Win32.SafeHandles.SafeRegistryHandle&)
emitted at 000000000cb6c620 to 000000000cb6c910 (code length 752)
[CustomAction]
Method void Microsoft.Win32.SafeHandles.SafeRegistryHandle:.ctor () emitted at
000000000cb6c930 to 000000000cb6c979 (code length 73) [CustomAction]
Method void System.NullReferenceException:.ctor () emitted at 000000000cb6c990
to 000000000cb6c9e4 (code length 84) [CustomAction]
Method (wrapper runtime-invoke) object object:runtime_invoke_void__this__
(object,intptr,intptr,intptr) emitted at 000000000cb6ca00 to 000000000cb6cb68
(code length 360) [CustomAction]
Method string System.Exception:get_Message () emitted at 000000000cb6cb80 to
000000000cb6cc70 (code length 240) [CustomAction]
Method (wrapper remoting-invoke-with-check) void
Microsoft.Deployment.WindowsInstaller.Session:Log (string) emitted at
000000000cb6cc90 to 000000000cb6cd45 (code length 181) [CustomAction]
Method (wrapper managed-to-native) object
object:__icall_wrapper_mono_thread_get_undeniable_exception () emitted at
000000000c872280 to 000000000c87237d (code length 253) [rundll32.exe]
Method (wrapper remoting-invoke-with-check) void
Microsoft.Deployment.WindowsInstaller.InstallerHandle:Close () emitted at
000000000cb6cd70 to 000000000cb6ce19 (code length 169) [CustomAction]
Method void Microsoft.Deployment.WindowsInstaller.InstallerHandle:Close ()
emitted at 000000000cb6ce50 to 000000000cb6ce9d (code length 77) [CustomAction]
Method (wrapper remoting-invoke-with-check) void
Microsoft.Deployment.WindowsInstaller.InstallerHandle:Dispose () emitted at
000000000cb6cec0 to 000000000cb6cf69 (code length 169) [CustomAction]
Method void Microsoft.Deployment.WindowsInstaller.Session:Dispose (bool)
emitted at 000000000cb6cfa0 to 000000000cb6d07e (code length 222)
[CustomAction]
Method void System.Runtime.InteropServices.Marshal:GetNativeVariantForObject
(object,intptr) emitted at 000000000c872390 to 000000000c872451 (code length
193) [rundll32.exe]
...
011c:fixme:mscoree:corruntimehost_UnloadDomain stub 000000000007BED0
011c:fixme:mscoree:corruntimehost_Stop stub 000000000007BED0
011c:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-localization-l1-2-1.dll" : builtin
011c:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-fibers-l1-1-1.dll" : builtin
011c:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-synch-l1-2-0.dll" : builtin
011c:trace:loaddll:free_modref Unloaded module
L"C:\users\focht\Temp\msi96a0.tmp" : native
...
Method (wrapper runtime-invoke) object
<Module>:runtime_invoke_void_object_intptr_byte (object,intptr,intptr,intptr)
emitted at 000000000c873160 to 000000000c8732f0 (code length 400)
[rundll32.exe] 
...
011c:trace:mscoree:CorExitProcess (0)
011c:trace:mscoree:CLRMetaHost_ExitProcess 0
converting method (wrapper managed-to-native) void System.Environment:Exit
(int)
Method (wrapper managed-to-native) void System.Environment:Exit (int) emitted
at 000000000c873300 to 000000000c8733f5 (code length 245) [rundll32.exe]
...
011c:trace:mscoree:CorExitProcess (0)
011c:trace:mscoree:CLRMetaHost_ExitProcess 0
011c:trace:mscoree:_CorDllMain (000000000CB50000, 0, 0000000000000001)
011c:trace:mscoree:_CorDllMain (000000000CB20000, 0, 0000000000000001)
011c:trace:mscoree:_CorDllMain (000000000C880000, 0, 0000000000000001)
011c:trace:mscoree:_CorDllMain (0000000010000000, 0, 0000000000000001)
011c:trace:mscoree:_CorDllMain (00000000036D0000, 0, 0000000000000001)
011c:trace:mscoree:DllMain (0000000001390000, 0, 0000000000000001)
0108:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-localization-l1-2-1.dll" : builtin
0108:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-fibers-l1-1-1.dll" : builtin
0108:trace:loaddll:free_modref Unloaded module
L"C:\windows\system32\api-ms-win-core-synch-l1-2-0.dll" : builtin
0108:trace:loaddll:free_modref Unloaded module
L"C:\users\focht\Temp\msi96a0.tmp" : native
0024:err:msi:custom_get_thread_return Invalid Return Code 68793744
0024:err:msi:ITERATE_Actions Execution halted, action
L"DeleteInvalidPathRegistryKey" returned 1603
...
--- snip ---

The exception in managed code (NullReferenceException) is also present with
native .NET Framework = ok.

The installer runs a custom action on MSI CA server side which re-launches the
custom action as a separate 'rundll32' process. That custom 'rundll32' CA
server creates an app domain to run managed code. It communicates using pipes
between the MSI custom action server which hosts the CA dll and the 'rundll32'
CA server which hosts the managed bits.

Pretty much this code 'InvokeOutOfProcManagedCustomAction':

https://github.com/wixtoolset/wix3/blob/2ed6d7906c87bb65b75cfa0020e38a1b1dff...

The 'Invalid Return Code 68845504' is the interesting part. The return code is
remoted across the app installer's own custom action server to the MSI custom
action server before app domain shutdown.

https://github.com/wixtoolset/wix3/blob/2ed6d7906c87bb65b75cfa0020e38a1b1dff...

--- snip ---
bool InvokeManagedCustomAction(MSIHANDLE hSession, _AppDomain* pAppDomain,
        const wchar_t* szEntryPoint, int* piResult)
{
        VARIANT vResult;
        ::VariantInit(&vResult);

const bool f64bit = (sizeof(void*) == sizeof(LONGLONG));
        const wchar_t* szMsiAssemblyName   =
L"Microsoft.Deployment.WindowsInstaller";
        const wchar_t* szMsiCAProxyClass   =
L"Microsoft.Deployment.WindowsInstaller.CustomActionProxy";
        const wchar_t* szMsiCAInvokeMethod = (f64bit ? L"InvokeCustomAction64"
: L"InvokeCustomAction32");

_MethodInfo* pCAInvokeMethod;
        if (!GetMethod(hSession, pAppDomain, szMsiAssemblyName,
                szMsiCAProxyClass, szMsiCAInvokeMethod, &pCAInvokeMethod))
        {
                return false;
        }

HRESULT hr;
        VARIANT vNull;
        vNull.vt = VT_EMPTY;
        SAFEARRAY* saArgs = SafeArrayCreateVector(VT_VARIANT, 0, 3);
        VARIANT vSessionHandle;
        vSessionHandle.vt = VT_I4;
        vSessionHandle.intVal = hSession;
        LONG index = 0;
        hr = SafeArrayPutElement(saArgs, &index, &vSessionHandle);
        if (FAILED(hr)) goto LExit;
        VARIANT vEntryPoint;
        vEntryPoint.vt = VT_BSTR;
        vEntryPoint.bstrVal = SysAllocString(szEntryPoint);
        if (vEntryPoint.bstrVal == NULL)
        {
                hr = E_OUTOFMEMORY;
                goto LExit;
        }
        index = 1;
        hr = SafeArrayPutElement(saArgs, &index, &vEntryPoint);
        if (FAILED(hr)) goto LExit;
        VARIANT vRemotingFunctionPtr;
#pragma warning(push)
#pragma warning(disable:4127) // conditional expression is constant
        if (f64bit)
#pragma warning(pop)
        {
                vRemotingFunctionPtr.vt =  VT_I8;
                vRemotingFunctionPtr.llVal = (LONGLONG) (g_fRunningOutOfProc ?
MsiRemoteInvoke : NULL);
        }
        else
        {
                vRemotingFunctionPtr.vt =  VT_I4;
#pragma warning(push)
#pragma warning(disable:4302) // truncation
#pragma warning(disable:4311) // pointer truncation
                vRemotingFunctionPtr.lVal = (LONG) (g_fRunningOutOfProc ?
MsiRemoteInvoke : NULL);
#pragma warning(pop)
        }
        index = 2;
        hr = SafeArrayPutElement(saArgs, &index, &vRemotingFunctionPtr);
        if (FAILED(hr)) goto LExit;

hr = pCAInvokeMethod->Invoke_3(vNull, saArgs, &vResult);

LExit:
        SafeArrayDestroy(saArgs);
        pCAInvokeMethod->Release();

if (FAILED(hr))
        {
                Log(hSession, L"Failed to invoke custom action method. Error
code 0x%X", hr);
                return false;
        }

*piResult = vResult.intVal;
        return true;
}
--- snip ---

--- snip ---
pCAInvokeMethod->Invoke_3(vNull, saArgs, &vResult);
--- snip ---

'Microsoft.Deployment.WindowsInstaller.CustomActionProxy.InvokeCustomAction64'

That invoke is a proxy which then invokes the real managed custom action. The
result code should be propagated from the inner "real" managed method
invocation within 'CustomActionProxy.InvokeCustomAction'

I managed to break into the managed CA and dump the 'vResult' VARIANT after the
"proxy" 'InvokeCustomAction' (after returning from CLR/managed code).

--- snip ---
000000000022F270 09 00 00 00 00 00 00 00 20 DE 18 04
--- snip ---

vt: 0x9 = VT_DISPATCH
value: 0x0418DE20 = 68738592

That's obviously wrong. Debugging dynamically generated code/reflection
invocation is not very fun.

Unfortunately there is another bug with Wine-Mono that prevents a full trace
with 'WINE_MONO_TRACE=all'. Using that causes an internal crash in Wine-Mono
much earlier.

$ sha1sum Stream_Deck_4.9.3.13222.msi 
d54a6df51519c5028eeb27b8f1a577d50a62e375  Stream_Deck_4.9.3.13222.msi

$ du -sh Stream_Deck_4.9.3.13222.msi 
96M    Stream_Deck_4.9.3.13222.msi

$ wine --version
wine-6.4

Regards