Hello all,
I was working on a test for MSI type 1 custom actions, i.e. functions exported from a given DLL and called by the install script. I've tested both with an independent file and in the Wine test environment, and I've found that I can successfully export functions from the executable itself with DECLSPEC_EXPORT, and MSI will successfully execute the function, as the attached patch demonstrates.
The function returns ERROR_SUCCESS, and changing this to other values gets reflected in the install's error code appropriately. However, as soon as I add a function call to the custom action, the install fails with ERROR_INSTALL_FAILURE, and the function is never called—making it look, in all probability, like a page fault occurred. This is the case when compiling with mingw-w64, both inside the Wine test environment and in an independent file.
Notably I have no trouble calling DLLs taken from real MSI packages in this way. I also have no trouble calling the function if I use GetModuleHandle() and GetProcAddress() directly, nor do I run into any trouble whatsoever running the custom action on Linux. It is specifically this environment—running this function as part of a custom action, on Windows.
Does anyone know, or have a hunch, as to what the problem might be here? For what it's worth, it's known that native MSI uses a separate process to execute custom actions, although I don't know how this would lend any clue to what's going wrong.
--Zeb