On Thu, 2017-10-12 at 23:09 -0500, Zebediah Figura wrote:
I was working on a test for MSI type 1 custom actions, i.e. functions exported from a given DLL and called by the install script. I've tested both with an independent file and in the Wine test environment, and I've found that I can successfully export functions from the executable itself with DECLSPEC_EXPORT, and MSI will successfully execute the function, as the attached patch demonstrates.
The function returns ERROR_SUCCESS, and changing this to other values gets reflected in the install's error code appropriately. However, as soon as I add a function call to the custom action, the install fails with ERROR_INSTALL_FAILURE, and the function is never called—making it look, in all probability, like a page fault occurred. This is the case when compiling with mingw-w64, both inside the Wine test environment and in an independent file.
Notably I have no trouble calling DLLs taken from real MSI packages in this way. I also have no trouble calling the function if I use GetModuleHandle() and GetProcAddress() directly, nor do I run into any trouble whatsoever running the custom action on Linux. It is specifically this environment—running this function as part of a custom action, on Windows.
Does it fail on any function call or just calls back into the test execu table? Can you call a kernel API? Perhaps it's some sort of protection scheme.