The plot sickens...
Here's the background info. I have a file, power-structure-demo.exe which can be obtained from www.write-brain.com. Uwe has another file, tar2001.exe. Both have been compressed by the commercial Shrinker utility. When these exe's are run under Wine, Wine will spin.
A look at the +all debug messages shows that the app tries to write into its own code area in order to purposely throw an exception. Kind of like this:
void my_function() { try { *(char *)&my_function = *(char *)&my_function; } catch (access violation) { } }
Shrinker was written using Visual C++, since this article at MSJ (http://www.microsoft.com/msj/defaultframe.asp?page=/msj/0197/exception/excep...) describes the code's exception handler perfectly.
The Visual C++ exception handler, which gets control of the app from ntdll.dll, first calls a filter function corresponding to the catch clause to see if the handler associated with the catch will handle the exception. If so, it passes control to that handler. Otherwise it moves to the next catch clause.
Here's the weird thing. The filter function looks at the code which called the exception handler, which is in ntdll.dll. Under Windows, this code is FFD1 (call ecx). The filter function compares that code to the call ecx instruction. At first the only difference between Wine and Windows is this comparison. A register gets decremented if the comparison fails.
However, then the function calls ReadProcessMemory with a handle of -1 (what does that do?) to read 8 bytes, and compares some of the data returned to FFD1, and to 648B25 (which is the beginning of the next instruction after call ecx, mov esp,fs:[00000000]). Things rapidly deviate from there, resulting in Wine spinning and Windows continuing execution.
Given that the application appears to look into ntdll.dll's code to compare it against essentially a magic number, is there any hope of getting Shrinkered exe's to run under Wine?
--Rob