I've been doing some debugging of the exception handler and have come across a problem for which I'm not sure what the best solution is. It involves an application using the ESP for something other than the stack.
The current DIB code uses protection to detect when the application modifies the DIB bits and respond accordingly. It does this by disabling read and/or write access to pages and then catching exceptions when they are thrown so that it can keep the X11 and windows versions of the DIB seporate but reflect changes appropriately (There may be other pieces of Wine which use similar tactics).
The application I am debugging, however, uses the ESP as a counter/memory offset in it's updating of the DIB. While not the best selection of registers, the application relies on ESP not being used for anything else while the DIB updates take place (In Windows this assumption is true).
A made-up example of assembly code is as follows:
mov esp, [0x12345678] ; Save the ESP mov 0x128796, esp ; Set up ESP for the DIB modification mov 0, ebp loop: mov ebp, [sp + bp] ; Write out the byte to the DIB inc ebp sub esp, 0x10000 ; The high word is a counter jns loop mov [0x12345678], esp ; Restore the ESP
Now, one of the first things our exception handler in signal_i386.c does is set up the stack layout to deliver the exception to the application (see the setup_exception function), but in this case ESP definately does not point to the stack (and the application never intended for an exception to occurr). It instead contains a counter in the high word, and a memory offset in the low word. Because of this, there is a double fault which happens and Wine actually Segfaults as opposed to breaking into the debugger.
So, what I'm wondering is if anyone who has experiance with the exception handling code has any thoughts on how this problem can be fixed.
If your interested in actual application code which does this, Alpha Centauri is what I found the initial problem with. You can download the demo at http://www.firaxis.com/downloads/Demo/smac_demo1_1.exe. The assembly code in the application which causes the crash is located starting at around offset 0x0060d954.
As an interesting side-effect, if there are other applications which are using ESP in a "bad" manor, then we may be corrupting memory whenever we generate an exception within Wine. This issue would be very hard to track down, which makes me think that only exceptions which would normally be generated in Windows should cause Wine to modify the application stack.
Comments? Suggestions?
Glenn.