I've been doing some debugging of the exception handler and have come across a problem for which I'm not sure what the best solution is. It involves an application using the ESP for something other than the stack.
The current DIB code uses protection to detect when the application modifies the DIB bits and respond accordingly. It does this by disabling read and/or write access to pages and then catching exceptions when they are thrown so that it can keep the X11 and windows versions of the DIB seporate but reflect changes appropriately (There may be other pieces of Wine which use similar tactics).
The application I am debugging, however, uses the ESP as a counter/memory offset in it's updating of the DIB. While not the best selection of registers, the application relies on ESP not being used for anything else while the DIB updates take place (In Windows this assumption is true).
A made-up example of assembly code is as follows:
mov esp, [0x12345678] ; Save the ESP mov 0x128796, esp ; Set up ESP for the DIB modification mov 0, ebp loop: mov ebp, [sp + bp] ; Write out the byte to the DIB inc ebp sub esp, 0x10000 ; The high word is a counter jns loop mov [0x12345678], esp ; Restore the ESP
Now, one of the first things our exception handler in signal_i386.c does is set up the stack layout to deliver the exception to the application (see the setup_exception function), but in this case ESP definately does not point to the stack (and the application never intended for an exception to occurr). It instead contains a counter in the high word, and a memory offset in the low word. Because of this, there is a double fault which happens and Wine actually Segfaults as opposed to breaking into the debugger.
So, what I'm wondering is if anyone who has experiance with the exception handling code has any thoughts on how this problem can be fixed.
If your interested in actual application code which does this, Alpha Centauri is what I found the initial problem with. You can download the demo at http://www.firaxis.com/downloads/Demo/smac_demo1_1.exe. The assembly code in the application which causes the crash is located starting at around offset 0x0060d954.
As an interesting side-effect, if there are other applications which are using ESP in a "bad" manor, then we may be corrupting memory whenever we generate an exception within Wine. This issue would be very hard to track down, which makes me think that only exceptions which would normally be generated in Windows should cause Wine to modify the application stack.
Comments? Suggestions?
Glenn.
Glenn Wurster wrote:
Comments? Suggestions?
Our current DIB implementation is an ugly hack. We need a real DIB implementation, either as a complete GDI engine in Wine or as an X11 extension.
Some work has been done on a GDI engine in Wine by Transgaming a couple of years ago but it wasn't merged. Then, some weeks ago someone suggested to port the ReactOS GDI engine to Wine.
However, in my opinion implementing a full-fledged GDI engine in Wine would be rather hackish and sub-optimal for several reasons:
- It would need a lot of code and a lot of work. - The X11 engine is captable enough for most of our purposes (except this DIB thingie) so we shouldn't duplicate all that stuff which has been stable and working for decades unless it's absolutely necessary. - Sometimes using the X11 engine and sometimes our DIB engine for the same GDI functions could cause problems for some apps if the used algorithms differ slightly (probably more than a Wine/Windows discrepancy) - we could use our DIB engine for everything but this would eliminate network transparency.
And it would be yet another hack. It would probably work but something like this should *really* be solved on X side using SHM and a yet-to-be-written extension IMHO.
Comments? Suggestions? :)
Felix
Our current DIB implementation is an ugly hack. We need a real DIB implementation, either as a complete GDI engine in Wine or as an X11 extension.
But is it the DIB implementation that is really at fault here or is it the fact that we are adding extra exceptions and can't be sure what the esp will be when they fire, especially since they don't occurr on Windows. The DIB engine does need to know when bitmaps are modified directly by the application if it is to avoid extra work. Another example of a SEH exception is that of the DOS memory initialization (http://www.winehq.org/hypermail/wine-cvs/2005/04/0298.html). It relies now on an access violation to detect when to initalize the DOS and BIOS memory. If ESP was used for something other than the stack on the first access to DOS memory, then we'd experiance the same problem. There may be other examples outside of DIB in the code as well that I'm not aware of.
As far as I can tell, the problem is that we have extra exceptions firing when the application is not expecting them and they would not occurr on Windows. So, the two options are to enforce that Wine never throws exceptions other than those that occurr on windows (and I suspect people in general do not like this restriction for performance reasons) or to fix the exception handler to work in the face of a bad ESP. Perhaps exceptions fired due to wine can use a different stack so that they don't rely on an assumed state of the application?
Some work has been done on a GDI engine in Wine by Transgaming a couple of years ago but it wasn't merged. Then, some weeks ago someone suggested to port the ReactOS GDI engine to Wine.
Yes, I read that. The Transgaming engine that was discussed uses exceptions to detect when the application modifies a DIB, which is the problem we have right now. I don't know how the ReactOS GDI engine does it.
Glenn.
Glenn Wurster wrote:
Our current DIB implementation is an ugly hack. We need a real DIB implementation, either as a complete GDI engine in Wine or as an X11 extension.
But is it the DIB implementation that is really at fault here or is it the fact that we are adding extra exceptions and can't be sure what the esp will be when they fire, especially since they don't occurr on Windows.
Both are - the current DIB implementation wouldn't work without that extra exceptions.
The "real" DIB, used for GDI calls, is stored on X side but the DIB surface, used for direct access, is stored on Wine side and we need to keep them in sync by catching reads / writes to the DIB surface.
We need to get rid of that seperation - either the DIB should be stored fully on X side (which would require to add some features to X) or fully on Wine side (by implementing a GDI renderer in Wine).
And since we already got a full-flegded rendering engine in X11 we already use (and will continue to use unless we wanna drop network transparency) I think the right way would be to add the missing features to X (which would just be XPixmaps of arbitrary depth as we should be able to use SHM for direct memory access).
As far as I can tell, the problem is that we have extra exceptions firing when the application is not expecting them and they would not occurr on Windows. So, the two options are to enforce that Wine never throws exceptions other than those that occurr on windows (and I suspect people in general do not like this restriction for performance reasons) or to fix the exception handler to work in the face of a bad ESP. Perhaps exceptions fired due to wine can use a different stack so that they don't rely on an assumed state of the application?
I dunno if that's possible but it would be yet another hack to workaround limitations of the current DIB hack (which also has other limitations that just *can't* be worked around).
Some work has been done on a GDI engine in Wine by Transgaming a couple of years ago but it wasn't merged. Then, some weeks ago someone suggested to port the ReactOS GDI engine to Wine.
Yes, I read that. The Transgaming engine that was discussed uses exceptions to detect when the application modifies a DIB, which is the problem we have right now.
Does it? I see no reason for that...
I don't know how the ReactOS GDI engine does it.
I'm just talking about the render engine - a "real" GDI implementation doesn't need those exceptions (as you can observe on Windows).
Felix
From: "Felix Nawothnig" felix.nawothnig@t-online.de
We need to get rid of that seperation - either the DIB should be stored fully on X side (which would require to add some features to X) or fully on Wine side (by implementing a GDI renderer in Wine).
You can't have it fully on the X side, since you need direct memory access (non-synchronized I might add) to the DIB data.
The only way is to implement it in Wine.
Dimi Paun wrote:
We need to get rid of that seperation - either the DIB should be stored fully on X side (which would require to add some features to X) or fully on Wine side (by implementing a GDI renderer in Wine).
You can't have it fully on the X side, since you need direct memory access
Direct memory access can be done using X11 SHM, no?
(non-synchronized I might add) to the DIB data.
MSDN says:
Windows NT/2000/XP: You need to guarantee that the GDI subsystem has completed any drawing to a bitmap created by CreateDIBSection before you draw to the bitmap yourself. Access to the bitmap must be synchronized. Do this by calling the GdiFlush function. This applies to any use of the pointer to the bitmap bit values, including passing the pointer in calls to functions such as SetDIBits.
(and I assume they are talking to application developers, not to Wine devs... :)
Or is MSDN lying here?
Felix
From: "Felix Nawothnig" felix.nawothnig@t-online.de
You can't have it fully on the X side, since you need direct memory
access
Direct memory access can be done using X11 SHM, no?
But then you lose network transparency.
(non-synchronized I might add) to the DIB data.
MSDN says:
Windows NT/2000/XP: You need to guarantee that the GDI subsystem has completed any drawing to a bitmap created by CreateDIBSection before you draw to the bitmap yourself. Access to the bitmap must be synchronized. Do this by calling the GdiFlush function. This applies to any use of the pointer to the bitmap bit values, including passing the pointer in calls to functions such as SetDIBits.
(and I assume they are talking to application developers, not to Wine devs... :)
Or is MSDN lying here?
This may be true for NT/2000/XP but Win 3.x/9x didn't have such a requirement, and AFAIK, a lot of apps do non-synchronized access. So they have to support it, and as such, I'm not sure that many people bother to fix their apps.
We really need a DIB engine in Wine.
Dimi Paun wrote:
Direct memory access can be done using X11 SHM, no?
But then you lose network transparency.
I'd think that apps directly accessing the surface memory are either games or do some other kind of realtime graphics so network transparency isn't worth anything there (since we'd have to transfer the whole DIB all the time anyway).
Felix
On Sat, 23 Jul 2005, Felix Nawothnig wrote:
Dimi Paun wrote:
Direct memory access can be done using X11 SHM, no?
But then you lose network transparency.
I'd think that apps directly accessing the surface memory are either games or do some other kind of realtime graphics so network transparency isn't worth anything there (since we'd have to transfer the whole DIB all the time anyway).
I don't remember any names off-hand but I am very sure that I have encountered 'office-style' applications that alternated direct access to the DIB with 'gdi' accesses to it. So yes, preserving network transparency is very much worth it.
Felix Nawothnig wrote:
Dimi Paun wrote:
We need to get rid of that seperation - either the DIB should be stored fully on X side (which would require to add some features to X) or fully on Wine side (by implementing a GDI renderer in Wine).
You can't have it fully on the X side, since you need direct memory access
Direct memory access can be done using X11 SHM, no?
Sharing code with ReactOS is easier if it's not done in the X server.
//jakob
The "real" DIB, used for GDI calls, is stored on X side but the DIB surface, used for direct access, is stored on Wine side and we need to keep them in sync by catching reads / writes to the DIB surface.
We need to get rid of that seperation - either the DIB should be stored fully on X side (which would require to add some features to X) or fully on Wine side (by implementing a GDI renderer in Wine).
What about other instances where Wine generates an exception and Windows does not? Rewriting GDI may be able to solve the exception handler for this case, but really the problem is any exception being generated when ESP does not point exactly to the top of the stack.
As an example. What if I wanted to access BIOS memory and so I did something like the following in an application:
mov 0x12345678, [esp - 4] mov [BIOSMEM], eax mov eax, [esp - 8] sub esp, 8 call somefunc add esp, 8
Correct me if I'm wrong, but the current code does not initialize BIOS memory unless the program requests it. In this case, the exception is going to be raised on the 2nd instruction, causing [esp - 4] to be overwritten with information related to the exception. That's not really what we want.
So, regardless of whether or not GDI uses exceptions to track accesses, other areas of the code use exceptions in a similar manor to detect when an application tries to access memory. They may also use exceptions for other things that I'm not aware of (Perhaps floating point errors?).
GDI is a very good example of the problem of throwing an exception where the application never intended one to be thrown. If ESP is bad we clobber something. GDI, however, is not the only example. If we fix it so that GDI does not throw exceptions, then we have to fix it so that we only throw exceptions when windows throws exceptions. Nothing more, nothing less - otherwise we have the issue of potentially having a bad ESP and clobbering something. The issue of never throwing an exception unless Windows would have is a larger problem and involves more than just GDI code. We can't enforce ESP usage on windows programs, so we have to come to terms with the fact that they may not use the ESP register as we would like.
That is what I am trying to get at. I know that GDI is far from optimal, but there are areas of code other than GDI in which the interaction between ESP and exceptions generated within Wine could pose a problem.
More comments?
Glenn.
Glenn Wurster gwurster@scs.carleton.ca writes:
GDI is a very good example of the problem of throwing an exception where the application never intended one to be thrown. If ESP is bad we clobber something. GDI, however, is not the only example. If we fix it so that GDI does not throw exceptions, then we have to fix it so that we only throw exceptions when windows throws exceptions. Nothing more, nothing less - otherwise we have the issue of potentially having a bad ESP and clobbering something. The issue of never throwing an exception unless Windows would have is a larger problem and involves more than just GDI code. We can't enforce ESP usage on windows programs, so we have to come to terms with the fact that they may not use the ESP register as we would like.
Sure, but that's a very rare case, since messing with ESP means you can't debug the code on Windows either. It would of course be possible to use a separate stack for exceptions, in fact we used to do that, but it has other problems and it's not how Windows does it.
So yes, the general rule is to avoid having exceptions where Windows doesn't. Things like DOS memory could be fixed if we found an app that really breaks because of that, but until we do I wouldn't worry too much about it.
On Sat, Jul 23, 2005 at 10:22:09AM +0200, Alexandre Julliard wrote:
Glenn Wurster gwurster@scs.carleton.ca writes:
GDI is a very good example of the problem of throwing an exception where the application never intended one to be thrown. If ESP is bad we clobber something. GDI, however, is not the only example. If we fix it so that GDI does not throw exceptions, then we have to fix it so that we only throw exceptions when windows throws exceptions. Nothing more, nothing less - otherwise we have the issue of potentially having a bad ESP and clobbering something. The issue of never throwing an exception unless Windows would have is a larger problem and involves more than just GDI code. We can't enforce ESP usage on windows programs, so we have to come to terms with the fact that they may not use the ESP register as we would like.
Sure, but that's a very rare case, since messing with ESP means you can't debug the code on Windows either. It would of course be possible to use a separate stack for exceptions, in fact we used to do that, but it has other problems and it's not how Windows does it.
Actually, you can debug the code on Windows. Visual Studio lets you set breakpoints and single step through it quite nicely. I did that (on an XP computer at least) while examining this problem to see what the Windows behaviour was. I got no complaints. The only difference was that the applications exception handler would not be called. I don't know exactly how Windows does does the exception handling. It would be nice if Wine did not totally crash though.
So yes, the general rule is to avoid having exceptions where Windows doesn't. Things like DOS memory could be fixed if we found an app that really breaks because of that, but until we do I wouldn't worry too much about it.
I'll try and develop a patch for the current GDI engine so at least things work since it sounds like things are a long way off on any sort of new engine. If someone is indeed working on a new GDI engine, than I'd be interested in finding out more and perhaps helping out, although I don't have a lot of time on my hands.
Glenn.
Glenn Wurster wrote:
of new engine. If someone is indeed working on a new GDI engine, than I'd be interested in finding out more and perhaps helping out, although I don't have a lot of time on my hands.
Maybe the ReactOS GDI engine could be used for starters.
" Currently in ReactOS the port of Wine involves building most of the Winecomponents that sit above the core Win32 subsystem of gdi32/user32 and kernel32/ntdll."
-- http://www.winehq.com/?interview=14
//Jakob
-
Alexandre Julliard -
Dimi Paun -
Felix Nawothnig -
Francois Gouget -
Glenn Wurster -
Jakob Eriksson