http://bugs.winehq.org/show_bug.cgi?id=25853
--- Comment #11 from Anastasius Focht focht@gmx.net 2011-06-13 12:24:24 CDT --- Hello,
well I got a bit further but are being hampered by Wine's context restoration code :| It might be related to the actual problem ...
The obfuscation code chunks access the thread stack not only by explicitly reserving space, e.g. "lea esp,[esp-xx]", "sub esp,xx" ... but directly, e.g: "mov ss:[esp-xx], reg" above the current top of stack without adjusting esp at all. Some branch instructions are done in that way too: "jmp dword ptr ss:[esp-xx]"
The problem is the way Wine restores the context from return of exception handling (... -> RtlRaiseException -> __wine_call_from_regs) for instance when single stepping through the target code.
If you keep an eye of the memory area directly above current top of thread stack you can spot values for cs:eip/eflags and ds being written to the stack before the context return (the selector values are easily to spot). You could also place some watchpoint (hw breakpoint on write access) above top of stack to see it trigger when single stepping.
This context return code overwrites values that used to be "invisible" (previous calls scratch area, future locals, whatever). Also values used by direct "[esp-xx]" addressing without adjusting esp at all are affected.
Another problem arises if code reserves a chunk from top of stack (using sub esp,xx). Now these values are within local stack area and become "visible". If the code doesn't fully initialize the stack it sees these values and might get it wrong.
In Windows, the memory neighbouring the current thread stack top is not touched when single stepping/handling exceptions due to different technical architecture. There is a ring switch so "iretd" can fully restore ss:esp while Wine needs to manually restore frame without a ring switch.
The code responsible for this behaviour (emitted __wine_call_from_regs):
http://source.winehq.org/git/wine.git/blob/bea2be5ccebba363b69e8186a102c8bbe...
I don't know a different solution offhand to fully restore the frame without touching memory neighbouring stack top. Maybe Alexandre comes up with a clever idea ...
Regards