http://bugs.winehq.org/show_bug.cgi?id=11030
--- Comment #9 from Anastasius Focht focht@gmx.net 2009-05-09 10:53:37 --- Hello,
well after some hours of debugging it seems that Wine's paradigm of having some ntdll API serving only as thin wrappers, calling other ntdll API being the problem.
As already mentioned in other bug reports/analysis Xenocode hooks a considerable amount of native API (ntdll). The trampolines are used to redirect execution to "virtualized" functions (API completely emulated) or to functions that modify caller parameters in place and then use the original OS functionality to carry out actions.
Example of "good" case, run with relay thunks enabled to prevent API from being hooked by Xenocode.
--- snip --- ... 0009:Call KERNEL32.SetEnvironmentVariableW(007b7394 L"__VMAPI",001ead58 L"00810CB8") ret=007f79f7 0009:Ret KERNEL32.SetEnvironmentVariableW() retval=00000001 ret=007f79f7 ... 0009:Call ntdll.NtOpenFile(0032f050,00120089,0032f088,0032f0c8,00000001,00000021) ret=007c2b9e ... 0009:trace:ntdll:NtOpenFile handle=0x32f050 access=00120089 name=L"\??\C:\users\focht\Local Settings\Application Data\Xenocode\XSandbox\Neuro-Programmer 2\2.5.0.0\2008.11.19T21.07" objattr=00000040 root=(nil) sec=(nil) io=0x32f0c8 sharing=00000001 options=00000021 ... 0009:trace:ntdll:NtCreateFile handle=0x32f050 access=00120089 name=L"\??\C:\users\focht\Local Settings\Application Data\Xenocode\XSandbox\Neuro-Programmer 2\2.5.0.0\2008.11.19T21.07" objattr=00000040 root=(nil) sec=(nil) io=0x32f0c8 alloc_size=(nil) attr=00000000 sharing=00000001 disp=1 options=00000021 ea=(nil).0x00000000 ... 0009: create_file( access=00120089, attributes=00000040, sharing=00000001, create=1, options=00000021, attrs=00000000, objattr={rootdir=0000,sd={},name=L""}, filename="/home/focht/.wine/dosdevices/c:/users/focht/Local Settings/Application Data/Xenocode/XSandbox/Neuro-Programmer 2/2.5.0.0/2008.11.19T21.07" ) 0009: create_file() = 0 { handle=003c } 0009:trace:ntdll:NtOpenFile result=0 0009:Ret ntdll.NtOpenFile() retval=00000000 ret=007c2b9e 0009:Call ntdll.NtClose(0000003c) ret=007ea94d 0009: close_handle( handle=003c ) 0009: close_handle() = 0 0009:Ret ntdll.NtClose() retval=00000000 ret=007ea94d ... --- snip ---
In the "bad" case with no relay thunks active some ntdll API get hooked and some not.
NtOpenFile() vs. NtCreateFile(): Calls to Wine's NtOpenFile() go straight to NtCreateFile(). This is something that Xenocode doesn't expect. NtOpenFile() is not hooked while NtCreateFile() is hooked. This results in a recursive call chain, getting the stack eaten up. Same applies to NtQueryAttributesFile() and NtQueryFullAttributesFile().
Moving the code to internal (non exported) function and having both calling this function makes Xenocode happy. It goes much further - only to display error message box which is another problem deserving a separate bug.
---
Move code from NtCreateFile() to internal (non exported) function or just rename it and create another wrapper with original exported name. Have NtOpenFile() and NtCreateFile() call this internal function.
Move code from NtQueryFullAttributesFile() to internal (non exported) function or just rename it and create another wrapper with original exported name. Have NtQueryAttributesFile() and NtQueryFullAttributesFile() call this internal function.
Regards