https://bugs.winehq.org/show_bug.cgi?id=49828
Bug ID: 49828 Summary: MSVC 2019 executables with ASAN enabled fail to start Product: Wine Version: 5.17 Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: -unknown Assignee: wine-bugs@winehq.org Reporter: nyanpasu64@tuta.io Distribution: ---
Recent versions of MSVC allow building Windows .exes with ASAN (it may have already been possible using Clang.): https://devblogs.microsoft.com/cppblog/addresssanitizer-asan-for-windows-wit...
I took an old Win32/MFC application and built it using MSVC ASAN on Windows 10, to try to discover memory bugs on Windows and Wine.
- Program download at https://cdn.discordapp.com/attachments/653168829891084298/754540922972274790... . - Source at https://github.com/nyanpasu64/Dn-FamiTracker/tree/b3ba802f (permalink at https://cdn.discordapp.com/attachments/653168829891084298/754540828520611860... ). - To build, copy cmake_user_begin.cmake.example to cmake_user_begin.cmake, then run CMake in MSVC 32-bit Release mode. It may be possible using Visual Studio's .json file or msbuild, but I haven't tried yet.
On Windows, the program starts without problems. On wine-5.17, when I try to run this program, it crashes during the loading process.
nyanpasu64@dell-arch ~/apps> wine Dn_FamiTracker_ASAN.exe 0024:fixme:ntdll:EtwEventRegister ({6c6c766d-3846-4e6a-a4fb-5b530bd0f3fa}, 00401030, 00884590, 008845A8) stub. 0024:fixme:ntdll:EtwEventSetInformation (deadbeef, 2, 0084A41C, 34) stub ==32==AddressSanitizer CHECK failed: D:\agent_work\3\s\src\vctools\crt\asan\llvm\compiler-rt\lib\asan\asan_rtl.cc:401 "((!asan_init_is_running && "ASan init calls itself!")) != (0)" (0x0, 0x0) <empty stack>
`winedbg Dn_FamiTracker_ASAN.exe` doesn't work, it gets stuck on `0x00000000005c9701 EntryPoint+0xffffffffffffffff in dn_famitracker_asan: ret`.
If I instead `winedbg --gdb Dn_FamiTracker_ASAN.exe`, I get a pile of identical SIGTRAP with different backtraces. These are non-fatal and can be continued. I think these correspond to asan errors that would normally terminate the program, but unfortunately it doesn't print log messages when running in winedbg. I don't know how to fix that.
0x005c9701 in _sanitizer_print_stack_trace () from /home/nyanpasu64/.wine/dosdevices/z:/home/nyanpasu64/apps/Dn_FamiTracker_ASAN.exe
The last SIGTRAP is different: 0x7bc52379 in DbgBreakPoint@0 () from /home/nyanpasu64/.wine/dosdevices/c:/windows/syswow64/ntdll.dll
Afterwards the program starts. (It used to crash, but works with lib32-libpulse and lib32-mpg123 installed, unsure if that's why.)
https://bugs.winehq.org/show_bug.cgi?id=49828
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords| |download, source URL| |https://cdn.discordapp.com/ | |attachments/653168829891084 | |298/754540922972274790/Dn_F | |amiTracker_ASAN.exe
https://bugs.winehq.org/show_bug.cgi?id=49828
winetaste@gmx.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |winetaste@gmx.net
https://bugs.winehq.org/show_bug.cgi?id=49828
Roman Pišl rpisl@seznam.cz changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |rpisl@seznam.cz
--- Comment #1 from Roman Pišl rpisl@seznam.cz --- Maybe related: bug 50993.
https://bugs.winehq.org/show_bug.cgi?id=49828
Anastasius Focht focht@gmx.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |focht@gmx.net Ever confirmed|0 |1 Status|UNCONFIRMED |NEW URL|https://cdn.discordapp.com/ |https://web.archive.org/web |attachments/653168829891084 |/20211230172933/https://cdn |298/754540922972274790/Dn_F |.discordapp.com/attachments |amiTracker_ASAN.exe |/653168829891084298/7545409 | |22972274790/Dn_FamiTracker_ | |ASAN.exe
--- Comment #2 from Anastasius Focht focht@gmx.net --- Hello folks,
confirming. I assume you use LLVM MinGW to build Wine.
The address sanitizer init code hooks several API in TLS callback (before entry point) by trying different hooking techniques.
https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d...
https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d...
--- snip --- bool OverrideFunction( uptr old_func, uptr new_func, uptr *orig_old_func) { #if !SANITIZER_WINDOWS64 if (OverrideFunctionWithDetour(old_func, new_func, orig_old_func)) return true; #endif if (OverrideFunctionWithRedirectJump(old_func, new_func, orig_old_func)) return true; if (OverrideFunctionWithHotPatch(old_func, new_func, orig_old_func)) return true; if (OverrideFunctionWithTrampoline(old_func, new_func, orig_old_func)) return true; return false; } --- snip ---
ASAN hook via 'OverrideFunctionWithDetour' technique on Wine's 'kerne32.RaiseException':
--- snip --- ... 7B60CE89 | 90 | nop | 7B60CE8A | 90 | nop | 7B60CE8B | E9 F0D4FC84 | jmp <dn_famitracker_asan.___asan_wrap_RaiseExc... | <kernel32.RaiseException>: 7B60CE90 | EB F9 | jmp kernel32.7B60CE8B | 7B60CE92 | 55 | push ebp | 7B60CE93 | 8BEC | mov ebp,esp | 7B60CE95 | 5D | pop ebp | 7B60CE96 | FF25 DCA9637B | jmp dword ptr ds:[&_RaiseException@16] | 7B60CE9C | 90 | nop | 7B60CE9D | 90 | nop | 7B60CE9E | 90 | nop | 7B60CE9F | 90 | nop | ... --- snip ---
'OverrideFunctionWithDetour' expects a hotpatch prolog (0x8B,0xFF) and enough padding (5 x nop or 5 x int3) before. You can check the linked llvm source code for details.
kernel32 API which import from kernelbase get their hotpatch thunks automatically generated which allows the first hooking technique 'OverrideFunctionWithDetour' to succeed here.
https://source.winehq.org/git/wine.git/blob/58b1dc9f47310ba868cbcb3083de1b42...
Now to the problematic one(s) ...
ASAN can't hotpatch 'kernelbase.RaiseException' with any hooking technique.
(1) OverrideFunctionWithDetour
* no hotpatch entry * not enough preceding padding
(2) OverrideFunctionWithRedirectJump
* no relative jump at entry
(3) OverrideFunctionWithHotPatch
* first instruction needs to be at least two-byte opcode * not enough preceding padding
(4) OverrideFunctionWithTrampoline
* it can't decode the two-byte 0x89,0xe5 'mov ebp, esp' instruction that llvm/mingw emits
It only handles the 0x8B,0xec form:
--- snip --- case 0xEC8B: // 8B EC : mov ebp, esp --- snip ---
https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d...
But even then it wouldn't be able to decode another one:
0x83,0xE4,0xFC 'and esp,FFFFFFFC'
Note, it requires 5 bytes to be copied into the trampoline because they get overwritten by the jump, hence 4 instructions need to be decoded (rounded up).
--- snip --- ... 7B0103BC | C2 0400 | ret 4 | 7B0103BF | 90 | nop | kernelbase._RaiseException@16: 7B0103C0 | 55 | push ebp | 7B0103C1 | 89E5 | mov ebp,esp | 7B0103C3 | 56 | push esi | 7B0103C4 | 83E4 FC | and esp,FFFFFFFC | 7B0103C7 | 83EC 50 | sub esp,50 | 7B0103CA | 8B45 10 | mov eax,dword ptr ss:[ebp+10] | 7B0103CD | 8B4D 0C | mov ecx,dword ptr ss:[ebp+C] | 7B0103D0 | 8B55 08 | mov edx,dword ptr ss:[ebp+8] | 7B0103D3 | 891424 | mov dword ptr ss:[esp],edx | 7B0103D6 | 83E1 01 | and ecx,1 | 7B0103D9 | 894C24 04 | mov dword ptr ss:[esp+4],ecx | 7B0103DD | C74424 08 00000000 | mov dword ptr ss:[esp+8],0 | ... --- snip ---
Apart from fixing instruction decoding issues, having LLVM MinGW implementing 'ms_hook_prologue' attribute would have helped here as well but it was rather short-lived:
https://github.com/llvm/llvm-project/issues/10584
You can test/work around by running the app with +relay which uses relay thunks that have hotpatch prolog:
https://source.winehq.org/git/wine.git/blob/58b1dc9f47310ba868cbcb3083de1b42...
====
@Roman (comment #1)
--- quote --- Maybe related: bug 50993. --- quote ---
Well, if you turn bug reports into meta-issues a la "make ASAN executables work", then of course they are related. But that's not really good practice.
Stable download link via Internet Archive:
https://web.archive.org/web/20211230172933/https://cdn.discordapp.com/attach...
$ sha1sum Dn_FamiTracker_ASAN.exe 3ab8880ae1d536363353e753dc97b1fee6b8cc41 Dn_FamiTracker_ASAN.exe
$ du -sh Dn_FamiTracker_ASAN.exe 5.6M Dn_FamiTracker_ASAN.exe
Regards
https://bugs.winehq.org/show_bug.cgi?id=49828
Jacek Caban jacek@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jacek@codeweavers.com
--- Comment #3 from Jacek Caban jacek@codeweavers.com --- It should work better when clang is used in MSVC mode. clang emits 'movl %edi,%edi' prologue for RaiseException for me. If I read it right:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/X86/X86MCInst...
the decision is made based on target, so I don't see any way to enable that for llvm-mingw without patching LLVM.
Also LLD seems to supports /functionpadmin option, which we probably should use (but it may not be needed here).
If you have llvm-mingw installed, then it will be used by consigure to build Wine with mingw target. To force MSVC target, you may use something like: configure CROSSCC=clang (it may still use clang shipped with llvm-mingw, just in a different way)
https://bugs.winehq.org/show_bug.cgi?id=49828
Jacek Caban jacek@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |martin@martin.st
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #4 from Roman Pišl rpisl@seznam.cz ---
@Roman (comment #1)
--- quote --- Maybe related: bug 50993. --- quote ---
Well, if you turn bug reports into meta-issues a la "make ASAN executables work", then of course they are related. But that's not really good practice.
Agree. Also the title could be helpful, is this bug definitely about "MSVC 2019 32bit executables with ASAN enabled fail to start on Wine compiled with llvm-mingw"?
https://bugs.winehq.org/show_bug.cgi?id=49828
Anastasius Focht focht@gmx.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|MSVC 2019 executables with |MSVC 2019 32-bit |ASAN enabled fail to start |executables with ASAN | |enabled fail to start when | |Wine is built with LLVM | |MinGW due to missing | |hotpatch prolog | |(kernelbase.dll exports et | |al.)
--- Comment #5 from Anastasius Focht focht@gmx.net --- Hello folks,
@Jacek comment #3
--- quote --- If you have llvm-mingw installed, then it will be used by consigure to build Wine with mingw target. To force MSVC target, you may use something like: configure CROSSCC=clang (it may still use clang shipped with llvm-mingw, just in a different way) --- quote ---
It didn't work for me, clean build but the API entry still doesn't start with the 2-byte instruction.
I'm using Martin's 20211002 release with LLVM stable 13.0.0
https://github.com/mstorsjo/llvm-mingw/releases/tag/20211002
https://github.com/mstorsjo/llvm-mingw/releases/download/20211002/llvm-mingw...
'config.log'
--- snip --- CROSSCC='clang' CROSSCFLAGS='-g -O2' CROSSDEBUG='pdb' ... CROSSTARGET='i686-windows' --- snip ---
@Roman comment #4
I've adjusted the title to be more clear. Sure, there could be more bug reports for each of the issues:
(1) Make 'MSVC' mode default in Wine when LLVM MinGW is used (this one?).
(2) Add a bunch of missing instruction decodings to improve ASAN hooking (from bug 50993 and bug 49828 = this one) -> track upsteam project with it and resolve 'NOT OUR BUG' later.
(3) Fixing specific Wine API to add missing hotpatch decoration = bug 50993?
(4) Bug 50993 add link to bug 50735 because one of the issues mentioned was fixed there ("ntdll: Don't pretend that the whole address space is reserved on non-i386.").
Regards
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #6 from Martin Storsjö martin@martin.st --- Hmm, I'm trying to read up on what's going on here... So if I've read correctly:
- One might consider whether the MSVC-mode specific quirk in LLVM's codegen for x86_32 (https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/X86/X86MCInst...) should be enabled for mingw mode too? How does GCC behave with respect to to this quirk (and how does GCC built Wine behave with regards to this bug?)
- Prefer MSVC mode over mingw mode, when both modes are available for Wine to use. Overall this could work (and I could see why it could be preferred, as that's a more pristine toolchain/environment for Wine to use). But specifically for arm32 mode, one can't build Wine with Clang in MSVC mode, because Wine doesn't provide the compiler_rt builtins that are needed there (it's around a dozen of functions needed; integer divisions, conversions between 64 bit integers and float/double). But with an exception for 32 bit arm, I'd be fine with such a preference change.
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #7 from Jacek Caban jacek@codeweavers.com --- I think that current logic is fine. The usual use case for MSVC mode is to use distro provided clang - it should be as simple as installing default clang with the package manager. When llvm-mingw is installed, it's presumably for a reason and defaulting to it seems fine. It then falls into the same checks as GCC and we use it like that.
It came out due to a problem with making functions hotpatchable in clang-based builds (ideally both mingw and msvc, IMHO). GCC provides __ms_hook_prologue__ for that. At some point Chip implemented that for that attribute for clang, but it was later reverted. There were later efforts to implement it in MSVC compatible way. Now that I looked again, it seems that it's not yet complete on LLVM side, but some parts landed: https://reviews.llvm.org/D49366 https://reviews.llvm.org/D81301
https://bugs.winehq.org/show_bug.cgi?id=49828
Neko-san nekoNexus@protonmail.ch changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |nekoNexus@protonmail.ch
https://bugs.winehq.org/show_bug.cgi?id=49828
Bernhard Übelacker bernhardu@mailbox.org changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |bernhardu@mailbox.org
--- Comment #8 from Bernhard Übelacker bernhardu@mailbox.org --- Tried to place a merge request with llvm-project, which contains at least the `89 E5` and `83 E4 XX` instructions. https://github.com/llvm/llvm-project/pull/113085
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #9 from Roman Pišl rpisl@seznam.cz --- Hi Bernhard, thank you for pushing this upstream. I tried your llvm branch with Wine 9.21 and initialization of asan fails with:
AddressSanitizer: CHECK failed: asan_win.cpp:256 "((tsd_key_inited)) != (0)" (0x0, 0x0) (tid=488)
Are additional patches for Wine needed?
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #10 from Bernhard Übelacker bernhardu@mailbox.org --- (In reply to Roman Pišl from comment #9)
Hi Bernhard, thank you for pushing this upstream. I tried your llvm branch with Wine 9.21 and initialization of asan fails with:
AddressSanitizer: CHECK failed: asan_win.cpp:256 "((tsd_key_inited)) != (0)" (0x0, 0x0) (tid=488)
Are additional patches for Wine needed?
In following merge request this also came up, so I just linking there. Updated the last answer to this comment with my latest version:
https://gitlab.winehq.org/wine/wine/-/merge_requests/6791#note_86894
Unfortunately I cannot remember seeing this message. But most time I used a single instrumented and renamed exe file in a regular prefix.
https://bugs.winehq.org/show_bug.cgi?id=49828
--- Comment #11 from Roman Pišl rpisl@seznam.cz --- (In reply to Bernhard Übelacker from comment #10)
In following merge request this also came up, so I just linking there. Updated the last answer to this comment with my latest version:
https://gitlab.winehq.org/wine/wine/-/merge_requests/6791#note_86894
Unfortunately I cannot remember seeing this message. But most time I used a single instrumented and renamed exe file in a regular prefix.
Setting windows_hook_rtl_allocators=true helps and leads to a similar problem mentioned in MR. I will definitely follow this. Very interesting. Thanks!
-
WineHQ Bugzilla