[Bug 53032] New: winedevice.exe segfaults on exit (Segmentation fault) in Wine 7.9
https://bugs.winehq.org/show_bug.cgi?id=53032 Bug ID: 53032 Summary: winedevice.exe segfaults on exit (Segmentation fault) in Wine 7.9 Product: Wine Version: 7.9 Hardware: x86 OS: Linux Status: UNCONFIRMED Severity: major Priority: P2 Component: wineserver Assignee: wine-bugs(a)winehq.org Reporter: aros(a)gmx.com Distribution: --- This is a regression or maybe it's down to GCC 12.1 - I don't know. When exiting wine, winedevice.exe segfaults all the time. I cannot get a bt and my attempts of using gdb have been futile. (gdb) c Continuing. [LWP 495578 exited] [LWP 495577 exited] [LWP 495576 exited] [LWP 495573 exited] Thread 5 "winedevice.exe" received signal SIGSEGV, Segmentation fault. [Switching to LWP 495599] 0x7e68a2bf in ?? () (gdb) bt #0 0x7e68a2bf in ?? () Backtrace stopped: Cannot access memory at address 0x1fe Wine is built using: -O2 -march=pentium-m -m32 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Artem S. Tashkinov <aros(a)gmx.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|winedevice.exe segfaults on |winedevice.exe segfaults on |exit (Segmentation fault) |exit (Segmentation fault) |in Wine 7.9 |when built by GCC 12.1 | |(Fedora 36) --- Comment #1 from Artem S. Tashkinov <aros(a)gmx.com> --- This is an issue with GCC 12.1 in Fedora 36. I've just compiled Wine 7.8 and the issue persists. [????] Process 577885 (winedevice.exe) of user 1000 dumped core. Stack trace of thread 577888: #0 0x00000000f7d51e7d n/a (n/a + 0x0) #1 0x00000000f7d528f7 n/a (n/a + 0x0) ELF object binary architecture: Intel 80386 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #2 from Artem S. Tashkinov <aros(a)gmx.com> --- The issue is reproducible with -march=i686 as well. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #3 from Artem S. Tashkinov <aros(a)gmx.com> --- GCC 12.0.1 is also affected. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #4 from Artem S. Tashkinov <aros(a)gmx.com> --- I've built Wine with GCC 11.3 and the bug is no more. No idea who's responsible - it could be GCC, it could be Wine. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Rémi Bernon <rbernon(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rbernon(a)codeweavers.com --- Comment #5 from Rémi Bernon <rbernon(a)codeweavers.com> --- Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A log with WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb could be useful. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #6 from Artem S. Tashkinov <aros(a)gmx.com> --- (In reply to Rémi Bernon from comment #5)
Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A log with WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay, +hid,+wineusb could be useful.
I doubt this can work as it enables logging for wine, not for its dependent subprocess. OK, I've run WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb /tmp/wine-7.9/wine notepad &> /tmp/wine.log the last line is all good and fine: 0034:trace:service:main services.exe exited with code 0 Except a couple of seconds later when wineserver decides to shutdown there's a crash. I've no idea where to get logs for it. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #7 from Artem S. Tashkinov <aros(a)gmx.com> --- I've even tried running winedevice.exe directly, again, wine exits successfully and then a few seconds later winedevice.exe segfaults: WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb /tmp/wine-7.9/wine winedevice.exe &> /tmp/wine.log 003c:trace:service:main services.exe exited with code 0 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #8 from Artem S. Tashkinov <aros(a)gmx.com> --- Here's how I build Wine: export CFLAGS="-O2 -pipe -m32 -march=pentium-m" export CXXFLAGS="$CFLAGS" export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu" ./configure --prefix=/opt/wine --disable-tests && make -j16 This is 100% reproducible under Fedora 36. Maybe it's down to my Ryzen 7 5800X CPU and you have an Intel one, so you don't hit the issue. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #9 from Artem S. Tashkinov <aros(a)gmx.com> --- I've tried to $ gdb wine attach X $ gdb wine attach Y (twice because there are two winedevice.exe processes) Then I exit whatever wine application I've launched, e.g. notepad.exe and then I see: (gdb) bt #0 0xf7fe0142 in ?? () Backtrace stopped: Cannot access memory at address 0x31f86c (gdb) bt #0 0xf7f0f142 in ?? () Backtrace stopped: Cannot access memory at address 0x31f86c $ ps ax | grep wine 136801 pts/2 Sl+ 0:00 gdb ./wine 137098 ? Zsl 0:00 [winedevice.exe] <defunct> 137106 ? Zsl 0:00 [winedevice.exe] <defunct> 137394 pts/4 Sl+ 0:00 gdb ./wine Zombies! No idea what's going on. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #10 from Austin English <austinenglish(a)gmail.com> --- (In reply to Artem S. Tashkinov from comment #6)
(In reply to Rémi Bernon from comment #5)
Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A log with WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay, +hid,+wineusb could be useful.
I doubt this can work as it enables logging for wine, not for its dependent subprocess.
Untested, but you could try exporting WINEDEBUG before starting any wine process, in theory the dependent processes would get it from the environment. (In reply to Artem S. Tashkinov from comment #8)
Here's how I build Wine:
export CFLAGS="-O2 -pipe -m32 -march=pentium-m" export CXXFLAGS="$CFLAGS" export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu"
Why the mix of optimization flags (01/02)? Does omitting march change anything? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #11 from Rémi Bernon <rbernon(a)codeweavers.com> --- As you managed to get gdb attached, you can loading this file in gdb with https://github.com/rbernon/wine/raw/develop/tools/gdbinit.py to maybe get symbols. You can load it in gdb with `source <filename.py>` and then load the symbols with a new load-symbol-files command. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #12 from Artem S. Tashkinov <aros(a)gmx.com> --- (In reply to Austin English from comment #10)
(In reply to Artem S. Tashkinov from comment #6)
(In reply to Rémi Bernon from comment #5)
Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A log with WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay, +hid,+wineusb could be useful.
I doubt this can work as it enables logging for wine, not for its dependent subprocess.
Untested, but you could try exporting WINEDEBUG before starting any wine process, in theory the dependent processes would get it from the environment.
The problem is winedevice.exe is a background process you do not start, I've no idea how to catch the debug output in this case, it's gonna be output to /dev/null - it's never linked to console :(
(In reply to Artem S. Tashkinov from comment #8)
Here's how I build Wine:
export CFLAGS="-O2 -pipe -m32 -march=pentium-m" export CXXFLAGS="$CFLAGS" export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu"
Why the mix of optimization flags (01/02)? Does omitting march change anything?
LD doesn't have the -O2 optimization level. Will try to build without -march. Please wait. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #13 from Artem S. Tashkinov <aros(a)gmx.com> --- Created attachment 72415 --> https://bugs.winehq.org/attachment.cgi?id=72415 winedebug.log.xz OK, actually exporting the variable made wine spill debug messages to console and I've managed to save everything but I don't see any crash. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #14 from Artem S. Tashkinov <aros(a)gmx.com> --- Removing -march=pentium-m has fixed the issue. Looks like this is a GCC bug. I really don't know how to proceed further. GCC might miscompile anything including the main `wine` binary, libwine.so, winedevice.exe - I've no idea where to look. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Artem S. Tashkinov <aros(a)gmx.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://gcc.gnu.org/bugzill | |a/show_bug.cgi?id=105700 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Artem S. Tashkinov <aros(a)gmx.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|winedevice.exe segfaults on |winedevice.exe segfaults on |exit (Segmentation fault) |exit when built by GCC |when built by GCC 12.1 |>=12.0 and -march=pentium-m |(Fedora 36) | -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #15 from Artem S. Tashkinov <aros(a)gmx.com> --- The crash is in ntdll.dll.so but that's a huge library: $ ls -la ntdll*so* -rwxr-xr-x. 1 root root 926240 May 23 09:42 ntdll.dll.so.o2 -rwxr-xr-x. 1 root root 950816 May 23 09:46 ntdll.dll.so.pentiumm ~25KB of extra binary code. Here's another tidbit: even there's no crash (i.e. GCC 12: -O2 -m32, GCC 11.3: -O2 -march=pentium-m -m32), there's still something wrong when exiting Wine applications: 00b0:err:virtual:virtual_setup_exception nested exception on signal stack in thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0 I feel like there's a coding error which gets worse when using -march=pentium-m. I still have no clue how to debug all of this. Considering how easy it's to reproduce ( see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105700#c2 ), would be great if Wine hackers chimed in instead. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Bernhard Übelacker <bernhardu(a)mailbox.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bernhardu(a)mailbox.org --- Comment #16 from Bernhard Übelacker <bernhardu(a)mailbox.org> --- Created attachment 72435 --> https://bugs.winehq.org/attachment.cgi?id=72435 debugging-with-rr.txt I tried if I could reproduce it inside a qemu VM. And I guess I succeeded in getting at least the "stack overlow" variant. This combined with rr-debugger and an old but modified version of gdbinit.py I reached in winedevice.exe a segfault. (rr) bt #0 0x7e154db0 in SetupCloseLog () #1 0x7bc54b16 in call_dll_entry_point () #2 0x7bc59381 in MODULE_InitDLL () ... Some notes, the backtrace and singlestepping through SetupCloseLog showing the esp register are in attached file. It is a build without mingw compilers and from flags without much debug information, but function SetupCloseLog is quite short with just a few function calls. As far as I see the esp register has "just" the wrong value when the ret instruction is reached. Unfortunately I cannot point exactly to a single instruction which might be wrong. Maybe someone with better assembly knowledge can have a look? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #17 from Bernhard Übelacker <bernhardu(a)mailbox.org> --- I redid the steps inside a Fedore 35 VM, with gcc 11.3.1. There I can confirm the assembly for SetupCloseLog is equal. (Except two instructions swapped before the EnterCriticalSection.) But there the $esp value at the ret instruction is the same as at SetupCloseLog+0. But still a segfault is observable because the stack memory holding the return address for the ret instruction got overwritten here: (rr) stepi 0xf7b93f10 in __wine_syscall_dispatcher () 1: x/i $pc => 0xf7b93f10 <__wine_syscall_dispatcher+20>: mov %esp,0xc(%ecx) 3: *(void**)0x165ef4c = (void *) 0x7d8c9f9c <DllMain+124> (rr) bt #0 0xf7b93f10 in __wine_syscall_dispatcher () #1 0x7b087541 in CloseHandle () #2 0x7d8b9b6f in SetupCloseLog () #3 0x7d8c9f9c in DllMain () #4 0x7bc55216 in call_dll_entry_point () #5 0x7bc59ae1 in MODULE_InitDLL () #6 0x7bc5a07f in process_detach () ... (rr) stepi 0xf7b93f13 in __wine_syscall_dispatcher () 1: x/i $pc => 0xf7b93f13 <__wine_syscall_dispatcher+23>: mov %cs,0x10(%ecx) 3: *(void**)0x165ef4c = (void *) 0x165eeec @Artem: If you build with MinGW cross compilers, which versions do you have installed? @All: Is a MinGW-less build still considered fully supported? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #18 from Artem S. Tashkinov <aros(a)gmx.com> --- (In reply to Bernhard Übelacker from comment #17)
@Artem: If you build with MinGW cross compilers, which versions do you have installed?
I use GCC only. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #19 from Rafał Mużyło <galtgendo(a)o2.pl> ---
00b0:err:virtual:virtual_setup_exception nested exception on signal stack in thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0
As I've mentioned today in a couple other places, this is unrelated. I'm getting this relatively often as a random (not happening every time) crash on exit in various apps with gcc 11.2. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Zeb Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12(a)gmail.com --- Comment #20 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Rafał Mużyło from comment #19)
00b0:err:virtual:virtual_setup_exception nested exception on signal stack in thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0
As I've mentioned today in a couple other places, this is unrelated. I'm getting this relatively often as a random (not happening every time) crash on exit in various apps with gcc 11.2.
It's not unrelated, it's exactly what this bug is. I wouldn't be surprised if it's the exact same bug, despite the gcc version. It's likely that Wine is doing something undefined, that only works by chance. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #21 from Bernhard Übelacker <bernhardu(a)mailbox.org> --- This seems to be a more reliable callstack without the gdbinit.py magic, but instead just loading the loader/wine executable. I guess the important line is the free_delay_imports frame. As this is a non-PE build, might this stack get damaged by the stack unwinding - and therefore the wine call is no longer working? Following is all from one thread, just before the segfault: (rr) file /home/benutzer/wine-build/loader/wine (rr) bt #0 0xf7b7aaf2 in signal_exit_thread () from dlls/ntdll/ntdll.so #1 0xf7b95fdf in abort_thread () at include/winnt.h:2165 #2 0xf7b777ce in wait_select_reply () at dlls/ntdll/unix/server.c:324 #3 0xf7b7a459 in server_select () at dlls/ntdll/unix/server.c:639 #4 0xf7b7a508 in server_wait () at dlls/ntdll/unix/server.c:666 #5 0xf7b8663c in NtWaitForMultipleObjects () at dlls/ntdll/unix/sync.c:1428 #6 0xf7b7ac09 in __wine_syscall_dispatcher () from dlls/ntdll/ntdll.so #7 0x00000001 in ?? () #8 0x0155fd50 in ?? () #9 0xf7bc1340 in ?? () from dlls/ntdll/ntdll.so Backtrace stopped: previous frame inner to this frame (corrupt stack?) (rr) when Current event: 366461 ... (rr) bt #0 0xf7d1f610 in __longjmp_cancel () from /lib/libc.so.6 #1 0xf7d1f572 in __libc_longjmp () from /lib/libc.so.6 #2 0xf7d7b5a2 in unwind_stop () from /lib/libc.so.6 #3 0x7e1b439a in _Unwind_ForcedUnwind_Phase2 () at libgcc/unwind.inc:171 #4 0x7e1b46be in _Unwind_ForcedUnwind () at libgcc/unwind.inc:218 #5 0xf7d7b6fb in __pthread_unwind () from /lib/libc.so.6 #6 0xf7d738af in pthread_exit () from /lib/libc.so.6 #7 0xf7b929ed in pthread_exit_wrapper () at dlls/ntdll/unix/thread.c:1052 #8 0xf7b7aafa in signal_exit_thread () from dlls/ntdll/ntdll.so #9 0x0165f338 in ?? () #10 0xf7d723bd in start_thread () from /lib/libc.so.6 #11 0xf7df953a in clone () from /lib/libc.so.6 (rr) when Current event: 366462 ... (rr) bt #0 0xf7b7ab09 in __wine_syscall_dispatcher () from dlls/ntdll/ntdll.so #1 0x7bc3544c in NtClose () from dlls/ntdll/ntdll.dll.so #2 0x7b08a1a1 in CloseHandle () at dlls/kernelbase/process.c:421 #3 0x7e154ddf in SetupCloseLog () at dlls/setupapi/misc.c:1661 #4 0x7e164949 in DllMain () at dlls/setupapi/setupcab.c:453 #5 0x7bc54b16 in call_dll_entry_point () from dlls/ntdll/ntdll.dll.so #6 0x7bc59381 in MODULE_InitDLL () at dlls/ntdll/loader.c:1568 #7 0x7bc5992f in process_detach () at dlls/ntdll/loader.c:1714 #8 0x7bc59be5 in LdrUnloadDll () at dlls/ntdll/loader.c:3850 #9 0x7bc59c87 in LdrUnloadDll () at dlls/ntdll/loader.c:3833 #10 0x7b056009 in FreeLibrary () at dlls/kernelbase/loader.c:265 #11 0x7e5d8091 in free_delay_imports () at dlls/winecrt0/delay_load.c:78 #12 0xf7f04df2 in _dl_fini () at dl-fini.c:142 #13 0xf7d223a3 in __run_exit_handlers () from /lib/libc.so.6 #14 0xf7d224f7 in exit () from /lib/libc.so.6 #15 0xf7d7242e in start_thread () from /lib/libc.so.6 #16 0xf7df953a in clone () from /lib/libc.so.6 (rr) when Current event: 366462 What I do not yet understand why the free_delay_imports seems not to get called at all at my usual Debian system. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #22 from Rafał Mużyło <galtgendo(a)o2.pl> --- Little note: I'm getting those errors despite being on on a mingw build. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #23 from Bernhard Übelacker <bernhardu(a)mailbox.org> --- glibc checks for the thread being the last thread and then executes the exit() call I see in the backtrace. Maybe some race condition, which thread is the last one? https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/pthread_create.c;h=6c6... (This is 2.33, similar to Debians version. There were some changes to start_thread between 2.33 and 2.34 which is used in Fedora.) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Ken Sharp <imwellcushtymelike(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |download, source -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #24 from Artem S. Tashkinov <aros(a)gmx.com> --- As mentioned earlier with GCC 11.3 there's still something wrong: 0058:err:virtual:virtual_setup_exception stack overflow 260 bytes in thread 0058 addr 0x7bc7f4ed stack 0x1220efc (0x1220000-0x1221000-0x1320000) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #25 from Artem S. Tashkinov <aros(a)gmx.com> --- winedevice.exe for Wine 7.14 now segfaults even when being built with GCC 11.3 :-( Process 247198 (winedevice.exe) of user 1000 dumped core. Stack trace of thread 247203: #0 0x00000000f7cdac0d n/a (n/a + 0x0) #1 0x00000000f7cdb6bc n/a (n/a + 0x0) ELF object binary architecture: Intel 80386 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Kevin Puetz <PuetzKevinA(a)JohnDeere.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |PuetzKevinA(a)JohnDeere.com --- Comment #26 from Kevin Puetz <PuetzKevinA(a)JohnDeere.com> --- I'm seeing a similar crash-on-exit in our aarch64 builds of wine-7.0: winedevice.exe and rpcss.exe are both crashing on exit with a similar stack trace: This is gcc 9.2, on a yocto sysroot. #0 0x0000ffff9428db34 in handle_syscall_fault (rec=0x67f9ed08, context=0x67f9ee20) at ../wine-7.0/dlls/ntdll/unix/unix_private.h:70#1 segv_handler (signal=<optimized out>, siginfo=<optimized out>, sigcontext=0x67f9ee20) at ../wine-7.0/dlls/ntdll/unix/signal_arm64.c:872 #2 <signal handler called> #3 ULongToHandle (ul=<optimized out>) at ../wine-7.0/include/basetsd.h:166 #4 RtlEnterCriticalSection (crit=crit(a)entry=0x7bcc44f0 <loader_section>) at ../wine-7.0/dlls/ntdll/sync.c:418 #5 0x000000007bc57364 in LdrUnloadDll (hModule=0xffff93c50000 <__wine_spec_pe_header+24768>) at ../wine-7.0/dlls/ntdll/loader.c:3738 #6 LdrUnloadDll (hModule=0xffff93c50000 <__wine_spec_pe_header+24768>) at ../wine-7.0/dlls/ntdll/loader.c:3729 #7 0x000000007b04ee4c in FreeLibrary (module=0xffff93c50000 <__wine_spec_pe_header+24768>) at ../wine-7.0/dlls/kernelbase/loader.c:265 #8 0x0000ffff93e6d8f0 in free_delay_imports () at ../wine-7.0/dlls/winecrt0/delay_load.c:78 #9 0x0000ffff944ce0b8 in _dl_fini () at dl-fini.c:138 #10 0x0000ffff9432e9ec in __run_exit_handlers (status=status(a)entry=0, listp=0xffff944585e8 <__exit_funcs>, run_list_atexit=run_list_atexit(a)entry=true, run_dtors=run_dtors(a)entry=true) at exit.c:108 #11 0x0000ffff9432eb54 in __GI_exit (status=status(a)entry=0) at exit.c:139 #12 0x0000ffff9447aa48 in start_thread (arg=0xa3e4a6) at pthread_create.c:500 #13 0x0000ffff943c3f6c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 The path from _dl_fini -> free_delay_imports -> FreeLibrary seems suspect - By the time pthreads is calling exit (in glibc), because the last thread has returned, we have long since unwound Win32 side of things, including destroying the Teb for that thread via BaseThreadInitThunk -> RtlExitUserThread -> NtTerminateThread -> exit_thread -> virtual_free_teb. So by the time we get to free_delay_imports -> FreeLibrary -> LdrUnloadDll -> RtlEnterCriticalSection( &loader_section ), which does `crit->OwningThread = ULongToHandle(GetCurrentThreadId())`, calling GetCurrentThreadId is a use-after-free if the already-destroyed Teb. So it seems like free_delay_imports needed to happen earlier, back when when the Win32 aspect of this module was about to be freed (but still valid) (DLL_PROCESS_DETACH after the user's DllImport returned, or __wine_spec_exe_entry between the return from main and the call to ExitProcess()), not put off until libdl is cleaning up the ELF structures. But referencing it from those places would no longer allow delay_load.c to just be discarded from libwinecrt0.a when __wine_spec_delay_load is unreferenced by the spec.o file. So I'm not quite sure how best to achieve this order of operations. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Artem S. Tashkinov <aros(a)gmx.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|winedevice.exe segfaults on |winedevice.exe segfaults on |exit when built by GCC |exit when built with GCC |>=12.0 and -march=pentium-m | -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #27 from Kevin Puetz <PuetzKevinA(a)JohnDeere.com> --- To make debugging harder, once that least thread has exited (the starting point of this use-after-free), the process is now racing a timer started in wineserver's process_killed that will SIGKILL the process in 0.5s, preventing one from really stepping through any of this (since even an attached debugger can't block SIGKILL) https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/server/process.c#L996 https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/server/process.c#L615-61... -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #28 from Kevin Puetz <PuetzKevinA(a)JohnDeere.com> --- Created attachment 72849 --> https://bugs.winehq.org/attachment.cgi?id=72849 Test code exploring the load/unload order of delayload dlls In the attached example, there are 5 dlls, all of which trace the order of their DllMain calls to stderr, as well as their namesake function main is linked to a.dll, and /delayload linked to b.dll. e.dll is linked to b.dll and /delayload linked to d.dll. main also does a LoadLibrary/GetProcAddress/call/FreeLibrary on e() twice. The first time it frees it properly, the second it leaks it. The resulting trace (diffing PE vs ELF, both x64) is: --- MSVC2022 Win10 +++ winegcc 11.2 wine-7.0 a.DllMain(1) main() -- begin a() b.DllMain(1) b() c.DllMain(1) e.DllMain(1) c() d.DllMain(1) d() e() e.DllMain(0) c.DllMain(0) +e.destructor() +c.destructor() +d.destructor() c.DllMain(1) e.DllMain(1) c() +d.DllMain(1) d() e() main() -- return atexit handler +d.DllMain(0) e.DllMain(0) c.DllMain(0) -d.DllMain(0) b.DllMain(0) a.DllMain(0) +main.destructor() +a.destructor() +b.destructor() +e.destructor() +c.destructor() +d.destructor() So the first time through, a is loaded up front, and b is delay-loaded before its first call (as expected). Loading e.dll first loads c.dll, and d is delay loaded before its first call. Unloading e frees c, but not d. The second call to load e reloads c, but d was still loaded from the first time through. When main exits, e, c, b, and a all unload (in that order). This eppears to just be the reverse order of the still-active LoadLibrary calls, nothing based on one library's finalizer containing code to free another. So, to first appearences, MSVC's delayimp.lib just leaks the LoadLibrary call, and it's cleaned up like any other still-loaded module during process exit. Running these msvc-built .dll/.exe files in wine produces a the same trace, so it appears that wine's ntdll contains the code to clean this up the same way, and for PE files everythign matches. But building this same code with winegcc (and thus winecrt0.a) produces differences even prior to the (potential) crash. FreeLibray(e_dll) still omits d.DllMain(DLL_PROCESS_DETACH), but e.dll.so's matching *does* run the ELF finalizers (__attribute((destructor)) of d.dll.so, so apparently it did dlclose the handle for d.dll.so, just without doing the Win32 DLL_PROCESS_DETACH first. And, because we dumped the buitin d.dll.so, the second round of loading and calling e does a second LoadLibrary(d.dll) and we see d.DllMain(DLL_PROCESS_ATTACH) run again. This seems bad, since we've unbalanced DLL_PROCESS_{ATTACH,DETACH}. The DllMain call seems to be missing because of the free_lib_count check in LdrUnloadDll (https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/dlls/ntdll/loader.c#L374...). Since we've ended up making recursive FreeLibrary(e.dll) -> dlclose(e.dll.so) -> free_delay_imports -> FreeLibrary(d.dll.so), which would violate the loader_section lock, it skips skips calling process_detach, and the d.DllMain(DLL_PROCESS_DETACH) call is lost. And also, this change to the load order means d.DllMain(DLL_PROCESS_DETACH) is the fist thing run after atexit, instead of e (and c) preceding it (which would be the reverse order of the LoadLibrary calls), which differs from windows. So... after doing this more investigation of the actual behavior, it seems like free_delay_imports should simply not exist. Calling FreeLibrary during the unload of a .exe.so is pointless (everything gets properly cleaned up during LdrShutdownProcess anyway), and doing so during the unload of a .dll.so inherently leads to this this invalid recursion (and thus misses the DLL_PROCESS_DETACH). The safe thing to do would be to somehow decrement delayloaded dependencies during the recursive MODULE_DecRefCount, which would then lead to process_detach and MODULE_FlushModrefs calls unloading it sequentally (rather than recursively). But that would be seemingly require private API between ntdll and winecrt0, and while the result seems sound, it wouldn't match the apparent behavior of MSVC's delayimp.lib (which presumably omits freeing delayload libs for similar reasons). It's explicitly unsafe to call FreeLibrary from within DllMain, and there's no other unload hook available in a PE build, so there's just no way for delayimp.cpp to do something similar to what wine is trying here. https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloader...
It is not safe to call FreeLibrary from DllMain. For more information, see the Remarks section in DllMain.
And if it's not a part of the real PE/windows/MSVC implementation, then I'm not sure what wine is trying to match. This is very old code (mid 2005 and pre-1.0), though it seems like there was some amount of investigation in b2d35c3620be73ca662e73fed49a280b405d1f1c that lead to disabling it for MacOS but keeping it for other platforms (?). So any other insight is welcome... -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #29 from Kevin Puetz <PuetzKevinA(a)JohnDeere.com> --- looks like doing this was originally inside __wine_dll_main, (in 5ea0dd6d5f2b085574e8d69a90200669461b2a4b), rather than the ELF __attribute__((destructor)) where it moved (in d4098549bb126e1711d91dbf2d2d9a610967c127). That would not would not have had the TEB use-after-free, but would still have had the problems of being a recursive FreeLibrary call (free_lib_count existed even way back then) So that doesn't give any real hints either on why wine tries to unload when windows seemingly doesn't, other than it hasn't just always been that way - delayloading was added in 2000 (5e32d1662867e41aae8b3748f7ee375644e4766a), and code to free them came ~4.5 years later. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #30 from Artem S. Tashkinov <aros(a)gmx.com> --- In Wine 7.16 winedevice.exe no longer segfaults but still gives this error on exit: 0060:err:virtual:virtual_setup_exception stack overflow 260 bytes in thread 0060 addr 0x7bc8247d stack 0x1620efc (0x1620000-0x1621000-0x1720000) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Béla Gyebrószki <gyebro69(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gyebro69(a)gmail.com -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #31 from Kevin Puetz <PuetzKevinA(a)JohnDeere.com> --- Proposed fix: https://gitlab.winehq.org/wine/wine/-/merge_requests/750 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 --- Comment #32 from Artem S. Tashkinov <aros(a)gmx.com> --- (In reply to Kevin Puetz from comment #31)
Proposed fix: https://gitlab.winehq.org/wine/wine/-/merge_requests/750
Thanks a ton for the patch. Confirming fixed. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Zeb Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Component|wineserver |winecrt0 Fixed by SHA1| |b5d3759fc5852f083632792d621 | |3f54151a6467a Status|UNCONFIRMED |RESOLVED --- Comment #33 from Zeb Figura <z.figura12(a)gmail.com> --- Fixed by <https://source.winehq.org/git/wine.git/commitdiff/b5d3759fc5852f083632792d6213f54151a6467a>. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Alexandre Julliard <julliard(a)winehq.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #34 from Alexandre Julliard <julliard(a)winehq.org> --- Closing bugs fixed in 7.17. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Michael Stefaniuc <mstefani(a)winehq.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |7.0.x -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=53032 Michael Stefaniuc <mstefani(a)winehq.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|7.0.x |--- --- Comment #35 from Michael Stefaniuc <mstefani(a)winehq.org> --- Removing the 7.0.x milestone from bug fixes included in 7.0.2. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla