[Bug 53032] New: winedevice.exe segfaults on exit (Segmentation fault) in Wine 7.9

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Bug ID: 53032
           Summary: winedevice.exe segfaults on exit (Segmentation fault)
                    in Wine 7.9
           Product: Wine
           Version: 7.9
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: P2
         Component: wineserver
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

This is a regression or maybe it's down to GCC 12.1 - I don't know.

When exiting wine, winedevice.exe segfaults all the time.

I cannot get a bt and my attempts of using gdb have been futile.

(gdb) c
Continuing.
[LWP 495578 exited]
[LWP 495577 exited]
[LWP 495576 exited]
[LWP 495573 exited]

Thread 5 "winedevice.exe" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 495599]
0x7e68a2bf in ?? ()
(gdb) bt
#0  0x7e68a2bf in ?? ()
Backtrace stopped: Cannot access memory at address 0x1fe

Wine is built using: -O2 -march=pentium-m -m32

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|winedevice.exe segfaults on |winedevice.exe segfaults on
                   |exit (Segmentation fault)   |exit (Segmentation fault)
                   |in Wine 7.9                 |when built by GCC 12.1
                   |                            |(Fedora 36)

--- Comment #1 from Artem S. Tashkinov  ---
This is an issue with GCC 12.1 in Fedora 36.

I've just compiled Wine 7.8 and the issue persists.

[????] Process 577885 (winedevice.exe) of user 1000 dumped core.
Stack trace of thread 577888:
#0  0x00000000f7d51e7d n/a (n/a + 0x0)
#1  0x00000000f7d528f7 n/a (n/a + 0x0)
ELF object binary architecture: Intel 80386

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #2 from Artem S. Tashkinov  ---
The issue is reproducible with -march=i686 as well.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #3 from Artem S. Tashkinov  ---
GCC 12.0.1 is also affected.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #4 from Artem S. Tashkinov  ---
I've built Wine with GCC 11.3 and the bug is no more.

No idea who's responsible - it could be GCC, it could be Wine.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Rémi Bernon  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #5 from Rémi Bernon  ---
Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A log
with
WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb
could be useful.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #6 from Artem S. Tashkinov  ---
(In reply to Rémi Bernon from comment #5)
> Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A
> log with
> WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,
> +hid,+wineusb could be useful.

I doubt this can work as it enables logging for wine, not for its dependent
subprocess.

OK, I've run

WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb
/tmp/wine-7.9/wine notepad &> /tmp/wine.log

the last line is all good and fine:
0034:trace:service:main services.exe exited with code 0

Except a couple of seconds later when wineserver decides to shutdown there's a
crash. I've no idea where to get logs for it.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #7 from Artem S. Tashkinov  ---
I've even tried running winedevice.exe directly, again, wine exits successfully
and then a few seconds later winedevice.exe segfaults:

WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,+hid,+wineusb
/tmp/wine-7.9/wine winedevice.exe &> /tmp/wine.log

003c:trace:service:main services.exe exited with code 0

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #8 from Artem S. Tashkinov  ---
Here's how I build Wine:

export CFLAGS="-O2 -pipe -m32 -march=pentium-m"
export CXXFLAGS="$CFLAGS"
export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu"

./configure --prefix=/opt/wine --disable-tests && make -j16

This is 100% reproducible under Fedora 36.

Maybe it's down to my Ryzen 7 5800X CPU and you have an Intel one, so you don't
hit the issue.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #9 from Artem S. Tashkinov  ---
I've tried to

$ gdb wine
attach X

$ gdb wine
attach Y

(twice because there are two winedevice.exe processes)

Then I exit whatever wine application I've launched, e.g. notepad.exe and then
I see:

(gdb) bt
#0  0xf7fe0142 in ?? ()
Backtrace stopped: Cannot access memory at address 0x31f86c

(gdb) bt
#0  0xf7f0f142 in ?? ()
Backtrace stopped: Cannot access memory at address 0x31f86c

$ ps ax | grep wine
 136801 pts/2    Sl+    0:00 gdb ./wine
 137098 ?        Zsl    0:00 [winedevice.exe] <defunct>
 137106 ?        Zsl    0:00 [winedevice.exe] <defunct>
 137394 pts/4    Sl+    0:00 gdb ./wine

Zombies! No idea what's going on.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #10 from Austin English  ---
(In reply to Artem S. Tashkinov from comment #6)
> (In reply to Rémi Bernon from comment #5)
> > Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A
> > log with
> > WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,
> > +hid,+wineusb could be useful.
>
> I doubt this can work as it enables logging for wine, not for its dependent
> subprocess.

Untested, but you could try exporting WINEDEBUG before starting any wine
process, in theory the dependent processes would get it from the environment.

(In reply to Artem S. Tashkinov from comment #8)
> Here's how I build Wine:
>
> export CFLAGS="-O2 -pipe -m32 -march=pentium-m"
> export CXXFLAGS="$CFLAGS"
> export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu"

Why the mix of optimization flags (01/02)? Does omitting march change anything?

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #11 from Rémi Bernon  ---
As you managed to get gdb attached, you can loading this file in gdb with
https://github.com/rbernon/wine/raw/develop/tools/gdbinit.py to maybe get
symbols. You can load it in gdb with `source <filename.py>` and then load the
symbols with a new load-symbol-files command.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #12 from Artem S. Tashkinov  ---
(In reply to Austin English from comment #10)
> (In reply to Artem S. Tashkinov from comment #6)
> > (In reply to Rémi Bernon from comment #5)
> > > Hi, I built Wine with GCC 12.1 and didn't see anything like that so far. A
> > > log with
> > > WINEDEBUG=+loaddll,+seh,+process,+ntoskrnl,+service,+winedevice,+plugplay,
> > > +hid,+wineusb could be useful.
> >
> > I doubt this can work as it enables logging for wine, not for its dependent
> > subprocess.
>
> Untested, but you could try exporting WINEDEBUG before starting any wine
> process, in theory the dependent processes would get it from the environment.

The problem is winedevice.exe is a background process you do not start, I've no
idea how to catch the debug output in this case, it's gonna be output to
/dev/null - it's never linked to console :(
> (In reply to Artem S. Tashkinov from comment #8)
> > Here's how I build Wine:
> >
> > export CFLAGS="-O2 -pipe -m32 -march=pentium-m"
> > export CXXFLAGS="$CFLAGS"
> > export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu"
>
> Why the mix of optimization flags (01/02)? Does omitting march change
> anything?

LD doesn't have the -O2 optimization level.

Will try to build without -march. Please wait.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #13 from Artem S. Tashkinov  ---
Created attachment 72415
  --> https://bugs.winehq.org/attachment.cgi?id=72415
winedebug.log.xz

OK, actually exporting the variable made wine spill debug messages to console
and I've managed to save everything but I don't see any crash.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #14 from Artem S. Tashkinov  ---
Removing -march=pentium-m has fixed the issue.

Looks like this is a GCC bug.

I really don't know how to proceed further.

GCC might miscompile anything including the main `wine` binary, libwine.so,
winedevice.exe - I've no idea where to look.

[Bug 53032] winedevice.exe segfaults on exit (Segmentation fault) when built by GCC 12.1 (Fedora 36)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=105700

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|winedevice.exe segfaults on |winedevice.exe segfaults on
                   |exit (Segmentation fault)   |exit when built by GCC
                   |when built by GCC 12.1      |>=12.0 and -march=pentium-m
                   |(Fedora 36)                 |

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #15 from Artem S. Tashkinov  ---
The crash is in ntdll.dll.so but that's a huge library:

$ ls -la ntdll*so*
-rwxr-xr-x. 1 root root 926240 May 23 09:42 ntdll.dll.so.o2
-rwxr-xr-x. 1 root root 950816 May 23 09:46 ntdll.dll.so.pentiumm

~25KB of extra binary code.

Here's another tidbit: even there's no crash (i.e. GCC 12: -O2 -m32, GCC 11.3:
-O2 -march=pentium-m -m32), there's still something wrong when exiting Wine
applications:

00b0:err:virtual:virtual_setup_exception nested exception on signal stack in
thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0

I feel like there's a coding error which gets worse when using
-march=pentium-m.

I still have no clue how to debug all of this. Considering how easy it's to
reproduce ( see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105700#c2 ), would
be great if Wine hackers chimed in instead.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Bernhard Übelacker  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #16 from Bernhard Übelacker  ---
Created attachment 72435
  --> https://bugs.winehq.org/attachment.cgi?id=72435
debugging-with-rr.txt

I tried if I could reproduce it inside a qemu VM.
And I guess I succeeded in getting at least the "stack overlow" variant.

This combined with rr-debugger and an old but modified version of gdbinit.py
I reached in winedevice.exe a segfault.

(rr) bt
#0  0x7e154db0 in SetupCloseLog ()
#1  0x7bc54b16 in call_dll_entry_point ()
#2  0x7bc59381 in MODULE_InitDLL ()
...

Some notes, the backtrace and singlestepping through SetupCloseLog
showing the esp register are in attached file.

It is a build without mingw compilers and from flags without much debug
information, but function SetupCloseLog is quite short with just a few
function calls.

As far as I see the esp register has "just" the wrong value
when the ret instruction is reached.
Unfortunately I cannot point exactly to a single instruction
which might be wrong.
Maybe someone with better assembly knowledge can have a look?

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #17 from Bernhard Übelacker  ---
I redid the steps inside a Fedore 35 VM, with gcc 11.3.1.

There I can confirm the assembly for SetupCloseLog is equal.
(Except two instructions swapped before the EnterCriticalSection.)

But there the $esp value at the ret instruction is the same as
at SetupCloseLog+0. But still a segfault is observable because
the stack memory holding the return address for the ret instruction
got overwritten here:

(rr) stepi
0xf7b93f10 in __wine_syscall_dispatcher ()
1: x/i $pc
=> 0xf7b93f10 <__wine_syscall_dispatcher+20>:   mov    %esp,0xc(%ecx)
3: *(void**)0x165ef4c = (void *) 0x7d8c9f9c <DllMain+124>
(rr) bt
#0  0xf7b93f10 in __wine_syscall_dispatcher ()
#1  0x7b087541 in CloseHandle ()
#2  0x7d8b9b6f in SetupCloseLog ()
#3  0x7d8c9f9c in DllMain ()
#4  0x7bc55216 in call_dll_entry_point ()
#5  0x7bc59ae1 in MODULE_InitDLL ()
#6  0x7bc5a07f in process_detach ()
...
(rr) stepi
0xf7b93f13 in __wine_syscall_dispatcher ()
1: x/i $pc
=> 0xf7b93f13 <__wine_syscall_dispatcher+23>:   mov    %cs,0x10(%ecx)
3: *(void**)0x165ef4c = (void *) 0x165eeec

@Artem:
If you build with MinGW cross compilers, which versions do you have installed?

@All:
Is a MinGW-less build still considered fully supported?

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #18 from Artem S. Tashkinov  ---
(In reply to Bernhard Übelacker from comment #17)
> @Artem:
> If you build with MinGW cross compilers, which versions do you have
> installed?

I use GCC only.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #19 from Rafał Mużyło  ---
> 00b0:err:virtual:virtual_setup_exception nested exception on signal stack in thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0

As I've mentioned today in a couple other places, this is unrelated. I'm
getting this relatively often as a random (not happening every time) crash on
exit in various apps with gcc 11.2.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Zeb Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #20 from Zeb Figura  ---
(In reply to Rafał Mużyło from comment #19)
> > 00b0:err:virtual:virtual_setup_exception nested exception on signal stack in thread 00b0 addr 0xf7cc2e9b stack 0x7ff7f0e0
>
> As I've mentioned today in a couple other places, this is unrelated. I'm
> getting this relatively often as a random (not happening every time) crash
> on exit in various apps with gcc 11.2.

It's not unrelated, it's exactly what this bug is. I wouldn't be surprised if
it's the exact same bug, despite the gcc version. It's likely that Wine is
doing something undefined, that only works by chance.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #21 from Bernhard Übelacker  ---
This seems to be a more reliable callstack without the gdbinit.py magic,
but instead just loading the loader/wine executable.

I guess the important line is the free_delay_imports frame.
As this is a non-PE build, might this stack get damaged by the
stack unwinding - and therefore the wine call is no longer working?

Following is all from one thread, just before the segfault:

(rr) file /home/benutzer/wine-build/loader/wine
(rr) bt
#0  0xf7b7aaf2 in signal_exit_thread () from dlls/ntdll/ntdll.so
#1  0xf7b95fdf in abort_thread () at include/winnt.h:2165
#2  0xf7b777ce in wait_select_reply () at dlls/ntdll/unix/server.c:324
#3  0xf7b7a459 in server_select () at dlls/ntdll/unix/server.c:639
#4  0xf7b7a508 in server_wait () at dlls/ntdll/unix/server.c:666
#5  0xf7b8663c in NtWaitForMultipleObjects () at dlls/ntdll/unix/sync.c:1428
#6  0xf7b7ac09 in __wine_syscall_dispatcher () from dlls/ntdll/ntdll.so
#7  0x00000001 in ?? ()
#8  0x0155fd50 in ?? ()
#9  0xf7bc1340 in ?? () from dlls/ntdll/ntdll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(rr) when
Current event: 366461
...
(rr) bt
#0  0xf7d1f610 in __longjmp_cancel () from /lib/libc.so.6
#1  0xf7d1f572 in __libc_longjmp () from /lib/libc.so.6
#2  0xf7d7b5a2 in unwind_stop () from /lib/libc.so.6
#3  0x7e1b439a in _Unwind_ForcedUnwind_Phase2 () at libgcc/unwind.inc:171
#4  0x7e1b46be in _Unwind_ForcedUnwind () at libgcc/unwind.inc:218
#5  0xf7d7b6fb in __pthread_unwind () from /lib/libc.so.6
#6  0xf7d738af in pthread_exit () from /lib/libc.so.6
#7  0xf7b929ed in pthread_exit_wrapper () at dlls/ntdll/unix/thread.c:1052
#8  0xf7b7aafa in signal_exit_thread () from dlls/ntdll/ntdll.so
#9  0x0165f338 in ?? ()
#10 0xf7d723bd in start_thread () from /lib/libc.so.6
#11 0xf7df953a in clone () from /lib/libc.so.6
(rr) when
Current event: 366462
...
(rr) bt
#0  0xf7b7ab09 in __wine_syscall_dispatcher () from dlls/ntdll/ntdll.so
#1  0x7bc3544c in NtClose () from dlls/ntdll/ntdll.dll.so
#2  0x7b08a1a1 in CloseHandle () at dlls/kernelbase/process.c:421
#3  0x7e154ddf in SetupCloseLog () at dlls/setupapi/misc.c:1661
#4  0x7e164949 in DllMain () at dlls/setupapi/setupcab.c:453
#5  0x7bc54b16 in call_dll_entry_point () from dlls/ntdll/ntdll.dll.so
#6  0x7bc59381 in MODULE_InitDLL () at dlls/ntdll/loader.c:1568
#7  0x7bc5992f in process_detach () at dlls/ntdll/loader.c:1714
#8  0x7bc59be5 in LdrUnloadDll () at dlls/ntdll/loader.c:3850
#9  0x7bc59c87 in LdrUnloadDll () at dlls/ntdll/loader.c:3833
#10 0x7b056009 in FreeLibrary () at dlls/kernelbase/loader.c:265
#11 0x7e5d8091 in free_delay_imports () at dlls/winecrt0/delay_load.c:78
#12 0xf7f04df2 in _dl_fini () at dl-fini.c:142
#13 0xf7d223a3 in __run_exit_handlers () from /lib/libc.so.6
#14 0xf7d224f7 in exit () from /lib/libc.so.6
#15 0xf7d7242e in start_thread () from /lib/libc.so.6
#16 0xf7df953a in clone () from /lib/libc.so.6
(rr) when
Current event: 366462

What I do not yet understand why the free_delay_imports
seems not to get called at all at my usual Debian system.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #22 from Rafał Mużyło  ---
Little note: I'm getting those errors despite being on on a mingw build.

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #23 from Bernhard Übelacker  ---
glibc checks for the thread being the last thread and then executes
the exit() call I see in the backtrace.

Maybe some race condition, which thread is the last one?

https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/pthread_create.c;h=6c6...
(This is 2.33, similar to Debians version. There were some changes to
start_thread between 2.33 and 2.34 which is used in Fedora.)

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Ken Sharp  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |download, source

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #24 from Artem S. Tashkinov  ---
As mentioned earlier with GCC 11.3 there's still something wrong:

0058:err:virtual:virtual_setup_exception stack overflow 260 bytes in thread
0058 addr 0x7bc7f4ed stack 0x1220efc (0x1220000-0x1221000-0x1320000)

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #25 from Artem S. Tashkinov  ---
winedevice.exe for Wine 7.14 now segfaults even when being built with GCC 11.3
:-(

Process 247198 (winedevice.exe) of user 1000 dumped core.

Stack trace of thread 247203:
#0  0x00000000f7cdac0d n/a (n/a + 0x0)
#1  0x00000000f7cdb6bc n/a (n/a + 0x0)
ELF object binary architecture: Intel 80386

[Bug 53032] winedevice.exe segfaults on exit when built by GCC >=12.0 and -march=pentium-m

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Kevin Puetz  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #26 from Kevin Puetz  ---
I'm seeing a similar crash-on-exit in our aarch64 builds of wine-7.0:
winedevice.exe and rpcss.exe are both crashing on exit with a similar stack
trace: This is gcc 9.2, on a yocto sysroot.

#0  0x0000ffff9428db34 in handle_syscall_fault (rec=0x67f9ed08,
context=0x67f9ee20) at ../wine-7.0/dlls/ntdll/unix/unix_private.h:70#1 
segv_handler (signal=<optimized out>, siginfo=<optimized out>,
sigcontext=0x67f9ee20) at ../wine-7.0/dlls/ntdll/unix/signal_arm64.c:872
#2  <signal handler called>
#3  ULongToHandle (ul=<optimized out>) at ../wine-7.0/include/basetsd.h:166
#4  RtlEnterCriticalSection (crit=crit@entry=0x7bcc44f0 <loader_section>) at
../wine-7.0/dlls/ntdll/sync.c:418
#5  0x000000007bc57364 in LdrUnloadDll (hModule=0xffff93c50000
<__wine_spec_pe_header+24768>) at ../wine-7.0/dlls/ntdll/loader.c:3738
#6  LdrUnloadDll (hModule=0xffff93c50000 <__wine_spec_pe_header+24768>) at
../wine-7.0/dlls/ntdll/loader.c:3729
#7  0x000000007b04ee4c in FreeLibrary (module=0xffff93c50000
<__wine_spec_pe_header+24768>) at ../wine-7.0/dlls/kernelbase/loader.c:265
#8  0x0000ffff93e6d8f0 in free_delay_imports () at
../wine-7.0/dlls/winecrt0/delay_load.c:78
#9  0x0000ffff944ce0b8 in _dl_fini () at dl-fini.c:138
#10 0x0000ffff9432e9ec in __run_exit_handlers (status=status@entry=0,
listp=0xffff944585e8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true,
    run_dtors=run_dtors@entry=true) at exit.c:108
#11 0x0000ffff9432eb54 in __GI_exit (status=status@entry=0) at exit.c:139
#12 0x0000ffff9447aa48 in start_thread (arg=0xa3e4a6) at pthread_create.c:500
#13 0x0000ffff943c3f6c in thread_start () at
../sysdeps/unix/sysv/linux/aarch64/clone.S:78

The path from _dl_fini -> free_delay_imports -> FreeLibrary seems suspect - By
the time pthreads is calling exit (in glibc), because the last thread has
returned, we have long since unwound Win32 side of things, including destroying
the Teb for that thread via BaseThreadInitThunk -> RtlExitUserThread ->
NtTerminateThread -> exit_thread -> virtual_free_teb. So by the time we get to
free_delay_imports -> FreeLibrary -> LdrUnloadDll -> RtlEnterCriticalSection(
&loader_section ), which does `crit->OwningThread   =
ULongToHandle(GetCurrentThreadId())`, calling GetCurrentThreadId is a
use-after-free if the already-destroyed Teb.

So it seems like free_delay_imports needed to happen earlier, back when when
the Win32 aspect of this module was about to be freed (but still valid)
(DLL_PROCESS_DETACH after the user's DllImport returned, or
__wine_spec_exe_entry between the return from main and the call to
ExitProcess()), not put off until libdl is cleaning up the ELF structures.

But referencing it from those places would no longer allow delay_load.c to just
be discarded from libwinecrt0.a when __wine_spec_delay_load is unreferenced by
the spec.o file. So I'm not quite sure how best to achieve this order of
operations.

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|winedevice.exe segfaults on |winedevice.exe segfaults on
                   |exit when built by GCC      |exit when built with GCC
                   |>=12.0 and -march=pentium-m |

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #27 from Kevin Puetz  ---
To make debugging harder, once that least thread has exited (the starting point
of this use-after-free), the process is now racing a timer started in
wineserver's process_killed that will SIGKILL the process in 0.5s, preventing
one from really stepping through any of this (since even an attached debugger
can't block SIGKILL)

https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/server/process.c#L996
https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/server/process.c#L615-61...

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #28 from Kevin Puetz  ---
Created attachment 72849
  --> https://bugs.winehq.org/attachment.cgi?id=72849
Test code exploring the load/unload order of delayload dlls

In the attached example, there are 5 dlls, all of which trace the order of
their DllMain calls to stderr, as well as their namesake function

main is linked to a.dll, and /delayload linked to b.dll. e.dll is linked to
b.dll and /delayload linked to d.dll. main also does a
LoadLibrary/GetProcAddress/call/FreeLibrary on e() twice. The first time it
frees it properly, the second it leaks it.

The resulting trace (diffing PE vs ELF, both x64) is:

--- MSVC2022 Win10
+++ winegcc 11.2 wine-7.0
 a.DllMain(1)
 main() -- begin
 a()
 b.DllMain(1)
 b()
 c.DllMain(1)
 e.DllMain(1)
 c()
 d.DllMain(1)
 d()
 e()
 e.DllMain(0)
 c.DllMain(0)
+e.destructor()
+c.destructor()
+d.destructor()
 c.DllMain(1)
 e.DllMain(1)
 c()
+d.DllMain(1)
 d()
 e()
 main() -- return
 atexit handler
+d.DllMain(0)
 e.DllMain(0)
 c.DllMain(0)
-d.DllMain(0)
 b.DllMain(0)
 a.DllMain(0)
+main.destructor()
+a.destructor()
+b.destructor()
+e.destructor()
+c.destructor()
+d.destructor()

So the first time through, a is loaded up front, and b is delay-loaded before
its first call (as expected). Loading e.dll first loads c.dll, and d is delay
loaded before its first call. Unloading e frees c, but not d. The second call
to load e reloads c, but d was still loaded from the first time through. When
main exits, e, c, b, and a all unload (in that order). This eppears to just be
the reverse order of the still-active LoadLibrary calls, nothing based on one
library's finalizer containing code to free another. So, to first appearences,
MSVC's delayimp.lib just leaks the LoadLibrary call, and it's cleaned up like
any other still-loaded module during process exit. Running these msvc-built
.dll/.exe files in wine produces a the same trace, so it appears that wine's
ntdll contains the code to clean this up the same way, and for PE files
everythign matches.

But building this same code with winegcc (and thus winecrt0.a) produces
differences even prior to the (potential) crash. FreeLibray(e_dll) still omits
d.DllMain(DLL_PROCESS_DETACH), but e.dll.so's matching *does* run the ELF
finalizers (__attribute((destructor)) of d.dll.so, so apparently it did dlclose
the handle for d.dll.so, just without doing the Win32 DLL_PROCESS_DETACH first.
And, because we dumped the buitin d.dll.so, the second round of loading and
calling e does a second LoadLibrary(d.dll) and we see
d.DllMain(DLL_PROCESS_ATTACH) run again. This seems bad, since we've unbalanced
DLL_PROCESS_{ATTACH,DETACH}. The DllMain call seems to be missing because of
the free_lib_count check in LdrUnloadDll
(https://gitlab.winehq.org/wine/wine/-/blob/wine-7.0/dlls/ntdll/loader.c#L374...).
Since we've ended up making recursive FreeLibrary(e.dll) -> dlclose(e.dll.so)
-> free_delay_imports -> FreeLibrary(d.dll.so), which would violate the
loader_section lock, it skips skips calling process_detach, and the
d.DllMain(DLL_PROCESS_DETACH) call is lost.

And also, this change to the load order means d.DllMain(DLL_PROCESS_DETACH) is
the fist thing run after atexit, instead of e (and c) preceding it (which would
be the reverse order of the LoadLibrary calls), which differs from windows.

So... after doing this more investigation of the actual behavior, it seems like
free_delay_imports should simply not exist. Calling FreeLibrary during the
unload of a .exe.so is pointless (everything gets properly cleaned up during
LdrShutdownProcess anyway), and doing so during the unload of a .dll.so
inherently leads to this this invalid recursion (and thus misses the
DLL_PROCESS_DETACH). The safe thing to do would be to somehow decrement
delayloaded dependencies during the recursive MODULE_DecRefCount, which would
then lead to process_detach and MODULE_FlushModrefs calls unloading it
sequentally (rather than recursively). But that would be seemingly require
private API between ntdll and winecrt0, and while the result seems sound, it
wouldn't match the apparent behavior of MSVC's delayimp.lib (which presumably
omits freeing delayload libs for similar reasons). It's explicitly unsafe to
call FreeLibrary from within DllMain, and there's no other unload hook
available in a PE build, so there's just no way for delayimp.cpp to do
something similar to what wine is trying here.

https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloader...
> It is not safe to call FreeLibrary from DllMain. For more information, see the Remarks section in DllMain.

And if it's not a part of the real PE/windows/MSVC implementation, then I'm not
sure what wine is trying to match.

This is very old code (mid 2005 and pre-1.0), though it seems like there was
some amount of investigation in b2d35c3620be73ca662e73fed49a280b405d1f1c that
lead to disabling it for MacOS but keeping it for other platforms (?). So any
other insight is welcome...

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #29 from Kevin Puetz  ---
looks like doing this was originally inside __wine_dll_main, (in
5ea0dd6d5f2b085574e8d69a90200669461b2a4b), rather than the ELF
__attribute__((destructor)) where it moved (in
d4098549bb126e1711d91dbf2d2d9a610967c127).

That would not would not have had the TEB use-after-free, but would still have
had the problems of being a recursive FreeLibrary call (free_lib_count existed
even way back then)

So that doesn't give any real hints either on why wine tries to unload when
windows seemingly doesn't, other than it hasn't just always been that way -
delayloading was added in 2000 (5e32d1662867e41aae8b3748f7ee375644e4766a), and
code to free them came ~4.5 years later.

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #30 from Artem S. Tashkinov  ---
In Wine 7.16 winedevice.exe no longer segfaults but still gives this error on
exit:

0060:err:virtual:virtual_setup_exception stack overflow 260 bytes in thread
0060 addr 0x7bc8247d stack 0x1620efc (0x1620000-0x1621000-0x1720000)

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Béla Gyebrószki  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #31 from Kevin Puetz  ---
Proposed fix: https://gitlab.winehq.org/wine/wine/-/merge_requests/750

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

--- Comment #32 from Artem S. Tashkinov  ---
(In reply to Kevin Puetz from comment #31)
> Proposed fix: https://gitlab.winehq.org/wine/wine/-/merge_requests/750

Thanks a ton for the patch.

Confirming fixed.

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Zeb Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
          Component|wineserver                  |winecrt0
      Fixed by SHA1|                            |b5d3759fc5852f083632792d621
                   |                            |3f54151a6467a
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #33 from Zeb Figura  ---
Fixed by
https://source.winehq.org/git/wine.git/commitdiff/b5d3759fc5852f083632792d6213f54151a6467a.

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #34 from Alexandre Julliard  ---
Closing bugs fixed in 7.17.

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Michael Stefaniuc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |7.0.x

[Bug 53032] winedevice.exe segfaults on exit when built with GCC

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53032

Michael Stefaniuc  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|7.0.x                       |---

--- Comment #35 from Michael Stefaniuc  ---
Removing the 7.0.x milestone from bug fixes included in 7.0.2.