Debugging 64-bit Wine Apps with winedbg

[Tom Grubbe]

Hello, we're the developers of a large 64-bit Windows application that uses Wine 1.3.0 64-bit on Linux.  When testing began we discovered several crashes inside our windows code on startup and needed to debug them.  But the problem seems to be getting any kind of stack trace information from the 64-bit winedbg.  This used to work with the 32-bit version of winedbg.

Even if I write a 64-bit "hello world" app that intentionally crashes I still can't get a clean stack trace from winedbg.  We get a register dump and a stack dump but no back trace.  So in our app, we have to resort to calculating the address of the crash in our DLLs using a MAP file and adding a bunch of printfs to try and locate the source of the SEGV.  This is awkward, slow and just a bad way to debug.

So here's my question - Has anyone had any luck getting a debugger stack trace using the 64-bit winedbg?

Some relevant info:

* We build wine 64-bit 1.3.0 ourselves
* We're running on RHEL 5.5 (Tikanga) 2.6.18-194.el5 64-bit
* We get the same non-results from both release and debug windows binaries
* Using winedbg --gdb crashes gdb
* We tried using msvsmon.exe to attach to and debug the wine process using Visual Studio but it hasn't been helpful
* Setting WINEDEBUG to several debug channels has helped some but is difficult to sift through all the noise

So any info on strategies to debug 64-bit Wine applications is welcome, or if you need more info I'll be glad to supply it.

Thanks for your help!

--Tom Grubbe
ESRI

[GOUJON Alexandre]

Hi,

I'm not a winedbg expert but I think your mis-using it.

As it's your software, you can compile it from source and add debug info.
I know Visual Studio can create .pdb files but I don't know if winedbg 
can use it.

Personally, I'm using gcc mingw with -g and that perfectly works.
The easiest way is to install the linux package (e.g. mingw-w64 for the 
64 bits version or mingw32 what I'm using)

Then compile your software : (depends on your arch and the 32/64 bits 
version)
$ i586-mingw32msvc-gcc test.c -g -o test.exe

and run it through the debugger :
$ /home/alex/Projects/wine-git/wine 
/home/alex/Projects/wine-git/programs/winedbg/winedbg.exe.so test.exe

type "break main" or "break WinMain" on the console then return (you may 
have to select the proper main() )
(or break aFunctionName)
"cont" to continue until your breakpoint is reached
"list" to see the next lines of the code
"step" to step in
"next" to step over
"bt" to have the backtrace
"quit" to detach the process from the debugger and exit

you can also type "break" followed by a line number to add a breakpoint 
on the specified line of the current debugged source file

If you need more commands, G**gle is your friend.

I hope that that will help you.
---

If you have a page fault on create_alpha_bitmap, just type "set $BreakOnFirstChance=0" first

[Peter Urbanec]

On 23/09/10 06:51, Tom Grubbe wrote:
> problem seems to be getting any kind of stack trace information from 
> the 64-bit winedbg.  This used to work with the 32-bit version of winedbg.

I can confirm that wine64 winedbg can not produce valid backtraces for 
multi-threaded programs (.exe + .pdb) generated by the VS2005 64-bit 
compiler.

I get something like the following:

Backtracing for thread 001a in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68ceb3c7a0)

Backtracing for thread 0019 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68cec4c7a0)

Backtracing for thread 0018 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68ced5c320)

Backtracing for thread 0009 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68cef7deb0)
0x00007f0d2737b5ab __libc_read+0x2b in libpthread.so.0: syscall

Testing on Gentoo x86_64, with wine 1.3.2. Identical source code 
compiled with VS2005 32-bit compiler running under wine32 produces valid 
backtraces, although they suffer from bug #20617
> - Setting WINEDEBUG to several debug channels has helped some but is
>
> difficult to sift through all the noise

I don't get much joy from WINEDEBUG output because the crash I encounter 
appears to be in a DLL initialisation routine called / calling code from 
Microsoft.VC80.CRT redist code. I can't use the wine msvcrt/msvcp 
implementations because they are missing implementations for several 
functions.
> So any info on strategies to debug 64-bit Wine applications is welcome

I would also like to hear any tips for debugging under wine64. I'm 
finding that even the minidump files produced by wine64 are not much use 
in VS2005 or VS2008. At least the minidumps from wine32 can provide a 
little bit of info when loaded into VS2008 debugger.

I'm happy to provide test source code, 64-bit and 32-bit binaries and 
matching PDB files, if anyone is interested in looking at the issue.

Cheers,

Peter Urbanec

[Tom Grubbe]

It's good to know someone else is also seeing these problems.  In fact here's my tiny test app that should produce a backtrace but does not with winedbg 64-bit:

% cat Crash.cpp 
#include <stdio.h>

extern "C" void crash_func()
{
  char* p = 0;
  *p = 'x';
}

extern "C" int main()
{
  crash_func();
  return 0;
}

% wine64 /hex1/test/Crash/x64/Debug/Crash.exe 
fixme:heap:HeapSetInformation 0x2ad170690000 0 0x2ad17068fcb0 4
wine: Unhandled page fault on write access to 0x00000000 at address 0x140001046 (thread 0009), starting debugger...
Unhandled exception: page fault on write access to 0x00000000 in 64-bit code (0x0000000140001046).
Register dump:
 rip:0000000140001046 rsp:00002ad17068fca0 rbp:0000000140001220 eflags:00010206 (  R- --  I   - -P- )
 rax:0000000000000000 rbx:00002ad16cab9000 rcx:0000000000000000 rdx:00002ad170691230
 rsi:0000000000000000 rdi:00002ad17068fcb0  r8:00002ad1706919f0  r9:0000000000000004 r10:00002ad170691290
 r11:0000000010341ff0 r12:0000000000000000 r13:00002ad16cc72860 r14:00000001400000f0 r15:00002ad16cabc000
Stack dump:
0x00002ad17068fca0:  0000000000000000 cccccccccccccccc
0x00002ad17068fcb0:  00002ad17068fce0 000000014000107f
0x00002ad17068fcc0:  cccccccccccccccc cccccccccccccccc
0x00002ad17068fcd0:  cccccccccccccccc cccccccccccccccc
0x00002ad17068fce0:  00002ad16cfa5320 00000001400013d2
0x00002ad17068fcf0:  0000000100000001 0000000140006220
0x00002ad17068fd00:  00002ad16cab9000 00000001400018f5
0x00002ad17068fd10:  0000000000000000 0000000000000000
0x00002ad17068fd20:  0000000000000000 00002ad170690000
0x00002ad17068fd30:  0000000000000000 00005b5ac0000005
0x00002ad17068fd40:  00002ad17068e9e8 0000000000000000
0x00002ad17068fd50:  0000000000000000 000000014000122e
Backtrace:
0x0000000140001046: movb        $0x78,(%rax)
Modules:
Module  Address                                 Debug info      Name (14 modules)
PE              10200000-        1034f000       Deferred        msvcr90d
PE              40000000-        4000e000       Deferred        crash
ELF             7be00000-        7c102000       Deferred        <wine-loader>
ELF           3bdc200000-      3bdc41d000       Deferred        ld-linux-x86-64.so.2
ELF           3bdc600000-      3bdc957000       Deferred        libc.so.6
ELF           3bdca00000-      3bdcc83000       Deferred        libm.so.6
ELF           3bdce00000-      3bdd004000       Deferred        libdl.so.2
ELF           3bdd200000-      3bdd41b000       Deferred        libpthread.so.0
ELF           3be1200000-      3be140e000       Deferred        libgcc_s.so.1
ELF         2ad16c07d000-    2ad16c3ad000       Deferred        libwine.so.1
ELF         2ad16c3d4000-    2ad16c6b9000       Deferred        ntdll<elf>
  -PE      2ad16c3f0000-    2ad16c6b9000       \               ntdll
ELF         2ad16cc05000-    2ad16cfa6000       Deferred        kernel32<elf>
  -PE      2ad16cc20000-    2ad16cfa6000       \               kernel32
Threads:
process  tid      prio (all id:s are in hex)
00000008 (D) Z:\hex1\test\Crash\x64\Debug\Crash.exe
        00000009    0 <==
0000000e services.exe
        00000016    0
        00000015    0
        00000014    0
        00000010    0
        0000000f    0
00000011 winedevice.exe
        00000017    0
        00000013    0
        00000012    0
00000018 explorer.exe
        00000019    0
Backtrace:
err:seh:setup_exception nested exception on signal stack in thread 0009 eip 0000003bdd20bc05 esp 00002ad16cabef68 stack 0x2ad170592000-0x2ad170690000

--Tom Grubbe x2609

-----Original Message-----
From: Peter Urbanec [mailto:[e-mail]] 
Sent: Thursday, September 23, 2010 10:05 AM
To: [e-mail]
Subject: Re: Debugging 64-bit Wine Apps with winedbg

On 23/09/10 06:51, Tom Grubbe wrote:
> problem seems to be getting any kind of stack trace information from 
> the 64-bit winedbg.  This used to work with the 32-bit version of winedbg.

I can confirm that wine64 winedbg can not produce valid backtraces for 
multi-threaded programs (.exe + .pdb) generated by the VS2005 64-bit 
compiler.

I get something like the following:

Backtracing for thread 001a in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68ceb3c7a0)

Backtracing for thread 0019 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68cec4c7a0)

Backtracing for thread 0018 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68ced5c320)

Backtracing for thread 0009 in process 0008 (Z:\amd64\MultipleThreads.exe):
Backtrace:
=>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 
(0x00007f68cef7deb0)
0x00007f0d2737b5ab __libc_read+0x2b in libpthread.so.0: syscall

Testing on Gentoo x86_64, with wine 1.3.2. Identical source code 
compiled with VS2005 32-bit compiler running under wine32 produces valid 
backtraces, although they suffer from bug #20617
> - Setting WINEDEBUG to several debug channels has helped some but is
>
> difficult to sift through all the noise

I don't get much joy from WINEDEBUG output because the crash I encounter 
appears to be in a DLL initialisation routine called / calling code from 
Microsoft.VC80.CRT redist code. I can't use the wine msvcrt/msvcp 
implementations because they are missing implementations for several 
functions.
> So any info on strategies to debug 64-bit Wine applications is welcome

I would also like to hear any tips for debugging under wine64. I'm 
finding that even the minidump files produced by wine64 are not much use 
in VS2005 or VS2008. At least the minidumps from wine32 can provide a 
little bit of info when loaded into VS2008 debugger.

I'm happy to provide test source code, 64-bit and 32-bit binaries and 
matching PDB files, if anyone is interested in looking at the issue.

Cheers,

Peter Urbanec

[Eric Pouech]

> I'm happy to provide test source code, 64-bit and 32-bit binaries and 
> matching PDB files, if anyone is interested in looking at the issue.

sure send me the .exe+pdb (+source) I'll have a look at it
I only tested winedbg on 64bit with gcc

A+

[Peter Urbanec]

On 24/09/10 06:43, Eric Pouech wrote:
> sure send me the .exe+pdb (+source) I'll have a look at it

Source and binaries sent to Eric in private, so that the list isn't 
polluted with megabytes of binaries. If anyone else is interested in 
having a copy, please let me know and I'll forward it in private.

Cheers,

Peter Urbanec

[Eric Pouech]

Le 24/09/2010 05:12, Peter Urbanec a écrit :
> On 24/09/10 06:43, Eric Pouech wrote:
> > sure send me the .exe+pdb (+source) I'll have a look at it
>
> Source and binaries sent to Eric in private, so that the list isn't 
> polluted with megabytes of binaries. If anyone else is interested in 
> having a copy, please let me know and I'll forward it in private.
>
> Cheers,
> Peter Urbanec

with the patchset I've just sent, I get something like:
> Thread 0x0021 sleeping 200ms
> Thread 0x0022 sleeping 200ms
> Thread 0x0020 sleeping 240ms
> Thread 0x0023 sleeping 200ms
> Thread 0x0021 *** C R A S H I N G ***
> wine: Unhandled page fault on write access to 0x00000000 at address 
> 0x1400010b3 (thread 0021), starting debugger...
> Thread 0x0022 *** C R A S H I N G ***
> Thread 0x0023 *** C R A S H I N G ***
> WineDbg starting on pid 001f
> Thread 0x0020 *** C R A S H I N G ***
> First chance exception: page fault on write access to 0x00000000 in 
> 64-bit code (0x00000001400010b3).
> Register dump:
>  rip:00000001400010b3 rsp:00007feb8997fbb0 rbp:0000000140001a20 
> eflags:00010202 (  R- --  I   - - - )
>  rax:0000000000000000 rbx:00007feb8bb55000 rcx:0000000140084478 
> rdx:0000000140084480
>  rsi:00007feb8b937401 rdi:00007feb8997fbe0  r8:0000000000000001  
> r9:000000000010d640 r10:00007feb8997e850
>  r11:00007feb8997e820 r12:0000000000000000 r13:000000007b86f590 
> r14:00000001400000d8 r15:00007feb8bb50000
> Stack dump:
> 0x00007feb8997fbb0:  000000014006ed20 0000000000000020
> 0x00007feb8997fbc0:  00007feb8997fa60 0000000000000028
> 0x00007feb8997fbd0:  00007feb8997fc30 cccccccc00000020
> 0x00007feb8997fbe0:  00007feb8997fce0 0000000140001223
> 0x00007feb8997fbf0:  00007feb8997fc30 0000000000000010
> 0x00007feb8997fc00:  ccccccccffffffff 0000000140001014
> 0x00007feb8997fc10:  cccccccc00000004 00007feb8997fc84
> 0x00007feb8997fc20:  cccccccccccccccc cccccccccccccccc
> 0x00007feb8997fc30:  000000f0cccccc01 0000000000000000
> 0x00007feb8997fc40:  000000c8cccccc01 0000000000000000
> 0x00007feb8997fc50:  000000c8cccccc01 0000000000000000
> 0x00007feb8997fc60:  000000c8cccccc01 0000000000000000
> Backtrace:
> =>0 0x00000001400010b3 Boom::crash_func+0x83(void_ptr=0x7feb8997fc30, 
> arg_ref=0x7feb8997fc30, tid=32) 
> [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:29] 
> in multiplethreads (0x0000000140001a20)
>   1 0x0000000140001223 main+0x142(argc=1, argv=0x7feb89771130, 
> thisThread=0xfffffffffffffffe, threadHandle={0x1c, 0x20, 0x24}, 
> threadID=35, oldpri=0, a={{cause_crash=true, delay=240, 
> pointer=0x0(nil)}, {cause_crash=true, delay=200, pointer=0x0(nil)}, 
> {cause_crash=true, delay=200, pointer=0x0(nil)}, {cause_crash=true, 
> delay=200, pointer=0x0(nil)}}) 
> [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:63] 
> in multiplethreads (0x0000000140001a20)
>   2 0x0000000140001c5c __tmainCRTStartup+0x21b(winver=1281, mainret=0, 
> osver=2600, winminor=1, posvi=0x7feb8ba24070, osplatform=2, 
> managedapp=0, initret=0, winmajor=5) 
> [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c:327] in 
> multiplethreads (0x0000000140001a20)
>   3 0x0000000140001a2e mainCRTStartup+0xd() 
> [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c:195] in 
> multiplethreads (0x0000000140001a20)
>   4 0x000000007b86f64f start_process+0xbe(peb=0x7feb8bb55000) 
> [/home/eric/work/wine/dlls/kernel32/process.c:963] in kernel32 
> (0x0000000140001a20)
>   5 0x00007feb8abfe43e call_thread_func+0x6d(entry=<is not available>, 
> arg=<is not available>, frame=<is not available>) 
> [/home/eric/work/wine/dlls/ntdll/signal_x86_64.c:3036] in ntdll 
> (0x0000000140001a20)
>   6 0x00007feb8abfd73e call_thread_entry_point+0x29() in ntdll 
> (0x0000000140001a20)
>   7 0x00007feb8abd0da6 start_process+0x15(kernel_start=0x7feb8997fec8) 
> [/home/eric/work/wine/dlls/ntdll/loader.c:2610] in ntdll 
> (0x0000000140001a20)
>   8 0x00007feb8b611ea3 wine_call_on_stack+0x12() in libwine.so.1 
> (0x0000000140001a20)
>   9 0x00007feb8b611eb9 wine_switch_to_stack+0x8(func=0x7feb8997fec8, 
> arg=0x0(nil), stack=0x140084480) 
> [/home/eric/work/wine/libs/wine/port.c:83] in libwine.so.1 
> (0x0000000140001a20)
>   10 0x00007feb8abd5294 
> LdrInitializeThunk+0x3d3(kernel_start=0x7b86f590, unknown2=<is not 
> available>, unknown3=<is not available>, unknown4=<is not available>) 
> [/home/eric/work/wine/dlls/ntdll/loader.c:2666] in ntdll 
> (0x0000000140001a20)
>   11 0x000000007b874185 __wine_kernel_init+0x914() 
> [/home/eric/work/wine/dlls/kernel32/process.c:1163] in kernel32 
> (0x0000000140001a20)
>   12 0x00007feb8abd5824 __wine_process_init+0x253() 
> [/home/eric/work/wine/dlls/ntdll/loader.c:2876] in ntdll 
> (0x0000000140001a20)
>   13 0x00007feb8b610a0f wine_init+0x28e(argc=<is not available>, 
> argv=0x7fff9949d238, error="", error_size=<is not available>) 
> [/home/eric/work/wine/libs/wine/loader.c:711] in libwine.so.1 
> (0x0000000140001a20)
>   14 0x000000007bf00cf1 main+0x80(argc=2, argv=0x7fff9949d238) 
> [/home/eric/work/wine/loader/main.c:218] in <wine-loader> 
> (0x0000000140001a20)
>   15 0x00007feb8b087c4d __libc_start_main+0xfc(main=<is not 
> available>, argc=<is not available>, ubp_av=<is not available>, 
> init=<is not available>, fini=<is not available>, rtld_fini=<is not 
> available>, stack_end=0x7fff9949d228) 
> [/build/buildd/eglibc-2.11.1/csu/libc-start.c:226] in libc.so.6 
> (0x0000000140001a20)
>   16 0x000000007bf00ba9 _start+0x28() in <wine-loader> 
> (0x0000000140001a20)
> 0x00000001400010b3 Boom::crash_func+0x83 
> [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:29] 
> in multiplethreads: movq  %rax,0x0000000000000000
> Unable to open file ''

which is closer to what one should expect.
A+