Debugging 64-bit Wine Apps with winedbg
Hello, we're the developers of a large 64-bit Windows application that uses Wine 1.3.0 64-bit on Linux. When testing began we discovered several crashes inside our windows code on startup and needed to debug them. But the problem seems to be getting any kind of stack trace information from the 64-bit winedbg. This used to work with the 32-bit version of winedbg. Even if I write a 64-bit "hello world" app that intentionally crashes I still can't get a clean stack trace from winedbg. We get a register dump and a stack dump but no back trace. So in our app, we have to resort to calculating the address of the crash in our DLLs using a MAP file and adding a bunch of printfs to try and locate the source of the SEGV. This is awkward, slow and just a bad way to debug. So here's my question - Has anyone had any luck getting a debugger stack trace using the 64-bit winedbg? Some relevant info: * We build wine 64-bit 1.3.0 ourselves * We're running on RHEL 5.5 (Tikanga) 2.6.18-194.el5 64-bit * We get the same non-results from both release and debug windows binaries * Using winedbg --gdb crashes gdb * We tried using msvsmon.exe to attach to and debug the wine process using Visual Studio but it hasn't been helpful * Setting WINEDEBUG to several debug channels has helped some but is difficult to sift through all the noise So any info on strategies to debug 64-bit Wine applications is welcome, or if you need more info I'll be glad to supply it. Thanks for your help! --Tom Grubbe ESRI
Hi, I'm not a winedbg expert but I think your mis-using it. As it's your software, you can compile it from source and add debug info. I know Visual Studio can create .pdb files but I don't know if winedbg can use it. Personally, I'm using gcc mingw with -g and that perfectly works. The easiest way is to install the linux package (e.g. mingw-w64 for the 64 bits version or mingw32 what I'm using) Then compile your software : (depends on your arch and the 32/64 bits version) $ i586-mingw32msvc-gcc test.c -g -o test.exe and run it through the debugger : $ /home/alex/Projects/wine-git/wine /home/alex/Projects/wine-git/programs/winedbg/winedbg.exe.so test.exe type "break main" or "break WinMain" on the console then return (you may have to select the proper main() ) (or break aFunctionName) "cont" to continue until your breakpoint is reached "list" to see the next lines of the code "step" to step in "next" to step over "bt" to have the backtrace "quit" to detach the process from the debugger and exit you can also type "break" followed by a line number to add a breakpoint on the specified line of the current debugged source file If you need more commands, G**gle is your friend. I hope that that will help you. --- If you have a page fault on create_alpha_bitmap, just type "set $BreakOnFirstChance=0" first
On 23/09/10 06:51, Tom Grubbe wrote:
problem seems to be getting any kind of stack trace information from the 64-bit winedbg. This used to work with the 32-bit version of winedbg.
I can confirm that wine64 winedbg can not produce valid backtraces for multi-threaded programs (.exe + .pdb) generated by the VS2005 64-bit compiler. I get something like the following: Backtracing for thread 001a in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68ceb3c7a0) Backtracing for thread 0019 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68cec4c7a0) Backtracing for thread 0018 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68ced5c320) Backtracing for thread 0009 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68cef7deb0) 0x00007f0d2737b5ab __libc_read+0x2b in libpthread.so.0: syscall Testing on Gentoo x86_64, with wine 1.3.2. Identical source code compiled with VS2005 32-bit compiler running under wine32 produces valid backtraces, although they suffer from bug #20617
* Setting WINEDEBUG to several debug channels has helped some but is difficult to sift through all the noise
I don't get much joy from WINEDEBUG output because the crash I encounter appears to be in a DLL initialisation routine called / calling code from Microsoft.VC80.CRT redist code. I can't use the wine msvcrt/msvcp implementations because they are missing implementations for several functions.
So any info on strategies to debug 64-bit Wine applications is welcome
I would also like to hear any tips for debugging under wine64. I'm finding that even the minidump files produced by wine64 are not much use in VS2005 or VS2008. At least the minidumps from wine32 can provide a little bit of info when loaded into VS2008 debugger. I'm happy to provide test source code, 64-bit and 32-bit binaries and matching PDB files, if anyone is interested in looking at the issue. Cheers, Peter Urbanec
It's good to know someone else is also seeing these problems. In fact here's my tiny test app that should produce a backtrace but does not with winedbg 64-bit: % cat Crash.cpp #include <stdio.h> extern "C" void crash_func() { char* p = 0; *p = 'x'; } extern "C" int main() { crash_func(); return 0; } % wine64 /hex1/test/Crash/x64/Debug/Crash.exe fixme:heap:HeapSetInformation 0x2ad170690000 0 0x2ad17068fcb0 4 wine: Unhandled page fault on write access to 0x00000000 at address 0x140001046 (thread 0009), starting debugger... Unhandled exception: page fault on write access to 0x00000000 in 64-bit code (0x0000000140001046). Register dump: rip:0000000140001046 rsp:00002ad17068fca0 rbp:0000000140001220 eflags:00010206 ( R- -- I - -P- ) rax:0000000000000000 rbx:00002ad16cab9000 rcx:0000000000000000 rdx:00002ad170691230 rsi:0000000000000000 rdi:00002ad17068fcb0 r8:00002ad1706919f0 r9:0000000000000004 r10:00002ad170691290 r11:0000000010341ff0 r12:0000000000000000 r13:00002ad16cc72860 r14:00000001400000f0 r15:00002ad16cabc000 Stack dump: 0x00002ad17068fca0: 0000000000000000 cccccccccccccccc 0x00002ad17068fcb0: 00002ad17068fce0 000000014000107f 0x00002ad17068fcc0: cccccccccccccccc cccccccccccccccc 0x00002ad17068fcd0: cccccccccccccccc cccccccccccccccc 0x00002ad17068fce0: 00002ad16cfa5320 00000001400013d2 0x00002ad17068fcf0: 0000000100000001 0000000140006220 0x00002ad17068fd00: 00002ad16cab9000 00000001400018f5 0x00002ad17068fd10: 0000000000000000 0000000000000000 0x00002ad17068fd20: 0000000000000000 00002ad170690000 0x00002ad17068fd30: 0000000000000000 00005b5ac0000005 0x00002ad17068fd40: 00002ad17068e9e8 0000000000000000 0x00002ad17068fd50: 0000000000000000 000000014000122e Backtrace: 0x0000000140001046: movb $0x78,(%rax) Modules: Module Address Debug info Name (14 modules) PE 10200000- 1034f000 Deferred msvcr90d PE 40000000- 4000e000 Deferred crash ELF 7be00000- 7c102000 Deferred <wine-loader> ELF 3bdc200000- 3bdc41d000 Deferred ld-linux-x86-64.so.2 ELF 3bdc600000- 3bdc957000 Deferred libc.so.6 ELF 3bdca00000- 3bdcc83000 Deferred libm.so.6 ELF 3bdce00000- 3bdd004000 Deferred libdl.so.2 ELF 3bdd200000- 3bdd41b000 Deferred libpthread.so.0 ELF 3be1200000- 3be140e000 Deferred libgcc_s.so.1 ELF 2ad16c07d000- 2ad16c3ad000 Deferred libwine.so.1 ELF 2ad16c3d4000- 2ad16c6b9000 Deferred ntdll<elf> \-PE 2ad16c3f0000- 2ad16c6b9000 \ ntdll ELF 2ad16cc05000- 2ad16cfa6000 Deferred kernel32<elf> \-PE 2ad16cc20000- 2ad16cfa6000 \ kernel32 Threads: process tid prio (all id:s are in hex) 00000008 (D) Z:\hex1\test\Crash\x64\Debug\Crash.exe 00000009 0 <== 0000000e services.exe 00000016 0 00000015 0 00000014 0 00000010 0 0000000f 0 00000011 winedevice.exe 00000017 0 00000013 0 00000012 0 00000018 explorer.exe 00000019 0 Backtrace: err:seh:setup_exception nested exception on signal stack in thread 0009 eip 0000003bdd20bc05 esp 00002ad16cabef68 stack 0x2ad170592000-0x2ad170690000 --Tom Grubbe x2609 -----Original Message----- From: Peter Urbanec [mailto:winehq.org(a)urbanec.net] Sent: Thursday, September 23, 2010 10:05 AM To: wine-devel(a)winehq.org Subject: Re: Debugging 64-bit Wine Apps with winedbg On 23/09/10 06:51, Tom Grubbe wrote:
problem seems to be getting any kind of stack trace information from the 64-bit winedbg. This used to work with the 32-bit version of winedbg.
I can confirm that wine64 winedbg can not produce valid backtraces for multi-threaded programs (.exe + .pdb) generated by the VS2005 64-bit compiler. I get something like the following: Backtracing for thread 001a in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68ceb3c7a0) Backtracing for thread 0019 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68cec4c7a0) Backtracing for thread 0018 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68ced5c320) Backtracing for thread 0009 in process 0008 (Z:\amd64\MultipleThreads.exe): Backtrace: =>0 0x00007f68d08925ab __libc_read+0x2b() in libpthread.so.0 (0x00007f68cef7deb0) 0x00007f0d2737b5ab __libc_read+0x2b in libpthread.so.0: syscall Testing on Gentoo x86_64, with wine 1.3.2. Identical source code compiled with VS2005 32-bit compiler running under wine32 produces valid backtraces, although they suffer from bug #20617
* Setting WINEDEBUG to several debug channels has helped some but is difficult to sift through all the noise
I don't get much joy from WINEDEBUG output because the crash I encounter appears to be in a DLL initialisation routine called / calling code from Microsoft.VC80.CRT redist code. I can't use the wine msvcrt/msvcp implementations because they are missing implementations for several functions.
So any info on strategies to debug 64-bit Wine applications is welcome
I would also like to hear any tips for debugging under wine64. I'm finding that even the minidump files produced by wine64 are not much use in VS2005 or VS2008. At least the minidumps from wine32 can provide a little bit of info when loaded into VS2008 debugger. I'm happy to provide test source code, 64-bit and 32-bit binaries and matching PDB files, if anyone is interested in looking at the issue. Cheers, Peter Urbanec
I'm happy to provide test source code, 64-bit and 32-bit binaries and matching PDB files, if anyone is interested in looking at the issue. sure send me the .exe+pdb (+source) I'll have a look at it I only tested winedbg on 64bit with gcc
A+ -- Eric Pouech "The problem with designing something completely foolproof is to underestimate the ingenuity of a complete idiot." (Douglas Adams)
On 24/09/10 06:43, Eric Pouech wrote:
sure send me the .exe+pdb (+source) I'll have a look at it
Source and binaries sent to Eric in private, so that the list isn't polluted with megabytes of binaries. If anyone else is interested in having a copy, please let me know and I'll forward it in private. Cheers, Peter Urbanec
Le 24/09/2010 05:12, Peter Urbanec a écrit :
On 24/09/10 06:43, Eric Pouech wrote:
sure send me the .exe+pdb (+source) I'll have a look at it
Source and binaries sent to Eric in private, so that the list isn't polluted with megabytes of binaries. If anyone else is interested in having a copy, please let me know and I'll forward it in private.
Cheers,
Peter Urbanec
with the patchset I've just sent, I get something like:
Thread 0x0021 sleeping 200ms Thread 0x0022 sleeping 200ms Thread 0x0020 sleeping 240ms Thread 0x0023 sleeping 200ms Thread 0x0021 *** C R A S H I N G *** wine: Unhandled page fault on write access to 0x00000000 at address 0x1400010b3 (thread 0021), starting debugger... Thread 0x0022 *** C R A S H I N G *** Thread 0x0023 *** C R A S H I N G *** WineDbg starting on pid 001f Thread 0x0020 *** C R A S H I N G *** First chance exception: page fault on write access to 0x00000000 in 64-bit code (0x00000001400010b3). Register dump: rip:00000001400010b3 rsp:00007feb8997fbb0 rbp:0000000140001a20 eflags:00010202 ( R- -- I - - - ) rax:0000000000000000 rbx:00007feb8bb55000 rcx:0000000140084478 rdx:0000000140084480 rsi:00007feb8b937401 rdi:00007feb8997fbe0 r8:0000000000000001 r9:000000000010d640 r10:00007feb8997e850 r11:00007feb8997e820 r12:0000000000000000 r13:000000007b86f590 r14:00000001400000d8 r15:00007feb8bb50000 Stack dump: 0x00007feb8997fbb0: 000000014006ed20 0000000000000020 0x00007feb8997fbc0: 00007feb8997fa60 0000000000000028 0x00007feb8997fbd0: 00007feb8997fc30 cccccccc00000020 0x00007feb8997fbe0: 00007feb8997fce0 0000000140001223 0x00007feb8997fbf0: 00007feb8997fc30 0000000000000010 0x00007feb8997fc00: ccccccccffffffff 0000000140001014 0x00007feb8997fc10: cccccccc00000004 00007feb8997fc84 0x00007feb8997fc20: cccccccccccccccc cccccccccccccccc 0x00007feb8997fc30: 000000f0cccccc01 0000000000000000 0x00007feb8997fc40: 000000c8cccccc01 0000000000000000 0x00007feb8997fc50: 000000c8cccccc01 0000000000000000 0x00007feb8997fc60: 000000c8cccccc01 0000000000000000 Backtrace: =>0 0x00000001400010b3 Boom::crash_func+0x83(void_ptr=0x7feb8997fc30, arg_ref=0x7feb8997fc30, tid=32) [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:29] in multiplethreads (0x0000000140001a20) 1 0x0000000140001223 main+0x142(argc=1, argv=0x7feb89771130, thisThread=0xfffffffffffffffe, threadHandle={0x1c, 0x20, 0x24}, threadID=35, oldpri=0, a={{cause_crash=true, delay=240, pointer=0x0(nil)}, {cause_crash=true, delay=200, pointer=0x0(nil)}, {cause_crash=true, delay=200, pointer=0x0(nil)}, {cause_crash=true, delay=200, pointer=0x0(nil)}}) [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:63] in multiplethreads (0x0000000140001a20) 2 0x0000000140001c5c __tmainCRTStartup+0x21b(winver=1281, mainret=0, osver=2600, winminor=1, posvi=0x7feb8ba24070, osplatform=2, managedapp=0, initret=0, winmajor=5) [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c:327] in multiplethreads (0x0000000140001a20) 3 0x0000000140001a2e mainCRTStartup+0xd() [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c:195] in multiplethreads (0x0000000140001a20) 4 0x000000007b86f64f start_process+0xbe(peb=0x7feb8bb55000) [/home/eric/work/wine/dlls/kernel32/process.c:963] in kernel32 (0x0000000140001a20) 5 0x00007feb8abfe43e call_thread_func+0x6d(entry=<is not available>, arg=<is not available>, frame=<is not available>) [/home/eric/work/wine/dlls/ntdll/signal_x86_64.c:3036] in ntdll (0x0000000140001a20) 6 0x00007feb8abfd73e call_thread_entry_point+0x29() in ntdll (0x0000000140001a20) 7 0x00007feb8abd0da6 start_process+0x15(kernel_start=0x7feb8997fec8) [/home/eric/work/wine/dlls/ntdll/loader.c:2610] in ntdll (0x0000000140001a20) 8 0x00007feb8b611ea3 wine_call_on_stack+0x12() in libwine.so.1 (0x0000000140001a20) 9 0x00007feb8b611eb9 wine_switch_to_stack+0x8(func=0x7feb8997fec8, arg=0x0(nil), stack=0x140084480) [/home/eric/work/wine/libs/wine/port.c:83] in libwine.so.1 (0x0000000140001a20) 10 0x00007feb8abd5294 LdrInitializeThunk+0x3d3(kernel_start=0x7b86f590, unknown2=<is not available>, unknown3=<is not available>, unknown4=<is not available>) [/home/eric/work/wine/dlls/ntdll/loader.c:2666] in ntdll (0x0000000140001a20) 11 0x000000007b874185 __wine_kernel_init+0x914() [/home/eric/work/wine/dlls/kernel32/process.c:1163] in kernel32 (0x0000000140001a20) 12 0x00007feb8abd5824 __wine_process_init+0x253() [/home/eric/work/wine/dlls/ntdll/loader.c:2876] in ntdll (0x0000000140001a20) 13 0x00007feb8b610a0f wine_init+0x28e(argc=<is not available>, argv=0x7fff9949d238, error="", error_size=<is not available>) [/home/eric/work/wine/libs/wine/loader.c:711] in libwine.so.1 (0x0000000140001a20) 14 0x000000007bf00cf1 main+0x80(argc=2, argv=0x7fff9949d238) [/home/eric/work/wine/loader/main.c:218] in <wine-loader> (0x0000000140001a20) 15 0x00007feb8b087c4d __libc_start_main+0xfc(main=<is not available>, argc=<is not available>, ubp_av=<is not available>, init=<is not available>, fini=<is not available>, rtld_fini=<is not available>, stack_end=0x7fff9949d228) [/build/buildd/eglibc-2.11.1/csu/libc-start.c:226] in libc.so.6 (0x0000000140001a20) 16 0x000000007bf00ba9 _start+0x28() in <wine-loader> (0x0000000140001a20) 0x00000001400010b3 Boom::crash_func+0x83 [z:\eyeon\u_drive\projects\winedbg_testcases\multiplethreads\multiplethreads.cpp:29] in multiplethreads: movq %rax,0x0000000000000000 Unable to open file '' which is closer to what one should expect. A+
-- Eric Pouech "The problem with designing something completely foolproof is to underestimate the ingenuity of a complete idiot." (Douglas Adams)
participants (4)
-
Eric Pouech -
GOUJON Alexandre -
Peter Urbanec -
Tom Grubbe