http://bugs.winehq.org/show_bug.cgi?id=15206
Summary: Trouble with symbolic stack dumps and Chromium Product: Wine Version: CVS/GIT Platform: Other URL: http://build.chromium.org/buildbot/snapshots/chromium- rel-xp/ OS/Version: other Status: NEW Keywords: download, source Severity: normal Priority: P2 Component: dbghelp AssignedTo: wine-bugs@winehq.org ReportedBy: dank@kegel.com
Chromium has nightly builds with pdb files at http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/ and if you install corefonts, then current wine can install and run (though for me chromium hangs when loading web page).
However, getting symbolic stack dumps (sometimes?) fails. To get wine to find the symbol files, put them in the current directory (or the directory they were in when they were built, or a directory on Wine's PATH). But wine complains
trace:dbghelp_msc:pdb_init PDB(Z:\home\dank\snapshots\1712\chrome-win32-syms\chrome_dll.pdb): "Microsoft C/C++ MSF 7.00\r\n\x1aDS\x00\ trace:dbghelp_msc:pdb_init found DS/f for Z:\home\dank\snapshots\1712\chrome-win32-syms\chrome_dll.pdb: age=30 guid={c007e080-dbde-4fd9-a8dd-ae84cd0524a9} warn:dbghelp:module_find_cb Found L"Z:\home\dank\snapshots\1712\chrome-win32-syms\chrome_dll.pdb", but wrong age: 00000030 0000002c
(Does that mean the .pdb files are mismatched, or is this a wine bug?)
This patch disables the "age" check and seems to let symbolic stacks to be produced:
--- a/dlls/dbghelp/path.c +++ b/dlls/dbghelp/path.c @@ -560,7 +560,7 @@ static BOOL CALLBACK module_find_cb(PCWSTR buffer, PVOID use } if (pdb_lookup.age != mf->dw2) { - matched--; + //matched--; WARN("Found %s, but wrong age: %08x %08x\n", debugstr_w(buffer), pdb_lookup.age, mf->dw2); }
although now I don't seem to need the patch, so maybe I'm smoking crack.
Also, after about 12 lines of backtrace, winedbg seems to get stuck in an infinite loop. And before the backtrace there are hundreds of thousands of lines like fixme:dbghelp_msc:codeview_snarf Unsupported symbol id 1012
FWIW, here's my script to reproduce:
#!/bin/sh set -xe # This snapshot crashed for me; normally you'll want the latest one SNAPSHOT=1712
export WINE=$HOME/wine-git/wine export WINEPREFIX=$HOME/.wine cd $HOME mkdir -p pdb-bug cd pdb-bug
# Grab and unpack a snapshot of Chromium including debugging symbols wget -c http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/$SNAPSHOT/chrom... test -d chrome-win32 || unzip chrome-win32.zip wget -c http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/$SNAPSHOT/chrom... test -f chrome_exe.pdb || unzip -j chrome-win32-syms.zip
# Clean up .wine and install corefonts rm -rf $WINEPREFIX rm -f winetricks wget http://kegel.com/wine/winetricks sh winetricks -q corefonts
$WINE chrome-win32/chrome.exe --no-sandbox --single-process --disable-breakpad http://www.kegel.com 2>&1 | tee wine.log echo 'Now look at $PWD/wine.log to see whether a symbolic backtrace was produced.' echo 'Without the patch, it (sometimes?) fails with WARN "Found %s, but wrong age".'
http://bugs.winehq.org/show_bug.cgi?id=15206
Eric Pouech eric.pouech@orange.fr changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |eric.pouech@orange.fr URL| |http://build.chromium.org/bu |http://build.chromium.org/bu%7Cildbot/snapshots/chromium- |ildbot/snapshots/chromium- |rel-xp/ |rel-xp/ |
--- Comment #1 from Eric Pouech eric.pouech@orange.fr 2008-09-09 15:06:25 --- currently winedbg still forces a pdb file to have correct timestamp (or guid) and age
timestamp (or guid) is generated the first time the pdb file is created the age starts at 1 and gets incremented at each recompilation of the project (without erasing the .pdb file)
so for consistency reasons, it's still usefull to force timestamp & age mapping between module and .pdb file
the good news is that now dbghelp knows how to load .pdb files which don't fully match.
the attached patch let winedbg turn on this feature (it's off by default) set ! module + is the command to be used in winedbg (I'm still unhappy with this patch, so still looking for a better solution)
you can very easily check that they match (tried the build 1712 and they match) do $WINE/tools/winedump/winedump -j debug chrome-win32/chrome.exe and $WINE/tools/winedump/winedump chrome_exe.pdb Age is the same (30 in my case, ditto for .dll). maybe there are older .pdb is the path ? (IIRC we do a depth first search first, so if you have other .pdb files in your pdb-bug directory, including in sub-sub-sub directories), you may get the error. however the algo should only return the matching file if any (patches committed yesterday)
for the rest, if I remove the flow of unsupported types (102c, 1137, 1139), I get err:ntdll:RtlpWaitForCriticalSection section 0x7d9679b4 "?" wait timed out in thread 0009, blocked by 0000, retrying (60 sec) error, which needs further investigations
will look further next days A+
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #2 from Dan Kegel dank@kegel.com 2008-09-09 17:21:26 --- Grumble. I built a copy of Chromium to get line numbers in the pdb, and now when winedbg triggers, I get an assertion in wine's storage.c:
wine: Unhandled exception 0x80000003 at address 0x10505d3 (thread 0020), starting debugger... err:dbghelp_msc:pdb_process_types -Unknown type info version 20040203 storage.c:86: pool_alloc: Assertion `sizeof(struct pool_arena) + len <= pool->arena_size && len' failed. wine: Assertion failed at address 0xb7fa0410 (thread 0028), starting debugger...
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #3 from Dan Kegel dank@kegel.com 2008-09-09 17:24:35 --- Here's the backtrace from the assertion failure:
Backtrace: =>1 0xb7fa0410 (0x0033e978) 2 0xb7d02a01 abort+0x101() in libc.so.6 (0x0033eaa4) 3 0xb7cfa10e __assert_fail+0xee() in libc.so.6 (0x0033eae8) 4 0x7ec2714e pool_alloc+0x5e(pool=0x15c844, len=<register EDI not in topmost frame>) [/home/dank/wine-git/dlls/dbghelp/storage.c:88] in dbghelp (0x0033eb18) 5 0x7ec27357 vector_add+0x8a(v=<register EDI not in topmost frame>, pool=0x15c844) [/home/dank/wine-git/dlls/dbghelp/storage.c:172] in dbghelp (0x0033eb48) 6 0x7ec2cde7 symt_add_type+0x30(module=<is not available>, symt=<register ESI not in topmost frame>) [/home/dank/wine-git/dlls/dbghelp/type.c:133] in dbghelp (0x0033eb68) 7 0x7ec2d0ab symt_new_function_signature+0x6f(module=<register EDI not in topmost frame>, ret_type=(nil), call_conv=CV_CALL_THISCALL) [/home/dank/wine-git/dlls/dbghelp/type.c:322] in dbghelp (0x0033eb88) ...
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #4 from Dan Kegel dank@kegel.com 2008-09-09 17:32:39 --- len was 64k. This works around the latest problem, now I'm back on looking at the chromium bug:
--- a/dlls/dbghelp/module.c +++ b/dlls/dbghelp/module.c @@ -139,7 +139,7 @@ struct module* module_new(struct process* pcs, const WCHAR* name, get_module_type(type, virtual), mod_addr, mod_addr + size, debugstr_w(name));
- pool_init(&module->pool, 65536); + pool_init(&module->pool, 2*65536);
module->module.SizeOfStruct = sizeof(module->module); module->module.BaseOfImage = mod_addr;
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #5 from Dan Kegel dank@kegel.com 2008-09-09 17:50:58 --- Bug came right back, still overflowed the twice as big arena. Changed it to 1MB instead of 64K. Now it's been running for ten minutes trying to work its way past 150K+ lines of the sort fixme:dbghelp_msc:codeview_snarf Unsupported symbol id 1012 at 99% cpu. Guess I'm hitting quadratic behavior? The .pdb files I'm using are built from scratch here and are twice as big as the ones on the nightly builder.
oprofile says:
samples % image name app name symbol name ------------------------------------------------------------------------------- 28077516 61.2073 dbghelp.dll.so dbghelp.dll.so enum_modW64_64 28077516 100.000 dbghelp.dll.so dbghelp.dll.so enum_modW64_64 [self] ------------------------------------------------------------------------------- 8256676 17.9991 dbghelp.dll.so dbghelp.dll.so module_new 8256676 100.000 dbghelp.dll.so dbghelp.dll.so module_new [self] ------------------------------------------------------------------------------- 1341044 2.9234 no-vmlinux no-vmlinux (no symbols) 1341044 100.000 no-vmlinux no-vmlinux (no symbols) [self]
Not sure I trust oprofile on wine yet, but I guess there's a chance this is telling us where the .pdb loading bottleneck is...?
winedbg is at 15 minutes of cpu time, no backtrace yet, no end in sight...
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #6 from Dan Kegel dank@kegel.com 2008-09-09 17:56:09 --- Whew. At 20 minutes I got the stack dump finally.
No line numbers, still, though. Does Wine's stack dumper support line numbers on apps compiled with visual C++ 2005?
http://bugs.winehq.org/show_bug.cgi?id=15206
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #7 from Eric Pouech eric.pouech@orange.fr 2008-09-10 14:46:53 --- regarding comment #2: - you've hit one of the hard limit of the dbghelp's allocator; bumping the arena size is not a good idea as it's going to suck memory as crazy, as you've noticed afterwards <g> - the cleanest short term fix would be to change the vector handling in storage.c so that it no longer enforce the assert you've seen - the long term and clean fix is to no longer load all symbols & types at once (as we do know), but rather do it on a load on call basis, which will require a major overhall of dbghelp's internals
regarding comment #6: - with earlier MSVC versions, line number information worked fine - didn't check on that one. if you can provide me a small sample, I'll have a look
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #8 from Dan Kegel dank@kegel.com 2008-09-10 15:45:25 --- Created an attachment (id=15995) --> (http://bugs.winehq.org/attachment.cgi?id=15995) Example Visual C++ 2005 sp1 app showing lack of line numbers and fixme spew
OK, here is an app compiled with visual studio 2005 sp1, along with a matching plain old .pdb file, and source code (I removed the .ncb file because it was too big, hopefully nobody cares.)
To reproduce from my precompiled binary:
rm -rf ~/.wine sh winetricks vcrun2005sp1 tar -xjvf stackdemo.tar.bz2 cd stackdemo/release wine stackdemo.exe
One should see line numbers (and does in Visual Studio's debugger), but in Wine one sees Backtrace: =>1 0x0040102a func7+0x2a(arg=-42384329) in stackdemo (0x0032fe64) 2 0x00401076 func6+0x16(arg=-1263462938) in stackdemo (0x0032fe70) ... which is correct except for the missing line numbers, along with a whole load of crap of the sort err:dbghelp_msc:pdb_process_types -Unknown type info version 20040203 fixme:dbghelp_msc:codeview_snarf Unsupported symbol id 1012 fixme:dbghelp_msc:dump 00000000: 1e 00 12 10 04 00 00 00 00 00 00 00 00 00 00 00 ................ fixme:dbghelp_msc:dump 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ fixme:dbghelp_msc:codeview_snarf Unsupported symbol id 1139 fixme:dbghelp_msc:dump 00000000: 0e 00 39 11 0e 00 00 00 01 00 00 00 05 10 00 00 ..9............. which really gets annoying after a while.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #9 from Roy Shea royshea@gmail.com 2008-09-12 12:52:21 --- Created an attachment (id=16043) --> (http://bugs.winehq.org/attachment.cgi?id=16043) Patch allowing variable sized arenas in the pool allocator
(In reply to comment #7)
regarding comment #2:
- the cleanest short term fix would be to change the vector handling in
storage.c so that it no longer enforce the assert you've seen
Attached is a patch implementing this fix. Each arena now tracks a custom size. Most arenas continue to use the default size set by pool_init. Large allocation requests create an arena of custom size to satisfy the request.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #10 from Eric Pouech eric.pouech@orange.fr 2008-09-15 12:55:17 --- I had a look at it (no line number) line number info format has obviously changed, so we need some more reverse engineering to make it work no ETA on this one, until I get an idea of the format (and where it is)
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #11 from Dan Kegel dank@kegel.com 2008-09-15 13:21:27 --- Hmm, http://blogs.msdn.com/matt_pietrek/archive/2005/02/04/367494.aspx mentions the MSDIA sdk as a way of accessing these files. Perhaps fuzzing that would be enlightening.
MSDIA is documented here: http://msdn.microsoft.com/en-us/library/x93ctkx8.aspx and mentioned here https://www.gwydiondylan.org/cgi-bin/bugzilla/show_bug.cgi?id=7078#c11 Looks like MSDIA dlls are included in the visual c++ 2005 runtime redistributable.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #12 from Dan Kegel dank@kegel.com 2008-09-30 22:42:52 --- Eric sent a series of 19 patches that might fix this: http://winehq.org/pipermail/wine-patches/2008-September/062291.html
Hmm, was this bug a dup of bug 835?
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #13 from Austin English austinenglish@gmail.com 2008-10-01 15:09:27 --- (In reply to comment #12)
Eric sent a series of 19 patches that might fix this: http://winehq.org/pipermail/wine-patches/2008-September/062291.html
Hmm, was this bug a dup of bug 835?
Eric's patches went in today.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #14 from Eric Pouech eric.pouech@orange.fr 2008-10-02 14:34:30 --- in reply to #12, no bug 835 is about the jg vs ds pdb file formats (the dbghelp or winedump for the details about those) which are the basic structure of the pdb file (how the files is structured in streams of data) this current bug is about the content of the line info streams (ie content of the streams referring to line number info) A+
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #15 from Eric Pouech eric.pouech@orange.fr 2008-10-20 14:39:48 --- this should be fixed in current git please retest
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #16 from Dan Kegel dank@kegel.com 2008-12-16 21:54:28 --- The stackdemo test works fine, but in the real world, I'm still seeing it, this time with Chromium's unit tests. I'll attach a log.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #17 from Dan Kegel dank@kegel.com 2008-12-16 21:56:45 --- Created an attachment (id=18002) --> (http://bugs.winehq.org/attachment.cgi?id=18002) Log of ui_tests.exe crashing, then winedbg crashing while trying to dump stack?
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #18 from Dan Kegel dank@kegel.com 2008-12-17 17:29:09 --- You can reproduce this yourself 'easily' by building ui_tests on windows, then transferring the Debug directory to your linux box. Or I can send you a 640MB tarball.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #19 from Eric Pouech eric.pouech@orange.fr 2008-12-21 06:36:36 --- still looks like a memory exhaustion issue (we're recreating way to much information from the debug sections, and we've really to rethink the way we handle internal information in dbghelp to cope with such issues). You can send me the tar ball, but I doubt we can simply tweak the current memory allocation A+
http://bugs.winehq.org/show_bug.cgi?id=15206
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Version|CVS/GIT |unspecified
--- Comment #20 from Austin English austinenglish@gmail.com 2009-01-20 02:40:23 --- Removing deprecated CVS/GIT version tag. Please retest in current git. If the bug is still present in today's wine, but was not present in some earlier version of wine, please update version field to earliest known version of wine that had the bug. Thanks!
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #21 from Dan Kegel dank@kegel.com 2009-02-04 09:53:43 --- Looks like Alexandre changed the allocation scheme, http://www.winehq.org/pipermail/wine-cvs/2009-February/052704.html Perhaps this is fixed now, I need to retest.
http://bugs.winehq.org/show_bug.cgi?id=15206
Jerome Leclanche adys.wh@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |adys.wh@gmail.com
--- Comment #22 from Jerome Leclanche adys.wh@gmail.com 2009-06-12 00:38:27 --- (In reply to comment #21)
Looks like Alexandre changed the allocation scheme, http://www.winehq.org/pipermail/wine-cvs/2009-February/052704.html Perhaps this is fixed now, I need to retest.
Any result?
http://bugs.winehq.org/show_bug.cgi?id=15206
Dan Kegel dank@kegel.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED
--- Comment #23 from Dan Kegel dank@kegel.com 2009-06-12 11:09:11 --- Looks better -- there are line numbers now! Here's a sample stack:
Backtrace: =>0 0x02e49c03 DebugUtil::BreakDebugger+0x3() [c:\users\dank\documents\chromium\src\base\debug_util_win.cc:103] in chrome (0x0033cd14) 1 0x01d280e8 SilentRuntimeAssertHandler+0x8(str=*** Invalid address 0x0033cd24 *** ) [c:\users\dank\documents\chromium\src\chrome\common\logging_chrome.cc:41] in chrome (0x0033cd1c) 2 0x02e1f1e6 logging::LogMessage::~LogMessage+0x2f6() [c:\users\dank\documents\chromium\src\base\logging.cc:538] in chrome (0x0033ce88) 3 0x0341c22e ChildProcessHost::Iterator::Iterator+0x1ae(type=*** Invalid address 0x0033d058 *** ) [c:\users\dank\documents\chromium\src\chrome\common\child_process_host.cc:212] in chrome (0x0033d050) ...
Two problems remain: 1) those pesky "Invalid address" notes in the arguments 2) it still takes 20 minutes to handle one backtrace
but I guess those should be filed as separate bugs.
http://bugs.winehq.org/show_bug.cgi?id=15206
Alexandre Julliard julliard@winehq.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED
--- Comment #24 from Alexandre Julliard julliard@winehq.org 2009-06-19 11:00:33 --- Closing bugs fixed in 1.1.24.
http://bugs.winehq.org/show_bug.cgi?id=15206
--- Comment #25 from Dan Kegel dank@kegel.com 2009-10-23 05:50:47 --- See also bug 835; I link to a simple test case there that has examples from several compilers. It looks like there's more work to do -- the pdb support may have bitrotted, or maybe it never handled my simple test case?
-
wine-bugs＠winehq.org