April 2014 - Wine-Devel - list.winehq.org

The (broken) state of the WineTestBot
by Francois Gouget April 21, 2014

April 21, 2014

My apologies for this last bout of WineTestBot brokenness. It all started when the VM host froze late last Friday and could not be remotely rebooted. So Newman power-cycled it Monday morning. He also suggested upgrading the kernel in the hope that this would avoid further crashes which I agreed to. Then I decided to upgrade to QEMU 1.7 in the hope of fixing the Dr6 ntdll:exception failures, or at least be in a better position to do so. Of course that entails redoing all the live VM snapshots (1) but I was prepared to do so. Then things went south. The issues ---------- I initially ran into some SELinux incompatibilities and then into QEMU/Libvirt incompatibilities (2). Then the VMs were suspending after a few seconds which turned out to be because the disk was full. My fault: I keep too many VM backups so the new VM backups I created finished filling it. That corrupted some VMs which, ironically, I was able to restore from backup after deleting older backups. Now the host now has a sane amount of free disk space and monitor it closely. But the real problem is that now the VMs get corrupted after a bit of use. This manifests through a couple of symptoms: * Sometimes the build VM will detect EXT4 filesystem corruption, remount '/' as read-only and obviously stop working properly. * Sometimes no filesystem corruption is detected but the content of files gets corrupted. For instance this is what caused all the 'Missing build status line' errors when half of the WineTestBot Build.pl sscript got lost. I suspect this also caused a round of build failures related to memcmp() being missing. * Sometimes the wtbbuild ends up at the grub prompt and complains that it cannot find the filesystem. Since these are live snapshots this is presumably preceded by a crash+reboot of the guest. * Sometimes, after qemu has been properly stopped, a 'qemu-img check' finds a lot of errors. Sometimes not despite the VM being broken. * The Windows 2000 VM also sometimes goes south: it resets and fails to reboot complaining some checksum is corrupted (probably that of the Windows boot loader). Of course QEMU behaves while I'm updating a VM's live snapshot. It's only after I've spent time doing so and creating a backup that it break the VM (thus also casting doubt into the trustworthiness of the new backup). Further compounding the problems, once the WineTestBot is on a tear compiling a bunch of patches on the lone build VM it's unstoppable (3). It also tends to get stuck whenever I restart libvirt or the VM host (a known issue I was working on before this episode). The strange thing is that QEMU 1.7 seems to work fine on my test environment. However it now appears that QEMU has at least three very different codepaths: * User-mode emulation which is slow and should not be used. * kvm_intel, uses the Intel VMX instructions, is used on my (Intel) test environment, has the icebp bug, but seems to otherwise be reliable. * kvm_amd, uses the AMD SVM instructions, is used by the WineTestBot's (AMD) VM host, has the Dr6 bug, and has been unreliable since last weekend. As kvm_adm and kvm_intel are kernel modules, the kernel version might actually be more important than the QEMU one. Indeed my latest tests seem to indicate that reverting the kernel from the 3.13 to 3.2.0 fixes the VM corruption issues (I also tested 3.14rc7 which did not help). A corrollary is that any QEMU tests I can do in my home environment are likely to be poor predictors for what will happen on the production VM host. Indeed it's because QEMU 1.7.0 seems to work fine here that I decided to upgrade the VM host. Short term goal --------------- Restore the WineTestBot to a working state! The current hope is that just reverting to a pre-3.13 kernel, maybe 3.2.0 will do the trick. That would make it possible to stick with QEMU 1.7.0 (Debian 7.0 (Stable/Wheezy) has 1.1.2 which is really too old but Wheezy-Backports has moved on to 1.7.0 so there's no easily accessible 1.6.0 packages to go back to). Longer term goals ----------------- * Solve the VM host crashes. The memory has been tested quite extensively already, and I did a badblocks pass on the hard-drive. Neither found anything. I could then test the process using PrimeNet. but the MRTG graphs did not indicate a tendency to overheat or other such problems. It now seems the crashes may be caused by the 3.2.0 kernel, and probably specifically by the kvm/kvm_amd modules. So finding a more recent kernel that actually works might help. * While I hope the VM host will never crash again, it would be nice to be able to remotely power-cycle it. * It will also be necessary to fix the stability issues in the 3.13 and 3.14 kvm_amd module. However given that I don't have an AMD box at hand and already way too many other things to do I don't see how that's going to happen. Maybe through a bug report once I get a better handle on this. Still the VM host cannot remain stuck on 3.2.0 indefinitely. * Given that on AMD the Dr6 bug still seems to be present in QEMU 2.0/Linux 3.13, it still needs to be fixed. (And the icebp one on Intel would be nice too). * Fix the mysterious 'network timeout' errors we get while waiting for a WineTest task to complete. Unfortunately the first set of patches to tackle them were not really conclusive. So maybe switch to plan B, i.e. blindly reconnect to work around them. That would be quite unsatisfying though. * Then resume work on making the WineTestBot Engine (more) resilient to network outages and VM host crashes. I started patches for that but working on them it became clear that this was entangled with proper handling and diagnostics after 'network timeout' errors. * Then resume work on all the other features and bugs of the WineTestBot. (1) QEmu 1.7.0 cannot restore a 1.6.0 live snapshot made in qemu-system-x86_64 https://bugs.launchpad.net/qemu/+bug/1259499 (2) A known issue caused by QEMU changing the 'qemu-system-x86_64 -cpu help' output format which is parsed by Libvirt to figure out which kinds of CPUs can be emulated. (3) Bug 35946 - Cannot mark a VM for maintenance if it is running a task http://bugs.winehq.org/show_bug.cgi?id=35946 -- Francois Gouget <fgouget(a)codeweavers.com>

3 5

Re: msvcrt: Remove wine-specific exports
by Alexandre Julliard April 21, 2014

April 21, 2014

Piotr Caban <piotr(a)codeweavers.com> writes: > --- > dlls/msvcr100/msvcr100.spec | 4 ++-- > dlls/msvcr110/msvcr110.spec | 4 ++-- > dlls/msvcr70/msvcr70.spec | 4 ++-- > dlls/msvcr71/msvcr71.spec | 4 ++-- > dlls/msvcr80/msvcr80.spec | 4 ++-- > dlls/msvcr90/msvcr90.spec | 4 ++-- > dlls/msvcrt/mbcs.c | 7 ++---- > dlls/msvcrt/msvcrt.spec | 53 > --------------------------------------------- > dlls/msvcrt/string.c | 6 +++-- > 9 files changed, 18 insertions(+), 72 deletions(-) It doesn't work: ../../../tools/runtest -q -P wine -T ../../.. -M msvcrt.dll -p msvcrt_test.exe.so string && touch string.ok wine: Unhandled exception 0xc0000417 in thread 62 at address 0x7b83aa0e (thread 0062), starting debugger... Unhandled exception: 0xc0000417 in 32-bit code (0x7b83aa0e). Register dump: CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b EIP:7b83aa0e ESP:0032f9a4 EBP:0032fa18 EFLAGS:00000246( - -- I Z- -P- ) EAX:7b826b79 EBX:7b8b5dac ECX:00000000 EDX:00000000 ESI:c0000417 EDI:00000000 Stack dump: 0x0032f9a4: 0032f9c4 009d0152 7bcc4514 c0000417 0x0032f9b4: 00000001 00000000 7b83aa0e 00000000 0x0032f9c4: 0032fa40 687415e1 68718cdf 00b100b0 0x0032f9d4: 00b300b2 039c00b4 681c8cc0 00000001 0x0032f9e4: 687415e0 0032fa28 6802d1bc 00000001 0x0032f9f4: 687415e0 68718cdf 68718c75 0032fa40 Backtrace: =>0 0x7b83aa0e RaiseException+0x5e(code=<couldn't compute location>, flags=<couldn't compute location>, nbargs=<couldn't compute location>, args=<couldn't compute location>) [/home/julliard/wine/wine/dlls/kernel32/except.c:84] in kernel32 (0x0032fa18) 1 0x686c9b5d MSVCRT__invalid_parameter+0x8c() in msvcrt (0x0032fa78) 2 0x686da0c8 MSVCRT_memcpy_s+0x87(dest=0x6869112c, numberOfElements=0x8, src=0x68688415, count=0xf) [/home/julliard/wine/wine/dlls/msvcrt/heap.c:645] in msvcrt (0x0032fac8) 3 0x6865d547 test_memcpy_s+0x1f6() [/home/julliard/wine/wine/dlls/msvcrt/tests/string.c:581] in msvcrt_test (0x0032fb48) 4 0x686686c6 func_string+0xe55() [/home/julliard/wine/wine/dlls/msvcrt/tests/string.c:2737] in msvcrt_test (0x0032fd48) 5 0x6862b021 main+0x2f0(argc=<is not available>, argv=<is not available>) [/home/julliard/wine/wine/dlls/msvcrt/tests/../../../include/wine/test.h:584] in msvcrt_test (0x0032fe18) -- Alexandre Julliard julliard(a)winehq.org

1 0

Re: typo fixes - rebased - fixed
by Marvin April 20, 2014

April 20, 2014

Hi, While running your changed tests on Windows, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at https://testbot.winehq.org/JobDetails.pl?Key=6455 Your paranoid android. === w7u (32 bit msg) === msg.c:12058: Test failed: unexpected hwnd 0017014A msg.c:12080: Test failed: bad time 310a2/310b2

1 0

Re: typo fixes - rebased - fixed
by Marvin April 20, 2014

April 20, 2014

Hi, While running your changed tests on Windows, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at https://testbot.winehq.org/JobDetails.pl?Key=6454 Your paranoid android. === w2008s64 (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w7u (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w864 (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w2008s64 (64 bit moniker) === moniker: unhandled exception c0000005 at 000000000042024A === w864 (64 bit moniker) === moniker: unhandled exception c0000005 at 000000000042024A

1 0

Re: [PATCH 2/6] wined3d: Pass the volume texture level to the client libraries.
by Henri Verbeet April 20, 2014

April 20, 2014

On 17 April 2014 15:47, Stefan Dösinger <stefan(a)codeweavers.com> wrote: > The goal is to remove volumes and surfaces from the wined3d public API. > This extra information is needed for the transition. We got rid of this not too long ago, adding it back doesn't sound very attractive. If you really need a way to find out a volume's level, I'd prefer wined3d_volume_get_level() instead, mostly because it doesn't need changes in every D3D library. I'm not convinced this is really necessary though, or that starting in d3d9 as opposed to wined3d with this is necessarily the right approach.

2 5

Re: typo fixes - rebased - fixed
by Marvin April 20, 2014

April 20, 2014

Hi, While running your changed tests on Windows, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at https://testbot.winehq.org/JobDetails.pl?Key=6447 Your paranoid android. === w7u (32 bit msg) === msg.c:12058: Test failed: unexpected hwnd 00070140 msg.c:12080: Test failed: bad time 3118c/3119c

1 0

Re: typo fixes - rebased - fixed
by Marvin April 20, 2014

April 20, 2014

Hi, While running your changed tests on Windows, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at https://testbot.winehq.org/JobDetails.pl?Key=6446 Your paranoid android. === w2008s64 (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w7u (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w864 (32 bit moniker) === moniker: unhandled exception c0000005 at 004256A9 === w2008s64 (64 bit moniker) === moniker: unhandled exception c0000005 at 000000000042024A === w864 (64 bit moniker) === moniker: unhandled exception c0000005 at 000000000042024A

1 0

Re: typo fixes - rebased - fixed
by Nikolay Sivov April 20, 2014

April 20, 2014

> - * INT oDefautlValue[nrargs]; > + * INT oDefaultlValue[nrargs]; This is wrong.

1 0

Re: [PATCH 10/10] vbscript/tests: Added tests for convert functions
by Marvin April 18, 2014

April 18, 2014

Hi, While running your changed tests on Windows, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at https://testbot.winehq.org/JobDetails.pl?Key=6431 Your paranoid android. === w7u (32 bit run) === run.c:973: Test failed: api.vbs: L"CStr(CDate(1.5)) = 1899/12/31 12:00:00" run.c:973: Test failed: api.vbs: L"CStr(CDate(-1.5)) = 1899/12/29 12:00:00"

1 0

Re: mshtml: Added IHTMLTable::bgColor property implementation.
by Nikolay Sivov April 18, 2014

April 18, 2014

On 4/6/2014 18:48, Zhenbo Li wrote: > try 2: > Don't free a string too early. > > --- > dlls/mshtml/htmltable.c | 41 +++++++++++++++++++++++++++++++++++++---- > dlls/mshtml/tests/dom.c | 35 +++++++++++++++++++++++++++++++++++ > 2 files changed, 72 insertions(+), 4 deletions(-) > Also please check for variant_to_nscolor() return value.

2 1