TL;DR; - WineTest should no longer hit the 1.5 MB report size limit. - WineTest results have way too many NEW errors (bold orange lines): https://testbot.winehq.org/JobDetails.pl?Key=65643#k108 Fix them or at least make them not new as shown below. - There is work for everyone!
* Running WineTest on Wine regularly produced reports above the 1.5 MB limit. -> The spam-limiting patch seems to have resolved this issue as no report has been over the limit since it was committed (e4dbd567).
* WineTest sometimes hits the 30 minutes timeout on Windows which prevents it from reporting the results to test.winehq.org. Lately it has mostly impacted the w1064v1809* VMs. -> Where does WineTest spend time? Why? Can it be sped up?
* There are a number of cases where the TestBot and WineTest either skip a test because they think some dll is missing, or try to run the test despite some missing dependency (usually and indirect one), causing a 2 minutes stall. https://bugs.winehq.org/show_bug.cgi?id=48208 https://bugs.winehq.org/show_bug.cgi?id=31609
* WineTest: get_subtests() fails randomly https://bugs.winehq.org/show_bug.cgi?id=48061 - Some of these are caused by the previous issue. - But others are totally random and make no sense like when WineTest claims that on some days the w1064v1809-hi snapshot does not have opengl32 or winhttp! https://test.winehq.org/data/6b35cd2903bc9842ce54cee02caee2106625091d/win10_... https://test.winehq.org/data/be2b0b1cec5843f0145f376316d6c28507559910/win10_... -> This causes less trouble now that the TestBot better deals with intermittent failures. But it still causes some dlls to go completely untested on some days.
* Fix "always new" failure messages. - These will cause false positives because the TestBot can never match them to a previous occurence in the WineTest results. - To identify them pick any WineTest job and look for new errors. For instance: winstation.c:973: Test failed: unexpected foreground window 00000000000A01AC dxgi.c:5063: Test failed: Got unexpected message 0x31f, hwnd 00000000003E002C, wparam 0x1, lparam 0
- Then check past WineTest reports for that VM configuration, either on the TestBot or on test.winehq.org and try to figure out why the failure messaege is different. Usually it will be because the message contains a handle or pointer value. If that value is not really important remove it from the failure message.
-> Otherwise some form of the following patch may help: https://www.winehq.org/pipermail/wine-devel/2019-December/157168.html https://www.winehq.org/pipermail/wine-devel/2019-December/157169.html https://bugs.winehq.org/show_bug.cgi?id=48209
* Fix rare intermittent failures / those that never happen in WineTest. - The TestBot can only identify preexiting failures if they are present in one of the ~20 past WineTest results it has for each VM configuration. According to my calculations this means that if the failure rate is below 6.7% and the failure happens for your patch, there is a 25% chance the TestBot will blame you. - Example: xaudio2.c:67: Test failed: Callbacks called out of order: 1 -> Happened twice in ~60 (newtb*) to ~140 (all) runs.
TL;DR; - Part 2 of the todo list, focused on the web server side and TestBot reliability. - False positives caused by intermittent failures should be much rarer. - Two other false positive sources are close to being fixed.
* Avoid false positives when intermittent failures happen. https://bugs.winehq.org/show_bug.cgi?id=47998 -> Done.
* Provide an efficient way to detect new failures on test.winehq.org https://bugs.winehq.org/show_bug.cgi?id=48164 - This is needed to properly monitor the WineTest results and detect regressions so this is a pretty high priority issue. -> Now that the intermittent failures are out of the way I plan to fix a few small bugs that cause false positives and then work on this. (cue jokes about plans and battlefields...)
* Fix handling of child processes. - winetest_wait_child_process() is wrong in many ways. - But the TestBot also needs changes to detect when a child process crashed and avoid generating a false positive. -> Wrote WTBS test cases. The Wine and TestBot patches are in progress.
* Better handle patch series: - Don't mix up different versions of the same patch series https://bugs.winehq.org/show_bug.cgi?id=48353 -> Patches written, need more testing, queued behind other patches. - Don't overwrite existing patch series parts https://bugs.winehq.org/show_bug.cgi?id=47042 -> Patch written and tested.
* Automatically update the WinePrefix if a patch requires it - The TestBot prepares the WinePrefixes in advance so it does not waste time creating them on the fly for each patch. Unfortunately some patches require updating the WinePrefix and this causes them to fail. https://bugs.winehq.org/show_bug.cgi?id=48354 - The first step is to make use of shared Gecko and Mono installs to speed up the WinePrefix creation. Then it may become acceptable to create WinePrefixes on the fly. https://bugs.winehq.org/show_bug.cgi?id=48652 -> I have updated wt-daily to support shared Gecko and Mono installs (so I know how to do it). Now this just needs to be replicated in the TestBot.
* Limit the size of the task reports and logs - A recent infinite loop bug in kernel32:process caused it to generate 140+ MB of test failures. The TestBot then blindly downloaded the reports and parsed them. But besides the disk usage on the TestBot server this made the impacted job details pretty unusable. - So this has shown that it's important for the TestBot to protect itself against such issues. https://bugs.winehq.org/show_bug.cgi?id=48031 - This is also a prerequisite to letting developers set $WINEDEBUG on their jobs: https://bugs.winehq.org/show_bug.cgi?id=47997
* Fix error handling in the WineRun* scripts The current situation makes it hard to highlight the testbot.log errors on the job details page. https://bugs.winehq.org/show_bug.cgi?id=48658
* Add support for com program tests (e.g. chcp.com). https://bugs.winehq.org/show_bug.cgi?id=48090 -> The WTBS has a test case.
Part 3: It's full of VM (or a bit less so).
TL;DR; - The TestBot is operating on a reduced set of VMs. - A lot of VMs need to be restored or updated.
* debiant - This new Debian Testing VM is meant to solve some Mesa issues that plague the debian10 VM. - It has somewhat better results for win32 and somewhat worse for wow64. -> Let me know if it needs tweaking. -> Should it replace debian10 as the main Wine testing platform? -> Should debian10 be kept around?
* wxppro_2scr - Windows was blue-screening shortly after reverting to the live dual-screen snapshot (reverting to the single-screen snapshot still works). This looks like a QEmu issue and recreating the snapshot did not help. - So I changed that configuration to use a powered off snapshot. This means it has to go through the Windows boot every time it is used which should reliable enough these days. This is slower but it is not one of the main test VM configurations so it is not used that often.
* wvista was not producing usable WineTest results anymore - It was also not restored when vm1's disks got replaced. - Restore from backup. - Set up autologin and TestAgentd autostart. - Run Windows Update. - Make sure all troublemakers like Windows Update, Defender, etc are disabled. - Check that WineTest runs fine and investigate if not.
* wvistau64 was not producing usable WineTest results anymore - It was also not restored when vm1's disks got replaced. - Vista's support for Unicode scripts is no good so there it no point testing languages such as Arabic or Hebrew on it. - So restore wvistau64 from a backup made before the languages were installed. This should reduce the qcow2 size and maybe reduce revert times. - Set up autologin and TestAgentd autostart. - Run Windows Update. - Make sure all troublemakers like Windows Update, Defender, etc are disabled. - Reinstall a few simple languages as those should still be testable: en-CA, fr-FR, fr-CA, pt-BR, ru-RU -> Send me your wish list. - Check that WineTest runs fine and investigate if not.
* w7pro64 tended to lose network access - Its role got dropped to winetest, i.e. wine-devel patches are not run on it. - Symptom: network read timed out (wait2/connect:AgentVersion.h:0/9) The test VM has crashed, rebooted or lost connectivity (or the TestAgent server died) - Figure out why or just restore from backup. - Set up autologin and TestAgentd autostart. - Run Windows Update. - Add a dual-screen configuration for Zhiyi. - Check that WineTest runs fine and investigate if not.
* w7u tended to lose network access - It caused false positives and thus got temporarily retired. - Symptom: unable to open 'exe32.report' for reading: No such file or directory The test VM has crashed, rebooted or lost connectivity (or the TestAgent server died) - Figure out why or just restore from backup. - TestAgentd is too old (1.6), preventing capturing cmd errors. - Set up autologin and TestAgentd autostart. - Run Windows Update. - Install some languages, at the very least all the vistau64 ones: en-CA, fr-FR, fr-CA, pt-BR, ru-RU - If the scripts support allows it add more complex languages. - Check that WineTest runs fine and investigate if not.
* w8 crashes while running WineTest - Figure out why or just restore from backup. - Set up autologin and TestAgentd autostart. - Run Windows Update. - Check that WineTest runs fine and investigate if not.
* w2008s64 needs an update for msvcr80 - It lacks the latest version of msvcr80. - Restore from backup. - Set up autologin and TestAgentd autostart. - Run Windows Update -> upgrades msvcr80. - Figure out what to do for the missing mfplat components: https://bugs.winehq.org/show_bug.cgi?id=48430 This could also be fixed by taking care of bug 48208. - Check that WineTest runs fine and investigate if not.
* w1064 - The 1607+ updates were installed too late so that by the time they were added failures had accumulated. So while 1909 is likely to end up above the 70 test unit failures limit it's better to install it as soon as possible. - Restore from the pre-language backup. - Install 1909, preferably from ISO (not available yet) as this limits the size increase of the qcow2 file. - Run Windows Update. - Do not reinstall the languages (see below). - Check that WineTest runs fine and investigate if not.
* w1064 bis - Get a new Windows 10 license and create a new Windows 10 VM. - The reason for this is that we are starting to have too many test configurations for w1064: 5 versions + dual-screen + 8+ languages. Because they are all on the same VM we have to run them all on the same host one at a time, leading to long job run times. Splitting them accross two VMs will allow us to double the test rate. - Make sure the new VM is on the latest Windows 10 version. Don't keep snapshots of the other versions. - Disable all the troublemakers such as Windows Defender, automatic defragmentation, etc. Then take a 'base' snapshot. - Then set up the new VM with the languages that got removed from the above other w1064 VM and some more: https://bugs.winehq.org/show_bug.cgi?id=31785 ar-EG, en-CA, fr-FR, fr-CA, he-IL, ja-JP, ko-KO, pt-BR, ru-RU, zh-CN - Also add Thai, Tibetan, a Deseret script language, a Sinhala script language and Canadian Aboriginal syllabic language. - This still makes a lot of configurations for a single VM. So some locales may be offloaded to the vistau64 or w7u VMs and kept with a winetest or extra role on the w1064b VM so one can always check the behavior on the latest Windows version. - Check that WineTest runs fine and investigate if not. -> The TestBot still seems to keep up with the current set of VMs so the priority is lower than restoring VMs that have been taken out. But as VMs are restored this may become more urgent.
* build - The build VM has become somewhat redundant: the Wine VMs are totally capable of producing the Windows binaries. Plus most jobs will require running the tests on Wine which means the Windows binaries will be available 'for free'. But the Wine VM is more likely to have build failures after the daily Wine commits or to timeout when building or running the tests. So it's probably worth keeping a VM dedicated to producing the Windows binaries. - However instead of being a specific VM it could be a clone of the 'main' Wine VM with just a different IP address and role. If so virt-sysprep may allow automating the conversion of a Wine VM to a build VM and vice-versa. - If not cloned from the current Wine test VM, upgrade it to at least run the same Debian version. Then also reconfigure it to autostart TestAgentd on boot. -> But for now the build VM still seems to work ok so this is low priority.
* cw-gtx560 + Wine - Sometimes has way too many failures in Wine. This seems to mostly happen with the wow32 and wow64 builds these days (yet another nouveau issue?). These need to be investigated and fixed. If the fix is not system-specific it should be replicated to the other systems. https://test.winehq.org/data/errors.html -> Not really part of the TestBot so priority is low.
* cw-rx460 + Wine - The Wine test results were so bad it no longer is running them. Again this needs to be investigated, etc. https://bugs.winehq.org/show_bug.cgi?id=48092 https://bugs.winehq.org/show_bug.cgi?id=48093 -> Not really part of the TestBot so priority is low.
On Wed, 26 Feb 2020, Francois Gouget wrote: [...]
- debiant
-> Let me know if it needs tweaking. -> Should it replace debian10 as the main Wine testing platform? -> Should debian10 be kept around?
- This new Debian Testing VM is meant to solve some Mesa issues that plague the debian10 VM.
- It has somewhat better results for win32 and somewhat worse for wow64.
The Wine patches are now tested on debiant, the new Debian Testing VM, instead of debian10.
Debiant seems to have slightly better results though it's not easy to be sure because of all the random failures (fix your tests!). For now the debian10 VM is still doing the daily WineTest runs to provide a baseline for the current Debian Stable. That may stop at some point.
Also debiant now offers a choice of 75 locales to run the tests in. As usual, these are available when submitting a job on the website: all one has to do is pick the locale for any of the three Wine builds on the VM selection page. Let me know if you're interested in a locale that's not on the list.
Finally the 64 bit VMs can now all run 64 bit tests, even in their locale-specific configuration.
TL;DR; - 4th and final part. - vm1 got new SSDs, no longer times out. - Most items on this list are lower priority and so it may many moons before any of this actually happens.
* One of vm1's hard drive was starting to get bad sectors. Newman replaced it with two SSDs in RAID 1. -> This also fixed the msi:* timeouts on vm1. Yay!
* Not really hardware but still on the hosting side: figure out whether to upgrade the VM hosts to Debian Testing. Or at least the QEmu part. This would get us from QEmu 3.1 to 4.2. Maybe this would help with issues like the wxppro_2scr VM (see part 3) but then the TestBot environment would be less stable.
* Spec out and setup a new VM host to speed up Wine builds and for GPU testing with PCI-passthrough. https://bugs.winehq.org/show_bug.cgi?id=31786 - Ryzen 3900X (lots of threads for Wine builds) - Pick a motherboard that provides sensible IOMMU groupings. - 32 GB of RAM (so we can run up to 6 concurrent VMs). - 1 TB SSD. - Any cheap low power graphics card to run the host's console. - Steal cw-rx460's RX 460 graphics card? - Add a modern Nvidia graphics card later on. - Get a new Windows 10 license so we can set up a new VM dedicated to testing all patches against a 'real' GPU. - Reconfigure the Debian VM to run on this new 'real' GPU too.
* Figure out how to get more than one VM running at a time on the new box without having one VM cause timing issues in the tests running on the others. This also implies: - Making sure we don't try to run to VMs that try to access the same PCI-passthrough graphics card at the same time! - Making sure we don't try to run more VM cores than are available on the host.
* Since the new box will be taking care of the real-AMD GPU tests there will not be much point running WineTest on cw-rx460 anymore. So it could be reconfigured to replace vm2 (i.e overwritten with vm2's backup).
* Retire vm2 - Its CPU is roughly half as fast as the others despite having 8 cores. - For some reason it can no longer run XP (and Vista?) VMs. This followed the Debian 10 upgrade, i.e. it was likely caused by a QEmu upgrade. So it may be that another QEmu upgrade fixes it but it's not really worth bothering.
-
Francois Gouget