TL;DR; - WineTest should no longer hit the 1.5 MB report size limit. - WineTest results have way too many NEW errors (bold orange lines): https://testbot.winehq.org/JobDetails.pl?Key=65643#k108 Fix them or at least make them not new as shown below. - There is work for everyone!
* Running WineTest on Wine regularly produced reports above the 1.5 MB limit. -> The spam-limiting patch seems to have resolved this issue as no report has been over the limit since it was committed (e4dbd567).
* WineTest sometimes hits the 30 minutes timeout on Windows which prevents it from reporting the results to test.winehq.org. Lately it has mostly impacted the w1064v1809* VMs. -> Where does WineTest spend time? Why? Can it be sped up?
* There are a number of cases where the TestBot and WineTest either skip a test because they think some dll is missing, or try to run the test despite some missing dependency (usually and indirect one), causing a 2 minutes stall. https://bugs.winehq.org/show_bug.cgi?id=48208 https://bugs.winehq.org/show_bug.cgi?id=31609
* WineTest: get_subtests() fails randomly https://bugs.winehq.org/show_bug.cgi?id=48061 - Some of these are caused by the previous issue. - But others are totally random and make no sense like when WineTest claims that on some days the w1064v1809-hi snapshot does not have opengl32 or winhttp! https://test.winehq.org/data/6b35cd2903bc9842ce54cee02caee2106625091d/win10_... https://test.winehq.org/data/be2b0b1cec5843f0145f376316d6c28507559910/win10_... -> This causes less trouble now that the TestBot better deals with intermittent failures. But it still causes some dlls to go completely untested on some days.
* Fix "always new" failure messages. - These will cause false positives because the TestBot can never match them to a previous occurence in the WineTest results. - To identify them pick any WineTest job and look for new errors. For instance: winstation.c:973: Test failed: unexpected foreground window 00000000000A01AC dxgi.c:5063: Test failed: Got unexpected message 0x31f, hwnd 00000000003E002C, wparam 0x1, lparam 0
- Then check past WineTest reports for that VM configuration, either on the TestBot or on test.winehq.org and try to figure out why the failure messaege is different. Usually it will be because the message contains a handle or pointer value. If that value is not really important remove it from the failure message.
-> Otherwise some form of the following patch may help: https://www.winehq.org/pipermail/wine-devel/2019-December/157168.html https://www.winehq.org/pipermail/wine-devel/2019-December/157169.html https://bugs.winehq.org/show_bug.cgi?id=48209
* Fix rare intermittent failures / those that never happen in WineTest. - The TestBot can only identify preexiting failures if they are present in one of the ~20 past WineTest results it has for each VM configuration. According to my calculations this means that if the failure rate is below 6.7% and the failure happens for your patch, there is a 25% chance the TestBot will blame you. - Example: xaudio2.c:67: Test failed: Callbacks called out of order: 1 -> Happened twice in ~60 (newtb*) to ~140 (all) runs.