https://bugs.winehq.org/show_bug.cgi?id=39512
Bug ID: 39512 Summary: Make it possible to run multiple tests in a single task Product: Wine-Testbot Version: unspecified Hardware: x86 OS: Linux Status: NEW Severity: normal Priority: P2 Component: unknown Assignee: wine-bugs@winehq.org Reporter: fgouget@codeweavers.com Distribution: ---
Currently when a patch touches multiple tests the TestBot creates one task per test per VM, and even multiple jobs if the patch touches tests for multiple dlls.
For instance currently Wine patches are tested on four 32 bit VMs and five 64 bit VMs. So a patch that modified 2 tests in one dll and 3 in another would create 2 jobs and 72 tasks (2 build tasks, (2+3)*(4+5) 32 bit tasks, and (2+3)*5 64 bit tasks). The overall run time would obviously be quite long, thus earning this type of (otherwise quite legitimate) patch the 'test bomb' nickname.
The fix is to take the ability to build a WineTest executable with only a few tests (see bug 39065) further so that: * A WineTest executable containing only the modified tests is built. * The TestBot then creates only one job and one task per VM to run that test executable. * Just like in bug 39065 the tasks would have to specify a command line such that only the modified tests are run. That's not just for efficiency's sake but also to avoid getting test failures from unmodified tests. * Instead of having a constant 2 minutes timeout the tasks would also need to have a timeout that's in line with the number of tests to run. This could be something like: MIN(tests_count, 2) * 2 minutes + average_time * tests_count
Where average_time would be some configurable constant (probably something around 3 to 5 seconds).