https://bugs.winehq.org/show_bug.cgi?id=39512
Bug ID: 39512 Summary: Make it possible to run multiple tests in a single task Product: Wine-Testbot Version: unspecified Hardware: x86 OS: Linux Status: NEW Severity: normal Priority: P2 Component: unknown Assignee: wine-bugs@winehq.org Reporter: fgouget@codeweavers.com Distribution: ---
Currently when a patch touches multiple tests the TestBot creates one task per test per VM, and even multiple jobs if the patch touches tests for multiple dlls.
For instance currently Wine patches are tested on four 32 bit VMs and five 64 bit VMs. So a patch that modified 2 tests in one dll and 3 in another would create 2 jobs and 72 tasks (2 build tasks, (2+3)*(4+5) 32 bit tasks, and (2+3)*5 64 bit tasks). The overall run time would obviously be quite long, thus earning this type of (otherwise quite legitimate) patch the 'test bomb' nickname.
The fix is to take the ability to build a WineTest executable with only a few tests (see bug 39065) further so that: * A WineTest executable containing only the modified tests is built. * The TestBot then creates only one job and one task per VM to run that test executable. * Just like in bug 39065 the tasks would have to specify a command line such that only the modified tests are run. That's not just for efficiency's sake but also to avoid getting test failures from unmodified tests. * Instead of having a constant 2 minutes timeout the tasks would also need to have a timeout that's in line with the number of tests to run. This could be something like: MIN(tests_count, 2) * 2 minutes + average_time * tests_count
Where average_time would be some configurable constant (probably something around 3 to 5 seconds).
https://bugs.winehq.org/show_bug.cgi?id=39512
--- Comment #1 from François Gouget fgouget@codeweavers.com --- This would also allow fixing the dependency issues (see bug 31789): * Do a global rebuild, relying on Wine's build system dependencies to figure out what to rebuild. * See which test object files were modified. * Build a WineTest executable that only includes the relevant test executables and somehow bake into it the exact list of tests to run.
A side effect would be that while before changing a header file in /include would not have had any impact on which test to run, with this system we may end up re-running every single tests (which might actually be the right thing to do, though rarely).
https://bugs.winehq.org/show_bug.cgi?id=39512
--- Comment #2 from François Gouget fgouget@codeweavers.com --- The situation has changed a bit since the bug was opened: * Merging TestLauncher and WineTest is no longer a goal. * The TestBot now creates a single job no matter how many test modules (aka dll or program) have been modified which cuts down on the number of build tasks (only a small improvement from the too many tasks point of view).
So another solution that would not require special WineTest builds would be to simply upload all the test binaries, TestLauncher, and then send a batch file that runs TestLauncher for each test unit in sequence.
TestLauncher enforces the test unit timeout in the same way as WineTest so this would not be so different. There is also less risk of ending up with a bloated test binary with TestLauncher. In particular we would not be downloading the extra WineTest code from the build VM for each job.
https://bugs.winehq.org/show_bug.cgi?id=39512
François Gouget fgouget@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |WONTFIX
--- Comment #3 from François Gouget fgouget@codeweavers.com --- Running only one test on a VM before reverting that VM has a big advantage: test isolation. Also the TestBot is pretty much keeping up these days and where it does not it seems like running multiple VMs simultaneously would be a better option (bug 47855).
Note that for Wine tests we do run all the tests in one go since that allows us to do the, potentially pretty long, compilation only once. But this goes through a different mechanism.
Since this 'improvement' is not really in the cards anymore I'm closing this bug.
https://bugs.winehq.org/show_bug.cgi?id=39512
--- Comment #4 from Austin English austinenglish@gmail.com --- Closing.
https://bugs.winehq.org/show_bug.cgi?id=39512
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED
--- Comment #5 from Austin English austinenglish@gmail.com --- Closing.
-
wine-bugs＠winehq.org -
WineHQ Bugzilla