https://bugs.winehq.org/show_bug.cgi?id=47855
Bug ID: 47855 Summary: Run tests in more than one VM at a time Product: Wine-Testbot Version: unspecified Hardware: x86 OS: Linux Status: NEW Severity: normal Priority: P2 Component: unknown Assignee: wine-bugs@winehq.org Reporter: fgouget@codeweavers.com Distribution: ---
Some Wine tests have pretty stringent timing requirements:
* Some audio tests play a short sound bite and make sure the playback lasted for the expected time within a few tens of seconds as a way to ensure audio is played at the right rate. But VMs are not always very good at respecting precise timing requirements when there is other activity on the host.
* Some tests (msi) generate quite a bit of disk traffic and take a while to run. Running two of those simultaneously can sometimes push them over the 2 minutes limit, causing them to time out. That said there may be something else afoot here as these run times vary from ~20s to ~100s depending on the VM for seemingly no reason.
As a consequence the TestBot is configured to only do one thing at a time: run at most one test or revert one VM. This is at odds with hardware evolution and new TestBot requirements:
* SSDs are much faster than hard drives and thus should solve the disk bottleneck issues.
* The builds for the Wine VMs take a lot more time than just producing PE executables for the Windows VMs. To speed up compilation it is necessary to increase the parallelism which means buying a machine with as many cores as possible and assigning as many as possible to the Wine VMs.
* But while running single threaded tests in a single VM on a quad-core machine is bad enough, doing the same thing on a 16+ threads machine would truly be a waste.
So a way to be able to run tests in more than one VM at a time should be found. The first step would be to verify that these issues still exist (this means distinguishing them from the usual random failures). If they don't then problem solved!
Otherwise CPU-pinning is a solution that's usually mentioned in gaming circles. The questions are:
* Would it actually help?
* How to integrate it with the TestBot. Is this something that should be configured through Libvirt when restoring a live snapshot? Or is this something that should be defined 'statically' in the VM's configuration?
* What happens if two VMs are pinned to the same CPU core? Will Libvirt prevent us from starting both at the same time or is it up to the TestBot Engine to make sure this does not happen? How?
* What happens when a VM that's is pinned to CPU core 7 is moved to a 4 core host? Is the pinning ignored or does the administrator have to manually change its configuration?