On Wed, 4 May 2022, Rémi Bernon wrote: [...]
When a patch is submitted that is detected as potentially touching more than a single test, all the tests for the module are queued for testing. However this isn't done through WineTest, and instead they are all queued and tested separately, at least on the Windows VMs.
Wouldn't it be better to always run the tests through WineTest, and make it run all the tests that need checking at once?
That would certainly be more efficient time-wise.
Network-wise there's the issue that WineTest.exe is big because it always contains all the tests so it would cause more network traffic. But the traffic issue is probably minor and it is probably possible to tweak the builds to reduce the size.
But as you mentionned, the main issue is that the tests could interfere with each other which so far has been regarded as "polluting" the results. But we could see things differently.
Another question, unrelated to the performance problems, could we consider adding more Desktop/WM environments to the Debian VMs? I think it could be interesting to have to track down winex11 bugs, though it's probably likely to have several broken tests.
So far the main goal has been to avoid failures so the desktop environment has been optimized with that in mind (so fvwm with a carefully crafted configuration).
But again things have changed since then. Most importantly the TestBot can now distinguish old failures from new ones and I'm still working towards having a way to prevent the "always new" failures from causing false positives.
With those two in place running the tests in configurations known to cause failures is less of an issue.
One way support for multiple desktop environments could be done in the current framework would be to have one Linux VM per desktop environment. However that means compiling once per test environment which has an impact on performance. With a fast new server (or servers) that could work though.
The alternative would be to install multiple desktop environments in the same Debian VM (easy) and have the client-side TestBot script switch from one desktop environment to another based on the configuration to test. I'm not sure how that would work though.
I also intend at some point, when win32u conversion will be more settled, to finish sending my nulldrv patches, and I think it'd be nice to have a testbot flavor that could be configured to use it instead of the default graphics driver.
It sounds like that's just a matter of configuring the test environment to use nulldrv instead of the regular graphics driver (including possibly unsetting $DISPLAY). So that would be a bit like setting the locale and could probably be done through the missions mechanism without requiring a separate test environment.