On 5/5/22 10:37, Francois Gouget wrote:
On Wed, 4 May 2022, Zebediah Figura (she/her) wrote: [...]
This seems worth looking into.
Given that almost all the CPU performance gains come from high core counts nowadays I agree that it would be nice. But...
There's also a lot of tests that can't really be improved by multithreading *internally*, but also don't touch global state and hence could be run in parallel with anything else. We could construct a whitelist (or maybe there's even enough to construct a blacklist instead) of tests that winetest can run in parallel.
I don't think there's a way to automatically detect which test units can be run in parallel or even to have a heuristic that reliably identifies a subset that are safe to parallelize. (and reciprocally for a whitelist)
That means we'd need a handcrafted whitelist or blacklist and I'm not sure how maintainable that would be:
- A blacklist has the drawback that we'd always be playing catchup to add new tests.
- But I'm not even sure a whitelist would work better: any patch to a whitelisted test may require evicting it from the whitelist.
Obviously it'd have to be maintained manually, but my idea is that we can work towards a blacklist, and try to make it as small as possible, probably by gradually making tests parallelizable, and then by adding the requirement that new changes to tests avoid breaking that. Which I don't think is an unreasonable requirement to have or enforce.
Ultimately I don't think it'd be that bad, either.
Off the top of my head, tests I can think of that inherently can't be parallelized:
* MSI installation tests (msi:action and msi:install, although not msi:db and msi:format); Windows only allows one installer to be run at once.
* dinput and ntoskrnl tests, I think? Probably also setupapi? I'm not sure these couldn't be made independent of each other, but it's probably easier not to try.
* Tests which change display mode (some ddraw, d3d8, d3d9, dxgi, user32:sysparams). In many cases these are put into test units with other d3d tests which *are* parallelizable, but they could be split out. Although, that said:
* d3d tests in general are an odd case. We can't parallelize them if we might run out of GPU memory, although that hasn't been a concern yet and it won't be for llvmpipe. We also can't parallelize them on nouveau because of its threading problems. There are also a few tests that *shouldn't* break other tests but do because of driver bugs.
* Tests which care about the foreground window. In practice this includes some user32, d3d, dinput tests, probably others. Often it's only a couple of tests functions out of the whole test. (I wonder if we could improve things by creating custom window stations or desktops in many cases?)
* Tests which warp the cursor or depend on cursor position. This ends up being about the same set.
A quick skim doesn't bring up any other clear cases of tests that can't be parallelized. There are probably still a lot that need auditing and perhaps extra work to ensure that they can be parallelized, but I think that's work worth doing.
There are a decent number of tests that hardcode temporary file paths but could be made to use GetTempFileName() instead. Actually most such tests already use GetTempFileName(), I guess in order to be robust.
There's also a lot of tests that do touch global state, e.g. write to the registry, but don't write to parts of the registry that any other tests should be reading. advapi32:registry is one such example. Such tests can't run in parallel with *themselves*, but could be run in parallel with anything else.
(Stuff like services.exe or advapi32:service might fall into the same boat. These do touch global state, but in theory other tests shouldn't care that we're e.g. creating an advapi32 test service. Definitely easier just to blacklist those, though, at least to start with...)
One other thing that occurs to me while writing this is that instead of using a blacklist in winetest.exe, we could use global (win32) mutex objects in the relevant tests. That would also allow us to separate tests which can't be paralellized only with a set group of other tests (and run e.g. msi:action and d3d8:device at the same time), as well as have finer grained control than blacklisting a whole test file (e.g. we could grab WINETEST_DISPLAY_MODE_MUTEX around only the d3d8:device tests that mess with the display mode).