On modern CPUs, with a 30 second timeout per test, the wine conformance test suite runs in about 4-5 minutes. That's wonderfully fast, completely sufficient for all purposes...
... except one: running tests under valgrind, which imposes a roughly 10x penalty. 40-50 minutes to run the tests is a bit too long to expect developers to wait for feedback.
So I had another try at getting conformance tests to run in parallel, this time without any code modifications. I use WINETEST_WRAPPER="$HOME/alarm 30", where alarm is the output of compiling http://winezeug.googlecode.com/svn/trunk/patchwatcher/alarm.c
That file has a hardcoded whitelist of tests that are know to not call CreateWindow (directly or indirectly) or do anything else that might not work well in parallel. alarm takes an exclusive lock when running any test not on that list. Even with this fairly restrictive whitelist, the tests were fairly parallelizable (see below).
Because running tests in parallel is tricky, and makes our already somewhat shaky test suite less stable, I will run the tests the normal way first, and only if they pass without regressions will I then do a second run under Valgrind. And I'll ignore any test failures that happen under valgrind; all I care about in those runs are the valgrind warnings.
Here are some benchmarks on a quad processor:
-j time in seconds 1 255 2 155 3 125 4 105 20 79
With -j4, it looks like valgrind could run the tests in about ten minutes, which would be quick enough.
It would be nice if our test suite really ran in parallel some day, but the approximation I have now is probably good enough. - Dan
p.s.
Out of curiosity, I checked how many tests interfere with themselves if multiple copies are run. To do this, I ran with WINETEST_WRAPPER="$HOME/foowrap.sh" where foowrap.sh was
#!/bin/sh # Run same test three times in parallel $@ & sleep 1 $@ & $@
As you can imagine, that broke a lot of tests, many more than just running in parallel.