Would something like this be useful to include in Wine? If that's deemed interesting I can clean it up and submit it. I'm posting it here early in case it's useful for WineConf.
It's a tool which you can use with git bisect to find the patch that causes a regression in a conformance test. It's used like this:
git bisect start <bad> <good> git bisect run ./tools/testbisect dlls/gdi32/tests bitmap
Come back a bit later and you have the bad patch.
It can also be used with flaky tests by specifying the number of times to run a test. So if a test fails in 30% of the time you could only consider that it works if it succeeds 10 times in a row. You can even grep for a specific error if the test has other errors you don't care about.
Finally you can also use it without git just to figure out if a test is flaky or not. For instance:
$ ./tools/testbisect dlls/random/tests flaky 100 100 Failed 42 time(s) in 100 run(s)
-> definitely flaky
-
Francois Gouget