Am 07.09.2011 06:14, schrieb Dan Kegel:
Hi folks, I've been running "make test" a lot lately, and occasionally (it happened five times today), a test will fail out of the blue, in one out of twenty or a hundred runs. It would be bad form for buildbot to reject a patch because of a spurious test failure, so I've been making a list of tests that fail, adding to it each time I run into a test that fails occasionally, and skipping the tests on that list during "make test". This is all fine... but one of the tests that fails occasionally is ddraw/visual.ok (see http://bugs.winehq.org/show_bug.cgi?id=28301 ). It really doesn't feel right to skip that entirely just because a few particular pixel comparisons fail sometimes.
Plan A would be to fix the tests to not be flaky, but that's hard, and the buildbot needs a quick solution.
Plan B might be to conditionally compile the flaky tests away, e.g.:
Plan C could be to do what testbot does and remember possible fails, maybe including the out of the blue ones.