Hi folks, I've been running "make test" a lot lately, and occasionally (it happened five times today), a test will fail out of the blue, in one out of twenty or a hundred runs. It would be bad form for buildbot to reject a patch because of a spurious test failure, so I've been making a list of tests that fail, adding to it each time I run into a test that fails occasionally, and skipping the tests on that list during "make test". This is all fine... but one of the tests that fails occasionally is ddraw/visual.ok (see http://bugs.winehq.org/show_bug.cgi?id=28301 ). It really doesn't feel right to skip that entirely just because a few particular pixel comparisons fail sometimes.
Plan A would be to fix the tests to not be flaky, but that's hard, and the buildbot needs a quick solution.
Plan B might be to conditionally compile the flaky tests away, e.g.:
--- a/dlls/ddraw/tests/visual.c +++ b/dlls/ddraw/tests/visual.c @@ -1490,6 +1490,7 @@ static void D3D1_TextureMapBlendTest(void) ok(hr == D3D_OK, "IDirect3DDevice3_EndScene failed, hr = %08x\n", hr); }
+#ifndef WINE_SKIP_FLAKY_TEST_SEE_BUG_28301 color = D3D1_getPixelColor(DirectDraw1, Surface1, 5, 5); red = (color & 0x00ff0000) >> 16; green = (color & 0x0000ff00) >> 8; @@ -1513,6 +1514,7 @@ static void D3D1_TextureMapBlendTest(void) green = (color & 0x0000ff00) >> 8; blue = (color & 0x000000ff); ok(red == 0 && green == 0 && blue == 0xff, "Got color %08x, expected 000000ff or near\n", color); +#endif
Then my blacklist would mostly be a bunch of -D definitions in CFLAGS, one per flaky test. Or we could use a single define for all flaky tests.
I may try that privately, to increase buildbot's test coverage.
Would something like that be useful for other people, or should I just stick with private patches? - Dan