The buildbot does three things to get fast, reliable test runs: - it runs a 'safe' set of tests in the background, without DISPLAY set, in a different wineprefix; this saves three minutes - it has a blacklist of known flaky or broken tests, listing how it's broken and linking to the bug report - it kills wineserver after the run (since some tests leave processes around)
I hoisted these out of the buildbot config and into a separate script, http://code.google.com/p/winezeug/source/browse/trunk/buildbot/dotests.sh
As a bonus, you can ask it to run all the flaky tests instead of all the good ones, which might be useful if you're trolling for test bugs to fix.
Here's the current blacklist from that script:
# Blacklist format # test condition [bug] # Tests may appear multiple times in the list. # Supported conditions: # FLAKY - fails sometimes # CRASHY - crashes sometimes # SYS - always fails on some systems, not on others # HEAP - fails or crashes if warn+heap # NOTTY - test fails if output redirected # BAD64 - always fails on 64 bits
cat > /tmp/blacklist.txt <<_EOF_ dlls/kernel32/tests/heap.ok HEAP dlls/dsound/tests/ds3d8.ok SYS 8668 dlls/dsound/tests/ds3d.ok SYS 8668 dlls/dsound/tests/dsound8.ok SYS 8668 dlls/dsound/tests/dsound.ok SYS 8668 dlls/user32/tests/input.ok SYS 12053 dlls/user32/tests/msg.ok SYS 12053 dlls/user32/tests/win.ok SYS 12053 dlls/oleaut32/tests/tmarshal.ok BAD64 26768 dlls/ieframe/tests/ie.ok BAD64 26768 dlls/ws2_32/tests/sock.ok CRASHY 28012 dlls/wininet/tests/urlcache.ok FLAKY 28038 dlls/winmm/tests/wave.ok SYS 28048 dlls/winmm/tests/mci.ok SYS 28071 dlls/urlmon/tests/url.ok FLAKY 28108 dlls/winmm/tests/capture.ok SYS 28109 dlls/shell32/tests/shlfolder.ok SYS 28216 dlls/kernel32/tests/process.ok NOTTY 28220 dlls/d3dx9_36/tests/shader.ok HEAP 28255 dlls/kernel32/tests/pipe.ok HEAP 28257 programs/cmd/tests/batch.ok HEAP 28258 dlls/dsound/tests/ds3d8.ok HEAP 28260 dlls/dsound/tests/ds3d.ok HEAP 28260 dlls/mshtml/tests/style.ok HEAP 28262 dlls/winhttp/tests/winhttp.ok SYS 28267 programs/wscript/tests/run.ok BAD64 28285 dlls/msctf/tests/inputprocessor.ok FLAKY 28288 _EOF_