Wine's test suite is now nine years old, and thanks in large part to Ge's testbot, new tests tend to actually work on windows. There's a lot of passing tests at http://test.winehq.org, which is awesome. Wine developers routinely use it to avoid sending bad patches.
And yet, some bad patches do get sent. Running all the incoming patches from wine-patches though the reliable subset of the tests automatically has been finding about one real bug a day. Sometimes I can warn the author soon enough that he can send a revised patch before the bad one is committed. Automated testing ftw!
But sometimes bad patches don't fail during the few test runs currently done, get committed, and then start causing unrelated patches to fail randomly, poisoning future test runs until it the bug is fixed or the bad patch reverted. It would be nice if Alexandre reverted bad patches as soon as the buildbot started going red, without waiting for his usual daily batch of commits, and without waiting for someone to fix the bad patch. (Maybe this can be discussed at wineconf.)
Right, so that's the story for keeping new bugs out of the tree. And then there are all the old bugs. Getting them fixed sometime would be nice; until then, the buildbot just skips known bad tests.
Here are the tests that fail or crash sometimes for me when run normally on 32 bit systems. (This list is a lot bigger than the one I posted back in 2009, mostly because I'm running the tests a lot more, and noticing ones that only fail occasionally.)
http://bugs.winehq.org/show_bug.cgi?id=28301 dlls/ddraw/tests/visual.ok http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok http://bugs.winehq.org/show_bug.cgi?id=28220 dlls/kernel32/tests/process.ok http://bugs.winehq.org/show_bug.cgi?id=28295 dlls/mshtml/tests/htmldoc.ok http://bugs.winehq.org/show_bug.cgi?id=28290 dlls/shell32/tests/shelllink.ok http://bugs.winehq.org/show_bug.cgi?id=28323 dlls/urlmon/tests/protocol.ok http://bugs.winehq.org/show_bug.cgi?id=28108 dlls/urlmon/tests/url.ok http://bugs.winehq.org/show_bug.cgi?id=20149 dlls/user32/tests/static.ok http://bugs.winehq.org/show_bug.cgi?id=28038 dlls/wininet/tests/urlcache.ok http://bugs.winehq.org/show_bug.cgi?id=28102 dlls/ws2_32/tests/sock.ok
And another handful that fail or crash sometimes for me when run with WINEDEBUG=warn+heap:
http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d8.ok http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d.ok http://bugs.winehq.org/show_bug.cgi?id=28316 dlls/kernel32/tests/debugger.ok http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok http://bugs.winehq.org/show_bug.cgi?id=28262 dlls/mshtml/tests/style.ok http://bugs.winehq.org/show_bug.cgi?id=28258 programs/cmd/tests/batch.ok
There are also tests I don't run on the buildbot because they only pass on other systems with particular window managers or sound hardware or software:
http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d8.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound8.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound.ok http://bugs.winehq.org/show_bug.cgi?id=28216 dlls/shell32/tests/shlfolder.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/input.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/msg.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/win.ok http://bugs.winehq.org/show_bug.cgi?id=28267 dlls/winhttp/tests/winhttp.ok http://bugs.winehq.org/show_bug.cgi?id=28109 dlls/winmm/tests/capture.ok http://bugs.winehq.org/show_bug.cgi?id=28071 dlls/winmm/tests/mci.ok http://bugs.winehq.org/show_bug.cgi?id=28048 dlls/winmm/tests/wave.ok
More info at http://wiki.winehq.org/BuildBot - Dan