Wine's test suite is now nine years old, and thanks in large part to Ge's testbot, new tests tend to actually work on windows. There's a lot of passing tests at http://test.winehq.org, which is awesome. Wine developers routinely use it to avoid sending bad patches.
And yet, some bad patches do get sent. Running all the incoming patches from wine-patches though the reliable subset of the tests automatically has been finding about one real bug a day. Sometimes I can warn the author soon enough that he can send a revised patch before the bad one is committed. Automated testing ftw!
But sometimes bad patches don't fail during the few test runs currently done, get committed, and then start causing unrelated patches to fail randomly, poisoning future test runs until it the bug is fixed or the bad patch reverted. It would be nice if Alexandre reverted bad patches as soon as the buildbot started going red, without waiting for his usual daily batch of commits, and without waiting for someone to fix the bad patch. (Maybe this can be discussed at wineconf.)
Right, so that's the story for keeping new bugs out of the tree. And then there are all the old bugs. Getting them fixed sometime would be nice; until then, the buildbot just skips known bad tests.
Here are the tests that fail or crash sometimes for me when run normally on 32 bit systems. (This list is a lot bigger than the one I posted back in 2009, mostly because I'm running the tests a lot more, and noticing ones that only fail occasionally.)
http://bugs.winehq.org/show_bug.cgi?id=28301 dlls/ddraw/tests/visual.ok http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok http://bugs.winehq.org/show_bug.cgi?id=28220 dlls/kernel32/tests/process.ok http://bugs.winehq.org/show_bug.cgi?id=28295 dlls/mshtml/tests/htmldoc.ok http://bugs.winehq.org/show_bug.cgi?id=28290 dlls/shell32/tests/shelllink.ok http://bugs.winehq.org/show_bug.cgi?id=28323 dlls/urlmon/tests/protocol.ok http://bugs.winehq.org/show_bug.cgi?id=28108 dlls/urlmon/tests/url.ok http://bugs.winehq.org/show_bug.cgi?id=20149 dlls/user32/tests/static.ok http://bugs.winehq.org/show_bug.cgi?id=28038 dlls/wininet/tests/urlcache.ok http://bugs.winehq.org/show_bug.cgi?id=28102 dlls/ws2_32/tests/sock.ok
And another handful that fail or crash sometimes for me when run with WINEDEBUG=warn+heap:
http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d8.ok http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d.ok http://bugs.winehq.org/show_bug.cgi?id=28316 dlls/kernel32/tests/debugger.ok http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok http://bugs.winehq.org/show_bug.cgi?id=28262 dlls/mshtml/tests/style.ok http://bugs.winehq.org/show_bug.cgi?id=28258 programs/cmd/tests/batch.ok
There are also tests I don't run on the buildbot because they only pass on other systems with particular window managers or sound hardware or software:
http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d8.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound8.ok http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound.ok http://bugs.winehq.org/show_bug.cgi?id=28216 dlls/shell32/tests/shlfolder.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/input.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/msg.ok http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/win.ok http://bugs.winehq.org/show_bug.cgi?id=28267 dlls/winhttp/tests/winhttp.ok http://bugs.winehq.org/show_bug.cgi?id=28109 dlls/winmm/tests/capture.ok http://bugs.winehq.org/show_bug.cgi?id=28071 dlls/winmm/tests/mci.ok http://bugs.winehq.org/show_bug.cgi?id=28048 dlls/winmm/tests/wave.ok
More info at http://wiki.winehq.org/BuildBot - Dan
Dan Kegel dank@kegel.com writes:
But sometimes bad patches don't fail during the few test runs currently done, get committed, and then start causing unrelated patches to fail randomly, poisoning future test runs until it the bug is fixed or the bad patch reverted. It would be nice if Alexandre reverted bad patches as soon as the buildbot started going red, without waiting for his usual daily batch of commits, and without waiting for someone to fix the bad patch. (Maybe this can be discussed at wineconf.)
I'm not going to revert patches, that would make a mess of the history. Reverting is a serious event and should happen only when a change is completely wrong, not when it happens to not be 100% correct or to have some unwanted side effect.
There are already many checks done before commit, and adding more checks like you do is of course a very good idea. All together, that should ensure that no committed patch can cause dramatic breakage. But there will always be minor breakages, and flaky tests, and race conditions, and buildbot must be able to deal with them gracefully, without requiring me to scramble a revert.
On Fri, Sep 16, 2011 at 2:31 PM, Alexandre Julliard julliard@winehq.org wrote:
I'm not going to revert patches, that would make a mess of the history. Reverting is a serious event and should happen only when a change is completely wrong, not when it happens to not be 100% correct or to have some unwanted side effect.
The rpct4 change seems seriously wrong, offhand, given how widespread the breakage it sometimes causes is. (A lot of tests seemed to think that the C:\ drive didn't exist.)
There are already many checks done before commit, and adding more checks like you do is of course a very good idea. All together, that should ensure that no committed patch can cause dramatic breakage. But there will always be minor breakages, and flaky tests, and race conditions, and buildbot must be able to deal with them gracefully, without requiring me to scramble a revert.
It's kind of hard to gracefully handle it when dozens of tests fail randomly because of a bad commit.
Mercifully, this kind of breakage doesn't happen often, so I guess I'm ok with doing local reverts when it does. I'm just worried that I won't be able to get to it quickly enough, and the resulting barrage of test failures will turn people off. - Dan
On Fri, Sep 16, 2011 at 2:54 PM, Dan Kegel dank@kegel.com wrote:
Mercifully, this kind of breakage doesn't happen often, so I guess I'm ok with doing local reverts when it does. I'm just worried that I won't be able to get to it quickly enough, and the resulting barrage of test failures will turn people off.
I suppose when the daily git commit lands, I could run the test suite ten times over. That'd only take an extra hour. If any failures occur during the burn-in, I could have buildbot stop testing patches until the git test failure was worked around or fixed.
On Friday 16 September 2011 23:54:03 Dan Kegel wrote:
The rpct4 change seems seriously wrong, offhand, given how widespread the breakage it sometimes causes is. (A lot of tests seemed to think that the C:\ drive didn't exist.)
There's hope that once the buildbot works well enough that Alexandre trusts it and doesn't commit patches a patch like this doesn't get in. And if we have multiple machines it's less likely that a patch that breaks randomly e.g. due to a buffer overrun goes by unnoticed.
-
Alexandre Julliard -
Dan Kegel -
Stefan Dösinger