http://bugs.winehq.org/show_bug.cgi?id=9916
--- Comment #95 from Dan Kegel dank@kegel.com 2011-09-08 11:27:31 CDT --- Wine's tests are in a lot better shape than when this bug was filed, but still, odds are that "make test" will fail on the average machine.
I've been collecting a blacklist of bad and/or flaky tests. Here's one way to use it to measure how unreliable the flaky tests are on a particular system:
wget -c http://winezeug.googlecode.com/svn/trunk/buildbot/dotests.sh wget -c http://winezeug.googlecode.com/svn/trunk/buildbot/dotests_blacklist.txt rm -rf ~/.wine winetricks nocrashdialog while true do make testclean sh dotests.sh flakytests done
Out of 40 or so runs on one system, five dll/foo.ok files exhibited crashes or failures:
7 dlls/wininet/tests/urlcache.ok (bug 28038, workaround: clear cache between runs) 4 dlls/msctf/tests/inputprocessor.ok (bug 28288) 4 dlls/kernel32/tests/pipe.ok (bug 28257) 3 dlls/user32/tests/static.ok (bug 20149) 2 dlls/urlmon/tests/url.ok (bug 28108)
So about half of 40 runs of "make test" would probably fail because of flaky tests on this machine without manual workarounds.
Your mileage may vary. Other systems tickle other bugs.
That blacklist and script can also look for tests that have failed on any of my systems repeatably. Running 'sh dotests.sh badtests' twice on that same (64 bit) system, I got
2 dlls/ieframe/tests/ie.ok FAILED; blacklist says BAD64 , see bug 26768 . 2 dlls/kernel32/tests/process.ok FAILED; blacklist says NOTTY , see bug 28220 . 2 dlls/oleaut32/tests/tmarshal.ok FAILED; blacklist says BAD64 , see bug 26768 . 1 dlls/user32/tests/msg.ok FAILED; blacklist says SYS , see bug 12053 . 2 programs/wscript/tests/run.ok FAILED; blacklist says BAD64 , see bug 28285 .
so "make test" will always fail on this system because of msg tests (probable workaround: run a blessed window manager) and because win64 isn't finished. And a kernel test will always fail because I tend to redirect the output of "make test".