Thanks to everybody who responded to Jeremy's call to run winetest using the 'dotests' script at http://wiki.winehq.org/MakeTestFailures
We now have 35 reports, and have some good data on which test failures are common. See the Most Common Failures section at the bottom. Here's what it says right now:
The skipgood script above now also tallies tests by how many people they fail for. Currently, it shows:
35 shell32:shelllink 32 user32:msg 16 user32:input 14 ntdll:info 13 d3d9:visual 10 user32:win 8 ddraw:visual 7 urlmon:protocol 5 gdi32:font 4 winmm:wave 3 d3d9:query urlmon:url wininet:url 2 comctl32:tooltips ddraw:dsurface msxml3:domdoc quartz:referenceclock wininet:http 1 d3d8:visual d3d9:device dinput:device dsound:ds3d dsound:ds3d8 dsound:dsound ...
Let's ignore the tests that only fail for one person for the moment, and look at the others.
shell32:shelllink fails on all winetest reports from all machines, including on real Windows. Yet "make test" in shell32 succeeds on this test. Could somebody have a look at why it reliably fails under winetest? I filed this as bug 13311.
user32:msg and user32:input are probably people running non-blessed window managers; Alexandre's position is that we can ignore these, since upstream gnome's WM is fixed.
ntdll:info is a flaky test marked todo_wine that succeeds sometimes: "info.c:822: Test succeeded inside todo block: Expected to read 0 bytes, got 0"
d3d9:visual failures are all from people not running an approved graphics card, and can be ignored (though maybe we should tell users we only support nvidia Geforce 6 and higher for 3d graphics...)
user32:win appears to be nearly all people running unapproved window managers, though abraml-hardy-32 did get one failure "win.c:2555: Test failed: wrong active window (nil)" with KDE3 which should be investigated. [fixme: file bug?]
ddraw:visual has two kinds of errors. The most common is "visual.c:2045: Test failed: Got color 00dcdde6, expected 0000ff00". Less common is mstefani's failure, "visual.c:481: Test failed: Untransformed vertex with linear vertex fog has color 00ff0000". [fixme: file bugs?]
urlmon:protocol has three or so failure modes; most common is "protocol.c:1597: Test failed: Read failed: 8000000a (0 bytes)". [fixme: file bug?]
gdi32:font has several failure modes, not sure what's going on there. [fixme: file bug?]
winemm:wave might actually be a problem, perhaps Maarten can take a look. [fixme: file bug?]
I haven't looked at the failures that happen for only 2 or 3 people.