Re: [1/2] tests: Add skip_wine, a standard way to skip tests that would cause Wine to crash.
Francois Gouget <fgouget(a)free.fr> writes:
Currently there is no standard way of adding a test that works on Windows but is known to cause Wine to crash. For instance this is the case for bug 6827:
Bug 6827: The SimSig installer crashes http://bugs.winehq.org/show_bug.cgi?id=6827
This essentially means that such tests just don't get written, or get put inside '#if 0' or 'if (0)' sections so that they are never executed, not even on Windows. This means we won't get good Windows coverage until Wine has been 'fixed'.
It's not really better with skip_wine; since the test will never be run on Wine it's useless for regression testing, and we won't even notice when the bug gets fixed. If the test doesn't crash on Windows, the right way is to fix Wine to not crash either, that's usually not too hard (making the test succeed may be harder, but that's why we have todo_wine). -- Alexandre Julliard julliard(a)winehq.org
On Thu, 14 Dec 2006, Alexandre Julliard wrote: [...]
It's not really better with skip_wine; since the test will never be run on Wine it's useless for regression testing, and we won't even notice when the bug gets fixed.
I see some advantages though: * the code exists rather than not being written in the first place * it is compiled, which is better than putting it inside an '#if 0' * 'skip_wine' clearly indicates that the test needs to be fixed in Wine, which a basic '#if 0' or 'if (0)' does not do * the test statistics will clearly indicate that one of the tests fails in Wine, rather than show that every test succeeds * the test will be run on Windows
If the test doesn't crash on Windows, the right way is to fix Wine to not crash either, that's usually not too hard (making the test succeed may be harder, but that's why we have todo_wine).
I found quite a few other places in our tests where we could use skip_wine(), but I am really not sure fixing the crashes is alway that simple. See for instance dlls/winmm/tests/wave.c or dlls/rpcrt4/tests/ndr_marshall.c. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ You can have my guns when you pry them from my kids cold, dead hands.
Francois Gouget <fgouget(a)free.fr> writes:
I see some advantages though: * the code exists rather than not being written in the first place * it is compiled, which is better than putting it inside an '#if 0' * 'skip_wine' clearly indicates that the test needs to be fixed in Wine, which a basic '#if 0' or 'if (0)' does not do * the test statistics will clearly indicate that one of the tests fails in Wine, rather than show that every test succeeds * the test will be run on Windows
Doing it right would involve running the test on Wine inside an exception handler, and verify that it actually crashes. That's not trivial to do in the tests though. Also I'm concerned that adding a mechanism like this will encourage writing tests that pass garbage pointers, since in most cases this doesn't crash on Windows. But passing bad pointers isn't an interesting test unless a real app depends on it, and if it does then we have to prevent Wine from crashing, adding skip_wine in the test won't help the app. -- Alexandre Julliard julliard(a)winehq.org
participants (2)
-
Alexandre Julliard -
Francois Gouget