Re: [1/2] tests: Add skip_wine, a standard way to skip tests that would cause Wine to crash.

[Alexandre Julliard]

Francois Gouget  writes:
> Currently there is no standard way of adding a test that works on 
> Windows but is known to cause Wine to crash. For instance this is 
> the case for bug 6827:
>
> Bug 6827: The SimSig installer crashes
> http://bugs.winehq.org/show_bug.cgi?id=6827
>
> This essentially means that such tests just don't get written, or get 
> put inside '#if 0' or 'if (0)' sections so that they are never executed, 
> not even on Windows. This means we won't get good Windows coverage until 
> Wine has been 'fixed'.

It's not really better with skip_wine; since the test will never be
run on Wine it's useless for regression testing, and we won't even
notice when the bug gets fixed. If the test doesn't crash on Windows,
the right way is to fix Wine to not crash either, that's usually not
too hard (making the test succeed may be harder, but that's why we
have todo_wine).

[1/2] tests: Add skip_wine, a standard way to skip tests that would cause Wine to crash.

[Francois Gouget]

On Thu, 14 Dec 2006, Alexandre Julliard wrote:
[...]
> It's not really better with skip_wine; since the test will never be
> run on Wine it's useless for regression testing, and we won't even
> notice when the bug gets fixed.

I see some advantages though:
 * the code exists rather than not being written in the first place
 * it is compiled, which is better than putting it inside an '#if 0'
 * 'skip_wine' clearly indicates that the test needs to be fixed in 
   Wine, which a basic '#if 0' or 'if (0)' does not do
 * the test statistics will clearly indicate that one of the tests fails 
   in Wine, rather than show that every test succeeds
 * the test will be run on Windows
> If the test doesn't crash on Windows, the right way is to fix Wine to 
> not crash either, that's usually not too hard (making the test succeed 
> may be harder, but that's why we have todo_wine).

I found quite a few other places in our tests where we could use 
skip_wine(), but I am really not sure fixing the crashes is alway that 
simple. See for instance dlls/winmm/tests/wave.c or 
dlls/rpcrt4/tests/ndr_marshall.c.

[1/2] tests: Add skip_wine, a standard way to skip tests that would cause Wine to crash.

[Alexandre Julliard]

Francois Gouget  writes:
> I see some advantages though:
> - the code exists rather than not being written in the first place
> - it is compiled, which is better than putting it inside an '#if 0'
> - 'skip_wine' clearly indicates that the test needs to be fixed in 
> Wine, which a basic '#if 0' or 'if (0)' does not do
> - the test statistics will clearly indicate that one of the tests fails 
> in Wine, rather than show that every test succeeds
> - the test will be run on Windows

Doing it right would involve running the test on Wine inside an
exception handler, and verify that it actually crashes. That's not
trivial to do in the tests though.

Also I'm concerned that adding a mechanism like this will encourage
writing tests that pass garbage pointers, since in most cases this
doesn't crash on Windows. But passing bad pointers isn't an
interesting test unless a real app depends on it, and if it does then
we have to prevent Wine from crashing, adding skip_wine in the test
won't help the app.