Re: update to regression testing framework

[Paul Millar]

On Wed, 10 Apr 2002, Francois Gouget wrote:
> > o  If you set WINETEST_CONTINUE_ON_FAIL environment variable to 1, "make
>
> This looks like it is equivalent to make -k.

Oops, didn't know about that one :) I'll remove the 
WINETEST_CONTINUE_ON_FAIL code ...
> > o  The commands "named_ok", "set_name" and "unset_name" now exist (in C
> >     and Perl). You can use these to specify a name for tests either
> >     individually (via named_ok) or for a region (via set_name and
> >     unset_name)
>
> I'm not too sure about these:
>
> -ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
> +named_ok( "Adding atom "$name" via GlobalAddAtomA()", ($atom >=
>            0xc000) && ($atom <= 0xffff) && !defined($wine::err) );

Here's the rationale: when a test result is displayed (currently only on
failure or success in a todo block) the file name and line number are
included. If the test file is altered then any tests after the edit will
have different line numbers. This makes these tests look like new ones: an
undesirable behaviour but not too big an issue. Explicitly naming tests is
a way of making the test results line-number independent.
> Did you know that you can also do:
>
> ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err),
>     "could not find atom $name, atom=$atom, err=$wine::err" );
> This seems pretty similar and works in perl and C.

Yep. But I'd say there's a difference between an error message and a test
name: both describe what the test checks for, but labelling a successful
test by the error message seems a bit ugly to me.

(btw, naming test works under both Perl and C too :)
> +set_name( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
>  for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
>  {
>      ok( GlobalAddAtomA($i) && !defined($wine::err) );
>      ok( GlobalAddAtomW($i) && !defined($wine::err) );
>  }
> +unset_name();
>
> would be better as:
>
> named_tests( "Checking GlobalAddAtom[AW](i) i < 0xbffff") {
>     for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
>     {
>         ok( GlobalAddAtomA($i) && !defined($wine::err) );
>         ok( GlobalAddAtomW($i) && !defined($wine::err) );
>     }
> }

Agreed.
> But I am not sure it adds that much to:
>
> trace( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
> for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
> {
>     ok( GlobalAddAtomA($i) && !defined($wine::err) );
>     ok( GlobalAddAtomW($i) && !defined($wine::err) );
> }

You could use trace, but IMHO there's a difference between using trace and
named_tests. Trace messages have arbitrary content (and so arbitrary
meaning) and they may appear at an arbitrary point. A test name always
appears in the same line as the test result, producing lines like:

tests/atom.pl:Checking GlobalAddAtom[AW](i) i < 0xc0000: Test succeeded

IMHO, this is a lot cleaner than using trace messages, although there's
still the potential redundancy between a test's name and its error
message.

Thoughts?

----
Paul Millar