update to regression testing framework

[Paul Millar]

Hi,

The attached patch changes the way the regression testing works slightly.
The default behaviour should be the same, but there's some optional
"enhancements":

 o  If you set WINETEST_REPORT_SUCCESS environment variable to 1, 
    successful tests are reported as well as unsuccessful ones.

 o  If you set WINETEST_CONTINUE_ON_FAIL environment variable to 1, "make
    test" will run all tests from all modules, rather than failing on the
    first failing module.

 o  The commands "named_ok", "set_name" and "unset_name" now exist (in C 
    and Perl). You can use these to specify a name for tests either 
    individually (via named_ok) or for a region (via set_name and 
    unset_name)

The patch changes both the C and Perl frameworks. I'm pretty sure the Perl
is right but someone who actually knows Perl ought to check it :)

The second patch illustrates these by changing dlls/kernel/tests/atom.pl 
and dlls/kernel/tests/directory.c (apologies to Alexandre & Dmitry if I 
got the test semantics wrong).

HTH

----
Paul Millar

ChangeLog:
  Adding test names and some optional behaviour.

[Francois Gouget]

On Wed, 10 Apr 2002, Paul Millar wrote:
> Hi,
>
> The attached patch changes the way the regression testing works slightly.
> The default behaviour should be the same, but there's some optional
> "enhancements":
>
> o  If you set WINETEST_REPORT_SUCCESS environment variable to 1,
>     successful tests are reported as well as unsuccessful ones.

I don't know if this is really useful but why not.
> o  If you set WINETEST_CONTINUE_ON_FAIL environment variable to 1, "make
>     test" will run all tests from all modules, rather than failing on the
>     first failing module.

This looks like it is equivalent to make -k.
> o  The commands "named_ok", "set_name" and "unset_name" now exist (in C
>     and Perl). You can use these to specify a name for tests either
>     individually (via named_ok) or for a region (via set_name and
>     unset_name)

I'm not too sure about these:

-ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
+named_ok( "Adding atom \"$name\" via GlobalAddAtomA()", ($atom >=
           0xc000) && ($atom <= 0xffff) && !defined($wine::err) );

Did you know that you can also do:

ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err),
    "could not find atom $name, atom=$atom, err=$wine::err" );

This seems pretty similar and works in perl and C. Also about the
following:

+set_name( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
 for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
 {
     ok( GlobalAddAtomA($i) && !defined($wine::err) );
     ok( GlobalAddAtomW($i) && !defined($wine::err) );
 }
+unset_name();

would be better as:

named_tests( "Checking GlobalAddAtom[AW](i) i < 0xbffff") {
    for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
    {
        ok( GlobalAddAtomA($i) && !defined($wine::err) );
        ok( GlobalAddAtomW($i) && !defined($wine::err) );
    }
}

This way one would not accidentally forget to unset the name. But I am
not sure it adds that much to:

trace( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
{
    ok( GlobalAddAtomA($i) && !defined($wine::err) );
    ok( GlobalAddAtomW($i) && !defined($wine::err) );
}

Traces are not printed by default but will be if you run the test as:
runtest -v ... instead of runtest -q ...


--
Francois Gouget         fgouget(a)free.fr        http://fgouget.free.fr/
        It really galls me that most of the computer power in the world
                          is wasted on screen savers.
                     Chris Caldwell from the GIMPS project
                       http://www.mersenne.org/prime.htm

[Paul Millar]

On Wed, 10 Apr 2002, Francois Gouget wrote:
> > o  If you set WINETEST_CONTINUE_ON_FAIL environment variable to 1, "make
> >    This looks like it is equivalent to make -k.

Oops, didn't know about that one :) I'll remove the 
WINETEST_CONTINUE_ON_FAIL code ...
> > o  The commands "named_ok", "set_name" and "unset_name" now exist (in C
> >     and Perl). You can use these to specify a name for tests either
> >     individually (via named_ok) or for a region (via set_name and
> >     unset_name)
>
> I'm not too sure about these:
>
> -ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
> +named_ok( "Adding atom \"$name\" via GlobalAddAtomA()", ($atom >=
>            0xc000) && ($atom <= 0xffff) && !defined($wine::err) );

Here's the rationale: when a test result is displayed (currently only on
failure or success in a todo block) the file name and line number are
included. If the test file is altered then any tests after the edit will
have different line numbers. This makes these tests look like new ones: an
undesirable behaviour but not too big an issue. Explicitly naming tests is
a way of making the test results line-number independent.
> Did you know that you can also do:
>
> ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err),
>     "could not find atom $name, atom=$atom, err=$wine::err" );
> This seems pretty similar and works in perl and C.

Yep. But I'd say there's a difference between an error message and a test
name: both describe what the test checks for, but labelling a successful
test by the error message seems a bit ugly to me.

(btw, naming test works under both Perl and C too :)
> +set_name( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
>  for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
>  {
>      ok( GlobalAddAtomA($i) && !defined($wine::err) );
>      ok( GlobalAddAtomW($i) && !defined($wine::err) );
>  }
> +unset_name();
>
> would be better as:
>
> named_tests( "Checking GlobalAddAtom[AW](i) i < 0xbffff") {
>     for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
>     {
>         ok( GlobalAddAtomA($i) && !defined($wine::err) );
>         ok( GlobalAddAtomW($i) && !defined($wine::err) );
>     }
> }

Agreed.
> But I am not sure it adds that much to:
>
> trace( "Checking GlobalAddAtom[AW](i) i < 0xbffff");
> for ($i = 0x0001; ($i <= 0xbfff); $i += 27)
> {
>     ok( GlobalAddAtomA($i) && !defined($wine::err) );
>     ok( GlobalAddAtomW($i) && !defined($wine::err) );
> }

You could use trace, but IMHO there's a difference between using trace and
named_tests. Trace messages have arbitrary content (and so arbitrary
meaning) and they may appear at an arbitrary point. A test name always
appears in the same line as the test result, producing lines like:

tests/atom.pl:Checking GlobalAddAtom[AW](i) i < 0xc0000: Test succeeded

IMHO, this is a lot cleaner than using trace messages, although there's
still the potential redundancy between a test's name and its error
message.

Thoughts?

----
Paul Millar

[Francois Gouget]

On Thu, 11 Apr 2002, Paul Millar wrote:
[...]
> > -ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
> > +named_ok( "Adding atom \"$name\" via GlobalAddAtomA()", ($atom >=
> >            0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
>
> Here's the rationale: when a test result is displayed (currently only on
> failure or success in a todo block) the file name and line number are
> included. If the test file is altered then any tests after the edit will
> have different line numbers. This makes these tests look like new ones: an
> undesirable behaviour but not too big an issue. Explicitly naming tests is
> a way of making the test results line-number independent.

Ok, so the goal is to make it possible to automatically analyze the
test results. I see you point then.

[...]
> meaning) and they may appear at an arbitrary point. A test name always
> appears in the same line as the test result, producing lines like:
>
> tests/atom.pl:Checking GlobalAddAtom[AW](i) i < 0xc0000: Test succeeded
>
> IMHO, this is a lot cleaner than using trace messages, although there's
> still the potential redundancy between a test's name and its error
> message.

Yes, seems cleaner than test messages but there is the problem of
redundancy between the 'name' and the error message. Also, it seems that
for your purposes all tests should use named_ok otherwise you will not
be able to identify all tests. This means modifying all tests, and/or
just having ok. No?

--
Francois Gouget         fgouget(a)free.fr        http://fgouget.free.fr/
      Broadcast message : fin du monde dans cinq minutes, repentez vous !

[Tels]

-----BEGIN PGP SIGNED MESSAGE-----

Moin,

On 11-Apr-02 Francois Gouget carved into stone:
> On Thu, 11 Apr 2002, Paul Millar wrote:
> [...]
> > > -ok( ($atom >= 0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
> > > +named_ok( "Adding atom \"$name\" via GlobalAddAtomA()", ($atom >=
> > >            0xc000) && ($atom <= 0xffff) && !defined($wine::err) );
> >
> > Here's the rationale: when a test result is displayed (currently only on
> > failure or success in a todo block) the file name and line number are
> > included. If the test file is altered then any tests after the edit will
> > have different line numbers. This makes these tests look like new ones:
> > an
> > undesirable behaviour but not too big an issue. Explicitly naming tests
> > is
> > a way of making the test results line-number independent.
>
> Ok, so the goal is to make it possible to automatically analyze the
> test results. I see you point then.
>
> [...]
> > meaning) and they may appear at an arbitrary point. A test name always
> > appears in the same line as the test result, producing lines like:
> >
> > tests/atom.pl:Checking GlobalAddAtom[AW](i) i < 0xc0000: Test succeeded
> >
> > IMHO, this is a lot cleaner than using trace messages, although there's
> > still the potential redundancy between a test's name and its error
> > message.
>
> Yes, seems cleaner than test messages but there is the problem of
> redundancy between the 'name' and the error message. Also, it seems that
> for your purposes all tests should use named_ok otherwise you will not
> be able to identify all tests. This means modifying all tests, and/or
> just having ok. No?

Just a quick hint:

The perl core test frame work already has dealed (probably) with problems
like this. Check out the Test, Test::More and Test::Harness modules. Maybe
you can re-use the work done there?

Thanx in advance,

Tels

- --
 "Why do you go so slowly? Do you think this is some kind of game?"
 PGP key available on http://bloodgate.com/tels.asc or via email.
 perl -MDev::Bollocks -e'print Dev::Bollocks->rand(),"\n"'
 dynamically utilize value-added technologies


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl.

iQEVAwUBPLXrMHcLPEOTuEwVAQHJXgf+K/UeFqq4jnCu3jqAVDN7Wy657APhmWsG
ql3lhSqfjDYklE1Lvck0qzir0Eiz+9UycdbYswo+EX4OQB5NV2ut0DbRjWAhPqzI
hnWh8iuC8q7RZQUbPWMLoJaZI/P+v5sVzpMcp3YGXAoa3JZfMcF7XW5LfT2/twMu
XIFD5Kfly018SdJL1K1xxCjgt1zLUjVXjeYwhCj8KqJEX2tuo8rxKpk5wGD2z584
EGkjIqItXgyT5LJX2JXOGbmWsenUOeR3w1sXoOuJ+05A22eHibp5w1veWWvuL9Hr
vXpnSxWW1lqi/0IZ3UxPK+3XbxtViC4us9qLSluUVbSrUmlVMDVY0w==
=HC8H
-----END PGP SIGNATURE-----

[Alexandre Julliard]

Paul Millar <paulm(a)astro.gla.ac.uk> writes:
> Here's the rationale: when a test result is displayed (currently only on
> failure or success in a todo block) the file name and line number are
> included. If the test file is altered then any tests after the edit will
> have different line numbers. This makes these tests look like new ones: an
> undesirable behaviour but not too big an issue. Explicitly naming tests is
> a way of making the test results line-number independent.

I think it's much more useful to have line numbers, it makes it much
easier to find the offending test. Most tests scripts will contain
thousands of individual tests and you won't be able to come up with
meaningful names for them anyway. For the same reason, I don't think
displaying every successful test is really useful. Traces are a much
better way to see what's going on IMO.

-- 
Alexandre Julliard
julliard(a)winehq.com

[Paul Millar]

On 11 Apr 2002, Alexandre Julliard wrote:
> I think it's much more useful to have line numbers, it makes it much
> easier to find the offending test.

If they don't have a meaningful name, sure, but if the name is suitably
unique then grep -n, /, or Ctrl+S should work.
> Most tests scripts will contain
> thousands of individual tests and you won't be able to come up with
> meaningful names for them anyway.

A lot of those are the same test repeated for different parameters. If the
name incorporated the parameters (w/ a printf-style expansion) it wouldn't
be too hard. Also, any unmodified test would work as before.
> For the same reason, I don't think
> displaying every successful test is really useful. Traces are a much
> better way to see what's going on IMO.

It would be nice (for me anyway :) if tests were line-number independent.

As naming tests seems to be a stumbling block, I've attached a patch that
just allows announcement of successful test results. I'm hoping it's
acceptable. Named tests can wait for another day ...

----
Paul Millar