Re: We *really* need a development model change !

[Andriy Palamarchuk]

Simplicity is one of the goals of the testing
framework, but besides being simple the framework
should be powerful enough for such big project.

I'm looking for existing library which would suit our
needs.

Unit test framework "Check" looks promising. It
provides many  features which I'd love to use for our
tests, including:
- protection of the test address space - tests crashed
do not put down the whole test suite and are reported.
At the same time you can request to run tests in the
same address space what is good for debugging.
- grouping tests in suites, tree-like grouping of the
test suites. You can choose to run only subset of the
tests
- having "set up" and "tear down" sections which can
be used to set up tests and free resourses after.
- a few forms of output - from silent to detailed

Individual test looks like (section from manual):

START_TEST(test_create)
{
   fail_unless (money_amount(five_dollars) == 5,
           "Amount not set correctly on creation");
   fail_unless
(strcmp(money_currency(five_dollars),"USD") == 0,
           "Currency not set correctly on creation");
}
END_TEST

They also give information how to configure the tests
with autoconf.

The issues which need to be resolved:
1) License - GPL. Is it Ok to have the test suite
under GPL?
2) Environment - POSIX. The library uses POSIX calls
to manage processes, in particular functions fork,
_exit, write, read, close, getppid, getpid, pipe.
Calls "fork" , "_exit" and "getppid" are used only
when test runs in FORK mode. We don't need to have
address space protection under Windows - tests there
should not crash.
The rest of functions are used for communication
between the test launcher and tests and can be
implemented with using corresponding Win32 calls.
After implementing these changes the framework can be
used in pure Win32 environment, without Cygwin.

Let me know if you are interested. I'll start to work
on the port. Meantime we can start to use the
framework under Wine and on Windows with Cygwin.

More information about Check you can find here:
http://check.sourceforge.net/ - home page
http://sourceforge.net/project/showfiles.php?group_id=28255
 - tutorial

Thanks,
Andriy Palamarchuk

__________________________________________________
Do You Yahoo!?
Send your FREE holiday greetings online!
http://greetings.yahoo.com

We *really* need a development model change !

[Jeremy White]

Andriy (and all),

I think you have dismissed winetest much too quickly.

We spent a considerable amount of energy thinking
about a test harness (largely because one of our investors
felt passionately that it was vital), so not only did
we have the public conversations you saw on wine-devel,
we here at CodeWeavers also had some lengthy private
conversations.

I entered that process feeling much as you do:  that 'C' based
tests were the way and the truth and the light.

I was persuaded that a Perl based test was, in fact,
a viable alternative, if not a better alternative.

Several points:
     1.  John Sturtz worked very hard to make the
         Perl code able to take callbacks.
         AFAIK, it does that.

2.  The Perl code makes it easy to do pattern
         matching, hex dumps, and general text manipulation,
         which is a good thing.

3.  Perl is really easy to install and use in Windows;
         writing these tests in Windows is a total
         snap, and, I would argue, Windows is exactly
         where the tests should be written.

All that aside, I think the most important tasks
are a simple set:
     1.  Modify the makefiles so that 'make test'
         (or make regress, or pick your poison)
         does something interesting, and does
         so without much hassle.

2.  Define a standard for how tests should
         work.  For example, if I wanted to
         write a test for FunctionFoo(), I should
         be able to create wine/programs/winetest/tests/foo.pl,
         add 'foo.pl' to winetest/tests/Makefile.in,
         and my test should automatically become part
         of the regression suite.

(Note that adding tests/foo.c is just as trivial).

I believe that the defacto standard chosen
         at this point is that a test prints 'OK' if
         all went well, and it doesn't on error, and
         prints some clue as to what went wrong.

3.  Add a section to the documentation on how
         to do the same.

In my not so humble opinion, that is a very straightforward
task, that shouldn't take very long, and would be an
excellent beginning for shifting the development model.

Once we have in place the concept of a 'make regress',
we can start expanding on that to roll in new
testing tools; I just think it's a mistake to arbitrarily
discard a perfectly useful tool.

Jer

Andriy Palamarchuk wrote:
> Simplicity is one of the goals of the testing
> framework, but besides being simple the framework
> should be powerful enough for such big project.
>
> I'm looking for existing library which would suit our
> needs.
>
> Unit test framework "Check" looks promising. It
> provides many  features which I'd love to use for our
> tests, including:
> - protection of the test address space - tests crashed
>
> do not put down the whole test suite and are reported.
> At the same time you can request to run tests in the
> same address space what is good for debugging.
> - grouping tests in suites, tree-like grouping of the
>
> test suites. You can choose to run only subset of the
> tests
> - having "set up" and "tear down" sections which can
>
> be used to set up tests and free resourses after.
> - a few forms of output - from silent to detailed
>
> Individual test looks like (section from manual):
>
> START_TEST(test_create)
> {
>    fail_unless (money_amount(five_dollars) == 5,
>           "Amount not set correctly on creation");
>    fail_unless
> (strcmp(money_currency(five_dollars),"USD") == 0,
>           "Currency not set correctly on creation");
> }
> END_TEST
>
> They also give information how to configure the tests
> with autoconf.
>
> The issues which need to be resolved:
> - License - GPL. Is it Ok to have the test suite
>
> under GPL?
> 2) Environment - POSIX. The library uses POSIX calls
> to manage processes, in particular functions fork,
> _exit, write, read, close, getppid, getpid, pipe.
> Calls "fork" , "_exit" and "getppid" are used only
> when test runs in FORK mode. We don't need to have
> address space protection under Windows - tests there
> should not crash.
> The rest of functions are used for communication
> between the test launcher and tests and can be
> implemented with using corresponding Win32 calls.
> After implementing these changes the framework can be
> used in pure Win32 environment, without Cygwin.
>
> Let me know if you are interested. I'll start to work
> on the port. Meantime we can start to use the
> framework under Wine and on Windows with Cygwin.
>
> More information about Check you can find here:
> http://check.sourceforge.net/ - home page
> http://sourceforge.net/project/showfiles.php?group_id=28255
> - tutorial
>
> Thanks,
> Andriy Palamarchuk
>
> Do You Yahoo!?
> Send your FREE holiday greetings online!
> http://greetings.yahoo.com