Andriy (and all),
I think you have dismissed winetest much too quickly.
We spent a considerable amount of energy thinking about a test harness (largely because one of our investors felt passionately that it was vital), so not only did we have the public conversations you saw on wine-devel, we here at CodeWeavers also had some lengthy private conversations.
I entered that process feeling much as you do: that 'C' based tests were the way and the truth and the light.
I was persuaded that a Perl based test was, in fact, a viable alternative, if not a better alternative.
Several points: 1. John Sturtz worked very hard to make the Perl code able to take callbacks. AFAIK, it does that.
2. The Perl code makes it easy to do pattern matching, hex dumps, and general text manipulation, which is a good thing.
3. Perl is really easy to install and use in Windows; writing these tests in Windows is a total snap, and, I would argue, Windows is exactly where the tests should be written.
All that aside, I think the most important tasks are a simple set: 1. Modify the makefiles so that 'make test' (or make regress, or pick your poison) does something interesting, and does so without much hassle.
2. Define a standard for how tests should work. For example, if I wanted to write a test for FunctionFoo(), I should be able to create wine/programs/winetest/tests/foo.pl, add 'foo.pl' to winetest/tests/Makefile.in, and my test should automatically become part of the regression suite.
(Note that adding tests/foo.c is just as trivial).
I believe that the defacto standard chosen at this point is that a test prints 'OK' if all went well, and it doesn't on error, and prints some clue as to what went wrong.
3. Add a section to the documentation on how to do the same.
In my not so humble opinion, that is a very straightforward task, that shouldn't take very long, and would be an excellent beginning for shifting the development model.
Once we have in place the concept of a 'make regress', we can start expanding on that to roll in new testing tools; I just think it's a mistake to arbitrarily discard a perfectly useful tool.
Jer
Andriy Palamarchuk wrote:
Simplicity is one of the goals of the testing framework, but besides being simple the framework should be powerful enough for such big project.
I'm looking for existing library which would suit our needs.
Unit test framework "Check" looks promising. It provides many features which I'd love to use for our tests, including:
- protection of the test address space - tests crashed
do not put down the whole test suite and are reported. At the same time you can request to run tests in the same address space what is good for debugging.
- grouping tests in suites, tree-like grouping of the
test suites. You can choose to run only subset of the tests
- having "set up" and "tear down" sections which can
be used to set up tests and free resourses after.
- a few forms of output - from silent to detailed
Individual test looks like (section from manual):
START_TEST(test_create) { fail_unless (money_amount(five_dollars) == 5, "Amount not set correctly on creation"); fail_unless (strcmp(money_currency(five_dollars),"USD") == 0, "Currency not set correctly on creation"); } END_TEST
They also give information how to configure the tests with autoconf.
The issues which need to be resolved:
- License - GPL. Is it Ok to have the test suite
under GPL? 2) Environment - POSIX. The library uses POSIX calls to manage processes, in particular functions fork, _exit, write, read, close, getppid, getpid, pipe. Calls "fork" , "_exit" and "getppid" are used only when test runs in FORK mode. We don't need to have address space protection under Windows - tests there should not crash. The rest of functions are used for communication between the test launcher and tests and can be implemented with using corresponding Win32 calls. After implementing these changes the framework can be used in pure Win32 environment, without Cygwin.
Let me know if you are interested. I'll start to work on the port. Meantime we can start to use the framework under Wine and on Windows with Cygwin.
More information about Check you can find here: http://check.sourceforge.net/ - home page http://sourceforge.net/project/showfiles.php?group_id=28255
- tutorial
Thanks, Andriy Palamarchuk
Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com