--- Francois Gouget fgouget@free.fr wrote: [...]
A test unit can also exercise more than one aspect
of
one or more APIs. But, as a rule of thumb, a
specific
test should not exercise more than a couple to a handful related APIs (or up to a dozen in extreme cases). Also, more than one test could exercise different aspects a given API.
I don't see a problem in having complex, task-oriented tests. Simple tests check one module, coplex tasks help to check integration between components. Both are needed. I especially like to create tests which implement long data conversion chains where output should be equal to input :-) I still want to have well-structured tests.
Wine tests can write their output in any form they like. The only important criteria are:
One more criteria - restore the system state (if possible). We don't want to have misconfigured machine after running the test suite.
To each test we associate a file containing the reference output for that test.
Maintaining reference output file(s) is difficult because of: - EOLs conversions - keep all the files in sync - output files can be huge - test suite can have different output for different versions of Windows - one person can't run the suite on all Windows platforms, those who can, are afraid to touch fragile system of the output reference files.
I suggest to use explicit checks and print descriptive messages in case of falure. I agree, this approach is more labour-intensive, especially for tests using IPC. It is also much more maintainable as soon as you coded it. Everything, including differences between different Windows platforms is documented directly in the code! This gives much better control. E.g, it is possible to comment-out part of the test, still getting meaningful results what worked, what did not.
One more idea about reference output - it can be included in Perl script as in-line constant, so we can keep the output in the same file as the test itself.
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
I prefer to have descriptive comments and failure messages. It will be difficult to keep the description you suggest in sync with the test implementation.
- TEST_WINVER This contains the value of the '-winver' Wine
argument, or the Windows version if the test is being run in Windows.
We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...),
etc.
(Do we need to distinguish between Windows and Wine?)
Yes, we do. Sometimes we have implemented behavior only for one Windows version, e.g. NT. In this case switch -winver won't affect Wine behaviour.
- TEST_BATCH If true (equal to 1) or unset, then the test
should assume that it is being run from within the test framework and thus that it should be non-interactive. If TEST_BATCH is set to 0, then the
test can assume that it is being run in interactive mode, and thus ask questions to the user. Of course most tests will simply behave identically in both cases,
I strongly recommend to use only batch version. If somebody has reasons to play with different forms of input, he can use any of following easy options: a) hardcode it - original test is improved ;-) b) take the test code to his interactive application c) add interaction with user to his local copy of the tests
Running tests
In Wine:
'make tests' seems the best way to do things. But some tests may need to create windows. For instance I have a DIB test that creates a window, draws a number of DIBs in it and checks the bitmap bits of these DIBs and then exits. Thus it is
a non-interactive test. I am not really sure whether
the window actually needs to be made visible or not, but even if this particular exemple does not require it, I suspect that othersm checking for message
sequences
for instance, may need to make the window visible.
I have tests which show window to use window messaging. Idea about tests separation to cui and gui looks good. You can run cui tests while doing something else.
In Windows:
Hmmm, not sure how that is done. Run
'winetest.exe'?
Something like: perl run_all_tests.pl
The unit test application we develop will be a test of Win32 API in general, useful for all implementations. I'm for keeping this test in separate directory tree, not mixing it with Wine files.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com