- its exit code
- text output on either or both of stdout and stderr, both of which are
normally redirected to a file called 'xxx.out'.
A test succeeds if:
- its exit code is 0
- and its output, 'xxx.out' matches the reference output according to
the rules described later.
I think that it would be handy to use stderr for status/diagnostics, and only use stdout for reference checking. Chatty people like me get their statii, but the tests remain clean. Perhaps a WINETEST_DEBUG env variable would be a good addition as well (and since Alexandre controls the commits, my guess is it'll default to off <g>).
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
I hope that the cases where this would be needed would be few enough that we wouldn't need to build in a general purpose exception; we can just have a mytest.win95.test and a mytest.win98.test for the cases where it's needed.
[snipping chunks largely agreed with]
Test coverage
Each test should contain a section that looks something like:
I started trying to tweak Alexandre's patch to create some sample tests, and I learned the following: 1. Cygwin installation has *dramatically* improved. Getting a full working toolchain is no longer a big pain in the rear end, it's actually pretty easy.
2. Having '.test' files be implicitly Perl scripts is too limiting, IMHO. I hate to add another format, but I've been toying with YAFF (yet another file format) so that a '.test' file describes a test, as follows: # Comment lines script=name_of_my_perl_test_script invoke=name_of_c_test_or_shell_script status=-eq 0 pattern=.*OK compare=name_of_my_ref_file
where one of script or invoke is required, and status, if given, is a test expression that $0 is compared to, pattern is a regexp applied to stdout, if given, and compare is the name of a .ref.out file to compare the output with.
The default would be just a case of 'status=-eq 0'.
The nice thing about this approach is that you can handle the multiple version testing just by creating a new reference file and a new .test file.
In Windows:
Hmmm, not sure how that is done. Run 'winetest.exe'?
IMO, we should have a script that creates a 'winetest.zip', with simple batch files to make it easy to run a single test.
But, (as a new convert to the ease of use of Cygwin), I think we can just stick with 'make tests' for the full deal on Windows.
Jer