Jeremy White jwhite@codeweavers.com writes:
The normal behaviour of the tests is that they areoblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
The problem is that it's hard to support both types of tests from the makefile, so it would force tests without output to generate dummy output and reference files. I think it's better to make the default be no output and have tests that need output diffs do the extra work.
The entire samples directory that I submitted consistedof 2,705 bytes. For all CVS checkouts and 'normal' Wine use, there will be an insignificant penalty.
"Normal" Wine use should definitely include running regression tests. I don't want to be the only one to run them.
For example, say we have 70 Winelib/C tests that are bigand bloated and we want to fix that. We could define a new make target, say a '.test.so'. We could modify winetest to be able to load a .test.so file and call main() with the right parameters and suddenly all of our bloat is gone. We'd just need a rule to build a .test.so file from a .test.c file, and then we'd need to tweak our foo.test file to add a new command 'so_test=foo.test.so' instead of 'invoke=foo' and we're done. We could probably even eliminate the .spec file (only for console apps, obviously).
But without your .test file it's even easier to do, all you have to do is to hack the makefile. Plus you can get the dependencies right (how can make guess what file your .test is referencing?)
And another thing, what if I have an app that creates agraphical image and then uses Gimp or some other scripting process to take screen shots and then compares them to a known reference? In a makefile only solution, it'd be a pain to add a target to call some fancy script to perform a regression test. This case is already handled by the .test file.
You can do that from a Perl test too, but I don't think this should be encouraged. Tests should not depend on external programs any more than strictly necessary, otherwise people will soon need a complete Linux distribution on their Windows machine to run our tests.