Alexandre Julliard wrote:
Jeremy White jwhite@codeweavers.com writes:
The normal behaviour of the tests is that they are oblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
The problem is that it's hard to support both types of tests from the makefile, so it would force tests without output to generate dummy output and reference files. I think it's better to make the default be no output and have tests that need output diffs do the extra work.
Um...the way I constructed it, there is no additional effort required in the Makefile whatsoever. Now, if you get rid of my .test file, yes, then, doing this all in the Makefile is a pain in the neck.
"Normal" Wine use should definitely include running regression tests. I don't want to be the only one to run them.
Point taken; but I still believe that this is an issue not relevant to my proposed patch, but is a problem with using C tests as a whole. A solution to this problem will be just as appropriate with or without my patch.
But without your .test file it's even easier to do, all you have to do is to hack the makefile. Plus you can get the dependencies right (how can make guess what file your .test is referencing?)
If the goal is to make it easy for a newcomer to add a test, then requiring them to hack the Makefile is going to be a barrier. Asking them to copy and cut/paste a 'samples/foo.test' file will not be.
I do concede the point that my .test file hides dependencies, and a Makefile only approach can do that better.
You can do that from a Perl test too, but I don't think this should be encouraged. Tests should not depend on external programs any more than strictly necessary, otherwise people will soon need a complete Linux distribution on their Windows machine to run our tests.
I threw that out as an example, not necessarily a serious case. I think the bottom line question is whether or not the test scripts should be built such that we depend on a shell script (runtest) and a configuration file for each test (foo.test), or whether we rely completely on the Makefiles to hold the test configuration information.
I think either choice is valid, but let me lay down the gauntlet: replace my 'winetest/samples' directory with a Make based equivalent that allows for the same sorts of tests, and makes it similarly obvious to a newcomer how to add a new test. You are better with Make than I am; you may persuade me readily with a patch.
Jer
p.s. If you're just rejecting the .test files because they have long horizontal comment lines, we could negotiate that...<g>