Francois Gouget fgouget@free.fr writes:
So the test has to do the diff itself. And no calling 'diff' because it's not portable :-(
But you shouldn't need diff at all. You just have to write the test slightly differently to have checks instead of printfs.
The normal behaviour of the tests is that they are oblidged to provide an 0 exit code on success; I think that is an excellent standard. However, having the diff feature allows us to more rapdily adapt existing programs to become tests. Since it's done (and it's trivial code), I don't see the harm in leaving in the feature. We can hide it/discourage in in the (as yet unwritten) doco if you like.
A C test should just be a .spec file and a .c file. If they are all handled the same way it's actually quite simple. In 'Programming Windows 98' I have 148 executables/libraries and 171 C files. It's quite manageable.
I'm not sure about that; if you run make in libtest/ you end up with 70 files and more than 7Mb of disk space, for just 10 nearly empty programs. And the makefile is a big mess too.
The entire samples directory that I submitted consisted of 2,705 bytes. For all CVS checkouts and 'normal' Wine use, there will be an insignificant penalty. And yes, in the framework I submitted, if you choose to run 'make test', it chews disk space (and the Winelib tests are fairly slow to launch). But the Makefile.in is, IMO, clean and tight, and easy to replicate. And we don't *have* to use Winelib/C tests - the Perl tests work fine. Further, if we had 70 regression tests, that would be exactly 70 more than we have now! We can fix the big bloatedness later.
Finally, the more I think about my structure of using the '.test' files, the more I believe it is the right solution, because I think it gives us great long term flexibility.
For example, say we have 70 Winelib/C tests that are big and bloated and we want to fix that. We could define a new make target, say a '.test.so'. We could modify winetest to be able to load a .test.so file and call main() with the right parameters and suddenly all of our bloat is gone. We'd just need a rule to build a .test.so file from a .test.c file, and then we'd need to tweak our foo.test file to add a new command 'so_test=foo.test.so' instead of 'invoke=foo' and we're done. We could probably even eliminate the .spec file (only for console apps, obviously).
And another thing, what if I have an app that creates a graphical image and then uses Gimp or some other scripting process to take screen shots and then compares them to a known reference? In a makefile only solution, it'd be a pain to add a target to call some fancy script to perform a regression test. This case is already handled by the .test file.
And don't forget the Julian fries! The very best feature of the framework! <g>
Jer