I can't say I like your *.test files; I think this should be taken care of by the makefile directly.
I started down this road, but then I ended up needing PERL_TESTS and INVOKE_TESTS in the Makefile, and IMHO, that was uglier than the *.test file. This may simply be a lack of Makefile moxy on my part; the point where I threw up my hands in disgust was when I was requiring my Winelib/C tests to be compiled into a 'foo.itest' so that I could do a .itest.test.out dependency rule.
Also it seems the consensus is that we should simply check the exit status of the test, not compare the test output.
Sorry, this wasn't clear. That is the default behavior. I preserved the pattern and reference checking as an optional addition.
Flaws: 1. It has no provision for use on Windows. IMO the right way to fix this is to go through the effort of making Wine's ./configure script be intelligent enough to build 'Winelib' programs, so that in Windows a 'make test' should just work.
I'm not sure about that; I think it would be preferable to have a simple, self-contained environment. Running configure would require a lot of stuff to be installed on the Windows side.
Well, I was sufficiently impressed with the Cygwin install to think it wouldn't be that bad; if nothing else, I think we should try to have both.
2. I really don't like that a C/Winelib test requires its own directory (see samples/sample3). However, AFAICT, that was the only way to create a simple and clean build environment for the Winelib app.That's the problem with C tests. You cannot have one executable for each test, at least not with Winelib, so we need a way to build multiple tests inside a single executable. Not very hard, but there's a bit of work involved.
I would argue, that except perhaps for your objection to my *.test format, that the framework as I have proposed would be a good start. We can refine the Winelib stuff later.
It seems to me that the alternate is to spend another five weeks debating possible frameworks, and ending with none implemented.
Jer