100% agreed
well, I started thinking on that... there could several uses of a test harness for Wine. We could either define this is a good result or not. But this is a bit deeper than that. For examples, some cases should work in Win98 emulation but fail for Win2k... So, I had rather in mind to have a two passes approach pass 1 : let the scripts run and output the results pass 2 : analyze the results. This could be either comparing with a manually edited reference case, or reference cases could be generated from running the program under Win <put any version here>.
So, this imposes to have the test programs also (compile) and run under windows... This could even allow (from source code for example) to have 3 test caes : 1/ compiled and running under windows 2/ compiled under windows, but run with wine 3/ compiled as a winelib app
normally, there shouldn't be too many differences between 2 and 3... but, we all know what "normally" means...
Of course, we don't need to start with all that
in order to help the pass 2 (and even produce all possible reports), I had in mind (but didn't give a good look at it) to let the test scripts produce an XML output. This would allow the test harness to spit out any information (like used DLLs native/builtin, version of those, version of windows (emulated by wine or native...), plus the output by itself
analysis and report should be (partially) driven by some XSL/XSLT tools this may be a bit overkill to start but could be a reasonable target
on the test cases basis, I wrote a few of them (as any of us has done more or less), basically tackling the multimedia DLLs
however, it's also a good basis to reverse engineer at the API level (or better understand some un^^ poorly documented features of the APIs)
for developpers, the make test should be made more fine grain (IMO). Like (not necessarly thru make) test a DLL (if some modifications have been made to it) or even an entry point... (or a set of entry points). It's the way I started splitting the test cases (DLL / set of features...). This has drawbacks. For example, it's rather hard to test, say, Unicode support accross all DLLs
no many thoughts on this one either ... did you try the windows messages journaling API ? IIRC, some early MS test programs did use it.
A+