Eric Pouech Eric.Pouech@wanadoo.fr writes:
So, I had rather in mind to have a two passes approach pass 1 : let the scripts run and output the results pass 2 : analyze the results. This could be either comparing with a manually edited reference case, or reference cases could be generated from running the program under Win <put any version here>.
Yes, that's the idea; the test script will output a bunch of results which can then be compared against a reference run generated under Windows. We probably also want a standard config file (or a few of them to handle version differences) to ensure the test environment is the same for everybody.
So, this imposes to have the test programs also (compile) and run under windows... This could even allow (from source code for example) to have 3 test caes : 1/ compiled and running under windows 2/ compiled under windows, but run with wine 3/ compiled as a winelib app
The idea of using an interpreter like Perl is precisely that you don't need to compile anything to run tests. I think this is important because not everybody has a Windows compiler. It also allows using the exact same test script under Windows and Wine, so that you don't have to worry whether your Windows binary exactly matches your Winelib binary.
in order to help the pass 2 (and even produce all possible reports), I had in mind (but didn't give a good look at it) to let the test scripts produce an XML output. This would allow the test harness to spit out any information (like used DLLs native/builtin, version of those, version of windows (emulated by wine or native...), plus the output by itself analysis and report should be (partially) driven by some XSL/XSLT tools
I don't think we need anything fancy like that. The output should be simple ASCII that can be automatically compared with diff, and the test is considered a failure if diff finds any difference against the reference output. Everything should be automated as much as possible.
for developpers, the make test should be made more fine grain (IMO). Like (not necessarly thru make) test a DLL (if some modifications have been made to it) or even an entry point... (or a set of entry points).
Yes, there should be one set of test scenarios for each dll, and each scenario should test one entry point (or a few related ones); this way you can either run a single test, or do a make test in a dll directory to run the tests for this dll, or make test at the top-level which will simply iterate through all the dlls. The tests should ideally run fast enough that you can do a make test in the dll dir everytime you change something.
It's the way I started splitting the test cases (DLL / set of features...). This has drawbacks. For example, it's rather hard to test, say, Unicode support accross all DLLs
But you should never need to do that. If you change something in a core Unicode function in kernel, this will be tested by the kernel test scenarios. If the test scenarios don't find a problem but tests for higher level dlls fail, then your kernel test scenarios are buggy since they didn't spot the change.