Alexandre Julliard wrote:
Robert Shearman rob@codeweavers.com writes:
The way I see it, we have a choice between having a framework that uses the makefile to run individual tests of the parser without checking the content or a framework that runs every test in one go, but is capable of checking the output of the generated files. The only technical advantage that I can think of for the former is that it allows the tests to be performed in parallel, but I don't know how much of a benefit that is to Alexandre (who will be the one doing "make test" the most).
We definitely have to be able to run tests individually from make, so there can't be just a single script to run them all.
Still, it seems to me that most of these tests can just as well be done in the existing framework, as part of the rpcrt4 test for instance. This way we can not only make sure that the code compiles, but also that the generated code builds, and works the way it should.
I don't really like the idea of mixing tests of two different components into the same file. Also, when developing and trying to debug a regression I prefer to work on simpler IDL files (i.e. testing a particular type of statement in a few ways) than everything being in one file and having to work out what statement broke what.
The only thing that can't be tested that way is obviously the code that is expected to fail to build, and for this something like Rob's framework would work fine, even though I'm not quite convinced that we care that much about getting the failure cases exactly right.
I see the failure cases being important for two reasons: 1. That the line number is reported correctly. 2. That the error is being triggered by the right part of the statement and therefore makes sense to the user.
Of course the failures themselves are important in order for incorrect constructs to be detected before: 1. widl crashes during generation of output files. 2. widl generates files that can't be compiled or compile with warnings. 3. widl generates files that compile correctly but crash or raise an exception at runtime.