Dan Hipschman wrote:
On Mon, Apr 28, 2008 at 08:37:26PM +0100, Robert Shearman wrote:
This should aid in testing more-obscure parts of the parser that aren't necessarily valid when using RPC (and hence don't make sense being put in dlls/rpcrt4/tests/server.idl).
Obviously this is a good idea. I tried doing something similar a while ago, but it didn't get accepted on the first try and I never asked why, so I'll ask why now. See here:
http://winehq.org/pipermail/wine-patches/2006-September/030438.html
It has some benefits over the approach you're proposing so it's worth bringing up. One is that it can run the tests against MIDL, so we know the tests themselves are correct (ok, MIDL has been known to do things differently than the spec, but we may want to copy MIDL's behaviour anyway, and in any case it's easier to automatically validate the tests on MIDL than to do so by hand or by inspection).
The above linked patch of yours is what actually inspired me to add automated testing of widl. I don't see any reason why we can't run MIDL with the same framework. In fact, I think the framework I am proposing could better handle this case due to being able to expect specific errors/warnings that differ between MIDL and widl (and being able to cope with MIDL failing and widl succeeding, for example, due to bugs in MIDL).
Another is versatility in what it can test. Since the tests are shell scripts, not only can it test the success/failure of a parse, but it can check that all the correct files were created, and it can perform further tests on the output (e.g., if we have an import in the IDL file, we can grep the output to make sure a #include was generated for the import).
It already supports "todo"s, and the test scripts look similar to the usual Wine tests written in C.
Absolutely. The flexibility of your system being able to test the contents of files and do other checks for specific tests is a definite bonus.
On the other hand your method has some advantages over mine. You already provide a "make test" framework,
I don't see any reason why your framework can't also be plugged into "make test".
and including the expected result of the compilation in the IDL file is cleaner.
Again, that could also be done using your approach. It's just that it was necessary with my approach.
Alexandre may very well disagree with what I see as valid points (like testing on MIDL), but these are my suggestions anyway. Hope they're useful.
The way I see it, we have a choice between having a framework that uses the makefile to run individual tests of the parser without checking the content or a framework that runs every test in one go, but is capable of checking the output of the generated files. The only technical advantage that I can think of for the former is that it allows the tests to be performed in parallel, but I don't know how much of a benefit that is to Alexandre (who will be the one doing "make test" the most).