Alexandre Julliard wrote:
Dan Kegel dank@kegel.com writes:
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
In that case the test should use a TODO mechanism or equivalent, and it must still be possible to run make test without failure (but there would be an option to switch the failures on if you want).
The dejagnu test harness that is used by e.g. the gcc test suite allows to classify a test case as 'expected to fail'.
When you run the test, every test case either passes or fails, which results in a classification into four subsets:
PASS Test case was expected to pass, and it did FAIL Test case was expected to pass, but failed XPASS Test case was expected to fail, but passed XFAIL Test case was expected to fail, and it did
Only a test case in the FAIL category causes the whole test run to fail, and is reported even in the test run summary. The other categories are only reported as total numbers.
If you are getting nonzero FAIL numbers, you have introduced a regression. Nonzero PASS and XFAIL numbers are expected; if you get nonzero XPASS numbers you might look at the cases in question and decide whether you want to remove the 'expected to fail' flag.
This system works quite well in my experience with gcc, maybe something like this could be implemented for Wine as well ...
Bye, Ulrich