Dan Kegel dank@kegel.com writes:
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
In that case the test should use a TODO mechanism or equivalent, and it must still be possible to run make test without failure (but there would be an option to switch the failures on if you want).
That means you probably want to live with less than 100% success rates. The important thing when committing a new change to Wine (as opposed to a change to the test suite) is that it not cause any *new* failures. I bet that's what you meant.
No, what I mean is that you can't spot new failures if every test run shows hundreds of existing ones. The only way to find new failures is if you can do a successful test run before a change.
Imagine that you have 1000 tests, and a typical run shows 250 failures. Then you make a change, and you now see 248 failures. Does it mean you fixed 2 bugs, or does it mean you fixed 5 and introduced 3 new ones? You have no way of knowing without going through the 250 failures one by one. This is clearly not possible.