--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
Always succeed *under Windows*. Do you really,
really,
really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
The value of unit tests is exactly in failures!
The
more failures of unit tests we have - the better
test
developers do their work.
Well, I guess it's a philosophical point, but for me the value of the tests is for regression testing. If you allow the tests to fail you'll pretty soon have 90% of the tests fail somewhere, and this is completely useless except maybe as a list of things we still have to do. While if the tests usually succeed, as soon as something fails you know there's a regression and you have to fix it.
As Francois mention, this is why TODO tests exists. Even without TODO tests it is not wise to reject a perfectly correct patch from a Windows developer who even does not have Wine.
Without TODO construction compromise is still possible. E.g we can use conditional compilation or execution to select only tests which succeed for finding regressions. I believe there will be somebody, interested in mining "dirty" tests. Other option is to store failture lists and compare them from time to time in a search of regressions.
What you can do with my make test patch is run make test -k first, let it go through everything, and then run make test again and it will only run the tests that failed the first time, or that have been modified. This is a major gain. It could certainly be done some other way too without using make, but AFAICS your test harness would force to run all tests all the time (or to disable them manually). This is not acceptable once you have a large number of tests.
I don't see poing in selecting subsets of tests. From experience - even with pretty big number of tests it does not take long time to execute them.
OTOH a Wine test suite can happen (I hope) because this is something Wine developers need when they write code, so there is at least some motivation for them to write tests.
Exactly! I do not want to spend resources of Wine project on some "nice documentation". On the contrary, goal of this change is to invite external resources.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com