Andriy Palamarchuk apa3a@yahoo.com writes:
Always succeed *under Windows*. Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
The value of unit tests is exactly in failures! The more failures of unit tests we have - the better test developers do their work.
Well, I guess it's a philosophical point, but for me the value of the tests is for regression testing. If you allow the tests to fail you'll pretty soon have 90% of the tests fail somewhere, and this is completely useless except maybe as a list of things we still have to do. While if the tests usually succeed, as soon as something fails you know there's a regression and you have to fix it.
Wrong. In make files you can check only some compilation dependencies. You can't do anything about logical dependencies. I'm for running the whole suite of the tests by developers before submitting *any* patch and from time to time centrally.
What you can do with my make test patch is run make test -k first, let it go through everything, and then run make test again and it will only run the tests that failed the first time, or that have been modified. This is a major gain. It could certainly be done some other way too without using make, but AFAICS your test harness would force to run all tests all the time (or to disable them manually). This is not acceptable once you have a large number of tests.
This is why I want very much to keep it in separate directory. Even the tests structure, probably, should correspond more to logical Win32 API structure, not to Wine directory tree.
Well, the Wine directory tree corresponds (mostly) to the Win32 API structure. But this is again a philosophical point. I don't view the test suite as an API specification, but as a regression test suite for Wine.
The Win32 unit test suite has huge value by itself. All Win32 projects can contribute to it on the same rights. The ideal solution, probably, whould be to move completely separate project. We can do this later if the need arises.
This is like the documentation: yes it would be great to have good documentation and good specifications of the Win32 API. It's not going to happen, because there's no motivation behind it except "would be nice to have".
OTOH a Wine test suite can happen (I hope) because this is something Wine developers need when they write code, so there is at least some motivation for them to write tests. We can do it in a way that the Wine test suite is useable for more than testing Wine, but this has to be the main focus.