Andreas Mohr wrote:
I guess we really should change our development
model from trying tons of
programs to *systematically* testing functions and
Windows mechanisms now.
If we can show everyone where stuff is failing, it
might be a lot easier
to attract new people.
I *completely* support this idea. Benefits of such test suite are enormous. Existing developers can contribute a lot by adding test snippets for the functions they create. Now they create such snippets anyway and throw them away.
I attached a preview of the posting I intend to post
on *tons* of Windows
devel newsgroups ("Call For Volunteers"). That way
we might actually get
hold of hundreds of Windows developers helping us
implement a complete
test suite (complete tests of up to 12000 Windows
functions).
Not to mention the additional PR we might get out of
this...
Lets ask for help only after the suite structure is more/less defined and we'll be able to give people something to work on.
Comments: - Don't want to reinvent the weel. Is there any existing test suite framework we can use? Sorry, I can't suggest any for C but I'm very impressed with JUnit in Java. It is even Ok if the framework is GPLed or LGPLed - I don't think any company will make buziness based on the test suite. - I /personally/ prefer CL interface only for such suite - it would be better if the suite print summary information and information about failed tests only - make the test suite more "visible" for existing developers. Ask them to run the test suite before submitting a patch? - I think the suite test will consist from a few separate applications because different tests may have different requirements to GUI configuration, processes, etc. We need a way to run all the applications in one batch. - define variable which indicates whether the suite runs under Wine. Such indicator can be used for Wine "white-box" testing. - it would be greate to have functionality to support output comparison? For some functionality it is easier to write tests to compare output instead of doing explicit checks (e.g. tests, involving a few processes). The output can be redirected to file and files compared. If we use files we need to store files for Wine and a few versions of Windows :-( - the suite applications size will be pretty big. Is it better to move it to separate CVS tree? - what about running the suite weekly (or daily) automatically and publishing the results to wine-devel? - most developers on this list have access to one version of Windows. Is it difficult to create "testing farm" with remote access to a few versions of windows? This would help developers to test their code on a few platforms. Existing environments in the companies, involved in the project can be used. - I remember long time ago there was a post on wine-devel about using Perl or Perl-like language for unit testing. What is current status of that project?
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com