--- Andreas Mohr andi@rhlx01.fht-esslingen.de wrote:
On Wed, Dec 26, 2001 at 10:07:20AM -0800, Andriy Palamarchuk wrote:
Andreas Mohr wrote:
[... skipped ...]
- it would be better if the suite print summary
information and information about failed tests
only Yep. Current output is something like:
WINETEST:test:Loader_16:LoadModule:FAILED:01:[retval]
WINETEST:test:Loader_16:LoadModule:FAILED:12:[retval]
WINETEST:test:Loader_16:LoadModule:FAILED:13:[retval]
or, in case of success, only: WINETEST:test:Loader_16:LoadModule:OK
I mean something like: =================== Run: 1234 tests Failed: 2 Errors: 1
Fail 1: <....> Fail 2: <....> Error 1: <....> =================== In the example above failture means condition check failure, Error - exception.
I suggest to print nothing for successfull tests. At least this is the way I am accustomed with JUnit. We are not interested in successfull tests, are we? ;-)
This output is pretty useful, I think: It can be parsed *very* easily, and grepping for regressions is also pretty easy.
"WINETEST" exists to be able to distinguish this output from bogus Wine messages, "test" indicates that this is a test line output versus a warning message or similar output, "Loader_16" indicates testing of 16bit loader functionality, "LoadModule" - well... ;-) "FAILED" - obvious "01" - test number 01 failed. "[retval]" - contains the (wrong) return value of the function, if applicable.
Looks simple and the output is really useful. I just don't see any reason to show information about successfull tests. At least we can get short form of the output by clipping all "Ok" messages from your suggested form.
BTW, I think having a test suite wouldn't be about hunting regressions at first: just look at my LoadModule16 example and you'll see that we're still quite far from hunting regressions *only*. My guess is that we'll be shocked at how many functions fail in how many ways.
Agree, agree, agree... We can even use eXtreme Programming approaches :-) See http://xprogramming.com/ and other sites on the subj. I also like this article: http://members.pingnet.ch/gamma/junit.htm I use JUnit extensively and like the whole idea.
- make the test suite more "visible" for existing
developers. Ask them to run the test suite before submitting a patch?
No, I don't think so. I think it suffices if Alexandre runs the test suite before or after every large commit cycle. That way he'd be able to back out problematic patches. Asking developers to run the *whole* test suite for each patch could be pretty painful.
I don't see why running the unit tests is paintful. I'd estimate that it would not take more then 5 minutes to test all 12000 W32 functions. We also can keep tests for slow/rarely changed areas of API in separate "complete" suite.
I think the test suite is for developers, not for Alexandre (I meas as a team leader :-) or QA. This is why I want to increase "visibility" of unit tests. Again, the developers will more likely to contribute to the suite if they will remember about it.
I do not suggest to enforce the unit test usage because we'll always have developers/companies who don't want to do that. It would suffice to recomment before submitting a patch to check that we have the same (accidentally - less :-) number of failures as we had before or report any new bugs introduced. It is even Ok to have increased number of issues as soon as developer consiously makes decision to break something. Compact tests output I describe above also will help to quicky identify any changes in unit tests output.
We'd also need to pass a winver value to the test suite via command line in order to let the test app adapt to different windows environments (and thus also to different wine --winver settings !).
Sounds good.
- it would be greate to have functionality to
support
output comparison? For some functionality it is
easier
to write tests to compare output instead of doing explicit checks (e.g. tests, involving a few processes). The output can be redirected to file
and
files compared. If we use files we need to store
files
for Wine and a few versions of Windows :-(
Hmm, I don't quite get what exactly you're talking about.
Example: I have pretty big unit test for SystemParametersInfo function. Part of the test is to insure that WM_SETTINGCHANGE window message is fired when necessary. I have simple handler for the message which prints confirmation when the message received. I save output when I run tests under Windows and Wine and compare the output. Advantages - 1) simplicity, 2) I can see contents of the failure. To do explicit check I need to set up some communication (common variable, step counter etc) between the message handler and testing code. If these 2 code snippets are in different processes I need to use IPC to do explicit check?
Ideally I'd like to pring nothing to the screen - developer does not need to see all this information. The information can be saved to file, and I need to keep separate files for Wine, (a few versions of ?) Windows.
- what about running the suite weekly (or daily)
automatically and publishing the results to wine-devel?
Good idea ! Might prove worthwhile.
For this feature compact output is useful too.
- most developers on this list have access to one
version of Windows. Is it difficult to create
"testing
farm" with remote access to a few versions of
windows?
This would help developers to test their code on a
few
platforms. Existing environments in the companies, involved in the project can be used.
Hmm, why ? The idea is that hundreds (or hopefully thousands ?) of volunteer Windows developers create bazillions of test functions for specific API functions. That will happen on specific Windows version only, of course. Now we have a test framework for a specific API function on a specific Windows version. Now if there are behavioral conflicts on different Windows versions (functions behave differently), then I guess people will notice immediately and fix the test function immediately to support different behaviour of different windows versions. --> no problem at all.
I was thinking not about unit tests only. Sometimes I'd like to know how different version of Windows behaves. The only option I have is to ask somebody who has such version to run a test (honestly - up to now I was lazy to ask anybody :-). But you are right - it is not a big issue.
- I remember long time ago there was a post on
wine-devel about using Perl or Perl-like language
for
unit testing. What is current status of that project?
Hmm. That'd be programs/winetest/, right ?
Lazy me ;-)
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com