Alexandre Julliard wrote:
I don't really care about the output. My point is
that this output is
not necessary, a simple "assertion foo failed at
line xx" on failure
and no output at all on success would work just as
well. I've nothing
against such an output either, but I don't think it
justifies
introducing the complexity of reusing the Test
stuff.
I already created small library with api like in Test::Simple - functions ok, TODO, SKIP. The implementation can be easy changed to the behaviour you described. I also suggest for library to fail on successful TODO test. TODO tests will be conditional - they are TODO only under Wine, not on Windows.
Careful usage SKIP sections can easy cover differences between Windows platforms, using TODO - between different non-Microsoft Win32 implementations.
On other hand, SKIP sections were important for Test::Harness, which needed to control number of tests. For your goals conditional execution of the tests is sufficient.
Which approaches can you suggest to reach goals I believe are important:
1) bundling - are we going to have separate distributions - Wine with tests, tests only, Wine only? There are a lot of cases where only one of them is required.
2) development of the unit tests under Windows. Obviously, we don't need to have Wine itself when we work with unit tests on Windows. Plus, we need to create development environment, usable by Windows developers.
3) Organization of the unit tests in such way, so they can be used by other Win32 implementation projects. Conditional TODOs I suggested above will help to manage different TODO lists for different projects.
4) besides narrow-scope regression testing you suggested we need to run all the tests from time to time, even if it takes more than 1 hour to run the whole suite. The reasons for this are:
a) we need to run the tests on all Windows platforms to make sure the tests correctly document their behaviour
b) Wine should be reguraly tested by the whole testing suite. Some regressions will be caught in unit tests of the modules other than changed. In other cases changed module may not have tests at all, but will brake modules, dependent on it.
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Alexandre, attached is very simple framework which suits your testing model.
On any failure the framework prints an error message and exits with error code 1.
The framework features: * ok function - fails if condition is FALSE * conditional TODO sections - ok functions fail in TODO section if its condition is TRUE and TODO condition is TRUE. * get_w32_impl function which returns implementation of Win32 api the test is running on. In our case these are W32_MS, W32_WINE. Is used for conditional TODO
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Andriy Palamarchuk apa3a@yahoo.com writes:
Which approaches can you suggest to reach goals I believe are important:
- bundling - are we going to have separate
distributions - Wine with tests, tests only, Wine only? There are a lot of cases where only one of them is required.
My view is that everything is distributed with Wine, and we have a script on WineHQ that builds a zip of the test-only environment for use under Windows, and/or a script to fetch it from CVS on Windows.
- development of the unit tests under Windows.
Obviously, we don't need to have Wine itself when we work with unit tests on Windows. Plus, we need to create development environment, usable by Windows developers.
For Perl we need to ship a winetest.exe and a couple of scripts to run through the tests. For C we need to generate makefiles one way or another, including support for the major Windows compilers.
- Organization of the unit tests in such way, so they
can be used by other Win32 implementation projects. Conditional TODOs I suggested above will help to manage different TODO lists for different projects.
Looks good, though I would suggest having simply a TODO_WINE instead of making people write the same test thousands of times. Then we can add TODO_ODIN or whatever if the need arises. And I think the TODOs should be controlled by a command-line option, so you can switch them on under Wine too. But these are details, I think overall it looks quite good.
--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
- Organization of the unit tests in such way, so
they
can be used by other Win32 implementation
projects.
Conditional TODOs I suggested above will help to manage different TODO lists for different
projects.
Looks good, though I would suggest having simply a TODO_WINE instead of making people write the same test thousands of times. Then we can add TODO_ODIN or whatever if the need arises.
Good idea, updating the library.
And I think the TODOs should be controlled by a command-line option, so you can switch them on under Wine too.
Do not understand why we need to have the command line option. TODO_WINE will automatically detect when the application runs under Wine. Under Windows the code in TODO_WINE will be executed like there is no TODO.
But these are details, I think overall it looks quite good.
I'm also satisfied by the overall status. Don't have big experience in build process creation, leaving this part up to you. Otherwise I'm willing to work further on unit tests.time.
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Andriy Palamarchuk apa3a@yahoo.com writes:
Do not understand why we need to have the command line option. TODO_WINE will automatically detect when the application runs under Wine. Under Windows the code in TODO_WINE will be executed like there is no TODO.
The idea is that you might want to run the TODO_WINE tests under Wine in Windows mode, for instance to find a problem you'd like to work on. You could grep for TODO_WINE in the test scripts but this is less convenient IMO.
--- Alexandre Julliard julliard@winehq.com wrote:
The idea is that you might want to run the TODO_WINE tests under Wine in Windows mode, for instance to find a problem you'd like to work on. You could grep for TODO_WINE in the test scripts but this is less convenient IMO.
Understood. IMO grep is better - you can choose from list of TODO's. Lets leave it to the time when somebody needs the feature.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
-
Alexandre Julliard -
Andriy Palamarchuk