On 2 Jan 2002, Alexandre Julliard wrote:
Francois Gouget fgouget@free.fr writes:
[...]
IMO the test should not be printing successes or failures at all. If it can determine whether some result is OK or not, it should simply do an assert on the result. Printing things should be only for the cases where checking for failure is too complicated, and so we need to rely on the output comparison to detect failures.
I am not sure about using asserts. In the case where one calls the same API ten times with different parameters, it would probably be better to still do all ten tests even if the second call fails. This way the developper debugging things immediately knows if just one aspect failed or if its everything. If we call assert as soon as one item fails we don't have the information of whether the rest still works or not. Of course this assumes that a failure is not fatal for the rest of the tests. If it is, then an assert could make sense. But I agree tests could be written so as not to print anything unless there is a failure. Especially for simple cases like checking return codes, there is no need to print the return code and to check it against a reference file. Then a single 'Test: Ok' before the final return would make sure we don't get an empty output file if that is important.
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
No, there should always be a single .ref file IMO. Version checks should be done inside the test itself to make sure the output is always the same.
The same way there are cases where the simplest solution for checking the result is correct is to compare it against a reference file, I'm thinking that there are cases where the somplest way to make sure we behave the same way as a specific windows version is to compare the output to that windows version. But I must admit that I don't have such an example for now so we can start with the single '.ref' file for now. Then, if necessary, the above mechanism should be relatively simple to put in place.
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
This is already part of the Perl framework, you have to explicitly declare the functions you use. So we don't want to duplicate the information.
But in the case where a test has to call GetVersion() to handle differences between windows versions, we surely don't want to go on record as saying that this test tests GetVersion. Similarly, pretty much all GUI tests will call things like RegisterClassExA, CreateWindowExA, GetMessageA, ... then, finding the two or three tests that really test these will be pretty hard. Maybe the test framework could have an alias for "wine::declare", maybe "wine::test". Then APIs declared with "wine::test" are those that are being tested while those declared with "wine::declare" are just overhead.
- TEST_WINVER This contains the value of the '-winver' Wine argument, or the
Windows version if the test is being run in Windows. We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...), etc. (Do we need to distinguish between Windows and Wine?)
The test should use GetVersion() and friends IMO, no need for a separate variable.
Good point.
- and add two corresponding targets: 'make cui-tests' runs only those
tests that do not pop up windows, and 'make gui-tests' runs only those tests that do pop up windows
- 'make tests' would be 'tests: cui-tests gui-tests'
I don't think this complexity is necessary. You can always redirect the display if the windows annoy you. And tests should try to keep the windows hidden as much as possible.
Ok. I guess we can still split tests in two categories later if necessary.
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
I have attached two sample Perl scripts that were written some time ago by John Sturtz and myself. One is testing the atom functions and the other is creating a window. They should probably be simplified a bit to serve as documentation samples.
Nice :-)
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ "Utilisateur" (nom commun) : Mot utilisé par les informaticiens en lieu et place d'"idiot".