January 2002 - Wine-Devel - list.winehq.org

Re: We *really* need a development model change !

by Alexandre Julliard

Andriy Palamarchuk <apa3a(a)yahoo.com> writes: > I suggest to use explicit checks and print descriptive > messages in case of falure. I agree, this approach is > more labour-intensive, especially for tests using IPC. > It is also much more maintainable as soon as you coded > it. Everything, including differences between > different Windows platforms is documented directly in > the code! This gives much better control. E.g, it is > possible to comment-out part of the test, still > getting meaningful results what worked, what did not. I definitely agree here, having the code check everything itself is much better that having to compare the output. The drawback is that it makes tests more complex to write, so I'm not sure if we want to make it the rule for all tests. -- Alexandre Julliard julliard(a)winehq.com

Jan. 2, 2002

Re: We *really* need a development model change !

by Francois Gouget

On 2 Jan 2002, Alexandre Julliard wrote: > Francois Gouget <fgouget(a)free.fr> writes: [...] > IMO the test should not be printing successes or failures at all. If > it can determine whether some result is OK or not, it should simply do > an assert on the result. Printing things should be only for the cases > where checking for failure is too complicated, and so we need to rely > on the output comparison to detect failures. I am not sure about using asserts. In the case where one calls the same API ten times with different parameters, it would probably be better to still do all ten tests even if the second call fails. This way the developper debugging things immediately knows if just one aspect failed or if its everything. If we call assert as soon as one item fails we don't have the information of whether the rest still works or not. Of course this assumes that a failure is not fatal for the rest of the tests. If it is, then an assert could make sense. But I agree tests could be written so as not to print anything unless there is a failure. Especially for simple cases like checking return codes, there is no need to print the return code and to check it against a reference file. Then a single 'Test: Ok' before the final return would make sure we don't get an empty output file if that is important. > > Otherwise this file is either called: > > * 'xxx.ref' > > * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the > > Windows version being emulated. The winever-specific file takes > > precedence over the '.ref' file, and the '.ref' file, which should > > exist, serves as a fallback. > > No, there should always be a single .ref file IMO. Version checks > should be done inside the test itself to make sure the output is > always the same. The same way there are cases where the simplest solution for checking the result is correct is to compare it against a reference file, I'm thinking that there are cases where the somplest way to make sure we behave the same way as a specific windows version is to compare the output to that windows version. But I must admit that I don't have such an example for now so we can start with the single '.ref' file for now. Then, if necessary, the above mechanism should be relatively simple to put in place. > > Each test should contain a section that looks something like: > > > > # @START_TEST_COVERAGE@ > > # kernel32.CreateProcess > > # kernel32.GetCommandLineA > > # kernel32.GetCommandLineW > > # __getmainargs > > # __p__argc > > # __p_argv > > # __p_wargv > > # @END_TEST_COVERAGE@ > > This is already part of the Perl framework, you have to explicitly > declare the functions you use. So we don't want to duplicate the > information. But in the case where a test has to call GetVersion() to handle differences between windows versions, we surely don't want to go on record as saying that this test tests GetVersion. Similarly, pretty much all GUI tests will call things like RegisterClassExA, CreateWindowExA, GetMessageA, ... then, finding the two or three tests that really test these will be pretty hard. Maybe the test framework could have an alias for "wine::declare", maybe "wine::test". Then APIs declared with "wine::test" are those that are being tested while those declared with "wine::declare" are just overhead. > > * TEST_WINVER > > This contains the value of the '-winver' Wine argument, or the > > Windows version if the test is being run in Windows. We should mandate > > the use of specific values of winver so that test don't have to > > recognize all the synonyms of win2000 (nt2k...), etc. > > (Do we need to distinguish between Windows and Wine?) > > The test should use GetVersion() and friends IMO, no need for a > separate variable. Good point. > > * and add two corresponding targets: 'make cui-tests' runs only those > > tests that do not pop up windows, and 'make gui-tests' runs only those > > tests that do pop up windows > > * 'make tests' would be 'tests: cui-tests gui-tests' > > I don't think this complexity is necessary. You can always redirect > the display if the windows annoy you. And tests should try to keep the > windows hidden as much as possible. Ok. I guess we can still split tests in two categories later if necessary. > > What is needed most is a two sample tests: > > * one simple console based test > > * another one involving some GUI stuff > > I have attached two sample Perl scripts that were written some time > ago by John Sturtz and myself. One is testing the atom functions and > the other is creating a window. They should probably be simplified a > bit to serve as documentation samples. Nice :-) -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ "Utilisateur" (nom commun) : Mot utilisé par les informaticiens en lieu et place d'"idiot".

Jan. 2, 2002

Implementing Wine testing framework with Perl

by Andriy Palamarchuk

I execised with winetest to find more information about using Perl as a base for Wine testing framework. My findings so far: I built winetest without problems. Provided sample test script worked. The next my step would be to translate my own unit tests to Perl. My tests extensively use number constants, declared in C header file with defines and C structures. I worked with Perl before, but never created Perl extensions. After researching I did not find reliable way to automatically import C structures to Perl. Perl distribution provides some tools, but AFAICS the tools have limited possibilities and are not suitable for fully automatic conversion (see man pages for perlxs, perlxstut, h2xs, xsubpp). h2xs tool imported defines fine. I'm not sure how it handled functions. It does not do complete parsing of headers and files, generated files need manual editing. Another option - SWIG (www.swig.org) Can generate bindings for a few scripting languages directly from C sources "provided they are sufficiently clean" (.h?). I do not assume yet it can make sense from our sources. Are there Perl experts out there who knows how to handle this issue? Can we use other approach if automatic code generation is not possible? E.g. we can manually create binding code for used structures. C compiler is needed to create or change the bindings. Please, let me know if I missed something. Thanks, Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com

Jan. 2, 2002

Re: We *really* need a development model change !

by Jeremy White

> > >* its exit code > * text output on either or both of stdout and stderr, both of which are >normally redirected to a file called 'xxx.out'. > > A test succeeds if: > * its exit code is 0 > * and its output, 'xxx.out' matches the reference output according to >the rules described later. > I think that it would be handy to use stderr for status/diagnostics, and only use stdout for reference checking. Chatty people like me get their statii, but the tests remain clean. Perhaps a WINETEST_DEBUG env variable would be a good addition as well (and since Alexandre controls the commits, my guess is it'll default to off <g>). >Otherwise this file is either called: > * 'xxx.ref' > * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the >Windows version being emulated. The winever-specific file takes >precedence over the '.ref' file, and the '.ref' file, which should >exist, serves as a fallback. > I hope that the cases where this would be needed would be few enough that we wouldn't need to build in a general purpose exception; we can just have a mytest.win95.test and a mytest.win98.test for the cases where it's needed. [snipping chunks largely agreed with] > >Test coverage >------------- > > Each test should contain a section that looks something like: > I started trying to tweak Alexandre's patch to create some sample tests, and I learned the following: 1. Cygwin installation has *dramatically* improved. Getting a full working toolchain is no longer a big pain in the rear end, it's actually pretty easy. 2. Having '.test' files be implicitly Perl scripts is too limiting, IMHO. I hate to add another format, but I've been toying with YAFF (yet another file format) so that a '.test' file describes a test, as follows: # Comment lines script=name_of_my_perl_test_script invoke=name_of_c_test_or_shell_script status=-eq 0 pattern=.*OK compare=name_of_my_ref_file where one of script or invoke is required, and status, if given, is a test expression that $0 is compared to, pattern is a regexp applied to stdout, if given, and compare is the name of a .ref.out file to compare the output with. The default would be just a case of 'status=-eq 0'. The nice thing about this approach is that you can handle the multiple version testing just by creating a new reference file and a new .test file. >In Windows: > > Hmmm, not sure how that is done. Run 'winetest.exe'? > IMO, we should have a script that creates a 'winetest.zip', with simple batch files to make it easy to run a single test. But, (as a new convert to the ease of use of Cygwin), I think we can just stick with 'make tests' for the full deal on Windows. Jer

Jan. 2, 2002

Re: We *really* need a development model change !

by Andreas Mohr

On Wed, Jan 02, 2002 at 10:20:25AM -0800, Alexandre Julliard wrote: > Francois Gouget <fgouget(a)free.fr> writes: > > * it should be easy to correlate with the source of the test. For > > instance if a check fails, it would be a good idea to print a message > > that can easily be grepped in the source code, or even the line number > > for that check. But don't print line numbers for success messages, they > > will change whenever someone changes the test and would require an > > update to the reference files.. > > IMO the test should not be printing successes or failures at all. If > it can determine whether some result is OK or not, it should simply do > an assert on the result. Printing things should be only for the cases > where checking for failure is too complicated, and so we need to rely > on the output comparison to detect failures. Hmm, I don't know how you'd do that exactly. If we implement "strict" testing, then tons of functions will fail on Wine. And then we get an assert() every 20 seconds or what ?? More info needed here, I guess... > > Otherwise this file is either called: > > * 'xxx.ref' > > * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the > > Windows version being emulated. The winever-specific file takes > > precedence over the '.ref' file, and the '.ref' file, which should > > exist, serves as a fallback. > > No, there should always be a single .ref file IMO. Version checks > should be done inside the test itself to make sure the output is > always the same. . . . > > * TEST_WINVER > > This contains the value of the '-winver' Wine argument, or the > > Windows version if the test is being run in Windows. We should mandate > > the use of specific values of winver so that test don't have to > > recognize all the synonyms of win2000 (nt2k...), etc. > > (Do we need to distinguish between Windows and Wine?) > > The test should use GetVersion() and friends IMO, no need for a > separate variable. Doh ! Right ! Like Andriy already said: the tests itself should reflect the entire behaviour of the functions, and even the version differences. One additional step in the direction of very simple, self-behaving tests... > > * and add two corresponding targets: 'make cui-tests' runs only those > > tests that do not pop up windows, and 'make gui-tests' runs only those > > tests that do pop up windows > > * 'make tests' would be 'tests: cui-tests gui-tests' > > I don't think this complexity is necessary. You can always redirect > the display if the windows annoy you. And tests should try to keep the > windows hidden as much as possible. Hmm, why complexity ? Is it really that difficult to implement ? I'd say it's a useful feature, and it doesn't incur much penalty, so the feature/penalty quotient is high enough ;-) Avoiding visible Windows as much as possible would be nice to have, though. Hmm, OTOH: maybe it'd be better to use make targets tests-unattended and tests-visual instead (note that I'm writing the names the other way around). This of course means that even many GUI tests would fall under the "unattended" category, thus annoying window popups would have to be minimized. OTOH we already kid of decided that we don't want to care about GUI testing right now, so maybe we should really just use one test target for now. Splitting later should be easy anyway. -- Andreas Mohr Stauferstr. 6, D-71272 Renningen, Germany Tel. +49 7159 800604 http://home.nexgo.de/andi.mohr/

Jan. 2, 2002

Re: We *really* need a development model change !

by Alexandre Julliard

Francois Gouget <fgouget(a)free.fr> writes: > I think what we need with this is a couple of guidelines and > documentation for potential test writers, and maybe a couple of > extensions. The following is half a proposed documentation that we could > put in the Wine Developper Guide, and half a proposed specification for > some possible extensions. As usual, comments and suggestions are > welcome. Great job! > * it should be easy to correlate with the source of the test. For > instance if a check fails, it would be a good idea to print a message > that can easily be grepped in the source code, or even the line number > for that check. But don't print line numbers for success messages, they > will change whenever someone changes the test and would require an > update to the reference files.. IMO the test should not be printing successes or failures at all. If it can determine whether some result is OK or not, it should simply do an assert on the result. Printing things should be only for the cases where checking for failure is too complicated, and so we need to rely on the output comparison to detect failures. > Otherwise this file is either called: > * 'xxx.ref' > * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the > Windows version being emulated. The winever-specific file takes > precedence over the '.ref' file, and the '.ref' file, which should > exist, serves as a fallback. No, there should always be a single .ref file IMO. Version checks should be done inside the test itself to make sure the output is always the same. > Each test should contain a section that looks something like: > > # @START_TEST_COVERAGE@ > # kernel32.CreateProcess > # kernel32.GetCommandLineA > # kernel32.GetCommandLineW > # __getmainargs > # __p__argc > # __p_argv > # __p_wargv > # @END_TEST_COVERAGE@ This is already part of the Perl framework, you have to explicitly declare the functions you use. So we don't want to duplicate the information. > * TEST_WINVER > This contains the value of the '-winver' Wine argument, or the > Windows version if the test is being run in Windows. We should mandate > the use of specific values of winver so that test don't have to > recognize all the synonyms of win2000 (nt2k...), etc. > (Do we need to distinguish between Windows and Wine?) The test should use GetVersion() and friends IMO, no need for a separate variable. > * and add two corresponding targets: 'make cui-tests' runs only those > tests that do not pop up windows, and 'make gui-tests' runs only those > tests that do pop up windows > * 'make tests' would be 'tests: cui-tests gui-tests' I don't think this complexity is necessary. You can always redirect the display if the windows annoy you. And tests should try to keep the windows hidden as much as possible. > What is needed most is a two sample tests: > * one simple console based test > * another one involving some GUI stuff I have attached two sample Perl scripts that were written some time ago by John Sturtz and myself. One is testing the atom functions and the other is creating a window. They should probably be simplified a bit to serve as documentation samples. -- Alexandre Julliard julliard(a)winehq.com

Jan. 2, 2002

Re: We *really* need a development model change !

by Andriy Palamarchuk

--- Francois Gouget <fgouget(a)free.fr> wrote: [...] > A test unit can also exercise more than one aspect of > one or more APIs. But, as a rule of thumb, a specific > test should not exercise more than a couple to a > handful related APIs (or up to a dozen in extreme > cases). Also, more than one test could exercise > different aspects a given API. I don't see a problem in having complex, task-oriented tests. Simple tests check one module, coplex tasks help to check integration between components. Both are needed. I especially like to create tests which implement long data conversion chains where output should be equal to input :-) I still want to have well-structured tests. > Wine tests can write their output in any form they > like. The only important criteria are: One more criteria - restore the system state (if possible). We don't want to have misconfigured machine after running the test suite. > To each test we associate a file containing the > reference output for > that test. Maintaining reference output file(s) is difficult because of: - EOLs conversions - keep all the files in sync - output files can be huge - test suite can have different output for different versions of Windows - one person can't run the suite on all Windows platforms, those who can, are afraid to touch fragile system of the output reference files. I suggest to use explicit checks and print descriptive messages in case of falure. I agree, this approach is more labour-intensive, especially for tests using IPC. It is also much more maintainable as soon as you coded it. Everything, including differences between different Windows platforms is documented directly in the code! This gives much better control. E.g, it is possible to comment-out part of the test, still getting meaningful results what worked, what did not. One more idea about reference output - it can be included in Perl script as in-line constant, so we can keep the output in the same file as the test itself. > Each test should contain a section that looks > something like: > > # @START_TEST_COVERAGE@ > # kernel32.CreateProcess > # kernel32.GetCommandLineA > # kernel32.GetCommandLineW > # __getmainargs > # __p__argc > # __p_argv > # __p_wargv > # @END_TEST_COVERAGE@ I prefer to have descriptive comments and failure messages. It will be difficult to keep the description you suggest in sync with the test implementation. > * TEST_WINVER > This contains the value of the '-winver' Wine > argument, or the > Windows version if the test is being run in Windows. > We should mandate the use of specific values of > winver so that test don't have to > recognize all the synonyms of win2000 (nt2k...), etc. > (Do we need to distinguish between Windows and > Wine?) Yes, we do. Sometimes we have implemented behavior only for one Windows version, e.g. NT. In this case switch -winver won't affect Wine behaviour. > * TEST_BATCH > If true (equal to 1) or unset, then the test > should assume that it is being run from within the > test framework and thus that it should be > non-interactive. If TEST_BATCH is set to 0, then the > test can assume that it is being run in interactive > mode, and thus ask questions to the > user. Of course most tests will simply behave > identically in both cases, I strongly recommend to use only batch version. If somebody has reasons to play with different forms of input, he can use any of following easy options: a) hardcode it - original test is improved ;-) b) take the test code to his interactive application c) add interaction with user to his local copy of the tests > Running tests > ------------- > In Wine: > > 'make tests' seems the best way to do things. > But some tests may need to create windows. For > instance I have a DIB test that creates a window, > draws a number of DIBs in it and checks the > bitmap bits of these DIBs and then exits. Thus it is > a non-interactive test. I am not really sure whether > the window actually needs to be made > visible or not, but even if this particular exemple > does not require it, > I suspect that othersm checking for message sequences > for instance, may need to make the window visible. I have tests which show window to use window messaging. Idea about tests separation to cui and gui looks good. You can run cui tests while doing something else. > In Windows: > > Hmmm, not sure how that is done. Run 'winetest.exe'? Something like: perl run_all_tests.pl The unit test application we develop will be a test of Win32 API in general, useful for all implementations. I'm for keeping this test in separate directory tree, not mixing it with Wine files. Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com

Jan. 2, 2002

Re: We *really* need a development model change !

by Andriy Palamarchuk

--- Francois Gouget <fgouget(a)free.fr> wrote: [...] > What is a test I wonder if I'm the only one who favours using existing testing framework? Why to create something new if you have not reviewed existing options? Perl has big choice of tools. In previous messages I reported about choices for C. Are you afraid that it will be difficult to learn new API? We impose some conventions ourselves. All the frameworks I saw provide very simple API. Examples of usages of different frameworks: Perl module Test::Simple: ok( 1 == 1, '1 == 1' ); ok( 2 == 2, '2 == 2' ); Perl module Test::Unit: sub test_ok_1 { assert(1 == 1); assert(2 == 2); } C framewok Check: fail_unless(1 == 1, "1==1"); fail_unless(2 == 2, "2==2"); Sure, there is more code to structure the test suites and glue them together, but API is very simple and can be easy guessed from examples. I don't see developer spending more than a few minutes to learn the framework basics. Advantages we get using existing framework: 1) existing services of the framewok can be used. Some of the services which I'm interested in: - TODO tests (by default are not reported), SKIPPED tests (test is not executed for some conditions) - Test::Simple Perl module - powerful reporting capabilities - test code structuring (init, run, teardown, tests hierarchy) - individual tests application address space protection - Check C framework 2) the implementation of the API can be extended as we like without changing the API. We can use help of the framework developers. Conformance to the API is maintained by compilation process. The conventions you suggested can be changed only with changing the tests and can't be easy checked. Andriy Palamarchuk __________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com

Jan. 2, 2002

[PATCH] Async IO on regular files

by Martin Wilck

This patch implements asynchronous I/O on regular file objects. It is against the current (02/01/02) CVS version of wine. I know that async IO on regular files is pretty pointless at the moment, but it's better to have it working as expected, I guess ... The patch was successfully tested with a simple test program that checks the sequence of scheduled reqeuests and file positioning via the lpOverlapped->Offset and OffsetHigh fields. server/file.c: struct file: add two async_queue fields (read/write) for async IO. create_file_for_fd(): Intialize async queues. destroy_file(): destroy async queues. file_poll_event(): invoke async requests for overlapped files. file_queue_async(): implement the queue_async() method. file_get_info(): return FD_TYPE_OVERLAPPED for overlapped files. files/file.c: FILE_AsyncReadService(): Use pread() for fd's that support seeking. FILE_AsyncWriteService(): Use pwrite() for fd's that support seeking. Regards Martin -- Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:Martin.Wilck(a)Fujitsu-Siemens.com D-33106 Paderborn http://www.fujitsu-siemens.com/primergy

Jan. 2, 2002

Re: We *really* need a development model change !

by Andriy Palamarchuk

--- Alexandre Julliard <julliard(a)winehq.com> wrote: > In theory tests should be written under Windows yes. > In practice the > initial version of a test may be done on Windows, > but I'm sure people > will then modify the test under Wine without > bothering to retry under > Windows every time. What is the point of having Win32 API unit test which does not confirm to the API reference implementation?! I agree that some of developers don't have Windows installed, don't have time to recompile the tests under Windows, etc, but Win32 confirmance is purpose of the whole project. No big qualification is required to fix confirmance problems and a few volunteers with access to different Windows platforms can perform this task. Andriy __________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com

Jan. 2, 2002