Improving the regression testing infrastructure
I know regression testing isn't as interesting as doing real development, and so I appologize with once again spamming the list with regression questions... As I've been writing tests (only for the last week or so), I realized that as we get a lot more tests, it will be very difficult to keep track of what is being tested, and what is not. I'd like to propose a list of which functions for each DLL are being tested, which file tests it, as well as any comments on the current tests. Currently, there is no great way to do this. I may use several functions in a directed test which I am not actually testing (for instance I've found it necessary to retrieve the page-size using GetSystemInfo, but my tests have nothing to do with this function, and so writing a directed test for it is a task for another day). Thus just grepping for which functions are used in a test is not sufficient. Also, I have found that some functions, or even specific task functions are very difficult to test, and so I've been leaving these out so that perhaps people ore skilled than myself could take a crack at them late (A good example is testing inheritance with CreateThread. It is a task that is beyond my capabilities at the moment, at least until I can find a good way to test CreateProcess...this would be much easier if Windows had something equivalent to 'fork'). In any case, I've tried to be thorough about commenting what I do and don't test, but it would be a lot more convenient if this information was more easily parsed than trying to find my comments intersperesed through the tests. So does anyone thing that creating a easily parsable list with the state of the current tests is this a reasonable thing to do, and would creating a simple file in each /tests directory with the information be good enough? If so, I can add one to my next test, and update it with what I've done so far. In general, while Francois' presentation is a good place to get started, I think it'd be a lot easier if there was a document off of winehq with recommendations on how to build, test, and document tests. Lowering the difficulty threshold, is more likely to draw more people to do so. Thanks, .Geoff
On Fri, Mar 29, 2002 at 11:21:23AM -0000, Geoffrey Hausheer wrote:
I know regression testing isn't as interesting as doing real development, and so I appologize with once again spamming the list with regression questions... NP. I'm sure it helps :)
[...]
So does anyone thing that creating a easily parsable list with the state of the current tests is this a reasonable thing to do, and would creating a simple file in each /tests directory with the information be good enough? If so, I can add one to my next test, and update it with what I've done so far. I don't think you want a separate list for each test. Hmm, or OTOH, if you want that, then make sure to have a unique file name, e.g. mytest.testlist. Then you could simply do find . -name "*.testlist"|xargs grep "TESTS: MyFunc" to search *all* list files check whether a particular function is being tested already.
In general, while Francois' presentation is a good place to get started, I think it'd be a lot easier if there was a document off of winehq with recommendations on how to build, test, and document tests. Lowering the difficulty threshold, is more likely to draw more people to do so. Definitely.
Creating such a document is on my ToDo list, but let me alter winebootup first to make it submittable ;-) -- Andreas Mohr Stauferstr. 6, D-71272 Renningen, Germany Tel. +49 7159 800604 http://home.arcor.de/andi.mohr/
On Fri, 29 Mar 2002, Eric Pouech wrote:
at least until I can find a good way to test CreateProcess... a bit out of topic (from the question), but I have an embryonic CreateProcess test, so don't bother with that (and there is a lot to test here)
This is interesting. How do you test it? AFAICS, The problem with the current infrastructure is that you cannot have a second executable to start. So you have to start yourself, and then convince the test framework to let you do your thing rather than just running another test. You cannot just write a perl/bash script either as this would not work on Windows. If you put submit a CreateProcess test I would be happy to add checks for the quote / command line parsing. -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ The greatest programming project of all took six days; on the seventh day the programmer rested. We've been trying to debug the *&^%$#@ thing ever since. Moral: design before you implement.
Francois Gouget a écrit :
On Fri, 29 Mar 2002, Eric Pouech wrote:
at least until I can find a good way to test CreateProcess... a bit out of topic (from the question), but I have an embryonic CreateProcess test, so don't bother with that (and there is a lot to test here)
This is interesting. How do you test it? AFAICS, The problem with the current infrastructure is that you cannot have a second executable to start. So you have to start yourself, and then convince the test framework to let you do your thing rather than just running another test. You cannot just write a perl/bash script either as this would not work on Windows. If you put submit a CreateProcess test I would be happy to add checks for the quote / command line parsing.
as of today, the test restarts itself, with a bit of a different command line it's written in C, but didn't test it on windows yet (I know a few issues already) the global test goes as : 1/ the parent creates the child 2/ the parent waits for the child to terminate 3/ the child process stores most information it can gather in a temporary file 4/ the child terminates 5/ the parent does the final checking I did choose the temporary file (instead of registry for example) to be as independant as possible of other APIs (even if the format of the file is a .INI one, and as of today, the parent reads it with private profile API) one of the checking that won't be possible is all the path lookup (except if we decide to copy the exec in different places :-( I'll put something out when I have the test working in Windows A+
On Fri, 29 Mar 2002, Eric Pouech wrote: [...]
the global test goes as : 1/ the parent creates the child 2/ the parent waits for the child to terminate 3/ the child process stores most information it can gather in a temporary file 4/ the child terminates 5/ the parent does the final checking
For the parameter testing I was thinking of storing the expected argv parameters in environment variables. Then, we need a way to tell the child to do specific tests (maybe one more environment variable) and it would then compare the contents of the environment variables with what it got. But since we do not control the main/WinMain, we would have to rely on getmainargs or __argv, __argp tricks (which means parts of the test may only be possible in msvcrt). The approach above would probably not work testing environment variable passing. So alternately, since the two share the same code, you could instruct the child to expect to receive the data corresponding to global array 5. Could one of the approaches above replace the temporary file approach? -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ War doesn't determine who's right. War determines who's left.
Francois Gouget a écrit :
On Fri, 29 Mar 2002, Eric Pouech wrote: For the parameter testing I was thinking of storing the expected argv parameters in environment variables. Then, we need a way to tell the child to do specific tests (maybe one more environment variable) and it would then compare the contents of the environment variables with what it got.
But since we do not control the main/WinMain, we would have to rely on getmainargs or __argv, __argp tricks (which means parts of the test may only be possible in msvcrt).
I don't follow you... we got access to main (what is missing is just letting the test procs access argc/argv pair
The approach above would probably not work testing environment variable passing. So alternately, since the two share the same code, you could instruct the child to expect to receive the data corresponding to global array 5.
anyway, since the test program is a CUI, the proposed scheme doesn't test either the WinMain part only the argc/argv part of main (the rest of the parsing is either in msvcrt or shell32 (for the unicode part)) I decided to let the parent do the checking because: - you don't know what the stdout of the child will be (and if you'll be able to get the results on the console) - make process depends on result of the parent for success/failure anyway, any test would be compare data (or behavior) from parent and child which means that you need a way to pass data from one to the other for the reason explained above, I think it's better to let the parent do the final testing. anyway, it may show that the child being only the same process as the parent with different arguments fails for: - CUI and GUI testing - independance of crt0 as I said, I choose the temp file because it's independent of any of the other bits (I mainly use the unix C lib) other ways would include: - env variables (but you may have limitations on the length of the var...) - registry (that would be, in term of coding, much more cleaner, but would require handle inheritance for testing the process creation...) - shared mem... so, a mechanism used by two processes. so I'd rather test the process creation first, and then the IPC mechanisms A+
On Fri, 29 Mar 2002, Eric Pouech wrote:
Francois Gouget a écrit :
On Fri, 29 Mar 2002, Eric Pouech wrote: For the parameter testing I was thinking of storing the expected argv parameters in environment variables. Then, we need a way to tell the child to do specific tests (maybe one more environment variable) and it would then compare the contents of the environment variables with what it got.
But since we do not control the main/WinMain, we would have to rely on getmainargs or __argv, __argp tricks (which means parts of the test may only be possible in msvcrt).
I don't follow you... we got access to main (what is missing is just letting the test procs access argc/argv pair
True. So what we need is to add a function in the testing framework so that tests can get argc/argv/envp. Seems like a reasonable approach.
I decided to let the parent do the checking because: - you don't know what the stdout of the child will be (and if you'll be able to get the results on the console)
It should have the same stdout as the parent, unless you explicitely nuke them when calling CreateProcess.
- make process depends on result of the parent for success/failure
Yes, the child should have an exit code that indicates success and another one that indicates failure. The parent checks that and returns success or failure as appropriate. -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ Linux, WinNT, MS-DOS - also known as the Good, the Bad and the Ugly.
True. So what we need is to add a function in the testing framework so that tests can get argc/argv/envp. Seems like a reasonable approach. I was more thinking adding some wt_argc and wt_argv, with init in main so that we don't have to deal with the portability issues of the __argc, __ ARGC...
I decided to let the parent do the checking because: - you don't know what the stdout of the child will be (and if you'll be able to get the results on the console)
It should have the same stdout as the parent, unless you explicitely nuke them when calling CreateProcess. yup... but we want to test CreateProcess, and some test cases will change the stdout handle
- make process depends on result of the parent for success/failure
Yes, the child should have an exit code that indicates success and another one that indicates failure. The parent checks that and returns success or failure as appropriate. either way
On Fri, 29 Mar 2002, Eric Pouech wrote:
True. So what we need is to add a function in the testing framework so that tests can get argc/argv/envp. Seems like a reasonable approach. I was more thinking adding some wt_argc and wt_argv, with init in main so that we don't have to deal with the portability issues of the __argc, __ ARGC...
Yes I agree with the wt_argc, etc approach. Either variables or functions, or even a single function: int wt_getmainargs(char*** argv, char*** envp); These are all pretty much equivalent.
I decided to let the parent do the checking because: - you don't know what the stdout of the child will be (and if you'll be able to get the results on the console)
It should have the same stdout as the parent, unless you explicitely nuke them when calling CreateProcess. yup... but we want to test CreateProcess, and some test cases will change the stdout handle
But do you need to write to stdout for these tests. Do you need to write to stdout at all for that matter? -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ Dieu dit: "M-x Lumière". Et la lumière fut.
yup... but we want to test CreateProcess, and some test cases will change the stdout handle
But do you need to write to stdout for these tests. Do you need to write to stdout at all for that matter? that has to be defined. But how can you test the child gets the correct stdout ? either the child knows (by other ways) what stdout really is (a file...) and can test it, either it doesn't care to know, and let the parent check what the child just wrote. Again, stdout has not to be used as a generic way to pass all information back to parent, just the stdout inheritance.
A+
On 29 Mar 2002, Geoffrey Hausheer wrote: [...]
In general, while Francois' presentation is a good place to get started, I think it'd be a lot easier if there was a document off of winehq with recommendations on how to build, test, and document tests. Lowering the difficulty threshold, is more likely to draw more people to do so.
I agree. Documenting and finishing the test framework is on the list for Wine 0.9.0. I added the corresponding tasks yesterday: * 526 [NEW] - Document the regression testing framework http://wine.codeweavers.com/bugzilla/show_bug.cgi?id=526 * 530 [NEW] - Package the regression tests for Windows http://wine.codeweavers.com/bugzilla/show_bug.cgi?id=530 * 527 [NEW] - Running C regression tests on Windows with MSVC http://wine.codeweavers.com/bugzilla/show_bug.cgi?id=527 * 528 [NEW] - Running C regression tests on Windows with Cygwin/MinGW http://wine.codeweavers.com/bugzilla/show_bug.cgi?id=528 * 529 [NEW] - Running Perl regression tests on Windows http://wine.codeweavers.com/bugzilla/show_bug.cgi?id=529 -- Francois Gouget fgouget(a)free.fr http://fgouget.free.fr/ "Utilisateur" (nom commun) : Mot utilisé par les informaticiens en lieu et place d'"idiot".
participants (4)
-
Andreas Mohr -
Eric Pouech -
Francois Gouget -
Geoffrey Hausheer