Alexandre Julliard wrote:
Andreas Mohr andi@rhlx01.fht-esslingen.de writes:
Please comment on both my intended posting and the
way I programmed the first
version of the test suite (I'm not extremely happy
with the current program;
if you have any improvements, then get them here
ASAP !).
Look at programs/winetest, that's the tool we should
use to write
tests IMO.
Just looked at the tool. It only provides gateway from Perl to wine API functions, right?
Advantages I see in using script-based tests: 1) tests can be used without recompilation 2) Perl is familiar to many developers 3) programming in scripting language is usually easier than in C 4) Perl is widely used for testing purposes and I expect to find many testing tools in Perl.
Cons: 1) Many current Wine contributors don't know Perl 2) one more level of abstraction does not give significant advantages in this application. On the contrary, it is more difficult to locate cause of problems because developer has to go trough one more, often not familiar layer. Absense of strict typing in this layer will hurt a lot.
Advantages of using C-based tests: 1) compilation has to be used to run the tests. In some cases this is an advantage. Before running the tests you'd better be sure it at least compiles in both environments. 2) C is the most familiar to developers language and the language itself is simpler than Perl 3) Documentation for Win32 API is C-oriented. Examples, given in documentation are in C or easy translated to C 4) Developers already have some testing code snippets in C
Summary: Requirements to the unit testing tool: 1) should help to quickly create tests 2) easy to use, should help to involve as many developers as possible. 3) may be useful for developers, using Wine for there projects
Because of the goals I'm more inclined to use C-based test suite. IMHO it is better suited for existing Wine developers audience and will provide us much bigger code pool. I'm even ready to have tests in both - Perl and C.
The big question is a tool to test GUI. I did not find any OS Windows GUI testing frameworks :-(
Comments, suggestions?
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
On Thu, Dec 27, 2001 at 07:38:24AM -0800, Andriy Palamarchuk wrote:
Because of the goals I'm more inclined to use C-based test suite. IMHO it is better suited for existing Wine developers audience and will provide us much bigger code pool. I'm even ready to have tests in both - Perl and C.
Yes, IMHO we really need a C based test. After all in order to leverage the Windows developer group`s skills, we probably really shouldn't ask them to code it up in perl...
The best way IMHO would be to try to make C and perl based test output compatible. That way people familiar with perl coding could code it in perl, and other people would have choice ;)
Hmm, wait. Also creating perl based tests would mean that you need to have perl installed on Windows, too. Maybe that is too much of a disadvantage...
The big question is a tool to test GUI. I did not find any OS Windows GUI testing frameworks :-(
I don`t think we should even bother right now. Many windows functions are not GUI based, and those who are could at least get tested against their functional behaviour.
On Friday 28 December 2001 05:00, Andreas Mohr wrote:
Yes, IMHO we really need a C based test.
I'd go along with this. It seems that a variety of tests, written and contributed by a variety of people, and thus written in a variety of mutually inconsistent and collectively odd ways, is inevitable. Rather than people wasting their valuable time on rejigging any existing "modular extensible test framework" or writing anything new, the simplist way forward would surely be to document some simple input/output behaviours test programs should use.
Eg. as long as it is described what should and should not be sent by a test program to stdout+stderr, and what input (if any) should be supported on the command line and/or stdin, then you have achieved the only consistency that matters. A simple shell script could then, for example, control the level of debugging in any/all such tests by simply piping output(s) through egrep or whatever - stdout itself could be piped to /dev/null if necessary leaving only stderr, etc.
Let's face it - anyone adding new functionality to wine or working on anything non-trivial will be cooking up their own "foo.c" test programs on the fly when developing - the less work it takes for that developer to convert their foo.c into a test-foo32-api.c for use by test scripts, the better off everyone will be.
Let's also not forget, if that "standardised form" for input/output of the testing is as logical and uncomplicated as possible, more and more well-meaning code grunts will be capable of making useful contributions - be it clunky GUI wrappers for the testing suites (think what "make xconfig" did for linux kernel rebuilding), or by even taking the various foo.c test fragments from the wine developers and coaxing them into the standardised form on their behalfs.
If a "testing framework" is anything more complicated - the only people working on the test suites will be whoever defines/understands the "test framework specification" and the hard-core wine developers themselves. In other words, we won't get far.
Cheers, Geoff
On Fri, Dec 28, 2001 at 12:49:54PM +1300, Geoff Thorpe wrote:
[voting for a simple test interface]
If a "testing framework" is anything more complicated - the only people working on the test suites will be whoever defines/understands the "test framework specification" and the hard-core wine developers themselves. In other words, we won't get far.
Exactly.
We've got so many (up to 12000) functions to write tests for that we really can't afford having a complicated test environment. Just about everybody should be able to write tests for a function. And if this is the case, then we really should try to leverage as much existing Windows developer knowledge/skillset as possible in order to have a working test suite in place in just about an instant.
Andriy Palamarchuk apa3a@yahoo.com writes:
Just looked at the tool. It only provides gateway from Perl to wine API functions, right?
Yes, though a lot of higher-level stuff could be added to it once people start using it.
The C vs. Perl question has been debated already, please check the archive. The truth is that a lot of people are willing to setup a testing infrastructure, but nobody is willing to write the actual tests.
So right now we have a minimal infrastructure in Perl, and I'm not going to setup another one until someone starts using the existing one seriously and demonstrates whether it works or not in practice. The theory has been discussed to death already.
--- Alexandre Julliard julliard@winehq.com wrote:
The truth is that a lot of people are willing to setup a testing infrastructure, but nobody is willing to write the actual tests.
Counterexamples: 1) I suspect that you try to run the code you develop before committing it to CVS ;-) You work on core Wine functionality and the test code snippets you use would be invaluable as unit tests. However I can't find these tests anywhere in CVS tree :-P
2) Besides Perl testing framework there are a few testing applications (vartest, guitest) in CVS, but they are not merged in a single test suite.
3) I tried to submit unit test application for SystemParametersInfo with my first patch to this function, but the unit test was not accepted. I assumed that no unit test is necessary in CVS tree for such really simple function.
Summary - there are people who wants to develop unit tests. IMO the problem with unit test in the project is in: 1) Our attitude. We don't have *any* policy about unit tests. Developers are not asked about unit tests, from the documents on WineHQ it is not clear whether we need them at all.
2) Absense of infrastracture. We need to develop or choose unit testing framework, define unit test policies, add information about unit tests to Wine documentation, keep unit tests visible for developers all the time.
So right now we have a minimal infrastructure in Perl
No, ability to call W32 API functions is not considered a unit test infrastracture. I may say Wine has such infrastructure for C since 1993 :-). From this point of view Andreas test application provides more support for unit tests than plain Perl module.
In existing CVS tree we have much more tests in C than in Perl (both - a little more than nothing)
I'm not going to setup another one until someone starts using the existing one seriously and demonstrates whether it works or not in practice. The theory has been discussed to death already.
I agree with you - only usage of the framework can help to choose better one. I'll look more closely at C-based, try to combine our Perl module with Perl-based unit frameworks.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
No, ability to call W32 API functions is not considered a unit test infrastracture. I may say Wine has such infrastructure for C since 1993 :-). From this point of view Andreas test application provides more support for unit tests than plain Perl module.
Not really, because you need a lot more infrastructure to build C tests than to simply run Perl scripts. For Perl all you need is the winetest app that is in the tree, plus a bit of Makefile glue which is pretty trivial to do.
In fact here's a 10-minute hack to add a make test target. With that all you have to do is create a test script in dlls/xxx/tests/foo.test, put the expected output in tests/foo.test.ref (presumably generated by running the test under Windows), add your script to the makefile and run make test.
Index: Make.rules.in =================================================================== RCS file: /opt/cvs-commit/wine/Make.rules.in,v retrieving revision 1.95 diff -u -r1.95 Make.rules.in --- Make.rules.in 2001/12/14 23:14:22 1.95 +++ Make.rules.in 2001/12/30 18:00:41 @@ -47,6 +47,7 @@ RM = rm -f MV = mv MKDIR = mkdir -p +DIFF = diff -u C2MAN = @C2MAN@ MANSPECS = -w $(TOPSRCDIR)/dlls/gdi/gdi32.spec \ -w $(TOPSRCDIR)/dlls/user/user32.spec \ @@ -58,6 +59,8 @@ ALLLINTFLAGS = $(LINTFLAGS) $(DEFS) $(OPTIONS) $(DIVINCL) WINAPI_CHECK = $(TOPSRCDIR)/tools/winapi_check/winapi_check WINEBUILD = $(TOPOBJDIR)/tools/winebuild/winebuild +WINETEST = $(TOPOBJDIR)/programs/winetest/winetest +RUNTEST = $(TOPOBJDIR)/programs/winetest/runtest MAKEDEP = $(TOPOBJDIR)/tools/makedep WRC = $(TOPOBJDIR)/tools/wrc/wrc WMC = $(TOPOBJDIR)/tools/wmc/wmc @@ -95,7 +98,7 @@
# Implicit rules
-.SUFFIXES: .mc .rc .mc.rc .res .spec .spec.c .glue.c +.SUFFIXES: .mc .rc .mc.rc .res .spec .spec.c .glue.c .test .test.out .test.ref
.c.o: $(CC) -c $(ALLCFLAGS) -o $@ $< @@ -121,6 +124,12 @@ .c.ln: $(LINT) -c $(ALLLINTFLAGS) $< || ( $(RM) $@ && exit 1 )
+.test.test.out: + $(RUNTEST) $(TOPOBJDIR) $< > $@ + +.test.out.test.ref: + $(DIFF) $< $@ && touch $@ + .PHONY: all install uninstall clean distclean depend dummy
# 'all' target first in case the enclosing Makefile didn't define any target @@ -216,7 +225,7 @@ -cd `dirname $@` && $(RM) $(CLEAN_FILES)
clean:: $(SUBDIRS:%=%/__clean__) $(EXTRASUBDIRS:%=%/__clean__) - $(RM) $(CLEAN_FILES) $(GEN_C_SRCS) $(GEN_ASM_SRCS) $(RC_SRCS:.rc=.res) $(RC_SRCS16:.rc=.res) $(MC_SRCS:.mc=.mc.rc) $(PROGRAMS) + $(RM) $(CLEAN_FILES) $(GEN_C_SRCS) $(GEN_ASM_SRCS) $(RC_SRCS:.rc=.res) $(RC_SRCS16:.rc=.res) $(MC_SRCS:.mc=.mc.rc) $(TESTS:%=tests/%.out) $(PROGRAMS)
# Rules for installing
@@ -225,6 +234,13 @@
$(SUBDIRS:%=%/__uninstall__): dummy cd `dirname $@` && $(MAKE) uninstall + +# Rules for testing + +test:: $(WINETEST) $(TESTS:%=tests/%.out) $(TESTS:%=tests/%.ref) + +$(WINETEST): + cd $(TOPOBJDIR)/programs/winetest && $(MAKE) winetest
# Misc. rules
Index: Makefile.in =================================================================== RCS file: /opt/cvs-commit/wine/Makefile.in,v retrieving revision 1.103 diff -u -r1.103 Makefile.in --- Makefile.in 2001/11/23 23:04:58 1.103 +++ Makefile.in 2001/12/30 18:00:41 @@ -132,6 +132,9 @@ @cd dlls && $(MAKE) checklink @cd debugger && $(MAKE) checklink
+test:: + @cd dlls && $(MAKE) test + TAGS etags: etags `find $(TOPSRCDIR) -name '*.[chS]' -print | grep -v dbgmain`
Index: dlls/Makedll.rules.in =================================================================== RCS file: /opt/cvs-commit/wine/dlls/Makedll.rules.in,v retrieving revision 1.16 diff -u -r1.16 Makedll.rules.in --- dlls/Makedll.rules.in 2001/09/17 20:09:08 1.16 +++ dlls/Makedll.rules.in 2001/12/30 18:01:00 @@ -36,6 +36,10 @@ checklink:: lib$(MODULE).$(LIBEXT) $(CC) -o checklink $(TOPSRCDIR)/library/checklink.c -L. -l$(MODULE) $(ALL_LIBS) && $(RM) checklink
+# Rules for testing + +$(TESTS:%=tests/%.out): lib$(MODULE).$(LIBEXT) + # Rules for debug channels
debug_channels: dummy Index: dlls/Makefile.in =================================================================== RCS file: /opt/cvs-commit/wine/dlls/Makefile.in,v retrieving revision 1.109 diff -u -r1.109 Makefile.in --- dlls/Makefile.in 2001/11/06 17:52:37 1.109 +++ dlls/Makefile.in 2001/12/30 18:01:02 @@ -764,6 +764,9 @@
# Misc rules
+$(SUBDIRS:%=%/__test__): dummy + @cd `dirname $@` && $(MAKE) test + $(SUBDIRS:%=%/__checklink__): dummy @cd `dirname $@` && $(MAKE) checklink
@@ -773,6 +776,8 @@ install:: $(SUBDIRS:%=%/__install__)
uninstall:: $(SUBDIRS:%=%/__uninstall__) + +test:: $(SUBDIRS:%=%/__test__)
checklink:: $(SUBDIRS:%=%/__checklink__)
--- /dev/null Fri Dec 7 20:45:56 2001 +++ programs/winetest/runtest Sat Dec 29 17:00:48 2001 @@ -0,0 +1,9 @@ +#!/bin/sh +topobjdir="$1" +shift +perldir="$topobjdir/programs/winetest" +LD_LIBRARY_PATH="$topobjdir/dlls:$topobjdir:$LD_LIBRARY_PATH" +export LD_LIBRARY_PATH +WINESERVER="$topobjdir/server/wineserver" +export WINESERVER +exec $perldir/winetest -debugmsg -all -- -I $perldir $@
In fact here's a 10-minute hack to add a make test target. With that all you have to do is create a test script in dlls/xxx/tests/foo.test, put the expected output in tests/foo.test.ref (presumably generated by running the test under Windows), add your script to the makefile and run make test.
I've started playing with this, Alexandre, and I had a thought/question: why not put the tests under 'wine/tests'? I recognize the benefit of having a test immediately associated with the implementation. But, I would argue a) that not all tests are going to be specific to one dll b) by placing all the tests together, you make exporting a 'test' package to Windows simpler. c) You centralize the info and allow for good doco d) We can create and sustain a whole Windows make file hierarchy, which would be useful to a test writer in Windows.
(And yes, I ask because I am threatening to actually do some of the work; of course, I'll run out of time shortly, and it will be an empty threat...<g>).
Thoughts?
Jer
Jeremy White jwhite@codeweavers.com writes:
I've started playing with this, Alexandre, and I had a thought/question: why not put the tests under 'wine/tests'? I recognize the benefit of having a test immediately associated with the implementation. But, I would argue a) that not all tests are going to be specific to one dll
It seems to me that a given test should always be specific not only to a dll, but to a single or a few functions of this dll. When do you think this would not be the case?
This really goes with the dll separation strategy: Wine should no longer be viewed as a monolithic program, but more as a set of dll packages grouped in the same tar file. And at some point it could become desirable to split some dlls out of the main tree, or to have separate people maintain separate dlls independently. So I think the unit tests should be part of their respective dll.
b) by placing all the tests together, you make exporting a 'test' package to Windows simpler. c) You centralize the info and allow for good doco d) We can create and sustain a whole Windows make file hierarchy, which would be useful to a test writer in Windows.
I don't think we should maintain a Windows make hierarchy, at least not manually. If we have to ship Windows makefiles they should be generated from the Wine makefiles (or both types of makefile generated from some other source file). Asking people to keep two hierarchies in sync won't work.
I don't think we should maintain a Windows make hierarchy, at least not manually. If we have to ship Windows makefiles they should be generated from the Wine makefiles (or both types of makefile generated from some other source file). Asking people to keep two hierarchies in sync won't work.
I'm relatively neutral on the tests vs. dlls issue, and so I'm willing to defer to your judgement.
However, I think it's critical that this process somehow be set up to be trivial for use by a Windows geek. And requiring the Cygwin tool chain on Windows defeats the whole purpose. For example, I here at home have nothing but a totally brain dead Win98 partition. No compilers, nothing. (Okay, Diablo II, but that's it).
For me, at a minimum, I need to have a precompiled winetest.exe.
Ideally, we would have a 'winetest.zip' such that all I would have to do is install Perl, and then I'd have a nice set of sample test scripts I could run/modify/tweak to my hearts satisfaction. If I had a C compiler, I could also compile the C tests.
Hmm. What if I had a 'make export-tests' that created a template 'winetest.zip' file. Then I've just got to get a Windows winetest.exe file built and repackage the 'winetest.zip' file.
So, if we had *one* Windows machine with a full Cygwin/CVS/gmake toolchain, it could periodically build new 'winetest.zip' files and publish them as a separate download at winehq.com.
What do you think? If I extended your patch to add an export-tests target, would this be useful?
Jer
Jeremy White jwhite@codeweavers.com writes:
Ideally, we would have a 'winetest.zip' such that all I would have to do is install Perl, and then I'd have a nice set of sample test scripts I could run/modify/tweak to my hearts satisfaction.
Exactly, yes. If possible winetest.zip should also include perl.dll and the needed packages so that you don't even need to install perl at all. I think it may be a good idea to have a winetest.zip that contains winetest.exe and related files that you only have to install once, and another zip that contains the test themselves which you may need to upgrade more frequently (or that you could get directly from CVS if you have a CVS client on Windows).
Hmm. What if I had a 'make export-tests' that created a template 'winetest.zip' file. Then I've just got to get a Windows winetest.exe file built and repackage the 'winetest.zip' file. So, if we had *one* Windows machine with a full Cygwin/CVS/gmake toolchain, it could periodically build new 'winetest.zip' files and publish them as a separate download at winehq.com.
What do you think? If I extended your patch to add an export-tests target, would this be useful?
Sure, but I don't think a makefile target is the best way to do it. It would be better to simply have a script that could be run on WineHQ every night to rebuild the zip directly from CVS.
On 1 Jan 2002, Alexandre Julliard wrote:
Jeremy White jwhite@codeweavers.com writes:
I've started playing with this, Alexandre, and I had a thought/question: why not put the tests under 'wine/tests'? I recognize the benefit of having a test immediately associated with the implementation. But, I would argue a) that not all tests are going to be specific to one dll
It seems to me that a given test should always be specific not only to a dll, but to a single or a few functions of this dll. When do you think this would not be the case?
I can think of one case that I burn to put into the Wine testing framework: the command line/argv/argv handling. I think it would make sense to test simultaneously: * kernel32.CreateProcess It is clearly involved when the parent proces is a Wine process * kernel32.GetCommandLineA/W Clearly related * main and WinMain Not from a specific dll * msvcrt.__getmainargs One of the ways to retrieve the parameters * msvcrt.__argc, msvcrt.__argv, msvcrt.__wargv Other functions returning the process's parameters * shell32.CommandLineToArgvW Not involved int the process creation but performs exactly the same command line to argument array conversion that is involved in there. So I think it makes sense to test it in the same program.
But I believe that even that case can fit reasonably well in the test architecture. For instance we could to these tests in msvcrt/test since msvcrt depends on kernel32. Then it's just a matter of CommandLineToArgvW that we may want to separate.
[...]
b) by placing all the tests together, you make exporting a 'test' package to Windows simpler. c) You centralize the info and allow for good doco d) We can create and sustain a whole Windows make file hierarchy, which would be useful to a test writer in Windows.
I think the real rational for putting tests in a separate directory would be to make them completely separate from Wine. After all a Win32 conformance test suite could also benefit other projects like Odin, ReactOS?, and any other Win32 reimplementation. Thus there may be an argument that it could be beneficial to have a test suite that everyone could contribute to. It could even be under a different license, like GPL to ensure that all projects contribute to it fairly (and who wants a proprietary Win32 conformance suite anyway). Such a strategy would even mke more sense if there was already such a test suite out there.
But I'm not aware of any such test suite so I say we should start with an integrated test suite which seems more practical at the moment and we'll see when the time comes if we need to separate it from Wine.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ The nice thing about meditation is that it makes doing nothing quite respectable -- Paul Dean
On Wed, Jan 02, 2002 at 02:34:56AM -0800, Francois Gouget wrote:
I can think of one case that I burn to put into the Wine testing framework: the command line/argv/argv handling. I think it would make sense to test simultaneously:
Yes, yes, and again: yes !
Wine's cmdline handling is still *very* buggy ! (which is not too astonishing, given these tons of quirks in the Win32 API) I've had several programs already which didn't work due to cmdline problems. This is a prime target of API testing. Another prime target would be file system handling (I know of several functions that are still "broken").
On 30 Dec 2001, Alexandre Julliard wrote: [...]
In fact here's a 10-minute hack to add a make test target. With that all you have to do is create a test script in dlls/xxx/tests/foo.test, put the expected output in tests/foo.test.ref (presumably generated by running the test under Windows), add your script to the makefile and run make test.
I think what we need with this is a couple of guidelines and documentation for potential test writers, and maybe a couple of extensions. The following is half a proposed documentation that we could put in the Wine Developper Guide, and half a proposed specification for some possible extensions. As usual, comments and suggestions are welcome.
What is a test --------------
A test unit is an executable or script. You can name it anyway you like (please no space in the names, it's always annoying). All test units should be non-interactive. A test unit called xxx generates two outputs: * its exit code * text output on either or both of stdout and stderr, both of which are normally redirected to a file called 'xxx.out'.
A test succeeds if: * its exit code is 0 * and its output, 'xxx.out' matches the reference output according to the rules described later.
Reciprocally it fails if: * its exit code is non zero Either because one aspect of the test failed and thus the test unit decided to return a non-zero code to indicate failure, or because it crashed and thus the parent got a >= 128 error code. * or because its output differs from the reference output established on Windows
Under this model each test unit may actually be comprised of more than one process (for instance to test CreateProcess, inter-process messaging, inter-process DDE, etc.). All that counts is that the original process does not finish until the testing is complete so that the testing framework knows when to check the test output and move on. (There is no provision for hung tests. A time out based mechanism, with a large time out, like 5 minutes, could do the trick.)
A test unit can also exercise more than one aspect of one or more APIs. But, as a rule of thumb, a specific test should not exercise more than a couple to a handful related APIs (or up to a dozen in extreme cases). Also, more than one test could exercise different aspects a given API. So when running the Wine regression tests, if we find that 3 tests out of 50 failed, it means that three processes had an incorrect exit code or output out of fifty. One should then analyze in more details what went wrong during the execution of these processes to determine which specific API or aspect of an API misbehaved. This can be done either by looking at their output, by running them again with Wine traces, or even by running them in a debugger.
Test Output -----------
Wine tests can write their output in any form they like. The only important criteria are: * it should be reproducible from one run to the next: don't print pointer values. They are most likely to change in the next run and thus cannot be checked * it should be the same on a wide range of systems: don't print things like the screen resolution! * it should be easy to correlate with the source of the test. For instance if a check fails, it would be a good idea to print a message that can easily be grepped in the source code, or even the line number for that check. But don't print line numbers for success messages, they will change whenever someone changes the test and would require an update to the reference files.. * the output should not be empty (just in case the process may die with a 0 return code / fail to start before writing anything to the output) * finally it should be easy to read by the people who are going to be debugging the test when something goes wrong.
To each test we associate a file containing the reference output for that test. If the test's output consists of a single "Test Ok", then that file may be ommitted. (I am not sure if this shortcut is actually very needed/useful)
Otherwise this file is either called: * 'xxx.ref' * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
This second feature is probably best avoided as much as possible as multiple reference files are harder to maintain than a single reference file. But they maybe be useful for some APIs (can anyone think of any?). In any case I propose not to implement it until we actually find the need for it.
One may also create a file called 'xxx.ref.diff' (resp. 'xxx.win95.diff', etc.) which contains a diff between the test output in Windows and the test output in Wine. The goal is to: * make it unnecessary to tweak tests to not report known Wine shortcomings/bugs, or to remove these tests altogether * but not have a hundreds of tests that systematically fail due to these shortcomings either (I can think of at least one case related to command line passing). Because then you would not know if more aspects of these tests fail than usual or not.
The criteria to determine success/failure of a test unit xxx then becomes: xxx >xxx.out 2>&1 if the return code is != 0 then the test failed diff -u xxx.ref xxx.out >xxx.diff if there is no xxx.ref.diff && xxx.diff is not empty then the test failed if xxx.diff is different from xxx.ref.diff then the test failed otherwise the test is successful
The test framework can then return three numbers: * the number of failed tests * the number of tests with known bugs * the total number of tests
(and/or various other differences between these numbers)
Test coverage -------------
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
The goal of this section is to identify which APIs are being tested by a given test unit. Each API is specified as 'dll.api'. If the dll name is omitted, then the API is assumed to be in the dll in which the test unit is located (in the above example that would be msvcrt).
Note that we cannot simply extract the list of APIs being called by a given test. For instance most tests are likely to call APIs like printf. And yet, printf should not be recorded as being tested by a thousand tests.
We can then write a script that uses this information to: * list the tests that cover a given API * build a list of the APIs that have no associated tests * build all sorts of fancy and aybe useful statistics
(the above section would work just as well inside C-style comments, one way to handle this is to ignore leading non-alphanumeric characters)
Test environment ----------------
A test may need to know things about its environment although hopefully this will be relatively rare. So I propose to store some information in environment variables as this seems the least intrusive way to provide them with information:
* TEST_OUT The name of the test output file. This may be useful if a test needs to create new processes and to redirect their output to temporary files. If the child processes need to output infomation to the test output, then they can use this environment variable to open the rght file.
* TEST_WINVER This contains the value of the '-winver' Wine argument, or the Windows version if the test is being run in Windows. We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...), etc. (Do we need to distinguish between Windows and Wine?)
* TEST_BATCH If true (equal to 1) or unset, then the test should assume that it is being run from within the test framework and thus that it should be non-interactive. If TEST_BATCH is set to 0, then the test can assume that it is being run in interactive mode, and thus ask questions to the user. Of course most tests will simply behave identically in both cases, but in some cases an interactive mode may be useful. For instance the test concerning CommandLineToArgvW could have an interactive mode where it asks the user for a command line and prints the corresponding argc/argv. This would allow a developper to manually check how the API behaves for specific command lines.
I thought about passing these arguments on the command line but: * it seems less practical, especially since 'main' would have to parse it and store is somewhere * it may interfer with some tests (command line related tests, although only child processes should care about that) * it seems less expandable and flexible
Running tests -------------
In Wine:
'make tests' seems the best way to do things. But some tests may need to create windows. For instance I have a DIB test that creates a window, draws a number of DIBs in it and checks the bitmap bits of these DIBs and then exits. Thus it is a non-interactive test. I am not really sure whether the window actually needs to be made visible or not, but even if this particular exemple does not require it, I suspect that othersm checking for message sequences for instance, may need to make the window visible. And if the Wine test suite contains many such tests, then there will be windows popping up and down all the time and it would make it impossible to use the computer while the tests are running. So we may: * have to test lists in the Makefiles: cui-tests (those that don't pop up windows) and gui-tests (those that do) * and add two corresponding targets: 'make cui-tests' runs only those tests that do not pop up windows, and 'make gui-tests' runs only those tests that do pop up windows * 'make tests' would be 'tests: cui-tests gui-tests'
Of course, it should be noted that one way to deal with tests that pop up windows is to run them inside a VNC X server, a second X server or some other similar X trick.
In Windows:
Hmmm, not sure how that is done. Run 'winetest.exe'?
Writing tests -------------
This is where we describe which APIs are available to a test writer... if any. I believe that very little functionality is necessary/useful.
* test_failed(message) Sets a global variable to 1 to indicate failure and printf the specified message
* test_result() Returns 0 if the 'test_failed' was never called, and 1 otherwise.
* get_test_out() Returns the contents of $TEST_OUT
* get_test_batch() Returns true if the test is being run in batch mode and false owtherwise. Of course this is based on $TEST_BATCH
* get_test_winver() Returns the current Windows version, or the version being emulated if in Wine. This coould simply be based on $TEST_WINVER.
That's about all. As you can see this is pretty basic stuff and I am not sure more is needed. But if you have ideas...
What is needed most is a two sample tests: * one simple console based test * another one involving some GUI stuff
Then the documentation could use them as examples and discuss the interesting aspects.
I believe that all the above is fairly neutral as far as perl vs. C is concerned. Except for the compilation issues, and maybe the exact command to use to invoke a test, whether a test is written in C or in perl should not make any difference.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ 1 + e ^ ( i * pi ) = 0
On Wed, Jan 02, 2002 at 04:36:14AM -0800, Francois Gouget wrote:
I think what we need with this is a couple of guidelines and documentation for potential test writers, and maybe a couple of extensions. The following is half a proposed documentation that we could put in the Wine Developper Guide, and half a proposed specification for some possible extensions. As usual, comments and suggestions are welcome.
Good !
Well, I've read the whole damn thing, and I don't have many comments/flames/whatever to make (damn ! :)
The criteria to determine success/failure of a test unit xxx then becomes: xxx >xxx.out 2>&1 if the return code is != 0 then the test failed diff -u xxx.ref xxx.out >xxx.diff if there is no xxx.ref.diff && xxx.diff is not empty then the test failed if xxx.diff is different from xxx.ref.diff then the test failed otherwise the test is successful
Wow, someone must have been very bored ;)
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
No ! We need Win32 GUI, Win32 console and Win16.
Then the documentation could use them as examples and discuss the interesting aspects.
Yep ! (in a very simple way...)
I believe that all the above is fairly neutral as far as perl vs. C is concerned. Except for the compilation issues, and maybe the exact command to use to invoke a test, whether a test is written in C or in perl should not make any difference.
Great job ! (unfortunately I don't have much to add :-)
I'm damn sure some issues will still arise, but we can only find problems if we go ahead and implement it.
On Wed, 2 Jan 2002, Andreas Mohr wrote: [...]
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
No ! We need Win32 GUI, Win32 console and Win16.
Why is it necessary to separate Win16 from the rest? On Windows it could make sense: obviously you cannot run the 32bit tests on Windows 3.x. But in Wine if you are running the Win32 GUI tests, why not run the Win16 GUI tests? Is a 'Win16 console' test something that can exist? Would that be a DOS program? In any case couldn't they be run with all the Win32 console tests?
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ Linux: It is now safe to turn on your computer.
On Wed, Jan 02, 2002 at 02:46:50PM -0800, Francois Gouget wrote:
On Wed, 2 Jan 2002, Andreas Mohr wrote: [...]
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
No ! We need Win32 GUI, Win32 console and Win16.
Why is it necessary to separate Win16 from the rest? On Windows it could make sense: obviously you cannot run the 32bit tests on Windows 3.x. But in Wine if you are running the Win32 GUI tests, why not run the Win16 GUI tests? Is a 'Win16 console' test something that can exist? Would that be a DOS program? In any case couldn't they be run with all the Win32 console tests?
?? :)
We need to have a Win16 framework. Win16 is completely *different*, that's why. (thunking, different programming, ...)
--- Francois Gouget fgouget@free.fr wrote: [...]
What is a test
I wonder if I'm the only one who favours using existing testing framework? Why to create something new if you have not reviewed existing options?
Perl has big choice of tools. In previous messages I reported about choices for C.
Are you afraid that it will be difficult to learn new API? We impose some conventions ourselves. All the frameworks I saw provide very simple API.
Examples of usages of different frameworks:
Perl module Test::Simple: ok( 1 == 1, '1 == 1' ); ok( 2 == 2, '2 == 2' );
Perl module Test::Unit: sub test_ok_1 { assert(1 == 1); assert(2 == 2); }
C framewok Check: fail_unless(1 == 1, "1==1"); fail_unless(2 == 2, "2==2");
Sure, there is more code to structure the test suites and glue them together, but API is very simple and can be easy guessed from examples. I don't see developer spending more than a few minutes to learn the framework basics.
Advantages we get using existing framework: 1) existing services of the framewok can be used. Some of the services which I'm interested in: - TODO tests (by default are not reported), SKIPPED tests (test is not executed for some conditions) - Test::Simple Perl module - powerful reporting capabilities - test code structuring (init, run, teardown, tests hierarchy) - individual tests application address space protection - Check C framework
2) the implementation of the API can be extended as we like without changing the API. We can use help of the framework developers. Conformance to the API is maintained by compilation process. The conventions you suggested can be changed only with changing the tests and can't be easy checked.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
--- Francois Gouget fgouget@free.fr wrote: [...]
A test unit can also exercise more than one aspect
of
one or more APIs. But, as a rule of thumb, a
specific
test should not exercise more than a couple to a handful related APIs (or up to a dozen in extreme cases). Also, more than one test could exercise different aspects a given API.
I don't see a problem in having complex, task-oriented tests. Simple tests check one module, coplex tasks help to check integration between components. Both are needed. I especially like to create tests which implement long data conversion chains where output should be equal to input :-) I still want to have well-structured tests.
Wine tests can write their output in any form they like. The only important criteria are:
One more criteria - restore the system state (if possible). We don't want to have misconfigured machine after running the test suite.
To each test we associate a file containing the reference output for that test.
Maintaining reference output file(s) is difficult because of: - EOLs conversions - keep all the files in sync - output files can be huge - test suite can have different output for different versions of Windows - one person can't run the suite on all Windows platforms, those who can, are afraid to touch fragile system of the output reference files.
I suggest to use explicit checks and print descriptive messages in case of falure. I agree, this approach is more labour-intensive, especially for tests using IPC. It is also much more maintainable as soon as you coded it. Everything, including differences between different Windows platforms is documented directly in the code! This gives much better control. E.g, it is possible to comment-out part of the test, still getting meaningful results what worked, what did not.
One more idea about reference output - it can be included in Perl script as in-line constant, so we can keep the output in the same file as the test itself.
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
I prefer to have descriptive comments and failure messages. It will be difficult to keep the description you suggest in sync with the test implementation.
- TEST_WINVER This contains the value of the '-winver' Wine
argument, or the Windows version if the test is being run in Windows.
We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...),
etc.
(Do we need to distinguish between Windows and Wine?)
Yes, we do. Sometimes we have implemented behavior only for one Windows version, e.g. NT. In this case switch -winver won't affect Wine behaviour.
- TEST_BATCH If true (equal to 1) or unset, then the test
should assume that it is being run from within the test framework and thus that it should be non-interactive. If TEST_BATCH is set to 0, then the
test can assume that it is being run in interactive mode, and thus ask questions to the user. Of course most tests will simply behave identically in both cases,
I strongly recommend to use only batch version. If somebody has reasons to play with different forms of input, he can use any of following easy options: a) hardcode it - original test is improved ;-) b) take the test code to his interactive application c) add interaction with user to his local copy of the tests
Running tests
In Wine:
'make tests' seems the best way to do things. But some tests may need to create windows. For instance I have a DIB test that creates a window, draws a number of DIBs in it and checks the bitmap bits of these DIBs and then exits. Thus it is
a non-interactive test. I am not really sure whether
the window actually needs to be made visible or not, but even if this particular exemple does not require it, I suspect that othersm checking for message
sequences
for instance, may need to make the window visible.
I have tests which show window to use window messaging. Idea about tests separation to cui and gui looks good. You can run cui tests while doing something else.
In Windows:
Hmmm, not sure how that is done. Run
'winetest.exe'?
Something like: perl run_all_tests.pl
The unit test application we develop will be a test of Win32 API in general, useful for all implementations. I'm for keeping this test in separate directory tree, not mixing it with Wine files.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
I suggest to use explicit checks and print descriptive messages in case of falure. I agree, this approach is more labour-intensive, especially for tests using IPC. It is also much more maintainable as soon as you coded it. Everything, including differences between different Windows platforms is documented directly in the code! This gives much better control. E.g, it is possible to comment-out part of the test, still getting meaningful results what worked, what did not.
I definitely agree here, having the code check everything itself is much better that having to compare the output. The drawback is that it makes tests more complex to write, so I'm not sure if we want to make it the rule for all tests.
On Wed, Jan 02, 2002 at 11:40:40AM -0800, Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
I suggest to use explicit checks and print descriptive messages in case of falure. I agree, this approach is more labour-intensive, especially for tests using IPC. It is also much more maintainable as soon as you coded it. Everything, including differences between different Windows platforms is documented directly in the code! This gives much better control. E.g, it is possible to comment-out part of the test, still getting meaningful results what worked, what did not.
I definitely agree here, having the code check everything itself is much better that having to compare the output. The drawback is that it makes tests more complex to write, so I'm not sure if we want to make it the rule for all tests.
I think we do want to do this :-) It doesn't add significant overhead to the test procedure, and as far as "third-party" (aka "Windows") developers are concerned, they could just hack away at their specific test, and even if they miss version checking, then we could easily add this, I guess, as someone *will* notice and will enhance it properly.
The final attempt to solicit feedback for my suggestion to use existing testing framework.
Want to bring to your attention testing framework Test::Simple. I think you'll like this one the most because it implements exactly the ideas you suggested earlier, plus some more.
You can play with the examples unpacking file winetest.tar.gz to the existing winetest application directory.
1) look at file test1.pl. It implements exactly the functionality of existing test.pl module with using Test::Simple framework. The only change I made are descriptive error messages for the first few tests.
Output of test1.pl: ok 1 - Valid atom handle ok 2 - No error code defined ok 3 ok 4 - Succeed code ok 5 - Atom name ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 1..15
The basic usage is not more difficult than one you suggested, right?
2) test2.pl - very simple test script. Demonstrates TODO tests functionality. These are tests which are known to fail - you are notified if any of these succeeds by miracle. You'll see following output if you run the test:
1..4 ok 1 - Success not ok 2 # Failed test (test2.pl at line 8) not ok 3 # TODO Example of using TODO tests # Failed (TODO) test (test2.pl at line 12) ok 4 - Example of successfull TODO test # TODO Example of using TODO tests # Looks like you failed 1 tests of 4.
3) Things become even more interesting when Test::Simple is used with module Test::Harness. Test::Harness allows to run many tests at once and consolidate results of these tests. test_all.pl uses the module to run all the tests (currently test2.pl only). The output of the script:
test2.p.............# Failed test (test2.pl at line 8) # Looks like you failed 1 tests of 4. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED tests 2-3 Failed 2/4 tests, 50.00% okay Failed Test Status Wstat Total Fail Failed List of failed ------------------------------------------------------------------------------- test2.pl 1 256 4 2 50.00% 2-3 Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.
4) the framework has other nice features, like skipping tests. Useful in choosing platform-specific tests, gui vs cli, etc.
Is this functionality sufficient for winetest?
However I found a few issues with winetest: 1) For some reason running test_all.pl with winetest gives compilation error. I saw the same compilation error when I tried to use other Perl testing framework Test::Unit. 2) Compilation failure when I try to run test1.pl directly with Perl, like "perl test1.pl"
Look forward for your answer. Let me know if you need more information.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Running test1.pl returns to me :
[syl@snoop winetest]$ cd /c/winetest [syl@snoop winetest]$ perl test1.pl Can't locate wine.pm in @INC (@INC contains: /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at test1.pl line 8. BEGIN failed--compilation aborted at test1.pl line 8.---
Andriy Palamarchuk apa3a@yahoo.com a écrit : > The final attempt to solicit feedback for my
suggestion to use existing testing framework.
Want to bring to your attention testing framework Test::Simple. I think you'll like this one the most because it implements exactly the ideas you suggested earlier, plus some more.
You can play with the examples unpacking file winetest.tar.gz to the existing winetest application directory.
- look at file test1.pl. It implements exactly the
functionality of existing test.pl module with using Test::Simple framework. The only change I made are descriptive error messages for the first few tests.
Output of test1.pl: ok 1 - Valid atom handle ok 2 - No error code defined ok 3 ok 4 - Succeed code ok 5 - Atom name ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 1..15
The basic usage is not more difficult than one you suggested, right?
- test2.pl - very simple test script. Demonstrates
TODO tests functionality. These are tests which are known to fail - you are notified if any of these succeeds by miracle. You'll see following output if you run the test:
1..4 ok 1 - Success not ok 2 # Failed test (test2.pl at line 8) not ok 3 # TODO Example of using TODO tests # Failed (TODO) test (test2.pl at line 12) ok 4 - Example of successfull TODO test # TODO Example of using TODO tests # Looks like you failed 1 tests of 4.
- Things become even more interesting when
Test::Simple is used with module Test::Harness. Test::Harness allows to run many tests at once and consolidate results of these tests. test_all.pl uses the module to run all the tests (currently test2.pl only). The output of the script:
test2.p.............# Failed test (test2.pl at line 8) # Looks like you failed 1 tests of 4. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED tests 2-3 Failed 2/4 tests, 50.00% okay Failed Test Status Wstat Total Fail Failed List of failed
-------------------------------------------------------------------------------
test2.pl 1 256 4 2 50.00% 2-3 Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.
- the framework has other nice features, like
skipping tests. Useful in choosing platform-specific tests, gui vs cli, etc.
Is this functionality sufficient for winetest?
However I found a few issues with winetest:
- For some reason running test_all.pl with winetest
gives compilation error. I saw the same compilation error when I tried to use other Perl testing framework Test::Unit. 2) Compilation failure when I try to run test1.pl directly with Perl, like "perl test1.pl"
Look forward for your answer. Let me know if you need more information.
Andriy Palamarchuk
Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
ATTACHMENT part 2 application/x-gzip
name=winetest.tar.gz
___________________________________________________________ Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français ! Yahoo! Courrier : http://courrier.yahoo.fr
Andriy Palamarchuk apa3a@yahoo.com writes:
- look at file test1.pl. It implements exactly the
functionality of existing test.pl module with using Test::Simple framework. The only change I made are descriptive error messages for the first few tests.
Output of test1.pl: ok 1 - Valid atom handle ok 2 - No error code defined ok 3 ok 4 - Succeed code ok 5 - Atom name ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 1..15
The basic usage is not more difficult than one you suggested, right?
Yes, using ok() or assert() is pretty much the same. But it should not be printing all that stuff IMO, except if you explicitly ask it to when debugging a test for instance. The TODO/SKIP stuff is interesting, I agree we probably want something like that.
- Things become even more interesting when
Test::Simple is used with module Test::Harness. Test::Harness allows to run many tests at once and consolidate results of these tests. test_all.pl uses the module to run all the tests (currently test2.pl only). The output of the script:
test2.p.............# Failed test (test2.pl at line 8) # Looks like you failed 1 tests of 4. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED tests 2-3 Failed 2/4 tests, 50.00% okay Failed Test Status Wstat Total Fail Failed List of failed
test2.pl 1 256 4 2 50.00% 2-3 Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.
I really don't see a need for this kind of things. IMO we should enforce that tests always succeed, otherwise we can't do regression testing. And running the tests through the Makefile has the advantage that you can check dependencies and only run tests when something affecting them has changed.
Francois Gouget fgouget@free.fr writes:
I think what we need with this is a couple of guidelines and documentation for potential test writers, and maybe a couple of extensions. The following is half a proposed documentation that we could put in the Wine Developper Guide, and half a proposed specification for some possible extensions. As usual, comments and suggestions are welcome.
Great job!
- it should be easy to correlate with the source of the test. For
instance if a check fails, it would be a good idea to print a message that can easily be grepped in the source code, or even the line number for that check. But don't print line numbers for success messages, they will change whenever someone changes the test and would require an update to the reference files..
IMO the test should not be printing successes or failures at all. If it can determine whether some result is OK or not, it should simply do an assert on the result. Printing things should be only for the cases where checking for failure is too complicated, and so we need to rely on the output comparison to detect failures.
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
No, there should always be a single .ref file IMO. Version checks should be done inside the test itself to make sure the output is always the same.
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
This is already part of the Perl framework, you have to explicitly declare the functions you use. So we don't want to duplicate the information.
- TEST_WINVER This contains the value of the '-winver' Wine argument, or the
Windows version if the test is being run in Windows. We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...), etc. (Do we need to distinguish between Windows and Wine?)
The test should use GetVersion() and friends IMO, no need for a separate variable.
- and add two corresponding targets: 'make cui-tests' runs only those
tests that do not pop up windows, and 'make gui-tests' runs only those tests that do pop up windows
- 'make tests' would be 'tests: cui-tests gui-tests'
I don't think this complexity is necessary. You can always redirect the display if the windows annoy you. And tests should try to keep the windows hidden as much as possible.
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
I have attached two sample Perl scripts that were written some time ago by John Sturtz and myself. One is testing the atom functions and the other is creating a window. They should probably be simplified a bit to serve as documentation samples.
On Wed, Jan 02, 2002 at 10:20:25AM -0800, Alexandre Julliard wrote:
Francois Gouget fgouget@free.fr writes:
- it should be easy to correlate with the source of the test. For
instance if a check fails, it would be a good idea to print a message that can easily be grepped in the source code, or even the line number for that check. But don't print line numbers for success messages, they will change whenever someone changes the test and would require an update to the reference files..
IMO the test should not be printing successes or failures at all. If it can determine whether some result is OK or not, it should simply do an assert on the result. Printing things should be only for the cases where checking for failure is too complicated, and so we need to rely on the output comparison to detect failures.
Hmm, I don't know how you'd do that exactly. If we implement "strict" testing, then tons of functions will fail on Wine. And then we get an assert() every 20 seconds or what ?? More info needed here, I guess...
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
No, there should always be a single .ref file IMO. Version checks should be done inside the test itself to make sure the output is always the same.
. . .
- TEST_WINVER This contains the value of the '-winver' Wine argument, or the
Windows version if the test is being run in Windows. We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...), etc. (Do we need to distinguish between Windows and Wine?)
The test should use GetVersion() and friends IMO, no need for a separate variable.
Doh ! Right ! Like Andriy already said: the tests itself should reflect the entire behaviour of the functions, and even the version differences. One additional step in the direction of very simple, self-behaving tests...
- and add two corresponding targets: 'make cui-tests' runs only those
tests that do not pop up windows, and 'make gui-tests' runs only those tests that do pop up windows
- 'make tests' would be 'tests: cui-tests gui-tests'
I don't think this complexity is necessary. You can always redirect the display if the windows annoy you. And tests should try to keep the windows hidden as much as possible.
Hmm, why complexity ? Is it really that difficult to implement ? I'd say it's a useful feature, and it doesn't incur much penalty, so the feature/penalty quotient is high enough ;-) Avoiding visible Windows as much as possible would be nice to have, though.
Hmm, OTOH: maybe it'd be better to use make targets tests-unattended and tests-visual instead (note that I'm writing the names the other way around). This of course means that even many GUI tests would fall under the "unattended" category, thus annoying window popups would have to be minimized.
OTOH we already kid of decided that we don't want to care about GUI testing right now, so maybe we should really just use one test target for now. Splitting later should be easy anyway.
I execised with winetest to find more information about using Perl as a base for Wine testing framework.
My findings so far: I built winetest without problems. Provided sample test script worked.
The next my step would be to translate my own unit tests to Perl. My tests extensively use number constants, declared in C header file with defines and C structures.
I worked with Perl before, but never created Perl extensions. After researching I did not find reliable way to automatically import C structures to Perl.
Perl distribution provides some tools, but AFAICS the tools have limited possibilities and are not suitable for fully automatic conversion (see man pages for perlxs, perlxstut, h2xs, xsubpp). h2xs tool imported defines fine. I'm not sure how it handled functions. It does not do complete parsing of headers and files, generated files need manual editing.
Another option - SWIG (www.swig.org). Can generate bindings for a few scripting languages directly from C sources "provided they are sufficiently clean" (.h?). I do not assume yet it can make sense from our sources.
Are there Perl experts out there who knows how to handle this issue?
Can we use other approach if automatic code generation is not possible? E.g. we can manually create binding code for used structures. C compiler is needed to create or change the bindings.
Please, let me know if I missed something.
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
The next my step would be to translate my own unit tests to Perl. My tests extensively use number constants, declared in C header file with defines and C structures.
The defines can be translated to Perl constants; I once started writing a tool to do that automatically, I'll try do dig it out.
Structures should be converted with pack/unpack (cf. the win.pl example), we probably want to export the pack format strings for each structure from some Perl module too. The advantage of doing that is that the structures are not derived from the C code, which allow checking that the layout is correct. The drawback is that you need to write the format string for each structure, but you only need to do it once.
--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
The next my step would be to translate my own unit tests to Perl. My tests extensively use number constants, declared in C header file with defines
and
C structures.
The defines can be translated to Perl constants; I once started writing a tool to do that automatically, I'll try do dig it out.
h2xs in standard Perl installation filtered put defines to the .xs file. Try to run "h2xs winuser.h" and look in the generated files. The tool has some issues: - processes one .h file at a time - does not do full-fledged header parsing, so it can have problems with complex precompilator structures - generates C code for the constants. This approach looks more efficient than simple declaring but requires to regenerate and recompile the C file whenever header is changed.
We also can chage h2xs to the behavior we need.
Structures should be converted with pack/unpack (cf. the win.pl example), we probably want to export the pack format strings for each structure from some Perl module too. The advantage of doing that is that the structures are not derived from the C code, which allow checking that the layout is correct. The drawback is that you need to write the format string for each structure, but you only need to do it once.
Looks good.
Andriy
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
On 2 Jan 2002, Alexandre Julliard wrote:
Francois Gouget fgouget@free.fr writes:
[...]
IMO the test should not be printing successes or failures at all. If it can determine whether some result is OK or not, it should simply do an assert on the result. Printing things should be only for the cases where checking for failure is too complicated, and so we need to rely on the output comparison to detect failures.
I am not sure about using asserts. In the case where one calls the same API ten times with different parameters, it would probably be better to still do all ten tests even if the second call fails. This way the developper debugging things immediately knows if just one aspect failed or if its everything. If we call assert as soon as one item fails we don't have the information of whether the rest still works or not. Of course this assumes that a failure is not fatal for the rest of the tests. If it is, then an assert could make sense. But I agree tests could be written so as not to print anything unless there is a failure. Especially for simple cases like checking return codes, there is no need to print the return code and to check it against a reference file. Then a single 'Test: Ok' before the final return would make sure we don't get an empty output file if that is important.
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
No, there should always be a single .ref file IMO. Version checks should be done inside the test itself to make sure the output is always the same.
The same way there are cases where the simplest solution for checking the result is correct is to compare it against a reference file, I'm thinking that there are cases where the somplest way to make sure we behave the same way as a specific windows version is to compare the output to that windows version. But I must admit that I don't have such an example for now so we can start with the single '.ref' file for now. Then, if necessary, the above mechanism should be relatively simple to put in place.
Each test should contain a section that looks something like:
# @START_TEST_COVERAGE@ # kernel32.CreateProcess # kernel32.GetCommandLineA # kernel32.GetCommandLineW # __getmainargs # __p__argc # __p_argv # __p_wargv # @END_TEST_COVERAGE@
This is already part of the Perl framework, you have to explicitly declare the functions you use. So we don't want to duplicate the information.
But in the case where a test has to call GetVersion() to handle differences between windows versions, we surely don't want to go on record as saying that this test tests GetVersion. Similarly, pretty much all GUI tests will call things like RegisterClassExA, CreateWindowExA, GetMessageA, ... then, finding the two or three tests that really test these will be pretty hard. Maybe the test framework could have an alias for "wine::declare", maybe "wine::test". Then APIs declared with "wine::test" are those that are being tested while those declared with "wine::declare" are just overhead.
- TEST_WINVER This contains the value of the '-winver' Wine argument, or the
Windows version if the test is being run in Windows. We should mandate the use of specific values of winver so that test don't have to recognize all the synonyms of win2000 (nt2k...), etc. (Do we need to distinguish between Windows and Wine?)
The test should use GetVersion() and friends IMO, no need for a separate variable.
Good point.
- and add two corresponding targets: 'make cui-tests' runs only those
tests that do not pop up windows, and 'make gui-tests' runs only those tests that do pop up windows
- 'make tests' would be 'tests: cui-tests gui-tests'
I don't think this complexity is necessary. You can always redirect the display if the windows annoy you. And tests should try to keep the windows hidden as much as possible.
Ok. I guess we can still split tests in two categories later if necessary.
What is needed most is a two sample tests:
- one simple console based test
- another one involving some GUI stuff
I have attached two sample Perl scripts that were written some time ago by John Sturtz and myself. One is testing the atom functions and the other is creating a window. They should probably be simplified a bit to serve as documentation samples.
Nice :-)
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ "Utilisateur" (nom commun) : Mot utilisé par les informaticiens en lieu et place d'"idiot".
Francois Gouget fgouget@free.fr writes:
I am not sure about using asserts. In the case where one calls the same API ten times with different parameters, it would probably be better to still do all ten tests even if the second call fails. This way the developper debugging things immediately knows if just one aspect failed or if its everything. If we call assert as soon as one item fails we don't have the information of whether the rest still works or not.
Sure, it doesn't have to be the C library version of assert, it can be an equivalent that continues execution and only sets the exit status at the end.
On 2 Jan 2002, Alexandre Julliard wrote:
Francois Gouget fgouget@free.fr writes:
I am not sure about using asserts. In the case where one calls the same API ten times with different parameters, it would probably be better to still do all ten tests even if the second call fails. This way the developper debugging things immediately knows if just one aspect failed or if its everything. If we call assert as soon as one item fails we don't have the information of whether the rest still works or not.
Sure, it doesn't have to be the C library version of assert, it can be an equivalent that continues execution and only sets the exit status at the end.
Kind of like my proposed 'test_failed(message)' function :-) I agree then.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ Avoid the Gates of Hell - use Linux.
Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
- look at file test1.pl. It implements exactly the
functionality of existing test.pl module with using Test::Simple framework. The only change I made are descriptive error messages for the first few tests.
Output of test1.pl: ok 1 - Valid atom handle ok 2 - No error code defined ok 3 ok 4 - Succeed code ok 5 - Atom name ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 1..15
The basic usage is not more difficult than one you suggested, right?
Yes, using ok() or assert() is pretty much the same.
But it should not be printing all that stuff IMO, except if you explicitly ask it to when debugging a test for instance.
The 'ok' messages are used by the Test::Harness module to track progress of the test. E.g. it can report point of crash if the test silently dies by knowing the last reported test. You should use Test::Harness module to configure report as you like.
[...]
- Things become even more interesting when
Test::Simple is used with module Test::Harness. Test::Harness allows to run many tests at once and consolidate results of these tests. test_all.pl uses the module to run all the tests (currently test2.pl only). The output of the script:
test2.p.............# Failed test (test2.pl at line 8) # Looks like you failed 1 tests of 4. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED tests 2-3 Failed 2/4 tests, 50.00% okay Failed Test Status Wstat Total Fail Failed List
of
failed
test2.pl 1 256 4 2 50.00% 2-3 Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.
I really don't see a need for this kind of things. IMO we should enforce that tests always succeed, otherwise we can't do regression testing.
Always succeed *under Windows*. Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
The value of unit tests is exactly in failures! The more failures of unit tests we have - the better test developers do their work.
The whole programming methodology exists which dictates that you write tests first, then implement code which makes them succeed. Please, look at this short article to better understand my point of view: "Test Infected: Programmers Love Writing Tests" http://members.pingnet.ch/gamma/junit.htm
And running the tests through the Makefile has the advantage that you can check dependencies and only run tests when something affecting them has changed.
Wrong. In make files you can check only some compilation dependencies. You can't do anything about logical dependencies. I'm for running the whole suite of the tests by developers before submitting *any* patch and from time to time centrally.
Think about the Wine unit tests as a specification of Win32 API. When you get such reference the first thing you want to do is to find all the areas where Wine does not correspond to the specification, right? Then you repeat such check after changes to be sure nothing is broken. This is a great reference - self-controlling, growing with each usage.
This is why I want very much to keep it in separate directory. Even the tests structure, probably, should correspond more to logical Win32 API structure, not to Wine directory tree. The Win32 unit test suite has huge value by itself. All Win32 projects can contribute to it on the same rights. The ideal solution, probably, whould be to move completely separate project. We can do this later if the need arises.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
Always succeed *under Windows*. Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
The value of unit tests is exactly in failures! The more failures of unit tests we have - the better test developers do their work.
Well, I guess it's a philosophical point, but for me the value of the tests is for regression testing. If you allow the tests to fail you'll pretty soon have 90% of the tests fail somewhere, and this is completely useless except maybe as a list of things we still have to do. While if the tests usually succeed, as soon as something fails you know there's a regression and you have to fix it.
Wrong. In make files you can check only some compilation dependencies. You can't do anything about logical dependencies. I'm for running the whole suite of the tests by developers before submitting *any* patch and from time to time centrally.
What you can do with my make test patch is run make test -k first, let it go through everything, and then run make test again and it will only run the tests that failed the first time, or that have been modified. This is a major gain. It could certainly be done some other way too without using make, but AFAICS your test harness would force to run all tests all the time (or to disable them manually). This is not acceptable once you have a large number of tests.
This is why I want very much to keep it in separate directory. Even the tests structure, probably, should correspond more to logical Win32 API structure, not to Wine directory tree.
Well, the Wine directory tree corresponds (mostly) to the Win32 API structure. But this is again a philosophical point. I don't view the test suite as an API specification, but as a regression test suite for Wine.
The Win32 unit test suite has huge value by itself. All Win32 projects can contribute to it on the same rights. The ideal solution, probably, whould be to move completely separate project. We can do this later if the need arises.
This is like the documentation: yes it would be great to have good documentation and good specifications of the Win32 API. It's not going to happen, because there's no motivation behind it except "would be nice to have".
OTOH a Wine test suite can happen (I hope) because this is something Wine developers need when they write code, so there is at least some motivation for them to write tests. We can do it in a way that the Wine test suite is useable for more than testing Wine, but this has to be the main focus.
On 3 Jan 2002, Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
Always succeed *under Windows*. Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
The value of unit tests is exactly in failures! The more failures of unit tests we have - the better test developers do their work.
Well, I guess it's a philosophical point, but for me the value of the tests is for regression testing. If you allow the tests to fail you'll pretty soon have 90% of the tests fail somewhere, and this is completely useless except maybe as a list of things we still have to do. While if the tests usually succeed, as soon as something fails you know there's a regression and you have to fix it.
This is why the notion of 'TODO tests' or known differences (my '.ref.diff') is useful. This way when writing a test you don't have to restrict yourself to just what works at a given point in Wine, and still, the tests don't fail. But as soon as something changes in the failure mode, or even stops failing, then you know it.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ Nouvelle version : les anciens bogues ont été remplacés par de nouveaux.
Alexandre Julliard wrote:
Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
That means you probably want to live with less than 100% success rates. The important thing when committing a new change to Wine (as opposed to a change to the test suite) is that it not cause any *new* failures. I bet that's what you meant.
I'm so jazzed by the new emphasis on regression testing. There were sparks of it in previous years http://groups.google.com/groups?hl=en&selm=38CE7B2D.77204824%40alumni.ca... but it didn't catch on for some reason.
Francios, are your tests (from http://fgouget.free.fr/wine/booktesting-en.shtml) part of this framework yet? - Dan
On Thu, 3 Jan 2002, Dan Kegel wrote: [...]
Francios, are your tests (from http://fgouget.free.fr/wine/booktesting-en.shtml) part of this framework yet?
No and they will not be. The reason is that the source for these tests are part of books and as such it is all copyrighted material. Plus, the actual programs in these tests are almost all interactive so they are not suited for our automated testing purposes. But I will continue to maintain them (hmmm, whenever I have time) as they are a good complement, especially for testing winemaker and Winelib.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ Linux: It is now safe to turn on your computer.
On Thu, Jan 03, 2002 at 10:59:37AM -0800, Dan Kegel wrote:
Alexandre Julliard wrote:
Do you really, really, really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
OK, let me show my support for your view, too.
IMHO not committing any tests that fail is a high goal, but it's simply the wrong one ;-)
Right now we've got a *lot* of problems giving people something worthwhile to hack on: Getting started with Wine is very difficult.
By giving people 150 (or, as far as I'm concerned, it'll be many more) tests that fail on Wine, they even have *choice* in selecting the specific issue that they want to fix !
That's why a "perfect" test suite is bad.
Dan Kegel dank@kegel.com writes:
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
In that case the test should use a TODO mechanism or equivalent, and it must still be possible to run make test without failure (but there would be an option to switch the failures on if you want).
That means you probably want to live with less than 100% success rates. The important thing when committing a new change to Wine (as opposed to a change to the test suite) is that it not cause any *new* failures. I bet that's what you meant.
No, what I mean is that you can't spot new failures if every test run shows hundreds of existing ones. The only way to find new failures is if you can do a successful test run before a change.
Imagine that you have 1000 tests, and a typical run shows 250 failures. Then you make a change, and you now see 248 failures. Does it mean you fixed 2 bugs, or does it mean you fixed 5 and introduced 3 new ones? You have no way of knowing without going through the 250 failures one by one. This is clearly not possible.
Alexandre Julliard wrote:
Dan Kegel dank@kegel.com writes:
That's great to hear, but I think you have to modify your statement a bit -- you may want to commit new tests don't yet pass, if they show a real flaw in Wine.
In that case the test should use a TODO mechanism or equivalent, and it must still be possible to run make test without failure (but there would be an option to switch the failures on if you want).
The dejagnu test harness that is used by e.g. the gcc test suite allows to classify a test case as 'expected to fail'.
When you run the test, every test case either passes or fails, which results in a classification into four subsets:
PASS Test case was expected to pass, and it did FAIL Test case was expected to pass, but failed XPASS Test case was expected to fail, but passed XFAIL Test case was expected to fail, and it did
Only a test case in the FAIL category causes the whole test run to fail, and is reported even in the test run summary. The other categories are only reported as total numbers.
If you are getting nonzero FAIL numbers, you have introduced a regression. Nonzero PASS and XFAIL numbers are expected; if you get nonzero XPASS numbers you might look at the cases in question and decide whether you want to remove the 'expected to fail' flag.
This system works quite well in my experience with gcc, maybe something like this could be implemented for Wine as well ...
Bye, Ulrich
--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
Always succeed *under Windows*. Do you really,
really,
really think all the tests will succeed under Wine from day 1 and we will be able to maintain them failure-free?
Absolutely. There's a very simple way of enforcing that: I'm not comitting anything that causes make test to fail.
The value of unit tests is exactly in failures!
The
more failures of unit tests we have - the better
test
developers do their work.
Well, I guess it's a philosophical point, but for me the value of the tests is for regression testing. If you allow the tests to fail you'll pretty soon have 90% of the tests fail somewhere, and this is completely useless except maybe as a list of things we still have to do. While if the tests usually succeed, as soon as something fails you know there's a regression and you have to fix it.
As Francois mention, this is why TODO tests exists. Even without TODO tests it is not wise to reject a perfectly correct patch from a Windows developer who even does not have Wine.
Without TODO construction compromise is still possible. E.g we can use conditional compilation or execution to select only tests which succeed for finding regressions. I believe there will be somebody, interested in mining "dirty" tests. Other option is to store failture lists and compare them from time to time in a search of regressions.
What you can do with my make test patch is run make test -k first, let it go through everything, and then run make test again and it will only run the tests that failed the first time, or that have been modified. This is a major gain. It could certainly be done some other way too without using make, but AFAICS your test harness would force to run all tests all the time (or to disable them manually). This is not acceptable once you have a large number of tests.
I don't see poing in selecting subsets of tests. From experience - even with pretty big number of tests it does not take long time to execute them.
OTOH a Wine test suite can happen (I hope) because this is something Wine developers need when they write code, so there is at least some motivation for them to write tests.
Exactly! I do not want to spend resources of Wine project on some "nice documentation". On the contrary, goal of this change is to invite external resources.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
I don't see poing in selecting subsets of tests. From experience - even with pretty big number of tests it does not take long time to execute them.
A good testing suite is going to take a long time to run. My simple atom test takes about 3 seconds on my machine, 1000 such tests would take an hour. Now some of that is Perl overhead of course, C tests would likely run faster but you need to compile them first so it's not necessarily a gain.
In either case a full run will most likely take longer than compiling Wine itself. Running all tests all the time is simply not an option; just imagine if you had to rebuild all of Wine everytime you change something.
On Thursday 03 January 2002 07:54 am, Andriy Palamarchuk wrote:
Alexandre Julliard wrote:
The value of unit tests is exactly in failures! The more failures of unit tests we have - the better test developers do their work.
The whole programming methodology exists which dictates that you write tests first, then implement code which makes them succeed. Please, look at this short article to better understand my point of view: "Test Infected: Programmers Love Writing Tests" http://members.pingnet.ch/gamma/junit.htm
According to Extreme Programming Installed, the chapter on Unit Tests, page 97: "Everyone on the team releases code only when all the unit tests in the entire system run at 100 percent". So in theory there shouldn't be any failures since the code wouldn't make it into the CVS tree. The only way this could work in the face of missing functionality is that the tests for that functionality are not run until the functionality is implemented.
The value is when you add new functionality (and possibly new tests) and old tests break. Then you can pinpoint the changes that caused the old tests to break. Again, that can only work if all the old tests succeeded, which means you can't include tests that you know will fail in a release.
--Rob
- its exit code
- text output on either or both of stdout and stderr, both of which are
normally redirected to a file called 'xxx.out'.
A test succeeds if:
- its exit code is 0
- and its output, 'xxx.out' matches the reference output according to
the rules described later.
I think that it would be handy to use stderr for status/diagnostics, and only use stdout for reference checking. Chatty people like me get their statii, but the tests remain clean. Perhaps a WINETEST_DEBUG env variable would be a good addition as well (and since Alexandre controls the commits, my guess is it'll default to off <g>).
Otherwise this file is either called:
- 'xxx.ref'
- or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
Windows version being emulated. The winever-specific file takes precedence over the '.ref' file, and the '.ref' file, which should exist, serves as a fallback.
I hope that the cases where this would be needed would be few enough that we wouldn't need to build in a general purpose exception; we can just have a mytest.win95.test and a mytest.win98.test for the cases where it's needed.
[snipping chunks largely agreed with]
Test coverage
Each test should contain a section that looks something like:
I started trying to tweak Alexandre's patch to create some sample tests, and I learned the following: 1. Cygwin installation has *dramatically* improved. Getting a full working toolchain is no longer a big pain in the rear end, it's actually pretty easy.
2. Having '.test' files be implicitly Perl scripts is too limiting, IMHO. I hate to add another format, but I've been toying with YAFF (yet another file format) so that a '.test' file describes a test, as follows: # Comment lines script=name_of_my_perl_test_script invoke=name_of_c_test_or_shell_script status=-eq 0 pattern=.*OK compare=name_of_my_ref_file
where one of script or invoke is required, and status, if given, is a test expression that $0 is compared to, pattern is a regexp applied to stdout, if given, and compare is the name of a .ref.out file to compare the output with.
The default would be just a case of 'status=-eq 0'.
The nice thing about this approach is that you can handle the multiple version testing just by creating a new reference file and a new .test file.
In Windows:
Hmmm, not sure how that is done. Run 'winetest.exe'?
IMO, we should have a script that creates a 'winetest.zip', with simple batch files to make it easy to run a single test.
But, (as a new convert to the ease of use of Cygwin), I think we can just stick with 'make tests' for the full deal on Windows.
Jer
On 2002.01.02 14:12 Jeremy White wrote: [big snip]
- Cygwin installation has *dramatically* improved. Getting a full working toolchain is no longer a big pain in the rear end, it's actually pretty easy.
[big snip]
Well, as I mentioned the other day... I have recently built a linux cross mingw32 toolchain with the latest released binutils (maybe I should upgrade that, seems the mingw people also use a newer unstable binutils) and the latest released gcc (3.0.3) along with the MinGW w32api and mingw-runtime packages (both version 1.2).
There were a few issues such as building with multithread support that crept up into the build process and thus I have some patches for that if anyone is interested. Most of them patch the configure scripts and so on so that it uses threading for the target environment instead of trying to compile posix threads.
I will be contacting the MinGW team about this shortly.
-Dave
Hi folks, I ported a sample of my tests from C to the Perl framework. Want to share my experiences.
Good: 1) AFAICS technically you could do all the things you could do in C - working with structures, using callback functions, using threads. Existing problems probably can be ironed out. 2) additional processing of the test results can be done easier and quicker than in C
Bad: 1) constants must be explicitely declared or plain numbers used instead 2) no documentation on the framework yet. 3) in many cases data sizes must be specified manually - for structures processing, function parameters. 4) more difficult to track issues. than in C
Note, issues (1) and (2) probably can be fixed.
Summary: For this application C is higher-level language than Perl, provides more services to developer.
After trying the Perl framewok I still vote for using C for the unit tests. For more information about advantages C over Perl see my previous messages.
Note, with David Elliott improvements of MigGW toolchain it is now possible to use one more set of Win32 headers under *nix. Differences between header files can be found without using Windows.
I urge you to try to use the framework and share your experience. The sample Alexandre sent earlier helped me a lot. I attached my script.
Thank you, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
--- Andriy Palamarchuk apa3a@yahoo.com wrote:
Bad:
5) no types checking, so errors in values, calculated manually won't be caught :-(
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
On 27 Dec 2001, Alexandre Julliard wrote: [...]
The C vs. Perl question has been debated already, please check the archive. The truth is that a lot of people are willing to setup a testing infrastructure, but nobody is willing to write the actual tests.
I don't know if it is that noone wants to write tests or if it is that: * the current infrastructure has never been officially announced * the Wine project never officially said 'go ahead' write tests with our new testing infrastructure * there is no documentation on either how to write tests, no guidelines on how to write good tests, or on how one is supposed to run the Wine regression test suite * there is no sample test in any of the dlls/xxx directories. There is just one sample perl script in programs/winetest/test.pl * and there is also an autoconf test missing for '-lperl': gcc -shared -Wl,-Bsymbolic winetest.spec.o winetest.o wine.o -o winetest.so -L../../library -lwine `perl -MExtUtils::Embed -e ldopts` -lm -lutil -ldl /usr/bin/ld: cannot find -lperl
So I would say we still have work to do on the infrastructure, if nothing else in terms of documentation and 'advertising' it, if we are to expect people to start writing tests.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ $live{free} || die "";
On Thu, 27 Dec 2001, Andriy Palamarchuk wrote:
[... Perl/C pros and cons for testing]
I think you summarised the pros and cons of both options quite well. I would add just one thing against perl in its current form: AFAIK (but it may have changed since), the current perl framework does not support callback functions. This can be a problem for testing things like: * CreateThread: takes a function to be run in the new thread * timers: SetTimer(...,timerproc) * window procs: We can write quite a few tests that create some widget (a list, table, ...) and then sends messages to check the state of that widget (select an item in the list, then check which item is selected, whether the 'hot' item is still hot, etc.), and or the resulting sequence of other messages. While these are not 'GUI' tests in that they don't make sure that the list displays correctly (or at all), they check important properties of the widget implementation.
The big question is a tool to test GUI. I did not find any OS Windows GUI testing frameworks :-(
Andreas has already replied and i agree with him. But I'll basically repeat what he said to give it more weight :-)
GUI testing is not 'the big question'. It's irrelevant right now. And what I really don't want to happen, it to see us refuse to pick the low hanging fruits and starve to death because we don't have a ladder that lets us reach the fruits at the top of the tree.
In other words, there are thousands of APIs that we can test very easily and we should write tests for them NOW. We should not wait for the development of a hypothetical framework that would let us also test graphical (i.e. daoes it display right) issues. Testing all these APIs represents enough work to keep us busy for quite some time and can already benefit Wine greatly. So really, at this time, given the amount of effort required to even get something usable, "GUI testing" is irrelevant and should be postponed.
(of course if someone out there really wants to develop a GUI testing framework and donate it to Wine, go ahead, all I want is that we don't wait for one)
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ Linux: the choice of a GNU generation
On Thu, Dec 27, 2001 at 12:13:05PM -0800, Francois Gouget wrote:
On Thu, 27 Dec 2001, Andriy Palamarchuk wrote:
[... Perl/C pros and cons for testing]
I think you summarised the pros and cons of both options quite well. I would add just one thing against perl in its current form: AFAIK (but it may have changed since), the current perl framework does not support callback functions. This can be a problem for testing things like:
- CreateThread: takes a function to be run in the new thread
- timers: SetTimer(...,timerproc)
- window procs: We can write quite a few tests that create some widget
(a list, table, ...) and then sends messages to check the state of that widget (select an item in the list, then check which item is selected, whether the 'hot' item is still hot, etc.), and or the resulting sequence of other messages. While these are not 'GUI' tests in that they don't make sure that the list displays correctly (or at all), they check important properties of the widget implementation.
Not to mention any other "non-easy" tests, such as highly complicated interaction/message handling/whatever that probably would be rather problematic to implement in perl. Thus I guess that we might want to concentrate on a usable C based test code, while keeping a compatibility scheme for perl based tests in mind. In short: my test model would be usable from that point of view. You could concatenate the output of all C programs and perl scripts easily and get an easy way of determining whether overall compatibility is healthy.
The big question is a tool to test GUI. I did not find any OS Windows GUI testing frameworks :-(
Andreas has already replied and i agree with him. But I'll basically repeat what he said to give it more weight :-)
GUI testing is not 'the big question'. It's irrelevant right now. And what I really don't want to happen, it to see us refuse to pick the low hanging fruits and starve to death because we don't have a ladder that lets us reach the fruits at the top of the tree.
Yes. Again, the simple text based "OK"/"FAILED" result would be applicable easily here, too. After all it's only the result that matters... Thus I don't think we need to worry much about it, since most likely it'll fit into such an infrastructure easily.
In other words, there are thousands of APIs that we can test very easily and we should write tests for them NOW. We should not wait for the development of a hypothetical framework that would let us also test graphical (i.e. daoes it display right) issues.
Exactly. The most valuable part of API tests is not their infrastructure, but rather the fact that they already tested certain functions pretty well. E.g. we'd be able to port my test code to a different test framework terribly easily, as (hopefully ?) all the test conditions of LoadModule16 are known now due to my evaluation of this function.
Testing all these APIs represents enough work to keep us busy for quite some time and can already benefit Wine greatly. So really, at this time, given the amount of effort required to even get something usable, "GUI testing" is irrelevant and should be postponed.
My idea exactly. People kept talking about GUI test framework (also at Codeweavers), but I think we should really get started at all first.
(of course if someone out there really wants to develop a GUI testing framework and donate it to Wine, go ahead, all I want is that we don't wait for one)
Hehe, right :-)
OK, now that we discussed this: I have to admit that I haven't really looked at the perl winetest yet. *blush*
I think I'll improve my intended Usenet posting (which will go to truly relevant Windows devel newsgroups only, I'd say), and then I'll mail it to interested parties for review. Any comments/improvements to this text ?
In the meantime we should try to discuss more about what the test suite framework should look like, i.e. whether my approach is good/bad, what to possibly improve, what the output should look like and whether it's suitably parsable.
I looked at thread "Perl/Wine extension for perusal" ran on February, 2001. Want to bring some information from that thread to this discussion:
1) The discussion started from John Sturtz post, who created the Perl module for Win32 functions. Discussion what is better - C or Perl for unit testing started later as I understand there was no conclusion. Now I can assume that this topic was not "discussed to death" and we can do it now ;-)
2) One of arguments about the tool choice was its availability.
Currently you are free to use one of a few commercial compilers, free compiler lcc or gcc as part of Cygwin, mingw packages. One more problem - support for a few compilers environment for C. Perl is available as ActiveState and standard ports.
3) No existing unit tests frameworks were discussed.
4) There was a suggestion to use both - C and Perl tests
5) Was defined that existing application should not be run as part of unit test.
I think these are all points of that discussions, which will be interesting for this discussion. Feel free to correct me.
--- Andreas Mohr andi@rhlx01.fht-esslingen.de wrote:
On Thu, Dec 27, 2001 at 12:13:05PM -0800, Francois Gouget wrote:
The big question is a tool to test GUI. I did
not find
any OS Windows GUI testing frameworks :-(
Andreas has already replied and i agree with
him. But I'll basically
repeat what he said to give it more weight :-)
GUI testing is not 'the big question'. It's
irrelevant right now.
I'm convinced.
In the meantime we should try to discuss more about what the test suite framework should look like, i.e. whether my approach is good/bad, what to possibly improve, what the output should look like and whether it's suitably parsable.
Results of preliminary review of the unit testing frameworks. Some of these frameworks can run tests in a separate address space and can report crashes! Suprisingly I don't have too many options.
C frameworks: 1) Check. Problem - POSIX-based (under Windows needs Cygwin). 2) CUnit. Problem - very simple, in development. I think it is not worth trying. 3) cUnit. Problem - Linux only. 4) Autounit. Problem - POSIX-based 5) QMTest. Problem - needs Python.
C++ frameworks: 1) CPPUnit. Problem - in C++
Perl frameworks: - there are quite a few Perl modules for testing on CPAN, including port of JUnit to Perl.
Summary: What do you think - which ones we still can use despite the constraints? I'll review the chosen frameworks more closely.
Perl modules are Ok and I can review them in detail if we decide to go with Perl.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
- The discussion started from John Sturtz post, who
created the Perl module for Win32 functions. Discussion what is better - C or Perl for unit testing started later as I understand there was no conclusion. Now I can assume that this topic was not "discussed to death" and we can do it now ;-)
OK you are right, it was discussed to death inside CodeWeavers, but not all of that was public. Basically the argument is that some sort of scripting language is better than plain C for two reasons:
1. It is much easier to install under Windows than a full development environment, and we don't have to worry about supporting a dozen different compilers. We can simply provide a zip file containing the compiled script interpreter, and people can be up and running in seconds.
2. The scripts are independent from the compilation environment, which allows testing binary compatibility. In C you have to compile the tests under Wine using the Wine headers, which means you can't spot wrong definitions in the headers since the test will see the same definition as Wine itself. The only way around is to build tests under Windows and run them under Wine but this is a major pain. With a script you are guaranteed to run the exact same thing in both environments.
I started implementing a simple scripting language, but then John Sturtz showed that it was possible to leverage Perl to do the same thing, so I think it's the way to go.
There are probably a number of things you cannot do from Perl, like threads or exception handling, and for that we will want some kind of C framework too. But I believe we can already go a long way with the Perl stuff we have today. Maybe I'm wrong, maybe it's really unusable and we need to scrap it and redo a C environment from scratch; but we won't know that until we try to use it seriously.
On 2001.12.30 15:34 Alexandre Julliard wrote:
- The scripts are independent from the compilation environment, which
allows testing binary compatibility. In C you have to compile the tests under Wine using the Wine headers, which means you can't spot wrong definitions in the headers since the test will see the same definition as Wine itself. The only way around is to build tests under Windows and run them under Wine but this is a major pain. With a script you are guaranteed to run the exact same thing in both environments.
Well, if it helps any, I have recently built a linux-cross-mingw32 toolchain as RPMs.
For those not running on an RPM distro the specfiles should be easy enough to understand and do manually.
For those who are running RH7.2 (and probably 7.1 as well) you can have the binary packages. And for all other RPM distros the source RPMs should build without problems.
I have the following RPMs built:
i386-mingw32-binutils-2.11.2-0_biscuit_0 i386-mingw32-gcc-bootstrap-3.0.3-0_biscuit_0 (built from the same specfile as the normal gcc but only makes a C compiler good enough to build w32api and mingw-runtime properly). i386-mingw32-w32api-1.2-0_biscuit_0 i386-mingw32-mingw-runtime-1.2-0_biscuit_0 All the rest are part of the gcc build: i386-mingw32-gcc-3.0.3-0_biscuit_0 i386-mingw32-gcc-c++-3.0.3-0_biscuit_0 i386-mingw32-libstdc++-devel-3.0.3-0_biscuit_0 i386-mingw32-gcc-g77-3.0.3-0_biscuit_0 i386-mingw32-gcc-java-3.0.3-0_biscuit_0 (unfortunately it didn't compile the java runtime, so this one is useless until I figure that out)
If anyone is interested in this I can probably put it in kernelrpm.sourceforge.net temporarily before I contact the MinGW team and see if they would be interested in hosting these files on their sourceforge site.
Please note that this is not just a simple compile. I had to do quite a bit of patching (well, figuring out what to patch was the issue, the patches are tiny) to get it to build and work properly even including thread support.
I also have compiled wxMSW with this compiler and successfully built and tested the minimal, mdi, and taskbar test programs.
Of note is that the MDI program does not show any icons in the toolbar when run under Wine, but works perfectly under Windows (95osr2 in Win4Lin). Is anyone aware of this, might be related to IE5.5 displaying black squares instead of icons in its toolbar, and would be significantly easier to debug as I built all of this with debugging info (assuming winedbg can read it), plus you'd have the full sourcecode from wxwindows.org.
-Dave
[omitting comments about very nice "make test" framework]
On Sun, Dec 30, 2001 at 12:34:06PM -0800, Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
- The discussion started from John Sturtz post, who
created the Perl module for Win32 functions. Discussion what is better - C or Perl for unit testing started later as I understand there was no conclusion. Now I can assume that this topic was not "discussed to death" and we can do it now ;-)
OK you are right, it was discussed to death inside CodeWeavers, but not all of that was public. Basically the argument is that some sort
Hmm right. I really should have remembered the extent of these discussions.
of scripting language is better than plain C for two reasons:
- It is much easier to install under Windows than a full development
environment, and we don't have to worry about supporting a dozen different compilers. We can simply provide a zip file containing the compiled script interpreter, and people can be up and running in seconds.
That one goes to you, I guess.
There are probably a number of things you cannot do from Perl, like threads or exception handling, and for that we will want some kind of C framework too. But I believe we can already go a long way with the Perl stuff we have today. Maybe I'm wrong, maybe it's really unusable and we need to scrap it and redo a C environment from scratch; but we won't know that until we try to use it seriously.
...at which point we already have a huge data collection of "known expected function behaviour" that's just waiting for us to port it easily to C then or so...
Hmm, and different "winver" settings for testing of the behaviour of different Wine --winver settings and adapting to different Windows versions are possible with this framework, too ? (we'd very much want to have that, I guess)
Oh, and what about Win16 support of the test suite ? This is why I started this in the first place. I'm very much afraid of losing HUGE amounts of Win16 compatibility due to everybody using more and more Win32 programs only...
I haven't toyed with the "make test" environment yet (severe lack of time - exam time), so I don't know much about it. Unfortunately I'm afraid I won't have much time in the foreseeable future either.
Still, I think we should try to have a C "compatibility" layer, too, for two reasons: - people who aren't familiar with anything else - supporting diversity (what if someday we find out that perl "sucks" ? :-)
That's it for now,
Andreas (who'd really like to be able to contribute more to this now)
Andreas Mohr andi@rhlx01.fht-esslingen.de writes:
Hmm, and different "winver" settings for testing of the behaviour of different Wine --winver settings and adapting to different Windows versions are possible with this framework, too ?
In its current state it uses the default Wine config from ~/.wine so any config changes can be done this way (of course we will have to support -winver in the config file, but we need that anyway). We could provide a set of standardized configs to reduce variation, but it may in fact be preferable to let people run with the config they normally use, so that we have a wider coverage of the different options.
Oh, and what about Win16 support of the test suite ?
Not supported yet, though this could be added. In fact since the Perl layer already needs to convert the arguments, making it support 16-bit should be relatively easy, and probably a lot more transparent than doing it in C.
Still, I think we should try to have a C "compatibility" layer, too, for two reasons:
- people who aren't familiar with anything else
- supporting diversity (what if someday we find out that perl "sucks" ? :-)
If it turns out Perl doesn't work, we can always convert the tests to C, this should be relatively easy. Most of the tests will simply be a bunch of function calls and result checks, which is simple to do in any language. Besides, unless you want to do elaborate tests, you really don't need to know much Perl to be able to write tests with winetest.
Responding to Alexandre and Jeremy.
Alexandre Julliard wrote:
Basically the argument is that some sort of scripting language is better than plain C for two reasons:
- It is much easier to install under Windows than a full development
environment, and we don't have to worry about supporting a dozen different compilers. We can simply provide a zip file containing the compiled script interpreter, and people can be up and running in seconds.
Completely agree with this one.
- The scripts are independent from the compilation environment, which
allows testing binary compatibility. In C you have to compile the tests under Wine using the Wine headers, which means you can't spot wrong definitions in the headers since the test will see the same definition as Wine itself. The only way around is to build tests under Windows and run them under Wine but this is a major pain. With a script you are guaranteed to run the exact same thing in both environments.
It seems the problem is not so big. It will be sufficient to run the binary, compiled under Windows not more often than once a month.
There are probably a number of things you cannot do from Perl, like threads or exception handling, and for that we will want some kind of C framework too. But I believe we can already go a long way with the Perl stuff we have today.
Some technical challenges we can overcome by improving our Perl framework, other we will be handled with C.
Maybe I'm wrong, maybe it's really unusable and we need to scrap it and redo a C environment from scratch; but we won't know that until we try to use it seriously.
Whether Perl in convenient enough language in comparison with C remains to be decided.
I see following biggest problems with Perl:
1) as I mentioned before the first problem is in capturing audience of test creators. Even be Perl the best programming language ever: a) existing Wine developers are experts in C. Even those who knows Perl are more experienced in C. b) attracting new developers. We want to target developers who already knows or want to know Win32 API, right? Usually these people are not those who program in scripting languages. I'm not sure if we will be able to justify before them learning new language.
2) Wine targets problem "Compilation and execution of Win32 applications". C is the native language of this problem. All the problems and solutions can be easy expressed in C and require translation when converted to other languages. All the documentation, learning materials are C-oriented. BTW, tests in Perl address only execution of applications, not compilation.
The biggest advantage of scripting language in this application is easiness of installation. Most people who know or want to learn Win32 API already have to use some kind C development environment.
I'd prefer to lose 10 developers who don't want to learn C instead of losing one expert in Win32 API who thinks that Perl sucks ;-)
Andriy Palamarchuk
One more point in favour of C-based tests. They will be very useful in porting Wine to non-Intel platforms. C tests will help to test both - execution of Windows applications under processor emulator and compilation of Windows applications with Winelib.
- It is much easier to install under Windows than a
full development
environment, and we don't have to worry about
supporting a dozen
different compilers. We can simply provide a zip
file containing the
compiled script interpreter, and people can be up
and running in seconds.
We can create MinGW toolchain distribution, customized for the test application. Besides original MinGW tools (about 32Mb) this distribution can include command-line, GUI CVS clients, GUI test application launcher. Can you suggest any other tools?
The launcher will have functionality to run Wine test application, update it from CVS, build, create patch. Such distribution is mostly newbie-oriented.
With script-based tool it is still necessary to install CVS separately or we need to create our own distribution.
Another advantage of C-based tests is that they provide smooth path from tests creation to contributing to core Wine project.
Thanks, Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
Andriy Palamarchuk apa3a@yahoo.com writes:
It seems the problem is not so big. It will be sufficient to run the binary, compiled under Windows not more often than once a month.
But it won't compile. Once we have a simple environment in Wine where you run make test and everything happens automatically, people will use that. They won't bother to update all the makefiles etc. that you need in order to build on Windows. Then every time someone tries to build the tests under Windows they will have to fix a ton of problems before it works.
We simply cannot expect people to constantly dual-boot to run their tests in both environments, so we need a way to make sure that when code works on one platform it also works on the other without extra work. We could certainly build a Windows infrastructure that does everything automatically for C tests, but this is a massive amount of work.
BTW, tests in Perl address only execution of applications, not compilation.
Compilation is not an interesting case. 99% of it is tested by compiling Wine itself, and the remaining occasional problem is trivial to locate and fix. There's simply no need for regression testing of the compilation environment.
Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
It seems the problem is not so big. It will be sufficient to run the binary, compiled under Windows not more often than once a month.
But it won't compile. Once we have a simple environment in Wine where you run make test and everything happens automatically, people will use that. They won't bother to update all the makefiles etc. that you need in order to build on Windows. Then every time someone tries to build the tests under Windows they will have to fix a ton of problems before it works.
We simply cannot expect people to constantly dual-boot to run their tests in both environments, so we need a way to make sure that when code works on one platform it also works on the other without extra work. We could certainly build a Windows infrastructure that does everything automatically for C tests, but this is a massive amount of work.
Before I was confident the tests would be developed under Windows and then run under Wine. You described reverse situation. To create a test people will have to use Windows to check it works properly. Of course all the tests must succeed under Windows.
BTW, tests in Perl address only execution of applications, not compilation.
Compilation is not an interesting case. 99% of it is tested by compiling Wine itself, and the remaining occasional problem is trivial to locate and fix. There's simply no need for regression testing of the compilation environment.
You are right for current conditions. The situation may be more interesting if Wine will be ported to non-Intel architecture.
Ok, this week I'll create a few tests in Perl and share my unbiased experience :-)
Andriy
Andriy Palamarchuk apa3a@yahoo.com writes:
Before I was confident the tests would be developed under Windows and then run under Wine. You described reverse situation.
To create a test people will have to use Windows to check it works properly. Of course all the tests must succeed under Windows.
In theory tests should be written under Windows yes. In practice the initial version of a test may be done on Windows, but I'm sure people will then modify the test under Wine without bothering to retry under Windows every time.
--- Alexandre Julliard julliard@winehq.com wrote:
In theory tests should be written under Windows yes. In practice the initial version of a test may be done on Windows, but I'm sure people will then modify the test under Wine without bothering to retry under Windows every time.
What is the point of having Win32 API unit test which does not confirm to the API reference implementation?!
I agree that some of developers don't have Windows installed, don't have time to recompile the tests under Windows, etc, but Win32 confirmance is purpose of the whole project. No big qualification is required to fix confirmance problems and a few volunteers with access to different Windows platforms can perform this task.
Andriy
__________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com
-
Alexandre Julliard -
Andreas Mohr -
Andriy Palamarchuk -
Dan Kegel -
David Elliott -
Francois Gouget -
Geoff Thorpe -
Jeremy White -
Robert Baruch -
Sylvain Petreolle -
Ulrich Weigand