Spent a couple of days trying to port my test to C unit testing framework "Check".
Simple tests work fine. Problems start when I try to use multithreaded Wine tests. In one case the framework stucks on reading from pipe. Other code layouts have other issues. The cause may be in incorrect pipes handling, multi-process tests handling or interaction with Wine thread implementation.
I remember I also had some problems with Perl multithreaded tests...
I attached my unit test example, almost identical to the one I published before for Perl framework. There is also a whole working directory. If you want to play with the it, just unpack it to wine/programs directory.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Andriy Palamarchuk apa3a@yahoo.com writes:
Simple tests work fine. Problems start when I try to use multithreaded Wine tests. In one case the framework stucks on reading from pipe. Other code layouts have other issues. The cause may be in incorrect pipes handling, multi-process tests handling or interaction with Wine thread implementation.
I'm not sure if this is what happens, but if two threads try to send results down the same pipe you'll have problems. You need explicit threading support in the framework if you want to create threads (the Perl framework doesn't support threads either at this point, so if your test works there it's by pure luck).
Another potential problem is that the fork() option of the framework is going to cause major breakage. You cannot use fork inside a Winelib app. And in any case I don't think a GPLed framework is appropriate for Wine.
Hmm, I don't see my post in wine-devel. Alexandre, did I send the message directly to you? If yes, could you, please, forward to the mailing list? Thank you.
Alexandre Julliard wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
Simple tests work fine. Problems start when I try to use multithreaded Wine tests. In one case the framework stucks on reading from pipe. Other code layouts have other issues. The cause may be in incorrect pipes handling, multi-process tests handling or interaction with Wine thread implementation.
I'm not sure if this is what happens, but if two threads try to send results down the same pipe you'll have problems.
I think there is no competition for the pipe. New pipe is opened for each test. It looks like not all handles are closed to the write side of the pipe and read hangs. Can it happen because the handle is inherited to wineserver?
You need explicit threading support in the framework if you want to create threads (the Perl framework doesn't support threads either at this point, so if your test works there it's by pure luck).
:-( I need more than one thread to catch message on system parameter value change.
Another potential problem is that the fork() option of the framework is going to cause major breakage. You cannot use fork inside a Winelib app.
I really like memory protection which is provided by "fork()". We are going to need it (e.g. for TODO tests) :-/ The framework can be used in forkless mode, however we still need to remove that piping stuff.
Can we have the same advantages by using two-stage launching process? The first part coordinates everything and starts each test with "exec". Such implementation is less efficient but much more portable.
And in any case I don't think a GPLed framework is appropriate for Wine.
a) this is license not for Wine, but for Wine testing application. From my point of view GPL is very appropriate in this case. Any other OS projects, using the test application will have to contribute the changes back. On other hand the license won't hurt any commercial application because nobody will make business on it.
b) on the Check mailing list I saw request to change license to less restrictive one.
The main developer of the framework took into account this request. I also asked for less restrictive license, posted a couple bug reports, but have not had any feedback yet.
I have an idea for compromise - let's use both - Perl and C tests. Perl module Test::Harness can be used to manage both types of the tests. C test application can print feedback information in format Test::Harness needs. A library can be created for this. This gives all the advantages of both tools and choice, do not impose big constraints on developers. Test::Harness is very mature framework and has clearly defined interface.
I'm still for keeping all the tests as separate application. It is very easy to create test hierarchy for tests application if you want to have subsets of the tests.
Andriy Palamarchuk
Andriy Palamarchuk apa3a@yahoo.com writes:
:-( I need more than one thread to catch message on system parameter value change.
Not really, if you create a window your winproc will be called even if it's in the same thread. But we need to make threading work in any case.
I really like memory protection which is provided by "fork()". We are going to need it (e.g. for TODO tests) :-/ The framework can be used in forkless mode, however we still need to remove that piping stuff.
Can we have the same advantages by using two-stage launching process? The first part coordinates everything and starts each test with "exec". Such implementation is less efficient but much more portable.
IMO that's what we want, and the next step is to recognize that a simple shell script and a couple of makefile rules can do the "exec" job just as well, without having to worry about pipes or fork or whatever. I frankly don't see the need of a complex test harness to launch tests and print reports. Maybe when we have 1000 tests we will want some of that complexity, but for now it's only a waste of time.
a) this is license not for Wine, but for Wine testing application. From my point of view GPL is very appropriate in this case. Any other OS projects, using the test application will have to contribute the changes back. On other hand the license won't hurt any commercial application because nobody will make business on it.
Let's not debate licenses again. Anything that is included in the Wine distribution has to be under the Wine license. If you want a GPLed test suite you'll have to distribute it separately from Wine; but I think that would be a mistake.
--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
I frankly don't see the need of a complex test harness to launch tests and print reports. Maybe when we have 1000 tests we will want some of that complexity, but for now it's only a waste of time.
Almost all complexity you are talking about is already implemented for us. Usage of the framework is very simple, do not require from test writers anything. They are only required to correctly use Test::Simple (or Test::More). They don't need to remember about Test::Harness.
There are reasons to use Test::Harness: 1) control of TODO tests - I really want to use this feature. 2) control of SKIP tests - very useful for Wine-specific tests, choosing behavior, depending on Windows versions, etc. I need this feature too. 3) we already need to manage the test output. I'd estimate number of checks for my existing SystemParametersInfo unit test as: 25 (number of implemented actions) * 10 (minimal number of checks for each action) = 250 - 350 tests We'll definitely have huge number of tests. Why not pick up scaleable approach from very beginning?
Let's not debate licenses again. Anything that is included in the Wine distribution has to be under the Wine license. If you want a GPLed test suite you'll have to distribute it separately from Wine; but I think that would be a mistake.
IMHO you can distribute them together if these are 2 separate applications, right? I don't have big preferences about the license. Wine license will work fine.
We spent a lot of our valuable time and resources on the issue. Lets finalize the decisions and start to work.
Suggest decisions from the discussion: 1) unit tests are very important for the project 2) mixed C/Perl environment will be used for the tests development. Choosing the tool is a matter of personal preferences.
Comment: Alexandre, basing on the discussion above I think we both came to this. Do I understand you correctly? TODO: - create support in the build process
3) Test::Harness will be used to compile report for test batches
Comment - TODO: - fix problems in using Test::Harness (and some other modules, BTW) with winetest Perl framework - create simple framework for C to print report in format, suitable for Test::Harness. (I already started to work on this.) - change Test::Harness to run not only Perl scripts, but also other types of executables (can do this)
4) The unit test will be a separate application
Comment: Alexandre, we explicitely did not agree on this decision yet. You preferred to have unit tests spreaded over the Wine directory tree. The main argument for this was possibility of running subsets of test. I suggest to change Test::Harness (or create new module, based on it) to use tree-like tests organization. If you need to test module wine/dll/foo, you request to run tests hierarchy "wine/dll/foo".
TODO: - customize Test::Harness to use tree-like test suits structure (I can do this).
5) Existing Wine license will be used for the Unit test application
Please, let me know if I missed something or your point of view is not reflected.
Looking forward for your comments. Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Andriy Palamarchuk apa3a@yahoo.com writes:
Almost all complexity you are talking about is already implemented for us. Usage of the framework is very simple, do not require from test writers anything. They are only required to correctly use Test::Simple (or Test::More). They don't need to remember about Test::Harness.
But adapting the framework to do what we want is IMO more work than simply reimplementing from scratch the few features that we actually need. We don't really gain anything by making the effort of reusing all that code, since we don't need most of it.
There are reasons to use Test::Harness:
- control of TODO tests - I really want to use this
feature. 2) control of SKIP tests - very useful for Wine-specific tests, choosing behavior, depending on Windows versions, etc. I need this feature too.
Yes, I agree we want that. I think there are easy to implement no matter what we use, we don't really need Test::Harness for that.
- we already need to manage the test output. I'd
estimate number of checks for my existing SystemParametersInfo unit test as: 25 (number of implemented actions) * 10 (minimal number of checks for each action) = 250 - 350 tests We'll definitely have huge number of tests. Why not pick up scaleable approach from very beginning?
For me your SystemParametersInfo is one test, not 250. All I want to know if whether it passed or not, and if not what was the cause of the failure. I don't want to know about the details of the 250 individual checks.
Suggest decisions from the discussion:
- unit tests are very important for the project
- mixed C/Perl environment will be used for the tests
development. Choosing the tool is a matter of personal preferences.
I don't think I agree. For me the value of Perl is in that it makes it trivial to take the suite over the Windows; but if half the tests are in C we lose this advantage, and then we might as well do everything in C.
- Test::Harness will be used to compile report for
test batches
I don't see the need. What I want is a make-like system that keeps track of which tests have been run, which ones need to be re-run because they have been modified etc. I don't think there is any use in a report stating that 12.42% of the tests failed, this doesn't tell us anything.
- The unit test will be a separate application
You cannot put the whole test suite in a single application, you need to split things up some way. A decent test suite will probably be several times the size of the code it is testing; you don't want that in a single application.
Alexandre, we explicitely did not agree on this decision yet. You preferred to have unit tests spreaded over the Wine directory tree. The main argument for this was possibility of running subsets of test.
No, the argument is modularity. The tests for kernel32 have nothing to do with the tests for opengl, and have everything to do with the kernel32 code they are testing. So it seems logical to put them together.
Then when you change something in kernel32 you can change the test that is right next to it, run make test in the kernel32 directory and have it re-run the relevant tests, and then do cvs diff dlls/kernel32 and get a complete diff including the code and the test changes.
--- Alexandre Julliard julliard@winehq.com wrote:
Andriy Palamarchuk apa3a@yahoo.com writes:
[...]
But adapting the framework to do what we want is IMO more work than simply reimplementing from scratch the few features that we actually need. We don't really gain anything by making the effort of reusing all that code, since we don't need most of it.
Could you, please, list the additional features we need? I'll try to estimate amount of work necessary to implement them in Test::Harness.
Do you want to use architecture, completely different from Test::Harness + Test::Simple modules or you only want to replace Test::Harness?
Existing architecture: Individual test scripts (executables) are very simple (use Test::Simple). They only print messages like "ok 4 - action SPI_FOO # TODO not implemented" for each check. Test::Harness module parses output, has logic to analyze test results, creates overall report and details of failures, including crashes.
I don't want to know about the details of the 250 individual checks.
You got an impression that Test::Harness does not report individual failures. Sorry, I gave too simple demo.
Example of Test::Harness output for a few failures:
test2.p.............NOK 2# Failed test (test2.pl at line 8) # got: '1' # expected: '0' test2.p.............NOK 3# Failed test (test2.pl at line 9) # got: '1' # expected: '0' test2.p.............NOK 4# Failed test (test2.pl at line 10) test2.p.............ok 8/8# Looks like you failed 3 tests of 8.
[... summary report output is skipped ...]
Does this output look closer to the one you want? Let me know if you need any other information.
- mixed C/Perl environment will be used for the
tests
development. Choosing the tool is a matter of
personal
preferences.
I don't think I agree. For me the value of Perl is in that it makes it trivial to take the suite over the Windows; but if half the tests are in C we lose this advantage, and then we might as well do everything in C.
Sorry, misinterpreted your statement that threads won't be used for tests in Perl. Could you give your vision when C is used?
What I want is a make-like system that keeps track of which tests have been run, which ones need to be re-run because they have been modified etc.
This usage of make is fine with me. I just want to separate unit tests from main code and have centralized control. You still can call subset of unit tests from the build process.
You cannot put the whole test suite in a single application, you need to split things up some way. A decent test suite will probably be several times the size of the code it is testing; you don't want that in a single application.
We were not careful about the terms. Yes, the tests will be bunch of the executables. I assumed these executables to be parts of one test application. Completely agree with you.
You preferred to have unit tests spreaded over the Wine directory tree. The main argument for this was possibility of running
subsets
of test.
No, the argument is modularity.
I'm all for modular unit tests and there are a few ways to divide the tests between modules.
Then when you change something in kernel32 you can change the test that is right next to it, run make test in the kernel32 directory and have it re-run the relevant tests, and then do cvs diff dlls/kernel32 and get a complete diff including the code and the test changes.
Agree, this is not so convenient for separate test application.
Separate unit tests application has its own advantages: 1) Separate distributions are used for Wine and the test applications. The only case when we want them together - Wine development under *nix. In all other cases we need only one of them. 2) The unit tests will be mostly developed under Windows. The unit tests build process has extra Windows compatibility requirements. 3) Having unit tests as a separate application we create possibilities to collaborate with other W32 implementation OS projects (ODIN comes to mind).
All the tasks we mentioned can be implemented with any approach. I believe the tasks you described will be not much more difficult with my approach, especially if directory tree has parallel structure.
Andriy Palamarchuk
__________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/
Andriy Palamarchuk apa3a@yahoo.com writes:
Could you, please, list the additional features we need? I'll try to estimate amount of work necessary to implement them in Test::Harness.
Basically the features I get with my 10-line makefile hack: ability to run tests for modified dlls and their dependencies (for instance if you change something in user32 and run make test it doesn't run tests for kernel32), ability to remember which tests failed and run only these the second time around, trivial integration of new tests (just add one line in a makefile).
Do you want to use architecture, completely different from Test::Harness + Test::Simple modules or you only want to replace Test::Harness?
I think we would be much better off developing a simple infrastructure from scratch that does exactly what we want, than trying to bend an existing framework to our needs. This will also ensure that the framework remains as simple as possible, which is important since every extra feature, even one we don't use, can possibly introduce problems (like the pipes/fork issues have shown).
Does this output look closer to the one you want? Let me know if you need any other information.
I don't really care about the output. My point is that this output is not necessary, a simple "assertion foo failed at line xx" on failure and no output at all on success would work just as well. I've nothing against such an output either, but I don't think it justifies introducing the complexity of reusing the Test stuff.
Sorry, misinterpreted your statement that threads won't be used for tests in Perl. Could you give your vision when C is used?
In my vision we either use Perl everywhere, or C everywhere. If we use Perl there may be a need for a few little glue programs in C, but this doesn't require any C infrastructure.
I personally think Perl would be a better choice, but I seem to be pretty much the only one of this opinion. In any case the most important thing is to choose something that people are going to use, and so far the Perl stuff isn't a success in this respect. I'm not convinced a C infrastructure would fare better, but maybe we should try it and see.
On 8 Jan 2002, Alexandre Julliard wrote:
I personally think Perl would be a better choice, but I seem to be pretty much the only one of this opinion.
Well, if random opinions count here, I also would prefer Perl. As much as I hate Perl (I'm more in the Python camp), I'd hate writing regression tests in C much more.
On Tue, 8 Jan 2002, Ove Kaaven wrote:
Well, if random opinions count here, I also would prefer Perl. As much as I hate Perl (I'm more in the Python camp), I'd hate writing regression tests in C much more.
(Another Pythoner, cool :) )
But if we accept tests in C, we don't loose anything. If it's a pain to do them in C, we'll end up with just a handful of them which we can simply convert to Perl (which should be trivial since they are just a few). On the other hand, if we end up with a lot of C tests, means that people simply prefer C over Perl (for whatever reason), so we would still have won since we end up with all those tests.
-- Dimi.
On Tue, 8 Jan 2002, Alexandre Julliard wrote:
I personally think Perl would be a better choice, but I seem to be pretty much the only one of this opinion. In any case the most important thing is to choose something that people are going to use, and so far the Perl stuff isn't a success in this respect. I'm not convinced a C infrastructure would fare better, but maybe we should try it and see.
And this is an excellent point. I, for one, know enough Perl to know that I find it ugly as bloody hell, and as such I have no desire to learn it. Writing tests is a big pain in the ass in the first place, and I can tell you I would not do it in Perl if I was paid to do it.
Now, I kept quiet on this issue because I can see the merits of using a scripting language to write said tests. However, I would like to point out that the _hard_ problem is getting the tests written, it doesn't really matter in what language. If the infrastructure (compiler, libs, etc) is not present on the current platform (configure is the 'man'), it is trivial enough to simply not run them (or run some dummied up tests instead, something like true(1)).
So, bottom line, I think you should accept whatever tests you get. If the author gets a woody writing them in C, or using some test harness or another, let's just consider it the motivating factor behind writing the tests in the first place.
-- Dimi.
"Dimitrie O. Paun" dimi@cs.toronto.edu writes:
So, bottom line, I think you should accept whatever tests you get. If the author gets a woody writing them in C, or using some test harness or another, let's just consider it the motivating factor behind writing the tests in the first place.
Unfortunately that's not possible. If everybody uses his favorite test harness we will soon have more of them than actual tests. It's already going to be enough work maintaining one framework and making sure it always works both on Windows and Wine, we can't afford to have several.
On Tue, 8 Jan 2002, Alexandre Julliard wrote:
Unfortunately that's not possible. If everybody uses his favorite test harness we will soon have more of them than actual tests.
Certainly, I was exaggerating. However, we should accept tests written in C.
-- Dimi.
On 8 Jan 2002, Alexandre Julliard wrote:
"Dimitrie O. Paun" dimi@cs.toronto.edu writes:
So, bottom line, I think you should accept whatever tests you get. If the author gets a woody writing them in C, or using some test harness or another, let's just consider it the motivating factor behind writing the tests in the first place.
Unfortunately that's not possible. If everybody uses his favorite test harness we will soon have more of them than actual tests. It's already going to be enough work maintaining one framework and making sure it always works both on Windows and Wine, we can't afford to have several.
I agree that we should not have as many testing frameworks as test writers. But maybe two would be acceptable. I propose the following:
* you posted the beginning of a perl test framework. What is needed before it can be committed to CVS? AFAICS all it lacks is support for difference files, aka TODO tests. * so let's complete that framework and commit it to CVS * then if someone really wants a C testing framework, let them develop it. And then we can replace
test: <run-perl-tests>
with:
ptests: <run-perl-tests> ctests: <run-c-tests> tests: ptests ctests
The perl test framework will need a way to build a zip file of some sort with all the necessary stuff to run the perl tests on Windows. All we need is for this to not be confused when we add the C tests. The C tests will need such a functionality too. And it should package just the C tests and not the perl tests. So we should be able to have perl and C tests side by side, in such a way that they basically ignore each other.
Then we'll see after a little while which framework is being used. And if one of them is never used, then as Dimitrie said, we can convert its tests and remove it.
-- Francois Gouget fgouget@free.fr http://fgouget.free.fr/ In a world without fences who needs Gates?
On Tue, 8 Jan 2002, Francois Gouget wrote:
The perl test framework will need a way to build a zip file of some sort with all the necessary stuff to run the perl tests on Windows. All we need is for this to not be confused when we add the C tests. The C tests will need such a functionality too. And it should package just the C tests and not the perl tests.
In fact, this should be possible in C as well. Say configure can check if gcc can generate PE executables, and if so, we can compile (in Linux/*BSD/etc) the C tests as PE executable so that we actually run the exact same binary in Wine and Windows.
From the testers POV they should be as easy to run as the Perl tests (if
not easier), and all that is required is a cross-compiling gcc. You don't have that, no big problem, you simply can not package the C tests.
-- Dimi.
-
Alexandre Julliard -
Andriy Palamarchuk -
Dimitrie O. Paun -
Francois Gouget -
Ove Kaaven