Hello,
recent discussion on the list ignited my interest again. Francois has a very nice page http://fgouget.free.fr/wine/tests-en.shtml, let's try to make better use of it!
1. The precompiled binaries offered here would mean a great help, were they updated regularly.
2. Maybe better, a framework could be developed which would run all the tests, pack the results up and send it back to Francois for easy integration into his page.
3. The table on the page is too big for convenient use. I do not know about much structure to see; maybe a table is not even needed. Just version, date of test, if it is up to date with the binaries on the page, and a list of failing tests as links to the error messages would be enough. If it is not, we can still embed it into the giant API database at http://24.229.94.2 (why is it not linked from WineHQ?) and have a general SQL interface! :) Hmm, some diffing capabilities might be useful indeed...
4. We could recruit people with access to the different platforms and send them mails asking for a rerun whenever the new tests are compiled. Or advertise it on the devel list only...
What do you think? I am even willing to do some of the above, except for the build, as I have no constant access to Windows machines. Feri.
On Thu, 21 Aug 2003, Ferenc Wagner wrote:
recent discussion on the list ignited my interest again. Francois has a very nice page http://fgouget.free.fr/wine/tests-en.shtml, let's try to make better use of it!
Good, we need a new infusion of enthusiasm for this project :)
- The precompiled binaries offered here would mean a great help, were they updated regularly.
To achive this I think we need to refine the process to the point where we build one file (an .exe), which is downloaded automatically to some address. When we get there, we can enlist a bunch of people to regularly run this process. But if we require more manual intervention from them, they will do it a few times and give up.
- Maybe better, a framework could be developed which would run all the tests, pack the results up and send it back to Francois for easy integration into his page.
Yes! I proposed such a shell some time ago. Look for "Winetest Shell" at the bottom of the page: http://winehq.org/site/fun_projects
- The table on the page is too big for convenient use. I do not know about much structure to see; maybe a table is not even needed. Just version, date of test, if it is up to date with the binaries on the page, and a list of failing tests as links to the error messages would be enough. If it is not, we can still embed it into the giant API database at http://24.229.94.2 (why is it not linked from WineHQ?) and have a general SQL interface! :) Hmm, some diffing capabilities might be useful indeed...
Hope you're kiddin', right? :) We should simplify things, rather than complicate them. In general, all these entries should be green, we a few red ones. The red ones should be hyperlinks to a page listing all the problems (a cute little popup window would be nice). The cells in the big table should be rather small, so that everything fits on the screen.
- We could recruit people with access to the different platforms and send them mails asking for a rerun whenever the new tests are compiled. Or advertise it on the devel list only...
See 1. above...
"Dimitrie O. Paun" dimi@intelliware.ca writes:
On Thu, 21 Aug 2003, Ferenc Wagner wrote:
- The precompiled binaries offered here would mean a great help, were they updated regularly.
To achive this I think we need to refine the process to the point where we build one file (an .exe)
I understand that you are strongly attached to C and heavily biased against scripting languages ==> :) <==, but what I have in mind is a simple .BAT file. That even I could generate by a Perl script (sorry again!) from the build tree. Just zip it up with the individual .EXE's and let go. Extract into a directory, run TESTS.BAT, email RESULTS.TXT, delete directory. Is it still too much?
we can still embed it into the giant API database at http://24.229.94.2 (why is it not linked from WineHQ?) and have a general SQL interface! :)
Hope you're kiddin', right? :) We should simplify things, rather than complicate them.
Right. :) See the :). :) But the question is serious.
In general, all these entries should be green, with a few red ones.
In the spirit of the above, I proposed omitting (gasp!) the green ones... Is that an oversimplification? I have to admit http://www.astro.gla.ac.uk/users/paulm/WRT/wrt.php looks very nice. It is a pity it went dead two days before I added a new test.
Your comments are most welcome! Feri.
Hi Ferenc,
On Fri, 22 Aug 2003, Ferenc Wagner wrote:
In the spirit of the above, I proposed omitting (gasp!) the green ones... Is that an oversimplification? I have to admit http://www.astro.gla.ac.uk/users/paulm/WRT/wrt.php looks very nice. It is a pity it went dead two days before I added a new test.
Such is life :)
Yes, its a bit dead at the moment, but I'm working on bringing it back to life again. More news soon (hopefully).
---- Paul Millar
On Thu, 21 Aug 2003, Ferenc Wagner wrote:
Hello,
recent discussion on the list ignited my interest again. Francois has a very nice page http://fgouget.free.fr/wine/tests-en.shtml, let's try to make better use of it!
- The precompiled binaries offered here would mean a great help, were they updated regularly.
Absolutely true. Unfortunately I have not had time to update them for quite some time :-(. It would be great if someone volunteered.
You suggested a script to generate a zip file automatically. I have a couple of trivial scripts that should do 90% of the job.
* zipmsvctests Assumes that you compiled the tests on Wnidows using MSVC somehow. Then, as its name implies, it just zips them all together and throws in a batch file to run all the tests.
* runtests.bat Simply runs all the tests.
- Maybe better, a framework could be developed which would run all the tests, pack the results up and send it back to Francois for easy integration into his page.
I think a database or anything complex is way overkill. All we need is to FIX the BUGS and then it will all be trivial to manage.
Most of the complexity is that currently there are way too many bugs. This makes it hard to update the status page, and makes the table way too big. If all tests just compiled and succeeded, and we only had to deal with the occasional new test then results would be easy to merge and everything would fit in the page.
Francois Gouget fgouget@free.fr writes:
You suggested a script to generate a zip file automatically. I have a couple of trivial scripts that should do 90% of the job.
- zipmsvctests
Exactly what I envisioned. What do you think about putting it into the Wine CVS tree so that anyone compiling the tests could use it? Into tools/ for example?
- runtests.bat
Maybe we could do a bit more here, like redirecting the output into a file and echoing back a counter between the tests as a kind of "progress bar". I could probably write a Perl script to generate the above scripts, but I am not sure if it is wise to require Perl from the good soul trying to compile the tests for us. So maybe we are better including the results (as well) in the tree.
All we need is to FIX the BUGS and then it will all be trivial to manage.
Golden words! Feri.
"Dimitrie O. Paun" dimi@intelliware.ca writes:
On Wed, 27 Aug 2003, Jakob Eriksson wrote:
I really was looking forward to write a testing application that runs all the tests.
Yes, that would be very nice indeed. Such a test shell should be something like the JUnit stuff. Here are few things that I'd want in such an app: [...]
Well, much less than what you want, but maybe something:
http://afavant.elte.hu/~wferi/wine
Basically, I cross-compiled the tests I could (all except dsound), and run the attached script, which created the zip file on the above page. A testing soul have to create a directory, download a file, unzip it, run a .BAT file, and mail a .TXT file. Cleanup is optional. I will write a script which digests the resulting textfile and renders some HTML. What do you think? Feri.
On Thu, 28 Aug 2003, Ferenc Wagner wrote:
Basically, I cross-compiled the tests I could (all except dsound), and run the attached script, which created the zip file on the above page. A testing soul have to create a directory, download a file, unzip it, run a .BAT file, and mail a .TXT file. Cleanup is optional. I will write a script which digests the resulting textfile and renders some HTML. What do you think?
Very nice. In fact, this is good enough to bridge us to when winetests will be ready. Esentially, winetests should just present a nice user interface and the extract/run/email/cleanup automatically.
So yeah, now the focus should be in post-processing these results. Also, Jeremy can we setup a mailing lists/alias to where the results should be sent? This way we have a well-known entry point (e.g. wine-tests-results@winehq.org) and we've broken down the problem into simpler parts: -- creating the testing bundle (we'll need to integrate that in our build process, and regularly post fresh binary on our SF download site) -- the generation of the results is covered 80% by Feri's script, and work is underway to create a nice UI for it. -- post-processing the results, and generation of a nice HTML page (this needs integrating on WineHQ as well).
On Thu, 2003-08-28 at 16:47, Dimitrie O. Paun wrote:
So yeah, now the focus should be in post-processing these results. Also, Jeremy can we setup a mailing lists/alias to where the results should be sent? This way we have a well-known entry point (e.g. wine-tests-results@winehq.org) and we've broken down the problem into simpler parts:
Sure, if that is the address you want for the list I can probably put it up tomorrow sometime. I just want to be very sure before I start work on it.
-- post-processing the results, and generation of a nice HTML page (this needs integrating on WineHQ as well).
Not sure how you want to accomplish this, but integrating it into the site should not be hard, if the output is generated using the .template spec of the lostwages engine. If it need to be more dynamic than that, some PHP work will be involved.
On August 28, 2003 06:30 pm, Jeremy Newman wrote:
Sure, if that is the address you want for the list I can probably put it up tomorrow sometime. I just want to be very sure before I start work on it.
Well, wine-tests-results@winehq.org is OK AFAICT. Are there better proposals?
Not sure how you want to accomplish this, but integrating it into the site should not be hard, if the output is generated using the .template spec of the lostwages engine. If it need to be more dynamic than that, some PHP work will be involved.
No need for PHP -- the page will be static HTML generated externally through a Perl script I'd guess. We'll have to figure a way of integrating the above list with such a script. I'm wondering if we don't need a DB in the middle, something like so:
wine-tests-results --> parser --> DB --> generator --> .template
Something like MySQL should do, but it's a non-trivial setup. We need to figure out a few things: -- how do we trigger the parser based on email comming in from the list. Maybe run it as a user, and have things done through procmail. -- Jeremy, are you OK with setting up a simple database for this stuff? If so, we need to figure out a schema. -- We need to run the generator either from time to time, or when the DB gets updated. Any preference? -- Once you have the .template, what does it need to happen to have it appear on WineHQ?
On Fri, 2003-08-29 at 07:35, Dimitrie O. Paun wrote:
On August 28, 2003 06:30 pm, Jeremy Newman wrote:
Sure, if that is the address you want for the list I can probably put it up tomorrow sometime. I just want to be very sure before I start work on it.
Well, wine-tests-results@winehq.org is OK AFAICT. Are there better proposals?
Not sure how you want to accomplish this, but integrating it into the site should not be hard, if the output is generated using the .template spec of the lostwages engine. If it need to be more dynamic than that, some PHP work will be involved.
No need for PHP -- the page will be static HTML generated externally through a Perl script I'd guess. We'll have to figure a way of integrating the above list with such a script. I'm wondering if we don't need a DB in the middle, something like so:
wine-tests-results --> parser --> DB --> generator --> .template
Something like MySQL should do, but it's a non-trivial setup. We need to figure out a few things: -- how do we trigger the parser based on email comming in from the list. Maybe run it as a user, and have things done through procmail.
It can work similar to how the mailman works, you have the script parse from /etc/aliases. .procmail would work as well.
-- Jeremy, are you OK with setting up a simple database for this stuff? If so, we need to figure out a schema.
Mysql is already on the machine, so that is fine. My concern is DB bloat over time. Will this DB get out of control or will it remain a fixed size?
-- We need to run the generator either from time to time, or when the DB gets updated. Any preference?
Cron, or during the monthly build. Monthly build is preferred IMO. I'm jumpy about giving out any further shell access to the box.
-- Once you have the .template, what does it need to happen to have it appear on WineHQ?
Just needs to be in the templates/en dir somewhere, it can be another level deep as well. I can provide the full path to whoever writes this script.
On 29 Aug 2003, Jeremy Newman wrote:
-- Jeremy, are you OK with setting up a simple database for this stuff? If so, we need to figure out a schema.
Mysql is already on the machine, so that is fine. My concern is DB bloat over time. Will this DB get out of control or will it remain a fixed size?
I will grow, as new tests are added to the database. We can have a monthly cleanup task to get rid of old tests (and maybe remember some fun stats while we're at it).
-- We need to run the generator either from time to time, or when the DB gets updated. Any preference?
Cron, or during the monthly build. Monthly build is preferred IMO. I'm jumpy about giving out any further shell access to the box.
There's no need for shell access -- we just need to trigger the scrip. Doing it once per month is a bit too seldom. We can run the script every day, and make it smart to not do anything if the DB has not been updated. This is easy to do.
-- Once you have the .template, what does it need to happen to have it appear on WineHQ?
Just needs to be in the templates/en dir somewhere, it can be another level deep as well. I can provide the full path to whoever writes this script.
Perfect. As long as we don't have to put it in CVS, things are simple. The best would be for the script to just output to stdout, and you redirect that where you need to when you install it.
Hello guys,
some machinery is already in place (http://afavant.elte.hu/~wferi/wine). Actually I am not sure we need a database or so. What I have in mind:
Whenever somebody compiles the tests (with a Samba mount it does not matter whether cross- os MSVC) s/he runs the ziptests Bash script to produce a tests.zip, and we collect it somewhere tagged by the date, eg. 20030828.
People download it, do their job, and send the report to the list.
If the report is identical (modulo a couple of lines which contain the current working directory and similar -- just egrep -v it) to another for the same build and Windows version, we just acknowledge the name. If it is always the case, then forget it. If not, we may need a subarchitecture or a better egrep pattern, this is manual.
For a new set of data, we run the dissect Perl script, which gives a summary and one report file for each test. No need to rerun this ever.
The gather script assembles the summaries for the various architectures into a summary table, with links to the small report files. Rerun when a new architecture is added for this build.
We can have on directory for each build tag (like 20030828), and separate directories for each architecture in them.
gather dissect works here works here
,- win95 | ,- 20030828 -+-- nt4.0 | | | `- XP index.html -+ | ,- win98 | | `- 20030914 -+-- nt4.0 | `- XP
We can also have a 'latest' directory which simply links to the appropriate architecture subdirectories, and we can also gather a summary there. All the pages are static, and you can select on the main page.
Find the scripts attached. I do not really want to go further until I know where/how they will run. The desing is not finished :) Feri.
On Fri, 29 Aug 2003, Dimitrie O. Paun wrote:
On August 28, 2003 06:30 pm, Jeremy Newman wrote:
Sure, if that is the address you want for the list I can probably put it up tomorrow sometime. I just want to be very sure before I start work on it.
Well, wine-tests-results@winehq.org is OK AFAICT. Are there better proposals?
[...]
No need for PHP -- the page will be static HTML generated externally through a Perl script I'd guess. We'll have to figure a way of integrating the above list with such a script. I'm wondering if we don't need a DB in the middle, something like so:
wine-tests-results --> parser --> DB --> generator --> .template
Something like MySQL should do, but it's a non-trivial setup.
Do we really need a C program, a new mailing list, automated result submission, generating HTML and now a database just so we have something that says:
Test | Result ----------+--------- kernel32 | Passed msvcrt | Passed oleaut32 | Passed dsound | Passed ...
This all seems very much over-engineered to me. I'd really like to see that kind of energy spent on fixing the tests so that they actually work on Windows.
Francois Gouget fgouget@free.fr writes:
Do we really need a C program, a new mailing list, automated result submission, generating HTML and now a database just so we have something that says:
Test | Result ----------+--------- kernel32 | Passed msvcrt | Passed oleaut32 | Passed dsound | Passed ...
We want a little bit more than this... but I think you are right.
This all seems very much over-engineered to me. I'd really like to see that kind of energy spent on fixing the tests so that they actually work on Windows.
I would like to. I spent some patches against the tests, and they were refused because somebody remembered something. Now I need FACTS, and since nobody seemed to provide me with facts, I try to do something about that. I still can not say that results are pouring in :)
Now I also produced Makefiles, by the way... Feri.
Francois Gouget wrote:
Do we really need a C program, a new mailing list, automated result submission, generating HTML and now a database just so we have something that says:
Test | Result ----------+--------- kernel32 | Passed msvcrt | Passed oleaut32 | Passed dsound | Passed ...
This all seems very much over-engineered to me. I'd really like to see that kind of energy spent on fixing the tests so that they actually work on Windows.
Well, a new mailing list seems overkill to me. Since yesterday I have been working on a program to run the tests. I have it more than 50% done now. It runs all the tests and logs everything in a file. The test is one big EXE for the Windows tester to download. The reporting is uploaded automatically if the user wants to. The EXE is built with mingw crosscompiler, so no need for MSVC.
regards, Jakob
On August 29, 2003 07:47 pm, Jakob Eriksson wrote:
Well, a new mailing list seems overkill to me.
I don't see what the big deal is -- Jer can set one up in a few minutes, certainly much faster/easier to maintain than setting up some sort of download area. Moreover, it allows everybody interested in following these reports in subscribing to it. And all this for free.
Since yesterday I have been working on a program to run the tests. I have it more than 50% done now.
Nice -- I'll mark you down on the page as working on this.
It runs all the tests and logs everything in a file. The test is one big EXE for the Windows tester to download. The reporting is uploaded automatically if the user wants to. The EXE is built with mingw crosscompiler, so no need for MSVC.
Very nice. Where do you currently upload? If the email is a problem in terms of coding, Jon Bright donated his SMTP code... As was stated on the list, if SoBig could do it, so should we, no? :)
Dimitrie O. Paun wrote:
On August 29, 2003 07:47 pm, Jakob Eriksson wrote:
Well, a new mailing list seems overkill to me.
I don't see what the big deal is -- Jer can set one up in a few minutes, certainly much faster/easier to maintain than setting up some sort of download area. Moreover, it allows everybody interested in following these reports in subscribing to it. And all this for free.
I change my mind. I want a new mailing list! :-) What is the name of it?
Nice -- I'll mark you down on the page as working on this.
Thank you.
Very nice. Where do you currently upload? If the email is a problem in terms of coding, Jon Bright donated his SMTP code... As was stated on the list, if SoBig could do it, so should we, no? :)
I upload to my server at vmlinux.org
regards, Jakob
On August 30, 2003 05:59 am, Jakob Eriksson wrote:
I change my mind. I want a new mailing list! :-) What is the name of it?
Cool :-) It's wine-tests-results@winehq.org Jer, can you set it up please when you get a chance?
On Sat, 2003-08-30 at 09:27, Dimitrie O. Paun wrote:
On August 30, 2003 05:59 am, Jakob Eriksson wrote:
I change my mind. I want a new mailing list! :-) What is the name of it?
Cool :-) It's wine-tests-results@winehq.org Jer, can you set it up please when you get a chance?
http://www.winehq.org/mailman/listinfo/wine-tests-results
On Tue, 26 Aug 2003, Ferenc Wagner wrote:
Francois Gouget fgouget@free.fr writes:
You suggested a script to generate a zip file automatically. I have a couple of trivial scripts that should do 90% of the job.
- zipmsvctests
Exactly what I envisioned. What do you think about putting it into the Wine CVS tree so that anyone compiling the tests could use it? Into tools/ for example?
I would not mind but see below.
- runtests.bat
Maybe we could do a bit more here, like redirecting the output into a file and echoing back a counter between the tests as a kind of "progress bar".
Possibly but remember that batch files are erally restricted in what they can do. The batch file also has to run on both Win9x and NT*.
I could probably write a Perl script to generate the above scripts, but I am not sure if it is wise to require Perl from the good soul trying to compile the tests for us. So maybe we are better including the results (as well) in the tree.
I think that's the way to go. Anyone wishing to compile the tests with MSVC already needs to use perl scripts to generate the Visual Studio project files. These scripts could probably easily be modified to generate the above batch files too. That would be much more maintainable than putting hard-coded scripts into Wine.
Then we would run the script on a Unix machine and upload them some place so that Windows users can get them and build without installing perl.
All we need is to FIX the BUGS and then it will all be trivial to manage.
Golden words!
Yep, and it should not even be too hard since it's only the bugs in the tests that we need to fix in this case.
Ferenc Wagner wrote:
What do you think? I am even willing to do some of the above, except for the build, as I have no constant access to Windows machines.
The build should be done on linux, with "make crosstest",
(See http://bugs.winehq.com/show_bug.cgi?id=1602 )
regards, Jakob
Jakob Eriksson jakob@vmlinux.org writes:
Ferenc Wagner wrote:
What do you think? I am even willing to do some of the above, except for the build, as I have no constant access to Windows machines.
The build should be done on linux, with "make crosstest", (See http://bugs.winehq.com/show_bug.cgi?id=1602 )
Even if "make crosstest" were not broken, I can remember it produce tests which ran on NT based systems only. It was never confirmed by anybody else, though. And I am also a little uncertain if that would mean real conformance testing. Feri.
Ferenc Wagner wrote:
The build should be done on linux, with "make crosstest", (See http://bugs.winehq.com/show_bug.cgi?id=1602 )
Even if "make crosstest" were not broken, I can remember it produce tests which ran on NT based systems only. It was
I am pretty sure I have run such tests under Win95 before I killed my Win95- installation. Maybe I am mistaken.
never confirmed by anybody else, though. And I am also a little uncertain if that would mean real conformance testing.
Is the EXE behaving the same under Wine and Windows? That is what matters, not how it was generated.
regards, Jakob
On Sun, 24 Aug 2003, Ferenc Wagner wrote: [...]
Even if "make crosstest" were not broken, I can remember it produce tests which ran on NT based systems only. It was never confirmed by anybody else, though.
I had problems running the cross-compiled tests on Win9x too while they did work on NT4. I don't know why. Maybe the executables generated by MinGW call NT-only functions in their startup code?
Now the cross-compilation is broken as has been reported a few times. :-(
I don't know how to fix that though. I used to compile using MSVC which was a good test of Winelib issues (mostly header problems).
-
Dimitrie O. Paun -
Dimitrie O. Paun -
Ferenc Wagner -
Francois Gouget -
Jakob Eriksson -
Jeremy Newman -
Paul Millar