Hi,
(I've just remembered that there is a limit on the email size, so I'm doing it a bit differently now. The other email, including the screenshots, will probably come through as well)
Before sending this to wine-patches I would like people to have a look first.
This patch makes sure that missing tests are shown in the test results, both when there is only one system in a group (f.e. Win98) or multiple systems in a group.
On 'http://www.xs4all.nl/~pvriens/200708031000.old.single/' you can see the current state of the test results. When a test is missing a '.' is shown (this is of course different from a missing test where we don't have results at all but that's shown as a '.' as well).
On 'http://www.xs4all.nl/~pvriens/200708031000.old.group/' you can see the current state when a group of systems is involved. When tests are missing for a group, a '.' is shown with a link to the group overview. There is however no missing test output in the group overview (for example gdiplus and pdh)
'http://www.xs4all.nl/~pvriens/200708031000.new.single/' shows the new results page for single tests. Instead of a single '.' there is a failure shown (red box) and a skip border (as these tests have not run). There is still a '.' which could maybe better be a '?'. There is also a 'onMouseOver' hint when you go over the '.'.
'http://www.xs4all.nl/~pvriens/200708031000.new.group/' shows the new results when a group is involved. The main summary is still the same as in the old one, but the group results now also show the missing tests (see gdiplus and pdh again). Again there is a 'onMouseOver' hint in the group results.
Remember that this is still without winetest producing output about missing dll's.
Next patches will:
- add a missing dll as one of the reasons for missing tests - add some better clue's on the main summary for multiple systems in a group (not a single '.' but maybe the same 'failed and skipped').
As said, one of the thoughts I had was to replace the '.' with a '?' in case of missing tests for an unknown reason but I'm not sure about that yet.
Comments/remarks etc.. are welcome.
I'd also like to hear about other idea's for this page to make it useful for everybody. Apart from the fact that we probably should have more exposure to the fact that we have such a page/tool. We saw an immediate rise (slowed down again) in test results as soon as winetest was mentioned on 'http://tweakers.net/meuktracker/15817/Wine-0.9.42.html' (in Dutch). There it was also mentioned that some machines crashed because of the tests etc.. (maybe a wine-winetest component on bugzilla could help with that which we could reference on the main test.winehq.org/data/*/index.html pages).
Cheers,
Paul.
Paul Vriens wrote:
Hi,
(I've just remembered that there is a limit on the email size, so I'm doing it a bit differently now. The other email, including the screenshots, will probably come through as well)
Before sending this to wine-patches I would like people to have a look first.
This patch makes sure that missing tests are shown in the test results, both when there is only one system in a group (f.e. Win98) or multiple systems in a group.
On 'http://www.xs4all.nl/~pvriens/200708031000.old.single/' you can see the current state of the test results. When a test is missing a '.' is shown (this is of course different from a missing test where we don't have results at all but that's shown as a '.' as well).
On 'http://www.xs4all.nl/~pvriens/200708031000.old.group/' you can see the current state when a group of systems is involved. When tests are missing for a group, a '.' is shown with a link to the group overview. There is however no missing test output in the group overview (for example gdiplus and pdh)
'http://www.xs4all.nl/~pvriens/200708031000.new.single/' shows the new results page for single tests. Instead of a single '.' there is a failure shown (red box) and a skip border (as these tests have not run). There is still a '.' which could maybe better be a '?'. There is also a 'onMouseOver' hint when you go over the '.'.
'http://www.xs4all.nl/~pvriens/200708031000.new.group/' shows the new results when a group is involved. The main summary is still the same as in the old one, but the group results now also show the missing tests (see gdiplus and pdh again). Again there is a 'onMouseOver' hint in the group results.
Remember that this is still without winetest producing output about missing dll's.
Next patches will:
- add a missing dll as one of the reasons for missing tests
- add some better clue's on the main summary for multiple systems in a
group (not a single '.' but maybe the same 'failed and skipped').
As said, one of the thoughts I had was to replace the '.' with a '?' in case of missing tests for an unknown reason but I'm not sure about that yet.
Comments/remarks etc.. are welcome.
I'd also like to hear about other idea's for this page to make it useful for everybody. Apart from the fact that we probably should have more exposure to the fact that we have such a page/tool. We saw an immediate rise (slowed down again) in test results as soon as winetest was mentioned on 'http://tweakers.net/meuktracker/15817/Wine-0.9.42.html' (in Dutch). There it was also mentioned that some machines crashed because of the tests etc.. (maybe a wine-winetest component on bugzilla could help with that which we could reference on the main test.winehq.org/data/*/index.html pages).
Cheers,
Paul.
Forgot to mention that only the index.html file is on the mentioned pages. No result files (to cut down on size for me to push).
I also forgot to attach the patch. So here it is.
Cheers,
Paul
--- /wine/tools/winetest/gather 2007-08-03 17:20:09.000000000 +0200 +++ gather 2007-08-05 14:02:42.000000000 +0200 @@ -121,6 +121,22 @@ push @{$group->{tests}}, $testref; }
+# Find missing tests. After this exercise all testresults are available +# for all systems. +foreach my $group (@groups) { + next unless exists $group->{tests}; + foreach my $test (@{$group->{tests}}) { + foreach my $testname (sort keys %alltests) { + if (!exists $test->{results}->{$testname}) { + # Make sure missing tests are shown in the group results + $group->{digests}->{$testname} = "differ"; + # Mark this test as missing for an unknown reason + $test->{results}->{$testname} = ["test missing", "-", "-", "-"]; + } + } + } +} + # Write out the tables
my ($header); # same in thead and tfoot @@ -201,28 +217,30 @@ # Output a single cell of a test sub singletest { my ($test, $testname, $groupname) = @_; - - if (!exists $test->{results}->{$testname}) { - print OUT " <td class="note">.</td>\n"; + my $file = "$test->{dir}/$testname.txt"; + my ($count, $todo, $error, $skipped) = @{$test->{results}->{$testname}}; + if ($count eq "failed") { + my $msg = $todo eq "-"?"crash": + $todo == 258?"timeout":"failed"; + my $fail = -r "$datadir/$build/$file"?"<a href="$file">$msg</a>":$msg; + print OUT " <td class="note">$fail</td>\n"; + } elsif ($count eq "test missing") { + print OUT <<"EOF"; + <td class="skip_fail"><a + title="Test did not run for an unknown reason" + onMouseOver="Test did not run for an unknown reason"; + >.</a></td> +EOF } else { - my $file = "$test->{dir}/$testname.txt"; - my ($count, $todo, $error, $skipped) = @{$test->{results}->{$testname}}; - if ($count eq "failed") { - my $msg = $todo eq "-"?"crash": - $todo == 258?"timeout":"failed"; - my $fail = -r "$datadir/$build/$file"?"<a href="$file">$msg</a>":$msg; - print OUT " <td class="note">$fail</td>\n"; - } else { - my $class = $error?"fail":"pass"; - my $skip = $skipped?"skip_":""; - print OUT <<"EOF"; + my $class = $error?"fail":"pass"; + my $skip = $skipped?"skip_":""; + print OUT <<"EOF"; <td class="$skip$class"><a href="$file" title="$count tests, $todo TODO, $error errors, $skipped skipped" onMouseOver="refresh('$testname','$groupname $test->{tag}',$count,$todo,$error,$skipped);" >$error</a></td> EOF - } } }
Paul Vriens wrote:
Comments/remarks etc.. are welcome.
It is excellent that you are doing this, the reports are clearer for it. I will try to submit test results regularly.
regards, Jakob Eriksson
Jakob Eriksson wrote:
Paul Vriens wrote:
Comments/remarks etc.. are welcome.
It is excellent that you are doing this, the reports are clearer for it. I will try to submit test results regularly.
regards, Jakob Eriksson
Hi Jakob,
I think you ran a few win95 tests yesterday? What was wrong with for example advapi32crypt32/d3d8/d3d9 (none of them ran so it appeared)? Can you produce a list of things winetest complained about while extracting?
If it just says "Can't parse subtests output of *.dll", could you go to the wct directory (in %TEMP%) and execute the single *_test.exe? That should give a bit more information about why it can't get the subtests list.
Cheers.
Paul.
Jakob, Paul, we should have a regular site for uploading screenshots like these... so that google can find them too...
On 8/6/07, Jakob Eriksson jakob@vmlinux.org wrote:
Paul Vriens wrote:
Comments/remarks etc.. are welcome.
It is excellent that you are doing this, the reports are clearer for it. I will try to submit test results regularly.
regards, Jakob Eriksson
Paul Vriens paul.vriens.wine@gmail.com writes:
'http://www.xs4all.nl/~pvriens/200708031000.new.single/' shows the new results page for single tests. Instead of a single '.' there is a failure shown (red box) and a skip border (as these tests have not run). There is still a '.' which could maybe better be a '?'. There is also a 'onMouseOver' hint when you go over the '.'.
Hi Paul,
Sorry for being so late. As a quick note, onMouseOver should contain JavaScript code, not a freeform string. Isn't the title attribute enough?
-
Ferenc Wagner -
Jakob Eriksson -
Paul Vriens -
Robert Kim Wireless Internet Advisor