Re: [Tools 6/6] winetest: Issue a better description than '.' for the special failure modes. Write them as a note so they stand out better.

[Paul Vriens]

Francois Gouget wrote:
> ---
>  winetest/gather |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/winetest/gather b/winetest/gather
> index b502345..ee49f4a 100755
> --- a/winetest/gather
> +++ b/winetest/gather
> @@ -329,21 +329,21 @@ sub singletest($$$) {
>  EOF
>      } elsif ($count eq "winetest crash") {
>          print OUT <<"EOF";
> -      <td class="result fail also-skip"><a
> +      <td class="note"><a
>          title="Test did not run as winetest crashed"
> -        >.</a></td>
> +        >winetest</a></td>
>  EOF
>      } elsif ($count eq "file limit") {
>          print OUT <<"EOF";
> -      <td class="result fail also-skip"><a
> +      <td class="note"><a
>          title="Test is missing because of a partial report file"
> -        >.</a></td>
> +        >truncated</a></td>
>  EOF
>      } elsif ($count eq "test missing") {
>          print OUT <<"EOF";
> -      <td class="result fail also-skip"><a
> +      <td class="note"><a
>          title="Test did not run for an unknown reason"
> -        >.</a></td>
> +        >not run</a></td>
>  EOF
>      } else {
>          my $class = $error?"fail":$todo?"todo":"pass";
> Hi Francois,

We currently have these single '.' also when there are multiple failures for an 
OS that are actually the same failures.

For example the missing dll's. There are several dll's missing on win95 and 
win98 so it would be nice to have this reflected in the group piece of the page. 
I think green with a blue border looks much nicer than the single '.' we are 
currently having. Other thoughts?

The new page layout is now present and I must say it looks better although Wine 
looks a bit worse now because all the green is gone :-).

-- 
Cheers,

Paul.

[Tools 6/6] winetest: Issue a better description than '.' for the special failure modes. Write them as a note so they stand out better.

[Francois Gouget]

On Mon, 17 Mar 2008, Paul Vriens wrote:
[...]
> We currently have these single '.' also when there are multiple failures for
> an OS that are actually the same failures.

Yes, I've noticed that. Essentially we lose information when merging bug 
reports. Fix That's next on my todo list. Here's my (optimistic) todo 
list:

 * Fix merging report data so the group-level information is more 
accurate.
 * Collect statistics about the number of unit tests that failed and 
their percentage, both for individual reports and for groups.
 * For each build, collect global statistics about the group-level 
results, how many reports of each type are present, etc. Store that 
in a new machine readable file (a bit like the individual report's 
summary.txt files).
 * Add a new script that takes these build-level summaries and generates 
a proper top-level index file showing the percentage of failed unit 
tests and the count of reports for each group.

That last bit would make progress more visible, and would make it easier 
to find a build which has been tested on a given platform.
> For example the missing dll's. There are several dll's missing on win95 and
> win98 so it would be nice to have this reflected in the group piece of the
> page. I think green with a blue border looks much nicer than the single '.' we
> are currently having. Other thoughts?

Well, if, for a given test, all the reports in the group have the 
same state ('crashed', 'timed out', 'errors', etc.), then the group's 
cell should look exactly like each of the corresponding report cells. So 
in the missing Win95 dlls, this would indeed be a 'pass also-skip' cell. 
But if they all crashed, then it should be a white cell with 
the 'crashed' message.

If the cells have different states, for instance one says 'crashed' and 
the other 'timed out', or one says 'not run' and the other says errors 
occurred, then I think the group cell should say something like 'mixed' 
and be on a white background.

Maybe the 'filelimit' and 'truncated' cells should be ignored when 
merging, though we should really not have any of these either.
> The new page layout is now present and I must say it looks better although
> Wine looks a bit worse now because all the green is gone :-).

Yeah :-/ But I think it more accurately reflects the state of Wine. 
Hopefully it's also more motivating for the people fixing Wine to 
eliminate todos.

-- 
Francois Gouget <fgouget(a)free.fr>              http://fgouget.free.fr/
                           La terre est une b