On Mon, 17 Mar 2008, Paul Vriens wrote: [...]
We currently have these single '.' also when there are multiple failures for an OS that are actually the same failures.
Yes, I've noticed that. Essentially we lose information when merging bug reports. Fix That's next on my todo list. Here's my (optimistic) todo list:
* Fix merging report data so the group-level information is more accurate. * Collect statistics about the number of unit tests that failed and their percentage, both for individual reports and for groups. * For each build, collect global statistics about the group-level results, how many reports of each type are present, etc. Store that in a new machine readable file (a bit like the individual report's summary.txt files). * Add a new script that takes these build-level summaries and generates a proper top-level index file showing the percentage of failed unit tests and the count of reports for each group.
That last bit would make progress more visible, and would make it easier to find a build which has been tested on a given platform.
For example the missing dll's. There are several dll's missing on win95 and win98 so it would be nice to have this reflected in the group piece of the page. I think green with a blue border looks much nicer than the single '.' we are currently having. Other thoughts?
Well, if, for a given test, all the reports in the group have the same state ('crashed', 'timed out', 'errors', etc.), then the group's cell should look exactly like each of the corresponding report cells. So in the missing Win95 dlls, this would indeed be a 'pass also-skip' cell. But if they all crashed, then it should be a white cell with the 'crashed' message.
If the cells have different states, for instance one says 'crashed' and the other 'timed out', or one says 'not run' and the other says errors occurred, then I think the group cell should say something like 'mixed' and be on a white background.
Maybe the 'filelimit' and 'truncated' cells should be ignored when merging, though we should really not have any of these either.
The new page layout is now present and I must say it looks better although Wine looks a bit worse now because all the green is gone :-).
Yeah :-/ But I think it more accurately reflects the state of Wine. Hopefully it's also more motivating for the people fixing Wine to eliminate todos.