On 4/23/21 7:54 AM, Francois Gouget wrote:
I did not get to send this patch set yet so here's a preview:
Windows test patterns http://fgouget.free.fr/tmp/winepatterns/patterns-tb-win.html
Wine test patterns http://fgouget.free.fr/tmp/winepatterns/patterns-tb-wine.html
So for each test unit that had failures you have:
- one line per test configuration (report tag)
- one row per WineTest build (daily commit)
Each cell has a one character description of the test result (with more details in the tooltip) and is color coded to help identify patterns. Each 'failure' cell also links to the corresponding report page.
The main goal of these pages for me is to simplify detecting new failures in the WineTest results... and not just in cases where there was no failure before since there are so many cases with preexisting failures already. In the patterns an increase in the number of failures shows up as a color change which makes them easy to spot.
A secondary goal is to allow comparing the before/after results when a test configuration (VM or otherwise) is modified.
A side effect is that these can also help identify the different failure modes. For instance comctl32:monthcal fails on wednesdays, except in Korean where it's all the time: http://fgouget.free.fr/tmp/winepatterns/patterns-tb-win.html#comctl32:monthc...
To help with these tasks the page also sorts the test units to show those that look like they have new failures first. So starting the review from the top is most likely to point to the new issues.
Looks like a more sophisticated version of https://www.winehq.org/~jwhite/2deb8c2825af.html, which is definitely a nice resource when I'm trying to put effort into fixing test failures.
I guess the tests are color-coded by number of failures, modulo some constant? I like the idea. I will note though that some of those colours seem hard to tell apart, e.g. the shades of green in wine d3d9:device. Also I guess they aren't consistent across tests for some reason?