https://bugs.winehq.org/show_bug.cgi?id=48164
François Gouget fgouget@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED
--- Comment #1 from François Gouget fgouget@codeweavers.com --- This is done.
Problem -------
1. Detecting when a new failure slips past the TestBot.
https://test.winehq.org/data/patterns.html
-> The new page has one pattern per test with failures (including those with missing dlls, etc). As planned the failure count is color coded so changes are easily visible. -> All the patterns are on a single page which is easier to review than visiting one page per test. -> The tests are put in four categories, with the ones most likely to contain new failures coming first. This helps reduce the number of tests that need to be looked at to have a good chance of detecting the new failures.
2. Detecting when the results on a VM degrade.
-> At the top of the page a color coded pattern shows the number of failed test units for each VM. Changes in color make result degradation obvious. This also allows detecting when a VM stops reporting results (whether because it has too many failures, times out or other reasons).
3. Comparing the results of two machines of different platforms.
-> The pattern at the top of the page contains results for all platforms. The results are sorted by platform to make it easy to compare their result as a group, but the results of different platform are just a few lines apart.
4. Show only the failures, not all the lines with skips and todos.
-> This is the normal behavior of the new pattern page.
5. Have a page showing only the TestBot results.
-> Two pages show the results for the Windows and Wine TestBot VMs respectively.
https://test.winehq.org/data/patterns-tb-win.html https://test.winehq.org/data/patterns-tb-wine.html
Implementation tweaks ---------------------
* Instead of a single color gradient there is a chain of color gradients to help with patterns that have many different failure counts. This is configurable and currently goes like this: dark cyan -> green -> yellow -> red.
* For each test the page links to related commits. For each commit it show if it patched the test itself, a shared test resource or the Wine module. This simplifies reviewing potential culprits when a test starts failing.
* For each test the page links to the related bugs. This simplifies checking if someone has already analyzed the bug or tried fixing it. If the bug has a regression commit id that commit is shown as such.
* The page also shows bugs for tests that have no failure. This simplifies verifying that the bugs have been updated when a test is fixed.
Commits -------
There are about 27 commits spread between the two below:
commit bce49667e6418c326ade5704ee65c5bf8c6930d6 Author: Francois Gouget fgouget@codeweavers.com Date: Tue Apr 27 12:45:07 2021 +0200
winetest/build-patterns: Add a page showing failure patterns.
The new page shows the failure patterns across time and test configurations to help detect Wine changes that cause new failures, and identify when failures happen in specific configurations.
Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=48164 Signed-off-by: Francois Gouget fgouget@codeweavers.com Signed-off-by: Alexandre Julliard julliard@winehq.org
commit 0ef48b83ac84f3a59ed5521b24d5a7a8d1dcde17 Author: Francois Gouget fgouget@codeweavers.com Date: Wed May 19 03:15:13 2021 +0200
winetest/build-patterns: Show Potentially obsolete bugs.
Bugs related to a test that has no failure are likely to be out of date, except if they are related to memory issues.
Signed-off-by: Francois Gouget fgouget@codeweavers.com Signed-off-by: Alexandre Julliard julliard@winehq.org