https://bugs.winehq.org/show_bug.cgi?id=48164 François Gouget <fgouget(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #1 from François Gouget <fgouget(a)codeweavers.com> --- This is done. Problem ------- 1. Detecting when a new failure slips past the TestBot. https://test.winehq.org/data/patterns.html -> The new page has one pattern per test with failures (including those with missing dlls, etc). As planned the failure count is color coded so changes are easily visible. -> All the patterns are on a single page which is easier to review than visiting one page per test. -> The tests are put in four categories, with the ones most likely to contain new failures coming first. This helps reduce the number of tests that need to be looked at to have a good chance of detecting the new failures. 2. Detecting when the results on a VM degrade. -> At the top of the page a color coded pattern shows the number of failed test units for each VM. Changes in color make result degradation obvious. This also allows detecting when a VM stops reporting results (whether because it has too many failures, times out or other reasons). 3. Comparing the results of two machines of different platforms. -> The pattern at the top of the page contains results for all platforms. The results are sorted by platform to make it easy to compare their result as a group, but the results of different platform are just a few lines apart. 4. Show only the failures, not all the lines with skips and todos. -> This is the normal behavior of the new pattern page. 5. Have a page showing only the TestBot results. -> Two pages show the results for the Windows and Wine TestBot VMs respectively. https://test.winehq.org/data/patterns-tb-win.html https://test.winehq.org/data/patterns-tb-wine.html Implementation tweaks --------------------- * Instead of a single color gradient there is a chain of color gradients to help with patterns that have many different failure counts. This is configurable and currently goes like this: dark cyan -> green -> yellow -> red. * For each test the page links to related commits. For each commit it show if it patched the test itself, a shared test resource or the Wine module. This simplifies reviewing potential culprits when a test starts failing. * For each test the page links to the related bugs. This simplifies checking if someone has already analyzed the bug or tried fixing it. If the bug has a regression commit id that commit is shown as such. * The page also shows bugs for tests that have no failure. This simplifies verifying that the bugs have been updated when a test is fixed. Commits ------- There are about 27 commits spread between the two below: commit bce49667e6418c326ade5704ee65c5bf8c6930d6 Author: Francois Gouget <fgouget(a)codeweavers.com> Date: Tue Apr 27 12:45:07 2021 +0200 winetest/build-patterns: Add a page showing failure patterns. The new page shows the failure patterns across time and test configurations to help detect Wine changes that cause new failures, and identify when failures happen in specific configurations. Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=48164 Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> Signed-off-by: Alexandre Julliard <julliard(a)winehq.org> commit 0ef48b83ac84f3a59ed5521b24d5a7a8d1dcde17 Author: Francois Gouget <fgouget(a)codeweavers.com> Date: Wed May 19 03:15:13 2021 +0200 winetest/build-patterns: Show Potentially obsolete bugs. Bugs related to a test that has no failure are likely to be out of date, except if they are related to memory issues. Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> Signed-off-by: Alexandre Julliard <julliard(a)winehq.org> -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.