On Fri, 27 Mar 2020 at 16:24, Francois Gouget fgouget@codeweavers.com wrote:
So I'm resubmitting this patch hoping for some feedback since the last time it just dropped off the mailing list [1].
The issue is that some test failures include values that change on every run which prevents the TestBot from knowing whether they are new or not. In turn this means the TestBot systematically blames the author of whichever patch is being tested for these failures.
If the main goal is to stop the testbot from being ignored, and to limit the number of new failures sneaking in, would it make sense to start with something fairly blunt, like ignoring failures for tests on unreliable configurations? E.g., suppose ddraw:ddraw7 reliably passed on w1064v1507, but not w1064v1809, you'd then blacklist all of ddraw:ddraw7 on w1064v1809. That means you potentially ignore some ddraw:ddraw7 tests that are reliable, but it would still be an improvement over effectively ignoring everything. Perhaps more importantly, it also means you can incrementally work on reducing the size of the blacklist, instead of having to do that work upfront.