The TestBot now has a list of known failures which it can identify and link to the corresponding Wine bug.
The main benefit is that the TestBot knows your patches cannot be the cause of known failures, even if they look new [1]. The list of known failures is not yet complete [2] but you can expect that over the next few weeks the cases where the TestBot incorrectly claims your patch introduces new failures should get much rarer.
Failure tracking also provides you with other benefits:
The task full logs now have an index of the known failures they contain to help you quickly go to the first corresponding failure line. (and a pair of additional entries point you to the first and last failure)
Also the failure lines are annotated with the corresponding bug id(s). This id is a link that takes you to a page that: * Shows how the TestBot identified the failure. * Links to the Wine bug discussing the failure. * And show all the tasks and logs where the failure occurred. You can use this to quickly access these logs and look for common patterns.
You can also get a list of the known failures by clicking on the Failures entry in the TestBot's sidebar.
https://testbot.winehq.org/FailuresList.pl
For each failure you will see when it last happened and the status of the corresponding bug (new, closed, etc). This is a good way to identify failures that seem to be fixed.
Finally the job report emails also list the known failures that happened while testing your patch. Consider this a reminder that your help in fixing them would be greatly appreciated, particularly as you may already know the area since you patched it.
[1] The TestBot normally diffs the test run's failures against the past WineTest reports and considers any line marked as new (literally '+' in unified diff parlance), as a new failure.
This means any failure that's rare enough to not be present in any of the past WineTest runs of the same test configuration, will be flagged as new.
Failures where the message changes (beyond basic line number changes, addresses in exceptions, etc.) will also look new. This could be because a patch changed the failure message, but more often it is because the message itself contains a pointer, handle, uninitialized data, elapsed time, etc.
[2] The goal is not necessarily to track every test failure but to track all those that can cause false positives. So a test that fails systematically may not be tracked (although if it has a Wine bug it probably should be). But tests that tend to be incorrectly tagged as new due to the issues in [1] should all be.
The way I plan to get there is to monitor the TestBot's WineTest jobs for a while and add to the known failure list whenever new unknown failures appear there.