https://bugs.winehq.org/show_bug.cgi?id=48653
--- Comment #1 from François Gouget fgouget@codeweavers.com --- An alternative to the above property system would to be embed a .errors file in the commit message.
.errors can both store properties like the ones described above, and groups of errors. Those can be compared to the task's new errors, i.e. those generated by the WTBS patch, to ensure we got the expected result. Note that the comparison already takes care of ignoring changes like line numbers and variable parts like crash addresses, etc.
For instance: --- WTBS.Results --- p WTBS.Job.Remarks WTBS Unreported test failures (kernel32:comm). p WTBS.Job.Status completed p WTBS.win.Status completed p WTBS.win.TestUnits kernel32:comm p WTBS.wine.TestUnits kernel32:comm g 0 kernel32 n 0 comm.c:2210: Test failed: WTBS Simulate an unreported test failure g 0 Report validation errors n 0 kernel32:comm has unaccounted for failure messages n 0 kernel32:comm returned success despite having failures --- WTBS.Results ---
This scheme would still need some adjustments to allow for different results on Windows and Wine VMs (prefix the group names?). It may also prove hard to maintain if the TestBot error messages change too often.
Note: The WTBS convention is to put the test units being used for the test in parentheses at the end of the subject line. This may be leveraged to provide a good default for the TestUnits properties (the exception being test=module again).