This patchset provides support for tracking known failures in the TestBot results. The main goal is to avoid false positives but there are some side benefits as well: * The job reports sent to developers link to the relevant WineHQ bugs which gives them more exposure and may help get them fixed faster. * The failure details pages will eventually link to all the tasks where the failure happened, which could be valuable to find patterns. * The failure page tracks when it was last seen. This simplifies spotting bugs which are fixed and could be closed. (Or failures where the regular expression needs to be updated / fixed)
The patchset mostly follows the scheme described in bug 48912: https://bugs.winehq.org/show_bug.cgi?id=48912
There are some differences though: * The bug envisionned this scheme purely as a blacklist to avoid false positives. The implementation can certainly be used in this way but it is not limited to failures that cause false positives. Tracking other failures can provide the side benefits mentionned above. The only limit is the maintenance work required to track more failures.
* As a consequence the FailureBlacklist class was renamed to Failure, and the FailureBlacklistUses was renamed to TaskFailures since it links Tasks with the related Failures.
* The FailureBlacklistVMs purpose is not really described in the bug but I think the goal was to provide a way to deckare that the failure is expected to only happen on specific test configurations. Instead the Failures class has a ConfigRegExp regular expression which specifies which test configurations the failure applies to. The test 'configuration name' is of the form 'VM:logfile' which allows matching specific Windows versions (thanks to the TestBot VM naming scheme), specific Windows locales (part of the VM name), specific bitnesses (part of the log filename), or specific Wine locales (part of the log filename). For instance: ^w8 -> all Windows 8 VMs :exe -> all Windows tests (exe32.report and exe64.report) ^w10.*_ja.*: -> all Windows 10 Japanese test configurations win|wow -> all Wine test configurations (win32.report, ...) (win|wow).*_fr -> French Wine test configurations
* Changes in FailureBlacklists -> Failures - Primary key: (Bug, TestModule, TestUnit) -> Id Some issues generate quite a few failures. So in case trying to match them all results in a regular expression that's too long (either because of the database limit or for readability), this change makes it possible to add multiple Failure objects to track them all.
- TestModule -> ErrorGroup The .errors files split the errors in groups. Each group corresponds to a test module, but extra errors (too much output, missing summary lines, etc) are in a separate group. So I renamed TestModule to ErrorGroup as a reminder to the coder that this is meant to match the .errors group names, not just test modules.
- Name -> (BugStatus, BugDescription) Instead of having a name describing the failure, this patchset uses the corresponding WineHQ bug description. The TestBot also retrieves the bug status which allows checking that closed bugs dont happen anymore, or open bugs that should be closed.
- LastUsed -> (LastNew, LastOld) The LastUse field was split into LastNew and LastOld to track whether the failure would cause false positives without the known failures support.
* Changes in FailureBlacklistUses -> TaskFailures - (Bug, TestModule, TestUnit) -> FailureId This is a consequence of the Failure primary key change described above.
- () -> (TaskLog) This allows tracking which task log the failure was found in. This is quite important for Wine tasks since they run all the test configurations in just one task.
- () -> (NewCount, OldCount) This provides a count of matching failures and whether they would have been tagged as old or new without this patchset. This provides a quick way to know whether there is the same number of failures every time or whether the number of failures changes from one run to the next. It also provides a quick way to know how many failures to look for when jumping to the log.
Francois Gouget (6): testbot: Add tables to track the test failures. testbot/web: Add pages to add, edit and show known failures. testbot: Match the failures with the test logs. testbot/web: Link job details to the related known failures. testbot/WineSendLog: Take into account the known failures. testbot/UpdateFailures: Add a script to refresh the failures' bug information.
testbot/bin/UpdateFailures.pl | 192 ++++++ testbot/bin/WineSendLog.pl | 78 ++- testbot/ddl/update47.sql | 33 + testbot/ddl/winetestbot.sql | 32 + testbot/doc/INSTALL.txt | 5 + testbot/doc/winetestbot-schema.dia | 852 +++++++++++++++++++++--- testbot/lib/WineTestBot/CGI/PageBase.pm | 1 + testbot/lib/WineTestBot/Config.pm | 6 +- testbot/lib/WineTestBot/Failures.pm | 184 +++++ testbot/lib/WineTestBot/Jobs.pm | 6 + testbot/lib/WineTestBot/LogUtils.pm | 180 ++++- testbot/lib/WineTestBot/TaskFailures.pm | 120 ++++ testbot/lib/WineTestBot/Tasks.pm | 4 + testbot/web/FailureDetails.pl | 119 ++++ testbot/web/FailuresList.pl | 160 +++++ testbot/web/JobDetails.pl | 54 +- testbot/web/WineTestBot.css | 3 + 17 files changed, 1907 insertions(+), 122 deletions(-) create mode 100755 testbot/bin/UpdateFailures.pl create mode 100644 testbot/ddl/update47.sql create mode 100644 testbot/lib/WineTestBot/Failures.pm create mode 100644 testbot/lib/WineTestBot/TaskFailures.pm create mode 100644 testbot/web/FailureDetails.pl create mode 100644 testbot/web/FailuresList.pl