New subject: [tools 1/6] testbot: Add tables to track the test failures.

15 Jun 2022


      This patchset provides support for tracking known failures in the
TestBot results. The main goal is to avoid false positives but there
are some side benefits as well:
* The job reports sent to developers link to the relevant WineHQ bugs
  which gives them more exposure and may help get them fixed faster.
* The failure details pages will eventually link to all the tasks where
  the failure happened, which could be valuable to find patterns.
* The failure page tracks when it was last seen. This simplifies
  spotting bugs which are fixed and could be closed.
  (Or failures where the regular expression needs to be updated / fixed)
The patchset mostly follows the scheme described in bug 48912:
https://bugs.winehq.org/show_bug.cgi?id=48912
There are some differences though:
* The bug envisionned this scheme purely as a blacklist to avoid false
  positives. The implementation can certainly be used in this way but
  it is not limited to failures that cause false positives.
  Tracking other failures can provide the side benefits mentionned
  above. The only limit is the maintenance work required to track
  more failures.
* As a consequence the FailureBlacklist class was renamed to Failure,
  and the FailureBlacklistUses was renamed to TaskFailures since it
  links Tasks with the related Failures.
* The FailureBlacklistVMs purpose is not really described in the bug but 
  I think the goal was to provide a way to deckare that the failure is 
  expected to only happen on specific test configurations.
  Instead the Failures class has a ConfigRegExp regular expression
  which specifies which test configurations the failure applies to.
  The test 'configuration name' is of the form 'VM:logfile' which
  allows matching specific Windows versions (thanks to the TestBot VM
  naming scheme), specific Windows locales (part of the VM name),
  specific bitnesses (part of the log filename), or specific Wine
  locales (part of the log filename). For instance:
    ^w8            -> all Windows 8 VMs
    :exe           -> all Windows tests (exe32.report and exe64.report)
    ^w10.*_ja.*:   -> all Windows 10 Japanese test configurations
    win|wow        -> all Wine test configurations (win32.report, ...)
    (win|wow).*_fr -> French Wine test configurations
* Changes in FailureBlacklists -> Failures
  - Primary key: (Bug, TestModule, TestUnit) -> Id
    Some issues generate quite a few failures. So in case trying to
    match them all results in a regular expression that's too long
    (either because of the database limit or for readability), this
    change makes it possible to add multiple Failure objects to track
    them all.
- TestModule -> ErrorGroup
    The .errors files split the errors in groups. Each group
    corresponds to a test module, but extra errors (too much output,
    missing summary lines, etc) are in a separate group. So I renamed
    TestModule to ErrorGroup as a reminder to the coder that this is
    meant to match the .errors group names, not just test modules.
- Name -> (BugStatus, BugDescription)
    Instead of having a name describing the failure, this patchset uses
    the corresponding WineHQ bug description.
    The TestBot also retrieves the bug status which allows checking
    that closed bugs dont happen anymore, or open bugs that should be
    closed.
- LastUsed -> (LastNew, LastOld)
    The LastUse field was split into LastNew and LastOld to track
    whether the failure would cause false positives without the known
    failures support.
* Changes in FailureBlacklistUses -> TaskFailures
  - (Bug, TestModule, TestUnit) -> FailureId
    This is a consequence of the Failure primary key change described
    above.
- () -> (TaskLog)
    This allows tracking which task log the failure was found in. This
    is quite important for Wine tasks since they run all the test 
    configurations in just one task.
- () -> (NewCount, OldCount)
    This provides a count of matching failures and whether they would
    have been tagged as old or new without this patchset.
    This provides a quick way to know whether there is the same number
    of failures every time or whether the number of failures changes
    from one run to the next.
    It also provides a quick way to know how many failures to look for
    when jumping to the log.
Francois Gouget (6):
  testbot: Add tables to track the test failures.
  testbot/web: Add pages to add, edit and show known failures.
  testbot: Match the failures with the test logs.
  testbot/web: Link job details to the related known failures.
  testbot/WineSendLog: Take into account the known failures.
  testbot/UpdateFailures: Add a script to refresh the failures' bug
    information.
testbot/bin/UpdateFailures.pl           | 192 ++++++
 testbot/bin/WineSendLog.pl              |  78 ++-
 testbot/ddl/update47.sql                |  33 +
 testbot/ddl/winetestbot.sql             |  32 +
 testbot/doc/INSTALL.txt                 |   5 +
 testbot/doc/winetestbot-schema.dia      | 852 +++++++++++++++++++++---
 testbot/lib/WineTestBot/CGI/PageBase.pm |   1 +
 testbot/lib/WineTestBot/Config.pm       |   6 +-
 testbot/lib/WineTestBot/Failures.pm     | 184 +++++
 testbot/lib/WineTestBot/Jobs.pm         |   6 +
 testbot/lib/WineTestBot/LogUtils.pm     | 180 ++++-
 testbot/lib/WineTestBot/TaskFailures.pm | 120 ++++
 testbot/lib/WineTestBot/Tasks.pm        |   4 +
 testbot/web/FailureDetails.pl           | 119 ++++
 testbot/web/FailuresList.pl             | 160 +++++
 testbot/web/JobDetails.pl               |  54 +-
 testbot/web/WineTestBot.css             |   3 +
 17 files changed, 1907 insertions(+), 122 deletions(-)
 create mode 100755 testbot/bin/UpdateFailures.pl
 create mode 100644 testbot/ddl/update47.sql
 create mode 100644 testbot/lib/WineTestBot/Failures.pm
 create mode 100644 testbot/lib/WineTestBot/TaskFailures.pm
 create mode 100644 testbot/web/FailureDetails.pl
 create mode 100644 testbot/web/FailuresList.pl
-- 
2.30.2

[tools 0/6] Known failures tracking