[Bug 48653] New: Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

Bug ID: 48653
           Summary: Automate checking the WTBS jobs
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

The Wine TestBot test Suite (WTBS) is a set of Wine patches that can be used to
check how the TestBot handles test failures, timeouts, patch series, non-Wine
patches, etc:

https://github.com/fgouget/wine/commits/wtbsuite

The patches can either be used in isolation to test a specific aspect, or all
together using formail to do a broad check for TestBot regressions. However as
the test suite has become more complete the number of patches has grown and now
tops 90. So a full check for regressions means checking the results of about 90
TestBot  jobs for missing test failures, incorrect report validation messages,
etc. This makes it pretty time consuming.

So the goal is to automate at least the most basic checks.
To do so the plan is to add lines of the form 'WTBS.xxx=yyy' to the commit
messages of the WTBS patches. Each will define a property that should be
checked in the TestBot job and tasks.

Then a TestBot script can be run to go through every job, ignore the queued and
running ones and those have have no associated patch. For the others read the
associated patch and look for the WTBS.xxx lines. Note that for patch series,
only the last set of WTBS values should be taken into account (i.e. the ones
after the "Last patchset part" line).

Here is a sample of the possible properties to check:
* WTB.Job.Remarks
  The job 'title' which is derived from the patch subject line.

* WTBS.Job.Status
  Whether to expect badpatch, completed, etc.

For tasks the property names would be prefixed with the VM type: build, win
(for win32 and win64), wine. This allows specifying different results for each.

* WTBS.build.Status, WTBS.win.Status, WTBS.wine.Status
  Same as Job.Status but at the task level.

* WTBS.win.Failures, WTBS.wine.Failures
  The number of test failures expected from the result.
  Note that if the Wine Test Unit is buggy we may get more test failures than
expected. This is particularly an issue with Wine VMs if their mission include
test=module or test=all.
  So it may make sense to specify WTBS.win.Failures but leave out
WTBS.wine.Failures. Or maybe have a MinFailures property. Or grep the failures
whose message contains 'WTBS' and only compare those to this property.

* WTBS.win.NewFailures, WTBS.wine.NewFailures
  The number of new test failures as reported by LoadLogErrors() for each of
the task's report.

* WTBS.win.ReportFailures, WTBS.wine.ReportFailures
  In a number of cases what we're interested in is the report validation errors
produced by the TestBot. Just checking the number of such errors may not be
that useful. What we may want is a way to check that it reported missing
failures messages, or too much data being printed, etc. This requires being
able to specify a set of errors that are expected to be present for which this
basic property system is not very well suited.

* WTBS.win.TestUnits
  For each VM, collect the Step's FileName and the Task's CmdLineArg values to
determine the test unit being run, and verify that this is a superset of the
TestUnits list. This can be used to ensure that the TestBot ran the right set
of tests (and avoids failures when a new test is added to Wine).
  Note that doing the same thing for Wine VMs is harder because those can have
test=module in their mission(s), which changes greatly the list of tests being
run.

It may make sense to provide default values for some properties to not have to
type too much in the commit messages. For instance for the Status properties
one may expect the status to be 'completed'. But given the WTBS tests error
cases a lot that may not be all that useful.

Similarly it may be useful to be able to specify something like
'WTBS.all.Status' to specify the expected Status for the build, win and wine
VMs all at once. But again this may be overkill.

Finally the general rule should be that unspecified properties are not checked.
So if 'WTBS.wine.Failures' is not specified, then the failure count of Wine VMs
should not be checked.

[Bug 48653] Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

--- Comment #1 from François Gouget  ---
An alternative to the above property system would to be embed a .errors file in
the commit message.

.errors can both store properties like the ones described above, and groups of
errors. Those can be compared to the task's new errors, i.e. those generated by
the WTBS patch, to ensure we got the expected result. Note that the comparison
already takes care of ignoring changes like line numbers and variable parts
like crash addresses, etc.

For instance:
--- WTBS.Results ---
p WTBS.Job.Remarks WTBS Unreported test failures (kernel32:comm).
p WTBS.Job.Status completed
p WTBS.win.Status completed
p WTBS.win.TestUnits kernel32:comm
p WTBS.wine.TestUnits kernel32:comm
g 0 kernel32
n 0 comm.c:2210: Test failed: WTBS Simulate an unreported test failure
g 0 Report validation errors
n 0 kernel32:comm has unaccounted for failure messages
n 0 kernel32:comm returned success despite having failures
--- WTBS.Results ---

This scheme would still need some adjustments to allow for different results on
Windows and Wine VMs (prefix the group names?). It may also prove hard to
maintain if the TestBot error messages change too often.

Note: The WTBS convention is to put the test units being used for the test in
parentheses at the end of the subject line. This may be leveraged to provide a
good default for the TestUnits properties (the exception being test=module
again).

[Bug 48653] Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #2 from François Gouget  ---
This is done. See the testbot/tests/TestWTBS commits, starting with:

commit 51d7779e876241c0a39839bb6eee3837379ed092
Author: Francois Gouget 
Date:   Tue Jan 19 18:59:51 2021 +0100

testbot/TestWTBS: Automate checking the Wine TestBot test Suite results.

The Wine TestBot test Suite is a series of Wine pactches that can be
    used to verify that the TestBot handles test failures, timeouts, etc
    as expected.
    The TestWTBS script looks for directives in the commit message of
    these patches to automate a range of checks on the corresponding jobs
    and tasks.
    To simplify parsing the directives, reuse the TestBot's simple errors
    cache format.

For instance:
    ----- TestWTBS -----
    p job.Status completed
    p tasks.Status completed
    p tests.TestFailures 1
    ----- TestWTBS -----

Signed-off-by: Francois Gouget 
    Signed-off-by: Alexandre Julliard

[Bug 48653] Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #3 from François Gouget  ---
Closing.