[Bug 48653] New: Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

Bug ID: 48653
           Summary: Automate checking the WTBS jobs
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

The Wine TestBot test Suite (WTBS) is a set of Wine patches that can be used to
check how the TestBot handles test failures, timeouts, patch series, non-Wine
patches, etc:

https://github.com/fgouget/wine/commits/wtbsuite

The patches can either be used in isolation to test a specific aspect, or all
together using formail to do a broad check for TestBot regressions. However as
the test suite has become more complete the number of patches has grown and now
tops 90. So a full check for regressions means checking the results of about 90
TestBot  jobs for missing test failures, incorrect report validation messages,
etc. This makes it pretty time consuming.

So the goal is to automate at least the most basic checks.
To do so the plan is to add lines of the form 'WTBS.xxx=yyy' to the commit
messages of the WTBS patches. Each will define a property that should be
checked in the TestBot job and tasks.

Then a TestBot script can be run to go through every job, ignore the queued and
running ones and those have have no associated patch. For the others read the
associated patch and look for the WTBS.xxx lines. Note that for patch series,
only the last set of WTBS values should be taken into account (i.e. the ones
after the "Last patchset part" line).

Here is a sample of the possible properties to check:
* WTB.Job.Remarks
  The job 'title' which is derived from the patch subject line.

* WTBS.Job.Status
  Whether to expect badpatch, completed, etc.

For tasks the property names would be prefixed with the VM type: build, win
(for win32 and win64), wine. This allows specifying different results for each.

* WTBS.build.Status, WTBS.win.Status, WTBS.wine.Status
  Same as Job.Status but at the task level.

* WTBS.win.Failures, WTBS.wine.Failures
  The number of test failures expected from the result.
  Note that if the Wine Test Unit is buggy we may get more test failures than
expected. This is particularly an issue with Wine VMs if their mission include
test=module or test=all.
  So it may make sense to specify WTBS.win.Failures but leave out
WTBS.wine.Failures. Or maybe have a MinFailures property. Or grep the failures
whose message contains 'WTBS' and only compare those to this property.

* WTBS.win.NewFailures, WTBS.wine.NewFailures
  The number of new test failures as reported by LoadLogErrors() for each of
the task's report.

* WTBS.win.ReportFailures, WTBS.wine.ReportFailures
  In a number of cases what we're interested in is the report validation errors
produced by the TestBot. Just checking the number of such errors may not be
that useful. What we may want is a way to check that it reported missing
failures messages, or too much data being printed, etc. This requires being
able to specify a set of errors that are expected to be present for which this
basic property system is not very well suited.

* WTBS.win.TestUnits
  For each VM, collect the Step's FileName and the Task's CmdLineArg values to
determine the test unit being run, and verify that this is a superset of the
TestUnits list. This can be used to ensure that the TestBot ran the right set
of tests (and avoids failures when a new test is added to Wine).
  Note that doing the same thing for Wine VMs is harder because those can have
test=module in their mission(s), which changes greatly the list of tests being
run.

It may make sense to provide default values for some properties to not have to
type too much in the commit messages. For instance for the Status properties
one may expect the status to be 'completed'. But given the WTBS tests error
cases a lot that may not be all that useful.

Similarly it may be useful to be able to specify something like
'WTBS.all.Status' to specify the expected Status for the build, win and wine
VMs all at once. But again this may be overkill.

Finally the general rule should be that unspecified properties are not checked.
So if 'WTBS.wine.Failures' is not specified, then the failure count of Wine VMs
should not be checked.