[Bug 48655] New: Automate checking the WTBS emails

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48655

Bug ID: 48655
           Summary: Automate checking the WTBS emails
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

See bug 48653 for a description of the WTBS and how to check the jobs created
when running it on the TestBot.

When a 'wine-devel' job completes the TestBot sends one or two emails.
* One of the emails simply sends the task reports to the patch author.
* The other is only sent if a task generated new errors.

The point of the TestWTBS.emails script would be to check that:

1. The TestBot sent the task report email for every Wine patch and not for any
patch email. This would be based off of the WTBS.Patch.Disposition field.

2. The TestBot sent a 'found new errors' email only when new errors are
expected. This would be based off of the 'WTBS.*.NewFailures' property or the
expected errors list.

However for both these checks the script would need access to the emails sent
by the TestBot. The simplest option would be for the tester to set the
WinePatchToOverride property and save the emails to a separate mbox and give
the patch to that mbox to the script.

As in the TestWTBS.patches case, the script would only care about a few lines:

* The 'From' lines to delimit the emails.

* The 'Subject' lines to match them to TestBot jobs and thus to the set of WTBS
properties.
  Note that this would not rely on the 'TestBot job xxx results:' part since
the new failures email are lacking it. Instead it would match the rest of the
subject to the job's Remarks field.

* The 'I think I found new failures' line to identify emails reporting new
errors.

The script could later be expanded to check the content of the email or
attachments more in details if needed.

[Bug 48655] Automate checking the WTBS emails

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48655

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #1 from François Gouget  ---
This is done. See the testbot/tests/TestWTBS commits, starting with:

commit 394be2d89578ad17cfc972ad691f00260866d6f8
Author: Francois Gouget 
Date:   Tue Jan 26 04:38:44 2021 +0100

testbot/TestWTBS: Check the result and error emails.

When a job completes the TestBot sends an email with the results and
    another with a summary of the errors in case of new failures. This
    patch allows checking that these emails did get sent if appropriate
    and seem to contain the expected data.
    For this the emails should be saved to a file in mbox format and
    that file passed to TestWTBS using the --mbox option.

Signed-off-by: Francois Gouget 
    Signed-off-by: Alexandre Julliard

[Bug 48655] Automate checking the WTBS emails

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48655

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #2 from François Gouget  ---
Closing.