[Bug 48216] New: Add a WineTest size, time and test failures comparison page to the TestBot

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48216

Bug ID: 48216
           Summary: Add a WineTest size, time and test failures comparison
                    page to the TestBot
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

This would expend on the data collected for bug 48215.

This would be a new page that lets the user pick two (or three or more)
WineTest results and put them in the same size, run time and test unit failure
tables as the JobDetails page would if they were all from the same job.

For each column the user would pick a CommitId and then get a list of the
available task reports (for instance "wxppro (32 bit)", "debian10 (32 bit
Chinese:China)").

This would allow comparing a VM's 32 and 64 bit results (these run in separate
jobs). Or compare a VM's latest result against its past result(s). Or compare
the results of running the tests on Windows 8 with those of Windows 10 on the
same hardware. Or comparing a WineTest run that timed out with one that did
not.

In effect this would be similar to bug 15477 but implemented in the TestBot
rather than the on test.winehq.org, the advantage being that the TestBot has
the infrastructure needed to dynamically build pages.

One approach to implement this would be to pour all the report summary
information into a big table. Something like (JobId, StepNo, TaskNo, CommitId,
TestUnit, Failures, Size, RunTime). However given the performance issues with
our current Perl ORM the performance may well be unacceptably slow (see bug
45023).

Another approach would be to use the task-level summary files (described in bug
48215) and use a much smaller table to map commit ids to the tasks with
relevant results. So this table would just contain (JobId, StepNo, TaskNo,
CommitId) and would be used to know which tasks have results for a given commit
id, and once the tasks have been selected, where to find the summary file
containing the data to put in the tables.

[Bug 48216] Add a WineTest size, time and test failures comparison page to the TestBot

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48216

--- Comment #1 from François Gouget  ---
Note that tasks can have more than one report. So the (JobId, StepNo, TaskNo)
triplet is not enough to fully identify the report to use to build the table.
Instead it should be extended with the report name: (JobId, StepNo, TaskNo,
ReportBaseName).

But currently we never have more than one WineTest run per task because this
ensures each WineTest run starts from a clean environment. So the (JobId,
StepNo, TaskNo) triplet would be enough.

Adding the ReportName would not be hard but may make things harder in case the
report name changes for whatever reason. So it may be better to only add it
when actually needed.