[Bug 39413] New: Make it easier to monitor the TestBot activity

https://bugs.winehq.org/show_bug.cgi?id=39413

Bug ID: 39413
           Summary: Make it easier to monitor the TestBot activity
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

At the bottom of the main page there is a list of the VMs and their status.
However the information there is incomplete:
* When a VM is reverting there is no way to know for how long it has been
reverting.
* When a VM is running a task there is no link to the corresponding task.
* There is no history of events so if someone reports that something strange
happened there is no way to check(*).

So it would be nice to have something like BuildBot's Waterfall display.
http://buildbot.buildbot.net/waterfall

(*) Actually one could check the WineTestBot log but that has its own problems
and requires poring over lots of traces. See bug 39410.

[Bug 39413] Make it easier to monitor the TestBot activity

https://bugs.winehq.org/show_bug.cgi?id=39413

François Gouget  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #1 from François Gouget  ---
This is fixed.

The TestBot now has an activity page which is loosely modeled over the
BuildBot's waterfall display.

The Activity page makes it possible to see at a glance what the TestBot is
doing across VM hosts and jobs. It also shows at each step how many tasks are
runnable, queued or blocked on an offline VM, which helps figure out how it
made its decisions. The Activity page also points to the relevant job result
pages.

https://testbot.winehq.org/Activity.pl

The TestBot now also has a statistics page which shows its vital statistics
such as how many jobs it gets per hour, how long it takes to revert each VM,
etc. The statistics page also links to the Activity and Job results pages as
appropriate.

https://testbot.winehq.org/Stats.pl

Finally the TestBot now has a Munin plugin which makes it possible to see the
long term evolution of various parameters like jobs/hour, tasks/hour, VM host
utilization statistics, revert times, WineTest run times, etc.

So if a VM takes a long time to revert it will now be possible to see if it
happened gradually or on a specific date (e.g. after some change on that VM).
It will also be possible to see increases in a VM host utilization which would
show that rebalancing the workload accross VM hosts is needed or that it's time
to buy more hardware.

All that's missing is monitoring of the WineTest report sizes (a patch is
coming), or each VM virtual disk size.

As before the Munin site is on vm2 (so not publicly accessible :-( ).

Regarding the (*), there is now an administrator-only page to retrieve the log
for a given period of time.

[Bug 39413] Make it easier to monitor the TestBot activity

https://bugs.winehq.org/show_bug.cgi?id=39413

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #2 from Austin English  ---
Closing.