The main index page is nice to browse the jobs but does not allow getting a synthetic picture of what the TestBot is doing at a given time. Even the VM table at the bottom falls short as it only shows the current state and does not even show what task a given VM is running.
Each row of the new Activity page shows what each VM was doing at that time, with color coding, links to the relevant task for running VMs, and a count of the runnable and queued tasks. Going down the table shows past history, allowing one to see how the TestBot got in the current state. The VMs are also grouped by host which makes it easier to verify that the scheduler is obeying its operational parameters like the limits on simultaneous running tasks, reverting VMs, etc. While the activity page shows no sensitive information for now access to it is restricted to users with a TestBot account.
Signed-off-by: Francois Gouget fgouget@codeweavers.com --- testbot/lib/WineTestBot/Activity.pm | 199 ++++++++++++++++++++ testbot/lib/WineTestBot/CGI/PageBase.pm | 1 + testbot/lib/WineTestBot/Config.pm | 8 +- testbot/lib/WineTestBot/ConfigLocalTemplate.pl | 4 + testbot/lib/WineTestBot/RecordGroups.pm | 12 +- testbot/lib/WineTestBot/Records.pm | 24 ++- testbot/web/Activity.pl | 249 +++++++++++++++++++++++++ testbot/web/WineTestBot.css | 39 ++++ 8 files changed, 528 insertions(+), 8 deletions(-) create mode 100644 testbot/lib/WineTestBot/Activity.pm create mode 100644 testbot/web/Activity.pl
diff --git a/testbot/lib/WineTestBot/Activity.pm b/testbot/lib/WineTestBot/Activity.pm new file mode 100644 index 00000000..3a9081a3 --- /dev/null +++ b/testbot/lib/WineTestBot/Activity.pm @@ -0,0 +1,199 @@ +# -*- Mode: Perl; perl-indent-level: 2; indent-tabs-mode: nil -*- +# Copyright 2017 Francois Gouget +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA + +use strict; + +package WineTestBot::Activity; + +=head1 NAME + +WineTestBot::Activity - reconstruct the TestBot's activity from its history records. + +=cut + +use WineTestBot::Config; +use WineTestBot::Jobs; +use WineTestBot::RecordGroups; +use WineTestBot::Records; + +use vars qw (@ISA @EXPORT); + +require Exporter; +@ISA = qw(Exporter); +@EXPORT = qw(&GetActivity); + + +=pod +=over 12 + +=item C<GetActivity()> + +Loads the records for the specified VMs and processes them to build a structure +describing the TestBot activity. The structure is as follows: + + { <GroupNo1> => { + start => <StartTimestamp>, + end => <EndTimestamp>, + runnable => <RunnableTasksCount>, + queued => <QueuedTasksCount>, + statusvms => { + <VMName1> => { + vm => <VMObject>, + status => <VMStatus>, + value => <RecordValue>, + task => <TaskObjectIfAppopriate>, + start => <StartTimestamp>, + end => <EndTimestamp>, + rows => <NbRows>, + }, + <VMName2> => { + ... + }, + }, + }, + <GroupNo2> => { + ... + }, + ... + } + +=back +=cut + +sub GetActivity($) +{ + my ($VMs) = @_; + my ($Activity, $Counters) = ({}, {}); + + ### First load all the RecordGroups + my $RecordGroups = CreateRecordGroups(); + $Counters->{recordgroups} = $RecordGroups->GetItemsCount(); + foreach my $RecordGroup (@{$RecordGroups->GetItems()}) + { + $Activity->{$RecordGroup->Id} = { start => $RecordGroup->Timestamp }; + } + + ### And then load all the Records in one go + # Loading the whole table at once is more efficient than loading it piecemeal + # one RecordGroup at a time. + + my $Jobs = CreateJobs(); + my $Records = CreateRecords(); + $Counters->{records} = $Records->GetItemsCount(); + foreach my $Record (@{$Records->GetItems()}) + { + my $Group = $Activity->{$Record->RecordGroupId}; + if ($Record->Type eq "tasks" and $Record->Name eq "counters") + { + ($Group->{runnable}, $Group->{queued}) = split / /, $Record->Value; + } + elsif ($Record->Type eq "vmstatus") + { + # Ignore retired / deleted VMs + my ($RecordName, $RecordHost) = split / /, $Record->Name; + next if (!$VMs->ItemExists($RecordName)); + + my $StatusVMs = ( $Group->{statusvms} ||= {} ); + my $VMStatus = ( $StatusVMs->{$RecordName} ||= {} ); + + $VMStatus->{host} = $RecordHost; + $VMStatus->{vmstatus} = $VMStatus; + $VMStatus->{start} = $Group->{start}; + my ($Status, @Extra) = split / /, $Record->Value; + $VMStatus->{status} = $Status; + $VMStatus->{rows} = 1; + + if ($Status eq "running") + { + $VMStatus->{job} = $Jobs->GetItem($Extra[0]); + $VMStatus->{step} = $VMStatus->{job}->Steps->GetItem($Extra[1]) if ($VMStatus->{job}); + $VMStatus->{task} = $VMStatus->{step}->Tasks->GetItem($Extra[2]) if ($VMStatus->{step}); + } + elsif (@Extra) + { + # @Extra contains details about the current status, such as what + # type of process a dirty VM is running, or how it came to be dirty + # in the first place. + $VMStatus->{details} = join(" ", @Extra); + } + } + } + + ### Fill the holes in the table, compute end times, etc. + + my ($LastGroup, %LastStatusVMs); + foreach my $RecordGroup (sort CompareRecordGroups @{$RecordGroups->GetItems()}) + { + my $Group = $Activity->{$RecordGroup->Id}; + my $StatusVMs = $Group->{statusvms}; + next if (!$StatusVMs); + if ($LastGroup) + { + $LastGroup->{end} = $Group->{start}; + foreach my $Counter ('runnable', 'queued') + { + if (!exists $Group->{$Counter} and exists $LastGroup->{$Counter}) + { + $Group->{$Counter} = $LastGroup->{$Counter}; + } + } + } + $LastGroup = $Group; + + foreach my $VM (@{$VMs->GetItems()}) + { + my $LastVMStatus = $LastStatusVMs{$VM->Name} ? $LastStatusVMs{$VM->Name}->{$VM->Name} : undef; + + my $VMStatus = $StatusVMs->{$VM->Name}; + if ($VMStatus) + { + $LastVMStatus->{end} = $VMStatus->{start} if ($LastVMStatus); + } + elsif ($LastVMStatus) + { + $VMStatus = $StatusVMs->{$VM->Name} = $LastVMStatus; + $LastStatusVMs{$VM->Name}->{$VM->Name} = {merged => 1, vmstatus => $VMStatus}; + $VMStatus->{rows}++; + } + else + { + $VMStatus = $StatusVMs->{$VM->Name} = { + start => $Group->{start}, + status => "unknown", + rows => 1}; + $VMStatus->{vmstatus} = $VMStatus; + } + $LastStatusVMs{$VM->Name} = $StatusVMs; + } + } + $LastGroup->{end} = time() if ($LastGroup); + + foreach my $VM (@{$VMs->GetItems()}) + { + my $LastVMStatus = $LastStatusVMs{$VM->Name}->{$VM->Name}; + next if (!$LastVMStatus); + $LastVMStatus->{end} = time(); + if ($LastVMStatus->{status} eq "unknown") + { + $LastVMStatus->{status} = $VM->Status; + } + } + + return ($Activity, $Counters); +} + +1; diff --git a/testbot/lib/WineTestBot/CGI/PageBase.pm b/testbot/lib/WineTestBot/CGI/PageBase.pm index 4ae190e2..fe03a47e 100644 --- a/testbot/lib/WineTestBot/CGI/PageBase.pm +++ b/testbot/lib/WineTestBot/CGI/PageBase.pm @@ -265,6 +265,7 @@ EOF print " <li class='divider'> </li>\n"; print " <li><p><a href='", MakeSecureURL("/Submit.pl"), "'>Submit job</a></p></li>\n"; + print " <li><p><a href='/Activity.pl'>Activity</a></p></li>\n"; print " <li class='divider'> </li>\n"; print " <li><p><a href='", MakeSecureURL("/Logout.pl"), "'>Log out"; if (defined($Session)) diff --git a/testbot/lib/WineTestBot/Config.pm b/testbot/lib/WineTestBot/Config.pm index 8388a0ed..b1024b5c 100644 --- a/testbot/lib/WineTestBot/Config.pm +++ b/testbot/lib/WineTestBot/Config.pm @@ -35,8 +35,8 @@ use vars qw (@ISA @EXPORT @EXPORT_OK $UseSSL $LogDir $DataDir $BinDir $ProjectName $PatchesMailingList $LDAPServer $LDAPBindDN $LDAPSearchBase $LDAPSearchFilter $LDAPRealNameAttribute $LDAPEMailAttribute $AgentPort $Tunnel - $TunnelDefaults $JobPurgeDays $JobArchiveDays $WebHostName - $RegistrationQ $RegistrationARE); + $TunnelDefaults $PrettyHostNames $JobPurgeDays $JobArchiveDays + $WebHostName $RegistrationQ $RegistrationARE);
require Exporter; @ISA = qw(Exporter); @@ -49,8 +49,8 @@ require Exporter; $TagPrefix $ProjectName $PatchesMailingList $LDAPServer $LDAPBindDN $LDAPSearchBase $LDAPSearchFilter $LDAPRealNameAttribute $LDAPEMailAttribute $AgentPort $Tunnel - $TunnelDefaults $JobPurgeDays $JobArchiveDays $WebHostName - $RegistrationQ $RegistrationARE); + $TunnelDefaults $PrettyHostNames $JobPurgeDays $JobArchiveDays + $WebHostName $RegistrationQ $RegistrationARE); @EXPORT_OK = qw($DbDataSource $DbUsername $DbPassword);
if ($::RootDir !~ m=^/=) diff --git a/testbot/lib/WineTestBot/ConfigLocalTemplate.pl b/testbot/lib/WineTestBot/ConfigLocalTemplate.pl index 6ac6b44c..c7514179 100644 --- a/testbot/lib/WineTestBot/ConfigLocalTemplate.pl +++ b/testbot/lib/WineTestBot/ConfigLocalTemplate.pl @@ -105,5 +105,9 @@ $WineTestBot::Config::Tunnel = undef; # - local_username $WineTestBot::Config::TunnelDefaults = undef;
+# If set this remaps the hostnames returned by $VM->GetHost() into more +# user friendly hostnames. For instance: +# "gateway:port1" => "vm1" +$WineTestBot::Config::PrettyHostNames = undef;
1; diff --git a/testbot/lib/WineTestBot/RecordGroups.pm b/testbot/lib/WineTestBot/RecordGroups.pm index b33d16eb..fc1c2528 100644 --- a/testbot/lib/WineTestBot/RecordGroups.pm +++ b/testbot/lib/WineTestBot/RecordGroups.pm @@ -67,7 +67,7 @@ use vars qw (@ISA @EXPORT @PropertyDescriptors);
require Exporter; @ISA = qw(WineTestBot::WineTestBotCollection Exporter); -@EXPORT = qw(&CreateRecordGroups &SaveRecord); +@EXPORT = qw(&CreateRecordGroups &CompareRecordGroups &SaveRecord);
BEGIN @@ -93,6 +93,16 @@ sub CreateRecordGroups(;$) @PropertyDescriptors, $ScopeObject); }
+sub CompareRecordGroups($$) +{ + my ($a, $b) = @_; + + # The Id will wrap eventually so sort by Timestamp + # and only use the Id to break ties. + return $a->Timestamp <=> $b->Timestamp || + $a->Id <=> $b->Id; +} + =pod =over 12
diff --git a/testbot/lib/WineTestBot/Records.pm b/testbot/lib/WineTestBot/Records.pm index dd0fec97..ac1c0faf 100644 --- a/testbot/lib/WineTestBot/Records.pm +++ b/testbot/lib/WineTestBot/Records.pm @@ -67,7 +67,7 @@ use ObjectModel::EnumPropertyDescriptor; use ObjectModel::PropertyDescriptor; use WineTestBot::WineTestBotObjects;
-use vars qw (@ISA @EXPORT @PropertyDescriptors); +use vars qw (@ISA @EXPORT @PropertyDescriptors @FlatPropertyDescriptors);
require Exporter; @ISA = qw(WineTestBot::WineTestBotCollection Exporter); @@ -81,6 +81,10 @@ BEGIN CreateBasicPropertyDescriptor("Name", "Name", 1, 1, "A", 96), CreateBasicPropertyDescriptor("Value", "Value", !1, !1, "A", 64), ); + @FlatPropertyDescriptors = ( + CreateBasicPropertyDescriptor("RecordGroupId", "Record group id", 1, 1, "S", 6), + @PropertyDescriptors + ); }
sub CreateItem($) @@ -90,12 +94,26 @@ sub CreateItem($) return WineTestBot::Record->new($self); }
+=pod +=over 12 + +=item C<CreateRecords()> + +Creates a collection containing the records of the specified RecordGroup. In +this case the Record objects have no RecordGroupId property. + +If no RecordGroup is specified all the table records are returned and the +Record objects have a RecordGroupId property. + +=back +=cut + sub CreateRecords(;$$) { my ($ScopeObject, $RecordGroup) = @_; return WineTestBot::Records->new("Records", "Records", "Record", - @PropertyDescriptors, $ScopeObject, - $RecordGroup); + $RecordGroup ? @PropertyDescriptors : @FlatPropertyDescriptors, + $ScopeObject, $RecordGroup); }
=pod diff --git a/testbot/web/Activity.pl b/testbot/web/Activity.pl new file mode 100644 index 00000000..677272f4 --- /dev/null +++ b/testbot/web/Activity.pl @@ -0,0 +1,249 @@ +# -*- Mode: Perl; perl-indent-level: 2; indent-tabs-mode: nil -*- +# Shows the VM activity +# +# Copyright 2017 Francois Gouget +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA + +use strict; + +package ActivityPage; + +use POSIX qw(strftime); +use URI::Escape; + +use ObjectModel::CGI::Page; +use WineTestBot::Config; +use WineTestBot::Activity; +use WineTestBot::Log; +use WineTestBot::VMs; + +@ActivityPage::ISA = qw(ObjectModel::CGI::Page); + +sub _initialize($$$) +{ + my ($self, $Request, $RequiredRole) = @_; + + $self->{start} = Time(); + $self->SUPER::_initialize($Request, $RequiredRole); +} + +sub GeneratePage($) +{ + my ($self) = @_; + + $self->{Request}->headers_out->add("Refresh", "60"); + $self->SUPER::GeneratePage(); +} + +sub _GetHtmlTime($) +{ + my ($Timestamp) = @_; + return "<noscript><div>", + strftime("%d %H:%M:%S", localtime($Timestamp)), "</div></noscript>\n" . + "<script type='text/javascript'><!--\n" . + "ShowDateTime($Timestamp);\n" . + "//--></script>"; +} + +sub _GetHtmlDuration($) +{ + my ($Secs) = @_; + + return "" if ($Secs < 2); + my $Mins = int($Secs / 60); + my $Hours = int($Mins / 60); + + my @Parts; + push @Parts, "${Hours}h" if ($Hours); + push @Parts, "${Mins}m" if ($Mins %= 60); + push @Parts, "${Secs}s" if ($Secs %= 60); + return "<span class='RecordDuration'>". join(" ", @Parts) ."</span>"; +} + +sub _CompareVMs() +{ + my ($aHost, $bHost) = ($a->GetHost(), $b->GetHost()); + if ($PrettyHostNames) + { + $aHost = $PrettyHostNames->{$aHost} || $aHost; + $bHost = $PrettyHostNames->{$bHost} || $bHost; + } + return $aHost cmp $bHost || $a->Name cmp $b->Name; +} + +sub GenerateBody($) +{ + my ($self) = @_; + + print "<h1>${ProjectName} Test Bot activity</h1>\n"; + print "<div class='Content'>\n"; + + print <<"EOF"; +<script type='text/javascript'><!--\ +function Pad2(n) +{ + return n < 10 ? '0' + n : n; +} +function ShowDateTime(Sec1970) +{ + var Dt = new Date(Sec1970 * 1000); + document.write(Pad2(Dt.getDate()) + ' ' + Pad2(Dt.getHours()) + ':' + + Pad2(Dt.getMinutes()) + ':' + Pad2(Dt.getSeconds())); +} +//--></script> +EOF + + ### Get the sorted VMs list + + my $VMs = CreateVMs(); + $VMs->FilterEnabledRole(); + my @SortedVMs = sort _CompareVMs @{$VMs->GetItems()}; + + ### Generate the table header : one column per VM + + print "<div class='CollectionBlock'><table>\n"; + print "<thead><tr><th class='Record'>Time</th>\n"; + print "<th class='Record'><a class='title' title='Runnable / queued task count before scheduling'>Tasks</a></th>\n"; + foreach my $VM (@SortedVMs) + { + my $Host = $VM->GetHost(); + if ($PrettyHostNames and defined $PrettyHostNames->{$Host}) + { + $Host = $PrettyHostNames->{$Host}; + } + $Host = " on $Host" if ($Host ne ""); + print "<th class='Record'>", $VM->Name, "$Host</th>\n"; + } + print "</tr></thead>\n"; + + ### Generate the HTML table with the newest record first + + print "<tbody>\n"; + my ($Activity, $_Counters) = GetActivity($VMs); + foreach my $GroupNo (sort { $b <=> $a } keys %$Activity) + { + my $Group = $Activity->{$GroupNo}; + next if (!$Group->{statusvms}); + + print "<tr><td>", _GetHtmlTime($Group->{start}), "</td>"; + if ($Group->{runnable} or $Group->{queued}) + { + print "<td class='Record'>", ($Group->{runnable} || 0), " / ", ($Group->{queued} || 0), "</td>"; + } + else + { + print "<td class='Record'> </td>"; + } + + foreach my $Col (0..@SortedVMs-1) + { + my $VM = $SortedVMs[$Col]; + my $VMStatus = $Group->{statusvms}->{$VM->Name}; + next if ($VMStatus->{merged}); + + # Add borders to separate VM hosts + print "<td class='Record Record-$VMStatus->{status}"; + my $Host = $VM->GetHost(); + print " Record-left" if ($Col > 0 and $SortedVMs[$Col-1]->GetHost() ne $Host); + print " Record-right" if ($Col+1 < @SortedVMs and $SortedVMs[$Col+1]->GetHost() ne $Host); + print "'"; + print " rowspan='$VMStatus->{rows}'" if ($VMStatus->{rows} > 1); + print ">"; + + my $Label; + if ($VMStatus->{task}) + { + $Label = "<span class='RecordJob'>". $VMStatus->{job}->Id .":</span>"; + if ($VMStatus->{step}->Type eq "build") + { + $Label .= " Build"; + } + elsif ($VMStatus->{step}->Type eq "reconfig") + { + $Label .= " Reconfig"; + } + elsif ($VMStatus->{step}->Type eq "suite") + { + $Label .= " WineTest"; + } + else + { + $Label .= " ". $VMStatus->{step}->FileName; + if ($VMStatus->{task}->CmdLineArg =~ /^\w+$/ and + $Label =~ s/_(?:cross)?test.exe$//) + { + $Label .= ":". $VMStatus->{task}->CmdLineArg; + } + } + $Label = "<a href='/JobDetails.pl?Key=". $VMStatus->{job}->Id ."#k". ($VMStatus->{step}->No * 100 + $VMStatus->{task}->No) ."'>$Label</a>"; + } + elsif ($VMStatus->{status} eq "dirty") + { + $Label = $VMStatus->{details} || $VMStatus->{status}; + } + else + { + $Label = $VMStatus->{status}; + } + if ($VMStatus->{host} and $VMStatus->{host} ne $VM->GetHost()) + { + my $Host = $VMStatus->{host}; + # Here we keep the original hostname if the pretty one is empty + $Host = $PrettyHostNames->{$Host} || $Host if ($PrettyHostNames); + $Label = "<span class='RecordHost'>(on $Host)</span><br>$Label"; + } + print "$Label ", _GetHtmlDuration($VMStatus->{end} - $VMStatus->{start}); + print "</td>\n"; + } + print "</tr>\n"; + } + + ### Generate the table footer + + print "</tbody></table></div>\n"; +} + +sub GenerateFooter($) +{ + my ($self) = @_; + print "<p></p><div class='CollectionBlock'><table>\n"; + print "<thead><tr><th class='Record'>Legend</th></tr></thead>\n"; + print "<tbody><tr><td class='Record'>\n"; + + print "<p>The VM typically goes through these states: <span class='Record-off'>off</span>,<br>\n"; + print "<span class='Record-reverting'>reverting</span> to the proper test configuration,<br>\n"; + print "<span class='Record-sleeping'>sleeping</span> until the server can connect to it,<br>\n"; + print "<span class='Record-running'>running</span> a task (in which case it links to it),<br>\n"; + print "<span class='Record-dirty'>dirty</span> while the server is powering off the VM after a task or while it assesses its state on startup.</p>\n"; + + print "<p>If no time is indicated then the VM remained in that state for less than 2 seconds. The tasks column indicates the number of runnable / queued tasks before that scheduling round.</p>\n"; + + print "<p>The VM could also be <span class='Record-offline'>offline</span> due to a temporary issue,<br>\n"; + print "or until the administrator can look at it for <span class='Record-maintenance'>maintenance</span>,<br>\n"; + print "or <span class='Record-retired'>retired</span> prior to a possible<br>\n"; + print "<span class='Record-deleted'>deletion</span>.</p>\n"; + + print "</td></tr></tbody>\n"; + print "</tbody></table></div>\n"; + print "<p class='GeneralFooterText'>Generated in ", Elapsed($self->{start}), " s</p>\n"; +} + +package main; + +my $Request = shift; + +my $ActivityPage = ActivityPage->new($Request, "wine-devel"); +$ActivityPage->GeneratePage(); diff --git a/testbot/web/WineTestBot.css b/testbot/web/WineTestBot.css index 5d25c42f..bb80e7f2 100644 --- a/testbot/web/WineTestBot.css +++ b/testbot/web/WineTestBot.css @@ -282,6 +282,14 @@ h2 margin-left: 4px; }
+.GeneralFooterText +{ + text-align: right; + font-size: smaller; + font-style: italic; +} + + .Screenshot { display: block; @@ -330,3 +338,34 @@ pre .testfail { color: red; } .boterror { color: #e55600; } .canceled { color: black; } + +a.title { color:white; text-decoration: none; } + +th.Record { text-align: center; } +td.Record { text-align: center; } +.RecordHost { font-size: smaller; } +.RecordJob { font-size: smaller; } +.RecordDuration { } + +.Record-start { background-color: white; } +.Record-off { color: #c0c0c0; } +.Record-unknown { background-color: #c0c0c0; } +.Record-offline { background-color: red; } + +/* Greens for preparing VMs */ +.Record-reverting { background-color: #70ff54; } +.Record-sleeping { background-color: #a3ff7a; } +.Record-idle { background-color: #ccff99; } + +/* Yellows for running VMs */ +.Record-running { background-color: #ffcc00; } +.Record-dirty { background-color: #ffe633; } + +/* Blues for disabled VMs */ +.Record-maintenance { background-color: #adccef; } +.Record-retired { background-color: #72a7e4; } +.Record-deleted { background-color: #246bbc; } + +/* Special borders */ +.Record.Record-left { border-left: thin solid #601919; } +.Record.Record-right { border-right: thin solid #601919; }