[PATCH 1/3] testbot: Add a history Record when the Engine starts or stops.
A missing stop record will show that the TestBot Engine crashed (or was stopped abruptly). The gap between the stop and start records will also allow identifying the downtime. Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> --- testbot/bin/Engine.pl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/testbot/bin/Engine.pl b/testbot/bin/Engine.pl index 36a18ade..827879cf 100755 --- a/testbot/bin/Engine.pl +++ b/testbot/bin/Engine.pl @@ -749,6 +749,7 @@ sub main() LogMsg "Capping MaxRevertsWhileRunningVMs to MaxRevertingVMs ($MaxRevertsWhileRunningVMs)\n"; } $MaxVMsWhenIdle = $MaxActiveVMs if (!defined $MaxVMsWhenIdle); + SaveRecord('engine', 'start'); Cleanup(1); # Check for patches that arrived while the server was off. @@ -871,6 +872,7 @@ sub main() } } } + SaveRecord('engine', 'stop'); LogMsg "Normal WineTestBot Engine shutdown\n"; return 0; -- 2.15.1
Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> --- testbot/lib/WineTestBot/Activity.pm | 17 ++++++++++++++++- testbot/web/Activity.pl | 8 +++++++- testbot/web/WineTestBot.css | 1 + 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/testbot/lib/WineTestBot/Activity.pm b/testbot/lib/WineTestBot/Activity.pm index dcc58956..38cd556a 100644 --- a/testbot/lib/WineTestBot/Activity.pm +++ b/testbot/lib/WineTestBot/Activity.pm @@ -51,6 +51,7 @@ describing the TestBot activity. The structure is as follows: end => <EndTimestamp>, runnable => <RunnableTasksCount>, queued => <QueuedTasksCount>, + engine => <StartOrStop>, statusvms => { <VMName1> => { vm => <VMObject>, @@ -102,6 +103,20 @@ sub GetActivity($) { ($Group->{runnable}, $Group->{queued}) = split / /, $Record->Value; } + elsif ($Record->Type eq "engine" and $Record->Name =~ /^(?:start|stop)$/) + { + $Group->{engine} = $Record->Name; + foreach my $VM (@{$VMs->GetItems()}) + { + my $StatusVMs = ( $Group->{statusvms} ||= {} ); + my $VMStatus = ( $StatusVMs->{$VM->Name} ||= {} ); + $VMStatus->{vmstatus} = $VMStatus; + weaken($VMStatus->{vmstatus}); # avoid memory cycles + $VMStatus->{start} = $Group->{start}; + $VMStatus->{status} = "engine"; + $VMStatus->{rows} = 1; + } + } elsif ($Record->Type eq "vmstatus") { # Ignore retired / deleted VMs @@ -165,7 +180,7 @@ sub GetActivity($) { $LastVMStatus->{end} = $VMStatus->{start} if ($LastVMStatus); } - elsif ($LastVMStatus) + elsif ($LastVMStatus and $LastVMStatus->{status} ne "engine") { $VMStatus = $StatusVMs->{$VM->Name} = $LastVMStatus; $LastStatusVMs{$VM->Name}->{$VM->Name} = {merged => 1, vmstatus => $VMStatus}; diff --git a/testbot/web/Activity.pl b/testbot/web/Activity.pl index 677272f4..35843533 100644 --- a/testbot/web/Activity.pl +++ b/testbot/web/Activity.pl @@ -139,6 +139,12 @@ EOF next if (!$Group->{statusvms}); print "<tr><td>", _GetHtmlTime($Group->{start}), "</td>"; + if ($Group->{engine}) + { + print "<td class='Record RecordEngine'>$Group->{engine}</td>\n"; + print "<td colspan='", scalar(@SortedVMs), "'><hr></td>\n"; + next; + } if ($Group->{runnable} or $Group->{queued}) { print "<td class='Record'>", ($Group->{runnable} || 0), " / ", ($Group->{queued} || 0), "</td>"; @@ -229,7 +235,7 @@ sub GenerateFooter($) print "<span class='Record-running'>running</span> a task (in which case it links to it),<br>\n"; print "<span class='Record-dirty'>dirty</span> while the server is powering off the VM after a task or while it assesses its state on startup.</p>\n"; - print "<p>If no time is indicated then the VM remained in that state for less than 2 seconds. The tasks column indicates the number of runnable / queued tasks before that scheduling round.</p>\n"; + print "<p>If no time is indicated then the VM remained in that state for less than 2 seconds. The tasks column indicates the number of runnable / queued tasks before that scheduling round. A long horizontal bar indicates the TestBot server was restarted. </p>\n"; print "<p>The VM could also be <span class='Record-offline'>offline</span> due to a temporary issue,<br>\n"; print "or until the administrator can look at it for <span class='Record-maintenance'>maintenance</span>,<br>\n"; diff --git a/testbot/web/WineTestBot.css b/testbot/web/WineTestBot.css index 8c03a914..80de5668 100644 --- a/testbot/web/WineTestBot.css +++ b/testbot/web/WineTestBot.css @@ -344,6 +344,7 @@ a.title { color:white; text-decoration: none; } th.Record { text-align: center; } td.Record { text-align: center; } +.RecordEngine { font-style: italic; color: #601919; } .RecordHost { font-size: smaller; } .RecordJob { font-size: smaller; } .RecordDuration { } -- 2.15.1
The TestBot prepares VMs in advance so they are ready for the next Task. Some times it mispredicts which VM will be needed next. Higlight such cases on the Activity page so they can be identified and analyzed. Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> --- testbot/lib/WineTestBot/Activity.pm | 10 ++++++++++ testbot/web/Activity.pl | 6 ++++-- testbot/web/WineTestBot.css | 1 + 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/testbot/lib/WineTestBot/Activity.pm b/testbot/lib/WineTestBot/Activity.pm index 38cd556a..31930ed2 100644 --- a/testbot/lib/WineTestBot/Activity.pm +++ b/testbot/lib/WineTestBot/Activity.pm @@ -195,6 +195,16 @@ sub GetActivity($) $VMStatus->{vmstatus} = $VMStatus; weaken($VMStatus->{vmstatus}); # avoid memory cycles } + if ($LastVMStatus and $LastVMStatus->{status} ne $VMStatus->{status} and + # Ignore acts of administrator + $VMStatus->{status} !~ /^(?:maintenance|engine)$/ and + # And flag forbidden transitions + (($LastVMStatus->{status} eq "reverting" and $VMStatus->{status} ne "sleeping") or + ($LastVMStatus->{status} eq "sleeping" and $VMStatus->{status} !~ /^(?:idle|running)$/) or + ($LastVMStatus->{status} eq "idle" and $VMStatus->{status} ne "running"))) + { + $LastVMStatus->{mispredict} = 1; + } $LastStatusVMs{$VM->Name} = $StatusVMs; } } diff --git a/testbot/web/Activity.pl b/testbot/web/Activity.pl index 35843533..378c1105 100644 --- a/testbot/web/Activity.pl +++ b/testbot/web/Activity.pl @@ -160,11 +160,12 @@ EOF my $VMStatus = $Group->{statusvms}->{$VM->Name}; next if ($VMStatus->{merged}); - # Add borders to separate VM hosts + # Add borders to separate VM hosts and indicate anomalies. print "<td class='Record Record-$VMStatus->{status}"; my $Host = $VM->GetHost(); print " Record-left" if ($Col > 0 and $SortedVMs[$Col-1]->GetHost() ne $Host); print " Record-right" if ($Col+1 < @SortedVMs and $SortedVMs[$Col+1]->GetHost() ne $Host); + print " Record-miss" if ($VMStatus->{mispredict}); print "'"; print " rowspan='$VMStatus->{rows}'" if ($VMStatus->{rows} > 1); print ">"; @@ -235,7 +236,8 @@ sub GenerateFooter($) print "<span class='Record-running'>running</span> a task (in which case it links to it),<br>\n"; print "<span class='Record-dirty'>dirty</span> while the server is powering off the VM after a task or while it assesses its state on startup.</p>\n"; - print "<p>If no time is indicated then the VM remained in that state for less than 2 seconds. The tasks column indicates the number of runnable / queued tasks before that scheduling round. A long horizontal bar indicates the TestBot server was restarted. </p>\n"; + print "<p>If no time is indicated then the VM remained in that state for less than 2 seconds. The tasks column indicates the number of runnable / queued tasks before that scheduling round. A long horizontal bar indicates the TestBot server was restarted.</p>\n"; + print "<p>This <span class='Record Record-idle Record-miss'>border</span> indicates that the server threw away the VM's current state without using it.</p>\n"; print "<p>The VM could also be <span class='Record-offline'>offline</span> due to a temporary issue,<br>\n"; print "or until the administrator can look at it for <span class='Record-maintenance'>maintenance</span>,<br>\n"; diff --git a/testbot/web/WineTestBot.css b/testbot/web/WineTestBot.css index 80de5668..4d0548be 100644 --- a/testbot/web/WineTestBot.css +++ b/testbot/web/WineTestBot.css @@ -371,3 +371,4 @@ td.Record { text-align: center; } /* Special borders */ .Record.Record-left { border-left: thin solid #601919; } .Record.Record-right { border-right: thin solid #601919; } +.Record.Record-miss { border-top: thick dashed #ff6600; } -- 2.15.1
participants (1)
-
Francois Gouget