Francois Gouget : testbot/Engine: Put a VM offline if it repeatedly times out during revert.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: 3f26a883efbff95dde8c25fd6bef67f35bd85e3e
URL:    https://source.winehq.org/git/tools.git/?a=commit;h=3f26a883efbff95dde8c25fd...

Author: Francois Gouget <fgouget(a)codeweavers.com>
Date:   Fri Sep 20 15:04:53 2019 +0200

testbot/Engine: Put a VM offline if it repeatedly times out during revert.

Putting it in maintenance mode makes it unusable until an administrator
looks at it which could take a long time when quite often the timeout is
just caused by high load on the host.
Now that LibvirtTool 'rate-limits' how often it puts VMs that keep
having errors back online, we can simply put such VMs offline and let
LibvirtTool deal with them.

Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>

---

 testbot/lib/WineTestBot/Engine/Scheduler.pm | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/testbot/lib/WineTestBot/Engine/Scheduler.pm b/testbot/lib/WineTestBot/Engine/Scheduler.pm
index 569dff7..e9a3ff9 100644
--- a/testbot/lib/WineTestBot/Engine/Scheduler.pm
+++ b/testbot/lib/WineTestBot/Engine/Scheduler.pm
@@ -270,31 +270,28 @@ sub _CheckAndClassifyVMs()
       {
         # The child process got stuck!
         $FoundVMErrors = 1;
-        my $NewStatus = "dirty";
         if ($VM->Status eq "reverting" or $VM->Status eq "sleeping")
         {
           my $Errors = ($VM->Errors || 0) + 1;
           $VM->Errors($Errors);
-          if ($Errors >= $MaxVMErrors)
-          {
-            $NewStatus = "maintenance";
-            NotifyAdministrator("Putting the $VMKey VM in maintenance mode",
-                                "The last $Errors revert operations timed out.\n\n".
-                                "No further operation will be attempted until an administrator has put\n".
-                                "the VM back online.");
-            $Sched->{busyvms}->{$VMKey} = 1;
-          }
+          $VM->Status("offline");
+          NotifyAdministrator("Putting the $VMKey VM offline",
+                              "The last $Errors revert operations timed out.\n\n".
+                              "This may be because of some transient load on the VM host but if not\n".
+                              "an administrator should look into it. In any case the TestBot will try\n".
+                              "to regain access to the VM.");
+          $Sched->{busyvms}->{$VMKey} = 1;
         }
-        $VM->Status($NewStatus);
-        $VM->KillChild();
-        $VM->Save();
-        $VM->RecordResult($Sched->{records}, "boterror stuck process");
-        if ($NewStatus eq "dirty")
+        else
         {
+          $VM->Status("dirty");
           $Sched->{lambvms}->{$VMKey} = 1;
           $Host->{dirty}++;
           $Host->{active}++;
         }
+        $VM->KillChild();
+        $VM->Save();
+        $VM->RecordResult($Sched->{records}, "boterror stuck process");
       }
       elsif ($VM->Status =~ /^(?:dirty|running|reverting)$/)
       {