Francois Gouget : testbot/TestWTBS: Check that no extra test units have been run.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: d03ed1fe17bf4a73232a51bdff6080f5cc92f53d
URL:    https://source.winehq.org/git/tools.git/?a=commit;h=d03ed1fe17bf4a73232a51bd...

Author: Francois Gouget <fgouget(a)codeweavers.com>
Date:   Mon Feb 22 06:26:12 2021 +0100

testbot/TestWTBS: Check that no extra test units have been run.

TestUnits is now an exhaustive list of test units that are expected to
be run. Any extra test unit is reported as a failure.
To compensate one can now add a module name to accept any test unit from
that module. This allows dealing with the test=module configurations.

Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>

---

 testbot/tests/TestWTBS | 34 +++++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)

diff --git a/testbot/tests/TestWTBS b/testbot/tests/TestWTBS
index 8a77b00..1467c4b 100755
--- a/testbot/tests/TestWTBS
+++ b/testbot/tests/TestWTBS
@@ -1038,11 +1038,15 @@ p win.HasTask 0
 
 =item <tests.TestUnits>
 
-Checks that the TestBot ran at least the test units mentioned in this
-space-separated list on the VMs matching the specified category.
+Checks that the TestBot ran the test units mentioned in this space-separated
+list on the VMs matching the specified category. Each entry is either of the
+form module:unit, or just module in which case it matches any test unit of the
+specified module. Each entry in the list must match at least one test unit.
+Two entries can match the same test unit(s).
+This is useful when test=module, in which case the TestBot may run many tests.
 
 For instance:
-p win.TestUnits cabinet:extract cabinet:fdi
+p win.TestUnits cabinet:newunit cabinet
 
 =item <patch.Grep>
 =item <patch.GrepV>
@@ -1168,11 +1172,31 @@ sub CheckJobTree($;$)
         my $MissionInfo = $TestInfo->{$MissionType};
         next if (!CheckValue($MissionInfo->{TestUnits}));
 
-        foreach my $TestUnit (split / +/, $MissionInfo->{TestUnits})
+        foreach my $TestGrep (split / +/, $MissionInfo->{TestUnits})
         {
-          ok($TestUnits->{$MissionType}->{$TestUnit}, "Check that $TestUnit was tested by $MissionType VMs for job $JobId")
+          if ($TestGrep =~ /:/)
+          {
+            if (exists $TestUnits->{$MissionType}->{$TestGrep})
+            {
+              $TestUnits->{$MissionType}->{$TestGrep} = 0;
+              next;
+            }
+            fail("Check that $MissionType VMs ran $TestGrep for job $JobId")
+                or diag("TestUnits=", join(" ", sort keys %{$TestUnits->{$MissionType}}));
+          }
+          my $Matched;
+          foreach my $TestUnit (keys %{$TestUnits->{$MissionType}})
+          {
+            next if ($TestUnit !~/^\Q$TestGrep:\E/);
+            $TestUnits->{$MissionType}->{$TestUnit} = 0;
+            $Matched = 1;
+          }
+          ok($Matched, "Check that $MissionType VMs ran $TestGrep for job $JobId")
               or diag("TestUnits=", join(" ", sort keys %{$TestUnits->{$MissionType}}));
         }
+        my @ExtraUnits = grep { $TestUnits->{$MissionType}->{$_} } keys %{$TestUnits->{$MissionType}};
+        ok(!@ExtraUnits, "Check that $MissionType VMs ran no extra test units")
+            or diag("Extraneous test runs=". join(" ", sort @ExtraUnits));
       }
     }