The script does not know how many tests to expect but it can reasonably expect that $maxfailedtests is only a small fraction of the expected number of tests.
Signed-off-by: Francois Gouget fgouget@codeweavers.com --- In the past WineTest would always show the names of the skipped dlls. But that made the reports cumbersome for the TestBot (5549ad21b33c) so when given a list of tests to run (as opposed to a list of tests to skip), it only prints information about those. As a result dissect does not know how many tests were skipped (it never really did, it only had a count of skipped test executables).
It would probably make sense for WineTest to add a line like "skipped N test executables" in all cases so dissect can better check the report is complete.
But dissect is a server-side script and thus should not rely on the client sending truthful and correct information. So I think there is room for this patch or one like it.
This version of the patch does not guarantee that the report is complete but it rules out obvious mistakes.
The $maxfailedtests multiplier can be adjusted as desired as long as it's less than 10: $maxfailedtests can be up to 50+20 and we have about 725 test units.
An alternative would be to add a $mintestresults setting in winetest.conf. It would always be out of date but given that the number of tests would presumably always go up that should be ok. Setting it to something large like 700 would work as long as we have more than 710 tests (700+$maxuserskips; currently we seem to have about 725). --- winetest/dissect | 1 + 1 file changed, 1 insertion(+)
diff --git a/winetest/dissect b/winetest/dissect index 43f7790d2..da4ad636a 100755 --- a/winetest/dissect +++ b/winetest/dissect @@ -903,6 +903,7 @@ close_test_unit(1); close SUM or mydie "error writing to '$tmpdir/summary.txt': $!"; close IN;
+mydie "too few tests run (", @boxes-1, ")" if @boxes <= $maxfailedtests * 5; mydie "report reached file size limit (runaway test?)" if -s $report >= $maxfilesize;