Module: tools
Branch: master
Commit: a26e789fad7cc0486b6e967ad97b4113082aa24c
URL: https://source.winehq.org/git/tools.git/?a=commit;h=a26e789fad7cc0486b6e967…
Author: Francois Gouget <fgouget(a)codeweavers.com>
Date: Mon Feb 8 15:21:56 2021 +0100
testbot/reporttest: Match the tests to the wtbsuite.
Where a test mirrors a wtbsuite test, add the latter's commit message as
a comment to make it simpler to compare the two.
Also tweak some comments to better describe the tests.
Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>
---
testbot/src/reporttest/report.template | 30 ++++++++++++++++++++++++------
1 file changed, 24 insertions(+), 6 deletions(-)
diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index c08b4d4..8da3d82 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -209,8 +209,11 @@ Dll info:
xmllite=1.0.1018.0
Test output:
stub advapi32:cred
+----- WTBS Simple test patch
+----- Expected assessement: Success
advapi32:crypt start dlls/advapi32/tests/crypt.c -
+----- WTBS Simple test failure
----- A test unit with one failed test
----- Expected assessement: 1 failure
This also has a valid source file name which one can link to.
@@ -219,10 +222,11 @@ crypt.c:1: Test failed: Something wrong
advapi32:crypt:031c done (1) in 0s
advapi32:crypt_lmhash start dlls/advapi32/tests/crypt_lmhash.c -
+----- WTBS Simple diacritic failure
----- A test unit with two failed tests
------ Expected assessement: 2 failures
+----- Expected assessement: 2 failures despite exit code
crypt_lmhash.c:10: Test failed: Something wrong
-crypt_lmhash.c:20: Test failed: Second failure here
+crypt_lmhash.c:20: Test failed: Second failure with added diacritics: ??????????
07c0:crypt_lmhash: 95 tests executed (0 marked as todo, 2 failures), 0 skipped.
advapi32:crypt_lmhash:07c0 done (1) in 0s
@@ -246,7 +250,8 @@ Note that this also applies to the todo and skip counts.
advapi32:crypt_md5:087c done (3) in 0s
advapi32:crypt_sha start dlls/advapi32/tests/crypt_sha.c -
------ A test unit with a failed subprocess test ignored by the parent
+----- A test unit with a failed subprocess test ignored by the parent test
+----- summary but accounted for in the exit code.
----- Expected assessement: 1 failure
crypt_sha.c:20: Test failed: A failure here, presumably in the subprocess
0456:crypt_sha: 1 tests executed (0 marked as todo, 1 failure), 0 skipped.
@@ -256,6 +261,7 @@ subprocess.
advapi32:crypt_sha:08f8 done (3) in 0s
advapi32:eventlog start dlls/advapi32/tests/eventlog.c -
+----- WTBS Unreported test failures
----- A test unit with an unreported failed test
----- Expected assessement: 1 failure, "Unreported failure" error
eventlog.c:20: Test failed: Something wrong
@@ -263,6 +269,7 @@ eventlog.c:20: Test failed: Something wrong
advapi32:eventlog:088c done (3) in 4s
advapi32:lsa start dlls/advapi32/tests/lsa.c -
+----- WTBS Missing test summary line
----- A test unit with no test summary line
----- Expected assessement: "Missing test summary" error
advapi32:lsa:083c done (0) in 0s
@@ -308,6 +315,7 @@ files.c:20: Test failed: A failure in the main process
advpack:files:0880 done (3) in 0s
advpack:install start dlls/advpack/tests/install.c -
+----- WTBS Simple test crash
----- A test unit with a crash
----- Expected assessement: Crash
07b0:install: unhandled exception c0000005 at 0040167C
@@ -371,6 +379,8 @@ comcat:comcat start dlls/comcat/tests/comcat.c -
comcat:comcat:0948 done (0) in 0s
comctl32:animate start dlls/comctl32/tests/animate.c -
+----- WTBS Timeout in a test
+----- WTBS Timeout in a test - barely
----- A test unit with a timeout
----- Expected assessement: Timeout
comctl32:animate:0b14 done (258) in 42s
@@ -380,7 +390,10 @@ comctl32:button skipped dlls/comctl32/tests/button.c -
----- Expected assessement: Skipped by user
comctl32:comboex start dlls/comctl32/tests/comboex.c -
------ A test unit with failed, todo and skipped tests in the main process
+----- WTBS Simple success in a todo test
+----- WTBS Simple trace, ok, todo, skip sampler
+----- A test unit with a subprocess and failed, todo and skipped tests in the
+----- main process.
----- Expected assessement: 2 failures, 1 todo, 1 skip
comboex.c:40: Test failed: A failure in the main process
comboex.c:41: Test succeeded inside todo block: Success in the main process
@@ -394,6 +407,7 @@ comctl32:comboex:0c9c done (2) in 0s
comctl32:datetime start dlls/comctl32/tests/datetime.c -
----- A test unit reporting a failure but with no test failed line
----- Expected assessement: "Missing test failed line" error
+----- + TestFailures=2 (1 for the exit code + 1 for the missing failure line)
0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped.
comctl32:datetime:0998 done (1) in 0s
@@ -425,8 +439,10 @@ comctl32:ipaddress start dlls/comctl32/tests/ipaddress.c -
Issued upon seeing the next start line.
0998:ipaddress: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
+----- Keep this stub in case the previous test is mishandled
stub comctl32:listview
+----- WTBS A test unit with no start line
----- A test unit with a missing start line
----- Expected assessement: "Missing start line" error
0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
@@ -490,6 +506,7 @@ crashed process which makes sense since it did not crash.
comctl32:subclass:07b0 done (1) in 0s
comctl32:syslink start dlls/comctl32/tests/syslink.c -
+----- WTBS Garbled trace, ok, todo, skip sampler
----- A test unit with garbled test lines
----- Expected assessement: 2 failures, 1 todo, 1 skip
Multi-threaded and multi-process tests should really try to avoid races in
@@ -518,13 +535,14 @@ comctl32:toolbar done (-1073741819) in 0s
stub comctl32:tooltips
comctl32:trackbar start dlls/comctl32/tests/trackbar.c -
+----- WTBS Test failures, todos and skips in a subtest
----- A test unit with a subtest
+----- Expected assessement: 2 failures, 1 todo, 1 skip
trackbar.c:496: Subtest driver
driver.c:40: Test failed: Something wrong
driver.c:41: Test succeeded inside todo block: Something right!
driver.c:42: Test marked todo: Hello Wine!
driver.c:43: Tests skipped: Another time
------ Expected assessement: 2 failures, 1 todo, 1 skip
07b1:trackbar: 3 tests executed (1 marked as todo, 2 failures), 1 skipped.
comctl32:trackbar:07b1 done (2) in 0s
@@ -1011,7 +1029,7 @@ stub xcopy.exe:xcopy
stub xmllite:reader
xmllite:writer start dlls/xmllite/tests/writer.c -
------ A test unit with a misplaced test failed line
+----- A test unit with a misplaced (late) test failed line
----- Expected assessement: "Misplaced test failed line" error
See the similar scenario before but note the lack of the extra test summary line
here. Also beware of the end-of-file.
Module: tools
Branch: master
Commit: 5985ad320c525d44998564e88d6989cba9f5a925
URL: https://source.winehq.org/git/tools.git/?a=commit;h=5985ad320c525d44998564e…
Author: Francois Gouget <fgouget(a)codeweavers.com>
Date: Mon Feb 8 15:21:53 2021 +0100
testbot/reporttest: Remove tests for legacy TestLauncher and Wine versions.
Remove tests for line formats that neither the TestBot nor the WineTest
website support anymore. This is the case of lines missing pids, run
times or using signed exit codes.
Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>
---
testbot/src/reporttest/report.template | 64 +++++++---------------------------
1 file changed, 12 insertions(+), 52 deletions(-)
diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index b541e41..c08b4d4 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -313,29 +313,10 @@ advpack:install start dlls/advpack/tests/install.c -
07b0:install: unhandled exception c0000005 at 0040167C
advpack:install:07b0 done (-1073741819) in 0s
-amstream:amstream start dlls/amstream/tests/amstream.c -
------ No pid at all
------ Expected assessement: Success
-amstream: 152 tests executed (0 marked as todo, 0 failures), 0 skipped.
-amstream:amstream done (0) in 0s
-
-apphelp:apphelp start dlls/apphelp/tests/apphelp.c -
------ A test unit with no execution time
------ Expected assessement: Success
-0860:apphelp: 15 tests executed (0 marked as todo, 0 failures), 0 skipped.
-apphelp:apphelp:0860 done (0)
-
-atl:atl_ax start dlls/atl/tests/atl_ax.c -
------ Pid on the test summary line, but no pid on the done one
------ Expected assessement: Success
-0938:atl_ax: 12 tests executed (0 marked as todo, 0 failures), 0 skipped.
-atl:atl_ax done (0) in 0s
-
-atl:module start dlls/atl/tests/module.c -
------ Pid on the done line, but no pid on the test summary line
------ Expected assessement: Success
-module: 544 tests executed (0 marked as todo, 0 failures), 0 skipped.
-atl:module:0920 done (0) in 0s
+stub amstream:amstream
+stub apphelp:apphelp
+stub atl:atl_ax
+stub atl:module
atl:registrar start dlls/atl/tests/registrar.c -
----- A test unit with a foreign test summary line
@@ -481,10 +462,7 @@ comctl32:progress start dlls/comctl32/tests/progress.c -
0148:progress: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:progress:0148 done (0) in 9s
-comctl32:propsheet start dlls/comctl32/tests/propsheet.c -
------ A test unit with no main process test summary line and no pid anywhere
------ Expected assessement: "Missing test summary" error
-comctl32:propsheet done (0) in 9s
+stub comctl32:propsheet
comctl32:rebar start dlls/comctl32/tests/rebar.c -
----- A test unit with an unreported crash in a subprocess
@@ -499,22 +477,16 @@ subprocess crashes either. This is why the error is "Zero exit code" and not
07b0:rebar: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:rebar:07b0 done (0) in 0s
-comctl32:status start dlls/comctl32/tests/status.c -
------ A test unit with a subprocess crash but no pid
------ Expected assessement: "Zero exit code" error
-See the comctl32:rebar comment.
-status: unhandled exception c0000005 at 0040167C
-status: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
-comctl32:status:07b0 done (0) in 0s
+stub comctl32:status
comctl32:subclass start dlls/comctl32/tests/subclass.c -
------ A test unit with a subprocess crash, no pid and a non-zero exit code
+----- A test unit with a subprocess crash, and a non-zero exit code
----- Expected assessement: 1 failure
See the comctl32:rebar comment. Then note that here the main process does not
claim success. Also note that the exit code does not correspond to that of a
crashed process which makes sense since it did not crash.
-subclass: unhandled exception c0000005 at 0040167C
-subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+1234:subclass: unhandled exception c0000005 at 0040167C
+07b0:subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:subclass:07b0 done (1) in 0s
comctl32:syslink start dlls/comctl32/tests/syslink.c -
@@ -534,28 +506,16 @@ Do not cut 07b0:syslink: 2251 tests executed (1 marked as todo, 2 failures), 1 s
me off
comctl32:syslink:07b0 done (2) in 0s
-comctl32:tab start dlls/comctl32/tests/tab.c -
------ A test unit crash with no pid
------ Expected assessement: crashed
-tab: unhandled exception c0000005 at 0040167C
-comctl32:tab done (-1073741819) in 0s
+stub comctl32:tab
-comctl32:taskdialog start dlls/comctl32/tests/taskdialog.c -
------ A test unit crash with unsigned exit code (TestLauncher)
------ Expected assessement: crashed
-taskdialog: unhandled exception c0000005 at 0040167C
-comctl32:taskdialog done (3221225477) in 0s
+stub comctl32:taskdialog
comctl32:toolbar start dlls/comctl32/tests/toolbar.c -
----- A test unit crash with no unhandled exception line
----- Expected assessement: crashed
comctl32:toolbar done (-1073741819) in 0s
-comctl32:tooltips start dlls/comctl32/tests/toolbar.c -
------ A test unit crash with no unhandled exception line
------ and an unsigned exit code.
------ Expected assessement: crashed
-comctl32:tooltips done (3221225477) in 0s
+stub comctl32:tooltips
comctl32:trackbar start dlls/comctl32/tests/trackbar.c -
----- A test unit with a subtest
Module: tools
Branch: master
Commit: 7456be821eda0684e170a93aefc9ae04d56a5415
URL: https://source.winehq.org/git/tools.git/?a=commit;h=7456be821eda0684e170a93…
Author: Francois Gouget <fgouget(a)codeweavers.com>
Date: Mon Feb 8 15:21:03 2021 +0100
testbot/TestWTBS: Update the description in the usage message.
Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>
---
testbot/tests/TestWTBS | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/testbot/tests/TestWTBS b/testbot/tests/TestWTBS
index 1cbec2e..4a4aba9 100755
--- a/testbot/tests/TestWTBS
+++ b/testbot/tests/TestWTBS
@@ -165,7 +165,12 @@ if (defined $Usage)
}
print "Usage: $name0 [--jobs RANGES] [--mbox FILE] [--help]\n";
print "\n";
- print "Tests the Patches subject parser.\n";
+ print "Automate checking the Wine TestBot test Suite results.\n";
+ print "\n";
+ print "The expectations are described in the form of directives that specify\n";
+ print "expected job, task, patch or webpatch property values; and errors or strings in\n";
+ print "logs and reports.\n";
+ print "For more details, see the documentation in the $name0 source.\n";
print "\n";
print "Where:\n";
print " --jobs RANGES Only check the jobs in one of the specified comma-separated\n";