[tools 1/2] testbot/reporttest: Remove tests for legacy TestLauncher and Wine versions.
Remove tests for line formats that neither the TestBot nor the WineTest website support anymore. This is the case of lines missing pids, run times or using signed exit codes. Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> --- testbot/src/reporttest/report.template | 64 +++++--------------------- 1 file changed, 12 insertions(+), 52 deletions(-) diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template index b541e41d8..c08b4d47c 100644 --- a/testbot/src/reporttest/report.template +++ b/testbot/src/reporttest/report.template @@ -313,29 +313,10 @@ advpack:install start dlls/advpack/tests/install.c - 07b0:install: unhandled exception c0000005 at 0040167C advpack:install:07b0 done (-1073741819) in 0s -amstream:amstream start dlls/amstream/tests/amstream.c - ------ No pid at all ------ Expected assessement: Success -amstream: 152 tests executed (0 marked as todo, 0 failures), 0 skipped. -amstream:amstream done (0) in 0s - -apphelp:apphelp start dlls/apphelp/tests/apphelp.c - ------ A test unit with no execution time ------ Expected assessement: Success -0860:apphelp: 15 tests executed (0 marked as todo, 0 failures), 0 skipped. -apphelp:apphelp:0860 done (0) - -atl:atl_ax start dlls/atl/tests/atl_ax.c - ------ Pid on the test summary line, but no pid on the done one ------ Expected assessement: Success -0938:atl_ax: 12 tests executed (0 marked as todo, 0 failures), 0 skipped. -atl:atl_ax done (0) in 0s - -atl:module start dlls/atl/tests/module.c - ------ Pid on the done line, but no pid on the test summary line ------ Expected assessement: Success -module: 544 tests executed (0 marked as todo, 0 failures), 0 skipped. -atl:module:0920 done (0) in 0s +stub amstream:amstream +stub apphelp:apphelp +stub atl:atl_ax +stub atl:module atl:registrar start dlls/atl/tests/registrar.c - ----- A test unit with a foreign test summary line @@ -481,10 +462,7 @@ comctl32:progress start dlls/comctl32/tests/progress.c - 0148:progress: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:progress:0148 done (0) in 9s -comctl32:propsheet start dlls/comctl32/tests/propsheet.c - ------ A test unit with no main process test summary line and no pid anywhere ------ Expected assessement: "Missing test summary" error -comctl32:propsheet done (0) in 9s +stub comctl32:propsheet comctl32:rebar start dlls/comctl32/tests/rebar.c - ----- A test unit with an unreported crash in a subprocess @@ -499,22 +477,16 @@ subprocess crashes either. This is why the error is "Zero exit code" and not 07b0:rebar: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:rebar:07b0 done (0) in 0s -comctl32:status start dlls/comctl32/tests/status.c - ------ A test unit with a subprocess crash but no pid ------ Expected assessement: "Zero exit code" error -See the comctl32:rebar comment. -status: unhandled exception c0000005 at 0040167C -status: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. -comctl32:status:07b0 done (0) in 0s +stub comctl32:status comctl32:subclass start dlls/comctl32/tests/subclass.c - ------ A test unit with a subprocess crash, no pid and a non-zero exit code +----- A test unit with a subprocess crash, and a non-zero exit code ----- Expected assessement: 1 failure See the comctl32:rebar comment. Then note that here the main process does not claim success. Also note that the exit code does not correspond to that of a crashed process which makes sense since it did not crash. -subclass: unhandled exception c0000005 at 0040167C -subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. +1234:subclass: unhandled exception c0000005 at 0040167C +07b0:subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:subclass:07b0 done (1) in 0s comctl32:syslink start dlls/comctl32/tests/syslink.c - @@ -534,28 +506,16 @@ Do not cut 07b0:syslink: 2251 tests executed (1 marked as todo, 2 failures), 1 s me off comctl32:syslink:07b0 done (2) in 0s -comctl32:tab start dlls/comctl32/tests/tab.c - ------ A test unit crash with no pid ------ Expected assessement: crashed -tab: unhandled exception c0000005 at 0040167C -comctl32:tab done (-1073741819) in 0s +stub comctl32:tab -comctl32:taskdialog start dlls/comctl32/tests/taskdialog.c - ------ A test unit crash with unsigned exit code (TestLauncher) ------ Expected assessement: crashed -taskdialog: unhandled exception c0000005 at 0040167C -comctl32:taskdialog done (3221225477) in 0s +stub comctl32:taskdialog comctl32:toolbar start dlls/comctl32/tests/toolbar.c - ----- A test unit crash with no unhandled exception line ----- Expected assessement: crashed comctl32:toolbar done (-1073741819) in 0s -comctl32:tooltips start dlls/comctl32/tests/toolbar.c - ------ A test unit crash with no unhandled exception line ------ and an unsigned exit code. ------ Expected assessement: crashed -comctl32:tooltips done (3221225477) in 0s +stub comctl32:tooltips comctl32:trackbar start dlls/comctl32/tests/trackbar.c - ----- A test unit with a subtest -- 2.20.1
Where a test mirrors a wtbsuite test, add the latter's commit message as a comment to make it simpler to compare the two. Also tweak some comments to better describe the tests. Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com> --- testbot/src/reporttest/report.template | 30 ++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template index c08b4d47c..560c8b224 100644 --- a/testbot/src/reporttest/report.template +++ b/testbot/src/reporttest/report.template @@ -209,8 +209,11 @@ Dll info: xmllite=1.0.1018.0 Test output: stub advapi32:cred +----- WTBS Simple test patch +----- Expected assessement: Success advapi32:crypt start dlls/advapi32/tests/crypt.c - +----- WTBS Simple test failure ----- A test unit with one failed test ----- Expected assessement: 1 failure This also has a valid source file name which one can link to. @@ -219,10 +222,11 @@ crypt.c:1: Test failed: Something wrong advapi32:crypt:031c done (1) in 0s advapi32:crypt_lmhash start dlls/advapi32/tests/crypt_lmhash.c - +----- WTBS Simple diacritic failure ----- A test unit with two failed tests ------ Expected assessement: 2 failures +----- Expected assessement: 2 failures despite exit code crypt_lmhash.c:10: Test failed: Something wrong -crypt_lmhash.c:20: Test failed: Second failure here +crypt_lmhash.c:20: Test failed: Second failure with added diacritics: ?????????? 07c0:crypt_lmhash: 95 tests executed (0 marked as todo, 2 failures), 0 skipped. advapi32:crypt_lmhash:07c0 done (1) in 0s @@ -246,7 +250,8 @@ Note that this also applies to the todo and skip counts. advapi32:crypt_md5:087c done (3) in 0s advapi32:crypt_sha start dlls/advapi32/tests/crypt_sha.c - ------ A test unit with a failed subprocess test ignored by the parent +----- A test unit with a failed subprocess test ignored by the parent test +----- summary but accounted for in the exit code. ----- Expected assessement: 1 failure crypt_sha.c:20: Test failed: A failure here, presumably in the subprocess 0456:crypt_sha: 1 tests executed (0 marked as todo, 1 failure), 0 skipped. @@ -256,6 +261,7 @@ subprocess. advapi32:crypt_sha:08f8 done (3) in 0s advapi32:eventlog start dlls/advapi32/tests/eventlog.c - +----- WTBS Unreported test failures ----- A test unit with an unreported failed test ----- Expected assessement: 1 failure, "Unreported failure" error eventlog.c:20: Test failed: Something wrong @@ -263,6 +269,7 @@ eventlog.c:20: Test failed: Something wrong advapi32:eventlog:088c done (3) in 4s advapi32:lsa start dlls/advapi32/tests/lsa.c - +----- WTBS Missing test summary line ----- A test unit with no test summary line ----- Expected assessement: "Missing test summary" error advapi32:lsa:083c done (0) in 0s @@ -308,6 +315,7 @@ files.c:20: Test failed: A failure in the main process advpack:files:0880 done (3) in 0s advpack:install start dlls/advpack/tests/install.c - +----- WTBS Simple test crash ----- A test unit with a crash ----- Expected assessement: Crash 07b0:install: unhandled exception c0000005 at 0040167C @@ -371,6 +379,8 @@ comcat:comcat start dlls/comcat/tests/comcat.c - comcat:comcat:0948 done (0) in 0s comctl32:animate start dlls/comctl32/tests/animate.c - +----- WTBS Timeout in a test +----- WTBS Timeout in a test - barely ----- A test unit with a timeout ----- Expected assessement: Timeout comctl32:animate:0b14 done (258) in 42s @@ -380,7 +390,10 @@ comctl32:button skipped dlls/comctl32/tests/button.c - ----- Expected assessement: Skipped by user comctl32:comboex start dlls/comctl32/tests/comboex.c - ------ A test unit with failed, todo and skipped tests in the main process +----- WTBS Simple success in a todo test +----- WTBS Simple trace, ok, todo, skip sampler +----- A test unit with a subprocess and failed, todo and skipped tests in the +----- main process. ----- Expected assessement: 2 failures, 1 todo, 1 skip comboex.c:40: Test failed: A failure in the main process comboex.c:41: Test succeeded inside todo block: Success in the main process @@ -394,6 +407,7 @@ comctl32:comboex:0c9c done (2) in 0s comctl32:datetime start dlls/comctl32/tests/datetime.c - ----- A test unit reporting a failure but with no test failed line ----- Expected assessement: "Missing test failed line" error +----- + TestFailures=2 (1 for the exit code + 1 for the missing failure line) 0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped. comctl32:datetime:0998 done (1) in 0s @@ -425,8 +439,10 @@ comctl32:ipaddress start dlls/comctl32/tests/ipaddress.c - Issued upon seeing the next start line. 0998:ipaddress: 16 tests executed (0 marked as todo, 0 failure), 0 skipped. +----- Keep this stub in case the previous test is mishandled stub comctl32:listview +----- WTBS A test unit with no start line ----- A test unit with a missing start line ----- Expected assessement: "Missing start line" error 0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped. @@ -490,6 +506,7 @@ crashed process which makes sense since it did not crash. comctl32:subclass:07b0 done (1) in 0s comctl32:syslink start dlls/comctl32/tests/syslink.c - +----- WTBS Garbled trace, ok, todo, skip sampler ----- A test unit with garbled test lines ----- Expected assessement: 2 failures, 1 todo, 1 skip Multi-threaded and multi-process tests should really try to avoid races in @@ -518,13 +535,14 @@ comctl32:toolbar done (-1073741819) in 0s stub comctl32:tooltips comctl32:trackbar start dlls/comctl32/tests/trackbar.c - +----- WTBS Test failures, todos and skips in a subtest ----- A test unit with a subtest +----- Expected assessement: 2 failures, 1 todo, 1 skip trackbar.c:496: Subtest driver driver.c:40: Test failed: Something wrong driver.c:41: Test succeeded inside todo block: Something right! driver.c:42: Test marked todo: Hello Wine! driver.c:43: Tests skipped: Another time ------ Expected assessement: 2 failures, 1 todo, 1 skip 07b1:trackbar: 3 tests executed (1 marked as todo, 2 failures), 1 skipped. comctl32:trackbar:07b1 done (2) in 0s @@ -1011,7 +1029,7 @@ stub xcopy.exe:xcopy stub xmllite:reader xmllite:writer start dlls/xmllite/tests/writer.c - ------ A test unit with a misplaced test failed line +----- A test unit with a misplaced (late) test failed line ----- Expected assessement: "Misplaced test failed line" error See the similar scenario before but note the lack of the extra test summary line here. Also beware of the end-of-file. -- 2.20.1
participants (1)
-
Francois Gouget