Signed-off-by: Francois Gouget fgouget@codeweavers.com --- testbot/src/reporttest/report.template | 94 +++++++++++++------------- testbot/src/reporttest/reporttest.c | 8 +-- 2 files changed, 51 insertions(+), 51 deletions(-)
diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template index eb9999e3e..b541e41d8 100644 --- a/testbot/src/reporttest/report.template +++ b/testbot/src/reporttest/report.template @@ -211,7 +211,7 @@ Test output: stub advapi32:cred
advapi32:crypt start dlls/advapi32/tests/crypt.c - ------ A unit test with one failed test +----- A test unit with one failed test ----- Expected assessement: 1 failure This also has a valid source file name which one can link to. crypt.c:1: Test failed: Something wrong @@ -219,7 +219,7 @@ crypt.c:1: Test failed: Something wrong advapi32:crypt:031c done (1) in 0s
advapi32:crypt_lmhash start dlls/advapi32/tests/crypt_lmhash.c - ------ A unit test with two failed tests +----- A test unit with two failed tests ----- Expected assessement: 2 failures crypt_lmhash.c:10: Test failed: Something wrong crypt_lmhash.c:20: Test failed: Second failure here @@ -227,14 +227,14 @@ crypt_lmhash.c:20: Test failed: Second failure here advapi32:crypt_lmhash:07c0 done (1) in 0s
advapi32:crypt_md4 start dlls/advapi32/tests/crypt_md4.c - ------ A unit test with two test summary lines +----- A test unit with two test summary lines ----- Expected assessement: Success 0123:crypt_md4: 11 tests executed (0 marked as todo, 0 failures), 0 skipped. 0ca0:crypt_md4: 11 tests executed (0 marked as todo, 0 failures), 0 skipped. advapi32:crypt_md4:0ca0 done (0) in 0s
advapi32:crypt_md5 start dlls/advapi32/tests/crypt_md5.c - ------ A unit test with a failed subprocess test repeated in the parent +----- A test unit with a failed subprocess test repeated in the parent ----- Expected assessement: 1 failure (also acceptable: 2 failures) crypt_md5.c:20: Test failed: A failure here, presumably in the subprocess 0123:crypt_md5: 11 tests executed (0 marked as todo, 1 failure), 0 skipped. @@ -246,35 +246,35 @@ Note that this also applies to the todo and skip counts. advapi32:crypt_md5:087c done (3) in 0s
advapi32:crypt_sha start dlls/advapi32/tests/crypt_sha.c - ------ A unit test with a failed subprocess test ignored by the parent +----- A test unit with a failed subprocess test ignored by the parent ----- Expected assessement: 1 failure crypt_sha.c:20: Test failed: A failure here, presumably in the subprocess 0456:crypt_sha: 1 tests executed (0 marked as todo, 1 failure), 0 skipped. -The unit test counts on the framework to pick up the failure reported by the +The test unit counts on the framework to pick up the failure reported by the subprocess. 08f8:crypt_sha: 1 tests executed (0 marked as todo, 0 failures), 0 skipped. advapi32:crypt_sha:08f8 done (3) in 0s
advapi32:eventlog start dlls/advapi32/tests/eventlog.c - ------ A unit test with an unreported failed test +----- A test unit with an unreported failed test ----- Expected assessement: 1 failure, "Unreported failure" error eventlog.c:20: Test failed: Something wrong 088c:eventlog: 408 tests executed (0 marked as todo, 0 failures), 0 skipped. advapi32:eventlog:088c done (3) in 4s
advapi32:lsa start dlls/advapi32/tests/lsa.c - ------ A unit test with no test summary line +----- A test unit with no test summary line ----- Expected assessement: "Missing test summary" error advapi32:lsa:083c done (0) in 0s
advapi32:registry start dlls/advapi32/tests/registry.c - ------ A unit test with no main process test summary line +----- A test unit with no main process test summary line ----- Expected assessement: "Missing test summary" error 1234:registry: 765 tests executed (0 marked as todo, 0 failures), 0 skipped. advapi32:registry:0804 done (0) in 0s
advapi32:security start dlls/advapi32/tests/security.c - ------ A unit test with a misplaced but successful subprocess test summary line +----- A test unit with a misplaced but successful subprocess test summary line ----- Expected assessement: Success The subprocess may be a server process that exits automatically once the test is complete. That's ok as long as it's not meant to report failures. @@ -283,24 +283,24 @@ advapi32:security:0148 done (0) in 9s 0954:security: 18 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:service start dlls/advapi32/tests/service.c - ------ A unit test with a misplaced and failed subprocess test summary line +----- A test unit with a misplaced and failed subprocess test summary line ----- Expected assessement: "Misplaced test summary line" error Same scenario as above but the subprocess can actually report failures so the main process must wait for it. Ideally the error should be reported against -this unit test but reporting against the next one is also acceptable. +this test unit but reporting against the next one is also acceptable. 037c:service: 1930 tests executed (0 marked as todo, 0 failures), 0 skipped. advapi32:service:037c done (0) in 11s service.c:20: Test failed: A failure in the subprocess 4567:service: 10 tests executed (0 marked as todo, 1 failure), 0 skipped.
advpack:advpack start dlls/advpack/tests/advpack.c - ------ A standard successful unit test +----- A standard successful test unit ----- Expected assessement: Success, unless the previous test interferes 0da0:advpack: 99 tests executed (0 marked as todo, 0 failures), 0 skipped. advpack:advpack:0da0 done (0) in 0s
advpack:files start dlls/advpack/tests/files.c - ------ A unit test with a failed test in the main process before a subprocess +----- A test unit with a failed test in the main process before a subprocess ----- Expected assessement: 1 failure files.c:20: Test failed: A failure in the main process 4321:files: 99 tests executed (0 marked as todo, 0 failures), 0 skipped. @@ -308,7 +308,7 @@ files.c:20: Test failed: A failure in the main process advpack:files:0880 done (3) in 0s
advpack:install start dlls/advpack/tests/install.c - ------ A unit test with a crash +----- A test unit with a crash ----- Expected assessement: Crash 07b0:install: unhandled exception c0000005 at 0040167C advpack:install:07b0 done (-1073741819) in 0s @@ -320,7 +320,7 @@ amstream: 152 tests executed (0 marked as todo, 0 failures), 0 skipped. amstream:amstream done (0) in 0s
apphelp:apphelp start dlls/apphelp/tests/apphelp.c - ------ A unit test with no execution time +----- A test unit with no execution time ----- Expected assessement: Success 0860:apphelp: 15 tests executed (0 marked as todo, 0 failures), 0 skipped. apphelp:apphelp:0860 done (0) @@ -338,7 +338,7 @@ module: 544 tests executed (0 marked as todo, 0 failures), 0 skipped. atl:module:0920 done (0) in 0s
atl:registrar start dlls/atl/tests/registrar.c - ------ A unit test with a foreign test summary line +----- A test unit with a foreign test summary line ----- Expected assessement: "Misplaced test summary line" error 9876:alien: 10 tests executed (0 marked as todo, 1 failures), 0 skipped. 0922:registrar: 17 tests executed (0 marked as todo, 0 failures), 0 skipped. @@ -350,47 +350,47 @@ avifil32 is missing and thus no test will be run for it and no entry will be found in the test report.
browseui:autocomplete start dlls/browseui/tests/autocomplete.c - ------ A unit test with a skipped test +----- A test unit with a skipped test ----- Expected assessement: Success, 1 skipped autocomplete.c:42: Tests skipped: Another time 052c:autocomplete: 66 tests executed (0 marked as todo, 0 failures), 1 skipped. browseui:autocomplete:052c done (0) in 0s
browseui:progressdlg start dlls/browseui/tests/progressdlg.c - ------ A unit test with an unreported skipped test +----- A test unit with an unreported skipped test ----- Expected assessement: 1 skipped, "Unreported skip" error progressdlg.c:42: Tests skipped: Another time 0680:progressdlg: 4 tests executed (0 marked as todo, 0 failures), 0 skipped. browseui:progressdlg:0680 done (0) in 0s
cabinet:extract start dlls/cabinet/tests/extract.c - ------ A unit test with a missing skipped test line +----- A test unit with a missing skipped test line ----- Expected assessement: "Missing tests skipped line" error 0fd4:extract: 347 tests executed (0 marked as todo, 0 failures), 1 skipped. cabinet:extract:0fd4 done (0) in 0s
cabinet:fdi start dlls/cabinet/tests/fdi.c - ------ A unit test with a todo test +----- A test unit with a todo test ----- Expected assessement: Success, 1 todo fdi.c:42: Test marked todo: Hello Wine! 0690:fdi: 265 tests executed (1 marked as todo, 0 failures), 0 skipped. cabinet:fdi:0690 done (0) in 0s
cmd.exe:batch start programs/cmd/tests/batch.c - ------ A unit test with an unreported todo test +----- A test unit with an unreported todo test ----- Expected assessement: "Unreported todo line" error batch.c:42: Test marked todo: Hello Wine! 0ddc:batch: 1701 tests executed (0 marked as todo, 0 failures), 0 skipped. cmd.exe:batch:0ddc done (0) in 3s
comcat:comcat start dlls/comcat/tests/comcat.c - ------ A unit test with a missing todo test line +----- A test unit with a missing todo test line ----- Expected assessement: "Missing todo line" error 0948:comcat: 9 tests executed (1 marked as todo, 0 failures), 0 skipped. comcat:comcat:0948 done (0) in 0s
comctl32:animate start dlls/comctl32/tests/animate.c - ------ A unit test with a timeout +----- A test unit with a timeout ----- Expected assessement: Timeout comctl32:animate:0b14 done (258) in 42s
@@ -399,7 +399,7 @@ comctl32:button skipped dlls/comctl32/tests/button.c - ----- Expected assessement: Skipped by user
comctl32:comboex start dlls/comctl32/tests/comboex.c - ------ A unit test with failed, todo and skipped tests in the main process +----- A test unit with failed, todo and skipped tests in the main process ----- Expected assessement: 2 failures, 1 todo, 1 skip comboex.c:40: Test failed: A failure in the main process comboex.c:41: Test succeeded inside todo block: Success in the main process @@ -411,20 +411,20 @@ This line says nothing is wrong because it's from the subprocess comctl32:comboex:0c9c done (2) in 0s
comctl32:datetime start dlls/comctl32/tests/datetime.c - ------ A unit test reporting a failure but with no test failed line +----- A test unit reporting a failure but with no test failed line ----- Expected assessement: "Missing test failed line" error 0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped. comctl32:datetime:0998 done (1) in 0s
comctl32:dpa start dlls/comctl32/tests/dpa.c - ------ A unit test with a non-zero exit code +----- A test unit with a non-zero exit code ----- Expected assessement: "Non-zero exit code" error -This would cause things like 'make test' to consider the unit test failed. +This would cause things like 'make test' to consider the test unit failed. 0998:dpa: 16 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:dpa:0998 done (1) in 0s
comctl32:header start dlls/comctl32/tests/header.c - ------ A unit test with a zero exit code despite failures +----- A test unit with a zero exit code despite failures ----- Expected assessement: 1 failure, "Zero exit code" error A non-zero exit code is necessary for 'make test'. header.c:1: Test failed: Something wrong @@ -432,27 +432,27 @@ header.c:1: Test failed: Something wrong comctl32:header:0998 done (0) in 0s
comctl32:imagelist start - - ------ A unit test with no source filename +----- A test unit with no source filename ----- Expected assessement: Success TestLauncher does not set the source filename field. 0998:imagelist: 16 tests executed (0 marked as todo, 0 failure), 0 skipped. comctl32:imagelist:0998 done (0) in 0s
comctl32:ipaddress start dlls/comctl32/tests/ipaddress.c - ------ A unit test with a missing done line +----- A test unit with a missing done line ----- Expected assessement: "Missing done line" error Issued upon seeing the next start line. 0998:ipaddress: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
stub comctl32:listview
------ A unit test with a missing start line +----- A test unit with a missing start line ----- Expected assessement: "Missing start line" error 0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped. comctl32:misc:0998 done (0) in 0s
comctl32:monthcal start dlls/comctl32/tests/monthcal.c - ------ A unit test with a misplaced todo subprocess test summary line +----- A test unit with a misplaced todo subprocess test summary line ----- Expected assessement: 1 todo, "Misplaced summary line" error See the previous misplaced test summary line scenarios. 0148:monthcal: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. @@ -461,13 +461,13 @@ security.c:41: Test marked todo: Hello Wine! 0955:security: 18 tests executed (1 marked as todo, 0 failures), 0 skipped.
comctl32:mru start dlls/comctl32/tests/mru.c - ------ A standard successful unit test +----- A standard successful test unit ----- Expected assessement: Success, unless the previous test interferes 0148:mru: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:mru:0148 done (0) in 9s
comctl32:pager start dlls/comctl32/tests/pager.c - ------ A unit test with a misplaced subprocess skip test summary line +----- A test unit with a misplaced subprocess skip test summary line ----- Expected assessement: Success See the previous misplaced test summary line scenarios. 0148:pager: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. @@ -476,18 +476,18 @@ security.c:42: Tests skipped: Another time 0956:security: 18 tests executed (0 marked as todo, 0 failures), 1 skipped.
comctl32:progress start dlls/comctl32/tests/progress.c - ------ A standard successful unit test +----- A standard successful test unit ----- Expected assessement: Success 0148:progress: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:progress:0148 done (0) in 9s
comctl32:propsheet start dlls/comctl32/tests/propsheet.c - ------ A unit test with no main process test summary line and no pid anywhere +----- A test unit with no main process test summary line and no pid anywhere ----- Expected assessement: "Missing test summary" error comctl32:propsheet done (0) in 9s
comctl32:rebar start dlls/comctl32/tests/rebar.c - ------ A unit test with an unreported crash in a subprocess +----- A test unit with an unreported crash in a subprocess ----- Expected assessement: "Zero exit code" error Even in subprocesses crashes are equivalent to 'Test failed' errors and as such the main process cannot report success, i.e. it must have a non-zero exit @@ -500,7 +500,7 @@ subprocess crashes either. This is why the error is "Zero exit code" and not comctl32:rebar:07b0 done (0) in 0s
comctl32:status start dlls/comctl32/tests/status.c - ------ A unit test with a subprocess crash but no pid +----- A test unit with a subprocess crash but no pid ----- Expected assessement: "Zero exit code" error See the comctl32:rebar comment. status: unhandled exception c0000005 at 0040167C @@ -508,7 +508,7 @@ status: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:status:07b0 done (0) in 0s
comctl32:subclass start dlls/comctl32/tests/subclass.c - ------ A unit test with a subprocess crash, no pid and a non-zero exit code +----- A test unit with a subprocess crash, no pid and a non-zero exit code ----- Expected assessement: 1 failure See the comctl32:rebar comment. Then note that here the main process does not claim success. Also note that the exit code does not correspond to that of a @@ -518,7 +518,7 @@ subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped. comctl32:subclass:07b0 done (1) in 0s
comctl32:syslink start dlls/comctl32/tests/syslink.c - ------ A unit test with garbled test lines +----- A test unit with garbled test lines ----- Expected assessement: 2 failures, 1 todo, 1 skip Multi-threaded and multi-process tests should really try to avoid races in the report. But analysis tools should also try to be resilient. @@ -535,30 +535,30 @@ me off comctl32:syslink:07b0 done (2) in 0s
comctl32:tab start dlls/comctl32/tests/tab.c - ------ A unit test crash with no pid +----- A test unit crash with no pid ----- Expected assessement: crashed tab: unhandled exception c0000005 at 0040167C comctl32:tab done (-1073741819) in 0s
comctl32:taskdialog start dlls/comctl32/tests/taskdialog.c - ------ A unit test crash with unsigned exit code (TestLauncher) +----- A test unit crash with unsigned exit code (TestLauncher) ----- Expected assessement: crashed taskdialog: unhandled exception c0000005 at 0040167C comctl32:taskdialog done (3221225477) in 0s
comctl32:toolbar start dlls/comctl32/tests/toolbar.c - ------ A unit test crash with no unhandled exception line +----- A test unit crash with no unhandled exception line ----- Expected assessement: crashed comctl32:toolbar done (-1073741819) in 0s
comctl32:tooltips start dlls/comctl32/tests/toolbar.c - ------ A unit test crash with no unhandled exception line +----- A test unit crash with no unhandled exception line ----- and an unsigned exit code. ----- Expected assessement: crashed comctl32:tooltips done (3221225477) in 0s
comctl32:trackbar start dlls/comctl32/tests/trackbar.c - ------ A unit test with a subtest +----- A test unit with a subtest trackbar.c:496: Subtest driver driver.c:40: Test failed: Something wrong driver.c:41: Test succeeded inside todo block: Something right! @@ -1051,7 +1051,7 @@ stub xcopy.exe:xcopy stub xmllite:reader
xmllite:writer start dlls/xmllite/tests/writer.c - ------ A unit test with a misplaced test failed line +----- A test unit with a misplaced test failed line ----- Expected assessement: "Misplaced test failed line" error See the similar scenario before but note the lack of the extra test summary line here. Also beware of the end-of-file. diff --git a/testbot/src/reporttest/reporttest.c b/testbot/src/reporttest/reporttest.c index b2e7a78bf..f6ad9b315 100644 --- a/testbot/src/reporttest/reporttest.c +++ b/testbot/src/reporttest/reporttest.c @@ -75,8 +75,8 @@ int main(int argc, char** argv) { fprintf(stderr, "error: Outputting the lines of a specific test (%s) is not supported.\n", argv[i]); /* This would require adjusting the report template to use the real - * process pid. Plus some tests imply mixing the output of unit - * tests which would not make sense. + * process pid. Plus some tests imply mixing the output of test + * units which would not make sense. */ return 2; } @@ -232,7 +232,7 @@ int main(int argc, char** argv) char* unit = strchr(line, ':'); if (!unit) { - fprintf(stderr, "error: The line below does not have a unit!\n"); + fprintf(stderr, "error: The line below does not have a test unit!\n"); fputs(line, stdout); return 1; } @@ -240,7 +240,7 @@ int main(int argc, char** argv) unit++; unit[strlen(unit)-1] = '\0'; fprintf(logfile, "%s:%s start fake/%s/%s.c -\n", line+5, unit, line+5, unit); - fprintf(logfile, "----- A standard successful unit test\n"); + fprintf(logfile, "----- A standard successful test unit\n"); fprintf(logfile, "----- Expected assessment: Success\n"); fprintf(logfile, "1234:%s: 2 tests executed (0 marked as todo, 0 failures), 0 skipped.\n", unit); fprintf(logfile, "%s:%s:1234 done (0) in 0s\n", line+5, unit);