Francois Gouget : testbot/reporttest: Some fixes and additions for the report template.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: 1239052d91cdfcb770964b007ea5ae8d89a61dea
URL:    http://source.winehq.org/git/tools.git/?a=commit;h=1239052d91cdfcb770964b007...

Author: Francois Gouget <fgouget(a)codeweavers.com>
Date:   Sat Feb 18 11:03:59 2017 +0100

testbot/reporttest: Some fixes and additions for the report template.

Some tests had copy/paste errors which were caught by an actual parser.
Added a few more tests for misplaced lines, exceptions in subprocesses,
and races in the test output.

Signed-off-by: Francois Gouget <fgouget(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>

---

 testbot/src/reporttest/report.template | 121 ++++++++++++++++++++++++++-------
 1 file changed, 98 insertions(+), 23 deletions(-)

diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index 003d41e..cb83ad7 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -296,6 +296,7 @@ service.c:20: Test failed: A failure in the subprocess
 advpack:advpack start dlls/advpack/tests/advpack.c -
 ----- A standard successful unit test
 ----- Expected assessement: Success, unless the previous test interferes
+0da0:advpack: 99 tests executed (0 marked as todo, 0 failures), 0 skipped.
 advpack:advpack:0da0 done (0) in 0s
 
 advpack:files start dlls/advapi32/tests/crypt.c -
@@ -309,7 +310,7 @@ advpack:files:0880 done (3) in 0s
 advpack:install start dlls/advpack/tests/install.c -
 ----- A unit test with a crash
 ----- Expected assessement: Crash
-install: unhandled exception c0000005 at 0040167C
+07b0:install: unhandled exception c0000005 at 0040167C
 advpack:install:07b0 done (3221225477) in 0s
 
 amstream:amstream start dlls/amstream/tests/amstream.c -
@@ -339,13 +340,14 @@ atl:module:0920 done (0) in 0s
 atl:registrar start dlls/atl/tests/registrar.c -
 ----- A unit test with a foreign test summary line
 ----- Expected assessement: "Misplaced test summary line" error
-9876:service: 10 tests executed (0 marked as todo, 0 failures), 0 skipped.
+9876:service: 10 tests executed (0 marked as todo, 1 failures), 0 skipped.
 0922:registrar: 17 tests executed (0 marked as todo, 0 failures), 0 skipped.
 atl:registrar:0922 done (0) in 0s
 
------ Skipping all tests for a missing dll (see header)
------ Expected assessement: Skipped (FIXME: or Not Run?)
-avifil32:api skipped dlls/avifil32/tests/api.c -
+----- Skipping all tests for a missing dll (see the "Dll info:" header)
+----- Expected assessement: Not available
+avifil32 is missing and thus no test will be run for it and no entry will be
+found in the test report.
 
 browseui:autocomplete start dlls/browseui/tests/autocomplete.c -
 ----- A unit test with a skipped test
@@ -408,17 +410,16 @@ This line says nothing is wrong because it's from the subprocess
 comctl32:comboex:0c9c done (1) in 0s
 
 comctl32:datetime start dlls/comctl32/tests/datetime.c -
------ A unit test with a missing test failure line
+----- A unit test reporting a failure but with no test failed line
 ----- Expected assessement: "Missing test failed line" error
-datetime.c:1: Test failed: Something wrong
-0998:datetime: 651 tests executed (0 marked as todo, 0 failures), 0 skipped.
-comctl32:datetime:0998 done (0) in 0s
+0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped.
+comctl32:datetime:0998 done (1) in 0s
 
 comctl32:dpa start dlls/comctl32/tests/dpa.c -
 ----- A unit test with a non-zero exit code
 ----- Expected assessement: "Non-zero exit code" error
 This would cause things like 'make test' to consider the unit test failed.
-0b14:dpa: 16 tests executed (0 marked as todo, 0 failures), 0 skipped.
+0998:dpa: 16 tests executed (0 marked as todo, 0 failures), 0 skipped.
 comctl32:dpa:0998 done (1) in 0s
 
 comctl32:header start dlls/comctl32/tests/header.c -
@@ -426,7 +427,7 @@ comctl32:header start dlls/comctl32/tests/header.c -
 ----- Expected assessement: "Zero exit code" error
 A non-zero exit code is necessary for 'make test'.
 header.c:1: Test failed: Something wrong
-0b14:header: 16 tests executed (0 marked as todo, 1 failure), 0 skipped.
+0998:header: 16 tests executed (0 marked as todo, 1 failure), 0 skipped.
 comctl32:header:0998 done (0) in 0s
 
 comctl32:imagelist start - -
@@ -449,16 +450,90 @@ stub comctl32:listview
 0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
 comctl32:misc:0998 done (0) in 0s
 
-stub comctl32:monthcal
-stub comctl32:mru
-stub comctl32:pager
-stub comctl32:progress
-stub comctl32:propsheet
-stub comctl32:rebar
-stub comctl32:status
-stub comctl32:subclass
-stub comctl32:syslink
-stub comctl32:tab
+comctl32:monthcal start dlls/comctl32/tests/monthcal.c -
+----- A unit test with a misplaced todo subprocess test summary line
+----- Expected assessement: "Misplaced summary line" error
+See the previous misplaced test summary line scenarios.
+0148:monthcal: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:monthcal:0148 done (0) in 9s
+security.c:41: Test marked todo: Hello Wine!
+0955:security: 18 tests executed (1 marked as todo, 0 failures), 0 skipped.
+
+comctl32:mru start dlls/comctl32/tests/mru.c -
+----- A standard successful unit test
+----- Expected assessement: Success, unless the previous test interferes
+0148:mru: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:mru:0148 done (0) in 9s
+
+comctl32:pager start dlls/comctl32/tests/pager.c -
+----- A unit test with a misplaced subprocess skip test summary line
+----- Expected assessement: Success
+See the previous misplaced test summary line scenarios.
+0148:pager: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:pager:0148 done (0) in 9s
+security.c:42: Tests skipped: Another time
+0956:security: 18 tests executed (0 marked as todo, 0 failures), 1 skipped.
+
+comctl32:progress start dlls/comctl32/tests/progress.c -
+----- A standard successful unit test
+----- Expected assessement: Success
+0148:progress: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:progress:0148 done (0) in 9s
+
+comctl32:propsheet start dlls/comctl32/tests/propsheet.c -
+----- A unit test with no main process test summary line and no pid anywhere
+----- Expected assessement: "Missing test summary" error
+comctl32:propsheet done (0) in 9s
+
+comctl32:rebar start dlls/comctl32/tests/rebar.c -
+----- A unit test with an unreported crash in a subprocess
+----- Expected assessement: "Zero exit code" error
+Even in subprocesses crashes are equivalent to 'Test failed' errors and as
+such the main process cannot report success, i.e. it must have a non-zero exit
+code. Note also that since we don't require the main process to account for the
+subprocess failures in its test summary line, it's ok for it not to account for
+subprocess crashes either. This is why the error is "Zero exit code" and not
+"Unreported failure".
+1234:rebar: unhandled exception c0000005 at 0040167C
+07b0:rebar: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:rebar:07b0 done (0) in 0s
+
+comctl32:status start dlls/comctl32/tests/status.c -
+----- A unit test with a subprocess crash but no pid
+----- Expected assessement: "Zero exit code" error
+See the comctl32:rebar comment.
+status: unhandled exception c0000005 at 0040167C
+status: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:status:07b0 done (0) in 0s
+
+comctl32:subclass start dlls/comctl32/tests/subclass.c -
+----- A unit test with a subprocess crash, no pid and a non-zero exit code
+----- Expected assessement: 1 failure
+See the comctl32:rebar comment. Then note that here the main process does not
+claim success. Also note that the exit code does not correspond to that of a
+crashed process which makes sense since it did not crash.
+subclass: unhandled exception c0000005 at 0040167C
+subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+comctl32:subclass:07b0 done (1) in 0s
+
+comctl32:syslink start dlls/comctl32/tests/syslink.c -
+----- A unit test with a garbled test failed line
+----- Expected assessement: 1 failure
+Multi-threaded and multi-process tests should really try to avoid races in
+the report. But analysis tools should also try to be resilient.
+Do not cut syslink.c:40: Test failed: Something wrong
+me off
+07b0:syslink: 2251 tests executed (0 marked as todo, 1 failures), 0 skipped.
+comctl32:syslink:07b0 done (1) in 0s
+
+comctl32:tab start dlls/comctl32/tests/tab.c -
+----- A unit test with a garbled test summary line
+----- Expected assessement: Success
+See previous comment.
+Do not cut 07b0:tab: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
+me off
+comctl32:tab:07b0 done (0) in 0s
+
 stub comctl32:toolbar
 stub comctl32:tooltips
 stub comctl32:trackbar
@@ -946,8 +1021,8 @@ stub xmllite:reader
 stub xmllite:writer
 
 xmllite:writer start fake/source/writer.c -
------ A unit test with a misplaced and failed subprocess test summary line
------ Expected assessement: "Misplaced test summary line" error
+----- A unit test with a misplaced test failed line
+----- Expected assessement: "Misplaced test failed line" error
 See the similar scenario before but note the lack of the extra test summary line
 here. Also beware of the end-of-file.
 037c:writer: 1930 tests executed (0 marked as todo, 0 failures), 0 skipped.