Francois Gouget : testbot/reporttest: Test flaky failures.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: 596e9e291e5645767819a44921c586714dc4ecd1
URL:    https://gitlab.winehq.org/winehq/tools/-/commit/596e9e291e5645767819a44921c5...

Author: Francois Gouget 
Date:   Tue Sep 20 10:22:47 2022 +0200

testbot/reporttest: Test flaky failures.

---

testbot/src/reporttest/report.template | 10 +++++++++-
 testbot/src/reporttest/report.testwtbs |  2 ++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index 8c726315..51d9869e 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -671,7 +671,15 @@ fontdlg.c:30: unhandled exception c0000005 in child process 3333
 01bd:fontdlg: 10 tests executed (0 marked as todo, 0 as flaky, 0 as flaky, 6 failures), 0 skipped.
 comdlg32:fontdlg:01bd done (6) in 0s

-stub comdlg32:itemdlg
+comdlg32:itemdlg start dlls/comdlg32/tests/itemdlg.c
+----- WTBS Flaky failures sampler
+----- A test unit with flaky failures
+----- Expected assessement: 2 flaky tests, 0 failure
+itemdlg.c:1: Test marked flaky: A regular test which fails sometimes
+itemdlg.c:2: Test succeeded inside flaky todo block: A todo which succeeds sometimes
+031c:itemdlg: 108 tests executed (0 marked as todo, 2 as flaky, 0 failure), 0 skipped.
+comdlg32:itemdlg:031c done (0) in 0s
+
 stub comdlg32:printdlg
 stub credui:credui
 stub crypt32:base64
diff --git a/testbot/src/reporttest/report.testwtbs b/testbot/src/reporttest/report.testwtbs
index 64a5ca13..d7ed0ecd 100644
--- a/testbot/src/reporttest/report.testwtbs
+++ b/testbot/src/reporttest/report.testwtbs
@@ -77,6 +77,8 @@ n 0 fontdlg.c:10: unhandled exception c0000005 in child process 1111
 n 0 01bd:fontdlg: unhandled exception c0000005 at 00003333
 n 0 Unhandled exception: page fault on read access to 0x00000000 in 32-bit code (0x00003333).
 n 0 fontdlg.c:30: unhandled exception c0000005 in child process 3333
+n 0 itemdlg.c:1: Test marked flaky: A regular test which fails sometimes
+n 0 itemdlg.c:2: Test succeeded inside flaky todo block: A todo which succeeds sometimes

g 0 tests.report.xmllite
 n 0 writer.c:20: Test failed: A delayed failure, typically in a subprocess