[PATCH v2 0/9] MR3418: Draft: Add tryok() for tests that may need multiple attempts to succeed

[Francois Gouget (＠fgouget)]

This is meant to simplify testing conditions that generally hold true
but may occasionally fail due to interference from external factors
(such as processes that start / stop, network connections being
opened / closed, etc).
The trick is to loop a few times on the set of flaky conditions until
they succeed. During the last attempt all failures are recorded as
usual, while in the previous runs, the tryok() failures area ignored
but cause one more attempt to be made.

The simplest case looks like this:
  LOOP_ON_FLAKY_TESTS(3)
  {
    // ok() failures are never ignored and not retried
    ok(..., "check 1", ...);
    // tryok() failures are ignored except on the last attempt
    tryok(..., "check 2", ...);
  }

There is also:
* attempt_retry() which marks the current attempt as failed as if
  calling tryok(0), and returns true if another attempt can be made.
* attempt_failed() which returns true if an ok() call failed.

---
This is independent from the 'flaky' mechanism which adds some naming
constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS()
despite being unrelated to the flaky mechanism. The attempt_retry()
and attempt_failed() macro names also don't make it obvious that they
are related to tryok().

I think this mechanism is better than the flaky one because a flaky test
can go bad without anyone noticing, whereas if a tryok() starts failing
systematically it will cause a real failure.

The other side of that coin is that, unlike flaky, the tryok()
mechanism does not entirely eliminate the possibility of getting a
failure, it just reduces it; though by adjusting the maximum number of
attempts one can achieve an arbitrarily low failure rate. For instance
if an ok() call fails 10% of the time and one wants a maximum of 1 in
a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an
increased run time in the worst case.

This also limits the use of this mechanism to tests that have a
reasonably low failure rate to start with (otherwise one has to loop
too many times). Also note that there are cases where looping
essentially reduce the failure rate to zero. For instance
ieframe:webbrowser fails if IE creates a net session while the test is
counting them. But IE only creates the one net session on start up so
trying even one more time should guarantee that the test will succeed.
Other cases like scheduling delays and the creation of network
connections are more probabilistic in nature. Maybe a comment in test.h
should offer some guideline as to the target failure rate.

Eventually this may replace the flaky mechanism but that depends on
how well it works in practice and how practical it is to loop on flaky
tests. It seems to be going well in the few cases I looked at. But I
think this mechanism has value even if the two end up coexisting
indefinitely.

This MR uses the tryok() in some actual tests for illustration and testing purposes. The final MR will probably split most of those off to separate MRs.

--
  v2: mmdevapi/tests: Replace flaky with tryok() in the capture tests.
      mmdevapi/tests: Replace flaky with tryok() in the render tests.
      quartz/tests: Replace flaky() with tryok() to work around scheduling delays.
      DEBUG ieframe/tests: tryok() framework testing ground.
      ieframe/tests: Work around a network session race condition.
      advapi32/tests: Replace the custom loop with tryok() mechanism.

https://gitlab.winehq.org/wine/wine/-/merge_requests/3418

[PATCH v2 1/9] tests: Update the documentation.

[Francois Gouget]

From: Francois Gouget

---
 include/wine/test.h | 37 ++++++++++++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 5 deletions(-)

diff --git a/include/wine/test.h b/include/wine/test.h
index c28128d9f0c..c20c6736e2b 100644
--- a/include/wine/test.h
+++ b/include/wine/test.h
@@ -280,6 +280,13 @@ static void exit_process( int code )
 }

+/* Records the location of the next check.
+ * See the xxx_(file, line) macros.
+ *
+ * Parameters:
+ *   - file - source file name of the check
+ *   - line - source line number of the check
+ */
 void winetest_set_location( const char* file, int line )
 {
     struct tls_data *data = get_tls_data();
@@ -391,12 +398,13 @@ static LONG winetest_add_line( void )
 }

/*
- * Checks condition.
+ * Checks that a condition has the expected value, that is true except if
+ * preceded by a todo indicating the check is expected to fail.
+ *
  * Parameters:
- *   - condition - condition to check;
- *   - msg test description;
- *   - file - test application source code file name of the check
- *   - line - test application source code file line number of the check
+ *   - condition - true if the check succeeded, false otherwise;
+ *   - msg - failure message format;
+ *   - args - arguments for the failure message
  * Return:
  *   0 if condition does not have the expected value, 1 otherwise
  */
@@ -551,6 +559,18 @@ void winetest_win_skip( const char *msg, ... )
     va_end(valist);
 }

+/* If is_flaky is true, indicates that the next test may occasionally fail due
+ * to unavoidable outside race conditions. Such failures will be flagged as
+ * flaky so they can be ignored by automated testing tools.
+ *
+ * Notes:
+ * - This is not meant to paper over race conditions within the test itself.
+ *   Those are bugs and should be fixed.
+ * - This is not meant to be used for tests that normally succeed but
+ *   systematically fail on a specific platform or locale.
+ * - The failures should be rare to start with. If a test fails 25% of the time
+ *   it is probably wrong to start with.
+ */
 void winetest_start_flaky( int is_flaky )
 {
     struct tls_data *data = get_tls_data();
@@ -572,6 +592,13 @@ void winetest_end_flaky(void)
     data->flaky_level >>= 1;
 }

+/* If is_todo is true, indicates that the next test is expected to fail on this
+ * platform. This used to identify tests that are known to fail in Wine.
+ *
+ * Notes:
+ * - is_todo should never be true on Windows. To ensure this, always use
+ *   todo_wine_if().
+ */
 void winetest_start_todo( int is_todo )
 {
     struct tls_data *data = get_tls_data();

[PATCH v2 2/9] FIXME(traces) tests: Add tryok() for tests that may need multiple tries to succeed.

[Francois Gouget]

From: Francois Gouget

This is meant to simplify testing conditions that generally hold true
but may occasionally fail due to interference from external factors
(such as processes that start / stop, network connections being
opened / closed, etc).
The trick is to loop a few times on the set of flaky conditions until
they succeed. During the last attempt all failures are recorded as
usual, while in the previous runs, the tryok() failures area ignored
but cause one more attempt to be made.

The simplest case looks like this:
  LOOP_ON_FLAKY_TESTS(3)
  {
    // ok() failures are never ignored and not retried
    ok(..., "check 1", ...);
    // tryok() failures are ignored except on the last attempt
    tryok(..., "check 2", ...);
  }

There is also:
* attempt_retry() which marks the current attempt as failed as if
  calling tryok(0), and returns true if another attempt can be made.
* attempt_failed() which returns true if an ok() call failed.
---
This is independent from the 'flaky' mechanism which adds some naming
constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS()
despite being unrelated to the flaky mechanism. The attempt_retry()
and attempt_failed() macro names also don't make it obvious that they
are related to tryok().

I think this mechanism is better than the flaky one because a flaky test
can go bad without anyone noticing, whereas if a tryok() starts failing
systematically it will cause a real failure.

The other side of that coin is that, unlike flaky, the tryok()
mechanism does not entirely eliminate the possibility of getting a
failure, it just reduces it; though by adjusting the maximum number of
attempts one can achieve an arbitrarily low failure rate. For instance
if an ok() call fails 10% of the time and one wants a maximum of 1 in
a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an
increased run time in the worst case.

This also limits the use of this mechanism to tests that have a
reasonably low failure rate to start with (otherwise one has to loop
too many times). Also note that there are cases where looping
essentially reduce the failure rate to zero. For instance
ieframe:webbrowser fails if IE creates a net session while the test is
counting them. But IE only creates the one net session on start up so
trying even one more time should guarantee that the test will succeed.
Other cases like scheduling delays and the creation of network
connections are more probabilistic in nature. Maybe a comment in test.h
should offer some guideline as to the target failure rate.

Eventually this may replace the flaky mechanism but that depends on
how well it works in practice and how practical it is to loop on flaky
tests. It seems to be going well in the few cases I looked at. But I
think this mechanism has value even if the two end up coexisting
indefinitely.
---
 dlls/kernel32/tests/debugger.c |   2 +-
 include/wine/test.h            | 200 ++++++++++++++++++++++++++++++++-
 2 files changed, 195 insertions(+), 7 deletions(-)

diff --git a/dlls/kernel32/tests/debugger.c b/dlls/kernel32/tests/debugger.c
index b0c69052936..3ec3133e7a7 100644
--- a/dlls/kernel32/tests/debugger.c
+++ b/dlls/kernel32/tests/debugger.c
@@ -63,7 +63,7 @@ static void WINAPIV __WINE_PRINTF_ATTR(2, 3) test_child_ok(int condition, const
     va_list valist;

va_start(valist, msg);
-    winetest_vok(condition, msg, valist);
+    winetest_vok(condition, 0, msg, valist);
     va_end(valist);
     if (!condition) ++child_failures;
 }
diff --git a/include/wine/test.h b/include/wine/test.h
index c20c6736e2b..871f04c412c 100644
--- a/include/wine/test.h
+++ b/include/wine/test.h
@@ -80,10 +80,11 @@ extern void winetest_wait_child_process( HANDLE process );
 #endif

extern int broken( int condition );
-extern int winetest_vok( int condition, const char *msg, va_list ap );
+extern int winetest_vok( int condition, BOOL trying, const char *msg, va_list ap );
 extern void winetest_vskip( const char *msg, va_list ap );

extern void winetest_ok( int condition, const char *msg, ... ) __WINE_PRINTF_ATTR(2,3);
+extern void winetest_tryok( int condition, const char *msg, ... ) __WINE_PRINTF_ATTR(2,3);
 extern void winetest_skip( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2);
 extern void winetest_win_skip( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2);
 extern void winetest_trace( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2);
@@ -91,27 +92,46 @@ extern void winetest_trace( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2);
 extern void winetest_push_context( const char *fmt, ... ) __WINE_PRINTF_ATTR(1, 2);
 extern void winetest_pop_context(void);

+extern void winetest_start_attempt( int retries );
+extern int winetest_loop_attempt( void );
+extern void winetest_end_attempt( void );
+extern BOOL winetest_attempt_retry( BOOL condition );
+extern BOOL winetest_attempt_failed( void );
+
 #ifdef WINETEST_NO_LINE_NUMBERS
 # define subtest_(file, line)  (winetest_set_location(file, 0), 0) ? (void)0 : winetest_subtest
 # define ignore_exceptions_(file, line)  (winetest_set_location(file, 0), 0) ? (void)0 : winetest_ignore_exceptions
 # define ok_(file, line)       (winetest_set_location(file, 0), 0) ? (void)0 : winetest_ok
+# define tryok_(file, line)    (winetest_set_location(file, 0), 0) ? (void)0 : winetest_tryok
 # define skip_(file, line)     (winetest_set_location(file, 0), 0) ? (void)0 : winetest_skip
 # define win_skip_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_win_skip
 # define trace_(file, line)    (winetest_set_location(file, 0), 0) ? (void)0 : winetest_trace
 # define wait_child_process_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_wait_child_process
+# define start_attempt_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_start_attempt
+# define loop_attempt_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_loop_attempt
+# define end_attempt_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_end_attempt
+# define attempt_retry_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_attempt_retry
+# define attempt_failed_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_attempt_failed
 #else
 # define subtest_(file, line)  (winetest_set_location(file, line), 0) ? (void)0 : winetest_subtest
 # define ignore_exceptions_(file, line)  (winetest_set_location(file, line), 0) ? (void)0 : winetest_ignore_exceptions
 # define ok_(file, line)       (winetest_set_location(file, line), 0) ? (void)0 : winetest_ok
+# define tryok_(file, line)    (winetest_set_location(file, line), 0) ? (void)0 : winetest_tryok
 # define skip_(file, line)     (winetest_set_location(file, line), 0) ? (void)0 : winetest_skip
 # define win_skip_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_win_skip
 # define trace_(file, line)    (winetest_set_location(file, line), 0) ? (void)0 : winetest_trace
 # define wait_child_process_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_wait_child_process
+# define start_attempt_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_start_attempt
+# define loop_attempt_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_loop_attempt
+# define end_attempt_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_end_attempt
+# define attempt_retry_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_attempt_retry
+# define attempt_failed_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_attempt_failed
 #endif

#define subtest  subtest_(__FILE__, __LINE__)
 #define ignore_exceptions  ignore_exceptions_(__FILE__, __LINE__)
 #define ok       ok_(__FILE__, __LINE__)
+#define tryok    tryok_(__FILE__, __LINE__)
 #define skip     skip_(__FILE__, __LINE__)
 #define win_skip win_skip_(__FILE__, __LINE__)
 #define trace    trace_(__FILE__, __LINE__)
@@ -130,6 +150,17 @@ extern void winetest_pop_context(void);
 #define todo_wine               todo_if(!strcmp(winetest_platform, "wine"))
 #define todo_wine_if(is_todo)   todo_if((is_todo) && !strcmp(winetest_platform, "wine"))

+#define start_attempt    start_attempt_(__FILE__, __LINE__)
+#define loop_attempt     loop_attempt_(__FILE__, __LINE__)
+#define end_attempt      end_attempt_(__FILE__, __LINE__)
+#define attempt_retry    attempt_retry_(__FILE__, __LINE__)
+#define attempt_failed   attempt_failed_(__FILE__, __LINE__)
+#define LOOP_ON_FLAKY_TESTS(count) for (start_attempt((count) - 1); \
+                                        loop_attempt(); \
+                                        end_attempt())
+#define LOOP_ON_FLAKY_WINDOWS_TESTS(count) LOOP_ON_FLAKY_TESTS(strcmp(winetest_platform, "windows") == 0 ? (count) : 1)
+#define LOOP_ON_FLAKY_WINE_TESTS(count) LOOP_ON_FLAKY_TESTS(strcmp(winetest_platform, "windows") == 0 ? 1 : (count))
+

#ifndef ARRAY_SIZE
 # define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
@@ -240,6 +271,12 @@ static LONG muted_todo_successes; /* same as todo_successes but silent */
 /* counts how many times a given line printed a message */
 static LONG line_counters[16384];

+#define ATTEMPT_NONE     0
+#define ATTEMPT_STARTED  1
+#define ATTEMPT_RETRY    2
+#define ATTEMPT_BADOK    4
+#define ATTEMPT_FAILED   (ATTEMPT_RETRY | ATTEMPT_BADOK)
+
 /* The following data must be kept track of on a per-thread basis */
 struct tls_data
 {
@@ -253,6 +290,8 @@ struct tls_data
     char strings[2000];              /* buffer for debug strings */
     char context[8][128];            /* data to print before messages */
     unsigned int context_count;      /* number of context prefixes */
+    int attempt_status;
+    int attempt_retries;
 };
 static DWORD tls_index;

@@ -267,6 +306,8 @@ static struct tls_data *get_tls_data(void)
     {
         data = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof(*data));
         data->str_pos = data->strings;
+        data->attempt_status = ATTEMPT_NONE;
+        data->attempt_retries = 0;
         TlsSetValue(tls_index,data);
     }
     SetLastError(last_error);
@@ -403,12 +444,13 @@ static LONG winetest_add_line( void )
  *
  * Parameters:
  *   - condition - true if the check succeeded, false otherwise;
+ *   - trying - if non-zero the failure may be ignored if in an attempt loop;
  *   - msg - failure message format;
  *   - args - arguments for the failure message
  * Return:
  *   0 if condition does not have the expected value, 1 otherwise
  */
-int winetest_vok( int condition, const char *msg, va_list args )
+int winetest_vok( int condition, BOOL trying, const char *msg, va_list args )
 {
     struct tls_data *data = get_tls_data();

@@ -417,12 +459,23 @@ int winetest_vok( int condition, const char *msg, va_list args )
         if (condition)
         {
             winetest_print_lock();
-            if (data->flaky_level)
+            if (trying && data->attempt_retries > 0)
+            {
+                if (winetest_report_success ||
+                    (winetest_time && GetTickCount() >= winetest_last_time + 1000))
+                {
+                    winetest_print_context( data->flaky_level ? "Retry successful test inside flaky todo block: " : "Retry successful test inside todo block: ");
+                    vprintf(msg, args);
+                }
+                data->attempt_status |= ATTEMPT_RETRY;
+            }
+            else if (data->flaky_level)
             {
                 if (winetest_color) printf( color_dark_purple );
                 winetest_print_context( "Test succeeded inside flaky todo block: " );
                 vprintf(msg, args);
                 InterlockedIncrement(&flaky_failures);
+                data->attempt_status |= ATTEMPT_BADOK;
             }
             else
             {
@@ -430,6 +483,7 @@ int winetest_vok( int condition, const char *msg, va_list args )
                 winetest_print_context( "Test succeeded inside todo block: " );
                 vprintf(msg, args);
                 InterlockedIncrement(&todo_failures);
+                data->attempt_status |= ATTEMPT_BADOK;
             }
             if (winetest_color) printf( color_reset );
             winetest_print_unlock();
@@ -461,12 +515,23 @@ int winetest_vok( int condition, const char *msg, va_list args )
         if (!condition)
         {
             winetest_print_lock();
-            if (data->flaky_level)
+            if (trying && data->attempt_retries > 0)
+            {
+                if (winetest_report_success ||
+                    (winetest_time && GetTickCount() >= winetest_last_time + 1000))
+                {
+                    winetest_print_context( data->flaky_level ? "Retry flaky test: " : "Retry failed test: ");
+                    vprintf(msg, args);
+                }
+                data->attempt_status |= ATTEMPT_RETRY;
+            }
+            else if (data->flaky_level)
             {
                 if (winetest_color) printf( color_bright_purple );
                 winetest_print_context( "Test marked flaky: " );
                 vprintf(msg, args);
                 InterlockedIncrement(&flaky_failures);
+                data->attempt_status |= ATTEMPT_BADOK;
             }
             else
             {
@@ -474,6 +539,7 @@ int winetest_vok( int condition, const char *msg, va_list args )
                 winetest_print_context( "Test failed: " );
                 vprintf(msg, args);
                 InterlockedIncrement(&failures);
+                data->attempt_status |= ATTEMPT_BADOK;
             }
             if (winetest_color) printf( color_reset );
             winetest_print_unlock();
@@ -501,7 +567,7 @@ void winetest_ok( int condition, const char *msg, ... )
     va_list valist;

va_start(valist, msg);
-    winetest_vok(condition, msg, valist);
+    winetest_vok(condition, FALSE, msg, valist);
     va_end(valist);
 }

@@ -555,7 +621,129 @@ void winetest_win_skip( const char *msg, ... )
     if (strcmp(winetest_platform, "windows") == 0)
         winetest_vskip(msg, valist);
     else
-        winetest_vok(0, msg, valist);
+        winetest_vok(0, FALSE, msg, valist);
+    va_end(valist);
+}
+
+/* Starts a group of tests that has a low but non-zero probability of failing
+ * due to unavoidable outside interference. This means it may be necessary
+ * to retry a few times to get a successful run.
+ *
+ * Parameters:
+ * - retries - The number of times this group of tests should be retried if it
+ *             fails.
+ *
+ * Remarks:
+ * - Nesting attempts is not allowed and results in a failure.
+ * - This is not meant to paper over race conditions within the test itself.
+ *   Those are bugs and should be fixed.
+ * - The failures should be rare to start with. If a test fails 25% of the time
+ *   or needs to be repeated more than a handful of times to succeed reliably
+ *   it is probably wrong to start with.
+ *
+ * Examples:
+ *
+ *   LOOP_ON_FLAKY_TESTS(3)
+ *   {
+ *       ...
+ *       // ok() failures are never ignored and not retried
+ *       ok(..., "check 1", ...);
+ *
+ *       // ignore tryok() failures except on the last attempt
+ *       tryok(..., "check 2", ...);
+ *       ...
+ *   }
+ *
+ *   LOOP_ON_FLAKY_TESTS(5)
+ *   {
+ *       ...
+ *       // skip if true because we know the remaining tests will fail
+ *       // will not be skipped on the last try
+ *       if (attempt_retry(condition))
+ *          continue;
+ *       ok(..., "check 1", ...);
+ *
+ *       ...
+ *       // skip to the next attempt if a tryok() failed already
+ *       if (attempt_failed()) continue;
+ *       ...
+ *   }
+ */
+void winetest_start_attempt( int retries )
+{
+    struct tls_data *data = get_tls_data();
+    if (data->attempt_status != ATTEMPT_NONE)
+         winetest_ok(0, "nesting attempt blocks is not allowed (%x/%d)\n", data->attempt_status, data->attempt_retries);
+    data->attempt_retries = retries;
+    data->attempt_status = ATTEMPT_STARTED | ATTEMPT_RETRY;
+    printf("%s:%d status=%x retries=%d\n", __FUNCTION__, __LINE__, data->attempt_status, data->attempt_retries); // FIXME
+}
+
+int winetest_loop_attempt( void )
+{
+    struct tls_data *data = get_tls_data();
+
+    printf("%s:%d status=%x retries=%d\n", __FUNCTION__, __LINE__, data->attempt_status, data->attempt_retries); // FIXME
+    if (!(data->attempt_status & ATTEMPT_FAILED) || /* no failures */
+        (data->attempt_status & ATTEMPT_BADOK) ||
+        data->attempt_retries < 0)
+    {
+        printf("%s:%d exiting loop\n", __FUNCTION__, __LINE__); // FIXME
+        data->attempt_status = ATTEMPT_NONE;
+        data->attempt_retries = 0;
+        return FALSE;
+    }
+
+    winetest_push_context("r%d", data->attempt_retries);
+    data->attempt_status = ATTEMPT_STARTED;
+    return TRUE;
+}
+
+void winetest_end_attempt( void )
+{
+    struct tls_data *data = get_tls_data();
+
+    printf("%s:%d status=%x retries=%d\n", __FUNCTION__, __LINE__, data->attempt_status, data->attempt_retries);
+    winetest_pop_context();
+    data->attempt_retries--;
+}
+
+BOOL winetest_attempt_retry( BOOL condition )
+{
+    struct tls_data *data = get_tls_data();
+    if (data->attempt_status == ATTEMPT_NONE)
+        winetest_ok(0, "attempt_retry() should be in an attempt block\n");
+    if (!condition || !data->attempt_retries ||
+        (data->attempt_status & ATTEMPT_BADOK))
+        return FALSE;
+    data->attempt_status |= ATTEMPT_RETRY;
+    return TRUE;
+}
+
+/* Returns true if a tryok(), ok() or win_skip() call failed already */
+int winetest_attempt_failed( void )
+{
+    struct tls_data *data = get_tls_data();
+    if (data->attempt_status == ATTEMPT_NONE)
+        winetest_ok(0, "attempt_failed() should be in an attempt block\n");
+    return data->attempt_status & ATTEMPT_FAILED;
+}
+
+
+/* Performs a check that may need to be repeated to avoid failures caused by
+ * race conditions. See winetest_start_attempt() for details.
+ *
+ * Parameters: see winetest_ok()
+ */
+void winetest_tryok( int condition, const char *msg, ... )
+{
+    struct tls_data *data = get_tls_data();
+    va_list valist;
+
+    if (data->attempt_status == ATTEMPT_NONE)
+        winetest_ok(0, "tryok() should be in an attempt block\n");
+    va_start(valist, msg);
+    winetest_vok(condition, TRUE, msg, valist);
     va_end(valist);
 }

[PATCH v2 3/9] kernel32/tests: Use tryok() to fix a heap race with other processes.

[Francois Gouget]

From: Francois Gouget

If other processes allocate or free enough memory (e.g. by terminating),
the available physical memory and swap space reported by the system may
change while we are testing these values.
Wine-Bug: https://bugs.winehq.org//show_bug.cgi?id=54498
---
 dlls/kernel32/tests/heap.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/dlls/kernel32/tests/heap.c b/dlls/kernel32/tests/heap.c
index 59a82c0c579..003c30d5f00 100644
--- a/dlls/kernel32/tests/heap.c
+++ b/dlls/kernel32/tests/heap.c
@@ -3595,9 +3595,11 @@ static void test_GlobalMemoryStatus(void)

ok( memex.dwMemoryLoad == expect.dwMemoryLoad, "got dwMemoryLoad %lu\n", memex.dwMemoryLoad );
     ok( memex.ullTotalPhys == expect.ullTotalPhys, "got ullTotalPhys %#I64x\n", memex.ullTotalPhys );
-    ok( IS_WITHIN_RANGE( memex.ullAvailPhys, expect.ullAvailPhys ), "got ullAvailPhys %#I64x\n", memex.ullAvailPhys );
+    /* ullAvailPhys may change if a big process starts / stops */
+    tryok( IS_WITHIN_RANGE( memex.ullAvailPhys, expect.ullAvailPhys ), "got ullAvailPhys %#I64x\n", memex.ullAvailPhys );
     ok( memex.ullTotalPageFile == expect.ullTotalPageFile, "got ullTotalPageFile %#I64x\n", memex.ullTotalPageFile );
-    ok( IS_WITHIN_RANGE( memex.ullAvailPageFile, expect.ullAvailPageFile ), "got ullAvailPageFile %#I64x\n", memex.ullAvailPageFile );
+    /* ullAvailPageFile may change if a big process starts / stops */
+    tryok( IS_WITHIN_RANGE( memex.ullAvailPageFile, expect.ullAvailPageFile ), "got ullAvailPageFile %#I64x\n", memex.ullAvailPageFile );
     ok( memex.ullTotalVirtual == expect.ullTotalVirtual, "got ullTotalVirtual %#I64x\n", memex.ullTotalVirtual );
     ok( memex.ullAvailVirtual <= expect.ullAvailVirtual, "got ullAvailVirtual %#I64x\n", memex.ullAvailVirtual );
     ok( memex.ullAvailExtendedVirtual == 0, "got ullAvailExtendedVirtual %#I64x\n", memex.ullAvailExtendedVirtual );
@@ -3720,7 +3722,9 @@ START_TEST(heap)
     test_LocalAlloc();

test_GetPhysicallyInstalledSystemMemory();
-    test_GlobalMemoryStatus();
+
+    /* The system memory usage may change during the test */
+    LOOP_ON_FLAKY_TESTS(3) test_GlobalMemoryStatus();

if (pRtlGetNtGlobalFlags)
     {

[PATCH v2 4/9] advapi32/tests: Replace the custom loop with tryok() mechanism.

[Francois Gouget]

From: Francois Gouget

The custom loop was meant to work around the race condition with
services being started / stopped by the Windows while the test is
running. The tryok() + LOOP_ON_FLAKY_TESTS() mechanism provides the same
functionality in a simpler to use and standardized form.
---
 dlls/advapi32/tests/service.c | 123 ++++++++++++++++------------------
 1 file changed, 56 insertions(+), 67 deletions(-)

diff --git a/dlls/advapi32/tests/service.c b/dlls/advapi32/tests/service.c
index 8cc737207e0..c90237fad8f 100644
--- a/dlls/advapi32/tests/service.c
+++ b/dlls/advapi32/tests/service.c
@@ -1062,10 +1062,10 @@ static void test_query_svc(void)
     CloseServiceHandle(scm_handle);
 }

-static BOOL test_enum_svc(int attempt)
+static void test_enum_svc(void)
 {
     SC_HANDLE scm_handle;
-    BOOL ret, alldone = FALSE;
+    BOOL ret;
     DWORD bufsize, needed, returned, resume;
     DWORD neededA, returnedA;
     DWORD tempneeded, tempreturned, missing;
@@ -1266,9 +1266,8 @@ static BOOL test_enum_svc(int attempt)
        "Expected ERROR_MORE_DATA, got %ld\n", GetLastError());
     ok(neededA != 0xdeadbeef && neededA > 0, "Expected the needed buffer size for this one service\n");
     ok(returnedA == 0, "Expected no service returned, got %ld\n", returnedA);
-    if (neededA != needed && attempt)
-         goto retry; /* service start|stop race condition */
-    ok(neededA == needed, "Expected needed buffersize to be the same for A- and W-calls\n");
+    tryok(neededA == needed, "Expected needed buffersize to be the same for A- and W-calls\n");
+    if (attempt_failed()) goto cleanup; /* service start|stop race condition */

/* Store the needed bytes */
     tempneeded = needed;
@@ -1282,8 +1281,8 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, bufsize, &needed, &returned, NULL);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start|stop race condition */
     ok(ret, "Expected success, got error %lu\n", GetLastError());
     ok(needed == 0, "Expected needed buffer to be 0 as we are done\n");
     ok(returned != 0xdeadbeef && returned > 0, "Expected some returned services\n");
@@ -1299,8 +1298,8 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusA(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               servicesA, bufsize, &neededA, &returnedA, NULL);
     HeapFree(GetProcessHeap(), 0, servicesA);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     if (!ret && GetLastError() == ERROR_NOT_ENOUGH_MEMORY && GetACP() == CP_UTF8)
          win_skip("Skipping some tests due to EnumServicesStatusA()'s broken UTF-8 support\n");
     else
@@ -1321,10 +1320,10 @@ static BOOL test_enum_svc(int attempt)
     SetLastError(0xdeadbeef);
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, bufsize, &needed, &returned, NULL);
-    if (ret && needed == 0 && attempt)
+    if (attempt_retry(ret && needed == 0))
     {
         HeapFree(GetProcessHeap(), 0, services);
-        goto retry; /* service stop race condition */
+        goto cleanup; /* service stop race condition */
     }
     ok(!ret, "Expected failure\n");
     ok(needed != 0xdeadbeef && needed > 0, "Expected the needed buffer size\n");
@@ -1344,10 +1343,10 @@ static BOOL test_enum_svc(int attempt)
     SetLastError(0xdeadbeef);
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, bufsize, &needed, &returned, &resume);
-    if (ret && needed == 0 && attempt)
+    if (attempt_retry(ret && needed == 0))
     {
         HeapFree(GetProcessHeap(), 0, services);
-        goto retry; /* service stop race condition */
+        goto cleanup; /* service stop race condition */
     }
     ok(!ret, "Expected failure\n");
     ok(needed != 0xdeadbeef && needed > 0, "Expected the needed buffer size for the missing services, got %lx\n", needed);
@@ -1366,8 +1365,8 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, bufsize, &needed, &returned, &resume);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success, got error %lu\n", GetLastError());
     ok(needed == 0, "Expected 0 needed bytes as we are done, got %lu\n", needed);
     todo_wine ok(returned == missing, "Expected %lu remaining services, got %lu\n", missing, returned);
@@ -1392,8 +1391,8 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_ACTIVE,
                               services, needed, &needed, &returned, NULL);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

servicecountactive = returned;

@@ -1404,8 +1403,8 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_INACTIVE,
                               services, needed, &needed, &returned, NULL);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

servicecountinactive = returned;

@@ -1416,12 +1415,12 @@ static BOOL test_enum_svc(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, needed, &needed, &returned, NULL);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

/* Check if total is the same as active and inactive win32 services */
-    if (returned != servicecountactive + servicecountinactive && attempt)
-         goto retry; /* service start|stop race condition */
+    if (attempt_retry(returned != servicecountactive + servicecountinactive))
+         goto cleanup; /* service start|stop race condition */
     ok(returned == servicecountactive + servicecountinactive,
        "Expected service count %ld == %ld + %ld\n",
        returned, servicecountactive, servicecountinactive);
@@ -1436,8 +1435,8 @@ static BOOL test_enum_svc(int attempt)
     services = HeapAlloc(GetProcessHeap(), 0, needed);
     ret = EnumServicesStatusW(scm_handle, SERVICE_DRIVER | SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, needed, &needed, &returned, NULL);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success %lu\n", GetLastError());

/* Loop through all those returned drivers and services */
@@ -1482,22 +1481,19 @@ static BOOL test_enum_svc(int attempt)
     }
     HeapFree(GetProcessHeap(), 0, services);

-    if ((servicecountactive || servicecountinactive) && attempt)
-         goto retry; /* service start|stop race condition */
+    if (attempt_retry(servicecountactive || servicecountinactive))
+         goto cleanup; /* service start|stop race condition */
     ok(servicecountactive == 0, "Active services mismatch %lu\n", servicecountactive);
     ok(servicecountinactive == 0, "Inactive services mismatch %lu\n", servicecountinactive);

-    alldone = TRUE;
-
-retry:
+cleanup:
     CloseServiceHandle(scm_handle);
-    return alldone;
 }

-static BOOL test_enum_svc_ex(int attempt)
+static void test_enum_svc_ex(void)
 {
     SC_HANDLE scm_handle;
-    BOOL ret, alldone = FALSE;
+    BOOL ret;
     DWORD bufsize, needed, returned, resume;
     DWORD neededA, returnedA;
     DWORD tempneeded, tempreturned, missing;
@@ -1510,7 +1506,7 @@ static BOOL test_enum_svc_ex(int attempt)
     if (!pEnumServicesStatusExA)
     {
         win_skip( "EnumServicesStatusExA not available\n" );
-        return TRUE;
+        return;
     }

/* All NULL or wrong */
@@ -1681,9 +1677,8 @@ static BOOL test_enum_svc_ex(int attempt)
        "Expected ERROR_MORE_DATA, got %ld\n", GetLastError());
     ok(neededA != 0xdeadbeef && neededA > 0, "Expected the needed buffer size for this one service\n");
     ok(returnedA == 0, "Expected no service returned, got %ld\n", returnedA);
-    if (neededA != needed && attempt)
-         goto retry; /* service start|stop race condition */
     ok(neededA == needed, "Expected needed buffersize to be the same for A- and W-calls\n");
+    if (attempt_failed()) goto cleanup; /* service start|stop race condition */
     ok(GetLastError() == ERROR_MORE_DATA,
        "Expected ERROR_MORE_DATA, got %ld\n", GetLastError());

@@ -1698,8 +1693,8 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = EnumServicesStatusW(scm_handle, SERVICE_WIN32, SERVICE_STATE_ALL,
                               services, needed, &needed, &returned, NULL);
     HeapFree(GetProcessHeap(), 0, services);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success, got error %lu\n", GetLastError());
     ok(needed == 0, "Expected needed buffer to be 0 as we are done\n");
     ok(returned != 0xdeadbeef && returned > 0, "Expected some returned services\n");
@@ -1716,8 +1711,8 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_STATE_ALL,
                                  (BYTE*)exservices, bufsize, &needed, &returned, NULL, NULL);
     HeapFree(GetProcessHeap(), 0, exservices);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success, got error %lu\n", GetLastError());
     ok(needed == 0, "Expected needed buffer to be 0 as we are done\n");
     ok(returned == tempreturned, "Expected the same number of service from this function\n");
@@ -1736,10 +1731,10 @@ static BOOL test_enum_svc_ex(int attempt)
     SetLastError(0xdeadbeef);
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_STATE_ALL,
                                  (BYTE*)exservices, bufsize, &needed, &returned, NULL, NULL);
-    if (ret && needed == 0 && attempt)
+    if (attempt_retry(ret && needed == 0))
     {
         HeapFree(GetProcessHeap(), 0, exservices);
-        goto retry; /* service stop race condition */
+        goto cleanup; /* service stop race condition */
     }
     ok(!ret, "Expected failure\n");
     ok(needed != 0xdeadbeef && needed > 0, "Expected the needed buffer size for the missing services\n");
@@ -1759,10 +1754,10 @@ static BOOL test_enum_svc_ex(int attempt)
     SetLastError(0xdeadbeef);
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_STATE_ALL,
                                  (BYTE*)exservices, bufsize, &needed, &returned, &resume, NULL);
-    if (ret && needed == 0 && attempt)
+    if (attempt_retry(ret && needed == 0))
     {
         HeapFree(GetProcessHeap(), 0, exservices);
-        goto retry; /* service stop race condition */
+        goto cleanup; /* service stop race condition */
     }
     ok(!ret, "Expected failure\n");
     ok(needed != 0xdeadbeef && needed > 0, "Expected the needed buffer size for the missing services\n");
@@ -1781,8 +1776,8 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_STATE_ALL,
                                  (BYTE*)exservices, bufsize, &needed, &returned, &resume, NULL);
     HeapFree(GetProcessHeap(), 0, exservices);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success, got error %lu\n", GetLastError());
     ok(needed == 0, "Expected needed buffer to be 0 as we are done\n");
     ok(returned == missing, "Expected %lu services to be returned\n", missing);
@@ -1797,8 +1792,8 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_ACTIVE,
                                  (BYTE*)exservices, needed, &needed, &returned, NULL, NULL);
     HeapFree(GetProcessHeap(), 0, exservices);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

servicecountactive = returned;

@@ -1809,8 +1804,8 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_INACTIVE,
                                  (BYTE*)exservices, needed, &needed, &returned, NULL, NULL);
     HeapFree(GetProcessHeap(), 0, exservices);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

servicecountinactive = returned;

@@ -1821,12 +1816,12 @@ static BOOL test_enum_svc_ex(int attempt)
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32, SERVICE_STATE_ALL,
                                  (BYTE*)exservices, needed, &needed, &returned, NULL, NULL);
     HeapFree(GetProcessHeap(), 0, exservices);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */

/* Check if total is the same as active and inactive win32 services */
-    if (returned != servicecountactive + servicecountinactive && attempt)
-         goto retry; /* service start|stop race condition */
+    if (attempt_retry(returned != servicecountactive + servicecountinactive))
+         goto cleanup; /* service start|stop race condition */
     ok(returned == servicecountactive + servicecountinactive,
        "Expected service count %ld == %ld + %ld\n",
        returned, servicecountactive, servicecountinactive);
@@ -1838,8 +1833,8 @@ static BOOL test_enum_svc_ex(int attempt)
     exservices = HeapAlloc(GetProcessHeap(), 0, needed);
     ret = pEnumServicesStatusExW(scm_handle, 0, SERVICE_WIN32 | SERVICE_DRIVER,
                                  SERVICE_STATE_ALL, (BYTE*)exservices, needed, &needed, &returned, NULL, NULL);
-    if (!ret && GetLastError() == ERROR_MORE_DATA && attempt)
-        goto retry; /* service start race condition */
+    if (attempt_retry(!ret && GetLastError() == ERROR_MORE_DATA))
+        goto cleanup; /* service start race condition */
     ok(ret, "Expected success %lu\n", GetLastError());

/* Loop through all those returned drivers and services */
@@ -1908,16 +1903,13 @@ static BOOL test_enum_svc_ex(int attempt)
     }
     HeapFree(GetProcessHeap(), 0, exservices);

-    if ((servicecountactive || servicecountinactive) && attempt)
-         goto retry; /* service start|stop race condition */
+    if (attempt_retry(servicecountactive || servicecountinactive))
+         goto cleanup; /* service start|stop race condition */
     ok(servicecountactive == 0, "Active services mismatch %lu\n", servicecountactive);
     ok(servicecountinactive == 0, "Inactive services mismatch %lu\n", servicecountinactive);

-    alldone = TRUE;
-
-retry:
+cleanup:
     CloseServiceHandle(scm_handle);
-    return alldone;
 }

static void test_close(void)
@@ -2999,7 +2991,6 @@ START_TEST(service)
     SC_HANDLE scm_handle;
     int myARGC;
     char** myARGV;
-    int attempt;

myARGC = winetest_get_mainargs(&myARGV);
     GetFullPathNameA(myARGV[0], sizeof(selfname), selfname, NULL);
@@ -3034,10 +3025,8 @@ START_TEST(service)
     /* Services may start or stop between enumeration calls, leading to
      * inconsistencies and failures. So we may need a couple attempts.
      */
-    for (attempt = 2; attempt >= 0; attempt--)
-        if (test_enum_svc(attempt)) break;
-    for (attempt = 2; attempt >= 0; attempt--)
-        if (test_enum_svc_ex(attempt)) break;
+    LOOP_ON_FLAKY_WINDOWS_TESTS(3) test_enum_svc();
+    LOOP_ON_FLAKY_WINDOWS_TESTS(3) test_enum_svc_ex();

test_close();
     test_wow64();

[PATCH v2 5/9] ieframe/tests: Work around a network session race condition.

[Francois Gouget]

From: Francois Gouget

IE may create a network session for its own purposes, thus changing
the session count while we are testing it.

Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=54866
---
 dlls/ieframe/tests/webbrowser.c | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/dlls/ieframe/tests/webbrowser.c b/dlls/ieframe/tests/webbrowser.c
index 15ce0272654..adb43f481f6 100644
--- a/dlls/ieframe/tests/webbrowser.c
+++ b/dlls/ieframe/tests/webbrowser.c
@@ -4436,20 +4436,24 @@ static void test_SetQueryNetSessionCount(void)
 {
     LONG count, init_count;

-    init_count = pSetQueryNetSessionCount(SESSION_QUERY);
-    trace("init_count %ld\n", init_count);
+    /* IE may create a network session for its own purposes during the test */
+    LOOP_ON_FLAKY_WINDOWS_TESTS(3)
+    {
+        init_count = pSetQueryNetSessionCount(SESSION_QUERY);
+        trace("init_count %ld\n", init_count);

-    count = pSetQueryNetSessionCount(SESSION_INCREMENT);
-    ok(count == init_count + 1, "count = %ld\n", count);
+        count = pSetQueryNetSessionCount(SESSION_INCREMENT);
+        tryok(count == init_count + 1, "count = %ld\n", count);

-    count = pSetQueryNetSessionCount(SESSION_QUERY);
-    ok(count == init_count + 1, "count = %ld\n", count);
+        count = pSetQueryNetSessionCount(SESSION_QUERY);
+        tryok(count == init_count + 1, "count = %ld\n", count);

-    count = pSetQueryNetSessionCount(SESSION_DECREMENT);
-    ok(count == init_count, "count = %ld\n", count);
+        count = pSetQueryNetSessionCount(SESSION_DECREMENT);
+        tryok(count == init_count, "count = %ld\n", count);

-    count = pSetQueryNetSessionCount(SESSION_QUERY);
-    ok(count == init_count, "count = %ld\n", count);
+        count = pSetQueryNetSessionCount(SESSION_QUERY);
+        tryok(count == init_count, "count = %ld\n", count);
+    }
 }

static HRESULT WINAPI outer_QueryInterface(IUnknown *iface, REFIID riid, void **ppv)

[PATCH v2 6/9] DEBUG ieframe/tests: tryok() framework testing ground.

[Francois Gouget]

From: Francois Gouget

Add code to test various tryok() scenarios and make sure it behaves as
expected.
---
 dlls/ieframe/tests/webbrowser.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/dlls/ieframe/tests/webbrowser.c b/dlls/ieframe/tests/webbrowser.c
index adb43f481f6..684bd4b358c 100644
--- a/dlls/ieframe/tests/webbrowser.c
+++ b/dlls/ieframe/tests/webbrowser.c
@@ -4436,6 +4436,7 @@ static void test_SetQueryNetSessionCount(void)
 {
     LONG count, init_count;

+    trace("-----\n");
     /* IE may create a network session for its own purposes during the test */
     LOOP_ON_FLAKY_WINDOWS_TESTS(3)
     {
@@ -4454,6 +4455,37 @@ static void test_SetQueryNetSessionCount(void)
         count = pSetQueryNetSessionCount(SESSION_QUERY);
         tryok(count == init_count, "count = %ld\n", count);
     }
+
+    trace("----- fatal loop\n");
+    LOOP_ON_FLAKY_TESTS(5)
+    {
+        init_count = pSetQueryNetSessionCount(SESSION_QUERY);
+        trace("init_count %ld\n", init_count);
+
+        count = pSetQueryNetSessionCount(SESSION_QUERY);
+        ok(count == init_count + 1, "count = %ld\n", count);
+    }
+
+    trace("----- infinite todo loop\n");
+    LOOP_ON_FLAKY_TESTS(2)
+    {
+        init_count = pSetQueryNetSessionCount(SESSION_QUERY);
+        trace("init_count %ld\n", init_count);
+
+        count = pSetQueryNetSessionCount(SESSION_QUERY);
+        todo_wine tryok(count == init_count, "count = %ld\n", count);
+    }
+
+    trace("----- infinite todo loop\n");
+    LOOP_ON_FLAKY_TESTS(2)
+    {
+        init_count = pSetQueryNetSessionCount(SESSION_QUERY);
+        trace("init_count %ld\n", init_count);
+
+        count = pSetQueryNetSessionCount(SESSION_QUERY);
+        todo_wine tryok(count == init_count, "count = %ld\n", count);
+        if (attempt_failed()) break;
+    }
 }

static HRESULT WINAPI outer_QueryInterface(IUnknown *iface, REFIID riid, void **ppv)
@@ -4522,6 +4554,7 @@ START_TEST(webbrowser)
     OleInitialize(NULL);

test_SetQueryNetSessionCount();
+    if (1) return;

container_hwnd = create_container_window();

[PATCH v2 7/9] quartz/tests: Replace flaky() with tryok() to work around scheduling delays.

[Francois Gouget]

From: Francois Gouget

---
My understanding is that the failures in quartz:filtergraph are caused
by scheduling delays, except maybe the EOS test? If so, then that means
the issue is not in the test itself and comes from an external source
(other processes) which corresponds to a tryok() use case.
So far we've used flaky() to avoid failures. This commit is a test to
see if tryok() is a workable replacement mechanism.
---
 dlls/quartz/tests/filtergraph.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/dlls/quartz/tests/filtergraph.c b/dlls/quartz/tests/filtergraph.c
index 00232ea7a52..77f7cd0a9cb 100644
--- a/dlls/quartz/tests/filtergraph.c
+++ b/dlls/quartz/tests/filtergraph.c
@@ -430,8 +430,7 @@ static void test_state_change(IFilterGraph2 *graph)
     hr = IMediaControl_Pause(control);
     ok(SUCCEEDED(hr), "Got hr %#lx.\n", hr);
     hr = IMediaControl_GetState(control, 1000, &state);
-    flaky_wine
-    ok(hr == S_OK, "Got hr %#lx.\n", hr);
+    tryok(hr == S_OK, "Got hr %#lx.\n", hr);
     ok(state == State_Paused, "Got state %ld.\n", state);

hr = IMediaControl_Run(control);
@@ -520,13 +519,11 @@ static void test_media_event(IFilterGraph2 *graph)
             }
         }
     }
-    flaky_wine
-    ok(got_eos, "didn't get EOS\n");
+    tryok(got_eos, "didn't get EOS\n");

hr = IMediaSeeking_GetCurrentPosition(seeking, &current);
     ok(hr == S_OK, "Got hr %#lx.\n", hr);
-    flaky_wine
-    ok(current == stop, "expected %s, got %s\n", wine_dbgstr_longlong(stop), wine_dbgstr_longlong(current));
+    tryok(current == stop, "expected %s, got %s\n", wine_dbgstr_longlong(stop), wine_dbgstr_longlong(current));

hr = IMediaControl_Stop(control);
     ok(SUCCEEDED(hr), "Got hr %#lx.\n", hr);
@@ -547,8 +544,8 @@ static void rungraph(IFilterGraph2 *graph, BOOL video)
     if (video)
         test_basic_video(graph);
     test_media_seeking(graph);
-    test_state_change(graph);
-    test_media_event(graph);
+    LOOP_ON_FLAKY_WINE_TESTS(5) test_state_change(graph);
+    LOOP_ON_FLAKY_WINE_TESTS(5) test_media_event(graph);
 }

static HRESULT test_graph_builder_connect_file(WCHAR *filename, BOOL audio, BOOL video)

[PATCH v2 8/9] mmdevapi/tests: Replace flaky with tryok() in the render tests.

[Francois Gouget]

From: Francois Gouget

tryok() can be used to work around scheduling delays.

---
This provides an attempt_failed() use case.
---
 dlls/mmdevapi/tests/render.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/dlls/mmdevapi/tests/render.c b/dlls/mmdevapi/tests/render.c
index 8e000f03acb..b9955975158 100644
--- a/dlls/mmdevapi/tests/render.c
+++ b/dlls/mmdevapi/tests/render.c
@@ -2211,10 +2211,9 @@ static void test_worst_case(void)
         hr = IAudioClient_Start(ac);
         ok(hr == S_OK, "Start failed: %08lx\n", hr);

-        for(i = 0; i <= 99; i++){ /* 100 x 10ms = 1 second */
+        for(i = 0; i <= 99 && !attempt_failed(); i++){ /* 100 x 10ms = 1 second */
             r = WaitForSingleObject(event, 60 + defp / 10000);
-            flaky_wine
-            ok(r == WAIT_OBJECT_0, "Wait iteration %d gave %lx\n", i, r);
+            tryok(r == WAIT_OBJECT_0, "Wait iteration %d gave %lx\n", i, r);

/* the app has nearly one period time to feed data */
             Sleep((i % 10) * defp / 120000);
@@ -2413,7 +2412,7 @@ START_TEST(render)
     test_simplevolume();
     test_volume_dependence();
     test_session_creation();
-    test_worst_case();
+    LOOP_ON_FLAKY_WINE_TESTS(3) test_worst_case();
     test_endpointvolume();

IMMDevice_Release(dev);

[PATCH v2 9/9] mmdevapi/tests: Replace flaky with tryok() in the capture tests.

[Francois Gouget]

From: Francois Gouget

tryok() can be used to work around scheduling delays.

---
Note that the wine-specific aspect of flaky_wine is all moved to the wrapping
LOOP_ON_FLAKY_WINE_TESTS() loop. There is no flaky_windows anyway so in
case of a mix of flaky and flaky_wine one would simply use
LOOP_ON_FLAKY_TESTS().
---
 dlls/mmdevapi/tests/capture.c | 35 ++++++++++++-----------------------
 1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/dlls/mmdevapi/tests/capture.c b/dlls/mmdevapi/tests/capture.c
index f3b426a7f9a..7949523db3d 100644
--- a/dlls/mmdevapi/tests/capture.c
+++ b/dlls/mmdevapi/tests/capture.c
@@ -184,12 +184,10 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
      * whereas GetCurrentPadding grows when input is not consumed. */
     hr = IAudioCaptureClient_GetNextPacketSize(acc, &next);
     ok(hr == S_OK, "IAudioCaptureClient_GetNextPacketSize returns %08lx\n", hr);
-    flaky_wine
-    ok(next <  pad, "GetNextPacketSize %u vs. GCP %u\n", next, pad);
+    tryok(next <  pad, "GetNextPacketSize %u vs. GCP %u\n", next, pad);

hr = IAudioCaptureClient_GetBuffer(acc, &data, &frames, &flags, &pos, &qpc);
-    flaky_wine
-    ok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
+    tryok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
     ok(next == frames, "GetNextPacketSize %u vs. GetBuffer %u\n", next, frames);

if(hr == S_OK){
@@ -258,8 +256,7 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
     }

frames = period;
-    flaky_wine
-    ok(next == frames, "GetNextPacketSize %u vs. GetDevicePeriod %u\n", next, frames);
+    tryok(next == frames, "GetNextPacketSize %u vs. GetDevicePeriod %u\n", next, frames);

/* GetBufferSize is not a multiple of the period size! */
     hr = IAudioClient_GetBufferSize(ac, &next);
@@ -272,8 +269,7 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
     ok(hr == S_OK, "GetCurrentPadding call returns %08lx\n", hr);

hr = IAudioCaptureClient_GetBuffer(acc, &data, &frames, &flags, &pos, &qpc);
-    flaky_wine
-    ok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
+    tryok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);

trace("Overrun position %d pad %u flags %lx, amount of frames locked: %u\n",
           hr==S_OK ? (UINT)pos : -1, pad, flags, frames);
@@ -300,17 +296,14 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
     ok(hr == S_OK, "GetCurrentPadding call returns %08lx\n", hr);

hr = IAudioCaptureClient_GetBuffer(acc, &data, &frames, &flags, &pos, &qpc);
-    flaky_wine
-    ok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
+    tryok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);

trace("Cont'ed position %d pad %u flags %lx, amount of frames locked: %u\n",
           hr==S_OK ? (UINT)pos : -1, pad, flags, frames);

if(hr == S_OK){
-        flaky_wine
-        ok(pos == sum, "Position %u expected %u\n", (UINT)pos, sum);
-        flaky_wine
-        ok(!flags, "flags %lu\n", flags);
+        tryok(pos == sum, "Position %u expected %u\n", (UINT)pos, sum);
+        tryok(!flags, "flags %lu\n", flags);

hr = IAudioCaptureClient_ReleaseBuffer(acc, frames);
         ok(hr == S_OK, "Releasing buffer returns %08lx\n", hr);
@@ -324,20 +317,17 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
     ok(hr == S_OK, "Start on a stopped stream returns %08lx\n", hr);

hr = IAudioCaptureClient_GetBuffer(acc, &data, &frames, &flags, &pos, &qpc);
-    flaky_wine
-    ok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
+    tryok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);

hr = IAudioClient_GetCurrentPadding(ac, &pad);
     ok(hr == S_OK, "GetCurrentPadding call returns %08lx\n", hr);

trace("Restart position %d pad %u flags %lx, amount of frames locked: %u\n",
           hr==S_OK ? (UINT)pos : -1, pad, flags, frames);
-    flaky_wine
-    ok(pad > sum, "restarted GCP %u\n", pad); /* GCP is still near buffer size */
+    tryok(pad > sum, "restarted GCP %u\n", pad); /* GCP is still near buffer size */

if(frames){
-        flaky_wine
-        ok(pos == sum, "Position %u expected %u\n", (UINT)pos, sum);
+        tryok(pos == sum, "Position %u expected %u\n", (UINT)pos, sum);
         ok(!flags, "flags %lu\n", flags);

hr = IAudioCaptureClient_ReleaseBuffer(acc, frames);
@@ -376,8 +366,7 @@ static void test_capture(IAudioClient *ac, HANDLE handle, WAVEFORMATEX *wfx)
     ok(hr == S_OK, "GetCurrentPadding call returns %08lx\n", hr);

hr = IAudioCaptureClient_GetBuffer(acc, &data, &frames, &flags, &pos, &qpc);
-    flaky_wine
-    ok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
+    tryok(hr == S_OK, "Valid IAudioCaptureClient_GetBuffer returns %08lx\n", hr);
     trace("Running position %d pad %u flags %lx, amount of frames locked: %u\n",
           SUCCEEDED(hr) ? (UINT)pos : -1, pad, flags, frames);

@@ -550,7 +539,7 @@ static void test_audioclient(void)
     hr = IAudioClient_Stop(ac);
     ok(hr == S_FALSE, "Stop on a stopped stream returns %08lx\n", hr);

-    test_capture(ac, handle, pwfx);
+    LOOP_ON_FLAKY_WINE_TESTS(3) test_capture(ac, handle, pwfx);

cleanup:
     IAudioClient_Release(ac);

[PATCH v2 9/9] mmdevapi/tests: Replace flaky with tryok() in the capture tests.

[Marvin]

Hi,

It looks like your patch introduced the new failures shown below.
Please investigate and fix them before resubmitting your patch.
If they are not new, fixing them anyway would help a lot. Otherwise
please ask for the known failures list to be updated.

The tests also ran into some preexisting test failures. If you know how
to fix them that would be helpful. See the TestBot job for the details:

The full results can be found at:
https://testbot.winehq.org/JobDetails.pl?Key=135280

Your paranoid android.

=== w7u_2qxl (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w7u_adm (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w7u_el (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w8 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w8adm (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w864 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064v1507 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064v1809 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064_tsign (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w11pro64 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w7pro64 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w864 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064v1507 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064v1809 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064_2qxl (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064_adm (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w1064_tsign (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64_en_AE_u8 (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64_ar (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64_ja (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w10pro64_zh_CN (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w11pro64_amd (64 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0

=== w864 (32 bit report) ===

mmdevapi:
capture.c:285: Test failed: r0: Position 1344 last 896 frames 448
capture.c:305: Test failed: r0: Position 1792 expected 1344
capture.c:330: Test failed: r0: Position 2240 expected 1792

=== w1064v1507 (32 bit report) ===

mmdevapi:
capture.c:282: Test failed: r0: Position 3136 last 896 frames 448

=== w1064v1809 (32 bit report) ===

mmdevapi:
capture.c:282: Test failed: r0: Position 3136 last 896 frames 448

=== w1064_tsign (32 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64 (32 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w11pro64 (32 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w1064v1507 (64 bit report) ===

mmdevapi:
capture.c:282: Test failed: r0: Position 1792 last 896 frames 448

=== w1064v1809 (64 bit report) ===

mmdevapi:
capture.c:282: Test failed: r0: Position 2688 last 896 frames 448

=== w1064_2qxl (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w1064_adm (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w1064_tsign (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64 (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64_en_AE_u8 (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64_ar (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64_ja (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w10pro64_zh_CN (64 bit report) ===

mmdevapi:
capture.c:542: Test failed: nesting attempt blocks is not allowed (4/0)
capture.c:135: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:137: Test failed: r0: Waiting on event handle failed!
capture.c:187: Test failed: r0: GetNextPacketSize 0 vs. GCP 0
capture.c:190: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:259: Test failed: r0: GetNextPacketSize 0 vs. GetDevicePeriod 448
capture.c:272: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:299: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:314: Test failed: r0: Stop on a started stream returns 00000001
capture.c:317: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:320: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001
capture.c:327: Test failed: r0: restarted GCP 0
capture.c:339: Test failed: r0: Stop on a started stream returns 00000001
capture.c:361: Test failed: r0: Start on a stopped stream returns 88890014
capture.c:369: Test failed: r0: Valid IAudioCaptureClient_GetBuffer returns 08890001

=== w7u_2qxl (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w7u_el (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064v1507 (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064v1809 (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064_tsign (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64 (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w11pro64 (32 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064v1507 (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064v1809 (64 bit report) ===

mmdevapi:
render.c:1342: Test failed: GetBuffer large (20671) at iteration 5
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064_2qxl (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064_adm (64 bit report) ===

mmdevapi:
render.c:1342: Test failed: GetBuffer large (22500) at iteration 6
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w1064_tsign (64 bit report) ===

mmdevapi:
render.c:1342: Test failed: GetBuffer large (22500) at iteration 7
render.c:1342: Test failed: GetBuffer large (22500) at iteration 8
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64 (64 bit report) ===

mmdevapi:
render.c:1342: Test failed: GetBuffer large (22500) at iteration 6
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64_en_AE_u8 (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64_ar (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64_ja (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== w10pro64_zh_CN (64 bit report) ===

mmdevapi:
render.c:2415: Test failed: nesting attempt blocks is not allowed (4/0)

=== debian11 (32 bit report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit ar:MA report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit de report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit fr report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit he:IL report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit hi:IN report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit ja:JP report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11 (32 bit zh:CN report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11b (32 bit WoW report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

=== debian11b (64 bit WoW report) ===

ieframe:
webbrowser.c:4466: Test failed: r4: count = 0
webbrowser.c:4476: Test succeeded inside todo block: r0: count = 0

quartz:
filtergraph.c:433: Test failed: r0: Got hr 0x40237.

win32u:
win32u.c:1056: Test failed: res = win32u:win32u:09e8 done (0) in 0s 1337B

[Zebediah Figura (＠zfigura)]

FWIW, the quartz tests are really just a deadlock in Wine. It's solvable but it's not simple to solve. I really just need to be able to take the time to design and fix it properly.