From: Francois Gouget fgouget@codeweavers.com
This is meant to simplify testing conditions that generally hold true but may occasionally fail due to interference from external factors (such as processes that start / stop, network connections being opened / closed, etc).
The trick is loop a few times on the set of flaky conditions until they succeed. During the last attempt all failures are recorded as usual, while in the previous runs, tryok() failures are ignored but cause one more attempt to be run.
The simplest case looks like this:
LOOP_ON_FLAKY_TESTS(3) { // ok() failures are never ignored and not retried ok(..., "check 1", ...); // tryok() failures are ignored except on the last attempt tryok(..., "check 2", ...); }
There is also: * attempt_retry() which indicate a new attempt is needed without counting as a failure, and returns true if a new attempt is possible. * attempt_failed() which returns true if the current attempt failed.
--- This is independent from the 'flaky' mechanism which adds some naming constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS() despite being unrelated to the flaky mechanism. The attempt_retry() and attempt_failed() macro names also don't make it obvious that they are related to tryok().
I think this mechanism is better than the flaky one because a flaky test can go bad without anyone noticing, whereas if a tryok() starts failing systematically it will cause a real failure.
The other side of that coin is that, unlike flaky, the tryok() mechanism does not entirely eliminate the possibility of getting a failure, it just reduces it; though by adjusting the maximum number of attempts one can achieve an arbitrarily low failure rate. For instance if an ok() call fails 10% of the time and one wants a maximum of 1 in a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an increased run time in the worst case.
This also limits the use of this mechanism to tests that have a reasonably low failure rate to start with (otherwise one has to loop too many times). Also note that there are cases where looping essentially reduce the failure rate to zero. For instance ieframe:webbrowser fails if IE creates a net session while the test is counting them. But IE only creates the one net session on start up so trying even one more time should guarantee that the test will succeed. Other cases like scheduling delays and the creation of network connections are more probabilistic in nature. Maybe a comment in test.h should offer some guideline as to the target failure rate.
Eventually this may replace the flaky mechanism but that depends on how well it works in practice and how practical it is to loop on flaky tests. It seems to be going well in the cases I looked at. I think this mechanism has value even if the two end up coexisting indefinitely. --- dlls/kernel32/tests/debugger.c | 2 +- include/wine/test.h | 225 ++++++++++++++++++++++++++++++++- 2 files changed, 220 insertions(+), 7 deletions(-)
diff --git a/dlls/kernel32/tests/debugger.c b/dlls/kernel32/tests/debugger.c index b0c69052936..3ec3133e7a7 100644 --- a/dlls/kernel32/tests/debugger.c +++ b/dlls/kernel32/tests/debugger.c @@ -63,7 +63,7 @@ static void WINAPIV __WINE_PRINTF_ATTR(2, 3) test_child_ok(int condition, const va_list valist;
va_start(valist, msg); - winetest_vok(condition, msg, valist); + winetest_vok(condition, 0, msg, valist); va_end(valist); if (!condition) ++child_failures; } diff --git a/include/wine/test.h b/include/wine/test.h index d531d052383..de38835ff9d 100644 --- a/include/wine/test.h +++ b/include/wine/test.h @@ -80,10 +80,11 @@ extern void winetest_wait_child_process( HANDLE process ); #endif
extern int broken( int condition ); -extern int winetest_vok( int condition, const char *msg, va_list ap ); +extern int winetest_vok( int condition, BOOL trying, const char *msg, va_list ap ); extern void winetest_vskip( const char *msg, va_list ap );
extern void winetest_ok( int condition, const char *msg, ... ) __WINE_PRINTF_ATTR(2,3); +extern void winetest_tryok( int condition, const char *msg, ... ) __WINE_PRINTF_ATTR(2,3); extern void winetest_skip( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2); extern void winetest_win_skip( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2); extern void winetest_trace( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2); @@ -91,27 +92,46 @@ extern void winetest_trace( const char *msg, ... ) __WINE_PRINTF_ATTR(1,2); extern void winetest_push_context( const char *fmt, ... ) __WINE_PRINTF_ATTR(1, 2); extern void winetest_pop_context(void);
+extern void winetest_start_attempt( int retries ); +extern int winetest_loop_attempt( void ); +extern void winetest_end_attempt( void ); +extern BOOL winetest_attempt_retry( BOOL condition ); +extern BOOL winetest_attempt_failed( void ); + #ifdef WINETEST_NO_LINE_NUMBERS # define subtest_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_subtest # define ignore_exceptions_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_ignore_exceptions # define ok_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_ok +# define tryok_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_tryok # define skip_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_skip # define win_skip_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_win_skip # define trace_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_trace # define wait_child_process_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_wait_child_process +# define start_attempt_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_start_attempt +# define loop_attempt_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_loop_attempt +# define end_attempt_(file, line) (winetest_set_location(file, 0), 0) ? (void)0 : winetest_end_attempt +# define attempt_retry_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_attempt_retry +# define attempt_failed_(file, line) (winetest_set_location(file, 0), 0) ? 0 : winetest_attempt_failed #else # define subtest_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_subtest # define ignore_exceptions_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_ignore_exceptions # define ok_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_ok +# define tryok_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_tryok # define skip_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_skip # define win_skip_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_win_skip # define trace_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_trace # define wait_child_process_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_wait_child_process +# define start_attempt_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_start_attempt +# define loop_attempt_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_loop_attempt +# define end_attempt_(file, line) (winetest_set_location(file, line), 0) ? (void)0 : winetest_end_attempt +# define attempt_retry_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_attempt_retry +# define attempt_failed_(file, line) (winetest_set_location(file, line), 0) ? 0 : winetest_attempt_failed #endif
#define subtest subtest_(__FILE__, __LINE__) #define ignore_exceptions ignore_exceptions_(__FILE__, __LINE__) #define ok ok_(__FILE__, __LINE__) +#define tryok tryok_(__FILE__, __LINE__) #define skip skip_(__FILE__, __LINE__) #define win_skip win_skip_(__FILE__, __LINE__) #define trace trace_(__FILE__, __LINE__) @@ -130,6 +150,17 @@ extern void winetest_pop_context(void); #define todo_wine todo_if(!strcmp(winetest_platform, "wine")) #define todo_wine_if(is_todo) todo_if((is_todo) && !strcmp(winetest_platform, "wine"))
+#define start_attempt start_attempt_(__FILE__, __LINE__) +#define loop_attempt loop_attempt_(__FILE__, __LINE__) +#define end_attempt end_attempt_(__FILE__, __LINE__) +#define attempt_retry attempt_retry_(__FILE__, __LINE__) +#define attempt_failed attempt_failed_(__FILE__, __LINE__) +#define LOOP_ON_FLAKY_TESTS(count) for (start_attempt((count) - 1); \ + loop_attempt(); \ + end_attempt()) +#define LOOP_ON_FLAKY_WINDOWS_TESTS(count) LOOP_ON_FLAKY_TESTS(strcmp(winetest_platform, "windows") == 0 ? (count) : 1) +#define LOOP_ON_FLAKY_WINE_TESTS(count) LOOP_ON_FLAKY_TESTS(strcmp(winetest_platform, "windows") == 0 ? 1 : (count)) +
#ifndef ARRAY_SIZE # define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) @@ -202,6 +233,12 @@ static LONG muted_todo_successes; /* same as todo_successes but silent */ /* counts how many times a given line printed a message */ static LONG line_counters[16384];
+#define ATTEMPT_NONE 0 +#define ATTEMPT_STARTED 1 +#define ATTEMPT_RETRY 2 +#define ATTEMPT_BADOK 4 +#define ATTEMPT_FAILED (ATTEMPT_RETRY | ATTEMPT_BADOK) + /* The following data must be kept track of on a per-thread basis */ struct tls_data { @@ -215,6 +252,8 @@ struct tls_data char strings[2000]; /* buffer for debug strings */ char context[8][128]; /* data to print before messages */ unsigned int context_count; /* number of context prefixes */ + int attempt_status; + int attempt_retries; }; static DWORD tls_index;
@@ -229,6 +268,8 @@ static struct tls_data *get_tls_data(void) { data = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof(*data)); data->str_pos = data->strings; + data->attempt_status = ATTEMPT_NONE; + data->attempt_retries = 0; TlsSetValue(tls_index,data); } SetLastError(last_error); @@ -365,12 +406,13 @@ static LONG winetest_add_line( void ) * * Parameters: * - condition - true if the check succeeded, false otherwise; + * - trying - if non-zero the failure may be ignored if in an attempt loop; * - msg - failure message format; * - args - arguments for the failure message * Return: * 0 if condition does not have the expected value, 1 otherwise */ -int winetest_vok( int condition, const char *msg, va_list args ) +int winetest_vok( int condition, BOOL trying, const char *msg, va_list args ) { struct tls_data *data = get_tls_data();
@@ -379,12 +421,23 @@ int winetest_vok( int condition, const char *msg, va_list args ) if (condition) { winetest_print_lock(); - if (data->flaky_level) + if (trying && data->attempt_retries > 0) + { + if (winetest_report_success || + (winetest_time && GetTickCount() >= winetest_last_time + 1000)) + { + winetest_print_context( data->flaky_level ? "Retry successful test inside flaky todo block: " : "Retry successful test inside todo block: "); + vprintf(msg, args); + } + data->attempt_status |= ATTEMPT_RETRY; + } + else if (data->flaky_level) { if (winetest_color) printf( color_dark_purple ); winetest_print_context( "Test succeeded inside flaky todo block: " ); vprintf(msg, args); InterlockedIncrement(&flaky_failures); + data->attempt_status |= ATTEMPT_BADOK; } else { @@ -392,6 +445,7 @@ int winetest_vok( int condition, const char *msg, va_list args ) winetest_print_context( "Test succeeded inside todo block: " ); vprintf(msg, args); InterlockedIncrement(&todo_failures); + data->attempt_status |= ATTEMPT_BADOK; } if (winetest_color) printf( color_reset ); winetest_print_unlock(); @@ -423,12 +477,23 @@ int winetest_vok( int condition, const char *msg, va_list args ) if (!condition) { winetest_print_lock(); - if (data->flaky_level) + if (trying && data->attempt_retries > 0) + { + if (winetest_report_success || + (winetest_time && GetTickCount() >= winetest_last_time + 1000)) + { + winetest_print_context( data->flaky_level ? "Retry flaky test: " : "Retry failed test: "); + vprintf(msg, args); + } + data->attempt_status |= ATTEMPT_RETRY; + } + else if (data->flaky_level) { if (winetest_color) printf( color_bright_purple ); winetest_print_context( "Test marked flaky: " ); vprintf(msg, args); InterlockedIncrement(&flaky_failures); + data->attempt_status |= ATTEMPT_BADOK; } else { @@ -436,6 +501,7 @@ int winetest_vok( int condition, const char *msg, va_list args ) winetest_print_context( "Test failed: " ); vprintf(msg, args); InterlockedIncrement(&failures); + data->attempt_status |= ATTEMPT_BADOK; } if (winetest_color) printf( color_reset ); winetest_print_unlock(); @@ -463,7 +529,7 @@ void winetest_ok( int condition, const char *msg, ... ) va_list valist;
va_start(valist, msg); - winetest_vok(condition, msg, valist); + winetest_vok(condition, FALSE, msg, valist); va_end(valist); }
@@ -529,7 +595,154 @@ void winetest_win_skip( const char *msg, ... ) if (strcmp(winetest_platform, "windows") == 0) winetest_vskip(msg, valist); else - winetest_vok(0, msg, valist); + winetest_vok(0, FALSE, msg, valist); + va_end(valist); +} + +/* Starts a group of tests that has a low but non-zero probability of failing + * due to unavoidable outside interference. This means it may be necessary + * to retry a few times to get a successful run. + * + * Parameters: + * - retries - The number of times this group of tests should be retried if it + * fails. + * + * Remarks: + * - Nesting attempts is not allowed and results in a failure. + * - As a corollary, do not break out of an attempt loop. + * - This is not meant to paper over race conditions within the test itself. + * Those are bugs and should be fixed. + * - The failures should be rare to start with. If a test fails 25% of the time + * or needs to be repeated more than a handful of times to succeed reliably + * it is probably wrong. + * + * Examples: + * + * LOOP_ON_FLAKY_TESTS(3) + * { + * ... + * // ok() failures are never ignored and not retried + * ok(..., "check 1", ...); + * + * // ignore tryok() failures except on the last attempt + * tryok(..., "check 2", ...); + * ... + * } + * + * LOOP_ON_FLAKY_TESTS(5) + * { + * ... + * // if the conditions for running the remaining tests are not met, + * // pass true to indicate this attempt failed and a new one should + * // be made. attempt_retry() will return true if that is possible. + * if (attempt_retry(condition)) + * continue; + * ok(..., "check 1", ...); + * + * ... + * // skip if an ok() or tryok() failed already + * if (attempt_failed()) continue; + * ... + * } + */ +void winetest_start_attempt( int retries ) +{ + struct tls_data *data = get_tls_data(); + + if (data->attempt_status & ATTEMPT_STARTED) + winetest_ok(0, "nesting attempt blocks is not allowed (%x/%d)\n", data->attempt_status, data->attempt_retries); + data->attempt_retries = retries; + data->attempt_status = ATTEMPT_STARTED | ATTEMPT_RETRY; +} + +int winetest_loop_attempt( void ) +{ + struct tls_data *data = get_tls_data(); + + if (!(data->attempt_status & ATTEMPT_FAILED) || /* no failure */ + (data->attempt_status & ATTEMPT_BADOK) || /* unrecoverable failure */ + data->attempt_retries < 0) /* no retries left */ + { + data->attempt_status = ATTEMPT_NONE; + data->attempt_retries = 0; + return FALSE; /* exit loop */ + } + + winetest_push_context("r%d", data->attempt_retries); + data->attempt_status = ATTEMPT_STARTED; + return TRUE; +} + +void winetest_end_attempt( void ) +{ + struct tls_data *data = get_tls_data(); + + winetest_pop_context(); + data->attempt_retries--; +} + +/* Marks the current attempt as failed if condition is true and returns + * whether another attempt is needed and possible. + * + * Parameters: + * - condition - True to indicate that the conditions for running the + * remaining tests are not met and a new attempt should be made. + * This never counts as a test failure. + * + * Return: + * True if the current attempt failed and a new attempt can be made. + * The current attempt is considered to be failed if either condition is + * true or a tryok() call failed already. + * + * Remarks: + * - A new attempt is needed if either condition is true, or a tryok() call + * failed. + * - Plain ok() failures are final so no new attempt is needed after those. + * - This always returns false if not in an attempt loop. + */ +BOOL winetest_attempt_retry( BOOL condition ) +{ + struct tls_data *data = get_tls_data(); + + if (!(data->attempt_status & ATTEMPT_STARTED) || /* not in a retry loop */ + (data->attempt_status & ATTEMPT_BADOK) || /* unrecoverable failure */ + !data->attempt_retries || /* no retries left */ + (!condition && !(data->attempt_status & ATTEMPT_RETRY))) /* no need to retry */ + return FALSE; + data->attempt_status |= ATTEMPT_RETRY; + return TRUE; +} + +/* Returns true if the current attempt failed, whether because of a tryok() + * or a plain ok() call. + * + * Remarks: + * - This always returns false if not in an attempt loop. + */ +BOOL winetest_attempt_failed( void ) +{ + struct tls_data *data = get_tls_data(); + + return (data->attempt_status & ATTEMPT_STARTED) && + (data->attempt_status & ATTEMPT_FAILED); +} + + +/* Performs a check that may need to be repeated to avoid failures caused by + * race conditions. See winetest_start_attempt() for details. + * + * Parameters: see winetest_ok() + * + * Remarks: + * - If not in an attempt loop this behaves the same as a regular ok() call. + */ +void winetest_tryok( int condition, const char *msg, ... ) +{ + struct tls_data *data = get_tls_data(); + va_list valist; + + va_start(valist, msg); + winetest_vok(condition, data->attempt_status & ATTEMPT_STARTED, msg, valist); va_end(valist); }