This is meant to simplify testing conditions that generally hold true
but may occasionally fail due to interference from external factors
(such as processes that start / stop, network connections being
opened / closed, etc).
The trick is loop a few times on the set of flaky conditions until
they succeed. During the last attempt all failures are recorded as
usual, while in the previous runs, tryok() failures are ignored but
cause one more attempt to be run.
The simplest case looks like this:
LOOP_ON_FLAKY_TESTS(3)
{
// ok() failures are never ignored and not retried
ok(..., "check 1", ...);
// tryok() failures are ignored except on the last attempt
tryok(..., "check 2", ...);
}
There is also:
* attempt_retry() which indicate a new attempt is needed without
counting as a failure, and returns true if a new attempt is
possible.
* attempt_failed() which returns true if the current attempt failed.
---
This is independent from the 'flaky' mechanism which adds some naming
constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS()
despite being unrelated to the flaky mechanism. The attempt_retry()
and attempt_failed() macro names also don't make it obvious that they
are related to tryok().
I think this mechanism is better than the flaky one because a flaky test
can go bad without anyone noticing, whereas if a tryok() starts failing
systematically it will cause a real failure.
The other side of that coin is that, unlike flaky, the tryok()
mechanism does not entirely eliminate the possibility of getting a
failure, it just reduces it; though by adjusting the maximum number of
attempts one can achieve an arbitrarily low failure rate. For instance
if an ok() call fails 10% of the time and one wants a maximum of 1 in
a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an
increased run time in the worst case.
This also limits the use of this mechanism to tests that have a
reasonably low failure rate to start with (otherwise one has to loop
too many times). Also note that there are cases where looping
essentially reduce the failure rate to zero. For instance
ieframe:webbrowser fails if IE creates a net session while the test is
counting them. But IE only creates the one net session on start up so
trying even one more time should guarantee that the test will succeed.
Other cases like scheduling delays and the creation of network
connections are more probabilistic in nature. Maybe a comment in test.h
should offer some guideline as to the target failure rate.
Eventually this may replace the flaky mechanism but that depends on
how well it works in practice and how practical it is to loop on flaky
tests. It seems to be going well in the cases I looked at. I think this
mechanism has value even if the two end up coexisting indefinitely.
In addition to introducing the framework, this MR patches some tests to use uses the tryok() mechanism for illustration and testing purposes. There are more places where this mechanism could be used but I think it's best to see about those later to keep this MR simple and focused on the new mechanism.
--
v6: advapi32/tests: Replace the custom loop with the tryok() mechanism.
ntdll/tests: Use tryok() to fix a free disk space race with other processes.
kernel32/tests: Use tryok() to fix a heap race with other processes.
tests: Add tryok() for tests that sometimes get outside interference.
tests: Update the documentation.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3418
---
Note that this code never runs in d3d10+ for a 0 pixelformat.
Dxgi/tests/dxgi.c:2032 has a test that shows it results in an error.
--
v2: dxgi/tests: Test swapchains with zero dimensions.
wined3d: Move zero swapchain parameter fixup to wined3d_swapchain_state_init.
wined3d: Make wined3d_swapchain_desc in wined3d_swapchain_create const.
dxgi: Read back the swapchain size assigned by wined3d.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3538
When the path contains a mountpoint on Unix or a junction point to another drive on Windows, cmd.exe should show free space for the path instead of the root of the drive.
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/3610
On Fri Aug 18 02:25:13 2023 +0000, Zhiyi Zhang wrote:
> Doesn't the current MR contain only mf changes?
Sure, my comment was for initial patchset. I probably finalized comment/review too late, making this confusing.
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/3572#note_42739