This is meant to simplify testing conditions that generally hold true
but may occasionally fail due to interference from external factors
(such as processes that start / stop, network connections being
opened / closed, etc).
The trick is to loop a few times on the set of flaky conditions until
they succeed. During the last attempt all failures are recorded as
usual, while in the previous runs, the tryok() failures area ignored
but cause one more attempt to be made.
The simplest case looks like this:
LOOP_ON_FLAKY_TESTS(3)
{
// ok() failures are never ignored and not retried
ok(..., "check 1", ...);
// tryok() failures are ignored except on the last attempt
tryok(..., "check 2", ...);
}
There is also:
* attempt_retry() which marks the current attempt as failed as if
calling tryok(0), and returns true if another attempt can be made.
* attempt_failed() which returns true if an ok() call failed.
---
This is independent from the 'flaky' mechanism which adds some naming
constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS()
despite being unrelated to the flaky mechanism. The attempt_retry()
and attempt_failed() macro names also don't make it obvious that they
are related to tryok().
I think this mechanism is better than the flaky one because a flaky test
can go bad without anyone noticing, whereas if a tryok() starts failing
systematically it will cause a real failure.
The other side of that coin is that, unlike flaky, the tryok()
mechanism does not entirely eliminate the possibility of getting a
failure, it just reduces it; though by adjusting the maximum number of
attempts one can achieve an arbitrarily low failure rate. For instance
if an ok() call fails 10% of the time and one wants a maximum of 1 in
a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an
increased run time in the worst case.
This also limits the use of this mechanism to tests that have a
reasonably low failure rate to start with (otherwise one has to loop
too many times). Also note that there are cases where looping
essentially reduce the failure rate to zero. For instance
ieframe:webbrowser fails if IE creates a net session while the test is
counting them. But IE only creates the one net session on start up so
trying even one more time should guarantee that the test will succeed.
Other cases like scheduling delays and the creation of network
connections are more probabilistic in nature. Maybe a comment in test.h
should offer some guideline as to the target failure rate.
Eventually this may replace the flaky mechanism but that depends on
how well it works in practice and how practical it is to loop on flaky
tests. It seems to be going well in the few cases I looked at. But I
think this mechanism has value even if the two end up coexisting
indefinitely.
This MR uses the tryok() in some actual tests for illustration and testing purposes. The final MR will probably split most of those off to separate MRs.
--
v5: ieframe/tests: Work around a network session race condition.
advapi32/tests: Replace the custom loop with the tryok() mechanism.
ntdll/tests: Use tryok() to fix a free disk space race with other processes.
kernel32/tests: Use tryok() to fix a heap race with other processes.
tests: Add tryok() for tests that sometimes get outside interference.
tests: Update the documentation.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3418
It's possible that a state object pointer not in the topology node collection gets passed to
session_get_node_object(). Instead of returning the last node when the object is not found, we
should return a NULL so that the state of the last node is not changed by mistake.
--
v2: mf/tests: Test IMFMediaSession::Start().
mf/tests: Add a create_media_session() helper.
mf: Add seeking support for IMFMediaSession::Start().
mf: Add a session_flush_transform_output_nodes() helper.
mf: Make session_get_node_object() more robust.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3572
Signed-off-by: Torge Matthies <openglfreak(a)googlemail.com>
--
v2: loader: Add Default, Failed, and LastKnownGood values to HKLM\System\Select.
server: Create link from HKLM\System\CurrentControlSet to ControlSet001.
advapi32/tests: Add test for CurrentControlSet link.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3563
Windows 10 [received support](https://devblogs.microsoft.com/commandline/af_unix-comes-to-window… for AF_UNIX sockets in Insider Build 17063. This merge request adds basic support for AF_UNIX sockets to ws2_32 and wineserver.
Of particular note is the difficulty in handling `sun_path`. Most of the functions that allow for translating Windows paths to Unix paths are not accessible from ws2_32. I considered the following options:
* Pass the Windows path to wineserver and do the conversion there.
* This is, as far as I can tell, not possible without major rearchitecting. wineserver does not have functions to translate Windows paths to Unix paths, for obvious reasons.
* Obtain the current working directory of the requesting process and temporarily change directories to there.
* This only handles relative paths and fails for absolute paths, UNC paths, etc.
* Conditionally change directories based on whether the path is relative or not.
* This is error-prone and wineserver does not have the requisite functions to do this cleanly.
I ultimately decided to pass the translated Unix path to wineserver, which changes directories to `dirname(path)`. It then provides `bind` and `connect` with `basename(path)`. This is not threadsafe, but wineserver is not (currently) multithreaded.
Abstract sockets are not fully supported by this patch, matching the behavior of Windows.
--
v47: ws2_32/tests: Add test for AF_UNIX sockets fix.
server: Fix getsockname() and accept() on AF_UNIX sockets.
server: Introduce error when attempting to create a SOCK_DGRAM AF_UNIX socket.
server: Allow for deletion of socket files.
ws2_32: Add support for AF_UNIX sockets.
ntdll/unix: Add support for AF_UNIX sockets to multiple functions.
ws2_32: Add afunix.h header.
https://gitlab.winehq.org/wine/wine/-/merge_requests/2786
This is meant to simplify testing conditions that generally hold true
but may occasionally fail due to interference from external factors
(such as processes that start / stop, network connections being
opened / closed, etc).
The trick is to loop a few times on the set of flaky conditions until
they succeed. During the last attempt all failures are recorded as
usual, while in the previous runs, the tryok() failures area ignored
but cause one more attempt to be made.
The simplest case looks like this:
LOOP_ON_FLAKY_TESTS(3)
{
// ok() failures are never ignored and not retried
ok(..., "check 1", ...);
// tryok() failures are ignored except on the last attempt
tryok(..., "check 2", ...);
}
There is also:
* attempt_retry() which marks the current attempt as failed as if
calling tryok(0), and returns true if another attempt can be made.
* attempt_failed() which returns true if an ok() call failed.
---
This is independent from the 'flaky' mechanism which adds some naming
constraints. The loop macro is still called LOOP_ON_FLAKY_TESTS()
despite being unrelated to the flaky mechanism. The attempt_retry()
and attempt_failed() macro names also don't make it obvious that they
are related to tryok().
I think this mechanism is better than the flaky one because a flaky test
can go bad without anyone noticing, whereas if a tryok() starts failing
systematically it will cause a real failure.
The other side of that coin is that, unlike flaky, the tryok()
mechanism does not entirely eliminate the possibility of getting a
failure, it just reduces it; though by adjusting the maximum number of
attempts one can achieve an arbitrarily low failure rate. For instance
if an ok() call fails 10% of the time and one wants a maximum of 1 in
a million failure rate, use LOOP_ON_FLAKY_TESTS(6). The cost is an
increased run time in the worst case.
This also limits the use of this mechanism to tests that have a
reasonably low failure rate to start with (otherwise one has to loop
too many times). Also note that there are cases where looping
essentially reduce the failure rate to zero. For instance
ieframe:webbrowser fails if IE creates a net session while the test is
counting them. But IE only creates the one net session on start up so
trying even one more time should guarantee that the test will succeed.
Other cases like scheduling delays and the creation of network
connections are more probabilistic in nature. Maybe a comment in test.h
should offer some guideline as to the target failure rate.
Eventually this may replace the flaky mechanism but that depends on
how well it works in practice and how practical it is to loop on flaky
tests. It seems to be going well in the few cases I looked at. But I
think this mechanism has value even if the two end up coexisting
indefinitely.
This MR uses the tryok() in some actual tests for illustration and testing purposes. The final MR will probably split most of those off to separate MRs.
--
v4: mmdevapi/tests: Replace flaky with tryok() in the capture tests.
mmdevapi/tests: Replace flaky with tryok() in the render tests.
quartz/tests: Replace flaky() with tryok() to work around scheduling delays.
DEBUG ieframe/tests: tryok() framework testing ground.
ieframe/tests: Work around a network session race condition.
advapi32/tests: Replace the custom loop with tryok() mechanism.
ntdll/tests: Use tryok() to fix a free disk space race with other processes.
https://gitlab.winehq.org/wine/wine/-/merge_requests/3418
It's possible that a state object pointer not in the topology node collection gets passed to
session_get_node_object(). Instead of returning the last node when the object is not found, we
should return a NULL so that the state of the last node is not changed by mistake.
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/3572