[PATCH 0/1] MR988: dmloader/tests: Mark tests failing randomly on Windows as flaky.
From: Francois Gouget <fgouget(a)codeweavers.com> Wine-Bug: https://bugs.winehq.org//show_bug.cgi?id=51484 --- dlls/dmloader/tests/loader.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dlls/dmloader/tests/loader.c b/dlls/dmloader/tests/loader.c index 3d68a19c331..fb92bfc8a82 100644 --- a/dlls/dmloader/tests/loader.c +++ b/dlls/dmloader/tests/loader.c @@ -102,8 +102,10 @@ static void test_directory(void) (void**)&loader); ok(hr == S_OK, "Couldn't create Loader %#lx\n", hr); hr = IDirectMusicLoader_SetSearchDirectory(loader, &GUID_DirectMusicAllTypes, empty, 0); + flaky ok(hr == S_OK, "SetSearchDirectory failed with %#lx\n", hr); hr = IDirectMusicLoader_SetSearchDirectory(loader, &GUID_DirectMusicAllTypes, empty, 0); + flaky ok(hr == S_FALSE, "SetSearchDirectory failed with %#lx\n", hr); hr = IDirectMusicLoader_ScanDirectory(loader, &CLSID_DirectMusicContainer, con, NULL); ok(hr == S_FALSE, "ScanDirectory for \"con\" files failed with %#lx\n", hr); -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/988
@fgouget do we need to mark those tests as flaky? The test fails seldom and only on win7u this days https://test.winehq.org/data/patterns.html . And only when running the daily winetest run, never when running the test standalone. No clue what those linux_newtb-debian11-win32 test crashes are but those seem to happen after all the dmloader tests run successfully, the final line is always printed, e.g.: 05a0:loader: 104 tests executed (0 marked as todo, 0 failures), 0 skipped. The gitlab pipeline is not running the tests on Windows so I don't see a need to silence those dmloader tests. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/988#note_9942
* I have not been notified that the tests should no longer be run on Windows 7 so it does not matter whether the test only fails on Windows 7 of Windows 10 21H2. Besides, as noted in the bug, this failure also used to happen on Windows 10 1507. So that's two Windows versions. * The full WineTest suite should succeed, not just the individual tests. * The GitLab CI should run the tests on Windows too. When it does and a patch impacts multiple tests it may opt for running all the tests in one go at which time this failure may well happen and cause false positives. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/988#note_10009
Cool that it is fixed now on Win 10 1507. Do you know what changed on / for those VMs? Or when that stopped being flaky? And can that be applied to the Win7u VMs too? Come to think of it: there are 7 different flavours of newtb-win7u but the test is flaky only on 3 of them. That's very puzzling. Anyway I know why we mark tests as flaky. But in this case it is a big hammer for no benefit. That area of the code needs more work, most likely a rewrite and I would hate to silence some tests and slip a regression through the CI. While Win7 is still "supported" for tests it has no priority to be added to the CI; I've watched Alexandre's gitlab roadmap presentation. Iff those seldom win7u test failures are a problem for any CI I am willing to accept a patch that adds a: broken(hr == DMUS_E_LOADER_BADPATH) /* Flaky on some win7 */ With broken() being superior to flaky as it applies only to Windows results and not to Linux where the test is rock solid. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/988#note_10042
It seems like there are 3 Windows 10 1507 machines that had this issue: * Based on the bug one was cw-gtx560-1507 (bare metal, NVIDIA GTX560 GPU). I don't see any trace of this in my archives (which go back to 2022-04-26). In any case this machine was decommissioned on 2022-05-31 so it cannot happen there anymore. * In my archives I see this happening on cw-rx460-1507 (identical bare metal, AMD RX460 GPU), until a few days before it was decommissioned (the case and PSU were reused to build gpu1). Again that explains why it does not happen anymore. * And this also happened on newtb-w1064v1507-32. That snapshot was created on 2022-04-24 and the failure happened on the following dates: 2022-05-18, 2022-05-24 and 2022-06-16. So there is nothing to explain why it has not happened since. Maybe the failure is caused by a previous test that changed, or there's some subtle timing aspect that changed. Given that there could already almost be a month between two occurrences it's also hard to pinpoint a date when the issue 'stopped'.
While Win7 is still "supported" for tests it has no priority to be added to the CI; I've watched Alexandre's gitlab roadmap presentation.
Right. And yet nobody is responsible for deciding which configurations to no longer test on.
With broken() being superior to flaky as it applies only to Windows results
The proposal to have a flaky_windows() macro was rejected. But one can still achieve the same result with flaky_if(strcmp(winetest_platform, "windows") == 0) (but I give no guarantee that it would be accepted).
That area of the code needs more work, most likely a rewrite
Would you be willing to take on the cleanup of that test code? -- https://gitlab.winehq.org/wine/wine/-/merge_requests/988#note_10737
This merge request was closed by Francois Gouget. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/988
participants (3)
-
Francois Gouget -
Francois Gouget (＠fgouget)
-
Michael Stefaniuc (＠mstefani)