On Tue Aug 22 08:53:11 2023 +0000, Stefan Dösinger wrote:
One more option I can think of: Move every test that changes modes to a new file, e.g. d3d9/tests/modeset.c, skip it on gitlab and enable the rest.
I get it. What bothers me is this: suppose we have d3d9 running on CI, and a test breaks for some reason. We have four options:
(a) investigate and try to fix it (b) mark it as todo_wine and move on (c) disable the whole test unit on CI (d) ignore it, let it fail
I feel obliged to at least try to push for (a), especially in a case where we know that a game is going to be broken, but it's pretty clear by now that if we have time for investigating bugs in general, it's limited and can't be depended on. And it's also evident that (c) is seen as preferable to (d), although I don't know (or have forgotten) why this is the case for d3d and not anything else. And evidently (b) is seen as preferable to (c).
The upshot of all of this is that our broken tests are just going to pile up. What's worse is, if the failure is intermittent (as I understand these failures are, but maybe I'm mistaken?), the tests don't just get broken, triggering spurious failures for any developer *not* affected by the bug—they get crippled, which means that at worst the test is completely useless.
And I still have doubts that getting all of the other tests running under CI is worth making making some completely useless. Currently, we can say for sure that that test passes legitimately, even if it's only true of one machine in the world and that makes things worse for everyone. If we cripple the test, that no longer becomes true.
Anyway, with all of that said, I think specifically tagging bugs, as I was thinking of, is a way we can make the tests pass for everyone without crippling them, and it's something I can live with. It still means that bugs are going to pile up, but I guess that's unavoidable, and importantly I think it alleviates my other worries.
But, before all that...
Now re switching to some other WM, last time I proposed this a lot of user32 tests were failing elsewhere because the user32 tests were written to make Alexandre's fvwm2 happy. Not something that makes me happy, but keeping Alexandre's preferred WM in his preferred settings is the path of least resistence.
Mostly what I'm wondering at this point is, can we *upgrade* the WM? If fvwm2 is EOL, and its authors prescribe fvwm3, then we should probably be using fvwm3 *anyway*. Of course, a major version upgrade has its caveats, but it seems at least worth a try.