Francois Gouget fgouget@codeweavers.com writes:
- The other policy changes are still to be determined.
Thank you for raising the issue. I understand your frustration that after all these years we still can't achieve zero failures, but I'm not envisioning drastic policy changes at this point, particularly not extra constraints on developers.
We all know that fixing tests can be hard even for the best developers, so punishing people for not fixing tests, by reverting their changes or blocking their MRs, would be counterproductive IMHO.
There's a balance to strike between the efforts needed to fix the tests, and the efforts needed to develop new features and fix bugs in real apps. A successful test suite is important, but it's not important enough to sacrifice progress in other areas.
We also have to look at the positive side: we have millions of tests that are being executed and succeed on every commit, and they are catching many potential regressions. While the failures are annoying, they are only a tiny percentage of the tests, and they don't make the whole test suite useless. That's also why I don't want to disable an entire test file when only a couple of tests fail. However, it's OK to disable specific tests, using flaky() or todo(), if they are too hard to fix properly.
So I think we have to accept that test failures and regressions are the price to pay for making progress. We all need to do our best to minimize them and keep them under control, but given the complexity of what we are doing with Wine, we can't always achieve perfection. I'm also hoping that, as the pool of paid developers grows, we will have more room to allocate some paid developer time to fixing the tests.