Re: A case for policy changes to get to zero test failures

[Francois Gouget]

On Mon, 23 Oct 2023, Francois Gouget wrote:
[...]
> So the conclusion I have come to is that making further and lasting 
> progress will require policy changes to incentivize work on the tests.

* The goal is to get rid of the TestBot so the bridge to GitLab will not 
  be restored in any form. There will also be no further TestBot 
  development.

* Wine developers are on their own to figure out if their commits cause 
  new failures, and if they do they are on their own to figure out how 
  to reproduce the issues.

* The other policy changes are still to be determined.


-- 
Francois Gouget <fgouget(a)codeweavers.com>

A case for policy changes to get to zero test failures

[Alexandre Julliard]

Francois Gouget <fgouget(a)codeweavers.com> writes:
> * The other policy changes are still to be determined.

Thank you for raising the issue. I understand your frustration that
after all these years we still can't achieve zero failures, but I'm not
envisioning drastic policy changes at this point, particularly not extra
constraints on developers.

We all know that fixing tests can be hard even for the best developers,
so punishing people for not fixing tests, by reverting their changes or
blocking their MRs, would be counterproductive IMHO.

There's a balance to strike between the efforts needed to fix the tests,
and the efforts needed to develop new features and fix bugs in real
apps. A successful test suite is important, but it's not important
enough to sacrifice progress in other areas.

We also have to look at the positive side: we have millions of tests
that are being executed and succeed on every commit, and they are
catching many potential regressions. While the failures are annoying,
they are only a tiny percentage of the tests, and they don't make the
whole test suite useless. That's also why I don't want to disable an
entire test file when only a couple of tests fail. However, it's OK to
disable specific tests, using flaky() or todo(), if they are too hard to
fix properly.

So I think we have to accept that test failures and regressions are the
price to pay for making progress. We all need to do our best to minimize
them and keep them under control, but given the complexity of what we
are doing with Wine, we can't always achieve perfection. I'm also hoping
that, as the pool of paid developers grows, we will have more room to
allocate some paid developer time to fixing the tests.

-- 
Alexandre Julliard
julliard(a)winehq.org