On Fri, Sep 02, 2022 at 12:41:42PM +0200, Francois Gouget wrote:
On Thu, 1 Sep 2022, Zebediah Figura wrote: [...]
Why would we allow (intentionally) adding any flaky tests at all? My interpretation of "flaky" is that anything that currently has it is a bug and should be fixed.
Initially we will need to add a bunch to deal with the current ~170 test units (~20%) that fail on a regular basis. But yes, beyond that we may ban adding more flaky tests entirely. In that case it would still be useful to identify commits that add flaky tests but we would not need the Flaky-Test mechanism.
And should we need to add a flaky call anyway (maybe because a test is restructured so that a single flaky call is split in two) then maybe the author can ask Alexandre to manually cherry-pick from a GitLab repository (or such cases fall back to email).
The issue is that we don't know that the tests are flakey when they get committed; that only becomes apparent at some later stage.
Huw.