I pushed a merge request with the latest version so you can also comment there: https://gitlab.winehq.org/wine/wine/-/merge_requests/764
See also my flaky-tests branch for some use cases: https://gitlab.winehq.org/fgouget/wine/-/commits/flaky-tests/
On to the next question: do we need to take special steps to make sure one cannot just (sneakily?) add bad tests by prefixing them with flaky*?
One option is to do nothing and let reviewers deal with it. Certainly if they spot a flaky addition they would scrutinize the patch and ask questions as needed. And if bad flaky calls slip through, the developer's trustiness would be adjusted accordingly.
But I think we could have a CI check that requires specific explanations for commits with flaky additions:
- If there are more '^+.*flaky' lines in a commit than there are '^-.*flaky' ones, then that commit has a flaky addition. - If so the CI check would require a 'Flaky-Tests' line in the commit message where the submitter explains why the additional flaky is needed.
I mostly see a CI check as having three benefits: - Reduce the risk of reviewers missing a flaky addition. - Save a step for the reviewers by providing them with an explanation of why the flaky is needed from the get go. - Raise the inconvenience of using flaky so in most cases it's easier / less embarrassing to fix the test.
Flaky tests are really buggy tests. So we could also require a Wine-Bug line in addition to the Flaky-Tests one for tracking progress on fixing that bug.
That is unless Wine-Bug lines are only supposed to be used when fixing a bug. Another approach would be to put the bug URL on the Flaky-Tests line and leave the explanation in the bug. But that makes the commit message less directly useful and not self-sufficient.
Also that line could be called Flaky-Test instead of Flaky-Tests. Depends if it's about the test unit or the individual ok() calls.