The false positives rate remained pretty low for the TestBot over the past week, both in terms of impact to the merge requests and in the nightly WineTest results.
Despite having a lot more test configurations, last week only about 6% of MRs got a false positive versus 15% that got one from the GitLab CI's single Wine test environment.
That means less annoying emails from Marvin but also that when the TestBot now reports an issue it's really best to pay attention to it.
Here's an updated list of the test bugs causing the most false positives in the GitLab CI:
18 mmdevapi:capture -> bug 53433 15 ntdll:threadpool -> bug 54064 11 ntdll:pipe -> bug 54078 5 ws2_32:sock -> bug 54074 5 wldap32:parse -> bug 54075 5 user32:msg -> bug 54037 5 ole32:clipboard -> bug 54005 5 mmdevapi:render -> bug 54140
And here's an update to last week's statistics: (I attached the updated data files)
Adjusted MR False Positive rate Week | TestBot | GitLab CI 2022-11-14 | 21.9% | 8.3% 2022-11-21 | 8.0% | 21.6% 2022-11-28 | 14.7% | 28.4% 2022-12-05 | 8.5% | 24.5% 2022-12-12 | 6.1% | 15.2% 2022-12-19 | 0.0% | 15.0% (incomplete week)
Raw MR False Positive rate Week | TestBot | GitLab CI 2022-11-14 | 52.1% | 27.1% 2022-11-21 | 50.0% | 29.5% 2022-11-28 | 20.0% | 33.7% 2022-12-05 | 19.1% | 57.4% 2022-12-12 | 6.1% | 15.2% 2022-12-19 | 0.0% | 15.0% (incomplete week)
Test Units with "new" failures in the nightly WineTest results Week | Total | Deduplicated 2022-11-21 | 96 | 51 2022-11-28 | 82 | 67 2022-12-05 | 108 | 53 2022-12-12 | 47 | 42 2022-12-19 | 14 | 14