On Sat, 5 Oct 2019, Sven Baars wrote: [...]
so it must be due to some testbot change. I was actually under the assumption that you were checking the test log after testbot changes to see if the changes caused any failures, but I guess not.
That depends on the type of change. Most of the TestBot changes (commits to tool/testbot) have no impact on the test results. Only the VM changes would make a difference. So I normally check the results after changing the VMs. But it's not actually very practical because when VMs already have a significant number of failures, some of which are random, it's hard to say if a new run is really better or worse :-(
[...]
Since I already look at it quite often, I can just file bug reports when I see consistent new failures if you like.
Yes, I think that could be useful. In particular if the new failures are caused by changes in the tests it's useful to identify the bad commit, or at least the range of bad commits.
If the new failure is not caused by a change in the tests and there's no other obvious explanation, then maybe it's better to ask what's up on wine-dev.