I'm concerned by some of the recent events regarding WineTest: * A couple of times compilation of the official binaries broke. This got caught within a couple of days which is okayish. * But the winehq.org upgrade completely broke test.winehq.org and it took 10 days for anyone to notice! * Then there's the wininet:ftp, wininet:http & co breakage, also caused by the winehq.org upgrade. This time it took a whopping 24 days for anyone to notice! https://www.winehq.org/pipermail/wine-devel/2019-October/151834.html * There's also TestBot VMs that have almost completely stopped succeeding at posting WineTest results: vista, vistau64*, w8*, w864. I'm probably more to blame for this than anyone else but I would have expected someone to publicly wonder what's going on with these VMs. * Finally there's the gdi32:bitmap not-quite regression. I may have missed something but I did not see it mentioned before. https://www.winehq.org/pipermail/wine-devel/2019-October/152082.html My sinking feeling is: * Nobody looks at test.winehq.org. * Major WineTest regressions can sail through without anyone noticing. My feeling is that nobody feels responsible for making sure test.winehq.org works and for spotting regressions. Hence why I think we may need someone whose responsibility would be to monitor test.winehq.org, analyze the WineTest results, diagnose new issues and report them. I did something like that a few times but it takes quite a bit of time so I had to stop. Essentially it went something like this: * Every couple of weeks, open the results for the latest build: https://test.winehq.org/data/7d954f23356f0aaf49b1ef0c4bed83041cb41c08/ * For each line that has a failure on Windows, open the corresponding test page. For instance: https://test.winehq.org/data/tests/advapi32:service.html * If the tests start failing consistenly on a specific date on at least one platform, dig in to figure out why. For instance: https://test.winehq.org/data/tests/gdi32:bitmap.html https://test.winehq.org/data/tests/wininet:http.html * Likewise, if a test has been consistently getting 13 failures on a platform and now consistently gets 19 it may be because a commit introduced new failures. For instance: https://test.winehq.org/data/tests/advapi32:service.html This requires regular checks otherwise failure is the new normal and there is nothing to see. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ A black hole is just God dividing by zero.