On Wed, Feb 11, 2009 at 1:58 PM, Juan Lang juan.lang@gmail.com wrote:
The front page of test.winehq.org shows statistics about failed tests, but it doesn't seem to take into account the number of individual tests that passed and failed, rather the number of files that had any failures.
So, for example, about a week ago I got a fix committed for some failing mapi32 tests. Looking at the machines with test failures, before the fix was committed, 139 tests were run, with 134 of them failing, whereas after the fix was committed, the same number of tests were run, with only 6 of them failing. Nonetheless, the 4th of February shows a higher failure rate (14.6%) than the 3rd of February (12.4%).
I know other tests could have started failing in the interim, but it seems like we've been putting a fair amount of effort into reducing test failures lately, while the percent of failed tests isn't going down, at least not on the main page. If you look at a particular day's results, the numbers look a bit better over time.
I'm not sending a patch, because there may be different opinions on this. That is, perhaps some people like to see a statistic on the number of files with failing tests on any machine, which the front page appears to show, while others may like to see the number of failures in a particular file, which a day's results show. My own opinion is that it's hard to get motivated to fix something without some sort of positive feedback for it, so changing the front page would be better.
My own feeling is that there are far fewer failing tests now than there used to be, and I'd sure like to see that reflected somewhere at a quick glance. Thoughts?
We should leave the failing files percentage up (note the name change) and add a failing tests percentage next to it. The failing tests percentage should be total_test_failures / total_tests_run.