http://bugs.winehq.org/show_bug.cgi?id=23158
--- Comment #4 from Rosanne DiMesio dimesio@earthlink.net 2010-08-04 11:07:27 --- (In reply to comment #3)
I totally agree that the false Platinum ratings are a big problem. However, we could check for applications that have been reduced from a rating >= Bronze to Garbage.
That would miss a lot of real regressions that don't render an app completely unusable. It would also produce false positives for problems that are due to some other change in the user's system, such as upgrading to the kernel that killed World of Warcraft.
I'm not arguing against doing it, mind you; I just want to point out that no matter how you do it, the results of any such search will always have a large margin of error. As to whether "good enough" accuracy can be achieved, I suppose that depends on your purpose.
To combat the problem of false ratings, maybe we should let maintainers create a per-version checklist that is used when submitting test results. So in addition to the normal test results fields, a version could have a checkmarks saying 'Multiplayer tested', 'Printing tested' etc. This way we could automatically reduce the rating if something has not been tested and is known not to work.
Possibly, but that wouldn't help with unmaintained apps or apps with less-conscientious maintainers, and for complex apps like Word or Photoshop, the checklist could get unbearably long.
One idea I had would be to have the submission system check the version's bug links for open, confirmed bugs, and not allow a platinum rating if there are any affecting the Wine version tested. But I don't know how difficult that would be to implement.