On Thu, 2006-04-20 at 11:42 -0400, H. Verbeet wrote:
"The OpenGL specification is not pixel exact. It therefore does not guarantee an exact match between images produced by different GL implementations."
That's from the OpenGL specification, Appendix A. Also note that we wouldn't just be comparing different OpenGL implementations with eachother, but different OpenGL implementations with different DirectX implementations.
What about a test that compared image similarity? Output maybe different but only up to a certain point -- obvious cases of breakage like the screen being entirely black are easily detectable, and I'm sure there are some clever algorithms out there for comparing how close images are to each other. How far apart each pixel's color is maybe simple but maybe effective enough. Then mark the test as failed if above a certain high threshold (say 80% dissimilar).
The tricky details of animation/timing could simply be avoided by having the tests just draw one frame, which should be enough to detect when something is really out of whack.