This was the approach I intended to take. I'd sum the squares of the differences of each pixel, and check to see if the score is below some threshold. With carefuly chosen tests and some trial-and-error, it should be possible to determine conformance with reasonable accuracy.
On 4/20/06, Joseph Garvin k04jg02@kzoo.edu wrote:
On Thu, 2006-04-20 at 11:42 -0400, H. Verbeet wrote:
"The OpenGL specification is not pixel exact. It therefore does not guarantee an exact match between images produced by different GL implementations."
That's from the OpenGL specification, Appendix A. Also note that we wouldn't just be comparing different OpenGL implementations with eachother, but different OpenGL implementations with different DirectX implementations.
What about a test that compared image similarity? Output maybe different but only up to a certain point -- obvious cases of breakage like the screen being entirely black are easily detectable, and I'm sure there are some clever algorithms out there for comparing how close images are to each other. How far apart each pixel's color is maybe simple but maybe effective enough. Then mark the test as failed if above a certain high threshold (say 80% dissimilar).
The tricky details of animation/timing could simply be avoided by having the tests just draw one frame, which should be enough to detect when something is really out of whack.