SOC: D3D Test Suite
It doesn't look like there are many unit tests for D3D in general, but D3D8 and WineD3D don't seem to have any. The ones that D3D9 has also don't test the generated graphics for conformance. Would it be a viable/appropriate SOC project to put together a test suite for D3D, including graphical conformance tests? I should probably note that I have no experience using DirectX and minimal experience using OpenGL. Still, a whole summer should be enough time to put together a reasonably comprehensive test suite (perhaps excluding shaders). Thanks, Sagar Mittal
On 19/04/06, Sagar Mittal <sagar.mittal(a)gmail.com> wrote:
The ones that D3D9 has also don't test the generated graphics for conformance. That's because OpenGL implementations aren't guaranteed to produce identical outputs when given identical inputs.
Graphical conformance tests would probably be usefull, but they can't easily be automated afaik.
H. Verbeet wrote:
On 19/04/06, Sagar Mittal <sagar.mittal(a)gmail.com> wrote:
The ones that D3D9 has also don't test the generated graphics for conformance.
That's because OpenGL implementations aren't guaranteed to produce identical outputs when given identical inputs.
Graphical conformance tests would probably be usefull, but they can't easily be automated afaik.
How is that coming along? Codeweavers supposedly has utility to do just that?! regards, Jakob
On Thu, 2006-04-20 at 02:28 -0400, H. Verbeet wrote:
On 19/04/06, Sagar Mittal <sagar.mittal(a)gmail.com> wrote:
The ones that D3D9 has also don't test the generated graphics for conformance. That's because OpenGL implementations aren't guaranteed to produce identical outputs when given identical inputs.
If they don't haven't we just found a driver bug? Assuming you're testing on cards with the same capabilities (you would have to record what extensions should correspond to what results) I don't see why the output shouldn't be the same. As long as you keep the tests simple so that they don't try to do anything like dynamically adjusting detail based on load, and you do the tests with either the same aliasing/filtering or none, I would think results would have to be the same across cards in order for them to be considered OpenGL compliant.
"The OpenGL specification is not pixel exact. It therefore does not guarantee an exact match between images produced by different GL implementations." That's from the OpenGL specification, Appendix A. Also note that we wouldn't just be comparing different OpenGL implementations with eachother, but different OpenGL implementations with different DirectX implementations.
On Thu, 2006-04-20 at 11:42 -0400, H. Verbeet wrote:
"The OpenGL specification is not pixel exact. It therefore does not guarantee an exact match between images produced by different GL implementations."
That's from the OpenGL specification, Appendix A. Also note that we wouldn't just be comparing different OpenGL implementations with eachother, but different OpenGL implementations with different DirectX implementations.
What about a test that compared image similarity? Output maybe different but only up to a certain point -- obvious cases of breakage like the screen being entirely black are easily detectable, and I'm sure there are some clever algorithms out there for comparing how close images are to each other. How far apart each pixel's color is maybe simple but maybe effective enough. Then mark the test as failed if above a certain high threshold (say 80% dissimilar). The tricky details of animation/timing could simply be avoided by having the tests just draw one frame, which should be enough to detect when something is really out of whack.
This was the approach I intended to take. I'd sum the squares of the differences of each pixel, and check to see if the score is below some threshold. With carefuly chosen tests and some trial-and-error, it should be possible to determine conformance with reasonable accuracy. On 4/20/06, Joseph Garvin <k04jg02(a)kzoo.edu> wrote:
On Thu, 2006-04-20 at 11:42 -0400, H. Verbeet wrote:
"The OpenGL specification is not pixel exact. It therefore does not guarantee an exact match between images produced by different GL implementations."
That's from the OpenGL specification, Appendix A. Also note that we wouldn't just be comparing different OpenGL implementations with eachother, but different OpenGL implementations with different DirectX implementations.
What about a test that compared image similarity? Output maybe different but only up to a certain point -- obvious cases of breakage like the screen being entirely black are easily detectable, and I'm sure there are some clever algorithms out there for comparing how close images are to each other. How far apart each pixel's color is maybe simple but maybe effective enough. Then mark the test as failed if above a certain high threshold (say 80% dissimilar).
The tricky details of animation/timing could simply be avoided by having the tests just draw one frame, which should be enough to detect when something is really out of whack.
Joseph Garvin wrote:
...Assuming you're testing on cards with the same capabilities (you would have to record what extensions should correspond to what results) I don't see why the output shouldn't be the same.
I would expect alias/filtering to be driver-level dependent, with individual pixel outputs varying as optimizations are made. Since you'll never be using "exactly" the same driver in Linux as in Windows (and one that's not even close, if you only run open-source drivers), I think a conformance test would show a lot of inconsequential variances that aren't really things that need to be fixed. You would see similar variances between different Windows machines, depending on the hardware/driver as well, so there's no single "correct" value for which to test. -J
participants (5)
-
Evil Jay -
H. Verbeet -
Jakob Eriksson -
Joseph Garvin -
Sagar Mittal