Hi, Now that we're facing Direct3D 9 feature completion soon - VTF support coming in from Henri, High order patches and thread safety from me - the d3d work will change a bit soon. Instead of adding new features we'll debug games and improve performance. It would be good if we had some automated testing framework to support the process :-)
Unfortunately this isn't easy. Games do not render exactly, so verifying that a game doesn't show any visual regressions automatically may be more effort than reward. Here I think we'll have to rely on users testing the games regularly and testing regressions.
I am more concerned about the performance thing. It is very easy to make some performance improvement that does more harm than good. Also benchmarks take quite some time and are affected by many external parameters. So I'd like some automated and controlled performance testing environment. Instead of building one from scratch it would be cool if we could reuse existing software.
cxtest comes to my mind(www.cxtest.org). It is a test suit built by codeweavers for running regression tests on Windows apps. It can remote control applications using keyboard input and compares screenshots to reference ones. It could be used to start benchmarks automatically. A plus here is that some machines run nightly tests. However, those are VMs, so no 3D acceleration. A problem is also reading back the results. Does anyone know some applications for doing that? Windows or Linux apps, commercial or open source ones(prefered, though).
Then we need benchmarking apps. Here regular Windows benchmarks come to my mind(3DMarkXYZW), games with benchmark facilities(hl2, ut, ...). We can record tests, get a reference performance counter from windows and run nightly tests(or manually started tests). A problem we're facing is reading back the results. Taking screenshots and running OCR on them is error prone, we need a way to get the numbers from the benchmarks, not an image of the numbers.
Now we need a concrete collection of benchmarks. A mixture of popular Benchmarks like 3DMark would be good, and some more cientific benchmarks. So far I have:
3DMark: Popular benchmark for testing the overall performance, doesn't give very detailed results though. Controlling: Standard windows controls, keyboard events should do the job Result readback: Can write results to a file, but must take care about the licensing. Not everything is free, most likely a license is needed.
Half-Life 2: Nice facility of recordable timedemos. Can record average game demos, but also demos which stress specific features. Controlling: Command line parameters Result readback: Can be asked to copy the console into the clipboard
UT2004: Popular engine. Has a Linux port, but many mods only ship the d3d renderer. Controlling: Unknown Result readback: unknown.
DirectX sdk demos, nvidia / ati demos: Stress specific rendering features. Good for testing performance(with vsync removed), or a bad idea? Controlling, Readback: Source code is available and can be modified. License, especially possible benchmark clauses?
Other suggestions?
Another nice-to-have thing would be a game independent profiling method, simmilar to the +fps debug channel, just more detailed. Some games do not have built-in benchmarking, it would still be nice to use them. That means facing some new problems? How to tell loading from in-game rendering? How to remove control those games in a repeatable way, especially if there are some random elements in the game? More detailed data from wined3d could also be used to draw nice graphs for other benchmarks :-)
We'll need some hardware diversity too. It would be nice if the tests worked on Linux and MacOS, and a variety of hardware was used. I want at least nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI mach64 :-)
Any other comments or suggestions? I think once I'm done with the current exam season and my patch backlog is in I will retire from direct wined3d development for a few days / weeks to set up a testing environment, then start fixing games.