Hi,
Now that we're facing Direct3D 9 feature completion soon - VTF support coming
in from Henri, High order patches and thread safety from me - the d3d work
will change a bit soon. Instead of adding new features we'll debug games and
improve performance. It would be good if we had some automated testing
framework to support the process :-)
Unfortunately this isn't easy. Games do not render exactly, so verifying that
a game doesn't show any visual regressions automatically may be more effort
than reward. Here I think we'll have to rely on users testing the games
regularly and testing regressions.
I am more concerned about the performance thing. It is very easy to make some
performance improvement that does more harm than good. Also benchmarks take
quite some time and are affected by many external parameters. So I'd like
some automated and controlled performance testing environment. Instead of
building one from scratch it would be cool if we could reuse existing
software.
cxtest comes to my mind(www.cxtest.org). It is a test suit built by
codeweavers for running regression tests on Windows apps. It can remote
control applications using keyboard input and compares screenshots to
reference ones. It could be used to start benchmarks automatically. A plus
here is that some machines run nightly tests. However, those are VMs, so no
3D acceleration. A problem is also reading back the results. Does anyone know
some applications for doing that? Windows or Linux apps, commercial or open
source ones(prefered, though).
Then we need benchmarking apps. Here regular Windows benchmarks come to my
mind(3DMarkXYZW), games with benchmark facilities(hl2, ut, ...). We can
record tests, get a reference performance counter from windows and run
nightly tests(or manually started tests). A problem we're facing is reading
back the results. Taking screenshots and running OCR on them is error prone,
we need a way to get the numbers from the benchmarks, not an image of the
numbers.
Now we need a concrete collection of benchmarks. A mixture of popular
Benchmarks like 3DMark would be good, and some more cientific benchmarks. So
far I have:
3DMark:
Popular benchmark for testing the overall performance, doesn't give very
detailed results though.
Controlling: Standard windows controls, keyboard events should do the job
Result readback: Can write results to a file, but must take care about the
licensing. Not everything is free, most likely a license is needed.
Half-Life 2:
Nice facility of recordable timedemos. Can record average game demos, but also
demos which stress specific features.
Controlling: Command line parameters
Result readback: Can be asked to copy the console into the clipboard
UT2004:
Popular engine. Has a Linux port, but many mods only ship the d3d renderer.
Controlling: Unknown
Result readback: unknown.
DirectX sdk demos, nvidia / ati demos:
Stress specific rendering features. Good for testing performance(with vsync
removed), or a bad idea?
Controlling, Readback: Source code is available and can be modified. License,
especially possible benchmark clauses?
Other suggestions?
Another nice-to-have thing would be a game independent profiling method,
simmilar to the +fps debug channel, just more detailed. Some games do not
have built-in benchmarking, it would still be nice to use them. That means
facing some new problems? How to tell loading from in-game rendering? How to
remove control those games in a repeatable way, especially if there are some
random elements in the game? More detailed data from wined3d could also be
used to draw nice graphs for other benchmarks :-)
We'll need some hardware diversity too. It would be nice if the tests worked
on Linux and MacOS, and a variety of hardware was used. I want at least
nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI
mach64 :-)
Any other comments or suggestions? I think once I'm done with the current exam
season and my patch backlog is in I will retire from direct wined3d
development for a few days / weeks to set up a testing environment, then
start fixing games.