On Mon, Jun 25, 2007 at 03:40:42PM +0200, Stefan Dösinger wrote:
Hi, Now that we're facing Direct3D 9 feature completion soon - VTF support coming in from Henri, High order patches and thread safety from me - the d3d work will change a bit soon. Instead of adding new features we'll debug games and improve performance. It would be good if we had some automated testing framework to support the process :-)
Unfortunately this isn't easy. Games do not render exactly, so verifying that a game doesn't show any visual regressions automatically may be more effort than reward. Here I think we'll have to rely on users testing the games regularly and testing regressions.
cxtest comes to my mind(www.cxtest.org). It is a test suit built by codeweavers for running regression tests on Windows apps. It can remote control applications using keyboard input and compares screenshots to reference ones. It could be used to start benchmarks automatically. A plus here is that some machines run nightly tests. However, those are VMs, so no 3D acceleration. A problem is also reading back the results. Does anyone know some applications for doing that? Windows or Linux apps, commercial or open source ones(prefered, though).
Other suggestions?
cxtest has already impressive infratructure (installation, creation of clean environment) and nice scripting. However, it requires Metacity WM and VNC.
I'm personally using AutoHotkey(http://www.autohotkey.com) as an automatic testing framework backend. It's a Windows application, so it can be run through Wine itself and it's sandboxed from X11.
I can imagine Winebot packages to be easily converted to an automatic testing engine. Winebot actually covers automated installation and creation of clean independent environment.
Running applications with different log levels, tracing output, comparing screenshots with different levels of fuzzy comparaion - it's not faraway.
AFAIK cxtest has the better functionality now.
Regards Vit