Direct3D Game test framework

[Stefan Dösinger]

Hi,
Now that we're facing Direct3D 9 feature completion soon - VTF support coming 
in from Henri, High order patches and thread safety from me - the d3d work 
will change a bit soon. Instead of adding new features we'll debug games and 
improve performance. It would be good if we had some automated testing 
framework to support the process :-)

Unfortunately this isn't easy. Games do not render exactly, so verifying that 
a game doesn't show any visual regressions automatically may be more effort 
than reward. Here I think we'll have to rely on users testing the games 
regularly and testing regressions.

I am more concerned about the performance thing. It is very easy to make some 
performance improvement that does more harm than good. Also benchmarks take 
quite some time and are affected by many external parameters. So I'd like 
some automated and controlled performance testing environment. Instead of 
building one from scratch it would be cool if we could reuse existing 
software.

cxtest comes to my mind(www.cxtest.org). It is a test suit built by 
codeweavers for running regression tests on Windows apps. It can remote 
control applications using keyboard input and compares screenshots to 
reference ones. It could be used to start benchmarks automatically. A plus 
here is that some machines run nightly tests. However, those are VMs, so no 
3D acceleration. A problem is also reading back the results. Does anyone know 
some applications for doing that? Windows or Linux apps, commercial or open 
source ones(prefered, though).

Then we need benchmarking apps. Here regular Windows benchmarks come to my 
mind(3DMarkXYZW), games with benchmark facilities(hl2, ut, ...). We can 
record tests, get a reference performance counter from windows and run 
nightly tests(or manually started tests). A problem we're facing is reading 
back the results. Taking screenshots and running OCR on them is error prone, 
we need a way to get the numbers from the benchmarks, not an image of the 
numbers.

Now we need a concrete collection of benchmarks. A mixture of popular 
Benchmarks like 3DMark would be good, and some more cientific benchmarks. So 
far I have:

3DMark:
Popular benchmark for testing the overall performance, doesn't give very 
detailed results though.
Controlling: Standard windows controls, keyboard events should do the job
Result readback: Can write results to a file, but must take care about the 
licensing. Not everything is free, most likely a license is needed.

Half-Life 2:
Nice facility of recordable timedemos. Can record average game demos, but also 
demos which stress specific features.
Controlling: Command line parameters
Result readback: Can be asked to copy the console into the clipboard

UT2004:
Popular engine. Has a Linux port, but many mods only ship the d3d renderer.
Controlling: Unknown
Result readback: unknown.

DirectX sdk demos, nvidia / ati demos:
Stress specific rendering features. Good for testing performance(with vsync 
removed), or a bad idea?
Controlling, Readback: Source code is available and can be modified. License, 
especially possible benchmark clauses?

Other suggestions?

Another nice-to-have thing would be a game independent profiling method, 
simmilar to the +fps debug channel, just more detailed. Some games do not 
have built-in benchmarking, it would still be nice to use them. That means 
facing some new problems? How to tell loading from in-game rendering? How to 
remove control those games in a repeatable way, especially if there are some 
random elements in the game? More detailed data from wined3d could also be 
used to draw nice graphs for other benchmarks :-)

We'll need some hardware diversity too. It would be nice if the tests worked 
on Linux and MacOS, and a variety of hardware was used. I want at least 
nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI 
mach64 :-)

Any other comments or suggestions? I think once I'm done with the current exam 
season and my patch backlog is in I will retire from direct wined3d 
development for a few days / weeks to set up a testing environment, then 
start fixing games.

[H. Verbeet]

On 25/06/07, Stefan Dösinger  wrote:
> 3DMark:
> Popular benchmark for testing the overall performance, doesn't give very
> detailed results though.
> Controlling: Standard windows controls, keyboard events should do the job
> Result readback: Can write results to a file, but must take care about the
> licensing. Not everything is free, most likely a license is needed.

Afaik 3DMark can also generate performance graphs.
> Other suggestions?

We could always try writing something ourselves, I guess :-)
The problem there is going to be creating something that's
representative though.
> random elements in the game? More detailed data from wined3d could also be
> used to draw nice graphs for other benchmarks :-)

NVPerfSDK could potentially help there, if the license permits.
> We'll need some hardware diversity too. It would be nice if the tests worked
> on Linux and MacOS, and a variety of hardware was used. I want at least
> nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI
> mach64 :-)
>
> Any other comments or suggestions? I think once I'm done with the current exam
> season and my patch backlog is in I will retire from direct wined3d
> development for a few days / weeks to set up a testing environment, then
> start fixing games.

One issue I see is actually interpreting the results. When is a
performance drop large enough to be a problem? Sometimes a change will
slightly reduce performance for some applications, but significantly
improve it for others.

[Stefan Dösinger]

> Afaik 3DMark can also generate performance graphs.

I prefer to have the raw data myself, to do my own processing, so I'm not sure 
if that helps us. But graphcs are just a nice to have, we don't need them.
> We could always try writing something ourselves, I guess :-)
> The problem there is going to be creating something that's
> representative though.

Ya, I don't think our own tests will help us much.
> > random elements in the game? More detailed data from wined3d could also
> > be used to draw nice graphs for other benchmarks :-)
>
> NVPerfSDK could potentially help there, if the license permits.

I'm concerned about the NV in NVPerfSDK :-) Of course its a tool we can use, 
but we shouldn't depend on it.
> One issue I see is actually interpreting the results. When is a
> performance drop large enough to be a problem? Sometimes a change will
> slightly reduce performance for some applications, but significantly
> improve it for others.

Thats a good point, and I've been thinking about it a bit. All we can use is 
some heuristics. Throwing a warning on a single 0.01 fps drop is an overkill 
because there are minor differences between runs, even with exactly the same 
code. On the other hand, if a number of 0.5 fps drops sum up to a drop from 
80 to 75 fps it is something that should perhaps be investigated, so we 
should check against a fixed reference value rather than the last day's 
result.

Wether a performance drop is a real regression or an unavoidable side effect 
from something else, like a rendering correctness fix is something we have to 
decide per incident. The automated tests can only generate a warning(e.g. per 
mail). Then we have to look at the warning and decide what to do with it. We 
could either fix the regression, if it is one, or adjust the reference value.

[H. Verbeet]

On 25/06/07, Stefan Dösinger  wrote:
> > Afaik 3DMark can also generate performance graphs.
>
> I prefer to have the raw data myself, to do my own processing, so I'm not sure
> if that helps us. But graphcs are just a nice to have, we don't need them.

I think it has the ability to export frame timings to Excel format.
Never used that myself though, and I've got no idea if it even works
in Wine.

[Francois Gouget]

On Mon, 25 Jun 2007, Stefan Dösinger wrote:
[...]
> > One issue I see is actually interpreting the results. When is a
> > performance drop large enough to be a problem? Sometimes a change will
> > slightly reduce performance for some applications, but significantly
> > improve it for others.
>
> Thats a good point, and I've been thinking about it a bit. All we can use is 
> some heuristics. Throwing a warning on a single 0.01 fps drop is an overkill 
> because there are minor differences between runs, even with exactly the same 
> code. On the other hand, if a number of 0.5 fps drops sum up to a drop from 
> 80 to 75 fps it is something that should perhaps be investigated, so we 
> should check against a fixed reference value rather than the last day's 
> result.

I don't think we can automatically flag performance drops as 
regressions / errors.

First you would have to compare the performance of the current run 
against that of one of the previous runs. This runs counter to the norm 
which is that each run is self contained and completely independent of 
the other runs.

Second, there is no garantee that all users running the nightly tests 
will have an otherwise idle machine. For instance I run cxtests every 
night but on some days my machine may still be finishing a backup, or 
recording a movie, or compiling wine with gcc 2.95, etc. Each of those 
could cause significant performance variations that could be mistaken 
for a performance bug.

Also, if you only compare one run to the next you will only detect a 
performance drop on one day, which can easily be dismissed as noise due 
to the above, and everything will be normal from then on. So you have to 
do a more long term analysis.

So I think a more promising approach is to just collect data and 
generate graphs. Then we can have human beings look at the graphs and 
investigate when something looks weird, like if the performance on 90% 
of the machines drops around the same date.

You could also write some software to automatically analyze this data on 
the server but that seems overkill.

[Stefan Dösinger]

Hi,
With Francois' help I could build a test package for 3DMark2000 now:

http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2

It runs two tests, one at 640x480, another one at 1024x768, writes the results 
to a file and reads the file back. No benchmark results are sent to the 
server yet(just ok / fail).

Steps to run the test:
1) install cxtest from http://www.cxtest.org/download . I recommend installing 
it with a seperate user account, since the tests scripts mess with files in 
$HOME. cxtest will download the wine git tree and compile wine, so you need 
some free space. cxtest installs some sample test packages too(picasa, 
winzip, pptviewer).

It was a bit tricky for me to set cxtest up, since it needs Xvnc, has some 
problems(non-fatal it seems) with git 1.5, and a problem with the nvidia 
driver interfered(I couldn't run 2 Wine instances from different user 
accounts, I got a crash in libglcore). When you try to *record*, not play 
back a test, make sure to have $DISPLAY unset(vnc is used)

2) Download my test package and unpack it in ~/cxtest/

3) Run the test WITHOUT using vnc. Obviously you need $DISPLAY set, and 3D 
acceleration available:

~/cxtest/cxtest_run/bin/run.sh --run 
3dmark2000 --verbose --no-user --no-dependency  --wine 
~/cxtest/wine/wine --no-vnc

So whats left?
*) Make the Email system send custom test return data, not just success / 
failure
*) extend the server side infrastructure to handle that test data.
*) Write tests for more games

[Stefan Dösinger]

Am Donnerstag, 28. Juni 2007 18:05 schrieb Stefan Dösinger:
> So whats left?
> *) Make the Email system send custom test return data, not just success /
> failure

I found a simple solution for that: I just dump the results onto the log 
channel, which is sent to the server which can then grep the numbers out.
> *) extend the server side infrastructure to handle that test data.

Any volunteers? I tried to avoid web development as good as possible and I'm 
pretty crappy at it :-/
> *) Write tests for more games

Will write more :-)

[martin pilka]

> With Francois' help I could build a test package for 3DMark2000 now:
>
> http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
>
> It runs two tests, one at 640x480, another one at 1024x768, writes the results 
> to a file and reads the file back. No benchmark results are sent to the 
> server yet(just ok / fail).

Stefan, this link is not valid anymore, could you make the file available
again?

Thanks,
Martin

[Stefan Dösinger]

Am Dienstag, 31. Juli 2007 12:28 schrieb martin pilka:
> > With Francois' help I could build a test package for 3DMark2000 now:
> >
> > http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
> >
> > It runs two tests, one at 640x480, another one at 1024x768, writes the
> > results to a file and reads the file back. No benchmark results are sent
> > to the server yet(just ok / fail).
>
> Stefan, this link is not valid anymore, could you make the file available
> again?

Oops, I deleted the file accidentally when cleaning up old junk on that 
server. I've reuploaded it.

[martin pilka]

Hello Stefan,

link works, I was able to run your test. Installation is ok, however test
itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers
installed in order to run'. This happens without CxTest as well, with
latest WineHq. Note that DirectX7 or newer is correctly detected during
installation process.

Is it a bug in my configuration, or did I hit some Wine error?

Thanks,
Martin

Stefan Dösinger wrote:
> Am Dienstag, 31. Juli 2007 12:28 schrieb martin pilka:
> > > With Francois' help I could build a test package for 3DMark2000 now:
> > >
> > > http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
> > >
> > > It runs two tests, one at 640x480, another one at 1024x768, writes the
> > > results to a file and reads the file back. No benchmark results are sent
> > > to the server yet(just ok / fail).
> >
> > Stefan, this link is not valid anymore, could you make the file available
> > again?
>
> Oops, I deleted the file accidentally when cleaning up old junk on that 
> server. I've reuploaded it.

[Stefan Dösinger]

Am Dienstag, 31. Juli 2007 16:42 schrieb martin pilka:
> Hello Stefan,
>
> link works, I was able to run your test. Installation is ok, however test
> itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers
> installed in order to run'. This happens without CxTest as well, with
> latest WineHq. Note that DirectX7 or newer is correctly detected during
> installation process.
>
> Is it a bug in my configuration, or did I hit some Wine error?

Are you running on VNC? Do you have opengl support? This error occurs when I 
run on a system without opengl support.

[martin pilka]

Stefan Dösinger wrote:
> Am Dienstag, 31. Juli 2007 16:42 schrieb martin pilka:
> > Hello Stefan,
> >
> > link works, I was able to run your test. Installation is ok, however test
> > itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers
> > installed in order to run'. This happens without CxTest as well, with
> > latest WineHq. Note that DirectX7 or newer is correctly detected during
> > installation process.
> >
> > Is it a bug in my configuration, or did I hit some Wine error?
>
> Are you running on VNC? Do you have opengl support? This error occurs when I 
> run on a system without opengl support.

It is without CxTest and without VNC. I do have HW acceleration and OpenGL
(GLX) enabled. Attached is output of xdpyinfo.

I am going to try whether 3DMark2001 works.

Martin

[martin pilka]

Hello Stefan!
> It is without CxTest and without VNC. I do have HW acceleration and OpenGL
> (GLX) enabled. Attached is output of xdpyinfo.
>
> I am going to try whether 3DMark2001 works.

3DMark2001 behaves the same way. Note this is long lasting bug, I remember
on WineConf 2006 I was able to run 3DMark2000 under CrossOver, but only
after the fresh installation. Subsequent runs failed with the same error
we have today (DirectX detection failed). It is probably configuration
dependent (~/.wine).

However, I was able to run it on another machine (Mac OS X) using latest
CrossOver nightly build. There were some minor issues, but after a while I
was able to get benchmark.txt file. I submitted that file to CxTest server
several times, to test new Direct3D evaluation functionality. It is here:

http://www.cxtest.org/product-d3d-eval?id_product=3&m_from=7&d_from=...

Please have a look whether this is (roughly) what you need.

To make this useful, we need to solve 'DirectX detection failed' bug, so we
can run real tests nightly. Could you send me ~/.wine directory you are
using with latest WineHq?

Thanks,
Martin

[Stefan Dösinger]

> To make this useful, we need to solve 'DirectX detection failed' bug, so we
> can run real tests nightly. Could you send me ~/.wine directory you are
> using with latest WineHq?

Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi" 
perhaps? This disables D3D support altogether. You could try with a 
fresh .wine, it should run 3dmark2000 and 2001 out of the box.

I'm not sure wether I should really attempt to send you my ~/.wine, it is full 
with games and has a size of 51GB :-) . I could send you the .reg files 
though.

[martin pilka]

> Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi" 
> perhaps? This disables D3D support altogether. You could try with a 
> fresh .wine, it should run 3dmark2000 and 2001 out of the box.

I tried latest WineHq, it created ~/.wine directory and installed
3DMark2000 correctly, however when I try to run I get same complains:
"3DMark needs DirectX7 and proper drivers installed". Also, this is printed
on console:

cxtest@mpilka:~$ ~/cxtest/wine/wine ./.wine/drive_c/Program\
Files/MadOnion.com/3DMark2000/3DMark2000.exe
err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?

Note ~/.wine directory generated by Wine does not mention
HKCU/Software/Wine/Direct3D/DirectDrawRenderer key at all. Also, glxgears
runs fine, xdpyinfo says GL is there (ATIFGLEXTENSION, ATIFGLRXDRI, GLX,
SGI-GLX extensions are present).

Martin

[Stefan Dösinger]

Am Montag, 6. August 2007 16:49 schrieb martin pilka:
> > Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi"
> > perhaps? This disables D3D support altogether. You could try with a
> > fresh .wine, it should run 3dmark2000 and 2001 out of the box.
>
> I tried latest WineHq, it created ~/.wine directory and installed
> 3DMark2000 correctly, however when I try to run I get same complains:
> "3DMark needs DirectX7 and proper drivers installed". Also, this is printed
> on console:
>
> cxtest@mpilka:~$ ~/cxtest/wine/wine ./.wine/drive_c/Program\
> Files/MadOnion.com/3DMark2000/3DMark2000.exe
> err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
> err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
> err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?

is wined3d built?

[martin pilka]

> is wined3d built?

That was it - I was missing some dev libs. Thanks!

I fixed that, now I am receiving this error:

Not all of the E2 drivers were found. Check that you have 3DMark installed
properly.

Attached are the log files from installation and run.
Martin

[martin pilka]

> Hello Martin
>
> If you cd into the 3DMark2000 install directory and run it from there
> does it work?
>
> Tom

Thanks Tom, that was it. I got a bit further, the message I got this time is:

3DMark could not initialize your 3D Accelerator. Try to downgrade your
display settings.

I played with resolutions and xorg.conf file quite a bit, no joy so far,
could you send me your xorg.conf file?

Thanks,
Martin

[Tom Wickline]

Can you try this, set Wine to run in a virtual desktop of 1024x768 and
set X at 16bpp
and run 3DMark2000 in win98 mode. then cd into the install directory
and see if that works.

Tom

On 8/8/07, martin pilka  wrote:
> > Hello Martin
> >
> > If you cd into the 3DMark2000 install directory and run it from there
> > does it work?
> >
> > Tom
>
> Thanks Tom, that was it. I got a bit further, the message I got this time is:
>
> 3DMark could not initialize your 3D Accelerator. Try to downgrade your
> display settings.
>
> I played with resolutions and xorg.conf file quite a bit, no joy so far,
> could you send me your xorg.conf file?
>
> Thanks,
> Martin

[martin pilka]

Francois Gouget wrote:
> You could also write some software to automatically analyze this data on 
> the server but that seems overkill.

Yes, we tried this while ago and it was really overkill. I believe results
should be analyzed by man, but he needs user friendly formatted information
for this. We are trying to provide such information i.e. here:

http://www.cxtest.org/product-evaluation?id_product=3&id_failure=170&...

It is clear that there was a regression between 070109 and 070110.
Martin

[Vit Hrachovy]

On Mon, Jun 25, 2007 at 03:40:42PM +0200, Stefan Dösinger wrote:
> Hi,
> Now that we're facing Direct3D 9 feature completion soon - VTF support coming 
> in from Henri, High order patches and thread safety from me - the d3d work 
> will change a bit soon. Instead of adding new features we'll debug games and 
> improve performance. It would be good if we had some automated testing 
> framework to support the process :-)
>
> Unfortunately this isn't easy. Games do not render exactly, so verifying that 
> a game doesn't show any visual regressions automatically may be more effort 
> than reward. Here I think we'll have to rely on users testing the games 
> regularly and testing regressions.
>
> cxtest comes to my mind(www.cxtest.org). It is a test suit built by 
> codeweavers for running regression tests on Windows apps. It can remote 
> control applications using keyboard input and compares screenshots to 
> reference ones. It could be used to start benchmarks automatically. A plus 
> here is that some machines run nightly tests. However, those are VMs, so no 
> 3D acceleration. A problem is also reading back the results. Does anyone know 
> some applications for doing that? Windows or Linux apps, commercial or open 
> source ones(prefered, though).
>
> Other suggestions?

cxtest has already impressive infratructure (installation, creation of
clean environment) and nice scripting. However, it requires Metacity WM
and VNC.

I'm personally using AutoHotkey(http://www.autohotkey.com) as an
automatic testing framework backend. It's a Windows application, so it 
can be run through Wine itself and it's sandboxed from X11.

I can imagine Winebot packages to be easily converted to an automatic
testing engine. Winebot actually covers automated installation and
creation of clean independent environment.

Running applications with different log levels, tracing output,
comparing screenshots with different levels of fuzzy comparaion 
- it's not faraway.

AFAIK cxtest has the better functionality now.

Regards
Vit

[Stefan Dösinger]

> cxtest has already impressive infratructure (installation, creation of
> clean environment) and nice scripting. However, it requires Metacity WM
> and VNC.

The Metacity WM doesn't really concern me, the VNC thing however does. This 
makes game testing harder since VNC isn't a part of the average game setup. 
We don't have to transfer all the rendering screenshots over VNC though. I am 
CCing Martin Pilka on this, maybe he can give us some more insight.
> I'm personally using AutoHotkey(http://www.autohotkey.com) as an
> automatic testing framework backend. It's a Windows application, so it
> can be run through Wine itself and it's sandboxed from X11.

It looks interesting, this would also allow us to run automated tests on 
Windows, perhaps to track the effect of driver updates, etc.
> I can imagine Winebot packages to be easily converted to an automatic
> testing engine. Winebot actually covers automated installation and
> creation of clean independent environment.

I don't know if we have to automate the installations as well. Reinstalling 
the apps excludes a few factors and makes the tests more repeatable, on the 
other hand it is not something that occurs in the average gaming environment. 
Not reinstalling can bring up problems related to the game's state on the 
disk, like fragmenting game caches. When we see a problem we can reset the 
environment manually to exclude the disk content as the cause. Automated game 
installation is a nice to have, but not a strict requirement IMHO.

Not reinstalling would ease the use of some controller application that runs 
inside wine, since constant installing and removing of the app can leave 
garbage in the C drive and the registry over time. On the other hand, an 
external application could remove the wineprefix and start with a completely 
clean Windows environment.
> Running applications with different log levels, tracing output,
> comparing screenshots with different levels of fuzzy comparaion
> - it's not faraway.

How would fuzzy comparison work?

[Francois Gouget]

On Mon, 25 Jun 2007, Stefan Dösinger wrote:
> > cxtest has already impressive infratructure (installation, creation of
> > clean environment) and nice scripting. However, it requires Metacity WM
> > and VNC.
>
> The Metacity WM doesn't really concern me, the VNC thing however does. This 
> makes game testing harder since VNC isn't a part of the average game setup. 
> We don't have to transfer all the rendering screenshots over VNC though. I am 
> CCing Martin Pilka on this, maybe he can give us some more insight.

cxtest's run.sh script has a --no-vnc option. I know I have once run a 
test outside of VNC so VNC is not a hard requirement. cxtest will 
probably complain about it though, and you might lose some functionality 
like automatic screenshots on failures. The best would be to try it out.

[Stefan Dösinger]

> cxtest's run.sh script has a --no-vnc option. I know I have once run a
> test outside of VNC so VNC is not a hard requirement. cxtest will
> probably complain about it though, and you might lose some functionality
> like automatic screenshots on failures. The best would be to try it out.

--no-vnc works fine for me for playing tests, but it does not work for 
recording tests. Thats pretty bad because the vnc server does not even have 
opengl, not even a software renderer. Because of that the benchmarks refuse 
to start up at all. I tried to write the test scripts manually(I don't need 
many clicks after all), but finding out all the button IDs sounds bad :-/ 
I'll try my luck with keystrokes now...

[Steven Edwards]

On 6/27/07, Stefan Dösinger  wrote:
> --no-vnc works fine for me for playing tests, but it does not work for
> recording tests. Thats pretty bad because the vnc server does not even have
> opengl, not even a software renderer. Because of that the benchmarks refuse

Could we add an extension to vnc to use Mesa?

[H. Verbeet]

On 28/06/07, Steven Edwards  wrote:
> Could we add an extension to vnc to use Mesa?

I suspect it would be non-trivial, but more importantly I think it
would defeat the purpose of measuring performance on different
hardware configurations.

[Stefan Dösinger]

Am Donnerstag, 28. Juni 2007 09:09 schrieb H. Verbeet:
> On 28/06/07, Steven Edwards  wrote:
> > Could we add an extension to vnc to use Mesa?
>
> I suspect it would be non-trivial, but more importantly I think it
> would defeat the purpose of measuring performance on different
> hardware configurations.

The problem with vnc is recording a cxtest test(ie button clicks, keystrokes). 
That doesn't have to be fast, just beeing able to init glx is ok. But I agree 
on the non-trivial point. So far I've used manually coded keystrokes instead 
of button clicks, and it works, although it a bit of pain.

[H. Verbeet]

On 28/06/07, Stefan Dösinger  wrote:
> Am Donnerstag, 28. Juni 2007 09:09 schrieb H. Verbeet:
> > On 28/06/07, Steven Edwards  wrote:
> > > Could we add an extension to vnc to use Mesa?
> >
> > I suspect it would be non-trivial, but more importantly I think it
> > would defeat the purpose of measuring performance on different
> > hardware configurations.
>
> The problem with vnc is recording a cxtest test(ie button clicks, keystrokes).
> That doesn't have to be fast, just beeing able to init glx is ok. But I agree
> on the non-trivial point. So far I've used manually coded keystrokes instead
> of button clicks, and it works, although it a bit of pain.

I see. With eg. 3DMark you'd still have caps issues though.

[Stefan Dösinger]

> > The problem with vnc is recording a cxtest test(ie button clicks,
> > keystrokes). That doesn't have to be fast, just beeing able to init glx
> > is ok. But I agree on the non-trivial point. So far I've used manually
> > coded keystrokes instead of button clicks, and it works, although it a
> > bit of pain.
>
> I see. With eg. 3DMark you'd still have caps issues though.

Thats true, but fortunately I don't need the whole set of tests to run while 
recording. Its enough to get the results dialog to continue recording from 
there. For that I could fake some caps temporarily in wined3d.

[martin pilka]

Stefan Dösinger wrote:
> --no-vnc works fine for me for playing tests, but it does not work for 
> recording tests.

VNC was never hard requirement of CxTest, we are using it to guarantee same
testing environment everywhere. However, it is been a while I ran it without
VNC, let me check.

Martin

[martin pilka]

Hello Stefan!
> We don't have to transfer all the rendering screenshots over VNC though.
> I am CCing Martin Pilka on this, maybe he can give us some more insight.

I apologize for late answer - I was (and still am) ill. However, now I read
whole thread, let me summarize the goals you would expect from test framework:

* Ability to regularly benchmark certain parts of game (and thus Wine, which
is involved) in clean environment and submit raw data results to server

* On the server, process submitted data and present them via web in user
friendly form; support for pointing to regressions (and finding regression
patches) would be nice

Did I miss some important aspect? Feel free to comment.

I try package you created with Francois soon. Then I have a meeting with other
guys who also work on CxTest, we try to find way how CxTest could meet goals
we define here.

Martin

[Stefan Dösinger]

> - Ability to regularly benchmark certain parts of game (and thus Wine,
>
> which is involved) in clean environment and submit raw data results to
> server
> - On the server, process submitted data and present them via web in user
>
> friendly form; support for pointing to regressions (and finding regression
> patches) would be nice
>
> Did I miss some important aspect? Feel free to comment.

Yes, thats essentially what we need. I've been setting up a quick and dirty 
test site myself, with a few lines of php and gnuplot:

http://84.112.174.163/~stefan/laptop/3dmark2000/results.php

The whole setup is highly inflexible, doesn't contain any real test data yet, 
just quick and dirty(~100 lines of php and ~70 lines of gnuplot script).

[martin pilka]

Stefan Dösinger wrote:
> Yes, thats essentially what we need. I've been setting up a quick and
> dirty test site myself, with a few lines of php and gnuplot:
>
> http://84.112.174.163/~stefan/laptop/3dmark2000/results.php

Stefan,
could you put the link life again? Or send me screenshots of it. I am
working on small specification draft what we could do in first iteration.

Thanks,
Martin

Wine Direct3D progress

[H. Verbeet]

On 20/12/2007, Mirek Slugeň  wrote:
> OLD CONFIGURATION:
>  Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 6800 GS
>  Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS,
> SoftwareEmulation
>  Patches: 3DMark hack
>
> ACTUAL CONFIGURATION (bold results):
>  Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 7900 GT (nvidia
> 169.04)
>  Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS,
> SoftwareEmulation
>  Patches: 3DMark hack, small gothic patch, wined3d: rewrite
> IWineD3DSurfaceImpl_BltOverride colorfill codepath, wined3d: use
> IWineD3DDeviceImpl_ClearSurface in IWineD3DDeviceImpl_Clear

Changing the graphics card skews the results a bit :-/
Unfortunately it's pretty hard to say how much of an impact that has per test.

Wine Direct3D progress

[Christoph Frick]

On Thu, Dec 20, 2007 at 03:58:05PM +0100, H. Verbeet wrote:
> > OLD CONFIGURATION:
> >  Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 6800 GS
> >  Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS,
> > SoftwareEmulation
> >  Patches: 3DMark hack
> >
> > ACTUAL CONFIGURATION (bold results):
> >  Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 7900 GT (nvidia
> > 169.04)
> >  Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS,
> > SoftwareEmulation
> >  Patches: 3DMark hack, small gothic patch, wined3d: rewrite
> > IWineD3DSurfaceImpl_BltOverride colorfill codepath, wined3d: use
> > IWineD3DDeviceImpl_ClearSurface in IWineD3DDeviceImpl_Clear
>
> Changing the graphics card skews the results a bit :-/
> Unfortunately it's pretty hard to say how much of an impact that has per test.

and changing drivers (assumption) and the patches won't help either.

Wine Direct3D progress

[Stefan Dösinger]

Am Donnerstag, 20. Dezember 2007 16:45:11 schrieb Christoph Frick:
> > Changing the graphics card skews the results a bit :-/
> > Unfortunately it's pretty hard to say how much of an impact that has per
> > test.
>
> and changing drivers (assumption) and the patches won't help either.

Yeah, the different hardware setup limits the usefulness of the results. It 
would be really interesting to see this comparison with the same hardware and 
driver :-)

On a related note, it would be really cool if people could help with writing 
cxtest(www.cxtest.org) scripts for the various apps. This will allow 
automated performance testing and automated one-click regression testing if 
problems occur. I have once written such benchmarks for ut2004, half life 2 
and 3dmark2000, but more are definitly welcome. My time is too limited to 
take care of all this, so our D3D QA still relies on user testing for the 
most part.

Wine Direct3D progress

[Tom Wickline]

On Dec 20, 2007 9:33 AM, Mirek Slugeň  wrote:
> 3DMark 2006
>  game tests: 768/126/1149 points
>  game 1: 5.2/1.2/9.3 fps
>  game 2: 7.6/0.9/9.7 fps
>  hdr game tests: 298/167/1022 points
>  game 1: 1.976/1.4/8.8 fps
>  game 2: 3.989/1.9/11.6 fps
>  cpu tests: 2412/2426/2665 points
>  game 1: 0.782/0.794/0.855 fps
>  game 2: 1.190/1.186/1.329 fps
>
> Mirek Slugen

I ran 3DMark06 just now and got a score of 3464

Some System Specs:
Quad Core 2.4Ghz,  8GB RAM,  1.2TB HD Space,
640MB 8800GTS, Gentoo Linux

SM2.0 = 1662
HDR SM3 = 880
CPU = 2841

http://img409.imageshack.us/img409/7809/screenshotwinedesktop2oy0.png
http://img516.imageshack.us/img516/3826/screenshotwinedesktop3vu1.png
http://img521.imageshack.us/img521/252/screenshotwinedesktop4xo2.png

Tom

Wine Direct3D progress

[Tom Wickline]

On Dec 21, 2007 12:52 AM, Tom Wickline  wrote:
> 640MB 8800GTS, Gentoo Linux

That should say 8800GTX
> Tom

[martin pilka]

> > Running applications with different log levels, tracing output,
> > comparing screenshots with different levels of fuzzy comparaion
> > - it's not faraway.
>
> How would fuzzy comparison work?

See projects findimagedupes and gqview (duplicate window, hit 'D' in gqview
interface), both presented as packages in standard Debian. These fuzzy compare
images. You can find some images which can be used as input here:

http://dl.ifne.eu/image_matching.tar.gz

Results vary, findimagedupes is quite good, however its accuracy is below 100%.

Martin