I've added four benchmarks to wisotool: 3dmark03 3D Mark 03 (Futuremark, 2003) 3dmark06 3D Mark 06 (Futuremark, 2006) re5bench Resident Evil 5 Benchmark (Capcom, 2009) unigine_heaven Unigen Heaven 2 Benchmark (Unigen, 2010) Are there others worth using (besides, I suppose, 3dmark05)? (3dmark03 isn't working for me because I can't bypass the sysinfo crash, see http://bugs.winehq.org/show_bug.cgi?id=22392 )
You may be able to add the timedemo mode of a few games. Thanks, Scott Ritchie On 04/17/2010 10:37 AM, Dan Kegel wrote:
I've added four benchmarks to wisotool: 3dmark03 3D Mark 03 (Futuremark, 2003) 3dmark06 3D Mark 06 (Futuremark, 2006) re5bench Resident Evil 5 Benchmark (Capcom, 2009) unigine_heaven Unigen Heaven 2 Benchmark (Unigen, 2010)
Are there others worth using (besides, I suppose, 3dmark05)?
(3dmark03 isn't working for me because I can't bypass the sysinfo crash, see http://bugs.winehq.org/show_bug.cgi?id=22392 )
On Sat, Apr 17, 2010 at 3:57 PM, Henri Verbeet <hverbeet(a)gmail.com> wrote:
On 17 April 2010 19:37, Dan Kegel <dank(a)kegel.com> wrote:
I've added four benchmarks to wisotool: What exactly is the use-case for this?
I'm trying to find short, small benchmarks that approximate demanding scenes from popular games. Maybe we can use them for performance regression testing and/or comparing our performance to windows. wisotool is just a convenient way to install games and benchmarks. - Dan
On 18 April 2010 01:03, Dan Kegel <dank(a)kegel.com> wrote:
I'm trying to find short, small benchmarks that approximate demanding scenes from popular games. Maybe we can use them for performance regression testing and/or comparing our performance to windows. wisotool is just a convenient way to install games and benchmarks. Well yes, an automated performance testing system has been on the wined3d todo list for some time now. I even have a pretty good idea of how I'd want it to look and work. It's just never going to happen because there's always something more important.
The thing with benchmarks though is that for them to be of any use you need someone capable of interpreting the data. I guess that means the important question is who "we" are? I assume it's not any of the current main wined3d developers, since Stefan and I both obviously already have our set of applications we use for performance testing.
On Sat, Apr 17, 2010 at 4:33 PM, Henri Verbeet <hverbeet(a)gmail.com> wrote:
Well yes, an automated performance testing system has been on the wined3d todo list for some time now. I even have a pretty good idea of how I'd want it to look and work. It's just never going to happen because there's always something more important.
Would it worth writing up in a wiki page (along with a description of how to run your current tests manually)?
The thing with benchmarks though is that for them to be of any use you need someone capable of interpreting the data. I guess that means the important question is who "we" are? I assume it's not any of the current main wined3d developers, since Stefan and I both obviously already have our set of applications we use for performance testing.
When giving a Wine demo before a skeptical audience, it helps if a few widely used benchmarks at least run correctly. It gives potential Wine adopters a sense of comfort. For example, gamers might relate to the Resident Evil 5 Benchmark. (It looks pretty good, but its frame rate is low.) Business users might relate to Passmark. (Passmark 7 is pretty close to running, just needs gdiplus to get around one missing function. Its jet fighter test has some rendering issues, though. And it has a 64 bit version which might be fun to try.) - Dan
On Sat, Apr 17, 2010 at 12:50 PM, Dan Kegel <dank(a)kegel.com> wrote:
On Sat, Apr 17, 2010 at 4:33 PM, Henri Verbeet <hverbeet(a)gmail.com> wrote:
Well yes, an automated performance testing system has been on the wined3d todo list for some time now. I even have a pretty good idea of how I'd want it to look and work. It's just never going to happen because there's always something more important.
Would it worth writing up in a wiki page (along with a description of how to run your current tests manually)?
FWIW, there's http://wiki.winehq.org/BenchMark-0.9.5 / http://wiki.winehq.org/BenchMark-0.9.6, though a more generic 'WineBenchmark' wiki page would be nice. -- -Austin
On Sat, Apr 17, 2010 at 7:47 PM, Austin English <austinenglish(a)gmail.com> wrote:
On Sat, Apr 17, 2010 at 4:33 PM, Henri Verbeet <hverbeet(a)gmail.com> wrote:
Well yes, an automated performance testing system has been on the wined3d todo list for some time now. I even have a pretty good idea of how I'd want it to look and work. It's just never going to happen because there's always something more important.
FWIW, there's http://wiki.winehq.org/BenchMark-0.9.5 / http://wiki.winehq.org/BenchMark-0.9.6
I'm looking for something more automated. At the moment, my plan is to simply dedicate a machine and script a daily run of a couple benchmarks with +fps, and upload the resulting framerates to a web page so we can notice if somebody checks in something that slows them down. - Dan
On Sat, Apr 17, 2010 at 9:57 PM, Dan Kegel <dank(a)kegel.com> wrote:
On Sat, Apr 17, 2010 at 7:47 PM, Austin English <austinenglish(a)gmail.com> wrote:
On Sat, Apr 17, 2010 at 4:33 PM, Henri Verbeet <hverbeet(a)gmail.com> wrote:
Well yes, an automated performance testing system has been on the wined3d todo list for some time now. I even have a pretty good idea of how I'd want it to look and work. It's just never going to happen because there's always something more important.
FWIW, there's http://wiki.winehq.org/BenchMark-0.9.5 / http://wiki.winehq.org/BenchMark-0.9.6
I'm looking for something more automated.
At the moment, my plan is to simply dedicate a machine and script a daily run of a couple benchmarks with +fps, and upload the resulting framerates to a web page so we can notice if somebody checks in something that slows them down.
Sorry, I meant that would be a good wikipage to get an example from. -- -Austin
Am 18.04.2010 um 04:57 schrieb Dan Kegel:
At the moment, my plan is to simply dedicate a machine and script a daily run of a couple benchmarks with +fps, and upload the resulting framerates to a web page so we can notice if somebody checks in something that slows them down. - Dan I still have my cxtest scripts and php hackerish that runs a bunch of benchmarks, and I'm still running them. Unfortunately this stuff is a pain to maintain, for various reasons.
I'm testing 3dmark2k, 3dmark2k1, ut2004, halflife2 and team fortress 2. I am also running the Wine tests. The big requirement for the benchmark games is that I have a way to automatically read the results from the games. Running with +fps and collecting the output is imprecise IMO. The main problems are: *) Maintainance. The tf2 timedemo format changes every now and then, and similar issues *) The way I wrote the code. Very hacky. *) The test machines. I could use a server farm, but I don't have any. The boxes are standing in my living room. *) System updates(ubuntu) need to be installed and can affect performance. *) Unreliable apps. The steam login fails every now and then, requiring manual intervention.
On Sun, Apr 18, 2010 at 5:40 PM, Stefan Dösinger <stefandoesinger(a)gmx.at> wrote:
Am 18.04.2010 um 04:57 schrieb Dan Kegel:
At the moment, my plan is to simply dedicate a machine and script a daily run of a couple benchmarks with +fps, and upload the resulting framerates to a web page so we can notice if somebody checks in something that slows them down. - Dan I still have my cxtest scripts and php hackerish that runs a bunch of benchmarks, and I'm still running them. Unfortunately this stuff is a pain to maintain, for various reasons.
I'm testing 3dmark2k, 3dmark2k1, ut2004, halflife2 and team fortress 2. I am also running the Wine tests. The big requirement for the benchmark games is that I have a way to automatically read the results from the games. Running with +fps and collecting the output is imprecise IMO.
There's not much better than can be done. Some games use the same calculation as in Wine while others take into account previous results. Here I have adjusted the code to basically: current_fps = 0.1*last_fps + 0.9*frames/(current_time - prev_time) and adjusted the sample period to 1s. This made the fps a bit more accurate.
The main problems are:
*) Maintainance. The tf2 timedemo format changes every now and then, and similar issues *) The way I wrote the code. Very hacky. *) The test machines. I could use a server farm, but I don't have any. The boxes are standing in my living room. *) System updates(ubuntu) need to be installed and can affect performance. *) Unreliable apps. The steam login fails every now and then, requiring manual intervention.
FPS reading requires some hacks to the Wine code anyway e.g. a way to enable/disable fps dumping. Cxtest or autohotkey can be used to script the games but yeah it is hard to deal with things like connection errors. Roderick
participants (6)
-
Austin English -
Dan Kegel -
Henri Verbeet -
Roderick Colenbrander -
Scott Ritchie -
Stefan Dösinger