Benchmark's on the Wiki

[Tom Wickline]

Hello,

I put the Benchmark results that I posted to wine-devel back in April
on the Wiki.
http://wiki.winehq.org/BenchMark

I would appreciate any constitutive criticism to improve this page.

Tom

[Mitchell Mebane]

Tom Wickline wrote:
> Hello,
>
> I put the Benchmark results that I posted to wine-devel back in April
> on the Wiki.
> http://wiki.winehq.org/BenchMark
>
> I would appreciate any constitutive criticism to improve this page.
>
> Tom
>
> .

I'd be interested to see how 3DMark05 runs. Do Oliver's patches support 
it yet?

--Mitchell

[Tom Wickline]

On 7/26/05, Mitchell Mebane  wrote:
> I'd be interested to see how 3DMark05 runs. Do Oliver's patches support it
> yet?

Back in April 3DMark05 would install and load but there was a lack of 
Pixel Shader 2.0
support, so none of the test would run.

http://www.futuremark.com/products/3dmark05/?minreq

A quick look at : http://directxwine.sourceforge.net/
"pixel shaders should be implemented in the next release or two
depending on the amount of regression and other bugs that popup."

Oliver would you have a ETA for Pixel Shader 2/3 ?

Tom
> --Mitchell

[Oliver Stieber]

--- Tom Wickline  wrote:
> On 7/26/05, Mitchell Mebane  wrote:
> > I'd be interested to see how 3DMark05 runs. Do Oliver's patches support it
> > yet?
>
> Back in April 3DMark05 would install and load but there was a lack of 
> Pixel Shader 2.0
> support, so none of the test would run.
>
> http://www.futuremark.com/products/3dmark05/?minreq
>
> A quick look at : http://directxwine.sourceforge.net/
> "pixel shaders should be implemented in the next release or two
> depending on the amount of regression and other bugs that popup."
>
> Oliver would you have a ETA for Pixel Shader 2/3 ?

I hope to get vertexshaders into winecvs by Friday this week, it's more or less the same as
vertexshader support in d3d8 but with a load of parser fixes (it can now parse V2 and 3 shaders).
Pixelshaders are more or less identical to vertexshaders so they should follow soon afterwards.

To properly support version 2 and 3 shaders we need to use GL shader language instead of vertex
and fragment program, it expect it to take a couple of weeks to get a cross compiler up and
running. Blends also needs to be implemented using shaders to efficiently use VBO and the cards
onboard memory for animated models. This affect more applications / games than V2 and 3 shaders so
I'll try and develop it first.
> Tom
> > --Mitchell

___________________________________________________________ 
Does your mail provider give you FREE antivirus protection? 
Get Yahoo! Mail http://uk.mail.yahoo.com

[Anssi Hannula]

Tom Wickline wrote:
> Hello,
>
> I put the Benchmark results that I posted to wine-devel back in April
> on the Wiki.
> http://wiki.winehq.org/BenchMark
>
> I would appreciate any constitutive criticism to improve this page.

I think it would be easier to read if the wine and xp results were 
side-to-side in a table?

[Anssi Hannula]

Anssi Hannula wrote:
> Tom Wickline wrote:
> > I put the Benchmark results that I posted to wine-devel back in April
> > on the Wiki.
> > http://wiki.winehq.org/BenchMark
> >
> > I would appreciate any constitutive criticism to improve this page.
>
> I think it would be easier to read if the wine and xp results were 
> side-to-side in a table?

So Codeweavers' email/list server delayed my email for 11 hours so that 
Paul would beat me to it and propose the same idea before me? unfair :P

[Dmitry Timoshkov]

"Anssi Hannula"  wrote:
> So Codeweavers' email/list server delayed my email for 11 hours so that 
> Paul would beat me to it and propose the same idea before me? unfair :P

Are you a subscriber of the mailing list with the address you were writing
from?

[Anssi Hannula]

Dmitry Timoshkov wrote:
> "Anssi Hannula"  wrote:
> > So Codeweavers' email/list server delayed my email for 11 hours so that 
> > Paul would beat me to it and propose the same idea before me? unfair :P
>
> Are you a subscriber of the mailing list with the address you were writing
> from?

Oops, I thought I switched the subscriber address to this list when I 
switched others. Ok, that explains it :)

[Tom Wickline]

On 7/27/05, Anssi Hannula  wrote:
> Anssi Hannula wrote:
> So Codeweavers' email/list server delayed my email for 11 hours so that
> Paul would beat me to it and propose the same idea before me? unfair :P

:-)

Hello Anssi,

Is this what you guys had in mind? 
http://wiki.winehq.org/BenchMark

And just add a third column with +/- %'s

Any other Recommendations?

Tom
> --
> Anssi Hannula

[Christoph Frick]

On Thu, Jul 28, 2005 at 04:29:53AM -0400, Tom Wickline wrote:
> And just add a third column with +/- %'s

can the test runs be automated like the winetests? and so have them
updated automatically with different machines? or is there a way to not
only test "functionality" with tests but also "speed"? i guess this
would be nice to find bottlenecks.

(maybe this is already done - if so excuse my ignorance ;))

[Holger Dell]

hi,

Christoph Frick wrote on 07/28/2005 11:55 AM:
> or is there a way to not only test "functionality" with tests but also "speed"? i guess this
> would be nice to find bottlenecks.
>
> (maybe this is already done - if so excuse my ignorance ;))

has anyone ever tried to use grpof or any other profiling tool to 
analyze, which code eats most cpu time (=bottlenecks).
normally, if you develop something, you first go for functionality and 
after that for speed. it's not good for wine to hack for speed (like 
cedega does, as far as I know) and recognize later, that it was the 
wrong decision, because of incompatibilities or incompleteness. (my 
professor would say "..and now we can calculate bullshit in a nanosecond").
nevertheless, it would not be wrong to have some clearly arranged 
detailed time profiling of wine, when started with several different 
applications.

holger.

[Christoph Frick]

On Thu, Jul 28, 2005 at 05:34:08PM +0200, Holger Dell wrote:
> > or is there a way to not only test "functionality" with tests but
> > also "speed"? i guess this would be nice to find bottlenecks.
>
> has anyone ever tried to use grpof or any other profiling tool to
> analyze, which code eats most cpu time (=bottlenecks).

i would consider gprof a tool to optimize an application - but its a
little hard to profile an API with it. different apps will produces
different results and so using gprof with certain apps (mainstream)
results in _exaclty_ what you dont want (what transgaming does?).

thinking about it - also using the benchmarks as a mark is the same
(isnt that what all the driver developers do?).

but having winetests, that also do certain things (lets say application
"patterns") and then time them is IMHO a Good Thing - it can be coded
into the winetest framework, which already runs the tests on real
windows and on wine - then also time it and compare it.

especially in the directx area this would be good to compare to windows
(even if its sometimes simply not fair - if e.g. sound mixing is done in
hardware under windows and in the cpu on wine).

[Tom Wickline]

On 7/28/05, Holger Dell  wrote:
> has anyone ever tried to use grpof or any other profiling tool to
> analyze, which code eats most cpu time (=bottlenecks).
> normally, if you develop something, you first go for functionality and
> after that for speed. it's not good for wine to hack for speed (like
> cedega does, as far as I know) and recognize later, that it was the
> wrong decision, because of incompatibilities or incompleteness.

Anyone here use Cedega and have the time to waist and benchmark it
against Wine? Would a modern benchmark even run on there Warez? As im
a advocate of OSS I don't use there offerings.

Tom
> holger.

[Holger Dell]

hi,

Tom Wickline wrote on 07/28/2005 10:29 AM:
> Is this what you guys had in mind? 
> http://wiki.winehq.org/BenchMark

I don't think so. The table format I had expected is:

$(benchmark name) | $(wine result) | $(XP result) | $([any other 
interesting OS] result)
> Any other Recommendations?

Mac OS X benchmarking results (where available) would be cool.

holger.

[Michael Stefaniuc]

Holger Dell wrote:
> Tom Wickline wrote on 07/28/2005 10:29 AM:
> > Is this what you guys had in mind? http://wiki.winehq.org/BenchMark
>
> I don't think so. The table format I had expected is:
>
> $(benchmark name) | $(wine result) | $(XP result) | $([any other 
> interesting OS] result)
> > Any other Recommendations?
>
> Mac OS X benchmarking results (where available) would be cool.

I do not think that MacOS X runs on his hardware. Using an other box for 
that would be like comparing apples with oranges.

bye
    michael

[Tom Wickline]

On 7/28/05, Michael Stefaniuc  wrote:
> > Mac OS X benchmarking results (where available) would be cool.
>
> I do not think that MacOS X runs on his hardware. Using an other box for
> that would be like comparing apples with oranges.

That's one reason why I don't list Win2k or Win98 results, I have
Win2k and 98 on other boxes with totally different hardware. This way
everything is equal as far as hardware goes, because a benchmark
comparison is total crap unless its done on a duel boot system.

And im to lazy to set up duel boot NT,95,98,2K boxes :-) So XP is about it....

Tom
> bye
>         michael

[Holger Dell]

hi,

Michael Stefaniuc wrote on 07/28/2005 12:37 PM:
> I do not think that MacOS X runs on his hardware. Using an other box for 
> that would be like comparing apples with oranges.

hm, ok.
what about native-linux ut2004 and q3, do they have benchmarks on linux, 
too? would be interesting to have the native version vs. wine version as 
well, wouldn't it?

holger.

[Tom Wickline]

On 7/28/05, Holger Dell  wrote:
> hm, ok.
> what about native-linux ut2004 and q3, do they have benchmarks on linux,
> too? would be interesting to have the native version vs. wine version as
> well, wouldn't it?

Gentoo            Demo 1      Demo2
                       388.3fps      391.0fps

XP                  Demo 1      Demo2
                       355.7fps      353.3fps

Wine                Demo 1       Demo2     
                        322.5fps     319.0fps

ftp://ftp.idsoftware.com/idstuff/quake3/linux/

If you receive a error in check sums try this:
export _POSIX2_VERSION=199209

1024x768x24 fullscreen = off
~ "timedemo 1"  run the demo then ~ to see fps

Anyone care to share there scores :-)

Tom
> holger.

[Dimi Paun]

From: "Tom Wickline"
> Is this what you guys had in mind? 
> http://wiki.winehq.org/BenchMark
>
> And just add a third column with +/- %'s
>
> Any other Recommendations?

Why repeat the measurements? Something like this
would be more desirable:

Wine            XP
3DMark Result:      14,606 marks    13,332 marks
CPU Speed:          786 CPU Marks    650 CPU Marks
...

Or even better, factor out the measurement unit too:

Wine       XP       Wine vs. Win     Unit
3DMark Result:      14,606     13,332      +10%          marks
CPU Speed:          786        650         +21%          CPU Marks
...

Or you can include the unit with the measure:

Wine       XP       Wine vs. Win
3DMark Result (marks):      14,606     13,332      +10% 
CPU Speed (CPU Marks):      786        650         +21% 
...

I personally like the last one the best.

[Tom Wickline]

On 7/28/05, Dimi Paun  wrote:
>                                           Wine       XP          Wine vs. Win
>
> 3DMark Result (marks):      14,606     13,332      +10%
> CPU Speed (CPU Marks):      786        650           +21%
> ...
>
> I personally like the last one the best.

Okay #3 it is... Ill have time to do it the *correct* way on Saturday.
What is there now will at least make it easy to view on bench against the other.

Thanks,

Tom
> --
> Dimi Paun 
> Lattica, Inc.

[Francois Gouget]

On Thu, 28 Jul 2005, Dimi Paun wrote:
[...]
>                        Wine       XP       Wine vs. Win
>
> 3DMark Result (marks):      14,606     13,332      +10%
> CPU Speed (CPU Marks):      786        650         +21%
> ...

I would exchange the Wine and XP columns. That would make the percentage 
column more understandable.

[Tom Wickline]

On 7/28/05, Dimi Paun  wrote:
>                         Wine       XP       Wine vs. Win
>
> 3DMark Result (marks):      14,606     13,332      +10%
> CPU Speed (CPU Marks):      786        650         +21%
> ...
>
> I personally like the last one the best.

I'm not finished, but I thought I would ask if this going in the right
direction?
http://wiki.winehq.org/BenchMark

Tom
> --
> Dimi Paun 
> Lattica, Inc.

[Felix Nawothnig]

Tom Wickline wrote:
> I'm not finished, but I thought I would ask if this going in the right
> direction?
> http://wiki.winehq.org/BenchMark

Much better, although Windows should really be on the left side.

Felix

[Tom Wickline]

On 7/30/05, Felix Nawothnig  wrote:
> Tom Wickline wrote:
> > I'm not finished, but I thought I would ask if this going in the right
> > direction?
> > http://wiki.winehq.org/BenchMark
>
> Much better, although Windows should really be on the left side.

Why? its Wine against XP ... not XP against Wine.
But ill change it if others think it would be best :-)

Tom
> Felix

[Felix Nawothnig]

Tom Wickline wrote:
> > > I'm not finished, but I thought I would ask if this going in the right
> > > direction?
> > > http://wiki.winehq.org/BenchMark
> >
> > Much better, although Windows should really be on the left side.
>
> Why? its Wine against XP ... not XP against Wine.

Maybe it's just me (no it's not, someone else in this thread also 
suggested it ;) but when I looked at it I really expected XP being on 
the left side.

This is probably caused by the third column - a green +42% looks like 
the expected increase when switching from Windows to Wine for me.

Another explanation would be that we are reimplementing the Windows API 
- first there was Windows, then Wine was written.

I'd also like to note that this (Windows, Wine) is the same order as 
used on test.winehq.com:

95,98,ME => NT3,NT4,2K,XP,2K3 => Wine

(consider that this chronology is in terms of popularity with the 
families grouped afterwards).

It also implies that Wine will take over the world one day. :-)

Felix

[Mitchell Mebane]

Tom Wickline wrote:
> On 7/30/05, Felix Nawothnig  wrote:
> > Tom Wickline wrote:
> > > I'm not finished, but I thought I would ask if this going in the right
> > > direction?
> > > http://wiki.winehq.org/BenchMark
> >
> > Much better, although Windows should really be on the left side.
>
> Why? its Wine against XP ... not XP against Wine.
> But ill change it if others think it would be best :-)
>
> Tom
> > Felix
>
> .

Yes, in this case XP is the reference platform, so it should be on the 
left. Also, you could change the title of the comparison column to "Wine 
Difference" or some such.

--Mitchell

[Dimi Paun]

On Sat, 2005-07-30 at 05:43 -0400, Tom Wickline wrote:
> I'm not finished, but I thought I would ask if this going in the right
> direction?
> http://wiki.winehq.org/BenchMark

Looks great! I must admit that having XP on the left side may be
preferable, if it's not too much work.

[Tom Wickline]

On 7/30/05, Dimi Paun  wrote:
> Looks great! I must admit that having XP on the left side may be
> preferable, if it's not too much work.

Does this look presentable?

Any more recommendations? Anyone other than Mitchell think I should
change "Wine vs. XP" to "Wine Difference" ?

Tom
> --
> Dimi Paun 
> Lattica, Inc.

[Anssi Hannula]

Tom Wickline wrote:
> Does this look presentable?

Much better now.
> Any more recommendations?

The % are apparently calculated wrong.

You should use:
(Wine score / XP score) - 1

For example in the first CPU Marks you say XP 650, Wine 786, +18%

However, (786 / 650) - 1 = 0.2092307692307692 ~ +21%

[Anssi Hannula]

Anssi Hannula wrote:
> Tom Wickline wrote:
> > Any more recommendations?
>
> The % are apparently calculated wrong.
>
> You should use:
> (Wine score / XP score) - 1
>
> For example in the first CPU Marks you say XP 650, Wine 786, +18%
>
> However, (786 / 650) - 1 = 0.2092307692307692 ~ +21%

I don't know if you already read my post, but I see now some very 
strange numbers, like -326%. That would mean Wine had negative 3D marks :S

Just use the above formula (e.g. Wine's score always first), no matter 
if wine has a better or worse result than XP.

Wrong: 19197 / 5888 = 3.26 => -326%

Right: (5888 / 19197) - 1 = -0.69 = -69%

[Tom Wickline]

On 7/30/05, Anssi Hannula  wrote:
> I don't know if you already read my post, but I see now some very
> strange numbers, like -326%. That would mean Wine had negative 3D marks :S

:-)
> Just use the above formula (e.g. Wine's score always first), no matter
> if wine has a better or worse result than XP.
>
> Wrong: 19197 / 5888 = 3.26 => -326%
>
> Right: (5888 / 19197) - 1 = -0.69 = -69%

Yea I know, I was goofing off and put the wrong ones in just to see
what it looked like...... horrid to say the least...

If you have a couple free minutes can you go over my %'s to make sure
i didn't make a typo?

BTW, Thanks!

Tom
> --
> Anssi Hannula

[Dimi Paun]

On Sat, 2005-07-30 at 21:48 -0400, Tom Wickline wrote:
> Does this look presentable?

Looks very good. It's way more useful this way.

[Tom Wickline]

The DronezMark scores from  a couple days back are incorrect!
I had VSync set in XP and that was the reason for Wine having such a huge
% increase over XP. I re-ran the test on XP and Wine with sound set to 
Performance and VSync turned off. XP 1024x768x32 Wine 1024x768x24

Tom

[Paul Vriens]

On Wed, 2005-07-27 at 00:49, Tom Wickline wrote:
> Hello,
>
> I put the Benchmark results that I posted to wine-devel back in April
> on the Wiki.
> http://wiki.winehq.org/BenchMark
>
> I would appreciate any constitutive criticism to improve this page.
>
> Tom

Hi Tom,

maybe it's an idea to put the WinXP and Wine results next to each other,
to have some immediate idea of the differences. Maybe even add a diff
column showing how well/bad we do against WinXP (or any other version
tested).

Paul.

[Tom Wickline]

On 7/27/05, Paul Vriens  wrote:
> Hi Tom,

Hello Paul,
> maybe it's an idea to put the WinXP and Wine results next to each other,
> to have some immediate idea of the differences.

Okay, I'll try to do this by Saturday.
> Maybe even add a diff column showing how well/bad we do against WinXP (or any > other version tested).

What I would like to do in the end is have two benchmark pages. Wine
against XP and Wine old-version against wine current and show our
progression/regression  each six months or so. And have the diff
column with +/- %'s for both pages.

Thanks,

Tom
> Paul.

[Dimi Paun]

On Thu, 2005-07-28 at 00:35 -0400, Tom Wickline wrote:
> What I would like to do in the end is have two benchmark pages. Wine
> against XP and Wine old-version against wine current and show our
> progression/regression  each six months or so. And have the diff
> column with +/- %'s for both pages.

I think it would be best to have all these on one page, but
different columns, so we can easily compare across.