Fixing the conformance tests

[Paul Vriens]

Hi,

As we are slowly progressing towards a more green test.winehq.org there are 
still some conformance tests that fail either on all platforms or with a 
specific configuration (like IE7).

This email is merely to list those and see if the SME's can have a look in these 
tests. (I didn't put to much effort in stuff below XP and left out user32)

Tests with timeouts
===================
- dplayx:dplayx fails with a timeout on all windows servers
- msi:install fails with a timeout on a lot of windows servers

Tests that fail with IE7 installed
==================================
- hlink:hlink has a consistent 8 failures on servers with IE7
- mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others)
- shdocvw:webbrowser has 3 failures on servers with IE7.

Tests that fail starting with a specific OS (or level)
======================================================
- msi:package has 16 failures on XP-SP3+ and 16 extra on W2K3-SP2+
- ole32:storage32 has 24 failures on XP-SP3+ and 24 extra on W2K3-SP2+
(both the above have to do with access rights on files)

Tests that fail on Windows and succeed on Wine
==============================================
- ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine

Tests with failures on all Windows boxes
========================================
- riched20:editor has failures on all boxes
- urlmon:protocol either has timeouts or failures on all boxes
- urlmon:url fails with 6 consistent failures on all boxes

The above is a subset of all the failures and are not the easy ones. There are 
still loads of tests that fail on only 1 OS or even on 1 server.

[James Hawkins]

On Sat, Sep 6, 2008 at 5:19 AM, Paul Vriens  wrote:
> Hi,
>
> As we are slowly progressing towards a more green test.winehq.org there are
> still some conformance tests that fail either on all platforms or with a
> specific configuration (like IE7).
>
> This email is merely to list those and see if the SME's can have a look in these
> tests. (I didn't put to much effort in stuff below XP and left out user32)
> Tests with timeouts
> - dplayx:dplayx fails with a timeout on all windows servers

According to Ismael, he has a few fixes that cut the time needed in
half.  Ismael, what's the status on this?

http://winehq.org/pipermail/wine-devel/2008-August/068125.html
> - msi:install fails with a timeout on a lot of windows servers

Working on it, but it makes me seriously grumpy.
> Tests that fail with IE7 installed
> - hlink:hlink has a consistent 8 failures on servers with IE7

Mike M.'s gone, so someone is going to have to step up to the plate on
this one.  It won't be that hard.
> - mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others)
> - shdocvw:webbrowser has 3 failures on servers with IE7.

Jacek has stated he doesn't have much time or interest in fixing
these, so the plate calls again.
> Tests that fail starting with a specific OS (or level)
> - msi:package has 16 failures on XP-SP3+ and 16 extra on W2K3-SP2+
> - ole32:storage32 has 24 failures on XP-SP3+ and 24 extra on W2K3-SP2+
>
> (both the above have to do with access rights on files)

Pretty easy.  I have a patch to send in tomorrow.
> Tests that fail on Windows and succeed on Wine
> - ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine

Last attempt got no comments:

http://winehq.org/pipermail/wine-patches/2008-May/054547.html
> Tests with failures on all Windows boxes
> - riched20:editor has failures on all boxes

These are all pretty easy to fix.
> - urlmon:protocol either has timeouts or failures on all boxes

Jacek has said this is a known bug, but the fix is not easy.  Jacek,
can you elaborate on that?  Maybe someone else can take a look at it
if you're too busy.
> - urlmon:url fails with 6 consistent failures on all boxes

Just need someone to look at the results and write up a fix.
> The above is a subset of all the failures and are not the easy ones. There are
> still loads of tests that fail on only 1 OS or even on 1 server.

Besides the urlmon:protocol bug, the rest are all pretty easy to fix
(referring to the issues you brought up in this message).

Folks, fixing the tests is a relatively easy task.   It's also a great
way to become better acquainted with the code base and the development
process if you're new to working on Wine.  For current developers, if
you've written tests in the past, check out test.winehq.org every once
in a while and see if any of your tests are failing.  And yes, we do
care about failures in win9x.  The average number of test failures
today is 64.7, which is down from 72.4 a month ago*.  Let's get that
down to zero.

* Averages of each windows version were then averaged together.

[Paul Vriens]

James Hawkins wrote:
> On Sat, Sep 6, 2008 at 5:19 AM, Paul Vriens  wrote:
> > Hi,
> >
> > As we are slowly progressing towards a more green test.winehq.org there are
> > still some conformance tests that fail either on all platforms or with a
> > specific configuration (like IE7).
> >
> > This email is merely to list those and see if the SME's can have a look in these
> > tests. (I didn't put to much effort in stuff below XP and left out user32)
> > Tests with timeouts
> > - dplayx:dplayx fails with a timeout on all windows servers
>
> According to Ismael, he has a few fixes that cut the time needed in
> half.  Ismael, what's the status on this?
>
> http://winehq.org/pipermail/wine-devel/2008-August/068125.html

Unfortunately half won't do. I've tested this on an XP of mine and it 
already took 8 minutes before it crashed. (I'm very interested in the 
results on Wine once most of the dplayx stuff gets implemented.)
> > - mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others)
> > - shdocvw:webbrowser has 3 failures on servers with IE7.
>
> Jacek has stated he doesn't have much time or interest in fixing
> these, so the plate calls again.

Time is a valid argument. No interest isn't. How can we be sure to 
implement the correct stuff if the conformance tests fail? (At some 
point in the not to far future we could only have IE7 boxes where the 
tests are run).
> Folks, fixing the tests is a relatively easy task.   It's also a great
> way to become better acquainted with the code base and the development
> process if you're new to working on Wine.  For current developers, if
> you've written tests in the past, check out test.winehq.org every once
> in a while and see if any of your tests are failing.  And yes, we do
> care about failures in win9x.  The average number of test failures
> today is 64.7, which is down from 72.4 a month ago*.  Let's get that
> down to zero.

+1

[Ismael Barros]

On 9/6/08, James Hawkins  wrote:
> > Tests with timeouts
> > - dplayx:dplayx fails with a timeout on all windows servers
>
> According to Ismael, he has a few fixes that cut the time needed in
> half.  Ismael, what's the status on this?
>
> http://winehq.org/pipermail/wine-devel/2008-August/068125.html

I managed to cut the time (in 3/4, if I recall correctly) just
changing a default timeout value in a function, but times were still
bigger than desired (around 7 minutes for all tests, I think).

Then I tried to run all tests concurrently, but it looks like dplay
doesn't like concurrency too much, in my Windows XP I started seeing
~200 errors in tests that produce 0 errors when run secuentially. When
running like only the 4 first tests at a time, it somtimes produced
errors, sometimes didn't, and I'm afraid random results in tests are
not really liked.

This attempt to make the tests concurrent can be found at
http://repo.or.cz/w/wine/gsoc_dplay.git

I'll try to figure out how to solve this, but right now and until next
week I'll be busy with exams, and even after that I won't have much
time to spend on wine. If somebody wants to quickly get rid of those
errors for now, it'd be okay to comment the test calls out in the
START_TEST of tests/dplay.c, I'll re-enable them when I find a
solution.

About the errors in non-XP versions of Windows, I suppose I'll have to
get and virtualize them to run the tests against them.

Cheers
Ismael

[Kai Blin]

On Saturday 06 September 2008 22:21:15 James Hawkins wrote:
> > Tests that fail on Windows and succeed on Wine
> > - ws2_32:sock has 1 consistent failure on all Windows boxes but not on
> >
> > Wine
>
> Last attempt got no comments:
>
> http://winehq.org/pipermail/wine-patches/2008-May/054547.html

Actually, Alexandre commented on this on IRC. I don't really recall his 
comment, but he didn't like the approach. I'll have some time to look into 
this again tomorrow, but I remember I quit posting new patches to this as I 
was out of ideas on how to fix it. A couple of bug reports seem to be related 
to this, I need to dig around some more.

Cheers,
Kai