Fixing the conformance tests
Hi, As we are slowly progressing towards a more green test.winehq.org there are still some conformance tests that fail either on all platforms or with a specific configuration (like IE7). This email is merely to list those and see if the SME's can have a look in these tests. (I didn't put to much effort in stuff below XP and left out user32) Tests with timeouts =================== - dplayx:dplayx fails with a timeout on all windows servers - msi:install fails with a timeout on a lot of windows servers Tests that fail with IE7 installed ================================== - hlink:hlink has a consistent 8 failures on servers with IE7 - mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others) - shdocvw:webbrowser has 3 failures on servers with IE7. Tests that fail starting with a specific OS (or level) ====================================================== - msi:package has 16 failures on XP-SP3+ and 16 extra on W2K3-SP2+ - ole32:storage32 has 24 failures on XP-SP3+ and 24 extra on W2K3-SP2+ (both the above have to do with access rights on files) Tests that fail on Windows and succeed on Wine ============================================== - ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine Tests with failures on all Windows boxes ======================================== - riched20:editor has failures on all boxes - urlmon:protocol either has timeouts or failures on all boxes - urlmon:url fails with 6 consistent failures on all boxes The above is a subset of all the failures and are not the easy ones. There are still loads of tests that fail on only 1 OS or even on 1 server. -- Cheers, Paul.
On Sat, Sep 6, 2008 at 5:19 AM, Paul Vriens <paul.vriens.wine(a)gmail.com> wrote:
Hi,
As we are slowly progressing towards a more green test.winehq.org there are still some conformance tests that fail either on all platforms or with a specific configuration (like IE7).
This email is merely to list those and see if the SME's can have a look in these tests. (I didn't put to much effort in stuff below XP and left out user32)
Tests with timeouts =================== - dplayx:dplayx fails with a timeout on all windows servers
According to Ismael, he has a few fixes that cut the time needed in half. Ismael, what's the status on this? http://winehq.org/pipermail/wine-devel/2008-August/068125.html
- msi:install fails with a timeout on a lot of windows servers
Working on it, but it makes me seriously grumpy.
Tests that fail with IE7 installed ================================== - hlink:hlink has a consistent 8 failures on servers with IE7
Mike M.'s gone, so someone is going to have to step up to the plate on this one. It won't be that hard.
- mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others) - shdocvw:webbrowser has 3 failures on servers with IE7.
Jacek has stated he doesn't have much time or interest in fixing these, so the plate calls again.
Tests that fail starting with a specific OS (or level) ====================================================== - msi:package has 16 failures on XP-SP3+ and 16 extra on W2K3-SP2+ - ole32:storage32 has 24 failures on XP-SP3+ and 24 extra on W2K3-SP2+ (both the above have to do with access rights on files)
Pretty easy. I have a patch to send in tomorrow.
Tests that fail on Windows and succeed on Wine ============================================== - ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine
Last attempt got no comments: http://winehq.org/pipermail/wine-patches/2008-May/054547.html
Tests with failures on all Windows boxes ======================================== - riched20:editor has failures on all boxes
These are all pretty easy to fix.
- urlmon:protocol either has timeouts or failures on all boxes
Jacek has said this is a known bug, but the fix is not easy. Jacek, can you elaborate on that? Maybe someone else can take a look at it if you're too busy.
- urlmon:url fails with 6 consistent failures on all boxes
Just need someone to look at the results and write up a fix.
The above is a subset of all the failures and are not the easy ones. There are still loads of tests that fail on only 1 OS or even on 1 server.
Besides the urlmon:protocol bug, the rest are all pretty easy to fix (referring to the issues you brought up in this message). Folks, fixing the tests is a relatively easy task. It's also a great way to become better acquainted with the code base and the development process if you're new to working on Wine. For current developers, if you've written tests in the past, check out test.winehq.org every once in a while and see if any of your tests are failing. And yes, we do care about failures in win9x. The average number of test failures today is 64.7, which is down from 72.4 a month ago*. Let's get that down to zero. * Averages of each windows version were then averaged together. -- James Hawkins
James Hawkins wrote:
On Sat, Sep 6, 2008 at 5:19 AM, Paul Vriens <paul.vriens.wine(a)gmail.com> wrote:
Hi,
As we are slowly progressing towards a more green test.winehq.org there are still some conformance tests that fail either on all platforms or with a specific configuration (like IE7).
This email is merely to list those and see if the SME's can have a look in these tests. (I didn't put to much effort in stuff below XP and left out user32)
Tests with timeouts =================== - dplayx:dplayx fails with a timeout on all windows servers
According to Ismael, he has a few fixes that cut the time needed in half. Ismael, what's the status on this?
http://winehq.org/pipermail/wine-devel/2008-August/068125.html
Unfortunately half won't do. I've tested this on an XP of mine and it already took 8 minutes before it crashed. (I'm very interested in the results on Wine once most of the dplayx stuff gets implemented.)
- mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others) - shdocvw:webbrowser has 3 failures on servers with IE7.
Jacek has stated he doesn't have much time or interest in fixing these, so the plate calls again.
Time is a valid argument. No interest isn't. How can we be sure to implement the correct stuff if the conformance tests fail? (At some point in the not to far future we could only have IE7 boxes where the tests are run).
Folks, fixing the tests is a relatively easy task. It's also a great way to become better acquainted with the code base and the development process if you're new to working on Wine. For current developers, if you've written tests in the past, check out test.winehq.org every once in a while and see if any of your tests are failing. And yes, we do care about failures in win9x. The average number of test failures today is 64.7, which is down from 72.4 a month ago*. Let's get that down to zero.
+1 -- Cheers, Paul.
On 9/6/08, James Hawkins <truiken(a)gmail.com> wrote:
Tests with timeouts =================== - dplayx:dplayx fails with a timeout on all windows servers
According to Ismael, he has a few fixes that cut the time needed in half. Ismael, what's the status on this?
http://winehq.org/pipermail/wine-devel/2008-August/068125.html
I managed to cut the time (in 3/4, if I recall correctly) just changing a default timeout value in a function, but times were still bigger than desired (around 7 minutes for all tests, I think). Then I tried to run all tests concurrently, but it looks like dplay doesn't like concurrency too much, in my Windows XP I started seeing ~200 errors in tests that produce 0 errors when run secuentially. When running like only the 4 first tests at a time, it somtimes produced errors, sometimes didn't, and I'm afraid random results in tests are not really liked. This attempt to make the tests concurrent can be found at http://repo.or.cz/w/wine/gsoc_dplay.git I'll try to figure out how to solve this, but right now and until next week I'll be busy with exams, and even after that I won't have much time to spend on wine. If somebody wants to quickly get rid of those errors for now, it'd be okay to comment the test calls out in the START_TEST of tests/dplay.c, I'll re-enable them when I find a solution. About the errors in non-XP versions of Windows, I suppose I'll have to get and virtualize them to run the tests against them. Cheers Ismael
On Saturday 06 September 2008 22:21:15 James Hawkins wrote:
Tests that fail on Windows and succeed on Wine ============================================== - ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine
Last attempt got no comments:
http://winehq.org/pipermail/wine-patches/2008-May/054547.html
Actually, Alexandre commented on this on IRC. I don't really recall his comment, but he didn't like the approach. I'll have some time to look into this again tomorrow, but I remember I quit posting new patches to this as I was out of ideas on how to fix it. A couple of bug reports seem to be related to this, I need to dig around some more. Cheers, Kai -- Kai Blin WorldForge developer http://www.worldforge.org/ Wine developer http://wiki.winehq.org/KaiBlin Samba team member http://www.samba.org/samba/team/ -- Will code for cotton.
participants (4)
-
Ismael Barros -
James Hawkins -
Kai Blin -
Paul Vriens