Towards Zero Test Suite Failures
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 ) IMHO one of the hallmarks of 1.0 should be reliably getting zero test suite failures. That would make regressions stand out like sore thumbs instead of requiring developers to diff results before and after their changes, and might result in fewer regressions getting committed. How 'bout folks spend some time tracking the current six odd failures down and cleaning them up? - Dan -- Wine for Windows ISVs: http://kegel.com/wine/isv
On Jan 13, 2008 6:53 PM, Dan Kegel <dank(a)kegel.com> wrote:
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 )
IMHO one of the hallmarks of 1.0 should be reliably getting zero test suite failures. That would make regressions stand out like sore thumbs instead of requiring developers to diff results before and after their changes, and might result in fewer regressions getting committed.
How 'bout folks spend some time tracking the current six odd failures down and cleaning them up? - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
Sounds like a worthy goal IMO. It seems theres much more than 5 or 6 things which need work though: http://test.winehq.org/data/200801121000/ Perhaps we should try and have the nightly suite also pass all tests -- or is this too much of a frivolous goal? -- Zachary Goldberg Computer Science & Engineering Electrical Captain of Penn Electric Race Team School of Engineering at the University of Pennsylvania
On Jan 13, 2008 8:30 PM, Zachary Goldberg <zgold550(a)gmail.com> wrote:
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 ) ... How 'bout folks spend some time tracking the current six odd failures down and cleaning them up?
Sounds like a worthy goal IMO. It seems theres much more than 5 or 6 things which need work though: http://test.winehq.org/data/200801121000/
Not a pretty picture, is it? But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
Perhaps we should try and have the nightly suite also pass all tests -- or is this too much of a frivolous goal?
Getting all tests passing on Windows XP would be great, but it's a lot of work. It'd be less work if we focused on just the tests that are failing on *all* Win XP systems, but that's still at least 180 ok's that aren't ok. Let's at least fix the 5 or 10 that are failing on Wine, and try to keep them fixed. - Dan
Dan Kegel wrote:
On Jan 13, 2008 8:30 PM, Zachary Goldberg <zgold550(a)gmail.com> wrote:
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 ) ... How 'bout folks spend some time tracking the current six odd failures down and cleaning them up? Sounds like a worthy goal IMO. It seems theres much more than 5 or 6 things which need work though: http://test.winehq.org/data/200801121000/
Not a pretty picture, is it? But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
Are these tests passing (now) on Alexandre's machine? I thought he didn't apply patches that break the wine tests? (at least on his box).
Perhaps we should try and have the nightly suite also pass all tests -- or is this too much of a frivolous goal?
Getting all tests passing on Windows XP would be great, but it's a lot of work. It'd be less work if we focused on just the tests that are failing on *all* Win XP systems, but that's still
Why XP? Doesn't that mean our default should be XP as well?
at least 180 ok's that aren't ok. Let's at least fix the 5 or 10 that are failing on Wine, and try to keep them fixed. - Dan
-- Cheers, Paul.
[snipsnipsnip]
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 ) ... How 'bout folks spend some time tracking the current six odd failures down and cleaning them up?
Sounds like a worthy goal IMO. It seems theres much more than 5 or 6 things which need work though
Not a pretty picture, is it? But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
Are these tests passing (now) on Alexandre's machine? I thought he didn't apply patches that break the wine tests? (at least on his box).
A funny observation perhaps, but I doubt that Microsoft has actually employed such a stringent QA process when they developed the code whose bugs Wine now works around :) Cheers, Kuba
On 14/01/2008, Dan Kegel <dank(a)kegel.com> wrote:
On Jan 13, 2008 8:30 PM, Zachary Goldberg <zgold550(a)gmail.com> wrote:
On my machine, we've been hovering between five and ten test suite failures for some time (see http://bugs.winehq.org/show_bug.cgi?id=9916 ) ... How 'bout folks spend some time tracking the current six odd failures down and cleaning them up?
Sounds like a worthy goal IMO. It seems theres much more than 5 or 6 things which need work though: http://test.winehq.org/data/200801121000/
Not a pretty picture, is it? But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
But if you fix a test failure in Wine that is failing on Windows, then you are introducing a bug in Wine.
Perhaps we should try and have the nightly suite also pass all tests -- or is this too much of a frivolous goal?
Getting all tests passing on Windows XP would be great, but it's a lot of work. It'd be less work if we focused on just the tests that are failing on *all* Win XP systems, but that's still at least 180 ok's that aren't ok. Let's at least fix the 5 or 10 that are failing on Wine, and try to keep them fixed.
Ideally, all tests should pass on all platforms. However, there are issues with different service packs/hot fixes on each version of Windows, different user configuration (themes, UAC on/off, locale, etc.), different hardware and even different behaviour between different versions of Windows. I like the idea of getting the tests passing on Windows XP (SP2?) - that will then be the baseline for Wine. This should not cause regressions on other versions of Windows. Getting the tests passing on older verisons of Windows will be important for backward compatibility (e.g. if you have a program that only runs on Windows 3.1 or Windows 95), and will ensure that Wine does not break in those cases. - Reece
On Jan 14, 2008 5:41 AM, Reece Dunn <msclrhd(a)googlemail.com> wrote:
But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
But if you fix a test failure in Wine that is failing on Windows, then you are introducing a bug in Wine.
When fixing those 5 - 10 tests on Wine, one should also make sure they work on Windows.
I like the idea of getting the tests passing on Windows XP (SP2?) - that will then be the baseline for Wine. This should not cause regressions on other versions of Windows.
Yep. And at some point we should indeed make Windows XP the default personality in Wine. Seems like a 1.0 kind of thing. - Dan
On Jan 14, 2008 11:13 AM, Dan Kegel <dank(a)kegel.com> wrote:
On Jan 14, 2008 5:41 AM, Reece Dunn <msclrhd(a)googlemail.com> wrote:
But I was only referring to the tests that fail on Wine; there, we have control over both the test and the code, so if we can't get those tests passing, we're pretty weak :-)
But if you fix a test failure in Wine that is failing on Windows, then you are introducing a bug in Wine.
When fixing those 5 - 10 tests on Wine, one should also make sure they work on Windows.
I like the idea of getting the tests passing on Windows XP (SP2?) - that will then be the baseline for Wine. This should not cause regressions on other versions of Windows.
Yep. And at some point we should indeed make Windows XP the default personality in Wine. Seems like a 1.0 kind of thing. - Dan
Is there a reason there isn't a Linux column on that giant test.wine list? -- Zachary Goldberg Computer Science & Engineering Electrical Captain of Penn Electric Race Team School of Engineering at the University of Pennsylvania
Am Montag, 14. Januar 2008 14:41:35 schrieb Reece Dunn:
But if you fix a test failure in Wine that is failing on Windows, then you are introducing a bug in Wine. Not necessarily. Some tests are too strict in what they expect, and sometimes the Windows behavior is "wrong", in the way that some application expects something different and wouldn't run on this Windows installation either.
An example for a too strict test is a test that hardcodes paths that are specific to english locales(These cases should be fixed by now). Examples of "broken" Windows installations are VMware installations with Direct3D support enabled. VMWare has a D3D driver that works similarly to Wine, but has a few bugs that our tests stumble uppon.
On Mon, 14 Jan 2008, Stefan Dösinger wrote:
Am Montag, 14. Januar 2008 14:41:35 schrieb Reece Dunn:
But if you fix a test failure in Wine that is failing on Windows, then you are introducing a bug in Wine. Not necessarily. Some tests are too strict in what they expect, and sometimes the Windows behavior is "wrong", in the way that some application expects something different and wouldn't run on this Windows installation either.
But the only way to know is to check the Windows results. So while you're doing that you can as well try to fix the test on Windows. [...]
This Examples of "broken" Windows installations are VMware installations with Direct3D support enabled. VMWare has a D3D driver that works similarly to Wine, but has a few bugs that our tests stumble uppon.
<rant> This gets mentioned *every single time*. For what it's worth, as far as I know none of the tests I run (the 'fg-xxx' ones) has that D3D driver you speak of. They still have failures in the Direct3D tests but if you check the Windows XP results, you will see that my logs show only two Direct3D tests failing. That's very little compared to the 73 other tests that fail... for the Windows XP results alone. So I'd like people to focus a little bit less on these two Direct3D failures and a little bit more on the 73 other failures. That the tests are being run in VMware is no excuse for not looking at them. If these 73 failing tests were fixed that would be a huge step for the conformance test suite. </rant> -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ There are 10 types of people in the world... those who understand binary and those who don't.
On Jan 15, 2008 6:03 PM, Francois Gouget <fgouget(a)free.fr> wrote:
This Examples of "broken" Windows installations are VMware installations with Direct3D support enabled. VMWare has a D3D driver that works similarly to Wine, but has a few bugs that our tests stumble uppon.
<rant>
This gets mentioned *every single time*.
For what it's worth, as far as I know none of the tests I run (the 'fg-xxx' ones) has that D3D driver you speak of.
They still have failures in the Direct3D tests but if you check the Windows XP results, you will see that my logs show only two Direct3D tests failing. That's very little compared to the 73 other tests that fail... for the Windows XP results alone.
So I'd like people to focus a little bit less on these two Direct3D failures and a little bit more on the 73 other failures. That the tests are being run in VMware is no excuse for not looking at them. If these 73 failing tests were fixed that would be a huge step for the conformance test suite.
</rant>
+1 -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
participants (8)
-
Dan Kegel -
Francois Gouget -
Kuba Ober -
Paul Vriens -
Reece Dunn -
Stefan Dösinger -
Steven Edwards -
Zachary Goldberg