Sufficient 1.2 release criterion: passing all tests on all platforms?
Even without any new features, it seems to me that passing all tests on all platforms might all on its own merit a new stable release. That said, by the time we have that, we might well have 64 bit support working, too... - Dan
On Sun, Mar 8, 2009 at 4:43 PM, Dan Kegel <dank(a)kegel.com> wrote:
Even without any new features, it seems to me that passing all tests on all platforms might all on its own merit a new stable release.
Grouping platforms by age: 2000 and earlier have 75 rows with red or mixed, XP/2003/Vista/2008 have 19 rows with red or mixed. Given how most apps for the last while have been targeted at XP or Vista, it's quite tempting to say that even fixing just those 19 rows, i.e. passing all tests on XP, 2003, Vista, and Windows 2008 (and Wine), all 32 bits, would merit a new stable release. It almost feels within our grasp for midyear... how 'bout it? - Dan
Dan Kegel <dank(a)kegel.com> writes:
On Sun, Mar 8, 2009 at 4:43 PM, Dan Kegel <dank(a)kegel.com> wrote:
Even without any new features, it seems to me that passing all tests on all platforms might all on its own merit a new stable release.
Grouping platforms by age: 2000 and earlier have 75 rows with red or mixed, XP/2003/Vista/2008 have 19 rows with red or mixed.
Given how most apps for the last while have been targeted at XP or Vista, it's quite tempting to say that even fixing just those 19 rows, i.e. passing all tests on XP, 2003, Vista, and Windows 2008 (and Wine), all 32 bits, would merit a new stable release.
I don't think tests passing on Windows is a reason for a release, it has very little impact on the Wine code. In the vast majority of cases these are tests that already succeed on Wine and on some Windows versions, so fixing them only involves making them less strict. That's not something users care about. You could certainly make the argument that failing tests should block the release, but that doesn't imply that succeeding tests justify a release all by themselves. -- Alexandre Julliard julliard(a)winehq.org
On Sun, Mar 8, 2009 at 6:08 PM, Alexandre Julliard <julliard(a)winehq.org> wrote:
I don't think tests passing on Windows is a reason for a release, it has very little impact on the Wine code. In the vast majority of cases these are tests that already succeed on Wine and on some Windows versions, so fixing them only involves making them less strict. That's not something users care about.
You could certainly make the argument that failing tests should block the release, but that doesn't imply that succeeding tests justify a release all by themselves.
I've been itching to do another release for a while, since what we have now is a lot better than 1.0. Your position has been that what's blocking release is the lack of a new feature (you listed several, any of which you felt would suffice). How do you feel now? Are we close to having any of those features, and/or are you willing to consider dropping that requirement? - Dan
Dan Kegel <dank(a)kegel.com> writes:
I've been itching to do another release for a while, since what we have now is a lot better than 1.0. Your position has been that what's blocking release is the lack of a new feature (you listed several, any of which you felt would suffice).
How do you feel now? Are we close to having any of those features, and/or are you willing to consider dropping that requirement?
64-bit support isn't too far away, so if we put some more effort into it that should be achievable in the near future. There are a number of other issues that need some more time to mature, like the OpenGL memory thing. It seems we could reasonably start the release process 3 months from now. Of course that would put code freeze right in the middle of the Summer of Code... -- Alexandre Julliard julliard(a)winehq.org
Alexandre Julliard wrote:
Dan Kegel <dank(a)kegel.com> writes:
I've been itching to do another release for a while, since what we have now is a lot better than 1.0. Your position has been that what's blocking release is the lack of a new feature (you listed several, any of which you felt would suffice).
How do you feel now? Are we close to having any of those features, and/or are you willing to consider dropping that requirement?
64-bit support isn't too far away, so if we put some more effort into it that should be achievable in the near future. There are a number of other issues that need some more time to mature, like the OpenGL memory thing. It seems we could reasonably start the release process 3 months from now. Of course that would put code freeze right in the middle of the Summer of Code...
Starting the release process three months from now would be a really good thing. It would put us just in time for the next wave of distro releases (Ubuntu 9.10 among them), which would get 1.2 to millions of new desktops. As it stands, only 135,150 downloaded Wine 1.1.15 from the apt repository, so about 90% of our users are still on 1.0.1. I'm not too worried about tabling this year's summer of code until the next release, in part because we already have the previous summer of codes' work waiting to be released as it is. You're obviously right about requiring all tests to pass before making a release. I'll add that, in theory, if we had "enough" tests, this would prevent regressions entirely. I do have one question though: do we mean regressions relative to any beta Wine, or just regressions relative to 1.0.1? I prefer the less strict approach if it means more frequent releases, but I'm not sure it matters at this point. Thanks, Scott Ritchie
On Sun, Mar 8, 2009 at 8:08 PM, Scott Ritchie <scott(a)open-vote.org> wrote:
I do have one question though: do we mean regressions relative to any beta Wine, or just regressions relative to 1.0.1? I prefer the less strict approach if it means more frequent releases
I think the users will expect 1.2 to have zero regressions relative to 1.0.
2009/3/9 Scott Ritchie <scott(a)open-vote.org>:
Starting the release process three months from now would be a really good thing. It would put us just in time for the next wave of distro releases (Ubuntu 9.10 among them), which would get 1.2 to millions of new desktops. As it stands, only 135,150 downloaded Wine 1.1.15 from the apt repository, so about 90% of our users are still on 1.0.1.
Where are you getting your stats on Wine + Ubuntu users from?
Ben Klein wrote:
2009/3/9 Scott Ritchie <scott(a)open-vote.org>:
Starting the release process three months from now would be a really good thing. It would put us just in time for the next wave of distro releases (Ubuntu 9.10 among them), which would get 1.2 to millions of new desktops. As it stands, only 135,150 downloaded Wine 1.1.15 from the apt repository, so about 90% of our users are still on 1.0.1.
Where are you getting your stats on Wine + Ubuntu users from?
From my own web server statistics (for wine.budgetdedicated.com) and from the earlier estimate of users (eg popcon.ubuntu.com saying we have a large portion of Ubuntu users installing Wine)
Thanks, Scott Ritchie
On Sun, Mar 8, 2009 at 7:40 PM, Alexandre Julliard <julliard(a)winehq.org> wrote:
64-bit support isn't too far away, so if we put some more effort into it that should be achievable in the near future.... It seems we could reasonably start the release process 3 months from now.
That would be sweet. I wonder if this time around it'll go faster, what with all our improved test-foo and all.
Of course that would put code freeze right in the middle of the Summer of Code...
C'est la vie. Summer of Coders can use a branch if need be. - Dan
On Sun, Mar 8, 2009 at 9:43 AM, Dan Kegel <dank(a)kegel.com> wrote:
Even without any new features, it seems to me that passing all tests on all platforms might all on its own merit a new stable release.
By 'all platforms', do you mean all Windows versions, or Linux/OS X/BSD/Solaris? -- -Austin
On Sun, Mar 8, 2009 at 9:27 PM, Austin English <austinenglish(a)gmail.com> wrote:
Even without any new features, it seems to me that passing all tests on all platforms might all on its own merit a new stable release.
By 'all platforms', do you mean all Windows versions, or Linux/OS X/BSD/Solaris?
I meant Windows. But now that you ask, we do have a lot of platforms to consider. We simply can't provide the same level of support for them all. The gcc project defines three tiers of support. If we did that, it might look like this: We would define tiers for Windows conformance test validation, CPUs, and host operating systems, and maybe graphics cards. 1st tier: we run tests regularly, and all tests must pass for release. 2nd tier: we might run tests occasionally or regularly, but we will tolerate some failures. 3rd tier: we won't test ourselves, and will tolerate failures, but will accept bugfixes from advocates. Here's one possible set of definitions: For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit 2nd tier: Win XP 16 bit, Win 95, Win 98, Win ME, Win 7 32 and 64 bit 3rd tier: Win 3.1, DOS For CPUs: 1st tier: whatever our developers use, but mostly < 2 year old Intel and AMD chips, running apps in all three modes, 16, 32, and 64 bit (as supported by hw) 2nd tier: none 3rd tier: power pc, sparc, other less-common pentium-compatible chips For host OS: 1st tier: Linux 2nd tier: Mac OS X 3rd tier: Solaris, FreeBSD For graphics cards: 1st tier: Nvidia 8400 or higher 2nd tier: < 4 year old Nvidia, < 2 year old ATI, < 2 year old Intel 3rd tier: older nvidia I'm sure that will need adjusting, but it's a good starting point for discussion. - Dan
On Sun, Mar 8, 2009 at 4:02 PM, Dan Kegel <dank(a)kegel.com> wrote:
But now that you ask, we do have a lot of platforms to consider. We simply can't provide the same level of support for them all. The gcc project defines three tiers of support. If we did that, it might look like this: We would define tiers for Windows conformance test validation, CPUs, and host operating systems, and maybe graphics cards. 1st tier: we run tests regularly, and all tests must pass for release. 2nd tier: we might run tests occasionally or regularly, but we will tolerate some failures. 3rd tier: we won't test ourselves, and will tolerate failures, but will accept bugfixes from advocates.
+1
Here's one possible set of definitions:
For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit 2nd tier: Win XP 16 bit, Win 95, Win 98, Win ME, Win 7 32 and 64 bit 3rd tier: Win 3.1, DOS
Not sure exactly what you mean by Win XP 16-bit? The Win16 test suite on XP?
For CPUs: 1st tier: whatever our developers use, but mostly < 2 year old Intel and AMD chips, running apps in all three modes, 16, 32, and 64 bit (as supported by hw) 2nd tier: none 3rd tier: power pc, sparc, other less-common pentium-compatible chips
No argument there. Perhaps move 64-bit to 2nd tier, and move it up to 1st once we've got better support for it.
For host OS: 1st tier: Linux 2nd tier: Mac OS X 3rd tier: Solaris, FreeBSD
Having tested these often, I'd say OS X is more broken than FreeBSD. I'd swap those two around to be honest. Solaris/OpenBSD/NetBSD are tier 3 though. -- -Austin
On Sun, Mar 8, 2009 at 4:17 PM, Austin English <austinenglish(a)gmail.com> wrote:
On Sun, Mar 8, 2009 at 4:02 PM, Dan Kegel <dank(a)kegel.com> wrote:
But now that you ask, we do have a lot of platforms to consider. We simply can't provide the same level of support for them all. The gcc project defines three tiers of support. If we did that, it might look like this: We would define tiers for Windows conformance test validation, CPUs, and host operating systems, and maybe graphics cards. 1st tier: we run tests regularly, and all tests must pass for release. 2nd tier: we might run tests occasionally or regularly, but we will tolerate some failures. 3rd tier: we won't test ourselves, and will tolerate failures, but will accept bugfixes from advocates.
+1
Here's one possible set of definitions:
For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit 2nd tier: Win XP 16 bit, Win 95, Win 98, Win ME, Win 7 32 and 64 bit 3rd tier: Win 3.1, DOS
Not sure exactly what you mean by Win XP 16-bit? The Win16 test suite on XP?
For CPUs: 1st tier: whatever our developers use, but mostly < 2 year old Intel and AMD chips, running apps in all three modes, 16, 32, and 64 bit (as supported by hw) 2nd tier: none 3rd tier: power pc, sparc, other less-common pentium-compatible chips
No argument there. Perhaps move 64-bit to 2nd tier, and move it up to 1st once we've got better support for it.
For host OS: 1st tier: Linux 2nd tier: Mac OS X 3rd tier: Solaris, FreeBSD
Having tested these often, I'd say OS X is more broken than FreeBSD. I'd swap those two around to be honest. Solaris/OpenBSD/NetBSD are tier 3 though.
-- -Austin
Also might add in WINEDEBUG paramaters, e.g., make +heap a 1st/2nd tier. -- -Austin
On Sun, Mar 8, 2009 at 11:17 PM, Austin English <austinenglish(a)gmail.com> wrote:
For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit 2nd tier: Win XP 16 bit, Win 95, Win 98, Win ME, Win 7 32 and 64 bit 3rd tier: Win 3.1, DOS
Not sure exactly what you mean by Win XP 16-bit? The Win16 test suite on XP?
Right. (I forgot to specify, but I would not run the 16 bit tests on any other version of windows except perhaps tier 3's win 3.1. Win XP is probably enough.)
For CPUs: 1st tier: whatever our developers use, but mostly < 2 year old Intel and AMD chips, running apps in all three modes, 16, 32, and 64 bit (as supported by hw) 2nd tier: none 3rd tier: power pc, sparc, other less-common pentium-compatible chips
No argument there. Perhaps move 64-bit to 2nd tier, and move it up to 1st once we've got better support for it.
We'll see how Alexandre feels about it.
For host OS: 1st tier: Linux 2nd tier: Mac OS X 3rd tier: Solaris, FreeBSD
Having tested these often, I'd say OS X is more broken than FreeBSD. I'd swap those two around to be honest. Solaris/OpenBSD/NetBSD are tier 3 though.
You're probably right, but I'll bet most wine developers would probably leave FreeBSD in tier 3. So maybe tier 2 will be empty for this release; it depends on how strong the support is from FreeBSD advocates. Also, about +heap: that's probably just a plain old release criterion for tier one host os platforms. That and Valgrind. - Dan
Dan Kegel <dank(a)kegel.com> writes:
Here's one possible set of definitions:
For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit
Having tests pass on all these platforms is of course a worthwhile goal, but it can't be made a requirement for release unless we want it to take another 3 years... For release I think it's enough to have tests pass on Wine and on at least one Windows box. -- Alexandre Julliard julliard(a)winehq.org
On Mon, Mar 9, 2009 at 7:37 AM, Alexandre Julliard <julliard(a)winehq.org> wrote:
For Windows conformance test validation: 1st tier: Win XP 32 bit, Win 2003 32 bit, Win Vista 32 and 64 bit, Win 2008 32 bit
Having tests pass on all these platforms is of course a worthwhile goal, but it can't be made a requirement for release unless we want it to take another 3 years... For release I think it's enough to have tests pass on Wine and on at least one Windows box.
We're already passing fairly often on one XP box, so that should be nearly trivial. So that leaves us with: For Windows conformance test validation: 1st tier: Win XP 32 bit 2nd tier: 16 bit: Win XP 32 bit: Win 95/98/ME, Win 2003, Vista, Win 2008, Win 7 64 bit: Vista, Win 7 3rd tier: Win 3.1, DOS with the understanding that we don't have to pass on all machines (though passing on more than one sure would be nice). - Dan
participants (5)
-
Alexandre Julliard -
Austin English -
Ben Klein -
Dan Kegel -
Scott Ritchie