Wine regression testing: PIT_

[Molle Bestefich]

Regression testing with Wine is a pain in the butthole.

For very bad reasons: Stuff that is simple to fix, but haven't been.

I'd like to help improve that situation.


Example:
========

I want to find which patch ruins an application.
The application, according to one note, worked in 2003.
I decide to try and see how it really runs with the 2003 version of
Wine mentioned in the note.

I then spend several days pulling various releases, just trying to compile them.
They all fail miserably, because some fix required to compile
correctly with newer Linux versions is /missing/.

For example,
 - any Wine before 2004-01-02 won't work because it won't compile
against newer ALSA versions.
 - any Wine before 2003-12-01 needs patches to the build system before
'make depend' will even run.

I've been at it for a week and I've managed to fix the above two.
_They're really simple things, but they take a lot of time to figure out._


Currently I'm working with two other problems:
 - Wine fails to correctly invoke the debugger. Winedbg shows 'Usage:
blah blah'.  Seems 'wine' calls it with incorrect parameters.
 - Wine fails to start any application, complaining about 'kthreads'
something or the other.

I imagine that if I spend a few more weeks on the above problems, I
can find patches in Wine CVS that fixes the exact problems I'm seeing.

I've already wasted a week of my time on problems that someone else
already fixed.  Spending two or more weeks doing this is not exactly
on my fun-things-to-do-before-I-die list.  I'd like to see a general
solution to this entire problem.



What can be done to alleviate?
==============================

Once I find a patch which makes an earlier version actually compile,
I'd like to propose that for a backport.

If approved/applied, the next person that checks out the release would
check out a version containing the backported patches.

That would fix the problem and allow people to do proper regression
testing, so developers and QA people can spend their time doing
development and QA instead of redoing already done effort trying to
get an older version to run.

I'm not sure how this would work with CVS.  I'm from an SVN world
where tagging is a 0-cost tree-copy operation, and any patches can be
applied to the tag with history and all showing that the tag's been
modified.  If it can't work with CVS, perhaps the tags can be created
in Troy's SVN repo and patches applied there?  Not sure.  Suggestions,
please, people.

[Mike McCormack]

> I then spend several days pulling various releases, just trying to compile them.
> They all fail miserably, because some fix required to compile
> correctly with newer Linux versions is /missing/.
>
> For example,
>  - any Wine before 2004-01-02 won't work because it won't compile
> against newer ALSA versions.
>  - any Wine before 2003-12-01 needs patches to the build system before
> 'make depend' will even run.

To do a valid regression test, you really want to run it on a system 
that worked in 2003, so you probably should be compiling it and running 
it on Slackware 9.0 or rather than Slackware 10.1, or whatever system 
you were using back then.

For example, if you tried to use Wine from a couple of years ago on the 
latest Fedora Core, even if you get it to compile, it won't run, because 
of problems with exec-shield and who knows what else.

Mike

Very good points. It seems a lot of this has been accepted over the years  
on the basis that wine was alpha software and in that context , the past  
is the past - get the new release etc.

It seems that the result is , as you say, regression testing is a PITA and  
as a result often gets skipped. The code base has certainly advanced a lot  
but every release seems to create as many problems as it solves.

I managed to get an app working in Feb, luckily I tarballed the whole  
.wine installation since I am now unable to reinstall this app under any  
version of wine and make it work.

Wine is now officially beta and this is probably the stage at which some  
serious project management has to come into play.

Maybe someone has to take a week off from coding and review the overall  
picture. The core devs have a huge task so it must be made easier for  
people like yourself who clearly have the expertise to contribute to get  
involved.


A working wine installation seems to have the life expectancy of a  
butterfly. If any wider interest is to be generated: like people who  
actually want to use the code rather than rewrite it , it has to stabalise  
and be done more rigourously (for example documentation syncronisation -  
last time I looked the man page was still telling me how to setup the  
config file, and winever simply reports CVS , irrespective of what build I  
install from.). Hopefully beta is where this will start to happen.

Now is a crucial time to Linux , it has reached take-off velocity.

If wine is to contribute to the migration path, and there is a the  
opertunity for a great contribution, it seems to me this has to be  
adressed now.

I dont mean to imply nothing is being done behind the scenes, I'm sure you  
guys are already working on these issues.

Hopefully 0.9 will be a turning point for Wine . Many thanks for the work  
that has gone into getting this far.

Best regards.


On Wed, 26 Oct 2005 12:04:46 +0200, Molle Bestefich  
<molle.bestefich(a)gmail.com> wrote:
> Regression testing with Wine is a pain in the butthole.
>
> For very bad reasons: Stuff that is simple to fix, but haven't been.
>
> I'd like to help improve that situation.
>
> Example:
> ========
>
> I want to find which patch ruins an application.
> The application, according to one note, worked in 2003.
> I decide to try and see how it really runs with the 2003 version of
> Wine mentioned in the note.
>
> I then spend several days pulling various releases, just trying to  
> compile them.
> They all fail miserably, because some fix required to compile
> correctly with newer Linux versions is /missing/.
>
> For example,
>  - any Wine before 2004-01-02 won't work because it won't compile
> against newer ALSA versions.
>  - any Wine before 2003-12-01 needs patches to the build system before
> 'make depend' will even run.
>
> I've been at it for a week and I've managed to fix the above two.
> _They're really simple things, but they take a lot of time to figure  
> out._
>
> Currently I'm working with two other problems:
>  - Wine fails to correctly invoke the debugger. Winedbg shows 'Usage:
> blah blah'.  Seems 'wine' calls it with incorrect parameters.
>  - Wine fails to start any application, complaining about 'kthreads'
> something or the other.
>
> I imagine that if I spend a few more weeks on the above problems, I
> can find patches in Wine CVS that fixes the exact problems I'm seeing.
>
> I've already wasted a week of my time on problems that someone else
> already fixed.  Spending two or more weeks doing this is not exactly
> on my fun-things-to-do-before-I-die list.  I'd like to see a general
> solution to this entire problem.
>
> What can be done to alleviate?
> ==============================
>
> Once I find a patch which makes an earlier version actually compile,
> I'd like to propose that for a backport.
>
> If approved/applied, the next person that checks out the release would
> check out a version containing the backported patches.
>
> That would fix the problem and allow people to do proper regression
> testing, so developers and QA people can spend their time doing
> development and QA instead of redoing already done effort trying to
> get an older version to run.
>
> I'm not sure how this would work with CVS.  I'm from an SVN world
> where tagging is a 0-cost tree-copy operation, and any patches can be
> applied to the tag with history and all showing that the tag's been
> modified.  If it can't work with CVS, perhaps the tags can be created
> in Troy's SVN repo and patches applied there?  Not sure.  Suggestions,
> please, people.

[Marcus Meissner]

On Wed, Oct 26, 2005 at 01:04:30PM +0200, wino(a)piments.com wrote:
> Very good points. It seems a lot of this has been accepted over the years  
> on the basis that wine was alpha software and in that context , the past  
> is the past - get the new release etc.
>
> It seems that the result is , as you say, regression testing is a PITA and  
> as a result often gets skipped. The code base has certainly advanced a lot  
> but every release seems to create as many problems as it solves.
>
> I managed to get an app working in Feb, luckily I tarballed the whole  
> .wine installation since I am now unable to reinstall this app under any  
> version of wine and make it work.
>
> Wine is now officially beta and this is probably the stage at which some  
> serious project management has to come into play.
>
> Maybe someone has to take a week off from coding and review the overall  
> picture. The core devs have a huge task so it must be made easier for  
> people like yourself who clearly have the expertise to contribute to get  
> involved.

We adapt to the upstream packages. We cannot control their changes.

Like alsa using "interface" as keyword in their headers.
Or glibc changing.

Ciao, Marcus

[Molle Bestefich]

Mike McCormack wrote:
> To do a valid regression test, you really want to run it on a system
> that worked in 2003, so you probably should be compiling it and running
> it on Slackware 9.0 or rather than Slackware 10.1, or whatever system
> you were using back then.

Probably.  I don't require 100% accuracy however, so I won't be doing that.

It's obviously too much of a hassle to go find out which operating
systems were in use back then and install those on my PC just because
I really "should" in order to get picture perfect results, isn't it?
99.9% of the time, it's stuff in Wine that's broken, not stuff in Linux, right?
> For example, if you tried to use Wine from a couple of years ago on the
> latest Fedora Core, even if you get it to compile, it won't run, because
> of problems with exec-shield and who knows what else.

That's exactly what should be fixed.
Stuff that prevents older versions from running *at all*.

You know where the problems are;
I know where *some* of them are;
I'm willing to do volunteer work to rectify the situation for everyone
else that wants to do Wine regression testing, but need cooperation
from you guys before I can do any good.


wino(a)piments.com wrote:
> It seems that the result is, as you say, regression testing is a PITA and
> as a result often gets skipped.

I see that we agree :-)


Marcus Meissner wrote:
> We adapt to the upstream packages. We cannot control their changes.
>
> Like alsa using "interface" as keyword in their headers.
> Or glibc changing.

No, but we can adapt to them.

And if we are serious about wanting people to do regression testing
and finding the patches that breaks things, we SHOULD adapt to them.

It seems that you and others have lots of examples of things that
break Wine but is not Wine's fault.
Let's get those things *fixed* in older release tarballs, so that
people can do regression testing.

We actually /ask/ people to do regression testing, here f.x.:
http://www.winehq.org/site/docs/winedev-guide/x1344
to find a patch that breaks things.  Some developers has probably
recommended that approach on the mailing lists too, I know I have.

The approach is useless however, until these simple fixes are applied
to the tarballs (preferably through the versioning system).

[Jonathan Ernst]

Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
> The approach is useless however, until these simple fixes are applied
> to the tarballs (preferably through the versioning system).

How do you decide what things have to be fixed in old tarballs ? Do you
test each old tarballs every now and then and apply the corresponding
patches to them ?

What if patching the old tarball cause other regressions ?

What if someone wants to use an old tarball in an old system that would
be broken because of this patching ? You would have to maintain a lot of
old tarballs version...

I don't think it is feasible.

-- 
Jonathan Ernst <Jonathan(a)ErnstFamily.ch>

[Saulius Krasuckas]

* On Wed, 26 Oct 2005, Jonathan Ernst wrote:
> * Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
> > The approach is useless however, until these simple fixes are applied 
> > to the tarballs (preferably through the versioning system).
>
> How do you decide what things have to be fixed in old tarballs ? Do you
> test each old tarballs every now and then and apply the corresponding
> patches to them ?
>
> What if patching the old tarball cause other regressions ?

If that will be noted on wine-user the mailing list, it could be fixed.
> What if someone wants to use an old tarball in an old system that would 
> be broken because of this patching ?

Then that person won't use patching.  Actually, using Molle tactics mean 
to patch tarball only when the problems arise.  Such script even may use 
some database to collect patches for an old distros and a variety of a 
kernels.  

Plus it can be a reminder of when user/admin should do configuration using 
winecfg, via config file or by editing ~/.wine/system.reg .  Do you think 
I still remember, on what year and what month winecfg was introduced?  No 
way.  Such stuff should go into some databased feature, IMHO.

That would be like a two or more levels of ./configure. :-)
> You would have to maintain a lot of old tarballs version...
> I don't think it is feasible.

If such patcher-system/script isn't acceptable in the Wine tree, then it 
can go as a separate project, named for example Wine2Old, OldWine, etc. 
:-P

[Jonathan Ernst]

Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
> We actually /ask/ people to do regression testing, here f.x.:
> http://www.winehq.org/site/docs/winedev-guide/x1344
> to find a patch that breaks things.  Some developers has probably
> recommended that approach on the mailing lists too, I know I have.
>
> The approach is useless however, until these simple fixes are applied
> to the tarballs (preferably through the versioning system).

I just got a new idea:

Instead of appliying new patches to old versions which is unmaintainable
imho, we could have VMWare installs of Wine and people could download
the VMWare image of any Wine release and play it for free using the
VMWare player !

I guess the download size will be quite big, but a small distro with
only X, Wine, Gnome and KDE and required dependencies for example could
fit on half a CD I guess.

[Molle Bestefich]

Jonathan Ernst wrote:
> > The approach is useless however, until these simple fixes are applied
> > to the tarballs (preferably through the versioning system).
>
> How do you decide what things have to be fixed in old tarballs ? Do you
> test each old tarballs every now and then and apply the corresponding
> patches to them ?

Sure, I wouldn't mind checking out alpha releases from the past
year or so and testing if they compile.
> What if someone wants to use an old tarball in an old system that would
> be broken because of this patching ? You would have to maintain a lot of
> old tarballs version...

They'd check out the original tagged version.
> I don't think it is feasible.

I'll show you.

The KDE project tagged version 3.4.9.2 on 2005-10-11.
Since then they've spent two days doing cleanups and merging hotfixes
to that tag.

Too see what has happened to the 3.4.9.2 tag (only), do:
# svn log --stop-on-copy svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/

You'll see that the tagging operation is r469461 and the hotfixes etc.
are ensuing numbers.

To check out the HEAD of the tag, including hotfixes:
# svn co svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/ my_kde

To check out the original tag, no fixes:
# svn co -r469461 svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/ my_kde

Easy :-).

This is just an example to show it can be done all while keeping the
source in the repository (where it belongs, IMHO) and making checkout
of both the original tag and the fixed version as easy as sunday
morning.

I know that SVN doesn't fly around here, so please don't hit me on the
head for that one - it's just that I have no experience with CVS..


gslink wrote:
> This is the only solution that has worked in the past.  It requires a
> test suite be built that can easily test Windows, not Wine, and it
> requires that the tests be run before the next version comes out.

Not realistic.  Building a test suite will take years.
I disagree that it's the only solution that will work.



Saulius Krasuckas wrote:
> Then that person won't use patching.  Actually, using Molle tactics mean
> to patch tarball only when the problems arise.  Such script even may use
> some database to collect patches for an old distros and a variety of a
> kernels.

Right.  Except I'm still hoping that an entire new system isn't needed
for this.  Although it would obviously be more flexible, it would
probably be overkill?
> If such patcher-system/script isn't acceptable in the Wine tree, then it
> can go as a separate project, named for example Wine2Old, OldWine, etc.

Feasible albeit not as pretty.  And vastly more time consuming.



Lionel Ulmer wrote:
> > The application, according to one note, worked in 2003.
>
> The question now is: why was this application not tested more often to get
> the exact date of regression ?

I'm guessing that most people give up before I do =).

Or maybe they just haven't the time.  Which is where I'd like to help
regarding this thread..



Jonathan Ernst wrote:
> I just got a new idea:
>
> Instead of appliying new patches to old versions which is unmaintainable
> imho,

Bah :-).  Perhaps with CVS.  I don't think so but I don't really know.
> we could have VMWare installs of Wine and people could download
> the VMWare image of any Wine release and play it for free using the
> VMWare player !

Excellent idea!!
> I guess the download size will be quite big, but a small distro with
> only X, Wine, Gnome and KDE and required dependencies for example could
> fit on half a CD I guess.

Do you know of any miniature distro(s) that carry these and
(preferably) existed all along with Wine?  If you have suggestions,
I'd like to give it a shot right away.

We'd need a web page somewhere telling people what image works with
which Wine alphas/betas.

[Jonathan Ernst]

Le vendredi 28 octobre 2005 à 06:21 +0000, Molle Bestefich a écrit :
> Jonathan Ernst wrote:
> > we could have VMWare installs of Wine and people could download
> > the VMWare image of any Wine release and play it for free using the
> > VMWare player !
>
> Excellent idea!!
> > I guess the download size will be quite big, but a small distro with
> > only X, Wine, Gnome and KDE and required dependencies for example could
> > fit on half a CD I guess.
>
> Do you know of any miniature distro(s) that carry these and
> (preferably) existed all along with Wine?  If you have suggestions,
> I'd like to give it a shot right away.

If you remove the word processor I think you'll be under 30-40MB
uncompressed with these ones:

http://www.damnsmalllinux.org/
http://www.zenwalk.org/staticpages/index.php?page=20050321031055330

I didn't find a ready to use minidistro with gnome or/and kde as these
applications are really big. You could also try something like ubuntu
and install only the minimum reqs of Wine.

Thanks for your efforts; looking forward to testing your VMWare
machines !
-- 
Jonathan Ernst <Jonathan(a)ErnstFamily.ch>

[Molle Bestefich]

Jonathan Ernst wrote:
> If you remove the word processor I think you'll be under 30-40MB
> uncompressed with these ones:
>
> http://www.damnsmalllinux.org/
> http://www.zenwalk.org/staticpages/index.php?page=20050321031055330
>
> I didn't find a ready to use minidistro with gnome or/and kde as these
> applications are really big. You could also try something like ubuntu
> and install only the minimum reqs of Wine.
>
> Thanks for your efforts; looking forward to testing your VMWare
> machines !

For the record, I still think it's a better idea to make older
releases compileable by applying the minimalistic patches required. 
It has several advantages:

a.) Effort can easily be shared by multiple volunteers by simply
identifying individual patches needed to compile correctly.
b.) Patches are usually required not only by one release, but by one
release _and_ all prior releases.  Lots of prior releases can be fixed
at the same time, once a culprit has been identified.
c.) Hotfixing requires a lot less effort than the virtual machine approach.

Depending on how much effort the VM thingy is, I might only make the
one or two VMs that I need for my testing, and just modify th{at/ose}
as I go along.

[Saulius Krasuckas]

* On Thu, 27 Oct 2005, Jonathan Ernst wrote:
> * Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
> > We actually /ask/ people to do regression testing, here f.x.: 
> > http://www.winehq.org/site/docs/winedev-guide/x1344 to find a patch 
> > that breaks things.  Some developers has probably recommended that 
> > approach on the mailing lists too, I know I have.
> >
> > The approach is useless however, until these simple fixes are applied 
> > to the tarballs (preferably through the versioning system).
>
> Instead of appliying new patches to old versions which is unmaintainable 
> imho, we could have VMWare installs of Wine and people could download 
> the VMWare image of any Wine release and play it for free using the 
> VMWare player !
>
> I guess the download size will be quite big, but a small distro with 
> only X, Wine, Gnome and KDE and required dependencies for example could 
> fit on half a CD I guess.

One more (nice) way could be to run all the tested stuff (inside VMWare 
player?) on some remote X (?) server, so people should not download 
several large ISOs to test small software (instead they would upload small 
test-software packages into the server).  But I start feeling this would 
require not a less amount of bandwidth too, so YMMV. :)

[Lionel Ulmer]

On Wed, Oct 26, 2005 at 10:04:46AM +0000, Molle Bestefich wrote:
> I want to find which patch ruins an application.
> The application, according to one note, worked in 2003.
> I decide to try and see how it really runs with the 2003 version of
> Wine mentioned in the note.

The question now is: why was this application not tested more often to get
the exact date of regression ?

       Lionel

-- 
		 Lionel Ulmer - http://www.bbrox.org/