Regression testing with Wine is a pain in the butthole.
For very bad reasons: Stuff that is simple to fix, but haven't been.
I'd like to help improve that situation.
Example: ========
I want to find which patch ruins an application. The application, according to one note, worked in 2003. I decide to try and see how it really runs with the 2003 version of Wine mentioned in the note.
I then spend several days pulling various releases, just trying to compile them. They all fail miserably, because some fix required to compile correctly with newer Linux versions is /missing/.
For example, - any Wine before 2004-01-02 won't work because it won't compile against newer ALSA versions. - any Wine before 2003-12-01 needs patches to the build system before 'make depend' will even run.
I've been at it for a week and I've managed to fix the above two. _They're really simple things, but they take a lot of time to figure out._
Currently I'm working with two other problems: - Wine fails to correctly invoke the debugger. Winedbg shows 'Usage: blah blah'. Seems 'wine' calls it with incorrect parameters. - Wine fails to start any application, complaining about 'kthreads' something or the other.
I imagine that if I spend a few more weeks on the above problems, I can find patches in Wine CVS that fixes the exact problems I'm seeing.
I've already wasted a week of my time on problems that someone else already fixed. Spending two or more weeks doing this is not exactly on my fun-things-to-do-before-I-die list. I'd like to see a general solution to this entire problem.
What can be done to alleviate? ==============================
Once I find a patch which makes an earlier version actually compile, I'd like to propose that for a backport.
If approved/applied, the next person that checks out the release would check out a version containing the backported patches.
That would fix the problem and allow people to do proper regression testing, so developers and QA people can spend their time doing development and QA instead of redoing already done effort trying to get an older version to run.
I'm not sure how this would work with CVS. I'm from an SVN world where tagging is a 0-cost tree-copy operation, and any patches can be applied to the tag with history and all showing that the tag's been modified. If it can't work with CVS, perhaps the tags can be created in Troy's SVN repo and patches applied there? Not sure. Suggestions, please, people.
I then spend several days pulling various releases, just trying to compile them. They all fail miserably, because some fix required to compile correctly with newer Linux versions is /missing/.
For example,
- any Wine before 2004-01-02 won't work because it won't compile
against newer ALSA versions.
- any Wine before 2003-12-01 needs patches to the build system before
'make depend' will even run.
To do a valid regression test, you really want to run it on a system that worked in 2003, so you probably should be compiling it and running it on Slackware 9.0 or rather than Slackware 10.1, or whatever system you were using back then.
For example, if you tried to use Wine from a couple of years ago on the latest Fedora Core, even if you get it to compile, it won't run, because of problems with exec-shield and who knows what else.
Mike
Very good points. It seems a lot of this has been accepted over the years on the basis that wine was alpha software and in that context , the past is the past - get the new release etc.
It seems that the result is , as you say, regression testing is a PITA and as a result often gets skipped. The code base has certainly advanced a lot but every release seems to create as many problems as it solves.
I managed to get an app working in Feb, luckily I tarballed the whole .wine installation since I am now unable to reinstall this app under any version of wine and make it work.
Wine is now officially beta and this is probably the stage at which some serious project management has to come into play.
Maybe someone has to take a week off from coding and review the overall picture. The core devs have a huge task so it must be made easier for people like yourself who clearly have the expertise to contribute to get involved.
A working wine installation seems to have the life expectancy of a butterfly. If any wider interest is to be generated: like people who actually want to use the code rather than rewrite it , it has to stabalise and be done more rigourously (for example documentation syncronisation - last time I looked the man page was still telling me how to setup the config file, and winever simply reports CVS , irrespective of what build I install from.). Hopefully beta is where this will start to happen.
Now is a crucial time to Linux , it has reached take-off velocity.
If wine is to contribute to the migration path, and there is a the opertunity for a great contribution, it seems to me this has to be adressed now.
I dont mean to imply nothing is being done behind the scenes, I'm sure you guys are already working on these issues.
Hopefully 0.9 will be a turning point for Wine . Many thanks for the work that has gone into getting this far.
Best regards.
On Wed, 26 Oct 2005 12:04:46 +0200, Molle Bestefich molle.bestefich@gmail.com wrote:
Regression testing with Wine is a pain in the butthole.
For very bad reasons: Stuff that is simple to fix, but haven't been.
I'd like to help improve that situation.
Example:
I want to find which patch ruins an application. The application, according to one note, worked in 2003. I decide to try and see how it really runs with the 2003 version of Wine mentioned in the note.
I then spend several days pulling various releases, just trying to compile them. They all fail miserably, because some fix required to compile correctly with newer Linux versions is /missing/.
For example,
- any Wine before 2004-01-02 won't work because it won't compile
against newer ALSA versions.
- any Wine before 2003-12-01 needs patches to the build system before
'make depend' will even run.
I've been at it for a week and I've managed to fix the above two. _They're really simple things, but they take a lot of time to figure out._
Currently I'm working with two other problems:
- Wine fails to correctly invoke the debugger. Winedbg shows 'Usage:
blah blah'. Seems 'wine' calls it with incorrect parameters.
- Wine fails to start any application, complaining about 'kthreads'
something or the other.
I imagine that if I spend a few more weeks on the above problems, I can find patches in Wine CVS that fixes the exact problems I'm seeing.
I've already wasted a week of my time on problems that someone else already fixed. Spending two or more weeks doing this is not exactly on my fun-things-to-do-before-I-die list. I'd like to see a general solution to this entire problem.
What can be done to alleviate?
Once I find a patch which makes an earlier version actually compile, I'd like to propose that for a backport.
If approved/applied, the next person that checks out the release would check out a version containing the backported patches.
That would fix the problem and allow people to do proper regression testing, so developers and QA people can spend their time doing development and QA instead of redoing already done effort trying to get an older version to run.
I'm not sure how this would work with CVS. I'm from an SVN world where tagging is a 0-cost tree-copy operation, and any patches can be applied to the tag with history and all showing that the tag's been modified. If it can't work with CVS, perhaps the tags can be created in Troy's SVN repo and patches applied there? Not sure. Suggestions, please, people.
On Wed, Oct 26, 2005 at 01:04:30PM +0200, wino@piments.com wrote:
Very good points. It seems a lot of this has been accepted over the years on the basis that wine was alpha software and in that context , the past is the past - get the new release etc.
It seems that the result is , as you say, regression testing is a PITA and as a result often gets skipped. The code base has certainly advanced a lot but every release seems to create as many problems as it solves.
I managed to get an app working in Feb, luckily I tarballed the whole .wine installation since I am now unable to reinstall this app under any version of wine and make it work.
Wine is now officially beta and this is probably the stage at which some serious project management has to come into play.
Maybe someone has to take a week off from coding and review the overall picture. The core devs have a huge task so it must be made easier for people like yourself who clearly have the expertise to contribute to get involved.
We adapt to the upstream packages. We cannot control their changes.
Like alsa using "interface" as keyword in their headers. Or glibc changing.
Ciao, Marcus
Mike McCormack wrote:
To do a valid regression test, you really want to run it on a system that worked in 2003, so you probably should be compiling it and running it on Slackware 9.0 or rather than Slackware 10.1, or whatever system you were using back then.
Probably. I don't require 100% accuracy however, so I won't be doing that.
It's obviously too much of a hassle to go find out which operating systems were in use back then and install those on my PC just because I really "should" in order to get picture perfect results, isn't it? 99.9% of the time, it's stuff in Wine that's broken, not stuff in Linux, right?
For example, if you tried to use Wine from a couple of years ago on the latest Fedora Core, even if you get it to compile, it won't run, because of problems with exec-shield and who knows what else.
That's exactly what should be fixed. Stuff that prevents older versions from running *at all*.
You know where the problems are; I know where *some* of them are; I'm willing to do volunteer work to rectify the situation for everyone else that wants to do Wine regression testing, but need cooperation from you guys before I can do any good.
wino@piments.com wrote:
It seems that the result is, as you say, regression testing is a PITA and as a result often gets skipped.
I see that we agree :-)
Marcus Meissner wrote:
We adapt to the upstream packages. We cannot control their changes.
Like alsa using "interface" as keyword in their headers. Or glibc changing.
No, but we can adapt to them.
And if we are serious about wanting people to do regression testing and finding the patches that breaks things, we SHOULD adapt to them.
It seems that you and others have lots of examples of things that break Wine but is not Wine's fault. Let's get those things *fixed* in older release tarballs, so that people can do regression testing.
We actually /ask/ people to do regression testing, here f.x.: http://www.winehq.org/site/docs/winedev-guide/x1344 to find a patch that breaks things. Some developers has probably recommended that approach on the mailing lists too, I know I have.
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
How do you decide what things have to be fixed in old tarballs ? Do you test each old tarballs every now and then and apply the corresponding patches to them ?
What if patching the old tarball cause other regressions ?
What if someone wants to use an old tarball in an old system that would be broken because of this patching ? You would have to maintain a lot of old tarballs version...
I don't think it is feasible.
* On Wed, 26 Oct 2005, Jonathan Ernst wrote:
- Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
How do you decide what things have to be fixed in old tarballs ? Do you test each old tarballs every now and then and apply the corresponding patches to them ?
What if patching the old tarball cause other regressions ?
If that will be noted on wine-user the mailing list, it could be fixed.
What if someone wants to use an old tarball in an old system that would be broken because of this patching ?
Then that person won't use patching. Actually, using Molle tactics mean to patch tarball only when the problems arise. Such script even may use some database to collect patches for an old distros and a variety of a kernels.
Plus it can be a reminder of when user/admin should do configuration using winecfg, via config file or by editing ~/.wine/system.reg . Do you think I still remember, on what year and what month winecfg was introduced? No way. Such stuff should go into some databased feature, IMHO.
That would be like a two or more levels of ./configure. :-)
You would have to maintain a lot of old tarballs version... I don't think it is feasible.
If such patcher-system/script isn't acceptable in the Wine tree, then it can go as a separate project, named for example Wine2Old, OldWine, etc. :-P
Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
We actually /ask/ people to do regression testing, here f.x.: http://www.winehq.org/site/docs/winedev-guide/x1344 to find a patch that breaks things. Some developers has probably recommended that approach on the mailing lists too, I know I have.
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
I just got a new idea:
Instead of appliying new patches to old versions which is unmaintainable imho, we could have VMWare installs of Wine and people could download the VMWare image of any Wine release and play it for free using the VMWare player !
I guess the download size will be quite big, but a small distro with only X, Wine, Gnome and KDE and required dependencies for example could fit on half a CD I guess.
Jonathan Ernst wrote:
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
How do you decide what things have to be fixed in old tarballs ? Do you test each old tarballs every now and then and apply the corresponding patches to them ?
Sure, I wouldn't mind checking out alpha releases from the past year or so and testing if they compile.
What if someone wants to use an old tarball in an old system that would be broken because of this patching ? You would have to maintain a lot of old tarballs version...
They'd check out the original tagged version.
I don't think it is feasible.
I'll show you.
The KDE project tagged version 3.4.9.2 on 2005-10-11. Since then they've spent two days doing cleanups and merging hotfixes to that tag.
Too see what has happened to the 3.4.9.2 tag (only), do: # svn log --stop-on-copy svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/
You'll see that the tagging operation is r469461 and the hotfixes etc. are ensuing numbers.
To check out the HEAD of the tag, including hotfixes: # svn co svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/ my_kde
To check out the original tag, no fixes: # svn co -r469461 svn://anonsvn.kde.org/home/kde/tags/KDE/3.4.92/ my_kde
Easy :-).
This is just an example to show it can be done all while keeping the source in the repository (where it belongs, IMHO) and making checkout of both the original tag and the fixed version as easy as sunday morning.
I know that SVN doesn't fly around here, so please don't hit me on the head for that one - it's just that I have no experience with CVS..
gslink wrote:
This is the only solution that has worked in the past. It requires a test suite be built that can easily test Windows, not Wine, and it requires that the tests be run before the next version comes out.
Not realistic. Building a test suite will take years. I disagree that it's the only solution that will work.
Saulius Krasuckas wrote:
Then that person won't use patching. Actually, using Molle tactics mean to patch tarball only when the problems arise. Such script even may use some database to collect patches for an old distros and a variety of a kernels.
Right. Except I'm still hoping that an entire new system isn't needed for this. Although it would obviously be more flexible, it would probably be overkill?
If such patcher-system/script isn't acceptable in the Wine tree, then it can go as a separate project, named for example Wine2Old, OldWine, etc.
Feasible albeit not as pretty. And vastly more time consuming.
Lionel Ulmer wrote:
The application, according to one note, worked in 2003.
The question now is: why was this application not tested more often to get the exact date of regression ?
I'm guessing that most people give up before I do =).
Or maybe they just haven't the time. Which is where I'd like to help regarding this thread..
Jonathan Ernst wrote:
I just got a new idea:
Instead of appliying new patches to old versions which is unmaintainable imho,
Bah :-). Perhaps with CVS. I don't think so but I don't really know.
we could have VMWare installs of Wine and people could download the VMWare image of any Wine release and play it for free using the VMWare player !
Excellent idea!!
I guess the download size will be quite big, but a small distro with only X, Wine, Gnome and KDE and required dependencies for example could fit on half a CD I guess.
Do you know of any miniature distro(s) that carry these and (preferably) existed all along with Wine? If you have suggestions, I'd like to give it a shot right away.
We'd need a web page somewhere telling people what image works with which Wine alphas/betas.
Le vendredi 28 octobre 2005 à 06:21 +0000, Molle Bestefich a écrit :
Jonathan Ernst wrote:
we could have VMWare installs of Wine and people could download the VMWare image of any Wine release and play it for free using the VMWare player !
Excellent idea!!
I guess the download size will be quite big, but a small distro with only X, Wine, Gnome and KDE and required dependencies for example could fit on half a CD I guess.
Do you know of any miniature distro(s) that carry these and (preferably) existed all along with Wine? If you have suggestions, I'd like to give it a shot right away.
If you remove the word processor I think you'll be under 30-40MB uncompressed with these ones:
http://www.damnsmalllinux.org/ http://www.zenwalk.org/staticpages/index.php?page=20050321031055330
I didn't find a ready to use minidistro with gnome or/and kde as these applications are really big. You could also try something like ubuntu and install only the minimum reqs of Wine.
Thanks for your efforts; looking forward to testing your VMWare machines !
Jonathan Ernst wrote:
If you remove the word processor I think you'll be under 30-40MB uncompressed with these ones:
http://www.damnsmalllinux.org/ http://www.zenwalk.org/staticpages/index.php?page=20050321031055330
I didn't find a ready to use minidistro with gnome or/and kde as these applications are really big. You could also try something like ubuntu and install only the minimum reqs of Wine.
Thanks for your efforts; looking forward to testing your VMWare machines !
For the record, I still think it's a better idea to make older releases compileable by applying the minimalistic patches required. It has several advantages:
a.) Effort can easily be shared by multiple volunteers by simply identifying individual patches needed to compile correctly. b.) Patches are usually required not only by one release, but by one release _and_ all prior releases. Lots of prior releases can be fixed at the same time, once a culprit has been identified. c.) Hotfixing requires a lot less effort than the virtual machine approach.
Depending on how much effort the VM thingy is, I might only make the one or two VMs that I need for my testing, and just modify th{at/ose} as I go along.
* On Thu, 27 Oct 2005, Jonathan Ernst wrote:
- Le mercredi 26 octobre 2005 à 16:49 +0200, Molle Bestefich a écrit :
We actually /ask/ people to do regression testing, here f.x.: http://www.winehq.org/site/docs/winedev-guide/x1344 to find a patch that breaks things. Some developers has probably recommended that approach on the mailing lists too, I know I have.
The approach is useless however, until these simple fixes are applied to the tarballs (preferably through the versioning system).
Instead of appliying new patches to old versions which is unmaintainable imho, we could have VMWare installs of Wine and people could download the VMWare image of any Wine release and play it for free using the VMWare player !
I guess the download size will be quite big, but a small distro with only X, Wine, Gnome and KDE and required dependencies for example could fit on half a CD I guess.
One more (nice) way could be to run all the tested stuff (inside VMWare player?) on some remote X (?) server, so people should not download several large ISOs to test small software (instead they would upload small test-software packages into the server). But I start feeling this would require not a less amount of bandwidth too, so YMMV. :)
On Wed, Oct 26, 2005 at 10:04:46AM +0000, Molle Bestefich wrote:
I want to find which patch ruins an application. The application, according to one note, worked in 2003. I decide to try and see how it really runs with the 2003 version of Wine mentioned in the note.
The question now is: why was this application not tested more often to get the exact date of regression ?
Lionel
-
Jonathan Ernst -
Lionel Ulmer -
Marcus Meissner -
Mike McCormack -
Molle Bestefich -
Saulius Krasuckas -
wino＠piments.com