Recruit regression testers?
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea? Ivan.
Ivan Leo Puoti wrote:
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
Well that is one of the responsibilitys of an application maintainer. I know that Myst is one of those apps that worked for years and then stopped working about a year ago. I have tried to regression test it, and so has Duane Clark. At issue here is that so much time has gone by that there are/were multiple regressions that interfere with actually finding the problem. I am certain that if I had caught the regression(s) early it would still be working today. With that in mind I think recruiting additional maintainers for the AppDB should be of utmost priority. -- Tony Lambregts
tony_lambregts(a)telusplanet.net wrote:
Ivan Leo Puoti wrote:
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
Well that is one of the responsibilitys of an application maintainer. I know that Myst is one of those apps that worked for years and then stopped working about a year ago. I have tried to regression test it, and so has Duane Clark. At issue here is that so much time has gone by that there are/were multiple regressions that interfere with actually finding the problem. I am certain that if I had caught the regression(s) early it would still be working today. With that in mind I think recruiting additional maintainers for the AppDB should be of utmost priority.
And a unit test should be written after each fixed regression. regards, Jakob
On Mon, 2005-01-24 at 04:01 +0100, Ivan Leo Puoti wrote:
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
Ivan.
I know Fallout 1 and 2 have been broken for 7 months, and I've even pointed the exact patch to lionel over IRC: http://www.winehq.org/hypermail/wine-devel/2004/12/0125.html http://www.winehq.org/hypermail/wine-cvs/2004/06/0029.html The trouble is that I myself wasn't aware of it until it had been broken for six months or so. I agree that having good app maintainers (of which I now am for Fallout) is very important. Interestingly enough Fallout was one of the first games duplicated in the Wine wiki - I'm not sure why. Lionel, meanwhile, needs to digup his fallout CDs to debug the app :) Thanks a ton, Scott Ritchie
Lionel, meanwhile, needs to digup his fallout CDs to debug the app :)
Oh well, it's not exactly my CD that I need to find (I actually have a log showing the problem done with the Fallout installer on my CD), it's just that I need to find the time to do it. Lionel -- Lionel Ulmer - http://www.bbrox.org/
El lun, 24 de ene de 2005, a las 10:39, Lionel Ulmer escribio:
Lionel, meanwhile, needs to digup his fallout CDs to debug the app :)
Oh well, it's not exactly my CD that I need to find (I actually have a log showing the problem done with the Fallout installer on my CD), it's just that I need to find the time to do it.
It is the line what did the regression. Regards, Carlos. -- ___ _ \ | / Consulting | . |._ _ _| | ___ ___ ___ http://www.andago.com | || ' |/ . |<_> |/ . |/ . \__ GNU/Linux |_|_||_|_|\___|<___|\_. |\___/ _ \ __|\ \ / Carlos A. Lozano <___'/ | \ -_) __/\__ \ > < -_) [ carlos.lozano(a)andago.com ]\___|_| ____/ _/\_\___| [ calb(a)epsxe.com ] http://www.ePSXe.com
I can confirm that this patch fixes things, at least for DInput in Fallout THANKS! -Scott Ritchie On Sat, 2005-01-29 at 18:49 +0100, Carlos Lozano wrote:
El lun, 24 de ene de 2005, a las 10:39, Lionel Ulmer escribio:
Lionel, meanwhile, needs to digup his fallout CDs to debug the app :)
Oh well, it's not exactly my CD that I need to find (I actually have a log showing the problem done with the Fallout installer on my CD), it's just that I need to find the time to do it.
It is the line what did the regression.
Regards, Carlos.
On Mon, Jan 31, 2005 at 10:41:34AM -0800, Scott Ritchie wrote:
I can confirm that this patch fixes things, at least for DInput in Fallout
I promise that one of these days I will look into it and try to understand exactly what the problem is (basically what I tried to do or which application to fix to introduce a more stringent check for correspondance between Wine's and the required data types)... While waiting for this day (i.e. when the snow melts, it's too good for now to miss any possible day out :-) ), this patch can go in. Lionel -- Lionel Ulmer - http://www.bbrox.org/
On Monday 24 January 2005 04:01, Ivan Leo Puoti wrote:
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
Ivan.
Hi Ivan, this idea sounds quite good, though I've had a different concept in mind, for let's say more "mature wine releases". Though the whole releasing methods would need changes. I was thinking of a ie. wine beta1-rc release. Now we wanna make sure no new regressions have been introduced - but how? I thought of creating a global list of applications, which are known to be quite "tricky" and have been broken several times of the last years. Think of a big feature matrix showing all applications indicating their status "works" / "broken forever" / "broken since last release". And a new wine release would only be given out when all of these apps have been tested. This would require a high amount a people with a real motivation in background and probably a much longer time until the release is "ready". Do you think this idea is "overengineered"? Bye Bye Niko
So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
As an fyi, we have spent the past year or so developing a Wine visual regression testing tool, and I think we're getting to a point where it is starting to be useful. A very rudimentary web site is up at www.cxtest.org, if you want to go look. The basic idea is that we provide a test framework that is capable of doing visual clicks, both for X11 based apps, and for Wine based apps. This lets us automate the process of installing an app and putting it through some really basic tests (see the README for the WineHQ winemine test for an example of that). Our long term goal is to have a very large distributed testing grid, where people can run a cron job that runs Wine tests every night, and every morning we can look at www.cxtest.org and see the 'health' of the previous nights build. In theory, this should let us catch regressions the day they happen. The test tool is largely oriented around CrossOver at this point, but it's been designed to be used with either Wine from CrossOver or with WineHQ Wine, and should be readily adaptable to Wine. We'd appreciate folks that wanted to come and help. The few of us working on it are hanging out on #cxtest these days. Cheers, Jeremy
Nikolas Zimmermann wrote:
On Monday 24 January 2005 04:01, Ivan Leo Puoti wrote:
I think quite a few regressions get into releases, and while a few a tracked down some stay in the code for months if not years (I've heard of at least one game that worked some 2 years ago but doesn't now), while regression testing may seem trivial to a developer, it can be a challenging task for most users, especially if they are new to wine/Linux. So maybe we should recruit some people to get regression reports from users, find the patch that caused the regression, and report back to developers. Am I just to tired at time of writing or could this be a good idea?
Ivan.
Hi Ivan,
this idea sounds quite good, though I've had a different concept in mind, for let's say more "mature wine releases". Though the whole releasing methods would need changes. I was thinking of a ie. wine beta1-rc release. Now we wanna make sure no new regressions have been introduced - but how?
I thought of creating a global list of applications, which are known to be quite "tricky" and have been broken several times of the last years. Think of a big feature matrix showing all applications indicating their status "works" / "broken forever" / "broken since last release".
And a new wine release would only be given out when all of these apps have been tested. This would require a high amount a people with a real motivation in background and probably a much longer time until the release is "ready". Do you think this idea is "overengineered"?
We have a list of applications, it is called the AppDB. One of the reasons we have maintainers is so that they will catch regressions. In the long run when we go to a stable/unstable release cycle and every app that was "gold" in the previous stable release should work with the next stable release. IMO Having a list of progams that are gold (just work) is a requirement of 0.9. If a program does not have a maintainer then it cannot get on that list. I could entertain adding another field to the appdb to indicate that the App is "Supported by Wine developers" but I believe that would be redundant. As a side note we have a list of "Gold" and "Silver" Applications on the front page. I cannot remember where this list came from but in the long run we plan on replacing that static list with a with a dynamic list that uses the actual data from the AppDB. Currently for the "Gold" list only Acrobat Reader, Putty and FileZilla have maintainers. For the "Silver" list only WinZip, WinRar, WinMX and Snagit have mantainers. (BTW Jonathan Ernst is the maintainer for most of these apps. Thanks.) I would like to see maintainers for the remaining apps so we can make the switch. -- Tony Lambregts
On Mon, 2005-01-24 at 10:01 -0700, tony_lambregts(a)telusplanet.net wrote:
Currently for the "Gold" list only Acrobat Reader, Putty and FileZilla have maintainers. For the "Silver" list only WinZip, WinRar, WinMX and Snagit have mantainers. (BTW Jonathan Ernst is the maintainer for most of these apps. Thanks.) I would like to see maintainers for the remaining apps so we can make the switch.
--
Tony Lambregts
How about we just move those apps and redo the front page? I'd been meaning to submit a patch, but I got scared away by the non template stuff. The list of supported or working apps should be on a separate page anyway. -Scott Ritchie
Scott Ritchie wrote:
On Mon, 2005-01-24 at 10:01 -0700, tony_lambregts(a)telusplanet.net wrote:
Currently for the "Gold" list only Acrobat Reader, Putty and FileZilla have maintainers. For the "Silver" list only WinZip, WinRar, WinMX and Snagit have mantainers. (BTW Jonathan Ernst is the maintainer for most of these apps. Thanks.) I would like to see maintainers for the remaining apps so we can make the switch.
How about we just move those apps and redo the front page?
I'd been meaning to submit a patch, but I got scared away by the non template stuff. The list of supported or working apps should be on a separate page anyway.
I like the supported apps being on the front page myself. Its just that supported apps should (by definition) have maintainers. -- Tony Lambregts
participants (8)
-
Carlos Lozano -
Ivan Leo Puoti -
Jakob Eriksson -
Jeremy White -
Lionel Ulmer -
Nikolas Zimmermann -
Scott Ritchie -
tony_lambregts＠telusplanet.net