I noticed a probable regression in an app (kid pix 3) tonight. Which got me thinking: what are the easiest set of application regression tests we could throw together that go further than just verifying the installer runs?
I'm thinking not just of things that could be automated with cxtest, but also stuff that might not fit into that framework.
In the case of Kid Pix 3, a test that simply started the app, waited ten seconds, and made sure an error dialog didn't pop up would have detected the problem. ('Course, only a few people would be able to run that, as Kid Pix 3 is not a free download.) cxtest should be able to handle that.
A bit more complex would be to launch WinZip, have it compress one file, and make sure the compressed file was as expected. It shouldn't be too hard to make cxtest handle that.
And then on the somewhat ambitious end, now that Sun's JVM basically installs and runs trivial programs, we could run through the Java regression test suite at http://sourceware.org/mauve/ Heck, I might give that a shot myself. There's probably no reason to try to shove that into cxtest, is there? Or would it fit?
And beyond the looking glass, one could imagine running the OpenOffice 2.0 automated test suite. (That's unlikely to work well at the moment, given that OpenOffice 2.0's open dialog locks up reliably.) That would probably flush out a few problems. - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
On 3/18/06, Dan Kegel dank@kegel.com wrote:
Which got me thinking: what are the easiest set of application regression tests we could throw together that go further than just verifying the installer runs? ... And then on the somewhat ambitious end, now that Sun's JVM basically installs and runs trivial programs, we could run through the Java regression test suite at http://sourceware.org/mauve/ Heck, I might give that a shot myself.
OK, I looked. It has a big configure script. I tried running it under msys on wine, but that failed miserably. So I guess it's not easy, offhand (big surprise). I suppose there might be other ways to run parts of the mauve test suite on wine, but I'll pass for now.
I remember now a post from almost five years ago, proposing using Abiword as the app to test: http://www.winehq.com/hypermail/wine-users/2001/08/0086.html Wine wasn't ready for that back then, but it is now; abiword-2.0.8 works fine under current wine (newer versions run into http://bugs.winehq.org/show_bug.cgi?id=4887 ). I'll go check to see if they've got an automated regression test of their own yet... - Dan -- Wine for Windows ISVs: http://kegel.com/wine/isv
On Sun, 2006-03-19 at 09:09 -0800, Dan Kegel wrote:
OK, I looked. It has a big configure script. I tried running it under msys on wine, but that failed miserably. So I guess it's not easy, offhand (big surprise). I suppose there might be other ways to run parts of the mauve test suite on wine, but I'll pass for now.
I personally think this is a very exciting thing to get right. Being able to run an emulation layer like msys/cygwin would stress the hell out of the fundamental APIs, and would ensure a solid implementation in Wine. Not to mention that: -- they are OSS, making debugging a bit easier -- they have _lots_ of tests that can be run automatically -- said tests could dramatically enhance and broaden the available regression tests we have
Adding to this mix stuff like Java can only extend coverage. And we would be tapping into a huge barrage of tests.
On 3/19/06, Dimi Paun dimi@lattica.com wrote:
On Sun, 2006-03-19 at 09:09 -0800, Dan Kegel wrote:
I tried [mauve on msys on wine] ...
I personally think this is a very exciting thing to get right. Being able to run an emulation layer like msys/cygwin would stress the hell out of the fundamental APIs, and would ensure a solid implementation in Wine. Not to mention that: -- they are OSS, making debugging a bit easier -- they have _lots_ of tests that can be run automatically -- said tests could dramatically enhance and broaden the available regression tests we have
Totally. FWIW, the old, minimal snapshot of cygwin in msys is probably easier to support than the modern cygwin. (See http://www.mingw.org/MinGWiki/index.php/mSysFromScratch for how to build it, if you're interested.)
Adding to this mix stuff like Java can only extend coverage. And we would be tapping into a huge barrage of tests.
Yes, it would be an amazing amount of leverage, test-wise. Having msys working would make a whole new level of dogfooding possible (e.g. building windows mozilla/firefox on msys on wine). - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
Dan Kegel wrote:
On 3/19/06, Dimi Paun dimi@lattica.com wrote:
On Sun, 2006-03-19 at 09:09 -0800, Dan Kegel wrote:
I tried [mauve on msys on wine] ...
I personally think this is a very exciting thing to get right. Being able to run an emulation layer like msys/cygwin would stress the hell out of the fundamental APIs, and would ensure a solid implementation in Wine. Not to mention that: -- they are OSS, making debugging a bit easier -- they have _lots_ of tests that can be run automatically -- said tests could dramatically enhance and broaden the available regression tests we have
Totally. FWIW, the old, minimal snapshot of cygwin in msys is probably easier to support than the modern cygwin. (See http://www.mingw.org/MinGWiki/index.php/mSysFromScratch for how to build it, if you're interested.)
Adding to this mix stuff like Java can only extend coverage. And we would be tapping into a huge barrage of tests.
Yes, it would be an amazing amount of leverage, test-wise. Having msys working would make a whole new level of dogfooding possible (e.g. building windows mozilla/firefox on msys on wine).
- Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
Well, that would be good for dogfooding, but so would getting Dev-C++ to work perfectly on Wine -- it uses MingW/gcc to compile.
On 3/19/06, Segin segin2005@gmail.com wrote:
... getting Dev-C++ to work perfectly on Wine [would be good for dogfooding] -- it uses MingW/gcc to compile.
Sure, good idea. I encourage Dev-C++ users to test it with Wine and file bug reports at http://bugs.winehq.org for any problems they find. I just looked, and only found one report, http://bugs.winehq.org/show_bug.cgi?id=4483 which was really fixed already by wine-0.9.7.
In my own test of dev-c++, it looks like there are a few small cosmetic problems. I didn't try it on any real apps, though. I'll let IDE users do that. - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
Dimi Paun wrote:
Being able to run an emulation layer like msys/cygwin
In my experience (couple of years worth of different Cygwin versions, all relatively new stuff), the Cygwin kernel emulation stuff is extremely unstable itself. So that would probably get you a heck of a lot of false positives in terms of faults occurring.
There are a couple of other Windows-->Unix emulation kits readily available, so shouldn't be hard to find something that both runs reliably and can make ./configure scripts work...
On 3/20/06, Molle Bestefich molle.bestefich@gmail.com wrote:
Dimi Paun wrote:
Being able to run an emulation layer like msys/cygwin
In my experience (couple of years worth of different Cygwin versions, all relatively new stuff), the Cygwin kernel emulation stuff is extremely unstable itself.
unstable in the sense that it changes all the time, yes. msys, on the other hand, is working off a frozen snapshot of cygwin from long ago, and never changes. So it's stable in at least that sense. I have no idea how buggy it is. It works well enough to run configure scripts on real windows, though.
There are a couple of other Windows-->Unix emulation kits readily available, so shouldn't be hard to find something that both runs reliably and can make ./configure scripts work...
Oh, really? Got URLs? Have you tried any of them on wine? - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
Dan Kegel wrote:
Molle Bestefich wrote:
Dimi Paun wrote:
Being able to run an emulation layer like msys/cygwin
In my experience (couple of years worth of different Cygwin versions, all relatively new stuff), the Cygwin kernel emulation stuff is extremely unstable itself.
unstable in the sense that it changes all the time, yes.
No. Unstable in the sense that the core code is buggy.
msys, on the other hand, is working off a frozen snapshot of cygwin from long ago, and never changes. So it's stable in at least that sense. I have no idea how buggy it is. It works well enough to run configure scripts on real windows, though.
Ok. Cygwin does pretty well if you're using it as a single user / do not use many processes/threads while pushing it CPU-wise.
I'll agree that hobby usage (running a couple of Unix apps) is the most common use case for Cygwin, and it works well in that case.
Eg., Cygwin will be able to run a ./configure script just fine, too.
There are a couple of other Windows-->Unix emulation kits readily available, so shouldn't be hard to find something that both runs reliably and can make ./configure scripts work...
Oh, really? Got URLs?
MKS Toolkit http://www.mkssoftware.com/
Windows Services for Unix http://www.microsoft.com/technet/interopmigration/unix/sfu/
Have you tried any of them on wine?
Unfortunately, no.
On 3/22/06, Molle Bestefich molle.bestefich@gmail.com wrote:
Ok. Cygwin does pretty well if you're using it as a single user / do not use many processes/threads while pushing it CPU-wise.
Right -- and I think we can hew to that limitation when using msys as part of a possible wine regression test.
Eg., Cygwin will be able to run a ./configure script just fine, too.
Right, but modern cygwin is hard to support on wine, I think. The old snapshot in msys is probably easier.
There are a couple of other Windows-->Unix emulation kits readily available, so shouldn't be hard to find something that both runs reliably and can make ./configure scripts work...
Oh, really? Got URLs?
MKS Toolkit http://www.mkssoftware.com/
Commercial, not a free download. There's a way to apply for a trial, but it's not easy, and you can't just download it.
Windows Services for Unix http://www.microsoft.com/technet/interopmigration/unix/sfu/
I just tried it; it bombs on setup, complaining that the TCP/IP service isn't installed. But even if we fix that, downloading it requires a Microsoft Passport account and you have to fill out an intrusive questionaire. Also, there's no way of knowing when Microsoft will suspend free downloads. So I think that's out.
Which kind of leaves us with msys/mingw as the prime contender, I think. For starters, all we have to do is verify, say, that it can build and run a hello, world app on wine. I think that even works right now! - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
On 3/19/06, Dan Kegel dank@kegel.com wrote:
[Let's use Abiword as the app to test, as I proposed long ago:] http://www.winehq.com/hypermail/wine-users/2001/08/0086.html Wine wasn't ready for that back then, but it is now; abiword-2.0.8 works fine under current wine (newer versions run into http://bugs.winehq.org/show_bug.cgi?id=4887 ). I'll go check to see if they've got an automated regression test of their own yet...
They don't. So we'd have to write one. Shouldn't be hard to do - either as a cxtest script, or as a C program that sends Windows events and looks at the resulting files abiword creates, or some combination... - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
On 3/19/06, Dan Kegel dank@kegel.com wrote:
[Let's use Abiword as the app to test, as I proposed long ago:]
But before we do that, let's have a good gui test for wine's own notepad and wordpad. First things first, eh? - Dan
-- Wine for Windows ISVs: http://kegel.com/wine/isv
-
Dan Kegel -
Dimi Paun -
Molle Bestefich -
Segin