On Wed, Mar 25, 2009 at 9:47 AM, Francois Gouget fgouget@free.fr wrote:
On Tue, 24 Mar 2009, Austin English wrote: [...]
I'd like to implement an application test suite. It's something that's been discussed for Wine for quite a while, but has never been put into place.
As mentioned before, check out cxtest. In particular the way they are structured, with one script to install the application, and others testing particular aspects of it. You will also need to support dependencies at some point.
Yes, I've looked at cxtest a bit. I like Jeremy's line of thinking...keep cxtest for its infrastructure, but move toward AHK, which gains us windows compatibility.
But also check out Lei Zhang's experiments with AutoHotKey:
I've looked a bit at this code, should be useful.
[...]
My plan to implement it is to make a script, that can be used standalone, in conjunction with Patchwatcher, or any other script, to test wine against several real world applications. I make a very quick proof of concept (attached), which is based off of winetricks.
One very important feature is that:
A good testing framework *must* work on Windows.
Exactly why Cxtest is not sufficient.
Why? For the same reason that the conformance tests would not be even half as useful as they are if they did not run on Windows:
* Applications should install the same way in Wine and on Windows. Sure there's bound to be some differences but these should be investigated and either fixed, or documented and worked-around in the test script.
Of course. My very brief initial testing already found a couple of these sorts of bugs.
* This would make it possible to write tests on Windows for things that don't work yet in Wine. This can be particularly powerful in conjunction with Bugzilla: just like we sometimes ask for a conformance test, we could ask for an application test for a given Bugzilla report. Then, sprinkle a few todo_wine and we will immediately know when a bug is fixed.
You read my mind :-)...exactly what I'd like to see. AHK is powerful in this regard as A) it already works well on windows, B) allows for gui recording of scripts, which can be adjusted manually and rewritten. This allows an average user to download AHK (no need to install, has a portable version you can run), fire up foo.exe which doesn't work on wine, record a simple test, and file a bug. It greatly simplifies bug triagers' work, especially for applications in foreign languages (for some reason, there are quite a few Polish and Slovak bugs in bugzilla...).
* This makes it possible for non-Unix users to contribute. Ideally, once we have developped a test for an open-source Windows application, the developers for that application should be able to use our tests and take over their maintenance.
Of course.
So AutoHotKey is a step towards that goal. But you will need some infrastructure around it to download installers, setting up CDs, maybe start and stop virtual machines, and it is this part that worries me. In particular I don't think that bash+wget&co is the most appropriate toolset. What to use on Windows is a tricky issue though.
Right. For downloading, wget for windows exists, as well as several other small utilities (ftp is builtin). CDs are one that I've been thinking about, haven't decided on that yet. Virtual machines would be useful, unfortunately I don't know much about scripting for those (shouldn't be too difficult to learn. Hopefully a potential mentor would have some insight <g>).
For that matter, keep in mind that cxtest is a bash script as well http://cxtest.ifne.eu:82/transfer/cxtest.sh.
Other important features: * You should not need to update the scripts for every little change in Wine.
Of course. That is something that I've kept in mind, but not an easy task. It will take careful planning.
- Microsoft Office - We've broken this a few times in recent months,
so adding a test in daily git for it would help catch regressions sooner.
Just as a note, CodeWeavers tests Microsoft Office every day with cxtest. However this is with the CrossOver codebase so we only detect WineHQ regressions when it gets merged after each release (so roughly every two weeks). In any case it would be nice if these were detected sooner and more 'upstream'<g>.
Right. The problem is those tests don't come before wine releases, nor daily, which allows Office to break in binary releases, not just git. Knowing the problem exists is only part of the fix, and if that takes 2 weeks or more to find, the installer could break for 2-3 WineHQ releases.
- World of Warcraft - I don't play, but A) It's really popular, and
B) also gives us a chance to test wininet for regressions. 4. iTunes - this installers been broken a couple times. The bonjour service should allow for testing services related regressions.
The problem with these two is that they change every other day, so that you're likely to end up having to update your script pretty often. Also, as far as I understand it, Wow is really huge and regularly insists on downloading big update patches (which of course you won't have been able to checksum). This is not to say it shouldn't be done though.
That's why I put them low on the list ;-). WoW would be nice to have, but not necessary. iTunes would be very useful, and the problem of updating daily can be avoided by using mirrors. Verify original install from Apple.com, get SHA1SUM, then get that version from http://www.filehippo.com/, verifying filesize and sha1sum.
Some pointers to past discussions on the subject (far from exhaustive): http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html
Thanks. I've followed these discussions in the past, but a refresher is always good.
My perception is that this sort of project is needed pretty badly, but suffers from 2 main problems: No one wants to do it and those who have tried in the past aimed to high, crashed, and burned. I think I could do better because:
I want to do it.
I've tested _tons_ of windows applications (all that bugzilla testing as good for something).
I've got a more pragmatic view of what's possible. I'm not aiming to make winetest version 2.0 for windows. The AHK scripts will be runnable on windows, sure. But the main bulk of the script will be built for Wine/Unix. While that may not be ideal (the ahk scripts may not be tested daily on windows), it shouldn't be too hard for someone who wants to implement that on windows to do so. Windows doesn't change all that often, Wine does, so it follows that we should test it more often. If the test passes on Windows originally, and a curious party tests it every month or two (or more likely, when a bug is found/fixed), that should be plenty.
This wouldn't be the panacea of regression testing for Wine, but would be a major step in that direction.