On Tue, 24 Mar 2009, Austin English wrote: [...]
I'd like to implement an application test suite. It's something that's been discussed for Wine for quite a while, but has never been put into place.
As mentioned before, check out cxtest. In particular the way they are structured, with one script to install the application, and others testing particular aspects of it. You will also need to support dependencies at some point.
But also check out Lei Zhang's experiments with AutoHotKey:
http://code.google.com/p/yawt/
[...]
My plan to implement it is to make a script, that can be used standalone, in conjunction with Patchwatcher, or any other script, to test wine against several real world applications. I make a very quick proof of concept (attached), which is based off of winetricks.
One very important feature is that:
A good testing framework *must* work on Windows.
Why? For the same reason that the conformance tests would not be even half as useful as they are if they did not run on Windows:
* Applications should install the same way in Wine and on Windows. Sure there's bound to be some differences but these should be investigated and either fixed, or documented and worked-around in the test script.
* This would make it possible to write tests on Windows for things that don't work yet in Wine. This can be particularly powerful in conjunction with Bugzilla: just like we sometimes ask for a conformance test, we could ask for an application test for a given Bugzilla report. Then, sprinkle a few todo_wine and we will immediately know when a bug is fixed.
* This makes it possible for non-Unix users to contribute. Ideally, once we have developped a test for an open-source Windows application, the developers for that application should be able to use our tests and take over their maintenance.
So AutoHotKey is a step towards that goal. But you will need some infrastructure around it to download installers, setting up CDs, maybe start and stop virtual machines, and it is this part that worries me. In particular I don't think that bash+wget&co is the most appropriate toolset. What to use on Windows is a tricky issue though.
Other important features: * You should not need to update the scripts for every little change in Wine.
- Microsoft Office - We've broken this a few times in recent months,
so adding a test in daily git for it would help catch regressions sooner.
Just as a note, CodeWeavers tests Microsoft Office every day with cxtest. However this is with the CrossOver codebase so we only detect WineHQ regressions when it gets merged after each release (so roughly every two weeks). In any case it would be nice if these were detected sooner and more 'upstream'<g>.
- World of Warcraft - I don't play, but A) It's really popular, and
B) also gives us a chance to test wininet for regressions. 4. iTunes - this installers been broken a couple times. The bonjour service should allow for testing services related regressions.
The problem with these two is that they change every other day, so that you're likely to end up having to update your script pretty often. Also, as far as I understand it, Wow is really huge and regularly insists on downloading big update patches (which of course you won't have been able to checksum). This is not to say it shouldn't be done though.
Some pointers to past discussions on the subject (far from exhaustive): http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html