Howdy,
I'm planning on applying for Google Summer of Code 2009. I'm pretty sure most people reading this list already know me, but for those that don't, I frequently triage bugs in Bugzilla, help users on the forum, and a few other things. I've also done quite a bit of work testing wine on more obscure OS's (OpenSolaris, FreeBSD, OS X, NetBSD, OpenBSD).
I'd like to implement an application test suite. It's something that's been discussed for Wine for quite a while, but has never been put into place. I took a bit of time a while back and hacked couple a couple quick scripts as a test, and got a script to automatically install Firefox 3 in Wine using AutoHotKey (http://www.autohotkey.com/). I've made a couple more scripts as well, but they don't match Windows behavior because of bug (http://bugs.winehq.org/show_bug.cgi?id=10147) hint, hint (disclaimer, I haven't tested that script in a while, been busy with exams, so that bug might be fixed...I don't think it has though).
My plan to implement it is to make a script, that can be used standalone, in conjunction with Patchwatcher, or any other script, to test wine against several real world applications. I make a very quick proof of concept (attached), which is based off of winetricks. It's _REALLY_ rough at the moment, (I haven't had time to mess with it lately), but demonstrates the idea. The script setups a clean WINEPREFIX, installs autohotkey, removes the Z: symlink for braindead installers that search the entire hard drive, and sets up a link to it's install cache (essentially winetrickscache). From there, it makes a copy of this prefix, so on subsequent installers/applications, we can restore the clean prefix and save a few seconds.
From there, so far, it only installs firefox3. The plan is to add
several dozen different tests, for various functions of wine. Obviously, I'd like to start with platinum rated applications, to prevent regressions. From there, the current list of applications/functions to target are: 1. Firefox3 - useful for testing lots of other things, e.g., java/flash, etc. Easy to test. 2. Microsoft Office - We've broken this a few times in recent months, so adding a test in daily git for it would help catch regressions sooner. 3. World of Warcraft - I don't play, but A) It's really popular, and B) also gives us a chance to test wininet for regressions. 4. iTunes - this installers been broken a couple times. The bonjour service should allow for testing services related regressions. 5. Adobe suite - there haven't been too many regressions here that I can remember, but it's definitely an area we should watch out for.
Of course, we don't want to only test the installer's keystrokes/mouse inputs, so SHA1 checksumming of installed files would verify against corrupted files/setupapi regressions/etc. Once a few installer tests are in and working well, I plan to extend the tests further to test running the application, e.g, for photoshop by opening a file, inserting a circle, and saving it. This is a bit harder to pull off in Autohotkey in a way that's portable, so I'm saving that sort of thing for later.
Also interesting would be a real application's test suite. Dan suggested using Google Chrome's, since it's squeaky clean after using lots of Valgrind and Purify. That would be good, but I'd put that after getting the test suite to pass in Wine.
This should be very doable using Autohotkey's scripting functionality (pretty extensive - http://www.autohotkey.com/docs/Functions.htm, http://www.autohotkey.com/docs/Variables.htm) and good old fashioned shell scripting. I'm very familiar with bourne scripting, and currently maintain winetricks while Dan's taking a Wine vacation. I'm currently working on a build script to build Wine on several OS's, as well as running and submitting the conformance tests (http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh). [I got tired of maintaining the script over several different OS's/computers]. The upside is it would be easy to add on appinstall to this daily script. I typically set it running before leaving home for work/school. It spends an hour or so building wine and running the tests in a few different configurations. Adding appinstall would allow my computer to daily test several applications for installation, running, etc. This of course isn't as useful as Patchwatcher's testing of each single patch, but I don't currently have a spare machine that it can test on. Getting a script and tests ready to do so, however, would be great for Wine. Besides, I can easily setup the script to upload the results to a webserver, so we can daily see that each test is passing. If a test is failing, the information would then be publicly available for anyone to run 'git bisect'.
Wow, this turned into a really long e-mail, and I've got an exam to study for. I'd appreciate any comments, and let me know if there are any questions.