On Wed, Mar 25, 2009 at 12:33 AM, Austin English
<austinenglish@gmail.com> wrote:
Howdy,
I'm planning on applying for Google Summer of Code 2009. I'm pretty
sure most people reading this list already know me, but for those that
don't, I frequently triage bugs in Bugzilla, help users on the forum,
and a few other things. I've also done quite a bit of work testing
wine on more obscure OS's (OpenSolaris, FreeBSD, OS X, NetBSD,
OpenBSD).
I'd like to implement an application test suite. It's something that's
been discussed for Wine for quite a while, but has never been put into
place. I took a bit of time a while back and hacked couple a couple
quick scripts as a test, and got a script to automatically install
Firefox 3 in Wine using AutoHotKey (http://www.autohotkey.com/). I've
made a couple more scripts as well, but they don't match Windows
behavior because of bug (http://bugs.winehq.org/show_bug.cgi?id=10147)
hint, hint (disclaimer, I haven't tested that script in a while, been
busy with exams, so that bug might be fixed...I don't think it has
though).
My plan to implement it is to make a script, that can be used
standalone, in conjunction with Patchwatcher, or any other script, to
test wine against several real world applications. I make a very quick
proof of concept (attached), which is based off of winetricks. It's
_REALLY_ rough at the moment, (I haven't had time to mess with it
lately), but demonstrates the idea. The script setups a clean
WINEPREFIX, installs autohotkey, removes the Z: symlink for braindead
installers that search the entire hard drive, and sets up a link to
it's install cache (essentially winetrickscache). From there, it makes
a copy of this prefix, so on subsequent installers/applications, we
can restore the clean prefix and save a few seconds.
>From there, so far, it only installs firefox3. The plan is to add
several dozen different tests, for various functions of wine.
Obviously, I'd like to start with platinum rated applications, to
prevent regressions. From there, the current list of
applications/functions to target are:
1. Firefox3 - useful for testing lots of other things, e.g.,
java/flash, etc. Easy to test.
2. Microsoft Office - We've broken this a few times in recent months,
so adding a test in daily git for it would help catch regressions
sooner.
3. World of Warcraft - I don't play, but A) It's really popular, and
B) also gives us a chance to test wininet for regressions.
4. iTunes - this installers been broken a couple times. The bonjour
service should allow for testing services related regressions.
5. Adobe suite - there haven't been too many regressions here that I
can remember, but it's definitely an area we should watch out for.
Of course, we don't want to only test the installer's keystrokes/mouse
inputs, so SHA1 checksumming of installed files would verify against
corrupted files/setupapi regressions/etc. Once a few installer tests
are in and working well, I plan to extend the tests further to test
running the application, e.g, for photoshop by opening a file,
inserting a circle, and saving it. This is a bit harder to pull off in
Autohotkey in a way that's portable, so I'm saving that sort of thing
for later.
Also interesting would be a real application's test suite. Dan
suggested using Google Chrome's, since it's squeaky clean after using
lots of Valgrind and Purify. That would be good, but I'd put that
after getting the test suite to pass in Wine.
This should be very doable using Autohotkey's scripting functionality
(pretty extensive - http://www.autohotkey.com/docs/Functions.htm,
http://www.autohotkey.com/docs/Variables.htm) and good old fashioned
shell scripting. I'm very familiar with bourne scripting, and
currently maintain winetricks while Dan's taking a Wine vacation. I'm
currently working on a build script to build Wine on several OS's, as
well as running and submitting the conformance tests
(http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh). [I
got tired of maintaining the script over several different
OS's/computers]. The upside is it would be easy to add on appinstall
to this daily script. I typically set it running before leaving home
for work/school. It spends an hour or so building wine and running the
tests in a few different configurations. Adding appinstall would allow
my computer to daily test several applications for installation,
running, etc. This of course isn't as useful as Patchwatcher's testing
of each single patch, but I don't currently have a spare machine that
it can test on. Getting a script and tests ready to do so, however,
would be great for Wine. Besides, I can easily setup the script to
upload the results to a webserver, so we can daily see that each test
is passing. If a test is failing, the information would then be
publicly available for anyone to run 'git bisect'.
Wow, this turned into a really long e-mail, and I've got an exam to
study for. I'd appreciate any comments, and let me know if there are
any questions.
--
-Austin