Appinstall testing guide up on the wiki

[Austin English]

Howdy all,

James asked me to put up a guide for testing applications with
Appinstall. It's on the wiki (which seems to be down at the moment) at
http:/wiki.winehq.org/Appinstall_Testing. I had a couple people read
over it to make sure I didn't miss anything obvious (or omit stuff
that seems obvious to me). If you've got a few minutes and want to
write a test for your favorite downloadable application, give it a
try. AutoHotKey is pretty simple to pick up, and basic tests (e.g.,
making sure app launches and doesn't crash) is pretty simple to write.
Send me your tests and I'll add them to the test suite.

If anything in the guide is unclear, please let me know and I'll
update it. If everything looks good, I'll e-mail wine-users, so users
can test their favorite apps.

For those who haven't followed, Appinstall is my Summer of Code
project to automate application testing using AutoHotKey + a wrapper
shell script.

It handles the process of automatically downloading, running,
verifying and testing various applications under Wine, against
previously verified Windows behavior. It can test for regressions of
popular applications/features, as well as making sure bugs aren't
subtly fixed. The wrapper script handles fresh WINEPREFIX's,
winetricks dependencies, timeouts of runaway/hung tests, and parsing
of the output logs.

Source is at http://winezeug.googlecode.com/svn/trunk/appinstall/

[Steven Edwards]

Hi Austin,

On Sat, Aug 15, 2009 at 2:58 PM, Austin English wrote:
> It handles the process of automatically downloading, running,
> verifying and testing various applications under Wine, against
> previously verified Windows behavior. It can test for regressions of
> popular applications/features, as well as making sure bugs aren't
> subtly fixed. The wrapper script handles fresh WINEPREFIX's,
> winetricks dependencies, timeouts of runaway/hung tests, and parsing
> of the output logs.

I've just given this a cursory look but I like it. I think what we
need to do to insure that its used and that scripts are kept up to
date is encourage wine appdb submission of scripts for applications
that are listed as gold/platinum. My thinking is that if we get good
scripts in appdb then the app maintainer wont have to carry all the
work of regression testing it at each release. If someone finds a
regression they can use the script to save time as they rollback and
test prior winehq releases and work with that app advocate to track
down the issue.

[Matt Perry]

On Sat, Aug 15, 2009 at 11:58 AM, Austin English wrote:
> James asked me to put up a guide for testing applications with
> Appinstall.

The guide looks great. Will there be any service to facilitate sharing
of scripts between users?

Matt

[Austin English]

On Sat, Aug 15, 2009 at 5:47 PM, Matt Perry wrote:
> On Sat, Aug 15, 2009 at 11:58 AM, Austin English wrote:
> > James asked me to put up a guide for testing applications with
> > Appinstall.
>
> The guide looks great. Will there be any service to facilitate sharing
> of scripts between users?

Winezeug can easily handle this, though eventually it should probably
be on winehq somewhere. That was never really decided early on. The
general consensus was, get some code, make sure it works well, and
from there, we'll discuss moving it to winehq.

For now, send tests to me, possibly with a CC to wine-devel, if you
think more people would be interested.

[Nicklas Börjesson]

This looks way cool. If no one else have already started, I'll have a go with PS CS4.

So I looked through the CS2 script and I have two questions:
1. CS4, and many other, currently need some winetricks to install and work.
Can I use winetricks or do I have to break out the downloading and installing of those components and try and do the same in the script language(using run_wait and so forth)? Basically, I guess my question is if I can add winetricks to tools? That way, we only need to fix broken .msi downloads in one place?
2. CS4 is huge, and has approximately one billion files(well maybe not). Is there some special reason all those sha1sum checks aren't in a include file? If not, what should I call the include file? photoshopcs4_inc_sha1sum or something similar?

//Nicklas

[Austin English]

2009/8/16 Nicklas Börjesson :
> This looks way cool. If no one else have already started, I'll have a go with PS CS4.

Excellent. You may want to start with something simpler. CS4 is a
beast, and probably has a lot of hidden bugs. Start with something
that works well, to get used to coding up tests. Then work toward the
hard stuff.
> So I looked through the CS2 script and I have two questions:
> - CS4, and many other, currently need some winetricks to install and work.
>
> Can I use winetricks or do I have to break out the downloading and installing of those components and try and do the same in the script language(using run_wait and so forth)? Basically, I guess my question is if I can add winetricks to tools? That way, we only need to fix broken .msi downloads in one place?

No, actually. Winetricks shouldn't be used to work around wine bugs.
Yes, I know you could argue missing mfc42/corefonts is a wine bug, but
those are pretty big bugs and not ones that will subtly break/be
fixed. What you should do is write the test on windows, test it
thoroughly (watch out for race conditions), then test it on wine. This
often reveals subtle bugs (different window names, etc.) File any
new/missing bugs. If you can work around them while testing for them,
do so. If a bug is fatal, have the test exit. The test should be
written though, so that if the bug is fixed, the test can continue as
if on windows, and report a fixed bug (TODO_FIXED).

You can see plenty of examples of that in the CS 2 test. It checks to
see if the license window appears a second time, like on windows. If
so, it accepts the license a second time. If it doesn't appear, it
reports a TODO_FAILED. There are other examples elsewhere...
> - CS4 is huge, and has approximately one billion files(well maybe not). Is there some special reason all those sha1sum checks aren't in a include file? If not, what should I call the include file? photoshopcs4_inc_sha1sum or something similar?

I suppose you could put them in a include file. AHK supports optional
includes, so you could, for instance, have it sha1sum if the include
file is there, or skip it otherwise. Personally, I don't mind the
clutter, since it keeps everything in one file (and the sha1sum's are
bundled together, and Notepad++ lets me skip that long list.).