SOC 2009: Application Test Suite

[Austin English]

Howdy,

I'm planning on applying for Google Summer of Code 2009. I'm pretty
sure most people reading this list already know me, but for those that
don't, I frequently triage bugs in Bugzilla, help users on the forum,
and a few other things. I've also done quite a bit of work testing
wine on more obscure OS's (OpenSolaris, FreeBSD, OS X,  NetBSD,
OpenBSD).

I'd like to implement an application test suite. It's something that's
been discussed for Wine for quite a while, but has never been put into
place. I took a bit of time a while back and hacked couple a couple
quick scripts as a test, and got a script to automatically install
Firefox 3 in Wine using AutoHotKey (http://www.autohotkey.com/). I've
made a couple more scripts as well, but they don't match Windows
behavior because of bug (http://bugs.winehq.org/show_bug.cgi?id=10147)
hint, hint (disclaimer, I haven't tested that script in a while, been
busy with exams, so that bug might be fixed...I don't think it has
though).

My plan to implement it is to make a script, that can be used
standalone, in conjunction with Patchwatcher, or any other script, to
test wine against several real world applications. I make a very quick
proof of concept (attached), which is based off of winetricks. It's
_REALLY_ rough at the moment, (I haven't had time to mess with it
lately), but demonstrates the idea. The script setups a clean
WINEPREFIX, installs autohotkey, removes the Z: symlink for braindead
installers that search the entire hard drive, and sets up a link to
it's install cache (essentially winetrickscache). From there, it makes
a copy of this prefix, so on subsequent installers/applications, we
can restore the clean prefix and save a few seconds.

>From there, so far, it only installs firefox3. The plan is to add
several dozen different tests, for various functions of wine.
Obviously, I'd like to start with platinum rated applications, to
prevent regressions. From there, the current list of
applications/functions to target are:
1. Firefox3 - useful for testing lots of other things, e.g.,
java/flash, etc. Easy to test.
2. Microsoft Office - We've broken this a few times in recent months,
so adding a test in daily git for it would help catch regressions
sooner.
3. World of Warcraft - I don't play, but A) It's really popular, and
B) also gives us a chance to test wininet for regressions.
4. iTunes - this installers been broken a couple times. The bonjour
service should allow for testing services related regressions.
5. Adobe suite - there haven't been too many regressions here that I
can remember, but it's definitely an area we should watch out for.

Of course, we don't want to only test the installer's keystrokes/mouse
inputs, so SHA1 checksumming of installed files would verify against
corrupted files/setupapi regressions/etc. Once a few installer tests
are in and working well, I plan to extend the tests further to test
running the application, e.g, for photoshop by opening a file,
inserting a circle, and saving it. This is a bit harder to pull off in
Autohotkey in a way that's portable, so I'm saving that sort of thing
for later.

Also interesting would be a real application's test suite. Dan
suggested using Google Chrome's, since it's squeaky clean after using
lots of Valgrind and Purify. That would be good, but I'd put that
after getting the test suite to pass in Wine.

This should be very doable using Autohotkey's scripting functionality
(pretty extensive - http://www.autohotkey.com/docs/Functions.htm,
http://www.autohotkey.com/docs/Variables.htm) and good old fashioned
shell scripting. I'm very familiar with bourne scripting, and
currently maintain winetricks while Dan's taking a Wine vacation. I'm
currently working on a build script to build Wine on several OS's, as
well as running and submitting the conformance tests
(http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh). [I
got tired of maintaining the script over several different
OS's/computers]. The upside is it would be easy to add on appinstall
to this daily script. I typically set it running before leaving home
for work/school. It spends an hour or so building wine and running the
tests in a few different configurations. Adding appinstall would allow
my computer to daily test several applications for installation,
running, etc. This of course isn't as useful as Patchwatcher's testing
of each single patch, but I don't currently have a spare machine that
it can test on. Getting a script and tests ready to do so, however,
would be great for Wine. Besides, I can easily setup the script to
upload the results to a webserver, so we can daily see that each test
is passing. If a test is failing, the information would then be
publicly available for anyone to run 'git bisect'.

Wow, this turned into a really long e-mail, and I've got an exam to
study for. I'd appreciate any comments, and let me know if there are
any questions.

-- 
-Austin

[Roderick Colenbrander]

Hi Austin,

Not sure if you are aware of it but there is also cxtest which was written
by codeweavers under the gpl. See http://cxtest.ifne.eu:82/ it seems they
(still?) use it regulary to track regressions. I haven't looked at it and
don't know that autohotkey stuff but how do both differ? Wouldn't it be
better to continue with cxtest? Or perhaps it would be better to use
autohotkey (assuming it is a widely used app) and extend it so that cxtest
can be dropped.

Roderick

On Wed, Mar 25, 2009 at 12:33 AM, Austin English <austinenglish(a)gmail.com>wrote:
> Howdy,
>
> I'm planning on applying for Google Summer of Code 2009. I'm pretty
> sure most people reading this list already know me, but for those that
> don't, I frequently triage bugs in Bugzilla, help users on the forum,
> and a few other things. I've also done quite a bit of work testing
> wine on more obscure OS's (OpenSolaris, FreeBSD, OS X,  NetBSD,
> OpenBSD).
>
> I'd like to implement an application test suite. It's something that's
> been discussed for Wine for quite a while, but has never been put into
> place. I took a bit of time a while back and hacked couple a couple
> quick scripts as a test, and got a script to automatically install
> Firefox 3 in Wine using AutoHotKey (http://www.autohotkey.com/). I've
> made a couple more scripts as well, but they don't match Windows
> behavior because of bug (http://bugs.winehq.org/show_bug.cgi?id=10147)
> hint, hint (disclaimer, I haven't tested that script in a while, been
> busy with exams, so that bug might be fixed...I don't think it has
> though).
>
> My plan to implement it is to make a script, that can be used
> standalone, in conjunction with Patchwatcher, or any other script, to
> test wine against several real world applications. I make a very quick
> proof of concept (attached), which is based off of winetricks. It's
> _REALLY_ rough at the moment, (I haven't had time to mess with it
> lately), but demonstrates the idea. The script setups a clean
> WINEPREFIX, installs autohotkey, removes the Z: symlink for braindead
> installers that search the entire hard drive, and sets up a link to
> it's install cache (essentially winetrickscache). From there, it makes
> a copy of this prefix, so on subsequent installers/applications, we
> can restore the clean prefix and save a few seconds.
>
> From there, so far, it only installs firefox3. The plan is to add
> several dozen different tests, for various functions of wine.
> Obviously, I'd like to start with platinum rated applications, to
> prevent regressions. From there, the current list of
> applications/functions to target are:
> 1. Firefox3 - useful for testing lots of other things, e.g.,
> java/flash, etc. Easy to test.
> 2. Microsoft Office - We've broken this a few times in recent months,
> so adding a test in daily git for it would help catch regressions
> sooner.
> 3. World of Warcraft - I don't play, but A) It's really popular, and
> B) also gives us a chance to test wininet for regressions.
> 4. iTunes - this installers been broken a couple times. The bonjour
> service should allow for testing services related regressions.
> 5. Adobe suite - there haven't been too many regressions here that I
> can remember, but it's definitely an area we should watch out for.
>
> Of course, we don't want to only test the installer's keystrokes/mouse
> inputs, so SHA1 checksumming of installed files would verify against
> corrupted files/setupapi regressions/etc. Once a few installer tests
> are in and working well, I plan to extend the tests further to test
> running the application, e.g, for photoshop by opening a file,
> inserting a circle, and saving it. This is a bit harder to pull off in
> Autohotkey in a way that's portable, so I'm saving that sort of thing
> for later.
>
> Also interesting would be a real application's test suite. Dan
> suggested using Google Chrome's, since it's squeaky clean after using
> lots of Valgrind and Purify. That would be good, but I'd put that
> after getting the test suite to pass in Wine.
>
> This should be very doable using Autohotkey's scripting functionality
> (pretty extensive - http://www.autohotkey.com/docs/Functions.htm,
> http://www.autohotkey.com/docs/Variables.htm) and good old fashioned
> shell scripting. I'm very familiar with bourne scripting, and
> currently maintain winetricks while Dan's taking a Wine vacation. I'm
> currently working on a build script to build Wine on several OS's, as
> well as running and submitting the conformance tests
> (http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh). [I
> got tired of maintaining the script over several different
> OS's/computers]. The upside is it would be easy to add on appinstall
> to this daily script. I typically set it running before leaving home
> for work/school. It spends an hour or so building wine and running the
> tests in a few different configurations. Adding appinstall would allow
> my computer to daily test several applications for installation,
> running, etc. This of course isn't as useful as Patchwatcher's testing
> of each single patch, but I don't currently have a spare machine that
> it can test on. Getting a script and tests ready to do so, however,
> would be great for Wine. Besides, I can easily setup the script to
> upload the results to a webserver, so we can daily see that each test
> is passing. If a test is failing, the information would then be
> publicly available for anyone to run 'git bisect'.
>
> Wow, this turned into a really long e-mail, and I've got an exam to
> study for. I'd appreciate any comments, and let me know if there are
> any questions.
>
> --
> -Austin

[Jeremy White]

Roderick Colenbrander wrote:
> Hi Austin,
>
> Not sure if you are aware of it but there is also cxtest which was
> written by codeweavers under the gpl. See http://cxtest.ifne.eu:82/ it
> seems they (still?) use it regulary to track regressions. I haven't
> looked at it and don't know that autohotkey stuff but how do both
> differ? Wouldn't it be better to continue with cxtest? Or perhaps it
> would be better to use autohotkey (assuming it is a widely used app) and
> extend it so that cxtest can be dropped.

Actually, afair, the thinking was that autohotkey would be a nice
complement to cxtest.  That is, if you can write an autohotkey test
that works nicely on a single application on Windows, in theory, you
can drop it into a cxtest frame work and get all of the error tracking
and automated bisecting and fun stuff we've done with cxtest.

Also, I haven't pursued it a lot, but we now have cxtest running
the Mozilla and OpenOffice test suite against Wine.

You can see those results here:
  http://cxtest.ifne.eu:82/wine-error-ratio

They're hard to understand, but the IFNE guys would *love* to
have someone take an interest in it and explain it to them.

The numbers are somewhat encouraging; roughly 50,000 tests run,
and about 97% of them correctly (the majority of the delta
are just unknown).  What's nice is that, in theory, cxtest can
detect any regressions any auto-bisect for guilty patches.

Cheers,

Jeremy

[Vincent Povirk]

AutoHotkey is a Windows program so any tests written in it can also be
run on Windows. In fact, if you wrote everything in AHK, the entire
framework would be portable to any win32 system. But that may be more
crazy than you're going for.

I'm told cxtest does not work on Windows.

Vincent Povirk

[Francois Gouget]

On Tue, 24 Mar 2009, Austin English wrote:
[...]
> I'd like to implement an application test suite. It's something that's
> been discussed for Wine for quite a while, but has never been put into
> place.

As mentioned before, check out cxtest. In particular the way they are 
structured, with one script to install the application, and others 
testing particular aspects of it. You will also need to support 
dependencies at some point.

But also check out Lei Zhang's experiments with AutoHotKey:

http://code.google.com/p/yawt/


[...]
> My plan to implement it is to make a script, that can be used
> standalone, in conjunction with Patchwatcher, or any other script, to
> test wine against several real world applications. I make a very quick
> proof of concept (attached), which is based off of winetricks.

One very important feature is that:


     A good testing framework *must* work on Windows.


Why? For the same reason that the conformance tests would not be even 
half as useful as they are if they did not run on Windows:

 * Applications should install the same way in Wine and on Windows. Sure 
   there's bound to be some differences but these should be investigated 
   and either fixed, or documented and worked-around in the test script.

 * This would make it possible to write tests on Windows for things that 
   don't work yet in Wine. This can be particularly powerful in 
   conjunction with Bugzilla: just like we sometimes ask for a 
   conformance test, we could ask for an application test for a 
   given Bugzilla report. Then, sprinkle a few todo_wine and we will 
   immediately know when a bug is fixed.

 * This makes it possible for non-Unix users to contribute. Ideally, 
   once we have developped a test for an open-source Windows 
   application, the developers for that application should be able to 
   use our tests and take over their maintenance.

So AutoHotKey is a step towards that goal. But you will need some 
infrastructure around it to download installers, setting up CDs, maybe 
start and stop virtual machines, and it is this part that worries me. In 
particular I don't think that bash+wget&co is the most appropriate 
toolset. What to use on Windows is a tricky issue though.

Other important features:
 * You should not need to update the scripts for every little change in 
   Wine.
> 2. Microsoft Office - We've broken this a few times in recent months,
> so adding a test in daily git for it would help catch regressions
> sooner.

Just as a note, CodeWeavers tests Microsoft Office every day with 
cxtest. However this is with the CrossOver codebase so we only detect 
WineHQ regressions when it gets merged after each release (so roughly 
every two weeks). In any case it would be nice if these were detected 
sooner and more 'upstream'<g>.
> 3. World of Warcraft - I don't play, but A) It's really popular, and
> B) also gives us a chance to test wininet for regressions.
> 4. iTunes - this installers been broken a couple times. The bonjour
> service should allow for testing services related regressions.

The problem with these two is that they change every other day, so that 
you're likely to end up having to update your script pretty often. Also, 
as far as I understand it, Wow is really huge and regularly insists on 
downloading big update patches (which of course you won't have been able 
to checksum). This is not to say it shouldn't be done though.


Some pointers to past discussions on the subject (far from exhaustive):
http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html
http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html


-- 
Francois Gouget <fgouget(a)free.fr>              http://fgouget.free.fr/
  Nouvelle version : les anciens bogues ont été remplacés par de nouveaux.

[Austin English]

On Wed, Mar 25, 2009 at 9:47 AM, Francois Gouget <fgouget(a)free.fr> wrote:
> On Tue, 24 Mar 2009, Austin English wrote:
> [...]
> > I'd like to implement an application test suite. It's something that's
> > been discussed for Wine for quite a while, but has never been put into
> > place.
>
> As mentioned before, check out cxtest. In particular the way they are
> structured, with one script to install the application, and others
> testing particular aspects of it. You will also need to support
> dependencies at some point.

Yes, I've looked at cxtest a bit. I like Jeremy's line of
thinking...keep cxtest for its infrastructure, but move toward AHK,
which gains us windows compatibility.
> But also check out Lei Zhang's experiments with AutoHotKey:
>
> http://code.google.com/p/yawt/

I've looked a bit at this code, should be useful.
> [...]
> > My plan to implement it is to make a script, that can be used
> > standalone, in conjunction with Patchwatcher, or any other script, to
> > test wine against several real world applications. I make a very quick
> > proof of concept (attached), which is based off of winetricks.
>
> One very important feature is that:
>
>     A good testing framework *must* work on Windows.

Exactly why Cxtest is not sufficient.
> Why? For the same reason that the conformance tests would not be even
> half as useful as they are if they did not run on Windows:
>
>  * Applications should install the same way in Wine and on Windows. Sure
>   there's bound to be some differences but these should be investigated
>   and either fixed, or documented and worked-around in the test script.

Of course. My very brief initial testing already found a couple of
these sorts of bugs.
>  * This would make it possible to write tests on Windows for things that
>   don't work yet in Wine. This can be particularly powerful in
>   conjunction with Bugzilla: just like we sometimes ask for a
>   conformance test, we could ask for an application test for a
>   given Bugzilla report. Then, sprinkle a few todo_wine and we will
>   immediately know when a bug is fixed.

You read my mind :-)...exactly what I'd like to see. AHK is powerful
in this regard as A) it already works well on windows, B) allows for
gui recording of scripts, which can be adjusted manually and
rewritten. This allows an average user to download AHK (no need to
install, has a portable version you can run), fire up foo.exe which
doesn't work on wine, record a simple test, and file a bug. It greatly
simplifies bug triagers' work, especially for applications in foreign
languages (for some reason, there are quite a few Polish and Slovak
bugs in bugzilla...).
>  * This makes it possible for non-Unix users to contribute. Ideally,
>   once we have developped a test for an open-source Windows
>   application, the developers for that application should be able to
>   use our tests and take over their maintenance.

Of course.
> So AutoHotKey is a step towards that goal. But you will need some
> infrastructure around it to download installers, setting up CDs, maybe
> start and stop virtual machines, and it is this part that worries me. In
> particular I don't think that bash+wget&co is the most appropriate
> toolset. What to use on Windows is a tricky issue though.

Right. For downloading, wget for windows exists, as well as several
other small utilities (ftp is builtin). CDs are one that I've been
thinking about, haven't decided on that yet. Virtual machines would be
useful, unfortunately I don't know much about scripting for those
(shouldn't be too difficult to learn. Hopefully a potential mentor
would have some insight <g>).

For that matter, keep in mind that cxtest is a bash script as well
http://cxtest.ifne.eu:82/transfer/cxtest.sh.
> Other important features:
>  * You should not need to update the scripts for every little change in
>   Wine.

Of course. That is something that I've kept in mind, but not an easy
task. It will take careful planning.
> > 2. Microsoft Office - We've broken this a few times in recent months,
> > so adding a test in daily git for it would help catch regressions
> > sooner.
>
> Just as a note, CodeWeavers tests Microsoft Office every day with
> cxtest. However this is with the CrossOver codebase so we only detect
> WineHQ regressions when it gets merged after each release (so roughly
> every two weeks). In any case it would be nice if these were detected
> sooner and more 'upstream'<g>.

Right. The problem is those tests don't come before wine releases, nor
daily, which allows Office to break in binary releases, not just git.
Knowing the problem exists is only part of the fix, and if that takes
2 weeks or more to find, the installer could break for 2-3 WineHQ
releases.
> > 3. World of Warcraft - I don't play, but A) It's really popular, and
> > B) also gives us a chance to test wininet for regressions.
> > 4. iTunes - this installers been broken a couple times. The bonjour
> > service should allow for testing services related regressions.
>
> The problem with these two is that they change every other day, so that
> you're likely to end up having to update your script pretty often. Also,
> as far as I understand it, Wow is really huge and regularly insists on
> downloading big update patches (which of course you won't have been able
> to checksum). This is not to say it shouldn't be done though.

That's why I put them low on the list ;-). WoW would be nice to have,
but not necessary. iTunes would be very useful, and the problem of
updating daily can be avoided by using mirrors. Verify original
install from Apple.com, get SHA1SUM, then get that version from
http://www.filehippo.com/, verifying filesize and sha1sum.
> Some pointers to past discussions on the subject (far from exhaustive):
> http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html
> http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html

Thanks. I've followed these discussions in the past, but a refresher
is always good.

My perception is that this sort of project is needed pretty badly, but
suffers from 2 main problems: No one wants to do it and those who have
tried in the past aimed to high, crashed, and burned. I think I could
do better because:

I want to do it.

I've tested _tons_ of windows applications (all that bugzilla testing
as good for something).

I've got a more pragmatic view of what's possible. I'm not aiming to
make winetest version 2.0 for windows. The AHK scripts will be
runnable on windows, sure. But the main bulk of the script will be
built for Wine/Unix. While that may not be ideal (the ahk scripts may
not be tested daily on windows), it shouldn't be too hard for someone
who wants to implement that on windows to do so. Windows doesn't
change all that often, Wine does, so it follows that we should test it
more often. If the test passes on Windows originally, and a curious
party tests it every month or two (or more likely, when a bug is
found/fixed), that should be plenty.

This wouldn't be the panacea of regression testing for Wine, but would
be a major step in that direction.

-- 
-Austin

[Austin English]

On Wed, Mar 25, 2009 at 11:47 AM, Austin English
<austinenglish(a)gmail.com> wrote:

Here's a quick thrown together script to test firefox3.
> From the looks of things, we could do just about everything in
> autohotkey with enough elbow grease. It would only take a small
> wrapper script to run it under wine, or, if someone wanted to later
> extend it, to windows.

Script is attached. Below is commented more.

-------------------
#!/bin/sh
# sha1sum is available for windows too, neat!
#wine `pwd`/wget.exe ftp://ftp.gnupg.org/gcrypt/binary/sha1sum.exe #
4a578ecd09a2d0c8431bdd8cf3d5c5f3ddcddfc9

wget http://users.ugent.be/~bpuype/cgi-bin/fetch.pl?dl=wget/wget.exe #
801a6938e0289d14078487d10152f8ce39af125c

---------All we really need sh for is the initial wget. That's easy
enough to workaround, or could add an ftp.exe to wine and eliminate
that completely.
> From there, wine can use wget to get the other files:
> wine `pwd`/wget.exe
> http://www.autohotkey.com/download/AutoHotkey104800.zip #
> 593423c075c0e1f58224bd9b8929b14013e09c1a
> wine `pwd`/wget.exe
> "http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.7/win32/en-..."
> # 6452f56cb9808b41c402ea55324eea86b50b3b96

--------This isn't ahk/wine, but we could use ahk's silent installer
or put the exe itself somewhere, as it has no dependencies
unzip -d ahk AutoHotkey104800.zip

--------Again, not ahk/wine. We could have .ahk files stored on a
sever and just wget it instead.
cat > ahk/firefox.ahk <<_EOF_

Run, "C:\users\\$USER\My Documents\Firefox Setup 3.0.7.exe"
WinWait, Mozilla Firefox Setup, Welcome to the Mozilla Firefox Setup
Wizard, 100, ,
Send, {ENTER}
WinWait, Mozilla Firefox Setup, Choose the type of setup you prefer, 100, ,
Send, {ENTER}
WinWait, Mozilla Firefox Setup, Click Install to continue, 100, ,
Send, {ENTER}
WinWait, Mozilla Firefox Setup, Completing the Mozilla Firefox Setup
Wizard, 100, ,
Send, {SPACE}{ENTER}
exit
_EOF_

cd ahk
wine autohotkey.exe firefox.ahk

---------Here would be sha1sum running to verify firefox.exe's sha1sum
for instance, or perhaps a couple other files.
cd ..

rm -rf .wine ahk
rm wget.exe
rm AutoHotkey104800.zip
rm "Firefox Setup 3.0.7.exe"

exit
-----------------------------------------

It's a quick demo, but it gives an idea of what we can do with ahk.
With a proper implementation design, we can setup a tester than can
run on Windows or Wine, is easily extensible, and can rapidly test
real world applications.

-- 
-Austin

[Vincent Povirk]

On Wed, Mar 25, 2009 at 6:37 PM, Austin English <austinenglish(a)gmail.com> wrote:
> From the looks of things, we could do just about everything in
> autohotkey with enough elbow grease. It would only take a small
> wrapper script to run it under wine, or, if someone wanted to later
> extend it, to windows.
> Your current script doesn't do anything that can't be done in autohotkey.

[Austin English]

On Wed, Mar 25, 2009 at 7:42 PM, Vincent Povirk
<madewokherd+8cd9(a)gmail.com> wrote:
> On Wed, Mar 25, 2009 at 6:37 PM, Austin English <austinenglish(a)gmail.com> wrote:
> > From the looks of things, we could do just about everything in
> > autohotkey with enough elbow grease. It would only take a small
> > wrapper script to run it under wine, or, if someone wanted to later
> > extend it, to windows.
> Your current script doesn't do anything that can't be done in autohotkey.

I know (aside from the initial wget for the wget.exe file...I don't
see anything similar for AHK, unless we have that file already
downloaded, which is easily doable).

I was more interested in a quick proof of concept. The flexibility of
autohotkey allows for lots of different methods for doing such things,
and a method that involves the majority of code being run under AHK
itself (and therefore on wine AND windows) is very possible with
careful planning, which is part of what makes this proposal so
interesting.

-- 
-Austin

[Vít Hrachový]

Austin English wrote:
> I was more interested in a quick proof of concept. The flexibility of
> autohotkey allows for lots of different methods for doing such things,
> and a method that involves the majority of code being run under AHK
> itself (and therefore on wine AND windows) is very possible with
> careful planning, which is part of what makes this proposal so
> interesting.

Hi Austin,
IMO it would be nice to make clear line between *installation* testing 
and *functional* testing.

The first would answer the AppDB question if the program is installable 
(and uninstallable).

The second could try to cover some basic aspects of program 
functionality, say - Firefox is able to view http://winehq.org or so.

AutoHotkey even supports screenshots matching.

See *-repository/*/resources for some library of existing primitive 
AutoHotkey install/uninstall scripts. There are some game demos too.

http://winebot.sandbox.cz/tracker/browser/repositories

For example Wizardry 8 demo download is available here and should work:
http://wood.sandbox.cz/wizardry8-demo/wizardry8demo.exe

Heroes of Might and Magic III demo installs and runs well, too (beware 
of troubles with PulseAudio).

 From top of my head, Widelands and FreeCiv would be another nice 
candidates for free downloadable games with easy automation.

Hope it helps.
Hark

[Austin English]

2009/3/26 Vít Hrachový <vit.hrachovy(a)sandbox.cz>:
> Austin English wrote:
> > I was more interested in a quick proof of concept. The flexibility of
> > autohotkey allows for lots of different methods for doing such things,
> > and a method that involves the majority of code being run under AHK
> > itself (and therefore on wine AND windows) is very possible with
> > careful planning, which is part of what makes this proposal so
> > interesting.
>
> Hi Austin,
> IMO it would be nice to make clear line between *installation* testing and
> *functional* testing.

Agreed. However, I don't want to bite off more than I can chew. I
think getting some basic installation scripts that we can put into
constant use would be better than trying to get a script to test that
a game works (which could break on different desktop environments,
video cards/drivers/etc.).
> The second could try to cover some basic aspects of program functionality,
> say - Firefox is able to view http://winehq.org or so.
>
> AutoHotkey even supports screenshots matching.

Possible, but that seems a bit flaky, especially when testing windows vs wine.
> See *-repository/*/resources for some library of existing primitive
> AutoHotkey install/uninstall scripts. There are some game demos too.
>
> http://winebot.sandbox.cz/tracker/browser/repositories

Thanks!
> Heroes of Might and Magic III demo installs and runs well, too (beware of
> troubles with PulseAudio).

Exactly why we need to go for things that work well across the board.
Concentrate on the low hanging fruit, then work toward more difficult
stuff from there.

-- 
-Austin

[Scott Ritchie]

Austin English wrote:
> On Wed, Mar 25, 2009 at 7:42 PM, Vincent Povirk
> <madewokherd+8cd9(a)gmail.com> wrote:
> > On Wed, Mar 25, 2009 at 6:37 PM, Austin English <austinenglish(a)gmail.com> wrote:
> > > From the looks of things, we could do just about everything in
> > > autohotkey with enough elbow grease. It would only take a small
> > > wrapper script to run it under wine, or, if someone wanted to later
> > > extend it, to windows.
> > Your current script doesn't do anything that can't be done in autohotkey.
>
> I know (aside from the initial wget for the wget.exe file...I don't
> see anything similar for AHK, unless we have that file already
> downloaded, which is easily doable).
>
> I was more interested in a quick proof of concept. The flexibility of
> autohotkey allows for lots of different methods for doing such things,
> and a method that involves the majority of code being run under AHK
> itself (and therefore on wine AND windows) is very possible with
> careful planning, which is part of what makes this proposal so
> interesting.

You don't need wget - just have autohotkey install the web browser, open 
a webpage, and then download the files by point and click ;)

Thanks,
Scott Ritchie

[Austin English]

On Thu, Mar 26, 2009 at 6:33 AM, Scott Ritchie <scott(a)open-vote.org> wrote:
> You don't need wget - just have autohotkey install the web browser, open a
> webpage, and then download the files by point and click ;)

That's got a large potential for breaking on different resolutions/setups.

Vincent pointed out to me that AHK already has something
(http://www.autohotkey.com/docs/commands/UrlDownloadToFile.htm) which
should work on Windows/Wine.

-- 
-Austin