Proof of concept - The crashbot

[Bruno Jesus]

Hi, I would like your opinion about a project I may start based on
BuildBot (if the Dan/license allows).

It's named crashbot because it's intended to automate testing of
applications that crash in wine. The main goal is reduce the amount of
time wasted on retesting bugzilla applications that authors abandoned.

It's intented to be as easy as this:
1 - Add a new application by setting it's name, bug number and direct
download url;
2 - Add requirements to run using winetricks;
3 - Set the test schedule (interval in days/weeks between repetitions);
4 - See the results in a web page which will automatically update;
5 - Auto send an email if the application starts successfully.

The crashbot will run in a tmpfs to avoid disk burning and will ever
run applications in a clean prefix. It would extract the downloaded
program if necessary. Wine would be recompiled daily based on latest
git.

In the future I would like to add support to some sort of window
scripting with keyboard and mouse input so it can be used not only to
test startup crashs but any other kind of test (including the program
installation before run). I'm not targeting it to test games for now
because it would run in a very basic computer with an embedded video
board.

Thanks in advance and apologies if this already exists or was
discussed previously.

Best wishes,
Bruno

[Seth Shelnutt]

You might be interested in looking at Appinstall by Austin English. He has
done some great work on using autohotkey and doing application installs. I
started a GSOC project that wasn't accepted last year to improve upon it.
I've made improvements since then but I guess I never pushed it to my
github. I'll see about pushing the changes I've made tonight, in the
meantime here is my github of "apptester" [1]. I was working mostly with
python and using bash for wpkg. wpkg helps to automate the installation of
apps. For the actually testing I was still trying to find out a good way to
test a gui in an automated sort of fashion. Never really find a easy way
because windows menus aren't automation friendly. I also looked at adding
some of MSAA or Microsoft UI Automation support to wine but never got very
far.

Personally I think automated application testing is the next major testing
platform for wine. Test suites are the most important, but application
testing can mean knowing when new software works in wine and from a user
standpoint of view more "interesting" and applicable than what test suites
are getting passed.

-Seth Shelnutt

[1]https://github.com/Shelnutt2/apptester

On Fri, Sep 16, 2011 at 9:42 AM, Bruno Jesus  wrote:
> Hi, I would like your opinion about a project I may start based on
> BuildBot (if the Dan/license allows).
>
> It's named crashbot because it's intended to automate testing of
> applications that crash in wine. The main goal is reduce the amount of
> time wasted on retesting bugzilla applications that authors abandoned.
>
> It's intented to be as easy as this:
> 1 - Add a new application by setting it's name, bug number and direct
> download url;
> 2 - Add requirements to run using winetricks;
> 3 - Set the test schedule (interval in days/weeks between repetitions);
> 4 - See the results in a web page which will automatically update;
> 5 - Auto send an email if the application starts successfully.
>
> The crashbot will run in a tmpfs to avoid disk burning and will ever
> run applications in a clean prefix. It would extract the downloaded
> program if necessary. Wine would be recompiled daily based on latest
> git.
>
> In the future I would like to add support to some sort of window
> scripting with keyboard and mouse input so it can be used not only to
> test startup crashs but any other kind of test (including the program
> installation before run). I'm not targeting it to test games for now
> because it would run in a very basic computer with an embedded video
> board.
>
> Thanks in advance and apologies if this already exists or was
> discussed previously.
>
> Best wishes,
> Bruno
>
> --
> universe* god::bigbang (void); //and then it all began...

[Henri Verbeet]

On 16 September 2011 16:56, Seth Shelnutt  wrote:
> Personally I think automated application testing is the next major testing
> platform for wine. Test suites are the most important, but application
> testing can mean knowing when new software works in wine and from a user
> standpoint of view more "interesting" and applicable than what test suites
> are getting passed.

It's something we've been interested in for some time, but it's fairly
hard to make it work in a reliable way.

[Scott Ritchie]

On 09/16/2011 08:16 AM, Henri Verbeet wrote:
> On 16 September 2011 16:56, Seth Shelnutt  wrote:
> > Personally I think automated application testing is the next major testing
> > platform for wine. Test suites are the most important, but application
> > testing can mean knowing when new software works in wine and from a user
> > standpoint of view more "interesting" and applicable than what test suites
> > are getting passed.
>
> It's something we've been interested in for some time, but it's fairly
> hard to make it work in a reliable way.

I think it's slightly easier to do the inverse -- automate applications
to make sure they don't crash in future versions.

I'm going to tackle something similar on the daily Ubuntu builds with
winetricks' included selftest once Launchpad starts building them again.

Thanks,
Scott Ritchie