I would like to work on the Regression Suite project, but I'm not quite sure of the scope. Would I be choosing a good 30 or 40 applications that run well on wine, using wpkg to install them, and then test for failures? Because that would seem as though I am only doing installer testing, instead of actual application testing. I would also like to do some sort of automated regression testing for actual features you can easily test, if that would be possible.
Corey
On Wed, Mar 19, 2008 at 3:01 PM, Corey McClymonds galeru@gmail.com wrote:
I would like to work on the Regression Suite project, but I'm not quite sure of the scope. Would I be choosing a good 30 or 40 applications that run well on wine, using wpkg to install them, and then test for failures? Because that would seem as though I am only doing installer testing, instead of actual application testing. I would also like to do some sort of automated regression testing for actual features you can easily test, if that would be possible.
Corey
Let's just say you're going to write an installer tester. How do you define a failure? If you're doing silent installers, do you just look at the return value from the installer? If you installed app foo, and the installer created file bar on Windows with a checksum of qux, but not in Wine, does that count as a failure? How about if a registry key is missing on Wine?
On 3/19/08, Lei Zhang thestig@google.com wrote:
On Wed, Mar 19, 2008 at 3:01 PM, Corey McClymonds galeru@gmail.com wrote:
I would like to work on the Regression Suite project, but I'm not quite sure of the scope. Would I be choosing a good 30 or 40 applications that run well on wine, using wpkg to install them, and then test for failures? Because that would seem as though I am only doing installer testing, instead of actual application testing. I would also like to do some sort of automated regression testing for actual features you can easily test, if that would be possible.
Corey
Let's just say you're going to write an installer tester. How do you define a failure? If you're doing silent installers, do you just look at the return value from the installer? If you installed app foo, and the installer created file bar on Windows with a checksum of qux, but not in Wine, does that count as a failure? How about if a registry key is missing on Wine?
It depends if any functionality is lost. You could argue that if anything is different, then wine isn't implementing something correctly, but I would be fine with it completing and working as intended. I would probably do checks on all 3 things you talked about, and likely anything else that could be easily checked for by a computer.
On Wed, Mar 19, 2008 at 6:07 PM, Corey McClymonds galeru@gmail.com wrote:
On 3/19/08, Lei Zhang thestig@google.com wrote:
On Wed, Mar 19, 2008 at 3:01 PM, Corey McClymonds galeru@gmail.com wrote:
I would like to work on the Regression Suite project, but I'm not quite sure of the scope. Would I be choosing a good 30 or 40 applications that run well on wine, using wpkg to install them, and then test for failures? Because that would seem as though I am only doing installer testing, instead of actual application testing. I would also like to do some sort of automated regression testing for actual features you can easily test, if that would be possible.
Corey
Let's just say you're going to write an installer tester. How do you define a failure? If you're doing silent installers, do you just look at the return value from the installer? If you installed app foo, and the installer created file bar on Windows with a checksum of qux, but not in Wine, does that count as a failure? How about if a registry key is missing on Wine?
It depends if any functionality is lost. You could argue that if anything is different, then wine isn't implementing something correctly, but I would be fine with it completing and working as intended. I would probably do checks on all 3 things you talked about, and likely anything else that could be easily checked for by a computer.
Using a program like AutoHotkey to install a program and test its features would be a better route IMHO.
On Wed, 19 Mar 2008, Austin English wrote: [...]
Using a program like AutoHotkey to install a program and test its features would be a better route IMHO.
I agree but I'm not sure AutoHotKey is quite ready yet. But I think that getting a robust graphical testing environment would be hugely beneficial for Wine.
There are a couple of options to get there: * cxtest http://www.cxtest.org/ It has tools to record and replay clicks. One big drawback is that in its current form it is very Unixy, that is you need bash and VNC in order to record an application installation. This prevents running it on Windows which is really needed imho. On the other hand it can handle both Unix and Windows scripting. It is also missing support for some things like menus and combo boxes. However it has a great backend for automating and reporting the tests, identifying known issues and automatically grouping them together, etc.
* AutoHotKey http://www.autohotkey.com/ It runs on Windows and has a very nice 'record' feature: you can start it in record mode, install your application, stop the recording and presto, you have a script ready to replay. However it generates scripts that are coordinates based (i.e. left-click at 123x456), rather than 'controls' based (i.e. left-click on the 'Ok' button in the 'Step 3 windows'). That worries me as I'm not convinced these coordinates will match well when the application is run in Wine (potential font issues). I suspect that even on Windows there could be issues if the user changes the font size, DPI or stuff like that. However I don't know how commercial graphical regression testing tools handle that so maybe it's not such a big issue. Also AutoHotKey has not yet been integrated in a framework with a good backend.
So one thing that would be very good for Wine would be to improve AutoHotKey so it really does all that we need. It should be good enough that open-source projects would want to use it for their own regression testing on Windows, so that we would just reuse their scripts. However I don't know if it's suitable for a GSoC project. Also it may be more of an AutoHotKey GSoC than a Wine one.
-
Austin English -
Corey McClymonds -
Francois Gouget -
Lei Zhang