Howdy all,
I've been working on the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.
For those interested, here's the first 'real' script I've written. It sets up a test directory (C:\appinstall), downloads sha1sum.exe (if it doesn't exist), verifies its own sha1sum, downloads winscp.exe (portable version), sets up winscp.ini with its settings (I was going to use keyboard/mouse control, but ftp.winehq.org times out to quickly, and adjusting that setting with mouse/keyboard would be a lot more trouble), runs winscp, downloads welcome.msg, closes winscp, sha1sum's welcome.msg, and finally, deletes welcome.msg and winscp.ini.
It already found one bug :-): http://bugs.winehq.org/show_bug.cgi?id=18455. On windows, when selecting the file to download, winscp pops up a dialog to allow you to put in an alternate path to save the file. On windows, it's already selected, so one can simply type a different path. On Wine, it's not selected, breaking the script. It's an easy work around (sending {HOME}, followed by {SHIFT}{END}), it's interesting that the script allows us to find such subtle bugs.
Back to work...
On Wed, May 13, 2009 at 10:27 AM, Austin English austinenglish@gmail.com wrote:
Howdy all,
I've been working on the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.
For those interested, here's the first 'real' script I've written. It sets up a test directory (C:\appinstall), downloads sha1sum.exe (if it doesn't exist), verifies its own sha1sum, downloads winscp.exe (portable version), sets up winscp.ini with its settings (I was going to use keyboard/mouse control, but ftp.winehq.org times out to quickly, and adjusting that setting with mouse/keyboard would be a lot more trouble), runs winscp, downloads welcome.msg, closes winscp, sha1sum's welcome.msg, and finally, deletes welcome.msg and winscp.ini.
This should probably be done in a temporary directory instead of hard-coded to C:\appinstall.
It already found one bug :-): http://bugs.winehq.org/show_bug.cgi?id=18455. On windows, when selecting the file to download, winscp pops up a dialog to allow you to put in an alternate path to save the file. On windows, it's already selected, so one can simply type a different path. On Wine, it's not selected, breaking the script. It's an easy work around (sending {HOME}, followed by {SHIFT}{END}), it's interesting that the script allows us to find such subtle bugs.
Good catch! Hopefully the scripts will find many more bugs like this, as well as keep Wine from regressing.
On Wed, May 13, 2009 at 1:01 PM, James Hawkins truiken@gmail.com wrote:
On Wed, May 13, 2009 at 10:27 AM, Austin English austinenglish@gmail.com wrote:
Howdy all,
I've been working on the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.
For those interested, here's the first 'real' script I've written. It sets up a test directory (C:\appinstall), downloads sha1sum.exe (if it doesn't exist), verifies its own sha1sum, downloads winscp.exe (portable version), sets up winscp.ini with its settings (I was going to use keyboard/mouse control, but ftp.winehq.org times out to quickly, and adjusting that setting with mouse/keyboard would be a lot more trouble), runs winscp, downloads welcome.msg, closes winscp, sha1sum's welcome.msg, and finally, deletes welcome.msg and winscp.ini.
This should probably be done in a temporary directory instead of hard-coded to C:\appinstall.
Of course. I'm working on changing any temporary stuff to %TEMP%\appinstall.
The reason for C:\appinstall (more accurately, %SYSTEMDIR%, so it will work on Windows installs without a C:\ drive) is to cache downloads. That way sha1sum.exe, wget.exe, etc. don't have to be downloaded repeatedly (they are still sha1sum'ed before use). I'm thinking a bit ahead, that way we can test, e.g., Photoshop CS 3, which has a download available, but not one that is scriptable. That way, if someone running the script already has CS 3 in place, the test is run. Else, it will get skipped, saying 'photoshop isn't available, download manually and place in C:\appinstall if you want to test it'.
Austin English wrote:
Howdy all,
I've been working on the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.
You may have already had this idea, but today I was just thinking of the concept of running performance tests alongside Wine's conformance tests. Your test suite would be a great place for that - just integrate scripts for the various benchmarking tools. Bonus points if you can put the output from the benchmark programs themselves into some sort of machine parseable format. Then we could have a rough chart of performance data on various things as development continues; in particular we'd catch bugs that still behave correctly but are drastically inefficient.
Thanks, Scott Ritchie
On Sun, May 17, 2009 at 3:28 AM, Scott Ritchie scott@open-vote.org wrote:
Austin English wrote:
Howdy all,
I've been working on the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.
You may have already had this idea, but today I was just thinking of the concept of running performance tests alongside Wine's conformance tests. Your test suite would be a great place for that - just integrate scripts for the various benchmarking tools. Bonus points if you can put the output from the benchmark programs themselves into some sort of machine parseable format. Then we could have a rough chart of performance data on various things as development continues; in particular we'd catch bugs that still behave correctly but are drastically inefficient.
Possibly, I may look into doing that.
My current plan is to gather all the low hanging fruit. Small, simple applications give us a large range of stuff to test, without adding much difficulty/complexity. Once all that is done, more complex stuff can be added.
-
Austin English -
James Hawkins -
Scott Ritchie