Re: Make test drill, next steps, call for help with Winetest
I have written attached script to facilitate automated winetest runs. It needs a VirtualBox virtual machine with either Windows or a Wine platform configured to run autorun.inf files. The script downloads the latest winetest binary and puts it in an iso image, along with an autorun file to trigger its execution when the image is 'inserted' into the virtual drive. A snapshot is used to revert the vm to its original state, so no need to worry about winetest screwing up your installation. -Hans #!/bin/bash if [ $# -lt 2 ]; then echo "Usage: $0 <vmname> <tag>" echo "" echo "vmname: Virtual machine to use (must be in suspended state). echo "tag: Tag built from [-.0-9a-zA-Z] to identify test report. exit 1 fi TMPDIR=/tmp/winetest$$ LOGFILE=$TMPDIR/log mkdir -p $TMPDIR/master echo "downloading winetest binary" wget -P $TMPDIR http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/winetest-latest.exe >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi mv $TMPDIR/winetest-latest.exe $TMPDIR/master/winetest.exe echo "creating autorun.inf" cat > $TMPDIR/master/autorun.inf <<_EOF_ [autorun] open=winetest.exe -q -t $2 _EOF_ echo "creating cd image" mkisofs -o $TMPDIR/winetest.iso $TMPDIR/master >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "checking vm $1 state" VBoxManage showvminfo $1 | grep "^State:" | grep "saved" >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "creating snapshot for vm $1" VBoxManage snapshot $1 take before-winetest >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "starting vm $1" VBoxManage startvm $1 >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "waiting until vm $1 is up and running" RUNNING="false" while [ $RUNNING != "true" ] do VBoxManage showvminfo $1 | grep "^State:" | grep "running" >> $LOGFILE 2>&1 if [ $? -eq 0 ]; then RUNNING="true" fi sleep 1 done echo "attaching cd image to vm $1" VBoxManage controlvm $1 dvdattach $TMPDIR/winetest.iso >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "sleeping while vm $1 runs winetest" sleep 1800 # 30 minutes should be enough for winetest to finish echo "suspending vm $1" VBoxManage controlvm $1 savestate >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "waiting until vm $1 is suspended" SAVED="false" while [ $SAVED != "true" ] do VBoxManage showvminfo $1 | grep "^State:" | grep "saved" >> $LOGFILE 2>&1 if [ $? -eq 0 ]; then SAVED="true" fi sleep 1 done echo "reverting vm $1 to previous state" VBoxManage snapshot $1 discardcurrent -all >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi rm -rf $TMPDIR exit 0
On Mon, 15 Oct 2007, Hans Leidekker wrote: [...]
I have written attached script to facilitate automated winetest runs. It needs a VirtualBox virtual machine with either Windows or a Wine platform configured to run autorun.inf files. worry about winetest screwing up your installation. [...]
That's something I've wanted to do for some time and this finally spured me to action. So here's a script that will do essentially the same thing but with VMware Workstation. Like your script it downloads winetest using wget, then generates an ISO with an autorun.inf file so the tests will start automatically in the VM. It also uses snapshots so the state of the virtual machine is preserved. Typically you would use it like this: * I have not found a way to 'insert' the CD if the cdrom device is not already connected in the VM. So you must first ensure that's the case. You can do so in a 'winetest' snapshot, the script will automatically switch to that snapshot. * Creating a winetest snapshot is probably a good idea anyway, if only to ensure it's a relatively clean environment. * Then run the script as follows: winetest /path/to/virtual/machine.vmx --tag mytests --no-submit The --no-submit option is so that the tests don't submit their results to http://test.winehq.org/. The goal here is to avoid sending lots of duplicate results while you're experimenting. Once you're confident things work just as expected, remove the option. There are other options to specify which version of winetest.exe to grab, to set the timeout on the tests execution, etc. Things still on the todo list: * the script also grabs the winetest.exe signature and attempts to verify it. But I don't know where to find the corresponding public key (651FD487) so I don't know if this really works. * the script gets the 'latest' winetest.exe version by default, but when there are compilation problems I guess this may stay stuck at the same revision for some time. So it would be nice if the script could know which version it's getting so it could skip the tests if that version is too old, or has already been tested. * I wonder if this works with VMware Server? It would be nice as Server is free. The script uses snapshots quite a bit though so it may not work. Let me know how it goes if you try. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Any sufficiently advanced Operating System is indistinguishable from Linux
On Saturday 27 October 2007 14:03:02 Francois Gouget wrote:
That's something I've wanted to do for some time and this finally spured me to action. So here's a script that will do essentially the same thing but with VMware Workstation.
Cool! I think I'll borrow some of your nice extra options. I'm moving towards VirtualBox now, it's FOSS, as good as VMware for my purposes and less painful to keep running on a bleeding edge Linux distro. -Hans
Hans Leidekker wrote:
On Saturday 27 October 2007 14:03:02 Francois Gouget wrote:
That's something I've wanted to do for some time and this finally spured me to action. So here's a script that will do essentially the same thing but with VMware Workstation.
Cool! I think I'll borrow some of your nice extra options. I'm moving towards VirtualBox now, it's FOSS, as good as VMware for my purposes and less painful to keep running on a bleeding edge Linux distro.
I achieved much the same thing by putting a wget script in the Auto-startup folder of Windows. Then I copy in this vmware windows virtual machine and start it. Then Windows itself downloads winetest and upon completion, does a shutdown -h now. (Via cygwin.) I often find cygwin a relief when working with Windows. Just FYI. regards, Jakob
On Saturday 27 October 2007 22:29:00 Jakob Eriksson wrote:
I often find cygwin a relief when working with Windows.
Yes, I also run cygwin on my Windows guests. I have openssh running on them and export the Wine tree to the guests via samba so I can do: $ make crosstest $ ssh vm /cygdrive/w/dlls/wininet/tests/wininet_crosstest.exe ftp from my Linux development host. This works for most tests and it gives me a fast edit-compile-test cycle because I don't have to leave the shell I'm working in. -Hans
On Sat, 27 Oct 2007, Jakob Eriksson wrote: [...]
I achieved much the same thing by putting a wget script in the Auto-startup folder of Windows. Then I copy in this vmware windows virtual machine and start it. Then Windows itself downloads winetest and upon completion, does a shutdown -h now. (Via cygwin.) I often find cygwin a relief when working with Windows.
While I prefer the autorun approach because it has fewer dependencies on the Windows side and thus allows me to test in as clean a Windows as desired, your approach could be pretty useful for testing on a real Windows machine. Maybe if you post your script with some instructions it will inspire some people to set up automated testing on their real Windows machines. I'm sure it would make the Direct3D guys happy (if the testers have Nvidia ;-) ). -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Any sufficiently advanced Operating System is indistinguishable from Linux
Francois Gouget wrote:
While I prefer the autorun approach because it has fewer dependencies on the Windows side and thus allows me to test in as clean a Windows as desired, your approach could be pretty useful for testing on a real Windows machine. Maybe if you post your script with some instructions it will inspire some people to set up automated testing on their real Windows machines. I'm sure it would make the Direct3D guys happy (if the testers have Nvidia ;-) ).
I forgot... I did a C hack. cc -o runstuff.exe runstuff Then copied runstuff.exe to c:\win95\Start-Menu\Program\Autostart #include <stdlib.h> #include <unistd.h> int main () { chdir ("/tmp"); unlink ("winetest-latest.exe"); system ("wget http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/winetest-latest.exe"); system ("./winetest-latest.exe -q -c -t JakobAuto"); system ("shutdown -h now"); return 0; } About virtual machines starting in a clean state, nothing stops anyone from booting linux in between and "dd if=windows_part.bin of=/dev/hdb1" and then reboot into Windows. This way you could get consistent Windows regression testing on real hardware. regards, Jakob
On Sa, 2007-10-27 at 14:03 +0200, Francois Gouget wrote:
Things still on the todo list: * the script also grabs the winetest.exe signature and attempts to verify it. But I don't know where to find the corresponding public key (651FD487) so I don't know if this really works.
Paul's homepage: http://www.astro.gla.ac.uk/users/paulm/ $ gpg --list-keys paulm pub 1024D/BF0F03E9 2000-11-07 uid Paul Millar (Physics & Astronomy) sub 1024g/1034BACC 2000-11-07 pub 1024D/651FD487 2002-05-11 uid quisquiliae (automatic program for testing WINE) sub 1024g/420F3FE4 2002-05-11 (The email-address was stripped) My script loads the signature and verify the winetest-latest.exe. When gpg failed to verify the binary, a new winetest is available. The new binary is downloaded and checked again. When the check was ok, the winetest-lastest is copied to the destination and qemu is started for every image. To bad, that qemu 0.9.0 crash on "ntdll_text.exe exception" ... -- By by ... Detlef
On Sun, 28 Oct 2007, Detlef Riekenberg wrote:
On Sa, 2007-10-27 at 14:03 +0200, Francois Gouget wrote:
Things still on the todo list: * the script also grabs the winetest.exe signature and attempts to verify it. But I don't know where to find the corresponding public key (651FD487) so I don't know if this really works.
Paul's homepage: http://www.astro.gla.ac.uk/users/paulm/
$ gpg --list-keys paulm pub 1024D/BF0F03E9 2000-11-07 uid Paul Millar (Physics & Astronomy) sub 1024g/1034BACC 2000-11-07
When I run 'gpg --list-keys paulm' all I get is this first key, but the one I need is the one below... Same thing if I go to the http://wwwkeys.pgp.net/ URL listed on his page.
pub 1024D/651FD487 2002-05-11 uid quisquiliae (automatic program for testing WINE) sub 1024g/420F3FE4 2002-05-11 [...] My script loads the signature and verify the winetest-latest.exe. When gpg failed to verify the binary, a new winetest is available. The new binary is downloaded and checked again. When the check was ok, the winetest-lastest is copied to the destination and qemu is started for every image.
Sounds like a good idea.
To bad, that qemu 0.9.0 crash on "ntdll_text.exe exception" ...
Maybe you can try VirtualBox. Apparently it's GPL... Though if it's incompatible with qemu it would force you to make a choice between them. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Avoid the Gates of Hell - use Linux.
On Sat, 27 Oct 2007, Francois Gouget wrote: [...]
There are other options to specify which version of winetest.exe to grab, to set the timeout on the tests execution, etc.
Things still on the todo list: * the script also grabs the winetest.exe signature and attempts to verify it. But I don't know where to find the corresponding public key (651FD487) so I don't know if this really works.
Hmm, as pointed out by Detlef, there a link on Paul's home page pointing to his public key and the required key is in there. Then it's just a matter of telling gpg to use it which I have now done and the signature checking code seems to work as intended.
* the script gets the 'latest' winetest.exe version by default, but when there are compilation problems I guess this may stay stuck at the same revision for some time. So it would be nice if the script could know which version it's getting so it could skip the tests if that version is too old, or has already been tested.
The script can now use a local cache containing the already downloaded files, and can be told to skip the tests if it finds that a file has not changed since the last time it has been used in a given vm+snapshot combination. In fact I've been able to verify that it works in real life since winetest has not been updated since last monday :-( Compilation problems? I've attached the latest version of the script. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Hiroshima '45 - Czernobyl '86 - Windows '95
Francois Gouget wrote:
On Sat, 27 Oct 2007, Francois Gouget wrote: In fact I've been able to verify that it works in real life since winetest has not been updated since last monday :-( Compilation problems?
Yeah, I guess due to the new d3dx8 tests. Have a look at http://quisquiliae.physics.gla.ac.uk/crossbuild-logs/ for more details. -- Cheers, Paul.
participants (5)
-
Detlef Riekenberg -
Francois Gouget -
Hans Leidekker -
Jakob Eriksson -
Paul Vriens