On 11/30/2009 10:05 AM, Greg Geldorp wrote:
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service.
Inspired by all the talk about testing during WineConf 2009, I set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. In its current state you can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run it in VMs of your choice.
So, how does this work in practice? First, go to https://winetestbot.geldorp.nl and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload an .exe from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page.
The tests are run on VMware ESX VMs, so it's probably not all that useful to run sound and DirectDraw tests on them. Also, no Win9x VMs are available. I haven't set up 64-bit VMs yet, but those should be easy to add. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway.
For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines.
The plan for the future is to extend this a bit. Speaking from my own experience, I know that programmers like to be as lazy as possible. Which means that having to cross-compile your test before you can send it off to the bot will be seen as a bit of a hurdle. So the next step will be to allow you to upload the git diff instead of a Windows executable. The bot will apply your diff to a git tree and take care of creating the cross-compiled Windows executable, which it will then run on the Windows VMs. Step 3 will be to have the bot watch wine-patches for anything that might impact the tests and do the apply diff/cross-compile/run thing automatically. Obviously I'll be looking at Dan's patch watcher for this 3rd step.
Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first.
I'd like to thank Paul Vriens who acted as beta tester and made some valuable suggestions. I welcome feedback so if you have any ideas on how to improve this thing further please let me know.
Ge.
Let me be the first to thank Ge for this awesome piece of work.