Introducing WineTestBot
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service. Inspired by all the talk about testing during WineConf 2009, I set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. In its current state you can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run it in VMs of your choice. So, how does this work in practice? First, go to https://winetestbot.geldorp.nl and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload an .exe from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page. The tests are run on VMware ESX VMs, so it's probably not all that useful to run sound and DirectDraw tests on them. Also, no Win9x VMs are available. I haven't set up 64-bit VMs yet, but those should be easy to add. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway. For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines. The plan for the future is to extend this a bit. Speaking from my own experience, I know that programmers like to be as lazy as possible. Which means that having to cross-compile your test before you can send it off to the bot will be seen as a bit of a hurdle. So the next step will be to allow you to upload the git diff instead of a Windows executable. The bot will apply your diff to a git tree and take care of creating the cross-compiled Windows executable, which it will then run on the Windows VMs. Step 3 will be to have the bot watch wine-patches for anything that might impact the tests and do the apply diff/cross-compile/run thing automatically. Obviously I'll be looking at Dan's patch watcher for this 3rd step. Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first. I'd like to thank Paul Vriens who acted as beta tester and made some valuable suggestions. I welcome feedback so if you have any ideas on how to improve this thing further please let me know. Ge.
On 11/30/2009 10:05 AM, Greg Geldorp wrote:
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service.
Inspired by all the talk about testing during WineConf 2009, I set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. In its current state you can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run it in VMs of your choice.
So, how does this work in practice? First, go to https://winetestbot.geldorp.nl and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload an .exe from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page.
The tests are run on VMware ESX VMs, so it's probably not all that useful to run sound and DirectDraw tests on them. Also, no Win9x VMs are available. I haven't set up 64-bit VMs yet, but those should be easy to add. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway.
For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines.
The plan for the future is to extend this a bit. Speaking from my own experience, I know that programmers like to be as lazy as possible. Which means that having to cross-compile your test before you can send it off to the bot will be seen as a bit of a hurdle. So the next step will be to allow you to upload the git diff instead of a Windows executable. The bot will apply your diff to a git tree and take care of creating the cross-compiled Windows executable, which it will then run on the Windows VMs. Step 3 will be to have the bot watch wine-patches for anything that might impact the tests and do the apply diff/cross-compile/run thing automatically. Obviously I'll be looking at Dan's patch watcher for this 3rd step.
Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first.
I'd like to thank Paul Vriens who acted as beta tester and made some valuable suggestions. I welcome feedback so if you have any ideas on how to improve this thing further please let me know.
Ge.
Let me be the first to thank Ge for this awesome piece of work. -- Cheers, Paul.
On Mon, Nov 30, 2009 at 3:21 AM, Paul Vriens <paul.vriens.wine(a)gmail.com> wrote:
On 11/30/2009 10:05 AM, Greg Geldorp wrote:
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service.
Inspired by all the talk about testing during WineConf 2009, I set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. In its current state you can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run it in VMs of your choice.
So, how does this work in practice? First, go to https://winetestbot.geldorp.nl and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload an .exe from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page.
The tests are run on VMware ESX VMs, so it's probably not all that useful to run sound and DirectDraw tests on them. Also, no Win9x VMs are available. I haven't set up 64-bit VMs yet, but those should be easy to add. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway.
For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines.
The plan for the future is to extend this a bit. Speaking from my own experience, I know that programmers like to be as lazy as possible. Which means that having to cross-compile your test before you can send it off to the bot will be seen as a bit of a hurdle. So the next step will be to allow you to upload the git diff instead of a Windows executable. The bot will apply your diff to a git tree and take care of creating the cross-compiled Windows executable, which it will then run on the Windows VMs. Step 3 will be to have the bot watch wine-patches for anything that might impact the tests and do the apply diff/cross-compile/run thing automatically. Obviously I'll be looking at Dan's patch watcher for this 3rd step.
Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first.
I'd like to thank Paul Vriens who acted as beta tester and made some valuable suggestions. I welcome feedback so if you have any ideas on how to improve this thing further please let me know.
Ge.
Let me be the first to thank Ge for this awesome piece of work.
Seconded! On a slightly related note, would it be possible to add a 16-bit display VM? I finally got around to testing wine on a 16-bit display again, and several things have regressed. It would be good to test the testsuite on windows in that configuration to see what happens. -- -Austin
From: Austin English [mailto:austinenglish(a)gmail.com]
On a slightly related note, would it be possible to add a 16-bit display VM? I finally got around to testing wine on a 16-bit display again, and several things have regressed. It would be good to test the testsuite on windows in that configuration to see what happens.
I've added a VM that shows up on test.winehq.org as wtb-wxpprodpy16. Ge.
Let me be the first to thank Ge for this awesome piece of work.
Seconded!
It's really quite slick, for anyone that hasn't used it. It's a very nice way to quickly feel comfortable that the test you just wrote actually works in more places than just your mind. Thanks again! Cheers, Jeremy
This sounds great, one question though. Is there a way to contribute to your cloud so that members of the community could grow your windows vm testing army? On Mon, Nov 30, 2009 at 4:05 PM, Greg Geldorp <ggeldorp(a)vmware.com> wrote:
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service.
Inspired by all the talk about testing during WineConf 2009, I set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. In its current state you can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run it in VMs of your choice.
So, how does this work in practice? First, go to https://winetestbot.geldorp.nl and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload an .exe from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page.
The tests are run on VMware ESX VMs, so it's probably not all that useful to run sound and DirectDraw tests on them. Also, no Win9x VMs are available. I haven't set up 64-bit VMs yet, but those should be easy to add. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway.
For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines.
The plan for the future is to extend this a bit. Speaking from my own experience, I know that programmers like to be as lazy as possible. Which means that having to cross-compile your test before you can send it off to the bot will be seen as a bit of a hurdle. So the next step will be to allow you to upload the git diff instead of a Windows executable. The bot will apply your diff to a git tree and take care of creating the cross-compiled Windows executable, which it will then run on the Windows VMs. Step 3 will be to have the bot watch wine-patches for anything that might impact the tests and do the apply diff/cross-compile/run thing automatically. Obviously I'll be looking at Dan's patch watcher for this 3rd step.
Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first.
I'd like to thank Paul Vriens who acted as beta tester and made some valuable suggestions. I welcome feedback so if you have any ideas on how to improve this thing further please let me know.
Ge.
Hi Andrew,
From: Andrew Fenn
This sounds great, one question though. Is there a way to contribute to your cloud so that members of the community could grow your windows vm testing army?
Interesting idea, it's not something I considered when designing this. The bot currently depends on a VMware specific API. That API, VIX, supports VMware vSphere (ESX), VMware Workstation and VMware Server. I believe Workstation can be controlled only locally, but vSphere and Server are network-capable. So it wouldn't be a big deal to add support for remote vSphere and Server VMs (btw, Server is free as in beer). Other VMs like VirtualBox would take more work to support. If anyone is serious about contributing VMs please contact me and we'll see how to integrate them. Ge.
participants (5)
-
Andrew Fenn -
Austin English -
Greg Geldorp -
Jeremy White -
Paul Vriens