Hey all, I wanted to run my proposal for GSoC by everybody for some feedback and commentary, especially since student apps start in just 10 days. Any advice is greatly appreciated.
GSoC 2010 Proposal: Buildbot Automated Testing System For those who aren't aware, http://buildbot.net/ provides an automated testing suite. I propose to set up a Buildbot system for Wine, using both a periodic scheduler, and a try scheduler. I also intend to create an easy configuration scheme that should keep maintenance fairly simple. In addition the test suite should be modular and easily extensible, ideally building on AppInstall and the existing test suite, as well as providing a means for new tests to be hooked in.
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of bugs/regressions that pop up. It'd be really cool if Wine had something like this.
--Schedulers-- The periodic scheduler's purpose is to run the full test suite at particular time intervals. The timing on this one would need to be decided by the project. For example, it could be set up to run the full test suite half a week or so prior to the bi-weekly releases, or perhaps once a night if people care to read the logs that often.
The try scheduler serves a slightly different purpose, namely to test on a patch by patch basis. Initially, this means that the patch would have to be directly uploaded to the test machine, though perhaps integration with patchwatcher may be possible (though I'm told that patchwatcher is fragile at best). Naturally, the full gamut of tests is probably not required for every single patch, and so a more selective (and fast) subset of the test suite can be run on the codebase after each patch, perhaps as limited as a compilation check.
--Configuration-- Currently, one of the downsides of buildbot is that setup and configuration is all done by editing a python script that initializes all the various variables. This is clunky and adds a dependency to the person maintaining the system. Since the configuration script is full python, configuration can be moved out to a simple sh file (like most other Linux programs), and parsing done in the current configuration script.
--Test Suite-- I would like to break apart the test suite into several, modular, components that can then be strung together in some order for a particular test run. Naturally, I intend to take advantage of the current test suite, and make use of AppInstall, integrating the two together to provide a comprehensive test suite. The conformance tests are also something that could potentially be included. Tying into the previous section, I'd also like to make it easy to attach new tests / test suites as people come up with new and better ways to stress test various things. This is the portion that may wind up being too much for a single person to do over the summer, though I doubt it. Buildbot appears to have decent support for this kind of thing, so it may just be wrestling it into doing what we need.
Also, I should have a sample buildbot system ready to showcase soon. I'm currently testing it on a machine other than mine, but due to time constraints this weekend, I probably won't be able to send it out until early next week.
Thanks for taking the time to read this. Once again, any advice is greatly appreciated. If you'd like any more detail or information, feel free to ask. It's entirely possible I forgot to include something. I look forward to working with all of you in the months to come.
On Thu, Mar 18, 2010 at 9:25 PM, Arjun Comar mandaya@rose-hulman.edu wrote:
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of bugs/regressions that pop up. It'd be really cool if Wine had something like this.
To be clear, Wine *does* have something like this already, but it's kind of fragmented, and only a few people know how to run all the pieces. The value of Buildbot might be that it's a popular system, so more people will be able to participate in running more pieces of, and adding to, our automated build & test. - Dan
Right, sorry, didn't mean to make it seem like there was nothing in place already. Buildbot just seems more easily extensible.
On Fri, Mar 19, 2010 at 12:28 AM, Dan Kegel dank@kegel.com wrote:
On Thu, Mar 18, 2010 at 9:25 PM, Arjun Comar mandaya@rose-hulman.edu wrote:
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of
bugs/regressions
that pop up. It'd be really cool if Wine had something like this.
To be clear, Wine *does* have something like this already, but it's kind of fragmented, and only a few people know how to run all the pieces. The value of Buildbot might be that it's a popular system, so more people will be able to participate in running more pieces of, and adding to, our automated build & test.
- Dan
Sorry about the confusion!
I don't think it would hurt to run tests on the latest revisions coming out. What might be more useful (and more complicated) is to automate the building then running of popular windows programs on the latest versions. When a failure is detected you could then run an automated regression test to find the bad patch and submit it.
On Fri, Mar 19, 2010 at 11:50 AM, Arjun Comar mandaya@rose-hulman.edu wrote:
Right, sorry, didn't mean to make it seem like there was nothing in place already. Buildbot just seems more easily extensible.
On Fri, Mar 19, 2010 at 12:28 AM, Dan Kegel dank@kegel.com wrote:
On Thu, Mar 18, 2010 at 9:25 PM, Arjun Comar mandaya@rose-hulman.edu wrote:
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of bugs/regressions that pop up. It'd be really cool if Wine had something like this.
To be clear, Wine *does* have something like this already, but it's kind of fragmented, and only a few people know how to run all the pieces. The value of Buildbot might be that it's a popular system, so more people will be able to participate in running more pieces of, and adding to, our automated build & test.
- Dan
Sorry about the confusion!
Arjun Comar, Rose-Hulman '12
That's the idea. AppInstall already runs a suite of tests on various windows applications, its a matter of integrating that with the current test suite and automating the whole process.
On Fri, Mar 19, 2010 at 1:05 AM, Andrew Fenn andrewfenn@gmail.com wrote:
I don't think it would hurt to run tests on the latest revisions coming out. What might be more useful (and more complicated) is to automate the building then running of popular windows programs on the latest versions. When a failure is detected you could then run an automated regression test to find the bad patch and submit it.
On Fri, Mar 19, 2010 at 11:50 AM, Arjun Comar mandaya@rose-hulman.edu wrote:
Right, sorry, didn't mean to make it seem like there was nothing in place already. Buildbot just seems more easily extensible.
On Fri, Mar 19, 2010 at 12:28 AM, Dan Kegel dank@kegel.com wrote:
On Thu, Mar 18, 2010 at 9:25 PM, Arjun Comar mandaya@rose-hulman.edu wrote:
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of bugs/regressions that pop up. It'd be really cool if Wine had something like this.
To be clear, Wine *does* have something like this already, but it's kind of fragmented, and only a few people know how to run all the pieces. The value of Buildbot might be that it's a popular system, so more people will be able to participate in running more pieces of, and adding to, our automated build & test.
- Dan
Sorry about the confusion!
Arjun Comar, Rose-Hulman '12
Hi Arjun,
On 19-03-10 05:25, Arjun Comar wrote:
GSoC 2010 Proposal: Buildbot Automated Testing System For those who aren't aware, http://buildbot.net/ provides an automated testing suite. I propose to set up a Buildbot system for Wine, using both a periodic scheduler, and a try scheduler. I also intend to create an easy configuration scheme that should keep maintenance fairly simple. In addition the test suite should be modular and easily extensible, ideally building on AppInstall and the existing test suite, as well as providing a means for new tests to be hooked in.
I'm a bit opposed, since there are 3 or 4 such projects already. So there has to be big advantage to your system if you want to introduce another.
What would be the advantage of buildbot over say, gre' s winetestbot, which allows you to test your patches, the build project dan did, or cxtest?
Cheers, Maarten
On 3/19/2010 10:57, Maarten Lankhorst wrote:
Hi Arjun,
On 19-03-10 05:25, Arjun Comar wrote:
GSoC 2010 Proposal: Buildbot Automated Testing System For those who aren't aware, http://buildbot.net/ provides an automated testing suite. I propose to set up a Buildbot system for Wine, using both a periodic scheduler, and a try scheduler. I also intend to create an easy configuration scheme that should keep maintenance fairly simple. In addition the test suite should be modular and easily extensible, ideally building on AppInstall and the existing test suite, as well as providing a means for new tests to be hooked in.
I'm a bit opposed, since there are 3 or 4 such projects already. So there has to be big advantage to your system if you want to introduce another.
I'm agreed with that. Testing is very important and should be done some way of course, but the priority is to extend unit testsuite I think. There's a lot of missed or incomplete parts and a lot of partially implemented things that aren't tested at all with unit tests. Running high level tests for an applicaiton is good but requires additional time spent to figure out what a regression about. Btw, I don't think AppInstall is widely used in Wine currently, correct me if I'm wrong.
If we really want to have a testing-only project it should be about unit-tests IMO, selected by some priority. Personally I prefer a work on some valuable module accompanied with complete tests for it.
Cheers, Maarten
On Fri, Mar 19, 2010 at 12:57 AM, Maarten Lankhorst m.b.lankhorst@gmail.com wrote:
I'm a bit opposed, since there are 3 or 4 such projects already. So there has to be big advantage to your system if you want to introduce another.
What would be the advantage of buildbot over say, gre' s winetestbot, which allows you to test your patches, the build project dan did, or cxtest?
It's standardized, i.e. it's winning the distributed build-and-test harness race. So people familiar with it from other projects can expand our test infrastructure by adding a new slave without having to learn yet another strange harness that was cobbled together in sh or perl. There's even enough development going on in buildtestbot that it has a decent amount of polish.
(So, for instance, it would make it easy for a bunch of people to add new slaves in their bedrooms. Wait, that doesn't sound right :-) - Dan
-
Andrew Fenn -
Arjun Comar -
Dan Kegel -
Maarten Lankhorst -
Nikolay Sivov