Hey all, I wanted to run my proposal for GSoC by everybody for some feedback and commentary, especially since student apps start in just 10 days. Any advice is greatly appreciated.
GSoC 2010 Proposal: Buildbot Automated Testing System For those who aren't aware, http://buildbot.net/ provides an automated testing suite. I propose to set up a Buildbot system for Wine, using both a periodic scheduler, and a try scheduler. I also intend to create an easy configuration scheme that should keep maintenance fairly simple. In addition the test suite should be modular and easily extensible, ideally building on AppInstall and the existing test suite, as well as providing a means for new tests to be hooked in.
--Buildbot-- To keep from seeming pedantic, I'll be brief with this one. Automating testing suites are awesome and help keep down the number of bugs/regressions that pop up. It'd be really cool if Wine had something like this.
--Schedulers-- The periodic scheduler's purpose is to run the full test suite at particular time intervals. The timing on this one would need to be decided by the project. For example, it could be set up to run the full test suite half a week or so prior to the bi-weekly releases, or perhaps once a night if people care to read the logs that often.
The try scheduler serves a slightly different purpose, namely to test on a patch by patch basis. Initially, this means that the patch would have to be directly uploaded to the test machine, though perhaps integration with patchwatcher may be possible (though I'm told that patchwatcher is fragile at best). Naturally, the full gamut of tests is probably not required for every single patch, and so a more selective (and fast) subset of the test suite can be run on the codebase after each patch, perhaps as limited as a compilation check.
--Configuration-- Currently, one of the downsides of buildbot is that setup and configuration is all done by editing a python script that initializes all the various variables. This is clunky and adds a dependency to the person maintaining the system. Since the configuration script is full python, configuration can be moved out to a simple sh file (like most other Linux programs), and parsing done in the current configuration script.
--Test Suite-- I would like to break apart the test suite into several, modular, components that can then be strung together in some order for a particular test run. Naturally, I intend to take advantage of the current test suite, and make use of AppInstall, integrating the two together to provide a comprehensive test suite. The conformance tests are also something that could potentially be included. Tying into the previous section, I'd also like to make it easy to attach new tests / test suites as people come up with new and better ways to stress test various things. This is the portion that may wind up being too much for a single person to do over the summer, though I doubt it. Buildbot appears to have decent support for this kind of thing, so it may just be wrestling it into doing what we need.
Also, I should have a sample buildbot system ready to showcase soon. I'm currently testing it on a machine other than mine, but due to time constraints this weekend, I probably won't be able to send it out until early next week.
Thanks for taking the time to read this. Once again, any advice is greatly appreciated. If you'd like any more detail or information, feel free to ask. It's entirely possible I forgot to include something. I look forward to working with all of you in the months to come.