BuildBot is a popular free continuous build-and-test manager.
Dan Kegel and Jeremy White have set up a buildbot instance at http://winehq.org:8010 that monitors http://source.winehq.org/patches, builds wine from git with each new patch, and emails Dan with any errors. He then manually filters out the spurious messages, sends the good ones to the author of the code, cc'ing wine-devel and email@example.com. New flaky test results get sent to wine-tests-results so we can track them and fix them if they repeat.
Dan has contributed two linux boxes (one q9300, one i7) with Nvidia cards, and one (an a8-3850) with AMD graphics, all with proprietary drivers. Austin has contributed another Nvidia system, and Stefan has contributed an iMac with ATI graphics.
It tests first with
- i686 with -Werror
and if that passes, it also tests with the rest of the bots,
- i686 with valgrind on just the related DLL's tests
- i686 with -Werror on AMD graphics
- MacOSX with ATI r500 graphics (offline - tests aren't quite reliable enough there yet)
The buildbot skips tests known to be flaky; see http://code.google.com/p/winezeug/source/browse/trunk/buildbot/dotests_blacklist.txt for the list of flaky tests, and the associated bugzilla entries.
Dan's time to improve Buildbot is limited. He's hoping that, once the buildbot proves its usefulness, other folks will start helping maintain and extend the slave farm.
Dan's To do list:
- Recruit volunteers to run more build slaves
- Move buildbot scripts source tree from winezeug svn to a new git repo next to the one for testbot at wine.git.sourceforge.net
Other things we might want to do
- once we have another slave, turn back on the heap slave
- once we have even more slaves, maybe add a full valgrind check run, but only on releases
- Root out remaining causes of spurious test failures
- Once there have been no spurious emails for five days, cc them automatically to wine-devel
- Add -fstack-protector-all
Add static analysis, perhaps one of http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#C Initial goal would be to disallow // comments. (gcc doesn't help here; -ansi would do it, but that disallows 'inline'.)
- Improve coverage
- investigate blacklisted tests and see if they can be fixed
- figure out which tests can't stand virtual desktops, and run them only on slaves which have real monitors, or something
- Catch new errors that don't happen on every run
- for patches that affect core dlls (e.g. rpcrt4), run a select set of tests more heavily
- Add some way for developers to submit jobs without sending them to wine-patches (there is a way to do this, just need an authenticated web form)
- Add support for ignoring flaky tests in "make test" or wine/test.h? Ideally would work with winetest.exe as well.
- Set up second buildbot master for experimental builds; this one wouldn't cc wine-devel or the developer on failure. Only after showing no failures for, say, two weeks would slaves get promoted from the experimental master to the main master.
- Add a Mac buildslave
- Add a Windows (test-only) buildslave
- Add a clang slave
- Add a latest-gcc slave
- Add a latest-ubuntu slave
For more info:
BuildBot replaces Patchwatcher, which was an earlier bot written by DanKegel to check patches against the Wine ConformanceTests, automatically upon submission. It began to wither because the author became busy and writing a new test bot was simpler than maintaining a maze of custom shell scripts. The code to Patchwatcher can still be browsed or downloaded here.