WineTestBot
Or, for those who like to be buzzword compliant, the Wine Testing Cloud Service.
Inspired by all the talk about testing during WineConf 2009, I [Ge van Geldorp] set out to fill a gap in Wine testing. Most of the Wine developers don't have access to a battery of Windows machines to run the tests they write/modify on. As a result, there are quite a few regressions in the tests themselves. The WineTestBot provides a whole bunch of Windows machines, ready to accept and run your modified tests. You can submit Windows executables (most likely cross-compiled on a Linux box) to the bot, which will run them in VMs of your choice. Alternatively, you can submit source patches, these will be applied to a clean git tree, the appropriate test will be cross-compiled and then run in the VMs.
So, how does this work in practice? First, go to https://testbot.winehq.org and click on "Register" to register an account. Once your account is approved (you'll receive a notification via email) you can log in. After log in, the menu in the sidebar changes to include an item "Submit job". This allows you to upload a Windows .exe or a "git diff"-style patch from your local system to the bot. You can pick the VMs (Windows versions) you want to run the test on. From the "Home" page you can see the status of your job, by clicking on the job id you get a bit more info about the progress. Once the test is completed, the stdout generated by the test is retrieved from each VM and made available via that web page.
The bot also scans wine-patches submissions for changes to tests. If it finds a test change, it will apply the changes, build the affected test executables and run them on the base VMs. Results will be mailed back to the submitter. It will also try to determine if new failures were introduced, by comparing the test results to the most recent run using winetest.exe from WineHQ. If it thinks there are new failures, a message is sent to the author with a copy to wine-devel.
When handling wine-patches submissions, the bot will only process complete series, so it will queue up patches that appear to be part of a series. Then when the series is complete, it generates one big patch file by adding all the individual patches from the series together before submitting it. We'll have to see how robust this is, it is quite possible to mess up this algorithm.
Only patches that contain changes to tests are run. It checks to see if any dlls/<dllname>/tests/* files were changed, if so it is deemed a change to the tests. This should cover most cases, but again is not 100% correct all the time (if you only change a file in include/ then the tests depending on the header files don't get run).
The tests are run on VMware ESX VMs (for NT4 and above) and VMware Server VMs (for Win9x), so it's probably not all that useful to run sound and DirectDraw tests on them. Both 32-bit and 64-bit Windows versions are available. Before each test run, the VMs are reset to a clean snapshot, so no cruft from old tests will be left around. Your test shouldn't take more than 5 minutes, which is more than the WineTest framework allows anyway.
For each Windows version, there is a "base" VM and there can be additional VMs too. The idea is that the base VM is running the most up-to-date and most often used variation of that Windows version, so running your test on the base VMs should give you a good indication on how the test behaves. Additionally, you can run your tests on the other variations too. So, for example for XP the base VM is XP Professional SP3, but there are additional VMs like SP2, Media Center Edition, and a non-English version available too. Obviously, even when you use all available VMs you'll cover a very small subset of all possible configurations (most notably you won't test on a single physical system) so please do check http://test.winehq.org after your patch has been accepted to see if you introduced failures on other machines.
Security is a concern, but perhaps more for me than for you :-). I have to trust you not to abuse this system. If you want to do something naughty please do it on your own machine. There are some safeguards in the system, like the timeout. In general, you should not expect any privacy, I do have access to the consoles of the VMs and firewall logs. If you want to use this system for something else than Wine tests, please talk to me first.
I welcome feedback so if you have any ideas on how to improve this thing further please let me know.
The source (LGPL) is available via the SourceForge Wine repository, you can browse it at http://wine.git.sourceforge.net/git/gitweb.cgi?p=wine/winetestbot or download via git://wine.git.sourceforge.net/gitroot/wine/winetestbot
Physical Machines
TestBot currently consists of 3 physical machines:
Dell PowerEdge SC1435, quad-core AMD Opteron 2.2GHz, 16GB memory, 1.4TB storage, running VMware ESX 4.1. Hosts about half of the NT4+ VMs plus the website/engine VM.
Dell PowerEdge R310, quad-core Intel Xeon X3430 2.4GHz, 16GB memory, 550GB storage, running VMware ESX 4.1. Hosts the other half of the NT4+ VMs.
- Whitebox, quad-core Intel Core i7 975 3.33GHz, 6GB memory, 450GB storage, running VMware Server on top of openSuSE 11.3. Hosts the Win9x VMs.
See also
https://testbot.winehq.org/ - WineTestBot website
http://test.winehq.org/ - Wine test results
"Introducing WineTestBot" - the original wine-devel post by Ge van Geldorp