Diff for "UnitTestSuites"

Differences between revisions 17 and 18

Unit Test Suites

Since 2003 or so, Wine has used Test Driven Development to ensure that bugs don't creep into Wine's source tree. The tests in Wine's source tree are all Conformance Tests to make sure Wine matches the reference implementation for Win32/Win64. (There are also Win16 conformance tests at http://win16test.googlecode.com that should get merged into WineHQ's source tree someday.)

Wine's conformance test suite is great, but it's far from complete. We improve it by adding tests as we add new features or fix bugs. But tracking down bugs from user problem reports is laborious, as the observed symptoms are often very far removed from the cause and/or hard to reproduce. Wouldn't it be great if there were a source of bug reports that pinpoints the exact problem in Wine for us? That would make fixing the bug and adding a conformance test much easier.

Fortunately, Wine isn't the only project using Test Driven Development; every other large project in the world is doing it, too.

LanguageTestSuites for languages that already run on Wine are probably the easiest to run. It might take a little effort to use their results, though, as there's usually a big fluffy interpreter between the test and Wine.

Simply building open source apps on Wine using, say, mingw32 or Visual C++ is a good automated test of Wine. You'd be surprised how many bugs this can turn up!

Unit tests written in C or C++ are even more likely to be easy to run, and when they fail, to point directly to the Wine bug than user problem reports or language test suite failures.

AppBuildTest

AppBuildTest is the lame and maybe not final name for a set of debugged build scripts Dan Kegel is writing. The goal is to have a single shell script that can run in either Cygwin+Windows or Cygwin+Wine and successfully build an application from source and run the app's own automated test suite. The same script should succeed in both Windows and Wine.

(Not to be confused with AppInstall, Austin English's scripts that download and test prebuilt applications using new AutoHotKey based tests.)

Here's a list of open source projects with automated test suites that could be built and tested in Wine, along with a note on their appbuildtest (and appinstall) status. Projects with their own public buildbots or other continuous build-and-test rigs showing that their tests all pass on Windows are the best candidates for this list.

Gyp

Home page: http://gyp.googlecode.com
Buildbot: (used in Chromium's build, doesn't need separate one?)
Unit tests: http://code.google.com/p/gyp/wiki/GypTesting

Inkscape

Home page: http://inkscape.org
Buildbot: http://home.hccnet.nl/th.v.d.gronde/inkscape/ResultViewer.html
Unit tests: http://wiki.inkscape.org/wiki/index.php/TestingInkscape#Running_unit_tests
AppBuildTest script: http://kegel.com/wine/inkscape-download-and-build.sh.txt (only build, no test yet)
Bugs so far: http://bugs.winehq.org/buglist.cgi?quicksearch=inkscape

OpenOffice.org

Home page: http://openoffice.org
Buildbot: http://buildbot.go-oo.org/buildbot/waterfall
Win32 build instructions: http://wiki.services.openoffice.org/wiki/Documentation/Building_Guide/Building_on_Windows, http://wiki.services.openoffice.org/wiki/Documentation/Building_Guide/Building_on_Windows_with_MinGW

Subversion

Home page: http://subversion.tigris.org/
Buildbot: http://crest.ics.uci.edu/buildbot/
Win32 build instructions: http://svn.collab.net/repos/svn/trunk/INSTALL

Webkit

Home page: http://www.webkit.org
Buildbot: http://build.webkit.org
Win32 build instructions: http://webkit.org/building/tools.html

Projects without known buildbots

We should think twice about using these; if a project doesn't have a continuous build and test rig, chances are they might not have purged their test suite of flaky tests yet (since flakiness often only becomes apparent when you test continuously). If you find a live automated build and test results site for one of these, please add its URL and move it to the previous section.