Regression Testing
Sometimes, committing patches in Wine to fix bugs and introduce new features causes new problems. This is called a regression. But since the problem isn't usually noticed until a new release occurs, it can be difficult to know which patch caused the problem, since hundreds of patches may have been committed in this time. This is where regression testing comes in.
Regression testing allows the problem patch to be found. This is accomplished using the source control program git and its bisect command. For a more in-depth article on git and Wine, see GitWine. This article is only focused on regression testing.
Contents
Setting up the regression test
To begin, first make sure git is installed on your system:
git --version
git should be installed and have a version number of 1.4.1 or higher. If not, install git using your distribution's package manager. For example, to install git in Ubuntu use:
sudo apt-get install git-core
Now, to save lots of time during the regression testing, install ccache using your distribution's package manager. On Ubuntu, use:
sudo apt-get install ccache
Once git and ccache are installed, you can download the Wine source code using the following command:
git clone git://source.winehq.org/git/wine.git wine-git cd wine-git
This will take a few minutes, as it must download several megabytes (approximately 90 Mb of data). If you are behind a firewall, you may need to replace git: with http:.
Now that you have a copy of the repository, we can begin testing. First, make sure the bug hasn't already been fixed in the bleeding-edge development tree ("current git" a.k.a. the "HEAD"). To do this, you need to compile Wine.
Compiling Wine
First, we need to configure the setup:
CC="ccache gcc" ./configure --verbose
If you have a 64-bit processor, you will need to do different steps to avoid errors. See WineOn64bit for your distribution (Ubuntu, for instance, requires hand-linking 32-bit libraries), but generally you will want to run configure using something like this:
CC="ccache gcc -m32" ./configure --verbose
Either way, unless you've compiled Wine before, configure will likely return some errors due to missing dependencies. Be sure to settle all dependencies, and try ./configure again. Keep doing this until you receive no errors. To install needed dependencies in Ubuntu/Debian, run:
sudo apt-get build-dep wine wine-dev
or
sudo apt-get install git-core flex bison libstdc++6-dev libartsc0-dev libasound2-dev libaudio2 libaudiofile0 libc6-dev libesd-alsa0 libesd0-dev libgcc1 libgl1-mesa-glx libgl1-mesa-dev libglib2.0-0 libglu1-mesa-dev libgphoto2-2-dev libgphoto2-port0 libice6 liblcms1-dev libldap2-dev libsm6 libx11-6 libxau6 libxext6 libxml2-dev libxslt1.1 libxt6 libxxf86vm1 msttcorefonts build-essential
This will grab all needed dependencies, install and configure them automatically.
Once you've got all dependencies settled, run:
CC="ccache gcc" ./configure --verbose && make depend && make
This will take some time, depending on the speed of your hardware. Some computers take as few as 3 minutes, while some can take hours. This takes about 20 minutes on my machine. Once it's compiled, test for the bug. To be safe, we want to test with a clean .wine directory.
mv ~/.wine ~/.wine-backup
Install the application from scratch:
./wine program_install.exe
And run it:
./wine "C:\Program Files\Program Name\program.exe"
Hopefully, the bug has been already fixed, in which case you can keep using git, or you can wait until the next release. If not, you will have to perform the regression test.
Note 1: The bug may have been fixed by simply using a fresh .wine directory.
Note 2: You don't need an internet connection while doing the regression test, as all the required information is in your local repository.
Running the bisect
Suppose that you have a bug that was introduced when upgrading from Wine 0.9.36 to Wine 0.9.37 (these numbers are release tags; for a list, see here). The following command will setup git to test for the bug.
git bisect start git bisect good wine-0.9.36 git bisect bad wine-0.9.37
Sometimes, however, you may know exactly which component the bad patch was in, if so, you can tell git this. For example, to tell git that the problem is in the wined3d dll component, that it worked in Wine 0.9.38, and is broken in current git, use:
git bisect start dlls/wined3d git bisect good wine-0.9.38 git bisect bad
This will tell you that you have X number of commits to test after this. Don't worry, it won't take that many tests :-). What git will do is position the source halfway between the two end-points (good and bad). For example, if you have 100 commits to test, it will position the source at the 50th commit. If the version you get does not compile (or is broken in some other way), run:
git bisect skip
And run make again. Repeat as many times as necessary until it compiles. Note that old Wine versions can give trouble compiling because of changes to freetype, http://www.winehq.org/pipermail/wine-patches/2008-September/060764.html explains this and has a patch.
Test for your bug here. If the bug is still present, you know that the patch that is causing the problem is somewhere in the first (older) 50, and not in the last (newer) 50. To tell git this, issue the command:
git bisect bad
But if the bug was not present, you know that the patch was in the last 50. To tell git this, issue the command:
git bisect good
This will reposition the source code in the middle of the remaining patches. You will then recompile Wine and test for the bug again. After a few tests, git will eventually identify the bad patch. The output should look something like this:
a460a2df43aa8eae10b1db028c6020829b009c54 is first bad commit commit a460a2df43aa8eae10b1db028c6020829b009c54 Author: Stefan Doesinger <stefandoesinger@gmx.at> Date: Sat Jun 9 14:27:41 2007 +0200 wined3d: Store the gl information in a per adapter structure and initialize it only once. :040000 040000 d8ae35832fcdbca8de07acae73b9e21564ced413 1720cc38fb598110071c9ee4f21f8f61b6f764c3 M dlls
Note that it says:
a460a2df43aa8eae10b1db028c6020829b009c54 is first bad commit
If you see:
Bisecting: 0 revisions left to test after this
THEN YOU ARE NOT DONE YET!
Once that patch is identified, post this info into bugzilla in the bug report (assuming one is opened, if not, open one). Also be sure to add the author of the patch to the CC. This information will allow the developers to fix the problem much quicker.
If you test and all you get are "good" bisects (the symptom fails to show) and get a final bisect output that has "Release x.x.xx", like this:
b821e839bb33ac8f56939cc582010ecf4d9c25d4 is first bad commit
commit b821e839bb33ac8f56939cc582010ecf4d9c25d4
Author: Alexandre Julliard <julliard@winehq.org>
Date: Fri Mar 21 16:41:33 2008 +0100
Release 0.9.58.
:100644 100644 9123c03a3138b05cb8ebb5271f554bb76510cd09 3b88d88dcdfc8e148f8e7e0858bf0ca62c0bdff3 M ANNOUNCE
:100644 100644 900c8932cb5aad58e928a5c3192ab95967dc0664 e001d501bec609c31c5a91ab185a06ef1662f4c8 M ChangeLog
:100644 100644 5614cdd9b3c6af6d44027f067fcc423e49c49c91 36e035642c055736e2f833210f5db334a6706486 M VERSION
:100755 100755 a516335cb3e4c67298f285adbdb2ede09da133be 51a9fd0b9741b28a588e2dd3bb57ae2be8805af2 M configure
... then you need to reset your bisect and use that "Release x.x.xx" as your first good bisect (wine-x.x.xx). This particular bisect output is just a version tag, meaning that no real code was changed (and so cannot have made your problem appear). All the "good" bisects you've set tells GIT that all that code is good, until wine-x.x.xx . GIT eliminates all that code except for the version tag, which would be the latest revision before the next version tag.
Testing a broken compile
If compilation is broken on your platform, apply the same procedure as above.
With broken compiles, it's easy to automate the bisection with git bisect start and a test script. See the git-bisect man page for how to set this up.
Resetting the bisect
Once you've finished with this test, you may want to test something else. To reset your git repository, use:
git bisect reset
If you try to start a new bisect before doing this step, git bisect start will fail with "won't bisect on seeked tree"
Reverting the patch
If the bisect identifies a patch that a developer believes is harmless, you may be asked to check the result by reverting that patch.
First, reset the bisect if you haven't already:
git bisect reset
Then use the revert command with the sha1 id of the bad patch:
git show b821e839bb33ac8f56939cc582010ecf4d9c25d4 | patch -p1 -R
Rebuild Wine and test your program again.
When you're done testing, reset your tree to the latest Wine commit
git reset --hard HEAD
Patching your git tree
Now suppose a developer posts a patch that they want you to test. Though this may seem overwhelming, it's quite simple. Simply download the patch (likely from Bugzilla, or possibly from e-mail) and copy it to your ~/wine-git folder. Then run:
patch -p1 < patch_name.diff
This will patch the source code. It should return something similar to this:
dlls/wined3d has been patched.
Once this is done, simply compile again:
./configure && make depend && make
And rerun your program:
./wine "C:\Program Files\Program Name\program.exe"
See, wasn't that hard! Now if you're going to be testing any other programs, you don't want your tree cluttered with patches you were testing for another program. To reset (to undo all patches applied to) your git tree, run:
git reset --hard origin
This will reset all commits you've made. But since you're only testing and not doing actual patches, you won't lose any work.
Updating your git tree
Let's now assume that time has passed, and a bug you've been following has been reported to work in the latest git. If you want to test this, you need to update your git repository and compile the new version. To update, use:
git fetch ; git rebase origin
Once that's done, simply compile again:
./configure && make depend && make
And test. Your program should now work!
Troubleshooting
See TroubleshootingBisectProblems
If you are having trouble compiling or running an older version of Wine, odds are the instructions in ReverseRegressionTesting will be able to help you.
Disabling optimization to speed up compilation
When compiling Wine, a large portion of the time spent compiling is spent optimizing the generated code. You might be able to speed up compilation substantially by skipping optimization. To do this, pass CFLAGS="-g -O0" on the command line when you run configure. So:
CC="ccache gcc" CFLAGS="-g -O0" ./configure --verbose && make depend && make
On my Intel e7200 system with gcc-4.2.1, "make -j3" normally takes 9 minutes, but with optimization disabled, it takes only 5 minutes.
Disabling tests to speed up compilation
When compiling for regression tests, a large portion of the time spent compiling is spent compiling the conformance tests. While these tests are valuable, they are not needed for regression testing (we already know something broke, so conformance is irrelevant).
Alexandre committed a patch that allows one to disable building the test suite when regression testing. For Wine 1.1.9 and above, you can simply use:
./configure --disable-tests
to disable building the tests.
If your "good" version is older, a small hack is needed. This patch works for Wine up to around 1.1.3:
diff --git a/dlls/Makefile.in b/dlls/Makefile.in
index d7b0976..2b4b779 100644
--- a/dlls/Makefile.in
+++ b/dlls/Makefile.in
@@ -332,7 +332,7 @@ SUBDIRS = \
winequartz.drv \
winex11.drv
-BUILDSUBDIRS = $(BASEDIRS) $(EXTRADIRS) $(TESTSUBDIRS)
+BUILDSUBDIRS = $(BASEDIRS) $(EXTRADIRS)
INSTALLSUBDIRS = $(BASEDIRS) $(EXTRADIRS) $(IMPLIBSUBDIRS)
DOCSUBDIRS = $(BASEDIRS) $(EXTRADIRS)
diff --git a/programs/Makefile.in b/programs/Makefile.in
index 3c128e1..72dbdb8 100644
--- a/programs/Makefile.in
+++ b/programs/Makefile.in
@@ -41,7 +41,6 @@ SUBDIRS = \
winemenubuilder \
winemine \
winepath \
- winetest \
winevdm \
winhelp \
winver \
--
1.4.4.2
If you're using Wine between 1.1.3 and 1.1.9, use this patch instead:
diff --git a/dlls/Makefile.in b/dlls/Makefile.in
index 6680673..e998a38 100644
--- a/dlls/Makefile.in
+++ b/dlls/Makefile.in
@@ -9,9 +9,8 @@ INSTALLDIRS = $(DESTDIR)$(dlldir)
DLLSUBDIRS = @ALL_DLL_DIRS@
IMPLIBSUBDIRS = @ALL_IMPLIB_DIRS@
-TESTSUBDIRS = @ALL_TEST_DIRS@
-SUBDIRS = $(DLLSUBDIRS) $(IMPLIBSUBDIRS) $(TESTSUBDIRS)
-BUILDSUBDIRS = $(DLLSUBDIRS) $(TESTSUBDIRS)
+SUBDIRS = $(DLLSUBDIRS) $(IMPLIBSUBDIRS)
+BUILDSUBDIRS = $(DLLSUBDIRS)
INSTALLSUBDIRS = $(DLLSUBDIRS) $(IMPLIBSUBDIRS)
DOCSUBDIRS = $(DLLSUBDIRS)
@@ -445,8 +444,6 @@ CROSS_IMPLIBS = \
wsock32/libwsock32.a \
wtsapi32/libwtsapi32.a
-$(TESTSUBDIRS:%=%/__crosstest__): $(CROSS_IMPLIBS)
-
implib: $(IMPORT_LIBS)
.PHONY: implib
diff --git a/programs/Makefile.in b/programs/Makefile.in
index 7bd60b2..e5edef4 100644
--- a/programs/Makefile.in
+++ b/programs/Makefile.in
@@ -38,7 +38,3 @@ install install-lib:: install-progs$(DLLEXT) $(INSTALLDIRS)
uninstall::
-cd $(DESTDIR)$(bindir) && $(RM) wineapploader $(INSTALLPROGS)
-rmdir $(DESTDIR)$(dlldir)
-
-# Rules for testing
-
-check test:: $(SUBDIRS:%=%/__test__)
diff --git a/programs/winetest/Makefile.in b/programs/winetest/Makefile.in
index d153c04..3dd7bcd 100644
--- a/programs/winetest/Makefile.in
+++ b/programs/winetest/Makefile.in
@@ -21,10 +21,6 @@ SVG_SRCS = winetest.svg
@MAKE_PROG_RULES@
-ALL_TEST_DIRS = @ALL_TEST_DIRS@
-
-TESTBINS = $(ALL_TEST_DIRS:%/tests=%_test.exe)
-
@ALL_WINETEST_DEPENDS@
# Special rules
--
1.5.3.6
However, care must be taken when using these patches. git-bisect doesn't like you editing the tree when bisected. So, to properly bisect, while still disabling tests, follow this procedure (this assumes the patch is ~/wine-git/disable_test_new.diff and that the regression appeared between versions 1.1.4 and 1.1.5):
$ git bisect start $ git bisect bad wine-1.1.4 $ git bisect good wine-1.1.5 $ patch -p1 < disable_test_new.diff && CC="ccache gcc" ./configure && make depend && make && patch -p1 -R < disable_test_new.diff
Now test your app, and use 'git bisect good' or 'git bisect bad' to prepare the next test. After that, repeat the 4th step until you've reached the patch in question. (The 4th step applies the patch, compiles Wine and then reverses (undoes) the patch, so git doesn't think anything has changed.)