Regression Testing
Sometimes, committing patches in Wine to fix bugs and introduce new features causes new problems. This is called a regression. But since the problem isn't usually noticed until a new release occurs, it can be difficult to know which patch caused the problem, since hundreds of patches may have been committed in this time. This is where regression testing comes in.
Regression testing allows the problem patch to be found. This is accomplished using the source control program git and its bisect command. For a more indepth article on git and Wine, see GitWine. This article is only focused on regression testing.
Setup Info
To begin, first make sure git is installed on your system:
git --version
git should be installed an have a version number of 1.4.1 or higher. If not, install git using your distribution's package manager. For example, to install git in Ubuntu use:
sudo apt-get install git-core
Now, to save lots of time during the regression testing, install ccache using your distribution's package manager. On Ubuntu, use:
sudo apt-get install ccache
Once git and ccache are installed, you can download the Wine source code using the following command:
git clone git://source.winehq.org/git/wine.git wine-git cd wine-git
This will take a few minutes, as it must download several megabytes (approximately 90 Mb of data). If you are behind a firewall, you may need to replace git: with http:.
Now that you have a copy of the repository, we can begin testing.
Running the Bisect
Suppose that you have a bug that was introduced when upgrading from Wine 0.9.36 to Wine 0.9.37 (These numbers are release tags. For a list, see here) The following command will setup git to test for the bug.
git bisect start git bisect good wine-0.9.36 git bisect bad wine-0.9.37
Sometimes, however, you may know exactly which component the bad patch was in, if so, you can tell git this. For example, to tell git that the problem is in the wined3d dll component, that it worked in wine 0.9.38, and is broken in current git, use:
git bisect start dlls/wined3d git bisect good wine-0.9.38 git bisect bad
This will tell you that you have X number of commits to test after this. Don't worry, it won't take that many tests :-). What git will do is position the source in the middle of the commits that you are testing. For example, if you have 100 commits to test, it will position the source at the 50th commit. We're almost ready to compile Wine. First, we need to configure the setup:
CC="ccache gcc" ./configure --verbose
If you have a 64 bit processor, you will need to do different steps to avoid errors. See WineOn64bit for your distribution (Ubuntu, for instance, requires hand linking 32 bit libraries), but generally you will want to run configure using something like this:
CC="ccache gcc -m32" ./configure --verbose
Either way, unless you've compiled Wine before, configure will likely return some errors due to missing dependencies. Be sure to settle all dependencies, and try ./configure again. Keep doing this until you receive no errors. To install needed dependencies in Ubuntu/Debian, run:
sudo apt-get build-dep wine wine-dev
or
sudo apt-get install git-core flex bison libstdc++6-dev libartsc0-dev libasound2-dev libaudio2 libaudiofile0 libc6-dev libesd-alsa0 libesd0-dev libgcc1 libgl1-mesa-glx libgl1-mesa-dev libglib2.0-0 libglu1-mesa-dev libgphoto2-2-dev libgphoto2-port0 libice6 liblcms1-dev libldap2-dev libsm6 libx11-6 libxau6 libxext6 libxml2-dev libxslt1.1 libxt6 libxxf86vm1 msttcorefonts build-essential
This will grab all needed dependencies, install and configure them automatically. Once you've got all dependencies settled, run:
CC="ccache gcc" ./configure --verbose && make clean && make
This will take some time, depending on the speed of your hardware. Some computers take as few as 3 minutes, while some can take hours. This takes about 20 minutes on my machine. Once it's compiled, you're almost ready to test for the bug. Just to be safe, we want to test with a clean .wine directory.
mv ~/.wine ~/.wine-backup
Testing the Actual Program
Then install the program from scratch:
./wine program_install.exe
And run it:
./wine "C:\Program Files\Program Name\program.exe"
Test for your bug here. If the bug is still present, you know that the patch that is causing the problem is somewhere in the first 50, and not in the last 50. To tell git this, issue the command:
git bisect bad
But if the bug was not present, you know that the patch was in the last 50. To tell this git, issue the command:
git bisect good
This will reposition the source code in the middle of the remaining patches. You will then recompile Wine and test for the bug again. After a few tests, git will eventually identify the bad patch. It should look something like this:
a460a2df43aa8eae10b1db028c6020829b009c54 is first bad commit commit a460a2df43aa8eae10b1db028c6020829b009c54 Author: Stefan Doesinger <stefandoesinger@gmx.at> Date: Sat Jun 9 14:27:41 2007 +0200 wined3d: Store the gl information in a per adapter structure and initialize it only once. :040000 040000 d8ae35832fcdbca8de07acae73b9e21564ced413 1720cc38fb598110071c9ee4f21f8f61b6f764c3 M dlls
Note that it says:
a460a2df43aa8eae10b1db028c6020829b009c54 is first bad commit
If you see:
Bisecting: 0 revisions left to test after this
THEN YOU ARE NOT DONE YET!
Once that patch is identified, post this info into bugzilla in the bug report (assuming one is opened, if not, open one). Also be sure to add the author of the patch to the CC. This information will allow the developers to fix the problem much quicker.
If you test and all you get are "good" bisects (the symptom fails to show) and get a final bisect output that has "Release 0.9.xx", like this:
b821e839bb33ac8f56939cc582010ecf4d9c25d4 is first bad commit
commit b821e839bb33ac8f56939cc582010ecf4d9c25d4
Author: Alexandre Julliard <julliard@winehq.org>
Date: Fri Mar 21 16:41:33 2008 +0100
Release 0.9.58.
:100644 100644 9123c03a3138b05cb8ebb5271f554bb76510cd09 3b88d88dcdfc8e148f8e7e0858bf0ca62c0bdff3 M ANNOUNCE
:100644 100644 900c8932cb5aad58e928a5c3192ab95967dc0664 e001d501bec609c31c5a91ab185a06ef1662f4c8 M ChangeLog
:100644 100644 5614cdd9b3c6af6d44027f067fcc423e49c49c91 36e035642c055736e2f833210f5db334a6706486 M VERSION
:100755 100755 a516335cb3e4c67298f285adbdb2ede09da133be 51a9fd0b9741b28a588e2dd3bb57ae2be8805af2 M configure
... then you need to reset your bisect and use that "Release 0.9.xx" as your first good bisect (wine-0.9.xx). This particular bisect output is just a version tag, meaning that no real code was changed between version to make your problem appear. All the "good" bisects you've set tells GIT that all that code is good, until wine-0.9.xx . GIT eliminates all that code except for the version tag, which would be the latest revision before the next version tag.
Resetting the Bisect
Once you've finished with this test. You may want to test something else. To reset your git repository, use:
git bisect reset
If you try to start a new bisect before doing this step, git bisect start will fail with "won't bisect on seeked tree"
Patching Your Git Tree
Now suppose a developer posts a patch that they want you to test. Though this may seem overwhelming, it's quite simple. Simply download the patch (likely from bugzilla, or possibly from e-mail) and copy it to your ~/wine-git folder. Then run:
patch -p1 < patch_name.diff
This will patch the source code. It should return something similar to this:
dlls/wined3d has been patched.
Once this is done, simply compile again:
./configure && make clean && make depend && make
And rerun your program:
./wine "C:\Program Files\Program Name\program.exe"
See, wasn't that hard! Now if you're going to be testing any other programs, you don't want your tree cluttered with patches you were testing for another program. To reset your git tree to patch the Wine Git repository, run:
git reset --hard origin
This will reset all commits you've made. But since you're only testing and not doing actual patches, you won't lose any work.
Updating Git
Let's now assume that a week or so has passed, and a bug you've been following has been reported to work in the latest git. If you want to test this, you need to update your git repository and compile the new version. To update, use:
git fetch ; git rebase origin
Once that's done, simply compile again:
./configure && make clean && make depend && make
And test. Your program should now work!
Troubleshooting
See TroubleshootingBisectProblems
If you are having trouble compiling or running an older version of wine, odds are the instructions in ReverseRegressionTesting will be able to help you.
Disabling Optimization to Speed up Compliation
When compiling Wine, a large portion of the time spent compiling is spent optimizing the generated code. You might be able to speed up compilation substantially by skipping optimization. To do this, pass CFLAGS="-g -O0" on the commandline when you run configure. On my Intel e7200 system with gcc-4.2.1, "make -j3" normally takes 9 minutes, but with optimization disabled, it takes only 5 minutes.
Disabling Tests to Speed up Compliation
When compiling for regression tests, a large portion of the time spent compiling is spent compiling the conformance tests. While these tests are valuable, they are not needed for regression testing (we already know something broke, so conformance is irrelevant). Until a patch is accepted in the main wine tree to enable such an option, a small hack is needed:
diff --git a/dlls/Makefile.in b/dlls/Makefile.in
index d7b0976..2b4b779 100644
--- a/dlls/Makefile.in
+++ b/dlls/Makefile.in
@@ -332,7 +332,7 @@ SUBDIRS = \
winequartz.drv \
winex11.drv
-BUILDSUBDIRS = $(BASEDIRS) $(EXTRADIRS) $(TESTSUBDIRS)
+BUILDSUBDIRS = $(BASEDIRS) $(EXTRADIRS)
INSTALLSUBDIRS = $(BASEDIRS) $(EXTRADIRS) $(IMPLIBSUBDIRS)
DOCSUBDIRS = $(BASEDIRS) $(EXTRADIRS)
diff --git a/programs/Makefile.in b/programs/Makefile.in
index 3c128e1..72dbdb8 100644
--- a/programs/Makefile.in
+++ b/programs/Makefile.in
@@ -41,7 +41,6 @@ SUBDIRS = \
winemenubuilder \
winemine \
winepath \
- winetest \
winevdm \
winhelp \
winver \
--
1.4.4.2
However, care must be taken when using this patch. Git-bisect doesn't like you editing the tree when bisected. So, to properly bisect, while still disabling tests, follow this procedure (this assumes the patch is ~/wine-git/disable_test.diff and that the regression appeared between versions 0.9.50 and 0.9.52) :
Step 1: $ git bisect start Step 2: $ git bisect bad wine-0.9.52 Step 3: $ git bisect good wine-0.9.50 Step 4: $ patch -p1 < disable_test.diff && CC="ccache gcc" ./configure && make clean && make depend && make && patch -p1 -R < disable_test.diff
After that, test your app, and then use 'git bisect good' or 'git bisect bad' to prepare the next test. After that, repeat step 4 until you've reached the patch in question. For your convenience, the disabling test patch can be downloaded here.