Wine Conformance Tests

Wine builds a set of unit tests that check the behavior of a large portion of the Windows API. The purpose of these tests is to:

  • Document the behavior of the Windows API.
  • Make sure the Wine implementation conforms to the Windows behavior.
  • Detect and prevent Wine regressions.

To simplify running the tests, a single Windows executable containing all the tests (named WineTest) is updated daily and can be downloaded from the bottom of WineHQ's testing page. This helps collect results from a broader set of Windows, Unix and Mac environments than the WineTestBot farm can cover.

{i} Running WineTest requires no programming skills whatsoever. Simply download the executable, answer any prompts that come up, and let the test suite do the rest. This makes running the test suite one of the simplest ways that users can help improve Wine.

To ensure that the conformance test suite meets its goals:

  • Alexandre will not commit any patch that causes conformance test failures on his Wine development machine. This ensures the tests pass on at least one Wine environment and prevents regressions (but he won't say what his computer specs are to preserve the testing environment diversity).
  • All conformance test patches posted to the wine-patches mailing list are run on a set of Windows machines using the WineTestBot. Tests that fail there will not be committed. This ensures the correctness of the conformance tests, i.e. that what they describe is indeed the Windows behavior.

Running the tests manually

Wine developers will want to run specific subsets of the tests whenever they changed something to make sure they did not introduce regressions. Everyone else should probably use WineTest instead and thus skip to the next section.

To run specific conformance tests, go to the corresponding dll's tests directory and run 'make test' or 'make testname.ok'. For instance:

cd dlls/advapi32/tests
make test
make services.ok

Also note that if you are set up for cross compiling Wine with MingW, you can run 'make crosstest' to get a Windows executable for that test.

Writing conformance tests

The Wine Developer's Guide contains a section devoted to the writing of Conformance Tests

Cmd conformance tests are handled a bit differently.

Getting WineTest

WineTest is the best and easiest way to run the conformance tests because it will upload the results to a central location where everyone can look at the results. You can get the latest version of WineTest at:

You can also get older 32-bit versions of WineTest by going to the WineTest website, opening a specific build's page, and clicking on the commit id on that page's header.

If you build Wine yourself you will also find a WineLib version of WineTest in the programs/winetest directory.

Running WineTest on Windows

Here is how to run WineTest once you got it by following the above instructions:

  1. Run the WineTest executable.

  2. Enter a short tag for your test run. This is what will be displayed as the heading on the WineTest website.

  3. Provide your email address. This is needed so developers can contact you when investigating why a given test failed.
  4. Wait for the tests to run.
  5. Press OK to upload the data to the test results page.
  6. About 5-10 minutes later your results will be on the page for that test run.

See also the MakeTestFailures page for a (necessarily) out-of-date list of known test failures.

  • Insert an audio or mixed audio+data CD-ROM or DVD in a drive otherwise winmm:mcicda and ntdll:cdrom skip some tests.

Windows firewall

  • Starting with Windows XP SP2, the Windows firewall will likely ask whether to allow some of the tests to accept incoming network connections. There is no need to worry about it, and answering either way won't impact the results. The reason is that the firewall always authorizes connections originating from the local host which is all we need for the Wine tests.

Windows User Account Control

  • If you are running Windows Vista or greater with UAC turned on, you will get UAC prompts for some of the tests. The tests are supposed to gracefully deal with those but may fail to do so. To sidestep the issue you can run WineTest by right-clicking on it and selecting 'Run as administrator', or by starting from a shell running as administrator.

DirectX

  • Even with a fully updated Windows system you will be missing some !DirectX libraries, typically d3dcompiler_43 and the d3dx9_* dlls, thus causing some tests to be skipped. To avoid that install the latest version of DirectX.

Running WineTest in Wine

The procedure is the same as on Windows except that:

  1. You would use the wine script to run the 32-bit tests and wine64 to run the 64-bit ones.
  2. If you built Wine from source you already have WineTest and can run it with

wine ./programs/winetest/winetest.exe.so
  1. The WineTest website only accepts test results from unpatched Wine sources.

If you are running the tests unattended you will likely want to disable winedbg's 'crash dialog' to avoid long delays whenver a test crashes. To do so, run something like the command below before starting WineTest.

./wine "./programs/regedit/regedit.exe.so" - <<__EOF__
[HKEY_CURRENT_USER\Software\Wine\WineDbg]
"ShowCrashDialog"=dword:00000000
__EOF__

See also the MakeTestFailures page for a (necessarily) out-of-date list of known test failures.

Unix notes

  • To avoid skipped tests, in particular in gdiplus:font, it is recommended to install the following Microsoft fonts: Arial, Courier New, Microsoft Sans Serif and Times New Roman. On Debian most of them are provided by the ttf-mscorefonts-installer package. Alternatively, one should at least install the Liberation fonts.
  • Message-sequence tests usually fail because of window manager bugs. Alexandre has been working with window manager developers to get these fixed. At the moment, known good window managers are metacity 2.23.8 or later, fvwm 2.5.26 (older versions may be fine too on 32-bit platforms), probably KDE 3.5.8 or greater too. So if you are seeing message sequence failures, check to see if switching to one of the above window manager versions helps.
  • A missing libjpeg library will cause a number of test failures. So if you compiled Wine from source make sure the configure script did not complain about it. Also make sure that there is no mismatch between the headers and the library you link with, like using the libjpeg62 headers but linking with libjpeg.so.8. This is particularly important if you have both 32-bit and 64-bit versions of this library.
  • You need a functional ntlm_auth binary otherwise you will get some rpcrt4 test failures. Note that on some systems, ntlm_auth is present by default but does not work because there is no smb.conf file.
  • Insert an audio or mixed audio+data CD-ROM or DVD in a drive otherwise winmm:mcicda and ntdll:cdrom skip some tests.

Mac OS X notes

  • Apple's X11 server is buggy and will result in a lot of test failures, especially on Leopard or older. So you should install and use XQuartz instead.

  • Depending on the Mac OS X settings, the application firewall may ask whether to allow wineserver or some of the tests to accept incoming network connections. There is no need to worry about it, and answering either way won't impact the results. The reason is that the firewall always authorizes connections originating from the local address which is all we need for the Wine tests.


CategoryQualityAssurance

ConformanceTests (last edited 2012-12-10 17:28:07 by FrancoisGouget)