Wine Developer's Guide/Writing Conformance Tests

From WineHQ Wiki
Jump to: navigation, search

1 Introduction

The Windows API follows no standard, it is itself a de facto standard, and deviations from that standard, even small ones, often cause applications to crash or misbehave in some way.

The question becomes, “How do we ensure compliance with that standard?”. The answer is, “By using the API documentation available to us and backing that up with conformance tests.” Furthermore, a conformance test suite is the most accurate (if not necessarily the most complete) form of API documentation and can be used to supplement the Windows API documentation.

Writing a conformance test suite for more than 10000 APIs is no small undertaking. Fortunately it can prove very useful to the development of Wine way before it is complete.

  • The conformance test suite must run on Windows. This is necessary to provide a reasonable way to verify its accuracy. Furthermore the tests must pass successfully on all Windows platforms (tests not relevant to a given platform should be skipped).

    A consequence of this is that the test suite will provide a great way to detect variations in the API between different Windows versions. For instance, this can provide insights into the differences between the, often undocumented, Win9x and NT Windows families.

    However, one must remember that the goal of Wine is to run Windows applications on Linux, not to be a clone of any specific Windows version. So such variations must only be tested for when relevant to that goal.

  • Writing conformance tests is also an easy way to discover bugs in Wine. Of course, before fixing the bugs discovered in this way, one must first make sure that the new tests do pass successfully on at least the Wine Test Bot base VMs.

    Bugs discovered this way should also be easier to fix. Unlike some mysterious application crashes, when a conformance test fails, the expected behavior and APIs tested for are known thus greatly simplifying the diagnosis.

  • To detect regressions. Simply running the test suite regularly in Wine turns it into a great tool to detect regressions. When a test fails, one immediately knows what was the expected behavior and which APIs are involved. Thus regressions caught this way should be detected earlier, because it is easy to run all tests on a regular basis, and be easier to fix because of the reduced diagnosis work.

  • Tests written in advance of the Wine development (possibly even by non Wine developers) can also simplify the work of the future implementer by making it easier for him to check the correctness of his code.

  • Conformance tests will also come in handy when testing Wine on new (or not as widely used) architectures such as FreeBSD, Solaris x86 or even non-x86 systems. Even when the port does not involve any significant change in the thread management, exception handling or other low-level aspects of Wine, new architectures can expose subtle bugs that can be hard to diagnose when debugging regular (complex) applications.

2 What to test for?

The first thing to test for is the documented behavior of APIs and such as CreateFile. For instance one can create a file using a long pathname, check that the behavior is correct when the file already exists, try to open the file using the corresponding short pathname, convert the filename to Unicode and try to open it using CreateFileW, and all other things which are documented and that applications rely on.

While the testing framework is not specifically geared towards this type of tests, it is also possible to test the behavior of Windows messages. To do so, create a window, preferably a hidden one so that it does not steal the focus when running the tests, and send messages to that window or to controls in that window. Then, in the message procedure, check that you receive the expected messages and with the correct parameters.

For instance you could create an edit control and use WM_SETTEXT to set its contents, possibly check length restrictions, and verify the results using WM_GETTEXT. Similarly one could create a listbox and check the effect of LB_DELETESTRING on the number of items of the list, selected items list, highlighted item, etc. For concrete examples, see dlls/user32/tests/win.c and the related tests.

However, undocumented behavior should not be tested for unless there is an application that relies on this behavior, and in that case the test should mention that application, or unless one can strongly expect applications to rely on this behavior, typically APIs that return the required buffer size when the buffer pointer is NULL.

3 Running the tests in Wine

The simplest way to run the tests in Wine is to type 'make test' in the Wine sources top level directory. This will run all the Wine conformance tests.

The tests for a specific Wine library are located in a 'tests' subdirectory of the library directory. Each test is contained in a file (e.g. dlls/kernel32/tests/thread.c). Each file itself contains many checks concerning one or more related APIs.

So to run all the tests related to a given Wine library, go to the corresponding tests directory and type make test. This will compile the tests, run them, and create a xxx.ok file for each test that passes successfully. And if you only want to run the tests contained in the thread.c file of the kernel library, you would do:

$ cd dlls/kernel32/tests
$ make thread.ok

Note that if the test has already been run and is up to date (i.e. if neither the kernel library nor the thread.c file has changed since the thread.ok file was created), then make will say so. To force the test to be re-run, delete the thread.ok file, and run the make command again.

You can also run tests manually using a command similar to the following:

$ ../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
$ ../../../tools/runtest -P wine -p kernel32_test.exe.so thread.c
thread.c: 86 tests executed, 5 marked as todo, 0 failures.

The -P option defines the platform that is currently being tested and is used in conjunction with the todo statements (see below). Remove the -q option if you want the testing framework to report statistics about the number of successful and failed tests. Run runtest -h for more details.

4 Cross-compiling the tests with MinGW-w64

4.1 Setup of the MinGW-w64 cross-compiling environment

Here are some instructions to setup MinGW-w64 on different Linux distributions and *BSD.

4.1.1 Debian GNU/Linux like deb systems

This includes Debian GNU/Linux, Ubuntu, etc. The following step should probably work on any deb based system.

Run apt-get install mingw-w64.

4.1.2 Red Hat Linux like rpm systems

This includes Fedora, Red Hat Enterprise Linux, Mandrake, most probably SuSE Linux too, etc. But this list isn't exhaustive; The following step should probably work on any rpm based system.

Run yum install mingw64-gcc.

4.1.3 *BSD

The *BSD systems have in their ports collection only a port for the MinGW cross-compiling environment. Please see the documentation of your system about how to build and install a port.

4.2 Compiling the tests

Having the cross-compiling environment set up the generation of the Windows executables is easy by using the Wine build system.

If you had already run configure, then delete config.cache and re-run configure. You can then run make crosstest. To sum up:

$ rm config.cache
$ ./configure
$ make crosstest

5 Building and running the tests on Windows

5.1 Using pre-compiled binaries

The simplest solution is to visit test.winehq.org and download the latest version of winetest at the bottom of the page. This executable contains all the Wine conformance tests, runs them and reports the results.

5.2 With Visual C++

  • If you are using Visual Studio 6, make sure you have the processor pack. The processor pack fixes error C2520: conversion from unsigned __int64 to double not implemented, use signed __int64. However note that the processor pack is incompatible with Visual Studio 6.0 Standard Edition, and with the Visual Studio 6 Service Pack 6. If you are using Visual Studio 7 or greater you don't need it. In either case it is recommended to use the most recent compatible Visual Studio service pack. If are using Visual Studio Express and you need specific libraries like ntdll.lib that don't ship with Visual Studio Express, you will need the Windows Driver Development Kit (DDK) which is part of the Windows Developer Kit.

  • get the Wine sources

  • Run msvcmaker to generate Visual C++ project files for the tests. msvcmaker is a perl script so you may be able to run it on Windows.

    $ ./tools/winapi/msvcmaker --no-wine
  • If the previous steps were done on your Linux development machine, make the Wine sources accessible to the Windows machine on which you are going to compile them. Typically you would do this using Samba but copying them altogether would work too.

  • On the Windows machine, open the winetest.dsw workspace. This will load each test's project. For each test there are two configurations: one compiles the test with the Wine headers, and the other uses the Microsoft headers.

  • If you choose the “Win32 MSVC Headers” configuration, most of the tests will not compile with the regular Visual Studio headers. So to use this configuration, download and install a recent Platform SDK as well as the latest DirectX SDK. Then, configure Visual Studio to use these SDK headers and libraries. Alternately you could go to the Project > Settings... menu and modify the settings appropriately, but you would then have to redo this whenever you rerun msvcmaker.

  • Open the Build > Batch build... menu and select the tests and build configurations you want to build. Then click on Build.

  • To run a specific test from Visual C++, go to Project > Settings.... There select that test's project and build configuration and go to the Debug tab. There type the name of the specific test to run (e.g. thread) in the Program arguments field. Validate your change by clicking on Ok and start the test by clicking the red exclamation mark (or hitting F5 or any other usual method).

  • You can also run the tests from the command line. You will find them in either Output\Win32_Wine_Headers or Output\Win32_MSVC_Headers depending on the build method. So to run the kernel path tests you would do:

    C:\> cd dlls\kernel\tests\Output\Win32_MSVC_Headers
    C:\wine\dlls\kernel\tests\Output\Win32_MSVC_Headers> kernel32_test path

5.3 With MinGW-w64

Wine build system already has support for building tests with a MinGW-w64 cross-compiler. See the section above called “Setup of the MinGW-w64 cross-compiling environment” for instructions on how to set things up. When you have a MinGW-w64 environment installed all you need to do is rerun configure and it should detect the MinGW-w64 compiler and tools. Then run make crosstest to start building the tests.

5.4 Standalone, using the Microsoft C++ Toolkit

Sometimes it's nice to be able to build a new unit test on Windows without Wine, and without buying Microsoft Visual C++. Here's the simplest way to do that on a Windows system:

  • Download and install the free-as-in-beer Microsoft C++ Toolkit and the Microsoft Platform SDK.

  • Make a wine directory underneath your work directory, and copy the file wine/test.h from the Wine source tree there (you can download this file from the latest revision at http://source.winehq.org/git/?p=wine.git;a=blob_plain;f=include/wine/test.h).

  • Copy some existing test from the Wine source tree, or create your test program (say, mytest.c) using Notepad, being sure to begin it with #include <wine/test.h> following the usual Wine test style.

  • Finally, in a command prompt window, compile the test with the command

    C:\your\work\dir> cl -I. -DSTANDALONE -D_X86_ mytest.c
  • Once that's working, try running the program under Wine without recompiling it. See? No Wine source required at all, save for that one header, wine/test.h.

  • If you want to use the Microsoft C++ Toolkit under Wine, see CL Howto for some tips on making it easy to use it from the Linux command line.

6 Inside a Test

When writing new checks you can either modify an existing test file or add a new one. If your tests are related to the tests performed by an existing file, then add them to that file. Otherwise create a new .c file in the tests directory and add that file to the CTESTS variable in Makefile.in.

A new test file will look something like the following:

#include <wine/test.h>
#include <winbase.h>

/* Maybe auxiliary functions and definitions here */

START_TEST(paths)
{
   /* Write your checks there or put them in functions you will call from
    * there
    */
}

The test entry point is the START_TEST section. This is where execution will start. You can put all your tests in that section but it may be better to split related checks in functions you will call from the START_TEST section. The parameter to START_TEST must match the name of the C file. So in the above example the C file would be called paths.c.

Tests should start by including the wine/test.h header. This header will provide you access to all the testing framework functions. You can then include the windows header you need, but make sure to not include any Unix or Wine specific header: tests must compile on Windows.

You can use trace to print informational messages. Note that these messages will only be printed if runtest -v is being used.

trace("testing GlobalAddAtomA\n");
trace("foo=%d\n",foo);

Then just call functions and use ok to make sure that they behaved as expected:

ATOM atom = GlobalAddAtomA( "foobar" );
ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar\n" );
ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR\n" );

The first parameter of ok is an expression which must evaluate to true if the test was successful. The next parameter is a printf-compatible format string which is displayed in case the test failed, and the following optional parameters depend on the format string.

7 Writing good error messages

The message that is printed when a test fails is extremely important.

Someone will take your test, run it on a Windows platform that you don't have access to, and discover that it fails. They will then post an email with the output of the test, and in particular your error message. Someone, maybe you, will then have to figure out from this error message why the test failed.

If the error message contains all the relevant information that will be easy. If not, then it will require modifying the test, finding someone to compile it on Windows, sending the modified version to the original tester and waiting for his reply. In other words, it will be long and painful.

So how do you write a good error message? Let's start with an example of a bad error message:

ok(GetThreadPriorityBoost(curthread,&disabled)!=0,
   "GetThreadPriorityBoost Failed\n");

This will yield:

thread.c:123: Test failed: GetThreadPriorityBoost Failed

Did you notice how the error message provides no information about why the test failed? We already know from the line number exactly which test failed. In fact the error message gives strictly no information that cannot already be obtained by reading the code. In other words it provides no more information than an empty string!

Let's look at how to rewrite it:

BOOL rc;
...
rc=GetThreadPriorityBoost(curthread,&disabled);
ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d\n",
   rc,GetLastError(),disabled);

This will yield:

thread.c:123: Test failed: rc=0 error=120 disabled=0

When receiving such a message, one would check the source, see that it's a call to GetThreadPriorityBoost, that the test failed not because the API returned the wrong value, but because it returned an error code. Furthermore we see that GetLastError() returned 120 which winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of the problem is obvious: this Windows platform (here Windows 98) does not support this API and thus the test must be modified to detect such a condition and skip the test.

So a good error message should provide all the information which cannot be obtained by reading the source, typically the function return value, error codes, and any function output parameter. Even if more information is needed to fully understand a problem, systematically providing the above is easy and will help cut down the number of iterations required to get to a resolution.

It may also be a good idea to dump items that may be hard to retrieve from the source, like the expected value in a test if it is the result of an earlier computation, or comes from a large array of test values (e.g. index 112 of _pTestStrA in vartest.c). In that respect, for some tests you may want to define a macro such as the following:

#define eq(received, expected, label, type) \
        ok((received) == (expected), "%s: got " type " instead of " type "\n", (label),(received),(expected))

...

eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );

8 Handling platform issues

Some checks may be written before they pass successfully in Wine. Without some mechanism, such checks would potentially generate hundred of known failures for months each time the tests are being run. This would make it hard to detect new failures caused by a regression. or to detect that a patch fixed a long standing issue.

Thus the Wine testing framework has the concept of platforms and groups of checks can be declared as expected to fail on some of them. In the most common case, one would declare a group of tests as expected to fail in Wine. To do so, use the following construct:

todo_wine {
    SetLastError( 0xdeadbeef );
    ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0\n" );
}

On Windows the above check would be performed normally, but on Wine it would be expected to fail, and not cause the failure of the whole test. However. If that check were to succeed in Wine, it would cause the test to fail, thus making it easy to detect when something has changed that fixes a bug. Also note that todo checks are accounted separately from regular checks so that the testing statistics remain meaningful.

When writing tests you will also encounter differences between Windows 9x and Windows NT platforms. Such differences should be treated differently from the platform issues mentioned above. In particular you should remember that the goal of Wine is not to be a clone of any specific Windows version but to run Windows applications on Unix.

So, if an API returns a different error code on Windows 9x and Windows NT, your check should just verify that Wine returns one or the other:

ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);

If an API is only present on some Windows platforms, then use LoadLibrary and GetProcAddress to check if it is implemented and invoke it. Remember, tests must run on all Windows platforms. Similarly, conformance tests should not try to correlate the Windows version returned by GetVersion with whether given APIs are implemented or not. Again, the goal of Wine is to run Windows applications (which do not do such checks), and not be a clone of a specific Windows version.