Debugging Wine

Introduction

Typical debugging patterns

Wine provides a variety of methods for debugging problems. Most Wine developers prefer to use Wine's debug channels to find problems. You can read more about how to structure such messages in the developer's guide to debug logging.

The remainder of this page goes into details about the use of Wine's internal debugger, winedbg.

Processes and threads: in underlying OS and in Windows

Before going into the depths of debugging in Wine, here's a small overview of process and thread handling in Wine. It has to be clear that there are two different beasts: processes/threads from the Unix point of view and processes/threads from a Windows point of view.

Each Windows thread is implemented as a Unix thread, meaning that all threads of a same Windows process share the same (unix) address space.

In the following:

• W-process means a process in Windows terminology
• U-process means a process in Unix terminology
• W-thread means a thread in Windows terminology

A W-process is made of one or several W-threads. Each W-thread is mapped to one and only one U-process. All U-processes of a same W-process share the same address space.

Each Unix process can be identified by two values:

• the Unix process id (upid in the following)
• the Windows thread id (tid)

Each Windows process has also a Windows process id (wpid in the following). It must be clear that upid and wpid are different and shall not be used instead of the other.

Wpid and tid are defined (Windows) system wide. They must not be confused with process or thread handles which, as any handle, are indirections to system objects (in this case processes or threads). A same process can have several different handles on the same kernel object. The handles can be defined as local (the values is only valid in a process), or system wide (the same handle can be used by any W-process).

Wine, debugging and WineDbg

When talking of debugging in Wine, there are at least two levels to think of:

• the Windows debugging API.
• the Wine integrated debugger, dubbed winedbg.

Wine implements most of the Windows debugging API. The first part of the debugging APIs (in KERNEL32.DLL) allows a W-process, called the debugger, to control the execution of another W-process, the debuggee. To control means stopping/resuming execution, enabling/disabling single stepping, setting breakpoints, reading/writing debuggee memory... Another part of the debugging APIs resides in DBGHELP.DLL (and its ancestor IMAGEHLP.DLL) and lets a debugger look into symbols and types from any module (if the module has been compiled with the proper options).

winedbg is a Winelib application making use of these APIs (KERNEL32.DLL debugging API and class="libraryfile"DBGHELP.DLL) to allow debugging both any Wine or Winelib application as well as Wine itself (kernel and all DLLs).

Debugging Tutorials

These tutorials are aimed at people who know C and programming, but are just starting out with Wine development. They're meant to show you how to debug problems when apps don't work.

• Debugging Reason 3 - a simple "Unhandled exception" error message. Introduces debug tracing, the shell DLLs and SEH/exception tracing.
• Debugging PE Explorer - fixing a simple hang in the file open dialog (yet another shell bug). Introduces backtracing using winedbg and different types of result codes.
• Debugging Wild Metal Country - finding why a game crashes (due to DirectPlay) and how to confirm the bug.
• Fixing a tiny problem in the Common Controls hotkey control
• Tracking down speed problems in riched20
• Many bug reports by Anastasius Focht contain excellent descriptions of how he found the problem; see Focht's bug reports

WineDbg modes of invocation

Starting a process

Any application (either a Windows native executable, or a Winelib application) can be run through winedbg. Command line options and tricks are the same as for wine:

winedbg telnet.exe
winedbg hl.exe -windowed

Attaching

winedbg can also be launched without any command line argument: winedbg is started without any attached process. You can get a list of running W-processes (and their wpid) using the info process command, and then, with the attach command, pick up the wpid of the W-process you want to debug. This is a neat feature as it allows you to debug an already started application. Beware that wpid' printed in info process' commands are hexadecimal values (even if they don't have a 0x' prefix) but the attach command expects the 0x' prefix.

On exceptions

When something goes wrong, Windows tracks this as an exception. Exceptions exist for segmentation violation, stack overflow, division by zero, etc.

When an exception occurs, Wine checks if the W-process is debugged. If so, the exception event is sent to the debugger, which takes care of it: end of the story. This mechanism is part of the standard Windows debugging API.

If the W-process is not debugged, Wine tries to launch a debugger. This debugger (normally winedbg, see III Configuration for more details), at startup, attaches to the W-process which generated the exception event. In this case, you are able to look at the causes of the exception, and either fix the causes (and continue further the execution) or dig deeper to understand what went wrong.

If winedbg is the standard debugger, the pass and cont commands are the two ways to let the process go further for the handling of the exception event.

To be more precise on the way Wine (and Windows) generates exception events, when a fault occurs (segmentation violation, stack overflow...), the event is first sent to the debugger (this is known as a first chance exception). The debugger can give two answers:

continue the debugger had the ability to correct what's generated the exception, and is now able to continue process execution.

pass the debugger couldn't correct the cause of the first chance exception. Wine will now try to walk the list of exception handlers to see if one of them can handle the exception. If no exception handler is found, the exception is sent once again to the debugger to indicate the failure of the exception handling.

Note: since some of Wine code uses exceptions and try/catch blocks to provide some functionality, winedbg can be entered in such cases with segv exceptions. This happens, for example, with IsBadReadPtr function. In that case, the pass command shall be used, to let the handling of the exception to be done by the catch block in IsBadReadPtr.

Interrupting

You can stop the debugger while it's running by hitting Ctrl+C in its window. This will stop the debugged process, and let you manipulate the current context.

Quitting

Wine supports the new XP APIs, allowing for a debugger to detach from a program being debugged (see detach command).

Using the Wine Debugger

This section describes where to start debugging Wine. If at any point you get stuck and want to ask for help, please read the How to Report A Bug section of the Wine Users Guide for information on how to write useful bug reports.

Crashes

These usually show up like this:

Unhandled exception: page fault on write access to 0x00000000 in 32-bit code (0x0043369e).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
 EIP:0043369e ESP:0b3ee90c EBP:0b3ee938 EFLAGS:00010246(  R- --  I  Z- -P- )
 EAX:00000072 EBX:7b8acff4 ECX:00000000 EDX:6f727265
 ESI:7ba3b37c EDI:7ffa0000
Stack dump:
0x0b3ee90c:  7b82ced8 00000000 7ba3b348 7b884401
0x0b3ee91c:  7b883cdc 00000008 00000000 7bc36e7b
0x0b3ee92c:  7b8acff4 7b82ceb9 7b8acff4 0b3eea18
0x0b3ee93c:  7b82ce82 00000000 00000000 00000000
0x0b3ee94c:  00000000 0b3ee968 70d7ed7b 70c50000
0x0b3ee95c:  00000000 0b3eea40 7b87fd40 7b82d0d0
Backtrace:
=>0 0x0043369e in elementclient (+0x3369e) (0x0b3ee938)
  1 0x7b82ce82 CONSOLE_SendEventThread+0xe1(pmt=0x0(nil)) [/usr/src/debug/wine-1.5.14/dlls/kernel32/console.c:1989] in kernel32 (0x0b3eea18)
  2 0x7bc76320 call_thread_func_wrapper+0xb() in ntdll (0x0b3eea28)
  3 0x7bc7916e call_thread_func+0x7d(entry=0x7b82cda0, arg=0x0(nil), frame=0xb3eeb18) [/usr/src/debug/wine-1.5.14/dlls/ntdll/signal_i386.c:2522] in ntdll (0x0b3eeaf8)
  4 0x7bc762fe RtlRaiseException+0x21() in ntdll (0x0b3eeb18)
  5 0x7bc7f3da start_thread+0xe9(info=0x7ffa0fb8) [/usr/src/debug/wine-1.5.14/dlls/ntdll/thread.c:408] in ntdll (0x0b3ef368)
  6 0xf7597adf start_thread+0xce() in libpthread.so.0 (0x0b3ef468)
0x0043369e: movl    %edx,0x0(%ecx)
Modules:
Module  Address         Debug info  Name (143 modules)
PE    340000-  3af000   Deferred        speedtreert
PE    3b0000-  3d6000   Deferred        ftdriver
PE    3e0000-  3e6000   Deferred        immwrapper
PE    400000-  b87000   Export          elementclient
PE    b90000-  e04000   Deferred        elementskill
PE    e10000-  e42000   Deferred        ifc22
PE  10000000-10016000   Deferred        zlibwapi
ELF 41f75000-41f7e000   Deferred        librt.so.1
ELF 41ff9000-42012000   Deferred        libresolv.so.2
PE  48080000-480a8000   Deferred        msls31
PE  65340000-653d2000   Deferred        oleaut32
PE  70200000-70294000   Deferred        wininet
PE  702b0000-70328000   Deferred        urlmon
PE  70440000-704cf000   Deferred        mlang
PE  70bd0000-70c34000   Deferred        shlwapi
PE  70c50000-70ef3000   Deferred        mshtml
PE  71930000-719b8000   Deferred        shdoclc
PE  78130000-781cb000   Deferred        msvcr80
ELF 79afb000-7b800000   Deferred        libnvidia-glcore.so.304.51
ELF 7b800000-7ba3d000   Dwarf           kernel32<elf>
  \-PE  7b810000-7ba3d000   \               kernel32
ELF 7bc00000-7bcd5000   Dwarf           ntdll<elf>
  \-PE  7bc10000-7bcd5000   \               ntdll
ELF 7bf00000-7bf04000   Deferred        <wine-loader>
ELF 7c288000-7c400000   Deferred        libvorbisenc.so.2
PE  7c420000-7c4a7000   Deferred        msvcp80
ELF 7c56d000-7c5b6000   Deferred        dinput<elf>
  \-PE  7c570000-7c5b6000   \               dinput
ELF 7c5b6000-7c600000   Deferred        libdbus-1.so.3
ELF 7c70e000-7c715000   Deferred        libasyncns.so.0
ELF 7c715000-7c77e000   Deferred        libsndfile.so.1
ELF 7c77e000-7c7e5000   Deferred        libpulsecommon-1.1.so
ELF 7c7e5000-7c890000   Deferred        krnl386.exe16.so
PE  7c7f0000-7c890000   Deferred        krnl386.exe16
ELF 7c890000-7c900000   Deferred        ieframe<elf>
  \-PE  7c8a0000-7c900000   \               ieframe
ELF 7ca00000-7ca1a000   Deferred        rasapi32<elf>
  \-PE  7ca10000-7ca1a000   \               rasapi32
ELF 7ca1a000-7ca21000   Deferred        libnss_dns.so.2
ELF 7ca21000-7ca25000   Deferred        libnss_mdns4_minimal.so.2
ELF 7ca25000-7ca2d000   Deferred        libogg.so.0
ELF 7ca2d000-7ca5a000   Deferred        libvorbis.so.0
ELF 7cd5d000-7cd9c000   Deferred        libflac.so.8
ELF 7cd9c000-7cdea000   Deferred        libpulse.so.0
ELF 7cdfe000-7ce23000   Deferred        iphlpapi<elf>
  \-PE  7ce00000-7ce23000   \               iphlpapi
ELF 7cff1000-7cffd000   Deferred        libnss_nis.so.2
ELF 7d60d000-7d629000   Deferred        wsock32<elf>
  \-PE  7d610000-7d629000   \               wsock32
ELF 7d80d000-7d828000   Deferred        libnsl.so.1
ELF 7d8cf000-7d8db000   Deferred        libgsm.so.1
ELF 7d8db000-7d903000   Deferred        winepulse<elf>
  \-PE  7d8e0000-7d903000   \               winepulse
ELF 7d95c000-7d966000   Deferred        libwrap.so.0
ELF 7d966000-7d96d000   Deferred        libxtst.so.6
ELF 7d96d000-7d992000   Deferred        mmdevapi<elf>
  \-PE  7d970000-7d992000   \               mmdevapi
ELF 7d9b3000-7d9d0000   Deferred        msimtf<elf>
  \-PE  7d9c0000-7d9d0000   \               msimtf
ELF 7d9d0000-7d9e5000   Deferred        comm.drv16.so
PE  7d9e0000-7d9e5000   Deferred        comm.drv16
ELF 7da83000-7db5f000   Deferred        libgl.so.1
ELF 7db60000-7db63000   Deferred        libx11-xcb.so.1
ELF 7db63000-7db78000   Deferred        system.drv16.so
PE  7db70000-7db78000   Deferred        system.drv16
ELF 7db98000-7dca1000   Deferred        opengl32<elf>
  \-PE  7dbb0000-7dca1000   \               opengl32
ELF 7dca1000-7dcb6000   Deferred        vdmdbg<elf>
  \-PE  7dcb0000-7dcb6000   \               vdmdbg
ELF 7dcce000-7dd04000   Deferred        uxtheme<elf>
  \-PE  7dcd0000-7dd04000   \               uxtheme
ELF 7dd04000-7dd0a000   Deferred        libxfixes.so.3
ELF 7dd0a000-7dd15000   Deferred        libxcursor.so.1
ELF 7dd16000-7dd1f000   Deferred        libjson.so.0
ELF 7dd24000-7dd38000   Deferred        psapi<elf>
  \-PE  7dd30000-7dd38000   \               psapi
ELF 7dd78000-7dda1000   Deferred        libexpat.so.1
ELF 7dda1000-7ddd6000   Deferred        libfontconfig.so.1
ELF 7ddd6000-7dde6000   Deferred        libxi.so.6
ELF 7dde6000-7ddef000   Deferred        libxrandr.so.2
ELF 7ddef000-7de11000   Deferred        libxcb.so.1
ELF 7de11000-7df49000   Deferred        libx11.so.6
ELF 7df49000-7df5b000   Deferred        libxext.so.6
ELF 7df5b000-7df75000   Deferred        libice.so.6
ELF 7df75000-7e005000   Deferred        winex11<elf>
  \-PE  7df80000-7e005000   \               winex11
ELF 7e005000-7e0a5000   Deferred        libfreetype.so.6
ELF 7e0a5000-7e0c5000   Deferred        libtinfo.so.5
ELF 7e0c5000-7e0ea000   Deferred        libncurses.so.5
ELF 7e123000-7e1eb000   Deferred        crypt32<elf>
  \-PE  7e130000-7e1eb000   \               crypt32
ELF 7e1eb000-7e235000   Deferred        dsound<elf>
  \-PE  7e1f0000-7e235000   \               dsound
ELF 7e235000-7e2a7000   Deferred        ddraw<elf>
  \-PE  7e240000-7e2a7000   \               ddraw
ELF 7e2a7000-7e3e3000   Deferred        wined3d<elf>
  \-PE  7e2b0000-7e3e3000   \               wined3d
ELF 7e3e3000-7e417000   Deferred        d3d8<elf>
  \-PE  7e3f0000-7e417000   \               d3d8
ELF 7e417000-7e43b000   Deferred        imm32<elf>
  \-PE  7e420000-7e43b000   \               imm32
ELF 7e43b000-7e46f000   Deferred        ws2_32<elf>
  \-PE  7e440000-7e46f000   \               ws2_32
ELF 7e46f000-7e49a000   Deferred        msacm32<elf>
  \-PE  7e470000-7e49a000   \               msacm32
ELF 7e49a000-7e519000   Deferred        rpcrt4<elf>
  \-PE  7e4b0000-7e519000   \               rpcrt4
ELF 7e519000-7e644000   Deferred        ole32<elf>
  \-PE  7e530000-7e644000   \               ole32
ELF 7e644000-7e6f7000   Deferred        winmm<elf>
  \-PE  7e650000-7e6f7000   \               winmm
ELF 7e6f7000-7e7fa000   Deferred        comctl32<elf>
  \-PE  7e700000-7e7fa000   \               comctl32
ELF 7e7fa000-7ea23000   Deferred        shell32<elf>
  \-PE  7e810000-7ea23000   \               shell32
ELF 7ea23000-7eaf9000   Deferred        gdi32<elf>
  \-PE  7ea30000-7eaf9000   \               gdi32
ELF 7eafb000-7eaff000   Deferred        libnvidia-tls.so.304.51
ELF 7eaff000-7eb09000   Deferred        libxrender.so.1
ELF 7eb09000-7eb0f000   Deferred        libxxf86vm.so.1
ELF 7eb0f000-7eb18000   Deferred        libsm.so.6
ELF 7eb18000-7eb32000   Deferred        version<elf>
  \-PE  7eb20000-7eb32000   \               version
ELF 7eb32000-7ec87000   Deferred        user32<elf>
  \-PE  7eb40000-7ec87000   \               user32
ELF 7ec87000-7ecf1000   Deferred        advapi32<elf>
  \-PE  7ec90000-7ecf1000   \               advapi32
ELF 7ecf1000-7ed8f000   Deferred        msvcrt<elf>
  \-PE  7ed00000-7ed8f000   \               msvcrt
ELF 7ef8f000-7ef9c000   Deferred        libnss_files.so.2
ELF 7ef9c000-7efc7000   Deferred        libm.so.6
ELF 7efc8000-7efe5000   Deferred        libgcc_s.so.1
ELF 7efe5000-7f000000   Deferred        crtdll<elf>
  \-PE  7eff0000-7f000000   \               crtdll
ELF f73d0000-f73d4000   Deferred        libxinerama.so.1
ELF f73d4000-f73d8000   Deferred        libxau.so.6
ELF f73da000-f73df000   Deferred        libdl.so.2
ELF f73df000-f7591000   Dwarf           libc.so.6
ELF f7591000-f75ab000   Dwarf           libpthread.so.0
ELF f75ab000-f76ef000   Dwarf           libwine.so.1
ELF f7722000-f7728000   Deferred        libuuid.so.1
ELF f7729000-f774a000   Deferred        ld-linux.so.2
ELF f774a000-f774b000   Deferred        [vdso].so
Threads:
process  tid      prio (all id:s are in hex)
00000008 (D) C:\Perfect World Entertainment\Perfect World International\element\elementclient.exe
    00000031    0 <==
    00000035   15
    00000012    0
    00000021    0
    00000045    0
    00000044    0
    00000043    0
    00000038   15
    00000037    0
    00000036   15
    00000034    0
    00000033    0
    00000032    0
    00000027    0
    00000009    0
0000000e services.exe
    0000000b    0
    00000020    0
    00000017    0
    00000010    0
    0000000f    0
00000014 winedevice.exe
    0000001e    0
    0000001b    0
    00000016    0
    00000015    0
0000001c plugplay.exe
    00000022    0
    0000001f    0
    0000001d    0
00000023 explorer.exe
    00000024    0

Steps to debug a crash. You may stop at any step, but please report the bug and provide as much of the information gathered to the bug report as feasible.

1. Get the reason for the crash. This is usually a page fault, an unimplemented function in Wine, or the like. When reporting a crash, report this whole crashdump even if it doesn't make sense to you. (In this case it is page fault on write access to 0x00000000. Most likely Wine passed NULL to the application or the like.)

2. Determine the cause of the crash. Since this is usually a primary/secondary reaction to a failed or misbehaving Wine function, rerun Wine with the WINEDEBUG=+relay environment variable set. This will generate quite a lot of output, but usually the reason is located in the last calls. Those lines usually look like this:

   000d:Call advapi32.RegOpenKeyExW(00000090,7eb94da0 L"Patterns",00000000,00020019,0033f968) ret=7eb39af8
   ^^^^      ^^^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^
   |         |        |             |                 |           |                           |Return address.
   |         |        |             |                 |           |More arguments.
   |         |        |             |                 |Textual parameter.
   |         |        |             |Arguments.
   |         |        |Function called.
   |         |The module of the called function.
   |The thread in which the call was made.
   
   000d:Ret  advapi32.RegOpenKeyExW() retval=00000000 ret=7eb39af8
                                      ^^^^^^^^^^^^^^^
                                      |Return value is 32-bit and has the value 0.

3. If you have found a misbehaving Wine function, try to find out why it misbehaves. Find the function in the source code. Try to make sense of the arguments passed. Usually there is a WINE_DEFAULT_DEBUG_CHANNEL(channel); at the beginning of the source file. Rerun wine with the WINEDEBUG=+xyz,+relay environment variable set. Occasionally there are additional debug channels defined at the beginning of the source file in the form WINE_DECLARE_DEBUG_CHANNEL(channel); if so the offending function may also use one of these alternate channels. Look through the the function for TRACE_(channel)("... /n"); and add any additional channels to the command line.

4. Additional information on how to debug using the internal debugger can be found in programs/winedbg/README.

5. If this information isn't clear enough or if you want to know more about what's happening in the function itself, try running wine with WINEDEBUG=+all, which dumps ALL included debug information in wine. It is often necessary to limit the debug output produced. That can be done by piping the output through grep, or alternatively with registry keys. See the Configuring +relay behavior section for more information.

6. If even that isn't enough, add more debug output for yourself into the functions you find relevant. See The section on Debug Logging in this guide for more information. You might also try to run the program in gdb instead of using the Wine debugger. If you do that, use handle SIGSEGV nostop noprint to disable the handling of seg faults inside gdb (needed for Win16).

7. You can also set a breakpoint for that function. Start wine using winedbg instead of wine. Once the debugger is running enter break RegOpenKeyExW (replace by function you want to debug, case is relevant) to set a breakpoint. Then use continue to start normal program-execution. Wine will stop if it reaches the breakpoint. If the program isn't yet at the crashing call of that function, use continue again until you are about to enter that function. You may now proceed with single-stepping the function until you reach the point of crash. Use the other debugger commands to print registers and the like.

Program hangs, nothing happens

Start the program with winedbg instead of wine. When the program locks up switch to the winedbg terminal and press Ctrl+C. This will stop the program and let you debug the program as you would for a crash.

Program reports an error with a message box

Sometimes programs are reporting failure using more or less nondescript message boxes. We can debug this using the same method as Crashes, but there is one problem... For setting up a message box the program also calls Wine producing huge chunks of debug code.

Since the failure happens usually directly before setting up the message box you can start winedbg and set a breakpoint at MessageBoxA (called by win16 and win32 programs) and proceed with continue. With WINEDEBUG=+all Wine will now stop directly before setting up the message box. Proceed as explained above.

You can also run wine using WINEDEBUG=+relay wine program.exe 2>&1 | less -i and in less search for “MessageBox”.

Disassembling programs

You may also try to disassemble the offending program to check for undocumented features and/or use of them.

The best, freely available, disassembler for Win16 programs is Windows Codeback, archive name wcbxxx.zip (e.g. wcb105a.zip).

Disassembling win32 programs is possible using e.g. GoVest by Ansgar Trimborn. It can be found here.

You can also use the newer and better Interactive Disassembler (IDA) from DataRescue. Take a look in the AppDB for links to various versions of IDA.

Another popular disassembler is Windows Disassembler 32 from URSoft. Look for a file called w32dsm87.zip (or similar) on winsite.com or softpedia.com. It seems that Windows Disassembler 32 currently has problems working correctly under Wine, so use IDA or GoVest.

Also of considerable fame amongst disassemblers is SoftIce from NuMega. That software has since been acquired by CompuWare and made part of their Windows driver development suite. Newer versions of SoftIce needs to run as a Windows Service and therefore won't currently work under Wine.

If nothing works for you, you might try one of the disassemblers found in Google directory.

Understanding disassembled code is mostly a question of exercise. Most code out there uses standard C function entries (for it is usually written in C). Win16 function entries usually look like that:

push bp
mov bp, sp
... function code ..
retf XXXX   <--------- XXXX is number of bytes of arguments

This is a FAR function with no local storage. The arguments usually start at [bp+6] with increasing offsets. Note, that [bp+6] belongs to the rightmost argument, for exported win16 functions use the PASCAL calling convention. So, if we use strcmp(a,b) with a and b both 32-bit variables b would be at [bp+6] and a at [bp+10].

Most functions make also use of local storage in the stackframe:

enter 0086, 00
... function code ...
leave
retf XXXX

This does mostly the same as above, but also adds 0x86 bytes of stackstorage, which is accessed using [bp-xx]. Before calling a function, arguments are pushed on the stack using something like this:

push word ptr [bp-02]   <- will be at [bp+8]
push di                 <- will be at [bp+6]
call KERNEL.LSTRLEN

Here first the selector and then the offset to the passed string are pushed.

Sample debugging session

Let's debug the infamous Word SHARE.EXE message box:

|marcus@jet $ wine winword.exe
|            +---------------------------------------------+
|            | !  You must leave Windows and load SHARE.EXE|
|            |    before starting Word.                    |
|            +---------------------------------------------+

|marcus@jet $ WINEDEBUG=+relay,-debug wine winword.exe
|CallTo32(wndproc=0x40065bc0,hwnd=000001ac,msg=00000081,wp=00000000,lp=00000000)
|Win16 task 'winword': Breakpoint 1 at 0x01d7:0x001a
|CallTo16(func=0127:0070,ds=0927)
|Call WPROCS.24: TASK_RESCHEDULE() ret=00b7:1456 ds=0927
|Ret  WPROCS.24: TASK_RESCHEDULE() retval=0x8672 ret=00b7:1456 ds=0927
|CallTo16(func=01d7:001a,ds=0927)
|     AX=0000 BX=3cb4 CX=1f40 DX=0000 SI=0000 DI=0927 BP=0000 ES=11f7
|Loading symbols: /home/marcus/wine/wine...
|Stopped on breakpoint 1 at 0x01d7:0x001a
|In 16 bit mode.
|Wine-dbg>break MessageBoxA                          <---- Set Breakpoint
|Breakpoint 2 at 0x40189100 (MessageBoxA [msgbox.c:190])
|Wine-dbg>c                                            <---- Continue
|Call KERNEL.91: INITTASK() ret=0157:0022 ds=08a7
|     AX=0000 BX=3cb4 CX=1f40 DX=0000 SI=0000 DI=08a7 ES=11d7 EFL=00000286
|CallTo16(func=090f:085c,ds=0dcf,0x0000,0x0000,0x0000,0x0000,0x0800,0x0000,0x0000,0x0dcf)
|...                                                   <----- Much debug output
|Call KERNEL.136: GETDRIVETYPE(0x0000) ret=060f:097b ds=0927
                               ^^^^^^ Drive 0 (A:)
|Ret  KERNEL.136: GETDRIVETYPE() retval=0x0002 ret=060f:097b ds=0927
                                        ^^^^^^  DRIVE_REMOVEABLE
                        (It is a floppy diskdrive.)

|Call KERNEL.136: GETDRIVETYPE(0x0001) ret=060f:097b ds=0927
                               ^^^^^^ Drive 1 (B:)
|Ret  KERNEL.136: GETDRIVETYPE() retval=0x0000 ret=060f:097b ds=0927
                                        ^^^^^^  DRIVE_CANNOTDETERMINE
                        (I don't have drive B: assigned)

|Call KERNEL.136: GETDRIVETYPE(0x0002) ret=060f:097b ds=0927
                               ^^^^^^^ Drive 2 (C:)
|Ret  KERNEL.136: GETDRIVETYPE() retval=0x0003 ret=060f:097b ds=0927
                                        ^^^^^^ DRIVE_FIXED
                                               (specified as a hard disk)

|Call KERNEL.97: GETTEMPFILENAME(0x00c3,0x09278364"doc",0x0000,0927:8248) ret=060f:09b1 ds=0927
                                 ^^^^^^           ^^^^^        ^^^^^^^^^
                                 |                |            |buffer for fname
                                 |                |temporary name ~docXXXX.tmp
                                 |Force use of Drive C:.

|Warning: GetTempFileName returns 'C:~doc9281.tmp', which doesn't seem to be writable.
|Please check your configuration file if this generates a failure.

Whoops, it even detects that something is wrong!

|Ret  KERNEL.97: GETTEMPFILENAME() retval=0x9281 ret=060f:09b1 ds=0927
                                          ^^^^^^ Temporary storage ID

|Call KERNEL.74: OPENFILE(0x09278248"C:~doc9281.tmp",0927:82da,0x1012) ret=060f:09d8 ds=0927
                                    ^^^^^^^^^^^^^^^^ ^^^^^^^^^ ^^^^^^^
                                    |filename        |OFSTRUCT |open mode:

                                       OF_CREATE|OF_SHARE_EXCLUSIVE|OF_READWRITE

This fails, since my C: drive is in this case mounted readonly.

|Ret  KERNEL.74: OPENFILE() retval=0xffff ret=060f:09d8 ds=0927
                                   ^^^^^^ HFILE_ERROR16, yes, it failed.

|Call USER.1: MESSAGEBOX(0x0000,0x09278376"You must close Windows and load SHARE.EXE before you start Word.",0x00000000,0x1030) ret=060f:084f ds=0927

And MessageBox'ed.

|Stopped on breakpoint 2 at 0x40189100 (MessageBoxA [msgbox.c:190])
|190     {      <- the sourceline
In 32 bit mode.
Wine-dbg>

The code seems to find a writable harddisk and tries to create a file there. To work around this bug, you can define C: as a network drive, which is ignored by the code above.

Debugging Tips

Here are some additional debugging tips:

• If you have a program crashing at such an early loader phase that you can't use the Wine debugger normally, but Wine already executes the program's start code, then you may use a special trick. You should do a WINEDEBUG=+relay wine program

  to get a listing of the functions the program calls in its start function. Now you do a

  winedbg winfile.exe

  This way, you get into winedbg. Now you can set a breakpoint on any function the program calls in the start function and just type c to bypass the eventual calls of Winfile to this function until you are finally at the place where this function gets called by the crashing start function. Now you can proceed with your debugging as usual.

• If you try to run a program and it quits after showing an error message box, the problem can usually be identified in the return value of one of the functions executed before MessageBox(). That's why you should re-run the program with e.g. WINEDEBUG=+relay wine program_name &>relmsg

  Then do a more relmsg and search for the last occurrence of a call to the string "MESSAGEBOX". This is a line like

  Call USER.1: MESSAGEBOX(0x0000,0x01ff1246 "Runtime error 219 at 0004:1056.",0x00000000,0x1010) ret=01f7:2160 ds=01ff

  In my example the lines before the call to MessageBox() look like that:

  Call KERNEL.96: FREELIBRARY(0x0347) ret=01cf:1033 ds=01ff
  CallTo16(func=033f:0072,ds=01ff,0x0000)
  Ret  KERNEL.96: FREELIBRARY() retval=0x0001 ret=01cf:1033 ds=01ff
  Call KERNEL.96: FREELIBRARY(0x036f) ret=01cf:1043 ds=01ff
  CallTo16(func=0367:0072,ds=01ff,0x0000)
  Ret  KERNEL.96: FREELIBRARY() retval=0x0001 ret=01cf:1043 ds=01ff
  Call KERNEL.96: FREELIBRARY(0x031f) ret=01cf:105c ds=01ff
  CallTo16(func=0317:0072,ds=01ff,0x0000)
  Ret  KERNEL.96: FREELIBRARY() retval=0x0001 ret=01cf:105c ds=01ff
  Call USER.171: WINHELP(0x02ac,0x01ff05b4 "COMET.HLP",0x0002,0x00000000) ret=01cf:1070 ds=01ff
  CallTo16(func=0117:0080,ds=01ff)
  Call WPROCS.24: TASK_RESCHEDULE() ret=00a7:0a2d ds=002b
  Ret  WPROCS.24: TASK_RESCHEDULE() retval=0x0000 ret=00a7:0a2d ds=002b
  Ret  USER.171: WINHELP() retval=0x0001 ret=01cf:1070 ds=01ff
  Call KERNEL.96: FREELIBRARY(0x01be) ret=01df:3e29 ds=01ff
  Ret  KERNEL.96: FREELIBRARY() retval=0x0000 ret=01df:3e29 ds=01ff
  Call KERNEL.52: FREEPROCINSTANCE(0x02cf00ba) ret=01f7:1460 ds=01ff
  Ret  KERNEL.52: FREEPROCINSTANCE() retval=0x0001 ret=01f7:1460 ds=01ff
  Call USER.1: MESSAGEBOX(0x0000,0x01ff1246 "Runtime error 219 at 0004:1056.",0x00000000,0x1010) ret=01f7:2160 ds=01ff

  I think that the call to MessageBox() in this example is not caused by a wrong result value of some previously executed function (it's happening quite often like that), but instead the message box complains about a runtime error at 0x0004:0x1056.

  As the segment value of the address is only 4, I think that that is only an internal program value. But the offset address reveals something quite interesting: offset 1056 is very close to the return address of FREELIBRARY():

  Call KERNEL.96: FREELIBRARY(0x031f) ret=01cf:105c ds=01ff
                                               ^^^^

  Provided that segment 0x0004 is indeed segment 0x1cf, we now we can use IDA to disassemble the part that caused the error. We just have to find the address of the call to FreeLibrary(). Some lines before that the runtime error occurred. But be careful! In some cases you don't have to disassemble the main program, but instead some DLL called by it in order to find the correct place where the runtime error occurred. That can be determined by finding the origin of the segment value (in this case 0x1cf).

• If you have created a relay file of some crashing program and want to set a breakpoint at a certain location which is not yet available as the program loads the breakpoint segment during execution, you may set a breakpoint to GetVersion16/32 as those functions are called very often. Then do a c until you are able to set this breakpoint without error message.

Some basic debugger usages

After starting your program with

winedbg myprog.exe

the program loads and you get a prompt at the program starting point. Then you can set breakpoints:

b RoutineName      (by routine name) OR
b *0x812575        (by address)

Then you hit c (continue) to run the program. It stops at the breakpoint. You can type

step               (to step one line) OR
stepi              (to step one machine instruction at a time;
                    here, it helps to know the basic 386
                    instruction set)
info reg           (to see registers)
info stack         (to see hex values in the stack)
info local         (to see local variables)
list line number   (to list source code)
x variable name    (to examine a variable; only works if code
                    is not compiled with optimization)
x 0x4269978        (to examine a memory location)
?                  (help)
q                  (quit)

By hitting Enter, you repeat the last command.

Useful programs

Some useful programs:

GoVest: govest.zip is available from https://www.oocities.com/govest/.

Simple win32 disassembler that works well with Wine.

IDA:

IDA Pro is highly recommended, but is not free. DataRescue does however make trial versions available.

Take a look in the AppDB for links to various versions of IDA.

XRAY: https://garbo.uwasa.fi/pub/pc/sysinfo/xray15.zip

Traces DOS calls (Int 21h, DPMI, ...). Use it with Windows to correct file management problems etc.

pedump: https://pedump.me/

Dumps the imports and exports of a PE (Portable Executable) DLL.

winedump: (included in wine tree)

Dumps the imports and exports of a PE (Portable Executable) DLL.

Useful memory addresses

Wine uses several different kinds of memory addresses.

Win32/“normal” Wine addresses/Linux: linear addresses.

Linear addresses can be everything from 0x0 up to 0xffffffff. In Wine on Linux they are often around e.g. 0x08000000, 0x00400000 (std. Win32 program load address), 0x40000000. Every Win32 process has its own private 4GB address space (that is, from 0x0 up to 0xffffffff).

Win16 “enhanced mode”: segmented addresses.

These are the “normal” Win16 addresses, called SEGPTR. They have a segment:offset notation, e.g. 0x01d7:0x0012. The segment part usually is a “selector”, which always has the lowest 3 bits set. Some sample selectors are 0x1f7, 0x16f, 0x8f. If these bits are set except for the lowest bit, as e.g. with 0x1f6,xi then it might be a handle to global memory. Just set the lowest bit to get the selector in these cases. A selector kind of “points” to a certain linear (see above) base address. It has more or less three important attributes: segment base address, segment limit, segment access rights.

Example:

Selector 0x1f7 (0x40320000, 0x0000ffff, r-x) So 0x1f7 has a base address of 0x40320000, the segment's last address is 0x4032ffff (limit 0xffff), and it's readable and executable. So an address of 0x1f7:0x2300 would be the linear address of 0x40322300.

DOS/Win16 “standard mode”

They, too, have a segment:offset notation. But they are completely different from “normal” Win16 addresses, as they just represent at most 1MB of memory: the segment part can be anything from 0 to 0xffff, and it's the same with the offset part.

Now the strange thing is the calculation that's behind these addresses: just calculate segment*16 + offset in order to get a “linear DOS” address. So e.g. 0x0f04:0x3628 results in 0xf040 + 0x3628 = 0x12668. And the highest address you can get is 0xfffff (1MB), of course.

Configuration

Windows Debugging configuration

The Windows debugging API uses a registry entry to know which debugger to invoke when an unhandled exception occurs (see On exceptions for some details). Two values in key

[MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug]

determine the behavior:

Debugger This is the command line used to launch the debugger (it uses two printf formats (%ld) to pass context dependent information to the debugger). You should put here a complete path to your debugger (winedbg can of course be used, but any other Windows debugging API aware debugger will do). The path to the debugger you choose to use must be reachable via one of the DOS drives configured under /dosdevices in your WINEPREFIX or ~/.wine folder.

Auto If this value is zero, a message box will ask the user if he/she wishes to launch the debugger when an unhandled exception occurs. Otherwise, the debugger is automatically started.

A regular Wine registry looks like:

[MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug] 957636538
"Auto"=dword:00000001
"Debugger"="winedbg %ld %ld"

WineDbg configuration

winedbg can be configured through a number of options. Those options are stored in the registry, on a per user basis. The key is (in my registry)

[HKCU\\Software\\Wine\\WineDbg]

Those options can be read/written while inside winedbg, as part of the debugger expressions. To refer to one of these options, its name must be prefixed by a $ sign. For example,

set $BreakAllThreadsStartup = 1

sets the option BreakAllThreadsStartup to TRUE.

All the options are read from the registry when winedbg starts (if no corresponding value is found, a default value is used), and are written back to the registry when winedbg exits (hence, all modifications to those options are automatically saved when winedbg terminates).

Here's the list of all options:

BreakAllThreadsStartup Set to TRUE if at all threads start-up the debugger stops set to FALSE if only at the first thread startup of a given process the debugger stops. FALSE by default.

BreakOnCritSectTimeOut Set to TRUE if the debugger stops when a critical section times out (5 minutes); TRUE by default.

BreakOnAttach Set to TRUE if when winedbg attaches to an existing process after an unhandled exception, winedbg shall be entered on the first attach event. Since the attach event is meaningless in the context of an exception event (the next event which is the exception event is of course relevant), that option is likely to be FALSE.

BreakOnFirstChance An exception can generate two debug events. The first one is passed to the debugger (known as a first chance) just after the exception. The debugger can then decide either to resume execution (see winedbg's cont command) or pass the exception up to the exception handler chain in the program (if it exists) (winedbg implements this through the pass command). If none of the exception handlers takes care of the exception, the exception event is sent again to the debugger (known as last chance exception). You cannot pass on a last exception. When the BreakOnFirstChance exception is TRUE, then winedbg is entered for both first and last chance exceptions (to FALSE, it's only entered for last chance exceptions).

AlwaysShowThunk Set to TRUE if the debugger, when looking up for a symbol from its name, displays all the thunks with that name. The default value (FALSE) allows not to have to choose between a symbol and all the import thunks from all the DLLs using that symbols.

Configuring +relay behaviour

When setting WINEDEBUG to +relay and debugging, you might get a lot of output. You can limit the output by configuring the value RelayExclude in the registry, located under the key [HKCU\\Software\\Wine\\Debug]

Set the value of RelayExclude to a semicolon-separated list of calls to exclude, e.g. "RtlEnterCriticalSection;RtlLeaveCriticalSection;kernel32.97;kernel32.98".

RelayInclude is an option similar to RelayExclude, except that functions listed here will be the only ones included in the output.

If your application runs too slow with +relay to get meaningful output and you're stuck with multi-GB relay log files, but you're not sure what to exclude, here's a trick to get you started. First, run your application for a minute or so, piping its output to a file on disk:

WINEDEBUG=+relay wine appname.exe &>relay.log

Then run this command to see which calls are performed the most:

awk -F'(' '{print $1}' < relay.log | awk '{print $2}' | sort | uniq -c | sort

Exclude the bottom-most calls with RelayExclude after making sure that they are irrelevant, then run your application again.

WineDbg Expressions and Variables

Expressions

Expressions in Wine Debugger are mostly written in a C form. However, there are a few discrepancies:

• Identifiers can take a '!' in their names. This allow mainly to access symbols from different DLLs like USER32CreateWindowExA.
• In cast operation, when specifying a structure or an union, you must use the struct or union keyword (even if your program uses a typedef).

When specifying an identifier by its name, if several symbols with the same name exist, the debugger will prompt for the symbol you want to use. Pick up the one you want from its number.

In lots of cases, you can also use regular expressions for looking for a symbol.

winedbg defines its own set of variables. The configuration variables from above are part of them. Some others include:

$ThreadId ID of the W-thread currently examined by the debugger

$ProcessId ID of the W-thread currently examined by the debugger

registers All CPU registers are also available, using '$' as a prefix. You can use info regs to get a list of available CPU registers.

The $ThreadId and $ProcessId variables can be handy to set conditional breakpoints on a given thread or process.

WineDbg Command Reference

Misc

Flow control

cont, step, next, stepi, nexti can be postfixed by a number (N), meaning that the command must be executed N times.

Breakpoints, watch points

You can use the symbol EntryPoint to stand for the entry point of the DLL.

When setting a break/watch-point by id, if the symbol cannot be found (for example, the symbol is contained in a not yet loaded module), winedbg will recall the name of the symbol and will try to set the breakpoint each time a new module is loaded (until it succeeds).

Stack manipulation

Directory & source file manipulation

You can specify the end target (to change the 10 lines value) using the ','. For example:

Displaying

A display is an expression that's evaluated and printed after the execution of any winedbg command.

winedbg will automatically detect if the expression you entered contains a local variable. If so, display will only be shown if the context is still in the same function as the one the debugger was in when the display expression was entered.

Disassembly

Memory (reading, writing, typing)

fmt is either letter or count letter (without a space between count and letter), where letter can be

s an ASCII string

u a Unicode UTF16 string

i instructions (disassemble)

x 32-bit unsigned hexadecimal integer

d 32-bit signed decimal integer

w 16-bit unsigned hexadecimal integer

c character (only printable 0x20-0x7f are actually printed)

b 8-bit unsigned hexadecimal integer

g GUID

Information on Wine internals

Debug channels

It is possible to turn on and off debug messages as you are debugging using the set command (only for debug channels specified in WINEDEBUG environment variable). See Debug Logging for more details on debug channels.

Other debuggers

GDB mode

WineDbg can act as a remote monitor for GDB. This allows to use all the power of GDB, but while debugging wine and/or any Win32 application. To enable this mode, just add --gdb to winedbg command line. You'll end up on a GDB prompt. You'll have to use the GDB commands (not WineDbg ones).

However, some limitation in GDB while debugging wine (see below) don't appear in this mode:

• GDB will correctly present Win32 thread information and breakpoint behavior
• Moreover, it also provides support for the Dwarf II debug format (which became the default format (instead of stabs) in gcc 3.1).

A few Wine extensions available through the monitor command.

Graphical frontends to gdb

This section will describe how you can debug Wine using the GDB mode of winedbg and some graphical front ends to GDB for those of you who really like graphical debuggers.

DDD

Use the following steps, in this order:

1. Start the Wine debugger with a command line like: winedbg --gdb --no-start name of exe to debug.exe optional parameters

2. Start ddd

3. In ddd, use Open File or Open Program to point to the Wine executable.

4. In the output of above command, there's a line like target remote localhost:12345

   Copy that line and paste into ddd command pane (the one with the (gdb) prompt)

The program should now be loaded and up and running.

kdbg

Use the following steps, in this order:

1. Start the Wine debugger with a command line like: winedbg --gdb --no-start name of exe to debug.exe optional parameters

2. In the output of above command, there's a line like target remote localhost:12345

   Start kdbg with

   kdbg -r localhost:12345 wine

   localhost:12345 is not a fixed value, but has been printed in first step. “wine” should also be the full path to the Wine executable.

The program should now be loaded and up and running.

Using other Unix debuggers

You can also use other debuggers (like gdb), but you must be aware of a few items:

You need to attach the unix debugger to the correct unix process (representing the correct windows thread) (you can “guess” it from a ps fax command for example. When running the emulator, usually the first two upids are for the Windows application running the desktop, the first thread of the application is generally the third upid; when running a Winelib program, the first thread of the application is generally the first upid)

Note: If you plan to used gdb for a multi-threaded Wine application (native or Winelib), then gdb will be able to handle the multiple threads directly only if:

• Wine is running on the pthread model (it won't work in the kthread one). See the Wine architecture documentation for further details.
• gdb supports the multi-threading (you need at least version 5.0 for that).

In the unfortunate case (no direct thread support in gdb because one of the above conditions is false), you'll have to spawn a different gdb session for each Windows thread you wish to debug (which means no synchronization for debugging purposes between the various threads).

Here's how to get info about the current execution status of a certain Wine process:

Change into your Wine source dir and enter:

$ gdb wine

Switch to another console and enter ps ax | grep wine to find all wine processes. Inside gdb, repeat for all Wine processes:

(gdb) attach wpid

with wpid being the process ID of one of the Wine processes. Use

(gdb) bt

to get the backtrace of the current Wine process, i.e. the function call history. That way you can find out what the current process is doing right now. And then you can use several times:

(gdb) n

or maybe even

(gdb) b SomeFunction

and

(gdb) c

to set a breakpoint at a certain function and continue up to that function. Finally you can enter

(gdb) detach

to detach from the Wine process.

Using other Windows debuggers

You can use any Windows debugging API compliant debugger with Wine. Some reports have been made of success with VisualStudio debugger (in remote mode, only the hub runs in Wine). GoVest fully runs in Wine.

Main differences between winedbg and regular Unix debuggers