Wine and Valgrind
Valgrind is a set of tools aimed at finding bugs and performance problems in programs. By default, it catches reads of uninitialized memory, accesses to inaccessible memory, and memory leaks. It's useful for debugging wine itself, as well as windows apps running on Wine.
You need Valgrind 3.8.0 or newer to avoid Valgrind Bug #275673 (v3.8 was released in late summer 2012, but just in case there are still a few people that haven't upgraded).
To get the full power of Valgrind while debugging Wine, you need to build Wine from source after installing Valgrind. You'll also want to switch off certain compiler optimizations; the "developer tools" section of the BuildingWine page has more details.
Valgrind will work on MacOSX (starting with partial support in MacOSX v10.8 "Mountain Lion"). If you want line numbers with Valgrind on Mac, don't forget to tell Xcode to generate a .dSYM file, though you can also use gdb after the fact to look up line numbers.
In applications built with the debug version of the MS Visual C/C++ runtime, memory allocated via malloc() is always initialized. This obscures any uninitialized memory accesses that would normally appear from Valgrind (see Memory Management and the Debug Heap at MSDN for the full story).
Running Wine under Valgrind
To analyze a program running on Wine with Valgrind, just call valgrind with the appropriate options in front of the normal wine command. For example:
valgrind --trace-children=yes --vex-iropt-register-updates=allregs-at-mem-access --workaround-gcc296-bugs=yes wine foo.exe
Note that this command will also trace the wineserver if foo.exe is the first Wine application running. Unless you're specifically looking to debug wineserver (usually you'll be more interested in specific dlls), this is a waste of resources. Valgrind can be pretty intensive on the CPU so you can speed things up quite a bit by not starting the wineserver under Valgrind.
To do that, just start a dummy program under Wine first (e.g. notepad or winemine), then without closing that program, start your desired program through wine and valgrind as you normally would.
Valgrind Options Relevant for Wine
Over the years, several Valgrind options have turned out to be necessary, or at least very helpful, when analyzing Wine. Two in particular are necessary for Wine to function well under Valgrind:
--trace-children=yes is necessary because wine starts new processes for individual programs
--vex-iropt-register-updates=allregs-at-mem-access ensures that bitmap-related code, such as GetBitmapBits(), works under Valgrind
Several more are useful for suppressing known bugs or exceptions in Wine:
This option can be repeated with multiple files
--partial-loads-ok=yes silences warnings for when bit-alignments overlap both addressable and illegal ranges. In particular, try this if the last three bytes of allocations throw a lot of "Invalid Read of Size N" warnings.
There's a good chance these are false positives from vectorized string operations (Valgrind Bug Report #264936 has more info). In the Valgrind bug report, someone also mentions using GCC with the -fno-builtin-strdup compiler option to make the warnings disappear. It hasn't been tested with Wine, but the "Compiler Optimizations" section of BuildingWine has more details about changing compiler flags while building Wine.
Controlling Memory Leak Reports
There are a few more Valgrind options that let you fine-tune how much it reports about memory leaks. By default, Valgrind only gives a count of memory leaks detected at the end of execution.
To get more info, you want to pass the --leak-check=full flag when running Valgrind. This will give you individual reports about certain types of memory leaks (but not all). Once this option is set, you can use several other options to specify which kinds of memory leaks you want detailed reports for:
--show-leak-kinds=definite,possible only reports unfreed blocks with no pointers or that depend on interior pointers (the default)
Finally, by passing the --track-origins=yes option, you can ask Valgrind to give you stack backtraces to the point where a leaky memory block was allocated. This is terrifically helpful for debugging, but it has the downside of making valgrind run even slower.
Conformance Tests under Valgrind
To run Wine's ConformanceTests under Valgrind, you mainly need to set the WINETEST_WRAPPER and VALGRIND_OPTS variables, e.g...
export VALGRIND_OPTS="--trace-children=yes --track-origins=yes --gen-suppressions=all --suppressions=$HOME/valgrind-suppressions-ignore --suppressions=$HOME/valgrind-suppressions-external --num-callers=20 --workaround-gcc296-bugs=yes --vex-iropt-register-updates=allregs-at-mem-access" WINETEST_WRAPPER=valgrind make -k test > test.log 2>&1
Changes to Wine Source
There are also some tweaks you can make to the Wine source code itself that might help you uncover other problems. First, the Wine source defines two constants that can be quickly changed and might reveal more bugs:
MAX_FREE_PENDING controls how long freed blocks are kept out of circulation and is set to 1024 by default. If you think your app is hanging on to freed pointers longer than it should, increasing it and recompiling wine might help Valgrind detect use-after-free errors.
HEAP_TAIL_EXTRA_SIZE gives the size of the redzone at the end of each block. If your app nests really big structs within another struct or array, you might need to increase this to catch overrun errors. Watch out though because increasing it beyond 32 or so bytes can cause strange false positives sometimes.
Additional Wine patches
There are also a few patches that have accumulated on Bugzilla over the years to make Wine play better with Valgrind. You might be able to get some use out of them for debugging (or even refine them and help get them accepted into the wine master branch):
Clear slots in new argv (Bug #14359)
Co-operate with valgrind at execv (Bug #14361)
Enhance loader and wine_main_preload_info for valgrind (Bug #14364)
Do not touch high address space (initial stack, user_space_limit) (Bug #14365)
Co-operate with valgrind when switching stacks (Bug #14366)
Adjust timeout interval for virtulaizers (Bug #14370)
If you want to use one of Valgrind's thread checking tools too (DRD or Helgrind), you're strongly recommended to apply the patch from Bug Report #24164 before building Wine. This patch informs these tools that some functions, like EnterCriticalSection() and LeaveCriticalSection(), are synchronization primitives. It also suppresses uninteresting memory access conflicts in Wine itself, and tithout this patch, either tool will report too many warnings to be useful.
Debugging Mozilla with Valgrind has a good case-study showing how to use Valgrind
Wine bugs involving or detected by Valgrind
Valgrind bugs with Wine mentioned in the comments