View source for Make Test Failures

== Tracking "make test" failures ==

== Build and run Winetest to upload fresh results ==
See ConformanceTests for information on running the test suite.

== Known Good Window Managers ==
Message-sequence tests usually fail because of window manager bugs. Alexandre has been working with window manager developers to get these fixed. At the moment, known good window managers are:

* metacity 2.23.8 or later (see http://bugs.winehq.org/show_bug.cgi?id=9916#c69 for how to build and run that)
* fvwm 2.5.26.  (Alexandre says: "fvwm 2.4.20 may also work on 32-bit, the bug I fixed was for 64-bit platforms".)
* KDE in Gutsy seems to be ok, too, though Alexandre hasn't blessed it
So - if you are seeing message sequence failures, check to see if switching to one of the above window manager versions helps.

== Known Good Graphics Cards ==
The only graphics cards known to pass all d3d9 tests are Nvidia GeForce 6 and higher.  (Note: a few TODO's are expected, that's ok.)
For instance, here are lspci strings of cards that have passed the tests:
* nVidia Corporation G70 [GeForce 7600 GT] (rev a1)
* nVidia Corporation G80 [GeForce 8800 GTX] (rev a2)
* nVidia Corporation NV40 [GeForce 6800] (rev a1)
* nVidia Corporation NV41GL [Quadro FX 1400] (rev a2)
* nVidia Corporation GeForce 9800 GTX+ (rev a2)

ATI cards are known to fail the tests due to bugs like
* http://ati.cchtml.com/show_bug.cgi?id=1031
* http://ati.cchtml.com/show_bug.cgi?id=1030
ATI is willing to fix bugs that affect games badly, but aren't eager to
fix conformance test failures that don't correspond to a bug in a popular game.

== Flaky tests ==

[http://code.google.com/p/winezeug/source/browse/trunk/patchwatcher/patchwatcher.sh#63 patchwatcher] ignores failures in the a number of tests because they seem to fail intermittantly.  The blacklist as of 27 Aug 2008 is:
* comctl32:tooltips.c
* d3d9:device.c
* d3d9:visual.c
* ddraw:visual.c
* kernel32:thread.c
* urlmon:protocol.c
* urlmon:url.c
* user32:msg.c
* user32:input.c
* user32:monitor.c
* wininet:http.c

More flaky tests:
* ole32:marshal.c - random timeout, crash or success
* ole32:clipboard.c - even after fix in 1.1.31
* oleaut32:tmarshal.c and likely some more ole* tests
* winmm:wave - sometimes `waveOutGetPosition(0): returned 10787 samples (10787 bytes), should be 22050 (22050 bytes)`, followed by `SMPTE/MIDI/TICKS test failed`

Some tests fail with WINEDEBUG set. A few of these are a WONTFIX:
* crypt32:sip.c - +relay - "+relay adds a wrapper around functions, so it's normal that addresses no longer match." - [[User:AlexandreJulliard|Alexandre Julliard]]
* rpcrt4:cstub.c - +relay - "Functions pointers are changed by +relay, same issue as bug 17517." - [[User:AlexandreJulliard|Alexandre Julliard]]
* user32:class.c - +relay - "Functions pointers are changed by +relay, same issue as bug 17517." - [[User:AlexandreJulliard|Alexandre Julliard]]

== Known bugs ==
We currently have a couple general bugs related to "make test" failures on Wine:
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=9916 bug 9916]</strike>: "make test" usually fails
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12074 bug 12074]</strike>:  The conformance tests fail on Windows
* [http://bugs.winehq.org/show_bug.cgi?id=7915 bug 7915]: Wine tests fail on Mac OS X
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=7645 bug 7645]</strike>:  Wine tests fail under FreeBSD and other problems
As well as several specific failures:

* <strike>[http://bugs.winehq.org/show_bug.cgi?id=10221 bug 10221]</strike>:  d3d9: visual.c:8694: Test failed: TSSARGTEMP test returned color 0x00ff0000, expected 0x00FFFF00
* [http://bugs.winehq.org/show_bug.cgi?id=19183 bug 19183]:  d3d9: visual flakey test failures
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12730 bug 12730]</strike>:  gdi32: some tests fail when X is run in 16 bit mode, but not 32 bit fails
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12768 bug 12768]</strike>:  gdi32: metafile.ok fails in a virtual desktop
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12925 bug 12925]</strike>:  d3d8: Visual test fails
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=17416 bug 17416]</strike>:  urlmon: url sometimes fails wit 2 errors

and several platform-specific failures (hmm, maybe we went a bit overboard with bug reports here):
* [http://bugs.winehq.org/show_bug.cgi?id=12840 bug 12840]:  kernel32: change.ok test fails in PC-BSD but not Linux
* [http://bugs.winehq.org/show_bug.cgi?id=12854 bug 12854]:  ntdll change.ok test fails in PC-BSD and OS X but not Linux
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12858 bug 12858]</strike>:  winmm: wave.ok test fails in PC-BSD but not Linux
* [http://bugs.winehq.org/show_bug.cgi?id=12891 bug 12891]:  ntdll: exception.ok test fails in PC-BSD, Solaris and OS X but not Linux
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12894 bug 12894]</strike>:  kernel32: thread test fails in PC-BSD but not Linux
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=12896 bug 12896]</strike>:  kernel32: pipe.ok hangs in PC-BSD but not Linux
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=13230 bug 13230]</strike>:  msi: msi.ok fails in Opensolaris
* <strike>[http://bugs.winehq.org/show_bug.cgi?id=15538 bug 15538]</strike>:  gdi32: path.ok test succeeds in todo block in PC-BSD and OS X

== Automated test reports ==
See also ConformanceTests for how to look at or contribute to the automated test result reporting system.

When you run the 'dotests' script, your results should show up at http://test.winehq.org within a few seconds.

You can use the perl script http://kegel.com/wine/skipgood.txt to ignore tests which pass everywhere. 
Its output for wine-1.0-rc2 as of May 24 is at http://kegel.com/wine/failing-1.0-rc2.html .
The skipgood script above now also tallies tests by how many people they fail for; see below.

== Most Common Failures on XP ==
As of 2011/08/24, the tests that fail the most on machines running XP are:

 gdi32:font urlmon:protocol urlmon:url user32:cursoricon ws2_32:sock



== Most Common Failures on Wine ==
As of 1.0-rc2, here are the tests that fail on some Wine systems, and how many people they fail for:
 14 user32:msg
 12 user32:win
 9 user32:input
 6 dsound:ds3d dsound:ds3d8 kernel32:path user32:monitor
 5 kernel32:process msi:install msi:package riched20:editor
 4 comctl32:listview ddraw:visual dinput:device gdi32:font gdi32:metafile kernel32:profile kernel32:sync kernel32:virtual snmpapi:util urlmon:protocol user32:class wininet:internet
 3 comctl32:monthcal ddraw:d3d kernel32:volume msi:msi setupapi:devinst shell32:shlfolder user32:menu userenv:userenv
 2 advapi32:registry advapi32:security advapi32:service comctl32:tooltips d3d8:visual d3d9:visual ddraw:dsurface dsound:dsound kernel32:version msvcrt:time ntdll:exception psapi:psapi_main qedit:mediadet
 1 advpack:advpack advpack:install ...[many deleted]... wininet:http winmm:wave winspool.drv:info


Whew, that's a lot of data.  I looked at all the ones that failed for five or more people:

user32:msg, user32:win, and user32:input are probably people running non-blessed window managers; Alexandre's position is that we can ignore these, since upstream gnome's WM is fixed.

dsound:ds3d is mostly "ds3d.c:891: Test failed: The primary volume changed from -27 to -960" (worrisome, random)
plus "ds3d.c:896: Test failed: The primary pan changed from -5594 to -5595" (rounding error, just add slop of 1?).
dsound:ds3d8 is the same.

kernel32:path is mostly "path.c:899:TMP=c:\windows\temp"..."path.c:1178: Test failed: expected buf[0] upper case letter got c",
probably people whose ~/.wine is old and has a windir that starts with a lowercase drive letter?!
Has that changed recently?

msi:install is mostly "install.c:4414: Test failed: File installed".  

msi:package is mostly "package.c:1661: Test failed: wrong return val (0)"

riched20:editor has several failures.

d3d9:visual failures are all from people not running an approved graphics card, and can be ignored (though maybe we should tell users we only support nvidia Geforce 6 and higher for 3d graphics...)

Here are some older notes from a previous build:

kernel32:process is mostly "process.c:1237: Test failed: Wrong cursor position"

ddraw:visual has two kinds of errors.  The most common is "visual.c:2045: Test failed: Got color 00dcdde6, expected 0000ff00".  Less common is mstefani's failure, "visual.c:481: Test failed: Untransformed vertex with linear vertex fog has color 00ff0000".  [fixme: file bugs?]

urlmon:protocol has three or so failure modes; most common is "protocol.c:1597: Test failed: Read failed: 8000000a (0 bytes)".  [fixme: file bug?]

gdi32:font has several failure modes, not sure what's going on there.  [fixme: file bug?]

----
[[Category:QualityAssurance]]