WineHQ

Wine TestBot VMs: Difference between revisions

(→‎Windows configuration: Add Delivery Optimization note)
(Remove the unmaintained failing tests list)
(One intermediate revision by the same user not shown)
Line 405: Line 405:
This is the lastest audio API for Windows 7+. It may be installed through this [https://www.microsoft.com/en-us/download/details.aspx?id=35 download].
This is the lastest audio API for Windows 7+. It may be installed through this [https://www.microsoft.com/en-us/download/details.aspx?id=35 download].


== Failing tests ==
== Unix configuration ==
=== comctl32 ===
listview.c:5045: Test failed: NM_HOVER allow test: the msg sequence is not complete: expected 0087 - actual 0000
listview.c:5050: Test failed: NM_HOVER block test: the msg sequence is not complete: expected 0087 - actual 0000


See bug [http://bugs.winehq.org/show_bug.cgi?id=24631 bug 24631]. These errors still happen if the VM is given only one processor. Looking at test.winehq.org, they happen on the following systems:
* Use the fvwm window manager
comctl32 5.82.6001.18000 newtb-wvista (admin, elevated)
: Rationale: Most window managers are buggy, in particular the GNOME and KDE ones, and cause some tests to fail or even crash. fvwm is the one known 'good' windows manager.
comctl32 5.82.7601.17514 fg-win7u64-1spie9adm (admin, not elevated)
comctl32 5.82.7601.17514 w7-ie9-sp1-64-64 (admin, elevated)
comctl32 5.82.9200.16384 vpovirk-kwin8-wow64 (admin, elevated)


But they never happen on a bunch of other very similar fg-win7u64-* systems:
* Use a Windows-compatible fvwm configuration
comctl32 5.82.7600.16385 fg-win7u64-0sp (admin, elevated)
: Rationale: Some tests verify that Windows applications get the right sequence of messages when windows lose/gain focus, etc. The sequence of messages depends in part on the window manager and cannot be right if the Unix window manager does not behave like Windows does, particularly wrt. focus-follows-mouse, transient window decorations, etc.
comctl32 5.82.7601.17514 fg-win7u64-1sp (admin, elevated)
: Style * ClickToFocus
comctl32 5.82.7601.17514 fg-win7u64-1spie9 (admin, elevated)
: Style * DontStackTransientParent
comctl32 5.82.7601.17514 fg-win7u64-1spie9-fr (admin, elevated)
: Style * DecorateTransient
comctl32 5.82.7601.17514 fg-win7u64-1spie9-ja (admin, elevated)
comctl32 5.82.7601.17514 fg-win7u64-1spie9usr (user, not elevated)
comctl32 5.82.7601.17514 fg-win7u64-1spie9t64 (admin, elevated)


All together this seems to rule out randomness, a comctl32.dll version issue and an elevated/not elevated privilege issue. So I don't see what the actual issue could be (fg).
* Install ttf-mscorefonts-installer (non-free) or fonts-liberation
 
: Rationale: No Windows system would ship without a base set of fonts of various families (Serif, Sans Serif). Since the lack of such fonts is not a valid Windows configuration, it is reasonable for tests (e.g. gdiplus:font) to depend on them and fail if they are missing. Furthermore ttf-mscorefonts-installer is preferred because it provides Arial which some tests need (but they skip if Arial is missing).
 
 
propsheet.c:543: Test failed: The Add button has not been pressed!
 
See [http://bugs.winehq.org/show_bug.cgi?id=33718 bug 33718]. This failure is clearly random and impacts various Windows XP, Windows 7 VMs as well as the new WineTestBot Windows Vista VM.
 
 
propsheet.c:746: Test failed: property sheet with custom window proc: the msg 0x001c was expected, but got msg 0x0047 instead
 
See [http://bugs.winehq.org/show_bug.cgi?id=33719 bug 33719]. This failure happens on the new WineTestBot Windows Vista VM and also on the following three VMs. Unfortunately this does not point to a clear source for this issue:
5.81.4807.2300 fg-win2000-ie55
5.82.6002.18305 wtb-wvistaadm
5.82.7601.17514 fg-win7u64-1sp
 
 
=== dinput8 ===
device.c:112: Test failed: GetDeviceData() failed: 00000000
 
  The test.winehq.org data shows that this failure has been happening intermittently on the fg-win2003es-* and fg-win7u64-* VMware VMs for a long time. It is also random on the new WineTestBot's Windows XP VM.
* When it happens there is one 'Tests skipped: We're not able to inject input into Windows dinput8 with events' message instead of two.
 
=== dsound ===
capture.c:405: Test failed: GetStatus: bad status: 1
...
 
  This only happens on the new WineTestBot Windows XP VM. Switching the sound card from ac97 to es1370 does not help. See also the winmm:wave test failures.
* -> Maybe upgrade the VM.
 
=== mmdevapi ===
render.c:1003: Test failed: Position 6336 too far after playing 100ms
 
  This failure only happens in the new WineTestBot Windows 7 VM (i.e. test.winehq.org does not show it happening on any other system). It happens quite frequently but is not systematic either (probably a bit under 50%) so it's likely a race or a timing issue.
 
=== ntdll ===
exception.c:621: Test failed: B0 flag is not set in Dr6
exception.c:629: Test failed: BS flag is not set in Dr6
exception.c:637: Test failed: B0 flag is not set in Dr6
exception.c:646: Test failed: BS flag is not set in Dr6
 
  This is caused by a QEMU bug which is fixed in QEMU 1.7.0. However that version introduces another bug ([https://bugs.launchpad.net/qemu/+bug/1119686 bug 1119686]) which causes ntdll:exception to crash altogether rather early. So upgrading does not particularly help.
 
=== user32 ===
menu.c:1222: ANSI tests: string 0028FB98 hbm BF050D4A txt 0028FD18
menu.c:1400: Test failed: InsertMenuItem should have failed.
menu.c:1222: Unicode tests: string 0028FB98 hbm BF050D4A txt 0028FE18
menu.c:1400: Test failed: InsertMenuItem should have failed.
 
 
  See [http://bugs.winehq.org/show_bug.cgi?id=33720 bug 33720]. This seems to only happen on Vista and Windows 7, and only in 32-bit tests.
 
=== wininet ===
http.c:3543: Test failed: Got string: 2
http.c:3544: Test failed: unexpected size 14
http.c:3543: Test failed: Got string: 2
http.c:3544: Test failed: unexpected size 14
http: Timeout
 
  FIXME: To analyze.
 
=== winmm ===
wave.c:498: Test failed: waveOutGetPosition(0): returned 13219 bytes, should be 13230
wave.c:498: Test failed: waveOutGetPosition(0): returned 21937 bytes, should be 22050
wave.c:509: Test failed: waveOutGetPosition(0): returned 21943 samples (21943 bytes), should be 22050 (22050 bytes)
wave.c:522: Test failed: waveOutGetPosition(0): returned 995 ms, (21954 bytes), should be 1000 (22050 bytes)
wave.c:535: Test failed: waveOutGetPosition(0): SMPTE test failed
wave.c:546: Test failed: waveOutGetPosition(0): MIDI test failed
wave.c:557: Test failed: waveOutGetPosition(0): TICKS test failed
 
  This only happens on the new WineTestBot Windows XP VM. Switching the sound card from ac97 to es1370 does not help.
* -> Maybe upgrade the VM.
 
mci.c:1120: Test failed: got 0001 instead of MCI_NOTIFY_xyz 0004 from command play (aborted by close)
 
  This is an intermittent bug which looks like it may be a timing issue. It also happened on the following system:
5.2.3790.1830 fg-win2003es-1sp
5.2.3790.3959 fg-win2003es-2sp
5.2.3790.3959 fg-win2003es-ie8
5.2.3790.3959 fg-win2003es-last
6.1.7600.16385 fg-win7u64-0sp
6.1.7601.17514 fg-win7u64-1sp
6.1.7601.17514 fg-win7u64-1spie9
 
 
=== ws2_32 ===
sock.c:528: Test failed: oob_server (974): not at the OOB mark: 1
 
  This is a long-standing bug which also happens on VMware test VMs. This still happens on single-processor VMs.
 
sock.c:5892: Test failed: failed to get completion status 0
sock.c:5893: Test failed: Last error was 995
sock.c:5895: Test failed: Number of bytes transferred is 0
sock.c:5897: Test failed: Internal status is c0000120
sock.c:5958: Test failed: Last error was 995
sock.c:5962: Test failed: Internal status is c0000120
 
  These bugs go away if limiting the VM to a single CPU.

Revision as of 15:20, 15 October 2018

WineTestBot VMs

This page describes the new WineTestBot VMs and how they are configured.

Virtualization

Libvirt

Here is a list of libvirt bugs relevant to the WineTestBot:

  • Libvirt bug 910711 - snapshot-revert causes spurious hardware mismatch errors
  • This may actually be a QEMU bug as it's the switch from QEMU 1.1.2 to 1.3.90 which seems to trigger this issue.
  • virt-manager bug 912810 - VM display is garbled if in 16-bpp

QEMU / KVM

Here is a list of QEMU / KVM bugs relevant to the WineTestBot:

Spice QXL driver

Here is a list of the Spice QXL driver bugs relevant to the WineTestBot:

Net::SSH2

We use Net::SSH2 for the file transfers between the WineTestBot and the VMs.


VM hardware and drivers

Compatibility tables

Sound

Windows ac97 es1370 ich6 sb16
2000 ok broken (1) no driver broken (2)
XP 52 test failures 60 test failures no driver broken (3)
2003 ok no driver no driver no driver
Vista ok 3 test failures no driver no driver
Vista 64-bit no driver no driver broken (4) no driver
2008 64-bit no driver no driver broken (5) no driver
7 no driver no driver ok no driver
7 64-bit no driver no driver ok no driver

Notes:

  1. With the es1370 driver dsound:capture causes Windows 2000 to freeze.
  2. Windows 2000 does not detect the Sound Blaster 16 card but its driver can be installed manually. However the driver then fails to load with code 10.
  3. Windows XP does not detect the PCI Sound Blaster 16 card but its driver can be installed manually. However that then breaks the network cards (rtl8139, e1000 and virtio).
  4. The High Definition Audio Device driver (6.0.6000.16386) fails to load with code 10.
  5. The High Definition Audio Device driver (6.0.6001.18000) fails to load with code 10.
  6. pcspk does not count as a sound card.


Video

Windows cirrus qxl vga vmvga
2000 317 test failures vga fallback (1) vga fallback (1) vga fallback (1)
XP 236 test failures 117 test failures (2) empty driver name ok
2003 ok (5, 7) ok ok ok (5)
Vista vga fallback; 234 test failures ok (2) ok ok (3)
Vista 64-bit vga fallback; 233 test failures 1 test failure ok ok (4)
2008 64-bit vga fallback; 233 test failures broken (6) ok ok (4)
7 vga fallback; 234 test failures ok (2) ok ok (4)
7 64-bit vga fallback; 234 test failures ok (2) ok ok (4)

Notes:

  1. The Windows 2000 fallback driver only support the standard VGA mode: 640x480x4.
  2. Using the 'Red Hat QXL GPU' 6.1.0.10016 driver.
  3. Using the 'VMware SVGA II' 11.9.1.0 XPDM driver from VMware Tools 9.0.1-913578.
  4. Using the 'VMware SVGA II' 11.6.0.38 XPDM driver from VMware Tools 8.3.17-870839.
  5. Using the default VGA driver.
  6. The 'Red Hat QXL GPU' 6.1.0.10016 driver fails to load with code 39.
  7. Operates in 24bpp only.
  8. None of the Windows versions has a xen driver.


Network card

Windows e1000 ne2k_pci pcnet rtl8139 virtio
2000 no driver ok (1) ok crash (2) no driver
XP no driver ok ok ok 1 test failure
2003 ok ok ok ok ???
Vista 64-bit ok no driver no driver ok 1 test failure
2008 64-bit ok no driver no driver ok 1 test failure
7 ok no driver no driver ok 1 test failure

Notes:

  1. With the ne2k_pci card network transfers cause high CPU usage and are slow (~10Mbps).
  2. The rtl8139 card causes Windows 2000 to crash on startup.

Processors

Ideally all VMs would be configured with two processors by default. However this currently seems to trigger quite a few race conditions in our tests (e.g. in ws2_32:sock) so they have currently all been configured with only one processor, except for the w864 VM. This issue will be revisited when the WineTestBot is in service.

Virtual disk

Some tests require reasonable disk performance otherwise they time out (in particular the msi tests). For these we need to make sure to use QEMU's paravirtualized 'virtio' disk. The driver comes from KVM/Fedora: VirtIO drivers for Windows from Fedora.

QXL graphics driver

Some VMs use the QXL paravirtualized graphics card together with the driver from Spice's Windows Guest Tools pack. Note that Spice currently only has an XPDM driver which means that on Windows 7 one must follow the procedure below to use the Spice driver (from Spice's WinQXL page):

  • In a console with elevated privileges, run:
bcdedit.exe -set loadoptions DDISABLE_INTEGRITY_CHECKS
bcdedit.exe -set TESTSIGNING ON
  • Mount spice-guest-tools-0.3.iso
  • Then run spice-guest-tools-0.3.exe from the CD to install the driver.
  • Windows will warn about each driver because they are unsigned but will still use any you install.

Also on Windows Vista Spice's Windows Guest Tools pack (0.3) does not install the QXL driver. It is possible to work around that by manually installing the driver directly from c:\Program Files\SPICE Guest Tools\drivers. One can either pick the Windows XP or Windows 7 one, they seem to behave identically. However the problem is that this immediately shows a very visible transparency/alpha blending issue in the Desktop Gadgets area (see (bug 61124). That does not seem to impact the tests though that configuration has a bunch of d3d8:device test failures like on Windows XP.

VMware graphics driver

QEMU can also emulate the paravirtualized VMware graphics card. However so far the VMware graphics driver has not shown to reduce the number failing tests or to improve the performance. Thus it is not used for now. Still here is how to install the driver on Windows 7 for further testing.

  • These instructions are based on the I roughly followed 1 to comment in a discussion thread on Linux KVM.
  • First go to VMware's site to download a VMware Tools ISO.
  • Do not pick the latest VMware version because it only works with the VMware 7 hardware which QEMU 1.1.2 does not implement. The driver would install but then fail to load with error code 43 when it does not recognize the graphics card.
  • Pick the ISO from the esx/4.1latest/windows directory which corresponds to VMware Tools 8.3.17.
  • Mount the ISO and run 'setup.exe /a' and extract the drivers to c:\vmwaretools.
  • Then right-click on the desktop background, and follow this sequence of buttons Screen Resolution -> Advanced Settings -> Properties -> Driver -> Update Driver -> Browse my computer.
  • Pick the 'C:\vmwaretools\Common\VMware\Drivers\video' directory (or Common64 for the 64-bit drivers). This will install the 'VMware SVGA II' driver. This is in fact the legacy XPDM driver but it will still work in Windows 7.
  • Do not pick the 'wddm_video' directory! It contains the same 'VMware SVGA 3D' that requires the VMware 7 hardware that is not supported by QEMU.
  • Note that this procedure will not work in Windows 8 because it completely dropped support for the XPDM drivers. This means it is impossible to use the VMware drivers with QEMU on Windows 8.

Network card

The paravirtualized 'Virtio' network card causes errors (see Bug 1119281). Thus the VMs use either the e1000 or rtl8139 network cards with the standard Windows drivers.

Windows configuration

  • Start Internet Explorer and answer the initial configuration questions.
Rationale: It is not known if leaving these questions unanswered would modify the test results. However this would no correspond to any configuration that Windows applications would expect to run in so in doubt it's best to answer them with something as close as possible to 'default' answers.
  • Disable the Internet Explorer automatic updates
Starting with version 10 Internet Explorer has its own independent update mechanism. The rationale for disabling it is the same as for Windows Update.
  • Disable screensaver.
Rationale: The VMs can remain idle for a long time so that the screensaver would possibly be running before before we start the tests. The screensaver could also trigger during the long WineTest.exe runs. In either case it seems better nor to risk having them interfere. Also we don't want half a dozen VMs to waste time on animated screensavers while waiting for a test to run.
  • Disable disk and computer suspend.
Rationale: The VMs may remain idle for quite some time waiting for a test to run. We would not want them to 'suspend to RAM' as it's not clear they would successfully wake up when called upon. Similarly suspending the disk seems pointless and just asking for trouble.
  • Disable Windows update.
Rationale: After each test the VM is restored to a clean snapshot. So any work done by the Windows update process would be lost at that point. So the only effect it could have is messing the test environment and wasting computing resources.
  • Disable Delivery Optimization.
Rationale: It seems like this service can use network bandwidth to exchange windows update data with the other Windows machines on the LAN even when Windows update is disabled.
  • Disable restore points.
Rationale: Our MSI tests normally disable restore points so should not need this. However restore points are redundant with the VM snapshots and just waste disk space.
  • Disable search indexing.
Rationale: Any indexing will have to be redone when the VM is reverted to the clean snapshot after the test. So all it can do is trash the disk, likely causing tests to time out. Indexing is disabled by disabling the corresponding service.
  • Disable Windows Defender
Rationale: When restoring an old snapshot it starts a full disk scan which slows the tests.
Windows 10: Set [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender] DisableAntiSpyware=DWORD:1
  • Disable Windows Telemetry and the Customer Experience Improvement Program
One may want to avoid KB3068708, KB3075249, KB3080149. Also, in the Task Scheduler, disable the tasks in Microsoft > Windows > Application Experience and Customer Experience Improvement Program.
Rationale: They involve tasks that get run on boot and cause a significant amount of disk activity.
  • Hide the Windows 10 upgrade notifications
Rationale: Avoid windows popping up during the user32:msg tests.
  • Activate Windows!
Preferably once the hardware configuration is pretty much settled down to avoid having to reactivate again.

The following configuration configuration options are not really important for the WineTestBot VMs but can help optimize real Windows partitions used for running WineTest.

  • Configure auto-login
Rationale: This is necessary to get the tests to start automatically on boot.
Windows 10: control userpasswords2
  • Disable hibernation
Rationale: This helps save a lot of space when taking the image of the Windows partition.
Windows 10: powercfg -h off
  • Disable swap
Rationale: This helps save a lot of space when taking the image of the Windows partition.


See also VMware's recommendations to Optimize Windows Guest Operating System Performance.

Missing Windows dlls

Even with the latest Windows version and all updates some of the dlls Wine tests are missing. This section describes the recommended way to install them.

atl80

Install the x86/x64 Visual C++ 2005 SP1 redistributable. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

Note that atl80.dll is not found in c:\Windows\System32. Instead look for this set of dlls in c:\Windows\WinSxS\*.vc80.*.

atl90

This one is not tested yet, but for completeness it's best to install the x86/x64 Visual C++ 2008 SP1 redistributable. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

Note that atl90.dll is not found in c:\Windows\System32. Instead look for this set of dlls in c:\Windows\WinSxS\*.vc90.*.

atl100, msvcp100, msvcr100

Install the x86/x64 Visual C++ 2010 SP1 redistributable. Note that it only supports Windows XP or greater. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

atl110, msvcp110, msvcr110

Install the x86 and x64 Visual C++ 2012 Update 4 redistributable. Note that it only supports Windows XP or greater. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

msvcp120, msvcr120

Install the x86 and x64 Visual C++ 2013 redistributable. Note that it only supports Windows XP or greater. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

msvcp140, msvcr140, ucrtbase

Install the x86 and x64 Visual C++ 2015 redistributable. Note that it only supports Windows XP SP3 or greater. You may want to run Windows Update afterwards to get the latest fixes for this set of dlls.

d3d8

It may look like this file is missing when running the 64-bit tests on a 64-bit version of Windows. This is because Direct3D 8 is deprecated and no 64-bit support for it is provided by Windows.

d3dcompiler_43, d3dx9_36, xinput1_3

To get these dlls it is necessary to install the optional Direct X components, even on the latest Windows systems.

d3dxof

If these are missing on a 32-bit system install the optional Direct X components as documented above.

It may look like this file is missing when running the 64-bit tests on a 64-bit version of Windows. FIXME: Why? Is it obsolete like Direct3D 8?

dmband, dmime, dplayx

If these are missing on a 32-bit system install the optional Direct X components as documented above.

It may look like these files are missing when running the 64-bit tests on a 64-bit version of Windows. This is because DirectMusic and DirectPlay are deprecated and no 64-bit support for them is provided by Windows.

localspl, localui, spoolss

FIXME

msxml4

MSXML4 SP3 is available here. You may also want to apply the KB973685 update. Note that there is no 64-bit version.

msxml6

Wine implements the msxml6 dll but has no test for it yet. Still one can get MSXML6 SP1 here.

qmgr

FIXME

services.exe

It may look like this file is missing when running the 32-bit tests on a 64-bit Windows system. This is because 64-bit Windows only has the 64-bit version of this file.

xaudio2_7, xaudio2_8

This is the lastest audio API for Windows 7+. It may be installed through this download.

Unix configuration

  • Use the fvwm window manager
Rationale: Most window managers are buggy, in particular the GNOME and KDE ones, and cause some tests to fail or even crash. fvwm is the one known 'good' windows manager.
  • Use a Windows-compatible fvwm configuration
Rationale: Some tests verify that Windows applications get the right sequence of messages when windows lose/gain focus, etc. The sequence of messages depends in part on the window manager and cannot be right if the Unix window manager does not behave like Windows does, particularly wrt. focus-follows-mouse, transient window decorations, etc.
Style * ClickToFocus
Style * DontStackTransientParent
Style * DecorateTransient
  • Install ttf-mscorefonts-installer (non-free) or fonts-liberation
Rationale: No Windows system would ship without a base set of fonts of various families (Serif, Sans Serif). Since the lack of such fonts is not a valid Windows configuration, it is reasonable for tests (e.g. gdiplus:font) to depend on them and fail if they are missing. Furthermore ttf-mscorefonts-installer is preferred because it provides Arial which some tests need (but they skip if Arial is missing).
This page was last edited on 15 October 2018, at 15:20.