Difference between revisions of "AppDB Test Results Guidelines"

From WineHQ Wiki
Jump to: navigation, search
Line 43: Line 43:
* '''Gallium Nine''' (D3D9 state tracker)
* '''Gallium Nine''' (D3D9 state tracker)
* '''xliveless'''
* '''xliveless'''
are ''not'' considered to be valid workarounds ''wrt'' to the WineHQ AppDB tests. ''These libraries are not directly supported by the WineHQ Project''. You are of course free to discuss and contrast the benefits of using any these libraries vs. using the stock Wine libraries. ''But these additional tests must be constrained to the '''Extra Comments''' section, of a Test Submission.''
are ''not'' considered to be valid workarounds ''wrt'' to WineHQ AppDB tests. ''These libraries are not directly supported by the WineHQ Project''. You are of course free to discuss and contrast the benefits of using any these libraries vs. using the stock Wine libraries. ''But these additional tests must be constrained to the '''Extra Comments''' section, of a Test Submission.''
''The overall rating, of a Windows game you are testing, should '''not''' be based on the use of any of these unofficial libraries''.
''The overall rating, of a Windows game you are testing, should '''not''' be based on the use of any of these unofficial libraries''.

Revision as of 13:54, 16 July 2019

The following guidelines are for submitting test results, including new versions and new applications, to the WineHQ Applications Database (AppDB). Failure to follow these guidelines may result in your test results being rejected and/or deleted.

1 Submitting Test Reports

1.1 Vanilla Wine Only

Do not submit test results for patched versions of Wine or third-party wrappers (such as PlayOnLinux or Wineskin). These are not a true reflection of what Wine can achieve and so do not help in identifying where Wine may need to be improved or fixed.

Tango-style info icon.svg In September, 2015, Wine-staging became an official branch of the Wine project, and test results are now accepted for it. Since the current version field does not differentiate between the development and staging branches, please note in the Extra Comments field if you have used wine-staging.

Distribution packages. Test results for distro Wine packages are accepted, even though they may include unapproved patches. The AppDB allows you to select your distribution when submitting a test result so that other users can see if a distribution package may actually cause a problem. If you are using a non-official repository, such as a Ubuntu PPA, include the name of it in the "Additional Comments" section of the report to help identify problems specific to that source.

1.2 Language

To be most useful to the maximum audience the preferred language for the AppDB test results is English. If you submit test results in another language, you must also provide an English translation in that test report.

The exception to this rule of course applies to proper nouns: if your application is called "Диверсанты" for example, then it will be accepted. It's usually a good idea to provide a Latinized version of the title too to make searching easier. Утопия (Pathologic) for example. If an application is given multiple names in different languages then please use the English name. Additional names can be added to keywords to help when searching.

Grammar: Capitalize the beginning of sentences and proper nouns, use punctuation, use newlines - don't crowd all the text together... basically make the text readable. Remember that the edit box is rich text if you need to use emphasis or highlighting.

1.3 Legal

Do not include copyrighted content from another source unless it is released under a GPL- or CC-style license such as with Wikipedia (remember to abide by the license by including the URL). This includes application descriptions, guides, reviews, and the like.

1.4 Formatting

Due to limitations of the AppDB system, avoid less than (<), greater than (>) symbols and <code> tags. They tend to get eaten during preview or suppressed. Use parenthesis instead of symbols.

1.5 Check the Notes

The maintainers should be keeping an eye on all the test results, bug reports and comments coming in to their app. When workarounds are found, a maintainer will add notes to the application. Read what these notes say. Sometimes extensive workarounds will get your application to work. A maintainer may simply reject your test results if you have not attempted to use these workarounds.

Remember to state what workarounds you have used in the comments section at the bottom of the test results submission form. This will show the maintainer that you have read the notes and that your tests are a true reflection of Wine.

1.6 Never Paste Console Logs

Console logs are for debugging. They look hideous in test results and do not belong there. If the console logs are relevant to a bug report, then attach them to the bug report.

A line or two of Wine messages is acceptable if it relates to a specific failure during testing. This helps others confirm if they are encountering the same problem and establishes the basis of a genuine bug in Wine instead of a fault with one tester's system.

1.7 Be Descriptive and Clear

"Everything I tested" may mean a lot to you when submitting a test result, but it means very little to other users and even less to someone trying to diagnose bugs. Be as descriptive as possible: list everything you tested. It doesn't matter if the list is pages long - it's better than saying very little. Tests should include printing (to cups-pdf at least), built-in help, and links or menu items that open external content such as a link that opens the default web browser.

Not saying enough is the main reason why test results are rejected. Nobody wants that to happen.

1.8 Include All Workarounds

Third-party, non-official Microsoft dynamic link libraries: e.g.

  • DXVK (D3D11/D3D10 - Vulkan translation layer)
  • D9VK (D3D11/D3D10/D3D9 - Vulkan translation layer)
  • Gallium Nine (D3D9 state tracker)
  • xliveless

are not considered to be valid workarounds wrt to WineHQ AppDB tests. These libraries are not directly supported by the WineHQ Project. You are of course free to discuss and contrast the benefits of using any these libraries vs. using the stock Wine libraries. But these additional tests must be constrained to the Extra Comments section, of a Test Submission.

The overall rating, of a Windows game you are testing, should not be based on the use of any of these unofficial libraries.

You will sometimes need to use workarounds to get an applications working, such as third-party components installed via winetricks.

Always include a list of the workarounds you have used in the comments section of your test result. This helps other users to get the application working. Do this even if there is a HOW TO listing the required workarounds. The HOW TO can be edited as Wine changes, but the test results shouldn't need to be. This helps developers spot regressions.

By definition an application cannot be Platinum if workarounds have been used. Marking your test results as Platinum just because it's your favorite or it works perfectly with workarounds gives a false representation of the capabilities of Wine and is not helpful in any way. This is a common complaint from Wine users. If an application is marked as Platinum then developers will take no notice of that application; there is no need to if it works perfectly.

If you are really determined to make it easy for others then attempt to find the minimum number of workarounds needed. As Wine improves the number of workarounds should decrease also.

Note that changing the Windows version reported to the program being tested (via winetricks or winecfg) is usually not considered a workaround if the program's specifications require a Windows version that is not Wine's default (currently Windows XP).

1.9 Select the Correct Wine Version

Always select the Wine version you are actually testing. This is obtained from via wine --version.

Dialog-warning.svg Never submit a test result for a Wine version that is not listed. If the version you are using is not on the list then it is no longer supported. Update your Wine version and submit a new test result.

Since at present the version field does not differentiate between the development and staging branches, please note in the Extra Comments field if you have used wine-staging.

If you are sticking with an old version of Wine because it works better then there is no need to keep submitting test results. If you have found a regression in a newer version then you should open a bug report to get it fixed. If the application is broken then submit a test result showing this.

Remember that new test results should show how well applications work in stable or new development releases of Wine, which is why they are the only versions listed when submitting new results.

1.10 Be Confident!

If you find that an application does not work at all for you yet it is rated gold for platinum, or works differently from all the other test reports, and you have followed all the notes and advice to get it to work correctly: be confident! Your test results should show how the application works for you. Remember that Wine is tested on a wide range of environments and it is inevitable that someone will see different results.

Remember to state in the test results what you did to get that far. Providing your system specs should also help determine why your experience is different, especially GPU/video card and driver for 3D applications.

2 Submitting New Applications

When submitting a new application make sure the entry will make sense to other users.


Enter the name of the application (adding an English or Latinized name in brackets will help when searching for the application).
Add useful keywords which will help the search engine such as an English or Latinized translation, or both, of the application name.
Choose the correct developer* of the application from the drop-down list or add it using the Developer name/URL fields. The plain "URL" field is for a link to the official product page of the application itself. Include the link protocol (http://…) in URLs.
If a significant third-party component is used, such as the Unreal 4 game engine or Apple QuickTime 5, then include it in the description since these often have their own bugs and workarounds (link to their relevant AppDB entries if they have one). A maintainer can trim a verbose description if necessary and move it into a separate note if too large for the category pages.

*Note that the developer is not necessarily the publisher, distributor, or seller. There may be multiple developers and they may change over time. For example Baldur's Gate may indicate Interplay or Black Isle Studios on the box but it was developed by BioWare. Baldur's Gate: Enhanced Edition was developed by Overhaul Games, distributed by Beamdog, and sold through GOG.com and Valve Software's Steam service. If there are multiple developers then use the most prominent one and include the complete list in the application description. The developer is desired because their technology may be reused across several programs so a bug and workaround for one may be applicable to others. The developer may be listed in the Help:About dialog, credits, or in files with names such as "readme", "license", "EULA", or "FAQ".


Enter the application version in the description - that is what the "Version Name" is for. This doesn't include sequels to an application series. For example: Bioshock 2 is a sequel to Bioshock, not just a new version of the original game.
Use the file name of the application as its name unless it is a command-line utility and doesn't have a separate name. If it is part of a well-defined collection of separate programs that don't have meaningful individual names then use the name of the collection and list each separate program individually in your test report.
Enter the same information into the keywords. It serves no useful purpose.
Submit entries for cracked or pirated software. Nobody in Wine is interested in illegal software. Using a NoCD is sometimes required to get a game with copy protection working. Make it clear in your test results that this is what you have done.
Submit a duplicate. If you find a problem with the name, multiple applications with same name, version, or assigned developer of a program then contact the maintainer or add a comment about it and submit your test data against the current "incorrect" entry.

3 Submitting New Versions

The Version Name is the letter and/or number of a program that identifies it from other releases or updates of the same program. It should be that of the main program, not included files and utilities (list the versions of significant secondary programs in test reports if they don't warrant their own AppDB entry). The version identifier may be found in the Help>About dialog, on the main menu of games, or in a file with a name like "readme", "changes", or "history".

The version name should not be too specific or too general. If the developer releases a new version every few days then it is unlikely that there will be many test results for any of them unless it is very popular. Applications developed with the "release early and often" policy usually only have minor changes between versions and thus minimal impact on Wine compatibility. Do submit separate versions for different architectures (16-bit, 32-bit, 64-bit), changes that significantly affect behavior (new installer), and distribution formats (CD/DVD, GOG.com, Steam) which affect installation and DRM. Do not submit separate versions for updates that only affect data, such as tax tables for tax software, unless they cause a significant change in behavior or an error that only occurs with Wine.

Do not use the version description field for test report summaries, HOW-TOs, reviews, or opinions.

Good examples:
Version Number Version Description
1.x All releases of the 1.x series. Specify exact version, GPU1, and driver when submitting test reports.
1.1+ Release 1.1 series or later using the new installer.
2013 2013 release (all data updates)
8R2 Service Pack 2
2.x (Steam) All Steam releases of the 2.x series. Specify exact version when submitting test reports.
1.x (GOG 32-bit) All 32-bit releases of the 1.x series from GOG.com. Specify exact version when submitting test reports.
Latest2 Specify version and sha1sum of game.exe when submitting test reports.
Unknown3 Specify date downloaded and sha1sum of app.exe when submitting test reports.

1Video card data is important for many 3D applications (especially games). Wine isn't going to make a GPU that doesn't meet an application's minimum requirements suddenly work. In addition, some failures can be misleading due to vague error messages caused by a GPU or driver problem.

2Latest is acceptable is an application automatically updates itself, and blocks older versions. Some financial and MMORPG games will block old version of software from accessing their server making them obsolete. Some Steam games do this which is why "Steam" is often accepted. "Latest" should only be used in this way and should never be used for software where multiple version are available.

3Unknown is acceptable for software that either does not provide version information, or is so badly broken that version information cannot be found. A maintainer can change this later if necessary. If the software is this badly broken you should be filing a bug report for it.

Bad examples
Version Number Version Description Latest version This will run 100% perfect, just launch it using Wine's "Wine Windows Program Loader", the graphics may lag a tiny bit; But, the Application runs. Doesn't work because of bug #1
2013-06-13 The version I downloaded today.

It can be difficult to choose the correct license since some applications have multiple releases with different licenses. For example, a retail product may have a time-limited trial mode (upgradeable with registration fee) and an unlimited free mode with a reduced feature set. In general, use the most restrictive license for the entry. If there are substantial differences between different releases, such as a freeware "beta" and a retail "full" release, then created different version entries for them.

The "Download" field is a URL to the page where a specific version can be downloaded. It does not have to be a direct link to a file (zip, exe, msi). This field is shown in the "Other apps affected" summary page for bug links, making it convenient for developers to download a specific version for analysis from a bug report.

4 Bugs

At some point you will find a bug and will need to log it in Bugzilla. Read Bugs thoroughly on details on how to do that and always attach the bug report.