Contents
1. Introduction
Writing cmd tests is a bit different than writing standard wine tests.
The tests are not written in C, but directly in the form of batch files, which are processed by wine cmd, then compared line by line to an associated expected result file.
Currently, we do the equivalent of
cmd /c test_builtins.cmd == test_builtins.cmd.exp
2. Adding tests
Find the relevant test section (--- Testing foo ---), or create it if necessary.
Enter the code to be processed in the .cmd file, and the expected result in the associated .exp file.
You might have to use some special keywords.
3. Dealing with missing or extra lines
If a test outputs a different number of lines in Wine than on Windows the actual output will get out of sync with the expected file, causing all the tests that follow to fail. To avoid that the test environment treats all lines that start with '---' as resynchronization points. That is, if it encounters a line starting with '---' in the output, it will read lines from the expected file until it finds the matching '---' line, and vice versa.
For this reason all section and subsection headers should start with three dashes in order to cut short the cascade of failures when this unexpectedly happens.
Also if you know that Wine is buggy and outputs an extra line (for instance), you can have it output a '---' line right after that test and match it with '@todo_wine@---'. This will mark the forced resynchronization as expected and the tests that follow will not be impacted. For instance:
Command |
Windows Output |
Wine Output |
Matched by |
buggy_command |
result |
result |
result |
|
|
error message |
@todo_wine@--- |
echo --- |
--- |
--- |
|
echo next test |
next test |
next test |
next test |
4. Keywords Handling
Both the .cmd and .exp files can contain some keywords, that are expanded or treated specially at runtime.
Only @space@ and @tab@ are usable in .cmd files.
4.1. Whitespace Characters
These keywords can be used in both the .cmd and .exp files:
Keyword |
substituted by |
Comments |
@space@ |
space character (' ') |
Should only be used for trailing spaces, where they can be difficult to spot |
@tab@ |
tab character ('\t') |
Should always be used instead of regular tab characters, for improved visibility |
4.2. Environment
The following keywords are relevant to the test environment in general:
Keyword |
substituted by |
Example |
@drive@ |
initial drive letter |
C: |
@pwd@ |
initial working directory |
C:\Documents and Settings\wine\ |
@path@ |
initial path |
\Documents and Settings\wine\ |
@shortpath@ |
initial path with short names |
\DOCUME~1\wine\ |
4.3. Test Helpers
The following keywords are not really substituted, but act as markers/indicators
Keyword |
Usage |
@todo_wine@ |
Used at the start of an expected line to indicate a current failure on wine, so the test suite can run successfully. |
@or_broken@ |
Used to specify alternate/broken values returned by different windows versions (e.g. when a specific feature isn't available, etc.) |
See the testing section of the Wine Developer's Guide for more information
4.4. Usage Example
@todo_wine@expected val@space@@or_broken@foo @tab@bar@or_broken@baz
specifies
- a failure on wine (@todo_wine@)
- normal value is 'expected val '
- other windows versions can return 'foo \tbar'
- still other windows versions can return 'baz'
5. Remarks / Tips & Tricks
5.1. Error Messages
Don't try to match error messages: they are language/locale specific and won't work on all (multilingual) windows versions.
Ex: don't use
echo foo_expr -> foo_val@or_broken@Echo is OFF
but use
echo 'foo_expr' -> 'foo_val'@or_broken@''
instead
5.2. Temporary Files
If you need temporary batch/data files, don't add them to git, but create them at runtime, e.g. using echo foo>>tmpFile, and delete them afterwards.
The best way is generally to use names like foo, bar or baz, or even better a mkdir foobar & cd foobar at the start and cd .. & rd /s/q foobar at the end of your test section, if appropriate.
5.3. Processed File
The result file (processed .cmd file) and error file are deleted after usage. If you want to check their contents (e.g. for debugging tests), you can temporarily comment/delete the lines
DeleteFileA("test.out");
DeleteFileA("test.err");
in programs/cmd/tests/batch.c