4 Writing Test Suites
4.1 Support for test suite authors
The
test_server
module provides som very nice function to support the test suite author. This include
- Starting and stopping slave or peer nodes
- Capturing and checking stdout output
- Retrieving and flushing process message queue
- Watchdog timers
- Checking that a function crashes
- Checking that a function succeds at least m out of n times
- Checking .app files
Please turn to the reference manual for the
test_server
module for details about these functions.4.2 Test suites
A test suite is an ordinary Erlang module that contains test cases. It's recommended that the module has a name on the form *_SUITE.erl. Otherwise, the directory function will not find the modules (by default).
For some of the test server support, the test server include file
test_server.hrl
must be included. Never include it with the full path, since that path is not valid when compiling on non-clearcase platforms. Use the compiler include directive instead.The special function
all(suite)
in each module is called to get the test specification for that module. The function typically returns a list of test cases in that module, but any test specification could be returned. Please turn to the chapter about test specifications for details about this.4.3 Init per test case
In each test suite module, the functions
init_per_testcase/2
andfin_per_testcase/2
must be implemented.
init_per_testcase
is called before each test case in the test suite, giving a (limited) possibility for initialization.
fin_per_testcase/2
is called after each test case is completed, giving a possibility to clean up.The first argument to these functions is the name of the test case. This can be used to do individual initiation and cleanup for each test cases.
The second argument is called
Config
.init_per_testcase/2
may modify this parameter or just return it as is. Whatever is retuned byinit_per_testcase/2
is given asConfig
parameter to the test case itself.The return value of
fin_per_testcase/2
is ignored by the test server.4.4 Test cases
The smallest unit that the test server is concerned with is a test case. Each test case can in turn test many things, for example make several calls to the same interface function with different parameters.
It is possible to put many or few tests into each test case. How many things each test case tests is up to the author, but here are some things to keep in mind.
Very small test cases often leads to more code, since initialization has to be duplicated. Larger code, especially with a lot of duplication, increases maintenance and reduces readability.
Larger test cases make it harder to tell what went wrong if it fails, and force us to skip larger portions of test code if a specific part fails. These effects are accentuated when running on multiple platforms because test cases often have to be skipped.
A test case generally consists of three parts, the documentation part, the specification part and the execution part. These are implemented as three clauses of the same function.
The documentation clause matches the argument '
doc
' and returns a list for strings describing what the test case tests.The specification clause matches the argument '
suite
' and returns the test specification for this particular test case. If the test specification is an empty list, this indicates that the test case is a leaf test case, i.e. one to be executed.Note that the specification clause of a test case is executed on the test server controller host. This means that if target is remote, the specification clause is probably executed on a different platform than the one tested.
The execution clause implements the actual test case. It takes one argument,
Config
, which contain configuration information likedata_dir
andpriv_dir
. See Data and Private Directories for more information about these.The
Config
variable can also contain thenodenames
key, if requested by therequire_nodenames
command in the test suite specification file. AllConfig
items should be extracted using the?config
macro. This is to ensure future compability if theConfig
format changes. See the reference manual fortest_server
for details about this macro.If the execution clause crashes or exits, it is considered a failure. An execution clause that returns to the caller, no matter the returned value, is considered a success. An exception to this is the return value
{skip,Reason}
. If this is returned, the test case is considered skipped.4.5 Data and Private Directories
The data directory (
data_dir
) is the directory where the test module has its own files needed for the testing. A compiler test case may have source files to feed into the compiler, a release upgrade test case may have some old and new release of something. A graphics test case may have some icons and a test case doing a lot of math with bignums might store the correct answers there. The name of thedata_dir
is the the name of the test suite and then "_data". For example,"some_path/foo_SUITE.beam"
has the data directory"some_path/foo_SUITE_data/"
.When using the test server framework
ts
, automatic compilation of code in the data directory can be obtained by placing a makefile source called Makefile.src in the data directory. Makefile.src will be converted to a valid makefile byts
when the test suite is run. See the reference manual for thets
module for details about the syntax of Makefile.src.The
priv_dir
is the test suite's private directory. This directory should be used when a test case needs to write to files. The name of the private directory is generated by the test server, which also creates the directory.Warning: Do not depend on current directory to be writable, or to point to anything in particular. All scratch files are to be written in the
priv_dir
, and all data files found indata_dir
. If the current directory has to be something specific, it must be set withfile:set_cwd/1
.4.6 Execution environment
Each time a test case is about to be executed, a new process is created with
spawn_link
. This is so that each test should be independent of earlier tests, with respect to process flags, process links, messages in the queue, servers having registered the process, etc. As little as possible is done to change that process' context, from what is created by plain spawn. Here is a list of differences:
- It has a link to the test server. If this link is removed, the test server will not know when the test case is finished, just wait infinitely.
- It often holds a few items in the process dictionary, all with names starting with '
test_server_
'. This is to keep track of if/where a test case fails.
- There is a top-level catch. All of the test case code is catched, so that the location of a crash can be reported back to the test server. If the test case process is killed by another process (thus the catch code is never executed) the test server is not able to tell where the test case was executing.
- It has a special group leader implemented by the test server. This way the test server is able to capture the io that the test case provokes. This is also used by some of the test server support functions.
There is no time limit for a test case, unless the test case itself imposes such a limit, by calling
test_server:timetrap/1
for example. The call can be made in each test case, or in theinit_per_testcase/2
function. Make sure to call the correspondingtest_server:timetrap_cancel/1
function as well, e.g in thefin_per_testcase/2
function, or else the test cases will always fail.4.7 Illegal dependencies
It is hard to write test suites that test much functionality, no implementation and is platform independent. Noted below are some of the more frequent dependency mistakes from our experience with running the OTP test suites.
- Depending on current directory, and writing there:
This is the most common error in test suites. The test suites think that current directory is what the author had as current directory when the test case was developed. Many test cases even try to write scratch files to this directory. If the current directory has to be set to something in particular, please usefile:set_cwd/1
to set it. And use thedata_dir
andpriv_dir
to locate data and scratch files.
- Depending on the clearcase paths and files:
When the test suites are run, they are not (always) run in a clearcase environment, but in the installed version of the product. Some paths are different or don't exist (such as the "lib/stdlib" path that becomes "lib/stdlib-1.3" and the "autoconf" directory which doesn't exist).
- Depending on running applications:
Some test suites rely on "their" application to be started before testing begins (I wonder how they test application startup). The only applications that are started from the beginning are the kernel, stdlib and sasl applications. All others have to be explicitely started, e.g. in the init function of a conf case..
- Depending on execution order:
There is no way of telling in which order the test cases are going to be run, so a test case can't depend on a server being started by a test case that runs "before". This has to be so for several reasons.
The user may specify the order at will, and maybe some particular order is better suited sometimes. Secondly, if the user just specifies a directory, the order of execution will depend on which order the files are listed in by the operating system, and that does vary among systems. Thirdly, if a user wishes to run only a subset of a test suite, there is no way one test case could successfully depend on another.
Execution order dependencies are allowed only within a con case, where you can depend on the initiation function to be called before all other test cases, and the cleanup function to be called at the end..
- Depending on the emulator:
Usecode:objfile_extension/0
instead of hardcoding ".beam". Functions that are emulator dependant should be tested in separate test cases.
- Depending on Unix:
Running unix commands through unix:cmd or os:cmd are likely not to work on non-unix platforms. Especially VxWorks has problems with these...
- Depending on NFS file system:
There are many different file systems that may be encountered when running test suites. Some operations are not supported or behave differently on different file systems. Too much dependencies in this area will surely break the test case on different platforms.
Here are the major file systems that need to be supported: NFS (Network file system), UFS (Unix file system) , MVFS (clearcase's file system), LOFS (loopback file system), TMPFS (ram based file system for /tmp/), NTFS (Windows NT), FAT (DOS partitions), FAT95 (DOS partitions on windows 95), VxWorks FAT (FAT on VxWorks).
Some things to keep in mind with theese file systems:
- All of these, except plain FAT, support long filenames. That's ok to depend on.
- The order in which files are sorted in the directories is undefined, on most of these. Don't depend on that.
- Open files may not be deleted or moved on many of these. Don't depend on that.
- Directories may not be overwritten by files and vice versa on many of these. Don't depend on that.
- Seeking before the start of the file is not allowed in MVFS, don't depend on this if you wish to run in clearcase.
- Named pipes are not supported by MVFS and FAT, try to place them in /tmp/ or in the private directory.
- Depending on other test cases:
Invoking a test case from another not only tests the same thing twice, but also makes it harder to follow what exactly is being tested. Also, if the called test case fails for some reason, so will the caller. This way one error gives cause to several error reports, which is less than ideal.
- Failure to crash or exit when things go wrong:
Making requests without checking that the return value indicates success may be ok if the test case will fail at a later stage, but it is never acceptable just to print an error message (into the log file), and return ok. Such test cases are worse than nothing since they create a false sense of security.
- Messing up for following test cases:
Test cases should restore as much of the execution environment as possible, so that the following test cases will not crash because of execution order of the test cases. Close down nodes and processes started, and be sure to restore the Erlang path, even if the test case crashes.
One particularly nasty case I have seen is when the test case loads new (e.g. cover compiled) modules instead of the originals. If this has to be done (doubtful), it must be encapsulated in a conf case in the test specification, so that the test case is 100% certain that it restores the modules to the originals. There is no use testing if the test cases operate on something else than we deliver! Again, this is worse than nothing since it may give a false sense of security (or flag nonexistent problems).
4.8 OS execution environment
It is unfortunate, but not all of Erlang is configurable from within Erlang. This causes some specific problems with testing, where different test cases assume that the OS environment is the same as the author's during development.
In order to run the test suites, a few of these parameters have had to be fixed. Theoretically, all tests should be run in all variations of these modes, but in practice the test cases rely on many of these choices to stay the same. This means that, in reality, we only test the product in this particular mode.
The maybe hardest aspect of this is the OS path, which cannot be set from within Erlang. The best bet is maybe not to depend at all on the path, for example when starting port programs, but to specify the full path instead.
An easier problem is the Erlang path, since it can be manipulated from within Erlang. Typically it is set from the Erlang command line just the same.
In the choice between non-distributed, distributed with short names and distributed with long names, short names were chosen. This seems to satisfy most test suite requirements, but there could be test cases where this is inadequate.
The
start_sasl
boot script was chosen. Only servers that are started by this script can be assumed to be running.The current directory is another issue already debated elsewhere. Test cases may not assume anything about where the current directory is pointing. A test suite that wish to refer to the current directory must set it first.
Another problem arises when running in a clearcase view, and the test cases start slave nodes. These slave nodes must run the same version of Erlang (the one that is being tested) and also need access to the clearcase view.