3 Test Structure and Test Specifications
3.1 Test structure
A test consists of a set of test cases. Each test case is implemented as an erlang function. An erlang module implementing one or more test cases is called a test suite.
3.2 Test specifications
A test specification is a specification of which test suites and test cases to run and which to skip. A test specification can also group several test cases into conf cases with init and cleanup functions (see section about configuration cases below). In a test there can be test specifications on three different levels:
The top level is a test specification file which roughly specifies what to test for a whole application. The test specification in such a file is encapsulated in a topcase command.
Then there is a test specification for each test suite, specifying which test cases to run within the suite. The test specification for a test suite is returned from the
all(suite)
function in the test suite module.And finally there can be a test specification per test case, specifying sub test cases to run. The test specification for a test case is returned from the specification clause of the test case.
When a test starts, the total test specification is built in a tree fashion, starting from the top level test specification.
The following are the valid elements of a test specification. The specification can be one of these elements or a list with any combination of the elements:
{Mod, Case}
- This specifies the test case Mod:Case/1
{dir, Dir}
- This specifies all modules
*_SUITE
in the directoryDir
{dir, Dir, Pattern}
- This specifies all modules
Pattern*
in the directoryDir
{conf, Init, TestSpec, Fin}
- This is a configuration case. In a test specification file,
Init
andFin
must be{Mod,Func}
. Inside a module they can also be justFunc
. See the section named Configuration Cases below for more information about this.{make, Init, TestSpec, Fin}
- This is a special version of a conf case which is only used by the test server framwork
ts
.Init
andFin
are make and unmake functions for a data directory.TestSpec
is the test specification for the test suite owning the data directory in question. If the make function fails, all tests in the test suite are skipped. The difference between this "make case" and a normal conf case is that for the make case,Init
andFin
are given with arguments ({Mod,Func,Args}
), and that they are executed on the controller node (i.e. not on target).Case
- This can only be used inside a module, i.e. not a test specification file. It specifies the test case
CurrentModule:Case
.3.3 Test Specification Files
A test specification file is a text file containing the top level test specification (a topcase command), and possibly one or more additional commands. A "command" in a test specification file means a key-value tuple ended by a dot-newline sequence.
The following commands are valid:
{topcase, TestSpec}
- This command is mandatory in all test specification files.
TestSpec
is the top level test specification of a test.{skip, {Mod, Comment}}
- This specifies that all cases in the module
Mod
shall be skipped.Comment
is a string.{skip, {Mod, Case, Comment}}
- This specifies that the case
Mod:Case
shall be skipped.{skip, {Mod, CaseList, Comment}}
- This specifies that all cases
Mod:Case
, whereCase
is inCaseList
, shall be skipped.{nodes, Nodes}
Nodes
is a list of nodenames available to the test suite. It will be added to theConfig
argument to all test cases.Nodes
is a list of atoms.{require_nodenames, Num}
- Specifies how many nodenames the test suite will need. Theese will be automaticly generated and inserted into the
Config
argument to all test cases.Num
is an integer.{hosts, Hosts}
- This is a list of available hosts on which to start slave nodes. It is used when the
{remote, true}
option is given to thetest_server:start_node/3
function. Also, if{require_nodenames, Num}
is contained in a test specification file, the generated nodenames will be spread over all hosts given in thisHosts
list. The hostnames are atoms or strings.{diskless, true}
- Adds
{diskless, true}
to theConfig
argument to all test cases. This is kept for backwards compatiblilty and should not be used. Use a configuration case instead.All test specification files shall have the extension ".spec". If special test specification files are needed for the Windows, VxWorks or OSE/Delta platforms, additional files with the extension ".spec.win", ".spec.vxworks" and ".spec.ose" shall be used. This is useful e.g. if some test cases shall be skippped on these platforms.
Some examples for test specification files can be found in the Examples section of this user's guide.
3.4 Configuration cases
If a group of test cases need the same initiation, a so called configuration or conf case can be used. A conf case consists of an initiation function, the group of test cases needing this initiation and a cleanup or finalization function.
If the init function in a conf case fails or returns
{skip,Comment}
, the rest of the test cases in the conf case (including the cleanup function) are skipped. If the init function succeeds, the cleanup function will always be called, even if some of the test cases inbetween failed.Both the init function and the cleanup function in a conf case get the
Config
parameter as only argument. This parameter can be modified or returned as is. Whatever is returned by the init function is given asConfig
parameter to the rest of the test cases in the conf case, including the cleanup function.If the
Config
parameter is changed by the init function, it must be restored by the cleanup function. Whatever is returned by the cleanup function will be given to the next test case called.3.5 Skipping test cases
It is possible to skip certain test cases, for example if you know beforehand that a specific test case fails. This might be functionality which isn't yet implemented, a bug that is known but not yet fixed or some functionality which doesn't work or isn't applicable on a spesific platform.
There are several different ways to state that a test case should be skipped:
- Using the
{skip,What}
command in a test specification file
- Returning
{skip,Reason}
from theinit_per_testcase/2
function
- Returning
{skip,Reason}
from the specification clause of the test case
- Returning
{skip,Reason}
from the execution clause of the test case
The latter of course means that the execution clause is actually called, so the author must make sure that the test case is not run. For more information about the different clauses in a test case, see the chapter about writing test cases.
When a test case is skipped, it will be noted as
SKIPPED
in the HTML log.