[Erlang Systems]

1 Test Server Basics

1.1 Introduction

Testing when using the test server is done by running test suites. A test suite is a number of test cases, where each test case tests one or more things. The test case is the smallest unit that the test server deals with. One or more test cases are grouped together into one ordinary Erlang module, which is called a test suite. Several test suite modules can be grouped together in special test specification files representing whole application and/or system test "jobs".

The test suite Erlang module must follow a certain interface, which is specified by the test server. See the section on writing test suites for details about this.

Each test case is considered a success if it returns to the caller, no matter what the returned value is. An exception to this is the return value {skip, Reason} which indicates that the test case is skipped. A failure is specified as a crash, no matter what the crash reason is.

As a test suite runs, all information (including output to stdout) is recorded in several different log files. A minimum of information is displayed to the user console. This only include start and stop information, plus a note for each failed test case.

The result from each test case is recorded in an HTML log file which is created for each test run. Every test case gets one row in a table presenting total time, whether the case was successful or not, if it was skipped, and possibly also a comment. The HTML file has links to each test case's logfile, wich may be viewed from e.g. Netscape or any other HTML capable browser.

The Test Server consists of three parts:

1.2 Definition of terms

conf(iguration) case
This is a group of test cases which need some specific configuration. A conf case contains an initiation function which sets up a specific configuration, one or more test cases using this configuration, and a cleanup function which restores the configuration. A conf case is specified in a test specification like this: {conf,InitFunc,ListOfCases,CleanupFunc}
datadir
Data directory for a test suite. This directory contains any files used by the test suite, e.g. additional erlang modules, c code or data files. If the data directory contains code which must be compiled before the test suite is run, it should also contain a makefile source called Makefile.src defining how to compile.
documentation clause
One of the function clauses in a test case. This clause shall return a list of strings describing what the test case tests.
execution clause
One of the function clauses in a test case. This clause implements the actual test case, i.e. calls the functions that shall be tested and checks results. The clause shall crash if it fails.
major log file
This is the test suites log file.
Makefile.src
This file is used by the test server framework to generate a makefile for a datadir. It contains some special characters which are replaced according to the platform currently tested.
minor log file
This is a separate log file for each test case.
privdir
Private directory for a test suite. This directory should be used when the test suite needs to write to files.
skip case
A test case which shall be skipped.
specification clause
One of the function clauses in a test case. This clause shall return an empty list, a test specification or {skip,Reason}. If an empty list is returned, it means that the test case shall be executed, and so it must also have an execution clause. Note that the specification clause is always executed on the controller node, i.e. not on the target node.
test case
A single test included in a test suite. Typically it tests one function in a module or application. A test case is implemented as a function in a test suite module. The function can have three clauses, the documentation-, specification- and execution clause.
test specification
A specification of which test suites and test cases to run. There can be test specifications on three different levels in a test. The top level is a test specification file which roughly specifies what to test for a whole application. Then there is a test specification for each test suite returned from the all(suite) function in the suite. And there can also be a test specification returned from the specification clause of a test case.
test specification file
This is a text file containing the test specification for an application. The file has the extension ".spec" or ".spec.Platform", where Platform is e.g. "vxworks" or "ose".
test suite
An erlang module containing a collection of test cases for a specific application or module.
topcase
The first "command" in a test specification file. This command contains the test specification, like this: {topcase,TestSpecification}

Copyright © 1991-2002 Ericsson Utvecklings AB