[Erlang Systems]

5 Running Test Suites

5.1 Using the test server framework ts

ts provides a high level operator inteface for testing of OTP. It adds the following features to the test server:

5.1.1 Setup

To be able to run ts, you must first `install' ts for the current environment. This is done by calling ts:install/0/1/2. A file called `variables' is created and used by ts when running test suites. It is not recommended to edit this file, but it is possible to alter if ts gets the wrong idea about your environment.

ts:install/0 is used if the target platform is the same as the controller host, i.e. if you run on "local target" and no options are needed. Then running ts:install/0 ts will run an autoconf script for your current environment and set up the necessary variables needed by the test suites.

ts:install/1 or ts:install/2 is used if the target platform is different from the controller host, i.e. if you run on "remote target" or if special options are required for your system. VxWorks and OSE/Delta are currently supported as remote target platforms.

See the reference manual for detailed information about ts:install/0/1/2.

Some of the common variables in the 'variables' file are discribed below. Do not make any assumptions as of what is found in this file, as it may change at any time.

longnames
Set to true if the system is using fully qualified nodenames.
platform_id
This is the currently installed platform identification string.
platform_filename
This is the name used to create the final save directory for test runs.
platform_label
This is the string presented in the generated test results index page.
rsh_name
This is the rsh program to use when starting slave or peer nodes on a remote host.
erl_flags
Compile time flags used when compiling test suites.
erl_release
The Erlang/OTP release being tested.
'EMULATOR'
The emulator being tested (e.g. beam)
'CPU'
The CPU in the machine running the tests, e.g. sparc.
target_host
The target host name
os
The target operating system, e.g. solaris2.8
target
The current target platform, e.g. sparc-sun-solaris2.8

5.1.2 Running tests

After installing ts, you can run your test with the ts:run/0/1/2/3/4 functions. These functions, however, require a special directory structure to be able to find your test suites. Both the test server and all tests must be located under your $TESTROOT directory. The test server implementation shall be located in the directory $TESTROOT/test_server and for each application there must be a directory named $TESTROOT/<application>_test containing the .spec file and all test suites and data directories for the application. Note that there shall only be one .spec file for each application.

$TESTROOT/test_server must be the current directory when calling the ts:run/* function.

All available tests can be found with ts:tests(). This will list all applications for which a test specification file ../<application>_test/<application>.spec can be found.

To run all these tests, use ts:run().

To run one or some of the tests, use ts:run(Tests), where Tests is the name of the application you want to test, or a list of such names.

To run one test suite within a test, use ts:run(Test,Suite).

To run one test case within a suite, use ts:run(Test,Suite,Case)

To all these functions, you can also add a list of options. Please turn to the reference manual for the ts module to see the valid options to use.

The function ts:help() displays some simple help for the functions in ts. Use this for quick reference.

5.2 Log files

As the execution of the test suites go on, events are logged in four different ways:

Typically the operator, who may run hundreds or thousands of test cases, doesn't want to fill the screen with details about/from the specific test cases. By default, the operator will only see:

This is enough for the operator to know, and if he wants to dig in deeper into a specific test case result, he can do so by following the links in the HTML presentation to take a look in the major or minor log files.

A detailed report of the entire test suite is stored in the major logfile, the exact reason for failure, time spent etc.

The HTML log file is a summary of the major log file, but gives a much better overview of the test run. It also has links to every test case's log file for quick viewing with a HTML browser.

The minor log file contain full details of every single test case, each one in a separate file. This way the files should be easy to compare with previous test runs, even if the set of test cases change.

Which information that goes where is user configurable via the test server controller. Three threshold values determine what comes out on screen, and in the major or minor log files. The contents that goes to the HTML log file is fixed, and cannot be altered.

5.3 Using the test server controller

The test server controller provides a low level interface to all the test server functionality. It is possible to use this interface directly, but it is recommended to use the ts framework instead. If ts does not meet your needs, you could concider writing your own framework on top of test server controller. Some information about how to do this can be found in the section named "Writing you own test server framework" in this user's guide.

If you still really would like to use the test server controller directly, please turn to the the reference manual to see all available functions. Also note that ts:r/0/1/2 is a simple wrapper function for test_server_ctrl:add_dir/1, test_server_ctrl:add_module/1 and test_server_ctrl:add_case/2. ts:r/0/1/2 does not require an installation of ts or any specific directory structure for your tests.


Copyright © 1991-2002 Ericsson Utvecklings AB