5 Running Test Suites
5.1 Using the test server framework ts
ts
provides a high level operator inteface for testing of OTP. It adds the following features to the test server:
- Automatic compilation of test suites and data directories
- Collection of files in central directories and creation of additional HTML pages for better overview.
- Single command interface for running all available tests
- Spawns a new node with correct parameters before starting the test server
- Atomatically creates the parameter file needed when running tests on remote target
5.1.1 Setup
To be able to run
ts
, you must first `install'ts
for the current environment. This is done by callingts:install/0/1/2
. A file called `variables' is created and used byts
when running test suites. It is not recommended to edit this file, but it is possible to alter ifts
gets the wrong idea about your environment.
ts:install/0
is used if the target platform is the same as the controller host, i.e. if you run on "local target" and no options are needed. Then runningts:install/0
ts
will run an autoconf script for your current environment and set up the necessary variables needed by the test suites.
ts:install/1
orts:install/2
is used if the target platform is different from the controller host, i.e. if you run on "remote target" or if special options are required for your system. VxWorks and OSE/Delta are currently supported as remote target platforms.See the reference manual for detailed information about
ts:install/0/1/2
.Some of the common variables in the 'variables' file are discribed below. Do not make any assumptions as of what is found in this file, as it may change at any time.
longnames
- Set to true if the system is using fully qualified nodenames.
platform_id
- This is the currently installed platform identification string.
platform_filename
- This is the name used to create the final save directory for test runs.
platform_label
- This is the string presented in the generated test results index page.
rsh_name
- This is the rsh program to use when starting slave or peer nodes on a remote host.
erl_flags
- Compile time flags used when compiling test suites.
erl_release
- The Erlang/OTP release being tested.
'EMULATOR'
- The emulator being tested (e.g. beam)
'CPU'
- The CPU in the machine running the tests, e.g. sparc.
target_host
- The target host name
os
- The target operating system, e.g. solaris2.8
target
- The current target platform, e.g. sparc-sun-solaris2.8
5.1.2 Running tests
After installing
ts
, you can run your test with thets:run/0/1/2/3/4
functions. These functions, however, require a special directory structure to be able to find your test suites. Both the test server and all tests must be located under your $TESTROOT directory. The test server implementation shall be located in the directory$TESTROOT/test_server
and for each application there must be a directory named$TESTROOT/<application>_test
containing the .spec file and all test suites and data directories for the application. Note that there shall only be one .spec file for each application.
$TESTROOT/test_server
must be the current directory when calling thets:run/*
function.All available tests can be found with
ts:tests()
. This will list all applications for which a test specification file../<application>_test/<application>.spec
can be found.To run all these tests, use
ts:run()
.To run one or some of the tests, use
ts:run(Tests)
, whereTests
is the name of the application you want to test, or a list of such names.To run one test suite within a test, use
ts:run(Test,Suite)
.To run one test case within a suite, use
ts:run(Test,Suite,Case)
To all these functions, you can also add a list of options. Please turn to the reference manual for the
ts
module to see the valid options to use.The function
ts:help()
displays some simple help for the functions ints
. Use this for quick reference.5.2 Log files
As the execution of the test suites go on, events are logged in four different ways:
- Text to the operator's console.
- Suite related information is sent to the major log file.
- Case related information is sent to the minor log file.
- The HTML log file gets updated with test results.
Typically the operator, who may run hundreds or thousands of test cases, doesn't want to fill the screen with details about/from the specific test cases. By default, the operator will only see:
- A confirmation that the test has started.
- A small note about each failed test case.
- A summary of all the run test cases.
- A confirmation that the test run is complete
- Some special information like error reports and progress reports, printouts written with erlang:display/1 or io:format/3 specifically addressed to somewhere other than
standard_io
.
This is enough for the operator to know, and if he wants to dig in deeper into a specific test case result, he can do so by following the links in the HTML presentation to take a look in the major or minor log files.
A detailed report of the entire test suite is stored in the major logfile, the exact reason for failure, time spent etc.
The HTML log file is a summary of the major log file, but gives a much better overview of the test run. It also has links to every test case's log file for quick viewing with a HTML browser.
The minor log file contain full details of every single test case, each one in a separate file. This way the files should be easy to compare with previous test runs, even if the set of test cases change.
Which information that goes where is user configurable via the test server controller. Three threshold values determine what comes out on screen, and in the major or minor log files. The contents that goes to the HTML log file is fixed, and cannot be altered.
5.3 Using the test server controller
The test server controller provides a low level interface to all the test server functionality. It is possible to use this interface directly, but it is recommended to use the
ts
framework instead. Ifts
does not meet your needs, you could concider writing your own framework on top of test server controller. Some information about how to do this can be found in the section named "Writing you own test server framework" in this user's guide.If you still really would like to use the test server controller directly, please turn to the the reference manual to see all available functions. Also note that
ts:r/0/1/2
is a simple wrapper function fortest_server_ctrl:add_dir/1
,test_server_ctrl:add_module/1
andtest_server_ctrl:add_case/2
.ts:r/0/1/2
does not require an installation ofts
or any specific directory structure for your tests.