2007 2011 Ericsson AB, All Rights Reserved The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/. Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Initial Developer of the Original Code is Ericsson AB. The OTP Test Server Framework Mattias Nilsson ts.xml
ts Test Server Framework for testing OTP

This is a framework for testing OTP. The ts module implements the interface to all the functionality in the framework.

The framework is built on top of the Test Server Controller, test_server_ctrl, and provides a high level operator interface. The main features added by the framework are:

Automatic compilation of test suites and data directories Collection of files in central directories and creation of additional HTML pages for better overview. Single command interface for running all available tests Spawns a new node with correct parameters before starting the test server Atomatically creates the parameter file needed when running tests on remote target

More information about the Test Server Framework and how to run test cases can be found in the Test Server User's Guide.

For writing you own test server framework, please turn to the reference manual for the Test Server Controller and chapter named "Writing your own test server framework" in the Test Server User's Guide.

SETUP

To be able to run ts, you must first `install' ts for the current environment. This is done by calling ts:install/0/1/2. A file called `variables' is created and used by ts when running test suites. It is not recommended to edit this file, but it is possible to alter if ts gets the wrong idea about your environment.

ts:install/0 is used if the target platform is the same as the controller host, i.e. if you run on "local target" and no options are needed. Then running ts:install/0 ts will run an autoconf script for your current environment and set up the necessary variables needed by the test suites.

ts:install/1 or ts:install/2 is used if the target platform is different from the controller host, i.e. if you run on "remote target" or if special options are required for your system.

See the reference manual for detailed information about ts:install/0/1/2.

Some of the common variables in the 'variables' file are described below. Do not make any assumptions as of what is found in this file, as it may change at any time.

longnames

Set to true if the system is using fully qualified nodenames. platform_id

This is the currently installed platform identification string. platform_filename

This is the name used to create the final save directory for test runs. platform_label

This is the string presented in the generated test results index page. rsh_name

This is the rsh program to use when starting slave or peer nodes on a remote host. erl_flags

Compile time flags used when compiling test suites. erl_release

The Erlang/OTP release being tested. 'EMULATOR'

The emulator being tested (e.g. beam) 'CPU'

The CPU in the machine running the tests, e.g. sparc. target_host

The target host name os

The target operating system, e.g. solaris2.8 target

The current target platform, e.g. sparc-sun-solaris2.8

RUNNING TESTS

After installing ts, you can run your test with the ts:run/0/1/2/3/4 functions. These functions, however, require a special directory structure to be able to find your test suites. Both the test server and all tests must be located under your $TESTROOT directory. The test server implementation shall be located in the directory $TESTROOT/test_server and for each application there must be a directory named _test]]> containing the .spec file and all test suites and data directories for the application. Note that there shall only be one .spec file for each application.

$TESTROOT/test_server must be the current directory when calling the ts:run/* function.

All available tests can be found with ts:tests(). This will list all applications for which a test specification file _test/.spec]]> can be found.

To run all these tests, use ts:run().

To run one or some of the tests, use ts:run(Tests), where Tests is the name of the application you want to test, or a list of such names.

To run one test suite within a test, use ts:run(Test,Suite).

To run one test case within a suite, use ts:run(Test,Suite,Case)

To all these functions, you can also add a list of options. Please turn to the reference manual for the ts module to see the valid options to use.

The function ts:help() displays some simple help for the functions in ts. Use this for quick reference.

LOG FILES

As the execution of the test suites go on, events are logged in four different ways:

Text to the operator's console. Suite related information is sent to the major log file. Case related information is sent to the minor log file. The HTML log file gets updated with test results.

Typically the operator, who may run hundreds or thousands of test cases, doesn't want to fill the screen with details about/from the specific test cases. By default, the operator will only see:

A confirmation that the test has started. A small note about each failed test case. A summary of all the run test cases. A confirmation that the test run is complete Some special information like error reports and progress reports, printouts written with erlang:display/1 or io:format/3 specifically addressed to somewhere other than standard_io.

This is enough for the operator to know, and if he wants to dig in deeper into a specific test case result, he can do so by following the links in the HTML presentation to take a look in the major or minor log files.

A detailed report of the entire test suite is stored in the major logfile, the exact reason for failure, time spent etc.

The HTML log file is a summary of the major log file, but gives a much better overview of the test run. It also has links to every test case's log file for quick viewing with a HTML browser.

The minor log file contain full details of every single test case, each one in a separate file. This way the files should be easy to compare with previous test runs, even if the set of test cases change.

Which information that goes where is user configurable via the test server controller. Three threshold values determine what comes out on screen, and in the major or minor log files. The contents that goes to the HTML log file is fixed, and cannot be altered.

install() -> ok | {error, Reason} install(TargetSystem) -> ok | {error, Reason} install(Opts) -> ok | {error, Reason} install(TargetSystem,Opts) -> ok | {error, Reason} Installs the Test Server Framework TargetSystem = {Architecture, TargetHost} Architecture = atom() or string() e.g. "ose" or "vxworks_ppc603" TargetHost = atom() or string() The name of the target host Opts = list()

Installs and configures the Test Server Framework for running test suites. If a remote host is to be used, the TargetSystem argument must be given so that "cross installation" can be done. Installation is required for any of the functions in ts to work.

Opts may be one or more of

{longnames, Bool}

Use fully qualified hostnames for test_server and slave nodes. Bool is true or false (default). {verbose, Level}

Verbosity level for test server output, set to 0, 1 or 2, where 0 is quiet(default). {hosts, Hosts}

This is a list of available hosts on which to start slave nodes. It is used when the {remote, true} option is given to the test_server:start_node/3 function. Also, if {require_nodenames, Num} is contained in a test specification file, the generated nodenames will be spread over all hosts given in this Hosts list. The hostnames are given as atoms or strings. {slavetargets, SlaveTarges}

For VxWorks only. This is a list of available hosts where slave nodes can be started. This is necessary because only one node can run per host in the VxWorks environment. This is not the same as {hosts, Hosts} because it is used for all slave nodes - not only the ones started with {remote, true}. The hostnames are given as atoms or strings. {crossroot, TargetErlRoot}

Erlang root directory on target host

This option is mandatory for remote targets {master, {MasterHost, MasterCookie}}

If target is remote and the target node is started as a slave node, this option indicates which master and cookie to use. The given master will also be used as master for slave nodes started with test_server:start_node/3. It is expected that the erl_boot_server is started on the master node before the test is run. If this option is not given, the test server controller node is used as master and the erl_boot_server is automatically started. {erl_start_args, ArgString}

Additional arguments to be used when starting the test server controller node. ArgString will be appended to the command line when starting the erlang node. Note that this will only affect the startup of the controller node, i.e. not the target node or any slave nodes startet from a test case. {ipv6_hosts, HostList}

This option will be inserted in the Config parameter for each test case. HostList is a list of hosts supporting IPv6. help() -> ok Presents simple help on the functions in ts

Presents simple help on the functions in ts. Useful for quick reference.

 tests() -> Tests Returns the list of available tests

Returns the list of available tests. This is actually just a list of all test specification files found by looking up "../*_test/*.spec".

In each ../Name_test/ directory there should be one test specification file named Name.spec.

 run() -> ok | {error, Reason} run([all_tests|Opts]) run(Specs) run(Specs, Opts) run(Spec, Module) run(Spec, Module, Opts) run(Spec, Module, Case) run(Spec, Module, Case, Opts) Runs (specified) test suite(s) Specs = Spec | [Spec] Spec = atom() Module = atom() Case = atom() Opts = [Opt] Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec} Level = integer(); 0 means silent Vars = list() of key-value tuples CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file. Reason = term()

This function runs test suite(s)/case(s). To be able to run any tests, ts:install must first be called to create the variables file needed. To run a whole test specification, only specify the name of the test specification, and all test suite modules belonging to that test spec will be run. To run a single module in a test specification, use the Module argument to specify the name of the module to run and all test cases in that module will be run, and to run a specified test case, specify the name of the test case using the Case argument. If called with no argument, all test specifications available will be run. Use ts:tests/0 to see the available test specifications.

If the batch option is not given, a new xterm is started (unix) when ts:run is called.

The verbose option sets the verbosity level for test server output. This has the same effect as if given to ts:install/1/2

The vars option can be used for adding configuration variables that are not in the variables file generated during installation. Can be any of the Opts valid for ts:install/1/2.

The keep_topcase option forces ts to keep the topcase in your test specification file as is. This option can only be used if you don't give the Module or Case parameters to ts:run. The keep_topcase option is necessary if your topcase contains anything other than _test"}]]>. If the option is not used, ts will modify your topcase.

The cover and cover_details options indicates that the test shall be run with code coverage analysis. cover_details means that analysis shall be done on the most detailed level. If the test is run with a remote target, this option creates a list of uncovered lines in each cover compiled module. If the test is run with a local target, each cover compiled module will be analysed with cover:analyse_to_file/1. The cover options will only create an overview of all cover compiled modules with the number of covered and not covered lines.

The CoverFile which can be given with the cover and cover_details options must be the filename of a file listing modules to be excluded from or included in the cover compilation. By default, ts believes that Spec is the name of an OTP application and that all modules in this application shall be cover compiled. The CoverFile can exclude modules that belong to the application and add modules that don't belong to the application. The file can have the following entries:

{exclude, all | ExcludeModuleList}. {include, IncludeModuleList}.

Note that each line must end with a full stop. ExcludeModuleList and IncludeModuleList are lists of atoms, where each atom is a module name.

If the cover or cover_details options are given on their own, the directory _test]]> is searched for a CoverFile named .cover]]>. If this file is not found, Spec is assumed to be the name of an OTP application, and all modules in the ebin directory for the application are cover compiled. The ebin directory is found by adding ebin to code:lib_dir(Spec).

The same cover compiled code will be loaded on all slave or peer nodes started with test_server:start_node/3. The exception is nodes that run an old version of OTP. If the loading fails, the node will simply not be a part of the coverage analysis. Note that slave and peer nodes must be stopped with test_server:stop_node/1 for the node to be part of the coverage analysis, else the test server will not be able to fetch coverage data from the node.

The trace option is used to turn on call trace on target and on slave or peer nodes started with test_server:start_node/3. TraceSpec can be the name of a trace information file, or a list of elements like the ones in a trace information file. Please turn to the reference manual for test_server_ctrl:trc/1 for details about the trace information file.

cross_cover_analyse(Level) -> ok cross_cover_analyse([Level]) -> ok Analyse cover data collected from all tests

Analyse cover data collected from all tests.

See test_server_ctrl:cross_cover_analyse/1

r() -> ok r(Opts) -> ok r(SpecOrSuite) -> ok r(SpecOrSuite,Opts) -> ok r(Suite,Case) -> ok r(Suite,Case,Opts) -> ok Run test suite or test case without tsinstalled SpecOrSuite = Spec | Suite Spec = string() "Name.spec" or "Name.spec.OsType", where OsType is vxworks Suite = atom() Case = atom() Opts = [Opt] Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile} Cover = cover | cover_details AppOrCoverFile = Application | CoverFile Application = atom() OTP application to cover compile CoverFile = string() Name of file listing modules to exclude from or include in cover compilation

This function can be used to run a test suites or test cases directly, without any of the additional features added by the test server framework. It is simply a wrapper function for the add_dir, add_spec, add_module and add_case functions in test_server_ctrl:

r() -> add_dir(".")

r(Spec) -> add_spec(Spec)

r(Suite) -> add_module(Suite)

r(Suite,Case) -> add_case(Suite,Case)

To use this function, it is required that the test suite is compiled and in the code path of the node where the function is called. The function can be used without having ts installed.

For information about the cover and cover_details options, see test_server_ctrl:cover/2/3.

 index() -> ok | {error, Reason} Updates local index page Reason = term()

This function updates the local index page. This can be useful if a previous test run was not completed and the index is incomplete.

 clean() -> ok clean(all) -> ok Cleans up the log directories created when running tests.

This function cleans up log directories created when running test cases. clean/0 cleans up all but the last run of each application. clean/1 cleans up all test runs found.

 estone() -> ok | {error, Reason} estone(Opts) -> ok Runs the EStone test

This function runs the EStone test. It is a shortcut for running the test suite estone_SUITE in the kernel application.

Opts is the same as the Opts argument for the ts:run functions.
Makfile.src in Data Directory

If a data directory contains code which must be compiled before the test suite is run, a makefile source called Makefile.src can be placed in the data directory. This file will be converted to a valid makefile by ts:run/0/1/2/3/4.

The reason for generating the makefile is that you can use variables from the variables file which was generated by ts:install/0/1/2. All occurrences of @Key@ in Makefile.src is substituted by the Value from {Key,Value} found in the variables file. Example:

Cut from variables:

... {'EMULATOR',"beam"}. {'CFLAGS',"-g -O2"}. {'LD',"$(CC) $(CFLAGS)"}. {'CC',"gcc"}. ...

Makefile.src for compiling erlang code could look something like this:

EFLAGS=+debug_info all: ordsets1.@EMULATOR@ ordsets1.@EMULATOR@: ordsets1.erl erlc $(EFLAGS) ordsets1.erl

Makefile.src for compiling c code could look something like this:

CC = @CC@ LD = @LD@ CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@ CROSSLDFLAGS = @CROSSLDFLAGS@ PROGS = nfs_check@exe@ all: $(PROGS) nfs_check@exe@: nfs_check@obj@ $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@ nfs_check@obj@: nfs_check.c $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c