<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">
<erlref>
<header>
<copyright>
<year>2007</year>
<year>2008</year>
<holder>Ericsson AB, All Rights Reserved</holder>
</copyright>
<legalnotice>
The contents of this file are subject to the Erlang Public License,
Version 1.1, (the "License"); you may not use this file except in
compliance with the License. You should have received a copy of the
Erlang Public License along with this software. If not, it can be
retrieved online at http://www.erlang.org/.
Software distributed under the License is distributed on an "AS IS"
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
the License for the specific language governing rights and limitations
under the License.
The Initial Developer of the Original Code is Ericsson AB.
</legalnotice>
<title>The OTP Test Server Framework</title>
<prepared>Mattias Nilsson</prepared>
<responsible></responsible>
<docno></docno>
<approved></approved>
<checked></checked>
<date></date>
<rev></rev>
<file>ts.xml</file>
</header>
<module>ts</module>
<modulesummary>Test Server Framework for testing OTP</modulesummary>
<description>
<p>This is a framework for testing OTP. The <c>ts</c> module
implements the interface to all the functionality in the
framework.
</p>
<p>The framework is built on top of the Test Server Controller,
<c>test_server_ctrl</c>, and provides a high level operator
interface. The main features added by the framework are:
</p>
<list type="bulleted">
<item>Automatic compilation of test suites and data directories
</item>
<item>Collection of files in central directories and creation of
additional HTML pages for better overview.
</item>
<item>Single command interface for running all available tests
</item>
<item>Spawns a new node with correct parameters before starting
the test server
</item>
<item>Atomatically creates the parameter file needed when
running tests on remote target
</item>
</list>
<p>More information about the Test Server Framework and how to run
test cases can be found in the Test Server User's Guide.
</p>
<p>For writing you own test server framework, please turn to the
reference manual for the Test Server Controller and chapter named
"Writing your own test server framework" in the Test Server User's
Guide.
</p>
<p>SETUP</p>
<p>To be able to run <c>ts</c>, you must first `install'
<c>ts</c> for the current environment. This is done by calling
<c>ts:install/0/1/2</c>. A file called `variables' is created
and used by <c>ts</c> when running test suites. It is not
recommended to edit this file, but it is possible to alter if
<c>ts</c> gets the wrong idea about your environment.
</p>
<p><c>ts:install/0</c> is used if the target platform is the
same as the controller host, i.e. if you run on "local target"
and no options are needed. Then running <c>ts:install/0</c><c>ts</c>
will run an autoconf script for your current
environment and set up the necessary variables needed by the
test suites.
</p>
<p><c>ts:install/1</c> or <c>ts:install/2</c> is used if the
target platform is different from the controller host, i.e. if
you run on "remote target" or if special options are required
for your system. VxWorks is currently supported
as remote target platform.
</p>
<p>See the reference manual for detailed information about
<c>ts:install/0/1/2</c>.
</p>
<p>Some of the common variables in the 'variables' file are
described below. Do not make any assumptions as of what is found
in this file, as it may change at any time.
</p>
<list>
<item><c>longnames</c><br></br>
Set to true if the system is using fully qualified
nodenames.
</item>
<item><c>platform_id</c><br></br>
This is the currently installed platform identification
string.
</item>
<item><c>platform_filename</c><br></br>
This is the name used to create the final save directory
for test runs.
</item>
<item><c>platform_label</c><br></br>
This is the string presented in the generated test
results index page.
</item>
<item><c>rsh_name</c><br></br>
This is the rsh program to use when starting slave or
peer nodes on a remote host.
</item>
<item><c>erl_flags</c><br></br>
Compile time flags used when compiling test suites.
</item>
<item><c>erl_release</c><br></br>
The Erlang/OTP release being tested.
</item>
<item><c>'EMULATOR'</c><br></br>
The emulator being tested (e.g. beam)
</item>
<item><c>'CPU'</c><br></br>
The CPU in the machine running the tests, e.g. sparc.
</item>
<item><c>target_host</c><br></br>
The target host name
</item>
<item><c>os</c><br></br>
The target operating system, e.g. solaris2.8
</item>
<item><c>target</c><br></br>
The current target platform, e.g. sparc-sun-solaris2.8
</item>
</list>
<p>RUNNING TESTS</p>
<p>After installing <c>ts</c>, you can run your test with the
<c>ts:run/0/1/2/3/4</c> functions. These functions, however,
require a special directory structure to be able to find your
test suites. Both the test server and all tests must be located
under your $TESTROOT directory. The test server implementation
shall be located in the directory <c>$TESTROOT/test_server</c>
and for each application there must be a directory named
<c><![CDATA[$TESTROOT/<application>_test]]></c> containing the .spec file
and all test suites and data directories for the
application. Note that there shall only be one .spec file for
each application.
</p>
<p><c>$TESTROOT/test_server</c> must be the current directory
when calling the <c>ts:run/*</c> function.
</p>
<p>All available tests can be found with <c>ts:tests()</c>. This
will list all applications for which a test specification file
<c><![CDATA[../<application>_test/<application>.spec]]></c> can be found.
</p>
<p>To run all these tests, use <c>ts:run()</c>.
</p>
<p>To run one or some of the tests, use <c>ts:run(Tests)</c>,
where <c>Tests</c> is the name of the application you want to
test, or a list of such names.
</p>
<p>To run one test suite within a test, use
<c>ts:run(Test,Suite)</c>.
</p>
<p>To run one test case within a suite, use
<c>ts:run(Test,Suite,Case)</c></p>
<p>To all these functions, you can also add a list of
options. Please turn to the reference manual for the <c>ts</c>
module to see the valid options to use.
</p>
<p>The function <c>ts:help()</c> displays some simple help for
the functions in <c>ts</c>. Use this for quick reference.
</p>
<p>LOG FILES</p>
<p>As the execution of the test suites go on, events are logged in
four different ways:
</p>
<list type="bulleted">
<item>Text to the operator's console.</item>
<item>Suite related information is sent to the major log file.</item>
<item>Case related information is sent to the minor log file.</item>
<item>The HTML log file gets updated with test results.</item>
</list>
<p>Typically the operator, who may run hundreds or thousands of
test cases, doesn't want to fill the screen with details
about/from the specific test cases. By default, the operator will
only see:
</p>
<list type="bulleted">
<item>A confirmation that the test has started.
</item>
<item>A small note about each failed test case.
</item>
<item>A summary of all the run test cases.
</item>
<item>A confirmation that the test run is complete
</item>
<item>Some special information like error reports and progress
reports, printouts written with erlang:display/1 or io:format/3
specifically addressed to somewhere other than
<c>standard_io</c>.</item>
</list>
<p>This is enough for the operator to know, and if he wants to dig
in deeper into a specific test case result, he can do so by
following the links in the HTML presentation to take a look in the
major or minor log files.
</p>
<p>A detailed report of the entire test suite is stored in the
major logfile, the exact reason for failure, time spent etc.
</p>
<p>The HTML log file is a summary of the major log file, but gives
a much better overview of the test run. It also has links to every
test case's log file for quick viewing with a HTML browser.
</p>
<p>The minor log file contain full details of every single test
case, each one in a separate file. This way the files should be
easy to compare with previous test runs, even if the set of test
cases change.
</p>
<p>Which information that goes where is user configurable via the
test server controller. Three threshold values determine what
comes out on screen, and in the major or minor log files. The
contents that goes to the HTML log file is fixed, and cannot be
altered.
</p>
</description>
<funcs>
<func>
<name>install() -> ok | {error, Reason}</name>
<name>install(TargetSystem) -> ok | {error, Reason}</name>
<name>install(Opts) -> ok | {error, Reason}</name>
<name>install(TargetSystem,Opts) -> ok | {error, Reason}</name>
<fsummary>Installs the Test Server Framework</fsummary>
<type>
<v>TargetSystem = {Architecture, TargetHost}</v>
<v>Architecture = atom() or string()</v>
<d>e.g. "ose" or "vxworks_ppc603"</d>
<v>TargetHost = atom() or string()</v>
<d>The name of the target host</d>
<v>Opts = list()</v>
</type>
<desc>
<p>Installs and configures the Test Server Framework for
running test suites. If a remote host is to be used, the
<c>TargetSystem</c> argument must be given so that "cross
installation" can be done. This should be used for testing on
VxWorks. Installation is required for any of the
functions in <c>ts</c> to work.
</p>
<p>Opts may be one or more of
</p>
<list>
<item><c>{longnames, Bool}</c><br></br>
Use fully qualified hostnames for test_server and
slave nodes. Bool is <c>true</c> or <c>false</c> (default).
</item>
<item><c>{verbose, Level}</c><br></br>
Verbosity level for test server output, set to 0, 1 or
2, where 0 is quiet(default).
</item>
<item><c>{hosts, Hosts}</c><br></br>
This is a list of available hosts on which to start
slave nodes. It is used when the <c>{remote, true}</c>
option is given to the <c>test_server:start_node/3</c>
function. Also, if <c>{require_nodenames, Num}</c> is
contained in a test specification file, the generated
nodenames will be spread over all hosts given in this
<c>Hosts</c> list. The hostnames are given as atoms or
strings.
</item>
<item><c>{slavetargets, SlaveTarges}</c><br></br>
For VxWorks only. This is a list of
available hosts where slave nodes can be started. This is
necessary because only one node can run per host in the
VxWorks environment. This is not the same as
<c>{hosts, Hosts}</c> because it is used for all slave nodes
- not only the ones started with <c>{remote, true}</c>. The
hostnames are given as atoms or strings.
</item>
<item><c>{crossroot, TargetErlRoot}</c><br></br>
Erlang root directory on target host
<br></br>
This option is mandatory for remote targets
</item>
<item><c>{master, {MasterHost, MasterCookie}}</c><br></br>
If target is remote and the target node is started as
a slave node, this option
indicates which master and cookie to use. The given master
will also be used as master for slave nodes started with
<c>test_server:start_node/3</c>. It is expected that the
<c>erl_boot_server</c> is started on the master node before
the test is run. If this option is not given, the test
server controller node is used as master and the
<c>erl_boot_server</c> is automatically started.
</item>
<item><c>{erl_start_args, ArgString}</c><br></br>
Additional arguments to be used when starting the test
server controller node. <c>ArgString</c> will be appended to
the command line when starting the erlang node. Note that
this will only affect the startup of the <em>controller node</em>,
i.e. not the target node or any slave nodes
startet from a test case.
</item>
<item><c>{ipv6_hosts, HostList}</c><br></br>
This option will be inserted in the
<c>Config</c> parameter for each test case. <c>HostList</c>
is a list of hosts supporting IPv6.
</item>
</list>
</desc>
</func>
<func>
<name>help() -> ok</name>
<fsummary>Presents simple help on the functions in <c>ts</c></fsummary>
<desc>
<p>Presents simple help on the functions in <c>ts</c>. Useful
for quick reference.</p>
</desc>
</func>
<func>
<name>tests() -> Tests</name>
<fsummary>Returns the list of available tests</fsummary>
<desc>
<p>Returns the list of available tests. This is actually just
a list of all test specification files found by looking up
"../*_test/*.spec".
</p>
<p>In each ../Name_test/ directory there should be one test
specification file named Name.spec.</p>
</desc>
</func>
<func>
<name>run() -> ok | {error, Reason}</name>
<name>run([all_tests|Opts])</name>
<name>run(Specs)</name>
<name>run(Specs, Opts)</name>
<name>run(Spec, Module)</name>
<name>run(Spec, Module, Opts)</name>
<name>run(Spec, Module, Case)</name>
<name>run(Spec, Module, Case, Opts)</name>
<fsummary>Runs (specified) test suite(s)</fsummary>
<type>
<v>Specs = Spec | [Spec]</v>
<v>Spec = atom()</v>
<v>Module = atom()</v>
<v>Case = atom()</v>
<v>Opts = [Opt]</v>
<v>Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec}</v>
<v>Level = integer(); 0 means silent</v>
<v>Vars = list() of key-value tuples</v>
<v>CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.</v>
<v>Reason = term()</v>
</type>
<desc>
<p>This function runs test suite(s)/case(s). To be able to run
any tests, ts:install must first be called to create the
<c>variables</c> file needed. To run a whole test specification,
only specify the name of the test specification, and all test
suite modules belonging to that test spec will be run. To run
a single module in a test specification, use the Module
argument to specify the name of the module to run and all test
cases in that module will be run, and to run a specified test
case, specify the name of the test case using the Case
argument. If called with no argument, all test specifications
available will be run. Use ts:tests/0 to see the available
test specifications.
</p>
<p>If the <c>batch</c> option is not given, a new xterm is
started (unix) when <c>ts:run</c> is called.
</p>
<p>The <c>verbose</c> option sets the verbosity level for test
server output. This has the same effect as if given to
<c>ts:install/1/2</c></p>
<p>The <c>vars</c> option can be used for adding configuration
variables that are not in the <c>variables</c> file generated
during installation. Can be any of the <c>Opts</c> valid for
<c>ts:install/1/2</c>.
</p>
<p>The <c>keep_topcase</c> option forces <c>ts</c> to keep the
topcase in your test specification file as is. This option can
only be used if you don't give the <c>Module</c> or
<c>Case</c> parameters to <c>ts:run</c>. The
<c>keep_topcase</c> option is necessary if your topcase
contains anything other than <c><![CDATA[{dir,"../<Name>_test"}]]></c>. If
the option is not used, <c>ts</c> will modify your topcase.
</p>
<p>The <c>cover</c> and <c>cover_details</c> options indicates
that the test shall be run with code coverage
analysis. <c>cover_details</c> means that analysis shall be
done on the most detailed level. If the test is run with a
remote target, this option creates a list of uncovered lines
in each cover compiled module. If the test is run with a local
target, each cover compiled module will be analysed with
<c>cover:analyse_to_file/1</c>. The <c>cover</c> options will
only create an overview of all cover compiled modules with the
number of covered and not covered lines.
</p>
<p>The <c>CoverFile</c> which can be given with the
<c>cover</c> and <c>cover_details</c> options must be the
filename of a file listing modules to be excluded from or
included in the cover compilation. By default, <c>ts</c>
believes that <c>Spec</c> is the name of an OTP application
and that all modules in this application shall be cover
compiled. The <c>CoverFile</c> can exclude modules that belong
to the application and add modules that don't belong to the
application. The file can have the following entries:</p>
<code type="none">
{exclude, all | ExcludeModuleList}.
{include, IncludeModuleList}. </code>
<p>Note that each line must end with a full
stop. <c>ExcludeModuleList</c> and <c>IncludeModuleList</c>
are lists of atoms, where each atom is a module name.
</p>
<p>If the <c>cover</c> or <c>cover_details</c> options are
given on their own, the directory <c><![CDATA[../<Spec>_test]]></c> is
searched for a <c>CoverFile</c> named <c><![CDATA[<Spec>.cover]]></c>. If
this file is not found, <c>Spec</c> is assumed to be the name
of an OTP application, and all modules in the <c>ebin</c>
directory for the application are cover compiled. The
<c>ebin</c> directory is found by adding <c>ebin</c> to
<c>code:lib_dir(Spec)</c>.
</p>
<p>The same cover compiled code will be loaded on all slave or
peer nodes started with <c>test_server:start_node/3</c>. The
exception is nodes that run an old version of OTP. If the loading
fails, the node will simply not be a part of the coverage
analysis. Note that slave and peer nodes must be stopped with
<c>test_server:stop_node/1</c> for the node to be part of the
coverage analysis, else the test server will not be able to
fetch coverage data from the node.
</p>
<p>The <c>trace</c> option is used to turn on call trace on
target and on slave or peer nodes started with
<c>test_server:start_node/3</c>. <c>TraceSpec</c> can be the
name of a trace information file, or a list of elements like
the ones in a trace information file. Please turn to the
reference manual for <c>test_server_ctrl:trc/1</c> for details
about the trace information file.
</p>
</desc>
</func>
<func>
<name>cross_cover_analyse(Level) -> ok</name>
<name>cross_cover_analyse([Level]) -> ok</name>
<fsummary>Analyse cover data collected from all tests</fsummary>
<desc>
<p>Analyse cover data collected from all tests.
</p>
<p>See test_server_ctrl:cross_cover_analyse/1
</p>
</desc>
</func>
<func>
<name>r() -> ok</name>
<name>r(Opts) -> ok</name>
<name>r(SpecOrSuite) -> ok</name>
<name>r(SpecOrSuite,Opts) -> ok</name>
<name>r(Suite,Case) -> ok</name>
<name>r(Suite,Case,Opts) -> ok</name>
<fsummary>Run test suite or test case without <c>ts</c>installed</fsummary>
<type>
<v>SpecOrSuite = Spec | Suite</v>
<v>Spec = string()</v>
<d>"Name.spec" or "Name.spec.OsType", where OsType is vxworks</d>
<v>Suite = atom()</v>
<v>Case = atom()</v>
<v>Opts = [Opt]</v>
<v>Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}</v>
<v>Cover = cover | cover_details</v>
<v>AppOrCoverFile = Application | CoverFile</v>
<v>Application = atom()</v>
<d>OTP application to cover compile</d>
<v>CoverFile = string()</v>
<d>Name of file listing modules to exclude from or include in cover compilation</d>
</type>
<desc>
<p>This function can be used to run a test suites or test
cases directly, without any of the additional features added
by the test server framework. It is simply a wrapper function
for the <c>add_dir</c>, <c>add_spec</c>, <c>add_module</c> and
<c>add_case</c> functions in <c>test_server_ctrl</c>:
</p>
<p><c>r() -> add_dir(".")</c> <br></br>
<c>r(Spec) -> add_spec(Spec)</c> <br></br>
<c>r(Suite) -> add_module(Suite)</c> <br></br>
<c>r(Suite,Case) -> add_case(Suite,Case)</c></p>
<p>To use this function, it is required that the test suite is
compiled and in the code path of the node where the function
is called. The function can be used without having <c>ts</c>
installed.
</p>
<p>For information about the <c>cover</c> and
<c>cover_details</c> options, see <c>test_server_ctrl:cover/2/3</c>.</p>
</desc>
</func>
<func>
<name>index() -> ok | {error, Reason}</name>
<fsummary>Updates local index page</fsummary>
<type>
<v>Reason = term()</v>
</type>
<desc>
<p>This function updates the local index page. This can be
useful if a previous test run was not completed and the index
is incomplete.</p>
</desc>
</func>
<func>
<name>clean() -> ok</name>
<name>clean(all) -> ok</name>
<fsummary>Cleans up the log directories created when running tests. </fsummary>
<desc>
<p>This function cleans up log directories created when
running test cases. <c>clean/0</c> cleans up all but the last
run of each application. <c>clean/1</c> cleans up all test
runs found.</p>
</desc>
</func>
<func>
<name>estone() -> ok | {error, Reason}</name>
<name>estone(Opts) -> ok</name>
<fsummary>Runs the EStone test</fsummary>
<desc>
<p>This function runs the EStone test. It is a shortcut for
running the test suite <c>estone_SUITE</c> in the
<c>kernel</c> application.
</p>
<p><c>Opts</c> is the same as the <c>Opts</c> argument for the
<c>ts:run</c> functions.</p>
</desc>
</func>
</funcs>
<section>
<title>Makfile.src in Data Directory</title>
<p>If a data directory contains code which must be compiled before
the test suite is run, a makefile source called
<c>Makefile.src</c> can be placed in the data directory. This file
will be converted to a valid makefile by <c>ts:run/0/1/2/3/4</c>.
</p>
<p>The reason for generating the makefile is that you can use
variables from the <c>variables</c> file which was generated by
<c>ts:install/0/1/2</c>. All occurrences of <c>@Key@</c> in
<c>Makefile.src</c> is substituted by the <c>Value</c> from
<c>{Key,Value}</c> found in the <c>variables</c> file. Example:
</p>
<p>Cut from <c>variables</c>:</p>
<code type="none">
...
{'EMULATOR',"beam"}.
{'CFLAGS',"-g -O2"}.
{'LD',"$(CC) $(CFLAGS)"}.
{'CC',"gcc"}.
...
</code>
<p><c>Makefile.src</c> for compiling erlang code could look
something like this:</p>
<code type="none">
EFLAGS=+debug_info
all: ordsets1.@EMULATOR@
ordsets1.@EMULATOR@: ordsets1.erl
erlc $(EFLAGS) ordsets1.erl
</code>
<p><c>Makefile.src</c> for compiling c code could look
something like this:</p>
<code type="none">
CC = @CC@
LD = @LD@
CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@
CROSSLDFLAGS = @CROSSLDFLAGS@
PROGS = nfs_check@exe@
all: $(PROGS)
nfs_check@exe@: nfs_check@obj@
$(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@
nfs_check@obj@: nfs_check.c
$(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c
</code>
</section>
</erlref>
