aboutsummaryrefslogtreecommitdiffstats
path: root/lib/test_server/doc/src/ts.xml
diff options
context:
space:
mode:
authorBjörn Gustavsson <[email protected]>2016-02-18 15:55:37 +0100
committerBjörn Gustavsson <[email protected]>2016-02-18 15:55:37 +0100
commit24f0d567ceffb0f1555bc3032b7801ba377e5a3b (patch)
tree7ddc913ec28740fb152babae474b42cb5d554b11 /lib/test_server/doc/src/ts.xml
parent6b9b39aca9a25a328f1dc4553120f7c8f26b5648 (diff)
parentdcda9b507bf14391c8bed91bfa9c56355342b681 (diff)
downloadotp-24f0d567ceffb0f1555bc3032b7801ba377e5a3b.tar.gz
otp-24f0d567ceffb0f1555bc3032b7801ba377e5a3b.tar.bz2
otp-24f0d567ceffb0f1555bc3032b7801ba377e5a3b.zip
Merge branch 'bjorn/remove-test_server/OTP-12705'
* bjorn/remove-test_server/OTP-12705: Remove test_server as a standalone application Erlang mode for Emacs: Include ct.hrl instead test_server.hrl Remove out-commented references to the test_server applications Makefiles: Remove test_server from include path and code path Eliminate use of test_server.hrl and test_server_line.hrl
Diffstat (limited to 'lib/test_server/doc/src/ts.xml')
-rw-r--r--lib/test_server/doc/src/ts.xml568
1 files changed, 0 insertions, 568 deletions
diff --git a/lib/test_server/doc/src/ts.xml b/lib/test_server/doc/src/ts.xml
deleted file mode 100644
index 60dfdbc545..0000000000
--- a/lib/test_server/doc/src/ts.xml
+++ /dev/null
@@ -1,568 +0,0 @@
-<?xml version="1.0" encoding="utf-8" ?>
-<!DOCTYPE erlref SYSTEM "erlref.dtd">
-
-<erlref>
- <header>
- <copyright>
- <year>2007</year>
- <year>2013</year>
- <holder>Ericsson AB, All Rights Reserved</holder>
- </copyright>
- <legalnotice>
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
- The Initial Developer of the Original Code is Ericsson AB.
- </legalnotice>
-
- <title>The OTP Test Server Framework</title>
- <prepared>Mattias Nilsson</prepared>
- <responsible></responsible>
- <docno></docno>
- <approved></approved>
- <checked></checked>
- <date></date>
- <rev></rev>
- <file>ts.xml</file>
- </header>
- <module>ts</module>
- <modulesummary>Test Server Framework for testing OTP</modulesummary>
- <description>
- <p>This is a framework for testing OTP. The <c>ts</c> module
- implements the interface to all the functionality in the
- framework.
- </p>
- <p>The framework is built on top of the Test Server Controller,
- <c>test_server_ctrl</c>, and provides a high level operator
- interface. The main features added by the framework are:
- </p>
- <list type="bulleted">
- <item>Automatic compilation of test suites and data directories
- </item>
- <item>Collection of files in central directories and creation of
- additional HTML pages for better overview.
- </item>
- <item>Single command interface for running all available tests
- </item>
- <item>Spawns a new node with correct parameters before starting
- the test server
- </item>
- <item>Atomatically creates the parameter file needed when
- running tests on remote target
- </item>
- </list>
- <p>More information about the Test Server Framework and how to run
- test cases can be found in the Test Server User's Guide.
- </p>
- <p>For writing you own test server framework, please turn to the
- reference manual for the Test Server Controller and chapter named
- "Writing your own test server framework" in the Test Server User's
- Guide.
- </p>
- <p>SETUP</p>
- <p>To be able to run <c>ts</c>, you must first `install'
- <c>ts</c> for the current environment. This is done by calling
- <c>ts:install/0/1/2</c>. A file called `variables' is created
- and used by <c>ts</c> when running test suites. It is not
- recommended to edit this file, but it is possible to alter if
- <c>ts</c> gets the wrong idea about your environment.
- </p>
- <p><c>ts:install/0</c> is used if the target platform is the
- same as the controller host, i.e. if you run on "local target"
- and no options are needed. Then running <c>ts:install/0</c> <c>ts</c>
- will run an autoconf script for your current
- environment and set up the necessary variables needed by the
- test suites.
- </p>
- <p><c>ts:install/1</c> or <c>ts:install/2</c> is used if the
- target platform is different from the controller host, i.e. if
- you run on "remote target" or if special options are required
- for your system.
- </p>
- <p>See the reference manual for detailed information about
- <c>ts:install/0/1/2</c>.
- </p>
- <p>Some of the common variables in the 'variables' file are
- described below. Do not make any assumptions as of what is found
- in this file, as it may change at any time.
- </p>
- <list>
- <item><c>longnames</c><br></br>
- Set to true if the system is using fully qualified
- nodenames.
- </item>
- <item><c>platform_id</c><br></br>
- This is the currently installed platform identification
- string.
- </item>
- <item><c>platform_filename</c><br></br>
- This is the name used to create the final save directory
- for test runs.
- </item>
- <item><c>platform_label</c><br></br>
- This is the string presented in the generated test
- results index page.
- </item>
- <item><c>rsh_name</c><br></br>
- This is the rsh program to use when starting slave or
- peer nodes on a remote host.
- </item>
- <item><c>erl_flags</c><br></br>
- Compile time flags used when compiling test suites.
- </item>
- <item><c>erl_release</c><br></br>
- The Erlang/OTP release being tested.
- </item>
- <item><c>'EMULATOR'</c><br></br>
- The emulator being tested (e.g. beam)
- </item>
- <item><c>'CPU'</c><br></br>
- The CPU in the machine running the tests, e.g. sparc.
- </item>
- <item><c>target_host</c><br></br>
- The target host name
- </item>
- <item><c>os</c><br></br>
- The target operating system, e.g. solaris2.8
- </item>
- <item><c>target</c><br></br>
- The current target platform, e.g. sparc-sun-solaris2.8
- </item>
- </list>
- <p>RUNNING TESTS</p>
- <p>After installing <c>ts</c>, you can run your test with the
- <c>ts:run/0/1/2/3/4</c> functions. These functions, however,
- require a special directory structure to be able to find your
- test suites. Both the test server and all tests must be located
- under your $TESTROOT directory. The test server implementation
- shall be located in the directory <c>$TESTROOT/test_server</c>
- and for each application there must be a directory named
- <c><![CDATA[$TESTROOT/<application>_test]]></c> containing the .spec file
- and all test suites and data directories for the
- application. Note that there shall only be one .spec file for
- each application.
- </p>
- <p><c>$TESTROOT/test_server</c> must be the current directory
- when calling the <c>ts:run/*</c> function.
- </p>
- <p>All available tests can be found with <c>ts:tests()</c>. This
- will list all applications for which a test specification file
- <c><![CDATA[../<application>_test/<application>.spec]]></c> can be found.
- </p>
- <p>To run all these tests, use <c>ts:run()</c>.
- </p>
- <p>To run one or some of the tests, use <c>ts:run(Tests)</c>,
- where <c>Tests</c> is the name of the application you want to
- test, or a list of such names.
- </p>
- <p>To run one test suite within a test, use
- <c>ts:run(Test,Suite)</c>.
- </p>
- <p>To run one test case within a suite, use
- <c>ts:run(Test,Suite,Case)</c></p>
- <p>To all these functions, you can also add a list of
- options. Please turn to the reference manual for the <c>ts</c>
- module to see the valid options to use.
- </p>
- <p>The function <c>ts:help()</c> displays some simple help for
- the functions in <c>ts</c>. Use this for quick reference.
- </p>
- <p>LOG FILES</p>
- <p>As the execution of the test suites go on, events are logged in
- four different ways:
- </p>
- <list type="bulleted">
- <item>Text to the operator's console.</item>
- <item>Suite related information is sent to the major log file.</item>
- <item>Case related information is sent to the minor log file.</item>
- <item>The HTML log file gets updated with test results.</item>
- </list>
- <p>Typically the operator, who may run hundreds or thousands of
- test cases, doesn't want to fill the screen with details
- about/from the specific test cases. By default, the operator will
- only see:
- </p>
- <list type="bulleted">
- <item>A confirmation that the test has started.
- </item>
- <item>A small note about each failed test case.
- </item>
- <item>A summary of all the run test cases.
- </item>
- <item>A confirmation that the test run is complete
- </item>
- <item>Some special information like error reports and progress
- reports, printouts written with erlang:display/1 or io:format/3
- specifically addressed to somewhere other than
- <c>standard_io</c>.</item>
- </list>
- <p>This is enough for the operator to know, and if he wants to dig
- in deeper into a specific test case result, he can do so by
- following the links in the HTML presentation to take a look in the
- major or minor log files.
- </p>
- <p>A detailed report of the entire test suite is stored in the
- major logfile, the exact reason for failure, time spent etc.
- </p>
- <p>The HTML log file is a summary of the major log file, but gives
- a much better overview of the test run. It also has links to every
- test case's log file for quick viewing with a HTML browser.
- </p>
- <p>The minor log file contain full details of every single test
- case, each one in a separate file. This way the files should be
- easy to compare with previous test runs, even if the set of test
- cases change.
- </p>
- <p>Which information that goes where is user configurable via the
- test server controller. Three threshold values determine what
- comes out on screen, and in the major or minor log files. The
- contents that goes to the HTML log file is fixed, and cannot be
- altered.
- </p>
-
- </description>
- <funcs>
- <func>
- <name>install() -> ok | {error, Reason}</name>
- <name>install(TargetSystem) -> ok | {error, Reason}</name>
- <name>install(Opts) -> ok | {error, Reason}</name>
- <name>install(TargetSystem,Opts) -> ok | {error, Reason}</name>
- <fsummary>Installs the Test Server Framework</fsummary>
- <type>
- <v>TargetSystem = {Architecture, TargetHost}</v>
- <v>Architecture = atom() or string()</v>
- <d>e.g. "ose" or "vxworks_ppc603"</d>
- <v>TargetHost = atom() or string()</v>
- <d>The name of the target host</d>
- <v>Opts = list()</v>
- </type>
- <desc>
- <p>Installs and configures the Test Server Framework for
- running test suites. If a remote host is to be used, the
- <c>TargetSystem</c> argument must be given so that "cross
- installation" can be done. Installation is required for
- any of the functions in <c>ts</c> to work.
- </p>
- <p>Opts may be one or more of
- </p>
- <list>
- <item><c>{longnames, Bool}</c><br></br>
- Use fully qualified hostnames for test_server and
- slave nodes. Bool is <c>true</c> or <c>false</c> (default).
- </item>
- <item><c>{verbose, Level}</c><br></br>
- Verbosity level for test server output, set to 0, 1 or
- 2, where 0 is quiet(default).
- </item>
- <item><c>{hosts, Hosts}</c><br></br>
- This is a list of available hosts on which to start
- slave nodes. It is used when the <c>{remote, true}</c>
- option is given to the <c>test_server:start_node/3</c>
- function. Also, if <c>{require_nodenames, Num}</c> is
- contained in a test specification file, the generated
- nodenames will be spread over all hosts given in this
- <c>Hosts</c> list. The hostnames are given as atoms or
- strings.
- </item>
- <item><c>{slavetargets, SlaveTarges}</c><br></br>
- For VxWorks only. This is a list of
- available hosts where slave nodes can be started. This is
- necessary because only one node can run per host in the
- VxWorks environment. This is not the same as
- <c>{hosts, Hosts}</c> because it is used for all slave nodes
- - not only the ones started with <c>{remote, true}</c>. The
- hostnames are given as atoms or strings.
- </item>
- <item><c>{crossroot, TargetErlRoot}</c><br></br>
- Erlang root directory on target host
- <br></br>
-This option is mandatory for remote targets
- </item>
- <item><c>{master, {MasterHost, MasterCookie}}</c><br></br>
- If target is remote and the target node is started as
- a slave node, this option
- indicates which master and cookie to use. The given master
- will also be used as master for slave nodes started with
- <c>test_server:start_node/3</c>. It is expected that the
- <c>erl_boot_server</c> is started on the master node before
- the test is run. If this option is not given, the test
- server controller node is used as master and the
- <c>erl_boot_server</c> is automatically started.
- </item>
- <item><c>{erl_start_args, ArgString}</c><br></br>
- Additional arguments to be used when starting the test
- server controller node. <c>ArgString</c> will be appended to
- the command line when starting the erlang node. Note that
- this will only affect the startup of the <em>controller node</em>,
- i.e. not the target node or any slave nodes
- startet from a test case.
- </item>
- <item><c>{ipv6_hosts, HostList}</c><br></br>
- This option will be inserted in the
- <c>Config</c> parameter for each test case. <c>HostList</c>
- is a list of hosts supporting IPv6.
- </item>
- </list>
- </desc>
- </func>
- <func>
- <name>help() -> ok</name>
- <fsummary>Presents simple help on the functions in <c>ts</c></fsummary>
- <desc>
- <p>Presents simple help on the functions in <c>ts</c>. Useful
- for quick reference.</p>
- </desc>
- </func>
- <func>
- <name>tests() -> Tests</name>
- <fsummary>Returns the list of available tests</fsummary>
- <desc>
- <p>Returns the list of available tests. This is actually just
- a list of all test specification files found by looking up
- "../*_test/*.spec".
- </p>
- <p>In each ../Name_test/ directory there should be one test
- specification file named Name.spec.</p>
- </desc>
- </func>
- <func>
- <name>run() -> ok | {error, Reason}</name>
- <name>run([all_tests|Opts])</name>
- <name>run(Specs)</name>
- <name>run(Specs, Opts)</name>
- <name>run(Spec, Module)</name>
- <name>run(Spec, Module, Opts)</name>
- <name>run(Spec, Module, Case)</name>
- <name>run(Spec, Module, Case, Opts)</name>
- <fsummary>Runs (specified) test suite(s)</fsummary>
- <type>
- <v>Specs = Spec | [Spec]</v>
- <v>Spec = atom()</v>
- <v>Module = atom()</v>
- <v>Case = atom()</v>
- <v>Opts = [Opt]</v>
- <v>Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec}</v>
- <v>Level = integer(); 0 means silent</v>
- <v>Vars = list() of key-value tuples</v>
- <v>CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.</v>
- <v>Reason = term()</v>
- </type>
- <desc>
- <p>This function runs test suite(s)/case(s). To be able to run
- any tests, ts:install must first be called to create the
- <c>variables</c> file needed. To run a whole test specification,
- only specify the name of the test specification, and all test
- suite modules belonging to that test spec will be run. To run
- a single module in a test specification, use the Module
- argument to specify the name of the module to run and all test
- cases in that module will be run, and to run a specified test
- case, specify the name of the test case using the Case
- argument. If called with no argument, all test specifications
- available will be run. Use ts:tests/0 to see the available
- test specifications.
- </p>
- <p>If the <c>batch</c> option is not given, a new xterm is
- started (unix) when <c>ts:run</c> is called.
- </p>
- <p>The <c>verbose</c> option sets the verbosity level for test
- server output. This has the same effect as if given to
- <c>ts:install/1/2</c></p>
- <p>The <c>vars</c> option can be used for adding configuration
- variables that are not in the <c>variables</c> file generated
- during installation. Can be any of the <c>Opts</c> valid for
- <c>ts:install/1/2</c>.
- </p>
- <p>The <c>keep_topcase</c> option forces <c>ts</c> to keep the
- topcase in your test specification file as is. This option can
- only be used if you don't give the <c>Module</c> or
- <c>Case</c> parameters to <c>ts:run</c>. The
- <c>keep_topcase</c> option is necessary if your topcase
- contains anything other than <c><![CDATA[{dir,"../<Name>_test"}]]></c>. If
- the option is not used, <c>ts</c> will modify your topcase.
- </p>
- <p>The <c>cover</c> and <c>cover_details</c> options indicates
- that the test shall be run with code coverage
- analysis. <c>cover_details</c> means that analysis shall be
- done on the most detailed level. If the test is run with a
- remote target, this option creates a list of uncovered lines
- in each cover compiled module. If the test is run with a local
- target, each cover compiled module will be analysed with
- <c>cover:analyse_to_file/1</c>. The <c>cover</c> options will
- only create an overview of all cover compiled modules with the
- number of covered and not covered lines.
- </p>
- <p>The <c>CoverFile</c> which can be given with the
- <c>cover</c> and <c>cover_details</c> options must be the
- filename of a file listing modules to be excluded from or
- included in the cover compilation. By default, <c>ts</c>
- believes that <c>Spec</c> is the name of an OTP application
- and that all modules in this application shall be cover
- compiled. The <c>CoverFile</c> can exclude modules that belong
- to the application and add modules that don't belong to the
- application. The file can have the following entries:</p>
- <code type="none">
-{exclude, all | ExcludeModuleList}.
-{include, IncludeModuleList}. </code>
- <p>Note that each line must end with a full
- stop. <c>ExcludeModuleList</c> and <c>IncludeModuleList</c>
- are lists of atoms, where each atom is a module name.
- </p>
- <p>If the <c>cover</c> or <c>cover_details</c> options are
- given on their own, the directory <c><![CDATA[../<Spec>_test]]></c> is
- searched for a <c>CoverFile</c> named <c><![CDATA[<Spec>.cover]]></c>. If
- this file is not found, <c>Spec</c> is assumed to be the name
- of an OTP application, and all modules in the <c>ebin</c>
- directory for the application are cover compiled. The
- <c>ebin</c> directory is found by adding <c>ebin</c> to
- <c>code:lib_dir(Spec)</c>.
- </p>
- <p>The same cover compiled code will be loaded on all slave or
- peer nodes started with <c>test_server:start_node/3</c>. The
- exception is nodes that run an old version of OTP. If the loading
- fails, the node will simply not be a part of the coverage
- analysis. Note that slave and peer nodes must be stopped with
- <c>test_server:stop_node/1</c> for the node to be part of the
- coverage analysis, else the test server will not be able to
- fetch coverage data from the node.
- </p>
- <p>The <c>trace</c> option is used to turn on call trace on
- target and on slave or peer nodes started with
- <c>test_server:start_node/3</c>. <c>TraceSpec</c> can be the
- name of a trace information file, or a list of elements like
- the ones in a trace information file. Please turn to the
- reference manual for <c>test_server_ctrl:trc/1</c> for details
- about the trace information file.
- </p>
- </desc>
- </func>
- <func>
- <name>cross_cover_analyse(Level) -> ok</name>
- <name>cross_cover_analyse([Level]) -> ok</name>
- <fsummary>Analyse cover data collected from all tests</fsummary>
- <desc>
- <p>Analyse cover data collected from all tests.
- </p>
- <p>See test_server_ctrl:cross_cover_analyse/2
- </p>
- </desc>
- </func>
- <func>
- <name>r() -> ok</name>
- <name>r(Opts) -> ok</name>
- <name>r(SpecOrSuite) -> ok</name>
- <name>r(SpecOrSuite,Opts) -> ok</name>
- <name>r(Suite,Case) -> ok</name>
- <name>r(Suite,Case,Opts) -> ok</name>
- <fsummary>Run test suite or test case without <c>ts</c>installed</fsummary>
- <type>
- <v>SpecOrSuite = Spec | Suite</v>
- <v>Spec = string()</v>
- <d>"Name.spec" or "Name.spec.OsType", where OsType is vxworks</d>
- <v>Suite = atom()</v>
- <v>Case = atom()</v>
- <v>Opts = [Opt]</v>
- <v>Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}</v>
- <v>Cover = cover | cover_details</v>
- <v>AppOrCoverFile = Application | CoverFile</v>
- <v>Application = atom()</v>
- <d>OTP application to cover compile</d>
- <v>CoverFile = string()</v>
- <d>Name of file listing modules to exclude from or include in cover compilation</d>
- </type>
- <desc>
- <p>This function can be used to run a test suites or test
- cases directly, without any of the additional features added
- by the test server framework. It is simply a wrapper function
- for the <c>add_dir</c>, <c>add_spec</c>, <c>add_module</c> and
- <c>add_case</c> functions in <c>test_server_ctrl</c>:
- </p>
- <p><c>r() -> add_dir(".")</c> <br></br>
-<c>r(Spec) -> add_spec(Spec)</c> <br></br>
-<c>r(Suite) -> add_module(Suite)</c> <br></br>
-<c>r(Suite,Case) -> add_case(Suite,Case)</c></p>
- <p>To use this function, it is required that the test suite is
- compiled and in the code path of the node where the function
- is called. The function can be used without having <c>ts</c>
- installed.
- </p>
- <p>For information about the <c>cover</c> and
- <c>cover_details</c> options, see <c>test_server_ctrl:cover/2/3</c>.</p>
- </desc>
- </func>
- <func>
- <name>estone() -> ok | {error, Reason}</name>
- <name>estone(Opts) -> ok</name>
- <fsummary>Runs the EStone test</fsummary>
- <desc>
- <p>This function runs the EStone test. It is a shortcut for
- running the test suite <c>estone_SUITE</c> in the
- <c>kernel</c> application.
- </p>
- <p><c>Opts</c> is the same as the <c>Opts</c> argument for the
- <c>ts:run</c> functions.</p>
- </desc>
- </func>
- </funcs>
-
- <section>
- <title>Makfile.src in Data Directory</title>
- <p>If a data directory contains code which must be compiled before
- the test suite is run, a makefile source called
- <c>Makefile.src</c> can be placed in the data directory. This file
- will be converted to a valid makefile by <c>ts:run/0/1/2/3/4</c>.
- </p>
- <p>The reason for generating the makefile is that you can use
- variables from the <c>variables</c> file which was generated by
- <c>ts:install/0/1/2</c>. All occurrences of <c>@Key@</c> in
- <c>Makefile.src</c> is substituted by the <c>Value</c> from
- <c>{Key,Value}</c> found in the <c>variables</c> file. Example:
- </p>
- <p>Cut from <c>variables</c>:</p>
- <code type="none">
- ...
- {'EMULATOR',"beam"}.
- {'CFLAGS',"-g -O2"}.
- {'LD',"$(CC) $(CFLAGS)"}.
- {'CC',"gcc"}.
- ...
- </code>
- <p><c>Makefile.src</c> for compiling erlang code could look
- something like this:</p>
- <code type="none">
- EFLAGS=+debug_info
-
- all: ordsets1.@EMULATOR@
-
- ordsets1.@EMULATOR@: ordsets1.erl
- erlc $(EFLAGS) ordsets1.erl
- </code>
- <p><c>Makefile.src</c> for compiling c code could look
- something like this:</p>
- <code type="none">
- CC = @CC@
- LD = @LD@
- CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@
- CROSSLDFLAGS = @CROSSLDFLAGS@
-
- PROGS = nfs_check@exe@
-
- all: $(PROGS)
-
- nfs_check@exe@: nfs_check@obj@
- $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@
-
- nfs_check@obj@: nfs_check.c
- $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c
- </code>
- </section>
-</erlref>
-