diff options
Diffstat (limited to 'lib/test_server/doc/src/ts.xml')
| -rw-r--r-- | lib/test_server/doc/src/ts.xml | 592 |
1 files changed, 592 insertions, 0 deletions
diff --git a/lib/test_server/doc/src/ts.xml b/lib/test_server/doc/src/ts.xml new file mode 100644 index 0000000000..0f91d3eea2 --- /dev/null +++ b/lib/test_server/doc/src/ts.xml @@ -0,0 +1,592 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE erlref SYSTEM "erlref.dtd"> + +<erlref> + <header> + <copyright> + <year>2007</year> + <year>2008</year> + <holder>Ericsson AB, All Rights Reserved</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + The Initial Developer of the Original Code is Ericsson AB. + </legalnotice> + + <title>The OTP Test Server Framework</title> + <prepared>Mattias Nilsson</prepared> + <responsible></responsible> + <docno></docno> + <approved></approved> + <checked></checked> + <date></date> + <rev></rev> + <file>ts.xml</file> + </header> + <module>ts</module> + <modulesummary>Test Server Framework for testing OTP</modulesummary> + <description> + <p>This is a framework for testing OTP. The <c>ts</c> module + implements the interface to all the functionality in the + framework. + </p> + <p>The framework is built on top of the Test Server Controller, + <c>test_server_ctrl</c>, and provides a high level operator + interface. The main features added by the framework are: + </p> + <list type="bulleted"> + <item>Automatic compilation of test suites and data directories + </item> + <item>Collection of files in central directories and creation of + additional HTML pages for better overview. + </item> + <item>Single command interface for running all available tests + </item> + <item>Spawns a new node with correct parameters before starting + the test server + </item> + <item>Atomatically creates the parameter file needed when + running tests on remote target + </item> + </list> + <p>More information about the Test Server Framework and how to run + test cases can be found in the Test Server User's Guide. + </p> + <p>For writing you own test server framework, please turn to the + reference manual for the Test Server Controller and chapter named + "Writing your own test server framework" in the Test Server User's + Guide. + </p> + <p>SETUP</p> + <p>To be able to run <c>ts</c>, you must first `install' + <c>ts</c> for the current environment. This is done by calling + <c>ts:install/0/1/2</c>. A file called `variables' is created + and used by <c>ts</c> when running test suites. It is not + recommended to edit this file, but it is possible to alter if + <c>ts</c> gets the wrong idea about your environment. + </p> + <p><c>ts:install/0</c> is used if the target platform is the + same as the controller host, i.e. if you run on "local target" + and no options are needed. Then running <c>ts:install/0</c><c>ts</c> + will run an autoconf script for your current + environment and set up the necessary variables needed by the + test suites. + </p> + <p><c>ts:install/1</c> or <c>ts:install/2</c> is used if the + target platform is different from the controller host, i.e. if + you run on "remote target" or if special options are required + for your system. VxWorks is currently supported + as remote target platform. + </p> + <p>See the reference manual for detailed information about + <c>ts:install/0/1/2</c>. + </p> + <p>Some of the common variables in the 'variables' file are + described below. Do not make any assumptions as of what is found + in this file, as it may change at any time. + </p> + <list> + <item><c>longnames</c><br></br> + Set to true if the system is using fully qualified + nodenames. + </item> + <item><c>platform_id</c><br></br> + This is the currently installed platform identification + string. + </item> + <item><c>platform_filename</c><br></br> + This is the name used to create the final save directory + for test runs. + </item> + <item><c>platform_label</c><br></br> + This is the string presented in the generated test + results index page. + </item> + <item><c>rsh_name</c><br></br> + This is the rsh program to use when starting slave or + peer nodes on a remote host. + </item> + <item><c>erl_flags</c><br></br> + Compile time flags used when compiling test suites. + </item> + <item><c>erl_release</c><br></br> + The Erlang/OTP release being tested. + </item> + <item><c>'EMULATOR'</c><br></br> + The emulator being tested (e.g. beam) + </item> + <item><c>'CPU'</c><br></br> + The CPU in the machine running the tests, e.g. sparc. + </item> + <item><c>target_host</c><br></br> + The target host name + </item> + <item><c>os</c><br></br> + The target operating system, e.g. solaris2.8 + </item> + <item><c>target</c><br></br> + The current target platform, e.g. sparc-sun-solaris2.8 + </item> + </list> + <p>RUNNING TESTS</p> + <p>After installing <c>ts</c>, you can run your test with the + <c>ts:run/0/1/2/3/4</c> functions. These functions, however, + require a special directory structure to be able to find your + test suites. Both the test server and all tests must be located + under your $TESTROOT directory. The test server implementation + shall be located in the directory <c>$TESTROOT/test_server</c> + and for each application there must be a directory named + <c><![CDATA[$TESTROOT/<application>_test]]></c> containing the .spec file + and all test suites and data directories for the + application. Note that there shall only be one .spec file for + each application. + </p> + <p><c>$TESTROOT/test_server</c> must be the current directory + when calling the <c>ts:run/*</c> function. + </p> + <p>All available tests can be found with <c>ts:tests()</c>. This + will list all applications for which a test specification file + <c><![CDATA[../<application>_test/<application>.spec]]></c> can be found. + </p> + <p>To run all these tests, use <c>ts:run()</c>. + </p> + <p>To run one or some of the tests, use <c>ts:run(Tests)</c>, + where <c>Tests</c> is the name of the application you want to + test, or a list of such names. + </p> + <p>To run one test suite within a test, use + <c>ts:run(Test,Suite)</c>. + </p> + <p>To run one test case within a suite, use + <c>ts:run(Test,Suite,Case)</c></p> + <p>To all these functions, you can also add a list of + options. Please turn to the reference manual for the <c>ts</c> + module to see the valid options to use. + </p> + <p>The function <c>ts:help()</c> displays some simple help for + the functions in <c>ts</c>. Use this for quick reference. + </p> + <p>LOG FILES</p> + <p>As the execution of the test suites go on, events are logged in + four different ways: + </p> + <list type="bulleted"> + <item>Text to the operator's console.</item> + <item>Suite related information is sent to the major log file.</item> + <item>Case related information is sent to the minor log file.</item> + <item>The HTML log file gets updated with test results.</item> + </list> + <p>Typically the operator, who may run hundreds or thousands of + test cases, doesn't want to fill the screen with details + about/from the specific test cases. By default, the operator will + only see: + </p> + <list type="bulleted"> + <item>A confirmation that the test has started. + </item> + <item>A small note about each failed test case. + </item> + <item>A summary of all the run test cases. + </item> + <item>A confirmation that the test run is complete + </item> + <item>Some special information like error reports and progress + reports, printouts written with erlang:display/1 or io:format/3 + specifically addressed to somewhere other than + <c>standard_io</c>.</item> + </list> + <p>This is enough for the operator to know, and if he wants to dig + in deeper into a specific test case result, he can do so by + following the links in the HTML presentation to take a look in the + major or minor log files. + </p> + <p>A detailed report of the entire test suite is stored in the + major logfile, the exact reason for failure, time spent etc. + </p> + <p>The HTML log file is a summary of the major log file, but gives + a much better overview of the test run. It also has links to every + test case's log file for quick viewing with a HTML browser. + </p> + <p>The minor log file contain full details of every single test + case, each one in a separate file. This way the files should be + easy to compare with previous test runs, even if the set of test + cases change. + </p> + <p>Which information that goes where is user configurable via the + test server controller. Three threshold values determine what + comes out on screen, and in the major or minor log files. The + contents that goes to the HTML log file is fixed, and cannot be + altered. + </p> + + </description> + <funcs> + <func> + <name>install() -> ok | {error, Reason}</name> + <name>install(TargetSystem) -> ok | {error, Reason}</name> + <name>install(Opts) -> ok | {error, Reason}</name> + <name>install(TargetSystem,Opts) -> ok | {error, Reason}</name> + <fsummary>Installs the Test Server Framework</fsummary> + <type> + <v>TargetSystem = {Architecture, TargetHost}</v> + <v>Architecture = atom() or string()</v> + <d>e.g. "ose" or "vxworks_ppc603"</d> + <v>TargetHost = atom() or string()</v> + <d>The name of the target host</d> + <v>Opts = list()</v> + </type> + <desc> + <p>Installs and configures the Test Server Framework for + running test suites. If a remote host is to be used, the + <c>TargetSystem</c> argument must be given so that "cross + installation" can be done. This should be used for testing on + VxWorks or OSE/Delta. Installation is required for any of the + functions in <c>ts</c> to work. + </p> + <p>Opts may be one or more of + </p> + <list> + <item><c>{longnames, Bool}</c><br></br> + Use fully qualified hostnames for test_server and + slave nodes. Bool is <c>true</c> or <c>false</c> (default). + </item> + <item><c>{verbose, Level}</c><br></br> + Verbosity level for test server output, set to 0, 1 or + 2, where 0 is quiet(default). + </item> + <item><c>{hosts, Hosts}</c><br></br> + This is a list of available hosts on which to start + slave nodes. It is used when the <c>{remote, true}</c> + option is given to the <c>test_server:start_node/3</c> + function. Also, if <c>{require_nodenames, Num}</c> is + contained in a test specification file, the generated + nodenames will be spread over all hosts given in this + <c>Hosts</c> list. The hostnames are given as atoms or + strings. + </item> + <item><c>{slavetargets, SlaveTarges}</c><br></br> + For VxWorks and OSE/Delta only. This is a list of + available hosts where slave nodes can be started. This is + necessary because only one node can run per host in the + VxWorks environment. This is not the same as + <c>{hosts, Hosts}</c> because it is used for all slave nodes + - not only the ones started with <c>{remote, true}</c>. The + hostnames are given as atoms or strings. + </item> + <item><c>{crossroot, TargetErlRoot}</c><br></br> + Erlang root directory on target host + <br></br> +This option is mandatory for remote targets + </item> + <item><c>{master, {MasterHost, MasterCookie}}</c><br></br> + If target is remote and the target node is started as + a slave node, this option + indicates which master and cookie to use. The given master + will also be used as master for slave nodes started with + <c>test_server:start_node/3</c>. It is expected that the + <c>erl_boot_server</c> is started on the master node before + the test is run. If this option is not given, the test + server controller node is used as master and the + <c>erl_boot_server</c> is automatically started. + </item> + <item><c>{erl_start_args, ArgString}</c><br></br> + Additional arguments to be used when starting the test + server controller node. <c>ArgString</c> will be appended to + the command line when starting the erlang node. Note that + this will only affect the startup of the <em>controller node</em>, + i.e. not the target node or any slave nodes + startet from a test case. + </item> + <item><c>{ipv6_hosts, HostList}</c><br></br> + This option will be inserted in the + <c>Config</c> parameter for each test case. <c>HostList</c> + is a list of hosts supporting IPv6. + </item> + </list> + </desc> + </func> + <func> + <name>help() -> ok</name> + <fsummary>Presents simple help on the functions in <c>ts</c></fsummary> + <desc> + <p>Presents simple help on the functions in <c>ts</c>. Useful + for quick reference.</p> + </desc> + </func> + <func> + <name>tests() -> Tests</name> + <fsummary>Returns the list of available tests</fsummary> + <desc> + <p>Returns the list of available tests. This is actually just + a list of all test specification files found by looking up + "../*_test/*.spec". + </p> + <p>In each ../Name_test/ directory there should be one test + specification file named Name.spec.</p> + </desc> + </func> + <func> + <name>run() -> ok | {error, Reason}</name> + <name>run([all_tests|Opts])</name> + <name>run(Specs)</name> + <name>run(Specs, Opts)</name> + <name>run(Spec, Module)</name> + <name>run(Spec, Module, Opts)</name> + <name>run(Spec, Module, Case)</name> + <name>run(Spec, Module, Case, Opts)</name> + <fsummary>Runs (specified) test suite(s)</fsummary> + <type> + <v>Specs = Spec | [Spec]</v> + <v>Spec = atom()</v> + <v>Module = atom()</v> + <v>Case = atom()</v> + <v>Opts = [Opt]</v> + <v>Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec}</v> + <v>Level = integer(); 0 means silent</v> + <v>Vars = list() of key-value tuples</v> + <v>CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.</v> + <v>Reason = term()</v> + </type> + <desc> + <p>This function runs test suite(s)/case(s). To be able to run + any tests, ts:install must first be called to create the + <c>variables</c> file needed. To run a whole test specification, + only specify the name of the test specification, and all test + suite modules belonging to that test spec will be run. To run + a single module in a test specification, use the Module + argument to specify the name of the module to run and all test + cases in that module will be run, and to run a specified test + case, specify the name of the test case using the Case + argument. If called with no argument, all test specifications + available will be run. Use ts:tests/0 to see the available + test specifications. + </p> + <p>If the <c>batch</c> option is not given, a new xterm is + started (unix) when <c>ts:run</c> is called. + </p> + <p>The <c>verbose</c> option sets the verbosity level for test + server output. This has the same effect as if given to + <c>ts:install/1/2</c></p> + <p>The <c>vars</c> option can be used for adding configuration + variables that are not in the <c>variables</c> file generated + during installation. Can be any of the <c>Opts</c> valid for + <c>ts:install/1/2</c>. + </p> + <p>The <c>keep_topcase</c> option forces <c>ts</c> to keep the + topcase in your test specification file as is. This option can + only be used if you don't give the <c>Module</c> or + <c>Case</c> parameters to <c>ts:run</c>. The + <c>keep_topcase</c> option is necessary if your topcase + contains anything other than <c><![CDATA[{dir,"../<Name>_test"}]]></c>. If + the option is not used, <c>ts</c> will modify your topcase. + </p> + <p>The <c>cover</c> and <c>cover_details</c> options indicates + that the test shall be run with code coverage + analysis. <c>cover_details</c> means that analysis shall be + done on the most detailed level. If the test is run with a + remote target, this option creates a list of uncovered lines + in each cover compiled module. If the test is run with a local + target, each cover compiled module will be analysed with + <c>cover:analyse_to_file/1</c>. The <c>cover</c> options will + only create an overview of all cover compiled modules with the + number of covered and not covered lines. + </p> + <p>The <c>CoverFile</c> which can be given with the + <c>cover</c> and <c>cover_details</c> options must be the + filename of a file listing modules to be excluded from or + included in the cover compilation. By default, <c>ts</c> + believes that <c>Spec</c> is the name of an OTP application + and that all modules in this application shall be cover + compiled. The <c>CoverFile</c> can exclude modules that belong + to the application and add modules that don't belong to the + application. The file can have the following entries:</p> + <code type="none"> +{exclude, all | ExcludeModuleList}. +{include, IncludeModuleList}. </code> + <p>Note that each line must end with a full + stop. <c>ExcludeModuleList</c> and <c>IncludeModuleList</c> + are lists of atoms, where each atom is a module name. + </p> + <p>If the <c>cover</c> or <c>cover_details</c> options are + given on their own, the directory <c><![CDATA[../<Spec>_test]]></c> is + searched for a <c>CoverFile</c> named <c><![CDATA[<Spec>.cover]]></c>. If + this file is not found, <c>Spec</c> is assumed to be the name + of an OTP application, and all modules in the <c>ebin</c> + directory for the application are cover compiled. The + <c>ebin</c> directory is found by adding <c>ebin</c> to + <c>code:lib_dir(Spec)</c>. + </p> + <p>The same cover compiled code will be loaded on all slave or + peer nodes started with <c>test_server:start_node/3</c>. The + exception is nodes that run an old version of OTP. If the loading + fails, the node will simply not be a part of the coverage + analysis. Note that slave and peer nodes must be stopped with + <c>test_server:stop_node/1</c> for the node to be part of the + coverage analysis, else the test server will not be able to + fetch coverage data from the node. + </p> + <p>The <c>trace</c> option is used to turn on call trace on + target and on slave or peer nodes started with + <c>test_server:start_node/3</c>. <c>TraceSpec</c> can be the + name of a trace information file, or a list of elements like + the ones in a trace information file. Please turn to the + reference manual for <c>test_server_ctrl:trc/1</c> for details + about the trace information file. + </p> + </desc> + </func> + <func> + <name>cross_cover_analyse(Level) -> ok</name> + <name>cross_cover_analyse([Level]) -> ok</name> + <fsummary>Analyse cover data collected from all tests</fsummary> + <desc> + <p>Analyse cover data collected from all tests. + </p> + <p>See test_server_ctrl:cross_cover_analyse/1 + </p> + </desc> + </func> + <func> + <name>r() -> ok</name> + <name>r(Opts) -> ok</name> + <name>r(SpecOrSuite) -> ok</name> + <name>r(SpecOrSuite,Opts) -> ok</name> + <name>r(Suite,Case) -> ok</name> + <name>r(Suite,Case,Opts) -> ok</name> + <fsummary>Run test suite or test case without <c>ts</c>installed</fsummary> + <type> + <v>SpecOrSuite = Spec | Suite</v> + <v>Spec = string()</v> + <d>"Name.spec" or "Name.spec.OsType", where OsType is vxworks</d> + <v>Suite = atom()</v> + <v>Case = atom()</v> + <v>Opts = [Opt]</v> + <v>Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}</v> + <v>Cover = cover | cover_details</v> + <v>AppOrCoverFile = Application | CoverFile</v> + <v>Application = atom()</v> + <d>OTP application to cover compile</d> + <v>CoverFile = string()</v> + <d>Name of file listing modules to exclude from or include in cover compilation</d> + </type> + <desc> + <p>This function can be used to run a test suites or test + cases directly, without any of the additional features added + by the test server framework. It is simply a wrapper function + for the <c>add_dir</c>, <c>add_spec</c>, <c>add_module</c> and + <c>add_case</c> functions in <c>test_server_ctrl</c>: + </p> + <p><c>r() -> add_dir(".")</c> <br></br> +<c>r(Spec) -> add_spec(Spec)</c> <br></br> +<c>r(Suite) -> add_module(Suite)</c> <br></br> +<c>r(Suite,Case) -> add_case(Suite,Case)</c></p> + <p>To use this function, it is required that the test suite is + compiled and in the code path of the node where the function + is called. The function can be used without having <c>ts</c> + installed. + </p> + <p>For information about the <c>cover</c> and + <c>cover_details</c> options, see <c>test_server_ctrl:cover/2/3</c>.</p> + </desc> + </func> + <func> + <name>index() -> ok | {error, Reason}</name> + <fsummary>Updates local index page</fsummary> + <type> + <v>Reason = term()</v> + </type> + <desc> + <p>This function updates the local index page. This can be + useful if a previous test run was not completed and the index + is incomplete.</p> + </desc> + </func> + <func> + <name>clean() -> ok</name> + <name>clean(all) -> ok</name> + <fsummary>Cleans up the log directories created when running tests. </fsummary> + <desc> + <p>This function cleans up log directories created when + running test cases. <c>clean/0</c> cleans up all but the last + run of each application. <c>clean/1</c> cleans up all test + runs found.</p> + </desc> + </func> + <func> + <name>estone() -> ok | {error, Reason}</name> + <name>estone(Opts) -> ok</name> + <fsummary>Runs the EStone test</fsummary> + <desc> + <p>This function runs the EStone test. It is a shortcut for + running the test suite <c>estone_SUITE</c> in the + <c>kernel</c> application. + </p> + <p><c>Opts</c> is the same as the <c>Opts</c> argument for the + <c>ts:run</c> functions.</p> + </desc> + </func> + </funcs> + + <section> + <title>Makfile.src in Data Directory</title> + <p>If a data directory contains code which must be compiled before + the test suite is run, a makefile source called + <c>Makefile.src</c> can be placed in the data directory. This file + will be converted to a valid makefile by <c>ts:run/0/1/2/3/4</c>. + </p> + <p>The reason for generating the makefile is that you can use + variables from the <c>variables</c> file which was generated by + <c>ts:install/0/1/2</c>. All occurrences of <c>@Key@</c> in + <c>Makefile.src</c> is substituted by the <c>Value</c> from + <c>{Key,Value}</c> found in the <c>variables</c> file. Example: + </p> + <p>Cut from <c>variables</c>:</p> + <code type="none"> + ... + {'EMULATOR',"beam"}. + {'CFLAGS',"-g -O2"}. + {'LD',"$(CC) $(CFLAGS)"}. + {'CC',"gcc"}. + ... + </code> + <p><c>Makefile.src</c> for compiling erlang code could look + something like this:</p> + <code type="none"> + EFLAGS=+debug_info + + all: ordsets1.@EMULATOR@ + + ordsets1.@EMULATOR@: ordsets1.erl + erlc $(EFLAGS) ordsets1.erl + </code> + <p><c>Makefile.src</c> for compiling c code could look + something like this:</p> + <code type="none"> + CC = @CC@ + LD = @LD@ + CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@ + CROSSLDFLAGS = @CROSSLDFLAGS@ + + PROGS = nfs_check@exe@ + + all: $(PROGS) + + nfs_check@exe@: nfs_check@obj@ + $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@ + + nfs_check@obj@: nfs_check.c + $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c + </code> + </section> +</erlref> + |