path: root/lib/test_server/doc/src/ts.xml

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">

<erlref>
  <header>
    <copyright>
      <year>2007</year>
      <year>2013</year>
      <holder>Ericsson AB, All Rights Reserved</holder>
    </copyright>
    <legalnotice>
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
 
      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.

  The Initial Developer of the Original Code is Ericsson AB.
    </legalnotice>

    <title>The OTP Test Server Framework</title>
    <prepared>Mattias Nilsson</prepared>
    <responsible></responsible>
    <docno></docno>
    <approved></approved>
    <checked></checked>
    <date></date>
    <rev></rev>
    <file>ts.xml</file>
  </header>
  <module>ts</module>
  <modulesummary>Test Server Framework for testing OTP</modulesummary>
  <description>
    <p>This is a framework for testing OTP. The <c>ts</c> module
      implements the interface to all the functionality in the
      framework.
      </p>
    <p>The framework is built on top of the Test Server Controller,
      <c>test_server_ctrl</c>, and provides a high level operator
      interface. The main features added by the framework are:
      </p>
    <list type="bulleted">
      <item>Automatic compilation of test suites and data directories
      </item>
      <item>Collection of files in central directories and creation of
       additional HTML pages for better overview.
      </item>
      <item>Single command interface for running all available tests
      </item>
      <item>Spawns a new node with correct parameters before starting
       the test server
      </item>
      <item>Atomatically creates the parameter file needed when
       running tests on remote target
      </item>
    </list>
    <p>More information about the Test Server Framework and how to run
      test cases can be found in the Test Server User's Guide.
      </p>
    <p>For writing you own test server framework, please turn to the
      reference manual for the Test Server Controller and chapter named
      "Writing your own test server framework" in the Test Server User's
      Guide.
      </p>
      <p>SETUP</p>
    <p>To be able to run <c>ts</c>, you must first `install'
        <c>ts</c> for the current environment. This is done by calling
        <c>ts:install/0/1/2</c>. A file called `variables' is created
        and used by <c>ts</c> when running test suites. It is not
        recommended to edit this file, but it is possible to alter if
        <c>ts</c> gets the wrong idea about your environment.
        </p>
    <p><c>ts:install/0</c> is used if the target platform is the
        same as the controller host, i.e. if you run on "local target"
        and no options are needed. Then running <c>ts:install/0</c> <c>ts</c>
	will run an autoconf script for your current
        environment and set up the necessary variables needed by the
        test suites.
        </p>
    <p><c>ts:install/1</c> or <c>ts:install/2</c> is used if the
        target platform is different from the controller host, i.e. if
        you run on "remote target" or if special options are required
        for your system.
        </p>
      <p>See the reference manual for detailed information about
        <c>ts:install/0/1/2</c>.
        </p>
    <p>Some of the common variables in the 'variables' file are
        described below. Do not make any assumptions as of what is found
        in this file, as it may change at any time.
        </p>
    <list>
        <item><c>longnames</c><br></br>
        Set to true if the system is using fully qualified
         nodenames.
        </item>
        <item><c>platform_id</c><br></br>
        This is the currently installed platform identification
         string.
        </item>
        <item><c>platform_filename</c><br></br>
        This is the name used to create the final save directory
         for test runs.
        </item>
        <item><c>platform_label</c><br></br>
        This is the string presented in the generated test
         results index page.
        </item>
        <item><c>rsh_name</c><br></br>
        This is the rsh program to use when starting slave or
         peer nodes on a remote host.
        </item>
        <item><c>erl_flags</c><br></br>
        Compile time flags used when compiling test suites.
        </item>
        <item><c>erl_release</c><br></br>
        The Erlang/OTP release being tested.
        </item>
        <item><c>'EMULATOR'</c><br></br>
        The emulator being tested (e.g. beam)
        </item>
        <item><c>'CPU'</c><br></br>
        The CPU in the machine running the tests, e.g. sparc.
        </item>
        <item><c>target_host</c><br></br>
        The target host name
        </item>
        <item><c>os</c><br></br>
        The target operating system, e.g. solaris2.8
        </item>
        <item><c>target</c><br></br>
        The current target platform, e.g. sparc-sun-solaris2.8
        </item>
      </list>
      <p>RUNNING TESTS</p>
      <p>After installing <c>ts</c>, you can run your test with the
        <c>ts:run/0/1/2/3/4</c> functions. These functions, however,
        require a special directory structure to be able to find your
        test suites. Both the test server and all tests must be located
        under your $TESTROOT directory. The test server implementation
        shall be located in the directory <c>$TESTROOT/test_server</c>
        and for each application there must be a directory named
        <c><![CDATA[$TESTROOT/<application>_test]]></c> containing the .spec file
        and all test suites and data directories for the
        application. Note that there shall only be one .spec file for
        each application.
        </p>
      <p><c>$TESTROOT/test_server</c> must be the current directory
        when calling the <c>ts:run/*</c> function.
        </p>
      <p>All available tests can be found with <c>ts:tests()</c>. This
        will list all applications for which a test specification file
        <c><![CDATA[../<application>_test/<application>.spec]]></c> can be found.
        </p>
      <p>To run all these tests, use <c>ts:run()</c>.
        </p>
      <p>To run one or some of the tests, use <c>ts:run(Tests)</c>,
        where <c>Tests</c> is the name of the application you want to
        test, or a list of such names.
        </p>
      <p>To run one test suite within a test, use
        <c>ts:run(Test,Suite)</c>.
        </p>
      <p>To run one test case within a suite, use
        <c>ts:run(Test,Suite,Case)</c></p>
      <p>To all these functions, you can also add a list of
        options. Please turn to the reference manual for the <c>ts</c>
        module to see the valid options to use.
        </p>
      <p>The function <c>ts:help()</c> displays some simple help for
        the functions in <c>ts</c>. Use this for quick reference.
        </p>
      <p>LOG FILES</p>
      <p>As the execution of the test suites go on, events are logged in
      four different ways:
      </p>
    <list type="bulleted">
      <item>Text to the operator's console.</item>
      <item>Suite related information is sent to the major log file.</item>
      <item>Case related information is sent to the minor log file.</item>
      <item>The HTML log file gets updated with test results.</item>
    </list>
    <p>Typically the operator, who may run hundreds or thousands of
      test cases, doesn't want to fill the screen with details
      about/from the specific test cases. By default, the operator will
      only see:
      </p>
    <list type="bulleted">
      <item>A confirmation that the test has started.
      </item>
      <item>A small note about each failed test case.
      </item>
      <item>A summary of all the run test cases.
      </item>
      <item>A confirmation that the test run is complete
      </item>
      <item>Some special information like error reports and progress
       reports, printouts written with erlang:display/1 or io:format/3
       specifically addressed to somewhere other than
      <c>standard_io</c>.</item>
    </list>
    <p>This is enough for the operator to know, and if he wants to dig
      in deeper into a specific test case result, he can do so by
      following the links in the HTML presentation to take a look in the
      major or minor log files.
      </p>
    <p>A detailed report of the entire test suite is stored in the
      major logfile, the exact reason for failure, time spent etc.
      </p>
    <p>The HTML log file is a summary of the major log file, but gives
      a much better overview of the test run. It also has links to every
      test case's log file for quick viewing with a HTML browser.
      </p>
    <p>The minor log file contain full details of every single test
      case, each one in a separate file. This way the files should be
      easy to compare with previous test runs, even if the set of test
      cases change.
      </p>
    <p>Which information that goes where is user configurable via the
      test server controller. Three threshold values determine what
      comes out on screen, and in the major or minor log files. The
      contents that goes to the HTML log file is fixed, and cannot be
      altered.
      </p>

  </description>
  <funcs>
    <func>
      <name>install() -> ok | {error, Reason}</name>
      <name>install(TargetSystem) -> ok | {error, Reason}</name>
      <name>install(Opts) -> ok | {error, Reason}</name>
      <name>install(TargetSystem,Opts) -> ok | {error, Reason}</name>
      <fsummary>Installs the Test Server Framework</fsummary>
      <type>
        <v>TargetSystem = {Architecture, TargetHost}</v>
        <v>Architecture = atom() or string()</v>
        <d>e.g. "ose" or "vxworks_ppc603"</d>
        <v>TargetHost = atom() or string()</v>
        <d>The name of the target host</d>
        <v>Opts = list()</v>
      </type>
      <desc>
        <p>Installs and configures the Test Server Framework for
          running test suites. If a remote host is to be used, the
          <c>TargetSystem</c> argument must be given so that "cross
		  installation" can be done. Installation is required for
		  any of the functions in <c>ts</c> to work.
          </p>
        <p>Opts may be one or more of
          </p>
        <list>
          <item><c>{longnames, Bool}</c><br></br>
          Use fully qualified hostnames for test_server and
           slave nodes. Bool is <c>true</c> or <c>false</c> (default).
          </item>
          <item><c>{verbose, Level}</c><br></br>
          Verbosity level for test server output, set to 0, 1 or
           2, where 0 is quiet(default).
          </item>
          <item><c>{hosts, Hosts}</c><br></br>
          This is a list of available hosts on which to start
           slave nodes. It is used when the <c>{remote, true}</c>
           option is given to the <c>test_server:start_node/3</c>
           function. Also, if <c>{require_nodenames, Num}</c> is
           contained in a test specification file, the generated
           nodenames will be spread over all hosts given in this
          <c>Hosts</c> list. The hostnames are given as atoms or
           strings.
          </item>
          <item><c>{slavetargets, SlaveTarges}</c><br></br>
          For VxWorks only. This is a list of
           available hosts where slave nodes can be started. This is
           necessary because only one node can run per host in the
           VxWorks environment. This is not the same as
          <c>{hosts, Hosts}</c> because it is used for all slave nodes
           - not only the ones started with <c>{remote, true}</c>. The
           hostnames are given as atoms or strings.
          </item>
          <item><c>{crossroot, TargetErlRoot}</c><br></br>
          Erlang root directory on target host
                    <br></br>
This option is mandatory for remote targets
          </item>
          <item><c>{master, {MasterHost, MasterCookie}}</c><br></br>
          If target is remote and the target node is started as
           a slave node, this option
           indicates which master and cookie to use. The given master
           will also be used as master for slave nodes started with
          <c>test_server:start_node/3</c>. It is expected that the
          <c>erl_boot_server</c> is started on the master node before
           the test is run. If this option is not given, the test
           server controller node is used as master and the
          <c>erl_boot_server</c> is automatically started.
          </item>
          <item><c>{erl_start_args, ArgString}</c><br></br>
          Additional arguments to be used when starting the test
           server controller node. <c>ArgString</c> will be appended to
           the command line when starting the erlang node. Note that
           this will only affect the startup of the <em>controller node</em>, 
	   i.e. not the target node or any slave nodes
           startet from a test case.
          </item>
          <item><c>{ipv6_hosts, HostList}</c><br></br>
          This option will be inserted in the
          <c>Config</c> parameter for each test case. <c>HostList</c>
           is a list of hosts supporting IPv6.
          </item>
        </list>
      </desc>
    </func>
    <func>
      <name>help() -> ok</name>
      <fsummary>Presents simple help on the functions in <c>ts</c></fsummary>
      <desc>
        <p>Presents simple help on the functions in <c>ts</c>. Useful
          for quick reference.</p>
      </desc>
    </func>
    <func>
      <name>tests() -> Tests</name>
      <fsummary>Returns the list of available tests</fsummary>
      <desc>
        <p>Returns the list of available tests. This is actually just
          a list of all test specification files found by looking up
          "../*_test/*.spec".
          </p>
        <p>In each ../Name_test/ directory there should be one test
          specification file named Name.spec.</p>
      </desc>
    </func>
    <func>
      <name>run() -> ok | {error, Reason}</name>
      <name>run([all_tests|Opts])</name>
      <name>run(Specs)</name>
      <name>run(Specs, Opts)</name>
      <name>run(Spec, Module)</name>
      <name>run(Spec, Module, Opts)</name>
      <name>run(Spec, Module, Case)</name>
      <name>run(Spec, Module, Case, Opts)</name>
      <fsummary>Runs (specified) test suite(s)</fsummary>
      <type>
        <v>Specs = Spec | [Spec]</v>
        <v>Spec = atom()</v>
        <v>Module = atom()</v>
        <v>Case = atom()</v>
        <v>Opts = [Opt]</v>
        <v>Opt = batch | verbose | {verbose, Level} | {vars, Vars} |  keep_topcase | cover | cover_details |{cover,CoverFile} |  {cover_details,CoverFile} | {trace, TraceSpec}</v>
        <v>Level = integer(); 0 means silent</v>
        <v>Vars = list() of key-value tuples</v>
        <v>CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.</v>
        <v>Reason = term()</v>
      </type>
      <desc>
        <p>This function runs test suite(s)/case(s). To be able to run
          any tests, ts:install must first be called to create the
          <c>variables</c> file needed.  To run a whole test specification,
          only specify the name of the test specification, and all test
          suite modules belonging to that test spec will be run.  To run
          a single module in a test specification, use the Module
          argument to specify the name of the module to run and all test
          cases in that module will be run, and to run a specified test
          case, specify the name of the test case using the Case
          argument. If called with no argument, all test specifications
          available will be run. Use ts:tests/0 to see the available
          test specifications.
          </p>
        <p>If the <c>batch</c> option is not given, a new xterm is
          started (unix) when <c>ts:run</c> is called.
          </p>
        <p>The <c>verbose</c> option sets the verbosity level for test
          server output. This has the same effect as if given to
          <c>ts:install/1/2</c></p>
        <p>The <c>vars</c> option can be used for adding configuration
          variables that are not in the <c>variables</c> file generated
          during installation. Can be any of the <c>Opts</c> valid for
          <c>ts:install/1/2</c>.
          </p>
        <p>The <c>keep_topcase</c> option forces <c>ts</c> to keep the
          topcase in your test specification file as is. This option can
          only be used if you don't give the <c>Module</c> or
          <c>Case</c> parameters to <c>ts:run</c>. The
          <c>keep_topcase</c> option is necessary if your topcase
          contains anything other than <c><![CDATA[{dir,"../<Name>_test"}]]></c>. If
          the option is not used, <c>ts</c> will modify your topcase.
          </p>
        <p>The <c>cover</c> and <c>cover_details</c> options indicates
          that the test shall be run with code coverage
          analysis. <c>cover_details</c> means that analysis shall be
          done on the most detailed level. If the test is run with a
          remote target, this option creates a list of uncovered lines
          in each cover compiled module. If the test is run with a local
          target, each cover compiled module will be analysed with
          <c>cover:analyse_to_file/1</c>. The <c>cover</c> options will
          only create an overview of all cover compiled modules with the
          number of covered and not covered lines.
          </p>
        <p>The <c>CoverFile</c> which can be given with the
          <c>cover</c> and <c>cover_details</c> options must be the
          filename of a file listing modules to be excluded from or
          included in the cover compilation. By default, <c>ts</c>
          believes that <c>Spec</c> is the name of an OTP application
          and that all modules in this application shall be cover
          compiled. The <c>CoverFile</c> can exclude modules that belong
          to the application and add modules that don't belong to the
          application. The file can have the following entries:</p>
        <code type="none">
{exclude, all | ExcludeModuleList}.
{include, IncludeModuleList}.        </code>
        <p>Note that each line must end with a full
          stop. <c>ExcludeModuleList</c> and <c>IncludeModuleList</c>
          are lists of atoms, where each atom is a module name.
          </p>
        <p>If the <c>cover</c> or <c>cover_details</c> options are
          given on their own, the directory <c><![CDATA[../<Spec>_test]]></c> is
          searched for a <c>CoverFile</c> named <c><![CDATA[<Spec>.cover]]></c>. If
          this file is not found, <c>Spec</c> is assumed to be the name
          of an OTP application, and all modules in the <c>ebin</c>
          directory for the application are cover compiled. The
          <c>ebin</c> directory is found by adding <c>ebin</c> to
          <c>code:lib_dir(Spec)</c>.
          </p>
        <p>The same cover compiled code will be loaded on all slave or
          peer nodes started with <c>test_server:start_node/3</c>. The
          exception is nodes that run an old version of OTP. If the loading 
          fails, the node will simply not be a part of the coverage
          analysis. Note that slave and peer nodes must be stopped with
          <c>test_server:stop_node/1</c> for the node to be part of the
          coverage analysis, else the test server will not be able to
          fetch coverage data from the node.
          </p>
        <p>The <c>trace</c> option is used to turn on call trace on
          target and on slave or peer nodes started with
          <c>test_server:start_node/3</c>. <c>TraceSpec</c> can be the
          name of a trace information file, or a list of elements like
          the ones in a trace information file. Please turn to the
          reference manual for <c>test_server_ctrl:trc/1</c> for details
          about the trace information file.
          </p>
      </desc>
    </func>
    <func>
      <name>cross_cover_analyse(Level) -> ok</name>
      <name>cross_cover_analyse([Level]) -> ok</name>
      <fsummary>Analyse cover data collected from all tests</fsummary>
      <desc>
        <p>Analyse cover data collected from all tests.
          </p>
        <p>See test_server_ctrl:cross_cover_analyse/2
          </p>
      </desc>
    </func>
    <func>
      <name>r() -> ok</name>
      <name>r(Opts) -> ok</name>
      <name>r(SpecOrSuite) -> ok</name>
      <name>r(SpecOrSuite,Opts) -> ok</name>
      <name>r(Suite,Case) -> ok</name>
      <name>r(Suite,Case,Opts) -> ok</name>
      <fsummary>Run test suite or test case without <c>ts</c>installed</fsummary>
      <type>
        <v>SpecOrSuite = Spec | Suite</v>
        <v>Spec = string()</v>
        <d>"Name.spec" or "Name.spec.OsType", where OsType is vxworks</d>
        <v>Suite = atom()</v>
        <v>Case = atom()</v>
        <v>Opts = [Opt]</v>
        <v>Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}</v>
        <v>Cover = cover | cover_details</v>
        <v>AppOrCoverFile = Application | CoverFile</v>
        <v>Application = atom()</v>
        <d>OTP application to cover compile</d>
        <v>CoverFile = string()</v>
        <d>Name of file listing modules to exclude from or include in cover compilation</d>
      </type>
      <desc>
        <p>This function can be used to run a test suites or test
          cases directly, without any of the additional features added
          by the test server framework. It is simply a wrapper function
          for the <c>add_dir</c>, <c>add_spec</c>, <c>add_module</c> and 
          <c>add_case</c> functions in <c>test_server_ctrl</c>:
          </p>
        <p><c>r() -> add_dir(".")</c>          <br></br>
<c>r(Spec) -> add_spec(Spec)</c>          <br></br>
<c>r(Suite) -> add_module(Suite)</c>          <br></br>
<c>r(Suite,Case) -> add_case(Suite,Case)</c></p>
        <p>To use this function, it is required that the test suite is
          compiled and in the code path of the node where the function
          is called. The function can be used without having <c>ts</c>
          installed.
          </p>
        <p>For information about the <c>cover</c> and
          <c>cover_details</c> options, see <c>test_server_ctrl:cover/2/3</c>.</p>
      </desc>
    </func>
    <func>
      <name>estone() -> ok | {error, Reason}</name>
      <name>estone(Opts) -> ok</name>
      <fsummary>Runs the EStone test</fsummary>
      <desc>
        <p>This function runs the EStone test. It is a shortcut for
          running the test suite <c>estone_SUITE</c> in the
          <c>kernel</c> application.
          </p>
        <p><c>Opts</c> is the same as the <c>Opts</c> argument for the
          <c>ts:run</c> functions.</p>
      </desc>
    </func>
  </funcs>

  <section>
    <title>Makfile.src in Data Directory</title>
    <p>If a data directory contains code which must be compiled before
      the test suite is run, a makefile source called
      <c>Makefile.src</c> can be placed in the data directory. This file
      will be converted to a valid makefile by <c>ts:run/0/1/2/3/4</c>.
      </p>
    <p>The reason for generating the makefile is that you can use
      variables from the <c>variables</c> file which was generated by
      <c>ts:install/0/1/2</c>. All occurrences of <c>@Key@</c> in
      <c>Makefile.src</c> is substituted by the <c>Value</c> from
      <c>{Key,Value}</c> found in the <c>variables</c> file. Example:
      </p>
    <p>Cut from <c>variables</c>:</p>
    <code type="none">
    ...
    {'EMULATOR',"beam"}.
    {'CFLAGS',"-g -O2"}.
    {'LD',"$(CC) $(CFLAGS)"}.
    {'CC',"gcc"}.
    ...
    </code>
    <p><c>Makefile.src</c> for compiling erlang code could look
      something like this:</p>
    <code type="none">
    EFLAGS=+debug_info
    
    all: ordsets1.@EMULATOR@
    
    ordsets1.@EMULATOR@: ordsets1.erl
           erlc $(EFLAGS) ordsets1.erl
    </code>
    <p><c>Makefile.src</c> for compiling c code could look
      something like this:</p>
    <code type="none">
    CC = @CC@
    LD = @LD@
    CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@
    CROSSLDFLAGS = @CROSSLDFLAGS@
    
    PROGS = nfs_check@exe@
    
    all: $(PROGS)
    
    nfs_check@exe@: nfs_check@obj@
           $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@

    nfs_check@obj@: nfs_check.c
           $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c
    </code>
  </section>
</erlref>