The R13B03 release.OTP_R13B03

1 files changed, 592 insertions, 0 deletions

diff --git a/lib/test_server/doc/src/ts.xml b/lib/test_server/doc/src/ts.xml
new file mode 100644
index 0000000000..0f91d3eea2
--- /dev/null
+++ b/lib/test_server/doc/src/ts.xml
@@ -0,0 +1,592 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE erlref SYSTEM "erlref.dtd">
+
+<erlref>
+  <header>
+    <copyright>
+      <year>2007</year>
+      <year>2008</year>
+      <holder>Ericsson AB, All Rights Reserved</holder>
+    </copyright>
+    <legalnotice>
+  The contents of this file are subject to the Erlang Public License,
+  Version 1.1, (the "License"); you may not use this file except in
+  compliance with the License. You should have received a copy of the
+  Erlang Public License along with this software. If not, it can be
+  retrieved online at http://www.erlang.org/.
+
+  Software distributed under the License is distributed on an "AS IS"
+  basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+  the License for the specific language governing rights and limitations
+  under the License.
+
+  The Initial Developer of the Original Code is Ericsson AB.
+    </legalnotice>
+
+    <title>The OTP Test Server Framework</title>
+    <prepared>Mattias Nilsson</prepared>
+    <responsible></responsible>
+    <docno></docno>
+    <approved></approved>
+    <checked></checked>
+    <date></date>
+    <rev></rev>
+    <file>ts.xml</file>
+  </header>
+  <module>ts</module>
+  <modulesummary>Test Server Framework for testing OTP</modulesummary>
+  <description>
+    <p>This is a framework for testing OTP. The <c>ts</c> module
+      implements the interface to all the functionality in the
+      framework.
+      </p>
+    <p>The framework is built on top of the Test Server Controller,
+      <c>test_server_ctrl</c>, and provides a high level operator
+      interface. The main features added by the framework are:
+      </p>
+    <list type="bulleted">
+      <item>Automatic compilation of test suites and data directories
+      </item>
+      <item>Collection of files in central directories and creation of
+       additional HTML pages for better overview.
+      </item>
+      <item>Single command interface for running all available tests
+      </item>
+      <item>Spawns a new node with correct parameters before starting
+       the test server
+      </item>
+      <item>Atomatically creates the parameter file needed when
+       running tests on remote target
+      </item>
+    </list>
+    <p>More information about the Test Server Framework and how to run
+      test cases can be found in the Test Server User's Guide.
+      </p>
+    <p>For writing you own test server framework, please turn to the
+      reference manual for the Test Server Controller and chapter named
+      "Writing your own test server framework" in the Test Server User's
+      Guide.
+      </p>
+      <p>SETUP</p>
+    <p>To be able to run <c>ts</c>, you must first `install'
+        <c>ts</c> for the current environment. This is done by calling
+        <c>ts:install/0/1/2</c>. A file called `variables' is created
+        and used by <c>ts</c> when running test suites. It is not
+        recommended to edit this file, but it is possible to alter if
+        <c>ts</c> gets the wrong idea about your environment.
+        </p>
+    <p><c>ts:install/0</c> is used if the target platform is the
+        same as the controller host, i.e. if you run on "local target"
+        and no options are needed. Then running <c>ts:install/0</c><c>ts</c> 
+	will run an autoconf script for your current
+        environment and set up the necessary variables needed by the
+        test suites.
+        </p>
+    <p><c>ts:install/1</c> or <c>ts:install/2</c> is used if the
+        target platform is different from the controller host, i.e. if
+        you run on "remote target" or if special options are required
+        for your system. VxWorks is currently supported
+        as remote target platform.
+        </p>
+      <p>See the reference manual for detailed information about
+        <c>ts:install/0/1/2</c>.
+        </p>
+    <p>Some of the common variables in the 'variables' file are
+        described below. Do not make any assumptions as of what is found
+        in this file, as it may change at any time.
+        </p>
+    <list>
+        <item><c>longnames</c><br></br>
+        Set to true if the system is using fully qualified
+         nodenames.
+        </item>
+        <item><c>platform_id</c><br></br>
+        This is the currently installed platform identification
+         string.
+        </item>
+        <item><c>platform_filename</c><br></br>
+        This is the name used to create the final save directory
+         for test runs.
+        </item>
+        <item><c>platform_label</c><br></br>
+        This is the string presented in the generated test
+         results index page.
+        </item>
+        <item><c>rsh_name</c><br></br>
+        This is the rsh program to use when starting slave or
+         peer nodes on a remote host.
+        </item>
+        <item><c>erl_flags</c><br></br>
+        Compile time flags used when compiling test suites.
+        </item>
+        <item><c>erl_release</c><br></br>
+        The Erlang/OTP release being tested.
+        </item>
+        <item><c>'EMULATOR'</c><br></br>
+        The emulator being tested (e.g. beam)
+        </item>
+        <item><c>'CPU'</c><br></br>
+        The CPU in the machine running the tests, e.g. sparc.
+        </item>
+        <item><c>target_host</c><br></br>
+        The target host name
+        </item>
+        <item><c>os</c><br></br>
+        The target operating system, e.g. solaris2.8
+        </item>
+        <item><c>target</c><br></br>
+        The current target platform, e.g. sparc-sun-solaris2.8
+        </item>
+      </list>
+      <p>RUNNING TESTS</p>
+      <p>After installing <c>ts</c>, you can run your test with the
+        <c>ts:run/0/1/2/3/4</c> functions. These functions, however,
+        require a special directory structure to be able to find your
+        test suites. Both the test server and all tests must be located
+        under your $TESTROOT directory. The test server implementation
+        shall be located in the directory <c>$TESTROOT/test_server</c>
+        and for each application there must be a directory named
+        <c><![CDATA[$TESTROOT/<application>_test]]></c> containing the .spec file
+        and all test suites and data directories for the
+        application. Note that there shall only be one .spec file for
+        each application.
+        </p>
+      <p><c>$TESTROOT/test_server</c> must be the current directory
+        when calling the <c>ts:run/*</c> function.
+        </p>
+      <p>All available tests can be found with <c>ts:tests()</c>. This
+        will list all applications for which a test specification file
+        <c><![CDATA[../<application>_test/<application>.spec]]></c> can be found.
+        </p>
+      <p>To run all these tests, use <c>ts:run()</c>.
+        </p>
+      <p>To run one or some of the tests, use <c>ts:run(Tests)</c>,
+        where <c>Tests</c> is the name of the application you want to
+        test, or a list of such names.
+        </p>
+      <p>To run one test suite within a test, use
+        <c>ts:run(Test,Suite)</c>.
+        </p>
+      <p>To run one test case within a suite, use
+        <c>ts:run(Test,Suite,Case)</c></p>
+      <p>To all these functions, you can also add a list of
+        options. Please turn to the reference manual for the <c>ts</c>
+        module to see the valid options to use.
+        </p>
+      <p>The function <c>ts:help()</c> displays some simple help for
+        the functions in <c>ts</c>. Use this for quick reference.
+        </p>
+      <p>LOG FILES</p>
+      <p>As the execution of the test suites go on, events are logged in
+      four different ways:
+      </p>
+    <list type="bulleted">
+      <item>Text to the operator's console.</item>
+      <item>Suite related information is sent to the major log file.</item>
+      <item>Case related information is sent to the minor log file.</item>
+      <item>The HTML log file gets updated with test results.</item>
+    </list>
+    <p>Typically the operator, who may run hundreds or thousands of
+      test cases, doesn't want to fill the screen with details
+      about/from the specific test cases. By default, the operator will
+      only see:
+      </p>
+    <list type="bulleted">
+      <item>A confirmation that the test has started.
+      </item>
+      <item>A small note about each failed test case.
+      </item>
+      <item>A summary of all the run test cases.
+      </item>
+      <item>A confirmation that the test run is complete
+      </item>
+      <item>Some special information like error reports and progress
+       reports, printouts written with erlang:display/1 or io:format/3
+       specifically addressed to somewhere other than
+      <c>standard_io</c>.</item>
+    </list>
+    <p>This is enough for the operator to know, and if he wants to dig
+      in deeper into a specific test case result, he can do so by
+      following the links in the HTML presentation to take a look in the
+      major or minor log files.
+      </p>
+    <p>A detailed report of the entire test suite is stored in the
+      major logfile, the exact reason for failure, time spent etc.
+      </p>
+    <p>The HTML log file is a summary of the major log file, but gives
+      a much better overview of the test run. It also has links to every
+      test case's log file for quick viewing with a HTML browser.
+      </p>
+    <p>The minor log file contain full details of every single test
+      case, each one in a separate file. This way the files should be
+      easy to compare with previous test runs, even if the set of test
+      cases change.
+      </p>
+    <p>Which information that goes where is user configurable via the
+      test server controller. Three threshold values determine what
+      comes out on screen, and in the major or minor log files. The
+      contents that goes to the HTML log file is fixed, and cannot be
+      altered.
+      </p>
+
+  </description>
+  <funcs>
+    <func>
+      <name>install() -> ok | {error, Reason}</name>
+      <name>install(TargetSystem) -> ok | {error, Reason}</name>
+      <name>install(Opts) -> ok | {error, Reason}</name>
+      <name>install(TargetSystem,Opts) -> ok | {error, Reason}</name>
+      <fsummary>Installs the Test Server Framework</fsummary>
+      <type>
+        <v>TargetSystem = {Architecture, TargetHost}</v>
+        <v>Architecture = atom() or string()</v>
+        <d>e.g. "ose" or "vxworks_ppc603"</d>
+        <v>TargetHost = atom() or string()</v>
+        <d>The name of the target host</d>
+        <v>Opts = list()</v>
+      </type>
+      <desc>
+        <p>Installs and configures the Test Server Framework for
+          running test suites. If a remote host is to be used, the
+          <c>TargetSystem</c> argument must be given so that "cross
+          installation" can be done. This should be used for testing on
+          VxWorks or OSE/Delta. Installation is required for any of the
+          functions in <c>ts</c> to work.
+          </p>
+        <p>Opts may be one or more of
+          </p>
+        <list>
+          <item><c>{longnames, Bool}</c><br></br>
+          Use fully qualified hostnames for test_server and
+           slave nodes. Bool is <c>true</c> or <c>false</c> (default).
+          </item>
+          <item><c>{verbose, Level}</c><br></br>
+          Verbosity level for test server output, set to 0, 1 or
+           2, where 0 is quiet(default).
+          </item>
+          <item><c>{hosts, Hosts}</c><br></br>
+          This is a list of available hosts on which to start
+           slave nodes. It is used when the <c>{remote, true}</c>
+           option is given to the <c>test_server:start_node/3</c>
+           function. Also, if <c>{require_nodenames, Num}</c> is
+           contained in a test specification file, the generated
+           nodenames will be spread over all hosts given in this
+          <c>Hosts</c> list. The hostnames are given as atoms or
+           strings.
+          </item>
+          <item><c>{slavetargets, SlaveTarges}</c><br></br>
+          For VxWorks and OSE/Delta only. This is a list of
+           available hosts where slave nodes can be started. This is
+           necessary because only one node can run per host in the
+           VxWorks environment. This is not the same as
+          <c>{hosts, Hosts}</c> because it is used for all slave nodes
+           - not only the ones started with <c>{remote, true}</c>. The
+           hostnames are given as atoms or strings.
+          </item>
+          <item><c>{crossroot, TargetErlRoot}</c><br></br>
+          Erlang root directory on target host
+                    <br></br>
+This option is mandatory for remote targets
+          </item>
+          <item><c>{master, {MasterHost, MasterCookie}}</c><br></br>
+          If target is remote and the target node is started as
+           a slave node, this option
+           indicates which master and cookie to use. The given master
+           will also be used as master for slave nodes started with
+          <c>test_server:start_node/3</c>. It is expected that the
+          <c>erl_boot_server</c> is started on the master node before
+           the test is run. If this option is not given, the test
+           server controller node is used as master and the
+          <c>erl_boot_server</c> is automatically started.
+          </item>
+          <item><c>{erl_start_args, ArgString}</c><br></br>
+          Additional arguments to be used when starting the test
+           server controller node. <c>ArgString</c> will be appended to
+           the command line when starting the erlang node. Note that
+           this will only affect the startup of the <em>controller node</em>, 
+	   i.e. not the target node or any slave nodes
+           startet from a test case.
+          </item>
+          <item><c>{ipv6_hosts, HostList}</c><br></br>
+          This option will be inserted in the
+          <c>Config</c> parameter for each test case. <c>HostList</c>
+           is a list of hosts supporting IPv6.
+          </item>
+        </list>
+      </desc>
+    </func>
+    <func>
+      <name>help() -> ok</name>
+      <fsummary>Presents simple help on the functions in <c>ts</c></fsummary>
+      <desc>
+        <p>Presents simple help on the functions in <c>ts</c>. Useful
+          for quick reference.</p>
+      </desc>
+    </func>
+    <func>
+      <name>tests() -> Tests</name>
+      <fsummary>Returns the list of available tests</fsummary>
+      <desc>
+        <p>Returns the list of available tests. This is actually just
+          a list of all test specification files found by looking up
+          "../*_test/*.spec".
+          </p>
+        <p>In each ../Name_test/ directory there should be one test
+          specification file named Name.spec.</p>
+      </desc>
+    </func>
+    <func>
+      <name>run() -> ok | {error, Reason}</name>
+      <name>run([all_tests|Opts])</name>
+      <name>run(Specs)</name>
+      <name>run(Specs, Opts)</name>
+      <name>run(Spec, Module)</name>
+      <name>run(Spec, Module, Opts)</name>
+      <name>run(Spec, Module, Case)</name>
+      <name>run(Spec, Module, Case, Opts)</name>
+      <fsummary>Runs (specified) test suite(s)</fsummary>
+      <type>
+        <v>Specs = Spec | [Spec]</v>
+        <v>Spec = atom()</v>
+        <v>Module = atom()</v>
+        <v>Case = atom()</v>
+        <v>Opts = [Opt]</v>
+        <v>Opt = batch | verbose | {verbose, Level} | {vars, Vars} |  keep_topcase | cover | cover_details |{cover,CoverFile} |  {cover_details,CoverFile} | {trace, TraceSpec}</v>
+        <v>Level = integer(); 0 means silent</v>
+        <v>Vars = list() of key-value tuples</v>
+        <v>CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.</v>
+        <v>Reason = term()</v>
+      </type>
+      <desc>
+        <p>This function runs test suite(s)/case(s). To be able to run
+          any tests, ts:install must first be called to create the
+          <c>variables</c> file needed.  To run a whole test specification,
+          only specify the name of the test specification, and all test
+          suite modules belonging to that test spec will be run.  To run
+          a single module in a test specification, use the Module
+          argument to specify the name of the module to run and all test
+          cases in that module will be run, and to run a specified test
+          case, specify the name of the test case using the Case
+          argument. If called with no argument, all test specifications
+          available will be run. Use ts:tests/0 to see the available
+          test specifications.
+          </p>
+        <p>If the <c>batch</c> option is not given, a new xterm is
+          started (unix) when <c>ts:run</c> is called.
+          </p>
+        <p>The <c>verbose</c> option sets the verbosity level for test
+          server output. This has the same effect as if given to
+          <c>ts:install/1/2</c></p>
+        <p>The <c>vars</c> option can be used for adding configuration
+          variables that are not in the <c>variables</c> file generated
+          during installation. Can be any of the <c>Opts</c> valid for
+          <c>ts:install/1/2</c>.
+          </p>
+        <p>The <c>keep_topcase</c> option forces <c>ts</c> to keep the
+          topcase in your test specification file as is. This option can
+          only be used if you don't give the <c>Module</c> or
+          <c>Case</c> parameters to <c>ts:run</c>. The
+          <c>keep_topcase</c> option is necessary if your topcase
+          contains anything other than <c><![CDATA[{dir,"../<Name>_test"}]]></c>. If
+          the option is not used, <c>ts</c> will modify your topcase.
+          </p>
+        <p>The <c>cover</c> and <c>cover_details</c> options indicates
+          that the test shall be run with code coverage
+          analysis. <c>cover_details</c> means that analysis shall be
+          done on the most detailed level. If the test is run with a
+          remote target, this option creates a list of uncovered lines
+          in each cover compiled module. If the test is run with a local
+          target, each cover compiled module will be analysed with
+          <c>cover:analyse_to_file/1</c>. The <c>cover</c> options will
+          only create an overview of all cover compiled modules with the
+          number of covered and not covered lines.
+          </p>
+        <p>The <c>CoverFile</c> which can be given with the
+          <c>cover</c> and <c>cover_details</c> options must be the
+          filename of a file listing modules to be excluded from or
+          included in the cover compilation. By default, <c>ts</c>
+          believes that <c>Spec</c> is the name of an OTP application
+          and that all modules in this application shall be cover
+          compiled. The <c>CoverFile</c> can exclude modules that belong
+          to the application and add modules that don't belong to the
+          application. The file can have the following entries:</p>
+        <code type="none">
+{exclude, all | ExcludeModuleList}.
+{include, IncludeModuleList}.        </code>
+        <p>Note that each line must end with a full
+          stop. <c>ExcludeModuleList</c> and <c>IncludeModuleList</c>
+          are lists of atoms, where each atom is a module name.
+          </p>
+        <p>If the <c>cover</c> or <c>cover_details</c> options are
+          given on their own, the directory <c><![CDATA[../<Spec>_test]]></c> is
+          searched for a <c>CoverFile</c> named <c><![CDATA[<Spec>.cover]]></c>. If
+          this file is not found, <c>Spec</c> is assumed to be the name
+          of an OTP application, and all modules in the <c>ebin</c>
+          directory for the application are cover compiled. The
+          <c>ebin</c> directory is found by adding <c>ebin</c> to
+          <c>code:lib_dir(Spec)</c>.
+          </p>
+        <p>The same cover compiled code will be loaded on all slave or
+          peer nodes started with <c>test_server:start_node/3</c>. The
+          exception is nodes that run an old version of OTP. If the loading 
+          fails, the node will simply not be a part of the coverage
+          analysis. Note that slave and peer nodes must be stopped with
+          <c>test_server:stop_node/1</c> for the node to be part of the
+          coverage analysis, else the test server will not be able to
+          fetch coverage data from the node.
+          </p>
+        <p>The <c>trace</c> option is used to turn on call trace on
+          target and on slave or peer nodes started with
+          <c>test_server:start_node/3</c>. <c>TraceSpec</c> can be the
+          name of a trace information file, or a list of elements like
+          the ones in a trace information file. Please turn to the
+          reference manual for <c>test_server_ctrl:trc/1</c> for details
+          about the trace information file.
+          </p>
+      </desc>
+    </func>
+    <func>
+      <name>cross_cover_analyse(Level) -> ok</name>
+      <name>cross_cover_analyse([Level]) -> ok</name>
+      <fsummary>Analyse cover data collected from all tests</fsummary>
+      <desc>
+        <p>Analyse cover data collected from all tests.
+          </p>
+        <p>See test_server_ctrl:cross_cover_analyse/1
+          </p>
+      </desc>
+    </func>
+    <func>
+      <name>r() -> ok</name>
+      <name>r(Opts) -> ok</name>
+      <name>r(SpecOrSuite) -> ok</name>
+      <name>r(SpecOrSuite,Opts) -> ok</name>
+      <name>r(Suite,Case) -> ok</name>
+      <name>r(Suite,Case,Opts) -> ok</name>
+      <fsummary>Run test suite or test case without <c>ts</c>installed</fsummary>
+      <type>
+        <v>SpecOrSuite = Spec | Suite</v>
+        <v>Spec = string()</v>
+        <d>"Name.spec" or "Name.spec.OsType", where OsType is vxworks</d>
+        <v>Suite = atom()</v>
+        <v>Case = atom()</v>
+        <v>Opts = [Opt]</v>
+        <v>Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}</v>
+        <v>Cover = cover | cover_details</v>
+        <v>AppOrCoverFile = Application | CoverFile</v>
+        <v>Application = atom()</v>
+        <d>OTP application to cover compile</d>
+        <v>CoverFile = string()</v>
+        <d>Name of file listing modules to exclude from or include in cover compilation</d>
+      </type>
+      <desc>
+        <p>This function can be used to run a test suites or test
+          cases directly, without any of the additional features added
+          by the test server framework. It is simply a wrapper function
+          for the <c>add_dir</c>, <c>add_spec</c>, <c>add_module</c> and 
+          <c>add_case</c> functions in <c>test_server_ctrl</c>:
+          </p>
+        <p><c>r() -> add_dir(".")</c>          <br></br>
+<c>r(Spec) -> add_spec(Spec)</c>          <br></br>
+<c>r(Suite) -> add_module(Suite)</c>          <br></br>
+<c>r(Suite,Case) -> add_case(Suite,Case)</c></p>
+        <p>To use this function, it is required that the test suite is
+          compiled and in the code path of the node where the function
+          is called. The function can be used without having <c>ts</c>
+          installed.
+          </p>
+        <p>For information about the <c>cover</c> and
+          <c>cover_details</c> options, see <c>test_server_ctrl:cover/2/3</c>.</p>
+      </desc>
+    </func>
+    <func>
+      <name>index() -> ok | {error, Reason}</name>
+      <fsummary>Updates local index page</fsummary>
+      <type>
+        <v>Reason = term()</v>
+      </type>
+      <desc>
+        <p>This function updates the local index page. This can be
+          useful if a previous test run was not completed and the index
+          is incomplete.</p>
+      </desc>
+    </func>
+    <func>
+      <name>clean() -> ok</name>
+      <name>clean(all) -> ok</name>
+      <fsummary>Cleans up the log directories created when running tests. </fsummary>
+      <desc>
+        <p>This function cleans up log directories created when
+          running test cases. <c>clean/0</c> cleans up all but the last
+          run of each application.  <c>clean/1</c> cleans up all test
+          runs found.</p>
+      </desc>
+    </func>
+    <func>
+      <name>estone() -> ok | {error, Reason}</name>
+      <name>estone(Opts) -> ok</name>
+      <fsummary>Runs the EStone test</fsummary>
+      <desc>
+        <p>This function runs the EStone test. It is a shortcut for
+          running the test suite <c>estone_SUITE</c> in the
+          <c>kernel</c> application.
+          </p>
+        <p><c>Opts</c> is the same as the <c>Opts</c> argument for the
+          <c>ts:run</c> functions.</p>
+      </desc>
+    </func>
+  </funcs>
+
+  <section>
+    <title>Makfile.src in Data Directory</title>
+    <p>If a data directory contains code which must be compiled before
+      the test suite is run, a makefile source called
+      <c>Makefile.src</c> can be placed in the data directory. This file
+      will be converted to a valid makefile by <c>ts:run/0/1/2/3/4</c>.
+      </p>
+    <p>The reason for generating the makefile is that you can use
+      variables from the <c>variables</c> file which was generated by
+      <c>ts:install/0/1/2</c>. All occurrences of <c>@Key@</c> in
+      <c>Makefile.src</c> is substituted by the <c>Value</c> from
+      <c>{Key,Value}</c> found in the <c>variables</c> file. Example:
+      </p>
+    <p>Cut from <c>variables</c>:</p>
+    <code type="none">
+    ...
+    {'EMULATOR',"beam"}.
+    {'CFLAGS',"-g -O2"}.
+    {'LD',"$(CC) $(CFLAGS)"}.
+    {'CC',"gcc"}.
+    ...
+    </code>
+    <p><c>Makefile.src</c> for compiling erlang code could look
+      something like this:</p>
+    <code type="none">
+    EFLAGS=+debug_info
+    
+    all: ordsets1.@EMULATOR@
+    
+    ordsets1.@EMULATOR@: ordsets1.erl
+           erlc $(EFLAGS) ordsets1.erl
+    </code>
+    <p><c>Makefile.src</c> for compiling c code could look
+      something like this:</p>
+    <code type="none">
+    CC = @CC@
+    LD = @LD@
+    CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@
+    CROSSLDFLAGS = @CROSSLDFLAGS@
+    
+    PROGS = nfs_check@exe@
+    
+    all: $(PROGS)
+    
+    nfs_check@exe@: nfs_check@obj@
+           $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@
+
+    nfs_check@obj@: nfs_check.c
+           $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c
+    </code>
+  </section>
+</erlref>
+