From 84adefa331c4159d432d22840663c38f155cd4c1 Mon Sep 17 00:00:00 2001 From: Erlang/OTP Date: Fri, 20 Nov 2009 14:54:40 +0000 Subject: The R13B03 release. --- lib/test_server/doc/src/ts.xml | 592 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 592 insertions(+) create mode 100644 lib/test_server/doc/src/ts.xml (limited to 'lib/test_server/doc/src/ts.xml') diff --git a/lib/test_server/doc/src/ts.xml b/lib/test_server/doc/src/ts.xml new file mode 100644 index 0000000000..0f91d3eea2 --- /dev/null +++ b/lib/test_server/doc/src/ts.xml @@ -0,0 +1,592 @@ + + + + +
+ + 2007 + 2008 + Ericsson AB, All Rights Reserved + + + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + The Initial Developer of the Original Code is Ericsson AB. + + + The OTP Test Server Framework + Mattias Nilsson + + + + + + + ts.xml +
+ ts + Test Server Framework for testing OTP + +

This is a framework for testing OTP. The ts module + implements the interface to all the functionality in the + framework. +

+

The framework is built on top of the Test Server Controller, + test_server_ctrl, and provides a high level operator + interface. The main features added by the framework are: +

+ + Automatic compilation of test suites and data directories + + Collection of files in central directories and creation of + additional HTML pages for better overview. + + Single command interface for running all available tests + + Spawns a new node with correct parameters before starting + the test server + + Atomatically creates the parameter file needed when + running tests on remote target + + +

More information about the Test Server Framework and how to run + test cases can be found in the Test Server User's Guide. +

+

For writing you own test server framework, please turn to the + reference manual for the Test Server Controller and chapter named + "Writing your own test server framework" in the Test Server User's + Guide. +

+

SETUP

+

To be able to run ts, you must first `install' + ts for the current environment. This is done by calling + ts:install/0/1/2. A file called `variables' is created + and used by ts when running test suites. It is not + recommended to edit this file, but it is possible to alter if + ts gets the wrong idea about your environment. +

+

ts:install/0 is used if the target platform is the + same as the controller host, i.e. if you run on "local target" + and no options are needed. Then running ts:install/0ts + will run an autoconf script for your current + environment and set up the necessary variables needed by the + test suites. +

+

ts:install/1 or ts:install/2 is used if the + target platform is different from the controller host, i.e. if + you run on "remote target" or if special options are required + for your system. VxWorks is currently supported + as remote target platform. +

+

See the reference manual for detailed information about + ts:install/0/1/2. +

+

Some of the common variables in the 'variables' file are + described below. Do not make any assumptions as of what is found + in this file, as it may change at any time. +

+ + longnames

+ Set to true if the system is using fully qualified + nodenames. + + platform_id

+ This is the currently installed platform identification + string. + + platform_filename

+ This is the name used to create the final save directory + for test runs. + + platform_label

+ This is the string presented in the generated test + results index page. + + rsh_name

+ This is the rsh program to use when starting slave or + peer nodes on a remote host. + + erl_flags

+ Compile time flags used when compiling test suites. + + erl_release

+ The Erlang/OTP release being tested. + + 'EMULATOR'

+ The emulator being tested (e.g. beam) + + 'CPU'

+ The CPU in the machine running the tests, e.g. sparc. + + target_host

+ The target host name + + os

+ The target operating system, e.g. solaris2.8 + + target

+ The current target platform, e.g. sparc-sun-solaris2.8 + + +

RUNNING TESTS

+

After installing ts, you can run your test with the + ts:run/0/1/2/3/4 functions. These functions, however, + require a special directory structure to be able to find your + test suites. Both the test server and all tests must be located + under your $TESTROOT directory. The test server implementation + shall be located in the directory $TESTROOT/test_server + and for each application there must be a directory named + _test]]> containing the .spec file + and all test suites and data directories for the + application. Note that there shall only be one .spec file for + each application. +

+

$TESTROOT/test_server must be the current directory + when calling the ts:run/* function. +

+

All available tests can be found with ts:tests(). This + will list all applications for which a test specification file + _test/.spec]]> can be found. +

+

To run all these tests, use ts:run(). +

+

To run one or some of the tests, use ts:run(Tests), + where Tests is the name of the application you want to + test, or a list of such names. +

+

To run one test suite within a test, use + ts:run(Test,Suite). +

+

To run one test case within a suite, use + ts:run(Test,Suite,Case)

+

To all these functions, you can also add a list of + options. Please turn to the reference manual for the ts + module to see the valid options to use. +

+

The function ts:help() displays some simple help for + the functions in ts. Use this for quick reference. +

+

LOG FILES

+

As the execution of the test suites go on, events are logged in + four different ways: +

+ + Text to the operator's console. + Suite related information is sent to the major log file. + Case related information is sent to the minor log file. + The HTML log file gets updated with test results. + +

Typically the operator, who may run hundreds or thousands of + test cases, doesn't want to fill the screen with details + about/from the specific test cases. By default, the operator will + only see: +

+ + A confirmation that the test has started. + + A small note about each failed test case. + + A summary of all the run test cases. + + A confirmation that the test run is complete + + Some special information like error reports and progress + reports, printouts written with erlang:display/1 or io:format/3 + specifically addressed to somewhere other than + standard_io. + +

This is enough for the operator to know, and if he wants to dig + in deeper into a specific test case result, he can do so by + following the links in the HTML presentation to take a look in the + major or minor log files. +

+

A detailed report of the entire test suite is stored in the + major logfile, the exact reason for failure, time spent etc. +

+

The HTML log file is a summary of the major log file, but gives + a much better overview of the test run. It also has links to every + test case's log file for quick viewing with a HTML browser. +

+

The minor log file contain full details of every single test + case, each one in a separate file. This way the files should be + easy to compare with previous test runs, even if the set of test + cases change. +

+

Which information that goes where is user configurable via the + test server controller. Three threshold values determine what + comes out on screen, and in the major or minor log files. The + contents that goes to the HTML log file is fixed, and cannot be + altered. +

+ + + + + install() -> ok | {error, Reason} + install(TargetSystem) -> ok | {error, Reason} + install(Opts) -> ok | {error, Reason} + install(TargetSystem,Opts) -> ok | {error, Reason} + Installs the Test Server Framework + + TargetSystem = {Architecture, TargetHost} + Architecture = atom() or string() + e.g. "ose" or "vxworks_ppc603" + TargetHost = atom() or string() + The name of the target host + Opts = list() + + +

Installs and configures the Test Server Framework for + running test suites. If a remote host is to be used, the + TargetSystem argument must be given so that "cross + installation" can be done. This should be used for testing on + VxWorks or OSE/Delta. Installation is required for any of the + functions in ts to work. +

+

Opts may be one or more of +

+ + {longnames, Bool}

+ Use fully qualified hostnames for test_server and + slave nodes. Bool is true or false (default). + + {verbose, Level}

+ Verbosity level for test server output, set to 0, 1 or + 2, where 0 is quiet(default). + + {hosts, Hosts}

+ This is a list of available hosts on which to start + slave nodes. It is used when the {remote, true} + option is given to the test_server:start_node/3 + function. Also, if {require_nodenames, Num} is + contained in a test specification file, the generated + nodenames will be spread over all hosts given in this + Hosts list. The hostnames are given as atoms or + strings. + + {slavetargets, SlaveTarges}

+ For VxWorks and OSE/Delta only. This is a list of + available hosts where slave nodes can be started. This is + necessary because only one node can run per host in the + VxWorks environment. This is not the same as + {hosts, Hosts} because it is used for all slave nodes + - not only the ones started with {remote, true}. The + hostnames are given as atoms or strings. + + {crossroot, TargetErlRoot}

+ Erlang root directory on target host +

+This option is mandatory for remote targets + + {master, {MasterHost, MasterCookie}}

+ If target is remote and the target node is started as + a slave node, this option + indicates which master and cookie to use. The given master + will also be used as master for slave nodes started with + test_server:start_node/3. It is expected that the + erl_boot_server is started on the master node before + the test is run. If this option is not given, the test + server controller node is used as master and the + erl_boot_server is automatically started. + + {erl_start_args, ArgString}

+ Additional arguments to be used when starting the test + server controller node. ArgString will be appended to + the command line when starting the erlang node. Note that + this will only affect the startup of the controller node, + i.e. not the target node or any slave nodes + startet from a test case. + + {ipv6_hosts, HostList}

+ This option will be inserted in the + Config parameter for each test case. HostList + is a list of hosts supporting IPv6. + + + + + + help() -> ok + Presents simple help on the functions in ts + +

Presents simple help on the functions in ts. Useful + for quick reference.

+ + + + tests() -> Tests + Returns the list of available tests + +

Returns the list of available tests. This is actually just + a list of all test specification files found by looking up + "../*_test/*.spec". +

+

In each ../Name_test/ directory there should be one test + specification file named Name.spec.

+ + + + run() -> ok | {error, Reason} + run([all_tests|Opts]) + run(Specs) + run(Specs, Opts) + run(Spec, Module) + run(Spec, Module, Opts) + run(Spec, Module, Case) + run(Spec, Module, Case, Opts) + Runs (specified) test suite(s) + + Specs = Spec | [Spec] + Spec = atom() + Module = atom() + Case = atom() + Opts = [Opt] + Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec} + Level = integer(); 0 means silent + Vars = list() of key-value tuples + CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file. + Reason = term() + + +

This function runs test suite(s)/case(s). To be able to run + any tests, ts:install must first be called to create the + variables file needed. To run a whole test specification, + only specify the name of the test specification, and all test + suite modules belonging to that test spec will be run. To run + a single module in a test specification, use the Module + argument to specify the name of the module to run and all test + cases in that module will be run, and to run a specified test + case, specify the name of the test case using the Case + argument. If called with no argument, all test specifications + available will be run. Use ts:tests/0 to see the available + test specifications. +

+

If the batch option is not given, a new xterm is + started (unix) when ts:run is called. +

+

The verbose option sets the verbosity level for test + server output. This has the same effect as if given to + ts:install/1/2

+

The vars option can be used for adding configuration + variables that are not in the variables file generated + during installation. Can be any of the Opts valid for + ts:install/1/2. +

+

The keep_topcase option forces ts to keep the + topcase in your test specification file as is. This option can + only be used if you don't give the Module or + Case parameters to ts:run. The + keep_topcase option is necessary if your topcase + contains anything other than _test"}]]>. If + the option is not used, ts will modify your topcase. +

+

The cover and cover_details options indicates + that the test shall be run with code coverage + analysis. cover_details means that analysis shall be + done on the most detailed level. If the test is run with a + remote target, this option creates a list of uncovered lines + in each cover compiled module. If the test is run with a local + target, each cover compiled module will be analysed with + cover:analyse_to_file/1. The cover options will + only create an overview of all cover compiled modules with the + number of covered and not covered lines. +

+

The CoverFile which can be given with the + cover and cover_details options must be the + filename of a file listing modules to be excluded from or + included in the cover compilation. By default, ts + believes that Spec is the name of an OTP application + and that all modules in this application shall be cover + compiled. The CoverFile can exclude modules that belong + to the application and add modules that don't belong to the + application. The file can have the following entries:

+ +{exclude, all | ExcludeModuleList}. +{include, IncludeModuleList}. +

Note that each line must end with a full + stop. ExcludeModuleList and IncludeModuleList + are lists of atoms, where each atom is a module name. +

+

If the cover or cover_details options are + given on their own, the directory _test]]> is + searched for a CoverFile named .cover]]>. If + this file is not found, Spec is assumed to be the name + of an OTP application, and all modules in the ebin + directory for the application are cover compiled. The + ebin directory is found by adding ebin to + code:lib_dir(Spec). +

+

The same cover compiled code will be loaded on all slave or + peer nodes started with test_server:start_node/3. The + exception is nodes that run an old version of OTP. If the loading + fails, the node will simply not be a part of the coverage + analysis. Note that slave and peer nodes must be stopped with + test_server:stop_node/1 for the node to be part of the + coverage analysis, else the test server will not be able to + fetch coverage data from the node. +

+

The trace option is used to turn on call trace on + target and on slave or peer nodes started with + test_server:start_node/3. TraceSpec can be the + name of a trace information file, or a list of elements like + the ones in a trace information file. Please turn to the + reference manual for test_server_ctrl:trc/1 for details + about the trace information file. +

+ + + + cross_cover_analyse(Level) -> ok + cross_cover_analyse([Level]) -> ok + Analyse cover data collected from all tests + +

Analyse cover data collected from all tests. +

+

See test_server_ctrl:cross_cover_analyse/1 +

+ + + + r() -> ok + r(Opts) -> ok + r(SpecOrSuite) -> ok + r(SpecOrSuite,Opts) -> ok + r(Suite,Case) -> ok + r(Suite,Case,Opts) -> ok + Run test suite or test case without tsinstalled + + SpecOrSuite = Spec | Suite + Spec = string() + "Name.spec" or "Name.spec.OsType", where OsType is vxworks + Suite = atom() + Case = atom() + Opts = [Opt] + Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile} + Cover = cover | cover_details + AppOrCoverFile = Application | CoverFile + Application = atom() + OTP application to cover compile + CoverFile = string() + Name of file listing modules to exclude from or include in cover compilation + + +

This function can be used to run a test suites or test + cases directly, without any of the additional features added + by the test server framework. It is simply a wrapper function + for the add_dir, add_spec, add_module and + add_case functions in test_server_ctrl: +

+

r() -> add_dir(".")

+r(Spec) -> add_spec(Spec)

+r(Suite) -> add_module(Suite)

+r(Suite,Case) -> add_case(Suite,Case)

+

To use this function, it is required that the test suite is + compiled and in the code path of the node where the function + is called. The function can be used without having ts + installed. +

+

For information about the cover and + cover_details options, see test_server_ctrl:cover/2/3.

+ + + + index() -> ok | {error, Reason} + Updates local index page + + Reason = term() + + +

This function updates the local index page. This can be + useful if a previous test run was not completed and the index + is incomplete.

+ + + + clean() -> ok + clean(all) -> ok + Cleans up the log directories created when running tests. + +

This function cleans up log directories created when + running test cases. clean/0 cleans up all but the last + run of each application. clean/1 cleans up all test + runs found.

+ + + + estone() -> ok | {error, Reason} + estone(Opts) -> ok + Runs the EStone test + +

This function runs the EStone test. It is a shortcut for + running the test suite estone_SUITE in the + kernel application. +

+

Opts is the same as the Opts argument for the + ts:run functions.

+ + + + +
+ Makfile.src in Data Directory +

If a data directory contains code which must be compiled before + the test suite is run, a makefile source called + Makefile.src can be placed in the data directory. This file + will be converted to a valid makefile by ts:run/0/1/2/3/4. +

+

The reason for generating the makefile is that you can use + variables from the variables file which was generated by + ts:install/0/1/2. All occurrences of @Key@ in + Makefile.src is substituted by the Value from + {Key,Value} found in the variables file. Example: +

+

Cut from variables:

+ + ... + {'EMULATOR',"beam"}. + {'CFLAGS',"-g -O2"}. + {'LD',"$(CC) $(CFLAGS)"}. + {'CC',"gcc"}. + ... + +

Makefile.src for compiling erlang code could look + something like this:

+ + EFLAGS=+debug_info + + all: ordsets1.@EMULATOR@ + + ordsets1.@EMULATOR@: ordsets1.erl + erlc $(EFLAGS) ordsets1.erl + +

Makefile.src for compiling c code could look + something like this:

+ + CC = @CC@ + LD = @LD@ + CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@ + CROSSLDFLAGS = @CROSSLDFLAGS@ + + PROGS = nfs_check@exe@ + + all: $(PROGS) + + nfs_check@exe@: nfs_check@obj@ + $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@ + + nfs_check@obj@: nfs_check.c + $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c + +
+ + -- cgit v1.2.3