The Common Test Framework provides a high level operator interface for testing. It adds the following features to the Erlang/OTP Test Server:
When Common Test starts, it will automatically attempt to compile any suites included in the specified tests. If particular suites have been specified, only those suites will be compiled. If a particular test object directory has been specified (meaning all suites in this directory should be part of the test), Common Test runs make:all/1 in the directory to compile the suites.
If compilation should fail for one or more suites, the compilation errors are printed to tty and the operator is asked if the test run should proceed without the missing suites, or be aborted. If the operator chooses to proceed, it is noted in the HTML log which tests have missing suites.
Any help module (i.e. regular Erlang module with name not ending with "_SUITE") that resides in the same test object directory as a suite which is part of the test, will also be automatically compiled. A help module will not be mistaken for a test suite (unless it has a "_SUITE" name of course). All help modules in a particular test object directory are compiled no matter if all or only particular suites in the directory are part of the test.
If test suites or help modules include header files stored in other
locations than the test directory, you may specify these include directories
by means of the
Common Test will pass all include directories (specified either with the
It is also possible to specify include directories in test specifications (see below).
If the user wants to run all test suites for a test object (or OTP application)
by specifying only the top directory (e.g. with the
It is possible to disable the automatic compilation feature by using the
The
Examples:
It is also possible to combine the
This has the same effect as calling:
Other flags that may be used with
Directories passed to Common Test may have either relative or absolute paths.
Arbitrary start flags to the Erlang Runtime System may also be passed as
parameters to
Note how in this example, the absolute path of the
For more information about the
The web based GUI, VTS, is started with the
From the GUI you can run tests and view the result and the logs.
Note that
Example:
Note that the browser must run as a separate OS process or VTS will hang!
If no specific browser start command is specified, Firefox will
be the default browser on Unix platforms and Internet Explorer on Windows.
If Common Test fails to start a browser automatically, or
Common Test provides an Erlang API for running tests. The main (and most
flexible) function for specifying and executing tests is called
is with
For detailed documentation, please see the
You can start Common Test in an interactive shell mode where no automatic testing is performed. Instead, in this mode, Common Test starts its utility processes, installs configuration data (if any), and waits for the user to call functions (typically test case support functions) from the Erlang shell.
The shell mode is useful e.g. for debugging test suites, for analysing and debugging the SUT during "simulated" test case execution, and for trying out various operations during test suite development.
To invoke the interactive shell mode, you can start an Erlang shell
manually and call
If no config file is given with the
If any functions using "required config data" (e.g. ct_telnet or
ct_ftp functions) are to be called from the erlang shell, config
data must first be required with
Example:
1> ct:require(unix_telnet, unix).
ok
2> ct_telnet:open(unix_telnet).
{ok,<0.105.0>}
4> ct_telnet:cmd(unix_telnet, "ls .").
{ok,["ls .","file1 ...",...]}
Everything that Common Test normally prints in the test case logs,
will in the interactive mode be written to a log named
If you wish to exit the interactive mode (e.g. to start an
automated test run with
By means of
If no extra options are given with the
Common Test enables the Debugger auto attach feature, which means
that for every new interpreted test case function that starts to execute,
a new trace window will automatically pop up. (This is because each test
case executes on a dedicated Erlang process). Whenever a new test case starts,
Common Test will attempt to close the inactive trace window of the previous
test case. However, if you prefer that Common Test leaves inactive trace
windows, use the
The step functionality can be used together with the
The most flexible way to specify what to test, is to use a so
called test specification. A test specification is a sequence of
Erlang terms. The terms may be declared in a text file or passed
to the test server at runtime as a list
(see
With configuration terms it is possible to e.g. label the test
run (similar to
With test specification terms it is possible to state exactly which tests should run and in which order. A test term specifies either one or more suites, one or more test case groups, or one or more test cases in a group or suite.
An arbitrary number of test terms may be declared in sequence.
Common Test will by default compile the terms into one or more tests
to be performed in one resulting test run. Note that a term that
specifies a set of test cases will "swallow" one that only
specifies a subset of these cases. E.g. the result of merging
one term that specifies that all cases in suite S should be
executed, with another term specifying only test case X and Y in
S, is a test of all cases in S. However, if a term specifying
test case X and Y in S is merged with a term specifying case Z
in S, the result is a test of X, Y and Z in S. To disable this
behaviour, it is possible in test specification to set the
A test term can also specify one or more test suites, groups, or test cases to be skipped. Skipped suites, groups and cases are not executed and show up in the HTML test log files as SKIPPED.
When a test case group is specified, the resulting test
executes the
With the
Below is the test specification syntax. Test specifications can
be used to run tests both in a single test host environment and
in a distributed Common Test environment (Large Scale
Testing). The node parameters in the init term are only
relevant in the latter (see the
Config terms:
{node, NodeAlias, Node}.
{init, InitOptions}.
{init, [NodeAlias], InitOptions}.
{label, Label}.
{label, NodeRefs, Label}.
{multiply_timetraps, N}.
{multiply_timetraps, NodeRefs, N}.
{scale_timetraps, Bool}.
{scale_timetraps, NodeRefs, Bool}.
{cover, CoverSpecFile}.
{cover, NodeRefs, CoverSpecFile}.
{include, IncludeDirs}.
{include, NodeRefs, IncludeDirs}.
{config, ConfigFiles}.
{config, NodeRefs, ConfigFiles}.
{userconfig, {CallbackModule, ConfigStrings}}.
{userconfig, NodeRefs, {CallbackModule, ConfigStrings}}.
{alias, DirAlias, Dir}.
{merge_tests, Bool}.
{logdir, LogDir}.
{logdir, NodeRefs, LogDir}.
{event_handler, EventHandlers}.
{event_handler, NodeRefs, EventHandlers}.
{event_handler, EventHandlers, InitArgs}.
{event_handler, NodeRefs, EventHandlers, InitArgs}.
{ct_hooks, CTHModules}.
{ct_hooks, NodeRefs, CTHModules}.
{enable_builtin_hooks, Bool}.
Test terms:
{suites, DirRef, Suites}.
{suites, NodeRefs, DirRef, Suites}.
{groups, DirRef, Suite, Groups}.
{groups, NodeRefsDirRef, Suite, Groups}.
{groups, DirRef, Suite, GroupSpec, {cases,Cases}}.
{groups, NodeRefsDirRef, Suite, GroupSpec, {cases,Cases}}.
{cases, DirRef, Suite, Cases}.
{cases, NodeRefs, DirRef, Suite, Cases}.
{skip_suites, DirRef, Suites, Comment}.
{skip_suites, NodeRefs, DirRef, Suites, Comment}.
{skip_cases, DirRef, Suite, Cases, Comment}.
{skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}.
Types:
NodeAlias = atom()
InitOptions = term()
Node = node()
NodeRef = NodeAlias | Node | master
NodeRefs = all_nodes | [NodeRef] | NodeRef
N = integer()
Bool = true | false
CoverSpecFile = string()
IncludeDirs = string() | [string()]
ConfigFiles = string() | [string()]
DirAlias = atom()
Dir = string()
LogDir = string()
EventHandlers = atom() | [atom()]
InitArgs = [term()]
CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]
CTHModule = atom()
CTHInitArgs = term()
DirRef = DirAlias | Dir
Suites = atom() | [atom()] | all
Suite = atom()
Groups = GroupSpec | [GroupSpec] | all
GroupSpec = Group | {Group,Properties} | {Group,Properties,GroupSpec}
Group = atom()
Cases = atom() | [atom()] | all
Comment = string() | ""
Example:
{logdir, "/home/test/logs"}.
{config, "/home/test/t1/cfg/config.cfg"}.
{config, "/home/test/t2/cfg/config.cfg"}.
{config, "/home/test/t3/cfg/config.cfg"}.
{alias, t1, "/home/test/t1"}.
{alias, t2, "/home/test/t2"}.
{alias, t3, "/home/test/t3"}.
{suites, t1, all}.
{skip_suites, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}.
{skip_cases, t1, t1A_SUITE, [test3,test4], "Irrelevant"}.
{skip_cases, t1, t1C_SUITE, [test1], "Ignore"}.
{suites, t2, [t2B_SUITE,t2C_SUITE]}.
{cases, t2, t2A_SUITE, [test4,test1,test7]}.
{skip_suites, t3, all, "Not implemented"}.
The example specifies the following:
It is possible to specify initialization options for nodes defined in the
test specification. Currently, there are options to start the node and/or to
evaluate any function on the node.
See the
It is possible for the user to provide a test specification that
includes (for Common Test) unrecognizable terms. If this is desired,
the
As the execution of the test suites proceed, events are logged in four different ways:
Typically the operator, who may run hundreds or thousands of test cases, doesn't want to fill the console with details about, or printouts from, the specific test cases. By default, the operator will only see:
If/when the operator wants to dig deeper into the general results, or
the result of a specific test case, he should do so by
following the links in the HTML presentation and take a look in the
major or minor log files. The "all_runs.html" page is a practical
starting point usually. It's located in
An "index.html" page is written for each test run (i.e. stored in the "ct_run" directory tagged with node name, date and time). This file gives a short overview of all individual tests performed in the same test run. The test names follow this convention:
On the test run index page there is a link to the Common Test Framework log file in which information about imported configuration data and general test progress is written. This log file is useful to get snapshot information about the test run during execution. It can also be very helpful when analyzing test results or debugging test suites.
On the test run index page it is noted if a test has missing suites (i.e. suites that Common Test has failed to compile). Names of the missing suites can be found in the Common Test Framework log file.
The major logfile shows a detailed report of the test run. It includes test suite and test case names, execution time, the exact reason for failures etc. The information is available in both a file with textual and with HTML representation. The HTML file shows a summary which gives a good overview of the test run. It also has links to each individual test case log file for quick viewing with an HTML browser.
The minor log files contain full details of every single test
case, each one in a separate file. This way, it should be
straightforward to compare the latest results to that of previous
test runs, even if the set of test cases changes. If SASL is running,
its logs will also be printed to the current minor log file by the
The full name of the minor log file (i.e. the name of the file
including the absolute directory path) can be read during execution
of the test case. It comes as value in the tuple
Which information goes where is user configurable via the test server controller. Three threshold values determine what comes out on screen, and in the major or minor log files. See the OTP Test Server manual for information. The contents that goes to the HTML log file is fixed however and cannot be altered.
The log files are written continously during a test run and links are always created initially when a test starts. This makes it possible to follow test progress simply by refreshing pages in the HTML browser. Statistics totals are not presented until a test is complete however.
With the
With
With
For example, if a test is started with:
then printouts during the test made by successive calls to
instead of each
Common Test uses a CSS file to control the look of the HTML
files generated during test runs. If, for some reason, the
log files are not displayed correctly in the HTML browser of your
choice, or you prefer the "pre Common Test v1.6 look"
of the log files (i.e. not using CSS), use the start flag/option
Common Test includes an optional feature to allow
user HTML style sheets for customizing printouts. The
functions in
div.sys_config { background:blue; color:white }
div.sys_state { background:yellow; color:black }
div.error { background:red; color:white }
To install the CSS file (Common Test inlines the definition in the
HTML code), the name may be provided when executing
$ ct_run -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css
Categories in a CSS file installed with the
It is also possible to install style sheets on a per suite and per test case basis. Example:
-module(my_SUITE).
...
suite() -> [..., {stylesheet,"suite_categories.css"}, ...].
...
my_testcase(_) ->
...
ct:log(sys_config, "Test node version: ~p", [VersionInfo]),
...
ct:log(sys_state, "Connections: ~p", [ConnectionInfo]),
...
ct:pal(error, "Error ~p detected! Info: ~p", [SomeFault,ErrorInfo]),
ct:fail(SomeFault).
If the style sheet is installed as in this example, the categories are
private to the suite in question. They can be used by all test cases in the
suite, but can not be used by other suites. A suite private style sheet,
if specified, will be used in favour of a global style sheet (one specified
with the
In a tuple
The
You can order Common Test to repeat the tests you specify. You can choose
to repeat tests a certain number of times, repeat tests for a specific period of time,
or repeat tests until a particular stop time is reached. If repetition is controlled by
means of time, it is also possible to specify what action Common Test should
take upon timeout. Either Common Test performs all tests in the current run before stopping,
or it stops as soon as the current test job is finished. Repetition can be activated by
means of
The duration time,
When timeout occurs, Common Test will never abort the test run immediately, since
this might leave the system under test in an undefined, and possibly bad, state.
Instead Common Test will finish the current test job, or the complete test
run, before stopping. The latter is the default behaviour. The
Log files from every single repeated test run is saved in normal Common Test fashion (see above). Common Test may later support an optional feature to only store the last (and possibly the first) set of logs of repeated test runs, but for now the user must be careful not to run out of disk space if tests are repeated during long periods of time.
Note that for each test run that is part of a repeated session, information about the particular test run is printed in the Common Test Framework Log. There you can read the repetition number, remaining time, etc.
Example 1:
$ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -force_stop
Here the suites in test directory to1, followed by the suites in to2, will be executed
in one test run. A timeout event will occur after 10 minutes. As long as there is time
left, Common Test will repeat the test run (i.e. starting over with the to1 test).
When the timeout occurs, Common Test will stop as soon as the current job is finished
(because of the
Example 2:
$ date
Fri Sep 28 15:00:00 MEST 2007
$ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -until 160000
Here the same test run as in the example above will be executed (and possibly repeated). In this example, however, the timeout will occur after 1 hour and when that happens, Common Test will finish the entire test run before stopping (i.e. the to1 and to2 test will always both be executed in the same test run).
Example 3:
$ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -repeat 5
Here the test run, including both the to1 and the to2 test, will be repeated 5 times.
This feature should not be confused with the
The protocol handling processes in Common Test, implemented by ct_telnet, ct_ftp etc,
do verbose printing to the test case logs. This can be switched off by means
of the
ct_run -silent_connections [conn_types]
where
Example:
ct_run ... -silent_connections telnet ftp
switches off logging for telnet and ftp connections.
ct_run ... -silent_connections
switches off logging for all connection types.
Basic and important information such as opening and closing a connection, fatal communication error and reconnection attempts will always be printed even if logging has been suppressed for the connection type in question. However, operations such as sending and receiving data may be performed silently.
It is possible to also specify
The
-module(my_SUITE).
...
suite() -> [..., {silent_connections,[telnet,ftp]}, ...].
...
my_testcase1() ->
[{silent_connections,[ftp]}].
my_testcase1(_) ->
...
my_testcase2(_) ->
...
In this example,
The
Note that in the current Common Test version, the