<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">
<erlref>
<header>
<copyright>
<year>2003</year><year>2016</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
</legalnotice>
<title>Common Test</title>
<prepared>Kenneth Lundin, Peter Andersson</prepared>
<responsible>Peter Andersson</responsible>
<docno></docno>
<approved></approved>
<checked></checked>
<date>2003-10-21</date>
<rev>PA1</rev>
<file>common_test_app.sgml</file>
</header>
<module>common_test</module>
<modulesummary>A framework for automated testing of any target nodes.</modulesummary>
<description>
<p>The <c>Common Test</c> framework is an environment for
implementing and performing automatic and semi-automatic execution of
test cases.</p>
<p>In brief, <c>Common Test</c> supports:</p>
<list>
<item>Automated execution of test suites (sets of test cases)</item>
<item>Logging of events during execution</item>
<item>HTML presentation of test suite results</item>
<item>HTML presentation of test suite code</item>
<item>Support functions for test suite authors</item>
<item>Step-by-step execution of test cases</item>
</list>
<p>The following section describes the mandatory and optional test suite
functions that <c>Common Test</c> calls during test execution.
For more details, see section
<seealso marker="write_test_chapter">Writing Test Suites</seealso>
in the User's Guide.</p>
</description>
<section>
<title>Test Case Callback Functions</title>
<p>The following functions define the callback interface
for a test suite.</p>
</section>
<funcs>
<func>
<name>Module:all() -> Tests | {skip,Reason} </name>
<fsummary>Returns the list of all test case groups and test cases
in the module.</fsummary>
<type>
<v>Tests = [TestCase | {group,GroupName} | {group,GroupName,Properties} | {group,GroupName,Properties,SubGroups}]</v>
<v>TestCase = atom()</v>
<v>GroupName = atom()</v>
<v>Properties = [parallel | sequence | Shuffle | {RepeatType,N}] | default</v>
<v>SubGroups = [{GroupName,Properties} | {GroupName,Properties,SubGroups}]</v>
<v>Shuffle = shuffle | {shuffle,Seed}</v>
<v>Seed = {integer(),integer(),integer()}</v>
<v>RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail</v>
<v>N = integer() | forever</v>
<v>Reason = term()</v>
</type>
<desc>
<p>MANDATORY</p>
<p>Returns the list of all test cases and test case groups in the
test suite module to be executed. This list also specifies the
order the cases and groups are executed by <c>Common Test</c>.
A test case is represented by an atom,
the name of the test case function. A test case group is
represented by a <c>group</c> tuple, where <c>GroupName</c>,
an atom, is the name of the group (defined in
<seealso marker="#Module:groups-0"><c>groups/0</c></seealso>).
Execution properties for groups can also be specified, both
for a top-level group and for any of its subgroups.
Group execution properties specified here override
properties in the group definition (see
<seealso marker="#Module:groups-0"><c>groups/0</c></seealso>).
(With value <c>default</c>, the group definition properties
are used).</p>
<p>If <c>{skip,Reason}</c> is returned, all test cases
in the module are skipped and <c>Reason</c>
is printed on the HTML result page.</p>
<p>For details on groups, see section
<seealso marker="write_test_chapter#test_case_groups">Test Case
Groups</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:groups() -> GroupDefs</name>
<fsummary>Returns a list of test case group definitions.</fsummary>
<type>
<v>GroupDefs = [Group]</v>
<v>Group = {GroupName,Properties,GroupsAndTestCases}</v>
<v>GroupName = atom()</v>
<v>Properties = [parallel | sequence | Shuffle | {RepeatType,N}]</v>
<v>GroupsAndTestCases = [Group | {group,GroupName} | TestCase]</v>
<v>TestCase = atom()</v>
<v>Shuffle = shuffle | {shuffle,Seed}</v>
<v>Seed = {integer(),integer(),integer()}</v>
<v>RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail</v>
<v>N = integer() | forever</v>
</type>
<desc>
<p>OPTIONAL</p>
<p>Defines test case groups. For details, see section
<seealso marker="write_test_chapter#test_case_groups">Test Case
Groups</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:suite() -> [Info] </name>
<fsummary>Test suite info function (providing default data
for the suite).</fsummary>
<type>
<v>Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}</v>
<v>Time = TimeVal | TimeFunc</v>
<v>TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()}</v>
<v>TimeFunc = {Mod,Func,Args} | Fun</v>
<v>MilliSec = integer()</v>
<v>Mod = atom()</v>
<v>Func = atom()</v>
<v>Args = list()</v>
<v>Fun = fun()</v>
<v>Required = Key | {Key,SubKeys} | {Key,SubKey} | {Key,SubKey,SubKeys}</v>
<v>Key = atom()</v>
<v>SubKeys = SubKey | [SubKey]</v>
<v>SubKey = atom()</v>
<v>Name = atom()</v>
<v>UserData = term()</v>
<v>Conns = [atom()]</v>
<v>CSSFile = string()</v>
<v>CTHs = [CTHModule |</v>
<v> {CTHModule, CTHInitArgs} |</v>
<v> {CTHModule, CTHInitArgs, CTHPriority}]</v>
<v>CTHModule = atom()</v>
<v>CTHInitArgs = term()</v>
</type>
<desc>
<p>OPTIONAL</p>
<p>The test suite information function. Returns a list of tagged
tuples specifying various properties related to the execution of
this test suite (common for all test cases in the suite).</p>
<p>Tag <c>timetrap</c> sets the maximum time that each
test case is allowed to execute (including
<seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso>
and
<seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso>).
If the timetrap time is exceeded, the test case fails with reason
<c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to
set a new timetrap by returning a <c>TimeVal</c>. It can also be
used to trigger a timetrap time-out by, at some point, returning a
value other than a <c>TimeVal</c>. For details, see section
<seealso marker="write_test_chapter#timetraps">Timetrap Time-Outs</seealso>
in the User's Guide.</p>
<p>Tag <c>require</c> specifies configuration variables
required by test cases (or configuration functions)
in the suite. If the required configuration variables are not found
in any of the configuration files, all test cases are skipped.
For details about the <c>require</c> functionality, see funtion
<seealso marker="ct#require-1"><c>ct:require/1,2</c></seealso>.</p>
<p>With <c>userdata</c>, the user can
specify any test suite-related information, which can be
read by calling
<seealso marker="ct#userdata-2"><c>ct:userdata/2</c></seealso>.</p>
<p>Tag <c>ct_hooks</c> specifies the
<seealso marker="ct_hooks_chapter">Common Test Hooks</seealso>
to be run with this suite.</p>
<p>Other tuples than the ones defined are ignored.</p>
<p>For details about the test suite information function, see section
<seealso marker="write_test_chapter#suite">Test
Suite Information Function</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:init_per_suite(Config) -> NewConfig | {skip,Reason} |
{skip_and_save,Reason,SaveConfig}</name>
<fsummary>Test suite initializations.</fsummary>
<type>
<v>Config = NewConfig = SaveConfig = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
<v>Reason = term()</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined, then <seealso
marker="#Module:end_per_suite-1"><c>end_per_suite/1</c></seealso>
must also be defined.</p>
<p>This configuration function is called as the first function in the
suite. It typically contains initializations that are common for
all test cases in the suite, and that must only be done
once. Parameter <c>Config</c> is the configuration data
that can be modified. Whatever is returned from this
function is specified as <c>Config</c> to all configuration functions
and test cases in the suite.</p>
<p>If <c>{skip,Reason}</c>
is returned, all test cases in the suite are skipped
and <c>Reason</c> is printed in the overview log for the suite.</p>
<p>For information on <c>save_config</c> and <c>skip_and_save</c>,
see section
<seealso marker="dependencies_chapter#save_config">Saving
Configuration Data</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:end_per_suite(Config) -> term() |
{save_config,SaveConfig}</name>
<fsummary>Test suite finalization.</fsummary>
<type>
<v>Config = SaveConfig = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined, then <seealso
marker="#Module:init_per_suite-1"><c>init_per_suite/1</c></seealso>
must also be defined.</p>
<p>This function is called as the last test case in the
suite. It is meant to be used for cleaning up after
<seealso marker="#Module:init_per_suite-1"><c>init_per_suite/1</c></seealso>.</p>
<p>For information on <c>save_config</c>, see section
<seealso marker="dependencies_chapter#save_config">Saving
Configuration Data</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:group(GroupName) -> [Info] </name>
<fsummary>Test case group information function (providing default data
for a test case group, that is, its test cases and
subgroups).</fsummary>
<type>
<v>Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}</v>
<v>Time = TimeVal | TimeFunc</v>
<v>TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()}</v>
<v>TimeFunc = {Mod,Func,Args} | Fun</v>
<v>MilliSec = integer()</v>
<v>Mod = atom()</v>
<v>Func = atom()</v>
<v>Args = list()</v>
<v>Fun = fun()</v>
<v>Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys}</v>
<v>Key = atom()</v>
<v>SubKeys = SubKey | [SubKey]</v>
<v>SubKey = atom()</v>
<v>Name = atom()</v>
<v>UserData = term()</v>
<v>Conns = [atom()]</v>
<v>CSSFile = string()</v>
<v>CTHs = [CTHModule |</v>
<v> {CTHModule, CTHInitArgs} |</v>
<v> {CTHModule, CTHInitArgs, CTHPriority}]</v>
<v>CTHModule = atom()</v>
<v>CTHInitArgs = term()</v>
</type>
<desc>
<p>OPTIONAL</p>
<p>The test case group information function. It is supposed to
return a list of tagged tuples that specify various properties
related to the execution of a test case group (that is, its test
cases and subgroups). Properties set by
<seealso marker="#Module:group-1"><c>group/1</c></seealso> override
properties with the same key that have been set previously by
<seealso marker="#Module:suite-0"><c>suite/0</c></seealso>.</p>
<p>Tag <c>timetrap</c> sets the maximum time that each
test case is allowed to execute (including
<seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso>
and
<seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso>).
If the timetrap time is
exceeded, the test case fails with reason
<c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to
set a new timetrap by returning a <c>TimeVal</c>. It can also be
used to trigger a timetrap time-out by, at some point, returning a
value other than a <c>TimeVal</c>. For details, see section
<seealso marker="write_test_chapter#timetraps">Timetrap
Time-Outs</seealso> in the User's Guide.</p>
<p>Tag <c>require</c> specifies configuration variables
required by test cases (or configuration functions)
in the suite. If the required configuration variables are not found
in any of the configuration files, all test cases in this group are
skipped. For details about the <c>require</c> functionality, see
function
<seealso marker="ct#require-1"><c>ct:require/1,2</c></seealso>.</p>
<p>With <c>userdata</c>, the user can
specify any test case group related information that can be
read by calling
<seealso marker="ct#userdata-2"><c>ct:userdata/2</c></seealso>.</p>
<p>Tag <c>ct_hooks</c> specifies the
<seealso marker="ct_hooks_chapter">Common Test Hooks</seealso>
to be run with this suite.</p>
<p>Other tuples than the ones defined are ignored.</p>
<p>For details about the test case group information function,
see section <seealso marker="write_test_chapter#group_info">Group
Information Function</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:init_per_group(GroupName, Config) -> NewConfig |
{skip,Reason}</name>
<fsummary>Test case group initializations.</fsummary>
<type>
<v>GroupName = atom()</v>
<v>Config = NewConfig = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
<v>Reason = term()</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined, then <seealso
marker="#Module:end_per_group-2"><c>end_per_group/2</c></seealso>
must also be defined.</p>
<p>This configuration function is called before execution of a
test case group. It typically contains initializations that are
common for all test cases and subgroups in the group, and that
must only be performed once. <c>GroupName</c> is the name of the
group, as specified in the group definition (see
<seealso marker="#Module:groups-0"><c>groups/0</c></seealso>).
Parameter <c>Config</c> is the configuration data that can be
modified.
The return value of this function is given as <c>Config</c>
to all test cases and subgroups in the group.</p>
<p>If <c>{skip,Reason}</c>
is returned, all test cases in the group are skipped and
<c>Reason</c> is printed in the overview log for the group.</p>
<p>For information about test case groups, see section
<seealso marker="write_test_chapter#test_case_groups">Test Case
Groups</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:end_per_group(GroupName, Config) -> term() |
{return_group_result,Status}</name>
<fsummary>Test case group finalization.</fsummary>
<type>
<v>GroupName = atom()</v>
<v>Config = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
<v>Status = ok | skipped | failed</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined, then <seealso
marker="#Module:init_per_group-2"><c>init_per_group/2</c></seealso>
must also be defined.</p>
<p>This function is called after the execution of a test case group
is finished. It is meant to be used for cleaning up after
<seealso marker="#Module:init_per_group-2"><c>init_per_group/2</c></seealso>.
A status value for a nested subgroup can be returned with
<c>{return_group_result,Status}</c>. The status can be retrieved in
<seealso marker="#Module:end_per_group-2"><c>end_per_group/2</c></seealso>
for the group on the level above. The status is also used by
<c>Common Test</c> for deciding if execution of a group is to
proceed if property <c>sequence</c> or <c>repeat_until_*</c>
is set.</p>
<p>For details about test case groups, see section
<seealso marker="write_test_chapter#test_case_groups">Test Case
Groups</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason} | {skip,Reason}</name>
<fsummary>Test case initializations.</fsummary>
<type>
<v> TestCase = atom()</v>
<v> Config = NewConfig = [{Key,Value}]</v>
<v> Key = atom()</v>
<v> Value = term()</v>
<v> Reason = term()</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined,
then <seealso marker="#Module:end_per_testcase-2">
<c>end_per_testcase/2</c></seealso> must also be
defined.</p>
<p>This function is called before each test case. Argument
<c>TestCase</c> is the test case name, and
<c>Config</c> (list of key-value tuples) is the configuration
data that can be modified. The <c>NewConfig</c> list returned
from this function is given as <c>Config</c> to the test case.
If <c>{fail,Reason}</c> is returned, the test case is
marked as failed without being executed.</p>
<p>If <c>{skip,Reason}</c> is returned, the test case is skipped
and <c>Reason</c> is printed in the overview log for the suite.</p>
</desc>
</func>
<func>
<name>Module:end_per_testcase(TestCase, Config) -> term() | {fail,Reason} | {save_config,SaveConfig}</name>
<fsummary>Test case finalization.</fsummary>
<type>
<v>TestCase = atom()</v>
<v>Config = SaveConfig = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
<v>Reason = term()</v>
</type>
<desc>
<p>OPTIONAL; if this function is defined,
then <seealso marker="#Module:init_per_testcase-2">
<c>init_per_testcase/2</c></seealso> must also be
defined.</p>
<p>This function is called after each test case, and can be used
to clean up after
<seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso>
and the test case. Any return value (besides <c>{fail,Reason}</c>
and <c>{save_config,SaveConfig}</c>) is ignored. By returning
<c>{fail,Reason}</c>, <c>TestCase</c> is marked as faulty (even
though it was successful in the sense that it returned
a value instead of terminating).</p>
<p>For information on <c>save_config</c>, see section
<seealso marker="dependencies_chapter#save_config">Saving
Configuration Data</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:Testcase() -> [Info] </name>
<fsummary>Test case information function.</fsummary>
<type>
<v>Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns}</v>
<v>Time = TimeVal | TimeFunc</v>
<v>TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()}</v>
<v>TimeFunc = {Mod,Func,Args} | Fun</v>
<v>MilliSec = integer()</v>
<v>Mod = atom()</v>
<v>Func = atom()</v>
<v>Args = list()</v>
<v>Fun = fun()</v>
<v>Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys}</v>
<v>Key = atom()</v>
<v>SubKeys = SubKey | [SubKey]</v>
<v>SubKey = atom()</v>
<v>Name = atom()</v>
<v>UserData = term()</v>
<v>Conns = [atom()]</v>
</type>
<desc>
<p>OPTIONAL</p>
<p>The test case information function. It is supposed to
return a list of tagged tuples that specify various properties
related to the execution of this particular test case.
Properties set by
<seealso marker="#Module:Testcase-0"><c>Testcase/0</c></seealso>
override properties set previously for the test case by
<seealso marker="#Module:group-1"><c>group/1</c></seealso> or
<seealso marker="#Module:suite-0"><c>suite/0</c></seealso>.</p>
<p>Tag <c>timetrap</c> sets the maximum time that the
test case is allowed to execute. If the timetrap time is
exceeded, the test case fails with reason <c>timetrap_timeout</c>.
<seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso>
and
<seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso>
are included in the timetrap time.
A <c>TimeFunc</c> function can be used to
set a new timetrap by returning a <c>TimeVal</c>. It can also be
used to trigger a timetrap time-out by, at some point, returning a
value other than a <c>TimeVal</c>. For details, see section
<seealso marker="write_test_chapter#timetraps">Timetrap
Time-Outs</seealso> in the User's Guide.</p>
<p>Tag <c>require</c> specifies configuration variables
that are required by the test case (or <c>init_per_testcase/2</c>
or <c>end_per_testcase/2</c>).
If the required configuration variables are not found in any of the
configuration files, the test case is skipped. For details about
the <c>require</c> functionality, see function
<seealso marker="ct#require-1"><c>ct:require/1,2</c></seealso>.</p>
<p>If <c>timetrap</c> or <c>require</c> is not set, the
default values specified by
<seealso marker="#Module:suite-0"><c>suite/0</c></seealso> (or
<seealso marker="#Module:group-1"><c>group/1</c></seealso>) are used.</p>
<p>With <c>userdata</c>, the user can specify any test case-related
information that can be read by calling
<seealso marker="ct#userdata-3"><c>ct:userdata/3</c></seealso>.</p>
<p>Other tuples than the ones defined are ignored.</p>
<p>For details about the test case information function, see section
<seealso marker="write_test_chapter#info_function">Test
Case Information Function</seealso> in the User's Guide.</p>
</desc>
</func>
<func>
<name>Module:Testcase(Config) -> term() | {skip,Reason} | {comment,Comment} | {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit() </name>
<fsummary>A test case.</fsummary>
<type>
<v>Config = SaveConfig = [{Key,Value}]</v>
<v>Key = atom()</v>
<v>Value = term()</v>
<v>Reason = term()</v>
<v>Comment = string()</v>
</type>
<desc>
<p>MANDATORY</p>
<p>The implementation of a test case. Call the functions to test and
check the result. If something fails, ensure the
function causes a runtime error or call
<seealso marker="ct#fail-1"><c>ct:fail/1,2</c></seealso>
(which also causes the test case process to terminate).</p>
<p>Elements from the <c>Config</c> list can, for example, be read
with <c>proplists:get_value/2</c> in STDLIB
(or the macro <c>?config</c> defined in <c>ct.hrl</c>).</p>
<p>If you decide not to run the test case after all, return
<c>{skip,Reason}</c>. <c>Reason</c> is then
printed in field <c>Comment</c> on the HTML result page.</p>
<p>To print some information in field <c>Comment</c> on the HTML
result page, return <c>{comment,Comment}</c>.</p>
<p>If the function returns anything else, the test case is
considered successful. The return value always gets printed
in the test case log file.</p>
<p>For details about test case implementation, see section
<seealso marker="write_test_chapter#test_cases">Test Cases</seealso>
in the User's Guide.</p>
<p>For information on <c>save_config</c> and <c>skip_and_save</c>,
see section
<seealso marker="dependencies_chapter#save_config">Saving
Configuration Data</seealso> in the User's Guide.</p>
</desc>
</func>
</funcs>
</erlref>