aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src/basics_chapter.xml
diff options
context:
space:
mode:
authorPeter Andersson <[email protected]>2016-03-10 14:43:09 +0100
committerPeter Andersson <[email protected]>2016-03-10 14:43:09 +0100
commit629ba3086de251e33339e7ce846c568451d27300 (patch)
tree760e07b5c4409c73d005593c8f9931bc379fa241 /lib/common_test/doc/src/basics_chapter.xml
parent9cafc664bbf75fa9b6d3666f85e672605c96d1f1 (diff)
parent00c18b1568eb629fcfa715295d7e99accf9c74b1 (diff)
downloadotp-629ba3086de251e33339e7ce846c568451d27300.tar.gz
otp-629ba3086de251e33339e7ce846c568451d27300.tar.bz2
otp-629ba3086de251e33339e7ce846c568451d27300.zip
Merge branch 'maint'
Diffstat (limited to 'lib/common_test/doc/src/basics_chapter.xml')
-rw-r--r--lib/common_test/doc/src/basics_chapter.xml230
1 files changed, 117 insertions, 113 deletions
diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml
index a01e3a9272..1a5a686fa0 100644
--- a/lib/common_test/doc/src/basics_chapter.xml
+++ b/lib/common_test/doc/src/basics_chapter.xml
@@ -31,74 +31,74 @@
</header>
<marker id="basics"></marker>
<section>
- <title>Introduction</title>
+ <title>General</title>
- <p>The <em>Common Test</em> framework (CT) is a tool which supports
- implementation and automated execution of test cases towards arbitrary
- types of target systems. The CT framework is based on the OTP Test
- Server and it's the main tool being used in all testing- and verification
- activities that are part of Erlang/OTP system development- and maintenance.
+ <p>The <c>Common Test</c> framework is a tool that supports
+ implementation and automated execution of test cases to any
+ types of target systems. <c>Common Test</c> is the main tool being used
+ in all testing- and verification activities that are part of Erlang/OTP
+ system development and maintenance.
</p>
- <p>Test cases can be executed individually or in batches. Common Test
- also features a distributed testing mode with central control and logging
- (a feature that makes it possible to test multiple systems independently in
- one common session, useful e.g. for running automated large-scale regression
- tests).
+ <p>Test cases can be executed individually or in batches. <c>Common Test</c>
+ also features a distributed testing mode with central control and logging.
+ With this feature, multiple systems can be tested independently in
+ one common session. This is useful, for example, when running automated
+ large-scale regression tests.
</p>
<p>
- The SUT (System Under Test) may consist of one or several target
- nodes. CT contains a generic test server which, together with
- other test utilities, is used to perform test case execution.
- It is possible to start the tests from a GUI or from the OS- or
+ The System Under Test (SUT) can consist of one or more target
+ nodes. <c>Common Test</c> contains a generic test server that,
+ together with other test utilities, is used to perform test case execution.
+ The tests can be started from a GUI, from the OS shell, or from an
Erlang shell. <em>Test suites</em> are files (Erlang
modules) that contain the <em>test cases</em> (Erlang functions)
to be executed. <em>Support modules</em> provide functions
- that the test cases utilize in order to carry out the tests.
+ that the test cases use to do the tests.
</p>
- <p>In a black-box testing scenario, CT based test programs connect to
- the target system(s) via standard O&amp;M and CLI protocols. CT
+ <p>In a black-box testing scenario, <c>Common Test</c>-based test programs connect to
+ the target system(s) through standard O&amp;M and CLI protocols. <c>Common Test</c>
provides implementations of, and wrapper interfaces to, some of these
- protocols (most of which exist as stand-alone components and
+ protocols (most of which exist as standalone components and
applications in OTP). The wrappers simplify configuration and add
- verbosity for logging purposes. CT will be continously extended with
- useful support modules. (Note however that it's
- a straightforward task to use any arbitrary Erlang/OTP component
- for testing purposes with Common Test, without needing a CT wrapper
- for it. It's as simple as calling Erlang functions). There
- are a number of target independent interfaces supported in CT, such as
- Generic Telnet, FTP, etc, which can be specialized or used
- directly for controlling instruments, traffic load generators, etc.
+ verbosity for logging purposes. <c>Common Test</c> is continously extended with
+ useful support modules. However, notice that it is
+ a straightforward task to use any Erlang/OTP component
+ for testing purposes with <c>Common Test</c>, without needing a <c>Common Test</c>
+ wrapper for it. It is as simple as calling Erlang functions. A number of
+ target-independent interfaces are supported in <c>Common Test</c>, such as
+ Generic Telnet and FTP. These can be specialized or used
+ directly for controlling instruments, traffic load generators, and so on.
</p>
- <p>Common Test is also a very useful tool for white-box testing Erlang
- code (e.g. module testing), since the test programs can call exported Erlang
- functions directly and there's very little overhead required for
+ <p><c>Common Test</c> is also a very useful tool for white-box testing Erlang
+ code (for example, module testing), as the test programs can call exported Erlang
+ functions directly. there is very little overhead required for
implementing basic test suites and executing simple tests. For black-box
- testing Erlang software, Erlang RPC as well as standard O&amp;M interfaces
- can for example be used.
+ testing Erlang software, Erlang RPC and standard O&amp;M interfaces
+ can be used for example.
</p>
- <p>A test case can handle several connections towards one or
- several target systems, instruments and traffic generators in
- parallel in order to perform the necessary actions for a
- test. The handling of many connections in parallel is one of
- the major strengths of Common Test (thanks to the efficient
- support for concurrency in the Erlang runtime system - which CT users
- can take great advantage of!).
+ <p>A test case can handle several connections to one or
+ more target systems, instruments, and traffic generators in
+ parallel to perform the necessary actions for a test.
+ The handling of many connections in parallel is one of
+ the major strengths of <c>Common Test</c>, thanks to the efficient
+ support for concurrency in the Erlang runtime system, which <c>Common Test</c>
+ users can take great advantage of.
</p>
</section>
<section>
<title>Test Suite Organisation</title>
<p>
- The test suites are organized in test directories and each test suite
- may have a separate data directory. Typically, these files and directories
- are version controlled similarly to other forms of source code (possibly by
- means of a version control system like GIT or Subversion). However, CT does
- not itself put any requirements on (or has any form of awareness of)
+ Test suites are organized in test directories and each test suite
+ can have a separate data directory. Typically, these files and directories
+ are version-controlled similar to other forms of source code (possibly by
+ a version control system like GIT or Subversion). However, <c>Common Test</c>
+ does not itself put any requirements on (or has any awareness of)
possible file and directory versions.
</p>
</section>
@@ -109,8 +109,8 @@
Support libraries contain functions that are useful for all test suites,
or for test suites in a specific functional area or subsystem.
In addition to the general support libraries provided by the
- CT framework, and the various libraries and applications provided by
- Erlang/OTP, there might also be a need for customized (user specific)
+ <c>Common Test</c> framework, and the various libraries and applications provided by
+ Erlang/OTP, there can also be a need for customized (user specific)
support libraries.
</p>
</section>
@@ -121,118 +121,122 @@
Testing is performed by running test suites (sets of test cases) or
individual test cases. A test suite is implemented as an Erlang module named
<c><![CDATA[<suite_name>_SUITE.erl]]></c> which contains a number of test cases.
- A test case is an Erlang function which tests one or more things.
- The test case is the smallest unit that the CT test server deals with.
+ A test case is an Erlang function that tests one or more things.
+ The test case is the smallest unit that the <c>Common Test</c> test server deals with.
</p>
<p>
- Subsets of test cases, called test case groups, may also be defined. A test case
+ Subsets of test cases, called test case groups, can also be defined. A test case
group can have execution properties associated with it. Execution properties
- specify whether the test cases in the group should be executed in
- random order, in parallel, in sequence, and if the execution of the group
- should be repeated. Test case groups may also be nested (i.e. a group may,
- besides test cases, contain sub-groups).
+ specify if the test cases in the group are to be executed in
+ random order, in parallel, or in sequence, and if the execution of the group
+ is to be repeated. Test case groups can also be nested (that is, a group can,
+ besides test cases, contain subgroups).
</p>
<p>
- Besides test cases and groups, the test suite may also contain configuration
+ Besides test cases and groups, the test suite can also contain configuration
functions. These functions are meant to be used for setting up (and verifying)
- environment and state on the SUT (and/or the CT host node), required for
- the tests to execute correctly. Examples of operations: Opening a connection
- to the SUT, initializing a database, running an installation script, etc.
- Configuration may be performed per suite, per test case group and per
- individual test case.
+ environment and state in the SUT (and/or the <c>Common Test</c> host node),
+ required for the tests to execute correctly. Examples of operations are:
+ Opening a connection to the SUT, initializing a database, running an installation
+ script, and so on. Configuration can be performed per suite, per test case group,
+ and per individual test case.
</p>
<p>
The test suite module must conform to a
<seealso marker="common_test">callback interface</seealso>
- specified by the CT test server. See the
- <seealso marker="write_test_chapter#intro">Writing Test Suites</seealso> chapter
- for more information.
+ specified by the <c>Common Test</c> test server. For details, see section
+ <seealso marker="write_test_chapter#intro">Writing Test Suites</seealso>.
</p>
<p>
A test case is considered successful if it returns to the caller, no matter
- what the returned value is. A few return values have special meaning however
- (such as <c>{skip,Reason}</c> which indicates that the test case is skipped,
- <c>{comment,Comment}</c> which prints a comment in the log for the test case and
- <c>{save_config,Config}</c> which makes the CT test server pass <c>Config</c> to
- the next test case).
+ what the returned value is. However, a few return values have special meaning
+ as follows:</p>
+ <list type="bulleted">
+ <item><c>{skip,Reason}</c> indicates that the test case is skipped.</item>
+ <item><c>{comment,Comment}</c> prints a comment in the log for the test case.</item>
+ <item><c>{save_config,Config}</c> makes the <c>Common Test</c> test server pass
+ <c>Config</c> to the next test case.</item>
+ </list>
+ <p>
A test case failure is specified as a runtime error (a crash), no matter what
the reason for termination is. If you use Erlang pattern matching effectively,
- you can take advantage of this property. The result will be concise and
+ you can take advantage of this property. The result is concise and
readable test case functions that look much more like scripts than actual programs.
- Simple example:
+ A simple example:
</p>
<pre>
- session(_Config) ->
- {started,ServerId} = my_server:start(),
- {clients,[]} = my_server:get_clients(ServerId),
- MyId = self(),
- connected = my_server:connect(ServerId, MyId),
- {clients,[MyId]} = my_server:get_clients(ServerId),
- disconnected = my_server:disconnect(ServerId, MyId),
- {clients,[]} = my_server:get_clients(ServerId),
- stopped = my_server:stop(ServerId).
- </pre>
+ session(_Config) ->
+ {started,ServerId} = my_server:start(),
+ {clients,[]} = my_server:get_clients(ServerId),
+ MyId = self(),
+ connected = my_server:connect(ServerId, MyId),
+ {clients,[MyId]} = my_server:get_clients(ServerId),
+ disconnected = my_server:disconnect(ServerId, MyId),
+ {clients,[]} = my_server:get_clients(ServerId),
+ stopped = my_server:stop(ServerId).</pre>
<p>
As a test suite runs, all information (including output to <c>stdout</c>) is
- recorded in several different log files. A minimum of information is displayed
+ recorded in many different log files. A minimum of information is displayed
in the user console (only start and stop information, plus a note
for each failed test case).
</p>
<p>
The result from each test case is recorded in a dedicated HTML log file, created
for the particular test run. An overview page displays each test case represented
- by row in a table showing total execution time, whether the case was successful,
- failed or skipped, plus an optional user comment. (For a failed test case, the
- reason for termination is also printed in the comment field). The overview page
+ by a table row showing total execution time, if the case was successful,
+ failed, or skipped, plus an optional user comment. For a failed test case, the
+ reason for termination is also printed in the comment field. The overview page
has a link to each test case log file, providing simple navigation with any standard
HTML browser.
</p>
</section>
<section>
+<marker id="External_Interfaces"></marker>
<title>External Interfaces</title>
<p>
- The CT test server requires that the test suite defines and exports the
+ The <c>Common Test</c> test server requires that the test suite defines and exports the
following mandatory or optional callback functions:
</p>
<taglist>
- <tag>all()</tag>
- <item>Returns a list of all test cases and groups in the suite. (Mandatory)</item>
- <tag>suite()</tag>
- <item>Info function used to return properties for the suite. (Optional)</item>
- <tag>groups()</tag>
- <item>For declaring test case groups. (Optional)</item>
- <tag>init_per_suite(Config)</tag>
- <item>Suite level configuration function, executed before the first
- test case. (Optional)</item>
- <tag>end_per_suite(Config)</tag>
- <item>Suite level configuration function, executed after the last
- test case. (Optional)</item>
- <tag>group(GroupName)</tag>
- <item>Info function used to return properties for a test case group. (Optional)</item>
- <tag>init_per_group(GroupName, Config)</tag>
- <item>Configuration function for a group, executed before the first
- test case. (Optional)</item>
- <tag>end_per_group(GroupName, Config)</tag>
- <item>Configuration function for a group, executed after the last
- test case. (Optional)</item>
- <tag>init_per_testcase(TestCase, Config)</tag>
- <item>Configuration function for a testcase, executed before each
- test case. (Optional)</item>
- <tag>end_per_testcase(TestCase, Config)</tag>
- <item>Configuration function for a testcase, executed after each
- test case. (Optional)</item>
+ <tag><c>all()</c></tag>
+ <item><p>Returns a list of all test cases and groups in the suite. (Mandatory)</p></item>
+ <tag><c>suite()</c></tag>
+ <item><p>Information function used to return properties for the suite. (Optional)</p></item>
+ <tag><c>groups()</c></tag>
+ <item><p>For declaring test case groups. (Optional)</p></item>
+ <tag><c>init_per_suite(Config)</c></tag>
+ <item><p>Suite level configuration function, executed before the first
+ test case. (Optional)</p></item>
+ <tag><c>end_per_suite(Config)</c></tag>
+ <item><p>Suite level configuration function, executed after the last
+ test case. (Optional)</p></item>
+ <tag><c>group(GroupName)</c></tag>
+ <item><p>Information function used to return properties for a test case group. (Optional)</p></item>
+ <tag><c>init_per_group(GroupName, Config)</c></tag>
+ <item><p>Configuration function for a group, executed before the first
+ test case. (Optional)</p></item>
+ <tag><c>end_per_group(GroupName, Config)</c></tag>
+ <item><p>Configuration function for a group, executed after the last
+ test case. (Optional)</p></item>
+ <tag><c>init_per_testcase(TestCase, Config)</c></tag>
+ <item><p>Configuration function for a testcase, executed before each
+ test case. (Optional)</p></item>
+ <tag><c>end_per_testcase(TestCase, Config)</c></tag>
+ <item><p>Configuration function for a testcase, executed after each
+ test case. (Optional)</p></item>
</taglist>
<p>
- For each test case the CT test server expects these functions:
+ For each test case, the <c>Common Test</c> test server expects the
+ following functions:
</p>
<taglist>
<tag>Testcasename()</tag>
- <item>Info function that returns a list of test case properties. (Optional)</item>
+ <item><p>Information function that returns a list of test case properties. (Optional)</p></item>
<tag>Testcasename(Config)</tag>
- <item>The actual test case function.</item>
+ <item><p>The test case function.</p></item>
</taglist>
</section>
</chapter>