The Common Test framework (CT) is a tool which supports implementation and automated execution of test cases towards arbitrary types of target systems. The CT framework is based on the OTP Test Server and it's the main tool being used in all testing- and verification activities that are part of Erlang/OTP system development- and maintenance.
Test cases can be executed individually or in batches. Common Test also features a distributed testing mode with central control and logging (a feature that makes it possible to test multiple systems independently in one common session, useful e.g. for running automated large-scale regression tests).
The SUT (System Under Test) may consist of one or several target nodes. CT contains a generic test server which, together with other test utilities, is used to perform test case execution. It is possible to start the tests from a GUI or from the OS- or Erlang shell. Test suites are files (Erlang modules) that contain the test cases (Erlang functions) to be executed. Support modules provide functions that the test cases utilize in order to carry out the tests.
In a black-box testing scenario, CT based test programs connect to the target system(s) via standard O&M and CLI protocols. CT provides implementations of, and wrapper interfaces to, some of these protocols (most of which exist as stand-alone components and applications in OTP). The wrappers simplify configuration and add verbosity for logging purposes. CT will be continously extended with useful support modules. (Note however that it's a straightforward task to use any arbitrary Erlang/OTP component for testing purposes with Common Test, without needing a CT wrapper for it. It's as simple as calling Erlang functions). There are a number of target independent interfaces supported in CT, such as Generic Telnet, FTP, etc, which can be specialized or used directly for controlling instruments, traffic load generators, etc.
Common Test is also a very useful tool for white-box testing Erlang code (e.g. module testing), since the test programs can call exported Erlang functions directly and there's very little overhead required for implementing basic test suites and executing simple tests. For black-box testing Erlang software, Erlang RPC as well as standard O&M interfaces can for example be used.
A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of the major strengths of Common Test (thanks to the efficient support for concurrency in the Erlang runtime system - which CT users can take great advantage of!).
The test suites are organized in test directories and each test suite may have a separate data directory. Typically, these files and directories are version controlled similarly to other forms of source code (possibly by means of a version control system like GIT or Subversion). However, CT does not itself put any requirements on (or has any form of awareness of) possible file and directory versions.
Support libraries contain functions that are useful for all test suites, or for test suites in a specific functional area or subsystem. In addition to the general support libraries provided by the CT framework, and the various libraries and applications provided by Erlang/OTP, there might also be a need for customized (user specific) support libraries.
Testing is performed by running test suites (sets of test cases) or
individual test cases. A test suite is implemented as an Erlang module named
Subsets of test cases, called test case groups, may also be defined. A test case group can have execution properties associated with it. Execution properties specify whether the test cases in the group should be executed in random order, in parallel, in sequence, and if the execution of the group should be repeated. Test case groups may also be nested (i.e. a group may, besides test cases, contain sub-groups).
Besides test cases and groups, the test suite may also contain configuration functions. These functions are meant to be used for setting up (and verifying) environment and state on the SUT (and/or the CT host node), required for the tests to execute correctly. Examples of operations: Opening a connection to the SUT, initializing a database, running an installation script, etc. Configuration may be performed per suite, per test case group and per individual test case.
The test suite module must conform to a
A test case is considered successful if it returns to the caller, no matter
what the returned value is. A few return values have special meaning however
(such as
session(_Config) ->
{started,ServerId} = my_server:start(),
{clients,[]} = my_server:get_clients(ServerId),
MyId = self(),
connected = my_server:connect(ServerId, MyId),
{clients,[MyId]} = my_server:get_clients(ServerId),
disconnected = my_server:disconnect(ServerId, MyId),
{clients,[]} = my_server:get_clients(ServerId),
stopped = my_server:stop(ServerId).
As a test suite runs, all information (including output to
The result from each test case is recorded in a dedicated HTML log file, created for the particular test run. An overview page displays each test case represented by row in a table showing total execution time, whether the case was successful, failed or skipped, plus an optional user comment. (For a failed test case, the reason for termination is also printed in the comment field). The overview page has a link to each test case log file, providing simple navigation with any standard HTML browser.
The CT test server requires that the test suite defines and exports the following mandatory or optional callback functions:
For each test case the CT test server expects these functions: