The
Test cases can be executed individually or in batches.
The System Under Test (SUT) can consist of one or more target
nodes.
In a black-box testing scenario,
A test case can handle several connections to one or
more target systems, instruments, and traffic generators in
parallel to perform the necessary actions for a test.
The handling of many connections in parallel is one of
the major strengths of
Test suites are organized in test directories and each test suite
can have a separate data directory. Typically, these files and directories
are version-controlled similar to other forms of source code (possibly by
a version control system like GIT or Subversion). However,
Support libraries contain functions that are useful for all test suites,
or for test suites in a specific functional area or subsystem.
In addition to the general support libraries provided by the
Testing is performed by running test suites (sets of test cases) or
individual test cases. A test suite is implemented as an Erlang module named
Sets of test cases, called test case groups, can also be defined. A test case group can have execution properties associated with it. Execution properties specify if the test cases in the group are to be executed in random order, in parallel, or in sequence, and if the execution of the group is to be repeated. Test case groups can also be nested (that is, a group can, besides test cases, contain subgroups).
Besides test cases and groups, the test suite can also contain configuration
functions. These functions are meant to be used for setting up (and verifying)
environment and state in the SUT (and/or the
The test suite module must conform to a
A test case is considered successful if it returns to the caller, no matter what the returned value is. However, a few return values have special meaning as follows:
A test case failure is specified as a runtime error (a crash), no matter what the reason for termination is. If you use Erlang pattern matching effectively, you can take advantage of this property. The result is concise and readable test case functions that look much more like scripts than actual programs. A simple example:
session(_Config) ->
{started,ServerId} = my_server:start(),
{clients,[]} = my_server:get_clients(ServerId),
MyId = self(),
connected = my_server:connect(ServerId, MyId),
{clients,[MyId]} = my_server:get_clients(ServerId),
disconnected = my_server:disconnect(ServerId, MyId),
{clients,[]} = my_server:get_clients(ServerId),
stopped = my_server:stop(ServerId).
As a test suite runs, all information (including output to
The result from each test case is recorded in a dedicated HTML log file, created for the particular test run. An overview page displays each test case represented by a table row showing total execution time, if the case was successful, failed, or skipped, plus an optional user comment. For a failed test case, the reason for termination is also printed in the comment field. The overview page has a link to each test case log file, providing simple navigation with any standard HTML browser.
The
Returns a list of all test cases and groups in the suite. (Mandatory)
Information function used to return properties for the suite. (Optional)
For declaring test case groups. (Optional)
Suite level configuration function, executed before the first test case. (Optional)
Suite level configuration function, executed after the last test case. (Optional)
Information function used to return properties for a test case group. (Optional)
Configuration function for a group, executed before the first test case. (Optional)
Configuration function for a group, executed after the last test case. (Optional)
Configuration function for a testcase, executed before each test case. (Optional)
Configuration function for a testcase, executed after each test case. (Optional)
For each test case, the
Information function that returns a list of test case properties. (Optional)
The test case function.