The Common Test Hook (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of hooks before and after all test suite calls. CTHs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!
In brief, Common Test Hooks allows you to:
The following sections describe how to use CTHs, when they are run and how to manipulate your test results in a CTH
When executing within a CTH all timetraps are shutoff. So if your CTH never returns, the entire test run will be stalled!
There are multiple ways to install a CTH in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run there are three different ways to accomplish that.
You can also add CTHs within a test suite. This is done by returning
By default each installation of a CTH will cause a new instance of it
to be activated. This can cause problems if you want to be able to
override CTHs in test specifications while still having them in the
suite info function. The
By default each CTH installed will be executed in the order which
they are installed for init calls, and then reversed for end calls.
This is not always wanted so common_test allows
the user to specify a priority for each hook. The priority can either
be specified in the CTH
Once the CTH is installed into a certain test run it will be there until
its scope is expired. The scope of a CTH depends on when it is
installed.
The
CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.
It's possible in the CTH to read configuration data values
by calling
The CT hook functions may call any of the logging functions available
in the
It is through CTHs possible to manipulate the results of tests and configuration functions. The main purpose of doing this with CTHs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for a CTH follow a common interface, this interface is described below.
Common Test will always call all available hook functions, even pre- and post
hooks for configuration functions that are not implemented in the suite.
For example,
It is possible in a CTH to hook in behaviour before
pre_init_per_suite(SuiteName, Config, CTHState) ->
case db:connect() of
{error,_Reason} ->
{{fail, "Could not connect to DB"}, CTHState};
{ok, Handle} ->
{[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
end.
If using multiple CTHs, the first part of the return tuple will be
used as input for the next CTH. So in the case above the next CTH might
get
It is also possible in a CTH to hook in behaviour after
The return value of the CTH function is always a combination of an
result for the suite/group/test and an updated
post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
case db:check_consistency() of
true ->
%% DB is good, pass the test.
{proplists:delete(tc_status, Config), CTHState};
false ->
%% DB is not good, mark as skipped instead of failing
{{skip, "DB is inconsisten!"}, CTHState}
end;
post_end_per_testcase(_TC, Config, Return, CTHState) ->
%% Do nothing if tc does not crash.
{Return, CTHState}.
Recovering from a testcase failure using CTHs should only be done as a last resort. If used wrongly it could become very difficult to determine which tests pass or fail in a test run
After any post hook has been executed for all installed CTHs,
CTHs can be used to synchronize test runs with external user applications.
The init function may e.g. start and/or communicate with an application that
has the purpose of preparing the SUT for an upcoming test run, or maybe
initialize a database for saving test data to during the test run. The
terminate function may similarly order such an application to reset the SUT
after the test run, and/or tell the application to finish active sessions
and terminate.
Any system error- or progress reports generated during the init- or
termination stage will be saved in the
In order to ensure that Common Test doesn't start executing tests, or
closes its log files and shuts down, before the external application
is ready for it, Common Test may be synchronized with the application.
During startup and shutdown, Common Test can be suspended, simply by
having a CTH evaluate a
The CTH below will log information about a test run into a format
parseable by
%%% @doc Common Test Example Common Test Hook module.
-module(example_cth).
%% Callbacks
-export([id/1]).
-export([init/2]).
-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).
-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).
-export([pre_init_per_testcase/3]).
-export([post_end_per_testcase/4]).
-export([on_tc_fail/3]).
-export([on_tc_skip/3]).
-export([terminate/1]).
-record(state, { file_handle, total, suite_total, ts, tcs, data }).
%% @doc Return a unique id for this CTH.
id(Opts) ->
proplists:get_value(filename, Opts, "/tmp/file.log").
%% @doc Always called before any other callback function. Use this to initiate
%% any common state.
init(Id, Opts) ->
{ok,D} = file:open(Id,[write]),
{ok, #state{ file_handle = D, total = 0, data = [] }}.
%% @doc Called before init_per_suite is called.
pre_init_per_suite(Suite,Config,State) ->
{Config, State#state{ suite_total = 0, tcs = [] }}.
%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
{Return, State}.
%% @doc Called before end_per_suite.
pre_end_per_suite(Suite,Config,State) ->
{Config, State}.
%% @doc Called after end_per_suite.
post_end_per_suite(Suite,Config,Return,State) ->
Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
{Return, State#state{ data = [Data | State#state.data] ,
total = State#state.total + State#state.suite_total } }.
%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called after each end_per_group.
pre_end_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each end_per_group.
post_end_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called before each test case.
pre_init_per_testcase(TC,Config,State) ->
{Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
%% @doc Called after each test case.
post_end_per_testcase(TC,Config,Return,State) ->
TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
{Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
State.
%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.
on_tc_skip(TC, Reason, State) ->
State.
%% @doc Called when the scope of the CTH is done
terminate(State) ->
io:format(State#state.file_handle, "~p.~n",
[{test_run, State#state.total, State#state.data}]),
file:close(State#state.file_handle),
ok.
Common Test is delivered with a couple of general purpose CTHs that
can be enabled by the user to provide some generic testing functionality.
Some of these are enabled by default when starting running common_test,
they can be disabled by setting