This feature is in alpha release right now. This means that the interface may change in the future and that there may be bugs. We encourage you to use this feature, but be prepared that there might be bugs and that the interface might change inbetween releases.
The Suite Callback (henceforth called SCB) framework allows extensions of the default behaviour of Common Test by means of callbacks before and after all test suite calls. SCBs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!
In brief, Suite Callbacks allows you to:
The following sections describe how to use SCBs, when they are run and how to manipulate your test results in an SCB
When executing within an SCB all timetraps are shutoff. So if your SCB never returns, the entire test run will be stalled!
There are multiple ways to install an SCB in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want an SCB to be present in all test suites within your testrun there are three different ways to accomplish that.
You can also add SCBs within a test suite. This is done by returning
By default each installation of an SCB will cause a new instance of it
to be activated. This can cause problems if you want to be able to
override SCBs in testspecifications while still having them in the
suite info function. The
Once the SCB is installed into a certain test run it will be there until
it's scope is expired. The scope of an SCB depends on when it is
installed.
The
CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.
It is through SCB's possible to manipulate the results of tests and configuration functions. The main purpose of doing this with SCBs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for an SCB follow a common interface, this interface is described below.
It is possible in an SCB to hook in behaviour before
pre_init_per_suite(SuiteName, Config, SCBState) ->
case db:connect() of
{error,_Reason} ->
{{fail, "Could not connect to DB"}, SCBState};
{ok, Handle} ->
{[{db_handle, Handle} | Config], SCBState#state{ handle = Handle }}
end.
It is also possible in an SCB to hook in behaviour after
The return value of the SCB function is always a combination of an
result for the suite/group/test and an updated
post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, SCBState) ->
case db:check_consistency() of
true ->
%% DB is good, pass the test.
{proplists:delete(tc_status, Config), SCBState};
false ->
%% DB is not good, mark as skipped instead of failing
{{skip, "DB is inconsisten!"}, SCBState}
end;
post_end_per_testcase(_TC, Config, Return, SCBState) ->
%% Do nothing if tc does not crash.
{Return, SCBState}.
After any post hook has been executed for all installed SCBs,
The SCB below will log information about a test run into a format
parseable by
%%% @doc Common Test Example Suite Callback module.
-module(example_scb).
%% Suite Callbacks
-export([id/1]).
-export([init/2]).
-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).
-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).
-export([pre_init_per_testcase/3]).
-export([post_end_per_testcase/4]).
-export([on_tc_fail/3]).
-export([on_tc_skip/3]).
-export([terminate/1]).
-record(state, { file_handle, total, suite_total, ts, tcs, data }).
%% @doc Return a unique id for this SCB.
id(Opts) ->
proplists:get_value(filename, Opts, "/tmp/file.log").
%% @doc Always called before any other callback function. Use this to initiate
%% any common state.
init(Id, Opts) ->
{ok,D} = file:open(Id,[write]),
#state{ file_handle = D, total = 0, data = [] }.
%% @doc Called before init_per_suite is called.
pre_init_per_suite(Suite,Config,State) ->
{Config, State#state{ suite_total = 0, tcs = [] }}.
%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
{Return, State}.
%% @doc Called before end_per_suite.
pre_end_per_suite(Suite,Config,State) ->
{Config, State}.
%% @doc Called after end_per_suite.
post_end_per_suite(Suite,Config,Return,State) ->
Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
{Return, State#state{ data = [Data | State#state.data] ,
total = State#state.total + State#state.suite_total } }.
%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called after each end_per_group.
pre_end_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each end_per_group.
post_end_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called before each test case.
pre_init_per_testcase(TC,Config,State) ->
{Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
%% @doc Called after each test case.
post_end_per_testcase(TC,Config,Return,State) ->
TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
{Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
State.
%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.
on_tc_skip(TC, Reason, State) ->
State.
%% @doc Called when the scope of the SCB is done
terminate(State) ->
io:format(State#state.file_handle, "~p.~n",
[{test_run, State#state.total, State#state.data}]),
file:close(State#state.file_handle),
ok.