The Common Test Hook (CTH) framework allows
extensions of the default behavior of
In brief, CTH allows you to do the following:
The following sections describe how to use CTHs, when they are run, and how to manipulate the test results in a CTH.
When executing within a CTH, all timetraps are shut off. So if your CTH never returns, the entire test run is stalled.
A CTH can be installed in multiple ways in your test run. You can do it for all tests in a run, for specific test suites, and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run, there are three ways to accomplish that, as follows:
CTHs can also be added within a test suite. This is done by returning
In this case,
By default, each installation of a CTH causes a new instance of it
to be activated. This can cause problems if you want to override
CTHs in test specifications while still having them in the
suite information function. The
By default, each CTH installed is executed in the order that
they are installed for init calls, and then reversed for end calls.
This is not always desired, so
Once the CTH is installed into a certain test run it remains there until
its scope is expired. The scope of a CTH depends on when it is
installed, see the following table.
Function
CTHs are run with the same process scoping as normal test suites,
that is, a different process executes the
Configuration data values in the CTH can be read
by calling
The CT hook functions can call any logging function
in the
Through CTHs the results of tests and configuration functions can be manipulated. The main purpose to do this with CTHs is to allow common patterns to be abstracted out from test suites and applied to multiple test suites without duplicating any code. All the callback functions for a CTH follow a common interface described hereafter.
In a CTH, the behavior can be hooked in before the following functions:
This is done in the CTH functions called pre_<name of function>.
These functions take the same three arguments,
To let the test suite continue on executing, return the configuration
list that you want the test to use as the result. To skip or
fail the test, return a tuple with
Example:
pre_init_per_suite(SuiteName, Config, CTHState) ->
case db:connect() of
{error,_Reason} ->
{{fail, "Could not connect to DB"}, CTHState};
{ok, Handle} ->
{[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
end.
If you use multiple CTHs, the first part of the return tuple is
used as input for the next CTH. So in the previous example the next CTH can
get
In a CTH, behavior can be hooked in after the following functions:
This is done in the CTH functions called
The return value of the CTH function is always a combination of a
result for the suite/group/test and an updated
Example:
post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
case db:check_consistency() of
true ->
%% DB is good, pass the test.
{proplists:delete(tc_status, Config), CTHState};
false ->
%% DB is not good, mark as skipped instead of failing
{{skip, "DB is inconsisten!"}, CTHState}
end;
post_end_per_testcase(_TC, Config, Return, CTHState) ->
%% Do nothing if tc does not crash.
{Return, CTHState}.
Do recover from a testcase failure using CTHs only a last resort. If used wrongly, it can be very difficult to determine which tests that pass or fail in a test run.
After any post hook has been executed for all installed CTHs,
CTHs can be used to synchronize test runs with external user applications.
The init function can, for example, start and/or communicate with an application that
has the purpose of preparing the SUT for an upcoming test run, or
initialize a database for saving test data to during the test run. The
terminate function can similarly order such an application to reset the SUT
after the test run, and/or tell the application to finish active sessions
and terminate.
Any system error- or progress reports generated during the init- or
termination stage are saved in the
To ensure that
The following CTH logs information about a test run into a format
parseable by
%%% @doc Common Test Example Common Test Hook module.
-module(example_cth).
%% Callbacks
-export([id/1]).
-export([init/2]).
-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).
-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).
-export([pre_init_per_testcase/3]).
-export([post_init_per_testcase/4]).
-export([pre_end_per_testcase/3]).
-export([post_end_per_testcase/4]).
-export([on_tc_fail/3]).
-export([on_tc_skip/3]).
-export([terminate/1]).
-record(state, { file_handle, total, suite_total, ts, tcs, data }).
%% @doc Return a unique id for this CTH.
id(Opts) ->
proplists:get_value(filename, Opts, "/tmp/file.log").
%% @doc Always called before any other callback function. Use this to initiate
%% any common state.
init(Id, Opts) ->
{ok,D} = file:open(Id,[write]),
{ok, #state{ file_handle = D, total = 0, data = [] }}.
%% @doc Called before init_per_suite is called.
pre_init_per_suite(Suite,Config,State) ->
{Config, State#state{ suite_total = 0, tcs = [] }}.
%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
{Return, State}.
%% @doc Called before end_per_suite.
pre_end_per_suite(Suite,Config,State) ->
{Config, State}.
%% @doc Called after end_per_suite.
post_end_per_suite(Suite,Config,Return,State) ->
Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
{Return, State#state{ data = [Data | State#state.data] ,
total = State#state.total + State#state.suite_total } }.
%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called before each end_per_group.
pre_end_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each end_per_group.
post_end_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called before each init_per_testcase.
pre_init_per_testcase(TC,Config,State) ->
{Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
%% Called after each init_per_testcase (immediately before the test case).
post_init_per_testcase(TC,Config,Return,State) ->
{Return, State}
%% @doc Called before each end_per_testcase (immediately after the test case).
pre_end_per_testcase(TC,Config,State) ->
{Config, State}.
%% @doc Called after each end_per_testcase.
post_end_per_testcase(TC,Config,Return,State) ->
TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
{Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
State.
%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.
on_tc_skip(TC, Reason, State) ->
State.
%% @doc Called when the scope of the CTH is done
terminate(State) ->
io:format(State#state.file_handle, "~p.~n",
[{test_run, State#state.total, State#state.data}]),
file:close(State#state.file_handle),
ok.
Built-in
Captures all
Not built-in
Captures all test results and outputs them as surefire
XML into a file. The created file is by default
called
If option
gives an URL attribute value similar to
Surefire XML can, for example, be used by Jenkins to display test results.