From 2ef840647acadb489d54332f6a218dcf2e629ff9 Mon Sep 17 00:00:00 2001
From: tmanevik
- The Common Test Hook (henceforth called CTH) framework allows
- extensions of the default behaviour of Common Test by means of hooks
- before and after all test suite calls. CTHs allow advanced Common Test
- users to abstract out behaviour which is common to multiple test suites
- without littering all test suites with library calls. Some example
- usages are: logging, starting and monitoring external systems,
- building C files needed by the tests and much more! In brief, Common Test Hooks allows you to:
-
In brief, CTH allows you to do the following:
+ +The following sections describe how to use CTHs, when they are run - and how to manipulate your test results in a CTH
+The following sections describe how to use CTHs, when they are run, + and how to manipulate the test results in a CTH.
-When executing within a CTH all timetraps are shutoff. So - if your CTH never returns, the entire test run will be stalled!
+When executing within a CTH, all timetraps are shut off. So + if your CTH never returns, the entire test run is stalled.
There are multiple ways to install a CTH in your test run. You can do it - for all tests in a run, for specific test suites and for specific groups +
A CTH can be installed in multiple ways in your test run. You can do it + for all tests in a run, for specific test suites, and for specific groups within a test suite. If you want a CTH to be present in all test suites - within your test run there are three different ways to accomplish that. + within your test run, there are three ways to accomplish that, as follows:
-You can also add CTHs within a test suite. This is done by returning
-
CTHs can also be added within a test suite. This is done by returning
+
In this case,
By default each installation of a CTH will cause a new instance of it - to be activated. This can cause problems if you want to be able to - override CTHs in test specifications while still having them in the - suite info function. The +
By default, each installation of a CTH causes a new instance of it
+ to be activated. This can cause problems if you want to override
+ CTHs in test specifications while still having them in the
+ suite information function. The
By default each CTH installed will be executed in the order which +
By default, each CTH installed is executed in the order that
they are installed for init calls, and then reversed for end calls.
- This is not always wanted so common_test allows
+ This is not always desired, so
Once the CTH is installed into a certain test run it will be there until +
Once the CTH is installed into a certain test run it remains there until
its scope is expired. The scope of a CTH depends on when it is
- installed.
- The
CTHs are run with the same process scoping as normal test suites - i.e. a different process will execute the init_per_suite hooks then the - init_per_group or per_testcase hooks. So if you want to spawn a - process in the CTH you cannot link with the CTH process as it will exit - after the post hook ends. Also if you for some reason need an ETS - table with your CTH, you will have to spawn a process which handles - it.
+CTHs are run with the same process scoping as normal test suites,
+ that is, a different process executes the
It's possible in the CTH to read configuration data values
- by calling
The CT hook functions may call any of the logging functions available +
Configuration data values in the CTH can be read
+ by calling
+
The CT hook functions can call any logging function
in the
It is through CTHs possible to manipulate the results of tests and - configuration functions. The main purpose of doing this with CTHs is to - allow common patterns to be abstracted out from test test suites and applied to - multiple test suites without duplicating any code. All of the callback - functions for a CTH follow a common interface, this interface is - described below.
- -Common Test will always call all available hook functions, even pre- and post
- hooks for configuration functions that are not implemented in the suite.
+
Through CTHs the results of tests and configuration functions can be manipulated. + The main purpose to do this with CTHs is to allow common + patterns to be abstracted out from test suites and applied to + multiple test suites without duplicating any code. All the callback + functions for a CTH follow a common interface described hereafter.
+ +
- It is possible in a CTH to hook in behaviour before
-
This is done in the CTH functions called pre_<name of function>.
- All of these functions take the same three arguments:
pre_init_per_suite(SuiteName, Config, CTHState) ->
- case db:connect() of
- {error,_Reason} ->
- {{fail, "Could not connect to DB"}, CTHState};
- {ok, Handle} ->
- {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
- end.
- If using multiple CTHs, the first part of the return tuple will be
- used as input for the next CTH. So in the case above the next CTH might
+ These functions take the same three arguments,
To let the test suite continue on executing, return the configuration
+ list that you want the test to use as the result. To skip or
+ fail the test, return a tuple with
Example:
+
+ pre_init_per_suite(SuiteName, Config, CTHState) ->
+ case db:connect() of
+ {error,_Reason} ->
+ {{fail, "Could not connect to DB"}, CTHState};
+ {ok, Handle} ->
+ {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
+ end.
+
+ If you use multiple CTHs, the first part of the return tuple is
+ used as input for the next CTH. So in the previous example the next CTH can
get
It is also possible in a CTH to hook in behaviour after
-
In a CTH, behavior can be hooked in after the following functions:
+
+ This is done in the CTH functions called
The return value of the CTH function is always a combination of an +
The return value of the CTH function is always a combination of a
result for the suite/group/test and an updated
post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
- case db:check_consistency() of
- true ->
- %% DB is good, pass the test.
- {proplists:delete(tc_status, Config), CTHState};
- false ->
- %% DB is not good, mark as skipped instead of failing
- {{skip, "DB is inconsisten!"}, CTHState}
- end;
-post_end_per_testcase(_TC, Config, Return, CTHState) ->
- %% Do nothing if tc does not crash.
- {Return, CTHState}.
-
- Recovering from a testcase failure using CTHs should only be done as - a last resort. If used wrongly it could become very difficult to - determine which tests pass or fail in a test run
Example:
+
+ post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
+ case db:check_consistency() of
+ true ->
+ %% DB is good, pass the test.
+ {proplists:delete(tc_status, Config), CTHState};
+ false ->
+ %% DB is not good, mark as skipped instead of failing
+ {{skip, "DB is inconsisten!"}, CTHState}
+ end;
+ post_end_per_testcase(_TC, Config, Return, CTHState) ->
+ %% Do nothing if tc does not crash.
+ {Return, CTHState}.
+
+ Do recover from a testcase failure using CTHs only a last resort. + If used wrongly, it can be very difficult to determine which tests that + pass or fail in a test run.
After any post hook has been executed for all installed CTHs,
CTHs can be used to synchronize test runs with external user applications.
- The init function may e.g. start and/or communicate with an application that
- has the purpose of preparing the SUT for an upcoming test run, or maybe
+ The init function can, for example, start and/or communicate with an application that
+ has the purpose of preparing the SUT for an upcoming test run, or
initialize a database for saving test data to during the test run. The
- terminate function may similarly order such an application to reset the SUT
+ terminate function can similarly order such an application to reset the SUT
after the test run, and/or tell the application to finish active sessions
and terminate.
Any system error- or progress reports generated during the init- or
- termination stage will be saved in the
-
In order to ensure that Common Test doesn't start executing tests, or + +
To ensure that
The CTH below will log information about a test run into a format
- parseable by
The following CTH logs information about a test run into a format
+ parseable by
%%% @doc Common Test Example Common Test Hook module.
--module(example_cth).
-
-%% Callbacks
--export([id/1]).
--export([init/2]).
-
--export([pre_init_per_suite/3]).
--export([post_init_per_suite/4]).
--export([pre_end_per_suite/3]).
--export([post_end_per_suite/4]).
-
--export([pre_init_per_group/3]).
--export([post_init_per_group/4]).
--export([pre_end_per_group/3]).
--export([post_end_per_group/4]).
-
--export([pre_init_per_testcase/3]).
--export([post_end_per_testcase/4]).
-
--export([on_tc_fail/3]).
--export([on_tc_skip/3]).
-
--export([terminate/1]).
-
--record(state, { file_handle, total, suite_total, ts, tcs, data }).
-
-%% @doc Return a unique id for this CTH.
-id(Opts) ->
- proplists:get_value(filename, Opts, "/tmp/file.log").
-
-%% @doc Always called before any other callback function. Use this to initiate
-%% any common state.
-init(Id, Opts) ->
- {ok,D} = file:open(Id,[write]),
- {ok, #state{ file_handle = D, total = 0, data = [] }}.
-
-%% @doc Called before init_per_suite is called.
-pre_init_per_suite(Suite,Config,State) ->
- {Config, State#state{ suite_total = 0, tcs = [] }}.
-
-%% @doc Called after init_per_suite.
-post_init_per_suite(Suite,Config,Return,State) ->
- {Return, State}.
-
-%% @doc Called before end_per_suite.
-pre_end_per_suite(Suite,Config,State) ->
- {Config, State}.
-
-%% @doc Called after end_per_suite.
-post_end_per_suite(Suite,Config,Return,State) ->
- Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
- {Return, State#state{ data = [Data | State#state.data] ,
- total = State#state.total + State#state.suite_total } }.
-
-%% @doc Called before each init_per_group.
-pre_init_per_group(Group,Config,State) ->
- {Config, State}.
-
-%% @doc Called after each init_per_group.
-post_init_per_group(Group,Config,Return,State) ->
- {Return, State}.
-
-%% @doc Called after each end_per_group.
-pre_end_per_group(Group,Config,State) ->
- {Config, State}.
-
-%% @doc Called after each end_per_group.
-post_end_per_group(Group,Config,Return,State) ->
- {Return, State}.
-
-%% @doc Called before each test case.
-pre_init_per_testcase(TC,Config,State) ->
- {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
-
-%% @doc Called after each test case.
-post_end_per_testcase(TC,Config,Return,State) ->
- TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
- {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
-
-%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
-%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
-on_tc_fail(TC, Reason, State) ->
- State.
-
-%% @doc Called when a test case is skipped by either user action
-%% or due to an init function failing.
-on_tc_skip(TC, Reason, State) ->
- State.
-
-%% @doc Called when the scope of the CTH is done
-terminate(State) ->
- io:format(State#state.file_handle, "~p.~n",
- [{test_run, State#state.total, State#state.data}]),
- file:close(State#state.file_handle),
- ok.
+
+ %%% @doc Common Test Example Common Test Hook module.
+ -module(example_cth).
+
+ %% Callbacks
+ -export([id/1]).
+ -export([init/2]).
+
+ -export([pre_init_per_suite/3]).
+ -export([post_init_per_suite/4]).
+ -export([pre_end_per_suite/3]).
+ -export([post_end_per_suite/4]).
+
+ -export([pre_init_per_group/3]).
+ -export([post_init_per_group/4]).
+ -export([pre_end_per_group/3]).
+ -export([post_end_per_group/4]).
+
+ -export([pre_init_per_testcase/3]).
+ -export([post_end_per_testcase/4]).
+
+ -export([on_tc_fail/3]).
+ -export([on_tc_skip/3]).
+
+ -export([terminate/1]).
+
+ -record(state, { file_handle, total, suite_total, ts, tcs, data }).
+
+ %% @doc Return a unique id for this CTH.
+ id(Opts) ->
+ proplists:get_value(filename, Opts, "/tmp/file.log").
+
+ %% @doc Always called before any other callback function. Use this to initiate
+ %% any common state.
+ init(Id, Opts) ->
+ {ok,D} = file:open(Id,[write]),
+ {ok, #state{ file_handle = D, total = 0, data = [] }}.
+
+ %% @doc Called before init_per_suite is called.
+ pre_init_per_suite(Suite,Config,State) ->
+ {Config, State#state{ suite_total = 0, tcs = [] }}.
+
+ %% @doc Called after init_per_suite.
+ post_init_per_suite(Suite,Config,Return,State) ->
+ {Return, State}.
+
+ %% @doc Called before end_per_suite.
+ pre_end_per_suite(Suite,Config,State) ->
+ {Config, State}.
+
+ %% @doc Called after end_per_suite.
+ post_end_per_suite(Suite,Config,Return,State) ->
+ Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
+ {Return, State#state{ data = [Data | State#state.data] ,
+ total = State#state.total + State#state.suite_total } }.
+
+ %% @doc Called before each init_per_group.
+ pre_init_per_group(Group,Config,State) ->
+ {Config, State}.
+
+ %% @doc Called after each init_per_group.
+ post_init_per_group(Group,Config,Return,State) ->
+ {Return, State}.
+
+ %% @doc Called after each end_per_group.
+ pre_end_per_group(Group,Config,State) ->
+ {Config, State}.
+
+ %% @doc Called after each end_per_group.
+ post_end_per_group(Group,Config,Return,State) ->
+ {Return, State}.
+
+ %% @doc Called before each test case.
+ pre_init_per_testcase(TC,Config,State) ->
+ {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
+
+ %% @doc Called after each test case.
+ post_end_per_testcase(TC,Config,Return,State) ->
+ TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
+ {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
+
+ %% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
+ %% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
+ on_tc_fail(TC, Reason, State) ->
+ State.
+
+ %% @doc Called when a test case is skipped by either user action
+ %% or due to an init function failing.
+ on_tc_skip(TC, Reason, State) ->
+ State.
+
+ %% @doc Called when the scope of the CTH is done
+ terminate(State) ->
+ io:format(State#state.file_handle, "~p.~n",
+ [{test_run, State#state.total, State#state.data}]),
+ file:close(State#state.file_handle),
+ ok.
Common Test is delivered with a couple of general purpose CTHs that
- can be enabled by the user to provide some generic testing functionality.
- Some of these are enabled by default when starting running common_test,
- they can be disabled by setting
Built-in
+Captures all
Not built-in
+Captures all test results and outputs them as surefire
+ XML into a file. The created file is by default
+ called
If option
gives an URL attribute value similar to
+ +Surefire XML can, for example, be used by Jenkins to display test + results.
+