20112012 Ericsson AB. All Rights Reserved. The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/. Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Common Test Hooks Lukas Larsson ct_hooks_chapter.xml
General

The Common Test Hook (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of hooks before and after all test suite calls. CTHs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!

In brief, Common Test Hooks allows you to:

Manipulate the runtime config before each suite configuration call Manipulate the return of all suite configuration calls and in extension the result of the test themselves.

The following sections describe how to use CTHs, when they are run and how to manipulate your test results in a CTH

When executing within a CTH all timetraps are shutoff. So if your CTH never returns, the entire test run will be stalled!
Installing a CTH

There are multiple ways to install a CTH in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run there are three different ways to accomplish that.

Add -ct_hooks as an argument to ct_run. To add multiple CTHs using this method append them to each other using the keyword and, i.e. ct_run -ct_hooks cth1 [{debug,true}] and cth2 .... Add the ct_hooks tag to your Test Specification Add the ct_hooks tag to your call to ct:run_test/1

You can also add CTHs within a test suite. This is done by returning {ct_hooks,[CTH]} in the config list from suite/0, init_per_suite/1 or init_per_group/2. CTH in this case can be either only the module name of the CTH or a tuple with the module name and the initial arguments and optionally the hook priority of the CTH. Eg: {ct_hooks,[my_cth_module]} or {ct_hooks,[{my_cth_module,[{debug,true}]}]} or {ct_hooks,[{my_cth_module,[{debug,true}],500}]}

Overriding CTHs

By default each installation of a CTH will cause a new instance of it to be activated. This can cause problems if you want to be able to override CTHs in test specifications while still having them in the suite info function. The id/1 callback exists to address this problem. By returning the same id in both places, Common Test knows that this CTH has already been installed and will not try to install it again.

CTH Execution order

By default each CTH installed will be executed in the order which they are installed for init calls, and then reversed for end calls. This is not always wanted so common_test allows the user to specify a priority for each hook. The priority can either be specified in the CTH init/2 function or when installing the hook. The priority given at installation will override the priority returned by the CTH.

CTH Scope

Once the CTH is installed into a certain test run it will be there until its scope is expired. The scope of a CTH depends on when it is installed. The init/2 is called at the beginning of the scope and the terminate/1 function is called when the scope ends.

CTH Installed in CTH scope begins before CTH scope ends after ct_run the first test suite is to be run. the last test suite has been run. ct:run_test the first test suite is to be run. the last test suite has been run. Test Specification the first test suite is to be run. the last test suite has been run. suite/0 pre_init_per_suite/3 is called. post_end_per_suite/4 has been called for that test suite. init_per_suite/1 post_init_per_suite/4 is called. post_end_per_suite/4 has been called for that test suite. init_per_group/2 post_init_per_group/4 is called. post_end_per_group/4 has been called for that group. Scope of a CTH
CTH Processes and Tables

CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.

External configuration data and Logging

It's possible in the CTH to read configuration data values by calling ct:get_config/1/2/3 (as explained in the External configuration data chapter). The config variables in question must, as always, first have been required by means of a suite-, group-, or test case info function, or the ct:require/1/2 function. Note that the latter can also be used in CT hook functions.

The CT hook functions may call any of the logging functions available in the ct interface to print information to the log files, or to add comments in the suite overview page.

Manipulating tests

It is through CTHs possible to manipulate the results of tests and configuration functions. The main purpose of doing this with CTHs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for a CTH follow a common interface, this interface is described below.

Common Test will always call all available hook functions, even pre- and post hooks for configuration functions that are not implemented in the suite. For example, pre_init_per_suite(x_SUITE, ...) and post_init_per_suite(x_SUITE, ...) will be called for test suite x_SUITE, even if it doesn't export init_per_suite/1. This feature makes it possible to use hooks as configuration fallbacks, or even completely replace all configuration functions with hook functions.

Pre Hooks

It is possible in a CTH to hook in behaviour before init_per_suite, init_per_group, init_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called pre_<name of function>. All of these functions take the same three arguments: Name, Config and CTHState. The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the test suite to continue on executing you should return the config list which you want the test to use as the result. If you for some reason want to skip/fail the test, return a tuple with skip or fail and a reason as the result. Example:

pre_init_per_suite(SuiteName, Config, CTHState) -> case db:connect() of {error,_Reason} -> {{fail, "Could not connect to DB"}, CTHState}; {ok, Handle} -> {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} end.
Post Hooks

It is also possible in a CTH to hook in behaviour after init_per_suite, init_per_group, end_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called post_<name of function>. All of these function take the same four arguments: Name, Config, Return and CTHState. Config in this case is the same Config as the testcase is called with. Return is the value returned by the testcase. If the testcase failed by crashing, Return will be {'EXIT',{{Error,Reason},Stacktrace}}.

The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the callback to not affect the outcome of the test you should return the Return data as it is given to the CTH. You can also modify the result of the test. By returning the Config list with the tc_status element removed you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip the test case in the post hook. Example:

post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -> case db:check_consistency() of true -> %% DB is good, pass the test. {proplists:delete(tc_status, Config), CTHState}; false -> %% DB is not good, mark as skipped instead of failing {{skip, "DB is inconsisten!"}, CTHState} end; post_end_per_testcase(_TC, Config, Return, CTHState) -> %% Do nothing if tc does not crash. {Return, CTHState}. Recovering from a testcase failure using CTHs should only be done as a last resort. If used wrongly it could become very difficult to determine which tests pass or fail in a test run
Skip and Fail hooks

After any post hook has been executed for all installed CTHs, on_tc_fail or on_tc_skip might be called if the testcase failed or was skipped respectively. You cannot affect the outcome of the tests any further at this point.

Example CTH

The CTH below will log information about a test run into a format parseable by file:consult/1.

%%% @doc Common Test Example Common Test Hook module. -module(example_cth). %% Callbacks -export([id/1]). -export([init/2]). -export([pre_init_per_suite/3]). -export([post_init_per_suite/4]). -export([pre_end_per_suite/3]). -export([post_end_per_suite/4]). -export([pre_init_per_group/3]). -export([post_init_per_group/4]). -export([pre_end_per_group/3]). -export([post_end_per_group/4]). -export([pre_init_per_testcase/3]). -export([post_end_per_testcase/4]). -export([on_tc_fail/3]). -export([on_tc_skip/3]). -export([terminate/1]). -record(state, { file_handle, total, suite_total, ts, tcs, data }). %% @doc Return a unique id for this CTH. id(Opts) -> proplists:get_value(filename, Opts, "/tmp/file.log"). %% @doc Always called before any other callback function. Use this to initiate %% any common state. init(Id, Opts) -> {ok,D} = file:open(Id,[write]), {ok, #state{ file_handle = D, total = 0, data = [] }}. %% @doc Called before init_per_suite is called. pre_init_per_suite(Suite,Config,State) -> {Config, State#state{ suite_total = 0, tcs = [] }}. %% @doc Called after init_per_suite. post_init_per_suite(Suite,Config,Return,State) -> {Return, State}. %% @doc Called before end_per_suite. pre_end_per_suite(Suite,Config,State) -> {Config, State}. %% @doc Called after end_per_suite. post_end_per_suite(Suite,Config,Return,State) -> Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)}, {Return, State#state{ data = [Data | State#state.data] , total = State#state.total + State#state.suite_total } }. %% @doc Called before each init_per_group. pre_init_per_group(Group,Config,State) -> {Config, State}. %% @doc Called after each init_per_group. post_init_per_group(Group,Config,Return,State) -> {Return, State}. %% @doc Called after each end_per_group. pre_end_per_group(Group,Config,State) -> {Config, State}. %% @doc Called after each end_per_group. post_end_per_group(Group,Config,Return,State) -> {Return, State}. %% @doc Called before each test case. pre_init_per_testcase(TC,Config,State) -> {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }. %% @doc Called after each test case. post_end_per_testcase(TC,Config,Return,State) -> TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)}, {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }. %% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group, %% post_end_per_group and post_end_per_testcase if the suite, group or test case failed. on_tc_fail(TC, Reason, State) -> State. %% @doc Called when a test case is skipped by either user action %% or due to an init function failing. on_tc_skip(TC, Reason, State) -> State. %% @doc Called when the scope of the CTH is done terminate(State) -> io:format(State#state.file_handle, "~p.~n", [{test_run, State#state.total, State#state.data}]), file:close(State#state.file_handle), ok.
Built-in CTHs

Common Test is delivered with a couple of general purpose CTHs that can be enabled by the user to provide some generic testing functionality. Some of these are enabled by default when starting running common_test, they can be disabled by setting enable_builtin_hooks to false on the command line or in the test specification. In the table below there is a list of all current CTHs which are delivered with Common Test.

CTH Name Is Built-in Description cth_log_redirect yes Captures all error_logger and SASL logging events and prints them to the current test case log. If an event can not be associated with a testcase it will be printed in the common test framework log. This will happen for testcases which are run in parallel and events which occur inbetween testcases. You can configure the level of SASL events report using the normal SASL mechanisms. cth_surefire no Captures all test results and outputs them as surefire XML into a file. The file which is created is by default called junit_report.xml. The name can be by setting the path option for this hook. e.g. -ct_hooks cth_surefire [{path,"/tmp/report.xml"}] Surefire XML can forinstance be used by Jenkins to display test results.