20112011 Ericsson AB. All Rights Reserved. The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/. Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Suite Callbacks Lukas Larsson suite_callbacks_chapter.xml
General

This feature is in alpha release right now. This means that the interface may change in the future and that there may be bugs. We encourage you to use this feature, but be prepared that there might be bugs and that the interface might change inbetween releases.

The Suite Callback (henceforth called SCB) framework allows extensions of the default behaviour of Common Test by means of callbacks before and after all test suite calls. SCBs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!

In brief, Suite Callbacks allows you to:

Manipulate the runtime config before each suite configuration calls Manipulate the return of all suite configuration calls and in extension the result of the test themselves.

The following sections describe how to use SCBs, when they are run and how to manipulate your test results in an SCB

When executing within an SCB all timetraps are shutoff. So if your SCB never returns, the entire test run will be stalled!
Installing an SCB

There are multiple ways to install an SCB in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want an SCB to be present in all test suites within your testrun there are three different ways to accomplish that.

Add -suite_callbacks as an argument to ct_run. To add multiple SCBs using this method append them to each other using the keyword and, i.e. ct_run -suite_callbacks scb1 [{debug,true}] and scb2 .... Add the suite_callbacks tag to your Test Specification Add the suite_callbacks tag to your call to ct:run_test/1

You can also add SCBs within a test suite. This is done by returning {suite_callbacks,[SCB]} in the config list from suite/0, init_per_suite/1 or init_per_group/2. SCB in this case can be either only the module name of the SCB or a tuple with the module name and the initial arguments to the SCB. Eg: {suite_callbacks,[my_scb_module]} or {suite_callbacks,[{my_scb_module,[{debug,true}]}]}

Overriding SCBs

By default each installation of an SCB will cause a new instance of it to be activated. This can cause problems if you want to be able to override SCBs in testspecifications while still having them in the suite info function. The id/1 callback exists to address this problem. By returning the same id in both places, Common Test knows that this SCB has already been installed and will not try to install it again.

SCB Scope

Once the SCB is installed into a certain test run it will be there until it's scope is expired. The scope of an SCB depends on when it is installed. The init/2 is called at the beginning of the scope and the terminate/1 function is called when the scope ends.

SCB Installed in SCB scope begins before SCB scope ends after ct_run the first test suite is to be run. the last test suite has been run. ct:run_test the first test suite is to be run. the last test suite has been run. Test Specification the first test suite is to be run. the last test suite has been run. suite/0 pre_init_per_suite/3 is called. post_end_per_suite/4 has been called for that test suite. init_per_suite/1 post_init_per_suite/4 is called. post_end_per_suite/4 has been called for that test suite. init_per_group/2 post_init_per_group/4 is called. post_end_per_group/4 has been called for that group. Scope of an SCB
CTH Processes and Tables

CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.

Manipulating tests

It is through SCB's possible to manipulate the results of tests and configuration functions. The main purpose of doing this with SCBs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for an SCB follow a common interface, this interface is described below.

Pre test manipulation

It is possible in an SCB to hook in behaviour before init_per_suite, init_per_group, init_per_testcase, end_per_group and end_per_suite. This is done in the SCB functions called pre_<name of function>. All of these function take the same three arguments: Name, Config and SCBState. The return value of the SCB function is always a combination of an result for the suite/group/test and an updated SCBState. If you want the test suite to continue on executing you should return the config list which you want the test to use as the result. If you for some reason want to skip/fail the test, return a tuple with skip or fail and a reason as the result. Example:

pre_init_per_suite(SuiteName, Config, SCBState) -> case db:connect() of {error,_Reason} -> {{fail, "Could not connect to DB"}, SCBState}; {ok, Handle} -> {[{db_handle, Handle} | Config], SCBState#state{ handle = Handle }} end.
Post test manipulation

It is also possible in an SCB to hook in behaviour after init_per_suite, init_per_group, end_per_testcase, end_per_group and end_per_suite. This is done in the SCB functions called post_<name of function>. All of these function take the same four arguments: Name, Config, Return and SCBState. Config in this case is the same Config as the testcase is called with. Return is the value returned by the testcase. If the testcase failed by crashing, Return will be {'EXIT',{{Error,Reason},Stacktrace}}.

The return value of the SCB function is always a combination of an result for the suite/group/test and an updated SCBState. If you want the callback to not affect the outcome of the test you should return the Return data as it is given to the SCB. You can also modify the result of the test. By returning the Config list with the tc_status element removed you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip the test case in the post hook. Example:

post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, SCBState) -> case db:check_consistency() of true -> %% DB is good, pass the test. {proplists:delete(tc_status, Config), SCBState}; false -> %% DB is not good, mark as skipped instead of failing {{skip, "DB is inconsisten!"}, SCBState} end; post_end_per_testcase(_TC, Config, Return, SCBState) -> %% Do nothing if tc does not crash. {Return, SCBState}. Recovering from a testcase failure using SCBs should only be done as a last resort. If used wrongly it could become very difficult to determine which tests pass or fail in a test run
Skip and Fail

After any post hook has been executed for all installed SCBs, on_tc_fail or on_tc_skip might be called if the testcase failed or was skipped respectively. You cannot affect the outcome of the tests any further at this point.

Example SCB

The SCB below will log information about a test run into a format parseable by file:consult/1.

%%% @doc Common Test Example Suite Callback module. -module(example_scb). %% Suite Callbacks -export([id/1]). -export([init/2]). -export([pre_init_per_suite/3]). -export([post_init_per_suite/4]). -export([pre_end_per_suite/3]). -export([post_end_per_suite/4]). -export([pre_init_per_group/3]). -export([post_init_per_group/4]). -export([pre_end_per_group/3]). -export([post_end_per_group/4]). -export([pre_init_per_testcase/3]). -export([post_end_per_testcase/4]). -export([on_tc_fail/3]). -export([on_tc_skip/3]). -export([terminate/1]). -record(state, { file_handle, total, suite_total, ts, tcs, data }). %% @doc Return a unique id for this SCB. id(Opts) -> proplists:get_value(filename, Opts, "/tmp/file.log"). %% @doc Always called before any other callback function. Use this to initiate %% any common state. init(Id, Opts) -> {ok,D} = file:open(Id,[write]), #state{ file_handle = D, total = 0, data = [] }. %% @doc Called before init_per_suite is called. pre_init_per_suite(Suite,Config,State) -> {Config, State#state{ suite_total = 0, tcs = [] }}. %% @doc Called after init_per_suite. post_init_per_suite(Suite,Config,Return,State) -> {Return, State}. %% @doc Called before end_per_suite. pre_end_per_suite(Suite,Config,State) -> {Config, State}. %% @doc Called after end_per_suite. post_end_per_suite(Suite,Config,Return,State) -> Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)}, {Return, State#state{ data = [Data | State#state.data] , total = State#state.total + State#state.suite_total } }. %% @doc Called before each init_per_group. pre_init_per_group(Group,Config,State) -> {Config, State}. %% @doc Called after each init_per_group. post_init_per_group(Group,Config,Return,State) -> {Return, State}. %% @doc Called after each end_per_group. pre_end_per_group(Group,Config,State) -> {Config, State}. %% @doc Called after each end_per_group. post_end_per_group(Group,Config,Return,State) -> {Return, State}. %% @doc Called before each test case. pre_init_per_testcase(TC,Config,State) -> {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }. %% @doc Called after each test case. post_end_per_testcase(TC,Config,Return,State) -> TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)}, {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }. %% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group, %% post_end_per_group and post_end_per_testcase if the suite, group or test case failed. on_tc_fail(TC, Reason, State) -> State. %% @doc Called when a test case is skipped by either user action %% or due to an init function failing. on_tc_skip(TC, Reason, State) -> State. %% @doc Called when the scope of the SCB is done terminate(State) -> io:format(State#state.file_handle, "~p.~n", [{test_run, State#state.total, State#state.data}]), file:close(State#state.file_handle), ok.