path: root/lib/common_test/doc/src/ct_hooks_chapter.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2011</year><year>2013</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at
 
          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.

    </legalnotice>

    <title>Common Test Hooks</title>
    <prepared>Lukas Larsson</prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>ct_hooks_chapter.xml</file>
  </header>

  <section>
    <marker id="general"></marker>
    <title>General</title>
    <p>
      The <em>Common Test Hook</em> (henceforth called CTH) framework allows 
      extensions of the default behaviour of Common Test by means of hooks 
      before and after all test suite calls. CTHs allow advanced Common Test
      users to abstract out behaviour which is common to multiple test suites
      without littering all test suites with library calls. Some example 
      usages are: logging, starting and monitoring external systems, 
      building C files needed by the tests and much more!</p>

    <p>In brief, Common Test Hooks allows you to:</p>

    <list>
      <item>Manipulate the runtime config before each suite 
      configuration call</item>
      <item>Manipulate the return of all suite configuration calls and in 
      extension the result of the test themselves.</item>
    </list>
    
    <p>The following sections describe how to use CTHs, when they are run 
      and how to manipulate your test results in a CTH</p>

    <warning><p>When executing within a CTH all timetraps are shutoff. So
	if your CTH never returns, the entire test run will be stalled!</p>
    </warning>

  </section>
  
  <section>
    <marker id="installing"></marker>
    <title>Installing a CTH</title>
    <p>There are multiple ways to install a CTH in your test run. You can do it
      for all tests in a run, for specific test suites and for specific groups 
      within a test suite. If you want a CTH to be present in all test suites 
      within your test run there are three different ways to accomplish that.
    </p>

    <list>
      <item>Add <c>-ct_hooks</c> as an argument to 
      <seealso marker="run_test_chapter#ct_run">ct_run</seealso>. 
      To add multiple CTHs using this method append them to each other
      using the keyword <c>and</c>, i.e. 
      <c>ct_run -ct_hooks cth1 [{debug,true}] and cth2 ...</c>.</item>
      <item>Add the <c>ct_hooks</c> tag to your 
      <seealso marker="run_test_chapter#test_specifications">
      Test Specification</seealso></item>
      <item>Add the <c>ct_hooks</c> tag to your call to 
      <seealso marker="ct#run_test-1">ct:run_test/1</seealso></item>
    </list>

    <p>You can also add CTHs within a test suite. This is done by returning
    <c>{ct_hooks,[CTH]}</c> in the config list from 
    <seealso marker="common_test#Module:suite-0">suite/0</seealso>,
    <seealso marker="common_test#Module:init_per_suite-1">
      init_per_suite/1</seealso> or
      <seealso marker="common_test#Module:init_per_group-2">
    init_per_group/2</seealso>. <c>CTH</c> in this case can be either
    only the module name of the CTH or a tuple with the module name and the
    initial arguments and optionally the hook priority of the CTH. Eg:
    <c>{ct_hooks,[my_cth_module]}</c> or 
    <c>{ct_hooks,[{my_cth_module,[{debug,true}]}]}</c> or 
    <c>{ct_hooks,[{my_cth_module,[{debug,true}],500}]}</c>
    </p>

    <section>
      <title>Overriding CTHs</title>
      <p>By default each installation of a CTH will cause a new instance of it
	to be activated. This can cause problems if you want to be able to 
	override CTHs in test specifications while still having them in the
	suite info function. The 
	<seealso marker="ct_hooks#Module:id-1">id/1</seealso>
	callback exists to address this problem. By returning the same
	<c>id</c> in both places, Common Test knows that this CTH
	has already been installed and will not try to install it again.</p>
    </section>
   
    <section>
      <title>CTH Execution order</title>
      <p>By default each CTH installed will be executed in the order which
      they are installed for init calls, and then reversed for end calls.
      This is not always wanted so common_test allows
      the user to specify a priority for each hook. The priority can either
      be specified in the CTH <seealso marker="ct_hooks#Module:init-2">init/2
      </seealso> function or when installing the hook. The priority given at
      installation will override the priority returned by the CTH. </p>
    </section>
  </section>

  <section>
    <marker id="scope"/>
    <title>CTH Scope</title>
    <p>Once the CTH is installed into a certain test run it will be there until
      its scope is expired. The scope of a CTH depends on when it is 
      installed.
      The <seealso marker="ct_hooks#Module:init-2">init/2</seealso> is 
      called at the beginning of the scope and the 
      <seealso marker="ct_hooks#Module:terminate-1">terminate/1
    </seealso> function is called when the scope ends.</p>
    <table>
      <row>
	<cell><em>CTH Installed in</em></cell>
	<cell><em>CTH scope begins before</em></cell>
	<cell><em>CTH scope ends after</em></cell>
      </row>
      <row>
	<cell><seealso marker="run_test_chapter#ct_run">ct_run</seealso></cell>
	<cell>the first test suite is to be run.</cell>
	<cell>the last test suite has been run.</cell>
      </row>
      <row>
	<cell><seealso marker="ct#run_test-1">ct:run_test</seealso></cell>
	<cell>the first test suite is to be run.</cell>
	<cell>the last test suite has been run.</cell>
      </row>
      <row>
	<cell><seealso marker="run_test_chapter#test_specifications">
	  Test Specification</seealso></cell>
	<cell>the first test suite is to be run.</cell>
	<cell>the last test suite has been run.</cell>
      </row>
      <row>
	<cell><seealso marker="common_test#Module:suite-0">suite/0
	</seealso></cell>
	<cell><seealso marker="ct_hooks#Module:pre_init_per_suite-3">
	    pre_init_per_suite/3</seealso> is called.</cell>
	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
	  post_end_per_suite/4</seealso> has been called for that test suite.</cell>
      </row>
      <row>
	<cell><seealso marker="common_test#Module:init_per_suite-1">
	  init_per_suite/1</seealso></cell>
	<cell><seealso marker="ct_hooks#Module:post_init_per_suite-4">
	    post_init_per_suite/4</seealso> is called.</cell>
	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
	  post_end_per_suite/4</seealso> has been called for that test suite.</cell>
      </row>
      <row>
	<cell><seealso marker="common_test#Module:init_per_group-2">
	  init_per_group/2</seealso></cell>
	<cell><seealso marker="ct_hooks#Module:post_init_per_group-4">
	    post_init_per_group/4</seealso> is called.</cell>
	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
	  post_end_per_group/4</seealso> has been called for that group.</cell>
      </row>
    <tcaption>Scope of a CTH</tcaption>
    </table>
    
    <section>
      <title>CTH Processes and Tables</title>
      <p>CTHs are run with the same process scoping as normal test suites
	i.e. a different process will execute the init_per_suite hooks then the
	init_per_group or per_testcase hooks. So if you want to spawn a 
	process in the CTH you cannot link with the CTH process as it will exit 
	after the post hook ends. Also if you for some reason need an ETS 
	table with your CTH, you will have to spawn a process which handles 
	it.</p>
    </section>
    
    <section>
      <title>External configuration data and Logging</title>
      <p>It's possible in the CTH to read configuration data values
	by calling <seealso marker="ct#get_config-1"><c>ct:get_config/1/2/3</c></seealso> (as explained in the
	<seealso marker="config_file_chapter#require_config_data">
	  External configuration data</seealso>
	chapter). The config variables in question must, as always, first have been
	<c>required</c> by means of a suite-, group-, or test case info function,
	or the <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso> function. Note that the latter can also be used
	in CT hook functions.</p>
      <p>The CT hook functions may call any of the logging functions available
	in the <c>ct</c> interface to print information to the log files, or to
	add comments in the suite overview page.
      </p>
    </section>    

  </section>

  <section>
    <marker id="manipulating"/>
    <title>Manipulating tests</title>
    <p>It is through CTHs possible to manipulate the results of tests and 
    configuration functions. The main purpose of doing this with CTHs is to
    allow common patterns to be abstracted out from test test suites and applied to
    multiple test suites without duplicating any code. All of the callback
    functions for a CTH follow a common interface, this interface is 
    described below.</p>

    <p>Common Test will always call all available hook functions, even pre- and post
      hooks for configuration functions that are not implemented in the suite.
      For example, <c>pre_init_per_suite(x_SUITE, ...)</c> and
      <c>post_init_per_suite(x_SUITE, ...)</c> will be called for test suite
      <c>x_SUITE</c>, even if it doesn't export <c>init_per_suite/1</c>. This feature
      makes it possible to use hooks as configuration fallbacks, or even
      completely replace all configuration functions with hook functions.</p>

    <section>
      <marker id="pre"/>
      <title>Pre Hooks</title>
      <p>
	It is possible in a CTH to hook in behaviour before 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_testcase</seealso>, 
	<seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and 
	<seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. 
	This is done in the CTH functions called pre_&lt;name of function&gt;.
	All of these functions take the same three arguments: <c>Name</c>, 
	<c>Config</c> and <c>CTHState</c>. The return value of the CTH function
	is always a combination of an result for the suite/group/test and an 
	updated <c>CTHState</c>. If you want the test suite to continue on 
	executing you should return the config list which you want the test to 
	use as the result. If you for some reason want to skip/fail the test, 
	return a tuple with <c>skip</c> or <c>fail</c> and a reason as the 
	result. Example:
      </p>
      <code>pre_init_per_suite(SuiteName, Config, CTHState) -&gt;
  case db:connect() of
    {error,_Reason} -&gt;
      {{fail, "Could not connect to DB"}, CTHState};
    {ok, Handle} -&gt;
      {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
  end.</code>
  <note><p>If using multiple CTHs, the first part of the return tuple will be
  used as input for the next CTH. So in the case above the next CTH might
  get <c>{fail,Reason}</c> as the second parameter. If you have many CTHs
  which interact, it might be a good idea to not let each CTH return
  <c>fail</c> or <c>skip</c>. Instead return that an action should be taken
  through the <c>Config</c> list and implement a CTH which at the end takes
  the correct action.</p></note>
	
    </section>
    
    <section>
      <marker id="post"/>
      <title>Post Hooks</title>
      <p>It is also possible in a CTH to hook in behaviour after 
      <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
      <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, 
      <seealso marker="common_test#Module:init_per_suite-1">end_per_testcase</seealso>, 
      <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and 
      <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>.
      This is done in the CTH functions called post_&lt;name of function&gt;. 
      All of these function take the same four arguments: <c>Name</c>, 
      <c>Config</c>, <c>Return</c> and <c>CTHState</c>. <c>Config</c> in this
      case is the same <c>Config</c> as the testcase is called with. 
      <c>Return</c> is the value returned by the testcase. If the testcase 
      failed by crashing, <c>Return</c> will be 
      <c>{'EXIT',{{Error,Reason},Stacktrace}}</c>.</p>
      
      <p>The return value of the CTH function is always a combination of an
	result for the suite/group/test and an updated <c>CTHState</c>. If
	you want the callback to not affect the outcome of the test you should
	return the <c>Return</c> data as it is given to the CTH. You can also
	modify the result of the test. By returning the <c>Config</c> list
	with the <c>tc_status</c> element removed you can recover from a test 
	failure. As in all the pre hooks, it is also possible to fail/skip
	the test case in the post hook. Example: </p>

      <code>post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -&gt;
  case db:check_consistency() of
    true ->
      %% DB is good, pass the test.
      {proplists:delete(tc_status, Config), CTHState};
    false ->
      %% DB is not good, mark as skipped instead of failing
      {{skip, "DB is inconsisten!"}, CTHState}
  end;
post_end_per_testcase(_TC, Config, Return, CTHState) -&gt;
  %% Do nothing if tc does not crash.
  {Return, CTHState}.</code>

      <note><p>Recovering from a testcase failure using CTHs should only be done as
	a last resort. If used wrongly it could become very difficult to 
	determine which tests pass or fail in a test run</p></note>
  
    </section>

    <section>
      <marker id="skip_n_fail"/>
      <title>Skip and Fail hooks</title>
      <p>
	After any post hook has been executed for all installed CTHs, 
	<seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_fail</seealso>
	or <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_skip</seealso> 
	might be called if the testcase failed or was skipped 
	respectively. You cannot affect the outcome of the tests any further at 
	this point. 
      </p>
    </section>

  </section>

  <section>
    <marker id="synchronizing"/>
    <title>Synchronizing external user applications with Common Test</title>
    <p>CTHs can be used to synchronize test runs with external user applications.
    The init function may e.g. start and/or communicate with an application that
    has the purpose of preparing the SUT for an upcoming test run, or maybe
    initialize a database for saving test data to during the test run. The
    terminate function may similarly order such an application to reset the SUT
    after the test run, and/or tell the application to finish active sessions
    and terminate.
    Any system error- or progress reports generated during the init- or
    termination stage will be saved in the 
    <seealso marker="run_test_chapter#pre_post_test_io_log">Pre-
    and post test I/O log</seealso>. (This is also true for any printouts made
    with <c>ct:log/2</c> and <c>ct:pal/2</c>).</p>
    <p>In order to ensure that Common Test doesn't start executing tests, or
    closes its log files and shuts down, before the external application
    is ready for it, Common Test may be synchronized with the application. 
    During startup and shutdown, Common Test can be suspended, simply by
    having a CTH evaluate a <c>receive</c> expression in the init- or terminate
    function. The macros <c>?CT_HOOK_INIT_PROCESS</c> (the process executing the hook
    init function) and <c>?CT_HOOK_TERMINATE_PROCESS</c> (the process executing
    the hook terminate function), each specifies the name of the correct Common Test
    process to send a message to in order to return from the <c>receive</c>.
    These macros are defined in <c>ct.hrl</c>.
    </p>
  </section>

  <section>
     <marker id="example"/>
     <title>Example CTH</title>
     <p>The CTH below will log information about a test run into a format 
       parseable by <seealso marker="kernel:file#consult-1">file:consult/1</seealso>.
     </p>
     <code>%%% @doc Common Test Example Common Test Hook module.
-module(example_cth).

%% Callbacks
-export([id/1]).
-export([init/2]).

-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).

-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).

-export([pre_init_per_testcase/3]).
-export([post_end_per_testcase/4]).

-export([on_tc_fail/3]).
-export([on_tc_skip/3]).

-export([terminate/1]).

-record(state, { file_handle, total, suite_total, ts, tcs, data }).

%% @doc Return a unique id for this CTH.
id(Opts) ->
  proplists:get_value(filename, Opts, "/tmp/file.log").

%% @doc Always called before any other callback function. Use this to initiate
%% any common state. 
init(Id, Opts) ->
    {ok,D} = file:open(Id,[write]),
    {ok, #state{ file_handle = D, total = 0, data = [] }}.

%% @doc Called before init_per_suite is called. 
pre_init_per_suite(Suite,Config,State) ->
    {Config, State#state{ suite_total = 0, tcs = [] }}.

%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
    {Return, State}.

%% @doc Called before end_per_suite. 
pre_end_per_suite(Suite,Config,State) ->
    {Config, State}.

%% @doc Called after end_per_suite. 
post_end_per_suite(Suite,Config,Return,State) ->
    Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
    {Return, State#state{ data = [Data | State#state.data] ,
                          total = State#state.total + State#state.suite_total } }.

%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
    {Config, State}.

%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
    {Return, State}.

%% @doc Called after each end_per_group. 
pre_end_per_group(Group,Config,State) ->
    {Config, State}.

%% @doc Called after each end_per_group. 
post_end_per_group(Group,Config,Return,State) ->
    {Return, State}.

%% @doc Called before each test case.
pre_init_per_testcase(TC,Config,State) ->
    {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.

%% @doc Called after each test case.
post_end_per_testcase(TC,Config,Return,State) ->
    TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
    {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.

%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
    State.

%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.  
on_tc_skip(TC, Reason, State) ->
    State.

%% @doc Called when the scope of the CTH is done
terminate(State) ->
    io:format(State#state.file_handle, "~p.~n",
               [{test_run, State#state.total, State#state.data}]),
    file:close(State#state.file_handle),
    ok.</code>
  </section>

  <section>
    <marker id="builtin_cths"/>
    <title>Built-in CTHs</title>
    <p>Common Test is delivered with a couple of general purpose CTHs that
    can be enabled by the user to provide some generic testing functionality.
    Some of these are enabled by default when starting running common_test,
    they can be disabled by setting <c>enable_builtin_hooks</c> to
    <c>false</c> on the command line or in the test specification. In the
    table below there is a list of all current CTHs which are delivered with
    Common Test.</p>

    <table>
      <row>
	<cell align="left"><em>CTH Name</em></cell>
	<cell align="left"><em>Is Built-in</em></cell>
	<cell align="left"><em>Description</em></cell>
      </row>
      <row>
	<cell align="left">cth_log_redirect</cell>
	<cell align="left">yes</cell>
	<cell align="left">Captures all error_logger and SASL logging events and prints them
	to the current test case log. If an event can not be associated with a
	testcase it will be printed in the common test framework log. This will
	happen for testcases which are run in parallel and events which occur
	inbetween testcases. You can configure the level of
	<seealso marker="sasl:sasl_app">SASL</seealso> events report
	using the normal SASL mechanisms. </cell>
      </row>
      <row>
	<cell align="left">cth_surefire</cell>
	<cell align="left">no</cell>
	<cell align="left"><p>Captures all test results and outputs them as surefire
	XML into a file. The file which is created is by default
	called junit_report.xml. The file name can be changed by
	setting the <c>path</c> option for this hook, e.g.</p>

	<code>-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]</code>

	<p>If the <c>url_base</c> option is set, an additional
	attribute named <c>url</c> will be added to each
	<c>testsuite</c> and <c>testcase</c> XML element. The value will
	be constructed from the <c>url_base</c> and a relative path
	to the test suite or test case log respectively, e.g.</p>

	<code>-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]</code>
	<p>will give a url attribute value similar to</p>

	<code>"http://myserver.com/[email protected]_11.19.39/
x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code>

	<p>Surefire XML can for instance be used by Jenkins to display test
	results.</p></cell>
      </row>
    </table>

  </section>

</chapter>