Rename Suite Callback to Common Test hook in documentation

1 files changed, 401 insertions, 0 deletions

diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml
new file mode 100644
index 0000000000..fc5ab48e1b
--- /dev/null
+++ b/lib/common_test/doc/src/ct_hooks_chapter.xml
@@ -0,0 +1,401 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+  <header>
+    <copyright>
+      <year>2011</year><year>2011</year>
+      <holder>Ericsson AB. All Rights Reserved.</holder>
+    </copyright>
+    <legalnotice>
+      The contents of this file are subject to the Erlang Public License,
+      Version 1.1, (the "License"); you may not use this file except in
+      compliance with the License. You should have received a copy of the
+      Erlang Public License along with this software. If not, it can be
+      retrieved online at http://www.erlang.org/.
+
+      Software distributed under the License is distributed on an "AS IS"
+      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+      the License for the specific language governing rights and limitations
+      under the License.
+
+    </legalnotice>
+
+    <title>Common Test Hooks</title>
+    <prepared>Lukas Larsson</prepared>
+    <docno></docno>
+    <date></date>
+    <rev></rev>
+    <file>ct_hooks_chapter.xml</file>
+  </header>
+
+  <marker id="general"></marker>
+  <section>
+    <title>General</title>
+    <warning><p>This feature is in alpha release right now. This means that the 
+	interface may change in the future and that there may be bugs. We 
+	encourage you to use this feature, but be prepared 
+	that there might be bugs and that the interface might change
+	inbetween releases.</p></warning>
+    <p>
+      The <em>Common Test Hook</em> (henceforth called CTH) framework allows 
+      extensions of the default behaviour of Common Test by means of hooks 
+      before and after all test suite calls. CTHs allow advanced Common Test
+      users to abstract out behaviour which is common to multiple test suites
+      without littering all test suites with library calls. Some example 
+      usages are: logging, starting and monitoring external systems, 
+      building C files needed by the tests and much more!</p>
+
+    <p>In brief, Common Test Hooks allows you to:</p>
+
+    <list>
+      <item>Manipulate the runtime config before each suite 
+      configuration call</item>
+      <item>Manipulate the return of all suite configuration calls and in 
+      extension the result of the test themselves.</item>
+    </list>
+    
+    <p>The following sections describe how to use CTHs, when they are run 
+      and how to manipulate your test results in a CTH</p>
+
+    <warning><p>When executing within a CTH all timetraps are shutoff. So
+	if your CTH never returns, the entire test run will be stalled!</p>
+    </warning>
+
+  </section>
+  
+  <marker id="installing"></marker>
+  <section>
+    <title>Installing a CTH</title>
+    <p>There are multiple ways to install a CTH in your test run. You can do it
+      for all tests in a run, for specific test suites and for specific groups 
+      within a test suite. If you want a CTH to be present in all test suites 
+      within your test run there are three different ways to accomplish that.
+    </p>
+
+    <list>
+      <item>Add <c>-ct_hooks</c> as an argument to 
+      <seealso marker="run_test_chapter#ct_run">ct_run</seealso>. 
+      To add multiple CTHs using this method append them to each other
+      using the keyword <c>and</c>, i.e. 
+      <c>ct_run -ct_hooks cth1 [{debug,true}] and cth2 ...</c>.</item>
+      <item>Add the <c>ct_hooks</c> tag to your 
+      <seealso marker="run_test_chapter#test_specifications">
+      Test Specification</seealso></item>
+      <item>Add the <c>ct_hooks</c> tag to your call to 
+      <seealso marker="ct#run_test-1">ct:run_test/1</seealso></item>
+    </list>
+
+    <p>You can also add CTHs within a test suite. This is done by returning
+    <c>{ct_hooks,[CTH]}</c> in the config list from 
+    <seealso marker="common_test#Module:suite-0">suite/0</seealso>,
+    <seealso marker="common_test#Module:init_per_suite-1">
+      init_per_suite/1</seealso> or
+      <seealso marker="common_test#Module:init_per_group-2">
+    init_per_group/2</seealso>. <c>CTH</c> in this case can be either
+    only the module name of the CTH or a tuple with the module name and the
+    initial arguments to the CTH. Eg:
+    <c>{ct_hooks,[my_cth_module]}</c> or 
+    <c>{ct_hooks,[{my_cth_module,[{debug,true}]}]}</c></p>
+
+    <section>
+      <title>Overriding CTHs</title>
+      <p>By default each installation of a CTH will cause a new instance of it
+	to be activated. This can cause problems if you want to be able to 
+	override CTHs in test specifications while still having them in the
+	suite info function. The 
+	<seealso marker="ct_hooks#Module:id-1">id/1</seealso>
+	callback exists to address this problem. By returning the same
+	<c>id</c> in both places, Common Test knows that this CTH
+	has already been installed and will not try to install it again.</p>
+    </section>
+
+  </section>
+
+  <marker id="scope"/>
+  <section>
+    <title>CTH Scope</title>
+    <p>Once the CTH is installed into a certain test run it will be there until
+      its scope is expired. The scope of a CTH depends on when it is 
+      installed.
+      The <seealso marker="ct_hooks#Module:init-2">init/2</seealso> is 
+      called at the beginning of the scope and the 
+      <seealso marker="ct_hooks#Module:terminate-1">terminate/1
+    </seealso> function is called when the scope ends.</p>
+    <table>
+      <row>
+	<cell><em>CTH Installed in</em></cell>
+	<cell><em>CTH scope begins before</em></cell>
+	<cell><em>CTH scope ends after</em></cell>
+      </row>
+      <row>
+	<cell><seealso marker="run_test_chapter#ct_run">ct_run</seealso></cell>
+	<cell>the first test suite is to be run.</cell>
+	<cell>the last test suite has been run.</cell>
+      </row>
+      <row>
+	<cell><seealso marker="ct#run_test-1">ct:run_test</seealso></cell>
+	<cell>the first test suite is to be run.</cell>
+	<cell>the last test suite has been run.</cell>
+      </row>
+      <row>
+	<cell><seealso marker="run_test_chapter#test_specifications">
+	  Test Specification</seealso></cell>
+	<cell>the first test suite is to be run.</cell>
+	<cell>the last test suite has been run.</cell>
+      </row>
+      <row>
+	<cell><seealso marker="common_test#Module:suite-0">suite/0
+	</seealso></cell>
+	<cell><seealso marker="ct_hooks#Module:pre_init_per_suite-3">
+	    pre_init_per_suite/3</seealso> is called.</cell>
+	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
+	  post_end_per_suite/4</seealso> has been called for that test suite.</cell>
+      </row>
+      <row>
+	<cell><seealso marker="common_test#Module:init_per_suite-1">
+	  init_per_suite/1</seealso></cell>
+	<cell><seealso marker="ct_hooks#Module:post_init_per_suite-4">
+	    post_init_per_suite/4</seealso> is called.</cell>
+	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
+	  post_end_per_suite/4</seealso> has been called for that test suite.</cell>
+      </row>
+      <row>
+	<cell><seealso marker="common_test#Module:init_per_group-2">
+	  init_per_group/2</seealso></cell>
+	<cell><seealso marker="ct_hooks#Module:post_init_per_group-4">
+	    post_init_per_group/4</seealso> is called.</cell>
+	<cell><seealso marker="ct_hooks#Module:post_end_per_suite-4">
+	  post_end_per_group/4</seealso> has been called for that group.</cell>
+      </row>
+    <tcaption>Scope of a CTH</tcaption>
+    </table>
+    
+    <section>
+      <title>CTH Processes and Tables</title>
+      <p>CTHs are run with the same process scoping as normal test suites
+	i.e. a different process will execute the init_per_suite hooks then the
+	init_per_group or per_testcase hooks. So if you want to spawn a 
+	process in the CTH you cannot link with the CTH process as it will exit 
+	after the post hook ends. Also if you for some reason need an ETS 
+	table with your CTH, you will have to spawn a process which handles 
+	it.</p>
+    </section>
+    
+  </section>
+
+  <marker id="manipulating"/>
+  <section>
+    <title>Manipulating tests</title>
+    <p>It is through CTHs possible to manipulate the results of tests and 
+    configuration functions. The main purpose of doing this with CTHs is to
+    allow common patterns to be abstracted out from test test suites and applied to
+    multiple test suites without duplicating any code. All of the callback
+    functions for a CTH follow a common interface, this interface is 
+    described below.</p>
+
+    <p>It is only possible to hook into test function which exists in the test 
+      suite. So in order for a CTH to hook in before 
+      <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
+      the <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso> 
+      function must exist in the test suite.</p>
+
+    <marker id="pre"/>
+    <section>
+      <title>Pre Hooks</title>
+      <p>
+	It is possible in a CTH to hook in behaviour before 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_testcase</seealso>, 
+	<seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and 
+	<seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. 
+	This is done in the CTH functions called pre_&lt;name of function&gt;.
+	All of these functions take the same three arguments: <c>Name</c>, 
+	<c>Config</c> and <c>CTHState</c>. The return value of the CTH function
+	is always a combination of an result for the suite/group/test and an 
+	updated <c>CTHState</c>. If you want the test suite to continue on 
+	executing you should return the config list which you want the test to 
+	use as the result. If you for some reason want to skip/fail the test, 
+	return a tuple with <c>skip</c> or <c>fail</c> and a reason as the 
+	result. Example:
+      </p>
+      <code>pre_init_per_suite(SuiteName, Config, CTHState) -&gt;
+  case db:connect() of
+    {error,_Reason} -&gt;
+      {{fail, "Could not connect to DB"}, CTHState};
+    {ok, Handle} -&gt;
+      {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
+  end.</code>
+	
+    </section>
+    
+    <marker id="post"/>
+    <section>
+      <title>Post Hooks</title>
+      <p>It is also possible in a CTH to hook in behaviour after 
+      <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
+      <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, 
+      <seealso marker="common_test#Module:init_per_suite-1">end_per_testcase</seealso>, 
+      <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and 
+      <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>.
+      This is done in the CTH functions called post_&lt;name of function&gt;. 
+      All of these function take the same four arguments: <c>Name</c>, 
+      <c>Config</c>, <c>Return</c> and <c>CTHState</c>. <c>Config</c> in this
+      case is the same <c>Config</c> as the testcase is called with. 
+      <c>Return</c> is the value returned by the testcase. If the testcase 
+      failed by crashing, <c>Return</c> will be 
+      <c>{'EXIT',{{Error,Reason},Stacktrace}}</c>.</p>
+      
+      <p>The return value of the CTH function is always a combination of an
+	result for the suite/group/test and an updated <c>CTHState</c>. If
+	you want the callback to not affect the outcome of the test you should
+	return the <c>Return</c> data as it is given to the CTH. You can also
+	modify the result of the test. By returning the <c>Config</c> list
+	with the <c>tc_status</c> element removed you can recover from a test 
+	failure. As in all the pre hooks, it is also possible to fail/skip
+	the test case in the post hook. Example: </p>
+
+      <code>post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -&gt;
+  case db:check_consistency() of
+    true ->
+      %% DB is good, pass the test.
+      {proplists:delete(tc_status, Config), CTHState};
+    false ->
+      %% DB is not good, mark as skipped instead of failing
+      {{skip, "DB is inconsisten!"}, CTHState}
+  end;
+post_end_per_testcase(_TC, Config, Return, CTHState) -&gt;
+  %% Do nothing if tc does not crash.
+  {Return, CTHState}.</code>
+
+      <note>Recovering from a testcase failure using CTHs should only be done as
+	a last resort. If used wrongly it could become very difficult to 
+	determine which tests pass or fail in a test run</note>
+  
+    </section>
+
+    <marker id="skip_n_fail"/>
+    <section>
+      <title>Skip and Fail hooks</title>
+      <p>
+	After any post hook has been executed for all installed CTHs, 
+	<seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_fail</seealso>
+	or <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_skip</seealso> 
+	might be called if the testcase failed or was skipped 
+	respectively. You cannot affect the outcome of the tests any further at 
+	this point. 
+      </p>
+    </section>
+
+  </section>
+
+  <marker id="example"/>
+  <section>
+     <title>Example CTH</title>
+     <p>The CTH below will log information about a test run into a format 
+       parseable by <seealso marker="kernel:file#consult-1">file:consult/1</seealso>.
+     </p>
+     <code>%%% @doc Common Test Example Common Test Hook module.
+-module(example_cth).
+
+%% Callbacks
+-export([id/1]).
+-export([init/2]).
+
+-export([pre_init_per_suite/3]).
+-export([post_init_per_suite/4]).
+-export([pre_end_per_suite/3]).
+-export([post_end_per_suite/4]).
+
+-export([pre_init_per_group/3]).
+-export([post_init_per_group/4]).
+-export([pre_end_per_group/3]).
+-export([post_end_per_group/4]).
+
+-export([pre_init_per_testcase/3]).
+-export([post_end_per_testcase/4]).
+
+-export([on_tc_fail/3]).
+-export([on_tc_skip/3]).
+
+-export([terminate/1]).
+
+-record(state, { file_handle, total, suite_total, ts, tcs, data }).
+
+%% @doc Return a unique id for this CTH.
+id(Opts) ->
+  proplists:get_value(filename, Opts, "/tmp/file.log").
+
+%% @doc Always called before any other callback function. Use this to initiate
+%% any common state. 
+init(Id, Opts) ->
+    {ok,D} = file:open(Id,[write]),
+    #state{ file_handle = D, total = 0, data = [] }.
+
+%% @doc Called before init_per_suite is called. 
+pre_init_per_suite(Suite,Config,State) ->
+    {Config, State#state{ suite_total = 0, tcs = [] }}.
+
+%% @doc Called after init_per_suite.
+post_init_per_suite(Suite,Config,Return,State) ->
+    {Return, State}.
+
+%% @doc Called before end_per_suite. 
+pre_end_per_suite(Suite,Config,State) ->
+    {Config, State}.
+
+%% @doc Called after end_per_suite. 
+post_end_per_suite(Suite,Config,Return,State) ->
+    Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
+    {Return, State#state{ data = [Data | State#state.data] ,
+                          total = State#state.total + State#state.suite_total } }.
+
+%% @doc Called before each init_per_group.
+pre_init_per_group(Group,Config,State) ->
+    {Config, State}.
+
+%% @doc Called after each init_per_group.
+post_init_per_group(Group,Config,Return,State) ->
+    {Return, State}.
+
+%% @doc Called after each end_per_group. 
+pre_end_per_group(Group,Config,State) ->
+    {Config, State}.
+
+%% @doc Called after each end_per_group. 
+post_end_per_group(Group,Config,Return,State) ->
+    {Return, State}.
+
+%% @doc Called before each test case.
+pre_init_per_testcase(TC,Config,State) ->
+    {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
+
+%% @doc Called after each test case.
+post_end_per_testcase(TC,Config,Return,State) ->
+    TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
+    {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
+
+%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
+%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
+on_tc_fail(TC, Reason, State) ->
+    State.
+
+%% @doc Called when a test case is skipped by either user action
+%% or due to an init function failing.  
+on_tc_skip(TC, Reason, State) ->
+    State.
+
+%% @doc Called when the scope of the CTH is done
+terminate(State) ->
+    io:format(State#state.file_handle, "~p.~n",
+               [{test_run, State#state.total, State#state.data}]),
+    file:close(State#state.file_handle),
+    ok.</code>
+  </section>
+
+</chapter>
+
+
+
+