diff options
Diffstat (limited to 'lib/common_test/doc/src/ct_hooks_chapter.xml')
-rw-r--r-- | lib/common_test/doc/src/ct_hooks_chapter.xml | 401 |
1 files changed, 401 insertions, 0 deletions
diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml new file mode 100644 index 0000000000..fc5ab48e1b --- /dev/null +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -0,0 +1,401 @@ +<?xml version="1.0" encoding="UTF-8" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2011</year><year>2011</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Common Test Hooks</title> + <prepared>Lukas Larsson</prepared> + <docno></docno> + <date></date> + <rev></rev> + <file>ct_hooks_chapter.xml</file> + </header> + + <marker id="general"></marker> + <section> + <title>General</title> + <warning><p>This feature is in alpha release right now. This means that the + interface may change in the future and that there may be bugs. We + encourage you to use this feature, but be prepared + that there might be bugs and that the interface might change + inbetween releases.</p></warning> + <p> + The <em>Common Test Hook</em> (henceforth called CTH) framework allows + extensions of the default behaviour of Common Test by means of hooks + before and after all test suite calls. CTHs allow advanced Common Test + users to abstract out behaviour which is common to multiple test suites + without littering all test suites with library calls. Some example + usages are: logging, starting and monitoring external systems, + building C files needed by the tests and much more!</p> + + <p>In brief, Common Test Hooks allows you to:</p> + + <list> + <item>Manipulate the runtime config before each suite + configuration call</item> + <item>Manipulate the return of all suite configuration calls and in + extension the result of the test themselves.</item> + </list> + + <p>The following sections describe how to use CTHs, when they are run + and how to manipulate your test results in a CTH</p> + + <warning><p>When executing within a CTH all timetraps are shutoff. So + if your CTH never returns, the entire test run will be stalled!</p> + </warning> + + </section> + + <marker id="installing"></marker> + <section> + <title>Installing a CTH</title> + <p>There are multiple ways to install a CTH in your test run. You can do it + for all tests in a run, for specific test suites and for specific groups + within a test suite. If you want a CTH to be present in all test suites + within your test run there are three different ways to accomplish that. + </p> + + <list> + <item>Add <c>-ct_hooks</c> as an argument to + <seealso marker="run_test_chapter#ct_run">ct_run</seealso>. + To add multiple CTHs using this method append them to each other + using the keyword <c>and</c>, i.e. + <c>ct_run -ct_hooks cth1 [{debug,true}] and cth2 ...</c>.</item> + <item>Add the <c>ct_hooks</c> tag to your + <seealso marker="run_test_chapter#test_specifications"> + Test Specification</seealso></item> + <item>Add the <c>ct_hooks</c> tag to your call to + <seealso marker="ct#run_test-1">ct:run_test/1</seealso></item> + </list> + + <p>You can also add CTHs within a test suite. This is done by returning + <c>{ct_hooks,[CTH]}</c> in the config list from + <seealso marker="common_test#Module:suite-0">suite/0</seealso>, + <seealso marker="common_test#Module:init_per_suite-1"> + init_per_suite/1</seealso> or + <seealso marker="common_test#Module:init_per_group-2"> + init_per_group/2</seealso>. <c>CTH</c> in this case can be either + only the module name of the CTH or a tuple with the module name and the + initial arguments to the CTH. Eg: + <c>{ct_hooks,[my_cth_module]}</c> or + <c>{ct_hooks,[{my_cth_module,[{debug,true}]}]}</c></p> + + <section> + <title>Overriding CTHs</title> + <p>By default each installation of a CTH will cause a new instance of it + to be activated. This can cause problems if you want to be able to + override CTHs in test specifications while still having them in the + suite info function. The + <seealso marker="ct_hooks#Module:id-1">id/1</seealso> + callback exists to address this problem. By returning the same + <c>id</c> in both places, Common Test knows that this CTH + has already been installed and will not try to install it again.</p> + </section> + + </section> + + <marker id="scope"/> + <section> + <title>CTH Scope</title> + <p>Once the CTH is installed into a certain test run it will be there until + its scope is expired. The scope of a CTH depends on when it is + installed. + The <seealso marker="ct_hooks#Module:init-2">init/2</seealso> is + called at the beginning of the scope and the + <seealso marker="ct_hooks#Module:terminate-1">terminate/1 + </seealso> function is called when the scope ends.</p> + <table> + <row> + <cell><em>CTH Installed in</em></cell> + <cell><em>CTH scope begins before</em></cell> + <cell><em>CTH scope ends after</em></cell> + </row> + <row> + <cell><seealso marker="run_test_chapter#ct_run">ct_run</seealso></cell> + <cell>the first test suite is to be run.</cell> + <cell>the last test suite has been run.</cell> + </row> + <row> + <cell><seealso marker="ct#run_test-1">ct:run_test</seealso></cell> + <cell>the first test suite is to be run.</cell> + <cell>the last test suite has been run.</cell> + </row> + <row> + <cell><seealso marker="run_test_chapter#test_specifications"> + Test Specification</seealso></cell> + <cell>the first test suite is to be run.</cell> + <cell>the last test suite has been run.</cell> + </row> + <row> + <cell><seealso marker="common_test#Module:suite-0">suite/0 + </seealso></cell> + <cell><seealso marker="ct_hooks#Module:pre_init_per_suite-3"> + pre_init_per_suite/3</seealso> is called.</cell> + <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> + post_end_per_suite/4</seealso> has been called for that test suite.</cell> + </row> + <row> + <cell><seealso marker="common_test#Module:init_per_suite-1"> + init_per_suite/1</seealso></cell> + <cell><seealso marker="ct_hooks#Module:post_init_per_suite-4"> + post_init_per_suite/4</seealso> is called.</cell> + <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> + post_end_per_suite/4</seealso> has been called for that test suite.</cell> + </row> + <row> + <cell><seealso marker="common_test#Module:init_per_group-2"> + init_per_group/2</seealso></cell> + <cell><seealso marker="ct_hooks#Module:post_init_per_group-4"> + post_init_per_group/4</seealso> is called.</cell> + <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> + post_end_per_group/4</seealso> has been called for that group.</cell> + </row> + <tcaption>Scope of a CTH</tcaption> + </table> + + <section> + <title>CTH Processes and Tables</title> + <p>CTHs are run with the same process scoping as normal test suites + i.e. a different process will execute the init_per_suite hooks then the + init_per_group or per_testcase hooks. So if you want to spawn a + process in the CTH you cannot link with the CTH process as it will exit + after the post hook ends. Also if you for some reason need an ETS + table with your CTH, you will have to spawn a process which handles + it.</p> + </section> + + </section> + + <marker id="manipulating"/> + <section> + <title>Manipulating tests</title> + <p>It is through CTHs possible to manipulate the results of tests and + configuration functions. The main purpose of doing this with CTHs is to + allow common patterns to be abstracted out from test test suites and applied to + multiple test suites without duplicating any code. All of the callback + functions for a CTH follow a common interface, this interface is + described below.</p> + + <p>It is only possible to hook into test function which exists in the test + suite. So in order for a CTH to hook in before + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, + the <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso> + function must exist in the test suite.</p> + + <marker id="pre"/> + <section> + <title>Pre Hooks</title> + <p> + It is possible in a CTH to hook in behaviour before + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">init_per_testcase</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and + <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. + This is done in the CTH functions called pre_<name of function>. + All of these functions take the same three arguments: <c>Name</c>, + <c>Config</c> and <c>CTHState</c>. The return value of the CTH function + is always a combination of an result for the suite/group/test and an + updated <c>CTHState</c>. If you want the test suite to continue on + executing you should return the config list which you want the test to + use as the result. If you for some reason want to skip/fail the test, + return a tuple with <c>skip</c> or <c>fail</c> and a reason as the + result. Example: + </p> + <code>pre_init_per_suite(SuiteName, Config, CTHState) -> + case db:connect() of + {error,_Reason} -> + {{fail, "Could not connect to DB"}, CTHState}; + {ok, Handle} -> + {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} + end.</code> + + </section> + + <marker id="post"/> + <section> + <title>Post Hooks</title> + <p>It is also possible in a CTH to hook in behaviour after + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">end_per_testcase</seealso>, + <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and + <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. + This is done in the CTH functions called post_<name of function>. + All of these function take the same four arguments: <c>Name</c>, + <c>Config</c>, <c>Return</c> and <c>CTHState</c>. <c>Config</c> in this + case is the same <c>Config</c> as the testcase is called with. + <c>Return</c> is the value returned by the testcase. If the testcase + failed by crashing, <c>Return</c> will be + <c>{'EXIT',{{Error,Reason},Stacktrace}}</c>.</p> + + <p>The return value of the CTH function is always a combination of an + result for the suite/group/test and an updated <c>CTHState</c>. If + you want the callback to not affect the outcome of the test you should + return the <c>Return</c> data as it is given to the CTH. You can also + modify the result of the test. By returning the <c>Config</c> list + with the <c>tc_status</c> element removed you can recover from a test + failure. As in all the pre hooks, it is also possible to fail/skip + the test case in the post hook. Example: </p> + + <code>post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -> + case db:check_consistency() of + true -> + %% DB is good, pass the test. + {proplists:delete(tc_status, Config), CTHState}; + false -> + %% DB is not good, mark as skipped instead of failing + {{skip, "DB is inconsisten!"}, CTHState} + end; +post_end_per_testcase(_TC, Config, Return, CTHState) -> + %% Do nothing if tc does not crash. + {Return, CTHState}.</code> + + <note>Recovering from a testcase failure using CTHs should only be done as + a last resort. If used wrongly it could become very difficult to + determine which tests pass or fail in a test run</note> + + </section> + + <marker id="skip_n_fail"/> + <section> + <title>Skip and Fail hooks</title> + <p> + After any post hook has been executed for all installed CTHs, + <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_fail</seealso> + or <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_skip</seealso> + might be called if the testcase failed or was skipped + respectively. You cannot affect the outcome of the tests any further at + this point. + </p> + </section> + + </section> + + <marker id="example"/> + <section> + <title>Example CTH</title> + <p>The CTH below will log information about a test run into a format + parseable by <seealso marker="kernel:file#consult-1">file:consult/1</seealso>. + </p> + <code>%%% @doc Common Test Example Common Test Hook module. +-module(example_cth). + +%% Callbacks +-export([id/1]). +-export([init/2]). + +-export([pre_init_per_suite/3]). +-export([post_init_per_suite/4]). +-export([pre_end_per_suite/3]). +-export([post_end_per_suite/4]). + +-export([pre_init_per_group/3]). +-export([post_init_per_group/4]). +-export([pre_end_per_group/3]). +-export([post_end_per_group/4]). + +-export([pre_init_per_testcase/3]). +-export([post_end_per_testcase/4]). + +-export([on_tc_fail/3]). +-export([on_tc_skip/3]). + +-export([terminate/1]). + +-record(state, { file_handle, total, suite_total, ts, tcs, data }). + +%% @doc Return a unique id for this CTH. +id(Opts) -> + proplists:get_value(filename, Opts, "/tmp/file.log"). + +%% @doc Always called before any other callback function. Use this to initiate +%% any common state. +init(Id, Opts) -> + {ok,D} = file:open(Id,[write]), + #state{ file_handle = D, total = 0, data = [] }. + +%% @doc Called before init_per_suite is called. +pre_init_per_suite(Suite,Config,State) -> + {Config, State#state{ suite_total = 0, tcs = [] }}. + +%% @doc Called after init_per_suite. +post_init_per_suite(Suite,Config,Return,State) -> + {Return, State}. + +%% @doc Called before end_per_suite. +pre_end_per_suite(Suite,Config,State) -> + {Config, State}. + +%% @doc Called after end_per_suite. +post_end_per_suite(Suite,Config,Return,State) -> + Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)}, + {Return, State#state{ data = [Data | State#state.data] , + total = State#state.total + State#state.suite_total } }. + +%% @doc Called before each init_per_group. +pre_init_per_group(Group,Config,State) -> + {Config, State}. + +%% @doc Called after each init_per_group. +post_init_per_group(Group,Config,Return,State) -> + {Return, State}. + +%% @doc Called after each end_per_group. +pre_end_per_group(Group,Config,State) -> + {Config, State}. + +%% @doc Called after each end_per_group. +post_end_per_group(Group,Config,Return,State) -> + {Return, State}. + +%% @doc Called before each test case. +pre_init_per_testcase(TC,Config,State) -> + {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }. + +%% @doc Called after each test case. +post_end_per_testcase(TC,Config,Return,State) -> + TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)}, + {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }. + +%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group, +%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed. +on_tc_fail(TC, Reason, State) -> + State. + +%% @doc Called when a test case is skipped by either user action +%% or due to an init function failing. +on_tc_skip(TC, Reason, State) -> + State. + +%% @doc Called when the scope of the CTH is done +terminate(State) -> + io:format(State#state.file_handle, "~p.~n", + [{test_run, State#state.total, State#state.data}]), + file:close(State#state.file_handle), + ok.</code> + </section> + +</chapter> + + + + |