diff options
Diffstat (limited to 'lib/common_test/doc/src/ct_hooks_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/ct_hooks_chapter.xml | 701 |
1 files changed, 363 insertions, 338 deletions
diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml index d9892c66f7..3eb1945a61 100644 --- a/lib/common_test/doc/src/ct_hooks_chapter.xml +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -30,177 +30,181 @@ <file>ct_hooks_chapter.xml</file> </header> - <marker id="general"></marker> <section> + <marker id="general"></marker> <title>General</title> <p> - The <em>Common Test Hook</em> (henceforth called CTH) framework allows - extensions of the default behaviour of Common Test by means of hooks - before and after all test suite calls. CTHs allow advanced Common Test - users to abstract out behaviour which is common to multiple test suites - without littering all test suites with library calls. Some example - usages are: logging, starting and monitoring external systems, - building C files needed by the tests and much more!</p> - - <p>In brief, Common Test Hooks allows you to:</p> - - <list> - <item>Manipulate the runtime config before each suite - configuration call</item> - <item>Manipulate the return of all suite configuration calls and in - extension the result of the test themselves.</item> + The <em>Common Test Hook (CTH)</em> framework allows + extensions of the default behavior of <c>Common Test</c> using hooks + before and after all test suite calls. CTHs allow advanced <c>Common Test</c> + users to abstract out behavior that is common to multiple test suites + without littering all test suites with library calls. this can be used + for logging, starting, and monitoring external systems, + building C files needed by the tests, and so on.</p> + + <p>In brief, CTH allows you to do the following:</p> + + <list type="bulleted"> + <item>Manipulate the runtime configuration before each suite + configuration call.</item> + <item>Manipulate the return of all suite configuration calls, and in + extension, the result of the tests themselves.</item> </list> - <p>The following sections describe how to use CTHs, when they are run - and how to manipulate your test results in a CTH</p> + <p>The following sections describe how to use CTHs, when they are run, + and how to manipulate the test results in a CTH.</p> - <warning><p>When executing within a CTH all timetraps are shutoff. So - if your CTH never returns, the entire test run will be stalled!</p> + <warning><p>When executing within a CTH, all timetraps are shut off. So + if your CTH never returns, the entire test run is stalled.</p> </warning> </section> - <marker id="installing"></marker> <section> + <marker id="installing"></marker> <title>Installing a CTH</title> - <p>There are multiple ways to install a CTH in your test run. You can do it - for all tests in a run, for specific test suites and for specific groups + <p>A CTH can be installed in multiple ways in your test run. You can do it + for all tests in a run, for specific test suites, and for specific groups within a test suite. If you want a CTH to be present in all test suites - within your test run there are three different ways to accomplish that. + within your test run, there are three ways to accomplish that, as follows: </p> - <list> + <list type="bulleted"> <item>Add <c>-ct_hooks</c> as an argument to <seealso marker="run_test_chapter#ct_run">ct_run</seealso>. - To add multiple CTHs using this method append them to each other - using the keyword <c>and</c>, i.e. + To add multiple CTHs using this method, append them to each other + using the keyword <c>and</c>, that is, <c>ct_run -ct_hooks cth1 [{debug,true}] and cth2 ...</c>.</item> - <item>Add the <c>ct_hooks</c> tag to your + <item>Add tag <c>ct_hooks</c> to your <seealso marker="run_test_chapter#test_specifications"> - Test Specification</seealso></item> - <item>Add the <c>ct_hooks</c> tag to your call to - <seealso marker="ct#run_test-1">ct:run_test/1</seealso></item> + Test Specification</seealso>.</item> + <item>Add tag <c>ct_hooks</c> to your call to + <seealso marker="ct#run_test-1">ct:run_test/1</seealso>.</item> </list> - <p>You can also add CTHs within a test suite. This is done by returning - <c>{ct_hooks,[CTH]}</c> in the config list from + <p>CTHs can also be added within a test suite. This is done by returning + <c>{ct_hooks,[CTH]}</c> in the configuration list from <seealso marker="common_test#Module:suite-0">suite/0</seealso>, <seealso marker="common_test#Module:init_per_suite-1"> - init_per_suite/1</seealso> or + init_per_suite/1</seealso>, or <seealso marker="common_test#Module:init_per_group-2"> - init_per_group/2</seealso>. <c>CTH</c> in this case can be either - only the module name of the CTH or a tuple with the module name and the - initial arguments and optionally the hook priority of the CTH. Eg: - <c>{ct_hooks,[my_cth_module]}</c> or - <c>{ct_hooks,[{my_cth_module,[{debug,true}]}]}</c> or - <c>{ct_hooks,[{my_cth_module,[{debug,true}],500}]}</c> - </p> + init_per_group/2</seealso>.</p> + + <p>In this case, <c>CTH</c> can either be only the module name of the CTH + or a tuple with the module name and the initial arguments, and optionally + the hook priority of the CTH. For example, one of the following:</p> + <list type="bulleted"> + <item><c>{ct_hooks,[my_cth_module]}</c></item> + <item><c>{ct_hooks,[{my_cth_module,[{debug,true}]}]}</c></item> + <item><c>{ct_hooks,[{my_cth_module,[{debug,true}],500}]}</c></item> + </list> <section> <title>Overriding CTHs</title> - <p>By default each installation of a CTH will cause a new instance of it - to be activated. This can cause problems if you want to be able to - override CTHs in test specifications while still having them in the - suite info function. The + <p>By default, each installation of a CTH causes a new instance of it + to be activated. This can cause problems if you want to override + CTHs in test specifications while still having them in the + suite information function. The <seealso marker="ct_hooks#Module:id-1">id/1</seealso> callback exists to address this problem. By returning the same - <c>id</c> in both places, Common Test knows that this CTH - has already been installed and will not try to install it again.</p> + <c>id</c> in both places, <c>Common Test</c> knows that this CTH + is already installed and does not try to install it again.</p> </section> <section> - <title>CTH Execution order</title> - <p>By default each CTH installed will be executed in the order which + <title>CTH Execution Order</title> + <p>By default, each CTH installed is executed in the order that they are installed for init calls, and then reversed for end calls. - This is not always wanted so common_test allows + This is not always desired, so <c>Common Test</c> allows the user to specify a priority for each hook. The priority can either - be specified in the CTH <seealso marker="ct_hooks#Module:init-2">init/2 - </seealso> function or when installing the hook. The priority given at - installation will override the priority returned by the CTH. </p> + be specified in the CTH function + <seealso marker="ct_hooks#Module:init-2">init/2</seealso> or when + installing the hook. The priority specified at installation overrides the + priority returned by the CTH.</p> </section> </section> - <marker id="scope"/> <section> + <marker id="scope"/> <title>CTH Scope</title> - <p>Once the CTH is installed into a certain test run it will be there until + <p>Once the CTH is installed into a certain test run it remains there until its scope is expired. The scope of a CTH depends on when it is - installed. - The <seealso marker="ct_hooks#Module:init-2">init/2</seealso> is - called at the beginning of the scope and the - <seealso marker="ct_hooks#Module:terminate-1">terminate/1 - </seealso> function is called when the scope ends.</p> + installed, see the following table. + Function <seealso marker="ct_hooks#Module:init-2">init/2</seealso> is + called at the beginning of the scope and function + <seealso marker="ct_hooks#Module:terminate-1">terminate/1</seealso> + is called when the scope ends.</p> <table> <row> - <cell><em>CTH Installed in</em></cell> + <cell><em>CTH installed in</em></cell> <cell><em>CTH scope begins before</em></cell> <cell><em>CTH scope ends after</em></cell> </row> <row> <cell><seealso marker="run_test_chapter#ct_run">ct_run</seealso></cell> - <cell>the first test suite is to be run.</cell> - <cell>the last test suite has been run.</cell> + <cell>the first test suite is to be run</cell> + <cell>the last test suite has been run</cell> </row> <row> <cell><seealso marker="ct#run_test-1">ct:run_test</seealso></cell> - <cell>the first test suite is to be run.</cell> - <cell>the last test suite has been run.</cell> + <cell>the first test suite is run</cell> + <cell>the last test suite has been run</cell> </row> <row> <cell><seealso marker="run_test_chapter#test_specifications"> Test Specification</seealso></cell> - <cell>the first test suite is to be run.</cell> - <cell>the last test suite has been run.</cell> + <cell>the first test suite is run</cell> + <cell>the last test suite has been run</cell> </row> <row> <cell><seealso marker="common_test#Module:suite-0">suite/0 </seealso></cell> <cell><seealso marker="ct_hooks#Module:pre_init_per_suite-3"> - pre_init_per_suite/3</seealso> is called.</cell> + pre_init_per_suite/3</seealso> is called</cell> <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> - post_end_per_suite/4</seealso> has been called for that test suite.</cell> + post_end_per_suite/4</seealso> has been called for that test suite</cell> </row> <row> <cell><seealso marker="common_test#Module:init_per_suite-1"> init_per_suite/1</seealso></cell> <cell><seealso marker="ct_hooks#Module:post_init_per_suite-4"> - post_init_per_suite/4</seealso> is called.</cell> + post_init_per_suite/4</seealso> is called</cell> <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> - post_end_per_suite/4</seealso> has been called for that test suite.</cell> + post_end_per_suite/4</seealso> has been called for that test suite</cell> </row> <row> <cell><seealso marker="common_test#Module:init_per_group-2"> init_per_group/2</seealso></cell> <cell><seealso marker="ct_hooks#Module:post_init_per_group-4"> - post_init_per_group/4</seealso> is called.</cell> + post_init_per_group/4</seealso> is called</cell> <cell><seealso marker="ct_hooks#Module:post_end_per_suite-4"> - post_end_per_group/4</seealso> has been called for that group.</cell> + post_end_per_group/4</seealso> has been called for that group</cell> </row> <tcaption>Scope of a CTH</tcaption> </table> <section> <title>CTH Processes and Tables</title> - <p>CTHs are run with the same process scoping as normal test suites - i.e. a different process will execute the init_per_suite hooks then the - init_per_group or per_testcase hooks. So if you want to spawn a - process in the CTH you cannot link with the CTH process as it will exit - after the post hook ends. Also if you for some reason need an ETS - table with your CTH, you will have to spawn a process which handles - it.</p> + <p>CTHs are run with the same process scoping as normal test suites, + that is, a different process executes the <c>init_per_suite</c> hooks then the + <c>init_per_group</c> or <c>per_testcase</c> hooks. So if you want to spawn a + process in the CTH, you cannot link with the CTH process, as it exits + after the post hook ends. Also, if you for some reason need an ETS + table with your CTH, you must spawn a process that handles it.</p> </section> <section> - <title>External configuration data and Logging</title> - <p>It's possible in the CTH to read configuration data values - by calling <seealso marker="ct#get_config-1"><c>ct:get_config/1/2/3</c></seealso> (as explained in the - <seealso marker="config_file_chapter#require_config_data"> - External configuration data</seealso> - chapter). The config variables in question must, as always, first have been - <c>required</c> by means of a suite-, group-, or test case info function, - or the <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso> function. Note that the latter can also be used - in CT hook functions.</p> - <p>The CT hook functions may call any of the logging functions available + <title>External Configuration Data and Logging</title> + <p>Configuration data values in the CTH can be read + by calling + <seealso marker="ct#get_config-1"><c>ct:get_config/1,2,3</c></seealso> + (as explained in section + <seealso marker="config_file_chapter#require_config_data">Requiring and Reading Configuration Data</seealso>). + The configuration variables in question must, as always, first have been + required by a suite-, group-, or test case information function, + or by function <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso>. + The latter can also be used in CT hook functions.</p> + <p>The CT hook functions can call any logging function in the <c>ct</c> interface to print information to the log files, or to add comments in the suite overview page. </p> @@ -208,307 +212,328 @@ </section> - <marker id="manipulating"/> <section> - <title>Manipulating tests</title> - <p>It is through CTHs possible to manipulate the results of tests and - configuration functions. The main purpose of doing this with CTHs is to - allow common patterns to be abstracted out from test test suites and applied to - multiple test suites without duplicating any code. All of the callback - functions for a CTH follow a common interface, this interface is - described below.</p> - - <p>Common Test will always call all available hook functions, even pre- and post - hooks for configuration functions that are not implemented in the suite. + <marker id="manipulating"/> + <title>Manipulating Tests</title> + <p>Through CTHs the results of tests and configuration functions can be manipulated. + The main purpose to do this with CTHs is to allow common + patterns to be abstracted out from test suites and applied to + multiple test suites without duplicating any code. All the callback + functions for a CTH follow a common interface described hereafter.</p> + + <p><c>Common Test</c> always calls all available hook functions, even pre- + and post hooks for configuration functions that are not implemented in the suite. For example, <c>pre_init_per_suite(x_SUITE, ...)</c> and - <c>post_init_per_suite(x_SUITE, ...)</c> will be called for test suite - <c>x_SUITE</c>, even if it doesn't export <c>init_per_suite/1</c>. This feature - makes it possible to use hooks as configuration fallbacks, or even - completely replace all configuration functions with hook functions.</p> + <c>post_init_per_suite(x_SUITE, ...)</c> are called for test suite + <c>x_SUITE</c>, even if it does not export <c>init_per_suite/1</c>. + With this feature hooks can be used as configuration fallbacks, and all + configuration functions can be replaced with hook functions.</p> - <marker id="pre"/> <section> + <marker id="pre"/> <title>Pre Hooks</title> <p> - It is possible in a CTH to hook in behaviour before - <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">init_per_testcase</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and - <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. + In a CTH, the behavior can be hooked in before the following functions:</p> + + <list type="bulleted"> + <item><seealso marker="common_test#Module:init_per_suite-1"><c>init_per_suite</c></seealso></item> + <item><seealso marker="common_test#Module:init_per_group-2"><c>init_per_group</c></seealso></item> + <item><seealso marker="common_test#Module:init_per_testcase-2"><c>init_per_testcase</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_testcase-2"><c>end_per_testcase</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_group-2"><c>end_per_group</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_suite-1"><c>end_per_suite</c></seealso></item> + </list> + + <p> This is done in the CTH functions called pre_<name of function>. - All of these functions take the same three arguments: <c>Name</c>, - <c>Config</c> and <c>CTHState</c>. The return value of the CTH function - is always a combination of an result for the suite/group/test and an - updated <c>CTHState</c>. If you want the test suite to continue on - executing you should return the config list which you want the test to - use as the result. If you for some reason want to skip/fail the test, - return a tuple with <c>skip</c> or <c>fail</c> and a reason as the - result. Example: - </p> - <code>pre_init_per_suite(SuiteName, Config, CTHState) -> - case db:connect() of - {error,_Reason} -> - {{fail, "Could not connect to DB"}, CTHState}; - {ok, Handle} -> - {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} - end.</code> - <note><p>If using multiple CTHs, the first part of the return tuple will be - used as input for the next CTH. So in the case above the next CTH might + These functions take the same three arguments, <c>Name</c>, + <c>Config</c>, and <c>CTHState</c>. The return value of the CTH function + is always a combination of a result for the suite/group/test and an + updated <c>CTHState</c>.</p> + + <p>To let the test suite continue on executing, return the configuration + list that you want the test to use as the result. To skip or + fail the test, return a tuple with <c>skip</c> or <c>fail</c>, and a reason + as the result.</p> + + <p><em>Example:</em></p> + <code> + pre_init_per_suite(SuiteName, Config, CTHState) -> + case db:connect() of + {error,_Reason} -> + {{fail, "Could not connect to DB"}, CTHState}; + {ok, Handle} -> + {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} + end.</code> + + <note><p>If you use multiple CTHs, the first part of the return tuple is + used as input for the next CTH. So in the previous example the next CTH can get <c>{fail,Reason}</c> as the second parameter. If you have many CTHs - which interact, it might be a good idea to not let each CTH return - <c>fail</c> or <c>skip</c>. Instead return that an action should be taken - through the <c>Config</c> list and implement a CTH which at the end takes - the correct action.</p></note> + interacting, do not let each CTH return <c>fail</c> or <c>skip</c>. + Instead, return that an action is to be taken through the <c>Config</c> + list and implement a CTH that, at the end, takes the correct action.</p></note> </section> - <marker id="post"/> <section> + <marker id="post"/> <title>Post Hooks</title> - <p>It is also possible in a CTH to hook in behaviour after - <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">init_per_group</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">end_per_testcase</seealso>, - <seealso marker="common_test#Module:init_per_suite-1">end_per_group</seealso> and - <seealso marker="common_test#Module:init_per_suite-1">end_per_suite</seealso>. - This is done in the CTH functions called post_<name of function>. - All of these function take the same four arguments: <c>Name</c>, - <c>Config</c>, <c>Return</c> and <c>CTHState</c>. <c>Config</c> in this + <p>In a CTH, behavior can be hooked in after the following functions:</p> + <list type="bulleted"> + <item><seealso marker="common_test#Module:init_per_suite-1"><c>init_per_suite</c></seealso></item> + <item><seealso marker="common_test#Module:init_per_group-2"><c>init_per_group</c></seealso></item> + <item><seealso marker="common_test#Module:init_per_testcase-2"><c>init_per_testcase</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_testcase-2"><c>end_per_testcase</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_group-2"><c>end_per_group</c></seealso></item> + <item><seealso marker="common_test#Module:end_per_suite-1"><c>end_per_suite</c></seealso></item> + </list> + + <p> + This is done in the CTH functions called <c>post_<name of function></c>. + These functions take the same four arguments, <c>Name</c>, + <c>Config</c>, <c>Return</c>, and <c>CTHState</c>. <c>Config</c> in this case is the same <c>Config</c> as the testcase is called with. <c>Return</c> is the value returned by the testcase. If the testcase - failed by crashing, <c>Return</c> will be + fails by crashing, <c>Return</c> is <c>{'EXIT',{{Error,Reason},Stacktrace}}</c>.</p> - <p>The return value of the CTH function is always a combination of an + <p>The return value of the CTH function is always a combination of a result for the suite/group/test and an updated <c>CTHState</c>. If - you want the callback to not affect the outcome of the test you should + you do not want the callback to affect the outcome of the test, return the <c>Return</c> data as it is given to the CTH. You can also - modify the result of the test. By returning the <c>Config</c> list - with the <c>tc_status</c> element removed you can recover from a test + modify the test result. By returning the <c>Config</c> list + with element <c>tc_status</c> removed, you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip - the test case in the post hook. Example: </p> - - <code>post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -> - case db:check_consistency() of - true -> - %% DB is good, pass the test. - {proplists:delete(tc_status, Config), CTHState}; - false -> - %% DB is not good, mark as skipped instead of failing - {{skip, "DB is inconsisten!"}, CTHState} - end; -post_end_per_testcase(_TC, Config, Return, CTHState) -> - %% Do nothing if tc does not crash. - {Return, CTHState}.</code> - - <note><p>Recovering from a testcase failure using CTHs should only be done as - a last resort. If used wrongly it could become very difficult to - determine which tests pass or fail in a test run</p></note> + the test case in the post hook.</p> + + <p><em>Example:</em></p> + <code> + post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) -> + case db:check_consistency() of + true -> + %% DB is good, pass the test. + {proplists:delete(tc_status, Config), CTHState}; + false -> + %% DB is not good, mark as skipped instead of failing + {{skip, "DB is inconsisten!"}, CTHState} + end; + post_end_per_testcase(_TC, Config, Return, CTHState) -> + %% Do nothing if tc does not crash. + {Return, CTHState}.</code> + + <note><p>Do recover from a testcase failure using CTHs only a last resort. + If used wrongly, it can be very difficult to determine which tests that + pass or fail in a test run.</p></note> </section> - <marker id="skip_n_fail"/> <section> - <title>Skip and Fail hooks</title> + <title>Skip and Fail Hooks</title> <p> After any post hook has been executed for all installed CTHs, <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_fail</seealso> - or <seealso marker="ct_hooks#Module:on_tc_fail-3">on_tc_skip</seealso> - might be called if the testcase failed or was skipped - respectively. You cannot affect the outcome of the tests any further at - this point. + or <seealso marker="ct_hooks#Module:on_tc_skip-3">on_tc_skip</seealso> + is called if the testcase failed or was skipped, respectively. + You cannot affect the outcome of the tests any further at this point. </p> </section> </section> - <marker id="synchronizing"/> <section> - <title>Synchronizing external user applications with Common Test</title> + <marker id="synchronizing"/> + <title>Synchronizing External User Applications with Common Test</title> <p>CTHs can be used to synchronize test runs with external user applications. - The init function may e.g. start and/or communicate with an application that - has the purpose of preparing the SUT for an upcoming test run, or maybe + The init function can, for example, start and/or communicate with an application that + has the purpose of preparing the SUT for an upcoming test run, or initialize a database for saving test data to during the test run. The - terminate function may similarly order such an application to reset the SUT + terminate function can similarly order such an application to reset the SUT after the test run, and/or tell the application to finish active sessions and terminate. Any system error- or progress reports generated during the init- or - termination stage will be saved in the - <seealso marker="run_test_chapter#pre_post_test_io_log">Pre- - and post test I/O log</seealso>. (This is also true for any printouts made + termination stage are saved in the + <seealso marker="run_test_chapter#pre_post_test_io_log">Pre- and Post Test I/O Log</seealso>. + (This is also true for any printouts made with <c>ct:log/2</c> and <c>ct:pal/2</c>).</p> - <p>In order to ensure that Common Test doesn't start executing tests, or + + <p>To ensure that <c>Common Test</c> does not start executing tests, or closes its log files and shuts down, before the external application - is ready for it, Common Test may be synchronized with the application. - During startup and shutdown, Common Test can be suspended, simply by + is ready for it, <c>Common Test</c> can be synchronized with the application. + During startup and shutdown, <c>Common Test</c> can be suspended, simply by having a CTH evaluate a <c>receive</c> expression in the init- or terminate function. The macros <c>?CT_HOOK_INIT_PROCESS</c> (the process executing the hook init function) and <c>?CT_HOOK_TERMINATE_PROCESS</c> (the process executing - the hook terminate function), each specifies the name of the correct Common Test - process to send a message to in order to return from the <c>receive</c>. + the hook terminate function) each specifies the name of the correct <c>Common Test</c> + process to send a message to. This is done to return from the <c>receive</c>. These macros are defined in <c>ct.hrl</c>. </p> </section> - <marker id="example"/> <section> + <marker id="example"/> <title>Example CTH</title> - <p>The CTH below will log information about a test run into a format - parseable by <seealso marker="kernel:file#consult-1">file:consult/1</seealso>. + <p>The following CTH logs information about a test run into a format + parseable by <seealso marker="kernel:file#consult-1">file:consult/1</seealso> + (in <c>Kernel</c>): </p> - <code>%%% @doc Common Test Example Common Test Hook module. --module(example_cth). - -%% Callbacks --export([id/1]). --export([init/2]). - --export([pre_init_per_suite/3]). --export([post_init_per_suite/4]). --export([pre_end_per_suite/3]). --export([post_end_per_suite/4]). - --export([pre_init_per_group/3]). --export([post_init_per_group/4]). --export([pre_end_per_group/3]). --export([post_end_per_group/4]). - --export([pre_init_per_testcase/3]). --export([post_end_per_testcase/4]). - --export([on_tc_fail/3]). --export([on_tc_skip/3]). - --export([terminate/1]). - --record(state, { file_handle, total, suite_total, ts, tcs, data }). - -%% @doc Return a unique id for this CTH. -id(Opts) -> - proplists:get_value(filename, Opts, "/tmp/file.log"). - -%% @doc Always called before any other callback function. Use this to initiate -%% any common state. -init(Id, Opts) -> - {ok,D} = file:open(Id,[write]), - {ok, #state{ file_handle = D, total = 0, data = [] }}. - -%% @doc Called before init_per_suite is called. -pre_init_per_suite(Suite,Config,State) -> - {Config, State#state{ suite_total = 0, tcs = [] }}. - -%% @doc Called after init_per_suite. -post_init_per_suite(Suite,Config,Return,State) -> - {Return, State}. - -%% @doc Called before end_per_suite. -pre_end_per_suite(Suite,Config,State) -> - {Config, State}. - -%% @doc Called after end_per_suite. -post_end_per_suite(Suite,Config,Return,State) -> - Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)}, - {Return, State#state{ data = [Data | State#state.data] , - total = State#state.total + State#state.suite_total } }. - -%% @doc Called before each init_per_group. -pre_init_per_group(Group,Config,State) -> - {Config, State}. - -%% @doc Called after each init_per_group. -post_init_per_group(Group,Config,Return,State) -> - {Return, State}. - -%% @doc Called after each end_per_group. -pre_end_per_group(Group,Config,State) -> - {Config, State}. - -%% @doc Called after each end_per_group. -post_end_per_group(Group,Config,Return,State) -> - {Return, State}. - -%% @doc Called before each test case. -pre_init_per_testcase(TC,Config,State) -> - {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }. - -%% @doc Called after each test case. -post_end_per_testcase(TC,Config,Return,State) -> - TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)}, - {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }. - -%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group, -%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed. -on_tc_fail(TC, Reason, State) -> - State. - -%% @doc Called when a test case is skipped by either user action -%% or due to an init function failing. -on_tc_skip(TC, Reason, State) -> - State. - -%% @doc Called when the scope of the CTH is done -terminate(State) -> - io:format(State#state.file_handle, "~p.~n", - [{test_run, State#state.total, State#state.data}]), - file:close(State#state.file_handle), - ok.</code> + <code> + %%% @doc Common Test Example Common Test Hook module. + -module(example_cth). + + %% Callbacks + -export([id/1]). + -export([init/2]). + + -export([pre_init_per_suite/3]). + -export([post_init_per_suite/4]). + -export([pre_end_per_suite/3]). + -export([post_end_per_suite/4]). + + -export([pre_init_per_group/3]). + -export([post_init_per_group/4]). + -export([pre_end_per_group/3]). + -export([post_end_per_group/4]). + + -export([pre_init_per_testcase/3]). + -export([post_init_per_testcase/4]). + -export([pre_end_per_testcase/3]). + -export([post_end_per_testcase/4]). + + -export([on_tc_fail/3]). + -export([on_tc_skip/3]). + + -export([terminate/1]). + + -record(state, { file_handle, total, suite_total, ts, tcs, data }). + + %% @doc Return a unique id for this CTH. + id(Opts) -> + proplists:get_value(filename, Opts, "/tmp/file.log"). + + %% @doc Always called before any other callback function. Use this to initiate + %% any common state. + init(Id, Opts) -> + {ok,D} = file:open(Id,[write]), + {ok, #state{ file_handle = D, total = 0, data = [] }}. + + %% @doc Called before init_per_suite is called. + pre_init_per_suite(Suite,Config,State) -> + {Config, State#state{ suite_total = 0, tcs = [] }}. + + %% @doc Called after init_per_suite. + post_init_per_suite(Suite,Config,Return,State) -> + {Return, State}. + + %% @doc Called before end_per_suite. + pre_end_per_suite(Suite,Config,State) -> + {Config, State}. + + %% @doc Called after end_per_suite. + post_end_per_suite(Suite,Config,Return,State) -> + Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)}, + {Return, State#state{ data = [Data | State#state.data] , + total = State#state.total + State#state.suite_total } }. + + %% @doc Called before each init_per_group. + pre_init_per_group(Group,Config,State) -> + {Config, State}. + + %% @doc Called after each init_per_group. + post_init_per_group(Group,Config,Return,State) -> + {Return, State}. + + %% @doc Called before each end_per_group. + pre_end_per_group(Group,Config,State) -> + {Config, State}. + + %% @doc Called after each end_per_group. + post_end_per_group(Group,Config,Return,State) -> + {Return, State}. + + %% @doc Called before each init_per_testcase. + pre_init_per_testcase(TC,Config,State) -> + {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }. + + %% Called after each init_per_testcase (immediately before the test case). + post_init_per_testcase(TC,Config,Return,State) -> + {Return, State} + +%% @doc Called before each end_per_testcase (immediately after the test case). + pre_end_per_testcase(TC,Config,State) -> + {Config, State}. + + %% @doc Called after each end_per_testcase. + post_end_per_testcase(TC,Config,Return,State) -> + TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)}, + {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }. + + %% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group, + %% post_end_per_group and post_end_per_testcase if the suite, group or test case failed. + on_tc_fail(TC, Reason, State) -> + State. + + %% @doc Called when a test case is skipped by either user action + %% or due to an init function failing. + on_tc_skip(TC, Reason, State) -> + State. + + %% @doc Called when the scope of the CTH is done + terminate(State) -> + io:format(State#state.file_handle, "~p.~n", + [{test_run, State#state.total, State#state.data}]), + file:close(State#state.file_handle), + ok.</code> </section> - <marker id="builtin_cths"/> <section> - <title>Built-in CTHs</title> - <p>Common Test is delivered with a couple of general purpose CTHs that - can be enabled by the user to provide some generic testing functionality. - Some of these are enabled by default when starting running common_test, - they can be disabled by setting <c>enable_builtin_hooks</c> to - <c>false</c> on the command line or in the test specification. In the - table below there is a list of all current CTHs which are delivered with - Common Test.</p> - - <table> - <row> - <cell align="left"><em>CTH Name</em></cell> - <cell align="left"><em>Is Built-in</em></cell> - <cell align="left"><em>Description</em></cell> - </row> - <row> - <cell align="left">cth_log_redirect</cell> - <cell align="left">yes</cell> - <cell align="left">Captures all error_logger and SASL logging events and prints them - to the current test case log. If an event can not be associated with a - testcase it will be printed in the common test framework log. This will - happen for testcases which are run in parallel and events which occur - inbetween testcases. You can configure the level of - <seealso marker="sasl:sasl_app">SASL</seealso> events report - using the normal SASL mechanisms. </cell> - </row> - <row> - <cell align="left">cth_surefire</cell> - <cell align="left">no</cell> - <cell align="left"><p>Captures all test results and outputs them as surefire - XML into a file. The file which is created is by default - called junit_report.xml. The file name can be changed by - setting the <c>path</c> option for this hook, e.g.</p> - - <code>-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]</code> - - <p>If the <c>url_base</c> option is set, an additional - attribute named <c>url</c> will be added to each - <c>testsuite</c> and <c>testcase</c> XML element. The value will - be constructed from the <c>url_base</c> and a relative path - to the test suite or test case log respectively, e.g.</p> - - <code>-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]</code> - <p>will give a url attribute value similar to</p> - - <code>"http://myserver.com/[email protected]_11.19.39/ -x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code> - - <p>Surefire XML can for instance be used by Jenkins to display test - results.</p></cell> - </row> - </table> + <marker id="builtin_cths"/> + <title>Built-In CTHs</title> + <p><c>Common Test</c> is delivered with some general-purpose CTHs that + can be enabled by the user to provide generic testing functionality. + Some of these CTHs are enabled by default when <c>common_test</c> is started to run. + They can be disabled by setting <c>enable_builtin_hooks</c> to + <c>false</c> on the command line or in the test specification. The following + two CTHs are delivered with <c>Common Test</c>:</p> + + <taglist> + <tag><c>cth_log_redirect</c></tag> + <item> + <p>Built-in</p> + <p>Captures all <c>error_logger</c> and <c>SASL</c> logging + events and prints them to the current test case log. If an event cannot be + associated with a test case, it is printed in the <c>Common Test</c> framework log. + This happens for test cases running in parallel and events occuring + in-between test cases. You can configure the level of + <seealso marker="sasl:sasl_app"><c>SASL</c></seealso> events report + using the normal <c>SASL</c> mechanisms.</p> + </item> + <tag><c>cth_surefire</c></tag> + <item> + <p>Not built-in</p> + <p>Captures all test results and outputs them as surefire + XML into a file. The created file is by default + called <c>junit_report.xml</c>. The file name can be changed by + setting option <c>path</c> for this hook, for example:</p> + + <p><c>-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]</c></p> + + <p>If option <c>url_base</c> is set, an extra + attribute named <c>url</c> is added to each + <c>testsuite</c> and <c>testcase</c> XML element. The value + is constructed from <c>url_base</c> and a relative path + to the test suite or test case log, respectively, for example:</p> + + <p><c>-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]</c></p> + + <p>gives an URL attribute value similar to</p> + + <p><c>"http://myserver.com/[email protected]_11.19.39/ +x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</c></p> + + <p>Surefire XML can, for example, be used by Jenkins to display test + results.</p> + </item> + </taglist> </section> |