diff options
Diffstat (limited to 'lib/common_test/doc/src/ct_suite_callbacks.xml')
-rw-r--r-- | lib/common_test/doc/src/ct_suite_callbacks.xml | 503 |
1 files changed, 480 insertions, 23 deletions
diff --git a/lib/common_test/doc/src/ct_suite_callbacks.xml b/lib/common_test/doc/src/ct_suite_callbacks.xml index 28a496a47b..cea5804825 100644 --- a/lib/common_test/doc/src/ct_suite_callbacks.xml +++ b/lib/common_test/doc/src/ct_suite_callbacks.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE erlref SYSTEM "erlref.dtd"> @@ -33,24 +33,43 @@ <file>suite_callback.sgml</file> </header> <module>ct_suite_callback</module> - <modulesummary>A callback interface on top of common test</modulesummary> + <modulesummary>A callback interface on top of Common Test</modulesummary> <description> - <p>The <em>Suite Callback</em> framework allows extensions of the default - behaviour of Common Test by means of callbacks before and after all - test suite calls. </p> + <warning><p>This feature is in alpha release right now. This means that the + interface may change in the future and that there may be bugs. We + encourage you to use this feature, but be prepared + that there might be bugs and that the interface might change + inbetween releases.</p></warning> + + <p>The <em>Suite Callback</em> (henceforth called SCB) framework allows + extensions of the default behaviour of Common Test by means of callbacks + before and after all test suite calls. It is meant for advanced users of + Common Test which want to abstract out behaviour which is common to + multiple test suites. </p> <p>In brief, Suite Callbacks allows you to:</p> <list> - <item>Manipulating the runtime config before each suite configuration calls</item> - <item>Manipulating the return of all suite configuration calls</item> + <item>Manipulate the runtime config before each suite + configuration calls</item> + <item>Manipulate the return of all suite configuration calls and in + extension the result of the test themselves.</item> </list> - <p>The following sections describe the mandatory and optional suite callback - functions Common Test will call during test execution. For more details - see <seealso marker="suite_callbacks">Suite Callbacks</seealso> in the User's Guide.</p> + <p>The following sections describe the mandatory and optional SCB + functions Common Test will call during test execution. For more details + see <seealso marker="suite_callbacks_chapter">Suite Callbacks</seealso> in + the User's Guide.</p> + + <p>For information about how to add a SCB to your suite see + <seealso marker="suite_callbacks_chapter#installing">Installing an SCB + </seealso> in the User's Guide.</p> + + <note><p>See the + <seealso marker="suite_callbacks_chapter#example">Example SCB</seealso> + in the User's Guide for a minimal example of an SCB. </p></note> </description> @@ -62,31 +81,469 @@ <funcs> <func> - <name>Module:init(Opts) -> {Id,State} </name> - <fsummary>asdas</fsummary> + <name>Module:init(Id, Opts) -> State</name> + <fsummary>Initiates the Suite Callback</fsummary> <type> + <v>Id = reference() | term()</v> <v>Opts = term()</v> - <v>Id = term()</v> - <v>GroupName = term()</v> + <v>State = term()</v> </type> <desc> <p> MANDATORY </p> + + <p>Always called before any other callback function. + Use this to initiate any common state. + It should return a state for this SCB.</p> + + <p><c>Id</c> is the return value of + <seealso marker="#Module:id-1">id/1</seealso>, or a <c>reference</c> + if <seealso marker="#Module:id-1">id/1</seealso> is not implemented. + </p> + + <p>For details about when init is called see + <seealso marker="suite_callbacks_chapter#scope">scope</seealso> + in the User's Guide.</p> + + </desc> + </func> + + <func> + <name>Module:pre_init_per_suite(SuiteName, Config, SCBState) -> + Result</name> + <fsummary>Called before init_per_suite</fsummary> + <type> + <v>SuiteName = atom()</v> + <v>Config = NewConfig = [{Key,Value}]</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {Return, NewSCBState}</v> + <v>Return = NewConfig | SkipOrFail</v> + <v>SkipOrFail = {fail, Reason} | {skip, Reason}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called before + <seealso marker="common_test#Module:init_per_suite-1"> + init_per_suite</seealso> if it exists. + It typically contains initialization/logging which needs to be done + before init_per_suite is called. + If <c>{skip,Reason}</c> or <c>{fail,Reason}</c> is returned, + init_per_suite and all test cases of the suite will be skipped and + Reason printed in the overview log of the suite.</p> - <p></p> + <p><c>SuiteName</c> is the name of the suite to be run.</p> + + <p><c>Config</c> is the original config list of the test suite.</p> + + <p><c>SCBState</c> is the current internal state of the SCB.</p> + + <p><c>Return</c> is the result of the init_per_suite function. + If it is <c>{skip,Reason}</c> or <c>{fail,Reason}</c> + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite + </seealso> will never be called, instead the initiation is considered + to be skipped/failed respectively. If a <c>NewConfig</c> list + is returned, <seealso marker="common_test#Module:init_per_suite-1"> + init_per_suite</seealso> will be called with that <c>NewConfig</c> list. + See <seealso marker="suite_callbacks_chapter#manipulating"> + Manipulating tests</seealso> in the User's Guide for more details.</p> + + + <p>Note that this function is only called if the SCB has been added + before init_per_suite is run, see + <seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> + in the User's Guide for details.</p> + </desc> + </func> + + <func> + <name>Module:post_init_per_suite(SuiteName, Config, Return, SCBState) -> + Result</name> + <fsummary>Called after init_per_suite</fsummary> + <type> + <v>SuiteName = atom()</v> + <v>Config = [{Key,Value}]</v> + <v>Return = NewReturn = Config | SkipOrFail | term()</v> + <v>SkipOrFail = {fail, Reason} | {skip, Reason} | term()</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewReturn, NewSCBState}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p><c>Return</c> is what + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite + </seealso> returned, i.e. {fail,Reason}, {skip,Reason}, a <c>Config</c> + list or a term describing how + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite + </seealso> failed.</p> + + <p><c>NewReturn</c> is the possibly modified return value of + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite + </seealso>. It is here possible to recover from a failure in + <seealso marker="common_test#Module:init_per_suite-1">init_per_suite + </seealso> by returning the <c>ConfigList</c> with the <c>tc_status</c> + element removed.</p> + + <p><c>SCBState</c> is the current internal state of the SCB.</p> + + <p>This function is called after + <seealso marker="common_test#Module:init_per_suite-1"> + init_per_suite</seealso> if it exists.</p> + + <p>Note that this function is only called if the SCB has been added + before or in init_per_suite, see + <seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> + in the User's Guide for details.</p> + </desc> + </func> + + <func> + <name>Module:pre_init_per_group(GroupName, Config, SCBState) -> + Result</name> + <fsummary>Called before init_per_group</fsummary> + <type> + <v>GroupName = atom()</v> + <v>Config = NewConfig = [{Key,Value}]</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewConfig | SkipOrFail, NewSCBState}</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> - <p> If <c>{skip,Reason}</c> is returned, all test cases - in the module will be skipped, and the <c>Reason</c> will - be printed on the HTML result page.</p> + <p>This function is called before + <seealso marker="common_test#Module:init_per_group-2"> + init_per_group</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:pre_init_per_suite-3"> + pre_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:init_per_group-2"> + init_per_group</seealso> instead.</p> + </desc> + </func> + + <func> + <name>Module:post_init_per_group(GroupName, Config, Return, SCBState) -> + Result</name> + <fsummary>Called after init_per_group</fsummary> + <type> + <v>GroupName = atom()</v> + <v>Config = [{Key,Value}]</v> + <v>Return = NewReturn = Config | SkipOrFail | term()</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewReturn, NewSCBState}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> - <p>For details on groups, see - <seealso marker="write_test_chapter#test_case_groups">Test case - groups</seealso> in the User's Guide.</p> + <p>This function is called after + <seealso marker="common_test#Module:init_per_group-2"> + init_per_group</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:post_init_per_suite-4"> + post_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:init_per_group-2"> + init_per_group</seealso> instead.</p> + </desc> + </func> + + <func> + <name>Module:pre_init_per_testcase(TestcaseName, Config, SCBState) -> + Result</name> + <fsummary>Called before init_per_testcase</fsummary> + <type> + <v>TestcaseName = atom()</v> + <v>Config = NewConfig = [{Key,Value}]</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewConfig | SkipOrFail, NewSCBState}</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> - </desc> + <desc> + <p> OPTIONAL </p> + + <p>This function is called before + <seealso marker="common_test#Module:init_per_testcase-2"> + init_per_testcase</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:pre_init_per_suite-3"> + pre_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:init_per_testcase-2"> + init_per_testcase</seealso> function instead.</p> + + <p>Note that it is not possible to add SCB's here right now, + that feature might be added later, + but it would right now break backwards compatability.</p> + </desc> + </func> + + <func> + <name>Module:post_end_per_testcase(TestcaseName, Config, Return, SCBState) + -> Result</name> + <fsummary>Called after end_per_testcase</fsummary> + <type> + <v>TestcaseName = atom()</v> + <v>Config = [{Key,Value}]</v> + <v>Return = NewReturn = Config | SkipOrFail | term()</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewReturn, NewSCBState}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called after + <seealso marker="common_test#Module:end_per_testcase-2"> + end_per_testcase</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:post_init_per_suite-4"> + post_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:end_per_testcase-2"> + end_per_testcase</seealso> function instead.</p> + </desc> + </func> + + <func> + <name>Module:pre_end_per_group(GroupName, Config, SCBState) -> + Result</name> + <fsummary>Called before end_per_group</fsummary> + <type> + <v>GroupName = atom()</v> + <v>Config = NewConfig = [{Key,Value}]</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewConfig | SkipOrFail, NewSCBState}</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called before + <seealso marker="common_test#Module:end_per_group-2"> + end_per_group</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:pre_init_per_suite-3"> + pre_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:end_per_group-2"> + end_per_group</seealso> function instead.</p> + </desc> + </func> + + <func> + <name>Module:post_end_per_group(GroupName, Config, Return, SCBState) -> + Result</name> + <fsummary>Called after end_per_group</fsummary> + <type> + <v>GroupName = atom()</v> + <v>Config = [{Key,Value}]</v> + <v>Return = NewReturn = Config | SkipOrFail | term()</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewReturn, NewSCBState}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called after + <seealso marker="common_test#Module:end_per_group-2"> + end_per_group</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:post_init_per_suite-4"> + post_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:end_per_group-2"> + end_per_group</seealso> function instead.</p> + </desc> + </func> + + <func> + <name>Module:pre_end_per_suite(SuiteName, Config, SCBState) -> + Result</name> + <fsummary>Called before end_per_suite</fsummary> + <type> + <v>SuiteName = atom()</v> + <v>Config = NewConfig = [{Key,Value}]</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewConfig | SkipOrFail, NewSCBState}</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called before + <seealso marker="common_test#Module:end_per_suite-1"> + end_per_suite</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:pre_init_per_suite-3"> + pre_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:end_per_suite-2"> + end_per_suite</seealso> function instead.</p> + </desc> + </func> + + <func> + <name>Module:post_end_per_suite(SuiteName, Config, Return, SCBState) -> + Result</name> + <fsummary>Called after end_per_suite</fsummary> + <type> + <v>SuiteName = atom()</v> + <v>Config = [{Key,Value}]</v> + <v>Return = NewReturn = Config | SkipOrFail | term()</v> + <v>SkipOrFail = {fail,Reason} | {skip, Reason}</v> + <v>SCBState = NewSCBState = term()</v> + <v>Result = {NewReturn, NewSCBState}</v> + <v>Key = atom()</v> + <v>Value = term()</v> + <v>Reason = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called after + <seealso marker="common_test#Module:end_per_suite-1"> + end_per_suite</seealso> if it exists. It behaves the same way as + <seealso marker="ct_suite_callback#Module:post_init_per_suite-4"> + post_init_per_suite</seealso>, but for the + <seealso marker="common_test#Module:end_per_suite-2"> + end_per_suite</seealso> function instead.</p> + </desc> + </func> + + <func> + <name>Module:on_tc_fail(TestcaseName, Reason, SCBState) -> + NewSCBState</name> + <fsummary>Called after the SCB scope ends</fsummary> + <type> + <v>TestcaseName = init_per_suite | end_per_suite | + init_per_group | end_per_group | atom()</v> + <v>Reason = term()</v> + <v>SCBState = NewSCBState = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called whenever a testcase fails. + It is called after the post function has been called for + the testcase which failed. i.e. + if init_per_suite fails this function is called after + <seealso marker="#Module:post_init_per_suite-4"> + post_init_per_suite</seealso>, and if a testcase fails it is called + after <seealso marker="#Module:post_end_per_testcase-4"> + post_end_per_testcase</seealso>.</p> + + <p>The data which comes with the Reason follows the same format as the + <seealso marker="event_handler_chapter#failreason">FailReason + </seealso> in the <seealso marker="event_handler_chapter#tc_done">tc_done</seealso> event. + See <seealso marker="event_handler_chapter#events">Event Handling + </seealso> in the User's Guide for details.</p> + </desc> + </func> + + <func> + <name>Module:on_tc_skip(TestcaseName, Reason, SCBState) -> + NewSCBState</name> + <fsummary>Called after the SCB scope ends</fsummary> + <type> + <v>TestcaseName = end_per_suite | init_per_group | + end_per_group | atom()</v> + <v>Reason = {tc_auto_skip | tc_user_skip, term()}</v> + <v>SCBState = NewSCBState = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called whenever a testcase is skipped. + It is called after the post function has been called for the + testcase which was skipped. + i.e. if init_per_group is skipped this function is called after + <seealso marker="#Module:post_init_per_suite-4">post_init_per_group + </seealso>, and if a testcase is skipped it is called after + <seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase + </seealso>.</p> + + <p>The data which comes with the Reason follows the same format as + <seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip + </seealso> and <seealso marker="event_handler_chapter#tc_user_skip"> + tc_user_skip</seealso> events. + See <seealso marker="event_handler_chapter#events">Event Handling + </seealso> in the User's Guide for details.</p> + </desc> + </func> + + <func> + <name>Module:terminate(SCBState)</name> + <fsummary>Called after the SCB scope ends</fsummary> + <type> + <v>SCBState = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>This function is called at the end of an SCB's + <seealso marker="suite_callbacks_chapter#scope">scope</seealso>. + </p> + </desc> + </func> + + <func> + <name>Module:id(Opts) -> Id</name> + <fsummary>Called before the init function of an SCB</fsummary> + <type> + <v>Opts = term()</v> + <v>Id = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + + <p>The <c>Id</c> is used to uniquely identify an SCB instance, + if two SCB's return the same <c>Id</c> the second SCB is ignored + and subsequent calls to the SCB will only be made to the first + instance. For more information see + <seealso marker="suite_callbacks_chapter#installing">Installing an SCB + </seealso> in the User's Guide. + </p> + + <p>This function should NOT have any side effects as it might + be called multiple times by Common Test.</p> + + <p>If not implemented the SCB will act as if this function returned a + call to <c>make_ref/0</c>.</p> + </desc> </func> - </funcs> </erlref> |