path: root/lib/common_test/doc/src/ct_suite_callbacks.xml

<?xml version="1.0" encoding="UTF-8" ?>

<!DOCTYPE erlref SYSTEM "erlref.dtd">

<erlref>
  <header>
    <copyright>
      <year>2010</year><year>2010</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.

      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.

    </legalnotice>

    <title>Suite Callbacks</title>
    <prepared>Lukas Larsson</prepared>
    <responsible>Lukas Larsson</responsible>
    <docno></docno>
    <approved></approved>
    <checked></checked>
    <date>2010-12-02</date>
    <rev>PA1</rev>
    <file>suite_callback.sgml</file>
  </header>
  <module>ct_suite_callback</module> 
  <modulesummary>A callback interface on top of Common Test</modulesummary>

  <description>

    <warning><p>This feature is in alpha release right now. This means that the 
	interface may change in the future and that there may be bugs. We 
	encourage you to use this feature, but be prepared 
	that there might be bugs and that the interface might change
	inbetween releases.</p></warning>

    <p>The <em>Suite Callback</em> (henceforth called SCB) framework allows 
      extensions of the default behaviour of Common Test by means of callbacks 
      before and after all test suite calls. It is meant for advanced users of
      Common Test which want to abstract out behaviour which is common to
      multiple test suites. </p>

    <p>In brief, Suite Callbacks allows you to:</p>

    <list>
      <item>Manipulate the runtime config before each suite 
      configuration calls</item>
      <item>Manipulate the return of all suite configuration calls and in 
      extension the result of the test themselves.</item>
    </list>
    
    <p>The following sections describe the mandatory and optional SCB
    functions Common Test will call during test execution. For more details
    see <seealso marker="suite_callbacks_chapter">Suite Callbacks</seealso> in 
    the User's Guide.</p>

    <p>For information about how to add a SCB to your suite see 
    <seealso marker="suite_callbacks_chapter#installing">Installing an SCB
    </seealso> in the User's Guide.</p>

    <note><p>See the
	<seealso marker="suite_callbacks_chapter#example">Example SCB</seealso>
	in the User's Guide for a minimal example of an SCB. </p></note>
    
  </description>

  <section>
    <title>CALLBACK FUNCTIONS</title>
    <p>The following functions define the callback interface
      for a suite callback.</p>
  </section>
  
  <funcs>
    <func>
      <name>Module:init(Id, Opts) -&gt; State</name>
      <fsummary>Initiates the Suite Callback</fsummary>
      <type>
	<v>Id = reference() | term()</v>
	<v>Opts = term()</v>
	<v>State = term()</v>
      </type>
      
      <desc>	
	<p> MANDATORY </p>

	<p>Always called before any other callback function. 
	  Use this to initiate any common state. 
	  It should return a state for this SCB.</p>

	<p><c>Id</c> is the return value of 
	  <seealso marker="#Module:id-1">id/1</seealso>, or a <c>reference</c>
	  if <seealso marker="#Module:id-1">id/1</seealso> is not implemented.
	</p>

	<p>For details about when init is called see
	  <seealso marker="suite_callbacks_chapter#scope">scope</seealso>
	  in the User's Guide.</p>
	      
      </desc>
    </func>

    <func>
      <name>Module:pre_init_per_suite(SuiteName, Config, SCBState) -&gt; 
              Result</name>
      <fsummary>Called before init_per_suite</fsummary>
      <type>
	<v>SuiteName = atom()</v>
	<v>Config = NewConfig = [{Key,Value}]</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {Return, NewSCBState}</v>
	<v>Return = NewConfig | SkipOrFail</v>
	<v>SkipOrFail = {fail, Reason} | {skip, Reason}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called before 
	<seealso marker="common_test#Module:init_per_suite-1">
	  init_per_suite</seealso> if it exists. 
	It typically contains initialization/logging which needs to be done 
	before init_per_suite is called. 
	If <c>{skip,Reason}</c> or <c>{fail,Reason}</c> is returned, 
	init_per_suite and all test cases of the suite will be skipped and
	Reason printed in the overview log of the suite.</p>
	
	<p><c>SuiteName</c> is the name of the suite to be run.</p>

	<p><c>Config</c> is the original config list of the test suite.</p>

	<p><c>SCBState</c> is the current internal state of the SCB.</p>

	<p><c>Return</c> is the result of the init_per_suite function.
	If it is <c>{skip,Reason}</c> or <c>{fail,Reason}</c> 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
	</seealso> will never be called, instead the initiation is considered 
	to be skipped/failed respectively. If a <c>NewConfig</c> list
	is returned, <seealso marker="common_test#Module:init_per_suite-1">
	init_per_suite</seealso> will be called with that <c>NewConfig</c> list.
	See <seealso marker="suite_callbacks_chapter#manipulating">
	Manipulating tests</seealso> in the User's Guide for more details.</p>
	
	
	<p>Note that this function is only called if the SCB has been added 
	before init_per_suite is run, see 
	<seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> 
	in the User's Guide for details.</p>
      </desc>
    </func>
    
    <func>
      <name>Module:post_init_per_suite(SuiteName, Config, Return, SCBState) -&gt; 
              Result</name>
      <fsummary>Called after init_per_suite</fsummary>
      <type>
	<v>SuiteName = atom()</v>
	<v>Config = [{Key,Value}]</v>
	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
	<v>SkipOrFail = {fail, Reason} | {skip, Reason} | term()</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewReturn, NewSCBState}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>
	
	<p><c>Return</c> is what 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
	</seealso> returned, i.e. {fail,Reason}, {skip,Reason}, a <c>Config</c>
	list or a term describing how 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
	</seealso> failed.</p>

	<p><c>NewReturn</c> is the possibly modified return value of
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
	</seealso>. It is here possible to recover from a failure in 
	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
	</seealso> by returning the <c>ConfigList</c> with the <c>tc_status</c>
	element removed.</p>

	<p><c>SCBState</c> is the current internal state of the SCB.</p>

	<p>This function is called after
	<seealso marker="common_test#Module:init_per_suite-1">
	init_per_suite</seealso> if it exists.</p>

	<p>Note that this function is only called if the SCB has been added 
	before or in init_per_suite, see 
	<seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> 
	in the User's Guide for details.</p>
      </desc>
    </func>
    
    <func>
      <name>Module:pre_init_per_group(GroupName, Config, SCBState) -&gt; 
              Result</name>
      <fsummary>Called before init_per_group</fsummary>
      <type>
	<v>GroupName = atom()</v>
	<v>Config = NewConfig = [{Key,Value}]</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>
	
	<p>This function is called before
	<seealso marker="common_test#Module:init_per_group-2">
	  init_per_group</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
	pre_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:init_per_group-2">
	init_per_group</seealso> instead.</p>
      </desc>
    </func>
    
    <func>
      <name>Module:post_init_per_group(GroupName, Config, Return, SCBState) -&gt; 
              Result</name>
      <fsummary>Called after init_per_group</fsummary>
      <type>
	<v>GroupName = atom()</v>
	<v>Config = [{Key,Value}]</v>
	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewReturn, NewSCBState}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called after
	<seealso marker="common_test#Module:init_per_group-2">
	  init_per_group</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
	post_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:init_per_group-2">
	init_per_group</seealso> instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:pre_init_per_testcase(TestcaseName, Config, SCBState) -&gt; 
              Result</name>
      <fsummary>Called before init_per_testcase</fsummary>
      <type>
	<v>TestcaseName = atom()</v>
	<v>Config = NewConfig = [{Key,Value}]</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called before
	<seealso marker="common_test#Module:init_per_testcase-2">
	  init_per_testcase</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
	pre_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:init_per_testcase-2">
	init_per_testcase</seealso> function instead.</p>

	<p>Note that it is not possible to add SCB's here right now, 
	that feature might be added later, 
	but it would right now break backwards compatability.</p>
      </desc>
    </func>

    <func>
      <name>Module:post_end_per_testcase(TestcaseName, Config, Return, SCBState)
               -&gt; Result</name>
      <fsummary>Called after end_per_testcase</fsummary>
      <type>
	<v>TestcaseName = atom()</v>
	<v>Config = [{Key,Value}]</v>
	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewReturn, NewSCBState}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called after
	<seealso marker="common_test#Module:end_per_testcase-2">
	  end_per_testcase</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
	post_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:end_per_testcase-2">
	end_per_testcase</seealso> function instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:pre_end_per_group(GroupName, Config, SCBState) -&gt; 
              Result</name>
      <fsummary>Called before end_per_group</fsummary>
      <type>
	<v>GroupName = atom()</v>
	<v>Config = NewConfig = [{Key,Value}]</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>
	
	<p>This function is called before
	<seealso marker="common_test#Module:end_per_group-2">
	end_per_group</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
	pre_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:end_per_group-2">
	end_per_group</seealso> function instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:post_end_per_group(GroupName, Config, Return, SCBState) -&gt; 
              Result</name>
      <fsummary>Called after end_per_group</fsummary>
      <type>
	<v>GroupName = atom()</v>
	<v>Config = [{Key,Value}]</v>
	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewReturn, NewSCBState}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called after
	<seealso marker="common_test#Module:end_per_group-2">
	  end_per_group</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
	post_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:end_per_group-2">
	end_per_group</seealso> function instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:pre_end_per_suite(SuiteName, Config, SCBState) -&gt; 
              Result</name>
      <fsummary>Called before end_per_suite</fsummary>
      <type>
	<v>SuiteName = atom()</v>
	<v>Config = NewConfig = [{Key,Value}]</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called before
	<seealso marker="common_test#Module:end_per_suite-1">
	  end_per_suite</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
	pre_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:end_per_suite-2">
	end_per_suite</seealso> function instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:post_end_per_suite(SuiteName, Config, Return, SCBState) -&gt; 
              Result</name>
      <fsummary>Called after end_per_suite</fsummary>
      <type>
	<v>SuiteName = atom()</v>
	<v>Config = [{Key,Value}]</v>
	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
	<v>SCBState = NewSCBState = term()</v>
	<v>Result = {NewReturn, NewSCBState}</v>
	<v>Key = atom()</v>
	<v>Value = term()</v>
	<v>Reason = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called after
	<seealso marker="common_test#Module:end_per_suite-1">
	  end_per_suite</seealso> if it exists. It behaves the same way as 
	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
	post_init_per_suite</seealso>, but for the
	<seealso marker="common_test#Module:end_per_suite-2">
	end_per_suite</seealso> function instead.</p>
      </desc>
    </func>

    <func>
      <name>Module:on_tc_fail(TestcaseName, Reason, SCBState) -&gt; 
              NewSCBState</name>
      <fsummary>Called after the SCB scope ends</fsummary>
      <type>
	<v>TestcaseName = init_per_suite | end_per_suite | 
	                  init_per_group | end_per_group | atom()</v>
	<v>Reason = term()</v>
	<v>SCBState = NewSCBState = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called whenever a testcase fails. 
	It is called after the post function has been called for 
	the testcase which failed. i.e. 
	if init_per_suite fails this function is called after 
	<seealso marker="#Module:post_init_per_suite-4">
	  post_init_per_suite</seealso>, and if a testcase fails it is called 
	  after <seealso marker="#Module:post_end_per_testcase-4">
	  post_end_per_testcase</seealso>.</p>

	<p>The data which comes with the Reason follows the same format as the
	<seealso marker="event_handler_chapter#failreason">FailReason
	</seealso> in the <seealso marker="event_handler_chapter#tc_done">tc_done</seealso> event.
	See <seealso marker="event_handler_chapter#events">Event Handling
	</seealso> in the User's Guide for details.</p>
      </desc>
    </func>

    <func>
      <name>Module:on_tc_skip(TestcaseName, Reason, SCBState) -&gt; 
              NewSCBState</name>
      <fsummary>Called after the SCB scope ends</fsummary>
      <type>
	<v>TestcaseName = end_per_suite | init_per_group | 
                          end_per_group | atom()</v>
	<v>Reason = {tc_auto_skip | tc_user_skip, term()}</v>
	<v>SCBState = NewSCBState = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called whenever a testcase is skipped. 
	It is called after the post function has been called for the 
	testcase which was skipped. 
	i.e. if init_per_group is skipped this function is called after 
	<seealso marker="#Module:post_init_per_suite-4">post_init_per_group
	</seealso>, and if a testcase is skipped it is called after 
	<seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase
	</seealso>.</p>

	<p>The data which comes with the Reason follows the same format as 
	<seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip
	</seealso> and <seealso marker="event_handler_chapter#tc_user_skip">
	tc_user_skip</seealso> events.
	See <seealso marker="event_handler_chapter#events">Event Handling
	</seealso> in the User's Guide for details.</p>
      </desc>
    </func>

    <func>
      <name>Module:terminate(SCBState)</name>
      <fsummary>Called after the SCB scope ends</fsummary>
      <type>
	<v>SCBState = term()</v>
      </type>
      
      <desc>	
	<p> OPTIONAL </p>

	<p>This function is called at the end of an SCB's 
	<seealso marker="suite_callbacks_chapter#scope">scope</seealso>. 
	</p>
      </desc>
    </func>

    <func>
      <name>Module:id(Opts) -&gt; Id</name>
      <fsummary>Called before the init function of an SCB</fsummary>
      <type>
	<v>Opts = term()</v>
	<v>Id = term()</v>
      </type>
      
      <desc>
	<p> OPTIONAL </p>
	
	<p>The <c>Id</c> is used to uniquely identify an SCB instance, 
	if two SCB's return the same <c>Id</c> the second SCB is ignored
	and subsequent calls to the SCB will only be made to the first
	instance. For more information see 
	<seealso marker="suite_callbacks_chapter#installing">Installing an SCB
	</seealso> in the User's Guide.
        </p>

	<p>This function should NOT have any side effects as it might 
	be called multiple times by Common Test.</p>
	
	<p>If not implemented the SCB will act as if this function returned a
	  call to <c>make_ref/0</c>.</p>
      </desc>
    </func>

  </funcs>

</erlref>