Add documentation for SCBs

1 files changed, 480 insertions, 23 deletions

diff --git a/lib/common_test/doc/src/ct_suite_callbacks.xml b/lib/common_test/doc/src/ct_suite_callbacks.xml
index 28a496a47b..cea5804825 100644
--- a/lib/common_test/doc/src/ct_suite_callbacks.xml
+++ b/lib/common_test/doc/src/ct_suite_callbacks.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="latin1" ?>
+<?xml version="1.0" encoding="UTF-8" ?>
 
 <!DOCTYPE erlref SYSTEM "erlref.dtd">
 
@@ -33,24 +33,43 @@
     <file>suite_callback.sgml</file>
   </header>
   <module>ct_suite_callback</module> 
-  <modulesummary>A callback interface on top of common test</modulesummary>
+  <modulesummary>A callback interface on top of Common Test</modulesummary>
 
   <description>
 
-    <p>The <em>Suite Callback</em> framework allows extensions of the default
-       behaviour of Common Test by means of callbacks before and after all 
-       test suite calls. </p>
+    <warning><p>This feature is in alpha release right now. This means that the 
+	interface may change in the future and that there may be bugs. We 
+	encourage you to use this feature, but be prepared 
+	that there might be bugs and that the interface might change
+	inbetween releases.</p></warning>
+
+    <p>The <em>Suite Callback</em> (henceforth called SCB) framework allows 
+      extensions of the default behaviour of Common Test by means of callbacks 
+      before and after all test suite calls. It is meant for advanced users of
+      Common Test which want to abstract out behaviour which is common to
+      multiple test suites. </p>
 
     <p>In brief, Suite Callbacks allows you to:</p>
 
     <list>
-      <item>Manipulating the runtime config before each suite configuration calls</item>
-      <item>Manipulating the return of all suite configuration calls</item>
+      <item>Manipulate the runtime config before each suite 
+      configuration calls</item>
+      <item>Manipulate the return of all suite configuration calls and in 
+      extension the result of the test themselves.</item>
     </list>
     
-    <p>The following sections describe the mandatory and optional suite callback
-      functions Common Test will call during test execution. For more details
-      see <seealso marker="suite_callbacks">Suite Callbacks</seealso> in the User's Guide.</p>
+    <p>The following sections describe the mandatory and optional SCB
+    functions Common Test will call during test execution. For more details
+    see <seealso marker="suite_callbacks_chapter">Suite Callbacks</seealso> in 
+    the User's Guide.</p>
+
+    <p>For information about how to add a SCB to your suite see 
+    <seealso marker="suite_callbacks_chapter#installing">Installing an SCB
+    </seealso> in the User's Guide.</p>
+
+    <note><p>See the
+	<seealso marker="suite_callbacks_chapter#example">Example SCB</seealso>
+	in the User's Guide for a minimal example of an SCB. </p></note>
     
   </description>
 
@@ -62,31 +81,469 @@
   
   <funcs>
     <func>
-      <name>Module:init(Opts) -> {Id,State} </name>
-      <fsummary>asdas</fsummary>
+      <name>Module:init(Id, Opts) -&gt; State</name>
+      <fsummary>Initiates the Suite Callback</fsummary>
       <type>
+	<v>Id = reference() | term()</v>
 	<v>Opts = term()</v>
-	<v>Id = term()</v>
-	<v>GroupName = term()</v>
+	<v>State = term()</v>
       </type>
       
       <desc>	
 	<p> MANDATORY </p>
+
+	<p>Always called before any other callback function. 
+	  Use this to initiate any common state. 
+	  It should return a state for this SCB.</p>
+
+	<p><c>Id</c> is the return value of 
+	  <seealso marker="#Module:id-1">id/1</seealso>, or a <c>reference</c>
+	  if <seealso marker="#Module:id-1">id/1</seealso> is not implemented.
+	</p>
+
+	<p>For details about when init is called see
+	  <seealso marker="suite_callbacks_chapter#scope">scope</seealso>
+	  in the User's Guide.</p>
+	      
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_init_per_suite(SuiteName, Config, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {Return, NewSCBState}</v>
+	<v>Return = NewConfig | SkipOrFail</v>
+	<v>SkipOrFail = {fail, Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before 
+	<seealso marker="common_test#Module:init_per_suite-1">
+	  init_per_suite</seealso> if it exists. 
+	It typically contains initialization/logging which needs to be done 
+	before init_per_suite is called. 
+	If <c>{skip,Reason}</c> or <c>{fail,Reason}</c> is returned, 
+	init_per_suite and all test cases of the suite will be skipped and
+	Reason printed in the overview log of the suite.</p>
 	
-	<p></p>
+	<p><c>SuiteName</c> is the name of the suite to be run.</p>
+
+	<p><c>Config</c> is the original config list of the test suite.</p>
+
+	<p><c>SCBState</c> is the current internal state of the SCB.</p>
+
+	<p><c>Return</c> is the result of the init_per_suite function.
+	If it is <c>{skip,Reason}</c> or <c>{fail,Reason}</c> 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> will never be called, instead the initiation is considered 
+	to be skipped/failed respectively. If a <c>NewConfig</c> list
+	is returned, <seealso marker="common_test#Module:init_per_suite-1">
+	init_per_suite</seealso> will be called with that <c>NewConfig</c> list.
+	See <seealso marker="suite_callbacks_chapter#manipulating">
+	Manipulating tests</seealso> in the User's Guide for more details.</p>
+	
+	
+	<p>Note that this function is only called if the SCB has been added 
+	before init_per_suite is run, see 
+	<seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> 
+	in the User's Guide for details.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:post_init_per_suite(SuiteName, Config, Return, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called after init_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail, Reason} | {skip, Reason} | term()</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewReturn, NewSCBState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+	
+	<p><c>Return</c> is what 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> returned, i.e. {fail,Reason}, {skip,Reason}, a <c>Config</c>
+	list or a term describing how 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> failed.</p>
+
+	<p><c>NewReturn</c> is the possibly modified return value of
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso>. It is here possible to recover from a failure in 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> by returning the <c>ConfigList</c> with the <c>tc_status</c>
+	element removed.</p>
+
+	<p><c>SCBState</c> is the current internal state of the SCB.</p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:init_per_suite-1">
+	init_per_suite</seealso> if it exists.</p>
+
+	<p>Note that this function is only called if the SCB has been added 
+	before or in init_per_suite, see 
+	<seealso marker="suite_callbacks_chapter#scope">SCB Scoping</seealso> 
+	in the User's Guide for details.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:pre_init_per_group(GroupName, Config, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
 	
-	<p> If <c>{skip,Reason}</c> is returned, all test cases
-          in the module will be skipped, and the <c>Reason</c> will
-          be printed on the HTML result page.</p>
+	<p>This function is called before
+	<seealso marker="common_test#Module:init_per_group-2">
+	  init_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_group-2">
+	init_per_group</seealso> instead.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:post_init_per_group(GroupName, Config, Return, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called after init_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewReturn, NewSCBState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
 
-	<p>For details on groups, see 
-	  <seealso marker="write_test_chapter#test_case_groups">Test case 
-	  groups</seealso> in the User's Guide.</p>
+	<p>This function is called after
+	<seealso marker="common_test#Module:init_per_group-2">
+	  init_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_group-2">
+	init_per_group</seealso> instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_init_per_testcase(TestcaseName, Config, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_testcase</fsummary>
+      <type>
+	<v>TestcaseName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
       
-    </desc>
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before
+	<seealso marker="common_test#Module:init_per_testcase-2">
+	  init_per_testcase</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_testcase-2">
+	init_per_testcase</seealso> function instead.</p>
+
+	<p>Note that it is not possible to add SCB's here right now, 
+	that feature might be added later, 
+	but it would right now break backwards compatability.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_testcase(TestcaseName, Config, Return, SCBState)
+               -&gt; Result</name>
+      <fsummary>Called after end_per_testcase</fsummary>
+      <type>
+	<v>TestcaseName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewReturn, NewSCBState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_testcase-2">
+	  end_per_testcase</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_testcase-2">
+	end_per_testcase</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_end_per_group(GroupName, Config, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called before end_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+	
+	<p>This function is called before
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_group(GroupName, Config, Return, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called after end_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewReturn, NewSCBState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_group-2">
+	  end_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_end_per_suite(SuiteName, Config, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called before end_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewSCBState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before
+	<seealso marker="common_test#Module:end_per_suite-1">
+	  end_per_suite</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_suite-2">
+	end_per_suite</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_suite(SuiteName, Config, Return, SCBState) -&gt; 
+              Result</name>
+      <fsummary>Called after end_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>SCBState = NewSCBState = term()</v>
+	<v>Result = {NewReturn, NewSCBState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_suite-1">
+	  end_per_suite</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_suite_callback#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_suite-2">
+	end_per_suite</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:on_tc_fail(TestcaseName, Reason, SCBState) -&gt; 
+              NewSCBState</name>
+      <fsummary>Called after the SCB scope ends</fsummary>
+      <type>
+	<v>TestcaseName = init_per_suite | end_per_suite | 
+	                  init_per_group | end_per_group | atom()</v>
+	<v>Reason = term()</v>
+	<v>SCBState = NewSCBState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called whenever a testcase fails. 
+	It is called after the post function has been called for 
+	the testcase which failed. i.e. 
+	if init_per_suite fails this function is called after 
+	<seealso marker="#Module:post_init_per_suite-4">
+	  post_init_per_suite</seealso>, and if a testcase fails it is called 
+	  after <seealso marker="#Module:post_end_per_testcase-4">
+	  post_end_per_testcase</seealso>.</p>
+
+	<p>The data which comes with the Reason follows the same format as the
+	<seealso marker="event_handler_chapter#failreason">FailReason
+	</seealso> in the <seealso marker="event_handler_chapter#tc_done">tc_done</seealso> event.
+	See <seealso marker="event_handler_chapter#events">Event Handling
+	</seealso> in the User's Guide for details.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:on_tc_skip(TestcaseName, Reason, SCBState) -&gt; 
+              NewSCBState</name>
+      <fsummary>Called after the SCB scope ends</fsummary>
+      <type>
+	<v>TestcaseName = end_per_suite | init_per_group | 
+                          end_per_group | atom()</v>
+	<v>Reason = {tc_auto_skip | tc_user_skip, term()}</v>
+	<v>SCBState = NewSCBState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called whenever a testcase is skipped. 
+	It is called after the post function has been called for the 
+	testcase which was skipped. 
+	i.e. if init_per_group is skipped this function is called after 
+	<seealso marker="#Module:post_init_per_suite-4">post_init_per_group
+	</seealso>, and if a testcase is skipped it is called after 
+	<seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase
+	</seealso>.</p>
+
+	<p>The data which comes with the Reason follows the same format as 
+	<seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip
+	</seealso> and <seealso marker="event_handler_chapter#tc_user_skip">
+	tc_user_skip</seealso> events.
+	See <seealso marker="event_handler_chapter#events">Event Handling
+	</seealso> in the User's Guide for details.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:terminate(SCBState)</name>
+      <fsummary>Called after the SCB scope ends</fsummary>
+      <type>
+	<v>SCBState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called at the end of an SCB's 
+	<seealso marker="suite_callbacks_chapter#scope">scope</seealso>. 
+	</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:id(Opts) -&gt; Id</name>
+      <fsummary>Called before the init function of an SCB</fsummary>
+      <type>
+	<v>Opts = term()</v>
+	<v>Id = term()</v>
+      </type>
+      
+      <desc>
+	<p> OPTIONAL </p>
+	
+	<p>The <c>Id</c> is used to uniquely identify an SCB instance, 
+	if two SCB's return the same <c>Id</c> the second SCB is ignored
+	and subsequent calls to the SCB will only be made to the first
+	instance. For more information see 
+	<seealso marker="suite_callbacks_chapter#installing">Installing an SCB
+	</seealso> in the User's Guide.
+        </p>
+
+	<p>This function should NOT have any side effects as it might 
+	be called multiple times by Common Test.</p>
+	
+	<p>If not implemented the SCB will act as if this function returned a
+	  call to <c>make_ref/0</c>.</p>
+      </desc>
     </func>
 
-    
   </funcs>
 
 </erlref>