Merge branch 'lukas/common_test/suite_callback/OTP-8851' into dev

* lukas/common_test/suite_callback/OTP-8851: (54 commits)
  Update example cth spec to reflect the implementation
  Cleanup code to fix dialyzer warning
  Rename Suite Callback to Common Test hook in documentation
  Rename Suite Callback to Common Test Hook in code and testcases
  Fix bug where the state of an SCB was altered when no on_tc_* existed
  Update SCBs to use a specific id/1 function for SCb overriding instead of returning it from init/1.
  Add documentation for SCBs
  Update suite callback test timeout so that beam debug test runs do not timeout
  Update test support so that if common test fails to be stopped in a suite, the ct node is restarted and the test case fails
  Add tests for SCB's with same id and fixed bug relating to it
  Add test suites for failing in init/1 function
  Added tests for starting SCB's with arguments and fixed a bug with how SCB's are parsed from the command line
  Add so that failures in SCB:init/1 causes the entire scb scope to fail
  Fix bug which caused scb to be added too much when initiated in the suite info function
  Started work on documenting suite callbacks, this is a partial commit
  Add locking mechanism for scb state when using parallel groups
  Add state update tests for suite callbacks
  Update ct_framework calls to allow manipulation of test results in end_tc without breaking backwards compatability (I hope)
  Add test case for init_per_suite recover scenario
  Add testcase for config updates
  ...

1 files changed, 556 insertions, 0 deletions

diff --git a/lib/common_test/doc/src/ct_hooks.xml b/lib/common_test/doc/src/ct_hooks.xml
new file mode 100644
index 0000000000..0d59ce3b22
--- /dev/null
+++ b/lib/common_test/doc/src/ct_hooks.xml
@@ -0,0 +1,556 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+
+<!DOCTYPE erlref SYSTEM "erlref.dtd">
+
+<erlref>
+  <header>
+    <copyright>
+      <year>2010</year><year>2010</year>
+      <holder>Ericsson AB. All Rights Reserved.</holder>
+    </copyright>
+    <legalnotice>
+      The contents of this file are subject to the Erlang Public License,
+      Version 1.1, (the "License"); you may not use this file except in
+      compliance with the License. You should have received a copy of the
+      Erlang Public License along with this software. If not, it can be
+      retrieved online at http://www.erlang.org/.
+
+      Software distributed under the License is distributed on an "AS IS"
+      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+      the License for the specific language governing rights and limitations
+      under the License.
+
+    </legalnotice>
+
+    <title>Common Test Hooks</title>
+    <prepared>Lukas Larsson</prepared>
+    <responsible>Lukas Larsson</responsible>
+    <docno></docno>
+    <approved></approved>
+    <checked></checked>
+    <date>2010-12-02</date>
+    <rev>PA1</rev>
+    <file>ct_hooks.sgml</file>
+  </header>
+  <module>ct_hooks</module> 
+  <modulesummary>A callback interface on top of Common Test</modulesummary>
+
+  <description>
+
+    <warning><p>This feature is in alpha release right now. This means that the 
+	interface may change in the future and that there may be bugs. We 
+	encourage you to use this feature, but be prepared 
+	that there might be bugs and that the interface might change
+	inbetween releases.</p></warning>
+
+    <p>The <em>Common Test Hook</em> (henceforth called CTH) framework allows 
+      extensions of the default behaviour of Common Test by means of callbacks 
+      before and after all test suite calls. It is meant for advanced users of
+      Common Test which want to abstract out behaviour which is common to
+      multiple test suites. </p>
+
+    <p>In brief, Common Test Hooks allows you to:</p>
+
+    <list>
+      <item>Manipulate the runtime config before each suite 
+      configuration call</item>
+      <item>Manipulate the return of all suite configuration calls and in 
+      extension the result of the test themselves.</item>
+    </list>
+    
+    <p>The following sections describe the mandatory and optional CTH
+    functions Common Test will call during test execution. For more details
+    see <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> in 
+    the User's Guide.</p>
+
+    <p>For information about how to add a CTH to your suite see 
+    <seealso marker="ct_hooks_chapter#installing">Installing a CTH
+    </seealso> in the User's Guide.</p>
+
+    <note><p>See the
+	<seealso marker="ct_hooks_chapter#example">Example CTH</seealso>
+	in the User's Guide for a minimal example of a CTH. </p></note>
+    
+  </description>
+
+  <section>
+    <title>CALLBACK FUNCTIONS</title>
+    <p>The following functions define the callback interface
+      for a Common Test Hook.</p>
+  </section>
+  
+  <funcs>
+    <func>
+      <name>Module:init(Id, Opts) -&gt; State</name>
+      <fsummary>Initiates the Common Test Hook</fsummary>
+      <type>
+	<v>Id = reference() | term()</v>
+	<v>Opts = term()</v>
+	<v>State = term()</v>
+      </type>
+      
+      <desc>	
+	<p> MANDATORY </p>
+
+	<p>Always called before any other callback function. 
+	  Use this to initiate any common state. 
+	  It should return a state for this CTH.</p>
+
+	<p><c>Id</c> is the return value of 
+	  <seealso marker="#Module:id-1">id/1</seealso>, or a <c>reference</c>
+	  (created using 
+	  <seealso marker="erts:erlang#make_ref-0">make_ref/0</seealso>)
+	  if <seealso marker="#Module:id-1">id/1</seealso> is not implemented.
+	</p>
+
+	<p>For details about when init is called see
+	  <seealso marker="ct_hooks_chapter#scope">scope</seealso>
+	  in the User's Guide.</p>
+	      
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_init_per_suite(SuiteName, Config, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {Return, NewCTHState}</v>
+	<v>Return = NewConfig | SkipOrFail</v>
+	<v>SkipOrFail = {fail, Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before 
+	<seealso marker="common_test#Module:init_per_suite-1">
+	  init_per_suite</seealso> if it exists. 
+	It typically contains initialization/logging which needs to be done 
+	before init_per_suite is called. 
+	If <c>{skip,Reason}</c> or <c>{fail,Reason}</c> is returned, 
+	init_per_suite and all test cases of the suite will be skipped and
+	Reason printed in the overview log of the suite.</p>
+	
+	<p><c>SuiteName</c> is the name of the suite to be run.</p>
+
+	<p><c>Config</c> is the original config list of the test suite.</p>
+
+	<p><c>CTHState</c> is the current internal state of the CTH.</p>
+
+	<p><c>Return</c> is the result of the init_per_suite function.
+	If it is <c>{skip,Reason}</c> or <c>{fail,Reason}</c> 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> will never be called, instead the initiation is considered 
+	to be skipped/failed respectively. If a <c>NewConfig</c> list
+	is returned, <seealso marker="common_test#Module:init_per_suite-1">
+	init_per_suite</seealso> will be called with that <c>NewConfig</c> list.
+	See <seealso marker="ct_hooks_chapter#pre">
+	Pre Hooks</seealso> in the User's Guide for more details.</p>
+	
+	
+	<p>Note that this function is only called if the CTH has been added 
+	before init_per_suite is run, see 
+	<seealso marker="ct_hooks_chapter#scope">CTH Scoping</seealso> 
+	in the User's Guide for details.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:post_init_per_suite(SuiteName, Config, Return, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called after init_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail, Reason} | {skip, Reason} | term()</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewReturn, NewCTHState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:init_per_suite-1">
+	  init_per_suite</seealso> if it exists. It typically contains extra 
+	checks to make sure that all the correct dependencies have
+	been started correctly.</p>
+	
+	<p><c>Return</c> is what 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> returned, i.e. {fail,Reason}, {skip,Reason}, a <c>Config</c>
+	list or a term describing how 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> failed.</p>
+
+	<p><c>NewReturn</c> is the possibly modified return value of
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso>. It is here possible to recover from a failure in 
+	<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
+	</seealso> by returning the <c>ConfigList</c> with the <c>tc_status</c>
+	element removed. See <seealso marker="ct_hooks_chapter#post">
+	Post Hooks</seealso> in the User's Guide for more details.</p>
+
+	<p><c>CTHState</c> is the current internal state of the CTH.</p>
+
+	<p>Note that this function is only called if the CTH has been added 
+	before or in init_per_suite, see 
+	<seealso marker="ct_hooks_chapter#scope">CTH Scoping</seealso> 
+	in the User's Guide for details.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:pre_init_per_group(GroupName, Config, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+	
+	<p>This function is called before
+	<seealso marker="common_test#Module:init_per_group-2">
+	  init_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_group-2">
+	init_per_group</seealso> instead.</p>
+      </desc>
+    </func>
+    
+    <func>
+      <name>Module:post_init_per_group(GroupName, Config, Return, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called after init_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewReturn, NewCTHState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:init_per_group-2">
+	  init_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_group-2">
+	init_per_group</seealso> instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_init_per_testcase(TestcaseName, Config, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called before init_per_testcase</fsummary>
+      <type>
+	<v>TestcaseName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before
+	<seealso marker="common_test#Module:init_per_testcase-2">
+	  init_per_testcase</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:init_per_testcase-2">
+	init_per_testcase</seealso> function instead.</p>
+
+	<p>Note that it is not possible to add CTH's here right now, 
+	that feature might be added later, 
+	but it would right now break backwards compatability.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_testcase(TestcaseName, Config, Return, CTHState)
+               -&gt; Result</name>
+      <fsummary>Called after end_per_testcase</fsummary>
+      <type>
+	<v>TestcaseName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewReturn, NewCTHState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_testcase-2">
+	  end_per_testcase</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_testcase-2">
+	end_per_testcase</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_end_per_group(GroupName, Config, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called before end_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+	
+	<p>This function is called before
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_group(GroupName, Config, Return, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called after end_per_group</fsummary>
+      <type>
+	<v>GroupName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewReturn, NewCTHState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_group-2">
+	  end_per_group</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_group-2">
+	end_per_group</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:pre_end_per_suite(SuiteName, Config, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called before end_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = NewConfig = [{Key,Value}]</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called before
+	<seealso marker="common_test#Module:end_per_suite-1">
+	  end_per_suite</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
+	pre_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_suite-2">
+	end_per_suite</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:post_end_per_suite(SuiteName, Config, Return, CTHState) -&gt; 
+              Result</name>
+      <fsummary>Called after end_per_suite</fsummary>
+      <type>
+	<v>SuiteName = atom()</v>
+	<v>Config = [{Key,Value}]</v>
+	<v>Return = NewReturn = Config | SkipOrFail | term()</v>
+	<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
+	<v>CTHState = NewCTHState = term()</v>
+	<v>Result = {NewReturn, NewCTHState}</v>
+	<v>Key = atom()</v>
+	<v>Value = term()</v>
+	<v>Reason = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called after
+	<seealso marker="common_test#Module:end_per_suite-1">
+	  end_per_suite</seealso> if it exists. It behaves the same way as 
+	<seealso marker="ct_hooks#Module:post_init_per_suite-4">
+	post_init_per_suite</seealso>, but for the
+	<seealso marker="common_test#Module:end_per_suite-2">
+	end_per_suite</seealso> function instead.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:on_tc_fail(TestcaseName, Reason, CTHState) -&gt; 
+              NewCTHState</name>
+      <fsummary>Called after the CTH scope ends</fsummary>
+      <type>
+	<v>TestcaseName = init_per_suite | end_per_suite | 
+	                  init_per_group | end_per_group | atom()</v>
+	<v>Reason = term()</v>
+	<v>CTHState = NewCTHState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called whenever a testcase fails. 
+	It is called after the post function has been called for 
+	the testcase which failed. i.e. 
+	if init_per_suite fails this function is called after 
+	<seealso marker="#Module:post_init_per_suite-4">
+	  post_init_per_suite</seealso>, and if a testcase fails it is called 
+	  after <seealso marker="#Module:post_end_per_testcase-4">
+	  post_end_per_testcase</seealso>.</p>
+
+	<p>The data which comes with the Reason follows the same format as the
+	<seealso marker="event_handler_chapter#failreason">FailReason
+	</seealso> in the <seealso marker="event_handler_chapter#tc_done">tc_done</seealso> event.
+	See <seealso marker="event_handler_chapter#events">Event Handling
+	</seealso> in the User's Guide for details.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:on_tc_skip(TestcaseName, Reason, CTHState) -&gt; 
+              NewCTHState</name>
+      <fsummary>Called after the CTH scope ends</fsummary>
+      <type>
+	<v>TestcaseName = end_per_suite | init_per_group | 
+                          end_per_group | atom()</v>
+	<v>Reason = {tc_auto_skip | tc_user_skip, term()}</v>
+	<v>CTHState = NewCTHState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called whenever a testcase is skipped. 
+	It is called after the post function has been called for the 
+	testcase which was skipped. 
+	i.e. if init_per_group is skipped this function is called after 
+	<seealso marker="#Module:post_init_per_suite-4">post_init_per_group
+	</seealso>, and if a testcase is skipped it is called after 
+	<seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase
+	</seealso>.</p>
+
+	<p>The data which comes with the Reason follows the same format as 
+	<seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip
+	</seealso> and <seealso marker="event_handler_chapter#tc_user_skip">
+	tc_user_skip</seealso> events.
+	See <seealso marker="event_handler_chapter#events">Event Handling
+	</seealso> in the User's Guide for details.</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:terminate(CTHState)</name>
+      <fsummary>Called after the CTH scope ends</fsummary>
+      <type>
+	<v>CTHState = term()</v>
+      </type>
+      
+      <desc>	
+	<p> OPTIONAL </p>
+
+	<p>This function is called at the end of a CTH's 
+	<seealso marker="ct_hooks_chapter#scope">scope</seealso>. 
+	</p>
+      </desc>
+    </func>
+
+    <func>
+      <name>Module:id(Opts) -&gt; Id</name>
+      <fsummary>Called before the init function of a CTH</fsummary>
+      <type>
+	<v>Opts = term()</v>
+	<v>Id = term()</v>
+      </type>
+      
+      <desc>
+	<p> OPTIONAL </p>
+	
+	<p>The <c>Id</c> is used to uniquely identify a CTH instance, 
+	if two CTH's return the same <c>Id</c> the second CTH is ignored
+	and subsequent calls to the CTH will only be made to the first
+	instance. For more information see 
+	<seealso marker="ct_hooks_chapter#installing">Installing a CTH
+	</seealso> in the User's Guide.
+        </p>
+
+	<p>This function should NOT have any side effects as it might 
+	be called multiple times by Common Test.</p>
+	
+	<p>If not implemented the CTH will act as if this function returned a
+	  call to <c>make_ref/0</c>.</p>
+      </desc>
+    </func>
+
+  </funcs>
+
+</erlref>
+
+