Merge branch 'peppe/common_test_r14_dev_2' into dev

* peppe/common_test_r14_dev_2:
  Misc documentation updates
  Add documentation for run_test program
  Step vsn for test_server to 3.4
  Make {repeat*,N} property in group execute the group N times exactly
  Fix so that ct_run converts relative diretories in the code path to absolute
  Fix bug in handling framework:end_tc timeouts
  Fix bug that prevents the interactive shell mode to start properly
  Fix failing multiply timetrap test case
  Minor fixes in code and test suites
  Add support for executing pre-loaded suites (e.g. modules loaded as binaries)
  Add test suite for remote loading of binary suites
  Fix error with {repeat,0} property in groups causing double iterations
  Add support for config info functions (e.g. init_per_suite/0)
  Add support for dynamic timetrap handling
  Have end_per_testcase run even after timetrap_timeout and abort_testcase
  Flush old DOWN messages in demonitor
  Add groups in test specifications
  Add new tests for test case groups and test specifications
  Improve and fix various test suites
  Add event_handler_init start flag that can pass init arguments to event handlers
  ...

OTP-8703 peppe/common_test_r14_dev_2

1 files changed, 56 insertions, 22 deletions

diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index d731d18783..207df7f5b5 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -4,7 +4,7 @@
 <chapter>
   <header>
     <copyright>
-      <year>2003</year><year>2009</year>
+      <year>2003</year><year>2010</year>
       <holder>Ericsson AB. All Rights Reserved.</holder>
     </copyright>
     <legalnotice>
@@ -13,12 +13,12 @@
       compliance with the License. You should have received a copy of the
       Erlang Public License along with this software. If not, it can be
       retrieved online at http://www.erlang.org/.
-    
+
       Software distributed under the License is distributed on an "AS IS"
       basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
       the License for the specific language governing rights and limitations
       under the License.
-    
+
     </legalnotice>
 
     <title>Running Test Suites</title>
@@ -97,25 +97,32 @@
       the <c><![CDATA[{auto_compile,false}]]></c> option with 
       <c><![CDATA[ct:run_test/1]]></c>. With automatic compilation
       disabled, the user is responsible for compiling the test suite modules 
-      (and any help modules) before the test run. Common Test will only verify 
-      that the specified test suites exist before starting the tests.</p> 
+      (and any help modules) before the test run. If the modules can not be loaded
+      from the local file system during startup of Common Test, the user needs to
+      pre-load the modules before starting the test. Common Test will only verify
+      that the specified test suites exist (i.e. that they are, or can be, loaded).
+      This is useful e.g. if the test suites are transferred and loaded as binaries via
+      RPC from a remote node.</p>
   </section>
 
   <section>
-    <title>Running tests from the UNIX command line</title>
+    <title>Running tests from the OS command line</title>
     
-    <p>The script <c>run_test</c> can be used for running tests from
-      the Unix/Linux command line, e.g.
+    <p>The <c>run_test</c> program can be used for running tests from
+      the OS command line, e.g.
     </p>
     <list>
       <item><c><![CDATA[run_test -config <configfilenames> -dir <dirs>]]></c></item>
       <item><c><![CDATA[run_test -config <configfilenames> -suite <suiteswithfullpath>]]></c>
       </item>
+      <item><c><![CDATA[run_test -userconfig <callbackmodulename> <configfilenames> -suite <suiteswithfullpath>]]></c>
+      </item>
       <item><c><![CDATA[run_test -config <configfilenames> -suite <suitewithfullpath>
 	      -group <groupnames> -case <casenames>]]></c></item>
     </list>
     <p>Examples:</p>
     <p><c>$ run_test -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST</c></p>
+    <p><c>$ run_test -userconfig ct_config_xml $CFGS/sys1.xml $CFGS/sys2.xml -dir $SYS1_TEST $SYS2_TEST</c></p>
     <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE $SYS2_TEST/config_SUITE</c></p>
     <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE -case start stop</c></p>
     <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE -group installation -case start stop</c></p>
@@ -136,8 +143,14 @@
 	<seealso marker="cover_chapter#cover">Code Coverage Analysis</seealso>).</item>
       <item><c><![CDATA[-event_handler <event_handlers>]]></c>, to install 
         <seealso marker="event_handler_chapter#event_handling">event handlers</seealso>.</item>
+      <item><c><![CDATA[-event_handler_init <event_handlers>]]></c>, to install
+        <seealso marker="event_handler_chapter#event_handling">event handlers</seealso> including start arguments.</item>
       <item><c><![CDATA[-include]]></c>, specifies include directories (see above).</item>
       <item><c><![CDATA[-no_auto_compile]]></c>, disables the automatic test suite compilation feature (see above).</item>
+      <item><c><![CDATA[-multiply_timetraps <n>]]></c>, extends <seealso marker="write_test_chapter#timetraps">timetrap
+	  timeout</seealso> values.</item>
+      <item><c><![CDATA[-scale_timetraps <bool>]]></c>, enables automatic <seealso marker="write_test_chapter#timetraps">timetrap
+	  timeout</seealso> scaling.</item>
       <item><c><![CDATA[-repeat <n>]]></c>, tells Common Test to repeat the tests n times (see below).</item> 
       <item><c><![CDATA[-duration <time>]]></c>, tells Common Test to repeat the tests for duration of time (see below).</item> 
       <item><c><![CDATA[-until <stop_time>]]></c>, tells Common Test to repeat the tests until stop_time (see below).</item> 
@@ -165,7 +178,7 @@
 	     the current working directory of the Erlang Runtime System during the test run!</p>
     </note>
     
-    <p>For details on how to generate the <c>run_test</c> script, see the 
+    <p>For more information about the <c>run_test</c> program, see the
       <seealso marker="install_chapter#general">Installation</seealso> chapter.
     </p>
   </section>
@@ -174,7 +187,7 @@
     <title>Running tests from the Web based GUI</title>
     
     <p>The web based GUI, VTS, is started with the <c>run_test</c>
-      script. From the GUI you can load config files, and select
+      program. From the GUI you can load config files, and select
       directories, suites and cases to run. You can also state the
       config files, directories, suites and cases on the command line
       when starting the web based GUI.
@@ -198,10 +211,11 @@
       <p>Example:</p>
       <p><c><![CDATA[$ run_test -vts -browser 'firefox&']]></c></p>
       <p>Note that the browser must run as a separate OS process or VTS will hang!</p>
-      <p>If no specific browser start command is specified, netscape will
+      <p>If no specific browser start command is specified, Firefox will
         be the default browser on Unix platforms and Internet Explorer on Windows.
-	If Common Test fails to start a browser automatically, start your 
-	favourite browser manually instead and type in the URL that Common Test
+	If Common Test fails to start a browser automatically, or <c>'none'</c> is
+	specified as the value for -browser (i.e. <c>-browser none</c>), start your
+	favourite browser manually and type in the URL that Common Test
 	displays in the shell.</p>    
   </section>
   
@@ -211,7 +225,7 @@
     <p>Common Test provides an Erlang API for running tests. The main (and most
       flexible) function for specifying and executing tests is called
       <c>ct:run_test/1</c>. This function takes the same start parameters as
-      the <c>run_test</c> script described above, only the flags are instead
+      the <c>run_test</c> program described above, only the flags are instead
       given as options in a list of key-value tuples. E.g. a test specified 
       with <c>run_test</c> like:</p>
       <p><c>$ run_test -suite ./my_SUITE -logdir ./results</c></p> 
@@ -237,13 +251,14 @@
       manually and call <c>ct:install/1</c> to install any configuration
       data you might need (use <c>[]</c> as argument otherwise), then
       call <c>ct:start_interactive/0</c> to start Common Test. If you use
-      the <c>run_test</c> script, you may start the Erlang shell and Common Test
+      the <c>run_test</c> program, you may start the Erlang shell and Common Test
       in the same go by using the <c>-shell</c> and, optionally, the <c>-config</c>
-      flag:
+      and/or <c>-userconfig</c> flag. Examples:
     </p>
     <list>
       <item><c>run_test -shell</c></item>
-      <item><c><![CDATA[run_test -shell -config <configfilename>]]></c></item>
+      <item><c><![CDATA[run_test -shell -config cfg/db.cfg]]></c></item>
+      <item><c><![CDATA[run_test -shell -userconfig db_login testuser x523qZ]]></c></item>
     </list>
     
     <p>If no config file is given with the <c>run_test</c> command,
@@ -268,7 +283,8 @@
        2> ct_telnet:open(unix_telnet).
        {ok,&lt;0.105.0&gt;}
        4> ct_telnet:cmd(unix_telnet, "ls .").
-       {ok,["ls .","file1  ...",...]}</pre>
+       {ok,["ls .","file1  ...",...]}
+    </pre>
     
     <p>Everything that Common Test normally prints in the test case logs,
       will in the interactive mode be written to a log named
@@ -350,14 +366,21 @@
 
     <p>Below is the test specification syntax. Test specifications can
       be used to run tests both in a single test host environment and in 
-      a distributed Common Test environment. Node parameters are only relevant in the 
-      latter (see the chapter about running Common Test in distributed mode for information). 
-      For details on the event_handler term, see the 
+      a distributed Common Test environment (Large Scale Testing). The init term,
+      as well as node parameters, are only relevant in the latter (see the
+      <seealso marker="ct_master_chapter#test_specifications">Large Scale Testing</seealso>
+      chapter for information). For details on the event_handler term, see the
       <seealso marker="event_handler_chapter#event_handling">Event Handling</seealso>
       chapter.</p>
       <p>Config terms:</p>
     <pre>
       {node, NodeAlias, Node}.
+
+      {init, InitOptions}.
+      {init, [NodeAlias], InitOptions}.
+
+      {multiply_timetraps, N}.
+      {scale_timetraps, Bool}.
  
       {cover, CoverSpecFile}.
       {cover, NodeRef, CoverSpecFile}.
@@ -367,6 +390,9 @@
 
       {config, ConfigFiles}.
       {config, NodeRefs, ConfigFiles}.
+
+      {userconfig, {CallbackModule, ConfigStrings}}.
+      {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}.
       
       {alias, DirAlias, Dir}.
       
@@ -395,9 +421,12 @@
       <p>Types:</p>
     <pre>
       NodeAlias     = atom()
+      InitOptions   = term()
       Node          = node()
       NodeRef       = NodeAlias | Node | master
       NodeRefs      = all_nodes | [NodeRef] | NodeRef
+      N             = integer()
+      Bool          = true | false
       CoverSpecFile = string()
       IncludeDirs   = string() | [string()]
       ConfigFiles   = string() | [string()]
@@ -449,9 +478,14 @@
       <item>Secondly, the test for system t2 should run. The included suites are
 	t2B and t2C. Included are also test cases test4, test1 and test7 in suite
 	t2A. Note that the test cases will be executed in the specified order.</item>
-      <item>Lastly, all suites for systems t3 are to be completely skipped and this 
+      <item>Lastly, all suites for systems t3 are to be completely skipped and this
 	should be explicitly noted in the log files.</item>
     </list>
+    <p>It is possible to specify initialization options for nodes defined in the
+      test specification. Currently, there are options to start the node and/or to
+      evaluate any function on the node.
+      See the <seealso marker="ct_master_chapter#ct_slave">Automatic startup of
+      the test target nodes</seealso> chapter for details.</p>
     <p>It is possible for the user to provide a test specification that
       includes (for Common Test) unrecognizable terms. If this is desired,
       the <c>-allow_user_terms</c> flag should be used when starting tests with