Add documentation for group specifications and for the label option

2 files changed, 93 insertions, 38 deletions

diff --git a/lib/common_test/doc/src/run_test.xml b/lib/common_test/doc/src/run_test.xml
index d609c4287f..2f0a94afba 100644
--- a/lib/common_test/doc/src/run_test.xml
+++ b/lib/common_test/doc/src/run_test.xml
@@ -45,6 +45,11 @@
     flags start an Erlang node prepared for running Common Test in a
     particular mode.</p>
 
+  <p>There is an interface function that corresponds to this program,
+    called <c>ct:run_test/1</c>, for starting Common Test from the Erlang
+    shell (or an Erlang program). Please see the <c>ct</c> man page for
+    details.</p>
+
   <p><c>run_test</c> also accepts Erlang emulator flags. These are used
     when <c>run_test</c> calls <c>erl</c> to start the Erlang node
     (making it possible to e.g. add directories to the code server path,
@@ -72,6 +77,8 @@
     <p>it prints all valid start flags to stdout.</p>
   </description>
 
+  <marker id="run_test"></marker>
+
   <section>
     <title>Run tests from command line</title>
     <pre>
@@ -83,6 +90,7 @@
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
 	 ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
+	[-label Label]
 	[-logdir LogDir]
 	[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
 	[-stylesheet CSSFile]
@@ -107,6 +115,7 @@
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
          ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
+	[-label Label]
 	[-logdir LogDir]
 	[-allow_user_terms]
 	[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
@@ -128,12 +137,12 @@
     <title>Run tests in web based GUI</title>
     <pre>
         run_test -vts [-browser Browser]
+        [-dir TestDir1 TestDir2 .. TestDirN] |
+        [-suite Suite [[-group Group] [-case Case]]]
 	[-config ConfigFile1 ConfigFile2 .. ConfigFileN]
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
          ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
-        [-dir TestDir1 TestDir2 .. TestDirN] |
-        [-suite Suite [[-group Group] [-case Case]]]
 	[-include InclDir1 InclDir2 .. InclDirN]
 	[-no_auto_compile]
 	[-muliply_timetraps Multiplier]
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index 207df7f5b5..1efff25f5b 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -21,7 +21,7 @@
 
     </legalnotice>
 
-    <title>Running Test Suites</title>
+    <title>Running Tests</title>
     <prepared>Peter Andersson, Kenneth Lundin</prepared>
     <docno></docno>
     <date></date>
@@ -130,6 +130,8 @@
     <p>Other flags that may be used with <c>run_test</c>:</p>
     <list>
       <item><c><![CDATA[-logdir <dir>]]></c>, specifies where the HTML log files are to be written.</item>
+      <item><c><![CDATA[-label <name_of_test_run>]]></c>, associates the test run with a name that gets printed
+	in the overview HTML log files.</item>
       <item><c>-refresh_logs</c>, refreshes the top level HTML index files.</item>
       <item><c>-vts</c>, start web based GUI (see below).</item>
       <item><c>-shell</c>, start interactive shell mode (see below).</item>
@@ -335,43 +337,72 @@
     <marker id="test_specifications"></marker>
     <title>Using test specifications</title>
     
-    <p>The most expressive way to specify what to test is to use a so 
-      called test specification. A test specification is a sequence of 
-      Erlang terms. The terms may be declared in a text file or passed 
-      to the test server at runtime as a list (see <c>run_testspec/1</c>
-      in the manual page for <c>ct</c>). There are two general types 
-      of terms: configuration terms and test specification terms.</p>
-    <p>With configuration terms it is possible to import configuration 
-      data (similar to <c>run_test -config</c>), specify HTML log 
-      directories (similar to <c>run_test -logdir</c>), give aliases 
-      to test nodes and test directories (to make a specification
-      easier to read and maintain), enable code coverage analysis
-      (see the <seealso marker="cover_chapter#cover">Code Coverage 
-       Analysis</seealso> chapter) and specify event_handler plugins 
+    <p>The most flexible way to specify what to test, is to use a so
+      called test specification. A test specification is a sequence of
+      Erlang terms. The terms may be declared in a text file or passed
+      to the test server at runtime as a list
+      (see <c>run_testspec/1</c> in the manual page
+      for <c>ct</c>). There are two general types of terms:
+      configuration terms and test specification terms.</p>
+    <p>With configuration terms it is possible to e.g. label the test
+      run (similar to <c>run_test -label</c>), evaluate arbitrary expressions
+      before starting a test, import configuration
+      data (similar to
+      <c>run_test -config/-userconfig</c>), specify HTML log directories (similar
+      to
+      <c>run_test -logdir</c>), give aliases to test nodes and test
+      directories (to make a specification easier to read and
+      maintain), enable code coverage analysis (see
+      the <seealso marker="cover_chapter#cover">Code Coverage
+      Analysis</seealso> chapter) and specify event_handler plugins
       (see the <seealso marker="event_handler_chapter#event_handling">
-       Event Handling</seealso> chapter). There is also a term
-       for specifying include directories that should be passed on
-       to the compiler when automatic compilation is performed 
-       (similar to <c>run_test -include</c>, see above).</p>
-    <p>With test specification terms it is possible to state exactly which
-      tests should run and in which order. A test term specifies either
-      one or more suites or one or more test cases. An arbitrary number of test 
-      terms may be declared in sequence. A test term can also specify one or 
-      more test suites or test cases to be skipped. Skipped suites and cases 
-      are not executed and show up in the HTML test log as SKIPPED.</p>
-
-    <note><p>It is not yet possible to specify test case groups in
-      test specifications. This will be supported in a soon upcoming 
-      release.</p></note>
+      Event Handling</seealso> chapter). There is also a term for
+      specifying include directories that should be passed on to the
+      compiler when automatic compilation is performed (similar
+      to <c>run_test -include</c>, see above).</p>
+    <p>With test specification terms it is possible to state exactly
+      which tests should run and in which order. A test term specifies
+      either one or more suites, one or more test case groups, or one
+      or more test cases in a group or suite.</p>
+    <p>An arbitrary number of test terms may be declared in sequence.
+      Common Test will compile the terms into one or more tests to be
+      performed in one resulting test run. Note that a term that
+      specifies a set of test cases will "swallow" one that only
+      specifies a subset of these cases. E.g. the result of merging
+      one term that specifies that all cases in suite S should be
+      executed, with another term specifying only test case X and Y in
+      S, is a test of all cases in S. However, if a term specifying
+      test case X and Y in S is merged with a term specifying case Z
+      in S, the result is a test of X, Y and Z in S.</p>
+    <p>A test term can also specify one or more test suites, groups,
+      or test cases to be skipped. Skipped suites, groups and cases
+      are not executed and show up in the HTML test log files as
+      SKIPPED.</p>
+    <p>When a test case group is specified, the resulting test
+      executes the
+      <c>init_per_group</c> function, followed by all test cases and
+      sub groups (including their configuration functions), and
+      finally the <c>end_per_group</c> function. Also if particular
+      test cases in a group are specified, <c>init_per_group</c>
+      and <c>end_per_group</c> for the group in question are
+      called. If a group which is defined (in <c>Suite:group/0</c>) to
+      be a sub group of another group, is specified (or particular test
+      cases of a sub group are), Common Test will call the configuration
+      functions for the top level groups as well as for the sub group
+      in question (making it possible to pass configuration data all
+      the way from <c>init_per_suite</c> down to the test cases in the
+      sub group).</p>
 
     <p>Below is the test specification syntax. Test specifications can
-      be used to run tests both in a single test host environment and in 
-      a distributed Common Test environment (Large Scale Testing). The init term,
-      as well as node parameters, are only relevant in the latter (see the
-      <seealso marker="ct_master_chapter#test_specifications">Large Scale Testing</seealso>
-      chapter for information). For details on the event_handler term, see the
-      <seealso marker="event_handler_chapter#event_handling">Event Handling</seealso>
-      chapter.</p>
+      be used to run tests both in a single test host environment and
+      in a distributed Common Test environment (Large Scale
+      Testing). The node parameters in the init term are only
+      relevant in the latter (see the
+      <seealso marker="ct_master_chapter#test_specifications">Large
+      Scale Testing</seealso> chapter for information). For details on
+      the event_handler term, see the
+      <seealso marker="event_handler_chapter#event_handling">Event
+      Handling</seealso> chapter.</p>
       <p>Config terms:</p>
     <pre>
       {node, NodeAlias, Node}.
@@ -379,11 +410,17 @@
       {init, InitOptions}.
       {init, [NodeAlias], InitOptions}.
 
+      {label, Label}.
+      {label, NodeRefs, Label}.
+
       {multiply_timetraps, N}.
+      {multiply_timetraps, NodeRefs, N}.
+
       {scale_timetraps, Bool}.
+      {scale_timetraps, NodeRefs, Bool}.
  
       {cover, CoverSpecFile}.
-      {cover, NodeRef, CoverSpecFile}.
+      {cover, NodeRefs, CoverSpecFile}.
       
       {include, IncludeDirs}.
       {include, NodeRefs, IncludeDirs}.
@@ -409,6 +446,12 @@
       {suites, DirRef, Suites}.                                
       {suites, NodeRefs, DirRef, Suites}.
       
+      {groups, DirRef, Suite, Groups}.
+      {groups, NodeRefsDirRef, Suite, Groups}.
+
+      {groups, DirRef, Suite, Group, {cases,Cases}}.
+      {groups, NodeRefsDirRef, Suite, Group, {cases,Cases}}.
+
       {cases, DirRef, Suite, Cases}.                           
       {cases, NodeRefs, DirRef, Suite, Cases}.
 
@@ -437,6 +480,9 @@
       InitArgs      = [term()]
       DirRef        = DirAlias | Dir
       Suites        = atom() | [atom()] | all
+      Suite         = atom()
+      Groups        = atom() | [atom()] | all
+      Group         = atom()
       Cases         = atom() | [atom()] | all
       Comment       = string() | ""
     </pre>