Merge branch 'peppe/common_test/dev/OTP-8725' of super:peppe_otp into dev

* 'peppe/common_test/dev/OTP-8725' of super:peppe_otp:
  Fix bug that crashes common_test when running cover
  Bump version numbers for common_test and test_server
  Add documentation for group specifications and for the label option
  Improve handling of test case group specifications
  Add textured backgound to html log files
  Improve handling of test case group specifications
  Add new option to label test runs
  Implement handling of group specifications
  Fix problem with using the wrong logdir setting
  Rewrite handling of group definitions in Common Test test suites

2 files changed, 93 insertions, 38 deletions

diff --git a/lib/common_test/doc/src/run_test.xml b/lib/common_test/doc/src/run_test.xml
index d609c4287f..2f0a94afba 100644
--- a/lib/common_test/doc/src/run_test.xml
+++ b/lib/common_test/doc/src/run_test.xml
@@ -45,6 +45,11 @@
     flags start an Erlang node prepared for running Common Test in a
     particular mode.</p>
 
+  <p>There is an interface function that corresponds to this program,
+    called <c>ct:run_test/1</c>, for starting Common Test from the Erlang
+    shell (or an Erlang program). Please see the <c>ct</c> man page for
+    details.</p>
+
   <p><c>run_test</c> also accepts Erlang emulator flags. These are used
     when <c>run_test</c> calls <c>erl</c> to start the Erlang node
     (making it possible to e.g. add directories to the code server path,
@@ -72,6 +77,8 @@
     <p>it prints all valid start flags to stdout.</p>
   </description>
 
+  <marker id="run_test"></marker>
+
   <section>
     <title>Run tests from command line</title>
     <pre>
@@ -83,6 +90,7 @@
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
 	 ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
+	[-label Label]
 	[-logdir LogDir]
 	[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
 	[-stylesheet CSSFile]
@@ -107,6 +115,7 @@
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
          ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
+	[-label Label]
 	[-logdir LogDir]
 	[-allow_user_terms]
 	[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
@@ -128,12 +137,12 @@
     <title>Run tests in web based GUI</title>
     <pre>
         run_test -vts [-browser Browser]
+        [-dir TestDir1 TestDir2 .. TestDirN] |
+        [-suite Suite [[-group Group] [-case Case]]]
 	[-config ConfigFile1 ConfigFile2 .. ConfigFileN]
 	[-userconfig CallbackModule1 ConfigString1 and CallbackModule2
          ConfigString2 and .. and CallbackModuleN ConfigStringN]
 	[-decrypt_key Key] | [-decrypt_file KeyFile]
-        [-dir TestDir1 TestDir2 .. TestDirN] |
-        [-suite Suite [[-group Group] [-case Case]]]
 	[-include InclDir1 InclDir2 .. InclDirN]
 	[-no_auto_compile]
 	[-muliply_timetraps Multiplier]
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index 207df7f5b5..1efff25f5b 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -21,7 +21,7 @@
 
     </legalnotice>
 
-    <title>Running Test Suites</title>
+    <title>Running Tests</title>
     <prepared>Peter Andersson, Kenneth Lundin</prepared>
     <docno></docno>
     <date></date>
@@ -130,6 +130,8 @@
     <p>Other flags that may be used with <c>run_test</c>:</p>
     <list>
       <item><c><![CDATA[-logdir <dir>]]></c>, specifies where the HTML log files are to be written.</item>
+      <item><c><![CDATA[-label <name_of_test_run>]]></c>, associates the test run with a name that gets printed
+	in the overview HTML log files.</item>
       <item><c>-refresh_logs</c>, refreshes the top level HTML index files.</item>
       <item><c>-vts</c>, start web based GUI (see below).</item>
       <item><c>-shell</c>, start interactive shell mode (see below).</item>
@@ -335,43 +337,72 @@
     <marker id="test_specifications"></marker>
     <title>Using test specifications</title>
     
-    <p>The most expressive way to specify what to test is to use a so 
-      called test specification. A test specification is a sequence of 
-      Erlang terms. The terms may be declared in a text file or passed 
-      to the test server at runtime as a list (see <c>run_testspec/1</c>
-      in the manual page for <c>ct</c>). There are two general types 
-      of terms: configuration terms and test specification terms.</p>
-    <p>With configuration terms it is possible to import configuration 
-      data (similar to <c>run_test -config</c>), specify HTML log 
-      directories (similar to <c>run_test -logdir</c>), give aliases 
-      to test nodes and test directories (to make a specification
-      easier to read and maintain), enable code coverage analysis
-      (see the <seealso marker="cover_chapter#cover">Code Coverage 
-       Analysis</seealso> chapter) and specify event_handler plugins 
+    <p>The most flexible way to specify what to test, is to use a so
+      called test specification. A test specification is a sequence of
+      Erlang terms. The terms may be declared in a text file or passed
+      to the test server at runtime as a list
+      (see <c>run_testspec/1</c> in the manual page
+      for <c>ct</c>). There are two general types of terms:
+      configuration terms and test specification terms.</p>
+    <p>With configuration terms it is possible to e.g. label the test
+      run (similar to <c>run_test -label</c>), evaluate arbitrary expressions
+      before starting a test, import configuration
+      data (similar to
+      <c>run_test -config/-userconfig</c>), specify HTML log directories (similar
+      to
+      <c>run_test -logdir</c>), give aliases to test nodes and test
+      directories (to make a specification easier to read and
+      maintain), enable code coverage analysis (see
+      the <seealso marker="cover_chapter#cover">Code Coverage
+      Analysis</seealso> chapter) and specify event_handler plugins
       (see the <seealso marker="event_handler_chapter#event_handling">
-       Event Handling</seealso> chapter). There is also a term
-       for specifying include directories that should be passed on
-       to the compiler when automatic compilation is performed 
-       (similar to <c>run_test -include</c>, see above).</p>
-    <p>With test specification terms it is possible to state exactly which
-      tests should run and in which order. A test term specifies either
-      one or more suites or one or more test cases. An arbitrary number of test 
-      terms may be declared in sequence. A test term can also specify one or 
-      more test suites or test cases to be skipped. Skipped suites and cases 
-      are not executed and show up in the HTML test log as SKIPPED.</p>
-
-    <note><p>It is not yet possible to specify test case groups in
-      test specifications. This will be supported in a soon upcoming 
-      release.</p></note>
+      Event Handling</seealso> chapter). There is also a term for
+      specifying include directories that should be passed on to the
+      compiler when automatic compilation is performed (similar
+      to <c>run_test -include</c>, see above).</p>
+    <p>With test specification terms it is possible to state exactly
+      which tests should run and in which order. A test term specifies
+      either one or more suites, one or more test case groups, or one
+      or more test cases in a group or suite.</p>
+    <p>An arbitrary number of test terms may be declared in sequence.
+      Common Test will compile the terms into one or more tests to be
+      performed in one resulting test run. Note that a term that
+      specifies a set of test cases will "swallow" one that only
+      specifies a subset of these cases. E.g. the result of merging
+      one term that specifies that all cases in suite S should be
+      executed, with another term specifying only test case X and Y in
+      S, is a test of all cases in S. However, if a term specifying
+      test case X and Y in S is merged with a term specifying case Z
+      in S, the result is a test of X, Y and Z in S.</p>
+    <p>A test term can also specify one or more test suites, groups,
+      or test cases to be skipped. Skipped suites, groups and cases
+      are not executed and show up in the HTML test log files as
+      SKIPPED.</p>
+    <p>When a test case group is specified, the resulting test
+      executes the
+      <c>init_per_group</c> function, followed by all test cases and
+      sub groups (including their configuration functions), and
+      finally the <c>end_per_group</c> function. Also if particular
+      test cases in a group are specified, <c>init_per_group</c>
+      and <c>end_per_group</c> for the group in question are
+      called. If a group which is defined (in <c>Suite:group/0</c>) to
+      be a sub group of another group, is specified (or particular test
+      cases of a sub group are), Common Test will call the configuration
+      functions for the top level groups as well as for the sub group
+      in question (making it possible to pass configuration data all
+      the way from <c>init_per_suite</c> down to the test cases in the
+      sub group).</p>
 
     <p>Below is the test specification syntax. Test specifications can
-      be used to run tests both in a single test host environment and in 
-      a distributed Common Test environment (Large Scale Testing). The init term,
-      as well as node parameters, are only relevant in the latter (see the
-      <seealso marker="ct_master_chapter#test_specifications">Large Scale Testing</seealso>
-      chapter for information). For details on the event_handler term, see the
-      <seealso marker="event_handler_chapter#event_handling">Event Handling</seealso>
-      chapter.</p>
+      be used to run tests both in a single test host environment and
+      in a distributed Common Test environment (Large Scale
+      Testing). The node parameters in the init term are only
+      relevant in the latter (see the
+      <seealso marker="ct_master_chapter#test_specifications">Large
+      Scale Testing</seealso> chapter for information). For details on
+      the event_handler term, see the
+      <seealso marker="event_handler_chapter#event_handling">Event
+      Handling</seealso> chapter.</p>
       <p>Config terms:</p>
     <pre>
       {node, NodeAlias, Node}.
@@ -379,11 +410,17 @@
       {init, InitOptions}.
       {init, [NodeAlias], InitOptions}.
 
+      {label, Label}.
+      {label, NodeRefs, Label}.
+
       {multiply_timetraps, N}.
+      {multiply_timetraps, NodeRefs, N}.
+
       {scale_timetraps, Bool}.
+      {scale_timetraps, NodeRefs, Bool}.
  
       {cover, CoverSpecFile}.
-      {cover, NodeRef, CoverSpecFile}.
+      {cover, NodeRefs, CoverSpecFile}.
       
       {include, IncludeDirs}.
       {include, NodeRefs, IncludeDirs}.
@@ -409,6 +446,12 @@
       {suites, DirRef, Suites}.                                
       {suites, NodeRefs, DirRef, Suites}.
       
+      {groups, DirRef, Suite, Groups}.
+      {groups, NodeRefsDirRef, Suite, Groups}.
+
+      {groups, DirRef, Suite, Group, {cases,Cases}}.
+      {groups, NodeRefsDirRef, Suite, Group, {cases,Cases}}.
+
       {cases, DirRef, Suite, Cases}.                           
       {cases, NodeRefs, DirRef, Suite, Cases}.
 
@@ -437,6 +480,9 @@
       InitArgs      = [term()]
       DirRef        = DirAlias | Dir
       Suites        = atom() | [atom()] | all
+      Suite         = atom()
+      Groups        = atom() | [atom()] | all
+      Group         = atom()
       Cases         = atom() | [atom()] | all
       Comment       = string() | ""
     </pre>