Merge branch 'peppe/common_test/otp-10029' into maint

* peppe/common_test/otp-10029:
  Update documentation

OTP-9972

6 files changed, 118 insertions, 20 deletions

diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml
index 20141d2561..d43ac44ac5 100644
--- a/lib/common_test/doc/src/basics_chapter.xml
+++ b/lib/common_test/doc/src/basics_chapter.xml
@@ -141,8 +141,9 @@
       individual test case.
     </p>
     <p>
-      The test suite module must conform to a callback interface specified 
-      by the CT test server. See the 
+      The test suite module must conform to a
+      <seealso marker="common_test">callback interface</seealso>
+      specified by the CT test server. See the 
       <seealso marker="write_test_chapter#intro">Writing Test Suites</seealso> chapter
       for more information.
     </p>
diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml
index c5b4fd0073..cb06e570bd 100644
--- a/lib/common_test/doc/src/ct_hooks_chapter.xml
+++ b/lib/common_test/doc/src/ct_hooks_chapter.xml
@@ -189,6 +189,22 @@
 	it.</p>
     </section>
     
+    <section>
+      <title>External configuration data and Logging</title>
+      <p>It's possible in the CTH to read configuration data values
+	by calling <c>ct:get_config/1/2/3</c> (as explained in the
+	<seealso marker="config_file_chapter#require_config_data">
+	  External configuration data</seealso>
+	chapter). The config variables in question must, as always, first have been
+	<c>required</c> by means of a suite-, group-, or test case info function,
+	or the <c>ct:require/1/2</c> function. Note that the latter can also be used
+	in CT hook functions.</p>
+      <p>The CT hook functions may call any of the logging functions available
+	in the <c>ct</c> interface to print information to the log files, or to
+	add comments in the suite overview page.
+      </p>
+    </section>    
+
   </section>
 
   <marker id="manipulating"/>
@@ -201,11 +217,13 @@
     functions for a CTH follow a common interface, this interface is 
     described below.</p>
 
-    <p>It is only possible to hook into test function which exists in the test 
-      suite. So in order for a CTH to hook in before 
-      <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, 
-      the <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso> 
-      function must exist in the test suite.</p>
+    <p>Common Test will always call all available hook functions, even pre- and post
+      hooks for configuration functions that are not implemented in the suite.
+      For example, <c>pre_init_per_suite(x_SUITE, ...)</c> and
+      <c>post_init_per_suite(x_SUITE, ...)</c> will be called for test suite
+      <c>x_SUITE</c>, even if it doesn't export <c>init_per_suite/1</c>. This feature
+      makes it possible to use hooks as configuration fallbacks, or even
+      completely replace all configuration functions with hook functions.</p>
 
     <marker id="pre"/>
     <section>
diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml
index b01ab3667d..4d4521957c 100644
--- a/lib/common_test/doc/src/ct_run.xml
+++ b/lib/common_test/doc/src/ct_run.xml
@@ -103,6 +103,7 @@
 	[-no_auto_compile]
 	[-muliply_timetraps Multiplier]
 	[-scale_timetraps]
+	[-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
         [-repeat N [-force_stop]] |
         [-duration HHMMSS [-force_stop]] |
         [-until [YYMoMoDD]HHMMSS [-force_stop]]
@@ -130,6 +131,7 @@
 	[-no_auto_compile]
 	[-muliply_timetraps Multiplier]
 	[-scale_timetraps]
+	[-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
         [-repeat N [-force_stop]] |
         [-duration HHMMSS [-force_stop]] |
         [-until [YYMoMoDD]HHMMSS [-force_stop]]
@@ -150,6 +152,7 @@
 	[-no_auto_compile]
 	[-muliply_timetraps Multiplier]
 	[-scale_timetraps]
+	[-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
 	[-basic_html]</pre>
   </section>
   <section>
diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml
index 7de0912036..05e21241cb 100644
--- a/lib/common_test/doc/src/getting_started_chapter.xml
+++ b/lib/common_test/doc/src/getting_started_chapter.xml
@@ -98,7 +98,8 @@
     <section>
     <title>A simple test suite</title>
     <p>As you've seen in the basics chapter, the test suite module implements
-      callback functions (mandatory or optional) for various purposes, e.g:
+      <seealso marker="common_test">callback functions</seealso>
+      (mandatory or optional) for various purposes, e.g:
       <list>
 	<item>Init/end configuration function for the test suite</item>
 	<item>Init/end configuration function for a test case</item>
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index 848f278fa6..b3ec524b21 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -466,6 +466,9 @@
       
       {logdir, LogDir}.                                        
       {logdir, NodeRefs, LogDir}.
+
+      {create_priv_dir, PrivDirOption}.
+      {create_priv_dir, NodeRefs, PrivDirOption}.
       
       {event_handler, EventHandlers}.
       {event_handler, NodeRefs, EventHandlers}.
@@ -493,6 +496,9 @@
 
       {skip_suites, DirRef, Suites, Comment}.
       {skip_suites, NodeRefs, DirRef, Suites, Comment}.
+
+      {skip_groups, DirRef, Suite, GroupNames, Comment}.
+      {skip_groups, NodeRefs, DirRef, Suite, GroupNames, Comment}.
       
       {skip_cases, DirRef, Suite, Cases, Comment}.
       {skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}.
@@ -512,6 +518,7 @@
       DirAlias      = atom()
       Dir           = string()
       LogDir        = string()
+      PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc
       EventHandlers = atom() | [atom()]
       InitArgs      = [term()]
       CTHModules    = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]
@@ -521,8 +528,9 @@
       Suites        = atom() | [atom()] | all
       Suite         = atom()
       Groups        = GroupSpec | [GroupSpec] | all
-      GroupSpec     = Group | {Group,Properties} | {Group,Properties,GroupSpec}
-      Group         = atom()
+      GroupSpec     = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec}
+      GroupName     = atom()
+      GroupNames    = GroupName | [GroupName]
       Cases         = atom() | [atom()] | all
       Comment       = string() | ""
     </pre>
diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml
index c0ec26ddcc..09f4f8ed3b 100644
--- a/lib/common_test/doc/src/write_test_chapter.xml
+++ b/lib/common_test/doc/src/write_test_chapter.xml
@@ -29,7 +29,6 @@
     <file>write_test_chapter.xml</file>
   </header>
          
-
   <section>
     <marker id="intro"></marker>
     <title>Support for test suite authors</title>
@@ -63,7 +62,8 @@
       function in CT will not be able to locate it (at least not per default).
     </p>
 
-    <p>The <c>ct.hrl</c> header file must be included in all test suite files.
+    <p>It is also recommended that the <c>ct.hrl</c> header file is included
+      in all test suite modules.
     </p>
 
     <p>Each test suite module must export the function <c>all/0</c>
@@ -71,6 +71,12 @@
       to be executed in that module. 
     </p>
 
+    <p>The callback functions that the test suite should implement, and
+      which will be described in more detail below, are
+      all listed in the <seealso marker="common_test">common_test
+	reference manual page</seealso>.
+    </p>
+
   </section>
 
   <section>
@@ -113,6 +119,14 @@
     suite will be skipped automatically (so called <em>auto skipped</em>), 
     including <c>end_per_suite</c>.
     </p>
+    
+    <p>Note that if <c>init_per_suite</c> and <c>end_per_suite</c> do not exist
+      in the suite, Common Test calls dummy functions (with the same names)
+      instead, so that output generated by hook functions may be saved to the log
+      files for these dummies
+      (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test Hooks</seealso>
+      chapter for more information).
+    </p>
   </section>
 
   <marker id="per_testcase"/>
@@ -513,8 +527,7 @@
       execution is immediately stopped and the rest of the cases skipped.</p>
 
     <p>Before execution of a group begins, the configuration function
-    <c>init_per_group(GroupName, Config)</c> is called (the function is
-    mandatory if one or more test case groups are defined). The list of tuples 
+    <c>init_per_group(GroupName, Config)</c> is called. The list of tuples 
     returned from this function is passed to the test cases in the usual 
     manner by means of the <c>Config</c> argument. <c>init_per_group/2</c>
     is meant to be used for initializations common for the test cases in the
@@ -522,6 +535,14 @@
     <c>end_per_group(GroupName, Config</c> function is called. This function
     is meant to be used for cleaning up after <c>init_per_group/2</c>.</p>
 
+    <p>Whenever a group is executed, if <c>init_per_group</c> and
+      <c>end_per_group</c> do not exist in the suite, Common Test calls
+      dummy functions (with the same names) instead. Output generated by
+      hook functions will be saved to the log files for these dummies
+      (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test
+	Hooks</seealso> chapter for more information).
+    </p>
+
     <note><p><c>init_per_testcase/2</c> and <c>end_per_testcase/2</c>
     are always called for each individual test case, no matter if the case 
     belongs to a group or not.</p></note>
@@ -611,6 +632,25 @@
   </section>
 
   <section>
+    <title>Parallel test cases and IO</title>
+    <p>A parallel test case has a private IO server as its group leader. 
+      (Please see the Erlang Run-Time System Application documentation for
+      a description of the group leader concept). The
+      central IO server process that handles the output from regular test
+      cases and configuration functions, does not respond to IO messages
+      during execution of parallel groups. This is important to understand
+      in order to avoid certain traps, like this one:</p>
+    <p>If a process, <c>P</c>, is spawned during execution of e.g.
+      <c>init_per_suite/1</c>, it will inherit the group leader of the
+      <c>init_per_suite</c> process. This group leader is the central IO server
+      process mentioned above. If, at a later time, <em>during parallel test case
+      execution</em>, some event triggers process <c>P</c> to call
+      <c>io:format/1/2</c>, that call will never return (since the group leader
+      is in a non-responsive state) and cause <c>P</c> to hang.
+    </p>
+  </section>
+
+  <section>
     <title>Repeated groups</title>
     <marker id="repeated_groups"></marker>
     <p>A test case group may be repeated a certain number of times
@@ -745,7 +785,7 @@
     <marker id="data_priv_dir"></marker>
     <title>Data and Private Directories</title>
 
-    <p>The data directory (<c>data_dir</c>) is the directory where the
+    <p>The data directory, <c>data_dir</c>, is the directory where the
       test module has its own files needed for the testing. The name
       of the <c>data_dir</c> is the the name of the test suite followed
       by <c>"_data"</c>. For example,
@@ -769,12 +809,39 @@
     </p>
 -->
     <p>
-      The <c>priv_dir</c> is the test suite's private directory. This
-      directory should be used when a test case needs to write to
-      files. The name of the private directory is generated by the test
-      server, which also creates the directory.
+      <c>priv_dir</c> is the private directory for the test cases.
+      This directory may be used whenever a test case (or configuration function)
+      needs to write something to file. The name of the private directory is
+      generated by Common Test, which also creates the directory.
     </p>
-    
+    <p>By default, Common Test creates one central private directory
+      per test run that all test cases share. This may not always be suitable,
+      especially if the same test cases are executed multiple times during
+      a test run (e.g. if they belong to a test case group with repeat
+      property), and there's a risk that files in the private directory get
+      overwritten. Under these circumstances, it's possible to configure
+      Common Test to create one dedicated private directory per
+      test case and execution instead. This is accomplished by means of
+      the flag/option: <c>create_priv_dir</c> (to be used with the
+      <c>ct_run</c> program, the <c>ct:run_test/1</c> function, or
+      as test specification term). There are three possible values
+      for this option:
+      <list>
+	<item><c>auto_per_run</c></item>
+	<item><c>auto_per_tc</c></item>
+	<item><c>manual_per_tc</c></item>
+      </list>
+      The first value indicates the default priv_dir behaviour, i.e.
+      one private directory created per test run. The two latter
+      values tell Common Test to generate a unique test directory name
+      per test case and execution. If the auto version is used, <em>all</em>
+      private directories will be created automatically. This can obviously
+      become very inefficient for test runs with many test cases and/or
+      repetitions. Therefore, in case the manual version is instead used, the
+      test case must tell Common Test to create priv_dir when it needs it.
+      It does this by calling the function <c>ct:make_priv_dir/0</c>.
+      </p>
+
       <note><p>You should not depend on current working directory for
 	  reading and writing data files since this is not portable. All 
 	  scratch files are to be written in the <c>priv_dir</c> and all