aboutsummaryrefslogtreecommitdiffstats
path: root/lib
diff options
context:
space:
mode:
authorPeter Andersson <peppe@erlang.org>2012-03-29 00:11:17 +0200
committerPeter Andersson <peppe@erlang.org>2012-03-29 00:15:35 +0200
commitccc9e16381244760b3013e11b6fe8f1fe2a0a809 (patch)
treedef25caab3503b2a12670bec0337753923716dd0 /lib
parent1d1d4456e8bc1089ab632b189c6cc866a4ff59a3 (diff)
parent96b3f1cdf435954b826be3da8de714b13910d396 (diff)
downloadotp-ccc9e16381244760b3013e11b6fe8f1fe2a0a809.tar.gz
otp-ccc9e16381244760b3013e11b6fe8f1fe2a0a809.tar.bz2
otp-ccc9e16381244760b3013e11b6fe8f1fe2a0a809.zip
Merge branch 'peppe/common_test/otp-10029' into maint
* peppe/common_test/otp-10029: Update documentation OTP-9972
Diffstat (limited to 'lib')
-rw-r--r--lib/common_test/doc/src/basics_chapter.xml5
-rw-r--r--lib/common_test/doc/src/ct_hooks_chapter.xml28
-rw-r--r--lib/common_test/doc/src/ct_run.xml3
-rw-r--r--lib/common_test/doc/src/getting_started_chapter.xml3
-rw-r--r--lib/common_test/doc/src/run_test_chapter.xml12
-rw-r--r--lib/common_test/doc/src/write_test_chapter.xml87
6 files changed, 118 insertions, 20 deletions
diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml
index 20141d2561..d43ac44ac5 100644
--- a/lib/common_test/doc/src/basics_chapter.xml
+++ b/lib/common_test/doc/src/basics_chapter.xml
@@ -141,8 +141,9 @@
individual test case.
</p>
<p>
- The test suite module must conform to a callback interface specified
- by the CT test server. See the
+ The test suite module must conform to a
+ <seealso marker="common_test">callback interface</seealso>
+ specified by the CT test server. See the
<seealso marker="write_test_chapter#intro">Writing Test Suites</seealso> chapter
for more information.
</p>
diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml
index c5b4fd0073..cb06e570bd 100644
--- a/lib/common_test/doc/src/ct_hooks_chapter.xml
+++ b/lib/common_test/doc/src/ct_hooks_chapter.xml
@@ -189,6 +189,22 @@
it.</p>
</section>
+ <section>
+ <title>External configuration data and Logging</title>
+ <p>It's possible in the CTH to read configuration data values
+ by calling <c>ct:get_config/1/2/3</c> (as explained in the
+ <seealso marker="config_file_chapter#require_config_data">
+ External configuration data</seealso>
+ chapter). The config variables in question must, as always, first have been
+ <c>required</c> by means of a suite-, group-, or test case info function,
+ or the <c>ct:require/1/2</c> function. Note that the latter can also be used
+ in CT hook functions.</p>
+ <p>The CT hook functions may call any of the logging functions available
+ in the <c>ct</c> interface to print information to the log files, or to
+ add comments in the suite overview page.
+ </p>
+ </section>
+
</section>
<marker id="manipulating"/>
@@ -201,11 +217,13 @@
functions for a CTH follow a common interface, this interface is
described below.</p>
- <p>It is only possible to hook into test function which exists in the test
- suite. So in order for a CTH to hook in before
- <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>,
- the <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>
- function must exist in the test suite.</p>
+ <p>Common Test will always call all available hook functions, even pre- and post
+ hooks for configuration functions that are not implemented in the suite.
+ For example, <c>pre_init_per_suite(x_SUITE, ...)</c> and
+ <c>post_init_per_suite(x_SUITE, ...)</c> will be called for test suite
+ <c>x_SUITE</c>, even if it doesn't export <c>init_per_suite/1</c>. This feature
+ makes it possible to use hooks as configuration fallbacks, or even
+ completely replace all configuration functions with hook functions.</p>
<marker id="pre"/>
<section>
diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml
index b01ab3667d..4d4521957c 100644
--- a/lib/common_test/doc/src/ct_run.xml
+++ b/lib/common_test/doc/src/ct_run.xml
@@ -103,6 +103,7 @@
[-no_auto_compile]
[-muliply_timetraps Multiplier]
[-scale_timetraps]
+ [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
[-repeat N [-force_stop]] |
[-duration HHMMSS [-force_stop]] |
[-until [YYMoMoDD]HHMMSS [-force_stop]]
@@ -130,6 +131,7 @@
[-no_auto_compile]
[-muliply_timetraps Multiplier]
[-scale_timetraps]
+ [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
[-repeat N [-force_stop]] |
[-duration HHMMSS [-force_stop]] |
[-until [YYMoMoDD]HHMMSS [-force_stop]]
@@ -150,6 +152,7 @@
[-no_auto_compile]
[-muliply_timetraps Multiplier]
[-scale_timetraps]
+ [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc]
[-basic_html]</pre>
</section>
<section>
diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml
index 7de0912036..05e21241cb 100644
--- a/lib/common_test/doc/src/getting_started_chapter.xml
+++ b/lib/common_test/doc/src/getting_started_chapter.xml
@@ -98,7 +98,8 @@
<section>
<title>A simple test suite</title>
<p>As you've seen in the basics chapter, the test suite module implements
- callback functions (mandatory or optional) for various purposes, e.g:
+ <seealso marker="common_test">callback functions</seealso>
+ (mandatory or optional) for various purposes, e.g:
<list>
<item>Init/end configuration function for the test suite</item>
<item>Init/end configuration function for a test case</item>
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index 848f278fa6..b3ec524b21 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -466,6 +466,9 @@
{logdir, LogDir}.
{logdir, NodeRefs, LogDir}.
+
+ {create_priv_dir, PrivDirOption}.
+ {create_priv_dir, NodeRefs, PrivDirOption}.
{event_handler, EventHandlers}.
{event_handler, NodeRefs, EventHandlers}.
@@ -493,6 +496,9 @@
{skip_suites, DirRef, Suites, Comment}.
{skip_suites, NodeRefs, DirRef, Suites, Comment}.
+
+ {skip_groups, DirRef, Suite, GroupNames, Comment}.
+ {skip_groups, NodeRefs, DirRef, Suite, GroupNames, Comment}.
{skip_cases, DirRef, Suite, Cases, Comment}.
{skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}.
@@ -512,6 +518,7 @@
DirAlias = atom()
Dir = string()
LogDir = string()
+ PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc
EventHandlers = atom() | [atom()]
InitArgs = [term()]
CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]
@@ -521,8 +528,9 @@
Suites = atom() | [atom()] | all
Suite = atom()
Groups = GroupSpec | [GroupSpec] | all
- GroupSpec = Group | {Group,Properties} | {Group,Properties,GroupSpec}
- Group = atom()
+ GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec}
+ GroupName = atom()
+ GroupNames = GroupName | [GroupName]
Cases = atom() | [atom()] | all
Comment = string() | ""
</pre>
diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml
index c0ec26ddcc..09f4f8ed3b 100644
--- a/lib/common_test/doc/src/write_test_chapter.xml
+++ b/lib/common_test/doc/src/write_test_chapter.xml
@@ -29,7 +29,6 @@
<file>write_test_chapter.xml</file>
</header>
-
<section>
<marker id="intro"></marker>
<title>Support for test suite authors</title>
@@ -63,7 +62,8 @@
function in CT will not be able to locate it (at least not per default).
</p>
- <p>The <c>ct.hrl</c> header file must be included in all test suite files.
+ <p>It is also recommended that the <c>ct.hrl</c> header file is included
+ in all test suite modules.
</p>
<p>Each test suite module must export the function <c>all/0</c>
@@ -71,6 +71,12 @@
to be executed in that module.
</p>
+ <p>The callback functions that the test suite should implement, and
+ which will be described in more detail below, are
+ all listed in the <seealso marker="common_test">common_test
+ reference manual page</seealso>.
+ </p>
+
</section>
<section>
@@ -113,6 +119,14 @@
suite will be skipped automatically (so called <em>auto skipped</em>),
including <c>end_per_suite</c>.
</p>
+
+ <p>Note that if <c>init_per_suite</c> and <c>end_per_suite</c> do not exist
+ in the suite, Common Test calls dummy functions (with the same names)
+ instead, so that output generated by hook functions may be saved to the log
+ files for these dummies
+ (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test Hooks</seealso>
+ chapter for more information).
+ </p>
</section>
<marker id="per_testcase"/>
@@ -513,8 +527,7 @@
execution is immediately stopped and the rest of the cases skipped.</p>
<p>Before execution of a group begins, the configuration function
- <c>init_per_group(GroupName, Config)</c> is called (the function is
- mandatory if one or more test case groups are defined). The list of tuples
+ <c>init_per_group(GroupName, Config)</c> is called. The list of tuples
returned from this function is passed to the test cases in the usual
manner by means of the <c>Config</c> argument. <c>init_per_group/2</c>
is meant to be used for initializations common for the test cases in the
@@ -522,6 +535,14 @@
<c>end_per_group(GroupName, Config</c> function is called. This function
is meant to be used for cleaning up after <c>init_per_group/2</c>.</p>
+ <p>Whenever a group is executed, if <c>init_per_group</c> and
+ <c>end_per_group</c> do not exist in the suite, Common Test calls
+ dummy functions (with the same names) instead. Output generated by
+ hook functions will be saved to the log files for these dummies
+ (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test
+ Hooks</seealso> chapter for more information).
+ </p>
+
<note><p><c>init_per_testcase/2</c> and <c>end_per_testcase/2</c>
are always called for each individual test case, no matter if the case
belongs to a group or not.</p></note>
@@ -611,6 +632,25 @@
</section>
<section>
+ <title>Parallel test cases and IO</title>
+ <p>A parallel test case has a private IO server as its group leader.
+ (Please see the Erlang Run-Time System Application documentation for
+ a description of the group leader concept). The
+ central IO server process that handles the output from regular test
+ cases and configuration functions, does not respond to IO messages
+ during execution of parallel groups. This is important to understand
+ in order to avoid certain traps, like this one:</p>
+ <p>If a process, <c>P</c>, is spawned during execution of e.g.
+ <c>init_per_suite/1</c>, it will inherit the group leader of the
+ <c>init_per_suite</c> process. This group leader is the central IO server
+ process mentioned above. If, at a later time, <em>during parallel test case
+ execution</em>, some event triggers process <c>P</c> to call
+ <c>io:format/1/2</c>, that call will never return (since the group leader
+ is in a non-responsive state) and cause <c>P</c> to hang.
+ </p>
+ </section>
+
+ <section>
<title>Repeated groups</title>
<marker id="repeated_groups"></marker>
<p>A test case group may be repeated a certain number of times
@@ -745,7 +785,7 @@
<marker id="data_priv_dir"></marker>
<title>Data and Private Directories</title>
- <p>The data directory (<c>data_dir</c>) is the directory where the
+ <p>The data directory, <c>data_dir</c>, is the directory where the
test module has its own files needed for the testing. The name
of the <c>data_dir</c> is the the name of the test suite followed
by <c>"_data"</c>. For example,
@@ -769,12 +809,39 @@
</p>
-->
<p>
- The <c>priv_dir</c> is the test suite's private directory. This
- directory should be used when a test case needs to write to
- files. The name of the private directory is generated by the test
- server, which also creates the directory.
+ <c>priv_dir</c> is the private directory for the test cases.
+ This directory may be used whenever a test case (or configuration function)
+ needs to write something to file. The name of the private directory is
+ generated by Common Test, which also creates the directory.
</p>
-
+ <p>By default, Common Test creates one central private directory
+ per test run that all test cases share. This may not always be suitable,
+ especially if the same test cases are executed multiple times during
+ a test run (e.g. if they belong to a test case group with repeat
+ property), and there's a risk that files in the private directory get
+ overwritten. Under these circumstances, it's possible to configure
+ Common Test to create one dedicated private directory per
+ test case and execution instead. This is accomplished by means of
+ the flag/option: <c>create_priv_dir</c> (to be used with the
+ <c>ct_run</c> program, the <c>ct:run_test/1</c> function, or
+ as test specification term). There are three possible values
+ for this option:
+ <list>
+ <item><c>auto_per_run</c></item>
+ <item><c>auto_per_tc</c></item>
+ <item><c>manual_per_tc</c></item>
+ </list>
+ The first value indicates the default priv_dir behaviour, i.e.
+ one private directory created per test run. The two latter
+ values tell Common Test to generate a unique test directory name
+ per test case and execution. If the auto version is used, <em>all</em>
+ private directories will be created automatically. This can obviously
+ become very inefficient for test runs with many test cases and/or
+ repetitions. Therefore, in case the manual version is instead used, the
+ test case must tell Common Test to create priv_dir when it needs it.
+ It does this by calling the function <c>ct:make_priv_dir/0</c>.
+ </p>
+
<note><p>You should not depend on current working directory for
reading and writing data files since this is not portable. All
scratch files are to be written in the <c>priv_dir</c> and all