aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src
diff options
context:
space:
mode:
Diffstat (limited to 'lib/common_test/doc/src')
-rw-r--r--lib/common_test/doc/src/common_test_app.xml11
-rw-r--r--lib/common_test/doc/src/cover_chapter.xml112
-rw-r--r--lib/common_test/doc/src/ct_hooks_chapter.xml27
-rw-r--r--lib/common_test/doc/src/ct_run.xml2
-rw-r--r--lib/common_test/doc/src/run_test_chapter.xml11
-rw-r--r--lib/common_test/doc/src/write_test_chapter.xml52
6 files changed, 169 insertions, 46 deletions
diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml
index b6d4a633cb..151159ad69 100644
--- a/lib/common_test/doc/src/common_test_app.xml
+++ b/lib/common_test/doc/src/common_test_app.xml
@@ -4,7 +4,7 @@
<erlref>
<header>
<copyright>
- <year>2003</year><year>2012</year>
+ <year>2003</year><year>2013</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
@@ -170,7 +170,9 @@
<v> UserData = term()</v>
<v> Conns = [atom()]</v>
<v> CSSFile = string()</v>
- <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]</v>
+ <v> CTHs = [CTHModule |</v>
+ <v>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{CTHModule, CTHInitArgs} |</v>
+ <v>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{CTHModule, CTHInitArgs, CTHPriority}]</v>
<v> CTHModule = atom()</v>
<v> CTHInitArgs = term()</v>
</type>
@@ -297,8 +299,9 @@
<v> UserData = term()</v>
<v> Conns = [atom()]</v>
<v> CSSFile = string()</v>
- <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} |
- {CTHModule, CTHInitArgs, CTHPriority}]</v>
+ <v> CTHs = [CTHModule |</v>
+ <v> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{CTHModule, CTHInitArgs} |</v>
+ <v> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{CTHModule, CTHInitArgs, CTHPriority}]</v>
<v> CTHModule = atom()</v>
<v> CTHInitArgs = term()</v>
</type>
diff --git a/lib/common_test/doc/src/cover_chapter.xml b/lib/common_test/doc/src/cover_chapter.xml
index 803a71de07..4fa92d5583 100644
--- a/lib/common_test/doc/src/cover_chapter.xml
+++ b/lib/common_test/doc/src/cover_chapter.xml
@@ -108,6 +108,33 @@
specifications</seealso>).</p>
</section>
+ <marker id="cover_stop"></marker>
+ <section>
+ <title>Stopping the cover tool when tests are completed</title>
+ <p>By default the Cover tool is automatically stopped when the
+ tests are completed. This causes the original (non cover
+ compiled) modules to be loaded back in to the test node. If a
+ process at this point is still running old code of any of the
+ modules that are cover compiled, meaning that it has not done
+ any fully qualified function call after the cover compilation,
+ the process will now be killed. To avoid this it is possible to
+ set the value of the <c>cover_stop</c> option to
+ <c>false</c>. This means that the modules will stay cover
+ compiled, and it is therefore only recommended if the erlang
+ node(s) under test is terminated after the test is completed
+ or if cover can be manually stopped.</p>
+
+ <p>The option can be set by using the <c>-cover_stop</c> flag with
+ <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the
+ Opts argument to <c><seealso
+ marker="ct#run_test-1">ct:run_test/1</seealso></c>, or by adding
+ a <c>cover_stop</c> term in your test specification (see chapter
+ about <seealso
+ marker="run_test_chapter#test_specifications">test
+ specifications</seealso>).</p>
+
+ </section>
+
<section>
<title>The cover specification file</title>
<p>These are the terms allowed in a cover specification file:</p>
@@ -148,6 +175,11 @@
%% Specific modules to exclude in cover.
{excl_mods, Mods}.
+
+ %% Cross cover compilation
+ %% Tag = atom(), an identifier for a test run
+ %% Mod = [atom()], modules to compile for accumulated analysis
+ {cross,[{Tag,Mods}]}.
</pre>
<p>The <c>incl_dirs_r</c> and <c>excl_dirs_r</c> terms tell Common
@@ -163,6 +195,81 @@
specification file for Common Test).</p>
</section>
+ <marker id="cross_cover"/>
+ <section>
+ <title>Cross cover analysis</title>
+ <p>The cross cover mechanism allows cover analysis of modules
+ across multiple tests. It is useful if some code, e.g. a library
+ module, is used by many different tests and the accumulated cover
+ result is desirable.</p>
+
+ <p>This can of course also be achieved in a more customized way by
+ using the <c>export</c> parameter in the cover specification and
+ analysing the result off line, but the cross cover mechanism is a
+ build in solution which also provides the logging.</p>
+
+ <p>The mechanism is easiest explained via an example:</p>
+
+ <p>Let's say that there are two systems, <c>s1</c> and <c>s2</c>,
+ which are tested in separate test runs. System <c>s1</c> contains
+ a library module <c>m1</c> which is tested by the <c>s1</c> test
+ run and is included in <c>s1</c>'s cover specification:</p>
+
+<code type="none">
+s1.cover:
+ {incl_mods,[m1]}.</code>
+
+ <p>When analysing code coverage, the result for <c>m1</c> can be
+ seen in the cover log in the <c>s1</c> test result.</p>
+
+ <p>Now, let's imagine that since <c>m1</c> is a library module, it
+ is also used quite a bit by system <c>s2</c>. The <c>s2</c> test
+ run does not specifically test <c>m1</c>, but it might still be
+ interesting to see which parts of <c>m1</c> is actually covered by
+ the <c>s2</c> tests. To do this, <c>m1</c> could be included also
+ in <c>s2</c>'s cover specification:</p>
+
+<code type="none">
+s2.cover:
+ {incl_mods,[m1]}.</code>
+
+ <p>This would give an entry for <c>m1</c> also in the cover log
+ for the <c>s2</c> test run. The problem is that this would only
+ reflect the coverage by <c>s2</c> tests, not the accumulated
+ result over <c>s1</c> and <c>s2</c>. And this is where the cross
+ cover mechanism comes in handy.</p>
+
+ <p>If instead the cover specification for <c>s2</c> was like
+ this:</p>
+
+<code type="none">
+s2.cover:
+ {cross,[{s1,[m1]}]}.</code>
+
+ <p>then <c>m1</c> would be cover compiled in the <c>s2</c> test
+ run, but not shown in the coverage log. Instead, if
+ <c>ct_cover:cross_cover_analyse/2</c> is called after both
+ <c>s1</c> and <c>s2</c> test runs are completed, the accumulated
+ result for <c>m1</c> would be available in the cross cover log for
+ the <c>s1</c> test run.</p>
+
+ <p>The call to the analyse function must be like this:</p>
+
+<code type="none">
+ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code>
+
+ <p>where <c>S1LogDir</c> and <c>S2LogDir</c> are the directories
+ named <c>&lt;TestName&gt;.logs</c> for each test respectively.</p>
+
+ <p>Note the tags <c>s1</c> and <c>s2</c> which are used in the
+ cover specification file and in the call to
+ <c>ct_cover:cross_cover_analyse/2</c>. The point of these are only
+ to map the modules specified in the cover specification to the log
+ directory specified in the call to the analyse function. The name
+ of the tag has no meaning beyond this.</p>
+
+ </section>
+
<section>
<title>Logging</title>
<p>To view the result of a code coverage test, follow the
@@ -170,6 +277,11 @@
takes you to the code coverage overview page. If you have
successfully performed a detailed coverage analysis, you
find links to each individual module coverage page here.</p>
+
+ <p>If cross cover analysis has been performed, and there are
+ accumulated coverage results for the current test, then the -
+ "Coverdata collected over all tests" link will take you to these
+ results.</p>
</section>
</chapter>
diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml
index 27d56fd47d..fe871eb516 100644
--- a/lib/common_test/doc/src/ct_hooks_chapter.xml
+++ b/lib/common_test/doc/src/ct_hooks_chapter.xml
@@ -4,7 +4,7 @@
<chapter>
<header>
<copyright>
- <year>2011</year><year>2012</year>
+ <year>2011</year><year>2013</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
@@ -439,14 +439,14 @@ terminate(State) ->
<table>
<row>
- <cell><em>CTH Name</em></cell>
- <cell><em>Is Built-in</em></cell>
- <cell><em>Description</em></cell>
+ <cell align="left"><em>CTH Name</em></cell>
+ <cell align="left"><em>Is Built-in</em></cell>
+ <cell align="left"><em>Description</em></cell>
</row>
<row>
- <cell>cth_log_redirect</cell>
- <cell>yes</cell>
- <cell>Captures all error_logger and SASL logging events and prints them
+ <cell align="left">cth_log_redirect</cell>
+ <cell align="left">yes</cell>
+ <cell align="left">Captures all error_logger and SASL logging events and prints them
to the current test case log. If an event can not be associated with a
testcase it will be printed in the common test framework log. This will
happen for testcases which are run in parallel and events which occur
@@ -455,9 +455,9 @@ terminate(State) ->
using the normal SASL mechanisms. </cell>
</row>
<row>
- <cell>cth_surefire</cell>
- <cell>no</cell>
- <cell><p>Captures all test results and outputs them as surefire
+ <cell align="left">cth_surefire</cell>
+ <cell align="left">no</cell>
+ <cell align="left"><p>Captures all test results and outputs them as surefire
XML into a file. The file which is created is by default
called junit_report.xml. The file name can be changed by
setting the <c>path</c> option for this hook, e.g.</p>
@@ -467,13 +467,14 @@ terminate(State) ->
<p>If the <c>url_base</c> option is set, an additional
attribute named <c>url</c> will be added to each
<c>testsuite</c> and <c>testcase</c> XML element. The value will
- be a constructed from the <c>url_base</c> and a relative path
+ be constructed from the <c>url_base</c> and a relative path
to the test suite or test case log respectively, e.g.</p>
- <code>-ct_hooks cth_surefire [{url_base,"http://myserver.com/"}]</code>
+ <code>-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]</code>
<p>will give a url attribute value similar to</p>
- <code>"http://myserver.com/[email protected]_11.19.39/x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code>
+ <code>"http://myserver.com/[email protected]_11.19.39/
+x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code>
<p>Surefire XML can for instance be used by Jenkins to display test
results.</p></cell>
diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml
index c6749d6960..0750f560b3 100644
--- a/lib/common_test/doc/src/ct_run.xml
+++ b/lib/common_test/doc/src/ct_run.xml
@@ -104,6 +104,7 @@
[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
[-stylesheet CSSFile]
[-cover CoverCfgFile]
+ [-cover_stop Bool]
[-event_handler EvHandler1 EvHandler2 .. EvHandlerN] |
[-event_handler_init EvHandler1 InitArg1 and
EvHandler2 InitArg2 and .. EvHandlerN InitArgN]
@@ -138,6 +139,7 @@
[-silent_connections [ConnType1 ConnType2 .. ConnTypeN]]
[-stylesheet CSSFile]
[-cover CoverCfgFile]
+ [-cover_stop Bool]
[-event_handler EvHandler1 EvHandler2 .. EvHandlerN] |
[-event_handler_init EvHandler1 InitArg1 and
EvHandler2 InitArg2 and .. EvHandlerN InitArgN]
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml
index a0b2c96006..d5f5d89e05 100644
--- a/lib/common_test/doc/src/run_test_chapter.xml
+++ b/lib/common_test/doc/src/run_test_chapter.xml
@@ -4,7 +4,7 @@
<chapter>
<header>
<copyright>
- <year>2003</year><year>2012</year>
+ <year>2003</year><year>2013</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
@@ -155,6 +155,8 @@
<item><c><![CDATA[-stylesheet <css_file>]]></c>, points out a user HTML style sheet (see below).</item>
<item><c><![CDATA[-cover <cover_cfg_file>]]></c>, to perform code coverage test (see
<seealso marker="cover_chapter#cover">Code Coverage Analysis</seealso>).</item>
+ <item><c><![CDATA[-cover_stop <bool>]]></c>, to specify if the cover tool shall be stopped after the test is completed (see
+ <seealso marker="cover_chapter#cover_stop">Code Coverage Analysis</seealso>).</item>
<item><c><![CDATA[-event_handler <event_handlers>]]></c>, to install
<seealso marker="event_handler_chapter#event_handling">event handlers</seealso>.</item>
<item><c><![CDATA[-event_handler_init <event_handlers>]]></c>, to install
@@ -658,6 +660,9 @@
{cover, CoverSpecFile}.
{cover, NodeRefs, CoverSpecFile}.
+ {cover_stop, Bool}.
+ {cover_stop, NodeRefs, Bool}.
+
{include, IncludeDirs}.
{include, NodeRefs, IncludeDirs}.
@@ -747,7 +752,9 @@
PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc
EventHandlers = atom() | [atom()]
InitArgs = [term()]
- CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]
+ CTHModules = [CTHModule |
+ {CTHModule, CTHInitArgs} |
+ {CTHModule, CTHInitArgs, CTHPriority}]
CTHModule = atom()
CTHInitArgs = term()
Dir = string()
diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml
index 248d7de8b6..cc8d913994 100644
--- a/lib/common_test/doc/src/write_test_chapter.xml
+++ b/lib/common_test/doc/src/write_test_chapter.xml
@@ -4,7 +4,7 @@
<chapter>
<header>
<copyright>
- <year>2003</year><year>2012</year>
+ <year>2003</year><year>2013</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
@@ -982,38 +982,36 @@
<p>Example:</p>
<pre>
+ Some printouts during test case execution:
- Some printouts during test case execution:
+ io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]),
+ ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]),
+ ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]),
+ ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]),
+ ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]),
+ ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]),
- io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]),
- ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]),
- ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]),
- ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]),
- ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]),
- ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]),
+ If starting the test without specifying any verbosity levels:
- If starting the test without specifying any verbosity levels:
+ $ ct_run ...
- $ ct_run ...
+ the following gets printed:
- the following gets printed:
-
- 1. Standard IO, importance = 50
- 2. Uncategorized, importance = 50
- 3. Categorized info, importance = 50
- 5. Categorized error, importance = 75
- 6. Categorized error, importance = 99
-
- If starting the test with:
-
- $ ct_run -verbosity 1 and info 75
-
- the following gets printed:
+ 1. Standard IO, importance = 50
+ 2. Uncategorized, importance = 50
+ 3. Categorized info, importance = 50
+ 5. Categorized error, importance = 75
+ 6. Categorized error, importance = 99
+
+ If starting the test with:
+
+ $ ct_run -verbosity 1 and info 75
+
+ the following gets printed:
- 3. Categorized info, importance = 50
- 4. Categorized info, importance = 25
- 6. Categorized error, importance = 99
- </pre>
+ 3. Categorized info, importance = 50
+ 4. Categorized info, importance = 25
+ 6. Categorized error, importance = 99</pre>
<p>How categories can be mapped to CSS tags is documented in the
<seealso marker="run_test_chapter#html_stylesheet">Running Tests</seealso>