diff options
Diffstat (limited to 'lib/common_test/doc')
| -rw-r--r-- | lib/common_test/doc/src/common_test_app.xml | 11 | ||||
| -rw-r--r-- | lib/common_test/doc/src/cover_chapter.xml | 112 | ||||
| -rw-r--r-- | lib/common_test/doc/src/ct_hooks_chapter.xml | 27 | ||||
| -rw-r--r-- | lib/common_test/doc/src/ct_run.xml | 2 | ||||
| -rw-r--r-- | lib/common_test/doc/src/run_test_chapter.xml | 11 | ||||
| -rw-r--r-- | lib/common_test/doc/src/write_test_chapter.xml | 52 |
6 files changed, 169 insertions, 46 deletions
diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index b6d4a633cb..151159ad69 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -4,7 +4,7 @@ <erlref> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -170,7 +170,9 @@ <v> UserData = term()</v> <v> Conns = [atom()]</v> <v> CSSFile = string()</v> - <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]</v> + <v> CTHs = [CTHModule |</v> + <v> {CTHModule, CTHInitArgs} |</v> + <v> {CTHModule, CTHInitArgs, CTHPriority}]</v> <v> CTHModule = atom()</v> <v> CTHInitArgs = term()</v> </type> @@ -297,8 +299,9 @@ <v> UserData = term()</v> <v> Conns = [atom()]</v> <v> CSSFile = string()</v> - <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} | - {CTHModule, CTHInitArgs, CTHPriority}]</v> + <v> CTHs = [CTHModule |</v> + <v> {CTHModule, CTHInitArgs} |</v> + <v> {CTHModule, CTHInitArgs, CTHPriority}]</v> <v> CTHModule = atom()</v> <v> CTHInitArgs = term()</v> </type> diff --git a/lib/common_test/doc/src/cover_chapter.xml b/lib/common_test/doc/src/cover_chapter.xml index 803a71de07..4fa92d5583 100644 --- a/lib/common_test/doc/src/cover_chapter.xml +++ b/lib/common_test/doc/src/cover_chapter.xml @@ -108,6 +108,33 @@ specifications</seealso>).</p> </section> + <marker id="cover_stop"></marker> + <section> + <title>Stopping the cover tool when tests are completed</title> + <p>By default the Cover tool is automatically stopped when the + tests are completed. This causes the original (non cover + compiled) modules to be loaded back in to the test node. If a + process at this point is still running old code of any of the + modules that are cover compiled, meaning that it has not done + any fully qualified function call after the cover compilation, + the process will now be killed. To avoid this it is possible to + set the value of the <c>cover_stop</c> option to + <c>false</c>. This means that the modules will stay cover + compiled, and it is therefore only recommended if the erlang + node(s) under test is terminated after the test is completed + or if cover can be manually stopped.</p> + + <p>The option can be set by using the <c>-cover_stop</c> flag with + <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the + Opts argument to <c><seealso + marker="ct#run_test-1">ct:run_test/1</seealso></c>, or by adding + a <c>cover_stop</c> term in your test specification (see chapter + about <seealso + marker="run_test_chapter#test_specifications">test + specifications</seealso>).</p> + + </section> + <section> <title>The cover specification file</title> <p>These are the terms allowed in a cover specification file:</p> @@ -148,6 +175,11 @@ %% Specific modules to exclude in cover. {excl_mods, Mods}. + + %% Cross cover compilation + %% Tag = atom(), an identifier for a test run + %% Mod = [atom()], modules to compile for accumulated analysis + {cross,[{Tag,Mods}]}. </pre> <p>The <c>incl_dirs_r</c> and <c>excl_dirs_r</c> terms tell Common @@ -163,6 +195,81 @@ specification file for Common Test).</p> </section> + <marker id="cross_cover"/> + <section> + <title>Cross cover analysis</title> + <p>The cross cover mechanism allows cover analysis of modules + across multiple tests. It is useful if some code, e.g. a library + module, is used by many different tests and the accumulated cover + result is desirable.</p> + + <p>This can of course also be achieved in a more customized way by + using the <c>export</c> parameter in the cover specification and + analysing the result off line, but the cross cover mechanism is a + build in solution which also provides the logging.</p> + + <p>The mechanism is easiest explained via an example:</p> + + <p>Let's say that there are two systems, <c>s1</c> and <c>s2</c>, + which are tested in separate test runs. System <c>s1</c> contains + a library module <c>m1</c> which is tested by the <c>s1</c> test + run and is included in <c>s1</c>'s cover specification:</p> + +<code type="none"> +s1.cover: + {incl_mods,[m1]}.</code> + + <p>When analysing code coverage, the result for <c>m1</c> can be + seen in the cover log in the <c>s1</c> test result.</p> + + <p>Now, let's imagine that since <c>m1</c> is a library module, it + is also used quite a bit by system <c>s2</c>. The <c>s2</c> test + run does not specifically test <c>m1</c>, but it might still be + interesting to see which parts of <c>m1</c> is actually covered by + the <c>s2</c> tests. To do this, <c>m1</c> could be included also + in <c>s2</c>'s cover specification:</p> + +<code type="none"> +s2.cover: + {incl_mods,[m1]}.</code> + + <p>This would give an entry for <c>m1</c> also in the cover log + for the <c>s2</c> test run. The problem is that this would only + reflect the coverage by <c>s2</c> tests, not the accumulated + result over <c>s1</c> and <c>s2</c>. And this is where the cross + cover mechanism comes in handy.</p> + + <p>If instead the cover specification for <c>s2</c> was like + this:</p> + +<code type="none"> +s2.cover: + {cross,[{s1,[m1]}]}.</code> + + <p>then <c>m1</c> would be cover compiled in the <c>s2</c> test + run, but not shown in the coverage log. Instead, if + <c>ct_cover:cross_cover_analyse/2</c> is called after both + <c>s1</c> and <c>s2</c> test runs are completed, the accumulated + result for <c>m1</c> would be available in the cross cover log for + the <c>s1</c> test run.</p> + + <p>The call to the analyse function must be like this:</p> + +<code type="none"> +ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code> + + <p>where <c>S1LogDir</c> and <c>S2LogDir</c> are the directories + named <c><TestName>.logs</c> for each test respectively.</p> + + <p>Note the tags <c>s1</c> and <c>s2</c> which are used in the + cover specification file and in the call to + <c>ct_cover:cross_cover_analyse/2</c>. The point of these are only + to map the modules specified in the cover specification to the log + directory specified in the call to the analyse function. The name + of the tag has no meaning beyond this.</p> + + </section> + <section> <title>Logging</title> <p>To view the result of a code coverage test, follow the @@ -170,6 +277,11 @@ takes you to the code coverage overview page. If you have successfully performed a detailed coverage analysis, you find links to each individual module coverage page here.</p> + + <p>If cross cover analysis has been performed, and there are + accumulated coverage results for the current test, then the - + "Coverdata collected over all tests" link will take you to these + results.</p> </section> </chapter> diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml index 27d56fd47d..fe871eb516 100644 --- a/lib/common_test/doc/src/ct_hooks_chapter.xml +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2011</year><year>2012</year> + <year>2011</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -439,14 +439,14 @@ terminate(State) -> <table> <row> - <cell><em>CTH Name</em></cell> - <cell><em>Is Built-in</em></cell> - <cell><em>Description</em></cell> + <cell align="left"><em>CTH Name</em></cell> + <cell align="left"><em>Is Built-in</em></cell> + <cell align="left"><em>Description</em></cell> </row> <row> - <cell>cth_log_redirect</cell> - <cell>yes</cell> - <cell>Captures all error_logger and SASL logging events and prints them + <cell align="left">cth_log_redirect</cell> + <cell align="left">yes</cell> + <cell align="left">Captures all error_logger and SASL logging events and prints them to the current test case log. If an event can not be associated with a testcase it will be printed in the common test framework log. This will happen for testcases which are run in parallel and events which occur @@ -455,9 +455,9 @@ terminate(State) -> using the normal SASL mechanisms. </cell> </row> <row> - <cell>cth_surefire</cell> - <cell>no</cell> - <cell><p>Captures all test results and outputs them as surefire + <cell align="left">cth_surefire</cell> + <cell align="left">no</cell> + <cell align="left"><p>Captures all test results and outputs them as surefire XML into a file. The file which is created is by default called junit_report.xml. The file name can be changed by setting the <c>path</c> option for this hook, e.g.</p> @@ -467,13 +467,14 @@ terminate(State) -> <p>If the <c>url_base</c> option is set, an additional attribute named <c>url</c> will be added to each <c>testsuite</c> and <c>testcase</c> XML element. The value will - be a constructed from the <c>url_base</c> and a relative path + be constructed from the <c>url_base</c> and a relative path to the test suite or test case log respectively, e.g.</p> - <code>-ct_hooks cth_surefire [{url_base,"http://myserver.com/"}]</code> + <code>-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]</code> <p>will give a url attribute value similar to</p> - <code>"http://myserver.com/[email protected]_11.19.39/x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code> + <code>"http://myserver.com/[email protected]_11.19.39/ +x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code> <p>Surefire XML can for instance be used by Jenkins to display test results.</p></cell> diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml index c6749d6960..0750f560b3 100644 --- a/lib/common_test/doc/src/ct_run.xml +++ b/lib/common_test/doc/src/ct_run.xml @@ -104,6 +104,7 @@ [-silent_connections [ConnType1 ConnType2 .. ConnTypeN]] [-stylesheet CSSFile] [-cover CoverCfgFile] + [-cover_stop Bool] [-event_handler EvHandler1 EvHandler2 .. EvHandlerN] | [-event_handler_init EvHandler1 InitArg1 and EvHandler2 InitArg2 and .. EvHandlerN InitArgN] @@ -138,6 +139,7 @@ [-silent_connections [ConnType1 ConnType2 .. ConnTypeN]] [-stylesheet CSSFile] [-cover CoverCfgFile] + [-cover_stop Bool] [-event_handler EvHandler1 EvHandler2 .. EvHandlerN] | [-event_handler_init EvHandler1 InitArg1 and EvHandler2 InitArg2 and .. EvHandlerN InitArgN] diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index a0b2c96006..d5f5d89e05 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -155,6 +155,8 @@ <item><c><![CDATA[-stylesheet <css_file>]]></c>, points out a user HTML style sheet (see below).</item> <item><c><![CDATA[-cover <cover_cfg_file>]]></c>, to perform code coverage test (see <seealso marker="cover_chapter#cover">Code Coverage Analysis</seealso>).</item> + <item><c><![CDATA[-cover_stop <bool>]]></c>, to specify if the cover tool shall be stopped after the test is completed (see + <seealso marker="cover_chapter#cover_stop">Code Coverage Analysis</seealso>).</item> <item><c><![CDATA[-event_handler <event_handlers>]]></c>, to install <seealso marker="event_handler_chapter#event_handling">event handlers</seealso>.</item> <item><c><![CDATA[-event_handler_init <event_handlers>]]></c>, to install @@ -658,6 +660,9 @@ {cover, CoverSpecFile}. {cover, NodeRefs, CoverSpecFile}. + {cover_stop, Bool}. + {cover_stop, NodeRefs, Bool}. + {include, IncludeDirs}. {include, NodeRefs, IncludeDirs}. @@ -747,7 +752,9 @@ PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc EventHandlers = atom() | [atom()] InitArgs = [term()] - CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}] + CTHModules = [CTHModule | + {CTHModule, CTHInitArgs} | + {CTHModule, CTHInitArgs, CTHPriority}] CTHModule = atom() CTHInitArgs = term() Dir = string() diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index 248d7de8b6..cc8d913994 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -982,38 +982,36 @@ <p>Example:</p> <pre> + Some printouts during test case execution: - Some printouts during test case execution: + io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]), + ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]), + ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]), + ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]), + ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]), + ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]), - io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]), - ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]), - ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]), - ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]), - ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]), - ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]), + If starting the test without specifying any verbosity levels: - If starting the test without specifying any verbosity levels: + $ ct_run ... - $ ct_run ... + the following gets printed: - the following gets printed: - - 1. Standard IO, importance = 50 - 2. Uncategorized, importance = 50 - 3. Categorized info, importance = 50 - 5. Categorized error, importance = 75 - 6. Categorized error, importance = 99 - - If starting the test with: - - $ ct_run -verbosity 1 and info 75 - - the following gets printed: + 1. Standard IO, importance = 50 + 2. Uncategorized, importance = 50 + 3. Categorized info, importance = 50 + 5. Categorized error, importance = 75 + 6. Categorized error, importance = 99 + + If starting the test with: + + $ ct_run -verbosity 1 and info 75 + + the following gets printed: - 3. Categorized info, importance = 50 - 4. Categorized info, importance = 25 - 6. Categorized error, importance = 99 - </pre> + 3. Categorized info, importance = 50 + 4. Categorized info, importance = 25 + 6. Categorized error, importance = 99</pre> <p>How categories can be mapped to CSS tags is documented in the <seealso marker="run_test_chapter#html_stylesheet">Running Tests</seealso> |