diff options
Diffstat (limited to 'lib/common_test/doc/src/cover_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/cover_chapter.xml | 349 |
1 files changed, 172 insertions, 177 deletions
diff --git a/lib/common_test/doc/src/cover_chapter.xml b/lib/common_test/doc/src/cover_chapter.xml index f164fff0ad..094aa7d80c 100644 --- a/lib/common_test/doc/src/cover_chapter.xml +++ b/lib/common_test/doc/src/cover_chapter.xml @@ -33,256 +33,251 @@ <section> <marker id="cover"></marker> <title>General</title> - <p>Although Common Test was created primarly for the purpose of - black box testing, nothing prevents it from working perfectly as - a white box testing tool as well. This is especially true when + <p>Although <c>Common Test</c> was created primarily for + black-box testing, nothing prevents it from working perfectly as + a white-box testing tool as well. This is especially true when the application to test is written in Erlang. Then the test - ports are easily realized by means of Erlang function calls.</p> + ports are easily realized with Erlang function calls.</p> - <p>When white box testing an Erlang application, it is useful to - be able to measure the code coverage of the test. Common Test + <p>When white-box testing an Erlang application, it is useful to + be able to measure the code coverage of the test. <c>Common Test</c> provides simple access to the OTP Cover tool for this - purpose. Common Test handles all necessary communication with - the Cover tool (starting, compiling, analysing, etc). All the - Common Test user needs to do is to specify the extent of the + purpose. <c>Common Test</c> handles all necessary communication with + the Cover tool (starting, compiling, analysing, and so on). + The <c>Common Test</c> user only needs to specify the extent of the code coverage analysis.</p> </section> <section> - <title>Usage</title> - <p>To specify what modules should be included - in the code coverage test, you provide a cover specification - file. Using this file you can point out specific modules or - specify directories that contain modules which should all be - included in the analysis. You can also, in the same fashion, - specify modules that should be excluded from the analysis.</p> + <title>Use</title> + <p>To specify the modules to be included in the code coverage test, + provide a cover specification file. With this file you can point + out specific modules or specify directories containing modules to be + included in the analysis. You can also specify modules to be excluded + from the analysis.</p> <p>If you are testing a distributed Erlang application, it is likely that code you want included in the code coverage analysis - gets executed on an Erlang node other than the one Common Test - is running on. If this is the case you need to specify these - other nodes in the cover specification file or add them - dynamically to the code coverage set of nodes. See the - <c>ct_cover</c> page in the reference manual for details on the - latter.</p> + gets executed on another Erlang node than the one <c>Common Test</c> + is running on. If so, you must specify these other nodes in the + cover specification file or add them dynamically to the code coverage + set of nodes. For details on the latter, see module + <seealso marker="ct_cover"><c>ct_cover</c></seealso>.</p> <p>In the cover specification file you can also specify your required level of the code coverage analysis; <c>details</c> or <c>overview</c>. In detailed mode, you get a coverage overview - page, showing you per module and total coverage percentages, as - well as one HTML file printed for each module included in the - analysis that shows exactly what parts of the code have been + page, showing per module and total coverage percentages. + You also get an HTML file printed for each module included in the + analysis showing exactly what parts of the code have been executed during the test. In overview mode, only the code - coverage overview page gets printed.</p> + coverage overview page is printed.</p> <p>You can choose to export and import code coverage data between tests. If you specify the name of an export file in the cover - specification file, Common Test will export collected coverage - data to this file at the end of the test. You may similarly - specify that previously exported data should be imported and - included in the analysis for a test (you can specify multiple - import files). This way it is possible to analyse total code coverage - without necessarily running all tests at once.</p> - - <p>To activate the code coverage support, you simply specify the - name of the cover specification file as you start Common Test. - This you do either by using the <c>-cover</c> flag with <c>ct_run</c>. - Example:</p> - - <p><c>$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</c></p> + specification file, <c>Common Test</c> exports collected coverage + data to this file at the end of the test. You can similarly + specify previously exported data to be imported and + included in the analysis for a test (multiple import files can be specified). + This way, the total code coverage can be analyzed without necessarily + running all tests at once.</p> + + <p>To activate the code coverage support, specify the name of the cover + specification file as you start <c>Common Test</c>. + Do this by using flag <c>-cover</c> with + <seealso marker="ct_run"><c>ct_run</c></seealso>, + for example:</p> + <pre> + $ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</pre> - <p>You may also pass the cover specification file name in a - call to <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, by adding a <c>{cover,CoverSpec}</c> - tuple to the <c>Opts</c> argument. Also, you can of course - enable code coverage in your test specifications (read - more in the chapter about - <seealso marker="run_test_chapter#test_specifications">using test - specifications</seealso>).</p> + <p>You can also pass the cover specification file name in a + call to <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, + by adding a <c>{cover,CoverSpec}</c> tuple to argument <c>Opts</c>.</p> + <p>You can also enable code coverage in your test specifications (see section + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso> + in section Running Tests and Analyzing Results).</p> </section> <section> <marker id="cover_stop"></marker> - <title>Stopping the cover tool when tests are completed</title> - <p>By default the Cover tool is automatically stopped when the - tests are completed. This causes the original (non cover - compiled) modules to be loaded back in to the test node. If a - process at this point is still running old code of any of the + <title>Stopping the Cover Tool When Tests Are Completed</title> + <p>By default, the Cover tool is automatically stopped when the + tests are completed. This causes the original (non-cover + compiled) modules to be loaded back into the test node. If a + process at this point still runs old code of any of the modules that are cover compiled, meaning that it has not done any fully qualified function call after the cover compilation, - the process will now be killed. To avoid this it is possible to - set the value of the <c>cover_stop</c> option to - <c>false</c>. This means that the modules will stay cover - compiled, and it is therefore only recommended if the erlang - node(s) under test is terminated after the test is completed - or if cover can be manually stopped.</p> - - <p>The option can be set by using the <c>-cover_stop</c> flag with - <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the - Opts argument to <seealso - marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, or by adding - a <c>cover_stop</c> term in your test specification (see chapter - about <seealso - marker="run_test_chapter#test_specifications">test - specifications</seealso>).</p> + the process is killed. To avoid this, set the value of option + <c>cover_stop</c> to <c>false</c>. This means that the + modules stay cover compiled. Therefore, this is only recommended + if the Erlang nodes under test are terminated after the test is + completed, or if cover can be manually stopped.</p> + + <p>The option can be set by using flag <c>-cover_stop</c> with + <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to argument + <c>Opts</c> to + <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, + or by adding a <c>cover_stop</c> term in the test specification (see section + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso> + in section Running Tests and Analyzing Results).</p> </section> <section> - <title>The cover specification file</title> - <p>These are the terms allowed in a cover specification file:</p> + <title>The Cover Specification File</title> + <p>The following terms are allowed in a cover specification file:</p> <pre> - %% List of Nodes on which cover will be active during test. - %% Nodes = [atom()] - {nodes, Nodes}. - - %% Files with previously exported cover data to include in analysis. - %% CoverDataFiles = [string()] - {import, CoverDataFiles}. - - %% Cover data file to export from this session. - %% CoverDataFile = string() - {export, CoverDataFile}. - - %% Cover analysis level. - %% Level = details | overview - {level, Level}. - - %% Directories to include in cover. - %% Dirs = [string()] - {incl_dirs, Dirs}. - - %% Directories, including subdirectories, to include. - {incl_dirs_r, Dirs}. - - %% Specific modules to include in cover. - %% Mods = [atom()] - {incl_mods, Mods}. - - %% Directories to exclude in cover. - {excl_dirs, Dirs}. - - %% Directories, including subdirectories, to exclude. - {excl_dirs_r, Dirs}. - - %% Specific modules to exclude in cover. - {excl_mods, Mods}. - - %% Cross cover compilation - %% Tag = atom(), an identifier for a test run - %% Mod = [atom()], modules to compile for accumulated analysis - {cross,[{Tag,Mods}]}. - </pre> - - <p>The <c>incl_dirs_r</c> and <c>excl_dirs_r</c> terms tell Common - Test to search the given directories recursively and include - or exclude any module found during the search. The - <c>incl_dirs</c> and <c>excl_dirs</c> terms result in a - non-recursive search for modules (i.e. only modules found in - the given directories are included or excluded).</p> - <p><em>Note:</em> Directories containing Erlang modules that are - to be included in a code coverage test must exist in the code - server path, or the cover tool will fail to recompile the modules. - (It is not sufficient to specify these directories in the cover - specification file for Common Test).</p> + %% List of Nodes on which cover will be active during test. + %% Nodes = [atom()] + {nodes, Nodes}. + + %% Files with previously exported cover data to include in analysis. + %% CoverDataFiles = [string()] + {import, CoverDataFiles}. + + %% Cover data file to export from this session. + %% CoverDataFile = string() + {export, CoverDataFile}. + + %% Cover analysis level. + %% Level = details | overview + {level, Level}. + + %% Directories to include in cover. + %% Dirs = [string()] + {incl_dirs, Dirs}. + + %% Directories, including subdirectories, to include. + {incl_dirs_r, Dirs}. + + %% Specific modules to include in cover. + %% Mods = [atom()] + {incl_mods, Mods}. + + %% Directories to exclude in cover. + {excl_dirs, Dirs}. + + %% Directories, including subdirectories, to exclude. + {excl_dirs_r, Dirs}. + + %% Specific modules to exclude in cover. + {excl_mods, Mods}. + + %% Cross cover compilation + %% Tag = atom(), an identifier for a test run + %% Mod = [atom()], modules to compile for accumulated analysis + {cross,[{Tag,Mods}]}.</pre> + + <p>The terms <c>incl_dirs_r</c> and <c>excl_dirs_r</c> tell <c>Common + Test</c> to search the specified directories recursively and include + or exclude any module found during the search. The terms + <c>incl_dirs</c> and <c>excl_dirs</c> result in a + non-recursive search for modules (that is, only modules found in + the specified directories are included or excluded).</p> + <note><p>Directories containing Erlang modules to be included in a code + coverage test must exist in the code server path. Otherwise, + the Cover tool fails to recompile the modules. It is not sufficient to + specify these directories in the cover specification file for + <c>Common Test</c>.</p></note> </section> <section> <marker id="cross_cover"/> - <title>Cross cover analysis</title> + <title>Cross Cover Analysis</title> <p>The cross cover mechanism allows cover analysis of modules - across multiple tests. It is useful if some code, e.g. a library - module, is used by many different tests and the accumulated cover - result is desirable.</p> + across multiple tests. It is useful if some code, for example, a + library module, is used by many different tests and the accumulated + cover result is desirable.</p> - <p>This can of course also be achieved in a more customized way by - using the <c>export</c> parameter in the cover specification and - analysing the result off line, but the cross cover mechanism is a - build in solution which also provides the logging.</p> + <p>This can also be achieved in a more customized way by + using parameter <c>export</c> in the cover specification and + analysing the result off line. However, the cross cover mechanism is a + built-in solution that also provides logging.</p> - <p>The mechanism is easiest explained via an example:</p> + <p>The mechanism is easiest explained by an example:</p> - <p>Let's say that there are two systems, <c>s1</c> and <c>s2</c>, - which are tested in separate test runs. System <c>s1</c> contains - a library module <c>m1</c> which is tested by the <c>s1</c> test - run and is included in <c>s1</c>'s cover specification:</p> + <p>Assume that there are two systems, <c>s1</c> and <c>s2</c>, + that are tested in separate test runs. System <c>s1</c> contains + a library module <c>m1</c> tested by test run <c>s1</c> and + is included in the cover specification of <c>s1</c> as follows:</p> <code type="none"> -s1.cover: - {incl_mods,[m1]}.</code> + s1.cover: + {incl_mods,[m1]}.</code> <p>When analysing code coverage, the result for <c>m1</c> can be seen in the cover log in the <c>s1</c> test result.</p> - <p>Now, let's imagine that since <c>m1</c> is a library module, it - is also used quite a bit by system <c>s2</c>. The <c>s2</c> test - run does not specifically test <c>m1</c>, but it might still be - interesting to see which parts of <c>m1</c> is actually covered by - the <c>s2</c> tests. To do this, <c>m1</c> could be included also - in <c>s2</c>'s cover specification:</p> + <p>Now, imagine that as <c>m1</c> is a library module, it + is also often used by system <c>s2</c>. Test run <c>s2</c> + does not specifically test <c>m1</c>, but it can still be + interesting to see which parts of <c>m1</c> that are covered + by the <c>s2</c> tests. To do this, <c>m1</c> can be included also + in the cover specification of <c>s2</c> as follows:</p> <code type="none"> -s2.cover: - {incl_mods,[m1]}.</code> + s2.cover: + {incl_mods,[m1]}.</code> - <p>This would give an entry for <c>m1</c> also in the cover log - for the <c>s2</c> test run. The problem is that this would only - reflect the coverage by <c>s2</c> tests, not the accumulated - result over <c>s1</c> and <c>s2</c>. And this is where the cross + <p>This gives an entry for <c>m1</c> also in the cover log + for test run <c>s2</c>. The problem is that this only + reflects the coverage by <c>s2</c> tests, not the accumulated + result over <c>s1</c> and <c>s2</c>. This is where the cross cover mechanism comes in handy.</p> - <p>If instead the cover specification for <c>s2</c> was like - this:</p> + <p>If instead the cover specification for <c>s2</c> is like + the following:</p> <code type="none"> -s2.cover: - {cross,[{s1,[m1]}]}.</code> + s2.cover: + {cross,[{s1,[m1]}]}.</code> - <p>then <c>m1</c> would be cover compiled in the <c>s2</c> test - run, but not shown in the coverage log. Instead, if - <c>ct_cover:cross_cover_analyse/2</c> is called after both - <c>s1</c> and <c>s2</c> test runs are completed, the accumulated - result for <c>m1</c> would be available in the cross cover log for - the <c>s1</c> test run.</p> + <p>Then <c>m1</c> is cover compiled in test run <c>s2</c>, + but not shown in the coverage log. Instead, if + <seealso marker="ct_cover#cross_cover_analyse-2"><c>ct_cover:cross_cover_analyse/2</c></seealso> + is called after both <c>s1</c> and <c>s2</c> test runs are completed, + the accumulated result for <c>m1</c> is available in the cross cover + log for test run <c>s1</c>.</p> - <p>The call to the analyse function must be like this:</p> + <p>The call to the analyze function must be as follows:</p> <code type="none"> -ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code> + ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code> - <p>where <c>S1LogDir</c> and <c>S2LogDir</c> are the directories + <p>Here, <c>S1LogDir</c> and <c>S2LogDir</c> are the directories named <c><TestName>.logs</c> for each test respectively.</p> - <p>Note the tags <c>s1</c> and <c>s2</c> which are used in the + <p>Notice the tags <c>s1</c> and <c>s2</c>, which are used in the cover specification file and in the call to - <c>ct_cover:cross_cover_analyse/2</c>. The point of these are only + <c>ct_cover:cross_cover_analyse/2</c>. The purpose of these is only to map the modules specified in the cover specification to the log - directory specified in the call to the analyse function. The name - of the tag has no meaning beyond this.</p> + directory specified in the call to the analyze function. The tag name + has no meaning beyond this.</p> </section> <section> <title>Logging</title> <p>To view the result of a code coverage test, click the button - labled "COVER LOG" in the top level index page for the test run.</p> + labeled "COVER LOG" in the top-level index page for the test run.</p> - <p>Prior to Erlang/OTP 17.1, if your test run consisted of + <p>Before Erlang/OTP 17.1, if your test run consisted of multiple tests, cover would be started and stopped for each test - within the test run. Separate logs would be available via the + within the test run. Separate logs would be available through the "Coverage log" link on the test suite result pages. These links are still available, but now they all point to the same page as - the button on the top level index page. The log contains the - accumulated results for the complete test run. See the release - notes for more information about this change.</p> + the button on the top-level index page. The log contains the + accumulated results for the complete test run. For details about + this change, see the release notes.</p> - <p>The buttonc takes you to the code coverage overview page. If you - have successfully performed a detailed coverage analysis, you - find links to each individual module coverage page here.</p> + <p>The button takes you to the code coverage overview page. If you + have successfully performed a detailed coverage analysis, + links to each individual module coverage page are found here.</p> - <p>If cross cover analysis has been performed, and there are - accumulated coverage results for the current test, then the - - "Coverdata collected over all tests" link will take you to these + <p>If cross cover analysis is performed, and there are + accumulated coverage results for the current test, the link + "Coverdata collected over all tests" takes you to these results.</p> </section> |