1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
|
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">
<chapter>
<header>
<copyright>
<year>2006</year><year>2012</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
The contents of this file are subject to the Erlang Public License,
Version 1.1, (the "License"); you may not use this file except in
compliance with the License. You should have received a copy of the
Erlang Public License along with this software. If not, it can be
retrieved online at http://www.erlang.org/.
Software distributed under the License is distributed on an "AS IS"
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
the License for the specific language governing rights and limitations
under the License.
</legalnotice>
<title>Code Coverage Analysis</title>
<prepared>Peter Andersson</prepared>
<docno></docno>
<date></date>
<rev></rev>
<file>cover_chapter.xml</file>
</header>
<section>
<marker id="cover"></marker>
<title>General</title>
<p>Although Common Test was created primarly for the purpose of
black box testing, nothing prevents it from working perfectly as
a white box testing tool as well. This is especially true when
the application to test is written in Erlang. Then the test
ports are easily realized by means of Erlang function calls.</p>
<p>When white box testing an Erlang application, it is useful to
be able to measure the code coverage of the test. Common Test
provides simple access to the OTP Cover tool for this
purpose. Common Test handles all necessary communication with
the Cover tool (starting, compiling, analysing, etc). All the
Common Test user needs to do is to specify the extent of the
code coverage analysis.</p>
</section>
<section>
<title>Usage</title>
<p>To specify what modules should be included
in the code coverage test, you provide a cover specification
file. Using this file you can point out specific modules or
specify directories that contain modules which should all be
included in the analysis. You can also, in the same fashion,
specify modules that should be excluded from the analysis.</p>
<p>If you are testing a distributed Erlang application, it is
likely that code you want included in the code coverage analysis
gets executed on an Erlang node other than the one Common Test
is running on. If this is the case you need to specify these
other nodes in the cover specification file or add them
dynamically to the code coverage set of nodes. See the
<c>ct_cover</c> page in the reference manual for details on the
latter.</p>
<p>In the cover specification file you can also specify your
required level of the code coverage analysis; <c>details</c> or
<c>overview</c>. In detailed mode, you get a coverage overview
page, showing you per module and total coverage percentages, as
well as one HTML file printed for each module included in the
analysis that shows exactly what parts of the code have been
executed during the test. In overview mode, only the code
coverage overview page gets printed.</p>
<p><em>Note:</em> Currently, for Common Test to be able to print
code coverage HTML files for the modules included in the
analysis, the source code files of these modules must be
located in the same directory as the corresponding <c>.beam</c>
files. This is a limitation that will be removed later.</p>
<p>You can choose to export and import code coverage data between
tests. If you specify the name of an export file in the cover
specification file, Common Test will export collected coverage
data to this file at the end of the test. You may similarly
specify that previously exported data should be imported and
included in the analysis for a test (you can specify multiple
import files). This way it is possible to analyse total code coverage
without necessarily running all tests at once. Note that even if
you run separate tests in one test run, code coverage data will
not be passed on from one test to another unless you specify an
export file for Common Test to use for this purpose.</p>
<p>To activate the code coverage support, you simply specify the
name of the cover specification file as you start Common Test.
This you do either by using the <c>-cover</c> flag with <c>ct_run</c>.
Example:</p>
<p><c>$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</c></p>
<p>You may also pass the cover specification file name in a
call to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, by adding a <c>{cover,CoverSpec}</c>
tuple to the <c>Opts</c> argument. Also, you can of course
enable code coverage in your test specifications (read
more in the chapter about
<seealso marker="run_test_chapter#test_specifications">using test
specifications</seealso>).</p>
</section>
<section>
<marker id="cover_stop"></marker>
<title>Stopping the cover tool when tests are completed</title>
<p>By default the Cover tool is automatically stopped when the
tests are completed. This causes the original (non cover
compiled) modules to be loaded back in to the test node. If a
process at this point is still running old code of any of the
modules that are cover compiled, meaning that it has not done
any fully qualified function call after the cover compilation,
the process will now be killed. To avoid this it is possible to
set the value of the <c>cover_stop</c> option to
<c>false</c>. This means that the modules will stay cover
compiled, and it is therefore only recommended if the erlang
node(s) under test is terminated after the test is completed
or if cover can be manually stopped.</p>
<p>The option can be set by using the <c>-cover_stop</c> flag with
<c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the
Opts argument to <c><seealso
marker="ct#run_test-1">ct:run_test/1</seealso></c>, or by adding
a <c>cover_stop</c> term in your test specification (see chapter
about <seealso
marker="run_test_chapter#test_specifications">test
specifications</seealso>).</p>
</section>
<section>
<title>The cover specification file</title>
<p>These are the terms allowed in a cover specification file:</p>
<pre>
%% List of Nodes on which cover will be active during test.
%% Nodes = [atom()]
{nodes, Nodes}.
%% Files with previously exported cover data to include in analysis.
%% CoverDataFiles = [string()]
{import, CoverDataFiles}.
%% Cover data file to export from this session.
%% CoverDataFile = string()
{export, CoverDataFile}.
%% Cover analysis level.
%% Level = details | overview
{level, Level}.
%% Directories to include in cover.
%% Dirs = [string()]
{incl_dirs, Dirs}.
%% Directories, including subdirectories, to include.
{incl_dirs_r, Dirs}.
%% Specific modules to include in cover.
%% Mods = [atom()]
{incl_mods, Mods}.
%% Directories to exclude in cover.
{excl_dirs, Dirs}.
%% Directories, including subdirectories, to exclude.
{excl_dirs_r, Dirs}.
%% Specific modules to exclude in cover.
{excl_mods, Mods}.
</pre>
<p>The <c>incl_dirs_r</c> and <c>excl_dirs_r</c> terms tell Common
Test to search the given directories recursively and include
or exclude any module found during the search. The
<c>incl_dirs</c> and <c>excl_dirs</c> terms result in a
non-recursive search for modules (i.e. only modules found in
the given directories are included or excluded).</p>
<p><em>Note:</em> Directories containing Erlang modules that are
to be included in a code coverage test must exist in the code
server path, or the cover tool will fail to recompile the modules.
(It is not sufficient to specify these directories in the cover
specification file for Common Test).</p>
</section>
<section>
<title>Logging</title>
<p>To view the result of a code coverage test, follow the
"Coverage log" link on the test suite results page. This
takes you to the code coverage overview page. If you have
successfully performed a detailed coverage analysis, you
find links to each individual module coverage page here.</p>
</section>
</chapter>
|
