path: root/lib/common_test/doc/src/cover_chapter.xml

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2006</year><year>2014</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at
 
          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
    
    </legalnotice>

    <title>Code Coverage Analysis</title>
    <prepared>Peter Andersson</prepared>
     <docno></docno>
     <date></date>
     <rev></rev>
    <file>cover_chapter.xml</file>
  </header>
  
  <section>
    <marker id="cover"></marker>
      <title>General</title>
      <p>Although Common Test was created primarly for the purpose of
	black box testing, nothing prevents it from working perfectly as
	a white box testing tool as well.  This is especially true when
	the application to test is written in Erlang. Then the test
	ports are easily realized by means of Erlang function calls.</p>
      
      <p>When white box testing an Erlang application, it is useful to
	be able to measure the code coverage of the test. Common Test
	provides simple access to the OTP Cover tool for this
	purpose. Common Test handles all necessary communication with
	the Cover tool (starting, compiling, analysing, etc). All the
	Common Test user needs to do is to specify the extent of the
	code coverage analysis.</p>
  </section>
  
  <section>
    <title>Usage</title> 
    <p>To specify what modules should be included
    in the code coverage test, you provide a cover specification
    file. Using this file you can point out specific modules or
    specify directories that contain modules which should all be
    included in the analysis. You can also, in the same fashion,
    specify modules that should be excluded from the analysis.</p>
    
    <p>If you are testing a distributed Erlang application, it is
      likely that code you want included in the code coverage analysis
      gets executed on an Erlang node other than the one Common Test
      is running on. If this is the case you need to specify these
      other nodes in the cover specification file or add them
      dynamically to the code coverage set of nodes. See the
      <c>ct_cover</c> page in the reference manual for details on the
      latter.</p>

    <p>In the cover specification file you can also specify your
      required level of the code coverage analysis; <c>details</c> or
      <c>overview</c>. In detailed mode, you get a coverage overview
      page, showing you per module and total coverage percentages, as
      well as one HTML file printed for each module included in the
      analysis that shows exactly what parts of the code have been
      executed during the test. In overview mode, only the code
      coverage overview page gets printed.</p>

    <p>You can choose to export and import code coverage data between
      tests. If you specify the name of an export file in the cover
      specification file, Common Test will export collected coverage
      data to this file at the end of the test. You may similarly
      specify that previously exported data should be imported and
      included in the analysis for a test (you can specify multiple
      import files). This way it is possible to analyse total code coverage
      without necessarily running all tests at once.</p>

    <p>To activate the code coverage support, you simply specify the
      name of the cover specification file as you start Common Test.
      This you do either by using the <c>-cover</c> flag with <c>ct_run</c>.
      Example:</p>

    <p><c>$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</c></p>
    
    <p>You may also pass the cover specification file name in a
      call to <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, by adding a <c>{cover,CoverSpec}</c>
      tuple to the <c>Opts</c> argument. Also, you can of course 
      enable code coverage in your test specifications (read
      more in the chapter about 
      <seealso marker="run_test_chapter#test_specifications">using test
      specifications</seealso>).</p>
  </section>

  <section>
  <marker id="cover_stop"></marker>
    <title>Stopping the cover tool when tests are completed</title>
    <p>By default the Cover tool is automatically stopped when the
      tests are completed. This causes the original (non cover
      compiled) modules to be loaded back in to the test node. If a
      process at this point is still running old code of any of the
      modules that are cover compiled, meaning that it has not done
      any fully qualified function call after the cover compilation,
      the process will now be killed. To avoid this it is possible to
      set the value of the <c>cover_stop</c> option to
      <c>false</c>. This means that the modules will stay cover
      compiled, and it is therefore only recommended if the erlang
      node(s) under test is terminated after the test is completed
      or if cover can be manually stopped.</p>

    <p>The option can be set by using the <c>-cover_stop</c> flag with
      <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the
      Opts argument to <seealso
      marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, or by adding
      a <c>cover_stop</c> term in your test specification (see chapter
      about <seealso
      marker="run_test_chapter#test_specifications">test
      specifications</seealso>).</p>

  </section>

  <section>
    <title>The cover specification file</title>
    <p>These are the terms allowed in a cover specification file:</p>

    <pre>
      %% List of Nodes on which cover will be active during test.
      %% Nodes = [atom()]
      {nodes, Nodes}.       

      %% Files with previously exported cover data to include in analysis.
      %% CoverDataFiles = [string()]
      {import, CoverDataFiles}.

      %% Cover data file to export from this session.
      %% CoverDataFile = string()
      {export, CoverDataFile}.

      %% Cover analysis level.
      %% Level = details | overview
      {level, Level}.       

      %% Directories to include in cover.
      %% Dirs = [string()]
      {incl_dirs, Dirs}.

      %% Directories, including subdirectories, to include.
      {incl_dirs_r, Dirs}.

      %% Specific modules to include in cover.
      %% Mods = [atom()]
      {incl_mods, Mods}.

      %% Directories to exclude in cover.
      {excl_dirs, Dirs}.

      %% Directories, including subdirectories, to exclude.
      {excl_dirs_r, Dirs}.

      %% Specific modules to exclude in cover.
      {excl_mods, Mods}.

      %% Cross cover compilation
      %% Tag = atom(), an identifier for a test run
      %% Mod = [atom()], modules to compile for accumulated analysis
      {cross,[{Tag,Mods}]}.
    </pre>

    <p>The <c>incl_dirs_r</c> and <c>excl_dirs_r</c> terms tell Common
      Test to search the given directories recursively and include 
      or exclude any module found during the search. The
      <c>incl_dirs</c> and <c>excl_dirs</c> terms result in a
      non-recursive search for modules (i.e. only modules found in 
      the given directories are included or excluded).</p>
    <p><em>Note:</em> Directories containing Erlang modules that are
       to be included in a code coverage test must exist in the code
       server path, or the cover tool will fail to recompile the modules.
       (It is not sufficient to specify these directories in the cover 
       specification file for Common Test).</p>
  </section>

  <section>
  <marker id="cross_cover"/>
    <title>Cross cover analysis</title>
    <p>The cross cover mechanism allows cover analysis of modules
    across multiple tests. It is useful if some code, e.g. a library
    module, is used by many different tests and the accumulated cover
    result is desirable.</p>

    <p>This can of course also be achieved in a more customized way by
    using the <c>export</c> parameter in the cover specification and
    analysing the result off line, but the cross cover mechanism is a
    build in solution which also provides the logging.</p>

    <p>The mechanism is easiest explained via an example:</p>

    <p>Let's say that there are two systems, <c>s1</c> and <c>s2</c>,
    which are tested in separate test runs. System <c>s1</c> contains
    a library module <c>m1</c> which is tested by the <c>s1</c> test
    run and is included in <c>s1</c>'s cover specification:</p>

<code type="none">
s1.cover:
  {incl_mods,[m1]}.</code>

    <p>When analysing code coverage, the result for <c>m1</c> can be
    seen in the cover log in the <c>s1</c> test result.</p>

    <p>Now, let's imagine that since <c>m1</c> is a library module, it
    is also used quite a bit by system <c>s2</c>. The <c>s2</c> test
    run does not specifically test <c>m1</c>, but it might still be
    interesting to see which parts of <c>m1</c> is actually covered by
    the <c>s2</c> tests. To do this, <c>m1</c> could be included also
    in <c>s2</c>'s cover specification:</p>

<code type="none">
s2.cover:
  {incl_mods,[m1]}.</code>

    <p>This would give an entry for <c>m1</c> also in the cover log
    for the <c>s2</c> test run. The problem is that this would only
    reflect the coverage by <c>s2</c> tests, not the accumulated
    result over <c>s1</c> and <c>s2</c>. And this is where the cross
    cover mechanism comes in handy.</p>

    <p>If instead the cover specification for <c>s2</c> was like
    this:</p>

<code type="none">
s2.cover:
  {cross,[{s1,[m1]}]}.</code>

    <p>then <c>m1</c> would be cover compiled in the <c>s2</c> test
    run, but not shown in the coverage log. Instead, if
    <c>ct_cover:cross_cover_analyse/2</c> is called after both
    <c>s1</c> and <c>s2</c> test runs are completed, the accumulated
    result for <c>m1</c> would be available in the cross cover log for
    the <c>s1</c> test run.</p>

    <p>The call to the analyse function must be like this:</p>

<code type="none">
ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code>

    <p>where <c>S1LogDir</c> and <c>S2LogDir</c> are the directories
    named <c>&lt;TestName&gt;.logs</c> for each test respectively.</p>

    <p>Note the tags <c>s1</c> and <c>s2</c> which are used in the
    cover specification file and in the call to
    <c>ct_cover:cross_cover_analyse/2</c>. The point of these are only
    to map the modules specified in the cover specification to the log
    directory specified in the call to the analyse function. The name
    of the tag has no meaning beyond this.</p>

  </section>

  <section>
    <title>Logging</title>
    <p>To view the result of a code coverage test, click the button
      labled "COVER LOG" in the top level index page for the test run.</p>

    <p>Prior to Erlang/OTP 17.1, if your test run consisted of
      multiple tests, cover would be started and stopped for each test
      within the test run. Separate logs would be available via the
      "Coverage log" link on the test suite result pages. These links
      are still available, but now they all point to the same page as
      the button on the top level index page. The log contains the
      accumulated results for the complete test run. See the release
      notes for more information about this change.</p>

    <p>The buttonc takes you to the code coverage overview page. If you
      have successfully performed a detailed coverage analysis, you
      find links to each individual module coverage page here.</p>

    <p>If cross cover analysis has been performed, and there are
      accumulated coverage results for the current test, then the -
      "Coverdata collected over all tests" link will take you to these
      results.</p>
  </section>

</chapter>