aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src/basics_chapter.xml
blob: ff6ea6c557696d16bd1f5c7e1edf7c761fdcd766 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2003</year><year>2012</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.
    
      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.
    
    </legalnotice>

    <title>Common Test Basics</title>
    <prepared>Kenneth Lundin, Peter Andersson</prepared>
    <docno></docno>
    <date>2003-10-21</date>
    <rev></rev>
    <file>basics_chapter.xml</file>
  </header>
  <marker id="basics"></marker>
  <section>
    <title>Introduction</title>

    <p>The <em>Common Test</em> framework (CT) is a tool which supports
      implementation and automated execution of test cases towards arbitrary
      types of target systems. The CT framework is based on the OTP Test
      Server and it's the main tool being used in all testing- and verification
      activities that are part of Erlang/OTP system development- and maintenance.
    </p>

    <p>Test cases can be executed individually or in batches. Common Test
      also features a distributed testing mode with central control and logging
      (a feature that makes it possible to test multiple systems independently in
      one common session, useful e.g. for running automated large-scale regression
      tests).
    </p>

    <p>
      The SUT (System Under Test) may consist of one or several target
      nodes. CT contains a generic test server which, together with
      other test utilities, is used to perform test case execution. 
      It is possible to start the tests from a GUI or from the OS- or
      Erlang shell. <em>Test suites</em> are files (Erlang
      modules) that contain the <em>test cases</em> (Erlang functions)
      to be executed. <em>Support modules</em> provide functions
      that the test cases utilize in order to carry out the tests.
    </p>
    
    <p>In a black-box testing scenario, CT based test programs connect to
      the target system(s) via standard O&amp;M and CLI protocols. CT
      provides implementations of, and wrapper interfaces to, some of these
      protocols (most of which exist as stand-alone components and
      applications in OTP). The wrappers simplify configuration and add
      verbosity for logging purposes. CT will be continously extended with
      useful support modules. (Note however that it's
      a straightforward task to use any arbitrary Erlang/OTP component
      for testing purposes with Common Test, without needing a CT wrapper
      for it. It's as simple as calling Erlang functions). There
      are a number of target independent interfaces supported in CT, such as
      Generic Telnet, FTP, etc, which can be specialized or used
      directly for controlling instruments, traffic load generators, etc.
    </p> 
    
    <p>Common Test is also a very useful tool for white-box testing Erlang
      code (e.g. module testing), since the test programs can call exported Erlang
      functions directly and there's very little overhead required for
      implementing basic test suites and executing simple tests. For black-box
      testing Erlang software, Erlang RPC as well as standard O&amp;M interfaces
      can for example be used.
    </p>
    
    <p>A test case can handle several connections towards one or
      several target systems, instruments and traffic generators in
      parallel in order to perform the necessary actions for a
      test. The handling of many connections in parallel is one of
      the major strengths of Common Test (thanks to the efficient
      support for concurrency in the Erlang runtime system - which CT users
      can take great advantage of!).
    </p>
  </section>

  <section>
    <title>Test Suite Organisation</title>
    <p>
      The test suites are organized in test directories and each test suite
      may have a separate data directory. Typically, these files and directories
      are version controlled similarly to other forms of source code (possibly by
      means of a version control system like GIT or Subversion). However, CT does 
      not itself put any requirements on (or has any form of awareness of) 
      possible file and directory versions.
    </p>
  </section>

  <section>
    <title>Support Libraries</title>
    <p>
      Support libraries contain functions that are useful for all test suites,
      or for test suites in a specific functional area or subsystem.
      In addition to the general support libraries provided by the
      CT framework, and the various libraries and applications provided by
      Erlang/OTP, there might also be a need for customized (user specific) 
      support libraries. 
    </p>
  </section>

  <section>
    <title>Suites and Test Cases</title>
    <p>
      Testing is performed by running test suites (sets of test cases) or 
      individual test cases. A test suite is implemented as an Erlang module named 
      <c><![CDATA[<suite_name>_SUITE.erl]]></c> which contains a number of test cases.
      A test case is an Erlang function which tests one or more things. 
      The test case is the smallest unit that the CT test server deals with.
    </p>
    <p>
      Subsets of test cases, called test case groups, may also be defined. A test case 
      group can have execution properties associated with it. Execution properties 
      specify whether the test cases in the group should be executed in
      random order, in parallel, in sequence, and if the execution of the group 
      should be repeated. Test case groups may also be nested (i.e. a group may,
      besides test cases, contain sub-groups).
    </p>
    <p>
      Besides test cases and groups, the test suite may also contain configuration 
      functions. These functions are meant to be used for setting up (and verifying)
      environment and state on the SUT (and/or the CT host node), required for 
      the tests to execute correctly. Examples of operations: Opening a connection
      to the SUT, initializing a database, running an installation script, etc.
      Configuration may be performed per suite, per test case group and per 
      individual test case.
    </p>
    <p>
      The test suite module must conform to a
      <seealso marker="common_test">callback interface</seealso>
      specified by the CT test server. See the 
      <seealso marker="write_test_chapter#intro">Writing Test Suites</seealso> chapter
      for more information.
    </p>

    <p>
      A test case is considered successful if it returns to the caller, no matter 
      what the returned value is. A few return values have special meaning however
      (such as <c>{skip,Reason}</c> which indicates that the test case is skipped,
      <c>{comment,Comment}</c> which prints a comment in the log for the test case and 
      <c>{save_config,Config}</c> which makes the CT test server pass <c>Config</c> to 
      the next test case). 
      A test case failure is specified as a runtime error (a crash), no matter what 
      the reason for termination is. If you use Erlang pattern matching effectively,
      you can take advantage of this property. The result will be concise and 
      readable test case functions that look much more like scripts than actual programs. 
      Simple example:
    </p>
    <pre>
      session(_Config) ->
          {started,ServerId} = my_server:start(),
          {clients,[]} = my_server:get_clients(ServerId),
          MyId = self(),
          connected = my_server:connect(ServerId, MyId),
          {clients,[MyId]} = my_server:get_clients(ServerId),
          disconnected = my_server:disconnect(ServerId, MyId),
          {clients,[]} = my_server:get_clients(ServerId),
          stopped = my_server:stop(ServerId).
    </pre>
    <p>
      As a test suite runs, all information (including output to <c>stdout</c>) is 
      recorded in several different log files. A minimum of information is displayed 
      in the user console (only start and stop information, plus a note 
      for each failed test case).
    </p>
    <p>
      The result from each test case is recorded in a dedicated HTML log file, created 
      for the particular test run. An overview page displays each test case represented 
      by row in a table showing total execution time, whether the case was successful,
      failed or skipped, plus an optional user comment. (For a failed test case, the 
      reason for termination is also printed in the comment field). The overview page
      has a link to each test case log file, providing simple navigation with any standard
      HTML browser.
    </p>
    </section>

  <section>
    <title>External Interfaces</title>
    <p>
      The CT test server requires that the test suite defines and exports the 
      following mandatory or optional callback functions:
    </p>
    
    <taglist>
      <tag>all()</tag>
        <item>Returns a list of all test cases and groups in the suite. (Mandatory)</item>
      <tag>suite()</tag>
        <item>Info function used to return properties for the suite. (Optional)</item>
      <tag>groups()</tag>
        <item>For declaring test case groups. (Optional)</item>
      <tag>init_per_suite(Config)</tag>
        <item>Suite level configuration function, executed before the first 
	test case. (Optional)</item>
      <tag>end_per_suite(Config)</tag>
        <item>Suite level configuration function, executed after the last 
	test case. (Optional)</item>
      <tag>group(GroupName)</tag>
        <item>Info function used to return properties for a test case group. (Optional)</item>	
      <tag>init_per_group(GroupName, Config)</tag>
        <item>Configuration function for a group, executed before the first 
	test case. (Optional)</item>
      <tag>end_per_group(GroupName, Config)</tag>
        <item>Configuration function for a group, executed after the last 
	test case. (Optional)</item>
      <tag>init_per_testcase(TestCase, Config)</tag>
        <item>Configuration function for a testcase, executed before each 
	test case. (Optional)</item>
      <tag>end_per_testcase(TestCase, Config)</tag>
      <item>Configuration function for a testcase, executed after each 
      test case. (Optional)</item> 
    </taglist>
      <p>
	For each test case the CT test server expects these functions:
      </p>
      <taglist>
	<tag>Testcasename()</tag>
	  <item>Info function that returns a list of test case properties. (Optional)</item>
	<tag>Testcasename(Config)</tag> 
	  <item>The actual test case function.</item> 
      </taglist>
    </section>
</chapter>