aboutsummaryrefslogtreecommitdiffstats
path: root/lib/test_server/doc/src/write_framework_chapter.xml
blob: 8a20e9afec5f795febd1ad1050a61d4f510b9b82 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2002</year><year>2009</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.
    
      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.
    
    </legalnotice>

    <title>Write you own test server framework</title>
    <prepared>Siri Hansen</prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>write_framework_chapter.xml</file>
  </header>

  <section>
    <title>Introduction</title>
    <p>The test server controller can be interfaced from the operating
      system or from within Erlang. The nature of your new framework
      will decide which interface to use. If you want your framework to
      start a new node for each test, the operating system interface is
      very convenient. If your node is already started, going from
      within Erlang might be a more flexible solution.
      </p>
    <p>The two methods are described below.
      </p>
  </section>

  <section>
    <title>Interfacing the test server controller from Erlang</title>
    <p>Using the test server from Erlang means that you have to start
      the test server and then add test jobs. Use
      <c>test_server_ctrl:start/0</c> to start a local target or
      <c>test_server_ctrl:start/1</c> to start a remote target. The test
      server is stopped by <c>test_server_ctrl:stop/0</c>.
      </p>
    <p>The argument to <c>test_server_ctrl:start/1</c> is the name of a
      parameter file. The parameter file specifies what type of target
      to start and where to start it, as well as some additional
      parameters needed for different target types. See the reference
      manual for a detailed description of all valid parameters.
      </p>

    <section>
      <title>Adding test jobs</title>
      <p>There are many commands available for adding test cases to
        the test server's job queue:        <br></br>
</p>
      <list type="bulleted">
        <item>Single test case        <br></br>
<c>test_server_ctrl:add_case/2/3</c></item>
        <item>Multiple test cases from same suite        <br></br>
<c>test_server_ctrl:add_cases/2/3</c></item>
        <item>Test suite module or modules        <br></br>
<c>test_server_ctrl:add_module/1/2</c></item>
        <item>Some or all test suite modules in a directory        <br></br>
<c>test_server_ctrl:add_dir/2/3</c></item>
        <item>Test cases specified in a test specification file        <br></br>
<c>test_server_ctrl:add_spec/1</c></item>
      </list>
      <p>All test suites are given a unique name, which is usually
        given when the test suite is added to the job queue. In some
        cases, a default name is used, as in the case when a module is
        added without a specified name. The test job name is used to
        store logfiles, which are stored in the `name.logs' directory
        under the current directory.
        </p>
      <p>See the reference manual for details about the functions for
        adding test jobs.
        </p>
    </section>
  </section>

  <section>
    <title>Interfacing the test server controller from the operating system.</title>
    <p>The function <c>run_test/1</c> is your interface in the test
      server controller if you wish to use it from the operating
      system. You simply start an erlang shell and invoke this function
      with the <c>-s</c> option. <c>run_test/1</c> starts the test
      server, runs the test specified by the command line and stops the
      test server. The argument to <c>run_test/1</c> is a list of
      command line flags, typically
      <c>['KEY1', Value1, 'KEY2', Value2, ...]</c>. 
      The valid command line flags are listed in the reference manual
      for <c>test_server_ctrl</c>.
      </p>
    <p>A typical command line may look like this      <br></br>
<c>erl -noshell -s test_server_ctrl run_test KEY1 Value1 KEY2 Value2 ... -s erlang halt</c></p>
    <p>Or make an alias (this is for unix/tcsh)      <br></br>
<c>alias erl_test 'erl -noshell -s test_server_ctrl run_test \!* -s erlang halt'</c></p>
    <p>And then use it like this      <br></br>
<c>erl_test KEY1 Value1 KEY2 Value2 ...</c>      <br></br>
</p>

    <section>
      <title>An Example</title>
      <p>An example of starting a test run from the command line        <br></br>
</p>
      <p><c>erl -name test_srv -noshell -rsh /home/super/otp/bin/ctrsh </c>        <br></br>
<c>-pa /clearcase/otp/erts/lib/kernel/test </c>        <br></br>
<c>-boot start_sasl -sasl errlog_type error </c>        <br></br>
<c>-s test_server_ctrl run_test SPEC kernel.spec -s erlang halt</c>        <br></br>
</p>
    </section>
  </section>

  <section>
    <title>Framework callback functions</title>
    <p>By defining the environment variable
      <c>TEST_SERVER_FRAMEWORK</c> to a module name, the framework
      callback functions can be used. The framework callback functions
      are called by the test server in order let the framework interact
      with the execution of the tests and to keep the framework upto
      date with information about the test progress.
      </p>
    <p>The framework callback functions are described in the reference
      manual for <c>test_server_ctrl</c>.
      </p>
    <p>Note that this topic is in an early stage of development, and
      changes might occur.
      </p>
  </section>

  <section>
    <title>Other concerns</title>
    <p>Some things to think about when writing you own test server
      framework:
      </p>
    <list type="bulleted">
      <item><c>emulator version</c> - Make sure that the intended
       version of the emulator is started.
      </item>
      <item><c>operating system path</c> - If test cases use port
       programs, make sure the paths are correct.
      </item>
      <item><c>recompilation</c> - Make sure all test suites are fresh
       compiled.
      </item>
      <item><c>test_server.hrl</c> - Make sure the
      <c>test_server.hrl</c> file is in the include path when
       compiling test suites.
      </item>
      <item><c>running applications</c> - Some test suites require
       some applications to be running (e.g. sasl). Make sure they are
       started.
      </item>
    </list>
  </section>
</chapter>