aboutsummaryrefslogtreecommitdiffstats
path: root/lib/test_server/doc/src/test_server.xml
blob: 5ef43dd6287f6fe458fb10e36f2e34a521ead4cf (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">

<erlref>
  <header>
    <copyright>
      <year>2007</year>
      <year>2013</year>
      <holder>Ericsson AB, All Rights Reserved</holder>
    </copyright>
    <legalnotice>
  The contents of this file are subject to the Erlang Public License,
  Version 1.1, (the "License"); you may not use this file except in
  compliance with the License. You should have received a copy of the
  Erlang Public License along with this software. If not, it can be
  retrieved online at http://www.erlang.org/.

  Software distributed under the License is distributed on an "AS IS"
  basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
  the License for the specific language governing rights and limitations
  under the License.

  The Initial Developer of the Original Code is Ericsson AB.
    </legalnotice>

    <title>test_server</title>
    <prepared>Siri Hansen</prepared>
    <responsible></responsible>
    <docno></docno>
    <approved></approved>
    <checked></checked>
    <date></date>
    <rev></rev>
    <file>test_server_ref.sgml</file>
  </header>
  <module>test_server</module>
  <modulesummary>This module provides support for test suite authors.</modulesummary>
  <description>
    <p>The <c>test_server</c> module aids the test suite author by providing
      various support functions. The supported functionality includes:
      </p>
    <list type="bulleted">
      <item>Logging and timestamping
      </item>
      <item>Capturing output to stdout
      </item>
      <item>Retrieving and flushing the message queue of a process
      </item>
      <item>Watchdog timers, process sleep, time measurement and unit
       conversion
      </item>
      <item>Private scratch directory for all test suites
      </item>
      <item>Start and stop of slave- or peer nodes</item>
    </list>
    <p>For more information on how to write test cases and for
      examples, please see the Test Server User's Guide. 
      </p>
  </description>

  <section>
    <title>TEST SUITE SUPPORT FUNCTIONS</title>
    <p>The following functions are supposed to be used inside a test
      suite.
      </p>
  </section>
  <funcs>
    <func>
      <name>os_type() -> OSType</name>
      <fsummary>Returns the OS type of the target node</fsummary>
      <type>
        <v>OSType = term()</v>
        <d>This is the same as returned from <c>os:type/0</c></d>
      </type>
      <desc>
        <p>This function is equivalent to <c>os:type/0</c>. It is kept
        for backwards compatibility.</p>
      </desc>
    </func>
    <func>
      <name>fail()</name>
      <name>fail(Reason)</name>
      <fsummary>Makes the test case fail.</fsummary>
      <type>
        <v>Reason = term()</v>
        <d>The reason why the test case failed.</d>
      </type>
      <desc>
        <p>This will make the test suite fail with a given reason, or
          with <c>suite_failed</c> if no reason was given. Use this
          function if you want to terminate a test case, as this will
          make it easier to read the log- and HTML files. <c>Reason</c>
          will appear in the comment field in the HTML log.</p>
      </desc>
    </func>
    <func>
      <name>timetrap(Timout) -> Handle</name>
      <fsummary></fsummary>
      <type>
        <v>Timeout = integer() | {hours,H} | {minutes,M} | {seconds,S}</v>
        <v>H = M = S = integer()</v>
        <v>Pid = pid()</v>
        <d>The process that is to be timetrapped (<c>self()</c>by default)</d>
      </type>
      <desc>
        <p>Sets up a time trap for the current process. An expired
          timetrap kills the process with reason
          <c>timetrap_timeout</c>. The returned handle is to be given
          as argument to <c>timetrap_cancel</c> before the timetrap
          expires.  If <c>Timeout</c> is an integer, it is expected to
          be milliseconds.</p>
        <note>
          <p>If the current process is trapping exits, it will not be killed
            by the exit signal with reason <c>timetrap_timeout</c>.
            If this happens, the process will be sent an exit signal
            with reason <c>kill</c> 10 seconds later which will kill the
            process. Information about the timetrap timeout will in
            this case not be found in the test logs. However, the
            error_logger will be sent a warning.</p>
        </note>
      </desc>
    </func>
    <func>
      <name>timetrap_cancel(Handle) -> ok</name>
      <fsummary>Cancels a timetrap.</fsummary>
      <type>
        <v>Handle = term()</v>
        <d>Handle returned from <c>timetrap</c></d>
      </type>
      <desc>
        <p>This function cancels a timetrap. This must be done before
          the timetrap expires.</p>
      </desc>
    </func>
    <func>
      <name>timetrap_scale_factor() -> ScaleFactor</name>
      <fsummary>Returns the scale factor for timeouts.</fsummary>
      <type>
        <v>ScaleFactor = integer()</v>
      </type>
      <desc>
        <p>This function returns the scale factor by which all timetraps
	are scaled. It is normally 1, but can be greater than 1 if
	the test_server is running <c>cover</c>, using a larger amount of
	scheduler threads than the amount of logical processors on the
	system, running under purify, valgrind or in a debug-compiled
	emulator. The scale factor can be used if you need to scale you
	own timeouts in test cases with same factor as the test_server
	uses.</p>
      </desc>
    </func>
    <func>
      <name>sleep(MSecs) -> ok</name>
      <fsummary>Suspens the calling task for a specified time.</fsummary>
      <type>
        <v>MSecs = integer() | float() | infinity</v>
        <d>The number of milliseconds to sleep</d>
      </type>
      <desc>
        <p>This function suspends the calling process for at least the
          supplied number of milliseconds. There are two major reasons
          why you should use this function instead of
          <c>timer:sleep</c>, the first being that the module
          <c>timer</c> may be unavailable at the time the test suite is
          run, and the second that it also accepts floating point
          numbers.</p>
      </desc>
    </func>
    <func>
      <name>adjusted_sleep(MSecs) -> ok</name>
      <fsummary>Suspens the calling task for a specified time.</fsummary>
      <type>
        <v>MSecs = integer() | float() | infinity</v>
        <d>The default number of milliseconds to sleep</d>
      </type>
      <desc>
        <p>This function suspends the calling process for at least the
          supplied number of milliseconds. The function behaves the same
	  way as <c>test_server:sleep/1</c>, only <c>MSecs</c>
	  will be multiplied by the 'multiply_timetraps' value, if set,
	  and also automatically scaled up if 'scale_timetraps' is set
	  to true (which it is by default).</p>
      </desc>
    </func>
    <func>
      <name>hours(N) -> MSecs</name>
      <name>minutes(N) -> MSecs</name>
      <name>seconds(N) -> MSecs</name>
      <fsummary></fsummary>
      <type>
        <v>N = integer()</v>
        <d>Value to convert to milliseconds.</d>
      </type>
      <desc>
        <p>Theese functions convert <c>N</c> number of hours, minutes
          or seconds into milliseconds.
          </p>
        <p>Use this function when you want to
          <c>test_server:sleep/1</c> for a number of seconds, minutes or
          hours(!).</p>
      </desc>
    </func>
    <func>
      <name>format(Format) -> ok</name>
      <name>format(Format, Args)</name>
      <name>format(Pri, Format)</name>
      <name>format(Pri, Format, Args)</name>
      <fsummary></fsummary>
      <type>
        <v>Format = string()</v>
        <d>Format as described for <c>io_:format</c>.</d>
        <v>Args = list()</v>
        <d>List of arguments to format.</d>
      </type>
      <desc>
        <p>Formats output just like <c>io:format</c> but sends the
          formatted string to a logfile. If the urgency value,
          <c>Pri</c>, is lower than some threshold value, it will also
          be written to the test person's console. Default urgency is
          50, default threshold for display on the console is 1.
          </p>
        <p>Typically, the test person don't want to see everything a
          test suite outputs, but is merely interested in if the test
          cases succeeded or not, which the test server tells him. If he
          would like to see more, he could manually change the threshold
          values by using the <c>test_server_ctrl:set_levels/3</c>
          function.</p>
      </desc>
    </func>
    <func>
      <name>capture_start() -> ok</name>
      <name>capture_stop() -> ok</name>
      <name>capture_get() -> list()</name>
      <fsummary>Captures all output to stdout for a process.</fsummary>
      <desc>
        <p>These functions makes it possible to capture all output to
          stdout from a process started by the test suite. The list of
          characters captured can be purged by using <c>capture_get</c>.</p>
      </desc>
    </func>
    <func>
      <name>messages_get() -> list()</name>
      <fsummary>Empty the message queue.</fsummary>
      <desc>
        <p>This function will empty and return all the messages
          currently in the calling process' message queue.</p>
      </desc>
    </func>
    <func>
      <name>timecall(M, F, A) -> {Time, Value}</name>
      <fsummary>Measures the time needed to call a function.</fsummary>
      <type>
        <v>M = atom()</v>
        <d>The name of the module where the function resides.</d>
        <v>F = atom()</v>
        <d>The name of the function to call in the module.</d>
        <v>A = list()</v>
        <d>The arguments to supply the called function.</d>
        <v>Time = integer()</v>
        <d>The number of seconds it took to call the function.</d>
        <v>Value = term()</v>
        <d>Value returned from the called function.</d>
      </type>
      <desc>
        <p>This function measures the time (in seconds) it takes to
          call a certain function. The function call is <em>not</em>
          caught within a catch.</p>
      </desc>
    </func>
    <func>
      <name>do_times(N, M, F, A) -> ok</name>
      <name>do_times(N, Fun)</name>
      <fsummary>Calls MFA or Fun N times.</fsummary>
      <type>
        <v>N = integer()</v>
        <d>Number of times to call MFA.</d>
        <v>M = atom()</v>
        <d>Module name where the function resides.</d>
        <v>F = atom()</v>
        <d>Function name to call.</d>
        <v>A = list()</v>
        <d>Arguments to M:F.</d>
      </type>
      <desc>
        <p>Calls MFA or Fun N times. Useful for extensive testing of a
          sensitive function.</p>
      </desc>
    </func>
    <func>
      <name>m_out_of_n(M, N, Fun) -> ok | exit({m_out_of_n_failed, {R,left_to_do}}</name>
      <fsummary>Fault tolerant <c>do_times</c>.</fsummary>
      <type>
        <v>N = integer()</v>
        <d>Number of times to call the Fun.</d>
        <v>M = integer()</v>
        <d>Number of times to require a successful return.</d>
      </type>
      <desc>
        <p>Repeatedly evaluates the given function until it succeeds
          (doesn't crash) M times. If, after N times, M successful
          attempts have not been accomplished, the process crashes with
          reason {m_out_of_n_failed, {R,left_to_do}}, where R indicates
          how many cases that was still to be successfully completed.
          </p>
        <p>For example:
          </p>
        <p><c>m_out_of_n(1,4,fun() -> tricky_test_case() end)</c>          <br></br>
Tries to run tricky_test_case() up to 4 times, and is
          happy if it succeeds once.
          </p>
        <p><c>m_out_of_n(7,8,fun() -> clock_sanity_check() end)</c>          <br></br>
Tries running clock_sanity_check() up to 8 times,and
          allows the function to fail once.  This might be useful if
          clock_sanity_check/0 is known to fail if the clock crosses an
          hour boundary during the test (and the up to 8 test runs could
          never cross 2 boundaries)</p>
      </desc>
    </func>
    <func>
      <name>call_crash(M, F, A) -> Result</name>
      <name>call_crash(Time, M, F, A) -> Result</name>
      <name>call_crash(Time, Crash, M, F, A) -> Result</name>
      <fsummary>Calls MFA and succeeds if it crashes.</fsummary>
      <type>
        <v>Result = ok | exit(call_crash_timeout) | exit({wrong_crash_reason, Reason})</v>
        <v>Crash = term()</v>
        <d>Crash return from the function.</d>
        <v>Time = integer()</v>
        <d>Timeout in milliseconds.</d>
        <v>M = atom()</v>
        <d>Module name where the function resides.</d>
        <v>F = atom()</v>
        <d>Function name to call.</d>
        <v>A = list()</v>
        <d>Arguments to M:F.</d>
      </type>
      <desc>
        <p>Spawns a new process that calls MFA. The call is considered
          successful if the call crashes with the gives reason
          (<c>Crash</c>) or any reason if not specified. The call must
          terminate within the given time (default <c>infinity</c>), or
          it is considered a failure.</p>
      </desc>
    </func>
    <func>
      <name>temp_name(Stem) -> Name</name>
      <fsummary>Returns a unique filename.</fsummary>
      <type>
        <v>Stem = string()</v>
      </type>
      <desc>
        <p>Returns a unique filename starting with <c>Stem</c> with
          enough extra characters appended to make up a unique
          filename. The filename returned is guaranteed not to exist in
          the filesystem at the time of the call.</p>
      </desc>
    </func>
    <func>
      <name>break(Comment) -> ok</name>
      <fsummary>Cancel all timetraps and wait for call to continue/0.</fsummary>
      <type>
        <v>Comment = string()</v>
      </type>
      <desc>
        <p><c>Comment</c> is a string which will be written in
          the shell, e.g. explaining what to do.</p>
        <p>This function will cancel all timetraps and pause the
          execution of the test case until the user executes the
          <c>continue/0</c> function. It gives the user the opportunity
          to interact with the erlang node running the tests, e.g. for
          debugging purposes or for manually executing a part of the
          test case.</p>
        <p>When the <c>break/1</c> function is called, the shell will
          look something like this:</p>
        <code type="none"><![CDATA[
   --- SEMIAUTOMATIC TESTING ---
   The test case executes on process <0.51.0>


   "Here is a comment, it could e.g. instruct to pull out a card"


   -----------------------------

   Continue with --> test_server:continue().        ]]></code>
        <p>The user can now interact with the erlang node, and when
          ready call <c>test_server:continue().</c></p>
        <p>Note that this function can not be used if the test is
          executed with <c>ts:run/0/1/2/3/4</c> in <c>batch</c> mode.</p>
      </desc>
    </func>
    <func>
      <name>continue() -> ok</name>
      <fsummary>Continue after break/1.</fsummary>
      <desc>
        <p>This function must be called in order to continue after a
          test case has called <c>break/1</c>.</p>
      </desc>
    </func>
    <func>
      <name>run_on_shielded_node(Fun, CArgs) -> term()</name>
      <fsummary>Execute a function a shielded node.</fsummary>
      <type>
        <v>Fun = function() (arity 0)</v>
        <d>Function to execute on the shielded node.</d>
        <v>CArg = string()</v>
        <d>Extra command line arguments to use when starting the shielded node.</d>
      </type>
      <desc>
        <p><c>Fun</c> is executed in a process on a temporarily created
          hidden node with a proxy for communication with the test server
          node. The node is called a shielded node (should have been called
          a shield node). If <c>Fun</c> is successfully executed, the result
          is returned. A peer node (see <c>start_node/3</c>) started from
          the shielded node will be shielded from test server node, i.e.
          they will not be aware of each other. This is useful when you want
          to start nodes from earlier OTP releases than the OTP release of
          the test server node.</p>
        <p>Nodes from an earlier OTP release can normally not be started
          if the test server hasn't been started in compatibility mode
          (see the <c>+R</c> flag in the <c>erl(1)</c> documentation) of
          an earlier release. If a shielded node is started in compatibility
          mode of an earlier OTP release than the OTP release of the test
          server node, the shielded node can start nodes of an earlier OTP
          release.</p>
        <note>
          <p>You <em>must</em> make sure that nodes started by the shielded
            node never communicate directly with the test server node.</p>
        </note>
        <note>
          <p>Slave nodes always communicate with the test server node;
            therefore, <em>never</em> start <em>slave nodes</em> from the
            shielded node, <em>always</em> start <em>peer nodes</em>.</p>
        </note>
      </desc>
    </func>
    <func>
      <name>start_node(Name, Type, Options) -> {ok, Node} | {error, Reason}</name>
      <fsummary>Start a node.</fsummary>
      <type>
        <v>Name = atom() | string()</v>
        <d>Name of the slavenode to start (as given to -sname or -name)</d>
        <v>Type = slave | peer</v>
        <d>The type of node to start.</d>
        <v>Options = [{atom(), term()]</v>
        <d>Tuplelist of options</d>
      </type>
      <desc>
        <p>This functions starts a node, possibly on a remote machine,
          and guarantees cross architecture transparency. Type is set to
          either <c>slave</c> or <c>peer</c>.
          </p>
        <p><c>slave</c> means that the new node will have a master,
          i.e. the slave node will terminate if the master terminates,
          TTY output produced on the slave will be sent back to the
          master node and file I/O is done via the master. The master is
          normally the target node unless the target is itself a slave.
          </p>
        <p><c>peer</c> means that the new node is an independent node
          with no master.
          </p>
        <p><c>Options</c> is a tuplelist which can contain one or more
          of
          </p>
        <taglist>
          <tag><c>{remote, true}</c></tag>
          <item>Start the node on a remote host. If not specified, the
           node will be started on the local host.  Test cases that
           require a remote host will fail with a reasonable comment if
           no remote hosts are available at the time they are run.
          </item>
          <tag><c>{args, Arguments}</c></tag>
          <item>Arguments passed directly to the node. This is
           typically a string appended to the command line.
          </item>
          <tag><c>{wait, false}</c></tag>
          <item>Don't wait until the node is up. By default, this
           function does not return until the node is up and running,
           but this option makes it return as soon as the node start
           command is given..
                    <br></br>
Only valid for peer nodes
          </item>
          <tag><c>{fail_on_error, false}</c></tag>
          <item>Returns <c>{error, Reason}</c> rather than failing the
           test case.
                    <br></br>
Only valid for peer nodes. Note that slave nodes always
           act as if they had <c>fail_on_error=false</c></item>
          <tag><c>{erl, ReleaseList}</c></tag>
          <item>Use an Erlang emulator determined by ReleaseList when
           starting nodes, instead of the same emulator as the test
           server is running. ReleaseList is a list of specifiers,
           where a specifier is either {release, Rel}, {prog, Prog}, or
           'this'. Rel is either the name of a release, e.g., "r12b_patched"
           or 'latest'. 'this' means using the same emulator as the test
           server. Prog is the name of an emulator executable.  If the
           list has more than one element, one of them is picked
           randomly.  (Only works on Solaris and Linux, and the test server 
           gives warnings when it notices that nodes are not of the same
           version as itself.)
                    <br></br>
          <br></br>

           When specifying this option to run a previous release, use
          <c>is_release_available/1</c> function to test if the given
           release is available and skip the test case if not.
                    <br></br>
          <br></br>

           In order to avoid compatibility problems (may not appear right
           away), use a shielded node (see <c>run_on_shielded_node/2</c>)
           when starting nodes from different OTP releases than the test
           server.
          </item>
          <tag><c>{cleanup, false}</c></tag>
          <item>Tells the test server not to kill this node if it is
           still alive after the test case is completed. This is useful
           if the same node is to be used by a group of test cases.
          </item>
          <tag><c>{env, Env}</c></tag>
          <item><c>Env</c> should be a list of tuples <c>{Name, Val}</c>,
           where <c>Name</c> is the name of an environment variable, and
          <c>Val</c> is the value it is to have in the started node.
           Both <c>Name</c> and <c>Val</c> must be strings. The one
           exception is <c>Val</c> being the atom <c>false</c> (in
           analogy  with  <c>os:getenv/1</c>),  which  removes  the
           environment variable. Only valid for peer nodes. Not
           available on VxWorks.</item>
	  <tag><c>{start_cover, false}</c></tag>
	  <item>By default the test server will start cover on all nodes
	   when the test is run with code coverage analysis. To make
	   sure cover is not started on a new node, set this option to
	   <c>false</c>. This can be necessary if the connection to
	   the node at some point will be broken but the node is
	   expected to stay alive. The reason is that a remote cover
	   node can not continue to run without its main node. Another
	   solution would be to explicitly stop cover on the node
	   before breaking the connection, but in some situations (if
	   old code resides in one or more processes) this is not
	   possible.</item>
        </taglist>
      </desc>
    </func>
    <func>
      <name>stop_node(NodeName) -> bool()</name>
      <fsummary>Stops a node</fsummary>
      <type>
        <v>NodeName = term()</v>
        <d>Name of the node to stop</d>
      </type>
      <desc>
        <p>This functions stops a node previously started with
          <c>start_node/3</c>. Use this function to stop any node you
          start, or the test server will produce a warning message in
          the test logs, and kill the nodes automatically unless it was
          started with the <c>{cleanup, false}</c> option.</p>
      </desc>
    </func>
    <func>
      <name>is_commercial() -> bool()</name>
      <fsummary>Tests whether the emulator is commercially supported</fsummary>
      <desc>
        <p>This function test whether the emulator is commercially supported
	emulator. The tests for a commercially supported emulator could be more
	stringent (for instance, a commercial release should always contain
	documentation for all applications).</p>
      </desc>
    </func>

    <func>
      <name>is_release_available(Release) -> bool()</name>
      <fsummary>Tests whether a release is available</fsummary>
      <type>
        <v>Release = string() | atom()</v>
        <d>Release to test for</d>
      </type>
      <desc>
        <p>This function test whether the release given by
          <c>Release</c> (for instance, "r12b_patched") is available
          on the computer that the test_server controller is running on.
          Typically, you should skip the test case if not.</p>
        <p>Caution: This function may not be called from the <c>suite</c>
          clause of a test case, as the test_server will deadlock.</p>
      </desc>
    </func>
    <func>
      <name>is_native(Mod) -> bool()</name>
      <fsummary>Checks whether the module is natively compiled or not</fsummary>
      <type>
        <v>Mod = atom()</v>
        <d>A module name</d>
      </type>
      <desc>
        <p>Checks whether the module is natively compiled or not</p>
      </desc>
    </func>
    <func>
      <name>app_test(App) -> ok | test_server:fail()</name>
      <name>app_test(App,Mode)</name>
      <fsummary>Checks an applications .app file for obvious errors</fsummary>
      <type>
        <v>App = term()</v>
        <d>The name of the application to test</d>
        <v>Mode = pedantic | tolerant</v>
        <d>Default is pedantic</d>
      </type>
      <desc>
        <p>Checks an applications .app file for obvious errors.
          The following is checked:
          </p>
        <list type="bulleted">
          <item>required fields
          </item>
          <item>that all modules specified actually exists
          </item>
          <item>that all requires applications exists
          </item>
          <item>that no module included in the application has export_all
          </item>
          <item>that all modules in the ebin/ dir is included (If
          <c>Mode==tolerant</c> this only produces a warning, as all
           modules does not have to be included)</item>
        </list>
      </desc>
    </func>
    <func>
      <name>comment(Comment) -> ok</name>
      <fsummary>Print a comment on the HTML result page</fsummary>
      <type>
        <v>Comment = string()</v>
      </type>
      <desc>
        <p>The given String will occur in the comment field of the
          table on the HTML result page. If called several times, only
          the last comment is printed.  comment/1 is also overwritten by
          the return value {comment,Comment} from a test case or by
          fail/1 (which prints Reason as a comment).</p>
      </desc>
    </func>
  </funcs>

  <section>
    <title>TEST SUITE EXPORTS</title>
    <p>The following functions must be exported from a test suite
      module.
      </p>
  </section>
  <funcs>
    <func>
      <name>all(suite) -> TestSpec | {skip, Comment}</name>
      <fsummary>Returns the module's test specification</fsummary>
      <type>
        <v>TestSpec = list()</v>
        <v>Comment = string()</v>
        <d>This comment will be printed on the HTML result page</d>
      </type>
      <desc>
        <p>This function must return the test specification for the
          test suite module. The syntax of a test specification is
          described in the Test Server User's Guide.</p>
      </desc>
    </func>
    <func>
      <name>init_per_suite(Config0) -> Config1 | {skip, Comment}</name>
      <fsummary>Test suite initiation</fsummary>
      <type>
        <v>Config0 = Config1 = [tuple()]</v>
        <v>Comment = string()</v>
	<d>Describes why the suite is skipped</d>
      </type>
      <desc>
        <p>This function is called before all other test cases in the
          suite. <c>Config</c> is the configuration which can be modified
          here. Whatever is returned from this function is given as
          <c>Config</c> to the test cases.
          </p>
        <p>If this function fails, all test cases in the suite will be
          skipped.</p>
      </desc>
    </func>
    <func>
      <name>end_per_suite(Config) -> void()</name>
      <fsummary>Test suite finalization</fsummary>
      <type>
        <v>Config = [tuple()]</v>
      </type>
      <desc>
        <p>This function is called after the last test case in the
          suite, and can be used to clean up whatever the test cases
          have done. The return value is ignored.</p>
      </desc>
    </func>
    <func>
      <name>init_per_testcase(Case, Config0) -> Config1 | {skip, Comment}</name>
      <fsummary>Test case initiation</fsummary>
      <type>
        <v>Case = atom()</v>
        <v>Config0 = Config1 = [tuple()]</v>
        <v>Comment = string()</v>
	<d>Describes why the test case is skipped</d>
      </type>
      <desc>
        <p>This function is called before each test case. The
          <c>Case</c> argument is the name of the test case, and
          <c>Config</c> is the configuration which can be modified
          here. Whatever is returned from this function is given as
          <c>Config</c> to the test case.</p>
      </desc>
    </func>
    <func>
      <name>end_per_testcase(Case, Config) -> void()</name>
      <fsummary>Test case finalization</fsummary>
      <type>
        <v>Case = atom()</v>
        <v>Config = [tuple()]</v>
      </type>
      <desc>
        <p>This function is called after each test case, and can be
          used to clean up whatever the test case has done. The return
          value is ignored.</p>
      </desc>
    </func>
    <func>
      <name>Case(doc) -> [Decription]</name>
      <name>Case(suite) -> [] | TestSpec | {skip, Comment}</name>
      <name>Case(Config) -> {skip, Comment} | {comment, Comment} | Ok</name>
      <fsummary>A test case</fsummary>
      <type>
        <v>Description = string()</v>
        <d>Short description of the test case</d>
        <v>TestSpec = list()</v>
        <v>Comment = string()</v>
        <d>This comment will be printed on the HTML result page</d>
        <v>Ok = term()</v>
        <v>Config = [tuple()]</v>
        <d>Elements from the Config parameter can be read with the ?config macro, see section about test suite support macros</d>
      </type>
      <desc>
        <p>The <em>documentation clause</em> (argument <c>doc</c>) can
          be used for automatic generation of test documentation or test
          descriptions.
          </p>
        <p>The <em>specification clause</em> (argument <c>spec</c>)
          shall return an empty list, the test specification for the
          test case or <c>{skip,Comment}</c>. The syntax of a test
          specification is described in the Test Server User's Guide.
          </p>
        <p>The <em>execution clause</em> (argument <c>Config</c>) is
          only called if the specification clause returns an empty list.
          The execution clause is the real test case. Here you must call
          the functions you want to test, and do whatever you need to
          check the result. If something fails, make sure the process
          crashes or call <c>test_server:fail/0/1</c> (which also will
          cause the process to crash).
          </p>
        <p>You can return <c>{skip,Comment}</c> if you decide not to
          run the test case after all, e.g. if it is not applicable on
          this platform.
          </p>
        <p>You can return <c>{comment,Comment}</c> if you wish to
          print some information in the 'Comment' field on the HTML
          result page.
          </p>
        <p>If the execution clause returns anything else, it is
          considered a success, unless it is <c>{'EXIT',Reason}</c> or
          <c>{'EXIT',Pid,Reason}</c> which can't be distinguished from a
          crash, and thus will be considered a failure.
          </p>
        <p>A <em>conf test case</em> is a group of test cases with an
          init and a cleanup function. The init and cleanup functions
          are also test cases, but they have special rules:</p>
	  <list type="bulleted">
	  <item>They do not need a specification clause.</item>
	  <item>They must always have the execution clause.</item>
	  <item>They must return the <c>Config</c> parameter, a modified
          version of it or <c>{skip,Comment}</c> from the execution clause.</item>
	  <item>The cleanup function may also return a tuple 
	  <c>{return_group_result,Status}</c>, which is used to return the
	  status of the conf case to Test Server and/or to a conf case on a
	  higher level. (<c>Status = ok | skipped | failed</c>).</item>
	  <item><c>init_per_testcase</c> and <c>end_per_testcase</c> are
          not called before and after these functions.</item>
	  </list>
      </desc>
    </func>
  </funcs>

  <section>
    <title>TEST SUITE LINE NUMBERS</title>
    <p>If a test case fails, the test server can report the exact line
      number at which it failed. There are two ways of doing this,
      either by using the <c>line</c> macro or by using the
      <c>test_server_line</c> parse transform.
      </p>
    <p>The <c>line</c> macro is described under TEST SUITE SUPPORT
      MACROS below. The <c>line</c> macro will only report the last line
      executed when a test case failed.
      </p>
    <p>The <c>test_server_line</c> parse transform is activated by
      including the headerfile <c>test_server_line.hrl</c> in the test
      suite. When doing this, it is important that the
      <c>test_server_line</c> module is in the code path of the erlang
      node compiling the test suite. The parse transform will report a
      history of a maximum of 10 lines when a test case
      fails. Consecutive lines in the same function are not shown.
      </p>
    <p>The attribute <c>-no_lines(FuncList).</c> can be used in the
      test suite to exclude specific functions from the parse
      transform. This is necessary e.g. for functions that are executed
      on old (i.e. &lt;R10B) OTP releases. <c>FuncList = [{Func,Arity}]</c>.
      </p>
    <p>If both the <c>line</c> macro and the parse transform is used in
      the same module, the parse transform will overrule the macro.
      </p>
  </section>

  <section>
    <title>TEST SUITE SUPPORT MACROS</title>
    <p>There are some macros defined in the <c>test_server.hrl</c>
      that are quite useful for test suite programmers:
      </p>
    <p>The <em>line</em> macro, is quite
      essential when writing test cases. It tells the test server
      exactly what line of code that is being executed, so that it can
      report this line back if the test case fails. Use this macro at
      the beginning of every test case line of code.
      </p>
    <p>The <em>config</em> macro, is used to
      retrieve information from the <c>Config</c> variable sent to all
      test cases. It is used with two arguments, where the first is the
      name of the configuration variable you wish to retrieve, and the
      second is the <c>Config</c> variable supplied to the test case
      from the test server.
      </p>
    <p>Possible configuration variables include:</p>
    <list type="bulleted">
      <item><c>data_dir</c>  - Data file directory.</item>
      <item><c>priv_dir</c>  - Scratch file directory.</item>
      <item><c>nodes</c>     - Nodes specified in the spec file</item>
      <item><c>nodenames</c> - Generated nodenames.</item>
      <item>Whatever added by conf test cases or
      <c>init_per_testcase/2</c></item>
    </list>
    <p>Examples of the <c>line</c> and <c>config</c> macros can be
      seen in the Examples chapter in the user's guide.
      </p>
    <p>If the <c>line_trace</c> macro is defined, you will get a
      timestamp (<c>erlang:now()</c>) in your minor log for each
      <c>line</c> macro in your suite. This way you can at any time see
      which line is currently being executed, and when the line was
      called.
      </p>
    <p>The <c>line_trace</c> macro can also be used together with the
      <c>test_server_line</c> parse transform described above. A
      timestamp will then be written for each line in the suite, except
      for functions stated in the <c>-no_lines</c> attribute.
      </p>
    <p>The <c>line_trace</c> macro can e.g. be defined as a compile
      option, like this:
            <br></br>
<c>erlc -W -Dline_trace my_SUITE.erl</c></p>
  </section>
</erlref>