aboutsummaryrefslogblamecommitdiffstats
path: root/lib/test_server/doc/src/basics_chapter.xml
blob: 2e60d412e971e4cae94d49e5f25ee9073b8fef4c (plain) (tree)
1
2
3
4
5
6
7





                                        
                                        



































































































                                                                            
                                                      




                                                                    

                                                                  






























































                                                                         
                           


































                                                                      
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2002</year><year>2013</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.
    
      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.
    
    </legalnotice>

    <title>Test Server Basics</title>
    <prepared>Siri Hansen</prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>basics_chapter.xml</file>
  </header>

  <section>
    <title>Introduction</title>
    <p><em>Test Server</em> is a portable test tool for automated
    testing of Erlang programs and OTP applications. It provides an
    interface for running test programs directly with Test Server 
    as well as an interface for integrating Test Server
    with a framework application. The latter makes it possible to use
    Test Server as the engine of a higher level test tool
    application.</p>

    <p>It is strongly recommended that Test Server be used from inside
    a framework application, rather than interfaced directly for
    running test programs. Test Server can be pretty difficult to use
    since it's a very general and quite extensive and complex
    application. Furthermore, the <c>test_server_ctrl</c> functions
    are not meant to be used from within the actual test programs. The
    framework should handle communication with Test Server and deal
    with the more complex aspects of this interaction automatically so
    that a higher level interface may be provided for the tester. For
    test tool usage to be productive, a simpler, more intuitive and 
    (if required) more specific interface is required than what Test Server
    can provide.</p>

    <p>OTP delivers a general purpose framework for Test Server, called
    <em>Common Test</em>. This application is a tool well suited for
    automated black box testing of target systems of <em>any kind</em> 
    (not necessarily implemented in Erlang). Common Test is also a very
    useful tool for white box testing of Erlang programs and OTP
    applications. Unless a more specific functionality and/or user 
    interface is required (in which case you might need to implement 
    your own framework), Common Test should do the job for 
    you. Please read the Common Test User's Guide and reference manual 
    for more information.</p>

    <p>Under normal circumstances, knowledge about the Test Server
    application is not required for using the Common Test framework.
    However, if you want to use Test Server without a framework,
    or learn how to integrate it with your own framework, please read on...
    </p>
    </section>
    <section>
    <title>Getting started</title>
    <p>Testing when using Test Server is done by running test
      suites. A test suite is a number of test cases, where each test
      case tests one or more things. The test case is the smallest unit
      that the test server deals with. One or more test cases are
      grouped together into one ordinary Erlang module, which is called
      a test suite. Several test suite modules can be grouped together
      in special test specification files representing whole application
      and/or system test "jobs".
      </p>
    <p>The test suite Erlang module must follow a certain interface,
      which is specified by Test Server. See the section on writing
      test suites for details about this.
      </p>
    <p>Each test case is considered a success if it returns to the
      caller, no matter what the returned value is. An exception to this
      is the return value <c>{skip, Reason}</c> which indicates that the
      test case is skipped. A failure is specified as a crash, no matter
      what the crash reason is.
      </p>
    <p>As a test suite runs, all information (including output to
      stdout) is recorded in several different log files. A minimum of
      information is displayed to the user console. This only include
      start and stop information, plus a note for each failed test case.
      </p>
    <p>The result from each test case is recorded in an HTML log file
      which is created for each test run. Every test case gets one row
      in a table presenting total time, whether the case was successful
      or not, if it was skipped, and possibly also a comment. The HTML
      file has links to each test case's logfile, which may be viewed
      from e.g. Netscape or any other HTML capable browser.
      </p>
    <p>The Test Server consists of three parts: 
      </p>
    <list type="bulleted">
      <item>The part that executes the test suites and
       provides support for the test suite author is called
      <c>test_server</c>. This is described in the chapter about
       writing test cases in this user's guide, and in the reference
       manual for the <c>test_server</c> module.</item>
      <item>The controlling part, which provides the low level
       operator interface, starts and stops slave nodes and writes
       log files, is called
      <c>test_server_ctrl</c>. The Test Server Controller should not
       be used directly when running tests. Instead a framework built
       on top of it should be used. More information
       about how to write your own framework can be found
       in this user's guide and in the reference manual for the
      <c>test_server_ctrl</c> module.</item>
    </list>
  </section>

  <section>
    <title>Definition of terms</title>
    <taglist>
      <tag><em>conf(iguration) case</em></tag>
      <item>This is a group of test cases which need some specific
       configuration. A conf case contains an initiation function which
       sets up a specific configuration, one or more test cases using
       this configuration, and a cleanup function which restores the
       configuration. A conf case is specified in a test specification
       either like this:<c>{conf,InitFunc,ListOfCases,CleanupFunc}</c>,
       or this: <c>{conf,Properties,InitFunc,ListOfCases,CleanupFunc}</c>
       </item>
      <tag><em>datadir</em></tag>
      <item>Data directory for a test suite. This directory contains
       any files used by the test suite, e.g. additional erlang
       modules, c code or data files. If the data directory contains
       code which must be compiled before the test suite is run, it
       should also contain a makefile source called Makefile.src
       defining how to compile.
      </item>
      <tag><em>documentation clause</em></tag>
      <item>One of the function clauses in a test case. This clause
       shall return a list of strings describing what the test case
       tests.
      </item>
      <tag><em>execution clause</em></tag>
      <item>One of the function clauses in a test case. This clause
       implements the actual test case, i.e. calls the functions that
       shall be tested and checks results. The clause shall crash if it
       fails.
      </item>
      <tag><em>major log file</em></tag>
      <item>This is the test suites log file.
      </item>
      <tag><em>Makefile.src</em></tag>
      <item>This file is used by the test server framework to generate
       a makefile for a datadir. It contains some special characters
       which are replaced according to the platform currently tested.
      </item>
      <tag><em>minor log file</em></tag>
      <item>This is a separate log file for each test case.
      </item>
      <tag><em>privdir</em></tag>
      <item>Private directory for a test suite. This directory should
       be used when the test suite needs to write to files.
      </item>
      <tag><em>skip case</em></tag>
      <item>A test case which shall be skipped.
      </item>
      <tag><em>specification clause</em></tag>
      <item>One of the function clauses in a test case. This clause
       shall return an empty list, a test specification or
      <c>{skip,Reason}</c>. If an empty list is returned, it means
       that the test case shall be executed, and so it must also have
       an execution clause.
      </item>
      <tag><em>test case</em></tag>
      <item>A single test included in a test suite. Typically it tests
       one function in a module or application. A test case is
       implemented as a function in a test suite module. The function
       can have three clauses, the documentation-, specification- and
       execution clause.
      </item>
      <tag><em>test specification</em></tag>
      <item>A specification of which test suites and test cases to
       run. There can be test specifications on three different levels
       in a test. The top level is a test specification file which
       roughly specifies what to test for a whole application. Then
       there is a test specification for each test suite returned from
       the <c>all(suite)</c> function in the suite. And there can also
       be a test specification returned from the specification clause
       of a test case.
      </item>
      <tag><em>test specification file</em></tag>
      <item>This is a text file containing the test specification for
       an application. The file has the extension ".spec" or
       ".spec.Platform", where Platform is e.g. "vxworks".
      </item>
      <tag><em>test suite</em></tag>
      <item>An erlang module containing a collection of test cases for
       a specific application or module.
      </item>
      <tag><em>topcase</em></tag>
      <item>The first "command" in a test specification file. This
       command contains the test specification, like this:
      <c>{topcase,TestSpecification}</c></item>
    </taglist>
  </section>
</chapter>