aboutsummaryrefslogtreecommitdiffstats
path: root/lib/test_server/doc/src/basics_chapter.xml
diff options
context:
space:
mode:
Diffstat (limited to 'lib/test_server/doc/src/basics_chapter.xml')
-rw-r--r--lib/test_server/doc/src/basics_chapter.xml216
1 files changed, 216 insertions, 0 deletions
diff --git a/lib/test_server/doc/src/basics_chapter.xml b/lib/test_server/doc/src/basics_chapter.xml
new file mode 100644
index 0000000000..a96cc88075
--- /dev/null
+++ b/lib/test_server/doc/src/basics_chapter.xml
@@ -0,0 +1,216 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+ <header>
+ <copyright>
+ <year>2002</year><year>2009</year>
+ <holder>Ericsson AB. All Rights Reserved.</holder>
+ </copyright>
+ <legalnotice>
+ The contents of this file are subject to the Erlang Public License,
+ Version 1.1, (the "License"); you may not use this file except in
+ compliance with the License. You should have received a copy of the
+ Erlang Public License along with this software. If not, it can be
+ retrieved online at http://www.erlang.org/.
+
+ Software distributed under the License is distributed on an "AS IS"
+ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ the License for the specific language governing rights and limitations
+ under the License.
+
+ </legalnotice>
+
+ <title>Test Server Basics</title>
+ <prepared>Siri Hansen</prepared>
+ <docno></docno>
+ <date></date>
+ <rev></rev>
+ <file>basics_chapter.xml</file>
+ </header>
+
+ <section>
+ <title>Introduction</title>
+ <p><em>Test Server</em> is a portable test tool for automated
+ testing of Erlang programs and OTP applications. It provides an
+ interface for running test programs directly with Test Server
+ as well as an interface for integrating Test Server
+ with a framework application. The latter makes it possible to use
+ Test Server as the engine of a higher level test tool
+ application.</p>
+
+ <p>It is strongly recommended that Test Server be used from inside
+ a framework application, rather than interfaced directly for
+ running test programs. Test Server can be pretty difficult to use
+ since it's a very general and quite extensive and complex
+ application. Furthermore, the <c>test_server_ctrl</c> functions
+ are not meant to be used from within the actual test programs. The
+ framework should handle communication with Test Server and deal
+ with the more complex aspects of this interaction automatically so
+ that a higher level interface may be provided for the tester. For
+ test tool usage to be productive, a simpler, more intuitive and
+ (if required) more specific interface is required than what Test Server
+ can provide.</p>
+
+ <p>OTP delivers a general purpose framework for Test Server, called
+ <em>Common Test</em>. This application is a tool well suited for
+ automated black box testing of target systems of <em>any kind</em>
+ (not necessarily implemented in Erlang). Common Test is also a very
+ useful tool for white box testing of Erlang programs and OTP
+ applications. Unless a more specific functionality and/or user
+ interface is required (in which case you might need to implement
+ your own framework), Common Test should do the job for
+ you. Please read the Common Test User's Guide and reference manual
+ for more information.</p>
+
+ <p>Under normal circumstances, knowledge about the Test Server
+ application is not required for using the Common Test framework.
+ However, if you want to use Test Server without a framework,
+ or learn how to integrate it with your own framework, please read on...
+ </p>
+ </section>
+ <section>
+ <title>Getting started</title>
+ <p>Testing when using Test Server is done by running test
+ suites. A test suite is a number of test cases, where each test
+ case tests one or more things. The test case is the smallest unit
+ that the test server deals with. One or more test cases are
+ grouped together into one ordinary Erlang module, which is called
+ a test suite. Several test suite modules can be grouped together
+ in special test specification files representing whole application
+ and/or system test "jobs".
+ </p>
+ <p>The test suite Erlang module must follow a certain interface,
+ which is specified by Test Server. See the section on writing
+ test suites for details about this.
+ </p>
+ <p>Each test case is considered a success if it returns to the
+ caller, no matter what the returned value is. An exception to this
+ is the return value <c>{skip, Reason}</c> which indicates that the
+ test case is skipped. A failure is specified as a crash, no matter
+ what the crash reason is.
+ </p>
+ <p>As a test suite runs, all information (including output to
+ stdout) is recorded in several different log files. A minimum of
+ information is displayed to the user console. This only include
+ start and stop information, plus a note for each failed test case.
+ </p>
+ <p>The result from each test case is recorded in an HTML log file
+ which is created for each test run. Every test case gets one row
+ in a table presenting total time, whether the case was successful
+ or not, if it was skipped, and possibly also a comment. The HTML
+ file has links to each test case's logfile, which may be viewed
+ from e.g. Netscape or any other HTML capable browser.
+ </p>
+ <p>The Test Server consists of three parts:
+ </p>
+ <list type="bulleted">
+ <item>The part that executes the test suites on target and
+ provides support for the test suite author is called
+ <c>test_server</c>. This is described in the chapter about
+ writing test cases in this user's guide, and in the reference
+ manual for the <c>test_server</c> module.</item>
+ <item>The controlling part, which provides the low level
+ operator interface, starts and stops the target node (if remote
+ target) and slave nodes and writes log files, is called
+ <c>test_server_ctrl</c>. The Test Server Controller should not
+ be used directly when running tests. Instead a framework built
+ on top of it should be used. More information
+ about how to write your own framework can be found
+ in this user's guide and in the reference manual for the
+ <c>test_server_ctrl</c> module.</item>
+ </list>
+ </section>
+
+ <section>
+ <title>Definition of terms</title>
+ <taglist>
+ <tag><em>conf(iguration) case</em></tag>
+ <item>This is a group of test cases which need some specific
+ configuration. A conf case contains an initiation function which
+ sets up a specific configuration, one or more test cases using
+ this configuration, and a cleanup function which restores the
+ configuration. A conf case is specified in a test specification
+ either like this:<c>{conf,InitFunc,ListOfCases,CleanupFunc}</c>,
+ or this: <c>{conf,Properties,InitFunc,ListOfCases,CleanupFunc}</c>
+ </item>
+ <tag><em>datadir</em></tag>
+ <item>Data directory for a test suite. This directory contains
+ any files used by the test suite, e.g. additional erlang
+ modules, c code or data files. If the data directory contains
+ code which must be compiled before the test suite is run, it
+ should also contain a makefile source called Makefile.src
+ defining how to compile.
+ </item>
+ <tag><em>documentation clause</em></tag>
+ <item>One of the function clauses in a test case. This clause
+ shall return a list of strings describing what the test case
+ tests.
+ </item>
+ <tag><em>execution clause</em></tag>
+ <item>One of the function clauses in a test case. This clause
+ implements the actual test case, i.e. calls the functions that
+ shall be tested and checks results. The clause shall crash if it
+ fails.
+ </item>
+ <tag><em>major log file</em></tag>
+ <item>This is the test suites log file.
+ </item>
+ <tag><em>Makefile.src</em></tag>
+ <item>This file is used by the test server framework to generate
+ a makefile for a datadir. It contains some special characters
+ which are replaced according to the platform currently tested.
+ </item>
+ <tag><em>minor log file</em></tag>
+ <item>This is a separate log file for each test case.
+ </item>
+ <tag><em>privdir</em></tag>
+ <item>Private directory for a test suite. This directory should
+ be used when the test suite needs to write to files.
+ </item>
+ <tag><em>skip case</em></tag>
+ <item>A test case which shall be skipped.
+ </item>
+ <tag><em>specification clause</em></tag>
+ <item>One of the function clauses in a test case. This clause
+ shall return an empty list, a test specification or
+ <c>{skip,Reason}</c>. If an empty list is returned, it means
+ that the test case shall be executed, and so it must also have
+ an execution clause. Note that the specification clause is
+ always executed on the controller node, i.e. not on the target
+ node.
+ </item>
+ <tag><em>test case</em></tag>
+ <item>A single test included in a test suite. Typically it tests
+ one function in a module or application. A test case is
+ implemented as a function in a test suite module. The function
+ can have three clauses, the documentation-, specification- and
+ execution clause.
+ </item>
+ <tag><em>test specification</em></tag>
+ <item>A specification of which test suites and test cases to
+ run. There can be test specifications on three different levels
+ in a test. The top level is a test specification file which
+ roughly specifies what to test for a whole application. Then
+ there is a test specification for each test suite returned from
+ the <c>all(suite)</c> function in the suite. And there can also
+ be a test specification returned from the specification clause
+ of a test case.
+ </item>
+ <tag><em>test specification file</em></tag>
+ <item>This is a text file containing the test specification for
+ an application. The file has the extension ".spec" or
+ ".spec.Platform", where Platform is e.g. "vxworks".
+ </item>
+ <tag><em>test suite</em></tag>
+ <item>An erlang module containing a collection of test cases for
+ a specific application or module.
+ </item>
+ <tag><em>topcase</em></tag>
+ <item>The first "command" in a test specification file. This
+ command contains the test specification, like this:
+ <c>{topcase,TestSpecification}</c></item>
+ </taglist>
+ </section>
+</chapter>
+