diff options
Diffstat (limited to 'lib/test_server/doc/src/basics_chapter.xml')
| -rw-r--r-- | lib/test_server/doc/src/basics_chapter.xml | 216 |
1 files changed, 216 insertions, 0 deletions
diff --git a/lib/test_server/doc/src/basics_chapter.xml b/lib/test_server/doc/src/basics_chapter.xml new file mode 100644 index 0000000000..a96cc88075 --- /dev/null +++ b/lib/test_server/doc/src/basics_chapter.xml @@ -0,0 +1,216 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2002</year><year>2009</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Test Server Basics</title> + <prepared>Siri Hansen</prepared> + <docno></docno> + <date></date> + <rev></rev> + <file>basics_chapter.xml</file> + </header> + + <section> + <title>Introduction</title> + <p><em>Test Server</em> is a portable test tool for automated + testing of Erlang programs and OTP applications. It provides an + interface for running test programs directly with Test Server + as well as an interface for integrating Test Server + with a framework application. The latter makes it possible to use + Test Server as the engine of a higher level test tool + application.</p> + + <p>It is strongly recommended that Test Server be used from inside + a framework application, rather than interfaced directly for + running test programs. Test Server can be pretty difficult to use + since it's a very general and quite extensive and complex + application. Furthermore, the <c>test_server_ctrl</c> functions + are not meant to be used from within the actual test programs. The + framework should handle communication with Test Server and deal + with the more complex aspects of this interaction automatically so + that a higher level interface may be provided for the tester. For + test tool usage to be productive, a simpler, more intuitive and + (if required) more specific interface is required than what Test Server + can provide.</p> + + <p>OTP delivers a general purpose framework for Test Server, called + <em>Common Test</em>. This application is a tool well suited for + automated black box testing of target systems of <em>any kind</em> + (not necessarily implemented in Erlang). Common Test is also a very + useful tool for white box testing of Erlang programs and OTP + applications. Unless a more specific functionality and/or user + interface is required (in which case you might need to implement + your own framework), Common Test should do the job for + you. Please read the Common Test User's Guide and reference manual + for more information.</p> + + <p>Under normal circumstances, knowledge about the Test Server + application is not required for using the Common Test framework. + However, if you want to use Test Server without a framework, + or learn how to integrate it with your own framework, please read on... + </p> + </section> + <section> + <title>Getting started</title> + <p>Testing when using Test Server is done by running test + suites. A test suite is a number of test cases, where each test + case tests one or more things. The test case is the smallest unit + that the test server deals with. One or more test cases are + grouped together into one ordinary Erlang module, which is called + a test suite. Several test suite modules can be grouped together + in special test specification files representing whole application + and/or system test "jobs". + </p> + <p>The test suite Erlang module must follow a certain interface, + which is specified by Test Server. See the section on writing + test suites for details about this. + </p> + <p>Each test case is considered a success if it returns to the + caller, no matter what the returned value is. An exception to this + is the return value <c>{skip, Reason}</c> which indicates that the + test case is skipped. A failure is specified as a crash, no matter + what the crash reason is. + </p> + <p>As a test suite runs, all information (including output to + stdout) is recorded in several different log files. A minimum of + information is displayed to the user console. This only include + start and stop information, plus a note for each failed test case. + </p> + <p>The result from each test case is recorded in an HTML log file + which is created for each test run. Every test case gets one row + in a table presenting total time, whether the case was successful + or not, if it was skipped, and possibly also a comment. The HTML + file has links to each test case's logfile, which may be viewed + from e.g. Netscape or any other HTML capable browser. + </p> + <p>The Test Server consists of three parts: + </p> + <list type="bulleted"> + <item>The part that executes the test suites on target and + provides support for the test suite author is called + <c>test_server</c>. This is described in the chapter about + writing test cases in this user's guide, and in the reference + manual for the <c>test_server</c> module.</item> + <item>The controlling part, which provides the low level + operator interface, starts and stops the target node (if remote + target) and slave nodes and writes log files, is called + <c>test_server_ctrl</c>. The Test Server Controller should not + be used directly when running tests. Instead a framework built + on top of it should be used. More information + about how to write your own framework can be found + in this user's guide and in the reference manual for the + <c>test_server_ctrl</c> module.</item> + </list> + </section> + + <section> + <title>Definition of terms</title> + <taglist> + <tag><em>conf(iguration) case</em></tag> + <item>This is a group of test cases which need some specific + configuration. A conf case contains an initiation function which + sets up a specific configuration, one or more test cases using + this configuration, and a cleanup function which restores the + configuration. A conf case is specified in a test specification + either like this:<c>{conf,InitFunc,ListOfCases,CleanupFunc}</c>, + or this: <c>{conf,Properties,InitFunc,ListOfCases,CleanupFunc}</c> + </item> + <tag><em>datadir</em></tag> + <item>Data directory for a test suite. This directory contains + any files used by the test suite, e.g. additional erlang + modules, c code or data files. If the data directory contains + code which must be compiled before the test suite is run, it + should also contain a makefile source called Makefile.src + defining how to compile. + </item> + <tag><em>documentation clause</em></tag> + <item>One of the function clauses in a test case. This clause + shall return a list of strings describing what the test case + tests. + </item> + <tag><em>execution clause</em></tag> + <item>One of the function clauses in a test case. This clause + implements the actual test case, i.e. calls the functions that + shall be tested and checks results. The clause shall crash if it + fails. + </item> + <tag><em>major log file</em></tag> + <item>This is the test suites log file. + </item> + <tag><em>Makefile.src</em></tag> + <item>This file is used by the test server framework to generate + a makefile for a datadir. It contains some special characters + which are replaced according to the platform currently tested. + </item> + <tag><em>minor log file</em></tag> + <item>This is a separate log file for each test case. + </item> + <tag><em>privdir</em></tag> + <item>Private directory for a test suite. This directory should + be used when the test suite needs to write to files. + </item> + <tag><em>skip case</em></tag> + <item>A test case which shall be skipped. + </item> + <tag><em>specification clause</em></tag> + <item>One of the function clauses in a test case. This clause + shall return an empty list, a test specification or + <c>{skip,Reason}</c>. If an empty list is returned, it means + that the test case shall be executed, and so it must also have + an execution clause. Note that the specification clause is + always executed on the controller node, i.e. not on the target + node. + </item> + <tag><em>test case</em></tag> + <item>A single test included in a test suite. Typically it tests + one function in a module or application. A test case is + implemented as a function in a test suite module. The function + can have three clauses, the documentation-, specification- and + execution clause. + </item> + <tag><em>test specification</em></tag> + <item>A specification of which test suites and test cases to + run. There can be test specifications on three different levels + in a test. The top level is a test specification file which + roughly specifies what to test for a whole application. Then + there is a test specification for each test suite returned from + the <c>all(suite)</c> function in the suite. And there can also + be a test specification returned from the specification clause + of a test case. + </item> + <tag><em>test specification file</em></tag> + <item>This is a text file containing the test specification for + an application. The file has the extension ".spec" or + ".spec.Platform", where Platform is e.g. "vxworks". + </item> + <tag><em>test suite</em></tag> + <item>An erlang module containing a collection of test cases for + a specific application or module. + </item> + <tag><em>topcase</em></tag> + <item>The first "command" in a test specification file. This + command contains the test specification, like this: + <c>{topcase,TestSpecification}</c></item> + </taglist> + </section> +</chapter> + |