diff options
Diffstat (limited to 'lib/test_server/doc/src/write_test_chapter.xml')
| -rw-r--r-- | lib/test_server/doc/src/write_test_chapter.xml | 228 |
1 files changed, 0 insertions, 228 deletions
diff --git a/lib/test_server/doc/src/write_test_chapter.xml b/lib/test_server/doc/src/write_test_chapter.xml deleted file mode 100644 index c3e1881b8a..0000000000 --- a/lib/test_server/doc/src/write_test_chapter.xml +++ /dev/null @@ -1,228 +0,0 @@ -<?xml version="1.0" encoding="utf-8" ?> -<!DOCTYPE chapter SYSTEM "chapter.dtd"> - -<chapter> - <header> - <copyright> - <year>2002</year><year>2013</year> - <holder>Ericsson AB. All Rights Reserved.</holder> - </copyright> - <legalnotice> - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - - </legalnotice> - - <title>Writing Test Suites</title> - <prepared>Siri Hansen</prepared> - <docno></docno> - <date></date> - <rev></rev> - <file>write_test_chapter.xml</file> - </header> - - <section> - <title>Support for test suite authors</title> - <p>The <c>test_server</c> module provides some useful functions - to support the test suite author. This includes: - </p> - <list type="bulleted"> - <item>Starting and stopping slave or peer nodes</item> - <item>Capturing and checking stdout output</item> - <item>Retrieving and flushing process message queue</item> - <item>Watchdog timers</item> - <item>Checking that a function crashes</item> - <item>Checking that a function succeeds at least m out of n times</item> - <item>Checking .app files</item> - </list> - <p>Please turn to the reference manual for the <c>test_server</c> - module for details about these functions. - </p> - </section> - - <section> - <title>Test suites</title> - <p>A test suite is an ordinary Erlang module that contains test - cases. It's recommended that the module has a name on the form - *_SUITE.erl. Otherwise, the directory function will not find the - modules (by default). - </p> - <p>For some of the test server support, the test server include - file <c>test_server.hrl</c> must be included. Never include it - with the full path, for portability reasons. Use the compiler - include directive instead. - </p> - <p>The special function <c>all(suite)</c> in each module is called - to get the test specification for that module. The function - typically returns a list of test cases in that module, but any - test specification could be returned. Please see the chapter - about test specifications for details about this. - </p> - </section> - - <section> - <title>Init per test case</title> - <p>In each test suite module, the functions - <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> must be - implemented. - </p> - <p><c>init_per_testcase</c> is called before each test case in the - test suite, giving a (limited) possibility for initialization. - </p> - <p><c>end_per_testcase/2</c> is called after each test case is - completed, giving a possibility to clean up. - </p> - <p>The first argument to these functions is the name of the test - case. This can be used to do individual initialization and cleanup for - each test cases. - </p> - <p>The second argument is a list of tuples called - <c>Config</c>. The first element in a <c>Config</c> tuple - should be an atom - a key value to be used for searching. - <c>init_per_testcase/2</c> may modify the <c>Config</c> - parameter or just return it as is. Whatever is retuned by - <c>init_per_testcase/2</c> is given as <c>Config</c> parameter to - the test case itself. - </p> - <p>The return value of <c>end_per_testcase/2</c> is ignored by the - test server. - </p> - </section> - - <section> - <title>Test cases</title> - <p>The smallest unit that the test server is concerned with is a - test case. Each test case can in turn test many things, for - example make several calls to the same interface function with - different parameters. - </p> - <p>It is possible to put many or few tests into each test - case. How many things each test case tests is up to the author, - but here are some things to keep in mind. - </p> - <p>Very small test cases often leads to more code, since - initialization has to be duplicated. Larger code, especially with - a lot of duplication, increases maintenance and reduces - readability. - </p> - <p>Larger test cases make it harder to tell what went wrong if it - fails, and force us to skip larger portions of test code if a - specific part fails. These effects are accentuated when running on - multiple platforms because test cases often have to be skipped. - </p> - <p>A test case generally consists of three parts, the - documentation part, the specification part and the execution - part. These are implemented as three clauses of the same function. - </p> - <p>The documentation clause matches the argument '<c>doc</c>' and - returns a list for strings describing what the test case tests. - </p> - <p>The specification clause matches the argument '<c>suite</c>' - and returns the test specification for this particular test - case. If the test specification is an empty list, this indicates - that the test case is a leaf test case, i.e. one to be executed. - </p> - <p>The execution clause implements the actual test case. It takes - one argument, <c>Config</c>, which contain configuration - information like <c>data_dir</c> and <c>priv_dir</c>. See <seealso marker="#data_priv_dir">Data and Private Directories</seealso> for - more information about these. - </p> - <p>The <c>Config</c> variable can also contain the - <c>nodenames</c> key, if requested by the <c>require_nodenames</c> - command in the test suite specification file. All <c>Config</c> - items should be extracted using the <c>?config</c> macro. This is - to ensure future compatibility if the <c>Config</c> format - changes. See the reference manual for <c>test_server</c> for - details about this macro. - </p> - <p>If the execution clause crashes or exits, it is considered a - failure. If it returns <c>{skip,Reason}</c>, the test case is - considered skipped. If it returns <c>{comment,String}</c>, - the string will be added in the 'Comment' field on the HTML - result page. If the execution clause returns anything else, it is - considered a success, unless it is <c>{'EXIT',Reason}</c> or - <c>{'EXIT',Pid,Reason}</c> which can't be distinguished from a - crash, and thus will be considered a failure. - </p> - </section> - - <section> - <marker id="data_priv_dir"></marker> - <title>Data and Private Directories</title> - <p>The data directory (<c>data_dir</c>) is the directory where the test - module has its own files needed for the testing. A compiler test - case may have source files to feed into the compiler, a release - upgrade test case may have some old and new release of - something. A graphics test case may have some icons and a test - case doing a lot of math with bignums might store the correct - answers there. The name of the <c>data_dir</c> is the the name of - the test suite and then "_data". For example, - <c>"some_path/foo_SUITE.beam"</c> has the data directory - <c>"some_path/foo_SUITE_data/"</c>. - </p> - <p>The <c>priv_dir</c> is the test suite's private directory. This - directory should be used when a test case needs to write to - files. The name of the private directory is generated by the test - server, which also creates the directory. - </p> - <p><em>Warning:</em> Do not depend on current directory to be - writable, or to point to anything in particular. All scratch files - are to be written in the <c>priv_dir</c>, and all data files found - in <c>data_dir</c>. If the current directory has to be something - specific, it must be set with <c>file:set_cwd/1</c>. - </p> - </section> - - <section> - <title>Execution environment</title> - <p>Each time a test case is about to be executed, a new process is - created with <c>spawn_link</c>. This is so that the test case will - have no dependencies to earlier tests, with respect to process flags, - process links, messages in the queue, other processes having registered - the process, etc. As little as possible is done to change the initial - context of the process (what is created by plain spawn). Here is a - list of differences: - </p> - <list type="bulleted"> - <item>It has a link to the test server. If this link is removed, - the test server will not know when the test case is finished, - just wait infinitely. - </item> - <item>It often holds a few items in the process dictionary, all - with names starting with '<c>test_server_</c>'. This is to keep - track of if/where a test case fails. - </item> - <item>There is a top-level catch. All of the test case code is - catched, so that the location of a crash can be reported back to - the test server. If the test case process is killed by another - process (thus the catch code is never executed) the test server - is not able to tell where the test case was executing. - </item> - <item>It has a special group leader implemented by the test - server. This way the test server is able to capture the io that - the test case provokes. This is also used by some of the test - server support functions. - </item> - </list> - <p>There is no time limit for a test case, unless the test case - itself imposes such a limit, by calling - <c>test_server:timetrap/1</c> for example. The call can be made - in each test case, or in the <c>init_per_testcase/2</c> - function. Make sure to call the corresponding - <c>test_server:timetrap_cancel/1</c> function as well, e.g in the - <c>end_per_testcase/2</c> function, or else the test cases will - always fail. - </p> - </section> - -</chapter> - |