diff options
Diffstat (limited to 'lib/test_server/doc/src/test_server.xml')
| -rw-r--r-- | lib/test_server/doc/src/test_server.xml | 840 |
1 files changed, 840 insertions, 0 deletions
diff --git a/lib/test_server/doc/src/test_server.xml b/lib/test_server/doc/src/test_server.xml new file mode 100644 index 0000000000..6e75425862 --- /dev/null +++ b/lib/test_server/doc/src/test_server.xml @@ -0,0 +1,840 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE erlref SYSTEM "erlref.dtd"> + +<erlref> + <header> + <copyright> + <year>2007</year> + <year>2008</year> + <holder>Ericsson AB, All Rights Reserved</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + The Initial Developer of the Original Code is Ericsson AB. + </legalnotice> + + <title>test_server</title> + <prepared>Siri Hansen</prepared> + <responsible></responsible> + <docno></docno> + <approved></approved> + <checked></checked> + <date></date> + <rev></rev> + <file>test_server_ref.sgml</file> + </header> + <module>test_server</module> + <modulesummary>This module provides support for test suite authors.</modulesummary> + <description> + <p>The <c>test_server</c> module aids the test suite author by providing + various support functions. The supported functionality includes: + </p> + <list type="bulleted"> + <item>Logging and timestamping + </item> + <item>Capturing output to stdout + </item> + <item>Retrieving and flushing the message queue of a process + </item> + <item>Watchdog timers, process sleep, time measurement and unit + conversion + </item> + <item>Private scratch directory for all test suites + </item> + <item>Start and stop of slave- or peer nodes</item> + </list> + <p>For more information on how to write test cases and for + examples, please see the Test Server User's Guide. + </p> + </description> + + <section> + <title>TEST SUITE SUPPORT FUNCTIONS</title> + <p>The following functions are supposed to be used inside a test + suite. + </p> + </section> + <funcs> + <func> + <name>os_type() -> OSType</name> + <fsummary>Returns the OS type of the target node</fsummary> + <type> + <v>OSType = term()</v> + <d>This is the same as returned from <c>os:type/0</c></d> + </type> + <desc> + <p>This function can be called on controller or target node, and + it will always return the OS type of the target node.</p> + </desc> + </func> + <func> + <name>fail()</name> + <name>fail(Reason)</name> + <fsummary>Makes the test case fail.</fsummary> + <type> + <v>Reason = term()</v> + <d>The reason why the test case failed.</d> + </type> + <desc> + <p>This will make the test suite fail with a given reason, or + with <c>suite_failed</c> if no reason was given. Use this + function if you want to terminate a test case, as this will + make it easier to read the log- and HTML files. <c>Reason</c> + will appear in the comment field in the HTML log.</p> + </desc> + </func> + <func> + <name>timetrap(Timout) -> Handle</name> + <fsummary></fsummary> + <type> + <v>Timeout = integer() | {hours,H} | {minutes,M} | {seconds,S}</v> + <v>H = M = S = integer()</v> + <v>Pid = pid()</v> + <d>The process that is to be timetrapped (<c>self()</c>by default)</d> + </type> + <desc> + <p>Sets up a time trap for the current process. An expired + timetrap kills the process with reason + <c>timetrap_timeout</c>. The returned handle is to be given + as argument to <c>timetrap_cancel</c> before the timetrap + expires. If <c>Timeout</c> is an integer, it is expected to + be milliseconds.</p> + <note> + <p>If the current process is trapping exits, it will not be killed + by the exit signal with reason <c>timetrap_timeout</c>. + If this happens, the process will be sent an exit signal + with reason <c>kill</c> 10 seconds later which will kill the + process. Information about the timetrap timeout will in + this case not be found in the test logs. However, the + error_logger will be sent a warning.</p> + </note> + </desc> + </func> + <func> + <name>timetrap_cancel(Handle) -> ok</name> + <fsummary>Cancels a timetrap.</fsummary> + <type> + <v>Handle = term()</v> + <d>Handle returned from <c>timetrap</c></d> + </type> + <desc> + <p>This function cancels a timetrap. This must be done before + the timetrap expires.</p> + </desc> + </func> + <func> + <name>timetrap_scale_factor() -> ScaleFactor</name> + <fsummary>Returns the scale factor for timeouts.</fsummary> + <type> + <v>ScaleFactor = integer()</v> + </type> + <desc> + <p>This function returns the scale factor by which all timetraps + are scaled. It is normally 1, but can be greater than 1 if + the test_server is running <c>cover</c>, using a larger amount of + scheduler threads than the amount of logical processors on the + system, running under purify, valgrind or in a debug-compiled + emulator. The scale factor can be used if you need to scale you + own timeouts in test cases with same factor as the test_server + uses.</p> + </desc> + </func> + <func> + <name>sleep(MSecs) -> ok</name> + <fsummary>Suspens the calling task for a specified time.</fsummary> + <type> + <v>MSecs = integer() | float() | infinity</v> + <d>The number of milliseconds to sleep</d> + </type> + <desc> + <p>This function suspends the calling process for at least the + supplied number of milliseconds. There are two major reasons + why you should use this function instead of + <c>timer:sleep</c>, the first being that the module + <c>timer</c> may be unavailable at the time the test suite is + run, and the second that it also accepts floating point + numbers.</p> + </desc> + </func> + <func> + <name>hours(N) -> MSecs</name> + <name>minutes(N) -> MSecs</name> + <name>seconds(N) -> MSecs</name> + <fsummary></fsummary> + <type> + <v>N = integer()</v> + <d>Value to convert to milliseconds.</d> + </type> + <desc> + <p>Theese functions convert <c>N</c> number of hours, minutes + or seconds into milliseconds. + </p> + <p>Use this function when you want to + <c>test_server:sleep/1</c> for a number of seconds, minutes or + hours(!).</p> + </desc> + </func> + <func> + <name>format(Format) -> ok</name> + <name>format(Format, Args)</name> + <name>format(Pri,Format)</name> + <name>format(Pri, Format, Args)</name> + <fsummary></fsummary> + <type> + <v>Format = string()</v> + <d>Format as described for <c>io_:format</c>.</d> + <v>Args = list()</v> + <d>List of arguments to format.</d> + </type> + <desc> + <p>Formats output just like <c>io:format</c> but sends the + formatted string to a logfile. If the urgency value, + <c>Pri</c>, is lower than some threshold value, it will also + be written to the test person's console. Default urgency is + 50, default threshold for display on the console is 1. + </p> + <p>Typically, the test person don't want to see everything a + test suite outputs, but is merely interested in if the test + cases succeeded or not, which the test server tells him. If he + would like to see more, he could manually change the threshold + values by using the <c>test_server_ctrl:set_levels/3</c> + function.</p> + </desc> + </func> + <func> + <name>capture_start() -> ok</name> + <name>capture_stop() -> ok</name> + <name>capture_get() -> list()</name> + <fsummary>Captures all output to stdout for a process.</fsummary> + <desc> + <p>These functions makes it possible to capture all output to + stdout from a process started by the test suite. The list of + characters captured can be purged by using <c>capture_get</c>.</p> + </desc> + </func> + <func> + <name>messages_get() -> list()</name> + <fsummary>Empty the message queue.</fsummary> + <desc> + <p>This function will empty and return all the messages + currently in the calling process' message queue.</p> + </desc> + </func> + <func> + <name>timecall(M, F, A) -> {Time, Value}</name> + <fsummary>Measures the time needed to call a function.</fsummary> + <type> + <v>M = atom()</v> + <d>The name of the module where the function resides.</d> + <v>F = atom()</v> + <d>The name of the function to call in the module.</d> + <v>A = list()</v> + <d>The arguments to supply the called function.</d> + <v>Time = integer()</v> + <d>The number of seconds it took to call the function.</d> + <v>Value = term()</v> + <d>Value returned from the called function.</d> + </type> + <desc> + <p>This function measures the time (in seconds) it takes to + call a certain function. The function call is <em>not</em> + caught within a catch.</p> + </desc> + </func> + <func> + <name>do_times(N, M, F, A) -> ok</name> + <name>do_times(N, Fun)</name> + <fsummary>Calls MFA or Fun N times.</fsummary> + <type> + <v>N = integer()</v> + <d>Number of times to call MFA.</d> + <v>M = atom()</v> + <d>Module name where the function resides.</d> + <v>F = atom()</v> + <d>Function name to call.</d> + <v>A = list()</v> + <d>Arguments to M:F.</d> + </type> + <desc> + <p>Calls MFA or Fun N times. Useful for extensive testing of a + sensitive function.</p> + </desc> + </func> + <func> + <name>m_out_of_n(M, N, Fun) -> ok | exit({m_out_of_n_failed, {R,left_to_do}}</name> + <fsummary>Fault tolerant <c>do_times</c>.</fsummary> + <type> + <v>N = integer()</v> + <d>Number of times to call the Fun.</d> + <v>M = integer()</v> + <d>Number of times to require a successful return.</d> + </type> + <desc> + <p>Repeatedly evaluates the given function until it succeeds + (doesn't crash) M times. If, after N times, M successful + attempts have not been accomplished, the process crashes with + reason {m_out_of_n_failed, {R,left_to_do}}, where R indicates + how many cases that was still to be successfully completed. + </p> + <p>For example: + </p> + <p><c>m_out_of_n(1,4,fun() -> tricky_test_case() end)</c> <br></br> +Tries to run tricky_test_case() up to 4 times, and is + happy if it succeeds once. + </p> + <p><c>m_out_of_n(7,8,fun() -> clock_sanity_check() end)</c> <br></br> +Tries running clock_sanity_check() up to 8 times,and + allows the function to fail once. This might be useful if + clock_sanity_check/0 is known to fail if the clock crosses an + hour boundary during the test (and the up to 8 test runs could + never cross 2 boundaries)</p> + </desc> + </func> + <func> + <name>call_crash(M, F, A) -> Result</name> + <name>call_crash(Time, M, F, A) -> Result</name> + <name>call_crash(Time, Crash, M, F, A) -> Result</name> + <fsummary>Calls MFA and succeeds if it crashes.</fsummary> + <type> + <v>Result = ok | exit(call_crash_timeout) | exit({wrong_crash_reason, Reason})</v> + <v>Crash = term()</v> + <d>Crash return from the function.</d> + <v>Time = integer()</v> + <d>Timeout in milliseconds.</d> + <v>M = atom()</v> + <d>Module name where the function resides.</d> + <v>F = atom()</v> + <d>Function name to call.</d> + <v>A = list()</v> + <d>Arguments to M:F.</d> + </type> + <desc> + <p>Spawns a new process that calls MFA. The call is considered + successful if the call crashes with the gives reason + (<c>Crash</c>) or any reason if not specified. The call must + terminate within the given time (default <c>infinity</c>), or + it is considered a failure.</p> + </desc> + </func> + <func> + <name>temp_name(Stem) -> Name</name> + <fsummary>Returns a unique filename.</fsummary> + <type> + <v>Stem = string()</v> + </type> + <desc> + <p>Returns a unique filename starting with <c>Stem</c> with + enough extra characters appended to make up a unique + filename. The filename returned is guaranteed not to exist in + the filesystem at the time of the call.</p> + </desc> + </func> + <func> + <name>break(Comment) -> ok</name> + <fsummary>Cancel all timetraps and wait for call to continue/0.</fsummary> + <type> + <v>Comment = string()</v> + </type> + <desc> + <p><c>Comment</c> is a string which will be written in + the shell, e.g. explaining what to do.</p> + <p>This function will cancel all timetraps and pause the + execution of the test case until the user executes the + <c>continue/0</c> function. It gives the user the opportunity + to interact with the erlang node running the tests, e.g. for + debugging purposes or for manually executing a part of the + test case.</p> + <p>When the <c>break/1</c> function is called, the shell will + look something like this:</p> + <code type="none"><![CDATA[ + --- SEMIAUTOMATIC TESTING --- + The test case executes on process <0.51.0> + + + "Here is a comment, it could e.g. instruct to pull out a card" + + + ----------------------------- + + Continue with --> test_server:continue(). ]]></code> + <p>The user can now interact with the erlang node, and when + ready call <c>test_server:continue().</c></p> + <p>Note that this function can not be used if the test is + executed with <c>ts:run/0/1/2/3/4</c> in <c>batch</c> mode.</p> + </desc> + </func> + <func> + <name>continue() -> ok</name> + <fsummary>Continue after break/1.</fsummary> + <desc> + <p>This function must be called in order to continue after a + test case has called <c>break/1</c>.</p> + </desc> + </func> + <func> + <name>run_on_shielded_node(Fun, CArgs) -> term()</name> + <fsummary>Execute a function a shielded node.</fsummary> + <type> + <v>Fun = function() (arity 0)</v> + <d>Function to execute on the shielded node.</d> + <v>CArg = string()</v> + <d>Extra command line arguments to use when starting the shielded node.</d> + </type> + <desc> + <p><c>Fun</c> is executed in a process on a temporarily created + hidden node with a proxy for communication with the test server + node. The node is called a shielded node (should have been called + a shield node). If <c>Fun</c> is successfully executed, the result + is returned. A peer node (see <c>start_node/3</c>) started from + the shielded node will be shielded from test server node, i.e. + they will not be aware of each other. This is useful when you want + to start nodes from earlier OTP releases than the OTP release of + the test server node.</p> + <p>Nodes from an earlier OTP release can normally not be started + if the test server hasn't been started in compatibility mode + (see the <c>+R</c> flag in the <c>erl(1)</c> documentation) of + an earlier release. If a shielded node is started in compatibility + mode of an earlier OTP release than the OTP release of the test + server node, the shielded node can start nodes of an earlier OTP + release.</p> + <note> + <p>You <em>must</em> make sure that nodes started by the shielded + node never communicate directly with the test server node.</p> + </note> + <note> + <p>Slave nodes always communicate with the test server node; + therefore, <em>never</em> start <em>slave nodes</em> from the + shielded node, <em>always</em> start <em>peer nodes</em>.</p> + </note> + </desc> + </func> + <func> + <name>start_node(Name, Type, Options) -> {ok, Node} | {error, Reason}</name> + <fsummary>Start a node.</fsummary> + <type> + <v>Name = atom() | string()</v> + <d>Name of the slavenode to start (as given to -sname or -name)</d> + <v>Type = slave | peer</v> + <d>The type of node to start.</d> + <v>Options = [{atom(), term()]</v> + <d>Tuplelist of options</d> + </type> + <desc> + <p>This functions starts a node, possibly on a remote machine, + and guarantees cross architecture transparency. Type is set to + either <c>slave</c> or <c>peer</c>. + </p> + <p><c>slave</c> means that the new node will have a master, + i.e. the slave node will terminate if the master terminates, + TTY output produced on the slave will be sent back to the + master node and file I/O is done via the master. The master is + normally the target node unless the target is itself a slave. + </p> + <p><c>peer</c> means that the new node is an independent node + with no master. + </p> + <p><c>Options</c> is a tuplelist which can contain one or more + of + </p> + <taglist> + <tag><c>{remote, true}</c></tag> + <item>Start the node on a remote host. If not specified, the + node will be started on the local host (with some + exceptions, as for the case of VxWorks, where + all nodes are started on a remote host). Test cases that + require a remote host will fail with a reasonable comment if + no remote hosts are available at the time they are run. + </item> + <tag><c>{args, Arguments}</c></tag> + <item>Arguments passed directly to the node. This is + typically a string appended to the command line. + </item> + <tag><c>{wait, false}</c></tag> + <item>Don't wait until the node is up. By default, this + function does not return until the node is up and running, + but this option makes it return as soon as the node start + command is given.. + <br></br> +Only valid for peer nodes + </item> + <tag><c>{fail_on_error, false}</c></tag> + <item>Returns <c>{error, Reason}</c> rather than failing the + test case. + <br></br> +Only valid for peer nodes. Note that slave nodes always + act as if they had <c>fail_on_error=false</c></item> + <tag><c>{erl, ReleaseList}</c></tag> + <item>Use an Erlang emulator determined by ReleaseList when + starting nodes, instead of the same emulator as the test + server is running. ReleaseList is a list of specifiers, + where a specifier is either {release, Rel}, {prog, Prog}, or + 'this'. Rel is either the name of a release, e.g., "r12b_patched" + or 'latest'. 'this' means using the same emulator as the test + server. Prog is the name of an emulator executable. If the + list has more than one element, one of them is picked + randomly. (Only works on Solaris and Linux, and the test server + gives warnings when it notices that nodes are not of the same + version as itself.) + <br></br> + <br></br> + + When specifying this option to run a previous release, use + <c>is_release_available/1</c> function to test if the given + release is available and skip the test case if not. + <br></br> + <br></br> + + In order to avoid compatibility problems (may not appear right + away), use a shielded node (see <c>run_on_shielded_node/2</c>) + when starting nodes from different OTP releases than the test + server. + </item> + <tag><c>{cleanup, false}</c></tag> + <item>Tells the test server not to kill this node if it is + still alive after the test case is completed. This is useful + if the same node is to be used by a group of test cases. + </item> + <tag><c>{env, Env}</c></tag> + <item><c>Env</c> should be a list of tuples <c>{Name, Val}</c>, + where <c>Name</c> is the name of an environment variable, and + <c>Val</c> is the value it is to have in the started node. + Both <c>Name</c> and <c>Val</c> must be strings. The one + exception is <c>Val</c> being the atom <c>false</c> (in + analogy with <c>os:getenv/1</c>), which removes the + environment variable. Only valid for peer nodes. Not + available on VxWorks.</item> + </taglist> + </desc> + </func> + <func> + <name>stop_node(NodeName) -> bool()</name> + <fsummary>Stops a node</fsummary> + <type> + <v>NodeName = term()</v> + <d>Name of the node to stop</d> + </type> + <desc> + <p>This functions stops a node previously started with + <c>start_node/3</c>. Use this function to stop any node you + start, or the test server will produce a warning message in + the test logs, and kill the nodes automatically unless it was + started with the <c>{cleanup, false}</c> option.</p> + </desc> + </func> + <func> + <name>is_commercial() -> bool()</name> + <fsummary>Tests whether the emulator is commercially supported</fsummary> + <desc> + <p>This function test whether the emulator is commercially supported + emulator. The tests for a commercially supported emulator could be more + stringent (for instance, a commercial release should always contain + documentation for all applications).</p> + </desc> + </func> + + <func> + <name>is_release_available(Release) -> bool()</name> + <fsummary>Tests whether a release is available</fsummary> + <type> + <v>Release = string() | atom()</v> + <d>Release to test for</d> + </type> + <desc> + <p>This function test whether the release given by + <c>Release</c> (for instance, "r12b_patched") is available + on the computer that the test_server controller is running on. + Typically, you should skip the test case if not.</p> + <p>Caution: This function may not be called from the <c>suite</c> + clause of a test case, as the test_server will deadlock.</p> + </desc> + </func> + <func> + <name>is_native(Mod) -> bool()</name> + <fsummary>Checks whether the module is natively compiled or not</fsummary> + <type> + <v>Mod = atom()</v> + <d>A module name</d> + </type> + <desc> + <p>Checks whether the module is natively compiled or not</p> + </desc> + </func> + <func> + <name>app_test(App) -> ok | test_server:fail()</name> + <name>app_test(App,Mode)</name> + <fsummary>Checks an applications .app file for obvious errors</fsummary> + <type> + <v>App = term()</v> + <d>The name of the application to test</d> + <v>Mode = pedantic | tolerant</v> + <d>Default is pedantic</d> + </type> + <desc> + <p>Checks an applications .app file for obvious errors. + The following is checked: + </p> + <list type="bulleted"> + <item>required fields + </item> + <item>that all modules specified actually exists + </item> + <item>that all requires applications exists + </item> + <item>that no module included in the application has export_all + </item> + <item>that all modules in the ebin/ dir is included (If + <c>Mode==tolerant</c> this only produces a warning, as all + modules does not have to be included)</item> + </list> + </desc> + </func> + <func> + <name>comment(Comment) -> ok</name> + <fsummary>Print a comment on the HTML result page</fsummary> + <type> + <v>Comment = string()</v> + </type> + <desc> + <p>The given String will occur in the comment field of the + table on the HTML result page. If called several times, only + the last comment is printed. comment/1 is also overwritten by + the return value {comment,Comment} from a test case or by + fail/1 (which prints Reason as a comment).</p> + </desc> + </func> + </funcs> + + <section> + <title>TEST SUITE EXPORTS</title> + <p>The following functions must be exported from a test suite + module. + </p> + </section> + <funcs> + <func> + <name>all(suite) -> TestSpec | {skip, Comment}</name> + <fsummary>Returns the module's test specification</fsummary> + <type> + <v>TestSpec = list()</v> + <v>Comment = string()</v> + <d>This comment will be printed on the HTML result page</d> + </type> + <desc> + <p>This function must return the test specification for the + test suite module. The syntax of a test specification is + described in the Test Server User's Guide.</p> + </desc> + </func> + <func> + <name>init_per_suite(Config0) -> Config1 | {skip, Comment}</name> + <fsummary>Test suite initiation</fsummary> + <type> + <v>Config0 = Config1 = [tuple()]</v> + <v>Comment = string()</v> + <d>Describes why the suite is skipped</d> + </type> + <desc> + <p>This function is called before all other test cases in the + suite. <c>Config</c> is the configuration which can be modified + here. Whatever is returned from this function is given as + <c>Config</c> to the test cases. + </p> + <p>If this function fails, all test cases in the suite will be + skipped.</p> + </desc> + </func> + <func> + <name>end_per_suite(Config) -> void()</name> + <fsummary>Test suite finalization</fsummary> + <type> + <v>Config = [tuple()]</v> + </type> + <desc> + <p>This function is called after the last test case in the + suite, and can be used to clean up whatever the test cases + have done. The return value is ignored.</p> + </desc> + </func> + <func> + <name>init_per_testcase(Case, Config0) -> Config1 | {skip, Comment}</name> + <fsummary>Test case initiation</fsummary> + <type> + <v>Case = atom()</v> + <v>Config0 = Config1 = [tuple()]</v> + <v>Comment = string()</v> + <d>Describes why the test case is skipped</d> + </type> + <desc> + <p>This function is called before each test case. The + <c>Case</c> argument is the name of the test case, and + <c>Config</c> is the configuration which can be modified + here. Whatever is returned from this function is given as + <c>Config</c> to the test case.</p> + </desc> + </func> + <func> + <name>end_per_testcase(Case, Config) -> void()</name> + <fsummary>Test case finalization</fsummary> + <type> + <v>Case = atom()</v> + <v>Config = [tuple()]</v> + </type> + <desc> + <p>This function is called after each test case, and can be + used to clean up whatever the test case has done. The return + value is ignored.</p> + </desc> + </func> + <func> + <name>Case(doc) -> [Decription]</name> + <name>Case(suite) -> [] | TestSpec | {skip, Comment}</name> + <name>Case(Config) -> {skip, Comment} | {comment, Comment} | Ok</name> + <fsummary>A test case</fsummary> + <type> + <v>Description = string()</v> + <d>Short description of the test case</d> + <v>TestSpec = list()</v> + <v>Comment = string()</v> + <d>This comment will be printed on the HTML result page</d> + <v>Ok = term()</v> + <v>Config = [tuple()]</v> + <d>Elements from the Config parameter can be read with the ?config macro, see section about test suite support macros</d> + </type> + <desc> + <p>The <em>documentation clause</em> (argument <c>doc</c>) can + be used for automatic generation of test documentation or test + descriptions. + </p> + <p>The <em>specification clause</em> (argument <c>spec</c>) + shall return an empty list, the test specification for the + test case or <c>{skip,Comment}</c>. The syntax of a test + specification is described in the Test Server User's Guide. + </p> + <p><em>Note that the specification clause always is executed on the controller host.</em></p> + <p>The <em>execution clause</em> (argument <c>Config</c>) is + only called if the specification clause returns an empty list. + The execution clause is the real test case. Here you must call + the functions you want to test, and do whatever you need to + check the result. If something fails, make sure the process + crashes or call <c>test_server:fail/0/1</c> (which also will + cause the process to crash). + </p> + <p>You can return <c>{skip,Comment}</c> if you decide not to + run the test case after all, e.g. if it is not applicable on + this platform. + </p> + <p>You can return <c>{comment,Comment}</c> if you wish to + print some information in the 'Comment' field on the HTML + result page. + </p> + <p>If the execution clause returns anything else, it is + considered a success, unless it is <c>{'EXIT',Reason}</c> or + <c>{'EXIT',Pid,Reason}</c> which can't be distinguished from a + crash, and thus will be considered a failure. + </p> + <p>A <em>conf test case</em> is a group of test cases with an + init and a cleanup function. The init and cleanup functions + are also test cases, but they have special rules:</p> + <list type="bulleted"> + <item>They do not need a specification clause.</item> + <item>They must always have the execution clause.</item> + <item>They must return the <c>Config</c> parameter, a modified + version of it or <c>{skip,Comment}</c> from the execution clause.</item> + <item>The cleanup function may also return a tuple + <c>{return_group_result,Status}</c>, which is used to return the + status of the conf case to Test Server and/or to a conf case on a + higher level. (<c>Status = ok | skipped | failed</c>).</item> + <item><c>init_per_testcase</c> and <c>end_per_testcase</c> are + not called before and after these functions.</item> + </list> + </desc> + </func> + </funcs> + + <section> + <title>TEST SUITE LINE NUMBERS</title> + <p>If a test case fails, the test server can report the exact line + number at which it failed. There are two ways of doing this, + either by using the <c>line</c> macro or by using the + <c>test_server_line</c> parse transform. + </p> + <p>The <c>line</c> macro is described under TEST SUITE SUPPORT + MACROS below. The <c>line</c> macro will only report the last line + executed when a test case failed. + </p> + <p>The <c>test_server_line</c> parse transform is activated by + including the headerfile <c>test_server_line.hrl</c> in the test + suite. When doing this, it is important that the + <c>test_server_line</c> module is in the code path of the erlang + node compiling the test suite. The parse transform will report a + history of a maximum of 10 lines when a test case + fails. Consecutive lines in the same function are not shown. + </p> + <p>The attribute <c>-no_lines(FuncList).</c> can be used in the + test suite to exclude specific functions from the parse + transform. This is necessary e.g. for functions that are executed + on old (i.e. <R10B) OTP releases. <c>FuncList = [{Func,Arity}]</c>. + </p> + <p>If both the <c>line</c> macro and the parse transform is used in + the same module, the parse transform will overrule the macro. + </p> + </section> + + <section> + <title>TEST SUITE SUPPORT MACROS</title> + <p>There are some macros defined in the <c>test_server.hrl</c> + that are quite useful for test suite programmers: + </p> + <p>The <em>line</em> macro, is quite + essential when writing test cases. It tells the test server + exactly what line of code that is being executed, so that it can + report this line back if the test case fails. Use this macro at + the beginning of every test case line of code. + </p> + <p>The <em>config</em> macro, is used to + retrieve information from the <c>Config</c> variable sent to all + test cases. It is used with two arguments, where the first is the + name of the configuration variable you wish to retrieve, and the + second is the <c>Config</c> variable supplied to the test case + from the test server. + </p> + <p>Possible configuration variables include:</p> + <list type="bulleted"> + <item><c>data_dir</c> - Data file directory.</item> + <item><c>priv_dir</c> - Scratch file directory.</item> + <item><c>nodes</c> - Nodes specified in the spec file</item> + <item><c>nodenames</c> - Generated nodenames.</item> + <item>Whatever added by conf test cases or + <c>init_per_testcase/2</c></item> + </list> + <p>Examples of the <c>line</c> and <c>config</c> macros can be + seen in the Examples chapter in the user's guide. + </p> + <p>If the <c>line_trace</c> macro is defined, you will get a + timestamp (<c>erlang:now()</c>) in your minor log for each + <c>line</c> macro in your suite. This way you can at any time see + which line is currently being executed, and when the line was + called. + </p> + <p>The <c>line_trace</c> macro can also be used together with the + <c>test_server_line</c> parse transform described above. A + timestamp will then be written for each line in the suite, except + for functions stated in the <c>-no_lines</c> attribute. + </p> + <p>The <c>line_trace</c> macro can e.g. be defined as a compile + option, like this: + <br></br> +<c>erlc -W -Dline_trace my_SUITE.erl</c></p> + </section> +</erlref> + |