diff options
Diffstat (limited to 'lib/common_test/doc/src/write_test_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/write_test_chapter.xml | 170 |
1 files changed, 133 insertions, 37 deletions
diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index c0ec26ddcc..7b7e7af8ea 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -29,7 +29,6 @@ <file>write_test_chapter.xml</file> </header> - <section> <marker id="intro"></marker> <title>Support for test suite authors</title> @@ -63,7 +62,8 @@ function in CT will not be able to locate it (at least not per default). </p> - <p>The <c>ct.hrl</c> header file must be included in all test suite files. + <p>It is also recommended that the <c>ct.hrl</c> header file is included + in all test suite modules. </p> <p>Each test suite module must export the function <c>all/0</c> @@ -71,6 +71,12 @@ to be executed in that module. </p> + <p>The callback functions that the test suite should implement, and + which will be described in more detail below, are + all listed in the <seealso marker="common_test">common_test + reference manual page</seealso>. + </p> + </section> <section> @@ -113,6 +119,14 @@ suite will be skipped automatically (so called <em>auto skipped</em>), including <c>end_per_suite</c>. </p> + + <p>Note that if <c>init_per_suite</c> and <c>end_per_suite</c> do not exist + in the suite, Common Test calls dummy functions (with the same names) + instead, so that output generated by hook functions may be saved to the log + files for these dummies + (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test Hooks</seealso> + chapter for more information). + </p> </section> <marker id="per_testcase"/> @@ -513,8 +527,7 @@ execution is immediately stopped and the rest of the cases skipped.</p> <p>Before execution of a group begins, the configuration function - <c>init_per_group(GroupName, Config)</c> is called (the function is - mandatory if one or more test case groups are defined). The list of tuples + <c>init_per_group(GroupName, Config)</c> is called. The list of tuples returned from this function is passed to the test cases in the usual manner by means of the <c>Config</c> argument. <c>init_per_group/2</c> is meant to be used for initializations common for the test cases in the @@ -522,6 +535,14 @@ <c>end_per_group(GroupName, Config</c> function is called. This function is meant to be used for cleaning up after <c>init_per_group/2</c>.</p> + <p>Whenever a group is executed, if <c>init_per_group</c> and + <c>end_per_group</c> do not exist in the suite, Common Test calls + dummy functions (with the same names) instead. Output generated by + hook functions will be saved to the log files for these dummies + (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test + Hooks</seealso> chapter for more information). + </p> + <note><p><c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> are always called for each individual test case, no matter if the case belongs to a group or not.</p></note> @@ -611,6 +632,25 @@ </section> <section> + <title>Parallel test cases and IO</title> + <p>A parallel test case has a private IO server as its group leader. + (Please see the Erlang Run-Time System Application documentation for + a description of the group leader concept). The + central IO server process that handles the output from regular test + cases and configuration functions, does not respond to IO messages + during execution of parallel groups. This is important to understand + in order to avoid certain traps, like this one:</p> + <p>If a process, <c>P</c>, is spawned during execution of e.g. + <c>init_per_suite/1</c>, it will inherit the group leader of the + <c>init_per_suite</c> process. This group leader is the central IO server + process mentioned above. If, at a later time, <em>during parallel test case + execution</em>, some event triggers process <c>P</c> to call + <c>io:format/1/2</c>, that call will never return (since the group leader + is in a non-responsive state) and cause <c>P</c> to hang. + </p> + </section> + + <section> <title>Repeated groups</title> <marker id="repeated_groups"></marker> <p>A test case group may be repeated a certain number of times @@ -745,7 +785,7 @@ <marker id="data_priv_dir"></marker> <title>Data and Private Directories</title> - <p>The data directory (<c>data_dir</c>) is the directory where the + <p>The data directory, <c>data_dir</c>, is the directory where the test module has its own files needed for the testing. The name of the <c>data_dir</c> is the the name of the test suite followed by <c>"_data"</c>. For example, @@ -769,12 +809,39 @@ </p> --> <p> - The <c>priv_dir</c> is the test suite's private directory. This - directory should be used when a test case needs to write to - files. The name of the private directory is generated by the test - server, which also creates the directory. + <c>priv_dir</c> is the private directory for the test cases. + This directory may be used whenever a test case (or configuration function) + needs to write something to file. The name of the private directory is + generated by Common Test, which also creates the directory. </p> - + <p>By default, Common Test creates one central private directory + per test run that all test cases share. This may not always be suitable, + especially if the same test cases are executed multiple times during + a test run (e.g. if they belong to a test case group with repeat + property), and there's a risk that files in the private directory get + overwritten. Under these circumstances, it's possible to configure + Common Test to create one dedicated private directory per + test case and execution instead. This is accomplished by means of + the flag/option: <c>create_priv_dir</c> (to be used with the + <c>ct_run</c> program, the <c>ct:run_test/1</c> function, or + as test specification term). There are three possible values + for this option: + <list> + <item><c>auto_per_run</c></item> + <item><c>auto_per_tc</c></item> + <item><c>manual_per_tc</c></item> + </list> + The first value indicates the default priv_dir behaviour, i.e. + one private directory created per test run. The two latter + values tell Common Test to generate a unique test directory name + per test case and execution. If the auto version is used, <em>all</em> + private directories will be created automatically. This can obviously + become very inefficient for test runs with many test cases and/or + repetitions. Therefore, in case the manual version is instead used, the + test case must tell Common Test to create priv_dir when it needs it. + It does this by calling the function <c>ct:make_priv_dir/0</c>. + </p> + <note><p>You should not depend on current working directory for reading and writing data files since this is not portable. All scratch files are to be written in the <c>priv_dir</c> and all @@ -805,34 +872,63 @@ <marker id="timetraps"></marker> <title>Timetrap timeouts</title> <p>The default time limit for a test case is 30 minutes, unless a - <c>timetrap</c> is specified either by the suite info function - or a test case info function. The timetrap timeout value defined - in <c>suite/0</c> is the value that will be used for each test case - in the suite (as well as for the configuration functions - <c>init_per_suite/1</c> and <c>end_per_suite</c>). A timetrap timeout - value set with the test case info function will override the value set - by <c>suite/0</c>, but only for that particular test case.</p> - <p>It is also possible to set/reset a timetrap during test case (or - configuration function) execution. This is done by calling - <c>ct:timetrap/1</c>. This function will cancel the current timetrap - and start a new one.</p> + <c>timetrap</c> is specified either by the suite-, group-, + or test case info function. The timetrap timeout value defined by + <c>suite/0</c> is the value that will be used for each test case + in the suite (as well as for the configuration functions + <c>init_per_suite/1</c>, <c>end_per_suite/1</c>, <c>init_per_group/2</c>, + and <c>end_per_group/2</c>). A timetrap value defined by + <c>group(GroupName)</c> overrides one defined by <c>suite()</c> + and will be used for each test case in group <c>GroupName</c>, and any + of its sub-groups. If a timetrap value is defined by <c>group/1</c> + for a sub-group, it overrides that of its higher level groups. Timetrap + values set by individual test cases (by means of the test case info + function) overrides both group- and suite- level timetraps.</p> + + <p>It is also possible to dynamically set/reset a timetrap during the + excution of a test case, or configuration function. This is done by calling + <c>ct:timetrap/1</c>. This function cancels the current timetrap + and starts a new one (that stays active until timeout, or end of the + current function).</p> + <p>Timetrap values can be extended with a multiplier value specified at - startup with the <c>multiply_timetraps</c> option. It is also possible - to let Test Server decide to scale up timetrap timeout values - automatically, e.g. if tools such as cover or trace are running during - the test. This feature is disabled by default and can be enabled with - the <c>scale_timetraps</c> start option.</p> + startup with the <c>multiply_timetraps</c> option. It is also possible + to let the test server decide to scale up timetrap timeout values + automatically, e.g. if tools such as cover or trace are running during + the test. This feature is disabled by default and can be enabled with + the <c>scale_timetraps</c> start option.</p> + <p>If a test case needs to suspend itself for a time that also gets - multipled by <c>multiply_timetraps</c>, and possibly scaled up if - <c>scale_timetraps</c> is enabled, the function <c>ct:sleep/1</c> - may be called.</p> - <p>A function (<c>fun</c> or <c>MFA</c>) may be specified as timetrap value - in the suite- and test case info function, e.g:</p> - <p><c>{timetrap,{test_utils,get_timetrap_value,[?MODULE,system_start]}}</c></p> - <p>The function will be called initially by Common Test (before execution - of the suite or the test case) and must return a time value such as an - integer (millisec), or a <c>{SecMinOrHourTag,Time}</c> tuple. More - information can be found in the <c>common_test</c> reference manual.</p> + multipled by <c>multiply_timetraps</c> (and possibly also scaled up if + <c>scale_timetraps</c> is enabled), the function <c>ct:sleep/1</c> + may be used (instead of e.g. <c>timer:sleep/1</c>).</p> + + <p>A function (<c>fun/0</c> or <c>MFA</c>) may be specified as + timetrap value in the suite-, group- and test case info function, as + well as argument to the <c>ct:timetrap/1</c> function. Examples:</p> + + <p><c>{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}</c></p> + <p><c>ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)</c></p> + + <p>The user timetrap function may be used for two things:</p> + <list> + <item>To act as a timetrap - the timeout is triggered when the + function returns.</item> + <item>To return a timetrap time value (other than a function).</item> + </list> + <p>Before execution of the timetrap function (which is performed + on a parallel, dedicated timetrap process), Common Test cancels + any previously set timer for the test case or configuration function. + When the timetrap function returns, the timeout is triggered, <em>unless</em> + the return value is a valid timetrap time, such as an integer, + or a <c>{SecMinOrHourTag,Time}</c> tuple (see the + <seealso marker="common_test">common_test reference manual</seealso> for + details). If a time value is returned, a new timetrap is started + to generate a timeout after the specified time.</p> + + <p>The user timetrap function may of course return a time value after a delay, + and if so, the effective timetrap time is the delay time <em>plus</em> the + returned time.</p> </section> <section> |