diff options
author | Erlang/OTP <[email protected]> | 2010-06-11 09:08:08 +0000 |
---|---|---|
committer | Erlang/OTP <[email protected]> | 2010-06-11 09:08:08 +0000 |
commit | c0f23851df50d6215981805102754b590c05acec (patch) | |
tree | 5a202fe2abc7cd77e9808930758b81d05957fab6 /lib/common_test/doc/src/run_test_chapter.xml | |
parent | a28aa321007f62ae4dce415cd190778c3ac184e7 (diff) | |
parent | 2d7ba88ebbb59a473dbcefd0f9dee1f1b816935e (diff) | |
download | otp-c0f23851df50d6215981805102754b590c05acec.tar.gz otp-c0f23851df50d6215981805102754b590c05acec.tar.bz2 otp-c0f23851df50d6215981805102754b590c05acec.zip |
Merge branch 'peppe/common_test_r14_dev_2' into dev
* peppe/common_test_r14_dev_2:
Misc documentation updates
Add documentation for run_test program
Step vsn for test_server to 3.4
Make {repeat*,N} property in group execute the group N times exactly
Fix so that ct_run converts relative diretories in the code path to absolute
Fix bug in handling framework:end_tc timeouts
Fix bug that prevents the interactive shell mode to start properly
Fix failing multiply timetrap test case
Minor fixes in code and test suites
Add support for executing pre-loaded suites (e.g. modules loaded as binaries)
Add test suite for remote loading of binary suites
Fix error with {repeat,0} property in groups causing double iterations
Add support for config info functions (e.g. init_per_suite/0)
Add support for dynamic timetrap handling
Have end_per_testcase run even after timetrap_timeout and abort_testcase
Flush old DOWN messages in demonitor
Add groups in test specifications
Add new tests for test case groups and test specifications
Improve and fix various test suites
Add event_handler_init start flag that can pass init arguments to event handlers
...
OTP-8703 peppe/common_test_r14_dev_2
Diffstat (limited to 'lib/common_test/doc/src/run_test_chapter.xml')
-rw-r--r-- | lib/common_test/doc/src/run_test_chapter.xml | 78 |
1 files changed, 56 insertions, 22 deletions
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index d731d18783..207df7f5b5 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2009</year> + <year>2003</year><year>2010</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -13,12 +13,12 @@ compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/. - + Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - + </legalnotice> <title>Running Test Suites</title> @@ -97,25 +97,32 @@ the <c><![CDATA[{auto_compile,false}]]></c> option with <c><![CDATA[ct:run_test/1]]></c>. With automatic compilation disabled, the user is responsible for compiling the test suite modules - (and any help modules) before the test run. Common Test will only verify - that the specified test suites exist before starting the tests.</p> + (and any help modules) before the test run. If the modules can not be loaded + from the local file system during startup of Common Test, the user needs to + pre-load the modules before starting the test. Common Test will only verify + that the specified test suites exist (i.e. that they are, or can be, loaded). + This is useful e.g. if the test suites are transferred and loaded as binaries via + RPC from a remote node.</p> </section> <section> - <title>Running tests from the UNIX command line</title> + <title>Running tests from the OS command line</title> - <p>The script <c>run_test</c> can be used for running tests from - the Unix/Linux command line, e.g. + <p>The <c>run_test</c> program can be used for running tests from + the OS command line, e.g. </p> <list> <item><c><![CDATA[run_test -config <configfilenames> -dir <dirs>]]></c></item> <item><c><![CDATA[run_test -config <configfilenames> -suite <suiteswithfullpath>]]></c> </item> + <item><c><![CDATA[run_test -userconfig <callbackmodulename> <configfilenames> -suite <suiteswithfullpath>]]></c> + </item> <item><c><![CDATA[run_test -config <configfilenames> -suite <suitewithfullpath> -group <groupnames> -case <casenames>]]></c></item> </list> <p>Examples:</p> <p><c>$ run_test -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST</c></p> + <p><c>$ run_test -userconfig ct_config_xml $CFGS/sys1.xml $CFGS/sys2.xml -dir $SYS1_TEST $SYS2_TEST</c></p> <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE $SYS2_TEST/config_SUITE</c></p> <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE -case start stop</c></p> <p><c>$ run_test -suite $SYS1_TEST/setup_SUITE -group installation -case start stop</c></p> @@ -136,8 +143,14 @@ <seealso marker="cover_chapter#cover">Code Coverage Analysis</seealso>).</item> <item><c><![CDATA[-event_handler <event_handlers>]]></c>, to install <seealso marker="event_handler_chapter#event_handling">event handlers</seealso>.</item> + <item><c><![CDATA[-event_handler_init <event_handlers>]]></c>, to install + <seealso marker="event_handler_chapter#event_handling">event handlers</seealso> including start arguments.</item> <item><c><![CDATA[-include]]></c>, specifies include directories (see above).</item> <item><c><![CDATA[-no_auto_compile]]></c>, disables the automatic test suite compilation feature (see above).</item> + <item><c><![CDATA[-multiply_timetraps <n>]]></c>, extends <seealso marker="write_test_chapter#timetraps">timetrap + timeout</seealso> values.</item> + <item><c><![CDATA[-scale_timetraps <bool>]]></c>, enables automatic <seealso marker="write_test_chapter#timetraps">timetrap + timeout</seealso> scaling.</item> <item><c><![CDATA[-repeat <n>]]></c>, tells Common Test to repeat the tests n times (see below).</item> <item><c><![CDATA[-duration <time>]]></c>, tells Common Test to repeat the tests for duration of time (see below).</item> <item><c><![CDATA[-until <stop_time>]]></c>, tells Common Test to repeat the tests until stop_time (see below).</item> @@ -165,7 +178,7 @@ the current working directory of the Erlang Runtime System during the test run!</p> </note> - <p>For details on how to generate the <c>run_test</c> script, see the + <p>For more information about the <c>run_test</c> program, see the <seealso marker="install_chapter#general">Installation</seealso> chapter. </p> </section> @@ -174,7 +187,7 @@ <title>Running tests from the Web based GUI</title> <p>The web based GUI, VTS, is started with the <c>run_test</c> - script. From the GUI you can load config files, and select + program. From the GUI you can load config files, and select directories, suites and cases to run. You can also state the config files, directories, suites and cases on the command line when starting the web based GUI. @@ -198,10 +211,11 @@ <p>Example:</p> <p><c><![CDATA[$ run_test -vts -browser 'firefox&']]></c></p> <p>Note that the browser must run as a separate OS process or VTS will hang!</p> - <p>If no specific browser start command is specified, netscape will + <p>If no specific browser start command is specified, Firefox will be the default browser on Unix platforms and Internet Explorer on Windows. - If Common Test fails to start a browser automatically, start your - favourite browser manually instead and type in the URL that Common Test + If Common Test fails to start a browser automatically, or <c>'none'</c> is + specified as the value for -browser (i.e. <c>-browser none</c>), start your + favourite browser manually and type in the URL that Common Test displays in the shell.</p> </section> @@ -211,7 +225,7 @@ <p>Common Test provides an Erlang API for running tests. The main (and most flexible) function for specifying and executing tests is called <c>ct:run_test/1</c>. This function takes the same start parameters as - the <c>run_test</c> script described above, only the flags are instead + the <c>run_test</c> program described above, only the flags are instead given as options in a list of key-value tuples. E.g. a test specified with <c>run_test</c> like:</p> <p><c>$ run_test -suite ./my_SUITE -logdir ./results</c></p> @@ -237,13 +251,14 @@ manually and call <c>ct:install/1</c> to install any configuration data you might need (use <c>[]</c> as argument otherwise), then call <c>ct:start_interactive/0</c> to start Common Test. If you use - the <c>run_test</c> script, you may start the Erlang shell and Common Test + the <c>run_test</c> program, you may start the Erlang shell and Common Test in the same go by using the <c>-shell</c> and, optionally, the <c>-config</c> - flag: + and/or <c>-userconfig</c> flag. Examples: </p> <list> <item><c>run_test -shell</c></item> - <item><c><![CDATA[run_test -shell -config <configfilename>]]></c></item> + <item><c><![CDATA[run_test -shell -config cfg/db.cfg]]></c></item> + <item><c><![CDATA[run_test -shell -userconfig db_login testuser x523qZ]]></c></item> </list> <p>If no config file is given with the <c>run_test</c> command, @@ -268,7 +283,8 @@ 2> ct_telnet:open(unix_telnet). {ok,<0.105.0>} 4> ct_telnet:cmd(unix_telnet, "ls ."). - {ok,["ls .","file1 ...",...]}</pre> + {ok,["ls .","file1 ...",...]} + </pre> <p>Everything that Common Test normally prints in the test case logs, will in the interactive mode be written to a log named @@ -350,14 +366,21 @@ <p>Below is the test specification syntax. Test specifications can be used to run tests both in a single test host environment and in - a distributed Common Test environment. Node parameters are only relevant in the - latter (see the chapter about running Common Test in distributed mode for information). - For details on the event_handler term, see the + a distributed Common Test environment (Large Scale Testing). The init term, + as well as node parameters, are only relevant in the latter (see the + <seealso marker="ct_master_chapter#test_specifications">Large Scale Testing</seealso> + chapter for information). For details on the event_handler term, see the <seealso marker="event_handler_chapter#event_handling">Event Handling</seealso> chapter.</p> <p>Config terms:</p> <pre> {node, NodeAlias, Node}. + + {init, InitOptions}. + {init, [NodeAlias], InitOptions}. + + {multiply_timetraps, N}. + {scale_timetraps, Bool}. {cover, CoverSpecFile}. {cover, NodeRef, CoverSpecFile}. @@ -367,6 +390,9 @@ {config, ConfigFiles}. {config, NodeRefs, ConfigFiles}. + + {userconfig, {CallbackModule, ConfigStrings}}. + {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}. {alias, DirAlias, Dir}. @@ -395,9 +421,12 @@ <p>Types:</p> <pre> NodeAlias = atom() + InitOptions = term() Node = node() NodeRef = NodeAlias | Node | master NodeRefs = all_nodes | [NodeRef] | NodeRef + N = integer() + Bool = true | false CoverSpecFile = string() IncludeDirs = string() | [string()] ConfigFiles = string() | [string()] @@ -449,9 +478,14 @@ <item>Secondly, the test for system t2 should run. The included suites are t2B and t2C. Included are also test cases test4, test1 and test7 in suite t2A. Note that the test cases will be executed in the specified order.</item> - <item>Lastly, all suites for systems t3 are to be completely skipped and this + <item>Lastly, all suites for systems t3 are to be completely skipped and this should be explicitly noted in the log files.</item> </list> + <p>It is possible to specify initialization options for nodes defined in the + test specification. Currently, there are options to start the node and/or to + evaluate any function on the node. + See the <seealso marker="ct_master_chapter#ct_slave">Automatic startup of + the test target nodes</seealso> chapter for details.</p> <p>It is possible for the user to provide a test specification that includes (for Common Test) unrecognizable terms. If this is desired, the <c>-allow_user_terms</c> flag should be used when starting tests with |