diff options
Diffstat (limited to 'lib/common_test/doc/src/run_test_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/run_test_chapter.xml | 752 |
1 files changed, 559 insertions, 193 deletions
diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index 30486d3eec..b804f134c6 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -119,7 +119,7 @@ <item><c><![CDATA[ct_run -userconfig <callbackmodulename> <configfilenames> -suite <suiteswithfullpath>]]></c> </item> <item><c><![CDATA[ct_run -config <configfilenames> -suite <suitewithfullpath> - -group <groupnames> -case <casenames>]]></c></item> + -group <groups> -case <casenames>]]></c></item> </list> <p>Examples:</p> <p><c>$ ct_run -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST</c></p> @@ -137,6 +137,8 @@ <p><c>$ ct_run -suite ./testdir/x_SUITE ./testdir/y_SUITE</c></p> + <p>For more details on <seealso marker="run_test_chapter#group_execution">test case group execution</seealso>, please see below.</p> + <p>Other flags that may be used with <c>ct_run</c>:</p> <list> <item><c><![CDATA[-logdir <dir>]]></c>, specifies where the HTML log files are to be written.</item> @@ -153,6 +155,8 @@ <item><c><![CDATA[-stylesheet <css_file>]]></c>, points out a user HTML style sheet (see below).</item> <item><c><![CDATA[-cover <cover_cfg_file>]]></c>, to perform code coverage test (see <seealso marker="cover_chapter#cover">Code Coverage Analysis</seealso>).</item> + <item><c><![CDATA[-cover_stop <bool>]]></c>, to specify if the cover tool shall be stopped after the test is completed (see + <seealso marker="cover_chapter#cover_stop">Code Coverage Analysis</seealso>).</item> <item><c><![CDATA[-event_handler <event_handlers>]]></c>, to install <seealso marker="event_handler_chapter#event_handling">event handlers</seealso>.</item> <item><c><![CDATA[-event_handler_init <event_handlers>]]></c>, to install @@ -178,6 +182,8 @@ <item><c><![CDATA[-basic_html]]></c>, switches off html enhancements that might not be compatible with older browsers.</item> <item><c><![CDATA[-logopts <opts>]]></c>, makes it possible to modify aspects of the logging behaviour, see <seealso marker="run_test_chapter#logopts">Log options</seealso> below.</item> + <item><c><![CDATA[-verbosity <levels>]]></c>, sets <seealso marker="write_test_chapter#logging">verbosity levels + for printouts</seealso>.</item> </list> <note><p>Directories passed to Common Test may have either relative or absolute paths.</p></note> @@ -196,62 +202,232 @@ the current working directory of the Erlang Runtime System during the test run!</p> </note> - <p>For more information about the <c>ct_run</c> program, see the - <seealso marker="install_chapter#general">Installation</seealso> chapter. - </p> - </section> - - <section> - <title>Running tests from the Web based GUI</title> - - <p>The web based GUI, VTS, is started with the <c>ct_run</c> - program. From the GUI you can load config files, and select - directories, suites and cases to run. You can also state the - config files, directories, suites and cases on the command line - when starting the web based GUI. - </p> - + <p>The <c>ct_run</c> program sets the exit status before shutting down. The following values + are defined:</p> <list> - <item><c>ct_run -vts</c></item> - <item><c><![CDATA[ct_run -vts -config <configfilename>]]></c></item> - <item><c><![CDATA[ct_run -vts -config <configfilename> -suite <suitewithfullpath> - -case <casename>]]></c></item> + <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto-skipped test cases.</item> + <item><c>1</c> indicates that one or more test cases have failed, or have been auto-skipped.</item> + <item><c>2</c> indicates that the test execution has failed because of e.g. compilation errors, an + illegal return value from an info function, etc.</item> </list> + <p>If auto-skipped test cases should not affect the exit status, you may change the default + behaviour using start flag:</p> + <pre>-exit_status ignore_config</pre> - <p>From the GUI you can run tests and view the result and the logs. + <p>For more information about the <c>ct_run</c> program, see the + <seealso marker="ct_run#top">Reference Manual</seealso> and the + <seealso marker="install_chapter#general">Installation</seealso> chapter. </p> - - <p>Note that <c>ct_run -vts</c> will try to open the Common Test start - page in an existing web browser window or start the browser if it is - not running. Which browser should be started may be specified with - the browser start command option:</p> - <p><c><![CDATA[ct_run -vts -browser <browser_start_cmd>]]></c></p> - <p>Example:</p> - <p><c><![CDATA[$ ct_run -vts -browser 'firefox&']]></c></p> - <p>Note that the browser must run as a separate OS process or VTS will hang!</p> - <p>If no specific browser start command is specified, Firefox will - be the default browser on Unix platforms and Internet Explorer on Windows. - If Common Test fails to start a browser automatically, or <c>'none'</c> is - specified as the value for -browser (i.e. <c>-browser none</c>), start your - favourite browser manually and type in the URL that Common Test - displays in the shell.</p> </section> - + <section> <title>Running tests from the Erlang shell or from an Erlang program</title> <p>Common Test provides an Erlang API for running tests. The main (and most flexible) function for specifying and executing tests is called - <c>ct:run_test/1</c>. This function takes the same start parameters as - the <c>ct_run</c> program described above, only the flags are instead + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>. + This function takes the same start parameters as + the <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + program described above, only the flags are instead given as options in a list of key-value tuples. E.g. a test specified with <c>ct_run</c> like:</p> + <p><c>$ ct_run -suite ./my_SUITE -logdir ./results</c></p> - <p>is with <c>ct:run_test/1</c> specified as:</p> + <p>is with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> specified as:</p> <p><c>1> ct:run_test([{suite,"./my_SUITE"},{logdir,"./results"}]).</c></p> - <p>For detailed documentation, please see the <c>ct</c> manual page.</p> + + <p>The function returns the test result, represented by the tuple: + <c>{Ok,Failed,{UserSkipped,AutoSkipped}}</c>, where each element is an + integer. If test execution fails, the function returns the tuple: + <c>{error,Reason}</c>, where the term <c>Reason</c> explains the + failure.</p> + + <section> + <title>Releasing the Erlang shell</title> + <p>During execution of tests, started with + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + the Erlang shell process, controlling stdin, will remain the top + level process of the Common Test system of processes. The result + is that the Erlang shell is not available for interaction during + the test run. If this is not desirable, maybe because the shell is needed + for debugging purposes or for interaction with the SUT during test + execution, you may set the <c>release_shell</c> start option to + <c>true</c> (in the call to <c>ct:run_test/1</c> or by + using the corresponding test specification term, see below). This will + make Common Test release the shell immediately after the test suite + compilation stage. To accomplish this, a test runner process + is spawned to take control of the test execution, and the effect is that + <c>ct:run_test/1</c> returns the pid of this process rather than the + test result - which instead is printed to tty at the end of the test run.</p> + <note><p>Note that in order to use the + <c><seealso marker="ct#break-1">ct:break/1/2</seealso></c> and + <c><seealso marker="ct#continue-0">ct:continue/0/1</seealso></c> functions, + <c>release_shell</c> <em>must</em> be set to <c>true</c>.</p></note> + </section> + + <p>For detailed documentation about + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + please see the + <c><seealso marker="ct#run_test-1">ct</seealso></c> manual page.</p> </section> + <marker id="group_execution"></marker> + <section> + <title>Test case group execution</title> + + <p>With the <c>ct_run</c> flag, or <c>ct:run_test/1</c> option <c>group</c>, + one or more test case groups can be specified, optionally in combination + with specific test cases. The syntax for specifying groups is as follows + (on the command line):</p> + + <pre> + <![CDATA[$ ct_run -group <group_names_or_paths> [-case <cases>]]]></pre> + <p>or (in the Erlang shell):</p> + <pre> + <![CDATA[1> ct:run_test([{group,GroupsNamesOrPaths}, {case,Cases}]).]]></pre> + + <p>The <c>group_names_or_paths</c> parameter specifies either one + or more group names and/or one or more group paths. At start up, + Common Test will search for matching groups in the group definitions + tree (i.e. the list returned from <c>Suite:groups/0</c>, please see the + <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso> + chapter for details). + Given a group name, say <c>g</c>, Common Test will search for all paths + that lead to <c>g</c>. By path here we mean a sequence of nested groups, + all of which have to be followed in order to get from the top level + group to <c>g</c>. Actually, what Common Test needs to do in order to + execute the test cases in group <c>g</c>, is to call the + <c>init_per_group/2</c> function for each group in the path to + <c>g</c>, as well as all corresponding <c>end_per_group/2</c> + functions afterwards. The obvious reason for this is that the configuration + of a test case in <c>g</c> (and its <c>Config</c> input data) depends on + <c>init_per_testcase(TestCase, Config)</c> and its return value, which + in turn depends on <c>init_per_group(g, Config)</c> and its return value, + which in turn depends on <c>init_per_group/2</c> of the group above + <c>g</c>, etc, all the way up to the top level group.</p> + + <p>As you may have already realized, this means that if there is more than + one way to locate a group (and its test cases) in a path, the result of the + group search operation is a number of tests, all of which will be performed. + Common Test actually interprets a group specification that consists of a + single name this way:</p> + + <p>"Search and find all paths in the group definitions tree that lead + to the specified group and, for each path, create a test which (1) executes + all configuration functions in the path to the specified group, then (2) + executes all - or all matching - test cases in this group, as well as (3) + all - or all matching - test cases in all sub groups of the group". + </p> + + <p>It is also possible for the user to specify a specific group path with + the <c>group_names_or_paths</c> parameter. With this type of specification it's + possible to avoid execution of unwanted groups (in otherwise matching paths), + and/or the execution of sub groups. The syntax of the group path is a list of + group names in the path, e.g. on the command line: + </p> + <p><c>$ ct_run -suite "./x_SUITE" -group [g1,g3,g4] -case tc1 tc5</c></p> + <p>or similarly in the Erlang shell (requires a list within the groups list):</p> + <p><c>1> ct:run_test([{suite,"./x_SUITE"}, {group,[[g1,g3,g4]]}, {testcase,[tc1,tc5]}]).</c></p> + + <p>The last group in the specified path will be the terminating group in + the test, i.e. no sub groups following this group will be executed. In the + example above, <c>g4</c> is the terminating group, hence Common Test will + execute a test that calls all init configuration functions in the path to + <c>g4</c>, i.e. <c>g1..g3..g4</c>. It will then call test cases <c>tc1</c> + and <c>tc5</c> in <c>g4</c> and finally all end configuration functions in order + <c>g4..g3..g1</c>.</p> + + <p>Note that the group path specification doesn't necessarily + have to include <em>all</em> groups in the path to the terminating group. + Common Test will search for all matching paths if given an incomplete group + path.</p> + + <p>Note also that it's possible to combine group names and group paths with the + <c>group_names_or_paths</c> parameter. Each element is treated as + an individual specification in combination with the <c>cases</c> parameter. + See examples below.</p> + + <p>Examples:</p> + <pre> + -module(x_SUITE). + ... + %% The group definitions: + groups() -> + [{top1,[],[tc11,tc12, + {sub11,[],[tc12,tc13]}, + {sub12,[],[tc14,tc15, + {sub121,[],[tc12,tc16]}]}]}, + + {top2,[],[{group,sub21},{group,sub22}]}, + {sub21,[],[tc21,{group,sub2X2}]}, + {sub22,[],[{group,sub221},tc21,tc22,{group,sub2X2}]}, + {sub221,[],[tc21,tc23]}, + {sub2X2,[],[tc21,tc24]}]. + </pre> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group all</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,all}]).</c></p> + <p>Two tests will be executed, one for all cases and all sub groups under <c>top1</c>, + and one for all under <c>top2</c>. (We would get the same result with + <c>-group top1 top2</c>, or <c>{group,[top1,top2]}</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}]).</c></p> + <p>This will execute one test for all cases and sub groups under <c>top1</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1 -case tc12</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc12]}]).</c></p> + <p>This will run a test that executes <c>tc12</c> in <c>top1</c> and any sub group + under <c>top1</c> where it can be found (<c>sub11</c> and <c>sub121</c>).</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [top1] -case tc12</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[top1]]}, {testcase,[tc12]}]).</c></p> + <p>This will execute <c>tc12</c> <em>only</em> in group <c>top1</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1 -case tc16</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc16]}]).</c></p> + <p>This will search <c>top1</c> and all its sub groups for <c>tc16</c> and the result + will be that this test case executes in group <c>sub121</c>. (The specific path: + <c>-group [sub121]</c> or <c>{group,[[sub121]]}</c>, would have given + us the same result in this example).</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group sub12 [sub12]</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[sub12,[sub12]]}]).</c></p> + <p>This will execute two tests, one that includes all cases and sub groups under + <c>sub12</c>, and one with <em>only</em> the test cases in <c>sub12</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group sub2X2</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[sub2X2]}]).</c></p> + <p>In this example, Common Test will find and execute two tests, one for the path from + <c>top2</c> to <c>sub2X2</c> via <c>sub21</c>, and one from <c>top2</c> to <c>sub2X2</c> + via <c>sub22</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [sub21,sub2X2]</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub21,sub2X2]]}]).</c></p> + <p>Here, by specifying the unique path: <c>top2 -> sub21 -> sub2X2</c>, only one test + is executed. The second possible path from <c>top2</c> to <c>sub2X2</c> (above) + will be discarded.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [sub22] -case tc22 tc21</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub22]]}, {testcase,[tc22,tc21]}]).</c></p> + <p>In this example only the test cases for <c>sub22</c> will be executed, and in + reverse order compared to the group definition.</p> + <br></br> + + <p>If a test case that belongs to a group (according to the group definition), is executed + without a group specification, i.e. simply by means of (command line):</p> + <p><c>$ ct_run -suite "my_SUITE" -case my_tc</c></p> + <p>or (Erlang shell):</p> + <p><c>1> ct:run_test([{suite,"my_SUITE"}, {testcase,my_tc}]).</c></p> + <p>then Common Test ignores the group definition and executes the test case in the scope of the + test suite only (no group configuration functions are called).</p> + + <p>The group specification feature, exactly as it has been presented in this section, can also + be used in <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso> (with some extra features added). Please see below.</p> + </section> + + <section> <title>Running the interactive shell mode</title> @@ -266,9 +442,9 @@ for trying out various operations during test suite development.</p> <p>To invoke the interactive shell mode, you can start an Erlang shell - manually and call <c>ct:install/1</c> to install any configuration + manually and call <c><seealso marker="ct#install-1">ct:install/1</seealso></c> to install any configuration data you might need (use <c>[]</c> as argument otherwise), then - call <c>ct:start_interactive/0</c> to start Common Test. If you use + call <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> to start Common Test. If you use the <c>ct_run</c> program, you may start the Erlang shell and Common Test in the same go by using the <c>-shell</c> and, optionally, the <c>-config</c> and/or <c>-userconfig</c> flag. Examples: @@ -287,7 +463,8 @@ <p>If any functions using "required config data" (e.g. ct_telnet or ct_ftp functions) are to be called from the erlang shell, config - data must first be required with <c>ct:require/[1,2]</c>. This is + data must first be required with <c><seealso marker="ct#require-1"> + ct:require/1/2</seealso></c>. This is equivalent to a <c>require</c> statement in the <seealso marker="write_test_chapter#suite">Test Suite Info Function</seealso> or in the <seealso @@ -314,11 +491,11 @@ is not supported.</p> <p>If you wish to exit the interactive mode (e.g. to start an - automated test run with <c>ct:run_test/1</c>), call the function - <c>ct:stop_interactive/0</c>. This shuts down the + automated test run with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), call the function + <c><seealso marker="ct#stop_interactive-0">ct:stop_interactive/0</seealso></c>. This shuts down the running <c>ct</c> application. Associations between configuration names and data created with <c>require</c> are - consequently deleted. <c>ct:start_interactive/0</c> will get you + consequently deleted. <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> will get you back into interactive mode, but the previous state is not restored.</p> </section> @@ -326,7 +503,7 @@ <title>Step by step execution of test cases with the Erlang Debugger</title> <p>By means of <c>ct_run -step [opts]</c>, or by passing the - <c>{step,Opts}</c> option to <c>ct:run_test/1</c>, it is possible + <c>{step,Opts}</c> option to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, it is possible to get the Erlang Debugger started automatically and use its graphical interface to investigate the state of the current test case and to execute it step by step and/or set execution breakpoints.</p> @@ -352,35 +529,38 @@ <marker id="test_specifications"></marker> <section> - <title>Using test specifications</title> + <title>Test Specifications</title> <p>The most flexible way to specify what to test, is to use a so called test specification. A test specification is a sequence of - Erlang terms. The terms may be declared in a text file or passed - to the test server at runtime as a list - (see <c>run_testspec/1</c> in the manual page - for <c>ct</c>). There are two general types of terms: - configuration terms and test specification terms.</p> + Erlang terms. The terms are normally declared in a text file (see + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), but + may also be passed to Common Test on the form of a list (see + <c><seealso marker="ct#run_testspec-1">ct:run_testspec/1</seealso></c>). + There are two general types of terms: configuration terms and test + specification terms.</p> <p>With configuration terms it is possible to e.g. label the test run (similar to <c>ct_run -label</c>), evaluate arbitrary expressions - before starting a test, import configuration - data (similar to - <c>ct_run -config/-userconfig</c>), specify HTML log directories (similar - to - <c>ct_run -logdir</c>), give aliases to test nodes and test - directories (to make a specification easier to read and - maintain), enable code coverage analysis (see - the <seealso marker="cover_chapter#cover">Code Coverage - Analysis</seealso> chapter) and specify event_handler plugins - (see the <seealso marker="event_handler_chapter#event_handling"> - Event Handling</seealso> chapter). There is also a term for - specifying include directories that should be passed on to the - compiler when automatic compilation is performed (similar - to <c>ct_run -include</c>, see above).</p> + before starting the test, import configuration data (similar to + <c>ct_run -config/-userconfig</c>), specify the top level HTML log + directory (similar to <c>ct_run -logdir</c>), enable code coverage + analysis (similar to <c>ct_run -cover</c>), install Common Test Hooks + (similar to <c>ct_run -ch_hooks</c>), install event_handler plugins + (similar to <c>ct_run -event_handler</c>), specify include directories + that should be passed to the compiler for automatic compilation + (similar to <c>ct_run -include</c>), disable the auto compilation + feature (similar to <c>ct_run -no_auto_compile</c>), set verbosity + levels (similar to <c>ct_run -verbosity</c>), and more.</p> + <p>Configuration terms can be combined with <c>ct_run</c> start flags, + or <c>ct:run_test/1</c> options. The result will for some flags/options + and terms be that the values are merged (e.g. configuration files, + include directories, verbosity levels, silent connections), and for + others that the start flags/options override the test specification + terms (e.g. log directory, label, style sheet, auto compilation).</p> <p>With test specification terms it is possible to state exactly which tests should run and in which order. A test term specifies - either one or more suites, one or more test case groups, or one - or more test cases in a group or suite.</p> + either one or more suites, one or more test case groups (possibly nested), + or one or more test cases in a group (or in multiple groups) or in a suite.</p> <p>An arbitrary number of test terms may be declared in sequence. Common Test will by default compile the terms into one or more tests to be performed in one resulting test run. Note that a term that @@ -391,49 +571,69 @@ S, is a test of all cases in S. However, if a term specifying test case X and Y in S is merged with a term specifying case Z in S, the result is a test of X, Y and Z in S. To disable this - behaviour, it is possible in test specification to set the - <c>merge_tests</c> term to <c>false</c>.</p> + behaviour, i.e. to instead perform each test sequentially in a "script-like" + manner, the term <c>merge_tests</c> can be set to <c>false</c> in + the test specification.</p> <p>A test term can also specify one or more test suites, groups, or test cases to be skipped. Skipped suites, groups and cases - are not executed and show up in the HTML test log files as + are not executed and show up in the HTML log files as SKIPPED.</p> <p>When a test case group is specified, the resulting test - executes the - <c>init_per_group</c> function, followed by all test cases and - sub groups (including their configuration functions), and + executes the <c>init_per_group</c> function, followed by all test + cases and sub groups (including their configuration functions), and finally the <c>end_per_group</c> function. Also if particular test cases in a group are specified, <c>init_per_group</c> and <c>end_per_group</c> for the group in question are called. If a group which is defined (in <c>Suite:group/0</c>) to - be a sub group of another group, is specified (or particular test + be a sub group of another group, is specified (or if particular test cases of a sub group are), Common Test will call the configuration functions for the top level groups as well as for the sub group in question (making it possible to pass configuration data all the way from <c>init_per_suite</c> down to the test cases in the sub group).</p> - - <p>With the <c>GroupSpec</c> element (below) it's possible to specify - group execution properties that will override those specified in the + <p>The test specification utilizes the same mechanism for specifying + test case groups by means of names and paths, as explained in the + <seealso marker="run_test_chapter#group_execution">Group Execution</seealso> + section above, with the addition of the <c>GroupSpec</c> element + described next.</p> + <p>The <c>GroupSpec</c> element makes it possible to specify + group execution properties that will override those in the group definition (i.e. in <c>groups/0</c>). Execution properties for sub-groups may be overridden as well. This feature makes it possible to change properties of groups at the time of execution, - without even having to edit the test suite. More detailed documentation, - and examples, can be found in the - <seealso marker="write_test_chapter#test_case_groups"> + without even having to edit the test suite. The very same + feature is available for <c>group</c> elements in the <c>Suite:all/0</c> + list. Therefore, more detailed documentation, and examples, can be + found in the <seealso marker="write_test_chapter#test_case_groups"> Test case groups</seealso> chapter.</p> <p>Below is the test specification syntax. Test specifications can be used to run tests both in a single test host environment and in a distributed Common Test environment (Large Scale - Testing). The node parameters in the init term are only + Testing). The node parameters in the <c>init</c> term are only relevant in the latter (see the <seealso marker="ct_master_chapter#test_specifications">Large - Scale Testing</seealso> chapter for information). For details on - the event_handler term, see the + Scale Testing</seealso> chapter for information). For more information + about the various terms, please see the corresponding sections in the + User's Guide, such as e.g. the + <seealso marker="run_test_chapter#ct_run"><c>ct_run</c> + program</seealso> for an overview of available start flags + (since most flags have a corresponding configuration term), and + more detailed explanation of e.g. + <seealso marker="write_test_chapter#logging">Logging</seealso> + (for the <c>verbosity</c>, <c>stylesheet</c> and <c>basic_html</c> terms), + <seealso marker="config_file_chapter#top">External Configuration Data</seealso> + (for the <c>config</c> and <c>userconfig</c> terms), <seealso marker="event_handler_chapter#event_handling">Event - Handling</seealso> chapter.</p> + Handling</seealso> (for the <c>event_handler</c> term), + <seealso marker="ct_hooks_chapter#installing">Common Test Hooks</seealso> + (for the <c>ct_hooks</c> term), etc.</p> <p>Config terms:</p> <pre> + {merge_tests, Bool}. + + {define, Constant, Value}. + {node, NodeAlias, Node}. {init, InitOptions}. @@ -442,6 +642,15 @@ {label, Label}. {label, NodeRefs, Label}. + {verbosity, VerbosityLevels}. + {verbosity, NodeRefs, VerbosityLevels}. + + {stylesheet, CSSFile}. + {stylesheet, NodeRefs, CSSFile}. + + {silent_connections, ConnTypes}. + {silent_connections, NodeRefs, ConnTypes}. + {multiply_timetraps, N}. {multiply_timetraps, NodeRefs, N}. @@ -451,22 +660,29 @@ {cover, CoverSpecFile}. {cover, NodeRefs, CoverSpecFile}. + {cover_stop, Bool}. + {cover_stop, NodeRefs, Bool}. + {include, IncludeDirs}. {include, NodeRefs, IncludeDirs}. + {auto_compile, Bool}, + {auto_compile, NodeRefs, Bool}, + {config, ConfigFiles}. + {config, ConfigDir, ConfigBaseNames}. {config, NodeRefs, ConfigFiles}. + {config, NodeRefs, ConfigDir, ConfigBaseNames}. {userconfig, {CallbackModule, ConfigStrings}}. {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}. - {alias, DirAlias, Dir}. - - {merge_tests, Bool}. - {logdir, LogDir}. {logdir, NodeRefs, LogDir}. + {logopts, LogOpts}. + {logopts, NodeRefs, LogOpts}. + {create_priv_dir, PrivDirOption}. {create_priv_dir, NodeRefs, PrivDirOption}. @@ -479,83 +695,177 @@ {ct_hooks, NodeRefs, CTHModules}. {enable_builtin_hooks, Bool}. - </pre> + + {basic_html, Bool}. + {basic_html, NodeRefs, Bool}. + + {release_shell, Bool}.</pre> + <p>Test terms:</p> <pre> - {suites, DirRef, Suites}. - {suites, NodeRefs, DirRef, Suites}. + {suites, Dir, Suites}. + {suites, NodeRefs, Dir, Suites}. - {groups, DirRef, Suite, Groups}. - {groups, NodeRefsDirRef, Suite, Groups}. + {groups, Dir, Suite, Groups}. + {groups, NodeRefs, Dir, Suite, Groups}. - {groups, DirRef, Suite, GroupSpec, {cases,Cases}}. - {groups, NodeRefsDirRef, Suite, GroupSpec, {cases,Cases}}. + {groups, Dir, Suite, Groups, {cases,Cases}}. + {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}. - {cases, DirRef, Suite, Cases}. - {cases, NodeRefs, DirRef, Suite, Cases}. + {cases, Dir, Suite, Cases}. + {cases, NodeRefs, Dir, Suite, Cases}. - {skip_suites, DirRef, Suites, Comment}. - {skip_suites, NodeRefs, DirRef, Suites, Comment}. + {skip_suites, Dir, Suites, Comment}. + {skip_suites, NodeRefs, Dir, Suites, Comment}. - {skip_groups, DirRef, Suite, GroupNames, Comment}. - {skip_groups, NodeRefs, DirRef, Suite, GroupNames, Comment}. + {skip_groups, Dir, Suite, GroupNames, Comment}. + {skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}. - {skip_cases, DirRef, Suite, Cases, Comment}. - {skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}. - </pre> + {skip_cases, Dir, Suite, Cases, Comment}. + {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}.</pre> + <p>Types:</p> <pre> - NodeAlias = atom() - InitOptions = term() - Node = node() - NodeRef = NodeAlias | Node | master - NodeRefs = all_nodes | [NodeRef] | NodeRef - N = integer() - Bool = true | false - CoverSpecFile = string() - IncludeDirs = string() | [string()] - ConfigFiles = string() | [string()] - DirAlias = atom() - Dir = string() - LogDir = string() - PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc - EventHandlers = atom() | [atom()] - InitArgs = [term()] - CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}] - CTHModule = atom() - CTHInitArgs = term() - DirRef = DirAlias | Dir - Suites = atom() | [atom()] | all - Suite = atom() - Groups = GroupSpec | [GroupSpec] | all - GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec} - GroupName = atom() - GroupNames = GroupName | [GroupName] - Cases = atom() | [atom()] | all - Comment = string() | "" - </pre> - <p>Example:</p> + Bool = true | false + Constant = atom() + Value = term() + NodeAlias = atom() + Node = node() + NodeRef = NodeAlias | Node | master + NodeRefs = all_nodes | [NodeRef] | NodeRef + InitOptions = term() + Label = atom() | string() + VerbosityLevels = integer() | [{Category,integer()}] + Category = atom() + CSSFile = string() + ConnTypes = all | [atom()] + N = integer() + CoverSpecFile = string() + IncludeDirs = string() | [string()] + ConfigFiles = string() | [string()] + ConfigDir = string() + ConfigBaseNames = string() | [string()] + CallbackModule = atom() + ConfigStrings = string() | [string()] + LogDir = string() + LogOpts = [term()] + PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc + EventHandlers = atom() | [atom()] + InitArgs = [term()] + CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}] + CTHModule = atom() + CTHInitArgs = term() + Dir = string() + Suites = atom() | [atom()] | all + Suite = atom() + Groups = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all + GroupPath = [GroupName] + GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec} + GroupName = atom() + GroupNames = GroupName | [GroupName] + Cases = atom() | [atom()] | all + Comment = string() | ""</pre> + + <p>The difference between the <c>config</c> terms above, is that with + <c>ConfigDir</c>, <c>ConfigBaseNames</c> is a list of base names, + i.e. without directory paths. <c>ConfigFiles</c> must be full names, + including paths. E.g, these two terms have the same meaning:</p> <pre> - {logdir, "/home/test/logs"}. + {config, ["/home/testuser/tests/config/nodeA.cfg", + "/home/testuser/tests/config/nodeB.cfg"]}. + + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.</pre> + + <note><p>Any relative paths specified in the test specification, will be + relative to the directory which contains the test specification file, if + <c>ct_run -spec TestSpecFile ...</c> or + <c>ct:run:test([{spec,TestSpecFile},...])</c> + executes the test. The path will be relative to the top level log directory, if + <c>ct:run:testspec(TestSpec)</c> executes the test.</p></note> - {config, "/home/test/t1/cfg/config.cfg"}. - {config, "/home/test/t2/cfg/config.cfg"}. - {config, "/home/test/t3/cfg/config.cfg"}. + <p>The <c>define</c> term introduces a constant, which is used to + replace the name <c>Constant</c> with <c>Value</c>, wherever it's found in + the test specification. This replacement happens during an initial iteration + through the test specification. Constants may be used anywhere in the test + specification, e.g. in arbitrary lists and tuples, and even in strings + and inside the value part of other constant definitions! A constant can + also be part of a node name, but that is the only place where a constant + can be part of an atom.</p> + + <note><p>For the sake of readability, the name of the constant must always + begin with an upper case letter, or a <c>$</c>, <c>?</c>, or <c>_</c>. + This also means that it must always be single quoted (obviously, since + the constant name is actually an atom, not text).</p></note> + + <p>The main benefit of constants is that they can be used to reduce the size + (and avoid repetition) of long strings, such as file paths. Compare these + terms:</p> + + <pre> + %% 1a. no constant + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "/home/testuser/tests/suites", all}. - {alias, t1, "/home/test/t1"}. - {alias, t2, "/home/test/t2"}. - {alias, t3, "/home/test/t3"}. + %% 1b. with constant + {define, 'TESTDIR', "/home/testuser/tests"}. + {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "'TESTDIR'/suites", all}. + + %% 2a. no constants + {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}. + + %% 2b. with constants + {define, 'NODE', testnode}. + {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}. + {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.</pre> + + <p>Constants make the test specification term <c>alias</c>, in previous + versions of Common Test, redundant. This term has been deprecated but will + remain supported in upcoming Common Test releases. Replacing <c>alias</c> + terms with <c>define</c> is strongly recommended though! Here's an example + of such a replacement:</p> + + <pre> + %% using the old alias term + {config, "/home/testuser/tests/config/nodeA.cfg"}. + {alias, suite_dir, "/home/testuser/tests/suites"}. + {groups, suite_dir, x_SUITE, group1}. + + %% replacing with constants + {define, 'TestDir', "/home/testuser/tests"}. + {define, 'CfgDir', "'TestDir'/config"}. + {define, 'SuiteDir', "'TestDir'/suites"}. + {config, 'CfgDir', "nodeA.cfg"}. + {groups, 'SuiteDir', x_SUITE, group1}.</pre> + + <p>Actually, constants could well replace the <c>node</c> term too, but + this still has declarative value, mainly when used in combination + with <c>NodeRefs == all_nodes</c> (see types above).</p> + + <p>Here follows a simple test specification example:</p> + <pre> + {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + + {logdir, "'Top'/logs"}. - {suites, t1, all}. - {skip_suites, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}. - {skip_cases, t1, t1A_SUITE, [test3,test4], "Irrelevant"}. - {skip_cases, t1, t1C_SUITE, [test1], "Ignore"}. + {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}. - {suites, t2, [t2B_SUITE,t2C_SUITE]}. - {cases, t2, t2A_SUITE, [test4,test1,test7]}. + {suites, 'T1', all}. + {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}. - {skip_suites, t3, all, "Not implemented"}. - </pre> + {suites, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, 'T2', t2A_SUITE, [test4,test1,test7]}. + + {skip_suites, 'T3', all, "Not implemented"}.</pre> + <p>The example specifies the following:</p> <list> <item>The specified logdir directory will be used for storing @@ -563,8 +873,6 @@ date and time).</item> <item>The variables in the specified test system config files will be imported for the test.</item> - <item>Aliases are given for three test system directories. The suites in - this example are stored in "/home/test/tX/test".</item> <item>The first test to run includes all suites for system t1. Excluded from the test are however the t1B and t1D suites. Also test cases test3 and test4 in t1A as well as the test1 case in t1C are excluded from @@ -575,9 +883,9 @@ <item>Lastly, all suites for systems t3 are to be completely skipped and this should be explicitly noted in the log files.</item> </list> - <p>It is possible to specify initialization options for nodes defined in the - test specification. Currently, there are options to start the node and/or to - evaluate any function on the node. + <p>With the <c>init</c> term it's possible to specify initialization options + for nodes defined in the test specification. Currently, there are options + to start the node and/or to evaluate any function on the node. See the <seealso marker="ct_master_chapter#ct_slave">Automatic startup of the test target nodes</seealso> chapter for details.</p> <p>It is possible for the user to provide a test specification that @@ -586,11 +894,49 @@ <c>ct_run</c>. This forces Common Test to ignore unrecognizable terms. Note that in this mode, Common Test is not able to check the specification for errors as efficiently as if the scanner runs in default mode. - If <c>ct:run_test/1</c> is used for starting the tests, the relaxed scanner + If <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> is used for starting the tests, the relaxed scanner mode is enabled by means of the tuple: <c>{allow_user_terms,true}</c></p> </section> + + <section> + <title>Running tests from the Web based GUI</title> + + <p>The web based GUI, VTS, is started with the + <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + program. From the GUI you can load config files, and select + directories, suites and cases to run. You can also state the + config files, directories, suites and cases on the command line + when starting the web based GUI. + </p> + + <list> + <item><c>ct_run -vts</c></item> + <item><c><![CDATA[ct_run -vts -config <configfilename>]]></c></item> + <item><c><![CDATA[ct_run -vts -config <configfilename> -suite <suitewithfullpath> + -case <casename>]]></c></item> + </list> + + <p>From the GUI you can run tests and view the result and the logs. + </p> + + <p>Note that <c>ct_run -vts</c> will try to open the Common Test start + page in an existing web browser window or start the browser if it is + not running. Which browser should be started may be specified with + the browser start command option:</p> + <p><c><![CDATA[ct_run -vts -browser <browser_start_cmd>]]></c></p> + <p>Example:</p> + <p><c><![CDATA[$ ct_run -vts -browser 'firefox&']]></c></p> + <p>Note that the browser must run as a separate OS process or VTS will hang!</p> + <p>If no specific browser start command is specified, Firefox will + be the default browser on Unix platforms and Internet Explorer on Windows. + If Common Test fails to start a browser automatically, or <c>'none'</c> is + specified as the value for -browser (i.e. <c>-browser none</c>), start your + favourite browser manually and type in the URL that Common Test + displays in the shell.</p> + </section> <section> + <marker id="log_files"></marker> <title>Log files</title> <p>As the execution of the test suites proceed, events are logged in @@ -718,17 +1064,30 @@ <p>instead of each <c>x</c> printed on a new line, which is the default behaviour.</p> </section> + <section> + <marker id="table_sorting"></marker> + <title>Sorting HTML table columns</title> + <p>By clicking the name in the column header of any table (e.g. "Ok", "Case", "Time", etc), + the table rows are sorted in whatever order makes sense for the type of value (e.g. + numerical for "Ok" or "Time", and alphabetical for "Case"). The sorting is performed + by means of JavaScript code, automatically inserted into the HTML log files. Common Test + uses the <url href="http://jquery.com">jQuery</url> library and the + <url href="http://tablesorter.com">tablesorter</url> plugin, with customized sorting + functions, for this implementation.</p> + </section> </section> <section> <marker id="html_stylesheet"></marker> <title>HTML Style Sheets</title> - <p>Common Test uses a CSS file to control the look of the HTML - files generated during test runs. If, for some reason, the - log files are not displayed correctly in the HTML browser of your - choice, or you prefer the "pre Common Test v1.6 look" - of the log files (i.e. not using CSS), use the start flag/option - <c>basic_html</c> to revert to the old style.</p> + <p>Common Test uses an HTML Style Sheet (CSS file) to control the look of + the HTML log files generated during test runs. If, for some reason, the + log files are not displayed correctly in the browser of your + choice, or you prefer a more primitive ("pre Common Test v1.6") look + of the logs, use the start flag/option:</p> + <pre>basic_html</pre> + <p>This disables the use of Style Sheets, as well as JavaScripts (see + table sorting above).</p> <p>Common Test includes an <em>optional</em> feature to allow user HTML style sheets for customizing printouts. The @@ -881,75 +1240,82 @@ <section> <marker id="silent_connections"></marker> <title>Silent Connections</title> - <p>The protocol handling processes in Common Test, implemented by ct_telnet, ct_ftp etc, - do verbose printing to the test case logs. This can be switched off by means - of the <c>-silent_connections</c> flag:</p> + <p>The protocol handling processes in Common Test, implemented by ct_telnet, + ct_ssh, ct_ftp etc, do verbose printing to the test case logs. This can be switched off + by means of the <c>-silent_connections</c> flag:</p> <pre> ct_run -silent_connections [conn_types] </pre> - <p>where <c>conn_types</c> specifies <c>telnet, ftp, rpc</c> and/or <c>snmp</c>.</p> + <p>where <c>conn_types</c> specifies <c>ssh, telnet, ftp, rpc</c> and/or <c>snmp</c>.</p> <p>Example:</p> <pre> - ct_run ... -silent_connections telnet ftp</pre> - <p>switches off logging for telnet and ftp connections.</p> + ct_run ... -silent_connections ssh telnet</pre> + <p>switches off logging for ssh and telnet connections.</p> <pre> ct_run ... -silent_connections</pre> <p>switches off logging for all connection types.</p> - <p>Basic and important information such as opening and closing a connection, - fatal communication error and reconnection attempts will always be printed even - if logging has been suppressed for the connection type in question. However, operations - such as sending and receiving data may be performed silently.</p> + <p>Fatal communication error and reconnection attempts will always be printed even + if logging has been suppressed for the connection type in question. However, operations + such as sending and receiving data will be performed silently.</p> <p>It is possible to also specify <c>silent_connections</c> in a test suite. This is accomplished by returning a tuple, <c>{silent_connections,ConnTypes}</c>, in the <c>suite/0</c> or test case info list. If <c>ConnTypes</c> is a list of atoms - (<c>telnet, ftp, rpc</c> and/or <c>snmp</c>), output for any corresponding connections + (<c>ssh, telnet, ftp, rpc</c> and/or <c>snmp</c>), output for any corresponding connections will be suppressed. Full logging is per default enabled for any connection of type not specified in <c>ConnTypes</c>. Hence, if <c>ConnTypes</c> is the empty list, logging is enabled for all connections.</p> - <p>The <c>silent_connections</c> setting returned from a test case info function overrides, - for the test case in question, any setting made with <c>suite/0</c> (which is the setting - used for all cases in the suite). Example:</p> + <p>Example:</p> <pre> -module(my_SUITE). + + suite() -> [..., {silent_connections,[telnet,ssh]}, ...]. + ... - suite() -> [..., {silent_connections,[telnet,ftp]}, ...]. - ... + my_testcase1() -> - [{silent_connections,[ftp]}]. + [{silent_connections,[ssh]}]. + my_testcase1(_) -> - ... + ... + my_testcase2(_) -> - ... + ... </pre> <p>In this example, <c>suite/0</c> tells Common Test to suppress - printouts from telnet and ftp connections. This is valid for + printouts from telnet and ssh connections. This is valid for all test cases. However, <c>my_testcase1/0</c> specifies that - for this test case, only ftp should be silent. The result is + for this test case, only ssh should be silent. The result is that <c>my_testcase1</c> will get telnet info (if any) printed - in the log, but not ftp info. <c>my_testcase2</c> will get no + in the log, but not ssh info. <c>my_testcase2</c> will get no info from either connection printed.</p> - <p>The <c>-silent_connections</c> tag (or - <c>silent_connections</c> tagged tuple in the call to - <c>ct:run_test/1</c>) overrides any settings in the test - suite.</p> + <p><c>silent_connections</c> may also be specified with a term + in a test specification + (see <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso>). Connections provided with the + <c>silent_connections</c> start flag/option, will be merged with + any connections listed in the test specification.</p> + + <p>The <c>silent_connections</c> start flag/option and test + specification term, overrides any settings made by the info functions + inside the test suite.</p> - <p>Note that in the current Common Test version, the + <note><p>Note that in the current Common Test version, the <c>silent_connections</c> feature only works for telnet - connections. Support for other connection types will be added - in future Common Test versions.</p> + and ssh connections! Support for other connection types will be added + in future Common Test versions.</p></note> </section> </chapter> |