From 84adefa331c4159d432d22840663c38f155cd4c1 Mon Sep 17 00:00:00 2001 From: Erlang/OTP Date: Fri, 20 Nov 2009 14:54:40 +0000 Subject: The R13B03 release. --- lib/common_test/doc/src/run_test_chapter.xml | 805 +++++++++++++++++++++++++++ 1 file changed, 805 insertions(+) create mode 100644 lib/common_test/doc/src/run_test_chapter.xml (limited to 'lib/common_test/doc/src/run_test_chapter.xml') diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml new file mode 100644 index 0000000000..d731d18783 --- /dev/null +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -0,0 +1,805 @@ + + + + +
+ + 20032009 + Ericsson AB. All Rights Reserved. + + + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + + + Running Test Suites + Peter Andersson, Kenneth Lundin + + + + run_test_chapter.xml +
+ +
+ Using the Common Test Framework + +

The Common Test Framework provides a high level + operator interface for testing. It adds the following features to + the Erlang/OTP Test Server:

+ + + Automatic compilation of test suites (and help modules). + Creation of additional HTML pages for better overview. + Single command interface for running all available tests. + Handling of configuration files specifying data related to + the System Under Test (and any other variable data). + Mode for running multiple independent test sessions in parallel with + central control and configuration. + +
+ +
+ Automatic compilation of test suites and help modules +

When Common Test starts, it will automatically attempt to compile any + suites included in the specified tests. If particular + suites have been specified, only those suites will be compiled. If a + particular test object directory has been specified (meaning all suites + in this directory should be part of the test), Common Test runs + make:all/1 in the directory to compile the suites.

+ +

If compilation should fail for one or more suites, the compilation errors + are printed to tty and the operator is asked if the test run should proceed + without the missing suites, or be aborted. If the operator chooses to proceed, + it is noted in the HTML log which tests have missing suites.

+ +

Any help module (i.e. regular Erlang module with name not ending with + "_SUITE") that resides in the same test object directory as a suite + which is part of the test, will also be automatically compiled. A help + module will not be mistaken for a test suite (unless it has a "_SUITE" + name of course). All help modules in a particular test object directory + are compiled no matter if all or only particular suites in the directory + are part of the test.

+ +

If test suites or help modules include header files stored in other + locations than the test directory, you may specify these include directories + by means of the flag with , + or the option with . + In addition to this, an include path may be specified with an OS + environment variable; . Example (bash):

+ +

$ export CT_INCLUDE_PATH=~testuser/common_suite_files/include:~testuser/common_lib_files/include

+ +

Common Test will pass all include directories (specified either with the + flag/option, or the + variable, or both) to the compiler.

+ +

It is also possible to specify include directories in test specifications + (see below).

+ +

If the user wants to run all test suites for a test object (or OTP application) + by specifying only the top directory (e.g. with the dir start flag/option), + Common Test will primarily look for test suite modules in a subdirectory named + test. If this subdirectory doesn't exist, the specified top directory + is assumed to be the actual test directory, and test suites will be read from + there instead.

+ +

It is possible to disable the automatic compilation feature by using the + flag with , or + the option with + . With automatic compilation + disabled, the user is responsible for compiling the test suite modules + (and any help modules) before the test run. Common Test will only verify + that the specified test suites exist before starting the tests.

+
+ +
+ Running tests from the UNIX command line + +

The script run_test can be used for running tests from + the Unix/Linux command line, e.g. +

+ + -dir ]]> + -suite ]]> + + -suite + -group -case ]]> + +

Examples:

+

$ run_test -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST

+

$ run_test -suite $SYS1_TEST/setup_SUITE $SYS2_TEST/config_SUITE

+

$ run_test -suite $SYS1_TEST/setup_SUITE -case start stop

+

$ run_test -suite $SYS1_TEST/setup_SUITE -group installation -case start stop

+ +

Other flags that may be used with run_test:

+ + ]]>, specifies where the HTML log files are to be written. + -refresh_logs, refreshes the top level HTML index files. + -vts, start web based GUI (see below). + -shell, start interactive shell mode (see below). + -step [step_opts], step through test cases using the Erlang Debugger (see below). + ]]>, use test specification as input (see below). + -allow_user_terms, allows user specific terms in a test specification (see below). + -silent_connections [conn_types], tells Common Test to suppress printouts for + specified connections (see below). + ]]>, points out a user HTML style sheet (see below). + ]]>, to perform code coverage test (see + Code Coverage Analysis). + ]]>, to install + event handlers. + , specifies include directories (see above). + , disables the automatic test suite compilation feature (see above). + ]]>, tells Common Test to repeat the tests n times (see below). + ]]>, tells Common Test to repeat the tests for duration of time (see below). + ]]>, tells Common Test to repeat the tests until stop_time (see below). + -force_stop, on timeout, the test run will be aborted when current test job is finished (see below). + ]]>, provides a decryption key for + encrypted configuration files. + ]]>, points out a file containing a decryption key for + encrypted configuration files. + , switches off html enhancements that might not be compatible with older browsers. + + +

Directories passed to Common Test may have either relative or absolute paths.

 + +

Arbitrary start flags to the Erlang Runtime System may also be passed as + parameters to run_test. It is, for example, useful to be able to + pass directories that should be added to the Erlang code server search path + with the -pa or -pz flag. If you have common help- or library + modules for test suites (separately compiled), stored in other directories + than the test suite directories, these help/lib directories are preferrably + added to the code path this way. Example:

+

$ run_test -dir ./chat_server -logdir ./chat_server/testlogs -pa $PWD/chat_server/ebin

+

Note how in this example, the absolute path of the chat_server/ebin + directory is passed to the code server. This is essential since relative + paths are stored by the code server as relative, and Common Test changes + the current working directory of the Erlang Runtime System during the test run!

+ + +

For details on how to generate the run_test script, see the + Installation chapter. +

+
+ +
+ Running tests from the Web based GUI + +

The web based GUI, VTS, is started with the run_test + script. From the GUI you can load config files, and select + directories, suites and cases to run. You can also state the + config files, directories, suites and cases on the command line + when starting the web based GUI. +

+ + + run_test -vts + ]]> + -suite + -case ]]> + + +

From the GUI you can run tests and view the result and the logs. +

+ +

Note that run_test -vts will try to open the Common Test start + page in an existing web browser window or start the browser if it is + not running. Which browser should be started may be specified with + the browser start command option:

+

]]>

+

Example:

+

+

Note that the browser must run as a separate OS process or VTS will hang!

+

If no specific browser start command is specified, netscape will + be the default browser on Unix platforms and Internet Explorer on Windows. + If Common Test fails to start a browser automatically, start your + favourite browser manually instead and type in the URL that Common Test + displays in the shell.

+
+ +
+ Running tests from the Erlang shell or from an Erlang program + +

Common Test provides an Erlang API for running tests. The main (and most + flexible) function for specifying and executing tests is called + ct:run_test/1. This function takes the same start parameters as + the run_test script described above, only the flags are instead + given as options in a list of key-value tuples. E.g. a test specified + with run_test like:

+

$ run_test -suite ./my_SUITE -logdir ./results

+

is with ct:run_test/1 specified as:

+

1> ct:run_test([{suite,"./my_SUITE"},{logdir,"./results"}]).

+

For detailed documentation, please see the ct manual page.

+
+ +
+ Running the interactive shell mode + +

You can start Common Test in an interactive shell mode where no + automatic testing is performed. Instead, in this mode, Common Test + starts its utility processes, installs configuration data (if any), + and waits for the user to call functions (typically test case support + functions) from the Erlang shell.

+ +

The shell mode is useful e.g. for debugging test suites, for analysing + and debugging the SUT during "simulated" test case execution, and + for trying out various operations during test suite development.

+ +

To invoke the interactive shell mode, you can start an Erlang shell + manually and call ct:install/1 to install any configuration + data you might need (use [] as argument otherwise), then + call ct:start_interactive/0 to start Common Test. If you use + the run_test script, you may start the Erlang shell and Common Test + in the same go by using the -shell and, optionally, the -config + flag: +

+ + run_test -shell + ]]> + + +

If no config file is given with the run_test command, + a warning will be displayed. If Common Test has been run from the same + directory earlier, the same config file(s) will be used + again. If Common Test has not been run from this directory before, no + config files will be available.

+ +

If any functions using "required config data" (e.g. ct_telnet or + ct_ftp functions) are to be called from the erlang shell, config + data must first be required with ct:require/[1,2]. This is + equivalent to a require statement in the Test Suite Info + Function or in the Test Case Info + Function.

+ +

Example:

+ 
 
+       1> ct:require(unix_telnet, unix).
+       ok
+       2> ct_telnet:open(unix_telnet).
+       {ok,<0.105.0>}
+       4> ct_telnet:cmd(unix_telnet, "ls .").
+       {ok,["ls .","file1  ...",...]}
+ +

Everything that Common Test normally prints in the test case logs, + will in the interactive mode be written to a log named + ctlog.html in the ]]> + directory. A link to this file will be available in the file + named last_interactive.html in the directory from which + you executed run_test. Currently, specifying a different + root directory for the logs than the current working directory, + is not supported.

+ +

If you wish to exit the interactive mode (e.g. to start an + automated test run with ct:run_test/1), call the function + ct:stop_interactive/0. This shuts down the + running ct application. Associations between + configuration names and data created with require are + consequently deleted. ct:start_interactive/0 will get you + back into interactive mode, but the previous state is not restored.

+
+ +
+ Step by step execution of test cases with the Erlang Debugger + +

By means of run_test -step [opts], or by passing the + {step,Opts} option to ct:run_test/1, it is possible + to get the Erlang Debugger started automatically and use its + graphical interface to investigate the state of the current test + case and to execute it step by step and/or set execution breakpoints.

+

If no extra options are given with the step flag/option, + breakpoints will be set automatically on the test cases that + are to be executed by Common Test, and those functions only. If + the step option config is specified, breakpoints will + also be initially set on the configuration functions in the suite, i.e. + init_per_suite/1, end_per_suite/1, init_per_testcase/2 + and end_per_testcase/2.

+

Common Test enables the Debugger auto attach feature, which means + that for every new interpreted test case function that starts to execute, + a new trace window will automatically pop up. (This is because each test + case executes on a dedicated Erlang process). Whenever a new test case starts, + Common Test will attempt to close the inactive trace window of the previous + test case. However, if you prefer that Common Test leaves inactive trace + windows, use the keep_inactive option.

+

The step functionality can be used together with the suite and + the suite + case/testcase flag/option, but not together + with dir.

+
+ +
+ + Using test specifications + +

The most expressive way to specify what to test is to use a so + called test specification. A test specification is a sequence of + Erlang terms. The terms may be declared in a text file or passed + to the test server at runtime as a list (see run_testspec/1 + in the manual page for ct). There are two general types + of terms: configuration terms and test specification terms.

+

With configuration terms it is possible to import configuration + data (similar to run_test -config), specify HTML log + directories (similar to run_test -logdir), give aliases + to test nodes and test directories (to make a specification + easier to read and maintain), enable code coverage analysis + (see the Code Coverage + Analysis chapter) and specify event_handler plugins + (see the + Event Handling chapter). There is also a term + for specifying include directories that should be passed on + to the compiler when automatic compilation is performed + (similar to run_test -include, see above).

+

With test specification terms it is possible to state exactly which + tests should run and in which order. A test term specifies either + one or more suites or one or more test cases. An arbitrary number of test + terms may be declared in sequence. A test term can also specify one or + more test suites or test cases to be skipped. Skipped suites and cases + are not executed and show up in the HTML test log as SKIPPED.

+ +

It is not yet possible to specify test case groups in + test specifications. This will be supported in a soon upcoming + release.

 + +

Below is the test specification syntax. Test specifications can + be used to run tests both in a single test host environment and in + a distributed Common Test environment. Node parameters are only relevant in the + latter (see the chapter about running Common Test in distributed mode for information). + For details on the event_handler term, see the + Event Handling + chapter.

+

Config terms:

+ 
+      {node, NodeAlias, Node}.
+ 
+      {cover, CoverSpecFile}.
+      {cover, NodeRef, CoverSpecFile}.
+      
+      {include, IncludeDirs}.
+      {include, NodeRefs, IncludeDirs}.
+
+      {config, ConfigFiles}.
+      {config, NodeRefs, ConfigFiles}.
+      
+      {alias, DirAlias, Dir}.
+      
+      {logdir, LogDir}.                                        
+      {logdir, NodeRefs, LogDir}.
+      
+      {event_handler, EventHandlers}.
+      {event_handler, NodeRefs, EventHandlers}.
+      {event_handler, EventHandlers, InitArgs}.
+      {event_handler, NodeRefs, EventHandlers, InitArgs}.
+
+

Test terms:

+ 
+      {suites, DirRef, Suites}.                                
+      {suites, NodeRefs, DirRef, Suites}.
+      
+      {cases, DirRef, Suite, Cases}.                           
+      {cases, NodeRefs, DirRef, Suite, Cases}.
+
+      {skip_suites, DirRef, Suites, Comment}.
+      {skip_suites, NodeRefs, DirRef, Suites, Comment}.
+      
+      {skip_cases, DirRef, Suite, Cases, Comment}.
+      {skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}.
+
+

Types:

+ 
+      NodeAlias     = atom()
+      Node          = node()
+      NodeRef       = NodeAlias | Node | master
+      NodeRefs      = all_nodes | [NodeRef] | NodeRef
+      CoverSpecFile = string()
+      IncludeDirs   = string() | [string()]
+      ConfigFiles   = string() | [string()]
+      DirAlias      = atom()
+      Dir           = string()
+      LogDir        = string()
+      EventHandlers = atom() | [atom()]
+      InitArgs      = [term()]
+      DirRef        = DirAlias | Dir
+      Suites        = atom() | [atom()] | all
+      Cases         = atom() | [atom()] | all
+      Comment       = string() | ""
+
+

Example:

+ 
+      {logdir, "/home/test/logs"}.
+      
+      {config, "/home/test/t1/cfg/config.cfg"}.
+      {config, "/home/test/t2/cfg/config.cfg"}.
+      {config, "/home/test/t3/cfg/config.cfg"}.
+      
+      {alias, t1, "/home/test/t1"}.
+      {alias, t2, "/home/test/t2"}.
+      {alias, t3, "/home/test/t3"}.
+      
+      {suites, t1, all}.
+      {skip_suites, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}.
+      {skip_cases, t1, t1A_SUITE, [test3,test4], "Irrelevant"}.
+      {skip_cases, t1, t1C_SUITE, [test1], "Ignore"}.
+      
+      {suites, t2, [t2B_SUITE,t2C_SUITE]}.
+      {cases, t2, t2A_SUITE, [test4,test1,test7]}.
+      
+      {skip_suites, t3, all, "Not implemented"}.
+
+

The example specifies the following:

+ + The specified logdir directory will be used for storing + the HTML log files (in subdirectories tagged with node name, + date and time). + The variables in the specified test system config files will be + imported for the test. + Aliases are given for three test system directories. The suites in + this example are stored in "/home/test/tX/test". + The first test to run includes all suites for system t1. Excluded from + the test are however the t1B and t1D suites. Also test cases test3 and + test4 in t1A as well as the test1 case in t1C are excluded from + the test. + Secondly, the test for system t2 should run. The included suites are + t2B and t2C. Included are also test cases test4, test1 and test7 in suite + t2A. Note that the test cases will be executed in the specified order. + Lastly, all suites for systems t3 are to be completely skipped and this + should be explicitly noted in the log files. + +

It is possible for the user to provide a test specification that + includes (for Common Test) unrecognizable terms. If this is desired, + the -allow_user_terms flag should be used when starting tests with + run_test. This forces Common Test to ignore unrecognizable terms. + Note that in this mode, Common Test is not able to check the specification + for errors as efficiently as if the scanner runs in default mode. + If ct:run_test/1 is used for starting the tests, the relaxed scanner + mode is enabled by means of the tuple: {allow_user_terms,true}

+
+ +
+ Log files + +

As the execution of the test suites proceed, events are logged in + four different ways:

+ + + Text to the operator's console. + Suite related information is sent to the major log file. + Case related information is sent to the minor log file. + The HTML overview log file gets updated with test results. + A link to all runs executed from a certain directory is written in + the log named "all_runs.html" and direct links to all tests (the + latest results) are written to the top level "index.html". + + +

Typically the operator, who may run hundreds or thousands of + test cases, doesn't want to fill the console with details + about, or printouts from, the specific test cases. By default, the + operator will only see:

+ + + A confirmation that the test has started and information about how + many test cases will be executed totally. + A small note about each failed test case. + A summary of all the run test cases. + A confirmation that the test run is complete. + Some special information like error reports and progress + reports, printouts written with erlang:display/1, or io:format/3 + specifically addressed to a receiver other than standard_io + (e.g. the default group leader process 'user'). + + +

If/when the operator wants to dig deeper into the general results, or + the result of a specific test case, he should do so by + following the links in the HTML presentation and take a look in the + major or minor log files. The "all_runs.html" page is a practical + starting point usually. It's located in logdir and contains + a link to each test run including a quick overview (date and time, + node name, number of tests, test names and test result totals).

+ +

An "index.html" page is written for each test run (i.e. stored in + the "ct_run" directory tagged with node name, date and time). This + file gives a short overview of all individual tests performed in the + same test run. The test names follow this convention:

+ + TopLevelDir.TestDir (all suites in TestDir executed) + TopLevelDir.TestDir:suites (specific suites were executed) + TopLevelDir.TestDir.Suite (all cases in Suite executed) + TopLevelDir.TestDir.Suite:cases (specific test cases were executed) + TopLevelDir.TestDir.Suite.Case (only Case was executed) + + +

On the test run index page there is a link to the Common Test + Framework log file in which information about imported + configuration data and general test progress is written. This + log file is useful to get snapshot information about the test + run during execution. It can also be very helpful when + analyzing test results or debugging test suites.

+ +

On the test run index page it is noted if a test has missing + suites (i.e. suites that Common Test has failed to + compile). Names of the missing suites can be found in the + Common Test Framework log file.

+ +

The major logfile shows a detailed report of the test run. It + includes test suite and test case names, execution time, the + exact reason for failures etc. The information is available in both + a file with textual and with HTML representation. The HTML file shows a + summary which gives a good overview of the test run. It also has links + to each individual test case log file for quick viewing with an HTML + browser.

+ +

The minor log file contain full details of every single test + case, each one in a separate file. This way the files should + be easy to compare with previous test runs, even if the set of + test cases change.

+ +

Which information goes where is user configurable via the + test server controller. Three threshold values determine what + comes out on screen, and in the major or minor log files. See + the OTP Test Server manual for information. The contents that + goes to the HTML log file is fixed however and cannot be altered.

+ +

The log files are written continously during a test run and links are + always created initially when a test starts. This makes it possible + to follow test progress simply by refreshing pages in the HTML browser. + Statistics totals are not presented until a test is complete however.

+ +
+ +
+ + HTML Style Sheets +

Common Test includes the optional feature to use + HTML style sheets (CSS) for customizing user printouts. The + functions in ct that print to a test case HTML log + file (log/3 and pal/3) accept Category + as first argument. With this argument it's possible to + specify a category that can be mapped to a selector in a CSS + definition. This is useful especially for coloring text + differently depending on the type of (or reason for) the + printout. Say you want one color for test system + configuration information, a different one for test system + state information and finally one for errors detected by the + test case functions. The corresponding style sheet may + look like this:

+ + 
+<style>
+  div.ct_internal { background:lightgrey; color:black }
+  div.default     { background:lightgreen; color:black }
+  div.sys_config  { background:blue; color:white }
+  div.sys_state   { background:yellow; color:black }
+  div.error       { background:red; color:white }
+</style>
+
+ +

To install the CSS file (Common Test inlines the definition in the + HTML code), the name may be provided when executing run_test. + Example:

+ + 
+	  $ run_test -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css
+
+ +

Categories in a CSS file installed with the -stylesheet flag + are on a global test level in the sense that they can be used in any + suite which is part of the test run.

+ +

It is also possible to install style sheets on a per suite and + per test case basis. Example:

+ + 
+	  -module(my_SUITE).
+	  ...
+	  suite() -> [..., {stylesheet,"suite_categories.css"}, ...].
+	  ...
+	  my_testcase(_) ->
+	      ...
+	      ct:log(sys_config, "Test node version: ~p", [VersionInfo]),
+	      ...
+	      ct:log(sys_state, "Connections: ~p", [ConnectionInfo]),
+	      ...
+	      ct:pal(error, "Error ~p detected! Info: ~p", [SomeFault,ErrorInfo]),
+	      ct:fail(SomeFault).
+
+ +

If the style sheet is installed as in this example, the categories are + private to the suite in question. They can be used by all test cases in the + suite, but can not be used by other suites. A suite private style sheet, + if specified, will be used in favour of a global style sheet (one specified + with the -stylesheet flag). A stylesheet tuple (as returned by suite/0 + above) can also be returned from a test case info function. In this case the + categories specified in the style sheet can only be used in that particular + test case. A test case private style sheet is used in favour of a suite or + global level style sheet. +

+ +

In a tuple {stylesheet,CSSFile}, if CSSFile is specified + with a path, e.g. "$TEST/styles/categories.css", this full + name will be used to locate the file. If only the file name is specified + however, e.g. "categories.css", then the CSS file is assumed to be located + in the data directory, data_dir, of the suite. The latter usage is + recommended since it is portable compared to hard coding path names in the + suite!

+ +

The Category argument in the example above may have the + value (atom) sys_config (white on blue), sys_state + (black on yellow) or error (white on red).

+ +

If the Category argument is not specified, Common Test will + use the CSS selector div.default for the + printout. For this reason a user supplied style sheet must + include this selector. Also the selector + div.ct_internal must be included. Hence a minimal + user style sheet should look like this (which is also the + default style sheet Common Test uses if no user CSS file is + provided):

+ 
+	  <style>
+	  div.ct_internal { background:lightgrey; color:black }
+	  div.default     { background:lightgreen; color:black }
+	  </style>
+
+
+ +
+ + Repeating tests +

You can order Common Test to repeat the tests you specify. You can choose + to repeat tests a certain number of times, repeat tests for a specific period of time, + or repeat tests until a particular stop time is reached. If repetition is controlled by + means of time, it is also possible to specify what action Common Test should + take upon timeout. Either Common Test performs all tests in the current run before stopping, + or it stops as soon as the current test job is finished. Repetition can be activated by + means of run_test start flags, or tuples in the ct:run:test/1 + option list argument. The flags (options in parenthesis) are:

+ + -repeat N ({repeat,N}), where N is a positive integer. + -duration DurTime ({duration,DurTime}), where DurTime is the duration, see below. + -until StopTime ({until,StopTime}), where StopTime is finish time, see below. + -force_stop ({force_stop,true}) + +

The duration time, DurTime, is specified as HHMMSS. Example: + -duration 012030 or {duration,"012030"}, means the tests will + be executed and (if time allows) repeated, until timeout occurs after 1 h, 20 min + and 30 secs. + StopTime can be specified as HHMMSS and is then interpreted as a time today + (or possibly tomorrow). StopTime can also be specified as YYMoMoDDHHMMSS. + Example: -until 071001120000 or {until,"071001120000"}, which means the tests + will be executed and (if time allows) repeated, until 12 o'clock on the 1st of Oct 2007.

+ +

When timeout occurs, Common Test will never abort the test run immediately, since + this might leave the system under test in an undefined, and possibly bad, state. + Instead Common Test will finish the current test job, or the complete test + run, before stopping. The latter is the default behaviour. The force_stop + flag/option tells Common Test to stop as soon as the current test job is finished. + Note that since Common Test always finishes off the current test job or test session, + the time specified with duration or until is never definitive!

+ +

Log files from every single repeated test run is saved in normal Common Test fashion (see above). + Common Test may later support an optional feature to only store the last (and possibly + the first) set of logs of repeated test runs, but for now the user must be careful not + to run out of disk space if tests are repeated during long periods of time.

+ +

Note that for each test run that is part of a repeated session, information about the + particular test run is printed in the Common Test Framework Log. There you can read + the repetition number, remaining time, etc.

+ +

Example 1:

+ 
+          $ run_test -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -force_stop
+

Here the suites in test directory to1, followed by the suites in to2, will be executed + in one test run. A timeout event will occur after 10 minutes. As long as there is time + left, Common Test will repeat the test run (i.e. starting over with the to1 test). + When the timeout occurs, Common Test will stop as soon as the current job is finished + (because of the force_stop flag). As a result, the specified test run might be + aborted after the to1 test and before the to2 test.

+ +

Example 2:

+ 
+          $ date
+	  Fri Sep 28 15:00:00 MEST 2007
+
+          $ run_test -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -until 160000
+

Here the same test run as in the example above will be executed (and possibly repeated). + In this example, however, the timeout will occur after 1 hour and when that happens, + Common Test will finish the entire test run before stopping (i.e. the to1 and to2 test + will always both be executed in the same test run).

+ +

Example 3:

+ 
+          $ run_test -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -repeat 5
+

Here the test run, including both the to1 and the to2 test, will be repeated 5 times.

+ +

This feature should not be confused with the repeat property of a test + case group. The options described here are used to repeat execution of entire test runs, + while the repeat property of a test case group makes it possible to repeat + execution of sets of test cases within a suite. For more information about the latter, + see the Writing Test Suites + chapter.

 +
+ +
+ + Silent Connections +

The protocol handling processes in Common Test, implemented by ct_telnet, ct_ftp etc, + do verbose printing to the test case logs. This can be switched off by means + of the -silent_connections flag:

+ + 
+	run_test -silent_connections [conn_types]
+
+ +

where conn_types specifies telnet, ftp, rpc and/or snmp.

+ +

Example:

+ + 
+	run_test ... -silent_connections telnet ftp
+

switches off logging for telnet and ftp connections.

+ + 
+	run_test ... -silent_connections
+

switches off logging for all connection types.

+ + +

Basic and important information such as opening and closing a connection, + fatal communication error and reconnection attempts will always be printed even + if logging has been suppressed for the connection type in question. However, operations + such as sending and receiving data may be performed silently.

+ +

It is possible to also specify silent_connections in a test suite. This is + accomplished by returning a tuple, {silent_connections,ConnTypes}, in the + suite/0 or test case info list. If ConnTypes is a list of atoms + (telnet, ftp, rpc and/or snmp), output for any corresponding connections + will be suppressed. Full logging is per default enabled for any connection of type not + specified in ConnTypes. Hence, if ConnTypes is the empty list, logging + is enabled for all connections.

+ +

The silent_connections setting returned from a test case info function overrides, + for the test case in question, any setting made with suite/0 (which is the setting + used for all cases in the suite). Example:

+ + 
+	
+	-module(my_SUITE).
+	...
+	suite() -> [..., {silent_connections,[telnet,ftp]}, ...].
+	...
+	my_testcase1() ->
+	[{silent_connections,[ftp]}].
+	my_testcase1(_) ->
+	...
+	my_testcase2(_) ->
+	...
+
+ +

In this example, suite/0 tells Common Test to suppress + printouts from telnet and ftp connections. This is valid for + all test cases. However, my_testcase1/0 specifies that + for this test case, only ftp should be silent. The result is + that my_testcase1 will get telnet info (if any) printed + in the log, but not ftp info. my_testcase2 will get no + info from either connection printed.

+ +

The -silent_connections tag (or + silent_connections tagged tuple in the call to + ct:run_test/1) overrides any settings in the test + suite.

+ +

Note that in the current Common Test version, the + silent_connections feature only works for telnet + connections. Support for other connection types will be added + in future Common Test versions.

+ +
+ + + -- cgit v1.2.3