diff options
Diffstat (limited to 'lib/common_test/doc/src')
26 files changed, 1119 insertions, 191 deletions
diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index 99161ce68a..57233a7f6c 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -47,6 +47,7 @@ CT_MODULES = \ ct_snmp \ unix_telnet \ ct_slave \ + ct_property_test \ ct_netconfc CT_XML_FILES = $(CT_MODULES:=.xml) diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index ff6ea6c557..12c6143a1a 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/book.xml b/lib/common_test/doc/src/book.xml index 2c48664b26..f3f28999d7 100644 --- a/lib/common_test/doc/src/book.xml +++ b/lib/common_test/doc/src/book.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE book SYSTEM "book.dtd"> <book xmlns:xi="http://www.w3.org/2001/XInclude"> <header> <copyright> - <year>2003</year><year>2009</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index 151159ad69..ea4522c40b 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE erlref SYSTEM "erlref.dtd"> <erlref> @@ -99,11 +99,11 @@ be executed by Common Test. A test case is represented by an atom, the name of the test case function. A test case group is represented by a <c>group</c> tuple, where <c>GroupName</c>, - an atom, is the name of the group (defined in <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). + an atom, is the name of the group (defined in <seealso marker="#Module:groups-0"><c>groups/0</c></seealso>). Execution properties for groups may also be specified, both for a top level group and for any of its sub-groups. Group execution properties specified here, will override - properties in the group definition (see <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). + properties in the group definition (see <seealso marker="#Module:groups-0"><c>groups/0</c></seealso>). (With value <c>default</c>, the group definition properties will be used).</p> @@ -186,8 +186,8 @@ test cases in the suite).</p> <p>The <c>timetrap</c> tag sets the maximum time each - test case is allowed to execute (including <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> - and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c>). If the timetrap time is + test case is allowed to execute (including <seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso> + and <seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso>). If the timetrap time is exceeded, the test case fails with reason <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to set a new timetrap by returning a <c>TimeVal</c>. It may also be @@ -203,11 +203,11 @@ in any of the configuration files, all test cases are skipped. For more information about the 'require' functionality, see the reference manual for the function - <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> + <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso>.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test suite related information which can be - read by calling <c><seealso marker="ct#userdata-2">ct:userdata/2</seealso></c>.</p> + read by calling <seealso marker="ct#userdata-2"><c>ct:userdata/2</c></seealso>.</p> <p>The <c>ct_hooks</c> tag specifies which <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> @@ -266,7 +266,7 @@ <p>This function is called as the last test case in the suite. It is meant to be used for cleaning up after - <c><seealso marker="#Module:init_per_suite-1">init_per_suite/1</seealso></c>. + <seealso marker="#Module:init_per_suite-1"><c>init_per_suite/1</c></seealso>. For information on <c>save_config</c>, please see <seealso marker="dependencies_chapter#save_config">Dependencies between Test Cases and Suites</seealso> in the User's Guide.</p> @@ -313,13 +313,13 @@ return a list of tagged tuples that specify various properties related to the execution of a test case group (i.e. its test cases and sub-groups). Properties set by - <c><seealso marker="#Module:group-1">group/1</seealso></c> override + <seealso marker="#Module:group-1"><c>group/1</c></seealso> override properties with the same key that have been previously set by - <c><seealso marker="#Module:suite-0">suite/0</seealso></c>.</p> + <seealso marker="#Module:suite-0"><c>suite/0</c></seealso>.</p> <p>The <c>timetrap</c> tag sets the maximum time each - test case is allowed to execute (including <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> - and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c>). If the timetrap time is + test case is allowed to execute (including <seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso> + and <seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso>). If the timetrap time is exceeded, the test case fails with reason <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to set a new timetrap by returning a <c>TimeVal</c>. It may also be @@ -334,11 +334,11 @@ in any of the configuration files, all test cases in this group are skipped. For more information about the 'require' functionality, see the reference manual for the function - <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> + <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso>.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test case group related information which can be - read by calling <c><seealso marker="ct#userdata-2">ct:userdata/2</seealso></c>.</p> + read by calling <seealso marker="ct#userdata-2"><c>ct:userdata/2</c></seealso>.</p> <p>The <c>ct_hooks</c> tag specifies which <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> @@ -371,7 +371,7 @@ test case group. It typically contains initializations which are common for all test cases and sub-groups in the group, and which shall only be performed once. <c>GroupName</c> is the name of the - group, as specified in the group definition (see <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). The + group, as specified in the group definition (see <seealso marker="#Module:groups-0"><c>groups/0</c></seealso>). The <c>Config</c> parameter is the configuration data which can be modified here. The return value of this function is given as <c>Config</c> to all test cases and sub-groups in the group. If <c>{skip,Reason}</c> @@ -400,10 +400,10 @@ <p> OPTIONAL </p> <p>This function is called after the execution of a test case group is finished. - It is meant to be used for cleaning up after <c><seealso marker="#Module:init_per_group-2">init_per_group/2</seealso></c>. + It is meant to be used for cleaning up after <seealso marker="#Module:init_per_group-2"><c>init_per_group/2</c></seealso>. By means of <c>{return_group_result,Status}</c>, it is possible to return a status value for a nested sub-group. The status can be retrieved in - <c><seealso marker="#Module:end_per_group-2">end_per_group/2</seealso></c> for the group on the level above. The status will also + <seealso marker="#Module:end_per_group-2"><c>end_per_group/2</c></seealso> for the group on the level above. The status will also be used by Common Test for deciding if execution of a group should proceed in case the property <c>sequence</c> or <c>repeat_until_*</c> is set.</p> @@ -454,7 +454,7 @@ <p> OPTIONAL </p> <p> This function is called after each test case, and can be used - to clean up after <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> and the test case. + to clean up after <seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso> and the test case. Any return value (besides <c>{fail,Reason}</c> and <c>{save_config,SaveConfig}</c>) is ignored. By returning <c>{fail,Reason}</c>, <c>TestCase</c> will be marked as failed (even though it was actually successful in the sense that it returned @@ -496,15 +496,15 @@ <p>This is the test case info function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of this particular test case. - Properties set by <c><seealso marker="#Module:Testcase-0">Testcase/0</seealso></c> override + Properties set by <seealso marker="#Module:Testcase-0"><c>Testcase/0</c></seealso> override properties that have been previously set for the test case - by <c><seealso marker="#Module:group-1">group/1</seealso></c> or <c><seealso marker="#Module:suite-0">suite/0</seealso></c>.</p> + by <seealso marker="#Module:group-1"><c>group/1</c></seealso> or <seealso marker="#Module:suite-0"><c>suite/0</c></seealso>.</p> <p>The <c>timetrap</c> tag sets the maximum time the test case is allowed to execute. If the timetrap time is exceeded, the test case fails with reason - <c>timetrap_timeout</c>. <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> - and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c> are included in the + <c>timetrap_timeout</c>. <seealso marker="#Module:init_per_testcase-2"><c>init_per_testcase/2</c></seealso> + and <seealso marker="#Module:end_per_testcase-2"><c>end_per_testcase/2</c></seealso> are included in the timetrap time. A <c>TimeFunc</c> function can be used to set a new timetrap by returning a <c>TimeVal</c>. It may also be used to trigger a timetrap timeout by, at some point, returning a @@ -518,15 +518,15 @@ configuration files, the test case is skipped. For more information about the 'require' functionality, see the reference manual for the function - <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> + <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso>.</p> <p>If <c>timetrap</c> and/or <c>require</c> is not set, the - default values specified by <c><seealso marker="#Module:suite-0">suite/0</seealso></c> (or - <c><seealso marker="#Module:group-1">group/1</seealso></c>) will be used.</p> + default values specified by <seealso marker="#Module:suite-0"><c>suite/0</c></seealso> (or + <seealso marker="#Module:group-1"><c>group/1</c></seealso>) will be used.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test case related information which can be - read by calling <c><seealso marker="ct#userdata-3">ct:userdata/3</seealso></c>.</p> + read by calling <seealso marker="ct#userdata-3"><c>ct:userdata/3</c></seealso>.</p> <p>Other tuples than the ones defined will simply be ignored.</p> @@ -554,7 +554,7 @@ <p>This is the implementation of a test case. Here you must call the functions you want to test, and do whatever you need to check the result. If something fails, make sure the - function causes a runtime error, or call <c><seealso marker="ct#fail-1">ct:fail/1/2</seealso></c> + function causes a runtime error, or call <seealso marker="ct#fail-1"><c>ct:fail/1/2</c></seealso> (which also causes the test case process to terminate).</p> <p>Elements from the <c>Config</c> list can e.g. be read @@ -582,7 +582,7 @@ Test Cases and Suites</seealso> in the User's Guide.</p> </desc> </func> - + </funcs> </erlref> diff --git a/lib/common_test/doc/src/config_file_chapter.xml b/lib/common_test/doc/src/config_file_chapter.xml index d90adf8d7b..d3068db17b 100644 --- a/lib/common_test/doc/src/config_file_chapter.xml +++ b/lib/common_test/doc/src/config_file_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2004</year><year>2012</year> + <year>2004</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -80,7 +80,7 @@ test is skipped (unless a default value has been specified, see the <seealso marker="write_test_chapter#info_function">test case info function</seealso> chapter for details). There is also a function - <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> which can be called from a test case + <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso> which can be called from a test case in order to check if a specific variable is available. The return value from this function must be checked explicitly and appropriate action be taken depending on the result (e.g. to skip the test case @@ -90,7 +90,7 @@ info-list should look like this: <c>{require,CfgVarName}</c> or <c>{require,AliasName,CfgVarName}</c>. The arguments <c>AliasName</c> and <c>CfgVarName</c> are the same as the - arguments to <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> which are described in the + arguments to <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso> which are described in the reference manual for <seealso marker="ct">ct</seealso>. <c>AliasName</c> becomes an alias for the configuration variable, and can be used as reference to the configuration data value. @@ -103,7 +103,7 @@ (or test case) and improve readability.</item> </list> <p>To read the value of a config variable, use the function - <c><seealso marker="ct#get_config-1">get_config/1/2/3</seealso></c> + <seealso marker="ct#get_config-1"><c>get_config/1/2/3</c></seealso> which is also described in the reference manual for <seealso marker="ct">ct</seealso>.</p> <p>Example:</p> @@ -121,7 +121,7 @@ <section> <title>Using configuration variables defined in multiple files</title> <p>If a configuration variable is defined in multiple files and you - want to access all possible values, you may use the <c><seealso marker="ct#get_config-3">ct:get_config/3</seealso></c> + want to access all possible values, you may use the <seealso marker="ct#get_config-3"><c>ct:get_config/3</c></seealso> function and specify <c>all</c> in the options list. The values will then be returned in a list and the order of the elements corresponds to the order that the config files were specified at startup. Please see @@ -133,7 +133,7 @@ <marker id="encrypted_config_files"></marker> <p>It is possible to encrypt configuration files containing sensitive data if these files must be stored in open and shared directories.</p> - <p>Call <c><seealso marker="ct#encrypt_config_file-2">ct:encrypt_config_file/2/3</seealso></c> to have Common Test encrypt a + <p>Call <seealso marker="ct#encrypt_config_file-2"><c>ct:encrypt_config_file/2/3</c></seealso> to have Common Test encrypt a specified file using the DES3 function in the OTP <c>crypto</c> application. The encrypted file can then be used as a regular configuration file, in combination with other encrypted files or normal text files. The key @@ -142,7 +142,7 @@ <c>decrypt_file</c> flag/option, or a key file in a predefined location.</p> <p>Common Test also provides decryption functions, - <c><seealso marker="ct#decrypt_config_file-2">ct:decrypt_config_file/2/3</seealso></c>, for recreating the original text + <seealso marker="ct#decrypt_config_file-2"><c>ct:decrypt_config_file/2/3</c></seealso>, for recreating the original text files.</p> <p>Please see the <seealso marker="ct">ct</seealso> reference manual for @@ -152,8 +152,8 @@ <section> <title>Opening connections by using configuration data</title> <p>There are two different methods for opening a connection - by means of the support functions in e.g. <c><seealso marker="ct_ssh">ct_ssh</seealso></c>, <c><seealso marker="ct_ftp">ct_ftp</seealso></c>, - and <c><seealso marker="ct_telnet">ct_telnet</seealso></c>:</p> + by means of the support functions in e.g. <seealso marker="ct_ssh"><c>ct_ssh</c></seealso>, <seealso marker="ct_ftp"><c>ct_ftp</c></seealso>, + and <seealso marker="ct_telnet"><c>ct_telnet</c></seealso>:</p> <list> <item>Using a configuration target name (an alias) as reference.</item> <item>Using the configuration variable as reference.</item> diff --git a/lib/common_test/doc/src/cover_chapter.xml b/lib/common_test/doc/src/cover_chapter.xml index 736486350b..accb94e1a9 100644 --- a/lib/common_test/doc/src/cover_chapter.xml +++ b/lib/common_test/doc/src/cover_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2006</year><year>2013</year> + <year>2006</year><year>2014</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -81,10 +81,7 @@ specify that previously exported data should be imported and included in the analysis for a test (you can specify multiple import files). This way it is possible to analyse total code coverage - without necessarily running all tests at once. Note that even if - you run separate tests in one test run, code coverage data will - not be passed on from one test to another unless you specify an - export file for Common Test to use for this purpose.</p> + without necessarily running all tests at once.</p> <p>To activate the code coverage support, you simply specify the name of the cover specification file as you start Common Test. @@ -94,7 +91,7 @@ <p><c>$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</c></p> <p>You may also pass the cover specification file name in a - call to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, by adding a <c>{cover,CoverSpec}</c> + call to <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, by adding a <c>{cover,CoverSpec}</c> tuple to the <c>Opts</c> argument. Also, you can of course enable code coverage in your test specifications (read more in the chapter about @@ -102,8 +99,8 @@ specifications</seealso>).</p> </section> - <marker id="cover_stop"></marker> <section> + <marker id="cover_stop"></marker> <title>Stopping the cover tool when tests are completed</title> <p>By default the Cover tool is automatically stopped when the tests are completed. This causes the original (non cover @@ -120,8 +117,8 @@ <p>The option can be set by using the <c>-cover_stop</c> flag with <c>ct_run</c>, by adding <c>{cover_stop,true|false}</c> to the - Opts argument to <c><seealso - marker="ct#run_test-1">ct:run_test/1</seealso></c>, or by adding + Opts argument to <seealso + marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, or by adding a <c>cover_stop</c> term in your test specification (see chapter about <seealso marker="run_test_chapter#test_specifications">test @@ -189,8 +186,8 @@ specification file for Common Test).</p> </section> - <marker id="cross_cover"/> <section> + <marker id="cross_cover"/> <title>Cross cover analysis</title> <p>The cross cover mechanism allows cover analysis of modules across multiple tests. It is useful if some code, e.g. a library @@ -266,10 +263,20 @@ ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).</code> <section> <title>Logging</title> - <p>To view the result of a code coverage test, follow the - "Coverage log" link on the test suite results page. This - takes you to the code coverage overview page. If you have - successfully performed a detailed coverage analysis, you + <p>To view the result of a code coverage test, click the button + labled "COVER LOG" in the top level index page for the test run.</p> + + <p>Prior to Erlang/OTP 17.1, if your test run consisted of + multiple tests, cover would be started and stopped for each test + within the test run. Separate logs would be available via the + "Coverage log" link on the test suite result pages. These links + are still available, but now they all point to the same page as + the button on the top level index page. The log contains the + accumulated results for the complete test run. See the release + notes for more information about this change.</p> + + <p>The buttonc takes you to the code coverage overview page. If you + have successfully performed a detailed coverage analysis, you find links to each individual module coverage page here.</p> <p>If cross cover analysis has been performed, and there are diff --git a/lib/common_test/doc/src/ct_hooks.xml b/lib/common_test/doc/src/ct_hooks.xml index b3e713c77f..cab6dfea51 100644 --- a/lib/common_test/doc/src/ct_hooks.xml +++ b/lib/common_test/doc/src/ct_hooks.xml @@ -450,12 +450,15 @@ </func> <func> - <name>Module:on_tc_fail(TestcaseName, Reason, CTHState) -> + <name>Module:on_tc_fail(TestName, Reason, CTHState) -> NewCTHState</name> <fsummary>Called after the CTH scope ends</fsummary> <type> - <v>TestcaseName = init_per_suite | end_per_suite | - init_per_group | end_per_group | atom()</v> + <v>TestName = init_per_suite | end_per_suite | + {init_per_group,GroupName} | {end_per_group,GroupName} | + {FuncName,GroupName} | FuncName</v> + <v>FuncName = atom()</v> + <v>GroupName = atom()</v> <v>Reason = term()</v> <v>CTHState = NewCTHState = term()</v> </type> @@ -463,14 +466,16 @@ <desc> <p> OPTIONAL </p> - <p>This function is called whenever a testcase fails. - It is called after the post function has been called for - the testcase which failed. i.e. - if init_per_suite fails this function is called after + <p>This function is called whenever a test case (or config function) + fails. It is called after the post function has been called for + the failed test case. I.e. if init_per_suite fails, this function + is called after <seealso marker="#Module:post_init_per_suite-4"> - post_init_per_suite</seealso>, and if a testcase fails it is called + post_init_per_suite</seealso>, and if a test case fails, it is called after <seealso marker="#Module:post_end_per_testcase-4"> - post_end_per_testcase</seealso>.</p> + post_end_per_testcase</seealso>. If the failed test case belongs + to a test case group, the first argument is a tuple + <c>{FuncName,GroupName}</c>, otherwise simply the function name.</p> <p>The data which comes with the Reason follows the same format as the <seealso marker="event_handler_chapter#failreason">FailReason @@ -481,12 +486,15 @@ </func> <func> - <name>Module:on_tc_skip(TestcaseName, Reason, CTHState) -> + <name>Module:on_tc_skip(TestName, Reason, CTHState) -> NewCTHState</name> <fsummary>Called after the CTH scope ends</fsummary> <type> - <v>TestcaseName = end_per_suite | init_per_group | - end_per_group | atom()</v> + <v>TestName = init_per_suite | end_per_suite | + {init_per_group,GroupName} | {end_per_group,GroupName} | + {FuncName,GroupName} | FuncName</v> + <v>FuncName = atom()</v> + <v>GroupName = atom()</v> <v>Reason = {tc_auto_skip | tc_user_skip, term()}</v> <v>CTHState = NewCTHState = term()</v> </type> @@ -494,14 +502,17 @@ <desc> <p> OPTIONAL </p> - <p>This function is called whenever a testcase is skipped. - It is called after the post function has been called for the - testcase which was skipped. - i.e. if init_per_group is skipped this function is called after - <seealso marker="#Module:post_init_per_suite-4">post_init_per_group - </seealso>, and if a testcase is skipped it is called after - <seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase - </seealso>.</p> + <p>This function is called whenever a test case (or config function) + is skipped. It is called after the post function has been called + for the skipped test case. I.e. if init_per_group is skipped, this + function is called after + <seealso marker="#Module:post_init_per_group-4"> + post_init_per_group</seealso>, and if a test case is skipped, + it is called after + <seealso marker="#Module:post_end_per_testcase-4"> + post_end_per_testcase</seealso>. If the skipped test case belongs to a + test case group, the first argument is a tuple <c>{FuncName,GroupName}</c>, + otherwise simply the function name.</p> <p>The data which comes with the Reason follows the same format as <seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml index fe871eb516..60cd9be918 100644 --- a/lib/common_test/doc/src/ct_hooks_chapter.xml +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -192,12 +192,12 @@ <section> <title>External configuration data and Logging</title> <p>It's possible in the CTH to read configuration data values - by calling <c><seealso marker="ct#get_config-1">ct:get_config/1/2/3</seealso></c> (as explained in the + by calling <seealso marker="ct#get_config-1"><c>ct:get_config/1/2/3</c></seealso> (as explained in the <seealso marker="config_file_chapter#require_config_data"> External configuration data</seealso> chapter). The config variables in question must, as always, first have been <c>required</c> by means of a suite-, group-, or test case info function, - or the <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> function. Note that the latter can also be used + or the <seealso marker="ct#require-1"><c>ct:require/1/2</c></seealso> function. Note that the latter can also be used in CT hook functions.</p> <p>The CT hook functions may call any of the logging functions available in the <c>ct</c> interface to print information to the log files, or to @@ -252,13 +252,13 @@ {ok, Handle} -> {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} end.</code> - <note>If using multiple CTHs, the first part of the return tuple will be + <note><p>If using multiple CTHs, the first part of the return tuple will be used as input for the next CTH. So in the case above the next CTH might get <c>{fail,Reason}</c> as the second parameter. If you have many CTHs which interact, it might be a good idea to not let each CTH return <c>fail</c> or <c>skip</c>. Instead return that an action should be taken through the <c>Config</c> list and implement a CTH which at the end takes - the correct action. </note> + the correct action.</p></note> </section> @@ -301,9 +301,9 @@ post_end_per_testcase(_TC, Config, Return, CTHState) -> %% Do nothing if tc does not crash. {Return, CTHState}.</code> - <note>Recovering from a testcase failure using CTHs should only be done as + <note><p>Recovering from a testcase failure using CTHs should only be done as a last resort. If used wrongly it could become very difficult to - determine which tests pass or fail in a test run</note> + determine which tests pass or fail in a test run</p></note> </section> @@ -322,6 +322,34 @@ post_end_per_testcase(_TC, Config, Return, CTHState) -> </section> + <marker id="synchronizing"/> + <section> + <title>Synchronizing external user applications with Common Test</title> + <p>CTHs can be used to synchronize test runs with external user applications. + The init function may e.g. start and/or communicate with an application that + has the purpose of preparing the SUT for an upcoming test run, or maybe + initialize a database for saving test data to during the test run. The + terminate function may similarly order such an application to reset the SUT + after the test run, and/or tell the application to finish active sessions + and terminate. + Any system error- or progress reports generated during the init- or + termination stage will be saved in the + <seealso marker="run_test_chapter#pre_post_test_io_log">Pre- + and post test I/O log</seealso>. (This is also true for any printouts made + with <c>ct:log/2</c> and <c>ct:pal/2</c>).</p> + <p>In order to ensure that Common Test doesn't start executing tests, or + closes its log files and shuts down, before the external application + is ready for it, Common Test may be synchronized with the application. + During startup and shutdown, Common Test can be suspended, simply by + having a CTH evaluate a <c>receive</c> expression in the init- or terminate + function. The macros <c>?CT_HOOK_INIT_PROCESS</c> (the process executing the hook + init function) and <c>?CT_HOOK_TERMINATE_PROCESS</c> (the process executing + the hook terminate function), each specifies the name of the correct Common Test + process to send a message to in order to return from the <c>receive</c>. + These macros are defined in <c>ct.hrl</c>. + </p> + </section> + <marker id="example"/> <section> <title>Example CTH</title> diff --git a/lib/common_test/doc/src/ct_master_chapter.xml b/lib/common_test/doc/src/ct_master_chapter.xml index 9e848e99bb..37a0805055 100644 --- a/lib/common_test/doc/src/ct_master_chapter.xml +++ b/lib/common_test/doc/src/ct_master_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2006</year><year>2012</year> + <year>2006</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -220,6 +220,7 @@ <p>The default <seealso marker="ct_slave">ct_slave</seealso> callback module, which is part of the Common Test application, has the following features: + </p> <list> <item>Starting Erlang target nodes on local or remote hosts (ssh is used for communication). @@ -237,7 +238,6 @@ Functions can be given as a list of {Module, Function, Arguments} tuples. </item> </list> - </p> <p>Note that it is possible to specify an <c>eval</c> term for the node as well as <c>startup_functions</c> in the <c>node_start</c> options list. In this case first the node will be started, then the <c>startup_functions</c> are diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml index c87c765ae7..d8e79ca80e 100644 --- a/lib/common_test/doc/src/ct_run.xml +++ b/lib/common_test/doc/src/ct_run.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE comref SYSTEM "comref.dtd"> <comref> @@ -36,8 +36,6 @@ OS command line. </comsummary> - <marker id="top"></marker> - <description> <p>The <c>ct_run</c> program is automatically installed with Erlang/OTP and Common Test (please see the Installation chapter in the Common @@ -48,7 +46,7 @@ particular mode.</p> <p>There is an interface function that corresponds to this program, - called <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, for starting Common Test from the Erlang + called <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, for starting Common Test from the Erlang shell (or an Erlang program). Please see the <c>ct</c> man page for details.</p> @@ -83,9 +81,9 @@ <p>it prints all valid start flags to stdout.</p> </description> - <marker id="ct_run"></marker> <section> + <marker id="ct_run"></marker> <title>Run tests from command line</title> <pre> ct_run [-dir TestDir1 TestDir2 .. TestDirN] | @@ -110,6 +108,7 @@ EvHandler2 InitArg2 and .. EvHandlerN InitArgN] [-include InclDir1 InclDir2 .. InclDirN] [-no_auto_compile] + [-abort_if_missing_suites] [-muliply_timetraps Multiplier] [-scale_timetraps] [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] @@ -146,6 +145,7 @@ EvHandler2 InitArg2 and .. EvHandlerN InitArgN] [-include InclDir1 InclDir2 .. InclDirN] [-no_auto_compile] + [-abort_if_missing_suites] [-muliply_timetraps Multiplier] [-scale_timetraps] [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] @@ -173,6 +173,7 @@ [-decrypt_key Key] | [-decrypt_file KeyFile] [-include InclDir1 InclDir2 .. InclDirN] [-no_auto_compile] + [-abort_if_missing_suites] [-muliply_timetraps Multiplier] [-scale_timetraps] [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml index 8aad552285..e6c7025b12 100644 --- a/lib/common_test/doc/src/dependencies_chapter.xml +++ b/lib/common_test/doc/src/dependencies_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2006</year><year>2009</year> + <year>2006</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml index 10a9b52d39..f39f391818 100644 --- a/lib/common_test/doc/src/event_handler_chapter.xml +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2006</year><year>2012</year> + <year>2006</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -59,12 +59,21 @@ Event handlers plugged into this manager will receive the events from all the test nodes as well as information from the CT Master server itself.</p> + + <p>User specific event handlers may be plugged into a Common Test event + manager, either by telling Common Test to install them before the test + run (see below), or by adding the handlers dynamically during the test + run by means of + <c>gen_event:add_handler/3</c> or <c>gen_event:add_sup_handler/3</c>. + In the latter scenario, the reference of the Common Test event manager is + required. To get it, call <c>ct:get_event_mgr_ref/0</c> or (on the CT + Master node) <c>ct_master:get_event_mgr_ref/0</c>.</p> </section> <section> <marker id="usage"></marker> <title>Usage</title> <p>Event handlers may be installed by means of an <c>event_handler</c> - start flag (<c>ct_run</c>) or option (<c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), where the + start flag (<c>ct_run</c>) or option (<seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>), where the argument specifies the names of one or more event handler modules. Example:</p> <p><c>$ ct_run -suite test/my_SUITE -event_handler handlers/my_evh1 @@ -78,7 +87,7 @@ example).</p> <p>An event_handler tuple in the argument <c>Opts</c> has the following - definition (see also <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> in the reference manual):</p> + definition (see also <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso> in the reference manual):</p> <pre> {event_handler,EventHandlers} @@ -193,7 +202,8 @@ <p><c>GroupName = unknown | atom()</c>, name of the group (unknown if init- or end function times out).</p> <p><c>GroupProperties = list()</c>, list of execution properties for the group.</p> - <p><c>Result = ok | {skipped,SkipReason} | {failed,FailReason}</c>, the result.</p> + <p><c>Result = ok | {auto_skipped,SkipReason} | {skipped,SkipReason} | {failed,FailReason}</c>, + the result.</p> <marker id="skipreason"/> <p><c>SkipReason = {require_failed,RequireInfo} | {require_failed_in_suite0,RequireInfo} | @@ -224,10 +234,15 @@ <c>end_per_testcase</c> for the case failed. </p></item> + <item> <marker id="tc_auto_skip"></marker> - <item><c>#event{name = tc_auto_skip, data = {Suite,Func,Reason}}</c> + <c>#event{name = tc_auto_skip, data = {Suite,TestName,Reason}}</c> <p><c>Suite = atom()</c>, the name of the suite.</p> - <p><c>Func = atom()</c>, the name of the test case or configuration function.</p> + <p><c>TestName = init_per_suite | end_per_suite | + {init_per_group,GroupName} | {end_per_group,GroupName} | + {FuncName,GroupName} | FuncName</c></p> + <p><c>FuncName = atom()</c>, the name of the test case or configuration function.</p> + <p><c>GroupName = atom()</c>, the name of the test case group.</p> <p><c>Reason = {failed,FailReason} | {require_failed_in_suite0,RequireInfo}</c>, reason for auto skipping <c>Func</c>.</p> @@ -248,18 +263,26 @@ <c>init_per_group</c>, a failed <c>require</c> in <c>suite/0</c>, or a failed test case in a sequence. Note that this event is never received as a result of a test case getting skipped because of <c>init_per_testcase</c> failing, since that information is carried with - the <c>tc_done</c> event. + the <c>tc_done</c> event. If a failed test case belongs to a test case group, the second + data element is a tuple <c>{FuncName,GroupName}</c>, otherwise simply the function name. </p></item> - + + <item> <marker id="tc_user_skip"></marker> - <item><c>#event{name = tc_user_skip, data = {Suite,TestCase,Comment}}</c> - <p><c>Suite = atom()</c>, name of the suite.</p> - <p><c>TestCase = atom()</c>, name of the test case.</p> + <c>#event{name = tc_user_skip, data = {Suite,TestName,Comment}}</c> + <p><c>Suite = atom()</c>, the name of the suite.</p> + <p><c>TestName = init_per_suite | end_per_suite | + {init_per_group,GroupName} | {end_per_group,GroupName} | + {FuncName,GroupName} | FuncName</c></p> + <p><c>FuncName = atom()</c>, the name of the test case or configuration function.</p> + <p><c>GroupName = atom()</c>, the name of the test case group.</p> <p><c>Comment = string()</c>, reason for skipping the test case.</p> <p>This event specifies that a test case has been skipped by the user. It is only ever received if the skip was declared in a test specification. Otherwise, user skip information is received as a <c>{skipped,SkipReason}</c> - result in the <c>tc_done</c> event for the test case. + result in the <c>tc_done</c> event for the test case. If a skipped test case belongs + to a test case group, the second data element is a tuple <c>{FuncName,GroupName}</c>, + otherwise simply the function name. </p></item> <item><c>#event{name = test_stats, data = {Ok,Failed,Skipped}}</c> @@ -308,7 +331,7 @@ manager can look like.</p> <note><p>To ensure that printouts to standard out (or printouts made with - <c><seealso marker="ct#log-2">ct:log/2/3</seealso></c> or <c><seealso marker="ct:pal-2">ct:pal/2/3</seealso></c>) get written to the test case log + <seealso marker="ct#log-2"><c>ct:log/2/3</c></seealso> or <seealso marker="ct:pal-2"><c>ct:pal/2/3</c></seealso>) get written to the test case log file, and not to the Common Test framework log, you can syncronize with the Common Test server by matching on the <c>tc_start</c> and <c>tc_done</c> events. In the period between these events, all IO gets directed to the diff --git a/lib/common_test/doc/src/example_chapter.xml b/lib/common_test/doc/src/example_chapter.xml index 2333f92989..36781d152c 100644 --- a/lib/common_test/doc/src/example_chapter.xml +++ b/lib/common_test/doc/src/example_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/fascicules.xml b/lib/common_test/doc/src/fascicules.xml index 38df4a1709..c4a28a699a 100644 --- a/lib/common_test/doc/src/fascicules.xml +++ b/lib/common_test/doc/src/fascicules.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE fascicules SYSTEM "fascicules.dtd"> <fascicules> diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml index 3cf04bb1a2..445c731d01 100644 --- a/lib/common_test/doc/src/getting_started_chapter.xml +++ b/lib/common_test/doc/src/getting_started_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2007</year><year>2012</year> + <year>2007</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -61,13 +61,11 @@ <title>Test case execution</title> <p>Execution of test cases is handled this way:</p> - <p> <image file="tc_execution.gif"> <icaption> Successful vs unsuccessful test case execution. </icaption> </image> - </p> <p>For each test case that Common Test is told to execute, it spawns a dedicated process on which the test case function in question starts @@ -90,7 +88,7 @@ <p>As you can understand from the illustration above, Common Test requires that a test case generates a runtime error to indicate failure (e.g. by causing a bad match error or by calling <c>exit/1</c>, preferrably - through the <c><seealso marker="ct#fail-1">ct:fail/1,2</seealso></c> help function). A succesful execution is + through the <seealso marker="ct#fail-1"><c>ct:fail/1,2</c></seealso> help function). A succesful execution is indicated by means of a normal return from the test case function. </p> </section> @@ -100,13 +98,15 @@ <p>As you've seen in the basics chapter, the test suite module implements <seealso marker="common_test">callback functions</seealso> (mandatory or optional) for various purposes, e.g: + </p> <list> <item>Init/end configuration function for the test suite</item> <item>Init/end configuration function for a test case</item> <item>Init/end configuration function for a test case group</item> <item>Test cases</item> </list> - The configuration functions are optional and if you don't need them for + <p> + The configuration functions are optional and if you don't need them for your test, a test suite with one simple test case could look like this: </p> <pre> @@ -136,13 +136,11 @@ "lower level"). The data flow looks like this: </p> - <p> <image file="config.gif"> <icaption> Config data flow in the suite. </icaption> </image> - </p> <p> Here's an example of a test suite which uses configuration functions @@ -203,13 +201,11 @@ shows the log file structure: </p> - <p> <image file="html_logs.gif"> <icaption> HTML log file structure. </icaption> </image> - </p> </section> <section> diff --git a/lib/common_test/doc/src/install_chapter.xml b/lib/common_test/doc/src/install_chapter.xml index 89c497962d..7f8c606324 100644 --- a/lib/common_test/doc/src/install_chapter.xml +++ b/lib/common_test/doc/src/install_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2007</year><year>2010</year> + <year>2007</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -56,13 +56,13 @@ shell script version run_test, however, this script needs to be generated first, according to the instructions below.</p> - <p><note>Before reading on, please note that since Common Test version + <note><p>Before reading on, please note that since Common Test version 1.5, the run_test shell script is no longer required for starting tests with Common Test from the OS command line. The ct_run program (descibed above) is the new recommended command line interface for Common Test. The shell script exists mainly for legacy reasons and may not be updated in future releases of Common Test. It may even be removed. - </note></p> + </p></note> <p>Optional step to generate a shell script for starting Common Test:</p> <p>To generate the run_test shell script, navigate to the diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index 2d6bcc0d8b..472e3b7833 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> @@ -32,6 +32,818 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.10.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + A fault in the Common Test logger process, that caused + the application to crash when running on a long name + node, has been corrected.</p> + <p> + Own Id: OTP-12643</p> + </item> + <item> + <p> + A 'wait_for_prompt' option in ct_telnet:expect/3 has been + introduced which forces the function to not return until + a prompt string has been received, even if other expect + patterns have already been found.</p> + <p> + Own Id: OTP-12688 Aux Id: seq12818 </p> + </item> + <item> + <p> + If the last expression in a test case causes a timetrap + timeout, the stack trace is ignored and not printed to + the test case log file. This happens because the + {Suite,TestCase,Line} info is not available in the stack + trace in this scenario, due to tail call elimination. + Common Test has been modified to handle this situation by + inserting a {Suite,TestCase,last_expr} tuple in the + correct place and printing the stack trace as expected.</p> + <p> + Own Id: OTP-12697 Aux Id: seq12848 </p> + </item> + <item> + <p> + Fixed a buffer problem in ct_netconfc which could cause + that some messages where buffered forever.</p> + <p> + Own Id: OTP-12698 Aux Id: seq12844 </p> + </item> + <item> + <p> + The VTS mode in Common Test has been modified to use a + private version of the Webtool application (ct_webtool).</p> + <p> + Own Id: OTP-12704 Aux Id: OTP-10922 </p> + </item> + <item> + <p> + Add possibility to add user capabilities in + <c>ct_netconfc:hello/3</c>.</p> + <p> + Own Id: OTP-12707 Aux Id: seq12846 </p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.10</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The tests overview file, index.html, did not always get + updated correctly after a new test run. This was because + of a bug in the Common Test log cache mechanism which has + now been corrected.</p> + <p> + Own Id: OTP-11400</p> + </item> + <item> + <p> + When a successful test case returns, Common Test should, + according to the documentation, send a tc_done event to + the event handlers with Result = ok in the data field. + However, Common Test sets Result to the return value of + the test case instead. Common Test has been modified now + to comply with the documentation.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-12279 Aux Id: seq12737, OTP-12531 </p> + </item> + <item> + <p> + A ct_telnet:expect/3 call could never be aborted before + an idle_timeout, even if total_timeout had been set to a + lower value (i.e. a shorter time). This problem has been + fixed.</p> + <p> + Own Id: OTP-12335</p> + </item> + <item> + <p> + The undocumented return value {skipped,Reason} from + config functions and test cases was handled + inconsistently. Test cases were e.g. reported as + "skipped" to CT Hook functions, but "successful" to event + handlers. Now, the above return value is consistently + handled the same way as {skip,Reason} and this has also + been documented.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-12359 Aux Id: seq12760 </p> + </item> + <item> + <p> + The Erlang source code to HTML generator would sometimes + fail because epp:parse_erl_form/1 could not find and + expand required macros in included header files. The + problem has been solved by making sure common_test always + passes the full include path to epp. Also, a bug that + could cause erl_syntax:revert/1 to fail because of a + badly formed syntax tree has been corrected.</p> + <p> + Own Id: OTP-12419</p> + </item> + <item> + <p> + A missing group option in the ct_run help text has been + added.</p> + <p> + Own Id: OTP-12433 Aux Id: seq12788 </p> + </item> + <item> + <p> + Printouts by means of ct:log/2/3 or ct:pal/2/3 from the + hook functions on_tc_fail/2 and on_tc_skip/2 would (quite + unexpectedly) end up in the "unexpected i/o" log file + instead of in the test case log file. This behaviour has + been changed so that now, all printouts (including stdio + printouts) from these hook functions will be routed to + the test case log file.</p> + <p> + Own Id: OTP-12468</p> + </item> + <item> + <p> + ct_netconfc:action/3 will now - if the return type is + void - accept an RPC reply on the form + {ok,[simple_xml()]}, and in this event return only the + atom ok.</p> + <p> + Own Id: OTP-12491 Aux Id: seq12797 </p> + </item> + <item> + <p> + OTP-11971 erroneously changed the handling of relative + paths for incl_dirs specified in the cover spec file. + This is now corrected so these are expected to be + relative to the directory where the cover spec file + itself is stored</p> + <p> + Own Id: OTP-12498 Aux Id: OTP-11971 </p> + </item> + <item> + <p> + Some test cases have been updated to use ct:sleep/1 + instead of timer:sleep/1. The reason being that the sleep + times need to be scaled to compensate for slow execution + (e.g. when cover is running).</p> + <p> + Own Id: OTP-12574</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Common Test now exports a function, + ct:get_event_mgr_ref/0, that returns the name of the + Common Test event manager. This makes it possible to plug + in event handlers to the event manager while tests are + running (using the gen_event API).</p> + <p> + Own Id: OTP-12506 Aux Id: seq12802 </p> + </item> + <item> + <p> + When a test case (or configuration function) fails + because of an exit signal from a linked process, Common + Test previously passed only the reason for process + termination to the CT post hook functions and the event + handlers (in the tc_done event). This has been changed so + that now the tuple {'EXIT',ReasonForProcessTermination} + is passed instead. This makes it much easier in the CT + post hook functions to distinguish a failure of this sort + from other types of errors and from the return value of a + successful test case.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-12531 Aux Id: OTP-12279 </p> + </item> + <item> + <p> + A new feature has been introduced in ct_telnet:get_data/1 + that makes it possible to automatically poll the telnet + connection in case an incomplete string (one that has not + yet been terminated by a newline) remains in the receive + buffer. The polling is controlled by two new telnet + config values, which are documented in the ct_telnet + reference manual. The polling mechanism is disabled by + default (making the get_data/1 function backwards + compatible).</p> + <p> + Own Id: OTP-12627</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.9</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The source code to html code generator in Test Server + (and Common Test) would fail to generate anchors in the + html code for functions with non-expandable macros, + resulting in bad html links to such functions. This + correction lets the code generator ignore macros that + can't be expanded (i.e. not pre-process them), so that + correct anchors will always be produced.</p> + <p> + Own Id: OTP-11766 Aux Id: seq12556 </p> + </item> + <item> + <p> + OTP-11971 erroneously changed the handling of relative + paths (import/export files) specified in the cover spec + file. This is now corrected so these are expected to be + relative to the directory where the cover spec file + itself is stored.</p> + <p> + Own Id: OTP-12031</p> + </item> + <item> + <p> + Common Test would sometimes crash while trying to print + large amounts of SASL reports to log on a computer with a + slow file system. This problem (due to an error in IO + message buffering in ct_logs) has been fixed.</p> + <p> + Own Id: OTP-12159</p> + </item> + <item> + <p> + The common_test telnet client, ct_telnet and friends, had + some unstable test cases. Some of these were caused by + the unix_telnet callback sending an extra newline after + sending the password. This caused the sever to send an + extra prompt back which confused the tests. The extra + newline is no longer sent.</p> + <p> + Also, debug printouts and logging from the telnet client + is improved, and some test cases are slightly modified in + order to stabilize the test.</p> + <p> + Own Id: OTP-12329</p> + </item> + <item> + <p> + ct_netconfc did not expect the return value + {error,timeout} from ssh_connection:subsystem/4. This has + been corrected.</p> + <p> + Own Id: OTP-12334</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A new option, <c>{newline,boolean()}</c> is added to all + functions in <c>ct_telnet</c> that send data (command + strings) to the telnet server. By default, + <c>ct_telnet</c> adds a newline to all command strings, + and by setting the new option to <c>false</c> this + behavior is turned off.</p> + <p> + Own Id: OTP-12252 Aux Id: seq12730 </p> + </item> + <item> + <p> + Distribute <c>autoconf</c> helpers to applications at + build time instead of having multiple identical copies + committed in the repository.</p> + <p> + Own Id: OTP-12348</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.8.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Ticket OTP-11971 introduced a runtime dependency towards + test_server-3.7.1, since the interface between + test_server and common_test was changed. Erroneously, the + common_test.app file was not updated according to this. + This has now been corrected.</p> + <p> + Own Id: OTP-12037</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Warning: this is experimental and may disappear or change + without previous warning.</p> + <p> + Experimental support for running Quickcheck and PropEr + tests from common_test suites is added to common_test. + See the reference manual for the new module + <c>ct_property_testing</c>.</p> + <p> + Experimental property tests are added under + <c>lib/{inet,ssh}/test/property_test</c>. They can be run + directly or from the commont_test suites + <c>inet/ftp_property_test_SUITE.erl</c> and + <c>ssh/test/ssh_property_test_SUITE.erl</c>.</p> + <p> + See the code in the <c>test</c> directories and the man + page for details.</p> + <p> + (Thanks to Tuncer Ayaz for a patch adding Triq)</p> + <p> + Own Id: OTP-12119</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.8.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Substrings in long telnet messages would sometimes get + wrongly reversed. This error has been corrected.</p> + <p> + Own Id: OTP-11871 Aux Id: seq12581 </p> + </item> + <item> + <p> + The basic_html logging mode in Common Test (for + compatibility with old browsers) generated HTML code with + unbalanced tags. This has been fixed.</p> + <p> + Own Id: OTP-11917 Aux Id: seq12598 </p> + </item> + <item> + <p> + The mechanism for running code cover analysis with + common_test has been improved. Earlier, if a test run + consisted of multiple tests, cover would be started and + stopped for each test. This would give "intermediate" + cover logs available from the "Coverage log" link on the + test suite result pages. To accumulate cover data over + all tests, the 'export' option had to be used in the + cover spec file. This was not well documented, and the + functionality was quite confusing.</p> + <p> + Using the 'nodes' option in the cover spec file would + fail when the test run consisted of multiple tests, since + the specified nodes would only be included in the cover + analysis of the first test.</p> + <p> + The repeated compilation and analysis of the same modules + was also very time consuming.</p> + <p> + To overcome these problems, ct will now only cover + compile and analyze modules once per test run, i.e. once + for each cover spec file. The log file is available via a + new button on the top level index page. The old "Coverage + log" links on the test suite result pages still exist, + but they all point to the same log containing the + accumulated result.</p> + <p> + Own Id: OTP-11971</p> + </item> + <item> + <p> + If multiple tests would run simultaneously on different + Erlang nodes, writing their logs to the same directory, + then there would often be entries in the all_runs.html + log file showing incomplete results (all zeroes) upon + completion. This problem was caused by a bug in the + Common Test log cache mechanism, which has been fixed.</p> + <p> + Own Id: OTP-11988 Aux Id: seq12611 </p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.8</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The error generated if a test case process received an + exit from a linked process while executing + init_per_testcase/2, was handled incorrectly by Common + Test. The problem has been solved, and Common Test now + reports this type of error correctly, with proper error + reason and exit location as well.</p> + <p> + Own Id: OTP-11643</p> + </item> + <item> + <p> + Running a parallel test case group with two or more + instances of the same test case would result in identical + log file names, and one test case instance would + overwrite the log file of another. This problem has been + solved.</p> + <p> + Own Id: OTP-11644</p> + </item> + <item> + <p> + Application upgrade (appup) files are corrected for the + following applications: </p> + <p> + <c>asn1, common_test, compiler, crypto, debugger, + dialyzer, edoc, eldap, erl_docgen, et, eunit, gs, hipe, + inets, observer, odbc, os_mon, otp_mibs, parsetools, + percept, public_key, reltool, runtime_tools, ssh, + syntax_tools, test_server, tools, typer, webtool, wx, + xmerl</c></p> + <p> + A new test utility for testing appup files is added to + test_server. This is now used by most applications in + OTP.</p> + <p> + (Thanks to Tobias Schlager)</p> + <p> + Own Id: OTP-11744</p> + </item> + <item> + <p> + The <c>cth_surefire</c> hook would crash in + <c>pre_init_per_suite/3</c> if a previous hook returned + <c>{skip,Reason}</c> or <c>{fail,Reason}</c> instead of a + <c>Config</c> list. This error has been corrected, and + <c>cth_surefire</c> will now simply propagate the + received <c>InitData</c> value instead.</p> + <p> + Own Id: OTP-11811</p> + </item> + <item> + <p> + Specs of return values are corrected for + <c>ct_netconfc:get/2,3</c>, + <c>ct_netconfc:get_config/3,4</c>, + <c>ct_netconfc:action/2,3</c>, + <c>ct_netconfc:send_rpc/2,3</c> and + <c>ct_netconfc:send/2,3</c>.</p> + <p> + Own Id: OTP-11834 Aux Id: seq12574 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + ct_telnet can now log all communication taking place + during a telnet session. Previously, only information + about ct_telnet operations and commands, as well as + explicitly requested data from the server, was logged.</p> + <p> + Furthermore, a logging mechanism based on an Error Logger + event handler and a dedicated Common Test hook, + <c>cth_conn_log</c>, now makes it possible to print data + for individual connections to separate log files. Please + see the <c>ct_telnet</c> reference manual for more + information and examples.</p> + <p> + Important note: A new argument, <c>ConnName</c> has been + added to the <c>unix_telnet:connect/5</c> callback + function. This forces users that use private ct_telnet + callback modules to update their code according to + <c>unix_telnet:connect/6</c>. Please see the + <c>unix_telnet</c> reference manual and source code + module for details.</p> + <p> + Own Id: OTP-11440 Aux Id: seq12457 </p> + </item> + <item> + <p> + A new timeout option has been introduced for the + <c>ct_telnet:expect/3</c> function. With + <c>{total_timeout,Time}</c> it's possible to set a time + limit for the complete expect operation. After + <c>Time</c> milliseconds, <c>expect/3</c> returns + <c>{error,timeout}</c>. The default value used if + <c>total_timeout</c> is not specified, is infinity (i.e. + no time limit). Please see the <c>ct_telnet</c> reference + manual for more information.</p> + <p> + Own Id: OTP-11689</p> + </item> + <item> + <p> + Some function specs are corrected or moved and some edoc + comments are corrected in order to allow use of edoc. + (Thanks to Pierre Fenoll)</p> + <p> + Own Id: OTP-11702</p> + </item> + <item> + <p> + Test case group name information has been added to the + data sent with <c>tc_user_skip</c> and + <c>tc_auto_skip</c> event messages, as well as the data + passed in calls to the CT Hook functions + <c>on_tc_skip/3</c> and <c>on_tc_fail/3</c>. The + modification only affects the function name + element/argument. This value remains an atom if the test + case in question does not belong to a test case group. + Otherwise a tuple <c>{FuncName,GroupName}</c> + (<c>{atom(),atom()}</c>) is passed instead.</p> + <p> + Note that this change may (depending on the patterns used + for matching) require modifications of user event + handlers and hook modules. Please see the Event Handling + chapter in the Common Test User's Guide, and the + reference manual for <c>ct_hooks</c>, for details.</p> + <p> + Note also that the Test Server framework callback + function <c>report/2</c> has been modified. This change + only affects users with test frameworks interfacing Test + Server rather than Common Test. See the + <c>test_server_ctrl</c> reference manual for details.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-11732 Aux Id: seq12541 </p> + </item> + <item> + <p> + If Common Test can't prompt the user to abort or continue + the test run when one or more test suites fail to + compile, a new option, + <c>{abort_if_missing_suites,Bool}</c>, can be used to + specify whether it should proceed with the test run, or + stop execution. The default value of <c>Bool</c> is + <c>false</c> (i.e. to proceed even if suites are + missing).</p> + <p> + Own Id: OTP-11769</p> + </item> + </list> + </section> + + + <section><title>Known Bugs and Problems</title> + <list> + <item> + <p> + common_test: Fix problems reported by Dialyzer.</p> + <p> + Own Id: OTP-11525</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.7.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Return values from group and testcase info functions are + now properly checked, and associated test cases are auto + skipped if a return value is invalid.</p> + <p> + Own Id: OTP-10631 Aux Id: kunagi-345 [256] </p> + </item> + <item> + <p>The way Common Test handles skipping of test cases has + been updated. In previous versions, returning + <c>{skip,Reason}</c> from a configuration function (such + as init_per_suite or init_per_group), resulted in all + affected test cases getting skipped with status + <c>auto_skipped</c>. This was inappropriate, since this + status is supposed to be used to inform that Common Test + has taken the initiative to skip something (e.g. a test + case group if init_per_group failed). Therefore, in this + version of Common Test, whenever the user skips a suite, + group, or individual test case (by means of a + configuration function or test specification term), the + affected test cases get the status <c>user_skipped</c> + instead.</p> <p>This update has meant a few changes that + may affect Common Test users in various ways: <list> + <item>The test results and statistics will be affected, + which is important to know when running regression tests + and comparing results to previous test runs.</item> + <item>Users that read or parse the textual log file + <c>suite.log</c> will notice that an auto skipped + function is now reported as <c>auto_skipped</c> rather + than <c>skipped</c> as before.</item> <item>When + <c>require</c> fails in an info function (such as suite/0 + or group/1), all affected configuration functions and + test cases are marked as <c>auto_skipped</c>.</item> + <item>If Common Test detects an error in the test suite + (such as e.g. an invalid all/0 function), all affected + configuration functions and test cases are marked as + <c>auto_skipped</c>.</item> <item>If a repeated test run + session reaches a deadline with <c>force_stop</c> + enabled, all remaining test cases are marked as + <c>auto_skipped</c> rather than <c>user_skipped</c> as + before.</item> <item>The event messages that Common Test + generates during test runs have been affected by this + update. For details see OTP-11524.</item> </list> </p> + <p> + Own Id: OTP-11305 Aux Id: OTP-11524 </p> + </item> + <item> + <p> + Returning {skip, Reason} from a pre_end_per_group/3 user + hook function would result in an exit in the Common Test + cth_log_redirect hook. This problem has been solved.</p> + <p> + Own Id: OTP-11409 Aux Id: seq12446 </p> + </item> + <item> + <p> + When the netconf server did not respond to the + close-session request, the call to + ct_netconfc:close_session/2 would hang forever waiting + for the netconf client to terminate. This has been + corrected. The client will now always terminate (and take + down the connection) if the close-session request times + out.</p> + <p> + Own Id: OTP-11478</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix cth_log_redirect.erl to fulfill gen_event behaviour. + Thanks to Roberto Aloi.</p> + <p> + Own Id: OTP-11401</p> + </item> + <item> + <p> + The first argument of the CT hook callback function + <c>on_tc_skip/3</c> has been modified. When this function + is called for <c>init_per_group</c> or + <c>end_per_group</c>, the value of the first argument is + now <c>{init_per_group,GroupName}</c> or + <c>{end_per_group,GroupName}</c>.</p> + <p> + Own Id: OTP-11523</p> + </item> + <item> + <p>The following modifications have been made to the + event messages that Common Test sends during test + execution: <list> <item>For the <c>tc_auto_skip</c> + event, the value of the <c>Func</c> element has changed + from <c>end_per_group</c> to + <c>{end_per_group,GroupName}</c>.</item> <item>When + <c>require</c> fails in an info function, such as suite/0 + or group/1, the init configuration function is now + reported as <c>auto_skipped</c> intead of <c>skipped</c>, + with the <c>tc_done</c> event.</item> <item>When + <c>require</c> fails in an info function because of a + configuration name already in use, the <c>tc_done</c> + event now reports the error with a tuple (of size 2) + tagged <c>failed</c> instead of <c>skipped</c>.</item> + </list> Please see the Event Handling chapter in the + Common Test User's Guide for reference. </p> + <p> + Own Id: OTP-11524 Aux Id: OTP-11305 </p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.7.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Documentation is added for ct_netconfc:send and + ct_netconfc:send_rpc.</p> + <p> + Own Id: OTP-11132</p> + </item> + <item> + <p> + ct_netconfc:create_subscription only allowed one XML + element inside the 'filter' element. According to RFC5277 + it should be allowed to add any number of elements inside + the filter, so this is now corrected.</p> + <p> + Own Id: OTP-11166</p> + </item> + <item> + <p> + The error handler installed by the Common Test hook + cth_log_redirect did not respond to init:stop/1/2. This + has been corrected.</p> + <p> + Own Id: OTP-11175 Aux Id: seq12356 </p> + </item> + <item> + <p> + Calling ct:pal/2 or ct:print/2 when Common Test was not + running, would cause an exit. This has been changed and + the string is now simply printed to stdout instead.</p> + <p> + Own Id: OTP-11176</p> + </item> + <item> + <p> + Fixed problem with the cth_log_redirect hook making calls + to an undefined function in ct_logs.</p> + <p> + Own Id: OTP-11238</p> + </item> + <item> + <p> + When running tests with the 'repeat' option, the Common + Test utility process did not always terminate quickly + enough after a test run, causing the start of the next + run to fail. A monitor is now used to ensure termination + of the utility process after each test run.</p> + <p> + Own Id: OTP-11244 Aux Id: seq12396 </p> + </item> + <item> + <p> + Test Server installed an error handler (test_server_h) + only to be able to write the name of the current test + case to stdout whenever it received an error- or progress + report. This functionality was not useful and has been + removed. The built-in Common Test hook, cth_log_redirect, + has instead been improved to now also tag all error- and + progress reports in the log with suite-, group-, and/or + test case name.</p> + <p> + Own Id: OTP-11263 Aux Id: seq12251 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A new log, the "Pre- and Post Test I/O Log", has been + introduced, which makes it possible to capture error- and + progress reports, as well as printouts made with ct:log/2 + and ct:pal/2, before and after a test run. (Some minor + improvements of the logging system have been made at the + same time). Links to the new log are found on the Common + Test Framework Log page. The Common Test User's Guide has + been updated with information about the new log and also + with a new section on how to synchronize external + applications with Common Test by means of the CT Hook + init and terminate functions.</p> + <p> + Own Id: OTP-11272</p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.7.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/doc/src/notes_history.xml b/lib/common_test/doc/src/notes_history.xml index e4085d5431..34ac4fd15b 100644 --- a/lib/common_test/doc/src/notes_history.xml +++ b/lib/common_test/doc/src/notes_history.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2004</year><year>2009</year> + <year>2004</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/part.xml b/lib/common_test/doc/src/part.xml index a74185221d..fe73918975 100644 --- a/lib/common_test/doc/src/part.xml +++ b/lib/common_test/doc/src/part.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE part SYSTEM "part.dtd"> <part xmlns:xi="http://www.w3.org/2001/XInclude"> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/part_notes.xml b/lib/common_test/doc/src/part_notes.xml index 506677a7a5..13ca499196 100644 --- a/lib/common_test/doc/src/part_notes.xml +++ b/lib/common_test/doc/src/part_notes.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE part SYSTEM "part.dtd"> <part xmlns:xi="http://www.w3.org/2001/XInclude"> <header> <copyright> - <year>2004</year><year>2009</year> + <year>2004</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/part_notes_history.xml b/lib/common_test/doc/src/part_notes_history.xml index e658810e7a..f995a70b7f 100644 --- a/lib/common_test/doc/src/part_notes_history.xml +++ b/lib/common_test/doc/src/part_notes_history.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE part SYSTEM "part.dtd"> <part> <header> <copyright> - <year>2004</year><year>2009</year> + <year>2004</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/ref_man.xml b/lib/common_test/doc/src/ref_man.xml index 6fede88434..c266b70d00 100644 --- a/lib/common_test/doc/src/ref_man.xml +++ b/lib/common_test/doc/src/ref_man.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE application SYSTEM "application.dtd"> <application xmlns:xi="http://www.w3.org/2001/XInclude"> <header> <copyright> - <year>2003</year><year>2012</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -78,6 +78,7 @@ <xi:include href="unix_telnet.xml"/> <xi:include href="ct_slave.xml"/> <xi:include href="ct_hooks.xml"/> + <xi:include href="ct_property_test.xml"/> </application> diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index afaed29626..864f82cb63 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> @@ -21,7 +21,7 @@ </legalnotice> - <title>Running Tests</title> + <title>Running Tests and Analyzing Results</title> <prepared>Peter Andersson, Kenneth Lundin</prepared> <docno></docno> <date></date> @@ -59,7 +59,15 @@ <p>If compilation should fail for one or more suites, the compilation errors are printed to tty and the operator is asked if the test run should proceed without the missing suites, or be aborted. If the operator chooses to proceed, - it is noted in the HTML log which tests have missing suites.</p> + it is noted in the HTML log which tests have missing suites. If Common Test is + unable to prompt the user after compilation failure (if Common Test doesn't + control stdin), the test run will proceed automatically without the missing + suites. This behaviour can however be modified with the + <c><![CDATA[ct_run]]></c> flag <c><![CDATA[-abort_if_missing_suites]]></c>, + or the <c><![CDATA[ct:run_test/1]]></c> option + <c><![CDATA[{abort_if_missing_suites,TrueOrFalse}]]></c>. If + <c><![CDATA[abort_if_missing_suites]]></c> is set (to true), the test run + will stop immediately if some suites fail to compile.</p> <p>Any help module (i.e. regular Erlang module with name not ending with "_SUITE") that resides in the same test object directory as a suite @@ -105,8 +113,8 @@ RPC from a remote node.</p> </section> - <marker id="ct_run"></marker> <section> + <marker id="ct_run"></marker> <title>Running tests from the OS command line</title> <p>The <c>ct_run</c> program can be used for running tests from @@ -167,6 +175,7 @@ <seealso marker="ct_hooks_chapter#builtin_cths">Built-in Common Test Hooks</seealso>. Default is <c>true</c>.</item> <item><c><![CDATA[-include]]></c>, specifies include directories (see above).</item> <item><c><![CDATA[-no_auto_compile]]></c>, disables the automatic test suite compilation feature (see above).</item> + <item><c><![CDATA[-abort_if_missing_suites]]></c>, aborts the test run if one or more suites fail to compile (see above).</item> <item><c><![CDATA[-multiply_timetraps <n>]]></c>, extends <seealso marker="write_test_chapter#timetraps">timetrap timeout</seealso> values.</item> <item><c><![CDATA[-scale_timetraps <bool>]]></c>, enables automatic <seealso marker="write_test_chapter#timetraps">timetrap @@ -205,17 +214,17 @@ <p>The <c>ct_run</c> program sets the exit status before shutting down. The following values are defined:</p> <list> - <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto-skipped test cases.</item> - <item><c>1</c> indicates that one or more test cases have failed, or have been auto-skipped.</item> + <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto skipped test cases.</item> + <item><c>1</c> indicates that one or more test cases have failed, or have been auto skipped.</item> <item><c>2</c> indicates that the test execution has failed because of e.g. compilation errors, an illegal return value from an info function, etc.</item> </list> - <p>If auto-skipped test cases should not affect the exit status, you may change the default + <p>If auto skipped test cases should not affect the exit status, you may change the default behaviour using start flag:</p> <pre>-exit_status ignore_config</pre> <p>For more information about the <c>ct_run</c> program, see the - <seealso marker="ct_run#top">Reference Manual</seealso> and the + <seealso marker="ct_run">Reference Manual</seealso> and the <seealso marker="install_chapter#general">Installation</seealso> chapter. </p> </section> @@ -225,15 +234,15 @@ <p>Common Test provides an Erlang API for running tests. The main (and most flexible) function for specifying and executing tests is called - <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>. + <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>. This function takes the same start parameters as - the <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + the <seealso marker="run_test_chapter#ct_run"><c>ct_run</c></seealso> program described above, only the flags are instead given as options in a list of key-value tuples. E.g. a test specified with <c>ct_run</c> like:</p> <p><c>$ ct_run -suite ./my_SUITE -logdir ./results</c></p> - <p>is with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> specified as:</p> + <p>is with <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso> specified as:</p> <p><c>1> ct:run_test([{suite,"./my_SUITE"},{logdir,"./results"}]).</c></p> <p>The function returns the test result, represented by the tuple: @@ -245,7 +254,7 @@ <section> <title>Releasing the Erlang shell</title> <p>During execution of tests, started with - <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, the Erlang shell process, controlling stdin, will remain the top level process of the Common Test system of processes. The result is that the Erlang shell is not available for interaction during @@ -260,19 +269,19 @@ <c>ct:run_test/1</c> returns the pid of this process rather than the test result - which instead is printed to tty at the end of the test run.</p> <note><p>Note that in order to use the - <c><seealso marker="ct#break-1">ct:break/1/2</seealso></c> and - <c><seealso marker="ct#continue-0">ct:continue/0/1</seealso></c> functions, + <seealso marker="ct#break-1"><c>ct:break/1/2</c></seealso> and + <seealso marker="ct#continue-0"><c>ct:continue/0/1</c></seealso> functions, <c>release_shell</c> <em>must</em> be set to <c>true</c>.</p></note> </section> <p>For detailed documentation about - <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, please see the - <c><seealso marker="ct#run_test-1">ct</seealso></c> manual page.</p> + <seealso marker="ct#run_test-1"><c>ct</c></seealso> manual page.</p> </section> - <marker id="group_execution"></marker> <section> + <marker id="group_execution"></marker> <title>Test case group execution</title> <p>With the <c>ct_run</c> flag, or <c>ct:run_test/1</c> option <c>group</c>, @@ -442,9 +451,9 @@ for trying out various operations during test suite development.</p> <p>To invoke the interactive shell mode, you can start an Erlang shell - manually and call <c><seealso marker="ct#install-1">ct:install/1</seealso></c> to install any configuration + manually and call <seealso marker="ct#install-1"><c>ct:install/1</c></seealso> to install any configuration data you might need (use <c>[]</c> as argument otherwise), then - call <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> to start Common Test. If you use + call <seealso marker="ct#start_interactive-0"><c>ct:start_interactive/0</c></seealso> to start Common Test. If you use the <c>ct_run</c> program, you may start the Erlang shell and Common Test in the same go by using the <c>-shell</c> and, optionally, the <c>-config</c> and/or <c>-userconfig</c> flag. Examples: @@ -463,8 +472,8 @@ <p>If any functions using "required config data" (e.g. ct_telnet or ct_ftp functions) are to be called from the erlang shell, config - data must first be required with <c><seealso marker="ct#require-1"> - ct:require/1/2</seealso></c>. This is + data must first be required with <seealso marker="ct#require-1"><c> + ct:require/1/2</c></seealso>. This is equivalent to a <c>require</c> statement in the <seealso marker="write_test_chapter#suite">Test Suite Info Function</seealso> or in the <seealso @@ -491,11 +500,11 @@ is not supported.</p> <p>If you wish to exit the interactive mode (e.g. to start an - automated test run with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), call the function - <c><seealso marker="ct#stop_interactive-0">ct:stop_interactive/0</seealso></c>. This shuts down the + automated test run with <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>), call the function + <seealso marker="ct#stop_interactive-0"><c>ct:stop_interactive/0</c></seealso>. This shuts down the running <c>ct</c> application. Associations between configuration names and data created with <c>require</c> are - consequently deleted. <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> will get you + consequently deleted. <seealso marker="ct#start_interactive-0"><c>ct:start_interactive/0</c></seealso> will get you back into interactive mode, but the previous state is not restored.</p> </section> @@ -503,7 +512,7 @@ <title>Step by step execution of test cases with the Erlang Debugger</title> <p>By means of <c>ct_run -step [opts]</c>, or by passing the - <c>{step,Opts}</c> option to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, it is possible + <c>{step,Opts}</c> option to <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>, it is possible to get the Erlang Debugger started automatically and use its graphical interface to investigate the state of the current test case and to execute it step by step and/or set execution breakpoints.</p> @@ -527,17 +536,17 @@ with <c>dir</c>.</p> </section> - <marker id="test_specifications"></marker> <section> + <marker id="test_specifications"></marker> <title>Test Specifications</title> <section> <title>General description</title> <p>The most flexible way to specify what to test, is to use a so called test specification. A test specification is a sequence of Erlang terms. The terms are normally declared in one or more text files - (see <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), but + (see <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso>), but may also be passed to Common Test on the form of a list (see - <c><seealso marker="ct#run_testspec-1">ct:run_testspec/1</seealso></c>). + <seealso marker="ct#run_testspec-1"><c>ct:run_testspec/1</c></seealso>). There are two general types of terms: configuration terms and test specification terms.</p> <p>With configuration terms it is possible to e.g. label the test @@ -589,8 +598,8 @@ Common Test will either execute one test run per specification file, or join the files and perform all tests within one single test run. The first behaviour is the default one. The latter requires that the start - flag/option <c>join_suites</c> is provided, e.g. - <c>run_test -spec ./my_tests1.ts ./my_tests2.ts -join_suites</c>.</p> + flag/option <c>join_specs</c> is provided, e.g. + <c>run_test -spec ./my_tests1.ts ./my_tests2.ts -join_specs</c>.</p> <p>Joining a number of specifications, or running them separately, can also be accomplished with (and may be combined with) test specification @@ -744,6 +753,9 @@ {auto_compile, Bool}, {auto_compile, NodeRefs, Bool}, + {abort_if_missing_suites, Bool}, + {abort_if_missing_suites, NodeRefs, Bool}, + {config, ConfigFiles}. {config, ConfigDir, ConfigBaseNames}. {config, NodeRefs, ConfigFiles}. @@ -989,7 +1001,7 @@ <c>ct_run</c>. This forces Common Test to ignore unrecognizable terms. Note that in this mode, Common Test is not able to check the specification for errors as efficiently as if the scanner runs in default mode. - If <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> is used + If <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso> is used for starting the tests, the relaxed scanner mode is enabled by means of the tuple: <c>{allow_user_terms,true}</c></p> </section> @@ -999,7 +1011,7 @@ <title>Running tests from the Web based GUI</title> <p>The web based GUI, VTS, is started with the - <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + <seealso marker="run_test_chapter#ct_run"><c>ct_run</c></seealso> program. From the GUI you can load config files, and select directories, suites and cases to run. You can also state the config files, directories, suites and cases on the command line @@ -1087,18 +1099,18 @@ </list> <p>On the test run index page there is a link to the Common Test - Framework log file in which information about imported - configuration data and general test progress is written. This - log file is useful to get snapshot information about the test - run during execution. It can also be very helpful when - analyzing test results or debugging test suites.</p> + Framework Log file in which information about imported + configuration data and general test progress is written. This + log file is useful to get snapshot information about the test + run during execution. It can also be very helpful when + analyzing test results or debugging test suites.</p> <p>On the test run index page it is noted if a test has missing suites (i.e. suites that Common Test has failed to compile). Names of the missing suites can be found in the - Common Test Framework log file.</p> + Common Test Framework Log file.</p> - <p>The major logfile shows a detailed report of the test run. It + <p>The major log file shows a detailed report of the test run. It includes test suite and test case names, execution time, the exact reason for failures etc. The information is available in both a file with textual and with HTML representation. The HTML file shows a @@ -1172,6 +1184,40 @@ <url href="http://tablesorter.com">tablesorter</url> plugin, with customized sorting functions, for this implementation.</p> </section> + + <section> + <title>The Unexpected I/O Log</title> + <p>On the test suites overview page you find a link to the Unexpected I/O Log. + In this log, Common Test saves printouts made with + <c>ct:log/2</c> and <c>ct:pal/2</c>, as well as captured system error- and + progress reports, that cannot be associated with particular test cases and + therefore cannot be written to individual test case log files. This happens e.g. + if a log printout is made from an external process (not a test case process), + or if an error- or progress report comes in, during a short interval while Common + Test is not executing a test case or configuration function, <em>or</em> while + Common Test is currently executing a parallell test case group.</p> + </section> + + <section> + <marker id="pre_post_test_io_log"></marker> + <title>The Pre- and Post Test I/O Log</title> + <p>On the Common Test Framework Log page you find links to the so called + Pre- and Post Test I/O Log. In this log, Common Test saves printouts made with + <c>ct:log/2</c> and <c>ct:pal/2</c>, as well as captured system error- + and progress reports, that take place before - and after - the actual test run. + Examples of this are printouts from a CT hook init- or terminate function, or + progress reports generated when an OTP application is started from a CT hook + init function. Another example is an error report generated due to + a failure when an external application is stopped from a CT hook terminate function. + All information in these examples ends up in the Pre- and Post Test I/O Log. + For more information on how to synchronize test runs with external user + applications, please see the + <seealso marker="ct_hooks_chapter#synchronizing">Synchronizing</seealso> + section in the Common Test Hooks chapter.</p> + <p>Note that logging to file with <c>ct:log/2</c> or <c>ct:pal/2</c> + only works when Common Test is running. Printouts with <c>ct:pal/2</c> + are however always displayed on screen.</p> + </section> </section> <section> diff --git a/lib/common_test/doc/src/test_structure_chapter.xml b/lib/common_test/doc/src/test_structure_chapter.xml index b9ca59135d..3c9e159436 100644 --- a/lib/common_test/doc/src/test_structure_chapter.xml +++ b/lib/common_test/doc/src/test_structure_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2006</year><year>2010</year> + <year>2006</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/why_test_chapter.xml b/lib/common_test/doc/src/why_test_chapter.xml index 95ff614f87..db94c6a6cb 100644 --- a/lib/common_test/doc/src/why_test_chapter.xml +++ b/lib/common_test/doc/src/why_test_chapter.xml @@ -1,10 +1,10 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> - <year>2003</year><year>2009</year> + <year>2003</year><year>2013</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index cc8d913994..055fb90641 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="latin1" ?> +<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> @@ -129,8 +129,8 @@ </p> </section> - <marker id="per_testcase"/> <section> + <marker id="per_testcase"/> <title>Init and end per test case</title> <p>Each test suite module can contain the optional configuration functions @@ -173,7 +173,7 @@ </p> <p>The <c>end_per_testcase/2</c> function is called even after a - test case terminates due to a call to <c><seealso marker="ct#abort_current_testcase-1">ct:abort_current_testcase/1</seealso></c>, + test case terminates due to a call to <seealso marker="ct#abort_current_testcase-1"><c>ct:abort_current_testcase/1</c></seealso>, or after a timetrap timeout. However, <c>end_per_testcase</c> will then execute on a different process than the test case function, and in this situation, <c>end_per_testcase</c> will @@ -243,8 +243,8 @@ <note><p>The test case function argument <c>Config</c> should not be confused with the information that can be retrieved from - configuration files (using <c><seealso marker="ct#get_config-1"> - ct:get_config/1/2</seealso></c>). The Config argument + configuration files (using <seealso marker="ct#get_config-1"><c> + ct:get_config/1/2</c></seealso>). The Config argument should be used for runtime configuration of the test suite and the test cases, while configuration files should typically contain data related to the SUT. These two types of configuration data are handled @@ -303,7 +303,7 @@ <item> <p> Use this to specify arbitrary data related to the testcase. This - data can be retrieved at any time using the <c><seealso marker="ct#userdata-3">ct:userdata/3</seealso></c> + data can be retrieved at any time using the <seealso marker="ct#userdata-3"><c>ct:userdata/3</c></seealso> utility function. </p> </item> @@ -348,8 +348,8 @@ </taglist> <p>See the <seealso marker="config_file_chapter#require_config_data">Config files</seealso> - chapter and the <c><seealso marker="ct#require-1"> - ct:require/1/2</seealso></c> function in the + chapter and the <seealso marker="ct#require-1"><c> + ct:require/1/2</c></seealso> function in the <seealso marker="ct">ct</seealso> reference manual for more information about <c>require</c>.</p> @@ -826,14 +826,16 @@ Common Test to create one dedicated private directory per test case and execution instead. This is accomplished by means of the flag/option: <c>create_priv_dir</c> (to be used with the - <c>ct_run</c> program, the <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> function, or + <c>ct_run</c> program, the <seealso marker="ct#run_test-1"><c>ct:run_test/1</c></seealso> function, or as test specification term). There are three possible values for this option: + </p> <list> <item><c>auto_per_run</c></item> <item><c>auto_per_tc</c></item> <item><c>manual_per_tc</c></item> </list> + <p> The first value indicates the default priv_dir behaviour, i.e. one private directory created per test run. The two latter values tell Common Test to generate a unique test directory name @@ -842,7 +844,7 @@ become very inefficient for test runs with many test cases and/or repetitions. Therefore, in case the manual version is instead used, the test case must tell Common Test to create priv_dir when it needs it. - It does this by calling the function <c><seealso marker="ct#make_priv_dir-0">ct:make_priv_dir/0</seealso></c>. + It does this by calling the function <seealso marker="ct#make_priv_dir-0"><c>ct:make_priv_dir/0</c></seealso>. </p> <note><p>You should not depend on current working directory for @@ -890,7 +892,7 @@ <p>It is also possible to dynamically set/reset a timetrap during the excution of a test case, or configuration function. This is done by calling - <c><seealso marker="ct#timetrap-1">ct:timetrap/1</seealso></c>. This function cancels the current timetrap + <seealso marker="ct#timetrap-1"><c>ct:timetrap/1</c></seealso>. This function cancels the current timetrap and starts a new one (that stays active until timeout, or end of the current function).</p> @@ -903,12 +905,12 @@ <p>If a test case needs to suspend itself for a time that also gets multipled by <c>multiply_timetraps</c> (and possibly also scaled up if - <c>scale_timetraps</c> is enabled), the function <c><seealso marker="ct#sleep-1">ct:sleep/1</seealso></c> + <c>scale_timetraps</c> is enabled), the function <seealso marker="ct#sleep-1"><c>ct:sleep/1</c></seealso> may be used (instead of e.g. <c>timer:sleep/1</c>).</p> <p>A function (<c>fun/0</c> or <c>MFA</c>) may be specified as timetrap value in the suite-, group- and test case info function, as - well as argument to the <c><seealso marker="ct#timetrap-1">ct:timetrap/1</seealso></c> function. Examples:</p> + well as argument to the <seealso marker="ct#timetrap-1"><c>ct:timetrap/1</c></seealso> function. Examples:</p> <p><c>{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}</c></p> <p><c>ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)</c></p> |