diff options
Diffstat (limited to 'lib/common_test/doc/src')
| -rw-r--r-- | lib/common_test/doc/src/Makefile | 22 | ||||
| -rw-r--r-- | lib/common_test/doc/src/common_test_app.xml | 28 | ||||
| -rw-r--r-- | lib/common_test/doc/src/notes.xml | 302 | ||||
| -rw-r--r-- | lib/common_test/doc/src/write_test_chapter.xml | 81 |
4 files changed, 387 insertions, 46 deletions
diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index d9651f13b0..2ec6952710 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -158,18 +158,18 @@ include $(ERL_TOP)/make/otp_release_targets.mk release_docs_spec: docs - $(INSTALL_DIR) $(RELSYSDIR)/doc/pdf - $(INSTALL_DATA) $(TOP_PDF_FILE) $(RELSYSDIR)/doc/pdf - $(INSTALL_DIR) $(RELSYSDIR)/doc/html + $(INSTALL_DIR) "$(RELSYSDIR)/doc/pdf" + $(INSTALL_DATA) $(TOP_PDF_FILE) "$(RELSYSDIR)/doc/pdf" + $(INSTALL_DIR) "$(RELSYSDIR)/doc/html" $(INSTALL_DATA) $(HTMLDIR)/* \ - $(RELSYSDIR)/doc/html - $(INSTALL_DATA) $(INFO_FILE) $(RELSYSDIR) - $(INSTALL_DIR) $(RELEASE_PATH)/man/man1 - $(INSTALL_DATA) $(MAN1DIR)/* $(RELEASE_PATH)/man/man1 - $(INSTALL_DIR) $(RELEASE_PATH)/man/man3 - $(INSTALL_DATA) $(MAN3DIR)/* $(RELEASE_PATH)/man/man3 - $(INSTALL_DIR) $(RELEASE_PATH)/man/man6 - $(INSTALL_DATA) $(MAN6DIR)/* $(RELEASE_PATH)/man/man6 + "$(RELSYSDIR)/doc/html" + $(INSTALL_DATA) $(INFO_FILE) "$(RELSYSDIR)" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man1" + $(INSTALL_DATA) $(MAN1DIR)/* "$(RELEASE_PATH)/man/man1" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man3" + $(INSTALL_DATA) $(MAN3DIR)/* "$(RELEASE_PATH)/man/man3" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man6" + $(INSTALL_DATA) $(MAN6DIR)/* "$(RELEASE_PATH)/man/man6" release_spec: diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index c7f6c7ce5c..6babdb93af 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -187,9 +187,13 @@ test case is allowed to execute (including <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c>). If the timetrap time is exceeded, the test case fails with reason - <c>timetrap_timeout</c>. If a <c>TimeFunc</c> function is specified, - it will be called initially and must return a value on - <c>TimeVal</c> format.</p> + <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details). + </p> <p>The <c>require</c> tag specifies configuration variables that are required by test cases (and/or configuration functions) @@ -313,9 +317,12 @@ test case is allowed to execute (including <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c>). If the timetrap time is exceeded, the test case fails with reason - <c>timetrap_timeout</c>. If a <c>TimeFunc</c> function is specified, - it will be called initially and must return a value on - <c>TimeVal</c> format.</p> + <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details).</p> <p>The <c>require</c> tag specifies configuration variables that are required by test cases (and/or configuration functions) @@ -494,9 +501,12 @@ exceeded, the test case fails with reason <c>timetrap_timeout</c>. <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> are included in the - timetrap time. If a <c>TimeFunc</c> function is specified, - it will be called before the test case (or <c>init_per_testcase/2</c>) - and must return a value on <c>TimeVal</c> format.</p> + timetrap time. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details).</p> <p>The <c>require</c> tag specifies configuration variables that are required by the test case (and/or <c>init/end_per_testcase/2</c>). diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index da0b6b2d65..050b8774ca 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -32,6 +32,308 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.6.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Common Test adds the test suite directories to the code + path before executing the tests. These directories should + also be removed from the code path at the end of the test + run, which, prior to this fix, was not performed.</p> + <p> + Own Id: OTP-9595</p> + </item> + <item> + <p> + An entry is now created in the index.html file (i.e. the + overview file for the test run) for each repeated test + during a test run. This was previously not the case. Note + that in the top level (logdir) index file, however, only + the last test result is listed. For example, given the + test spec: + [{merge_tests,false},{dirs,"test1"},{dirs,"test1"}]. In + the index file for the test run (under + Logdir/ct_run.Node.Date.Time), both tests are listed. In + the top level index file (under Logdir), only the last + test is listed (one has to find the previous results + through the all_runs.html file).</p> + <p> + Own Id: OTP-9634 Aux Id: seq11924 </p> + </item> + <item> + <p> + After a test case timeout or abortion, the + end_per_testcase function executes on a new dedicated + process. The group leader for this process should be set + to the IO server for the test case, which was not done + properly. The result of this error was that no warnings + about end_per_testcase failing or timing out were ever + printed in the test case log. Also, help functions such + as e.g. test_server:stop_node/1, attempting to + synchronize with the IO server, would hang. The fault has + been corrected.</p> + <p> + Own Id: OTP-9666</p> + </item> + <item> + <p> + The ct:get_status/0 function would cause the calling + process to receive 'DOWN' messages if no tests were + running at the time of the call. This bug has been fixed.</p> + <p> + Own Id: OTP-9830 Aux Id: seq11975 </p> + </item> + <item> + <p> + A deadlock situation could occur if Common Test is + forwarding error_handler printouts to Test Server at the + same time a new test case is starting. This error has + been fixed.</p> + <p> + Own Id: OTP-9894</p> + </item> + <item> + <p> + A link to the ct_run program is now created, as expected, + in the installation bin directory (default + /usr/local/bin) during 'make install'.</p> + <p> + Own Id: OTP-9898</p> + </item> + <item> + <p> + Using the repeat, duration or until option with + ct:run_test/1, would cause an infinite loop. This has + been fixed.</p> + <p> + Own Id: OTP-9899</p> + </item> + <item> + <p> + Two or more test cases executing in parallel and printing + to screen at the same time with ct:pal/2/3 or + ct:print/2/3 could write into each other's "slots" and + create a mess of mixed strings. In order to avoid this, + only a single IO message is now ever sent per printout + call.</p> + <p> + Own Id: OTP-9900 Aux Id: OTP-9904 </p> + </item> + <item> + <p> + When a test case was killed because of a timetrap + timeout, the current location (suite, case and line) was + not printed correctly in the log files. This has been + corrected.</p> + <p> + Own Id: OTP-9930 Aux Id: seq12002 </p> + </item> + <item> + <p> + The wrong exit location was printed in the log file when + ct:fail/1 or ct_fail/2 was called.</p> + <p> + Own Id: OTP-9933 Aux Id: seq12002 </p> + </item> + <item> + <p> + Test Server and Common Test would add new error handlers + with each test run and fail to remove previously added + ones. In the case of Test Server, this would only happen + if SASL was not running on the test node. This has been + fixed.</p> + <p> + Own Id: OTP-9941 Aux Id: seq12009 </p> + </item> + <item> + <p> + If a test case process was terminated due to an exit + signal from a linked process, Test Server failed to + report the correct name of the suite and case to the + framework. This has been corrected.</p> + <p> + Own Id: OTP-9958 Aux Id: OTP-9855 </p> + </item> + <item> + <p> + When starting a test with ct_run and adding a directory + to the code path using -pa or -pz (preceding -erl_args), + Common Test would delete any existing directory in the + code path with the same base name (see + filename:basename/1) as the directory being added. This + has been fixed.</p> + <p> + Own Id: OTP-9964</p> + </item> + <item> + <p> + If passing two or more directories with the same base + name (see filename:basename/1) to Common Test with ct_run + -pa, only one of the directories would actually be added.</p> + <p> + Own Id: OTP-9975 Aux Id: seq12019 </p> + </item> + <item> + <p> + Configuration data required by the group info function + was deleted before the call to post_end_per_group, which + made it impossible for the hook function to read and use + the data in question. This has been fixed.</p> + <p> + Own Id: OTP-9989</p> + </item> + <item> + <p> + Disabling built-in hooks in a test specification was + ignored, this has now been fixed.</p> + <p> + Own Id: OTP-10009</p> + </item> + <item> + <p> + Various typographical errors corrected in documentation + for common_test, driver, erl_driver and windows + installation instructions. (Thanks to Tuncer Ayaz)</p> + <p> + Own Id: OTP-10037</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A new optional feature has been introduced that enables + Common Test to generate priv_dir directory names that are + unique for each test case or config function. The name of + the option/flag is 'create_priv_dir' and it can be set to + value 'auto_per_run' (which is the default, existing, + behaviour), or 'auto_per_tc' or 'manual_per_tc'. If + 'auto_per_tc' is used, Test Server creates a dedicated + priv_dir automatically for each test case (which can be + very expensive in case of many and/or repeated cases). If + 'manual_per_tc' is used, the user needs to create the + priv_dir explicitly by calling the new function + ct:make_priv_dir/0.</p> + <p> + Own Id: OTP-9659 Aux Id: seq11930 </p> + </item> + <item> + <p> + A column for test case group name has been added to the + suite overview HTML log file.</p> + <p> + Own Id: OTP-9730 Aux Id: seq11952 </p> + </item> + <item> + <p> + It is now possible to use the post_end_per_testcase CT + hook function to print a comment for a test case in the + overview log file, even if the test case gets killed by a + timetrap or unknown exit signal, or if the + end_per_testcase function times out.</p> + <p> + Own Id: OTP-9855 Aux Id: seq11979 </p> + </item> + <item> + <p> + The pre- and post CT hook functions are now always called + for all configuration functions, even for configuration + functions that are not implemented in the test suite.</p> + <p> + Own Id: OTP-9880 Aux Id: seq11993 </p> + </item> + <item> + <p> + Common Test will now print error information (with a time + stamp) in the test case log file immediately when a test + case fails. This makes it easier to see when, in time, + the fault actually occured, and aid the job of locating + relevant trace and debug printouts in the log.</p> + <p> + Own Id: OTP-9904 Aux Id: seq11985, OTP-9900 </p> + </item> + <item> + <p> + Test Server has been modified to check the SASL + errlog_type parameter when receiving an error logger + event, so that it doesn't print reports of type that the + user has disabled.</p> + <p> + Own Id: OTP-9955 Aux Id: seq12013 </p> + </item> + <item> + <p> + The test specification term 'skip_groups' was implemented + in Common Test v1.6. It was never documented however, + which has now been attended to. Please see the Test + Specifications chapter in the User's Guide for + information.</p> + <p> + Own Id: OTP-9972</p> + </item> + <item> + <p> + The Common Test Master has been updated to use a CSS + style sheet for the html log files.</p> + <p> + Own Id: OTP-9973</p> + </item> + <item> + <p> + If the init_per_group/2 and end_per_group/2 functions are + not implemented in the test suite, Common Test calls it's + own local init- and end functions - previously named + ct_init_per_group/2 and ct_end_per_group/2 - when a group + is executed. These functions have been renamed + init_per_group/2 and end_per_group/2 respectively. Note + that this may affect any user event handler identifying + events by the old names.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-9986 Aux Id: OTP-9992 </p> + </item> + <item> + <p> + By specifying a user defined function ({M,F,A} or fun) as + timetrap value, either by means of an info function or by + calling ct:timetrap/1, it is now possible to set a + timetrap that will be triggered when the user function + returns.</p> + <p> + Own Id: OTP-9988 Aux Id: OTP-9501, seq11894 </p> + </item> + <item> + <p> + If the optional configuration functions init_per_suite/1 + and end_per_suite/1 are not implemented in the test + suite, local Common Test versions of these functions are + called instead, and will be displayed in the overview log + file. Any printouts made by the pre- or + post_init_per_suite and pre- or post_end_per_suite hook + functions are saved in the log files for these functions.</p> + <p> + Own Id: OTP-9992</p> + </item> + <item> + <p> + A hook has been added to common test which outputs + surefire XML for usage together with CI tools such as + Jenkins. To enable the hook pass '-ct_hooks cth_surefire' + to ct_run. See the CTH documentation for more details.</p> + <p> + Own Id: OTP-9995</p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.6</title> <section><title>Improvements and New Features</title> diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index eec7628b6f..7b7e7af8ea 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -872,34 +872,63 @@ <marker id="timetraps"></marker> <title>Timetrap timeouts</title> <p>The default time limit for a test case is 30 minutes, unless a - <c>timetrap</c> is specified either by the suite info function - or a test case info function. The timetrap timeout value defined - in <c>suite/0</c> is the value that will be used for each test case - in the suite (as well as for the configuration functions - <c>init_per_suite/1</c> and <c>end_per_suite</c>). A timetrap timeout - value set with the test case info function will override the value set - by <c>suite/0</c>, but only for that particular test case.</p> - <p>It is also possible to set/reset a timetrap during test case (or - configuration function) execution. This is done by calling - <c>ct:timetrap/1</c>. This function will cancel the current timetrap - and start a new one.</p> + <c>timetrap</c> is specified either by the suite-, group-, + or test case info function. The timetrap timeout value defined by + <c>suite/0</c> is the value that will be used for each test case + in the suite (as well as for the configuration functions + <c>init_per_suite/1</c>, <c>end_per_suite/1</c>, <c>init_per_group/2</c>, + and <c>end_per_group/2</c>). A timetrap value defined by + <c>group(GroupName)</c> overrides one defined by <c>suite()</c> + and will be used for each test case in group <c>GroupName</c>, and any + of its sub-groups. If a timetrap value is defined by <c>group/1</c> + for a sub-group, it overrides that of its higher level groups. Timetrap + values set by individual test cases (by means of the test case info + function) overrides both group- and suite- level timetraps.</p> + + <p>It is also possible to dynamically set/reset a timetrap during the + excution of a test case, or configuration function. This is done by calling + <c>ct:timetrap/1</c>. This function cancels the current timetrap + and starts a new one (that stays active until timeout, or end of the + current function).</p> + <p>Timetrap values can be extended with a multiplier value specified at - startup with the <c>multiply_timetraps</c> option. It is also possible - to let Test Server decide to scale up timetrap timeout values - automatically, e.g. if tools such as cover or trace are running during - the test. This feature is disabled by default and can be enabled with - the <c>scale_timetraps</c> start option.</p> + startup with the <c>multiply_timetraps</c> option. It is also possible + to let the test server decide to scale up timetrap timeout values + automatically, e.g. if tools such as cover or trace are running during + the test. This feature is disabled by default and can be enabled with + the <c>scale_timetraps</c> start option.</p> + <p>If a test case needs to suspend itself for a time that also gets - multipled by <c>multiply_timetraps</c>, and possibly scaled up if - <c>scale_timetraps</c> is enabled, the function <c>ct:sleep/1</c> - may be called.</p> - <p>A function (<c>fun</c> or <c>MFA</c>) may be specified as timetrap value - in the suite- and test case info function, e.g:</p> - <p><c>{timetrap,{test_utils,get_timetrap_value,[?MODULE,system_start]}}</c></p> - <p>The function will be called initially by Common Test (before execution - of the suite or the test case) and must return a time value such as an - integer (millisec), or a <c>{SecMinOrHourTag,Time}</c> tuple. More - information can be found in the <c>common_test</c> reference manual.</p> + multipled by <c>multiply_timetraps</c> (and possibly also scaled up if + <c>scale_timetraps</c> is enabled), the function <c>ct:sleep/1</c> + may be used (instead of e.g. <c>timer:sleep/1</c>).</p> + + <p>A function (<c>fun/0</c> or <c>MFA</c>) may be specified as + timetrap value in the suite-, group- and test case info function, as + well as argument to the <c>ct:timetrap/1</c> function. Examples:</p> + + <p><c>{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}</c></p> + <p><c>ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)</c></p> + + <p>The user timetrap function may be used for two things:</p> + <list> + <item>To act as a timetrap - the timeout is triggered when the + function returns.</item> + <item>To return a timetrap time value (other than a function).</item> + </list> + <p>Before execution of the timetrap function (which is performed + on a parallel, dedicated timetrap process), Common Test cancels + any previously set timer for the test case or configuration function. + When the timetrap function returns, the timeout is triggered, <em>unless</em> + the return value is a valid timetrap time, such as an integer, + or a <c>{SecMinOrHourTag,Time}</c> tuple (see the + <seealso marker="common_test">common_test reference manual</seealso> for + details). If a time value is returned, a new timetrap is started + to generate a timeout after the specified time.</p> + + <p>The user timetrap function may of course return a time value after a delay, + and if so, the effective timetrap time is the delay time <em>plus</em> the + returned time.</p> </section> <section> |