diff options
Diffstat (limited to 'lib/common_test/doc')
| -rw-r--r-- | lib/common_test/doc/src/ct_hooks.xml | 5 | ||||
| -rw-r--r-- | lib/common_test/doc/src/event_handler_chapter.xml | 15 | ||||
| -rw-r--r-- | lib/common_test/doc/src/notes.xml | 120 | ||||
| -rw-r--r-- | lib/common_test/doc/src/run_test_chapter.xml | 6 |
4 files changed, 136 insertions, 10 deletions
diff --git a/lib/common_test/doc/src/ct_hooks.xml b/lib/common_test/doc/src/ct_hooks.xml index b3e713c77f..859ff9df14 100644 --- a/lib/common_test/doc/src/ct_hooks.xml +++ b/lib/common_test/doc/src/ct_hooks.xml @@ -485,8 +485,9 @@ NewCTHState</name> <fsummary>Called after the CTH scope ends</fsummary> <type> - <v>TestcaseName = end_per_suite | init_per_group | - end_per_group | atom()</v> + <v>TestcaseName = end_per_suite | {init_per_group,GroupName} | + {end_per_group,GroupName} | atom()</v> + <v>GroupName = atom()</v> <v>Reason = {tc_auto_skip | tc_user_skip, term()}</v> <v>CTHState = NewCTHState = term()</v> </type> diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml index 65cc2a4042..47d0ba59fb 100644 --- a/lib/common_test/doc/src/event_handler_chapter.xml +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -193,7 +193,8 @@ <p><c>GroupName = unknown | atom()</c>, name of the group (unknown if init- or end function times out).</p> <p><c>GroupProperties = list()</c>, list of execution properties for the group.</p> - <p><c>Result = ok | {skipped,SkipReason} | {failed,FailReason}</c>, the result.</p> + <p><c>Result = ok | {auto_skipped,SkipReason} | {skipped,SkipReason} | {failed,FailReason}</c>, + the result.</p> <marker id="skipreason"/> <p><c>SkipReason = {require_failed,RequireInfo} | {require_failed_in_suite0,RequireInfo} | @@ -228,7 +229,9 @@ <marker id="tc_auto_skip"></marker> <c>#event{name = tc_auto_skip, data = {Suite,Func,Reason}}</c> <p><c>Suite = atom()</c>, the name of the suite.</p> - <p><c>Func = atom()</c>, the name of the test case or configuration function.</p> + <p><c>Func = atom() | {end_per_group,GroupName}</c>, the name of the test case + or configuration function.</p> + <p><c>GroupName = atom()</c>, name of the group.</p> <p><c>Reason = {failed,FailReason} | {require_failed_in_suite0,RequireInfo}</c>, reason for auto skipping <c>Func</c>.</p> @@ -254,9 +257,11 @@ <item> <marker id="tc_user_skip"></marker> - <c>#event{name = tc_user_skip, data = {Suite,TestCase,Comment}}</c> - <p><c>Suite = atom()</c>, name of the suite.</p> - <p><c>TestCase = atom()</c>, name of the test case.</p> + <c>#event{name = tc_user_skip, data = {Suite,Func,Comment}}</c> + <p><c>Suite = atom()</c>, the name of the suite.</p> + <p><c>Func = atom() | {end_per_group,GroupName}</c>, the name of the test case + or configuration function.</p> + <p><c>GroupName = atom()</c>, name of the group.</p> <p><c>Comment = string()</c>, reason for skipping the test case.</p> <p>This event specifies that a test case has been skipped by the user. It is only ever received if the skip was declared in a test specification. diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index b4a5c610cf..f10d5f85bf 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -32,6 +32,126 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.7.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Return values from group and testcase info functions are + now properly checked, and associated test cases are auto + skipped if a return value is invalid.</p> + <p> + Own Id: OTP-10631 Aux Id: kunagi-345 [256] </p> + </item> + <item> + <p>The way Common Test handles skipping of test cases has + been updated. In previous versions, returning + <c>{skip,Reason}</c> from a configuration function (such + as init_per_suite or init_per_group), resulted in all + affected test cases getting skipped with status + <c>auto_skipped</c>. This was inappropriate, since this + status is supposed to be used to inform that Common Test + has taken the initiative to skip something (e.g. a test + case group if init_per_group failed). Therefore, in this + version of Common Test, whenever the user skips a suite, + group, or individual test case (by means of a + configuration function or test specification term), the + affected test cases get the status <c>user_skipped</c> + instead.</p> <p>This update has meant a few changes that + may affect Common Test users in various ways: <list> + <item>The test results and statistics will be affected, + which is important to know when running regression tests + and comparing results to previous test runs.</item> + <item>Users that read or parse the textual log file + <c>suite.log</c> will notice that an auto skipped + function is now reported as <c>auto_skipped</c> rather + than <c>skipped</c> as before.</item> <item>When + <c>require</c> fails in an info function (such as suite/0 + or group/1), all affected configuration functions and + test cases are marked as <c>auto_skipped</c>.</item> + <item>If Common Test detects an error in the test suite + (such as e.g. an invalid all/0 function), all affected + configuration functions and test cases are marked as + <c>auto_skipped</c>.</item> <item>If a repeated test run + session reaches a deadline with <c>force_stop</c> + enabled, all remaining test cases are marked as + <c>auto_skipped</c> rather than <c>user_skipped</c> as + before.</item> <item>The event messages that Common Test + generates during test runs have been affected by this + update. For details see OTP-11524.</item> </list> </p> + <p> + Own Id: OTP-11305 Aux Id: OTP-11524 </p> + </item> + <item> + <p> + Returning {skip, Reason} from a pre_end_per_group/3 user + hook function would result in an exit in the Common Test + cth_log_redirect hook. This problem has been solved.</p> + <p> + Own Id: OTP-11409 Aux Id: seq12446 </p> + </item> + <item> + <p> + When the netconf server did not respond to the + close-session request, the call to + ct_netconfc:close_session/2 would hang forever waiting + for the netconf client to terminate. This has been + corrected. The client will now always terminate (and take + down the connection) if the close-session request times + out.</p> + <p> + Own Id: OTP-11478</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix cth_log_redirect.erl to fulfill gen_event behaviour. + Thanks to Roberto Aloi.</p> + <p> + Own Id: OTP-11401</p> + </item> + <item> + <p> + The first argument of the CT hook callback function + <c>on_tc_skip/3</c> has been modified. When this function + is called for <c>init_per_group</c> or + <c>end_per_group</c>, the value of the first argument is + now <c>{init_per_group,GroupName}</c> or + <c>{end_per_group,GroupName}</c>.</p> + <p> + Own Id: OTP-11523</p> + </item> + <item> + <p>The following modifications have been made to the + event messages that Common Test sends during test + execution: <list> <item>For the <c>tc_auto_skip</c> + event, the value of the <c>Func</c> element has changed + from <c>end_per_group</c> to + <c>{end_per_group,GroupName}</c>.</item> <item>When + <c>require</c> fails in an info function, such as suite/0 + or group/1, the init configuration function is now + reported as <c>auto_skipped</c> intead of <c>skipped</c>, + with the <c>tc_done</c> event.</item> <item>When + <c>require</c> fails in an info function because of a + configuration name already in use, the <c>tc_done</c> + event now reports the error with a tuple (of size 2) + tagged <c>failed</c> instead of <c>skipped</c>.</item> + </list> Please see the Event Handling chapter in the + Common Test User's Guide for reference. </p> + <p> + Own Id: OTP-11524 Aux Id: OTP-11305 </p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.7.3</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index 03d98eb371..44fe73d24f 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -205,12 +205,12 @@ <p>The <c>ct_run</c> program sets the exit status before shutting down. The following values are defined:</p> <list> - <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto-skipped test cases.</item> - <item><c>1</c> indicates that one or more test cases have failed, or have been auto-skipped.</item> + <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto skipped test cases.</item> + <item><c>1</c> indicates that one or more test cases have failed, or have been auto skipped.</item> <item><c>2</c> indicates that the test execution has failed because of e.g. compilation errors, an illegal return value from an info function, etc.</item> </list> - <p>If auto-skipped test cases should not affect the exit status, you may change the default + <p>If auto skipped test cases should not affect the exit status, you may change the default behaviour using start flag:</p> <pre>-exit_status ignore_config</pre> |