diff options
Diffstat (limited to 'lib/test_server/doc/src/notes.xml')
| -rw-r--r-- | lib/test_server/doc/src/notes.xml | 1694 |
1 files changed, 0 insertions, 1694 deletions
diff --git a/lib/test_server/doc/src/notes.xml b/lib/test_server/doc/src/notes.xml deleted file mode 100644 index b48bda94d0..0000000000 --- a/lib/test_server/doc/src/notes.xml +++ /dev/null @@ -1,1694 +0,0 @@ -<?xml version="1.0" encoding="utf-8" ?> -<!DOCTYPE chapter SYSTEM "chapter.dtd"> - -<chapter> - <header> - <copyright> - <year>2004</year><year>2013</year> - <holder>Ericsson AB. All Rights Reserved.</holder> - </copyright> - <legalnotice> - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - - </legalnotice> - - <title>APPLICATION Release Notes</title> - <prepared>Peter Andersson</prepared> - <responsible>Peter Andersson</responsible> - <docno></docno> - <approved></approved> - <checked></checked> - <date>2007-11-30</date> - <rev>A</rev> - <file>notes.xml</file> - </header> - -<section><title>Test_Server 3.9.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - When generating Makefile from Makefile.src, - ts_lib:get_arg/4 earlier removed all spaces in the - extracted argument. The code was probably meant for - removing leading and trailing spaces only, and is now - corrected to do so.</p> - <p> - Own Id: OTP-13015</p> - </item> - <item> - <p> - With the Common Test 'create_priv_dir' start option set - to 'auto_per_tc', the name of the priv directory for a - configuration function could clash with the name of the - priv directory for a test case, which would cause Test - Server failure. This error has been corrected.</p> - <p> - Own Id: OTP-13181</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.9</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The status of an aborted test due to test suite - compilation error has changed from 'auto_skipped' to - 'failed'. This affects both the textual log file, event - handling and CT hook callbacks. The logging of - compilation failures has also been improved, especially - in the case of multiple test suites failing compilation.</p> - <p> - Own Id: OTP-10816</p> - </item> - <item> - <p> - The Test Server source code parser (erl2html2) failed to - handle the macro tuple in the syntax tree returned by - epp_dodger. This error has been corrected.</p> - <p> - Own Id: OTP-12740</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The Test Server application has been marked as obsolete - and will be removed from OTP in the next major release - (OTP 19.0).</p> - <p> - Own Id: OTP-10923 Aux Id: OTP-12705 </p> - </item> - <item> - <p> - When running OTP tests using the ts interface, it is now - possible to specify so called test categories per OTP - application. A test category is represented by a CT test - specification and defines an arbitrary subset of existing - test suites, groups and cases. Examples of test - categories are 'smoke' (smoke tests) and 'bench' - (benchmarks). (Call ts:help() for more info). Also, - functions for reading terms from the current test - specification during test, ct:get_testspec_terms/0 and - ct:get_testspec_terms/1, have been implemented.</p> - <p> - Own Id: OTP-11962</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.8.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - If the last expression in a test case causes a timetrap - timeout, the stack trace is ignored and not printed to - the test case log file. This happens because the - {Suite,TestCase,Line} info is not available in the stack - trace in this scenario, due to tail call elimination. - Common Test has been modified to handle this situation by - inserting a {Suite,TestCase,last_expr} tuple in the - correct place and printing the stack trace as expected.</p> - <p> - Own Id: OTP-12697 Aux Id: seq12848 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.8</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - When installing test suites in a cross compilation - environment, ts_install was not able to read the values - of the environment variables specified in the - configuration file. This has been fixed.</p> - <p> - Own Id: OTP-11441</p> - </item> - <item> - <p> - Printouts by means of ct:log/2/3 or ct:pal/2/3 from the - hook functions on_tc_fail/2 and on_tc_skip/2 would (quite - unexpectedly) end up in the "unexpected i/o" log file - instead of in the test case log file. This behaviour has - been changed so that now, all printouts (including stdio - printouts) from these hook functions will be routed to - the test case log file.</p> - <p> - Own Id: OTP-12468</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The format of the information printed on top of the test - case (and configuration function) log file has been - slightly modified, mainly in order to make the start - configuration data easier to read and interpret.</p> - <p> - Own Id: OTP-12518 Aux Id: seq12808 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.7.2</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The source code to html code generator in Test Server - (and Common Test) would fail to generate anchors in the - html code for functions with non-expandable macros, - resulting in bad html links to such functions. This - correction lets the code generator ignore macros that - can't be expanded (i.e. not pre-process them), so that - correct anchors will always be produced.</p> - <p> - Own Id: OTP-11766 Aux Id: seq12556 </p> - </item> - <item> - <p> - Make sure to install .hrl files when needed</p> - <p> - Own Id: OTP-12197</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Distribute <c>autoconf</c> helpers to applications at - build time instead of having multiple identical copies - committed in the repository.</p> - <p> - Own Id: OTP-12348</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.7.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The mechanism for running code cover analysis with - common_test has been improved. Earlier, if a test run - consisted of multiple tests, cover would be started and - stopped for each test. This would give "intermediate" - cover logs available from the "Coverage log" link on the - test suite result pages. To accumulate cover data over - all tests, the 'export' option had to be used in the - cover spec file. This was not well documented, and the - functionality was quite confusing.</p> - <p> - Using the 'nodes' option in the cover spec file would - fail when the test run consisted of multiple tests, since - the specified nodes would only be included in the cover - analysis of the first test.</p> - <p> - The repeated compilation and analysis of the same modules - was also very time consuming.</p> - <p> - To overcome these problems, ct will now only cover - compile and analyze modules once per test run, i.e. once - for each cover spec file. The log file is available via a - new button on the top level index page. The old "Coverage - log" links on the test suite result pages still exist, - but they all point to the same log containing the - accumulated result.</p> - <p> - Own Id: OTP-11971</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.7</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Application upgrade (appup) files are corrected for the - following applications: </p> - <p> - <c>asn1, common_test, compiler, crypto, debugger, - dialyzer, edoc, eldap, erl_docgen, et, eunit, gs, hipe, - inets, observer, odbc, os_mon, otp_mibs, parsetools, - percept, public_key, reltool, runtime_tools, ssh, - syntax_tools, test_server, tools, typer, webtool, wx, - xmerl</c></p> - <p> - A new test utility for testing appup files is added to - test_server. This is now used by most applications in - OTP.</p> - <p> - (Thanks to Tobias Schlager)</p> - <p> - Own Id: OTP-11744</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Calls to erlang:open_port/2 with 'spawn' are updated to - handle space in the command path.</p> - <p> - Own Id: OTP-10842</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.6.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>The way Common Test handles skipping of test cases has - been updated. In previous versions, returning - <c>{skip,Reason}</c> from a configuration function (such - as init_per_suite or init_per_group), resulted in all - affected test cases getting skipped with status - <c>auto_skipped</c>. This was inappropriate, since this - status is supposed to be used to inform that Common Test - has taken the initiative to skip something (e.g. a test - case group if init_per_group failed). Therefore, in this - version of Common Test, whenever the user skips a suite, - group, or individual test case (by means of a - configuration function or test specification term), the - affected test cases get the status <c>user_skipped</c> - instead.</p> <p>This update has meant a few changes that - may affect Common Test users in various ways:</p> <list> - <item>The test results and statistics will be affected, - which is important to know when running regression tests - and comparing results to previous test runs.</item> - <item>Users that read or parse the textual log file - <c>suite.log</c> will notice that an auto skipped - function is now reported as <c>auto_skipped</c> rather - than <c>skipped</c> as before.</item> <item>When - <c>require</c> fails in an info function (such as suite/0 - or group/1), all affected configuration functions and - test cases are marked as <c>auto_skipped</c>.</item> - <item>If Common Test detects an error in the test suite - (such as e.g. an invalid all/0 function), all affected - configuration functions and test cases are marked as - <c>auto_skipped</c>.</item> <item>If a repeated test run - session reaches a deadline with <c>force_stop</c> - enabled, all remaining test cases are marked as - <c>auto_skipped</c> rather than <c>user_skipped</c> as - before.</item> <item>The event messages that Common Test - generates during test runs have been affected by this - update. For details see OTP-11524.</item> </list> - <p> - Own Id: OTP-11305 Aux Id: OTP-11524 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.6.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Test Server installed an error handler (test_server_h) - only to be able to write the name of the current test - case to stdout whenever it received an error- or progress - report. This functionality was not useful and has been - removed. The built-in Common Test hook, cth_log_redirect, - has instead been improved to now also tag all error- and - progress reports in the log with suite-, group-, and/or - test case name.</p> - <p> - Own Id: OTP-11263 Aux Id: seq12251 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - A new log, the "Pre- and Post Test I/O Log", has been - introduced, which makes it possible to capture error- and - progress reports, as well as printouts made with ct:log/2 - and ct:pal/2, before and after a test run. (Some minor - improvements of the logging system have been made at the - same time). Links to the new log are found on the Common - Test Framework Log page. The Common Test User's Guide has - been updated with information about the new log and also - with a new section on how to synchronize external - applications with Common Test by means of the CT Hook - init and terminate functions.</p> - <p> - Own Id: OTP-11272</p> - </item> - </list> - </section> - - - <section><title>Known Bugs and Problems</title> - <list> - <item> - <p> - Test Server: Report auto_skipped in major log.</p> - <p> - Own Id: OTP-11297</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.6.2</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Some unused code related to remote targets is removed, - and documentation is updated.</p> - <p> - Own Id: OTP-10607 Aux Id: kunagi-338 [249] </p> - </item> - <item> - <p> - A bug in test_server_gl caused io requests containing - invalid data (i.e. not unicode:chardata()) to hang, since - no io reply was sent. This has been corrected.</p> - <p> - Own Id: OTP-10991</p> - </item> - <item> - <p> - Common Test would, in case of timetrap error, print a - warning in the log if end_per_testcase wasn't implemented - in the suite, even though it's an optional function. This - printout has been removed.</p> - <p> - Own Id: OTP-11052</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The '-force_stop' flag to use with time-limited repeats - of test runs can now be used with a new 'skip_rest' - option which causes the rest of the test cases in the - ongoing test job to be skipped when the time limit is - reached. E.g. 'ct_run -spec xxx -duration 010000 - -force_stop skip_rest'</p> - <p> - Own Id: OTP-10856 Aux Id: OTP-10832 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.6.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The unicode update of test_server for R16A introduced a - few potential errors when logging to files. Sometimes ~tp - or ~ts was used for formatting also when writing to files - that were not opened with the {encoding,utf8} option. If - then the argument contained unicode characters above 255, - the file descriptor would crash. This has been corrected - by the following modifications:</p> <list> <item> Since the - 'unexpected_io' log file is used only when the test case - HTML file is not available (e.g. between test cases), - this file is now also a HTML file and as other - test_server HTML logs it is always UTF-8 encoded </item> - <item> Since it is possible to change which information - is going to which log file (with - test_server_ctrl:set_levels/3), we do not have full - control over which information is written to which file. - This means that any printout could be written to the - 'major' log file (suite.log), which was earlier encoded - as latin1. To avoid crashing this file descriptor due to - unicode strings, the 'major' log file is now also encoded - in UTF-8 (possible incopatibility). </item> <item> The - cross_cover.info file is no longer a text file which can - be read with file:consult/1, instead it is written as a - pure binary file using term_to_binary when writing and - binary_to_term when reading. </item> <item> The encoding - of the file named 'last_name', which only content is the - path to the last run.<timestamp> directory, is now - dependent on the file name mode of the VM. If file names - are expected to be unicode, then the 'last_name' file is - UTF-8 encoded, else it is latin1 encoded. </item> </list> - <p> - Also, ~tp has been changed back to ~p unless it is - somehow likely that the argument includes strings. It is - not obvious that this is the correct thing to do, but - some decission had to be taken...</p> - <p> - Own Id: OTP-10780</p> - </item> - <item> - <p> - Using the force_stop flag/option to interrupt a test run - caused a crash in Common Test. This problem has been - solved.</p> - <p> - Own Id: OTP-10832</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.6</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Line numbering of erlang files that were not correctly - indented could be wrong after coverting to html with - erl2html2:convert/[2,3] (the source code pointed to from - the test case). This has been corrected.</p> - <p> - Also, there are now link targets for each line and not - only for each 10th line, and link targets for functions - now include the arity and not only the function name - (e.g. func/1 has a link target "func-1").</p> - <p> - Own Id: OTP-9710 Aux Id: seq11945, kunagi-201 [112] </p> - </item> - <item> - <p> - Severe errors detected by <c>test_server</c> (e.g. if log - files directories cannot be created) will now be reported - to <c>common_test</c> and noted in the <c>common_test</c> - logs.</p> - <p> - Own Id: OTP-9769 Aux Id: kunagi-202 [113] </p> - </item> - <item> - <p> - The earlier undocumented cross cover feature for - accumulating cover data over multiple tests has now been - fixed and documented.</p> - <p> - Own Id: OTP-9870 Aux Id: kunagi-206 [117] </p> - </item> - <item> - <p> - If the test suite itself was included in code coverage - analysis, then the test_server would not manage to set - data_dir correctly for the test. This has been corrected.</p> - <p> - Own Id: OTP-9956 Aux Id: kunagi-207 [118] </p> - </item> - <item> - <p> - Any call to test_server:break/1 should cancel all active - timetramps. However, in some cases - Suite:end_per_testcase/2 is executed on a different - process than the test case itself, and if - test_server:break/1 was called from there, the timetraps - were not cancelled. This has been corrected.</p> - <p> - Own Id: OTP-10046 Aux Id: kunagi-174 [85] </p> - </item> - <item> - <p>When a test case failed because of a timetrap time - out, the <c>Config</c> data for the case was lost in the - following call to <c>end_per_testcase/2</c>, and also in - calls to the CT Hook function - <c>post_end_per_testcase/4</c>. This problem has been - solved and the <c>Config</c> data is now correctly passed - to the above functions after a timetrap timeout - failure.</p> - <p> - Own Id: OTP-10070 Aux Id: kunagi-175 [86] </p> - </item> - <item> - <p>In test_server, the same process would supervise the - currently running test case and be group leader (and IO - server) for the test case. Furthermore, when running - parallel test cases, new temporary supervisor/group - leader processes were spawned and the process that was - group leader for sequential test cases would not be - active. That would lead to several problems:</p> - <p>* Processes started by init_per_suite will inherit the - group leader of the init_per_suite process (and that - group leader would not process IO requests when parallel - test cases was running). If later a parallel test case - caused such a processto print using (for example) - io:format/2, the calling would hang.</p> - <p>* Similarly, if a process was spawned from a parallel - test case, it would inherit the temporary group leader - for that parallel test case. If that spawned process - later - when the group of parallel tests have finished - - attempted to print something, its group leader would be - dead and there would be <c>badarg</c> exception.</p> - <p>Those problems have been solved by having group - leaders separate from the processes that supervises the - test cases, and keeping temporary group leader process - for parallel test cases alive until no more process in - the system use them as group leaders.</p> - <p>Also, a new <c>unexpected_io.log</c> log file - (reachable from the summary page of each test suite) has - been introduced. All unexpected IO will be printed into - it(for example, IO to a group leader for a parallel test - case that has finished).</p> - <p> - Own Id: OTP-10101 Aux Id: OTP-10125 </p> - </item> - <item> - <p> - The stability of <c>common_test</c> and - <c>test_server</c> when running test cases in parallel - has been improved.</p> - <p> - Own Id: OTP-10480 Aux Id: kunagi-318 [229] </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Added a general framework for executing benchmarks of - Erlang/OTP. Benchmarks for the Erlang VM and mnesia have - been incorporated in the framework. </p> - <p> - For details about how to add more benchmarks see - $ERL_TOP/HOWTO/BENCHMARKS.md in the source distribution.</p> - <p> - Own Id: OTP-10156</p> - </item> - <item> - <p> - Update common test modules to handle Unicode:</p> <list> - <item> Use UTF-8 encoding for all HTML files, except the - HTML version of the test suite generated with - erl2html2:convert, which will have the same encoding as - the original test suite (.erl) file. </item> <item> - Encode link targets in HTML files with - test_server_ctrl:uri_encode/1. </item> <item> Use unicode - modifier 't' with ~s when appropriate. </item> <item> Use - unicode:characters_to_list and - unicode:characters_to_binary for conversion between - binaries and strings instead of binary_to_list and - list_to_binary. </item> </list> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.5.3</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - test_server_h will now recognize info_reports written by - ct connection handlers (according to the description in - cth_conn_log) and ignore them as they will be completely - handled by by ct_conn_log_h.</p> - <p> - Earlier test_server_h would print a tag (testcase name) - before forwarding the report to error_logger_tty_h. This - would cause lots of tags in the log with no info report - following (since error_logger_tty_h did not handle them).</p> - <p> - Own Id: OTP-10571</p> - </item> - </list> - </section> - - - <section><title>Known Bugs and Problems</title> - <list> - <item> - <p> - Restore Config data if lost when test case fails.</p> - <p> - Own Id: OTP-10070 Aux Id: kunagi-175 [86] </p> - </item> - <item> - <p> - IO server error in test_server.</p> - <p> - Own Id: OTP-10125 Aux Id: OTP-10101, kunagi-177 [88] </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.5.2</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The documentation has been updated with the latest - changes for the test_server_ctrl:report/2 function.</p> - <p> - Own Id: OTP-10086 Aux Id: seq12066 </p> - </item> - <item> - <p> - The ct:get_status/0 function failed to report status if a - parallel test case group was running at the time of the - call. This has been fixed and the return value for the - function has been updated. Please see the ct reference - manual for details.</p> - <p> - Own Id: OTP-10172</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - It is now possible to sort the HTML tables by clicking on - the header elements. In order to reset a sorted table, - the browser window should simply be refreshed. This - feature requires that the browser supports javascript, - and has javascript execution enabled. If the 'ct_run - -basic_html' flag is used, no javascript code is included - in the generated HTML code.</p> - <p> - Own Id: OTP-9896 Aux Id: seq12034, OTP-9835 </p> - </item> - <item> - <p> - Verbosity levels for log printouts has been added. This - makes it possible to specify preferred verbosity for - different categories of log printouts, as well as general - printouts (such as standard IO), to allow control over - which strings get printed and which get ignored. New - versions of the Common Test logging functions, ct:log, - ct:pal and ct:print, have been introduced, with a new - Importance argument added. The Importance value is - compared to the verbosity level at runtime. More - information can be found in the chapter about Logging in - the Common Test User's Guide.</p> - <p> - Own Id: OTP-10067 Aux Id: seq12050 </p> - </item> - <item> - <p> - The Erlang/OTP test runner ts has been extended to allow - cross compilation of test suites. To cross compile the - test suites first follow the normal cross compilation - procedures and release the tests on the build host. Then - install ts using an xcomp specification file and compile - test suites using ts:compile_testcases/0. For more - details see $ERL_TOP/xcomp/README.md.</p> - <p> - Own Id: OTP-10074</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.5.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - After a test case timeout or abortion, the - end_per_testcase function executes on a new dedicated - process. The group leader for this process should be set - to the IO server for the test case, which was not done - properly. The result of this error was that no warnings - about end_per_testcase failing or timing out were ever - printed in the test case log. Also, help functions such - as e.g. test_server:stop_node/1, attempting to - synchronize with the IO server, would hang. The fault has - been corrected.</p> - <p> - Own Id: OTP-9666</p> - </item> - <item> - <p> - A deadlock situation could occur if Common Test is - forwarding error_handler printouts to Test Server at the - same time a new test case is starting. This error has - been fixed.</p> - <p> - Own Id: OTP-9894</p> - </item> - <item> - <p> - When a test case was killed because of a timetrap - timeout, the current location (suite, case and line) was - not printed correctly in the log files. This has been - corrected.</p> - <p> - Own Id: OTP-9930 Aux Id: seq12002 </p> - </item> - <item> - <p> - Test Server and Common Test would add new error handlers - with each test run and fail to remove previously added - ones. In the case of Test Server, this would only happen - if SASL was not running on the test node. This has been - fixed.</p> - <p> - Own Id: OTP-9941 Aux Id: seq12009 </p> - </item> - <item> - <p> - If a test case process was terminated due to an exit - signal from a linked process, Test Server failed to - report the correct name of the suite and case to the - framework. This has been corrected.</p> - <p> - Own Id: OTP-9958 Aux Id: OTP-9855 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - A new optional feature has been introduced that enables - Common Test to generate priv_dir directory names that are - unique for each test case or config function. The name of - the option/flag is 'create_priv_dir' and it can be set to - value 'auto_per_run' (which is the default, existing, - behaviour), or 'auto_per_tc' or 'manual_per_tc'. If - 'auto_per_tc' is used, Test Server creates a dedicated - priv_dir automatically for each test case (which can be - very expensive in case of many and/or repeated cases). If - 'manual_per_tc' is used, the user needs to create the - priv_dir explicitly by calling the new function - ct:make_priv_dir/0.</p> - <p> - Own Id: OTP-9659 Aux Id: seq11930 </p> - </item> - <item> - <p> - A column for test case group name has been added to the - suite overview HTML log file.</p> - <p> - Own Id: OTP-9730 Aux Id: seq11952 </p> - </item> - <item> - <p> - It is now possible to use the post_end_per_testcase CT - hook function to print a comment for a test case in the - overview log file, even if the test case gets killed by a - timetrap or unknown exit signal, or if the - end_per_testcase function times out.</p> - <p> - Own Id: OTP-9855 Aux Id: seq11979 </p> - </item> - <item> - <p> - Common Test will now print error information (with a time - stamp) in the test case log file immediately when a test - case fails. This makes it easier to see when, in time, - the fault actually occured, and aid the job of locating - relevant trace and debug printouts in the log.</p> - <p> - Own Id: OTP-9904 Aux Id: seq11985, OTP-9900 </p> - </item> - <item> - <p> - Test Server has been modified to check the SASL - errlog_type parameter when receiving an error logger - event, so that it doesn't print reports of type that the - user has disabled.</p> - <p> - Own Id: OTP-9955 Aux Id: seq12013 </p> - </item> - <item> - <p> - If an application cannot be found by ts it is - automatically skipped when testing.</p> - <p> - Own Id: OTP-9971</p> - </item> - <item> - <p> - By specifying a user defined function ({M,F,A} or fun) as - timetrap value, either by means of an info function or by - calling ct:timetrap/1, it is now possible to set a - timetrap that will be triggered when the user function - returns.</p> - <p> - Own Id: OTP-9988 Aux Id: OTP-9501, seq11894 </p> - </item> - <item> - <p> - If the optional configuration functions init_per_suite/1 - and end_per_suite/1 are not implemented in the test - suite, local Common Test versions of these functions are - called instead, and will be displayed in the overview log - file. Any printouts made by the pre- or - post_init_per_suite and pre- or post_end_per_suite hook - functions are saved in the log files for these functions.</p> - <p> - Own Id: OTP-9992</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.5</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The test case group info function has been implemented in - Common Test. Before execution of a test case group, a - call is now made to <c>TestSuite:group(GroupName)</c>. - The function returns a list of test properties, e.g. to - specify timetrap values, require configuration data, etc - (analogue to the test suite- and test case info - function). The scope of the properties set by - <c>group(GroupName)</c> is all test cases and sub-groups - of group <c>GroupName</c>.</p> - <p> - Own Id: OTP-9235</p> - </item> - <item> - <p> - The look of the HTML log files generated by Common Test - and Test Server has been improved (and made easier to - customize) by means of a CSS file.</p> - <p> - Own Id: OTP-9706</p> - </item> - </list> - </section> - - - <section><title>Known Bugs and Problems</title> - <list> - <item> - <p> - Fix problems in CT/TS due to line numbers in exceptions.</p> - <p> - Own Id: OTP-9203</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4.5</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - An error in how comments are colored in the test suite - overview html log file has been corrected. As result, a - new framework callback function, format_comment/1, has - been introduced.</p> - <p> - Own Id: OTP-9237</p> - </item> - <item> - <p> - Test Server did not release SASL TTY handlers - (sasl_report_tty_h and error_logger_tty_h) properly after - each test run. This error has been fixed.</p> - <p> - Own Id: OTP-9311</p> - </item> - <item> - <p> - Automatically generated init- and end-configuration - functions for test case groups caused incorrect execution - order of test cases. This has been corrected.</p> - <p> - Own Id: OTP-9369</p> - </item> - <item> - <p> - If ct:log/2 was called with bad arguments, this could - cause the Common Test IO handling process to crash. This - fault has been corrected.</p> - <p> - Own Id: OTP-9371 Aux Id: OTP-8933 </p> - </item> - <item> - <p> - A bug has been fixed that made Test Server call the - end_tc/3 framework function with an incorrect module name - as first argument.</p> - <p> - Own Id: OTP-9379 Aux Id: seq11863 </p> - </item> - <item> - <p> - If end_per_testcase caused a timetrap timeout, the actual - test case status was discarded and the test case logged - as successful (even if the case had actually failed - before the call to end_per_testcase). This fault has been - fixed.</p> - <p> - Own Id: OTP-9397</p> - </item> - <item> - <p> - If a timetrap timeout occured during execution of of a - function in a lib module (i.e. a function called directly - or indirectly from a test case), the Suite argument in - the end_tc/3 framework callback function would not - correctly contain the name of the test suite, but the lib - module. (This would only happen if the lib module was - compiled with ct.hrl included). This error has been - solved.</p> - <p> - Own Id: OTP-9398</p> - </item> - <item> - <p> - Add a proplist() type</p> - <p> - Recently I was adding specs to an API and found that - there is no canonical proplist() type defined. (Thanks to - Ryan Zezeski)</p> - <p> - Own Id: OTP-9499</p> - </item> - <item> - <p> XML files have been corrected. </p> - <p> - Own Id: OTP-9550 Aux Id: OTP-9541 </p> - </item> - <item> - <p> - If a test suite would start with a test case group - defined without the init_per_group/2 and end_per_group/2 - function, init_per_suite/1 would not execute initially - and logging of the test run would fail. This error has - been fixed.</p> - <p> - Own Id: OTP-9584</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - A new option, 'logopts', has been introduced, to make it - possible to modify some aspects of the logging behaviour - in Common Test (or Test Server). For example, whenever an - io printout is made, test_server adds newline (\n) to the - end of the output string. This may not always be a - preferred action and can therefore be disabled by means - of "ct_run ... -logopts no_nl" (or ct:run_test([..., - {logopts,[no_nl]}])). A new framework callback function, - get_logopts/0, has been introduced (see the ct_framework - module for details).</p> - <p> - Own Id: OTP-9372 Aux Id: OTP-9396 </p> - </item> - <item> - <p> - A new option, 'logopts', has been introduced, to make it - possible to modify some aspects of the logging behaviour - in Common Test (or Test Server). For example, if the html - version of the test suite source code should not be - generated during the test run (and consequently be - unavailable in the log file system), the feature may be - disabled by means of "ct_run ... -logopts no_src" (or - ct:run_test([..., {logopts,[no_src]}])). A new framework - callback function, get_logopts/0, has been introduced - (see the ct_framework module for details).</p> - <p> - Own Id: OTP-9396 Aux Id: seq11869, OTP-9372 </p> - </item> - <item> - <p> - It is now possible to use a tuple {M,F,A}, or a fun, as - timetrap specification in the suite info function or test - case info functions. The function must return a valid - timeout value, as documented in the common_test man page - and in the User's Guide.</p> - <p> - Own Id: OTP-9501 Aux Id: seq11894 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - It was previously not possible to use timetrap value - 'infinity' with ct:timetrap/1. This has been fixed.</p> - <p> - Own Id: OTP-9159</p> - </item> - <item> - <p> - A bug that made it impossible to cancel the previous - timetrap when calling ct:timetrap/1 has been corrected.</p> - <p> - Own Id: OTP-9233 Aux Id: OTP-9159 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - When running tests with auto-compilation disabled, Common - Test could only display the test suite source code on - html format in the test case log if the source file was - located in the same directory as the pre-compiled suite. - This has been modified so that Common Test now tries to - locate the source file by means of the test suite module - info (Suite:module_info/1). As a result, a suite may now - be compiled to a different output directory (e.g. - $MYTEST/bin) than the source code directory (e.g. - $MYTEST/src), without the source-code-to-html generation - being affected.</p> - <p> - Own Id: OTP-9138</p> - </item> - <item> - <p> - It is now possible to return a tuple {fail,Reason} from - init_per_testcase/2. The result is that the associated - test case gets logged as failed without ever executing.</p> - <p> - Own Id: OTP-9160 Aux Id: seq11502 </p> - </item> - <item> - <p> - Added DragonflyBSD check in test_server configure.</p> - <p> - Own Id: OTP-9249</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Updated the ts*.config files to contain information - relevant to testing Erlang/OTP in an open source - environment.</p> - <p> - Own Id: OTP-9017</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Alpha release of Common Test Hooks (CTH). CTHs allow the - users of common test to abtract out common behaviours - from test suites in a much more elegant and flexible way - than was possible before. Note that the addition of this - feature may introduce minor changes in the undocumented - behaviour of the interface inbetween common_test and - test_server.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-8851</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4.2</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p>Miscellaneous updates</p> - <p> - Own Id: OTP-8976</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Returning {return_group_result,failed} from end_per_group - in a group that is part of a sequence, did not cause the - proceeding cases (or groups) to get skipped. This has - been fixed.</p> - <p> - Own Id: OTP-8753 Aux Id: seq11644 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Common Test has been updated to handle start options and - test specification terms for test case groups (and test - cases in groups). Also, an option named 'label', has been - added that associates the test run with a name that - Common Test prints in the overview HTML logs.</p> - <p> - Own Id: OTP-8725 Aux Id: OTP-8727 </p> - </item> - <item> - <p> - It is now possible to skip all tests in a suite, or a - group, by returning {fail,Reason} from the end_tc/5 - framework function for init_per_suite, or init_per_group.</p> - <p> - Own Id: OTP-8805 Aux Id: seq11664 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Returning {fail,Reason} from the framework end_tc - function was not handled properly by Test Server for all - test suite functions.</p> - <p> - Own Id: OTP-8492 Aux Id: seq11502 </p> - </item> - <item> - <p> - If the framework end_tc function would hang and get - aborted by Test Server, there was no indication of - failure in the logs. This has been fixed.</p> - <p> - Own Id: OTP-8682 Aux Id: seq11504 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - It is now possible for the Test Server framework end_tc - function to change the status of the test case from ok or - auto-skipped to failed by returning {fail,Reason}.</p> - <p> - Own Id: OTP-8495 Aux Id: seq11502 </p> - </item> - <item> - <p> - Test Server will now call the end_per_testcase/2 function - even if the test case has been terminated explicitly - (with abort_current_testcase/1), or after a timetrap - timeout. Under these circumstances the return value of - end_per_testcase is completely ignored. Therefore the - function will not be able to change the reason for test - case termination by returning {fail,Reason}, nor will it - be able to save data with {save_config,Data}.</p> - <p> - Own Id: OTP-8500 Aux Id: seq11521 </p> - </item> - <item> - <p> - Previously, a repeat property of a test case group - specified the number of times the group should be - repeated after the main test run. I.e. {repeat,N} would - case the group to execute 1+N times. To be consistent - with the behaviour of the run_test repeat option, this - has been changed. N now specifies the absolute number of - executions instead.</p> - <p> - Own Id: OTP-8689 Aux Id: seq11502 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3.6</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The Test Server parse transform did not handle bit string - comprehensions. This has been fixed.</p> - <p> - Own Id: OTP-8458 Aux Id: OTP-8311 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The tc_status value in the Config list for a test case - that has failed because of a timetrap timeout, has - changed from {tc_status,timeout} to - {tc_status,timetrap_timeout}.</p> - <p> - Own Id: OTP-8302</p> - </item> - <item> - <p>The documentation is now possible to build in an open - source environment after a number of bugs are fixed and - some features are added in the documentation build - process. </p> - <p>- The arity calculation is updated.</p> - <p>- The module prefix used in the function names for - bif's are removed in the generated links so the links - will look like - "http://www.erlang.org/doc/man/erlang.html#append_element-2" - instead of - "http://www.erlang.org/doc/man/erlang.html#erlang:append_element-2".</p> - <p>- Enhanced the menu positioning in the html - documentation when a new page is loaded.</p> - <p>- A number of corrections in the generation of man - pages (thanks to Sergei Golovan)</p> - <p>- The legal notice is taken from the xml book file so - OTP's build process can be used for non OTP - applications.</p> - <p> - Own Id: OTP-8343</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3.5</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - If the init_per_testcase/2 function fails, the test case - now gets marked and counted as auto skipped, not user - skipped (which would previously happen).</p> - <p> - Own Id: OTP-8289</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The documentation is now built with open source tools - (xsltproc and fop) that exists on most platforms. One - visible change is that the frames are removed.</p> - <p> - Own Id: OTP-8201</p> - </item> - <item> - <p> - It is now possible to fail a test case from the - end_per_testcase/2 function, by returning {fail,Reason}.</p> - <p> - Own Id: OTP-8284</p> - </item> - <item> - <p> - It is now possible to fail a test case by having the - end_tc/3 framework function return {fail,Reason} for the - test case.</p> - <p> - Own Id: OTP-8285</p> - </item> - <item> - <p> - The test_server framework API (e.g. the end_tc/3 - function) has been modified. See the test_server_ctrl - documentation for details.</p> - <p> - Own Id: OTP-8286 Aux Id: OTP-8285, OTP-8287 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - When running a suite starting with a test case group, - Test Server crashed if init_per_suite/1 exited or - returned skip. This has been fixed.</p> - <p> - Own Id: OTP-8105 Aux Id: OTP-8089 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Various updates and fixes in Common Test and Test Server.</p> - <p> - Own Id: OTP-8045 Aux Id: OTP-8089,OTP-8105,OTP-8163 </p> - </item> - <item> - <p> - Errors in coverage data collection and analysis were - difficult to detect. The logging has been improved so - that more information about e.g. imported and missing - modules is printed to the html log files.</p> - <p> - Own Id: OTP-8163 Aux Id: seq11374 </p> - </item> - <item> - <p> - The Common Test HTML overview pages have been improved. - It is now possible to see if a test case has been skipped - explicitly or because a configuration function has - failed. Also, the history page (all_runs.html) now has - scrolling text displaying the test names. The old format - (showing names as a truncated string) can still be - generated by means of the flag/option 'basic_html'.</p> - <p> - Own Id: OTP-8177</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3.2</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Various corrections and improvements of Common Test and - Test Server.</p> - <p> - Own Id: OTP-7981</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3.1</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Minor updates and corrections.</p> - <p> - Own Id: OTP-7897</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.3</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The conf case in Test Server has been extended with - properties that make it possible to execute test cases in - parallel, in sequence and in shuffled order. It is now - also possible to repeat test cases according to different - criterias. The properties can be combined, making it - possible to e.g. repeat a conf case a certain number of - times and execute the test cases in different (random) - order every time. The properties are specified in a list - in the conf case definition: {conf, Properties, InitCase, - TestCases, EndCase}. The available properties are: - parallel, sequence, shuffle, repeat, repeat_until_all_ok, - repeat_until_any_ok, repeat_until_any_fail, - repeat_until_all_fail.</p> - <p> - Own Id: OTP-7511 Aux Id: OTP-7839 </p> - </item> - <item> - <p>The test server starts Cover on nodes of the same - version as the test server itself only.</p> - <p> - Own Id: OTP-7699</p> - </item> - <item> - <p> - The Erlang mode for Emacs has been updated with new and - modified skeletons for Common Test and TS. Syntax for - test case groups in Common Test (and conf cases with - properties in TS) has been added and a new minimal Common - Test suite skeleton has been introduced.</p> - <p> - Own Id: OTP-7856</p> - </item> - </list> - </section> - -</section> -<section><title>Test_Server 3.2.4.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - The step functionality in Common Test (based on - interaction with Debugger) was broken. This has been - fixed, and some new step features have also been added. - Please see the Common Test User's Guide for details.</p> - <p> - Own Id: OTP-7800 Aux Id: seq11106 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.2.4</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Miscellaneous updates.</p> - <p> - Own Id: OTP-7527</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.2.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - When a testcase terminated due to a timetrap, io sent to - the group leader from framework:end_tc/3 (using - ct:pal/2/3 or ct:log/2/3) would cause deadlock. This has - been fixed.</p> - <p> - Own Id: OTP-7447 Aux Id: seq11010 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Various updates and improvements, plus some minor bug - fixes, have been implemented in Common Test and Test - Server.</p> - <p> - Own Id: OTP-7112</p> - </item> - <item> - <p> - It is now possible, by means of the new function - ct:abort_current_testcase/1 or - test_server_ctrl:abort_current_testcase/1, to abort the - currently executing test case.</p> - <p> - Own Id: OTP-7518 Aux Id: OTP-7112 </p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.2.2</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p><c>erlang:system_info/1</c> now accepts the - <c>logical_processors</c>, and <c>debug_compiled</c> - arguments. For more info see the, <c>erlang(3)</c> - documentation.</p> <p>The scale factor returned by - <c>test_server:timetrap_scale_factor/0</c> is now also - effected if the emulator uses a larger amount of - scheduler threads than the amount of logical processors - on the system. </p> - <p> - Own Id: OTP-7175</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.2.1</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - When init_per_suite or end_per_suite terminated due to - runtime failure, test_server failed to format the line - number information properly and crashed. This error has - now been fixed.</p> - <p> - Own Id: OTP-7091</p> - </item> - </list> - </section> - -</section> - -<section><title>Test_Server 3.2.0</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Test Server is a portable test server for automated - application testing. The server can run test suites on - local or remote targets and log progress and results to - HTML pages. The main purpose of Test Server is to act as - engine inside customized test tools. A callback interface - for such framework applications is provided.</p> - <p> - Own Id: OTP-6989</p> - </item> - </list> - </section> - -</section> - -</chapter> - |