aboutsummaryrefslogtreecommitdiffstats
path: root/lib/test_server/src/test_server.erl
AgeCommit message (Collapse)Author
2014-06-16[test_server] Fix bug introduced in 5a3c466Siri Hansen
This was detected by dialyzer - an '=' character too much had erronously been inserted.
2014-06-05Improve cover analysis via common_testSiri Hansen
This addresses several bugs in common_test (ct) when using the cover analysis mechanism: In a ct test run, one can give a cover spec file which indicates that cover analysis shall be run, including a number of modules and a number of nodes. During the ct test run, multiple jobs may be started in test_server, and when the cover option is used, test_server would cover compile and analyse all given modules for each job. This commit instead allows the compilation and analysis to be explicitly ordered by ct for each test run. This way each module will only be cover compiled and analysed once. The cover log will be located in the common_test log directory (ct_run.<timestamp>), and the "Coverage log" link in each suite.log.html will point to this file. A new button is also added in the top level ct index file, which points to the cover log. This change also reduces the need of using the 'export' and 'import' options, since there is no longer any need to accumulate cover data over multiple test_server jobs. However, these options may still be used for importing and exporting cover data in order to store or accumulate data from multiple ct test runs. The 'nodes' option was earlier only used by ct to start cover on the given nodes before starting the first test_server job. After this job was completed, test_server would stop cover completely and then start it again for the next job without any knowledge of the 'nodes' options. For the next test_server jobs cover would therefore no longer be running on these nodes. Explcit calls to ct_cover:add_nodes had to be done in order to collect data through all test_server jobs. This bug has now been solved, since cover is no longer stopped between each test_server job. Finally, ct no longer stores cover data using ct_util:set_testdata. This was earlier used by ct_cover:add_nodes to make sure no node was added twice.This did, however, cause some problems when ct and cover were out of sync. ct could belive that a node was running cover and thus reject adding this node, while in reality cover had been stopped on the node (e.g. by test_server) so no cover data was collected. ct_cover:add_nodes will now instead use cover:which_nodes to check if a node is already running.
2014-05-20Change internal format of CoverInfo in test_serverSiri Hansen
This is in preparation of a future change of the cover API between common_test and test_server.
2014-04-30Fix problem with comments getting lost when running parallel test casesPeter Andersson
OTP-11898
2014-03-12test_server: Allow init_per_ function to throw skipDan Gudmundsson
Allow all init_per_[suite|group|testcase] to throw (or exit) {skip, Reason} to explicitly ignore tests without failing. It is easier to catch it in one place instead of writing a try in each init_per_* to return {skip, Reason}.
2014-02-24Introduce appup test utilityTobias Schlager
This adds a test function similar to the app file test function existing in test_server. The code is mostly taken from the asn1, snmp and inets test suites. Low-level upgrade instruction checking is stripped. Library application's appup files get specialized treatment, since their content is almost static.
2014-02-13test_server: Allow cover to run on itselfBjörn Gustavsson
2014-02-07Fix problem with logging exits that happen in init_per_testcasePeter Andersson
2013-11-18Correct tests cases that fail because of modified eventsPeter Andersson
2013-11-18Correct various bugs related to auto_skip and groupsPeter Andersson
2013-08-28Remove the obsolete test_server_h error handlerPeter Andersson
2013-05-29Remove incorrect warning about missing end_per_testcasePeter Andersson
OTP-11052
2013-03-04[test_server] Remove unused code related to remote targetsSiri Hansen
After removing the functionality for remote target nodes in R16A, there was still some unused code left and some documentation that was not updated. This is now completed.
2013-02-04[test_server] Don't write unicode strings to latin1 log filesSiri Hansen
The unicode update of test_server for R16A introduced a few potential errors when logging to files. Sometimes ~tp or ~ts was used for formatting also when writing to files that were not opened with the {encoding,utf8} option. If then the argument contained unicode characters above 255, the file descriptor would crash. This has been corrected by the following modifications: * Since the 'unexpected_io' log file is used only when the test case HTML file is not available (e.g. between test cases), this file is now also a HTML file and as other test_server HTML logs it is always UTF-8 encoded * Since it is possible to change which information is going to which log file (with test_server_ctrl:set_levels/3), we do not have full control over which information is written to which file. This means that any printout could be written to the 'major' log file (suite.log), which was earlier encoded as latin1. To avoid crashing this file descriptor due to unicode strings, the 'major' log file is now also encoded in UTF-8 (possible incopatibility). * The cross_cover.info file is no longer a text file which can be read with file:consult/1, instead it is written as a pure binary file using term_to_binary when writing and binary_to_term when reading. * The encoding of the file named 'last_name', which only content is the path to the last run.<timestamp> directory, is now dependent on the file name mode of the VM. If file names are expected to be unicode, then the 'last_name' file is UTF-8 encoded, else it is latin1 encoded. Also, ~tp is changed back to ~p unless it is somehow likely that the argument includes strings. It is not obvious that this is the correct thing to do, but some decission had to be taken...
2013-01-25[test_server] Update test_server to handle unicodeSiri Hansen
* Use UTF-8 encoding for all HTML files, except the HTML version of the test suite generated with erl2html2:convert, which will have the same encoding as the original test suite (.erl) file. * Encode link targets in HTML files, allowing both special characters like "/", "&", "?" etc, and latin1 or unicode characters. * Use unicode modifier 't' with ~s and ~p when appropriate. * Use unicode:characters_to_list and unicode:characters_to_binary for conversion between binaries and strings instead of binary_to_list and list_to_binary.
2013-01-10Merge branch 'siri/cross-cover/OTP-9870'Siri Hansen
* siri/cross-cover/OTP-9870: [test_server] Don't use print/3 in cross_cover_analyse [common_test] Add documentation for cross cover analysis [common_test] Add test case for cross cover mechanism [test_server] Update documentation about cross cover [test_server] Add test of code cover mechanism [test_server,common_test] Fix cross cover mechansim
2013-01-09test_server: Remove support for packagesBjörn Gustavsson
2012-12-20[test_server] Stop cover on node after node is terminatedSiri Hansen
Before terminating slave nodes, test_server calls cover:flush/1 to fetch data from the node without actually stopping cover on this node. If cover is not stopped for the node and a new node with the same name is started, then cover will be started on the new node. To avoid this test_server now calls cover:stop/1 after the slave node is terminated.
2012-12-04[test_server,common_test] Fix cross cover mechansimSiri Hansen
Update the interface for cross cover analysis (collection of cover data over multiple tests) so it can be used via common_test and directly with test server. The concept of 'application' in the cross cover interface is removed and replaced with an arbitrary Tag=atom() which identifies a test run.
2012-11-16[test_server] Cancel timetrap for break in end_per_testcaseSiri Hansen
Any call to test_server:break/1 should cancel all active timetramps. In some cases, Suite:end_per_testcase/2 is executed on a different process than the test case itself, and if test_server:break/1 would be called from there, the timetraps would not be cancelled. This has been corrected.
2012-11-08Merge branch 'bjorn/ct/minor-corrections'Björn Gustavsson
* bjorn/ct/minor-corrections: test_server_io: Correct a few comments Handle ct:abort_current_testcase/1 when executing parallel groups Make sure that "Cover analysing..." is written to stdout Make sure that "Testing..." is not written if testing has finished
2012-11-08Remove stale support for remote target nodesBjörn Gustavsson
The support for remote target nodes has not worked in a long time. We are unlikely to ever need remot target node support and if we'll need it will be easier to start over from scratch.
2012-11-07Make sure that "Cover analysing..." is written to stdoutBjörn Gustavsson
If it is written with io:fwrite/2 it will be written to the unexpected_io log, where it will not be of any use.
2012-11-07[test_server] Add Config to run_test_case_msgloop/1 from the startSiri Hansen
Earlier, #st.config was set to undefined when entering this loop. Then it was updated when the test case itself started. This means that if the test case process crashed during test case initiation, the parent process would not know the Config. This commit saves the initial Config value in the parent process (run_test_case_msgloop/1) from the start.
2012-10-31Merge branch 'siri/cover-tests'Siri Hansen
* siri/cover-tests: (21 commits) [common_test] Extend timer for flushing error logger [cover] Allow reconnection if node has been disconnected or down [cover] Don't kill remote nodes when connection to main node is lost [test_server] Add option {start_cover,false} to test_server:start_node Use code:lib_dir instead of code:which to get application directory [common_test] Add test for OTP-9956 Include all kernel modules in code coverage analysis [common_test] Add test suite for code coverage support [common_test, test_server] Don't flush cover if cover is not running [common_test] Add option cover_stop [test_server] Allow cross cover analysis when testing through common_test [test_server] Start cover in test_server:wait_for_node [test_server] Multiply timers with timetrap_scale_factor when starting nodes Include all stdlib modules in code coverage analysis [test_server] Include all test_server modules in code coverage analysis Skip epp_SUITE:otp_8911 if cover is running [common_test] Start cover on slave nodes if running cover tests [common_test] Don't stop cover before stopping slave node [test_server] Don't stop cover after test is finished [cover] Add support for test_server ... OTP-10427
2012-10-30[test_server] Add option {start_cover,false} to test_server:start_nodeSiri Hansen
By default the test server will start cover on all nodes when the test is run with code coverage analysis. To make sure cover is not started on a new node, this option can be set. This can be useful if the test itself starts cover or if the new node for some reason can not run cover compiled code.
2012-10-30[common_test] Add option cover_stopSiri Hansen
By default, test_server will always stop cover (and thus load back the non cover compiled original beam files) after a test with code coverage analysis is completed. The new option allows the common_test user to specify that cover shall not be stopped. This can be useful if there are processes still running old code, i.e. processes that have not done any fully qualified function call after the cover compilation, since loading the original code then will kill those processes. This is only recommended if the erlang node is to be stopped after the test run, or if cover can be manually stopped.
2012-10-30[test_server] Start cover in test_server:wait_for_nodeSiri Hansen
Without this change, there is no support in test_server for starting cover on a node started with option {wait,false}.
2012-10-30[test_server] Don't stop cover after test is finishedSiri Hansen
This commit removes all calls to cover:stop from test_server. This is to avoid crash in some processes due to old code. The correction shall be further refined for general use. There will, for instance, be an option to turn off cover:stop/0 after completed test run, so the default behaviour is backwards compatible.
2012-10-26Simplify managment of test_server_locBjörn Gustavsson
Before line numbers were included in exceptions (in R15), there were parse transforms and macros that would keep the test_server_loc process dictionary variable updated; therefore there is a mod_loc/1 function that will normalize the representation of locations. Simplify the code as following: * 'test_server_loc' should always contain a list with one or more tuples. ({M,F} or {M,F,Line}) * At the beginning of each test case, set 'test_server_loc' to [{Module,Func}] of the the currently executing test case. * Stop updating 'test_server_loc', except when an exception occurs. (It used to be updated when running 'init_per_testcase' and so on.) * Remove the mod_loc/1 function.
2012-10-26Keep the information about the current test case consistentBjörn Gustavsson
Three pieces of information could be out of sync in testcase supervisor process at the time when a timetrap occurred: * test_server_loc (process dictionary) - may indicate that a framework function is executing * test_server_init_or_end_conf (process dictionary) - indicates whether init_per_testcase or end_per_testcase is executing * The current configuration (#st.config) - set using a synchronous call There could be in a crash in spawn_fw_call/7 because the current configuration was not defined when it was expected to. To avoid the problem, introduce set_tc_state/2 (and a corresponding message) to allow setting both an indication what the testcase is executing (e.g. init_per_testcase, framework call, and so on), and the current configuration. Use only that information to handle a timetrap timeout (and aborted testcase and the other reasons for the testcase process to terminate). Completely remove test_server_init_or_end_conf.
2012-10-26test_server: Eliminate the Loc argument for do_end_tc_call()Björn Gustavsson
The Loc argument was use to determine the name of the currently executing test case, in case that the M and F arguments did not specify a test case. Since a previous commit makes sure that the M and F arguments *are* valied, we can eliminate the Loc argument.
2012-10-26Refactor the handling of the make_priv_dir messageBjörn Gustavsson
The code is not actually shorter, but it avoids duplicating the code for producing an error tuple when theres is no priv_dir tuple in the config data.
2012-10-26test_server: Refactor run_test_case_msgloop()Björn Gustavsson
Keeping all state as separate function arguments does not make it easy to add more pieces of data to the state. Case in point is the CurrConf argument where up to three separate items have been shoehorned in. Instead of the arguments, introduce a state record to hold all of the different pieces of state (taking care to separate the former CurrConf variable into separate fields in the record). While at it, fix a bug. The name of the currently executing test case (module and function) used to be stored in a tuple in the CurrConf variable, and it would only be available after 'set_curr_conf' message has been received. Depending on timing, it was possible in rare circumstances for an EXIT signal to arrive before the 'set_curr_conf' message. Avoid this problem by putting the current test case name into the record from the beginning. That also means that we can eliminate the workaround of getting the currently executing test case from the stack trace using the get_mf/1.
2012-10-23Introduce test_server_io and test_server_glBjörn Gustavsson
2012-10-08test_server: Refactor timetrap_scale_factor/0Björn-Egil Dahlberg
2012-08-23Merge branch 'maint'Peter Andersson
2012-08-23Solve problems with verbosity levels and parallel test casesPeter Andersson
2012-08-23Merge branch 'maint'Peter Andersson
2012-08-23Merge remote branch 'origin/peppe/common_test/break_and_continue' into maintPeter Andersson
* origin/peppe/common_test/break_and_continue: Fix error in documentation Implement support for test case execution break/continue Conflicts: lib/common_test/src/ct.erl lib/common_test/src/ct_run.erl OTP-10127 OTP-10172
2012-08-23Merge branch 'maint'Peter Andersson
2012-08-08Implement support for test case execution break/continuePeter Andersson
2012-07-19test_server: Remove VxWorksBjörn-Egil Dahlberg
2012-07-13Implement verbosity levels and parameter for log printout importancePeter Andersson
2012-03-30Update copyright yearsBjörn-Egil Dahlberg
2012-03-22Fix various problems with the user timetrap implementationPeter Andersson
2012-03-21Implement support for user controlled timetrapsPeter Andersson
OTP-9988
2012-03-15Merge remote branch 'origin/peppe/common_test/otp-9666' into maintPeter Andersson
* origin/peppe/common_test/otp-9666: Set correct group leader for end_per_testcase process after tc timeout OTP-9666
2012-03-14Set correct group leader for end_per_testcase process after tc timeoutPeter Andersson
Also make sure that warnings about failure or timeout of end_per_testcase (after a test case timeout) get printed in the test case log file. OTP-9666
2012-03-13Make it possible to use unique priv_dir namesPeter Andersson
Introduce the optional feature to have Test Server generate priv_dir directory names that are unique for each test case or config function. The name of the option/flag is 'unique_priv_dir' and it can be set to value 'auto' or 'manual'. If auto, Test Server creates each priv_dir automatically (can be expensive in case of many and/or repeated cases). If manual, the user needs to create the priv_dir explicitly by calling ct:make_priv_dir/0.