aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/test
AgeCommit message (Collapse)Author
2017-05-04Update copyright yearRaimo Niskanen
2017-03-20[ct] Add 'keep_logs' optionSiri Hansen
If setting the value for this option to an integer, N, common_test will remove all ct_run.* directories in the current log directory, except the N newest. The default value for the 'keep_logs' option is 'all', which means that no logs will be deleted. 'keep_logs' can be used in combination with refresh_logs, or in a normal common_test test run.
2017-03-06Merge branch 'siri/ct_hooks/callbacks-on-skip/OTP-10599' into maintSiri Hansen
* siri/ct_hooks/callbacks-on-skip/OTP-10599: [ct] Update built-in ct hooks with new Suite parameter [cth_surefire] Handle skips from test spec [ct] Remove excessive skipped/failed tag in hook function [ct] Send tc_start event on force_stop and failed sequence [ct] Correctly handle process died in init and end_per_testcase [ct] Fix faulty hook callbacks for skipped tests Add dummy end_per_suite/1 [ct] Fix hooks and fail when one of init/end_per_* does not exit [ct] Add Suite argument to hook callback functions [ct] Fix function_clause in ct_framework when hook function crashes
2017-02-22Merge branch 'siri/ct/get_dirs_from_testspec/OTP-14132' into maintSiri Hansen
* siri/ct/get_dirs_from_testspec/OTP-14132: [ct] Add ct_testspec:get_tests/1
2017-02-21[cth_surefire] Handle skips from test specSiri Hansen
When a test case is skipped from the test spec, there is no call to the common_test hook functions 'pre_init_per_testcase' and 'post_end_per_testcase'. This causes cth_surefire to erroneously mark the previous test case as skipped in the xml report. The actually skipped test case is not present in the xml report at all. This is now corrected. OTP-14129
2017-02-20[ct] Remove excessive skipped/failed tag in hook functionSiri Hansen
When skipping or failing in a configuration function, the configuration function itself would get {auto_skipped,Reason}, {skipped,Reason} or {failed,Reason} in the hook callbacks on_tc_skip or on_tc_fail. The other test cases that were skipped as a result of this would only get Reason in the hook callbacks. This is now corrected.
2017-02-20[ct] Send tc_start event on force_stop and failed sequenceSiri Hansen
If test cases were skipped because of option {force_stop,skip_rest} or because of a failed sequence, then no tc_start event would be sent, only only tc_done. This is now corrected.
2017-02-20[ct] Correctly handle process died in init and end_per_testcaseSiri Hansen
Documentation says that a test case is skipped if init_per_testcase fails. This is correct if init_per_testcase fails in a normal way, but if it gets an exit signal causing the process to die, then the case was earlier marked as failed with reason {skip,...}. This is now corrected, so the case is auto skipped. If end_per_testcase crashes, it does not change the result of a test case, it only adds a warning in the comment field. But if it gets an exit signal causing the process to die while running end_per_testcase, then the case was earlier marked as failed in addition to this warning. This is now corrected, so the result of the test case is not changed by an exit signal during end_per_testcase.
2017-02-20[ct] Fix faulty hook callbacks for skipped testsSiri Hansen
The following bugs are corrected: * Tests that were skipped before calling pre_init_per_* got faulty calls to the corresponding post_init_per_*. E.g. if a test was skipped because suite/0 returned a 'require' statement that was not fulfilled, then post_init_per_suite would be called, even though pre_init_per_suite and init_per_suite were not called. * Tests that were skipped before or in init_per_testcase got faulty calls to pre_end_per_testcase and post_end_per_testcase (end_per_testcase is not called in these situations). Test are added to make sure that the expected callbacks, and only those, are called when tests are skipped in different ways. Conflicts: lib/common_test/test/ct_hooks_SUITE.erl
2017-02-20[ct] Fix hooks and fail when one of init/end_per_* does not exitSiri Hansen
The following bugs are corrected: - if init_per_suite is exported from a test suite, but not end_per_suite, then pre/post_end_per_suite will be called with Suite=ct_framework instead of the correct suite name. - if end_per_group is exported from a suite, but not init_per_group, then end_per_group is never called. According to the documentation, if implementing an init config function, you must also implement the end config function, so the two scenarios above are really not allowed. To make this more visible, common_test will now mark the non-exported config function as failed with reason 'undef' if the other function is exported. For example, if init_per_suite is exported, but not end_per_suite, then end_per_suite will be marked as failed with reason undef. (If none of them exist, then they will both be marked as passed since the default functions in ct_framework are called instead.) All hook functions are always called with the correct suite name, i.e. never with Suite=ct_framework. Conflicts: lib/common_test/test/ct_hooks_SUITE.erl
2017-02-20[ct] Add Suite argument to hook callback functionsSiri Hansen
An extra argument, Suite, is added as the first argument to each of the following hook callback functions: - pre_init_per_group - post_init_per_group - pre_end_per_group - post_end_per_group - pre_init_per_testcase - post_init_per_testcase - pre_end_per_testcase - post_end_per_testcase - on_tc_fail - on_tc_skip For backwards compatibility, if the new function is not exported from a hook callback module, common_test will fall back to the old interface and call the function without the Suite argument. The reason for adding the new argument is that if a test suite is skipped by a 'skip_suites' statement in the test specification, then there will be no call to pre/post_init_per_suite, and thus the hook has no other way of knowing which Suite is skipped when it gets the on_tc_skip callback. The other callbacks are updated for symmetry.
2017-01-17Merge branch 'egil/cuddle-tests' into maintBjörn-Egil Dahlberg
* egil/cuddle-tests: stdlib: Increase timetrap for rand_SUITE common_test: Increase timetrap for cth_hooks_SUITE compiler: Increase timetrap timeouts for lc_SUITE
2017-01-11[ct] Add ct_testspec:get_tests/1Siri Hansen
This API function is used by rebar3 instead of implementing test spec parsing again.
2016-12-20[ct] Add tests for .app and .appup filesSiri Hansen
2016-12-02Add heading option to log functionsPeter Andersson
2016-11-11common_test: Increase timetrap for cth_hooks_SUITEBjörn-Egil Dahlberg
* valgrind needs a lot of time
2016-08-30make, ct_make: Handle warning attributes in source filesBjörn Gustavsson
epp learned to handle the -warning() directive in 14d72f02, but make and ct_make were not updated to expect a {warning,_} return value from epp:parse_erl_form/1.
2016-07-08[ct test] Extent node startup timeouts in netconfc_remote_SUITESiri Hansen
2016-07-08[ct test] Monitor netconf server during transactionSiri Hansen
Netconf client tests fail every now and then with 'table_trans_timeout' with no obvious reason. A monitor of the maint netconf server process is now added - in case this process crashes during the transaction.
2016-06-22[ct test] Skip system time related tests on TimeWarpingOSSiri Hansen
Tests using system time can be skipped on test hosts set up to do a lot of time warps.
2016-06-18[ct test] Catch call to ssh:start/0Siri Hansen
The ssh application is not installed on all test hosts. Catching the call to ssh:start/0 allows the the netconf tests to be nicely skipped if ssh does not exist.
2016-06-15Merge branch 'siri/cuddle-master'Siri Hansen
* siri/cuddle-master: Add short sleep when killing slave nodes [ct test] Do reload_config instead of get_config
2016-06-14Merge branch 'peppe/common_test/OTP-13615'Peter Andersson
* peppe/common_test/OTP-13615: Fix problem with incorrect type of timestamps OTP-13615
2016-06-13Fix problem with incorrect type of timestampsPeter Andersson
OTP-13615
2016-06-09[ct test] Do reload_config instead of get_configSiri Hansen
The test case config_dynamic_SUITE:test_localtime_update sometimes failed on slow machines, since the initial value of 'localtime' is fetched with ct:get_config instead of ct:reload_config. The value is then from the test run start, not from the time of the ct:get_config call. This is now corrected.
2016-06-09Merge branch 'siri/cuddle-master'Siri Hansen
* siri/cuddle-master: [ct test] Wait for event receiver to be unregistered [ct test] Flush messages from old client after close in telnet server [ct test] Use monitor instead of ct:sleep [ct test] Check that crypto is loaded, or skip netconf tests [ct test] Reduce amount of data in netconfc1_SUITE:get_a_lot
2016-06-09[ct test] Wait for event receiver to be unregisteredSiri Hansen
To avoid badarg when two tests are run within the same test case.
2016-06-09[ct test] Flush messages from old client after close in telnet serverSiri Hansen
Since the same erlang process is used for subsequent telnet connections, messages from old clients must be flushed after close.
2016-06-09[ct test] Use monitor instead of ct:sleepSiri Hansen
ct_gen_conn_SUITE_data/conn_SUITE.erl uses a lot of ct:sleep(100) when waiting for processes to exit. This commit removes all the sleeps and introducess monitor instead.
2016-06-09[ct test] Check that crypto is loaded, or skip netconf testsSiri Hansen
Tests for ct_netconfc are failed instead of skipped if crypto can not be loaded. This is now corrected.
2016-06-09[ct test] Reduce amount of data in netconfc1_SUITE:get_a_lotSiri Hansen
This test case often fails with table_trans_timeout in the netconf server (ns.erl) on slow machines. The amount of data was originally reduced by commit 976214f8d738d4852348496df79f84264d899aba, but was faulty reverted by commit 171d7e2a161ef9270240aff0fa15a285df21c1ef. It is now reduced again...
2016-06-08Merge branch 'peppe/common_test/log_tests'Peter Andersson
* peppe/common_test/log_tests: Write suite that tests the API for printing to log
2016-06-08Merge branch 'peppe/common_test/fix_failing_tests'Peter Andersson
* peppe/common_test/fix_failing_tests: Make sure test node shuts down before skipping suite Measure file i/o overhead and skip test if the speed is too slow Skip test cases if cover or debug is running
2016-06-07Write suite that tests the API for printing to logPeter Andersson
2016-06-07Make sure test node shuts down before skipping suitePeter Andersson
2016-06-06Measure file i/o overhead and skip test if the speed is too slowPeter Andersson
2016-06-01Merge branch 'siri/cuddle-master'Siri Hansen
* siri/cuddle-master: Change ct:sleep to timer:sleep where scaling is not wanted Don't generate ssh key files in netconf client test Ignore data received in break mode in test telnet_server Wait for process to die before next call Retry ct_telnet:get_data if no data is received after short wait Don't throw rest of line when NOP is received in test telnet_server Change ct:sleep to timer:sleep in telnet_server
2016-06-01Skip test cases if cover or debug is runningPeter Andersson
2016-05-26Change ct:sleep to timer:sleep where scaling is not wantedSiri Hansen
2016-05-26Don't generate ssh key files in netconf client testSiri Hansen
Generating the dsa files can be very slow on some machines. Use hardcoded files instead.
2016-05-26Ignore data received in break mode in test telnet_serverSiri Hansen
The only command handled in break mode is 'q' = 'quit break mode'. Other data would earlier cause a function_clause exception. Other data could e.g. be a NOP poll (keep alive). To fix the problem, any data received in break mode, except 'q', will now be ignored.
2016-05-25Retry ct_telnet:get_data if no data is received after short waitSiri Hansen
ct_telnet_own_server_SUITE:large_string tests that the client can receive a chopped up string. To make sure the right thing is tested, a SHORT timer is used before calling ct_telnet:get_data, but on some slow machines the the timer is too short to allow data to be received. To overcome this, the test now re-tries ct_telnet:get_data a few times before giving up.
2016-05-20Don't throw rest of line when NOP is received in test telnet_serverSiri Hansen
If telnet command NOP (No Operation) was received in the same tcp package as other data, then the rest of the data would be regarded further telnet commands (to proceed IAC, Interprete As Command) and would never get to telnet_server:do_handle_data/2. This is now corrected.
2016-05-04Merge branch 'maint-18'Henrik Nord
Conflicts: OTP_VERSION lib/common_test/test/ct_hooks_SUITE_data/cth/tests/ct_update_config_SUITE.erl lib/common_test/vsn.mk
2016-05-04Change ct:sleep to timer:sleep in telnet_serverSiri Hansen
The telnet_server is run on the main test node which will scale and multiply timers on some test hosts. The other side of the telnet test (the client) is run on the slave, which does not inherit timer scaling and multiplication. Therefore, it is better to use timer:sleep in the server.
2016-05-04Merge branch 'siri/cuddle-master'Siri Hansen
* siri/cuddle-master: Extend timer in test Update crashdump_viewer_SUITE with new release numbers Set longer timetrap when creating dsa files Check that ssh application exists before starting netconf tests Improve error control when starting ssh in netconf test
2016-05-03Merge branch 'peppe/ct_misc_18_patches' into maint-18Erlang/OTP
* peppe/ct_misc_18_patches: Add flag/option for disabling the character escaping functionality Fix bug using the wrong lists search function Fix bug with clashing timestamp values Fix problem with stylesheet tags getting escaped Skip pre/post test IO suite if cover or debug is running Tweak pre_post_io test case to run without failing Fix various log related problems
2016-05-02Fix bug with clashing timestamp valuesPeter Andersson
2016-05-02Skip pre/post test IO suite if cover or debug is runningPeter Andersson
OTP-13535 The return value of ct:get_timetrap_info/0 has been modified.
2016-05-02Tweak pre_post_io test case to run without failingPeter Andersson