diff options
Diffstat (limited to 'lib/common_test/doc/src/dependencies_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/dependencies_chapter.xml | 382 |
1 files changed, 194 insertions, 188 deletions
diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml index fb758d90df..d96a41473f 100644 --- a/lib/common_test/doc/src/dependencies_chapter.xml +++ b/lib/common_test/doc/src/dependencies_chapter.xml @@ -33,217 +33,220 @@ <section> <title>General</title> <p>When creating test suites, it is strongly recommended to not - create dependencies between test cases, i.e. letting test cases + create dependencies between test cases, that is, letting test cases depend on the result of previous test cases. There are various - reasons for this, for example:</p> + reasons for this, such as, the following:</p> - <list> + <list type="bulleted"> <item>It makes it impossible to run test cases individually.</item> - <item>It makes it impossible to run test cases in different order.</item> - <item>It makes debugging very difficult (since a fault could be + <item>It makes it impossible to run test cases in a different order.</item> + <item>It makes debugging difficult (as a fault can be the result of a problem in a different test case than the one failing).</item> - <item>There exists no good and explicit ways to declare dependencies, so - it may be very difficult to see and understand these in test suite + <item>There are no good and explicit ways to declare dependencies, so + it can be difficult to see and understand these in test suite code and in test logs.</item> - <item>Extending, restructuring and maintaining test suites with + <item>Extending, restructuring, and maintaining test suites with test case dependencies is difficult.</item> </list> <p>There are often sufficient means to work around the need for test case dependencies. Generally, the problem is related to the state of - the system under test (SUT). The action of one test case may alter the state - of the system and for some other test case to run properly, the new state + the System Under Test (SUT). The action of one test case can change the + system state. For some other test case to run properly, this new state must be known.</p> <p>Instead of passing data between test cases, it is recommended that the test cases read the state from the SUT and perform assertions - (i.e. let the test case run if the state is as expected, otherwise reset or fail) - and/or use the state to set variables necessary for the test case to execute - properly. Common actions can often be implemented as library functions for - test cases to call to set the SUT in a required state. (Such common actions - may of course also be separately tested if necessary, to ensure they are - working as expected). It is sometimes also possible, but not always desirable, - to group tests together in one test case, i.e. let a test case perform a - "scenario" test (a test that consists of subtests).</p> - - <p>Consider for example a server application under test. The following + (that is, let the test case run if the state is as expected, otherwise reset or fail). + It is also recommended to use the state to set variables necessary for the + test case to execute properly. Common actions can often be implemented as + library functions for test cases to call to set the SUT in a required state. + (Such common actions can also be separately tested, if necessary, + to ensure that they work as expected). It is sometimes also possible, + but not always desirable, to group tests together in one test case, that is, + let a test case perform a "scenario" test (a test consisting of subtests).</p> + + <p>Consider, for example, a server application under test. The following functionality is to be tested:</p> - <list> - <item>Starting the server.</item> - <item>Configuring the server.</item> - <item>Connecting a client to the server.</item> - <item>Disconnecting a client from the server.</item> - <item>Stopping the server.</item> + <list type="bulleted"> + <item>Starting the server</item> + <item>Configuring the server</item> + <item>Connecting a client to the server</item> + <item>Disconnecting a client from the server</item> + <item>Stopping the server</item> </list> - <p>There are obvious dependencies between the listed functions. We can't configure - the server if it hasn't first been started, we can't connect a client until - the server has been properly configured, etc. If we want to have one test - case for each of the functions, we might be tempted to try to always run the + <p>There are obvious dependencies between the listed functions. The server cannot + be configured if it has not first been started, a client connot be connectd until + the server is properly configured, and so on. If we want to have one test + case for each function, we might be tempted to try to always run the test cases in the stated order and carry possible data (identities, handles, - etc) between the cases and therefore introduce dependencies between them. - To avoid this we could consider starting and stopping the server for every test. - We would implement the start and stop action as common functions that may be - called from init_per_testcase and end_per_testcase. (We would of course test - the start and stop functionality separately). The configuration could perhaps also - be implemented as a common function, maybe grouped with the start function. - Finally the testing of connecting and disconnecting a client may be grouped into - one test case. The resulting suite would look something like this:</p> - + and so on) between the cases and therefore introduce dependencies between them.</p> + + <p>To avoid this, we can consider starting and stopping the server for every test. + We can thus implement the start and stop action as common functions to be + called from + <seealso marker="common_test_app#Module:init_per_testcase-2"><c>init_per_testcase</c></seealso> and + <seealso marker="common_test_app#Module:end_per_testcase-2"><c>end_per_testcase</c></seealso>. + (Remember to test the start and stop functionality separately.) + The configuration can also be implemented as a common function, maybe grouped + with the start function. Finally, the testing of connecting and disconnecting a + client can be grouped into one test case. The resulting suite can look as + follows:</p> <pre> - -module(my_server_SUITE). - -compile(export_all). - -include_lib("ct.hrl"). + -module(my_server_SUITE). + -compile(export_all). + -include_lib("ct.hrl"). + + %%% init and end functions... - %%% init and end functions... + suite() -> [{require,my_server_cfg}]. - suite() -> [{require,my_server_cfg}]. + init_per_testcase(start_and_stop, Config) -> + Config; - init_per_testcase(start_and_stop, Config) -> - Config; + init_per_testcase(config, Config) -> + [{server_pid,start_server()} | Config]; - init_per_testcase(config, Config) -> - [{server_pid,start_server()} | Config]; + init_per_testcase(_, Config) -> + ServerPid = start_server(), + configure_server(), + [{server_pid,ServerPid} | Config]. - init_per_testcase(_, Config) -> - ServerPid = start_server(), - configure_server(), - [{server_pid,ServerPid} | Config]. + end_per_testcase(start_and_stop, _) -> + ok; - end_per_testcase(start_and_stop, _) -> - ok; + end_per_testcase(_, _) -> + ServerPid = ?config(server_pid), + stop_server(ServerPid). - end_per_testcase(_, _) -> - ServerPid = ?config(server_pid), - stop_server(ServerPid). + %%% test cases... - %%% test cases... + all() -> [start_and_stop, config, connect_and_disconnect]. - all() -> [start_and_stop, config, connect_and_disconnect]. + %% test that starting and stopping works + start_and_stop(_) -> + ServerPid = start_server(), + stop_server(ServerPid). - %% test that starting and stopping works - start_and_stop(_) -> - ServerPid = start_server(), - stop_server(ServerPid). + %% configuration test + config(Config) -> + ServerPid = ?config(server_pid, Config), + configure_server(ServerPid). - %% configuration test - config(Config) -> - ServerPid = ?config(server_pid, Config), - configure_server(ServerPid). + %% test connecting and disconnecting client + connect_and_disconnect(Config) -> + ServerPid = ?config(server_pid, Config), + {ok,SessionId} = my_server:connect(ServerPid), + ok = my_server:disconnect(ServerPid, SessionId). - %% test connecting and disconnecting client - connect_and_disconnect(Config) -> - ServerPid = ?config(server_pid, Config), - {ok,SessionId} = my_server:connect(ServerPid), - ok = my_server:disconnect(ServerPid, SessionId). + %%% common functions... - %%% common functions... + start_server() -> + {ok,ServerPid} = my_server:start(), + ServerPid. - start_server() -> - {ok,ServerPid} = my_server:start(), - ServerPid. + stop_server(ServerPid) -> + ok = my_server:stop(), + ok. - stop_server(ServerPid) -> - ok = my_server:stop(), - ok. + configure_server(ServerPid) -> + ServerCfgData = ct:get_config(my_server_cfg), + ok = my_server:configure(ServerPid, ServerCfgData), + ok.</pre> - configure_server(ServerPid) -> - ServerCfgData = ct:get_config(my_server_cfg), - ok = my_server:configure(ServerPid, ServerCfgData), - ok. - </pre> </section> <section> <marker id="save_config"></marker> - <title>Saving configuration data</title> + <title>Saving Configuration Data</title> - <p>There might be situations where it is impossible, or infeasible at least, to - implement independent test cases. Maybe it is simply not possible to read the - SUT state. Maybe resetting the SUT is impossible and it takes much too long + <p>Sometimes it is impossible, or infeasible, to + implement independent test cases. Maybe it is not possible to read the + SUT state. Maybe resetting the SUT is impossible and it takes too long time to restart the system. In situations where test case dependency is necessary, CT offers a structured way to carry data from one test case to the next. The - same mechanism may also be used to carry data from one test suite to the next.</p> + same mechanism can also be used to carry data from one test suite to the next.</p> <p>The mechanism for passing data is called <c>save_config</c>. The idea is that - one test case (or suite) may save the current value of Config - or any list of - key-value tuples - so that it can be read by the next executing test case - (or test suite). The configuration data is not saved permanently but can only - be passed from one case (or suite) to the next.</p> + one test case (or suite) can save the current value of <c>Config</c>, or any list of + key-value tuples, so that the next executing test case (or test suite) can read it. + The configuration data is not saved permanently but can only be passed from one + case (or suite) to the next.</p> - <p>To save <c>Config</c> data, return the tuple:</p> + <p>To save <c>Config</c> data, return tuple <c>{save_config,ConfigList}</c> + from <c>end_per_testcase</c> or from the main test case function.</p> - <p><c>{save_config,ConfigList}</c></p> - - <p>from <c>end_per_testcase</c> or from the main test case function. To read data - saved by a previous test case, use the <c>config</c> macro with a - <c>saved_config</c> key:</p> + <p>To read data saved by a previous test case, use macro <c>config</c> with a + <c>saved_config</c> key as follows:</p> <p><c>{Saver,ConfigList} = ?config(saved_config, Config)</c></p> <p><c>Saver</c> (<c>atom()</c>) is the name of the previous test case (where the - data was saved). The <c>config</c> macro may be used to extract particular data + data was saved). The <c>config</c> macro can be used to extract particular data also from the recalled <c>ConfigList</c>. It is strongly recommended that <c>Saver</c> is always matched to the expected name of the saving test case. - This way problems due to restructuring of the test suite may be avoided. Also it - makes the dependency more explicit and the test suite easier to read and maintain.</p> + This way, problems because of restructuring of the test suite can be avoided. + Also, it makes the dependency more explicit and the test suite easier to read + and maintain.</p> <p>To pass data from one test suite to another, the same mechanism is used. The data - should be saved by the <c>end_per_suite</c> function and read by <c>init_per_suite</c> + is to be saved by finction + <seealso marker="common_test_app#Module:end_per_suite-1"><c>end_per_suite</c></seealso> + and read by function + <seealso marker="common_test_app#Module:init_per_suite-1"><c>init_per_suite</c></seealso> in the suite that follows. When passing data between suites, <c>Saver</c> carries the name of the test suite.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> - -module(server_b_SUITE). - -compile(export_all). - -include_lib("ct.hrl"). - - %%% init and end functions... - - init_per_suite(Config) -> - %% read config saved by previous test suite - {server_a_SUITE,OldConfig} = ?config(saved_config, Config), - %% extract server identity (comes from server_a_SUITE) - ServerId = ?config(server_id, OldConfig), - SessionId = connect_to_server(ServerId), - [{ids,{ServerId,SessionId}} | Config]. - - end_per_suite(Config) -> - %% save config for server_c_SUITE (session_id and server_id) - {save_config,Config} - - %%% test cases... - - all() -> [allocate, deallocate]. - - allocate(Config) -> - {ServerId,SessionId} = ?config(ids, Config), - {ok,Handle} = allocate_resource(ServerId, SessionId), - %% save handle for deallocation test - NewConfig = [{handle,Handle}], - {save_config,NewConfig}. - - deallocate(Config) -> - {ServerId,SessionId} = ?config(ids, Config), - {allocate,OldConfig} = ?config(saved_config, Config), - Handle = ?config(handle, OldConfig), - ok = deallocate_resource(ServerId, SessionId, Handle). - </pre> - - <p>It is also possible to save <c>Config</c> data from a test case that is to be - skipped. To accomplish this, return the following tuple:</p> - - <p><c>{skip_and_save,Reason,ConfigList}</c></p> - - <p>The result will be that the test case is skipped with <c>Reason</c> printed to - the log file (as described in previous chapters), and <c>ConfigList</c> is saved - for the next test case. <c>ConfigList</c> may be read by means of - <c>?config(saved_config, Config)</c>, as described above. <c>skip_and_save</c> - may also be returned from <c>init_per_suite</c>, in which case the saved data can + -module(server_b_SUITE). + -compile(export_all). + -include_lib("ct.hrl"). + + %%% init and end functions... + + init_per_suite(Config) -> + %% read config saved by previous test suite + {server_a_SUITE,OldConfig} = ?config(saved_config, Config), + %% extract server identity (comes from server_a_SUITE) + ServerId = ?config(server_id, OldConfig), + SessionId = connect_to_server(ServerId), + [{ids,{ServerId,SessionId}} | Config]. + + end_per_suite(Config) -> + %% save config for server_c_SUITE (session_id and server_id) + {save_config,Config} + + %%% test cases... + + all() -> [allocate, deallocate]. + + allocate(Config) -> + {ServerId,SessionId} = ?config(ids, Config), + {ok,Handle} = allocate_resource(ServerId, SessionId), + %% save handle for deallocation test + NewConfig = [{handle,Handle}], + {save_config,NewConfig}. + + deallocate(Config) -> + {ServerId,SessionId} = ?config(ids, Config), + {allocate,OldConfig} = ?config(saved_config, Config), + Handle = ?config(handle, OldConfig), + ok = deallocate_resource(ServerId, SessionId, Handle).</pre> + + <p>To save <c>Config</c> data from a test case that is to be + skipped, return tuple + <c>{skip_and_save,Reason,ConfigList}</c>.</p> + + <p>The result is that the test case is skipped with <c>Reason</c> printed to + the log file (as described earlier) and <c>ConfigList</c> is saved + for the next test case. <c>ConfigList</c> can be read using + <c>?config(saved_config, Config)</c>, as described earlier. <c>skip_and_save</c> + can also be returned from <c>init_per_suite</c>. In this case, the saved data can be read by <c>init_per_suite</c> in the suite that follows.</p> </section> @@ -251,60 +254,63 @@ <marker id="sequences"></marker> <title>Sequences</title> - <p>It is possible that test cases depend on each other so that - if one case fails, the following test(s) should not be executed. + <p>Sometimes test cases depend on each other so that + if one case fails, the following tests are not to be executed. Typically, if the <c>save_config</c> facility is used and a test case that is expected to save data crashes, the following - case can not run. CT offers a way to declare such dependencies, + case cannot run. <c>Common Test</c> offers a way to declare such dependencies, called sequences.</p> <p>A sequence of test cases is defined as a test case group - with a <c>sequence</c> property. Test case groups are defined by - means of the <c>groups/0</c> function in the test suite (see the - <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso> - chapter for details).</p> - - <p>For example, if we would like to make sure that if <c>allocate</c> - in <c>server_b_SUITE</c> (above) crashes, <c>deallocate</c> is skipped, - we may define a sequence like this:</p> + with a <c>sequence</c> property. Test case groups are defined + through function <c>groups/0</c> in the test suite (for details, see section + <seealso marker="write_test_chapter#test_case_groups">Test Case Groups</seealso>.</p> + + <p>For example, to ensure that if <c>allocate</c> + in <c>server_b_SUITE</c> crashes, <c>deallocate</c> is skipped, + the following sequence can be defined:</p> <pre> - groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre> + groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre> - <p>Let's also assume the suite contains the test case <c>get_resource_status</c>, - which is independent of the other two cases, then the <c>all</c> function could - look like this:</p> + <p>Assume that the suite contains the test case <c>get_resource_status</c> + that is independent of the other two cases, then function <c>all</c> can + look as follows:</p> <pre> - all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre> + all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre> <p>If <c>alloc</c> succeeds, <c>dealloc</c> is also executed. If <c>alloc</c> fails - however, <c>dealloc</c> is not executed but marked as SKIPPED in the html log. - <c>get_resource_status</c> will run no matter what happens to the <c>alloc_and_dealloc</c> + however, <c>dealloc</c> is not executed but marked as <c>SKIPPED</c> in the HTML log. + <c>get_resource_status</c> runs no matter what happens to the <c>alloc_and_dealloc</c> cases.</p> - <p>Test cases in a sequence will be executed in order until they have all succeeded or - until one case fails. If one fails, all following cases in the sequence are skipped. - The cases in the sequence that have succeeded up to that point are reported as successful - in the log. An arbitrary number of sequences may be specified. Example:</p> + <p>Test cases in a sequence are executed in order until all succeed or + one fails. If one fails, all following cases in the sequence are skipped. + The cases in the sequence that have succeeded up to that point are reported as + successful in the log. Any number of sequences can be specified.</p> + <p><em>Example:</em></p> <pre> - groups() -> [{scenarioA, [sequence], [testA1, testA2]}, - {scenarioB, [sequence], [testB1, testB2, testB3]}]. - - all() -> [test1, - test2, - {group,scenarioA}, - test3, - {group,scenarioB}, - test4].</pre> - - <p>It is possible to have sub-groups in a sequence group. Such sub-groups can have - any property, i.e. they are not required to also be sequences. If you want the status - of the sub-group to affect the sequence on the level above, return - <c>{return_group_result,Status}</c> from <c>end_per_group/2</c>, as described in the - <seealso marker="write_test_chapter#repeated_groups">Repeated groups</seealso> - chapter. A failed sub-group (<c>Status == failed</c>) will cause the execution of a + groups() -> [{scenarioA, [sequence], [testA1, testA2]}, + {scenarioB, [sequence], [testB1, testB2, testB3]}]. + + all() -> [test1, + test2, + {group,scenarioA}, + test3, + {group,scenarioB}, + test4].</pre> + + <p>A sequence group can have subgroups. Such subgroups can have + any property, that is, they are not required to also be sequences. If you want the + status of the subgroup to affect the sequence on the level above, return + <c>{return_group_result,Status}</c> from + <seealso marker="common_test_app#Module:end_per_group-2"><c>end_per_group/2</c></seealso>, + as described in section + <seealso marker="write_test_chapter#repeated_groups">Repeated Groups</seealso> + in Writing Test Suites. + A failed subgroup (<c>Status == failed</c>) causes the execution of a sequence to fail in the same way a test case does.</p> </section> </chapter> |