aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src/dependencies_chapter.xml
diff options
context:
space:
mode:
Diffstat (limited to 'lib/common_test/doc/src/dependencies_chapter.xml')
-rw-r--r--lib/common_test/doc/src/dependencies_chapter.xml309
1 files changed, 309 insertions, 0 deletions
diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml
new file mode 100644
index 0000000000..8aad552285
--- /dev/null
+++ b/lib/common_test/doc/src/dependencies_chapter.xml
@@ -0,0 +1,309 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+ <header>
+ <copyright>
+ <year>2006</year><year>2009</year>
+ <holder>Ericsson AB. All Rights Reserved.</holder>
+ </copyright>
+ <legalnotice>
+ The contents of this file are subject to the Erlang Public License,
+ Version 1.1, (the "License"); you may not use this file except in
+ compliance with the License. You should have received a copy of the
+ Erlang Public License along with this software. If not, it can be
+ retrieved online at http://www.erlang.org/.
+
+ Software distributed under the License is distributed on an "AS IS"
+ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ the License for the specific language governing rights and limitations
+ under the License.
+
+ </legalnotice>
+
+ <title>Dependencies between Test Cases and Suites</title>
+ <prepared>Peter Andersson</prepared>
+ <docno></docno>
+ <date></date>
+ <rev></rev>
+ <file>dependencies_chapter.xml</file>
+ </header>
+
+ <section>
+ <title>General</title>
+ <p>When creating test suites, it is strongly recommended to not
+ create dependencies between test cases, i.e. letting test cases
+ depend on the result of previous test cases. There are various
+ reasons for this, for example:</p>
+
+ <list>
+ <item>It makes it impossible to run test cases individually.</item>
+ <item>It makes it impossible to run test cases in different order.</item>
+ <item>It makes debugging very difficult (since a fault could be
+ the result of a problem in a different test case than the one failing).</item>
+ <item>There exists no good and explicit ways to declare dependencies, so
+ it may be very difficult to see and understand these in test suite
+ code and in test logs.</item>
+ <item>Extending, restructuring and maintaining test suites with
+ test case dependencies is difficult.</item>
+ </list>
+
+ <p>There are often sufficient means to work around the need for test
+ case dependencies. Generally, the problem is related to the state of
+ the system under test (SUT). The action of one test case may alter the state
+ of the system and for some other test case to run properly, the new state
+ must be known.</p>
+
+ <p>Instead of passing data between test cases, it is recommended
+ that the test cases read the state from the SUT and perform assertions
+ (i.e. let the test case run if the state is as expected, otherwise reset or fail)
+ and/or use the state to set variables necessary for the test case to execute
+ properly. Common actions can often be implemented as library functions for
+ test cases to call to set the SUT in a required state. (Such common actions
+ may of course also be separately tested if necessary, to ensure they are
+ working as expected). It is sometimes also possible, but not always desirable,
+ to group tests together in one test case, i.e. let a test case perform a
+ "scenario" test (a test that consists of subtests).</p>
+
+ <p>Consider for example a server application under test. The following
+ functionality is to be tested:</p>
+
+ <list>
+ <item>Starting the server.</item>
+ <item>Configuring the server.</item>
+ <item>Connecting a client to the server.</item>
+ <item>Disconnecting a client from the server.</item>
+ <item>Stopping the server.</item>
+ </list>
+
+ <p>There are obvious dependencies between the listed functions. We can't configure
+ the server if it hasn't first been started, we can't connect a client until
+ the server has been properly configured, etc. If we want to have one test
+ case for each of the functions, we might be tempted to try to always run the
+ test cases in the stated order and carry possible data (identities, handles,
+ etc) between the cases and therefore introduce dependencies between them.
+ To avoid this we could consider starting and stopping the server for every test.
+ We would implement the start and stop action as common functions that may be
+ called from init_per_testcase and end_per_testcase. (We would of course test
+ the start and stop functionality separately). The configuration could perhaps also
+ be implemented as a common function, maybe grouped with the start function.
+ Finally the testing of connecting and disconnecting a client may be grouped into
+ one test case. The resulting suite would look something like this:</p>
+
+
+ <pre>
+ -module(my_server_SUITE).
+ -compile(export_all).
+ -include_lib("ct.hrl").
+
+ %%% init and end functions...
+
+ suite() -> [{require,my_server_cfg}].
+
+ init_per_testcase(start_and_stop, Config) ->
+ Config;
+
+ init_per_testcase(config, Config) ->
+ [{server_pid,start_server()} | Config];
+
+ init_per_testcase(_, Config) ->
+ ServerPid = start_server(),
+ configure_server(),
+ [{server_pid,ServerPid} | Config].
+
+ end_per_testcase(start_and_stop, _) ->
+ ok;
+
+ end_per_testcase(_, _) ->
+ ServerPid = ?config(server_pid),
+ stop_server(ServerPid).
+
+ %%% test cases...
+
+ all() -> [start_and_stop, config, connect_and_disconnect].
+
+ %% test that starting and stopping works
+ start_and_stop(_) ->
+ ServerPid = start_server(),
+ stop_server(ServerPid).
+
+ %% configuration test
+ config(Config) ->
+ ServerPid = ?config(server_pid, Config),
+ configure_server(ServerPid).
+
+ %% test connecting and disconnecting client
+ connect_and_disconnect(Config) ->
+ ServerPid = ?config(server_pid, Config),
+ {ok,SessionId} = my_server:connect(ServerPid),
+ ok = my_server:disconnect(ServerPid, SessionId).
+
+ %%% common functions...
+
+ start_server() ->
+ {ok,ServerPid} = my_server:start(),
+ ServerPid.
+
+ stop_server(ServerPid) ->
+ ok = my_server:stop(),
+ ok.
+
+ configure_server(ServerPid) ->
+ ServerCfgData = ct:get_config(my_server_cfg),
+ ok = my_server:configure(ServerPid, ServerCfgData),
+ ok.
+ </pre>
+ </section>
+
+ <section>
+ <marker id="save_config"></marker>
+ <title>Saving configuration data</title>
+
+ <p>There might be situations where it is impossible, or infeasible at least, to
+ implement independent test cases. Maybe it is simply not possible to read the
+ SUT state. Maybe resetting the SUT is impossible and it takes much too long
+ to restart the system. In situations where test case dependency is necessary,
+ CT offers a structured way to carry data from one test case to the next. The
+ same mechanism may also be used to carry data from one test suite to the next.</p>
+
+ <p>The mechanism for passing data is called <c>save_config</c>. The idea is that
+ one test case (or suite) may save the current value of Config - or any list of
+ key-value tuples - so that it can be read by the next executing test case
+ (or test suite). The configuration data is not saved permanently but can only
+ be passed from one case (or suite) to the next.</p>
+
+ <p>To save <c>Config</c> data, return the tuple:</p>
+
+ <p><c>{save_config,ConfigList}</c></p>
+
+ <p>from <c>end_per_testcase</c> or from the main test case function. To read data
+ saved by a previous test case, use the <c>config</c> macro with a
+ <c>saved_config</c> key:</p>
+
+ <p><c>{Saver,ConfigList} = ?config(saved_config, Config)</c></p>
+
+ <p><c>Saver</c> (<c>atom()</c>) is the name of the previous test case (where the
+ data was saved). The <c>config</c> macro may be used to extract particular data
+ also from the recalled <c>ConfigList</c>. It is strongly recommended that
+ <c>Saver</c> is always matched to the expected name of the saving test case.
+ This way problems due to restructuring of the test suite may be avoided. Also it
+ makes the dependency more explicit and the test suite easier to read and maintain.</p>
+
+ <p>To pass data from one test suite to another, the same mechanism is used. The data
+ should be saved by the <c>end_per_suite</c> function and read by <c>init_per_suite</c>
+ in the suite that follows. When passing data between suites, <c>Saver</c> carries the
+ name of the test suite.</p>
+
+ <p>Example:</p>
+
+ <pre>
+ -module(server_b_SUITE).
+ -compile(export_all).
+ -include_lib("ct.hrl").
+
+ %%% init and end functions...
+
+ init_per_suite(Config) ->
+ %% read config saved by previous test suite
+ {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
+ %% extract server identity (comes from server_a_SUITE)
+ ServerId = ?config(server_id, OldConfig),
+ SessionId = connect_to_server(ServerId),
+ [{ids,{ServerId,SessionId}} | Config].
+
+ end_per_suite(Config) ->
+ %% save config for server_c_SUITE (session_id and server_id)
+ {save_config,Config}
+
+ %%% test cases...
+
+ all() -> [allocate, deallocate].
+
+ allocate(Config) ->
+ {ServerId,SessionId} = ?config(ids, Config),
+ {ok,Handle} = allocate_resource(ServerId, SessionId),
+ %% save handle for deallocation test
+ NewConfig = [{handle,Handle}],
+ {save_config,NewConfig}.
+
+ deallocate(Config) ->
+ {ServerId,SessionId} = ?config(ids, Config),
+ {allocate,OldConfig} = ?config(saved_config, Config),
+ Handle = ?config(handle, OldConfig),
+ ok = deallocate_resource(ServerId, SessionId, Handle).
+ </pre>
+
+ <p>It is also possible to save <c>Config</c> data from a test case that is to be
+ skipped. To accomplish this, return the following tuple:</p>
+
+ <p><c>{skip_and_save,Reason,ConfigList}</c></p>
+
+ <p>The result will be that the test case is skipped with <c>Reason</c> printed to
+ the log file (as described in previous chapters), and <c>ConfigList</c> is saved
+ for the next test case. <c>ConfigList</c> may be read by means of
+ <c>?config(saved_config, Config)</c>, as described above. <c>skip_and_save</c>
+ may also be returned from <c>init_per_suite</c>, in which case the saved data can
+ be read by <c>init_per_suite</c> in the suite that follows.</p>
+ </section>
+
+ <section>
+ <marker id="sequences"></marker>
+ <title>Sequences</title>
+
+ <p>It is possible that test cases depend on each other so that
+ if one case fails, the following test(s) should not be executed.
+ Typically, if the <c>save_config</c> facility is used and a test
+ case that is expected to save data crashes, the following
+ case can not run. CT offers a way to declare such dependencies,
+ called sequences.</p>
+
+ <p>A sequence of test cases is defined as a test case group
+ with a <c>sequence</c> property. Test case groups are defined by
+ means of the <c>groups/0</c> function in the test suite (see the
+ <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso>
+ chapter for details).</p>
+
+ <p>For example, if we would like to make sure that if <c>allocate</c>
+ in <c>server_b_SUITE</c> (above) crashes, <c>deallocate</c> is skipped,
+ we may define a sequence like this:</p>
+
+ <pre>
+ groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre>
+
+ <p>Let's also assume the suite contains the test case <c>get_resource_status</c>,
+ which is independent of the other two cases, then the <c>all</c> function could
+ look like this:</p>
+
+ <pre>
+ all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre>
+
+ <p>If <c>alloc</c> succeeds, <c>dealloc</c> is also executed. If <c>alloc</c> fails
+ however, <c>dealloc</c> is not executed but marked as SKIPPED in the html log.
+ <c>get_resource_status</c> will run no matter what happens to the <c>alloc_and_dealloc</c>
+ cases.</p>
+
+ <p>Test cases in a sequence will be executed in order until they have all succeeded or
+ until one case fails. If one fails, all following cases in the sequence are skipped.
+ The cases in the sequence that have succeeded up to that point are reported as successful
+ in the log. An arbitrary number of sequences may be specified. Example:</p>
+
+ <pre>
+ groups() -> [{scenarioA, [sequence], [testA1, testA2]},
+ {scenarioB, [sequence], [testB1, testB2, testB3]}].
+
+ all() -> [test1,
+ test2,
+ {group,scenarioA},
+ test3,
+ {group,scenarioB},
+ test4].</pre>
+
+ <p>It is possible to have sub-groups in a sequence group. Such sub-groups can have
+ any property, i.e. they are not required to also be sequences. If you want the status
+ of the sub-group to affect the sequence on the level above, return
+ <c>{return_group_result,Status}</c> from <c>end_per_group/2</c>, as described in the
+ <seealso marker="write_test_chapter#repeated_groups">Repeated groups</seealso>
+ chapter. A failed sub-group (<c>Status == failed</c>) will cause the execution of a
+ sequence to fail in the same way a test case does.</p>
+ </section>
+</chapter>