The R13B03 release.OTP_R13B03

1 files changed, 309 insertions, 0 deletions

diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml
new file mode 100644
index 0000000000..8aad552285
--- /dev/null
+++ b/lib/common_test/doc/src/dependencies_chapter.xml
@@ -0,0 +1,309 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+  <header>
+    <copyright>
+      <year>2006</year><year>2009</year>
+      <holder>Ericsson AB. All Rights Reserved.</holder>
+    </copyright>
+    <legalnotice>
+      The contents of this file are subject to the Erlang Public License,
+      Version 1.1, (the "License"); you may not use this file except in
+      compliance with the License. You should have received a copy of the
+      Erlang Public License along with this software. If not, it can be
+      retrieved online at http://www.erlang.org/.
+    
+      Software distributed under the License is distributed on an "AS IS"
+      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+      the License for the specific language governing rights and limitations
+      under the License.
+    
+    </legalnotice>
+
+    <title>Dependencies between Test Cases and Suites</title>
+    <prepared>Peter Andersson</prepared>
+    <docno></docno>
+    <date></date>
+    <rev></rev>
+    <file>dependencies_chapter.xml</file>
+  </header>
+
+  <section>
+    <title>General</title>
+    <p>When creating test suites, it is strongly recommended to not
+      create dependencies between test cases, i.e. letting test cases
+      depend on the result of previous test cases. There are various
+      reasons for this, for example:</p>
+
+    <list>
+      <item>It makes it impossible to run test cases individually.</item>
+      <item>It makes it impossible to run test cases in different order.</item>
+      <item>It makes debugging very difficult (since a fault could be
+	the result of a problem in a different test case than the one failing).</item>
+      <item>There exists no good and explicit ways to declare dependencies, so 
+	it may be very difficult to see and understand these in test suite 
+	code and in test logs.</item>
+      <item>Extending, restructuring and maintaining test suites with 
+	test case dependencies is difficult.</item>      
+    </list>
+    
+    <p>There are often sufficient means to work around the need for test 
+      case dependencies. Generally, the problem is related to the state of 
+      the system under test (SUT). The action of one test case may alter the state 
+      of the system and for some other test case to run properly, the new state 
+      must be known.</p>
+
+    <p>Instead of passing data between test cases, it is recommended
+      that the test cases read the state from the SUT and perform assertions
+      (i.e. let the test case run if the state is as expected, otherwise reset or fail)
+      and/or use the state to set variables necessary for the test case to execute
+      properly. Common actions can often be implemented as library functions for
+      test cases to call to set the SUT in a required state. (Such common actions
+      may of course also be separately tested if necessary, to ensure they are 
+      working as expected). It is sometimes also possible, but not always desirable, 
+      to group tests together in one test case, i.e. let a test case perform a 
+      "scenario" test (a test that consists of subtests).</p>
+
+    <p>Consider for example a server application under test. The following 
+      functionality is to be tested:</p>
+    
+    <list>
+      <item>Starting the server.</item>
+      <item>Configuring the server.</item>
+      <item>Connecting a client to the server.</item>
+      <item>Disconnecting a client from the server.</item>
+      <item>Stopping the server.</item>
+    </list>
+
+    <p>There are obvious dependencies between the listed functions. We can't configure 
+      the server if it hasn't first been started, we can't connect a client until 
+      the server has been properly configured, etc. If we want to have one test 
+      case for each of the functions, we might be tempted to try to always run the
+      test cases in the stated order and carry possible data (identities, handles,
+      etc) between the cases and therefore introduce dependencies between them. 
+      To avoid this we could consider starting and stopping the server for every test.
+      We would implement the start and stop action as common functions that may be 
+      called from init_per_testcase and end_per_testcase. (We would of course test 
+      the start and stop functionality separately). The configuration could perhaps also
+      be implemented as a common function, maybe grouped with the start function.
+      Finally the testing of connecting and disconnecting a client may be grouped into
+      one test case. The resulting suite would look something like this:</p>
+
+
+    <pre>      
+      -module(my_server_SUITE).
+      -compile(export_all).
+      -include_lib("ct.hrl").
+
+      %%% init and end functions...
+
+      suite() -> [{require,my_server_cfg}].
+
+      init_per_testcase(start_and_stop, Config) ->
+          Config;
+
+      init_per_testcase(config, Config) ->
+          [{server_pid,start_server()} | Config];
+
+      init_per_testcase(_, Config) ->
+          ServerPid = start_server(),
+          configure_server(),
+          [{server_pid,ServerPid} | Config].
+
+      end_per_testcase(start_and_stop, _) ->
+          ok;
+
+      end_per_testcase(_, _) ->
+          ServerPid = ?config(server_pid),
+          stop_server(ServerPid).
+
+      %%% test cases...
+
+      all() -> [start_and_stop, config, connect_and_disconnect].
+
+      %% test that starting and stopping works
+      start_and_stop(_) ->
+          ServerPid = start_server(),
+          stop_server(ServerPid).
+
+      %% configuration test
+      config(Config) ->
+          ServerPid = ?config(server_pid, Config),
+          configure_server(ServerPid).
+
+      %% test connecting and disconnecting client
+      connect_and_disconnect(Config) ->
+          ServerPid = ?config(server_pid, Config),
+          {ok,SessionId} = my_server:connect(ServerPid),
+          ok = my_server:disconnect(ServerPid, SessionId).
+
+      %%% common functions...
+
+      start_server() ->
+          {ok,ServerPid} = my_server:start(),
+          ServerPid.
+
+      stop_server(ServerPid) ->
+          ok = my_server:stop(),
+          ok.
+
+      configure_server(ServerPid) ->
+          ServerCfgData = ct:get_config(my_server_cfg),
+          ok = my_server:configure(ServerPid, ServerCfgData),
+          ok.
+      </pre>
+    </section>
+
+    <section>
+    <marker id="save_config"></marker>
+      <title>Saving configuration data</title>
+
+      <p>There might be situations where it is impossible, or infeasible at least, to
+	implement independent test cases. Maybe it is simply not possible to read the 
+	SUT state. Maybe resetting the SUT is impossible and it takes much too long
+	to restart the system. In situations where test case dependency is necessary,
+	CT offers a structured way to carry data from one test case to the next. The
+	same mechanism may also be used to carry data from one test suite to the next.</p>
+
+      <p>The mechanism for passing data is called <c>save_config</c>. The idea is that
+	one test case (or suite) may save the current value of Config - or any list of
+	key-value tuples - so that it can be read by the next executing test case 
+	(or test suite). The configuration data is not saved permanently but can only 
+	be passed from one case (or suite) to the next.</p>
+
+      <p>To save <c>Config</c> data, return the	tuple:</p>
+
+      <p><c>{save_config,ConfigList}</c></p>
+      
+      <p>from <c>end_per_testcase</c> or from the main test case function. To read data 
+	saved by a previous test case, use the <c>config</c> macro with	a 
+	<c>saved_config</c> key:</p>
+      
+      <p><c>{Saver,ConfigList} = ?config(saved_config, Config)</c></p>
+
+      <p><c>Saver</c> (<c>atom()</c>) is the name of the previous test case (where the
+	data was saved). The <c>config</c> macro may be used to extract particular data
+	also from the recalled <c>ConfigList</c>. It is strongly recommended that 
+	<c>Saver</c> is always matched to the expected name of the saving test case. 
+	This way problems due to restructuring of the test suite may be avoided. Also it 
+	makes the dependency more explicit and the test suite easier to read and maintain.</p>
+
+      <p>To pass data from one test suite to another, the same mechanism is used. The data
+	should be saved by the <c>end_per_suite</c> function and read by <c>init_per_suite</c>
+	in the suite that follows. When passing data between suites, <c>Saver</c> carries the 
+	name of the test suite.</p>
+
+      <p>Example:</p>
+      
+      <pre>
+	-module(server_b_SUITE).
+	-compile(export_all).
+	-include_lib("ct.hrl").
+
+	%%% init and end functions...
+
+	init_per_suite(Config) ->
+	    %% read config saved by previous test suite
+	    {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
+	    %% extract server identity (comes from server_a_SUITE)
+	    ServerId = ?config(server_id, OldConfig),
+	    SessionId = connect_to_server(ServerId),
+	    [{ids,{ServerId,SessionId}} | Config].
+
+	end_per_suite(Config) ->
+	    %% save config for server_c_SUITE (session_id and server_id)
+	    {save_config,Config}
+
+	%%% test cases...
+
+	all() -> [allocate, deallocate].
+
+	allocate(Config) ->
+	    {ServerId,SessionId} = ?config(ids, Config),
+	    {ok,Handle} = allocate_resource(ServerId, SessionId),
+	    %% save handle for deallocation test
+	    NewConfig = [{handle,Handle}],
+	    {save_config,NewConfig}.
+
+	deallocate(Config) ->
+	    {ServerId,SessionId} = ?config(ids, Config),
+	    {allocate,OldConfig} = ?config(saved_config, Config),
+	    Handle = ?config(handle, OldConfig),
+	    ok = deallocate_resource(ServerId, SessionId, Handle). 
+	</pre>
+
+      <p>It is also possible to save <c>Config</c> data from a test case that is to be
+        skipped. To accomplish this, return the following tuple:</p>
+
+      <p><c>{skip_and_save,Reason,ConfigList}</c></p>
+
+      <p>The result will be that the test case is skipped with <c>Reason</c> printed to
+      the log file (as described in previous chapters), and <c>ConfigList</c> is saved 
+      for the next test case. <c>ConfigList</c> may be read by means of 
+      <c>?config(saved_config, Config)</c>, as described above. <c>skip_and_save</c>
+      may also be returned from <c>init_per_suite</c>, in which case the saved data can
+      be read by <c>init_per_suite</c> in the suite that follows.</p>
+    </section>
+
+    <section>
+    <marker id="sequences"></marker>
+      <title>Sequences</title>
+
+      <p>It is possible that test cases depend on each other so that
+	if one case fails, the following test(s) should not be executed.
+        Typically, if the <c>save_config</c> facility is used and a test 
+	case that is expected to save data crashes, the following 
+	case can not run. CT offers a way to declare such dependencies, 
+	called sequences.</p>
+
+      <p>A sequence of test cases is defined as a test case group
+        with a <c>sequence</c> property. Test case groups are defined by 
+	means of the <c>groups/0</c> function in the test suite (see the 
+	<seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso>
+	chapter for details).</p>
+
+      <p>For example, if we would like to make sure that if <c>allocate</c>
+	in <c>server_b_SUITE</c> (above) crashes, <c>deallocate</c> is skipped,
+	we may define a sequence like this:</p>
+      
+      <pre>
+	groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre>
+
+      <p>Let's also assume the suite contains the test case <c>get_resource_status</c>, 
+	which is independent of the other two cases, then the <c>all</c> function could 
+	look like this:</p>
+
+      <pre>
+	all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre>
+
+      <p>If <c>alloc</c> succeeds, <c>dealloc</c> is also executed. If <c>alloc</c> fails
+	however, <c>dealloc</c> is not executed but marked as SKIPPED in the html log. 
+	<c>get_resource_status</c> will run no matter what happens to the <c>alloc_and_dealloc</c>
+	cases.</p>
+
+      <p>Test cases in a sequence will be executed in order until they have all succeeded or 
+	until one case fails. If one fails, all following cases in the sequence are skipped.
+	The cases in the sequence that have succeeded up to that point are reported as successful
+	in the log. An arbitrary number of sequences may be specified. Example:</p>
+
+      <pre>
+	groups() -> [{scenarioA, [sequence], [testA1, testA2]}, 
+	             {scenarioB, [sequence], [testB1, testB2, testB3]}].
+
+	all() -> [test1, 
+	          test2, 
+	          {group,scenarioA}, 
+	          test3, 
+	          {group,scenarioB}, 
+	          test4].</pre>
+
+      <p>It is possible to have sub-groups in a sequence group. Such sub-groups can have 
+        any property, i.e. they are not required to also be sequences. If you want the status
+	of the sub-group to affect the sequence on the level above, return 
+	<c>{return_group_result,Status}</c> from <c>end_per_group/2</c>, as described in the
+	<seealso marker="write_test_chapter#repeated_groups">Repeated groups</seealso>
+	chapter. A failed sub-group (<c>Status == failed</c>) will cause the execution of a 
+	sequence to fail in the	same way a test case does.</p>
+    </section>
+</chapter>