aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src/dependencies_chapter.xml
diff options
context:
space:
mode:
authorPeter Andersson <[email protected]>2016-03-10 14:43:09 +0100
committerPeter Andersson <[email protected]>2016-03-10 14:43:09 +0100
commit629ba3086de251e33339e7ce846c568451d27300 (patch)
tree760e07b5c4409c73d005593c8f9931bc379fa241 /lib/common_test/doc/src/dependencies_chapter.xml
parent9cafc664bbf75fa9b6d3666f85e672605c96d1f1 (diff)
parent00c18b1568eb629fcfa715295d7e99accf9c74b1 (diff)
downloadotp-629ba3086de251e33339e7ce846c568451d27300.tar.gz
otp-629ba3086de251e33339e7ce846c568451d27300.tar.bz2
otp-629ba3086de251e33339e7ce846c568451d27300.zip
Merge branch 'maint'
Diffstat (limited to 'lib/common_test/doc/src/dependencies_chapter.xml')
-rw-r--r--lib/common_test/doc/src/dependencies_chapter.xml382
1 files changed, 194 insertions, 188 deletions
diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml
index fb758d90df..d96a41473f 100644
--- a/lib/common_test/doc/src/dependencies_chapter.xml
+++ b/lib/common_test/doc/src/dependencies_chapter.xml
@@ -33,217 +33,220 @@
<section>
<title>General</title>
<p>When creating test suites, it is strongly recommended to not
- create dependencies between test cases, i.e. letting test cases
+ create dependencies between test cases, that is, letting test cases
depend on the result of previous test cases. There are various
- reasons for this, for example:</p>
+ reasons for this, such as, the following:</p>
- <list>
+ <list type="bulleted">
<item>It makes it impossible to run test cases individually.</item>
- <item>It makes it impossible to run test cases in different order.</item>
- <item>It makes debugging very difficult (since a fault could be
+ <item>It makes it impossible to run test cases in a different order.</item>
+ <item>It makes debugging difficult (as a fault can be
the result of a problem in a different test case than the one failing).</item>
- <item>There exists no good and explicit ways to declare dependencies, so
- it may be very difficult to see and understand these in test suite
+ <item>There are no good and explicit ways to declare dependencies, so
+ it can be difficult to see and understand these in test suite
code and in test logs.</item>
- <item>Extending, restructuring and maintaining test suites with
+ <item>Extending, restructuring, and maintaining test suites with
test case dependencies is difficult.</item>
</list>
<p>There are often sufficient means to work around the need for test
case dependencies. Generally, the problem is related to the state of
- the system under test (SUT). The action of one test case may alter the state
- of the system and for some other test case to run properly, the new state
+ the System Under Test (SUT). The action of one test case can change the
+ system state. For some other test case to run properly, this new state
must be known.</p>
<p>Instead of passing data between test cases, it is recommended
that the test cases read the state from the SUT and perform assertions
- (i.e. let the test case run if the state is as expected, otherwise reset or fail)
- and/or use the state to set variables necessary for the test case to execute
- properly. Common actions can often be implemented as library functions for
- test cases to call to set the SUT in a required state. (Such common actions
- may of course also be separately tested if necessary, to ensure they are
- working as expected). It is sometimes also possible, but not always desirable,
- to group tests together in one test case, i.e. let a test case perform a
- "scenario" test (a test that consists of subtests).</p>
-
- <p>Consider for example a server application under test. The following
+ (that is, let the test case run if the state is as expected, otherwise reset or fail).
+ It is also recommended to use the state to set variables necessary for the
+ test case to execute properly. Common actions can often be implemented as
+ library functions for test cases to call to set the SUT in a required state.
+ (Such common actions can also be separately tested, if necessary,
+ to ensure that they work as expected). It is sometimes also possible,
+ but not always desirable, to group tests together in one test case, that is,
+ let a test case perform a "scenario" test (a test consisting of subtests).</p>
+
+ <p>Consider, for example, a server application under test. The following
functionality is to be tested:</p>
- <list>
- <item>Starting the server.</item>
- <item>Configuring the server.</item>
- <item>Connecting a client to the server.</item>
- <item>Disconnecting a client from the server.</item>
- <item>Stopping the server.</item>
+ <list type="bulleted">
+ <item>Starting the server</item>
+ <item>Configuring the server</item>
+ <item>Connecting a client to the server</item>
+ <item>Disconnecting a client from the server</item>
+ <item>Stopping the server</item>
</list>
- <p>There are obvious dependencies between the listed functions. We can't configure
- the server if it hasn't first been started, we can't connect a client until
- the server has been properly configured, etc. If we want to have one test
- case for each of the functions, we might be tempted to try to always run the
+ <p>There are obvious dependencies between the listed functions. The server cannot
+ be configured if it has not first been started, a client connot be connectd until
+ the server is properly configured, and so on. If we want to have one test
+ case for each function, we might be tempted to try to always run the
test cases in the stated order and carry possible data (identities, handles,
- etc) between the cases and therefore introduce dependencies between them.
- To avoid this we could consider starting and stopping the server for every test.
- We would implement the start and stop action as common functions that may be
- called from init_per_testcase and end_per_testcase. (We would of course test
- the start and stop functionality separately). The configuration could perhaps also
- be implemented as a common function, maybe grouped with the start function.
- Finally the testing of connecting and disconnecting a client may be grouped into
- one test case. The resulting suite would look something like this:</p>
-
+ and so on) between the cases and therefore introduce dependencies between them.</p>
+
+ <p>To avoid this, we can consider starting and stopping the server for every test.
+ We can thus implement the start and stop action as common functions to be
+ called from
+ <seealso marker="common_test_app#Module:init_per_testcase-2"><c>init_per_testcase</c></seealso> and
+ <seealso marker="common_test_app#Module:end_per_testcase-2"><c>end_per_testcase</c></seealso>.
+ (Remember to test the start and stop functionality separately.)
+ The configuration can also be implemented as a common function, maybe grouped
+ with the start function. Finally, the testing of connecting and disconnecting a
+ client can be grouped into one test case. The resulting suite can look as
+ follows:</p>
<pre>
- -module(my_server_SUITE).
- -compile(export_all).
- -include_lib("ct.hrl").
+ -module(my_server_SUITE).
+ -compile(export_all).
+ -include_lib("ct.hrl").
+
+ %%% init and end functions...
- %%% init and end functions...
+ suite() -> [{require,my_server_cfg}].
- suite() -> [{require,my_server_cfg}].
+ init_per_testcase(start_and_stop, Config) ->
+ Config;
- init_per_testcase(start_and_stop, Config) ->
- Config;
+ init_per_testcase(config, Config) ->
+ [{server_pid,start_server()} | Config];
- init_per_testcase(config, Config) ->
- [{server_pid,start_server()} | Config];
+ init_per_testcase(_, Config) ->
+ ServerPid = start_server(),
+ configure_server(),
+ [{server_pid,ServerPid} | Config].
- init_per_testcase(_, Config) ->
- ServerPid = start_server(),
- configure_server(),
- [{server_pid,ServerPid} | Config].
+ end_per_testcase(start_and_stop, _) ->
+ ok;
- end_per_testcase(start_and_stop, _) ->
- ok;
+ end_per_testcase(_, _) ->
+ ServerPid = ?config(server_pid),
+ stop_server(ServerPid).
- end_per_testcase(_, _) ->
- ServerPid = ?config(server_pid),
- stop_server(ServerPid).
+ %%% test cases...
- %%% test cases...
+ all() -> [start_and_stop, config, connect_and_disconnect].
- all() -> [start_and_stop, config, connect_and_disconnect].
+ %% test that starting and stopping works
+ start_and_stop(_) ->
+ ServerPid = start_server(),
+ stop_server(ServerPid).
- %% test that starting and stopping works
- start_and_stop(_) ->
- ServerPid = start_server(),
- stop_server(ServerPid).
+ %% configuration test
+ config(Config) ->
+ ServerPid = ?config(server_pid, Config),
+ configure_server(ServerPid).
- %% configuration test
- config(Config) ->
- ServerPid = ?config(server_pid, Config),
- configure_server(ServerPid).
+ %% test connecting and disconnecting client
+ connect_and_disconnect(Config) ->
+ ServerPid = ?config(server_pid, Config),
+ {ok,SessionId} = my_server:connect(ServerPid),
+ ok = my_server:disconnect(ServerPid, SessionId).
- %% test connecting and disconnecting client
- connect_and_disconnect(Config) ->
- ServerPid = ?config(server_pid, Config),
- {ok,SessionId} = my_server:connect(ServerPid),
- ok = my_server:disconnect(ServerPid, SessionId).
+ %%% common functions...
- %%% common functions...
+ start_server() ->
+ {ok,ServerPid} = my_server:start(),
+ ServerPid.
- start_server() ->
- {ok,ServerPid} = my_server:start(),
- ServerPid.
+ stop_server(ServerPid) ->
+ ok = my_server:stop(),
+ ok.
- stop_server(ServerPid) ->
- ok = my_server:stop(),
- ok.
+ configure_server(ServerPid) ->
+ ServerCfgData = ct:get_config(my_server_cfg),
+ ok = my_server:configure(ServerPid, ServerCfgData),
+ ok.</pre>
- configure_server(ServerPid) ->
- ServerCfgData = ct:get_config(my_server_cfg),
- ok = my_server:configure(ServerPid, ServerCfgData),
- ok.
- </pre>
</section>
<section>
<marker id="save_config"></marker>
- <title>Saving configuration data</title>
+ <title>Saving Configuration Data</title>
- <p>There might be situations where it is impossible, or infeasible at least, to
- implement independent test cases. Maybe it is simply not possible to read the
- SUT state. Maybe resetting the SUT is impossible and it takes much too long
+ <p>Sometimes it is impossible, or infeasible, to
+ implement independent test cases. Maybe it is not possible to read the
+ SUT state. Maybe resetting the SUT is impossible and it takes too long time
to restart the system. In situations where test case dependency is necessary,
CT offers a structured way to carry data from one test case to the next. The
- same mechanism may also be used to carry data from one test suite to the next.</p>
+ same mechanism can also be used to carry data from one test suite to the next.</p>
<p>The mechanism for passing data is called <c>save_config</c>. The idea is that
- one test case (or suite) may save the current value of Config - or any list of
- key-value tuples - so that it can be read by the next executing test case
- (or test suite). The configuration data is not saved permanently but can only
- be passed from one case (or suite) to the next.</p>
+ one test case (or suite) can save the current value of <c>Config</c>, or any list of
+ key-value tuples, so that the next executing test case (or test suite) can read it.
+ The configuration data is not saved permanently but can only be passed from one
+ case (or suite) to the next.</p>
- <p>To save <c>Config</c> data, return the tuple:</p>
+ <p>To save <c>Config</c> data, return tuple <c>{save_config,ConfigList}</c>
+ from <c>end_per_testcase</c> or from the main test case function.</p>
- <p><c>{save_config,ConfigList}</c></p>
-
- <p>from <c>end_per_testcase</c> or from the main test case function. To read data
- saved by a previous test case, use the <c>config</c> macro with a
- <c>saved_config</c> key:</p>
+ <p>To read data saved by a previous test case, use macro <c>config</c> with a
+ <c>saved_config</c> key as follows:</p>
<p><c>{Saver,ConfigList} = ?config(saved_config, Config)</c></p>
<p><c>Saver</c> (<c>atom()</c>) is the name of the previous test case (where the
- data was saved). The <c>config</c> macro may be used to extract particular data
+ data was saved). The <c>config</c> macro can be used to extract particular data
also from the recalled <c>ConfigList</c>. It is strongly recommended that
<c>Saver</c> is always matched to the expected name of the saving test case.
- This way problems due to restructuring of the test suite may be avoided. Also it
- makes the dependency more explicit and the test suite easier to read and maintain.</p>
+ This way, problems because of restructuring of the test suite can be avoided.
+ Also, it makes the dependency more explicit and the test suite easier to read
+ and maintain.</p>
<p>To pass data from one test suite to another, the same mechanism is used. The data
- should be saved by the <c>end_per_suite</c> function and read by <c>init_per_suite</c>
+ is to be saved by finction
+ <seealso marker="common_test_app#Module:end_per_suite-1"><c>end_per_suite</c></seealso>
+ and read by function
+ <seealso marker="common_test_app#Module:init_per_suite-1"><c>init_per_suite</c></seealso>
in the suite that follows. When passing data between suites, <c>Saver</c> carries the
name of the test suite.</p>
- <p>Example:</p>
+ <p><em>Example:</em></p>
<pre>
- -module(server_b_SUITE).
- -compile(export_all).
- -include_lib("ct.hrl").
-
- %%% init and end functions...
-
- init_per_suite(Config) ->
- %% read config saved by previous test suite
- {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
- %% extract server identity (comes from server_a_SUITE)
- ServerId = ?config(server_id, OldConfig),
- SessionId = connect_to_server(ServerId),
- [{ids,{ServerId,SessionId}} | Config].
-
- end_per_suite(Config) ->
- %% save config for server_c_SUITE (session_id and server_id)
- {save_config,Config}
-
- %%% test cases...
-
- all() -> [allocate, deallocate].
-
- allocate(Config) ->
- {ServerId,SessionId} = ?config(ids, Config),
- {ok,Handle} = allocate_resource(ServerId, SessionId),
- %% save handle for deallocation test
- NewConfig = [{handle,Handle}],
- {save_config,NewConfig}.
-
- deallocate(Config) ->
- {ServerId,SessionId} = ?config(ids, Config),
- {allocate,OldConfig} = ?config(saved_config, Config),
- Handle = ?config(handle, OldConfig),
- ok = deallocate_resource(ServerId, SessionId, Handle).
- </pre>
-
- <p>It is also possible to save <c>Config</c> data from a test case that is to be
- skipped. To accomplish this, return the following tuple:</p>
-
- <p><c>{skip_and_save,Reason,ConfigList}</c></p>
-
- <p>The result will be that the test case is skipped with <c>Reason</c> printed to
- the log file (as described in previous chapters), and <c>ConfigList</c> is saved
- for the next test case. <c>ConfigList</c> may be read by means of
- <c>?config(saved_config, Config)</c>, as described above. <c>skip_and_save</c>
- may also be returned from <c>init_per_suite</c>, in which case the saved data can
+ -module(server_b_SUITE).
+ -compile(export_all).
+ -include_lib("ct.hrl").
+
+ %%% init and end functions...
+
+ init_per_suite(Config) ->
+ %% read config saved by previous test suite
+ {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
+ %% extract server identity (comes from server_a_SUITE)
+ ServerId = ?config(server_id, OldConfig),
+ SessionId = connect_to_server(ServerId),
+ [{ids,{ServerId,SessionId}} | Config].
+
+ end_per_suite(Config) ->
+ %% save config for server_c_SUITE (session_id and server_id)
+ {save_config,Config}
+
+ %%% test cases...
+
+ all() -> [allocate, deallocate].
+
+ allocate(Config) ->
+ {ServerId,SessionId} = ?config(ids, Config),
+ {ok,Handle} = allocate_resource(ServerId, SessionId),
+ %% save handle for deallocation test
+ NewConfig = [{handle,Handle}],
+ {save_config,NewConfig}.
+
+ deallocate(Config) ->
+ {ServerId,SessionId} = ?config(ids, Config),
+ {allocate,OldConfig} = ?config(saved_config, Config),
+ Handle = ?config(handle, OldConfig),
+ ok = deallocate_resource(ServerId, SessionId, Handle).</pre>
+
+ <p>To save <c>Config</c> data from a test case that is to be
+ skipped, return tuple
+ <c>{skip_and_save,Reason,ConfigList}</c>.</p>
+
+ <p>The result is that the test case is skipped with <c>Reason</c> printed to
+ the log file (as described earlier) and <c>ConfigList</c> is saved
+ for the next test case. <c>ConfigList</c> can be read using
+ <c>?config(saved_config, Config)</c>, as described earlier. <c>skip_and_save</c>
+ can also be returned from <c>init_per_suite</c>. In this case, the saved data can
be read by <c>init_per_suite</c> in the suite that follows.</p>
</section>
@@ -251,60 +254,63 @@
<marker id="sequences"></marker>
<title>Sequences</title>
- <p>It is possible that test cases depend on each other so that
- if one case fails, the following test(s) should not be executed.
+ <p>Sometimes test cases depend on each other so that
+ if one case fails, the following tests are not to be executed.
Typically, if the <c>save_config</c> facility is used and a test
case that is expected to save data crashes, the following
- case can not run. CT offers a way to declare such dependencies,
+ case cannot run. <c>Common Test</c> offers a way to declare such dependencies,
called sequences.</p>
<p>A sequence of test cases is defined as a test case group
- with a <c>sequence</c> property. Test case groups are defined by
- means of the <c>groups/0</c> function in the test suite (see the
- <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso>
- chapter for details).</p>
-
- <p>For example, if we would like to make sure that if <c>allocate</c>
- in <c>server_b_SUITE</c> (above) crashes, <c>deallocate</c> is skipped,
- we may define a sequence like this:</p>
+ with a <c>sequence</c> property. Test case groups are defined
+ through function <c>groups/0</c> in the test suite (for details, see section
+ <seealso marker="write_test_chapter#test_case_groups">Test Case Groups</seealso>.</p>
+
+ <p>For example, to ensure that if <c>allocate</c>
+ in <c>server_b_SUITE</c> crashes, <c>deallocate</c> is skipped,
+ the following sequence can be defined:</p>
<pre>
- groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre>
+ groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre>
- <p>Let's also assume the suite contains the test case <c>get_resource_status</c>,
- which is independent of the other two cases, then the <c>all</c> function could
- look like this:</p>
+ <p>Assume that the suite contains the test case <c>get_resource_status</c>
+ that is independent of the other two cases, then function <c>all</c> can
+ look as follows:</p>
<pre>
- all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre>
+ all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre>
<p>If <c>alloc</c> succeeds, <c>dealloc</c> is also executed. If <c>alloc</c> fails
- however, <c>dealloc</c> is not executed but marked as SKIPPED in the html log.
- <c>get_resource_status</c> will run no matter what happens to the <c>alloc_and_dealloc</c>
+ however, <c>dealloc</c> is not executed but marked as <c>SKIPPED</c> in the HTML log.
+ <c>get_resource_status</c> runs no matter what happens to the <c>alloc_and_dealloc</c>
cases.</p>
- <p>Test cases in a sequence will be executed in order until they have all succeeded or
- until one case fails. If one fails, all following cases in the sequence are skipped.
- The cases in the sequence that have succeeded up to that point are reported as successful
- in the log. An arbitrary number of sequences may be specified. Example:</p>
+ <p>Test cases in a sequence are executed in order until all succeed or
+ one fails. If one fails, all following cases in the sequence are skipped.
+ The cases in the sequence that have succeeded up to that point are reported as
+ successful in the log. Any number of sequences can be specified.</p>
+ <p><em>Example:</em></p>
<pre>
- groups() -> [{scenarioA, [sequence], [testA1, testA2]},
- {scenarioB, [sequence], [testB1, testB2, testB3]}].
-
- all() -> [test1,
- test2,
- {group,scenarioA},
- test3,
- {group,scenarioB},
- test4].</pre>
-
- <p>It is possible to have sub-groups in a sequence group. Such sub-groups can have
- any property, i.e. they are not required to also be sequences. If you want the status
- of the sub-group to affect the sequence on the level above, return
- <c>{return_group_result,Status}</c> from <c>end_per_group/2</c>, as described in the
- <seealso marker="write_test_chapter#repeated_groups">Repeated groups</seealso>
- chapter. A failed sub-group (<c>Status == failed</c>) will cause the execution of a
+ groups() -> [{scenarioA, [sequence], [testA1, testA2]},
+ {scenarioB, [sequence], [testB1, testB2, testB3]}].
+
+ all() -> [test1,
+ test2,
+ {group,scenarioA},
+ test3,
+ {group,scenarioB},
+ test4].</pre>
+
+ <p>A sequence group can have subgroups. Such subgroups can have
+ any property, that is, they are not required to also be sequences. If you want the
+ status of the subgroup to affect the sequence on the level above, return
+ <c>{return_group_result,Status}</c> from
+ <seealso marker="common_test_app#Module:end_per_group-2"><c>end_per_group/2</c></seealso>,
+ as described in section
+ <seealso marker="write_test_chapter#repeated_groups">Repeated Groups</seealso>
+ in Writing Test Suites.
+ A failed subgroup (<c>Status == failed</c>) causes the execution of a
sequence to fail in the same way a test case does.</p>
</section>
</chapter>