diff options
Diffstat (limited to 'lib/common_test')
| -rw-r--r-- | lib/common_test/doc/src/common_test_app.xml | 222 | ||||
| -rw-r--r-- | lib/common_test/doc/src/write_test_chapter.xml | 66 |
2 files changed, 222 insertions, 66 deletions
diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index f58b2ab0a9..c7f6c7ce5c 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -69,12 +69,25 @@ <funcs> <func> - <name>Module:all() -> TestCases | {skip,Reason} </name> - <fsummary>Returns the list of all test cases in the module.</fsummary> + <name>Module:all() -> Tests | {skip,Reason} </name> + <fsummary>Returns the list of all test case groups and test cases + in the module.</fsummary> <type> - <v>TestCases = [atom() | {group,GroupName}]</v> - <v>Reason = term()</v> + <v>Tests = [TestCase | {group,GroupName} | + {group,GroupName,Properties} | + {group,GroupName,Properties,SubGroups}]</v> + <v>TestCase = atom()</v> <v>GroupName = atom()</v> + <v>Properties = [parallel | sequence | Shuffle | {RepeatType,N}] | + default</v> + <v>SubGroups = [{GroupName,Properties} | + {GroupName,Properties,SubGroups}]</v> + <v>Shuffle = shuffle | {shuffle,Seed}</v> + <v>Seed = {integer(),integer(),integer()}</v> + <v>RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | + repeat_until_any_ok | repeat_until_any_fail</v> + <v>N = integer() | forever</v> + <v>Reason = term()</v> </type> <desc> @@ -85,9 +98,14 @@ This list also specifies the order the cases and groups will be executed by Common Test. A test case is represented by an atom, the name of the test case function. A test case group is - represented by a <c>{group,GroupName}</c> tuple, where - <c>GroupName</c>, an atom, is the name of the group (defined - with <c>groups/0</c>).</p> + represented by a <c>group</c> tuple, where <c>GroupName</c>, + an atom, is the name of the group (defined in <c>groups/0</c>). + Execution properties for groups may also be specified, both + for a top level group and for any of its sub-groups. + Group execution properties specified here, will override + properties in the group definition (see <c>groups/0</c>). + (With value <c>default</c>, the group definition properties + will be used).</p> <p> If <c>{skip,Reason}</c> is returned, all test cases in the module will be skipped, and the <c>Reason</c> will @@ -120,14 +138,16 @@ <desc> <p> OPTIONAL </p> - <p>See <seealso marker="write_test_chapter#test_case_groups">Test case + <p>Function for defining test case groups. Please see + <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso> in the User's Guide for details.</p> </desc> </func> <func> <name>Module:suite() -> [Info] </name> - <fsummary>Test suite info function (providing default data for the suite).</fsummary> + <fsummary>Test suite info function (providing default data + for the suite).</fsummary> <type> <v> Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | @@ -160,11 +180,11 @@ <p>This is the test suite info function. It is supposed to return a list of tagged tuples that specify various properties - regarding the execution of this test suite (common for all + related to the execution of this test suite (common for all test cases in the suite).</p> <p>The <c>timetrap</c> tag sets the maximum time each - test case is allowed to take (including <c>init_per_testcase/2</c> + test case is allowed to execute (including <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c>). If the timetrap time is exceeded, the test case fails with reason <c>timetrap_timeout</c>. If a <c>TimeFunc</c> function is specified, @@ -172,9 +192,9 @@ <c>TimeVal</c> format.</p> <p>The <c>require</c> tag specifies configuration variables - that are required by test cases in the suite. If the required - configuration variables are not found in any of the - configuration files, all test cases are skipped. For more + that are required by test cases (and/or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases are skipped. For more information about the 'require' functionality, see the reference manual for the function <c>ct:require/[1,2]</c>.</p> @@ -196,8 +216,9 @@ </func> <func> - <name>Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | {skip_and_save,Reason,SaveConfig}</name> - <fsummary>Test suite initialization.</fsummary> + <name>Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | + {skip_and_save,Reason,SaveConfig}</name> + <fsummary>Test suite initializations.</fsummary> <type> <v> Config = NewConfig = SaveConfig = [{Key,Value}]</v> <v> Key = atom()</v> @@ -208,15 +229,15 @@ <p> OPTIONAL </p> - <p>This function is called as the first function in the - suite. It typically contains initialization which is common for + <p>This configuration function is called as the first function in the + suite. It typically contains initializations which are common for all test cases in the suite, and which shall only be done - once. The <c>Config</c> parameter is the configuration + once. The <c>Config</c> parameter is the configuration data which can be modified here. Whatever is returned from this function is given as <c>Config</c> to all configuration functions and test cases in the suite. If <c>{skip,Reason}</c> - is returned, all test cases in the suite will be skipped and <c>Reason</c> - printed in the overview log for the suite.</p> + is returned, all test cases in the suite will be skipped + and <c>Reason</c> printed in the overview log for the suite.</p> <p>For information on <c>save_config</c> and <c>skip_and_save</c>, please see <seealso marker="dependencies_chapter#save_config">Dependencies @@ -224,29 +245,106 @@ </desc> </func> - <func> - <name>Module:end_per_suite(Config) -> void() | {save_config,SaveConfig}</name> - <fsummary>Test suite finalization. </fsummary> - <type> - <v> Config = SaveConfig = [{Key,Value}]</v> - <v> Key = atom()</v> - <v> Value = term()</v> - </type> + <func> + <name>Module:end_per_suite(Config) -> void() | + {save_config,SaveConfig}</name> + <fsummary>Test suite finalization. </fsummary> + <type> + <v> Config = SaveConfig = [{Key,Value}]</v> + <v> Key = atom()</v> + <v> Value = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + <p>This function is called as the last test case in the + suite. It is meant to be used for cleaning up after + <c>init_per_suite/1</c>. + For information on <c>save_config</c>, please see + <seealso marker="dependencies_chapter#save_config">Dependencies + between Test Cases and Suites</seealso> in the User's Guide.</p> + </desc> + </func> + + <func> + <name>Module:group(GroupName) -> [Info] </name> + <fsummary>Test case group info function (providing default data + for a test case group, i.e. its test cases and sub-groups).</fsummary> + <type> + <v> Info = {timetrap,Time} | {require,Required} | + {require,Name,Required} | {userdata,UserData} | + {silent_connections,Conns} | {stylesheet,CSSFile} | + {ct_hooks, CTHs}</v> + <v> Time = TimeVal | TimeFunc</v> + <v> TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | + {hours,integer()}</v> + <v> TimeFunc = {Mod,Func,Args} | Fun</v> + <v> MilliSec = integer()</v> + <v> Mod = atom()</v> + <v> Func = atom()</v> + <v> Args = list()</v> + <v> Fun = fun()</v> + <v> Required = Key | {Key,SubKeys}</v> + <v> Key = atom()</v> + <v> SubKeys = SubKey | [SubKey]</v> + <v> SubKey = atom()</v> + <v> Name = atom()</v> + <v> UserData = term()</v> + <v> Conns = [atom()]</v> + <v> CSSFile = string()</v> + <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} | + {CTHModule, CTHInitArgs, CTHPriority}]</v> + <v> CTHModule = atom()</v> + <v> CTHInitArgs = term()</v> + </type> <desc> + <p> OPTIONAL </p> - <p>This function is called as the last test case in the - suite. It is meant to be used for cleaning up after <c>init_per_suite/1</c>. - For information on <c>save_config</c>, please see - <seealso marker="dependencies_chapter#save_config">Dependencies between - Test Cases and Suites</seealso> in the User's Guide.</p> - </desc> + <p>This is the test case group info function. It is supposed to + return a list of tagged tuples that specify various properties + related to the execution of a test case group (i.e. its test cases + and sub-groups). Properties set by <c>groups/1</c> override + properties with the same key that have been previously set by + <c>suite/0</c>.</p> + + <p>The <c>timetrap</c> tag sets the maximum time each + test case is allowed to execute (including <c>init_per_testcase/2</c> + and <c>end_per_testcase/2</c>). If the timetrap time is + exceeded, the test case fails with reason + <c>timetrap_timeout</c>. If a <c>TimeFunc</c> function is specified, + it will be called initially and must return a value on + <c>TimeVal</c> format.</p> + + <p>The <c>require</c> tag specifies configuration variables + that are required by test cases (and/or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases in this group are skipped. + For more information about the 'require' functionality, see the + reference manual for the function + <c>ct:require/[1,2]</c>.</p> + + <p>With <c>userdata</c>, it is possible for the user to + specify arbitrary test case group related information which can be + read by calling <c>ct:userdata/2</c>.</p> + + <p>The <c>ct_hooks</c> tag specifies which + <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> + are to be run together with this suite.</p> + + <p>Other tuples than the ones defined will simply be ignored.</p> + + <p>For more information about the test case group info function, + see <seealso marker="write_test_chapter#suite">Test + case group info function</seealso> in the User's Guide.</p> + </desc> </func> <func> - <name>Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason}</name> - <fsummary>Test case group initialization.</fsummary> + <name>Module:init_per_group(GroupName, Config) -> NewConfig | + {skip,Reason}</name> + <fsummary>Test case group initializations.</fsummary> <type> <v> GroupName = atom()</v> <v> Config = NewConfig = [{Key,Value}]</v> @@ -258,16 +356,16 @@ <p> OPTIONAL </p> - <p>This function is called before execution of a test case group. - It typically contains initialization which is common for - all test cases in the group, and which shall only be performed - once. <c>GroupName</c> is the name of the group, as specified in - the group definition (see <c>groups/0</c>). The <c>Config</c> - parameter is the configuration which can be modified here. - Whatever is returned from this function is given as <c>Config</c> - to all test cases in the group. If <c>{skip,Reason}</c> is returned, - all test cases in the group will be skipped and <c>Reason</c> printed - in the overview log for the group.</p> + <p>This configuration function is called before execution of a + test case group. It typically contains initializations which are + common for all test cases and sub-groups in the group, and which + shall only be performed once. <c>GroupName</c> is the name of the + group, as specified in the group definition (see <c>groups/0</c>). The + <c>Config</c> parameter is the configuration data which can be modified + here. The return value of this function is given as <c>Config</c> + to all test cases and sub-groups in the group. If <c>{skip,Reason}</c> + is returned, all test cases in the group will be skipped and + <c>Reason</c> printed in the overview log for the group.</p> <p>For information about test case groups, please see <seealso marker="write_test_chapter#test_case_groups">Test case @@ -276,7 +374,8 @@ </func> <func> - <name>Module:end_per_group(GroupName, Config) -> void() | {return_group_result,Status}</name> + <name>Module:end_per_group(GroupName, Config) -> void() | + {return_group_result,Status}</name> <fsummary>Test case group finalization.</fsummary> <type> <v> GroupName = atom()</v> @@ -305,7 +404,7 @@ <func> <name>Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason} | {skip,Reason}</name> - <fsummary>Test case initialization.</fsummary> + <fsummary>Test case initializations.</fsummary> <type> <v> TestCase = atom()</v> <v> Config = NewConfig = [{Key,Value}]</v> @@ -385,10 +484,13 @@ <p>This is the test case info function. It is supposed to return a list of tagged tuples that specify various properties - regarding the execution of this particular test case.</p> + related to the execution of this particular test case. + Properties set by <c>Testcase/0</c> override + properties that have been previously set for the test case + by <c>group/1</c> or <c>suite/0</c>.</p> <p>The <c>timetrap</c> tag sets the maximum time the - test case is allowed to take. If the timetrap time is + test case is allowed to execute. If the timetrap time is exceeded, the test case fails with reason <c>timetrap_timeout</c>. <c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> are included in the @@ -397,16 +499,16 @@ and must return a value on <c>TimeVal</c> format.</p> <p>The <c>require</c> tag specifies configuration variables - that are required by the test case. If the required - configuration variables are not found in any of the + that are required by the test case (and/or <c>init/end_per_testcase/2</c>). + If the required configuration variables are not found in any of the configuration files, the test case is skipped. For more information about the 'require' functionality, see the reference manual for the function <c>ct:require/[1,2]</c>.</p> <p>If <c>timetrap</c> and/or <c>require</c> is not set, the - default values specified in the <c>suite/0</c> return list - will be used.</p> + default values specified by <c>suite/0</c> (or + <c>group/1</c>) will be used.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test case related information which can be @@ -438,12 +540,12 @@ <p>This is the implementation of a test case. Here you must call the functions you want to test, and do whatever you need to check the result. If something fails, make sure the - function causes a runtime error, or call <c>ct:fail/[0,1]</c> - (which also causes the test case process to crash).</p> + function causes a runtime error, or call <c>ct:fail/1/2</c> + (which also causes the test case process to terminate).</p> - <p>Elements from the <c>Config</c> parameter can be read - with the <c>?config</c> macro. The <c>config</c> - macro is defined in <c>ct.hrl</c></p> + <p>Elements from the <c>Config</c> list can e.g. be read + with <c>proplists:get_value/2</c> (or the macro <c>?config</c> + defined in <c>ct.hrl</c>).</p> <p>You can return <c>{skip,Reason}</c> if you decide not to run the test case after all. <c>Reason</c> will then be diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index e35888e68f..b8487ea2d4 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -68,7 +68,7 @@ <p>Each test suite module must export the function <c>all/0</c> which returns the list of all test case groups and test cases - in that module. + to be executed in that module. </p> </section> @@ -450,11 +450,65 @@ <pre> all() -> [testcase1, {group,group1}, testcase2, {group,group2}].</pre> - <p>Properties may be combined so that e.g. if <c>shuffle</c>, - <c>repeat_until_any_fail</c> and <c>sequence</c> are all specified, the test - cases in the group will be executed repeatedly and in random order until - a test case fails, when execution is immediately stopped and the rest of - the cases skipped.</p> + <p>It is also possible to specify execution properties with a group + tuple in <c>all/0</c>: <c>{group,GroupName,Properties}</c>. These + properties will override those specified in the group definition (see + <c>groups/0</c> above). This way, it's possible to run the same set of tests, + but with different properties, without having to make copies of the group + definition in question.</p> + + <p>If a group contains sub-groups, the execution properties for these may + also be specified in the group tuple: + <c>{group,GroupName,Properties,SubGroups}</c>, where <c>SubGroups</c> + is a list of tuples, <c>{GroupName,Properties}</c>, or + <c>{GroupName,Properties,SubGroups}</c>, representing the sub-groups. + Any sub-groups defined in <c>group/0</c> for a group, that are not specified + in the <c>SubGroups</c> list, will simply execute with their pre-defined + properties.</p> + + <p>Example:</p> + <pre> + groups() -> {tests1, [], [{tests2, [], [t2a,t2b]}, + {tests3, [], [t31,t3b]}]}.</pre> + <p>To execute group 'tests1' twice with different properties for 'tests2' + each time:</p> + <pre> + all() -> + [{group, tests1, default, [{tests2, [parallel]}]}, + {group, tests1, default, [{tests2, [shuffle,{repeat,10}]}]}].</pre> + <p>Note that this is equivalent to this specification:</p> + <pre> + all() -> + [{group, tests1, default, [{tests2, [parallel]}, + {tests3, default}]}, + {group, tests1, default, [{tests2, [shuffle,{repeat,10}]}, + {tests3, default}]}].</pre> + <p>The value <c>default</c> states that the pre-defined properties + should be used.</p> + <p>Here's an example of how to override properties in a scenario + with deeply nested groups:</p> + <pre> + groups() -> + [{tests1, [], [{group, tests2}]}, + {tests2, [], [{group, tests3}]}, + {tests3, [{repeat,2}], [t3a,t3b,t3c]}]. + + all() -> + [{group, tests1, default, + [{tests2, default, + [{tests3, [parallel,{repeat,100}]}]}]}].</pre> + + <p>The syntax described above may also be used in Test Specifications + in order to change properties of groups at the time of execution, + without even having to edit the test suite (please see the + <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso> chapter for more info).</p> + + <p>As illustrated above, properties may be combined. If e.g. + <c>shuffle</c>, <c>repeat_until_any_fail</c> and <c>sequence</c> + are all specified, the test cases in the group will be executed + repeatedly, and in random order, until a test case fails. Then + execution is immediately stopped and the rest of the cases skipped.</p> <p>Before execution of a group begins, the configuration function <c>init_per_group(GroupName, Config)</c> is called (the function is |