From 283aa83c7ee83221aeef590afb56a24507bbe35b Mon Sep 17 00:00:00 2001 From: Peter Andersson Date: Tue, 20 Nov 2012 17:33:03 +0100 Subject: Document the new group search feature --- lib/common_test/doc/src/run_test_chapter.xml | 196 ++++++++++++++++++++++++--- 1 file changed, 180 insertions(+), 16 deletions(-) (limited to 'lib/common_test/doc/src/run_test_chapter.xml') diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index ea62df27cc..a0b2c96006 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -119,7 +119,7 @@ -suite ]]> -suite - -group -case ]]> + -group -case ]]>

Examples:

$ ct_run -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST

@@ -137,6 +137,8 @@

$ ct_run -suite ./testdir/x_SUITE ./testdir/y_SUITE

+

For more details on test case group execution, please see below.

+

Other flags that may be used with ct_run:

]]>, specifies where the HTML log files are to be written. @@ -267,6 +269,163 @@ ct manual page.

+ +
+ Test case group execution + +

With the ct_run flag, or ct:run_test/1 option group, + one or more test case groups can be specified, optionally in combination + with specific test cases. The syntax for specifying groups is as follows + (on the command line):

+ + 
+       [-case ]]]>
+

or (in the Erlang shell):

+ 
+       ct:run_test([{group,GroupsNamesOrPaths}, {case,Cases}]).]]>
+ +

The group_names_or_paths parameter specifies either one + or more group names and/or one or more group paths. At start up, + Common Test will search for matching groups in the group definitions + tree (i.e. the list returned from Suite:groups/0, please see the + Test case groups + chapter for details). + Given a group name, say g, Common Test will search for all paths + that lead to g. By path here we mean a sequence of nested groups, + all of which have to be followed in order to get from the top level + group to g. Actually, what Common Test needs to do in order to + execute the test cases in group g, is to call the + init_per_group/2 function for each group in the path to + g, as well as all corresponding end_per_group/2 + functions afterwards. The obvious reason for this is that the configuration + of a test case in g (and its Config input data) depends on + init_per_testcase(TestCase, Config) and its return value, which + in turn depends on init_per_group(g, Config) and its return value, + which in turn depends on init_per_group/2 of the group above + g, etc, all the way up to the top level group.

+ +

As you may have already realized, this means that if there is more than + one way to locate a group (and its test cases) in a path, the result of the + group search operation is a number of tests, all of which will be performed. + Common Test actually interprets a group specification that consists of a + single name this way:

+ +

"Search and find all paths in the group definitions tree that lead + to the specified group and, for each path, create a test which (1) executes + all configuration functions in the path to the specified group, then (2) + executes all - or all matching - test cases in this group, as well as (3) + all - or all matching - test cases in all sub groups of the group". +

+ +

It is also possible for the user to specify a specific group path with + the group_names_or_paths parameter. With this type of specification it's + possible to avoid execution of unwanted groups (in otherwise matching paths), + and/or the execution of sub groups. The syntax of the group path is a list of + group names in the path, e.g. on the command line: +

+

$ ct_run -suite "./x_SUITE" -group [g1,g3,g4] -case tc1 tc5

+

or similarly in the Erlang shell (requires a list within the groups list):

+

1> ct:run_test([{suite,"./x_SUITE"}, {group,[[g1,g3,g4]]}, {testcase,[tc1,tc5]}]).

+ +

The last group in the specified path will be the terminating group in + the test, i.e. no sub groups following this group will be executed. In the + example above, g4 is the terminating group, hence Common Test will + execute a test that calls all init configuration functions in the path to + g4, i.e. g1..g3..g4. It will then call test cases tc1 + and tc5 in g4 and finally all end configuration functions in order + g4..g3..g1.

+ +

Note that the group path specification doesn't necessarily + have to include all groups in the path to the terminating group. + Common Test will search for all matching paths if given an incomplete group + path.

+ +

Note also that it's possible to combine group names and group paths with the + group_names_or_paths parameter. Each element is treated as + an individual specification in combination with the cases parameter. + See examples below.

+ +

Examples:

+ 
+      -module(x_SUITE).
+      ...
+      %% The group definitions:      
+      groups() ->
+        [{top1,[],[tc11,tc12,
+	           {sub11,[],[tc12,tc13]},
+	           {sub12,[],[tc14,tc15,
+			      {sub121,[],[tc12,tc16]}]}]},
+
+         {top2,[],[{group,sub21},{group,sub22}]},
+         {sub21,[],[tc21,{group,sub2X2}]},
+         {sub22,[],[{group,sub221},tc21,tc22,{group,sub2X2}]},
+         {sub221,[],[tc21,tc23]},
+         {sub2X2,[],[tc21,tc24]}].
+
+

+

$ ct_run -suite "x_SUITE" -group all

+

1> ct:run_test([{suite,"x_SUITE"}, {group,all}]).

+

Two tests will be executed, one for all cases and all sub groups under top1, + and one for all under top2. (We would get the same result with + -group top1 top2, or {group,[top1,top2]}.

+

+

$ ct_run -suite "x_SUITE" -group top1

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}]).

+

This will execute one test for all cases and sub groups under top1.

+

+

$ ct_run -suite "x_SUITE" -group top1 -case tc12

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc12]}]).

+

This will run a test that executes tc12 in top1 and any sub group + under top1 where it can be found (sub11 and sub121).

+

+

$ ct_run -suite "x_SUITE" -group [top1] -case tc12

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[[top1]]}, {testcase,[tc12]}]).

+

This will execute tc12 only in group top1.

+

+

$ ct_run -suite "x_SUITE" -group top1 -case tc16

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc16]}]).

+

This will search top1 and all its sub groups for tc16 and the result + will be that this test case executes in group sub121. (The specific path: + -group [sub121] or {group,[[sub121]]}, would have given + us the same result in this example).

+

+

$ ct_run -suite "x_SUITE" -group sub12 [sub12]

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[sub12,[sub12]]}]).

+

This will execute two tests, one that includes all cases and sub groups under + sub12, and one with only the test cases in sub12.

+

+

$ ct_run -suite "x_SUITE" -group sub2X2

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[sub2X2]}]).

+

In this example, Common Test will find and execute two tests, one for the path from + top2 to sub2X2 via sub21, and one from top2 to sub2X2 + via sub22.

+

+

$ ct_run -suite "x_SUITE" -group [sub21,sub2X2]

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub21,sub2X2]]}]).

+

Here, by specifying the unique path: top2 -> sub21 -> sub2X2, only one test + is executed. The second possible path from top2 to sub2X2 (above) + will be discarded.

+

+

$ ct_run -suite "x_SUITE" -group [sub22] -case tc22 tc21

+

1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub22]]}, {testcase,[tc22,tc21]}]).

+

In this example only the test cases for sub22 will be executed, and in + reverse order compared to the group definition.

+

+ +

If a test case that belongs to a group (according to the group definition), is executed + without a group specification, i.e. simply by means of (command line):

+

$ ct_run -suite "my_SUITE" -case my_tc

+

or (Erlang shell):

+

1> ct:run_test([{suite,"my_SUITE"}, {testcase,my_tc}]).

+

then Common Test ignores the group definition and executes the test case in the scope of the + test suite only (no group configuration functions are called).

+ +

The group specification feature, exactly as it has been presented in this section, can also + be used in Test + Specifications (with some extra features added). Please see below.

+
+ +
Running the interactive shell mode @@ -398,8 +557,8 @@ terms (e.g. log directory, label, style sheet, auto compilation).

With test specification terms it is possible to state exactly which tests should run and in which order. A test term specifies - either one or more suites, one or more test case groups, or one - or more test cases in a group or suite.

+ either one or more suites, one or more test case groups (possibly nested), + or one or more test cases in a group (or in multiple groups) or in a suite.

An arbitrary number of test terms may be declared in sequence. Common Test will by default compile the terms into one or more tests to be performed in one resulting test run. Note that a term that @@ -418,28 +577,32 @@ are not executed and show up in the HTML log files as SKIPPED.

When a test case group is specified, the resulting test - executes the - init_per_group function, followed by all test cases and - sub groups (including their configuration functions), and + executes the init_per_group function, followed by all test + cases and sub groups (including their configuration functions), and finally the end_per_group function. Also if particular test cases in a group are specified, init_per_group and end_per_group for the group in question are called. If a group which is defined (in Suite:group/0) to - be a sub group of another group, is specified (or particular test + be a sub group of another group, is specified (or if particular test cases of a sub group are), Common Test will call the configuration functions for the top level groups as well as for the sub group in question (making it possible to pass configuration data all the way from init_per_suite down to the test cases in the sub group).

- -

With the GroupSpec element (below) it's possible to specify - group execution properties that will override those specified in the +

The test specification utilizes the same mechanism for specifying + test case groups by means of names and paths, as explained in the + Group Execution + section above, with the addition of the GroupSpec element + described next.

+

The GroupSpec element makes it possible to specify + group execution properties that will override those in the group definition (i.e. in groups/0). Execution properties for sub-groups may be overridden as well. This feature makes it possible to change properties of groups at the time of execution, - without even having to edit the test suite. More detailed documentation, - and examples, can be found in the - + without even having to edit the test suite. The very same + feature is available for group elements in the Suite:all/0 + list. Therefore, more detailed documentation, and examples, can be + found in the Test case groups chapter.

Below is the test specification syntax. Test specifications can @@ -541,8 +704,8 @@ {groups, Dir, Suite, Groups}. {groups, NodeRefs, Dir, Suite, Groups}. - {groups, Dir, Suite, GroupSpec, {cases,Cases}}. - {groups, NodeRefs, Dir, Suite, GroupSpec, {cases,Cases}}. + {groups, Dir, Suite, Groups, {cases,Cases}}. + {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}. {cases, Dir, Suite, Cases}. {cases, NodeRefs, Dir, Suite, Cases}. @@ -590,7 +753,8 @@ Dir = string() Suites = atom() | [atom()] | all Suite = atom() - Groups = GroupSpec | [GroupSpec] | all + Groups = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all + GroupPath = [GroupName] GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec} GroupName = atom() GroupNames = GroupName | [GroupName] -- cgit v1.2.3