From 9230b38f318d916258168fafb2878f225fb3052b Mon Sep 17 00:00:00 2001 From: Peter Andersson Date: Thu, 24 Jan 2013 12:20:02 +0100 Subject: Update documentation OTP-9881 --- lib/common_test/doc/src/run_test_chapter.xml | 817 +++++++++++++++------------ 1 file changed, 456 insertions(+), 361 deletions(-) (limited to 'lib/common_test/doc/src/run_test_chapter.xml') diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index d5f5d89e05..35f89153d3 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -530,374 +530,469 @@
Test Specifications - -

The most flexible way to specify what to test, is to use a so - called test specification. A test specification is a sequence of - Erlang terms. The terms are normally declared in a text file (see - ct:run_test/1), but - may also be passed to Common Test on the form of a list (see - ct:run_testspec/1). - There are two general types of terms: configuration terms and test - specification terms.

-

With configuration terms it is possible to e.g. label the test - run (similar to ct_run -label), evaluate arbitrary expressions - before starting the test, import configuration data (similar to - ct_run -config/-userconfig), specify the top level HTML log - directory (similar to ct_run -logdir), enable code coverage - analysis (similar to ct_run -cover), install Common Test Hooks - (similar to ct_run -ch_hooks), install event_handler plugins - (similar to ct_run -event_handler), specify include directories - that should be passed to the compiler for automatic compilation - (similar to ct_run -include), disable the auto compilation - feature (similar to ct_run -no_auto_compile), set verbosity - levels (similar to ct_run -verbosity), and more.

-

Configuration terms can be combined with ct_run start flags, - or ct:run_test/1 options. The result will for some flags/options - and terms be that the values are merged (e.g. configuration files, - include directories, verbosity levels, silent connections), and for - others that the start flags/options override the test specification - terms (e.g. log directory, label, style sheet, auto compilation).

-

With test specification terms it is possible to state exactly - which tests should run and in which order. A test term specifies - either one or more suites, one or more test case groups (possibly nested), - or one or more test cases in a group (or in multiple groups) or in a suite.

-

An arbitrary number of test terms may be declared in sequence. - Common Test will by default compile the terms into one or more tests - to be performed in one resulting test run. Note that a term that - specifies a set of test cases will "swallow" one that only - specifies a subset of these cases. E.g. the result of merging - one term that specifies that all cases in suite S should be - executed, with another term specifying only test case X and Y in - S, is a test of all cases in S. However, if a term specifying - test case X and Y in S is merged with a term specifying case Z - in S, the result is a test of X, Y and Z in S. To disable this - behaviour, i.e. to instead perform each test sequentially in a "script-like" - manner, the term merge_tests can be set to false in - the test specification.

-

A test term can also specify one or more test suites, groups, - or test cases to be skipped. Skipped suites, groups and cases - are not executed and show up in the HTML log files as - SKIPPED.

-

When a test case group is specified, the resulting test - executes the init_per_group function, followed by all test - cases and sub groups (including their configuration functions), and - finally the end_per_group function. Also if particular - test cases in a group are specified, init_per_group - and end_per_group for the group in question are - called. If a group which is defined (in Suite:group/0) to - be a sub group of another group, is specified (or if particular test - cases of a sub group are), Common Test will call the configuration - functions for the top level groups as well as for the sub group - in question (making it possible to pass configuration data all - the way from init_per_suite down to the test cases in the - sub group).

-

The test specification utilizes the same mechanism for specifying - test case groups by means of names and paths, as explained in the - Group Execution - section above, with the addition of the GroupSpec element - described next.

-

The GroupSpec element makes it possible to specify - group execution properties that will override those in the - group definition (i.e. in groups/0). Execution properties for - sub-groups may be overridden as well. This feature makes it possible to - change properties of groups at the time of execution, - without even having to edit the test suite. The very same - feature is available for group elements in the Suite:all/0 - list. Therefore, more detailed documentation, and examples, can be - found in the - Test case groups chapter.

- -

Below is the test specification syntax. Test specifications can - be used to run tests both in a single test host environment and - in a distributed Common Test environment (Large Scale - Testing). The node parameters in the init term are only - relevant in the latter (see the - Large - Scale Testing chapter for information). For more information - about the various terms, please see the corresponding sections in the - User's Guide, such as e.g. the - ct_run - program for an overview of available start flags - (since most flags have a corresponding configuration term), and - more detailed explanation of e.g. - Logging - (for the verbosity, stylesheet and basic_html terms), - External Configuration Data - (for the config and userconfig terms), - Event - Handling (for the event_handler term), - Common Test Hooks - (for the ct_hooks term), etc.

-

Config terms:

- 
-      {merge_tests, Bool}.
-
-      {define, Constant, Value}.
-
-      {node, NodeAlias, Node}.
-
-      {init, InitOptions}.
-      {init, [NodeAlias], InitOptions}.
-
-      {label, Label}.
-      {label, NodeRefs, Label}.
-
-      {verbosity, VerbosityLevels}.
-      {verbosity, NodeRefs, VerbosityLevels}.
-
-      {stylesheet, CSSFile}.
-      {stylesheet, NodeRefs, CSSFile}.
-
-      {silent_connections, ConnTypes}.
-      {silent_connections, NodeRefs, ConnTypes}.
-
-      {multiply_timetraps, N}.
-      {multiply_timetraps, NodeRefs, N}.
-
-      {scale_timetraps, Bool}.
-      {scale_timetraps, NodeRefs, Bool}.
- 
-      {cover, CoverSpecFile}.
-      {cover, NodeRefs, CoverSpecFile}.
-      
-      {cover_stop, Bool}.
-      {cover_stop, NodeRefs, Bool}.
-
-      {include, IncludeDirs}.
-      {include, NodeRefs, IncludeDirs}.
-
-      {auto_compile, Bool},
-      {auto_compile, NodeRefs, Bool},
-
-      {config, ConfigFiles}.
-      {config, ConfigDir, ConfigBaseNames}.
-      {config, NodeRefs, ConfigFiles}.
-      {config, NodeRefs, ConfigDir, ConfigBaseNames}.
-
-      {userconfig, {CallbackModule, ConfigStrings}}.
-      {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}.
-      
-      {logdir, LogDir}.                                        
-      {logdir, NodeRefs, LogDir}.
-
-      {logopts, LogOpts}.
-      {logopts, NodeRefs, LogOpts}.
-
-      {create_priv_dir, PrivDirOption}.
-      {create_priv_dir, NodeRefs, PrivDirOption}.
-      
-      {event_handler, EventHandlers}.
-      {event_handler, NodeRefs, EventHandlers}.
-      {event_handler, EventHandlers, InitArgs}.
-      {event_handler, NodeRefs, EventHandlers, InitArgs}.
-
-      {ct_hooks, CTHModules}.
-      {ct_hooks, NodeRefs, CTHModules}.
-
-      {enable_builtin_hooks, Bool}.
-      
-      {basic_html, Bool}.
-      {basic_html, NodeRefs, Bool}.
+    

+	General description
+	
The most flexible way to specify what to test, is to use a so
+	called test specification. A test specification is a sequence of
+	Erlang terms. The terms are normally declared in one or more text files
+	(see ct:run_test/1), but
+	may also be passed to Common Test on the form of a list (see
+	ct:run_testspec/1).
+	There are two general types of terms: configuration terms and test
+	specification terms.

+	
With configuration terms it is possible to e.g. label the test
+	run (similar to ct_run -label), evaluate arbitrary expressions
+	before starting the test, import configuration data (similar to
+	ct_run -config/-userconfig), specify the top level HTML log
+	directory (similar to ct_run -logdir), enable code coverage
+	analysis (similar to ct_run -cover), install Common Test Hooks
+	(similar to ct_run -ch_hooks), install event_handler plugins
+	(similar to ct_run -event_handler), specify include directories
+	that should be passed to the compiler for automatic compilation
+	(similar to ct_run -include), disable the auto compilation
+	feature (similar to ct_run -no_auto_compile), set verbosity
+	levels (similar to ct_run -verbosity), and more.

+	
Configuration terms can be combined with ct_run start flags,
+	or ct:run_test/1 options. The result will for some flags/options
+	and terms be that the values are merged (e.g. configuration files,
+	include directories, verbosity levels, silent connections), and for
+	others that the start flags/options override the test specification
+	terms (e.g. log directory, label, style sheet, auto compilation).

+	
With test specification terms it is possible to state exactly
+	which tests should run and in which order. A test term specifies
+	either one or more suites, one or more test case groups (possibly nested),
+	or one or more test cases in a group (or in multiple groups) or in a suite.

+	
An arbitrary number of test terms may be declared in sequence.
+	Common Test will by default compile the terms into one or more tests 
+	to be performed in one resulting test run. Note that a term that
+	specifies a set of test cases will "swallow" one that only
+	specifies a subset of these cases. E.g. the result of merging
+	one term that specifies that all cases in suite S should be
+	executed, with another term specifying only test case X and Y in
+	S, is a test of all cases in S. However, if a term specifying
+	test case X and Y in S is merged with a term specifying case Z
+	in S, the result is a test of X, Y and Z in S. To disable this
+	behaviour, i.e. to instead perform each test sequentially in a "script-like"
+	manner, the term merge_tests can be set to false in
+	the test specification.

+	
A test term can also specify one or more test suites, groups,
+	or test cases to be skipped. Skipped suites, groups and cases
+	are not executed and show up in the HTML log files as
+	SKIPPED.

+    

+    

+        Using multiple test specification files
+	
+	
When multiple test specification files are given at startup (either
+	with ct_run -spec file1 file2 ... or 
+	ct:run_test([{spec, [File1,File2,...]}])), 
+	Common Test will either execute one test run per specification file, or
+	join the files and perform all tests within one single test run. The first
+	behaviour is the default one. The latter requires that the start
+	flag/option join_suites is provided, e.g.
+	run_test -spec ./my_tests1.ts ./my_tests2.ts -join_suites.

+	
+	
Joining a number of specifications, or running them separately, can
+	also be accomplished with (and may be combined with) test specification
+	file inclusion, described next.

+    

+    

+      Test specification file inclusion
+        
With the specs term (see syntax below), it's possible to have
+	a test specification include other specifications. An included
+	specification may either be joined with the source specification,
+	or used to produce a separate test run (like with the join_specs
+	start flag/option above). Example:

+	
+	%% In specification file "a.spec"
+	{specs, join, ["b.spec", "c.spec"]}.
+	{specs, separate, ["d.spec", "e.spec"]}.
+	%% Config and test terms follow
+	...

+	
In this example, the test terms defined in files "b.spec" and "c.spec"
+	will be joined with the terms in the source specification "a.spec"
+	(if any). The inclusion of specifications "d.spec" and
+	"e.spec" will result in two separate, and independent, test runs (i.e.
+	one for each included specification).

+	
Note that the join option does not imply that the test terms
+	will be merged (see merge_tests above), only that all tests are
+	executed in one single test run.

+	
Joined specifications share common configuration settings, such as
+	the list of config files or include directories.
+	For configuration that can not be combined, such as settings for logdir
+	or verbosity, it is up to the user to ensure there are no clashes
+	when the test specifications are joined. Specifications included with
+	the separate option, do not share configuration settings with the
+	source specification. This is useful e.g. if there are clashing
+	configuration settings in included specifications, making it impossible
+	to join them.

+	
If {merge_tests,true} is set in the source specification
+	(which is the default setting), terms in joined specifications will be
+	merged with terms in the source specification (according to the
+	description of merge_tests above).

+	
Note that it is always the merge_tests setting in the source
+	specification that is used when joined with other specifications.
+	Say e.g. that a source specification A, with tests TA1 and TA2, has
+	{merge_tests,false} set, and it includes another specification,
+	B, with tests TB1 and TB2, that has {merge_tests,true} set.
+	The result will be that the test series: TA1,TA2,merge(TB1,TB2),
+	is executed. The opposite merge_tests settings would result in the
+	following the test series: merge(merge(TA1,TA2),TB1,TB2).

+	
The specs term may of course be used to nest specifications,
+	i.e. have one specification include other specifications, which in turn
+	include others, etc.

+    

+      

+	Test case groups
+
+	
When a test case group is specified, the resulting test
+	executes the init_per_group function, followed by all test
+	cases and sub groups (including their configuration functions), and
+	finally the end_per_group function. Also if particular
+	test cases in a group are specified, init_per_group
+	and end_per_group for the group in question are
+	called. If a group which is defined (in Suite:group/0) to
+	be a sub group of another group, is specified (or if particular test
+	cases of a sub group are), Common Test will call the configuration
+	functions for the top level groups as well as for the sub group
+	in question (making it possible to pass configuration data all
+	the way from init_per_suite down to the test cases in the
+	sub group).

+	
The test specification utilizes the same mechanism for specifying
+	test case groups by means of names and paths, as explained in the
+	Group Execution
+	section above, with the addition of the GroupSpec element
+	described next.

+	
The GroupSpec element makes it possible to specify
+	group execution properties that will override those in the
+	group definition (i.e. in groups/0). Execution properties for
+	sub-groups may be overridden as well. This feature makes it possible to
+	change properties of groups at the time of execution,
+	without even having to edit the test suite. The very same
+	feature is available for group elements in the Suite:all/0
+	list. Therefore, more detailed documentation, and examples, can be
+	found in the 
+	Test case groups chapter.

+      

 
-      {release_shell, Bool}.
+
+ Test specification syntax + +

Below is the test specification syntax. Test specifications can + be used to run tests both in a single test host environment and + in a distributed Common Test environment (Large Scale + Testing). The node parameters in the init term are only + relevant in the latter (see the + Large + Scale Testing chapter for information). For more information + about the various terms, please see the corresponding sections in the + User's Guide, such as e.g. the + ct_run + program for an overview of available start flags + (since most flags have a corresponding configuration term), and + more detailed explanation of e.g. + Logging + (for the verbosity, stylesheet and basic_html terms), + External Configuration Data + (for the config and userconfig terms), + Event + Handling (for the event_handler term), + Common Test Hooks + (for the ct_hooks term), etc.

+
+

Config terms:

+ 
+	{merge_tests, Bool}.
+	
+	{define, Constant, Value}.
+	
+	{specs, InclSpecsOption, TestSpecs}.
+	
+	{node, NodeAlias, Node}.
+	
+	{init, InitOptions}.
+	{init, [NodeAlias], InitOptions}.
+	
+	{label, Label}.
+	{label, NodeRefs, Label}.
+	
+	{verbosity, VerbosityLevels}.
+	{verbosity, NodeRefs, VerbosityLevels}.
+	
+	{stylesheet, CSSFile}.
+	{stylesheet, NodeRefs, CSSFile}.
+	
+	{silent_connections, ConnTypes}.
+	{silent_connections, NodeRefs, ConnTypes}.
+	
+	{multiply_timetraps, N}.
+	{multiply_timetraps, NodeRefs, N}.
+	
+	{scale_timetraps, Bool}.
+	{scale_timetraps, NodeRefs, Bool}.
+	
+	{cover, CoverSpecFile}.
+	{cover, NodeRefs, CoverSpecFile}.
+	
+	{cover_stop, Bool}.
+	{cover_stop, NodeRefs, Bool}.
+	
+	{include, IncludeDirs}.
+	{include, NodeRefs, IncludeDirs}.
+	
+	{auto_compile, Bool},
+	{auto_compile, NodeRefs, Bool},
+	
+	{config, ConfigFiles}.
+	{config, ConfigDir, ConfigBaseNames}.
+	{config, NodeRefs, ConfigFiles}.
+	{config, NodeRefs, ConfigDir, ConfigBaseNames}.
+	
+	{userconfig, {CallbackModule, ConfigStrings}}.
+	{userconfig, NodeRefs, {CallbackModule, ConfigStrings}}.
+	
+	{logdir, LogDir}.                                        
+	{logdir, NodeRefs, LogDir}.
+	
+	{logopts, LogOpts}.
+	{logopts, NodeRefs, LogOpts}.
+	
+	{create_priv_dir, PrivDirOption}.
+	{create_priv_dir, NodeRefs, PrivDirOption}.
+	
+	{event_handler, EventHandlers}.
+	{event_handler, NodeRefs, EventHandlers}.
+	{event_handler, EventHandlers, InitArgs}.
+	{event_handler, NodeRefs, EventHandlers, InitArgs}.
+	
+	{ct_hooks, CTHModules}.
+	{ct_hooks, NodeRefs, CTHModules}.
+	
+	{enable_builtin_hooks, Bool}.
+	
+	{basic_html, Bool}.
+	{basic_html, NodeRefs, Bool}.
+	
+        {release_shell, Bool}.
+

Test terms:

- 
-      {suites, Dir, Suites}.                                
-      {suites, NodeRefs, Dir, Suites}.
-      
-      {groups, Dir, Suite, Groups}.
-      {groups, NodeRefs, Dir, Suite, Groups}.
-
-      {groups, Dir, Suite, Groups, {cases,Cases}}.
-      {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}.
-
-      {cases, Dir, Suite, Cases}.                           
-      {cases, NodeRefs, Dir, Suite, Cases}.
-
-      {skip_suites, Dir, Suites, Comment}.
-      {skip_suites, NodeRefs, Dir, Suites, Comment}.
-
-      {skip_groups, Dir, Suite, GroupNames, Comment}.
-      {skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}.
-      
-      {skip_cases, Dir, Suite, Cases, Comment}.
-      {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}.
- + 
+	{suites, Dir, Suites}.                                
+	{suites, NodeRefs, Dir, Suites}.
+	
+	{groups, Dir, Suite, Groups}.
+	{groups, NodeRefs, Dir, Suite, Groups}.
+	
+	{groups, Dir, Suite, Groups, {cases,Cases}}.
+	{groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}.
+	
+	{cases, Dir, Suite, Cases}.                           
+	{cases, NodeRefs, Dir, Suite, Cases}.
+	
+	{skip_suites, Dir, Suites, Comment}.
+	{skip_suites, NodeRefs, Dir, Suites, Comment}.
+	
+	{skip_groups, Dir, Suite, GroupNames, Comment}.
+	{skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}.
+	
+	{skip_cases, Dir, Suite, Cases, Comment}.
+        {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}.
+

Types:

- 
-      Bool            = true | false
-      Constant        = atom()
-      Value           = term()
-      NodeAlias       = atom()
-      Node            = node()
-      NodeRef         = NodeAlias | Node | master
-      NodeRefs        = all_nodes | [NodeRef] | NodeRef
-      InitOptions     = term()
-      Label           = atom() | string()
-      VerbosityLevels = integer() | [{Category,integer()}]
-      Category        = atom()
-      CSSFile         = string()
-      ConnTypes       = all | [atom()]
-      N               = integer()
-      CoverSpecFile   = string()
-      IncludeDirs     = string() | [string()]
-      ConfigFiles     = string() | [string()]
-      ConfigDir       = string()
-      ConfigBaseNames = string() | [string()]
-      CallbackModule  = atom()
-      ConfigStrings   = string() | [string()]
-      LogDir          = string()
-      LogOpts         = [term()]
-      PrivDirOption   = auto_per_run | auto_per_tc | manual_per_tc
-      EventHandlers   = atom() | [atom()]
-      InitArgs        = [term()]
-      CTHModules      = [CTHModule |
-                         {CTHModule, CTHInitArgs} |
-                         {CTHModule, CTHInitArgs, CTHPriority}]
-      CTHModule       = atom()
-      CTHInitArgs     = term()
-      Dir             = string()
-      Suites          = atom() | [atom()] | all
-      Suite           = atom()
-      Groups          = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all
-      GroupPath       = [GroupName]
-      GroupSpec       = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec}
-      GroupName       = atom()
-      GroupNames      = GroupName | [GroupName]
-      Cases           = atom() | [atom()] | all
-      Comment         = string() | ""
- -

The difference between the config terms above, is that with - ConfigDir, ConfigBaseNames is a list of base names, - i.e. without directory paths. ConfigFiles must be full names, - including paths. E.g, these two terms have the same meaning:

- 
-      {config, ["/home/testuser/tests/config/nodeA.cfg",
-                "/home/testuser/tests/config/nodeB.cfg"]}.
-
-      {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.
- -

Any relative paths specified in the test specification, will be - relative to the directory which contains the test specification file, if - ct_run -spec TestSpecFile ... or - ct:run:test([{spec,TestSpecFile},...]) - executes the test. The path will be relative to the top level log directory, if - ct:run:testspec(TestSpec) executes the test.

 - -

The define term introduces a constant, which is used to - replace the name Constant with Value, wherever it's found in - the test specification. This replacement happens during an initial iteration - through the test specification. Constants may be used anywhere in the test - specification, e.g. in arbitrary lists and tuples, and even in strings - and inside the value part of other constant definitions! A constant can - also be part of a node name, but that is the only place where a constant - can be part of an atom.

- -

For the sake of readability, the name of the constant must always - begin with an upper case letter, or a $, ?, or _. - This also means that it must always be single quoted (obviously, since - the constant name is actually an atom, not text).

 - -

The main benefit of constants is that they can be used to reduce the size - (and avoid repetition) of long strings, such as file paths. Compare these - terms:

- - 
-      %% 1a. no constant
-      {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.
-      {suites, "/home/testuser/tests/suites", all}.
-      
-      %% 1b. with constant
-      {define, 'TESTDIR', "/home/testuser/tests"}.
-      {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}.
-      {suites, "'TESTDIR'/suites", all}.
-
-      %% 2a. no constants
-      {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}.
-      {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}.
-
-      %% 2b. with constants
-      {define, 'NODE', testnode}.
-      {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}.
-      {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}.
-      {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.
- -

Constants make the test specification term alias, in previous - versions of Common Test, redundant. This term has been deprecated but will - remain supported in upcoming Common Test releases. Replacing alias - terms with define is strongly recommended though! Here's an example - of such a replacement:

+ 
+	Bool            = true | false
+	Constant        = atom()
+	Value           = term()
+	InclSpecsOption = join | separate
+	TestSpecs       = string() | [string()]
+	NodeAlias       = atom()
+	Node            = node()
+	NodeRef         = NodeAlias | Node | master
+	NodeRefs        = all_nodes | [NodeRef] | NodeRef
+	InitOptions     = term()
+	Label           = atom() | string()
+	VerbosityLevels = integer() | [{Category,integer()}]
+	Category        = atom()
+	CSSFile         = string()
+	ConnTypes       = all | [atom()]
+	N               = integer()
+	CoverSpecFile   = string()
+	IncludeDirs     = string() | [string()]
+	ConfigFiles     = string() | [string()]
+	ConfigDir       = string()
+	ConfigBaseNames = string() | [string()]
+	CallbackModule  = atom()
+	ConfigStrings   = string() | [string()]
+	LogDir          = string()
+	LogOpts         = [term()]
+	PrivDirOption   = auto_per_run | auto_per_tc | manual_per_tc
+	EventHandlers   = atom() | [atom()]
+	InitArgs        = [term()]
+	CTHModules      = [CTHModule |
+	                   {CTHModule, CTHInitArgs} |
+	                   {CTHModule, CTHInitArgs, CTHPriority}]
+	CTHModule       = atom()
+	CTHInitArgs     = term()
+	Dir             = string()
+	Suites          = atom() | [atom()] | all
+	Suite           = atom()
+	Groups          = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all
+	GroupPath       = [GroupName]
+	GroupSpec       = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec}
+	GroupName       = atom()
+	GroupNames      = GroupName | [GroupName]
+	Cases           = atom() | [atom()] | all
+        Comment         = string() | ""
+ +
+

The difference between the config terms above, is that with + ConfigDir, ConfigBaseNames is a list of base names, + i.e. without directory paths. ConfigFiles must be full names, + including paths. E.g, these two terms have the same meaning:

+ 
+	  {config, ["/home/testuser/tests/config/nodeA.cfg",
+	            "/home/testuser/tests/config/nodeB.cfg"]}.
+	  
+	  {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.
+ +

Any relative paths specified in the test specification, will be + relative to the directory which contains the test specification file, if + ct_run -spec TestSpecFile ... or + ct:run:test([{spec,TestSpecFile},...]) + executes the test. The path will be relative to the top level log directory, if + ct:run:testspec(TestSpec) executes the test.

 +
- 
-      %% using the old alias term
-      {config, "/home/testuser/tests/config/nodeA.cfg"}.
-      {alias, suite_dir, "/home/testuser/tests/suites"}.
-      {groups, suite_dir, x_SUITE, group1}.
-
-      %% replacing with constants
-      {define, 'TestDir', "/home/testuser/tests"}.
-      {define, 'CfgDir', "'TestDir'/config"}.
-      {define, 'SuiteDir', "'TestDir'/suites"}.
-      {config, 'CfgDir', "nodeA.cfg"}.
-      {groups, 'SuiteDir', x_SUITE, group1}.
- -

Actually, constants could well replace the node term too, but - this still has declarative value, mainly when used in combination - with NodeRefs == all_nodes (see types above).

- -

Here follows a simple test specification example:

- 
-      {define, 'Top', "/home/test"}.
-      {define, 'T1', "'Top'/t1"}.
-      {define, 'T2', "'Top'/t2"}.
-      {define, 'T3', "'Top'/t3"}.
-      {define, 'CfgFile', "config.cfg"}.
+	

+	  Constants
+
+	  
The define term introduces a constant, which is used to
+	  replace the name Constant with Value, wherever it's found in
+	  the test specification. This replacement happens during an initial iteration
+	  through the test specification. Constants may be used anywhere in the test
+	  specification, e.g. in arbitrary lists and tuples, and even in strings
+	  and inside the value part of other constant definitions! A constant can
+	  also be part of a node name, but that is the only place where a constant
+	  can be part of an atom.

+	  
+	  
For the sake of readability, the name of the constant must always
+	  begin with an upper case letter, or a $, ?, or _.
+	  This also means that it must always be single quoted (obviously, since
+	  the constant name is actually an atom, not text).

+	  
+	  
The main benefit of constants is that they can be used to reduce the size
+	  (and avoid repetition) of long strings, such as file paths. Compare these
+	  terms:

+	  
+	  
+	    %% 1a. no constant
+	    {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.
+	    {suites, "/home/testuser/tests/suites", all}.
+	    
+	    %% 1b. with constant
+	    {define, 'TESTDIR', "/home/testuser/tests"}.
+	    {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}.
+	    {suites, "'TESTDIR'/suites", all}.
+	    
+	    %% 2a. no constants
+	    {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}.
+	    {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}.
+	    
+	    %% 2b. with constants
+	    {define, 'NODE', testnode}.
+	    {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}.
+	    {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}.
+	    {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.

+
+	    
Constants make the test specification term alias, in previous
+	    versions of Common Test, redundant. This term has been deprecated but will
+	    remain supported in upcoming Common Test releases. Replacing alias
+	    terms with define is strongly recommended though! Here's an example
+	    of such a replacement:

+	    
+	    
+	      %% using the old alias term
+	      {config, "/home/testuser/tests/config/nodeA.cfg"}.
+	      {alias, suite_dir, "/home/testuser/tests/suites"}.
+	      {groups, suite_dir, x_SUITE, group1}.
+	      
+	      %% replacing with constants
+	      {define, 'TestDir', "/home/testuser/tests"}.
+	      {define, 'CfgDir', "'TestDir'/config"}.
+	      {define, 'SuiteDir', "'TestDir'/suites"}.
+	      {config, 'CfgDir', "nodeA.cfg"}.
+	      {groups, 'SuiteDir', x_SUITE, group1}.

+	      
+	      
Actually, constants could well replace the node term too, but
+	      this still has declarative value, mainly when used in combination
+	      with NodeRefs == all_nodes (see types above).

+	

 
-      {logdir, "'Top'/logs"}.
-      
-      {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}.
-      
-      {suites, 'T1', all}.
-      {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
-      {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
-      {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}.
-      
-      {suites, 'T2', [t2B_SUITE,t2C_SUITE]}.
-      {cases, 'T2', t2A_SUITE, [test4,test1,test7]}.
-      
-      {skip_suites, 'T3', all, "Not implemented"}.
+
+ Example + +

Here follows a simple test specification example:

+ 
+	    {define, 'Top', "/home/test"}.
+	    {define, 'T1', "'Top'/t1"}.
+	    {define, 'T2', "'Top'/t2"}.
+	    {define, 'T3', "'Top'/t3"}.
+	    {define, 'CfgFile', "config.cfg"}.
+	    
+	    {logdir, "'Top'/logs"}.
+	    
+	    {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}.
+	    
+	    {suites, 'T1', all}.
+	    {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
+	    {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
+	    {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}.
+	    
+	    {suites, 'T2', [t2B_SUITE,t2C_SUITE]}.
+	    {cases, 'T2', t2A_SUITE, [test4,test1,test7]}.
+	    
+	    {skip_suites, 'T3', all, "Not implemented"}.
+ +

The example specifies the following:

+ + The specified logdir directory will be used for storing + the HTML log files (in subdirectories tagged with node name, + date and time). + The variables in the specified test system config files will be + imported for the test. + The first test to run includes all suites for system t1. Excluded from + the test are however the t1B and t1D suites. Also test cases test3 and + test4 in t1A as well as the test1 case in t1C are excluded from + the test. + Secondly, the test for system t2 should run. The included suites are + t2B and t2C. Included are also test cases test4, test1 and test7 in suite + t2A. Note that the test cases will be executed in the specified order. + Lastly, all suites for systems t3 are to be completely skipped and this + should be explicitly noted in the log files. + +
-

The example specifies the following:

- - The specified logdir directory will be used for storing - the HTML log files (in subdirectories tagged with node name, - date and time). - The variables in the specified test system config files will be - imported for the test. - The first test to run includes all suites for system t1. Excluded from - the test are however the t1B and t1D suites. Also test cases test3 and - test4 in t1A as well as the test1 case in t1C are excluded from - the test. - Secondly, the test for system t2 should run. The included suites are - t2B and t2C. Included are also test cases test4, test1 and test7 in suite - t2A. Note that the test cases will be executed in the specified order. - Lastly, all suites for systems t3 are to be completely skipped and this - should be explicitly noted in the log files. - -

With the init term it's possible to specify initialization options - for nodes defined in the test specification. Currently, there are options - to start the node and/or to evaluate any function on the node. - See the Automatic startup of - the test target nodes chapter for details.

-

It is possible for the user to provide a test specification that - includes (for Common Test) unrecognizable terms. If this is desired, - the -allow_user_terms flag should be used when starting tests with - ct_run. This forces Common Test to ignore unrecognizable terms. - Note that in this mode, Common Test is not able to check the specification - for errors as efficiently as if the scanner runs in default mode. - If ct:run_test/1 is used for starting the tests, the relaxed scanner - mode is enabled by means of the tuple: {allow_user_terms,true}

+
+ The init term +

With the init term it's possible to specify initialization options + for nodes defined in the test specification. Currently, there are options + to start the node and/or to evaluate any function on the node. + See the Automatic startup of + the test target nodes chapter for details.

+
+
+ User specific terms +

It is possible for the user to provide a test specification that + includes (for Common Test) unrecognizable terms. If this is desired, + the -allow_user_terms flag should be used when starting tests with + ct_run. This forces Common Test to ignore unrecognizable terms. + Note that in this mode, Common Test is not able to check the specification + for errors as efficiently as if the scanner runs in default mode. + If ct:run_test/1 is used + for starting the tests, the relaxed scanner + mode is enabled by means of the tuple: {allow_user_terms,true}

+
-- cgit v1.2.3