From 2ef840647acadb489d54332f6a218dcf2e629ff9 Mon Sep 17 00:00:00 2001 From: tmanevik Date: Wed, 18 Nov 2015 18:24:10 +0100 Subject: Common Test: Editorial changes 1 Conflicts: lib/common_test/doc/src/ct_hooks_chapter.xml lib/common_test/doc/src/event_handler_chapter.xml lib/common_test/doc/src/run_test_chapter.xml --- lib/common_test/doc/src/ct_master_chapter.xml | 291 +++++++++++++------------- 1 file changed, 147 insertions(+), 144 deletions(-) (limited to 'lib/common_test/doc/src/ct_master_chapter.xml') diff --git a/lib/common_test/doc/src/ct_master_chapter.xml b/lib/common_test/doc/src/ct_master_chapter.xml index 16492f39b8..99555164e0 100644 --- a/lib/common_test/doc/src/ct_master_chapter.xml +++ b/lib/common_test/doc/src/ct_master_chapter.xml @@ -22,7 +22,7 @@ - Using Common Test for Large Scale Testing + Using Common Test for Large-Scale Testing Peter Andersson @@ -33,217 +33,220 @@
General -

Large scale automated testing requires running multiple independent +

Large-scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running - a number of Common Test nodes on one or more hosts, testing - different target systems. Configuring, starting and controlling the + some Common Test nodes on one or more hosts, testing + different target systems. Configuring, starting, and controlling the test nodes independently can be a cumbersome operation. To aid - this kind of automated large scale testing, CT offers a master test - node component, CT Master, that handles central configuration and control - in a system of distributed CT nodes.

+ this kind of automated large-scale testing, Common Test offers a master + test node component, Common Test Master, which handles central configuration and control + in a system of distributed Common Test nodes.

-

The CT Master server runs on one dedicated Erlang node and uses distributed - Erlang to communicate with any number of CT test nodes, each hosting a regular - CT server. Test specifications are used as input to specify what to test on which +

The Common Test Master server runs on one dedicated Erlang node and uses distributed + Erlang to communicate with any number of Common Test test nodes, each hosting a regular + Common Test server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.

-

The CT Master server writes progress information to HTML log files similarly - to the regular CT server. The logs contain test statistics and links to the - log files written by each independent CT server.

+

The Common Test Master server writes progress information to HTML log files similarly + to the regular Common Test server. The logs contain test statistics and links to the + log files written by each independent Common Test server.

-

The CT master API is exported by the ct_master module.

+

The Common Test Master API is exported by module + ct_master.

- Usage -

CT Master requires all test nodes to be on the same network and share a common - file system. As of this date, CT Master can not start test nodes - automatically. The nodes must have been started in advance for CT Master to be + Use +

Common Test Master requires all test nodes to be on the same network and share a common + file system. Common Test Master cannot start test nodes + automatically. The nodes must be started in advance for Common Test Master to be able to start test sessions on them.

-

Tests are started by calling:

- -

ct_master:run(TestSpecs) or - ct_master:run(TestSpecs, InclNodes, ExclNodes)

+

Tests are started by calling + ct_master:run(TestSpecs) or + ct_master:run(TestSpecs, InclNodes, ExclNodes)

TestSpecs is either the name of a test specification file (string) or a list - of test specifications. In case of a list, the specifications will be handled (and + of test specifications. If it is a list, the specifications are handled (and the corresponding tests executed) in sequence. An element in a TestSpecs list - can also be list of test specifications. The specifications in such a list will be - merged into one combined specification prior to test execution. For example:

- -

ct_master:run(["ts1","ts2",["ts3","ts4"]])

+ can also be list of test specifications. The specifications in such a list are + merged into one combined specification before test execution.

-

means first the tests specified by "ts1" will run, then the tests specified by "ts2" +

Example:

+ 
+ ct_master:run(["ts1","ts2",["ts3","ts4"]])
+ +

Here, the tests specified by "ts1" run first, then the tests specified by "ts2", and finally the tests specified by both "ts3" and "ts4".

-

The InclNodes argument to run/3 is a list of node names. The run/3 - function runs the tests in TestSpecs just like run/1 but will also - take any test in TestSpecs that's not explicitly tagged with a particular - node name and execute it on the nodes listed in InclNodes. By using run/3 - this way it is possible to use any test specification, with or without node information, - in a large scale test environment! ExclNodes is a list of nodes that should be - excluded from the test. I.e. tests that have been specified in the test specification - to run on a particular node will not be performed if that node is at runtime - listed in ExclNodes.

- -

If CT Master fails initially to connect to any of the test nodes specified in a - test specification or in the InclNodes list, the operator will be prompted with - the option to either start over again (after manually checking the status of the - node(s) in question), to run without the missing nodes, or to abort the operation.

+

The InclNodes argument to run/3 is a list of node names. Function + run/3 runs the tests in TestSpecs just like run/1, but also + takes any test in TestSpecs, which is not explicitly tagged with a particular + node name, and execute it on the nodes listed in InclNodes. By using run/3 + this way, any test specification can be used, with or without node information, + in a large-scale test environment.

+ +

ExclNodes is a list of nodes to be + excluded from the test. That is, tests that are specified in the test specification + to run on a particular node are not performed if that node is + listed in ExclNodes at runtime.

-

When tests start, CT Master prints information to console about the nodes that are - involved. CT Master also reports when tests finish, successfully or unsuccessfully. If - connection is lost to a node, the test on that node is considered finished. CT Master - will not attempt to reestablish contact with the failing node. At any time to get the - current status of the test nodes, call the function:

+

If Common Test Master fails initially to connect to any of the test nodes specified in a + test specification or in the InclNodes list, the operator is prompted with + the option to either start over again (after manually checking the status of the + nodes in question), to run without the missing nodes, or to abort the operation.

-

ct_master:progress()

+

When tests start, Common Test Master displays information to console about the involved nodes. + Common Test Master also reports when tests finish, successfully or unsuccessfully. If + connection is lost to a node, the test on that node is considered finished. Common Test Master + does not attempt to re-establish contact with the failing node.

-

To stop one or more tests, use:

+

At any time, to get the current status of the test nodes, call function + ct_master:progress().

-

ct_master:abort() (stop all) or ct_master:abort(Nodes)

+

To stop one or more tests, use function + ct_master:abort() (to stop all) or + ct_master:abort(Nodes).

-

For detailed information about the ct_master API, please see the - manual page for this module.

+

For details about the Common Test Master API, see module + ct_master.

Test Specifications -

The test specifications used as input to CT Master are fully compatible with the - specifications used as input to the regular CT server. The syntax is described in the - Running Test Suites - chapter.

+

The test specifications used as input to Common Test Master are fully compatible with the + specifications used as input to the regular Common Test server. The syntax is described in section + Test Specifications + in section Running Tests and Analyzing Results.

All test specification terms can have a NodeRefs element. This element specifies which node or nodes a configuration operation or a test is to be executed - on. NodeRefs is defined as:

+ on. NodeRefs is defined as follows:

NodeRefs = all_nodes | [NodeRef] | NodeRef

- -

where

NodeRef = NodeAlias | node() | master

A NodeAlias (atom()) is used in a test specification as a - reference to a node name (so the actual node name only needs to be declared once, - which can of course also be achieved using constants). - The alias is declared with a node term:

+ reference to a node name (so the node name only needs to be declared once, + which also can be achieved using constants). + The alias is declared with a node term as follows:

{node, NodeAlias, NodeName}

-

If NodeRefs has the value all_nodes, the operation or test will - be performed on all given test nodes. (Declaring a term without a NodeRefs - element actually has the same effect). If NodeRefs has the value - master, the operation is only performed on the CT Master node (namely set +

If NodeRefs has the value all_nodes, the operation or test + is performed on all specified test nodes. (Declaring a term without a NodeRefs + element has the same effect). If NodeRefs has the value + master, the operation is only performed on the Common Test Master node (namely set the log directory or install an event handler).

-

Consider the example in the - Running Test Suites - chapter, now extended with node information and intended to be executed by the - CT Master:

+

Consider the example in section + Test Specifications + in section Running Tests and Analysing Results, + now extended with node information and intended to be executed by + Common Test Master:

-      {define, 'Top', "/home/test"}.
-      {define, 'T1', "'Top'/t1"}.
-      {define, 'T2', "'Top'/t2"}.
-      {define, 'T3', "'Top'/t3"}.
-      {define, 'CfgFile', "config.cfg"}.
-      {define, 'Node', ct_node}.
-
-      {node, node1, 'Node@host_x'}.
-      {node, node2, 'Node@host_y'}.
-
-      {logdir, master, "'Top'/master_logs"}.
-      {logdir, "'Top'/logs"}.
-      
-      {config, node1, "'T1'/'CfgFile'"}.
-      {config, node2, "'T2'/'CfgFile'"}.
-      {config, "'T3'/'CfgFile'"}.
-      
-      {suites, node1, 'T1', all}.
-      {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
-      {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
-      {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.
-      
-      {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
-      {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.
-      
-      {skip_suites, 'T3', all, "Not implemented"}.
+ {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + {define, 'Node', ct_node}. + + {node, node1, 'Node@host_x'}. + {node, node2, 'Node@host_y'}. + + {logdir, master, "'Top'/master_logs"}. + {logdir, "'Top'/logs"}. + + {config, node1, "'T1'/'CfgFile'"}. + {config, node2, "'T2'/'CfgFile'"}. + {config, "'T3'/'CfgFile'"}. + + {suites, node1, 'T1', all}. + {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}. + + {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}. + + {skip_suites, 'T3', all, "Not implemented"}.

This example specifies the same tests as the original example. But - now if started with a call to ct_master:run(TestSpecName), the - t1 test will be executed on node ct_node@host_x (node1), the - t2 test on ct_node@host_y (node2) and the t3 test on both - node1 and node2. The t1 config file will only be read on - node1 and the t2 config file only on node2, while the t3 config file - will be read on both node1 and node2. Both test nodes will write log - files to the same directory. (The CT Master node will however use a - different log directory than the test nodes).

+ now if started with a call to ct_master:run(TestSpecName), test + t1 is executed on node ct_node@host_x (node1), test + t2 on ct_node@host_y (node2) and test t3 + on both node1 and node2. Configuration file t1 is only read on + node1 and configuration file t2 only on node2, while the + configuration file t3 is read on both node1 and node2. + Both test nodes write log files to the same directory. (However, the Common Test Master + node uses a different log directory than the test nodes.)

If the test session is instead started with a call to ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x]), - the result is that the t1 test does not run on - ct_node@host_x (or any other node) while the t3 test runs on + the result is that test t1 does not run on + ct_node@host_x (or any other node) while test t3 runs on both ct_node@host_y and ct_node@host_z.

A nice feature is that a test specification that includes node - information can still be used as input to the regular Common Test server - (as described in the - Running Test Suites - chapter). The result is that any test specified to run on a node with the same - name as the Common Test node in question (typically ct@somehost if started - with the ct_run program), will be performed. Tests without explicit - node association will always be performed too of course!

+ information can still be used as input to the regular Common Test server + (as described in section + Test Specifications). + The result is that any test specified to run on a node with the same + name as the Common Test node in question (typically ct@somehost if started + with the ct_run program), is performed. Tests without explicit + node association are always performed too, of course.

- Automatic startup of test target nodes + Automatic Startup of Test Target Nodes -

It is possible to automatically start, and perform initial actions, on - test target nodes by using the test specification term init.

-

Currently, two sub-terms are supported, node_start and eval.

-

Example:

+

Initial actions can be started and performed automatically on + test target nodes using test specification term init.

+

Two subterms are supported, node_start and eval.

+

Example:

-     {node, node1, node1@host1}.
-     {node, node2, node1@host2}.
-     {node, node3, node2@host2}.
-     {node, node4, node1@host3}.
-     {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
-     {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
-     {init, node4, {eval, {module, function, []}}}.
+ {node, node1, node1@host1}. + {node, node2, node1@host2}. + {node, node3, node2@host2}. + {node, node4, node1@host3}. + {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}. + {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}. + {init, node4, {eval, {module, function, []}}}.

This test specification declares that node1@host1 is to be started using the user callback function callback_module:my_slave_callback/0, and nodes - node1@host2 and node2@host2 will be started with the default callback - module ct_slave. The given user name and password is used to log into remote - host host2. Also, the function module:function/0 will be evaluated on - node1@host3, and the result of this call will be printed to the log.

+ node1@host2 and node2@host2 are to be started with the default callback + module ct_slave. The specified username and password are used to log on to remote + host host2. Also, function module:function/0 is evaluated on + node1@host3, and the result of this call is printed to the log.

-

The default ct_slave callback module, - which is part of the Common Test application, has the following features: +

The default callback module ct_slave, + has the following features:

- + Starting Erlang target nodes on local or remote hosts - (ssh is used for communication). + (application SSH is used for communication). - Ability to start an Erlang emulator with additional flags + Ability to start an Erlang emulator with more flags (any flags supported by erl are supported). - Supervision of a node being started by means of internal callback - functions. Used to prevent hanging nodes. (Configurable). + Supervision of a node being started using internal callback + functions. Used to prevent hanging nodes. (Configurable.) - Monitoring of the master node by the slaves. A slave node may be - stopped in case the master node terminates. (Configurable). + Monitoring of the master node by the slaves. A slave node can be + stopped if the master node terminates. (Configurable.) - Execution of user functions after a slave node is started. - Functions can be given as a list of {Module, Function, Arguments} tuples. + Execution of user functions after a slave node is started. Functions can + be specified as a list of {Module, Function, Arguments} tuples. -

Note that it is possible to specify an eval term for the node as well - as startup_functions in the node_start options list. In this - case first the node will be started, then the startup_functions are +

An eval term for the node and + startup_functions in the node_start options list can be specified. + In this case, the node is started first, then the startup_functions are executed, and finally functions specified with eval are called. -

+

-- cgit v1.2.3