diff options
Diffstat (limited to 'lib/common_test/doc/src/ct_master_chapter.xml')
| -rw-r--r-- | lib/common_test/doc/src/ct_master_chapter.xml | 291 |
1 files changed, 147 insertions, 144 deletions
diff --git a/lib/common_test/doc/src/ct_master_chapter.xml b/lib/common_test/doc/src/ct_master_chapter.xml index 16492f39b8..99555164e0 100644 --- a/lib/common_test/doc/src/ct_master_chapter.xml +++ b/lib/common_test/doc/src/ct_master_chapter.xml @@ -22,7 +22,7 @@ </legalnotice> - <title>Using Common Test for Large Scale Testing</title> + <title>Using Common Test for Large-Scale Testing</title> <prepared>Peter Andersson</prepared> <docno></docno> <date></date> @@ -33,217 +33,220 @@ <section> <marker id="general"></marker> <title>General</title> - <p>Large scale automated testing requires running multiple independent + <p>Large-scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running - a number of Common Test nodes on one or more hosts, testing - different target systems. Configuring, starting and controlling the + some <c>Common Test</c> nodes on one or more hosts, testing + different target systems. Configuring, starting, and controlling the test nodes independently can be a cumbersome operation. To aid - this kind of automated large scale testing, CT offers a master test - node component, CT Master, that handles central configuration and control - in a system of distributed CT nodes.</p> + this kind of automated large-scale testing, <c>Common Test</c> offers a master + test node component, <c>Common Test</c> Master, which handles central configuration and control + in a system of distributed <c>Common Test</c> nodes.</p> - <p>The CT Master server runs on one dedicated Erlang node and uses distributed - Erlang to communicate with any number of CT test nodes, each hosting a regular - CT server. Test specifications are used as input to specify what to test on which + <p>The <c>Common Test</c> Master server runs on one dedicated Erlang node and uses distributed + Erlang to communicate with any number of <c>Common Test</c> test nodes, each hosting a regular + <c>Common Test</c> server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.</p> - <p>The CT Master server writes progress information to HTML log files similarly - to the regular CT server. The logs contain test statistics and links to the - log files written by each independent CT server.</p> + <p>The <c>Common Test</c> Master server writes progress information to HTML log files similarly + to the regular <c>Common Test</c> server. The logs contain test statistics and links to the + log files written by each independent <c>Common Test</c> server.</p> - <p>The CT master API is exported by the <c>ct_master</c> module.</p> + <p>The <c>Common Test</c> Master API is exported by module + <seealso marker="ct_master"><c>ct_master</c></seealso>.</p> </section> <section> - <title>Usage</title> - <p>CT Master requires all test nodes to be on the same network and share a common - file system. As of this date, CT Master can not start test nodes - automatically. The nodes must have been started in advance for CT Master to be + <title>Use</title> + <p><c>Common Test</c> Master requires all test nodes to be on the same network and share a common + file system. <c>Common Test</c> Master cannot start test nodes + automatically. The nodes must be started in advance for <c>Common Test</c> Master to be able to start test sessions on them.</p> - <p>Tests are started by calling:</p> - - <p><c>ct_master:run(TestSpecs)</c> or - <c>ct_master:run(TestSpecs, InclNodes, ExclNodes)</c></p> + <p>Tests are started by calling + <seealso marker="ct_master#run-1"><c>ct_master:run(TestSpecs)</c></seealso> or + <seealso marker="ct_master#run-3"><c>ct_master:run(TestSpecs, InclNodes, ExclNodes)</c></seealso></p> <p><c>TestSpecs</c> is either the name of a test specification file (string) or a list - of test specifications. In case of a list, the specifications will be handled (and + of test specifications. If it is a list, the specifications are handled (and the corresponding tests executed) in sequence. An element in a <c>TestSpecs</c> list - can also be list of test specifications. The specifications in such a list will be - merged into one combined specification prior to test execution. For example:</p> - - <p><c>ct_master:run(["ts1","ts2",["ts3","ts4"]])</c></p> + can also be list of test specifications. The specifications in such a list are + merged into one combined specification before test execution.</p> - <p>means first the tests specified by "ts1" will run, then the tests specified by "ts2" + <p><em>Example:</em></p> + <pre> + ct_master:run(["ts1","ts2",["ts3","ts4"]])</pre> + + <p>Here, the tests specified by "ts1" run first, then the tests specified by "ts2", and finally the tests specified by both "ts3" and "ts4".</p> - <p>The <c>InclNodes</c> argument to <c>run/3</c> is a list of node names. The <c>run/3</c> - function runs the tests in <c>TestSpecs</c> just like <c>run/1</c> but will also - take any test in <c>TestSpecs</c> that's not explicitly tagged with a particular - node name and execute it on the nodes listed in <c>InclNodes</c>. By using <c>run/3</c> - this way it is possible to use any test specification, with or without node information, - in a large scale test environment! <c>ExclNodes</c> is a list of nodes that should be - excluded from the test. I.e. tests that have been specified in the test specification - to run on a particular node will not be performed if that node is at runtime - listed in <c>ExclNodes</c>.</p> - - <p>If CT Master fails initially to connect to any of the test nodes specified in a - test specification or in the <c>InclNodes</c> list, the operator will be prompted with - the option to either start over again (after manually checking the status of the - node(s) in question), to run without the missing nodes, or to abort the operation.</p> + <p>The <c>InclNodes</c> argument to <c>run/3</c> is a list of node names. Function + <c>run/3</c> runs the tests in <c>TestSpecs</c> just like <c>run/1</c>, but also + takes any test in <c>TestSpecs</c>, which is not explicitly tagged with a particular + node name, and execute it on the nodes listed in <c>InclNodes</c>. By using <c>run/3</c> + this way, any test specification can be used, with or without node information, + in a large-scale test environment.</p> + + <p><c>ExclNodes</c> is a list of nodes to be + excluded from the test. That is, tests that are specified in the test specification + to run on a particular node are not performed if that node is + listed in <c>ExclNodes</c> at runtime.</p> - <p>When tests start, CT Master prints information to console about the nodes that are - involved. CT Master also reports when tests finish, successfully or unsuccessfully. If - connection is lost to a node, the test on that node is considered finished. CT Master - will not attempt to reestablish contact with the failing node. At any time to get the - current status of the test nodes, call the function:</p> + <p>If <c>Common Test</c> Master fails initially to connect to any of the test nodes specified in a + test specification or in the <c>InclNodes</c> list, the operator is prompted with + the option to either start over again (after manually checking the status of the + nodes in question), to run without the missing nodes, or to abort the operation.</p> - <p><c>ct_master:progress()</c></p> + <p>When tests start, <c>Common Test</c> Master displays information to console about the involved nodes. + <c>Common Test</c> Master also reports when tests finish, successfully or unsuccessfully. If + connection is lost to a node, the test on that node is considered finished. <c>Common Test</c> Master + does not attempt to re-establish contact with the failing node.</p> - <p>To stop one or more tests, use:</p> + <p>At any time, to get the current status of the test nodes, call function + <seealso marker="ct_master#progress-0"><c>ct_master:progress()</c></seealso>.</p> - <p><c>ct_master:abort()</c> (stop all) or <c>ct_master:abort(Nodes)</c></p> + <p>To stop one or more tests, use function + <seealso marker="ct_master#abort-0"><c>ct_master:abort()</c></seealso> (to stop all) or + <seealso marker="ct_master#abort-1"><c>ct_master:abort(Nodes)</c></seealso>.</p> - <p>For detailed information about the <c>ct_master</c> API, please see the - <seealso marker="ct_master">manual page</seealso> for this module.</p> + <p>For details about the <c>Common Test</c> Master API, see module + <seealso marker="ct_master"><c>ct_master</c></seealso>.</p> </section> <section> <marker id="test_specifications"></marker> <title>Test Specifications</title> - <p>The test specifications used as input to CT Master are fully compatible with the - specifications used as input to the regular CT server. The syntax is described in the - <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> - chapter.</p> + <p>The test specifications used as input to <c>Common Test</c> Master are fully compatible with the + specifications used as input to the regular <c>Common Test</c> server. The syntax is described in section + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso> + in section Running Tests and Analyzing Results.</p> <p>All test specification terms can have a <c>NodeRefs</c> element. This element specifies which node or nodes a configuration operation or a test is to be executed - on. <c>NodeRefs</c> is defined as:</p> + on. <c>NodeRefs</c> is defined as follows:</p> <p><c>NodeRefs = all_nodes | [NodeRef] | NodeRef</c></p> - - <p>where</p> <p><c>NodeRef = NodeAlias | node() | master</c></p> <p>A <c>NodeAlias</c> (<c>atom()</c>) is used in a test specification as a - reference to a node name (so the actual node name only needs to be declared once, - which can of course also be achieved using constants). - The alias is declared with a <c>node</c> term:</p> + reference to a node name (so the node name only needs to be declared once, + which also can be achieved using constants). + The alias is declared with a <c>node</c> term as follows:</p> <p><c>{node, NodeAlias, NodeName}</c></p> - <p>If <c>NodeRefs</c> has the value <c>all_nodes</c>, the operation or test will - be performed on all given test nodes. (Declaring a term without a <c>NodeRefs</c> - element actually has the same effect). If <c>NodeRefs</c> has the value - <c>master</c>, the operation is only performed on the CT Master node (namely set + <p>If <c>NodeRefs</c> has the value <c>all_nodes</c>, the operation or test + is performed on all specified test nodes. (Declaring a term without a <c>NodeRefs</c> + element has the same effect). If <c>NodeRefs</c> has the value + <c>master</c>, the operation is only performed on the <c>Common Test</c> Master node (namely set the log directory or install an event handler).</p> - <p>Consider the example in the - <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> - chapter, now extended with node information and intended to be executed by the - CT Master:</p> + <p>Consider the example in section + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso> + in section Running Tests and Analysing Results, + now extended with node information and intended to be executed by + <c>Common Test</c> Master:</p> <pre> - {define, 'Top', "/home/test"}. - {define, 'T1', "'Top'/t1"}. - {define, 'T2', "'Top'/t2"}. - {define, 'T3', "'Top'/t3"}. - {define, 'CfgFile', "config.cfg"}. - {define, 'Node', ct_node}. - - {node, node1, 'Node@host_x'}. - {node, node2, 'Node@host_y'}. - - {logdir, master, "'Top'/master_logs"}. - {logdir, "'Top'/logs"}. - - {config, node1, "'T1'/'CfgFile'"}. - {config, node2, "'T2'/'CfgFile'"}. - {config, "'T3'/'CfgFile'"}. - - {suites, node1, 'T1', all}. - {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. - {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. - {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}. - - {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}. - {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}. - - {skip_suites, 'T3', all, "Not implemented"}.</pre> + {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + {define, 'Node', ct_node}. + + {node, node1, 'Node@host_x'}. + {node, node2, 'Node@host_y'}. + + {logdir, master, "'Top'/master_logs"}. + {logdir, "'Top'/logs"}. + + {config, node1, "'T1'/'CfgFile'"}. + {config, node2, "'T2'/'CfgFile'"}. + {config, "'T3'/'CfgFile'"}. + + {suites, node1, 'T1', all}. + {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}. + + {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}. + + {skip_suites, 'T3', all, "Not implemented"}.</pre> <p>This example specifies the same tests as the original example. But - now if started with a call to <c>ct_master:run(TestSpecName)</c>, the - t1 test will be executed on node <c>ct_node@host_x</c> (node1), the - t2 test on <c>ct_node@host_y</c> (node2) and the t3 test on both - node1 and node2. The t1 config file will only be read on - node1 and the t2 config file only on node2, while the t3 config file - will be read on both node1 and node2. Both test nodes will write log - files to the same directory. (The CT Master node will however use a - different log directory than the test nodes).</p> + now if started with a call to <c>ct_master:run(TestSpecName)</c>, test + <c>t1</c> is executed on node <c>ct_node@host_x</c> (<c>node1</c>), test + <c>t2</c> on <c>ct_node@host_y</c> (<c>node2</c>) and test <c>t3</c> + on both <c>node1</c> and <c>node2</c>. Configuration file <c>t1</c> is only read on + <c>node1</c> and configuration file <c>t2</c> only on <c>node2</c>, while the + configuration file <c>t3</c> is read on both <c>node1</c> and <c>node2</c>. + Both test nodes write log files to the same directory. (However, the <c>Common Test</c> Master + node uses a different log directory than the test nodes.)</p> <p>If the test session is instead started with a call to <c>ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x])</c>, - the result is that the t1 test does not run on - <c>ct_node@host_x</c> (or any other node) while the t3 test runs on + the result is that test <c>t1</c> does not run on + <c>ct_node@host_x</c> (or any other node) while test <c>t3</c> runs on both <c>ct_node@host_y</c> and <c>ct_node@host_z</c>.</p> <p>A nice feature is that a test specification that includes node - information can still be used as input to the regular Common Test server - (as described in the - <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> - chapter). The result is that any test specified to run on a node with the same - name as the Common Test node in question (typically <c>ct@somehost</c> if started - with the <c>ct_run</c> program), will be performed. Tests without explicit - node association will always be performed too of course!</p> + information can still be used as input to the regular <c>Common Test</c> server + (as described in section + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso>). + The result is that any test specified to run on a node with the same + name as the <c>Common Test</c> node in question (typically <c>ct@somehost</c> if started + with the <c>ct_run</c> program), is performed. Tests without explicit + node association are always performed too, of course.</p> </section> <section> - <title>Automatic startup of test target nodes</title> + <title>Automatic Startup of Test Target Nodes</title> <marker id="ct_slave"></marker> - <p>It is possible to automatically start, and perform initial actions, on - test target nodes by using the test specification term <c>init</c>.</p> - <p>Currently, two sub-terms are supported, <c>node_start</c> and <c>eval</c>.</p> - <p>Example:</p> + <p>Initial actions can be started and performed automatically on + test target nodes using test specification term <c>init</c>.</p> + <p>Two subterms are supported, <c>node_start</c> and <c>eval</c>.</p> + <p><em>Example:</em></p> <pre> - {node, node1, node1@host1}. - {node, node2, node1@host2}. - {node, node3, node2@host2}. - {node, node4, node1@host3}. - {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}. - {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}. - {init, node4, {eval, {module, function, []}}}.</pre> + {node, node1, node1@host1}. + {node, node2, node1@host2}. + {node, node3, node2@host2}. + {node, node4, node1@host3}. + {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}. + {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}. + {init, node4, {eval, {module, function, []}}}.</pre> <p>This test specification declares that <c>node1@host1</c> is to be started using the user callback function <c>callback_module:my_slave_callback/0</c>, and nodes - <c>node1@host2</c> and <c>node2@host2</c> will be started with the default callback - module <c>ct_slave</c>. The given user name and password is used to log into remote - host <c>host2</c>. Also, the function <c>module:function/0</c> will be evaluated on - <c>node1@host3</c>, and the result of this call will be printed to the log.</p> + <c>node1@host2</c> and <c>node2@host2</c> are to be started with the default callback + module <c>ct_slave</c>. The specified username and password are used to log on to remote + host <c>host2</c>. Also, function <c>module:function/0</c> is evaluated on + <c>node1@host3</c>, and the result of this call is printed to the log.</p> - <p>The default <seealso marker="ct_slave">ct_slave</seealso> callback module, - which is part of the Common Test application, has the following features: + <p>The default callback module <seealso marker="ct_slave">ct_slave</seealso>, + has the following features: </p> - <list> + <list type="bulleted"> <item>Starting Erlang target nodes on local or remote hosts - (ssh is used for communication). + (application <c>SSH</c> is used for communication). </item> - <item>Ability to start an Erlang emulator with additional flags + <item>Ability to start an Erlang emulator with more flags (any flags supported by <c>erl</c> are supported). </item> - <item>Supervision of a node being started by means of internal callback - functions. Used to prevent hanging nodes. (Configurable). + <item>Supervision of a node being started using internal callback + functions. Used to prevent hanging nodes. (Configurable.) </item> - <item>Monitoring of the master node by the slaves. A slave node may be - stopped in case the master node terminates. (Configurable). + <item>Monitoring of the master node by the slaves. A slave node can be + stopped if the master node terminates. (Configurable.) </item> - <item>Execution of user functions after a slave node is started. - Functions can be given as a list of {Module, Function, Arguments} tuples. + <item>Execution of user functions after a slave node is started. Functions can + be specified as a list of <c>{Module, Function, Arguments}</c> tuples. </item> </list> - <p>Note that it is possible to specify an <c>eval</c> term for the node as well - as <c>startup_functions</c> in the <c>node_start</c> options list. In this - case first the node will be started, then the <c>startup_functions</c> are + <note><p>An <c>eval</c> term for the node and + <c>startup_functions</c> in the <c>node_start</c> options list can be specified. + In this case, the node is started first, then the <c>startup_functions</c> are executed, and finally functions specified with <c>eval</c> are called. - </p> + </p></note> </section> </chapter> |