diff options
Diffstat (limited to 'lib/common_test/doc/src/ct_master_chapter.xml')
-rw-r--r-- | lib/common_test/doc/src/ct_master_chapter.xml | 199 |
1 files changed, 199 insertions, 0 deletions
diff --git a/lib/common_test/doc/src/ct_master_chapter.xml b/lib/common_test/doc/src/ct_master_chapter.xml new file mode 100644 index 0000000000..79288cfe4c --- /dev/null +++ b/lib/common_test/doc/src/ct_master_chapter.xml @@ -0,0 +1,199 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2006</year><year>2009</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Using Common Test for Large Scale Testing</title> + <prepared>Peter Andersson</prepared> + <docno></docno> + <date></date> + <rev></rev> + <file>ct_master_chapter.xml</file> + </header> + + <section> + <title>General</title> + <p>Large scale automated testing requires running multiple independent + test sessions in parallel. This is accomplished by running + a number of Common Test nodes on one or more hosts, testing + different target systems. Configuring, starting and controlling the + test nodes independently can be a cumbersome operation. To aid + this kind of automated large scale testing, CT offers a master test + node component, CT Master, that handles central configuration and control + in a system of distributed CT nodes.</p> + + <p>The CT Master server runs on one dedicated Erlang node and uses distributed + Erlang to communicate with any number of CT test nodes, each hosting a regular + CT server. Test specifications are used as input to specify what to test on which + test nodes, using what configuration.</p> + + <p>The CT Master server writes progress information to HTML log files similarly + to the regular CT server. The logs contain test statistics and links to the + log files written by each independent CT server.</p> + + <p>The CT master API is exported by the <c>ct_master</c> module.</p> + </section> + <section> + <title>Usage</title> + <p>CT Master requires all test nodes to be on the same network and share a common + file system. As of this date, CT Master can not start test nodes + automatically. The nodes must have been started in advance for CT Master to be + able to start test sessions on them.</p> + + <p>Tests are started by calling:</p> + + <p><c>ct_master:run(TestSpecs)</c> or + <c>ct_master:run(TestSpecs, InclNodes, ExclNodes)</c></p> + + <p><c>TestSpecs</c> is either the name of a test specification file (string) or a list + of test specifications. In case of a list, the specifications will be handled (and + the corresponding tests executed) in sequence. An element in a <c>TestSpecs</c> list + can also be list of test specifications. The specifications in such a list will be + merged into one combined specification prior to test execution. For example:</p> + + <p><c>ct_master:run(["ts1","ts2",["ts3","ts4"]])</c></p> + + <p>means first the tests specified by "ts1" will run, then the tests specified by "ts2" + and finally the tests specified by both "ts3" and "ts4".</p> + + <p>The <c>InclNodes</c> argument to <c>run/3</c> is a list of node names. The <c>run/3</c> + function runs the tests in <c>TestSpecs</c> just like <c>run/1</c> but will also + take any test in <c>TestSpecs</c> that's not explicitly tagged with a particular + node name and execute it on the nodes listed in <c>InclNodes</c>. By using <c>run/3</c> + this way it is possible to use any test specification, with or without node information, + in a large scale test environment! <c>ExclNodes</c> is a list of nodes that should be + excluded from the test. I.e. tests that have been specified in the test specification + to run on a particular node will not be performed if that node is at runtime + listed in <c>ExclNodes</c>.</p> + + <p>If CT Master fails initially to connect to any of the test nodes specified in a + test specification or in the <c>InclNodes</c> list, the operator will be prompted with + the option to either start over again (after manually checking the status of the + node(s) in question), to run without the missing nodes, or to abort the operation.</p> + + <p>When tests start, CT Master prints information to console about the nodes that are + involved. CT Master also reports when tests finish, successfully or unsuccessfully. If + connection is lost to a node, the test on that node is considered finished. CT Master + will not attempt to reestablish contact with the failing node. At any time to get the + current status of the test nodes, call the function:</p> + + <p><c>ct_master:progress()</c></p> + + <p>To stop one or more tests, use:</p> + + <p><c>ct_master:abort()</c> (stop all) or <c>ct_master:abort(Nodes)</c></p> + + <p>For detailed information about the <c>ct_master</c> API, please see the + manual page for this module.</p> + </section> + <section> + <title>Test Specifications</title> + <p>The test specifications used as input to CT Master are fully compatible with the + specifications used as input to the regular CT server. The syntax is described in the + <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> + chapter.</p> + + <p>All test specification terms can have a <c>NodeRefs</c> element. This element + specifies which node or nodes a configuration operation or a test is to be executed + on. <c>NodeRefs</c> is defined as:</p> + + <p><c>NodeRefs = all_nodes | [NodeRef] | NodeRef</c></p> + + <p>where</p> + + <p><c>NodeRef = NodeAlias | node() | master</c></p> + + <p>A <c>NodeAlias</c> (<c>atom()</c>) is used in a test specification as a + reference to a node name (so the actual node name only needs to be declared once). + The alias is declared with a <c>node</c> term:</p> + + <p><c>{node, NodeAlias, NodeName}</c></p> + + <p>If <c>NodeRefs</c> has the value <c>all_nodes</c>, the operation or test will + be performed on all given test nodes. (Declaring a term without a <c>NodeRefs</c> + element actually has the same effect). If <c>NodeRefs</c> has the value + <c>master</c>, the operation is only performed on the CT Master node (namely set + the log directory or install an event handler).</p> + + <p>Consider the example in the + <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> + chapter, now extended with node information and intended to be executed by the + CT Master:</p> + + <pre> + {node, node1, ct_node@host_x}. + {node, node2, ct_node@host_y}. + + {logdir, master, "/home/test/master_logs"}. + {logdir, "/home/test/logs"}. + + {config, node1, "/home/test/t1/cfg/config.cfg"}. + {config, node2, "/home/test/t2/cfg/config.cfg"}. + {config, "/home/test/t3/cfg/config.cfg"}. + + {alias, t1, "/home/test/t1"}. + {alias, t2, "/home/test/t2"}. + {alias, t3, "/home/test/t3"}. + + {suites, node1, t1, all}. + {skip_suites, node1, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, node1, t1, t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, node1, t1, t1C_SUITE, [test1], "Ignore"}. + + {suites, node2, t2, [t2B_SUITE,t2C_SUITE]}. + {cases, node2, t2, t2A_SUITE, [test4,test1,test7]}. + + {skip_suites, t3, all, "Not implemented"}. + </pre> + + <p>This example specifies the same tests as the original example. But + now if started with a call to <c>ct_master:run(TestSpecName)</c>, the + t1 test will be executed on node <c>ct_node@host_x</c> (node1), the + t2 test on <c>ct_node@host_y</c> (node2) and the t3 test on both + node1 and node2. The t1 config file will only be read on + node1 and the t2 config file only on node2, while the t3 config file + will be read on both node1 and node2. Both test nodes will write log + files to the same directory. (The CT Master node will however use a + different log directory than the test nodes).</p> + + <p>If the test session is instead started with a call to + <c>ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x])</c>, + the result is that the t1 test does not run on + <c>ct_node@host_x</c> (or any other node) while the t3 test runs on + <c>ct_node@host_y</c> and <c>ct_node@host_z</c>.</p> + + <p>A nice feature is that a test specification that includes node + information can still be used as input to the regular Common Test server + (as described in the + <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso> + chapter). The result is that any test specified to run on a node with the same + name as the Common Test node in question (typically <c>ct@somehost</c> if started + with the <c>run_test</c> script), will be performed. Tests without explicit + node association will always be performed too of course!</p> + + <note><p>It is recommended that absolute paths are used for log directories, + config files and test directory aliases in the test specifications so that + current working directory settings are not important.</p></note> + </section> + +</chapter> + + |