path: root/lib/common_test/doc/src/ct_master_chapter.xml

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2006</year><year>2013</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.

      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.

    </legalnotice>

    <title>Using Common Test for Large Scale Testing</title>
    <prepared>Peter Andersson</prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>ct_master_chapter.xml</file>
  </header>

  <section>
    <marker id="general"></marker>
    <title>General</title>
    <p>Large scale automated testing requires running multiple independent 
      test sessions in parallel. This is accomplished by running
      a number of Common Test nodes on one or more hosts, testing
      different target systems. Configuring, starting and controlling the
      test nodes independently can be a cumbersome operation. To aid
      this kind of automated large scale testing, CT offers a master test 
      node component, CT Master, that handles central configuration and control
      in a system of distributed CT nodes.</p>

    <p>The CT Master server runs on one dedicated Erlang node and uses distributed
      Erlang to communicate with any number of CT test nodes, each hosting a regular
      CT server. Test specifications are used as input to specify what to test on which 
      test nodes, using what configuration.</p> 

    <p>The CT Master server writes progress information to HTML log files similarly 
      to the regular CT server. The logs contain test statistics and links to the 
      log files written by each independent CT server.</p>

    <p>The CT master API is exported by the <c>ct_master</c> module.</p>
  </section>
  <section>
    <title>Usage</title>
    <p>CT Master requires all test nodes to be on the same network and share a common 
      file system. As of this date, CT Master can not start test nodes
      automatically. The nodes must have been started in advance for CT Master to be 
      able to start test sessions on them.</p>
      
    <p>Tests are started by calling:</p>
    
    <p><c>ct_master:run(TestSpecs)</c> or 
       <c>ct_master:run(TestSpecs, InclNodes, ExclNodes)</c></p>
    
    <p><c>TestSpecs</c> is either the name of a test specification file (string) or a list 
      of test specifications. In case of a list, the specifications will be handled (and
      the corresponding tests executed) in sequence. An element in a <c>TestSpecs</c> list 
      can also be list of test specifications. The specifications in such a list will be 
      merged into one combined specification prior to test execution. For example:</p>
      
    <p><c>ct_master:run(["ts1","ts2",["ts3","ts4"]])</c></p>

    <p>means first the tests specified by "ts1" will run, then the tests specified by "ts2" 
      and finally the tests specified by both "ts3" and "ts4".</p>

    <p>The <c>InclNodes</c> argument to <c>run/3</c> is a list of node names. The <c>run/3</c> 
      function runs the tests in <c>TestSpecs</c> just like <c>run/1</c> but will also 
      take any test in <c>TestSpecs</c> that's not explicitly tagged with a particular 
      node name and execute it on the nodes listed in <c>InclNodes</c>. By using <c>run/3</c> 
      this way it is possible to use any test specification, with or without node information, 
      in a large scale test environment! <c>ExclNodes</c> is a list of nodes that should be
      excluded from the test. I.e. tests that have been specified in the test specification 
      to run on a particular node will not be performed if that node is at runtime
      listed in <c>ExclNodes</c>.</p>

      <p>If CT Master fails initially to connect to any of the test nodes specified in a 
      test specification or in the <c>InclNodes</c> list, the operator will be prompted with 
      the option to either start over again (after manually checking the status of the 
      node(s) in question), to run without the missing nodes, or to abort the operation.</p>

      <p>When tests start, CT Master prints information to console about the nodes that are 
      involved. CT Master also reports when tests finish, successfully or unsuccessfully. If
      connection is lost to a node, the test on that node is considered finished. CT Master 
      will not attempt to reestablish contact with the failing node. At any time to get the
      current status of the test nodes, call the function:</p>

      <p><c>ct_master:progress()</c></p>

      <p>To stop one or more tests, use:</p>

      <p><c>ct_master:abort()</c> (stop all) or <c>ct_master:abort(Nodes)</c></p>

      <p>For detailed information about the <c>ct_master</c> API, please see the 
      <seealso marker="ct_master">manual page</seealso> for this module.</p>
  </section>
  <section>
    <marker id="test_specifications"></marker>
    <title>Test Specifications</title>
    <p>The test specifications used as input to CT Master are fully compatible with the
    specifications used as input to the regular CT server. The syntax is described in the 
    <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso>
    chapter.</p>

    <p>All test specification terms can have a <c>NodeRefs</c> element. This element
    specifies which node or nodes a configuration operation or a test is to be executed 
    on. <c>NodeRefs</c> is defined as:</p>
    
    <p><c>NodeRefs = all_nodes | [NodeRef] | NodeRef</c></p>
    
    <p>where</p>

    <p><c>NodeRef = NodeAlias | node() | master</c></p>

    <p>A <c>NodeAlias</c> (<c>atom()</c>) is used in a test specification as a 
    reference to a node name (so the actual node name only needs to be declared once,
      which can of course also be achieved using constants). 
    The alias is declared with a <c>node</c> term:</p>

    <p><c>{node, NodeAlias, NodeName}</c></p>

    <p>If <c>NodeRefs</c> has the value <c>all_nodes</c>, the operation or test will
    be performed on all given test nodes. (Declaring a term without a <c>NodeRefs</c> 
    element actually has the same effect). If <c>NodeRefs</c> has the value 
    <c>master</c>, the operation is only performed on the CT Master node (namely set 
    the log directory or install an event handler).</p>

    <p>Consider the example in the 
    <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso>
    chapter, now extended with node information and intended to be executed by the
    CT Master:</p>

    <pre>
      {define, 'Top', "/home/test"}.
      {define, 'T1', "'Top'/t1"}.
      {define, 'T2', "'Top'/t2"}.
      {define, 'T3', "'Top'/t3"}.
      {define, 'CfgFile', "config.cfg"}.
      {define, 'Node', ct_node}.

      {node, node1, 'Node@host_x'}.
      {node, node2, 'Node@host_y'}.

      {logdir, master, "'Top'/master_logs"}.
      {logdir, "'Top'/logs"}.
      
      {config, node1, "'T1'/'CfgFile'"}.
      {config, node2, "'T2'/'CfgFile'"}.
      {config, "'T3'/'CfgFile'"}.
      
      {suites, node1, 'T1', all}.
      {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
      {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
      {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.
      
      {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
      {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.
      
      {skip_suites, 'T3', all, "Not implemented"}.</pre>

    <p>This example specifies the same tests as the original example. But 
    now if started with a call to <c>ct_master:run(TestSpecName)</c>, the 
    t1 test will be executed on node <c>ct_node@host_x</c> (node1), the
    t2 test on <c>ct_node@host_y</c> (node2) and the t3 test on both
    node1 and node2. The t1 config file will only be read on
    node1 and the t2 config file only on node2, while the t3 config file
    will be read on both node1 and node2. Both test nodes will write log 
    files to the same directory. (The CT Master node will however use a 
    different log directory than the test nodes).</p>

    <p>If the test session is instead started with a call to 
    <c>ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x])</c>, 
    the result is that the t1 test does not run on 
    <c>ct_node@host_x</c> (or any other node) while the t3 test runs on
    <c>ct_node@host_y</c> and <c>ct_node@host_z</c>.</p>
    
    <p>A nice feature is that a test specification that includes node 
      information can still be used as input to the regular Common Test server 
      (as described in the
    <seealso marker="run_test_chapter#test_specifications">Running Test Suites</seealso>
    chapter). The result is that any test specified to run on a node with the same
    name as the Common Test node in question (typically <c>ct@somehost</c> if started
    with the <c>ct_run</c> program), will be performed. Tests without explicit
    node association will always be performed too of course!</p>
  </section>

  <section>
    <title>Automatic startup of test target nodes</title>
    <marker id="ct_slave"></marker>
    <p>Is is possible to automatically start, and perform initial actions, on
      test target nodes by using the test specification term <c>init</c>.</p>
    <p>Currently, two sub-terms are supported, <c>node_start</c> and <c>eval</c>.</p>
    <p>Example:</p>
    <pre>
     {node, node1, node1@host1}.
     {node, node2, node1@host2}.
     {node, node3, node2@host2}.
     {node, node4, node1@host3}.
     {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
     {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
     {init, node4, {eval, {module, function, []}}}.</pre>

    <p>This test specification declares that <c>node1@host1</c> is to be started using
      the user callback function <c>callback_module:my_slave_callback/0</c>, and nodes
      <c>node1@host2</c> and <c>node2@host2</c> will be started with the default callback
      module <c>ct_slave</c>. The given user name and password is used to log into remote
      host <c>host2</c>. Also, the function <c>module:function/0</c> will be evaluated on
      <c>node1@host3</c>, and the result of this call will be printed to the log.</p>

    <p>The default <seealso marker="ct_slave">ct_slave</seealso> callback module,
      which is part of the Common Test application, has the following features:
    </p>
    <list>
     <item>Starting Erlang target nodes on local or remote hosts
       (ssh is used for communication).
     </item>
     <item>Ability to start an Erlang emulator with additional flags
       (any flags supported by <c>erl</c> are supported).
     </item>
     <item>Supervision of a node being started by means of internal callback
        functions. Used to prevent hanging nodes. (Configurable).
     </item>
     <item>Monitoring of the master node by the slaves. A slave node may be
        stopped in case the master node terminates. (Configurable).
     </item>
     <item>Execution of user functions after a slave node is started.
        Functions can be given as a list of {Module, Function, Arguments} tuples.
     </item>
    </list>
    <p>Note that it is possible to specify an <c>eval</c> term for the node as well
    as <c>startup_functions</c> in the <c>node_start</c> options list. In this
    case first the node will be started, then the <c>startup_functions</c> are
    executed, and finally functions specified with <c>eval</c> are called.
    </p>
  </section>

</chapter>