diff options
author | Erlang/OTP <[email protected]> | 2009-11-20 14:54:40 +0000 |
---|---|---|
committer | Erlang/OTP <[email protected]> | 2009-11-20 14:54:40 +0000 |
commit | 84adefa331c4159d432d22840663c38f155cd4c1 (patch) | |
tree | bff9a9c66adda4df2106dfd0e5c053ab182a12bd /lib/common_test/doc/src/event_handler_chapter.xml | |
download | otp-84adefa331c4159d432d22840663c38f155cd4c1.tar.gz otp-84adefa331c4159d432d22840663c38f155cd4c1.tar.bz2 otp-84adefa331c4159d432d22840663c38f155cd4c1.zip |
The R13B03 release.OTP_R13B03
Diffstat (limited to 'lib/common_test/doc/src/event_handler_chapter.xml')
-rw-r--r-- | lib/common_test/doc/src/event_handler_chapter.xml | 310 |
1 files changed, 310 insertions, 0 deletions
diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml new file mode 100644 index 0000000000..a550810850 --- /dev/null +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -0,0 +1,310 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2006</year><year>2009</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Event Handling</title> + <prepared>Peter Andersson</prepared> + <docno></docno> + <date></date> + <rev></rev> + <file>event_handler_chapter.xml</file> + </header> + + <section> + <marker id="event_handling"></marker> + <title>General</title> + <p>It is possible for the operator of a Common Test system to receive + event notifications continously during a test run. It is reported e.g. + when a test case starts and stops, what the current count of successful, + failed and skipped cases is, etc. This information can be used for + different purposes such as logging progress and results on + other format than HTML, saving statistics to a database for report + generation and test system supervision.</p> + + <p>Common Test has a framework for event handling which is based on + the OTP event manager concept and gen_event behaviour. When the Common Test + server starts, it spawns an event manager. During test execution the + manager gets a notification from the server every time something + of potential interest happens. Any event handler plugged into the + event manager can match on events of interest, take action, or maybe + simply pass the information on. Event handlers are Erlang modules + implemented by the Common Test user according to the gen_event + behaviour (see the OTP User's Guide and Reference Manual for more + information).</p> + + <p>As already described, a Common Test server always starts an event manager. + The server also plugs in a default event handler which has as its only + purpose to relay notifications to a globally registered CT Master + event manager (if a CT Master server is running in the system). + The CT Master also spawns an event manager at startup. + Event handlers plugged into this manager will receive the events from + all the test nodes as well as information from the CT Master server + itself.</p> + </section> + <section> + <title>Usage</title> + <p>Event handlers may be installed by means of an <c>event_handler</c> + start flag (<c>run_test</c>) or option (<c>ct:run_test/1</c>), where the + argument specifies the names of one or more event handler modules. + Example:</p> + <p><c>$ run_test -suite test/my_SUITE -event_handler handlers/my_evh1 + handlers/my_evh2 -pa $PWD/handlers</c></p> + <p>All event handler modules must have gen_event behaviour. Note also that + these modules must be precompiled, and that their locations must be + added explicitly to the Erlang code server search path (like in the + example).</p> + + <p>It is not possible to specify start arguments to the event handlers when + using the <c>run_test</c> script. You may however pass along start arguments + if you use the <c>ct:run_test/1</c> function. An event_handler tuple in the argument + <c>Opts</c> has the following definition (see also <c>ct:run_test/1</c> in the + reference manual):</p> + + <pre> + {event_handler,EventHandlers} + + EventHandlers = EH | [EH] + EH = atom() | {atom(),InitArgs} | {[atom()],InitArgs} + InitArgs = [term()] + </pre> + + <p>Example:</p> + <pre> + 1> ct:run_test([{suite,"test/my_SUITE"},{event_handler,[my_evh1,{my_evh2,[node()]}]}]). + </pre> + <p>This will install two event handlers for the <c>my_SUITE</c> test. Event handler + <c>my_evh1</c> is started with <c>[]</c> as argument to the init function. Event handler + <c>my_evh2</c> is started with the name of the current node in the init argument list.</p> + + <p>Event handlers can also be plugged in by means of + <seealso marker="run_test_chapter#test_specifications">test specification</seealso> + terms:</p> + + <p><c>{event_handler, EventHandlers}</c>, or</p> + <p><c>{event_handler, EventHandlers, InitArgs}</c>, or</p> + <p><c>{event_handler, NodeRefs, EventHandlers}</c>, or</p> + <p><c>{event_handler, NodeRefs, EventHandlers, InitArgs}</c></p> + + <p><c>EventHandlers</c> is a list of module names. Before a test + session starts, the init function of each plugged in event handler + is called (with the InitArgs list as argument or [] if + no start arguments are given).</p> + + <p>To plug a handler into the CT Master event manager, specify + <c>master</c> as the node in <c>NodeRefs</c>.</p> + + <p>For an event handler to be able to match on events, the module must + include the header file <c>ct_event.hrl</c>. An event is a record with the + following definition:</p> + + <p><c>#event{name, node, data}</c></p> + + <p><c>name</c> is the label (type) of the event. <c>node</c> is the name of the + node the event has originated from (only relevant for CT Master event handlers). + <c>data</c> is specific for the particular event.</p> + + <p><em>General events:</em></p> + + <list> + <item><c>#event{name = start_logging, data = LogDir}</c> + <p><c>LogDir = string()</c>, top level log directory for the test run.</p> + <p>Indicates that the logging process of Common Test + has started successfully and is ready to receive IO + messages.</p></item> + + <item><c>#event{name = stop_logging, data = []}</c> + <p>Indicates that the logging process of Common Test + has been shut down at the end of the test run. + </p></item> + + <item><c>#event{name = test_start, data = {StartTime,LogDir}}</c> + <p><c>StartTime = {date(),time()}</c>, test run start date and time.</p> + <p><c>LogDir = string()</c>, top level log directory for the test run.</p> + <p>This event indicates that Common Test has finished initial preparations + and will begin executing test cases. + </p></item> + + <item><c>#event{name = test_done, data = EndTime}</c> + <p><c>EndTime = {date(),time()}</c>, date and time the test run finished.</p> + <p>This indicates that the last test case has been executed and + Common Test is shutting down. + </p></item> + + <item><c>#event{name = start_info, data = {Tests,Suites,Cases}}</c> + <p><c>Tests = integer()</c>, the number of tests.</p> + <p><c>Suites = integer()</c>, the total number of suites.</p> + <p><c>Cases = integer() | unknown</c>, the total number of test cases.</p> + <p>Initial test run information that can be interpreted as: "This test + run will execute <c>Tests</c> separate tests, in total containing + <c>Cases</c> number of test cases, in <c>Suites</c> number of suites". + Note that if a test case group with a repeat property exists in any test, + the total number of test cases can not be calculated (unknown). + </p></item> + + <item><c>#event{name = tc_start, data = {Suite,FuncOrGroup}}</c> + <p><c>Suite = atom()</c>, name of the test suite.</p> + <p><c>FuncOrGroup = Func | {Conf,GroupName,GroupProperties}</c></p> + <p><c>Func = atom()</c>, name of test case or configuration function.</p> + <p><c>Conf = init_per_group | end_per_group</c>, group configuration function.</p> + <p><c>GroupName = atom()</c>, name of the group.</p> + <p><c>GroupProperties = list()</c>, list of execution properties for the group.</p> + <p>This event informs about the start of a test case, or a group configuration + function. The event is sent also for <c>init_per_suite</c> and <c>end_per_suite</c>, + but not for <c>init_per_testcase</c> and <c>end_per_testcase</c>. If a group + configuration function is starting, the group name and execution properties + are also given. + </p></item> + + <item><c>#event{name = tc_done, data = {Suite,FuncOrGroup,Result}}</c> + <p><c>Suite = atom()</c>, name of the suite.</p> + <p><c>FuncOrGroup = Func | {Conf,GroupName,GroupProperties}</c></p> + <p><c>Func = atom()</c>, name of test case or configuration function.</p> + <p><c>Conf = init_per_group | end_per_group</c>, group configuration function.</p> + <p><c>GroupName = unknown | atom()</c>, name of the group + (unknown if init- or end function times out).</p> + <p><c>GroupProperties = list()</c>, list of execution properties for the group.</p> + <p><c>Result = ok | {skipped,SkipReason} | {failed,FailReason}</c>, the result.</p> + <p><c>SkipReason = {require_failed,RequireInfo} | + {require_failed_in_suite0,RequireInfo} | + {failed,{Suite,init_per_testcase,FailInfo}} | + UserTerm</c>, + the reason why the case has been skipped.</p> + <p><c>FailReason = {error,FailInfo} | + {error,{RunTimeError,StackTrace}} | + {timetrap_timeout,integer()} | + {failed,{Suite,end_per_testcase,FailInfo}}</c>, reason for failure.</p> + <p><c>RequireInfo = {not_available,atom()}</c>, why require has failed.</p> + <p><c>FailInfo = {timetrap_timeout,integer()} | + {RunTimeError,StackTrace} | + UserTerm</c>, + detailed information about an error.</p> + <p><c>RunTimeError = term()</c>, a run-time error, e.g. badmatch, undef, etc.</p> + <p><c>StackTrace = list()</c>, list of function calls preceeding a run-time error.</p> + <p><c>UserTerm = term()</c>, arbitrary data specified by user, or <c>exit/1</c> info.</p> + <p>This event informs about the end of a test case or a configuration function (see the + <c>tc_start</c> event for details on the FuncOrGroup element). With this event comes the + final result of the function in question. It is possible to determine on the top level + of <c>Result</c> if the function was successful, skipped (by the user), or if it failed. + It is of course possible to dig deeper and also perform pattern matching on the various + reasons for skipped or failed. Note that <c>{'EXIT',Reason}</c> tuples have been translated into + <c>{error,Reason}</c>. Note also that if a <c>{failed,{Suite,end_per_testcase,FailInfo}</c> + result is received, it actually means the test case was successful, but that + <c>end_per_testcase</c> for the case failed. + </p></item> + + <item><c>#event{name = tc_auto_skip, data = {Suite,Func,Reason}}</c> + <p><c>Suite = atom()</c>, the name of the suite.</p> + <p><c>Func = atom()</c>, the name of the test case or configuration function.</p> + <p><c>Reason = {failed,FailReason} | + {require_failed_in_suite0,RequireInfo}</c>, + reason for auto skipping <c>Func</c>.</p> + <p><c>FailReason = {Suite,ConfigFunc,FailInfo}} | + {Suite,FailedCaseInSequence}</c>, reason for failure.</p> + <p><c>RequireInfo = {not_available,atom()}</c>, why require has failed.</p> + <p><c>ConfigFunc = init_per_suite | init_per_group</c></p> + <p><c>FailInfo = {timetrap_timeout,integer()} | + {RunTimeError,StackTrace} | + bad_return | UserTerm</c>, + detailed information about an error.</p> + <p><c>FailedCaseInSequence = atom()</c>, name of a case that has failed in a sequence.</p> + <p><c>RunTimeError = term()</c>, a run-time error, e.g. badmatch, undef, etc.</p> + <p><c>StackTrace = list()</c>, list of function calls preceeding a run-time error.</p> + <p><c>UserTerm = term()</c>, arbitrary data specified by user, or <c>exit/1</c> info.</p> + <p>This event gets sent for every test case or configuration function that Common Test + has skipped automatically because of either a failed <c>init_per_suite</c> or + <c>init_per_group</c>, a failed <c>require</c> in <c>suite/0</c>, or a failed test case + in a sequence. Note that this event is never received as a result of a test case getting + skipped because of <c>init_per_testcase</c> failing, since that information is carried with + the <c>tc_done</c> event. + </p></item> + + <item><c>#event{name = tc_user_skip, data = {Suite,TestCase,Comment}}</c> + <p><c>Suite = atom()</c>, name of the suite.</p> + <p><c>TestCase = atom()</c>, name of the test case.</p> + <p><c>Comment = string()</c>, reason for skipping the test case.</p> + <p>This event specifies that a test case has been skipped by the user. + It is only ever received if the skip was declared in a test specification. + Otherwise, user skip information is received as a <c>{skipped,SkipReason}</c> + result in the <c>tc_done</c> event for the test case. + </p></item> + + <item><c>#event{name = test_stats, data = {Ok,Failed,Skipped}}</c> + <p><c>Ok = integer()</c>, the current number of successful test cases.</p> + <p><c>Failed = integer()</c>, the current number of failed test cases.</p> + <p><c>Skipped = {UserSkipped,AutoSkipped}</c></p> + <p><c>UserSkipped = integer()</c>, the current number of user skipped test cases.</p> + <p><c>AutoSkipped = integer()</c>, the current number of auto skipped test cases.</p> + <p>This is a statistics event with the current count of successful, skipped + and failed test cases so far. This event gets sent after the end of each test case, + immediately following the <c>tc_done</c> event. + </p></item> + </list> + + <p><em>Internal events:</em></p> + + <list> + <item><c>#event{name = start_make, data = Dir}</c> + <p><c>Dir = string()</c>, running make in this directory.</p> + <p>An internal event saying that Common Test will start compiling + modules in directory <c>Dir</c>. + </p></item> + + <item><c>#event{name = finished_make, data = Dir}</c> + <p><c>Dir = string()</c>, finished running make in this directory.</p> + <p>An internal event saying that Common Test is finished compiling + modules in directory <c>Dir</c>. + </p></item> + + <item><c>#event{name = start_write_file, data = FullNameFile}</c> + <p><c>FullNameFile = string(), full name of the file.</c></p> + <p>An internal event used by the Common Test Master process to + synchronize particular file operations. + </p></item> + + <item><c>#event{name = finished_write_file, data = FullNameFile}</c> + <p><c>FullNameFile = string(), full name of the file.</c></p> + <p>An internal event used by the Common Test Master process to + synchronize particular file operations. + </p></item> + + </list> + + <p>The events are also documented in <c>ct_event.erl</c>. This module + may serve as an example of what an event handler for the CT event + manager can look like.</p> + + <note><p>To ensure that printouts to standard out (or printouts made with + <c>ct:log/2/3</c> or <c>ct:pal/2/3</c>) get written to the test case log + file, and not to the Common Test framework log, you can syncronize + with the Common Test server by matching on the <c>tc_start</c> and <c>tc_done</c> + events. In the period between these events, all IO gets directed to the + test case log file. These events are sent synchronously to avoid potential + timing problems (e.g. that the test case log file gets closed just before + an IO message from an external process gets through). Knowing this, you + need to be careful that your <c>handle_event/2</c> callback function doesn't + stall the test execution, possibly causing unexpected behaviour as a result.</p></note> +</section> +</chapter> + + + + |