From 2ef840647acadb489d54332f6a218dcf2e629ff9 Mon Sep 17 00:00:00 2001 From: tmanevik Date: Wed, 18 Nov 2015 18:24:10 +0100 Subject: Common Test: Editorial changes 1 Conflicts: lib/common_test/doc/src/ct_hooks_chapter.xml lib/common_test/doc/src/event_handler_chapter.xml lib/common_test/doc/src/run_test_chapter.xml --- lib/common_test/doc/src/event_handler_chapter.xml | 384 ++++++++++++---------- 1 file changed, 215 insertions(+), 169 deletions(-) (limited to 'lib/common_test/doc/src/event_handler_chapter.xml') diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml index 78e5bb5e70..31128a7114 100644 --- a/lib/common_test/doc/src/event_handler_chapter.xml +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -33,144 +33,171 @@
General -

It is possible for the operator of a Common Test system to receive - event notifications continously during a test run. It is reported e.g. - when a test case starts and stops, what the current count of successful, - failed and skipped cases is, etc. This information can be used for - different purposes such as logging progress and results on - other format than HTML, saving statistics to a database for report - generation and test system supervision.

- -

Common Test has a framework for event handling which is based on - the OTP event manager concept and gen_event behaviour. When the Common Test - server starts, it spawns an event manager. During test execution the - manager gets a notification from the server every time something - of potential interest happens. Any event handler plugged into the - event manager can match on events of interest, take action, or maybe - simply pass the information on. Event handlers are Erlang modules - implemented by the Common Test user according to the gen_event - behaviour (see the OTP User's Guide and Reference Manual for more - information).

- -

As already described, a Common Test server always starts an event manager. - The server also plugs in a default event handler which has as its only - purpose to relay notifications to a globally registered CT Master - event manager (if a CT Master server is running in the system). - The CT Master also spawns an event manager at startup. - Event handlers plugged into this manager will receive the events from - all the test nodes as well as information from the CT Master server - itself.

- -

User specific event handlers may be plugged into a Common Test event - manager, either by telling Common Test to install them before the test - run (see below), or by adding the handlers dynamically during the test - run by means of - gen_event:add_handler/3 or gen_event:add_sup_handler/3. - In the latter scenario, the reference of the Common Test event manager is - required. To get it, call ct:get_event_mgr_ref/0 or (on the CT - Master node) ct_master:get_event_mgr_ref/0.

+

The operator of a Common Test system can receive + event notifications continuously during a test run. For example, + Common Test reports when a test case starts and stops, + the current count of successful, failed, and skipped cases, and so on. + This information can be used for different purposes such as logging progress + and results in another format than HTML, saving statistics to a database + for report generation, and test system supervision.

+ +

Common Test has a framework for event handling based on + the OTP event manager concept and gen_event behavior. + When the Common Test server starts, it spawns an event manager. + During test execution the manager gets a notification from the server + when something of potential interest happens. Any event handler plugged into + the event manager can match on events of interest, take action, or + pass the information on. The event handlers are Erlang modules + implemented by the Common Test user according to the gen_event + behavior (for details, see module + stdlib:gen_event and + section + gen_event Behaviour + in OTP Design Principles in the System Documentation). +

+ +

A Common Test server always starts an event manager. + The server also plugs in a default event handler, which only + purpose is to relay notifications to a globally registered Common Test + Master event manager (if a Common Test Master server is running in the system). + The Common Test Master also spawns an event manager at startup. + Event handlers plugged into this manager receives the events from + all the test nodes, plus information from the Common Test Master server. +

+ +

User-specific event handlers can be plugged into a Common Test event + manager, either by telling Common Test to install them before the test + run (described later), or by adding the handlers dynamically during the test + run using + stdlib:gen_event:add_handler/3 or + stdlib:gen_event:add_sup_handler/3. + In the latter scenario, the reference of the Common Test event manager is + required. To get it, call + ct:get_event_mgr_ref/0 + or (on the Common Test Master node) + ct_master:get_event_mgr_ref/0.

- Usage -

Event handlers may be installed by means of an event_handler - start flag (ct_run) or option (ct:run_test/1), where the - argument specifies the names of one or more event handler modules. - Example:

+ Use +

Event handlers can be installed by an event_handler start flag + (ct_run) or option + ct:run_test/1, where the + argument specifies the names of one or more event handler modules.

+ +

Example:

$ ct_run -suite test/my_SUITE -event_handler handlers/my_evh1 handlers/my_evh2 -pa $PWD/handlers

-

Use the option instead of - to pass start arguments to the event handler - init function.

-

All event handler modules must have gen_event behaviour. Note also that - these modules must be precompiled, and that their locations must be - added explicitly to the Erlang code server search path (like in the - example).

-

An event_handler tuple in the argument Opts has the following - definition (see also ct:run_test/1 in the reference manual):

+

To pass start arguments to the event handler init function, use option + instead of + .

+ +

All event handler modules must have gen_event behavior. + These modules must be precompiled and their locations must be + added explicitly to the Erlang code server search path (as in the previous + example).

 + +

An event_handler tuple in argument Opts has the following definition + (see ct:run_test/1):

-    {event_handler,EventHandlers}
+ {event_handler,EventHandlers}
 
-    EventHandlers = EH | [EH]
-    EH = atom() | {atom(),InitArgs} | {[atom()],InitArgs}
-    InitArgs = [term()]
+ EventHandlers = EH | [EH] + EH = atom() | {atom(),InitArgs} | {[atom()],InitArgs} + InitArgs = [term()] -

Example:

+

In the following example, two event handlers for the my_SUITE test are installed:

-    1> ct:run_test([{suite,"test/my_SUITE"},{event_handler,[my_evh1,{my_evh2,[node()]}]}]).
-

This will install two event handlers for the my_SUITE test. Event handler - my_evh1 is started with [] as argument to the init function. Event handler - my_evh2 is started with the name of the current node in the init argument list.

+ 1> ct:run_test([{suite,"test/my_SUITE"},{event_handler,[my_evh1,{my_evh2,[node()]}]}]). +

Event handler my_evh1 is started with [] as argument to the init function. + Event handler my_evh2 is started with the name of the current node in the init argument list.

-

Event handlers can also be plugged in by means of +

Event handlers can also be plugged in using one of the following test specification terms:

- -

{event_handler, EventHandlers}, or

-

{event_handler, EventHandlers, InitArgs}, or

-

{event_handler, NodeRefs, EventHandlers}, or

-

{event_handler, NodeRefs, EventHandlers, InitArgs}

+ + {event_handler, EventHandlers} + {event_handler, EventHandlers, InitArgs} + {event_handler, NodeRefs, EventHandlers} + {event_handler, NodeRefs, EventHandlers, InitArgs} +

EventHandlers is a list of module names. Before a test session starts, the init function of each plugged in event handler - is called (with the InitArgs list as argument or [] if - no start arguments are given).

+ is called (with the InitArgs list as argument or [] if + no start arguments are specified).

-

To plug a handler into the CT Master event manager, specify +

To plug in a handler to the Common Test Master event manager, specify master as the node in NodeRefs.

-

For an event handler to be able to match on events, the module must +

To be able to match on events, the event handler module must include the header file ct_event.hrl. An event is a record with the following definition:

#event{name, node, data}

-

name is the label (type) of the event. node is the name of the - node the event has originated from (only relevant for CT Master event handlers). - data is specific for the particular event.

+ + name +

Label (type) of the event.

 + node +

Name of the node that the event originated from + (only relevant for Common Test Master event handlers).

 + data +

Specific for the event.

 + + -

General events:

+
+ General Events - - #event{name = start_logging, data = LogDir} -

LogDir = string(), top level log directory for the test run.

-

Indicates that the logging process of Common Test - has started successfully and is ready to receive IO +

The general events are as follows:

+ + + #event{name = start_logging, data = LogDir} + +

LogDir = string(), top-level log directory for the test run.

+

This event indicates that the logging process of Common Test + has started successfully and is ready to receive I/O messages.

 - #event{name = stop_logging, data = []} -

Indicates that the logging process of Common Test - has been shut down at the end of the test run. + #event{name = stop_logging, data = []} + +

This event indicates that the logging process of Common Test + was shut down at the end of the test run.

- #event{name = test_start, data = {StartTime,LogDir}} + #event{name = test_start, data = {StartTime,LogDir}} +

StartTime = {date(),time()}, test run start date and time.

-

LogDir = string(), top level log directory for the test run.

-

This event indicates that Common Test has finished initial preparations - and will begin executing test cases. +

LogDir = string(), top-level log directory for the test run.

+

This event indicates that Common Test has finished initial preparations + and begins executing test cases.

- #event{name = test_done, data = EndTime} + #event{name = test_done, data = EndTime} +

EndTime = {date(),time()}, date and time the test run finished.

-

This indicates that the last test case has been executed and - Common Test is shutting down. +

This event indicates that the last test case has been executed and + Common Test is shutting down.

- #event{name = start_info, data = {Tests,Suites,Cases}} -

Tests = integer(), the number of tests.

-

Suites = integer(), the total number of suites.

-

Cases = integer() | unknown, the total number of test cases.

-

Initial test run information that can be interpreted as: "This test - run will execute Tests separate tests, in total containing + #event{name = start_info, data = {Tests,Suites,Cases}} + +

Tests = integer(), number of tests.

+

Suites = integer(), total number of suites.

+

Cases = integer() | unknown, total number of test cases.

+

This event gives initial test run information that can be interpreted as: + "This test run will execute Tests separate tests, in total containing Cases number of test cases, in Suites number of suites". - Note that if a test case group with a repeat property exists in any test, - the total number of test cases can not be calculated (unknown). + However, if a test case group with a repeat property exists in any test, + the total number of test cases cannot be calculated (unknown).

- #event{name = tc_start, data = {Suite,FuncOrGroup}} + #event{name = tc_start, data = {Suite,FuncOrGroup}} +

Suite = atom(), name of the test suite.

FuncOrGroup = Func | {Conf,GroupName,GroupProperties}

Func = atom(), name of test case or configuration function.

@@ -180,23 +207,24 @@

This event informs about the start of a test case, or a group configuration function. The event is sent also for init_per_suite and end_per_suite, but not for init_per_testcase and end_per_testcase. If a group - configuration function is starting, the group name and execution properties - are also given. + configuration function starts, the group name and execution properties + are also specified.

- #event{name = tc_logfile, data = {{Suite,Func},LogFileName}} + #event{name = tc_logfile, data = {{Suite,Func},LogFileName}} +

Suite = atom(), name of the test suite.

Func = atom(), name of test case or configuration function.

-

LogFileName = string(), full name of test case log file.

+

LogFileName = string(), full name of the test case log file.

This event is sent at the start of each test case (and configuration function except init/end_per_testcase) and carries information about the - full name (i.e. the file name including the absolute directory path) of + full name (that is, the file name including the absolute directory path) of the current test case log file.

+ #event{name = tc_done, data = {Suite,FuncOrGroup,Result}} - - #event{name = tc_done, data = {Suite,FuncOrGroup,Result}} +

Suite = atom(), name of the suite.

FuncOrGroup = Func | {Conf,GroupName,GroupProperties}

Func = atom(), name of test case or configuration function.

@@ -211,34 +239,37 @@ {require_failed_in_suite0,RequireInfo} | {failed,{Suite,init_per_testcase,FailInfo}} | UserTerm, - the reason why the case has been skipped.

+ why the case was skipped.

FailReason = {error,FailInfo} | {error,{RunTimeError,StackTrace}} | {timetrap_timeout,integer()} | {failed,{Suite,end_per_testcase,FailInfo}}, reason for failure.

-

RequireInfo = {not_available,atom() | tuple()}, why require has failed.

+

RequireInfo = {not_available,atom() | tuple()}, why require failed.

FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | UserTerm, - detailed information about an error.

-

RunTimeError = term(), a run-time error, e.g. badmatch, undef, etc.

-

StackTrace = list(), list of function calls preceeding a run-time error.

-

UserTerm = term(), arbitrary data specified by user, or exit/1 info.

-

This event informs about the end of a test case or a configuration function (see the - tc_start event for details on the FuncOrGroup element). With this event comes the - final result of the function in question. It is possible to determine on the top level - of Result if the function was successful, skipped (by the user), or if it failed. - It is of course possible to dig deeper and also perform pattern matching on the various - reasons for skipped or failed. Note that {'EXIT',Reason} tuples have been translated into - {error,Reason}. Note also that if a {failed,{Suite,end_per_testcase,FailInfo} - result is received, it actually means the test case was successful, but that + error details.

+

RunTimeError = term(), a runtime error, for example, + badmatch or undef.

+

StackTrace = list(), list of function calls preceding a runtime error.

+

UserTerm = term(), any data specified by user, or exit/1 information.

+

This event informs about the end of a test case or a configuration function (see event + tc_start for details on element FuncOrGroup). With this event + comes the final result of the function in question. It is possible to determine on the + top level of Result if the function was successful, skipped (by the user), + or if it failed.

+

It is also possible to dig deeper and, for example, perform pattern matching + on the various reasons for skipped or failed. Notice that {'EXIT',Reason} tuples + are translated into {error,Reason}. + Notice also that if a {failed,{Suite,end_per_testcase,FailInfo} + result is received, the test case was successful, but end_per_testcase for the case failed.

+ #event{name = tc_auto_skip, data = {Suite,TestName,Reason}} - #event{name = tc_auto_skip, data = {Suite,TestName,Reason}}

Suite = atom(), the name of the suite.

TestName = init_per_suite | end_per_suite | {init_per_group,GroupName} | {end_per_group,GroupName} | @@ -247,101 +278,116 @@

GroupName = atom(), the name of the test case group.

Reason = {failed,FailReason} | {require_failed_in_suite0,RequireInfo}, - reason for auto skipping Func.

+ reason for auto-skipping Func.

FailReason = {Suite,ConfigFunc,FailInfo}} | {Suite,FailedCaseInSequence}, reason for failure.

-

RequireInfo = {not_available,atom() | tuple()}, why require has failed.

+

RequireInfo = {not_available,atom() | tuple()}, why require failed.

ConfigFunc = init_per_suite | init_per_group

FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | bad_return | UserTerm, - detailed information about an error.

-

FailedCaseInSequence = atom(), name of a case that has failed in a sequence.

-

RunTimeError = term(), a run-time error, e.g. badmatch, undef, etc.

-

StackTrace = list(), list of function calls preceeding a run-time error.

-

UserTerm = term(), arbitrary data specified by user, or exit/1 info.

-

This event gets sent for every test case or configuration function that Common Test + error details.

+

FailedCaseInSequence = atom(), the name of a case that failed in a sequence.

+

RunTimeError = term(), a runtime error, for example badmatch or + undef.

+

StackTrace = list(), list of function calls preceeding a runtime error.

+

UserTerm = term(), any data specified by user, or exit/1 information.

+

This event is sent for every test case or configuration function that Common Test has skipped automatically because of either a failed init_per_suite or init_per_group, a failed require in suite/0, or a failed test case - in a sequence. Note that this event is never received as a result of a test case getting - skipped because of init_per_testcase failing, since that information is carried with - the tc_done event. If a failed test case belongs to a test case group, the second - data element is a tuple {FuncName,GroupName}, otherwise simply the function name. + in a sequence. Notice that this event is never received as a result of a test case getting + skipped because of init_per_testcase failing, as that information is carried with + event tc_done. If a failed test case belongs to a test case group, the second + data element is a tuple {FuncName,GroupName}, otherwise only the function name.

+ #event{name = tc_user_skip, data = {Suite,TestName,Comment}} - - #event{name = tc_user_skip, data = {Suite,TestName,Comment}} +

Suite = atom(), the name of the suite.

TestName = init_per_suite | end_per_suite | {init_per_group,GroupName} | {end_per_group,GroupName} | {FuncName,GroupName} | FuncName

FuncName = atom(), the name of the test case or configuration function.

GroupName = atom(), the name of the test case group.

-

Comment = string(), reason for skipping the test case.

-

This event specifies that a test case has been skipped by the user. - It is only ever received if the skip was declared in a test specification. +

Comment = string(), why the test case was skipped.

+

This event specifies that a test case was skipped by the user. + It is only received if the skip is declared in a test specification. Otherwise, user skip information is received as a {skipped,SkipReason} - result in the tc_done event for the test case. If a skipped test case belongs + result in event tc_done for the test case. If a skipped test case belongs to a test case group, the second data element is a tuple {FuncName,GroupName}, - otherwise simply the function name. + otherwise only the function name.

- #event{name = test_stats, data = {Ok,Failed,Skipped}} -

Ok = integer(), the current number of successful test cases.

-

Failed = integer(), the current number of failed test cases.

+ #event{name = test_stats, data = {Ok,Failed,Skipped}} + +

Ok = integer(), current number of successful test cases.

+

Failed = integer(), current number of failed test cases.

Skipped = {UserSkipped,AutoSkipped}

-

UserSkipped = integer(), the current number of user skipped test cases.

-

AutoSkipped = integer(), the current number of auto skipped test cases.

-

This is a statistics event with the current count of successful, skipped - and failed test cases so far. This event gets sent after the end of each test case, - immediately following the tc_done event. +

UserSkipped = integer(), current number of user-skipped test cases.

+

AutoSkipped = integer(), current number of auto-skipped test cases.

+

This is a statistics event with current count of successful, skipped, + and failed test cases so far. This event is sent after the end of each test case, + immediately following event tc_done.

- + +
+ +
+ Internal Events -

Internal events:

+

The internal events are as follows:

- - #event{name = start_make, data = Dir} + + #event{name = start_make, data = Dir} +

Dir = string(), running make in this directory.

-

An internal event saying that Common Test will start compiling +

This internal event says that Common Test starts compiling modules in directory Dir.

- #event{name = finished_make, data = Dir} + #event{name = finished_make, data = Dir} +

Dir = string(), finished running make in this directory.

-

An internal event saying that Common Test is finished compiling +

This internal event says that Common Test is finished compiling modules in directory Dir.

- #event{name = start_write_file, data = FullNameFile} + #event{name = start_write_file, data = FullNameFile} +

FullNameFile = string(), full name of the file.

-

An internal event used by the Common Test Master process to +

This internal event is used by the Common Test Master process to synchronize particular file operations.

- #event{name = finished_write_file, data = FullNameFile} + #event{name = finished_write_file, data = FullNameFile} +

FullNameFile = string(), full name of the file.

-

An internal event used by the Common Test Master process to +

This internal event is used by the Common Test Master process to synchronize particular file operations.

- - + +
+
+ Notes +

The events are also documented in ct_event.erl. This module - may serve as an example of what an event handler for the CT event + can serve as an example of what an event handler for the Common Test event manager can look like.

-

To ensure that printouts to standard out (or printouts made with - ct:log/2/3 or ct:pal/2/3) get written to the test case log - file, and not to the Common Test framework log, you can syncronize - with the Common Test server by matching on the tc_start and tc_done - events. In the period between these events, all IO gets directed to the +

To ensure that printouts to stdout (or printouts made with + ct:log/2,3 or + ct:pal,2,3) get written to the test case log + file, and not to the Common Test framework log, you can synchronize + with the Common Test server by matching on evvents tc_start and tc_done. + In the period between these events, all I/O is directed to the test case log file. These events are sent synchronously to avoid potential - timing problems (e.g. that the test case log file gets closed just before - an IO message from an external process gets through). Knowing this, you - need to be careful that your handle_event/2 callback function doesn't - stall the test execution, possibly causing unexpected behaviour as a result.

 + timing problems (for example, that the test case log file is closed just before + an I/O message from an external process gets through). Knowing this, you + need to be careful that your handle_event/2 callback function does not + stall the test execution, possibly causing unexpected behavior as a result.

+
-- cgit v1.2.3