From 84adefa331c4159d432d22840663c38f155cd4c1 Mon Sep 17 00:00:00 2001 From: Erlang/OTP Date: Fri, 20 Nov 2009 14:54:40 +0000 Subject: The R13B03 release. --- lib/common_test/doc/src/common_test_app.xml | 448 ++++++++++++++++++++++++++++ 1 file changed, 448 insertions(+) create mode 100644 lib/common_test/doc/src/common_test_app.xml (limited to 'lib/common_test/doc/src/common_test_app.xml') diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml new file mode 100644 index 0000000000..7b52883f8a --- /dev/null +++ b/lib/common_test/doc/src/common_test_app.xml @@ -0,0 +1,448 @@ + + + + +
+ + 20032009 + Ericsson AB. All Rights Reserved. + + + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + + + Common Test + Kenneth Lundin, Peter Andersson + Peter Andersson + + + + 2003-10-21 + PA1 + common_test_app.sgml +
+ common_test + A framework for automated testing of arbitrary target nodes + + + +

The Common Test framework is an environment for + implementing and performing automatic and semi-automatic execution of + test cases. + + Common Test uses the OTP Test Server as engine for test case + execution and logging.

+ +

In brief, Common Test supports:

+ + + Automated execution of test suites (sets of test cases). + Logging of the events during execution. + HTML presentation of test suite results. + HTML presentation of test suite code. + Support functions for test suite authors. + Step by step execution of test cases. + + +

The following sections describe the mandatory and optional test suite + functions Common Test will call during test execution. For more details + see Common Test User's + Guide.

+ + + +
+ TEST CASE CALLBACK FUNCTIONS +

The following functions define the callback interface + for a test suite.

+
+ + + + Module:all() -> TestCases | {skip,Reason} + Returns the list of all test cases in the module. + + TestCases = [atom() | {group,GroupName}] + Reason = term() + GroupName = atom() + + + +

MANDATORY

+ +

This function must return the list of all test cases and test + case groups in the test suite module that are to be executed. + This list also specifies the order the cases and groups will + be executed by Common Test. A test case is represented by an atom, + the name of the test case function. A test case group is + represented by a {group,GroupName} tuple, where + GroupName, an atom, is the name of the group (defined + with groups/0).

+ +

If {skip,Reason} is returned, all test cases + in the module will be skipped, and the Reason will + be printed on the HTML result page.

+ +

For details on groups, see + Test case + groups in the User's Guide.

+ + + + + + Module:groups() -> GroupDefs + Returns a list of test case group definitions. + + GroupDefs = [Group] + Group = {GroupName,Properties,GroupsAndTestCases} + GroupName = atom() + Properties = [parallel | sequence | Shuffle | {RepeatType,N}] + GroupsAndTestCases = [Group | {group,GroupName} | TestCase] + TestCase = atom() + Shuffle = shuffle | {shuffle,Seed} + Seed = {integer(),integer(),integer()} + RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | + repeat_until_any_ok | repeat_until_any_fail + N = integer() | forever + + + +

OPTIONAL

+ +

See Test case + groups in the User's Guide for details.

+ + + + + Module:suite() -> [Info] + Test suite info function (providing default data for the suite). + + Info = {timetrap,Time} | {require,Required} | + {require,Name,Required} | {userdata,UserData} | + {silent_connections,Conns} | {stylesheet,CSSFile} + Time = MilliSec | {seconds,integer()} | {minutes,integer()} + | {hours,integer()} + MilliSec = integer() + Required = Key | {Key,SubKeys} + Key = atom() + SubKeys = SubKey | [SubKey] + SubKey = atom() + Name = atom() + UserData = term() + Conns = [atom()] + CSSFile = string() + + + +

OPTIONAL

+ +

This is the test suite info function. It is supposed to + return a list of tagged tuples that specify various properties + regarding the execution of this test suite (common for all + test cases in the suite).

+ +

The timetrap tag sets the maximum time each + test case is allowed to take (including init_per_testcase/2 + and end_per_testcase/2). If the timetrap time is + exceeded, the test case fails with reason + timetrap_timeout.

+ +

The require tag specifies configuration variables + that are required by test cases in the suite. If the required + configuration variables are not found in any of the + configuration files, all test cases are skipped. For more + information about the 'require' functionality, see the + reference manual for the function + ct:require/[1,2].

+ +

With userdata, it is possible for the user to + specify arbitrary test suite related information which can be + read by calling ct:userdata/2.

+ +

Other tuples than the ones defined will simply be ignored.

+ +

For more information about the test suite info function, + see Test + suite info function in the User's Guide.

+ + + + + Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | {skip_and_save,Reason,SaveConfig} + Test suite initialization. + + Config = NewConfig = SaveConfig = [{Key,Value}] + Key = atom() + Value = term() + Reason = term() + + + +

OPTIONAL

+ +

This function is called as the first function in the + suite. It typically contains initialization which is common for + all test cases in the suite, and which shall only be done + once. The Config parameter is the configuration + which can be modified here. Whatever is returned from this + function is given as Config to all configuration functions + and test cases in the suite. If {skip,Reason} + is returned, all test cases in the suite will be skipped and Reason + printed in the overview log for the suite.

+

For information on save_config and skip_and_save, + please see + Dependencies + between Test Cases and Suites in the User's Guide.

+ + + + + Module:end_per_suite(Config) -> void() | {save_config,SaveConfig} + Test suite finalization. + + Config = SaveConfig = [{Key,Value}] + Key = atom() + Value = term() + + + +

OPTIONAL

+ +

This function is called as the last test case in the + suite. It is meant to be used for cleaning up after init_per_suite/1. + For information on save_config, please see + Dependencies between + Test Cases and Suites in the User's Guide.

+ + + + + Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason} + Test case group initialization. + + GroupName = atom() + Config = NewConfig = [{Key,Value}] + Key = atom() + Value = term() + Reason = term() + + + +

MANDATORY (only if one or more groups are defined)

+ +

This function is called before execution of a test case group. + It typically contains initialization which is common for + all test cases in the group, and which shall only be performed + once. GroupName is the name of the group, as specified in + the group definition (see groups/0). The Config + parameter is the configuration which can be modified here. + Whatever is returned from this function is given as Config + to all test cases in the group. If {skip,Reason} is returned, + all test cases in the group will be skipped and Reason printed + in the overview log for the group.

+ +

For information about test case groups, please see + Test case + groups chapter in the User's Guide.

+ + + + + Module:end_per_group(GroupName, Config) -> void() | {return_group_result,Status} + Test case group finalization. + + GroupName = atom() + Config = [{Key,Value}] + Key = atom() + Value = term() + Status = ok | skipped | failed + + + +

MANDATORY (only if one or more groups are defined)

+ +

This function is called after the execution of a test case group is finished. + It is meant to be used for cleaning up after init_per_group/2. + By means of {return_group_result,Status}, it is possible to return a + status value for a nested sub-group. The status can be retrieved in + end_per_group/2 for the group on the level above. The status will also + be used by Common Test for deciding if execution of a group should proceed in + case the property sequence or repeat_until_* is set.

+ +

For more information about test case groups, please see + Test case + groups chapter in the User's Guide.

+ + + + + Module:init_per_testcase(TestCase, Config) -> NewConfig | {skip,Reason} + Test case initialization. + + TestCase = atom() + Config = NewConfig = [{Key,Value}] + Key = atom() + Value = term() + Reason = term() + + + +

OPTIONAL

+ +

This function is called before each test case. The + TestCase argument is the name of the test case, and + Config is the configuration which can be modified + here. Whatever is returned from this function is given as + Config to the test case. If {skip,Reason} is returned, + the test case will be skipped and Reason printed + in the overview log for the suite.

+ + + + + Module:end_per_testcase(TestCase, Config) -> void() | {fail,Reason} | {save_config,SaveConfig} + Test case finalization. + + TestCase = atom() + Config = SaveConfig = [{Key,Value}] + Key = atom() + Value = term() + Reason = term() + + + +

OPTIONAL

+ +

This function is called after each test case, and can be used + to clean up after init_per_testcase/2 and the test case. + Any return value (besides {fail,Reason} and {save_config,SaveConfig}) + is ignored. By returning {fail,Reason}, TestCase will be marked as + failed (even though it was actually successful in the sense that it returned + a value instead of terminating). For information on save_config, please see + Dependencies between + Test Cases and Suites in the User's Guide

+ + + + + Module:testcase() -> [Info] + Test case info function. + + Info = {timetrap,Time} | {require,Required} | + {require,Name,Required} | {userdata,UserData} | + {silent_connections,Conns} + Time = MilliSec | {seconds,integer()} | {minutes,integer()} + | {hours,integer()} + MilliSec = integer() + Required = Key | {Key,SubKeys} + Key = atom() + SubKeys = SubKey | [SubKey] + SubKey = atom() + Name = atom() + UserData = term() + Conns = [atom()] + + + + +

OPTIONAL

+ +

This is the test case info function. It is supposed to + return a list of tagged tuples that specify various properties + regarding the execution of this particular test case.

+ +

The timetrap tag sets the maximum time the + test case is allowed to take. If the timetrap time is + exceeded, the test case fails with reason + timetrap_timeout. init_per_testcase/2 + and end_per_testcase/2 are included in the + timetrap time.

+ +

The require tag specifies configuration variables + that are required by the test case. If the required + configuration variables are not found in any of the + configuration files, the test case is skipped. For more + information about the 'require' functionality, see the + reference manual for the function + ct:require/[1,2].

+ +

If timetrap and/or require is not set, the + default values specified in the suite/0 return list + will be used.

+ +

With userdata, it is possible for the user to + specify arbitrary test case related information which can be + read by calling ct:userdata/3.

+ +

Other tuples than the ones defined will simply be ignored.

+ +

For more information about the test case info function, + see Test + case info function in the User's Guide.

+ + + + + + Module:testcase(Config) -> void() | {skip,Reason} | {comment,Comment} | {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit() + A test case + + Config = SaveConfig = [{Key,Value}] + Key = atom() + Value = term() + Reason = term() + Comment = string() + + + +

MANDATORY

+ +

This is the implementation of a test case. Here you must + call the functions you want to test, and do whatever you + need to check the result. If something fails, make sure the + function causes a runtime error, or call ct:fail/[0,1] + (which also causes the test case process to crash).

+ +

Elements from the Config parameter can be read + with the ?config macro. The config + macro is defined in ct.hrl

+ +

You can return {skip,Reason} if you decide not to + run the test case after all. Reason will then be + printed in 'Comment' field on the HTML result page.

+ +

You can return {comment,Comment} if you wish to + print some information in the 'Comment' field on the HTML + result page.

+ +

If the function returns anything else, the test case is + considered successful. (The return value always gets printed + in the test case log file).

+ +

For more information about test case implementation, please + see Test + cases in the User's Guide.

+ +

For information on save_config and skip_and_save, please see + Dependencies between + Test Cases and Suites in the User's Guide.

+ + + + + + + + -- cgit v1.2.3