diff options
Diffstat (limited to 'lib/common_test/doc/src')
22 files changed, 2506 insertions, 525 deletions
diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index 3ea6ae65d5..99161ce68a 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2003-2011. All Rights Reserved. +# Copyright Ericsson AB 2003-2012. All Rights Reserved. # # The contents of this file are subject to the Erlang Public License, # Version 1.1, (the "License"); you may not use this file except in @@ -46,7 +46,8 @@ CT_MODULES = \ ct_rpc \ ct_snmp \ unix_telnet \ - ct_slave + ct_slave \ + ct_netconfc CT_XML_FILES = $(CT_MODULES:=.xml) @@ -61,6 +62,7 @@ XML_PART_FILES = part.xml XML_CHAPTER_FILES = \ basics_chapter.xml \ + getting_started_chapter.xml \ install_chapter.xml \ write_test_chapter.xml \ test_structure_chapter.xml \ @@ -80,7 +82,10 @@ MAKE_EDOC = make_edoc BOOK_FILES = book.xml -GIF_FILES = +GIF_FILES = \ + tc_execution.gif \ + config.gif \ + html_logs.gif INSTALL_NOTES = ../../notes.html @@ -119,7 +124,7 @@ $(HTMLDIR)/%.gif: %.gif docs: pdf html man -$(CT_XML_FILES): +$(CT_XML_FILES): %.xml: ../../src/%.erl escript $(DOCGEN)/priv/bin/xml_from_edoc.escript -preprocess true -i $(XMERL_DIR)/include \ -i ../../../test_server/include -i ../../include \ -i ../../../../erts/lib/kernel/include -i ../../../../lib/kernel/include \ @@ -138,12 +143,6 @@ man: $(MAN6_FILES) $(MAN3_FILES) $(MAN1_FILES) debug opt: -# -# checkout make.dep before generating new dependecies -# -#make_doc_depend: xml -# docdepend > make.dep - clean clean_docs: rm -f $(CT_XML_FILES) rm -rf $(HTMLDIR)/* @@ -160,28 +159,19 @@ include $(ERL_TOP)/make/otp_release_targets.mk release_docs_spec: docs - $(INSTALL_DIR) $(RELSYSDIR)/doc/pdf - $(INSTALL_DATA) $(TOP_PDF_FILE) $(RELSYSDIR)/doc/pdf - $(INSTALL_DIR) $(RELSYSDIR)/doc/html + $(INSTALL_DIR) "$(RELSYSDIR)/doc/pdf" + $(INSTALL_DATA) $(TOP_PDF_FILE) "$(RELSYSDIR)/doc/pdf" + $(INSTALL_DIR) "$(RELSYSDIR)/doc/html" $(INSTALL_DATA) $(HTMLDIR)/* \ - $(RELSYSDIR)/doc/html - $(INSTALL_DATA) $(INFO_FILE) $(RELSYSDIR) - $(INSTALL_DIR) $(RELEASE_PATH)/man/man1 - $(INSTALL_DATA) $(MAN1DIR)/* $(RELEASE_PATH)/man/man1 - $(INSTALL_DIR) $(RELEASE_PATH)/man/man3 - $(INSTALL_DATA) $(MAN3DIR)/* $(RELEASE_PATH)/man/man3 - $(INSTALL_DIR) $(RELEASE_PATH)/man/man6 - $(INSTALL_DATA) $(MAN6DIR)/* $(RELEASE_PATH)/man/man6 + "$(RELSYSDIR)/doc/html" + $(INSTALL_DATA) $(INFO_FILE) "$(RELSYSDIR)" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man1" + $(INSTALL_DATA) $(MAN1DIR)/* "$(RELEASE_PATH)/man/man1" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man3" + $(INSTALL_DATA) $(MAN3DIR)/* "$(RELEASE_PATH)/man/man3" + $(INSTALL_DIR) "$(RELEASE_PATH)/man/man6" + $(INSTALL_DATA) $(MAN6DIR)/* "$(RELEASE_PATH)/man/man6" release_spec: release_tests_spec: - -# ---------------------------------------------------- -# Include dependency -# ---------------------------------------------------- - -include make.dep -# DO NOT DELETE - - diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index c1bb365b1f..ff6ea6c557 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2009</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,54 +28,65 @@ <rev></rev> <file>basics_chapter.xml</file> </header> - + <marker id="basics"></marker> <section> <title>Introduction</title> - <p> - The Common Test framework (CT) is a tool which can support - implementation and automated execution of test cases towards different - types of target systems. The framework is based on the OTP Test - Server. Test cases can be run individually or in batches. Common - Test also features a distributed testing mode with central - control and logging. This feature makes it possible to test - multiple systems independently in one common session. This - can be very useful e.g. for running automated large-scale regression - tests. + <p>The <em>Common Test</em> framework (CT) is a tool which supports + implementation and automated execution of test cases towards arbitrary + types of target systems. The CT framework is based on the OTP Test + Server and it's the main tool being used in all testing- and verification + activities that are part of Erlang/OTP system development- and maintenance. + </p> + + <p>Test cases can be executed individually or in batches. Common Test + also features a distributed testing mode with central control and logging + (a feature that makes it possible to test multiple systems independently in + one common session, useful e.g. for running automated large-scale regression + tests). </p> <p> The SUT (System Under Test) may consist of one or several target - nodes. CT contains a generic test server which together with - other test utilities is used to perform test case execution. - It is possible to start the tests from the CT GUI or from an OS- or - Erlang shell prompt. <em>Test suites</em> are files (Erlang + nodes. CT contains a generic test server which, together with + other test utilities, is used to perform test case execution. + It is possible to start the tests from a GUI or from the OS- or + Erlang shell. <em>Test suites</em> are files (Erlang modules) that contain the <em>test cases</em> (Erlang functions) to be executed. <em>Support modules</em> provide functions that the test cases utilize in order to carry out the tests. </p> - <p> - The main idea is that CT based test programs connect to - the target system(s) via standard O&M interfaces. CT - provides implementations and wrappers of some of these O&M - interfaces and will be extended with more interfaces later. - There are a number of target independent interfaces - supported in CT such as Generic Telnet, FTP etc. which can be - specialized or used directly for controlling instruments, - traffic generators etc.</p> + <p>In a black-box testing scenario, CT based test programs connect to + the target system(s) via standard O&M and CLI protocols. CT + provides implementations of, and wrapper interfaces to, some of these + protocols (most of which exist as stand-alone components and + applications in OTP). The wrappers simplify configuration and add + verbosity for logging purposes. CT will be continously extended with + useful support modules. (Note however that it's + a straightforward task to use any arbitrary Erlang/OTP component + for testing purposes with Common Test, without needing a CT wrapper + for it. It's as simple as calling Erlang functions). There + are a number of target independent interfaces supported in CT, such as + Generic Telnet, FTP, etc, which can be specialized or used + directly for controlling instruments, traffic load generators, etc. + </p> <p>Common Test is also a very useful tool for white-box testing Erlang - code since the test programs can call Erlang API functions directly. - For black-box testing Erlang software, Erlang RPC as well as - standard O&M interfaces can be used. + code (e.g. module testing), since the test programs can call exported Erlang + functions directly and there's very little overhead required for + implementing basic test suites and executing simple tests. For black-box + testing Erlang software, Erlang RPC as well as standard O&M interfaces + can for example be used. </p> <p>A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of - the major strengths of Common Test! + the major strengths of Common Test (thanks to the efficient + support for concurrency in the Erlang runtime system - which CT users + can take great advantage of!). </p> </section> @@ -130,8 +141,9 @@ individual test case. </p> <p> - The test suite module must conform to a callback interface specified - by the CT test server. See the + The test suite module must conform to a + <seealso marker="common_test">callback interface</seealso> + specified by the CT test server. See the <seealso marker="write_test_chapter#intro">Writing Test Suites</seealso> chapter for more information. </p> @@ -186,7 +198,7 @@ <taglist> <tag>all()</tag> - <item>Returns a list of all test cases in the suite. (Mandatory)</item> + <item>Returns a list of all test cases and groups in the suite. (Mandatory)</item> <tag>suite()</tag> <item>Info function used to return properties for the suite. (Optional)</item> <tag>groups()</tag> @@ -197,12 +209,14 @@ <tag>end_per_suite(Config)</tag> <item>Suite level configuration function, executed after the last test case. (Optional)</item> + <tag>group(GroupName)</tag> + <item>Info function used to return properties for a test case group. (Optional)</item> <tag>init_per_group(GroupName, Config)</tag> <item>Configuration function for a group, executed before the first - test case. (Mandatory if groups are defined)</item> + test case. (Optional)</item> <tag>end_per_group(GroupName, Config)</tag> <item>Configuration function for a group, executed after the last - test case. (Mandatory if groups are defined)</item> + test case. (Optional)</item> <tag>init_per_testcase(TestCase, Config)</tag> <item>Configuration function for a testcase, executed before each test case. (Optional)</item> diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index f58b2ab0a9..b6d4a633cb 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -4,7 +4,7 @@ <erlref> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -69,12 +69,25 @@ <funcs> <func> - <name>Module:all() -> TestCases | {skip,Reason} </name> - <fsummary>Returns the list of all test cases in the module.</fsummary> + <name>Module:all() -> Tests | {skip,Reason} </name> + <fsummary>Returns the list of all test case groups and test cases + in the module.</fsummary> <type> - <v>TestCases = [atom() | {group,GroupName}]</v> - <v>Reason = term()</v> + <v>Tests = [TestCase | {group,GroupName} | + {group,GroupName,Properties} | + {group,GroupName,Properties,SubGroups}]</v> + <v>TestCase = atom()</v> <v>GroupName = atom()</v> + <v>Properties = [parallel | sequence | Shuffle | {RepeatType,N}] | + default</v> + <v>SubGroups = [{GroupName,Properties} | + {GroupName,Properties,SubGroups}]</v> + <v>Shuffle = shuffle | {shuffle,Seed}</v> + <v>Seed = {integer(),integer(),integer()}</v> + <v>RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | + repeat_until_any_ok | repeat_until_any_fail</v> + <v>N = integer() | forever</v> + <v>Reason = term()</v> </type> <desc> @@ -85,9 +98,14 @@ This list also specifies the order the cases and groups will be executed by Common Test. A test case is represented by an atom, the name of the test case function. A test case group is - represented by a <c>{group,GroupName}</c> tuple, where - <c>GroupName</c>, an atom, is the name of the group (defined - with <c>groups/0</c>).</p> + represented by a <c>group</c> tuple, where <c>GroupName</c>, + an atom, is the name of the group (defined in <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). + Execution properties for groups may also be specified, both + for a top level group and for any of its sub-groups. + Group execution properties specified here, will override + properties in the group definition (see <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). + (With value <c>default</c>, the group definition properties + will be used).</p> <p> If <c>{skip,Reason}</c> is returned, all test cases in the module will be skipped, and the <c>Reason</c> will @@ -120,14 +138,16 @@ <desc> <p> OPTIONAL </p> - <p>See <seealso marker="write_test_chapter#test_case_groups">Test case + <p>Function for defining test case groups. Please see + <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso> in the User's Guide for details.</p> </desc> </func> <func> <name>Module:suite() -> [Info] </name> - <fsummary>Test suite info function (providing default data for the suite).</fsummary> + <fsummary>Test suite info function (providing default data + for the suite).</fsummary> <type> <v> Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | @@ -142,7 +162,7 @@ <v> Func = atom()</v> <v> Args = list()</v> <v> Fun = fun()</v> - <v> Required = Key | {Key,SubKeys}</v> + <v> Required = Key | {Key,SubKeys} | {Key,SubKey} | {Key,SubKey,SubKeys}</v> <v> Key = atom()</v> <v> SubKeys = SubKey | [SubKey]</v> <v> SubKey = atom()</v> @@ -160,28 +180,32 @@ <p>This is the test suite info function. It is supposed to return a list of tagged tuples that specify various properties - regarding the execution of this test suite (common for all + related to the execution of this test suite (common for all test cases in the suite).</p> <p>The <c>timetrap</c> tag sets the maximum time each - test case is allowed to take (including <c>init_per_testcase/2</c> - and <c>end_per_testcase/2</c>). If the timetrap time is + test case is allowed to execute (including <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> + and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c>). If the timetrap time is exceeded, the test case fails with reason - <c>timetrap_timeout</c>. If a <c>TimeFunc</c> function is specified, - it will be called initially and must return a value on - <c>TimeVal</c> format.</p> + <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details). + </p> <p>The <c>require</c> tag specifies configuration variables - that are required by test cases in the suite. If the required - configuration variables are not found in any of the - configuration files, all test cases are skipped. For more + that are required by test cases (and/or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases are skipped. For more information about the 'require' functionality, see the reference manual for the function - <c>ct:require/[1,2]</c>.</p> + <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test suite related information which can be - read by calling <c>ct:userdata/2</c>.</p> + read by calling <c><seealso marker="ct#userdata-2">ct:userdata/2</seealso></c>.</p> <p>The <c>ct_hooks</c> tag specifies which <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> @@ -196,8 +220,9 @@ </func> <func> - <name>Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | {skip_and_save,Reason,SaveConfig}</name> - <fsummary>Test suite initialization.</fsummary> + <name>Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | + {skip_and_save,Reason,SaveConfig}</name> + <fsummary>Test suite initializations.</fsummary> <type> <v> Config = NewConfig = SaveConfig = [{Key,Value}]</v> <v> Key = atom()</v> @@ -208,15 +233,15 @@ <p> OPTIONAL </p> - <p>This function is called as the first function in the - suite. It typically contains initialization which is common for + <p>This configuration function is called as the first function in the + suite. It typically contains initializations which are common for all test cases in the suite, and which shall only be done - once. The <c>Config</c> parameter is the configuration + once. The <c>Config</c> parameter is the configuration data which can be modified here. Whatever is returned from this function is given as <c>Config</c> to all configuration functions and test cases in the suite. If <c>{skip,Reason}</c> - is returned, all test cases in the suite will be skipped and <c>Reason</c> - printed in the overview log for the suite.</p> + is returned, all test cases in the suite will be skipped + and <c>Reason</c> printed in the overview log for the suite.</p> <p>For information on <c>save_config</c> and <c>skip_and_save</c>, please see <seealso marker="dependencies_chapter#save_config">Dependencies @@ -224,29 +249,110 @@ </desc> </func> - <func> - <name>Module:end_per_suite(Config) -> void() | {save_config,SaveConfig}</name> - <fsummary>Test suite finalization. </fsummary> - <type> - <v> Config = SaveConfig = [{Key,Value}]</v> - <v> Key = atom()</v> - <v> Value = term()</v> - </type> + <func> + <name>Module:end_per_suite(Config) -> void() | + {save_config,SaveConfig}</name> + <fsummary>Test suite finalization. </fsummary> + <type> + <v> Config = SaveConfig = [{Key,Value}]</v> + <v> Key = atom()</v> + <v> Value = term()</v> + </type> + + <desc> + <p> OPTIONAL </p> + <p>This function is called as the last test case in the + suite. It is meant to be used for cleaning up after + <c><seealso marker="#Module:init_per_suite-1">init_per_suite/1</seealso></c>. + For information on <c>save_config</c>, please see + <seealso marker="dependencies_chapter#save_config">Dependencies + between Test Cases and Suites</seealso> in the User's Guide.</p> + </desc> + </func> + + <func> + <name>Module:group(GroupName) -> [Info] </name> + <fsummary>Test case group info function (providing default data + for a test case group, i.e. its test cases and sub-groups).</fsummary> + <type> + <v> Info = {timetrap,Time} | {require,Required} | + {require,Name,Required} | {userdata,UserData} | + {silent_connections,Conns} | {stylesheet,CSSFile} | + {ct_hooks, CTHs}</v> + <v> Time = TimeVal | TimeFunc</v> + <v> TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | + {hours,integer()}</v> + <v> TimeFunc = {Mod,Func,Args} | Fun</v> + <v> MilliSec = integer()</v> + <v> Mod = atom()</v> + <v> Func = atom()</v> + <v> Args = list()</v> + <v> Fun = fun()</v> + <v> Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys}</v> + <v> Key = atom()</v> + <v> SubKeys = SubKey | [SubKey]</v> + <v> SubKey = atom()</v> + <v> Name = atom()</v> + <v> UserData = term()</v> + <v> Conns = [atom()]</v> + <v> CSSFile = string()</v> + <v> CTHs = [CTHModule | {CTHModule, CTHInitArgs} | + {CTHModule, CTHInitArgs, CTHPriority}]</v> + <v> CTHModule = atom()</v> + <v> CTHInitArgs = term()</v> + </type> <desc> + <p> OPTIONAL </p> - <p>This function is called as the last test case in the - suite. It is meant to be used for cleaning up after <c>init_per_suite/1</c>. - For information on <c>save_config</c>, please see - <seealso marker="dependencies_chapter#save_config">Dependencies between - Test Cases and Suites</seealso> in the User's Guide.</p> - </desc> + <p>This is the test case group info function. It is supposed to + return a list of tagged tuples that specify various properties + related to the execution of a test case group (i.e. its test cases + and sub-groups). Properties set by + <c><seealso marker="#Module:group-1">group/1</seealso></c> override + properties with the same key that have been previously set by + <c><seealso marker="#Module:suite-0">suite/0</seealso></c>.</p> + + <p>The <c>timetrap</c> tag sets the maximum time each + test case is allowed to execute (including <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> + and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c>). If the timetrap time is + exceeded, the test case fails with reason + <c>timetrap_timeout</c>. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details).</p> + + <p>The <c>require</c> tag specifies configuration variables + that are required by test cases (and/or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases in this group are skipped. + For more information about the 'require' functionality, see the + reference manual for the function + <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> + + <p>With <c>userdata</c>, it is possible for the user to + specify arbitrary test case group related information which can be + read by calling <c><seealso marker="ct#userdata-2">ct:userdata/2</seealso></c>.</p> + + <p>The <c>ct_hooks</c> tag specifies which + <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> + are to be run together with this suite.</p> + + <p>Other tuples than the ones defined will simply be ignored.</p> + + <p>For more information about the test case group info function, + see <seealso marker="write_test_chapter#suite">Test + case group info function</seealso> in the User's Guide.</p> + </desc> </func> <func> - <name>Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason}</name> - <fsummary>Test case group initialization.</fsummary> + <name>Module:init_per_group(GroupName, Config) -> NewConfig | + {skip,Reason}</name> + <fsummary>Test case group initializations.</fsummary> <type> <v> GroupName = atom()</v> <v> Config = NewConfig = [{Key,Value}]</v> @@ -258,16 +364,16 @@ <p> OPTIONAL </p> - <p>This function is called before execution of a test case group. - It typically contains initialization which is common for - all test cases in the group, and which shall only be performed - once. <c>GroupName</c> is the name of the group, as specified in - the group definition (see <c>groups/0</c>). The <c>Config</c> - parameter is the configuration which can be modified here. - Whatever is returned from this function is given as <c>Config</c> - to all test cases in the group. If <c>{skip,Reason}</c> is returned, - all test cases in the group will be skipped and <c>Reason</c> printed - in the overview log for the group.</p> + <p>This configuration function is called before execution of a + test case group. It typically contains initializations which are + common for all test cases and sub-groups in the group, and which + shall only be performed once. <c>GroupName</c> is the name of the + group, as specified in the group definition (see <c><seealso marker="#Module:groups-0">groups/0</seealso></c>). The + <c>Config</c> parameter is the configuration data which can be modified + here. The return value of this function is given as <c>Config</c> + to all test cases and sub-groups in the group. If <c>{skip,Reason}</c> + is returned, all test cases in the group will be skipped and + <c>Reason</c> printed in the overview log for the group.</p> <p>For information about test case groups, please see <seealso marker="write_test_chapter#test_case_groups">Test case @@ -276,7 +382,8 @@ </func> <func> - <name>Module:end_per_group(GroupName, Config) -> void() | {return_group_result,Status}</name> + <name>Module:end_per_group(GroupName, Config) -> void() | + {return_group_result,Status}</name> <fsummary>Test case group finalization.</fsummary> <type> <v> GroupName = atom()</v> @@ -290,10 +397,10 @@ <p> OPTIONAL </p> <p>This function is called after the execution of a test case group is finished. - It is meant to be used for cleaning up after <c>init_per_group/2</c>. + It is meant to be used for cleaning up after <c><seealso marker="#Module:init_per_group-2">init_per_group/2</seealso></c>. By means of <c>{return_group_result,Status}</c>, it is possible to return a status value for a nested sub-group. The status can be retrieved in - <c>end_per_group/2</c> for the group on the level above. The status will also + <c><seealso marker="#Module:end_per_group-2">end_per_group/2</seealso></c> for the group on the level above. The status will also be used by Common Test for deciding if execution of a group should proceed in case the property <c>sequence</c> or <c>repeat_until_*</c> is set.</p> @@ -305,7 +412,7 @@ <func> <name>Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason} | {skip,Reason}</name> - <fsummary>Test case initialization.</fsummary> + <fsummary>Test case initializations.</fsummary> <type> <v> TestCase = atom()</v> <v> Config = NewConfig = [{Key,Value}]</v> @@ -344,7 +451,7 @@ <p> OPTIONAL </p> <p> This function is called after each test case, and can be used - to clean up after <c>init_per_testcase/2</c> and the test case. + to clean up after <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> and the test case. Any return value (besides <c>{fail,Reason}</c> and <c>{save_config,SaveConfig}</c>) is ignored. By returning <c>{fail,Reason}</c>, <c>TestCase</c> will be marked as failed (even though it was actually successful in the sense that it returned @@ -370,7 +477,7 @@ <v> Func = atom()</v> <v> Args = list()</v> <v> Fun = fun()</v> - <v> Required = Key | {Key,SubKeys}</v> + <v> Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys}</v> <v> Key = atom()</v> <v> SubKeys = SubKey | [SubKey]</v> <v> SubKey = atom()</v> @@ -385,32 +492,38 @@ <p>This is the test case info function. It is supposed to return a list of tagged tuples that specify various properties - regarding the execution of this particular test case.</p> + related to the execution of this particular test case. + Properties set by <c><seealso marker="#Module:Testcase-0">Testcase/0</seealso></c> override + properties that have been previously set for the test case + by <c><seealso marker="#Module:group-1">group/1</seealso></c> or <c><seealso marker="#Module:suite-0">suite/0</seealso></c>.</p> <p>The <c>timetrap</c> tag sets the maximum time the - test case is allowed to take. If the timetrap time is + test case is allowed to execute. If the timetrap time is exceeded, the test case fails with reason - <c>timetrap_timeout</c>. <c>init_per_testcase/2</c> - and <c>end_per_testcase/2</c> are included in the - timetrap time. If a <c>TimeFunc</c> function is specified, - it will be called before the test case (or <c>init_per_testcase/2</c>) - and must return a value on <c>TimeVal</c> format.</p> + <c>timetrap_timeout</c>. <c><seealso marker="#Module:init_per_testcase-2">init_per_testcase/2</seealso></c> + and <c><seealso marker="#Module:end_per_testcase-2">end_per_testcase/2</seealso></c> are included in the + timetrap time. A <c>TimeFunc</c> function can be used to + set a new timetrap by returning a <c>TimeVal</c>. It may also be + used to trigger a timetrap timeout by, at some point, returning a + value other than a <c>TimeVal</c>. (See the + <seealso marker="write_test_chapter#timetraps">User's Guide</seealso> + for details).</p> <p>The <c>require</c> tag specifies configuration variables - that are required by the test case. If the required - configuration variables are not found in any of the + that are required by the test case (and/or <c>init/end_per_testcase/2</c>). + If the required configuration variables are not found in any of the configuration files, the test case is skipped. For more information about the 'require' functionality, see the reference manual for the function - <c>ct:require/[1,2]</c>.</p> + <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c>.</p> <p>If <c>timetrap</c> and/or <c>require</c> is not set, the - default values specified in the <c>suite/0</c> return list - will be used.</p> + default values specified by <c><seealso marker="#Module:suite-0">suite/0</seealso></c> (or + <c><seealso marker="#Module:group-1">group/1</seealso></c>) will be used.</p> <p>With <c>userdata</c>, it is possible for the user to specify arbitrary test case related information which can be - read by calling <c>ct:userdata/3</c>.</p> + read by calling <c><seealso marker="ct#userdata-3">ct:userdata/3</seealso></c>.</p> <p>Other tuples than the ones defined will simply be ignored.</p> @@ -438,12 +551,12 @@ <p>This is the implementation of a test case. Here you must call the functions you want to test, and do whatever you need to check the result. If something fails, make sure the - function causes a runtime error, or call <c>ct:fail/[0,1]</c> - (which also causes the test case process to crash).</p> + function causes a runtime error, or call <c><seealso marker="ct#fail-1">ct:fail/1/2</seealso></c> + (which also causes the test case process to terminate).</p> - <p>Elements from the <c>Config</c> parameter can be read - with the <c>?config</c> macro. The <c>config</c> - macro is defined in <c>ct.hrl</c></p> + <p>Elements from the <c>Config</c> list can e.g. be read + with <c>proplists:get_value/2</c> (or the macro <c>?config</c> + defined in <c>ct.hrl</c>).</p> <p>You can return <c>{skip,Reason}</c> if you decide not to run the test case after all. <c>Reason</c> will then be diff --git a/lib/common_test/doc/src/config.gif b/lib/common_test/doc/src/config.gif Binary files differnew file mode 100644 index 0000000000..ac8006c4fb --- /dev/null +++ b/lib/common_test/doc/src/config.gif diff --git a/lib/common_test/doc/src/config_file_chapter.xml b/lib/common_test/doc/src/config_file_chapter.xml index 6fc6638bf7..d90adf8d7b 100644 --- a/lib/common_test/doc/src/config_file_chapter.xml +++ b/lib/common_test/doc/src/config_file_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2004</year><year>2010</year> + <year>2004</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -29,6 +29,8 @@ <file>config_file_chapter.xml</file> </header> + <marker id="top"></marker> + <section> <title>General</title> @@ -78,7 +80,7 @@ test is skipped (unless a default value has been specified, see the <seealso marker="write_test_chapter#info_function">test case info function</seealso> chapter for details). There is also a function - <c>ct:require/[1,2]</c> which can be called from a test case + <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> which can be called from a test case in order to check if a specific variable is available. The return value from this function must be checked explicitly and appropriate action be taken depending on the result (e.g. to skip the test case @@ -88,7 +90,7 @@ info-list should look like this: <c>{require,CfgVarName}</c> or <c>{require,AliasName,CfgVarName}</c>. The arguments <c>AliasName</c> and <c>CfgVarName</c> are the same as the - arguments to <c>ct:require/[1,2]</c> which are described in the + arguments to <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> which are described in the reference manual for <seealso marker="ct">ct</seealso>. <c>AliasName</c> becomes an alias for the configuration variable, and can be used as reference to the configuration data value. @@ -101,7 +103,8 @@ (or test case) and improve readability.</item> </list> <p>To read the value of a config variable, use the function - <c>get_config/[1,2,3]</c> which is also described in the reference + <c><seealso marker="ct#get_config-1">get_config/1/2/3</seealso></c> + which is also described in the reference manual for <seealso marker="ct">ct</seealso>.</p> <p>Example:</p> <pre> @@ -118,7 +121,7 @@ <section> <title>Using configuration variables defined in multiple files</title> <p>If a configuration variable is defined in multiple files and you - want to access all possible values, you may use the <c>ct:get_config/3</c> + want to access all possible values, you may use the <c><seealso marker="ct#get_config-3">ct:get_config/3</seealso></c> function and specify <c>all</c> in the options list. The values will then be returned in a list and the order of the elements corresponds to the order that the config files were specified at startup. Please see @@ -130,7 +133,7 @@ <marker id="encrypted_config_files"></marker> <p>It is possible to encrypt configuration files containing sensitive data if these files must be stored in open and shared directories.</p> - <p>Call <c>ct:encrypt_config_file/[2,3]</c> to have Common Test encrypt a + <p>Call <c><seealso marker="ct#encrypt_config_file-2">ct:encrypt_config_file/2/3</seealso></c> to have Common Test encrypt a specified file using the DES3 function in the OTP <c>crypto</c> application. The encrypted file can then be used as a regular configuration file, in combination with other encrypted files or normal text files. The key @@ -139,7 +142,7 @@ <c>decrypt_file</c> flag/option, or a key file in a predefined location.</p> <p>Common Test also provides decryption functions, - <c>ct:decrypt_config_file/[2,3]</c>, for recreating the original text + <c><seealso marker="ct#decrypt_config_file-2">ct:decrypt_config_file/2/3</seealso></c>, for recreating the original text files.</p> <p>Please see the <seealso marker="ct">ct</seealso> reference manual for @@ -149,8 +152,8 @@ <section> <title>Opening connections by using configuration data</title> <p>There are two different methods for opening a connection - by means of the support functions in e.g. <c>ct_ssh</c>, <c>ct_ftp</c>, - and <c>ct_telnet</c>:</p> + by means of the support functions in e.g. <c><seealso marker="ct_ssh">ct_ssh</seealso></c>, <c><seealso marker="ct_ftp">ct_ftp</seealso></c>, + and <c><seealso marker="ct_telnet">ct_telnet</seealso></c>:</p> <list> <item>Using a configuration target name (an alias) as reference.</item> <item>Using the configuration variable as reference.</item> @@ -295,7 +298,7 @@ <pre> [{ftp_host, [{ftp, "targethost"}, {username, "tester"}, {password, "letmein"}]}, - {lm_directory, "/test/loadmodules"}]</pre> + {lm_directory, "/test/loadmodules"}]</pre> </section> diff --git a/lib/common_test/doc/src/cover_chapter.xml b/lib/common_test/doc/src/cover_chapter.xml index b7162cb542..803a71de07 100644 --- a/lib/common_test/doc/src/cover_chapter.xml +++ b/lib/common_test/doc/src/cover_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2006</year><year>2011</year> + <year>2006</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -100,7 +100,7 @@ <p><c>$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec</c></p> <p>You may also pass the cover specification file name in a - call to <c>ct:run_test/1</c>, by adding a <c>{cover,CoverSpec}</c> + call to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, by adding a <c>{cover,CoverSpec}</c> tuple to the <c>Opts</c> argument. Also, you can of course enable code coverage in your test specifications (read more in the chapter about diff --git a/lib/common_test/doc/src/ct_hooks.xml b/lib/common_test/doc/src/ct_hooks.xml index f9fc1858d0..b3e713c77f 100644 --- a/lib/common_test/doc/src/ct_hooks.xml +++ b/lib/common_test/doc/src/ct_hooks.xml @@ -5,7 +5,7 @@ <erlref> <header> <copyright> - <year>2010</year><year>2011</year> + <year>2010</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -37,12 +37,6 @@ <description> - <warning><p>This feature is in alpha release right now. This means that the - interface may change in the future and that there may be bugs. We - encourage you to use this feature, but be prepared - that there might be bugs and that the interface might change - inbetween releases.</p></warning> - <p>The <em>Common Test Hook</em> (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of callbacks before and after all test suite calls. It is meant for advanced users of @@ -117,11 +111,12 @@ </func> <func> - <name>Module:pre_init_per_suite(SuiteName, Config, CTHState) -> + <name>Module:pre_init_per_suite(SuiteName, InitData, CTHState) -> Result</name> <fsummary>Called before init_per_suite</fsummary> <type> <v>SuiteName = atom()</v> + <v>InitData = Config | SkipOrFail</v> <v>Config = NewConfig = [{Key,Value}]</v> <v>CTHState = NewCTHState = term()</v> <v>Result = {Return, NewCTHState}</v> @@ -146,7 +141,8 @@ <p><c>SuiteName</c> is the name of the suite to be run.</p> - <p><c>Config</c> is the original config list of the test suite.</p> + <p><c>InitData</c> is the original config list of the test suite, or + a <c>SkipOrFail</c> tuple if a previous CTH has returned this.</p> <p><c>CTHState</c> is the current internal state of the CTH.</p> @@ -218,11 +214,12 @@ </func> <func> - <name>Module:pre_init_per_group(GroupName, Config, CTHState) -> + <name>Module:pre_init_per_group(GroupName, InitData, CTHState) -> Result</name> <fsummary>Called before init_per_group</fsummary> <type> <v>GroupName = atom()</v> + <v>InitData = Config | SkipOrFail</v> <v>Config = NewConfig = [{Key,Value}]</v> <v>CTHState = NewCTHState = term()</v> <v>Result = {NewConfig | SkipOrFail, NewCTHState}</v> @@ -275,11 +272,12 @@ </func> <func> - <name>Module:pre_init_per_testcase(TestcaseName, Config, CTHState) -> + <name>Module:pre_init_per_testcase(TestcaseName, InitData, CTHState) -> Result</name> <fsummary>Called before init_per_testcase</fsummary> <type> <v>TestcaseName = atom()</v> + <v>InitData = Config | SkipOrFail</v> <v>Config = NewConfig = [{Key,Value}]</v> <v>CTHState = NewCTHState = term()</v> <v>Result = {NewConfig | SkipOrFail, NewCTHState}</v> @@ -336,11 +334,12 @@ </func> <func> - <name>Module:pre_end_per_group(GroupName, Config, CTHState) -> + <name>Module:pre_end_per_group(GroupName, EndData, CTHState) -> Result</name> <fsummary>Called before end_per_group</fsummary> <type> <v>GroupName = atom()</v> + <v>EndData = Config | SkipOrFail</v> <v>Config = NewConfig = [{Key,Value}]</v> <v>CTHState = NewCTHState = term()</v> <v>Result = {NewConfig | SkipOrFail, NewCTHState}</v> @@ -393,11 +392,12 @@ </func> <func> - <name>Module:pre_end_per_suite(SuiteName, Config, CTHState) -> + <name>Module:pre_end_per_suite(SuiteName, EndData, CTHState) -> Result</name> <fsummary>Called before end_per_suite</fsummary> <type> <v>SuiteName = atom()</v> + <v>EndData = Config | SkipOrFail</v> <v>Config = NewConfig = [{Key,Value}]</v> <v>CTHState = NewCTHState = term()</v> <v>Result = {NewConfig | SkipOrFail, NewCTHState}</v> diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml index dbb4310040..27d56fd47d 100644 --- a/lib/common_test/doc/src/ct_hooks_chapter.xml +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2011</year><year>2011</year> + <year>2011</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -32,11 +32,6 @@ <marker id="general"></marker> <section> <title>General</title> - <warning><p>This feature is in alpha release right now. This means that the - interface may change in the future and that there may be bugs. We - encourage you to use this feature, but be prepared - that there might be bugs and that the interface might change - inbetween releases.</p></warning> <p> The <em>Common Test Hook</em> (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of hooks @@ -113,9 +108,10 @@ </section> <section> - <title>CTH Priority</title> + <title>CTH Execution order</title> <p>By default each CTH installed will be executed in the order which - they are installed. This is not always wanted so common_test allows + they are installed for init calls, and then reversed for end calls. + This is not always wanted so common_test allows the user to specify a priority for each hook. The priority can either be specified in the CTH <seealso marker="ct_hooks#Module:init-2">init/2 </seealso> function or when installing the hook. The priority given at @@ -193,6 +189,22 @@ it.</p> </section> + <section> + <title>External configuration data and Logging</title> + <p>It's possible in the CTH to read configuration data values + by calling <c><seealso marker="ct#get_config-1">ct:get_config/1/2/3</seealso></c> (as explained in the + <seealso marker="config_file_chapter#require_config_data"> + External configuration data</seealso> + chapter). The config variables in question must, as always, first have been + <c>required</c> by means of a suite-, group-, or test case info function, + or the <c><seealso marker="ct#require-1">ct:require/1/2</seealso></c> function. Note that the latter can also be used + in CT hook functions.</p> + <p>The CT hook functions may call any of the logging functions available + in the <c>ct</c> interface to print information to the log files, or to + add comments in the suite overview page. + </p> + </section> + </section> <marker id="manipulating"/> @@ -205,11 +217,13 @@ functions for a CTH follow a common interface, this interface is described below.</p> - <p>It is only possible to hook into test function which exists in the test - suite. So in order for a CTH to hook in before - <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso>, - the <seealso marker="common_test#Module:init_per_suite-1">init_per_suite</seealso> - function must exist in the test suite.</p> + <p>Common Test will always call all available hook functions, even pre- and post + hooks for configuration functions that are not implemented in the suite. + For example, <c>pre_init_per_suite(x_SUITE, ...)</c> and + <c>post_init_per_suite(x_SUITE, ...)</c> will be called for test suite + <c>x_SUITE</c>, even if it doesn't export <c>init_per_suite/1</c>. This feature + makes it possible to use hooks as configuration fallbacks, or even + completely replace all configuration functions with hook functions.</p> <marker id="pre"/> <section> @@ -238,6 +252,13 @@ {ok, Handle} -> {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }} end.</code> + <note>If using multiple CTHs, the first part of the return tuple will be + used as input for the next CTH. So in the case above the next CTH might + get <c>{fail,Reason}</c> as the second parameter. If you have many CTHs + which interact, it might be a good idea to not let each CTH return + <c>fail</c> or <c>skip</c>. Instead return that an action should be taken + through the <c>Config</c> list and implement a CTH which at the end takes + the correct action. </note> </section> @@ -405,6 +426,62 @@ terminate(State) -> ok.</code> </section> + <marker id="builtin_cths"/> + <section> + <title>Built-in CTHs</title> + <p>Common Test is delivered with a couple of general purpose CTHs that + can be enabled by the user to provide some generic testing functionality. + Some of these are enabled by default when starting running common_test, + they can be disabled by setting <c>enable_builtin_hooks</c> to + <c>false</c> on the command line or in the test specification. In the + table below there is a list of all current CTHs which are delivered with + Common Test.</p> + + <table> + <row> + <cell><em>CTH Name</em></cell> + <cell><em>Is Built-in</em></cell> + <cell><em>Description</em></cell> + </row> + <row> + <cell>cth_log_redirect</cell> + <cell>yes</cell> + <cell>Captures all error_logger and SASL logging events and prints them + to the current test case log. If an event can not be associated with a + testcase it will be printed in the common test framework log. This will + happen for testcases which are run in parallel and events which occur + inbetween testcases. You can configure the level of + <seealso marker="sasl:sasl_app">SASL</seealso> events report + using the normal SASL mechanisms. </cell> + </row> + <row> + <cell>cth_surefire</cell> + <cell>no</cell> + <cell><p>Captures all test results and outputs them as surefire + XML into a file. The file which is created is by default + called junit_report.xml. The file name can be changed by + setting the <c>path</c> option for this hook, e.g.</p> + + <code>-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]</code> + + <p>If the <c>url_base</c> option is set, an additional + attribute named <c>url</c> will be added to each + <c>testsuite</c> and <c>testcase</c> XML element. The value will + be a constructed from the <c>url_base</c> and a relative path + to the test suite or test case log respectively, e.g.</p> + + <code>-ct_hooks cth_surefire [{url_base,"http://myserver.com/"}]</code> + <p>will give a url attribute value similar to</p> + + <code>"http://myserver.com/[email protected]_11.19.39/x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"</code> + + <p>Surefire XML can for instance be used by Jenkins to display test + results.</p></cell> + </row> + </table> + + </section> + </chapter> diff --git a/lib/common_test/doc/src/ct_master_chapter.xml b/lib/common_test/doc/src/ct_master_chapter.xml index f4f0ecad62..9e848e99bb 100644 --- a/lib/common_test/doc/src/ct_master_chapter.xml +++ b/lib/common_test/doc/src/ct_master_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2006</year><year>2010</year> + <year>2006</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -124,7 +124,8 @@ <p><c>NodeRef = NodeAlias | node() | master</c></p> <p>A <c>NodeAlias</c> (<c>atom()</c>) is used in a test specification as a - reference to a node name (so the actual node name only needs to be declared once). + reference to a node name (so the actual node name only needs to be declared once, + which can of course also be achieved using constants). The alias is declared with a <c>node</c> term:</p> <p><c>{node, NodeAlias, NodeName}</c></p> @@ -141,30 +142,32 @@ CT Master:</p> <pre> - {node, node1, ct_node@host_x}. - {node, node2, ct_node@host_y}. - - {logdir, master, "/home/test/master_logs"}. - {logdir, "/home/test/logs"}. + {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + {define, 'Node', ct_node}. + + {node, node1, 'Node@host_x'}. + {node, node2, 'Node@host_y'}. + + {logdir, master, "'Top'/master_logs"}. + {logdir, "'Top'/logs"}. - {config, node1, "/home/test/t1/cfg/config.cfg"}. - {config, node2, "/home/test/t2/cfg/config.cfg"}. - {config, "/home/test/t3/cfg/config.cfg"}. + {config, node1, "'T1'/'CfgFile'"}. + {config, node2, "'T2'/'CfgFile'"}. + {config, "'T3'/'CfgFile'"}. - {alias, t1, "/home/test/t1"}. - {alias, t2, "/home/test/t2"}. - {alias, t3, "/home/test/t3"}. + {suites, node1, 'T1', all}. + {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}. - {suites, node1, t1, all}. - {skip_suites, node1, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}. - {skip_cases, node1, t1, t1A_SUITE, [test3,test4], "Irrelevant"}. - {skip_cases, node1, t1, t1C_SUITE, [test1], "Ignore"}. + {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}. - {suites, node2, t2, [t2B_SUITE,t2C_SUITE]}. - {cases, node2, t2, t2A_SUITE, [test4,test1,test7]}. - - {skip_suites, t3, all, "Not implemented"}. - </pre> + {skip_suites, 'T3', all, "Not implemented"}.</pre> <p>This example specifies the same tests as the original example. But now if started with a call to <c>ct_master:run(TestSpecName)</c>, the @@ -190,10 +193,6 @@ name as the Common Test node in question (typically <c>ct@somehost</c> if started with the <c>ct_run</c> program), will be performed. Tests without explicit node association will always be performed too of course!</p> - - <note><p>It is recommended that absolute paths are used for log directories, - config files and test directory aliases in the test specifications so that - current working directory settings are not important.</p></note> </section> <section> diff --git a/lib/common_test/doc/src/ct_run.xml b/lib/common_test/doc/src/ct_run.xml index 9045646733..c6749d6960 100644 --- a/lib/common_test/doc/src/ct_run.xml +++ b/lib/common_test/doc/src/ct_run.xml @@ -4,7 +4,7 @@ <comref> <header> <copyright> - <year>2007</year><year>2010</year> + <year>2007</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -36,6 +36,8 @@ OS command line. </comsummary> + <marker id="top"></marker> + <description> <p>The <c>ct_run</c> program is automatically installed with Erlang/OTP and Common Test (please see the Installation chapter in the Common @@ -46,7 +48,7 @@ particular mode.</p> <p>There is an interface function that corresponds to this program, - called <c>ct:run_test/1</c>, for starting Common Test from the Erlang + called <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, for starting Common Test from the Erlang shell (or an Erlang program). Please see the <c>ct</c> man page for details.</p> @@ -72,6 +74,10 @@ following <c>-erl_args</c> on the command line. These directories are added to the code path normally (i.e. on specified form)</p> + <p>Exit status is set before the program ends. Value <c>0</c> indicates a successful + test result, <c>1</c> indicates one or more failed or auto-skipped test cases, and + <c>2</c> indicates test execution failure.</p> + <p>If <c>ct_run</c> is called with option:</p> <pre>-help</pre> <p>it prints all valid start flags to stdout.</p> @@ -84,15 +90,17 @@ <pre> ct_run [-dir TestDir1 TestDir2 .. TestDirN] | [[-dir TestDir] -suite Suite1 Suite2 .. SuiteN - [[-group Group1 Group2 .. GroupN] [-case Case1 Case2 .. CaseN]]] + [[-group Groups1 Groups2 .. GroupsN] [-case Case1 Case2 .. CaseN]]] [-step [config | keep_inactive]] [-config ConfigFile1 ConfigFile2 .. ConfigFileN] [-userconfig CallbackModule1 ConfigString1 and CallbackModule2 - ConfigString2 and .. and CallbackModuleN ConfigStringN] + ConfigString2 and .. CallbackModuleN ConfigStringN] [-decrypt_key Key] | [-decrypt_file KeyFile] [-label Label] [-logdir LogDir] [-logopts LogOpts] + [-verbosity GenVLevel | [Category1 VLevel1 and + Category2 VLevel2 and .. CategoryN VLevelN]] [-silent_connections [ConnType1 ConnType2 .. ConnTypeN]] [-stylesheet CSSFile] [-cover CoverCfgFile] @@ -103,10 +111,15 @@ [-no_auto_compile] [-muliply_timetraps Multiplier] [-scale_timetraps] + [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] [-repeat N [-force_stop]] | [-duration HHMMSS [-force_stop]] | [-until [YYMoMoDD]HHMMSS [-force_stop]] - [-basic_html]</pre> + [-basic_html] + [-ct_hooks CTHModule1 CTHOpts1 and CTHModule2 CTHOpts2 and .. + CTHModuleN CTHOptsN] + [-exit_status ignore_config] + </pre> </section> <section> <title>Run tests using test specification</title> @@ -119,6 +132,8 @@ [-label Label] [-logdir LogDir] [-logopts LogOpts] + [-verbosity GenVLevel | [Category1 VLevel1 and + Category2 VLevel2 and .. CategoryN VLevelN]] [-allow_user_terms] [-silent_connections [ConnType1 ConnType2 .. ConnTypeN]] [-stylesheet CSSFile] @@ -130,10 +145,15 @@ [-no_auto_compile] [-muliply_timetraps Multiplier] [-scale_timetraps] + [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] [-repeat N [-force_stop]] | [-duration HHMMSS [-force_stop]] | [-until [YYMoMoDD]HHMMSS [-force_stop]] - [-basic_html]</pre> + [-basic_html] + [-ct_hooks CTHModule1 CTHOpts1 and CTHModule2 CTHOpts2 and .. + CTHModuleN CTHOptsN] + [-exit_status ignore_config] + </pre> </section> <section> <title>Run tests in web based GUI</title> @@ -145,11 +165,14 @@ [-userconfig CallbackModule1 ConfigString1 and CallbackModule2 ConfigString2 and .. and CallbackModuleN ConfigStringN] [-logopts LogOpts] + [-verbosity GenVLevel | [Category1 VLevel1 and + Category2 VLevel2 and .. CategoryN VLevelN]] [-decrypt_key Key] | [-decrypt_file KeyFile] [-include InclDir1 InclDir2 .. InclDirN] [-no_auto_compile] [-muliply_timetraps Multiplier] [-scale_timetraps] + [-create_priv_dir auto_per_run | auto_per_tc | manual_per_tc] [-basic_html]</pre> </section> <section> diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml index b41b233ce6..10a9b52d39 100644 --- a/lib/common_test/doc/src/event_handler_chapter.xml +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2006</year><year>2011</year> + <year>2006</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -64,7 +64,7 @@ <marker id="usage"></marker> <title>Usage</title> <p>Event handlers may be installed by means of an <c>event_handler</c> - start flag (<c>ct_run</c>) or option (<c>ct:run_test/1</c>), where the + start flag (<c>ct_run</c>) or option (<c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), where the argument specifies the names of one or more event handler modules. Example:</p> <p><c>$ ct_run -suite test/my_SUITE -event_handler handlers/my_evh1 @@ -78,7 +78,7 @@ example).</p> <p>An event_handler tuple in the argument <c>Opts</c> has the following - definition (see also <c>ct:run_test/1</c> in the reference manual):</p> + definition (see also <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> in the reference manual):</p> <pre> {event_handler,EventHandlers} @@ -174,6 +174,16 @@ are also given. </p></item> + <item><c>#event{name = tc_logfile, data = {{Suite,Func},LogFileName}}</c> + <p><c>Suite = atom()</c>, name of the test suite.</p> + <p><c>Func = atom()</c>, name of test case or configuration function.</p> + <p><c>LogFileName = string()</c>, full name of test case log file.</p> + <p>This event is sent at the start of each test case (and configuration function + except <c>init/end_per_testcase</c>) and carries information about the + full name (i.e. the file name including the absolute directory path) of + the current test case log file. + </p></item> + <marker id="tc_done"/> <item><c>#event{name = tc_done, data = {Suite,FuncOrGroup,Result}}</c> <p><c>Suite = atom()</c>, name of the suite.</p> @@ -195,7 +205,7 @@ {error,{RunTimeError,StackTrace}} | {timetrap_timeout,integer()} | {failed,{Suite,end_per_testcase,FailInfo}}</c>, reason for failure.</p> - <p><c>RequireInfo = {not_available,atom()}</c>, why require has failed.</p> + <p><c>RequireInfo = {not_available,atom() | tuple()}</c>, why require has failed.</p> <p><c>FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | UserTerm</c>, @@ -223,7 +233,7 @@ reason for auto skipping <c>Func</c>.</p> <p><c>FailReason = {Suite,ConfigFunc,FailInfo}} | {Suite,FailedCaseInSequence}</c>, reason for failure.</p> - <p><c>RequireInfo = {not_available,atom()}</c>, why require has failed.</p> + <p><c>RequireInfo = {not_available,atom() | tuple()}</c>, why require has failed.</p> <p><c>ConfigFunc = init_per_suite | init_per_group</c></p> <p><c>FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | @@ -298,7 +308,7 @@ manager can look like.</p> <note><p>To ensure that printouts to standard out (or printouts made with - <c>ct:log/2/3</c> or <c>ct:pal/2/3</c>) get written to the test case log + <c><seealso marker="ct#log-2">ct:log/2/3</seealso></c> or <c><seealso marker="ct:pal-2">ct:pal/2/3</seealso></c>) get written to the test case log file, and not to the Common Test framework log, you can syncronize with the Common Test server by matching on the <c>tc_start</c> and <c>tc_done</c> events. In the period between these events, all IO gets directed to the diff --git a/lib/common_test/doc/src/example_chapter.xml b/lib/common_test/doc/src/example_chapter.xml index f269dba2cd..2333f92989 100644 --- a/lib/common_test/doc/src/example_chapter.xml +++ b/lib/common_test/doc/src/example_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2010</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,7 +28,8 @@ <rev></rev> <file>example_chapter.xml</file> </header> - + + <marker id="top"></marker> <section> <title>Test suite example</title> diff --git a/lib/common_test/doc/src/filestruct.gif b/lib/common_test/doc/src/filestruct.gif Binary files differindex 2b06833d0b..2b06833d0b 100755..100644 --- a/lib/common_test/doc/src/filestruct.gif +++ b/lib/common_test/doc/src/filestruct.gif diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml new file mode 100644 index 0000000000..3cf04bb1a2 --- /dev/null +++ b/lib/common_test/doc/src/getting_started_chapter.xml @@ -0,0 +1,276 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2007</year><year>2012</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Getting Started</title> + <prepared>Peter Andersson</prepared> + <docno></docno> + <date>2011-12-12</date> + <rev></rev> + <file>getting_started_chapter.xml</file> + </header> + + <section> + <title>Are you new around here?</title> + <p> + The purpose of this short chapter is to, with a "learning by example" + approach, give the newcomer a chance to get started quickly writing and + executing some first simple tests. The chapter will introduce some of the + basics, but leave most explanations and details for the later + chapters in this User's Guide. Hopefully though, after this chapter, you + will be inspired and unintimidated enough to go on and get into the + nitty-gritty that follows in this rather heavy User's Guide! If you're + not much into "learning by example" and prefer to get into more technical + detail right away, go ahead and skip to the next chapter. Again, the basics + presented here will be covered in detail in later chapters. + </p> + <p> + This chapter also tries to demonstrate how dead simple it actually is + to write a very basic (yet for many module testing purposes, often sufficiently + complex) test suite, and execute its test cases. This is not necessarily + obvious when you read the rest of the chapters in the User's Guide. + </p> + <p> + A quick note before we start: In order to understand what's discussed and + examplified here, it is recommended that you first read through the + opening <seealso marker="basics_chapter#basics">Common Test Basics</seealso> + chapter. + </p> + </section> + + <section> + <title>Test case execution</title> + <p>Execution of test cases is handled this way:</p> + + <p> + <image file="tc_execution.gif"> + <icaption> + Successful vs unsuccessful test case execution. + </icaption> + </image> + </p> + + <p>For each test case that Common Test is told to execute, it spawns a + dedicated process on which the test case function in question starts + running. (In parallel to the test case process, an idle waiting timer + process is started which is linked to the test case process. If the timer + process runs out of waiting time, it sends an exit signal to terminate + the test case process and this is what's called a <em>timetrap</em>). + </p> + <p>In scenario 1, the test case process terminates normally after case A has + finished executing its test code without detecting any errors. The test + case function simply returns a value and Common Test logs the test case as + successful. + </p> + <p>In scenario 2, an error is detected during test case execution + which causes the test case B function to generate an exception. + This causes the test case process to exit with reason + other than normal, and as a result, Common Test will log this as an + unsuccessful test case. + </p> + <p>As you can understand from the illustration above, Common Test requires + that a test case generates a runtime error to indicate failure (e.g. + by causing a bad match error or by calling <c>exit/1</c>, preferrably + through the <c><seealso marker="ct#fail-1">ct:fail/1,2</seealso></c> help function). A succesful execution is + indicated by means of a normal return from the test case function. + </p> + </section> + + <section> + <title>A simple test suite</title> + <p>As you've seen in the basics chapter, the test suite module implements + <seealso marker="common_test">callback functions</seealso> + (mandatory or optional) for various purposes, e.g: + <list> + <item>Init/end configuration function for the test suite</item> + <item>Init/end configuration function for a test case</item> + <item>Init/end configuration function for a test case group</item> + <item>Test cases</item> + </list> + The configuration functions are optional and if you don't need them for + your test, a test suite with one simple test case could look like this: + </p> + <pre> + -module(my1st_SUITE). + -compile(export_all). + + all() -> + [mod_exists]. + + mod_exists(_) -> + {module,mymod} = code:load_file(mymod).</pre> + <p> + In this example we check that the <c>mymod</c> module exists (i.e. can be + successfully loaded by the code server). If the operation fails, we will + get a bad match error which terminates the test case. + </p> + </section> + + <section> + <title>A test suite with configuration functions</title> + <p> + If we need to perform configuration operations in order to run our test, we + implement configuration functions in our suite. The result from a + configuration function is configuration data, or simply <em><c>Config</c></em>. + This is a list of key-value tuples which get passed from the configuration + function to the test cases (possibly through configuration functions on + "lower level"). The data flow looks like this: + </p> + + <p> + <image file="config.gif"> + <icaption> + Config data flow in the suite. + </icaption> + </image> + </p> + + <p> + Here's an example of a test suite which uses configuration functions + to open and close a log file for the test cases (an operation that would + be unnecessary and irrelevant to perform by each test case): + </p> + <pre> + -module(check_log_SUITE). + -export([all/0, init_per_suite/1, end_per_suite/1]). + -export([check_restart_result/1, check_no_errors/1]). + + -define(value(Key,Config), proplists:get_value(Key,Config)). + + all() -> [check_restart_result, check_no_errors]. + + init_per_suite(InitConfigData) -> + [{logref,open_log()} | InitConfigData]. + + end_per_suite(ConfigData) -> + close_log(?value(logref, ConfigData)). + + check_restart_result(ConfigData) -> + TestData = read_log(restart, ?value(logref, ConfigData)), + {match,_Line} = search_for("restart successful", TestData). + + check_no_errors(ConfigData) -> + TestData = read_log(all, ?value(logref, ConfigData)), + case search_for("error", TestData) of + {match,Line} -> ct:fail({error_found_in_log,Line}); + nomatch -> ok + end.</pre> + <p> + In this example we have test cases that verify, by parsing a + log file, that our SUT has performed a successful restart and + that no unexpected errors have been printed. + </p> + + <p>To execute the test cases in the test suite above, we could type this on + the Unix/Linux command line (assuming for this example that the suite module + is in the current working directory): + </p> + <pre> + $ ct_run -dir .</pre> + <p>or</p> + <pre> + $ ct_run -suite check_log_SUITE</pre> + + <p>If we want to use the Erlang shell to run our test, we could evaluate this call: + </p> + <pre> + 1> ct:run_test([{dir, "."}]).</pre> + <p>or</p> + <pre> + 1> ct:run_test([{suite, "check_log_SUITE"}]).</pre> + <p> + The result from running our test is printed in log files in HTML format + (stored in unique log directories on different level). This illustration + shows the log file structure: + </p> + + <p> + <image file="html_logs.gif"> + <icaption> + HTML log file structure. + </icaption> + </image> + </p> + </section> + + <section> + <title>What happens next?</title> + + <p>Well, you might already be asking yourself questions such as:</p> + + <list> + <item>"How and where can I specify variable data for my tests that mustn't + be hard-coded in the test suites (such as host names, addresses, + user login data, etc)?" The + <seealso marker="config_file_chapter#top">External Configuration Data</seealso> + chapter will give you that information. + </item> + <item>"Is there a way to declare a number of different tests and run them + in one session without having to write my own scripts? And can such + declarations be used for regression testing?" The + <seealso marker="run_test_chapter#test_specifications">Test Specifications</seealso> + chapter answers these questions. + </item> + <item>"Can test cases and/or test runs be automatically repeated?" Learn more about + <seealso marker="write_test_chapter#test_case_groups">Test Case Groups</seealso> + and also read about start flags/options in the + <seealso marker="run_test_chapter#ct_run">Running Tests</seealso> chapter and + the Reference Manual. + </item> + <item>"Will Common Test execute my test cases in sequence or in parallel?" The + <seealso marker="write_test_chapter#test_case_groups">Test Case Groups</seealso> + section in the Running Tests chapter will give you the answer. + </item> + <item>"What's the syntax for timetraps (mentioned above), and how do I set them?" + This is explained in the + <seealso marker="write_test_chapter#timetraps">Timetrap Timeouts</seealso> + part of the Writing Test Suites chapter. + </item> + <item>"What functions are available for logging and printing?" Check the + <seealso marker="write_test_chapter#logging">Logging</seealso> + section in the Writing Test Suites chapter. + </item> + <item>"I need data files for my tests. Where do I store them preferrably?" + You should read about + <seealso marker="write_test_chapter#data_priv_dir">Data and Private + Directories</seealso> for information about this. + </item> + <item>"May I start with a test suite example, please?" + <seealso marker="example_chapter#top">Sure!</seealso> + </item> + </list> + <p>You will probably want to get started on your own first test suites now, while + at the same time digging deeper into the Common Test User's Guide and Reference Manual. + You will find that there's lots more to learn about the things that have been introduced + in this chapter. You will of course also be presented many more useful features, such as the + ones listed above. Have fun! + </p> + </section> +</chapter> + + + + + + + + diff --git a/lib/common_test/doc/src/html_logs.gif b/lib/common_test/doc/src/html_logs.gif Binary files differnew file mode 100644 index 0000000000..3a3fd86bde --- /dev/null +++ b/lib/common_test/doc/src/html_logs.gif diff --git a/lib/common_test/doc/src/make.dep b/lib/common_test/doc/src/make.dep deleted file mode 100644 index e34075888d..0000000000 --- a/lib/common_test/doc/src/make.dep +++ /dev/null @@ -1,27 +0,0 @@ -# ---------------------------------------------------- -# >>>> Do not edit this file <<<< -# This file was automaticly generated by -# /home/otp/bin/docdepend -# ---------------------------------------------------- - - -# ---------------------------------------------------- -# TeX files that the DVI file depend on -# ---------------------------------------------------- - -book.dvi: basics_chapter.tex book.tex common_test_app.tex \ - config_file_chapter.tex cover_chapter.tex \ - ct.tex ct_cover.tex ct_ftp.tex ct_master.tex \ - ct_master_chapter.tex ct_rpc.tex ct_snmp.tex \ - ct_ssh.tex ct_telnet.tex dependencies_chapter.tex \ - event_handler_chapter.tex example_chapter.tex \ - install_chapter.tex part.tex ref_man.tex run_test.tex \ - run_test_chapter.tex test_structure_chapter.tex \ - unix_telnet.tex why_test_chapter.tex write_test_chapter.tex - -# ---------------------------------------------------- -# Source inlined when transforming from source to LaTeX -# ---------------------------------------------------- - -book.tex: ref_man.xml - diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index af96ef621f..8c3b13951d 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2004</year><year>2011</year> + <year>2004</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -32,6 +32,839 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.6.3.1</title> + + <section><title>Known Bugs and Problems</title> + <list> + <item> + <p> + The following corrections/changes are done in the + cth_surefire hook:</p> + <p> + <list> <item> Earlier there would always be a + 'properties' element under the 'testsuites' element. This + would exist even if there were no 'property' element + inside it. This has been changed so if there are no + 'property' elements to display, then there will not be a + 'properties' element either. </item> <item> The XML file + will now (unless other is specified) be stored in the top + log directory. Earlier, the default directory would be + the current working directory for the erlang node, which + would mostly, but not always, be the top log directory. + </item> <item> The 'hostname' attribute in the + 'testsuite' element would earlier never have the correct + value. This has been corrected. </item> <item> The + 'errors' attribute in the 'testsuite' element would + earlier display the number of failed testcases. This has + been changed and will now always have the value 0, while + the 'failures' attribute will show the number of failed + testcases. </item> <item> A new attribute 'skipped' is + added to the 'testsuite' element. This will display the + number of skipped testcases. These would earlier be + included in the number of failed test cases. </item> + <item> The total number of tests displayed by the 'tests' + attribute in the 'testsuite' element would earlier + include init/end_per_suite and init/end_per_group. This + is no longer the case. The 'tests' attribute will now + only count "real" test cases. </item> <item> Earlier, + auto skipped test cases would have no value in the 'log' + attribute. This is now corrected. </item> <item> A new + attributes 'log' is added to the 'testsuite' element. + </item> <item> A new option named 'url_base' is added for + this hook. If this option is used, a new attribute named + 'url' will be added to the 'testcase' and 'testsuite' + elements. </item> </list></p> + <p> + Own Id: OTP-10589</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.6.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The ct:run_test/1 option 'config' only worked with a + single config file, not a list of files. This has been + fixed.</p> + <p> + Own Id: OTP-10495</p> + </item> + <item> + <p> + ct_netconfc:close_session sometimes returned + {error,closed} because the ssh connection was closed + (from the server side) before the rpc-reply was received + by the client. This is normal and can not be helped. It + has been corrected so the return will be 'ok' in this + case. Other error situations will still give + {error,Reason}.</p> + <p> + Own Id: OTP-10510 Aux Id: kunagi-320 [231] </p> + </item> + <item> + <p> + ct_netconfc:close_session sometimes returned + {error,closed} or (if the connection was named) + {error,{process_down,Pid,normal}} because the ssh + connection was closed (from the server side) before the + rpc-reply was received by the client. This is normal and + can not be helped. It has been corrected so the return + will be 'ok' in this situation.</p> + <p> + Own Id: OTP-10570</p> + </item> + <item> + <p> + Fix bug where ct:require of same name with same config + would return name_in_use.</p> + <p> + Own Id: OTP-10572</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A new test case group search functionality has been + implemented that makes Common Test search automatically + through the group definitions tree (the return value of + groups/0) and create tests for all paths of nested groups + that match the specification. It also allows for + specifying unique paths to sub groups in order to avoid + execution of unwanted tests. This new feature can be used + whenever starting a test run by means of the ct_run + program, the ct:run_test/1 API function, or a Test + Specification. Details can be found in the Test Case + Group Execution section in the Running Tests chapter.</p> + <p> + Own Id: OTP-10466 Aux Id: kunagi-276 [187] </p> + </item> + </list> + </section> + + + <section><title>Known Bugs and Problems</title> + <list> + <item> + <p> + Restore Config data if lost when test case fails.</p> + <p> + Own Id: OTP-10070 Aux Id: kunagi-175 [86] </p> + </item> + <item> + <p> + IO server error in test_server.</p> + <p> + Own Id: OTP-10125 Aux Id: OTP-10101, kunagi-177 [88] </p> + </item> + <item> + <p> + Faulty connection handling in common_test.</p> + <p> + Own Id: OTP-10126 Aux Id: kunagi-178 [89] </p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.6.2.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The interactive mode (ct_run -shell) would not start + properly. This error has been fixed.</p> + <p> + Own Id: OTP-10414</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.6.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + If a CT hook function caused a crash, this could in some + situations cause Common Test to terminate due to an + illegal IO operation. This error has been corrected.</p> + <p> + Own Id: OTP-10050 Aux Id: seq12039 </p> + </item> + <item> + <p> + The Common Test documentation states that timetraps are + never active during execution of CT hook functions. This + was only true for post hook functions, not for pre hook + functions. The code for CT hooks has been modified to + behave according to the documentation.</p> + <p> + Own Id: OTP-10069</p> + </item> + <item> + <p> + If a CT hook function would call the exit/1 or throw/1 + BIF (possibly indirectly, e.g. as a result of a timeout + in gen_server:call/3), Common Test would hang. This + problem has been fixed.</p> + <p> + Own Id: OTP-10072 Aux Id: seq12053 </p> + </item> + <item> + <p> + The documentation has been updated with information about + how to deal with chaining of hooks which return + fail/skip.</p> + <p> + Own Id: OTP-10077 Aux Id: seq12048 </p> + </item> + <item> + <p> + When ct_hooks called the id/1 functions of multiple + hooks, it would reverse the order of the hooks and call + the proceeding init/2 calls in the wrong order. This has + been fixed.</p> + <p> + Own Id: OTP-10135</p> + </item> + <item> + <p> + The surefire hook now correctly handles autoskipped + initialization and test functions.</p> + <p> + Own Id: OTP-10158</p> + </item> + <item> + <p> + The ct:get_status/0 function failed to report status if a + parallel test case group was running at the time of the + call. This has been fixed and the return value for the + function has been updated. Please see the ct reference + manual for details.</p> + <p> + Own Id: OTP-10172</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + The support for "silent connections" has been updated to + include ssh. Also, a silent_connections term has been + added to the set of test specification terms.</p> + <p> + Own Id: OTP-9625 Aux Id: seq11918 </p> + </item> + <item> + <p> + It is now possible to specify an arbitrarily large tuple + as the requires config data when using require and + ct:get_config. See the ct:get_config and ct:require + reference manual pages for details about which keys are + allowed.</p> + <p> + This change introduces a backwards incompatability in the + <c>ct:require/2</c> interface. Previously when doing + <c>ct:require(a_name,{key,subkey})</c>, a_name would be + associated with key. This has been changed to that + <c>a_name</c> is associated with <c>subkey</c>. This + change also effects using <c>require</c> in an + suite/group/testcase info function.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-9626 Aux Id: seq11920 </p> + </item> + <item> + <p> + The ct_run program now sets the OS process exit status + before it ends. Value 0 indicates a successful test + result, 1 indicates one or more failed or auto-skipped + test cases, and 2 indicates test execution failure.</p> + <p> + Own Id: OTP-9865 Aux Id: OTP-10087 </p> + </item> + <item> + <p> + It is now possible to sort the HTML tables by clicking on + the header elements. In order to reset a sorted table, + the browser window should simply be refreshed. This + feature requires that the browser supports javascript, + and has javascript execution enabled. If the 'ct_run + -basic_html' flag is used, no javascript code is included + in the generated HTML code.</p> + <p> + Own Id: OTP-9896 Aux Id: seq12034, OTP-9835 </p> + </item> + <item> + <p> + A netconf client, ct_netconfc, is added to common_test. + It supports basic netconf functionality over SSH. In + order to allow testing of both success and failure cases, + it is intentionally written to allow non-standard + behavior.</p> + <p> + Own Id: OTP-10025</p> + </item> + <item> + <p> + The test specification term {define,Constant,Value} has + been introduced, which makes it possible to replace + constant names (atom()) with values (term()) in arbitrary + test specification terms. The 'define' makes the (now + deprecated) 'alias' term obsolete. More details, + including examples, can be found in the Test + Specifications chapter in the Common Test User's Guide.</p> + <p> + Own Id: OTP-10049</p> + </item> + <item> + <p> + Verbosity levels for log printouts has been added. This + makes it possible to specify preferred verbosity for + different categories of log printouts, as well as general + printouts (such as standard IO), to allow control over + which strings get printed and which get ignored. New + versions of the Common Test logging functions, ct:log, + ct:pal and ct:print, have been introduced, with a new + Importance argument added. The Importance value is + compared to the verbosity level at runtime. More + information can be found in the chapter about Logging in + the Common Test User's Guide.</p> + <p> + Own Id: OTP-10067 Aux Id: seq12050 </p> + </item> + <item> + <p> + The return values of ct:run_test/1 and ct:run_testspec/1 + have been changed from an uninformative 'ok' (independent + of the test outcome) to a value, + {Ok,Failed,{UserSkipped,AutoSkipped}} (all integers), + that presents the final test case result, or a value, + {error,Reason}, that informs about fatal test execution + failure. See details in the reference manual for ct.</p> + <p> + Own Id: OTP-10087 Aux Id: OTP-9865 </p> + </item> + <item> + <p> + The test specification syntax has been updated with new + and missing terms, such as 'define', 'verbosity', + 'auto_compile', 'stylesheet', 'silent_connections', + 'basic_html' and 'release_shell'. See the Test + Specification chapter in the Common Test User's Guide for + details.</p> + <p> + Own Id: OTP-10089 Aux Id: OTP-10049 </p> + </item> + <item> + <p> + It is now possible to pause execution of a test case, by + calling the ct:break/1/2 function. Execution is resumed + with a call to ct:continue/0/1. Break/continue also works + for test cases executing in parallel. See the ct + reference manual for details.</p> + <p> + Own Id: OTP-10127</p> + </item> + <item> + <p> + It is now possible to send user defined events from a + testcase which will be picked up by the installed event + handler.</p> + <p> + Own Id: OTP-10157</p> + </item> + <item> + <p> + A new start option, release_shell, for ct:run_test/1, has + been added, which makes Common Test release the shell + process after the test suite compilation phase is + finished. For details, see the Running Tests chapter in + the User's Guide.</p> + <p> + Own Id: OTP-10248 Aux Id: OTP-10127 </p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.6.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Common Test adds the test suite directories to the code + path before executing the tests. These directories should + also be removed from the code path at the end of the test + run, which, prior to this fix, was not performed.</p> + <p> + Own Id: OTP-9595</p> + </item> + <item> + <p> + An entry is now created in the index.html file (i.e. the + overview file for the test run) for each repeated test + during a test run. This was previously not the case. Note + that in the top level (logdir) index file, however, only + the last test result is listed. For example, given the + test spec: + [{merge_tests,false},{dirs,"test1"},{dirs,"test1"}]. In + the index file for the test run (under + Logdir/ct_run.Node.Date.Time), both tests are listed. In + the top level index file (under Logdir), only the last + test is listed (one has to find the previous results + through the all_runs.html file).</p> + <p> + Own Id: OTP-9634 Aux Id: seq11924 </p> + </item> + <item> + <p> + After a test case timeout or abortion, the + end_per_testcase function executes on a new dedicated + process. The group leader for this process should be set + to the IO server for the test case, which was not done + properly. The result of this error was that no warnings + about end_per_testcase failing or timing out were ever + printed in the test case log. Also, help functions such + as e.g. test_server:stop_node/1, attempting to + synchronize with the IO server, would hang. The fault has + been corrected.</p> + <p> + Own Id: OTP-9666</p> + </item> + <item> + <p> + The ct:get_status/0 function would cause the calling + process to receive 'DOWN' messages if no tests were + running at the time of the call. This bug has been fixed.</p> + <p> + Own Id: OTP-9830 Aux Id: seq11975 </p> + </item> + <item> + <p> + A deadlock situation could occur if Common Test is + forwarding error_handler printouts to Test Server at the + same time a new test case is starting. This error has + been fixed.</p> + <p> + Own Id: OTP-9894</p> + </item> + <item> + <p> + A link to the ct_run program is now created, as expected, + in the installation bin directory (default + /usr/local/bin) during 'make install'.</p> + <p> + Own Id: OTP-9898</p> + </item> + <item> + <p> + Using the repeat, duration or until option with + ct:run_test/1, would cause an infinite loop. This has + been fixed.</p> + <p> + Own Id: OTP-9899</p> + </item> + <item> + <p> + Two or more test cases executing in parallel and printing + to screen at the same time with ct:pal/2/3 or + ct:print/2/3 could write into each other's "slots" and + create a mess of mixed strings. In order to avoid this, + only a single IO message is now ever sent per printout + call.</p> + <p> + Own Id: OTP-9900 Aux Id: OTP-9904 </p> + </item> + <item> + <p> + When a test case was killed because of a timetrap + timeout, the current location (suite, case and line) was + not printed correctly in the log files. This has been + corrected.</p> + <p> + Own Id: OTP-9930 Aux Id: seq12002 </p> + </item> + <item> + <p> + The wrong exit location was printed in the log file when + ct:fail/1 or ct_fail/2 was called.</p> + <p> + Own Id: OTP-9933 Aux Id: seq12002 </p> + </item> + <item> + <p> + Test Server and Common Test would add new error handlers + with each test run and fail to remove previously added + ones. In the case of Test Server, this would only happen + if SASL was not running on the test node. This has been + fixed.</p> + <p> + Own Id: OTP-9941 Aux Id: seq12009 </p> + </item> + <item> + <p> + If a test case process was terminated due to an exit + signal from a linked process, Test Server failed to + report the correct name of the suite and case to the + framework. This has been corrected.</p> + <p> + Own Id: OTP-9958 Aux Id: OTP-9855 </p> + </item> + <item> + <p> + When starting a test with ct_run and adding a directory + to the code path using -pa or -pz (preceding -erl_args), + Common Test would delete any existing directory in the + code path with the same base name (see + filename:basename/1) as the directory being added. This + has been fixed.</p> + <p> + Own Id: OTP-9964</p> + </item> + <item> + <p> + If passing two or more directories with the same base + name (see filename:basename/1) to Common Test with ct_run + -pa, only one of the directories would actually be added.</p> + <p> + Own Id: OTP-9975 Aux Id: seq12019 </p> + </item> + <item> + <p> + Configuration data required by the group info function + was deleted before the call to post_end_per_group, which + made it impossible for the hook function to read and use + the data in question. This has been fixed.</p> + <p> + Own Id: OTP-9989</p> + </item> + <item> + <p> + Disabling built-in hooks in a test specification was + ignored, this has now been fixed.</p> + <p> + Own Id: OTP-10009</p> + </item> + <item> + <p> + Various typographical errors corrected in documentation + for common_test, driver, erl_driver and windows + installation instructions. (Thanks to Tuncer Ayaz)</p> + <p> + Own Id: OTP-10037</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A new optional feature has been introduced that enables + Common Test to generate priv_dir directory names that are + unique for each test case or config function. The name of + the option/flag is 'create_priv_dir' and it can be set to + value 'auto_per_run' (which is the default, existing, + behaviour), or 'auto_per_tc' or 'manual_per_tc'. If + 'auto_per_tc' is used, Test Server creates a dedicated + priv_dir automatically for each test case (which can be + very expensive in case of many and/or repeated cases). If + 'manual_per_tc' is used, the user needs to create the + priv_dir explicitly by calling the new function + ct:make_priv_dir/0.</p> + <p> + Own Id: OTP-9659 Aux Id: seq11930 </p> + </item> + <item> + <p> + A column for test case group name has been added to the + suite overview HTML log file.</p> + <p> + Own Id: OTP-9730 Aux Id: seq11952 </p> + </item> + <item> + <p> + It is now possible to use the post_end_per_testcase CT + hook function to print a comment for a test case in the + overview log file, even if the test case gets killed by a + timetrap or unknown exit signal, or if the + end_per_testcase function times out.</p> + <p> + Own Id: OTP-9855 Aux Id: seq11979 </p> + </item> + <item> + <p> + The pre- and post CT hook functions are now always called + for all configuration functions, even for configuration + functions that are not implemented in the test suite.</p> + <p> + Own Id: OTP-9880 Aux Id: seq11993 </p> + </item> + <item> + <p> + Common Test will now print error information (with a time + stamp) in the test case log file immediately when a test + case fails. This makes it easier to see when, in time, + the fault actually occured, and aid the job of locating + relevant trace and debug printouts in the log.</p> + <p> + Own Id: OTP-9904 Aux Id: seq11985, OTP-9900 </p> + </item> + <item> + <p> + Test Server has been modified to check the SASL + errlog_type parameter when receiving an error logger + event, so that it doesn't print reports of type that the + user has disabled.</p> + <p> + Own Id: OTP-9955 Aux Id: seq12013 </p> + </item> + <item> + <p> + The test specification term 'skip_groups' was implemented + in Common Test v1.6. It was never documented however, + which has now been attended to. Please see the Test + Specifications chapter in the User's Guide for + information.</p> + <p> + Own Id: OTP-9972</p> + </item> + <item> + <p> + The Common Test Master has been updated to use a CSS + style sheet for the html log files.</p> + <p> + Own Id: OTP-9973</p> + </item> + <item> + <p> + If the init_per_group/2 and end_per_group/2 functions are + not implemented in the test suite, Common Test calls it's + own local init- and end functions - previously named + ct_init_per_group/2 and ct_end_per_group/2 - when a group + is executed. These functions have been renamed + init_per_group/2 and end_per_group/2 respectively. Note + that this may affect any user event handler identifying + events by the old names.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-9986 Aux Id: OTP-9992 </p> + </item> + <item> + <p> + By specifying a user defined function ({M,F,A} or fun) as + timetrap value, either by means of an info function or by + calling ct:timetrap/1, it is now possible to set a + timetrap that will be triggered when the user function + returns.</p> + <p> + Own Id: OTP-9988 Aux Id: OTP-9501, seq11894 </p> + </item> + <item> + <p> + If the optional configuration functions init_per_suite/1 + and end_per_suite/1 are not implemented in the test + suite, local Common Test versions of these functions are + called instead, and will be displayed in the overview log + file. Any printouts made by the pre- or + post_init_per_suite and pre- or post_end_per_suite hook + functions are saved in the log files for these functions.</p> + <p> + Own Id: OTP-9992</p> + </item> + <item> + <p> + A hook has been added to common test which outputs + surefire XML for usage together with CI tools such as + Jenkins. To enable the hook pass '-ct_hooks cth_surefire' + to ct_run. See the CTH documentation for more details.</p> + <p> + Own Id: OTP-9995</p> + </item> + </list> + </section> + +</section> + +<section><title>Common_Test 1.6</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + A Getting Started chapter has been added to the Common + Test User's Guide.</p> + <p> + Own Id: OTP-9156</p> + </item> + <item> + <p> + The test case group info function has been implemented in + Common Test. Before execution of a test case group, a + call is now made to <c>TestSuite:group(GroupName)</c>. + The function returns a list of test properties, e.g. to + specify timetrap values, require configuration data, etc + (analogue to the test suite- and test case info + function). The scope of the properties set by + <c>group(GroupName)</c> is all test cases and sub-groups + of group <c>GroupName</c>.</p> + <p> + Own Id: OTP-9235</p> + </item> + <item> + <p> + Common Test hooks are now in a final supported version. + The Common Test hooks allow you to abstract out + initialization behaviour that is common to multiple test + suites into one place and also extend the behaviour of a + suite without changing the suite itself. For more + information see the Common Test user's guide.</p> + <p> + Own Id: OTP-9449</p> + </item> + <item> + <p> + A new built-in common test hook has been added which + captures error_logger and SASL event and prints them in + the testcase log. To disable this (and any other built-in + hooks) pass 'enable_builtin_hooks false' to common test.</p> + <p> + Own Id: OTP-9543</p> + </item> + <item> + <p> + Common Test now calls info functions also for the + <c>init/end_per_suite/1</c> and + <c>init/end_per_group/2</c> configuration functions. + These can be used e.g. to set timetraps and require + external configuration data relevant only for the + configuration functions in question (without affecting + properties set for groups and test cases in the suite). + The info function for <c>init/end_per_suite(Config)</c> + is <c>init/end_per_suite()</c>, and for + <c>init/end_per_group(GroupName,Config)</c> it's + <c>init/end_per_group(GroupName)</c>. Info functions can + not be used with <c>init/end_per_testcase(TestCase, + Config)</c>, since these configuration functions execute + on the test case process and will use the same properties + as the test case (i.e. properties set by the test case + info function, <c>TestCase()</c>).</p> + <p> + Own Id: OTP-9569</p> + </item> + <item> + <p> + It's now possible to read the full name of the test case + log file during execution. One way to do this is to + lookup it up as value of the key <c>tc_logfile</c> in the + test case <c>Config</c> list (which means it can also be + read by a pre- or post Common Test hook function). The + data is also sent with the event + <c>#event{name=tc_logfile,data={{Suite,Func},LogFileName}}</c>, + and can be read by any installed event handler.</p> + <p> + Own Id: OTP-9676 Aux Id: seq11941 </p> + </item> + <item> + <p> + The look of the HTML log files generated by Common Test + and Test Server has been improved (and made easier to + customize) by means of a CSS file.</p> + <p> + Own Id: OTP-9706</p> + </item> + <item> + <p> + Functions ct:fail(Format, Args) and ct:comment(Format, + Args) have been added in order to make printouts of + formatted error and comment strings easier (no need for + the user to call io_lib:format/2 explicitly).</p> + <p> + Own Id: OTP-9709 Aux Id: seq11951 </p> + </item> + <item> + <p> + The order in which ct hooks are executed for cleanup + hooks (i.e. *_end_per_* hooks) has been reversed.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-9774 Aux Id: seq11913 </p> + </item> + <item> + <p> + Printouts to stdout may be captured during test case + execution. This is useful in order to e.g. read and parse + tty printouts from the SUT during test case execution (if + necessary, say, to determine the outcome of the test). + The capturing session is started with + <c>ct:capture_start/0</c>, and stopped with + <c>ct:capture_stop/0</c>. The list of buffered strings is + read and purged with <c>ct:capture_get/0/1</c>. It's + possible to filter out printouts made with + <c>ct:log/2/3</c> and <c>ct:pal/2/3</c> from the captured + list of strings. This is done by calling + <c>capture_get/1</c> with a list of log categories to + exclude.</p> + <p> + Own Id: OTP-9775</p> + </item> + <item> + <p> + The syntax for specifying test case groups in the all/0 + list has been extended to include execution properties + for both groups and sub-groups. The properties specified + in all/0 for a group overrides the properties specified + in the group declaration (in groups/0). The main purpose + of this extension is to make it possible to run the same + set of tests, but with different properties, without + having to declare copies of the group in question. Also, + the same syntax may be used in test specifications in + order to change properties of groups at the time of + execution, without having to edit the test suite. Please + see the User's Guide for details and examples.</p> + <p> + Own Id: OTP-9809 Aux Id: OTP-9235 </p> + </item> + </list> + </section> + + + <section><title>Known Bugs and Problems</title> + <list> + <item> + <p> + Fix problems in CT/TS due to line numbers in exceptions.</p> + <p> + Own Id: OTP-9203</p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.5.5</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/doc/src/part.xml b/lib/common_test/doc/src/part.xml index 3284bcadaa..a74185221d 100644 --- a/lib/common_test/doc/src/part.xml +++ b/lib/common_test/doc/src/part.xml @@ -65,6 +65,7 @@ </description> <xi:include href="basics_chapter.xml"/> + <xi:include href="getting_started_chapter.xml"/> <xi:include href="install_chapter.xml"/> <xi:include href="write_test_chapter.xml"/> <xi:include href="test_structure_chapter.xml"/> diff --git a/lib/common_test/doc/src/ref_man.xml b/lib/common_test/doc/src/ref_man.xml index a9fdef7359..6fede88434 100644 --- a/lib/common_test/doc/src/ref_man.xml +++ b/lib/common_test/doc/src/ref_man.xml @@ -4,7 +4,7 @@ <application xmlns:xi="http://www.w3.org/2001/XInclude"> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -71,6 +71,7 @@ <xi:include href="ct_cover.xml"/> <xi:include href="ct_ftp.xml"/> <xi:include href="ct_ssh.xml"/> + <xi:include href="ct_netconfc.xml"/> <xi:include href="ct_rpc.xml"/> <xi:include href="ct_snmp.xml"/> <xi:include href="ct_telnet.xml"/> diff --git a/lib/common_test/doc/src/run_test_chapter.xml b/lib/common_test/doc/src/run_test_chapter.xml index d3c6847d85..a0b2c96006 100644 --- a/lib/common_test/doc/src/run_test_chapter.xml +++ b/lib/common_test/doc/src/run_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -119,7 +119,7 @@ <item><c><![CDATA[ct_run -userconfig <callbackmodulename> <configfilenames> -suite <suiteswithfullpath>]]></c> </item> <item><c><![CDATA[ct_run -config <configfilenames> -suite <suitewithfullpath> - -group <groupnames> -case <casenames>]]></c></item> + -group <groups> -case <casenames>]]></c></item> </list> <p>Examples:</p> <p><c>$ ct_run -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST</c></p> @@ -137,6 +137,8 @@ <p><c>$ ct_run -suite ./testdir/x_SUITE ./testdir/y_SUITE</c></p> + <p>For more details on <seealso marker="run_test_chapter#group_execution">test case group execution</seealso>, please see below.</p> + <p>Other flags that may be used with <c>ct_run</c>:</p> <list> <item><c><![CDATA[-logdir <dir>]]></c>, specifies where the HTML log files are to be written.</item> @@ -159,6 +161,8 @@ <seealso marker="event_handler_chapter#event_handling">event handlers</seealso> including start arguments.</item> <item><c><![CDATA[-ct_hooks <ct_hooks>]]></c>, to install <seealso marker="ct_hooks_chapter#installing">Common Test Hooks</seealso> including start arguments.</item> + <item><c><![CDATA[-enable_builtin_hooks <bool>]]></c>, to enable/disable + <seealso marker="ct_hooks_chapter#builtin_cths">Built-in Common Test Hooks</seealso>. Default is <c>true</c>.</item> <item><c><![CDATA[-include]]></c>, specifies include directories (see above).</item> <item><c><![CDATA[-no_auto_compile]]></c>, disables the automatic test suite compilation feature (see above).</item> <item><c><![CDATA[-multiply_timetraps <n>]]></c>, extends <seealso marker="write_test_chapter#timetraps">timetrap @@ -176,6 +180,8 @@ <item><c><![CDATA[-basic_html]]></c>, switches off html enhancements that might not be compatible with older browsers.</item> <item><c><![CDATA[-logopts <opts>]]></c>, makes it possible to modify aspects of the logging behaviour, see <seealso marker="run_test_chapter#logopts">Log options</seealso> below.</item> + <item><c><![CDATA[-verbosity <levels>]]></c>, sets <seealso marker="write_test_chapter#logging">verbosity levels + for printouts</seealso>.</item> </list> <note><p>Directories passed to Common Test may have either relative or absolute paths.</p></note> @@ -194,62 +200,232 @@ the current working directory of the Erlang Runtime System during the test run!</p> </note> - <p>For more information about the <c>ct_run</c> program, see the - <seealso marker="install_chapter#general">Installation</seealso> chapter. - </p> - </section> - - <section> - <title>Running tests from the Web based GUI</title> - - <p>The web based GUI, VTS, is started with the <c>ct_run</c> - program. From the GUI you can load config files, and select - directories, suites and cases to run. You can also state the - config files, directories, suites and cases on the command line - when starting the web based GUI. - </p> - + <p>The <c>ct_run</c> program sets the exit status before shutting down. The following values + are defined:</p> <list> - <item><c>ct_run -vts</c></item> - <item><c><![CDATA[ct_run -vts -config <configfilename>]]></c></item> - <item><c><![CDATA[ct_run -vts -config <configfilename> -suite <suitewithfullpath> - -case <casename>]]></c></item> + <item><c>0</c> indicates a successful testrun, i.e. one without failed or auto-skipped test cases.</item> + <item><c>1</c> indicates that one or more test cases have failed, or have been auto-skipped.</item> + <item><c>2</c> indicates that the test execution has failed because of e.g. compilation errors, an + illegal return value from an info function, etc.</item> </list> + <p>If auto-skipped test cases should not affect the exit status, you may change the default + behaviour using start flag:</p> + <pre>-exit_status ignore_config</pre> - <p>From the GUI you can run tests and view the result and the logs. + <p>For more information about the <c>ct_run</c> program, see the + <seealso marker="ct_run#top">Reference Manual</seealso> and the + <seealso marker="install_chapter#general">Installation</seealso> chapter. </p> - - <p>Note that <c>ct_run -vts</c> will try to open the Common Test start - page in an existing web browser window or start the browser if it is - not running. Which browser should be started may be specified with - the browser start command option:</p> - <p><c><![CDATA[ct_run -vts -browser <browser_start_cmd>]]></c></p> - <p>Example:</p> - <p><c><![CDATA[$ ct_run -vts -browser 'firefox&']]></c></p> - <p>Note that the browser must run as a separate OS process or VTS will hang!</p> - <p>If no specific browser start command is specified, Firefox will - be the default browser on Unix platforms and Internet Explorer on Windows. - If Common Test fails to start a browser automatically, or <c>'none'</c> is - specified as the value for -browser (i.e. <c>-browser none</c>), start your - favourite browser manually and type in the URL that Common Test - displays in the shell.</p> </section> - + <section> <title>Running tests from the Erlang shell or from an Erlang program</title> <p>Common Test provides an Erlang API for running tests. The main (and most flexible) function for specifying and executing tests is called - <c>ct:run_test/1</c>. This function takes the same start parameters as - the <c>ct_run</c> program described above, only the flags are instead + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>. + This function takes the same start parameters as + the <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + program described above, only the flags are instead given as options in a list of key-value tuples. E.g. a test specified with <c>ct_run</c> like:</p> + <p><c>$ ct_run -suite ./my_SUITE -logdir ./results</c></p> - <p>is with <c>ct:run_test/1</c> specified as:</p> + <p>is with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> specified as:</p> <p><c>1> ct:run_test([{suite,"./my_SUITE"},{logdir,"./results"}]).</c></p> - <p>For detailed documentation, please see the <c>ct</c> manual page.</p> + + <p>The function returns the test result, represented by the tuple: + <c>{Ok,Failed,{UserSkipped,AutoSkipped}}</c>, where each element is an + integer. If test execution fails, the function returns the tuple: + <c>{error,Reason}</c>, where the term <c>Reason</c> explains the + failure.</p> + + <section> + <title>Releasing the Erlang shell</title> + <p>During execution of tests, started with + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + the Erlang shell process, controlling stdin, will remain the top + level process of the Common Test system of processes. The result + is that the Erlang shell is not available for interaction during + the test run. If this is not desirable, maybe because the shell is needed + for debugging purposes or for interaction with the SUT during test + execution, you may set the <c>release_shell</c> start option to + <c>true</c> (in the call to <c>ct:run_test/1</c> or by + using the corresponding test specification term, see below). This will + make Common Test release the shell immediately after the test suite + compilation stage. To accomplish this, a test runner process + is spawned to take control of the test execution, and the effect is that + <c>ct:run_test/1</c> returns the pid of this process rather than the + test result - which instead is printed to tty at the end of the test run.</p> + <note><p>Note that in order to use the + <c><seealso marker="ct#break-1">ct:break/1/2</seealso></c> and + <c><seealso marker="ct#continue-0">ct:continue/0/1</seealso></c> functions, + <c>release_shell</c> <em>must</em> be set to <c>true</c>.</p></note> + </section> + + <p>For detailed documentation about + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, + please see the + <c><seealso marker="ct#run_test-1">ct</seealso></c> manual page.</p> </section> + <marker id="group_execution"></marker> + <section> + <title>Test case group execution</title> + + <p>With the <c>ct_run</c> flag, or <c>ct:run_test/1</c> option <c>group</c>, + one or more test case groups can be specified, optionally in combination + with specific test cases. The syntax for specifying groups is as follows + (on the command line):</p> + + <pre> + <![CDATA[$ ct_run -group <group_names_or_paths> [-case <cases>]]]></pre> + <p>or (in the Erlang shell):</p> + <pre> + <![CDATA[1> ct:run_test([{group,GroupsNamesOrPaths}, {case,Cases}]).]]></pre> + + <p>The <c>group_names_or_paths</c> parameter specifies either one + or more group names and/or one or more group paths. At start up, + Common Test will search for matching groups in the group definitions + tree (i.e. the list returned from <c>Suite:groups/0</c>, please see the + <seealso marker="write_test_chapter#test_case_groups">Test case groups</seealso> + chapter for details). + Given a group name, say <c>g</c>, Common Test will search for all paths + that lead to <c>g</c>. By path here we mean a sequence of nested groups, + all of which have to be followed in order to get from the top level + group to <c>g</c>. Actually, what Common Test needs to do in order to + execute the test cases in group <c>g</c>, is to call the + <c>init_per_group/2</c> function for each group in the path to + <c>g</c>, as well as all corresponding <c>end_per_group/2</c> + functions afterwards. The obvious reason for this is that the configuration + of a test case in <c>g</c> (and its <c>Config</c> input data) depends on + <c>init_per_testcase(TestCase, Config)</c> and its return value, which + in turn depends on <c>init_per_group(g, Config)</c> and its return value, + which in turn depends on <c>init_per_group/2</c> of the group above + <c>g</c>, etc, all the way up to the top level group.</p> + + <p>As you may have already realized, this means that if there is more than + one way to locate a group (and its test cases) in a path, the result of the + group search operation is a number of tests, all of which will be performed. + Common Test actually interprets a group specification that consists of a + single name this way:</p> + + <p>"Search and find all paths in the group definitions tree that lead + to the specified group and, for each path, create a test which (1) executes + all configuration functions in the path to the specified group, then (2) + executes all - or all matching - test cases in this group, as well as (3) + all - or all matching - test cases in all sub groups of the group". + </p> + + <p>It is also possible for the user to specify a specific group path with + the <c>group_names_or_paths</c> parameter. With this type of specification it's + possible to avoid execution of unwanted groups (in otherwise matching paths), + and/or the execution of sub groups. The syntax of the group path is a list of + group names in the path, e.g. on the command line: + </p> + <p><c>$ ct_run -suite "./x_SUITE" -group [g1,g3,g4] -case tc1 tc5</c></p> + <p>or similarly in the Erlang shell (requires a list within the groups list):</p> + <p><c>1> ct:run_test([{suite,"./x_SUITE"}, {group,[[g1,g3,g4]]}, {testcase,[tc1,tc5]}]).</c></p> + + <p>The last group in the specified path will be the terminating group in + the test, i.e. no sub groups following this group will be executed. In the + example above, <c>g4</c> is the terminating group, hence Common Test will + execute a test that calls all init configuration functions in the path to + <c>g4</c>, i.e. <c>g1..g3..g4</c>. It will then call test cases <c>tc1</c> + and <c>tc5</c> in <c>g4</c> and finally all end configuration functions in order + <c>g4..g3..g1</c>.</p> + + <p>Note that the group path specification doesn't necessarily + have to include <em>all</em> groups in the path to the terminating group. + Common Test will search for all matching paths if given an incomplete group + path.</p> + + <p>Note also that it's possible to combine group names and group paths with the + <c>group_names_or_paths</c> parameter. Each element is treated as + an individual specification in combination with the <c>cases</c> parameter. + See examples below.</p> + + <p>Examples:</p> + <pre> + -module(x_SUITE). + ... + %% The group definitions: + groups() -> + [{top1,[],[tc11,tc12, + {sub11,[],[tc12,tc13]}, + {sub12,[],[tc14,tc15, + {sub121,[],[tc12,tc16]}]}]}, + + {top2,[],[{group,sub21},{group,sub22}]}, + {sub21,[],[tc21,{group,sub2X2}]}, + {sub22,[],[{group,sub221},tc21,tc22,{group,sub2X2}]}, + {sub221,[],[tc21,tc23]}, + {sub2X2,[],[tc21,tc24]}]. + </pre> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group all</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,all}]).</c></p> + <p>Two tests will be executed, one for all cases and all sub groups under <c>top1</c>, + and one for all under <c>top2</c>. (We would get the same result with + <c>-group top1 top2</c>, or <c>{group,[top1,top2]}</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}]).</c></p> + <p>This will execute one test for all cases and sub groups under <c>top1</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1 -case tc12</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc12]}]).</c></p> + <p>This will run a test that executes <c>tc12</c> in <c>top1</c> and any sub group + under <c>top1</c> where it can be found (<c>sub11</c> and <c>sub121</c>).</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [top1] -case tc12</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[top1]]}, {testcase,[tc12]}]).</c></p> + <p>This will execute <c>tc12</c> <em>only</em> in group <c>top1</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group top1 -case tc16</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc16]}]).</c></p> + <p>This will search <c>top1</c> and all its sub groups for <c>tc16</c> and the result + will be that this test case executes in group <c>sub121</c>. (The specific path: + <c>-group [sub121]</c> or <c>{group,[[sub121]]}</c>, would have given + us the same result in this example).</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group sub12 [sub12]</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[sub12,[sub12]]}]).</c></p> + <p>This will execute two tests, one that includes all cases and sub groups under + <c>sub12</c>, and one with <em>only</em> the test cases in <c>sub12</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group sub2X2</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[sub2X2]}]).</c></p> + <p>In this example, Common Test will find and execute two tests, one for the path from + <c>top2</c> to <c>sub2X2</c> via <c>sub21</c>, and one from <c>top2</c> to <c>sub2X2</c> + via <c>sub22</c>.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [sub21,sub2X2]</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub21,sub2X2]]}]).</c></p> + <p>Here, by specifying the unique path: <c>top2 -> sub21 -> sub2X2</c>, only one test + is executed. The second possible path from <c>top2</c> to <c>sub2X2</c> (above) + will be discarded.</p> + <br></br> + <p><c>$ ct_run -suite "x_SUITE" -group [sub22] -case tc22 tc21</c></p> + <p><c>1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub22]]}, {testcase,[tc22,tc21]}]).</c></p> + <p>In this example only the test cases for <c>sub22</c> will be executed, and in + reverse order compared to the group definition.</p> + <br></br> + + <p>If a test case that belongs to a group (according to the group definition), is executed + without a group specification, i.e. simply by means of (command line):</p> + <p><c>$ ct_run -suite "my_SUITE" -case my_tc</c></p> + <p>or (Erlang shell):</p> + <p><c>1> ct:run_test([{suite,"my_SUITE"}, {testcase,my_tc}]).</c></p> + <p>then Common Test ignores the group definition and executes the test case in the scope of the + test suite only (no group configuration functions are called).</p> + + <p>The group specification feature, exactly as it has been presented in this section, can also + be used in <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso> (with some extra features added). Please see below.</p> + </section> + + <section> <title>Running the interactive shell mode</title> @@ -264,9 +440,9 @@ for trying out various operations during test suite development.</p> <p>To invoke the interactive shell mode, you can start an Erlang shell - manually and call <c>ct:install/1</c> to install any configuration + manually and call <c><seealso marker="ct#install-1">ct:install/1</seealso></c> to install any configuration data you might need (use <c>[]</c> as argument otherwise), then - call <c>ct:start_interactive/0</c> to start Common Test. If you use + call <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> to start Common Test. If you use the <c>ct_run</c> program, you may start the Erlang shell and Common Test in the same go by using the <c>-shell</c> and, optionally, the <c>-config</c> and/or <c>-userconfig</c> flag. Examples: @@ -285,7 +461,8 @@ <p>If any functions using "required config data" (e.g. ct_telnet or ct_ftp functions) are to be called from the erlang shell, config - data must first be required with <c>ct:require/[1,2]</c>. This is + data must first be required with <c><seealso marker="ct#require-1"> + ct:require/1/2</seealso></c>. This is equivalent to a <c>require</c> statement in the <seealso marker="write_test_chapter#suite">Test Suite Info Function</seealso> or in the <seealso @@ -312,11 +489,11 @@ is not supported.</p> <p>If you wish to exit the interactive mode (e.g. to start an - automated test run with <c>ct:run_test/1</c>), call the function - <c>ct:stop_interactive/0</c>. This shuts down the + automated test run with <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), call the function + <c><seealso marker="ct#stop_interactive-0">ct:stop_interactive/0</seealso></c>. This shuts down the running <c>ct</c> application. Associations between configuration names and data created with <c>require</c> are - consequently deleted. <c>ct:start_interactive/0</c> will get you + consequently deleted. <c><seealso marker="ct#start_interactive-0">ct:start_interactive/0</seealso></c> will get you back into interactive mode, but the previous state is not restored.</p> </section> @@ -324,7 +501,7 @@ <title>Step by step execution of test cases with the Erlang Debugger</title> <p>By means of <c>ct_run -step [opts]</c>, or by passing the - <c>{step,Opts}</c> option to <c>ct:run_test/1</c>, it is possible + <c>{step,Opts}</c> option to <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>, it is possible to get the Erlang Debugger started automatically and use its graphical interface to investigate the state of the current test case and to execute it step by step and/or set execution breakpoints.</p> @@ -350,35 +527,38 @@ <marker id="test_specifications"></marker> <section> - <title>Using test specifications</title> + <title>Test Specifications</title> <p>The most flexible way to specify what to test, is to use a so called test specification. A test specification is a sequence of - Erlang terms. The terms may be declared in a text file or passed - to the test server at runtime as a list - (see <c>run_testspec/1</c> in the manual page - for <c>ct</c>). There are two general types of terms: - configuration terms and test specification terms.</p> + Erlang terms. The terms are normally declared in a text file (see + <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c>), but + may also be passed to Common Test on the form of a list (see + <c><seealso marker="ct#run_testspec-1">ct:run_testspec/1</seealso></c>). + There are two general types of terms: configuration terms and test + specification terms.</p> <p>With configuration terms it is possible to e.g. label the test run (similar to <c>ct_run -label</c>), evaluate arbitrary expressions - before starting a test, import configuration - data (similar to - <c>ct_run -config/-userconfig</c>), specify HTML log directories (similar - to - <c>ct_run -logdir</c>), give aliases to test nodes and test - directories (to make a specification easier to read and - maintain), enable code coverage analysis (see - the <seealso marker="cover_chapter#cover">Code Coverage - Analysis</seealso> chapter) and specify event_handler plugins - (see the <seealso marker="event_handler_chapter#event_handling"> - Event Handling</seealso> chapter). There is also a term for - specifying include directories that should be passed on to the - compiler when automatic compilation is performed (similar - to <c>ct_run -include</c>, see above).</p> + before starting the test, import configuration data (similar to + <c>ct_run -config/-userconfig</c>), specify the top level HTML log + directory (similar to <c>ct_run -logdir</c>), enable code coverage + analysis (similar to <c>ct_run -cover</c>), install Common Test Hooks + (similar to <c>ct_run -ch_hooks</c>), install event_handler plugins + (similar to <c>ct_run -event_handler</c>), specify include directories + that should be passed to the compiler for automatic compilation + (similar to <c>ct_run -include</c>), disable the auto compilation + feature (similar to <c>ct_run -no_auto_compile</c>), set verbosity + levels (similar to <c>ct_run -verbosity</c>), and more.</p> + <p>Configuration terms can be combined with <c>ct_run</c> start flags, + or <c>ct:run_test/1</c> options. The result will for some flags/options + and terms be that the values are merged (e.g. configuration files, + include directories, verbosity levels, silent connections), and for + others that the start flags/options override the test specification + terms (e.g. log directory, label, style sheet, auto compilation).</p> <p>With test specification terms it is possible to state exactly which tests should run and in which order. A test term specifies - either one or more suites, one or more test case groups, or one - or more test cases in a group or suite.</p> + either one or more suites, one or more test case groups (possibly nested), + or one or more test cases in a group (or in multiple groups) or in a suite.</p> <p>An arbitrary number of test terms may be declared in sequence. Common Test will by default compile the terms into one or more tests to be performed in one resulting test run. Note that a term that @@ -389,39 +569,69 @@ S, is a test of all cases in S. However, if a term specifying test case X and Y in S is merged with a term specifying case Z in S, the result is a test of X, Y and Z in S. To disable this - behaviour, it is possible in test specification to set the - <c>merge_tests</c> term to <c>false</c>.</p> + behaviour, i.e. to instead perform each test sequentially in a "script-like" + manner, the term <c>merge_tests</c> can be set to <c>false</c> in + the test specification.</p> <p>A test term can also specify one or more test suites, groups, or test cases to be skipped. Skipped suites, groups and cases - are not executed and show up in the HTML test log files as + are not executed and show up in the HTML log files as SKIPPED.</p> <p>When a test case group is specified, the resulting test - executes the - <c>init_per_group</c> function, followed by all test cases and - sub groups (including their configuration functions), and + executes the <c>init_per_group</c> function, followed by all test + cases and sub groups (including their configuration functions), and finally the <c>end_per_group</c> function. Also if particular test cases in a group are specified, <c>init_per_group</c> and <c>end_per_group</c> for the group in question are called. If a group which is defined (in <c>Suite:group/0</c>) to - be a sub group of another group, is specified (or particular test + be a sub group of another group, is specified (or if particular test cases of a sub group are), Common Test will call the configuration functions for the top level groups as well as for the sub group in question (making it possible to pass configuration data all the way from <c>init_per_suite</c> down to the test cases in the sub group).</p> + <p>The test specification utilizes the same mechanism for specifying + test case groups by means of names and paths, as explained in the + <seealso marker="run_test_chapter#group_execution">Group Execution</seealso> + section above, with the addition of the <c>GroupSpec</c> element + described next.</p> + <p>The <c>GroupSpec</c> element makes it possible to specify + group execution properties that will override those in the + group definition (i.e. in <c>groups/0</c>). Execution properties for + sub-groups may be overridden as well. This feature makes it possible to + change properties of groups at the time of execution, + without even having to edit the test suite. The very same + feature is available for <c>group</c> elements in the <c>Suite:all/0</c> + list. Therefore, more detailed documentation, and examples, can be + found in the <seealso marker="write_test_chapter#test_case_groups"> + Test case groups</seealso> chapter.</p> <p>Below is the test specification syntax. Test specifications can be used to run tests both in a single test host environment and in a distributed Common Test environment (Large Scale - Testing). The node parameters in the init term are only + Testing). The node parameters in the <c>init</c> term are only relevant in the latter (see the <seealso marker="ct_master_chapter#test_specifications">Large - Scale Testing</seealso> chapter for information). For details on - the event_handler term, see the + Scale Testing</seealso> chapter for information). For more information + about the various terms, please see the corresponding sections in the + User's Guide, such as e.g. the + <seealso marker="run_test_chapter#ct_run"><c>ct_run</c> + program</seealso> for an overview of available start flags + (since most flags have a corresponding configuration term), and + more detailed explanation of e.g. + <seealso marker="write_test_chapter#logging">Logging</seealso> + (for the <c>verbosity</c>, <c>stylesheet</c> and <c>basic_html</c> terms), + <seealso marker="config_file_chapter#top">External Configuration Data</seealso> + (for the <c>config</c> and <c>userconfig</c> terms), <seealso marker="event_handler_chapter#event_handling">Event - Handling</seealso> chapter.</p> + Handling</seealso> (for the <c>event_handler</c> term), + <seealso marker="ct_hooks_chapter#installing">Common Test Hooks</seealso> + (for the <c>ct_hooks</c> term), etc.</p> <p>Config terms:</p> <pre> + {merge_tests, Bool}. + + {define, Constant, Value}. + {node, NodeAlias, Node}. {init, InitOptions}. @@ -430,6 +640,15 @@ {label, Label}. {label, NodeRefs, Label}. + {verbosity, VerbosityLevels}. + {verbosity, NodeRefs, VerbosityLevels}. + + {stylesheet, CSSFile}. + {stylesheet, NodeRefs, CSSFile}. + + {silent_connections, ConnTypes}. + {silent_connections, NodeRefs, ConnTypes}. + {multiply_timetraps, N}. {multiply_timetraps, NodeRefs, N}. @@ -442,18 +661,25 @@ {include, IncludeDirs}. {include, NodeRefs, IncludeDirs}. + {auto_compile, Bool}, + {auto_compile, NodeRefs, Bool}, + {config, ConfigFiles}. + {config, ConfigDir, ConfigBaseNames}. {config, NodeRefs, ConfigFiles}. + {config, NodeRefs, ConfigDir, ConfigBaseNames}. {userconfig, {CallbackModule, ConfigStrings}}. {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}. - {alias, DirAlias, Dir}. - - {merge_tests, Bool}. - {logdir, LogDir}. {logdir, NodeRefs, LogDir}. + + {logopts, LogOpts}. + {logopts, NodeRefs, LogOpts}. + + {create_priv_dir, PrivDirOption}. + {create_priv_dir, NodeRefs, PrivDirOption}. {event_handler, EventHandlers}. {event_handler, NodeRefs, EventHandlers}. @@ -462,77 +688,179 @@ {ct_hooks, CTHModules}. {ct_hooks, NodeRefs, CTHModules}. - </pre> + + {enable_builtin_hooks, Bool}. + + {basic_html, Bool}. + {basic_html, NodeRefs, Bool}. + + {release_shell, Bool}.</pre> + <p>Test terms:</p> <pre> - {suites, DirRef, Suites}. - {suites, NodeRefs, DirRef, Suites}. + {suites, Dir, Suites}. + {suites, NodeRefs, Dir, Suites}. - {groups, DirRef, Suite, Groups}. - {groups, NodeRefsDirRef, Suite, Groups}. + {groups, Dir, Suite, Groups}. + {groups, NodeRefs, Dir, Suite, Groups}. + + {groups, Dir, Suite, Groups, {cases,Cases}}. + {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}. - {groups, DirRef, Suite, Group, {cases,Cases}}. - {groups, NodeRefsDirRef, Suite, Group, {cases,Cases}}. + {cases, Dir, Suite, Cases}. + {cases, NodeRefs, Dir, Suite, Cases}. - {cases, DirRef, Suite, Cases}. - {cases, NodeRefs, DirRef, Suite, Cases}. + {skip_suites, Dir, Suites, Comment}. + {skip_suites, NodeRefs, Dir, Suites, Comment}. - {skip_suites, DirRef, Suites, Comment}. - {skip_suites, NodeRefs, DirRef, Suites, Comment}. + {skip_groups, Dir, Suite, GroupNames, Comment}. + {skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}. - {skip_cases, DirRef, Suite, Cases, Comment}. - {skip_cases, NodeRefs, DirRef, Suite, Cases, Comment}. - </pre> + {skip_cases, Dir, Suite, Cases, Comment}. + {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}.</pre> + <p>Types:</p> <pre> - NodeAlias = atom() - InitOptions = term() - Node = node() - NodeRef = NodeAlias | Node | master - NodeRefs = all_nodes | [NodeRef] | NodeRef - N = integer() - Bool = true | false - CoverSpecFile = string() - IncludeDirs = string() | [string()] - ConfigFiles = string() | [string()] - DirAlias = atom() - Dir = string() - LogDir = string() - EventHandlers = atom() | [atom()] - InitArgs = [term()] - CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}] - CTHModule = atom() - CTHInitArgs = term() - DirRef = DirAlias | Dir - Suites = atom() | [atom()] | all - Suite = atom() - Groups = atom() | [atom()] | all - Group = atom() - Cases = atom() | [atom()] | all - Comment = string() | "" - </pre> - <p>Example:</p> + Bool = true | false + Constant = atom() + Value = term() + NodeAlias = atom() + Node = node() + NodeRef = NodeAlias | Node | master + NodeRefs = all_nodes | [NodeRef] | NodeRef + InitOptions = term() + Label = atom() | string() + VerbosityLevels = integer() | [{Category,integer()}] + Category = atom() + CSSFile = string() + ConnTypes = all | [atom()] + N = integer() + CoverSpecFile = string() + IncludeDirs = string() | [string()] + ConfigFiles = string() | [string()] + ConfigDir = string() + ConfigBaseNames = string() | [string()] + CallbackModule = atom() + ConfigStrings = string() | [string()] + LogDir = string() + LogOpts = [term()] + PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc + EventHandlers = atom() | [atom()] + InitArgs = [term()] + CTHModules = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}] + CTHModule = atom() + CTHInitArgs = term() + Dir = string() + Suites = atom() | [atom()] | all + Suite = atom() + Groups = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all + GroupPath = [GroupName] + GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec} + GroupName = atom() + GroupNames = GroupName | [GroupName] + Cases = atom() | [atom()] | all + Comment = string() | ""</pre> + + <p>The difference between the <c>config</c> terms above, is that with + <c>ConfigDir</c>, <c>ConfigBaseNames</c> is a list of base names, + i.e. without directory paths. <c>ConfigFiles</c> must be full names, + including paths. E.g, these two terms have the same meaning:</p> + <pre> + {config, ["/home/testuser/tests/config/nodeA.cfg", + "/home/testuser/tests/config/nodeB.cfg"]}. + + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.</pre> + + <note><p>Any relative paths specified in the test specification, will be + relative to the directory which contains the test specification file, if + <c>ct_run -spec TestSpecFile ...</c> or + <c>ct:run:test([{spec,TestSpecFile},...])</c> + executes the test. The path will be relative to the top level log directory, if + <c>ct:run:testspec(TestSpec)</c> executes the test.</p></note> + + <p>The <c>define</c> term introduces a constant, which is used to + replace the name <c>Constant</c> with <c>Value</c>, wherever it's found in + the test specification. This replacement happens during an initial iteration + through the test specification. Constants may be used anywhere in the test + specification, e.g. in arbitrary lists and tuples, and even in strings + and inside the value part of other constant definitions! A constant can + also be part of a node name, but that is the only place where a constant + can be part of an atom.</p> + + <note><p>For the sake of readability, the name of the constant must always + begin with an upper case letter, or a <c>$</c>, <c>?</c>, or <c>_</c>. + This also means that it must always be single quoted (obviously, since + the constant name is actually an atom, not text).</p></note> + + <p>The main benefit of constants is that they can be used to reduce the size + (and avoid repetition) of long strings, such as file paths. Compare these + terms:</p> + <pre> - {logdir, "/home/test/logs"}. + %% 1a. no constant + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "/home/testuser/tests/suites", all}. - {config, "/home/test/t1/cfg/config.cfg"}. - {config, "/home/test/t2/cfg/config.cfg"}. - {config, "/home/test/t3/cfg/config.cfg"}. + %% 1b. with constant + {define, 'TESTDIR', "/home/testuser/tests"}. + {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "'TESTDIR'/suites", all}. + + %% 2a. no constants + {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}. + + %% 2b. with constants + {define, 'NODE', testnode}. + {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}. + {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.</pre> + + <p>Constants make the test specification term <c>alias</c>, in previous + versions of Common Test, redundant. This term has been deprecated but will + remain supported in upcoming Common Test releases. Replacing <c>alias</c> + terms with <c>define</c> is strongly recommended though! Here's an example + of such a replacement:</p> + + <pre> + %% using the old alias term + {config, "/home/testuser/tests/config/nodeA.cfg"}. + {alias, suite_dir, "/home/testuser/tests/suites"}. + {groups, suite_dir, x_SUITE, group1}. + + %% replacing with constants + {define, 'TestDir', "/home/testuser/tests"}. + {define, 'CfgDir', "'TestDir'/config"}. + {define, 'SuiteDir', "'TestDir'/suites"}. + {config, 'CfgDir', "nodeA.cfg"}. + {groups, 'SuiteDir', x_SUITE, group1}.</pre> + + <p>Actually, constants could well replace the <c>node</c> term too, but + this still has declarative value, mainly when used in combination + with <c>NodeRefs == all_nodes</c> (see types above).</p> + + <p>Here follows a simple test specification example:</p> + <pre> + {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + + {logdir, "'Top'/logs"}. - {alias, t1, "/home/test/t1"}. - {alias, t2, "/home/test/t2"}. - {alias, t3, "/home/test/t3"}. + {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}. - {suites, t1, all}. - {skip_suites, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}. - {skip_cases, t1, t1A_SUITE, [test3,test4], "Irrelevant"}. - {skip_cases, t1, t1C_SUITE, [test1], "Ignore"}. + {suites, 'T1', all}. + {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}. - {suites, t2, [t2B_SUITE,t2C_SUITE]}. - {cases, t2, t2A_SUITE, [test4,test1,test7]}. + {suites, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, 'T2', t2A_SUITE, [test4,test1,test7]}. - {skip_suites, t3, all, "Not implemented"}. - </pre> + {skip_suites, 'T3', all, "Not implemented"}.</pre> + <p>The example specifies the following:</p> <list> <item>The specified logdir directory will be used for storing @@ -540,8 +868,6 @@ date and time).</item> <item>The variables in the specified test system config files will be imported for the test.</item> - <item>Aliases are given for three test system directories. The suites in - this example are stored in "/home/test/tX/test".</item> <item>The first test to run includes all suites for system t1. Excluded from the test are however the t1B and t1D suites. Also test cases test3 and test4 in t1A as well as the test1 case in t1C are excluded from @@ -552,9 +878,9 @@ <item>Lastly, all suites for systems t3 are to be completely skipped and this should be explicitly noted in the log files.</item> </list> - <p>It is possible to specify initialization options for nodes defined in the - test specification. Currently, there are options to start the node and/or to - evaluate any function on the node. + <p>With the <c>init</c> term it's possible to specify initialization options + for nodes defined in the test specification. Currently, there are options + to start the node and/or to evaluate any function on the node. See the <seealso marker="ct_master_chapter#ct_slave">Automatic startup of the test target nodes</seealso> chapter for details.</p> <p>It is possible for the user to provide a test specification that @@ -563,11 +889,49 @@ <c>ct_run</c>. This forces Common Test to ignore unrecognizable terms. Note that in this mode, Common Test is not able to check the specification for errors as efficiently as if the scanner runs in default mode. - If <c>ct:run_test/1</c> is used for starting the tests, the relaxed scanner + If <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> is used for starting the tests, the relaxed scanner mode is enabled by means of the tuple: <c>{allow_user_terms,true}</c></p> </section> + + <section> + <title>Running tests from the Web based GUI</title> + + <p>The web based GUI, VTS, is started with the + <c><seealso marker="run_test_chapter#ct_run">ct_run</seealso></c> + program. From the GUI you can load config files, and select + directories, suites and cases to run. You can also state the + config files, directories, suites and cases on the command line + when starting the web based GUI. + </p> + + <list> + <item><c>ct_run -vts</c></item> + <item><c><![CDATA[ct_run -vts -config <configfilename>]]></c></item> + <item><c><![CDATA[ct_run -vts -config <configfilename> -suite <suitewithfullpath> + -case <casename>]]></c></item> + </list> + + <p>From the GUI you can run tests and view the result and the logs. + </p> + + <p>Note that <c>ct_run -vts</c> will try to open the Common Test start + page in an existing web browser window or start the browser if it is + not running. Which browser should be started may be specified with + the browser start command option:</p> + <p><c><![CDATA[ct_run -vts -browser <browser_start_cmd>]]></c></p> + <p>Example:</p> + <p><c><![CDATA[$ ct_run -vts -browser 'firefox&']]></c></p> + <p>Note that the browser must run as a separate OS process or VTS will hang!</p> + <p>If no specific browser start command is specified, Firefox will + be the default browser on Unix platforms and Internet Explorer on Windows. + If Common Test fails to start a browser automatically, or <c>'none'</c> is + specified as the value for -browser (i.e. <c>-browser none</c>), start your + favourite browser manually and type in the URL that Common Test + displays in the shell.</p> + </section> <section> + <marker id="log_files"></marker> <title>Log files</title> <p>As the execution of the test suites proceed, events are logged in @@ -640,10 +1004,25 @@ to each individual test case log file for quick viewing with an HTML browser.</p> - <p>The minor log file contain full details of every single test - case, each one in a separate file. This way the files should - be easy to compare with previous test runs, even if the set of - test cases change.</p> + <p>The minor log files contain full details of every single test + case, each one in a separate file. This way, it should be + straightforward to compare the latest results to that of previous + test runs, even if the set of test cases changes. If SASL is running, + its logs will also be printed to the current minor log file by the + <seealso marker="common_test:ct_hooks_chapter#builtin_cths"> + cth_log_redirect built-in hook</seealso>. + </p> + + <p>The full name of the minor log file (i.e. the name of the file + including the absolute directory path) can be read during execution + of the test case. It comes as value in the tuple + <c>{tc_logfile,LogFileName}</c> in the <c>Config</c> list (which means it + can also be read by a pre- or post Common Test hook function). Also, + at the start of a test case, this data is sent with an event + to any installed event handler. Please see the + <seealso marker="event_handler_chapter#event_handling">Event Handling</seealso> + chapter for details. + </p> <p>Which information goes where is user configurable via the test server controller. Three threshold values determine what @@ -680,13 +1059,33 @@ <p>instead of each <c>x</c> printed on a new line, which is the default behaviour.</p> </section> + <section> + <marker id="table_sorting"></marker> + <title>Sorting HTML table columns</title> + <p>By clicking the name in the column header of any table (e.g. "Ok", "Case", "Time", etc), + the table rows are sorted in whatever order makes sense for the type of value (e.g. + numerical for "Ok" or "Time", and alphabetical for "Case"). The sorting is performed + by means of JavaScript code, automatically inserted into the HTML log files. Common Test + uses the <url href="http://jquery.com">jQuery</url> library and the + <url href="http://tablesorter.com">tablesorter</url> plugin, with customized sorting + functions, for this implementation.</p> + </section> </section> <section> <marker id="html_stylesheet"></marker> <title>HTML Style Sheets</title> - <p>Common Test includes the <em>optional</em> feature to use - HTML style sheets (CSS) for customizing user printouts. The + <p>Common Test uses an HTML Style Sheet (CSS file) to control the look of + the HTML log files generated during test runs. If, for some reason, the + log files are not displayed correctly in the browser of your + choice, or you prefer a more primitive ("pre Common Test v1.6") look + of the logs, use the start flag/option:</p> + <pre>basic_html</pre> + <p>This disables the use of Style Sheets, as well as JavaScripts (see + table sorting above).</p> + + <p>Common Test includes an <em>optional</em> feature to allow + user HTML style sheets for customizing printouts. The functions in <c>ct</c> that print to a test case HTML log file (<c>log/3</c> and <c>pal/3</c>) accept <c>Category</c> as first argument. With this argument it's possible to @@ -700,22 +1099,16 @@ look like this:</p> <pre> -<style> - div.ct_internal { background:lightgrey; color:black } - div.default { background:lightgreen; color:black } - div.sys_config { background:blue; color:white } - div.sys_state { background:yellow; color:black } - div.error { background:red; color:white } -</style> - </pre> + div.sys_config { background:blue; color:white } + div.sys_state { background:yellow; color:black } + div.error { background:red; color:white }</pre> <p>To install the CSS file (Common Test inlines the definition in the HTML code), the name may be provided when executing <c>ct_run</c>. Example:</p> <pre> - $ ct_run -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css - </pre> + $ ct_run -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css</pre> <p>Categories in a CSS file installed with the <c>-stylesheet</c> flag are on a global test level in the sense that they can be used in any @@ -736,8 +1129,7 @@ ct:log(sys_state, "Connections: ~p", [ConnectionInfo]), ... ct:pal(error, "Error ~p detected! Info: ~p", [SomeFault,ErrorInfo]), - ct:fail(SomeFault). - </pre> + ct:fail(SomeFault).</pre> <p>If the style sheet is installed as in this example, the categories are private to the suite in question. They can be used by all test cases in the @@ -761,21 +1153,6 @@ <p>The <c>Category</c> argument in the example above may have the value (atom) <c>sys_config</c> (white on blue), <c>sys_state</c> (black on yellow) or <c>error</c> (white on red).</p> - - <p>If the <c>Category</c> argument is not specified, Common Test will - use the CSS selector <c>div.default</c> for the - printout. For this reason a user supplied style sheet must - include this selector. Also the selector - <c>div.ct_internal</c> must be included. Hence a minimal - user style sheet should look like this (which is also the - default style sheet Common Test uses if no user CSS file is - provided):</p> - <pre> - <style> - div.ct_internal { background:lightgrey; color:black } - div.default { background:lightgreen; color:black } - </style> - </pre> </section> <section> @@ -858,75 +1235,82 @@ <section> <marker id="silent_connections"></marker> <title>Silent Connections</title> - <p>The protocol handling processes in Common Test, implemented by ct_telnet, ct_ftp etc, - do verbose printing to the test case logs. This can be switched off by means - of the <c>-silent_connections</c> flag:</p> + <p>The protocol handling processes in Common Test, implemented by ct_telnet, + ct_ssh, ct_ftp etc, do verbose printing to the test case logs. This can be switched off + by means of the <c>-silent_connections</c> flag:</p> <pre> ct_run -silent_connections [conn_types] </pre> - <p>where <c>conn_types</c> specifies <c>telnet, ftp, rpc</c> and/or <c>snmp</c>.</p> + <p>where <c>conn_types</c> specifies <c>ssh, telnet, ftp, rpc</c> and/or <c>snmp</c>.</p> <p>Example:</p> <pre> - ct_run ... -silent_connections telnet ftp</pre> - <p>switches off logging for telnet and ftp connections.</p> + ct_run ... -silent_connections ssh telnet</pre> + <p>switches off logging for ssh and telnet connections.</p> <pre> ct_run ... -silent_connections</pre> <p>switches off logging for all connection types.</p> - <p>Basic and important information such as opening and closing a connection, - fatal communication error and reconnection attempts will always be printed even - if logging has been suppressed for the connection type in question. However, operations - such as sending and receiving data may be performed silently.</p> + <p>Fatal communication error and reconnection attempts will always be printed even + if logging has been suppressed for the connection type in question. However, operations + such as sending and receiving data will be performed silently.</p> <p>It is possible to also specify <c>silent_connections</c> in a test suite. This is accomplished by returning a tuple, <c>{silent_connections,ConnTypes}</c>, in the <c>suite/0</c> or test case info list. If <c>ConnTypes</c> is a list of atoms - (<c>telnet, ftp, rpc</c> and/or <c>snmp</c>), output for any corresponding connections + (<c>ssh, telnet, ftp, rpc</c> and/or <c>snmp</c>), output for any corresponding connections will be suppressed. Full logging is per default enabled for any connection of type not specified in <c>ConnTypes</c>. Hence, if <c>ConnTypes</c> is the empty list, logging is enabled for all connections.</p> - <p>The <c>silent_connections</c> setting returned from a test case info function overrides, - for the test case in question, any setting made with <c>suite/0</c> (which is the setting - used for all cases in the suite). Example:</p> + <p>Example:</p> <pre> -module(my_SUITE). + + suite() -> [..., {silent_connections,[telnet,ssh]}, ...]. + ... - suite() -> [..., {silent_connections,[telnet,ftp]}, ...]. - ... + my_testcase1() -> - [{silent_connections,[ftp]}]. + [{silent_connections,[ssh]}]. + my_testcase1(_) -> - ... + ... + my_testcase2(_) -> - ... + ... </pre> <p>In this example, <c>suite/0</c> tells Common Test to suppress - printouts from telnet and ftp connections. This is valid for + printouts from telnet and ssh connections. This is valid for all test cases. However, <c>my_testcase1/0</c> specifies that - for this test case, only ftp should be silent. The result is + for this test case, only ssh should be silent. The result is that <c>my_testcase1</c> will get telnet info (if any) printed - in the log, but not ftp info. <c>my_testcase2</c> will get no + in the log, but not ssh info. <c>my_testcase2</c> will get no info from either connection printed.</p> - <p>The <c>-silent_connections</c> tag (or - <c>silent_connections</c> tagged tuple in the call to - <c>ct:run_test/1</c>) overrides any settings in the test - suite.</p> + <p><c>silent_connections</c> may also be specified with a term + in a test specification + (see <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso>). Connections provided with the + <c>silent_connections</c> start flag/option, will be merged with + any connections listed in the test specification.</p> + + <p>The <c>silent_connections</c> start flag/option and test + specification term, overrides any settings made by the info functions + inside the test suite.</p> - <p>Note that in the current Common Test version, the + <note><p>Note that in the current Common Test version, the <c>silent_connections</c> feature only works for telnet - connections. Support for other connection types will be added - in future Common Test versions.</p> + and ssh connections! Support for other connection types will be added + in future Common Test versions.</p></note> </section> </chapter> diff --git a/lib/common_test/doc/src/tc_execution.gif b/lib/common_test/doc/src/tc_execution.gif Binary files differnew file mode 100644 index 0000000000..7c89d7be57 --- /dev/null +++ b/lib/common_test/doc/src/tc_execution.gif diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index e35888e68f..248d7de8b6 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2011</year> + <year>2003</year><year>2012</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -29,7 +29,6 @@ <file>write_test_chapter.xml</file> </header> - <section> <marker id="intro"></marker> <title>Support for test suite authors</title> @@ -48,7 +47,7 @@ module for details about these functions.</p> <p>The CT application also includes other modules named - <c><![CDATA[ct_<something>]]></c> that + <c><![CDATA[ct_<component>]]></c> that provide various support, mainly simplified use of communication protocols such as rpc, snmp, ftp, telnet, etc.</p> @@ -63,12 +62,19 @@ function in CT will not be able to locate it (at least not per default). </p> - <p>The <c>ct.hrl</c> header file must be included in all test suite files. + <p>It is also recommended that the <c>ct.hrl</c> header file is included + in all test suite modules. </p> <p>Each test suite module must export the function <c>all/0</c> which returns the list of all test case groups and test cases - in that module. + to be executed in that module. + </p> + + <p>The callback functions that the test suite should implement, and + which will be described in more detail below, are + all listed in the <seealso marker="common_test">common_test + reference manual page</seealso>. </p> </section> @@ -113,6 +119,14 @@ suite will be skipped automatically (so called <em>auto skipped</em>), including <c>end_per_suite</c>. </p> + + <p>Note that if <c>init_per_suite</c> and <c>end_per_suite</c> do not exist + in the suite, Common Test calls dummy functions (with the same names) + instead, so that output generated by hook functions may be saved to the log + files for these dummies + (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test Hooks</seealso> + chapter for more information). + </p> </section> <marker id="per_testcase"/> @@ -159,7 +173,7 @@ </p> <p>The <c>end_per_testcase/2</c> function is called even after a - test case terminates due to a call to <c>ct:abort_current_testcase/1</c>, + test case terminates due to a call to <c><seealso marker="ct#abort_current_testcase-1">ct:abort_current_testcase/1</seealso></c>, or after a timetrap timeout. However, <c>end_per_testcase</c> will then execute on a different process than the test case function, and in this situation, <c>end_per_testcase</c> will @@ -229,7 +243,8 @@ <note><p>The test case function argument <c>Config</c> should not be confused with the information that can be retrieved from - configuration files (using ct:get_config/[1,2]). The Config argument + configuration files (using <c><seealso marker="ct#get_config-1"> + ct:get_config/1/2</seealso></c>). The Config argument should be used for runtime configuration of the test suite and the test cases, while configuration files should typically contain data related to the SUT. These two types of configuration data are handled @@ -288,7 +303,7 @@ <item> <p> Use this to specify arbitrary data related to the testcase. This - data can be retrieved at any time using the <c>ct:userdata/3</c> + data can be retrieved at any time using the <c><seealso marker="ct#userdata-3">ct:userdata/3</seealso></c> utility function. </p> </item> @@ -324,7 +339,8 @@ <pre> testcase2() -> - [{require, unix_telnet, {unix, [telnet, username, password]}}, + [{require, unix_telnet, unix}, + {require, {unix, [telnet, username, password]}}, {default_config, unix, [{telnet, "my_telnet_host"}, {username, "aladdin"}, {password, "sesame"}]}}].</pre> @@ -332,7 +348,8 @@ </taglist> <p>See the <seealso marker="config_file_chapter#require_config_data">Config files</seealso> - chapter and the <c>ct:require/[1,2]</c> function in the + chapter and the <c><seealso marker="ct#require-1"> + ct:require/1/2</seealso></c> function in the <seealso marker="ct">ct</seealso> reference manual for more information about <c>require</c>.</p> @@ -369,10 +386,12 @@ <title>Test suite info function</title> <p>The <c>suite/0</c> function can be used in a test suite - module to set the default values for the <c>timetrap</c> and - <c>require</c> tags. If a test case info function also specifies - any of these tags, the default value is overruled. See above for - more information. + module to e.g. set a default <c>timetrap</c> value and to + <c>require</c> external configuration data. If a test case-, or + group info function also specifies any of the info tags, it + overrides the default values set by <c>suite/0</c>. See the test + case info function above, and group info function below, for more + details. </p> <p>Other options that may be specified with the suite info list are:</p> @@ -450,15 +469,68 @@ <pre> all() -> [testcase1, {group,group1}, testcase2, {group,group2}].</pre> - <p>Properties may be combined so that e.g. if <c>shuffle</c>, - <c>repeat_until_any_fail</c> and <c>sequence</c> are all specified, the test - cases in the group will be executed repeatedly and in random order until - a test case fails, when execution is immediately stopped and the rest of - the cases skipped.</p> + <p>It is also possible to specify execution properties with a group + tuple in <c>all/0</c>: <c>{group,GroupName,Properties}</c>. These + properties will override those specified in the group definition (see + <c>groups/0</c> above). This way, it's possible to run the same set of tests, + but with different properties, without having to make copies of the group + definition in question.</p> + + <p>If a group contains sub-groups, the execution properties for these may + also be specified in the group tuple: + <c>{group,GroupName,Properties,SubGroups}</c>, where <c>SubGroups</c> + is a list of tuples, <c>{GroupName,Properties}</c>, or + <c>{GroupName,Properties,SubGroups}</c>, representing the sub-groups. + Any sub-groups defined in <c>group/0</c> for a group, that are not specified + in the <c>SubGroups</c> list, will simply execute with their pre-defined + properties.</p> + + <p>Example:</p> + <pre> + groups() -> {tests1, [], [{tests2, [], [t2a,t2b]}, + {tests3, [], [t31,t3b]}]}.</pre> + <p>To execute group 'tests1' twice with different properties for 'tests2' + each time:</p> + <pre> + all() -> + [{group, tests1, default, [{tests2, [parallel]}]}, + {group, tests1, default, [{tests2, [shuffle,{repeat,10}]}]}].</pre> + <p>Note that this is equivalent to this specification:</p> + <pre> + all() -> + [{group, tests1, default, [{tests2, [parallel]}, + {tests3, default}]}, + {group, tests1, default, [{tests2, [shuffle,{repeat,10}]}, + {tests3, default}]}].</pre> + <p>The value <c>default</c> states that the pre-defined properties + should be used.</p> + <p>Here's an example of how to override properties in a scenario + with deeply nested groups:</p> + <pre> + groups() -> + [{tests1, [], [{group, tests2}]}, + {tests2, [], [{group, tests3}]}, + {tests3, [{repeat,2}], [t3a,t3b,t3c]}]. + + all() -> + [{group, tests1, default, + [{tests2, default, + [{tests3, [parallel,{repeat,100}]}]}]}].</pre> + + <p>The syntax described above may also be used in Test Specifications + in order to change properties of groups at the time of execution, + without even having to edit the test suite (please see the + <seealso marker="run_test_chapter#test_specifications">Test + Specifications</seealso> chapter for more info).</p> + + <p>As illustrated above, properties may be combined. If e.g. + <c>shuffle</c>, <c>repeat_until_any_fail</c> and <c>sequence</c> + are all specified, the test cases in the group will be executed + repeatedly, and in random order, until a test case fails. Then + execution is immediately stopped and the rest of the cases skipped.</p> <p>Before execution of a group begins, the configuration function - <c>init_per_group(GroupName, Config)</c> is called (the function is - mandatory if one or more test case groups are defined). The list of tuples + <c>init_per_group(GroupName, Config)</c> is called. The list of tuples returned from this function is passed to the test cases in the usual manner by means of the <c>Config</c> argument. <c>init_per_group/2</c> is meant to be used for initializations common for the test cases in the @@ -466,6 +538,14 @@ <c>end_per_group(GroupName, Config</c> function is called. This function is meant to be used for cleaning up after <c>init_per_group/2</c>.</p> + <p>Whenever a group is executed, if <c>init_per_group</c> and + <c>end_per_group</c> do not exist in the suite, Common Test calls + dummy functions (with the same names) instead. Output generated by + hook functions will be saved to the log files for these dummies + (see the <seealso marker="ct_hooks_chapter#manipulating">Common Test + Hooks</seealso> chapter for more information). + </p> + <note><p><c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> are always called for each individual test case, no matter if the case belongs to a group or not.</p></note> @@ -555,6 +635,25 @@ </section> <section> + <title>Parallel test cases and IO</title> + <p>A parallel test case has a private IO server as its group leader. + (Please see the Erlang Run-Time System Application documentation for + a description of the group leader concept). The + central IO server process that handles the output from regular test + cases and configuration functions, does not respond to IO messages + during execution of parallel groups. This is important to understand + in order to avoid certain traps, like this one:</p> + <p>If a process, <c>P</c>, is spawned during execution of e.g. + <c>init_per_suite/1</c>, it will inherit the group leader of the + <c>init_per_suite</c> process. This group leader is the central IO server + process mentioned above. If, at a later time, <em>during parallel test case + execution</em>, some event triggers process <c>P</c> to call + <c>io:format/1/2</c>, that call will never return (since the group leader + is in a non-responsive state) and cause <c>P</c> to hang. + </p> + </section> + + <section> <title>Repeated groups</title> <marker id="repeated_groups"></marker> <p>A test case group may be repeated a certain number of times @@ -641,10 +740,55 @@ </section> <section> + <marker id="group_info"></marker> + <title>Group info function</title> + + <p>The test case group info function, <c>group(GroupName)</c>, + serves the same purpose as the suite- and test case info + functions previously described in this chapter. The scope for + the group info, however, is all test cases and sub-groups in the + group in question (<c>GroupName</c>).</p> + <p>Example:</p> + <pre> + group(connection_tests) -> + [{require,login_data}, + {timetrap,1000}].</pre> + + <p>The group info properties override those set with the + suite info function, and may in turn be overridden by test + case info properties. Please see the test case info + function above for a list of valid info properties and more + general information.</p> + </section> + + <section> + <title>Info functions for init- and end-configuration</title> + <p>It is possible to use info functions also for the <c>init_per_suite</c>, + <c>end_per_suite</c>, <c>init_per_group</c>, and <c>end_per_group</c> + functions, and it works the same way as with info functions + for test cases (see above). This is useful e.g. for setting + timetraps and requiring external configuration data relevant + only for the configuration function in question (without + affecting properties set for groups and test cases in the suite).</p> + + <p>The info function <c>init/end_per_suite()</c> is called for + <c>init/end_per_suite(Config)</c>, and info function + <c>init/end_per_group(GroupName)</c> is called for + <c>init/end_per_group(GroupName,Config)</c>. Info functions + can not be used with <c>init/end_per_testcase(TestCase, Config)</c>, + however, since these configuration functions execute on the test case process + and will use the same properties as the test case (i.e. the properties + set by the test case info function, <c>TestCase()</c>). Please see the test case + info function above for a list of valid info properties and more + general information. + </p> + </section> + + <section> <marker id="data_priv_dir"></marker> <title>Data and Private Directories</title> - <p>The data directory (<c>data_dir</c>) is the directory where the + <p>The data directory, <c>data_dir</c>, is the directory where the test module has its own files needed for the testing. The name of the <c>data_dir</c> is the the name of the test suite followed by <c>"_data"</c>. For example, @@ -668,12 +812,39 @@ </p> --> <p> - The <c>priv_dir</c> is the test suite's private directory. This - directory should be used when a test case needs to write to - files. The name of the private directory is generated by the test - server, which also creates the directory. + <c>priv_dir</c> is the private directory for the test cases. + This directory may be used whenever a test case (or configuration function) + needs to write something to file. The name of the private directory is + generated by Common Test, which also creates the directory. </p> - + <p>By default, Common Test creates one central private directory + per test run that all test cases share. This may not always be suitable, + especially if the same test cases are executed multiple times during + a test run (e.g. if they belong to a test case group with repeat + property), and there's a risk that files in the private directory get + overwritten. Under these circumstances, it's possible to configure + Common Test to create one dedicated private directory per + test case and execution instead. This is accomplished by means of + the flag/option: <c>create_priv_dir</c> (to be used with the + <c>ct_run</c> program, the <c><seealso marker="ct#run_test-1">ct:run_test/1</seealso></c> function, or + as test specification term). There are three possible values + for this option: + <list> + <item><c>auto_per_run</c></item> + <item><c>auto_per_tc</c></item> + <item><c>manual_per_tc</c></item> + </list> + The first value indicates the default priv_dir behaviour, i.e. + one private directory created per test run. The two latter + values tell Common Test to generate a unique test directory name + per test case and execution. If the auto version is used, <em>all</em> + private directories will be created automatically. This can obviously + become very inefficient for test runs with many test cases and/or + repetitions. Therefore, in case the manual version is instead used, the + test case must tell Common Test to create priv_dir when it needs it. + It does this by calling the function <c><seealso marker="ct#make_priv_dir-0">ct:make_priv_dir/0</seealso></c>. + </p> + <note><p>You should not depend on current working directory for reading and writing data files since this is not portable. All scratch files are to be written in the <c>priv_dir</c> and all @@ -704,37 +875,159 @@ <marker id="timetraps"></marker> <title>Timetrap timeouts</title> <p>The default time limit for a test case is 30 minutes, unless a - <c>timetrap</c> is specified either by the suite info function - or a test case info function. The timetrap timeout value defined - in <c>suite/0</c> is the value that will be used for each test case - in the suite (as well as for the configuration functions - <c>init_per_suite/1</c> and <c>end_per_suite</c>). A timetrap timeout - value set with the test case info function will override the value set - by <c>suite/0</c>, but only for that particular test case.</p> - <p>It is also possible to set/reset a timetrap during test case (or - configuration function) execution. This is done by calling - <c>ct:timetrap/1</c>. This function will cancel the current timetrap - and start a new one.</p> + <c>timetrap</c> is specified either by the suite-, group-, + or test case info function. The timetrap timeout value defined by + <c>suite/0</c> is the value that will be used for each test case + in the suite (as well as for the configuration functions + <c>init_per_suite/1</c>, <c>end_per_suite/1</c>, <c>init_per_group/2</c>, + and <c>end_per_group/2</c>). A timetrap value defined by + <c>group(GroupName)</c> overrides one defined by <c>suite()</c> + and will be used for each test case in group <c>GroupName</c>, and any + of its sub-groups. If a timetrap value is defined by <c>group/1</c> + for a sub-group, it overrides that of its higher level groups. Timetrap + values set by individual test cases (by means of the test case info + function) override both group- and suite- level timetraps.</p> + + <p>It is also possible to dynamically set/reset a timetrap during the + excution of a test case, or configuration function. This is done by calling + <c><seealso marker="ct#timetrap-1">ct:timetrap/1</seealso></c>. This function cancels the current timetrap + and starts a new one (that stays active until timeout, or end of the + current function).</p> + <p>Timetrap values can be extended with a multiplier value specified at - startup with the <c>multiply_timetraps</c> option. It is also possible - to let Test Server decide to scale up timetrap timeout values - automatically, e.g. if tools such as cover or trace are running during - the test. This feature is disabled by default and can be enabled with - the <c>scale_timetraps</c> start option.</p> + startup with the <c>multiply_timetraps</c> option. It is also possible + to let the test server decide to scale up timetrap timeout values + automatically, e.g. if tools such as cover or trace are running during + the test. This feature is disabled by default and can be enabled with + the <c>scale_timetraps</c> start option.</p> + <p>If a test case needs to suspend itself for a time that also gets - multipled by <c>multiply_timetraps</c>, and possibly scaled up if - <c>scale_timetraps</c> is enabled, the function <c>ct:sleep/1</c> - may be called.</p> - <p>A function (<c>fun</c> or <c>MFA</c>) may be specified as timetrap value - in the suite- and test case info function, e.g:</p> - <p><c>{timetrap,{test_utils,get_timetrap_value,[?MODULE,system_start]}}</c></p> - <p>The function will be called initially by Common Test (before execution - of the suite or the test case) and must return a time value such as an - integer (millisec), or a <c>{SecMinOrHourTag,Time}</c> tuple. More - information can be found in the <c>common_test</c> reference manual.</p> + multipled by <c>multiply_timetraps</c> (and possibly also scaled up if + <c>scale_timetraps</c> is enabled), the function <c><seealso marker="ct#sleep-1">ct:sleep/1</seealso></c> + may be used (instead of e.g. <c>timer:sleep/1</c>).</p> + + <p>A function (<c>fun/0</c> or <c>MFA</c>) may be specified as + timetrap value in the suite-, group- and test case info function, as + well as argument to the <c><seealso marker="ct#timetrap-1">ct:timetrap/1</seealso></c> function. Examples:</p> + + <p><c>{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}</c></p> + <p><c>ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)</c></p> + + <p>The user timetrap function may be used for two things:</p> + <list> + <item>To act as a timetrap - the timeout is triggered when the + function returns.</item> + <item>To return a timetrap time value (other than a function).</item> + </list> + <p>Before execution of the timetrap function (which is performed + on a parallel, dedicated timetrap process), Common Test cancels + any previously set timer for the test case or configuration function. + When the timetrap function returns, the timeout is triggered, <em>unless</em> + the return value is a valid timetrap time, such as an integer, + or a <c>{SecMinOrHourTag,Time}</c> tuple (see the + <seealso marker="common_test">common_test reference manual</seealso> for + details). If a time value is returned, a new timetrap is started + to generate a timeout after the specified time.</p> + + <p>The user timetrap function may of course return a time value after a delay, + and if so, the effective timetrap time is the delay time <em>plus</em> the + returned time.</p> </section> <section> + <marker id="logging"></marker> + <title>Logging - categories and verbosity levels</title> + <p>Common Test provides three main functions for printing strings:</p> + <list> + <item><c>ct:log(Category, Importance, Format, Args)</c></item> + <item><c>ct:print(Category, Importance, Format, Args)</c></item> + <item><c>ct:pal(Category, Importance, Format, Args)</c></item> + </list> + <p>The <c>log/1/2/3/4</c> function will print a string to the test case + log file. The <c>print/1/2/3/4</c> function will print the string to screen, + and the <c>pal/1/2/3/4</c> function will print the same string both to file and + screen. (The functions are documented in the <c>ct</c> reference manual).</p> + + <p>The optional <c>Category</c> argument may be used to categorize the + log printout, and categories can be used for two things:</p> + <list> + <item>To compare the importance of the printout to a specific + verbosity level, and</item> + <item>to format the printout according to a user specific HTML + Style Sheet (CSS).</item> + </list> + + <p>The <c>Importance</c> argument specifies a level of importance + which, compared to a verbosity level (general and/or set per category), + determines if the printout should be visible or not. <c>Importance</c> + is an arbitrary integer in the range 0..99. Pre-defined constants + exist in the <c>ct.hrl</c> header file. The default importance level, + <c>?STD_IMPORTANCE</c> (used if the <c>Importance</c> argument is not + provided), is 50. This is also the importance used for standard IO, e.g. + from printouts made with <c>io:format/2</c>, <c>io:put_chars/1</c>, etc.</p> + + <p><c>Importance</c> is compared to a verbosity level set by means of the + <c>verbosity</c> start flag/option. The verbosity level can be set per + category and/or generally. The default verbosity level, <c>?STD_VERBOSITY</c>, + is 50, i.e. all standard IO gets printed. If a lower verbosity level is set, + standard IO printouts will be ignored. Common Test performs the following test:</p> + <pre>Importance >= (100-VerbosityLevel)</pre> + <p>This also means that verbosity level 0 effectively turns all logging off + (with the exception of printouts made by Common Test itself).</p> + + <p>The general verbosity level is not associated with any particular + category. This level sets the threshold for the standard IO printouts, + uncategorized <c>ct:log/print/pal</c> printouts, as well as + printouts for categories with undefined verbosity level.</p> + + <p>Example:</p> + <pre> + + Some printouts during test case execution: + + io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]), + ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]), + ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]), + ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]), + ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]), + ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]), + + If starting the test without specifying any verbosity levels: + + $ ct_run ... + + the following gets printed: + + 1. Standard IO, importance = 50 + 2. Uncategorized, importance = 50 + 3. Categorized info, importance = 50 + 5. Categorized error, importance = 75 + 6. Categorized error, importance = 99 + + If starting the test with: + + $ ct_run -verbosity 1 and info 75 + + the following gets printed: + + 3. Categorized info, importance = 50 + 4. Categorized info, importance = 25 + 6. Categorized error, importance = 99 + </pre> + + <p>How categories can be mapped to CSS tags is documented in the + <seealso marker="run_test_chapter#html_stylesheet">Running Tests</seealso> + chapter.</p> + + <p>The <c>Format</c> and <c>Args</c> arguments in <c>ct:log/print/pal</c> are + always passed on to the <c>io:format/3</c> function in <c>stdlib</c> + (please see the <c>io</c> manual page for details).</p> + + <p>For more information about log files, please see the + <seealso marker="run_test_chapter#log_files">Running Tests</seealso> chapter.</p> + </section> + + <section> <title>Illegal dependencies</title> <p>Even though it is highly efficient to write test suites with @@ -744,7 +1037,6 @@ Erlang/OTP test suites.</p> <list> - <item>Depending on current directory, and writing there:<br></br> <p>This is a common error in test suites. It is assumed that @@ -756,19 +1048,10 @@ </p> </item> - <item>Depending on the Clearcase (file version control system) - paths and files:<br></br> - - <p>The test suites are stored in Clearcase but are not - (necessarily) run within this environment. The directory - structure may vary from test run to test run. - </p> - </item> - <item>Depending on execution order:<br></br> - <p>During development of test suites, no assumption should be made - (preferrably) about the execution order of the test cases or suites. + <p>During development of test suites, no assumption should preferrably + be made about the execution order of the test cases or suites. E.g. a test case should not assume that a server it depends on, has already been started by a previous test case. There are several reasons for this: |
