The ct module provides the main interface for writing test cases. This includes for example, the following:

For details about these functions, see module ct.

The Common Test application also includes other modules named ]]>, which provide various support, mainly simplified use of communication protocols such as RPC, SNMP, FTP, Telnet, and others.

A test suite is an ordinary Erlang module that contains test cases. It is recommended that the module has a name on the form *_SUITE.erl. Otherwise, the directory and auto compilation function in Common Test cannot locate it (at least not by default).

It is also recommended that the ct.hrl header file is included in all test suite modules.

Each test suite module must export function all/0, which returns the list of all test case groups and test cases to be executed in that module.

The callback functions to be implemented by the test suite are all listed in module common_test . They are also described in more detail later in this User's Guide.

Each test suite module can contain the optional configuration functions init_per_suite/1 and end_per_suite/1. If the init function is defined, so must the end function be.

If init_per_suite exists, it is called initially before the test cases are executed. It typically contains initializations common for all test cases in the suite, which are only to be performed once. init_per_suite is recommended for setting up and verifying state and environment on the System Under Test (SUT) or the Common Test host node, or both, so that the test cases in the suite executes correctly. The following are examples of initial configuration operations:

end_per_suite is called as the final stage of the test suite execution (after the last test case has finished). The function is meant to be used for cleaning up after init_per_suite.

init_per_suite and end_per_suite execute on dedicated Erlang processes, just like the test cases do. The result of these functions is however not included in the test run statistics of successful, failed, and skipped cases.

The argument to init_per_suite is Config, that is, the same key-value list of runtime configuration data that each test case takes as input argument. init_per_suite can modify this parameter with information that the test cases need. The possibly modified Config list is the return value of the function.

If init_per_suite fails, all test cases in the test suite are skipped automatically (so called auto skipped), including end_per_suite.

Notice that if init_per_suite and end_per_suite do not exist in the suite, Common Test calls dummy functions (with the same names) instead, so that output generated by hook functions can be saved to the log files for these dummies. For details, see Common Test Hooks.

The smallest unit that the test server is concerned with is a test case. Each test case can test many things, for example, make several calls to the same interface function with different parameters.

The author can choose to put many or few tests into each test case. Some things to keep in mind follows:

The test case function takes one argument, Config, which contains configuration information such as data_dir and priv_dir. (For details about these, see section Data and Private Directories. The value of Config at the time of the call, is the same as the return value from init_per_testcase, mentioned earlier.

As parameter Config is a list of key-value tuples, that is, a data type called a property list, it can be handled by the proplists module. A value can, for example, be searched for and returned with function proplists:get_value/2. Also, or alternatively, the general lists module contains useful functions. Normally, the only operations performed on Config is insert (adding a tuple to the head of the list) and lookup. Common Test provides a simple macro named ?config, which returns a value of an item in Config given the key (exactly like proplists:get_value). Example: PrivDir = ?config(priv_dir, Config).

If the test case function crashes or exits purposely, it is considered failed. If it returns a value (no matter what value), it is considered successful. An exception to this rule is the return value {skip,Reason}. If this tuple is returned, the test case is considered skipped and is logged as such.

If the test case returns the tuple {comment,Comment}, the case is considered successful and Comment is printed in the overview log file. This is equal to calling ct:comment(Comment).

For each test case function there can be an extra function with the same name but without arguments. This is the test case information function. It is expected to return a list of tagged tuples that specifies various properties regarding the test case.

The following tags have special meaning:

 testcase1() -> 
     [{require, ftp},
      {default_config, ftp, [{ftp, "my_ftp_host"},
                             {username, "aladdin"},
                             {password, "sesame"}]}}].

 testcase2() -> 
     [{require, unix_telnet, unix},
      {require, {unix, [telnet, username, password]}},
      {default_config, unix, [{telnet, "my_telnet_host"},
                              {username, "aladdin"},
                              {password, "sesame"}]}}].

For more information about require, see section Requiring and Reading Configuration Data in section External Configuration Data and function ct:require/1/2.

If timetrap or require, or both, is not set specifically for a particular test case, default values specified by function suite/0 are used.

Tags other than the earlier mentioned are ignored by the test server.

An example of a test case information function follows:

 reboot_node() ->
     [
      {timetrap,{seconds,60}},
      {require,interfaces},
      {userdata,
          [{description,"System Upgrade: RpuAddition Normal RebootNode"},
           {fts,"http://someserver.ericsson.se/test_doc4711.pdf"}]}                  
     ].

Function suite/0 can, for example, be used in a test suite module to set a default timetrap value and to require external configuration data. If a test case, or a group information function also specifies any of the information tags, it overrides the default values set by suite/0. For details, see Test Case Information Function and Test Case Groups.

The following options can also be specified with the suite information list:

An example of the suite information function follows:

 suite() ->
     [
      {timetrap,{minutes,10}},
      {require,global_names},
      {userdata,[{info,"This suite tests database transactions."}]},
      {silent_connections,[telnet]},
      {stylesheet,"db_testing.css"}
     ].

A test case group is a set of test cases sharing configuration functions and execution properties. Test case groups are defined by function groups/0 according to the following syntax:

 groups() -> GroupDefs

 Types:

 GroupDefs = [GroupDef]
 GroupDef = {GroupName,Properties,GroupsAndTestCases}
 GroupName = atom()
 GroupsAndTestCases = [GroupDef | {group,GroupName} | TestCase]
 TestCase = atom()

GroupName is the name of the group and must be unique within the test suite module. Groups can be nested, by including a group definition within the GroupsAndTestCases list of another group. Properties is the list of execution properties for the group. The possible values are as follows:

 Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
 Shuffle = shuffle | {shuffle,Seed}
 Seed = {integer(),integer(),integer()}
 RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
              repeat_until_any_ok | repeat_until_any_fail
 N = integer() | forever

Explanations:

Example:

 groups() -> [{group1, [parallel], [test1a,test1b]},
              {group2, [shuffle,sequence], [test2a,test2b,test2c]}].

To specify in which order groups are to be executed (also with respect to test cases that are not part of any group), add tuples on the form {group,GroupName} to the all/0 list.

Example:

 all() -> [testcase1, {group,group1}, testcase2, {group,group2}].

Execution properties with a group tuple in all/0: {group,GroupName,Properties} can also be specified. These properties override those specified in the group definition (see groups/0 earlier). This way, the same set of tests can be run, but with different properties, without having to make copies of the group definition in question.

If a group contains subgroups, the execution properties for these can also be specified in the group tuple: {group,GroupName,Properties,SubGroups} Where, SubGroups is a list of tuples, {GroupName,Properties} or {GroupName,Properties,SubGroups} representing the subgroups. Any subgroups defined in group/0 for a group, that are not specified in the SubGroups list, executes with their predefined properties.

Example:

 groups() -> {tests1, [], [{tests2, [], [t2a,t2b]},
                           {tests3, [], [t31,t3b]}]}.

To execute group tests1 twice with different properties for tests2 each time:

 all() ->
    [{group, tests1, default, [{tests2, [parallel]}]},
     {group, tests1, default, [{tests2, [shuffle,{repeat,10}]}]}].

This is equivalent to the following specification:

 all() ->
    [{group, tests1, default, [{tests2, [parallel]},
                               {tests3, default}]},
     {group, tests1, default, [{tests2, [shuffle,{repeat,10}]},
                               {tests3, default}]}].

Value default states that the predefined properties are to be used.

The following example shows how to override properties in a scenario with deeply nested groups:

 groups() ->
    [{tests1, [], [{group, tests2}]},
     {tests2, [], [{group, tests3}]},
     {tests3, [{repeat,2}], [t3a,t3b,t3c]}].

 all() ->
    [{group, tests1, default, 
      [{tests2, default,
        [{tests3, [parallel,{repeat,100}]}]}]}].

The described syntax can also be used in test specifications to change group properties at the time of execution, without having to edit the test suite. For more information, see section Test Specifications in section Running Tests and Analyzing Results.

As illustrated, properties can be combined. If, for example, shuffle, repeat_until_any_fail, and sequence are all specified, the test cases in the group are executed repeatedly, and in random order, until a test case fails. Then execution is immediately stopped and the remaining cases are skipped.

Before execution of a group begins, the configuration function init_per_group(GroupName, Config) is called. The list of tuples returned from this function is passed to the test cases in the usual manner by argument Config. init_per_group/2 is meant to be used for initializations common for the test cases in the group. After execution of the group is finished, function end_per_group(GroupName, Config) is called. This function is meant to be used for cleaning up after init_per_group/2. If the init function is defined, so must the end function be.

Whenever a group is executed, if init_per_group and end_per_group do not exist in the suite, Common Test calls dummy functions (with the same names) instead. Output generated by hook functions are saved to the log files for these dummies. For more information, see section Manipulating Tests in section Common Test Hooks.

The properties for a group are always printed in the top of the HTML log for init_per_group/2. The total execution time for a group is included at the bottom of the log for end_per_group/2.

Test case groups can be nested so sets of groups can be configured with the same init_per_group/2 and end_per_group/2 functions. Nested groups can be defined by including a group definition, or a group name reference, in the test case list of another group.

Example:

 groups() -> [{group1, [shuffle], [test1a,
                                   {group2, [], [test2a,test2b]},
                                   test1b]},
              {group3, [], [{group,group4},
                            {group,group5}]},
              {group4, [parallel], [test4a,test4b]},
              {group5, [sequence], [test5a,test5b,test5c]}].

In the previous example, if all/0 returns group name references in the order [{group,group1},{group,group3}], the order of the configuration functions and test cases becomes the following (notice that init_per_testcase/2 and end_per_testcase/2: are also always called, but not included in this example for simplification):

 init_per_group(group1, Config) -> Config1  (*)
      test1a(Config1)
      init_per_group(group2, Config1) -> Config2
           test2a(Config2), test2b(Config2)
      end_per_group(group2, Config2)
      test1b(Config1)
 end_per_group(group1, Config1) 
 init_per_group(group3, Config) -> Config3
      init_per_group(group4, Config3) -> Config4
           test4a(Config4), test4b(Config4)  (**)
      end_per_group(group4, Config4)
      init_per_group(group5, Config3) -> Config5
           test5a(Config5), test5b(Config5), test5c(Config5)
      end_per_group(group5, Config5)
 end_per_group(group3, Config3)

(*) The order of test case test1a, test1b, and group2 is undefined, as group1 has a shuffle property.

(**) These cases are not executed in order, but in parallel.

Properties are not inherited from top-level groups to nested subgroups. For instance, in the previous example, the test cases in group2 are not executed in random order (which is the property of group1).

If a group has a parallel property, its test cases are spawned simultaneously and get executed in parallel. However, a test case is not allowed to execute in parallel with end_per_group/2, which means that the time to execute a parallel group is equal to the execution time of the slowest test case in the group. A negative side effect of running test cases in parallel is that the HTML summary pages are not updated with links to the individual test case logs until function end_per_group/2 for the group has finished.

A group nested under a parallel group starts executing in parallel with previous (parallel) test cases (no matter what properties the nested group has). However, as test cases are never executed in parallel with init_per_group/2 or end_per_group/2 of the same group, it is only after a nested group has finished that remaining parallel cases in the previous group become spawned.

A parallel test case has a private I/O server as its group leader. (For a description of the group leader concept, see ERTS). The central I/O server process, which handles the output from regular test cases and configuration functions, does not respond to I/O messages during execution of parallel groups. This is important to understand to avoid certain traps, like the following:

If a process, P, is spawned during execution of, for example, init_per_suite/1, it inherits the group leader of the init_per_suite process. This group leader is the central I/O server process mentioned earlier. If, at a later time, during parallel test case execution, some event triggers process P to call io:format/1/2, that call never returns (as the group leader is in a non-responsive state) and causes P to hang.

A test case group can be repeated a certain number of times (specified by an integer) or indefinitely (specified by forever). The repetition can also be stopped too early if any or all cases fail or succeed, that is, if any of the properties repeat_until_any_fail, repeat_until_any_ok, repeat_until_all_fail, or repeat_until_all_ok is used. If the basic repeat property is used, status of test cases is irrelevant for the repeat operation.

The status of a subgroup can be returned (ok or failed), to affect the execution of the group on the level above. This is accomplished by, in end_per_group/2, looking up the value of tc_group_properties in the Config list and checking the result of the test cases in the group. If status failed is to be returned from the group as a result, end_per_group/2 is to return the value {return_group_result,failed}. The status of a subgroup is taken into account by Common Test when evaluating if execution of a group is to be repeated or not (unless the basic repeat property is used).

The value of tc_group_properties is a list of status tuples, each with the key ok, skipped, and failed. The value of a status tuple is a list with names of test cases that have been executed with the corresponding status as result.

The following is an example of how to return the status from a group:

 end_per_group(_Group, Config) ->
     Status = ?config(tc_group_result, Config),
     case proplists:get_value(failed, Status) of
         [] ->                                   % no failed cases 
             {return_group_result,ok};
         _Failed ->                              % one or more failed
             {return_group_result,failed}
     end.

It is also possible, in end_per_group/2, to check the status of a subgroup (maybe to determine what status the current group is to return). This is as simple as illustrated in the previous example, only the group name is stored in a tuple {group_result,GroupName}, which can be searched for in the status lists.

Example:

 end_per_group(group1, Config) ->
     Status = ?config(tc_group_result, Config),
     Failed = proplists:get_value(failed, Status),
     case lists:member({group_result,group2}, Failed) of
           true ->
               {return_group_result,failed};
           false ->                                                    
               {return_group_result,ok}
     end; 
 ...

The order in which test cases in a group are executed is under normal circumstances the same as the order specified in the test case list in the group definition. With property shuffle set, however, Common Test instead executes the test cases in random order.

You can provide a seed value (a tuple of three integers) with the shuffle property {shuffle,Seed}. This way, the same shuffling order can be created every time the group is executed. If no seed value is specified, Common Test creates a "random" seed for the shuffling operation (using the return value of erlang:timestamp/0). The seed value is always printed to the init_per_group/2 log file so that it can be used to recreate the same execution order in a subsequent test run.

If a subgroup is specified in a group with a shuffle property, the execution order of this subgroup in relation to the test cases (and other subgroups) in the group, is random. The order of the test cases in the subgroup is however not random (unless the subgroup has a shuffle property).

The test case group information function, group(GroupName), serves the same purpose as the suite- and test case information functions previously described. However, the scope for the group information function, is all test cases and subgroups in the group in question (GroupName).

Example:

 group(connection_tests) ->
    [{require,login_data},
     {timetrap,1000}].

The group information properties override those set with the suite information function, and can in turn be overridden by test case information properties. For a list of valid information properties and more general information, see the Test Case Information Function.

Information functions can also be used for functions init_per_suite, end_per_suite, init_per_group, and end_per_group, and they work the same way as with the Test Case Information Function. This is useful, for example, for setting timetraps and requiring external configuration data relevant only for the configuration function in question (without affecting properties set for groups and test cases in the suite).

The information function init/end_per_suite() is called for init/end_per_suite(Config), and information function init/end_per_group(GroupName) is called for init/end_per_group(GroupName,Config). However, information functions cannot be used with init/end_per_testcase(TestCase, Config), as these configuration functions execute on the test case process and use the same properties as the test case (that is, the properties set by the test case information function, TestCase()). For a list of valid information properties and more general information, see the Test Case Information Function.

In the data directory, data_dir, the test module has its own files needed for the testing. The name of data_dir is the the name of the test suite followed by "_data". For example, "some_path/foo_SUITE.beam" has the data directory "some_path/foo_SUITE_data/". Use this directory for portability, that is, to avoid hardcoding directory names in your suite. As the data directory is stored in the same directory as your test suite, you can rely on its existence at runtime, even if the path to your test suite directory has changed between test suite implementation and execution.

priv_dir is the private directory for the test cases. This directory can be used whenever a test case (or configuration function) needs to write something to file. The name of the private directory is generated by Common Test, which also creates the directory.

By default, Common Test creates one central private directory per test run, shared by all test cases. This is not always suitable. Especially if the same test cases are executed multiple times during a test run (that is, if they belong to a test case group with property repeat) and there is a risk that files in the private directory get overwritten. Under these circumstances, Common Test can be configured to create one dedicated private directory per test case and execution instead. This is accomplished with the flag/option create_priv_dir (to be used with the ct_run program, the ct:run_test/1 function, or as test specification term). There are three possible values for this option as follows:

The first value indicates the default priv_dir behavior, that is, one private directory created per test run. The two latter values tell Common Test to generate a unique test directory name per test case and execution. If the auto version is used, all private directories are created automatically. This can become very inefficient for test runs with many test cases or repetitions, or both. Therefore, if the manual version is used instead, the test case must tell Common Test to create priv_dir when it needs it. It does this by calling the function ct:make_priv_dir/0.

Each test case is executed by a dedicated Erlang process. The process is spawned when the test case starts, and terminated when the test case is finished. The configuration functions init_per_testcase and end_per_testcase execute on the same process as the test case.

The configuration functions init_per_suite and end_per_suite execute, like test cases, on dedicated Erlang processes.

The default time limit for a test case is 30 minutes, unless a timetrap is specified either by the suite-, group-, or test case information function. The timetrap time-out value defined by suite/0 is the value that is used for each test case in the suite (and for the configuration functions init_per_suite/1, end_per_suite/1, init_per_group/2, and end_per_group/2). A timetrap value defined by group(GroupName) overrides one defined by suite() and is used for each test case in group GroupName, and any of its subgroups. If a timetrap value is defined by group/1 for a subgroup, it overrides that of its higher level groups. Timetrap values set by individual test cases (by the test case information function) override both group- and suite- level timetraps.

A timetrap can also be set or reset dynamically during the execution of a test case, or configuration function. This is done by calling ct:timetrap/1. This function cancels the current timetrap and starts a new one (that stays active until time-out, or end of the current function).

Timetrap values can be extended with a multiplier value specified at startup with option multiply_timetraps. It is also possible to let the test server decide to scale up timetrap time-out values automatically. That is, if tools such as cover or trace are running during the test. This feature is disabled by default and can be enabled with start option scale_timetraps.

If a test case needs to suspend itself for a time that also gets multipled by multiply_timetraps (and possibly also scaled up if scale_timetraps is enabled), the function ct:sleep/1 can be used (instead of, for example, timer:sleep/1).

A function (fun/0 or {Mod,Func,Args} (MFA) tuple) can be specified as timetrap value in the suite-, group- and test case information function, and as argument to function ct:timetrap/1.

Examples:

{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}

ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)

The user timetrap function can be used for two things as follows:

Before execution of the timetrap function (which is performed on a parallel, dedicated timetrap process), Common Test cancels any previously set timer for the test case or configuration function. When the timetrap function returns, the time-out is triggered, unless the return value is a valid timetrap time, such as an integer, or a {SecMinOrHourTag,Time} tuple (for details, see module common_test). If a time value is returned, a new timetrap is started to generate a time-out after the specified time.

The user timetrap function can return a time value after a delay. The effective timetrap time is then the delay time plus the returned time.

Common Test provides the following three main functions for printing strings:

The log/1,2,3,4,5 function prints a string to the test case log file. The print/1,2,3,4 function prints the string to screen. The pal/1,2,3,4 function prints the same string both to file and screen. The functions are described in module ct.

The optional Category argument can be used to categorize the log printout. Categories can be used for two things as follows:

Argument Importance specifies a level of importance that, compared to a verbosity level (general and/or set per category), determines if the printout is to be visible. Importance is any integer in the range 0..99. Predefined constants exist in the ct.hrl header file. The default importance level, ?STD_IMPORTANCE (used if argument Importance is not provided), is 50. This is also the importance used for standard I/O, for example, from printouts made with io:format/2, io:put_chars/1, and so on.

Importance is compared to a verbosity level set by the verbosity start flag/option. The level can be set per category or generally, or both. If verbosity is not set by the user, a level of 100 (?MAX_VERBOSITY = all printouts visible) is used as default value. Common Test performs the following test:

Importance >= (100-VerbosityLevel)

The constant ?STD_VERBOSITY has value 50 (see ct.hrl). At this level, all standard I/O gets printed. If a lower verbosity level is set, standard I/O printouts are ignored. Verbosity level 0 effectively turns all logging off (except from printouts made by Common Test itself).

The general verbosity level is not associated with any particular category. This level sets the threshold for the standard I/O printouts, uncategorized ct:log/print/pal printouts, and printouts for categories with undefined verbosity level.

Examples:

Some printouts during test case execution:

 io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]),
 ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]),
 ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]),
 ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]),
 ct:log(error, ?HI_IMPORTANCE, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]),
 ct:log(error, ?MAX_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]),

If starting the test with a general verbosity level of 50 (?STD_VERBOSITY):

 $ ct_run -verbosity 50

the following is printed:

 1. Standard IO, importance = 50
 2. Uncategorized, importance = 50
 3. Categorized info, importance = 50
 5. Categorized error, importance = 75
 6. Categorized error, importance = 99

If starting the test with:

 $ ct_run -verbosity 1 and info 75

the following is printed:

 3. Categorized info, importance = 50
 4. Categorized info, importance = 25
 6. Categorized error, importance = 99

Note that the category argument is not required in order to only specify the importance of a printout. Example:

ct:pal(?LOW_IMPORTANCE, "Info report: ~p", [Info])

Or perhaps in combination with constants:

-define(INFO, ?LOW_IMPORTANCE).
-define(ERROR, ?HI_IMPORTANCE).

ct:log(?INFO, "Info report: ~p", [Info])
ct:pal(?ERROR, "Error report: ~p", [Error])

The functions ct:set_verbosity/2 and ct:get_verbosity/1 may be used to modify and read verbosity levels during test execution.

The arguments Format and FormatArgs in ct:log/print/pal are always passed on to the STDLIB function io:format/3 (For details, see the io manual page).

ct:pal/4 and ct:log/5 add headers to strings being printed to the log file. The strings are also wrapped in div tags with a CSS class attribute, so that stylesheet formatting can be applied. To disable this feature for a printout (i.e. to get a result similar to using io:format/2), call ct:log/5 with the no_css option.

How categories can be mapped to CSS tags is documented in section HTML Style Sheets in section Running Tests and Analyzing Results.

Common Test will escape special HTML characters (<, > and &) in printouts to the log file made with ct:pal/4 and io:format/2. In order to print strings with HTML tags to the log, use the ct:log/3,4,5 function. The character escaping feature is per default disabled for ct:log/3,4,5 but can be enabled with the esc_chars option in the Opts list, see ct:log/3,4,5.

If the character escaping feature needs to be disabled (typically for backwards compatibility reasons), use the ct_run start flag -no_esc_chars, or the ct:run_test/1 start option {esc_chars,Bool} (this start option is also supported in test specifications).

For more information about log files, see section Log Files in section Running Tests and Analyzing Results.

Even though it is highly efficient to write test suites with the Common Test framework, mistakes can be made, mainly because of illegal dependencies. Some of the more frequent mistakes from our own experience with running the Erlang/OTP test suites follows: