diff options
Diffstat (limited to 'lib/common_test')
| -rw-r--r-- | lib/common_test/doc/src/Makefile | 6 | ||||
| -rw-r--r-- | lib/common_test/doc/src/basics_chapter.xml | 75 | ||||
| -rw-r--r-- | lib/common_test/doc/src/config.gif | bin | 0 -> 4963 bytes | |||
| -rw-r--r-- | lib/common_test/doc/src/getting_started_chapter.xml | 230 | ||||
| -rw-r--r-- | lib/common_test/doc/src/html_logs.gif | bin | 0 -> 10726 bytes | |||
| -rw-r--r-- | lib/common_test/doc/src/part.xml | 1 | ||||
| -rw-r--r-- | lib/common_test/doc/src/tc_execution.gif | bin | 0 -> 9561 bytes |
7 files changed, 280 insertions, 32 deletions
diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index 964f7c76c1..d9651f13b0 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -61,6 +61,7 @@ XML_PART_FILES = part.xml XML_CHAPTER_FILES = \ basics_chapter.xml \ + getting_started_chapter.xml \ install_chapter.xml \ write_test_chapter.xml \ test_structure_chapter.xml \ @@ -80,7 +81,10 @@ MAKE_EDOC = make_edoc BOOK_FILES = book.xml -GIF_FILES = +GIF_FILES = \ + tc_execution.gif \ + config.gif \ + html_logs.gif INSTALL_NOTES = ../../notes.html diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index c1bb365b1f..20141d2561 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -28,54 +28,65 @@ <rev></rev> <file>basics_chapter.xml</file> </header> - + <marker id="basics"></marker> <section> <title>Introduction</title> - <p> - The Common Test framework (CT) is a tool which can support - implementation and automated execution of test cases towards different - types of target systems. The framework is based on the OTP Test - Server. Test cases can be run individually or in batches. Common - Test also features a distributed testing mode with central - control and logging. This feature makes it possible to test - multiple systems independently in one common session. This - can be very useful e.g. for running automated large-scale regression - tests. + <p>The <em>Common Test</em> framework (CT) is a tool which supports + implementation and automated execution of test cases towards arbitrary + types of target systems. The CT framework is based on the OTP Test + Server and it's the main tool being used in all testing- and verification + activities that are part of Erlang/OTP system development- and maintenance. + </p> + + <p>Test cases can be executed individually or in batches. Common Test + also features a distributed testing mode with central control and logging + (a feature that makes it possible to test multiple systems independently in + one common session, useful e.g. for running automated large-scale regression + tests). </p> <p> The SUT (System Under Test) may consist of one or several target - nodes. CT contains a generic test server which together with - other test utilities is used to perform test case execution. - It is possible to start the tests from the CT GUI or from an OS- or - Erlang shell prompt. <em>Test suites</em> are files (Erlang + nodes. CT contains a generic test server which, together with + other test utilities, is used to perform test case execution. + It is possible to start the tests from a GUI or from the OS- or + Erlang shell. <em>Test suites</em> are files (Erlang modules) that contain the <em>test cases</em> (Erlang functions) to be executed. <em>Support modules</em> provide functions that the test cases utilize in order to carry out the tests. </p> - <p> - The main idea is that CT based test programs connect to - the target system(s) via standard O&M interfaces. CT - provides implementations and wrappers of some of these O&M - interfaces and will be extended with more interfaces later. - There are a number of target independent interfaces - supported in CT such as Generic Telnet, FTP etc. which can be - specialized or used directly for controlling instruments, - traffic generators etc.</p> + <p>In a black-box testing scenario, CT based test programs connect to + the target system(s) via standard O&M and CLI protocols. CT + provides implementations of, and wrapper interfaces to, some of these + protocols (most of which exist as stand-alone components and + applications in OTP). The wrappers simplify configuration and add + verbosity for logging purposes. CT will be continously extended with + useful support modules. (Note however that it's + a straightforward task to use any arbitrary Erlang/OTP component + for testing purposes with Common Test, without needing a CT wrapper + for it. It's as simple as calling Erlang functions). There + are a number of target independent interfaces supported in CT, such as + Generic Telnet, FTP, etc, which can be specialized or used + directly for controlling instruments, traffic load generators, etc. + </p> <p>Common Test is also a very useful tool for white-box testing Erlang - code since the test programs can call Erlang API functions directly. - For black-box testing Erlang software, Erlang RPC as well as - standard O&M interfaces can be used. + code (e.g. module testing), since the test programs can call exported Erlang + functions directly and there's very little overhead required for + implementing basic test suites and executing simple tests. For black-box + testing Erlang software, Erlang RPC as well as standard O&M interfaces + can for example be used. </p> <p>A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of - the major strengths of Common Test! + the major strengths of Common Test (thanks to the efficient + support for concurrency in the Erlang runtime system - which CT users + can take great advantage of!). </p> </section> @@ -186,7 +197,7 @@ <taglist> <tag>all()</tag> - <item>Returns a list of all test cases in the suite. (Mandatory)</item> + <item>Returns a list of all test cases and groups in the suite. (Mandatory)</item> <tag>suite()</tag> <item>Info function used to return properties for the suite. (Optional)</item> <tag>groups()</tag> @@ -197,12 +208,14 @@ <tag>end_per_suite(Config)</tag> <item>Suite level configuration function, executed after the last test case. (Optional)</item> + <tag>group(GroupName)</tag> + <item>Info function used to return properties for a test case group. (Optional)</item> <tag>init_per_group(GroupName, Config)</tag> <item>Configuration function for a group, executed before the first - test case. (Mandatory if groups are defined)</item> + test case. (Optional)</item> <tag>end_per_group(GroupName, Config)</tag> <item>Configuration function for a group, executed after the last - test case. (Mandatory if groups are defined)</item> + test case. (Optional)</item> <tag>init_per_testcase(TestCase, Config)</tag> <item>Configuration function for a testcase, executed before each test case. (Optional)</item> diff --git a/lib/common_test/doc/src/config.gif b/lib/common_test/doc/src/config.gif Binary files differnew file mode 100644 index 0000000000..ac8006c4fb --- /dev/null +++ b/lib/common_test/doc/src/config.gif diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml new file mode 100644 index 0000000000..7de0912036 --- /dev/null +++ b/lib/common_test/doc/src/getting_started_chapter.xml @@ -0,0 +1,230 @@ +<?xml version="1.0" encoding="latin1" ?> +<!DOCTYPE chapter SYSTEM "chapter.dtd"> + +<chapter> + <header> + <copyright> + <year>2007</year><year>2010</year> + <holder>Ericsson AB. All Rights Reserved.</holder> + </copyright> + <legalnotice> + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + </legalnotice> + + <title>Getting Started</title> + <prepared>Peter Andersson</prepared> + <docno></docno> + <date>2011-12-12</date> + <rev></rev> + <file>getting_started_chapter.xml</file> + </header> + + <section> + <title>Are you new around here?</title> + <p> + The purpose of this short chapter is to, with a "learning by example" + approach, give the newcomer a chance to get started quickly writing and + executing some first simple tests. The chapter will introduce some of the + basics, but leave most explanations and details for the later + chapters in this User's Guide. Hopefully though, after this chapter, you + will be inspired and unintimidated enough to go on and get into the + nitty-gritty that follows in this rather heavy User's Guide! If you're + not much into "learning by example" and prefer to get into more technical + detail right away, go ahead and skip to the next chapter. Again, the basics + presented here will be covered in detail in later chapters. + </p> + <p> + This chapter also tries to demonstrate how dead simple it actually is + to write a very basic (yet for many module testing purposes, often sufficiently + complex) test suite, and execute its test cases. This is not necessarily + obvious when you read the rest of the chapters in the User's Guide. + </p> + <p> + A quick note before we start: In order to understand what's discussed and + examplified here, it is recommended that you first read through the + opening <seealso marker="basics_chapter#basics">Common Test Basics</seealso> + chapter. + </p> + </section> + + <section> + <title>Test case execution</title> + <p>Execution of test cases is handled this way:</p> + + <p> + <image file="tc_execution.gif"> + <icaption> + Successful vs unsuccessful test case execution. + </icaption> + </image> + </p> + + <p>For each test case that Common Test is told to execute, it spawns a + dedicated process on which the test case function in question starts + running. (In parallel to the test case process, an idle waiting timer + process is started which is linked to the test case process. If the timer + process runs out of waiting time, it sends an exit signal to terminate + the test case process and this is what's called a <em>timetrap</em>). + </p> + <p>In scenario 1, the test case process terminates normally after case A has + finished executing its test code without detecting any errors. The test + case function simply returns a value and Common Test logs the test case as + successful. + </p> + <p>In scenario 2, an error is detected during test case execution + which causes the test case B function to generate an exception. + This causes the test case process to exit with reason + other than normal, and as a result, Common Test will log this as an + unsuccessful test case. + </p> + <p>As you can understand from the illustration above, Common Test requires + that a test case generates a runtime error to indicate failure (e.g. + by causing a bad match error or by calling <c>exit/1</c>, preferrably + through the <c>ct:fail/1/2</c> help function). A succesful execution is + indicated by means of a normal return from the test case function. + </p> + </section> + + <section> + <title>A simple test suite</title> + <p>As you've seen in the basics chapter, the test suite module implements + callback functions (mandatory or optional) for various purposes, e.g: + <list> + <item>Init/end configuration function for the test suite</item> + <item>Init/end configuration function for a test case</item> + <item>Init/end configuration function for a test case group</item> + <item>Test cases</item> + </list> + The configuration functions are optional and if you don't need them for + your test, a test suite with one simple test case could look like this: + </p> + <pre> + -module(my1st_SUITE). + -compile(export_all). + + all() -> + [mod_exists]. + + mod_exists(_) -> + {module,mymod} = code:load_file(mymod).</pre> + <p> + In this example we check that the <c>mymod</c> module exists (i.e. can be + successfully loaded by the code server). If the operation fails, we will + get a bad match error which terminates the test case. + </p> + </section> + + <section> + <title>A test suite with configuration functions</title> + <p> + If we need to perform configuration operations in order to run our test, we + implement configuration functions in our suite. The result from a + configuration function is configuration data, or simply <em><c>Config</c></em>. + This is a list of key-value tuples which get passed from the configuration + function to the test cases (possibly through configuration functions on + "lower level"). The data flow looks like this: + </p> + + <p> + <image file="config.gif"> + <icaption> + Config data flow in the suite. + </icaption> + </image> + </p> + + <p> + Here's an example of a test suite which uses configuration functions + to open and close a log file for the test cases (an operation that would + be unnecessary and irrelevant to perform by each test case): + </p> + <pre> + -module(check_log_SUITE). + -export([all/0, init_per_suite/1, end_per_suite/1]). + -export([check_restart_result/1, check_no_errors/1]). + + -define(value(Key,Config), proplists:get_value(Key,Config)). + + all() -> [check_restart_result, check_no_errors]. + + init_per_suite(InitConfigData) -> + [{logref,open_log()} | InitConfigData]. + + end_per_suite(ConfigData) -> + close_log(?value(logref, ConfigData)). + + check_restart_result(ConfigData) -> + TestData = read_log(restart, ?value(logref, ConfigData)), + {match,_Line} = search_for("restart successful", TestData). + + check_no_errors(ConfigData) -> + TestData = read_log(all, ?value(logref, ConfigData)), + case search_for("error", TestData) of + {match,Line} -> ct:fail({error_found_in_log,Line}); + nomatch -> ok + end.</pre> + <p> + In this example we have test cases that verify, by parsing a + log file, that our SUT has performed a successful restart and + that no unexpected errors have been printed. + </p> + + <p>To execute the test cases in the test suite above, we could type this on + the Unix/Linux command line (assuming for this example that the suite module + is in the current working directory): + </p> + <pre> + $ ct_run -dir .</pre> + <p>or</p> + <pre> + $ ct_run -suite check_log_SUITE</pre> + + <p>If we want to use the Erlang shell to run our test, we could evaluate this call: + </p> + <pre> + 1> ct:run_test([{dir, "."}]).</pre> + <p>or</p> + <pre> + 1> ct:run_test([{suite, "check_log_SUITE"}]).</pre> + <p> + The result from running our test is printed in log files on HTML format + (stored in unique log directories on different level). This illustration + shows the log file structure: + </p> + + <p> + <image file="html_logs.gif"> + <icaption> + HTML log file structure. + </icaption> + </image> + </p> + </section> + + <section> + <title>What happens next?</title> + <p> + You will find detailed information about the basics introduced here in this + chapter in the following chapters in the User's Guide, as well as + presentations of many more useful features. Have fun! + </p> + </section> +</chapter> + + + + + + + + diff --git a/lib/common_test/doc/src/html_logs.gif b/lib/common_test/doc/src/html_logs.gif Binary files differnew file mode 100644 index 0000000000..3a3fd86bde --- /dev/null +++ b/lib/common_test/doc/src/html_logs.gif diff --git a/lib/common_test/doc/src/part.xml b/lib/common_test/doc/src/part.xml index 3284bcadaa..a74185221d 100644 --- a/lib/common_test/doc/src/part.xml +++ b/lib/common_test/doc/src/part.xml @@ -65,6 +65,7 @@ </description> <xi:include href="basics_chapter.xml"/> + <xi:include href="getting_started_chapter.xml"/> <xi:include href="install_chapter.xml"/> <xi:include href="write_test_chapter.xml"/> <xi:include href="test_structure_chapter.xml"/> diff --git a/lib/common_test/doc/src/tc_execution.gif b/lib/common_test/doc/src/tc_execution.gif Binary files differnew file mode 100644 index 0000000000..7c89d7be57 --- /dev/null +++ b/lib/common_test/doc/src/tc_execution.gif |
