aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test
diff options
context:
space:
mode:
authorPeter Andersson <[email protected]>2011-12-12 03:10:51 +0100
committerPeter Andersson <[email protected]>2011-12-12 05:08:41 +0100
commite7fc8ebdf5fcf82c581026d7c52e23a53b05ba01 (patch)
tree6f0a5b5a038b5cc0f7f1fa44b890082431fa6bfb /lib/common_test
parent46ef0536a9e3442c3a54237b59b238e6b6f7a90a (diff)
downloadotp-e7fc8ebdf5fcf82c581026d7c52e23a53b05ba01.tar.gz
otp-e7fc8ebdf5fcf82c581026d7c52e23a53b05ba01.tar.bz2
otp-e7fc8ebdf5fcf82c581026d7c52e23a53b05ba01.zip
Update basics chapter and introduce getting started chapter
Diffstat (limited to 'lib/common_test')
-rw-r--r--lib/common_test/doc/src/Makefile6
-rw-r--r--lib/common_test/doc/src/basics_chapter.xml75
-rw-r--r--lib/common_test/doc/src/config.gifbin0 -> 4963 bytes
-rw-r--r--lib/common_test/doc/src/getting_started_chapter.xml230
-rw-r--r--lib/common_test/doc/src/html_logs.gifbin0 -> 10726 bytes
-rw-r--r--lib/common_test/doc/src/part.xml1
-rw-r--r--lib/common_test/doc/src/tc_execution.gifbin0 -> 9561 bytes
7 files changed, 280 insertions, 32 deletions
diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile
index 964f7c76c1..d9651f13b0 100644
--- a/lib/common_test/doc/src/Makefile
+++ b/lib/common_test/doc/src/Makefile
@@ -61,6 +61,7 @@ XML_PART_FILES = part.xml
XML_CHAPTER_FILES = \
basics_chapter.xml \
+ getting_started_chapter.xml \
install_chapter.xml \
write_test_chapter.xml \
test_structure_chapter.xml \
@@ -80,7 +81,10 @@ MAKE_EDOC = make_edoc
BOOK_FILES = book.xml
-GIF_FILES =
+GIF_FILES = \
+ tc_execution.gif \
+ config.gif \
+ html_logs.gif
INSTALL_NOTES = ../../notes.html
diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml
index c1bb365b1f..20141d2561 100644
--- a/lib/common_test/doc/src/basics_chapter.xml
+++ b/lib/common_test/doc/src/basics_chapter.xml
@@ -28,54 +28,65 @@
<rev></rev>
<file>basics_chapter.xml</file>
</header>
-
+ <marker id="basics"></marker>
<section>
<title>Introduction</title>
- <p>
- The Common Test framework (CT) is a tool which can support
- implementation and automated execution of test cases towards different
- types of target systems. The framework is based on the OTP Test
- Server. Test cases can be run individually or in batches. Common
- Test also features a distributed testing mode with central
- control and logging. This feature makes it possible to test
- multiple systems independently in one common session. This
- can be very useful e.g. for running automated large-scale regression
- tests.
+ <p>The <em>Common Test</em> framework (CT) is a tool which supports
+ implementation and automated execution of test cases towards arbitrary
+ types of target systems. The CT framework is based on the OTP Test
+ Server and it's the main tool being used in all testing- and verification
+ activities that are part of Erlang/OTP system development- and maintenance.
+ </p>
+
+ <p>Test cases can be executed individually or in batches. Common Test
+ also features a distributed testing mode with central control and logging
+ (a feature that makes it possible to test multiple systems independently in
+ one common session, useful e.g. for running automated large-scale regression
+ tests).
</p>
<p>
The SUT (System Under Test) may consist of one or several target
- nodes. CT contains a generic test server which together with
- other test utilities is used to perform test case execution.
- It is possible to start the tests from the CT GUI or from an OS- or
- Erlang shell prompt. <em>Test suites</em> are files (Erlang
+ nodes. CT contains a generic test server which, together with
+ other test utilities, is used to perform test case execution.
+ It is possible to start the tests from a GUI or from the OS- or
+ Erlang shell. <em>Test suites</em> are files (Erlang
modules) that contain the <em>test cases</em> (Erlang functions)
to be executed. <em>Support modules</em> provide functions
that the test cases utilize in order to carry out the tests.
</p>
- <p>
- The main idea is that CT based test programs connect to
- the target system(s) via standard O&amp;M interfaces. CT
- provides implementations and wrappers of some of these O&amp;M
- interfaces and will be extended with more interfaces later.
- There are a number of target independent interfaces
- supported in CT such as Generic Telnet, FTP etc. which can be
- specialized or used directly for controlling instruments,
- traffic generators etc.</p>
+ <p>In a black-box testing scenario, CT based test programs connect to
+ the target system(s) via standard O&amp;M and CLI protocols. CT
+ provides implementations of, and wrapper interfaces to, some of these
+ protocols (most of which exist as stand-alone components and
+ applications in OTP). The wrappers simplify configuration and add
+ verbosity for logging purposes. CT will be continously extended with
+ useful support modules. (Note however that it's
+ a straightforward task to use any arbitrary Erlang/OTP component
+ for testing purposes with Common Test, without needing a CT wrapper
+ for it. It's as simple as calling Erlang functions). There
+ are a number of target independent interfaces supported in CT, such as
+ Generic Telnet, FTP, etc, which can be specialized or used
+ directly for controlling instruments, traffic load generators, etc.
+ </p>
<p>Common Test is also a very useful tool for white-box testing Erlang
- code since the test programs can call Erlang API functions directly.
- For black-box testing Erlang software, Erlang RPC as well as
- standard O&amp;M interfaces can be used.
+ code (e.g. module testing), since the test programs can call exported Erlang
+ functions directly and there's very little overhead required for
+ implementing basic test suites and executing simple tests. For black-box
+ testing Erlang software, Erlang RPC as well as standard O&amp;M interfaces
+ can for example be used.
</p>
<p>A test case can handle several connections towards one or
several target systems, instruments and traffic generators in
parallel in order to perform the necessary actions for a
test. The handling of many connections in parallel is one of
- the major strengths of Common Test!
+ the major strengths of Common Test (thanks to the efficient
+ support for concurrency in the Erlang runtime system - which CT users
+ can take great advantage of!).
</p>
</section>
@@ -186,7 +197,7 @@
<taglist>
<tag>all()</tag>
- <item>Returns a list of all test cases in the suite. (Mandatory)</item>
+ <item>Returns a list of all test cases and groups in the suite. (Mandatory)</item>
<tag>suite()</tag>
<item>Info function used to return properties for the suite. (Optional)</item>
<tag>groups()</tag>
@@ -197,12 +208,14 @@
<tag>end_per_suite(Config)</tag>
<item>Suite level configuration function, executed after the last
test case. (Optional)</item>
+ <tag>group(GroupName)</tag>
+ <item>Info function used to return properties for a test case group. (Optional)</item>
<tag>init_per_group(GroupName, Config)</tag>
<item>Configuration function for a group, executed before the first
- test case. (Mandatory if groups are defined)</item>
+ test case. (Optional)</item>
<tag>end_per_group(GroupName, Config)</tag>
<item>Configuration function for a group, executed after the last
- test case. (Mandatory if groups are defined)</item>
+ test case. (Optional)</item>
<tag>init_per_testcase(TestCase, Config)</tag>
<item>Configuration function for a testcase, executed before each
test case. (Optional)</item>
diff --git a/lib/common_test/doc/src/config.gif b/lib/common_test/doc/src/config.gif
new file mode 100644
index 0000000000..ac8006c4fb
--- /dev/null
+++ b/lib/common_test/doc/src/config.gif
Binary files differ
diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml
new file mode 100644
index 0000000000..7de0912036
--- /dev/null
+++ b/lib/common_test/doc/src/getting_started_chapter.xml
@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+ <header>
+ <copyright>
+ <year>2007</year><year>2010</year>
+ <holder>Ericsson AB. All Rights Reserved.</holder>
+ </copyright>
+ <legalnotice>
+ The contents of this file are subject to the Erlang Public License,
+ Version 1.1, (the "License"); you may not use this file except in
+ compliance with the License. You should have received a copy of the
+ Erlang Public License along with this software. If not, it can be
+ retrieved online at http://www.erlang.org/.
+
+ Software distributed under the License is distributed on an "AS IS"
+ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ the License for the specific language governing rights and limitations
+ under the License.
+
+ </legalnotice>
+
+ <title>Getting Started</title>
+ <prepared>Peter Andersson</prepared>
+ <docno></docno>
+ <date>2011-12-12</date>
+ <rev></rev>
+ <file>getting_started_chapter.xml</file>
+ </header>
+
+ <section>
+ <title>Are you new around here?</title>
+ <p>
+ The purpose of this short chapter is to, with a "learning by example"
+ approach, give the newcomer a chance to get started quickly writing and
+ executing some first simple tests. The chapter will introduce some of the
+ basics, but leave most explanations and details for the later
+ chapters in this User's Guide. Hopefully though, after this chapter, you
+ will be inspired and unintimidated enough to go on and get into the
+ nitty-gritty that follows in this rather heavy User's Guide! If you're
+ not much into "learning by example" and prefer to get into more technical
+ detail right away, go ahead and skip to the next chapter. Again, the basics
+ presented here will be covered in detail in later chapters.
+ </p>
+ <p>
+ This chapter also tries to demonstrate how dead simple it actually is
+ to write a very basic (yet for many module testing purposes, often sufficiently
+ complex) test suite, and execute its test cases. This is not necessarily
+ obvious when you read the rest of the chapters in the User's Guide.
+ </p>
+ <p>
+ A quick note before we start: In order to understand what's discussed and
+ examplified here, it is recommended that you first read through the
+ opening <seealso marker="basics_chapter#basics">Common Test Basics</seealso>
+ chapter.
+ </p>
+ </section>
+
+ <section>
+ <title>Test case execution</title>
+ <p>Execution of test cases is handled this way:</p>
+
+ <p>
+ <image file="tc_execution.gif">
+ <icaption>
+ Successful vs unsuccessful test case execution.
+ </icaption>
+ </image>
+ </p>
+
+ <p>For each test case that Common Test is told to execute, it spawns a
+ dedicated process on which the test case function in question starts
+ running. (In parallel to the test case process, an idle waiting timer
+ process is started which is linked to the test case process. If the timer
+ process runs out of waiting time, it sends an exit signal to terminate
+ the test case process and this is what's called a <em>timetrap</em>).
+ </p>
+ <p>In scenario 1, the test case process terminates normally after case A has
+ finished executing its test code without detecting any errors. The test
+ case function simply returns a value and Common Test logs the test case as
+ successful.
+ </p>
+ <p>In scenario 2, an error is detected during test case execution
+ which causes the test case B function to generate an exception.
+ This causes the test case process to exit with reason
+ other than normal, and as a result, Common Test will log this as an
+ unsuccessful test case.
+ </p>
+ <p>As you can understand from the illustration above, Common Test requires
+ that a test case generates a runtime error to indicate failure (e.g.
+ by causing a bad match error or by calling <c>exit/1</c>, preferrably
+ through the <c>ct:fail/1/2</c> help function). A succesful execution is
+ indicated by means of a normal return from the test case function.
+ </p>
+ </section>
+
+ <section>
+ <title>A simple test suite</title>
+ <p>As you've seen in the basics chapter, the test suite module implements
+ callback functions (mandatory or optional) for various purposes, e.g:
+ <list>
+ <item>Init/end configuration function for the test suite</item>
+ <item>Init/end configuration function for a test case</item>
+ <item>Init/end configuration function for a test case group</item>
+ <item>Test cases</item>
+ </list>
+ The configuration functions are optional and if you don't need them for
+ your test, a test suite with one simple test case could look like this:
+ </p>
+ <pre>
+ -module(my1st_SUITE).
+ -compile(export_all).
+
+ all() ->
+ [mod_exists].
+
+ mod_exists(_) ->
+ {module,mymod} = code:load_file(mymod).</pre>
+ <p>
+ In this example we check that the <c>mymod</c> module exists (i.e. can be
+ successfully loaded by the code server). If the operation fails, we will
+ get a bad match error which terminates the test case.
+ </p>
+ </section>
+
+ <section>
+ <title>A test suite with configuration functions</title>
+ <p>
+ If we need to perform configuration operations in order to run our test, we
+ implement configuration functions in our suite. The result from a
+ configuration function is configuration data, or simply <em><c>Config</c></em>.
+ This is a list of key-value tuples which get passed from the configuration
+ function to the test cases (possibly through configuration functions on
+ "lower level"). The data flow looks like this:
+ </p>
+
+ <p>
+ <image file="config.gif">
+ <icaption>
+ Config data flow in the suite.
+ </icaption>
+ </image>
+ </p>
+
+ <p>
+ Here's an example of a test suite which uses configuration functions
+ to open and close a log file for the test cases (an operation that would
+ be unnecessary and irrelevant to perform by each test case):
+ </p>
+ <pre>
+ -module(check_log_SUITE).
+ -export([all/0, init_per_suite/1, end_per_suite/1]).
+ -export([check_restart_result/1, check_no_errors/1]).
+
+ -define(value(Key,Config), proplists:get_value(Key,Config)).
+
+ all() -> [check_restart_result, check_no_errors].
+
+ init_per_suite(InitConfigData) ->
+ [{logref,open_log()} | InitConfigData].
+
+ end_per_suite(ConfigData) ->
+ close_log(?value(logref, ConfigData)).
+
+ check_restart_result(ConfigData) ->
+ TestData = read_log(restart, ?value(logref, ConfigData)),
+ {match,_Line} = search_for("restart successful", TestData).
+
+ check_no_errors(ConfigData) ->
+ TestData = read_log(all, ?value(logref, ConfigData)),
+ case search_for("error", TestData) of
+ {match,Line} -> ct:fail({error_found_in_log,Line});
+ nomatch -> ok
+ end.</pre>
+ <p>
+ In this example we have test cases that verify, by parsing a
+ log file, that our SUT has performed a successful restart and
+ that no unexpected errors have been printed.
+ </p>
+
+ <p>To execute the test cases in the test suite above, we could type this on
+ the Unix/Linux command line (assuming for this example that the suite module
+ is in the current working directory):
+ </p>
+ <pre>
+ $ ct_run -dir .</pre>
+ <p>or</p>
+ <pre>
+ $ ct_run -suite check_log_SUITE</pre>
+
+ <p>If we want to use the Erlang shell to run our test, we could evaluate this call:
+ </p>
+ <pre>
+ 1> ct:run_test([{dir, "."}]).</pre>
+ <p>or</p>
+ <pre>
+ 1> ct:run_test([{suite, "check_log_SUITE"}]).</pre>
+ <p>
+ The result from running our test is printed in log files on HTML format
+ (stored in unique log directories on different level). This illustration
+ shows the log file structure:
+ </p>
+
+ <p>
+ <image file="html_logs.gif">
+ <icaption>
+ HTML log file structure.
+ </icaption>
+ </image>
+ </p>
+ </section>
+
+ <section>
+ <title>What happens next?</title>
+ <p>
+ You will find detailed information about the basics introduced here in this
+ chapter in the following chapters in the User's Guide, as well as
+ presentations of many more useful features. Have fun!
+ </p>
+ </section>
+</chapter>
+
+
+
+
+
+
+
+
diff --git a/lib/common_test/doc/src/html_logs.gif b/lib/common_test/doc/src/html_logs.gif
new file mode 100644
index 0000000000..3a3fd86bde
--- /dev/null
+++ b/lib/common_test/doc/src/html_logs.gif
Binary files differ
diff --git a/lib/common_test/doc/src/part.xml b/lib/common_test/doc/src/part.xml
index 3284bcadaa..a74185221d 100644
--- a/lib/common_test/doc/src/part.xml
+++ b/lib/common_test/doc/src/part.xml
@@ -65,6 +65,7 @@
</description>
<xi:include href="basics_chapter.xml"/>
+ <xi:include href="getting_started_chapter.xml"/>
<xi:include href="install_chapter.xml"/>
<xi:include href="write_test_chapter.xml"/>
<xi:include href="test_structure_chapter.xml"/>
diff --git a/lib/common_test/doc/src/tc_execution.gif b/lib/common_test/doc/src/tc_execution.gif
new file mode 100644
index 0000000000..7c89d7be57
--- /dev/null
+++ b/lib/common_test/doc/src/tc_execution.gif
Binary files differ