From e7fc8ebdf5fcf82c581026d7c52e23a53b05ba01 Mon Sep 17 00:00:00 2001 From: Peter Andersson Date: Mon, 12 Dec 2011 03:10:51 +0100 Subject: Update basics chapter and introduce getting started chapter --- lib/common_test/doc/src/Makefile | 6 +- lib/common_test/doc/src/basics_chapter.xml | 75 ++++--- lib/common_test/doc/src/config.gif | Bin 0 -> 4963 bytes .../doc/src/getting_started_chapter.xml | 230 +++++++++++++++++++++ lib/common_test/doc/src/html_logs.gif | Bin 0 -> 10726 bytes lib/common_test/doc/src/part.xml | 1 + lib/common_test/doc/src/tc_execution.gif | Bin 0 -> 9561 bytes 7 files changed, 280 insertions(+), 32 deletions(-) create mode 100644 lib/common_test/doc/src/config.gif create mode 100644 lib/common_test/doc/src/getting_started_chapter.xml create mode 100644 lib/common_test/doc/src/html_logs.gif create mode 100644 lib/common_test/doc/src/tc_execution.gif (limited to 'lib') diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index 964f7c76c1..d9651f13b0 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -61,6 +61,7 @@ XML_PART_FILES = part.xml XML_CHAPTER_FILES = \ basics_chapter.xml \ + getting_started_chapter.xml \ install_chapter.xml \ write_test_chapter.xml \ test_structure_chapter.xml \ @@ -80,7 +81,10 @@ MAKE_EDOC = make_edoc BOOK_FILES = book.xml -GIF_FILES = +GIF_FILES = \ + tc_execution.gif \ + config.gif \ + html_logs.gif INSTALL_NOTES = ../../notes.html diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index c1bb365b1f..20141d2561 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -28,54 +28,65 @@ basics_chapter.xml - +
Introduction -

- The Common Test framework (CT) is a tool which can support - implementation and automated execution of test cases towards different - types of target systems. The framework is based on the OTP Test - Server. Test cases can be run individually or in batches. Common - Test also features a distributed testing mode with central - control and logging. This feature makes it possible to test - multiple systems independently in one common session. This - can be very useful e.g. for running automated large-scale regression - tests. +

The Common Test framework (CT) is a tool which supports + implementation and automated execution of test cases towards arbitrary + types of target systems. The CT framework is based on the OTP Test + Server and it's the main tool being used in all testing- and verification + activities that are part of Erlang/OTP system development- and maintenance. +

+ +

Test cases can be executed individually or in batches. Common Test + also features a distributed testing mode with central control and logging + (a feature that makes it possible to test multiple systems independently in + one common session, useful e.g. for running automated large-scale regression + tests).

The SUT (System Under Test) may consist of one or several target - nodes. CT contains a generic test server which together with - other test utilities is used to perform test case execution. - It is possible to start the tests from the CT GUI or from an OS- or - Erlang shell prompt. Test suites are files (Erlang + nodes. CT contains a generic test server which, together with + other test utilities, is used to perform test case execution. + It is possible to start the tests from a GUI or from the OS- or + Erlang shell. Test suites are files (Erlang modules) that contain the test cases (Erlang functions) to be executed. Support modules provide functions that the test cases utilize in order to carry out the tests.

-

- The main idea is that CT based test programs connect to - the target system(s) via standard O&M interfaces. CT - provides implementations and wrappers of some of these O&M - interfaces and will be extended with more interfaces later. - There are a number of target independent interfaces - supported in CT such as Generic Telnet, FTP etc. which can be - specialized or used directly for controlling instruments, - traffic generators etc.

+

In a black-box testing scenario, CT based test programs connect to + the target system(s) via standard O&M and CLI protocols. CT + provides implementations of, and wrapper interfaces to, some of these + protocols (most of which exist as stand-alone components and + applications in OTP). The wrappers simplify configuration and add + verbosity for logging purposes. CT will be continously extended with + useful support modules. (Note however that it's + a straightforward task to use any arbitrary Erlang/OTP component + for testing purposes with Common Test, without needing a CT wrapper + for it. It's as simple as calling Erlang functions). There + are a number of target independent interfaces supported in CT, such as + Generic Telnet, FTP, etc, which can be specialized or used + directly for controlling instruments, traffic load generators, etc. +

Common Test is also a very useful tool for white-box testing Erlang - code since the test programs can call Erlang API functions directly. - For black-box testing Erlang software, Erlang RPC as well as - standard O&M interfaces can be used. + code (e.g. module testing), since the test programs can call exported Erlang + functions directly and there's very little overhead required for + implementing basic test suites and executing simple tests. For black-box + testing Erlang software, Erlang RPC as well as standard O&M interfaces + can for example be used.

A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of - the major strengths of Common Test! + the major strengths of Common Test (thanks to the efficient + support for concurrency in the Erlang runtime system - which CT users + can take great advantage of!).

@@ -186,7 +197,7 @@ all() - Returns a list of all test cases in the suite. (Mandatory) + Returns a list of all test cases and groups in the suite. (Mandatory) suite() Info function used to return properties for the suite. (Optional) groups() @@ -197,12 +208,14 @@ end_per_suite(Config) Suite level configuration function, executed after the last test case. (Optional) + group(GroupName) + Info function used to return properties for a test case group. (Optional) init_per_group(GroupName, Config) Configuration function for a group, executed before the first - test case. (Mandatory if groups are defined) + test case. (Optional) end_per_group(GroupName, Config) Configuration function for a group, executed after the last - test case. (Mandatory if groups are defined) + test case. (Optional) init_per_testcase(TestCase, Config) Configuration function for a testcase, executed before each test case. (Optional) diff --git a/lib/common_test/doc/src/config.gif b/lib/common_test/doc/src/config.gif new file mode 100644 index 0000000000..ac8006c4fb Binary files /dev/null and b/lib/common_test/doc/src/config.gif differ diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml new file mode 100644 index 0000000000..7de0912036 --- /dev/null +++ b/lib/common_test/doc/src/getting_started_chapter.xml @@ -0,0 +1,230 @@ + + + + +
+ + 20072010 + Ericsson AB. All Rights Reserved. + + + The contents of this file are subject to the Erlang Public License, + Version 1.1, (the "License"); you may not use this file except in + compliance with the License. You should have received a copy of the + Erlang Public License along with this software. If not, it can be + retrieved online at http://www.erlang.org/. + + Software distributed under the License is distributed on an "AS IS" + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + + + Getting Started + Peter Andersson + + 2011-12-12 + + getting_started_chapter.xml +
+ +
+ Are you new around here? +

+ The purpose of this short chapter is to, with a "learning by example" + approach, give the newcomer a chance to get started quickly writing and + executing some first simple tests. The chapter will introduce some of the + basics, but leave most explanations and details for the later + chapters in this User's Guide. Hopefully though, after this chapter, you + will be inspired and unintimidated enough to go on and get into the + nitty-gritty that follows in this rather heavy User's Guide! If you're + not much into "learning by example" and prefer to get into more technical + detail right away, go ahead and skip to the next chapter. Again, the basics + presented here will be covered in detail in later chapters. +

+

+ This chapter also tries to demonstrate how dead simple it actually is + to write a very basic (yet for many module testing purposes, often sufficiently + complex) test suite, and execute its test cases. This is not necessarily + obvious when you read the rest of the chapters in the User's Guide. +

+

+ A quick note before we start: In order to understand what's discussed and + examplified here, it is recommended that you first read through the + opening Common Test Basics + chapter. +

+
+ +
+ Test case execution +

Execution of test cases is handled this way:

+ +

+ + + Successful vs unsuccessful test case execution. + + +

+ +

For each test case that Common Test is told to execute, it spawns a + dedicated process on which the test case function in question starts + running. (In parallel to the test case process, an idle waiting timer + process is started which is linked to the test case process. If the timer + process runs out of waiting time, it sends an exit signal to terminate + the test case process and this is what's called a timetrap). +

+

In scenario 1, the test case process terminates normally after case A has + finished executing its test code without detecting any errors. The test + case function simply returns a value and Common Test logs the test case as + successful. +

+

In scenario 2, an error is detected during test case execution + which causes the test case B function to generate an exception. + This causes the test case process to exit with reason + other than normal, and as a result, Common Test will log this as an + unsuccessful test case. +

+

As you can understand from the illustration above, Common Test requires + that a test case generates a runtime error to indicate failure (e.g. + by causing a bad match error or by calling exit/1, preferrably + through the ct:fail/1/2 help function). A succesful execution is + indicated by means of a normal return from the test case function. +

+
+ +
+ A simple test suite +

As you've seen in the basics chapter, the test suite module implements + callback functions (mandatory or optional) for various purposes, e.g: + + Init/end configuration function for the test suite + Init/end configuration function for a test case + Init/end configuration function for a test case group + Test cases + + The configuration functions are optional and if you don't need them for + your test, a test suite with one simple test case could look like this: +

+ 
+      -module(my1st_SUITE).
+      -compile(export_all).
+
+      all() ->
+          [mod_exists].
+
+      mod_exists(_) ->
+          {module,mymod} = code:load_file(mymod).
+

+ In this example we check that the mymod module exists (i.e. can be + successfully loaded by the code server). If the operation fails, we will + get a bad match error which terminates the test case. +

+
+ +
+ A test suite with configuration functions +

+ If we need to perform configuration operations in order to run our test, we + implement configuration functions in our suite. The result from a + configuration function is configuration data, or simply Config. + This is a list of key-value tuples which get passed from the configuration + function to the test cases (possibly through configuration functions on + "lower level"). The data flow looks like this: +

+ +

+ + + Config data flow in the suite. + + +

+ +

+ Here's an example of a test suite which uses configuration functions + to open and close a log file for the test cases (an operation that would + be unnecessary and irrelevant to perform by each test case): +

+ 
+      -module(check_log_SUITE).
+      -export([all/0, init_per_suite/1, end_per_suite/1]).
+      -export([check_restart_result/1, check_no_errors/1]).
+      
+      -define(value(Key,Config), proplists:get_value(Key,Config)).
+
+      all() -> [check_restart_result, check_no_errors].
+
+      init_per_suite(InitConfigData) ->
+          [{logref,open_log()} | InitConfigData].
+
+      end_per_suite(ConfigData) ->
+          close_log(?value(logref, ConfigData)).
+
+      check_restart_result(ConfigData) ->
+          TestData = read_log(restart, ?value(logref, ConfigData)),
+          {match,_Line} = search_for("restart successful", TestData).
+      
+      check_no_errors(ConfigData) ->
+          TestData = read_log(all, ?value(logref, ConfigData)),
+          case search_for("error", TestData) of
+              {match,Line} -> ct:fail({error_found_in_log,Line});
+              nomatch -> ok
+          end.
+

+ In this example we have test cases that verify, by parsing a + log file, that our SUT has performed a successful restart and + that no unexpected errors have been printed. +

+ +

To execute the test cases in the test suite above, we could type this on + the Unix/Linux command line (assuming for this example that the suite module + is in the current working directory): +

+ 
+      $ ct_run -dir .
+

or

+ 
+    $ ct_run -suite check_log_SUITE
+ +

If we want to use the Erlang shell to run our test, we could evaluate this call: +

+ 
+      1> ct:run_test([{dir, "."}]).
+

or

+ 
+      1> ct:run_test([{suite, "check_log_SUITE"}]).
+

+ The result from running our test is printed in log files on HTML format + (stored in unique log directories on different level). This illustration + shows the log file structure: +

+ +

+ + + HTML log file structure. + + +

+
+ +
+ What happens next? +

+ You will find detailed information about the basics introduced here in this + chapter in the following chapters in the User's Guide, as well as + presentations of many more useful features. Have fun! +

+
+ + + + + + + + + diff --git a/lib/common_test/doc/src/html_logs.gif b/lib/common_test/doc/src/html_logs.gif new file mode 100644 index 0000000000..3a3fd86bde Binary files /dev/null and b/lib/common_test/doc/src/html_logs.gif differ diff --git a/lib/common_test/doc/src/part.xml b/lib/common_test/doc/src/part.xml index 3284bcadaa..a74185221d 100644 --- a/lib/common_test/doc/src/part.xml +++ b/lib/common_test/doc/src/part.xml @@ -65,6 +65,7 @@ + diff --git a/lib/common_test/doc/src/tc_execution.gif b/lib/common_test/doc/src/tc_execution.gif new file mode 100644 index 0000000000..7c89d7be57 Binary files /dev/null and b/lib/common_test/doc/src/tc_execution.gif differ -- cgit v1.2.3