aboutsummaryrefslogtreecommitdiffstats
path: root/lib/common_test/doc/src/why_test_chapter.xml
blob: cdac4e04b2b6aae09bde8fceb370c2e9cdb50a56 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2003</year><year>2016</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at
 
          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
    
    </legalnotice>

    <title>Some Thoughts about Testing</title>
    <prepared>Siri Hansen</prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>why_test_chapter.xml</file>
  </header>
         

  <section>
    <title>Goals</title>

    <p>It is not possible to prove that a program is correct by
    testing. On the contrary, it has been formally proven that it is
    impossible to prove programs in general by testing. Theoretical
    program proofs or plain examination of code can be viable options
    for those wishing to certify that a program is correct. The test
    server, as it is based on testing, cannot be used for
    certification. Its intended use is instead to (cost effectively)
    <em>find bugs</em>. A successful test suite is one that reveals a
    bug. If a test suite results in OK, then we know very little that
    we did not know before.</p>

  </section>

  <section>
    <title>What to Test</title>

    <p>
      There are many kinds of test suites. Some concentrate on
      calling every function or command (in the documented way) in 
      a certain interface.
      Some others do the same, but use all kinds of illegal
      parameters, and verify that the server stays alive and rejects
      the requests with reasonable error codes. Some test suites
      simulate an application (typically consisting of a few modules of
      an application), some try to do tricky requests in general, and some
      test suites even test internal functions with help of special
      Load Modules on target.</p>

    <p>Another interesting category of test suites is the one
      checking that fixed bugs do not reoccur. When a bugfix is introduced,
      a test case that checks for that specific bug is written
      and submitted to the affected test suites.</p>

    <p>Aim for finding bugs. Write whatever test that has the highest
      probability of finding a bug, now or in the future. Concentrate
      more on the critical parts. Bugs in critical subsystems are much
      more expensive than others.</p>

    <p>Aim for functionality testing rather than implementation
      details. Implementation details change quite often, and the test
      suites are to be long lived. Implementation details often differ
      on different platforms and versions. If implementation details
      must be tested, try to factor them out into separate test
      cases. These test cases can later be rewritten or skipped.</p>

    <p>Also, aim for testing everything once, no less, no more. It is
      not effective having every test case fail only because one
      function in the interface changed.</p>

  </section>

</chapter>