Add a testing HOWTO

1 files changed, 150 insertions, 0 deletions

diff --git a/HOWTO/TESTING.md b/HOWTO/TESTING.md
new file mode 100644
index 0000000000..34eaa68df8
--- /dev/null
+++ b/HOWTO/TESTING.md
@@ -0,0 +1,150 @@
+Testing Erlang/OTP
+==================
+
+Before you start testing you need to have the Erlang release which you
+are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for
+instructions on how to build an Erlang release.
+
+Short version
+-------------
+Move to the top directory of the Erlang release you want to test, i.e.
+cd /ldisk/work/otp
+
+    export ERL_TOP=`pwd`
+	./otp_build setup -a
+	export PATH=`pwd`/bin:$PATH
+	./otp_build tests
+	cd release/tests/test_server
+	erl -s ts install -s ts run all_tests -s init stop
+
+Where are the tests
+-------------------
+
+There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and
+they are located in different places throughout the source tree. Below is a list
+of the places where you can expect to find tests:
+
+* $ERL_TOP/lib/AppName/test/
+* $ERL_TOP/erts/test/
+* $ERL_TOP/erts/emulator/test/
+* $ERL_TOP/erts/epmd/test/
+
+Writing tests
+-------------
+
+All tests are [common_test][] suites and follow the same pattern as all
+[common_test][] suites. However, a couple of corner cases are
+handled by a test wrapper called `ts`. `ts` allows the test writer to put
+`Makefile.src` and `Makefile.first` files in the [data_dir][] and a special
+directory called `all_SUITE_data`.
+
+`Makefile.first` is run before any other Makefile and is typically used to
+generate .hrl files which are needed by other test suites. At the moment only
+the erl_interface tests use this feature and it should remain that way.
+
+`Makefile.src` is configured to a `Makefile` using the `variables` created when
+[configuring the tests][]. These `Makefile`s are later run by the test suite
+to compile whatever platform specific code the tests need to run.
+
+Releasing tests
+---------------
+
+If you cannot use [ct_run][] in the source tree you have to release the tests
+into a common test directory. The easiest way to do this is to use `otp_build`
+like this:
+
+    export ERL_TOP=`pwd`; ./otp_build tests
+
+This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you
+want to change the directory where the tests are released to use the `TESTROOT`
+environmental variable.
+
+In the `$TESTROOT` you should now see *_test folders. These folders contain
+everything needed to test Erlang/OTP and are platform independent; if you are
+testing Erlang on multiple platforms you just have to release on one and copy
+the tests to all other platforms.
+
+### Releasing cross tests
+
+For releasing tests in a cross compilation environment see [$ERL_TOP/HOWTO/INSTALL-CROSS.md][].
+
+Configuring and Running tests
+-----------------------------
+
+Running tests is done by first navigating to the `$TESTROOT/test_server` folder
+created when you released the tests and then start `erl` in that directory. The
+emulator flags specified will be used in the test runs. For example, if you want
+to test using async threads you have to supply `+A 10` to `erl` when you start it.
+
+To configure and run the tests `ts` is used. `ts` is a wrapper module to
+[common_test][] which takes care of configuration and build issues before
+[common_test][] is started.
+
+`ts` has a lot of special options and functions which can be usefull when
+testing Erlang/OTP. For a full listing issue `ts:help()` in the erlang shell.
+
+### Configuring the test environment
+
+Before running released tests you have to install them on the target system.
+Installing the tests is done by calling `ts:install().` in the Erlang shell
+which you intend to test from. `ts:install()` is basically a wrapper to a
+configure script and some Erlang code which figures out what your system looks
+like and what kind of emulator you are testing with. `ts:install()` can also
+take some arguments if necessary, see `ts:help()` for details.
+
+All variables created by `ts:install()` are found in
+`$TESTROOT/test_server/variables`.
+
+### Running the tests
+
+To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type:
+
+    ts:run().
+
+Note that running all tests will require several hours, so you may want to run
+the test cases for a single application
+
+    ts:run(Application, [batch]).
+
+or even part of the test suite for an application, for example
+
+    ts:run(emulator, bs, [batch]).
+
+to run all test suite modules starting with `bs` (i.e. all modules that test
+the bit syntax).
+
+To run a specific test case in a module, the full name of the module and test
+case must be specified:
+
+    ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]).
+
+Run `ts:help().` for more information.
+
+As of R14B02 it is also possibly to start all tests but the erl_interface tests
+by invoking Common Test directly from the released applications test directory,
+i.e.
+
+    cd $TESTROOT/test_server
+    $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse
+
+Running [ct_run][] from the command line still requires you to do the
+`ts:install()` step above.
+
+Examining the results
+---------------------
+
+Open the file `release/tests/test_server/index.html` in a web browser. Or open
+`release/tests/test_server/last_test.html` when a test suite is running to
+examine the results so far for the currently executing test suite (in R14B02 and
+later you want to open the `release/tests/test_server/all_runs.html` file to
+get to the currently running test)
+
+   [ct_run]: http://www.erlang.org/doc/man/ct_run.html
+   [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html
+   [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md
+   [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md#testing-the-cross-compiled-system
+   [common_test]: http://www.erlang.org/doc/man/ct.html
+   [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir
+   [configuring the tests]: #configuring-the-test-environment
+
+   [?TOC]: true