diff options
| -rw-r--r-- | HOWTO/TESTING.md | 150 |
1 files changed, 150 insertions, 0 deletions
diff --git a/HOWTO/TESTING.md b/HOWTO/TESTING.md new file mode 100644 index 0000000000..34eaa68df8 --- /dev/null +++ b/HOWTO/TESTING.md @@ -0,0 +1,150 @@ +Testing Erlang/OTP +================== + +Before you start testing you need to have the Erlang release which you +are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for +instructions on how to build an Erlang release. + +Short version +------------- +Move to the top directory of the Erlang release you want to test, i.e. +cd /ldisk/work/otp + + export ERL_TOP=`pwd` + ./otp_build setup -a + export PATH=`pwd`/bin:$PATH + ./otp_build tests + cd release/tests/test_server + erl -s ts install -s ts run all_tests -s init stop + +Where are the tests +------------------- + +There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and +they are located in different places throughout the source tree. Below is a list +of the places where you can expect to find tests: + +* $ERL_TOP/lib/AppName/test/ +* $ERL_TOP/erts/test/ +* $ERL_TOP/erts/emulator/test/ +* $ERL_TOP/erts/epmd/test/ + +Writing tests +------------- + +All tests are [common_test][] suites and follow the same pattern as all +[common_test][] suites. However, a couple of corner cases are +handled by a test wrapper called `ts`. `ts` allows the test writer to put +`Makefile.src` and `Makefile.first` files in the [data_dir][] and a special +directory called `all_SUITE_data`. + +`Makefile.first` is run before any other Makefile and is typically used to +generate .hrl files which are needed by other test suites. At the moment only +the erl_interface tests use this feature and it should remain that way. + +`Makefile.src` is configured to a `Makefile` using the `variables` created when +[configuring the tests][]. These `Makefile`s are later run by the test suite +to compile whatever platform specific code the tests need to run. + +Releasing tests +--------------- + +If you cannot use [ct_run][] in the source tree you have to release the tests +into a common test directory. The easiest way to do this is to use `otp_build` +like this: + + export ERL_TOP=`pwd`; ./otp_build tests + +This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you +want to change the directory where the tests are released to use the `TESTROOT` +environmental variable. + +In the `$TESTROOT` you should now see *_test folders. These folders contain +everything needed to test Erlang/OTP and are platform independent; if you are +testing Erlang on multiple platforms you just have to release on one and copy +the tests to all other platforms. + +### Releasing cross tests + +For releasing tests in a cross compilation environment see [$ERL_TOP/HOWTO/INSTALL-CROSS.md][]. + +Configuring and Running tests +----------------------------- + +Running tests is done by first navigating to the `$TESTROOT/test_server` folder +created when you released the tests and then start `erl` in that directory. The +emulator flags specified will be used in the test runs. For example, if you want +to test using async threads you have to supply `+A 10` to `erl` when you start it. + +To configure and run the tests `ts` is used. `ts` is a wrapper module to +[common_test][] which takes care of configuration and build issues before +[common_test][] is started. + +`ts` has a lot of special options and functions which can be usefull when +testing Erlang/OTP. For a full listing issue `ts:help()` in the erlang shell. + +### Configuring the test environment + +Before running released tests you have to install them on the target system. +Installing the tests is done by calling `ts:install().` in the Erlang shell +which you intend to test from. `ts:install()` is basically a wrapper to a +configure script and some Erlang code which figures out what your system looks +like and what kind of emulator you are testing with. `ts:install()` can also +take some arguments if necessary, see `ts:help()` for details. + +All variables created by `ts:install()` are found in +`$TESTROOT/test_server/variables`. + +### Running the tests + +To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type: + + ts:run(). + +Note that running all tests will require several hours, so you may want to run +the test cases for a single application + + ts:run(Application, [batch]). + +or even part of the test suite for an application, for example + + ts:run(emulator, bs, [batch]). + +to run all test suite modules starting with `bs` (i.e. all modules that test +the bit syntax). + +To run a specific test case in a module, the full name of the module and test +case must be specified: + + ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]). + +Run `ts:help().` for more information. + +As of R14B02 it is also possibly to start all tests but the erl_interface tests +by invoking Common Test directly from the released applications test directory, +i.e. + + cd $TESTROOT/test_server + $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse + +Running [ct_run][] from the command line still requires you to do the +`ts:install()` step above. + +Examining the results +--------------------- + +Open the file `release/tests/test_server/index.html` in a web browser. Or open +`release/tests/test_server/last_test.html` when a test suite is running to +examine the results so far for the currently executing test suite (in R14B02 and +later you want to open the `release/tests/test_server/all_runs.html` file to +get to the currently running test) + + [ct_run]: http://www.erlang.org/doc/man/ct_run.html + [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html + [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md + [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md#testing-the-cross-compiled-system + [common_test]: http://www.erlang.org/doc/man/ct.html + [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir + [configuring the tests]: #configuring-the-test-environment + + [?TOC]: true |