blob: 2aa9e4cd70648fd951ab71c8143507d188514dd7 (
plain) (
tree)
|
|
Benchmark framework
-------------------
This benchmark framework consists of the files:
bench.erl - see bench module below
bench.hrl - Defines some useful macros
all.erl - see all module below
bench module
-----------
The module bench is a generic module that measures execution time
of functions in callback modules and writes an html-report on the outcome.
When you execute the function bench:run/0 it will compile and run all
benchmark modules in the current directory.
all module
-----------
In the all module there is a function called releases/0 that you can
edit to contain all your erlang installations and then you can
run your benchmarks on several erlang versions using only one command i.e.
all:run().
Requirements on callback modules
---------------------------------
* A callback module must be named <callbackModuleName>_bm.erl
* The module must export the function benchmarks/0 that must return:
{Iterations, [Name1,Name2...]} where Iterations is the number of
times each benchmark should be run. Name1, Name2 and so one are the
name of exported functions in the module.
* The exported functions Name1 etc. must take one argument i.e. the number
of iterations and should return the atom ok.
* The functions in a benchmark module should represent different
ways/different sequential algorithms for doing something. And the
result will be how fast they are compared to each other.
Files created
--------------
Files that are created in the current directory are *.bmres and
index.html. The file(s) with the extension "bmres" are an intermediate
representation of the benchmark results and is only meant to be read
by the reporting mechanism defined in bench.erl. The index.html file
is the report telling you how good the benchmarks are in comparison to
each other. If you run your test on several erlang releases the
html-file will include the result for all versions.
Pitfalls
---------
To get meaningful measurements, you should make sure that:
* The total execution time is at least several seconds.
* That any time spent in setup before entering the measurement loop is very
small compared to the total time.
* That time spent by the loop itself is small compared to the total execution
time
Consider the following example of a benchmark function that does
a local function call.
local_call(0) -> ok;
local_call(Iter) ->
foo(), % Local function call
local_call(Iter-1).
The problem is that both "foo()" and "local_call(Iter-1)" takes about
the same amount of time. To get meaningful figures you'll need to make
sure that the loop overhead will not be visible. In this case we can
take help of a macro in bench.hrl to repeat the local function call
many times, making sure that time spent calling the local function is
relatively much longer than the time spent iterating. Of course, all
benchmarks in the same module must be repeated the same number of
times; thus external_call will look like
external_call(0) -> ok;
external_call(Iter) ->
?rep20(?MODULE:foo()),
external_call(Iter-1).
This technique is only necessary if the operation we are testing executes
really fast.
If you for instance want to test a sort routine we can keep it simple:
sorted(Iter) ->
do_sort(Iter, lists:seq(0, 63)).
do_sort(0, List) -> ok;
do_sort(Iter, List) ->
lists:sort(List),
do_sort(Iter-1, List).
The call to lists:seq/2 is only done once. The loop overhead in the
do_sort/2 function is small compared to the execution time of lists:sort/1.
Error handling
---------------
Any error enforced by a callback module will result in exit of the benchmark
program and an errormessage that should give a good idea of what is wrong.
|
