The R13B03 release.OTP_R13B03

1 files changed, 798 insertions, 0 deletions

diff --git a/lib/observer/doc/src/ttb_ug.xml b/lib/observer/doc/src/ttb_ug.xml
new file mode 100644
index 0000000000..44b7b08fd3
--- /dev/null
+++ b/lib/observer/doc/src/ttb_ug.xml
@@ -0,0 +1,798 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+  <header>
+    <copyright>
+      <year>2002</year><year>2009</year>
+      <holder>Ericsson AB. All Rights Reserved.</holder>
+    </copyright>
+    <legalnotice>
+      The contents of this file are subject to the Erlang Public License,
+      Version 1.1, (the "License"); you may not use this file except in
+      compliance with the License. You should have received a copy of the
+      Erlang Public License along with this software. If not, it can be
+      retrieved online at http://www.erlang.org/.
+    
+      Software distributed under the License is distributed on an "AS IS"
+      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+      the License for the specific language governing rights and limitations
+      under the License.
+    
+    </legalnotice>
+
+    <title>Trace Tool Builder</title>
+    <prepared></prepared>
+    <docno></docno>
+    <date></date>
+    <rev></rev>
+    <file>ttb_ug.xml</file>
+  </header>
+
+  <section>
+    <title>Introduction</title>
+    <p>The Trace Tool Builder is a base for building trace tools for
+      single node or distributed erlang systems. It requires the
+      <c>runtime_tools</c> application to be available on the traced
+      node.
+      </p>
+    <p>The main features of the Trace Tool Builder are:</p>
+    <list type="bulleted">
+      <item>Start tracing to file ports on several nodes with one
+       function call.</item>
+      <item>Write additional information to a trace information file,
+       which is read during formatting.</item>
+      <item>Restoring of previous configuration by maintaining a
+       history buffer and handling configuration files.</item>
+      <item>Some simple support for sequential tracing.</item>
+      <item>Formatting of binary trace logs and merging of logs from
+       multiple nodes.</item>
+    </list>
+    <p>Even though the intention of the Trace Tool Builder is to serve
+      as a base for tailor made trace tools, it is of course possible
+      to use it directly from the erlang shell. The application only
+      allows the use of file port tracer, so if you would like would
+      like to use other types of trace clients you will be better off
+      using <c>dbg</c> directly instead.</p>
+  </section>
+
+  <section>
+    <title>Getting Started</title>
+    <p>The <c>ttb</c> module is the interface to all functions in the
+      Trace Tool Builder. To get started the least you need to do is to
+      start a tracer with <c>ttb:tracer/0/1/2</c>, and set the required
+      trace flags on the processes you want to trace with
+      <c>ttb:p/2</c>. Then, when the tracing is completed, you must stop
+      the tracer with <c>ttb:stop/0/1</c> and format the trace log with
+      <c>ttb:format/1/2</c>.
+      </p>
+    <p><c>ttb:tracer/0/1/2</c> opens a file trace port on each node
+      that shall be traced. All trace messages will be written to this
+      port and end up in a binary file (the binary trace log).
+      </p>
+    <p><c>ttb:p/2</c> specifies which processes that shall be
+      traced. Trace flags given in this call specifies what to trace on
+      each process. You can call this function several times if you like
+      different trace flags to be set on different processes.
+      </p>
+    <p>If you want to trace function calls (i.e. if you have the
+      <c>call</c> trace flag set on any of your processes), you must
+      also set trace patterns on the required function(s) with
+      <c>ttb:tp</c> or <c>ttb:tpl</c>. A function is only traced if it
+      has a trace pattern. The trace pattern specifies how to trace the
+      function by using match specifications. Match specifications are
+      described in the User's Guide for the erlang runtime system
+      <c>erts</c>.
+      </p>
+    <p><c>ttb:stop/0/1</c> stops tracing on all nodes, deletes all
+      trace patterns and flushes the trace port buffer.
+      </p>
+    <p><c>ttb:format/1/2</c> translates the binary trace logs into
+      something readable. By default <c>ttb</c> presents each trace
+      message as a line of text, but you can also write your own handler
+      to make more complex interpretations of the trace information. A
+      trace log can even be presented graphically via the Event Tracer
+      application. Note that if you give the <c>format</c> option to
+      <c>ttb:stop/1</c> the formatting is automatically done when
+      stopping <c>ttb</c>.
+      </p>
+
+    <section>
+      <title>Example: Tracing the local node from the erlang shell</title>
+      <p>This small module is used in the example:</p>
+      <code type="none">
+-module(m).
+-export([f/0]).
+f() ->
+   receive 
+      From when pid(From) -> 
+         Now = erlang:now(),
+         From ! {self(),Now}
+   end.      </code>
+      <p>The following example shows the basic use of <c>ttb</c> from
+        the erlang shell. Default options are used both for starting the
+        tracer and for formatting. This gives a trace log named
+        <c>Node-ttb</c>, where <c>Node</c> is the name of the node.  The
+        default handler prints the formatted trace messages in the
+        shell.</p>
+      <code type="none"><![CDATA[
+(tiger@durin)47> %% First I spawn a process running my test function
+(tiger@durin)47> Pid = spawn(m,f,[]).
+<0.125.0>
+(tiger@durin)48> 
+(tiger@durin)48> %% Then I start a tracer...
+(tiger@durin)48> ttb:tracer().
+{ok,[tiger@durin]}
+(tiger@durin)49> 
+(tiger@durin)49> %% and activate the new process for tracing
+(tiger@durin)49> %% function calls and sent messages.
+(tiger@durin)49> ttb:p(Pid,[call,send]).
+{ok,[{<0.125.0>,[{matched,tiger@durin,1}]}]}
+(tiger@durin)50> 
+(tiger@durin)50> %% Here I set a trace pattern on erlang:now/0
+(tiger@durin)50> %% The trace pattern is a simple match spec
+(tiger@durin)50> %% generated by dbg:fun2ms/1. It indicates that 
+(tiger@durin)50> %% the return value shall be traced.
+(tiger@durin)50> MS = dbg:fun2ms(fun(_) -> return_trace() end).
+[{'_',[],[{return_trace}]}]
+(tiger@durin)51> ttb:tp(erlang,now,MS).
+{ok,[{matched,tiger@durin,1},{saved,1}]}
+(tiger@durin)52> 
+(tiger@durin)52> %% I run my test (i.e. send a message to
+(tiger@durin)52> %% my new process)
+(tiger@durin)52> Pid ! self().
+<0.72.0>
+(tiger@durin)53> 
+(tiger@durin)53> %% And then I have to stop ttb in order to flush
+(tiger@durin)53> %% the trace port buffer
+(tiger@durin)53> ttb:stop().
+stopped
+(tiger@durin)54> 
+(tiger@durin)54> %% Finally I format my trace log
+(tiger@durin)54> ttb:format("tiger@durin-ttb").
+({<0.125.0>,{m,f,0},tiger@durin}) call erlang:now()
+({<0.125.0>,{m,f,0},tiger@durin}) returned from erlang:now/0 ->
+{1031,133451,667611}
+({<0.125.0>,{m,f,0},tiger@durin}) <0.72.0> ! 
+{<0.125.0>,{1031,133451,667611}}
+ok      ]]></code>
+    </section>
+
+    <section>
+      <title>Example: Build your own tool</title>
+      <p>This small example shows a simple tool for "debug tracing",
+        i.e. tracing of function calls with return values.</p>
+      <code type="none"><![CDATA[
+-module(mydebug).
+-export([start/0,trc/1,stop/0,format/1]).
+-export([print/4]).
+
+%% Include ms_transform.hrl so that I can use dbg:fun2ms/2 to
+%% generate match specifications.
+-include_lib("stdlib/include/ms_transform.hrl").
+
+%%% -------------Tool API-------------
+%%% ----------------------------------
+%%% Star the "mydebug" tool
+start() ->
+    %% The options specify that the binary log shall be named
+    %% <Node>-debug_log and that the print/4 function in this
+    %% module shall be used as format handler
+    ttb:tracer(all,[{file,"debug_log"},{handler,{{?MODULE,print},0}}]),
+    %% All processes (existing and new) shall trace function calls
+    %% and include a timestamp in each trace message
+    ttb:p(all,[call,timestamp]).
+
+%%% Set trace pattern on function(s)
+trc(M) when atom(M) ->
+    trc({M,'_','_'});
+trc({M,F}) when atom(M), atom(F) ->
+    trc({M,F,'_'});
+trc({M,F,_A}=MFA) when atom(M), atom(F) ->
+    %% This match spec specifies that return values shall 
+    %% be traced. NOTE that ms_transform.hrl must be included
+    %% if dbg:fun2ms/1 shall be used!
+    MatchSpec = dbg:fun2ms(fun(_) -> return_trace() end),
+    ttb:tpl(MFA,MatchSpec).
+
+%%% Format a binary trace log
+format(File) ->
+    ttb:format(File).
+
+%%% Stop the "mydebug" tool
+stop() ->
+    ttb:stop().
+
+%%% --------Internal functions--------
+%%% ----------------------------------
+%%% Format handler
+print(_Out,end_of_trace,_TI,N) ->
+    N;
+print(Out,Trace,_TI,N) ->
+    do_print(Out,Trace,N),
+    N+1.
+
+do_print(Out,{trace_ts,P,call,{M,F,A},Ts},N) ->
+    io:format(Out,
+              "~w: ~w, ~w:~n"
+              "Call      : ~w:~w/~w~n"
+              "Arguments :~p~n~n",
+              [N,Ts,P,M,F,length(A),A]);
+do_print(Out,{trace_ts,P,return_from,{M,F,A},R,Ts},N) ->
+    io:format(Out,
+              "~w: ~w, ~w:~n"
+              "Return from  : ~w:~w/~w~n"
+              "Return value :~p~n~n",
+              [N,Ts,P,M,F,A,R]).      ]]></code>
+      <p>To distinguish trace logs produced with this tool from other
+        logs, the <c>file</c> option is used in <c>tracer/2</c>. The
+        logs will therefore be named <c>Node-debug_log</c>, where
+        <c>Node</c> is the name of the node where the log is produced.
+        </p>
+      <p>By using the <c>handler</c> option when starting the tracer,
+        the information about how to format the file is stored in the
+        trace information file (<c>.ti</c>). This is not necessary, as
+        it might be given at the time of formatting instead. It can
+        however be useful if you e.g. want to automatically format your
+        trace logs by using the <c>format</c> option in
+        <c>ttb:stop/1</c>. It also means that you don't need any
+        knowledge of the content of a binary log to be able to format it
+        the way it was intended. If the <c>handler</c> option is given
+        both when starting the tracer and when formatting, the one given
+        when formatting is used.
+        </p>
+      <p>The <c>call</c> trace flag is set on all processes. This
+        means that any function activated with the <c>trc/1</c> command
+        will be traced on all existing and new processes.
+        </p>
+    </section>
+  </section>
+
+  <section>
+    <title>Running the Trace Tool Builder against a remote node</title>
+    <p>The Observer application might not always be available on the
+      node that shall be traced (in the following called the "traced
+      node"). It is still possible to run the Trace Tool Builder from
+      another node (in the following called the "trace control node") as
+      long as
+      </p>
+    <list type="bulleted">
+      <item>The Observer application is available on the trace control node.</item>
+      <item>The Runtime Tools application is available on both the
+       trace control node and the traced node.</item>
+    </list>
+    <p>If the Trace Tool Builder shall be used against a remote node,
+      it is highly recommended to start the trace control node as
+      <em>hidden</em>. This way it can connect to the traced node
+      without the traced node "seeing" it, i.e. if the <c>nodes()</c>
+      BIF is called on the traced node, the trace control node will not
+      show. To start a hidden node, add the <c>-hidden</c> option to the
+      <c>erl</c> command, e.g.</p>
+    <code type="none">
+% erl -sname trace_control -hidden    </code>
+
+    <section>
+      <title>Diskless node</title>
+      <p>If the traced node is diskless, <c>ttb</c> must be started from
+        a trace control node with disk access, and the <c>file</c> option
+        must be given to the <c>tracer/2</c> function with the value
+        <c>{local, File}</c>, e.g.</p>
+      <code type="none">
+(trace_control@durin)1> ttb:tracer(mynode@diskless,[{file,{local,
+{wrap,"mytrace"}}}]).
+{ok,[mynode@diskless]}      </code>
+    </section>
+  </section>
+
+  <section>
+    <marker id="trace_info"></marker>
+    <title>Trace Information and the .ti File</title>
+    <p>In addition to the trace log file(s), a file with the extension
+      <c>.ti</c> is created when the Trace Tool Builder is started. This
+      is the trace information file. It is a binary file, and it
+      contains the process information, trace flags used, the name of
+      the node to which it belongs and all information written with the
+      <c>write_trace_info/2</c> function.
+      </p>
+    <p>To be able to use all this information during formatting, it is
+      important that the trace information file exists in the same
+      directory as the trace log, and that it has the same name as the
+      trace log with the additional extension <c>.ti</c>.
+      </p>
+    <p>Except for the process information, everything in the trace
+      information file is passed on to the handler function when
+      formatting. The <c>TI</c> parameter is a list of
+      <c>{Key,ValueList}</c> tuples. The keys <c>flags</c>,
+      <c>handler</c>, <c>file</c> and <c>node</c> are used for
+      information written directly by <c>ttb</c>.
+      </p>
+    <p>You can add information to the trace information file by
+      calling <c>write_trace_info/2</c>. Note that <c>ValueList</c>
+      always will be a list, and if you call <c>write_trace_info/2</c>
+      several times with the same <c>Key</c>, the <c>ValueList</c> will
+      be extended with a new value each time. Example:
+      </p>
+    <p><c>ttb:write_trace_info(mykey,1)</c> gives the entry
+      <c>{mykey,[1]}</c> in <c>TI</c>. Another call,
+      <c>ttb:write_trace_info(mykey,2)</c>, changes this entry to
+      <c>{mykey,[1,2]}</c>.
+      </p>
+  </section>
+
+  <section>
+    <title>Wrap Logs</title>
+    <p>If you want to limit the size of the trace logs, you can use
+      wrap logs. This works almost like a circular buffer. You can
+      specify the maximum number of binary logs and the maximum size of
+      each log. <c>ttb</c> will create a new binary log each time a log
+      reaches the maximum size. When the the maximum number of logs are
+      reached, the oldest log is deleted before a new one is created.
+      </p>
+    <p>Wrap logs can be formatted one by one or all at once. See
+      <seealso marker="#format">Formatting</seealso>.
+      </p>
+  </section>
+
+  <section>
+    <marker id="format"></marker>
+    <title>Formatting</title>
+    <p>Formatting can be done automatically when stopping <c>ttb</c>
+      (see <seealso marker="#fetch_format">Automatically collect and format logs from all nodes</seealso>), or explicitly by calling
+      the <c>ttb:format/1/2</c> function.
+      </p>
+    <p>Formatting means to read a binary log and present it in a
+      readable format. You can use the default format handler in
+      <c>ttb</c> to present each trace message as a line of text, or
+      write your own handler to make more complex interpretations of the
+      trace information. You can even use the Event Tracer <c>et</c> to
+      present the trace log graphically (see <seealso marker="#et_viewer">Presenting trace logs with Event Tracer</seealso>).
+      </p>
+    <p>The first argument to <c>ttb:format/1/2</c> specifies which
+      binary log(s) to format. This can be the name of one binary log, a
+      list of such logs or the name of a directory containing one or
+      more binary logs. If this argument indicates more than one log,
+      and the <c>timestamp</c> flag was set when tracing, the trace
+      messages from the different logs will be merged according to the
+      timestamps in each message.
+      </p>
+    <p>The second argument to <c>ttb:format/2</c> is a list of
+      options. The <c>out</c> option specifies the destination where the
+      formatted text shall be written. Default destination is
+      <c>standard_io</c>, but a filename can also be given. The
+      <c>handler</c> option specifies the format handler to use. If this
+      option is not given, the <c>handler</c> option given when starting
+      the tracer is used. If the <c>handler</c> option was not given
+      when starting the tracer either, a default handler is used, which
+      prints each trace message as a line of text.
+      </p>
+    <p>A format handler is a fun taking four arguments. This fun will
+      be called for each trace message in the binary log(s). A simple
+      example which only prints each trace message could be like this:</p>
+    <code type="none">
+fun(Fd, Trace, _TraceInfo, State) ->
+   io:format(Fd, "Trace: ~p~n", [Trace]),
+   State
+end.    </code>
+    <p><c>Fd</c> is the file descriptor for the destination file, or
+      the atom <c>standard_io</c>. <c>_TraceInfo</c> contains information
+      from the trace information file (see <seealso marker="#trace_info">Trace Information and the .ti File</seealso>). <c>State</c> is a state variable for the format
+      handler fun. The initial value of the <c>State</c> variable is
+      given with the handler option, e.g.</p>
+    <code type="none">
+ttb:format("tiger@durin-ttb", [{handler, {{Mod,Fun}, initial_state}}])
+                                                     ^^^^^^^^^^^^^    </code>
+    <p>Another format handler could be used to calculate time spent by
+      the garbage collector:</p>
+    <code type="none">
+fun(_Fd,{trace_ts,P,gc_start,_Info,StartTs},_TraceInfo,State) ->
+      [{P,StartTs}|State];
+   (Fd,{trace_ts,P,gc_end,_Info,EndTs},_TraceInfo,State) ->
+      {value,{P,StartTs}} = lists:keysearch(P,1,State),
+      Time = diff(StartTs,EndTs),
+      io:format("GC in process ~w: ~w milliseconds~n", [P,Time]),
+      State -- [{P,StartTs}]
+end    </code>
+    <p>A more refined version of this format handler is the function
+      <c>handle_gc/4</c> in the module <c>multitrace.erl</c> which can
+      be found in the <c>src</c> directory of the Observer application.
+      </p>
+    <p>By giving the format handler <c>et</c>, you can have the trace
+      log presented graphically with <c>et_viewer</c> in the Event
+      Tracer application (see <seealso marker="#et_viewer">Presenting trace logs with Event Tracer</seealso>).
+      </p>
+    <p>Wrap logs can be formatted one by one or all in one go. To
+      format one of the wrap logs in a set, give the exact name of the
+      file. To format the whole set of wrap logs, give the name with '*'
+      instead of the wrap count. An example:
+      </p>
+    <p>Start tracing:</p>
+    <code type="none">
+(tiger@durin)1> ttb:tracer(node(),[{file,{wrap,"trace"}}]).
+{ok,[tiger@durin]}
+(tiger@durin)2> ttb:p(...)
+...    </code>
+    <p>This will give a set of binary logs, like:</p>
+    <code type="none">
[email protected]
[email protected]
[email protected]
+...    </code>
+    <p>Format the whole set of logs:</p>
+    <code type="none">
+1> ttb:format("tiger@durin-trace.*.wrp").
+....
+ok
+2>    </code>
+    <p>Format only the first log:</p>
+    <code type="none">
+1> ttb:format("[email protected]").
+....
+ok
+2>    </code>
+    <p>To merge all wrap logs from two nodes:</p>
+    <code type="none">
+1> ttb:format(["tiger@durin-trace.*.wrp","lion@durin-trace.*.wrp"]).
+....
+ok
+2>    </code>
+
+    <section>
+      <marker id="et_viewer"></marker>
+      <title>Presenting trace logs with Event Tracer</title>
+      <p>For detailed information about the Event Tracer, please turn
+        to the User's Guide and Reference Manuals for the <c>et</c>
+        application.
+        </p>
+      <p>By giving the format handler <c>et</c>, you can have the
+        trace log presented graphically with <c>et_viewer</c> in the
+        Event Tracer application. <c>ttb</c> provides a few different
+        filters which can be selected from the Filter menu in the
+        <c>et_viewer</c> window. The filters are names according to the
+        type of actors they present (i.e. what each vertical line in the
+        sequence diagram represent). Interaction between actors is shown
+        as red arrows between two vertical lines, and activities within
+        an actor are shown as blue text to the right of the actors line.
+        </p>
+      <p>The <c>processes</c> filter is the only filter which will
+        show all trace messages from a trace log. Each vertical line in
+        the sequence diagram represents a process. Erlang messages,
+        spawn and link/unlink are typical interactions between
+        processes. Function calls, scheduling and garbage collection are
+        typical activities within a process. <c>processes</c> is the
+        default filter.
+        </p>
+      <p>The rest of the filters will only show function calls and
+        function returns. All other trace message are discarded. To get
+        the most out of these filters, <c>et_viewer</c> needs to known
+        the caller of each function and the time of return. This can be
+        obtained by using both the <c>call</c> and <c>return_to</c>
+        flags when tracing. Note that the <c>return_to</c> flag only
+        works with local call trace, i.e. when trace patterns are set
+        with <c>ttb:tpl</c>.
+        </p>
+      <p>The same result can be obtained by using the <c>call</c> flag
+        only and setting a match specification like this on local or
+        global function calls:</p>
+      <code type="none">
+1> dbg:fun2ms(fun(_) -> return_trace(),message(caller()) end).
+[{'_',[],[{return_trace},{message,{caller}}]}]      </code>
+      <p>This should however be done with care, since the
+        <c>{return_trace}</c> function in the match specification will
+        destroy tail recursiveness.
+        </p>
+      <p>The <c>modules</c> filter shows each module as a vertical
+        line in the sequence diagram. External function calls/returns
+        are shown as interactions between modules and internal function
+        calls/returns are shown as activities within a module.
+        </p>
+      <p>The <c>functions</c> filter shows each function as a vertical
+        line in the sequence diagram. A function calling itself is shown
+        as an activity within a function, and all other function calls
+        are shown as interactions between functions.
+        </p>
+      <p>The <c>mods_and_procs</c> and <c>funcs_and_procs</c> filters
+        are equivalent to the <c>modules</c> and <c>functions</c>
+        filters respectively, except that each module or function can
+        have several vertical lines, one for each process it resides on.
+        </p>
+      <p>As an example this module is used, and the function
+        <c>bar:f1()</c> is called from another module <c>foo</c>.</p>
+      <code type="none">
+-module(bar).
+-export([f1/0,f3/0]).
+f1() ->
+    f2(),
+    ok.
+f2() ->
+    spawn(?MODULE,f3,[]).
+f3() ->
+    ok.      </code>
+      <p>The <c>call</c> and <c>return_to</c> flags are used, and
+        trace pattern is set on local calls in module <c>bar</c>.
+        </p>
+      <p><c>ttb:format("tiger@durin-ttb", [{handler, et}])</c> gives the
+        following result:
+        </p>
+      <p></p>
+      <image file="et_processes.gif">
+        <icaption>Filter: "processes"</icaption>
+      </image>
+      <image file="et_modsprocs.gif">
+        <icaption>Filter: "mods_and_procs"</icaption>
+      </image>
+    </section>
+  </section>
+
+  <section>
+    <marker id="fetch_format"></marker>
+    <title>Automatically collect and format logs from all nodes</title>
+    <p>If the option <c>fetch</c> is given to the <c>ttb:stop/1</c>
+      function, trace logs and trace information files are fetched
+      from all nodes after tracing is stopped. The logs are stored in a
+      new directory named <c>ttb_upload-Timestamp</c> under the working
+      directory of the trace control node.
+      </p>
+    <p>If the option <c>format</c> is given to <c>ttb:stop/1</c>, the
+      trace logs are automatically formatted after tracing is
+      stopped. Note that <c>format</c> also implies <c>fetch</c>,
+      i.e. the trace logs will be collected from all nodes as for the
+      <c>fetch</c> option before they are formatted. All logs in the
+      upload directory are merged during formatting.
+      </p>
+  </section>
+
+  <section>
+    <title>History and Configuration Files</title>
+    <p>For the tracing functionality, <c>dbg</c> could be used instead
+      of the <c>ttb</c> for setting trace flags on processes and trace
+      patterns for call trace, i.e. the functions <c>p</c>, <c>tp</c>,
+      <c>tpl</c>, <c>ctp</c>, <c>ctpl</c> and <c>ctpg</c>. The only
+      thing added by <c>ttb</c> for these functions is that all calls
+      are stored in the history buffer and can be recalled and stored in
+      a configuration file. This makes it easy to setup the same trace
+      environment e.g. if you want to compare two test runs. It also
+      reduces the amount of typing when using <c>ttb</c> from the erlang
+      shell.
+      </p>
+    <p>Use <c>list_history/0</c> to see the content of the history
+      buffer, and <c>run_history/1</c> to re-execute one of the entries.
+      </p>
+    <p>The main purpose of the history buffer is the possibility to
+      create configuration files. Any function stored in the history
+      buffer can be written to a configuration file and used for
+      creating a specific configuration at any time with one single
+      function call.
+      </p>
+    <p>A configuration file is created or extended with
+      <c>write_config/2/3</c>. Configuration files are binary files
+      and can therefore only be read and written with functions provided
+      by <c>ttb</c>.
+      </p>
+    <p>You can write the complete content of the history buffer to a
+      config file by calling
+      <c>ttb:write_config(ConfigFile,all)</c>. And you can write
+      selected entries from the history by calling
+      <c>ttb:write_config(ConfigFile,NumList)</c>, where
+      <c>NumList</c> is a list of integers pointing out the history
+      entries to write.
+      </p>
+    <p>User defined entries can also be written to a config file by
+      calling the function
+      <c>ttb:write_config(ConfigFile,ConfigList)</c> where
+      <c>ConfigList</c> is a list of <c>{Module,Function,Args}</c>.
+      </p>
+    <p>Any existing file <c>ConfigFile</c> is deleted and a new file
+      is created when <c>write_config/2</c> is called. The option
+      <c>append</c> can be used if you wish to add something at the end
+      of an existing config file, e.g.
+      <c>ttb:write_config(ConfigFile,What,[append])</c>.
+      </p>
+
+    <section>
+      <title>Example: History and configuration files</title>
+      <p>See the content of the history buffer</p>
+      <code type="none"><![CDATA[
+(tiger@durin)191> ttb:tracer().                                    
+{ok,[tiger@durin]}
+(tiger@durin)192> ttb:p(self(),[garbage_collection,call]).               
+{ok,{[<0.1244.0>],[garbage_collection,call]}}
+(tiger@durin)193> ttb:tp(ets,new,2,[]).                                  
+{ok,[{matched,1}]}
+(tiger@durin)194> ttb:list_history().
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}}]      ]]></code>
+      <p>Execute an entry from the history buffer:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)195> ttb:ctp(ets,new,2).
+{ok,[{matched,1}]}
+(tiger@durin)196> ttb:list_history().
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}},
+ {4,{ttb,ctp,[ets,new,2]}}]
+(tiger@durin)197> ttb:run_history(3).
+ttb:tp(ets,new,2,[]) ->
+{ok,[{matched,1}]}      ]]></code>
+      <p>Write the content of the history buffer to a configuration
+        file:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)198> ttb:write_config("myconfig",all).
+ok
+(tiger@durin)199> ttb:list_config("myconfig").
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}},
+ {4,{ttb,ctp,[ets,new,2]}},
+ {5,{ttb,tp,[ets,new,2,[]]}}]      ]]></code>
+      <p>Extend an existing configuration:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)200> ttb:write_config("myconfig",[{ttb,tp,[ets,delete,1,[]]}],
+[append]).
+ok
+(tiger@durin)201> ttb:list_config("myconfig").
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}},
+ {4,{ttb,ctp,[ets,new,2]}},
+ {5,{ttb,tp,[ets,new,2,[]]}},
+ {6,{ttb,tp,[ets,delete,1,[]]}}]      ]]></code>
+      <p>Go back to a previous configuration after stopping Trace Tool
+        Builder:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)202> ttb:stop().
+ok
+(tiger@durin)203> ttb:run_config("myconfig").
+ttb:tracer(tiger@durin,[]) ->
+{ok,[tiger@durin]}
+
+ttb:p(<0.1244.0>,[garbage_collection,call]) ->
+{ok,{[<0.1244.0>],[garbage_collection,call]}}
+
+ttb:tp(ets,new,2,[]) ->
+{ok,[{matched,1}]}
+
+ttb:ctp(ets,new,2) ->
+{ok,[{matched,1}]}
+
+ttb:tp(ets,new,2,[]) ->
+{ok,[{matched,1}]}
+
+ttb:tp(ets,delete,1,[]) ->
+{ok,[{matched,1}]}
+
+ok      ]]></code>
+      <p>Write selected entries from the history buffer to a
+        configuration file:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)204> ttb:list_history().          
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}},
+ {4,{ttb,ctp,[ets,new,2]}},
+ {5,{ttb,tp,[ets,new,2,[]]}},
+ {6,{ttb,tp,[ets,delete,1,[]]}}]
+(tiger@durin)205> ttb:write_config("myconfig",[1,2,3,6]).
+ok
+(tiger@durin)206> ttb:list_config("myconfig").
+[{1,{ttb,tracer,[tiger@durin,[]]}},
+ {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
+ {3,{ttb,tp,[ets,new,2,[]]}},
+ {4,{ttb,tp,[ets,delete,1,[]]}}]
+(tiger@durin)207>       ]]></code>
+    </section>
+  </section>
+
+  <section>
+    <title>Sequential Tracing</title>
+    <p>To learn what sequential tracing is and how it can be used,
+      please turn to the reference manual for the
+      <em><c>seq_trace</c></em> module in the <em><c>kernel</c></em>
+      application.
+      </p>
+    <p>The support for sequential tracing provided by the Trace Tool
+      Builder includes </p>
+    <list type="bulleted">
+      <item>Initiation of the system tracer. This is automatically
+       done when a trace port is started with <c>ttb:tracer/0/1/2</c></item>
+      <item>Creation of match specifications which activates
+       sequential tracing</item>
+    </list>
+    <p>Starting sequential tracing requires that a tracer has been
+      started with the <c>ttb:tracer/0/1/2</c> function. Sequential
+      tracing can then either be started via a trigger function with a
+      match specification created with <c>ttb:seq_trigger_ms/0/1</c>,
+      or directly by using the <c>seq_trace</c> module in the
+      <c>kernel</c> application.
+      </p>
+
+    <section>
+      <title>Example: Sequential tracing</title>
+      <p>In the following example, the function
+        <c>dbg:get_tracer/0</c> is used as trigger for sequential
+        tracing:</p>
+      <code type="none"><![CDATA[
+(tiger@durin)110> ttb:tracer().                               
+{ok,[tiger@durin]}
+(tiger@durin)111> ttb:p(self(),call).                               
+{ok,{[<0.158.0>],[call]}}
+(tiger@durin)112> ttb:tp(dbg,get_tracer,0,ttb:seq_trigger_ms(send)).
+{ok,[{matched,1},{saved,1}]}
+(tiger@durin)113> dbg:get_tracer(), seq_trace:reset_trace().         
+true
+(tiger@durin)114> ttb:stop().                                       
+ok
+(tiger@durin)115> ttb:format("tiger@durin-ttb").                    
+({<0.158.0>,{shell,evaluator,3},tiger@durin}) call dbg:get_tracer()
+SeqTrace [0]: ({<0.158.0>,{shell,evaluator,3},tiger@durin}) 
+{<0.237.0>,dbg,tiger@durin} ! {<0.158.0>,{get_tracer,tiger@durin}} 
+[Serial: {0,1}]
+SeqTrace [0]: ({<0.237.0>,dbg,tiger@durin}) 
+{<0.158.0>,{shell,evaluator,3},tiger@durin} ! {dbg,{ok,#Port<0.222>}} 
+[Serial: {1,2}]
+ok
+(tiger@durin)116>       ]]></code>
+      <p>Starting sequential tracing with a trigger is actually more
+        useful if the trigger function is not called directly from the
+        shell, but rather implicitly within a larger system. When
+        calling a function from the shell, it is simpler to start
+        sequential tracing directly, e.g.</p>
+      <code type="none"><![CDATA[
+(tiger@durin)116> ttb:tracer().                               
+{ok,[tiger@durin]}
+(tiger@durin)117> seq_trace:set_token(send,true), dbg:get_tracer(), 
+seq_trace:reset_trace().
+true
+(tiger@durin)118> ttb:stop().
+ok
+(tiger@durin)119> ttb:format("tiger@durin-ttb").
+SeqTrace [0]: ({<0.158.0>,{shell,evaluator,3},tiger@durin}) 
+{<0.246.0>,dbg,tiger@durin} ! {<0.158.0>,{get_tracer,tiger@durin}} 
+[Serial: {0,1}]
+SeqTrace [0]: ({<0.246.0>,dbg,tiger@durin}) 
+{<0.158.0>,{shell,evaluator,3},tiger@durin} ! {dbg,{ok,#Port<0.229>}} 
+[Serial: {1,2}]
+ok
+(tiger@durin)120>       ]]></code>
+      <p>In both examples above, the <c>seq_trace:reset_trace/0</c>
+        resets the trace token immediately after the traced function in
+        order to avoid lots of trace messages due to the printouts in
+        the erlang shell.
+        </p>
+      <p>All functions in the <c>seq_trace</c> module, except
+        <c>set_system_tracer/1</c>, can be used after the trace port has
+        been started with <c>ttb:tracer/0/1/2</c>.
+        </p>
+    </section>
+  </section>
+
+  <section>
+    <title>Example: Multipurpose trace tool</title>
+    <p>The module <c>multitrace.erl</c> which can be found in the
+      <c>src</c> directory of the Observer application implements a
+      small tool with three possible trace settings. The trace messages
+      are written to binary files which can be formatted with the
+      function <em><c>multitrace:format/1/2</c></em>.
+      </p>
+    <taglist>
+      <tag><em><c>multitrace:debug(What)</c></em></tag>
+      <item>Start calltrace on all processes and trace the given
+       function(s). The format handler used is
+      <c>multitrace:handle_debug/4</c> which prints each call and
+       return. <c>What</c> must be an item or a list of items to trace,
+       given on the format <c>{Module,Function,Arity}</c>,
+      <c>{Module,Function}</c> or just <c>Module</c>.</item>
+      <tag><em><c>multitrace:gc(Procs)</c></em></tag>
+      <item>Trace garbage collection on the given process(es). The
+       format handler used is <c>multitrace:handle_gc/4</c> which
+       prints start and stop and the time spent for each GC.</item>
+      <tag><em><c>multitrace:schedule(Procs)</c></em></tag>
+      <item>Trace in- and out-scheduling on the given process(es). The
+       format handler used is <c>multitrace:handle_schedule/4</c> which
+       prints each in and out scheduling with process, timestamp and
+       current function. It also prints the total time each traced
+       process was scheduled in.</item>
+    </taglist>
+  </section>
+</chapter>
+