The Trace Tool Builder is a base for building trace tools for single node or distributed erlang systems. It requires the runtime_tools application to be available on the traced node.

The main features of the Trace Tool Builder are:

The intention of the Trace Tool Builder is to serve as a base for tailor made trace tools, but you may use it directly from the erlang shell (it may mimic dbg behaviour while still providing useful additions like match specification shortcuts). The application only allows the use of file port tracer, so if you would like to use other types of trace clients you will be better off using dbg directly instead.

The ttb module is the interface to all functions in the Trace Tool Builder. To get started the least you need to do is to start a tracer with ttb:tracer/0/1/2, and set the required trace flags on the processes you want to trace with ttb:p/2. Then, when the tracing is completed, you must stop the tracer with ttb:stop/0/1 and format the trace log with ttb:format/1/2 (as long as there is anything to format, of course).

ttb:tracer/0/1/2 opens a trace port on each node that shall be traced. By default, trace messages are written to binary files on remote nodes(the binary trace log).

ttb:p/2 specifies which processes shall be traced. Trace flags given in this call specify what to trace on each process. You can call this function several times if you like different trace flags to be set on different processes.

If you want to trace function calls (i.e. if you have the call trace flag set on any of your processes), you must also set trace patterns on the required function(s) with ttb:tp or ttb:tpl. A function is only traced if it has a trace pattern. The trace pattern specifies how to trace the function by using match specifications. Match specifications are described in the User's Guide for the erlang runtime system erts.

ttb:stop/0/1 stops tracing on all nodes, deletes all trace patterns and flushes the trace port buffer.

ttb:format/1/2 translates the binary trace logs into something readable. By default ttb presents each trace message as a line of text, but you can also write your own handler to make more complex interpretations of the trace information. A trace log can even be presented graphically via the Event Tracer application. Note that if you give the format option to ttb:stop/1 the formatting is automatically done when stopping ttb.

The Observer application might not always be available on the node that shall be traced (in the following called the "traced node"). It is still possible to run the Trace Tool Builder from another node (in the following called the "trace control node") as long as

If the Trace Tool Builder shall be used against a remote node, it is highly recommended to start the trace control node as hidden. This way it can connect to the traced node without the traced node "seeing" it, i.e. if the nodes() BIF is called on the traced node, the trace control node will not show. To start a hidden node, add the -hidden option to the erl command, e.g.

When setting up a trace, several features may be turned on:

In addition to the trace log file(s), a file with the extension .ti is created when the Trace Tool Builder is started. This is the trace information file. It is a binary file, and it contains the process information, trace flags used, the name of the node to which it belongs and all information written with the write_trace_info/2 function. .ti files are always fetched with other logs when the trace is stopped.

Except for the process information, everything in the trace information file is passed on to the handler function when formatting. The TI parameter is a list of {Key,ValueList} tuples. The keys flags, handler, file and node are used for information written directly by ttb.

You can add information to the trace information file by calling write_trace_info/2. Note that ValueList always will be a list, and if you call write_trace_info/2 several times with the same Key, the ValueList will be extended with a new value each time. Example:

ttb:write_trace_info(mykey,1) gives the entry {mykey,[1]} in TI. Another call, ttb:write_trace_info(mykey,2), changes this entry to {mykey,[1,2]}.

If you want to limit the size of the trace logs, you can use wrap logs. This works almost like a circular buffer. You can specify the maximum number of binary logs and the maximum size of each log. ttb will create a new binary log each time a log reaches the maximum size. When the the maximum number of logs are reached, the oldest log is deleted before a new one is created.

Note that the overall size of data generated by ttb may be greater than the wrap specification would suggest - if a traced node restarts and autoresume is enabled, old wrap log is always stored and a new one is created.

Wrap logs can be formatted one by one or all at once. See Formatting.

Formatting can be done automatically when stopping ttb (see Automatically collect and format logs from all nodes), or explicitly by calling the ttb:format/1/2 function.

Formatting means to read a binary log and present it in a readable format. You can use the default format handler in ttb to present each trace message as a line of text, or write your own handler to make more complex interpretations of the trace information. You can even use the Event Tracer et to present the trace log graphically (see Presenting trace logs with Event Tracer).

The first argument to ttb:format/1/2 specifies which binary log(s) to format. This is usually the name of a directory that ttb created during log fetch. Unless there is the disable_sort option provided, the logs from different files are always sorted according to timestamp in traces.

The second argument to ttb:format/2 is a list of options. The out option specifies the destination where the formatted text shall be written. Default destination is standard_io, but a filename can also be given. The handler option specifies the format handler to use. If this option is not given, the handler option given when starting the tracer is used. If the handler option was not given when starting the tracer either, a default handler is used, which prints each trace message as a line of text. The disable_sort option indicates that there logs should not be merged according to timestamp, but processed one file after another (this might be a bit faster).

A format handler is a fun taking four arguments. This fun will be called for each trace message in the binary log(s). A simple example which only prints each trace message could be like this:

Fd is the file descriptor for the destination file, or the atom standard_io. _TraceInfo contains information from the trace information file (see Trace Information and the .ti File). State is a state variable for the format handler fun. The initial value of the State variable is given with the handler option, e.g.

Another format handler could be used to calculate time spent by the garbage collector:

A more refined version of this format handler is the function handle_gc/4 in the module multitrace.erl which can be found in the src directory of the Observer application.

The actual trace message is passed as the second argument (Trace). The possible values of Trace are:

By giving the format handler ttb:get_et_handler(), you can have the trace log presented graphically with et_viewer in the Event Tracer application (see Presenting trace logs with Event Tracer).

You may always decide not to format the whole trace data contained in the fetch directory, but analyze single files instead. In order to do so, a single file (or list of files) have to be passed as the first argument to format/1/2.

Wrap logs can be formatted one by one or all in one go. To format one of the wrap logs in a set, give the exact name of the file. To format the whole set of wrap logs, give the name with '*' instead of the wrap count. An example:

Start tracing:

This will give a set of binary logs, like:

Format the whole set of logs:

Format only the first log:

To merge all wrap logs from two nodes:

For detailed information about the Event Tracer, please turn to the User's Guide and Reference Manuals for the et application.

By giving the format handler ttb:get_et_handler(), you can have the trace log presented graphically with et_viewer in the Event Tracer application. ttb provides a few different filters which can be selected from the Filter menu in the et_viewer window. The filters are names according to the type of actors they present (i.e. what each vertical line in the sequence diagram represent). Interaction between actors is shown as red arrows between two vertical lines, and activities within an actor are shown as blue text to the right of the actors line.

The processes filter is the only filter which will show all trace messages from a trace log. Each vertical line in the sequence diagram represents a process. Erlang messages, spawn and link/unlink are typical interactions between processes. Function calls, scheduling and garbage collection are typical activities within a process. processes is the default filter.

The rest of the filters will only show function calls and function returns. All other trace message are discarded. To get the most out of these filters, et_viewer needs to known the caller of each function and the time of return. This can be obtained by using both the call and return_to flags when tracing. Note that the return_to flag only works with local call trace, i.e. when trace patterns are set with ttb:tpl.

The same result can be obtained by using the call flag only and setting a match specification like this on local or global function calls:

1> dbg:fun2ms(fun(_) -> return_trace(),message(caller()) end). [{'_',[],[{return_trace},{message,{caller}}]}]

This should however be done with care, since the {return_trace} function in the match specification will destroy tail recursiveness.

The modules filter shows each module as a vertical line in the sequence diagram. External function calls/returns are shown as interactions between modules and internal function calls/returns are shown as activities within a module.

The functions filter shows each function as a vertical line in the sequence diagram. A function calling itself is shown as an activity within a function, and all other function calls are shown as interactions between functions.

The mods_and_procs and funcs_and_procs filters are equivalent to the modules and functions filters respectively, except that each module or function can have several vertical lines, one for each process it resides on.

In the next example, modules foo and bar are used:

-module(foo). -export([start/0,go/0]). start() -> spawn(?MODULE, go, []). go() -> receive stop -> ok; go -> bar:f1(), go() end. -module(bar). -export([f1/0,f3/0]). f1() -> f2(), ok. f2() -> spawn(?MODULE,f3,[]). f3() -> ok.

Now let's set up the trace.

(tiger@durin)1>%%First we retrieve the Pid to limit traced processes set (tiger@durin)1>Pid = foo:start(). (tiger@durin)2>%%Now we set up tracing (tiger@durin)2>ttb:tracer(). (tiger@durin)3>ttb:p(Pid, [call, return_to, procs, set_on_spawn]). (tiger@durin)4>ttb:tpl(bar, []). (tiger@durin)5>%%Invoke our test function and see output with et viewer (tiger@durin)5>Pid ! go. (tiger@durin)6>ttb:stop({format, {handler, ttb:get_et_handler()}}).

This should render a result similar to the following:

Filter: "processes" Filter: "mods_and_procs"

Note, that we can use ttb:start_trace/4 function to help us here:

(tiger@durin)1>Pid = foo:start(). (tiger@durin)2>ttb:start_trace([node()], [{bar,[]}], {Pid, [call, return_to, procs, set_on_spawn]} {handler, ttb:get_et_handler()}). (tiger@durin)3>Pid ! go. (tiger@durin)4>ttb:stop(format).

By default ttb:stop/1 fetches trace logs and trace information files from all nodes. The logs are stored in a new directory named ttb_upload-Filename-Timestamp under the working directory of the trace control node. Fetching may be disabled by providing the nofetch option to ttb:stop/1. User can specify a fetch directory of his choice passing the {fetch_dir, Dir} option.

If the option format is given to ttb:stop/1, the trace logs are automatically formatted after tracing is stopped.

For the tracing functionality, dbg could be used instead of the ttb for setting trace flags on processes and trace patterns for call trace, i.e. the functions p, tp, tpl, ctp, ctpl and ctpg. There are only two things added by ttb for these functions: all calls are stored in the history buffer and can be recalled and stored in a configuration file. This makes it easy to setup the same trace environment e.g. if you want to compare two test runs. It also reduces the amount of typing when using ttb from the erlang shell; shortcuts are provided for the most common match specifications (in order not to force the user to use dbg:fun2ms continually).

Use list_history/0 to see the content of the history buffer, and run_history/1 to re-execute one of the entries.

The main purpose of the history buffer is the possibility to create configuration files. Any function stored in the history buffer can be written to a configuration file and used for creating a specific configuration at any time with one single function call.

A configuration file is created or extended with write_config/2/3. Configuration files are binary files and can therefore only be read and written with functions provided by ttb.

You can write the complete content of the history buffer to a config file by calling ttb:write_config(ConfigFile,all). And you can write selected entries from the history by calling ttb:write_config(ConfigFile,NumList), where NumList is a list of integers pointing out the history entries to write. Moreover, the history buffer is always dumped to ttb_last_config when ttb:stop/0/1 is called.

User defined entries can also be written to a config file by calling the function ttb:write_config(ConfigFile,ConfigList) where ConfigList is a list of {Module,Function,Args}.

Any existing file ConfigFile is deleted and a new file is created when write_config/2 is called. The option append can be used if you wish to add something at the end of an existing config file, e.g. ttb:write_config(ConfigFile,What,[append]).

To learn what sequential tracing is and how it can be used, please turn to the reference manual for the seq_trace module in the kernel application.

The support for sequential tracing provided by the Trace Tool Builder includes

Starting sequential tracing requires that a tracer has been started with the ttb:tracer/0/1/2 function. Sequential tracing can then either be started via a trigger function with a match specification created with ttb:seq_trigger_ms/0/1, or directly by using the seq_trace module in the kernel application.

The module multitrace.erl which can be found in the src directory of the Observer application implements a small tool with three possible trace settings. The trace messages are written to binary files which can be formatted with the function multitrace:format/1/2.