aboutsummaryrefslogtreecommitdiffstats
path: root/lib/observer/doc/src/ttb_ug.xml
blob: 08093a9451094bd4dd75b7760fd60e71a7c1174e (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">

<chapter>
  <header>
    <copyright>
      <year>2002</year><year>2011</year>
      <holder>Ericsson AB. All Rights Reserved.</holder>
    </copyright>
    <legalnotice>
      The contents of this file are subject to the Erlang Public License,
      Version 1.1, (the "License"); you may not use this file except in
      compliance with the License. You should have received a copy of the
      Erlang Public License along with this software. If not, it can be
      retrieved online at http://www.erlang.org/.
    
      Software distributed under the License is distributed on an "AS IS"
      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
      the License for the specific language governing rights and limitations
      under the License.
    
    </legalnotice>

    <title>Trace Tool Builder</title>
    <prepared></prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
    <file>ttb_ug.xml</file>
  </header>

  <section>
    <title>Introduction</title>
    <p>The Trace Tool Builder is a base for building trace tools for
      single node or distributed erlang systems. It requires the
      <c>runtime_tools</c> application to be available on the traced
      node.
      </p>
    <p>The main features of the Trace Tool Builder are:</p>
    <list type="bulleted">
      <item>Start tracing to file ports on several nodes with one
       function call.</item>
      <item>Write additional information to a trace information file,
       which is read during formatting.</item>
      <item>Restoring of previous configuration by maintaining a
       history buffer and handling configuration files.</item>
      <item>Some simple support for sequential tracing.</item>
      <item>Formatting of binary trace logs and merging of logs from
       multiple nodes.</item>
    </list>
    <p>The intention of the Trace Tool Builder is to serve
      as a base for tailor made trace tools, but you may use it directly
      from the erlang shell (it may mimic <c>dbg</c> behaviour while
      still providing useful additions like match specification shortcuts).
      The application only
      allows the use of file port tracer, so if you would like
      to use other types of trace clients you will be better off
      using <c>dbg</c> directly instead.</p>
  </section>

  <section>
    <title>Getting Started</title>
    <p>The <c>ttb</c> module is the interface to all functions in the
      Trace Tool Builder. To get started the least you need to do is to
      start a tracer with <c>ttb:tracer/0/1/2</c>, and set the required
      trace flags on the processes you want to trace with
      <c>ttb:p/2</c>. Then, when the tracing is completed, you must stop
      the tracer with <c>ttb:stop/0/1</c> and format the trace log with
      <c>ttb:format/1/2</c> (as long as there is anything to format, of
      course).
      </p>
    <p><c>ttb:tracer/0/1/2</c> opens a trace port on each node
      that shall be traced. By default, trace messages are written
      to binary files on remote nodes(the binary trace log).
      </p>
    <p><c>ttb:p/2</c> specifies which processes shall be
      traced. Trace flags given in this call specify what to trace on
      each process. You can call this function several times if you like
      different trace flags to be set on different processes.
      </p>
    <p>If you want to trace function calls (i.e. if you have the
      <c>call</c> trace flag set on any of your processes), you must
      also set trace patterns on the required function(s) with
      <c>ttb:tp</c> or <c>ttb:tpl</c>. A function is only traced if it
      has a trace pattern. The trace pattern specifies how to trace the
      function by using match specifications. Match specifications are
      described in the User's Guide for the erlang runtime system
      <c>erts</c>.
      </p>
    <p><c>ttb:stop/0/1</c> stops tracing on all nodes, deletes all
      trace patterns and flushes the trace port buffer.
      </p>
    <p><c>ttb:format/1/2</c> translates the binary trace logs into
      something readable. By default <c>ttb</c> presents each trace
      message as a line of text, but you can also write your own handler
      to make more complex interpretations of the trace information. A
      trace log can even be presented graphically via the Event Tracer
      application. Note that if you give the <c>format</c> option to
      <c>ttb:stop/1</c> the formatting is automatically done when
      stopping <c>ttb</c>.
      </p>

    <section>
      <title>Example: Tracing the local node from the erlang shell</title>
      <p>This small module is used in the example:</p>
      <code type="none">
-module(m).
-export([f/0]).
f() ->
   receive 
      From when is_pid(From) ->
         Now = erlang:now(),
         From ! {self(),Now}
   end.      </code>
      <p>The following example shows the basic use of <c>ttb</c> from
        the erlang shell. Default options are used both for starting the
        tracer and for formatting (the custom fetch dir is however provided).
        This gives a trace log named <c>Node-ttb</c> in the newly-created
        directory, where <c>Node</c> is the name of the node.  The
        default handler prints the formatted trace messages in the
        shell.</p>
      <code type="none"><![CDATA[
(tiger@durin)47> %% First I spawn a process running my test function
(tiger@durin)47> Pid = spawn(m,f,[]).
<0.125.0>
(tiger@durin)48> 
(tiger@durin)48> %% Then I start a tracer...
(tiger@durin)48> ttb:tracer().
{ok,[tiger@durin]}
(tiger@durin)49> 
(tiger@durin)49> %% and activate the new process for tracing
(tiger@durin)49> %% function calls and sent messages.
(tiger@durin)49> ttb:p(Pid,[call,send]).
{ok,[{<0.125.0>,[{matched,tiger@durin,1}]}]}
(tiger@durin)50> 
(tiger@durin)50> %% Here I set a trace pattern on erlang:now/0
(tiger@durin)50> %% The trace pattern is a simple match spec
(tiger@durin)50> %% indicating that the return value should be
(tiger@durin)50> %% traced. Refer to the reference_manual for
(tiger@durin)50> %% the full list of match spec shortcuts
(tiger@durin)50> %% available.
(tiger@durin)51> ttb:tp(erlang,now,return).
{ok,[{matched,tiger@durin,1},{saved,1}]}
(tiger@durin)52> 
(tiger@durin)52> %% I run my test (i.e. send a message to
(tiger@durin)52> %% my new process)
(tiger@durin)52> Pid ! self().
<0.72.0>
(tiger@durin)53> 
(tiger@durin)53> %% And then I have to stop ttb in order to flush
(tiger@durin)53> %% the trace port buffer
(tiger@durin)53> ttb:stop([return, {fetch_dir, "fetch"}]).
{stopped, "fetch"}
(tiger@durin)54> 
(tiger@durin)54> %% Finally I format my trace log
(tiger@durin)54> ttb:format("fetch").
({<0.125.0>,{m,f,0},tiger@durin}) call erlang:now()
({<0.125.0>,{m,f,0},tiger@durin}) returned from erlang:now/0 ->
{1031,133451,667611}
({<0.125.0>,{m,f,0},tiger@durin}) <0.72.0> ! 
{<0.125.0>,{1031,133451,667611}}
ok      ]]></code>
    </section>

    <section>
      <title>Example: Build your own tool</title>
      <p>This small example shows a simple tool for "debug tracing",
        i.e. tracing of function calls with return values.</p>
      <code type="none"><![CDATA[
-module(mydebug).
-export([start/0,trc/1,stop/0,format/1]).
-export([print/4]).
%% Include ms_transform.hrl so that I can use dbg:fun2ms/2 to
%% generate match specifications.
-include_lib("stdlib/include/ms_transform.hrl").
%%% -------------Tool API-------------
%%% ----------------------------------
%%% Star the "mydebug" tool
start() ->
    %% The options specify that the binary log shall be named
    %% <Node>-debug_log and that the print/4 function in this
    %% module shall be used as format handler
    ttb:tracer(all,[{file,"debug_log"},{handler,{{?MODULE,print},0}}]),
    %% All processes (existing and new) shall trace function calls
    %% We want trace messages to be sorted upon format, which requires
    %% timestamp flag. The flag is however enabled by default in ttb.
    ttb:p(all,call).

%%% Set trace pattern on function(s)
trc(M) when is_atom(M) ->
    trc({M,'_','_'});
trc({M,F}) when is_atom(M), is_atom(F) ->
    trc({M,F,'_'});
trc({M,F,_A}=MFA) when is_atom(M), is_atom(F) ->
    %% This match spec shortcut specifies that return values shall
    %% be traced.
    MatchSpec = dbg:fun2ms(fun(_) -> return_trace() end),
    ttb:tpl(MFA,MatchSpec).

%%% Format a binary trace log
format(Dir) ->
    ttb:format(Dir).

%%% Stop the "mydebug" tool
stop() ->
    ttb:stop(return).

%%% --------Internal functions--------
%%% ----------------------------------
%%% Format handler
print(_Out,end_of_trace,_TI,N) ->
    N;
print(Out,Trace,_TI,N) ->
    do_print(Out,Trace,N),
    N+1.

do_print(Out,{trace_ts,P,call,{M,F,A},Ts},N) ->
    io:format(Out,
              "~w: ~w, ~w:~n"
              "Call      : ~w:~w/~w~n"
              "Arguments :~p~n~n",
              [N,Ts,P,M,F,length(A),A]);
do_print(Out,{trace_ts,P,return_from,{M,F,A},R,Ts},N) ->
    io:format(Out,
              "~w: ~w, ~w:~n"
              "Return from  : ~w:~w/~w~n"
              "Return value :~p~n~n",
              [N,Ts,P,M,F,A,R]).      ]]></code>
      <p>To distinguish trace logs produced with this tool from other
        logs, the <c>file</c> option is used in <c>tracer/2</c>. The
        logs will therefore be fetched to a directory named
        <c>ttb_upload_debug_log-YYYYMMDD-HHMMSS</c>
      </p>
      <p>By using the <c>handler</c> option when starting the tracer,
        the information about how to format the file is stored in the
        trace information file (<c>.ti</c>). This is not necessary, as
        it might be given at the time of formatting instead. It can
        however be useful if you e.g. want to automatically format your
        trace logs by using the <c>format</c> option in
        <c>ttb:stop/1</c>. It also means that you don't need any
        knowledge of the content of a binary log to be able to format it
        the way it was intended. If the <c>handler</c> option is given
        both when starting the tracer and when formatting, the one given
        when formatting is used.
        </p>
      <p>The <c>call</c> trace flag is set on all processes. This
        means that any function activated with the <c>trc/1</c> command
        will be traced on all existing and new processes.
        </p>
    </section>
  </section>

  <section>
    <title>Running the Trace Tool Builder against a remote node</title>
    <p>The Observer application might not always be available on the
      node that shall be traced (in the following called the "traced
      node"). It is still possible to run the Trace Tool Builder from
      another node (in the following called the "trace control node") as
      long as
      </p>
    <list type="bulleted">
      <item>The Observer application is available on the trace control node.</item>
      <item>The Runtime Tools application is available on both the
       trace control node and the traced node.</item>
    </list>
    <p>If the Trace Tool Builder shall be used against a remote node,
      it is highly recommended to start the trace control node as
      <em>hidden</em>. This way it can connect to the traced node
      without the traced node "seeing" it, i.e. if the <c>nodes()</c>
      BIF is called on the traced node, the trace control node will not
      show. To start a hidden node, add the <c>-hidden</c> option to the
      <c>erl</c> command, e.g.</p>
    <code type="none">
% erl -sname trace_control -hidden    </code>

    <section>
      <title>Diskless node</title>
      <p>If the traced node is diskless, <c>ttb</c> must be started from
        a trace control node with disk access, and the <c>file</c> option
        must be given to the <c>tracer/2</c> function with the value
        <c>{local, File}</c>, e.g.</p>
      <code type="none">
(trace_control@durin)1> ttb:tracer(mynode@diskless,{file,{local,
{wrap,"mytrace"}}}).
{ok,[mynode@diskless]}      </code>
    </section>
  </section>

  <section>
    <title>Additional tracing options</title>
    <p>When setting up a trace, several features may be turned on:</p>
      <list type="bulleted">
        <item>time-constrained tracing,</item>
        <item>overload protection,</item>
        <item>autoresuming.</item>
      </list>
      <section>
        <title>Time-constrained tracing</title>
        <p>Sometimes, it may be helpful to enable trace for a
          given period of time (i.e. to monitor a system for 24 hours
          or half of a second). This may be done by issuing additional
          <c>{timer, TimerSpec}</c> option. If <c>TimerSpec</c> has the
          form of <c>MSec</c>, the trace is stopped after <c>MSec</c>
          milliseconds using <c>ttb:stop/0</c>. If any additional options
          are provided (<c>TimerSpec = {MSec, Opts}</c>), <c>ttb:stop/1</c>
          is called instead with <c>Opts</c> as the arguments. The timer
          is started with <c>ttb:p/2</c>, so any trace patterns should
          be set up before. <c>ttb:start_trace/4</c>
          always sets up all pattern before invoking <c>ttb:p/2</c>.
          Note that due to network and processing delays the the period
          of tracing is approximate.
          The example below shows how to set up a trace which will be
          automatically stopped and formatted after 5 seconds
        </p><code>
(tiger@durin)1>ttb:start_trace([node()],
                               [{erlang, now,[]}],
                               {all, call},
                               [{timer, {5000, format}}]).
</code>
        </section>
        <section>
          <label>Overload protection</label>
          <p>When tracing live systems, special care needs to be always taken
            not to overload a node with too heavy tracing. <c>ttb</c> provides
            the <c>overload</c> option to help to address the problem.</p>
          <p><c>{overload, MSec, Module, Function}</c> instructs the ttb backend
            (called <c>observer_backend</c>, part of the <c>runtime_tools</c>
            application) to perform overload check every <c>MSec</c> milliseconds.
            If the check (namely <c>Module:Function(check)</c>) returns
            <c>true</c>, tracing is disabled on the selected node.</p>
          <p>Overload protection activated on one node does not
            affect other nodes, where the tracing continues as normal.
            <c>ttb:stop/0/1</c> fetches data from all clients, including everything
            that has been collected before overload protection was activated.
            Note that
            changing trace details (with <c>ttb:p</c> and <c>ttb:tp/tpl...</c>)
            once overload protection gets activated in one of the traced
            nodes is not permitted in order not to allow trace setup
            to be inconsistent between nodes.
          </p>
          <p><c>Module:Function</c> provided with the <c>overload</c> option must
            handle three calls: <c>init</c>, <c>check</c> and <c>stop</c>. <c>init</c>
            and <c>stop</c> allows to perform some setup and teardown required by
            the check. An overload check module could look like this (note that
            <c>check</c> is always called by the same process, so <c>put</c> and
            <c>get</c> are possible).
          </p><code>
-module(overload).
-export([check/1]).

check(init) ->
    Pid = sophisticated_module:start(),
    put(pid, Pid);
check(check) ->
    get(pid) ! is_overloaded,
    receive
        Reply ->
            Reply
    after 5000 ->
            true
    end;
check(stop) ->
    get(pid) ! stop.</code>
        </section>
        <section>
          <title>Autoresume</title>
          <p>It is possible that a node (probably a buggy one, hence traced)
            crashes. In order to automatically resume tracing on the node
            as soon as it gets back, <c>resume</c> has to be used. When
            it is, the failing node tries to reconnect
            to trace control node as soon as <c>runtime tools</c> is started.
            This implies that <c>runtime_tools</c> must be included in
            other node's startup chain (if it is not, one could still
            resume tracing by starting <c>runtime_tools</c> manually,
            i.e. by an RPC call).</p>
          <p>In order not to loose the data that the failing node stored
            up to the point of crash, the control node will try to fetch
            it before restarting trace. This must happen within the allowed
            time frame or is aborted (default is 10 seconds, can be customized with
            <c>{resume, MSec}</c>). The data fetched this way is then
            merged with all other traces.</p>
          <p>Autostart feature requires additional data to be stored on
            traced nodes. By default, the data is stored automatically
            to the file called "ttb_autostart.bin" in the traced node's cwd.
            Users may decide to change this behaviour (i.e. on diskless
            nodes) by specifying their own module to handle autostart data
            storage and retrieval (<c>ttb_autostart_module</c>
            environment variable of <c>runtime_tools</c>). Please see the
            ttb's reference manual to see the module's API. This example
            shows the default handler</p>
          <code>
-module(ttb_autostart).
-export([read_config/0,
         write_config/1,
         delete_config/0]).

-define(AUTOSTART_FILENAME, "ttb_autostart.bin").

delete_config() ->
    file:delete(?AUTOSTART_FILENAME).

read_config() ->
    case file:read_file(?AUTOSTART_FILENAME) of
        {ok, Data} -> {ok, binary_to_term(Data)};
        Error      -> Error
    end.

write_config(Data) ->
    file:write_file(?AUTOSTART_FILENAME, term_to_binary(Data)).
          </code>
          <p>Remember that file trace ports buffer the data
            by default. If the node crashes, trace messages are not
            flushed to the binary log. If the chance of failure is
            high, it might be a good idea to automatically flush
            the buffers every now and then. Passing <c>{flush, MSec}</c>
            as one of <c>ttb:tracer/2</c> option flushes all buffers
            every <c>MSec</c> milliseconds.</p>
        </section>
        <section>
          <title>dbg mode</title>
          <p>The <c>{shell, ShellType}</c> option allows to make <c>ttb</c>
            operation similar to <c>dbg</c>. Using <c>{shell, true}</c>
            displays all trace messages in the shell before storing them.
            <c>{shell, only}</c> additionally disables message storage
            (so that the tool behaves exactly like dbg). This is allowed
            only with ip trace ports (<c>{trace, {local, File}}</c>).
          </p>
          <p>The command <c>ttb:tracer(dbg)</c> is a shortcut for the pure-dbg
            mode (<c>{shell, only}</c>).</p>
        </section>
  </section>

  <section>
    <marker id="trace_info"></marker>
    <title>Trace Information and the .ti File</title>
    <p>In addition to the trace log file(s), a file with the extension
      <c>.ti</c> is created when the Trace Tool Builder is started. This
      is the trace information file. It is a binary file, and it
      contains the process information, trace flags used, the name of
      the node to which it belongs and all information written with the
      <c>write_trace_info/2</c> function. .ti files are always fetched
      with other logs when the trace is stopped.
    </p>
    <p>Except for the process information, everything in the trace
      information file is passed on to the handler function when
      formatting. The <c>TI</c> parameter is a list of
      <c>{Key,ValueList}</c> tuples. The keys <c>flags</c>,
      <c>handler</c>, <c>file</c> and <c>node</c> are used for
      information written directly by <c>ttb</c>.
      </p>
    <p>You can add information to the trace information file by
      calling <c>write_trace_info/2</c>. Note that <c>ValueList</c>
      always will be a list, and if you call <c>write_trace_info/2</c>
      several times with the same <c>Key</c>, the <c>ValueList</c> will
      be extended with a new value each time. Example:
      </p>
    <p><c>ttb:write_trace_info(mykey,1)</c> gives the entry
      <c>{mykey,[1]}</c> in <c>TI</c>. Another call,
      <c>ttb:write_trace_info(mykey,2)</c>, changes this entry to
      <c>{mykey,[1,2]}</c>.
      </p>
  </section>

  <section>
    <title>Wrap Logs</title>
    <p>If you want to limit the size of the trace logs, you can use
      wrap logs. This works almost like a circular buffer. You can
      specify the maximum number of binary logs and the maximum size of
      each log. <c>ttb</c> will create a new binary log each time a log
      reaches the maximum size. When the the maximum number of logs are
      reached, the oldest log is deleted before a new one is created.
    </p>
    <p>Note that the overall size of data generated by ttb may be greater
      than the wrap specification would suggest - if a traced node restarts
      and autoresume is enabled, old wrap log is always stored and
      a new one is created.
    </p>
    <p>Wrap logs can be formatted one by one or all at once. See
      <seealso marker="#format">Formatting</seealso>.
      </p>
  </section>

  <section>
    <marker id="format"></marker>
    <title>Formatting</title>
    <p>Formatting can be done automatically when stopping <c>ttb</c>
      (see <seealso marker="#fetch_format">Automatically collect and format logs from all nodes</seealso>), or explicitly by calling
      the <c>ttb:format/1/2</c> function.
      </p>
    <p>Formatting means to read a binary log and present it in a
      readable format. You can use the default format handler in
      <c>ttb</c> to present each trace message as a line of text, or
      write your own handler to make more complex interpretations of the
      trace information. You can even use the Event Tracer <c>et</c> to
      present the trace log graphically (see <seealso marker="#et_viewer">Presenting trace logs with Event Tracer</seealso>).
      </p>
    <p>The first argument to <c>ttb:format/1/2</c> specifies which
      binary log(s) to format. This is usually the name of a directory
      that ttb created during log fetch. Unless there is the <c>disable_sort</c>
      option provided, the logs from different files are always sorted
      according to timestamp in traces.
      </p>
    <p>The second argument to <c>ttb:format/2</c> is a list of
      options. The <c>out</c> option specifies the destination where the
      formatted text shall be written. Default destination is
      <c>standard_io</c>, but a filename can also be given. The
      <c>handler</c> option specifies the format handler to use. If this
      option is not given, the <c>handler</c> option given when starting
      the tracer is used. If the <c>handler</c> option was not given
      when starting the tracer either, a default handler is used, which
      prints each trace message as a line of text. The <c>disable_sort</c>
      option indicates that there logs should not be merged according to
      timestamp, but processed one file after another (this might be
      a bit faster).
      </p>
    <p>A format handler is a fun taking four arguments. This fun will
      be called for each trace message in the binary log(s). A simple
      example which only prints each trace message could be like this:</p>
    <code type="none">
fun(Fd, Trace, _TraceInfo, State) ->
   io:format(Fd, "Trace: ~p~n", [Trace]),
   State
end.    </code>
    <p><c>Fd</c> is the file descriptor for the destination file, or
      the atom <c>standard_io</c>. <c>_TraceInfo</c> contains information
      from the trace information file (see <seealso marker="#trace_info">Trace Information and the .ti File</seealso>). <c>State</c> is a state variable for the format
      handler fun. The initial value of the <c>State</c> variable is
      given with the handler option, e.g.</p>
    <code type="none">
ttb:format("tiger@durin-ttb", [{handler, {{Mod,Fun}, initial_state}}])
                                                     ^^^^^^^^^^^^^    </code>
    <p>Another format handler could be used to calculate time spent by
      the garbage collector:</p>
    <code type="none">
fun(_Fd,{trace_ts,P,gc_start,_Info,StartTs},_TraceInfo,State) ->
      [{P,StartTs}|State];
   (Fd,{trace_ts,P,gc_end,_Info,EndTs},_TraceInfo,State) ->
      {value,{P,StartTs}} = lists:keysearch(P,1,State),
      Time = diff(StartTs,EndTs),
      io:format("GC in process ~w: ~w milliseconds~n", [P,Time]),
      State -- [{P,StartTs}]
end    </code>
    <p>A more refined version of this format handler is the function
      <c>handle_gc/4</c> in the module <c>multitrace.erl</c> which can
      be found in the <c>src</c> directory of the Observer application.
      </p>
    <p>The actual trace message is passed as the second argument (<c>Trace</c>).
      The possible values of <c>Trace</c> are:</p>
      <list type="bulleted">
        <item>all trace messages described in <c>erlang:trace/3</c> documentation,
        </item>
        <item><c>{drop, N}</c> if ip tracer is used (see <c>dbg:trace_port/2</c>),
        </item>
        <item><c>end_of_trace</c> received once when all trace messages have
          been processed.</item>
      </list>
    <p>By giving the format handler <c>ttb:get_et_handler()</c>, you can have the trace
      log presented graphically with <c>et_viewer</c> in the Event
      Tracer application (see <seealso marker="#et_viewer">Presenting trace logs with Event Tracer</seealso>).
    </p>
    <p>You may always decide not to format the whole trace data contained
      in the fetch directory, but analyze single files instead. In order
      to do so, a single file (or list of files) have to be passed as
      the first argument to <c>format/1/2</c>.</p>
    <p>Wrap logs can be formatted one by one or all in one go. To
      format one of the wrap logs in a set, give the exact name of the
      file. To format the whole set of wrap logs, give the name with '*'
      instead of the wrap count. An example:
      </p>
    <p>Start tracing:</p>
    <code type="none">
(tiger@durin)1> ttb:tracer(node(),{file,{wrap,"trace"}}).
{ok,[tiger@durin]}
(tiger@durin)2> ttb:p(...)
...    </code>
    <p>This will give a set of binary logs, like:</p>
    <code type="none">
tiger@durin-trace.0.wrp
tiger@durin-trace.1.wrp
tiger@durin-trace.2.wrp
...    </code>
    <p>Format the whole set of logs:</p>
    <code type="none">
1> ttb:format("tiger@durin-trace.*.wrp").
....
ok
2>    </code>
    <p>Format only the first log:</p>
    <code type="none">
1> ttb:format("tiger@durin-trace.0.wrp").
....
ok
2>    </code>
    <p>To merge all wrap logs from two nodes:</p>
    <code type="none">
1> ttb:format(["tiger@durin-trace.*.wrp","lion@durin-trace.*.wrp"]).
....
ok
2>    </code>

    <section>
      <marker id="et_viewer"></marker>
      <title>Presenting trace logs with Event Tracer</title>
      <p>For detailed information about the Event Tracer, please turn
        to the User's Guide and Reference Manuals for the <c>et</c>
        application.
        </p>
      <p>By giving the format handler <c>ttb:get_et_handler()</c>, you can have the
        trace log presented graphically with <c>et_viewer</c> in the
        Event Tracer application. <c>ttb</c> provides a few different
        filters which can be selected from the Filter menu in the
        <c>et_viewer</c> window. The filters are names according to the
        type of actors they present (i.e. what each vertical line in the
        sequence diagram represent). Interaction between actors is shown
        as red arrows between two vertical lines, and activities within
        an actor are shown as blue text to the right of the actors line.
        </p>
      <p>The <c>processes</c> filter is the only filter which will
        show all trace messages from a trace log. Each vertical line in
        the sequence diagram represents a process. Erlang messages,
        spawn and link/unlink are typical interactions between
        processes. Function calls, scheduling and garbage collection are
        typical activities within a process. <c>processes</c> is the
        default filter.
        </p>
      <p>The rest of the filters will only show function calls and
        function returns. All other trace message are discarded. To get
        the most out of these filters, <c>et_viewer</c> needs to known
        the caller of each function and the time of return. This can be
        obtained by using both the <c>call</c> and <c>return_to</c>
        flags when tracing. Note that the <c>return_to</c> flag only
        works with local call trace, i.e. when trace patterns are set
        with <c>ttb:tpl</c>.
        </p>
      <p>The same result can be obtained by using the <c>call</c> flag
        only and setting a match specification like this on local or
        global function calls:</p>
      <code type="none">
1> dbg:fun2ms(fun(_) -> return_trace(),message(caller()) end).
[{'_',[],[{return_trace},{message,{caller}}]}]      </code>
      <p>This should however be done with care, since the
        <c>{return_trace}</c> function in the match specification will
        destroy tail recursiveness.
        </p>
      <p>The <c>modules</c> filter shows each module as a vertical
        line in the sequence diagram. External function calls/returns
        are shown as interactions between modules and internal function
        calls/returns are shown as activities within a module.
        </p>
      <p>The <c>functions</c> filter shows each function as a vertical
        line in the sequence diagram. A function calling itself is shown
        as an activity within a function, and all other function calls
        are shown as interactions between functions.
        </p>
      <p>The <c>mods_and_procs</c> and <c>funcs_and_procs</c> filters
        are equivalent to the <c>modules</c> and <c>functions</c>
        filters respectively, except that each module or function can
        have several vertical lines, one for each process it resides on.
        </p>
      <p>In the next example, modules <c>foo</c> and <c>bar</c> are used:</p>
      <code type="none">
-module(foo).
-export([start/0,go/0]).

start() ->
    spawn(?MODULE, go, []).

go() ->
    receive
        stop ->
            ok;
        go ->
            bar:f1(),
            go()
    end.
</code><code type="none">
-module(bar).
-export([f1/0,f3/0]).
f1() ->
    f2(),
    ok.
f2() ->
    spawn(?MODULE,f3,[]).
f3() ->
    ok.</code>

      <p>Now let's set up the trace.</p>
<code>
(tiger@durin)1>%%First we retrieve the Pid to limit traced processes set
(tiger@durin)1>Pid = foo:start().
(tiger@durin)2>%%Now we set up tracing
(tiger@durin)2>ttb:tracer().
(tiger@durin)3>ttb:p(Pid, [call, return_to, procs, set_on_spawn]).
(tiger@durin)4>ttb:tpl(bar, []).
(tiger@durin)5>%%Invoke our test function and see output with et viewer
(tiger@durin)5>Pid ! go.
(tiger@durin)6>ttb:stop({format, {handler, ttb:get_et_handler()}}).
</code>

      <p>This shoud render a result similar to the
        following:
        </p>
      <p></p>
      <image file="et_processes.gif">
        <icaption>Filter: "processes"</icaption>
      </image>
      <image file="et_modsprocs.gif">
        <icaption>Filter: "mods_and_procs"</icaption>
      </image>

      <p>Note, that we can use <c>ttb:start_trace/4</c> function to help
        us here:</p>
<code>
(tiger@durin)1>Pid = foo:start().
(tiger@durin)2>ttb:start_trace([node()],
                               [{bar,[]}],
                               {Pid, [call, return_to, procs, set_on_spawn]}
                               {handler, ttb:get_et_handler()}).
(tiger@durin)3>Pid ! go.
(tiger@durin)4>ttb:stop(format).
</code>

    </section>
  </section>

  <section>
    <marker id="fetch_format"></marker>
    <title>Automatically collect and format logs from all nodes</title>
    <p>By default <c>ttb:stop/1</c> fetches  trace logs and
      trace information files from all nodes. The logs are stored in a
      new directory named <c>ttb_upload-Filename-Timestamp</c> under the working
      directory of the trace control node. Fetching may be disabled by
      providing the <c>nofetch</c> option to <c>ttb:stop/1</c>. User can
      specify a fetch directory of his choice passing the
      <c>{fetch_dir, Dir}</c> option.
      </p>
    <p>If the option <c>format</c> is given to <c>ttb:stop/1</c>, the
      trace logs are automatically formatted after tracing is
      stopped.
    </p>
  </section>

  <section>
    <title>History and Configuration Files</title>
    <p>For the tracing functionality, <c>dbg</c> could be used instead
      of the <c>ttb</c> for setting trace flags on processes and trace
      patterns for call trace, i.e. the functions <c>p</c>, <c>tp</c>,
      <c>tpl</c>, <c>ctp</c>, <c>ctpl</c> and <c>ctpg</c>. There are only
      two things added by <c>ttb</c> for these functions:
      <list type="bulleted">
        <item>all calls  are stored in the history buffer and can be
          recalled and stored in a configuration file. This makes it
          easy to setup the same trace environment e.g. if you want to
          compare two test runs. It also reduces the amount of
          typing when using <c>ttb</c> from the erlang shell;</item>
        <item>shortcuts are provided for the most common match
          specifications (in order not to force the user to use
          <c>dbg:fun2ms</c> continually</item>).
      </list>
      </p>
    <p>Use <c>list_history/0</c> to see the content of the history
      buffer, and <c>run_history/1</c> to re-execute one of the entries.
      </p>
    <p>The main purpose of the history buffer is the possibility to
      create configuration files. Any function stored in the history
      buffer can be written to a configuration file and used for
      creating a specific configuration at any time with one single
      function call.
      </p>
    <p>A configuration file is created or extended with
      <c>write_config/2/3</c>. Configuration files are binary files
      and can therefore only be read and written with functions provided
      by <c>ttb</c>.
      </p>
    <p>You can write the complete content of the history buffer to a
      config file by calling
      <c>ttb:write_config(ConfigFile,all)</c>. And you can write
      selected entries from the history by calling
      <c>ttb:write_config(ConfigFile,NumList)</c>, where
      <c>NumList</c> is a list of integers pointing out the history
      entries to write. Moreover, the history buffer is always dumped
      to <c>ttb_last_config</c> when <c>ttb:stop/0/1</c> is called.
      </p>
    <p>User defined entries can also be written to a config file by
      calling the function
      <c>ttb:write_config(ConfigFile,ConfigList)</c> where
      <c>ConfigList</c> is a list of <c>{Module,Function,Args}</c>.
      </p>
    <p>Any existing file <c>ConfigFile</c> is deleted and a new file
      is created when <c>write_config/2</c> is called. The option
      <c>append</c> can be used if you wish to add something at the end
      of an existing config file, e.g.
      <c>ttb:write_config(ConfigFile,What,[append])</c>.
      </p>

    <section>
      <title>Example: History and configuration files</title>
      <p>See the content of the history buffer</p>
      <code type="none"><![CDATA[
(tiger@durin)191> ttb:tracer().                                    
{ok,[tiger@durin]}
(tiger@durin)192> ttb:p(self(),[garbage_collection,call]).               
{ok,{[<0.1244.0>],[garbage_collection,call]}}
(tiger@durin)193> ttb:tp(ets,new,2,[]).                                  
{ok,[{matched,1}]}
(tiger@durin)194> ttb:list_history().
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}}]      ]]></code>
      <p>Execute an entry from the history buffer:</p>
      <code type="none"><![CDATA[
(tiger@durin)195> ttb:ctp(ets,new,2).
{ok,[{matched,1}]}
(tiger@durin)196> ttb:list_history().
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}},
 {4,{ttb,ctp,[ets,new,2]}}]
(tiger@durin)197> ttb:run_history(3).
ttb:tp(ets,new,2,[]) ->
{ok,[{matched,1}]}      ]]></code>
      <p>Write the content of the history buffer to a configuration
        file:</p>
      <code type="none"><![CDATA[
(tiger@durin)198> ttb:write_config("myconfig",all).
ok
(tiger@durin)199> ttb:list_config("myconfig").
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}},
 {4,{ttb,ctp,[ets,new,2]}},
 {5,{ttb,tp,[ets,new,2,[]]}}]      ]]></code>
      <p>Extend an existing configuration:</p>
      <code type="none"><![CDATA[
(tiger@durin)200> ttb:write_config("myconfig",[{ttb,tp,[ets,delete,1,[]]}],
[append]).
ok
(tiger@durin)201> ttb:list_config("myconfig").
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}},
 {4,{ttb,ctp,[ets,new,2]}},
 {5,{ttb,tp,[ets,new,2,[]]}},
 {6,{ttb,tp,[ets,delete,1,[]]}}]      ]]></code>
      <p>Go back to a previous configuration after stopping Trace Tool
        Builder:</p>
      <code type="none"><![CDATA[
(tiger@durin)202> ttb:stop().
ok
(tiger@durin)203> ttb:run_config("myconfig").
ttb:tracer(tiger@durin,[]) ->
{ok,[tiger@durin]}

ttb:p(<0.1244.0>,[garbage_collection,call]) ->
{ok,{[<0.1244.0>],[garbage_collection,call]}}

ttb:tp(ets,new,2,[]) ->
{ok,[{matched,1}]}

ttb:ctp(ets,new,2) ->
{ok,[{matched,1}]}

ttb:tp(ets,new,2,[]) ->
{ok,[{matched,1}]}

ttb:tp(ets,delete,1,[]) ->
{ok,[{matched,1}]}

ok      ]]></code>
      <p>Write selected entries from the history buffer to a
        configuration file:</p>
      <code type="none"><![CDATA[
(tiger@durin)204> ttb:list_history().          
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}},
 {4,{ttb,ctp,[ets,new,2]}},
 {5,{ttb,tp,[ets,new,2,[]]}},
 {6,{ttb,tp,[ets,delete,1,[]]}}]
(tiger@durin)205> ttb:write_config("myconfig",[1,2,3,6]).
ok
(tiger@durin)206> ttb:list_config("myconfig").
[{1,{ttb,tracer,[tiger@durin,[]]}},
 {2,{ttb,p,[<0.1244.0>,[garbage_collection,call]]}},
 {3,{ttb,tp,[ets,new,2,[]]}},
 {4,{ttb,tp,[ets,delete,1,[]]}}]
(tiger@durin)207>       ]]></code>
    </section>
  </section>

  <section>
    <title>Sequential Tracing</title>
    <p>To learn what sequential tracing is and how it can be used,
      please turn to the reference manual for the
      <em><c>seq_trace</c></em> module in the <em><c>kernel</c></em>
      application.
      </p>
    <p>The support for sequential tracing provided by the Trace Tool
      Builder includes </p>
    <list type="bulleted">
      <item>Initiation of the system tracer. This is automatically
       done when a trace port is started with <c>ttb:tracer/0/1/2</c></item>
      <item>Creation of match specifications which activates
       sequential tracing</item>
    </list>
    <p>Starting sequential tracing requires that a tracer has been
      started with the <c>ttb:tracer/0/1/2</c> function. Sequential
      tracing can then either be started via a trigger function with a
      match specification created with <c>ttb:seq_trigger_ms/0/1</c>,
      or directly by using the <c>seq_trace</c> module in the
      <c>kernel</c> application.
      </p>

    <section>
      <title>Example: Sequential tracing</title>
      <p>In the following example, the function
        <c>dbg:get_tracer/0</c> is used as trigger for sequential
        tracing:</p>
      <code type="none"><![CDATA[
(tiger@durin)110> ttb:tracer().                               
{ok,[tiger@durin]}
(tiger@durin)111> ttb:p(self(),call).                               
{ok,{[<0.158.0>],[call]}}
(tiger@durin)112> ttb:tp(dbg,get_tracer,0,ttb:seq_trigger_ms(send)).
{ok,[{matched,1},{saved,1}]}
(tiger@durin)113> dbg:get_tracer(), seq_trace:reset_trace().         
true
(tiger@durin)114> ttb:stop(format).
({<0.158.0>,{shell,evaluator,3},tiger@durin}) call dbg:get_tracer()
SeqTrace [0]: ({<0.158.0>,{shell,evaluator,3},tiger@durin}) 
{<0.237.0>,dbg,tiger@durin} ! {<0.158.0>,{get_tracer,tiger@durin}} 
[Serial: {0,1}]
SeqTrace [0]: ({<0.237.0>,dbg,tiger@durin}) 
{<0.158.0>,{shell,evaluator,3},tiger@durin} ! {dbg,{ok,#Port<0.222>}} 
[Serial: {1,2}]
ok
(tiger@durin)116>       ]]></code>
      <p>Starting sequential tracing with a trigger is actually more
        useful if the trigger function is not called directly from the
        shell, but rather implicitly within a larger system. When
        calling a function from the shell, it is simpler to start
        sequential tracing directly, e.g.</p>
      <code type="none"><![CDATA[
(tiger@durin)116> ttb:tracer().                               
{ok,[tiger@durin]}
(tiger@durin)117> seq_trace:set_token(send,true), dbg:get_tracer(), 
seq_trace:reset_trace().
true
(tiger@durin)118> ttb:stop(format).
SeqTrace [0]: ({<0.158.0>,{shell,evaluator,3},tiger@durin}) 
{<0.246.0>,dbg,tiger@durin} ! {<0.158.0>,{get_tracer,tiger@durin}} 
[Serial: {0,1}]
SeqTrace [0]: ({<0.246.0>,dbg,tiger@durin}) 
{<0.158.0>,{shell,evaluator,3},tiger@durin} ! {dbg,{ok,#Port<0.229>}} 
[Serial: {1,2}]
ok
(tiger@durin)120>       ]]></code>
      <p>In both examples above, the <c>seq_trace:reset_trace/0</c>
        resets the trace token immediately after the traced function in
        order to avoid lots of trace messages due to the printouts in
        the erlang shell.
        </p>
      <p>All functions in the <c>seq_trace</c> module, except
        <c>set_system_tracer/1</c>, can be used after the trace port has
        been started with <c>ttb:tracer/0/1/2</c>.
        </p>
    </section>
  </section>

  <section>
    <title>Example: Multipurpose trace tool</title>
    <p>The module <c>multitrace.erl</c> which can be found in the
      <c>src</c> directory of the Observer application implements a
      small tool with three possible trace settings. The trace messages
      are written to binary files which can be formatted with the
      function <em><c>multitrace:format/1/2</c></em>.
      </p>
    <taglist>
      <tag><em><c>multitrace:debug(What)</c></em></tag>
      <item>Start calltrace on all processes and trace the given
       function(s). The format handler used is
      <c>multitrace:handle_debug/4</c> which prints each call and
       return. <c>What</c> must be an item or a list of items to trace,
       given on the format <c>{Module,Function,Arity}</c>,
      <c>{Module,Function}</c> or just <c>Module</c>.</item>
      <tag><em><c>multitrace:gc(Procs)</c></em></tag>
      <item>Trace garbage collection on the given process(es). The
       format handler used is <c>multitrace:handle_gc/4</c> which
       prints start and stop and the time spent for each GC.</item>
      <tag><em><c>multitrace:schedule(Procs)</c></em></tag>
      <item>Trace in- and out-scheduling on the given process(es). The
       format handler used is <c>multitrace:handle_schedule/4</c> which
       prints each in and out scheduling with process, timestamp and
       current function. It also prints the total time each traced
       process was scheduled in.</item>
    </taglist>
  </section>
</chapter>