From 37092dab15448ef6a078800e3ff0cc41880ea765 Mon Sep 17 00:00:00 2001 From: Lukas Larsson Date: Thu, 10 Dec 2015 11:10:46 +0100 Subject: erts: Implement tracer modules Add the possibility to use modules as trace data receivers. The functions in the module have to be nifs as otherwise complex trace probes will be very hard to handle (complex means trace probes for ports for example). This commit changes the way that the ptab->tracer field works from always being an immediate, to now be NIL if no tracer is present or else be the tuple {TracerModule, TracerState} where TracerModule is an atom that is later used to lookup the appropriate tracer callbacks to call and TracerState is just passed to the tracer callback. The default process and port tracers have been rewritten to use the new API. This commit also changes the order which trace messages are delivered to the potential tracer process. Any enif_send done in a tracer module may be delayed indefinitely because of lock order issues. If a message is delayed any other trace message send from that process is also delayed so that order is preserved for each traced entity. This means that for some trace events (i.e. send/receive) the events may come in an unintuitive order (receive before send) to the trace receiver. Timestamps are taken when the trace message is generated so trace messages from differented processes may arrive with the timestamp out of order. Both the erlang:trace and seq_trace:set_system_tracer accept the new tracer module tracers and also the backwards compatible arguments. OTP-10267 --- erts/doc/src/Makefile | 15 +- erts/doc/src/erl_nif.xml | 3 +- erts/doc/src/erl_tracer.xml | 324 ++++++++++++++++++++++++++++++++++++++++++++ erts/doc/src/erlang.xml | 46 ++++--- erts/doc/src/match_spec.xml | 11 +- erts/doc/src/ref_man.xml | 1 + erts/doc/src/specs.xml | 1 + 7 files changed, 365 insertions(+), 36 deletions(-) create mode 100644 erts/doc/src/erl_tracer.xml (limited to 'erts/doc/src') diff --git a/erts/doc/src/Makefile b/erts/doc/src/Makefile index e02e89238e..b96cbbce40 100644 --- a/erts/doc/src/Makefile +++ b/erts/doc/src/Makefile @@ -50,12 +50,14 @@ XML_REF1_FILES = epmd.xml \ XML_REF3_EFILES = \ erl_prim_loader.xml \ erlang.xml \ + erl_tracer.xml \ init.xml \ zlib.xml XML_REF3_FILES = \ driver_entry.xml \ erl_nif.xml \ + erl_tracer.xml \ erl_driver.xml \ erl_prim_loader.xml \ erlang.xml \ @@ -154,18 +156,9 @@ clean: rm -f $(SPECDIR)/* rm -f errs core *~ -$(SPECDIR)/specs_driver_entry.xml: +$(SPECDIR)/specs_%.xml: escript $(SPECS_EXTRACTOR) $(SPECS_FLAGS) \ - -o$(dir $@) -module driver_entry -$(SPECDIR)/specs_erl_nif.xml: - escript $(SPECS_EXTRACTOR) $(SPECS_FLAGS) \ - -o$(dir $@) -module erl_nif -$(SPECDIR)/specs_erl_driver.xml: - escript $(SPECS_EXTRACTOR) $(SPECS_FLAGS) \ - -o$(dir $@) -module erl_driver -$(SPECDIR)/specs_erts_alloc.xml: - escript $(SPECS_EXTRACTOR) $(SPECS_FLAGS) \ - -o$(dir $@) -module erts_alloc + -o$(dir $@) -module $(patsubst $(SPECDIR)/specs_%.xml,%,$@) # ---------------------------------------------------- # Release Target diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml index 5f56d8b595..7546f7ef81 100644 --- a/erts/doc/src/erl_nif.xml +++ b/erts/doc/src/erl_nif.xml @@ -566,8 +566,7 @@ typedef struct { typedef void ErlNifResourceDtor(ErlNifEnv* env, void* obj); -

The function prototype of a resource destructor function. - A destructor function is not allowed to call any term-making functions.

+

The function prototype of a resource destructor function.

ErlNifCharEncoding diff --git a/erts/doc/src/erl_tracer.xml b/erts/doc/src/erl_tracer.xml new file mode 100644 index 0000000000..1e8e78b25f --- /dev/null +++ b/erts/doc/src/erl_tracer.xml @@ -0,0 +1,324 @@ + + + + +
+ + 20162016 + Ericsson AB. All Rights Reserved. + + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + + erl_tracer + + + + +
+ erl_tracer + Erlang Tracer Behaviour + +

A behaviour module for implementing the back end of the erlang + tracing system. The functions in this module will be called whenever + a trace probe is triggered. Both the enabled and trace + functions are called in the context of the entity that triggered the + trace probe. + This means that the overhead by having the tracing enabled will be + greatly effected by how much time is spent in these functions. So do as + little work as possible in these functions.

+ +

All functions in this behaviour have to be implemented as NIF's. + This is a limitation that may the lifted in the future. + There is an example tracer module nif + implementation at the end of this page.

+ + +

Do not send messages or issue port commands to the Tracee + in any of the callbacks. Doing so is not allowed and can cause all + sorts of strange behaviour, including but not limited to infinite + recursions.

+ + + + + + + +

The different trace tags that the tracer will be called with. + Each trace tag is described in greater detail in + Module:trace/6 +

+ + + + + +

The process or port that the trace belongs to. +

+ + + + + +

The options for the tracee. + + timestamp + If not set to undefined, the tracer has been requested to + include a timestamp. + match_spec_result + If not set to true, the tracer has been requested to + include the output of a match specification that was run. + scheduler_id + Set to a number of the scheduler id is to be included by the tracer. + Otherwise it is set to undefined. + +

+ + + + + +

+ The state which is given when calling + erlang:trace(PidPortSpec,true,[{tracer,Module,TracerState}]). + The tracer state is an immutable value that is passed to erl_tracer callbacks and should + contain all the data that is needed to generate the trace event. +

+ + + + +
+ CALLBACK FUNCTIONS +

The following functions + should be exported from a erl_tracer callback module.

+
+ + + + Module:enabled(TraceTag, TracerState, Tracee) -> Result + Check if a trace event should be generated. + + TraceTag = trace_tag() | trace_status + TracerState = term() + Tracee = tracee() + Result = trace | discard | remove + + +

This callback will be called whenever a trace point is triggered. It + allows the tracer to decide whether a trace should be generated or not. + This check is made as early as possible in order to limit the amount of + overhead associated with tracing. If trace is returned the + necessary trace data will be created and the trace call-back of the tracer + will be called. If discard is returned, this trace call + will be discarded and no call to trace will be done. If + remove is returned, the VM will attempt to remove this tracer + from the tracee, together with any trace flags set on the tracee. +

+

trace_status is a special type of TraceTag which is used + to check if the tracer should still be active. It is called in multiple + scenarios, but most significantly it is used when tracing is started + using this tracer.

+

This function may be called multiple times per trace point, so it + is important that it is both fast and side effect free.

+ + + + + Module:trace(TraceTag, TracerState, Tracee, FirstTraceTerm, SecondTraceTerm, Opts) -> Result + Check if a trace event should be generated. + + TraceTag = trace_tag() + TracerState = term() + Tracee = tracee() + FirstTraceTerm = term() + SecondTraceTerm = term() | undefined + Opts = trace_opts() + Result = ok + + +

This callback will be called when a trace point is triggered and + the Module:enabled/3 + callback returned trace. In it any side effects needed by + the tracer should be done. The trace point payload is located in + the FirstTraceTerm and SecondTraceTerm. The content + of the TraceTerms depends on which TraceTag has been triggered. + The FirstTraceTerm and SecondTraceTerm correspond to the + fourth and fifth slot in the trace tuples described in + erlang:trace/3. + If the tuple only has four elements, SecondTraceTerm will be + undefined.

+ + + + Module:trace(seq_trace, TracerState, Label, SeqTraceInfo, undefined, Opts) -> Result + Check if a sequence trace event should be generated. + + TracerState = term() + Label = term() + SeqTraceInfo = term() + Opts = trace_opts() + Result = ok + + +

The TraceTag seq_trace is handled a little bit + differently. There is not Tracee for seq_trace, instead the + Label associated with the seq_trace event is given. + For more info on what Label and SeqTraceInfo can be + see the seq_trace manual.

+ + + +
+ + Erl Tracer Module example +

In the example below a tracer module with a nif backend sends a message + for each send trace tag containing only the sender and receiver. + Using this tracer module, a much more lightweight message tracer is + used that only records who sent messages to who.

+

Here is an example session using it on Linux.

+ 
+$ gcc -I erts-8.0/include/ -fPIC -shared -o erl_msg_tracer.so erl_msg_tracer.c
+$ erl
+Erlang/OTP 19 [DEVELOPMENT] [erts-8.0] [source-ed2b56b] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]
+
+Eshell V8.0  (abort with ^G)
+1> c(erl_msg_tracer), erl_msg_tracer:load().
+ok
+2> Tracer = spawn(fun F() -> receive M -> io:format("~p~n",[M]), F() end end).
+<0.37.0>
+3> erlang:trace(new, true, [send,{tracer, erl_msg_tracer, Tracer}]).
+0
+{<0.39.0>,<0.27.0>}
+4> {ok, D} = file:open("/tmp/tmp.data",[write]).
+{trace,#Port<0.486>,<0.40.0>}
+{trace,<0.40.0>,<0.21.0>}
+{trace,#Port<0.487>,<0.4.0>}
+{trace,#Port<0.488>,<0.4.0>}
+{trace,#Port<0.489>,<0.4.0>}
+{trace,#Port<0.490>,<0.4.0>}
+{ok,<0.40.0>}
+{trace,<0.41.0>,<0.27.0>}
+5>
+
+

erl_msg_tracer.erl

+ 
+-module(erl_msg_tracer).
+
+-export([enabled/3, trace/6, load/0]).
+
+load() ->
+    erlang:load_nif("erl_msg_tracer", []).
+
+enabled(_, _, _) ->
+    error.
+
+trace(_, _, _,_, _, _) ->
+    error.
+
+

erl_msg_tracer.c

+ 
+#include "erl_nif.h"
+
+/* NIF interface declarations */
+static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info);
+static int upgrade(ErlNifEnv* env, void** priv_data, void** old_priv_data, ERL_NIF_TERM load_info);
+static void unload(ErlNifEnv* env, void* priv_data);
+
+/* The NIFs: */
+static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+static ERL_NIF_TERM trace(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+
+static ErlNifFunc nif_funcs[] = {
+    {"enabled", 3, enabled},
+    {"trace", 6, trace}
+};
+
+ERL_NIF_INIT(erl_msg_tracer, nif_funcs, load, NULL, upgrade, unload)
+
+static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info)
+{
+    *priv_data = NULL;
+    return 0;
+}
+
+static void unload(ErlNifEnv* env, void* priv_data)
+{
+
+}
+
+static int upgrade(ErlNifEnv* env, void** priv_data, void** old_priv_data,
+		   ERL_NIF_TERM load_info)
+{
+    if (*old_priv_data != NULL || *priv_data != NULL) {
+	return -1; /* Don't know how to do that */
+    }
+    if (load(env, priv_data, load_info)) {
+	return -1;
+    }
+    return 0;
+}
+
+/*
+ * argv[0]: Trace Tag
+ * argv[1]: TracerState
+ * argv[2]: Tracee
+ */
+static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+    ErlNifPid to_pid;
+    if (enif_get_local_pid(env, argv[1], &to_pid))
+        if (!enif_is_process_alive(env, &to_pid))
+            /* tracer is dead so we should remove this trace point */
+            return enif_make_atom(env, "remove");
+
+    /* Only generate trace for when tracer != tracee */
+    if (enif_is_identical(argv[1], argv[2]))
+        return enif_make_atom(env, "discard");
+
+    /* Only trigger trace messages on 'send' */
+    if (enif_is_identical(enif_make_atom(env, "send"), argv[0]))
+        return enif_make_atom(env, "trace");
+
+    /* Have to answer trace_status */
+    if (enif_is_identical(enif_make_atom(env, "trace_status"), argv[0]))
+        return enif_make_atom(env, "trace");
+
+    return enif_make_atom(env, "discard");
+}
+
+/*
+ * argv[0]: Trace Tag, should only be 'send'
+ * argv[1]: TracerState, process to send {argv[2], argv[4]} to
+ * argv[2]: Tracee
+ * argv[3]: Message, ignored
+ * argv[4]: Recipient
+ * argv[5]: Options, ignored
+ */
+static ERL_NIF_TERM trace(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+    ErlNifPid to_pid;
+
+    if (enif_get_local_pid(env, argv[1], &to_pid)) {
+        ERL_NIF_TERM msg = enif_make_tuple3(env, enif_make_atom(env, "trace"), argv[2], argv[4]);
+        enif_send(env, &to_pid, NULL, msg);
+    }
+
+    return enif_make_atom(env, "ok");
+}
+
+
+ diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index ef577c82bf..86bdb1dfe6 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -8371,7 +8371,7 @@ timestamp() -> all -

Sets all trace flags except {tracer, Tracer} and +

Sets all trace flags except tracer and cpu_timestamp, which are in their nature different than the others.

 @@ -8529,12 +8529,20 @@ timestamp() ->

Specifies where to send the trace messages. Tracer must be the process identifier of a local process - or the port identifier - of a local port. If this flag is not given, trace - messages are sent to the process that called - erlang:trace/3.

+ or the port identifier of a local port.

+ + {tracer, TracerModule, TracerState} + +

Specifies that a tracer module should be called + instead of sending a trace message. The tracer module + can then ignore or change the trace message. For more details + on how to write a tracer module see + erl_tracer +

+

If no tracer is given, the calling process + will be receiving all of the trace messages

The effect of combining set_on_first_link with set_on_link is the same as having set_on_first_link alone. Likewise for @@ -8706,9 +8714,9 @@ timestamp() -> garbage collection.

 -

If the tracing process dies, the flags are silently - removed.

-

Only one process can trace a particular process. Therefore, +

If the tracing process/port dies or the tracer module returns + remove, the flags are silently removed.

+

Each process can only be traced by one tracer. Therefore, attempts to trace an already traced process fail.

Returns: A number indicating the number of processes that matched PidSpec. @@ -8716,7 +8724,7 @@ timestamp() -> identifier, the return value is 1. If PidSpec is all or existing, the return value is - the number of processes running, excluding tracer processes. + the number of processes running. If PidSpec is new, the return value is 0.

Failure: badarg if the specified arguments are @@ -8750,7 +8758,7 @@ timestamp() -> has not been traced by someone, but if this is the case, no trace messages have been delivered when the trace_delivered message arrives.

-

Notice that that Tracee must refer +

Notice that Tracee must refer to a process currently, or previously existing on the same node as the caller of erlang:trace_delivered(Tracee) resides on. @@ -8801,7 +8809,8 @@ timestamp() -> tracer -

Returns the identifier for process or port tracing this +

Returns the identifier for process, port or a tuple containing + the tracer module and tracer state tracing this process. If this process is not being traced, the return value is [].

@@ -8830,8 +8839,8 @@ timestamp() -> meta -

Returns the meta-trace tracer process or port for this - function, if it has one. If the function is not +

Returns the meta-trace tracer process, port or trace module + for this function, if it has one. If the function is not meta-traced, the returned value is false. If the function is meta-traced but has once detected that the tracer process is invalid, the returned value is [].

@@ -8999,13 +9008,12 @@ timestamp() -> the process, a return_to message is also sent when this function returns to its caller.

 - meta | {meta, Pid} + meta | {meta, Pid} | {meta, TracerModule, TracerState} +

Turns on or off meta-tracing for all types of function - calls. Trace messages are sent to the tracer process - or port Pid whenever any of the specified - functions are called, regardless of how they are called. - If no Pid is specified, + calls. Trace messages are sent to the tracer whenever any of + the specified functions are called. If no tracer is specified, self() is used as a default tracer process.

Meta-tracing traces all processes and does not care about the process trace flags set by trace/3, @@ -9013,7 +9021,7 @@ timestamp() -> [call, timestamp].

The match specification function {return_trace} works with meta-trace and sends its trace message to the - same tracer process.

+ same tracer.

 call_count diff --git a/erts/doc/src/match_spec.xml b/erts/doc/src/match_spec.xml index 975f01cf2c..b49e1483aa 100644 --- a/erts/doc/src/match_spec.xml +++ b/erts/doc/src/match_spec.xml @@ -287,7 +287,7 @@ can not be one of the atoms , or (unless, of course, they are registered names). can not be nor - . + . Returns and may only be used in the part when tracing.

@@ -298,7 +298,7 @@ be either a process identifier or a registered name and is given as the first argument to the match_spec function. can not be nor - . Returns + . Returns and may only be used in the part when tracing.

@@ -308,11 +308,14 @@ disable list is applied first, but effectively all changes are applied atomically. The trace flags are the same as for not including - but including . If a + but including . If a tracer is specified in both lists, the tracer in the enable list takes precedence. If no tracer is specified the same tracer as the process executing the match spec is - used. With three parameters to this function the first is + used. When using a tracer module + the module has to be loaded before the match specification is executed. + If it is not loaded the match will fail. + With three parameters to this function the first is either a process identifier or the registered name of a process to set trace flags on, the second is the disable list, and the third is the enable list. Returns diff --git a/erts/doc/src/ref_man.xml b/erts/doc/src/ref_man.xml index aa245ec08a..e45402a397 100644 --- a/erts/doc/src/ref_man.xml +++ b/erts/doc/src/ref_man.xml @@ -56,5 +56,6 @@ + diff --git a/erts/doc/src/specs.xml b/erts/doc/src/specs.xml index 41a3984659..ed6be650e5 100644 --- a/erts/doc/src/specs.xml +++ b/erts/doc/src/specs.xml @@ -2,6 +2,7 @@ + -- cgit v1.2.3