erts: Implement tracer modules

Add the possibility to use modules as trace data receivers. The functions
in the module have to be nifs as otherwise complex trace probes will be
very hard to handle (complex means trace probes for ports for example).

This commit changes the way that the ptab->tracer field works from always
being an immediate, to now be NIL if no tracer is present or else be
the tuple {TracerModule, TracerState} where TracerModule is an atom that
is later used to lookup the appropriate tracer callbacks to call and
TracerState is just passed to the tracer callback. The default process and
port tracers have been rewritten to use the new API.

This commit also changes the order which trace messages are delivered to the
potential tracer process. Any enif_send done in a tracer module may be delayed
indefinitely because of lock order issues. If a message is delayed any other
trace message send from that process is also delayed so that order is preserved
for each traced entity. This means that for some trace events (i.e. send/receive)
the events may come in an unintuitive order (receive before send) to the
trace receiver. Timestamps are taken when the trace message is generated so
trace messages from differented processes may arrive with the timestamp
out of order.

Both the erlang:trace and seq_trace:set_system_tracer accept the new tracer
module tracers and also the backwards compatible arguments.

OTP-10267

1 files changed, 7 insertions, 4 deletions

diff --git a/erts/doc/src/match_spec.xml b/erts/doc/src/match_spec.xml
index 975f01cf2c..b49e1483aa 100644
--- a/erts/doc/src/match_spec.xml
+++ b/erts/doc/src/match_spec.xml
@@ -287,7 +287,7 @@
         can <em>not</em> be one of the atoms <c><![CDATA[all]]></c>, <c><![CDATA[new]]></c> or 
         <c><![CDATA[existing]]></c> (unless, of course, they are registered names).
         <c><![CDATA[P2]]></c> can <em>not</em> be <c><![CDATA[cpu_timestamp]]></c> nor
-        <c><![CDATA[{tracer,_}]]></c>.
+        <c><![CDATA[tracer]]></c>.
         Returns <c><![CDATA[true]]></c> and may only be used in
         the <c><![CDATA[MatchBody]]></c> part when tracing.
         </p>
@@ -298,7 +298,7 @@
         be either a process identifier or a registered name and is given
         as the first argument to the match_spec function. 
         <c><![CDATA[P2]]></c> can <em>not</em> be <c><![CDATA[cpu_timestamp]]></c> nor
-        <c><![CDATA[{tracer,_}]]></c>. Returns
+        <c><![CDATA[tracer]]></c>. Returns
         <c><![CDATA[true]]></c> and may only be used in the <c><![CDATA[MatchBody]]></c> part
         when tracing.
         </p>
@@ -308,11 +308,14 @@
         disable list is applied first, but effectively all changes
         are applied atomically. The trace flags
         are the same as for <c><![CDATA[erlang:trace/3]]></c> not including
-        <c><![CDATA[cpu_timestamp]]></c> but including <c><![CDATA[{tracer,_}]]></c>. If a
+        <c><![CDATA[cpu_timestamp]]></c> but including <c><![CDATA[tracer]]></c>. If a
         tracer is specified in both lists, the tracer in the 
         enable list takes precedence. If no tracer is specified the
         same tracer as the process executing the match spec is 
-        used. With three parameters to this function the first is 
+        used. When using a <seealso marker="erl_tracer">tracer module</seealso>
+        the module has to be loaded before the match specification is executed.
+        If it is not loaded the match will fail.
+        With three parameters to this function the first is
         either a process identifier or the registered name of a
         process to set trace flags on, the second is the disable
         list, and the third is the enable list. Returns