Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4

Since it's been quite a while since I've written C code, *and* I
haven't done any significant hacking on the VM itself in years, it's
quite likely that I haven't done things in 100% proper style.  Or
my co-collaborators Dustin Sallings (CouchBase) or Michal Ptaszek
(Erlang Solutions).  My intent for this patch is to start discussion
and review of DTrace support for consideration for the R15 release.

For additional background on the motivation for this work, please
see the slides for the presentation at the Erlang User Conference 2011
in Stockholm:
  https://www.erlang-factory.com/upload/presentations/462/euc2011-draft2.pdf

Changes relative to dtrace-review2 branch:
    * Fix errors in OTP test suite 'kernel' file_SUITE
    * Fix errors in OTP test suite 'kernel' prim_file_SUITE
    * Fix bad pointer bug in efile_drv.c flush_write()
    * Move the DTrace material from the top of `README.md` into a
      new file, `README.dtrace.md`

Changes since last push to GitHub (relative to commit 5828a4fb28, which
was the former `dtrace-review1` branch):
    * Rebased onto 14 Nov 2011's "master" branch
    * Recent changes to the async task queuing mechanism means that
      the async worker queue length is not available.  A bogus value
      of -1 is hard-coded until there's a good way to peek into the
      new queue structure and find the queue length.
    * Small fixes based on review comments by Mikael Pettersson,
      Andrew Thompson, and Andreas Schultz.

Add autoconf support: use "./configure --enable-dtrace" on all supported
platforms:
    * OS X Snow Leopard or later
    * Solaris 10 or OpenSolaris
    * Linux, via SystemTap's DTrace compatibility packages
    * FreeBSD 9.0RC1.  FreeBSD 8 and earlier do not have support
      for USDT, DTrace's User-land Statically Defined Tracing.

See the file `erts/emulator/beam/erlang_dtrace.d` for the definition
of all DTrace probes in the virtual machine so far.

Example D scripts can be found in `lib/dtrace/examples`.  Note that if
you see the error message `{name of probe} does not match any probes`,
then there is no Erlang VM process + DTrace probes running.  To fix,
start a DTrace-enabled VM or remove `-q` from the `dtrace` command line.

The `lib/dtrace` directory contains a small code-only OTP application
that contains code that allows Erlang code to trigger a DTrace probe.
Dynamic creation & deletion of DTrace probes is not currently
supported, so the `dtrace:p()` function is hacked to allow a variable
number of arguments (up to four integers and up to four strings) to be
used.  See the comments at the top of `lib/dtrace/src/dtrace.c` for
more detail.

One feature that may be controversial is the notion I've introduced
of a special process dictionary key that can be used by Erlang code to
tag I/O operations for an application-specific purpose.  Right now,
that tag's name is `dtrace_utag`.  The dictionary keys used by `sys`
and other modules start with a dollar sign.  Perhaps there is some
convention (but not a dollar sign?) that this tag should use?

The purpose of the process dictionary key is to allow the tag to
be included in trace messages, e.g. for file I/O, without changing the
API of the `file.erl` module's functions.  For example, here's a use
of the tag when calling the `file:rename/2` function:

    (bar@sbb2)1> put(dtrace_utag, "GGOOOAAALL!!!!!").
    undefined

    (bar@sbb2)2> dtrace:init().
    ok

    %% Now start both the `user-probe.d` and `efile_drv.d` D scripts
    %% found in the `lib/dtrace/examples` directory.

    (bar@sbb2)3> dtrace:p(7, 8, 9, "one", "four").
    true

    %% The output from the `user-probe.d` script:
    <0.40.0> GGOOOAAALL!!!!! 7 8 9 0 'one' 'four' '' ''

    (bar@sbb2)4> file:rename("old-name", "new-name").
    {error,enoent}

    %% The output from the `efile_drv.d` script:
    async I/O pool port #Port<0.59> queue len 1
    async I/O pool port #Port<0.59> queue len 0
    efile_drv enter tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | args: old-name new-name , 0 0 (port #Port<0.59>)
    async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_entry
    async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_return
    efile_drv return tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | errno 2

I'm not exactly happy with this choice of tagging, namely using
`put(dtrace_utag, Tag::list())`.  But this is an experiment, so
we'll see how it goes.  I can't imagine changing the API for
all file.erl functions in order pass the tag explicitly.

Some modules have some extensive (ab)use of the C preprocessor to
reduce the amount of #ifdefs that clutter the code.  In several places,
I have not #ifdef'ed automatic variables because of clutter.  For the
same reason, there are a handful of cases where I added DTrace-related
members to a struct definition without an #ifdef.  I feel that the
result is easier to read than earlier drafts where I did use many more
`https://github.com/slfritchie/otp/tree/dtrace-experiment+michal2` if
you're curious.)  I expect there may be some debate about whether the
bloat of the affected structs is worthwhile.  I erred on adding stuff
to structs, especially in the efile_drv.c driver, not having a full
grasp on what was thread-safe and what was not ... so I erred on the
side of caution.

The efile_drv.c has a work-around for a crazy GCC optimization bug.
Thank goodness for Google, I dunno how I would've found a work-around
for this silly thing.  Many thanks to Trond Norbye for writing clearly
about the problem in a membase Git repo commit message.

/*
 * A note on probe naming: if "__" appears in a provider probe
 * definition, then two things happen during compilation:
 *
 *    1. The "__" will turn into a hypen, "-", for the probe name.
 *    2. The "__" will turn into a single underscore, "_", for the
 *       macro names and function definitions that the compiler and
 *       C developers will see.
 *
 * We'll try to use the following naming convention.  We're a bit
 * limited because, as a USDT probe, we can only specify the 4th part
 * of the probe name, e.g. erlang*:::mumble.  The 2nd part of the
 * probe name is always going to be "beam" or "beam.smp", and the 3rd
 * part of the probe name will always be the name of the function
 * that's calling the probe.
 *
 * So, all probes will be have names defined in this file using the
 * convention category__name or category__sub_category__name.  This
 * will translate to probe names of category-name or
 * category-sub_category-name.
 *
 * Each of "category", "sub_category", and "name" may have underscores
 * but may not have hyphens.
 */

Add tentative support for sequential tracing sending, queueing, and
receiving a message.  I don't believe I've fully covered all the major
places where it would be useful to have the sequential trace token info
in a probe -- guidance from the OTP team would be helpful, if there's
time to do that kind of review.

Add global variable `erts_this_node_sysname`.

4 files changed, 383 insertions, 0 deletions

diff --git a/lib/dtrace/c_src/Makefile b/lib/dtrace/c_src/Makefile
new file mode 100644
index 0000000000..f3320bb766
--- /dev/null
+++ b/lib/dtrace/c_src/Makefile
@@ -0,0 +1,4 @@
+#
+# Invoke with GNU make or clearmake -C gnu.
+#
+include $(ERL_TOP)/make/run_make.mk
diff --git a/lib/dtrace/c_src/Makefile.in b/lib/dtrace/c_src/Makefile.in
new file mode 100644
index 0000000000..ed13684a95
--- /dev/null
+++ b/lib/dtrace/c_src/Makefile.in
@@ -0,0 +1,152 @@
+#
+# %CopyrightBegin%
+#
+# Copyright Scott Lystig Fritchie 2011. All Rights Reserved.
+#
+# The contents of this file are subject to the Erlang Public License,
+# Version 1.1, (the "License"); you may not use this file except in
+# compliance with the License. You should have received a copy of the
+# Erlang Public License along with this software. If not, it can be
+# retrieved online at http://www.erlang.org/.
+#
+# Software distributed under the License is distributed on an "AS IS"
+# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+# the License for the specific language governing rights and limitations
+# under the License.
+#
+# %CopyrightEnd%
+#
+include $(ERL_TOP)/make/target.mk
+include $(ERL_TOP)/make/$(TARGET)/otp.mk
+include $(ERL_TOP)/make/$(TARGET)/otp_ded.mk
+
+# ----------------------------------------------------
+# Items from top-level configure
+# ----------------------------------------------------
+DTRACE_ENABLED=@DTRACE_ENABLED@
+DTRACE_ENABLED_2STEP=@DTRACE_ENABLED_2STEP@
+
+# ----------------------------------------------------
+# Application version
+# ----------------------------------------------------
+include ../vsn.mk
+VSN=$(DTRACE_VSN)
+
+# ----------------------------------------------------
+# The following variables differ between systems.
+# Set by configure.
+# ----------------------------------------------------
+CC = $(DED_CC)
+LD = $(DED_LD)
+SHELL = /bin/sh
+LIBS = $(DED_LIBS)
+LDFLAGS += $(DED_LDFLAGS)
+CFLAGS = $(DED_CFLAGS)
+
+DTRACE_LIBNAME = dtrace
+
+
+INCLUDES = $(DED_INCLUDES)
+
+ifeq ($(TYPE),debug)
+TYPEMARKER = .debug
+TYPE_FLAGS = $(subst -O3,,$(subst -O2,,$(CFLAGS))) -DDEBUG
+else
+ifeq ($(TYPE),valgrind)
+TYPEMARKER = .valgrind
+TYPE_FLAGS = $(subst -O3,,$(subst -O2,,$(CFLAGS))) -DVALGRIND
+else
+TYPEMARKER =
+TYPE_FLAGS = $(CFLAGS)
+endif
+endif
+
+ALL_CFLAGS = $(TYPE_FLAGS) $(INCLUDES) -I$(OBJDIR) \
+    -I$(ERL_TOP)/erts/emulator/$(TARGET)
+
+# ----------------------------------------------------
+# Release directory specification
+# ----------------------------------------------------
+RELSYSDIR = $(RELEASE_PATH)/lib/dtrace-$(VSN)
+
+# ----------------------------------------------------
+# Misc Macros
+# ----------------------------------------------------
+OBJS = $(OBJDIR)/dtrace$(TYPEMARKER).o
+## NIF_MAKEFILE = $(PRIVDIR)/Makefile
+
+# Higher-level makefiles says that we can only compile on UNIX flavors
+NIF_LIB = $(LIBDIR)/dtrace$(TYPEMARKER).so
+
+ifeq ($(HOST_OS),)
+HOST_OS := $(shell $(ERL_TOP)/erts/autoconf/config.guess)
+endif
+
+# ----------------------------------------------------
+# Targets
+# ----------------------------------------------------
+
+debug opt valgrind: $(OBJDIR) $(LIBDIR) $(NIF_LIB)
+
+ifdef DTRACE_ENABLED
+DTRACE_USER_HEADER=$(OBJDIR)/dtrace_user.h
+$(OBJDIR)/dtrace_user.h: ./dtrace_user.d
+	dtrace -h -C $(INCLUDES) \
+	  -s ./dtrace_user.d \
+	  -o ./dtrace_user.tmp
+	sed -e '/^#define[ 	]*ERLANG_[A-Z0-9_]*(.*)/y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' ./dtrace_user.tmp > $@
+	rm ./dtrace_user.tmp
+else
+DTRACE_USER_HEADER=
+endif
+
+ifdef DTRACE_ENABLED_2STEP
+OBJS += $(OBJDIR)/dtrace_user.o
+$(OBJDIR)/dtrace_user.o: $(OBJS) $(OBJDIR)/dtrace_user.h
+	touch $(OBJDIR)/erlang_dtrace.c
+	$(CC) $(CFLAGS) -c -o $@ $(OBJDIR)/erlang_dtrace.c
+	# The object file created above is immediately clobbered below.
+	# But creating it above avoids chicken-and-egg problem with OBJS
+	dtrace -G -C \
+	  -s ./dtrace_user.d \
+	  -o $@ $(OBJS)
+endif
+
+$(OBJDIR):
+	-@mkdir -p $(OBJDIR)
+
+$(LIBDIR):
+	-@mkdir -p $(LIBDIR)
+
+$(OBJDIR)/%$(TYPEMARKER).o: %.c $(DTRACE_USER_HEADER)
+	$(INSTALL_DIR) $(OBJDIR)
+	$(CC) -c -o $@ $(ALL_CFLAGS) $<
+
+$(LIBDIR)/dtrace$(TYPEMARKER).so: $(OBJS)
+	$(INSTALL_DIR) $(LIBDIR)
+	$(LD) $(LDFLAGS) -o $@ $^ $(LDLIBS)
+
+clean:
+	rm -f $(LIBDIR)/dtrace.so
+	rm -f $(LIBDIR)/dtrace.debug.so
+	rm -f $(LIBDIR)/dtrace.valgrind.so
+	rm -f $(OBJDIR)/dtrace.o
+	rm -f $(OBJDIR)/dtrace.debug.o
+	rm -f $(OBJDIR)/dtrace.valgrind.o
+	rm -f core *~
+
+docs:
+
+# ----------------------------------------------------
+# Release Target
+# ----------------------------------------------------
+include $(ERL_TOP)/make/otp_release_targets.mk
+
+release_spec: opt
+	$(INSTALL_DIR) $(RELSYSDIR)/priv/obj
+	$(INSTALL_DIR) $(RELSYSDIR)/priv/lib
+	#  $(INSTALL_DATA) $(NIF_MAKEFILE) $(RELSYSDIR)/priv/obj
+	$(INSTALL_PROGRAM) $(OBJS) $(RELSYSDIR)/priv/obj
+	$(INSTALL_PROGRAM) $(NIF_LIB) $(RELSYSDIR)/priv/lib
+
+release_docs_spec:
diff --git a/lib/dtrace/c_src/dtrace.c b/lib/dtrace/c_src/dtrace.c
new file mode 100644
index 0000000000..c9d25ece9c
--- /dev/null
+++ b/lib/dtrace/c_src/dtrace.c
@@ -0,0 +1,174 @@
+/*
+ * %CopyrightBegin%
+ *
+ * Copyright Scott Lystig Fritchie 2011. All Rights Reserved.
+ *
+ * The contents of this file are subject to the Erlang Public License,
+ * Version 1.1, (the "License"); you may not use this file except in
+ * compliance with the License. You should have received a copy of the
+ * Erlang Public License along with this software. If not, it can be
+ * retrieved online at http://www.erlang.org/.
+ *
+ * Software distributed under the License is distributed on an "AS IS"
+ * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ * the License for the specific language governing rights and limitations
+ * under the License.
+ *
+ * %CopyrightEnd%
+ */
+
+/*
+ * Purpose:  Dynamically loadable NIF library for DTrace
+ */
+
+
+#include "erl_nif.h"
+#include "config.h"
+#include "sys.h"
+#define DTRACE_DRIVER_SKIP_FUNC_DECLARATIONS
+#include "dtrace-wrapper.h"
+#ifdef  HAVE_DTRACE
+#include "dtrace_user.h"
+#endif
+
+void dtrace_nifenv_str(ErlNifEnv *env, char *process_buf);
+void get_string_maybe(ErlNifEnv *env, const ERL_NIF_TERM term, char **ptr, char *buf, int bufsiz);
+
+#ifdef VALGRIND
+    #  include <valgrind/memcheck.h>
+#endif
+
+#ifdef __GNUC__
+    #  define INLINE __inline__
+#else
+    #  define INLINE
+#endif
+
+#define MESSAGE_BUFSIZ 1024
+
+/* NIF interface declarations */
+static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info);
+
+/* The NIFs: */
+static ERL_NIF_TERM available(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+static ERL_NIF_TERM user_trace_s1(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+static ERL_NIF_TERM user_trace_i4s4(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+
+static ErlNifFunc nif_funcs[] = {
+    {"available", 0, available},
+    {"user_trace_s1", 1, user_trace_s1},
+    {"user_trace_i4s4", 9, user_trace_i4s4}
+};
+
+ERL_NIF_INIT(dtrace, nif_funcs, load, NULL, NULL, NULL)
+
+static ERL_NIF_TERM atom_true;
+static ERL_NIF_TERM atom_false;
+static ERL_NIF_TERM atom_error;
+static ERL_NIF_TERM atom_not_available;
+static ERL_NIF_TERM atom_badarg;
+static ERL_NIF_TERM atom_ok;
+
+static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info)
+{
+    atom_true = enif_make_atom(env,"true");
+    atom_false = enif_make_atom(env,"false");
+    atom_error = enif_make_atom(env,"error");
+    atom_not_available = enif_make_atom(env,"not_available");
+    atom_badarg = enif_make_atom(env,"badarg");
+    atom_ok = enif_make_atom(env,"ok");
+
+    return 0;
+}
+
+static ERL_NIF_TERM available(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+#ifdef	HAVE_DTRACE
+    return atom_true;
+#else
+    return atom_false;
+#endif
+}
+
+static ERL_NIF_TERM user_trace_s1(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+#ifdef	HAVE_DTRACE
+    ErlNifBinary message_bin;
+    DTRACE_CHARBUF(messagebuf, MESSAGE_BUFSIZ + 1);
+
+    if (DTRACE_ENABLED(user_trace_s1)) {
+	if (!enif_inspect_iolist_as_binary(env, argv[0], &message_bin) ||
+	    message_bin.size > MESSAGE_BUFSIZ) {
+	    return atom_badarg;
+	}
+	memcpy(messagebuf, (char *) message_bin.data, message_bin.size);
+        messagebuf[message_bin.size] = '\0';
+	DTRACE1(user_trace_s1, messagebuf);
+	return atom_true;
+    } else {
+	return atom_false;
+    }
+#else
+    return atom_error;
+#endif
+}
+
+void
+get_string_maybe(ErlNifEnv *env,
+                 const ERL_NIF_TERM term, char **ptr, char *buf, int bufsiz)
+{
+    ErlNifBinary str_bin;
+
+    if (!enif_inspect_iolist_as_binary(env, term, &str_bin) ||
+        str_bin.size > bufsiz) {
+        *ptr = NULL;
+    } else {
+        memcpy(buf, (char *) str_bin.data, str_bin.size);
+        buf[str_bin.size] = '\0';
+        *ptr = buf;
+    }
+}
+
+static ERL_NIF_TERM user_trace_i4s4(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+#ifdef	HAVE_DTRACE
+    DTRACE_CHARBUF(procbuf, 32 + 1);
+    DTRACE_CHARBUF(user_tagbuf, MESSAGE_BUFSIZ + 1);
+    char *utbuf = NULL;
+    ErlNifSInt64 i1, i2, i3, i4;
+    DTRACE_CHARBUF(messagebuf1, MESSAGE_BUFSIZ + 1);
+    DTRACE_CHARBUF(messagebuf2, MESSAGE_BUFSIZ + 1);
+    DTRACE_CHARBUF(messagebuf3, MESSAGE_BUFSIZ + 1);
+    DTRACE_CHARBUF(messagebuf4, MESSAGE_BUFSIZ + 1);
+    char *mbuf1 = NULL, *mbuf2 = NULL, *mbuf3 = NULL, *mbuf4 = NULL;
+
+    if (DTRACE_ENABLED(user_trace_i4s4)) {
+	dtrace_nifenv_str(env, procbuf);
+        get_string_maybe(env, argv[0], &utbuf,
+                         user_tagbuf, sizeof(user_tagbuf)-1);
+        if (! enif_get_int64(env, argv[1], &i1))
+            i1 = 0;
+        if (! enif_get_int64(env, argv[2], &i2))
+            i2 = 0;
+        if (! enif_get_int64(env, argv[3], &i3))
+            i3 = 0;
+        if (! enif_get_int64(env, argv[4], &i4))
+            i4 = 0;
+        get_string_maybe(env, argv[5], &mbuf1,
+                         messagebuf1, sizeof(messagebuf1)-1);
+        get_string_maybe(env, argv[6], &mbuf2,
+                         messagebuf2, sizeof(messagebuf2)-1);
+        get_string_maybe(env, argv[7], &mbuf3,
+                         messagebuf3, sizeof(messagebuf3)-1);
+        get_string_maybe(env, argv[8], &mbuf4,
+                         messagebuf4, sizeof(messagebuf4)-1);
+	DTRACE10(user_trace_i4s4, procbuf, utbuf,
+		 i1, i2, i3, i4, mbuf1, mbuf2, mbuf3, mbuf4);
+	return atom_true;
+    } else {
+	return atom_false;
+    }
+#else
+    return atom_error;
+#endif
+}
diff --git a/lib/dtrace/c_src/dtrace_user.d b/lib/dtrace/c_src/dtrace_user.d
new file mode 100644
index 0000000000..45d3ef3b66
--- /dev/null
+++ b/lib/dtrace/c_src/dtrace_user.d
@@ -0,0 +1,53 @@
+/*
+ * %CopyrightBegin%
+ *
+ * Copyright Scott Lystig Fritchie 2011.
+ * All Rights Reserved.
+ *
+ * The contents of this file are subject to the Erlang Public License,
+ * Version 1.1, (the "License"); you may not use this file except in
+ * compliance with the License. You should have received a copy of the
+ * Erlang Public License along with this software. If not, it can be
+ * retrieved online at http://www.erlang.org/.
+ *
+ * Software distributed under the License is distributed on an "AS IS"
+ * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ * the License for the specific language governing rights and limitations
+ * under the License.
+ *
+ * %CopyrightEnd%
+ */
+
+provider erlang {
+    /**
+     * Send a single string to a probe.
+     *
+     * @param NUL-terminated string
+     */
+    probe user_trace__s1(char* message);
+
+    /**
+     * Multi-purpose probe: up to 4 NUL-terminated strings and 4
+     * 64-bit integer arguments.
+     *
+     * @param proc, the PID (string form) of the sending process
+     * @param user_tag, the user tag of the sender
+     * @param i1, integer
+     * @param i2, integer
+     * @param i3, integer
+     * @param i4, integer
+     * @param s1, string/iolist. D's arg6 is NULL if not given by Erlang
+     * @param s2, string/iolist. D's arg7 is NULL if not given by Erlang
+     * @param s3, string/iolist. D's arg8 is NULL if not given by Erlang
+     * @param s4, string/iolist. D's arg9 is NULL if not given by Erlang
+     */
+    probe user_trace__i4s4(char *proc, char *user_tag,
+                           int i1, int i2, int i3, int i4,
+                           char *s1, char *s2, char *s3, char *s4);
+};
+
+#pragma D attributes Evolving/Evolving/Common provider erlang provider
+#pragma D attributes Private/Private/Common provider erlang module
+#pragma D attributes Private/Private/Common provider erlang function
+#pragma D attributes Evolving/Evolving/Common provider erlang name
+#pragma D attributes Evolving/Evolving/Common provider erlang args