| Age | Commit message (Collapse) | Author |
|---|
|
OTP-10106
OTP-10107
|
|
|
|
|
|
Since it's been quite a while since I've written C code, *and* I
haven't done any significant hacking on the VM itself in years, it's
quite likely that I haven't done things in 100% proper style. Or
my co-collaborators Dustin Sallings (CouchBase) or Michal Ptaszek
(Erlang Solutions). My intent for this patch is to start discussion
and review of DTrace support for consideration for the R15 release.
For additional background on the motivation for this work, please
see the slides for the presentation at the Erlang User Conference 2011
in Stockholm:
https://www.erlang-factory.com/upload/presentations/462/euc2011-draft2.pdf
Changes relative to dtrace-review2 branch:
* Fix errors in OTP test suite 'kernel' file_SUITE
* Fix errors in OTP test suite 'kernel' prim_file_SUITE
* Fix bad pointer bug in efile_drv.c flush_write()
* Move the DTrace material from the top of `README.md` into a
new file, `README.dtrace.md`
Changes since last push to GitHub (relative to commit 5828a4fb28, which
was the former `dtrace-review1` branch):
* Rebased onto 14 Nov 2011's "master" branch
* Recent changes to the async task queuing mechanism means that
the async worker queue length is not available. A bogus value
of -1 is hard-coded until there's a good way to peek into the
new queue structure and find the queue length.
* Small fixes based on review comments by Mikael Pettersson,
Andrew Thompson, and Andreas Schultz.
Add autoconf support: use "./configure --enable-dtrace" on all supported
platforms:
* OS X Snow Leopard or later
* Solaris 10 or OpenSolaris
* Linux, via SystemTap's DTrace compatibility packages
* FreeBSD 9.0RC1. FreeBSD 8 and earlier do not have support
for USDT, DTrace's User-land Statically Defined Tracing.
See the file `erts/emulator/beam/erlang_dtrace.d` for the definition
of all DTrace probes in the virtual machine so far.
Example D scripts can be found in `lib/dtrace/examples`. Note that if
you see the error message `{name of probe} does not match any probes`,
then there is no Erlang VM process + DTrace probes running. To fix,
start a DTrace-enabled VM or remove `-q` from the `dtrace` command line.
The `lib/dtrace` directory contains a small code-only OTP application
that contains code that allows Erlang code to trigger a DTrace probe.
Dynamic creation & deletion of DTrace probes is not currently
supported, so the `dtrace:p()` function is hacked to allow a variable
number of arguments (up to four integers and up to four strings) to be
used. See the comments at the top of `lib/dtrace/src/dtrace.c` for
more detail.
One feature that may be controversial is the notion I've introduced
of a special process dictionary key that can be used by Erlang code to
tag I/O operations for an application-specific purpose. Right now,
that tag's name is `dtrace_utag`. The dictionary keys used by `sys`
and other modules start with a dollar sign. Perhaps there is some
convention (but not a dollar sign?) that this tag should use?
The purpose of the process dictionary key is to allow the tag to
be included in trace messages, e.g. for file I/O, without changing the
API of the `file.erl` module's functions. For example, here's a use
of the tag when calling the `file:rename/2` function:
(bar@sbb2)1> put(dtrace_utag, "GGOOOAAALL!!!!!").
undefined
(bar@sbb2)2> dtrace:init().
ok
%% Now start both the `user-probe.d` and `efile_drv.d` D scripts
%% found in the `lib/dtrace/examples` directory.
(bar@sbb2)3> dtrace:p(7, 8, 9, "one", "four").
true
%% The output from the `user-probe.d` script:
<0.40.0> GGOOOAAALL!!!!! 7 8 9 0 'one' 'four' '' ''
(bar@sbb2)4> file:rename("old-name", "new-name").
{error,enoent}
%% The output from the `efile_drv.d` script:
async I/O pool port #Port<0.59> queue len 1
async I/O pool port #Port<0.59> queue len 0
efile_drv enter tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | args: old-name new-name , 0 0 (port #Port<0.59>)
async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_entry
async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_return
efile_drv return tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | errno 2
I'm not exactly happy with this choice of tagging, namely using
`put(dtrace_utag, Tag::list())`. But this is an experiment, so
we'll see how it goes. I can't imagine changing the API for
all file.erl functions in order pass the tag explicitly.
Some modules have some extensive (ab)use of the C preprocessor to
reduce the amount of #ifdefs that clutter the code. In several places,
I have not #ifdef'ed automatic variables because of clutter. For the
same reason, there are a handful of cases where I added DTrace-related
members to a struct definition without an #ifdef. I feel that the
result is easier to read than earlier drafts where I did use many more
`https://github.com/slfritchie/otp/tree/dtrace-experiment+michal2` if
you're curious.) I expect there may be some debate about whether the
bloat of the affected structs is worthwhile. I erred on adding stuff
to structs, especially in the efile_drv.c driver, not having a full
grasp on what was thread-safe and what was not ... so I erred on the
side of caution.
The efile_drv.c has a work-around for a crazy GCC optimization bug.
Thank goodness for Google, I dunno how I would've found a work-around
for this silly thing. Many thanks to Trond Norbye for writing clearly
about the problem in a membase Git repo commit message.
/*
* A note on probe naming: if "__" appears in a provider probe
* definition, then two things happen during compilation:
*
* 1. The "__" will turn into a hypen, "-", for the probe name.
* 2. The "__" will turn into a single underscore, "_", for the
* macro names and function definitions that the compiler and
* C developers will see.
*
* We'll try to use the following naming convention. We're a bit
* limited because, as a USDT probe, we can only specify the 4th part
* of the probe name, e.g. erlang*:::mumble. The 2nd part of the
* probe name is always going to be "beam" or "beam.smp", and the 3rd
* part of the probe name will always be the name of the function
* that's calling the probe.
*
* So, all probes will be have names defined in this file using the
* convention category__name or category__sub_category__name. This
* will translate to probe names of category-name or
* category-sub_category-name.
*
* Each of "category", "sub_category", and "name" may have underscores
* but may not have hyphens.
*/
Add tentative support for sequential tracing sending, queueing, and
receiving a message. I don't believe I've fully covered all the major
places where it would be useful to have the sequential trace token info
in a probe -- guidance from the OTP team would be helpful, if there's
time to do that kind of review.
Add global variable `erts_this_node_sysname`.
|
|
|
|
|
|
* Make generate and depend.mk automatically.
* Do not make depend.mk for targets clean and generate.
* Remove old replaced CREATE_DIRS cruft.
* Fiercer remove by target clean.
* Move depend.mk to Target/Type/Flavor directory.
|
|
|
|
Use a make timestamp file to condense dependencies to some
part(s) of erts/lib_src build results.
|
|
Convert the recursive make for zlib into a make include file.
|
|
Still does not run, just compiles.
|
|
|
|
|
|
* rickard/generic-thr-queue/OTP-9632:
Use generic lock-free queue for async threads
Use generic lock-free queue for misc aux work
Implement generic lock-free queue
|
|
* rickard/alloc-opt/OTP-7775:
Optimize memory allocation
Conflicts:
erts/aclocal.m4
erts/emulator/hipe/hipe_bif_list.m4
erts/preloaded/ebin/erl_prim_loader.beam
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/init.beam
erts/preloaded/ebin/otp_ring0.beam
erts/preloaded/ebin/prim_file.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/ebin/prim_zip.beam
erts/preloaded/ebin/zlib.beam
|
|
The implementation of an ERTS internal, generic, many to one, lock-free
queue for communication between threads. The many to one scenario is
very common in ERTS, so it can be used in a lot of places in the future.
Changing to this queue from a lock based queue, however, often requires
some redesigning. This since we have often used the lock of the queue
to protect other information too.
|
|
A number of memory allocation optimizations have been implemented. Most
optimizations reduce contention caused by synchronization between
threads during allocation and deallocation of memory. Most notably:
* Synchronization of memory management in scheduler specific allocator
instances has been rewritten to use lock-free synchronization.
* Synchronization of memory management in scheduler specific
pre-allocators has been rewritten to use lock-free synchronization.
* The 'mseg_alloc' memory segment allocator now use scheduler specific
instances instead of one instance. Apart from reducing contention
this also ensures that memory allocators always create memory
segments on the local NUMA node on a NUMA system.
|
|
* dev:
erl_bif_types: Fix types for lists:key{search,find,member}/3
Fix build problems on MacOS 10.7 (Lion)
|
|
On MacOS 10.7 (Lion) with Xcode 4.1 installed, the default
C compiler is llvm-gcc-4.2. That compiler compiles beam_emu.c
incorrectly, resulting in an emulator that will not start.
The problem can be worked around by turning off all optimizations,
but that will significantly degrade the performance of the run-time
system. The problem can also be worked around manually like this:
CC=gcc-4.2 ./configure
To allow a working system to be built "out of the box", have
the configure script set up the emulator Makefile so that gcc-4.2
will be used to compile beam_emu.c if the default compiler is
llvm-based. All other C files will still be compiled with the
default compiler.
|
|
To avoid issues with parallel make and to slightly speed up the
build process, avoid a recursive make by replacing pcre/Makefile.in
with pcre/pcre.mk and including it from the main emulator Makefile.
|
|
|
|
|
|
|
|
|
|
|
|
Giving the beam_makeops script access to the external word
size (=the size of instruction words) will allow it to pack
more operands into a word for the 64 bits emulator.
|
|
|
|
|
|
* rickard/cpu-groups/OTP-8861:
Generalize reader groups
Move cpu topology functionality into erl_cpu_topology.[ch]
Do not use more reader groups for schedulers than schedulers
Conflicts:
erts/emulator/beam/erl_init.c
|
|
|
|
Make sure that an update to erts/emulator/tools/make_tables will
force all generated files to be re-generated.
|
|
Added erlang:system_info(build_type) which makes it
easier to chose drivers, NIF libraries, etc based
on build type of the runtime system.
|
|
elib_malloc is an alternate memory allocator that
is no longer possible to build.
|
|
* bjorn/remove-obsolete-driver-support/OTP-8758:
Remove obsolete/driver.h and the associated functionality
|
|
The obsolete/driver.h header file has been obsolete
since R8B.
Remove that file, along with obsolete thread APIs for
drivers in the emulator and the test cases.
|
|
Large parts of the ethread library have been rewritten. The
ethread library is an Erlang runtime system internal, portable
thread library used by the runtime system itself.
Most notable improvement is a reader optimized rwlock
implementation which dramatically improve the performance of
read-lock/read-unlock operations on multi processor systems by
avoiding ping-ponging of the rwlock cache lines. The reader
optimized rwlock implementation is used by miscellaneous
rwlocks in the runtime system that are known to be read-locked
frequently, and can be enabled on ETS tables by passing the
`{read_concurrency, true}' option upon table creation. See the
documentation of `ets:new/2' for more information.
The ethread library can now also use the libatomic_ops library
for atomic memory accesses. This makes it possible for the
Erlang runtime system to utilize optimized atomic operations
on more platforms than before. Use the
`--with-libatomic_ops=PATH' configure command line argument
when specifying where the libatomic_ops installation is
located. The libatomic_ops library can be downloaded from:
http://www.hpl.hp.com/research/linux/atomic_ops/
The changed API of the ethread library has also caused
modifications in the Erlang runtime system. Preparations for
the to come "delayed deallocation" feature has also been done
since it depends on the ethread library.
Note: When building for x86, the ethread library will now use
instructions that first appeared on the pentium 4 processor. If
you want the runtime system to be compatible with older
processors (back to 486) you need to pass the
`--enable-ethread-pre-pentium4-compatibility' configure command
line argument when configuring the system.
|
|
The experimental Standalone Erlang (SAE) support based on
Joe Armstrong's work has long been broken. Remove the remaining
code and Makefile rules.
|
|
|
|
|
|
|
|
* gc/hipe_darwin_amd64:
Fix hipe memory allocation problems on darwin/amd64
Porting x86 darwin fixes to amd64 darwin hipe asm/m4 code
Automatically enable hipe for darwin/amd64 builds
Allow configure to enable_hipe for darwin/amd64 builds
OTP-8416 HiPE now works in the 64-bit emulator on Mac OS X. (Thanks to
Geoff Cant.)
|
|
tile-cc 2.0.1.78377 when compiling the runtime system.
|
|
HiPE on amd64 needs to be able to allocate memory in the 0x20000000+
range so that compiled code will be addressable with 32bit pointers
(when sign extended). By default, when building 64bit executables on
OS X, PAGEZERO occupies the first 4GB of virtual memory. The PAGEZERO
segment cannot be mmap(ed) or vm_allocate(d) into.
To avoid this problem we use the -pagezero_size linker flag to reduce
the size of PAGEZERO and free up the 0x20000000+ space.
|
|
checking if CC equaled gcc. That is, the makefiles failed to
detect gcc C compilers with other command line names than gcc.
`configure' now substitute GCC into the makefiles. If CC is a gcc
C compiler, GCC will have the value yes. (Thanks to
Jean-S�bastien P�dron)
|
|
The Apple compiler creates DWARF debugging symbols in a bundle (ie. a
directory) named <name>.dSYM
In order for make clean to work correctly, the rm command needs -r
|
|
|
