aboutsummaryrefslogtreecommitdiffstats
path: root/erts/emulator
AgeCommit message (Collapse)Author
2012-03-22Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4Scott Lystig Fritchie
Since it's been quite a while since I've written C code, *and* I haven't done any significant hacking on the VM itself in years, it's quite likely that I haven't done things in 100% proper style. Or my co-collaborators Dustin Sallings (CouchBase) or Michal Ptaszek (Erlang Solutions). My intent for this patch is to start discussion and review of DTrace support for consideration for the R15 release. For additional background on the motivation for this work, please see the slides for the presentation at the Erlang User Conference 2011 in Stockholm: https://www.erlang-factory.com/upload/presentations/462/euc2011-draft2.pdf Changes relative to dtrace-review2 branch: * Fix errors in OTP test suite 'kernel' file_SUITE * Fix errors in OTP test suite 'kernel' prim_file_SUITE * Fix bad pointer bug in efile_drv.c flush_write() * Move the DTrace material from the top of `README.md` into a new file, `README.dtrace.md` Changes since last push to GitHub (relative to commit 5828a4fb28, which was the former `dtrace-review1` branch): * Rebased onto 14 Nov 2011's "master" branch * Recent changes to the async task queuing mechanism means that the async worker queue length is not available. A bogus value of -1 is hard-coded until there's a good way to peek into the new queue structure and find the queue length. * Small fixes based on review comments by Mikael Pettersson, Andrew Thompson, and Andreas Schultz. Add autoconf support: use "./configure --enable-dtrace" on all supported platforms: * OS X Snow Leopard or later * Solaris 10 or OpenSolaris * Linux, via SystemTap's DTrace compatibility packages * FreeBSD 9.0RC1. FreeBSD 8 and earlier do not have support for USDT, DTrace's User-land Statically Defined Tracing. See the file `erts/emulator/beam/erlang_dtrace.d` for the definition of all DTrace probes in the virtual machine so far. Example D scripts can be found in `lib/dtrace/examples`. Note that if you see the error message `{name of probe} does not match any probes`, then there is no Erlang VM process + DTrace probes running. To fix, start a DTrace-enabled VM or remove `-q` from the `dtrace` command line. The `lib/dtrace` directory contains a small code-only OTP application that contains code that allows Erlang code to trigger a DTrace probe. Dynamic creation & deletion of DTrace probes is not currently supported, so the `dtrace:p()` function is hacked to allow a variable number of arguments (up to four integers and up to four strings) to be used. See the comments at the top of `lib/dtrace/src/dtrace.c` for more detail. One feature that may be controversial is the notion I've introduced of a special process dictionary key that can be used by Erlang code to tag I/O operations for an application-specific purpose. Right now, that tag's name is `dtrace_utag`. The dictionary keys used by `sys` and other modules start with a dollar sign. Perhaps there is some convention (but not a dollar sign?) that this tag should use? The purpose of the process dictionary key is to allow the tag to be included in trace messages, e.g. for file I/O, without changing the API of the `file.erl` module's functions. For example, here's a use of the tag when calling the `file:rename/2` function: (bar@sbb2)1> put(dtrace_utag, "GGOOOAAALL!!!!!"). undefined (bar@sbb2)2> dtrace:init(). ok %% Now start both the `user-probe.d` and `efile_drv.d` D scripts %% found in the `lib/dtrace/examples` directory. (bar@sbb2)3> dtrace:p(7, 8, 9, "one", "four"). true %% The output from the `user-probe.d` script: <0.40.0> GGOOOAAALL!!!!! 7 8 9 0 'one' 'four' '' '' (bar@sbb2)4> file:rename("old-name", "new-name"). {error,enoent} %% The output from the `efile_drv.d` script: async I/O pool port #Port<0.59> queue len 1 async I/O pool port #Port<0.59> queue len 0 efile_drv enter tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | args: old-name new-name , 0 0 (port #Port<0.59>) async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_entry async I/O worker tag={1,110} | RENAME (12) | efile_drv-int_return efile_drv return tag={1,110} user tag GGOOOAAALL!!!!! | RENAME (12) | errno 2 I'm not exactly happy with this choice of tagging, namely using `put(dtrace_utag, Tag::list())`. But this is an experiment, so we'll see how it goes. I can't imagine changing the API for all file.erl functions in order pass the tag explicitly. Some modules have some extensive (ab)use of the C preprocessor to reduce the amount of #ifdefs that clutter the code. In several places, I have not #ifdef'ed automatic variables because of clutter. For the same reason, there are a handful of cases where I added DTrace-related members to a struct definition without an #ifdef. I feel that the result is easier to read than earlier drafts where I did use many more `https://github.com/slfritchie/otp/tree/dtrace-experiment+michal2` if you're curious.) I expect there may be some debate about whether the bloat of the affected structs is worthwhile. I erred on adding stuff to structs, especially in the efile_drv.c driver, not having a full grasp on what was thread-safe and what was not ... so I erred on the side of caution. The efile_drv.c has a work-around for a crazy GCC optimization bug. Thank goodness for Google, I dunno how I would've found a work-around for this silly thing. Many thanks to Trond Norbye for writing clearly about the problem in a membase Git repo commit message. /* * A note on probe naming: if "__" appears in a provider probe * definition, then two things happen during compilation: * * 1. The "__" will turn into a hypen, "-", for the probe name. * 2. The "__" will turn into a single underscore, "_", for the * macro names and function definitions that the compiler and * C developers will see. * * We'll try to use the following naming convention. We're a bit * limited because, as a USDT probe, we can only specify the 4th part * of the probe name, e.g. erlang*:::mumble. The 2nd part of the * probe name is always going to be "beam" or "beam.smp", and the 3rd * part of the probe name will always be the name of the function * that's calling the probe. * * So, all probes will be have names defined in this file using the * convention category__name or category__sub_category__name. This * will translate to probe names of category-name or * category-sub_category-name. * * Each of "category", "sub_category", and "name" may have underscores * but may not have hyphens. */ Add tentative support for sequential tracing sending, queueing, and receiving a message. I don't believe I've fully covered all the major places where it would be useful to have the sequential trace token info in a probe -- guidance from the OTP team would be helpful, if there's time to do that kind of review. Add global variable `erts_this_node_sysname`.
2012-03-22Merge branch 'raimo/close-ports-on-halt/OTP-9985' into maintRaimo Niskanen
* raimo/close-ports-on-halt/OTP-9985: erts: Basic test of erlang:halt/0..2 erts: Document erlang:halt/2 and update erlang:halt/0,1 erts: Implement erlang:halt/2 stdlib: Stop working around erlang:halt not flushing erts: Make erlang:halt/0,1 close ports and flush async threads erts: Remove forgotten and unused function erl_exit0 erts: Implement erl_halt erts: Remove unused ntbuild.erl
2012-03-21erts: Basic test of erlang:halt/0..2Raimo Niskanen
2012-03-21erts: Implement erlang:halt/2Raimo Niskanen
2012-03-21erts: Make erlang:halt/0,1 close ports and flush async threadsRaimo Niskanen
2012-03-21erts: Remove forgotten and unused function erl_exit0Raimo Niskanen
2012-03-21erts: Implement erl_haltRaimo Niskanen
2012-03-20Merge branch 'sverk/zlib_port_leak' into maintSverker Eriksson
* sverk/zlib_port_leak: Fix port leaks in zlib OTP-9981
2012-03-20Merge branch 'ps/inet6-less-build-fix' into maintGustav Simonsson
* ps/inet6-less-build-fix: Add missing HAVE_IN6 && AF_INET6 ifdef OTP-9996
2012-03-20Merge branch 'lukas/kernel/sendfile_fixes' into maintLukas Larsson
* lukas/kernel/sendfile_fixes: Fix reselecting bug on OS X Fix bug when sending long files using select Fix ifdef to check if we are on OS X Add test case for sending multiple small files on same connection Fix memory leak when sendfile process crashes Extend timeout for windows Skip sendfile suite if solaris 8
2012-03-20Fix reselecting bug on OS XLukas Larsson
Since stop_select is called at an arbitrary point in the future it would sometime not be alled before the tcp driver started selecting on the fd. So now ERL_DRV_USE_NO_CALLBACK is used so that the stop_select call is never made. This seems to only have happened OS X.
2012-03-20Fix bug when sending long files using selectLukas Larsson
The return value from efile_sendfile was not consistent inbetween platforms. The API should now be working as it was intended. OTP-9994
2012-03-20Fix ifdef to check if we are on OS XLukas Larsson
2012-03-20Fix memory leak when sendfile process crashesLukas Larsson
We use the fact that file_flush is called when there is data in the driver queue when a port is closed to ensure that all data is cleaned up as it should. OTP-9993
2012-03-20Merge branch 'bjorn/kernel/erts_debug_size-fix/OTP-9991' into maintBjörn Gustavsson
* bjorn/kernel/erts_debug_size-fix/OTP-9991: Improve erts_debug_SUITE Fix calculation of fun size in erts_debug:size/1
2012-03-20Improve erts_debug_SUITEBjörn Gustavsson
Test erts_debug:size/1 too. Also test funs and shared terms.
2012-03-19Merge branch 'ph/fix-enfile/OTP-9990' into maintBjörn-Egil Dahlberg
* ph/fix-enfile/OTP-9990: Update preloaded prim_inet.beam gen_tcp: Use correct specs for accept and listen prim_inet: Catch system_limit in open_port Fix returned error from gen_tcp:accept/1,2 when running out of ports
2012-03-15Fix port leaks in zlibSverker Eriksson
zip,unzip,gzip,gunzip,compress and uncompress did not close the port when given invalid data argument. Fixed by putting close in "try-after". A closed port could in same cases get stuck in limbo due to data in io-queue. Fixed by adding a flush callback.
2012-03-15Merge branch 'rickard/barriers/OTP-9922' into maintRickard Green
* rickard/barriers/OTP-9922: Remove newly introduced excessive barrier in erts_thr_progress_later() Fix implementation with ddrb barriers
2012-03-14erts: Do not rebuild all emulator when preloaded changesRaimo Niskanen
2012-03-12Remove newly introduced excessive barrier in erts_thr_progress_later()Rickard Green
2012-02-21Add missing HAVE_IN6 && AF_INET6 ifdefPeer Stritzinger
This prevents building for operating systems that do not yet have IPv6 compatible socket API. Error can be reproduced by (cross)building for a os that doesn't have these headers. This will abort with a error about incomplete type 'struct in6_addr'. All other uses of IPv6 API in inet_drv.c are conditionalized like in the patch.
2012-02-21erts: Fail binary_to_term if bignum arity is too largeSverker Eriksson
2012-02-21erts: Fix bignum-bug in ETS with compressed optionSverker Eriksson
A large 64-bit immediate number will be stored as SMALL_BIG_EXT by ETS compressed format. When uncompressing, the SMALL_BIG_EXT was first decoded as as bignum (by bytes_to_big) and then turned into a small (by big_norm). This works for normal "binary_to_term" as decoded_size() over-estimates the needed heap size. But for ETS no over-estimation is done as the real term size is known and stored in DbTerm. Fixed by preventing bytes_to_big() from writing bignum digit when the number is seen to fit in an immediate.
2012-02-20Merge branch 'rickard/barriers/OTP-9922' into maintRickard Green
* rickard/barriers/OTP-9922: Reduce thread progress read operations in handle_aux_work() Misc memory barrier fixes
2012-02-19Reduce thread progress read operations in handle_aux_work()Rickard Green
2012-02-19Misc memory barrier fixesRickard Green
- Document barrier semantics - Introduce ddrb suffix on atomic ops - Barrier macros for both non-SMP and SMP case - Make the thread progress API a bit more intuitive
2012-02-16Merge branch 'jz/erts-remove-unused-var' into maintHenrik Nord
* jz/erts-remove-unused-var: erts: Remove unused variable OTP-9926
2012-02-16Merge branch 'rc/spell-registered' into maintHenrik Nord
* rc/spell-registered: Correct spelling of "registered" in various places in the source code OTP-9925
2012-02-14Merge branch 'jz/reduce-smp-locking-time-range' into maintHenrik Nord
* jz/reduce-smp-locking-time-range: erts: reduce smp locking time range in erts_garbage_collect OTP-9912
2012-02-07Merge branch 'dgud/sched-work-time/OTP-9858' into maintPatrik Nyblom
* dgud/sched-work-time/OTP-9858: emulator: Document and test scheduler_wall_time Implement statistics(scheduler_wall_time)
2012-02-07Merge branch 'bjorn/hipe-test-cases' into maintBjörn Gustavsson
* bjorn/hipe-test-cases: debugger tests: Increase time trap timeout fprof_SUITE: Skip create_file_slow/1 if libraries are native etop_SUITE: Skip a test case if libraries are native-compiled Skip all inviso test cases if libraries are native-compiled erl_prim_loader_SUITE: Handle native-compiled erl_boot_server re_SUITE: Skip error_handling/1 if 're' is native code shell_SUITE: Handle different EXIT reasons from native code sofs_SUITE: Handle different EXIT reasons from native code erl_eval_SUITE: Skip test of parameterized modules if native code trace_port_SUITE: Skip test cases if 'lists' is native-compiled call_trace_SUITE: Skip test cases if libraries are native-compiled
2012-02-06emulator: Document and test scheduler_wall_timeDan Gudmundsson
2012-01-31erts: Correct incorrect assertion in erl_alloc_util.cBjörn Gustavsson
The incorrect assertion would fire when '+Muramv true' has been given, i.e. when a reallocation always forces a move. Change the assertion to not call the lock checker if the mutex is not used (because the allocator will only be accessed from a specific scheduler and thus no lock is needed).
2012-01-26trace_port_SUITE: Skip test cases if 'lists' is native-compiledBjörn Gustavsson
2012-01-26call_trace_SUITE: Skip test cases if libraries are native-compiledBjörn Gustavsson
Skip test cases that trace lists or filename if those modules are native-compiled. While we are at it, also eliminate two compilation warnings.
2012-01-26Implement statistics(scheduler_wall_time)Rickard Green
2012-01-25Fix crash in trace_info({M,F,A}, Flags) when M:F/A has native codeBjörn Gustavsson
2012-01-25Merge branch 'raimo/parallel-make/OTP-9857' into maintRaimo Niskanen
* raimo/parallel-make/OTP-9857: hipe: Fix rtl dependencies erts: Simplify toplevel makefile due to better dependencies erts: Fix make omissions for void emulators erts: Fix dependencies between targets generate and depend erts: Cleanup dependencies for target generate erts,tools: Fix parallel make for erts/lib_src erts: Fix parallel make for emulator zlib Conflicts: erts/emulator/Makefile.in
2012-01-24Merge branch 'egil/fix-profiler-pid' into maintBjörn-Egil Dahlberg
* egil/fix-profiler-pid: erts: Do not profile profiler pid (non smp) erts: Do not profile profiler pid (smp) Revert "erts: Do not profile system profiler pid"
2012-01-23erts: Fix make omissions for void emulatorsRaimo Niskanen
2012-01-23erts: Fix dependencies between targets generate and dependRaimo Niskanen
* Make generate and depend.mk automatically. * Do not make depend.mk for targets clean and generate. * Remove old replaced CREATE_DIRS cruft. * Fiercer remove by target clean. * Move depend.mk to Target/Type/Flavor directory.
2012-01-23Merge branch 'jz/error-logic-efile_sendfile' into maintHenrik Nord
* jz/error-logic-efile_sendfile: erts: minor fix for unnecessary condition OTP-9872
2012-01-23Merge branch 'jz/sendfile_chunk_size' into maintHenrik Nord
* jz/sendfile_chunk_size: erts: change SENDFILE_CHUNK_SIZE from signed to unsigned Conflicts: erts/emulator/drivers/unix/unix_efile.c OTP-9872
2012-01-18Merge branch 'raimo/64-bit-efile_drv/OTP-9820' into maintErlang/OTP
* raimo/64-bit-efile_drv/OTP-9820: file_SUITE: Assume 64-bit windows has enough memory erts: Badarg if port output overflows iov_len prim_file_SUITE: large_file success depends on wordsize file_SUITE: Increase timeout for large_file file_SITE,prim_file_SUITE: Accept old OpenBSD pecularity prim_file_SUITE: large_write - meaner test data and check result content file_SUITE: large_write - check mem size before creating huge binary erts: rewrite efile_writev to handle partial writes correctly erts: Bugfix - driver_deq freed wrong length due to short type (int) prim_file_SUITE: Add large_write/1 file_SUITE: Add large_write/1 file_SUITE: Refactor large_file/1 file_SUITE: Fix unix_free/1
2012-01-17erts: Do not profile profiler pid (non smp)Björn-Egil Dahlberg
2012-01-16erts: Do not profile profiler pid (smp)Björn-Egil Dahlberg
2012-01-16Revert "erts: Do not profile system profiler pid"Björn-Egil Dahlberg
This reverts commit 4a8ce05083b9c88b94560f400370dbc656893b6e. * Above commit has a lock order violation
2012-01-16erts: Cleanup dependencies for target generateRaimo Niskanen
2012-01-13erts,tools: Fix parallel make for erts/lib_srcRaimo Niskanen
Use a make timestamp file to condense dependencies to some part(s) of erts/lib_src build results.