aboutsummaryrefslogtreecommitdiffstats
path: root/erts/emulator/beam/erl_nif.c
AgeCommit message (Collapse)Author
2014-11-04Fix gc-related problem with dirty NIFsSteve Vinoski
Ensure that the return value from a dirty NIF call is made part of the GC rootset. Add a new regression test to nif_SUITE. Thanks to Daniel Goertzen for reporting the error and providing a test case, and to Sverker Eriksson for making test case reproducible and finding the fix.
2014-09-12erts: Remove enif_have_dirty_schedulers()Sverker Eriksson
and add 'dirty_scheduler_support' to ErlNifSysInfo
2014-09-05Merge branch 'vinoski/enif-schedule-nif' into maintRickard Green
OTP-12128 * vinoski/enif-schedule-nif: Fix leak of NIF exports Use separate allocation type for NIF export
2014-09-05Use separate allocation type for NIF exportRickard Green
2014-09-04Merge branch 'sverk/nif-inspect-copy-bug/OTP-9828' into maintSverker Eriksson
* sverk/nif-inspect-copy-bug/OTP-9828: erts: Fix bug with enif_make_copy reallocating writable binary Conflicts: erts/emulator/test/nif_SUITE.erl
2014-09-03erts: Fix bug with enif_make_copy reallocating writable binarySverker Eriksson
that could invalidate a pointer received from an earlier call to enif_inspect_binary. Solution: Emasculate writable binary at enif_inspect_binary. There are room for optimizations here as we now do an unconditional emasculation even though enif_make_copy is not called later in the NIF.
2014-08-28add enif_schedule_nif() to NIF APISteve Vinoski
In the #erlang IRC channel Anthony Ramine once mentioned the idea of allowing a NIF to use an emulator trap, similar to a BIF trap, to schedule another NIF for execution. This is exactly how dirty NIFs were implemented for Erlang/OTP 17.0, so this commit refactors and generalizes that dirty NIF code to support a new enif_schedule_nif() API function. The enif_schedule_nif() function allows a long-running NIF to be broken into separate NIF invocations. The NIF first executes part of the long-running task, then calls enif_schedule_nif() to schedule a NIF for later execution to continue the task. Any number of NIFs can be scheduled in this manner, one after another. Since the emulator regains control between invocations, this helps avoid problems caused by native code tying up scheduler threads for too long. The enif_schedule_nif() function also replaces the original experimental dirty NIF API. The function takes a flags parameter that a caller can use to indicate the NIF should be scheduled onto either a dirty CPU scheduler thread, a dirty I/O scheduler thread, or scheduled as a regular NIF on a regular scheduler thread. With this change, the original experimental enif_schedule_dirty_nif(), enif_schedule_dirty_nif_finalizer() and enif_dirty_nif_finalizer() API functions are no longer needed and have been removed. Explicit scheduling of a dirty NIF finalization function is no longer necessary; if an application wants similar functionality, it can have a dirty NIF just invoke enif_schedule_nif() to schedule a non-dirty NIF to complete its task. Lift the restriction that dirty NIFs can't call enif_make_badarg() to raise an exception. This was a problem with the original dirty NIF API because it forced developers to get and check all incoming arguments in a regular NIF, and then schedule the dirty NIF which then had to get all the arguments again. Now, the argument checking can be done in the dirty NIF and it can call enif_make_badarg() itself to flag incorrect arguments. Extend the ErlNifFunc struct with a new flags field that allows NIFs to be declared as dirty. The default value for this field is 0, indicating a regular NIF, so it's backwards compatible with all existing statically initialized ErlNifFunc struct instances, and so such instances require no code changes. Defining the flags field with a value of ERL_NIF_DIRTY_JOB_CPU_BOUND indicates that the NIF should execute on a dirty CPU scheduler thread, or defining it with a value of ERL_NIF_DIRTY_JOB_IO_BOUND indicates that the NIF should execute on a dirty I/O scheduler thread. Any other flags field value causes a NIF library loading error. Extend the ErlNifEntry struct with a new options field that indicates whether a NIF library was built with support for optional features such as dirty NIFs. When a NIF library is loaded, the runtime checks the options field to ensure compatibility. If a NIF library built with dirty NIF support is loaded into a runtime that does not support dirty NIFs, and the library defines one or more ErlNifFunc entries with non-zero flags fields indicating dirty NIFs, a NIF library loading error results. There is no error if a NIF library built with dirty NIF support is loaded into a runtime that does not support dirty NIFs but the library does not have any dirty NIFs. It is also not an error if a library without dirty NIF support is loaded into a runtime built with dirty NIF support. Add documentation and tests for enif_schedule_nif().
2014-03-28prevent NIF purge during dirty NIF executionSteve Vinoski
Reference-count the NIF before and after invoking a NIF on dirty schedulers to prevent having the NIF purged during the call.
2014-03-28fix dirty NIF invalid memory readSteve Vinoski
Dirty NIF support used an Export structure to facilitate calls to dirty NIFs and finalizers, but Export isn't large enough to hold all necessary data. This was causing an invalid memory read in beam_emu.c past the end of the Export object. Add a local extended Export struct to erl_nif.c that can hold all the necessary data.
2014-03-21Introduce minimum allowed major driver and nif versions on loadRickard Green
2014-02-24Merge branch 'lukas/ose/master/OTP-11334'Lukas Larsson
* lukas/ose/master/OTP-11334: (71 commits) erts: Fix unix efile assert ose: Use -O2 when building ose: Expand OSE docs ose: Add dummy ttsl driver ose: Cleanup cleanup of mutex selection defines ose: Polish mmap configure checks ose: Add ose specific x-compile flags ose: Updating fd_driver and spawn_driver for OSE ose: Updating event and signal API for OSE ose: Cleanup of mutex selection defines win32: Compile erl_log.exe ose: Remove uneccesary define ose: Fix ssl configure test for osx erts: Fix sys_msg_dispatcher assert ose: Fix broken doc links ose: Thread priorities configurable from lmconf ose: Yielding the cpu is done "the OSE" way ose: Start using ppdata for tse key ose: Do not use spinlocks on OSE ose: Fix support for crypto ... Conflicts: lib/crypto/c_src/crypto.c
2014-02-24Added support for ENEA OSELukas Larsson
This port has support for both non-smp and smp. It contains a new way to do io checking in which erts_poll_wait receives the payload of the polled entity. This has implications for all linked-in drivers.
2014-02-24Merge branch 'vinoski/ds2'Rickard Green
* vinoski/ds2: further enhancements for dirty schedulers allow optional whitespace in dirty scheduler erl options
2014-02-24further enhancements for dirty schedulersSteve Vinoski
Add support for setting the number of dirty CPU schedulers online via erlang:system_flag/2. Assuming the emulator is built with dirty schedulers enabled, the number of dirty CPU schedulers online may not be less than 1, nor greater than the number of dirty CPU schedulers available, nor greater than the number of normal schedulers online. Dirty CPU scheduler threads that are taken offline via system_flag/2 are suspended. The number of dirty CPU schedulers online may be adjusted independently of the number of normal schedulers online, but if system_flag/2 is used to set the number of normal schedulers online to a value less than the current number of normal schedulers online, the number of dirty CPU schedulers online is decreased proportionally. Likewise, if the number of normal schedulers online is increased, the number of dirty CPU schedulers online is increased proportionally. For example, if 8 normal schedulers and 4 dirty CPU schedulers are online, and system_flag/2 is called to set the number of normal schedulers online to 4, the number of dirty CPU schedulers online is also decreased by half, to 2. Subsequently setting the number of normal schedulers online back to 8 also sets the number of dirty CPU schedulers online back to 4. Augment the system_flag/2 documentation in the erlang man page to explain this relationship between schedulers_online and dirty_cpu_schedulers_online. Also ensure that all dirty CPU and I/O schedulers are suspended when multi-scheduling is blocked via system_flag/2, and brought back online when multi-scheduling is unblocked. Add Rickard Green's rewritten check_enqueue_in_prio_queue() function that inspects process state more thoroughly to determine if to enqueue it and if so on what queue, including dirty queues when appropriate. Make sure dirty NIF jobs do not trigger erlang:system_monitor long_schedule messages. Add more dirty scheduler testing to the scheduler test suite. Remove the erts_no_dirty_cpu_schedulers_online global variable, since it's no longer needed. Execute dirty NIFs on a normal scheduler thread while multi-scheduling blocking is in effect. Evacuate any dirty jobs residing in the dirty run queues over to a normal run queue when multi-scheduling is blocked. Allow dirty schedulers to execute aux work. Set the dirty run queues halt_in_progress flag when halting the normal schedulers. Change dirty scheduler numbers to a structure including both scheduler number and type, either dirty CPU or dirty I/O. Add some assertions to ensure that dirty CPU schedulers operate only on dirty CPU scheduler process flags, and the same for dirty I/O schedulers.
2014-02-19Merge branch 'sverk/crypto/hmac-context-reuse-bug'Sverker Eriksson
OTP-11722 OTP-11724 * sverk/crypto/hmac-context-reuse-bug: crypto: Fix bug when using old hmac context erts: Fix NIF bug when load/upgrade fails after enif_open_resource_type Conflicts: erts/emulator/test/nif_SUITE.erl
2014-02-05erts: Fix NIF bug when load/upgrade fails after enif_open_resource_typeSverker Eriksson
..has been successfully called. Opened resource types (created or taken-over) were left "hanging" leading both to memory leakage and other more strange and serious behavior. Now a proper rollback is done.
2014-01-29erts: Change 'size' argument of enif_get_map_size from int* to size_t*Sverker Eriksson
2014-01-29erts: Fix map iterator bug when reverting from end of map positionSverker Eriksson
and simplify code by ignoring h_limit which is always zero.
2014-01-29erts: Simplify some map iterator codeSverker Eriksson
2014-01-29erts: Remove use of h_limit which is always zero.Sverker Eriksson
2014-01-29erts: Let enif_map_iterator_next/prev return 0 to signal end of map.Sverker Eriksson
2014-01-29erts: Do not allow map iterator created without mapSverker Eriksson
2014-01-29erts: Increase version for NIF API and reject experimental v2.5Sverker Eriksson
2014-01-29erts: Remove enif_find_map_valueSverker Eriksson
as it does the same thing as enif_get_map_value. Replace with placeholder to be ABI backward compatible on Windows as long as enif_find_map_value is not called.
2014-01-29erts: Add NIFs for MapsBjörn-Egil Dahlberg
- int enif_is_map(ErlNifEnv* env, ERL_NIF_TERM map) - int enif_get_map_size(ErlNifEnv *env, ERL_NIF_TERM, int*) - ERL_NIF_TERM enif_make_new_map(ErlNifEnv *env) - int enif_make_map_put(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM value, ERL_NIF_TERM* map_out) - int enif_get_map_value(ErlNifEnv *env, ERL_NIF_TERM map, ERL_NIF_TERM key, ERL_NIF_TERM* value) - int enif_find_map_value(ErlNifEnv *env, ERL_NIF_TERM map, ERL_NIF_TERM key, ERL_NIF_TERM* value) - int enif_make_map_update(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM value, ERL_NIF_TERM* map_out) - int enif_make_map_remove(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM* map_out) - int enif_map_iterator_create(ErlNifEnv *env, ERL_NIF_TERM map, ErlNifMapIterator *iter) - void enif_map_iterator_destroy(ErlNifEnv *env, ErlNifMapIterator *iter) - int enif_map_iterator_next(ErlNifEnv *env, ErlNifMapIterator *iter) - int enif_map_iterator_get_pair(ErlNifEnv *env, ErlNifMapIterator *iter, ERL_NIF_TERM *key, ERL_NIF_TERM *value)
2014-01-28initial support for dirty schedulers and dirty NIFsSteve Vinoski
Add initial support for dirty schedulers. There are two types of dirty schedulers: CPU schedulers and I/O schedulers. By default, there are as many dirty CPU schedulers as there are normal schedulers and as many dirty CPU schedulers online as normal schedulers online. There are 10 dirty I/O schedulers (similar to the choice of 10 as the default for async threads). By default, dirty schedulers are disabled and conditionally compiled out. To enable them, you must pass --enable-dirty-schedulers to the top-level configure script when building Erlang/OTP. Current dirty scheduler support requires the emulator to be built with SMP support. This restriction will be lifted in the future. You can specify the number of dirty schedulers with the command-line options +SDcpu (for dirty CPU schedulers) and +SDio (for dirty I/O schedulers). The +SDcpu option is similar to the +S option in that it takes two numbers separated by a colon: C1:C2, where C1 specifies the number of dirty schedulers available and C2 specifies the number of dirty schedulers online. The +SDPcpu option allows numbers of dirty CPU schedulers available and dirty CPU schedulers online to be specified as percentages, similar to the existing +SP option for normal schedulers. The number of dirty CPU schedulers created and dirty CPU schedulers online may not exceed the number of normal schedulers created and normal schedulers online, respectively. The +SDio option takes only a single number specifying the number of dirty I/O schedulers available and online. There is no support yet for programmatically changing at run time the number of dirty CPU schedulers online via erlang:system_flag/2. Also, changing the number of normal schedulers online via erlang:system_flag(schedulers_online, NewSchedulersOnline) should ensure that there are no more dirty CPU schedulers than normal schedulers, but this is not yet implemented. You can retrieve the number of dirty schedulers by passing dirty_cpu_schedulers, dirty_cpu_schedulers_online, or dirty_io_schedulers to erlang:system_info/1. Currently only NIFs are able to access dirty scheduler functionality. Neither drivers nor BIFs currently support dirty schedulers. This restriction will be addressed in the future. If dirty scheduler support is present in the runtime, the initial status line Erlang prints before presenting its interactive prompt will include the indicator "[ds:C1:C2:I]" where "ds" indicates "dirty schedulers", "C1" indicates the number of dirty CPU schedulers available, "C2" indicates the number of dirty CPU schedulers online, and "I" indicates the number of dirty I/O schedulers. Document The dirty NIF API in the erl_nif man page. The API closely follows Rickard Green's presentation slides from his talk "Future Extensions to the Native Interface", presented at the 2011 Erlang Factory held in the San Francisco Bay Area. Rickard's slides are available online at http://bit.ly/1m34UHB . Document the new erl command-line options, the additions to erlang:system_info/1, and also add the erlang:system_flag/2 dirty scheduler documentation even though it's not yet implemented. To determine whether the dirty NIF API is available, native code can check to see whether the C preprocessor macro ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined. To check if dirty schedulers are available at run time, native code can call the boolean enif_have_dirty_schedulers() function, and Erlang code can call erlang:system_info(dirty_cpu_schedulers), which raises badarg if no dirty scheduler support is available. Add a simple dirty NIF test to the emulator NIF suite.
2013-12-16erts: Add 'extra' argument to erts_convert_filename_to_encodingSverker Eriksson
2013-12-13erts: Refactor remove erts_sys_dll_open2Sverker Eriksson
2013-09-19erts, crypto: Support NIF library with unicode filename on windowsSverker Eriksson
2013-09-19erts: Fix compiler warningSverker Eriksson
2013-09-13erts: Fix loading of NIF library with unicode in pathSverker Eriksson
2013-08-21erts: Add option to include nifs staticallyLukas Larsson
Both crypto and asn1 are supported.
2013-07-11Fix variable ‘rp_had_locks’ set but not used warningRickard Green
2013-02-13erts: Add enif_consume_timesliceSverker Eriksson
2013-01-28Merge branch 'sverk/enc_atom-opt'Sverker Eriksson
* sverk/enc_atom-opt: erts: Optimize atom encoding to use memcpy for pure ascii erts: Refactor erts_atom_get to use ErtsAtomEncoding
2013-01-25Update copyright yearsBjörn-Egil Dahlberg
2013-01-25erts: Refactor erts_atom_get to use ErtsAtomEncodingSverker Eriksson
instead of 'is_latin1' boolean argument.
2013-01-23Merge branch 'sverk/r16/utf8-atoms'Sverker Eriksson
* sverk/r16/utf8-atoms: erl_interface: Fix bug when transcoding atoms from and to UTF8 erl_interface: Changed erlang_char_encoding interface erts: Testcase doing unicode atom printout with ~w erl_interface: even more utf8 atom stuff erts: Fix bug in analyze_utf8 causing faulty latin1 detection Add UTF-8 node name support for epmd workaround... Fix merge conflict with hasse UTF-8 atom documentation test case erl_interface: utf8 atoms continued Add utf8 atom distribution test cases atom fixes for NIFs and atom_to_binary UTF-8 support for distribution Implement UTF-8 atom support for jinterface erl_interface: Enable decode of unicode atoms stdlib: Fix printing of unicode atoms erts: Change internal representation of atoms to utf8 erts: Refactor rename DFLAG(S)_INTERNAL_TAGS for conformity Conflicts: erts/emulator/beam/io.c OTP-10753
2013-01-16atom fixes for NIFs and atom_to_binarySverker Eriksson
2013-01-08erts: Change internal representation of atoms to utf8Sverker Eriksson
2012-12-07Merge branch 'rickard/port-optimizations/OTP-10336' into ↵Rickard Green
rickard/r16/port-optimizations/OTP-10336 * rickard/port-optimizations/OTP-10336: Change annotate level for emacs-22 in cerl Update etp-commands Add documentation on communication in Erlang Add support for busy port message queue Add driver callback epilogue Implement true asynchronous signaling between processes and ports Add erl_drv_[send|output]_term Move busy port flag Use rwlock for driver list Optimize management of port tasks Improve configuration of process and port tables Remove R9 compatibility features Use ptab functionality also for ports Prepare for use of ptab functionality also for ports Atomic port state Generalize process table implementation Implement functionality for delaying thread progress from unmanaged threads Conflicts: erts/doc/src/erl_driver.xml erts/doc/src/erlang.xml erts/emulator/beam/beam_bif_load.c erts/emulator/beam/beam_bp.c erts/emulator/beam/beam_emu.c erts/emulator/beam/bif.c erts/emulator/beam/copy.c erts/emulator/beam/erl_alloc.c erts/emulator/beam/erl_alloc.types erts/emulator/beam/erl_bif_info.c erts/emulator/beam/erl_bif_port.c erts/emulator/beam/erl_bif_trace.c erts/emulator/beam/erl_init.c erts/emulator/beam/erl_message.c erts/emulator/beam/erl_port_task.c erts/emulator/beam/erl_process.c erts/emulator/beam/erl_process.h erts/emulator/beam/erl_process_lock.c erts/emulator/beam/erl_trace.c erts/emulator/beam/export.h erts/emulator/beam/global.h erts/emulator/beam/io.c erts/emulator/sys/unix/sys.c erts/emulator/sys/vxworks/sys.c erts/emulator/test/port_SUITE.erl erts/etc/unix/cerl.src erts/preloaded/ebin/erlang.beam erts/preloaded/ebin/prim_inet.beam erts/preloaded/src/prim_inet.erl lib/hipe/cerl/erl_bif_types.erl lib/kernel/doc/src/inet.xml lib/kernel/src/inet.erl
2012-12-07Implement true asynchronous signaling between processes and portsRickard Green
2012-12-03Prepare for use of ptab functionality also for portsRickard Green
2012-10-09crypto: Make unloading of crypto saferSverker Eriksson
Facts: crypto nif-lib registers callback functions that openssl uses for memory management and thread synchronization. The callback functions can only be set once, openssl does not allow changing the callback functions. Problem: If openssl is dynamicly linked to crypto, you might get s scenario where the crypto lib is unloaded while leaving openssl loaded with its old pointers to the unloaded crypto code intact. If crypto is then reloaded (by init:restart() for example), the crypto nif-lib might get relocated at a different address. crypto calls openssl which in turn calls the old invalid callback functions...kaboom. Solution: Break apart the callback functions into a separate dynamic lib that crypto loads with dlopen. When crypto is unloaded the callback lib is left in place to be reused if/when crypto is loaded again.
2012-08-20Merge branch 'maint'Patrik Nyblom
Conflicts: erts/doc/src/erlang.xml erts/preloaded/ebin/init.beam lib/kernel/doc/src/os.xml lib/stdlib/test/filename_SUITE.erl
2012-08-17Relocate bodies of DTrace probes to the statically-linked VM.Scott Lystig Fritchie
Due to various operating systems (in both the DTrace and SystemTap worlds) not fully supporting DTrace probes (or SystemTap-compatibility mode probes) in shared libraries, we relocate those probes to the statically-linked virtual machine. This could be seen as pollution of the pristine VM by a (yet) experimental feature. However: 1. This code can be eliminated completely by the C preprocessor. 2. Leaving the probes in the dyntrace NIF shared library simply does not work correctly on too many platforms. *Many* thanks to Macneil Shonle at Basho for assisting when my RSI-injured fingers gave out. Tested on: * CentOS 5, SystemTap 1.3 * Solaris 10 (note) * Solaris 11 * OpenIndiana 151 * SmartOS 20120809T221258Z * FreeBSD 9.0-RELEASE (note) I had hoped to be able to test CentOS 6 + SystemTap 1.7, but the details of dealing with all dependencies for a 2.6.32-279.5.1.el6.x86_64 kernel are too time consuming right now. (note: Solaris 10 and FreeBSD 9.0-RELEASE can take a long time to compile)
2012-06-25Change the data structures for breakpointsBjörn Gustavsson
Change the data structures for breakpoints to make it possible (in a future commit) to manage breakpoints without taking down the system to single-scheduling mode. The current "breakpoint wheel" data structure (a circular, double-linked list of breakpoints) was invented before the SMP emulator. To support it in the SMP emulator, there is essentially one breakpoint wheel per scheduler. As more breakpoint types have been added, the implementation has become messy and hard to understand and maintain. Therefore, the time for a rewrite has come. Use one struct to hold all breakpoint data for a breakpoint in a function. Use a flag field to indicate what different type of break actions that are enabled.
2012-04-27Merge branch 'rickard/proc-sched/OTP-9892'Rickard Green
* rickard/proc-sched/OTP-9892: Teach etp-commands to understand new emulator internal data structures Optimize process state changes Optimize process table access Implement possibility to use ordinary mutexes as process locks Conflicts: erts/emulator/beam/erl_alloc.types
2012-04-16Optimize process table accessRickard Green
2012-03-30Merge branch 'maint'Björn-Egil Dahlberg