| Age | Commit message (Collapse) | Author |
|---|
|
Ensure that the return value from a dirty NIF call is made part of the GC
rootset. Add a new regression test to nif_SUITE.
Thanks to Daniel Goertzen for reporting the error and providing a test
case, and to Sverker Eriksson for making test case reproducible and finding
the fix.
|
|
and add 'dirty_scheduler_support' to ErlNifSysInfo
|
|
OTP-12128
* vinoski/enif-schedule-nif:
Fix leak of NIF exports
Use separate allocation type for NIF export
|
|
|
|
* sverk/nif-inspect-copy-bug/OTP-9828:
erts: Fix bug with enif_make_copy reallocating writable binary
Conflicts:
erts/emulator/test/nif_SUITE.erl
|
|
that could invalidate a pointer received from an earlier
call to enif_inspect_binary.
Solution: Emasculate writable binary at enif_inspect_binary. There are
room for optimizations here as we now do an unconditional emasculation
even though enif_make_copy is not called later in the NIF.
|
|
In the #erlang IRC channel Anthony Ramine once mentioned the idea of
allowing a NIF to use an emulator trap, similar to a BIF trap, to schedule
another NIF for execution. This is exactly how dirty NIFs were implemented
for Erlang/OTP 17.0, so this commit refactors and generalizes that dirty
NIF code to support a new enif_schedule_nif() API function.
The enif_schedule_nif() function allows a long-running NIF to be broken
into separate NIF invocations. The NIF first executes part of the
long-running task, then calls enif_schedule_nif() to schedule a NIF for
later execution to continue the task. Any number of NIFs can be scheduled
in this manner, one after another. Since the emulator regains control
between invocations, this helps avoid problems caused by native code tying
up scheduler threads for too long.
The enif_schedule_nif() function also replaces the original experimental
dirty NIF API. The function takes a flags parameter that a caller can use
to indicate the NIF should be scheduled onto either a dirty CPU scheduler
thread, a dirty I/O scheduler thread, or scheduled as a regular NIF on a
regular scheduler thread. With this change, the original experimental
enif_schedule_dirty_nif(), enif_schedule_dirty_nif_finalizer() and
enif_dirty_nif_finalizer() API functions are no longer needed and have been
removed. Explicit scheduling of a dirty NIF finalization function is no
longer necessary; if an application wants similar functionality, it can
have a dirty NIF just invoke enif_schedule_nif() to schedule a non-dirty
NIF to complete its task.
Lift the restriction that dirty NIFs can't call enif_make_badarg() to raise
an exception. This was a problem with the original dirty NIF API because it
forced developers to get and check all incoming arguments in a regular NIF,
and then schedule the dirty NIF which then had to get all the arguments
again. Now, the argument checking can be done in the dirty NIF and it can
call enif_make_badarg() itself to flag incorrect arguments.
Extend the ErlNifFunc struct with a new flags field that allows NIFs to be
declared as dirty. The default value for this field is 0, indicating a
regular NIF, so it's backwards compatible with all existing statically
initialized ErlNifFunc struct instances, and so such instances require no
code changes. Defining the flags field with a value of
ERL_NIF_DIRTY_JOB_CPU_BOUND indicates that the NIF should execute on a
dirty CPU scheduler thread, or defining it with a value of
ERL_NIF_DIRTY_JOB_IO_BOUND indicates that the NIF should execute on a dirty
I/O scheduler thread. Any other flags field value causes a NIF library
loading error.
Extend the ErlNifEntry struct with a new options field that indicates
whether a NIF library was built with support for optional features such as
dirty NIFs. When a NIF library is loaded, the runtime checks the options
field to ensure compatibility. If a NIF library built with dirty NIF
support is loaded into a runtime that does not support dirty NIFs, and the
library defines one or more ErlNifFunc entries with non-zero flags fields
indicating dirty NIFs, a NIF library loading error results. There is no
error if a NIF library built with dirty NIF support is loaded into a
runtime that does not support dirty NIFs but the library does not have any
dirty NIFs. It is also not an error if a library without dirty NIF support
is loaded into a runtime built with dirty NIF support.
Add documentation and tests for enif_schedule_nif().
|
|
Reference-count the NIF before and after invoking a NIF on dirty schedulers
to prevent having the NIF purged during the call.
|
|
Dirty NIF support used an Export structure to facilitate calls to dirty
NIFs and finalizers, but Export isn't large enough to hold all necessary
data. This was causing an invalid memory read in beam_emu.c past the end of
the Export object. Add a local extended Export struct to erl_nif.c that can
hold all the necessary data.
|
|
|
|
* lukas/ose/master/OTP-11334: (71 commits)
erts: Fix unix efile assert
ose: Use -O2 when building
ose: Expand OSE docs
ose: Add dummy ttsl driver
ose: Cleanup cleanup of mutex selection defines
ose: Polish mmap configure checks
ose: Add ose specific x-compile flags
ose: Updating fd_driver and spawn_driver for OSE
ose: Updating event and signal API for OSE
ose: Cleanup of mutex selection defines
win32: Compile erl_log.exe
ose: Remove uneccesary define
ose: Fix ssl configure test for osx
erts: Fix sys_msg_dispatcher assert
ose: Fix broken doc links
ose: Thread priorities configurable from lmconf
ose: Yielding the cpu is done "the OSE" way
ose: Start using ppdata for tse key
ose: Do not use spinlocks on OSE
ose: Fix support for crypto
...
Conflicts:
lib/crypto/c_src/crypto.c
|
|
This port has support for both non-smp and smp.
It contains a new way to do io checking in which erts_poll_wait
receives the payload of the polled entity. This has implications
for all linked-in drivers.
|
|
* vinoski/ds2:
further enhancements for dirty schedulers
allow optional whitespace in dirty scheduler erl options
|
|
Add support for setting the number of dirty CPU schedulers online via
erlang:system_flag/2. Assuming the emulator is built with dirty schedulers
enabled, the number of dirty CPU schedulers online may not be less than 1,
nor greater than the number of dirty CPU schedulers available, nor greater
than the number of normal schedulers online. Dirty CPU scheduler threads
that are taken offline via system_flag/2 are suspended. The number of dirty
CPU schedulers online may be adjusted independently of the number of normal
schedulers online, but if system_flag/2 is used to set the number of normal
schedulers online to a value less than the current number of normal
schedulers online, the number of dirty CPU schedulers online is decreased
proportionally. Likewise, if the number of normal schedulers online is
increased, the number of dirty CPU schedulers online is increased
proportionally. For example, if 8 normal schedulers and 4 dirty CPU
schedulers are online, and system_flag/2 is called to set the number of
normal schedulers online to 4, the number of dirty CPU schedulers online is
also decreased by half, to 2. Subsequently setting the number of normal
schedulers online back to 8 also sets the number of dirty CPU schedulers
online back to 4. Augment the system_flag/2 documentation in the erlang man
page to explain this relationship between schedulers_online and
dirty_cpu_schedulers_online.
Also ensure that all dirty CPU and I/O schedulers are suspended when
multi-scheduling is blocked via system_flag/2, and brought back online when
multi-scheduling is unblocked.
Add Rickard Green's rewritten check_enqueue_in_prio_queue() function that
inspects process state more thoroughly to determine if to enqueue it and if
so on what queue, including dirty queues when appropriate.
Make sure dirty NIF jobs do not trigger erlang:system_monitor long_schedule
messages.
Add more dirty scheduler testing to the scheduler test suite.
Remove the erts_no_dirty_cpu_schedulers_online global variable, since it's
no longer needed.
Execute dirty NIFs on a normal scheduler thread while multi-scheduling
blocking is in effect. Evacuate any dirty jobs residing in the dirty run
queues over to a normal run queue when multi-scheduling is blocked.
Allow dirty schedulers to execute aux work.
Set the dirty run queues halt_in_progress flag when halting the normal
schedulers.
Change dirty scheduler numbers to a structure including both scheduler
number and type, either dirty CPU or dirty I/O. Add some assertions to
ensure that dirty CPU schedulers operate only on dirty CPU scheduler
process flags, and the same for dirty I/O schedulers.
|
|
OTP-11722
OTP-11724
* sverk/crypto/hmac-context-reuse-bug:
crypto: Fix bug when using old hmac context
erts: Fix NIF bug when load/upgrade fails after enif_open_resource_type
Conflicts:
erts/emulator/test/nif_SUITE.erl
|
|
..has been successfully called.
Opened resource types (created or taken-over) were left "hanging"
leading both to memory leakage and other more strange and serious behavior.
Now a proper rollback is done.
|
|
|
|
and simplify code by ignoring h_limit which is always zero.
|
|
|
|
|
|
|
|
|
|
|
|
as it does the same thing as enif_get_map_value.
Replace with placeholder to be ABI backward compatible on Windows
as long as enif_find_map_value is not called.
|
|
- int enif_is_map(ErlNifEnv* env, ERL_NIF_TERM map)
- int enif_get_map_size(ErlNifEnv *env, ERL_NIF_TERM, int*)
- ERL_NIF_TERM enif_make_new_map(ErlNifEnv *env)
- int enif_make_map_put(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM value, ERL_NIF_TERM* map_out)
- int enif_get_map_value(ErlNifEnv *env, ERL_NIF_TERM map, ERL_NIF_TERM key, ERL_NIF_TERM* value)
- int enif_find_map_value(ErlNifEnv *env, ERL_NIF_TERM map, ERL_NIF_TERM key, ERL_NIF_TERM* value)
- int enif_make_map_update(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM value, ERL_NIF_TERM* map_out)
- int enif_make_map_remove(ErlNifEnv *env, ERL_NIF_TERM map_in, ERL_NIF_TERM key, ERL_NIF_TERM* map_out)
- int enif_map_iterator_create(ErlNifEnv *env, ERL_NIF_TERM map, ErlNifMapIterator *iter)
- void enif_map_iterator_destroy(ErlNifEnv *env, ErlNifMapIterator *iter)
- int enif_map_iterator_next(ErlNifEnv *env, ErlNifMapIterator *iter)
- int enif_map_iterator_get_pair(ErlNifEnv *env, ErlNifMapIterator *iter, ERL_NIF_TERM *key, ERL_NIF_TERM *value)
|
|
Add initial support for dirty schedulers.
There are two types of dirty schedulers: CPU schedulers and I/O
schedulers. By default, there are as many dirty CPU schedulers as there are
normal schedulers and as many dirty CPU schedulers online as normal
schedulers online. There are 10 dirty I/O schedulers (similar to the choice
of 10 as the default for async threads).
By default, dirty schedulers are disabled and conditionally compiled
out. To enable them, you must pass --enable-dirty-schedulers to the
top-level configure script when building Erlang/OTP.
Current dirty scheduler support requires the emulator to be built with SMP
support. This restriction will be lifted in the future.
You can specify the number of dirty schedulers with the command-line
options +SDcpu (for dirty CPU schedulers) and +SDio (for dirty I/O
schedulers). The +SDcpu option is similar to the +S option in that it takes
two numbers separated by a colon: C1:C2, where C1 specifies the number of
dirty schedulers available and C2 specifies the number of dirty schedulers
online. The +SDPcpu option allows numbers of dirty CPU schedulers available
and dirty CPU schedulers online to be specified as percentages, similar to
the existing +SP option for normal schedulers. The number of dirty CPU
schedulers created and dirty CPU schedulers online may not exceed the
number of normal schedulers created and normal schedulers online,
respectively. The +SDio option takes only a single number specifying the
number of dirty I/O schedulers available and online. There is no support
yet for programmatically changing at run time the number of dirty CPU
schedulers online via erlang:system_flag/2. Also, changing the number of
normal schedulers online via erlang:system_flag(schedulers_online,
NewSchedulersOnline) should ensure that there are no more dirty CPU
schedulers than normal schedulers, but this is not yet implemented. You can
retrieve the number of dirty schedulers by passing dirty_cpu_schedulers,
dirty_cpu_schedulers_online, or dirty_io_schedulers to
erlang:system_info/1.
Currently only NIFs are able to access dirty scheduler
functionality. Neither drivers nor BIFs currently support dirty
schedulers. This restriction will be addressed in the future.
If dirty scheduler support is present in the runtime, the initial status
line Erlang prints before presenting its interactive prompt will include
the indicator "[ds:C1:C2:I]" where "ds" indicates "dirty schedulers", "C1"
indicates the number of dirty CPU schedulers available, "C2" indicates the
number of dirty CPU schedulers online, and "I" indicates the number of
dirty I/O schedulers.
Document The dirty NIF API in the erl_nif man page. The API closely follows
Rickard Green's presentation slides from his talk "Future Extensions to the
Native Interface", presented at the 2011 Erlang Factory held in the San
Francisco Bay Area. Rickard's slides are available online at
http://bit.ly/1m34UHB .
Document the new erl command-line options, the additions to
erlang:system_info/1, and also add the erlang:system_flag/2 dirty scheduler
documentation even though it's not yet implemented.
To determine whether the dirty NIF API is available, native code can check
to see whether the C preprocessor macro ERL_NIF_DIRTY_SCHEDULER_SUPPORT is
defined. To check if dirty schedulers are available at run time, native
code can call the boolean enif_have_dirty_schedulers() function, and Erlang
code can call erlang:system_info(dirty_cpu_schedulers), which raises
badarg if no dirty scheduler support is available.
Add a simple dirty NIF test to the emulator NIF suite.
|
|
|
|
|
|
|
|
|
|
|
|
Both crypto and asn1 are supported.
|
|
|
|
|
|
* sverk/enc_atom-opt:
erts: Optimize atom encoding to use memcpy for pure ascii
erts: Refactor erts_atom_get to use ErtsAtomEncoding
|
|
|
|
instead of 'is_latin1' boolean argument.
|
|
* sverk/r16/utf8-atoms:
erl_interface: Fix bug when transcoding atoms from and to UTF8
erl_interface: Changed erlang_char_encoding interface
erts: Testcase doing unicode atom printout with ~w
erl_interface: even more utf8 atom stuff
erts: Fix bug in analyze_utf8 causing faulty latin1 detection
Add UTF-8 node name support for epmd
workaround...
Fix merge conflict with hasse
UTF-8 atom documentation
test case
erl_interface: utf8 atoms continued
Add utf8 atom distribution test cases
atom fixes for NIFs and atom_to_binary
UTF-8 support for distribution
Implement UTF-8 atom support for jinterface
erl_interface: Enable decode of unicode atoms
stdlib: Fix printing of unicode atoms
erts: Change internal representation of atoms to utf8
erts: Refactor rename DFLAG(S)_INTERNAL_TAGS for conformity
Conflicts:
erts/emulator/beam/io.c
OTP-10753
|
|
|
|
|
|
rickard/r16/port-optimizations/OTP-10336
* rickard/port-optimizations/OTP-10336:
Change annotate level for emacs-22 in cerl
Update etp-commands
Add documentation on communication in Erlang
Add support for busy port message queue
Add driver callback epilogue
Implement true asynchronous signaling between processes and ports
Add erl_drv_[send|output]_term
Move busy port flag
Use rwlock for driver list
Optimize management of port tasks
Improve configuration of process and port tables
Remove R9 compatibility features
Use ptab functionality also for ports
Prepare for use of ptab functionality also for ports
Atomic port state
Generalize process table implementation
Implement functionality for delaying thread progress from unmanaged threads
Conflicts:
erts/doc/src/erl_driver.xml
erts/doc/src/erlang.xml
erts/emulator/beam/beam_bif_load.c
erts/emulator/beam/beam_bp.c
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.c
erts/emulator/beam/copy.c
erts/emulator/beam/erl_alloc.c
erts/emulator/beam/erl_alloc.types
erts/emulator/beam/erl_bif_info.c
erts/emulator/beam/erl_bif_port.c
erts/emulator/beam/erl_bif_trace.c
erts/emulator/beam/erl_init.c
erts/emulator/beam/erl_message.c
erts/emulator/beam/erl_port_task.c
erts/emulator/beam/erl_process.c
erts/emulator/beam/erl_process.h
erts/emulator/beam/erl_process_lock.c
erts/emulator/beam/erl_trace.c
erts/emulator/beam/export.h
erts/emulator/beam/global.h
erts/emulator/beam/io.c
erts/emulator/sys/unix/sys.c
erts/emulator/sys/vxworks/sys.c
erts/emulator/test/port_SUITE.erl
erts/etc/unix/cerl.src
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/src/prim_inet.erl
lib/hipe/cerl/erl_bif_types.erl
lib/kernel/doc/src/inet.xml
lib/kernel/src/inet.erl
|
|
|
|
|
|
Facts:
crypto nif-lib registers callback functions that openssl uses
for memory management and thread synchronization. The callback
functions can only be set once, openssl does not allow changing the
callback functions.
Problem:
If openssl is dynamicly linked to crypto, you might get s scenario
where the crypto lib is unloaded while leaving openssl loaded
with its old pointers to the unloaded crypto code intact.
If crypto is then reloaded (by init:restart() for example), the crypto
nif-lib might get relocated at a different address. crypto calls
openssl which in turn calls the old invalid callback functions...kaboom.
Solution:
Break apart the callback functions into a separate dynamic lib that
crypto loads with dlopen. When crypto is unloaded the callback lib is
left in place to be reused if/when crypto is loaded again.
|
|
Conflicts:
erts/doc/src/erlang.xml
erts/preloaded/ebin/init.beam
lib/kernel/doc/src/os.xml
lib/stdlib/test/filename_SUITE.erl
|
|
Due to various operating systems (in both the DTrace and SystemTap
worlds) not fully supporting DTrace probes (or SystemTap-compatibility
mode probes) in shared libraries, we relocate those probes to the
statically-linked virtual machine. This could be seen as pollution
of the pristine VM by a (yet) experimental feature. However:
1. This code can be eliminated completely by the C preprocessor.
2. Leaving the probes in the dyntrace NIF shared library simply
does not work correctly on too many platforms.
*Many* thanks to Macneil Shonle at Basho for assisting when my
RSI-injured fingers gave out.
Tested on:
* CentOS 5, SystemTap 1.3
* Solaris 10 (note)
* Solaris 11
* OpenIndiana 151
* SmartOS 20120809T221258Z
* FreeBSD 9.0-RELEASE (note)
I had hoped to be able to test CentOS 6 + SystemTap 1.7, but the
details of dealing with all dependencies for a 2.6.32-279.5.1.el6.x86_64
kernel are too time consuming right now.
(note: Solaris 10 and FreeBSD 9.0-RELEASE can take a long time to compile)
|
|
Change the data structures for breakpoints to make it possible
(in a future commit) to manage breakpoints without taking down the
system to single-scheduling mode.
The current "breakpoint wheel" data structure (a circular,
double-linked list of breakpoints) was invented before the
SMP emulator. To support it in the SMP emulator, there is essentially
one breakpoint wheel per scheduler. As more breakpoint types have
been added, the implementation has become messy and hard to understand
and maintain.
Therefore, the time for a rewrite has come. Use one struct to hold
all breakpoint data for a breakpoint in a function. Use a flag field
to indicate what different type of break actions that are enabled.
|
|
* rickard/proc-sched/OTP-9892:
Teach etp-commands to understand new emulator internal data structures
Optimize process state changes
Optimize process table access
Implement possibility to use ordinary mutexes as process locks
Conflicts:
erts/emulator/beam/erl_alloc.types
|
|
|
|
|
