| Age | Commit message (Collapse) | Author |
|---|
|
Renamed the current preloaded net module to prim_net
and removed the deprecated functions (call, cast, ...).
Introduce a "new" net module (in kernel) as an interface
module to the (preloaded) prim_net. This one also contains
the deprecated functions (call, cast, ...).
OTP-15765
|
|
The config options --[en|dis]able-esock now works as
expected.
|
|
Preliminary work to make socket configurable (enable and disable).
OTP-15658
|
|
Previously, all ETS tables used centralized counter variables to keep
track of the number of items stored and the amount of memory
consumed. These counters can cause scalability problems (especially on
big NUMA systems). This commit adds an implementation of a
decentralized counter and modifies the implementation of ETS so that
ETS tables of type ordered_set with write_concurrency enabled use the
decentralized counter. [Experiments][1] indicate that this change
substantially improves the scalability of ETS ordered_set tables with
write_concurrency enabled in scenarios with frequent `ets:insert/2`
and `ets:delete/2` calls.
The new counter is implemented in the module erts_flxctr
(`erts_flxctr.h` and `erts_flxctr.c`). The module has the suffix
flxctr as it contains the implementation of a flexible counter (i.e.,
counter instances can be configured to be either centralized or
decentralized). Counters that are configured to be centralized are
implemented with a single counter variable which is modified with
atomic operations. Decentralized counters are spread over several
cache lines (how many can be configured with the parameter
`+dcg`). The scheduler threads are mapped to cache lines so that there
is no single point of contention when decentralized counters are
updated. The thread progress functionality of the Erlang VM is
utilized to implement support for linearizable snapshots of
decentralized counters. The snapshot functionality is used by the
`ets:info/1` and `ets:info/2` functions.
[1]: http://winsh.me/ets_catree_benchmark/flxctr_res.html
|
|
* rickard/make-fixes-21/OTP-15657:
Remove own configured RM make variable
|
|
* rickard/make-fixes-20/OTP-15657:
Remove own configured RM make variable
|
|
* rickard/make-fixes-19/OTP-15657:
Remove own configured RM make variable
|
|
* rickard/make-fixes-18/OTP-15657:
Remove own configured RM make variable
|
|
* rickard/make-fixes-17/OTP-15657:
Remove own configured RM make variable
|
|
Instead rely on gnu make's pre-defined RM variable which should
equal 'rm -f'
|
|
as called in crypto.erl and erlang.erl.
|
|
bmk/20190204/socket_as_nif/OTP-14831
|
|
The nif callback functions (nif_open) now instead cause an
'nosup' exception if called (instead of badarg).
The basic type uint16_t, uint32_t and int32_t (C99) replaced
"own" (that is, defined by "us") types Uint16, Uint32 and
Sint32.
The point of this is that our Windows build system seems to
be a bit lacking when it comes to types...
Removed "some stuff" that was if-defed. Different solution
when win32 support for sockets has been improved.
Make sure the socket_*.c util modules are not included
in the building for windows.
OTP-15526
|
|
Added preliminary, and temporary, windows adaptions.
Basically they amount to letting all nif-callback functions
returning badarg for all calls if on windows (this has
been accomplished by if-defing the nif-code; if win32 then
badarg else do-something).
OTP-15526
|
|
|
|
|
|
|
|
* maint:
Implement a tab for persistent terms in crashdump viewer
Add tests of persistent terms for crashdump_viewer
Add a persistent term storage
Refactor releasing of literals
Extend the sharing-preserving routines to optionally copy literals
Conflicts:
erts/emulator/Makefile.in
erts/emulator/beam/erl_process_dump.c
erts/preloaded/ebin/erts_internal.beam
erts/preloaded/ebin/init.beam
lib/sasl/src/systools_make.erl
|
|
Persistent terms are useful for storing Erlang terms that are never
or infrequently updated. They have the following advantages:
* Constant time access. A persistent term is not copied when it is
looked up. The constant factor is lower than for ETS, and no locks
are taken when looking up a term.
* Persistent terms are not copied in garbage collections.
* There is only ever one copy of a persistent term (until it is
deleted). That makes them useful for storing configuration data
that needs to be easily accessible by all processes.
Persistent terms have the following drawbacks:
* Updates are expensive. The hash table holding the keys for the
persistent terms are updated whenever a persistent term is added,
updated or deleted.
* Updating or deleting a persistent term triggers a "global GC", which
will schedule a heap scan of all processes to search the heap of all
processes for the deleted term. If a process still holds a reference
to the deleted term, the process will be garbage collected and the
term copied to the heap of the process. This global GC can make the
system less responsive for some time.
Three BIFs (implemented in C in the emulator) is the entire
interface to the persistent term functionality:
* put(Key, Value) to store a persistent term.
* get(Key) to look up a persistent term.
* erase(Key) to delete a persistent term.
There are also two additional BIFs to obtain information about
persistent terms:
* info() to return a map with information about persistent terms.
* get() to return a list of a {Key,Value} tuples for all persistent
terms. (The values are not copied.)
|
|
* richcarl/erts/erl_init-cleanup/OTP-15336:
sasl: Order systools_make:preloaded modules alphabetically
Update preloaded modules
Move calling on_load for preloaded modules to erl_init
Make erl_init.c pass the boot module to erl_init.beam
Remove obsolete comment text
Remove undocumented and unused -# display_items emulator option
Remove broken and undocumented boot function emulator option
Replace remaining references to otp_ring0 with erl_init
Drop otp_ring0, using erl_init instead
Update preloaded modules
Add erl_init module
Conflicts:
erts/emulator/beam/erl_init.c
erts/preloaded/ebin/erl_prim_loader.beam
erts/preloaded/ebin/erl_tracer.beam
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/erts_code_purger.beam
erts/preloaded/ebin/erts_dirty_process_signal_handler.beam
erts/preloaded/ebin/erts_internal.beam
erts/preloaded/ebin/erts_literal_area_collector.beam
erts/preloaded/ebin/init.beam
erts/preloaded/ebin/otp_ring0.beam
erts/preloaded/ebin/prim_buffer.beam
erts/preloaded/ebin/prim_eval.beam
erts/preloaded/ebin/prim_file.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/ebin/prim_zip.beam
erts/preloaded/ebin/zlib.beam
|
|
Added preliminary support for function recvmsg. At the moment
this only works on *nix (Windows has another function, WSARecvMsg,
which has a slightly different API).
Also we have "no" cmsg decode at the moment (just level and type).
OTP-14831
|
|
Started to move the common stuff, such as common utility functions,
debug and encode / decode of basic types.
OTP-14831
|
|
OTP-14831
|
|
|
|
The current ETS ordered_set implementation can quickly become a
scalability bottleneck on multicore machines when an application updates
an ordered_set table from concurrent processes [1][2]. The current
implementation is based on an AVL tree protected from concurrent writes
by a single readers-writer lock. Furthermore, the current implementation
has an optimization, called the stack optimization [3], that can improve
the performance when only a single process accesses a table but can
cause bad scalability even in read-only scenarios. It is possible to
pass the option {write_concurrency, true} to ets:new/2 when creating an
ETS table of type ordered_set but this option has no effect for tables
of type ordered_set without this commit. The new ETS ordered_set
implementation, added by this commit, is only activated when one passes
the options ordered_set and {write_concurrency, true} to the ets:new/2
function. Thus, the previous ordered_set implementation (from here on
called the default implementation) can still be used in applications
that do not benefit from the new implementation. The benchmark results
on the following web page show that the new implementation is many times
faster than the old implementation in some scenarios and that the old
implementation is still better than the new implementation in some
scenarios.
http://winsh.me/ets_catree_benchmark/ets_ca_tree_benchmark_results.html
The new implementation is expected to scale better than the default
implementation when concurrent processes use the following ETS
operations to operate on a table:
delete/2, delete_object/2, first/1, insert/2 (single object),
insert_new/2 (single object), lookup/2, lookup_element/2, member/2,
next/2, take/2 and update_element/3 (single object).
Currently, the new implementation does not have scalable support for the
other operations (e.g., select/2). However, when these operations are
used infrequently, the new implantation may still scale better than the
default implementation as the benchmark results at the URL above shows.
Description of the New Implementation
----------------------------------
The new implementation is based on a data structure which is called the
contention adapting search tree (CA tree for short). The following
publication contains a detailed description of the CA tree:
A Contention Adapting Approach to Concurrent Ordered Sets
Journal of Parallel and Distributed Computing, 2018
Kjell Winblad and Konstantinos Sagonas
https://doi.org/10.1016/j.jpdc.2017.11.007
http://www.it.uu.se/research/group/languages/software/ca_tree/catree_proofs.pdf
A discussion of how the CA tree can be used as an ETS back-end can be
found in another publication [1]. The CA tree is a data structure that
dynamically changes its synchronization granularity based on detected
contention. Internally, the CA tree uses instances of a sequential data
structure to store items. The CA tree implementation contained in this
commit uses the same AVL tree implementation as is used for the default
ordered set implementation. This AVL tree implementation is reused so
that much of the existing code to implement the ETS operations can be
reused.
Tests
-----
The ETS tests in `lib/stdlib/test/ets_SUITE.erl` have been extended to
also test the new ordered_set implementation. The function
ets_SUITE:throughput_benchmark/0 has also been added to this file. This
function can be used to measure and compare the performance of the
different ETS table types and options. This function writes benchmark
data to standard output that can be visualized by the HTML page
`lib/stdlib/test/ets_SUITE_data/visualize_throughput.html`.
[1]
More Scalable Ordered Set for ETS Using Adaptation.
In Thirteenth ACM SIGPLAN workshop on Erlang (2014).
Kjell Winblad and Konstantinos Sagonas.
https://doi.org/10.1145/2633448.2633455
http://www.it.uu.se/research/group/languages/software/ca_tree/erlang_paper.pdf
[2]
On the Scalability of the Erlang Term Storage
In Twelfth ACM SIGPLAN workshop on Erlang (2013)
Kjell Winblad, David Klaftenegger and Konstantinos Sagonas
https://doi.org/10.1145/2505305.2505308
http://winsh.me/papers/erlang_workshop_2013.pdf
[3]
The stack optimization works by keeping one preallocated stack instance
in every ordered_set table. This stack is updated so that it contains
the search path in some read operations (e.g., ets:next/2). This makes
it possible for a subsequent ets:next/2 to avoid traversing some nodes
in some cases. Unfortunately, the preallocated stack needs to be flagged
so that it is not updated concurrently by several threads which cause
bad scalability.
|
|
|
|
|
|
|
|
Floating point exceptions has been disabled since 2011 on macOS
(fa0f8d2c29b) and on Linux since 2014 (c7ddafbe6dbc) because
there were unresolved stability issues.
Floating point exceptions are not disabled by default on FreeBSD,
and if OTP is compiled with gcc (as opposed to clang) floating
point exceptions will be used.
81a6adab693a introduced a bug in erts/emulator/Makefile.in which would
cause the building of OTP to fail if floating point exceptions were
enabled. The bug was not noticed because it turns out that none of our
daily build machines has floating point exceptions enabled.
Since floating point exceptions is not tested, we should not expect
them to work reliably on any platform. Therefore, turn off floating
point exceptions unconditionally in erts/configure.in. For the moment
we will keep the code in the runtime system that handles floating
point exceptions.
(This commit also fixes the bug in erts/emulator/Makefile.in, in
case floating point exceptions ever become reliable and enabled.)
https://bugs.erlang.org/browse/ERL-620
|
|
This commit replaces the old memory instrumentation with a new
implementation that scans carriers instead of wrapping
erts_alloc/erts_free. The old implementation could not extract
information without halting the emulator, had considerable runtime
overhead, and the memory maps it produced were noisy and lacked
critical information.
Since the new implementation walks through existing data structures
there's no longer a need to start the emulator with special flags to
get information about carrier utilization/fragmentation. Memory
fragmentation is also easier to diagnose as it's presented on a
per-carrier basis which eliminates the need to account for "holes"
between mmap segments.
To help track allocations, each allocation can now be tagged with
what it is and who allocated it at the cost of one extra word per
allocation. This is controlled on a per-allocator basis with the
+M<S>atags option, and is enabled by default for binary_alloc and
driver_alloc (which is also used by NIFs).
|
|
Communication between Erlang processes has conceptually always been
performed through asynchronous signaling. The runtime system
implementation has however previously preformed most operation
synchronously. In a system with only one true thread of execution, this
is not problematic (often the opposite). In a system with multiple threads
of execution (as current runtime system implementation with SMP support)
it becomes problematic. This since it often involves locking of structures
when updating them which in turn cause resource contention. Utilizing
true asynchronous communication often avoids these resource contention
issues.
The case that triggered this change was contention on the link lock due
to frequent updates of the monitor trees during communication with a
frequently used server. The signal order delivery guarantees of the
language makes it hard to change the implementation of only some signals
to use true asynchronous signaling. Therefore the implementations
of (almost) all signals have been changed.
Currently the following signals have been implemented as true
asynchronous signals:
- Message signals
- Exit signals
- Monitor signals
- Demonitor signals
- Monitor triggered signals (DOWN, CHANGE, etc)
- Link signals
- Unlink signals
- Group leader signals
All of the above already defined as asynchronous signals in the
language. The implementation of messages signals was quite
asynchronous to begin with, but had quite strict delivery constraints
due to the ordering guarantees of signals between a pair of processes.
The previously used message queue partitioned into two halves has been
replaced by a more general signal queue partitioned into three parts
that service all kinds of signals. More details regarding the signal
queue can be found in comments in the erl_proc_sig_queue.h file.
The monitor and link implementations have also been completely replaced
in order to fit the new asynchronous signaling implementation as good
as possible. More details regarding the new monitor and link
implementations can be found in the erl_monitor_link.h file.
|
|
putenv(3) and friends aren't thread-safe regardless of how you slice
it; a global lock around all environment operations (like before)
keeps things safe as far as our own operations go, but we have
absolutely no control over what libc or a library dragged in by a
driver/NIF does -- they're free to call getenv(3) or putenv(3)
without honoring our lock.
This commit solves this by setting up an "emulated" environment which
can't be touched without going through our interfaces. Third-party
libraries can still shoot themselves in the foot but benign uses of
os:putenv/2 will no longer risk crashing the emulator.
|
|
This improves the latency of file operations as dirty schedulers
are a bit more eager to run jobs than async threads, and use a
single global queue rather than per-thread queues, eliminating the
risk of a job stalling behind a long-running job on the same thread
while other async threads sit idle.
There's no such thing as a free lunch though; the lowered latency
comes at the cost of increased busy-waiting which may have an
adverse effect on some applications. This behavior can be tweaked
with the +sbwt flag, but unfortunately it affects all types of
schedulers and not just dirty ones. We plan to add type-specific
flags at a later stage.
sendfile has been moved to inet_drv to lessen the effect of a nasty
race; the cooperation between inet_drv and efile has never been
airtight and the socket dying at the wrong time (Regardless of
reason) could result in fd aliasing. Moving it to the inet driver
makes it impossible to trigger this by closing the socket in the
middle of a sendfile operation, while still allowing it to be
aborted -- something that can't be done if it stays in the file
driver.
The race still occurs if the controlling process dies in the short
window between dispatching the sendfile operation and the dup(2)
call in the driver, but it's much less likely to happen now.
A proper fix is in the works.
--
Notable functional differences:
* The use_threads option for file:sendfile/5 no longer has any
effect.
* The file-specific DTrace probes have been removed. The same
effect can be achieved with normal tracing together with the
nif__entry/nif__return probes to track scheduling.
--
OTP-14256
|
|
|
|
* lukas/erts/pgo/OTP-14604:
erts: Only do PGO if gcc supports -fprofile-correction
|
|
|
|
* lukas/erts/poll-thread/OTP-14346:
erts: Make a copy of erl_poll.c to help debuggers
|
|
Especially lldb seems to get very confused when the
same source file is used by two different object files.
|
|
OTP-14327
OTP-14340
* bjorn/erts/pack-with-opcode/OTP-14325:
Pack operands for combined instructions into the instruction word
beam_makeops: Use named arguments for the code generation functions
Optimize packing for "optional use" operands
beam_makeops: Print the instruction name for fatal packing errors
Introduce a syntax for marking operands as "optional use"
beam_makeops: Refactor parsing of specific instructions
Optimize instruction prefetch
Pack operands into the instruction word
Use 32-bits pointers to C code
Move LD flags for hipe from Makefile.in to configure.in
beam_disasm: Correct printing of y registers
ops.tab: Slightly optimize badmatch on a Y register
macros.tab: Fix assertion in SET_I_REL()
|
|
* lukas/erts/beam-emu-vars:
erts: Add makefile target to check emu register allocation
|
|
On 64-bit machines where the C code is always at address below 4Gb,
pack one or more operands into the instruction word.
|
|
We want the flags to be available for other tests in configure.in.
|
|
|
|
|
|
|
|
* lukas/erts/pgo/OTP-14604:
Add support for building a pgo beam_emu
|
|
|
|
|
|
|
|
All operations will now yield appropriately, allowing them to be used
freely in concurrent applications. This commit also deprecates the
functions listed below, although they won't raise deprecation
warnings until OTP 21:
zlib:adler32
zlib:crc32
zlib:inflateChunk
zlib:getBufSize
zlib:setBufSize
The behavior of throwing an error when a dictionary is required for
decompression has also been deprecated.
|
