| Age | Commit message (Collapse) | Author |
|---|
|
|
|
* rickard/time_api/OTP-11997: (22 commits)
Update primary bootstrap
inets: Suppress deprecated warning on erlang:now/0
inets: Cleanup of multiple copies of functions Add inets_lib with common functions used by multiple modules
inets: Update comments
Suppress deprecated warning on erlang:now/0
Use new time API and be back-compatible in inets Remove unused functions and removed redundant test
asn1 test SUITE: Eliminate use of now/0
Disable deprecated warning on erlang:now/0 in diameter_lib
Use new time API and be back-compatible in ssh
Replace all calls to now/0 in CT with new time API functions
test_server: Replace usage of erlang:now() with usage of new API
Replace usage of erlang:now() with usage of new API
Replace usage of erlang:now() with usage of new API
Replace usage of erlang:now() with usage of new API
Replace usage of erlang:now() with usage of new API
otp_SUITE: Warn for calls to erlang:now/0
Replace usage of erlang:now() with usage of new API
Multiple timer wheels
Erlang based BIF timer implementation for scalability
Implement ethread events with timeout
...
Conflicts:
bootstrap/bin/start.boot
bootstrap/bin/start_clean.boot
bootstrap/lib/compiler/ebin/beam_asm.beam
bootstrap/lib/compiler/ebin/compile.beam
bootstrap/lib/kernel/ebin/auth.beam
bootstrap/lib/kernel/ebin/dist_util.beam
bootstrap/lib/kernel/ebin/global.beam
bootstrap/lib/kernel/ebin/hipe_unified_loader.beam
bootstrap/lib/kernel/ebin/inet_db.beam
bootstrap/lib/kernel/ebin/inet_dns.beam
bootstrap/lib/kernel/ebin/inet_res.beam
bootstrap/lib/kernel/ebin/os.beam
bootstrap/lib/kernel/ebin/pg2.beam
bootstrap/lib/stdlib/ebin/dets.beam
bootstrap/lib/stdlib/ebin/dets_utils.beam
bootstrap/lib/stdlib/ebin/erl_tar.beam
bootstrap/lib/stdlib/ebin/escript.beam
bootstrap/lib/stdlib/ebin/file_sorter.beam
bootstrap/lib/stdlib/ebin/otp_internal.beam
bootstrap/lib/stdlib/ebin/qlc.beam
bootstrap/lib/stdlib/ebin/random.beam
bootstrap/lib/stdlib/ebin/supervisor.beam
bootstrap/lib/stdlib/ebin/timer.beam
erts/aclocal.m4
erts/emulator/beam/bif.c
erts/emulator/beam/erl_bif_info.c
erts/emulator/beam/erl_db_hash.c
erts/emulator/beam/erl_init.c
erts/emulator/beam/erl_process.h
erts/emulator/beam/erl_thr_progress.c
erts/emulator/beam/utils.c
erts/emulator/sys/unix/sys.c
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/erts_internal.beam
erts/preloaded/ebin/init.beam
erts/preloaded/src/erts_internal.erl
lib/common_test/test/ct_hooks_SUITE_data/cth/tests/empty_cth.erl
lib/diameter/src/base/diameter_lib.erl
lib/kernel/src/os.erl
lib/ssh/test/ssh_basic_SUITE.erl
system/doc/efficiency_guide/advanced.xml
|
|
|
|
The old time API is based on erlang:now/0. The major issue with
erlang:now/0 is that it was intended to be used for so many
unrelated things. This tied these unrelated operations together
and unnecessarily caused performance, scalability as well as
accuracy, and precision issues for operations that do not need
to have such issues. The new API spreads different functionality
over multiple functions in order to improve on this.
The new API consists of a number of new BIFs:
- erlang:convert_time_unit/3
- erlang:monotonic_time/0
- erlang:monotonic_time/1
- erlang:system_time/0
- erlang:system_time/1
- erlang:time_offset/0
- erlang:time_offset/1
- erlang:timestamp/0
- erlang:unique_integer/0
- erlang:unique_integer/1
- os:system_time/0
- os:system_time/1
and a number of extensions of existing BIFs:
- erlang:monitor(time_offset, clock_service)
- erlang:system_flag(time_offset, finalize)
- erlang:system_info(os_monotonic_time_source)
- erlang:system_info(time_offset)
- erlang:system_info(time_warp_mode)
- erlang:system_info(time_correction)
- erlang:system_info(start_time)
See the "Time and Time Correction in Erlang" chapter of the
ERTS User's Guide for more information.
|
|
and usage
|
|
* sverk/hipe-inline-reserve-trap-frame:
erts: Extend usage of ASM macro to avoid including asm macros in C code
erts: Make hipe_{un}reserve_beam_trap_frame INLINE
|
|
|
|
except the reference counter 'refc', as different callers
have different strategies regarding the lifetime of the binary.
|
|
In the #erlang IRC channel Anthony Ramine once mentioned the idea of
allowing a NIF to use an emulator trap, similar to a BIF trap, to schedule
another NIF for execution. This is exactly how dirty NIFs were implemented
for Erlang/OTP 17.0, so this commit refactors and generalizes that dirty
NIF code to support a new enif_schedule_nif() API function.
The enif_schedule_nif() function allows a long-running NIF to be broken
into separate NIF invocations. The NIF first executes part of the
long-running task, then calls enif_schedule_nif() to schedule a NIF for
later execution to continue the task. Any number of NIFs can be scheduled
in this manner, one after another. Since the emulator regains control
between invocations, this helps avoid problems caused by native code tying
up scheduler threads for too long.
The enif_schedule_nif() function also replaces the original experimental
dirty NIF API. The function takes a flags parameter that a caller can use
to indicate the NIF should be scheduled onto either a dirty CPU scheduler
thread, a dirty I/O scheduler thread, or scheduled as a regular NIF on a
regular scheduler thread. With this change, the original experimental
enif_schedule_dirty_nif(), enif_schedule_dirty_nif_finalizer() and
enif_dirty_nif_finalizer() API functions are no longer needed and have been
removed. Explicit scheduling of a dirty NIF finalization function is no
longer necessary; if an application wants similar functionality, it can
have a dirty NIF just invoke enif_schedule_nif() to schedule a non-dirty
NIF to complete its task.
Lift the restriction that dirty NIFs can't call enif_make_badarg() to raise
an exception. This was a problem with the original dirty NIF API because it
forced developers to get and check all incoming arguments in a regular NIF,
and then schedule the dirty NIF which then had to get all the arguments
again. Now, the argument checking can be done in the dirty NIF and it can
call enif_make_badarg() itself to flag incorrect arguments.
Extend the ErlNifFunc struct with a new flags field that allows NIFs to be
declared as dirty. The default value for this field is 0, indicating a
regular NIF, so it's backwards compatible with all existing statically
initialized ErlNifFunc struct instances, and so such instances require no
code changes. Defining the flags field with a value of
ERL_NIF_DIRTY_JOB_CPU_BOUND indicates that the NIF should execute on a
dirty CPU scheduler thread, or defining it with a value of
ERL_NIF_DIRTY_JOB_IO_BOUND indicates that the NIF should execute on a dirty
I/O scheduler thread. Any other flags field value causes a NIF library
loading error.
Extend the ErlNifEntry struct with a new options field that indicates
whether a NIF library was built with support for optional features such as
dirty NIFs. When a NIF library is loaded, the runtime checks the options
field to ensure compatibility. If a NIF library built with dirty NIF
support is loaded into a runtime that does not support dirty NIFs, and the
library defines one or more ErlNifFunc entries with non-zero flags fields
indicating dirty NIFs, a NIF library loading error results. There is no
error if a NIF library built with dirty NIF support is loaded into a
runtime that does not support dirty NIFs but the library does not have any
dirty NIFs. It is also not an error if a library without dirty NIF support
is loaded into a runtime built with dirty NIF support.
Add documentation and tests for enif_schedule_nif().
|
|
|
|
|
|
* egil/gc-increase-tenure-rate/OTP-11617:
erts: Increase gc tenure rate
|
|
as we don't use it and instead have the feature to disable GC
during trapping BIFs.
|
|
The garbage collector tries to maintain the previous heap block size
during a minor gc, i.e. 'need' is not utilized in determining the size of
the new heap, instead it relies on tenure and garbage to be sufficiently large.
In instances during intense growing with exlusively live data on the heap
coupled with delayed tenure, fullsweeps would be triggered directly after
a minor gc to make room for 'need' since the new heap would be full.
To remedy this, the tenure of terms on the minor heap will always happen
(if it is below the high watermark) instead of every other minor gc.
|
|
* rickard/garbage_collect/OTP-11388:
Parallel check_process_code when code_server purge a module
Functionality for disabling garbage collection
Use asynchronous check_process_code in code_parallel_SUITE
Execution of system tasks in context of another process
Conflicts:
bootstrap/lib/kernel/ebin/hipe_unified_loader.beam
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/erts_internal.beam
|
|
The test needs to be false to fail, not true. This was noticed with the
following warning, where marking erl_assert_error as non-returning didn't
help:
beam/erl_gc.c:2772:2: warning: variable 'refc' is used uninitialized whenever switch default is taken [-Wsometimes-uninitialized]
default:
^~~~~~~
beam/erl_gc.c:2775:26: note: uninitialized use occurs here
ERTS_CHK_OFFHEAP_ASSERT(refc >= 1);
^~~~
beam/erl_gc.c:2738:11: note: expanded from macro 'ERTS_CHK_OFFHEAP_ASSERT'
if (!(EXP)) \
^
beam/erl_gc.c:2759:18: note: initialize the variable 'refc' to silence this warning
erts_aint_t refc;
^
= 0
|
|
Being able to disable garbage collection over context
switches vastly simplifies implementation of yielding
native code that builds large or complex data structures
on the heap. This since the heap can be left in an
inconsistent state over the context switch.
|
|
|
|
|
|
Rewrite and extend of Happi's initial work
Extra_root to process structure to enable GC of state - Changed the
process structure to point to a separate struct, the struct also
contains a destructor function to allow for proper cleanup.
Rewrote encode_size_struct and enc_term to have internal versions with
reduction counters which will result in interrupt for later restart
when the counter reaches zero - removed the EWA_STACK from Happis
version and directly save the ESTACK's and WSTACK's in the above
mentioned struct (or array thereof) that are pointed out from the
process structure. The destructor will take care of the deallocation
in case of process death.
Added ESTACK and WSTACK macros to save and restore stack and to change
allocator, which makes the previously mentioned stack-save work.
Rewrote enc_term to not store pointers on the stack, and use one WSTACK
for commands etc and another ESTACK for Eterms - Slightly different than
Happis version to make halfword code simpler.
Rewrote encode_size_struct2 so that it does not store pointers on the
stack, also switched to ESTACK instead of WSTACK, this also handles
halfword correctly.
Added interfaces for chunkwise compression, that are
used from term_to_binary/2 when the compressed option is given.
|
|
|
|
|
|
|
|
Using %p, %bpx and %bex for pointers, UWords and Uint.
|
|
OTP-10415
* sverk/egil/r16/new-alloc-header-scheme/OTP-10273: (42 commits)
erts: Make ll main mbc fit into 2pow size
erts: Clear entire mseg cache upon request
erts: Reduce max heap sizes
tests: Refactor away ?line macro in beam_SUITE
tests: Fix heap_sizes check
tests: Refactor away ?line macro in process_SUITE
tests: Use new correct min_bin_vheap_size in test
erts: Set super alignment (256kb) and limits for sbct (8Mb) and lmbcs (128Mb)
erts: Do not cache segments that are misaligned
erts: Add mseg cache for large sbc segments
erts: Reintroduce mseg options amcbf and rmcbf
erts: Optimize erl_alloc_util.c by substitute MBC_BLK_SZ
erts: Fix bug when allocating size near sbc_threshold
erts: Make gc sizes fit into MB Carrier blocks
erts: Force sbmbc to be disabled in a crude way
erts: Fix new header scheme for win64
erts: Fix mseg cache. Malplaced NULL pointer
erts: Remove unused mseg options amcbf and rmcbf
erts: Use aligned bits as constant in mseg_alloc
erts: Don't let zero be considered a power of two
...
Conflicts:
erts/emulator/test/process_SUITE.erl
|
|
Reduces max heap sizes to max addressable memory space. In reality this could be
even less since we need at least twice as much adressable memory to do a garbage
collection.
The rest of the system also uses memory thus further constraining heap memory
space and in the 64 bit case we really only have 48 bits mappable memory.
|
|
* Account for block header size in gc-sizes
* Also slow down growth to 20% instead of 25% when
size threshold is reached
|
|
rickard/r16/port-optimizations/OTP-10336
* rickard/port-optimizations/OTP-10336:
Change annotate level for emacs-22 in cerl
Update etp-commands
Add documentation on communication in Erlang
Add support for busy port message queue
Add driver callback epilogue
Implement true asynchronous signaling between processes and ports
Add erl_drv_[send|output]_term
Move busy port flag
Use rwlock for driver list
Optimize management of port tasks
Improve configuration of process and port tables
Remove R9 compatibility features
Use ptab functionality also for ports
Prepare for use of ptab functionality also for ports
Atomic port state
Generalize process table implementation
Implement functionality for delaying thread progress from unmanaged threads
Conflicts:
erts/doc/src/erl_driver.xml
erts/doc/src/erlang.xml
erts/emulator/beam/beam_bif_load.c
erts/emulator/beam/beam_bp.c
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.c
erts/emulator/beam/copy.c
erts/emulator/beam/erl_alloc.c
erts/emulator/beam/erl_alloc.types
erts/emulator/beam/erl_bif_info.c
erts/emulator/beam/erl_bif_port.c
erts/emulator/beam/erl_bif_trace.c
erts/emulator/beam/erl_init.c
erts/emulator/beam/erl_message.c
erts/emulator/beam/erl_port_task.c
erts/emulator/beam/erl_process.c
erts/emulator/beam/erl_process.h
erts/emulator/beam/erl_process_lock.c
erts/emulator/beam/erl_trace.c
erts/emulator/beam/export.h
erts/emulator/beam/global.h
erts/emulator/beam/io.c
erts/emulator/sys/unix/sys.c
erts/emulator/sys/vxworks/sys.c
erts/emulator/test/port_SUITE.erl
erts/etc/unix/cerl.src
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/src/prim_inet.erl
lib/hipe/cerl/erl_bif_types.erl
lib/kernel/doc/src/inet.xml
lib/kernel/src/inet.erl
|
|
|
|
|
|
Conflicts:
erts/doc/src/erlang.xml
|
|
When erts_bs_append() calls the garbage collector, it has set
the PB_ACTIVE_WRITER flag in the ProcBin for the binary object
is about to be appended to. Therefore, it is expected that the
garbage collector should neither move nor shrink the binary
object.
But if the garbage collector does a minor collection (thereby
clearing the PB_ACTIVE_WRITER flag in the ProcBin) and there is
still not enough heap space, there will be a second major
garbage collection. During the major collection (since the
the PB_ACTIVE_WRITER flag was cleared), the binary object may
be shrunk or moved (depending on sizes and how many other
writable binaries there are in the process).
Avoid the problem by clearing the PB_ACTIVE_WRITER flags in
a separate pass after the garbage collection(s).
Thanks to Denis Titoruk for helping us find this bug.
|
|
|
|
|
|
|
|
|
|
User tags in a dynamic trace enabled VM are spread throughout the system
in the same way as seq_trace tokens. This is used by the file module
and various other modules to get hold of the tag from the user process
without changing the protocol.
|
|
|
|
Add probes to the virtual machine, except (mostly) the efile_drv.c
driver and other file I/O-related source files.
|
|
* jz/reduce-smp-locking-time-range:
erts: reduce smp locking time range in erts_garbage_collect
OTP-9912
|
|
get_now should be use out of process status locking, then
it can reduce smp locking time range in here.
|
|
Seen causing segv on sparc with hibernate_native_SUITE:basic:
nstack_walk_frame_ra (nsp=0x0, sdesc=0x6c8f2c) at hipe_risc_gc.h:105
gensweep_nstack (p=0x6a6930, ptr_old_htop=0xffbfea50, ptr_n_htop=0xffbfea4c) at hipe/hipe_gc.c:224
do_minor (p=0x6a6930, new_sz=233, objv=0x6a6984, nobj=3) at beam/erl_gc.c:949
minor_collection (p=0x6a6930, need=3, objv=0x6a6984, nobj=3, recl=0xffbfece4) at beam/erl_gc.c:811
erts_garbage_collect (p=0x6a6930, need=3, objv=0x6a6984, nobj=3) at beam/erl_gc.c:379
erts_send_message (sender=0x6a6048, receiver=0x6a6930, receiver_locks=0xffbfedf8, message=6132762, flags=0) at beam/erl_message.c:922
do_send (p=0x6a6048, to=515, msg=6132762, suspend=1) at beam/bif.c:2052
erl_send (p=0x6a6048, to=515, msg=6132762) at beam/bif.c:2151
send_2 (A__p=0x6a6048, BIF__ARGS=0x6a609c) at beam/bif.c:2146
nbif_send_2 ()
|
|
To simplify the implementation of literal pools (constant pools)
for the R12 release, a shortcut was taken regarding binaries --
all binaries would be stored as heap binaries regardless of size.
To allow a module containing literals to be unloaded, literal
terms are copied when sent to another process. That means that
huge literal binaries will also be copied if they are sent to
another process, which could be surprising.
Another problem is that the arity field in the header for the heap
object may not be wide enough to handle big binaries.
Therefore, bite the bullet and allow refc binaries to be stored
in literal pools. In short, the following need to be changed:
* Each loaded module needs a MSO list, linking all refc binaries
in the literal pool.
* When check_process_code/2 copies literals to a process heap,
it must link each referenced binary into the MSO list for the
process and increment the reference counter for the binary.
* purge_module/1 must decrement the reference counter for each
refc binary in the literal pool.
|
|
* rickard/thr-progress-block/OTP-9631:
Replace system block with thread progress block
|
|
The ERTS internal system block functionality has been replaced by
new functionality for blocking the system. The old system block
functionality had contention issues and complexity issues. The
new functionality piggy-backs on thread progress tracking functionality
needed by newly introduced lock-free synchronization in the runtime
system. When the functionality for blocking the system isn't used
there is more or less no overhead at all. This since the functionality
for tracking thread progress is there and needed anyway.
|
|
|
|
Store arguments for a trap in the X register array to allow
traps to have any number of arguments.
|
|
We discovered that if a single Erlang process tried to grow above 32
GB (i.e., more 64-bit words than can be counted by a 32-bit number),
the VM failed to find the next larger heap size, even though there
were plenty more heap sizes left to pick from and even though we had a
lot more memory available on the machine. (Obviously, this is only
applicable on 64-bit Erlang.)
It turned out to be due to some 'int' variables in the heap resizing
parts of erl_gc.c not being properly updated to 'Uint' or 'Sint'. Once
that was fixed, I got segfaults instead as soon as the heap got larger
than 2^32 words, due to even more 'int' declarations in the same file,
but now in the GC code.
After fixing this as well, I successfully ran an Erlang node in which
a single Erlang process had a heap so large that I'm not at liberty to
divulge the exact size, but I think the scientific term is
"humongous", and I'm confident that there are no further immediate
problems with very very large individual process heaps.
|
|
* ms/beam-fix-format-specifiers-in-erl_exit-msg:
Fix format specifiers in erl_exit messages
OTP-9262
|
|
Fix an error message by using an unsigned integer specifier as seen in
a tweet by @metabrew:
#erlang VM crashed with "no next heap size found: -2090496108,
offset 0", suddenly allocated all available RAM
Also correct mis-typed string formats in bif.c.
|
