| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
* egil/gc-increase-tenure-rate/OTP-11617:
erts: Increase gc tenure rate
|
|
as we don't use it and instead have the feature to disable GC
during trapping BIFs.
|
|
The garbage collector tries to maintain the previous heap block size
during a minor gc, i.e. 'need' is not utilized in determining the size of
the new heap, instead it relies on tenure and garbage to be sufficiently large.
In instances during intense growing with exlusively live data on the heap
coupled with delayed tenure, fullsweeps would be triggered directly after
a minor gc to make room for 'need' since the new heap would be full.
To remedy this, the tenure of terms on the minor heap will always happen
(if it is below the high watermark) instead of every other minor gc.
|
|
* rickard/garbage_collect/OTP-11388:
Parallel check_process_code when code_server purge a module
Functionality for disabling garbage collection
Use asynchronous check_process_code in code_parallel_SUITE
Execution of system tasks in context of another process
Conflicts:
bootstrap/lib/kernel/ebin/hipe_unified_loader.beam
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/erts_internal.beam
|
|
The test needs to be false to fail, not true. This was noticed with the
following warning, where marking erl_assert_error as non-returning didn't
help:
beam/erl_gc.c:2772:2: warning: variable 'refc' is used uninitialized whenever switch default is taken [-Wsometimes-uninitialized]
default:
^~~~~~~
beam/erl_gc.c:2775:26: note: uninitialized use occurs here
ERTS_CHK_OFFHEAP_ASSERT(refc >= 1);
^~~~
beam/erl_gc.c:2738:11: note: expanded from macro 'ERTS_CHK_OFFHEAP_ASSERT'
if (!(EXP)) \
^
beam/erl_gc.c:2759:18: note: initialize the variable 'refc' to silence this warning
erts_aint_t refc;
^
= 0
|
|
Being able to disable garbage collection over context
switches vastly simplifies implementation of yielding
native code that builds large or complex data structures
on the heap. This since the heap can be left in an
inconsistent state over the context switch.
|
|
|
|
|
|
Rewrite and extend of Happi's initial work
Extra_root to process structure to enable GC of state - Changed the
process structure to point to a separate struct, the struct also
contains a destructor function to allow for proper cleanup.
Rewrote encode_size_struct and enc_term to have internal versions with
reduction counters which will result in interrupt for later restart
when the counter reaches zero - removed the EWA_STACK from Happis
version and directly save the ESTACK's and WSTACK's in the above
mentioned struct (or array thereof) that are pointed out from the
process structure. The destructor will take care of the deallocation
in case of process death.
Added ESTACK and WSTACK macros to save and restore stack and to change
allocator, which makes the previously mentioned stack-save work.
Rewrote enc_term to not store pointers on the stack, and use one WSTACK
for commands etc and another ESTACK for Eterms - Slightly different than
Happis version to make halfword code simpler.
Rewrote encode_size_struct2 so that it does not store pointers on the
stack, also switched to ESTACK instead of WSTACK, this also handles
halfword correctly.
Added interfaces for chunkwise compression, that are
used from term_to_binary/2 when the compressed option is given.
|
|
|
|
|
|
|
|
Using %p, %bpx and %bex for pointers, UWords and Uint.
|
|
OTP-10415
* sverk/egil/r16/new-alloc-header-scheme/OTP-10273: (42 commits)
erts: Make ll main mbc fit into 2pow size
erts: Clear entire mseg cache upon request
erts: Reduce max heap sizes
tests: Refactor away ?line macro in beam_SUITE
tests: Fix heap_sizes check
tests: Refactor away ?line macro in process_SUITE
tests: Use new correct min_bin_vheap_size in test
erts: Set super alignment (256kb) and limits for sbct (8Mb) and lmbcs (128Mb)
erts: Do not cache segments that are misaligned
erts: Add mseg cache for large sbc segments
erts: Reintroduce mseg options amcbf and rmcbf
erts: Optimize erl_alloc_util.c by substitute MBC_BLK_SZ
erts: Fix bug when allocating size near sbc_threshold
erts: Make gc sizes fit into MB Carrier blocks
erts: Force sbmbc to be disabled in a crude way
erts: Fix new header scheme for win64
erts: Fix mseg cache. Malplaced NULL pointer
erts: Remove unused mseg options amcbf and rmcbf
erts: Use aligned bits as constant in mseg_alloc
erts: Don't let zero be considered a power of two
...
Conflicts:
erts/emulator/test/process_SUITE.erl
|
|
Reduces max heap sizes to max addressable memory space. In reality this could be
even less since we need at least twice as much adressable memory to do a garbage
collection.
The rest of the system also uses memory thus further constraining heap memory
space and in the 64 bit case we really only have 48 bits mappable memory.
|
|
* Account for block header size in gc-sizes
* Also slow down growth to 20% instead of 25% when
size threshold is reached
|
|
rickard/r16/port-optimizations/OTP-10336
* rickard/port-optimizations/OTP-10336:
Change annotate level for emacs-22 in cerl
Update etp-commands
Add documentation on communication in Erlang
Add support for busy port message queue
Add driver callback epilogue
Implement true asynchronous signaling between processes and ports
Add erl_drv_[send|output]_term
Move busy port flag
Use rwlock for driver list
Optimize management of port tasks
Improve configuration of process and port tables
Remove R9 compatibility features
Use ptab functionality also for ports
Prepare for use of ptab functionality also for ports
Atomic port state
Generalize process table implementation
Implement functionality for delaying thread progress from unmanaged threads
Conflicts:
erts/doc/src/erl_driver.xml
erts/doc/src/erlang.xml
erts/emulator/beam/beam_bif_load.c
erts/emulator/beam/beam_bp.c
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.c
erts/emulator/beam/copy.c
erts/emulator/beam/erl_alloc.c
erts/emulator/beam/erl_alloc.types
erts/emulator/beam/erl_bif_info.c
erts/emulator/beam/erl_bif_port.c
erts/emulator/beam/erl_bif_trace.c
erts/emulator/beam/erl_init.c
erts/emulator/beam/erl_message.c
erts/emulator/beam/erl_port_task.c
erts/emulator/beam/erl_process.c
erts/emulator/beam/erl_process.h
erts/emulator/beam/erl_process_lock.c
erts/emulator/beam/erl_trace.c
erts/emulator/beam/export.h
erts/emulator/beam/global.h
erts/emulator/beam/io.c
erts/emulator/sys/unix/sys.c
erts/emulator/sys/vxworks/sys.c
erts/emulator/test/port_SUITE.erl
erts/etc/unix/cerl.src
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/src/prim_inet.erl
lib/hipe/cerl/erl_bif_types.erl
lib/kernel/doc/src/inet.xml
lib/kernel/src/inet.erl
|
|
|
|
|
|
Conflicts:
erts/doc/src/erlang.xml
|
|
When erts_bs_append() calls the garbage collector, it has set
the PB_ACTIVE_WRITER flag in the ProcBin for the binary object
is about to be appended to. Therefore, it is expected that the
garbage collector should neither move nor shrink the binary
object.
But if the garbage collector does a minor collection (thereby
clearing the PB_ACTIVE_WRITER flag in the ProcBin) and there is
still not enough heap space, there will be a second major
garbage collection. During the major collection (since the
the PB_ACTIVE_WRITER flag was cleared), the binary object may
be shrunk or moved (depending on sizes and how many other
writable binaries there are in the process).
Avoid the problem by clearing the PB_ACTIVE_WRITER flags in
a separate pass after the garbage collection(s).
Thanks to Denis Titoruk for helping us find this bug.
|
|
|
|
|
|
|
|
|
|
User tags in a dynamic trace enabled VM are spread throughout the system
in the same way as seq_trace tokens. This is used by the file module
and various other modules to get hold of the tag from the user process
without changing the protocol.
|
|
|
|
Add probes to the virtual machine, except (mostly) the efile_drv.c
driver and other file I/O-related source files.
|
|
* jz/reduce-smp-locking-time-range:
erts: reduce smp locking time range in erts_garbage_collect
OTP-9912
|
|
get_now should be use out of process status locking, then
it can reduce smp locking time range in here.
|
|
Seen causing segv on sparc with hibernate_native_SUITE:basic:
nstack_walk_frame_ra (nsp=0x0, sdesc=0x6c8f2c) at hipe_risc_gc.h:105
gensweep_nstack (p=0x6a6930, ptr_old_htop=0xffbfea50, ptr_n_htop=0xffbfea4c) at hipe/hipe_gc.c:224
do_minor (p=0x6a6930, new_sz=233, objv=0x6a6984, nobj=3) at beam/erl_gc.c:949
minor_collection (p=0x6a6930, need=3, objv=0x6a6984, nobj=3, recl=0xffbfece4) at beam/erl_gc.c:811
erts_garbage_collect (p=0x6a6930, need=3, objv=0x6a6984, nobj=3) at beam/erl_gc.c:379
erts_send_message (sender=0x6a6048, receiver=0x6a6930, receiver_locks=0xffbfedf8, message=6132762, flags=0) at beam/erl_message.c:922
do_send (p=0x6a6048, to=515, msg=6132762, suspend=1) at beam/bif.c:2052
erl_send (p=0x6a6048, to=515, msg=6132762) at beam/bif.c:2151
send_2 (A__p=0x6a6048, BIF__ARGS=0x6a609c) at beam/bif.c:2146
nbif_send_2 ()
|
|
To simplify the implementation of literal pools (constant pools)
for the R12 release, a shortcut was taken regarding binaries --
all binaries would be stored as heap binaries regardless of size.
To allow a module containing literals to be unloaded, literal
terms are copied when sent to another process. That means that
huge literal binaries will also be copied if they are sent to
another process, which could be surprising.
Another problem is that the arity field in the header for the heap
object may not be wide enough to handle big binaries.
Therefore, bite the bullet and allow refc binaries to be stored
in literal pools. In short, the following need to be changed:
* Each loaded module needs a MSO list, linking all refc binaries
in the literal pool.
* When check_process_code/2 copies literals to a process heap,
it must link each referenced binary into the MSO list for the
process and increment the reference counter for the binary.
* purge_module/1 must decrement the reference counter for each
refc binary in the literal pool.
|
|
* rickard/thr-progress-block/OTP-9631:
Replace system block with thread progress block
|
|
The ERTS internal system block functionality has been replaced by
new functionality for blocking the system. The old system block
functionality had contention issues and complexity issues. The
new functionality piggy-backs on thread progress tracking functionality
needed by newly introduced lock-free synchronization in the runtime
system. When the functionality for blocking the system isn't used
there is more or less no overhead at all. This since the functionality
for tracking thread progress is there and needed anyway.
|
|
|
|
Store arguments for a trap in the X register array to allow
traps to have any number of arguments.
|
|
We discovered that if a single Erlang process tried to grow above 32
GB (i.e., more 64-bit words than can be counted by a 32-bit number),
the VM failed to find the next larger heap size, even though there
were plenty more heap sizes left to pick from and even though we had a
lot more memory available on the machine. (Obviously, this is only
applicable on 64-bit Erlang.)
It turned out to be due to some 'int' variables in the heap resizing
parts of erl_gc.c not being properly updated to 'Uint' or 'Sint'. Once
that was fixed, I got segfaults instead as soon as the heap got larger
than 2^32 words, due to even more 'int' declarations in the same file,
but now in the GC code.
After fixing this as well, I successfully ran an Erlang node in which
a single Erlang process had a heap so large that I'm not at liberty to
divulge the exact size, but I think the scientific term is
"humongous", and I'm confident that there are no further immediate
problems with very very large individual process heaps.
|
|
* ms/beam-fix-format-specifiers-in-erl_exit-msg:
Fix format specifiers in erl_exit messages
OTP-9262
|
|
Fix an error message by using an unsigned integer specifier as seen in
a tweet by @metabrew:
#erlang VM crashed with "no next heap size found: -2090496108,
offset 0", suddenly allocated all available RAM
Also correct mis-typed string formats in bif.c.
|
|
|
|
|
|
This commit fixes four related bugs:
- calling hibernate/3 using a dynamic call would fail with badarg
as hibernate/3 as a BIF was not implemented. hibernate/3 is generally
provided as a Beam instruction, and code is translated to use this
instruction when loaded.
- calling hibernate/3 from HiPE would fail with badarg because this
would call the aforementioned BIF which was not implemented.
- calling hibernate/3 with some HiPE-native garbage in the process heap
would randomly crash at the next garbage collect. This bug only
happened in a complex, yet reproduceable scenarios, where native code
calls beam code that calls hibernate/3, and the process has some
garbage when being hibernated and the process generates garbage when
awaken.
- when entering HiPE, the process current_function can be set and be
inaccurate.
The fix is three folded:
- hibernate_3 BIF now actually works instead of throwing a badarg. While
hibernate_3 BIF was (usually) not called from BEAM, it is called from
HiPE. hibernate behaviour is very close to the scheduler and this is why
it is implemented as an instruction in BEAM. The fix consists in doing
the actual hibernation (through the now exported erts_hibernate
function) and setting the process flag to TRAP as well as the process
status to P_WAITING. On BIF epilogue in both BEAM and HiPE, this status
is tested on TRAP and if set, the scheduler is invoked. The i_hibernate
instruction and translation code is now redundant and could be deleted.
- hibernation now also empties the HiPE native stack, with a new
function hipe_empty_nstack provided by Mikael Pettersson.
- when entering HiPE through hipe_mode_switch, p->current is cleared,
as suggested by Mikael Pettersson. p->current normally hold a pointer to
the {M,F,A} of the current function if it exists. When hibernating, it
is set to {erlang,hibernate,3}, and all stdlib hibernate tests
(gen_server_SUITE:hibernate/1, proc_lib_suite:hibernate/1, etc.)
actually rely on this information. Clearing p->current fixes the tests
and avoids the surprise one might have when querying the process info
of a process that hibernated and woke up in a native function.
Non-regression tests are provided, a test for the dynamic call as well
as a Makefile-handled duplication of the hibernate_SUITE into
hibernate_native_SUITE for the HiPE case.
|
|
|
|
This will reduce the risk of integer wrapping in bin vheap counting.
The vheap size series will now use the golden ratio instead of doubling
and fibonacci sequences.
OTP #8730
|
|
OTP #8730
|
|
|
|
Merging the three off-heap lists (binaries, funs and externals) into
one list. This reduces memory consumption by two words (pointers) per
ETS object.
|
|
New NIF features:
Send messages from a NIF, or from thread created by NIF, to any local
process (enif_send)
Store terms between NIF calls (enif_alloc_env, enif_make_copy)
Create binary terms with user defined memory management
(enif_make_resource_binary)
|
