| Age | Commit message (Collapse) | Author |
|---|
|
* sverk/dec_term-bin-overhead/OTP-12554:
erts: Add missing binary offheap overhead in binary_to_term
|
|
|
|
Adding ERTS_SWORD_MAX to a pointer does not work
as a way to disable a bound check.
Remove the hp_end from ErtsHeapFactory as it isn't really used anyway.
|
|
flatmap: Small map
hashmap: Large map
map: flatmap or hashmap
|
|
Strategy: Calculate an over estimation of heap size that will give
such a low probability for overflow, that "it will not happen".
Scary assumption 1: Uniformly distributed hash values.
Scary assumption 2: Tree size is normally distributed (right?)
|
|
|
|
with over estimation of heap size.
|
|
|
|
|
|
|
|
Binary offheap overhead is used to trigger GC when a process is
referring "too much" binary offheap data.
Offheap binaries created from external format (binary_to_term,
distributed messages or compacted ets tables) were not accounted for.
Example: A process receiving a lot of binary data in distributed messages,
while not building much terms on its heap, could cause an extensive
memory consumption for garbage binaries.
|
|
Conflicts:
erts/emulator/hipe/hipe_bif0.c
|
|
Bignums are artifically restricted in size. Arithmetic and logical
operations check the sizes of resulting bignums, and turn oversize
results into system_limit exceptions.
However, this check is not performed when bignums are constructed by
binary matching. The consequence is that such matchings can construct
oversize bignums that satisfy is_integer/1 yet don't work. Performing
arithmetic such as Term - 0 fails with a system_limit exception. Worse,
performing a logical operation such as Term band Term results in [].
The latter occurs because the size checking (e.g. in erts_band()) is
a simple ASSERT(is_not_nil(...)) on the result of the bignum operation,
which internally is [] (NIL) in the case of oversize results. However,
ASSERT is a no-op in release builds, so the error goes unnoticed and []
is returned as the result of the band/2.
This patch addresses this by preventing oversize bignums from entering
the VM via binary matching:
- the internal bytes_to_big() procedure is augmented to return NIL for
oversize results, just like big_norm()
- callers of bytes_to_big() are augmented to check for NIL returns and
signal errors in those cases
- erts_bs_get_integer_2() can only fail with badmatch, so that is the
Erlang-level result of oversize bignums from binary matches
- big_SUITE.erl is extended with a test case that fails without this
fix (no error signalled) and passes with it (badmatch occurs)
Credit goes to Nico Kruber for the initial bug report.
|
|
* sverk/yielding-distr-send/OTP-12232:
erts: Add constant TERM_TO_BINARY_MEMCPY_FACTOR
erts: Optimize some repeated calls to {E,W}STACK_PUSH
erts: Yield in term_to_binary when encoding big maps
erts: Remove unnecessary goto for fun encoding
erts: Yield in term_to_binary while copying large binaries
erts: Implement yielding for distributed send of large messages
|
|
and do not piggyback on B2T_MEMCPY_FACTOR
|
|
* sverk/hipe-inline-reserve-trap-frame:
erts: Extend usage of ASM macro to avoid including asm macros in C code
erts: Make hipe_{un}reserve_beam_trap_frame INLINE
|
|
|
|
except the reference counter 'refc', as different callers
have different strategies regarding the lifetime of the binary.
|
|
|
|
|
|
|
|
|
|
Applies also to distributed send.
|
|
Use same mechanism as term_to_binary to yield
while encoding large messages for distributed send.
|
|
Replace the 'erlang:binary_to_term/1' and 'erlang:binary_to_term/2'
Erlang wrappers taking care of failure after yield with management
of this in the hidden yield BIF.
|
|
|
|
to be: 116,Arity, K1,V1,K2,V2,...,Kn,Vn
instead of: 116,Arity, K1,K2,...,Kn, V1,V2,....,Vn
We think this will be better for future internal map structures
like HAMT. Would be bad if we need to iterate twice over HAMT
in term_to_binary, one for keys and one for values.
|
|
* sverk/hipe-disable-gc-bug:
erts: Fix heap overwrite by hipe "trap frames" when GC is disabled
ASSERT that GC is not tried with "need" when GC is disabled
|
|
by trapping BIFs like term_to_binary and binary_to_term.
|
|
This change was triggered by the OSE float printing function
not working exactly the same way as linux/win32. But it is
also a good one in general as it cuts size in more than half
for floats.
|
|
'maps_head' was not restored when yielding. Risk for crash increases with
size and number of maps in term.
|
|
erl_drv_output_term() and erl_drv_send_term() can send messages
containing maps with the use of the new ERL_DRV_MAP.
The driver API minor version is updated as new functionality is added.
|
|
Maps may be encoded with keys in arbitrary order. This is fine,
as long as keys are unique.
|
|
* erlang:term_to_binary/1
* erlang:binary_to_term/1
|
|
|
|
* sverk/term2bin-simplify:
erts: Refactor ESTACK & WSTACK to use a struct easy to "export"
erts: Fix benign ESTACK/WSTACK typo
erts: Fix compiler warnings for NO_JUMP_TABLE
erts: Run binary_SUITE:trapping even for 32bit
erts: Extend binary_SUITE:ttb_trap to also cover binary_to_term
erts: Remove the extra_root feature from the process structure
erts: Simplify term_to_binary by removing saved ESTACK from root set
|
|
This is not a clean refactor. It changes the behaviour slightly
of E/WSTACK_RESTORE. The allocated stack from E/WSTACK_SAVE is used
as-is and not copied into default_stack. This will hopefully fix an
illusive memory leak that valgrind is reporting.
|
|
We disabled GC (in 522a29666088d5) during trapping and don't need to
include the saved ESTACK as part of root set.
|
|
for 32-bit integers (INTEGER_EXT) on 64-bit architectures.
|
|
* sverk/trapping-bin2term:
erts: Adjust term_to_binary reduction factors
erts: Yield after trapping term_to_binary if gc has been ordered
erts: Let term_to_binary disable gc while trapping
erts: Improve stress of binary_to_term in binary_SUITE
erts: Fix bug in binary_to_term for compressed on halfword
erts: Fix crash when binary_to_term throws badarg
erts: Trapping memcpy in binary_to_term
erts: Cleanup code for trapping binary_to_term
erts: Add erlang wrappers to binary_to_term
trapping uncompress
trapping size calculation
trapping binary_to_term/2
trapping STRING_EXT
trapping lists and tuples
trapping binary_to_term passing binary_SUITE
Parallel check_process_code when code_server purge a module
Functionality for disabling garbage collection
Use asynchronous check_process_code in code_parallel_SUITE
Execution of system tasks in context of another process
Conflicts:
erts/emulator/beam/external.c
erts/emulator/beam/sys.h
erts/emulator/test/binary_SUITE.erl
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/erts_internal.beam
|
|
Made them powers of 2 for faster calculations.
500 encoded terms per reductions seemed a bit much, lowered to 32.
TERM_TO_BINARY_SIZE_FACTOR was not used in practice as it was only
applied to small binaries.
Lowered from 500kb to 256kb compressed bytes per trap call.
|
|
or if "too much" offheap binaries has been built
|
|
as an attempt to improve performance
|
|
|
|
after it has built off_heap data
and then done at least one trap call.
The undo mechanism in dec_term does not work
if we build the magic binary after any other off_heap data.
|
|
|
|
|
|
to not expose the trapping BIF in the stacktrace
when it throws badarg.
|
|
|
|
|
