aboutsummaryrefslogtreecommitdiffstats
path: root/erts/emulator/beam/external.c
AgeCommit message (Collapse)Author
2015-06-18Change license text to APLv2Bruce Yinhe
2015-06-15erts: Remove hashmap probabilistic heap overestimationSverker Eriksson
by adding a dynamic heap factory. "binary_to_term" is now a hybrid solution with both a call to decoded_size() to calculate needed heap space AND possible dynamic allocation of more heap space if needed for big maps. The heap size returned from decoded_size() is guaranteed to be sufficient for all term heap data except for hashmap nodes. All hashmap nodes are created at the end of dec_term() by invoking the heap factory interface that may allocate more heap space on process heap or in fragments. With this commit it is no longer guaranteed that a message is confined to only one heap fragment.
2015-03-30erts: Refactor encode_size_struct_intSverker Eriksson
to handle the "start of list" case in one place and not seven. Note that this commit reverts (47d6fd3ccf35) back to using WSTACK and pushing raw pointers. We disable GC while yielding, so this should not be a problem.
2015-03-26Merge branch 'sverk/hamt-encode-size-bug/OTP-12585'Sverker Eriksson
* sverk/hamt-encode-size-bug/OTP-12585: erts: Fix bug in term_to_binary size estimation for hamt erts: Optimize term_to_binary size estimation
2015-03-25erts: Fix bug in term_to_binary size estimation for hamtSverker Eriksson
2015-03-25erts: Optimize term_to_binary size estimationSverker Eriksson
for tuples and maps containing ascii strings (lists).
2015-03-25erts: Combine flat and hash maps under one unifying tagBjörn-Egil Dahlberg
2015-03-24erts: Remove HAMT_SUBTAG_NODE_ARRAYSverker Eriksson
This will also fix a bug in term_to_binary treating full nodes as tuples and emiting LIST_EXT for leafs.
2015-03-24erts: Fix bug in binary_to_term for hamt when yieldingSverker Eriksson
Must save hamt_list in context.
2015-03-20Merge branch 'sverk/dec_term-bin-overhead/OTP-12554'Sverker Eriksson
* sverk/dec_term-bin-overhead/OTP-12554: erts: Add missing binary offheap overhead in binary_to_term
2015-03-19erts: Fix bug in binary_to_term with more than one big mapSverker Eriksson
2015-03-19erts: Fix map bug in dec_term for 32-bit debug VMSverker Eriksson
Adding ERTS_SWORD_MAX to a pointer does not work as a way to disable a bound check. Remove the hp_end from ErtsHeapFactory as it isn't really used anyway.
2015-03-12erts: Refactor maps naming conventionSverker Eriksson
flatmap: Small map hashmap: Large map map: flatmap or hashmap
2015-03-12erts: Tweak over estimation of hashmap size for binary_to_termSverker Eriksson
Strategy: Calculate an over estimation of heap size that will give such a low probability for overflow, that "it will not happen". Scary assumption 1: Uniformly distributed hash values. Scary assumption 2: Tree size is normally distributed (right?)
2015-03-12erts: Reject duplicate keys for hamt in binary_to_termSverker Eriksson
2015-03-12First stab at binary_to_term for hamtSverker Eriksson
with over estimation of heap size.
2015-03-12erts: Remove erl_hashmap.[ch] filesBjörn-Egil Dahlberg
2015-03-12erts: Refactor erl_hashmap header includesBjörn-Egil Dahlberg
2015-03-12erts: Add term_to_binary for hashmapSverker Eriksson
2015-03-12erts: Add missing binary offheap overhead in binary_to_termSverker Eriksson
Binary offheap overhead is used to trigger GC when a process is referring "too much" binary offheap data. Offheap binaries created from external format (binary_to_term, distributed messages or compacted ets tables) were not accounted for. Example: A process receiving a lot of binary data in distributed messages, while not building much terms on its heap, could cause an extensive memory consumption for garbage binaries.
2015-03-12Merge branch 'maint'Henrik Nord
Conflicts: erts/emulator/hipe/hipe_bif0.c
2015-02-04don't create oversize bignums in binary matchingMikael Pettersson
Bignums are artifically restricted in size. Arithmetic and logical operations check the sizes of resulting bignums, and turn oversize results into system_limit exceptions. However, this check is not performed when bignums are constructed by binary matching. The consequence is that such matchings can construct oversize bignums that satisfy is_integer/1 yet don't work. Performing arithmetic such as Term - 0 fails with a system_limit exception. Worse, performing a logical operation such as Term band Term results in []. The latter occurs because the size checking (e.g. in erts_band()) is a simple ASSERT(is_not_nil(...)) on the result of the bignum operation, which internally is [] (NIL) in the case of oversize results. However, ASSERT is a no-op in release builds, so the error goes unnoticed and [] is returned as the result of the band/2. This patch addresses this by preventing oversize bignums from entering the VM via binary matching: - the internal bytes_to_big() procedure is augmented to return NIL for oversize results, just like big_norm() - callers of bytes_to_big() are augmented to check for NIL returns and signal errors in those cases - erts_bs_get_integer_2() can only fail with badmatch, so that is the Erlang-level result of oversize bignums from binary matches - big_SUITE.erl is extended with a test case that fails without this fix (no error signalled) and passes with it (badmatch occurs) Credit goes to Nico Kruber for the initial bug report.
2014-11-03Merge branch 'sverk/yielding-distr-send/OTP-12232'Sverker Eriksson
* sverk/yielding-distr-send/OTP-12232: erts: Add constant TERM_TO_BINARY_MEMCPY_FACTOR erts: Optimize some repeated calls to {E,W}STACK_PUSH erts: Yield in term_to_binary when encoding big maps erts: Remove unnecessary goto for fun encoding erts: Yield in term_to_binary while copying large binaries erts: Implement yielding for distributed send of large messages
2014-11-03erts: Add constant TERM_TO_BINARY_MEMCPY_FACTORSverker Eriksson
and do not piggyback on B2T_MEMCPY_FACTOR
2014-10-24Merge branch 'sverk/hipe-inline-reserve-trap-frame'Sverker Eriksson
* sverk/hipe-inline-reserve-trap-frame: erts: Extend usage of ASM macro to avoid including asm macros in C code erts: Make hipe_{un}reserve_beam_trap_frame INLINE
2014-10-24erts: Make hipe_{un}reserve_beam_trap_frame INLINESverker Eriksson
2014-09-04erts: Refactor binary allocation interface to also initialize BinarySverker Eriksson
except the reference counter 'refc', as different callers have different strategies regarding the lifetime of the binary.
2014-09-03erts: Fix bug in term_to_binary that reallocates binary with wrong sizeSverker Eriksson
2014-08-29erts: Optimize some repeated calls to {E,W}STACK_PUSHSverker Eriksson
2014-08-29erts: Yield in term_to_binary when encoding big mapsSverker Eriksson
2014-08-29erts: Remove unnecessary goto for fun encodingSverker Eriksson
2014-08-29erts: Yield in term_to_binary while copying large binariesSverker Eriksson
Applies also to distributed send.
2014-08-29erts: Implement yielding for distributed send of large messagesSverker Eriksson
Use same mechanism as term_to_binary to yield while encoding large messages for distributed send.
2014-06-05Replace erlang:binary_to_term() Erlang wrappersRickard Green
Replace the 'erlang:binary_to_term/1' and 'erlang:binary_to_term/2' Erlang wrappers taking care of failure after yield with management of this in the hidden yield BIF.
2014-05-13HiPE wrappers for BIFs disabling GCSverker Eriksson
2014-03-14erts: Change external format for mapsSverker Eriksson
to be: 116,Arity, K1,V1,K2,V2,...,Kn,Vn instead of: 116,Arity, K1,K2,...,Kn, V1,V2,....,Vn We think this will be better for future internal map structures like HAMT. Would be bad if we need to iterate twice over HAMT in term_to_binary, one for keys and one for values.
2014-02-25Merge branch 'sverk/hipe-disable-gc-bug'Sverker Eriksson
* sverk/hipe-disable-gc-bug: erts: Fix heap overwrite by hipe "trap frames" when GC is disabled ASSERT that GC is not tried with "need" when GC is disabled
2014-02-25erts: Fix heap overwrite by hipe "trap frames" when GC is disabledSverker Eriksson
by trapping BIFs like term_to_binary and binary_to_term.
2014-02-24erts: Set default external enc to use new float schemeLukas Larsson
This change was triggered by the OSE float printing function not working exactly the same way as linux/win32. But it is also a good one in general as it cuts size in more than half for floats.
2014-02-03erts: Fix bug in binary_to_term for mapsSverker Eriksson
'maps_head' was not restored when yielding. Risk for crash increases with size and number of maps in term.
2014-01-29erts: Add map construction to driver APISverker Eriksson
erl_drv_output_term() and erl_drv_send_term() can send messages containing maps with the use of the new ERL_DRV_MAP. The driver API minor version is updated as new functionality is added.
2014-01-29erts: Let erlang:binary_to_term/1 handle unsorted MapsBjörn-Egil Dahlberg
Maps may be encoded with keys in arbitrary order. This is fine, as long as keys are unique.
2014-01-29erts: Update maps serializing to use term orderBjörn-Egil Dahlberg
* erlang:term_to_binary/1 * erlang:binary_to_term/1
2014-01-28erts: Teach term_to_binary/1 and binary_to_term/1 Map encodingBjörn-Egil Dahlberg
2014-01-20Merge branch 'sverk/term2bin-simplify'Sverker Eriksson
* sverk/term2bin-simplify: erts: Refactor ESTACK & WSTACK to use a struct easy to "export" erts: Fix benign ESTACK/WSTACK typo erts: Fix compiler warnings for NO_JUMP_TABLE erts: Run binary_SUITE:trapping even for 32bit erts: Extend binary_SUITE:ttb_trap to also cover binary_to_term erts: Remove the extra_root feature from the process structure erts: Simplify term_to_binary by removing saved ESTACK from root set
2014-01-17erts: Refactor ESTACK & WSTACK to use a struct easy to "export"Sverker Eriksson
This is not a clean refactor. It changes the behaviour slightly of E/WSTACK_RESTORE. The allocated stack from E/WSTACK_SAVE is used as-is and not copied into default_stack. This will hopefully fix an illusive memory leak that valgrind is reporting.
2014-01-10erts: Simplify term_to_binary by removing saved ESTACK from root setSverker Eriksson
We disabled GC (in 522a29666088d5) during trapping and don't need to include the saved ESTACK as part of root set.
2014-01-08erts: Remove overestimation of heap space in binary_to_termSverker Eriksson
for 32-bit integers (INTEGER_EXT) on 64-bit architectures.
2013-12-12Merge branch 'sverk/trapping-bin2term' into OTP_R16B03Sverker Eriksson
* sverk/trapping-bin2term: erts: Adjust term_to_binary reduction factors erts: Yield after trapping term_to_binary if gc has been ordered erts: Let term_to_binary disable gc while trapping erts: Improve stress of binary_to_term in binary_SUITE erts: Fix bug in binary_to_term for compressed on halfword erts: Fix crash when binary_to_term throws badarg erts: Trapping memcpy in binary_to_term erts: Cleanup code for trapping binary_to_term erts: Add erlang wrappers to binary_to_term trapping uncompress trapping size calculation trapping binary_to_term/2 trapping STRING_EXT trapping lists and tuples trapping binary_to_term passing binary_SUITE Parallel check_process_code when code_server purge a module Functionality for disabling garbage collection Use asynchronous check_process_code in code_parallel_SUITE Execution of system tasks in context of another process Conflicts: erts/emulator/beam/external.c erts/emulator/beam/sys.h erts/emulator/test/binary_SUITE.erl erts/preloaded/ebin/erlang.beam erts/preloaded/ebin/erts_internal.beam
2013-11-21erts: Adjust term_to_binary reduction factorsR16B02_yielding_binary_to_termSverker Eriksson
Made them powers of 2 for faster calculations. 500 encoded terms per reductions seemed a bit much, lowered to 32. TERM_TO_BINARY_SIZE_FACTOR was not used in practice as it was only applied to small binaries. Lowered from 500kb to 256kb compressed bytes per trap call.