aboutsummaryrefslogtreecommitdiffstats
path: root/erts/emulator/beam/external.c
AgeCommit message (Collapse)Author
2015-02-04don't create oversize bignums in binary matchingMikael Pettersson
Bignums are artifically restricted in size. Arithmetic and logical operations check the sizes of resulting bignums, and turn oversize results into system_limit exceptions. However, this check is not performed when bignums are constructed by binary matching. The consequence is that such matchings can construct oversize bignums that satisfy is_integer/1 yet don't work. Performing arithmetic such as Term - 0 fails with a system_limit exception. Worse, performing a logical operation such as Term band Term results in []. The latter occurs because the size checking (e.g. in erts_band()) is a simple ASSERT(is_not_nil(...)) on the result of the bignum operation, which internally is [] (NIL) in the case of oversize results. However, ASSERT is a no-op in release builds, so the error goes unnoticed and [] is returned as the result of the band/2. This patch addresses this by preventing oversize bignums from entering the VM via binary matching: - the internal bytes_to_big() procedure is augmented to return NIL for oversize results, just like big_norm() - callers of bytes_to_big() are augmented to check for NIL returns and signal errors in those cases - erts_bs_get_integer_2() can only fail with badmatch, so that is the Erlang-level result of oversize bignums from binary matches - big_SUITE.erl is extended with a test case that fails without this fix (no error signalled) and passes with it (badmatch occurs) Credit goes to Nico Kruber for the initial bug report.
2014-09-03erts: Fix bug in term_to_binary that reallocates binary with wrong sizeSverker Eriksson
2014-06-05Replace erlang:binary_to_term() Erlang wrappersRickard Green
Replace the 'erlang:binary_to_term/1' and 'erlang:binary_to_term/2' Erlang wrappers taking care of failure after yield with management of this in the hidden yield BIF.
2014-05-13HiPE wrappers for BIFs disabling GCSverker Eriksson
2014-03-14erts: Change external format for mapsSverker Eriksson
to be: 116,Arity, K1,V1,K2,V2,...,Kn,Vn instead of: 116,Arity, K1,K2,...,Kn, V1,V2,....,Vn We think this will be better for future internal map structures like HAMT. Would be bad if we need to iterate twice over HAMT in term_to_binary, one for keys and one for values.
2014-02-25Merge branch 'sverk/hipe-disable-gc-bug'Sverker Eriksson
* sverk/hipe-disable-gc-bug: erts: Fix heap overwrite by hipe "trap frames" when GC is disabled ASSERT that GC is not tried with "need" when GC is disabled
2014-02-25erts: Fix heap overwrite by hipe "trap frames" when GC is disabledSverker Eriksson
by trapping BIFs like term_to_binary and binary_to_term.
2014-02-24erts: Set default external enc to use new float schemeLukas Larsson
This change was triggered by the OSE float printing function not working exactly the same way as linux/win32. But it is also a good one in general as it cuts size in more than half for floats.
2014-02-03erts: Fix bug in binary_to_term for mapsSverker Eriksson
'maps_head' was not restored when yielding. Risk for crash increases with size and number of maps in term.
2014-01-29erts: Add map construction to driver APISverker Eriksson
erl_drv_output_term() and erl_drv_send_term() can send messages containing maps with the use of the new ERL_DRV_MAP. The driver API minor version is updated as new functionality is added.
2014-01-29erts: Let erlang:binary_to_term/1 handle unsorted MapsBjörn-Egil Dahlberg
Maps may be encoded with keys in arbitrary order. This is fine, as long as keys are unique.
2014-01-29erts: Update maps serializing to use term orderBjörn-Egil Dahlberg
* erlang:term_to_binary/1 * erlang:binary_to_term/1
2014-01-28erts: Teach term_to_binary/1 and binary_to_term/1 Map encodingBjörn-Egil Dahlberg
2014-01-20Merge branch 'sverk/term2bin-simplify'Sverker Eriksson
* sverk/term2bin-simplify: erts: Refactor ESTACK & WSTACK to use a struct easy to "export" erts: Fix benign ESTACK/WSTACK typo erts: Fix compiler warnings for NO_JUMP_TABLE erts: Run binary_SUITE:trapping even for 32bit erts: Extend binary_SUITE:ttb_trap to also cover binary_to_term erts: Remove the extra_root feature from the process structure erts: Simplify term_to_binary by removing saved ESTACK from root set
2014-01-17erts: Refactor ESTACK & WSTACK to use a struct easy to "export"Sverker Eriksson
This is not a clean refactor. It changes the behaviour slightly of E/WSTACK_RESTORE. The allocated stack from E/WSTACK_SAVE is used as-is and not copied into default_stack. This will hopefully fix an illusive memory leak that valgrind is reporting.
2014-01-10erts: Simplify term_to_binary by removing saved ESTACK from root setSverker Eriksson
We disabled GC (in 522a29666088d5) during trapping and don't need to include the saved ESTACK as part of root set.
2014-01-08erts: Remove overestimation of heap space in binary_to_termSverker Eriksson
for 32-bit integers (INTEGER_EXT) on 64-bit architectures.
2013-12-12Merge branch 'sverk/trapping-bin2term' into OTP_R16B03Sverker Eriksson
* sverk/trapping-bin2term: erts: Adjust term_to_binary reduction factors erts: Yield after trapping term_to_binary if gc has been ordered erts: Let term_to_binary disable gc while trapping erts: Improve stress of binary_to_term in binary_SUITE erts: Fix bug in binary_to_term for compressed on halfword erts: Fix crash when binary_to_term throws badarg erts: Trapping memcpy in binary_to_term erts: Cleanup code for trapping binary_to_term erts: Add erlang wrappers to binary_to_term trapping uncompress trapping size calculation trapping binary_to_term/2 trapping STRING_EXT trapping lists and tuples trapping binary_to_term passing binary_SUITE Parallel check_process_code when code_server purge a module Functionality for disabling garbage collection Use asynchronous check_process_code in code_parallel_SUITE Execution of system tasks in context of another process Conflicts: erts/emulator/beam/external.c erts/emulator/beam/sys.h erts/emulator/test/binary_SUITE.erl erts/preloaded/ebin/erlang.beam erts/preloaded/ebin/erts_internal.beam
2013-11-21erts: Adjust term_to_binary reduction factorsR16B02_yielding_binary_to_termSverker Eriksson
Made them powers of 2 for faster calculations. 500 encoded terms per reductions seemed a bit much, lowered to 32. TERM_TO_BINARY_SIZE_FACTOR was not used in practice as it was only applied to small binaries. Lowered from 500kb to 256kb compressed bytes per trap call.
2013-11-21erts: Yield after trapping term_to_binary if gc has been orderedSverker Eriksson
or if "too much" offheap binaries has been built
2013-11-21erts: Let term_to_binary disable gc while trappingSverker Eriksson
as an attempt to improve performance
2013-11-18erts: Fix bug in binary_to_term for compressed on halfwordSverker Eriksson
2013-11-18erts: Fix crash when binary_to_term throws badargSverker Eriksson
after it has built off_heap data and then done at least one trap call. The undo mechanism in dec_term does not work if we build the magic binary after any other off_heap data.
2013-11-18erts: Trapping memcpy in binary_to_termSverker Eriksson
2013-11-18erts: Cleanup code for trapping binary_to_termSverker Eriksson
2013-11-18erts: Add erlang wrappers to binary_to_termSverker Eriksson
to not expose the trapping BIF in the stacktrace when it throws badarg.
2013-11-18trapping uncompressSverker Eriksson
2013-11-18trapping size calculationSverker Eriksson
2013-11-18trapping binary_to_term/2Sverker Eriksson
2013-11-18trapping STRING_EXTSverker Eriksson
2013-11-18trapping lists and tuplesSverker Eriksson
2013-11-18trapping binary_to_term passing binary_SUITESverker Eriksson
2013-11-18erts: Fix bug in binary_to_term for binaries larger than 2^31Sverker Eriksson
2013-11-18erts: Fix bugs in binary_to_term for invalid bitstringsSverker Eriksson
<<131, 77, Len:32, Bits:8, Data/binary>> badarg if Bits > 8 Used to return internally inconsistent bitstring badarg if Len==0 and Bits > 0 Used to return invalid *huge* binary (size = (Uint)-1) badarg if Bits==0 and Len > 0 Used to return valid binary as if Bits was 8
2013-07-11Merge branch 'jw/fix-float-middle-endian/OTP-11201' into maintFredrik Gustafsson
* jw/fix-float-middle-endian/OTP-11201: Fix binary construction on floating point middle-endian machines Fix binary matching on floating point middle-endian machines Fix erlang:phash2() on floating point middle-endian machines Fix external term format BIFs on floating point middle-endian machines
2013-06-14erts: Fix memory leak in term_to_binary/2 with compressed optionSverker Eriksson
2013-06-14Fix external term format BIFs on floating point middle-endian machinesJohannes Weißl
This complements 933e701 (OTP-10209). Simple error example: 1> <<131,70,63,240,0,0,0,0,0,0>> = term_to_binary(1.0, [{minor_version,1}]). ** exception error: no match of right hand side value <<131,70,0,0,0,0,63,240,0,0>> 2> 1.0 = binary_to_term(<<131,70,63,240,0,0,0,0,0,0>>). ** exception error: no match of right hand side value 5.299808824e-315 But roundtrip always works: 3> 1.0 = binary_to_term(term_to_binary(1.0, [{minor_version,1}])). 1.0
2013-06-10term_to_binary: Remove debug code and set production trap levelsPatrik Nyblom
2013-06-06Teach erl_gc:offset_rootset about extra_rootPatrik Nyblom
2013-06-05Teach external.c to handle reallocs before compressionPatrik Nyblom
2013-06-05Make all steps ofterm_to_binary work in chunks and yieldPatrik Nyblom
Rewrite and extend of Happi's initial work Extra_root to process structure to enable GC of state - Changed the process structure to point to a separate struct, the struct also contains a destructor function to allow for proper cleanup. Rewrote encode_size_struct and enc_term to have internal versions with reduction counters which will result in interrupt for later restart when the counter reaches zero - removed the EWA_STACK from Happis version and directly save the ESTACK's and WSTACK's in the above mentioned struct (or array thereof) that are pointed out from the process structure. The destructor will take care of the deallocation in case of process death. Added ESTACK and WSTACK macros to save and restore stack and to change allocator, which makes the previously mentioned stack-save work. Rewrote enc_term to not store pointers on the stack, and use one WSTACK for commands etc and another ESTACK for Eterms - Slightly different than Happis version to make halfword code simpler. Rewrote encode_size_struct2 so that it does not store pointers on the stack, also switched to ESTACK instead of WSTACK, this also handles halfword correctly. Added interfaces for chunkwise compression, that are used from term_to_binary/2 when the compressed option is given.
2013-06-04Make term_to_binary yield (trap).happi
2013-04-15erts: Remove unnecessary utf8 analysis step in external decodingSverker Eriksson
Faulty utf8 atoms are rejected anyway later when the atom is created.
2013-01-28Merge branch 'sverk/enc_atom-opt'Sverker Eriksson
* sverk/enc_atom-opt: erts: Optimize atom encoding to use memcpy for pure ascii erts: Refactor erts_atom_get to use ErtsAtomEncoding
2013-01-25Merge branch 'pan/fix-compiler-warnings-clang-and-new-gcc'Patrik Nyblom
* pan/fix-compiler-warnings-clang-and-new-gcc: Fix compiler warnings from GCC 4.7.1 on ARCH Linux Fix clang compiler warnings on FreeBSD in erts
2013-01-25Update copyright yearsBjörn-Egil Dahlberg
2013-01-25erts: Optimize atom encoding to use memcpy for pure asciiSverker Eriksson
2013-01-25erts: Refactor erts_atom_get to use ErtsAtomEncodingSverker Eriksson
instead of 'is_latin1' boolean argument.
2013-01-23Fix clang compiler warnings on FreeBSD in ertsPatrik Nyblom
The following are deliberately left, as I have only a list of compiler warnings and no system to test on: hipe/hipe_x86_signal.c:264:5: warning: no previous prototype for function '_sigaction' [-Wmissing-prototypes] int __SIGACTION(int signum, const struct sigaction *act, struct sigaction *oldact) ^ hipe/hipe_x86_signal.c:222:21: note: expanded from macro '__SIGACTION' ^ 1 warning generated. sys/unix/sys_float.c:835:16: warning: declaration of 'struct exception' will not be visible outside of this function [-Wvisibility] matherr(struct exception *exc) ^ sys/unix/sys_float.c:835:1: warning: no previous prototype for function 'matherr' [-Wmissing-prototypes] matherr(struct exception *exc) ^ 2 warnings generated. drivers/unix/unix_efile.c:1504:11: warning: implicit declaration of function 'sendfile' [-Wimplicit-function-declaration] retval = sendfile(in_fd, out_fd, *offset, SENDFILE_CHUNK_SIZE, ^ 1 warning generated.
2013-01-16UTF-8 support for distributionRickard Green