don't create oversize bignums in binary matching

Bignums are artifically restricted in size.  Arithmetic and logical
operations check the sizes of resulting bignums, and turn oversize
results into system_limit exceptions.

However, this check is not performed when bignums are constructed by
binary matching.  The consequence is that such matchings can construct
oversize bignums that satisfy is_integer/1 yet don't work.  Performing
arithmetic such as Term - 0 fails with a system_limit exception.  Worse,
performing a logical operation such as Term band Term results in [].
The latter occurs because the size checking (e.g. in erts_band()) is
a simple ASSERT(is_not_nil(...)) on the result of the bignum operation,
which internally is [] (NIL) in the case of oversize results.  However,
ASSERT is a no-op in release builds, so the error goes unnoticed and []
is returned as the result of the band/2.

This patch addresses this by preventing oversize bignums from entering
the VM via binary matching:

- the internal bytes_to_big() procedure is augmented to return NIL for
  oversize results, just like big_norm()
- callers of bytes_to_big() are augmented to check for NIL returns and
  signal errors in those cases
- erts_bs_get_integer_2() can only fail with badmatch, so that is the
  Erlang-level result of oversize bignums from binary matches
- big_SUITE.erl is extended with a test case that fails without this
  fix (no error signalled) and passes with it (badmatch occurs)

Credit goes to Nico Kruber for the initial bug report.

1 files changed, 12 insertions, 2 deletions

diff --git a/erts/emulator/test/big_SUITE.erl b/erts/emulator/test/big_SUITE.erl
index 413bd3bcae..3193d56e2a 100644
--- a/erts/emulator/test/big_SUITE.erl
+++ b/erts/emulator/test/big_SUITE.erl
@@ -23,7 +23,7 @@
 	 init_per_group/2,end_per_group/2]).
 -export([t_div/1, eq_28/1, eq_32/1, eq_big/1, eq_math/1, big_literals/1,
 	 borders/1, negative/1, big_float_1/1, big_float_2/1,
-	 shift_limit_1/1, powmod/1, system_limit/1, otp_6692/1]).
+	 shift_limit_1/1, powmod/1, system_limit/1, toobig/1, otp_6692/1]).
 
 %% Internal exports.
 -export([eval/1]).
@@ -40,7 +40,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}].
 all() -> 
     [t_div, eq_28, eq_32, eq_big, eq_math, big_literals,
      borders, negative, {group, big_float}, shift_limit_1,
-     powmod, system_limit, otp_6692].
+     powmod, system_limit, toobig, otp_6692].
 
 groups() -> 
     [{big_float, [], [big_float_1, big_float_2]}].
@@ -370,6 +370,16 @@ maxbig() ->
 
 id(I) -> I.
 
+toobig(Config) when is_list(Config) ->
+    ?line {'EXIT',{{badmatch,_},_}} = (catch toobig()),
+    ok.
+
+toobig() ->
+    A = erlang:term_to_binary(lists:seq(1000000, 2200000)),
+    ASize = erlang:bit_size(A),
+    <<ANr:ASize>> = A, % should fail
+    ANr band ANr.
+
 otp_6692(suite) ->
     [];
 otp_6692(doc) ->