beam_validator: fix incorrect assumptions about GC guard BIFs

The beam_validator pass incorrectly assumes that a GC guard
BIF (such as length/1) may first do a garbage collection
and then fail. That assumption is not correct (guards BIF
only do garbage collection when it is known that the BIF
call will succeed), and will cause the compiler to reject
valid programs.

Modify the beam_validator to assume that if the branch is
taken for a gc_bif instruction, all registers are unchanged
and no garbage collection has occurred. Also add a comment
in the emulator about that assumption.

1 files changed, 4 insertions, 0 deletions

diff --git a/erts/emulator/beam/erl_bif_guard.c b/erts/emulator/beam/erl_bif_guard.c
index 8b47db10dd..9e15f4e399 100644
--- a/erts/emulator/beam/erl_bif_guard.c
+++ b/erts/emulator/beam/erl_bif_guard.c
@@ -318,6 +318,10 @@ double_to_integer(Process* p, double x)
  * The following code is used when a guard that may build on the
  * heap is called directly. They must not use HAlloc(), but must
  * do a garbage collection if there is insufficient heap space.
+ *
+ * Important note: All error checking MUST be done before doing
+ * a garbage collection. The compiler assumes that all registers
+ * are still valid if a guard BIF generates an exception.
  */
 
 #define ERTS_NEED_GC(p, need) ((HEAP_LIMIT((p)) - HEAP_TOP((p))) <= (need))