Teach erts_bs_append not to dump core

When erts_bs_append() calls the garbage collector, it has set
the PB_ACTIVE_WRITER flag in the ProcBin for the binary object
is about to be appended to. Therefore, it is expected that the
garbage collector should neither move nor shrink the binary
object.

But if the garbage collector does a minor collection (thereby
clearing the PB_ACTIVE_WRITER flag in the ProcBin) and there is
still not enough heap space, there will be a second major
garbage collection. During the major collection (since the
the PB_ACTIVE_WRITER flag was cleared), the binary object may
be shrunk or moved (depending on sizes and how many other
writable binaries there are in the process).

Avoid the problem by clearing the PB_ACTIVE_WRITER flags in
a separate pass after the garbage collection(s).

Thanks to Denis Titoruk for helping us find this bug.

1 files changed, 14 insertions, 1 deletions

diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c
index 52a6e52e6c..e2689f58c3 100644
--- a/erts/emulator/beam/erl_gc.c
+++ b/erts/emulator/beam/erl_gc.c
@@ -336,6 +336,19 @@ erts_gc_after_bif_call(Process* p, Eterm result, Eterm* regs, Uint arity)
     return result;
 }
 
+static ERTS_INLINE void reset_active_writer(Process *p)
+{
+    struct erl_off_heap_header* ptr;
+    ptr = MSO(p).first;
+    while (ptr) {
+	if (ptr->thing_word == HEADER_PROC_BIN) {	
+	    ProcBin *pbp = (ProcBin*) ptr;
+	    pbp->flags &= ~PB_ACTIVE_WRITER;
+	}
+	ptr = ptr->next;
+    }
+}
+
 /*
  * Garbage collect a process.
  *
@@ -395,6 +408,7 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj)
 	    DTRACE2(gc_minor_end, pidbuf, reclaimed_now);
 	}
     }
+    reset_active_writer(p);
 
     /*
      * Finish.
@@ -2181,7 +2195,6 @@ link_live_proc_bin(struct shrink_cand_data *shrink,
 
 
 	if (pbp->flags & PB_ACTIVE_WRITER) {
-	    pbp->flags &= ~PB_ACTIVE_WRITER;
 	    shrink->no_of_active++;
 	}
 	else { /* inactive */