Replace io_list_len() with erts_iolist_size()

The io_list_len() function returns an int, where a negative return
value indicates a type error. One problem is that an int only consists
of 32 bits in a 64-bit emulator. Changing the return type to Sint
will solve that problem, but in the 32-bit emulator, a large iolist
and a iolist with a type error will both return a negative number.
(Noticed by Jon Meredith.)

Another problem is that for iolists whose total size exceed the
word size, the result would be truncated, leading to a subsequent
buffer overflow and emulator crash.

Therefore, introduce the new erts_iolist_size() function which
returns a status indication and writes the result size through
a passed pointer. If the result size does not fit in a word,
return an overflow indication.

1 files changed, 5 insertions, 4 deletions

diff --git a/erts/emulator/beam/erl_bif_port.c b/erts/emulator/beam/erl_bif_port.c
index fbc92b9730..3fd35dd963 100644
--- a/erts/emulator/beam/erl_bif_port.c
+++ b/erts/emulator/beam/erl_bif_port.c
@@ -1,7 +1,7 @@
 /*
  * %CopyrightBegin%
  *
- * Copyright Ericsson AB 2001-2010. All Rights Reserved.
+ * Copyright Ericsson AB 2001-2011. All Rights Reserved.
  *
  * The contents of this file are subject to the Erlang Public License,
  * Version 1.1, (the "License"); you may not use this file except in
@@ -996,6 +996,7 @@ static byte* convert_environment(Process* p, Eterm env)
     Eterm* hp;
     Uint heap_size;
     int n;
+    Uint size;
     byte* bytes;
 
     if ((n = list_length(env)) < 0) {
@@ -1039,15 +1040,15 @@ static byte* convert_environment(Process* p, Eterm env)
     if (is_not_nil(env)) {
 	goto done;
     }
-    if ((n = io_list_len(all)) < 0) {
+    if (erts_iolist_size(all, &size)) {
 	goto done;
     }
 
     /*
      * Put the result in a binary (no risk for a memory leak that way).
      */
-    (void) erts_new_heap_binary(p, NULL, n, &bytes);
-    io_list_to_buf(all, (char*)bytes, n);
+    (void) erts_new_heap_binary(p, NULL, size, &bytes);
+    io_list_to_buf(all, (char*)bytes, size);
 
  done:
     erts_free(ERTS_ALC_T_TMP, temp_heap);