Merge branch 'maint'

* maint:
  Add clearer warnings about misuse of NIF and driver functionality
  Fix erlang:memory(ets)

Conflicts:
	erts/emulator/beam/erl_db.c

4 files changed, 138 insertions, 43 deletions

diff --git a/erts/doc/src/driver_entry.xml b/erts/doc/src/driver_entry.xml
index a2efdf3ebc..929c485c36 100644
--- a/erts/doc/src/driver_entry.xml
+++ b/erts/doc/src/driver_entry.xml
@@ -34,6 +34,29 @@
   <lib>driver_entry</lib>
   <libsummary>The driver-entry structure used by erlang drivers.</libsummary>
   <description>
+    <marker id="WARNING"/>
+      <warning><p><em>Use this functionality with extreme care!</em></p>
+      <p>A driver callback is executed as a direct extension of the
+      native code of the VM. Execution is not made in a safe environment.
+      The VM can <em>not</em> provide the same services as provided when
+      executing Erlang code, such as preemptive scheduling or memory
+      protection. If the driver callback function doesn't behave well,
+      the whole VM will misbehave.</p>
+      <list>
+	<item><p>A driver callback that crash will crash the whole VM.</p></item>
+	<item><p>An erroneously implemented driver callback might cause
+	a VM internal state inconsistency which may cause a crash of the VM,
+	or miscellaneous misbehaviors of the VM at any point after the call
+	to the driver callback.</p></item>
+	<item><p>A driver callback that do
+	<seealso marker="erl_driver#lengthy_work">lengthy work</seealso>
+	before returning will degrade responsiveness of the VM,
+	and may cause miscellaneous strange behaviors. Such strange behaviors
+	include, but are not limited to, extreme memory usage, and bad load
+	balancing between schedulers. Strange behaviors that might occur due
+	to lengthy work may also vary between OTP releases.</p></item>
+      </list>
+      </warning>
     <p>
       As of erts version 5.9 (OTP release R15B) the driver interface
       has been changed with larger types for the callbacks
diff --git a/erts/doc/src/erl_driver.xml b/erts/doc/src/erl_driver.xml
index 187c263b60..e16fd744c0 100644
--- a/erts/doc/src/erl_driver.xml
+++ b/erts/doc/src/erl_driver.xml
@@ -34,6 +34,32 @@
   <lib>erl_driver</lib>
   <libsummary>API functions for an Erlang driver</libsummary>
   <description>
+    <p>An Erlang driver is a library containing a set of native driver
+      callback functions that the Erlang VM calls when certain
+      events occur. There may be multiple instances of a driver, each
+      instance is associated with an Erlang port.</p>
+    <marker id="WARNING"/>
+      <warning><p><em>Use this functionality with extreme care!</em></p>
+      <p>A driver callback is executed as a direct extension of the
+      native code of the VM. Execution is not made in a safe environment.
+      The VM can <em>not</em> provide the same services as provided when
+      executing Erlang code, such as preemptive scheduling or memory
+      protection. If the driver callback function doesn't behave well,
+      the whole VM will misbehave.</p>
+      <list>
+	<item><p>A driver callback that crash will crash the whole VM.</p></item>
+	<item><p>An erroneously implemented driver callback might cause
+	a VM internal state inconsistency which may cause a crash of the VM,
+	or miscellaneous misbehaviors of the VM at any point after the call
+	to the driver callback.</p></item>
+	<item><p>A driver callback that do <seealso marker="#lengthy_work">lengthy
+	work</seealso> before returning will degrade responsiveness of the VM,
+	and may cause miscellaneous strange behaviors. Such strange behaviors
+	include, but are not limited to, extreme memory usage, and bad load
+	balancing between schedulers. Strange behaviors that might occur due
+	to lengthy work may also vary between OTP releases.</p></item>
+      </list>
+      </warning>
     <p>As of erts version 5.5.3 the driver interface has been extended
       (see <seealso marker="driver_entry#extended_marker">extended marker</seealso>).
       The extended interface introduce
@@ -53,16 +79,12 @@
     <p>The driver calls back to the emulator, using the API
       functions declared in <c>erl_driver.h</c>. They are used for
       outputting data from the driver, using timers, etc.</p>
-    <p>A driver is a library with a set of function that the emulator
-      calls, in response to Erlang functions and message
-      sending. There may be multiple instances of a driver, each
-      instance is connected to an Erlang port. Every port has a port
-      owner process. Communication with the port is normally done
-      through the port owner process.</p>
-    <p>Most of the functions take the <c>port</c> handle as an
-      argument. This identifies the driver instance. Note that this
-      port handle must be stored by the driver, it is not given when
-      the driver is called from the emulator (see
+    <p>Each driver instance is associated with a port. Every port
+      has a port owner process. Communication with the port is normally
+      done through the port owner process. Most of the functions take
+      the <c>port</c> handle as an argument. This identifies the driver
+      instance. Note that this port handle must be stored by the driver,
+      it is not given when the driver is called from the emulator (see
       <seealso marker="driver_entry#emulator">driver_entry</seealso>).</p>
     <p>Some of the functions take a parameter of type
       <c>ErlDrvBinary</c>, a driver binary. It should be both
@@ -129,6 +151,21 @@
         are <em>only</em> thread safe when used in a runtime
         system with SMP support.</p>
     </note>
+     <p><marker id="lengthy_work"/>
+     As mentioned in the <seealso marker="#WARNING">warning</seealso> text at
+     the beginning of this document it is of vital importance that a driver callback
+     does return relatively fast. It is hard to give an exact maximum amount
+     of time that a driver callback is allowed to work, but as a rule of thumb
+     a well behaving driver callback should return before a millisecond has
+     passed. This can be achieved using different approaches.
+     If you have full control over the code that are to execute in the driver
+     callback, the best approach is to divide the work into multiple chunks of
+     work and trigger multiple calls to the
+     <seealso marker="driver_entry#timeout">timeout callback</seealso> using
+     zero timeouts. This might, however, not always be possible, e.g. when
+     calling third party libraries. In this case you typically want to dispatch
+     the work to another thread. Information about thread primitives can be
+     found below.</p>
   </description>
 
   <section>
diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml
index f484e9eaf7..f00f7b9f46 100644
--- a/erts/doc/src/erl_nif.xml
+++ b/erts/doc/src/erl_nif.xml
@@ -34,30 +34,6 @@
   <lib>erl_nif</lib>
   <libsummary>API functions for an Erlang NIF library</libsummary>
   <description>
-     <note><p>The NIF concept is officially supported from R14B. NIF source code
-     written for earlier experimental versions might need adaption to run on R14B.</p>
-    <p>No incompatible changes between <em>R14B</em> and R14A.</p>
-    <p>Incompatible changes between <em>R14A</em> and R13B04:</p>
-  <list>
-    <item>Environment argument removed for <c>enif_alloc</c>,
-    <c>enif_realloc</c>, <c>enif_free</c>, <c>enif_alloc_binary</c>,
-    <c>enif_realloc_binary</c>, <c>enif_release_binary</c>,
-    <c>enif_alloc_resource</c>, <c>enif_release_resource</c>,
-    <c>enif_is_identical</c> and <c>enif_compare</c>.</item>
-    <item>Character encoding argument added to <c>enif_get_atom</c>
-    and <c>enif_make_existing_atom</c>.</item>
-    <item>Module argument added to <c>enif_open_resource_type</c>
-      while changing name spaces of resource types from global to module local.</item>
-  </list>
-  <p>Incompatible changes between <em>R13B04</em> and R13B03:</p>
-  <list>
-    <item>The function prototypes of the NIFs have  changed to expect <c>argc</c> and <c>argv</c>
-	  arguments. The arity of a NIF is by that no longer limited to 3.</item>
-    <item><c>enif_get_data</c> renamed as <c>enif_priv_data</c>.</item>
-    <item><c>enif_make_string</c> got a third argument for character encoding.</item>
-  </list>
-  </note>
-
     <p>A NIF library contains native implementation of some functions
     of an Erlang module. The native implemented functions (NIFs) are
     called like any other functions without any difference to the
@@ -67,6 +43,57 @@
     is to throw an exception. But it can also be used as a fallback
     implementation if the NIF library is not implemented for some
     architecture.</p>     
+    <marker id="WARNING"/>
+      <warning><p><em>Use this functionality with extreme care!</em></p>
+      <p>A native function is executed as a direct extension of the
+      native code of the VM. Execution is not made in a safe environment.
+      The VM can <em>not</em> provide the same services as provided when
+      executing Erlang code, such as preemptive scheduling or memory
+      protection. If the native function doesn't behave well, the whole
+      VM will misbehave.</p>
+      <list>
+	<item><p>A native function that crash will crash the whole VM.</p></item>
+	<item><p>An erroneously implemented native function might cause
+	a VM internal state inconsistency which may cause a crash of the VM,
+	or miscellaneous misbehaviors of the VM at any point after the call
+	to the native function.</p></item>
+	<item><p>A native function that do <seealso marker="#lengthy_work">lengthy
+	work</seealso> before returning will degrade responsiveness of the VM,
+	and may cause miscellaneous strange behaviors. Such strange behaviors
+	include, but are not limited to, extreme memory usage, and bad load
+	balancing between schedulers. Strange behaviors that might occur due
+	to lengthy work may also vary between OTP releases.</p></item>
+      </list>
+      </warning>
+
+     <p>The NIF concept is officially supported from R14B. NIF source code
+     written for earlier experimental versions might need adaption to run on R14B
+     or later versions:</p>
+     <list>
+       <item>No incompatible changes between <em>R14B</em> and R14A.</item>
+       <item>Incompatible changes between <em>R14A</em> and R13B04:
+       <list>
+	 <item>Environment argument removed for <c>enif_alloc</c>,
+	 <c>enif_realloc</c>, <c>enif_free</c>, <c>enif_alloc_binary</c>,
+	 <c>enif_realloc_binary</c>, <c>enif_release_binary</c>,
+	 <c>enif_alloc_resource</c>, <c>enif_release_resource</c>,
+	 <c>enif_is_identical</c> and <c>enif_compare</c>.</item>
+	 <item>Character encoding argument added to <c>enif_get_atom</c>
+	 and <c>enif_make_existing_atom</c>.</item>
+	 <item>Module argument added to <c>enif_open_resource_type</c>
+	 while changing name spaces of resource types from global to module local.</item>
+       </list>
+     </item>
+     <item>Incompatible changes between <em>R13B04</em> and R13B03:
+     <list>
+       <item>The function prototypes of the NIFs have  changed to expect <c>argc</c> and <c>argv</c>
+       arguments. The arity of a NIF is by that no longer limited to 3.</item>
+       <item><c>enif_get_data</c> renamed as <c>enif_priv_data</c>.</item>
+       <item><c>enif_make_string</c> got a third argument for character encoding.</item>
+     </list>
+     </item>
+    </list>
+
     <p>A minimal example of a NIF library can look like this:</p>
       <p/>
       <code type="none">
@@ -136,7 +163,23 @@ ok
      then retrieved by calling <seealso marker="#enif_priv_data">enif_priv_data</seealso>.</p>
     <p>There is no way to explicitly unload a NIF library. A library will be
      automatically unloaded when the module code that it belongs to is purged
-     by the code server.</p> 
+     by the code server.</p>
+
+     <p><marker id="lengthy_work"/>
+     As mentioned in the <seealso marker="#WARNING">warning</seealso> text at
+     the beginning of this document it is of vital importance that a native function
+     does return relatively fast. It is hard to give an exact maximum amount
+     of time that a native function is allowed to work, but as a rule of thumb
+     a well behaving native function should return to its caller before a
+     millisecond has passed. This can be achieved using different approaches.
+     If you have full control over the code that are to execute in the native
+     function, the best approach is to divide the work into multiple chunks of
+     work and call the native function multiple times. This might, however,
+     not always be possible, e.g. when calling third party libraries. In this
+     case you typically want to dispatch the work to another thread, return
+     from the native function, and wait for the result. The thread can send
+     the result back to the calling thread using message passing. Information
+     about thread primitives can be found below.</p>
   </description>
   <section>
   <title>FUNCTIONALITY</title>
@@ -266,10 +309,6 @@ ok
       mutable.</p>
       <p>The library initialization callbacks <c>load</c>, <c>reload</c> and
       <c>upgrade</c> are all thread-safe even for shared state data.</p>
-      <p>Avoid doing lengthy work in NIF calls as that may degrade the
-      responsiveness of the VM. NIFs are called directly by the same scheduler
-      thread that executed the calling Erlang code. The calling scheduler will thus
-      be blocked from doing any other work until the NIF returns.</p>
       </item>
     </taglist>
   </section>
diff --git a/erts/emulator/beam/erl_db.c b/erts/emulator/beam/erl_db.c
index 4c30905495..1ba1048afa 100644
--- a/erts/emulator/beam/erl_db.c
+++ b/erts/emulator/beam/erl_db.c
@@ -251,7 +251,6 @@ free_dbtable(void *vtb)
 #endif
 	ASSERT(is_immed(tb->common.heir_data));
 	erts_db_free(ERTS_ALC_T_DB_TABLE, tb, (void *) tb, sizeof(DbTable));
-	ERTS_ETS_MISC_MEM_ADD(-sizeof(DbTable));
 }
 
 static void schedule_free_dbtable(DbTable* tb)
@@ -1423,7 +1422,6 @@ BIF_RETTYPE ets_new_2(BIF_ALIST_2)
 	erts_smp_atomic_init_nob(&init_tb.common.memory_size, 0);
 	tb = (DbTable*) erts_db_alloc(ERTS_ALC_T_DB_TABLE,
 				      &init_tb, sizeof(DbTable));
-	ERTS_ETS_MISC_MEM_ADD(sizeof(DbTable));
 	erts_smp_atomic_init_nob(&tb->common.memory_size,
 				 erts_smp_atomic_read_nob(&init_tb.common.memory_size));
     }
@@ -2867,7 +2865,6 @@ void init_db(void)
     meta_pid_to_tab = (DbTable*) erts_db_alloc(ERTS_ALC_T_DB_TABLE,
 					       &init_tb,
 					       sizeof(DbTable));
-    ERTS_ETS_MISC_MEM_ADD(sizeof(DbTable));
     erts_smp_atomic_init_nob(&meta_pid_to_tab->common.memory_size,
 			     erts_smp_atomic_read_nob(&init_tb.common.memory_size));
 
@@ -2899,7 +2896,6 @@ void init_db(void)
     meta_pid_to_fixed_tab = (DbTable*) erts_db_alloc(ERTS_ALC_T_DB_TABLE,
 						     &init_tb,
 						     sizeof(DbTable));
-    ERTS_ETS_MISC_MEM_ADD(sizeof(DbTable));
     erts_smp_atomic_init_nob(&meta_pid_to_fixed_tab->common.memory_size,
 			     erts_smp_atomic_read_nob(&init_tb.common.memory_size));