Ensure NIF term creation disallows illegal values

Add a check to enif_make_double to see if its double argument is
infinity or NaN, returning a badarg exception if it is. Change the
erl_nif documentation to specify that enif_make_double returns a
badarg exception if its double argument is either infinity or NaN. Add
tests to nif_SUITE for this change.

Add checks to the enif_make* functions for atoms to prevent the
creation of atoms whose name lengths are greater than the allowed
maximum atom length. The enif_make_atom and enif_make_atom_len
functions now return a badarg exception if the input string is too
long. The enif_make_existing_atom and enif_make_existing_atom_len
functions return false if the input string is too long. Change the
erl_nif documentation to reflect the changes to these functions. Add
tests to nif_SUITE for these changes.

Add a field to ErlNifEnv to track that a NIF has raised an exception
via enif_make_badarg. If a NIF calls enif_make_badarg but then ignores
its return value and instead tries to return a non-exception term as
its return value, the runtime still raises a badarg. This is needed to
prevent enif_make_badarg values resulting from calls to
enif_make_double, enif_make_atom, or enif_make_atom_len from being
erroneously stored within other terms and returned from a NIF. Calling
enif_make_badarg but not returning its return value has been
documented as being illegal ever since enif_make_badarg was added, but
the runtime has not enforced it until now. Add tests for regular and
dirty NIFs to ensure that calls to enif_make_badarg result in badarg
exceptions even if a NIF fails to return the result of
enif_make_badarg as its return value. Add documentation to
enif_make_badarg to specify that calling it raises a badarg even if a
NIF ignores its return value.

1 files changed, 16 insertions, 7 deletions

diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml
index 3de94be9ff..feba6daaa0 100644
--- a/erts/doc/src/erl_nif.xml
+++ b/erts/doc/src/erl_nif.xml
@@ -898,12 +898,14 @@ typedef enum {
     <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_make_atom(ErlNifEnv* env, const char* name)</nametext></name>
       <fsummary>Create an atom term</fsummary>
       <desc><p>Create an atom term from the null-terminated C-string <c>name</c>
-      with iso-latin-1 encoding.</p></desc>
+      with iso-latin-1 encoding. If the length of <c>name</c> exceeds the maximum length
+      allowed for an atom, <c>enif_make_atom</c> returns a <c>badarg</c> exception.</p></desc>
     </func>
     <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_make_atom_len(ErlNifEnv* env, const char* name, size_t len)</nametext></name>
       <fsummary>Create an atom term</fsummary>
       <desc><p>Create an atom term from the string <c>name</c> with length <c>len</c>.
-      Null-characters are treated as any other characters.</p></desc>
+      Null-characters are treated as any other characters. If <c>len</c> is greater than the maximum length
+      allowed for an atom, <c>enif_make_atom</c> returns a <c>badarg</c> exception.</p></desc>
     </func>
     <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_make_badarg(ErlNifEnv* env)</nametext></name>
       <fsummary>Make a badarg exception.</fsummary>
@@ -911,8 +913,10 @@ typedef enum {
       an associated exception reason in <c>env</c>. If
       <c>enif_make_badarg</c> is called, the term it returns <em>must</em>
       be returned from the function that called it. No other return value
-      is allowed. Also, the term returned from <c>enif_make_badarg</c> may
-      be passed only to
+      is allowed. Once a NIF or any function it calls invokes <c>enif_make_badarg</c>,
+      the runtime ensures that a <c>badarg</c> exception is raised when the NIF
+      returns, even if the NIF attempts to return a non-exception term instead.
+      Also, the term returned from <c>enif_make_badarg</c> may be passed only to
       <seealso marker="#enif_is_exception">enif_is_exception</seealso> and
       not to any other NIF API function.</p></desc>
     </func>
@@ -931,7 +935,9 @@ typedef enum {
     </func>
     <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_make_double(ErlNifEnv* env, double d)</nametext></name>
       <fsummary>Create a floating-point term</fsummary>
-      <desc><p>Create a floating-point term from a <c>double</c>.</p></desc>
+      <desc><p>Create a floating-point term from a <c>double</c>. If the <c>double</c> argument is
+      not finite or is NaN, <c>enif_make_double</c> returns a <c>badarg</c> exception.</p>
+      </desc>
     </func>
     <func><name><ret>int</ret><nametext>enif_make_existing_atom(ErlNifEnv* env, const char* name, ERL_NIF_TERM* atom, ErlNifCharEncoding encode)</nametext></name>
       <fsummary>Create an existing atom term</fsummary>
@@ -939,7 +945,8 @@ typedef enum {
       the null-terminated C-string <c>name</c> with encoding
       <seealso marker="#ErlNifCharEncoding">encode</seealso>. If the atom
       already exists store the term in <c>*atom</c> and return true, otherwise
-      return false.</p></desc>
+      return false. If the length of <c>name</c> exceeds the maximum length
+      allowed for an atom, <c>enif_make_existing_atom</c> returns false.</p></desc>
     </func>
     <func><name><ret>int</ret><nametext>enif_make_existing_atom_len(ErlNifEnv* env, const char* name, size_t len, ERL_NIF_TERM* atom, ErlNifCharEncoding encoding)</nametext></name>
       <fsummary>Create an existing atom term</fsummary>
@@ -947,7 +954,9 @@ typedef enum {
       string <c>name</c> with length <c>len</c> and encoding
       <seealso marker="#ErlNifCharEncoding">encode</seealso>. Null-characters
       are treated as any other characters. If the atom already exists store the term
-      in <c>*atom</c> and return true, otherwise return false.</p></desc>
+      in <c>*atom</c> and return true, otherwise return false. If <c>len</c> is greater
+      than the maximum length allowed for an atom, <c>enif_make_existing_atom_len</c>
+      returns false.</p></desc>
     </func>
     <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_make_int(ErlNifEnv* env, int i)</nametext></name>
       <fsummary>Create an integer term</fsummary>