Merge branch 'sverk/r16/utf8-atoms'

* sverk/r16/utf8-atoms:
  erl_interface: Fix bug when transcoding atoms from and to UTF8
  erl_interface: Changed erlang_char_encoding interface
  erts: Testcase doing unicode atom printout with ~w
  erl_interface: even more utf8 atom stuff
  erts: Fix bug in analyze_utf8 causing faulty latin1 detection
  Add UTF-8 node name support for epmd
  workaround...
  Fix merge conflict with hasse
  UTF-8 atom documentation
  test case
  erl_interface: utf8 atoms continued
  Add utf8 atom distribution test cases
  atom fixes for NIFs and atom_to_binary
  UTF-8 support for distribution
  Implement UTF-8 atom support for jinterface
  erl_interface: Enable decode of unicode atoms
  stdlib: Fix printing of unicode atoms
  erts: Change internal representation of atoms to utf8
  erts: Refactor rename DFLAG(S)_INTERNAL_TAGS for conformity

Conflicts:
	erts/emulator/beam/io.c

OTP-10753

1 files changed, 282 insertions, 6 deletions

diff --git a/erts/doc/src/erl_dist_protocol.xml b/erts/doc/src/erl_dist_protocol.xml
index 6c725fc82d..0252187be5 100644
--- a/erts/doc/src/erl_dist_protocol.xml
+++ b/erts/doc/src/erl_dist_protocol.xml
@@ -547,13 +547,289 @@ If Result > 0, the packet only consists of [119, Result].
 -->
 
     </section>
-
+    <marker id="distribution_handshake"/>
       <section>
-	<title>Handshake</title>
-	<p>
-	  The handshake is discussed in detail in the internal documentation for 
-	  the kernel (Erlang) application.
-	</p>
+	<title>Distribution Handshake</title>
+	  <p>
+	    This section describes the distribution handshake protocol
+	    introduced in the OTP-R6 release of Erlang/OTP. This
+	    description was previously located in
+	    <c>$ERL_TOP/lib/kernel/internal_doc/distribution_handshake.txt</c>,
+	    and has more or less been copied and "formatted" here. It has been
+	    more or less unchanged since the year 1999, but the handshake
+	    should not have changed much since then either.
+	  </p>
+	<section>
+	  <title>General</title>
+	  <p>
+	    The TCP/IP distribution uses a handshake which expects a
+	    connection based protocol, i.e. the protocol does not include
+	    any authentication after the handshake procedure.
+	  </p>
+	  <p>
+	    This is not entirely safe, as it is vulnerable against takeover
+	    attacks, but it is a tradeoff between fair safety and performance.
+	  </p>
+	  <p>
+	    The cookies are never sent in cleartext and the handshake procedure
+	    expects the client (called A) to be the first one to prove that it can 
+	    generate a sufficient digest. The digest is generated with the 
+	    MD5 message digest algorithm and the challenges are expected to be very
+	    random numbers.
+	  </p>
+	</section>
+	<section>
+	  <title>Definitions</title>
+	  <p>
+	    A challenge is a 32 bit integer number in big endian order. Below the function
+	    <c>gen_challenge()</c> returns a random 32 bit integer used as a challenge.
+	  </p>
+	  <p>
+	    A digest is a (16 bytes) MD5 hash of the Challenge (as text) concatenated
+	    with the cookie (as text). Below, the function <c>gen_digest(Challenge, Cookie)</c>
+	    generates a digest as described above.
+	  </p>
+	  <p>
+	    An out_cookie is the cookie used in outgoing communication to a certain node,
+	    so that A's out_cookie for B should correspond with B's in_cookie for A and 
+	    the other way around. A's out_cookie for B and A's in_cookie for B need <em>NOT</em>
+	    be the same. Below the function <c>out_cookie(Node)</c> returns the current
+	    node's out_cookie for <c>Node</c>.
+	  </p>
+	  <p>
+	    An in_cookie is the cookie expected to be used by another node when 
+	    communicating with us, so that A's in_cookie for B corresponds with B's 
+	    out_cookie for A. Below the function <c>in_cookie(Node)</c> returns the current
+	    node's <c>in_cookie</c> for <c>Node</c>.
+	  </p>
+	  <p>
+	    The cookies are text strings that can be viewed as passwords.
+	  </p>
+	  <p>
+	    Every message in the handshake starts with a 16 bit big endian integer 
+	    which contains the length of the message (not counting the two initial bytes).
+	    In erlang this corresponds to the <c>gen_tcp</c> option <c>{packet, 2}</c>. Note that after 
+	    the handshake, the distribution switches to 4 byte packet headers.
+	  </p>
+
+	</section>
+	<section>
+	  <title>The Handshake in Detail</title>
+	  <p>
+	    Imagine two nodes, node A, which initiates the handshake and node B, which
+	    accepts the connection.
+	  </p>
+	  <taglist>
+	    <tag>1) connect/accept</tag>
+	    <item><p>A connects to B via TCP/IP and B accepts the connection.</p></item>
+	    <tag>2) send_name/receive_name</tag>
+	    <item><p>A sends an initial identification to B. B receives the message.
+	    The message looks like this (every "square" being one byte and the packet
+	    header removed):
+	  </p>
+<pre>
++---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+
+|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN|
++---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+
+</pre>
+	  <p>
+	    The 'n' is just a message tag.
+	    Version0 and Version1 is the distribution version selected by node A,
+            based on information from EPMD. (16 bit big endian)
+	    Flag0 ... Flag3 are capability flags, the capabilities defined in
+	    <c>$ERL_TOP/lib/kernel/include/dist.hrl</c>.
+            (32 bit big endian)
+	    Name0 ... NameN is the full nodename of A, as a string of bytes (the
+            packet length denotes how long it is).
+	  </p></item>
+	  <tag>3) recv_status/send_status</tag>
+	  <item><p>B sends a status message to A, which indicates
+	    if the connection is allowed. The following status codes are defined:</p>
+	    <taglist>
+	      <tag><c>ok</c></tag>
+	      <item>The handshake will continue.</item>
+	      <tag><c>ok_simultaneous</c></tag>
+	      <item>The handshake will continue, but A is informed that B
+              has another ongoing connection attempt that will be
+	      shut down (simultaneous connect where A's name is 
+	      greater than B's name, compared literally).</item>
+	      <tag><c>nok</c></tag>
+	      <item>The handshake will not continue, as B already has an ongoing handshake
+	      which it itself has initiated. (simultaneous connect where B's name is 
+	      greater than A's).</item>
+	      <tag><c>not_allowed</c></tag>
+	      <item>The connection is disallowed for some (unspecified) security 
+              reason.</item>
+	      <tag><c>alive</c></tag>
+	      <item>A connection to the node is already active, which either means
+	      that node A is confused or that the TCP connection breakdown
+	      of a previous node with this name has not yet reached node B.
+	      See 3B below.</item>
+	    </taglist>
+	    <p>This is the format of the status message:</p>
+<pre>
++---+-------+-------+-...-+-------+
+|'s'|Status0|Status1| ... |StatusN|
++---+-------+-------+-...-+-------+
+</pre>
+	    <p>
+	      's' is the message tag Status0 ... StatusN is the status as a string (not terminated)
+	    </p>
+	  </item>
+	  <tag>3B) send_status/recv_status</tag>
+	  <item><p>If status was 'alive', node A will answer with
+	  another status message containing either 'true' which means that the
+	  connection should continue (The old connection from this node is broken), or
+	  <c>'false'</c>, which simply means that the connection should be closed, the 
+	  connection attempt was a mistake.</p></item>
+	  <tag>4) recv_challenge/send_challenge</tag>
+	  <item><p>If the status was <c>ok</c> or <c>ok_simultaneous</c>,
+	  The handshake continues with B sending A another message, the challenge.
+	  The challenge contains the same type of information as the "name" message 
+	  initially sent from A to B, with the addition of a 32 bit challenge:</p>
+<pre>
++---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-...-+-----+
+|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3|Name0|Name1| ... |NameN|
++---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-... +-----+
+</pre>
+          <p>
+	    Where Chal0 ... Chal3 is the challenge as a 32 bit big endian integer
+	    and the other fields are B's version, flags and full nodename.
+	  </p></item>
+	  <tag>5) send_challenge_reply/recv_challenge_reply</tag>
+	  <item><p>Now A has generated a digest and its own challenge. Those are
+	  sent together in a package to B:</p>
+<pre>
++---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
+|'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
++---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
+</pre>
+          <p>
+	    Where 'r' is the tag, Chal0 ... Chal3 is A's challenge for B to handle and
+	    Dige0 ... Dige15 is the digest that A constructed from the challenge B sent
+	    in the previous step.
+	  </p></item>
+	  <tag>6) recv_challenge_ack/send_challenge_ack</tag>
+	  <item><p>B checks that the digest received from A is correct and generates a
+	  digest from the challenge received from  A. The digest is then sent to A. The
+	  message looks like this:</p>
+<pre>
++---+-----+-----+-----+-----+-...-+------+
+|'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
++---+-----+-----+-----+-----+-...-+------+
+</pre>
+          <p>
+	    Where 'a' is the tag and Dige0 ... Dige15 is the digest calculated by B
+	    for A's challenge.</p></item>
+	  <tag>7)</tag>
+	  <item><p>A checks the digest from B and the connection is up.</p></item>
+	</taglist>
+	</section>
+	<section>
+	  <title>Semigraphic View</title>
+<pre>
+A (initiator)						B (acceptor)
+
+TCP connect -----------------------------------------&gt;	
+							TCP accept
+
+send_name   -----------------------------------------&gt;
+							recv_name
+
+	    &lt;----------------------------------------	send_status
+recv_status
+(if status was 'alive'
+ send_status - - - - - - - - - - - - - - - - - - - -&gt;
+							recv_status)
+							ChB = gen_challenge()
+		          (ChB)
+	    &lt;----------------------------------------	send_challenge
+recv_challenge
+
+ChA = gen_challenge(),
+OCA = out_cookie(B),
+DiA = gen_digest(ChB,OCA)
+			  (ChA, DiA)
+send_challenge_reply --------------------------------&gt;
+							recv_challenge_reply
+							ICB = in_cookie(A),
+							check:
+							DiA == gen_digest
+								(ChB, ICB) ?
+							- if OK:
+	    						 OCB = out_cookie(A),
+							 DiB = gen_digest
+			(DiB)					(ChA, OCB)
+	    &lt;-----------------------------------------	 send_challenge_ack
+recv_challenge_ack					 DONE
+ICA = in_cookie(B),                                     - else
+check:                                                   CLOSE
+DiB == gen_digest(ChA,ICA) ?
+- if OK
+ DONE
+- else
+ CLOSE
+</pre>
+	</section>
+	<marker id="dflags"/>
+	<section>
+	  <title>The Currently Defined Distribution Flags</title>
+	  <p>
+	    Currently (OTP-R16) the following capability flags are defined:
+	  </p>
+<pre>
+%% The node should be published and part of the global namespace
+-define(DFLAG_PUBLISHED,1).
+
+%% The node implements an atom cache (obsolete)
+-define(DFLAG_ATOM_CACHE,2).
+
+%% The node implements extended (3 * 32 bits) references. This is
+%% required today. If not present connection will be refused.
+-define(DFLAG_EXTENDED_REFERENCES,4).
+
+%% The node implements distributed process monitoring.
+-define(DFLAG_DIST_MONITOR,8).
+
+%% The node uses separate tag for fun's (lambdas) in the distribution protocol.
+-define(DFLAG_FUN_TAGS,16#10).
+
+%% The node implements distributed named process monitoring.
+-define(DFLAG_DIST_MONITOR_NAME,16#20).
+
+%% The (hidden) node implements atom cache (obsolete)
+-define(DFLAG_HIDDEN_ATOM_CACHE,16#40).
+
+%% The node understand new fun-tags
+-define(DFLAG_NEW_FUN_TAGS,16#80).
+
+%% The node is capable of handling extended pids and ports. This is
+%% required today. If not present connection will be refused.
+-define(DFLAG_EXTENDED_PIDS_PORTS,16#100).
+
+%%
+-define(DFLAG_EXPORT_PTR_TAG,16#200).
+
+%%
+-define(DFLAG_BIT_BINARIES,16#400).
+
+%% The node understands new float format
+-define(DFLAG_NEW_FLOATS,16#800).
+
+%%
+-define(DFLAG_UNICODE_IO,16#1000).
+
+%% The node implements atom cache in distribution header.
+-define(DFLAG_DIST_HDR_ATOM_CACHE,16#2000).
+
+%% The node understand the SMALL_ATOM_EXT tag
+-define(DFLAG_SMALL_ATOM_TAGS, 16#4000).
+
+%% The node understand UTF-8 encoded atoms
+-define(DFLAG_UTF8_ATOMS, 16#10000).
+
+</pre>
+	</section>
       </section>
 
       <section>