Merge branch 'sverk/r16/utf8-atoms'

* sverk/r16/utf8-atoms:
  erl_interface: Fix bug when transcoding atoms from and to UTF8
  erl_interface: Changed erlang_char_encoding interface
  erts: Testcase doing unicode atom printout with ~w
  erl_interface: even more utf8 atom stuff
  erts: Fix bug in analyze_utf8 causing faulty latin1 detection
  Add UTF-8 node name support for epmd
  workaround...
  Fix merge conflict with hasse
  UTF-8 atom documentation
  test case
  erl_interface: utf8 atoms continued
  Add utf8 atom distribution test cases
  atom fixes for NIFs and atom_to_binary
  UTF-8 support for distribution
  Implement UTF-8 atom support for jinterface
  erl_interface: Enable decode of unicode atoms
  stdlib: Fix printing of unicode atoms
  erts: Change internal representation of atoms to utf8
  erts: Refactor rename DFLAG(S)_INTERNAL_TAGS for conformity

Conflicts:
	erts/emulator/beam/io.c

OTP-10753

3 files changed, 4 insertions, 216 deletions

diff --git a/lib/kernel/include/dist.hrl b/lib/kernel/include/dist.hrl
index 5b52f6f294..91e13d99a9 100644
--- a/lib/kernel/include/dist.hrl
+++ b/lib/kernel/include/dist.hrl
@@ -36,3 +36,4 @@
 -define(DFLAG_UNICODE_IO,16#1000).
 -define(DFLAG_DIST_HDR_ATOM_CACHE,16#2000).
 -define(DFLAG_SMALL_ATOM_TAGS, 16#4000).
+-define(DFLAG_UTF8_ATOMS, 16#10000).
diff --git a/lib/kernel/internal_doc/distribution_handshake.txt b/lib/kernel/internal_doc/distribution_handshake.txt
index 6a3ee22ed3..d00c4ceb02 100644
--- a/lib/kernel/internal_doc/distribution_handshake.txt
+++ b/lib/kernel/internal_doc/distribution_handshake.txt
@@ -1,215 +1 @@
-HOW THE DISTRIBUTION HANDSHAKE WORKS
-------------------------------------
-
-This document describes the distribution handshake introduced in 
-the R6 release of Erlang/OTP.
-
-GENERAL
--------
-
-The TCP/IP distribution uses a handshake which expects a
-connection based protocol, i.e. the protocol does not include
-any authentication after the handshake procedure.
-
-This is not entirely safe, as it is vulnerable against takeover
-attacks, but it is a tradeoff between fair safety and performance.
-
-The cookies are never sent in cleartext and the handshake procedure
-expects the client (called A) to be the first one to prove that it can 
-generate a sufficient digest. The digest is generated with the 
-MD5 message digest algorithm and the challenges are expected to be very
-random numbers.
-
-DEFINITIONS
------------
-
-A challenge is a 32 bit integer number in big endian order. Below the function
-gen_challenge() returns a random 32 bit integer used as a challenge.
-
-A digest is a (16 bytes) MD5 hash of [the Challenge (as text) concatenated
-with the cookie (as text)]. Below, the function gen_digest(Challenge, Cookie)
-generates a digest as described above.
-
-An out_cookie is the cookie used in outgoing communication to a certain node,
-so that A's out_cookie for B should correspond with B's in_cookie for A and 
-the other way around. A's out_cookie for B and A's in_cookie for B need *NOT*
-be the same. Below the function out_cookie(Node) returns the current
-node's out_cookie for Node.
-
-An in_cookie is the cookie expected to be used by another node when 
-communicating with us, so that A's in_cookie for B corresponds with B's 
-out_cookie for A. Below the function in_cookie(Node) returns the current
-node's in_cookie for Node.
-
-The cookies are text strings that can be viewed as passwords.
-
-Every message in the handshake starts with a 16 bit big endian integer 
-which contains the length of the message (not counting the two initial bytes).
-In erlang this corresponds to the gen_tcp option {packet, 2}. Note that after 
-the handshake, the distribution switches to 4 byte packet headers.
-
-THE HANDSHAKE IN DETAIL
------------------------
- 
-Imagine two nodes, node A, which initiates the handshake and node B, which
-accepts the connection.
-
-1) connect/accept: A connects to B via TCP/IP and B accepts the connection.
-
-2) send_name/receive_name: A sends an initial identification to B. 
-B receives the message. The message looks
-like this (every "square" being one byte and the packet header removed):
-
-+---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+
-|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN|
-+---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+
-
-The 'n' is just a message tag,
-Version0 & Version1 is the distribution version selected by node A,
-                    based on information from EPMD. (16 bit big endian)
-Flag0 ... Flag3 are capability flags, the capabilities defined in dist.hrl.
-                (32 bit big endian)
-Name0 ... NameN is the full nodename of A, as a string of bytes (the
-                packet length denotes how long it is).
-
-3) recv_status/send_status: B sends a status message to A, which indicates
-if the connection is allowed. Four different status codes are defined:
-ok: The handshake will continue.
-ok_simultaneous: The handshake will continue, but A is informed that B
-                 has another ongoing connection attempt that will be
-		 shut down (simultaneous connect where A's name is 
-		 greater than B's name, compared literally),
-nok: The handshake will not continue, as B already has an ongoing handshake
-     which it itself has initiated. (simultaneous connect where B's name is 
-     greater than A's)
-not_allowed: The connection is disallowed for some (unspecified) security 
-             reason.
-alive: A connection to the node is already active, which either means
-       that node A is confused or that the TCP connection breakdown
-       of a previous node with this name has not yet reached node B.
-       See 3B below.
-
-This is the format of the status message:
-
-+---+-------+-------+-...-+-------+
-|'s'|Status0|Status1| ... |StatusN|
-+---+-------+-------+-...-+-------+
-
-'s' is the message tag
-Status0 ... StatusN is the status as a string (not terminated)
-
-3B) send_status/recv_status: If status was 'alive', node A will answer with
-another status message containing either 'true' which means that the
-connection should continue (The old connection from this node is broken), or
-'false', which simply means that the connection should be closed, the 
-connection attempt was a mistake.
-
-4) recv_challenge/send_challenge: If the status was 'ok' or 'ok_simultaneous',
-The handshake continues with B sending A another message, the challenge.
-The challenge contains the same type of information as the "name" message 
-initially sent from A to B, with the addition of a 32 bit challenge:
-
-+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+---
-|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3|
-+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+---
-   ------+-----+-...-+-----+
-    Name0|Name1| ... |NameN|
-   ------+-----+-... +-----+
-
-Where Chal0 ... Chal3 is the challenge as a 32 bit big endian integer
-and the other fields are B's version, flags and full nodename.
-
-5) send_challenge_reply/recv_challenge_reply: Now A has generated
-a digest and its own challenge. Those are sent together in a package
-to B:
-
-+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
-|'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
-+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
-
-Where 'r' is the tag, Chal0 ... Chal3 is A's challenge for B to handle and
-Dige0 ... Dige15 is the digest that A constructed from the challenge B sent
-in the previous step.
-
-6) recv_challenge_ack/send_challenge_ack: B checks that the digest received
-from A is correct and generates a digest from the challenge received from 
-A. The digest is then sent to A. The message looks like this:
-
-+---+-----+-----+-----+-----+-...-+------+
-|'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
-+---+-----+-----+-----+-----+-...-+------+
-
-Where 'a' is the tag and Dige0 ... Dige15 is the digest calculated by B
-for A's challenge.
-
-7) A checks the digest from B and the connection is up.
-
-SEMIGRAPHIC VIEW
-----------------
-
-A (initiator)						B (acceptor)
-
-TCP connect ----------------------------------------->	
-							TCP accept
-
-send_name   ----------------------------------------->	
-							recv_name
-
-	    <----------------------------------------	send_status
-recv_status
-(if status was 'alive'
- send_status - - - - - - - - - - - - - - - - - - - ->   
-							recv_status)
-							ChB = gen_challenge()
-		          (ChB)
-	    <----------------------------------------	send_challenge
-recv_challenge
-
-ChA = gen_challenge(),
-OCA = out_cookie(B),
-DiA = gen_digest(ChB,OCA)
-			  (ChA, DiA)
-send_challenge_reply -------------------------------->
-							recv_challenge_reply
-							ICB = in_cookie(A),
-							check:
-							DiA == gen_digest
-								(ChB, ICB) ?
-							- if OK:
-	    						 OCB = out_cookie(A),
-							 DiB = gen_digest
-			(DiB)					(ChA, OCB)
-	    <-----------------------------------------	 send_challenge_ack
-recv_challenge_ack					 DONE
-ICA = in_cookie(B),                                     - else
-check:                                                   CLOSE
-DiB == gen_digest(ChA,ICA) ?
-- if OK
- DONE
-- else
- CLOSE
-
-
-THE CURRENTLY DEFINED FLAGS
----------------------------
-Currently the following capability flags are defined:
-
-%% The node should be published and part of the global namespace
--define(DFLAG_PUBLISHED,1).
-
-%% The node implements an atom cache
--define(DFLAG_ATOM_CACHE,2).
-
-%% The node implements extended (3 * 32 bits) references
--define(DFLAG_EXTENDED_REFERENCES,4).
-
-%% The node implements distributed process monitoring.
--define(DFLAG_DIST_MONITOR,8).
- 
-%% The node uses separate tag for fun's (lambdas) in the distribution protocol.
--define(DFLAG_FUN_TAGS,16).
- 
-An R6 erlang node implements all of the above, while a C or Java node only
-implements DFLAG_EXTENDED_REFERENCES.
-
-Last modified 1999-11-08 -- Patrik Nyblom, OTP
+This information has been moved to the "Distribution Protocol" chapter of "ERTS User's Guide".
diff --git a/lib/kernel/src/dist_util.erl b/lib/kernel/src/dist_util.erl
index e3511988a6..bbb212cebe 100644
--- a/lib/kernel/src/dist_util.erl
+++ b/lib/kernel/src/dist_util.erl
@@ -115,7 +115,8 @@ make_this_flags(RequestType, OtherNode) ->
 	 ?DFLAG_NEW_FLOATS bor
 	 ?DFLAG_UNICODE_IO bor
 	 ?DFLAG_DIST_HDR_ATOM_CACHE bor
-	 ?DFLAG_SMALL_ATOM_TAGS).
+	 ?DFLAG_SMALL_ATOM_TAGS bor
+	 ?DFLAG_UTF8_ATOMS).
 
 handshake_other_started(#hs_data{request_type=ReqType}=HSData0) ->
     {PreOtherFlags,Node,Version} = recv_name(HSData0),