From b15688d40d5147c1122aaad3b82495fbbc4dede8 Mon Sep 17 00:00:00 2001 From: Rickard Green Date: Sat, 19 Jan 2013 00:45:16 +0100 Subject: UTF-8 atom documentation --- erts/doc/src/erl_dist_protocol.xml | 288 ++++++++++++++++++++- erts/doc/src/erl_ext_dist.xml | 121 ++++++++- erts/doc/src/erlang.xml | 10 +- lib/kernel/internal_doc/distribution_handshake.txt | 216 +--------------- 4 files changed, 398 insertions(+), 237 deletions(-) diff --git a/erts/doc/src/erl_dist_protocol.xml b/erts/doc/src/erl_dist_protocol.xml index 6c725fc82d..0252187be5 100644 --- a/erts/doc/src/erl_dist_protocol.xml +++ b/erts/doc/src/erl_dist_protocol.xml @@ -547,13 +547,289 @@ If Result > 0, the packet only consists of [119, Result]. --> - +
- Handshake -

- The handshake is discussed in detail in the internal documentation for - the kernel (Erlang) application. -

+ Distribution Handshake +

+ This section describes the distribution handshake protocol + introduced in the OTP-R6 release of Erlang/OTP. This + description was previously located in + $ERL_TOP/lib/kernel/internal_doc/distribution_handshake.txt, + and has more or less been copied and "formatted" here. It has been + more or less unchanged since the year 1999, but the handshake + should not have changed much since then either. +

+
+ General +

+ The TCP/IP distribution uses a handshake which expects a + connection based protocol, i.e. the protocol does not include + any authentication after the handshake procedure. +

+

+ This is not entirely safe, as it is vulnerable against takeover + attacks, but it is a tradeoff between fair safety and performance. +

+

+ The cookies are never sent in cleartext and the handshake procedure + expects the client (called A) to be the first one to prove that it can + generate a sufficient digest. The digest is generated with the + MD5 message digest algorithm and the challenges are expected to be very + random numbers. +

+
+
+ Definitions +

+ A challenge is a 32 bit integer number in big endian order. Below the function + gen_challenge() returns a random 32 bit integer used as a challenge. +

+

+ A digest is a (16 bytes) MD5 hash of the Challenge (as text) concatenated + with the cookie (as text). Below, the function gen_digest(Challenge, Cookie) + generates a digest as described above. +

+

+ An out_cookie is the cookie used in outgoing communication to a certain node, + so that A's out_cookie for B should correspond with B's in_cookie for A and + the other way around. A's out_cookie for B and A's in_cookie for B need NOT + be the same. Below the function out_cookie(Node) returns the current + node's out_cookie for Node. +

+

+ An in_cookie is the cookie expected to be used by another node when + communicating with us, so that A's in_cookie for B corresponds with B's + out_cookie for A. Below the function in_cookie(Node) returns the current + node's in_cookie for Node. +

+

+ The cookies are text strings that can be viewed as passwords. +

+

+ Every message in the handshake starts with a 16 bit big endian integer + which contains the length of the message (not counting the two initial bytes). + In erlang this corresponds to the gen_tcp option {packet, 2}. Note that after + the handshake, the distribution switches to 4 byte packet headers. +

+ +
+
+ The Handshake in Detail +

+ Imagine two nodes, node A, which initiates the handshake and node B, which + accepts the connection. +

+ + 1) connect/accept +

A connects to B via TCP/IP and B accepts the connection.

 + 2) send_name/receive_name +

A sends an initial identification to B. B receives the message. + The message looks like this (every "square" being one byte and the packet + header removed): +

+
++---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+
+|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN|
++---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+
+
+

+ The 'n' is just a message tag. + Version0 and Version1 is the distribution version selected by node A, + based on information from EPMD. (16 bit big endian) + Flag0 ... Flag3 are capability flags, the capabilities defined in + $ERL_TOP/lib/kernel/include/dist.hrl. + (32 bit big endian) + Name0 ... NameN is the full nodename of A, as a string of bytes (the + packet length denotes how long it is). +

+ 3) recv_status/send_status +

B sends a status message to A, which indicates + if the connection is allowed. The following status codes are defined:

+ + ok + The handshake will continue. + ok_simultaneous + The handshake will continue, but A is informed that B + has another ongoing connection attempt that will be + shut down (simultaneous connect where A's name is + greater than B's name, compared literally). + nok + The handshake will not continue, as B already has an ongoing handshake + which it itself has initiated. (simultaneous connect where B's name is + greater than A's). + not_allowed + The connection is disallowed for some (unspecified) security + reason. + alive + A connection to the node is already active, which either means + that node A is confused or that the TCP connection breakdown + of a previous node with this name has not yet reached node B. + See 3B below. + +

This is the format of the status message:

+
++---+-------+-------+-...-+-------+
+|'s'|Status0|Status1| ... |StatusN|
++---+-------+-------+-...-+-------+
+
+

+ 's' is the message tag Status0 ... StatusN is the status as a string (not terminated) +

+ + 3B) send_status/recv_status +

If status was 'alive', node A will answer with + another status message containing either 'true' which means that the + connection should continue (The old connection from this node is broken), or + 'false', which simply means that the connection should be closed, the + connection attempt was a mistake.

 + 4) recv_challenge/send_challenge +

If the status was ok or ok_simultaneous, + The handshake continues with B sending A another message, the challenge. + The challenge contains the same type of information as the "name" message + initially sent from A to B, with the addition of a 32 bit challenge:

+
++---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-...-+-----+
+|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3|Name0|Name1| ... |NameN|
++---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-... +-----+
+
+

+ Where Chal0 ... Chal3 is the challenge as a 32 bit big endian integer + and the other fields are B's version, flags and full nodename. +

+ 5) send_challenge_reply/recv_challenge_reply +

Now A has generated a digest and its own challenge. Those are + sent together in a package to B:

+
++---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
+|'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
++---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
+
+

+ Where 'r' is the tag, Chal0 ... Chal3 is A's challenge for B to handle and + Dige0 ... Dige15 is the digest that A constructed from the challenge B sent + in the previous step. +

+ 6) recv_challenge_ack/send_challenge_ack +

B checks that the digest received from A is correct and generates a + digest from the challenge received from A. The digest is then sent to A. The + message looks like this:

+
++---+-----+-----+-----+-----+-...-+------+
+|'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15|
++---+-----+-----+-----+-----+-...-+------+
+
+

+ Where 'a' is the tag and Dige0 ... Dige15 is the digest calculated by B + for A's challenge.

 + 7) +

A checks the digest from B and the connection is up.

 + +
+
+ Semigraphic View +
+A (initiator)						B (acceptor)
+
+TCP connect ----------------------------------------->	
+							TCP accept
+
+send_name   ----------------------------------------->
+							recv_name
+
+	    <----------------------------------------	send_status
+recv_status
+(if status was 'alive'
+ send_status - - - - - - - - - - - - - - - - - - - ->
+							recv_status)
+							ChB = gen_challenge()
+		          (ChB)
+	    <----------------------------------------	send_challenge
+recv_challenge
+
+ChA = gen_challenge(),
+OCA = out_cookie(B),
+DiA = gen_digest(ChB,OCA)
+			  (ChA, DiA)
+send_challenge_reply -------------------------------->
+							recv_challenge_reply
+							ICB = in_cookie(A),
+							check:
+							DiA == gen_digest
+								(ChB, ICB) ?
+							- if OK:
+	    						 OCB = out_cookie(A),
+							 DiB = gen_digest
+			(DiB)					(ChA, OCB)
+	    <-----------------------------------------	 send_challenge_ack
+recv_challenge_ack					 DONE
+ICA = in_cookie(B),                                     - else
+check:                                                   CLOSE
+DiB == gen_digest(ChA,ICA) ?
+- if OK
+ DONE
+- else
+ CLOSE
+
+
+ +
+ The Currently Defined Distribution Flags +

+ Currently (OTP-R16) the following capability flags are defined: +

+
+%% The node should be published and part of the global namespace
+-define(DFLAG_PUBLISHED,1).
+
+%% The node implements an atom cache (obsolete)
+-define(DFLAG_ATOM_CACHE,2).
+
+%% The node implements extended (3 * 32 bits) references. This is
+%% required today. If not present connection will be refused.
+-define(DFLAG_EXTENDED_REFERENCES,4).
+
+%% The node implements distributed process monitoring.
+-define(DFLAG_DIST_MONITOR,8).
+
+%% The node uses separate tag for fun's (lambdas) in the distribution protocol.
+-define(DFLAG_FUN_TAGS,16#10).
+
+%% The node implements distributed named process monitoring.
+-define(DFLAG_DIST_MONITOR_NAME,16#20).
+
+%% The (hidden) node implements atom cache (obsolete)
+-define(DFLAG_HIDDEN_ATOM_CACHE,16#40).
+
+%% The node understand new fun-tags
+-define(DFLAG_NEW_FUN_TAGS,16#80).
+
+%% The node is capable of handling extended pids and ports. This is
+%% required today. If not present connection will be refused.
+-define(DFLAG_EXTENDED_PIDS_PORTS,16#100).
+
+%%
+-define(DFLAG_EXPORT_PTR_TAG,16#200).
+
+%%
+-define(DFLAG_BIT_BINARIES,16#400).
+
+%% The node understands new float format
+-define(DFLAG_NEW_FLOATS,16#800).
+
+%%
+-define(DFLAG_UNICODE_IO,16#1000).
+
+%% The node implements atom cache in distribution header.
+-define(DFLAG_DIST_HDR_ATOM_CACHE,16#2000).
+
+%% The node understand the SMALL_ATOM_EXT tag
+-define(DFLAG_SMALL_ATOM_TAGS, 16#4000).
+
+%% The node understand UTF-8 encoded atoms
+-define(DFLAG_UTF8_ATOMS, 16#10000).
+
+
+
diff --git a/erts/doc/src/erl_ext_dist.xml b/erts/doc/src/erl_ext_dist.xml index fd2da2cfe3..28afea8b29 100644 --- a/erts/doc/src/erl_ext_dist.xml +++ b/erts/doc/src/erl_ext_dist.xml @@ -119,10 +119,39 @@ Data + + +

As of ERTS version 5.10 (OTP-R16) support + for UTF-8 encoded atoms has been introduced in the external format. + However, only characters that can be encoded using Latin1 (ISO-8859-1) + are currently supported in atoms. The support for UTF-8 encoded atoms + in the external format has been implemented in order to be able to support + all Unicode characters in atoms in some future release. Full + support for Unicode atoms will not happen before OTP-R18, and might + be introduced even later than that. Until full Unicode support for + atoms has been introduced, it is an error to pass atoms containing + characters that cannot be encoded in Latin1, and the behavior is + undefined.

+

When the + DFLAG_UTF8_ATOMS + distribution flag has been exchanged between both nodes in the + distribution handshake, + all atoms in the distribution header will be encoded in UTF-8; otherwise, + all atoms in the distribution header will be encoded in Latin1. The two + new tags ATOM_UTF8_EXT, and + SMALL_ATOM_UTF8_EXT + will only be used if the DFLAG_UTF8_ATOMS distribution flag has + been exchanged between nodes, or if an atom containing characters + that cannot be encoded in Latin1 is encountered. +

+

The maximum number of allowed characters in an atom is 255. In the + UTF-8 case each character may need 4 bytes to be encoded. +

+
-
- + +
Distribution header

As of erts version 5.7.2 the old atom cache protocol was @@ -219,8 +248,7 @@

The least significant bit in that half byte is the LongAtoms flag. If it is set, 2 bytes are used for atom lengths instead of - 1 byte in the distribution header. However, the current emulator - cannot handle long atoms, so it will currently always be 0. + 1 byte in the distribution header.

After the Flags field follow the AtomCacheRefs. The @@ -247,15 +275,25 @@

InternalSegmentIndex together with the SegmentIndex completely identify the location of an atom cache entry in the - atom cache. Length is number of one byte characters that - the atom text consists of. Length is a two byte big endian integer + atom cache. Length is number of bytes that AtomText + consists of. Length is a two byte big endian integer if the LongAtoms flag has been set, otherwise a one byte - integer. Subsequent CachedAtomRefs with the same + integer. When the + DFLAG_UTF8_ATOMS + distribution flag has been exchanged between both nodes in the + distribution handshake, + characters in AtomText is encoded in UTF-8; otherwise, + encoded in Latin1. Subsequent CachedAtomRefs with the same SegmentIndex and InternalSegmentIndex as this NewAtomCacheRef will refer to this atom until a new NewAtomCacheRef with the same SegmentIndex and InternalSegmentIndex appear.

+

+ For more information on encoding of atoms, see + note on UTF-8 encoded atoms + in the beginning of this document. +

If the NewCacheEntryFlag for the next AtomCacheRef has not been set, a CachedAtomRef on the following format @@ -383,9 +421,9 @@

An atom is stored with a 2 byte unsigned length in big-endian order, - followed by Len numbers of 8 bit characters that forms the - AtomName. - Note: The maximum allowed value for Len is 255. + followed by Len numbers of 8 bit Latin1 characters that forms + the AtomName. + Note: The maximum allowed value for Len is 255.

@@ -754,12 +792,14 @@

An atom is stored with a 1 byte unsigned length, - followed by Len numbers of 8 bit characters that + followed by Len numbers of 8 bit Latin1 characters that forms the AtomName. Longer atoms can be represented by ATOM_EXT. Note the SMALL_ATOM_EXT was introduced in erts version 5.7.2 and - require a small atom distribution flag exchanged in the distribution - handshake. + require an exchange of the + DFLAG_SMALL_ATOM_TAGS + distribution flag in the + distribution handshake.

@@ -1007,7 +1047,62 @@ This term is used in minor version 1 of the external format.

+
+ + ATOM_UTF8_EXT + + + + 1 + 2 + Len + + + 118 + Len + AtomName + +
+

+ An atom is stored with a 2 byte unsigned length in big-endian order, + followed by Len bytes containing the AtomName encoded + in UTF-8. +

+

+ For more information on encoding of atoms, see + note on UTF-8 encoded atoms + in the beginning of this document. +

+
+
+ + SMALL_ATOM_UTF8_EXT + + + + 1 + 1 + Len + + + 119 + Len + AtomName + +
+

+ An atom is stored with a 1 byte unsigned length, + followed by Len bytes containing the AtomName encoded + in UTF-8. Longer atoms encoded in UTF-8 can be represented using + ATOM_UTF8_EXT. +

+

+ For more information on encoding of atoms, see + note on UTF-8 encoded atoms + in the beginning of this document. +

+
diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 5002c48ca1..0077c0096c 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -277,7 +277,9 @@ the binary contains Unicode characters greater than 16#FF. In a future release, such Unicode characters might be allowed and binary_to_atom(Binary, utf8) - will not fail in that case.

+ will not fail in that case. For more information on Unicode support in atoms + see note on UTF-8 encoded atoms + in the chapter about the external term format in the ERTS User's Guide.

 > binary_to_atom(<<"Erlang">>, latin1).
@@ -1647,9 +1649,11 @@ os_prompt%

Returns the atom whose text representation is String.

String may only contain ISO-latin-1 - characterns (i.e. numbers below 256) as the current + characters (i.e. numbers below 256) as the current implementation does not allow unicode characters >= 256 in - atoms.

+ atoms. For more information on Unicode support in atoms + see note on UTF-8 encoded atoms + in the chapter about the external term format in the ERTS User's Guide.

 > list_to_atom("Erlang").
 'Erlang'
diff --git a/lib/kernel/internal_doc/distribution_handshake.txt b/lib/kernel/internal_doc/distribution_handshake.txt index 6a3ee22ed3..d00c4ceb02 100644 --- a/lib/kernel/internal_doc/distribution_handshake.txt +++ b/lib/kernel/internal_doc/distribution_handshake.txt @@ -1,215 +1 @@ -HOW THE DISTRIBUTION HANDSHAKE WORKS ------------------------------------- - -This document describes the distribution handshake introduced in -the R6 release of Erlang/OTP. - -GENERAL -------- - -The TCP/IP distribution uses a handshake which expects a -connection based protocol, i.e. the protocol does not include -any authentication after the handshake procedure. - -This is not entirely safe, as it is vulnerable against takeover -attacks, but it is a tradeoff between fair safety and performance. - -The cookies are never sent in cleartext and the handshake procedure -expects the client (called A) to be the first one to prove that it can -generate a sufficient digest. The digest is generated with the -MD5 message digest algorithm and the challenges are expected to be very -random numbers. - -DEFINITIONS ------------ - -A challenge is a 32 bit integer number in big endian order. Below the function -gen_challenge() returns a random 32 bit integer used as a challenge. - -A digest is a (16 bytes) MD5 hash of [the Challenge (as text) concatenated -with the cookie (as text)]. Below, the function gen_digest(Challenge, Cookie) -generates a digest as described above. - -An out_cookie is the cookie used in outgoing communication to a certain node, -so that A's out_cookie for B should correspond with B's in_cookie for A and -the other way around. A's out_cookie for B and A's in_cookie for B need *NOT* -be the same. Below the function out_cookie(Node) returns the current -node's out_cookie for Node. - -An in_cookie is the cookie expected to be used by another node when -communicating with us, so that A's in_cookie for B corresponds with B's -out_cookie for A. Below the function in_cookie(Node) returns the current -node's in_cookie for Node. - -The cookies are text strings that can be viewed as passwords. - -Every message in the handshake starts with a 16 bit big endian integer -which contains the length of the message (not counting the two initial bytes). -In erlang this corresponds to the gen_tcp option {packet, 2}. Note that after -the handshake, the distribution switches to 4 byte packet headers. - -THE HANDSHAKE IN DETAIL ------------------------ - -Imagine two nodes, node A, which initiates the handshake and node B, which -accepts the connection. - -1) connect/accept: A connects to B via TCP/IP and B accepts the connection. - -2) send_name/receive_name: A sends an initial identification to B. -B receives the message. The message looks -like this (every "square" being one byte and the packet header removed): - -+---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+ -|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN| -+---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+ - -The 'n' is just a message tag, -Version0 & Version1 is the distribution version selected by node A, - based on information from EPMD. (16 bit big endian) -Flag0 ... Flag3 are capability flags, the capabilities defined in dist.hrl. - (32 bit big endian) -Name0 ... NameN is the full nodename of A, as a string of bytes (the - packet length denotes how long it is). - -3) recv_status/send_status: B sends a status message to A, which indicates -if the connection is allowed. Four different status codes are defined: -ok: The handshake will continue. -ok_simultaneous: The handshake will continue, but A is informed that B - has another ongoing connection attempt that will be - shut down (simultaneous connect where A's name is - greater than B's name, compared literally), -nok: The handshake will not continue, as B already has an ongoing handshake - which it itself has initiated. (simultaneous connect where B's name is - greater than A's) -not_allowed: The connection is disallowed for some (unspecified) security - reason. -alive: A connection to the node is already active, which either means - that node A is confused or that the TCP connection breakdown - of a previous node with this name has not yet reached node B. - See 3B below. - -This is the format of the status message: - -+---+-------+-------+-...-+-------+ -|'s'|Status0|Status1| ... |StatusN| -+---+-------+-------+-...-+-------+ - -'s' is the message tag -Status0 ... StatusN is the status as a string (not terminated) - -3B) send_status/recv_status: If status was 'alive', node A will answer with -another status message containing either 'true' which means that the -connection should continue (The old connection from this node is broken), or -'false', which simply means that the connection should be closed, the -connection attempt was a mistake. - -4) recv_challenge/send_challenge: If the status was 'ok' or 'ok_simultaneous', -The handshake continues with B sending A another message, the challenge. -The challenge contains the same type of information as the "name" message -initially sent from A to B, with the addition of a 32 bit challenge: - -+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+--- -|'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3| -+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+--- - ------+-----+-...-+-----+ - Name0|Name1| ... |NameN| - ------+-----+-... +-----+ - -Where Chal0 ... Chal3 is the challenge as a 32 bit big endian integer -and the other fields are B's version, flags and full nodename. - -5) send_challenge_reply/recv_challenge_reply: Now A has generated -a digest and its own challenge. Those are sent together in a package -to B: - -+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+ -|'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15| -+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+ - -Where 'r' is the tag, Chal0 ... Chal3 is A's challenge for B to handle and -Dige0 ... Dige15 is the digest that A constructed from the challenge B sent -in the previous step. - -6) recv_challenge_ack/send_challenge_ack: B checks that the digest received -from A is correct and generates a digest from the challenge received from -A. The digest is then sent to A. The message looks like this: - -+---+-----+-----+-----+-----+-...-+------+ -|'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15| -+---+-----+-----+-----+-----+-...-+------+ - -Where 'a' is the tag and Dige0 ... Dige15 is the digest calculated by B -for A's challenge. - -7) A checks the digest from B and the connection is up. - -SEMIGRAPHIC VIEW ----------------- - -A (initiator) B (acceptor) - -TCP connect -----------------------------------------> - TCP accept - -send_name -----------------------------------------> - recv_name - - <---------------------------------------- send_status -recv_status -(if status was 'alive' - send_status - - - - - - - - - - - - - - - - - - - -> - recv_status) - ChB = gen_challenge() - (ChB) - <---------------------------------------- send_challenge -recv_challenge - -ChA = gen_challenge(), -OCA = out_cookie(B), -DiA = gen_digest(ChB,OCA) - (ChA, DiA) -send_challenge_reply --------------------------------> - recv_challenge_reply - ICB = in_cookie(A), - check: - DiA == gen_digest - (ChB, ICB) ? - - if OK: - OCB = out_cookie(A), - DiB = gen_digest - (DiB) (ChA, OCB) - <----------------------------------------- send_challenge_ack -recv_challenge_ack DONE -ICA = in_cookie(B), - else -check: CLOSE -DiB == gen_digest(ChA,ICA) ? -- if OK - DONE -- else - CLOSE - - -THE CURRENTLY DEFINED FLAGS ---------------------------- -Currently the following capability flags are defined: - -%% The node should be published and part of the global namespace --define(DFLAG_PUBLISHED,1). - -%% The node implements an atom cache --define(DFLAG_ATOM_CACHE,2). - -%% The node implements extended (3 * 32 bits) references --define(DFLAG_EXTENDED_REFERENCES,4). - -%% The node implements distributed process monitoring. --define(DFLAG_DIST_MONITOR,8). - -%% The node uses separate tag for fun's (lambdas) in the distribution protocol. --define(DFLAG_FUN_TAGS,16). - -An R6 erlang node implements all of the above, while a C or Java node only -implements DFLAG_EXTENDED_REFERENCES. - -Last modified 1999-11-08 -- Patrik Nyblom, OTP +This information has been moved to the "Distribution Protocol" chapter of "ERTS User's Guide". -- cgit v1.2.3