This description is far from complete. It will be updated if the protocol is updated. However, the protocols, both from Erlang nodes to the Erlang Port Mapper Daemon (EPMD) and between Erlang nodes are stable since many years.
The distribution protocol can be divided into four parts:
Low-level socket connection (1)
Handshake, interchange node name, and authenticate (2)
Authentication (done by
Connected (4)
A node fetches the port number of another node through the EPMD (at the other host) to initiate a connection request.
For each host, where a distributed Erlang node is running, also an EPMD is to be running. The EPMD can be started explicitly or automatically as a result of the Erlang node startup.
By default the EPMD listens on port 4369.
(3) and (4) above are performed at the same level but the
The integers in all multibyte fields are in big-endian order.
The Erlang Distribution protocol is not by itself secure and does not
aim to be so. In order to get secure distribution the distributed nodes
should be configured to use distribution over tls.
See the
The requests served by the EPMD are summarized in the following figure.
Each request
When a distributed node is started it registers itself in the EPMD.
The message
The port number on which the node accept connection requests.
77 = normal Erlang node, 72 = hidden node (C-node), ...
0 = TCP/IPv4, ...
The highest distribution version that this node can handle. The value in Erlang/OTP R6B and later is 5.
The lowest distribution version that this node can handle. The value in Erlang/OTP R6B and later is 5.
The length (in bytes) of field
The node name as an UTF-8 encoded string of
The length of field
Extra field of
The connection created to the EPMD must be kept as long as the node is a distributed node. When the connection is closed, the node is automatically unregistered from the EPMD.
The response message
Result = 0 -> ok, result > 0 -> error.
A node unregisters itself from the EPMD by closing the TCP connection to EPMD established when the node was registered.
When one node wants to connect to another node it starts with
a
where N =
or
If
The EPMD closes the socket when it has sent the information.
This request is used through the Erlang function
The response for a
io:format("name ~ts at port ~p~n", [NodeName, Port]).
This request is not really used, it is to be regarded as a debug feature.
The response for a
io:format("active name ~ts at port ~p, fd = ~p~n",
[NodeName, Port, Fd]).
or
io:format("old/unused name ~ts at port ~p, fd = ~p ~n",
[NodeName, Port, Fd]).
This request kills the running EPMD. It is almost never used.
The response for a
where
where n =
The current implementation of Erlang does not care if the connection to the EPMD is broken.
The response for a
where
A negative response can look as follows:
where
This section describes the distribution handshake protocol introduced
in Erlang/OTP R6. This description was previously located in
The TCP/IP distribution uses a handshake that expects a connection-based protocol, that is, the protocol does not include any authentication after the handshake procedure.
This is not entirely safe, as it is vulnerable against takeover attacks, but it is a tradeoff between fair safety and performance.
The cookies are never sent in cleartext and the handshake procedure
expects the client (called
A challenge is a 32-bit integer in big-endian order. Below the function
A digest is a (16 bytes) MD5 hash of the challenge (as text)
concatenated with the cookie (as text). Below, the function
An
An
The cookies are text strings that can be viewed as passwords.
Every message in the handshake starts with a 16-bit big-endian integer,
which contains the message length (not counting the two initial bytes).
In Erlang this corresponds to option
Imagine two nodes,
+---+--------+--------+-----+-----+-----+-----+-----+-----+-...-+-----+ |'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Name0|Name1| ... |NameN| +---+--------+--------+-----+-----+-----+-----+-----+-----+-... +-----+
'n' is the message tag. 'Version0' and 'Version1' is the
distribution version selected by
The handshake will continue.
The handshake will continue, but
The handshake will not continue, as
The connection is disallowed for some (unspecified) security reason.
A connection to the node is already active, which either means
that node
The format of the status message is as follows:
+---+-------+-------+-...-+-------+ |'s'|Status0|Status1| ... |StatusN| +---+-------+-------+-...-+-------+
's' is the message tag. 'Status0' ... 'StatusN' is the status as a string (not terminated).
If status was
If the status was
+---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-...-+-----+ |'n'|Version0|Version1|Flag0|Flag1|Flag2|Flag3|Chal0|Chal1|Chal2|Chal3|Name0|Name1| ... |NameN| +---+--------+--------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-... +-----+
'Chal0' ... 'Chal3' is the challenge as a 32-bit big-endian integer
and the other fields are
Now
+---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+ |'r'|Chal0|Chal1|Chal2|Chal3|Dige0|Dige1|Dige2|Dige3| ... |Dige15| +---+-----+-----+-----+-----+-----+-----+-----+-----+-...-+------+
'r' is the tag. 'Chal0' ... 'Chal3' is
+---+-----+-----+-----+-----+-...-+------+ |'a'|Dige0|Dige1|Dige2|Dige3| ... |Dige15| +---+-----+-----+-----+-----+-...-+------+
'a' is the tag. 'Dige0' ... 'Dige15' is the digest calculated by
A (initiator) B (acceptor)
TCP connect ------------------------------------>
TCP accept
send_name -------------------------------------->
recv_name
<---------------------------------------------- send_status
recv_status
(if status was 'alive'
send_status - - - - - - - - - - - - - - - - - ->
recv_status)
ChB = gen_challenge()
(ChB)
<---------------------------------------------- send_challenge
recv_challenge
ChA = gen_challenge(),
OCA = out_cookie(B),
DiA = gen_digest(ChB, OCA)
(ChA, DiA)
send_challenge_reply --------------------------->
recv_challenge_reply
ICB = in_cookie(A),
check:
DiA == gen_digest (ChB, ICB)?
- if OK:
OCB = out_cookie(A),
DiB = gen_digest (ChA, OCB)
(DiB)
<----------------------------------------------- send_challenge_ack
recv_challenge_ack DONE
ICA = in_cookie(B), - else:
check: CLOSE
DiB == gen_digest(ChA, ICA)?
- if OK:
DONE
- else:
CLOSE
The following capability flags are defined:
The node is to be published and part of the global namespace.
The node implements an atom cache (obsolete).
The node implements extended (3 × 32 bits) references. This is required today. If not present, the connection is refused.
The node implements distributed process monitoring.
The node uses separate tag for funs (lambdas) in the distribution protocol.
The node implements distributed named process monitoring.
The (hidden) node implements atom cache (obsolete).
The node understand new fun tags.
The node can handle extended pids and ports. This is required today. If not present, the connection is refused.
The node understands new float format.
The node implements atom cache in distribution header.
The node understand the
The node understand UTF-8 encoded atoms.
The node understand the map tag.
The node understand big node creation.
As from ERTS 5.7.2 the runtime system passes a distribution flag
in the handshake stage that enables the use of a
Equal to d + n + m.
A tuple passed using the external format of Erlang.
The message sent to another node using the '!' (in external format).
Notice that
Notice that
Nodes with an ERTS version earlier than 5.7.2 does not pass the distribution flag that enables the distribution header. Messages passed between nodes have in this case the following format:
Equal to 1 + n + m.
Equal to
A tuple passed using the external format of Erlang.
The message sent to another node using the '!' (in external format).
Notice that
The
Followed by
Followed by
Followed by
Followed by
None, but the version number was increased anyway.
These are only recognized by Erlang nodes, not by hidden nodes.
We include