20032017
Ericsson AB. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Distributed Erlang
distributed.xml
Distributed Erlang System
A distributed Erlang system consists of a number of
Erlang runtime systems communicating with each other. Each such
runtime system is called a node. Message passing between
processes at different nodes, as well as links and monitors, are
transparent when pids are used. Registered names, however, are
local to each node. This means that the node must be specified as well
when sending messages, and so on, using registered names.
The distribution mechanism is implemented using TCP/IP sockets.
How to implement an alternative carrier is described in the
ERTS User's Guide.
Starting a distributed node without also specifying
-proto_dist inet_tls
will expose the node to attacks that may give the attacker
complete access to the node and in extension the cluster.
When using un-secure distributed nodes, make sure that the
network is configured to keep potential attackers out.
See the
Using SSL for Erlang Distribution User's Guide
for details on how to setup a secure distributed node.
Node Connections
The nodes in a distributed Erlang system are loosely connected.
The first time the name of another node is used, for example, if
spawn(Node,M,F,A) or net_adm:ping(Node) is called,
a connection attempt to that node is made.
Connections are by default transitive. If a node A connects to
node B, and node B has a connection to node C, then node A
also tries to connect to node C. This feature can be turned off by
using the command-line flag -connect_all false, see the
erl(1) manual page in ERTS.
If a node goes down, all connections to that node are removed.
Calling erlang:disconnect_node(Node) forces disconnection
of a node.
The list of (visible) nodes currently connected to is returned by
nodes().
epmd
The Erlang Port Mapper Daemon epmd is automatically
started at every host where an Erlang node is started. It is
responsible for mapping the symbolic node names to machine
addresses. See the
epmd(1) manual page in ERTS.
Hidden Nodes
In a distributed Erlang system, it is sometimes useful to
connect to a node without also connecting to all other nodes.
An example is some kind of O&M functionality used to
inspect the status of a system, without disturbing it. For this
purpose, a hidden node can be used.
A hidden node is a node started with the command-line flag
-hidden. Connections between hidden nodes and other nodes
are not transitive, they must be set up explicitly. Also, hidden
nodes does not show up in the list of nodes returned by
nodes(). Instead, nodes(hidden) or
nodes(connected) must be used. This means, for example,
that the hidden node is not added to the set of nodes that
global is keeping track of.
This feature was added in Erlang 5.0/OTP R7.
C Nodes
A C node is a C program written to act as a hidden node
in a distributed Erlang system. The library Erl_Interface
contains functions for this purpose. For more information about
C nodes, see the
Erl_Interface application and
Interoperability Tutorial..
Security
Authentication determines which nodes are allowed to communicate
with each other. In a network of different Erlang nodes, it is
built into the system at the lowest possible level. Each node has
its own magic cookie, which is an Erlang atom.
When a node tries to connect to another node, the magic cookies
are compared. If they do not match, the connected node rejects
the connection.
At start-up, a node has a random atom assigned as its magic
cookie and the cookie of other nodes is assumed to be
nocookie. The first action of the Erlang network
authentication server (auth) is then to read a file named
$HOME/.erlang.cookie. If the file does not exist, it is
created. The UNIX permissions mode of the file is set to octal
400 (read-only by user) and its contents are a random string. An
atom Cookie is created from the contents of the file and
the cookie of the local node is set to this using
erlang:set_cookie(node(), Cookie). This also makes
the local node assume that all other nodes have the same cookie
Cookie.
Thus, groups of users with identical cookie files get Erlang
nodes that can communicate freely and without interference from
the magic cookie system. Users who want to run nodes on separate
file systems must make certain that their cookie files are
identical on the different file systems.
For a node Node1 with magic cookie Cookie to be
able to connect to, or accept a connection from, another node
Node2 with a different cookie DiffCookie,
the function erlang:set_cookie(Node2, DiffCookie) must
first be called at Node1. Distributed systems with
multiple user IDs can be handled in this way.
The default when a connection is established between two nodes,
is to immediately connect all other visible nodes as well. This
way, there is always a fully connected network. If there are
nodes with different cookies, this method can be inappropriate
and the command-line flag -connect_all false must be set,
see the erl(1)
manual page in ERTS.
The magic cookie of the local node is retrieved by calling
erlang:get_cookie().
Distribution BIFs
Some useful BIFs for distributed programming
(for more information, see the
erlang(3) manual page in ERTS:
| BIF |
Description |
| erlang:disconnect_node(Node) |
Forces the disconnection of a node. |
| erlang:get_cookie() |
Returns the magic cookie of the current node. |
| is_alive() |
Returns true if the runtime system is a node and can connect to other nodes, false otherwise. |
| monitor_node(Node, true|false) |
Monitors the status of
Node. A message{nodedown, Node} is received
if the connection to it is lost. |
| node() |
Returns the name of the current node. Allowed in guards. |
| node(Arg) |
Returns the node where Arg, a pid, reference, or port, is located. |
| nodes() |
Returns a list of all visible nodes this node is connected to. |
| nodes(Arg) |
Depending on Arg,
this function can return a list not only of visible nodes,
but also hidden nodes and previously known nodes, and so on. |
| erlang:set_cookie(Node, Cookie) |
Sets the magic cookie used
when connecting to Node. If Node is the
current node, Cookie is used when connecting to
all new nodes. |
| spawn[_link|_opt](Node, Fun) |
Creates a process at a remote node. |
| spawn[_link|opt](Node, Module, FunctionName, Args) |
Creates a process at a remote node. |
Distribution BIFs
Distribution Command-Line Flags
Examples of command-line flags used for distributed programming
(for more information, see the erl(1)
manual page in ERTS:
| Command-Line Flag |
Description |
| -connect_all false |
Only explicit connection
set-ups are used. |
| -hidden |
Makes a node into a hidden node. |
| -name Name |
Makes a runtime system into a node, using long node names. |
| -setcookie Cookie |
Same as calling erlang:set_cookie(node(), Cookie). |
| -sname Name |
Makes a runtime system into a node, using short node names. |
Distribution Command-Line Flags
Distribution Modules
Examples of modules useful for distributed programming:
In the Kernel application:
| Module |
Description |
| global |
A global name registration facility. |
| global_group |
Grouping nodes to global name registration groups. |
| net_adm |
Various Erlang net administration routines. |
| net_kernel |
Erlang networking kernel. |
Kernel Modules Useful For Distribution.
In the STDLIB application:
| Module |
Description |
| slave |
Start and control of slave nodes. |
STDLIB Modules Useful For Distribution.