aboutsummaryrefslogtreecommitdiffstats
path: root/erts/doc/src/alt_dist.xml
diff options
context:
space:
mode:
authorErlang/OTP <[email protected]>2009-11-20 14:54:40 +0000
committerErlang/OTP <[email protected]>2009-11-20 14:54:40 +0000
commit84adefa331c4159d432d22840663c38f155cd4c1 (patch)
treebff9a9c66adda4df2106dfd0e5c053ab182a12bd /erts/doc/src/alt_dist.xml
downloadotp-84adefa331c4159d432d22840663c38f155cd4c1.tar.gz
otp-84adefa331c4159d432d22840663c38f155cd4c1.tar.bz2
otp-84adefa331c4159d432d22840663c38f155cd4c1.zip
The R13B03 release.OTP_R13B03
Diffstat (limited to 'erts/doc/src/alt_dist.xml')
-rw-r--r--erts/doc/src/alt_dist.xml1099
1 files changed, 1099 insertions, 0 deletions
diff --git a/erts/doc/src/alt_dist.xml b/erts/doc/src/alt_dist.xml
new file mode 100644
index 0000000000..9a68b3cf40
--- /dev/null
+++ b/erts/doc/src/alt_dist.xml
@@ -0,0 +1,1099 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+ <header>
+ <copyright>
+ <year>2000</year><year>2009</year>
+ <holder>Ericsson AB. All Rights Reserved.</holder>
+ </copyright>
+ <legalnotice>
+ The contents of this file are subject to the Erlang Public License,
+ Version 1.1, (the "License"); you may not use this file except in
+ compliance with the License. You should have received a copy of the
+ Erlang Public License along with this software. If not, it can be
+ retrieved online at http://www.erlang.org/.
+
+ Software distributed under the License is distributed on an "AS IS"
+ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ the License for the specific language governing rights and limitations
+ under the License.
+
+ </legalnotice>
+
+ <title>How to implement an alternative carrier for the Erlang distribution</title>
+ <prepared>Patrik Nyblom</prepared>
+ <responsible></responsible>
+ <docno></docno>
+ <approved></approved>
+ <checked></checked>
+ <date>2000-10-17</date>
+ <rev>PA2</rev>
+ <file>alt_dist.xml</file>
+ </header>
+ <p>This document describes how one can implement ones own carrier
+ protocol for the Erlang distribution. The distribution is normally
+ carried by the TCP/IP protocol. What's explained here is the method for
+ replacing TCP/IP with another protocol. </p>
+ <p>The document is a step by step explanation of the <c><![CDATA[uds_dist]]></c> example
+ application (seated in the kernel applications <c><![CDATA[examples]]></c> directory).
+ The <c><![CDATA[uds_dist]]></c> application implements distribution over Unix domain
+ sockets and is written for the Sun Solaris 2 operating environment. The
+ mechanisms are however general and applies to any operating system Erlang
+ runs on. The reason the C code is not made portable, is simply readability.</p>
+ <note><p>This document was written a long time ago. Most of it is still
+ valid, but some things have changed since it was first written.
+ Most notably the driver interface. There have been some updates
+ to the documentation of the driver presented in this documentation,
+ but more could be done and are planned for the future. The
+ reader is encouraged to also read the
+ <seealso marker="erl_driver">erl_driver</seealso>, and the
+ <seealso marker="erl_driver">driver_entry</seealso> documentation.
+ </p></note>
+
+ <section>
+ <title>Introduction</title>
+ <p>To implement a new carrier for the Erlang distribution, one must first
+ make the protocol available to the Erlang machine, which involves writing
+ an Erlang driver. There is no way one can use a port program,
+ there <em>has</em> to
+ be an Erlang driver. Erlang drivers can either be statically
+ linked
+ to the emulator, which can be an alternative when using the open source
+ distribution of Erlang, or dynamically loaded into the Erlang machines
+ address space, which is the only alternative if a precompiled version of
+ Erlang is to be used. </p>
+ <p>Writing an Erlang driver is by no means easy. The driver is written
+ as a couple of call-back functions called by the Erlang emulator when
+ data is sent to the driver or the driver has any data available on a file
+ descriptor. As the driver call-back routines execute in the main
+ thread of the Erlang machine, the call-back functions can perform
+ no blocking activity whatsoever. The call-backs should only set up
+ file descriptors for waiting and/or read/write available data. All
+ I/O has to be non blocking. Driver call-backs are however executed
+ in sequence, why a global state can safely be updated within the
+ routines. </p>
+ <p>When the driver is implemented, one would preferably write an
+ Erlang interface for the driver to be able to test the
+ functionality of the driver separately. This interface can then
+ be used by the distribution module which will cover the details of
+ the protocol from the <c><![CDATA[net_kernel]]></c>. The easiest path is to
+ mimic the <c><![CDATA[inet]]></c> and <c><![CDATA[inet_tcp]]></c> interfaces, but a lot of
+ functionality in those modules need not be implemented. In the
+ example application, only a few of the usual interfaces are
+ implemented, and they are much simplified.</p>
+ <p>When the protocol is available to Erlang through a driver and an
+ Erlang interface module, a distribution module can be
+ written. The distribution module is a module with well defined
+ call-backs, much like a <c><![CDATA[gen_server]]></c> (there is no compiler support
+ for checking the call-backs though). The details of finding other
+ nodes (i.e. talking to epmd or something similar), creating a
+ listen port (or similar), connecting to other nodes and performing
+ the handshakes/cookie verification are all implemented by this
+ module. There is however a utility module, <c><![CDATA[dist_util]]></c>, that
+ will do most of the hard work of handling handshakes, cookies,
+ timers and ticking. Using <c><![CDATA[dist_util]]></c> makes implementing a
+ distribution module much easier and that's what we are doing in
+ the example application.</p>
+ <p>The last step is to create boot scripts to make the protocol
+ implementation available at boot time. The implementation can be
+ debugged by starting the distribution when all of the system is
+ running, but in a real system the distribution should start very
+ early, why a boot-script and some command line parameters are
+ necessary. This last step also implies that the Erlang code in the
+ interface and distribution modules is written in such a way that
+ it can be run in the startup phase. Most notably there can be no
+ calls to the <c><![CDATA[application]]></c> module or to any modules not
+ loaded at boot-time (i.e. only <c><![CDATA[kernel]]></c>, <c><![CDATA[stdlib]]></c> and the
+ application itself can be used).</p>
+ </section>
+
+ <section>
+ <title>The driver</title>
+ <p>Although Erlang drivers in general may be beyond the scope of this
+ document, a brief introduction seems to be in place.</p>
+
+ <section>
+ <title>Drivers in general</title>
+ <p>An Erlang driver is a native code module written in C (or
+ assembler) which serves as an interface for some special operating
+ system service. This is a general mechanism that is used
+ throughout the Erlang emulator for all kinds of I/O. An Erlang
+ driver can be dynamically linked (or loaded) to the Erlang
+ emulator at runtime by using the <c><![CDATA[erl_ddll]]></c> Erlang
+ module. Some of the drivers in OTP are however statically linked
+ to the runtime system, but that's more an optimization than a
+ necessity.</p>
+ <p>The driver data-types and the functions available to the driver
+ writer are defined in the header file <c><![CDATA[erl_driver.h]]></c> (there
+ is also an deprecated version called <c><![CDATA[driver.h]]></c>, don't use
+ that one.) seated in Erlang's include directory (and in
+ $ERL_TOP/erts/emulator/beam in the source code
+ distribution). Refer to that file for function prototypes etc.</p>
+ <p>When writing a driver to make a communications protocol available
+ to Erlang, one should know just about everything worth knowing
+ about that particular protocol. All operation has to be non
+ blocking and all possible situations should be accounted for in
+ the driver. A non stable driver will affect and/or crash the
+ whole Erlang runtime system, which is seldom what's wanted. </p>
+ <p>The emulator calls the driver in the following situations:</p>
+ <list type="bulleted">
+ <item>When the driver is loaded. This call-back has to have a
+ special name and will inform the emulator of what call-backs should
+ be used by returning a pointer to a <c><![CDATA[ErlDrvEntry]]></c> struct,
+ which should be properly filled in (see below).</item>
+ <item>When a port to the driver is opened (by a <c><![CDATA[open_port]]></c>
+ call from Erlang). This routine should set up internal data
+ structures and return an opaque data entity of the type
+ <c><![CDATA[ErlDrvData]]></c>, which is a data-type large enough to hold a
+ pointer. The pointer returned by this function will be the first
+ argument to all other call-backs concerning this particular
+ port. It is usually called the port handle. The emulator only
+ stores the handle and does never try to interpret it, why it can
+ be virtually anything (well anything not larger than a pointer
+ that is) and can point to anything if it is a pointer. Usually
+ this pointer will refer to a structure holding information about
+ the particular port, as i t does in our example.</item>
+ <item>When an Erlang process sends data to the port. The data will
+ arrive as a buffer of bytes, the interpretation is not defined,
+ but is up to the implementor. This call-back returns nothing to the
+ caller, answers are sent to the caller as messages (using a
+ routine called <c><![CDATA[driver_output]]></c> available to all
+ drivers). There is also a way to talk in a synchronous way to
+ drivers, described below. There can be an additional call-back
+ function for handling data that is fragmented (sent in a deep
+ io-list). That interface will get the data in a form suitable for
+ Unix <c><![CDATA[writev]]></c> rather than in a single buffer. There is no
+ need for a distribution driver to implement such a call-back, so
+ we wont.</item>
+ <item>When a file descriptor is signaled for input. This call-back
+ is called when the emulator detects input on a file descriptor
+ which the driver has marked for monitoring by using the interface
+ <c><![CDATA[driver_select]]></c>. The mechanism of driver select makes it
+ possible to read non blocking from file descriptors by calling
+ <c><![CDATA[driver_select]]></c> when reading is needed and then do the actual
+ reading in this call-back (when reading is actually possible). The
+ typical scenario is that <c><![CDATA[driver_select]]></c> is called when an
+ Erlang process orders a read operation, and that this routine
+ sends the answer when data is available on the file descriptor.</item>
+ <item>When a file descriptor is signaled for output. This call-back
+ is called in a similar way as the previous, but when writing to a
+ file descriptor is possible. The usual scenario is that Erlang
+ orders writing on a file descriptor and that the driver calls
+ <c><![CDATA[driver_select]]></c>. When the descriptor is ready for output,
+ this call-back is called an the driver can try to send the
+ output. There may of course be queuing involved in such
+ operations, and there are some convenient queue routines available
+ to the driver writer to use in such situations.</item>
+ <item>When a port is closed, either by an Erlang process or by the
+ driver calling one of the <c><![CDATA[driver_failure_XXX]]></c> routines. This
+ routine should clean up everything connected to one particular
+ port. Note that when other call-backs call a
+ <c><![CDATA[driver_failure_XXX]]></c> routine, this routine will be
+ immediately called and the call-back routine issuing the error can
+ make no more use of the data structures for the port, as this
+ routine surely has freed all associated data and closed all file
+ descriptors. If the queue utility available to driver writes is
+ used, this routine will however <em>not</em> be called until the
+ queue is empty.</item>
+ <item>When an Erlang process calls <c>erlang:port_control/3</c>,
+ which is a synchronous interface to drivers. The control interface
+ is used to set driver options, change states of ports etc. We'll
+ use this interface quite a lot in our example.</item>
+ <item>When a timer expires. The driver can set timers with the
+ function <c><![CDATA[driver_set_timer]]></c>. When such timers expire, a
+ specific call-back function is called. We will not use timers in
+ our example.</item>
+ <item>When the whole driver is unloaded. Every resource allocated
+ by the driver should be freed.</item>
+ </list>
+ </section>
+
+ <section>
+ <title>The distribution driver's data structures</title>
+ <p>The driver used for Erlang distribution should implement a
+ reliable, order maintaining, variable length packet oriented
+ protocol. All error correction, re-sending and such need to be
+ implemented in the driver or by the underlying communications
+ protocol. If the protocol is stream oriented (as is the case with
+ both TCP/IP and our streamed Unix domain sockets), some mechanism
+ for packaging is needed. We will use the simple method of having a
+ header of four bytes containing the length of the package in a big
+ endian 32 bit integer (as Unix domain sockets only can be used
+ between processes on the same machine, we actually don't need to
+ code the integer in some special endianess, but I'll do it anyway
+ because in most situation you do need to do it. Unix domain
+ sockets are reliable and order maintaining, so we don't need to
+ implement resends and such in our driver.</p>
+ <p>Lets start writing our example Unix domain sockets driver by
+ declaring prototypes and filling in a static ErlDrvEntry
+ structure.</p>
+ <code type="none"><![CDATA[
+( 1) #include <stdio.h>
+( 2) #include <stdlib.h>
+( 3) #include <string.h>
+( 4) #include <unistd.h>
+( 5) #include <errno.h>
+( 6) #include <sys/types.h>
+( 7) #include <sys/stat.h>
+( 8) #include <sys/socket.h>
+( 9) #include <sys/un.h>
+(10) #include <fcntl.h>
+
+(11) #define HAVE_UIO_H
+(12) #include "erl_driver.h"
+
+(13) /*
+(14) ** Interface routines
+(15) */
+(16) static ErlDrvData uds_start(ErlDrvPort port, char *buff);
+(17) static void uds_stop(ErlDrvData handle);
+(18) static void uds_command(ErlDrvData handle, char *buff, int bufflen);
+(19) static void uds_input(ErlDrvData handle, ErlDrvEvent event);
+(20) static void uds_output(ErlDrvData handle, ErlDrvEvent event);
+(21) static void uds_finish(void);
+(22) static int uds_control(ErlDrvData handle, unsigned int command,
+(23) char* buf, int count, char** res, int res_size);
+
+(24) /* The driver entry */
+(25) static ErlDrvEntry uds_driver_entry = {
+(26) NULL, /* init, N/A */
+(27) uds_start, /* start, called when port is opened */
+(28) uds_stop, /* stop, called when port is closed */
+(29) uds_command, /* output, called when erlang has sent */
+(30) uds_input, /* ready_input, called when input
+(31) descriptor ready */
+(32) uds_output, /* ready_output, called when output
+(33) descriptor ready */
+(34) "uds_drv", /* char *driver_name, the argument
+(35) to open_port */
+(36) uds_finish, /* finish, called when unloaded */
+(37) NULL, /* void * that is not used (BC) */
+(38) uds_control, /* control, port_control callback */
+(39) NULL, /* timeout, called on timeouts */
+(40) NULL, /* outputv, vector output interface */
+(41) NULL, /* ready_async callback */
+(42) NULL, /* flush callback */
+(43) NULL, /* call callback */
+(44) NULL, /* event callback */
+(45) ERL_DRV_EXTENDED_MARKER, /* Extended driver interface marker */
+(46) ERL_DRV_EXTENDED_MAJOR_VERSION, /* Major version number */
+(47) ERL_DRV_EXTENDED_MINOR_VERSION, /* Minor version number */
+(48) ERL_DRV_FLAG_SOFT_BUSY, /* Driver flags. Soft busy flag is
+(49) required for distribution drivers */
+(50) NULL, /* Reserved for internal use */
+(51) NULL, /* process_exit callback */
+(52) NULL /* stop_select callback */
+(53) };]]></code>
+ <p>On line 1 to 10 we have included the OS headers needed for our
+ driver. As this driver is written for Solaris, we know that the
+ header <c><![CDATA[uio.h]]></c> exists, why we can define the preprocessor
+ variable <c><![CDATA[HAVE_UIO_H]]></c> before we include <c><![CDATA[erl_driver.h]]></c>
+ at line 12. The definition of <c><![CDATA[HAVE_UIO_H]]></c> will make the
+ I/O vectors used in Erlang's driver queues to correspond to the
+ operating systems ditto, which is very convenient.</p>
+ <p>The different call-back functions are declared ("forward
+ declarations") on line 16 to 23.</p>
+ <p>The driver structure is similar for statically linked in
+ drivers and dynamically loaded. However some of the fields
+ should be left empty (i.e. initialized to NULL) in the
+ different types of drivers. The first field (the <c><![CDATA[init]]></c>
+ function pointer) is always left blank in a dynamically loaded
+ driver, which can be seen on line 26. The NULL on line 37
+ should always be there, the field is no longer used and is
+ retained for backward compatibility. We use no timers in this
+ driver, why no call-back for timers is needed. The <c>outputv</c> field
+ (line 40) can be used to implement an interface similar to
+ Unix <c><![CDATA[writev]]></c> for output. The Erlang runtime
+ system could previously not use <c>outputv</c> for the
+ distribution, but since erts version 5.7.2 it can.
+ Since this driver was written before erts version 5.7.2 it does
+ not use the <c>outputv</c> callback. Using the <c>outputv</c>
+ callback is preferred since it reduces copying of data. (We
+ will however use scatter/gather I/O internally in the driver).</p>
+ <p>As of erts version 5.5.3 the driver interface was extended with
+ version control and the possibility to pass capability information.
+ Capability flags are present at line 48. As of erts version 5.7.4
+ the
+ <seealso marker="driver_entry#driver_flags">ERL_DRV_FLAG_SOFT_BUSY</seealso>
+ flag is required for drivers that are to be used by the distribution.
+ The soft busy flag implies that the driver is capable of handling
+ calls to the <c>output</c> and <c>outputv</c> callbacks even though
+ it has marked itself as busy. This has always been a requirement
+ on drivers used by the distribution, but there have previously not
+ been any capability information available about this. For more
+ information see
+ <seealso marker="erl_driver#set_busy_port">set_busy_port()</seealso>).
+</p>
+ <p>This driver was written before the runtime system had SMP support.
+ The driver will still function in the runtime system with SMP support,
+ but performance will suffer from lock contention on the driver lock
+ used for the driver. This can be alleviated by reviewing and perhaps
+ rewriting the code so that each instance of the driver safely can
+ execute in parallel. When instances safely can execute in parallel it
+ is safe to enable instance specific locking on the driver. This is done
+ by passing
+ <seealso marker="driver_entry#driver_flags">ERL_DRV_FLAG_USE_PORT_LOCKING</seealso>
+ as a driver flag. This is left as an exercise for the reader.</p>
+ <p>Our defined call-backs thus are:</p>
+ <list type="bulleted">
+ <item>uds_start, which shall initiate data for a port. We wont
+ create any actual sockets here, just initialize data structures.</item>
+ <item>uds_stop, the function called when a port is closed.</item>
+ <item>uds_command, which will handle messages from Erlang. The
+ messages can either be plain data to be sent or more subtle
+ instructions to the driver. We will use this function mostly for
+ data pumping.</item>
+ <item>uds_input, this is the call-back which is called when we have
+ something to read from a socket.</item>
+ <item>uds_output, this is the function called when we can write to a
+ socket.</item>
+ <item>uds_finish, which is called when the driver is unloaded. A
+ distribution driver will actually (or hopefully) never be unloaded,
+ but we include this for completeness. Being able to clean up after
+ oneself is always a good thing.</item>
+ <item>uds_control, the <c>erlang:port_control/2</c> call-back, which
+ will be used a lot in this implementation.</item>
+ </list>
+ <p>The ports implemented by this driver will operate in two major
+ modes, which i will call the <em>command</em> and <em>data</em>
+ modes. In command mode, only passive reading and writing (like
+ gen_tcp:recv/gen_tcp:send) can be
+ done, and this is the mode the port will be in during the
+ distribution handshake. When the connection is up, the port will
+ be switched to data mode and all data will be immediately read and
+ passed further to the Erlang emulator. In data mode, no data
+ arriving to the uds_command will be interpreted, but just packaged
+ and sent out on the socket. The uds_control call-back will do the
+ switching between those two modes.</p>
+ <p>While the <c><![CDATA[net_kernel]]></c> informs different subsystems that the
+ connection is coming up, the port should accept data to send, but
+ not receive any data, to avoid that data arrives from another node
+ before every kernel subsystem is prepared to handle it. We have a
+ third mode for this intermediate stage, lets call it the
+ <em>intermediate</em> mode.</p>
+ <p>Lets define an enum for the different types of ports we have:</p>
+ <code type="none"><![CDATA[
+( 1) typedef enum {
+( 2) portTypeUnknown, /* An uninitialized port */
+( 3) portTypeListener, /* A listening port/socket */
+( 4) portTypeAcceptor, /* An intermidiate stage when accepting
+( 5) on a listen port */
+( 6) portTypeConnector, /* An intermediate stage when connecting */
+( 7) portTypeCommand, /* A connected open port in command mode */
+( 8) portTypeIntermediate, /* A connected open port in special
+( 9) half active mode */
+(10) portTypeData /* A connectec open port in data mode */
+(11) } PortType; ]]></code>
+ <p>Lets look at the different types:</p>
+ <list type="bulleted">
+ <item>portTypeUnknown - The type a port has when it's opened, but
+ not actually bound to any file descriptor.</item>
+ <item>portTypeListener - A port that is connected to a listen
+ socket. This port will not do especially much, there will be no data
+ pumping done on this socket, but there will be read data available
+ when one is trying to do an accept on the port.</item>
+ <item>portTypeAcceptor - This is a port that is to represent the
+ result of an accept operation. It is created when one wants to
+ accept from a listen socket, and it will be converted to a
+ portTypeCommand when the accept succeeds.</item>
+ <item>portTypeConnector - Very similar to portTypeAcceptor, an
+ intermediate stage between the request for a connect operation and
+ that the socket is really connected to an accepting ditto in the
+ other end. As soon as the sockets are connected, the port will
+ switch type to portTypeCommand.</item>
+ <item>portTypeCommand - A connected socket (or accepted socket if
+ you want) that is in the command mode mentioned earlier.</item>
+ <item>portTypeIntermediate - The intermediate stage for a connected
+ socket. There should be no processing of input for this socket.</item>
+ <item>portTypeData - The mode where data is pumped through the port
+ and the uds_command routine will regard every call as a call where
+ sending is wanted. In this mode all input available will be read and
+ sent to Erlang as soon as it arrives on the socket, much like in the
+ active mode of a <c><![CDATA[gen_tcp]]></c> socket.</item>
+ </list>
+ <p>Now lets look at the state we'll need for our ports. One can note
+ that not all fields are used for all types of ports and that one
+ could save some space by using unions, but that would clutter the
+ code with multiple indirections, so i simply use one struct for
+ all types of ports, for readability.</p>
+ <code type="none"><![CDATA[
+( 1) typedef unsigned char Byte;
+( 2) typedef unsigned int Word;
+
+( 3) typedef struct uds_data {
+( 4) int fd; /* File descriptor */
+( 5) ErlDrvPort port; /* The port identifier */
+( 6) int lockfd; /* The file descriptor for a lock file in
+( 7) case of listen sockets */
+( 8) Byte creation; /* The creation serial derived from the
+( 9) lockfile */
+(10) PortType type; /* Type of port */
+(11) char *name; /* Short name of socket for unlink */
+(12) Word sent; /* Bytes sent */
+(13) Word received; /* Bytes received */
+(14) struct uds_data *partner; /* The partner in an accept/listen pair */
+(15) struct uds_data *next; /* Next structure in list */
+(16) /* The input buffer and it's data */
+(17) int buffer_size; /* The allocated size of the input buffer */
+(18) int buffer_pos; /* Current position in input buffer */
+(19) int header_pos; /* Where the current header is in the
+(20) input buffer */
+(21) Byte *buffer; /* The actual input buffer */
+(22) } UdsData; ]]></code>
+ <p>This structure is used for all types of ports although some
+ fields are useless for some types. The least memory consuming
+ solution would be to arrange this structure as a union of
+ structures, but the multiple indirections in the code to
+ access a field in such a structure will clutter the code to
+ much for an example.</p>
+ <p>Let's look at the fields in our structure:</p>
+ <list type="bulleted">
+ <item>fd - The file descriptor of the socket associated with the
+ port.</item>
+ <item>port - The port identifier for the port which this structure
+ corresponds to. It is needed for most <c><![CDATA[driver_XXX]]></c>
+ calls from the driver back to the emulator.</item>
+ <item>
+ <p>lockfd - If the socket is a listen socket, we use a separate
+ (regular) file for two purposes:</p>
+ <list type="bulleted">
+ <item>We want a locking mechanism that gives no race
+ conditions, so that we can be sure of if another Erlang
+ node uses the listen socket name we require or if the
+ file is only left there from a previous (crashed)
+ session.</item>
+ <item>
+ <p>We store the <em>creation</em> serial number in the
+ file. The <em>creation</em> is a number that should
+ change between different instances of different Erlang
+ emulators with the same name, so that process
+ identifiers from one emulator won't be valid when sent
+ to a new emulator with the same distribution name. The
+ creation can be between 0 and 3 (two bits) and is stored
+ in every process identifier sent to another node. </p>
+ <p>In a system with TCP based distribution, this data is
+ kept in the <em>Erlang port mapper daemon</em>
+ (<c><![CDATA[epmd]]></c>), which is contacted when a distributed
+ node starts. The lock-file and a convention for the UDS
+ listen socket's name will remove the need for
+ <c><![CDATA[epmd]]></c> when using this distribution module. UDS
+ is always restricted to one host, why avoiding a port
+ mapper is easy.</p>
+ </item>
+ </list>
+ </item>
+ <item>creation - The creation number for a listen socket, which is
+ calculated as (the value found in the lock-file + 1) rem
+ 4. This creation value is also written back into the
+ lock-file, so that the next invocation of the emulator will
+ found our value in the file.</item>
+ <item>type - The current type/state of the port, which can be one
+ of the values declared above.</item>
+ <item>name - The name of the socket file (the path prefix
+ removed), which allows for deletion (<c><![CDATA[unlink]]></c>) when the
+ socket is closed.</item>
+ <item>sent - How many bytes that have been sent over the
+ socket. This may wrap, but that's no problem for the
+ distribution, as the only thing that interests the Erlang
+ distribution is if this value has changed (the Erlang
+ net_kernel <em>ticker</em> uses this value by calling the
+ driver to fetch it, which is done through the
+ <c>erlang:port_control</c> routine).</item>
+ <item>received - How many bytes that are read (received) from the
+ socket, used in similar ways as <c><![CDATA[sent]]></c>.</item>
+ <item>partner - A pointer to another port structure, which is
+ either the listen port from which this port is accepting a
+ connection or the other way around. The "partner relation"
+ is always bidirectional.</item>
+ <item>next - Pointer to next structure in a linked list of all
+ port structures. This list is used when accepting
+ connections and when the driver is unloaded.</item>
+ <item>buffer_size, buffer_pos, header_pos, buffer - data for input
+ buffering. Refer to the source code (in the kernel/examples
+ directory) for details about the input buffering. That
+ certainly goes beyond the scope of this document.</item>
+ </list>
+ </section>
+
+ <section>
+ <title>Selected parts of the distribution driver implementation</title>
+ <p>The distribution drivers implementation is not completely
+ covered in this text, details about buffering and other things
+ unrelated to driver writing are not explained. Likewise are
+ some peculiarities of the UDS protocol not explained in
+ detail. The chosen protocol is not important.</p>
+ <p>Prototypes for the driver call-back routines can be found in
+ the <c><![CDATA[erl_driver.h]]></c> header file.</p>
+ <p>The driver initialization routine is (usually) declared with a
+ macro to make the driver easier to port between different
+ operating systems (and flavours of systems). This is the only
+ routine that has to have a well defined name. All other
+ call-backs are reached through the driver structure. The macro
+ to use is named <c><![CDATA[DRIVER_INIT]]></c> and takes the driver name
+ as parameter.</p>
+ <code type="none"><![CDATA[
+(1) /* Beginning of linked list of ports */
+(2) static UdsData *first_data;
+
+
+(3) DRIVER_INIT(uds_drv)
+(4) {
+(5) first_data = NULL;
+(6) return &uds_driver_entry;
+(7) } ]]></code>
+ <p>The routine initializes the single global data structure and
+ returns a pointer to the driver entry. The routine will be
+ called when <c><![CDATA[erl_ddll:load_driver]]></c> is called from Erlang.</p>
+ <p>The <c><![CDATA[uds_start]]></c> routine is called when a port is opened
+ from Erlang. In our case, we only allocate a structure and
+ initialize it. Creating the actual socket is left to the
+ <c><![CDATA[uds_command]]></c> routine.</p>
+ <code type="none"><![CDATA[
+( 1) static ErlDrvData uds_start(ErlDrvPort port, char *buff)
+( 2) {
+( 3) UdsData *ud;
+( 4)
+( 5) ud = ALLOC(sizeof(UdsData));
+( 6) ud->fd = -1;
+( 7) ud->lockfd = -1;
+( 8) ud->creation = 0;
+( 9) ud->port = port;
+(10) ud->type = portTypeUnknown;
+(11) ud->name = NULL;
+(12) ud->buffer_size = 0;
+(13) ud->buffer_pos = 0;
+(14) ud->header_pos = 0;
+(15) ud->buffer = NULL;
+(16) ud->sent = 0;
+(17) ud->received = 0;
+(18) ud->partner = NULL;
+(19) ud->next = first_data;
+(20) first_data = ud;
+(21)
+(22) return((ErlDrvData) ud);
+(23) } ]]></code>
+ <p>Every data item is initialized, so that no problems will arise
+ when a newly created port is closed (without there being any
+ corresponding socket). This routine is called when
+ <c><![CDATA[open_port({spawn, "uds_drv"},[])]]></c> is called from Erlang.</p>
+ <p>The <c><![CDATA[uds_command]]></c> routine is the routine called when an
+ Erlang process sends data to the port. All asynchronous
+ commands when the port is in <em>command mode</em> as well as
+ the sending of all data when the port is in <em>data mode</em>
+ is handled in this9s routine. Let's have a look at it:</p>
+ <code type="none"><![CDATA[
+( 1) static void uds_command(ErlDrvData handle, char *buff, int bufflen)
+( 2) {
+( 3) UdsData *ud = (UdsData *) handle;
+
+( 4) if (ud->type == portTypeData || ud->type == portTypeIntermediate) {
+( 5) DEBUGF(("Passive do_send %d",bufflen));
+( 6) do_send(ud, buff + 1, bufflen - 1); /* XXX */
+( 7) return;
+( 8) }
+( 9) if (bufflen == 0) {
+(10) return;
+(11) }
+(12) switch (*buff) {
+(13) case 'L':
+(14) if (ud->type != portTypeUnknown) {
+(15) driver_failure_posix(ud->port, ENOTSUP);
+(16) return;
+(17) }
+(18) uds_command_listen(ud,buff,bufflen);
+(19) return;
+(20) case 'A':
+(21) if (ud->type != portTypeUnknown) {
+(22) driver_failure_posix(ud->port, ENOTSUP);
+(23) return;
+(24) }
+(25) uds_command_accept(ud,buff,bufflen);
+(26) return;
+(27) case 'C':
+(28) if (ud->type != portTypeUnknown) {
+(29) driver_failure_posix(ud->port, ENOTSUP);
+(30) return;
+(31) }
+(32) uds_command_connect(ud,buff,bufflen);
+(33) return;
+(34) case 'S':
+(35) if (ud->type != portTypeCommand) {
+(36) driver_failure_posix(ud->port, ENOTSUP);
+(37) return;
+(38) }
+(39) do_send(ud, buff + 1, bufflen - 1);
+(40) return;
+(41) case 'R':
+(42) if (ud->type != portTypeCommand) {
+(43) driver_failure_posix(ud->port, ENOTSUP);
+(44) return;
+(45) }
+(46) do_recv(ud);
+(47) return;
+(48) default:
+(49) return;
+(50) }
+(51) } ]]></code>
+ <p>The command routine takes three parameters; the handle
+ returned for the port by <c><![CDATA[uds_start]]></c>, which is a pointer
+ to the internal port structure, the data buffer and the length
+ of the data buffer. The buffer is the data sent from Erlang
+ (a list of bytes) converted to an C array (of bytes). </p>
+ <p>If Erlang sends i.e. the list <c><![CDATA[[$a,$b,$c]]]></c> to the port,
+ the <c><![CDATA[bufflen]]></c> variable will be <c><![CDATA[3]]></c> ant the
+ <c><![CDATA[buff]]></c> variable will contain <c><![CDATA[{'a','b','c'}]]></c> (no
+ null termination). Usually the first byte is used as an
+ opcode, which is the case in our driver to (at least when the
+ port is in command mode). The opcodes are defined as:</p>
+ <list type="bulleted">
+ <item>'L'&lt;socketname&gt;: Create and listen on socket with the
+ given name.</item>
+ <item>'A'&lt;listennumber as 32 bit bigendian&gt;: Accept from the
+ listen socket identified by the given identification
+ number. The identification number is retrieved with the
+ uds_control routine.</item>
+ <item>'C'&lt;socketname&gt;: Connect to the socket named
+ &lt;socketname&gt;.</item>
+ <item>'S'&lt;data&gt;: Send the data &lt;data&gt; on the
+ connected/accepted socket (in command mode). The sending is
+ acked when the data has left this process.</item>
+ <item>'R': Receive one packet of data.</item>
+ </list>
+ <p>One may wonder what is meant by "one packet of data" in the
+ 'R' command. This driver always sends data packeted with a 4
+ byte header containing a big endian 32 bit integer that
+ represents the length of the data in the packet. There is no
+ need for different packet sizes or some kind of streamed
+ mode, as this driver is for the distribution only. One may
+ wonder why the header word is coded explicitly in big endian
+ when an UDS socket is local to the host. The answer simply is
+ that I see it as a good practice when writing a distribution
+ driver, as distribution in practice usually cross the host
+ boundaries. </p>
+ <p>On line 4-8 we handle the case where the port is in data or
+ intermediate mode, the rest of the routine handles the
+ different commands. We see (first on line 15) that the routine
+ uses the <c><![CDATA[driver_failure_posix()]]></c> routine to report
+ errors. One important thing to remember is that the failure
+ routines make a call to our <c><![CDATA[uds_stop]]></c> routine, which
+ will remove the internal port data. The handle (and the casted
+ handle <c><![CDATA[ud]]></c>) is therefore <em>invalid pointers</em> after a
+ <c><![CDATA[driver_failure]]></c> call and we should <em>immediately return</em>. The runtime system will send exit signals to all
+ linked processes.</p>
+ <p>The uds_input routine gets called when data is available on a
+ file descriptor previously passed to the <c><![CDATA[driver_select]]></c>
+ routine. Typically this happens when a read command is issued
+ and no data is available. Lets look at the <c><![CDATA[do_recv]]></c>
+ routine:</p>
+ <code type="none"><![CDATA[
+( 1) static void do_recv(UdsData *ud)
+( 2) {
+( 3) int res;
+( 4) char *ibuf;
+( 5) for(;;) {
+( 6) if ((res = buffered_read_package(ud,&ibuf)) < 0) {
+( 7) if (res == NORMAL_READ_FAILURE) {
+( 8) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 1);
+( 9) } else {
+(10) driver_failure_eof(ud->port);
+(11) }
+(12) return;
+(13) }
+(14) /* Got a package */
+(15) if (ud->type == portTypeCommand) {
+(16) ibuf[-1] = 'R'; /* There is always room for a single byte
+(17) opcode before the actual buffer
+(18) (where the packet header was) */
+(19) driver_output(ud->port,ibuf - 1, res + 1);
+(20) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ,0);
+(21) return;
+(22) } else {
+(23) ibuf[-1] = DIST_MAGIC_RECV_TAG; /* XXX */
+(24) driver_output(ud->port,ibuf - 1, res + 1);
+(25) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ,1);
+(26) }
+(27) }
+(28) } ]]></code>
+ <p>The routine tries to read data until a packet is read or the
+ <c><![CDATA[buffered_read_package]]></c> routine returns a
+ <c><![CDATA[NORMAL_READ_FAILURE]]></c> (an internally defined constant for
+ the module that means that the read operation resulted in an
+ <c><![CDATA[EWOULDBLOCK]]></c>). If the port is in command mode, the
+ reading stops when one package is read, but if it is in data
+ mode, the reading continues until the socket buffer is empty
+ (read failure). If no more data can be read and more is wanted
+ (always the case when socket is in data mode) driver_select is
+ called to make the <c><![CDATA[uds_input]]></c> call-back be called when
+ more data is available for reading.</p>
+ <p>When the port is in data mode, all data is sent to Erlang in a
+ format that suits the distribution, in fact the raw data will
+ never reach any Erlang process, but will be
+ translated/interpreted by the emulator itself and then
+ delivered in the correct format to the correct processes. In
+ the current emulator version, received data should be tagged
+ with a single byte of 100. Thats what the macro
+ <c><![CDATA[DIST_MAGIC_RECV_TAG]]></c> is defined to. The tagging of data
+ in the distribution will possibly change in the future.</p>
+ <p>The <c><![CDATA[uds_input]]></c> routine will handle other input events
+ (like nonblocking <c><![CDATA[accept]]></c>), but most importantly handle
+ data arriving at the socket by calling <c><![CDATA[do_recv]]></c>:</p>
+ <code type="none"><![CDATA[
+( 1) static void uds_input(ErlDrvData handle, ErlDrvEvent event)
+( 2) {
+( 3) UdsData *ud = (UdsData *) handle;
+
+( 4) if (ud->type == portTypeListener) {
+( 5) UdsData *ad = ud->partner;
+( 6) struct sockaddr_un peer;
+( 7) int pl = sizeof(struct sockaddr_un);
+( 8) int fd;
+
+( 9) if ((fd = accept(ud->fd, (struct sockaddr *) &peer, &pl)) < 0) {
+(10) if (errno != EWOULDBLOCK) {
+(11) driver_failure_posix(ud->port, errno);
+(12) return;
+(13) }
+(14) return;
+(15) }
+(16) SET_NONBLOCKING(fd);
+(17) ad->fd = fd;
+(18) ad->partner = NULL;
+(19) ad->type = portTypeCommand;
+(20) ud->partner = NULL;
+(21) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(22) driver_output(ad->port, "Aok",3);
+(23) return;
+(24) }
+(25) do_recv(ud);
+(26) } ]]></code>
+ <p>The important line here is the last line in the function, the
+ <c><![CDATA[do_read]]></c> routine is called to handle new input. The rest
+ of the function handles input on a listen socket, which means
+ that there should be possible to do an accept on the
+ socket, which is also recognized as a read event.</p>
+ <p>The output mechanisms are similar to the input. Lets first
+ look at the <c><![CDATA[do_send]]></c> routine:</p>
+ <code type="none"><![CDATA[
+( 1) static void do_send(UdsData *ud, char *buff, int bufflen)
+( 2) {
+( 3) char header[4];
+( 4) int written;
+( 5) SysIOVec iov[2];
+( 6) ErlIOVec eio;
+( 7) ErlDrvBinary *binv[] = {NULL,NULL};
+
+( 8) put_packet_length(header, bufflen);
+( 9) iov[0].iov_base = (char *) header;
+(10) iov[0].iov_len = 4;
+(11) iov[1].iov_base = buff;
+(12) iov[1].iov_len = bufflen;
+(13) eio.iov = iov;
+(14) eio.binv = binv;
+(15) eio.vsize = 2;
+(16) eio.size = bufflen + 4;
+(17) written = 0;
+(18) if (driver_sizeq(ud->port) == 0) {
+(19) if ((written = writev(ud->fd, iov, 2)) == eio.size) {
+(20) ud->sent += written;
+(21) if (ud->type == portTypeCommand) {
+(22) driver_output(ud->port, "Sok", 3);
+(23) }
+(24) return;
+(25) } else if (written < 0) {
+(26) if (errno != EWOULDBLOCK) {
+(27) driver_failure_eof(ud->port);
+(28) return;
+(29) } else {
+(30) written = 0;
+(31) }
+(32) } else {
+(33) ud->sent += written;
+(34) }
+(35) /* Enqueue remaining */
+(36) }
+(37) driver_enqv(ud->port, &eio, written);
+(38) send_out_queue(ud);
+(39) } ]]></code>
+ <p>This driver uses the <c><![CDATA[writev]]></c> system call to send data
+ onto the socket. A combination of writev and the driver output
+ queues is very convenient. An <em>ErlIOVec</em> structure
+ contains a <em>SysIOVec</em> (which is equivalent to the
+ <c><![CDATA[struct iovec]]></c> structure defined in <c><![CDATA[uio.h]]></c>. The
+ ErlIOVec also contains an array of <em>ErlDrvBinary</em>
+ pointers, of the same length as the number of buffers in the
+ I/O vector itself. One can use this to allocate the binaries
+ for the queue "manually" in the driver, but we'll just fill
+ the binary array with NULL values (line 7) , which will make
+ the runtime system allocate it's own buffers when we call
+ <c><![CDATA[driver_enqv]]></c> (line 37).</p>
+ <p></p>
+ <p>The routine builds an I/O vector containing the header bytes
+ and the buffer (the opcode has been removed and the buffer
+ length decreased by the output routine). If the queue is
+ empty, we'll write the data directly to the socket (or at
+ least try to). If any data is left, it is stored in the queue
+ and then we try to send the queue (line 38). An ack is sent
+ when the message is delivered completely (line 22). The
+ <c><![CDATA[send_out_queue]]></c> will send acks if the sending is
+ completed there. If the port is in command mode, the Erlang
+ code serializes the send operations so that only one packet
+ can be waiting for delivery at a time. Therefore the ack can
+ be sent simply whenever the queue is empty.</p>
+ <p></p>
+ <p>A short look at the <c><![CDATA[send_out_queue]]></c> routine:</p>
+ <code type="none"><![CDATA[
+( 1) static int send_out_queue(UdsData *ud)
+( 2) {
+( 3) for(;;) {
+( 4) int vlen;
+( 5) SysIOVec *tmp = driver_peekq(ud->port, &vlen);
+( 6) int wrote;
+( 7) if (tmp == NULL) {
+( 8) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_WRITE, 0);
+( 9) if (ud->type == portTypeCommand) {
+(10) driver_output(ud->port, "Sok", 3);
+(11) }
+(12) return 0;
+(13) }
+(14) if (vlen > IO_VECTOR_MAX) {
+(15) vlen = IO_VECTOR_MAX;
+(16) }
+(17) if ((wrote = writev(ud->fd, tmp, vlen)) < 0) {
+(18) if (errno == EWOULDBLOCK) {
+(19) driver_select(ud->port, (ErlDrvEvent) ud->fd,
+(20) DO_WRITE, 1);
+(21) return 0;
+(22) } else {
+(23) driver_failure_eof(ud->port);
+(24) return -1;
+(25) }
+(26) }
+(27) driver_deq(ud->port, wrote);
+(28) ud->sent += wrote;
+(29) }
+(30) } ]]></code>
+ <p>What we do is simply to pick out an I/O vector from the queue
+ (which is the whole queue as an <em>SysIOVec</em>). If the I/O
+ vector is to long (IO_VECTOR_MAX is defined to 16), the vector
+ length is decreased (line 15), otherwise the <c><![CDATA[writev]]></c>
+ (line 17) call will
+ fail. Writing is tried and anything written is dequeued (line
+ 27). If the write fails with <c><![CDATA[EWOULDBLOCK]]></c> (note that all
+ sockets are in nonblocking mode), <c><![CDATA[driver_select]]></c> is
+ called to make the <c><![CDATA[uds_output]]></c> routine be called when
+ there is space to write again.</p>
+ <p>We will continue trying to write until the queue is empty or
+ the writing would block.</p>
+ <p>The routine above are called from the <c><![CDATA[uds_output]]></c>
+ routine, which looks like this:</p>
+ <code type="none"><![CDATA[
+( 1) static void uds_output(ErlDrvData handle, ErlDrvEvent event)
+( 2) {
+( 3) UdsData *ud = (UdsData *) handle;
+( 4) if (ud->type == portTypeConnector) {
+( 5) ud->type = portTypeCommand;
+( 6) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_WRITE, 0);
+( 7) driver_output(ud->port, "Cok",3);
+( 8) return;
+( 9) }
+(10) send_out_queue(ud);
+(11) } ]]></code>
+ <p>The routine is simple, it first handles the fact that the
+ output select will concern a socket in the business of
+ connecting (and the connecting blocked). If the socket is in
+ a connected state it simply sends the output queue, this
+ routine is called when there is possible to write to a socket
+ where we have an output queue, so there is no question what to
+ do.</p>
+ <p>The driver implements a control interface, which is a
+ synchronous interface called when Erlang calls
+ <c><![CDATA[erlang:port_control/3]]></c>. This is the only interface
+ that can control the driver when it is in data mode and it may
+ be called with the following opcodes:</p>
+ <list type="bulleted">
+ <item>'C': Set port in command mode.</item>
+ <item>'I': Set port in intermediate mode.</item>
+ <item>'D': Set port in data mode.</item>
+ <item>'N': Get identification number for listen port, this
+ identification number is used in an accept command to the
+ driver, it is returned as a big endian 32 bit integer, which
+ happens to be the file identifier for the listen socket.</item>
+ <item>'S': Get statistics, which is the number of bytes received,
+ the number of bytes sent and the number of bytes pending in
+ the output queue. This data is used when the distribution
+ checks that a connection is alive (ticking). The statistics
+ is returned as 3 32 bit big endian integers.</item>
+ <item>'T': Send a tick message, which is a packet of length
+ 0. Ticking is done when the port is in data mode, so the
+ command for sending data cannot be used (besides it ignores
+ zero length packages in command mode). This is used by the
+ ticker to send dummy data when no other traffic is present.
+ <em>Note</em> that it is important that the interface for
+ sending ticks is not blocking. This implementation uses
+ <c>erlang:port_control/3</c> which does not block the caller.
+ If <c>erlang:port_command</c> is used, use
+ <c>erlang:port_command/3</c> and pass <c>[force]</c> as
+ option list; otherwise, the caller can be blocked indefinitely
+ on a busy port and prevent the system from taking down a
+ connection that is not functioning.</item>
+ <item>'R': Get creation number of listen socket, which is used to
+ dig out the number stored in the lock file to differentiate
+ between invocations of Erlang nodes with the same name.\011 </item>
+ </list>
+ <p>The control interface gets a buffer to return its value in,
+ but is free to allocate it's own buffer is the provided one is
+ to small. Here is the code for <c><![CDATA[uds_control]]></c>:</p>
+ <code type="none"><![CDATA[
+( 1) static int uds_control(ErlDrvData handle, unsigned int command,
+( 2) char* buf, int count, char** res, int res_size)
+( 3) {
+( 4) /* Local macro to ensure large enough buffer. */
+( 5) #define ENSURE(N) \\
+( 6) do { \\
+( 7) if (res_size < N) { \\
+( 8) *res = ALLOC(N); \\
+( 9) } \\
+(10) } while(0)
+
+(11) UdsData *ud = (UdsData *) handle;
+
+(12) switch (command) {
+(13) case 'S':
+(14) {
+(15) ENSURE(13);
+(16) **res = 0;
+(17) put_packet_length((*res) + 1, ud->received);
+(18) put_packet_length((*res) + 5, ud->sent);
+(19) put_packet_length((*res) + 9, driver_sizeq(ud->port));
+(20) return 13;
+(21) }
+(22) case 'C':
+(23) if (ud->type < portTypeCommand) {
+(24) return report_control_error(res, res_size, "einval");
+(25) }
+(26) ud->type = portTypeCommand;
+(27) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(28) ENSURE(1);
+(29) **res = 0;
+(30) return 1;
+(31) case 'I':
+(32) if (ud->type < portTypeCommand) {
+(33) return report_control_error(res, res_size, "einval");
+(34) }
+(35) ud->type = portTypeIntermediate;
+(36) driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(37) ENSURE(1);
+(38) **res = 0;
+(39) return 1;
+(40) case 'D':
+(41) if (ud->type < portTypeCommand) {
+(42) return report_control_error(res, res_size, "einval");
+(43) }
+(44) ud->type = portTypeData;
+(45) do_recv(ud);
+(46) ENSURE(1);
+(47) **res = 0;
+(48) return 1;
+(49) case 'N':
+(50) if (ud->type != portTypeListener) {
+(51) return report_control_error(res, res_size, "einval");
+(52) }
+(53) ENSURE(5);
+(54) (*res)[0] = 0;
+(55) put_packet_length((*res) + 1, ud->fd);
+(56) return 5;
+(57) case 'T': /* tick */
+(58) if (ud->type != portTypeData) {
+(59) return report_control_error(res, res_size, "einval");
+(60) }
+(61) do_send(ud,"",0);
+(62) ENSURE(1);
+(63) **res = 0;
+(64) return 1;
+(65) case 'R':
+(66) if (ud->type != portTypeListener) {
+(67) return report_control_error(res, res_size, "einval");
+(68) }
+(69) ENSURE(2);
+(70) (*res)[0] = 0;
+(71) (*res)[1] = ud->creation;
+(72) return 2;
+(73) default:
+(74) return report_control_error(res, res_size, "einval");
+(75) }
+(76) #undef ENSURE
+(77) } ]]></code>
+ <p>The macro <c><![CDATA[ENSURE]]></c> (line 5 to 10) is used to ensure that
+ the buffer is large enough for our answer. We switch on the
+ command and take actions, there is not much to say about this
+ routine. Worth noting is that we always has read select active
+ on a port in data mode (achieved by calling <c><![CDATA[do_recv]]></c> on
+ line 45), but turn off read selection in intermediate and
+ command modes (line 27 and 36).</p>
+ <p>The rest of the driver is more or less UDS specific and not of
+ general interest.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Putting it all together</title>
+ <p>To test the distribution, one can use the
+ <c><![CDATA[net_kernel:start/1]]></c> function, which is useful as it starts
+ the distribution on a running system, where tracing/debugging
+ can be performed. The <c><![CDATA[net_kernel:start/1]]></c> routine takes a
+ list as it's single argument. The lists first element should be
+ the node name (without the "@hostname") as an atom, and the second (and
+ last) element should be one of the atoms <c><![CDATA[shortnames]]></c> or
+ <c><![CDATA[longnames]]></c>. In the example case <c><![CDATA[shortnames]]></c> is
+ preferred. </p>
+ <p>For net kernel to find out which distribution module to use, the
+ command line argument <c><![CDATA[-proto_dist]]></c> is used. The argument
+ is followed by one or more distribution module names, with the
+ "_dist" suffix removed, i.e. uds_dist as a distribution module
+ is specified as <c><![CDATA[-proto_dist uds]]></c>.</p>
+ <p>If no epmd (TCP port mapper daemon) is used, one should also
+ specify the command line option <c><![CDATA[-no_epmd]]></c>, which will make
+ Erlang skip the epmd startup, both as a OS process and as an
+ Erlang ditto.</p>
+ <p>The path to the directory where the distribution modules reside
+ must be known at boot, which can either be achieved by
+ specifying <c><![CDATA[-pa <path>]]></c> on the command line or by building
+ a boot script containing the applications used for your
+ distribution protocol (in the uds_dist protocol, it's only the
+ uds_dist application that needs to be added to the script).</p>
+ <p>The distribution will be started at boot if all the above is
+ specified and an <c><![CDATA[-sname <name>]]></c> flag is present at the
+ command line, here follows two examples: </p>
+ <pre>
+$ <input>erl -pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin -proto_dist uds -no_epmd</input>
+Erlang (BEAM) emulator version 5.0
+
+Eshell V5.0 (abort with ^G)
+1> <input>net_kernel:start([bing,shortnames]).</input>
+{ok,&lt;0.30.0>}
+(bing@hador)2></pre>
+ <p>...</p>
+ <pre>
+$ <input>erl -pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin -proto_dist uds \\ </input>
+<input> -no_epmd -sname bong</input>
+Erlang (BEAM) emulator version 5.0
+
+Eshell V5.0 (abort with ^G)
+(bong@hador)1></pre>
+ <p>One can utilize the ERL_FLAGS environment variable to store the
+ complicated parameters in:</p>
+ <pre>
+$ <input>ERL_FLAGS=-pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin \\ </input>
+<input> -proto_dist uds -no_epmd</input>
+$ <input>export ERL_FLAGS</input>
+$ <input>erl -sname bang</input>
+Erlang (BEAM) emulator version 5.0
+
+Eshell V5.0 (abort with ^G)
+(bang@hador)1></pre>
+ <p>The <c><![CDATA[ERL_FLAGS]]></c> should preferably not include the name of
+ the node.</p>
+ </section>
+</chapter>
+