The R13B03 release.OTP_R13B03

1 files changed, 1099 insertions, 0 deletions

diff --git a/erts/doc/src/alt_dist.xml b/erts/doc/src/alt_dist.xml
new file mode 100644
index 0000000000..9a68b3cf40
--- /dev/null
+++ b/erts/doc/src/alt_dist.xml
@@ -0,0 +1,1099 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+  <header>
+    <copyright>
+      <year>2000</year><year>2009</year>
+      <holder>Ericsson AB. All Rights Reserved.</holder>
+    </copyright>
+    <legalnotice>
+      The contents of this file are subject to the Erlang Public License,
+      Version 1.1, (the "License"); you may not use this file except in
+      compliance with the License. You should have received a copy of the
+      Erlang Public License along with this software. If not, it can be
+      retrieved online at http://www.erlang.org/.
+    
+      Software distributed under the License is distributed on an "AS IS"
+      basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+      the License for the specific language governing rights and limitations
+      under the License.
+    
+    </legalnotice>
+
+    <title>How to implement an alternative carrier for  the Erlang distribution</title>
+    <prepared>Patrik Nyblom</prepared>
+    <responsible></responsible>
+    <docno></docno>
+    <approved></approved>
+    <checked></checked>
+    <date>2000-10-17</date>
+    <rev>PA2</rev>
+    <file>alt_dist.xml</file>
+  </header>
+  <p>This document describes how one can implement ones own carrier
+    protocol for the Erlang distribution. The distribution is normally
+    carried by the TCP/IP protocol. What's explained here is the method for 
+    replacing TCP/IP with another protocol. </p>
+  <p>The document is a step by step explanation of the <c><![CDATA[uds_dist]]></c> example 
+    application (seated in the kernel applications <c><![CDATA[examples]]></c> directory). 
+    The <c><![CDATA[uds_dist]]></c> application implements distribution over Unix domain 
+    sockets and is written for the Sun Solaris 2 operating environment. The 
+    mechanisms are however general and applies to any operating system Erlang 
+    runs on. The reason the C code is not made portable, is simply readability.</p>
+  <note><p>This document was written a long time ago. Most of it is still
+           valid, but some things have changed since it was first written.
+	   Most notably the driver interface. There have been some updates
+	   to the documentation of the driver presented in this documentation,
+	   but more could be done and are planned for the future. The
+	   reader is encouraged to also read the
+	   <seealso marker="erl_driver">erl_driver</seealso>, and the
+	   <seealso marker="erl_driver">driver_entry</seealso> documentation.
+   </p></note>
+
+  <section>
+    <title>Introduction</title>
+    <p>To implement a new carrier for the Erlang distribution, one must first
+      make the protocol available to the Erlang machine, which involves writing 
+      an Erlang driver. There is no way one can use a port program,
+      there <em>has</em> to
+      be an Erlang driver. Erlang drivers can either be statically
+      linked
+      to the emulator, which can be an alternative when using the open source
+      distribution of Erlang, or dynamically loaded into the Erlang machines
+      address space, which is the only alternative if a precompiled version of 
+      Erlang is to be used. </p>
+    <p>Writing an Erlang driver is by no means easy. The driver is written 
+      as a couple of call-back functions called by the Erlang emulator when
+      data is sent to the driver or the driver has any data available on a file
+      descriptor. As the driver call-back routines execute in the main
+      thread of the Erlang machine, the call-back functions can perform
+      no blocking activity whatsoever. The call-backs should only set up
+      file descriptors for waiting and/or read/write available data. All
+      I/O has to be non blocking. Driver call-backs are however executed
+      in sequence, why a global state can safely be updated within the
+      routines. </p>
+    <p>When the driver is implemented, one would preferably write an
+      Erlang interface for the driver to be able to test the
+      functionality of the driver separately. This interface can then
+      be used by the distribution module which will cover the details of
+      the protocol from the <c><![CDATA[net_kernel]]></c>. The easiest path is to
+      mimic the <c><![CDATA[inet]]></c> and <c><![CDATA[inet_tcp]]></c> interfaces, but a lot of
+      functionality in those modules need not be implemented. In the
+      example application, only a few of the usual interfaces are
+      implemented, and they are much simplified.</p>
+    <p>When the protocol is available to Erlang through a driver and an
+      Erlang interface module, a distribution module can be
+      written. The distribution module is a module with well defined
+      call-backs, much like a <c><![CDATA[gen_server]]></c> (there is no compiler support
+      for checking the call-backs though). The details of finding other
+      nodes (i.e. talking to epmd or something similar), creating a
+      listen port (or similar), connecting to other nodes and performing
+      the handshakes/cookie verification are all implemented by this
+      module. There is however a utility module, <c><![CDATA[dist_util]]></c>, that
+      will do most of the hard work of handling handshakes, cookies,
+      timers and ticking. Using <c><![CDATA[dist_util]]></c> makes implementing a
+      distribution module much easier and that's what we are doing in
+      the example application.</p>
+    <p>The last step is to create boot scripts to make the protocol
+      implementation available at boot time. The implementation can be
+      debugged by starting the distribution when all of the system is
+      running, but in a real system the distribution should start very
+      early, why a boot-script and some command line parameters are
+      necessary. This last step also implies that the Erlang code in the 
+      interface and distribution modules is written in such a way that
+      it can be run in the startup phase. Most notably there can be no
+      calls to the <c><![CDATA[application]]></c> module or to any modules not
+      loaded at boot-time (i.e. only <c><![CDATA[kernel]]></c>, <c><![CDATA[stdlib]]></c> and the
+      application itself can be used).</p>
+  </section>
+
+  <section>
+    <title>The driver</title>
+    <p>Although Erlang drivers in general may be beyond the scope of this
+      document, a brief introduction seems to be in place.</p>
+
+    <section>
+      <title>Drivers in general</title>
+      <p>An Erlang driver is a native code module written in C (or
+        assembler) which serves as an interface for some special operating
+        system service. This is a general mechanism that is used
+        throughout the Erlang emulator for all kinds of I/O. An Erlang
+        driver can be dynamically linked (or loaded) to the Erlang
+        emulator at runtime by using the <c><![CDATA[erl_ddll]]></c> Erlang
+        module. Some of the drivers in OTP are however statically linked
+        to the runtime system, but that's more an optimization than a
+        necessity.</p>
+      <p>The driver data-types and the functions available to the driver
+        writer are defined in the header file <c><![CDATA[erl_driver.h]]></c> (there
+        is also an deprecated version called <c><![CDATA[driver.h]]></c>, don't use
+        that one.) seated in Erlang's include directory (and in
+        $ERL_TOP/erts/emulator/beam in the source code
+        distribution). Refer to that file for function prototypes etc.</p>
+      <p>When writing a driver to make a communications protocol available
+        to Erlang, one should know just about everything worth knowing
+        about that particular protocol. All operation has to be non
+        blocking and all possible situations should be accounted for in
+        the driver. A non stable driver will affect and/or crash the
+        whole Erlang runtime system, which is seldom what's wanted. </p>
+      <p>The emulator calls the driver in the following situations:</p>
+      <list type="bulleted">
+        <item>When the driver is loaded. This call-back has to have a
+         special name and will inform the emulator of what call-backs should
+         be used by returning a pointer to a <c><![CDATA[ErlDrvEntry]]></c> struct,
+         which should be properly filled in (see below).</item>
+        <item>When a port to the driver is opened (by a <c><![CDATA[open_port]]></c>
+         call from Erlang). This routine should set up internal data
+         structures and return an opaque data entity of the type
+        <c><![CDATA[ErlDrvData]]></c>, which is a data-type large enough to hold a
+         pointer. The pointer returned by this function will be the first
+         argument to all other call-backs concerning this particular
+         port. It is usually called the port handle. The emulator only
+         stores the handle and does never try to interpret it, why it can
+         be virtually anything (well anything not larger than a pointer
+         that is) and can point to anything if it is a pointer. Usually
+         this pointer will refer to a structure holding information about
+         the particular port, as i t does in our example.</item>
+        <item>When an Erlang process sends data to the port. The data will
+         arrive as a buffer of bytes, the interpretation is not defined,
+         but is up to the implementor. This call-back returns nothing to the
+         caller, answers are sent to the caller as messages (using a
+         routine called <c><![CDATA[driver_output]]></c> available to all
+         drivers). There is also a way to talk in a synchronous way to
+         drivers, described below. There can be an additional call-back
+         function for handling data that is fragmented (sent in a deep
+         io-list). That interface will get the data in a form suitable for
+         Unix <c><![CDATA[writev]]></c> rather than in a single buffer. There is no
+         need for a distribution driver to implement such a call-back, so
+         we wont.</item>
+        <item>When a file descriptor is signaled for input. This call-back
+         is called when the emulator detects input on a file descriptor
+         which the driver has marked for monitoring by using the interface
+        <c><![CDATA[driver_select]]></c>. The mechanism of driver select makes it
+         possible to read non blocking from file descriptors by calling
+        <c><![CDATA[driver_select]]></c> when reading is needed and then do the actual
+         reading in this call-back (when reading is actually possible). The
+         typical scenario is that <c><![CDATA[driver_select]]></c> is called when an 
+         Erlang process orders a read operation, and that this routine
+         sends the answer when data is available on the file descriptor.</item>
+        <item>When a file descriptor is signaled for output. This call-back
+         is called in a similar way as the previous, but when writing to a
+         file descriptor is possible. The usual scenario is that Erlang
+         orders writing on a file descriptor and that the driver calls
+        <c><![CDATA[driver_select]]></c>. When the descriptor is ready for output,
+         this call-back is called an the driver can try to send the
+         output. There may of course be queuing involved in such
+         operations, and there are some convenient queue routines available
+         to the driver writer to use in such situations.</item>
+        <item>When a port is closed, either by an Erlang process or by the
+         driver calling one of the <c><![CDATA[driver_failure_XXX]]></c> routines. This
+         routine should clean up everything connected to one particular
+         port. Note that when other call-backs call a
+        <c><![CDATA[driver_failure_XXX]]></c> routine, this routine will be
+         immediately called and the call-back routine issuing the error can
+         make no more use of the data structures for the port, as this
+         routine surely has freed all associated data and closed all file
+         descriptors. If the queue utility available to driver writes is
+         used, this routine will however <em>not</em> be called until the
+         queue is empty.</item>
+        <item>When an Erlang process calls <c>erlang:port_control/3</c>,
+         which is a synchronous interface to drivers. The control interface
+         is used to set driver options, change states of ports etc. We'll
+         use this interface quite a lot in our example.</item>
+        <item>When a timer expires. The driver can set timers with the
+         function <c><![CDATA[driver_set_timer]]></c>. When such timers expire, a
+         specific call-back function is called. We will not use timers in
+         our example.</item>
+        <item>When the whole driver is unloaded. Every resource allocated
+         by the driver should be freed.</item>
+      </list>
+    </section>
+
+    <section>
+      <title>The distribution driver's data structures</title>
+      <p>The driver used for Erlang distribution should implement a
+        reliable, order maintaining, variable length packet oriented
+        protocol. All error correction, re-sending and such need to be
+        implemented in the driver or by the underlying communications
+        protocol. If the protocol is stream oriented (as is the case with
+        both TCP/IP and our streamed Unix domain sockets), some mechanism
+        for packaging is needed. We will use the simple method of having a
+        header of four bytes containing the length of the package in a big
+        endian 32 bit integer (as Unix domain sockets only can be used
+        between processes on the same machine, we actually don't need to
+        code the integer in some special endianess, but I'll do it anyway
+        because in most situation you do need to do it. Unix domain
+        sockets are reliable and order maintaining, so we don't need to
+        implement resends and such in our driver.</p>
+      <p>Lets start writing our example Unix domain sockets driver by
+        declaring prototypes and filling in a static ErlDrvEntry
+        structure.</p>
+      <code type="none"><![CDATA[
+( 1) #include <stdio.h>
+( 2) #include <stdlib.h>
+( 3) #include <string.h>
+( 4) #include <unistd.h>
+( 5) #include <errno.h>
+( 6) #include <sys/types.h>
+( 7) #include <sys/stat.h>
+( 8) #include <sys/socket.h>
+( 9) #include <sys/un.h>
+(10) #include <fcntl.h>
+
+(11) #define HAVE_UIO_H
+(12) #include "erl_driver.h"
+
+(13) /*
+(14) ** Interface routines
+(15) */
+(16) static ErlDrvData uds_start(ErlDrvPort port, char *buff);
+(17) static void uds_stop(ErlDrvData handle);
+(18) static void uds_command(ErlDrvData handle, char *buff, int bufflen);
+(19) static void uds_input(ErlDrvData handle, ErlDrvEvent event);
+(20) static void uds_output(ErlDrvData handle, ErlDrvEvent event);
+(21) static void uds_finish(void);
+(22) static int uds_control(ErlDrvData handle, unsigned int command, 
+(23)                        char* buf, int count, char** res, int res_size);
+
+(24) /* The driver entry */
+(25) static ErlDrvEntry uds_driver_entry = {
+(26)     NULL,                            /* init, N/A */
+(27)     uds_start,                       /* start, called when port is opened */
+(28)     uds_stop,                        /* stop, called when port is closed */
+(29)     uds_command,                     /* output, called when erlang has sent */
+(30)     uds_input,                       /* ready_input, called when input
+(31)                                         descriptor ready */
+(32)     uds_output,                      /* ready_output, called when output 
+(33)                                         descriptor ready */
+(34)     "uds_drv",                       /* char *driver_name, the argument 
+(35)                                         to open_port */
+(36)     uds_finish,                      /* finish, called when unloaded */
+(37)     NULL,                            /* void * that is not used (BC) */
+(38)     uds_control,                     /* control, port_control callback */
+(39)     NULL,                            /* timeout, called on timeouts */
+(40)     NULL,                            /* outputv, vector output interface */
+(41)     NULL,                            /* ready_async callback */
+(42)     NULL,                            /* flush callback */
+(43)     NULL,                            /* call callback */
+(44)     NULL,                            /* event callback */
+(45)     ERL_DRV_EXTENDED_MARKER,         /* Extended driver interface marker */
+(46)     ERL_DRV_EXTENDED_MAJOR_VERSION,  /* Major version number */
+(47)     ERL_DRV_EXTENDED_MINOR_VERSION,  /* Minor version number */
+(48)     ERL_DRV_FLAG_SOFT_BUSY,          /* Driver flags. Soft busy flag is
+(49)                                         required for distribution drivers */
+(50)     NULL,                            /* Reserved for internal use */
+(51)     NULL,                            /* process_exit callback */
+(52)     NULL                             /* stop_select callback */
+(53) };]]></code>
+      <p>On line 1 to 10 we have included the OS headers needed for our
+        driver. As this driver is written for Solaris, we know that the
+        header <c><![CDATA[uio.h]]></c> exists, why we can define the preprocessor
+        variable <c><![CDATA[HAVE_UIO_H]]></c> before we include <c><![CDATA[erl_driver.h]]></c> 
+        at line 12. The definition of <c><![CDATA[HAVE_UIO_H]]></c> will make the
+        I/O vectors used in Erlang's driver queues to correspond to the
+        operating systems ditto, which is very convenient.</p>
+      <p>The different call-back functions are declared ("forward
+        declarations") on line 16 to 23.</p>
+      <p>The driver structure is similar for statically linked in
+        drivers and dynamically loaded. However some of the fields
+        should be left empty (i.e. initialized to NULL) in the
+        different types of drivers. The first field (the <c><![CDATA[init]]></c>
+        function pointer) is always left blank in a dynamically loaded
+        driver, which can be seen on line 26. The NULL on line 37
+        should always be there, the field is no longer used and is
+        retained for backward compatibility. We use no timers in this
+        driver, why no call-back for timers is needed. The <c>outputv</c> field
+        (line 40) can be used to implement an interface similar to
+        Unix <c><![CDATA[writev]]></c> for output. The Erlang runtime
+	system could previously not use <c>outputv</c> for the
+	distribution, but since erts version 5.7.2 it can.
+	Since this driver was written before erts version 5.7.2 it does 
+	not use the <c>outputv</c> callback. Using the <c>outputv</c>
+	callback is preferred since it reduces copying of data. (We
+	will however use scatter/gather I/O internally in the driver).</p>
+      <p>As of erts version 5.5.3 the driver interface was extended with
+         version control and the possibility to pass capability information.
+	 Capability flags are present at line 48. As of erts version 5.7.4
+	 the
+	 <seealso marker="driver_entry#driver_flags">ERL_DRV_FLAG_SOFT_BUSY</seealso>
+	 flag is required for drivers that are to be used by the distribution.
+	 The soft busy flag implies that the driver is capable of handling
+	 calls to the <c>output</c> and <c>outputv</c> callbacks even though
+	 it has marked itself as busy. This has always been a requirement
+	 on drivers used by the distribution, but there have previously not
+	 been any capability information available about this. For more
+	 information see
+         <seealso marker="erl_driver#set_busy_port">set_busy_port()</seealso>).
+</p>
+      <p>This driver was written before the runtime system had SMP support.
+         The driver will still function in the runtime system with SMP support,
+	 but performance will suffer from lock contention on the driver lock
+	 used for the driver. This can be alleviated by reviewing and perhaps
+	 rewriting the code so that each instance of the driver safely can
+	 execute in parallel. When instances safely can execute in parallel it
+	 is safe to enable instance specific locking on the driver. This is done
+	 by passing
+	 <seealso marker="driver_entry#driver_flags">ERL_DRV_FLAG_USE_PORT_LOCKING</seealso>
+	 as a driver flag. This is left as an exercise for the reader.</p>
+      <p>Our defined call-backs thus are:</p>
+      <list type="bulleted">
+        <item>uds_start, which shall initiate data for a port. We wont
+         create any actual sockets here, just initialize data structures.</item>
+        <item>uds_stop, the function called when a port is closed.</item>
+        <item>uds_command, which will handle messages from Erlang. The
+         messages can either be plain data to be sent or more subtle
+         instructions to the driver. We will use this function mostly for
+         data pumping.</item>
+        <item>uds_input, this is the call-back which is called when we have
+         something to read from a socket.</item>
+        <item>uds_output, this is the function called when we can write to a
+         socket.</item>
+        <item>uds_finish, which is called when the driver is unloaded. A
+         distribution driver will actually (or hopefully) never be unloaded,
+         but we include this for completeness. Being able to clean up after
+         oneself is always a good thing.</item>
+        <item>uds_control, the <c>erlang:port_control/2</c> call-back, which
+         will be used a lot in this implementation.</item>
+      </list>
+      <p>The ports implemented by this driver will operate in two major
+        modes, which i will call the <em>command</em> and <em>data</em>
+        modes. In command mode, only passive reading and writing (like
+        gen_tcp:recv/gen_tcp:send) can be
+        done, and this is the mode the port will be in during the
+        distribution handshake. When the connection is up, the port will
+        be switched to data mode and all data will be immediately read and
+        passed further to the Erlang emulator. In data mode, no data
+        arriving to the uds_command will be interpreted, but just packaged
+        and sent out on the socket. The uds_control call-back will do the
+        switching between those two modes.</p>
+      <p>While the <c><![CDATA[net_kernel]]></c> informs different subsystems that the
+        connection is coming up, the port should accept data to send, but
+        not receive any data, to avoid that data arrives from another node
+        before every kernel subsystem is prepared to handle it. We have a
+        third mode for this intermediate stage, lets call it the
+        <em>intermediate</em> mode.</p>
+      <p>Lets define an enum for the different types of ports we have:</p>
+      <code type="none"><![CDATA[
+( 1) typedef enum { 
+( 2)     portTypeUnknown,      /* An uninitialized port */
+( 3)     portTypeListener,     /* A listening port/socket */
+( 4)     portTypeAcceptor,     /* An intermidiate stage when accepting 
+( 5)                              on a listen port */
+( 6)     portTypeConnector,    /* An intermediate stage when connecting */
+( 7)     portTypeCommand,      /* A connected open port in command mode */
+( 8)     portTypeIntermediate, /* A connected open port in special
+( 9)                              half active mode */
+(10)     portTypeData          /* A connectec open port in data mode */ 
+(11) } PortType;      ]]></code>
+      <p>Lets look at the different types:</p>
+      <list type="bulleted">
+        <item>portTypeUnknown - The type a port has when it's opened, but
+         not actually bound to any file descriptor.</item>
+        <item>portTypeListener - A port that is connected to a listen
+         socket. This port will not do especially much, there will be no data
+         pumping done on this socket, but there will be read data available
+         when one is trying to do an accept on the port.</item>
+        <item>portTypeAcceptor - This is a port that is to represent the
+         result of an accept operation. It is created when one wants to
+         accept from a listen socket, and it will be converted to a
+         portTypeCommand when the accept succeeds.</item>
+        <item>portTypeConnector - Very similar to portTypeAcceptor, an
+         intermediate stage between the request for a connect operation and
+         that the socket is really connected to an accepting ditto in the
+         other end. As soon as the sockets are connected, the port will
+         switch type to portTypeCommand.</item>
+        <item>portTypeCommand - A connected socket (or accepted socket if
+         you want) that is in the command mode mentioned earlier.</item>
+        <item>portTypeIntermediate - The intermediate stage for a connected
+         socket. There should be no processing of input for this socket.</item>
+        <item>portTypeData - The mode where data is pumped through the port
+         and the uds_command routine will regard every call as a call where
+         sending is wanted. In this mode all input available will be read and
+         sent to Erlang as soon as it arrives on the socket, much like in the
+         active mode of a <c><![CDATA[gen_tcp]]></c> socket.</item>
+      </list>
+      <p>Now lets look at the state we'll need for our ports. One can note
+        that not all fields are used for all types of ports and that one
+        could save some space by using unions, but that would clutter the
+        code with multiple indirections, so i simply use one struct for
+        all types of ports, for readability.</p>
+      <code type="none"><![CDATA[
+( 1) typedef unsigned char Byte;
+( 2) typedef unsigned int Word;
+
+( 3) typedef struct uds_data {
+( 4)     int fd;                   /* File descriptor */
+( 5)     ErlDrvPort port;          /* The port identifier */
+( 6)     int lockfd;               /* The file descriptor for a lock file in 
+( 7)                                  case of listen sockets */
+( 8)     Byte creation;            /* The creation serial derived from the 
+( 9)                                  lockfile */
+(10)     PortType type;            /* Type of port */
+(11)     char *name;               /* Short name of socket for unlink */
+(12)     Word sent;                /* Bytes sent */
+(13)     Word received;            /* Bytes received */
+(14)     struct uds_data *partner; /* The partner in an accept/listen pair */
+(15)     struct uds_data *next;    /* Next structure in list */
+(16)     /* The input buffer and it's data */
+(17)     int buffer_size;          /* The allocated size of the input buffer */
+(18)     int buffer_pos;           /* Current position in input buffer */
+(19)     int header_pos;           /* Where the current header is in the 
+(20)                                  input buffer */
+(21)     Byte *buffer;             /* The actual input buffer */
+(22) } UdsData;      ]]></code>
+      <p>This structure is used for all types of ports although some
+        fields are useless for some types. The least memory consuming
+        solution would be to arrange this structure as a union of
+        structures, but the multiple indirections in the code to
+        access a field in such a structure will clutter the code to
+        much for an example.</p>
+      <p>Let's look at the fields in our structure:</p>
+      <list type="bulleted">
+        <item>fd - The file descriptor of the socket associated with the
+         port.</item>
+        <item>port - The port identifier for the port which this structure
+         corresponds to. It is needed for most <c><![CDATA[driver_XXX]]></c>
+         calls from the driver back to the emulator.</item>
+        <item>
+          <p>lockfd - If the socket is a listen socket, we use a separate
+            (regular) file for two purposes:</p>
+          <list type="bulleted">
+            <item>We want a locking mechanism that gives no race
+             conditions, so that we can be sure of if another Erlang
+             node uses the listen socket name we require or if the
+             file is only left there from a previous (crashed)
+             session.</item>
+            <item>
+              <p>We store the <em>creation</em> serial number in the
+                file. The <em>creation</em> is a number that should
+                change between different instances of different Erlang
+                emulators with the same name, so that process
+                identifiers from one emulator won't be valid when sent
+                to a new emulator with the same distribution name. The
+                creation can be between 0 and 3 (two bits) and is stored
+                in every process identifier sent to another node. </p>
+              <p>In a system with TCP based distribution, this data is
+                kept in the <em>Erlang port mapper daemon</em>
+                (<c><![CDATA[epmd]]></c>), which is contacted when a distributed
+                node starts. The lock-file and a convention for the UDS
+                listen socket's name will remove the need for
+                <c><![CDATA[epmd]]></c> when using this distribution module. UDS
+                is always restricted to one host, why avoiding a port
+                mapper is easy.</p>
+            </item>
+          </list>
+        </item>
+        <item>creation - The creation number for a listen socket, which is
+         calculated as (the value found in the lock-file + 1) rem
+         4. This creation value is also written back into the
+         lock-file, so that the next invocation of the emulator will
+         found our value in the file.</item>
+        <item>type - The current type/state of the port, which can be one
+         of the values declared above.</item>
+        <item>name - The name of the socket file (the path prefix
+         removed), which allows for deletion (<c><![CDATA[unlink]]></c>) when the
+         socket is closed.</item>
+        <item>sent - How many bytes that have been sent over the
+         socket. This may wrap, but that's no problem for the
+         distribution, as the only thing that interests the Erlang
+         distribution is if this value has changed (the Erlang
+         net_kernel <em>ticker</em> uses this value by calling the
+         driver to fetch it, which is done through the
+	 <c>erlang:port_control</c> routine).</item>
+        <item>received - How many bytes that are read (received) from the
+         socket, used in similar ways as <c><![CDATA[sent]]></c>.</item>
+        <item>partner - A pointer to another port structure, which is
+         either the listen port from which this port is accepting a
+         connection or the other way around. The "partner relation"
+         is always bidirectional.</item>
+        <item>next - Pointer to next structure in a linked list of all
+         port structures. This list is used when accepting
+         connections and when the driver is unloaded.</item>
+        <item>buffer_size, buffer_pos, header_pos, buffer - data for input
+         buffering. Refer to the source code (in the kernel/examples
+         directory) for details about the input buffering. That
+         certainly goes beyond the scope of this document.</item>
+      </list>
+    </section>
+
+    <section>
+      <title>Selected parts of the distribution driver implementation</title>
+      <p>The distribution drivers implementation is not completely
+        covered in this text, details about buffering and other things
+        unrelated to driver writing are not explained. Likewise are
+        some peculiarities of the UDS protocol not explained in
+        detail. The chosen protocol is not important.</p>
+      <p>Prototypes for the driver call-back routines can be found in
+        the <c><![CDATA[erl_driver.h]]></c> header file.</p>
+      <p>The driver initialization routine is (usually) declared with a
+        macro to make the driver easier to port between different
+        operating systems (and flavours of systems). This is the only
+        routine that has to have a well defined name. All other
+        call-backs are reached through the driver structure. The macro
+        to use is named <c><![CDATA[DRIVER_INIT]]></c> and takes the driver name
+        as parameter.</p>
+      <code type="none"><![CDATA[
+(1) /* Beginning of linked list of ports */
+(2) static UdsData *first_data;
+
+
+(3) DRIVER_INIT(uds_drv)
+(4) {
+(5)     first_data = NULL;
+(6)     return &uds_driver_entry;
+(7) }      ]]></code>
+      <p>The routine initializes the single global data structure and
+        returns a pointer to the driver entry. The routine will be
+        called when <c><![CDATA[erl_ddll:load_driver]]></c> is called from Erlang.</p>
+      <p>The <c><![CDATA[uds_start]]></c> routine is called when a port is opened
+        from Erlang. In our case, we only allocate a structure and
+        initialize it. Creating the actual socket is left to the
+        <c><![CDATA[uds_command]]></c> routine.</p>
+      <code type="none"><![CDATA[
+( 1) static ErlDrvData uds_start(ErlDrvPort port, char *buff)
+( 2) {
+( 3)     UdsData *ud;
+( 4)     
+( 5)     ud = ALLOC(sizeof(UdsData));
+( 6)     ud->fd = -1;
+( 7)     ud->lockfd = -1;
+( 8)     ud->creation = 0;
+( 9)     ud->port = port;
+(10)     ud->type = portTypeUnknown;
+(11)     ud->name = NULL;
+(12)     ud->buffer_size = 0;
+(13)     ud->buffer_pos = 0;
+(14)     ud->header_pos = 0;
+(15)     ud->buffer = NULL;
+(16)     ud->sent = 0;
+(17)     ud->received = 0;
+(18)     ud->partner = NULL;
+(19)     ud->next = first_data;
+(20)     first_data = ud;
+(21)     
+(22)     return((ErlDrvData) ud);
+(23) }      ]]></code>
+      <p>Every data item is initialized, so that no problems will arise
+        when a newly created port is closed (without there being any
+        corresponding socket). This routine is called when
+        <c><![CDATA[open_port({spawn, "uds_drv"},[])]]></c> is called from Erlang.</p>
+      <p>The <c><![CDATA[uds_command]]></c> routine is the routine called when an
+        Erlang process sends data to the port. All asynchronous
+        commands when the port is in <em>command mode</em> as well as 
+        the sending of all data when the port is in <em>data mode</em>
+        is handled in this9s routine. Let's have a look at it:</p>
+      <code type="none"><![CDATA[
+( 1) static void uds_command(ErlDrvData handle, char *buff, int bufflen)
+( 2) {
+( 3)     UdsData *ud = (UdsData *) handle;
+
+( 4)     if (ud->type == portTypeData || ud->type == portTypeIntermediate) {
+( 5)         DEBUGF(("Passive do_send %d",bufflen));
+( 6)         do_send(ud, buff + 1, bufflen - 1); /* XXX */
+( 7)         return;
+( 8)     } 
+( 9)     if (bufflen == 0) {
+(10)         return;
+(11)     }
+(12)     switch (*buff) {
+(13)     case 'L':
+(14)         if (ud->type != portTypeUnknown) {
+(15)             driver_failure_posix(ud->port, ENOTSUP);
+(16)             return;
+(17)         }
+(18)         uds_command_listen(ud,buff,bufflen);
+(19)         return;
+(20)     case 'A':
+(21)         if (ud->type != portTypeUnknown) {
+(22)             driver_failure_posix(ud->port, ENOTSUP);
+(23)             return;
+(24)         }
+(25)         uds_command_accept(ud,buff,bufflen);
+(26)         return;
+(27)     case 'C':
+(28)         if (ud->type != portTypeUnknown) {
+(29)             driver_failure_posix(ud->port, ENOTSUP);
+(30)             return;
+(31)         }
+(32)         uds_command_connect(ud,buff,bufflen);
+(33)         return;
+(34)     case 'S':
+(35)         if (ud->type != portTypeCommand) {
+(36)             driver_failure_posix(ud->port, ENOTSUP);
+(37)             return;
+(38)         }
+(39)         do_send(ud, buff + 1, bufflen - 1);
+(40)         return;
+(41)     case 'R':
+(42)         if (ud->type != portTypeCommand) {
+(43)             driver_failure_posix(ud->port, ENOTSUP);
+(44)             return;
+(45)         }
+(46)         do_recv(ud);
+(47)         return;
+(48)     default:
+(49)         return;
+(50)     }
+(51) }      ]]></code>
+      <p>The command routine takes three parameters; the handle
+        returned for the port by <c><![CDATA[uds_start]]></c>, which is a pointer
+        to the internal port structure, the data buffer and the length
+        of the data buffer. The buffer is the data sent from Erlang
+        (a list of bytes) converted to an C array (of bytes). </p>
+      <p>If Erlang sends i.e. the list <c><![CDATA[[$a,$b,$c]]]></c> to the port,
+        the <c><![CDATA[bufflen]]></c> variable will be <c><![CDATA[3]]></c> ant the
+        <c><![CDATA[buff]]></c> variable will contain <c><![CDATA[{'a','b','c'}]]></c> (no
+        null termination). Usually the first byte is used as an
+        opcode, which is the case in our driver to (at least when the
+        port is in command mode). The opcodes are defined as:</p>
+      <list type="bulleted">
+        <item>'L'&lt;socketname&gt;: Create and listen on socket with the
+         given name.</item>
+        <item>'A'&lt;listennumber as 32 bit bigendian&gt;: Accept from the
+         listen socket identified by the given identification
+         number. The identification number is retrieved with the
+         uds_control routine.</item>
+        <item>'C'&lt;socketname&gt;: Connect to the socket named
+         &lt;socketname&gt;.</item>
+        <item>'S'&lt;data&gt;: Send the data &lt;data&gt; on the
+         connected/accepted socket (in command mode). The sending is
+         acked when the data has left this process.</item>
+        <item>'R': Receive one packet of data.</item>
+      </list>
+      <p>One may wonder what is meant by "one packet of data" in the
+        'R' command. This driver always sends data packeted with a 4
+        byte header containing a big endian 32 bit integer that
+        represents the length of the data in the packet. There is no
+        need for different packet sizes or some kind of streamed
+        mode, as this driver is for the distribution only. One may
+        wonder why the header word is coded explicitly in big endian
+        when an UDS socket is local to the host. The answer simply is
+        that I see it as a good practice when writing a distribution
+        driver, as distribution in practice usually cross the host
+        boundaries. </p>
+      <p>On line 4-8 we handle the case where the port is in data or
+        intermediate mode, the rest of the routine handles the
+        different commands. We see (first on line 15) that the routine
+        uses the <c><![CDATA[driver_failure_posix()]]></c> routine to report
+        errors. One important thing to remember is that the failure
+        routines make a call to our <c><![CDATA[uds_stop]]></c> routine, which
+        will remove the internal port data. The handle (and the casted
+        handle <c><![CDATA[ud]]></c>) is therefore <em>invalid pointers</em> after a
+        <c><![CDATA[driver_failure]]></c> call and we should <em>immediately return</em>. The runtime system will send exit signals to all
+        linked processes.</p>
+      <p>The uds_input routine gets called when data is available on a
+        file descriptor previously passed to the <c><![CDATA[driver_select]]></c>
+        routine. Typically this happens when a read command is issued
+        and no data is available. Lets look at the <c><![CDATA[do_recv]]></c>
+        routine:</p>
+      <code type="none"><![CDATA[
+( 1) static void do_recv(UdsData *ud)
+( 2) {
+( 3)     int res;
+( 4)     char *ibuf;
+( 5)     for(;;) {
+( 6)         if ((res = buffered_read_package(ud,&ibuf)) < 0) {
+( 7)             if (res == NORMAL_READ_FAILURE) {
+( 8)                 driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 1);
+( 9)             } else {
+(10)                 driver_failure_eof(ud->port);
+(11)             }
+(12)             return;
+(13)         }
+(14)         /* Got a package */
+(15)         if (ud->type == portTypeCommand) {
+(16)             ibuf[-1] = 'R'; /* There is always room for a single byte 
+(17)                                opcode before the actual buffer 
+(18)                                (where the packet header was) */
+(19)             driver_output(ud->port,ibuf - 1, res + 1);
+(20)             driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ,0);
+(21)             return;
+(22)         } else {
+(23)             ibuf[-1] = DIST_MAGIC_RECV_TAG; /* XXX */
+(24)             driver_output(ud->port,ibuf - 1, res + 1);
+(25)             driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ,1);
+(26)         }
+(27)     }
+(28) }      ]]></code>
+      <p>The routine tries to read data until a packet is read or the
+        <c><![CDATA[buffered_read_package]]></c> routine returns a
+        <c><![CDATA[NORMAL_READ_FAILURE]]></c> (an internally defined constant for
+        the module that means that the read operation resulted in an
+        <c><![CDATA[EWOULDBLOCK]]></c>). If the port is in command mode, the
+        reading stops when one package is read, but if it is in data
+        mode, the reading continues until the socket buffer is empty
+        (read failure). If no more data can be read and more is wanted
+        (always the case when socket is in data mode) driver_select is
+        called to make the <c><![CDATA[uds_input]]></c> call-back be called when
+        more data is available for reading.</p>
+      <p>When the port is in data mode, all data is sent to Erlang in a
+        format that suits the distribution, in fact the raw data will
+        never reach any Erlang process, but will be
+        translated/interpreted by the emulator itself and then
+        delivered in the correct format to the correct processes. In
+        the current emulator version, received data should be tagged
+        with a single byte of 100. Thats what the macro
+        <c><![CDATA[DIST_MAGIC_RECV_TAG]]></c> is defined to. The tagging of data
+        in the distribution will possibly change in the future.</p>
+      <p>The <c><![CDATA[uds_input]]></c> routine will handle other input events
+        (like nonblocking <c><![CDATA[accept]]></c>), but most importantly handle
+        data arriving at the socket by calling <c><![CDATA[do_recv]]></c>:</p>
+      <code type="none"><![CDATA[
+( 1) static void uds_input(ErlDrvData handle, ErlDrvEvent event)
+( 2) {
+( 3)     UdsData *ud = (UdsData *) handle;
+
+( 4)     if (ud->type == portTypeListener) {
+( 5)         UdsData *ad = ud->partner;
+( 6)         struct sockaddr_un peer;
+( 7)         int pl = sizeof(struct sockaddr_un);
+( 8)         int fd;
+
+( 9)         if ((fd = accept(ud->fd, (struct sockaddr *) &peer, &pl)) < 0) {
+(10)             if (errno != EWOULDBLOCK) {
+(11)                 driver_failure_posix(ud->port, errno);
+(12)                 return;
+(13)             }
+(14)             return;
+(15)         }
+(16)         SET_NONBLOCKING(fd);
+(17)         ad->fd = fd;
+(18)         ad->partner = NULL;
+(19)         ad->type = portTypeCommand;
+(20)         ud->partner = NULL;
+(21)         driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(22)         driver_output(ad->port, "Aok",3);
+(23)         return;
+(24)     }
+(25)     do_recv(ud);
+(26) }      ]]></code>
+      <p>The important line here is the last line in the function, the
+        <c><![CDATA[do_read]]></c> routine is called to handle new input. The rest
+        of the function handles input on a listen socket, which means
+        that there should be possible to do an accept on the
+        socket, which is also recognized as a read event.</p>
+      <p>The output mechanisms are similar to the input. Lets first
+        look at the <c><![CDATA[do_send]]></c> routine:</p>
+      <code type="none"><![CDATA[
+( 1) static void do_send(UdsData *ud, char *buff, int bufflen) 
+( 2) {
+( 3)     char header[4];
+( 4)     int written;
+( 5)     SysIOVec iov[2];
+( 6)     ErlIOVec eio;
+( 7)     ErlDrvBinary *binv[] = {NULL,NULL};
+
+( 8)     put_packet_length(header, bufflen);
+( 9)     iov[0].iov_base = (char *) header;
+(10)     iov[0].iov_len = 4;
+(11)     iov[1].iov_base = buff;
+(12)     iov[1].iov_len = bufflen;
+(13)     eio.iov = iov;
+(14)     eio.binv = binv;
+(15)     eio.vsize = 2;
+(16)     eio.size = bufflen + 4;
+(17)     written = 0;
+(18)     if (driver_sizeq(ud->port) == 0) {
+(19)         if ((written = writev(ud->fd, iov, 2)) == eio.size) {
+(20)             ud->sent += written;
+(21)             if (ud->type == portTypeCommand) {
+(22)                 driver_output(ud->port, "Sok", 3);
+(23)             }
+(24)             return;
+(25)         } else if (written < 0) {
+(26)             if (errno != EWOULDBLOCK) {
+(27)                 driver_failure_eof(ud->port);
+(28)                 return;
+(29)             } else {
+(30)                 written = 0;
+(31)             }
+(32)         } else {
+(33)             ud->sent += written;
+(34)         }
+(35)         /* Enqueue remaining */
+(36)     }
+(37)     driver_enqv(ud->port, &eio, written);
+(38)     send_out_queue(ud);
+(39) }      ]]></code>
+      <p>This driver uses the <c><![CDATA[writev]]></c> system call to send data
+        onto the socket. A combination of writev and the driver output
+        queues is very convenient. An <em>ErlIOVec</em> structure
+        contains a <em>SysIOVec</em> (which is equivalent to the
+        <c><![CDATA[struct iovec]]></c> structure defined in <c><![CDATA[uio.h]]></c>. The
+        ErlIOVec also contains an array of <em>ErlDrvBinary</em>
+        pointers, of the same length as the number of buffers in the
+        I/O vector itself. One can use this to allocate the binaries
+        for the queue "manually" in the driver, but we'll just fill
+        the binary array with NULL values (line 7) , which will make
+        the runtime system allocate it's own buffers when we call
+        <c><![CDATA[driver_enqv]]></c> (line 37).</p>
+      <p></p>
+      <p>The routine builds an I/O vector containing the header bytes
+        and the buffer (the opcode has been removed and the buffer
+        length decreased by the output routine). If the queue is
+        empty, we'll write the data directly to the socket (or at
+        least try to). If any data is left, it is stored in the queue
+        and then we try to send the queue (line 38). An ack is sent
+        when the message is delivered completely (line 22). The
+        <c><![CDATA[send_out_queue]]></c> will send acks if the sending is
+        completed there. If the port is in command mode, the Erlang
+        code serializes the send operations so that only one packet
+        can be waiting for delivery at a time. Therefore the ack can
+        be sent simply whenever the queue is empty.</p>
+      <p></p>
+      <p>A short look at the <c><![CDATA[send_out_queue]]></c> routine:</p>
+      <code type="none"><![CDATA[
+( 1) static int send_out_queue(UdsData *ud)
+( 2) {
+( 3)     for(;;) {
+( 4)         int vlen;
+( 5)         SysIOVec *tmp = driver_peekq(ud->port, &vlen);
+( 6)         int wrote;
+( 7)         if (tmp == NULL) {
+( 8)             driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_WRITE, 0);
+( 9)             if (ud->type == portTypeCommand) {
+(10)                 driver_output(ud->port, "Sok", 3);
+(11)             }
+(12)             return 0;
+(13)         }
+(14)         if (vlen > IO_VECTOR_MAX) {
+(15)             vlen = IO_VECTOR_MAX;
+(16)         } 
+(17)         if ((wrote = writev(ud->fd, tmp, vlen)) < 0) {
+(18)             if (errno == EWOULDBLOCK) {
+(19)                 driver_select(ud->port, (ErlDrvEvent) ud->fd, 
+(20)                               DO_WRITE, 1);
+(21)                 return 0;
+(22)             } else {
+(23)                 driver_failure_eof(ud->port);
+(24)                 return -1;
+(25)             }
+(26)         }
+(27)         driver_deq(ud->port, wrote);
+(28)         ud->sent += wrote;
+(29)     }
+(30) }      ]]></code>
+      <p>What we do is simply to pick out an I/O vector from the queue
+        (which is the whole queue as an <em>SysIOVec</em>). If the I/O
+        vector is to long (IO_VECTOR_MAX is defined to 16), the vector
+        length is decreased (line 15), otherwise the <c><![CDATA[writev]]></c>
+        (line 17) call will
+        fail. Writing is tried and anything written is dequeued (line
+        27). If the write fails with <c><![CDATA[EWOULDBLOCK]]></c> (note that all
+        sockets are in nonblocking mode), <c><![CDATA[driver_select]]></c> is
+        called to make the <c><![CDATA[uds_output]]></c> routine be called when
+        there is space to write again.</p>
+      <p>We will continue trying to write until the queue is empty or
+        the writing would block.</p>
+      <p>The routine above are called from the <c><![CDATA[uds_output]]></c>
+        routine, which looks like this:</p>
+      <code type="none"><![CDATA[
+( 1) static void uds_output(ErlDrvData handle, ErlDrvEvent event)
+( 2) {
+( 3)    UdsData *ud = (UdsData *) handle;
+( 4)    if (ud->type == portTypeConnector) {
+( 5)        ud->type = portTypeCommand;
+( 6)        driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_WRITE, 0);
+( 7)        driver_output(ud->port, "Cok",3);
+( 8)        return;
+( 9)    }
+(10)    send_out_queue(ud);
+(11) }      ]]></code>
+      <p>The routine is simple, it first handles the fact that the
+        output select will concern a socket in the business of
+        connecting (and the connecting blocked). If the socket is in
+        a connected state it simply sends the output queue, this
+        routine is called when there is possible to write to a socket
+        where we have an output queue, so there is no question what to
+        do.</p>
+      <p>The driver implements a control interface, which is a
+        synchronous interface called when Erlang calls
+        <c><![CDATA[erlang:port_control/3]]></c>. This is the only interface
+        that can control the driver when it is in data mode and it may
+        be called with the following opcodes:</p>
+      <list type="bulleted">
+        <item>'C': Set port in command mode.</item>
+        <item>'I': Set port in intermediate mode.</item>
+        <item>'D': Set port in data mode.</item>
+        <item>'N': Get identification number for listen port, this
+         identification number is used in an accept command to the
+         driver, it is returned as a big endian 32 bit integer, which
+         happens to be the file identifier for the listen socket.</item>
+        <item>'S': Get statistics, which is the number of bytes received,
+         the number of bytes sent and the number of bytes pending in
+         the output queue. This data is used when the distribution
+         checks that a connection is alive (ticking). The statistics
+         is returned as 3 32 bit big endian integers.</item>
+        <item>'T': Send a tick message, which is a packet of length
+         0. Ticking is done when the port is in data mode, so the
+         command for sending data cannot be used (besides it ignores
+         zero length packages in command mode). This is used by the
+         ticker to send dummy data when no other traffic is present.
+	 <em>Note</em> that it is important that the interface for
+	 sending ticks is not blocking. This implementation uses
+	 <c>erlang:port_control/3</c> which does not block the caller.
+	 If <c>erlang:port_command</c> is used, use
+	 <c>erlang:port_command/3</c> and pass <c>[force]</c> as
+	 option list; otherwise, the caller can be blocked indefinitely
+	 on a busy port and prevent the system from taking down a
+	 connection that is not functioning.</item>
+        <item>'R': Get creation number of listen socket, which is used to
+         dig out the number stored in the lock file to differentiate
+         between invocations of Erlang nodes with the same name.\011  </item>
+      </list>
+      <p>The control interface gets a buffer to return its value in,
+        but is free to allocate it's own buffer is the provided one is
+        to small. Here is the code for <c><![CDATA[uds_control]]></c>:</p>
+      <code type="none"><![CDATA[
+( 1) static int uds_control(ErlDrvData handle, unsigned int command, 
+( 2)                        char* buf, int count, char** res, int res_size)
+( 3) {
+( 4) /* Local macro to ensure large enough buffer. */
+( 5) #define ENSURE(N)                               \\
+( 6)    do {                                         \\
+( 7)        if (res_size < N) {                      \\
+( 8)            *res = ALLOC(N);                     \\
+( 9)        }                                        \\
+(10)    } while(0)
+
+(11)    UdsData *ud = (UdsData *) handle;
+
+(12)    switch (command) {
+(13)    case 'S':
+(14)        {
+(15)            ENSURE(13);
+(16)            **res = 0;
+(17)            put_packet_length((*res) + 1, ud->received);
+(18)            put_packet_length((*res) + 5, ud->sent);
+(19)            put_packet_length((*res) + 9, driver_sizeq(ud->port));
+(20)            return 13;
+(21)        }
+(22)    case 'C':
+(23)        if (ud->type < portTypeCommand) {
+(24)            return report_control_error(res, res_size, "einval");
+(25)        }
+(26)        ud->type = portTypeCommand;
+(27)        driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(28)        ENSURE(1);
+(29)        **res = 0;
+(30)        return 1;
+(31)    case 'I':
+(32)        if (ud->type < portTypeCommand) {
+(33)            return report_control_error(res, res_size, "einval");
+(34)        }
+(35)        ud->type = portTypeIntermediate;
+(36)        driver_select(ud->port, (ErlDrvEvent) ud->fd, DO_READ, 0);
+(37)        ENSURE(1);
+(38)        **res = 0;
+(39)        return 1;
+(40)    case 'D':
+(41)        if (ud->type < portTypeCommand) {
+(42)            return report_control_error(res, res_size, "einval");
+(43)        }
+(44)        ud->type = portTypeData;
+(45)        do_recv(ud);
+(46)        ENSURE(1);
+(47)        **res = 0;
+(48)        return 1;
+(49)    case 'N':
+(50)        if (ud->type != portTypeListener) {
+(51)            return report_control_error(res, res_size, "einval");
+(52)        }
+(53)        ENSURE(5);
+(54)        (*res)[0] = 0;
+(55)        put_packet_length((*res) + 1, ud->fd);
+(56)        return 5;
+(57)    case 'T': /* tick */
+(58)        if (ud->type != portTypeData) {
+(59)            return report_control_error(res, res_size, "einval");
+(60)        }
+(61)        do_send(ud,"",0);
+(62)        ENSURE(1);
+(63)        **res = 0;
+(64)        return 1;
+(65)    case 'R':
+(66)        if (ud->type != portTypeListener) {
+(67)            return report_control_error(res, res_size, "einval");
+(68)        }
+(69)        ENSURE(2);
+(70)        (*res)[0] = 0;
+(71)        (*res)[1] = ud->creation;
+(72)        return 2;
+(73)    default:
+(74)        return report_control_error(res, res_size, "einval");
+(75)    }
+(76) #undef ENSURE
+(77) }      ]]></code>
+      <p>The macro <c><![CDATA[ENSURE]]></c> (line 5 to 10) is used to ensure that
+        the buffer is large enough for our answer. We switch on the
+        command and take actions, there is not much to say about this
+        routine. Worth noting is that we always has read select active
+        on a port in data mode (achieved by calling <c><![CDATA[do_recv]]></c> on
+        line 45), but turn off read selection in intermediate and
+        command modes (line 27 and 36).</p>
+      <p>The rest of the driver is more or less UDS specific and not of
+        general interest.</p>
+    </section>
+  </section>
+
+  <section>
+    <title>Putting it all together</title>
+    <p>To test the distribution, one can use the
+      <c><![CDATA[net_kernel:start/1]]></c> function, which is useful as it starts
+      the distribution on a running system, where tracing/debugging
+      can be performed. The <c><![CDATA[net_kernel:start/1]]></c> routine takes a
+      list as it's single argument. The lists first element should be
+      the node name (without the "@hostname") as an atom, and the second (and
+      last) element should be one of the atoms <c><![CDATA[shortnames]]></c> or 
+      <c><![CDATA[longnames]]></c>. In the example case <c><![CDATA[shortnames]]></c> is
+      preferred. </p>
+    <p>For net kernel to find out which distribution module to use, the
+      command line argument <c><![CDATA[-proto_dist]]></c> is used. The argument
+      is followed by one or more distribution module names, with the
+      "_dist" suffix removed, i.e. uds_dist as a distribution module
+      is specified as <c><![CDATA[-proto_dist uds]]></c>.</p>
+    <p>If no epmd (TCP port mapper daemon) is used, one should also
+      specify the command line option <c><![CDATA[-no_epmd]]></c>, which will make
+      Erlang skip the epmd startup, both as a OS process and as an
+      Erlang ditto.</p>
+    <p>The path to the directory where the distribution modules reside
+      must be known at boot, which can either be achieved by
+      specifying <c><![CDATA[-pa <path>]]></c> on the command line or by building
+      a boot script containing the applications used for your
+      distribution protocol (in the uds_dist protocol, it's only the
+      uds_dist application that needs to be added to the script).</p>
+    <p>The distribution will be started at boot if all the above is
+      specified and an <c><![CDATA[-sname <name>]]></c> flag is present at the
+      command line, here follows two examples: </p>
+    <pre>
+$ <input>erl -pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin -proto_dist uds -no_epmd</input>
+Erlang (BEAM) emulator version 5.0 
+ 
+Eshell V5.0  (abort with ^G)
+1> <input>net_kernel:start([bing,shortnames]).</input>
+{ok,&lt;0.30.0>}
+(bing@hador)2></pre>
+    <p>...</p>
+    <pre>
+$ <input>erl -pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin -proto_dist uds \\ </input>
+<input>      -no_epmd -sname bong</input>
+Erlang (BEAM) emulator version 5.0 
+ 
+Eshell V5.0  (abort with ^G)
+(bong@hador)1></pre>
+    <p>One can utilize the ERL_FLAGS environment variable to store the
+      complicated parameters in:</p>
+    <pre>
+$ <input>ERL_FLAGS=-pa $ERL_TOP/lib/kernel/examples/uds_dist/ebin \\ </input>
+<input>      -proto_dist uds -no_epmd</input>
+$ <input>export ERL_FLAGS</input>
+$ <input>erl -sname bang</input>
+Erlang (BEAM) emulator version 5.0 
+ 
+Eshell V5.0  (abort with ^G)
+(bang@hador)1></pre>
+    <p>The <c><![CDATA[ERL_FLAGS]]></c> should preferably not include the name of
+      the node.</p>
+  </section>
+</chapter>
+