From 78fad16ef7c5477239bc0b51125fabfe6567039d Mon Sep 17 00:00:00 2001 From: Rickard Green Date: Fri, 30 Jun 2017 16:32:42 +0200 Subject: Support for distribution controller processes --- erts/doc/src/alt_dist.xml | 653 +++++++++++++++++++++++++++++++++++++++++++++- erts/doc/src/erlang.xml | 154 +++++++++++ 2 files changed, 795 insertions(+), 12 deletions(-) (limited to 'erts/doc') diff --git a/erts/doc/src/alt_dist.xml b/erts/doc/src/alt_dist.xml index be969a8267..3d87e9dcdb 100644 --- a/erts/doc/src/alt_dist.xml +++ b/erts/doc/src/alt_dist.xml @@ -47,23 +47,25 @@ runs on. The reason the C code is not made portable, is simply readability.

- -

This section was written a long time ago. Most of it is still - valid, but some things have changed since then. - Most notably is the driver interface. Some updates have been made - to the documentation of the driver presented here, - but more can be done and is planned for the future. - The reader is encouraged to read the - erl_driver and - driver_entry - documentation also.

- -
Introduction

To implement a new carrier for the Erlang distribution, the main steps are as follows.

+

+ As of ERTS version 10.0 support for distribution controller + processes has been introduced. That is, the traffic over a + distribution channel can be managed by a process instead of + only by a port. This makes it possible to implement large + parts of the logic in Erlang code, and you perhaps do not + even need a new driver for the protocol. One example could + be Erlang distribution over UDP using gen_udp (your + Erlang code will of course have to take care of retranspissions, + etc in this example). That is, depending on what you want + to do you perhaps do not need to implement a driver at all + and can then skip the driver related sections below. +

+
Writing an Erlang Driver

First, the protocol must be available to the Erlang machine, which @@ -151,8 +153,635 @@

+
+ + Distribution Module +

+ The distribution module expose an API that net_kernel call + in order to manage connections to other nodes. The module name + should have the suffix _dist. +

+

+ The module needs to create some kind of listening entity (process + or port) and an acceptor process that accepts incoming connections + using the listening entity. For each connection, the module at least + needs to create one connection supervisor process, which also is + responsible for the handshake when setting up the connection, and + a distribution controller (process or port) responsible for + transport of data over the connection. The distribution controller + and the connection supervisor process should be linked together + so both of them are cleaned up when the connection is taken down. +

+

+ Note that there need to be exactly one distribution controller + per connection. A process or port can only be distribution + controller for one connection. The registration as distribution + controller cannot be undone. It will stick until the distribution + controller terminates. The distribution controller should not + ignore exit signals. It is allowed to trap exits, but it should + then voluntarily terminate when an exit signal is received. +

+ +
+ + Exported Callback Functions + +

+ The following functions are mandatory: +

+ + listen(Name) ->
  {ok, {Listen, Address, Creation}} | {error, Error} + +

+ listen/1 is called once in order to listen for incoming + connection requests. The call is made when the distribution is brought + up. The argument Name is the part of the node name before + the @ sign in the full node name. It can be either an atom or a + string. +

+

+ The return value consists of a Listen handle (which is + later passed to the accept/1 + callback), Address which is a #net_address{} record + with information about the address for the node (the + #net_address{} record is defined in + kernel/include/net_address.hrl), and Creation which + (currently) is an integer 1, 2, or 3. +

+

+ If epmd is to be used + for node discovery, you typically want to use the (unfortunately + undocumented) erl_epmd module (part of the kernel + application) in order to register the listen port with epmd + and retrieve Creation to use. +

+ + + accept(Listen) ->
  AcceptorPid + +

+ accept/1 should spawn a process that accepts connections. This + process should preferably execute on max priority. The process + identifier of this process should be returned. +

+

+ The Listen argument will be the same as the Listen handle + part of the return value of the + listen/1 callback above. + accept/1 is called only once when the distribution protocol is + started. +

+

+ The caller of this function is a representative for net_kernel + (this may or may not be the process registered as net_kernel) + and is in this document identified as Kernel. + When a connection has been accepted by the acceptor process, it needs + to inform Kernel about the accepted connection. This is done by + passing a message on the form: +

+ +

+ DistController is either the process or port identifier + of the distribution controller for the connection. The + distribution controller should be created by the acceptor + processes when a new connection is accepted. Its job is to + dispatch traffic on the connection. +

+ Kernel responds with one of the following messages: + + {Kernel, controller, SupervisorPid} + +

+ The request was accepted and SupervisorPid is the + process identifier of the connection supervisor process + (which is created in the + accept_connection/5 + callback). +

+ + {Kernel, unsupported_protocol} + +

+ The request was rejected. This is a fatal error. The acceptor + process should terminate. +

+ + +

+ When an accept sequence has been completed the acceptor process + is expected to continue accepting further requests. +

+ + + accept_connection(AcceptorPid, DistCtrl, MyNode, Allowed, SetupTime) ->
  ConnectionSupervisorPid + +

+ accept_connection/5 should spawn a process that will + perform the Erlang distribution handshake for the connection. + If the handshake successfully completes it should continue to + function as a connection supervisor. This process + should preferably execute on max priority. +

+

The arguments:

+ + AcceptorPid + +

+ Process identifier of the process created by the + accept/1 + callback. +

+ + DistCtrl + +

The identifier of the distribution controller identifier + created by the acceptor process. To be passed along to + dist_util:handshake_other_started(HsData). +

+ + MyNode + +

+ Node name of this node. To be passed along to + dist_util:handshake_other_started(HsData). +

+ + Allowed + +

+ To be passed along to + dist_util:handshake_other_started(HsData). +

+ + SetupTime + +

+ Time used for creating a setup timer by a + call to dist_util:start_timer(SetupTime). + The timer should be passed along to + dist_util:handshake_other_started(HsData). +

+ + +

+ The created process should provide callbacks and other + information needed for the handshake in a + #hs_data{} + record and call dist_util:handshake_other_started(HsData) + with this record. +

+

+ dist_util:handshake_other_started(HsData) will perform + the handshake and if the handshake successfully completes this + process will then continue in a connection supervisor loop + as long as the connection is up. +

+ + + setup(Node, Type, MyNode, LongOrShortNames, SetupTime) ->
  ConnectionSupervisorPid + +

+ setup/5 should spawn a process that connects to + Node. When connection has been established it should + perform the Erlang distribution handshake for the connection. + If the handshake successfully completes it should continue to + function as a connection supervisor. This process + should preferably execute on max priority. +

+

The arguments:

+ + Node + +

+ Node name of remote node. To be passed along to + dist_util:handshake_we_started(HsData). +

+ + Type + +

+ Connection type. To be passed along to + dist_util:handshake_we_started(HsData). +

+ + MyNode + +

+ Node name of this node. To be passed along to + dist_util:handshake_we_started(HsData). +

+ + LongOrShortNames + +

+ Either the atom longnames or + the atom shortnames indicating + whether long or short names is used. +

+ + SetupTime + +

+ Time used for creating a setup timer by a + call to dist_util:start_timer(SetupTime). + The timer should be passed along to + dist_util:handshake_we_started(HsData). +

+ + +

+ The caller of this function is a representative for net_kernel + (this may or may not be the process registered as net_kernel) + and is in this document identified as Kernel. +

+

+ This function should, besides spawning the connection supervisor, + also create a distribution controller. The distribution + controller is either a process or a port which is responsible + for dispatching traffic. +

+

+ The created process should provide callbacks and other + information needed for the handshake in a + #hs_data{} + record and call dist_util:handshake_we_started(HsData) + with this record. +

+

+ dist_util:handshake_we_started(HsData) will perform + the handshake and the handshake successfully completes this + process will then continue in a connection supervisor loop + as long as the connection is up. +

+ + + close(Listen) ->
  void() + +

+ Called in order to close the Listen handle + that originally was passed from the + listen/1 callback. +

+ + select(NodeName) ->
  boolean() + +

Return true if the host name part + of the NodeName is valid for use + with this protocol; otherwise, false. +

+ + + + +

+ There are also two optional functions that may be + exported: +

+ + setopts(Listen, Opts) ->
  ok | {error, Error} + +

+ The argument Listen is the handle originally passed + from the + listen/1 callback. + The argument Opts is a list of options to set on future + connections. +

+ + + getopts(Listen, Opts) ->
  {ok, OptionValues} | {error, Error} + +

+ The argument Listen is the handle originally passed + from the + listen/1 callback. + The argument Opts is a list of options to read for future + connections. +

+ + + +
+
+ + The #hs_data{} Record +

+ The dist_util:handshake_we_started/1 and + dist_util:handshake_other_started/1 functions + takes a #hs_data{} record as argument. There + are quite a lot of fields in this record that you + need to set. The record is defined in + kernel/include/dist_util.hrl. Not documented + fields should not be set, i.e., should be left as + undefined. +

+

+ The following #hs_data{} record fields need + to be set unless otherwise stated:

+ + kernel_pid + +

+ Process identifier of the Kernel process. That is, + the process that called either + setup/5 or + accept_connection/5. +

+ + + other_node + +

Name of the other node. This field is only + mandatory when this node initiates the connection. + That is, when connection is set up via + setup/5. +

+ + + this_node + +

+ The node name of this node. +

+ + + socket + +

+ The identifier of the distribution controller. +

+ + + timer + +

+ The timer created using dist_util:start_timer/1. +

+ + + allowed + +

Information passed as Allowed to + accept_connection/5. This field is only + mandatory when the remote node initiated the + connection. That is, when the connection is set + up via + accept_connection/5. +

+ + + f_send + +

+ A fun with the following signature: +

+ ok | {error, Error}]]> +

+ where DistCtrlr is the identifier of + the distribution controller and Data + is io data to pass to the other side. +

+

Only used during handshake phase.

+ + + f_recv + +

+ A fun with the following signature: +

+ {ok, Packet} | {error, Reason}]]> +

+ where DistCtrlr is the identifier of the distribution + controller. + If Length is 0, all available bytes should be + returned. If Length > 0, exactly Length bytes + should be returned, or an error; possibly discarding less + than Length bytes of data when the connection is + closed from the other side. + It is used for passive receive of data from the + other end. +

+

Only used during handshake phase.

+ + + f_setopts_pre_nodeup + +

+ A fun with the following signature: +

+ ok | {error, Error}]]> +

+ where DistCtrlr is the identifier of + the distribution controller. Called just + before the distribution channel is taken up + for normal traffic. +

+

Only used during handshake phase.

+ + + f_setopts_post_nodeup + +

+ A fun with the following signature: +

+ ok | {error, Error}]]> +

+ where DistCtrlr is the identifier of + the distribution controller. Called just + after distribution channel has been taken + up for normal traffic. +

+

Only used during handshake phase.

+ + + f_getll + +

+ A fun with the following signature: +

+ ID]]> +

+ where DistCtrlr is the identifier of + the distribution controller and ID is + the identifier of the low level entity that + handles the connection (often DistCtrlr + itself). +

+

Only used during handshake phase.

+ + + f_address + +

+ A fun with the following signature: +

+ NetAddress]]> +

+ where DistCtrlr is the identifier of + the distribution controller, Node + is the node name of the node on the other end, + and NetAddress is a #net_address{} + record with information about the address + for the Node on the other end of the + connection. The #net_address{} record + is defined in + kernel/include/net_address.hrl. +

+

Only used during handshake phase.

+ + + mf_tick + +

+ A fun with the following signature: +

+ void()]]> +

+ where DistCtrlr is the identifier + of the distribution controller. This + function should send information over + the connection that is not interpreted + by the other end while increasing the + statistics of received packets on the + other end. This is usually implemented by + sending an empty packet. +

+

+ It is of vital importance that this operation + does not block the caller for a long time. + This since it is called from the connection + supervisor. +

+

Used when connection is up.

+ + + mf_getstat + +

+ A fun with the following signature: +

+ {ok, Received, Sent, PendSend}]]> +

+ where DistCtrlr is the identifier + of the distribution controller, Received + is received packets, Sent is + sent packets, and PendSend is + amount of packets in queue to be sent + or a boolean() indicating whether + there are packets in queue to be sent. +

+

+ It is of vital importance that this operation + does not block the caller for a long time. + This since it is called from the connection + supervisor. +

+

Used when connection is up.

+ + + request_type + +

+ The request Type as passed to + setup/5. + This is only mandatory when the connection has + been initiated by this node. That is, the connection + is set up via setup/5. +

+ + + mf_setopts + +

+ A fun with the following signature: +

+ ok | {error, Error}]]> +

+ where DistCtrlr is the identifier + of the distribution controller and Opts + is a list of options to set on the connection. +

+

This function is optional. Used when connection is up.

+ + + mf_getopts + +

+ A fun with the following signature: +

+ {ok, OptionValues} | {error, Error}]]> +

+ where DistCtrlr is the identifier + of the distribution controller and Opts + is a list of options to read for the connection. +

+

This function is optional. Used when connection is up.

+ + + f_handshake_complete + +

+ A fun with the following signature: +

+ void()]]> +

+ where DistCtrlr is the identifier + of the distribution controller, Node is + the node name of the node connected at the other + end, and DHandle is a distribution handle + needed by a distribution controller process when + calling the following BIFs: +

+ +

erlang:dist_ctrl_get_data/1

 +

erlang:dist_ctrl_get_data_notification/1

 +

erlang:dist_ctrl_input_handler/2

 +

erlang:dist_ctrl_put_data/2

 + +

+ This function is called when the handshake has + completed and the distribution channel is up. + The distribution controller can begin dispatching + traffic over the channel. This function is optional. +

+

Only used during handshake phase.

+ + + +
+ +
+ + Enable Your Distribution Module + +

For net_kernel to find out which distribution module to use, + the erl command-line argument -proto_dist is used. It + is followed by one or more distribution module names, with suffix + "_dist" removed. That is, gen_tcp_dist as a distribution module + is specified as -proto_dist gen_tcp.

+ +

If no epmd (TCP port mapper daemon) is used, also command-line + option -no_epmd is to be specified, which makes + Erlang skip the epmd startup, both as an OS process and as an + Erlang ditto.

+
+ +
+
The Driver + + +

This section was written a long time ago. Most of it is still + valid, but some things have changed since then. Some updates have + been made to the documentation of the driver presented here, + but more can be done and is planned for the future. + The reader is encouraged to read the + erl_driver and + driver_entry + documentation also.

+ +

Although Erlang drivers in general can be beyond the scope of this section, a brief introduction seems to be in place.

diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 105734d5b2..8b0f97f30f 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -181,6 +181,14 @@ + + + + +

An opaque handle identifing a distribution channel.

+ + + @@ -1214,6 +1222,152 @@ end + + + Get distribution channel data to pass to another node. + +

+ Get distribution channel data from the local node that is + to be passed to the remote node. The distribution channel + is identified by DHandle. If no data + is available, the atom none is returned. One + can request to be informed by a message when more + data is available by calling + erlang:dist_ctrl_get_data_notification(DHandle). +

+

+ The data retreived from this function needs to be delivered + as is to the node on the other end in the exact same order, + with no loss of data what so ever, as retrived from this + function. +

+

+ Only the process registered as distribution + controller for the distribution channel identified by + DHandle is allowed to call this + function. +

+

+ This function is used when implementing an alternative + distribution carrier using processes as distribution + controllers. DHandle is retrived + via the callback + f_handshake_complete. + More information about this can be found in the documentation of + ERTS + User's Guide ➜ How to implement an Alternative Carrier + for the Erlang Distribution ➜ Distribution Module. +

+ + + + + + Request notification about available outgoing distribution channel data. + +

+ Request notification when more data is available to + fetch using + erlang:dist_ctrl_get_data(DHandle) + for the distribution channel identified by + DHandle. When more data is present, + the caller will be sent the message dist_data. + Once a dist_data messages has been sent, no + more dist_data messages will be sent until + the dist_ctrl_get_data_notification/1 function has been called + again. +

+

+ Only the process registered as distribution + controller for the distribution channel identified by + DHandle is allowed to call this + function. +

+

+ This function is used when implementing an alternative + distribution carrier using processes as distribution + controllers. DHandle is retrived + via the callback + f_handshake_complete. + More information about this can be found in the documentation of + ERTS + User's Guide ➜ How to implement an Alternative Carrier + for the Erlang Distribution ➜ Distribution Module. +

+ + + + + + Register distribution channel input handler process. + +

+ Register an alternate input handler process for the + distribution channel identified by DHandle. + Once this function has been called, InputHandler + is the only process allowed to call + erlang:dist_ctrl_put_data(DHandle, Data) + with the DHandle identifing this distribution + channel. +

+

+ Only the process registered as distribution + controller for the distribution channel identified by + DHandle is allowed to call this + function. +

+

+ This function is used when implementing an alternative + distribution carrier using processes as distribution + controllers. DHandle is retrived + via the callback + f_handshake_complete. + More information about this can be found in the documentation of + ERTS + User's Guide ➜ How to implement an Alternative Carrier + for the Erlang Distribution ➜ Distribution Module. +

+ + + + + + Pass data into the VM from a distribution channel. + +

+ Deliver distribution channel data from a remote node to the + local node. +

+

+ The data passed to the VM using this function needs to be + passed in the exact same order, and with no loss of data + what so ever, as sent from the node on the other end. +

+

+ Only the process registered as distribution + controller for the distribution channel identified by + DHandle is allowed to call this + function unless an alternate input handler process + has been registered using + erlang:dist_ctrl_input_handler(DHandle, InputHandler). + If an alternate input handler has been registered, only + the registered input handler process is allowed to call + this function. +

+

+ This function is used when implementing an alternative + distribution carrier using processes as distribution + controllers. DHandle is retrived + via the callback + f_handshake_complete. + More information about this can be found in the documentation of + ERTS + User's Guide ➜ How to implement an Alternative Carrier + for the Erlang Distribution ➜ Distribution Module. +

+ + + Return the Nth element of a tuple. -- cgit v1.2.3