Spread a server over three nodes, one of which terminates a peer
connection, the other two to handle requests. Terminate transport on one
of the server nodes and ensure that answers come only from the other
two.

To be able to restrict how many AVPs will be examined (from the front of
a message) when looking for Session-Id, and to decide what to do with if
the AVP isn't found. Options are specified as a map of the following
form.

  #{search => non_neg_integer(),
    default => discard | mfa(),
    dispatch => list() | mfa()}

The search member says how many AVPs to examine at most, from the front
of the message. If the optional value of a Session-Id is not the name of
a connected node then the default member determines what to do with the
request, handle it locally (the default), discard it, or invoke an MFA
on the Session-Id | false (if none was found) and diameter_packet record
to return a node() | false; if the latter then the request is discarded.

If a node is identified then the dispatch MFA is invoked on the node and
the request MFA (as three arguments), a list Opts being equivalent to
the MFA {erlang, spawn_opt, [Opts]}, and the default being the empty
list.

Integer- or list-valued options are equivalent to the corresponding map
with a single value.

Limiting the search is to avoid searching messages containing many AVPs
for a Session-Id that is known to occur near the header, since section
8.8 of RFC 6733 says this:

   When present, the Session-Id SHOULD appear immediately
   following the Diameter header (see Section 3).

There's no guarantee, but in practice it may well be known that peers
are respecting the RFC, and in that case limiting the search is a
defense against searching messages from a malicious peer unnecessarily.
The search is unlimited by default.

A default is only used when a search fails to locate a Session-Id, and
can be to discard the message, or have a node() or false be returned
from an MFA applied to the diameter_packet in question. The local node
is chosen by default.

That is, of functions that can be configured as spawn_opt MFAs in
transport configuration.

This commits adds the spawn_local described in the parent commit, and a
route_session that assumes that the local node initiates all sessions
with Session-Id returned by diameter:session_id/1, and handles incoming
requests on the node on which the id in question was returned,
diameter:session_id/1 using node() as optional value in the Session-Id
format.

The possibility of configuring an MFA as spawn_opt was added in commit
fd285079, the callback being passed an arity-0 fun to be applied in an
appropriate handler process. Replace the fun by a tuple to be passed to
diameter_traffic:request/1, to avoid passing funs between nodes when
handler processes are remote.

A list-valued spawn_opt is now equivalent to the following configured as
{spawn_opt, {Mod, spawn_local, [Opts]}}.

  spawn_local(ReqT, Opts) ->
      spawn_opt(diameter_traffic, request, [ReqT], Opts).

ReqT is passed by diameter and contains information that the callback
may want to decide where to handle the request in question (which wasn't
accessible with a fun), but this information isn't exposed in a
documented way. The intention is instead to add an own callback
implementation to make use of the information.

Note that application lookup now takes place in the watchdog process in
both the list-valued (or no configuration) and mfa-valued cases. Whether
this is good, bad, or (probably) inconsequential remains to be seen.

The badmatch in the grandparent commit is ssl not being installed:

=== Reason: no match of right hand side value {ssl,non_existing,
                                                    ["non_existing"]}
  in function  diameter_app_SUITE:app/1 (diameter_app_SUITE.erl, line 266)
  in call from diameter_app_SUITE:'-xref/1-fun-3-'/2 (diameter_app_SUITE.erl, line 214)
  in call from lists:'-filter/2-lc$^0/1-0-'/2 (lists.erl, line 1286)
  in call from diameter_app_SUITE:xref/1 (diameter_app_SUITE.erl, line 214)
  in call from test_server:ts_tc/3 (test_server.erl, line 1545)
  in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1063)
  in call from test_server:run_test_case_eval/9 (test_server.erl, line 995)

Tweak the failure a bit more, without fixing it.

See this failure in nightly test on one host, indicating that
code:which/1 is returning something other than the expected path:

=== Ended at 2018-06-08 05:10:41
=== Location: [{lists,nth,170},
              {diameter_app_SUITE,app,265},
              {diameter_app_SUITE,'-xref/1-fun-3-',214},
              {lists,'-filter/2-lc$^0/1-0-',1286},
              {diameter_app_SUITE,xref,214},
              {test_server,ts_tc,1545},
              {test_server,run_test_case_eval1,1063},
              {test_server,run_test_case_eval,995}]
=== Reason: no function clause matching lists:nth(2,[]) (lists.erl, line 170)
  in function  diameter_app_SUITE:app/1 (diameter_app_SUITE.erl, line 265)
  in call from diameter_app_SUITE:'-xref/1-fun-3-'/2 (diameter_app_SUITE.erl, line 214)
  in call from lists:'-filter/2-lc$^0/1-0-'/2 (lists.erl, line 1286)
  in call from diameter_app_SUITE:xref/1 (diameter_app_SUITE.erl, line 214)
  in call from test_server:ts_tc/3 (test_server.erl, line 1545)
  in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1063)
  in call from test_server:run_test_case_eval/9 (test_server.erl, line 995)

Commit fae8ca0c inadvertently removed the monitor at add/1 and
add_new/1. As a result, process death did not remove associations,
causing table diameter_reg to leak entries and stop/start of a service
to fail.

Add a testcase to detect the problem, which existing testcases miss.

* anders/diameter/sctp/OTP-10889:
  Make unordered delivery configurable

Changing the default in the parent commit is possibly a bit dangerous,
even if the motivation still holds. Take a step back and make unordered
delivery a matter of configuration, without changing the default:
configuration is {unordered, boolean() | pos_integer()}, with false the
default, and N equivalent to OS =< N, where OS is the number of outbound
streams negotiated on the association in question.

A user can mess with this by configuring an sctp_default_send_param of
their own, but unordered sending is them from start, not only after the
second message reception.