<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE chapter SYSTEM "chapter.dtd"> <chapter> <header> <copyright> <year>1997</year><year>2017</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. </legalnotice> <title>Supervisor Behaviour</title> <prepared></prepared> <docno></docno> <date></date> <rev></rev> <file>sup_princ.xml</file> </header> <p>This section should be read with the <seealso marker="stdlib:supervisor">supervisor(3)</seealso> manual page in STDLIB, where all details about the supervisor behaviour is given.</p> <section> <title>Supervision Principles</title> <p>A supervisor is responsible for starting, stopping, and monitoring its child processes. The basic idea of a supervisor is that it is to keep its child processes alive by restarting them when necessary.</p> <p>Which child processes to start and monitor is specified by a list of <seealso marker="#spec">child specifications</seealso>. The child processes are started in the order specified by this list, and terminated in the reversed order.</p> </section> <section> <title>Example</title> <p>The callback module for a supervisor starting the server from <seealso marker="gen_server_concepts#ex">gen_server Behaviour</seealso> can look as follows:</p> <marker id="ex"></marker> <code type="none"> -module(ch_sup). -behaviour(supervisor). -export([start_link/0]). -export([init/1]). start_link() -> supervisor:start_link(ch_sup, []). init(_Args) -> SupFlags = #{strategy => one_for_one, intensity => 1, period => 5}, ChildSpecs = [#{id => ch3, start => {ch3, start_link, []}, restart => permanent, shutdown => brutal_kill, type => worker, modules => [cg3]}], {ok, {SupFlags, ChildSpecs}}.</code> <p>The <c>SupFlags</c> variable in the return value from <c>init/1</c> represents the <seealso marker="#flags">supervisor flags</seealso>.</p> <p>The <c>ChildSpecs</c> variable in the return value from <c>init/1</c> is a list of <seealso marker="#spec">child specifications</seealso>.</p> </section> <section> <marker id="flags"/> <title>Supervisor Flags</title> <p>This is the type definition for the supervisor flags:</p> <code type="none"><![CDATA[ sup_flags() = #{strategy => strategy(), % optional intensity => non_neg_integer(), % optional period => pos_integer()} % optional strategy() = one_for_all | one_for_one | rest_for_one | simple_one_for_one]]></code> <list type="bulleted"> <item> <p><c>strategy</c> specifies the <seealso marker="#strategy">restart strategy</seealso>.</p> </item> <item> <p><c>intensity</c> and <c>period</c> specify the <seealso marker="#max_intensity">maximum restart intensity</seealso>.</p> </item> </list> </section> <section> <marker id="strategy"/> <title>Restart Strategy</title> <p> The restart strategy is specified by the <c>strategy</c> key in the supervisor flags map returned by the callback function <c>init</c>:</p> <code type="none"> SupFlags = #{strategy => Strategy, ...}</code> <p>The <c>strategy</c> key is optional in this map. If it is not given, it defaults to <c>one_for_one</c>.</p> <section> <title>one_for_one</title> <p>If a child process terminates, only that process is restarted.</p> <marker id="sup4"></marker> <image file="../design_principles/sup4.gif"> <icaption>One_For_One Supervision</icaption> </image> </section> <section> <title>one_for_all</title> <p>If a child process terminates, all other child processes are terminated, and then all child processes, including the terminated one, are restarted.</p> <marker id="sup5"></marker> <image file="../design_principles/sup5.gif"> <icaption>One_For_All Supervision</icaption> </image> </section> <section> <title>rest_for_one</title> <p>If a child process terminates, the rest of the child processes (that is, the child processes after the terminated process in start order) are terminated. Then the terminated child process and the rest of the child processes are restarted.</p> </section> <section> <title>simple_one_for_one</title> <p>See <seealso marker="#simple">simple-one-for-one supervisors</seealso>.</p> </section> </section> <section> <marker id="max_intensity"></marker> <title>Maximum Restart Intensity</title> <p>The supervisors have a built-in mechanism to limit the number of restarts which can occur in a given time interval. This is specified by the two keys <c>intensity</c> and <c>period</c> in the supervisor flags map returned by the callback function <c>init</c>:</p> <code type="none"> SupFlags = #{intensity => MaxR, period => MaxT, ...}</code> <p>If more than <c>MaxR</c> number of restarts occur in the last <c>MaxT</c> seconds, the supervisor terminates all the child processes and then itself. The termination reason for the supervisor itself in that case will be <c>shutdown</c>.</p> <p>When the supervisor terminates, then the next higher-level supervisor takes some action. It either restarts the terminated supervisor or terminates itself.</p> <p>The intention of the restart mechanism is to prevent a situation where a process repeatedly dies for the same reason, only to be restarted again.</p> <p>The keys <c>intensity</c> and <c>period</c> are optional in the supervisor flags map. If they are not given, they default to <c>1</c> and <c>5</c>, respectively.</p> <section> <title>Tuning the intensity and period</title> <p>The default values are 1 restart per 5 seconds. This was chosen to be safe for most systems, even with deep supervision hierarchies, but you will probably want to tune the settings for your particular use case.</p> <p>First, the intensity decides how big bursts of restarts you want to tolerate. For example, you might want to accept a burst of at most 5 or 10 attempts, even within the same second, if it results in a successful restart.</p> <p>Second, you need to consider the sustained failure rate, if crashes keep happening but not often enough to make the supervisor give up. If you set intensity to 10 and set the period as low as 1, the supervisor will allow child processes to keep restarting up to 10 times per second, forever, filling your logs with crash reports until someone intervenes manually.</p> <p>You should therefore set the period to be long enough that you can accept that the supervisor keeps going at that rate. For example, if you have picked an intensity value of 5, then setting the period to 30 seconds will give you at most one restart per 6 seconds for any longer period of time, which means that your logs won't fill up too quickly, and you will have a chance to observe the failures and apply a fix.</p> <p>These choices depend a lot on your problem domain. If you don't have real time monitoring and ability to fix problems quickly, for example in an embedded system, you might want to accept at most one restart per minute before the supervisor should give up and escalate to the next level to try to clear the error automatically. On the other hand, if it is more important that you keep trying even at a high failure rate, you might want a sustained rate of as much as 1-2 restarts per second.</p> <p>Avoiding common mistakes:</p> <list type="bulleted"> <item> <p>Do not forget to consider the burst rate. If you set intensity to 1 and period to 6, it gives the same sustained error rate as 5/30 or 10/60, but will not allow even 2 restart attempts in quick succession. This is probably not what you wanted.</p> </item> <item> <p>Do not set the period to a very high value if you want to tolerate bursts. If you set intensity to 5 and period to 3600 (one hour), the supervisor will allow a short burst of 5 restarts, but then gives up if it sees another single restart almost an hour later. You probably want to regard those crashes as separate incidents, so setting the period to 5 or 10 minutes will be more reasonable.</p> </item> <item> <p>If your application has multiple levels of supervision, then do not simply set the restart intensities to the same values on all levels. Keep in mind that the total number of restarts (before the top level supervisor gives up and terminates the application) will be the product of the intensity values of all the supervisors above the failing child process.</p> <p>For example, if the top level allows 10 restarts, and the next level also allows 10, a crashing child below that level will be restarted 100 times, which is probably excessive. Allowing at most 3 restarts for the top level supervisor might be a better choice in this case.</p> </item> </list> </section> </section> <section> <marker id="spec"></marker> <title>Child Specification</title> <p>The type definition for a child specification is as follows:</p> <code type="none"><![CDATA[ child_spec() = #{id => child_id(), % mandatory start => mfargs(), % mandatory restart => restart(), % optional shutdown => shutdown(), % optional type => worker(), % optional modules => modules()} % optional child_id() = term() mfargs() = {M :: module(), F :: atom(), A :: [term()]} modules() = [module()] | dynamic restart() = permanent | transient | temporary shutdown() = brutal_kill | timeout() worker() = worker | supervisor]]></code> <list type="bulleted"> <item> <p><c>id</c> is used to identify the child specification internally by the supervisor.</p> <p>The <c>id</c> key is mandatory.</p> <p>Note that this identifier occasionally has been called "name". As far as possible, the terms "identifier" or "id" are now used but in order to keep backwards compatibility, some occurences of "name" can still be found, for example in error messages.</p> </item> <item> <p><c>start</c> defines the function call used to start the child process. It is a module-function-arguments tuple used as <c>apply(M, F, A)</c>.</p> <p>It is to be (or result in) a call to any of the following:</p> <list type="bulleted"> <item><c>supervisor:start_link</c></item> <item><c>gen_server:start_link</c></item> <item><c>gen_statem:start_link</c></item> <item><c>gen_event:start_link</c></item> <item>A function compliant with these functions. For details, see the <c>supervisor(3)</c> manual page.</item> </list> <p>The <c>start</c> key is mandatory.</p> </item> <item> <p><c>restart</c> defines when a terminated child process is to be restarted.</p> <list type="bulleted"> <item>A <c>permanent</c> child process is always restarted.</item> <item>A <c>temporary</c> child process is never restarted (not even when the supervisor restart strategy is <c>rest_for_one</c> or <c>one_for_all</c> and a sibling death causes the temporary process to be terminated).</item> <item>A <c>transient</c> child process is restarted only if it terminates abnormally, that is, with an exit reason other than <c>normal</c>, <c>shutdown</c>, or <c>{shutdown,Term}</c>.</item> </list> <p>The <c>restart</c> key is optional. If it is not given, the default value <c>permanent</c> will be used.</p> </item> <item> <marker id="shutdown"></marker> <p><c>shutdown</c> defines how a child process is to be terminated.</p> <list type="bulleted"> <item><c>brutal_kill</c> means that the child process is unconditionally terminated using <c>exit(Child, kill)</c>.</item> <item>An integer time-out value means that the supervisor tells the child process to terminate by calling <c>exit(Child, shutdown)</c> and then waits for an exit signal back. If no exit signal is received within the specified time, the child process is unconditionally terminated using <c>exit(Child, kill)</c>.</item> <item>If the child process is another supervisor, it is to be set to <c>infinity</c> to give the subtree enough time to shut down. It is also allowed to set it to <c>infinity</c>, if the child process is a worker. See the warning below:</item> </list> <warning> <p>Be careful when setting the shutdown time to <c>infinity</c> when the child process is a worker. Because, in this situation, the termination of the supervision tree depends on the child process; it must be implemented in a safe way and its cleanup procedure must always return.</p> </warning> <p>The <c>shutdown</c> key is optional. If it is not given, and the child is of type <c>worker</c>, the default value <c>5000</c> will be used; if the child is of type <c>supervisor</c>, the default value <c>infinity</c> will be used.</p> </item> <item> <p><c>type</c> specifies if the child process is a supervisor or a worker.</p> <p>The <c>type</c> key is optional. If it is not given, the default value <c>worker</c> will be used.</p> </item> <item> <p><c>modules</c> are to be a list with one element <c>[Module]</c>, where <c>Module</c> is the name of the callback module, if the child process is a supervisor, gen_server, gen_statem. If the child process is a gen_event, the value shall be <c>dynamic</c>.</p> <p>This information is used by the release handler during upgrades and downgrades, see <seealso marker="release_handling">Release Handling</seealso>.</p> <p>The <c>modules</c> key is optional. If it is not given, it defaults to <c>[M]</c>, where <c>M</c> comes from the child's start <c>{M,F,A}</c>.</p> </item> </list> <p><em>Example:</em> The child specification to start the server <c>ch3</c> in the previous example look as follows:</p> <code type="none"> #{id => ch3, start => {ch3, start_link, []}, restart => permanent, shutdown => brutal_kill, type => worker, modules => [ch3]}</code> <p>or simplified, relying on the default values:</p> <code type="none"> #{id => ch3, start => {ch3, start_link, []} shutdown => brutal_kill}</code> <p>Example: A child specification to start the event manager from the chapter about <seealso marker="events#mgr">gen_event</seealso>:</p> <code type="none"> #{id => error_man, start => {gen_event, start_link, [{local, error_man}]}, modules => dynamic}</code> <p>Both server and event manager are registered processes which can be expected to be always accessible. Thus they are specified to be <c>permanent</c>.</p> <p><c>ch3</c> does not need to do any cleaning up before termination. Thus, no shutdown time is needed, but <c>brutal_kill</c> is sufficient. <c>error_man</c> can need some time for the event handlers to clean up, thus the shutdown time is set to 5000 ms (which is the default value).</p> <p>Example: A child specification to start another supervisor:</p> <code type="none"> #{id => sup, start => {sup, start_link, []}, restart => transient, type => supervisor} % will cause default shutdown=>infinity</code> </section> <section> <marker id="super_tree"></marker> <title>Starting a Supervisor</title> <p>In the previous example, the supervisor is started by calling <c>ch_sup:start_link()</c>:</p> <code type="none"> start_link() -> supervisor:start_link(ch_sup, []).</code> <p><c>ch_sup:start_link</c> calls function <c>supervisor:start_link/2</c>, which spawns and links to a new process, a supervisor.</p> <list type="bulleted"> <item>The first argument, <c>ch_sup</c>, is the name of the callback module, that is, the module where the <c>init</c> callback function is located.</item> <item>The second argument, <c>[]</c>, is a term that is passed as is to the callback function <c>init</c>. Here, <c>init</c> does not need any indata and ignores the argument.</item> </list> <p>In this case, the supervisor is not registered. Instead its pid must be used. A name can be specified by calling <c>supervisor:start_link({local, Name}, Module, Args)</c> or <c>supervisor:start_link({global, Name}, Module, Args)</c>.</p> <p>The new supervisor process calls the callback function <c>ch_sup:init([])</c>. <c>init</c> shall return <c>{ok, {SupFlags, ChildSpecs}}</c>:</p> <code type="none"> init(_Args) -> SupFlags = #{}, ChildSpecs = [#{id => ch3, start => {ch3, start_link, []}, shutdown => brutal_kill}], {ok, {SupFlags, ChildSpecs}}.</code> <p>The supervisor then starts all its child processes according to the child specifications in the start specification. In this case there is one child process, <c>ch3</c>.</p> <p><c>supervisor:start_link</c> is synchronous. It does not return until all child processes have been started.</p> </section> <section> <title>Adding a Child Process</title> <p>In addition to the static supervision tree, dynamic child processes can be added to an existing supervisor with the following call:</p> <code type="none"> supervisor:start_child(Sup, ChildSpec)</code> <p><c>Sup</c> is the pid, or name, of the supervisor. <c>ChildSpec</c> is a <seealso marker="#spec">child specification</seealso>.</p> <p>Child processes added using <c>start_child/2</c> behave in the same way as the other child processes, with the an important exception: if a supervisor dies and is recreated, then all child processes that were dynamically added to the supervisor are lost.</p> </section> <section> <title>Stopping a Child Process</title> <p>Any child process, static or dynamic, can be stopped in accordance with the shutdown specification:</p> <code type="none"> supervisor:terminate_child(Sup, Id)</code> <p>The child specification for a stopped child process is deleted with the following call:</p> <code type="none"> supervisor:delete_child(Sup, Id)</code> <p><c>Sup</c> is the pid, or name, of the supervisor. <c>Id</c> is the value associated with the <c>id</c> key in the <seealso marker="#spec">child specification</seealso>.</p> <p>As with dynamically added child processes, the effects of deleting a static child process is lost if the supervisor itself restarts.</p> </section> <section> <marker id="simple"/> <title>Simplified one_for_one Supervisors</title> <p>A supervisor with restart strategy <c>simple_one_for_one</c> is a simplified <c>one_for_one</c> supervisor, where all child processes are dynamically added instances of the same process.</p> <p>The following is an example of a callback module for a <c>simple_one_for_one</c> supervisor:</p> <code type="none"> -module(simple_sup). -behaviour(supervisor). -export([start_link/0]). -export([init/1]). start_link() -> supervisor:start_link(simple_sup, []). init(_Args) -> SupFlags = #{strategy => simple_one_for_one, intensity => 0, period => 1}, ChildSpecs = [#{id => call, start => {call, start_link, []}, shutdown => brutal_kill}], {ok, {SupFlags, ChildSpecs}}.</code> <p>When started, the supervisor does not start any child processes. Instead, all child processes are added dynamically by calling:</p> <code type="none"> supervisor:start_child(Sup, List)</code> <p><c>Sup</c> is the pid, or name, of the supervisor. <c>List</c> is an arbitrary list of terms, which are added to the list of arguments specified in the child specification. If the start function is specified as <c>{M, F, A}</c>, the child process is started by calling <c>apply(M, F, A++List)</c>.</p> <p>For example, adding a child to <c>simple_sup</c> above:</p> <code type="none"> supervisor:start_child(Pid, [id1])</code> <p>The result is that the child process is started by calling <c>apply(call, start_link, []++[id1])</c>, or actually:</p> <code type="none"> call:start_link(id1)</code> <p>A child under a <c>simple_one_for_one</c> supervisor can be terminated with the following:</p> <code type="none"> supervisor:terminate_child(Sup, Pid)</code> <p><c>Sup</c> is the pid, or name, of the supervisor and <c>Pid</c> is the pid of the child.</p> <p>Because a <c>simple_one_for_one</c> supervisor can have many children, it shuts them all down asynchronously. This means that the children will do their cleanup in parallel and therefore the order in which they are stopped is not defined.</p> </section> <section> <title>Stopping</title> <p>Since the supervisor is part of a supervision tree, it is automatically terminated by its supervisor. When asked to shut down, it terminates all child processes in reversed start order according to the respective shutdown specifications, and then terminates itself.</p> </section> </chapter>