An important feature of the Erlang programming language is the ability to change module code in runtime, code replacement, as described in the Erlang Reference Manual.
Based on this feature, the OTP application SASL provides a framework for upgrading and downgrading between different versions of an entire release in runtime. This is called release handling.
The framework consists of:
The minimal system based on Erlang/OTP, enabling release handling, thus consists of the Kernel, STDLIB, and SASL applications.
Step 1) A release is created as described in
Step 2) The release is transferred to and installed at target environment. For information of how to install the first target system, see System Principles.
Step 3) Modifications, for example, error corrections, are made to the code in the development environment.
Step 4) At some point, it is time to make a new version
of release. The relevant
Step 5) For each modified application, an
Step 6) Based on the
Step 7) A new release package is made and transferred to the target system.
Step 8) The new release package is unpacked using the release handler.
Step 9) The new version of the release is installed,
also using the release handler. This is done by evaluating the
instructions in
Complicated or circular dependencies can make it difficult or even impossible to decide in which order things must be done without risking runtime errors during an upgrade or downgrade. Dependencies can be:
During release handling, non-affected processes continue normal execution. This can lead to time-outs or other problems. For example, new processes created in the time window between suspending processes using a certain module, and loading a new version of this module, can execute old code.
It is thus recommended that code is changed in as small steps as possible, and always kept backwards compatible.
For release handling to work properly, the runtime system must
have knowledge about which release it is running. It
must also be able to change (in runtime) which boot script and
system configuration file to use if the system is
rebooted, for example, by
For system reboots to work properly, it is also required that
the system is started with heartbeat monitoring, see the
Other requirements:
The boot script included in a release package must be
generated from the same
Information about applications is fetched from the script when an upgrade or downgrade is performed.
The system must be configured using only one system
configuration file, called
If found, this file is automatically included when a release package is created.
All versions of a release, except the first one, must
contain a
If found, this file is automatically included when a release package is created.
If the system consists of several Erlang nodes, each node can use
its own version of the release. The release handler is a locally
registered process and must be called at each node where an
upgrade or downgrade is required. A release handling
instruction,
OTP supports a set of release handling instructions
that are used when creating
Some of the most frequently used instructions are described in
this section. The complete list of instructions is included in the
First, some definitions:
For a process implemented using an OTP behaviour, the behaviour module is the residence module for that process. The callback module is a functional module.
If a simple extension has been made to a functional module, it is sufficient to load the new version of the module into the system, and remove the old version. This is called simple code replacement and for this the following instruction is used:
{load_module, Module}
If a more complex change has been made, for example, a change
to the format of the internal state of a
This is called synchronized code replacement and for this the following instructions are used:
{update, Module, {advanced, Extra}}
{update, Module, supervisor}
When a module is to be updated, the release handler finds which processes that are using the module by traversing the supervision tree of each running application and checking all the child specifications:
{Id, StartFunc, Restart, Shutdown, Type, Modules}
A process uses a module if the name is listed in
If
The release handler suspends, asks for code change, and
resumes processes by calling the functions
If a new module is introduced, the following instruction is used:
{add_module, Module}
The instruction loads the module and is necessary when running Erlang in embedded mode. It is not strictly required when running Erlang in interactive (default) mode, since the code server then automatically searches for and loads unloaded modules.
The opposite of
{delete_module, Module}
Any process, in any application, with
The following is the instruction for adding an application:
{add_application, Application}
Adding an application means that the modules defined by
the
The following is the instruction for removing an application:
{remove_application, Application}
Removing an application means that the application is stopped,
the modules are unloaded using a number of
The following is the instruction for restarting an application:
{restart_application, Application}
Restarting an application means that the application is stopped
and then started again similar to using the instructions
To call an arbitrary function from the release handler, the following instruction is used:
{apply, {M, F, A}}
The release handler evalutes
This instruction is used when changing to a new emulator
version, or when any of the core applications Kernel,
STDLIB, or SASL is upgraded. If a system reboot
is needed for another reason, the
This instruction requires that the system is started with
heartbeat monitoring, see the
The
When the release handler encounters the instruction, it first
generates a temporary boot file, which starts the new versions
of the emulator and the core applications, and the old version
of all other applications. Then it shuts down
the current emulator by calling
This mechanism causes the new versions of the emulator and core applications to run with the old version of other applications during startup. Thus, take extra care to avoid incompatibility. Incompatible changes in the core applications can in some situations be necessary. If possible, such changes are preceded by deprecation over two major releases before the actual change. To ensure the application is not crashed by an incompatible change, always remove any call to deprecated functions as soon as possible.
An info report is written when the upgrade is completed. To
programmatically find out if the upgrade is complete,
call
The new release version must be made permanent when the new emulator is operational. Otherwise, the old version will be used if there is a new system reboot.
On UNIX, the release handler tells the
This instruction is not related to upgrades of ERTS or any of the core applications. It can be used by any application to force a restart of the emulator after all upgrade instructions are executed.
A relup script can only have one
When the release handler encounters the instruction, it shuts
down the emulator by calling
To define how to upgrade/downgrade between the current version
and previous versions of an application, an
application upgrade file, or in short an
{Vsn,
[{UpFromVsn1, InstructionsU1},
...,
{UpFromVsnK, InstructionsUK}],
[{DownToVsn1, InstructionsD1},
...,
{DownToVsnK, InstructionsDK}]}.
For information about the syntax and contents of the
Example: Consider the release
-module(ch3).
-behaviour(gen_server).
-export([start_link/0]).
-export([alloc/0, free/1]).
-export([available/0]).
-export([init/1, handle_call/3, handle_cast/2]).
start_link() ->
gen_server:start_link({local, ch3}, ch3, [], []).
alloc() ->
gen_server:call(ch3, alloc).
free(Ch) ->
gen_server:cast(ch3, {free, Ch}).
available() ->
gen_server:call(ch3, available).
init(_Args) ->
{ok, channels()}.
handle_call(alloc, _From, Chs) ->
{Ch, Chs2} = alloc(Chs),
{reply, Ch, Chs2};
handle_call(available, _From, Chs) ->
N = available(Chs),
{reply, N, Chs}.
handle_cast({free, Ch}, Chs) ->
Chs2 = free(Ch, Chs),
{noreply, Chs2}.
A new version of the
{application, ch_app,
[{description, "Channel allocator"},
{vsn, "2"},
{modules, [ch_app, ch_sup, ch3]},
{registered, [ch3]},
{applications, [kernel, stdlib, sasl]},
{mod, {ch_app,[]}}
]}.
To upgrade
{"2",
[{"1", [{load_module, ch3}]}],
[{"1", [{load_module, ch3}]}]
}.
To define how to upgrade/downgrade between the new version and
previous versions of a release, a release upgrade file,
or in short
This file does not need to be created manually, it can be
generated by
If the
For details about the syntax and contents of the release upgrade
file, see the
Example, continued from the previous section: You have a
new version "2" of
{release,
{"ch_rel", "B"},
{erts, "5.3"},
[{kernel, "2.9"},
{stdlib, "1.12"},
{sasl, "1.10"},
{ch_app, "2"}]
}.
Now the
1> systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"]).
ok
This generates a
Both the old and new versions of the
1> systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"],
[{path,["../ch_rel-1",
"../ch_rel-1/lib/ch_app-1/ebin"]}]).
ok
When you have made a new version of a release, a release package can be created with this new version and transferred to the target environment.
To install the new version of the release in runtime,
the release handler is used. This is a process belonging
to the SASL application, which handles unpacking, installation,
and removal of release packages. It is communicated through
the
Assuming there is an operational target system with
installation root directory
First, unpack the release package. The files are then extracted from the package:
release_handler:unpack_release(ReleaseName) => {ok, Vsn}
A directory
An unpacked release can be installed. The release
handler then evaluates the instructions in
release_handler:install_release(Vsn) => {ok, FromVsn, []}
If an error occurs during the installation, the system is rebooted using the old version of the release. If installation succeeds, the system is afterwards using the new version of the release, but if anything happens and the system is rebooted, it starts using the previous version again.
To be made the default version, the newly installed release must be made permanent, which means the previous version becomes old:
release_handler:make_permanent(Vsn) => ok
The system keeps information about which versions are old and
permanent in the files
To downgrade from
release_handler:install_release(FromVsn) => {ok, Vsn, []}
An installed, but not permanent, release can be removed.
Information about the release is then deleted from
release_handler:remove_release(Vsn) => ok
Step 1) Create a target system as described in
System Principles of the first version
[].
Step 2) Start the system as a simple target system. In
reality, it is to be started as an embedded system. However, using
% cd $ROOT % bin/erl -boot $ROOT/releases/A/start -config $ROOT/releases/A/sys ...
Step 3) In another Erlang shell, generate start scripts and
create a release package for the new version
1> systools:make_script("ch_rel-2").
ok
2> systools:make_tar("ch_rel-2").
ok
The new release package now also contains version "2" of
% tar tf ch_rel-2.tar
lib/kernel-2.9/ebin/kernel.app
lib/kernel-2.9/ebin/application.beam
...
lib/stdlib-1.12/ebin/stdlib.app
lib/stdlib-1.12/ebin/beam_lib.beam
...
lib/sasl-1.10/ebin/sasl.app
lib/sasl-1.10/ebin/sasl.beam
...
lib/ch_app-2/ebin/ch_app.app
lib/ch_app-2/ebin/ch_app.beam
lib/ch_app-2/ebin/ch_sup.beam
lib/ch_app-2/ebin/ch3.beam
releases/B/start.boot
releases/B/relup
releases/B/sys.config
releases/B/ch_rel-2.rel
releases/ch_rel-2.rel
Step 4) Copy the release package
Step 5) In the running target system, unpack the release package:
1> release_handler:unpack_release("ch_rel-2").
{ok,"B"}
The new application version
Under
Step 6) Check if the function
2> ch3:available(). ** exception error: undefined function ch3:available/0
Step 7) Install the new release. The instructions in
3> release_handler:install_release("B").
{ok,"A",[]}
4> ch3:available().
3
5> code:which(ch3).
".../lib/ch_app-2/ebin/ch3.beam"
6> code:which(ch_sup).
".../lib/ch_app-1/ebin/ch_sup.beam"
Processes in
Step 8) If the target system is now rebooted, it uses version "A" again. The "B" version must be made permanent, to be used when the system is rebooted.
7> release_handler:make_permanent("B").
ok
When a new version of a release is installed, the application specifications are automatically updated for all loaded applications.
The information about the new application specifications is
fetched from the boot script included in the release package.
Thus, it is important that the boot script is generated from
the same
Specifically, the application configuration parameters are automatically updated according to (in increasing priority order):
This means that parameter values set in the other system
configuration files and values set using
When an installed release is made permanent, the system process
After the installation, the application controller compares the old and new configuration parameters for all running applications and call the callback function:
Module:config_change(Changed, New, Removed)
The function is optional and can be omitted when implementing an application callback module.