An important feature of the Erlang programming language is the ability to change module code in run-time, code replacement, as described in Erlang Reference Manual.
Based on this feature, the OTP application SASL provides a framework for upgrading and downgrading between different versions of an entire release in run-time. This is what we call release handling.
The framework consists of off-line support (
Note that the minimal system based on Erlang/OTP, enabling release handling, thus consists of Kernel, STDLIB and SASL.
A release is created as described in the previous chapter
Modifications, for example error corrections, are made to the code in the development environment.
At some point it is time to make a new version of release.
The relevant
For each modified application, an
Based on the
A new release package is made and transferred to the target system.
The new release package is unpacked using the release handler.
The new version of the release is installed, also using
the release handler. This is done by evaluating
the instructions in
If the installation fails, the system may be rebooted. The old release version is then automatically used.
If the installation succeeds, the new version is made the default version, which should now be used in case of a system reboot.
The next chapter,
Complicated or circular dependencies can make it difficult or even impossible to decide in which order things must be done without risking run-time errors during an upgrade or downgrade. Dependencies may be:
During release handling, non-affected processes continue normal execution. This may lead to timeouts or other problems. For example, new processes created in the time window between suspending processes using a certain module and loading a new version of this module, may execute old code.
It is therefore recommended that code is changed in as small steps as possible, and always kept backwards compatible.
For release handling to work properly, the runtime system needs
to have knowledge about which release it is currently running. It
must also be able to change (in run-time) which boot script and
system configuration file should be used if the system is
rebooted, for example by
For system reboots to work properly, it is also required that
the system is started with heart beat monitoring, see
Other requirements:
The boot script included in a release package must be
generated from the same
Information about applications are fetched from the script when an upgrade or downgrade is performed.
The system must be configured using one and only one system
configuration file, called
If found, this file is automatically included when a release package is created.
All versions of a release, except the first one, must
contain a
If found, this file is automatically included when a release package is created.
If the system consists of several Erlang nodes, each node may use
its own version of the release. The release handler is a locally
registered process and must be called at each node where an
upgrade or downgrade is required. There is a release handling
instruction that can be used to synchronize the release handler
processes at a number of nodes:
OTP supports a set of release handling instructions
that is used when creating
Here, some of the most frequently used instructions are
described. The complete list of instructions is found in
First, some definitions:
The module where a process has its tail-recursive loop function(s). If the tail-recursive loop functions are implemented in several modules, all those modules are residence modules for the process.
A module which is not a residence module for any process.
Note that for a process implemented using an OTP behaviour, the behaviour module is the residence module for that process. The callback module is a functional module.
If a simple extension has been made to a functional module, it is sufficient to simply load the new version of the module into the system, and remove the old version. This is called simple code replacement and for this the following instruction is used:
{load_module, Module}
If a more complex change has been made, for example a change to the format of the internal state of a gen_server, simple code replacement is not sufficient. Instead it is necessary to suspend the processes using the module (to avoid that they try to handle any requests before the code replacement is completed), ask them to transform the internal state format and switch to the new version of the module, remove the old version and last, resume the processes. This is called synchronized code replacement and for this the following instructions are used:
{update, Module, {advanced, Extra}}
{update, Module, supervisor}
The release handler finds the processes using a module to update by traversing the supervision tree of each running application and checking all the child specifications:
{Id, StartFunc, Restart, Shutdown, Type, Modules}
A process is using a module if the name is listed in
If
The release handler suspends, asks for code change, and
resumes processes by calling the functions
If a new module is introduced, the following instruction is used:
{add_module, Module}
The instruction loads the module and is absolutely necessary when running Erlang in embedded mode. It is not strictly required when running Erlang in interactive (default) mode, since the code server automatically searches for and loads unloaded modules.
The opposite of
{delete_module, Module}
Note that any process, in any application, with
Instruction for adding an application:
{add_application, Application}
Adding an application means that the modules defined by
the
Instruction for removing an application:
{remove_application, Application}
Removing an application means that the application is stopped,
the modules are unloaded using a number of
Instruction for removing an application:
{restart_application, Application}
Restarting an application means that the application is stopped
and then started again similar to using the instructions
To call an arbitrary function from the release handler, the following instruction is used:
{apply, {M, F, A}}
The release handler will evalute
This instruction is used when changing to a new emulator
version, or if a system reboot is needed for some other reason.
Requires that the system is started with heart beat
monitoring, see
When the release handler encounters the instruction, it shuts
down the current emulator by calling
On UNIX, the release handler tells the heart program which
command to use to reboot the system. Note that the environment
variable
To define how to upgrade/downgrade between the current version
and previous versions of an application, we create an
application upgrade file, or in short
{Vsn,
[{UpFromVsn1, InstructionsU1},
...,
{UpFromVsnK, InstructionsUK}],
[{DownToVsn1, InstructionsD1},
...,
{DownToVsnK, InstructionsDK}]}.
The syntax and contents of the
In the chapter
Example: Consider the release
(Hint: When trying out the example, make the changes in a copy of the original directory, so that the first versions are still available.)
-module(ch3).
-behaviour(gen_server).
-export([start_link/0]).
-export([alloc/0, free/1]).
-export([available/0]).
-export([init/1, handle_call/3, handle_cast/2]).
start_link() ->
gen_server:start_link({local, ch3}, ch3, [], []).
alloc() ->
gen_server:call(ch3, alloc).
free(Ch) ->
gen_server:cast(ch3, {free, Ch}).
available() ->
gen_server:call(ch3, available).
init(_Args) ->
{ok, channels()}.
handle_call(alloc, _From, Chs) ->
{Ch, Chs2} = alloc(Chs),
{reply, Ch, Chs2};
handle_call(available, _From, Chs) ->
N = available(Chs),
{reply, N, Chs}.
handle_cast({free, Ch}, Chs) ->
Chs2 = free(Ch, Chs),
{noreply, Chs2}.
A new version of the
{application, ch_app,
[{description, "Channel allocator"},
{vsn, "2"},
{modules, [ch_app, ch_sup, ch3]},
{registered, [ch3]},
{applications, [kernel, stdlib, sasl]},
{mod, {ch_app,[]}}
]}.
To upgrade
{"2",
[{"1", [{load_module, ch3}]}],
[{"1", [{load_module, ch3}]}]
}.
To define how to upgrade/downgrade between the new version and
previous versions of a release, we create a release upgrade file, or in short
This file does not need to be created manually, it can be
generated by
If the
The syntax and contents of the release upgrade file are
described in detail in
Example, continued from the previous section. We have a new
version "2" of
{release,
{"ch_rel", "B"},
{erts, "5.3"},
[{kernel, "2.9"},
{stdlib, "1.12"},
{sasl, "1.10"},
{ch_app, "2"}]
}.
Now the
1> systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"]).
ok
This will generate a
Note that both the old and new versions of the
1> systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"],
[{path,["../ch_rel-1",
"../ch_rel-1/lib/ch_app-1/ebin"]}]).
ok
When we have made a new version of a release, a release package can be created with this new version and transferred to the target environment.
To install the new version of the release in run-time,
the release handler is used. This is a process belonging
to the SASL application, that handles unpacking, installation,
and removal of release packages. It is interfaced through
the module
Assuming there is a target system up and running with
installation root directory
The first action is to unpack the release package, the files are then extracted from the package:
release_handler:unpack_release(ReleaseName) => {ok, Vsn}
A directory
An unpacked release can be installed. The release
handler then evaluates the instructions in
release_handler:install_release(Vsn) => {ok, FromVsn, []}
If an error occurs during the installation, the system is rebooted using the old version of the release. If installation succeeds, the system is afterwards using the new version of the release, but should anything happen and the system is rebooted, it would start using the previous version again. To be made the default version, the newly installed release must be made permanent, which means the previous version becomes old:
release_handler:make_permanent(Vsn) => ok
The system keeps information about which versions are old and
permanent in the files
To downgrade from
release_handler:install_release(FromVsn) => {ok, Vsn, []}
An installed, but not permanent, release can be removed.
Information about the release is then deleted from
release_handler:remove_release(Vsn) => ok
Example, continued from the previous sections:
1) Create a target system as described in System Principles of the first version
[].
2) Start the system as a simple target system. Note that in
reality, it should be started as an embedded system. However,
using
% cd $ROOT % bin/erl -boot $ROOT/releases/A/start -config $ROOT/releases/A/sys ...
3) In another Erlang shell, generate start scripts and create a
release package for the new version
1> systools:make_script("ch_rel-2").
ok
2> systools:make_tar("ch_rel-2").
ok
The new release package now contains version "2" of
% tar tf ch_rel-2.tar
lib/kernel-2.9/ebin/kernel.app
lib/kernel-2.9/ebin/application.beam
...
lib/stdlib-1.12/ebin/stdlib.app
lib/stdlib-1.12/ebin/beam_lib.beam
...
lib/sasl-1.10/ebin/sasl.app
lib/sasl-1.10/ebin/sasl.beam
...
lib/ch_app-2/ebin/ch_app.app
lib/ch_app-2/ebin/ch_app.beam
lib/ch_app-2/ebin/ch_sup.beam
lib/ch_app-2/ebin/ch3.beam
releases/B/start.boot
releases/B/relup
releases/B/sys.config
releases/ch_rel-2.rel
4) Copy the release package
5) In the running target system, unpack the release package:
1> release_handler:unpack_release("ch_rel-2").
{ok,"B"}
The new application version
Under
6) Check if the function
2> ch3:available(). ** exception error: undefined function ch3:available/0
7) Install the new release. The instructions in
3> release_handler:install_release("B").
{ok,"A",[]}
4> ch3:available().
3
5> code:which(ch3).
".../lib/ch_app-2/ebin/ch3.beam"
6> code:which(ch_sup).
".../lib/ch_app-1/ebin/ch_sup.beam"
Note that processes in
8) If the target system is now rebooted, it will use version "A" again. The "B" version must be made permanent, in order to be used when the system is rebooted.
7> release_handler:make_permanent("B").
ok
When a new version of a release is installed, the application specifications are automatically updated for all loaded applications.
The information about the new application specifications are
fetched from the boot script included in the release package.
It is therefore important that the boot script is generated from
the same
Specifically, the application configuration parameters are automatically updated according to (in increasing priority order):
This means that parameter values set in the other system
configuration files, as well as values set using
When an installed release is made permanent, the system process
After the installation, the application controller will compare the old and new configuration parameters for all running applications and call the callback function:
Module:config_change(Changed, New, Removed)
The function is optional and may be omitted when implementing an application callback module.