diff options
537 files changed, 29212 insertions, 10984 deletions
diff --git a/.gitignore b/.gitignore index 18a54c21ca..e27b5b12ff 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ # emacs *~ +lib/tools/emacs/*.elc # vim .*.sw[a-z] @@ -381,6 +382,7 @@ JAVADOC-GENERATED /system/doc/installation_guide/INSTALL.xml /system/doc/installation_guide/INSTALL-CROSS.xml /system/doc/installation_guide/INSTALL-WIN32.xml +/system/doc/installation_guide/OTP-PATCH-APPLY.xml /system/doc/installation_guide/MARKDOWN.xml # test_server diff --git a/HOWTO/INSTALL.md b/HOWTO/INSTALL.md index 53b1b8cd8a..7efea77d4c 100644 --- a/HOWTO/INSTALL.md +++ b/HOWTO/INSTALL.md @@ -417,22 +417,33 @@ important when building Erlang/OTP. By default the VM will refuse to build if native atomic memory operations are not available. Erlang/OTP itself provides implementations of native atomic memory operations -that can be used when compiling with a `gcc` compatible compiler on 32-bit -and 64-bit x86, 32-bit and 64-bit SPARC V9, and 32-bit PowerPC. When compiling -with a `gcc` compatible compiler on other architectures, the VM may be able to -make use of native atomic operations using the `__sync_*` primitives, but this -should only be used as a last resort since this wont give you optimal -performance. When compiling on Windows using a MicroSoft Visual C++ compiler -native atomic memory operations are provided by Windows APIs. - -You are recommended to use the native atomic implementation provided by -Erlang/OTP, or the API provided by Windows. If these do not provide native -atomics on your platform, you are recommended to build and install -[libatomic_ops][] before building Erlang/OTP. The `libatomic_ops` library -provides native atomic memory operations for a variety of platforms and -compilers. When building Erlang/OTP you need to inform the build system of -where the `libatomic_ops` library is installed using the -`--with-libatomic_ops=PATH` configure switch. +that can be used when compiling with a `gcc` compatible compiler for 32/64-bit +x86, 32/64-bit SPARC V9, 32-bit PowerPC, or 32-bit Tile. When compiling with +a `gcc` compatible compiler for other architectures, the VM may be able to make +use of native atomic operations using the `__atomic_*` builtins (may be +available when using a `gcc` of at least version 4.7) and/or using the +`__sync_*` builtins (may be available when using a `gcc` of at least version +4.1). If only the `gcc`'s `__sync_*` builtins are available, the performance +will suffer. Such a configuration should only be used as a last resort. When +compiling on Windows using a MicroSoft Visual C++ compiler native atomic +memory operations are provided by Windows APIs. + +Native atomic implementation in the order preferred: +1. The implementation provided by Erlang/OTP. +2. The API provided by Windows. +3. The implementation based on the `gcc` `__atomic_*` builtins. +4. If none of the above are available for your architecture/compiler, you + are recommended to build and install [libatomic_ops][] before building + Erlang/OTP. The `libatomic_ops` library provides native atomic memory + operations for a variety of architectures and compilers. When building + Erlang/OTP you need to inform the build system of where the + `libatomic_ops` library is installed using the + `--with-libatomic_ops=PATH` `configure` switch. +5. As a last resort, the implementation solely based on the `gcc` + `__sync_*` builtins. This will however cause lots of expensive and + unnecessary memory barrier instructions to be issued. That is, + performance will suffer. The `configure` script will warn at the end + of its execution if it cannot find any other alternative than this. ### Building ### @@ -849,7 +860,7 @@ Copyright and License %CopyrightBegin% -Copyright Ericsson AB 1998-2014. All Rights Reserved. +Copyright Ericsson AB 1998-2015. All Rights Reserved. The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in diff --git a/HOWTO/OTP-PATCH-APPLY.md b/HOWTO/OTP-PATCH-APPLY.md new file mode 100644 index 0000000000..2aa31629ef --- /dev/null +++ b/HOWTO/OTP-PATCH-APPLY.md @@ -0,0 +1,144 @@ +Patching OTP Applications +========================= + +Introduction +------------ + +This document describes the process of patching an existing OTP +installation with one or more Erlang/OTP applications of newer versions +than already installed. The tool `otp_patch_apply` is available for this +specific purpose. It resides in the top directory of the Erlang/OTP +source tree. + +The `otp_patch_apply` tool utilizes the [runtime_dependencies][] tag in +the [application resource file][]. This information is used to determine +if the patch can be installed in the given Erlang/OTP installation +directory. + +Read more about the [version handling][] introduced in Erlang/OTP release +17, which also describes how to determine if an installation includes one +or more patched applications. + +If you want to apply patches of multiple OTP applications that resides +in different OTP versions, you have to apply these patches in multiple +steps. It is only possible to apply multiple OTP applications from the +same OTP version at once. + +Prerequisites +------------- + +It's assumed that the reader is familiar with +[building and installing Erlang/OTP][]. To be able to patch an +application, the following must exist: + +* An Erlang/OTP installation. + +* An Erlang/OTP source tree containing the updated applications that + you want to patch into the existing Erlang/OTP installation. + +Using otp\_patch\_apply +----------------------- + +> *WARNING*: Patching applications is a one-way process. +> Create a backup of your OTP installation directory before +> proceeding. + +First of all, build the OTP source tree at `$ERL_TOP` containing +the updated applications. + +> *NOTE*: Before applying a patch you need to do a *full* build +> of OTP in the source directory. + +If you are building in `git` you first need to generate the +`configure` scripts: + + $ ./otp_build autoconf + +Configure and build all applications in OTP: + + $ configure + $ make + +or + + $ ./otp_build configure + $ ./otp_build boot -a + +If you have installed documentation in the OTP installation, also +build the documentation: + + $ make docs + +After the successful build it's time to patch. The source tree directory, +the directory of the installation and the applications to patch are given +as arguments to `otp_patch_apply`. The dependencies of each application +are validated against the applications in the installation and the other +applications given as arguments. If a dependency error is detected, the +script will be aborted. + +The `otp_patch_apply` syntax: + + $ otp_patch_apply -s <Dir> -i <Dir> [-l <Dir>] [-c] [-f] [-h] \ + [-n] [-v] <App1> [... <AppN>] + + -s <Dir> -- OTP source directory that contains build results. + -i <Dir> -- OTP installation directory to patch. + -l <Dir> -- Alternative OTP source library directory path(s) + containing build results of OTP applications. + Multiple paths should be colon separated. + -c -- Cleanup (remove) old versions of applications + patched in the installation. + -f -- Force patch of application(s) even though + dependencies are not fulfilled (should only be + considered in a test environment). + -h -- Print help then exit. + -n -- Do not install documentation. + -v -- Print version then exit. + <AppX> -- Application to patch. + + Environment Variable: + ERL_LIBS -- Alternative OTP source library directory path(s) + containing build results of OTP applications. + Multiple paths should be colon separated. + +> *NOTE*: The complete build environment is required while running +> `otp_patch_apply`. + +> *NOTE*: All source directories identified by `-s` and `-l` should +> contain build results of OTP applications. + +For example, if the user wants to install patched versions of `mnesia` +and `ssl` built in `/home/me/git/otp` into the OTP installation +located in `/opt/erlang/my_otp` type + + $ otp_patch_apply -s /home/me/git/otp -i /opt/erlang/my_otp \ + mnesia ssl + +> *NOTE*: If the list of applications contains core applications, +> i.e `erts`, `kernel`, `stdlib` or `sasl`, the `Install` script in +> the patched Erlang/OTP installation must be rerun. + +The patched applications are appended to the list of installed +applications. Take a look at +`<InstallDir>/releases/OTP-REL/installed_application_versions`. + +Sanity check +------------ + +The application dependencies can be checked using the Erlang shell. +Application dependencies are verified among installed applications by +`otp_patch_apply`, but these are not necessarily those actually loaded. +By calling `system_information:sanity_check()` one can validate +dependencies among applications actually loaded. + + 1> system_information:sanity_check(). + ok + +Please take a look at the reference of [sanity_check()][] for more +information. + +[application resource file]: kernel:app +[runtime_dependencies]: kernel:app#runtime_dependencies +[building and installing Erlang/OTP]: INSTALL.md +[version handling]: ../system_principles/versions +[sanity_check()]: runtime_tools:system_information#sanity_check-0 diff --git a/OTP_VERSION b/OTP_VERSION index 1fbc0d2431..9d7cce5373 100644 --- a/OTP_VERSION +++ b/OTP_VERSION @@ -1 +1 @@ -18.0-rc0 +18.0-rc1 diff --git a/bootstrap/lib/compiler/ebin/beam_asm.beam b/bootstrap/lib/compiler/ebin/beam_asm.beam Binary files differindex bcbf1f6c2a..eea00c7267 100644 --- a/bootstrap/lib/compiler/ebin/beam_asm.beam +++ b/bootstrap/lib/compiler/ebin/beam_asm.beam diff --git a/bootstrap/lib/compiler/ebin/beam_block.beam b/bootstrap/lib/compiler/ebin/beam_block.beam Binary files differindex 8790340543..74969880e0 100644 --- a/bootstrap/lib/compiler/ebin/beam_block.beam +++ b/bootstrap/lib/compiler/ebin/beam_block.beam diff --git a/bootstrap/lib/compiler/ebin/beam_bool.beam b/bootstrap/lib/compiler/ebin/beam_bool.beam Binary files differindex 6454719448..0fa7612114 100644 --- a/bootstrap/lib/compiler/ebin/beam_bool.beam +++ b/bootstrap/lib/compiler/ebin/beam_bool.beam diff --git a/bootstrap/lib/compiler/ebin/beam_dead.beam b/bootstrap/lib/compiler/ebin/beam_dead.beam Binary files differindex 93188416d6..dfeec25a9c 100644 --- a/bootstrap/lib/compiler/ebin/beam_dead.beam +++ b/bootstrap/lib/compiler/ebin/beam_dead.beam diff --git a/bootstrap/lib/compiler/ebin/beam_peep.beam b/bootstrap/lib/compiler/ebin/beam_peep.beam Binary files differindex 214fa56911..7a7856e247 100644 --- a/bootstrap/lib/compiler/ebin/beam_peep.beam +++ b/bootstrap/lib/compiler/ebin/beam_peep.beam diff --git a/bootstrap/lib/compiler/ebin/beam_receive.beam b/bootstrap/lib/compiler/ebin/beam_receive.beam Binary files differindex aec35c43eb..78c9eb1c39 100644 --- a/bootstrap/lib/compiler/ebin/beam_receive.beam +++ b/bootstrap/lib/compiler/ebin/beam_receive.beam diff --git a/bootstrap/lib/compiler/ebin/cerl_inline.beam b/bootstrap/lib/compiler/ebin/cerl_inline.beam Binary files differindex 24a36e139d..a089535e59 100644 --- a/bootstrap/lib/compiler/ebin/cerl_inline.beam +++ b/bootstrap/lib/compiler/ebin/cerl_inline.beam diff --git a/bootstrap/lib/compiler/ebin/cerl_trees.beam b/bootstrap/lib/compiler/ebin/cerl_trees.beam Binary files differindex f36cc31458..45f76f5c89 100644 --- a/bootstrap/lib/compiler/ebin/cerl_trees.beam +++ b/bootstrap/lib/compiler/ebin/cerl_trees.beam diff --git a/bootstrap/lib/compiler/ebin/compile.beam b/bootstrap/lib/compiler/ebin/compile.beam Binary files differindex 5df7d4aa86..48a9d55bf0 100644 --- a/bootstrap/lib/compiler/ebin/compile.beam +++ b/bootstrap/lib/compiler/ebin/compile.beam diff --git a/bootstrap/lib/compiler/ebin/compiler.app b/bootstrap/lib/compiler/ebin/compiler.app index d1caa3a348..26e357a204 100644 --- a/bootstrap/lib/compiler/ebin/compiler.app +++ b/bootstrap/lib/compiler/ebin/compiler.app @@ -69,5 +69,5 @@ {registered, []}, {applications, [kernel, stdlib]}, {env, []}, - {runtime_dependencies, ["stdlib-2.0","kernel-3.0","hipe-3.10.3","erts-6.0", + {runtime_dependencies, ["stdlib-2.0","kernel-3.0","hipe-3.10.3","erts-7.0", "crypto-3.3"]}]}. diff --git a/bootstrap/lib/compiler/ebin/core_parse.beam b/bootstrap/lib/compiler/ebin/core_parse.beam Binary files differindex 6ea05bf28f..f0e746cbd0 100644 --- a/bootstrap/lib/compiler/ebin/core_parse.beam +++ b/bootstrap/lib/compiler/ebin/core_parse.beam diff --git a/bootstrap/lib/compiler/ebin/sys_core_fold.beam b/bootstrap/lib/compiler/ebin/sys_core_fold.beam Binary files differindex 5b485a6a94..25930bc5f1 100644 --- a/bootstrap/lib/compiler/ebin/sys_core_fold.beam +++ b/bootstrap/lib/compiler/ebin/sys_core_fold.beam diff --git a/bootstrap/lib/compiler/ebin/sys_pre_expand.beam b/bootstrap/lib/compiler/ebin/sys_pre_expand.beam Binary files differindex 0ad8bc7ae4..f727e5c1b9 100644 --- a/bootstrap/lib/compiler/ebin/sys_pre_expand.beam +++ b/bootstrap/lib/compiler/ebin/sys_pre_expand.beam diff --git a/bootstrap/lib/compiler/ebin/v3_codegen.beam b/bootstrap/lib/compiler/ebin/v3_codegen.beam Binary files differindex 2f39db14c7..e01c356f39 100644 --- a/bootstrap/lib/compiler/ebin/v3_codegen.beam +++ b/bootstrap/lib/compiler/ebin/v3_codegen.beam diff --git a/bootstrap/lib/compiler/ebin/v3_core.beam b/bootstrap/lib/compiler/ebin/v3_core.beam Binary files differindex dff562a748..492b73e543 100644 --- a/bootstrap/lib/compiler/ebin/v3_core.beam +++ b/bootstrap/lib/compiler/ebin/v3_core.beam diff --git a/bootstrap/lib/compiler/ebin/v3_kernel.beam b/bootstrap/lib/compiler/ebin/v3_kernel.beam Binary files differindex 20612f3407..5f42d5acb6 100644 --- a/bootstrap/lib/compiler/ebin/v3_kernel.beam +++ b/bootstrap/lib/compiler/ebin/v3_kernel.beam diff --git a/bootstrap/lib/compiler/ebin/v3_life.beam b/bootstrap/lib/compiler/ebin/v3_life.beam Binary files differindex 0eb0cb6caf..3c977ceb0e 100644 --- a/bootstrap/lib/compiler/ebin/v3_life.beam +++ b/bootstrap/lib/compiler/ebin/v3_life.beam diff --git a/bootstrap/lib/kernel/ebin/application.beam b/bootstrap/lib/kernel/ebin/application.beam Binary files differindex b8f7f156fe..b0b9e3d12c 100644 --- a/bootstrap/lib/kernel/ebin/application.beam +++ b/bootstrap/lib/kernel/ebin/application.beam diff --git a/bootstrap/lib/kernel/ebin/application_master.beam b/bootstrap/lib/kernel/ebin/application_master.beam Binary files differindex 4ff90f7eff..d8018f0a40 100644 --- a/bootstrap/lib/kernel/ebin/application_master.beam +++ b/bootstrap/lib/kernel/ebin/application_master.beam diff --git a/bootstrap/lib/kernel/ebin/auth.beam b/bootstrap/lib/kernel/ebin/auth.beam Binary files differindex bf5dea6d84..74dde4fd1c 100644 --- a/bootstrap/lib/kernel/ebin/auth.beam +++ b/bootstrap/lib/kernel/ebin/auth.beam diff --git a/bootstrap/lib/kernel/ebin/code.beam b/bootstrap/lib/kernel/ebin/code.beam Binary files differindex 5428f2cf0c..93bf7ad032 100644 --- a/bootstrap/lib/kernel/ebin/code.beam +++ b/bootstrap/lib/kernel/ebin/code.beam diff --git a/bootstrap/lib/kernel/ebin/code_server.beam b/bootstrap/lib/kernel/ebin/code_server.beam Binary files differindex aa75ae9bd1..7b0517c500 100644 --- a/bootstrap/lib/kernel/ebin/code_server.beam +++ b/bootstrap/lib/kernel/ebin/code_server.beam diff --git a/bootstrap/lib/kernel/ebin/disk_log.beam b/bootstrap/lib/kernel/ebin/disk_log.beam Binary files differindex bc73a8e71c..3ccdfaea05 100644 --- a/bootstrap/lib/kernel/ebin/disk_log.beam +++ b/bootstrap/lib/kernel/ebin/disk_log.beam diff --git a/bootstrap/lib/kernel/ebin/dist_ac.beam b/bootstrap/lib/kernel/ebin/dist_ac.beam Binary files differindex 25b155418c..7027118b3f 100644 --- a/bootstrap/lib/kernel/ebin/dist_ac.beam +++ b/bootstrap/lib/kernel/ebin/dist_ac.beam diff --git a/bootstrap/lib/kernel/ebin/dist_util.beam b/bootstrap/lib/kernel/ebin/dist_util.beam Binary files differindex 3be91b2b06..c92373c68e 100644 --- a/bootstrap/lib/kernel/ebin/dist_util.beam +++ b/bootstrap/lib/kernel/ebin/dist_util.beam diff --git a/bootstrap/lib/kernel/ebin/erl_boot_server.beam b/bootstrap/lib/kernel/ebin/erl_boot_server.beam Binary files differindex 6e56a1061d..91899fe231 100644 --- a/bootstrap/lib/kernel/ebin/erl_boot_server.beam +++ b/bootstrap/lib/kernel/ebin/erl_boot_server.beam diff --git a/bootstrap/lib/kernel/ebin/erts_debug.beam b/bootstrap/lib/kernel/ebin/erts_debug.beam Binary files differindex 17320342bc..1acdba0178 100644 --- a/bootstrap/lib/kernel/ebin/erts_debug.beam +++ b/bootstrap/lib/kernel/ebin/erts_debug.beam diff --git a/bootstrap/lib/kernel/ebin/file.beam b/bootstrap/lib/kernel/ebin/file.beam Binary files differindex 18ba4449b8..32a3a27993 100644 --- a/bootstrap/lib/kernel/ebin/file.beam +++ b/bootstrap/lib/kernel/ebin/file.beam diff --git a/bootstrap/lib/kernel/ebin/file_io_server.beam b/bootstrap/lib/kernel/ebin/file_io_server.beam Binary files differindex f85f130258..be820676ed 100644 --- a/bootstrap/lib/kernel/ebin/file_io_server.beam +++ b/bootstrap/lib/kernel/ebin/file_io_server.beam diff --git a/bootstrap/lib/kernel/ebin/global.beam b/bootstrap/lib/kernel/ebin/global.beam Binary files differindex 22bbaa0a9b..2fb529adb4 100644 --- a/bootstrap/lib/kernel/ebin/global.beam +++ b/bootstrap/lib/kernel/ebin/global.beam diff --git a/bootstrap/lib/kernel/ebin/global_group.beam b/bootstrap/lib/kernel/ebin/global_group.beam Binary files differindex d11f60015a..85306ef123 100644 --- a/bootstrap/lib/kernel/ebin/global_group.beam +++ b/bootstrap/lib/kernel/ebin/global_group.beam diff --git a/bootstrap/lib/kernel/ebin/group.beam b/bootstrap/lib/kernel/ebin/group.beam Binary files differindex 5ec6322df2..0f72880f49 100644 --- a/bootstrap/lib/kernel/ebin/group.beam +++ b/bootstrap/lib/kernel/ebin/group.beam diff --git a/bootstrap/lib/kernel/ebin/inet_db.beam b/bootstrap/lib/kernel/ebin/inet_db.beam Binary files differindex dac744c265..bb2a6cb5e0 100644 --- a/bootstrap/lib/kernel/ebin/inet_db.beam +++ b/bootstrap/lib/kernel/ebin/inet_db.beam diff --git a/bootstrap/lib/kernel/ebin/inet_dns.beam b/bootstrap/lib/kernel/ebin/inet_dns.beam Binary files differindex 01d3b08362..0c5b6c73e1 100644 --- a/bootstrap/lib/kernel/ebin/inet_dns.beam +++ b/bootstrap/lib/kernel/ebin/inet_dns.beam diff --git a/bootstrap/lib/kernel/ebin/inet_res.beam b/bootstrap/lib/kernel/ebin/inet_res.beam Binary files differindex 8cb11656f6..f35242f49a 100644 --- a/bootstrap/lib/kernel/ebin/inet_res.beam +++ b/bootstrap/lib/kernel/ebin/inet_res.beam diff --git a/bootstrap/lib/kernel/ebin/kernel.app b/bootstrap/lib/kernel/ebin/kernel.app index 3388f28e91..e3084fde09 100644 --- a/bootstrap/lib/kernel/ebin/kernel.app +++ b/bootstrap/lib/kernel/ebin/kernel.app @@ -115,6 +115,6 @@ {applications, []}, {env, [{error_logger, tty}]}, {mod, {kernel, []}}, - {runtime_dependencies, ["erts-6.1.2", "stdlib-2.0", "sasl-2.4"]} + {runtime_dependencies, ["erts-7.0", "stdlib-2.0", "sasl-2.4"]} ] }. diff --git a/bootstrap/lib/kernel/ebin/os.beam b/bootstrap/lib/kernel/ebin/os.beam Binary files differindex d16dbbac73..9eaf7dfe5d 100644 --- a/bootstrap/lib/kernel/ebin/os.beam +++ b/bootstrap/lib/kernel/ebin/os.beam diff --git a/bootstrap/lib/kernel/ebin/pg2.beam b/bootstrap/lib/kernel/ebin/pg2.beam Binary files differindex 6abf358157..63bedc72d7 100644 --- a/bootstrap/lib/kernel/ebin/pg2.beam +++ b/bootstrap/lib/kernel/ebin/pg2.beam diff --git a/bootstrap/lib/stdlib/ebin/beam_lib.beam b/bootstrap/lib/stdlib/ebin/beam_lib.beam Binary files differindex b0f2c784da..463e571164 100644 --- a/bootstrap/lib/stdlib/ebin/beam_lib.beam +++ b/bootstrap/lib/stdlib/ebin/beam_lib.beam diff --git a/bootstrap/lib/stdlib/ebin/c.beam b/bootstrap/lib/stdlib/ebin/c.beam Binary files differindex 3fed93c41a..a08df44eb7 100644 --- a/bootstrap/lib/stdlib/ebin/c.beam +++ b/bootstrap/lib/stdlib/ebin/c.beam diff --git a/bootstrap/lib/stdlib/ebin/dets.beam b/bootstrap/lib/stdlib/ebin/dets.beam Binary files differindex f5fac6e8a0..04dde86c0d 100644 --- a/bootstrap/lib/stdlib/ebin/dets.beam +++ b/bootstrap/lib/stdlib/ebin/dets.beam diff --git a/bootstrap/lib/stdlib/ebin/dets_utils.beam b/bootstrap/lib/stdlib/ebin/dets_utils.beam Binary files differindex b4b243cd39..f2d1d86a60 100644 --- a/bootstrap/lib/stdlib/ebin/dets_utils.beam +++ b/bootstrap/lib/stdlib/ebin/dets_utils.beam diff --git a/bootstrap/lib/stdlib/ebin/dets_v9.beam b/bootstrap/lib/stdlib/ebin/dets_v9.beam Binary files differindex 360c95ee8f..9d543abc94 100644 --- a/bootstrap/lib/stdlib/ebin/dets_v9.beam +++ b/bootstrap/lib/stdlib/ebin/dets_v9.beam diff --git a/bootstrap/lib/stdlib/ebin/digraph_utils.beam b/bootstrap/lib/stdlib/ebin/digraph_utils.beam Binary files differindex fdc6741439..458c6d2895 100644 --- a/bootstrap/lib/stdlib/ebin/digraph_utils.beam +++ b/bootstrap/lib/stdlib/ebin/digraph_utils.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_lint.beam b/bootstrap/lib/stdlib/ebin/erl_lint.beam Binary files differindex 34af4c99f4..606666808d 100644 --- a/bootstrap/lib/stdlib/ebin/erl_lint.beam +++ b/bootstrap/lib/stdlib/ebin/erl_lint.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_tar.beam b/bootstrap/lib/stdlib/ebin/erl_tar.beam Binary files differindex 1df8ab89f8..e525b3738b 100644 --- a/bootstrap/lib/stdlib/ebin/erl_tar.beam +++ b/bootstrap/lib/stdlib/ebin/erl_tar.beam diff --git a/bootstrap/lib/stdlib/ebin/escript.beam b/bootstrap/lib/stdlib/ebin/escript.beam Binary files differindex 53f963ce79..83a6a5cdb2 100644 --- a/bootstrap/lib/stdlib/ebin/escript.beam +++ b/bootstrap/lib/stdlib/ebin/escript.beam diff --git a/bootstrap/lib/stdlib/ebin/ets.beam b/bootstrap/lib/stdlib/ebin/ets.beam Binary files differindex 6325142290..6f2085cf22 100644 --- a/bootstrap/lib/stdlib/ebin/ets.beam +++ b/bootstrap/lib/stdlib/ebin/ets.beam diff --git a/bootstrap/lib/stdlib/ebin/file_sorter.beam b/bootstrap/lib/stdlib/ebin/file_sorter.beam Binary files differindex 62062c6a56..a2a82fb77d 100644 --- a/bootstrap/lib/stdlib/ebin/file_sorter.beam +++ b/bootstrap/lib/stdlib/ebin/file_sorter.beam diff --git a/bootstrap/lib/stdlib/ebin/filename.beam b/bootstrap/lib/stdlib/ebin/filename.beam Binary files differindex 31cc4a7309..59d18f9a86 100644 --- a/bootstrap/lib/stdlib/ebin/filename.beam +++ b/bootstrap/lib/stdlib/ebin/filename.beam diff --git a/bootstrap/lib/stdlib/ebin/gen.beam b/bootstrap/lib/stdlib/ebin/gen.beam Binary files differindex 61259fd267..ccaabd8087 100644 --- a/bootstrap/lib/stdlib/ebin/gen.beam +++ b/bootstrap/lib/stdlib/ebin/gen.beam diff --git a/bootstrap/lib/stdlib/ebin/gen_event.beam b/bootstrap/lib/stdlib/ebin/gen_event.beam Binary files differindex d46e225182..d7d25774f4 100644 --- a/bootstrap/lib/stdlib/ebin/gen_event.beam +++ b/bootstrap/lib/stdlib/ebin/gen_event.beam diff --git a/bootstrap/lib/stdlib/ebin/io_lib.beam b/bootstrap/lib/stdlib/ebin/io_lib.beam Binary files differindex af31d1b283..67d2f9570a 100644 --- a/bootstrap/lib/stdlib/ebin/io_lib.beam +++ b/bootstrap/lib/stdlib/ebin/io_lib.beam diff --git a/bootstrap/lib/stdlib/ebin/io_lib_format.beam b/bootstrap/lib/stdlib/ebin/io_lib_format.beam Binary files differindex b284a99f70..f4483c908e 100644 --- a/bootstrap/lib/stdlib/ebin/io_lib_format.beam +++ b/bootstrap/lib/stdlib/ebin/io_lib_format.beam diff --git a/bootstrap/lib/stdlib/ebin/io_lib_pretty.beam b/bootstrap/lib/stdlib/ebin/io_lib_pretty.beam Binary files differindex 7265b0e865..eed02e5fa5 100644 --- a/bootstrap/lib/stdlib/ebin/io_lib_pretty.beam +++ b/bootstrap/lib/stdlib/ebin/io_lib_pretty.beam diff --git a/bootstrap/lib/stdlib/ebin/otp_internal.beam b/bootstrap/lib/stdlib/ebin/otp_internal.beam Binary files differindex 9812be8bd9..d144d72394 100644 --- a/bootstrap/lib/stdlib/ebin/otp_internal.beam +++ b/bootstrap/lib/stdlib/ebin/otp_internal.beam diff --git a/bootstrap/lib/stdlib/ebin/qlc.beam b/bootstrap/lib/stdlib/ebin/qlc.beam Binary files differindex abc4d0d33d..5dfd03d3b7 100644 --- a/bootstrap/lib/stdlib/ebin/qlc.beam +++ b/bootstrap/lib/stdlib/ebin/qlc.beam diff --git a/bootstrap/lib/stdlib/ebin/qlc_pt.beam b/bootstrap/lib/stdlib/ebin/qlc_pt.beam Binary files differindex da6056d3c5..e9c359400b 100644 --- a/bootstrap/lib/stdlib/ebin/qlc_pt.beam +++ b/bootstrap/lib/stdlib/ebin/qlc_pt.beam diff --git a/bootstrap/lib/stdlib/ebin/random.beam b/bootstrap/lib/stdlib/ebin/random.beam Binary files differindex ad6e553935..f576b310df 100644 --- a/bootstrap/lib/stdlib/ebin/random.beam +++ b/bootstrap/lib/stdlib/ebin/random.beam diff --git a/bootstrap/lib/stdlib/ebin/re.beam b/bootstrap/lib/stdlib/ebin/re.beam Binary files differindex f30fc894d1..9e140def2c 100644 --- a/bootstrap/lib/stdlib/ebin/re.beam +++ b/bootstrap/lib/stdlib/ebin/re.beam diff --git a/bootstrap/lib/stdlib/ebin/shell.beam b/bootstrap/lib/stdlib/ebin/shell.beam Binary files differindex 37cb1626c2..c0450dab5b 100644 --- a/bootstrap/lib/stdlib/ebin/shell.beam +++ b/bootstrap/lib/stdlib/ebin/shell.beam diff --git a/bootstrap/lib/stdlib/ebin/stdlib.app b/bootstrap/lib/stdlib/ebin/stdlib.app index 295d208e42..08135da5af 100644 --- a/bootstrap/lib/stdlib/ebin/stdlib.app +++ b/bootstrap/lib/stdlib/ebin/stdlib.app @@ -102,7 +102,7 @@ dets]}, {applications, [kernel]}, {env, []}, - {runtime_dependencies, ["sasl-2.4","kernel-3.0.2","erts-6.2","crypto-3.3", + {runtime_dependencies, ["sasl-2.4","kernel-3.0.2","erts-7.0","crypto-3.3", "compiler-5.0"]} ]}. diff --git a/bootstrap/lib/stdlib/ebin/supervisor.beam b/bootstrap/lib/stdlib/ebin/supervisor.beam Binary files differindex 473aac8acc..fcf9868a09 100644 --- a/bootstrap/lib/stdlib/ebin/supervisor.beam +++ b/bootstrap/lib/stdlib/ebin/supervisor.beam diff --git a/bootstrap/lib/stdlib/ebin/timer.beam b/bootstrap/lib/stdlib/ebin/timer.beam Binary files differindex c12b0986df..8f34d08a0b 100644 --- a/bootstrap/lib/stdlib/ebin/timer.beam +++ b/bootstrap/lib/stdlib/ebin/timer.beam diff --git a/bootstrap/lib/stdlib/ebin/unicode.beam b/bootstrap/lib/stdlib/ebin/unicode.beam Binary files differindex 138a1ce28d..16557d5631 100644 --- a/bootstrap/lib/stdlib/ebin/unicode.beam +++ b/bootstrap/lib/stdlib/ebin/unicode.beam diff --git a/bootstrap/lib/stdlib/ebin/zip.beam b/bootstrap/lib/stdlib/ebin/zip.beam Binary files differindex 33402a7c58..e80b6ae0cd 100644 --- a/bootstrap/lib/stdlib/ebin/zip.beam +++ b/bootstrap/lib/stdlib/ebin/zip.beam diff --git a/erts/aclocal.m4 b/erts/aclocal.m4 index 264a1726db..83f735d332 100644 --- a/erts/aclocal.m4 +++ b/erts/aclocal.m4 @@ -1,7 +1,7 @@ dnl dnl %CopyrightBegin% dnl -dnl Copyright Ericsson AB 1998-2013. All Rights Reserved. +dnl Copyright Ericsson AB 1998-2015. All Rights Reserved. dnl dnl The contents of this file are subject to the Erlang Public License, dnl Version 1.1, (the "License"); you may not use this file except in @@ -246,31 +246,31 @@ lbl1: return 1; lbl2: return 2; -],ac_cv_prog_emu_cc=$CC,ac_cv_prog_emu_cc=no) +],ac_cv_prog_emu_cc="$CC",ac_cv_prog_emu_cc=no) -if test $ac_cv_prog_emu_cc = no; then +if test "$ac_cv_prog_emu_cc" = no; then for ac_progname in emu_cc.sh gcc-4.2 gcc; do IFS="${IFS= }"; ac_save_ifs="$IFS"; IFS=":" ac_dummy="$PATH" for ac_dir in $ac_dummy; do test -z "$ac_dir" && ac_dir=. - if test -f $ac_dir/$ac_progname; then - ac_cv_prog_emu_cc=$ac_dir/$ac_progname + if test -f "$ac_dir/$ac_progname"; then + ac_cv_prog_emu_cc="$ac_dir/$ac_progname" break fi done IFS="$ac_save_ifs" - if test $ac_cv_prog_emu_cc != no; then + if test "$ac_cv_prog_emu_cc" != no; then break fi done fi -if test $ac_cv_prog_emu_cc != no; then - save_CC=$CC +if test "$ac_cv_prog_emu_cc" != no; then + save_CC="$CC" save_CFLAGS=$CFLAGS save_CPPFLAGS=$CPPFLAGS - CC=$ac_cv_prog_emu_cc + CC="$ac_cv_prog_emu_cc" CFLAGS="" CPPFLAGS="" AC_TRY_COMPILE([],[ @@ -291,17 +291,17 @@ if test $ac_cv_prog_emu_cc != no; then return 1; lbl2: return 2; - ],ac_cv_prog_emu_cc=$CC,ac_cv_prog_emu_cc=no) + ],ac_cv_prog_emu_cc="$CC",ac_cv_prog_emu_cc=no) CC=$save_CC CFLAGS=$save_CFLAGS CPPFLAGS=$save_CPPFLAGS fi ]) -if test $ac_cv_prog_emu_cc = no; then +if test "$ac_cv_prog_emu_cc" = no; then AC_DEFINE(NO_JUMP_TABLE,[],[Defined if no found C compiler can handle jump tables]) - EMU_CC=$CC + EMU_CC="$CC" else - EMU_CC=$ac_cv_prog_emu_cc + EMU_CC="$ac_cv_prog_emu_cc" fi AC_SUBST(EMU_CC) ]) @@ -724,6 +724,179 @@ esac ])# AC_C_DOUBLE_MIDDLE_ENDIAN +AC_DEFUN(ERL_MONOTONIC_CLOCK, +[ + AC_CACHE_CHECK([for clock_gettime() with monotonic clock type], erl_cv_clock_gettime_monotonic, + [ + for clock_type in CLOCK_HIGHRES CLOCK_MONOTONIC CLOCK_MONOTONIC_PRECISE; do + AC_TRY_COMPILE([ +#include <time.h> + ], + [ + struct timespec ts; + long long result; + clock_gettime($clock_type,&ts); + result = ((long long) ts.tv_sec) * 1000000000LL + + ((long long) ts.tv_nsec); + ], + erl_cv_clock_gettime_monotonic=$clock_type, + erl_cv_clock_gettime_monotonic=no) + test $erl_cv_clock_gettime_monotonic = no || break + done + ]) + + AC_CHECK_FUNCS([clock_getres clock_get_attributes gethrtime]) + + AC_CACHE_CHECK([for mach clock_get_time() with monotonic clock type], erl_cv_mach_clock_get_time_monotonic, + [ + AC_TRY_COMPILE([ +#include <mach/clock.h> +#include <mach/mach.h> + ], + [ + kern_return_t res; + clock_serv_t clk_srv; + mach_timespec_t time_spec; + + host_get_clock_service(mach_host_self(), SYSTEM_CLOCK, &clk_srv); + res = clock_get_time(clk_srv, &time_spec); + mach_port_deallocate(mach_task_self(), clk_srv); + ], + erl_cv_mach_clock_get_time_monotonic=yes, + erl_cv_mach_clock_get_time_monotonic=no) + ]) + + case $erl_cv_clock_gettime_monotonic-$ac_cv_func_gethrtime-$erl_cv_mach_clock_get_time_monotonic-$host_os in + *-*-*-win32) + erl_monotonic_clock_func=WindowsAPI + ;; + CLOCK_*-*-*-linux*) + if test X$cross_compiling != Xyes; then + linux_kernel_vsn_=`uname -r` + case $linux_kernel_vsn_ in + [[0-1]].*|2.[[0-5]]|2.[[0-5]].*) + erl_monotonic_clock_func=times + ;; + *) + erl_monotonic_clock_func=clock_gettime + ;; + esac + else + case X$erl_xcomp_linux_clock_gettime_correction in + X) + AC_MSG_WARN([result clock_gettime guessed because of cross compilation]) + erl_monotonic_clock_func=clock_gettime + ;; + Xyes|Xno) + if test $erl_xcomp_linux_clock_gettime_correction = yes; then + erl_monotonic_clock_func=clock_gettime + else + erl_monotonic_clock_func=times + fi + ;; + *) + AC_MSG_ERROR([Bad erl_xcomp_linux_clock_gettime_correction value: $erl_xcomp_linux_clock_gettime_correction]) + ;; + esac + fi + ;; + no-no-no-linux*) + erl_monotonic_clock_func=times + ;; + CLOCK_*-*-*-*) + erl_monotonic_clock_func=clock_gettime + ;; + no-yes-*-*) + erl_monotonic_clock_func=gethrtime + ;; + no-no-yes-*) + erl_monotonic_clock_func=mach_clock_get_time + ;; + no-no-no-*) + erl_monotonic_clock_func=none + ;; + esac + + erl_monotonic_clock_lib= + erl_monotonic_clock_id= + case $erl_monotonic_clock_func in + clock_gettime) + erl_monotonic_clock_id="$erl_cv_clock_gettime_monotonic" + AC_CHECK_LIB(rt, clock_gettime, [erl_monotonic_clock_lib="-lrt"]) + ;; + mach_clock_get_time) + erl_monotonic_clock_id=SYSTEM_CLOCK + ;; + *) + ;; + esac + +]) + +AC_DEFUN(ERL_WALL_CLOCK, +[ + AC_CACHE_CHECK([for clock_gettime() with wall clock type], erl_cv_clock_gettime_wall, + [ + for clock_type in CLOCK_REALTIME; do + AC_TRY_COMPILE([ +#include <time.h> + ], + [ + struct timespec ts; + long long result; + clock_gettime($clock_type,&ts); + result = ((long long) ts.tv_sec) * 1000000000LL + + ((long long) ts.tv_nsec); + ], + erl_cv_clock_gettime_wall=$clock_type, + erl_cv_clock_gettime_wall=no) + test $erl_cv_clock_gettime_wall = no || break + done + ]) + + AC_CHECK_FUNCS([clock_getres clock_get_attributes gettimeofday]) + + AC_CACHE_CHECK([for mach clock_get_time() with wall clock type], erl_cv_mach_clock_get_time_wall, + [ + AC_TRY_COMPILE([ +#include <mach/clock.h> +#include <mach/mach.h> + ], + [ + kern_return_t res; + clock_serv_t clk_srv; + mach_timespec_t time_spec; + + host_get_clock_service(mach_host_self(), CALENDAR_CLOCK, &clk_srv); + res = clock_get_time(clk_srv, &time_spec); + mach_port_deallocate(mach_task_self(), clk_srv); + ], + erl_cv_mach_clock_get_time_wall=yes, + erl_cv_mach_clock_get_time_wall=no) + ]) + + erl_wall_clock_id= + case $erl_cv_clock_gettime_wall-$erl_cv_mach_clock_get_time_wall-$ac_cv_func_gettimeofday-$host_os in + *-*-*-win32) + erl_wall_clock_func=WindowsAPI + ;; + no-yes-*-*) + erl_wall_clock_func=mach_clock_get_time + erl_wall_clock_id=CALENDAR_CLOCK + ;; + CLOCK_*-*-*-*) + erl_wall_clock_func=clock_gettime + erl_wall_clock_id=$erl_cv_clock_gettime_wall + ;; + no-no-yes-*) + erl_wall_clock_func=gettimeofday + ;; + *) + erl_wall_clock_func=none + ;; + esac +]) + dnl ---------------------------------------------------------------------- dnl dnl LM_CHECK_THR_LIB @@ -908,24 +1081,226 @@ AC_SUBST(ERTS_INTERNAL_X_LIBS) ]) -AC_DEFUN(ETHR_CHK_SYNC_OP, +AC_DEFUN(ETHR_CHK_GCC_ATOMIC_OP__, [ - AC_MSG_CHECKING([for $3-bit $1()]) - case "$2" in - "1") sync_call="$1(&var);";; - "2") sync_call="$1(&var, ($4) 0);";; - "3") sync_call="$1(&var, ($4) 0, ($4) 0);";; + # $1 - atomic_op + + for atomic_bit_size in 32 64 128; do + case $atomic_bit_size in + 32) gcc_atomic_type="$gcc_atomic_type32";; + 64) gcc_atomic_type="$gcc_atomic_type64";; + 128) gcc_atomic_type="$gcc_atomic_type128";; + esac + gcc_atomic_lockfree="int x[[(2*__atomic_always_lock_free(sizeof($gcc_atomic_type), 0))-1]]" + case $1 in + __sync_add_and_fetch | __sync_fetch_and_and | __sync_fetch_and_or) + atomic_call="volatile $gcc_atomic_type var; $gcc_atomic_type res = $1(&var, ($gcc_atomic_type) 0);" + ;; + __sync_val_compare_and_swap) + atomic_call="volatile $gcc_atomic_type var; $gcc_atomic_type res = $1(&var, ($gcc_atomic_type) 0, ($gcc_atomic_type) 0);" + ;; + __atomic_store_n) + atomic_call="$gcc_atomic_lockfree; volatile $gcc_atomic_type var; $1(&var, ($gcc_atomic_type) 0, __ATOMIC_RELAXED); $1(&var, ($gcc_atomic_type) 0, __ATOMIC_RELEASE);" + ;; + __atomic_load_n) + atomic_call="$gcc_atomic_lockfree; volatile $gcc_atomic_type var; $gcc_atomic_type res = $1(&var, __ATOMIC_RELAXED); res = $1(&var, __ATOMIC_ACQUIRE);" + ;; + __atomic_add_fetch| __atomic_fetch_and | __atomic_fetch_or) + atomic_call="$gcc_atomic_lockfree; volatile $gcc_atomic_type var; $gcc_atomic_type res = $1(&var, ($gcc_atomic_type) 0, __ATOMIC_RELAXED); res = $1(&var, ($gcc_atomic_type) 0, __ATOMIC_ACQUIRE); res = $1(&var, ($gcc_atomic_type) 0, __ATOMIC_RELEASE);" + ;; + __atomic_compare_exchange_n) + atomic_call="$gcc_atomic_lockfree; volatile $gcc_atomic_type var; $gcc_atomic_type val; int res = $1(&var, &val, ($gcc_atomic_type) 0, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); res = $1(&var, &val, ($gcc_atomic_type) 0, 0, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE);" + ;; + *) + AC_MSG_ERROR([Internal error: missing implementation for $1]) + ;; + esac + eval atomic${atomic_bit_size}_call=\"$atomic_call\" + done + + AC_CACHE_CHECK([for 32-bit $1()], ethr_cv_32bit_$1, + [ + ethr_cv_32bit_$1=no + AC_TRY_LINK([], [$atomic32_call], [ethr_cv_32bit_$1=yes]) + ]) + AC_CACHE_CHECK([for 64-bit $1()], ethr_cv_64bit_$1, + [ + ethr_cv_64bit_$1=no + AC_TRY_LINK([], [$atomic64_call], [ethr_cv_64bit_$1=yes]) + ]) + AC_CACHE_CHECK([for 128-bit $1()], ethr_cv_128bit_$1, + [ + ethr_cv_128bit_$1=no + AC_TRY_LINK([], [$atomic128_call], [ethr_cv_128bit_$1=yes]) + ]) + + case $ethr_cv_128bit_$1-$ethr_cv_64bit_$1-$ethr_cv_32bit_$1 in + no-no-no) + have_atomic_ops=0;; + no-no-yes) + have_atomic_ops=4;; + no-yes-no) + have_atomic_ops=8;; + no-yes-yes) + have_atomic_ops=12;; + yes-no-no) + have_atomic_ops=16;; + yes-no-yes) + have_atomic_ops=20;; + yes-yes-no) + have_atomic_ops=24;; + yes-yes-yes) + have_atomic_ops=28;; + esac + AC_DEFINE_UNQUOTED([ETHR_HAVE_$1], [$have_atomic_ops], [Define as a bitmask corresponding to the word sizes that $1() can handle on your system]) +]) + +AC_DEFUN(ETHR_CHK_IF_NOOP, +[ + ethr_test_filename="chk_if_$1$3_noop_config1test.$$" + cat > "${ethr_test_filename}.c" <<EOF +int +my_test(void) +{ + $1$2; + return 0; +} +EOF + $CC -O3 $ETHR_DEFS -c "${ethr_test_filename}.c" -o "${ethr_test_filename}1.o" + cat > "${ethr_test_filename}.c" <<EOF +int +my_test(void) +{ + ; + return 0; +} +EOF + $CC -O3 $ETHR_DEFS -c "${ethr_test_filename}.c" -o "${ethr_test_filename}2.o" + if diff "${ethr_test_filename}1.o" "${ethr_test_filename}2.o" >/dev/null 2>&1; then + ethr_$1$3_noop=yes + else + ethr_$1$3_noop=no + fi + rm -f "${ethr_test_filename}.c" "${ethr_test_filename}1.o" "${ethr_test_filename}2.o" +]) + +AC_DEFUN(ETHR_CHK_GCC_ATOMIC_OPS, +[ + AC_CHECK_SIZEOF(short) + AC_CHECK_SIZEOF(int) + AC_CHECK_SIZEOF(long) + AC_CHECK_SIZEOF(long long) + AC_CHECK_SIZEOF(__int128_t) + + if test "$ac_cv_sizeof_short" = "4"; then + gcc_atomic_type32="short" + elif test "$ac_cv_sizeof_int" = "4"; then + gcc_atomic_type32="int" + elif test "$ac_cv_sizeof_long" = "4"; then + gcc_atomic_type32="long" + else + AC_MSG_ERROR([No 32-bit type found]) + fi + + if test "$ac_cv_sizeof_int" = "8"; then + gcc_atomic_type64="int" + elif test "$ac_cv_sizeof_long" = "8"; then + gcc_atomic_type64="long" + elif test "$ac_cv_sizeof_long_long" = "8"; then + gcc_atomic_type64="long long" + else + AC_MSG_ERROR([No 64-bit type found]) + fi + + if test "$ac_cv_sizeof___int128_t" = "16"; then + gcc_atomic_type128="__int128_t" + else + gcc_atomic_type128="#error " + fi + AC_CACHE_CHECK([for a working __sync_synchronize()], ethr_cv___sync_synchronize, + [ + ethr_cv___sync_synchronize=no + AC_TRY_LINK([], + [ __sync_synchronize(); ], + [ethr_cv___sync_synchronize=yes]) + if test $ethr_cv___sync_synchronize = yes; then + # + # Old gcc versions on at least x86 have a buggy + # __sync_synchronize() which does not emit a + # memory barrier. We try to detect this by + # compiling to assembly with and without + # __sync_synchronize() and compare the results. + # + ETHR_CHK_IF_NOOP(__sync_synchronize, [()], []) + if test $ethr___sync_synchronize_noop = yes; then + # Got a buggy implementation of + # __sync_synchronize... + ethr_cv___sync_synchronize="no; buggy implementation" + fi + fi + ]) + + if test "$ethr_cv___sync_synchronize" = "yes"; then + have_sync_synchronize_value="~0" + else + have_sync_synchronize_value="0" + fi + AC_DEFINE_UNQUOTED([ETHR_HAVE___sync_synchronize], [$have_sync_synchronize_value], [Define as a bitmask corresponding to the word sizes that __sync_synchronize() can handle on your system]) + + ETHR_CHK_GCC_ATOMIC_OP__(__sync_add_and_fetch) + ETHR_CHK_GCC_ATOMIC_OP__(__sync_fetch_and_and) + ETHR_CHK_GCC_ATOMIC_OP__(__sync_fetch_and_or) + ETHR_CHK_GCC_ATOMIC_OP__(__sync_val_compare_and_swap) + + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_store_n) + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_load_n) + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_add_fetch) + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_fetch_and) + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_fetch_or) + ETHR_CHK_GCC_ATOMIC_OP__(__atomic_compare_exchange_n) + + ethr_have_gcc_native_atomics=no + ethr_arm_dbm_instr_val=0 + case "$GCC-$host_cpu" in + yes-arm*) + AC_CACHE_CHECK([for ARM DMB instruction], ethr_cv_arm_dbm_instr, + [ + ethr_cv_arm_dbm_instr=no + AC_TRY_LINK([], + [ + __asm__ __volatile__("dmb sy" : : : "memory"); + __asm__ __volatile__("dmb st" : : : "memory"); + ], + [ethr_cv_arm_dbm_instr=yes]) + ]) + if test $ethr_cv_arm_dbm_instr = yes; then + ethr_arm_dbm_instr_val=1 + test $ethr_cv_64bit___atomic_compare_exchange_n = yes && + ethr_have_gcc_native_atomics=yes + fi;; + *) + ;; esac - have_sync_op=no - AC_TRY_LINK([], - [ - $4 res; - volatile $4 var; - res = $sync_call - ], - [have_sync_op=yes]) - test $have_sync_op = yes && $5 - AC_MSG_RESULT([$have_sync_op]) + AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION], [$ethr_arm_dbm_instr_val], [Define as a boolean indicating whether you have a gcc compatible compiler capable of generating the ARM DMB instruction, and are compiling for an ARM processor with ARM DMB instruction support, or not]) + test $ethr_cv_32bit___sync_val_compare_and_swap = yes && + ethr_have_gcc_native_atomics=yes + test $ethr_cv_64bit___sync_val_compare_and_swap = yes && + ethr_have_gcc_native_atomics=yes + if test "$ethr_cv___sync_synchronize" = "yes"; then + test $ethr_cv_64bit___atomic_compare_exchange_n = yes && + ethr_have_gcc_native_atomics=yes + test $ethr_cv_32bit___atomic_compare_exchange_n = yes && + ethr_have_gcc_native_atomics=yes + fi + ethr_have_gcc_atomic_builtins=0 + if test $ethr_have_gcc_native_atomics = yes; then + ethr_native_atomic_implementation=gcc_sync + test $ethr_cv_32bit___atomic_compare_exchange_n = yes && ethr_have_gcc_atomic_builtins=1 + test $ethr_cv_64bit___atomic_compare_exchange_n = yes && ethr_have_gcc_atomic_builtins=1 + test $ethr_have_gcc_atomic_builtins = 1 && ethr_native_atomic_implementation=gcc_atomic_sync + fi + AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC___ATOMIC_BUILTINS], [$ethr_have_gcc_atomic_builtins], [Define as a boolean indicating whether you have a gcc __atomic builtins or not]) + test $ethr_have_gcc_native_atomics = yes && ethr_have_native_atomics=yes ]) AC_DEFUN(ETHR_CHK_INTERLOCKED, @@ -1005,6 +1380,16 @@ AC_ARG_ENABLE(prefer-gcc-native-ethr-impls, test $enable_prefer_gcc_native_ethr_impls = yes && AC_DEFINE(ETHR_PREFER_GCC_NATIVE_IMPLS, 1, [Define if you prefer gcc native ethread implementations]) +AC_ARG_ENABLE(trust-gcc-atomic-builtins-memory-barriers, + AS_HELP_STRING([--enable-trust-gcc-atomic-builtins-memory-barriers], + [trust gcc atomic builtins memory barriers]), +[ case "$enableval" in + yes) trust_gcc_atomic_builtins_mbs=1 ;; + *) trust_gcc_atomic_builtins_mbs=0 ;; + esac ], trust_gcc_atomic_builtins_mbs=0) + +AC_DEFINE_UNQUOTED(ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS, [$trust_gcc_atomic_builtins_mbs], [Define as a boolean indicating whether you trust gcc's __atomic_* builtins memory barrier implementations, or not]) + AC_ARG_WITH(libatomic_ops, AS_HELP_STRING([--with-libatomic_ops=PATH], [specify and prefer usage of libatomic_ops in the ethread library])) @@ -1016,12 +1401,33 @@ AC_ARG_WITH(with_sparc_memory_order, LM_CHECK_THR_LIB ERL_INTERNAL_LIBS +ERL_MONOTONIC_CLOCK + +case $erl_monotonic_clock_func in + clock_gettime) + AC_DEFINE(ETHR_HAVE_CLOCK_GETTIME_MONOTONIC, [1], [Define if you have a clock_gettime() with a monotonic clock]) + ;; + mach_clock_get_time) + AC_DEFINE(ETHR_HAVE_MACH_CLOCK_GET_TIME, [1], [Define if you have a mach clock_get_time() with a monotonic clock]) + ;; + gethrtime) + AC_DEFINE(ETHR_HAVE_GETHRTIME, [1], [Define if you have a monotonic gethrtime()]) + ;; + *) + ;; +esac + +if test "x$erl_monotonic_clock_id" != "x"; then + AC_DEFINE_UNQUOTED(ETHR_MONOTONIC_CLOCK_ID, [$erl_monotonic_clock_id], [Define to the monotonic clock id to use]) +fi + +ethr_native_atomic_implementation=none ethr_have_native_atomics=no ethr_have_native_spinlock=no ETHR_THR_LIB_BASE="$THR_LIB_NAME" ETHR_THR_LIB_BASE_TYPE="$THR_LIB_TYPE" ETHR_DEFS="$THR_DEFS" -ETHR_X_LIBS="$THR_LIBS $ERTS_INTERNAL_X_LIBS" +ETHR_X_LIBS="$THR_LIBS $ERTS_INTERNAL_X_LIBS $erl_monotonic_clock_lib" ETHR_LIBS= ETHR_LIB_NAME= @@ -1100,7 +1506,10 @@ case "$THR_LIB_NAME" in ETHR_CHK_INTERLOCKED([_InterlockedCompareExchange128], [4], [__int64], AC_DEFINE_UNQUOTED(ETHR_HAVE__INTERLOCKEDCOMPAREEXCHANGE128, 1, [Define if you have _InterlockedCompareExchange128()])) fi - test "$ethr_have_native_atomics" = "yes" && ethr_have_native_spinlock=yes + if test "$ethr_have_native_atomics" = "yes"; then + ethr_native_atomic_implementation=windows + ethr_have_native_spinlock=yes + fi ;; pthread|ose_threads) @@ -1329,6 +1738,50 @@ case "$THR_LIB_NAME" in AC_DEFINE(ETHR_HAVE_PTHREAD_ATTR_SETGUARDSIZE, 1, \ [Define if you have the pthread_attr_setguardsize function.])) + if test "x$erl_monotonic_clock_id" != "x"; then + AC_MSG_CHECKING(whether pthread_cond_timedwait() can use the monotonic clock $erl_monotonic_clock_id for timeout) + pthread_cond_timedwait_monotonic=no + AC_TRY_LINK([ + #if defined(ETHR_NEED_NPTL_PTHREAD_H) + # include <nptl/pthread.h> + #elif defined(ETHR_HAVE_MIT_PTHREAD_H) + # include <pthread/mit/pthread.h> + #elif defined(ETHR_HAVE_PTHREAD_H) + # include <pthread.h> + #endif + #ifdef ETHR_TIME_WITH_SYS_TIME + # include <time.h> + # include <sys/time.h> + #else + # ifdef ETHR_HAVE_SYS_TIME_H + # include <sys/time.h> + # else + # include <time.h> + # endif + #endif + #if defined(ETHR_HAVE_MACH_CLOCK_GET_TIME) + # include <mach/clock.h> + # include <mach/mach.h> + #endif + ], + [ + int res; + pthread_condattr_t attr; + pthread_cond_t cond; + struct timespec cond_timeout; + pthread_mutex_t mutex; + res = pthread_condattr_init(&attr); + res = pthread_condattr_setclock(&attr, ETHR_MONOTONIC_CLOCK_ID); + res = pthread_cond_init(&cond, &attr); + res = pthread_cond_timedwait(&cond, &mutex, &cond_timeout); + ], + [pthread_cond_timedwait_monotonic=yes]) + AC_MSG_RESULT([$pthread_cond_timedwait_monotonic]) + if test $pthread_cond_timedwait_monotonic = yes; then + AC_DEFINE(ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC, [1], [Define if pthread_cond_timedwait() can be used with a monotonic clock]) + fi + fi + linux_futex=no AC_MSG_CHECKING([for Linux futexes]) AC_TRY_LINK([ @@ -1400,54 +1853,11 @@ case "$THR_LIB_NAME" in fi ## test "x$THR_LIB_NAME" = "xpthread" - AC_CHECK_SIZEOF(int) - AC_CHECK_SIZEOF(long) - AC_CHECK_SIZEOF(long long) - AC_CHECK_SIZEOF(__int128_t) - - if test "$ac_cv_sizeof_int" = "4"; then - int32="int" - elif test "$ac_cv_sizeof_long" = "4"; then - int32="long" - elif test "$ac_cv_sizeof_long_long" = "4"; then - int32="long long" - else - AC_MSG_ERROR([No 32-bit type found]) - fi - - if test "$ac_cv_sizeof_int" = "8"; then - int64="int" - elif test "$ac_cv_sizeof_long" = "8"; then - int64="long" - elif test "$ac_cv_sizeof_long_long" = "8"; then - int64="long long" - else - AC_MSG_ERROR([No 64-bit type found]) - fi - - int128=no - if test "$ac_cv_sizeof___int128_t" = "16"; then - int128="__int128_t" - fi - if test "X$disable_native_ethr_impls" = "Xyes"; then ethr_have_native_atomics=no else - ETHR_CHK_SYNC_OP([__sync_val_compare_and_swap], [3], [32], [$int32], AC_DEFINE(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP32, 1, [Define if you have __sync_val_compare_and_swap() for 32-bit integers])) - test "$have_sync_op" = "yes" && ethr_have_native_atomics=yes - ETHR_CHK_SYNC_OP([__sync_add_and_fetch], [2], [32], [$int32], AC_DEFINE(ETHR_HAVE___SYNC_ADD_AND_FETCH32, 1, [Define if you have __sync_add_and_fetch() for 32-bit integers])) - ETHR_CHK_SYNC_OP([__sync_fetch_and_and], [2], [32], [$int32], AC_DEFINE(ETHR_HAVE___SYNC_FETCH_AND_AND32, 1, [Define if you have __sync_fetch_and_and() for 32-bit integers])) - ETHR_CHK_SYNC_OP([__sync_fetch_and_or], [2], [32], [$int32], AC_DEFINE(ETHR_HAVE___SYNC_FETCH_AND_OR32, 1, [Define if you have __sync_fetch_and_or() for 32-bit integers])) - - ETHR_CHK_SYNC_OP([__sync_val_compare_and_swap], [3], [64], [$int64], AC_DEFINE(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64, 1, [Define if you have __sync_val_compare_and_swap() for 64-bit integers])) - test "$have_sync_op" = "yes" && ethr_have_native_atomics=yes - ETHR_CHK_SYNC_OP([__sync_add_and_fetch], [2], [64], [$int64], AC_DEFINE(ETHR_HAVE___SYNC_ADD_AND_FETCH64, 1, [Define if you have __sync_add_and_fetch() for 64-bit integers])) - ETHR_CHK_SYNC_OP([__sync_fetch_and_and], [2], [64], [$int64], AC_DEFINE(ETHR_HAVE___SYNC_FETCH_AND_AND64, 1, [Define if you have __sync_fetch_and_and() for 64-bit integers])) - ETHR_CHK_SYNC_OP([__sync_fetch_and_or], [2], [64], [$int64], AC_DEFINE(ETHR_HAVE___SYNC_FETCH_AND_OR64, 1, [Define if you have __sync_fetch_and_or() for 64-bit integers])) - - if test $int128 != no; then - ETHR_CHK_SYNC_OP([__sync_val_compare_and_swap], [3], [128], [$int128], AC_DEFINE(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP128, 1, [Define if you have __sync_val_compare_and_swap() for 128-bit integers])) - fi + + ETHR_CHK_GCC_ATOMIC_OPS([]) AC_MSG_CHECKING([for a usable libatomic_ops implementation]) case "x$with_libatomic_ops" in @@ -1497,6 +1907,7 @@ case "$THR_LIB_NAME" in #endif ], [ethr_have_native_atomics=yes + ethr_native_atomic_implementation=libatomic_ops ethr_have_libatomic_ops=yes]) AC_MSG_RESULT([$ethr_have_libatomic_ops]) if test $ethr_have_libatomic_ops = yes; then @@ -1528,15 +1939,19 @@ case "$THR_LIB_NAME" in *) AC_MSG_ERROR([Unsupported Sparc memory order: $with_sparc_memory_order]);; esac + ethr_native_atomic_implementation=ethread ethr_have_native_atomics=yes;; i86pc | i*86 | x86_64 | amd64) if test "$enable_x86_out_of_order" = "yes"; then AC_DEFINE(ETHR_X86_OUT_OF_ORDER, 1, [Define if x86/x86_64 out of order instructions should be synchronized]) fi + ethr_native_atomic_implementation=ethread ethr_have_native_atomics=yes;; macppc | ppc | powerpc | "Power Macintosh") + ethr_native_atomic_implementation=ethread ethr_have_native_atomics=yes;; tile) + ethr_native_atomic_implementation=ethread ethr_have_native_atomics=yes;; *) ;; @@ -1709,7 +2124,6 @@ AC_SUBST(ETHR_X86_SSE2_ASM) ]) - dnl ---------------------------------------------------------------------- dnl dnl ERL_TIME_CORRECTION @@ -1725,93 +2139,54 @@ dnl work... dnl AC_DEFUN(ERL_TIME_CORRECTION, -[if test x$ac_cv_func_gethrtime = x; then - AC_CHECK_FUNC(gethrtime) -fi -if test x$clock_gettime_correction = xunknown; then - AC_TRY_COMPILE([#include <time.h>], - [struct timespec ts; - long long result; - clock_gettime(CLOCK_MONOTONIC,&ts); - result = ((long long) ts.tv_sec) * 1000000000LL + - ((long long) ts.tv_nsec);], - clock_gettime_compiles=yes, - clock_gettime_compiles=no) -else - clock_gettime_compiles=no -fi - - -AC_CACHE_CHECK([how to correct for time adjustments], erl_cv_time_correction, [ -case $clock_gettime_correction in - yes) - erl_cv_time_correction=clock_gettime;; - no|unknown) - case $ac_cv_func_gethrtime in - yes) - erl_cv_time_correction=hrtime ;; - no) - case $host_os in - linux*) - case $clock_gettime_correction in - unknown) - if test x$clock_gettime_compiles = xyes; then - if test X$cross_compiling != Xyes; then - linux_kernel_vsn_=`uname -r` - case $linux_kernel_vsn_ in - [[0-1]].*|2.[[0-5]]|2.[[0-5]].*) - erl_cv_time_correction=times ;; - *) - erl_cv_time_correction=clock_gettime;; - esac - else - case X$erl_xcomp_linux_clock_gettime_correction in - X) - erl_cv_time_correction=cross;; - Xyes|Xno) - if test $erl_xcomp_linux_clock_gettime_correction = yes; then - erl_cv_time_correction=clock_gettime - else - erl_cv_time_correction=times - fi;; - *) - AC_MSG_ERROR([Bad erl_xcomp_linux_clock_gettime_correction value: $erl_xcomp_linux_clock_gettime_correction]);; - esac - fi - else - erl_cv_time_correction=times - fi - ;; - *) - erl_cv_time_correction=times ;; - esac - ;; - *) - erl_cv_time_correction=none ;; - esac - ;; - esac - ;; + +ERL_WALL_CLOCK + +case $erl_wall_clock_func in + mach_clock_get_time) + AC_DEFINE(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME, [1], [Define if you want to implement erts_os_system_time() using mach clock_get_time()]) + ;; + clock_gettime) + AC_DEFINE(OS_SYSTEM_TIME_USING_CLOCK_GETTIME, [1], [Define if you want to implement erts_os_system_time() using clock_gettime()]) + ;; + gettimeofday) + AC_DEFINE(OS_SYSTEM_TIME_GETTIMEOFDAY, [1], [Define if you want to implement erts_os_system_time() using gettimeofday()]) + ;; + *) + ;; esac -]) -xrtlib="" -case $erl_cv_time_correction in +if test "x$erl_wall_clock_id" != "x"; then + AC_DEFINE_UNQUOTED(WALL_CLOCK_ID_STR, ["$erl_wall_clock_id"], [Define as a string of wall clock id to use]) + AC_DEFINE_UNQUOTED(WALL_CLOCK_ID, [$erl_wall_clock_id], [Define to wall clock id to use]) +fi + +ERL_MONOTONIC_CLOCK + +case $erl_monotonic_clock_func in times) - AC_DEFINE(CORRECT_USING_TIMES,[], - [Define if you do not have a high-res. timer & want to use times() instead]) + AC_DEFINE(OS_MONOTONIC_TIME_USING_TIMES, [1], [Define if you want to implement erts_os_monotonic_time() using times()]) ;; - clock_gettime|cross) - if test $erl_cv_time_correction = cross; then - erl_cv_time_correction=clock_gettime - AC_MSG_WARN([result clock_gettime guessed because of cross compilation]) - fi - xrtlib="-lrt" - AC_DEFINE(GETHRTIME_WITH_CLOCK_GETTIME,[1], - [Define if you want to use clock_gettime to simulate gethrtime]) + mach_clock_get_time) + AC_DEFINE(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME, [1], [Define if you want to implement erts_os_monotonic_time() using mach clock_get_time()]) + ;; + clock_gettime) + AC_DEFINE(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME, [1], [Define if you want to implement erts_os_monotonic_time() using clock_gettime()]) + ;; + gethrtime) + AC_DEFINE(OS_MONOTONIC_TIME_USING_GETHRTIME, [1], [Define if you want to implement erts_os_monotonic_time() using gethrtime()]) + ;; + *) ;; esac + +xrtlib="$erl_monotonic_clock_lib" +if test "x$erl_monotonic_clock_id" != "x"; then + AC_DEFINE_UNQUOTED(MONOTONIC_CLOCK_ID_STR, ["$erl_monotonic_clock_id"], [Define as a string of monotonic clock id to use]) + AC_DEFINE_UNQUOTED(MONOTONIC_CLOCK_ID, [$erl_monotonic_clock_id], [Define to monotonic clock id to use]) +fi + dnl dnl Check if gethrvtime is working, and if to use procfs ioctl dnl or (yet to be written) write to the procfs ctl file. @@ -1884,6 +2259,7 @@ case X$erl_xcomp_gethrvtime_procfs_ioctl in esac ]) +LIBRT=$xrtlib case $erl_gethrvtime in procfs_ioctl) AC_DEFINE(HAVE_GETHRVTIME_PROCFS_IOCTL,[1], @@ -1950,15 +2326,13 @@ case $erl_gethrvtime in cross) erl_clock_gettime_cpu_time=no AC_MSG_WARN([result no guessed because of cross compilation]) - LIBRT=$xrtlib ;; *) - LIBRT=$xrtlib ;; esac - AC_SUBST(LIBRT) ;; esac +AC_SUBST(LIBRT) ])dnl dnl ---------------------------------------------------------------------- diff --git a/erts/autoconf/win32.config.cache.static b/erts/autoconf/win32.config.cache.static index b387db2b22..b3328e5414 100755 --- a/erts/autoconf/win32.config.cache.static +++ b/erts/autoconf/win32.config.cache.static @@ -221,7 +221,6 @@ ac_cv_type_size_t=${ac_cv_type_size_t=yes} ac_cv_type_uid_t=${ac_cv_type_uid_t=no} ac_cv_type_void_p=${ac_cv_type_void_p=yes} ac_cv_working_alloca_h=${ac_cv_working_alloca_h=no} -erl_cv_time_correction=${erl_cv_time_correction=none} erts_cv___after_morecore_hook_can_track_malloc=${erts_cv___after_morecore_hook_can_track_malloc=no} erts_cv_fwrite_unlocked=${erts_cv_fwrite_unlocked=no} erts_cv_have__end_symbol=${erts_cv_have__end_symbol=no} diff --git a/erts/autoconf/win64.config.cache.static b/erts/autoconf/win64.config.cache.static index a8a2bfb59c..c7d92c7000 100755 --- a/erts/autoconf/win64.config.cache.static +++ b/erts/autoconf/win64.config.cache.static @@ -262,7 +262,6 @@ ac_cv_type_signal=${ac_cv_type_signal=void} ac_cv_type_size_t=${ac_cv_type_size_t=yes} ac_cv_type_uid_t=${ac_cv_type_uid_t=no} ac_cv_working_alloca_h=${ac_cv_working_alloca_h=no} -erl_cv_time_correction=${erl_cv_time_correction=none} erts_cv___after_morecore_hook_can_track_malloc=${erts_cv___after_morecore_hook_can_track_malloc=no} erts_cv_fwrite_unlocked=${erts_cv_fwrite_unlocked=no} erts_cv_have__end_symbol=${erts_cv_have__end_symbol=no} diff --git a/erts/configure.in b/erts/configure.in index f248cbe34d..481dfe405e 100644 --- a/erts/configure.in +++ b/erts/configure.in @@ -2,7 +2,7 @@ dnl Process this file with autoconf to produce a configure script. -*-m4-*- dnl %CopyrightBegin% dnl -dnl Copyright Ericsson AB 1997-2014. All Rights Reserved. +dnl Copyright Ericsson AB 1997-2015. All Rights Reserved. dnl dnl The contents of this file are subject to the Erlang Public License, dnl Version 1.1, (the "License"); you may not use this file except in @@ -1105,10 +1105,31 @@ if test $ERTS_BUILD_SMP_EMU = yes; then case "$ethr_have_native_atomics-$smp_require_native_atomics-$ethr_have_native_spinlock" in yes-*) + if test "$ethr_native_atomic_implementation" = "gcc_sync"; then + test -f "$ERL_TOP/erts/CONF_INFO" || + echo "" > "$ERL_TOP/erts/CONF_INFO" + cat >> $ERL_TOP/erts/CONF_INFO <<EOF + + WARNING: + Only gcc's __sync_* builtins available for + atomic memory access. This will cause lots + of expensive and unnecessary memory barrier + instructions to be issued which will make + the performance of the runtime system + suffer. You are *strongly* advised to + upgrade to a gcc version that supports the + __atomic_* builtins (at least gcc version + 4.7) or build with libatomic_ops. See the + "Atomic Memory Operations and the VM" + chapter of \$ERL_TOP/HOWTO/INSTALL.md for + more information. + +EOF + fi ;; no-yes-*) - AC_MSG_ERROR([No native atomic implementation found. See Configuring section in INSTALL.md for more information.]) + AC_MSG_ERROR([No native atomic implementation found. See the \"Atomic Memory Operations and the VM\" chapter of \$ERL_TOP/HOWTO/INSTALL.md for more information.]) ;; no-no-yes) diff --git a/erts/doc/src/Makefile b/erts/doc/src/Makefile index e8b856c3ff..a83aa9b875 100644 --- a/erts/doc/src/Makefile +++ b/erts/doc/src/Makefile @@ -177,6 +177,8 @@ release_docs_spec: docs $(INSTALL_DIR) "$(RELSYSDIR)/doc/html" $(INSTALL_DATA) $(HTMLDIR)/* \ "$(RELSYSDIR)/doc/html" + $(INSTALL_DATA) $(ERL_TOP)/erts/example/time_compat.erl \ + "$(RELSYSDIR)/doc/html" $(INSTALL_DATA) $(INFO_FILE) "$(RELSYSDIR)" $(INSTALL_DIR) "$(RELEASE_PATH)/man/man3" $(INSTALL_DATA) $(MAN3DIR)/* "$(RELEASE_PATH)/man/man3" diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index 16000191dc..ea94a4e82b 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -495,24 +495,35 @@ <c><![CDATA[werl]]></c>, not <c><![CDATA[erl]]></c> (<c><![CDATA[oldshell]]></c>). Note also that <c><![CDATA[Ctrl-Break]]></c> is used instead of <c><![CDATA[Ctrl-C]]></c> on Windows.</p> </item> - <tag><marker id="+c"><c><![CDATA[+c]]></c></marker></tag> - <item> - <p>Disable compensation for sudden changes of system time.</p> - <p>Normally, <c><![CDATA[erlang:now/0]]></c> will not immediately reflect - sudden changes in the system time, in order to keep timers - (including <c><![CDATA[receive-after]]></c>) working. Instead, the time - maintained by <c><![CDATA[erlang:now/0]]></c> is slowly adjusted towards - the new system time. (Slowly means in one percent adjustments; - if the time is off by one minute, the time will be adjusted - in 100 minutes.)</p> - <p>When the <c><![CDATA[+c]]></c> option is given, this slow adjustment - will not take place. Instead <c><![CDATA[erlang:now/0]]></c> will always - reflect the current system time. Note that timers are based - on <c><![CDATA[erlang:now/0]]></c>. If the system time jumps, timers - then time out at the wrong time.</p> - <p><em>NOTE</em>: You can check whether the adjustment is enabled or - disabled by calling - <seealso marker="erlang#system_info_tolerant_timeofday">erlang:system_info(tolerant_timeofday)</seealso>.</p> + <tag><marker id="+c"><c><![CDATA[+c true | false]]></c></marker></tag> + <item> + <p>Enable or disable + <seealso marker="time_correction#Time_Correction">time correction</seealso>:</p> + <taglist> + <tag><c>true</c></tag> + <item><p>Enable time correction. This is the default if + time correction is supported on the specific platform.</p></item> + + <tag><c>false</c></tag> + <item><p>Disable time correction.</p></item> + </taglist> + <p>For backwards compatibility, the boolean value can be omitted. + This is interpreted as <c>+c false</c>. + </p> + </item> + <tag><marker id="+C_"><c><![CDATA[+C no_time_warp | single_time_warp | multi_time_warp]]></c></marker></tag> + <item> + <p>Set + <seealso marker="time_correction#Time_Warp_Modes">time warp mode</seealso>: + </p> + <taglist> + <tag><c>no_time_warp</c></tag> + <item><p><seealso marker="time_correction#No_Time_Warp_Mode">No Time Warp Mode</seealso> (the default)</p></item> + <tag><c>single_time_warp</c></tag> + <item><p><seealso marker="time_correction#Single_Time_Warp_Mode">Single Time Warp Mode</seealso></p></item> + <tag><c>multi_time_warp</c></tag> + <item><p><seealso marker="time_correction#Multi_Time_Warp_Mode">Multi Time Warp Mode</seealso></p></item> + </taglist> </item> <tag><c><![CDATA[+d]]></c></tag> <item> @@ -588,6 +599,11 @@ <p>Sets the default binary virtual heap size of processes to the size <c><![CDATA[Size]]></c>.</p> </item> + <tag><c><![CDATA[+hpds Size]]></c></tag> + <item> + <p>Sets the initial process dictionary size of processes to the size + <c><![CDATA[Size]]></c>.</p> + </item> <tag><c><![CDATA[+K true | false]]></c></tag> <item> <p>Enables or disables the kernel poll functionality if diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 07b5fdd039..ba5f80a9c1 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -58,7 +58,78 @@ </datatype> <datatype> <name name="timestamp"></name> - <desc><p>See <seealso marker="#now/0">now/0</seealso>.</p> + <desc><p>See <seealso marker="#timestamp/0">erlang:timestamp/0</seealso>.</p> + </desc> + </datatype> + <marker id="type_time_unit"/> + <datatype> + <name name="time_unit"></name> + <desc><p>Currently supported time unit representations:</p> + <taglist> + <tag><c>PartsPerSecond :: integer() >= 1</c></tag> + <item><p>Time unit expressed in parts per second. That is, + the time unit equals <c>1/PartsPerSecond</c> second.</p></item> + + <tag><c>seconds</c></tag> + <item><p>Symbolic representation of the time unit + represented by the integer <c>1</c>.</p></item> + + <tag><c>milli_seconds</c></tag> + <item><p>Symbolic representation of the time unit + represented by the integer <c>1000</c>.</p></item> + + <tag><c>micro_seconds</c></tag> + <item><p>Symbolic representation of the time unit + represented by the integer <c>1000000</c>.</p></item> + + <tag><c>nano_seconds</c></tag> + <item><p>Symbolic representation of the time unit + represented by the integer <c>1000000000</c>.</p></item> + + <tag><c>native</c></tag> + <item><p>Symbolic representation of the native time unit + used by the Erlang runtime system.</p> + + <p>The <c>native</c> time unit is determined at + runtime system start, and will remain the same until + the runtime system terminates. If a runtime system + is stopped and then started again (even on the same + machine), the <c>native</c> time unit of the new + runtime system instance may differ from the + <c>native</c> time unit of the old runtime system + instance.</p> + + <p>One can get an approximation of the <c>native</c> + time unit by calling <c>erlang:convert_time_unit(1, + seconds, native)</c>. The result equals the number + of whole <c>native</c> time units per second. In case + the number of <c>native</c> time units per second does + not add up to a whole number, the result will be + rounded downwards.</p> + + <note> + <p>The value of the <c>native</c> time unit gives + you more or less no information at all about the + quality of time values. It sets a limit for + the + <seealso marker="time_correction#Time_Resolution">resolution</seealso> + as well as for the + <seealso marker="time_correction#Time_Precision">precision</seealso> + of time values, + but it gives absolutely no information at all about the + <seealso marker="time_correction#Time_Accuracy">accuracy</seealso> + of time values. The resolution of the <c>native</c> time + unit and the resolution of time values may differ + significantly.</p> + </note> + </item> + + </taglist> + + <p>The <c>time_unit/0</c> type may be extended. Use + <seealso marker="#convert_time_unit/3"><c>erlang:convert_time_unit/3</c></seealso> + in order to convert time values between time units.</p> + </desc> </datatype> </datatypes> @@ -585,6 +656,22 @@ </desc> </func> <func> + <name name="convert_time_unit" arity="3"/> + <fsummary>Convert time unit of a time value</fsummary> + <desc> + <p>Converts the <c><anno>Time</anno></c> value of time unit + <c><anno>FromUnit</anno></c> to the corresponding + <c><anno>ConvertedTime</anno></c> value of time unit + <c><anno>ToUnit</anno></c>. The result is rounded + using the floor function.</p> + + <warning><p>You may lose accuracy and precision when converting + between time units. In order to minimize such loss, collect all + data at <c>native</c> time unit and do the conversion on the end + result.</p></warning> + </desc> + </func> + <func> <name name="crc32" arity="1"/> <fsummary>Compute crc32 (IEEE 802.3) checksum</fsummary> <desc> @@ -1357,7 +1444,7 @@ true <name name="get" arity="1"/> <fsummary>Return a value from the process dictionary</fsummary> <desc> - <p>Returns the value <c><anno>Val</anno></c>associated with <c><anno>Key</anno></c> in + <p>Returns the value <c><anno>Val</anno></c> associated with <c><anno>Key</anno></c> in the process dictionary, or <c>undefined</c> if <c><anno>Key</anno></c> does not exist.</p> <pre> @@ -2205,14 +2292,15 @@ os_prompt% </pre> </func> <func> <name name="make_ref" arity="0"/> - <fsummary>Return an almost unique reference</fsummary> + <fsummary>Return a unique reference</fsummary> <desc> - <p>Returns an almost unique reference.</p> - <p>The returned reference will re-occur after approximately 2^82 - calls; therefore it is unique enough for practical purposes.</p> - <pre> -> <input>make_ref().</input> -#Ref<0.0.0.135></pre> + <p>Return a <seealso marker="doc/efficiency_guide:advanced#unique_references">unique + reference</seealso>. The reference is unique among + connected nodes.</p> + <warning><p>Known issue: When a node is restarted multiple + times with the same node name, references created + on a newer node can be mistaken for a reference + created on an older node with the same node name.</p></warning> </desc> </func> <func> @@ -2513,97 +2601,178 @@ os_prompt% </pre> </desc> </func> <func> - <name name="monitor" arity="2"/> + <name name="monitor" arity="2" clause_i="1"/> + <name name="monitor" arity="2" clause_i="2"/> + <type name="registered_name"/> + <type name="registered_process_identifier"/> + <type name="monitor_process_identifier"/> <fsummary>Start monitoring</fsummary> <desc> - <p>The calling process starts monitoring <c><anno>Item</anno></c> which is - an object of type <c><anno>Type</anno></c>.</p> - <p>Currently only processes can be monitored, i.e. the only - allowed <c><anno>Type</anno></c> is <c>process</c>, but other types may be - allowed in the future.</p> - <p><c><anno>Item</anno></c> can be:</p> - <taglist> - <tag><c>pid()</c></tag> - <item> - <p>The pid of the process to monitor.</p> - </item> - <tag><c>{RegName, Node}</c></tag> - <item> - <p>A tuple consisting of a registered name of a process and - a node name. The process residing on the node <c>Node</c> - with the registered name <c>RegName</c> will be monitored.</p> - </item> - <tag><c>RegName</c></tag> - <item> - <p>The process locally registered as <c>RegName</c> will be - monitored.</p> - </item> - </taglist> - <note> - <p>When a process is monitored by registered name, the process - that has the registered name at the time when - <c>monitor/2</c> is called will be monitored. + <p>Send a monitor request of type <c><anno>Type</anno></c> to the + entity identified by <c><anno>Item</anno></c>. The caller of + <c>monitor/2</c> will later be notified by a monitor message on the + following format if the monitored state is changed:</p> + <code type="none">{Tag, <anno>MonitorRef</anno>, <anno>Type</anno>, Object, Info}</code> + <note><p>The monitor request is an asynchronous signal. That is, it + takes time before the signal reach its destination.</p></note> + <p>Currently valid <c><anno>Type</anno></c>s:</p> + <taglist> + <tag><marker id="monitor_process"/><c>process</c></tag> + <item> + <p>Monitor the existence of the process identified by + <c><anno>Item</anno></c>. Currently valid + <c><anno>Item</anno></c>s in combination with the + <c>process <anno>Type</anno></c>:</p> + <taglist> + <tag><c>pid()</c></tag> + <item> + <p>The process identifier of the process to monitor.</p> + </item> + <tag><c>{RegisteredName, Node}</c></tag> + <item> + <p>A tuple consisting of a registered name of a process and + a node name. The process residing on the node <c>Node</c> + with the registered name <c>{RegisteredName, Node}</c> will + be monitored.</p> + </item> + <tag><c>RegisteredName</c></tag> + <item> + <p>The process locally registered as <c>RegisteredName</c> + will become monitored.</p> + </item> + </taglist> + <note><p>When a process is monitored by registered name, the + process that has the registered name at the time when the + monitor request reach its destination will be monitored. The monitor will not be effected, if the registered name is - unregistered.</p> - </note> - <p>A <c>'DOWN'</c> message will be sent to the monitoring - process if <c><anno>Item</anno></c> dies, if <c><anno>Item</anno></c> does not exist, - or if the connection is lost to the node which <c><anno>Item</anno></c> - resides on. A <c>'DOWN'</c> message has the following pattern:</p> - <code type="none"> -{'DOWN', MonitorRef, Type, Object, Info}</code> - <p>where <c>MonitorRef</c> and <c>Type</c> are the same as - described above, and:</p> - <taglist> - <tag><c>Object</c></tag> - <item> - <p>A reference to the monitored object:</p> - <list type="bulleted"> - <item>the pid of the monitored process, if <c><anno>Item</anno></c> was - specified as a pid.</item> - <item><c>{RegName, Node}</c>, if <c><anno>Item</anno></c> was specified as - <c>{RegName, Node}</c>.</item> - <item><c>{RegName, Node}</c>, if <c><anno>Item</anno></c> was specified as - <c>RegName</c>. <c>Node</c> will in this case be the - name of the local node (<c>node()</c>).</item> - </list> - </item> - <tag><c>Info</c></tag> - <item> - <p>Either the exit reason of the process, <c>noproc</c> - (non-existing process), or <c>noconnection</c> (no - connection to <c><anno>Node</anno></c>).</p> - </item> - </taglist> - <note> - <p>If/when <c>monitor/2</c> is extended (e.g. to - handle other item types than <c>process</c>), other - possible values for <c>Object</c>, and <c>Info</c> in the - <c>'DOWN'</c> message will be introduced.</p> - </note> - <p>The monitoring is turned off either when the <c>'DOWN'</c> - message is sent, or when - <seealso marker="#demonitor/1">demonitor/1</seealso> - is called.</p> - <p>If an attempt is made to monitor a process on an older node - (where remote process monitoring is not implemented or one - where remote process monitoring by registered name is not - implemented), the call fails with <c>badarg</c>.</p> - <p>Making several calls to <c>monitor/2</c> for the same - <c><anno>Item</anno></c> is not an error; it results in as many, completely - independent, monitorings.</p> + unregistered, or unregistered and later registered on another + process.</p></note> + <p>The monitor is triggered either when the monitored process + terminates, is non existing, or if the connection to it is + lost. In the case the connection to it is lost, we do not know + if it still exist or not. After this type of monitor has been + triggered, the monitor is automatically removed.</p> + <p>When the monitor is triggered a <c>'DOWN'</c> message will + be sent to the monitoring process. A <c>'DOWN'</c> message has + the following pattern:</p> + <code type="none">{'DOWN', MonitorRef, Type, Object, Info}</code> + <p>where <c>MonitorRef</c> and <c>Type</c> are the same as + described above, and:</p> + <taglist> + <tag><c>Object</c></tag> + <item> + <p>equals:</p> + <taglist> + <tag><c><anno>Item</anno></c></tag> + <item>If <c><anno>Item</anno></c> was specified by a + pid.</item> + <tag><c>{RegisteredName, Node}</c></tag> + <item>If <c><anno>Item</anno></c> was specified as + <c>RegisteredName</c>, or <c>{RegisteredName, Node}</c> + where <c>Node</c> corresponds to the node that the + monitored process resides on.</item> + </taglist> + </item> + <tag><c>Info</c></tag> + <item> + <p>Either the exit reason of the process, <c>noproc</c> + (non-existing process), or <c>noconnection</c> (no + connection to the node where the monitored process + resides).</p></item> + </taglist> + <p>The monitoring is turned off either when the <c>'DOWN'</c> + message is sent, or when + <seealso marker="#demonitor/1">demonitor/1</seealso> + is called.</p> + <p>If an attempt is made to monitor a process on an older node + (where remote process monitoring is not implemented or one + where remote process monitoring by registered name is not + implemented), the call fails with <c>badarg</c>.</p> + <note> + <p>The format of the <c>'DOWN'</c> message changed in the 5.2 + version of the emulator (OTP release R9B) for monitor + <em>by registered name</em>. The <c>Object</c> element of + the <c>'DOWN'</c> message could in earlier versions + sometimes be the pid of the monitored process and sometimes + be the registered name. Now the <c>Object</c> element is + always a tuple consisting of the registered name and + the node name. Processes on new nodes (emulator version 5.2 + or greater) will always get <c>'DOWN'</c> messages on + the new format even if they are monitoring processes on old + nodes. Processes on old nodes will always get <c>'DOWN'</c> + messages on the old format.</p> + </note> + </item> + <tag><marker id="monitor_time_offset"/><c>time_offset</c></tag> + <item> + <p>Monitor changes in + <seealso marker="#time_offset/0">time offset</seealso> + between + <seealso marker="time_correction#Erlang_Monotonic_Time">Erlang + monotonic time</seealso> and + <seealso marker="time_correction#Erlang_System_Time">Erlang + system time</seealso>. There is only one valid + <c><anno>Item</anno></c> in combination with the + <c>time_offset <anno>Type</anno></c>, namely the atom + <c>clock_service</c>. Note that the atom <c>clock_service</c> is + <em>not</em> the registered name of a process. In this specific + case it serves as an identifier of the runtime system internal + clock service at current runtime system instance.</p> + + <p>The monitor is triggered when the time offset is changed. + This either if the time offset value is changed, or if the + offset is changed from preliminary to final during + <seealso marker="#system_flag_time_offset">finalization + of the time offset</seealso> when the + <seealso marker="time_correction#Single_Time_Warp_Mode">single + time warp mode</seealso> is used. When a change from preliminary + to final time offset is made, the monitor will be triggered once + regardless of whether the time offset value was changed due to + the finalization or not.</p> + + <p>If the runtime system is in + <seealso marker="time_correction#Multi_Time_Warp_Mode">multi + time warp mode</seealso>, the time offset will be changed when + the runtime system detects that the + <seealso marker="time_correction#OS_System_Time">OS system + time</seealso> has changed. The runtime system will, however, + not detect this immediately when it happens. A task checking + the time offset is scheduled to execute at least once a minute, + so under normal operation this should be detected within a + minute, but during heavy load it might take longer time.</p> + + <p>The monitor will <em>not</em> be automatically removed + after it has been triggered. That is, repeated changes of + the time offset will trigger the monitor repeatedly.</p> + + <p>When the monitor is triggered a <c>'CHANGE'</c> message will + be sent to the monitoring process. A <c>'CHANGE'</c> message has + the following pattern:</p> + <code type="none">{'CHANGE', MonitorRef, Type, Item, NewTimeOffset}</code> + <p>where <c>MonitorRef</c>, <c><anno>Type</anno></c>, and + <c><anno>Item</anno></c> are the same as described above, and + <c>NewTimeOffset</c> is the new time offset.</p> + + <p>When the <c>'CHANGE'</c> message has been received you are + guaranteed not to retrieve the old time offset when calling + <seealso marker="#time_offset/0"><c>erlang:time_offset()</c></seealso>. + Note that you may observe the change of the time offset + when calling <c>erlang:time_offset()</c> before you + get the <c>'CHANGE'</c> message.</p> + + </item> + </taglist> + <p>Making several calls to <c>monitor/2</c> for the same + <c><anno>Item</anno></c> and/or <c><anno>Type</anno></c> is not + an error; it results in many, completely independent, + monitorings.</p> + <p>The monitor functionality is expected to be extended. That is, + other <c><anno>Type</anno></c>s and <c><anno>Item</anno></c>s + are expected to be supported in the future.</p> <note> - <p>The format of the <c>'DOWN'</c> message changed in the 5.2 - version of the emulator (OTP release R9B) for monitor <em>by registered name</em>. The <c>Object</c> element of - the <c>'DOWN'</c> message could in earlier versions - sometimes be the pid of the monitored process and sometimes - be the registered name. Now the <c>Object</c> element is - always a tuple consisting of the registered name and - the node name. Processes on new nodes (emulator version 5.2 - or greater) will always get <c>'DOWN'</c> messages on - the new format even if they are monitoring processes on old - nodes. Processes on old nodes will always get <c>'DOWN'</c> - messages on the old format.</p> + <p>If/when <c>monitor/2</c> is extended, other + possible values for <c>Tag</c>, <c>Object</c>, and + <c>Info</c> in the monitor message will be introduced.</p> </note> </desc> </func> @@ -2654,6 +2823,51 @@ os_prompt% </pre> </desc> </func> <func> + <name name="monotonic_time" arity="0"/> + <fsummary>Current Erlang monotonic time</fsummary> + <desc> + <p>Returns the current + <seealso marker="time_correction#Erlang_Monotonic_Time">Erlang + monotonic time</seealso> in <c>native</c> + <seealso marker="#type_time_unit">time unit</seealso>. This + is a monotonically increasing time since some unspecified point in + time.</p> + + <note><p>This is a + <seealso marker="time_correction#Monotonically_Increasing">monotonically increasing</seealso> time, but <em>not</em> a + <seealso marker="time_correction#Strictly_Monotonically_Increasing">strictly monotonically increasing</seealso> + time. That is, consecutive calls to + <c>erlang:monotonic_time/0</c> may produce the same result.</p> + + <p>Different runtime system instances will use different + unspecified points in time as base for their Erlang monotonic clocks. + That is, it is <em>pointless</em> comparing monotonic times from + different runtime system instances. Different runtime system instances + may also place this unspecified point in time different relative + runtime system start. It may be placed in the future (time at start + will be a negative value), the past (time at start will be a + positive value), or the runtime system start (time at start will + be zero). The monotonic time as of runtime system start can be + retrieved by calling + <seealso marker="#system_info_start_time"><c>erlang:system_info(start_time)</c></seealso>.</p></note> + </desc> + </func> + <func> + <name name="monotonic_time" arity="1"/> + <fsummary>Current Erlang monotonic time</fsummary> + <desc> + <p>Returns the current + <seealso marker="time_correction#Erlang_Monotonic_Time">Erlang + monotonic time</seealso> converted + into the <c><anno>Unit</anno></c> passed as argument.</p> + + <p>Same as calling + <seealso marker="#convert_time_unit/3"><c>erlang:convert_time_unit</c></seealso><c>(</c><seealso marker="#monotonic_time/0"><c>erlang:monotonic_time()</c></seealso><c>, + native, <anno>Unit</anno>)</c> + however optimized for commonly used <c><anno>Unit</anno></c>s.</p> + </desc> + </func> + <func> <name name="nif_error" arity="1"/> <fsummary>Stop execution with a given reason</fsummary> <desc> @@ -2748,6 +2962,13 @@ os_prompt% </pre> <type name="timestamp"/> <fsummary>Elapsed time since 00:00 GMT</fsummary> <desc> + <warning><p><em>This function is deprecated! Do not use it!</em> + See the users guide chapter + <seealso marker="time_correction">Time and Time Correction</seealso> + for more information. Specifically the + <seealso marker="time_correction#Dos_and_Donts">Dos and Dont's</seealso> + section for information on what to use instead of <c>erlang:now/0</c>. + </p></warning> <p>Returns the tuple <c>{MegaSecs, Secs, MicroSecs}</c> which is the elapsed time since 00:00 GMT, January 1, 1970 (zero hour) on the assumption that the underlying OS supports this. @@ -2760,10 +2981,6 @@ os_prompt% </pre> <p>It can only be used to check the local time of day if the time-zone info of the underlying operating system is properly configured.</p> - <p>If you do not need the return value to be unique and - monotonically increasing, use - <seealso marker="kernel:os#timestamp/0">os:timestamp/0</seealso> - instead to avoid some overhead.</p> </desc> </func> <func> @@ -5510,6 +5727,35 @@ ok <p>Returns the old value of the flag.</p> </desc> </func> + <marker id="system_flag_time_offset"/> + <func> + <name name="system_flag" arity="2" clause_i="12"/> + <fsummary>Finalize the Time Offset</fsummary> + <desc> + <p>Finalizes the <seealso marker="#time_offset/0">time offset</seealso> + when the <seealso marker="time_correction#Single_Time_Warp_Mode">single + time warp mode</seealso> is being used. If another time warp mode than + the "single time warp mode" is used, the time offset state will be left + unchanged.</p> + <p>Returns the old state identifier. That is, if:</p> + <list> + <item><p><c>preliminary</c> is returned, finalization was + performed and the time offset is now final.</p></item> + + <item><p><c>final</c> is returned, the time offset was + already in the final state. This either due to another + <c>erlang:system_flag(time_offset, finalize)</c> call, or + due to the + <seealso marker="time_correction#No_Time_Warp_Mode">no + time warp mode</seealso> being used.</p></item> + + <item><p><c>volatile</c> is returned, the time offset + cannot be finalized due to the + <seealso marker="time_correction#Multi_Time_Warp_Mode">multi + time warp mode</seealso> being used.</p></item> + </list> + </desc> + </func> <func> <name name="system_info" arity="1" clause_i="1"/> <name name="system_info" arity="1" clause_i="2"/> @@ -5790,6 +6036,17 @@ ok <name name="system_info" arity="1" clause_i="53"/> <name name="system_info" arity="1" clause_i="54"/> <name name="system_info" arity="1" clause_i="55"/> + <name name="system_info" arity="1" clause_i="56"/> + <name name="system_info" arity="1" clause_i="57"/> + <name name="system_info" arity="1" clause_i="58"/> + <name name="system_info" arity="1" clause_i="59"/> + <name name="system_info" arity="1" clause_i="60"/> + <name name="system_info" arity="1" clause_i="61"/> + <name name="system_info" arity="1" clause_i="62"/> + <name name="system_info" arity="1" clause_i="63"/> + <name name="system_info" arity="1" clause_i="64"/> + <name name="system_info" arity="1" clause_i="65"/> + <name name="system_info" arity="1" clause_i="66"/> <fsummary>Information about the system</fsummary> <desc> <p>Returns various information about the current system @@ -6177,6 +6434,123 @@ ok documentation of versions in the system principles guide</seealso>.</p> </item> + <tag><marker id="system_info_os_monotonic_time_source"><c>os_monotonic_time_source</c></marker></tag> + <item> + <p>Returns a list containing information about the source of + <seealso marker="erts:time_correction#OS_Monotonic_Time">OS + monotonic time</seealso> that is used by the runtime system.</p> + <p>In case <c>[]</c> is returned, no OS monotonic time is + available. The list contains two-tuples with <c>Key</c>s + as first element, and <c>Value</c>s as second element. The + order if these tuples is undefined. Currently the following + tuples may be part of the list, but more tuples may be + introduced in the future:</p> + <taglist> + <tag><c>{function, Function}</c></tag> + <item><p><c>Function</c> is the name of the funcion + used. This tuple always exist if OS monotonic time is + available to the runtime system.</p></item> + + <tag><c>{clock_id, ClockId}</c></tag> + <item><p>This tuple only exist if <c>Function</c> + can be used with different clocks. <c>ClockId</c> + corresponds to the clock identifer used when calling + <c>Function</c>.</p></item> + + <tag><c>{resolution, OsMonotonicTimeResolution}</c></tag> + <item><p>Highest possible + <seealso marker="time_correction#Time_Resolution">resolution</seealso> + of current OS monotonic time source as parts per + second. If no resolution information can be retreived + from the OS, <c>OsMonotonicTimeResolution</c> will be + set to the resolution of the time unit of + <c>Function</c>s return value. That is, the actual + resolution may be lower than + <c>OsMonotonicTimeResolution</c>. Also note that + the resolution does not say anything about the + <seealso marker="time_correction#Time_Accuracy">accuracy</seealso>, + and that the + <seealso marker="time_correction#Time_Precision">precision</seealso> + might not align with the resolution. You do, + however, know that the precision won't be + better than + <c>OsMonotonicTimeResolution</c>.</p></item> + + <tag><c>{extended, Extended}</c></tag> + <item><p><c>Extended</c> equals <c>yes</c> if + the range of time values has been extended; + otherwise, <c>Extended</c> equals <c>no</c>. The + range needs to be extended if <c>Function</c> + returns values that wrap fast. This typically + is the case when the return value is a 32-bit + value.</p></item> + + <tag><c>{parallel, Parallel}</c></tag> + <item><p><c>Parallel</c> equals <c>yes</c> if + <c>Function</c> is called in parallel from multiple + threads. If it is not called in parallel, because + calls needs to be serialized, <c>Parallel</c> equals + <c>no</c>.</p></item> + + <tag><c>{time, OsMonotonicTime}</c></tag> + <item><p><c>OsMonotonicTime</c> equals current OS + monotonic time in <c>native</c> + <seealso marker="#type_time_unit">time unit</seealso>.</p></item> + </taglist> + </item> + <tag><marker id="system_info_os_system_time_source"><c>os_system_time_source</c></marker></tag> + <item> + <p>Returns a list containing information about the source of + <seealso marker="erts:time_correction#OS_System_Time">OS + system time</seealso> that is used by the runtime system.</p> + <p>The list contains two-tuples with <c>Key</c>s + as first element, and <c>Value</c>s as second element. The + order if these tuples is undefined. Currently the following + tuples may be part of the list, but more tuples may be + introduced in the future:</p> + <taglist> + <tag><c>{function, Function}</c></tag> + <item><p><c>Function</c> is the name of the funcion + used.</p></item> + + <tag><c>{clock_id, ClockId}</c></tag> + <item><p>This tuple only exist if <c>Function</c> + can be used with different clocks. <c>ClockId</c> + corresponds to the clock identifer used when calling + <c>Function</c>.</p></item> + + <tag><c>{resolution, OsSystemTimeResolution}</c></tag> + <item><p>Highest possible + <seealso marker="time_correction#Time_Resolution">resolution</seealso> + of current OS system time source as parts per + second. If no resolution information can be retreived + from the OS, <c>OsSystemTimeResolution</c> will be + set to the resolution of the time unit of + <c>Function</c>s return value. That is, the actual + resolution may be lower than + <c>OsSystemTimeResolution</c>. Also note that + the resolution does not say anything about the + <seealso marker="time_correction#Time_Accuracy">accuracy</seealso>, + and that the + <seealso marker="time_correction#Time_Precision">precision</seealso> + might not align with the resolution. You do, + however, know that the precision won't be + better than + <c>OsSystemTimeResolution</c>.</p></item> + + <tag><c>{parallel, Parallel}</c></tag> + <item><p><c>Parallel</c> equals <c>yes</c> if + <c>Function</c> is called in parallel from multiple + threads. If it is not called in parallel, because + calls needs to be serialized, <c>Parallel</c> equals + <c>no</c>.</p></item> + + <tag><c>{time, OsSystemTime}</c></tag> + <item><p><c>OsSystemTime</c> equals current OS + system time in <c>native</c> + <seealso marker="#type_time_unit">time unit</seealso>.</p></item> + </taglist> + </item> <tag><marker id="system_info_port_parallelism"><c>port_parallelism</c></marker></tag> <item><p>Returns the default port parallelism scheduling hint used. For more information see the @@ -6302,6 +6676,11 @@ ok <p>Returns <c>true</c> if the emulator has been compiled with smp support; otherwise, <c>false</c>.</p> </item> + <tag><marker id="system_info_start_time"/><c>start_time</c></tag> + <item><p>The <seealso marker="#monotonic_time/0">Erlang monotonic + time</seealso> in <c>native</c> + <seealso marker="#type_time_unit">time unit</seealso> at the + time when current Erlang runtime system instance started.</p></item> <tag><c>system_version</c></tag> <item> <p>Returns a string containing version number and @@ -6325,12 +6704,64 @@ ok (<seealso marker="erts:erl_driver#driver_async">driver_async()</seealso>) as an integer.</p> </item> + <tag><marker id="system_info_time_correction"/><c>time_correction</c></tag> + <item><p>Returns a boolean value indicating whether + <seealso marker="time_correction#Time_Correction">time correction</seealso> + is enabled or not. + </p></item> + <tag><marker id="system_info_time_offset"/><c>time_offset</c></tag> + <item><p>Returns the state of the time offset:</p> + <taglist> + <tag><c>preliminary</c></tag> + <item><p>The time offset is preliminary, and will be changed + at a later time when being finalized. The preliminary time offset + is used during the preliminary phase of the + <seealso marker="time_correction#Single_Time_Warp_Mode">single + time warp mode</seealso>.</p></item> + + <tag><c>final</c></tag> + <item><p>The time offset is final. This + either due to the use of the + <seealso marker="time_correction#No_Time_Warp_Mode">no + time warp mode</seealso>, or due to the time offset having + been finalized when using the + <seealso marker="time_correction#Single_Time_Warp_Mode">single + time warp mode</seealso>.</p></item> + + <tag><c>volatile</c></tag> + <item><p>The time offset is volatile. That is, it may + change at any time. This due to the + <seealso marker="time_correction#Multi_Time_Warp_Mode">multi + time warp mode</seealso> being used.</p></item> + </taglist> + </item> + <tag><marker id="system_info_time_warp_mode"/><c>time_warp_mode</c></tag> + <item><p>Returns a value identifying the + <seealso marker="time_correction#Time_Warp_Modes">time warp + mode</seealso> being used:</p> + <taglist> + <tag><c>no_time_warp</c></tag> + <item><p>The <seealso marker="time_correction#No_Time_Warp_Mode">no + time warp mode</seealso> is being used.</p></item> + + <tag><c>single_time_warp</c></tag> + <item><p>The <seealso marker="time_correction#Single_Time_Warp_Mode">single + time warp mode</seealso> is being used.</p></item> + + <tag><c>multi_time_warp</c></tag> + <item><p>The <seealso marker="time_correction#Multi_Time_Warp_Mode">multi + time warp mode</seealso> is being used.</p></item> + </taglist> + </item> <tag><marker id="system_info_tolerant_timeofday"><c>tolerant_timeofday</c></marker></tag> <item> - <p>Returns whether compensation for sudden changes of system - time is <c>enabled</c> or <c>disabled</c>.</p> - <p>See also <seealso marker="erts:erl#+c">+c</seealso> - command line flag.</p> + <p>Returns whether a pre erts-7.0 backwards compatible compensation + for sudden changes of system time is <c>enabled</c> or <c>disabled</c>. + Such compensation is <c>enabled</c> when the + <seealso marker="#system_info_time_offset">time offset</seealso> is + <c>final</c>, and + <seealso marker="#system_info_time_correction">time correction</seealso> + is enabled.</p> </item> <tag><c>trace_control_word</c></tag> <item> @@ -6609,7 +7040,44 @@ ok </note> </desc> </func> + <func> + <name name="system_time" arity="0"/> + <fsummary>Current Erlang system time</fsummary> + <desc> + <p>Returns current + <seealso marker="time_correction#Erlang_System_Time">Erlang system time</seealso> + in <c>native</c> + <seealso marker="#type_time_unit">time unit</seealso>.</p> + + <p>Calling <c>erlang:system_time()</c> is equivalent to: + <seealso marker="#monotonic_time/0"><c>erlang:monotonic_time()</c></seealso><c> + + + </c><seealso marker="#time_offset/0"><c>erlang:time_offset()</c></seealso>.</p> + + <note><p>This time is <em>not</em> a monotonically increasing time + in the general case. For more information, see the documentation of + <seealso marker="time_correction#Time_Warp_Modes">time warp modes</seealso> in the + ERTS User's Guide.</p></note> + </desc> + </func> + <func> + <name name="system_time" arity="1"/> + <fsummary>Current Erlang system time</fsummary> + <desc> + <p>Returns current + <seealso marker="time_correction#Erlang_System_Time">Erlang system time</seealso> + converted into the <c><anno>Unit</anno></c> passed as argument.</p> + + <p>Calling <c>erlang:system_time(<anno>Unit</anno>)</c> is equivalent to: + <seealso marker="#convert_time_unit/3"><c>erlang:convert_time_unit</c></seealso><c>(</c><seealso marker="#system_time/0"><c>erlang:system_time()</c></seealso><c>, + native, <anno>Unit</anno>)</c>.</p> + <note><p>This time is <em>not</em> a monotonically increasing time + in the general case. For more information, see the documentation of + <seealso marker="time_correction#Time_Warp_Modes">time warp modes</seealso> in the + ERTS User's Guide.</p></note> + </desc> + </func> <func> <name name="term_to_binary" arity="1"/> <fsummary>Encode a term to an Erlang external term format binary</fsummary> @@ -6686,6 +7154,88 @@ ok </desc> </func> <func> + <name name="time_offset" arity="0"/> + <fsummary>Current time offset</fsummary> + <desc> + <p>Returns the current time offset between + <seealso marker="time_correction#Erlang_Monotonic_Time">Erlang monotonic time</seealso> + and + <seealso marker="time_correction#Erlang_System_Time">Erlang system time</seealso> in + <c>native</c> <seealso marker="#type_time_unit">time unit</seealso>. + Current time offset added to an Erlang monotonic time gives + corresponding Erlang system time.</p> + + <p>The time offset may or may not change during operation depending + on the <seealso marker="time_correction#Time_Warp_Modes">time + warp mode</seealso> used.</p> + + <note> + <p>A change in time offset may be observed at slightly + different points in time by different processes.</p> + + <p>If the runtime system is in + <seealso marker="time_correction#Multi_Time_Warp_Mode">multi + time warp mode</seealso>, the time offset will be changed when + the runtime system detects that the + <seealso marker="time_correction#OS_System_Time">OS system + time</seealso> has changed. The runtime system will, however, + not detect this immediately when it happens. A task checking + the time offset is scheduled to execute at least once a minute, + so under normal operation this should be detected within a + minute, but during heavy load it might take longer time.</p> + </note> + </desc> + </func> + <func> + <name name="time_offset" arity="1"/> + <fsummary>Current time offset</fsummary> + <desc> + <p>Returns the current time offset between + <seealso marker="time_correction#Erlang_Monotonic_Time">Erlang monotonic time</seealso> + and + <seealso marker="time_correction#Erlang_System_Time">Erlang system time</seealso> + converted into the <c><anno>Unit</anno></c> passed as argument.</p> + + <p>Same as calling + <seealso marker="#convert_time_unit/3"><c>erlang:convert_time_unit</c></seealso><c>(</c><seealso marker="#time_offset/0"><c>erlang:time_offset()</c></seealso><c>, native, <anno>Unit</anno>)</c> + however optimized for commonly used <c><anno>Unit</anno></c>s.</p> + </desc> + </func> + <func> + <name name="timestamp" arity="0"/> + <type name="timestamp"/> + <fsummary>Current Erlang System time</fsummary> + <desc> + <p>Returns current + <seealso marker="time_correction#Erlang_System_Time">Erlang system time</seealso> + on the format <c>{MegaSecs, Secs, MicroSecs}</c>. This format is + the same that <seealso marker="kernel:os#timestamp/0"><c>os:timestamp/0</c></seealso> + and the now deprecated <seealso marker="#now/0"><c>erlang:now/0</c></seealso> + uses. The reason for the existence of <c>erlang:timestamp()</c> is + purely to simplify usage for existing code that assumes this timestamp + format. Current Erlang system time can more efficiently be retrieved in + the time unit of your choice using + <seealso marker="#system_time/1"><c>erlang:system_time/1</c></seealso>.</p> + + <p>The <c>erlang:timestamp()</c> BIF is equivalent to:</p><code type="none"> +timestamp() -> + ErlangSystemTime = erlang:system_time(micro_seconds), + MegaSecs = ErlangSystemTime div 1000000000000, + Secs = ErlangSystemTime div 1000000 - MegaSecs*1000000, + MicroSecs = ErlangSystemTime rem 1000000, + {MegaSecs, Secs, MicroSecs}.</code> + <p>It however use a native implementation which does + not build garbage on the heap and with slightly better + performance.</p> + + <note><p>This time is <em>not</em> a monotonically increasing time + in the general case. For more information, see the documentation of + <seealso marker="time_correction#Time_Warp_Modes">time warp modes</seealso> in the + ERTS User's Guide.</p></note> + </desc> + + </func> + <func> <name name="tl" arity="1"/> <fsummary>Tail of a list</fsummary> <desc> @@ -7452,6 +8002,100 @@ ok </desc> </func> <func> + <name name="unique_integer" arity="0"/> + <fsummary>Get a unique integer value</fsummary> + <desc> + <p>Generates and returns an + <seealso marker="doc/efficiency_guide:advanced#unique_integers">integer + unique on current runtime system instance</seealso>. The same as calling + <seealso marker="#unique_integer/1"><c>erlang:unique_integer([])</c></seealso>.</p> + </desc> + </func> + <func> + <name name="unique_integer" arity="1"/> + <fsummary>Get a unique integer value</fsummary> + <desc> + <p>Generates and returns an + <seealso marker="doc/efficiency_guide:advanced#unique_integers">integer + unique on current runtime system + instance</seealso>. The integer is unique in the + sense that this BIF, using the same set of + modifiers, will not return the same integer more + than once on the current runtime system instance. + Each integer value can of course be constructed + by other means.</p> + + <p>By default, i.e. when <c>[]</c> is passed as + <c><anno>ModifierList</anno></c>, both negative and + positive integers will be returned. This is order + to be able to utilize the range of integers that do + not need to be heap allocated as much as possible. + By default the returned integers are also only + guaranteed to be unique, i.e., any integer returned + may be either smaller, or larger than previously + returned integers.</p> + + <p>Currently valid <c><anno>Modifier</anno></c>s:</p> + <taglist> + + <tag>positive</tag> + <item><p>Return only positive integers.</p> + <p>Note that by passing the <c>positive</c> modifier + you will get heap allocated integers (big-nums) + quicker.</p> + </item> + + <tag>monotonic</tag> + <item><p>Return + <seealso marker="time_correction#Strictly_Monotonically_Increasing">strictly + monotonically increasing</seealso> integers + corresponding to creation time. That is, the integer + returned will always be larger than previously + returned integers on the current runtime system + instance.</p> + <p>These values can be used when ordering events + on the runtime system instance. That is, if both + <c>X = erlang:unique_integer([monotonic])</c> and + <c>Y = erlang:unique_integer([monotonic])</c> are + executed by different processes (or the same + process) on the same runtime system instance and + <c>X < Y</c> we know that <c>X</c> was created + before <c>Y</c>.</p> + <warning><p>Strictly monotonically increasing values + are inherently quite expensive to generate and scales + poorly. This since the values needs to be + synchronized. That is, do not pass the <c>monotonic</c> + modifier unless you really need strictly monotonically + increasing values.</p></warning> + </item> + + </taglist> + + <p>All currently valid <c><anno>Modifier</anno></c>s + can be combined. Repeated (valid) + <c><anno>Modifier</anno></c>s in the <c>ModifierList</c> + are ignored.</p> + + <note><p>Note that the set of integers returned by + <c>unique_integer/1</c> using diffrent sets of + <c><anno>Modifier</anno></c>s <em>will overlap</em>. + For example, by calling <c>unique_integer([monotonic])</c>, + and <c>unique_integer([positive, monotonic])</c> + repeatedly, you will eventually see some integers being + returned by both calls.</p></note> + + <p>Failures:</p> + <taglist> + <tag><c>badarg</c></tag> + <item>if <c><anno>ModifierList</anno></c> is not a + proper list.</item> + <tag><c>badarg</c></tag> + <item>if <c><anno>Modifier</anno></c> is not a + valid modifier.</item> + </taglist> + </desc> + </func> + <func> <name name="unlink" arity="1"/> <fsummary>Remove a link, if there is one, to another process or port</fsummary> <desc> diff --git a/erts/doc/src/time_correction.xml b/erts/doc/src/time_correction.xml index 7f7c28fc30..979a37d7ff 100644 --- a/erts/doc/src/time_correction.xml +++ b/erts/doc/src/time_correction.xml @@ -21,8 +21,8 @@ </legalnotice> - <title>Time and time correction in Erlang</title> - <prepared>Patrik Nyblom</prepared> + <title>Time and Time Correction in Erlang</title> + <prepared></prepared> <responsible></responsible> <docno></docno> <approved></approved> @@ -31,6 +31,205 @@ <rev>PA1</rev> <file>time_correction.xml</file> </header> + + <section> + <title>New Extended Time Functionality</title> + <note><p>As of OTP 18 (ERTS version 7.0) the time functionality of + Erlang has been extended. This both includes a + <seealso marker="#The_New_Time_API">new API</seealso> + for time, as well as + <seealso marker="#Time_Warp_Modes">time warp + modes</seealso> which alters the behavior of the system when + system time changes.</p> + <p>The <seealso marker="#No_Time_Warp_Mode">default + time warp mode</seealso> has the same behavior as before, and the + old API will still work, so you are not required to change + anything unless you want to. However, <em>you are strongly + encouraged to use the new API</em> instead of the old API based + on <seealso marker="erlang#now/0"><c>erlang:now/0</c></seealso>. + <c>erlang:now/0</c> has been deprecated since it is and forever + will be a scalability bottleneck. By using the new API you will + automatically get scalability and performance improvements. This + will also enable you to use the + <seealso marker="#Multi_Time_Warp_Mode">multi time warp mode</seealso> + which improves accuracy, and precision of time measurements.</p></note> + </section> + + <section> + <title>Some Terminology</title> + <p>In order to make it easier to understand this document we first + define some terminology. This is a mixture of our own terminology + (Erlang/OS system time, Erlang/OS monotonic time, time warp) + and globally accepted terminology.</p> + + <marker id="Monotonically_Increasing"/> + <section> + <title>Monotonically Increasing</title> + <p>In a monotonically increasing sequence of values, all values + that have a predecessor are either larger than, or equal to its + predecessor.</p> + </section> + + <marker id="Strictly_Monotonically_Increasing"/> + <section> + <title>Strictly Monotonically Increasing</title> + <p>In a strictly monotonically increasing sequence of values, + all values that have a predecessor are larger than its + predecessor.</p> + </section> + + <marker id="UT1"/> + <section> + <title>UT1</title> + <p>Universal Time. Based on the rotation of the earth. Conceptually + mean solar time at 0° longitude.</p> + </section> + + <marker id="UTC"/> + <section> + <title>UTC</title> + <p>Coordinated Universal Time. UTC almost align with + <seealso marker="#UT1">UT1</seealso>, however, UTC uses the + SI definition of a second which is not exactly of the same length + as the second used by UT1. This means that UTC slowly drifts from + UT1. In order to keep UTC relatively in sync with UT1, leap seconds + are inserted, and potentially also deleted. That is, an UTC day may + be 86400, 86401, or 86399 seconds long.</p> + </section> + + <marker id="POSIX_Time"/> + <section> + <title>POSIX Time</title> + <p>Time since + <url href="http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html#tag_21_03_00_17">Epoch</url>. + Epoch is defined to be 00:00:00 <seealso marker="#UTC">UTC</seealso>, + January 1, 1970. + <url href="http://pubs.opengroup.org/onlinepubs/009604499/basedefs/xbd_chap04.html#tag_04_14">A day in POSIX time</url> + is defined to be exactly 86400 seconds long. Strangely enough + Epoch is defined to be a time in UTC, and UTC have another + definition of how long a day is. Quoting the Open Group + <url href="http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap04.html#tag_21_04_15">"POSIX time is therefore not necessarily UTC, despite its appearance"</url>. The effect of this is that when an UTC leap second is + inserted, POSIX time either stops for a second, or repeats the + last second. If an UTC leap second would be deleted (has never + happened yet), POSIX time would make a one second leap forward.</p> + </section> + + <marker id="Time_Resolution"/> + <section> + <title>Time Resolution</title> + <p>The shortest time interval that can be distinguished when + reading time values.</p> + </section> + + <marker id="Time_Precision"/> + <section> + <title>Time Precision</title> + <p>The shortest time interval that can be be distinguished + repeatedly and reliably when reading time values. Precision + is limited by the + <seealso marker="#Time_Resolution">resolution</seealso>, but + resolution and precision might differ significantly.</p> + </section> + + <marker id="Time_Accuracy"/> + <section> + <title>Time Accuracy</title> + <p>The correctness of time values.</p> + </section> + + <marker id="OS_System_Time"/> + <section> + <title>OS System Time</title> + <p>The operating systems view of + <seealso marker="#POSIX_Time">POSIX time</seealso>. It can be + retrieved by calling + <seealso marker="kernel:os#system_time/0"><c>os:system_time()</c></seealso>. + This may or may not be an accurate view of POSIX time. This time + may typically be adjusted both backwards and forwards without + limitation. That is, huge leaps both backwards and forwards in time + may be observed. You can get information about the Erlang runtime + system's source of OS system time by calling + <seealso marker="erlang#system_info_os_system_time_source"><c>erlang:system_info(os_system_time_source)</c></seealso>.</p> + </section> + + <marker id="OS_Monotonic_Time"/> + <section> + <title>OS Monotonic Time</title> + <p>A monotonically increasing time provided by the operating + system. This time does not leap and have a relatively steady + frequency although not completely correct. However, it is not + uncommon that the OS monotonic time stops if the system is + suspended. This time typically increase since some unspecified + point in time that is not connected to + <seealso marker="#OS_System_Time">OS system time</seealso>. Note that + this type of time is not necessarily provided by all operating + systems. You can get information about the Erlang runtime + system's source of OS monotonic time by calling + <seealso marker="erlang#system_info_os_monotonic_time_source"><c>erlang:system_info(os_monotonic_time_source)</c></seealso>.</p> + </section> + + <marker id="Erlang_System_Time"/> + <section> + <title>Erlang System Time</title> + <p>The Erlang runtime systems view of + <seealso marker="#POSIX_Time">POSIX time</seealso>. It can be + retrieved by calling + <seealso marker="erlang#system_time/0"><c>erlang:system_time()</c></seealso>. + This time may or may not be an accurate view of POSIX time, and may + or may not align with <seealso marker="#OS_System_Time">OS system + time</seealso>. The <seealso marker="#Time_Warp_Modes">time + warp mode</seealso> determines how it behaves when OS system + time suddenly change.</p> + </section> + + <marker id="Erlang_Monotonic_Time"/> + <section> + <title>Erlang Monotonic Time</title> + <p>A monotonically increasing time provided by the + Erlang runtime system. The Erlang monotonic time increase since + some unspecified point in time. It can be retrieved by calling + <seealso marker="erlang#monotonic_time/0"><c>erlang:monotonic_time()</c></seealso>. + The + <seealso marker="#Time_Accuracy">accuracy</seealso>, and + <seealso marker="#Time_Precision">precision</seealso> of Erlang + monotonic time heavily depends on the accuracy and precision of + <seealso marker="#OS_Monotonic_Time">OS monotonic time</seealso>, + the accuracy and precision of + <seealso marker="#OS_System_Time">OS system time</seealso> as well + as on the + <seealso marker="#Time_Warp_Modes">time warp mode</seealso> + used. On a system that is lacking OS monotonic time, the Erlang + monotonic time can only guarantee monotonicity and can more or less + not give any other guarantees. The frequency adjustments made to + the Erlang monotonic time depends on the time warp mode + used.</p> + + <p>Internally in the runtime system the Erlang monotonic + time is the "time engine" that is used for more or less + everything that has anything to do with time. All timers + regardless of it is a <c>receive ... after</c> timer, BIF timer, + or a timer in the <c>timer</c> module are triggered + relative Erlang monotonic time. Even + <seealso marker="#Erlang_System_Time">Erlang system + time</seealso> is based on Erlang monotonic time. + By adding current Erlang monotonic time with current time + offset you get current Erlang system time. Current time + offset can be retrieved by calling + <seealso marker="erlang#time_offset/0"><c>erlang:time_offset/0</c></seealso>. + </p> + </section> + + <marker id="Time_Warp"/> + <section> + <title>Time Warp</title> + <p>A time warp is a leap forwards or backwards in time.</p> + </section> + + </section> + + <section> + <title>Introduction</title> + <p>Time is vital to an Erlang program and, more importantly, <em>correct</em> time is vital to an Erlang program. As Erlang is a language with soft real time properties and we have the possibility to express @@ -83,192 +282,589 @@ microsecond resolution or much less, but generally it has a drift that is not to be ignored.</p> - <p>So we have this monotonic ticking and we have the wall clock - time. Two unreliable times that together can give us an estimate of - an actual wall clock time that does not jump around and that - monotonically moves forward. If the tick counter has a high - resolution, this is fairly easy to do, if the counter has a low - resolution, it's more expensive, but still doable down to - frequencies of 50-60 Hz (of the tick counter).</p> - - <p>So the corrected time is the nearest approximation of an atomic - clock that is available on the computer. We want it to have the - following properties:</p> - <taglist> - <tag>Monotonic</tag> - <item>The clock should not move backwards</item> - <tag>Intervals should be near the truth</tag> - <item>We want the actual time (as measured by an atomic clock or - an astronomer) that passes between two time stamps, T1 and T2, to be as - near to T2 - T1 as possible.</item> - <tag>Tight coupling to the wall clock</tag> - <item>We want a timer that is to be fired when the wall clock - reaches a time in the future, to fire as near to that point in - time as possible</item> - </taglist> - <p>To meet all the criteria, we have to utilize both times in such a - way that Erlangs "corrected time" moves slightly slower or slightly - faster than the wall clock to get in sync with it. The word - "slightly" means a maximum of 1% difference to the wall clock time, - meaning that a sudden change in the wall clock of one minute, takes - 100 minutes to fix, by letting all "corrected time" move 1% slower - or faster.</p> - - <p>Needless to say, correcting for a faulty handling of daylight - saving time may be disturbing to a user comparing wall clock - time to for example calendar:now_to_local_time(erlang:now()). But - calendar:now_to_local_time/1 is not supposed to be used for presenting wall - clock time to the user.</p> - - <p>Time correction is not perfect, but it saves you from the havoc - of clocks jumping around, which would make timers in your program - fire far to late or far to early and could bring your whole system - to it's knees (or worse) just because someone detected a small error - in the wall clock time of the server where your program runs. So - while it might be confusing, it is still a really good feature of - Erlang and you should not throw it away using time functions which - may give you higher benchmark results, not unless you really know - what you're doing.</p> + </section> + <marker id="Time_Correction"/> <section> - <title>What does time correction mean in my system?</title> - <p>Time correction means that Erlang estimates a time from current - and previous settings of the wall clock, and it uses a fairly - exact tick counter to detect when the wall clock time has jumped - for some reason, slowly adjusting to the new value.</p> - - <p>In practice, this means that the difference between two calls - to time corrected functions, like erlang:now(), might differ up to - one percent from the corresponding calls to non time corrected - functions (like os:timestamp()). Furthermore, if comparing - calendar:local_time/0 to calendar:now_to_local_time(erlang:now()), - you might temporarily see a difference, depending on how well kept your - system is.</p> - - <p>It is important to understand that it is (to the program) - always unknown if it is the wall clock time that moves in the - wrong pace or the Erlang corrected time. The only way to determine - that, is to have an external source of universally correct time. If - some such source is available, the wall clock time can be kept - nearly perfect at all times, and no significant difference will be - detected between erlang:now/0's pace and the wall clock's.</p> - - <p>Still, the time correction will mean that your system keeps - it's real time characteristics very well, even when the wall clock - is unreliable.</p> + <title>Time Correction</title> + <p>If time correction is enabled, the Erlang runtime system + will make use of both + <seealso marker="#OS_System_Time">OS system time</seealso> + and <seealso marker="#OS_Monotonic_Time">OS monotonic time</seealso>, + in order to make adjustments of the frequency of the Erlang + monotonic clock. Time correction will ensure that + <seealso marker="#Erlang_Monotonic_Time">Erlang monotonic time</seealso> + will not warp, and that the frequency is relatively accurate. + The type of adjustments made to the frequency depends on the + time warp mode used. This will be discussed in more details in + the <seealso marker="#Time_Warp_Modes">time warp modes</seealso> + section below.</p> + + <p>By default time correction will be enabled if support for + it on the specific platform exist. Support for it includes + both an OS monotonic time provided by the OS, and an + implementation in the Erlang runtime system utilizing the + OS monotonic time. You can check if your system has support + for OS monotonic time by calling + <seealso marker="erlang#system_info_os_monotonic_time_source"><c>erlang:system_info(os_monotonic_time_source)</c></seealso>, + and you can check if time correction is enabled on your + system by calling + <seealso marker="erlang#system_info_time_correction"><c>erlang:system_info(time_correction)</c></seealso>.</p> + + <p>Time correction is enabled or disabled by passing the + <seealso marker="erl#+c"><c>+c [true|false]</c></seealso> + command line argument to <c>erl</c>.</p> + + <p>If time correction is disabled, Erlang monotonic time + may warp forwards, it may stop and even freeze for extended + periods of time, and there are no guarantees that the frequency + of the Erlang monotonic clock is accurate or stable.</p> + + <p><em>You typically never want to disable time correction</em>. + Previously there was a performance penalty associated with time + correction, but nowadays it is most often the other way around. + By disabling time correction you are likely to get bad scalability, + bad performance, and bad time measurements.</p> </section> + + + <marker id="Time_Warp_Safe_Code"/> <section> - <title>Where does Erlang use corrected time?</title> - <p>For all functionality where real time characteristics are - desirable, time correction is used. This basically means:</p> - <taglist> - <tag>erlang:now/0</tag> - <item>The infamous erlang:now/0 function uses time correction so - that differences between two "now-timestamps" will correspond to - other timeouts in the system. erlang:now/0 also holds other - properties, discussed later.</item> - <tag>receive ... after</tag> - <item>Timeouts on receive uses time correction to determine a - stable timeout interval.</item> - <tag>The timer module</tag> - <item>As the timer module uses other built in functions which - deliver corrected time, the timer module itself works with - corrected time.</item> - <tag>erlang:start_timer/3 and erlang:send_after/3</tag> - <item>The timer BIF's work with corrected time, so that they - will not fire prematurely or too late due to changes in the wall - clock time.</item> - </taglist> - - <p>All other functionality in the system where erlang:now/0 or any - other time corrected functionality is used, will of course - automatically benefit from it, as long as it's not "optimized" to - use some other time stamp function (like os:timestamp/0).</p> - - <p>Modules like calendar and functions like erlang:localtime/0 use - the wall clock time as it is currently set on the system. They - will not use corrected time. However, if you use a now-value and - convert it to local time, you will get a corrected local time - value, which may or may not be what you want. Typically older code - tend to use erlang:now/0 as a wall clock time, which is usually - correct (at least when testing), but might surprise you when - compared to other times in the system.</p> + <title>Time Warp Safe Code</title> + <p>Time warp safe code is code that is able to handle + a time warp of + <seealso marker="#Erlang_System_Time">Erlang system time</seealso>. + </p> + + <p><seealso marker="erlang#now/0"><c>erlang:now/0</c></seealso> + behaves very bad when Erlang system time warps. When Erlang + system time do a time warp backwards, the values returned + from <c>erlang:now/0</c> will freeze (if you disregard the + micro second increments made due to the actual call) until + OS system time reach the point of the last value returned by + <c>erlang:now/0</c>. This freeze might continue for very + long periods of time. It might take years, decades, + and even longer than this until the freeze stops.</p> + + <p>All uses of <c>erlang:now/0</c> are not necessarily + time warp unsafe. If you do not use it to get time, it + will be time warp safe. However <em>all uses of + <c>erlang:now/0</c> are suboptimal</em> from a performance + and scalability perspective. So you really want to replace + the usage of it with other functionality. For examples + of how to replace the usage of <c>erlang:now/0</c>, + see the <seealso marker="#Dos_and_Donts">Dos and Donts</seealso> + section.</p> </section> + + <marker id="Time_Warp_Modes"/> <section> - <title>What is erlang:now/0 really?</title> - <p>erlang:now/0 is a function designed to serve multiple purposes - (or a multi-headed beast if you're a VM designer). It is expected - to hold the following properties:</p> - <taglist> - <tag>Monotonic</tag> - <item>erlang:now() never jumps backwards - it always moves - forward</item> - <tag>Interval correct</tag> - <item>The interval between two erlang:now() calls is expected to - correspond to the correct time in real life (as defined by an - atomic clock, or better)</item> - <tag>Absolute correctness</tag> - <item>The erlang:now/0 value should be possible to convert to an - absolute and correct date-time, corresponding to the real world - date and time (the wall clock)</item> - <tag>System correspondence</tag> - <item>The erlang:now/0 value converted to a date-time is - expected to correspond to times given by other programs on the - system (or by functions like os:timestamp/0)</item> - <tag>Unique</tag> - <item>No two calls to erlang:now on one Erlang node should - return the same value</item> - </taglist> - <p>All these requirements are possible to uphold at the same - time if (and only if):</p> - <taglist> - <tag>The wall clock time of the system is perfect</tag> - <item>The system (Operating System) time needs to be perfectly - in sync with the actual time as defined by an atomic clock or - a better time source. A good installation using NTP, and that is - up to date before Erlang starts, will have properties that for - most users and programs will be near indistinguishable from the - perfect time. Note that any larger corrections to the time done - by hand, or after Erlang has started, will partly (or - temporarily) invalidate some of the properties, as the time is - no longer perfect.</item> - <tag>Less than one call per microsecond to erlang:now/0 is - done</tag> - <item>This means that at <em>any</em> microsecond interval in - time, there can be no more than one call to erlang:now/0 in the - system. However, for the system not to loose it's properties - completely, it's enough that it on average is no more than one - call per microsecond (in one Erlang node).</item> - </taglist> - <p>The uniqueness property of erlang:now/0 is the most limiting - property. It means that erlang:now() maintains a global state and - that there is a hard-to-check property of the system that needs to - be maintained. For most applications this is still not a problem, - but a future system might very well manage to violate the - frequency limit on the calls globally. The uniqueness property is - also quite useless, as there are globally unique references that - provide a much better unique value to programs. However the - property will need to be maintained unless a really subtle - backward compatibility issue is to be introduced.</p> + <title>Time Warp Modes</title> + + <p>Current <seealso marker="#Erlang_System_Time">Erlang system + time</seealso> is determined by adding current + <seealso marker="erlang#monotonic_time/0">Erlang monotonic time</seealso> + with current + <seealso marker="erlang#time_offset/0">time offset</seealso>. The + time offset is managed differently depending on which time + warp mode you use. The time warp mode is set by passing the + <seealso marker="erl#+C_"><c>+C + [no_time_warp|single_time_warp|multi_time_warp]</c></seealso> + command line argument to <c>erl</c>.</p> + + <marker id="No_Time_Warp_Mode"/> + <section> + <title>No Time Warp Mode</title> + <p>The time offset is determined at runtime system start + and will after this not change. This is the default behavior. + Not because it is the best mode (which it isn't). It is + default only because this is how the runtime system always + has behaved until ERTS version 7.0, and you have to ensure + that your Erlang code that may execute during a time warp is + <seealso marker="#Time_Warp_Safe_Code">time warp safe</seealso> + before you can enable other modes.</p> + + <p>Since the time offset is not allowed to change, time + correction needs to adjust the frequency of the Erlang + monotonic clock in order to smoothly align Erlang system + time with OS system time. A big downside of this approach + is that we on purpose will use a faulty frequency on the + Erlang monotonic clock if adjustments are needed. This + error may be as big as 1%. This error will show up in all + time measurements in the runtime system.</p> + + <p>If time correction is not enabled, the Erlang monotonic + time will freeze when the OS system time leap backwards. + The freeze of the monotonic time will continue until + OS system time catch up. The freeze may continue for + a very long time. When OS system time leaps forwards, + Erlang monotonic time will also leap forward.</p> + </section> + + <marker id="Single_Time_Warp_Mode"/> + <section> + <title>Single Time Warp Mode</title> + <p>This mode is more or less a backwards compatibility mode + as of its introduction.</p> + <p>On an embedded system it is not uncommon that the system + has no power supply at all, not even a battery, when it is + shut off. The system clock on such a system will typically + be way off when the system boots. If the + <seealso marker="#No_Time_Warp_Mode">no time warp mode</seealso> + is used, and the Erlang runtime system is started before + the OS system time has been corrected, the Erlang system + time may be wrong for a very long time, even centuries or + more.</p> + <p>If you for some reason need to use Erlang code that + is not + <seealso marker="#Time_Warp_Safe_Code">time warp safe</seealso>, + and you need to start the Erlang runtime system before the OS + system time has been corrected, you may want to use the single + time warp mode. Note that there are limitations to when you can + execute time warp unsafe code using this mode. If it is possible + to only utilize time warp safe code, it is much better to use + the <seealso marker="#Multi_Time_Warp_Mode">multi time warp + mode</seealso> instead. + </p> + + <p>Using the single time warp mode, the time offset is + handled in two phases:</p> + + <taglist> + <tag>Preliminary Phase</tag> + <item> + <p>The preliminary phase starts when the runtime + system starts. A preliminary time offset based on + current OS system time is determined. This offset will + from now on be fixed during the whole preliminary phase.</p> + + <p>If time correction is enabled, the Erlang + monotonic clock will only use the OS monotonic time as + time source during this phase. That is, during the + preliminary phase changes in OS system time will have + no effect on Erlang system time and/or Erlang + monotonic time what so ever.</p> + + <p>If time correction is disabled, changes in OS system + time will effect the monotonic clock the same way as + when the <seealso marker="#No_Time_Warp_Mode">no time warp + mode</seealso> is used.</p> + </item> + + <tag>Final Phase</tag> + <item> + + <p>The final phase begin when the user finalize the time + offset by calling + <seealso marker="erlang#system_flag_time_offset"><c>erlang:system_flag(time_offset, finalize)</c></seealso>. + The finalization can only be performed once. + </p> + + <p>During finalization, the time offset is adjusted and + fixated so that current Erlang system time align with + current OS system time. Since the time offset + may be changed, the Erlang system time may do + a time warp at this point. The time offset will from + now on be fixed until the runtime system terminates. + If time correction has been enabled, the time correction + also begins when this phase begins. When the system is + in the final phase it behaves exactly as in the + <seealso marker="#No_Time_Warp_Mode">no time warp + mode</seealso>.</p> + + </item> + </taglist> + + <p>In order for this to work properly there are two + requirements that the user needs to ensure are + satisfied:</p> + + <taglist> + <tag>Forward Time Warp</tag> + <item><p>The time warp made when finalizing the time offset + can only be done forwards without encountering problems. + This implies that the user has to ensure that the OS + system time is set to a time earlier or equal to actual + POSIX time before starting the Erlang runtime system. If + you are not completely sure the OS system time is correct, + set it to a time that is guaranteed to be earlier than + actual POSIX time before starting the Erlang runtime + system just to be safe.</p></item> + + <tag>Finalize Correct OS System Time</tag> + <item><p>The OS system time needs to be correct when the + the user finalizes the time offset.</p></item> + </taglist> + + <p>If these requirements are not fulfilled, the system + may behave very bad. + </p> + + <p>Assuming that the requirements above are fulfilled, + time correction is enabled, and that the OS system time + is adjusted using some time adjustment protocol like NTP + or similar, only small adjustments of the Erlang monotonic + time should be needed in order to keep system times + aligned after finilization. As long as the system is not + suspended, the largest adjustments needed should be for + inserted (or deleted) leap seconds.</p> + + <warning><p>In order to be able to use this mode you have + to ensure that all Erlang code that will execute in + both phases are + <seealso marker="#Time_Warp_Safe_Code">time warp + safe</seealso>.</p> + <p>Code that only execute in the final phase does not have + to be able to cope with the time warp.</p></warning> + + </section> + + <marker id="Multi_Time_Warp_Mode"/> + <section> + <title>Multi Time Warp Mode</title> + + <p><em>Multi time warp mode in combination with time + correction is the preferred configuration</em>. This since, + on almost all platforms, the Erlang runtime system will have + better performance, will scale better, will behave better, + and since the accuracy, and precision of time measurements + will be better. Only Erlang runtime systems executing on + ancient platforms will benefit from another configuration.</p> + + <p>The time offset may change at any time without limitations. + That is, Erlang system time may perform time warps both + forwards and backwards at <em>any</em> time. Since we align + the Erlang system time with the OS system time by changing + the time offset, we can enable a time correction that tries + to adjust the frequency of the Erlang monotonic clock to be as + correct as possible. This will make time measurements using + the Erlang monotonic time more accurate and precise.</p> + + <p>If time correction is disabled, Erlang monotonic time + will leap forward if OS system time leaps forward. If the + OS system time leaps backwards, Erlang monotonic time will + stop briefly but it does not freeze for extended periods + of time. This since the time offset is changed in order to + align Erlang system time with OS system time.</p> + + <warning><p>In order to be able to use this mode you have + to ensure that all Erlang code that will execute on the + runtime system is + <seealso marker="#Time_Warp_Safe_Code">time warp + safe</seealso>.</p></warning> + </section> </section> + + <marker id="The_New_Time_API"/> <section> - <title>Should I use erlang:now/0 or os:timestamp/0</title> - <p>The simple answer is to use erlang:now/0 for everything where - you want to keep real time characteristics, but use os:timestamp - for things like logs, user communication and debugging (typically - timer:ts uses os:timestamp, as it is a test tool, not a real world - application API). The benefit of using os:timestamp/0 is that it's - faster and does not involve any global state (unless the operating - system has one). The downside is that it will be vulnerable to wall - clock time changes.</p> + <title>The New Time API</title> + + <p>The old time API is based on + <seealso marker="erlang#now/0"><c>erlang:now/0</c></seealso>. + The major issue with <c>erlang:now/0</c> is that it was + intended to be used for so many unrelated things. This + tied these unrelated operations together and unnecessarily + caused performance, scalability as well as accuracy, and + precision issues for operations that do not need to have + such issues. The new API spreads different functionality + over multiple functions in order to improve on this.</p> + + <p>In order to be backwards compatible <c>erlang:now/0</c> will + remain as is, but <em>you are strongly discouraged from using + it</em>. A lot of uses of <c>erlang:now/0</c> will also + prevent you from using the new + <seealso marker="#Multi_Time_Warp_Mode">multi time warp + mode</seealso> which is an important part of this + new time functionality improvement.</p> + + <p>Some of the new BIFs on some systems, perhaps surprisingly, + return negative integer values on a newly started run time + system. This is not a bug, but a memory usage optimization.</p> + + <p>The new API consists of a number of new BIFs:</p> + <list> + <item><p><seealso marker="erlang#convert_time_unit/3"><c>erlang:convert_time_unit/3</c></seealso></p></item> + <item><p><seealso marker="erlang#monotonic_time/0"><c>erlang:monotonic_time/0</c></seealso></p></item> + <item><p><seealso marker="erlang#monotonic_time/1"><c>erlang:monotonic_time/1</c></seealso></p></item> + <item><p><seealso marker="erlang#system_time/0"><c>erlang:system_time/0</c></seealso></p></item> + <item><p><seealso marker="erlang#system_time/1"><c>erlang:system_time/1</c></seealso></p></item> + <item><p><seealso marker="erlang#time_offset/0"><c>erlang:time_offset/0</c></seealso></p></item> + <item><p><seealso marker="erlang#time_offset/1"><c>erlang:time_offset/1</c></seealso></p></item> + <item><p><seealso marker="erlang#timestamp/0"><c>erlang:timestamp/0</c></seealso></p></item> + <item><p><seealso marker="erlang#unique_integer/0"><c>erlang:unique_integer/0</c></seealso></p></item> + <item><p><seealso marker="erlang#unique_integer/1"><c>erlang:unique_integer/1</c></seealso></p></item> + <item><p><seealso marker="kernel:os#system_time/0"><c>os:system_time/0</c></seealso></p></item> + <item><p><seealso marker="kernel:os#system_time/1"><c>os:system_time/1</c></seealso></p></item> + </list> + <p>and a number of extensions of existing BIFs:</p> + <list> + <item><p><seealso marker="erlang#monitor/2"><c>erlang:monitor(time_offset, clock_service)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_flag_time_offset"><c>erlang:system_flag(time_offset, finalize)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_os_monotonic_time_source"><c>erlang:system_info(os_monotonic_time_source)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_os_system_time_source"><c>erlang:system_info(os_system_time_source)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_time_offset"><c>erlang:system_info(time_offset)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_time_warp_mode"><c>erlang:system_info(time_warp_mode)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_time_correction"><c>erlang:system_info(time_correction)</c></seealso></p></item> + <item><p><seealso marker="erlang#system_info_start_time"><c>erlang:system_info(start_time)</c></seealso></p></item> + </list> + + <marker id="The_New_Erlang_Monotonic_Time"/> + <section> + <title>The New Erlang Monotonic Time</title> + <p>The Erlang monotonic time as such is new as of ERTS + version 7.0. It has been introduced in order to be able + to detach time measurements such as elapsed time from + calender time. It is very common that one is interested + in measuring elapsed time or specifying a time relative + to another point in time without having any need to know + what the involved times are in UTC or any other + globally defined time scale. By introducing a time scale + that has a local definition of where it starts, it is + possible to manage time that do not concern calender + time on that time scale. Erlang monotonic time use + such a time scale with a locally defined start.</p> + + <p>The introduction of Erlang monotonic time gives us + the possibility to adjust the two Erlang times (Erlang + monotonic time and Erlang system time) separately. By + doing this, accuracy of elapsed time does not have to + suffer just because the system time happened to be + wrong at some point in time. Separate adjustments + of the two times are only performed in the time warp + modes, and only fully separated in the + <seealso marker="#Multi_Time_Warp_Mode">multi + time warp mode</seealso>. All other modes than the + multi time warp mode are there for backwards + compatibility reasons, and when using these the + accuracy of Erlang monotonic time suffer since + the adjustments of Erlang monotonic time in these + modes are more or less tied to the Erlang system + time.</p> + + <p>The adjustment of system time could have been made + smother than using a time warp approach, but we think + that would be a bad choice. Since we are able to + express and measure time that aren't connected to + calender time by the use of Erlang monotonic time, it + is better to expose the change in Erlang system time + immediately. This since it makes it possible for the + Erlang applications executing on the system to react + on the change in system time as soon as possible. This + is also more or less exactly how most OSes handle this + (OS monotonic time and OS system time). By adjusting + system time smoothly we would just hide the fact that + system time changed and make it harder for the Erlang + applications to react to the change in a sensible way.</p> + + <p>In order to be able to react to a change in Erlang + system time you have to be able to detect that it + happened. The change in Erlang system time occurs when + current time offset is changed. We have therefore + introduced the possibility to monitor the time offset + using + <seealso marker="erlang#monitor/2"><c>erlang:monitor(time_offset, clock_service)</c></seealso>. A process monitoring the time + offset will be sent a message on the following format + when the time offset is changed:</p> + <code type="none">{'CHANGE', MonitorReference, time_offset, clock_service, NewTimeOffset}</code> + </section> + + <marker id="Unique_Values"/> + <section> + <title>Unique Values</title> + <p>Besides reporting time <c>erlang:now/0</c> also + produce unique and strictly monotonically increasing + values. In order to detach this functionality from + time measurements we have introduced + <seealso marker="erlang#unique_integer/1"><c>erlang:unique_integer()</c></seealso>. + </p> + </section> + + <marker id="Dos_and_Donts"/> + <section> + <title>Dos and Don'ts</title> + <p>Previously <c>erlang:now/0</c> was the only option for doing + quite a lot of things. We will look at a few different things + <c>erlang:now/0</c> could be used for, and how you want to do + this using the new API:</p> + + <marker id="Dos_and_Donts_Retrieve_Erlang_System_Time"/> + <section> + <title>Retrieve Erlang System Time</title> + <dont> + <p> + use <c>erlang:now/0</c> in order to retrieve current Erlang + system time. + </p> + </dont> + <do> + <p> + use + <seealso marker="erlang#system_time/1"><c>erlang:system_time/1</c></seealso> + in order to retrieve current Erlang system time on the + <seealso marker="erlang#type_time_unit">time unit</seealso> + of your choice.</p> + <p>If you want the same format as returned by <c>erlang:now/0</c>, use + <seealso marker="erlang#timestamp/0"><c>erlang:timestamp/0</c></seealso>. + </p> + </do> + </section> + + <marker id="Dos_and_Donts_Measure_Elapsed_Time"/> + <section> + <title>Measure Elapsed Time</title> + <dont> + <p> + take timestamps with <c>erlang:now/0</c> and calculate + the difference in time with + <seealso marker="stdlib:timer#now_diff/2"><c>timer:now_diff/2</c></seealso>. + </p> + </dont> + <do> + <p> + take timestamps with + <seealso marker="erlang#monotonic_time/0"><c>erlang:monotonic_time/0</c></seealso> + and calculate the time difference using ordinary subtraction. + The result will be in <c>native</c> + <seealso marker="erlang#type_time_unit">time unit</seealso>. + If you want to convert the + result to another time unit you can do this using + <seealso marker="erlang#convert_time_unit/3"><c>erlang:convert_time_unit/3</c></seealso>. + </p> + <p>Another easier way of doing this is to use + <seealso marker="erlang#monotonic_time/1"><c>erlang:monotonic_time/1</c></seealso> + with desired time unit. However, you may lose accuracy, + and precision this way. + </p> + </do> + </section> + + <marker id="Dos_and_Donts_Determine_Order_of_Events"/> + <section> + <title>Determine Order of Events</title> + <dont> + <p> + determine the order of events by saving a timestamp + with <c>erlang:now/0</c> when the event happens. + </p> + </dont> + <do> + <p> + determine the order of events by saving the integer + returned by + <seealso marker="erlang#unique_integer/1"><c>erlang:unique_integer([monotonic])</c></seealso> + when the event happens. These integers will be strictly + monotonically ordered on current runtime system instance + corresponding to creation time. + </p> + </do> + </section> + + <marker id="Dos_and_Donts_Determine_Order_of_Events_With_Time_of_the_Event"/> + <section> + <title>Determine Order of Events With Time of the Event</title> + <dont> + <p> + determine the order of events by saving a timestamp + with <c>erlang:now/0</c> when the event happens. + </p> + </dont> + <do> + <p> + determine the order of events by saving a tuple + containing + <seealso marker="erlang#monotonic_time/0">monotonic time</seealso> + and a <seealso marker="erlang#unique_integer/1">strictly + monotonically increasing integer</seealso> like this:</p> + <code type="none"> +Time = erlang:monotonic_time(), +UMI = erlang:unique_integer([monotonic]), +EventTag = {Time, UMI}</code> + <p>These tuples will be strictly monotonically ordered + on the current runtime system instance according to + creation time. Note that it is important that the + monotonic time is in the first element (the most + significant element when comparing 2-tuples). Using + the monotonic time in the tuples, you can calculate time + between events.</p> + <p>If you are interested in the Erlang system time at the + time when the event occurred you can also save the time + offset before or after saving the events using + <seealso marker="erlang#time_offset/0"><c>erlang:time_offset/0</c></seealso>. + Erlang monotonic time added with the time + offset corresponds to Erlang system time.</p> + <p>If you are executing in a mode where time offset + may change and you want to be able to get the actual + Erlang system time when the event occurred you can + save the time offset as a third element in the tuple + (the least significant element when comparing 3-tuples).</p> + </do> + </section> + + <marker id="Dos_and_Donts_Create_a_Unique_Name"/> + <section> + <title>Create a Unique Name</title> + <dont> + <p> + use the values returned from <c>erlang:now/0</c> + in order to create a name unique on the current + runtime system instance. + </p> + </dont> + <do> + <p> + use the value returned from + <seealso marker="erlang#unique_integer/0"><c>erlang:unique_integer/0</c></seealso> + in order to create a name unique on the current runtime system + instance. If you only want positive integers, you can use + <seealso marker="erlang#unique_integer/1"><c>erlang:unique_integer([positive])</c></seealso>. + </p> + </do> + </section> + + <marker id="Dos_and_Donts_Seed_Random_Number_Generation_With_a_Unique_Value"/> + <section> + <title>Seed Random Number Generation With a Unique Value</title> + <dont> + <p> + seed random number generation using <c>erlang:now()</c>. + </p> + </dont> + <do> + <p> + seed random number generation using a combination of + <seealso marker="erlang#monotonic_time/0"><c>erlang:monotonic_time()</c></seealso>, + <seealso marker="erlang#time_offset/0"><c>erlang:time_offset()</c></seealso>, + <seealso marker="erlang#unique_integer/0"><c>erlang:unique_integer()</c></seealso>, and other functionality. + </p> + </do> + </section> + + <p>To sum this section up: <em>Don't use <c>erlang:now/0</c>!</em></p> + </section> </section> + + <marker id="Supporting_Both_New_and_Old_OTP_Releases"/> <section> - <title>Turning off time correction</title> - <p>If, for some reason, time correction causes trouble and you are - absolutely confident that the wall clock on the system is nearly - perfect, you can turn off time correction completely by giving the - <c>+c</c> option to <c>erl</c>. The probability for this being a - good idea, is very low.</p> + <title>Supporting Both New and Old OTP Releases</title> + <p>Your code may be required to be able to run on a variety + of OTP installations of different OTP releases. If so, you + can not just use the new API out of the box, since it will + not be available on old pre OTP 18 releases. The solution + is <em>not</em> to avoid using the new API, since your + code then won't be able to benefit from the scalability + and accuracy improvements made. Instead you want to use the + new API when available, and fall back on <c>erlang:now/0</c> + when it is not available. Fortunately almost all of the new + API can easily be implemented using existing primitives + (except for + <seealso marker="erlang#system_info_start_time"><c>erlang:system_info(start_time)</c></seealso>, + <seealso marker="erlang#system_info_os_monotonic_time_source"><c>erlang:system_info(os_monotonic_time_source)</c></seealso>, and + <seealso marker="erlang#system_info_os_system_time_source"><c>erlang:system_info(os_system_time_source)</c></seealso>). + By wrapping the API with functions that fall back on + <c>erlang:now/0</c> when the new API is not available, + and using these wrappers instead of using the API directly + the problem is solved. These wrappers can for example + be implemented as in + <url href="time_compat.erl"><c>$ERL_TOP/erts/example/time_compat.erl</c></url>.</p> </section> </chapter> - diff --git a/erts/doc/src/zlib.xml b/erts/doc/src/zlib.xml index 673b743e2e..1f10ddef6d 100644 --- a/erts/doc/src/zlib.xml +++ b/erts/doc/src/zlib.xml @@ -99,7 +99,7 @@ list_to_binary([Compressed|Last])</pre> <datatype> <name name="zwindowbits"/> <desc> - <p>Normally in the range <c>-15..-9 | 9..15</c>.</p> + <p>Normally in the range <c>-15..-8 | 8..15</c>.</p> </desc> </datatype> </datatypes> @@ -149,7 +149,7 @@ list_to_binary([Compressed|Last])</pre> currently the only supported method is <c>deflated</c>.</p> <p>The <c><anno>WindowBits</anno></c> parameter is the base two logarithm of the window size (the size of the history buffer). It - should be in the range 9 through 15. Larger values + should be in the range 8 through 15. Larger values of this parameter result in better compression at the expense of memory usage. The default value is 15 if <c>deflateInit/2</c>. A negative <c><anno>WindowBits</anno></c> @@ -288,7 +288,7 @@ list_to_binary([B1,B2])</pre> <p>Initialize decompression session on zlib stream.</p> <p>The <c><anno>WindowBits</anno></c> parameter is the base two logarithm of the maximum window size (the size of the history buffer). - It should be in the range 9 through 15. + It should be in the range 8 through 15. The default value is 15 if <c>inflateInit/1</c> is used. If a compressed stream with a larger window size is given as input, inflate() will throw the <c>data_error</c> diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index a632faf57d..b4a17e76e7 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -754,7 +754,8 @@ RUN_OBJS = \ $(OBJDIR)/erl_bif_ddll.o $(OBJDIR)/erl_bif_guard.o \ $(OBJDIR)/erl_bif_info.o $(OBJDIR)/erl_bif_op.o \ $(OBJDIR)/erl_bif_os.o $(OBJDIR)/erl_bif_lists.o \ - $(OBJDIR)/erl_bif_trace.o $(OBJDIR)/erl_bif_wrap.o \ + $(OBJDIR)/erl_bif_trace.o $(OBJDIR)/erl_bif_unique.o \ + $(OBJDIR)/erl_bif_wrap.o \ $(OBJDIR)/erl_trace.o $(OBJDIR)/copy.o \ $(OBJDIR)/utils.o $(OBJDIR)/bif.o \ $(OBJDIR)/io.o $(OBJDIR)/erl_printf_term.o\ @@ -887,7 +888,8 @@ OS_OBJS += $(OBJDIR)/erl_mseg.o \ $(OBJDIR)/erl_mmap.o \ $(OBJDIR)/erl_$(ERLANG_OSTYPE)_sys_ddll.o \ $(OBJDIR)/erl_mtrace_sys_wrap.o \ - $(OBJDIR)/erl_sys_common_misc.o + $(OBJDIR)/erl_sys_common_misc.o \ + $(OBJDIR)/erl_os_monotonic_time_extender.o HIPE_ARCH64_OBJS=$(OBJDIR)/hipe_bif64.o diff --git a/erts/emulator/beam/atom.names b/erts/emulator/beam/atom.names index c097866c7e..ae3f30d82f 100644 --- a/erts/emulator/beam/atom.names +++ b/erts/emulator/beam/atom.names @@ -109,6 +109,7 @@ atom bag atom band atom big atom bif_return_trap +atom bif_timer_server atom binary atom binary_bin_to_list_trap atom binary_copy_trap @@ -144,9 +145,11 @@ atom catchlevel atom cd atom cdr atom cflags +atom CHANGE='CHANGE' atom characters_to_binary_int atom characters_to_list_int atom clear +atom clock_service atom close atom closed atom code @@ -156,6 +159,7 @@ atom compat_rel atom compile atom compressed atom config_h +atom convert_time_unit atom connect atom connected atom connection_closed @@ -237,7 +241,7 @@ atom first atom firstline atom flags atom flush -atom flush_monitor_message +atom flush_monitor_messages atom force atom format_cpu_topology atom free @@ -345,6 +349,8 @@ atom message_queue_len atom messages atom meta atom meta_match_spec +atom micro_seconds +atom milli_seconds atom min_heap_size atom min_bin_vheap_size atom minor_version @@ -355,12 +361,15 @@ atom monitored_by atom monitor atom monitor_nodes atom monitors +atom monotonic atom more atom multi_scheduling atom multiline +atom nano_seconds atom name atom named_table atom namelist +atom native atom native_addresses atom Neq='=/=' atom Neqeq='/=' @@ -451,6 +460,7 @@ atom ports atom port_count atom port_limit atom port_op +atom positive atom print atom priority atom private @@ -510,6 +520,7 @@ atom schedulers_online atom scheme atom scientific atom scope +atom seconds atom sensitive atom sequential_tracer atom sequential_trace_token @@ -555,6 +566,7 @@ atom term_to_binary_trap atom this atom thread_pool_size atom threads +atom time_offset atom timeout atom timeout_value atom Times='*' diff --git a/erts/emulator/beam/beam_debug.c b/erts/emulator/beam/beam_debug.c index a3cd08834f..6bb987985d 100644 --- a/erts/emulator/beam/beam_debug.c +++ b/erts/emulator/beam/beam_debug.c @@ -579,9 +579,29 @@ print_op(int to, void *to_arg, int op, int size, BeamInstr* addr) unpacked = ap; ap = addr + size; switch (op) { - case op_i_select_val_rfI: - case op_i_select_val_xfI: - case op_i_select_val_yfI: + case op_i_select_val_lins_rfI: + case op_i_select_val_lins_xfI: + case op_i_select_val_lins_yfI: + { + int n = ap[-1]; + int ix = n; + + while (ix--) { + erts_print(to, to_arg, "%T ", (Eterm) ap[0]); + ap++; + size++; + } + ix = n; + while (ix--) { + erts_print(to, to_arg, "f(" HEXF ") ", (Eterm) ap[0]); + ap++; + size++; + } + } + break; + case op_i_select_val_bins_rfI: + case op_i_select_val_bins_xfI: + case op_i_select_val_bins_yfI: { int n = ap[-1]; @@ -598,13 +618,19 @@ print_op(int to, void *to_arg, int op, int size, BeamInstr* addr) case op_i_select_tuple_arity_yfI: { int n = ap[-1]; + int ix = n; - while (n > 0) { + while (ix--) { Uint arity = arityval(ap[0]); - erts_print(to, to_arg, " {%d} f(" HEXF ")", arity, ap[1]); - ap += 2; - size += 2; - n--; + erts_print(to, to_arg, "{%d} ", arity, ap[1]); + ap++; + size++; + } + ix = n; + while (ix--) { + erts_print(to, to_arg, "f(" HEXF ") ", ap[0]); + ap++; + size++; } } break; diff --git a/erts/emulator/beam/beam_emu.c b/erts/emulator/beam/beam_emu.c index 3af7c43abf..8fcdc72895 100644 --- a/erts/emulator/beam/beam_emu.c +++ b/erts/emulator/beam/beam_emu.c @@ -699,7 +699,7 @@ void** beam_ops; Fail; \ } -#define IsMap(Src, Fail) if (is_not_map(Src)) { Fail; } +#define IsMap(Src, Fail) if (!is_map(Src)) { Fail; } #define HasMapField(Src, Key, Fail) if (has_not_map_field(Src, Key)) { Fail; } @@ -2138,19 +2138,18 @@ void process_main(void) NextPF(0, next); } - { Eterm select_val2; - OpCase(i_select_tuple_arity2_yfAfAf): + OpCase(i_select_tuple_arity2_yfAAff): select_val2 = yb(Arg(0)); goto do_select_tuple_arity2; - OpCase(i_select_tuple_arity2_xfAfAf): + OpCase(i_select_tuple_arity2_xfAAff): select_val2 = xb(Arg(0)); goto do_select_tuple_arity2; - OpCase(i_select_tuple_arity2_rfAfAf): + OpCase(i_select_tuple_arity2_rfAAff): select_val2 = r(0); I--; @@ -2161,22 +2160,22 @@ void process_main(void) select_val2 = *tuple_val(select_val2); goto do_select_val2; - OpCase(i_select_val2_yfcfcf): + OpCase(i_select_val2_yfccff): select_val2 = yb(Arg(0)); goto do_select_val2; - OpCase(i_select_val2_xfcfcf): + OpCase(i_select_val2_xfccff): select_val2 = xb(Arg(0)); goto do_select_val2; - OpCase(i_select_val2_rfcfcf): + OpCase(i_select_val2_rfccff): select_val2 = r(0); I--; do_select_val2: if (select_val2 == Arg(2)) { - I += 2; - } else if (select_val2 == Arg(4)) { + I += 3; + } else if (select_val2 == Arg(3)) { I += 4; } @@ -2203,20 +2202,50 @@ void process_main(void) do_select_tuple_arity: if (is_tuple(select_val)) { select_val = *tuple_val(select_val); - goto do_binary_search; + goto do_linear_search; + } + SET_I((BeamInstr *) Arg(1)); + Goto(*I); + + OpCase(i_select_val_lins_xfI): + select_val = xb(Arg(0)); + goto do_linear_search; + + OpCase(i_select_val_lins_yfI): + select_val = yb(Arg(0)); + goto do_linear_search; + + OpCase(i_select_val_lins_rfI): + select_val = r(0); + I--; + + do_linear_search: { + BeamInstr *vs = &Arg(3); + int ix = 0; + + for(;;) { + if (vs[ix+0] >= select_val) { ix += 0; break; } + if (vs[ix+1] >= select_val) { ix += 1; break; } + ix += 2; + } + + if (vs[ix] == select_val) { + I += ix + Arg(2) + 2; } + SET_I((BeamInstr *) Arg(1)); Goto(*I); + } - OpCase(i_select_val_xfI): + OpCase(i_select_val_bins_xfI): select_val = xb(Arg(0)); goto do_binary_search; - OpCase(i_select_val_yfI): + OpCase(i_select_val_bins_yfI): select_val = yb(Arg(0)); goto do_binary_search; - OpCase(i_select_val_rfI): + OpCase(i_select_val_bins_rfI): select_val = r(0); I--; @@ -2363,7 +2392,7 @@ void process_main(void) } OpCase(i_has_map_fields_fsI): { - map_t* mp; + flatmap_t* mp; Eterm map; Eterm field; Eterm *ks; @@ -2371,22 +2400,34 @@ void process_main(void) Uint sz,n; GetArg1(1, map); + n = (Uint)Arg(2); + fs = &Arg(3); /* pattern fields */ - /* this instruction assumes Arg1 is a map, - * i.e. that it follows a test is_map if needed. - */ + /* get term from field? */ + if (is_hashmap(map)) { + Uint32 hx; + while(n--) { + field = *fs++; + hx = hashmap_make_hash(field); + if (!erts_hashmap_get(hx,field,map)) { + SET_I((BeamInstr *) Arg(0)); + goto has_map_fields_fail; + } + } + goto has_map_fields_ok; + } + + ASSERT(is_flatmap(map)); - mp = (map_t *)map_val(map); - sz = map_get_size(mp); + mp = (flatmap_t *)flatmap_val(map); + sz = flatmap_get_size(mp); if (sz == 0) { SET_I((BeamInstr *) Arg(0)); goto has_map_fields_fail; } - ks = map_get_keys(mp); - n = (Uint)Arg(2); - fs = &Arg(3); /* pattern fields */ + ks = flatmap_get_keys(mp); ASSERT(n>0); @@ -2404,7 +2445,7 @@ void process_main(void) SET_I((BeamInstr *) Arg(0)); goto has_map_fields_fail; } - +has_map_fields_ok: I += 4 + Arg(2); has_map_fields_fail: ASSERT(VALID_INSTR(*I)); @@ -2431,12 +2472,8 @@ do { \ OpCase(i_get_map_elements_fsI): { Eterm map; - map_t *mp; - Eterm field; - Eterm *ks; - Eterm *vs; BeamInstr *fs; - Uint sz,n; + Uint sz, n; GetArg1(1, map); @@ -2444,36 +2481,56 @@ do { \ * i.e. that it follows a test is_map if needed. */ - mp = (map_t *)map_val(map); - sz = map_get_size(mp); - - if (sz == 0) { - SET_I((BeamInstr *) Arg(0)); - goto get_map_elements_fail; - } - n = (Uint)Arg(2) / 2; fs = &Arg(3); /* pattern fields and target registers */ - ks = map_get_keys(mp); - vs = map_get_values(mp); - while(sz) { - field = (Eterm)*fs; - if (EQ(field,*ks)) { - PUT_TERM_REG(*vs, fs[1]); - n--; + if (is_flatmap(map)) { + flatmap_t *mp; + Eterm *ks; + Eterm *vs; + + mp = (flatmap_t *)flatmap_val(map); + sz = flatmap_get_size(mp); + + if (sz == 0) { + SET_I((BeamInstr *) Arg(0)); + goto get_map_elements_fail; + } + + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + while(sz) { + if (EQ((Eterm)*fs,*ks)) { + PUT_TERM_REG(*vs, fs[1]); + n--; + fs += 2; + /* no more values to fetch, we are done */ + if (n == 0) break; + } + ks++; sz--; + vs++; + } + + if (n) { + SET_I((BeamInstr *) Arg(0)); + goto get_map_elements_fail; + } + } else { + const Eterm *v; + Uint32 hx; + ASSERT(is_hashmap(map)); + while(n--) { + hx = hashmap_make_hash((Eterm)*fs); + if ((v = erts_hashmap_get(hx,(Eterm)*fs, map)) == NULL) { + SET_I((BeamInstr *) Arg(0)); + goto get_map_elements_fail; + } + PUT_TERM_REG(*v, fs[1]); fs += 2; - /* no more values to fetch, we are done */ - if (n == 0) break; } - ks++; sz--; - vs++; } - if (n) { - SET_I((BeamInstr *) Arg(0)); - goto get_map_elements_fail; - } I += 4 + Arg(2); get_map_elements_fail: @@ -2772,6 +2829,7 @@ get_map_elements_fail: } PreFetch(1, next); ASSERT(!ERTS_PROC_IS_EXITING(c_p)); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(c_p); reg[0] = r(0); result = (*bf)(c_p, reg, I); ASSERT(!ERTS_PROC_IS_EXITING(c_p) || is_non_value(result)); @@ -3553,7 +3611,7 @@ get_map_elements_fail: vbf = (BifFunction) Arg(0); PROCESS_MAIN_CHK_LOCKS(c_p); bif_nif_arity = I[-1]; - ASSERT(bif_nif_arity <= 3); + ASSERT(bif_nif_arity <= 4); ERTS_SMP_UNREQ_PROC_MAIN_LOCK(c_p); reg[0] = r(0); { @@ -6414,55 +6472,69 @@ new_fun(Process* p, Eterm* reg, ErlFunEntry* fe, int num_free) static int has_not_map_field(Eterm map, Eterm key) { - map_t* mp; - Eterm* keys; - Uint i; - Uint n; - - mp = (map_t *)map_val(map); - keys = map_get_keys(mp); - n = map_get_size(mp); - if (is_immed(key)) { - for (i = 0; i < n; i++) { - if (keys[i] == key) { - return 0; + Uint32 hx; + if (is_flatmap(map)) { + flatmap_t* mp; + Eterm* keys; + Uint i; + Uint n; + + mp = (flatmap_t *)flatmap_val(map); + keys = flatmap_get_keys(mp); + n = flatmap_get_size(mp); + if (is_immed(key)) { + for (i = 0; i < n; i++) { + if (keys[i] == key) { + return 0; + } } - } - } else { - for (i = 0; i < n; i++) { - if (EQ(keys[i], key)) { - return 0; + } else { + for (i = 0; i < n; i++) { + if (EQ(keys[i], key)) { + return 0; + } } } + return 1; } - return 1; + ASSERT(is_hashmap(map)); + hx = hashmap_make_hash(key); + return erts_hashmap_get(hx,key,map) ? 0 : 1; } static Eterm get_map_element(Eterm map, Eterm key) { - map_t *mp; - Eterm* ks, *vs; - Uint i; - Uint n; - - mp = (map_t *)map_val(map); - ks = map_get_keys(mp); - vs = map_get_values(mp); - n = map_get_size(mp); - if (is_immed(key)) { - for (i = 0; i < n; i++) { - if (ks[i] == key) { - return vs[i]; + Uint32 hx; + const Eterm *vs; + if (is_flatmap(map)) { + flatmap_t *mp; + Eterm *ks; + Uint i; + Uint n; + + mp = (flatmap_t *)flatmap_val(map); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + n = flatmap_get_size(mp); + if (is_immed(key)) { + for (i = 0; i < n; i++) { + if (ks[i] == key) { + return vs[i]; + } } - } - } else { - for (i = 0; i < n; i++) { - if (EQ(ks[i], key)) { - return vs[i]; + } else { + for (i = 0; i < n; i++) { + if (EQ(ks[i], key)) { + return vs[i]; + } } } + return THE_NON_VALUE; } - return THE_NON_VALUE; + ASSERT(is_hashmap(map)); + hx = hashmap_make_hash(key); + vs = erts_hashmap_get(hx,key,map); + return vs ? *vs : THE_NON_VALUE; } #define GET_TERM(term, dest) \ @@ -6495,7 +6567,30 @@ new_map(Process* p, Eterm* reg, BeamInstr* I) Eterm *mhp,*thp; Eterm *E; BeamInstr *ptr; - map_t *mp; + flatmap_t *mp; + ErtsHeapFactory factory; + + ptr = &Arg(4); + + if (n > 2*MAP_SMALL_MAP_LIMIT) { + if (HeapWordsLeft(p) < n) { + erts_garbage_collect(p, n, reg, Arg(2)); + } + + mhp = p->htop; + thp = p->htop; + E = p->stop; + + for (i = 0; i < n/2; i++) { + GET_TERM(*ptr++, *mhp++); + GET_TERM(*ptr++, *mhp++); + } + + p->htop = mhp; + + factory.p = p; + return erts_hashmap_from_array(&factory, thp, n/2, 0); + } if (HeapWordsLeft(p) < need) { erts_garbage_collect(p, need, reg, Arg(2)); @@ -6504,12 +6599,11 @@ new_map(Process* p, Eterm* reg, BeamInstr* I) thp = p->htop; mhp = thp + 1 + n/2; E = p->stop; - ptr = &Arg(4); keys = make_tuple(thp); *thp++ = make_arityval(n/2); - mp = (map_t *)mhp; mhp += MAP_HEADER_SIZE; - mp->thing_word = MAP_HEADER; + mp = (flatmap_t *)mhp; mhp += MAP_HEADER_FLATMAP_SZ; + mp->thing_word = MAP_HEADER_FLATMAP; mp->size = n/2; mp->keys = keys; @@ -6518,7 +6612,7 @@ new_map(Process* p, Eterm* reg, BeamInstr* I) GET_TERM(*ptr++, *mhp++); } p->htop = mhp; - return make_map(mp); + return make_flatmap(mp); } static Eterm @@ -6528,7 +6622,7 @@ update_map_assoc(Process* p, Eterm* reg, Eterm map, BeamInstr* I) Uint num_old; Uint num_updates; Uint need; - map_t *old_mp, *mp; + flatmap_t *old_mp, *mp; Eterm res; Eterm* hp; Eterm* E; @@ -6538,12 +6632,44 @@ update_map_assoc(Process* p, Eterm* reg, Eterm map, BeamInstr* I) Eterm new_key; Eterm* kp; - if (is_not_map(map)) { - return THE_NON_VALUE; + new_p = &Arg(5); + num_updates = Arg(4) / 2; + + if (is_not_flatmap(map)) { + Uint32 hx; + Eterm val; + + /* apparently the compiler does not emit is_map instructions, + * bad compiler */ + + if (is_not_hashmap(map)) + return THE_NON_VALUE; + + res = map; + E = p->stop; + while(num_updates--) { + /* assoc can't fail */ + GET_TERM(new_p[0], new_key); + GET_TERM(new_p[1], val); + hx = hashmap_make_hash(new_key); + + res = erts_hashmap_insert(p, hx, new_key, val, res, 0); + if (p->mbuf) { + Uint live = Arg(3); + reg[live] = res; + erts_garbage_collect(p, 0, reg, live+1); + res = reg[live]; + } + + E = p->stop; + + new_p += 2; + } + return res; } - old_mp = (map_t *) map_val(map); - num_old = map_get_size(old_mp); + old_mp = (flatmap_t *) flatmap_val(map); + num_old = flatmap_get_size(old_mp); /* * If the old map is empty, create a new map. @@ -6558,14 +6684,13 @@ update_map_assoc(Process* p, Eterm* reg, Eterm map, BeamInstr* I) * update list are new). */ - num_updates = Arg(4) / 2; - need = 2*(num_old+num_updates) + 1 + MAP_HEADER_SIZE; + need = 2*(num_old+num_updates) + 1 + MAP_HEADER_FLATMAP_SZ; if (HeapWordsLeft(p) < need) { Uint live = Arg(3); reg[live] = map; erts_garbage_collect(p, need, reg, live+1); map = reg[live]; - old_mp = (map_t *)map_val(map); + old_mp = (flatmap_t *)flatmap_val(map); } /* @@ -6596,16 +6721,15 @@ update_map_assoc(Process* p, Eterm* reg, Eterm map, BeamInstr* I) kp = p->htop + 1; /* Point to first key */ hp = kp + num_old + num_updates; - res = make_map(hp); - mp = (map_t *)hp; - hp += MAP_HEADER_SIZE; - mp->thing_word = MAP_HEADER; + res = make_flatmap(hp); + mp = (flatmap_t *)hp; + hp += MAP_HEADER_FLATMAP_SZ; + mp->thing_word = MAP_HEADER_FLATMAP; mp->keys = make_tuple(kp-1); - old_vals = map_get_values(old_mp); - old_keys = map_get_keys(old_mp); + old_vals = flatmap_get_values(old_mp); + old_keys = flatmap_get_keys(old_mp); - new_p = &Arg(5); GET_TERM(*new_p, new_key); n = num_updates; @@ -6691,8 +6815,19 @@ update_map_assoc(Process* p, Eterm* reg, Eterm map, BeamInstr* I) n = kp - p->htop - 1; /* Actual number of keys/values */ *p->htop = make_arityval(n); + p->htop = hp; mp->size = n; - p->htop = hp; + + /* The expensive case, need to build a hashmap */ + if (n > MAP_SMALL_MAP_LIMIT) { + res = erts_hashmap_from_ks_and_vs(p,flatmap_get_keys(mp),flatmap_get_values(mp),n); + if (p->mbuf) { + Uint live = Arg(3); + reg[live] = res; + erts_garbage_collect(p, 0, reg, live+1); + res = reg[live]; + } + } return res; } @@ -6707,7 +6842,7 @@ update_map_exact(Process* p, Eterm* reg, Eterm map, BeamInstr* I) Uint i; Uint num_old; Uint need; - map_t *old_mp, *mp; + flatmap_t *old_mp, *mp; Eterm res; Eterm* hp; Eterm* E; @@ -6716,12 +6851,48 @@ update_map_exact(Process* p, Eterm* reg, Eterm map, BeamInstr* I) BeamInstr* new_p; Eterm new_key; - if (is_not_map(map)) { - return THE_NON_VALUE; + new_p = &Arg(5); + n = Arg(4) / 2; /* Number of values to be updated */ + ASSERT(n > 0); + + if (is_not_flatmap(map)) { + Uint32 hx; + Eterm val; + + /* apparently the compiler does not emit is_map instructions, + * bad compiler */ + + if (is_not_hashmap(map)) + return THE_NON_VALUE; + + res = map; + E = p->stop; + while(n--) { + /* assoc can't fail */ + GET_TERM(new_p[0], new_key); + GET_TERM(new_p[1], val); + hx = hashmap_make_hash(new_key); + + res = erts_hashmap_insert(p, hx, new_key, val, res, 1); + if (is_non_value(res)) + return res; + + if (p->mbuf) { + Uint live = Arg(3); + reg[live] = res; + erts_garbage_collect(p, 0, reg, live+1); + res = reg[live]; + } + + E = p->stop; + + new_p += 2; + } + return res; } - old_mp = (map_t *) map_val(map); - num_old = map_get_size(old_mp); + old_mp = (flatmap_t *) flatmap_val(map); + num_old = flatmap_get_size(old_mp); /* * If the old map is empty, create a new map. @@ -6735,13 +6906,13 @@ update_map_exact(Process* p, Eterm* reg, Eterm map, BeamInstr* I) * Allocate the exact heap space needed. */ - need = num_old + MAP_HEADER_SIZE; + need = num_old + MAP_HEADER_FLATMAP_SZ; if (HeapWordsLeft(p) < need) { Uint live = Arg(3); reg[live] = map; erts_garbage_collect(p, need, reg, live+1); map = reg[live]; - old_mp = (map_t *)map_val(map); + old_mp = (flatmap_t *)flatmap_val(map); } /* @@ -6751,23 +6922,20 @@ update_map_exact(Process* p, Eterm* reg, Eterm map, BeamInstr* I) hp = p->htop; E = p->stop; - old_vals = map_get_values(old_mp); - old_keys = map_get_keys(old_mp); + old_vals = flatmap_get_values(old_mp); + old_keys = flatmap_get_keys(old_mp); - res = make_map(hp); - mp = (map_t *)hp; - hp += MAP_HEADER_SIZE; - mp->thing_word = MAP_HEADER; + res = make_flatmap(hp); + mp = (flatmap_t *)hp; + hp += MAP_HEADER_FLATMAP_SZ; + mp->thing_word = MAP_HEADER_FLATMAP; mp->size = num_old; mp->keys = old_mp->keys; /* Get array of key/value pairs to be updated */ - new_p = &Arg(5); GET_TERM(*new_p, new_key); /* Update all values */ - n = Arg(4) / 2; /* Number of values to be updated */ - ASSERT(n > 0); for (i = 0; i < num_old; i++) { if (!EQ(*old_keys, new_key)) { /* Not same keys */ diff --git a/erts/emulator/beam/beam_load.c b/erts/emulator/beam/beam_load.c index 07654f6d5c..02689e5b19 100644 --- a/erts/emulator/beam/beam_load.c +++ b/erts/emulator/beam/beam_load.c @@ -3321,9 +3321,10 @@ gen_select_tuple_arity(LoaderState* stp, GenOpArg S, GenOpArg Fail, { GenOp* op; + GenOpArg *tmp; int arity = Size.val + 3; int size = Size.val / 2; - int i; + int i, j, align = 0; /* * Verify the validity of the list. @@ -3338,9 +3339,37 @@ gen_select_tuple_arity(LoaderState* stp, GenOpArg S, GenOpArg Fail, } /* + * Use a special-cased instruction if there are only two values. + */ + if (size == 2) { + NEW_GENOP(stp, op); + op->next = NULL; + op->op = genop_i_select_tuple_arity2_6; + GENOP_ARITY(op, arity - 1); + op->a[0] = S; + op->a[1] = Fail; + op->a[2].type = TAG_u; + op->a[2].val = Rest[0].val; + op->a[3].type = TAG_u; + op->a[3].val = Rest[2].val; + op->a[4] = Rest[1]; + op->a[5] = Rest[3]; + + return op; + } + + /* * Generate the generic instruction. + * Assumption: + * Few different tuple arities to select on (fewer than 20). + * Use linear scan approach. */ + align = 1; + + arity += 2*align; + size += align; + NEW_GENOP(stp, op); op->next = NULL; op->op = genop_i_select_tuple_arity_3; @@ -3348,39 +3377,36 @@ gen_select_tuple_arity(LoaderState* stp, GenOpArg S, GenOpArg Fail, op->a[0] = S; op->a[1] = Fail; op->a[2].type = TAG_u; - op->a[2].val = Size.val / 2; - for (i = 0; i < Size.val; i += 2) { - op->a[i+3].type = TAG_v; - op->a[i+3].val = make_arityval(Rest[i].val); - op->a[i+4] = Rest[i+1]; - } + op->a[2].val = size; - /* - * Sort the values to make them useful for a binary search. - */ + tmp = (GenOpArg *) erts_alloc(ERTS_ALC_T_LOADER_TMP, sizeof(GenOpArg)*(arity-2*align)); - qsort(op->a+3, size, 2*sizeof(GenOpArg), - (int (*)(const void *, const void *)) genopargcompare); -#ifdef DEBUG - for (i = 3; i < arity-2; i += 2) { - ASSERT(op->a[i].val < op->a[i+2].val); + for (i = 3; i < arity - 2*align; i+=2) { + tmp[i-3].type = TAG_v; + tmp[i-3].val = make_arityval(Rest[i-3].val); + tmp[i-2] = Rest[i-2]; } -#endif /* - * Use a special-cased instruction if there are only two values. + * Sort the values to make them useful for a sentinel search */ - if (size == 2) { - op->op = genop_i_select_tuple_arity2_6; - op->arity--; - op->a[2].type = TAG_u; - op->a[2].val = arityval(op->a[3].val); - op->a[3] = op->a[4]; - op->a[4].type = TAG_u; - op->a[4].val = arityval(op->a[5].val); - op->a[5] = op->a[6]; + + qsort(tmp, size - align, 2*sizeof(GenOpArg), + (int (*)(const void *, const void *)) genopargcompare); + + j = 3; + for (i = 3; i < arity - 2*align; i += 2) { + op->a[j] = tmp[i-3]; + op->a[j + size] = tmp[i-2]; + j++; } + erts_free(ERTS_ALC_T_LOADER_TMP, (void *) tmp); + + op->a[j].type = TAG_u; + op->a[j].val = ~((BeamInstr)0); + op->a[j+size] = Fail; + return op; } @@ -3602,45 +3628,109 @@ gen_select_val(LoaderState* stp, GenOpArg S, GenOpArg Fail, GenOpArg Size, GenOpArg* Rest) { GenOp* op; + GenOpArg *tmp; int arity = Size.val + 3; int size = Size.val / 2; - int i; + int i, j, align = 0; + + if (size == 2) { + + /* + * Use a special-cased instruction if there are only two values. + */ + + NEW_GENOP(stp, op); + op->next = NULL; + op->op = genop_i_select_val2_6; + GENOP_ARITY(op, arity - 1); + op->a[0] = S; + op->a[1] = Fail; + op->a[2] = Rest[0]; + op->a[3] = Rest[2]; + op->a[4] = Rest[1]; + op->a[5] = Rest[3]; + + return op; + + } else if (size > 10) { + + /* binary search instruction */ + + NEW_GENOP(stp, op); + op->next = NULL; + op->op = genop_i_select_val_bins_3; + GENOP_ARITY(op, arity); + op->a[0] = S; + op->a[1] = Fail; + op->a[2].type = TAG_u; + op->a[2].val = size; + for (i = 3; i < arity; i++) { + op->a[i] = Rest[i-3]; + } + + /* + * Sort the values to make them useful for a binary search. + */ + + qsort(op->a+3, size, 2*sizeof(GenOpArg), + (int (*)(const void *, const void *)) genopargcompare); +#ifdef DEBUG + for (i = 3; i < arity-2; i += 2) { + ASSERT(op->a[i].val < op->a[i+2].val); + } +#endif + return op; + } + + /* linear search instruction */ + + align = 1; + + arity += 2*align; + size += align; NEW_GENOP(stp, op); op->next = NULL; - op->op = genop_i_select_val_3; + op->op = genop_i_select_val_lins_3; GENOP_ARITY(op, arity); op->a[0] = S; op->a[1] = Fail; op->a[2].type = TAG_u; op->a[2].val = size; - for (i = 3; i < arity; i++) { - op->a[i] = Rest[i-3]; + + tmp = (GenOpArg *) erts_alloc(ERTS_ALC_T_LOADER_TMP, sizeof(GenOpArg)*(arity-2*align)); + + for (i = 3; i < arity - 2*align; i++) { + tmp[i-3] = Rest[i-3]; } /* - * Sort the values to make them useful for a binary search. + * Sort the values to make them useful for a sentinel search */ - qsort(op->a+3, size, 2*sizeof(GenOpArg), - (int (*)(const void *, const void *)) genopargcompare); -#ifdef DEBUG - for (i = 3; i < arity-2; i += 2) { - ASSERT(op->a[i].val < op->a[i+2].val); + qsort(tmp, size - align, 2*sizeof(GenOpArg), + (int (*)(const void *, const void *)) genopargcompare); + + j = 3; + for (i = 3; i < arity - 2*align; i += 2) { + op->a[j] = tmp[i-3]; + op->a[j+size] = tmp[i-2]; + j++; } -#endif - /* - * Use a special-cased instruction if there are only two values. - */ - if (size == 2) { - op->op = genop_i_select_val2_6; - op->arity--; - op->a[2] = op->a[3]; - op->a[3] = op->a[4]; - op->a[4] = op->a[5]; - op->a[5] = op->a[6]; + erts_free(ERTS_ALC_T_LOADER_TMP, (void *) tmp); + + /* add sentinel */ + + op->a[j].type = TAG_u; + op->a[j].val = ~((BeamInstr)0); + op->a[j+size] = Fail; + +#ifdef DEBUG + for (i = 0; i < size - 1; i++) { + ASSERT(op->a[i+3].val <= op->a[i+4].val); } +#endif return op; } @@ -4982,7 +5072,8 @@ get_tag_and_value(LoaderState* stp, Uint len_code, arity = count/sizeof(Eterm); *result = new_literal(stp, &hp, arity+1); - (void) bytes_to_big(bigbuf, count, neg, hp); + if (is_nil(bytes_to_big(bigbuf, count, neg, hp))) + goto load_error; if (bigbuf != default_buf) { erts_free(ERTS_ALC_T_LOADER_TMP, (void *) bigbuf); diff --git a/erts/emulator/beam/benchmark.h b/erts/emulator/beam/benchmark.h index 904564a96b..7fc3933f3d 100644 --- a/erts/emulator/beam/benchmark.h +++ b/erts/emulator/beam/benchmark.h @@ -141,10 +141,10 @@ extern unsigned long long major_gc; /* (Assuming Solaris) */ -#define BM_TIMER_T hrtime_t -#define BM_START_TIMER(t) system_clock = sys_gethrtime() +#define BM_TIMER_T ErtsMonotonicTime +#define BM_START_TIMER(t) system_clock = ERTS_MONOTONIC_TO_NSEC(erts_os_monotonic_time()) #define BM_STOP_TIMER(t) do { \ - BM_TIMER_T tmp = (sys_gethrtime() - system_clock) - timer_time; \ + BM_TIMER_T tmp = (ERTS_MONOTONIC_TO_NSEC(erts_os_monotonic_time()) - system_clock) - timer_time; \ t##_time += (tmp > 0 ? tmp : 0); \ } while(0) diff --git a/erts/emulator/beam/bif.c b/erts/emulator/beam/bif.c index 49996e7f0b..cc20ec7440 100644 --- a/erts/emulator/beam/bif.c +++ b/erts/emulator/beam/bif.c @@ -42,16 +42,22 @@ #define ERTS_PTAB_WANT_BIF_IMPL__ #include "erl_ptab.h" #include "erl_bits.h" +#include "erl_bif_unique.h" -static Export* flush_monitor_message_trap = NULL; +static Export* flush_monitor_messages_trap = NULL; static Export* set_cpu_topology_trap = NULL; static Export* await_proc_exit_trap = NULL; static Export* await_port_send_result_trap = NULL; Export* erts_format_cpu_topology_trap = NULL; static Export dsend_continue_trap_export; +Export *erts_convert_time_unit_trap = NULL; + static Export *await_sched_wall_time_mod_trap; static erts_smp_atomic32_t sched_wall_time; +static erts_smp_mtx_t ports_snapshot_mtx; +erts_smp_atomic_t erts_dead_ports_ptr; /* To store dying ports during snapshot */ + #define DECL_AM(S) Eterm AM_ ## S = am_atom_put(#S, sizeof(#S) - 1) /* @@ -393,7 +399,7 @@ remote_demonitor(Process *c_p, DistEntry *dep, Eterm ref, Eterm to) return res; } -static int demonitor(Process *c_p, Eterm ref) +static int demonitor(Process *c_p, Eterm ref, Eterm *multip) { ErtsMonitor *mon = NULL; /* The monitor entry to delete */ Process *rp; /* Local target process */ @@ -417,65 +423,73 @@ static int demonitor(Process *c_p, Eterm ref) goto done; } - if (mon->type != MON_ORIGIN) { - res = ERTS_DEMONITOR_BADARG; - goto done; - } - to = mon->pid; - - if (is_atom(to)) { - /* Monitoring a name at node to */ - ASSERT(is_node_name_atom(to)); - dep = erts_sysname_to_connected_dist_entry(to); - ASSERT(dep != erts_this_dist_entry); - if (dep) - deref_de = 1; - } else { - ASSERT(is_pid(to)); - dep = pid_dist_entry(to); - } - if (dep != erts_this_dist_entry) { - res = remote_demonitor(c_p, dep, ref, to); - /* remote_demonitor() unlocks link lock on c_p */ - unlock_link = 0; - } - else { /* Local monitor */ - if (deref_de) { - deref_de = 0; - erts_deref_dist_entry(dep); + switch (mon->type) { + case MON_TIME_OFFSET: + *multip = am_true; + erts_demonitor_time_offset(ref); + res = ERTS_DEMONITOR_TRUE; + break; + case MON_ORIGIN: + to = mon->pid; + *multip = am_false; + if (is_atom(to)) { + /* Monitoring a name at node to */ + ASSERT(is_node_name_atom(to)); + dep = erts_sysname_to_connected_dist_entry(to); + ASSERT(dep != erts_this_dist_entry); + if (dep) + deref_de = 1; + } else { + ASSERT(is_pid(to)); + dep = pid_dist_entry(to); } - dep = NULL; - rp = erts_pid2proc_opt(c_p, - ERTS_PROC_LOCK_MAIN|ERTS_PROC_LOCK_LINK, - to, - ERTS_PROC_LOCK_LINK, - ERTS_P2P_FLG_ALLOW_OTHER_X); - mon = erts_remove_monitor(&ERTS_P_MONITORS(c_p), ref); + if (dep != erts_this_dist_entry) { + res = remote_demonitor(c_p, dep, ref, to); + /* remote_demonitor() unlocks link lock on c_p */ + unlock_link = 0; + } + else { /* Local monitor */ + if (deref_de) { + deref_de = 0; + erts_deref_dist_entry(dep); + } + dep = NULL; + rp = erts_pid2proc_opt(c_p, + ERTS_PROC_LOCK_MAIN|ERTS_PROC_LOCK_LINK, + to, + ERTS_PROC_LOCK_LINK, + ERTS_P2P_FLG_ALLOW_OTHER_X); + mon = erts_remove_monitor(&ERTS_P_MONITORS(c_p), ref); #ifndef ERTS_SMP - ASSERT(mon); + ASSERT(mon); #else - if (!mon) - res = ERTS_DEMONITOR_FALSE; - else + if (!mon) + res = ERTS_DEMONITOR_FALSE; + else #endif - { - res = ERTS_DEMONITOR_TRUE; - erts_destroy_monitor(mon); - } - if (rp) { - ErtsMonitor *rmon; - rmon = erts_remove_monitor(&ERTS_P_MONITORS(rp), ref); - if (rp != c_p) - erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK); - if (rmon != NULL) - erts_destroy_monitor(rmon); - } - else { - ERTS_SMP_ASSERT_IS_NOT_EXITING(c_p); - } + { + res = ERTS_DEMONITOR_TRUE; + erts_destroy_monitor(mon); + } + if (rp) { + ErtsMonitor *rmon; + rmon = erts_remove_monitor(&ERTS_P_MONITORS(rp), ref); + if (rp != c_p) + erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK); + if (rmon != NULL) + erts_destroy_monitor(rmon); + } + else { + ERTS_SMP_ASSERT_IS_NOT_EXITING(c_p); + } + } + break; + default: + res = ERTS_DEMONITOR_BADARG; + *multip = am_false; + break; } - done: if (unlock_link) @@ -492,7 +506,8 @@ static int demonitor(Process *c_p, Eterm ref) BIF_RETTYPE demonitor_1(BIF_ALIST_1) { - switch (demonitor(BIF_P, BIF_ARG_1)) { + Eterm multi; + switch (demonitor(BIF_P, BIF_ARG_1, &multi)) { case ERTS_DEMONITOR_FALSE: case ERTS_DEMONITOR_TRUE: BIF_RET(am_true); @@ -510,6 +525,7 @@ BIF_RETTYPE demonitor_1(BIF_ALIST_1) BIF_RETTYPE demonitor_2(BIF_ALIST_2) { Eterm res = am_true; + Eterm multi = am_false; int info = 0; int flush = 0; Eterm list = BIF_ARG_2; @@ -532,13 +548,18 @@ BIF_RETTYPE demonitor_2(BIF_ALIST_2) if (is_not_nil(list)) goto badarg; - switch (demonitor(BIF_P, BIF_ARG_1)) { + switch (demonitor(BIF_P, BIF_ARG_1, &multi)) { case ERTS_DEMONITOR_FALSE: if (info) res = am_false; - if (flush) - BIF_TRAP2(flush_monitor_message_trap, BIF_P, BIF_ARG_1, res); + if (flush) { + flush_messages: + BIF_TRAP3(flush_monitor_messages_trap, BIF_P, + BIF_ARG_1, multi, res); + } case ERTS_DEMONITOR_TRUE: + if (multi == am_true && flush) + goto flush_messages; BIF_RET(res); case ERTS_DEMONITOR_YIELD_TRUE: ERTS_BIF_YIELD_RETURN(BIF_P, am_true); @@ -597,14 +618,12 @@ erts_queue_monitor_message(Process *p, } static BIF_RETTYPE -local_pid_monitor(Process *p, Eterm target) +local_pid_monitor(Process *p, Eterm target, Eterm mon_ref, int bool) { BIF_RETTYPE ret; - Eterm mon_ref; Process *rp; ErtsProcLocks p_locks = ERTS_PROC_LOCK_MAIN|ERTS_PROC_LOCK_LINK; - mon_ref = erts_make_ref(p); ERTS_BIF_PREP_RET(ret, mon_ref); if (target == p->common.id) { return ret; @@ -617,12 +636,18 @@ local_pid_monitor(Process *p, Eterm target) if (!rp) { erts_smp_proc_unlock(p, ERTS_PROC_LOCK_LINK); p_locks &= ~ERTS_PROC_LOCK_LINK; - erts_queue_monitor_message(p, &p_locks, - mon_ref, am_process, target, am_noproc); + if (bool) + ret = am_false; + else + erts_queue_monitor_message(p, &p_locks, + mon_ref, am_process, target, am_noproc); } else { ASSERT(rp != p); + if (bool) + ret = am_true; + erts_add_monitor(&ERTS_P_MONITORS(p), MON_ORIGIN, mon_ref, target, NIL); erts_add_monitor(&ERTS_P_MONITORS(rp), MON_TARGET, mon_ref, p->common.id, NIL); @@ -746,13 +771,28 @@ BIF_RETTYPE monitor_2(BIF_ALIST_2) int deref_de = 0; /* Only process monitors are implemented */ - if (BIF_ARG_1 != am_process) { + switch (BIF_ARG_1) { + case am_time_offset: { + Eterm ref; + if (BIF_ARG_2 != am_clock_service) + goto error; + ref = erts_make_ref(BIF_P); + erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_LINK); + erts_add_monitor(&ERTS_P_MONITORS(BIF_P), MON_TIME_OFFSET, + ref, am_clock_service, NIL); + erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCK_LINK); + erts_monitor_time_offset(BIF_P->common.id, ref); + BIF_RET(ref); + } + case am_process: + break; + default: goto error; } if (is_internal_pid(target)) { local_pid: - ret = local_pid_monitor(BIF_P, target); + ret = local_pid_monitor(BIF_P, target, erts_make_ref(BIF_P), 0); } else if (is_external_pid(target)) { dep = external_pid_dist_entry(target); if (dep == erts_this_dist_entry) @@ -795,6 +835,25 @@ BIF_RETTYPE monitor_2(BIF_ALIST_2) return ret; } +BIF_RETTYPE erts_internal_monitor_process_2(BIF_ALIST_2) +{ + if (is_not_internal_pid(BIF_ARG_1)) { + if (is_external_pid(BIF_ARG_1) + && (external_pid_dist_entry(BIF_ARG_1) + == erts_this_dist_entry)) { + BIF_RET(am_false); + } + goto badarg; + } + + if (is_not_internal_ref(BIF_ARG_2)) + goto badarg; + + BIF_RET(local_pid_monitor(BIF_P, BIF_ARG_1, BIF_ARG_2, 1)); + +badarg: + BIF_ERROR(BIF_P, BADARG); +} /**********************************************************************/ /* this is a combination of the spawn and link BIFs */ @@ -3542,91 +3601,6 @@ BIF_RETTYPE self_0(BIF_ALIST_0) /**********************************************************************/ -/* - New representation of refs in R9, see erl_term.h - - In the first data word, only the usual 18 bits are used. Ordinarily, - in "long refs" all words are used (in other words, practically never - wrap around), but for compatibility with older nodes, "short refs" - exist. Short refs come into being by being converted from the old - external format for refs (tag REFERENCE_EXT). Short refs are - converted back to the old external format. - - When converting a long ref to the external format in the case of - preparing for sending to an older node, the ref is truncated by only - using the first word (with 18 significant bits), and using the old tag - REFERENCE_EXT. - - When comparing refs or different size, only the parts up to the length - of the shorter operand are used. This has the desirable effect that a - long ref sent to an old node and back will be treated as equal to - the original, although some of the bits have been lost. - - The hash value for a ref always considers only the first word, since - in the above scenario, the original and the copy should have the same - hash value. -*/ - -static Uint32 reference0; /* Initialized in erts_init_bif */ -static Uint32 reference1; -static Uint32 reference2; -static erts_smp_spinlock_t make_ref_lock; -static erts_smp_mtx_t ports_snapshot_mtx; -erts_smp_atomic_t erts_dead_ports_ptr; /* To store dying ports during snapshot */ - -void -erts_make_ref_in_array(Uint32 ref[ERTS_MAX_REF_NUMBERS]) -{ - erts_smp_spin_lock(&make_ref_lock); - - reference0++; - if (reference0 >= MAX_REFERENCE) { - reference0 = 0; - reference1++; - if (reference1 == 0) { - reference2++; - } - } - - ref[0] = reference0; - ref[1] = reference1; - ref[2] = reference2; - - erts_smp_spin_unlock(&make_ref_lock); -} - -Eterm erts_make_ref_in_buffer(Eterm buffer[REF_THING_SIZE]) -{ - Eterm* hp = buffer; - Uint32 ref[ERTS_MAX_REF_NUMBERS]; - - erts_make_ref_in_array(ref); - write_ref_thing(hp, ref[0], ref[1], ref[2]); - return make_internal_ref(hp); -} - -Eterm erts_make_ref(Process *p) -{ - Eterm* hp; - Uint32 ref[ERTS_MAX_REF_NUMBERS]; - - ERTS_SMP_LC_ASSERT(ERTS_PROC_LOCK_MAIN & erts_proc_lc_my_proc_locks(p)); - - hp = HAlloc(p, REF_THING_SIZE); - - erts_make_ref_in_array(ref); - write_ref_thing(hp, ref[0], ref[1], ref[2]); - - return make_internal_ref(hp); -} - -BIF_RETTYPE make_ref_0(BIF_ALIST_0) -{ - return erts_make_ref(BIF_P); -} - -/**********************************************************************/ - /* return the time of day */ BIF_RETTYPE time_0(BIF_ALIST_0) @@ -4607,6 +4581,28 @@ BIF_RETTYPE system_flag_2(BIF_ALIST_2) break; } #endif + } else if (BIF_ARG_1 == am_time_offset + && ERTS_IS_ATOM_STR("finalize", BIF_ARG_2)) { + ErtsTimeOffsetState res; + erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCK_MAIN); + res = erts_finalize_time_offset(); + erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_MAIN); + switch (res) { + case ERTS_TIME_OFFSET_PRELIMINARY: { + DECL_AM(preliminary); + BIF_RET(AM_preliminary); + } + case ERTS_TIME_OFFSET_FINAL: { + DECL_AM(final); + BIF_RET(AM_final); + } + case ERTS_TIME_OFFSET_VOLATILE: { + DECL_AM(volatile); + BIF_RET(AM_volatile); + } + default: + ERTS_INTERNAL_ERROR("Unknown state"); + } } else if (ERTS_IS_ATOM_STR("scheduling_statistics", BIF_ARG_1)) { int what; if (ERTS_IS_ATOM_STR("disable", BIF_ARG_2)) @@ -4758,7 +4754,7 @@ BIF_RETTYPE bump_reductions_1(BIF_ALIST_1) } BIF_RETTYPE erts_internal_cmp_term_2(BIF_ALIST_2) { - int res = CMP_TERM(BIF_ARG_1,BIF_ARG_2); + Sint res = CMP_TERM(BIF_ARG_1,BIF_ARG_2); /* ensure -1, 0, 1 result */ if (res < 0) { @@ -4894,11 +4890,6 @@ void erts_init_trap_export(Export* ep, Eterm m, Eterm f, Uint a, void erts_init_bif(void) { - reference0 = 0; - reference1 = 0; - reference2 = 0; - - erts_smp_spinlock_init(&make_ref_lock, "make_ref"); erts_smp_mtx_init(&ports_snapshot_mtx, "ports_snapshot"); erts_smp_atomic_init_nob(&erts_dead_ports_ptr, (erts_aint_t) NULL); @@ -4919,9 +4910,13 @@ void erts_init_bif(void) am_erts_internal, am_dsend_continue_trap, 1, dsend_continue_trap_1); - flush_monitor_message_trap = erts_export_put(am_erlang, - am_flush_monitor_message, - 2); + flush_monitor_messages_trap = erts_export_put(am_erts_internal, + am_flush_monitor_messages, + 3); + + erts_convert_time_unit_trap = erts_export_put(am_erlang, + am_convert_time_unit, + 3); set_cpu_topology_trap = erts_export_put(am_erlang, am_set_cpu_topology, diff --git a/erts/emulator/beam/bif.h b/erts/emulator/beam/bif.h index 7b69b39511..d461c3f479 100644 --- a/erts/emulator/beam/bif.h +++ b/erts/emulator/beam/bif.h @@ -21,6 +21,7 @@ #define __BIF_H__ extern Export* erts_format_cpu_topology_trap; +extern Export *erts_convert_time_unit_trap; #define BIF_RETTYPE Eterm @@ -30,10 +31,12 @@ extern Export* erts_format_cpu_topology_trap; #define BIF_ALIST_1 Process* A__p, Eterm* BIF__ARGS #define BIF_ALIST_2 Process* A__p, Eterm* BIF__ARGS #define BIF_ALIST_3 Process* A__p, Eterm* BIF__ARGS +#define BIF_ALIST_4 Process* A__p, Eterm* BIF__ARGS #define BIF_ARG_1 (BIF__ARGS[0]) #define BIF_ARG_2 (BIF__ARGS[1]) #define BIF_ARG_3 (BIF__ARGS[2]) +#define BIF_ARG_4 (BIF__ARGS[3]) #define ERTS_IS_PROC_OUT_OF_REDS(p) \ ((p)->fcalls > 0 \ diff --git a/erts/emulator/beam/bif.tab b/erts/emulator/beam/bif.tab index 1d0d214e77..471f687101 100644 --- a/erts/emulator/beam/bif.tab +++ b/erts/emulator/beam/bif.tab @@ -92,6 +92,8 @@ bif erlang:loaded/0 bif erlang:localtime/0 bif erlang:localtime_to_universaltime/2 bif erlang:make_ref/0 +bif erlang:unique_integer/0 +bif erlang:unique_integer/1 bif erlang:md5/1 bif erlang:md5_init/0 bif erlang:md5_update/2 @@ -104,6 +106,13 @@ ubif erlang:node/1 ubif erlang:node/0 bif erlang:nodes/1 bif erlang:now/0 +bif erlang:monotonic_time/0 +bif erlang:monotonic_time/1 +bif erlang:system_time/0 +bif erlang:system_time/1 +bif erlang:time_offset/0 +bif erlang:time_offset/1 +bif erlang:timestamp/0 bif erlang:open_port/2 @@ -157,6 +166,17 @@ bif erts_internal:request_system_task/3 bif erts_internal:check_process_code/2 bif erts_internal:map_to_tuple_keys/1 +bif erts_internal:map_type/1 +bif erts_internal:map_hashmap_children/1 + +bif erts_internal:time_unit/0 + +bif erts_internal:get_bif_timer_servers/0 +bif erts_internal:create_bif_timer/0 +bif erts_internal:access_bif_timer/1 + +bif erts_internal:monitor_process/2 +bif erts_internal:is_system_process/1 # inet_db support bif erlang:port_set_data/2 @@ -200,11 +220,6 @@ bif math:sqrt/1 bif math:atan2/2 bif math:pow/2 -bif erlang:start_timer/3 -bif erlang:send_after/3 -bif erlang:cancel_timer/1 -bif erlang:read_timer/1 - bif erlang:make_tuple/2 bif erlang:append_element/2 bif erlang:make_tuple/3 @@ -348,6 +363,8 @@ bif os:getenv/0 bif os:getenv/1 bif os:getpid/0 bif os:timestamp/0 +bif os:system_time/0 +bif os:system_time/1 # # Bifs in the erl_ddll module (the module actually does not exist) @@ -613,6 +630,8 @@ bif erlang:fun_info_mfa/1 # bif erlang:get_keys/0 +bif ets:update_counter/4 +bif erts_debug:map_info/1 # # Obsolete diff --git a/erts/emulator/beam/big.c b/erts/emulator/beam/big.c index de7d370938..a4ea9c59ca 100644 --- a/erts/emulator/beam/big.c +++ b/erts/emulator/beam/big.c @@ -1577,6 +1577,46 @@ Eterm erts_sint64_to_big(Sint64 x, Eterm **hpp) return make_big(hp); } +Eterm +erts_uint64_array_to_big(Uint **hpp, int neg, int len, Uint64 *array) +{ + Uint *headerp; + int i, pot_digits, digits; + + headerp = *hpp; + + pot_digits = digits = 0; + for (i = 0; i < len; i++) { +#if defined(ARCH_32) || HALFWORD_HEAP + Uint low_val = array[i] & ((Uint) 0xffffffff); + Uint high_val = (array[i] >> 32) & ((Uint) 0xffffffff); + BIG_DIGIT(headerp, pot_digits) = low_val; + pot_digits++; + if (low_val) + digits = pot_digits; + BIG_DIGIT(headerp, pot_digits) = high_val; + pot_digits++; + if (high_val) + digits = pot_digits; +#else + Uint val = array[i]; + BIG_DIGIT(headerp, pot_digits) = val; + pot_digits++; + if (val) + digits = pot_digits; +#endif + } + + if (neg) + *headerp = make_neg_bignum_header(digits); + else + *headerp = make_pos_bignum_header(digits); + + *hpp = headerp + 1 + digits; + + return make_big(headerp); +} + /* ** Convert a bignum to a double float */ @@ -1900,6 +1940,8 @@ Eterm bytes_to_big(byte *xp, dsize_t xsz, int xsgn, Eterm *r) *rwp = d; rwp++; } + if (rsz > BIG_ARITY_MAX) + return NIL; if (xsgn) { *r = make_neg_bignum_header(rsz); } diff --git a/erts/emulator/beam/big.h b/erts/emulator/beam/big.h index da31876d75..4e4611de16 100644 --- a/erts/emulator/beam/big.h +++ b/erts/emulator/beam/big.h @@ -104,6 +104,9 @@ typedef Uint dsize_t; /* Vector size type */ : ERTS_UINT64_BIG_HEAP_SIZE__((X) >= 0 ? (X) : -(Uint64)(X))) #define ERTS_UINT64_HEAP_SIZE(X) \ (IS_USMALL(0, (X)) ? 0 : ERTS_UINT64_BIG_HEAP_SIZE__((X))) +#define ERTS_MAX_SINT64_HEAP_SIZE (1 + 2) +#define ERTS_MAX_UINT64_HEAP_SIZE (1 + 2) +#define ERTS_UINT64_ARRAY_TO_BIG_MAX_HEAP_SZ(LEN) (2*(LEN)+1) #else @@ -111,6 +114,9 @@ typedef Uint dsize_t; /* Vector size type */ (IS_SSMALL((X)) ? 0 : (1 + 1)) #define ERTS_UINT64_HEAP_SIZE(X) \ (IS_USMALL(0, (X)) ? 0 : (1 + 1)) +#define ERTS_MAX_SINT64_HEAP_SIZE (1 + 1) +#define ERTS_MAX_UINT64_HEAP_SIZE (1 + 1) +#define ERTS_UINT64_ARRAY_TO_BIG_MAX_HEAP_SZ(LEN) ((LEN)+1) #endif @@ -156,6 +162,7 @@ int term_to_Uint(Eterm, Uint*); int term_to_UWord(Eterm, UWord*); int term_to_Sint(Eterm, Sint*); #if HAVE_INT64 +Eterm erts_uint64_array_to_big(Uint **, int, int, Uint64 *); int term_to_Uint64(Eterm, Uint64*); int term_to_Sint64(Eterm, Sint64*); #endif diff --git a/erts/emulator/beam/break.c b/erts/emulator/beam/break.c index 4ede2c9d7d..e2fa572546 100644 --- a/erts/emulator/beam/break.c +++ b/erts/emulator/beam/break.c @@ -661,7 +661,6 @@ erl_crash_dump_v(char *file, int line, char* fmt, va_list args) { #ifdef ERTS_SMP ErtsThrPrgrData tpd_buf; /* in case we aren't a managed thread... */ - int bc; #endif int fd; size_t envsz; @@ -681,7 +680,7 @@ erl_crash_dump_v(char *file, int line, char* fmt, va_list args) /* Order all managed threads to block, this has to be done first to guarantee that this is the only thread to generate crash dump. */ - bc = erts_thr_progress_fatal_error_block(&tpd_buf); + erts_thr_progress_fatal_error_block(&tpd_buf); #ifdef ERTS_THR_HAVE_SIG_FUNCS /* diff --git a/erts/emulator/beam/copy.c b/erts/emulator/beam/copy.c index 0010f6a440..4d12dae787 100644 --- a/erts/emulator/beam/copy.c +++ b/erts/emulator/beam/copy.c @@ -127,6 +127,52 @@ Uint size_object(Eterm obj) obj = *bptr; break; } + case MAP_SUBTAG: + switch (MAP_HEADER_TYPE(hdr)) { + case MAP_HEADER_TAG_FLATMAP_HEAD : + { + Uint n; + flatmap_t *mp; + mp = (flatmap_t*)flatmap_val_rel(obj,base); + ptr = (Eterm *)mp; + n = flatmap_get_size(mp) + 1; + sum += n + 2; + ptr += 2; /* hdr + size words */ + while (n--) { + obj = *ptr++; + if (!IS_CONST(obj)) { + ESTACK_PUSH(s, obj); + } + } + goto pop_next; + } + case MAP_HEADER_TAG_HAMT_HEAD_BITMAP : + case MAP_HEADER_TAG_HAMT_HEAD_ARRAY : + case MAP_HEADER_TAG_HAMT_NODE_BITMAP : + { + Eterm *head; + Uint sz; + head = hashmap_val_rel(obj, base); + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + sum += 1 + sz + header_arity(hdr); + head += 1 + header_arity(hdr); + + if (sz == 0) { + goto pop_next; + } + while(sz-- > 1) { + obj = head[sz]; + if (!IS_CONST(obj)) { + ESTACK_PUSH(s, obj); + } + } + obj = head[0]; + } + break; + default: + erl_exit(ERTS_ABORT_EXIT, "size_object: bad hashmap type %d\n", MAP_HEADER_TYPE(hdr)); + } + break; case SUB_BINARY_SUBTAG: { Eterm real_bin; @@ -154,25 +200,7 @@ Uint size_object(Eterm obj) goto pop_next; } break; - case MAP_SUBTAG: - { - Uint n; - map_t *mp; - mp = (map_t*)map_val_rel(obj,base); - ptr = (Eterm *)mp; - n = map_get_size(mp) + 1; - sum += n + 2; - ptr += 2; /* hdr + size words */ - while (n--) { - obj = *ptr++; - if (!IS_CONST(obj)) { - ESTACK_PUSH(s, obj); - } - } - goto pop_next; - } - break; - case BIN_MATCHSTATE_SUBTAG: + case BIN_MATCHSTATE_SUBTAG: erl_exit(ERTS_ABORT_EXIT, "size_object: matchstate term not allowed"); default: @@ -340,15 +368,6 @@ Eterm copy_struct(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap) } } break; - case MAP_SUBTAG: - { - i = map_get_size(objp) + 3; - *argp = make_map_rel(htop, dst_base); - while (i--) { - *htop++ = *objp++; - } - } - break; case REFC_BINARY_SUBTAG: { ProcBin* pb; @@ -459,7 +478,7 @@ Eterm copy_struct(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap) { ExternalThing *etp = (ExternalThing *) htop; - i = thing_arityval(hdr) + 1; + i = thing_arityval(hdr) + 1; tp = htop; while (i--) { @@ -473,6 +492,28 @@ Eterm copy_struct(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap) *argp = make_external_rel(tp, dst_base); } break; + case MAP_SUBTAG: + tp = htop; + switch (MAP_HEADER_TYPE(hdr)) { + case MAP_HEADER_TAG_FLATMAP_HEAD : + i = flatmap_get_size(objp) + 3; + *argp = make_flatmap_rel(htop, dst_base); + while (i--) { + *htop++ = *objp++; + } + break; + case MAP_HEADER_TAG_HAMT_HEAD_BITMAP : + case MAP_HEADER_TAG_HAMT_HEAD_ARRAY : + *htop++ = *objp++; + case MAP_HEADER_TAG_HAMT_NODE_BITMAP : + i = 1 + hashmap_bitcount(MAP_HEADER_VAL(hdr)); + while (i--) { *htop++ = *objp++; } + *argp = make_hashmap_rel(tp, dst_base); + break; + default: + erl_exit(ERTS_ABORT_EXIT, "copy_struct: bad hashmap type %d\n", MAP_HEADER_TYPE(hdr)); + } + break; case BIN_MATCHSTATE_SUBTAG: erl_exit(ERTS_ABORT_EXIT, "copy_struct: matchstate term not allowed"); diff --git a/erts/emulator/beam/erl_alloc.types b/erts/emulator/beam/erl_alloc.types index 61def65235..e2f8da38b9 100644 --- a/erts/emulator/beam/erl_alloc.types +++ b/erts/emulator/beam/erl_alloc.types @@ -269,6 +269,7 @@ type BUSY_CALLER_TAB SHORT_LIVED SYSTEM busy_caller_table type BUSY_CALLER SHORT_LIVED SYSTEM busy_caller type PROC_SYS_TSK SHORT_LIVED PROCESSES proc_sys_task type PROC_SYS_TSK_QS SHORT_LIVED PROCESSES proc_sys_task_queues +type NEW_TIME_OFFSET SHORT_LIVED SYSTEM new_time_offset +if threads_no_smp # Need thread safe allocs, but std_alloc and fix_alloc are not; @@ -364,6 +365,7 @@ type AINFO_REQ STANDARD_LOW SYSTEM alloc_info_request type SCHED_WTIME_REQ STANDARD_LOW SYSTEM sched_wall_time_request type GC_INFO_REQ STANDARD_LOW SYSTEM gc_info_request type PORT_DATA_HEAP STANDARD_LOW SYSTEM port_data_heap +type BIF_TIMER_DATA LONG_LIVED_LOW SYSTEM bif_timer_data +else # "fullword" @@ -384,6 +386,7 @@ type AINFO_REQ SHORT_LIVED SYSTEM alloc_info_request type SCHED_WTIME_REQ SHORT_LIVED SYSTEM sched_wall_time_request type GC_INFO_REQ SHORT_LIVED SYSTEM gc_info_request type PORT_DATA_HEAP STANDARD SYSTEM port_data_heap +type BIF_TIMER_DATA LONG_LIVED SYSTEM bif_timer_data +endif diff --git a/erts/emulator/beam/erl_bif_binary.c b/erts/emulator/beam/erl_bif_binary.c index bd0d7c71cc..934904d58e 100644 --- a/erts/emulator/beam/erl_bif_binary.c +++ b/erts/emulator/beam/erl_bif_binary.c @@ -38,6 +38,7 @@ #include "big.h" #include "erl_binary.h" #include "erl_bits.h" +#include "erl_bif_unique.h" /* diff --git a/erts/emulator/beam/erl_bif_ddll.c b/erts/emulator/beam/erl_bif_ddll.c index 56cd2ba04f..fc4f819f56 100644 --- a/erts/emulator/beam/erl_bif_ddll.c +++ b/erts/emulator/beam/erl_bif_ddll.c @@ -45,6 +45,7 @@ #include "big.h" #include "dist.h" #include "erl_version.h" +#include "erl_bif_unique.h" #include "dtrace-wrapper.h" #ifdef ERTS_SMP diff --git a/erts/emulator/beam/erl_bif_guard.c b/erts/emulator/beam/erl_bif_guard.c index bbd8aa31d9..e7d84ebda1 100644 --- a/erts/emulator/beam/erl_bif_guard.c +++ b/erts/emulator/beam/erl_bif_guard.c @@ -459,23 +459,25 @@ Eterm erts_gc_byte_size_1(Process* p, Eterm* reg, Uint live) Eterm erts_gc_map_size_1(Process* p, Eterm* reg, Uint live) { Eterm arg = reg[live]; - if (is_map(arg)) { - map_t *mp = (map_t*)map_val(arg); - Uint size = map_get_size(mp); - if (IS_USMALL(0, size)) { - return make_small(size); - } else { - Eterm* hp; - if (ERTS_NEED_GC(p, BIG_UINT_HEAP_SIZE)) { - erts_garbage_collect(p, BIG_UINT_HEAP_SIZE, reg, live); - } - hp = p->htop; - p->htop += BIG_UINT_HEAP_SIZE; - return uint_to_big(size, hp); - } + Eterm* hp; + Uint size; + if (is_flatmap(arg)) { + flatmap_t *mp = (flatmap_t*)flatmap_val(arg); + size = flatmap_get_size(mp); + } else if (is_hashmap(arg)) { + size = hashmap_size(arg); } else { BIF_ERROR(p, BADARG); } + if (IS_USMALL(0, size)) { + return make_small(size); + } + if (ERTS_NEED_GC(p, BIG_UINT_HEAP_SIZE)) { + erts_garbage_collect(p, BIG_UINT_HEAP_SIZE, reg, live); + } + hp = p->htop; + p->htop += BIG_UINT_HEAP_SIZE; + return uint_to_big(size, hp); } Eterm erts_gc_abs_1(Process* p, Eterm* reg, Uint live) diff --git a/erts/emulator/beam/erl_bif_info.c b/erts/emulator/beam/erl_bif_info.c index d750e34be3..b2658a1fd6 100644 --- a/erts/emulator/beam/erl_bif_info.c +++ b/erts/emulator/beam/erl_bif_info.c @@ -42,6 +42,7 @@ #include "erl_cpu_topology.h" #include "erl_async.h" #include "erl_thr_progress.h" +#include "erl_bif_unique.h" #define ERTS_PTAB_WANT_DEBUG_FUNCS__ #include "erl_ptab.h" #ifdef HIPE @@ -540,6 +541,7 @@ pi_locks(Eterm info) switch (info) { case am_status: case am_priority: + case am_trap_exit: return ERTS_PROC_LOCK_STATUS; case am_links: case am_monitors: @@ -592,7 +594,7 @@ static Eterm pi_args[] = { am_min_bin_vheap_size, am_current_location, am_current_stacktrace, -}; +}; #define ERTS_PI_ARGS ((int) (sizeof(pi_args)/sizeof(Eterm))) @@ -2102,6 +2104,48 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) BIF_RET(am_opt); #endif BIF_RET(res); + } else if (BIF_ARG_1 == am_time_offset) { + switch (erts_time_offset_state()) { + case ERTS_TIME_OFFSET_PRELIMINARY: { + ERTS_DECL_AM(preliminary); + BIF_RET(AM_preliminary); + } + case ERTS_TIME_OFFSET_FINAL: { + ERTS_DECL_AM(final); + BIF_RET(AM_final); + } + case ERTS_TIME_OFFSET_VOLATILE: { + ERTS_DECL_AM(volatile); + BIF_RET(AM_volatile); + } + default: + ERTS_INTERNAL_ERROR("Invalid time offset state"); + } + } else if (ERTS_IS_ATOM_STR("os_monotonic_time_source", BIF_ARG_1)) { + BIF_RET(erts_monotonic_time_source(BIF_P)); + } else if (ERTS_IS_ATOM_STR("os_system_time_source", BIF_ARG_1)) { + BIF_RET(erts_system_time_source(BIF_P)); + } else if (ERTS_IS_ATOM_STR("time_correction", BIF_ARG_1)) { + BIF_RET(erts_has_time_correction() ? am_true : am_false); + } else if (ERTS_IS_ATOM_STR("start_time", BIF_ARG_1)) { + BIF_RET(erts_get_monotonic_start_time(BIF_P)); + } else if (ERTS_IS_ATOM_STR("time_warp_mode", BIF_ARG_1)) { + switch (erts_time_warp_mode()) { + case ERTS_NO_TIME_WARP_MODE: { + ERTS_DECL_AM(no_time_warp); + BIF_RET(AM_no_time_warp); + } + case ERTS_SINGLE_TIME_WARP_MODE: { + ERTS_DECL_AM(single_time_warp); + BIF_RET(AM_single_time_warp); + } + case ERTS_MULTI_TIME_WARP_MODE: { + ERTS_DECL_AM(multi_time_warp); + BIF_RET(AM_multi_time_warp); + } + default: + ERTS_INTERNAL_ERROR("Invalid time warp mode"); + } } else if (BIF_ARG_1 == am_allocated_areas) { res = erts_allocated_areas(NULL, NULL, BIF_P); BIF_RET(res); @@ -2703,9 +2747,11 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) BIF_RET(make_small(erts_db_get_max_tabs())); } else if (ERTS_IS_ATOM_STR("tolerant_timeofday",BIF_ARG_1)) { - BIF_RET(erts_disable_tolerant_timeofday - ? am_disabled - : am_enabled); + if (erts_has_time_correction() + && erts_time_offset_state() == ERTS_TIME_OFFSET_FINAL) { + BIF_RET(am_enabled); + } + BIF_RET(am_disabled); } else if (ERTS_IS_ATOM_STR("eager_check_io",BIF_ARG_1)) { BIF_RET(erts_eager_check_io ? am_true : am_false); @@ -3403,6 +3449,29 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1) else if (ERTS_IS_ATOM_STR("mmap", BIF_ARG_1)) { BIF_RET(erts_mmap_debug_info(BIF_P)); } + else if (ERTS_IS_ATOM_STR("unique_monotonic_integer_state", BIF_ARG_1)) { + BIF_RET(erts_debug_get_unique_monotonic_integer_state(BIF_P)); + } + else if (ERTS_IS_ATOM_STR("min_unique_monotonic_integer", BIF_ARG_1)) { + Sint64 value = erts_get_min_unique_monotonic_integer(); + if (IS_SSMALL(value)) + BIF_RET(make_small(value)); + else { + Uint hsz = ERTS_SINT64_HEAP_SIZE(value); + Eterm *hp = HAlloc(BIF_P, hsz); + BIF_RET(erts_sint64_to_big(value, &hp)); + } + } + else if (ERTS_IS_ATOM_STR("min_unique_integer", BIF_ARG_1)) { + Sint64 value = erts_get_min_unique_integer(); + if (IS_SSMALL(value)) + BIF_RET(make_small(value)); + else { + Uint hsz = ERTS_SINT64_HEAP_SIZE(value); + Eterm *hp = HAlloc(BIF_P, hsz); + BIF_RET(erts_sint64_to_big(value, &hp)); + } + } } else if (is_tuple(BIF_ARG_1)) { Eterm* tp = tuple_val(BIF_ARG_1); @@ -3597,6 +3666,58 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1) BIF_RET(erts_debug_reader_groups_map(BIF_P, (int) groups)); } + else if (ERTS_IS_ATOM_STR("internal_hash", tp[1])) { + Uint hash = (Uint) make_internal_hash(tp[2]); + Uint hsz = 0; + Eterm* hp; + erts_bld_uint(NULL, &hsz, hash); + hp = HAlloc(BIF_P,hsz); + return erts_bld_uint(&hp, NULL, hash); + } + else if (ERTS_IS_ATOM_STR("atom", tp[1])) { + Uint ix; + if (!term_to_Uint(tp[2], &ix)) + BIF_ERROR(BIF_P, BADARG); + while (ix >= atom_table_size()) { + char tmp[20]; + erts_snprintf(tmp, sizeof(tmp), "am%x", atom_table_size()); + erts_atom_put((byte *) tmp, strlen(tmp), ERTS_ATOM_ENC_LATIN1, 1); + } + return make_atom(ix); + } + + break; + } + case 3: { + if (ERTS_IS_ATOM_STR("check_time_config", tp[1])) { + int res, time_correction; + ErtsTimeWarpMode time_warp_mode; + if (tp[2] == am_true) + time_correction = !0; + else if (tp[2] == am_false) + time_correction = 0; + else + break; + if (ERTS_IS_ATOM_STR("no_time_warp", tp[3])) + time_warp_mode = ERTS_NO_TIME_WARP_MODE; + else if (ERTS_IS_ATOM_STR("single_time_warp", tp[3])) + time_warp_mode = ERTS_SINGLE_TIME_WARP_MODE; + else if (ERTS_IS_ATOM_STR("multi_time_warp", tp[3])) + time_warp_mode = ERTS_MULTI_TIME_WARP_MODE; + else + break; + res = erts_check_time_adj_support(time_correction, + time_warp_mode); + BIF_RET(res ? am_true : am_false); + } + else if (ERTS_IS_ATOM_STR("make_unique_integer", tp[1])) { + Eterm res = erts_debug_make_unique_integer(BIF_P, + tp[2], + tp[3]); + if (is_non_value(res)) + break; + BIF_RET(res); + } break; } default: @@ -3606,6 +3727,24 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1) BIF_ERROR(BIF_P, BADARG); } +BIF_RETTYPE erts_internal_is_system_process_1(BIF_ALIST_1) +{ + if (is_internal_pid(BIF_ARG_1)) { + Process *rp = erts_proc_lookup(BIF_ARG_1); + if (rp && (rp->static_flags & ERTS_STC_FLG_SYSTEM_PROC)) + BIF_RET(am_true); + BIF_RET(am_false); + } + + if (is_external_pid(BIF_ARG_1) + && external_pid_dist_entry(BIF_ARG_1) == erts_this_dist_entry) { + BIF_RET(am_false); + } + + BIF_ERROR(BIF_P, BADARG); +} + + static erts_smp_atomic_t hipe_test_reschedule_flag; @@ -3907,6 +4046,10 @@ BIF_RETTYPE erts_debug_set_internal_state_2(BIF_ALIST_2) #endif erl_exit(ERTS_DUMP_EXIT, "%T", BIF_ARG_2); } + else if (ERTS_IS_ATOM_STR("unique_monotonic_integer_state", BIF_ARG_1)) { + int res = erts_debug_set_unique_monotonic_integer_state(BIF_ARG_2); + BIF_RET(res ? am_true : am_false); + } } BIF_ERROR(BIF_P, BADARG); diff --git a/erts/emulator/beam/erl_bif_timer.c b/erts/emulator/beam/erl_bif_timer.c index 03ac97283c..0bd8d20c34 100644 --- a/erts/emulator/beam/erl_bif_timer.c +++ b/erts/emulator/beam/erl_bif_timer.c @@ -27,6 +27,7 @@ #include "error.h" #include "big.h" #include "erl_thr_progress.h" +#include "erl_bif_unique.h" /**************************************************************************** ** BIF Timer support @@ -480,7 +481,7 @@ setup_bif_timer(Uint32 xflags, tab_insert(btm); ASSERT(btm == tab_find(ref)); - btm->tm.active = 0; /* MUST be initalized */ + erts_init_timer(&btm->tm); erts_set_timer(&btm->tm, (ErlTimeoutProc) bif_timer_timeout, (ErlCancelProc) bif_timer_cleanup, @@ -489,8 +490,9 @@ setup_bif_timer(Uint32 xflags, return ref; } +BIF_RETTYPE old_send_after_3(BIF_ALIST_3); /* send_after(Time, Pid, Message) -> Ref */ -BIF_RETTYPE send_after_3(BIF_ALIST_3) +BIF_RETTYPE old_send_after_3(BIF_ALIST_3) { Eterm res; @@ -510,8 +512,9 @@ BIF_RETTYPE send_after_3(BIF_ALIST_3) } } +BIF_RETTYPE old_start_timer_3(BIF_ALIST_3); /* start_timer(Time, Pid, Message) -> Ref */ -BIF_RETTYPE start_timer_3(BIF_ALIST_3) +BIF_RETTYPE old_start_timer_3(BIF_ALIST_3) { Eterm res; @@ -531,8 +534,9 @@ BIF_RETTYPE start_timer_3(BIF_ALIST_3) } } +BIF_RETTYPE old_cancel_timer_1(BIF_ALIST_1); /* cancel_timer(Ref) -> false | RemainingTime */ -BIF_RETTYPE cancel_timer_1(BIF_ALIST_1) +BIF_RETTYPE old_cancel_timer_1(BIF_ALIST_1) { Eterm res; ErtsBifTimer *btm; @@ -569,8 +573,9 @@ BIF_RETTYPE cancel_timer_1(BIF_ALIST_1) BIF_RET(res); } +BIF_RETTYPE old_read_timer_1(BIF_ALIST_1); /* read_timer(Ref) -> false | RemainingTime */ -BIF_RETTYPE read_timer_1(BIF_ALIST_1) +BIF_RETTYPE old_read_timer_1(BIF_ALIST_1) { Eterm res; ErtsBifTimer *btm; @@ -652,7 +657,7 @@ erts_cancel_bif_timers(Process *p, ErtsProcLocks plocks) erts_smp_btm_rwunlock(); } -void erts_bif_timer_init(void) +static void erts_old_bif_timer_init(void) { int i; no_bif_timers = 0; @@ -703,3 +708,146 @@ erts_bif_timer_foreach(void (*func)(Eterm, Eterm, ErlHeapFragment *, void *), } } } + +typedef struct { + Uint ref_heap[REF_THING_SIZE]; + Eterm pid[1]; +} ErtsBifTimerServers; + +static ErtsBifTimerServers *bif_timer_servers; + +void erts_bif_timer_init(void) +{ + erts_old_bif_timer_init(); +} + +void +erts_bif_timer_start_servers(Eterm parent) +{ + Process *parent_proc; + Eterm *hp, btr_ref, arg_list_end; + ErlSpawnOpts so; + int i; + + bif_timer_servers = erts_alloc(ERTS_ALC_T_BIF_TIMER_DATA, + (sizeof(ErtsBifTimerServers) + + (sizeof(Eterm)*(erts_no_schedulers-1)))); + + so.flags = SPO_USE_ARGS|SPO_SYSTEM_PROC|SPO_PREFER_SCHED|SPO_OFF_HEAP_MSGS; + so.min_heap_size = H_MIN_SIZE; + so.min_vheap_size = BIN_VH_MIN_SIZE; + so.priority = PRIORITY_MAX; + so.max_gen_gcs = (Uint16) erts_smp_atomic32_read_nob(&erts_max_gen_gcs); + + /* + * Parent is "init" and schedulers have not yet been started, so it + * *should* be alive and well... + */ + ASSERT(is_internal_pid(parent)); + parent_proc = (Process *) erts_ptab_pix2intptr_ddrb(&erts_proc, + internal_pid_index(parent)); + ASSERT(parent_proc); + ASSERT(parent_proc->common.id == parent); + ASSERT(!ERTS_PROC_IS_EXITING(parent_proc)); + + erts_smp_proc_lock(parent_proc, ERTS_PROC_LOCK_MAIN); + + hp = HAlloc(parent_proc, 2*erts_no_schedulers + 2 + REF_THING_SIZE); + + btr_ref = erts_make_ref_in_buffer(hp); + hp += REF_THING_SIZE; + + arg_list_end = CONS(hp, btr_ref, NIL); + hp += 2; + + for (i = 0; i < erts_no_schedulers; i++) { + int sched = i+1; + Eterm arg_list = CONS(hp, make_small(i+1), arg_list_end); + hp += 2; + + so.scheduler = sched; /* Preferred scheduler */ + + bif_timer_servers->pid[i] = erl_create_process(parent_proc, + am_erts_internal, + am_bif_timer_server, + arg_list, + &so); + } + + erts_smp_proc_unlock(parent_proc, ERTS_PROC_LOCK_MAIN); + + hp = internal_ref_val(btr_ref); + for (i = 0; i < REF_THING_SIZE; i++) + bif_timer_servers->ref_heap[i] = hp[i]; +} + +BIF_RETTYPE +erts_internal_get_bif_timer_servers_0(BIF_ALIST_0) +{ + int i; + Eterm *hp, res = NIL; + + hp = HAlloc(BIF_P, erts_no_schedulers*2); + for (i = erts_no_schedulers-1; i >= 0; i--) { + res = CONS(hp, bif_timer_servers->pid[i], res); + hp += 2; + } + BIF_RET(res); +} + +BIF_RETTYPE +erts_internal_access_bif_timer_1(BIF_ALIST_1) +{ + int ix; + Uint32 *rdp; + Eterm ref, pid, *hp, res; + + if (is_not_internal_ref(BIF_ARG_1)) { + if (is_not_ref(BIF_ARG_1)) + BIF_ERROR(BIF_P, BADARG); + BIF_RET(am_undefined); + } + + rdp = internal_ref_numbers(BIF_ARG_1); + ix = (int) erts_get_ref_numbers_thr_id(rdp); + if (ix < 1 || erts_no_schedulers < ix) + BIF_RET(am_undefined); + + pid = bif_timer_servers->pid[ix-1]; + ASSERT(is_internal_pid(pid)); + + hp = HAlloc(BIF_P, 3 /* 2-tuple */ + REF_THING_SIZE); + for (ix = 0; ix < REF_THING_SIZE; ix++) + hp[ix] = bif_timer_servers->ref_heap[ix]; + ref = make_internal_ref(&hp[0]); + hp += REF_THING_SIZE; + + res = TUPLE2(hp, ref, pid); + BIF_RET(res); +} + +BIF_RETTYPE +erts_internal_create_bif_timer_0(BIF_ALIST_0) +{ + ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(BIF_P); + Eterm *hp, btr_ref, t_ref, pid, res; + int ix; + + hp = HAlloc(BIF_P, 4 /* 3-tuple */ + 2*REF_THING_SIZE); + for (ix = 0; ix < REF_THING_SIZE; ix++) + hp[ix] = bif_timer_servers->ref_heap[ix]; + btr_ref = make_internal_ref(&hp[0]); + hp += REF_THING_SIZE; + + t_ref = erts_sched_make_ref_in_buffer(esdp, hp); + hp += REF_THING_SIZE; + + ASSERT(erts_get_ref_numbers_thr_id(internal_ref_numbers(t_ref)) + == (Uint32) esdp->no); + + pid = bif_timer_servers->pid[((int) esdp->no) - 1]; + + res = TUPLE3(hp, btr_ref, pid, t_ref); + + BIF_RET(res); +} diff --git a/erts/emulator/beam/erl_bif_timer.h b/erts/emulator/beam/erl_bif_timer.h index 1197c176f5..c2f5dfd3c3 100644 --- a/erts/emulator/beam/erl_bif_timer.h +++ b/erts/emulator/beam/erl_bif_timer.h @@ -33,4 +33,5 @@ void erts_cancel_bif_timers(Process *p, ErtsProcLocks plocks); void erts_bif_timer_init(void); void erts_bif_timer_foreach(void (*func)(Eterm,Eterm,ErlHeapFragment *,void *), void *arg); +void erts_bif_timer_start_servers(Eterm); #endif diff --git a/erts/emulator/beam/erl_bif_trace.c b/erts/emulator/beam/erl_bif_trace.c index f5e582b1c5..ac57205c47 100644 --- a/erts/emulator/beam/erl_bif_trace.c +++ b/erts/emulator/beam/erl_bif_trace.c @@ -38,6 +38,7 @@ #include "beam_bp.h" #include "erl_binary.h" #include "erl_thr_progress.h" +#include "erl_bif_unique.h" #define DECL_AM(S) Eterm AM_ ## S = am_atom_put(#S, sizeof(#S) - 1) diff --git a/erts/emulator/beam/erl_bif_unique.c b/erts/emulator/beam/erl_bif_unique.c new file mode 100644 index 0000000000..57b0bab72f --- /dev/null +++ b/erts/emulator/beam/erl_bif_unique.c @@ -0,0 +1,556 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2014. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#include "sys.h" +#include "erl_vm.h" +#include "erl_alloc.h" +#include "export.h" +#include "bif.h" +#include "erl_bif_unique.h" + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Reference * +\* */ + +static union { + erts_atomic64_t count; + char align__[ERTS_CACHE_LINE_SIZE]; +} global_reference erts_align_attribute(ERTS_CACHE_LINE_SIZE); + + +/* + * ref[0] indicate thread creating reference as follows: + * + * - ref[0] == 0 => Non-scheduler thread; + * - else; ref[0] <= erts_no_schedulers => + * ordinary scheduler with id == ref[0]; + * - else; ref[0] <= erts_no_schedulers + * + erts_no_dirty_cpu_schedulers => + * dirty cpu scheduler with id == 'ref[0] - erts_no_schedulers'; + * - else => + * dirty io scheduler with id == 'ref[0] + * - erts_no_schedulers + * - erts_no_dirty_cpu_schedulers' + */ + +#ifdef DEBUG +static Uint32 max_thr_id; +#endif + +static void +init_reference(void) +{ +#ifdef DEBUG + max_thr_id = (Uint32) erts_no_schedulers; +#ifdef ERTS_DIRTY_SCHEDULERS + max_thr_id += (Uint32) erts_no_dirty_cpu_schedulers; + max_thr_id += (Uint32) erts_no_dirty_io_schedulers; +#endif +#endif + erts_atomic64_init_nob(&global_reference.count, 0); +} + +static ERTS_INLINE void +global_make_ref_in_array(Uint32 thr_id, Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + Uint64 value; + + value = (Uint64) erts_atomic64_inc_read_mb(&global_reference.count); + + erts_set_ref_numbers(ref, thr_id, value); +} + +static ERTS_INLINE void +make_ref_in_array(Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + ErtsSchedulerData *esdp = erts_get_scheduler_data(); + if (esdp) + erts_sched_make_ref_in_array(esdp, ref); + else + global_make_ref_in_array(0, ref); +} + +void +erts_make_ref_in_array(Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + make_ref_in_array(ref); +} + +Eterm erts_make_ref_in_buffer(Eterm buffer[REF_THING_SIZE]) +{ + Eterm* hp = buffer; + Uint32 ref[ERTS_MAX_REF_NUMBERS]; + + make_ref_in_array(ref); + write_ref_thing(hp, ref[0], ref[1], ref[2]); + return make_internal_ref(hp); +} + +Eterm erts_make_ref(Process *c_p) +{ + Eterm* hp; + Uint32 ref[ERTS_MAX_REF_NUMBERS]; + + ERTS_SMP_LC_ASSERT(ERTS_PROC_LOCK_MAIN & erts_proc_lc_my_proc_locks(c_p)); + + hp = HAlloc(c_p, REF_THING_SIZE); + + make_ref_in_array(ref); + write_ref_thing(hp, ref[0], ref[1], ref[2]); + + return make_internal_ref(hp); +} + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Unique Integer * +\* */ + +static struct { + union { + struct { + int left_shift; + int right_shift; + Uint64 mask; + Uint64 val0_max; + } o; + char align__[ERTS_CACHE_LINE_SIZE]; + } r; + union { + erts_atomic64_t val1; + char align__[ERTS_CACHE_LINE_SIZE]; + } w; +} unique_data erts_align_attribute(ERTS_CACHE_LINE_SIZE); + +static void +init_unique_integer(void) +{ + int bits; + unique_data.r.o.val0_max = (Uint64) erts_no_schedulers; +#ifdef ERTS_DIRTY_SCHEDULERS + unique_data.r.o.val0_max += (Uint64) erts_no_dirty_cpu_schedulers; + unique_data.r.o.val0_max += (Uint64) erts_no_dirty_io_schedulers; +#endif + bits = erts_fit_in_bits_int64(unique_data.r.o.val0_max); + unique_data.r.o.left_shift = bits; + unique_data.r.o.right_shift = 64 - bits; + unique_data.r.o.mask = (((Uint64) 1) << bits) - 1; + erts_atomic64_init_nob(&unique_data.w.val1, -1); +} + +#define ERTS_MAX_UNIQUE_INT_HEAP_SIZE ERTS_UINT64_ARRAY_TO_BIG_MAX_HEAP_SZ(2) + +static ERTS_INLINE Eterm +bld_unique_integer_term(Eterm **hpp, Uint *szp, + Uint64 val0, Uint64 val1, + int positive) +{ + Uint hsz; + Uint64 unique_val[2]; + + unique_val[0] = ((Uint64) val0); + unique_val[0] |= ((Uint64) val1) << unique_data.r.o.left_shift; + unique_val[1] = ((Uint64) val1) >> unique_data.r.o.right_shift; + unique_val[1] &= unique_data.r.o.mask; + + if (positive) { + unique_val[0]++; + if (unique_val[0] == 0) + unique_val[1]++; + } + else { + ASSERT(MIN_SMALL < 0); + if (unique_val[1] == 0 + && unique_val[0] < ((Uint64) -1*((Sint64) MIN_SMALL))) { + Sint64 s_unique_val = (Sint64) unique_val[0]; + s_unique_val += MIN_SMALL; + ASSERT(MIN_SMALL <= s_unique_val && s_unique_val < 0); + if (szp) + *szp = 0; + if (!hpp) + return THE_NON_VALUE; + return make_small((Sint) s_unique_val); + } + if (unique_val[0] < ((Uint64) -1*((Sint64) MIN_SMALL))) { + ASSERT(unique_val[1] != 0); + unique_val[1] -= 1; + } + unique_val[0] += MIN_SMALL; + } + + if (!unique_val[1]) { + if (unique_val[0] <= MAX_SMALL) { + if (szp) + *szp = 0; + if (!hpp) + return THE_NON_VALUE; + return make_small((Uint) unique_val[0]); + } + + if (szp) + *szp = ERTS_UINT64_HEAP_SIZE(unique_val[0]); + if (!hpp) + return THE_NON_VALUE; + return erts_uint64_to_big(unique_val[0], hpp); + } + else { + Eterm tmp, *tmp_hp, res; + DeclareTmpHeapNoproc(local_heap, 2*ERTS_MAX_UNIQUE_INT_HEAP_SIZE); + + UseTmpHeapNoproc(2*ERTS_MAX_UNIQUE_INT_HEAP_SIZE); + + tmp_hp = local_heap; + + tmp = erts_uint64_array_to_big(&tmp_hp, 0, 2, unique_val); + ASSERT(is_big(tmp)); + + hsz = big_arity(tmp) + 1; + + ASSERT(hsz <= ERTS_MAX_UNIQUE_INT_HEAP_SIZE); + + if (szp) + *szp = hsz; + + if (!hpp) + res = THE_NON_VALUE; + else { + int hix; + Eterm *hp = *hpp; + tmp_hp = big_val(tmp); + for (hix = 0; hix < hsz; hix++) + hp[hix] = tmp_hp[hix]; + + *hpp = hp + hsz; + res = make_big(hp); + } + + UnUseTmpHeapNoproc(2*ERTS_MAX_UNIQUE_INT_HEAP_SIZE); + + return res; + } +} + +static ERTS_INLINE Eterm unique_integer_bif(Process *c_p, int positive) +{ + ErtsSchedulerData *esdp; + Uint64 thr_id, unique; + Uint hsz; + Eterm *hp; + + esdp = ERTS_PROC_GET_SCHDATA(c_p); + thr_id = (Uint64) esdp->thr_id; + unique = esdp->unique++; + bld_unique_integer_term(NULL, &hsz, thr_id, unique, positive); + hp = hsz ? HAlloc(c_p, hsz) : NULL; + return bld_unique_integer_term(&hp, NULL, thr_id, unique, positive); +} + +Uint +erts_raw_unique_integer_heap_size(Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]) +{ + Uint sz; + bld_unique_integer_term(NULL, &sz, val[0], val[1], 0); + return sz; +} + +Eterm +erts_raw_make_unique_integer(Eterm **hpp, Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]) +{ + return bld_unique_integer_term(hpp, NULL, val[0], val[1], 0); +} + +void +erts_raw_get_unique_integer(Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]) +{ + ErtsSchedulerData *esdp = erts_get_scheduler_data(); + if (esdp) { + val[0] = (Uint64) esdp->thr_id; + val[1] = esdp->unique++; + } + else { + val[0] = (Uint64) 0; + val[1] = (Uint64) erts_atomic64_inc_read_nob(&unique_data.w.val1); + } +} + + +Sint64 +erts_get_min_unique_integer(void) +{ + return (Sint64) MIN_SMALL; +} + +/* --- Debug --- */ + +Eterm +erts_debug_make_unique_integer(Process *c_p, Eterm etval0, Eterm etval1) +{ + Uint64 val0, val1; + Uint hsz; + Eterm res, *hp, *end_hp; + + if (!term_to_Uint64(etval0, &val0)) + return THE_NON_VALUE; + + if (!term_to_Uint64(etval1, &val1)) + return THE_NON_VALUE; + + bld_unique_integer_term(NULL, &hsz, val0, val1, 0); + + hp = HAlloc(c_p, hsz); + end_hp = hp + hsz; + + res = bld_unique_integer_term(&hp, NULL, val0, val1, 0); + if (hp != end_hp) + ERTS_INTERNAL_ERROR("Heap allocation error"); + + return res; +} + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Strict Monotonic Counter * +\* */ + +static struct { + union { + erts_atomic64_t value; + char align__[ERTS_CACHE_LINE_SIZE]; + } w; +} raw_unique_monotonic_integer erts_align_attribute(ERTS_CACHE_LINE_SIZE); + +#if defined(ARCH_32) || HALFWORD_HEAP +# define ERTS_UNIQUE_MONOTONIC_OFFSET ERTS_SINT64_MIN +#else +# define ERTS_UNIQUE_MONOTONIC_OFFSET MIN_SMALL +#endif + +static void +init_unique_monotonic_integer(void) +{ + erts_atomic64_init_nob(&raw_unique_monotonic_integer.w.value, + (erts_aint64_t) -1); +} + +static ERTS_INLINE Uint64 +get_raw_unique_monotonic_integer(void) +{ + return (Uint64) erts_atomic64_inc_read_mb(&raw_unique_monotonic_integer.w.value); +} + +static ERTS_INLINE Uint +get_unique_monotonic_integer_heap_size(Uint64 raw, int positive) +{ + if (positive) { + Uint64 value = raw+1; + return ERTS_UINT64_HEAP_SIZE(value); + } + else { + Sint64 value = ((Sint64) raw) + ERTS_UNIQUE_MONOTONIC_OFFSET; + if (IS_SSMALL(value)) + return 0; +#if defined(ARCH_32) || HALFWORD_HEAP + return ERTS_SINT64_HEAP_SIZE(value); +#else + return ERTS_UINT64_HEAP_SIZE((Uint64) value); +#endif + } +} + +static ERTS_INLINE Eterm +make_unique_monotonic_integer_value(Eterm *hp, Uint hsz, Uint64 raw, int positive) +{ + Eterm res; +#ifdef DEBUG + Eterm *end_hp = hp + hsz; +#endif + + if (positive) { + Uint64 value = raw+1; + res = hsz ? erts_uint64_to_big(value, &hp) : make_small(value); + } + else { + Sint64 value = ((Sint64) raw) + ERTS_UNIQUE_MONOTONIC_OFFSET; + if (hsz == 0) + res = make_small(value); + else { +#if defined(ARCH_32) || HALFWORD_HEAP + res = erts_sint64_to_big(value, &hp); +#else + res = erts_uint64_to_big((Uint64) value, &hp); +#endif + } + } + + ASSERT(end_hp == hp); + + return res; +} + +static ERTS_INLINE Eterm +unique_monotonic_integer_bif(Process *c_p, int positive) +{ + Uint64 raw; + Uint hsz; + Eterm *hp; + + raw = get_raw_unique_monotonic_integer(); + hsz = get_unique_monotonic_integer_heap_size(raw, positive); + hp = hsz ? HAlloc(c_p, hsz) : NULL; + return make_unique_monotonic_integer_value(hp, hsz, raw, positive); +} + +Sint64 +erts_raw_get_unique_monotonic_integer(void) +{ + return get_raw_unique_monotonic_integer(); +} + +Uint +erts_raw_unique_monotonic_integer_heap_size(Sint64 raw) +{ + return get_unique_monotonic_integer_heap_size(raw, 0); +} + +Eterm +erts_raw_make_unique_monotonic_integer_value(Eterm **hpp, Sint64 raw) +{ + Uint hsz = get_unique_monotonic_integer_heap_size(raw, 0); + Eterm res = make_unique_monotonic_integer_value(*hpp, hsz, raw, 0); + *hpp += hsz; + return res; +} + +Sint64 +erts_get_min_unique_monotonic_integer(void) +{ + return ERTS_UNIQUE_MONOTONIC_OFFSET; +} + +/* --- Debug --- */ + +int +erts_debug_set_unique_monotonic_integer_state(Eterm et_value) +{ + Sint64 value; + + if (!term_to_Sint64(et_value, &value)) { + Uint64 uvalue; + if (!term_to_Uint64(et_value, &uvalue)) + return 0; + value = (Sint64) uvalue; + } + + erts_atomic64_set_mb(&raw_unique_monotonic_integer.w.value, + (erts_aint64_t) value); + return 1; +} + +Eterm +erts_debug_get_unique_monotonic_integer_state(Process *c_p) +{ + Uint64 value; + Eterm hsz, *hp; + + value = (Uint64) erts_atomic64_read_mb(&raw_unique_monotonic_integer.w.value); + + if (IS_USMALL(0, value)) + return make_small(value); + hsz = ERTS_UINT64_HEAP_SIZE(value); + hp = HAlloc(c_p, hsz); + return erts_uint64_to_big(value, &hp); +} + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Initilazation * +\* */ + +void +erts_bif_unique_init(void) +{ + init_reference(); + init_unique_monotonic_integer(); + init_unique_integer(); +} + +void +erts_sched_bif_unique_init(ErtsSchedulerData *esdp) +{ + esdp->unique = (Uint64) 0; + esdp->ref = (Uint64) 0; +} + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * The BIFs * +\* */ + + +BIF_RETTYPE make_ref_0(BIF_ALIST_0) +{ + BIF_RETTYPE res; + Eterm* hp; + + ERTS_SMP_LC_ASSERT(ERTS_PROC_LOCK_MAIN & erts_proc_lc_my_proc_locks(BIF_P)); + + hp = HAlloc(BIF_P, REF_THING_SIZE); + + res = erts_sched_make_ref_in_buffer(ERTS_PROC_GET_SCHDATA(BIF_P), hp); + + BIF_RET(res); +} + +BIF_RETTYPE unique_integer_0(BIF_ALIST_0) +{ + BIF_RET(unique_integer_bif(BIF_P, 0)); +} + +BIF_RETTYPE unique_integer_1(BIF_ALIST_1) +{ + Eterm modlist = BIF_ARG_1; + int monotonic = 0; + int positive = 0; + BIF_RETTYPE res; + + while (is_list(modlist)) { + Eterm *consp = list_val(modlist); + switch (CAR(consp)) { + case am_monotonic: + monotonic = 1; + break; + case am_positive: + positive = 1; + break; + default: + BIF_ERROR(BIF_P, BADARG); + } + modlist = CDR(consp); + } + + if (is_not_nil(modlist)) + BIF_ERROR(BIF_P, BADARG); + + if (monotonic) + res = unique_monotonic_integer_bif(BIF_P, positive); + else + res = unique_integer_bif(BIF_P, positive); + + BIF_RET(res); +} diff --git a/erts/emulator/beam/erl_bif_unique.h b/erts/emulator/beam/erl_bif_unique.h new file mode 100644 index 0000000000..cd001172a1 --- /dev/null +++ b/erts/emulator/beam/erl_bif_unique.h @@ -0,0 +1,131 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2014. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + +#ifndef ERTS_BIF_UNIQUE_H__ +#define ERTS_BIF_UNIQUE_H__ + +#include "erl_process.h" +#include "big.h" + +void erts_bif_unique_init(void); +void erts_sched_bif_unique_init(ErtsSchedulerData *esdp); + +/* reference */ +Eterm erts_make_ref(Process *); +Eterm erts_make_ref_in_buffer(Eterm buffer[REF_THING_SIZE]); +void erts_make_ref_in_array(Uint32 ref[ERTS_MAX_REF_NUMBERS]); + +/* strict monotonic counter */ + +#define ERTS_MAX_UNIQUE_MONOTONIC_INTEGER_HEAP_SIZE ERTS_MAX_UINT64_HEAP_SIZE + +/* + * Note that a raw value is an intermediate value that + * not necessarily correspond to the end result. + */ +Sint64 erts_raw_get_unique_monotonic_integer(void); +Uint erts_raw_unique_monotonic_integer_heap_size(Sint64 raw); +Eterm erts_raw_make_unique_monotonic_integer_value(Eterm **hpp, Sint64 raw); + +Sint64 erts_get_min_unique_monotonic_integer(void); + +int erts_debug_set_unique_monotonic_integer_state(Eterm et_value); +Eterm erts_debug_get_unique_monotonic_integer_state(Process *c_p); + +/* unique integer */ +#define ERTS_UNIQUE_INT_RAW_VALUES 2 +#define ERTS_MAX_UNIQUE_INT_HEAP_SIZE ERTS_UINT64_ARRAY_TO_BIG_MAX_HEAP_SZ(2) + +Uint erts_raw_unique_integer_heap_size(Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]); +Eterm erts_raw_make_unique_integer(Eterm **hpp, Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]); +void erts_raw_get_unique_integer(Uint64 val[ERTS_UNIQUE_INT_RAW_VALUES]); +Sint64 erts_get_min_unique_integer(void); + +Eterm erts_debug_make_unique_integer(Process *c_p, + Eterm etval0, + Eterm etval1); + + +ERTS_GLB_INLINE void erts_set_ref_numbers(Uint32 ref[ERTS_MAX_REF_NUMBERS], + Uint32 thr_id, Uint64 value); +ERTS_GLB_INLINE Uint32 erts_get_ref_numbers_thr_id(Uint32 ref[ERTS_MAX_REF_NUMBERS]); +ERTS_GLB_INLINE Uint64 erts_get_ref_numbers_value(Uint32 ref[ERTS_MAX_REF_NUMBERS]); +ERTS_GLB_INLINE void erts_sched_make_ref_in_array(ErtsSchedulerData *esdp, + Uint32 ref[ERTS_MAX_REF_NUMBERS]); +ERTS_GLB_INLINE Eterm erts_sched_make_ref_in_buffer(ErtsSchedulerData *esdp, + Eterm buffer[REF_THING_SIZE]); + +#if ERTS_GLB_INLINE_INCL_FUNC_DEF + +ERTS_GLB_INLINE void +erts_set_ref_numbers(Uint32 ref[ERTS_MAX_REF_NUMBERS], Uint32 thr_id, Uint64 value) +{ + /* + * We cannot use thread id in the first 18-bit word since + * the hash/phash/phash2 BIFs only hash on this word. If + * we did, we would get really poor hash values. Instead + * we have to shuffle the bits a bit. + */ + ASSERT(thr_id == (thr_id & ((Uint32) 0x3ffff))); + ref[0] = (Uint32) (value & ((Uint64) 0x3ffff)); + ref[1] = (((Uint32) (value & ((Uint64) 0xfffc0000))) + | (thr_id & ((Uint32) 0x3ffff))); + ref[2] = (Uint32) ((value >> 32) & ((Uint64) 0xffffffff)); +} + +ERTS_GLB_INLINE Uint32 +erts_get_ref_numbers_thr_id(Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + return ref[1] & ((Uint32) 0x3ffff); +} + +ERTS_GLB_INLINE Uint64 +erts_get_ref_numbers_value(Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + return (((((Uint64) ref[2]) & ((Uint64) 0xffffffff)) << 32) + | (((Uint64) ref[1]) & ((Uint64) 0xfffc0000)) + | (((Uint64) ref[0]) & ((Uint64) 0x3ffff))); +} + +ERTS_GLB_INLINE void +erts_sched_make_ref_in_array(ErtsSchedulerData *esdp, + Uint32 ref[ERTS_MAX_REF_NUMBERS]) +{ + Uint64 value; + + ASSERT(esdp); + value = esdp->ref++; + erts_set_ref_numbers(ref, (Uint32) esdp->thr_id, value); +} + +ERTS_GLB_INLINE Eterm +erts_sched_make_ref_in_buffer(ErtsSchedulerData *esdp, + Eterm buffer[REF_THING_SIZE]) +{ + Eterm* hp = buffer; + Uint32 ref[ERTS_MAX_REF_NUMBERS]; + + erts_sched_make_ref_in_array(esdp, ref); + write_ref_thing(hp, ref[0], ref[1], ref[2]); + return make_internal_ref(hp); +} + +#endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ + +#endif /* ERTS_BIF_UNIQUE_H__ */ diff --git a/erts/emulator/beam/erl_bits.c b/erts/emulator/beam/erl_bits.c index 71d31c01aa..5cc0a23dc9 100644 --- a/erts/emulator/beam/erl_bits.c +++ b/erts/emulator/beam/erl_bits.c @@ -403,7 +403,10 @@ erts_bs_get_integer_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuff words_needed = 1+WSIZE(bytes); hp = HeapOnlyAlloc(p, words_needed); res = bytes_to_big(LSB, bytes, sgn, hp); - if (is_small(res)) { + if (is_nil(res)) { + p->htop = hp; + res = THE_NON_VALUE; + } else if (is_small(res)) { p->htop = hp; } else if ((actual = bignum_header_arity(*hp)+1) < words_needed) { p->htop = hp + actual; diff --git a/erts/emulator/beam/erl_db.c b/erts/emulator/beam/erl_db.c index 4806befd99..fff892ae54 100644 --- a/erts/emulator/beam/erl_db.c +++ b/erts/emulator/beam/erl_db.c @@ -805,7 +805,7 @@ BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) list = BIF_ARG_3; } - if (!tb->common.meth->db_lookup_dbterm(tb, BIF_ARG_2, &handle)) { + if (!tb->common.meth->db_lookup_dbterm(BIF_P, tb, BIF_ARG_2, THE_NON_VALUE, &handle)) { cret = DB_ERROR_BADKEY; goto bail_out; } @@ -844,7 +844,7 @@ BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) } finalize: - tb->common.meth->db_finalize_dbterm(&handle); + tb->common.meth->db_finalize_dbterm(cret, &handle); bail_out: UnUseTmpHeap(2,BIF_P); @@ -863,14 +863,8 @@ bail_out: } } -/* -** update_counter(Tab, Key, Incr) -** update_counter(Tab, Key, {Upop}) -** update_counter(Tab, Key, [{Upop}]) -** Upop = {Pos,Incr} | {Pos,Incr,Threshold,WarpTo} -** Returns new value(s) (integer or [integer]) -*/ -BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) +static BIF_RETTYPE +do_update_counter(Process *p, Eterm arg1, Eterm arg2, Eterm arg3, Eterm arg4) { DbTable* tb; int cret = DB_ERROR_BADITEM; @@ -880,7 +874,7 @@ BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) Eterm* ret_list_currp = NULL; Eterm* ret_list_prevp = NULL; Eterm iter; - DeclareTmpHeap(cell,5,BIF_P); + DeclareTmpHeap(cell, 5, p); Eterm *tuple = cell+2; DbUpdateHandle handle; Uint halloc_size = 0; /* overestimated heap usage */ @@ -888,28 +882,29 @@ BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) Eterm* hstart; Eterm* hend; - if ((tb = db_get_table(BIF_P, BIF_ARG_1, DB_WRITE, LCK_WRITE_REC)) == NULL) { - BIF_ERROR(BIF_P, BADARG); + if ((tb = db_get_table(p, arg1, DB_WRITE, LCK_WRITE_REC)) == NULL) { + BIF_ERROR(p, BADARG); } - UseTmpHeap(5,BIF_P); + UseTmpHeap(5, p); if (!(tb->common.status & (DB_SET | DB_ORDERED_SET))) { goto bail_out; } - if (is_integer(BIF_ARG_3)) { /* Incr */ - upop_list = CONS(cell, TUPLE2(tuple, make_small(tb->common.keypos+1), - BIF_ARG_3), NIL); + if (is_integer(arg3)) { /* Incr */ + upop_list = CONS(cell, + TUPLE2(tuple, make_small(tb->common.keypos+1), arg3), + NIL); } - else if (is_tuple(BIF_ARG_3)) { /* {Upop} */ - upop_list = CONS(cell, BIF_ARG_3, NIL); + else if (is_tuple(arg3)) { /* {Upop} */ + upop_list = CONS(cell, arg3, NIL); } else { /* [{Upop}] (probably) */ - upop_list = BIF_ARG_3; + upop_list = arg3; ret_list_prevp = &ret; } - if (!tb->common.meth->db_lookup_dbterm(tb, BIF_ARG_2, &handle)) { + if (!tb->common.meth->db_lookup_dbterm(p, tb, arg2, arg4, &handle)) { goto bail_out; /* key not found */ } @@ -982,13 +977,13 @@ BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) if (ret_list_prevp) { /* Prepare to return a list */ ret = NIL; halloc_size += list_size; - hstart = HAlloc(BIF_P, halloc_size); + hstart = HAlloc(p, halloc_size); ret_list_currp = hstart; htop = hstart + list_size; hend = hstart + halloc_size; } else { - hstart = htop = HAlloc(BIF_P, halloc_size); + hstart = htop = HAlloc(p, halloc_size); } hend = hstart + halloc_size; @@ -1035,26 +1030,54 @@ BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) (is_list(ret) && (list_val(ret)+list_size)==ret_list_currp)); ASSERT(htop <= hend); - HRelease(BIF_P,hend,htop); + HRelease(p, hend, htop); finalize: - tb->common.meth->db_finalize_dbterm(&handle); + tb->common.meth->db_finalize_dbterm(cret, &handle); bail_out: - UnUseTmpHeap(5,BIF_P); + UnUseTmpHeap(5, p); db_unlock(tb, LCK_WRITE_REC); switch (cret) { case DB_ERROR_NONE: BIF_RET(ret); case DB_ERROR_SYSRES: - BIF_ERROR(BIF_P, SYSTEM_LIMIT); + BIF_ERROR(p, SYSTEM_LIMIT); default: - BIF_ERROR(BIF_P, BADARG); + BIF_ERROR(p, BADARG); break; } } +/* +** update_counter(Tab, Key, Incr) +** update_counter(Tab, Key, Upop) +** update_counter(Tab, Key, [{Upop}]) +** Upop = {Pos,Incr} | {Pos,Incr,Threshold,WarpTo} +** Returns new value(s) (integer or [integer]) +*/ +BIF_RETTYPE ets_update_counter_3(BIF_ALIST_3) +{ + return do_update_counter(BIF_P, BIF_ARG_1, BIF_ARG_2, BIF_ARG_3, THE_NON_VALUE); +} + +/* +** update_counter(Tab, Key, Incr, Default) +** update_counter(Tab, Key, Upop, Default) +** update_counter(Tab, Key, [{Upop}], Default) +** Upop = {Pos,Incr} | {Pos,Incr,Threshold,WarpTo} +** Returns new value(s) (integer or [integer]) +*/ +BIF_RETTYPE ets_update_counter_4(BIF_ALIST_4) +{ + if (is_not_tuple(BIF_ARG_4)) { + BIF_ERROR(BIF_P, BADARG); + } + return do_update_counter(BIF_P, BIF_ARG_1, BIF_ARG_2, BIF_ARG_3, BIF_ARG_4); +} + + /* ** The put BIF */ diff --git a/erts/emulator/beam/erl_db_hash.c b/erts/emulator/beam/erl_db_hash.c index c2157457a0..045c8ae135 100644 --- a/erts/emulator/beam/erl_db_hash.c +++ b/erts/emulator/beam/erl_db_hash.c @@ -174,7 +174,7 @@ static ERTS_INLINE void add_fixed_deletion(DbTableHash* tb, int ix) /* optimised version of make_hash (normal case? atomic key) */ #define MAKE_HASH(term) \ ((is_atom(term) ? (atom_tab(atom_val(term))->slot.bucket.hvalue) : \ - make_hash2(term)) % MAX_HASH) + make_internal_hash(term)) % MAX_HASH) #ifdef ERTS_SMP # define DB_HASH_LOCK_MASK (DB_HASH_LOCK_CNT-1) @@ -444,8 +444,11 @@ static int db_delete_all_objects_hash(Process* p, DbTable* tbl); #ifdef HARDDEBUG static void db_check_table_hash(DbTableHash *tb); #endif -static int db_lookup_dbterm_hash(DbTable *tbl, Eterm key, DbUpdateHandle* handle); -static void db_finalize_dbterm_hash(DbUpdateHandle* handle); +static int +db_lookup_dbterm_hash(Process *p, DbTable *tbl, Eterm key, Eterm obj, + DbUpdateHandle* handle); +static void +db_finalize_dbterm_hash(int cret, DbUpdateHandle* handle); static ERTS_INLINE void try_shrink(DbTableHash* tb) { @@ -2796,59 +2799,129 @@ static HashDbTerm* next(DbTableHash *tb, Uint *iptr, erts_smp_rwmtx_t** lck_ptr, return NULL; } -static int db_lookup_dbterm_hash(DbTable *tbl, Eterm key, DbUpdateHandle* handle) +static int +db_lookup_dbterm_hash(Process *p, DbTable *tbl, Eterm key, Eterm obj, + DbUpdateHandle* handle) { DbTableHash *tb = &tbl->hash; - HashDbTerm* b; - HashDbTerm** prevp; - int ix; HashValue hval; + HashDbTerm **bp, *b; erts_smp_rwmtx_t* lck; + int flags = 0; + + ASSERT(tb->common.status & DB_SET); hval = MAKE_HASH(key); - lck = WLOCK_HASH(tb,hval); - ix = hash_to_ix(tb, hval); - prevp = &BUCKET(tb, ix); - b = *prevp; + lck = WLOCK_HASH(tb, hval); + bp = &BUCKET(tb, hash_to_ix(tb, hval)); + b = *bp; - while (b != 0) { - if (has_live_key(tb,b,key,hval)) { - handle->tb = tbl; - handle->bp = (void**) prevp; - handle->dbterm = &b->dbterm; - handle->mustResize = 0; - handle->new_size = b->dbterm.size; - #if HALFWORD_HEAP - handle->abs_vec = NULL; - #endif - handle->lck = lck; - /* KEEP hval WLOCKED, db_finalize_dbterm_hash will WUNLOCK */ - return 1; - } - prevp = &b->next; - b = *prevp; + for (;;) { + if (b == NULL) { + break; + } + if (has_key(tb, b, key, hval)) { + if (b->hvalue != INVALID_HASH) { + goto Ldone; + } + break; + } + bp = &b->next; + b = *bp; } - WUNLOCK_HASH(lck); - return 0; + + if (obj == THE_NON_VALUE) { + WUNLOCK_HASH(lck); + return 0; + } + + { + Eterm *objp = tuple_val(obj); + int arity = arityval(*objp); + Eterm *htop, *hend; + + ASSERT(arity >= tb->common.keypos); + htop = HAlloc(p, arity + 1); + hend = htop + arity + 1; + sys_memcpy(htop, objp, sizeof(Eterm) * (arity + 1)); + htop[tb->common.keypos] = key; + obj = make_tuple(htop); + + if (b == NULL) { + HashDbTerm *q = new_dbterm(tb, obj); + + q->hvalue = hval; + q->next = NULL; + *bp = b = q; + + { + int nitems = erts_smp_atomic_inc_read_nob(&tb->common.nitems); + int nactive = NACTIVE(tb); + + if (nitems > nactive * (CHAIN_LEN + 1) && !IS_FIXED(tb)) { + grow(tb, nactive); + } + } + } else { + HashDbTerm *q, *next = b->next; + + ASSERT(b->hvalue == INVALID_HASH); + q = replace_dbterm(tb, b, obj); + q->next = next; + q->hvalue = hval; + *bp = b = q; + erts_smp_atomic_inc_nob(&tb->common.nitems); + } + + HRelease(p, hend, htop); + flags |= DB_NEW_OBJECT; + } + +Ldone: + handle->tb = tbl; + handle->bp = (void **)bp; + handle->dbterm = &b->dbterm; + handle->flags = flags; + handle->new_size = b->dbterm.size; +#if HALFWORD_HEAP + handle->abs_vec = NULL; +#endif + handle->lck = lck; + return 1; } /* Must be called after call to db_lookup_dbterm */ -static void db_finalize_dbterm_hash(DbUpdateHandle* handle) +static void +db_finalize_dbterm_hash(int cret, DbUpdateHandle* handle) { DbTable* tbl = handle->tb; - HashDbTerm* oldp = (HashDbTerm*) *(handle->bp); + DbTableHash *tb = &tbl->hash; + HashDbTerm **bp = (HashDbTerm **) handle->bp; + HashDbTerm *b = *bp; erts_smp_rwmtx_t* lck = (erts_smp_rwmtx_t*) handle->lck; - ERTS_SMP_LC_ASSERT(IS_HASH_WLOCKED(&tbl->hash,lck)); /* locked by db_lookup_dbterm_hash */ + ERTS_SMP_LC_ASSERT(IS_HASH_WLOCKED(tb, lck)); /* locked by db_lookup_dbterm_hash */ + + ASSERT((&b->dbterm == handle->dbterm) == !(tb->common.compress && handle->flags & DB_MUST_RESIZE)); - ASSERT((&oldp->dbterm == handle->dbterm) == !(tbl->common.compress && handle->mustResize)); + if (handle->flags & DB_NEW_OBJECT && cret != DB_ERROR_NONE) { + if (IS_FIXED(tb)) { + add_fixed_deletion(tb, hash_to_ix(tb, b->hvalue)); + b->hvalue = INVALID_HASH; + } else { + *bp = b->next; + free_term(tb, b); + } - if (handle->mustResize) { + WUNLOCK_HASH(lck); + erts_smp_atomic_dec_nob(&tb->common.nitems); + try_shrink(tb); + } else if (handle->flags & DB_MUST_RESIZE) { db_finalize_resize(handle, offsetof(HashDbTerm,dbterm)); WUNLOCK_HASH(lck); - free_term(&tbl->hash, oldp); + free_term(tb, b); } else { WUNLOCK_HASH(lck); diff --git a/erts/emulator/beam/erl_db_tree.c b/erts/emulator/beam/erl_db_tree.c index 720c0659c3..577da35b75 100644 --- a/erts/emulator/beam/erl_db_tree.c +++ b/erts/emulator/beam/erl_db_tree.c @@ -399,8 +399,11 @@ static int db_delete_all_objects_tree(Process* p, DbTable* tbl); #ifdef HARDDEBUG static void db_check_table_tree(DbTable *tbl); #endif -static int db_lookup_dbterm_tree(DbTable *, Eterm key, DbUpdateHandle*); -static void db_finalize_dbterm_tree(DbUpdateHandle*); +static int +db_lookup_dbterm_tree(Process *, DbTable *, Eterm key, Eterm obj, + DbUpdateHandle*); +static void +db_finalize_dbterm_tree(int cret, DbUpdateHandle *); /* ** Static variables @@ -2546,16 +2549,43 @@ static TreeDbTerm **find_node2(DbTableTree *tb, Eterm key) return this; } -static int db_lookup_dbterm_tree(DbTable *tbl, Eterm key, DbUpdateHandle* handle) +static int +db_lookup_dbterm_tree(Process *p, DbTable *tbl, Eterm key, Eterm obj, + DbUpdateHandle* handle) { DbTableTree *tb = &tbl->tree; TreeDbTerm **pp = find_node2(tb, key); - - if (pp == NULL) return 0; + int flags = 0; + + if (pp == NULL) { + if (obj == THE_NON_VALUE) { + return 0; + } else { + Eterm *objp = tuple_val(obj); + int arity = arityval(*objp); + Eterm *htop, *hend; + + ASSERT(arity >= tb->common.keypos); + htop = HAlloc(p, arity + 1); + hend = htop + arity + 1; + sys_memcpy(htop, objp, sizeof(Eterm) * (arity + 1)); + htop[tb->common.keypos] = key; + obj = make_tuple(htop); + + if (db_put_tree(tbl, obj, 1) != DB_ERROR_NONE) { + return 0; + } + + pp = find_node2(tb, key); + ASSERT(pp != NULL); + HRelease(p, hend, htop); + flags |= DB_NEW_OBJECT; + } + } handle->tb = tbl; handle->dbterm = &(*pp)->dbterm; - handle->mustResize = 0; + handle->flags = flags; handle->bp = (void**) pp; handle->new_size = (*pp)->dbterm.size; #if HALFWORD_HEAP @@ -2564,15 +2594,21 @@ static int db_lookup_dbterm_tree(DbTable *tbl, Eterm key, DbUpdateHandle* handle return 1; } -static void db_finalize_dbterm_tree(DbUpdateHandle* handle) +static void +db_finalize_dbterm_tree(int cret, DbUpdateHandle *handle) { - if (handle->mustResize) { - TreeDbTerm* oldp = (TreeDbTerm*) *handle->bp; + DbTable *tbl = handle->tb; + DbTableTree *tb = &tbl->tree; + TreeDbTerm *bp = (TreeDbTerm *) *handle->bp; + if (handle->flags & DB_NEW_OBJECT && cret != DB_ERROR_NONE) { + Eterm ret; + db_erase_tree(tbl, GETKEY(tb, bp->dbterm.tpl), &ret); + } else if (handle->flags & DB_MUST_RESIZE) { db_finalize_resize(handle, offsetof(TreeDbTerm,dbterm)); - reset_static_stack(&handle->tb->tree); + reset_static_stack(tb); - free_term(&handle->tb->tree, oldp); + free_term(tb, bp); } #ifdef DEBUG handle->dbterm = 0; diff --git a/erts/emulator/beam/erl_db_util.c b/erts/emulator/beam/erl_db_util.c index 7eb80e3bb1..0bf562d937 100644 --- a/erts/emulator/beam/erl_db_util.c +++ b/erts/emulator/beam/erl_db_util.c @@ -214,8 +214,8 @@ typedef enum { matchPushT, matchPushL, matchPushM, - matchPushK, matchPop, + matchSwap, matchBind, matchCmp, matchEqBin, @@ -225,12 +225,14 @@ typedef enum { matchEq, matchList, matchMap, + matchKey, matchSkip, matchPushC, matchConsA, /* Car is below Cdr */ matchConsB, /* Cdr is below Car (unusual) */ matchMkTuple, - matchMkMap, + matchMkFlatMap, + matchMkHashMap, matchCall0, matchCall1, matchCall2, @@ -1375,15 +1377,15 @@ restart: for (;;) { switch (t & _TAG_PRIMARY_MASK) { case TAG_PRIMARY_BOXED: - if (is_map(t)) { - num_iters = map_get_size(map_val(t)); + if (is_flatmap(t)) { + num_iters = flatmap_get_size(flatmap_val(t)); if (!structure_checked) { DMC_PUSH(text, matchMap); DMC_PUSH(text, num_iters); } structure_checked = 0; for (i = 0; i < num_iters; ++i) { - Eterm key = map_get_keys(map_val(t))[i]; + Eterm key = flatmap_get_keys(flatmap_val(t))[i]; if (db_is_variable(key) >= 0) { if (context.err_info) { add_dmc_err(context.err_info, @@ -1399,24 +1401,85 @@ restart: } goto error; } - DMC_PUSH(text, matchPushK); - ++(context.stack_used); + DMC_PUSH(text, matchKey); DMC_PUSH(text, dmc_private_copy(&context, key)); + { + int old_stack = ++(context.stack_used); + Eterm value = flatmap_get_values(flatmap_val(t))[i]; + res = dmc_one_term(&context, &heap, &stack, &text, + value); + ASSERT(res != retFail); + if (res == retRestart) { + goto restart; + } + if (old_stack != context.stack_used) { + ASSERT(old_stack + 1 == context.stack_used); + DMC_PUSH(text, matchSwap); + } + if (context.stack_used > context.stack_need) { + context.stack_need = context.stack_used; + } + DMC_PUSH(text, matchPop); + --(context.stack_used); + } } - if (context.stack_used > context.stack_need) { - context.stack_need = context.stack_used; + break; + } + if (is_hashmap(t)) { + DECLARE_WSTACK(wstack); + Eterm *kv; + num_iters = hashmap_size(t); + if (!structure_checked) { + DMC_PUSH(text, matchMap); + DMC_PUSH(text, num_iters); } - for (i = num_iters; i--; ) { - Eterm value = map_get_values(map_val(t))[i]; - DMC_PUSH(text, matchPop); - --(context.stack_used); - res = dmc_one_term(&context, &heap, &stack, &text, - value); - ASSERT(res != retFail); - if (res == retRestart) { - goto restart; + structure_checked = 0; + + hashmap_iterator_init(&wstack, t, 0); + + while ((kv=hashmap_iterator_next(&wstack)) != NULL) { + Eterm key = CAR(kv); + Eterm value = CDR(kv); + if (db_is_variable(key) >= 0) { + if (context.err_info) { + add_dmc_err(context.err_info, + "Variable found in map key.", + -1, 0UL, dmcError); + } + DESTROY_WSTACK(wstack); + goto error; + } else if (key == am_Underscore) { + if (context.err_info) { + add_dmc_err(context.err_info, + "Underscore found in map key.", + -1, 0UL, dmcError); + } + DESTROY_WSTACK(wstack); + goto error; + } + DMC_PUSH(text, matchKey); + DMC_PUSH(text, dmc_private_copy(&context, key)); + { + int old_stack = ++(context.stack_used); + res = dmc_one_term(&context, &heap, &stack, &text, + value); + ASSERT(res != retFail); + if (res == retRestart) { + DESTROY_WSTACK(wstack); + goto restart; + } + if (old_stack != context.stack_used) { + ASSERT(old_stack + 1 == context.stack_used); + DMC_PUSH(text, matchSwap); + } + if (context.stack_used > context.stack_need) { + context.stack_need = context.stack_used; + } + DMC_PUSH(text, matchPop); + --(context.stack_used); } } + DESTROY_WSTACK(wstack); break; } if (!is_tuple(t)) { @@ -1945,32 +2008,52 @@ restart: FAIL(); } n = *pc++; - if (map_get_size(map_val_rel(*ep, base)) < n) { - FAIL(); - } - ep = map_val_rel(*ep, base); + if (is_flatmap_rel(*ep,base)) { + if (flatmap_get_size(flatmap_val_rel(*ep, base)) < n) { + FAIL(); + } + } else { + ASSERT(is_hashmap_rel(*ep,base)); + if (hashmap_size_rel(*ep, base) < n) { + FAIL(); + } + } + ep = flatmap_val_rel(*ep, base); break; case matchPushM: if (!is_map_rel(*ep, base)) { FAIL(); } n = *pc++; - if (map_get_size(map_val_rel(*ep, base)) < n) { - FAIL(); - } - *sp++ = map_val_rel(*ep++, base); + if (is_flatmap_rel(*ep,base)) { + if (flatmap_get_size(flatmap_val_rel(*ep, base)) < n) { + FAIL(); + } + } else { + ASSERT(is_hashmap_rel(*ep,base)); + if (hashmap_size_rel(*ep, base) < n) { + FAIL(); + } + } + *sp++ = flatmap_val_rel(*ep++, base); break; - case matchPushK: + case matchKey: t = (Eterm) *pc++; - tp = erts_maps_get_rel(t, make_map_rel(ep, base), base); + tp = erts_maps_get_rel(t, make_flatmap_rel(ep, base), base); if (!tp) { FAIL(); } - *sp++ = tp; + *sp++ = ep; + ep = tp; break; case matchPop: ep = *(--sp); break; + case matchSwap: + tp = sp[-1]; + sp[-1] = sp[-2]; + sp[-2] = tp; + break; case matchBind: n = *pc++; variables[n].term = *ep++; @@ -2068,23 +2151,38 @@ restart: } *esp++ = t; break; - case matchMkMap: + case matchMkFlatMap: n = *pc++; - ehp = HAllocX(build_proc, 1 + MAP_HEADER_SIZE + n, HEAP_XTRA); + ehp = HAllocX(build_proc, 1 + MAP_HEADER_FLATMAP_SZ + n, HEAP_XTRA); t = *ehp++ = *--esp; { - map_t *m = (map_t *)ehp; - m->thing_word = MAP_HEADER; + flatmap_t *m = (flatmap_t *)ehp; + m->thing_word = MAP_HEADER_FLATMAP; m->size = n; m->keys = t; } - t = make_map(ehp); - ehp += MAP_HEADER_SIZE; + t = make_flatmap(ehp); + ehp += MAP_HEADER_FLATMAP_SZ; while (n--) { *ehp++ = *--esp; } *esp++ = t; break; + case matchMkHashMap: + n = *pc++; + esp -= 2*n; + ehp = HAllocX(build_proc, 2*n, HEAP_XTRA); + { + ErtsHeapFactory factory; + Uint ix; + factory.p = build_proc; + for (ix = 0; ix < 2*n; ix++){ + ehp[ix] = esp[ix]; + } + t = erts_hashmap_from_array(&factory, ehp, n, 0); + } + *esp++ = t; + break; case matchCall0: bif = (Eterm (*)(Process*, ...)) *pc++; t = (*bif)(build_proc, bif_args); @@ -2699,10 +2797,10 @@ Wterm db_do_read_element(DbUpdateHandle* handle, Sint position) } ASSERT(((DbTableCommon*)handle->tb)->compress); - ASSERT(!handle->mustResize); + ASSERT(!(handle->flags & DB_MUST_RESIZE)); handle->dbterm = db_alloc_tmp_uncompressed(&handle->tb->common, handle->dbterm); - handle->mustResize = 1; + handle->flags |= DB_MUST_RESIZE; return handle->dbterm->tpl[position]; } @@ -2735,11 +2833,11 @@ void db_do_update_element(DbUpdateHandle* handle, #endif return; } - if (!handle->mustResize) { + if (!(handle->flags & DB_MUST_RESIZE)) { if (handle->tb->common.compress) { handle->dbterm = db_alloc_tmp_uncompressed(&handle->tb->common, handle->dbterm); - handle->mustResize = 1; + handle->flags |= DB_MUST_RESIZE; oldval = handle->dbterm->tpl[position]; #if HALFWORD_HEAP old_base = NULL; @@ -2799,7 +2897,7 @@ both_size_set: /* write new value in old dbterm, finalize will make a flat copy */ handle->dbterm->tpl[position] = newval; - handle->mustResize = 1; + handle->flags |= DB_MUST_RESIZE; #if HALFWORD_HEAP if (old_base && newval_sz > 0) { @@ -3275,7 +3373,14 @@ int db_has_variable(Eterm node) { while(arity--) { ESTACK_PUSH(s,*(++tuple)); } - } + } else if (is_flatmap(node)) { + Eterm *values = flatmap_get_values(flatmap_val(node)); + Uint size = flatmap_get_size(flatmap_val(node)); + ESTACK_PUSH(s, ((flatmap_t *) flatmap_val(node))->keys); + while (size--) { + ESTACK_PUSH(s, *(values++)); + } + } break; case TAG_PRIMARY_IMMED1: if (node == am_Underscore || db_is_variable(node) >= 0) { @@ -3348,7 +3453,6 @@ static DMCRet dmc_one_term(DMCContext *context, Uint sz, sz2, sz3; Uint i, j; - switch (c & _TAG_PRIMARY_MASK) { case TAG_PRIMARY_IMMED1: if ((n = db_is_variable(c)) >= 0) { /* variable */ @@ -3436,7 +3540,10 @@ static DMCRet dmc_one_term(DMCContext *context, DMC_PUSH(*stack, c); break; case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE): - n = map_get_size(map_val(c)); + if (is_flatmap(c)) + n = flatmap_get_size(flatmap_val(c)); + else + n = hashmap_size(c); DMC_PUSH(*text, matchPushM); ++(context->stack_used); DMC_PUSH(*text, n); @@ -3727,30 +3834,87 @@ static DMCRet dmc_map(DMCContext *context, DMCHeap *heap, DMC_STACK_TYPE(UWord) *text, Eterm t, int *constant) { - map_t *m = (map_t *)map_val(t); - Eterm *values = map_get_values(m); - int nelems = map_get_size(m); + int nelems; int constant_values; DMCRet ret; + if (is_flatmap(t)) { + flatmap_t *m = (flatmap_t *)flatmap_val(t); + Eterm *values = flatmap_get_values(m); - ret = dmc_array(context, heap, text, values, nelems, &constant_values); - if (ret != retOk) { - return ret; - } - if (constant_values) { - *constant = 1; + nelems = flatmap_get_size(m); + ret = dmc_array(context, heap, text, values, nelems, &constant_values); + + if (ret != retOk) { + return ret; + } + if (constant_values) { + *constant = 1; + return retOk; + } + DMC_PUSH(*text, matchPushC); + DMC_PUSH(*text, dmc_private_copy(context, m->keys)); + if (++context->stack_used > context->stack_need) { + context->stack_need = context->stack_used; + } + DMC_PUSH(*text, matchMkFlatMap); + DMC_PUSH(*text, nelems); + context->stack_used -= nelems; + *constant = 0; + return retOk; + } else { + DECLARE_WSTACK(wstack); + Eterm *kv; + int c; + + ASSERT(is_hashmap(t)); + + hashmap_iterator_init(&wstack, t, 1); + constant_values = 1; + nelems = hashmap_size(t); + + while ((kv=hashmap_iterator_prev(&wstack)) != NULL) { + if ((ret = dmc_expr(context, heap, text, CDR(kv), &c)) != retOk) { + DESTROY_WSTACK(wstack); + return ret; + } + if (!c) + constant_values = 0; + } + + if (constant_values) { + *constant = 1; + DESTROY_WSTACK(wstack); + return retOk; + } + + *constant = 0; + + hashmap_iterator_init(&wstack, t, 1); + + while ((kv=hashmap_iterator_prev(&wstack)) != NULL) { + /* push key */ + if ((ret = dmc_expr(context, heap, text, CAR(kv), &c)) != retOk) { + DESTROY_WSTACK(wstack); + return ret; + } + if (c) { + do_emit_constant(context, text, CAR(kv)); + } + /* push value */ + if ((ret = dmc_expr(context, heap, text, CDR(kv), &c)) != retOk) { + DESTROY_WSTACK(wstack); + return ret; + } + if (c) { + do_emit_constant(context, text, CDR(kv)); + } + } + DMC_PUSH(*text, matchMkHashMap); + DMC_PUSH(*text, nelems); + context->stack_used -= nelems; + DESTROY_WSTACK(wstack); return retOk; } - DMC_PUSH(*text, matchPushC); - DMC_PUSH(*text, dmc_private_copy(context, m->keys)); - if (++context->stack_used > context->stack_need) { - context->stack_need = context->stack_used; - } - DMC_PUSH(*text, matchMkMap); - DMC_PUSH(*text, nelems); - context->stack_used -= nelems; - *constant = 0; - return retOk; } static DMCRet dmc_whole_expression(DMCContext *context, @@ -5302,6 +5466,12 @@ void db_match_dis(Binary *bp) ++t; erts_printf("Map\t%beu\n", n); break; + case matchKey: + ++t; + p = (Eterm) *t; + ++t; + erts_printf("Key\t%p (%T)\n", t, p); + break; case matchPushT: ++t; n = *t; @@ -5318,16 +5488,14 @@ void db_match_dis(Binary *bp) ++t; erts_printf("PushM\t%beu\n", n); break; - case matchPushK: - ++t; - p = (Eterm) *t; - ++t; - erts_printf("PushK\t%p (%T)\n", t, p); - break; case matchPop: ++t; erts_printf("Pop\n"); break; + case matchSwap: + ++t; + erts_printf("Swap\n"); + break; case matchBind: ++t; n = *t; @@ -5440,11 +5608,17 @@ void db_match_dis(Binary *bp) ++t; erts_printf("MkTuple\t%beu\n", n); break; - case matchMkMap: + case matchMkFlatMap: + ++t; + n = *t; + ++t; + erts_printf("MkFlatMap\t%beu\n", n); + break; + case matchMkHashMap: ++t; n = *t; ++t; - erts_printf("MkMapA\t%beu\n", n); + erts_printf("MkHashMap\t%beu\n", n); break; case matchOr: ++t; diff --git a/erts/emulator/beam/erl_db_util.h b/erts/emulator/beam/erl_db_util.h index 5ace93c8ed..ca206c7f58 100644 --- a/erts/emulator/beam/erl_db_util.h +++ b/erts/emulator/beam/erl_db_util.h @@ -76,6 +76,9 @@ typedef struct db_term { union db_table; typedef union db_table DbTable; +#define DB_MUST_RESIZE 1 +#define DB_NEW_OBJECT 2 + /* Info about a database entry while it's being updated * (by update_counter or update_element) */ @@ -84,7 +87,7 @@ typedef struct { DbTerm* dbterm; void** bp; /* {Hash|Tree}DbTerm** */ Uint new_size; - int mustResize; + int flags; void* lck; #if HALFWORD_HEAP unsigned char* abs_vec; /* [i] true if dbterm->tpl[i] is absolute Eterm */ @@ -183,15 +186,14 @@ typedef struct db_table_method void *arg); void (*db_check_table)(DbTable* tb); - /* Lookup a dbterm for updating. Return false if not found. - */ - int (*db_lookup_dbterm)(DbTable*, Eterm key, - DbUpdateHandle* handle); /* [out] */ + /* Lookup a dbterm for updating. Return false if not found. */ + int (*db_lookup_dbterm)(Process *, DbTable *, Eterm key, Eterm obj, + DbUpdateHandle* handle); - /* Must be called for each db_lookup_dbterm that returned true, - ** even if dbterm was not updated. - */ - void (*db_finalize_dbterm)(DbUpdateHandle* handle); + /* Must be called for each db_lookup_dbterm that returned true, even if + ** dbterm was not updated. If the handle was of a new object and cret is + ** not DB_ERROR_NONE, the object is removed from the table. */ + void (*db_finalize_dbterm)(int cret, DbUpdateHandle* handle); } DbTableMethod; diff --git a/erts/emulator/beam/erl_drv_thread.c b/erts/emulator/beam/erl_drv_thread.c index 31b05d22af..240faa823d 100644 --- a/erts/emulator/beam/erl_drv_thread.c +++ b/erts/emulator/beam/erl_drv_thread.c @@ -604,10 +604,12 @@ erl_drv_thread_create(char *name, ethr_thr_opts ethr_opts = ETHR_THR_OPTS_DEFAULT_INITER; ethr_thr_opts *use_opts; - if (!opts) + if (!opts && !name) use_opts = NULL; else { - ethr_opts.suggested_stack_size = opts->suggested_stack_size; + if(opts) + ethr_opts.suggested_stack_size = opts->suggested_stack_size; + ethr_opts.name = name; use_opts = ðr_opts; } diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index d1a7ee113b..4a116c0740 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -39,6 +39,7 @@ #include "hipe_mode_switch.h" #endif #include "dtrace-wrapper.h" +#include "erl_bif_unique.h" #define ERTS_INACT_WR_PB_LEAVE_MUCH_LIMIT 1 #define ERTS_INACT_WR_PB_LEAVE_MUCH_PERCENTAGE 20 @@ -96,10 +97,10 @@ typedef struct { static Uint setup_rootset(Process*, Eterm*, int, Rootset*); static void cleanup_rootset(Rootset *rootset); -static Uint combined_message_size(Process* p); +static Uint combined_message_size(Process* p, int off_heap_msgs); static void remove_message_buffers(Process* p); -static int major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl); -static int minor_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl); +static int major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl, int off_heap_msgs); +static int minor_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl, int off_heap_msgs); static void do_minor(Process *p, Uint new_sz, Eterm* objv, int nobj); static Eterm* sweep_rootset(Rootset *rootset, Eterm* htop, char* src, Uint src_size); static Eterm* sweep_one_area(Eterm* n_hp, Eterm* n_htop, char* src, Uint src_size); @@ -402,7 +403,9 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj) { Uint reclaimed_now = 0; int done = 0; + int off_heap_msgs; Uint ms1, s1, us1; + erts_aint32_t state; ErtsSchedulerData *esdp; #ifdef USE_VM_PROBES DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); @@ -419,7 +422,8 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj) trace_gc(p, am_gc_start); } - erts_smp_atomic32_read_bor_nob(&p->state, ERTS_PSFLG_GC); + state = erts_smp_atomic32_read_bor_nob(&p->state, ERTS_PSFLG_GC); + off_heap_msgs = state & ERTS_PSFLG_OFF_HEAP_MSGS; if (erts_system_monitor_long_gc != 0) { get_now(&ms1, &s1, &us1); } @@ -445,11 +449,11 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj) while (!done) { if ((FLAGS(p) & F_NEED_FULLSWEEP) != 0) { DTRACE2(gc_major_start, pidbuf, need); - done = major_collection(p, need, objv, nobj, &reclaimed_now); + done = major_collection(p, need, objv, nobj, &reclaimed_now, off_heap_msgs); DTRACE2(gc_major_end, pidbuf, reclaimed_now); } else { DTRACE2(gc_minor_start, pidbuf, need); - done = minor_collection(p, need, objv, nobj, &reclaimed_now); + done = minor_collection(p, need, objv, nobj, &reclaimed_now, off_heap_msgs); DTRACE2(gc_minor_end, pidbuf, reclaimed_now); } } @@ -832,7 +836,7 @@ erts_garbage_collect_literals(Process* p, Eterm* literals, } static int -minor_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl) +minor_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl, int off_heap_msgs) { Uint mature = HIGH_WATER(p) - HEAP_START(p); @@ -871,20 +875,22 @@ minor_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl) Uint size_after; Uint need_after; Uint stack_size = STACK_SZ_ON_HEAP(p); - Uint fragments = MBUF_SIZE(p) + combined_message_size(p); + Uint fragments = MBUF_SIZE(p) + combined_message_size(p, off_heap_msgs); Uint size_before = fragments + (HEAP_TOP(p) - HEAP_START(p)); Uint new_sz = next_heap_size(p, HEAP_SIZE(p) + fragments, 0); do_minor(p, new_sz, objv, nobj); - /* - * Copy newly received message onto the end of the new heap. - */ - ErtsGcQuickSanityCheck(p); - for (msgp = p->msg.first; msgp; msgp = msgp->next) { - if (msgp->data.attached) { - erts_move_msg_attached_data_to_heap(&p->htop, &p->off_heap, msgp); - ErtsGcQuickSanityCheck(p); + if (!off_heap_msgs) { + /* + * Copy newly received message onto the end of the new heap. + */ + ErtsGcQuickSanityCheck(p); + for (msgp = p->msg.first; msgp; msgp = msgp->next) { + if (msgp->data.attached) { + erts_move_msg_attached_data_to_heap(&p->htop, &p->off_heap, msgp); + ErtsGcQuickSanityCheck(p); + } } } ErtsGcQuickSanityCheck(p); @@ -1210,7 +1216,7 @@ do_minor(Process *p, Uint new_sz, Eterm* objv, int nobj) */ static int -major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl) +major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl, int off_heap_msgs) { Rootset rootset; Roots* roots; @@ -1223,8 +1229,7 @@ major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl) Uint oh_size = (char *) OLD_HTOP(p) - oh; Uint n; Uint new_sz; - Uint fragments = MBUF_SIZE(p) + combined_message_size(p); - ErlMessage *msgp; + Uint fragments = MBUF_SIZE(p) + combined_message_size(p, off_heap_msgs); size_before = fragments + (HEAP_TOP(p) - HEAP_START(p)); @@ -1434,13 +1439,16 @@ major_collection(Process* p, int need, Eterm* objv, int nobj, Uint *recl) ErtsGcQuickSanityCheck(p); - /* - * Copy newly received message onto the end of the new heap. - */ - for (msgp = p->msg.first; msgp; msgp = msgp->next) { - if (msgp->data.attached) { - erts_move_msg_attached_data_to_heap(&p->htop, &p->off_heap, msgp); - ErtsGcQuickSanityCheck(p); + if (!off_heap_msgs) { + ErlMessage *msgp; + /* + * Copy newly received message onto the end of the new heap. + */ + for (msgp = p->msg.first; msgp; msgp = msgp->next) { + if (msgp->data.attached) { + erts_move_msg_attached_data_to_heap(&p->htop, &p->off_heap, msgp); + ErtsGcQuickSanityCheck(p); + } } } @@ -1501,15 +1509,17 @@ adjust_after_fullsweep(Process *p, Uint size_before, int need, Eterm *objv, int * mbuf list. */ static Uint -combined_message_size(Process* p) +combined_message_size(Process* p, int off_heap_msgs) { - Uint sz = 0; + Uint sz; ErlMessage *msgp; - for (msgp = p->msg.first; msgp; msgp = msgp->next) { - if (msgp->data.attached) { + if (off_heap_msgs) + return 0; + + for (sz = 0, msgp = p->msg.first; msgp; msgp = msgp->next) { + if (msgp->data.attached) sz += erts_msg_attached_data_size(msgp); - } } return sz; } diff --git a/erts/emulator/beam/erl_gc.h b/erts/emulator/beam/erl_gc.h index bf0496c112..bd6dcc9078 100644 --- a/erts/emulator/beam/erl_gc.h +++ b/erts/emulator/beam/erl_gc.h @@ -55,7 +55,10 @@ do { \ nelts = header_arity(HDR); \ switch ((HDR) & _HEADER_SUBTAG_MASK) { \ case SUB_BINARY_SUBTAG: nelts++; break; \ - case MAP_SUBTAG: nelts+=map_get_size(PTR) + 1; break; \ + case MAP_SUBTAG: \ + if (is_flatmap_header(HDR)) nelts+=flatmap_get_size(PTR) + 1; \ + else nelts += hashmap_bitcount(MAP_HEADER_VAL(HDR)); \ + break; \ case FUN_SUBTAG: nelts+=((ErlFunThing*)(PTR))->num_free+1; break; \ } \ gval = make_boxed(HTOP); \ @@ -63,7 +66,6 @@ do { \ *HTOP++ = HDR; \ *PTR++ = gval; \ while (nelts--) *HTOP++ = *PTR++; \ - \ } while(0) #define in_area(ptr,start,nbytes) \ diff --git a/erts/emulator/beam/erl_init.c b/erts/emulator/beam/erl_init.c index 77445ef1ff..86d3416423 100644 --- a/erts/emulator/beam/erl_init.c +++ b/erts/emulator/beam/erl_init.c @@ -45,6 +45,7 @@ #include "erl_thr_queue.h" #include "erl_async.h" #include "erl_ptab.h" +#include "erl_bif_unique.h" #ifdef HIPE #include "hipe_mode_switch.h" /* for hipe_mode_switch_init() */ @@ -134,7 +135,9 @@ static void erl_init(int ncpu, int legacy_proc_tab, int port_tab_sz, int port_tab_sz_ignore_files, - int legacy_port_tab); + int legacy_port_tab, + int time_correction, + ErtsTimeWarpMode time_warp_mode); static erts_atomic_t exiting; @@ -185,12 +188,10 @@ static int no_dirty_io_schedulers; Uint32 verbose; /* See erl_debug.h for information about verbose */ #endif -int erts_disable_tolerant_timeofday; /* Time correction can be disabled it is - * not and/or it is too slow. - */ - int erts_atom_table_size = ATOM_LIMIT; /* Maximum number of atoms */ +int erts_pd_initial_size = 10; + int erts_modified_timing_level; int erts_no_crash_dump = 0; /* Use -d to suppress crash dump. */ @@ -266,6 +267,19 @@ this_rel_num(void) return this_rel; } +static ERTS_INLINE void +set_default_time_adj(int *time_correction_p, ErtsTimeWarpMode *time_warp_mode_p) +{ + *time_correction_p = 1; + *time_warp_mode_p = ERTS_NO_TIME_WARP_MODE; + if (!erts_check_time_adj_support(*time_correction_p, + *time_warp_mode_p)) { + *time_correction_p = 0; + ASSERT(erts_check_time_adj_support(*time_correction_p, + *time_warp_mode_p)); + } +} + /* * Common error printout function, all error messages * that don't go to the error logger go through here. @@ -281,13 +295,22 @@ static int early_init(int *argc, char **argv); void erts_short_init(void) { - int ncpu = early_init(NULL, NULL); + + int ncpu; + int time_correction; + ErtsTimeWarpMode time_warp_mode; + + set_default_time_adj(&time_correction, + &time_warp_mode); + ncpu = early_init(NULL, NULL); erl_init(ncpu, ERTS_DEFAULT_MAX_PROCESSES, 0, ERTS_DEFAULT_MAX_PORTS, 0, - 0); + 0, + time_correction, + time_warp_mode); erts_initialized = 1; } @@ -297,12 +320,15 @@ erl_init(int ncpu, int legacy_proc_tab, int port_tab_sz, int port_tab_sz_ignore_files, - int legacy_port_tab) + int legacy_port_tab, + int time_correction, + ErtsTimeWarpMode time_warp_mode) { init_benchmarking(); + erts_bif_unique_init(); erts_init_monitors(); - erts_init_time(); + erts_init_time(time_correction, time_warp_mode); erts_init_sys_common_misc(); erts_init_process(ncpu, proc_tab_sz, legacy_proc_tab); erts_init_scheduling(no_schedulers, @@ -313,6 +339,7 @@ erl_init(int ncpu, no_dirty_io_schedulers #endif ); + erts_late_init_time_sup(); erts_init_cpu_topology(); /* Must be after init_scheduling */ erts_init_gc(); /* Must be after init_scheduling */ erts_alloc_late_init(); @@ -362,12 +389,13 @@ erl_init(int ncpu, erl_nif_init(); } -static void +static Eterm erl_first_process_otp(char* modname, void* code, unsigned size, int argc, char** argv) { int i; Eterm start_mod; Eterm args; + Eterm res; Eterm* hp; Process parent; ErlSpawnOpts so; @@ -397,10 +425,11 @@ erl_first_process_otp(char* modname, void* code, unsigned size, int argc, char** hp += 2; args = CONS(hp, env, args); - so.flags = 0; - (void) erl_create_process(&parent, start_mod, am_start, args, &so); + so.flags = SPO_SYSTEM_PROC; + res = erl_create_process(&parent, start_mod, am_start, args, &so); erts_smp_proc_unlock(&parent, ERTS_PROC_LOCK_MAIN); erts_cleanup_empty_process(&parent); + return res; } Eterm @@ -506,9 +535,9 @@ void erts_usage(void) /* erts_fprintf(stderr, "-b func set the boot function (default boot)\n"); */ - erts_fprintf(stderr, "-c disable continuous date/time correction with\n"); - erts_fprintf(stderr, " respect to uptime\n"); - + erts_fprintf(stderr, "-c bool enable or disable time correction\n"); + erts_fprintf(stderr, "-C mode set time warp mode; valid modes are:\n"); + erts_fprintf(stderr, " no_time_warp|single_time_warp|multi_time_warp\n"); erts_fprintf(stderr, "-d don't write a crash dump for internally detected errors\n"); erts_fprintf(stderr, " (halt(String) will still produce a crash dump)\n"); erts_fprintf(stderr, "-fn[u|a|l] Control how filenames are interpreted\n"); @@ -516,6 +545,8 @@ void erts_usage(void) H_DEFAULT_SIZE); erts_fprintf(stderr, "-hmbs size set minimum binary virtual heap size in words (default %d)\n", VH_DEFAULT_SIZE); + erts_fprintf(stderr, "-hpds size initial process dictionary size (default %d)\n", + erts_pd_initial_size); /* erts_fprintf(stderr, "-i module set the boot module (default init)\n"); */ @@ -678,7 +709,6 @@ early_init(int *argc, char **argv) /* erts_sched_compact_load = 1; erts_printf_eterm_func = erts_printf_term; - erts_disable_tolerant_timeofday = 0; display_items = 200; erts_backtrace_depth = DEFAULT_BACKTRACE_SIZE; erts_async_max_threads = ERTS_DEFAULT_NO_ASYNC_THREADS; @@ -1184,7 +1214,11 @@ erl_start(int argc, char **argv) int port_tab_sz_ignore_files = 0; int legacy_proc_tab = 0; int legacy_port_tab = 0; + int time_correction; + ErtsTimeWarpMode time_warp_mode; + set_default_time_adj(&time_correction, + &time_warp_mode); envbufsz = sizeof(envbuf); if (erts_sys_getenv_raw(ERL_MAX_ETS_TABLES_ENV, envbuf, &envbufsz) == 0) @@ -1405,6 +1439,7 @@ erl_start(int argc, char **argv) * * h|ms - min_heap_size * h|mbs - min_bin_vheap_size + * h|pds - erts_pd_initial_size * */ if (has_prefix("mbs", sub_param)) { @@ -1422,6 +1457,14 @@ erl_start(int argc, char **argv) erts_usage(); } VERBOSE(DEBUG_SYSTEM, ("using minimum heap size %d\n", H_MIN_SIZE)); + } else if (has_prefix("pds", sub_param)) { + arg = get_arg(sub_param+3, argv[i+1], &i); + if ((erts_pd_initial_size = atoi(arg)) <= 0) { + erts_fprintf(stderr, "bad initial process dictionary size %s\n", arg); + erts_usage(); + } + VERBOSE(DEBUG_SYSTEM, ("using initial process dictionary size %d\n", + erts_pd_initial_size)); } else { /* backward compatibility */ arg = get_arg(argv[i]+2, argv[i+1], &i); @@ -1893,9 +1936,55 @@ erl_start(int argc, char **argv) } break; } + case 'C': + arg = get_arg(argv[i]+2, argv[i+1], &i); + if (sys_strcmp(arg, "no_time_warp") == 0) + time_warp_mode = ERTS_NO_TIME_WARP_MODE; + else if (sys_strcmp(arg, "single_time_warp") == 0) + time_warp_mode = ERTS_SINGLE_TIME_WARP_MODE; + else if (sys_strcmp(arg, "multi_time_warp") == 0) + time_warp_mode = ERTS_MULTI_TIME_WARP_MODE; + else { + erts_fprintf(stderr, + "Invalid time warp mode: %s\n", arg); + erts_usage(); + } + break; case 'c': - if (argv[i][2] == 0) { /* -c: documented option */ - erts_disable_tolerant_timeofday = 1; + if (sys_strcmp(argv[i]+2, "false") == 0) + goto time_correction_false; + else if (sys_strcmp(argv[i]+2, "true") == 0) + goto time_correction_true; +#ifdef ERTS_OPCODE_COUNTER_SUPPORT + else if (argv[i][2] == 'i') { /* -ci: undcoumented option*/ + count_instructions = 1; + } +#endif + else if (argv[i][2] == '\0') { + if (i + 1 >= argc) + goto time_correction_false; + else { + if (sys_strcmp(argv[i+1], "false") == 0) { + (void) get_arg(argv[i]+2, argv[i+1], &i); + goto time_correction_false; + } + else if (sys_strcmp(argv[i+1], "true") == 0) { + (void) get_arg(argv[i]+2, argv[i+1], &i); + time_correction_true: + time_correction = 1; + break; + } + else { + time_correction_false: + time_correction = 0; + break; + } + } + } + else { + arg = get_arg(argv[i]+2, argv[i+1], &i); + erts_fprintf(stderr, "Invalid time correnction value: %s\n", arg); + erts_usage(); } break; case 'W': @@ -1942,6 +2031,30 @@ erl_start(int argc, char **argv) i++; } + if (!erts_check_time_adj_support(time_correction, time_warp_mode)) { + char *time_correction_str = time_correction ? "Enabled" : "Disabled"; + char *time_warp_str = "undefined"; + switch (time_warp_mode) { + case ERTS_NO_TIME_WARP_MODE: + time_warp_str = "no"; + break; + case ERTS_SINGLE_TIME_WARP_MODE: + time_warp_str = "single"; + break; + case ERTS_MULTI_TIME_WARP_MODE: + time_warp_str = "multi"; + break; + default: + time_warp_str = "undefined"; + break; + } + erts_fprintf(stderr, "%s time correction with %s time warp mode " + "is not supported on this platform\n", + time_correction_str, + time_warp_str); + erts_usage(); + } + /* Output format on windows for sprintf defaults to three exponents. * We use two-exponent to mimic normal sprintf behaviour. */ @@ -1975,7 +2088,9 @@ erl_start(int argc, char **argv) legacy_proc_tab, port_tab_sz, port_tab_sz_ignore_files, - legacy_port_tab); + legacy_port_tab, + time_correction, + time_warp_mode); load_preloaded(); erts_end_staging_code_ix(); @@ -1983,7 +2098,11 @@ erl_start(int argc, char **argv) erts_initialized = 1; - erl_first_process_otp("otp_ring0", NULL, 0, boot_argc, boot_argv); + { + Eterm init = erl_first_process_otp("otp_ring0", NULL, 0, + boot_argc, boot_argv); + erts_bif_timer_start_servers(init); + } #ifdef ERTS_SMP erts_start_schedulers(); diff --git a/erts/emulator/beam/erl_lock_check.c b/erts/emulator/beam/erl_lock_check.c index b105ece6f1..261460d054 100644 --- a/erts/emulator/beam/erl_lock_check.c +++ b/erts/emulator/beam/erl_lock_check.c @@ -140,7 +140,6 @@ static erts_lc_lock_order_t erts_lock_order[] = { { "async_enq_mtx", NULL }, #ifdef ERTS_SMP { "atom_tab", NULL }, - { "make_ref", NULL }, { "misc_op_list_pre_alloc_lock", "address" }, { "message_pre_alloc_lock", "address" }, { "ptimer_pre_alloc_lock", "address", }, @@ -168,6 +167,8 @@ static erts_lc_lock_order_t erts_lock_order[] = { { "timer_wheel", NULL }, { "system_block", NULL }, { "timeofday", NULL }, + { "get_time", NULL }, + { "get_corrected_time", NULL }, { "breakpoints", NULL }, { "pollsets_lock", NULL }, { "pix_lock", "address" }, @@ -184,10 +185,8 @@ static erts_lc_lock_order_t erts_lock_order[] = { { "efile_drv dtrace mutex", NULL }, #endif { "mtrace_buf", NULL }, -#ifdef __WIN32__ #ifdef ERTS_SMP - { "sys_gethrtime", NULL }, -#endif + { "os_monotonic_time", NULL }, #endif { "erts_alloc_hard_debug", NULL }, { "hard_dbg_mseg", NULL }, diff --git a/erts/emulator/beam/erl_lock_count.c b/erts/emulator/beam/erl_lock_count.c index cf6996ea06..c6d8f4df95 100644 --- a/erts/emulator/beam/erl_lock_count.c +++ b/erts/emulator/beam/erl_lock_count.c @@ -104,17 +104,13 @@ static void lcnt_clear_stats(erts_lcnt_lock_stats_t *stats) { } static void lcnt_time(erts_lcnt_time_t *time) { -#if 0 || defined(HAVE_GETHRTIME) - SysHrTime hr_time; - hr_time = sys_gethrtime(); - time->s = (unsigned long)(hr_time / 1000000000LL); - time->ns = (unsigned long)(hr_time - 1000000000LL*time->s); -#else - SysTimeval tv; - sys_gettimeofday(&tv); - time->s = tv.tv_sec; - time->ns = tv.tv_usec*1000LL; -#endif + /* + * erts_sys_hrtime() is the highest resolution + * we could find, it may or may not be monotonic... + */ + ErtsMonotonicTime mtime = erts_sys_hrtime(); + time->s = (unsigned long) (mtime / 1000000000LL); + time->ns = (unsigned long) (mtime - 1000000000LL*time->s); } static void lcnt_time_diff(erts_lcnt_time_t *d, erts_lcnt_time_t *t1, erts_lcnt_time_t *t0) { diff --git a/erts/emulator/beam/erl_lock_count.h b/erts/emulator/beam/erl_lock_count.h index ffbb93da1b..09fadd7e9e 100644 --- a/erts/emulator/beam/erl_lock_count.h +++ b/erts/emulator/beam/erl_lock_count.h @@ -76,7 +76,7 @@ /* histogram */ #define ERTS_LCNT_HISTOGRAM_MAX_NS (((unsigned long)1LL << 28) - 1) -#if 0 || defined(HAVE_GETHRTIME) +#if 0 || defined(ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT) #define ERTS_LCNT_HISTOGRAM_SLOT_SIZE (30) #define ERTS_LCNT_HISTOGRAM_RSHIFT (0) #else diff --git a/erts/emulator/beam/erl_map.c b/erts/emulator/beam/erl_map.c index b2a16eb5ed..ab40f47919 100644 --- a/erts/emulator/beam/erl_map.c +++ b/erts/emulator/beam/erl_map.c @@ -16,6 +16,9 @@ * * %CopyrightEnd% * + * hashmaps are an adaption of Rich Hickeys Persistent HashMaps + * which were an adaption of Phil Bagwells - Hash Array Mapped Tries + * * Author: Björn-Egil Dahlberg */ @@ -62,39 +65,78 @@ * - erts_internal:map_to_tuple_keys/1 */ +#ifndef DECL_AM +#define DECL_AM(S) Eterm AM_ ## S = am_atom_put(#S, sizeof(#S) - 1) +#endif + +/* for hashmap_from_list/1 */ +typedef struct { + Uint32 hx; + Uint32 skip; + Uint i; + Eterm val; +} hxnode_t; + + +static Eterm flatmap_merge(Process *p, Eterm nodeA, Eterm nodeB); +static Eterm map_merge_mixed(Process *p, Eterm flat, Eterm tree, int swap_args); +static Eterm hashmap_merge(Process *p, Eterm nodeA, Eterm nodeB); +static Eterm hashmap_to_list(Process *p, Eterm map); +static Eterm hashmap_keys(Process *p, Eterm map); +static Eterm hashmap_values(Process *p, Eterm map); +static Eterm hashmap_delete(Process *p, Uint32 hx, Eterm key, Eterm node); +static Eterm flatmap_from_validated_list(Process *p, Eterm list, Uint size); +static Eterm hashmap_from_validated_list(Process *p, Eterm list, Uint size); +static Eterm hashmap_from_unsorted_array(ErtsHeapFactory*, hxnode_t *hxns, Uint n, int reject_dupkeys); +static Eterm hashmap_from_sorted_unique_array(ErtsHeapFactory*, hxnode_t *hxns, Uint n, int is_root); +static Eterm hashmap_from_chunked_array(ErtsHeapFactory*, hxnode_t *hxns, Uint n, int is_root); +static Eterm hashmap_info(Process *p, Eterm node); +static Eterm hashmap_bld_tuple_uint(Uint **hpp, Uint *szp, Uint n, Uint nums[]); +static int hxnodecmp(hxnode_t* a, hxnode_t* b); +static int hxnodecmpkey(hxnode_t* a, hxnode_t* b); + /* erlang:map_size/1 * the corresponding instruction is implemented in: * beam/erl_bif_guard.c */ BIF_RETTYPE map_size_1(BIF_ALIST_1) { - if (is_map(BIF_ARG_1)) { + if (is_flatmap(BIF_ARG_1)) { Eterm *hp; Uint hsz = 0; - map_t *mp = (map_t*)map_val(BIF_ARG_1); - Uint n = map_get_size(mp); + flatmap_t *mp = (flatmap_t*)flatmap_val(BIF_ARG_1); + Uint n = flatmap_get_size(mp); erts_bld_uint(NULL, &hsz, n); hp = HAlloc(BIF_P, hsz); BIF_RET(erts_bld_uint(&hp, NULL, n)); + } else if (is_hashmap(BIF_ARG_1)) { + Eterm *head, *hp, res; + Uint size, hsz=0; + + head = hashmap_val(BIF_ARG_1); + size = head[1]; + (void) erts_bld_uint(NULL, &hsz, size); + hp = HAlloc(BIF_P, hsz); + res = erts_bld_uint(&hp, NULL, size); + BIF_RET(res); } BIF_ERROR(BIF_P, BADARG); } -/* maps:to_list/1 - */ +/* maps:to_list/1 */ BIF_RETTYPE maps_to_list_1(BIF_ALIST_1) { - if (is_map(BIF_ARG_1)) { + if (is_flatmap(BIF_ARG_1)) { Uint n; Eterm* hp; Eterm *ks,*vs, res, tup; - map_t *mp = (map_t*)map_val(BIF_ARG_1); + flatmap_t *mp = (flatmap_t*)flatmap_val(BIF_ARG_1); - ks = map_get_keys(mp); - vs = map_get_values(mp); - n = map_get_size(mp); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + n = flatmap_get_size(mp); hp = HAlloc(BIF_P, (2 + 3) * n); res = NIL; @@ -104,6 +146,8 @@ BIF_RETTYPE maps_to_list_1(BIF_ALIST_1) { } BIF_RET(res); + } else if (is_hashmap(BIF_ARG_1)) { + return hashmap_to_list(BIF_P, BIF_ARG_1); } BIF_ERROR(BIF_P, BADARG); @@ -120,34 +164,41 @@ erts_maps_get_rel(Eterm key, Eterm map, Eterm *map_base) erts_maps_get(Eterm key, Eterm map) #endif { - Eterm *ks, *vs; - map_t *mp; - Uint n, i; + Uint32 hx; + if (is_flatmap_rel(map, map_base)) { + Eterm *ks, *vs; + flatmap_t *mp; + Uint n, i; - mp = (map_t *)map_val_rel(map, map_base); - n = map_get_size(mp); + mp = (flatmap_t *)flatmap_val_rel(map, map_base); + n = flatmap_get_size(mp); - if (n == 0) { - return NULL; - } + if (n == 0) { + return NULL; + } - ks = (Eterm *)tuple_val_rel(mp->keys, map_base) + 1; - vs = map_get_values(mp); + ks = (Eterm *)tuple_val_rel(mp->keys, map_base) + 1; + vs = flatmap_get_values(mp); - if (is_immed(key)) { - for (i = 0; i < n; i++) { - if (ks[i] == key) { - return &vs[i]; - } - } - } + if (is_immed(key)) { + for (i = 0; i < n; i++) { + if (ks[i] == key) { + return &vs[i]; + } + } + } - for (i = 0; i < n; i++) { - if (eq_rel(ks[i], NULL, key, map_base)) { - return &vs[i]; - } + for (i = 0; i < n; i++) { + if (eq_rel(ks[i], map_base, key, NULL)) { + return &vs[i]; + } + } + return NULL; } - return NULL; + ASSERT(is_hashmap_rel(map, map_base)); + hx = hashmap_make_hash(key); + + return erts_hashmap_get_rel(hx, key, map, map_base); } BIF_RETTYPE maps_find_2(BIF_ALIST_2) { @@ -164,7 +215,6 @@ BIF_RETTYPE maps_find_2(BIF_ALIST_2) { *hp++ = *value; BIF_RET(res); } - BIF_RET(am_error); } BIF_ERROR(BIF_P, BADARG); @@ -202,13 +252,8 @@ BIF_RETTYPE maps_get_2(BIF_ALIST_2) { */ BIF_RETTYPE maps_from_list_1(BIF_ALIST_1) { - Eterm *kv, item = BIF_ARG_1; - Eterm *hp, *thp,*vs, *ks, keys, res; - map_t *mp; - Uint size = 0, unused_size = 0; - Sint c = 0; - Sint idx = 0; - + Eterm item = BIF_ARG_1, res, *kv; + Uint size = 0; if (is_list(item) || is_nil(item)) { /* Calculate size and check validity */ @@ -229,368 +274,1085 @@ BIF_RETTYPE maps_from_list_1(BIF_ALIST_1) { if (is_not_nil(item)) goto error; - hp = HAlloc(BIF_P, 3 + 1 + (2 * size)); - thp = hp; + if (size > MAP_SMALL_MAP_LIMIT) { + BIF_RET(hashmap_from_validated_list(BIF_P, BIF_ARG_1, size)); + } else { + BIF_RET(flatmap_from_validated_list(BIF_P, BIF_ARG_1, size)); + } + } + +error: + + BIF_ERROR(BIF_P, BADARG); +} + +static Eterm flatmap_from_validated_list(Process *p, Eterm list, Uint size) { + Eterm *kv, item = list; + Eterm *hp, *thp,*vs, *ks, keys, res; + flatmap_t *mp; + Uint unused_size = 0; + Sint c = 0; + Sint idx = 0; + + + hp = HAlloc(p, 3 + 1 + (2 * size)); + thp = hp; + keys = make_tuple(hp); + *hp++ = make_arityval(size); + ks = hp; + hp += size; + mp = (flatmap_t*)hp; + res = make_flatmap(mp); + hp += MAP_HEADER_FLATMAP_SZ; + vs = hp; + + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = size; /* set later, might shrink*/ + mp->keys = keys; + + if (size == 0) + return res; + + /* first entry */ + kv = tuple_val(CAR(list_val(item))); + ks[0] = kv[1]; + vs[0] = kv[2]; + size = 1; + item = CDR(list_val(item)); + + /* insert sort key/value pairs */ + while(is_list(item)) { + + kv = tuple_val(CAR(list_val(item))); + + /* compare ks backwards + * idx represent word index to be written (hole position). + * We cannot copy the elements when searching since we might + * have an equal key. So we search for just the index first =( + * + * It is perhaps faster to move the values in the first pass. + * Check for uniqueness during insert phase and then have a + * second phace compacting the map if duplicates are found + * during insert. .. or do someother sort .. shell-sort perhaps. + */ + + idx = size; + + while(idx > 0 && (c = CMP_TERM(kv[1],ks[idx-1])) < 0) { idx--; } + + if (c == 0) { + /* last compare was equal, + * i.e. we have to release memory + * and overwrite that key/value + */ + ks[idx-1] = kv[1]; + vs[idx-1] = kv[2]; + unused_size++; + } else { + Uint i = size; + while(i > idx) { + ks[i] = ks[i-1]; + vs[i] = vs[i-1]; + i--; + } + ks[idx] = kv[1]; + vs[idx] = kv[2]; + size++; + } + item = CDR(list_val(item)); + } + + if (unused_size) { + /* the key tuple is embedded in the heap + * write a bignum to clear it. + */ + /* release values as normal since they are on the top of the heap */ + + ks[size] = make_pos_bignum_header(unused_size - 1); + HRelease(p, vs + size + unused_size, vs + size); + } + + *thp = make_arityval(size); + mp->size = size; + return res; +} + +#define swizzle32(D,S) \ + do { \ + (D) = ((S) & 0x0000000f) << 28 | ((S) & 0x000000f0) << 20 \ + | ((S) & 0x00000f00) << 12 | ((S) & 0x0000f000) << 4 \ + | ((S) & 0x000f0000) >> 4 | ((S) & 0x00f00000) >> 12 \ + | ((S) & 0x0f000000) >> 20 | ((S) & 0xf0000000) >> 28; \ + } while(0) + +#define maskval(V,L) (((V) >> ((7 - (L))*4)) & 0xf) +#define cdepth(V1,V2) (hashmap_clz((V1) ^ (V2)) >> 2) + +static Eterm hashmap_from_validated_list(Process *p, Eterm list, Uint size) { + Eterm item = list; + Eterm *hp; + Eterm *kv, res; + Eterm tmp[2]; + Uint32 sw, hx; + Uint ix = 0; + hxnode_t *hxns; + ErtsHeapFactory factory; + + ASSERT(size > 0); + + hp = HAlloc(p, (2 * size)); + + /* create tmp hx values and leaf ptrs */ + hxns = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP, size * sizeof(hxnode_t)); + + while(is_list(item)) { + res = CAR(list_val(item)); + kv = tuple_val(res); + hx = hashmap_restore_hash(tmp,0,kv[1]); + swizzle32(sw,hx); + hxns[ix].hx = sw; + hxns[ix].val = CONS(hp, kv[1], kv[2]); hp += 2; + hxns[ix].skip = 1; /* will be reassigned in from_array */ + hxns[ix].i = ix; + ix++; + item = CDR(list_val(item)); + } + + factory.p = p; + res = hashmap_from_unsorted_array(&factory, hxns, size, 0); + + erts_free(ERTS_ALC_T_TMP, (void *) hxns); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); + + if (hashmap_size(res) <= MAP_SMALL_MAP_LIMIT) { + DECLARE_WSTACK(wstack); + Eterm *kv, *ks, *vs; + flatmap_t *mp; + Eterm keys; + Uint n = hashmap_size(res); + + /* build flat structure */ + hp = HAlloc(p, 3 + 1 + (2 * n)); keys = make_tuple(hp); - *hp++ = make_arityval(size); + *hp++ = make_arityval(n); ks = hp; - hp += size; - mp = (map_t*)hp; - res = make_map(mp); - hp += MAP_HEADER_SIZE; + hp += n; + mp = (flatmap_t*)hp; + hp += MAP_HEADER_FLATMAP_SZ; vs = hp; - mp->thing_word = MAP_HEADER; - mp->size = size; /* set later, might shrink*/ + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = n; mp->keys = keys; - if (size == 0) - BIF_RET(res); + hashmap_iterator_init(&wstack, res, 0); - item = BIF_ARG_1; + while ((kv=hashmap_iterator_next(&wstack)) != NULL) { + *ks++ = CAR(kv); + *vs++ = CDR(kv); + } - /* first entry */ - kv = tuple_val(CAR(list_val(item))); - ks[0] = kv[1]; - vs[0] = kv[2]; - size = 1; - item = CDR(list_val(item)); + /* it cannot have multiple keys */ + erts_validate_and_sort_flatmap(mp); - /* insert sort key/value pairs */ - while(is_list(item)) { + DESTROY_WSTACK(wstack); + return make_flatmap(mp); + } - kv = tuple_val(CAR(list_val(item))); - - /* compare ks backwards - * idx represent word index to be written (hole position). - * We cannot copy the elements when searching since we might - * have an equal key. So we search for just the index first =( - * - * It is perhaps faster to move the values in the first pass. - * Check for uniqueness during insert phase and then have a - * second phace compacting the map if duplicates are found - * during insert. .. or do someother sort .. shell-sort perhaps. - */ + return res; +} + +Eterm erts_hashmap_from_array(ErtsHeapFactory* factory, Eterm *leafs, Uint n, + int reject_dupkeys) { + Uint32 sw, hx; + Uint ix; + hxnode_t *hxns; + Eterm res; + + /* create tmp hx values and leaf ptrs */ + hxns = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP, n * sizeof(hxnode_t)); + + for (ix = 0; ix < n; ix++) { + hx = hashmap_make_hash(*leafs); + swizzle32(sw,hx); + hxns[ix].hx = sw; + hxns[ix].val = make_list(leafs); + hxns[ix].skip = 1; + hxns[ix].i = ix; + leafs += 2; + } - idx = size; + res = hashmap_from_unsorted_array(factory, hxns, n, reject_dupkeys); - while(idx > 0 && (c = CMP_TERM(kv[1],ks[idx-1])) < 0) { idx--; } + erts_free(ERTS_ALC_T_TMP, (void *) hxns); - if (c == 0) { - /* last compare was equal, - * i.e. we have to release memory - * and overwrite that key/value - */ - ks[idx-1] = kv[1]; - vs[idx-1] = kv[2]; - unused_size++; - } else { - Uint i = size; - while(i > idx) { - ks[i] = ks[i-1]; - vs[i] = vs[i-1]; - i--; + return res; +} + + +Eterm erts_hashmap_from_ks_and_vs_extra(Process *p, Eterm *ks, Eterm *vs, Uint n, + Eterm key, Eterm value) { + Uint32 sw, hx; + Uint i,sz; + hxnode_t *hxns; + ErtsHeapFactory factory; + Eterm *hp, res; + + sz = (key == THE_NON_VALUE) ? n : (n + 1); + ASSERT(sz > MAP_SMALL_MAP_LIMIT); + hp = HAlloc(p, 2 * sz); + + /* create tmp hx values and leaf ptrs */ + hxns = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP, sz * sizeof(hxnode_t)); + + for(i = 0; i < n; i++) { + hx = hashmap_make_hash(ks[i]); + swizzle32(sw,hx); + hxns[i].hx = sw; + hxns[i].val = CONS(hp, ks[i], vs[i]); hp += 2; + hxns[i].skip = 1; /* will be reassigned in from_array */ + hxns[i].i = i; + } + + if (key != THE_NON_VALUE) { + hx = hashmap_make_hash(key); + swizzle32(sw,hx); + hxns[i].hx = sw; + hxns[i].val = CONS(hp, key, value); hp += 2; + hxns[i].skip = 1; + hxns[i].i = i; + } + + factory.p = p; + res = hashmap_from_unsorted_array(&factory, hxns, sz, 0); + + erts_free(ERTS_ALC_T_TMP, (void *) hxns); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); + + return res; +} + +static Eterm hashmap_from_unsorted_array(ErtsHeapFactory* factory, + hxnode_t *hxns, Uint n, + int reject_dupkeys) { + Uint jx = 0, ix = 0, lx, cx; + Eterm res; + + if (n == 0) { + Eterm *hp; + hp = erts_produce_heap(factory, 2, 0); + hp[0] = MAP_HEADER_HAMT_HEAD_BITMAP(0); + hp[1] = 0; + + return make_hashmap(hp); + } + + /* sort and compact array (remove non-unique entries) */ + qsort(hxns, n, sizeof(hxnode_t), (int (*)(const void *, const void *)) hxnodecmp); + + ix = 0, cx = 0; + while(ix < n - 1) { + if (hxns[ix].hx == hxns[ix+1].hx) { + + /* find region of equal hash values */ + jx = ix + 1; + while(jx < n && hxns[ix].hx == hxns[jx].hx) { jx++; } + /* find all correct keys from region + * (last in list but now hash sorted so we check highest id instead) */ + + /* resort with keys instead of hash value within region */ + + qsort(&hxns[ix], jx - ix, sizeof(hxnode_t), + (int (*)(const void *, const void *)) hxnodecmpkey); + + while(ix < jx) { + lx = ix; + while(ix < jx && EQ(CAR(list_val(hxns[ix].val)), CAR(list_val(hxns[lx].val)))) { + if (reject_dupkeys) + return THE_NON_VALUE; + + if (hxns[ix].i > hxns[lx].i) { + lx = ix; + } + ix++; } - ks[idx] = kv[1]; - vs[idx] = kv[2]; - size++; + hxns[cx].hx = hxns[lx].hx; + hxns[cx].val = hxns[lx].val; + cx++; } - item = CDR(list_val(item)); + ix = jx; + continue; + } + if (ix > cx) { + hxns[cx].hx = hxns[ix].hx; + hxns[cx].val = hxns[ix].val; } + cx++; + ix++; + } - if (unused_size) { - /* the key tuple is embedded in the heap - * write a bignum to clear it. - */ - /* release values as normal since they are on the top of the heap */ + if (ix < n) { + hxns[cx].hx = hxns[ix].hx; + hxns[cx].val = hxns[ix].val; + cx++; + } - ks[size] = make_pos_bignum_header(unused_size - 1); - HRelease(BIF_P, vs + size + unused_size, vs + size); - } + if (cx > 1) { + /* recursive decompose array */ + res = hashmap_from_sorted_unique_array(factory, hxns, cx, 0); + } else { + Eterm *hp; - *thp = make_arityval(size); - mp->size = size; - BIF_RET(res); + /* we only have one item, either because n was 1 or + * because we hade multiples of the same key. + * + * hash value has been swizzled, need to drag it down to get the + * correct slot. */ + + hp = erts_produce_heap(factory, HAMT_HEAD_BITMAP_SZ(1), 0); + hp[0] = MAP_HEADER_HAMT_HEAD_BITMAP(1 << ((hxns[0].hx >> 0x1c) & 0xf)); + hp[1] = 1; + hp[2] = hxns[0].val; + res = make_hashmap(hp); } -error: + return res; +} - BIF_ERROR(BIF_P, BADARG); +static Eterm hashmap_from_sorted_unique_array(ErtsHeapFactory* factory, + hxnode_t *hxns, Uint n, int lvl) { + Eterm res = NIL; + Uint i,ix,jx,elems; + Uint32 sw, hx; + Eterm val; + Eterm th[2]; + hxnode_t *tmp; + + ASSERT(lvl < 32); + ix = 0; + elems = 1; + while (ix < n - 1) { + if (hxns[ix].hx == hxns[ix+1].hx) { + jx = ix + 1; + while (jx < n && hxns[ix].hx == hxns[jx].hx) { jx++; } + tmp = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP, ((jx - ix)) * sizeof(hxnode_t)); + + for(i = 0; i < jx - ix; i++) { + val = hxns[i + ix].val; + hx = hashmap_restore_hash(th, lvl + 8, CAR(list_val(val))); + swizzle32(sw,hx); + tmp[i].hx = sw; + tmp[i].val = val; + tmp[i].i = i; + tmp[i].skip = 1; + } + + qsort(tmp, jx - ix, sizeof(hxnode_t), (int (*)(const void *, const void *)) hxnodecmp); + + hxns[ix].skip = jx - ix; + hxns[ix].val = hashmap_from_sorted_unique_array(factory, tmp, jx - ix, lvl + 8); + erts_free(ERTS_ALC_T_TMP, (void *) tmp); + ix = jx; + if (ix < n) { elems++; } + continue; + } + hxns[ix].skip = 1; + elems++; + ix++; + } + + res = hashmap_from_chunked_array(factory, hxns, elems, !lvl); + + ERTS_FACTORY_HOLE_CHECK(factory); + + return res; } -/* maps:is_key/2 - */ +#define HALLOC_EXTRA 200 +static Eterm hashmap_from_chunked_array(ErtsHeapFactory *factory, + hxnode_t *hxns, Uint n, int is_root) { + Uint ix, d, dn, dc, slot, elems; + Uint32 v, vp, vn, hdr; + Uint bp, sz; + DECLARE_ESTACK(stack); + Eterm res = NIL, *hp = NULL, *nhp; -BIF_RETTYPE maps_is_key_2(BIF_ALIST_2) { - if (is_map(BIF_ARG_2)) { - Eterm *ks, key; - map_t *mp; - Uint n,i; + ASSERT(n > 1); - mp = (map_t*)map_val(BIF_ARG_2); - key = BIF_ARG_1; - n = map_get_size(mp); - ks = map_get_keys(mp); + /* push initial nodes on the stack, + * this is the starting depth */ - if (n == 0) - BIF_RET(am_false); + ix = 0; + d = 0; + vp = hxns[ix].hx; + v = hxns[ix + hxns[ix].skip].hx; - if (is_immed(key)) { - for( i = 0; i < n; i++) { - if (ks[i] == key) { - BIF_RET(am_true); - } + ASSERT(vp > v); + slot = maskval(vp,d); + + while(slot == maskval(v,d)) { + ESTACK_PUSH(stack, 1 << slot); + d++; + slot = maskval(vp,d); + } + + res = hxns[ix].val; + + if (hxns[ix].skip > 1) { + dc = 7; + /* build collision nodes */ + while (dc > d) { + hp = erts_produce_heap(factory, HAMT_NODE_BITMAP_SZ(1), HALLOC_EXTRA); + hp[0] = MAP_HEADER_HAMT_NODE_BITMAP(1 << maskval(vp,dc)); + hp[1] = res; + res = make_hashmap(hp); + dc--; + } + } + + ESTACK_PUSH2(stack,res,1 << slot); + + /* all of the other nodes .. */ + elems = n - 2; /* remove first and last elements */ + while(elems--) { + hdr = ESTACK_POP(stack); + ix = ix + hxns[ix].skip; + + /* determine if node or subtree should be built by looking + * at the next value. */ + + vn = hxns[ix + hxns[ix].skip].hx; + dn = cdepth(v,vn); + ASSERT(v > vn); + + res = hxns[ix].val; + + if (hxns[ix].skip > 1) { + int wat = (d > dn) ? d : dn; + dc = 7; + /* build collision nodes */ + while (dc > wat) { + hp = erts_produce_heap(factory, HAMT_NODE_BITMAP_SZ(1), HALLOC_EXTRA); + hp[0] = MAP_HEADER_HAMT_NODE_BITMAP(1 << maskval(v,dc)); + hp[1] = res; + res = make_hashmap(hp); + dc--; } } - for( i = 0; i < n; i++) { - if (EQ(ks[i], key)) { - BIF_RET(am_true); + /* next depth is higher (implies collision) */ + if (d < dn) { + /* hdr is the popped one initially */ + while(d < dn) { + slot = maskval(v, d); + bp = 1 << slot; + ESTACK_PUSH(stack, hdr | bp); + d++; + hdr = 0; /* clear hdr for all other collisions */ } + + slot = maskval(v, d); + bp = 1 << slot; + /* no more collisions */ + ESTACK_PUSH2(stack,res,bp); + } else if (d == dn) { + /* no collisions at all */ + slot = maskval(v, d); + bp = 1 << slot; + ESTACK_PUSH2(stack,res,hdr | bp); + } else { + /* dn < n, we have a drop and we are done + * build nodes and subtree */ + while (dn != d) { + slot = maskval(v, d); + bp = 1 << slot; + /* OR bitposition before sz calculation to handle + * redundant collisions */ + hdr |= bp; + sz = hashmap_bitcount(hdr); + hp = erts_produce_heap(factory, HAMT_NODE_BITMAP_SZ(sz), HALLOC_EXTRA); + nhp = hp; + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hdr); + *hp++ = res; sz--; + while (sz--) { *hp++ = ESTACK_POP(stack); } + ASSERT((hp - nhp) < 18); + res = make_hashmap(nhp); + + /* we need to pop the next hdr and push if we don't need it */ + + hdr = ESTACK_POP(stack); + d--; + } + ESTACK_PUSH2(stack,res,hdr); } - BIF_RET(am_false); + + vp = v; + v = vn; + d = dn; + ERTS_FACTORY_HOLE_CHECK(factory); + } + + /* v and vp are reused from above */ + dn = cdepth(vp,v); + ix = ix + hxns[ix].skip; + res = hxns[ix].val; + + if (hxns[ix].skip > 1) { + dc = 7; + /* build collision nodes */ + while (dc > dn) { + hp = erts_produce_heap(factory, HAMT_NODE_BITMAP_SZ(1), HALLOC_EXTRA); + hp[0] = MAP_HEADER_HAMT_NODE_BITMAP(1 << maskval(v,dc)); + hp[1] = res; + res = make_hashmap(hp); + dc--; + } + } + + hdr = ESTACK_POP(stack); + /* pop remaining subtree if any */ + while (dn) { + slot = maskval(v, dn); + bp = 1 << slot; + /* OR bitposition before sz calculation to handle + * redundant collisions */ + hdr |= bp; + sz = hashmap_bitcount(hdr); + hp = erts_produce_heap(factory, HAMT_NODE_BITMAP_SZ(sz), HALLOC_EXTRA); + nhp = hp; + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hdr); + *hp++ = res; sz--; + + while (sz--) { *hp++ = ESTACK_POP(stack); } + res = make_hashmap(nhp); + hdr = ESTACK_POP(stack); + dn--; + } + + /* and finally the root .. */ + + slot = maskval(v, dn); + bp = 1 << slot; + hdr |= bp; + sz = hashmap_bitcount(hdr); + hp = erts_produce_heap(factory, sz + /* hdr + item */ (is_root ? 2 : 1), 0); + nhp = hp; + + if (is_root) { + *hp++ = (hdr == 0xffff) ? MAP_HEADER_HAMT_HEAD_ARRAY : MAP_HEADER_HAMT_HEAD_BITMAP(hdr); + *hp++ = n; + } else { + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hdr); + } + + *hp++ = res; sz--; + while (sz--) { *hp++ = ESTACK_POP(stack); } + + res = make_hashmap(nhp); + + ASSERT(ESTACK_COUNT(stack) == 0); + DESTROY_ESTACK(stack); + ERTS_FACTORY_HOLE_CHECK(factory); + return res; +} +#undef HALLOC_EXTRA + +static int hxnodecmpkey(hxnode_t *a, hxnode_t *b) { + return CMP_TERM(CAR(list_val(a->val)), CAR(list_val(b->val))); +} + +static int hxnodecmp(hxnode_t *a, hxnode_t *b) { + if (a->hx < b->hx) + return 1; + else if (a->hx == b->hx) + return 0; + else + return -1; +} + +/* maps:is_key/2 */ + +BIF_RETTYPE maps_is_key_2(BIF_ALIST_2) { + if (is_map(BIF_ARG_2)) { + BIF_RET(erts_maps_get(BIF_ARG_1, BIF_ARG_2) ? am_true : am_false); } BIF_ERROR(BIF_P, BADARG); } -/* maps:keys/1 - */ +/* maps:keys/1 */ BIF_RETTYPE maps_keys_1(BIF_ALIST_1) { - if (is_map(BIF_ARG_1)) { + if (is_flatmap(BIF_ARG_1)) { Eterm *hp, *ks, res = NIL; - map_t *mp; + flatmap_t *mp; Uint n; - mp = (map_t*)map_val(BIF_ARG_1); - n = map_get_size(mp); + mp = (flatmap_t*)flatmap_val(BIF_ARG_1); + n = flatmap_get_size(mp); if (n == 0) BIF_RET(res); hp = HAlloc(BIF_P, (2 * n)); - ks = map_get_keys(mp); + ks = flatmap_get_keys(mp); while(n--) { res = CONS(hp, ks[n], res); hp += 2; } BIF_RET(res); + } else if (is_hashmap(BIF_ARG_1)) { + BIF_RET(hashmap_keys(BIF_P, BIF_ARG_1)); } BIF_ERROR(BIF_P, BADARG); } -/* maps:merge/2 - */ +/* maps:merge/2 */ BIF_RETTYPE maps_merge_2(BIF_ALIST_2) { - if (is_map(BIF_ARG_1) && is_map(BIF_ARG_2)) { - Eterm *hp,*thp; - Eterm tup; - Eterm *ks,*vs,*ks1,*vs1,*ks2,*vs2; - map_t *mp1,*mp2,*mp_new; - Uint n1,n2,i1,i2,need,unused_size=0; - int c = 0; - - mp1 = (map_t*)map_val(BIF_ARG_1); - mp2 = (map_t*)map_val(BIF_ARG_2); - n1 = map_get_size(mp1); - n2 = map_get_size(mp2); - - need = MAP_HEADER_SIZE + 1 + 2*(n1 + n2); - - hp = HAlloc(BIF_P, need); - thp = hp; - tup = make_tuple(thp); - ks = hp + 1; hp += 1 + n1 + n2; - mp_new = (map_t*)hp; hp += MAP_HEADER_SIZE; - vs = hp; hp += n1 + n2; - - mp_new->thing_word = MAP_HEADER; - mp_new->size = 0; - mp_new->keys = tup; - - i1 = 0; i2 = 0; - ks1 = map_get_keys(mp1); - vs1 = map_get_values(mp1); - ks2 = map_get_keys(mp2); - vs2 = map_get_values(mp2); - - while(i1 < n1 && i2 < n2) { - c = CMP_TERM(ks1[i1],ks2[i2]); - if ( c == 0) { - /* use righthand side arguments map value, - * but advance both maps */ - *ks++ = ks2[i2]; - *vs++ = vs2[i2]; - i1++, i2++, unused_size++; - } else if ( c < 0) { - *ks++ = ks1[i1]; - *vs++ = vs1[i1]; - i1++; - } else { - *ks++ = ks2[i2]; - *vs++ = vs2[i2]; - i2++; - } + if (is_flatmap(BIF_ARG_1)) { + if (is_flatmap(BIF_ARG_2)) { + BIF_RET(flatmap_merge(BIF_P, BIF_ARG_1, BIF_ARG_2)); + } else if (is_hashmap(BIF_ARG_2)) { + /* Will always become a tree */ + BIF_RET(map_merge_mixed(BIF_P, BIF_ARG_1, BIF_ARG_2, 0)); } + } else if (is_hashmap(BIF_ARG_1)) { + if (is_hashmap(BIF_ARG_2)) { + BIF_RET(hashmap_merge(BIF_P, BIF_ARG_1, BIF_ARG_2)); + } else if (is_flatmap(BIF_ARG_2)) { + /* Will always become a tree */ + BIF_RET(map_merge_mixed(BIF_P, BIF_ARG_2, BIF_ARG_1, 1)); + } + } + BIF_ERROR(BIF_P, BADARG); +} - /* copy remaining */ - while (i1 < n1) { +static Eterm flatmap_merge(Process *p, Eterm nodeA, Eterm nodeB) { + Eterm *hp,*thp; + Eterm tup; + Eterm *ks,*vs,*ks1,*vs1,*ks2,*vs2; + flatmap_t *mp1,*mp2,*mp_new; + Uint n,n1,n2,i1,i2,need,unused_size=0; + Sint c = 0; + + mp1 = (flatmap_t*)flatmap_val(nodeA); + mp2 = (flatmap_t*)flatmap_val(nodeB); + n1 = flatmap_get_size(mp1); + n2 = flatmap_get_size(mp2); + + need = MAP_HEADER_FLATMAP_SZ + 1 + 2 * (n1 + n2); + + hp = HAlloc(p, need); + thp = hp; + tup = make_tuple(thp); + ks = hp + 1; hp += 1 + n1 + n2; + mp_new = (flatmap_t*)hp; hp += MAP_HEADER_FLATMAP_SZ; + vs = hp; hp += n1 + n2; + + mp_new->thing_word = MAP_HEADER_FLATMAP; + mp_new->size = 0; + mp_new->keys = tup; + + i1 = 0; i2 = 0; + ks1 = flatmap_get_keys(mp1); + vs1 = flatmap_get_values(mp1); + ks2 = flatmap_get_keys(mp2); + vs2 = flatmap_get_values(mp2); + + while(i1 < n1 && i2 < n2) { + c = CMP_TERM(ks1[i1],ks2[i2]); + if (c == 0) { + /* use righthand side arguments map value, + * but advance both maps */ + *ks++ = ks2[i2]; + *vs++ = vs2[i2]; + i1++, i2++, unused_size++; + } else if (c < 0) { *ks++ = ks1[i1]; *vs++ = vs1[i1]; i1++; - } - - while (i2 < n2) { + } else { *ks++ = ks2[i2]; *vs++ = vs2[i2]; i2++; } + } - if (unused_size) { - /* the key tuple is embedded in the heap, write a bignum to clear it. - * - * release values as normal since they are on the top of the heap - * size = n1 + n1 - unused_size - */ + /* copy remaining */ + while (i1 < n1) { + *ks++ = ks1[i1]; + *vs++ = vs1[i1]; + i1++; + } - *ks = make_pos_bignum_header(unused_size - 1); - HRelease(BIF_P, vs + unused_size, vs); - } + while (i2 < n2) { + *ks++ = ks2[i2]; + *vs++ = vs2[i2]; + i2++; + } - mp_new->size = n1 + n2 - unused_size; - *thp = make_arityval(n1 + n2 - unused_size); + if (unused_size) { + /* the key tuple is embedded in the heap, write a bignum to clear it. + * + * release values as normal since they are on the top of the heap + * size = n1 + n1 - unused_size + */ - BIF_RET(make_map(mp_new)); + *ks = make_pos_bignum_header(unused_size - 1); + HRelease(p, vs + unused_size, vs); } - BIF_ERROR(BIF_P, BADARG); -} -/* maps:new/2 - */ -BIF_RETTYPE maps_new_0(BIF_ALIST_0) { - Eterm* hp; - Eterm tup; - map_t *mp; + n = n1 + n2 - unused_size; + *thp = make_arityval(n); + mp_new->size = n; - hp = HAlloc(BIF_P, (MAP_HEADER_SIZE + 1)); - tup = make_tuple(hp); - *hp++ = make_arityval(0); + /* Reshape map to a hashmap if the map exceeds the limit */ - mp = (map_t*)hp; - mp->thing_word = MAP_HEADER; - mp->size = 0; - mp->keys = tup; + if (n > MAP_SMALL_MAP_LIMIT) { + Uint32 hx,sw; + Uint i; + Eterm res; + hxnode_t *hxns; + ErtsHeapFactory factory; - BIF_RET(make_map(mp)); -} + ks = flatmap_get_keys(mp_new); + vs = flatmap_get_values(mp_new); -/* maps:put/3 - */ + hp = HAlloc(p, 2 * n); -Eterm erts_maps_put(Process *p, Eterm key, Eterm value, Eterm map) { - Sint n,i; - Sint c = 0; - Eterm* hp, *shp; - Eterm *ks,*vs, res, tup; - map_t *mp = (map_t*)map_val(map); + hxns = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP,n * sizeof(hxnode_t)); - n = map_get_size(mp); + for (i = 0; i < n; i++) { + hx = hashmap_make_hash(ks[i]); + swizzle32(sw,hx); + hxns[i].hx = sw; + hxns[i].val = CONS(hp, ks[i], vs[i]); hp += 2; + hxns[i].skip = 1; + hxns[i].i = i; + } - if (n == 0) { - hp = HAlloc(p, MAP_HEADER_SIZE + 1 + 2); - tup = make_tuple(hp); - *hp++ = make_arityval(1); - *hp++ = key; - res = make_map(hp); - *hp++ = MAP_HEADER; - *hp++ = 1; - *hp++ = tup; - *hp++ = value; + factory.p = p; + res = hashmap_from_unsorted_array(&factory, hxns, n, 0); + + erts_free(ERTS_ALC_T_TMP, (void *) hxns); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); return res; } - ks = map_get_keys(mp); - vs = map_get_values(mp); + return make_flatmap(mp_new); +} + +static Eterm map_merge_mixed(Process *p, Eterm flat, Eterm tree, int swap_args) { + Eterm *ks, *vs, *hp, res; + flatmap_t *mp; + Uint n, i; + hxnode_t *hxns; + Uint32 sw, hx; + ErtsHeapFactory factory; + + /* convert flat to tree */ + + ASSERT(is_flatmap(flat)); + ASSERT(is_hashmap(tree)); + + mp = (flatmap_t*)flatmap_val(flat); + n = flatmap_get_size(mp); + + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + hp = HAlloc(p, 2 * n); + + hxns = (hxnode_t *)erts_alloc(ERTS_ALC_T_TMP, n * sizeof(hxnode_t)); + + for (i = 0; i < n; i++) { + hx = hashmap_make_hash(ks[i]); + swizzle32(sw,hx); + hxns[i].hx = sw; + hxns[i].val = CONS(hp, ks[i], vs[i]); hp += 2; + hxns[i].skip = 1; + hxns[i].i = i; + } + + factory.p = p; + res = hashmap_from_unsorted_array(&factory, hxns, n, 0); + + erts_free(ERTS_ALC_T_TMP, (void *) hxns); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); + + return swap_args ? hashmap_merge(p, tree, res) : hashmap_merge(p, res, tree); +} - /* only allocate for values, - * assume key-tuple will be intact +#define HALLOC_EXTRA 200 + +static Eterm hashmap_merge(Process *p, Eterm nodeA, Eterm nodeB) { +#define PSTACK_TYPE struct HashmapMergePStackType + struct HashmapMergePStackType { + Eterm *srcA, *srcB; + Uint32 abm, bbm, rbm; /* node bitmaps */ + int keepA; + int ix; + Eterm array[16]; + }; + PSTACK_DECLARE(s, 4); + struct HashmapMergePStackType* sp = PSTACK_PUSH(s); + Eterm *hp, *nhp; + Eterm hdrA, hdrB; + Eterm th[2]; + Uint32 ahx, bhx; + Uint size; /* total key-value counter */ + int keepA = 0; + unsigned lvl = 0; + Eterm res = THE_NON_VALUE; + + /* + * Strategy: Do depth-first traversal of both trees (at the same time) + * and merge each pair of nodes. */ - hp = HAlloc(p, MAP_HEADER_SIZE + n); - shp = hp; /* save hp, used if optimistic update fails */ - res = make_map(hp); - *hp++ = MAP_HEADER; - *hp++ = n; - *hp++ = mp->keys; - - if (is_immed(key)) { - for( i = 0; i < n; i ++) { - if (ks[i] == key) { - *hp++ = value; - vs++; - c = 1; + { + hashmap_head_t* a = (hashmap_head_t*) hashmap_val(nodeA); + hashmap_head_t* b = (hashmap_head_t*) hashmap_val(nodeB); + size = a->size + b->size; + } + +recurse: + + if (primary_tag(nodeA) == TAG_PRIMARY_BOXED && + primary_tag(nodeB) == TAG_PRIMARY_LIST) { + /* Avoid implementing this combination by switching places */ + Eterm tmp = nodeA; + nodeA = nodeB; + nodeB = tmp; + keepA = !keepA; + } + + switch (primary_tag(nodeA)) { + case TAG_PRIMARY_LIST: { + sp->srcA = list_val(nodeA); + switch (primary_tag(nodeB)) { + case TAG_PRIMARY_LIST: { /* LEAF + LEAF */ + sp->srcB = list_val(nodeB); + + if (EQ(CAR(sp->srcA), CAR(sp->srcB))) { + --size; + res = keepA ? nodeA : nodeB; } else { - *hp++ = *vs++; + ahx = hashmap_restore_hash(th, lvl, CAR(sp->srcA)); + bhx = hashmap_restore_hash(th, lvl, CAR(sp->srcB)); + sp->abm = 1 << hashmap_index(ahx); + sp->bbm = 1 << hashmap_index(bhx); + + sp->srcA = &nodeA; + sp->srcB = &nodeB; } + break; } - } else { - for( i = 0; i < n; i ++) { - if (EQ(ks[i], key)) { - *hp++ = value; - vs++; - c = 1; - } else { - *hp++ = *vs++; + case TAG_PRIMARY_BOXED: { /* LEAF + NODE */ + sp->srcB = boxed_val(nodeB); + ASSERT(is_header(*sp->srcB)); + hdrB = *sp->srcB++; + + ahx = hashmap_restore_hash(th, lvl, CAR(sp->srcA)); + sp->abm = 1 << hashmap_index(ahx); + sp->srcA = &nodeA; + switch(hdrB & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + sp->srcB++; + sp->bbm = 0xffff; + break; + + case HAMT_SUBTAG_HEAD_BITMAP: sp->srcB++; + case HAMT_SUBTAG_NODE_BITMAP: + sp->bbm = MAP_HEADER_VAL(hdrB); + break; + + default: + erl_exit(1, "bad header tag %ld\r\n", *sp->srcB & _HEADER_MAP_SUBTAG_MASK); + break; } + break; + } + default: + erl_exit(1, "bad primary tag %ld\r\n", nodeB); } + break; } + case TAG_PRIMARY_BOXED: { /* NODE + NODE */ + sp->srcA = boxed_val(nodeA); + hdrA = *sp->srcA++; + ASSERT(is_header(hdrA)); + switch (hdrA & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: { + sp->srcA++; + ASSERT(primary_tag(nodeB) == TAG_PRIMARY_BOXED); + sp->abm = 0xffff; + sp->srcB = boxed_val(nodeB); + hdrB = *sp->srcB++; + ASSERT(is_header(hdrB)); + switch (hdrB & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + sp->srcB++; + sp->bbm = 0xffff; + break; + case HAMT_SUBTAG_HEAD_BITMAP: sp->srcB++; + case HAMT_SUBTAG_NODE_BITMAP: + sp->bbm = MAP_HEADER_VAL(hdrB); + break; + default: + erl_exit(1, "bad header tag %ld\r\n", *sp->srcB & _HEADER_MAP_SUBTAG_MASK); + } + break; + } + case HAMT_SUBTAG_HEAD_BITMAP: sp->srcA++; + case HAMT_SUBTAG_NODE_BITMAP: { + ASSERT(primary_tag(nodeB) == TAG_PRIMARY_BOXED); + sp->abm = MAP_HEADER_VAL(hdrA); + sp->srcB = boxed_val(nodeB); + hdrB = *sp->srcB++; + ASSERT(is_header(hdrB)); + switch (hdrB & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + sp->srcB++; + sp->bbm = 0xffff; + break; + case HAMT_SUBTAG_HEAD_BITMAP: sp->srcB++; + case HAMT_SUBTAG_NODE_BITMAP: + sp->bbm = MAP_HEADER_VAL(hdrB); + break; - if (c) - return res; + default: + erl_exit(1, "bad header tag %ld\r\n", *sp->srcB & _HEADER_MAP_SUBTAG_MASK); + } + break; + } + default: + erl_exit(1, "bad primary tag %ld\r\n", nodeA); + } + break; + } + default: + erl_exit(1, "bad primary tag %ld\r\n", nodeA); + } - /* need to make a new tuple, - * use old hp since it needs to be recreated anyway. - */ - tup = make_tuple(shp); - *shp++ = make_arityval(n+1); + for (;;) { + if (is_value(res)) { /* We have a complete (sub-)tree or leaf */ + if (lvl == 0) + break; + + /* Pop from stack and continue build parent node */ + lvl--; + sp = PSTACK_POP(s); + sp->array[sp->ix++] = res; + res = THE_NON_VALUE; + if (sp->rbm) { + sp->srcA++; + sp->srcB++; + keepA = sp->keepA; + } + } else { /* Start build a node */ + sp->ix = 0; + sp->rbm = sp->abm | sp->bbm; + ASSERT(!(sp->rbm == 0 && lvl > 0)); + } - hp = HAlloc(p, 3 + n + 1); - res = make_map(hp); - *hp++ = MAP_HEADER; - *hp++ = n + 1; - *hp++ = tup; + while (sp->rbm) { + Uint32 next = sp->rbm & (sp->rbm-1); + Uint32 bit = sp->rbm ^ next; + sp->rbm = next; + if (sp->abm & bit) { + if (sp->bbm & bit) { + /* Bit clash. Push and resolve by recursive merge */ + if (sp->rbm) { + sp->keepA = keepA; + } + nodeA = *sp->srcA; + nodeB = *sp->srcB; + lvl++; + sp = PSTACK_PUSH(s); + goto recurse; + } else { + sp->array[sp->ix++] = *sp->srcA++; + } + } else { + ASSERT(sp->bbm & bit); + sp->array[sp->ix++] = *sp->srcB++; + } + } - ks = map_get_keys(mp); - vs = map_get_values(mp); + ASSERT(sp->ix == hashmap_bitcount(sp->abm | sp->bbm)); + if (lvl == 0) { + nhp = HAllocX(p, HAMT_HEAD_BITMAP_SZ(sp->ix), HALLOC_EXTRA); + hp = nhp; + *hp++ = (sp->ix == 16 ? MAP_HEADER_HAMT_HEAD_ARRAY + : MAP_HEADER_HAMT_HEAD_BITMAP(sp->abm | sp->bbm)); + *hp++ = size; + } else { + nhp = HAllocX(p, HAMT_NODE_BITMAP_SZ(sp->ix), HALLOC_EXTRA); + hp = nhp; + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(sp->abm | sp->bbm); + } + memcpy(hp, sp->array, sp->ix * sizeof(Eterm)); + res = make_boxed(nhp); + } + PSTACK_DESTROY(s); + return res; +} - ASSERT(n >= 0); +static int hash_cmp(Uint32 ha, Uint32 hb) +{ + int i; + for (i=0; i<8; i++) { + int cmp = (int)(ha & 0xF) - (int)(hb & 0xF); + if (cmp) + return cmp; + ha >>= 4; + hb >>= 4; + } + return 0; +} - /* copy map in order */ - while (n && ((c = CMP_TERM(*ks, key)) < 0)) { - *shp++ = *ks++; - *hp++ = *vs++; - n--; +int hashmap_key_hash_cmp(Eterm* ap, Eterm* bp) +{ + Eterm th[2]; + unsigned lvl = 0; + + if (ap && bp) { + ASSERT(CMP_TERM(CAR(ap), CAR(bp)) != 0); + for (;;) { + Uint32 ha = hashmap_restore_hash(th, lvl, CAR(ap)); + Uint32 hb = hashmap_restore_hash(th, lvl, CAR(bp)); + int cmp = hash_cmp(ha, hb); + if (cmp) + return cmp; + lvl += 8; + } } + return ap ? -1 : 1; +} + +/* maps:new/0 */ - *shp++ = key; - *hp++ = value; +BIF_RETTYPE maps_new_0(BIF_ALIST_0) { + Eterm* hp; + Eterm tup; + flatmap_t *mp; - ASSERT(n >= 0); + hp = HAlloc(BIF_P, (MAP_HEADER_FLATMAP_SZ + 1)); + tup = make_tuple(hp); + *hp++ = make_arityval(0); - while(n--) { - *shp++ = *ks++; - *hp++ = *vs++; - } - /* we have one word remaining - * this will work out fine once we get the size word - * in the header. - */ - *shp = make_pos_bignum_header(0); - return res; + mp = (flatmap_t*)hp; + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = 0; + mp->keys = tup; + + BIF_RET(make_flatmap(mp)); } +/* maps:put/3 */ + BIF_RETTYPE maps_put_3(BIF_ALIST_3) { if (is_map(BIF_ARG_3)) { BIF_RET(erts_maps_put(BIF_P, BIF_ARG_1, BIF_ARG_2, BIF_ARG_3)); @@ -598,81 +1360,87 @@ BIF_RETTYPE maps_put_3(BIF_ALIST_3) { BIF_ERROR(BIF_P, BADARG); } -/* maps:remove/3 - */ +/* maps:remove/3 */ int erts_maps_remove(Process *p, Eterm key, Eterm map, Eterm *res) { - Sint n; - Uint need; - Eterm *hp_start; - Eterm *thp, *mhp; - Eterm *ks, *vs, tup; - map_t *mp = (map_t*)map_val(map); + Uint32 hx; + if (is_flatmap(map)) { + Sint n; + Uint need; + Eterm *hp_start; + Eterm *thp, *mhp; + Eterm *ks, *vs, tup; + flatmap_t *mp = (flatmap_t*)flatmap_val(map); + + n = flatmap_get_size(mp); + + if (n == 0) { + *res = map; + return 1; + } - n = map_get_size(mp); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); - if (n == 0) { - *res = map; - return 1; - } + /* Assume key exists. + * Release allocated if it didn't. + * Allocate key tuple first. + */ - ks = map_get_keys(mp); - vs = map_get_values(mp); + need = n + 1 - 1 + 3 + n - 1; /* tuple - 1 + map - 1 */ + hp_start = HAlloc(p, need); + thp = hp_start; + mhp = thp + n; /* offset with tuple heap size */ - /* Assume key exists. - * Release allocated if it didn't. - * Allocate key tuple first. - */ + tup = make_tuple(thp); + *thp++ = make_arityval(n - 1); - need = n + 1 - 1 + 3 + n - 1; /* tuple - 1 + map - 1 */ - hp_start = HAlloc(p, need); - thp = hp_start; - mhp = thp + n; /* offset with tuple heap size */ + *res = make_flatmap(mhp); + *mhp++ = MAP_HEADER_FLATMAP; + *mhp++ = n - 1; + *mhp++ = tup; - tup = make_tuple(thp); - *thp++ = make_arityval(n - 1); - - *res = make_map(mhp); - *mhp++ = MAP_HEADER; - *mhp++ = n - 1; - *mhp++ = tup; - - if (is_immed(key)) { - while (1) { - if (*ks == key) { - goto found_key; - } else if (--n) { - *mhp++ = *vs++; - *thp++ = *ks++; - } else - break; - } - } else { - while(1) { - if (EQ(*ks, key)) { - goto found_key; - } else if (--n) { - *mhp++ = *vs++; - *thp++ = *ks++; - } else - break; + if (is_immed(key)) { + while (1) { + if (*ks == key) { + goto found_key; + } else if (--n) { + *mhp++ = *vs++; + *thp++ = *ks++; + } else + break; + } + } else { + while(1) { + if (EQ(*ks, key)) { + goto found_key; + } else if (--n) { + *mhp++ = *vs++; + *thp++ = *ks++; + } else + break; + } } - } - /* Not found, remove allocated memory - * and return previous map. - */ - HRelease(p, hp_start + need, hp_start); + /* Not found, remove allocated memory + * and return previous map. + */ + HRelease(p, hp_start + need, hp_start); - *res = map; - return 1; + *res = map; + return 1; found_key: - /* Copy rest of keys and values */ - if (--n) { - sys_memcpy(mhp, vs+1, n*sizeof(Eterm)); - sys_memcpy(thp, ks+1, n*sizeof(Eterm)); + /* Copy rest of keys and values */ + if (--n) { + sys_memcpy(mhp, vs+1, n*sizeof(Eterm)); + sys_memcpy(thp, ks+1, n*sizeof(Eterm)); + } + return 1; } + ASSERT(is_hashmap(map)); + hx = hashmap_make_hash(key); + *res = hashmap_delete(p, hx, key, map); return 1; } @@ -686,29 +1454,28 @@ BIF_RETTYPE maps_remove_2(BIF_ALIST_2) { BIF_ERROR(BIF_P, BADARG); } -/* maps:update/3 - */ - int erts_maps_update(Process *p, Eterm key, Eterm value, Eterm map, Eterm *res) { + Uint32 hx; + if (is_flatmap(map)) { Sint n,i; Eterm* hp,*shp; Eterm *ks,*vs; - map_t *mp = (map_t*)map_val(map); + flatmap_t *mp = (flatmap_t*)flatmap_val(map); - if ((n = map_get_size(mp)) == 0) { + if ((n = flatmap_get_size(mp)) == 0) { return 0; } - ks = map_get_keys(mp); - vs = map_get_values(mp); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); /* only allocate for values, * assume key-tuple will be intact */ - hp = HAlloc(p, MAP_HEADER_SIZE + n); + hp = HAlloc(p, MAP_HEADER_FLATMAP_SZ + n); shp = hp; - *hp++ = MAP_HEADER; + *hp++ = MAP_HEADER_FLATMAP; *hp++ = n; *hp++ = mp->keys; @@ -730,7 +1497,7 @@ int erts_maps_update(Process *p, Eterm key, Eterm value, Eterm map, Eterm *res) } } - HRelease(p, shp + MAP_HEADER_SIZE + n, shp); + HRelease(p, shp + MAP_HEADER_FLATMAP_SZ + n, shp); return 0; found_key: @@ -738,10 +1505,147 @@ found_key: vs++; if (++i < n) sys_memcpy(hp, vs, (n - i)*sizeof(Eterm)); - *res = make_map(shp); + *res = make_flatmap(shp); return 1; + } + + ASSERT(is_hashmap(map)); + hx = hashmap_make_hash(key); + *res = erts_hashmap_insert(p, hx, key, value, map, 1); + if (is_value(*res)) + return 1; + + return 0; } +Eterm erts_maps_put(Process *p, Eterm key, Eterm value, Eterm map) { + Uint32 hx; + Eterm res; + if (is_flatmap(map)) { + Sint n,i; + Sint c = 0; + Eterm* hp, *shp; + Eterm *ks, *vs, tup; + flatmap_t *mp = (flatmap_t*)flatmap_val(map); + + n = flatmap_get_size(mp); + + if (n == 0) { + hp = HAlloc(p, MAP_HEADER_FLATMAP_SZ + 1 + 2); + tup = make_tuple(hp); + *hp++ = make_arityval(1); + *hp++ = key; + res = make_flatmap(hp); + *hp++ = MAP_HEADER_FLATMAP; + *hp++ = 1; + *hp++ = tup; + *hp++ = value; + + return res; + } + + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + /* only allocate for values, + * assume key-tuple will be intact + */ + + hp = HAlloc(p, MAP_HEADER_FLATMAP_SZ + n); + shp = hp; /* save hp, used if optimistic update fails */ + res = make_flatmap(hp); + *hp++ = MAP_HEADER_FLATMAP; + *hp++ = n; + *hp++ = mp->keys; + + if (is_immed(key)) { + for( i = 0; i < n; i ++) { + if (ks[i] == key) { + *hp++ = value; + vs++; + c = 1; + } else { + *hp++ = *vs++; + } + } + } else { + for( i = 0; i < n; i ++) { + if (EQ(ks[i], key)) { + *hp++ = value; + vs++; + c = 1; + } else { + *hp++ = *vs++; + } + } + } + + if (c) + return res; + + /* the map will grow */ + + if (n >= MAP_SMALL_MAP_LIMIT) { + HRelease(p, shp + MAP_HEADER_FLATMAP_SZ + n, shp); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + res = erts_hashmap_from_ks_and_vs_extra(p,ks,vs,n,key,value); + + return res; + } + + /* still a small map. need to make a new tuple, + * use old hp since it needs to be recreated anyway. */ + + tup = make_tuple(shp); + *shp++ = make_arityval(n+1); + + hp = HAlloc(p, 3 + n + 1); + res = make_flatmap(hp); + *hp++ = MAP_HEADER_FLATMAP; + *hp++ = n + 1; + *hp++ = tup; + + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + ASSERT(n >= 0); + + /* copy map in order */ + while (n && ((c = CMP_TERM(*ks, key)) < 0)) { + *shp++ = *ks++; + *hp++ = *vs++; + n--; + } + + *shp++ = key; + *hp++ = value; + + ASSERT(n >= 0); + + while(n--) { + *shp++ = *ks++; + *hp++ = *vs++; + } + /* we have one word remaining + * this will work out fine once we get the size word + * in the header. + */ + *shp = make_pos_bignum_header(0); + return res; + } + ASSERT(is_hashmap(map)); + + hx = hashmap_make_hash(key); + res = erts_hashmap_insert(p, hx, key, value, map, 0); + ASSERT(is_hashmap(res)); + + return res; +} + +/* maps:update/3 */ + BIF_RETTYPE maps_update_3(BIF_ALIST_3) { if (is_map(BIF_ARG_3)) { Eterm res; @@ -753,41 +1657,769 @@ BIF_RETTYPE maps_update_3(BIF_ALIST_3) { } -/* maps:values/1 - */ +/* maps:values/1 */ BIF_RETTYPE maps_values_1(BIF_ALIST_1) { - if (is_map(BIF_ARG_1)) { + if (is_flatmap(BIF_ARG_1)) { Eterm *hp, *vs, res = NIL; - map_t *mp; + flatmap_t *mp; Uint n; - mp = (map_t*)map_val(BIF_ARG_1); - n = map_get_size(mp); + mp = (flatmap_t*)flatmap_val(BIF_ARG_1); + n = flatmap_get_size(mp); if (n == 0) BIF_RET(res); hp = HAlloc(BIF_P, (2 * n)); - vs = map_get_values(mp); + vs = flatmap_get_values(mp); while(n--) { res = CONS(hp, vs[n], res); hp += 2; } BIF_RET(res); + } else if (is_hashmap(BIF_ARG_1)) { + BIF_RET(hashmap_values(BIF_P, BIF_ARG_1)); } BIF_ERROR(BIF_P, BADARG); } -int erts_validate_and_sort_map(map_t* mp) +static Eterm hashmap_to_list(Process *p, Eterm node) { + DECLARE_WSTACK(stack); + Eterm *hp, *kv; + Eterm res = NIL; + + hp = HAlloc(p, hashmap_size(node) * (2 + 3)); + hashmap_iterator_init(&stack, node, 0); + while ((kv=hashmap_iterator_next(&stack)) != NULL) { + Eterm tup = TUPLE2(hp, CAR(kv), CDR(kv)); + hp += 3; + res = CONS(hp, tup, res); + hp += 2; + } + DESTROY_WSTACK(stack); + return res; +} + +void hashmap_iterator_init(ErtsWStack* s, Eterm node, int reverse) { + Eterm hdr = *hashmap_val(node); + Uint sz; + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + sz = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + break; + default: + erl_exit(1, "bad header"); + } + + WSTACK_PUSH3((*s), (UWord)THE_NON_VALUE, /* end marker */ + (UWord)(!reverse ? 0 : sz+1), + (UWord)node); +} + +Eterm* hashmap_iterator_next(ErtsWStack* s) { + Eterm node, *ptr, hdr; + Uint32 sz; + Uint idx; + + for (;;) { + ASSERT(!WSTACK_ISEMPTY((*s))); + node = (Eterm) WSTACK_POP((*s)); + if (is_non_value(node)) { + return NULL; + } + idx = (Uint) WSTACK_POP((*s)); + for (;;) { + ASSERT(is_boxed(node)); + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ptr++; + sz = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + ptr++; + case HAMT_SUBTAG_NODE_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(sz < 17); + break; + default: + erl_exit(1, "bad header"); + } + + idx++; + + if (idx <= sz) { + WSTACK_PUSH2((*s), (UWord)idx, (UWord)node); + + if (is_list(ptr[idx])) { + return list_val(ptr[idx]); + } + ASSERT(is_boxed(ptr[idx])); + node = ptr[idx]; + idx = 0; + } + else + break; /* and pop parent node */ + } + } +} + +Eterm* hashmap_iterator_prev(ErtsWStack* s) { + Eterm node, *ptr, hdr; + Uint32 sz; + Uint idx; + + for (;;) { + ASSERT(!WSTACK_ISEMPTY((*s))); + node = (Eterm) WSTACK_POP((*s)); + if (is_non_value(node)) { + return NULL; + } + idx = (Uint) WSTACK_POP((*s)); + for (;;) { + ASSERT(is_boxed(node)); + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ptr++; + sz = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + ptr++; + case HAMT_SUBTAG_NODE_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(sz < 17); + break; + default: + erl_exit(1, "bad header"); + } + + if (idx > sz) + idx = sz; + else + idx--; + + if (idx >= 1) { + WSTACK_PUSH2((*s), (UWord)idx, (UWord)node); + + if (is_list(ptr[idx])) { + return list_val(ptr[idx]); + } + ASSERT(is_boxed(ptr[idx])); + node = ptr[idx]; + idx = 17; + } + else + break; /* and pop parent node */ + } + } +} + +const Eterm * +#if HALFWORD_HEAP +erts_hashmap_get_rel(Uint32 hx, Eterm key, Eterm node, Eterm *map_base) +#else +erts_hashmap_get(Uint32 hx, Eterm key, Eterm node) +#endif { - Eterm *ks = map_get_keys(mp); - Eterm *vs = map_get_values(mp); - Uint sz = map_get_size(mp); + Eterm *ptr, hdr, *res; + Uint ix, lvl = 0; + Uint32 hval,bp; + DeclareTmpHeapNoproc(th,2); + UseTmpHeapNoproc(2); + + ASSERT(is_boxed(node)); + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + ASSERT(is_hashmap_header_head(hdr)); + ptr++; + + for (;;) { + hval = MAP_HEADER_VAL(hdr); + ix = hashmap_index(hx); + if (hval != 0xffff) { + bp = 1 << ix; + if (!(bp & hval)) { + /* not occupied */ + res = NULL; + break; + } + ix = hashmap_bitcount(hval & (bp - 1)); + } + node = ptr[ix+1]; + + if (is_list(node)) { /* LEAF NODE [K|V] */ + ptr = list_val(node); + + res = eq_rel(CAR(ptr), map_base, key, NULL) ? &(CDR(ptr)) : NULL; + break; + } + + hx = hashmap_shift_hash(th,hx,lvl,key); + + ASSERT(is_boxed(node)); + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + ASSERT(!is_hashmap_header_head(hdr)); + } + + UnUseTmpHeapNoproc(2); + return res; +} + +Eterm erts_hashmap_insert(Process *p, Uint32 hx, Eterm key, Eterm value, + Eterm map, int is_update) { + Uint size, upsz; + Eterm *hp, res = THE_NON_VALUE; + DECLARE_ESTACK(stack); + if (erts_hashmap_insert_down(hx, key, map, &size, &upsz, &stack, is_update)) { + hp = HAlloc(p, size); + res = erts_hashmap_insert_up(hp, key, value, &upsz, &stack); + } + + DESTROY_ESTACK(stack); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); + ERTS_HOLE_CHECK(p); + + return res; +} + + +int erts_hashmap_insert_down(Uint32 hx, Eterm key, Eterm node, Uint *sz, + Uint *update_size, ErtsEStack *sp, int is_update) { + Eterm *ptr; + Eterm hdr, ckey; + Eterm th[2]; + Uint32 ix, cix, bp, hval, chx; + Uint slot, lvl = 0, clvl; + Uint size = 0, n = 0; + + *update_size = 1; + + for (;;) { + switch(primary_tag(node)) { + case TAG_PRIMARY_LIST: /* LEAF NODE [K|V] */ + ptr = list_val(node); + ckey = CAR(ptr); + if (EQ(ckey, key)) { + *update_size = 0; + goto unroll; + } + if (is_update) { + return 0; + } + goto insert_subnodes; + case TAG_PRIMARY_BOXED: + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ix = hashmap_index(hx); + hx = hashmap_shift_hash(th,hx,lvl,key); + size += HAMT_HEAD_ARRAY_SZ; + ESTACK_PUSH2(*sp, ix, node); + node = ptr[ix+2]; + break; + case HAMT_SUBTAG_NODE_BITMAP: + hval = MAP_HEADER_VAL(hdr); + ix = hashmap_index(hx); + bp = 1 << ix; + slot = hashmap_bitcount(hval & (bp - 1)); + n = hashmap_bitcount(hval); + + ESTACK_PUSH4(*sp, n, bp, slot, node); + + /* occupied */ + if (bp & hval) { + hx = hashmap_shift_hash(th,hx,lvl,key); + node = ptr[slot+1]; + ASSERT(HAMT_NODE_BITMAP_SZ(n) <= 17); + size += HAMT_NODE_BITMAP_SZ(n); + break; + } + /* not occupied */ + if (is_update) { + return 0; + } + size += HAMT_NODE_BITMAP_SZ(n+1); + goto unroll; + case HAMT_SUBTAG_HEAD_BITMAP: + hval = MAP_HEADER_VAL(hdr); + ix = hashmap_index(hx); + bp = 1 << ix; + slot = hashmap_bitcount(hval & (bp - 1)); + n = hashmap_bitcount(hval); + + ESTACK_PUSH4(*sp, n, bp, slot, node); + + /* occupied */ + if (bp & hval) { + hx = hashmap_shift_hash(th,hx,lvl,key); + node = ptr[slot+2]; + ASSERT(HAMT_HEAD_BITMAP_SZ(n) <= 18); + size += HAMT_HEAD_BITMAP_SZ(n); + break; + } + /* not occupied */ + if (is_update) { + return 0; + } + size += HAMT_HEAD_BITMAP_SZ(n+1); + goto unroll; + default: + erl_exit(1, "bad header tag %ld\r\n", hdr & _HEADER_MAP_SUBTAG_MASK); + break; + } + break; + default: + erl_exit(1, "bad primary tag %p\r\n", node); + break; + } + } +insert_subnodes: + clvl = lvl; + chx = hashmap_restore_hash(th,clvl,ckey); + size += HAMT_NODE_BITMAP_SZ(2); + ix = hashmap_index(hx); + cix = hashmap_index(chx); + + while (cix == ix) { + ESTACK_PUSH4(*sp, 0, 1 << ix, 0, MAP_HEADER_HAMT_NODE_BITMAP(0)); + size += HAMT_NODE_BITMAP_SZ(1); + hx = hashmap_shift_hash(th,hx,lvl,key); + chx = hashmap_shift_hash(th,chx,clvl,ckey); + ix = hashmap_index(hx); + cix = hashmap_index(chx); + } + ESTACK_PUSH3(*sp, cix, ix, node); + +unroll: + *sz = size + /* res cons */ 2; + return 1; +} + +Eterm erts_hashmap_insert_up(Eterm *hp, Eterm key, Eterm value, + Uint *update_size, ErtsEStack *sp) { + Eterm node, *ptr, hdr; + Eterm res; + Eterm *nhp = NULL; + Uint32 ix, cix, bp, hval; + Uint slot, n; + /* Needed for halfword */ + DeclareTmpHeapNoproc(fake,1); + UseTmpHeapNoproc(1); + + res = CONS(hp, key, value); hp += 2; + + do { + node = ESTACK_POP(*sp); + switch(primary_tag(node)) { + case TAG_PRIMARY_LIST: + ix = (Uint32) ESTACK_POP(*sp); + cix = (Uint32) ESTACK_POP(*sp); + + nhp = hp; + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP((1 << ix) | (1 << cix)); + if (ix < cix) { + *hp++ = res; + *hp++ = node; + } else { + *hp++ = node; + *hp++ = res; + } + res = make_hashmap(nhp); + break; + case TAG_PRIMARY_HEADER: + /* subnodes, fake it */ + *fake = node; + node = make_boxed(fake); + case TAG_PRIMARY_BOXED: + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + slot = (Uint) ESTACK_POP(*sp); + nhp = hp; + n = HAMT_HEAD_ARRAY_SZ - 2; + *hp++ = MAP_HEADER_HAMT_HEAD_ARRAY; ptr++; + *hp++ = (*ptr++) + *update_size; + while(n--) { *hp++ = *ptr++; } + nhp[slot+2] = res; + res = make_hashmap(nhp); + break; + case HAMT_SUBTAG_NODE_BITMAP: + slot = (Uint) ESTACK_POP(*sp); + bp = (Uint32) ESTACK_POP(*sp); + n = (Uint32) ESTACK_POP(*sp); + hval = MAP_HEADER_VAL(hdr); + nhp = hp; + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hval | bp); ptr++; + + n -= slot; + while(slot--) { *hp++ = *ptr++; } + *hp++ = res; + if (hval & bp) { ptr++; n--; } + while(n--) { *hp++ = *ptr++; } + + res = make_hashmap(nhp); + break; + case HAMT_SUBTAG_HEAD_BITMAP: + slot = (Uint) ESTACK_POP(*sp); + bp = (Uint32) ESTACK_POP(*sp); + n = (Uint32) ESTACK_POP(*sp); + hval = MAP_HEADER_VAL(hdr); + nhp = hp; + *hp++ = MAP_HEADER_HAMT_HEAD_BITMAP(hval | bp); ptr++; + *hp++ = (*ptr++) + *update_size; + + n -= slot; + while(slot--) { *hp++ = *ptr++; } + *hp++ = res; + if (hval & bp) { ptr++; n--; } + while(n--) { *hp++ = *ptr++; } + + if ((hval | bp) == 0xffff) { + *nhp = MAP_HEADER_HAMT_HEAD_ARRAY; + } + res = make_hashmap(nhp); + break; + default: + erl_exit(1, "bad header tag %x\r\n", hdr & _HEADER_MAP_SUBTAG_MASK); + break; + } + break; + default: + erl_exit(1, "bad primary tag %x\r\n", primary_tag(node)); + break; + } + + } while(!ESTACK_ISEMPTY(*sp)); + + UnUseTmpHeapNoproc(1); + return res; +} + +static Eterm hashmap_keys(Process* p, Eterm node) { + DECLARE_WSTACK(stack); + hashmap_head_t* root; + Eterm *hp, *kv; + Eterm res = NIL; + + root = (hashmap_head_t*) boxed_val(node); + hp = HAlloc(p, root->size * 2); + hashmap_iterator_init(&stack, node, 0); + while ((kv=hashmap_iterator_next(&stack)) != NULL) { + res = CONS(hp, CAR(kv), res); + hp += 2; + } + DESTROY_WSTACK(stack); + return res; +} + +static Eterm hashmap_values(Process* p, Eterm node) { + DECLARE_WSTACK(stack); + hashmap_head_t* root; + Eterm *hp, *kv; + Eterm res = NIL; + + root = (hashmap_head_t*) boxed_val(node); + hp = HAlloc(p, root->size * 2); + hashmap_iterator_init(&stack, node, 0); + while ((kv=hashmap_iterator_next(&stack)) != NULL) { + res = CONS(hp, CDR(kv), res); + hp += 2; + } + DESTROY_WSTACK(stack); + return res; +} + +static Eterm hashmap_delete(Process *p, Uint32 hx, Eterm key, Eterm map) { + Eterm *hp = NULL, *nhp = NULL, *hp_end = NULL; + Eterm th[2]; + Eterm *ptr; + Eterm hdr, res = map, node = map; + Uint32 ix, bp, hval; + Uint slot, lvl = 0; + Uint size = 0, n = 0; + DECLARE_ESTACK(stack); + + for (;;) { + switch(primary_tag(node)) { + case TAG_PRIMARY_LIST: + if (EQ(CAR(list_val(node)), key)) { + goto unroll; + } + goto not_found; + case TAG_PRIMARY_BOXED: + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ix = hashmap_index(hx); + hx = hashmap_shift_hash(th,hx,lvl,key); + size += HAMT_HEAD_ARRAY_SZ; + ESTACK_PUSH2(stack, ix, node); + node = ptr[ix+2]; + break; + case HAMT_SUBTAG_NODE_BITMAP: + hval = MAP_HEADER_VAL(hdr); + ix = hashmap_index(hx); + bp = 1 << ix; + slot = hashmap_bitcount(hval & (bp - 1)); + n = hashmap_bitcount(hval); + + ESTACK_PUSH4(stack, n, bp, slot, node); + + /* occupied */ + if (bp & hval) { + hx = hashmap_shift_hash(th,hx,lvl,key); + node = ptr[slot+1]; + ASSERT(HAMT_NODE_BITMAP_SZ(n) <= 17); + size += HAMT_NODE_BITMAP_SZ(n); + break; + } + /* not occupied */ + goto not_found; + case HAMT_SUBTAG_HEAD_BITMAP: + hval = MAP_HEADER_VAL(hdr); + ix = hashmap_index(hx); + bp = 1 << ix; + slot = hashmap_bitcount(hval & (bp - 1)); + n = hashmap_bitcount(hval); + + ESTACK_PUSH4(stack, n, bp, slot, node); + + /* occupied */ + if (bp & hval) { + hx = hashmap_shift_hash(th,hx,lvl,key); + node = ptr[slot+2]; + ASSERT(HAMT_HEAD_BITMAP_SZ(n) <= 18); + size += HAMT_HEAD_BITMAP_SZ(n); + break; + } + /* not occupied */ + goto not_found; + default: + erl_exit(1, "bad header tag %ld\r\n", hdr & _HEADER_MAP_SUBTAG_MASK); + break; + } + break; + default: + erl_exit(1, "bad primary tag %p\r\n", node); + break; + } + } + +unroll: + /* the size is bounded and atleast one less than the previous size */ + size -= 1; + n = hashmap_size(map) - 1; + + if (n <= MAP_SMALL_MAP_LIMIT) { + DECLARE_WSTACK(wstack); + Eterm *kv, *ks, *vs; + flatmap_t *mp; + Eterm keys; + + DESTROY_ESTACK(stack); + + /* build flat structure */ + hp = HAlloc(p, 3 + 1 + (2 * n)); + keys = make_tuple(hp); + *hp++ = make_arityval(n); + ks = hp; + hp += n; + mp = (flatmap_t*)hp; + hp += MAP_HEADER_FLATMAP_SZ; + vs = hp; + + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = n; + mp->keys = keys; + + hashmap_iterator_init(&wstack, map, 0); + + while ((kv=hashmap_iterator_next(&wstack)) != NULL) { + if (EQ(CAR(kv),key)) + continue; + *ks++ = CAR(kv); + *vs++ = CDR(kv); + } + + /* it cannot have multiple keys */ + erts_validate_and_sort_flatmap(mp); + + DESTROY_WSTACK(wstack); + return make_flatmap(mp); + } + + hp = HAlloc(p, size); + hp_end = hp + size; + res = THE_NON_VALUE; + + do { + node = ESTACK_POP(stack); + + /* all nodes are things */ + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ix = (Uint) ESTACK_POP(stack); + nhp = hp; + if (res == THE_NON_VALUE) { + n = 16; + n -= ix; + *hp++ = MAP_HEADER_HAMT_HEAD_BITMAP(0xffff ^ (1 << ix)); ptr++; + *hp++ = (*ptr++) - 1; + while(ix--) { *hp++ = *ptr++; } + ptr++; n--; + while(n--) { *hp++ = *ptr++; } + res = make_hashmap(nhp); + } else { + n = 16; + *hp++ = MAP_HEADER_HAMT_HEAD_ARRAY; ptr++; + *hp++ = (*ptr++) - 1; + while(n--) { *hp++ = *ptr++; } + nhp[ix+2] = res; + res = make_hashmap(nhp); + } + break; + case HAMT_SUBTAG_NODE_BITMAP: + slot = (Uint) ESTACK_POP(stack); + bp = (Uint32) ESTACK_POP(stack); + n = (Uint32) ESTACK_POP(stack); + nhp = hp; + + /* bitmap change matrix + * res | none leaf bitmap + * ---------------------------- + * n=1 | remove remove keep + * n=2 | other keep keep + * n>2 | shrink keep keep + * + * other: (remember, n is 2) + * shrink if the other bitmap value is a bitmap node + * remove if the other bitmap value is a leaf + * + * remove: + * this bitmap node is removed, res is moved up in tree (could be none) + * this is a special case of shrink + * + * keep: + * the current path index is still used down in the tree, need to keep it + * copy as usual with the updated res + * + * shrink: + * the current path index is no longer used down in the tree, remove it (shrink) + */ + if (res == THE_NON_VALUE) { + if (n == 1) { + break; + } else if (n == 2) { + if (slot == 0) { + ix = 2; /* off by one 'cause hdr */ + } else { + ix = 1; /* off by one 'cause hdr */ + } + if (primary_tag(ptr[ix]) == TAG_PRIMARY_LIST) { + res = ptr[ix]; + } else { + hval = MAP_HEADER_VAL(hdr); + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hval ^ bp); + *hp++ = ptr[ix]; + res = make_hashmap(nhp); + } + } else { + /* n > 2 */ + hval = MAP_HEADER_VAL(hdr); + *hp++ = MAP_HEADER_HAMT_NODE_BITMAP(hval ^ bp); ptr++; + n -= slot; + while(slot--) { *hp++ = *ptr++; } + ptr++; n--; + while(n--) { *hp++ = *ptr++; } + res = make_hashmap(nhp); + } + } else if (primary_tag(res) == TAG_PRIMARY_LIST && n == 1) { + break; + } else { + /* res is bitmap or leaf && n > 1, keep */ + n -= slot; + *hp++ = *ptr++; + while(slot--) { *hp++ = *ptr++; } + *hp++ = res; + ptr++; n--; + while(n--) { *hp++ = *ptr++; } + res = make_hashmap(nhp); + } + break; + case HAMT_SUBTAG_HEAD_BITMAP: + slot = (Uint) ESTACK_POP(stack); + bp = (Uint32) ESTACK_POP(stack); + n = (Uint32) ESTACK_POP(stack); + nhp = hp; + + if (res != THE_NON_VALUE) { + *hp++ = *ptr++; + *hp++ = (*ptr++) - 1; + n -= slot; + while(slot--) { *hp++ = *ptr++; } + *hp++ = res; + ptr++; n--; + while(n--) { *hp++ = *ptr++; } + } else { + hval = MAP_HEADER_VAL(hdr); + *hp++ = MAP_HEADER_HAMT_HEAD_BITMAP(hval ^ bp); ptr++; + *hp++ = (*ptr++) - 1; + n -= slot; + while(slot--) { *hp++ = *ptr++; } + ptr++; n--; + while(n--) { *hp++ = *ptr++; } + } + res = make_hashmap(nhp); + break; + default: + erl_exit(1, "bad header tag %x\r\n", hdr & _HEADER_MAP_SUBTAG_MASK); + break; + } + } while(!ESTACK_ISEMPTY(stack)); + HRelease(p, hp_end, hp); +not_found: + DESTROY_ESTACK(stack); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(p); + ERTS_HOLE_CHECK(p); + return res; +} + + +int erts_validate_and_sort_flatmap(flatmap_t* mp) +{ + Eterm *ks = flatmap_get_keys(mp); + Eterm *vs = flatmap_get_values(mp); + Uint sz = flatmap_get_size(mp); Uint ix,jx; Eterm tmp; - int c; + Sint c; /* sort */ @@ -811,6 +2443,55 @@ int erts_validate_and_sort_map(map_t* mp) return 1; } +/* Really rough estimate of sqrt(x) + * Guaranteed not to be less than sqrt(x) + */ +static int int_sqrt_ceiling(Uint x) +{ + int n; + + if (x <= 2) + return x; + + n = erts_fit_in_bits_uint(x-1); + if (n & 1) { + /* Calc: sqrt(2^n) = 2^(n/2) * sqrt(2) ~= 2^(n/2) * 3 / 2 */ + return (1 << (n/2 - 1)) * 3; + } + else { + /* Calc: sqrt(2^n) = 2^(n/2) */ + return 1 << (n / 2); + } +} + +Uint hashmap_over_estimated_heap_size(Uint k) +{ + /* k is nr of key-value pairs. + N(k) is expected nr of nodes in hamt. + + Observation: + For uniformly distributed hash values, average of N varies between + 0.3*k and 0.4*k (with a beautiful sine curve) + and standard deviation of N is about sqrt(k)/3. + + Assuming normal probability distribution, we overestimate nr of nodes + by 15 std.devs above the average, which gives a probability for overrun + less than 1.0e-49 (same magnitude as a git SHA1 collision). + */ + Uint max_nodes = 2*k/5 + (15/3)*int_sqrt_ceiling(k); + return (k*2 + /* leaf cons cells */ + k + /* leaf list terms */ + max_nodes*2); /* headers + parent boxed terms */ +} + + +BIF_RETTYPE erts_debug_map_info_1(BIF_ALIST_1) { + if (is_hashmap(BIF_ARG_1)) { + BIF_RET(hashmap_info(BIF_P,BIF_ARG_1)); + } + BIF_ERROR(BIF_P, BADARG); +} + /* * erts_internal:map_to_tuple_keys/1 * @@ -818,9 +2499,220 @@ int erts_validate_and_sort_map(map_t* mp) */ BIF_RETTYPE erts_internal_map_to_tuple_keys_1(BIF_ALIST_1) { - if (is_map(BIF_ARG_1)) { - map_t *mp = (map_t*)map_val(BIF_ARG_1); + if (is_flatmap(BIF_ARG_1)) { + flatmap_t *mp = (flatmap_t*)flatmap_val(BIF_ARG_1); BIF_RET(mp->keys); } BIF_ERROR(BIF_P, BADARG); } + +/* + * erts_internal:map_type/1 + * + * Used in erts_debug:size/1 + */ + +BIF_RETTYPE erts_internal_map_type_1(BIF_ALIST_1) { + DECL_AM(hashmap); + DECL_AM(hashmap_node); + DECL_AM(flatmap); + if (is_flatmap(BIF_ARG_1)) { + BIF_RET(AM_flatmap); + } else if (is_hashmap(BIF_ARG_1)) { + Eterm hdr = *(boxed_val(BIF_ARG_1)); + ASSERT(is_header(hdr)); + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + case HAMT_SUBTAG_HEAD_BITMAP: + BIF_RET(AM_hashmap); + case HAMT_SUBTAG_NODE_BITMAP: + BIF_RET(AM_hashmap_node); + default: + erl_exit(1, "bad header"); + } + } + BIF_ERROR(BIF_P, BADARG); +} + +/* + * erts_internal:map_hashmap_children/1 + * + * Used in erts_debug:size/1 + */ + +BIF_RETTYPE erts_internal_map_hashmap_children_1(BIF_ALIST_1) { + if (is_hashmap(BIF_ARG_1)) { + Eterm node = BIF_ARG_1; + Eterm *ptr, hdr, *hp, res = NIL; + Uint sz = 0; + ptr = boxed_val(node); + hdr = *ptr; + + ASSERT(is_header(hdr)); + + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_NODE_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ptr += 1; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ptr += 2; + break; + case HAMT_SUBTAG_HEAD_ARRAY: + sz = 16; + ptr += 2; + break; + default: + erl_exit(1, "bad header\r\n"); + break; + } + ASSERT(sz < 17); + hp = HAlloc(BIF_P, 2*sz); + while(sz--) { res = CONS(hp, *ptr++, res); hp += 2; } + BIF_RET(res); + } + BIF_ERROR(BIF_P, BADARG); +} + + +static Eterm hashmap_info(Process *p, Eterm node) { + Eterm *hp; + Eterm res = NIL, info = NIL; + Eterm *ptr, tup, hdr; + Uint sz; + DECL_AM(depth); + DECL_AM(leafs); + DECL_AM(bitmaps); + DECL_AM(arrays); + Uint nleaf=0, nbitmap=0, narray=0; + Uint bitmap_usage[16], leaf_usage[16]; + Uint lvl = 0, clvl; + DECLARE_ESTACK(stack); + + for (sz = 0; sz < 16; sz++) { + bitmap_usage[sz] = 0; + leaf_usage[sz] = 0; + } + + ptr = boxed_val(node); + ESTACK_PUSH(stack, 0); + ESTACK_PUSH(stack, node); + do { + node = ESTACK_POP(stack); + clvl = ESTACK_POP(stack); + if (lvl < clvl) + lvl = clvl; + switch(primary_tag(node)) { + case TAG_PRIMARY_LIST: + nleaf++; + leaf_usage[clvl] += 1; + break; + case TAG_PRIMARY_BOXED: + ptr = boxed_val(node); + hdr = *ptr; + ASSERT(is_header(hdr)); + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_NODE_BITMAP: + nbitmap++; + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(sz < 17); + bitmap_usage[sz-1] += 1; + while(sz--) { + ESTACK_PUSH(stack, clvl + 1); + ESTACK_PUSH(stack, ptr[sz+1]); + } + break; + case HAMT_SUBTAG_HEAD_BITMAP: + nbitmap++; + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + bitmap_usage[sz-1] += 1; + while(sz--) { + ESTACK_PUSH(stack, clvl + 1); + ESTACK_PUSH(stack, ptr[sz+2]); + } + break; + case HAMT_SUBTAG_HEAD_ARRAY: + narray++; + sz = 16; + while(sz--) { + ESTACK_PUSH(stack, clvl + 1); + ESTACK_PUSH(stack, ptr[sz+2]); + } + break; + default: + erl_exit(1, "bad header\r\n"); + break; + } + } + } while(!ESTACK_ISEMPTY(stack)); + + + /* size */ + sz = 0; + hashmap_bld_tuple_uint(NULL,&sz,16,leaf_usage); + hashmap_bld_tuple_uint(NULL,&sz,16,bitmap_usage); + + /* alloc */ + hp = HAlloc(p, 2+3 + 3*(2+4) + sz); + + info = hashmap_bld_tuple_uint(&hp,NULL,16,leaf_usage); + tup = TUPLE3(hp, AM_leafs, make_small(nleaf),info); hp += 4; + res = CONS(hp, tup, res); hp += 2; + + info = hashmap_bld_tuple_uint(&hp,NULL,16,bitmap_usage); + tup = TUPLE3(hp, AM_bitmaps, make_small(nbitmap), info); hp += 4; + res = CONS(hp, tup, res); hp += 2; + + tup = TUPLE3(hp, AM_arrays, make_small(narray),NIL); hp += 4; + res = CONS(hp, tup, res); hp += 2; + + tup = TUPLE2(hp, AM_depth, make_small(lvl)); hp += 3; + res = CONS(hp, tup, res); hp += 2; + + DESTROY_ESTACK(stack); + ERTS_HOLE_CHECK(p); + return res; +} + +static Eterm hashmap_bld_tuple_uint(Uint **hpp, Uint *szp, Uint n, Uint nums[]) { + Eterm res = THE_NON_VALUE; + Eterm *ts = (Eterm *)erts_alloc(ERTS_ALC_T_TMP, n * sizeof(Eterm)); + Uint i; + + for (i = 0; i < n; i++) { + ts[i] = erts_bld_uint(hpp, szp, nums[i]); + } + res = erts_bld_tuplev(hpp, szp, n, ts); + erts_free(ERTS_ALC_T_TMP, (void *) ts); + return res; +} + + +/* implementation of builtin emulations */ + +#if !ERTS_AT_LEAST_GCC_VSN__(3, 4, 0) +/* Count leading zeros emulation */ +Uint32 hashmap_clz(Uint32 x) { + Uint32 y; + int n = 32; + y = x >>16; if (y != 0) {n = n -16; x = y;} + y = x >> 8; if (y != 0) {n = n - 8; x = y;} + y = x >> 4; if (y != 0) {n = n - 4; x = y;} + y = x >> 2; if (y != 0) {n = n - 2; x = y;} + y = x >> 1; if (y != 0) return n - 2; + return n - x; +} + +const Uint32 SK5 = 0x55555555, SK3 = 0x33333333; +const Uint32 SKF0 = 0xF0F0F0F, SKFF = 0xFF00FF; + +/* CTPOP emulation */ +Uint32 hashmap_bitcount(Uint32 x) { + x -= ((x >> 1 ) & SK5); + x = (x & SK3 ) + ((x >> 2 ) & SK3 ); + x = (x & SKF0) + ((x >> 4 ) & SKF0); + x += x >> 8; + return (x + (x >> 16)) & 0x3F; +} +#endif diff --git a/erts/emulator/beam/erl_map.h b/erts/emulator/beam/erl_map.h index 2e02ca4677..2cc6768bfc 100644 --- a/erts/emulator/beam/erl_map.h +++ b/erts/emulator/beam/erl_map.h @@ -22,13 +22,23 @@ #define __ERL_MAP_H__ #include "sys.h" + +/* instrinsic wrappers */ +#if ERTS_AT_LEAST_GCC_VSN__(3, 4, 0) +#define hashmap_clz(x) ((Uint32) __builtin_clz((unsigned int)(x))) +#define hashmap_bitcount(x) ((Uint32) __builtin_popcount((unsigned int) (x))) +#else +Uint32 hashmap_clz(Uint32 x); +Uint32 hashmap_bitcount(Uint32 x); +#endif + /* MAP */ -typedef struct map_s { +typedef struct flatmap_s { Eterm thing_word; Uint size; Eterm keys; /* tuple */ -} map_t; +} flatmap_t; /* map node * * ----------- @@ -42,39 +52,144 @@ typedef struct map_s { * ----------- */ +/* the head-node is a bitmap or array with an untagged size */ + + +#define hashmap_size(x) (((hashmap_head_t*) hashmap_val(x))->size) +#define hashmap_size_rel(RTERM, BASE) hashmap_size(rterm2wterm(RTERM, BASE)) +#define hashmap_make_hash(Key) make_internal_hash(Key) + +#define hashmap_restore_hash(Heap,Lvl,Key) \ + (((Lvl) < 8) ? hashmap_make_hash(Key) >> (4*(Lvl)) : hashmap_make_hash(CONS(Heap, make_small((Lvl)>>3), (Key))) >> (4*((Lvl) & 7))) +#define hashmap_shift_hash(Heap,Hx,Lvl,Key) \ + (((++(Lvl)) & 7) ? (Hx) >> 4 : hashmap_make_hash(CONS(Heap, make_small((Lvl)>>3), Key))) /* erl_term.h stuff */ -#define make_map(x) make_boxed((Eterm*)(x)) -#define make_map_rel(x, BASE) make_boxed_rel((Eterm*)(x),(BASE)) -#define is_map(x) (is_boxed((x)) && is_map_header(*boxed_val((x)))) -#define is_map_rel(RTERM,BASE) is_map(rterm2wterm(RTERM,BASE)) -#define is_not_map(x) (!is_map((x))) -#define is_map_header(x) (((x) & (_TAG_HEADER_MASK)) == _TAG_HEADER_MAP) -#define header_is_map(x) ((((x) & (_HEADER_SUBTAG_MASK)) == MAP_SUBTAG)) -#define map_val(x) (_unchecked_boxed_val((x))) -#define map_val_rel(RTERM, BASE) map_val(rterm2wterm(RTERM, BASE)) - -#define map_get_values(x) (((Eterm *)(x)) + 3) -#define map_get_keys(x) (((Eterm *)tuple_val(((map_t *)(x))->keys)) + 1) -#define map_get_size(x) (((map_t*)(x))->size) - -#define MAP_HEADER _make_header(1,_TAG_HEADER_MAP) -#define MAP_HEADER_SIZE (sizeof(map_t) / sizeof(Eterm)) - -Eterm erts_maps_put(Process *p, Eterm key, Eterm value, Eterm map); -int erts_maps_update(Process *p, Eterm key, Eterm value, Eterm map, Eterm *res); -int erts_maps_remove(Process *p, Eterm key, Eterm map, Eterm *res); -int erts_validate_and_sort_map(map_t* map); +#define flatmap_get_values(x) (((Eterm *)(x)) + 3) +#define flatmap_get_keys(x) (((Eterm *)tuple_val(((flatmap_t *)(x))->keys)) + 1) +#define flatmap_get_size(x) (((flatmap_t*)(x))->size) + +#ifdef DEBUG +#define MAP_SMALL_MAP_LIMIT (3) +#else +#define MAP_SMALL_MAP_LIMIT (32) +#endif + +struct ErtsWStack_; +struct ErtsEStack_; + +Eterm erts_maps_put(Process *p, Eterm key, Eterm value, Eterm map); +int erts_maps_update(Process *p, Eterm key, Eterm value, Eterm map, Eterm *res); +int erts_maps_remove(Process *p, Eterm key, Eterm map, Eterm *res); + +Eterm erts_hashmap_insert(Process *p, Uint32 hx, Eterm key, Eterm value, + Eterm node, int is_update); +int erts_hashmap_insert_down(Uint32 hx, Eterm key, Eterm node, Uint *sz, + Uint *upsz, struct ErtsEStack_ *sp, int is_update); +Eterm erts_hashmap_insert_up(Eterm *hp, Eterm key, Eterm value, + Uint *upsz, struct ErtsEStack_ *sp); + +int erts_validate_and_sort_flatmap(flatmap_t* map); +Uint hashmap_over_estimated_heap_size(Uint n); +void hashmap_iterator_init(struct ErtsWStack_* s, Eterm node, int reverse); +Eterm* hashmap_iterator_next(struct ErtsWStack_* s); +Eterm* hashmap_iterator_prev(struct ErtsWStack_* s); +int hashmap_key_hash_cmp(Eterm* ap, Eterm* bp); +Eterm erts_hashmap_from_array(ErtsHeapFactory*, Eterm *leafs, Uint n, int reject_dupkeys); + +#define erts_hashmap_from_ks_and_vs(P, KS, VS, N) \ + erts_hashmap_from_ks_and_vs_extra((P), (KS), (VS), (N), THE_NON_VALUE, THE_NON_VALUE); + +Eterm erts_hashmap_from_ks_and_vs_extra(Process *p, Eterm *ks, Eterm *vs, Uint n, + Eterm k, Eterm v); -#if HALFWORD_HEAP const Eterm * +#if HALFWORD_HEAP erts_maps_get_rel(Eterm key, Eterm map, Eterm *map_base); # define erts_maps_get(A, B) erts_maps_get_rel(A, B, NULL) #else -const Eterm * erts_maps_get(Eterm key, Eterm map); # define erts_maps_get_rel(A, B, B_BASE) erts_maps_get(A, B) #endif +const Eterm * +#if HALFWORD_HEAP +erts_hashmap_get_rel(Uint32 hx, Eterm key, Eterm node, Eterm *map_base); +# define erts_hashmap_get(Hx, K, M) erts_hashmap_get_rel(Hx, K, M, NULL) +#else +erts_hashmap_get(Uint32 hx, Eterm key, Eterm map); +# define erts_hashmap_get_rel(Hx, K, M, M_BASE) erts_hashmap_get(Hx, K, M) +#endif + +/* hamt nodes v2.0 + * + * node :: leaf | array | bitmap + * head + */ +typedef struct hashmap_head_s { + Eterm thing_word; + Uint size; + Eterm items[1]; +} hashmap_head_t; + +/* thing_word tagscheme + * Need two bits for map subtags + * + * Original HEADER representation: + * + * aaaaaaaaaaaaaaaa aaaaaaaaaatttt00 arity:26, tag:4 + * + * For maps we have: + * + * vvvvvvvvvvvvvvvv aaaaaaaamm111100 val:16, arity:8, mtype:2 + * + * unsure about trailing zeros + * + * map-tag: + * 00 - flat map tag (non-hamt) -> val:16 = #items + * 01 - map-node bitmap tag -> val:16 = bitmap + * 10 - map-head (array-node) -> val:16 = 0xffff + * 11 - map-head (bitmap-node) -> val:16 = bitmap + */ + +/* erl_map.h stuff */ + +#define is_hashmap_header_head(x) ((MAP_HEADER_TYPE(x) & (0x2))) + +#define MAKE_MAP_HEADER(Type,Arity,Val) \ + (_make_header(((((Uint16)(Val)) << MAP_HEADER_ARITY_SZ) | (Arity)) << MAP_HEADER_TAG_SZ | (Type) , _TAG_HEADER_MAP)) + +#define MAP_HEADER_FLATMAP \ + MAKE_MAP_HEADER(MAP_HEADER_TAG_FLATMAP_HEAD,0x1,0x0) + +#define MAP_HEADER_HAMT_HEAD_ARRAY \ + MAKE_MAP_HEADER(MAP_HEADER_TAG_HAMT_HEAD_ARRAY,0x1,0xffff) + +#define MAP_HEADER_HAMT_HEAD_BITMAP(Bmp) \ + MAKE_MAP_HEADER(MAP_HEADER_TAG_HAMT_HEAD_BITMAP,0x1,Bmp) + +#define MAP_HEADER_HAMT_NODE_BITMAP(Bmp) \ + MAKE_MAP_HEADER(MAP_HEADER_TAG_HAMT_NODE_BITMAP,0x0,Bmp) + +#define MAP_HEADER_FLATMAP_SZ (sizeof(flatmap_t) / sizeof(Eterm)) + +#define HAMT_NODE_ARRAY_SZ (17) +#define HAMT_HEAD_ARRAY_SZ (18) +#define HAMT_NODE_BITMAP_SZ(n) (1 + n) +#define HAMT_HEAD_BITMAP_SZ(n) (2 + n) + +/* 2 bits maps tag + 4 bits subtag + 2 ignore bits */ +#define _HEADER_MAP_SUBTAG_MASK (0xfc) +/* 1 bit map tag + 1 ignore bit + 4 bits subtag + 2 ignore bits */ +#define _HEADER_MAP_HASHMAP_HEAD_MASK (0xbc) + +#define HAMT_SUBTAG_NODE_BITMAP ((MAP_HEADER_TAG_HAMT_NODE_BITMAP << _HEADER_ARITY_OFFS) | MAP_SUBTAG) +#define HAMT_SUBTAG_HEAD_ARRAY ((MAP_HEADER_TAG_HAMT_HEAD_ARRAY << _HEADER_ARITY_OFFS) | MAP_SUBTAG) +#define HAMT_SUBTAG_HEAD_BITMAP ((MAP_HEADER_TAG_HAMT_HEAD_BITMAP << _HEADER_ARITY_OFFS) | MAP_SUBTAG) +#define HAMT_SUBTAG_HEAD_FLATMAP ((MAP_HEADER_TAG_FLATMAP_HEAD << _HEADER_ARITY_OFFS) | MAP_SUBTAG) + +#define hashmap_index(hash) (((Uint32)hash) & 0xf) + + #endif diff --git a/erts/emulator/beam/erl_message.c b/erts/emulator/beam/erl_message.c index 8870fac7d9..22cbae10d1 100644 --- a/erts/emulator/beam/erl_message.c +++ b/erts/emulator/beam/erl_message.c @@ -994,7 +994,7 @@ erts_send_message(Process* sender, #endif ); BM_SWAP_TIMER(send,system); - } else if (sender == receiver) { + } else if (sender == receiver && !(sender->flags & F_OFF_HEAP_MSGS)) { /* Drop message if receiver has a pending exit ... */ #ifdef ERTS_SMP ErtsProcLocks need_locks = (~(*receiver_locks) @@ -1146,3 +1146,15 @@ erts_deliver_exit_message(Eterm from, Process *to, ErtsProcLocks *to_locksp, } } +Eterm* erts_produce_heap(ErtsHeapFactory* factory, Uint need, Uint xtra) +{ + Eterm* res; + if (factory->p) { + res = HAllocX(factory->p, need, xtra); + } else { + res = factory->hp; + factory->hp += need; + } + return res; +} + diff --git a/erts/emulator/beam/erl_message.h b/erts/emulator/beam/erl_message.h index 0f3bb8d281..8713941769 100644 --- a/erts/emulator/beam/erl_message.h +++ b/erts/emulator/beam/erl_message.h @@ -68,6 +68,21 @@ struct erl_heap_fragment { Eterm mem[1]; /* Data */ }; +typedef struct { + Process* p; + Eterm* hp; +} ErtsHeapFactory; + +Eterm* erts_produce_heap(ErtsHeapFactory*, Uint need, Uint xtra); +#ifdef CHECK_FOR_HOLES +# define ERTS_FACTORY_HOLE_CHECK(f) do { \ + if ((f)->p) erts_check_for_holes((f)->p); \ + } while (0) +#else +# define ERTS_FACTORY_HOLE_CHECK(p) +#endif + + typedef struct erl_mesg { struct erl_mesg* next; /* Next message */ union { @@ -198,15 +213,25 @@ do { \ if ((M)->data.attached) { \ Uint need__ = erts_msg_attached_data_size((M)); \ if ((ST) - (HT) >= need__) { \ - Uint *htop__ = (HT); \ + Uint *htop__; \ + move__attached__msg__data____: \ + htop__ = (HT); \ erts_move_msg_attached_data_to_heap(&htop__, &MSO((P)), (M));\ ASSERT(htop__ - (HT) <= need__); \ (HT) = htop__; \ } \ else { \ + int off_heap_msgs__ = (int) (P)->flags & F_OFF_HEAP_MSGS; \ + if (!off_heap_msgs__) \ + need__ = 0; \ { SWPO ; } \ - (FC) -= erts_garbage_collect((P), 0, NULL, 0); \ + (FC) -= erts_garbage_collect((P), need__, NULL, 0); \ { SWPI ; } \ + if (off_heap_msgs__) { \ + ASSERT((M)->data.attached); \ + ASSERT((ST) - (HT) >= need__); \ + goto move__attached__msg__data____; \ + } \ } \ ASSERT(!(M)->data.attached); \ } \ diff --git a/erts/emulator/beam/erl_monitors.h b/erts/emulator/beam/erl_monitors.h index fb11dbbd22..9972890db7 100644 --- a/erts/emulator/beam/erl_monitors.h +++ b/erts/emulator/beam/erl_monitors.h @@ -82,6 +82,7 @@ /* Type tags for monitors */ #define MON_ORIGIN 1 #define MON_TARGET 3 +#define MON_TIME_OFFSET 7 /* Type tags for links */ #define LINK_PID 1 /* ...Or port */ @@ -103,7 +104,7 @@ typedef struct erts_monitor_or_link { typedef struct erts_monitor { struct erts_monitor *left, *right; Sint16 balance; - Uint16 type; /* MON_ORIGIN | MON_TARGET */ + Uint16 type; /* MON_ORIGIN | MON_TARGET | MON_TIME_OFFSET */ Eterm ref; Eterm pid; /* In case of distributed named monitor, this is the nodename atom in MON_ORIGIN process, otherwise a pid or diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index 198acfd128..660f446a52 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -36,6 +36,7 @@ #include "erl_thr_progress.h" #include "dtrace-wrapper.h" #include "erl_process.h" +#include "erl_bif_unique.h" #if defined(USE_DYNAMIC_TRACE) && (defined(USE_DTRACE) || defined(USE_SYSTEMTAP)) #define HAVE_USE_DTRACE 1 #endif @@ -1910,29 +1911,33 @@ int enif_is_map(ErlNifEnv* env, ERL_NIF_TERM term) int enif_get_map_size(ErlNifEnv* env, ERL_NIF_TERM term, size_t *size) { - if (is_map(term)) { - map_t *mp; - mp = (map_t*)map_val(term); - *size = map_get_size(mp); + if (is_flatmap(term)) { + flatmap_t *mp; + mp = (flatmap_t*)flatmap_val(term); + *size = flatmap_get_size(mp); return 1; } + else if (is_hashmap(term)) { + *size = hashmap_size(term); + return 1; + } return 0; } ERL_NIF_TERM enif_make_new_map(ErlNifEnv* env) { - Eterm* hp = alloc_heap(env,MAP_HEADER_SIZE+1); + Eterm* hp = alloc_heap(env,MAP_HEADER_FLATMAP_SZ+1); Eterm tup; - map_t *mp; + flatmap_t *mp; tup = make_tuple(hp); *hp++ = make_arityval(0); - mp = (map_t*)hp; - mp->thing_word = MAP_HEADER; + mp = (flatmap_t*)hp; + mp->thing_word = MAP_HEADER_FLATMAP; mp->size = 0; mp->keys = tup; - return make_map(mp); + return make_flatmap(mp); } int enif_make_map_put(ErlNifEnv* env, @@ -1941,7 +1946,7 @@ int enif_make_map_put(ErlNifEnv* env, Eterm value, Eterm *map_out) { - if (is_not_map(map_in)) { + if (!is_map(map_in)) { return 0; } flush_env(env); @@ -1956,7 +1961,7 @@ int enif_get_map_value(ErlNifEnv* env, Eterm *value) { const Eterm *ret; - if (is_not_map(map)) { + if (!is_map(map)) { return 0; } ret = erts_maps_get(key, map); @@ -1974,7 +1979,7 @@ int enif_make_map_update(ErlNifEnv* env, Eterm *map_out) { int res; - if (is_not_map(map_in)) { + if (!is_map(map_in)) { return 0; } @@ -1990,7 +1995,7 @@ int enif_make_map_remove(ErlNifEnv* env, Eterm *map_out) { int res; - if (is_not_map(map_in)) { + if (!is_map(map_in)) { return 0; } flush_env(env); @@ -2004,13 +2009,13 @@ int enif_map_iterator_create(ErlNifEnv *env, ErlNifMapIterator *iter, ErlNifMapIteratorEntry entry) { - if (is_map(map)) { - map_t *mp = (map_t*)map_val(map); + if (is_flatmap(map)) { + flatmap_t *mp = (flatmap_t*)flatmap_val(map); size_t offset; switch (entry) { case ERL_NIF_MAP_ITERATOR_HEAD: offset = 0; break; - case ERL_NIF_MAP_ITERATOR_TAIL: offset = map_get_size(mp) - 1; break; + case ERL_NIF_MAP_ITERATOR_TAIL: offset = flatmap_get_size(mp) - 1; break; default: goto error; } @@ -2019,14 +2024,37 @@ int enif_map_iterator_create(ErlNifEnv *env, */ iter->map = map; - iter->ks = ((Eterm *)map_get_keys(mp)) + offset; - iter->vs = ((Eterm *)map_get_values(mp)) + offset; - iter->t_limit = map_get_size(mp) + 1; + iter->u.flat.ks = ((Eterm *)flatmap_get_keys(mp)) + offset; + iter->u.flat.vs = ((Eterm *)flatmap_get_values(mp)) + offset; + iter->size = flatmap_get_size(mp); iter->idx = offset + 1; return 1; } - + else if (is_hashmap(map)) { + iter->map = map; + iter->size = hashmap_size(map); + iter->u.hash.wstack = erts_alloc(ERTS_ALC_T_NIF, sizeof(ErtsDynamicWStack)); + WSTACK_INIT(iter->u.hash.wstack, ERTS_ALC_T_NIF); + + switch (entry) { + case ERL_NIF_MAP_ITERATOR_HEAD: + iter->idx = 1; + hashmap_iterator_init(&iter->u.hash.wstack->ws, map, 0); + iter->u.hash.kv = hashmap_iterator_next(&iter->u.hash.wstack->ws); + break; + case ERL_NIF_MAP_ITERATOR_TAIL: + iter->idx = hashmap_size(map); + hashmap_iterator_init(&iter->u.hash.wstack->ws, map, 1); + iter->u.hash.kv = hashmap_iterator_prev(&iter->u.hash.wstack->ws); + break; + default: + goto error; + } + ASSERT(!!iter->u.hash.kv == (iter->idx >= 1 && + iter->idx <= iter->size)); + return 1; + } error: #ifdef DEBUG iter->map = THE_NON_VALUE; @@ -2036,48 +2064,97 @@ error: void enif_map_iterator_destroy(ErlNifEnv *env, ErlNifMapIterator *iter) { - /* not used */ + if (is_hashmap(iter->map)) { + WSTACK_DESTROY(iter->u.hash.wstack->ws); + erts_free(ERTS_ALC_T_NIF, iter->u.hash.wstack); + } + else + ASSERT(is_flatmap(iter->map)); + #ifdef DEBUG iter->map = THE_NON_VALUE; #endif - } int enif_map_iterator_is_tail(ErlNifEnv *env, ErlNifMapIterator *iter) { - ASSERT(iter && is_map(iter->map)); - ASSERT(iter->idx >= 0 && (iter->idx <= map_get_size(map_val(iter->map)) + 1)); - return (iter->t_limit == 1 || iter->idx == iter->t_limit); + ASSERT(iter); + if (is_flatmap(iter->map)) { + ASSERT(iter->idx >= 0); + ASSERT(iter->idx <= flatmap_get_size(flatmap_val(iter->map)) + 1); + return (iter->size == 0 || iter->idx > iter->size); + } + else { + ASSERT(is_hashmap(iter->map)); + return iter->idx > iter->size; + } } int enif_map_iterator_is_head(ErlNifEnv *env, ErlNifMapIterator *iter) { - ASSERT(iter && is_map(iter->map)); - ASSERT(iter->idx >= 0 && (iter->idx <= map_get_size(map_val(iter->map)) + 1)); - return (iter->t_limit == 1 || iter->idx == 0); + ASSERT(iter); + if (is_flatmap(iter->map)) { + ASSERT(iter->idx >= 0); + ASSERT(iter->idx <= flatmap_get_size(flatmap_val(iter->map)) + 1); + return (iter->size == 0 || iter->idx == 0); + } + else { + ASSERT(is_hashmap(iter->map)); + return iter->idx == 0; + } } int enif_map_iterator_next(ErlNifEnv *env, ErlNifMapIterator *iter) { - ASSERT(iter && is_map(iter->map)); - if (iter->idx < iter->t_limit) { - iter->idx++; - iter->ks++; - iter->vs++; + ASSERT(iter); + if (is_flatmap(iter->map)) { + if (iter->idx <= iter->size) { + iter->idx++; + iter->u.flat.ks++; + iter->u.flat.vs++; + } + return (iter->idx <= iter->size); + } + else { + ASSERT(is_hashmap(iter->map)); + + if (iter->idx <= hashmap_size(iter->map)) { + if (iter->idx < 1) { + hashmap_iterator_init(&iter->u.hash.wstack->ws, iter->map, 0); + } + iter->u.hash.kv = hashmap_iterator_next(&iter->u.hash.wstack->ws); + iter->idx++; + ASSERT(!!iter->u.hash.kv == (iter->idx <= iter->size)); + } + return iter->idx <= iter->size; } - return (iter->idx != iter->t_limit); } int enif_map_iterator_prev(ErlNifEnv *env, ErlNifMapIterator *iter) { - ASSERT(iter && is_map(iter->map)); - if (iter->idx > 0) { - iter->idx--; - iter->ks--; - iter->vs--; + ASSERT(iter); + if (is_flatmap(iter->map)) { + if (iter->idx > 0) { + iter->idx--; + iter->u.flat.ks--; + iter->u.flat.vs--; + } + return iter->idx > 0; + } + else { + ASSERT(is_hashmap(iter->map)); + + if (iter->idx > 0) { + if (iter->idx > iter->size) { + hashmap_iterator_init(&iter->u.hash.wstack->ws, iter->map, 1); + } + iter->u.hash.kv = hashmap_iterator_prev(&iter->u.hash.wstack->ws); + iter->idx--; + ASSERT(!!iter->u.hash.kv == (iter->idx > 0)); + } + return iter->idx > 0; } - return (iter->idx > 0); } int enif_map_iterator_get_pair(ErlNifEnv *env, @@ -2085,15 +2162,25 @@ int enif_map_iterator_get_pair(ErlNifEnv *env, Eterm *key, Eterm *value) { - ASSERT(iter && is_map(iter->map)); - if (iter->idx > 0 && iter->idx < iter->t_limit) { - ASSERT(iter->ks >= map_get_keys(map_val(iter->map)) && - iter->ks < (map_get_keys(map_val(iter->map)) + map_get_size(map_val(iter->map)))); - ASSERT(iter->vs >= map_get_values(map_val(iter->map)) && - iter->vs < (map_get_values(map_val(iter->map)) + map_get_size(map_val(iter->map)))); - *key = *(iter->ks); - *value = *(iter->vs); - return 1; + ASSERT(iter); + if (is_flatmap(iter->map)) { + if (iter->idx > 0 && iter->idx <= iter->size) { + ASSERT(iter->u.flat.ks >= flatmap_get_keys(flatmap_val(iter->map)) && + iter->u.flat.ks < (flatmap_get_keys(flatmap_val(iter->map)) + flatmap_get_size(flatmap_val(iter->map)))); + ASSERT(iter->u.flat.vs >= flatmap_get_values(flatmap_val(iter->map)) && + iter->u.flat.vs < (flatmap_get_values(flatmap_val(iter->map)) + flatmap_get_size(flatmap_val(iter->map)))); + *key = *(iter->u.flat.ks); + *value = *(iter->u.flat.vs); + return 1; + } + } + else { + ASSERT(is_hashmap(iter->map)); + if (iter->idx > 0 && iter->idx <= iter->size) { + *key = CAR(iter->u.hash.kv); + *value = CDR(iter->u.hash.kv); + return 1; + } } return 0; } diff --git a/erts/emulator/beam/erl_nif.h b/erts/emulator/beam/erl_nif.h index 849024453c..9b2b90c82d 100644 --- a/erts/emulator/beam/erl_nif.h +++ b/erts/emulator/beam/erl_nif.h @@ -201,10 +201,18 @@ typedef enum typedef struct /* All fields all internal and may change */ { ERL_NIF_TERM map; - ERL_NIF_UINT t_limit; + ERL_NIF_UINT size; ERL_NIF_UINT idx; - ERL_NIF_TERM *ks; - ERL_NIF_TERM *vs; + union { + struct { + ERL_NIF_TERM *ks; + ERL_NIF_TERM *vs; + }flat; + struct { + struct ErtsDynamicWStack_* wstack; + ERL_NIF_TERM* kv; + }hash; + }u; void* __spare__[2]; /* for future additions to be ABI compatible (same struct size) */ } ErlNifMapIterator; diff --git a/erts/emulator/beam/erl_printf_term.c b/erts/emulator/beam/erl_printf_term.c index c982dc2080..e0dfbd31b8 100644 --- a/erts/emulator/beam/erl_printf_term.c +++ b/erts/emulator/beam/erl_printf_term.c @@ -247,6 +247,17 @@ static int print_atom_name(fmtfn_t fn, void* arg, Eterm atom, long *dcount) #define PRT_PATCH_FUN_SIZE ((Eterm) 7) #define PRT_LAST_ARRAY_ELEMENT ((Eterm) 8) /* Note! Must be last... */ +#if 0 +static char *format_binary(Uint16 x, char *b) { + int z; + b[16] = '\0'; + for (z = 0; z < 16; z++) { + b[15-z] = ((x>>z) & 0x1) ? '1' : '0'; + } + return b; +} +#endif + static int print_term(fmtfn_t fn, void* arg, Eterm obj, long *dcount, Eterm* obj_base) /* ignored if !HALFWORD_HEAP */ @@ -554,28 +565,74 @@ print_term(fmtfn_t fn, void* arg, Eterm obj, long *dcount, } break; case MAP_DEF: - { - Uint n; - Eterm *ks, *vs; - map_t *mp = (map_t *)map_val(wobj); - n = map_get_size(mp); - ks = map_get_keys(mp); - vs = map_get_values(mp); - - PRINT_CHAR(res, fn, arg, '#'); - PRINT_CHAR(res, fn, arg, '{'); - WSTACK_PUSH(s, PRT_CLOSE_TUPLE); - if (n > 0) { - n--; - WSTACK_PUSH5(s, vs[n], PRT_TERM, PRT_ASSOC, ks[n], PRT_TERM); - while (n--) { - WSTACK_PUSH6(s, PRT_COMMA, vs[n], PRT_TERM, PRT_ASSOC, - ks[n], PRT_TERM); - } - } - } - break; - default: + if (is_flatmap(wobj)) { + Uint n; + Eterm *ks, *vs; + flatmap_t *mp = (flatmap_t *)flatmap_val(wobj); + n = flatmap_get_size(mp); + ks = flatmap_get_keys(mp); + vs = flatmap_get_values(mp); + + PRINT_CHAR(res, fn, arg, '#'); + PRINT_CHAR(res, fn, arg, '{'); + WSTACK_PUSH(s, PRT_CLOSE_TUPLE); + if (n > 0) { + n--; + WSTACK_PUSH5(s, vs[n], PRT_TERM, PRT_ASSOC, ks[n], PRT_TERM); + while (n--) { + WSTACK_PUSH6(s, PRT_COMMA, vs[n], PRT_TERM, PRT_ASSOC, + ks[n], PRT_TERM); + } + } + } else { + Uint n, mapval; + Eterm *head; + head = hashmap_val(wobj); + mapval = MAP_HEADER_VAL(*head); + switch (MAP_HEADER_TYPE(*head)) { + case MAP_HEADER_TAG_HAMT_HEAD_ARRAY: + case MAP_HEADER_TAG_HAMT_HEAD_BITMAP: + PRINT_STRING(res, fn, arg, "#<"); + PRINT_UWORD(res, fn, arg, 'x', 0, 1, mapval); + PRINT_STRING(res, fn, arg, ">{"); + WSTACK_PUSH(s,PRT_CLOSE_TUPLE); + n = hashmap_bitcount(mapval); + ASSERT(n < 17); + head += 2; + if (n > 0) { + n--; + WSTACK_PUSH(s, head[n]); + WSTACK_PUSH(s, PRT_TERM); + while (n--) { + WSTACK_PUSH(s, PRT_COMMA); + WSTACK_PUSH(s, head[n]); + WSTACK_PUSH(s, PRT_TERM); + } + } + break; + case MAP_HEADER_TAG_HAMT_NODE_BITMAP: + n = hashmap_bitcount(mapval); + head++; + PRINT_CHAR(res, fn, arg, '<'); + PRINT_UWORD(res, fn, arg, 'x', 0, 1, mapval); + PRINT_STRING(res, fn, arg, ">{"); + WSTACK_PUSH(s,PRT_CLOSE_TUPLE); + ASSERT(n < 17); + if (n > 0) { + n--; + WSTACK_PUSH(s, head[n]); + WSTACK_PUSH(s, PRT_TERM); + while (n--) { + WSTACK_PUSH(s, PRT_COMMA); + WSTACK_PUSH(s, head[n]); + WSTACK_PUSH(s, PRT_TERM); + } + } + break; + } + } + break; + default: PRINT_STRING(res, fn, arg, "<unknown:"); PRINT_POINTER(res, fn, arg, wobj); PRINT_CHAR(res, fn, arg, '>'); @@ -584,11 +641,11 @@ print_term(fmtfn_t fn, void* arg, Eterm obj, long *dcount, } L_done: - DESTROY_WSTACK(s); return res; } + int erts_printf_term(fmtfn_t fn, void* arg, ErlPfEterm term, long precision, ErlPfEterm* term_base) diff --git a/erts/emulator/beam/erl_process.c b/erts/emulator/beam/erl_process.c index ba09ee57c2..f74a2ee54c 100644 --- a/erts/emulator/beam/erl_process.c +++ b/erts/emulator/beam/erl_process.c @@ -43,6 +43,7 @@ #include "erl_async.h" #include "dtrace-wrapper.h" #include "erl_ptab.h" +#include "erl_bif_unique.h" #define ERTS_DELAYED_WAKEUP_INFINITY (~(Uint64) 0) @@ -701,8 +702,8 @@ init_sched_wall_time(ErtsSchedWallTime *swtp) static ERTS_INLINE Uint64 sched_wall_time_ts(void) { -#ifdef HAVE_GETHRTIME - return (Uint64) sys_gethrtime(); +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + return (Uint64) erts_os_monotonic_time(); #else Uint64 res; SysTimeval tv; @@ -2185,7 +2186,7 @@ aux_work_timeout_late_init(void) { aux_work_tmo->initialized = 1; if (erts_atomic32_read_nob(&aux_work_tmo->refc)) { - aux_work_tmo->timer.data.active = 0; + erts_init_timer(&aux_work_tmo->timer.data); erts_set_timer(&aux_work_tmo->timer.data, aux_work_timeout, NULL, @@ -2218,7 +2219,6 @@ aux_work_timeout(void *unused) if (refc != 1 || 1 != erts_atomic32_cmpxchg_relb(&aux_work_tmo->refc, 0, 1)) { /* Setup next timeout... */ - aux_work_tmo->timer.data.active = 0; erts_set_timer(&aux_work_tmo->timer.data, aux_work_timeout, NULL, @@ -2237,7 +2237,7 @@ setup_aux_work_timer(void) else #endif { - aux_work_tmo->timer.data.active = 0; + erts_init_timer(&aux_work_tmo->timer.data); erts_set_timer(&aux_work_tmo->timer.data, aux_work_timeout, NULL, @@ -2638,6 +2638,13 @@ thr_prgr_fin_wait(void *vssi) static void init_aux_work_data(ErtsAuxWorkData *awdp, ErtsSchedulerData *esdp, char *dawwp); +void +erts_interupt_aux_thread_timed(ErtsMonotonicTime timeout_time) +{ + /* TODO only poke when needed (based on timeout_time) */ + erts_sched_poke(ERTS_SCHED_SLEEP_INFO_IX(-1)); +} + static void * aux_thread(void *unused) { @@ -2646,6 +2653,11 @@ aux_thread(void *unused) erts_aint32_t aux_work; ErtsThrPrgrCallbacks callbacks; int thr_prgr_active = 1; + ErtsTimerWheel *timer_wheel = erts_default_timer_wheel; + ErtsNextTimeoutRef nxt_tmo_ref = erts_get_next_timeout_reference(timer_wheel); + + if (!timer_wheel) + ERTS_INTERNAL_ERROR("Missing aux timer wheel"); #ifdef ERTS_ENABLE_LOCK_CHECK { @@ -2669,6 +2681,7 @@ aux_thread(void *unused) sched_prep_spin_wait(ssi); while (1) { + ErtsMonotonicTime current_time; erts_aint32_t flgs; aux_work = erts_atomic32_read_acqb(&ssi->aux_work); @@ -2680,28 +2693,56 @@ aux_thread(void *unused) erts_thr_progress_leader_update(NULL); } - if (!aux_work) { - if (thr_prgr_active) - erts_thr_progress_active(NULL, thr_prgr_active = 0); - erts_thr_progress_prepare_wait(NULL); + if (aux_work) { + current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(nxt_tmo_ref)) { + if (!thr_prgr_active) + erts_thr_progress_active(NULL, thr_prgr_active = 1); + erts_bump_timers(timer_wheel, current_time); + } + } + else { + ErtsMonotonicTime timeout_time; + timeout_time = erts_check_next_timeout_time(timer_wheel, + ERTS_SEC_TO_MONOTONIC(10*60)); + current_time = erts_get_monotonic_time(); + if (current_time >= timeout_time) { + if (!thr_prgr_active) + erts_thr_progress_active(NULL, thr_prgr_active = 1); + } + else { + if (thr_prgr_active) + erts_thr_progress_active(NULL, thr_prgr_active = 0); + erts_thr_progress_prepare_wait(NULL); - ERTS_SCHED_FAIR_YIELD(); + ERTS_SCHED_FAIR_YIELD(); - flgs = sched_spin_wait(ssi, 0); + flgs = sched_spin_wait(ssi, 0); - if (flgs & ERTS_SSI_FLG_SLEEPING) { - ASSERT(flgs & ERTS_SSI_FLG_WAITING); - flgs = sched_set_sleeptype(ssi, ERTS_SSI_FLG_TSE_SLEEPING); if (flgs & ERTS_SSI_FLG_SLEEPING) { - int res; - ASSERT(flgs & ERTS_SSI_FLG_TSE_SLEEPING); ASSERT(flgs & ERTS_SSI_FLG_WAITING); - do { - res = erts_tse_wait(ssi->event); - } while (res == EINTR); + flgs = sched_set_sleeptype(ssi, ERTS_SSI_FLG_TSE_SLEEPING); + if (flgs & ERTS_SSI_FLG_SLEEPING) { + int res; + ASSERT(flgs & ERTS_SSI_FLG_TSE_SLEEPING); + ASSERT(flgs & ERTS_SSI_FLG_WAITING); + current_time = erts_get_monotonic_time(); + do { + Sint64 timeout; + if (current_time >= timeout_time) + break; + timeout = ERTS_MONOTONIC_TO_NSEC(timeout_time + - current_time + - 1) + 1; + res = erts_tse_twait(ssi->event, timeout); + current_time = erts_get_monotonic_time(); + } while (res == EINTR); + } } + erts_thr_progress_finalize_wait(NULL); } - erts_thr_progress_finalize_wait(NULL); + if (current_time >= timeout_time) + erts_bump_timers(timer_wheel, current_time); } flgs = sched_prep_spin_wait(ssi); @@ -2768,6 +2809,7 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) sched_wall_time_change(esdp, thr_prgr_active); while (1) { + ErtsMonotonicTime current_time; aux_work = erts_atomic32_read_acqb(&ssi->aux_work); if (aux_work) { @@ -2781,34 +2823,65 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) erts_thr_progress_leader_update(esdp); } - if (aux_work) + if (aux_work) { flgs = erts_smp_atomic32_read_acqb(&ssi->flags); + current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) { + if (!ERTS_SCHEDULER_IS_DIRTY(esdp) && !thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 1); + sched_wall_time_change(esdp, 1); + } + erts_bump_timers(esdp->timer_wheel, current_time); + } + } else { - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { - if (thr_prgr_active) { - erts_thr_progress_active(esdp, thr_prgr_active = 0); - sched_wall_time_change(esdp, 0); + ErtsMonotonicTime timeout_time; + timeout_time = erts_check_next_timeout_time(esdp->timer_wheel, + ERTS_SEC_TO_MONOTONIC(10*60)); + current_time = erts_get_monotonic_time(); + if (current_time >= timeout_time) { + if (!ERTS_SCHEDULER_IS_DIRTY(esdp) && !thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 1); + sched_wall_time_change(esdp, 1); } - erts_thr_progress_prepare_wait(esdp); } + else { + if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { + if (thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 0); + sched_wall_time_change(esdp, 0); + } + erts_thr_progress_prepare_wait(esdp); + } - ERTS_SCHED_FAIR_YIELD(); + ERTS_SCHED_FAIR_YIELD(); - flgs = sched_spin_wait(ssi, spincount); - if (flgs & ERTS_SSI_FLG_SLEEPING) { - ASSERT(flgs & ERTS_SSI_FLG_WAITING); - flgs = sched_set_sleeptype(ssi, ERTS_SSI_FLG_TSE_SLEEPING); + flgs = sched_spin_wait(ssi, spincount); if (flgs & ERTS_SSI_FLG_SLEEPING) { - int res; - ASSERT(flgs & ERTS_SSI_FLG_TSE_SLEEPING); ASSERT(flgs & ERTS_SSI_FLG_WAITING); - do { - res = erts_tse_wait(ssi->event); - } while (res == EINTR); + flgs = sched_set_sleeptype(ssi, ERTS_SSI_FLG_TSE_SLEEPING); + if (flgs & ERTS_SSI_FLG_SLEEPING) { + int res; + ASSERT(flgs & ERTS_SSI_FLG_TSE_SLEEPING); + ASSERT(flgs & ERTS_SSI_FLG_WAITING); + current_time = erts_get_monotonic_time(); + do { + Sint64 timeout; + if (current_time >= timeout_time) + break; + timeout = ERTS_MONOTONIC_TO_NSEC(timeout_time + - current_time + - 1) + 1; + res = erts_tse_twait(ssi->event, timeout); + current_time = erts_get_monotonic_time(); + } while (res == EINTR); + } } + if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) + erts_thr_progress_finalize_wait(esdp); } - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) - erts_thr_progress_finalize_wait(esdp); + if (current_time >= timeout_time) + erts_bump_timers(esdp->timer_wheel, current_time); } if (!(flgs & ERTS_SSI_FLG_WAITING)) { @@ -2841,7 +2914,6 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) else #endif { - erts_aint_t dt; erts_smp_atomic32_set_relb(&function_calls, 0); *fcalls = 0; @@ -2866,6 +2938,7 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) goto sys_aux_work; while (spincount-- > 0) { + ErtsMonotonicTime current_time; sys_poll_aux_work: @@ -2875,8 +2948,9 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) ASSERT(!erts_port_task_have_outstanding_io_tasks()); erl_sys_schedule(1); /* Might give us something to do */ - dt = erts_do_time_read_and_reset(); - if (dt) erts_bump_timer(dt); + current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) + erts_bump_timers(esdp->timer_wheel, current_time); sys_aux_work: #ifndef ERTS_SMP @@ -2991,8 +3065,11 @@ scheduler_wait(int *fcalls, ErtsSchedulerData *esdp, ErtsRunQueue *rq) erl_sys_schedule(0); - dt = erts_do_time_read_and_reset(); - if (dt) erts_bump_timer(dt); + { + ErtsMonotonicTime current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) + erts_bump_timers(esdp->timer_wheel, current_time); + } #ifndef ERTS_SMP if (rq->len == 0 && !rq->misc.start) @@ -5262,6 +5339,10 @@ init_scheduler_data(ErtsSchedulerData* esdp, int num, #else esdp->no = (Uint) num; #endif + + esdp->timer_wheel = erts_default_timer_wheel; + esdp->next_tmo_ref = erts_get_next_timeout_reference(esdp->timer_wheel); + esdp->ssi = ssi; esdp->current_process = NULL; esdp->current_port = NULL; @@ -5274,6 +5355,9 @@ init_scheduler_data(ErtsSchedulerData* esdp, int num, esdp->run_queue = runq; esdp->run_queue->scheduler = esdp; + esdp->thr_id = (Uint32) num; + erts_sched_bif_unique_init(esdp); + if (daww_ptr) { init_aux_work_data(&esdp->aux_work_data, esdp, *daww_ptr); #ifdef ERTS_SMP @@ -5834,6 +5918,13 @@ schedule_out_process(ErtsRunQueue *c_rq, erts_aint32_t state, Process *p, Proces int check_emigration_need; #endif +#ifdef ERTS_SMP + if ((p->static_flags & ERTS_STC_FLG_PREFER_SCHED) + && p->preferred_run_queue != RUNQ_READ_RQ(&p->run_queue)) { + RUNQ_SET_RQ(&p->run_queue, p->preferred_run_queue); + } +#endif + a = state; while (1) { @@ -5872,6 +5963,7 @@ schedule_out_process(ErtsRunQueue *c_rq, erts_aint32_t state, Process *p, Proces free_proxy_proc(proxy); erts_smp_runq_lock(c_rq); + return 0; #ifdef ERTS_DIRTY_SCHEDULERS @@ -6584,7 +6676,7 @@ suspend_scheduler(ErtsSchedulerData *esdp) int res; do { - res = erts_tse_wait(ssi->event); + res = erts_tse_twait(ssi->event, -1); } while (res == EINTR); } } @@ -6747,6 +6839,7 @@ suspend_scheduler(ErtsSchedulerData *esdp) erts_smp_mtx_unlock(&schdlr_sspnd.mtx); while (1) { + ErtsMonotonicTime current_time; erts_aint32_t qmask; erts_aint32_t flgs; @@ -6771,30 +6864,64 @@ suspend_scheduler(ErtsSchedulerData *esdp) } } - if (!aux_work) { - if (thr_prgr_active) { - erts_thr_progress_active(esdp, thr_prgr_active = 0); - sched_wall_time_change(esdp, 0); + if (aux_work) { + current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) { + if (!thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 1); + sched_wall_time_change(esdp, 1); + } + erts_bump_timers(esdp->timer_wheel, current_time); } - erts_thr_progress_prepare_wait(esdp); - flgs = sched_spin_suspended(ssi, - ERTS_SCHED_SUSPEND_SLEEP_SPINCOUNT); - if (flgs == (ERTS_SSI_FLG_SLEEPING - | ERTS_SSI_FLG_WAITING - | ERTS_SSI_FLG_SUSPENDED)) { - flgs = sched_set_suspended_sleeptype(ssi); + } + else { + ErtsMonotonicTime timeout_time; + timeout_time = erts_check_next_timeout_time(esdp->timer_wheel, + ERTS_SEC_TO_MONOTONIC(60*60)); + current_time = erts_get_monotonic_time(); + + if (current_time >= timeout_time) { + if (!thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 1); + sched_wall_time_change(esdp, 1); + } + } + else { + if (thr_prgr_active) { + erts_thr_progress_active(esdp, thr_prgr_active = 0); + sched_wall_time_change(esdp, 0); + } + erts_thr_progress_prepare_wait(esdp); + flgs = sched_spin_suspended(ssi, + ERTS_SCHED_SUSPEND_SLEEP_SPINCOUNT); if (flgs == (ERTS_SSI_FLG_SLEEPING - | ERTS_SSI_FLG_TSE_SLEEPING | ERTS_SSI_FLG_WAITING | ERTS_SSI_FLG_SUSPENDED)) { - int res; - - do { - res = erts_tse_wait(ssi->event); - } while (res == EINTR); + flgs = sched_set_suspended_sleeptype(ssi); + if (flgs == (ERTS_SSI_FLG_SLEEPING + | ERTS_SSI_FLG_TSE_SLEEPING + | ERTS_SSI_FLG_WAITING + | ERTS_SSI_FLG_SUSPENDED)) { + int res; + + current_time = erts_get_monotonic_time(); + do { + Sint64 timeout; + if (current_time >= timeout_time) + break; + timeout = ERTS_MONOTONIC_TO_NSEC(timeout_time + - current_time + - 1) + 1; + res = erts_tse_twait(ssi->event, timeout); + current_time = erts_get_monotonic_time(); + } while (res == EINTR); + } } + erts_thr_progress_finalize_wait(esdp); } - erts_thr_progress_finalize_wait(esdp); + + if (current_time >= timeout_time) + erts_bump_timers(esdp->timer_wheel, current_time); } flgs = sched_prep_spin_suspended(ssi, (ERTS_SSI_FLG_WAITING @@ -7613,6 +7740,9 @@ sched_thread_func(void *vesdp) ErtsThrPrgrCallbacks callbacks; ErtsSchedulerData *esdp = vesdp; Uint no = esdp->no; + + esdp->timer_wheel = erts_create_timer_wheel((int) no); + esdp->next_tmo_ref = erts_get_next_timeout_reference(esdp->timer_wheel); #ifdef ERTS_SMP ERTS_SCHED_SLEEP_INFO_IX(no - 1)->event = erts_tse_fetch(); callbacks.arg = (void *) esdp->ssi; @@ -7715,6 +7845,8 @@ sched_dirty_cpu_thread_func(void *vesdp) callbacks.wait = NULL; callbacks.finalize_wait = NULL; + esdp->thr_id += erts_no_schedulers; + erts_thr_progress_register_unmanaged_thread(&callbacks); #ifdef ERTS_ENABLE_LOCK_CHECK { @@ -7776,6 +7908,8 @@ sched_dirty_io_thread_func(void *vesdp) callbacks.wait = NULL; callbacks.finalize_wait = NULL; + esdp->thr_id += erts_no_schedulers + erts_no_dirty_cpu_schedulers; + erts_thr_progress_register_unmanaged_thread(&callbacks); #ifdef ERTS_ENABLE_LOCK_CHECK { @@ -8878,7 +9012,6 @@ Process *schedule(Process *p, int calls) { Process *proxy_p = NULL; ErtsRunQueue *rq; - erts_aint_t dt; ErtsSchedulerData *esdp; int context_reds; int fcalls; @@ -9008,11 +9141,13 @@ Process *schedule(Process *p, int calls) ERTS_SMP_CHK_NO_PROC_LOCKS; - dt = erts_do_time_read_and_reset(); - if (dt) { - erts_smp_runq_unlock(rq); - erts_bump_timer(dt); - erts_smp_runq_lock(rq); + { + ErtsMonotonicTime current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) { + erts_smp_runq_unlock(rq); + erts_bump_timers(esdp->timer_wheel, current_time); + erts_smp_runq_lock(rq); + } } BM_STOP_TIMER(system); @@ -9158,6 +9293,7 @@ Process *schedule(Process *p, int calls) else if (!ERTS_SCHEDULER_IS_DIRTY(esdp) && (fcalls > input_reductions && prepare_for_sys_schedule(esdp, !0))) { + ErtsMonotonicTime current_time; /* * Schedule system-level activities. */ @@ -9170,8 +9306,10 @@ Process *schedule(Process *p, int calls) #endif erts_smp_runq_unlock(rq); erl_sys_schedule(1); - dt = erts_do_time_read_and_reset(); - if (dt) erts_bump_timer(dt); + + current_time = erts_get_monotonic_time(); + if (current_time >= erts_next_timeout_time(esdp->next_tmo_ref)) + erts_bump_timers(esdp->timer_wheel, current_time); #ifdef ERTS_SMP erts_smp_runq_lock(rq); @@ -10476,7 +10614,10 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). int ix = so->scheduler-1; ASSERT(0 <= ix && ix < erts_no_run_queues); rq = ERTS_RUNQ_IX(ix); - state |= ERTS_PSFLG_BOUND; + if (!(so->flags & SPO_PREFER_SCHED)) { + /* Unsupported feature... */ + state |= ERTS_PSFLG_BOUND; + } } prio = (erts_aint32_t) so->priority; } @@ -10484,6 +10625,9 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). state |= (((prio & ERTS_PSFLGS_PRIO_MASK) << ERTS_PSFLGS_ACT_PRIO_OFFSET) | ((prio & ERTS_PSFLGS_PRIO_MASK) << ERTS_PSFLGS_USR_PRIO_OFFSET)); + if (so->flags & SPO_OFF_HEAP_MSGS) + state |= ERTS_PSFLG_OFF_HEAP_MSGS; + if (!rq) rq = erts_get_runq_proc(parent); @@ -10507,11 +10651,25 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). heap_need = arg_size; p->flags = erts_default_process_flags; + if (so->flags & SPO_OFF_HEAP_MSGS) + p->flags |= F_OFF_HEAP_MSGS; +#ifdef ERTS_SMP + p->preferred_run_queue = NULL; +#endif + p->static_flags = 0; + if (so->flags & SPO_SYSTEM_PROC) + p->static_flags |= ERTS_STC_FLG_SYSTEM_PROC; if (so->flags & SPO_USE_ARGS) { p->min_heap_size = so->min_heap_size; p->min_vheap_size = so->min_vheap_size; p->max_gen_gcs = so->max_gen_gcs; + if (so->flags & SPO_PREFER_SCHED) { +#ifdef ERTS_SMP + p->preferred_run_queue = rq; +#endif + p->static_flags |= ERTS_STC_FLG_PREFER_SCHED; + } } else { p->min_heap_size = H_MIN_SIZE; p->min_vheap_size = BIN_VH_MIN_SIZE; @@ -10589,7 +10747,7 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). #ifdef ERTS_SMP p->common.u.alive.ptimer = NULL; #else - sys_memset(&p->common.u.alive.tm, 0, sizeof(ErlTimer)); + erts_init_timer(&p->common.u.alive.tm); #endif p->common.u.alive.reg = NULL; @@ -10782,7 +10940,7 @@ void erts_init_empty_process(Process *p) #ifdef ERTS_SMP p->common.u.alive.ptimer = NULL; #else - memset(&(p->common.u.alive.tm), 0, sizeof(ErlTimer)); + erts_init_timer(&p->common.u.alive.tm); #endif p->next = NULL; p->off_heap.first = NULL; @@ -10832,6 +10990,8 @@ void erts_init_empty_process(Process *p) p->parent = NIL; p->approx_started = 0; + p->static_flags = 0; + p->common.u.alive.started_interval = 0; #ifdef HIPE @@ -10857,6 +11017,7 @@ void erts_init_empty_process(Process *p) p->pending_suspenders = NULL; p->pending_exit.reason = THE_NON_VALUE; p->pending_exit.bp = NULL; + p->preferred_run_queue = NULL; erts_proc_lock_init(p); erts_smp_proc_unlock(p, ERTS_PROC_LOCKS_ALL); RUNQ_SET_RQ(&p->run_queue, ERTS_RUNQ_IX(0)); @@ -11482,7 +11643,8 @@ static void doit_exit_monitor(ErtsMonitor *mon, void *vpcontext) ErtsMonitor *rmon; Process *rp; - if (mon->type == MON_ORIGIN) { + switch (mon->type) { + case MON_ORIGIN: /* We are monitoring someone else, we need to demonitor that one.. */ if (is_atom(mon->pid)) { /* remote by name */ ASSERT(is_node_name_atom(mon->pid)); @@ -11545,7 +11707,8 @@ static void doit_exit_monitor(ErtsMonitor *mon, void *vpcontext) } } } - } else { /* type == MON_TARGET */ + break; + case MON_TARGET: ASSERT(mon->type == MON_TARGET); ASSERT(is_pid(mon->pid) || is_internal_port(mon->pid)); if (is_internal_port(mon->pid)) { @@ -11604,6 +11767,12 @@ static void doit_exit_monitor(ErtsMonitor *mon, void *vpcontext) } } } + break; + case MON_TIME_OFFSET: + erts_demonitor_time_offset(mon->ref); + break; + default: + ERTS_INTERNAL_ERROR("Invalid monitor type"); } done: /* As the monitors are previously removed from the process, @@ -11762,6 +11931,9 @@ erts_do_exit_process(Process* p, Eterm reason) } #endif + if (p->static_flags & ERTS_STC_FLG_SYSTEM_PROC) + erl_exit(1, "System process %T terminated: %T\n", p->common.id, reason); + #ifdef ERTS_SMP ERTS_SMP_CHK_HAVE_ONLY_MAIN_PROC_LOCK(p); /* By locking all locks (main lock is already locked) when going diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index d12ac792af..743711cc3b 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -348,7 +348,7 @@ typedef struct { } ErtsRunQueueInfo; -#ifdef HAVE_GETHRTIME +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT # undef ERTS_HAVE_SCHED_UTIL_BALANCING_SUPPORT_OPT # define ERTS_HAVE_SCHED_UTIL_BALANCING_SUPPORT_OPT 1 #endif @@ -564,6 +564,8 @@ struct ErtsSchedulerData_ { Eterm* x_reg_array; /* X registers */ FloatDef* f_reg_array; /* Floating point registers. */ + ErtsTimerWheel *timer_wheel; + ErtsNextTimeoutRef next_tmo_ref; #ifdef ERTS_SMP ethr_tid tid; /* Thread id */ struct erl_bits_state erl_bits_state; /* erl_bits.c state */ @@ -590,6 +592,10 @@ struct ErtsSchedulerData_ { ErtsAuxWorkData aux_work_data; ErtsAtomCacheMap atom_cache_map; + Uint32 thr_id; + Uint64 unique; + Uint64 ref; + ErtsSchedAllocData alloc_data; Uint64 reductions; @@ -936,6 +942,8 @@ struct process { Eterm parent; /* Pid of process that created this process. */ erts_approx_time_t approx_started; /* Time when started. */ + Uint32 static_flags; /* Flags that do *not* change */ + /* This is the place, where all fields that differs between memory * architectures, have gone to. */ @@ -967,6 +975,7 @@ struct process { ErtsSchedulerData *scheduler_data; Eterm suspendee; ErtsPendingSuspend *pending_suspenders; + ErtsRunQueue *preferred_run_queue; erts_smp_atomic_t run_queue; #ifdef HIPE struct hipe_process_state_smp hipe_smp; @@ -1076,14 +1085,15 @@ void erts_check_for_holes(Process* p); #define ERTS_PSFLG_RUNNING_SYS ERTS_PSFLG_BIT(15) #define ERTS_PSFLG_PROXY ERTS_PSFLG_BIT(16) #define ERTS_PSFLG_DELAYED_SYS ERTS_PSFLG_BIT(17) +#define ERTS_PSFLG_OFF_HEAP_MSGS ERTS_PSFLG_BIT(18) #ifdef ERTS_DIRTY_SCHEDULERS -#define ERTS_PSFLG_DIRTY_CPU_PROC ERTS_PSFLG_BIT(18) -#define ERTS_PSFLG_DIRTY_IO_PROC ERTS_PSFLG_BIT(19) -#define ERTS_PSFLG_DIRTY_CPU_PROC_IN_Q ERTS_PSFLG_BIT(20) -#define ERTS_PSFLG_DIRTY_IO_PROC_IN_Q ERTS_PSFLG_BIT(21) -#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 22) +#define ERTS_PSFLG_DIRTY_CPU_PROC ERTS_PSFLG_BIT(19) +#define ERTS_PSFLG_DIRTY_IO_PROC ERTS_PSFLG_BIT(20) +#define ERTS_PSFLG_DIRTY_CPU_PROC_IN_Q ERTS_PSFLG_BIT(21) +#define ERTS_PSFLG_DIRTY_IO_PROC_IN_Q ERTS_PSFLG_BIT(22) +#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 23) #else -#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 18) +#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 19) #endif #define ERTS_PSFLGS_IN_PRQ_MASK (ERTS_PSFLG_IN_PRQ_MAX \ @@ -1098,6 +1108,12 @@ void erts_check_for_holes(Process* p); #define ERTS_PSFLGS_GET_PRQ_PRIO(PSFLGS) \ (((PSFLGS) >> ERTS_PSFLGS_USR_PRIO_OFFSET) & ERTS_PSFLGS_PRIO_MASK) +/* + * Static flags that do not change after process creation. + */ +#define ERTS_STC_FLG_SYSTEM_PROC (((Uint32) 1) << 0) +#define ERTS_STC_FLG_PREFER_SCHED (((Uint32) 1) << 1) + /* The sequential tracing token is a tuple of size 5: * * {Flags, Label, Serial, Sender} @@ -1125,6 +1141,9 @@ void erts_check_for_holes(Process* p); #define SPO_LINK 1 #define SPO_USE_ARGS 2 #define SPO_MONITOR 4 +#define SPO_OFF_HEAP_MSGS 8 +#define SPO_SYSTEM_PROC 16 +#define SPO_PREFER_SCHED 32 /* * The following struct contains options for a process to be spawned. @@ -1212,6 +1231,7 @@ extern struct erts_system_profile_flags_t erts_system_profile_flags; #define F_P2PNR_RESCHED (1 << 9) /* Process has been rescheduled via erts_pid2proc_not_running() */ #define F_FORCE_GC (1 << 10) /* Force gc at process in-scheduling */ #define F_DISABLE_GC (1 << 11) /* Disable GC */ +#define F_OFF_HEAP_MSGS (1 << 12) /* process trace_flags */ #define F_SENSITIVE (1 << 0) @@ -2228,6 +2248,8 @@ extern int erts_disable_proc_not_running_opt; void erts_smp_notify_inc_runq(ErtsRunQueue *runq); +void erts_interupt_aux_thread_timed(ErtsMonotonicTime timeout_time); + #ifdef ERTS_SMP void erts_sched_finish_poke(ErtsSchedulerSleepInfo *, erts_aint32_t); ERTS_GLB_INLINE void erts_sched_poke(ErtsSchedulerSleepInfo *ssi); diff --git a/erts/emulator/beam/erl_process_dict.c b/erts/emulator/beam/erl_process_dict.c index 3ce707efda..00761f2d0e 100644 --- a/erts/emulator/beam/erl_process_dict.c +++ b/erts/emulator/beam/erl_process_dict.c @@ -47,7 +47,7 @@ /* Hash constant macros */ #define MAX_HASH 1342177280UL -#define INITIAL_SIZE 10 +#define INITIAL_SIZE (erts_pd_initial_size) /* Hash utility macros */ #define HASH_RANGE(PDict) ((PDict)->homeSize + (PDict)->splitPosition) diff --git a/erts/emulator/beam/erl_term.c b/erts/emulator/beam/erl_term.c index 28cbe7004f..bc04d7b78e 100644 --- a/erts/emulator/beam/erl_term.c +++ b/erts/emulator/beam/erl_term.c @@ -86,11 +86,12 @@ unsigned tag_val_def(Wterm x) case (_TAG_HEADER_EXTERNAL_PID >> _TAG_PRIMARY_SIZE): return EXTERNAL_PID_DEF; case (_TAG_HEADER_EXTERNAL_PORT >> _TAG_PRIMARY_SIZE): return EXTERNAL_PORT_DEF; case (_TAG_HEADER_EXTERNAL_REF >> _TAG_PRIMARY_SIZE): return EXTERNAL_REF_DEF; + case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE): return MAP_DEF; case (_TAG_HEADER_REFC_BIN >> _TAG_PRIMARY_SIZE): return BINARY_DEF; case (_TAG_HEADER_HEAP_BIN >> _TAG_PRIMARY_SIZE): return BINARY_DEF; case (_TAG_HEADER_SUB_BIN >> _TAG_PRIMARY_SIZE): return BINARY_DEF; - case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE): return MAP_DEF; } + break; } case TAG_PRIMARY_IMMED1: { diff --git a/erts/emulator/beam/erl_term.h b/erts/emulator/beam/erl_term.h index 37014ccf94..095aa54ddd 100644 --- a/erts/emulator/beam/erl_term.h +++ b/erts/emulator/beam/erl_term.h @@ -147,21 +147,21 @@ struct erl_node_; /* Declared in erl_node_tables.h */ #define MAP_SUBTAG (0xF << _TAG_PRIMARY_SIZE) /* MAP */ -#define _TAG_HEADER_ARITYVAL (TAG_PRIMARY_HEADER|ARITYVAL_SUBTAG) -#define _TAG_HEADER_FUN (TAG_PRIMARY_HEADER|FUN_SUBTAG) -#define _TAG_HEADER_POS_BIG (TAG_PRIMARY_HEADER|POS_BIG_SUBTAG) -#define _TAG_HEADER_NEG_BIG (TAG_PRIMARY_HEADER|NEG_BIG_SUBTAG) -#define _TAG_HEADER_FLOAT (TAG_PRIMARY_HEADER|FLOAT_SUBTAG) -#define _TAG_HEADER_EXPORT (TAG_PRIMARY_HEADER|EXPORT_SUBTAG) -#define _TAG_HEADER_REF (TAG_PRIMARY_HEADER|REF_SUBTAG) -#define _TAG_HEADER_REFC_BIN (TAG_PRIMARY_HEADER|REFC_BINARY_SUBTAG) -#define _TAG_HEADER_HEAP_BIN (TAG_PRIMARY_HEADER|HEAP_BINARY_SUBTAG) -#define _TAG_HEADER_SUB_BIN (TAG_PRIMARY_HEADER|SUB_BINARY_SUBTAG) -#define _TAG_HEADER_EXTERNAL_PID (TAG_PRIMARY_HEADER|EXTERNAL_PID_SUBTAG) -#define _TAG_HEADER_EXTERNAL_PORT (TAG_PRIMARY_HEADER|EXTERNAL_PORT_SUBTAG) -#define _TAG_HEADER_EXTERNAL_REF (TAG_PRIMARY_HEADER|EXTERNAL_REF_SUBTAG) +#define _TAG_HEADER_ARITYVAL (TAG_PRIMARY_HEADER|ARITYVAL_SUBTAG) +#define _TAG_HEADER_FUN (TAG_PRIMARY_HEADER|FUN_SUBTAG) +#define _TAG_HEADER_POS_BIG (TAG_PRIMARY_HEADER|POS_BIG_SUBTAG) +#define _TAG_HEADER_NEG_BIG (TAG_PRIMARY_HEADER|NEG_BIG_SUBTAG) +#define _TAG_HEADER_FLOAT (TAG_PRIMARY_HEADER|FLOAT_SUBTAG) +#define _TAG_HEADER_EXPORT (TAG_PRIMARY_HEADER|EXPORT_SUBTAG) +#define _TAG_HEADER_REF (TAG_PRIMARY_HEADER|REF_SUBTAG) +#define _TAG_HEADER_REFC_BIN (TAG_PRIMARY_HEADER|REFC_BINARY_SUBTAG) +#define _TAG_HEADER_HEAP_BIN (TAG_PRIMARY_HEADER|HEAP_BINARY_SUBTAG) +#define _TAG_HEADER_SUB_BIN (TAG_PRIMARY_HEADER|SUB_BINARY_SUBTAG) +#define _TAG_HEADER_EXTERNAL_PID (TAG_PRIMARY_HEADER|EXTERNAL_PID_SUBTAG) +#define _TAG_HEADER_EXTERNAL_PORT (TAG_PRIMARY_HEADER|EXTERNAL_PORT_SUBTAG) +#define _TAG_HEADER_EXTERNAL_REF (TAG_PRIMARY_HEADER|EXTERNAL_REF_SUBTAG) #define _TAG_HEADER_BIN_MATCHSTATE (TAG_PRIMARY_HEADER|BIN_MATCHSTATE_SUBTAG) -#define _TAG_HEADER_MAP (TAG_PRIMARY_HEADER|MAP_SUBTAG) +#define _TAG_HEADER_MAP (TAG_PRIMARY_HEADER|MAP_SUBTAG) #define _TAG_HEADER_MASK 0x3F @@ -296,9 +296,11 @@ _ET_DECLARE_CHECKED(Uint,atom_val,Eterm) #define atom_val(x) _ET_APPLY(atom_val,(x)) /* header (arityval or thing) access methods */ -#define _make_header(sz,tag) ((Uint)(((sz) << _HEADER_ARITY_OFFS) + (tag))) +#define _make_header(sz,tag) ((Uint)(((Uint)(sz) << _HEADER_ARITY_OFFS) + (tag))) #define is_header(x) (((x) & _TAG_PRIMARY_MASK) == TAG_PRIMARY_HEADER) -#define _unchecked_header_arity(x) ((x) >> _HEADER_ARITY_OFFS) +//#define _unchecked_header_arity(x) ((x) >> _HEADER_ARITY_OFFS) +#define _unchecked_header_arity(x) \ + (is_map_header(x) ? MAP_HEADER_ARITY(x) : ((x) >> _HEADER_ARITY_OFFS)) _ET_DECLARE_CHECKED(Uint,header_arity,Eterm) #define header_arity(x) _ET_APPLY(header_arity,(x)) @@ -361,6 +363,7 @@ _ET_DECLARE_CHECKED(Uint,thing_subtag,Eterm) ((((x) & (_TAG_HEADER_MASK)) == _TAG_HEADER_REFC_BIN) || \ (((x) & (_TAG_HEADER_MASK)) == _TAG_HEADER_HEAP_BIN) || \ (((x) & (_TAG_HEADER_MASK)) == _TAG_HEADER_SUB_BIN)) + #define make_binary(x) make_boxed((Eterm*)(x)) #define is_binary(x) (is_boxed((x)) && is_binary_header(*boxed_val((x)))) #define is_not_binary(x) (!is_binary((x))) @@ -990,6 +993,43 @@ _ET_DECLARE_CHECKED(Uint32*,external_ref_data,Wterm) _ET_DECLARE_CHECKED(struct erl_node_*,external_ref_node,Eterm) #define external_ref_node(x) _ET_APPLY(external_ref_node,(x)) +/* maps */ + +#define MAP_HEADER_TAG_SZ (2) +#define MAP_HEADER_ARITY_SZ (8) +#define MAP_HEADER_VAL_SZ (16) + +#define MAP_HEADER_TAG_FLATMAP_HEAD (0x0) +#define MAP_HEADER_TAG_HAMT_NODE_BITMAP (0x1) +#define MAP_HEADER_TAG_HAMT_HEAD_ARRAY (0x2) +#define MAP_HEADER_TAG_HAMT_HEAD_BITMAP (0x3) + +#define MAP_HEADER_TYPE(Hdr) (((Hdr) >> (_HEADER_ARITY_OFFS)) & (0x3)) +#define MAP_HEADER_ARITY(Hdr) (((Hdr) >> (_HEADER_ARITY_OFFS + MAP_HEADER_TAG_SZ)) & (0xff)) +#define MAP_HEADER_VAL(Hdr) (((Hdr) >> (_HEADER_ARITY_OFFS + MAP_HEADER_TAG_SZ + MAP_HEADER_ARITY_SZ)) & (0xffff)) + +#define make_hashmap(x) make_boxed((Eterm*)(x)) +#define make_hashmap_rel make_boxed_rel +#define is_hashmap(x) (is_boxed((x)) && is_hashmap_header(*boxed_val((x)))) +#define is_not_hashmap(x) (!is_hashmap(x)) +#define is_hashmap_rel(RTERM,BASE) is_hashmap(rterm2wterm(RTERM,BASE)) +#define is_hashmap_header(x) (((x) & (_HEADER_MAP_HASHMAP_HEAD_MASK)) == HAMT_SUBTAG_HEAD_ARRAY) +#define hashmap_val(x) _unchecked_boxed_val((x)) +#define hashmap_val_rel(RTERM, BASE) hashmap_val(rterm2wterm(RTERM, BASE)) + +#define make_flatmap(x) make_boxed((Eterm*)(x)) +#define make_flatmap_rel(x, BASE) make_boxed_rel((Eterm*)(x),(BASE)) +#define is_flatmap(x) (is_boxed((x)) && is_flatmap_header(*boxed_val((x)))) +#define is_flatmap_rel(RTERM,BASE) is_flatmap(rterm2wterm(RTERM,BASE)) +#define is_not_flatmap(x) (!is_flatmap((x))) +#define is_flatmap_header(x) (((x) & (_HEADER_MAP_SUBTAG_MASK)) == HAMT_SUBTAG_HEAD_FLATMAP) +#define flatmap_val(x) (_unchecked_boxed_val((x))) +#define flatmap_val_rel(RTERM, BASE) flatmap_val(rterm2wterm(RTERM, BASE)) + +#define is_map_header(x) (((x) & (_TAG_HEADER_MASK)) == _TAG_HEADER_MAP) +#define is_map(x) (is_boxed((x)) && is_map_header(*boxed_val(x))) +#define is_map_rel(RTERM,BASE) is_map(rterm2wterm(RTERM,BASE)) + /* number tests */ #define is_integer(x) (is_small(x) || is_big(x)) @@ -1096,6 +1136,8 @@ _ET_DECLARE_CHECKED(Uint,y_reg_index,Uint) #define BIG_DEF 0xf #define SMALL_DEF 0x10 +#define FIRST_VACANT_TAG_DEF 0x11 + #if ET_DEBUG extern unsigned tag_val_def_debug(Wterm, const char*, unsigned); #define tag_val_def(x) tag_val_def_debug((x),__FILE__,__LINE__) diff --git a/erts/emulator/beam/erl_thr_progress.c b/erts/emulator/beam/erl_thr_progress.c index c2365c5cf7..4c9b00d2ee 100644 --- a/erts/emulator/beam/erl_thr_progress.c +++ b/erts/emulator/beam/erl_thr_progress.c @@ -1359,18 +1359,16 @@ void erts_thr_progress_fatal_error_wait(SWord timeout) { erts_aint32_t bc; SWord time_left = timeout; - SysTimeval to; + ErtsMonotonicTime timeout_time; /* * Counting poll intervals may give us a too long timeout - * if cpu is busy. If we got tolerant time of day we use it - * to prevent this. + * if cpu is busy. We use timeout time to try to prevent + * this. In case we havn't got time correction this may + * however fail too... */ - if (!erts_disable_tolerant_timeofday) { - erts_get_timeval(&to); - to.tv_sec += timeout / 1000; - to.tv_sec += timeout % 1000; - } + timeout_time = erts_get_monotonic_time(); + timeout_time += ERTS_MSEC_TO_MONOTONIC((ErtsMonotonicTime) timeout); while (1) { if (erts_milli_sleep(ERTS_THR_PRGR_FTL_ERR_BLCK_POLL_INTERVAL) == 0) @@ -1380,14 +1378,8 @@ erts_thr_progress_fatal_error_wait(SWord timeout) { break; /* Succefully blocked all managed threads */ if (time_left <= 0) break; /* Timeout */ - if (!erts_disable_tolerant_timeofday) { - SysTimeval now; - erts_get_timeval(&now); - if (now.tv_sec > to.tv_sec) - break; /* Timeout */ - if (now.tv_sec == to.tv_sec && now.tv_usec >= to.tv_usec) - break; /* Timeout */ - } + if (timeout_time <= erts_get_monotonic_time()) + break; /* Timeout */ } } diff --git a/erts/emulator/beam/erl_threads.h b/erts/emulator/beam/erl_threads.h index 1fd800d524..dc20ac207f 100644 --- a/erts/emulator/beam/erl_threads.h +++ b/erts/emulator/beam/erl_threads.h @@ -652,6 +652,8 @@ ERTS_GLB_INLINE void erts_tse_set(erts_tse_t *ep); ERTS_GLB_INLINE void erts_tse_reset(erts_tse_t *ep); ERTS_GLB_INLINE int erts_tse_wait(erts_tse_t *ep); ERTS_GLB_INLINE int erts_tse_swait(erts_tse_t *ep, int spincount); +ERTS_GLB_INLINE int erts_tse_twait(erts_tse_t *ep, Sint64 tmo); +ERTS_GLB_INLINE int erts_tse_stwait(erts_tse_t *ep, int spincount, Sint64 tmo); ERTS_GLB_INLINE int erts_tse_is_tmp(erts_tse_t *ep); ERTS_GLB_INLINE void erts_thr_set_main_status(int, int); ERTS_GLB_INLINE int erts_thr_get_main_status(void); @@ -3490,6 +3492,27 @@ ERTS_GLB_INLINE int erts_tse_swait(erts_tse_t *ep, int spincount) #endif } +ERTS_GLB_INLINE int erts_tse_twait(erts_tse_t *ep, Sint64 tmo) +{ +#ifdef USE_THREADS + return ethr_event_twait(&((ethr_ts_event *) ep)->event, + (ethr_sint64_t) tmo); +#else + return ENOTSUP; +#endif +} + +ERTS_GLB_INLINE int erts_tse_stwait(erts_tse_t *ep, int spincount, Sint64 tmo) +{ +#ifdef USE_THREADS + return ethr_event_stwait(&((ethr_ts_event *) ep)->event, + spincount, + (ethr_sint64_t) tmo); +#else + return ENOTSUP; +#endif +} + ERTS_GLB_INLINE int erts_tse_is_tmp(erts_tse_t *ep) { #ifdef USE_THREADS diff --git a/erts/emulator/beam/erl_time.h b/erts/emulator/beam/erl_time.h index 7ed1a395ad..cb7764addc 100644 --- a/erts/emulator/beam/erl_time.h +++ b/erts/emulator/beam/erl_time.h @@ -20,11 +20,16 @@ #ifndef ERL_TIME_H__ #define ERL_TIME_H__ -#define ERTS_SHORT_TIME_T_MAX ERTS_AINT32_T_MAX -#define ERTS_SHORT_TIME_T_MIN ERTS_AINT32_T_MIN -typedef erts_aint32_t erts_short_time_t; +#if defined(DEBUG) || 0 +#define ERTS_TIME_ASSERT(B) ERTS_ASSERT(B) +#else +#define ERTS_TIME_ASSERT(B) ((void) 1) +#endif + +typedef struct ErtsTimerWheel_ ErtsTimerWheel; +typedef erts_atomic64_t * ErtsNextTimeoutRef; +extern ErtsTimerWheel *erts_default_timer_wheel; -extern erts_smp_atomic32_t do_time; /* set at clock interrupt */ extern SysTimeval erts_first_emu_time; /* @@ -34,8 +39,8 @@ typedef struct erl_timer { struct erl_timer* next; /* next entry tiw slot or chain */ struct erl_timer* prev; /* prev entry tiw slot or chain */ Uint slot; /* slot in timer wheel */ - Uint count; /* number of loops remaining */ - int active; /* 1=activated, 0=deactivated */ + erts_smp_atomic_t wheel; + ErtsMonotonicTime timeout_pos; /* Timeout in absolute clock ticks */ /* called when timeout */ void (*timeout)(void*); /* called when cancel (may be NULL) */ @@ -62,7 +67,6 @@ union ErtsSmpPTimer_ { ErtsSmpPTimer *next; }; - void erts_create_smp_ptimer(ErtsSmpPTimer **timer_ref, Eterm id, ErlTimeoutProc timeout_func, @@ -70,36 +74,42 @@ void erts_create_smp_ptimer(ErtsSmpPTimer **timer_ref, void erts_cancel_smp_ptimer(ErtsSmpPTimer *ptimer); #endif +void erts_monitor_time_offset(Eterm id, Eterm ref); +int erts_demonitor_time_offset(Eterm ref); + +void erts_late_init_time_sup(void); + /* timer-wheel api */ -void erts_init_time(void); +ErtsTimerWheel *erts_create_timer_wheel(int); +ErtsNextTimeoutRef erts_get_next_timeout_reference(ErtsTimerWheel *); +void erts_init_time(int time_correction, ErtsTimeWarpMode time_warp_mode); void erts_set_timer(ErlTimer*, ErlTimeoutProc, ErlCancelProc, void*, Uint); void erts_cancel_timer(ErlTimer*); -void erts_bump_timer(erts_short_time_t); -Uint erts_timer_wheel_memory_size(void); Uint erts_time_left(ErlTimer *); -erts_short_time_t erts_next_time(void); +void erts_bump_timers(ErtsTimerWheel *, ErtsMonotonicTime); +Uint erts_timer_wheel_memory_size(void); #ifdef DEBUG void erts_p_slpq(void); #endif -ERTS_GLB_INLINE erts_short_time_t erts_do_time_read_and_reset(void); -ERTS_GLB_INLINE void erts_do_time_add(erts_short_time_t); +ErtsMonotonicTime erts_check_next_timeout_time(ErtsTimerWheel *, + ErtsMonotonicTime); + +ERTS_GLB_INLINE void erts_init_timer(ErlTimer *p); +ERTS_GLB_INLINE ErtsMonotonicTime erts_next_timeout_time(ErtsNextTimeoutRef); #if ERTS_GLB_INLINE_INCL_FUNC_DEF -ERTS_GLB_INLINE erts_short_time_t erts_do_time_read_and_reset(void) +ERTS_GLB_INLINE void erts_init_timer(ErlTimer *p) { - erts_short_time_t time = erts_smp_atomic32_xchg_acqb(&do_time, 0); - if (time < 0) - erl_exit(ERTS_ABORT_EXIT, "Internal time management error\n"); - return time; + erts_smp_atomic_init_nob(&p->wheel, (erts_aint_t) NULL); } -ERTS_GLB_INLINE void erts_do_time_add(erts_short_time_t elapsed) +ERTS_GLB_INLINE ErtsMonotonicTime erts_next_timeout_time(ErtsNextTimeoutRef nxt_tmo_ref) { - erts_smp_atomic32_add_relb(&do_time, elapsed); + return (ErtsMonotonicTime) erts_atomic64_read_acqb((erts_atomic64_t *) nxt_tmo_ref); } #endif /* #if ERTS_GLB_INLINE_INCL_FUNC_DEF */ @@ -121,25 +131,220 @@ void erts_get_now_cpu(Uint* megasec, Uint* sec, Uint* microsec); typedef UWord erts_approx_time_t; erts_approx_time_t erts_get_approx_time(void); -void erts_get_timeval(SysTimeval *tv); -erts_time_t erts_get_time(void); +int erts_has_time_correction(void); +int erts_check_time_adj_support(int time_correction, + ErtsTimeWarpMode time_warp_mode); + +ErtsTimeWarpMode erts_time_warp_mode(void); -ERTS_GLB_INLINE int erts_cmp_timeval(SysTimeval *t1p, SysTimeval *t2p); +typedef enum { + ERTS_TIME_OFFSET_PRELIMINARY, + ERTS_TIME_OFFSET_FINAL, + ERTS_TIME_OFFSET_VOLATILE +} ErtsTimeOffsetState; + +ErtsTimeOffsetState erts_time_offset_state(void); +ErtsTimeOffsetState erts_finalize_time_offset(void); +struct process; +Eterm erts_get_monotonic_start_time(struct process *c_p); +Eterm erts_monotonic_time_source(struct process*c_p); +Eterm erts_system_time_source(struct process*c_p); + +#ifdef SYS_CLOCK_RESOLUTION +#define ERTS_CLKTCK_RESOLUTION ((ErtsMonotonicTime) (SYS_CLOCK_RESOLUTION*1000)) +#else +#define ERTS_CLKTCK_RESOLUTION (erts_time_sup__.r.o.clktck_resolution) +#endif + +struct erts_time_sup_read_only__ { + ErtsMonotonicTime monotonic_time_unit; +#ifndef SYS_CLOCK_RESOLUTION + ErtsMonotonicTime clktck_resolution; +#endif +}; + +typedef struct { + union { + struct erts_time_sup_read_only__ o; + char align__[(((sizeof(struct erts_time_sup_read_only__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } r; +} ErtsTimeSupData; + +extern ErtsTimeSupData erts_time_sup__; + +ERTS_GLB_INLINE Uint64 +erts_time_unit_conversion(Uint64 value, + Uint32 from_time_unit, + Uint32 to_time_unit); #if ERTS_GLB_INLINE_INCL_FUNC_DEF -ERTS_GLB_INLINE int -erts_cmp_timeval(SysTimeval *t1p, SysTimeval *t2p) +ERTS_GLB_INLINE Uint64 +erts_time_unit_conversion(Uint64 value, + Uint32 from_time_unit, + Uint32 to_time_unit) { - if (t1p->tv_sec == t2p->tv_sec) { - if (t1p->tv_usec < t2p->tv_usec) - return -1; - else if (t1p->tv_usec > t2p->tv_usec) - return 1; - return 0; - } - return t1p->tv_sec < t2p->tv_sec ? -1 : 1; + Uint64 high, low, result; + if (value <= ~((Uint64) 0)/to_time_unit) + return (value*to_time_unit)/from_time_unit; + + low = value & ((Uint64) 0xffffffff); + high = (value >> 32) & ((Uint64) 0xffffffff); + + low *= to_time_unit; + high *= to_time_unit; + + high += (low >> 32) & ((Uint64) 0xffffffff); + low &= ((Uint64) 0xffffffff); + + result = high % from_time_unit; + high /= from_time_unit; + high <<= 32; + + result <<= 32; + result += low; + result /= from_time_unit; + result += high; + + return result; } -#endif /* #if ERTS_GLB_INLINE_INCL_FUNC_DEF */ +#endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ + +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + +/* + * If the monotonic time unit is a compile time constant, + * it is assumed (and need) to be a power of 10. + */ + +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT < 1000*1000 +# error Compile time time unit needs to be at least 1000000 +#endif + +#define ERTS_MONOTONIC_TIME_UNIT \ + ((ErtsMonotonicTime) ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT) + +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT == 1000*1000*1000 +/* Nano-second time unit */ + +#define ERTS_MONOTONIC_TO_SEC__(NSEC) ((NSEC) / (1000*1000*1000)) +#define ERTS_MONOTONIC_TO_MSEC__(NSEC) ((NSEC) / (1000*1000)) +#define ERTS_MONOTONIC_TO_USEC__(NSEC) ((NSEC) / 1000) +#define ERTS_MONOTONIC_TO_NSEC__(NSEC) (NSEC) + +#define ERTS_SEC_TO_MONOTONIC__(SEC) (((ErtsMonotonicTime) (SEC))*(1000*1000*1000)) +#define ERTS_MSEC_TO_MONOTONIC__(MSEC) (((ErtsMonotonicTime) (MSEC))*(1000*1000)) +#define ERTS_USEC_TO_MONOTONIC__(USEC) (((ErtsMonotonicTime) (USEC))*1000) +#define ERTS_NSEC_TO_MONOTONIC__(NSEC) ((ErtsMonotonicTime) (NSEC)) + +#elif ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT == 1000*1000 +/* Micro-second time unit */ + +#define ERTS_MONOTONIC_TO_SEC__(USEC) ((USEC) / (1000*1000)) +#define ERTS_MONOTONIC_TO_MSEC__(USEC) ((USEC) / 1000) +#define ERTS_MONOTONIC_TO_USEC__(USEC) (USEC) +#define ERTS_MONOTONIC_TO_NSEC__(USEC) ((USEC)*1000) + +#define ERTS_SEC_TO_MONOTONIC__(SEC) (((ErtsMonotonicTime) (SEC))*(1000*1000)) +#define ERTS_MSEC_TO_MONOTONIC__(MSEC) (((ErtsMonotonicTime) (MSEC))*1000) +#define ERTS_USEC_TO_MONOTONIC__(USEC) ((ErtsMonotonicTime) (USEC)) +#define ERTS_NSEC_TO_MONOTONIC__(NSEC) (((ErtsMonotonicTime) (NSEC))/1000) + +#else +#error Missing implementation for monotonic time unit +#endif + +#define ERTS_MONOTONIC_TO_CLKTCKS__(MON) \ + ((MON) / (ERTS_MONOTONIC_TIME_UNIT/ERTS_CLKTCK_RESOLUTION)) +#define ERTS_CLKTCKS_TO_MONOTONIC__(TCKS) \ + ((TCKS) * (ERTS_MONOTONIC_TIME_UNIT/ERTS_CLKTCK_RESOLUTION)) + +#else /* !ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT */ + +#define ERTS_MONOTONIC_TIME_UNIT (erts_time_sup__.r.o.monotonic_time_unit) + +#define ERTS_CONV_FROM_MON_UNIT___(M, TO) \ + ((ErtsMonotonicTime) \ + erts_time_unit_conversion((Uint64) (M), \ + (Uint32) ERTS_MONOTONIC_TIME_UNIT, \ + (Uint32) (TO))) + +#define ERTS_CONV_TO_MON_UNIT___(M, FROM) \ + ((ErtsMonotonicTime) \ + erts_time_unit_conversion((Uint64) (M), \ + (Uint32) (FROM), \ + (Uint32) ERTS_MONOTONIC_TIME_UNIT)) \ + +#define ERTS_MONOTONIC_TO_SEC__(M) \ + ERTS_CONV_FROM_MON_UNIT___((M), 1) +#define ERTS_MONOTONIC_TO_MSEC__(M) \ + ERTS_CONV_FROM_MON_UNIT___((M), 1000) +#define ERTS_MONOTONIC_TO_USEC__(M) \ + ERTS_CONV_FROM_MON_UNIT___((M), 1000*1000) +#define ERTS_MONOTONIC_TO_NSEC__(M) \ + ERTS_CONV_FROM_MON_UNIT___((M), 1000*1000*1000) + +#define ERTS_SEC_TO_MONOTONIC__(SEC) \ + ERTS_CONV_TO_MON_UNIT___((SEC), 1) +#define ERTS_MSEC_TO_MONOTONIC__(MSEC) \ + ERTS_CONV_TO_MON_UNIT___((MSEC), 1000) +#define ERTS_USEC_TO_MONOTONIC__(USEC) \ + ERTS_CONV_TO_MON_UNIT___((USEC), 1000*1000) +#define ERTS_NSEC_TO_MONOTONIC__(NSEC) \ + ERTS_CONV_TO_MON_UNIT___((NSEC), 1000*1000*1000) + +#define ERTS_MONOTONIC_TO_CLKTCKS__(MON) \ + ERTS_CONV_FROM_MON_UNIT___((MON), ERTS_CLKTCK_RESOLUTION) +#define ERTS_CLKTCKS_TO_MONOTONIC__(TCKS) \ + ERTS_CONV_TO_MON_UNIT___((TCKS), ERTS_CLKTCK_RESOLUTION) + +#endif /* !ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT */ + +#define ERTS_MSEC_TO_CLKTCKS__(MON) \ + ((MON) * (ERTS_CLKTCK_RESOLUTION/1000)) +#define ERTS_CLKTCKS_TO_MSEC__(TCKS) \ + ((TCKS) / (ERTS_CLKTCK_RESOLUTION/1000)) + +#define ERTS_MONOTONIC_TO_SEC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MONOTONIC_TO_SEC__((X))) +#define ERTS_MONOTONIC_TO_MSEC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MONOTONIC_TO_MSEC__((X))) +#define ERTS_MONOTONIC_TO_USEC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MONOTONIC_TO_USEC__((X))) +#define ERTS_MONOTONIC_TO_NSEC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MONOTONIC_TO_NSEC__((X))) +#define ERTS_SEC_TO_MONOTONIC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_SEC_TO_MONOTONIC__((X))) +#define ERTS_MSEC_TO_MONOTONIC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MSEC_TO_MONOTONIC__((X))) +#define ERTS_USEC_TO_MONOTONIC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_USEC_TO_MONOTONIC__((X))) +#define ERTS_NSEC_TO_MONOTONIC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_NSEC_TO_MONOTONIC__((X))) + +#define ERTS_MONOTONIC_TO_CLKTCKS(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MONOTONIC_TO_CLKTCKS__((X))) +#define ERTS_CLKTCKS_TO_MONOTONIC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_CLKTCKS_TO_MONOTONIC__((X))) + +#define ERTS_MSEC_TO_CLKTCKS(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_MSEC_TO_CLKTCKS__((X))) +#define ERTS_CLKTCKS_TO_MSEC(X) \ + (ERTS_TIME_ASSERT((X) >= 0), \ + ERTS_CLKTCKS_TO_MSEC__((X))) + #endif /* ERL_TIME_H__ */ diff --git a/erts/emulator/beam/erl_time_sup.c b/erts/emulator/beam/erl_time_sup.c index 3272a5326d..bbdedcc128 100644 --- a/erts/emulator/beam/erl_time_sup.c +++ b/erts/emulator/beam/erl_time_sup.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 1999-2012. All Rights Reserved. + * Copyright Ericsson AB 1999-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -18,60 +18,10 @@ */ /* -** Support routines for the timer wheel -** -** This code contains two strategies for dealing with -** date/time changes in the system. -** If the system has some kind of high resolution timer (HAVE_GETHRTIME), -** the high resolution timer is used to correct the time-of-day and the -** timeouts, the base source is the hrtimer, but at certain intervals the -** OS time-of-day is checked and if it is not within certain bounds, the -** delivered time gets slowly adjusted for each call until -** it corresponds to the system time (built-in adjtime...). -** The call gethrtime() is detected by autoconf on Unix, but other -** platforms may define it in erl_*_sys.h and implement -** their own high resolution timer. The high resolution timer -** strategy is (probably) best on all systems where the timer have -** a resolution higher or equal to gettimeofday (or what's implemented -** is sys_gettimeofday()). The actual resolution is the interesting thing, -** not the unit's thats used (i.e. on VxWorks, nanoseconds can be -** retrieved in terms of units, but the actual resolution is the same as -** for the clock ticks). -** If the systems best timer routine is kernel ticks returned from -** sys_times(), and the actual resolution of sys_gettimeofday() is -** better (like most unixes that does not have any realtime extensions), -** another strategy is used. The tolerant gettimeofday() corrects -** the value with respect to uptime (sys_times() return value) and checks -** for correction both when delivering timeticks and delivering nowtime. -** this strategy is slower, but accurate on systems without better timer -** routines. The kernel tick resolution is not enough to implement -** a gethrtime routine. On Linux and other non solaris unix-boxes the second -** strategy is used, on all other platforms we use the first. -** -** The following is expected (from sys.[ch] and erl_*_sys.h): -** -** 64 bit integers. So it is, and so it will be. -** -** sys_init_time(), will return the clock resolution in MS and -** that's about it. More could be added of course -** If the clock-rate is constant (i.e. 1 ms) one can define -** SYS_CLOCK_RESOLUTION (to 1), -** which makes erts_deliver_time/erts_time_remaining a bit faster. -** -** if HAVE_GETHRTIME is defined: -** sys_gethrtime() will return a SysHrTime (long long) representing -** nanoseconds, sys_init_hrtime() will do any initialization. -** else -** a long (64bit) integer type called Sint64 should be defined. -** -** sys_times() will return clock_ticks since start and -** fill in a SysTimes structure (struct tms). Instead of CLK_TCK, -** SYS_CLK_TCK is used to determine the resolution of kernel ticks. -** -** sys_gettimeofday() will take a SysTimeval (a struct timeval) as parameter -** and fill it in as gettimeofday(X,NULL). -** -*/ + * Support routines for the time + */ + +/* #define ERTS_TIME_CORRECTION_PRINT */ #ifdef HAVE_CONFIG_H # include "config.h" @@ -80,384 +30,1107 @@ #include "sys.h" #include "erl_vm.h" #include "global.h" - + static erts_smp_mtx_t erts_timeofday_mtx; - -static SysTimeval inittv; /* Used everywhere, the initial time-of-day */ +static erts_smp_mtx_t erts_get_time_mtx; static SysTimes t_start; /* Used in elapsed_time_both */ -static SysTimeval gtv; /* Used in wall_clock_elapsed_time_both */ -static SysTimeval then; /* Used in get_now */ -static SysTimeval last_emu_time; /* Used in erts_get_emu_time() */ -SysTimeval erts_first_emu_time; /* Used in erts_get_emu_time() */ +static ErtsMonotonicTime prev_wall_clock_elapsed; /* Used in wall_clock_elapsed_time_both */ +static ErtsMonotonicTime previous_now; /* Used in get_now */ -union { - erts_smp_atomic_t time; - char align[ERTS_CACHE_LINE_SIZE]; -} approx erts_align_attribute(ERTS_CACHE_LINE_SIZE); +static ErtsMonitor *time_offset_monitors = NULL; +static Uint no_time_offset_monitors = 0; + +#ifdef DEBUG +static int time_sup_initialized = 0; +#endif + +#define ERTS_MONOTONIC_TIME_KILO \ + ((ErtsMonotonicTime) 1000) +#define ERTS_MONOTONIC_TIME_MEGA \ + (ERTS_MONOTONIC_TIME_KILO*ERTS_MONOTONIC_TIME_KILO) +#define ERTS_MONOTONIC_TIME_GIGA \ + (ERTS_MONOTONIC_TIME_MEGA*ERTS_MONOTONIC_TIME_KILO) +#define ERTS_MONOTONIC_TIME_TERA \ + (ERTS_MONOTONIC_TIME_GIGA*ERTS_MONOTONIC_TIME_KILO) static void -init_approx_time(void) +schedule_send_time_offset_changed_notifications(ErtsMonotonicTime new_offset); + +/* + * NOTE! ERTS_MONOTONIC_TIME_START *need* to be a multiple + * of ERTS_MONOTONIC_TIME_UNIT. + */ + +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + +#ifdef ARCH_32 +/* + * Want to use a big-num of arity 2 as long as possible (584 years + * in the nano-second time unit case). + */ +#define ERTS_MONOTONIC_TIME_START \ + (((((((ErtsMonotonicTime) 1) << 32)-1) \ + / ERTS_MONOTONIC_TIME_UNIT) \ + * ERTS_MONOTONIC_TIME_UNIT) \ + + ERTS_MONOTONIC_TIME_UNIT) + +#else /* ARCH_64 */ + +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT <= 10*1000*1000 + +/* + * Using micro second time unit or lower. Start at zero since + * time will remain an immediate for a very long time anyway + * (1827 years in the 10 micro second case)... + */ +#define ERTS_MONOTONIC_TIME_START ((ErtsMonotonicTime) 0) + +#else /* ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT > 1000*1000 */ + +/* + * Want to use an immediate as long as possible (36 years in the + * nano-second time unit case). +*/ +#define ERTS_MONOTONIC_TIME_START \ + ((((ErtsMonotonicTime) MIN_SMALL) \ + / ERTS_MONOTONIC_TIME_UNIT) \ + * ERTS_MONOTONIC_TIME_UNIT) + +#endif /* ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT > 1000*1000 */ + +#endif /* ARCH_64 */ + +#define ERTS_MONOTONIC_OFFSET_NATIVE \ + (ERTS_MONOTONIC_TIME_START - ERTS_MONOTONIC_TIME_UNIT) +#define ERTS_MONOTONIC_OFFSET_NSEC \ + ERTS_MONOTONIC_TO_NSEC__(ERTS_MONOTONIC_OFFSET_NATIVE) +#define ERTS_MONOTONIC_OFFSET_USEC \ + ERTS_MONOTONIC_TO_USEC__(ERTS_MONOTONIC_OFFSET_NATIVE) +#define ERTS_MONOTONIC_OFFSET_MSEC \ + ERTS_MONOTONIC_TO_MSEC__(ERTS_MONOTONIC_OFFSET_NATIVE) +#define ERTS_MONOTONIC_OFFSET_SEC \ + ERTS_MONOTONIC_TO_SEC__(ERTS_MONOTONIC_OFFSET_NATIVE) + +#else /* ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT */ + +/* + * Initialized in erts_init_time_sup()... + */ + +#define ERTS_MONOTONIC_TIME_START (time_sup.r.o.start) +#define ERTS_MONOTONIC_OFFSET_NATIVE (time_sup.r.o.start_offset.native) +#define ERTS_MONOTONIC_OFFSET_NSEC (time_sup.r.o.start_offset.nsec) +#define ERTS_MONOTONIC_OFFSET_USEC (time_sup.r.o.start_offset.usec) +#define ERTS_MONOTONIC_OFFSET_MSEC (time_sup.r.o.start_offset.msec) +#define ERTS_MONOTONIC_OFFSET_SEC (time_sup.r.o.start_offset.sec) + +#endif /* ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT */ + +struct time_sup_read_only__ { + ErtsMonotonicTime (*get_time)(void); + int correction; + ErtsTimeWarpMode warp_mode; +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + ErtsMonotonicTime moffset; + int os_monotonic_time_disable; + char *os_monotonic_time_func; + char *os_monotonic_time_clock_id; + int os_monotonic_time_locked; + Uint64 os_monotonic_time_resolution; + Uint64 os_monotonic_time_extended; +#endif + char *os_system_time_func; + char *os_system_time_clock_id; + int os_system_time_locked; + Uint64 os_system_time_resolution; + Uint64 os_system_time_extended; +#if !ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + ErtsMonotonicTime start; + struct { + ErtsMonotonicTime native; + ErtsMonotonicTime nsec; + ErtsMonotonicTime usec; + ErtsMonotonicTime msec; + ErtsMonotonicTime sec; + } start_offset; +#endif + struct { + ErtsMonotonicTime large_diff; + ErtsMonotonicTime small_diff; + } adj; +}; + +typedef struct { +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + ErtsMonotonicTime drift; /* Correction for os monotonic drift */ +#endif + ErtsMonotonicTime error; /* Correction for error between system times */ +} ErtsMonotonicCorrection; + +typedef struct { + ErtsMonotonicTime erl_mtime; + ErtsMonotonicTime os_mtime; + ErtsMonotonicCorrection correction; +} ErtsMonotonicCorrectionInstance; + +#define ERTS_DRIFT_INTERVALS 5 +typedef struct { + struct { + struct { + ErtsMonotonicTime sys; + ErtsMonotonicTime mon; + } diff; + struct { + ErtsMonotonicTime sys; + ErtsMonotonicTime mon; + } time; + } intervals[ERTS_DRIFT_INTERVALS]; + struct { + ErtsMonotonicTime sys; + ErtsMonotonicTime mon; + } acc; + int ix; + int dirty_counter; +} ErtsMonotonicDriftData; + +typedef struct { + ErtsMonotonicCorrectionInstance prev; + ErtsMonotonicCorrectionInstance curr; +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + ErtsMonotonicDriftData drift; +#endif + ErtsMonotonicTime last_check; + int short_check_interval; +} ErtsMonotonicCorrectionData; + +struct time_sup_infrequently_changed__ { +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + struct { + erts_smp_rwmtx_t rwmtx; + ErlTimer timer; + ErtsMonotonicCorrectionData cdata; + } parmon; + ErtsMonotonicTime minit; +#endif + int finalized_offset; + ErtsSystemTime sinit; + ErtsMonotonicTime not_corrected_moffset; + erts_atomic64_t offset; +}; + +struct time_sup_frequently_changed__ { + ErtsMonotonicTime last_not_corrected_time; +}; + +static struct { + union { + struct time_sup_read_only__ o; + char align__[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(struct time_sup_read_only__))]; + } r; + union { + struct time_sup_infrequently_changed__ c; + char align__[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(struct time_sup_infrequently_changed__))]; + } inf; + union { + struct time_sup_frequently_changed__ c; + char align__[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(struct time_sup_frequently_changed__))]; + } f; +} time_sup erts_align_attribute(ERTS_CACHE_LINE_SIZE); + +ErtsTimeSupData erts_time_sup__ erts_align_attribute(ERTS_CACHE_LINE_SIZE); + +/* + * erts_get_approx_time() returns an *approximate* time + * in seconds. NOTE that this time may jump backwards!!! + */ +erts_approx_time_t +erts_get_approx_time(void) { - erts_smp_atomic_init_nob(&approx.time, 0); + ErtsSystemTime stime = erts_os_system_time(); + return (erts_approx_time_t) ERTS_MONOTONIC_TO_SEC(stime); } -static ERTS_INLINE erts_approx_time_t -get_approx_time(void) +static ERTS_INLINE void +init_time_offset(ErtsMonotonicTime offset) { - return (erts_approx_time_t) erts_smp_atomic_read_nob(&approx.time); + erts_atomic64_init_nob(&time_sup.inf.c.offset, (erts_aint64_t) offset); } static ERTS_INLINE void -update_approx_time(SysTimeval *tv) +set_time_offset(ErtsMonotonicTime offset) { - erts_approx_time_t new_secs = (erts_approx_time_t) tv->tv_sec; - erts_approx_time_t old_secs = get_approx_time(); - if (old_secs != new_secs) - erts_smp_atomic_set_nob(&approx.time, new_secs); + erts_atomic64_set_relb(&time_sup.inf.c.offset, (erts_aint64_t) offset); } -/* - * erts_get_approx_time() returns an *approximate* time - * in seconds. NOTE that this time may jump backwards!!! - */ -erts_approx_time_t -erts_get_approx_time(void) +static ERTS_INLINE ErtsMonotonicTime +get_time_offset(void) { - return get_approx_time(); + return (ErtsMonotonicTime) erts_atomic64_read_acqb(&time_sup.inf.c.offset); } -#ifdef HAVE_GETHRTIME -int erts_disable_tolerant_timeofday; +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT -static SysHrTime hr_init_time, hr_last_correction_check, - hr_correction, hr_last_time; +/* + * Time correction adjustments made due to + * error between Erlang system time and OS + * system time: + * - Large adjustment ~1% + * - Small adjustment ~0.05% + */ +#define ERTS_TCORR_ERR_UNIT 2048 +#define ERTS_TCORR_ERR_LARGE_ADJ 20 +#define ERTS_TCORR_ERR_SMALL_ADJ 1 + +#define ERTS_INIT_SHORT_INTERVAL_COUNTER 10 +#define ERTS_LONG_TIME_CORRECTION_CHECK ERTS_SEC_TO_MONOTONIC(60) +#define ERTS_SHORT_TIME_CORRECTION_CHECK ERTS_SEC_TO_MONOTONIC(15) + +#define ERTS_TIME_DRIFT_MAX_ADJ_DIFF ERTS_USEC_TO_MONOTONIC(50) +#define ERTS_TIME_DRIFT_MIN_ADJ_DIFF ERTS_USEC_TO_MONOTONIC(5) -static void init_tolerant_timeofday(void) +static ERTS_INLINE ErtsMonotonicTime +calc_corrected_erl_mtime(ErtsMonotonicTime os_mtime, + ErtsMonotonicCorrectionInstance *cip, + ErtsMonotonicTime *os_mdiff_p) { - /* Should be in sys.c */ -#if defined(HAVE_SYSCONF) && defined(_SC_NPROCESSORS_CONF) - if (sysconf(_SC_NPROCESSORS_CONF) > 1) { - char b[1024]; - int maj,min,build; - os_flavor(b,1024); - os_version(&maj,&min,&build); - if (!strcmp(b,"sunos") && maj <= 5 && min <= 7) { - erts_disable_tolerant_timeofday = 1; - } - } + ErtsMonotonicTime erl_mtime, diff = os_mtime - cip->os_mtime; + ERTS_TIME_ASSERT(diff >= 0); +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + diff += (cip->correction.drift*diff)/ERTS_MONOTONIC_TIME_UNIT; #endif - hr_init_time = sys_gethrtime(); - hr_last_correction_check = hr_last_time = hr_init_time; - hr_correction = 0; + erl_mtime = cip->erl_mtime; + erl_mtime += diff; + erl_mtime += cip->correction.error*(diff/ERTS_TCORR_ERR_UNIT); + if (os_mdiff_p) + *os_mdiff_p = diff; + return erl_mtime; } -static void get_tolerant_timeofday(SysTimeval *tv) +static ErtsMonotonicTime get_corrected_time(void) { - SysHrTime diff_time, curr; + ErtsMonotonicTime os_mtime; + ErtsMonotonicCorrectionData cdata; + ErtsMonotonicCorrectionInstance *cip; + + erts_smp_rwmtx_rlock(&time_sup.inf.c.parmon.rwmtx); - if (erts_disable_tolerant_timeofday) { - sys_gettimeofday(tv); - return; + os_mtime = erts_os_monotonic_time(); + + cdata = time_sup.inf.c.parmon.cdata; + + erts_smp_rwmtx_runlock(&time_sup.inf.c.parmon.rwmtx); + + if (os_mtime >= cdata.curr.os_mtime) + cip = &cdata.curr; + else { + if (os_mtime < cdata.prev.os_mtime) + erl_exit(ERTS_ABORT_EXIT, + "OS monotonic time stepped backwards\n"); + cip = &cdata.prev; } - *tv = inittv; - diff_time = ((curr = sys_gethrtime()) + hr_correction - hr_init_time) / 1000; - if (curr < hr_init_time) { - erl_exit(1,"Unexpected behaviour from operating system high " - "resolution timer"); + return calc_corrected_erl_mtime(os_mtime, cip, NULL); +} + +#ifdef ERTS_TIME_CORRECTION_PRINT + +static ERTS_INLINE void +print_correction(int change, + ErtsMonotonicTime sdiff, + ErtsMonotonicTime old_ecorr, + ErtsMonotonicTime old_dcorr, + ErtsMonotonicTime new_ecorr, + ErtsMonotonicTime new_dcorr, + Uint tmo) +{ + ErtsMonotonicTime usec_sdiff; + if (sdiff < 0) + usec_sdiff = -1*ERTS_MONOTONIC_TO_USEC(-1*sdiff); + else + usec_sdiff = ERTS_MONOTONIC_TO_USEC(sdiff); + + if (!change) + fprintf(stderr, + "sdiff = %lld usec : [ec=%lld ppm, dc=%lld ppb] : " + "tmo = %lld msec\r\n", + (long long) usec_sdiff, + (long long) (1000000*old_ecorr) / ERTS_TCORR_ERR_UNIT, + (long long) (1000000000*old_dcorr) / ERTS_MONOTONIC_TIME_UNIT, + (long long) tmo); + else + fprintf(stderr, + "sdiff = %lld usec : [ec=%lld ppm, dc=%lld ppb] " + "-> [ec=%lld ppm, dc=%lld ppb] : tmo = %lld msec\r\n", + (long long) usec_sdiff, + (long long) (1000000*old_ecorr) / ERTS_TCORR_ERR_UNIT, + (long long) (1000000000*old_dcorr) / ERTS_MONOTONIC_TIME_UNIT, + (long long) (1000000*new_ecorr) / ERTS_TCORR_ERR_UNIT, + (long long) (1000000000*new_dcorr) / ERTS_MONOTONIC_TIME_UNIT, + (long long) tmo); + +} + +#endif + +static void +check_time_correction(void *unused) +{ +#ifndef ERTS_TIME_CORRECTION_PRINT +# define ERTS_PRINT_CORRECTION +#else +# ifdef ERTS_HAVE_CORRECTED_OS_MONOTONIC +# define ERTS_PRINT_CORRECTION \ + print_correction(set_new_correction, \ + sdiff, \ + cip->correction.error, \ + 0, \ + new_correction.error, \ + 0, \ + timeout) +# else +# define ERTS_PRINT_CORRECTION \ + print_correction(set_new_correction, \ + sdiff, \ + cip->correction.error, \ + cip->correction.drift, \ + new_correction.error, \ + new_correction.drift, \ + timeout) +# endif +#endif + ErtsMonotonicCorrectionData cdata; + ErtsMonotonicCorrection new_correction; + ErtsMonotonicCorrectionInstance *cip; + ErtsMonotonicTime mdiff, sdiff, os_mtime, erl_mtime, os_stime, + erl_stime, time_offset; + Uint timeout; + int set_new_correction, begin_short_intervals = 0; + + erts_smp_rwmtx_rlock(&time_sup.inf.c.parmon.rwmtx); + + ASSERT(time_sup.inf.c.finalized_offset); + + erts_os_times(&os_mtime, &os_stime); + + cdata = time_sup.inf.c.parmon.cdata; + + erts_smp_rwmtx_runlock(&time_sup.inf.c.parmon.rwmtx); + + if (os_mtime < cdata.curr.os_mtime) + erl_exit(ERTS_ABORT_EXIT, + "OS monotonic time stepped backwards\n"); + cip = &cdata.curr; + + erl_mtime = calc_corrected_erl_mtime(os_mtime, cip, &mdiff); + time_offset = get_time_offset(); + erl_stime = erl_mtime + time_offset; + + sdiff = erl_stime - os_stime; + + new_correction = cip->correction; + + if (time_sup.r.o.warp_mode == ERTS_MULTI_TIME_WARP_MODE + && (sdiff < -2*time_sup.r.o.adj.small_diff + || 2*time_sup.r.o.adj.small_diff < sdiff)) { + /* System time diff exeeded limits; change time offset... */ + time_offset -= sdiff; + sdiff = 0; + set_time_offset(time_offset); + schedule_send_time_offset_changed_notifications(time_offset); + begin_short_intervals = 1; + if (cdata.curr.correction.error == 0) + set_new_correction = 0; + else { + set_new_correction = 1; + new_correction.error = 0; + } + } + else if (cdata.curr.correction.error == 0) { + if (sdiff < -time_sup.r.o.adj.small_diff) { + set_new_correction = 1; + if (sdiff < -time_sup.r.o.adj.large_diff) + new_correction.error = ERTS_TCORR_ERR_LARGE_ADJ; + else + new_correction.error = ERTS_TCORR_ERR_SMALL_ADJ; + } + else if (sdiff > time_sup.r.o.adj.small_diff) { + set_new_correction = 1; + if (sdiff > time_sup.r.o.adj.large_diff) + new_correction.error = -ERTS_TCORR_ERR_LARGE_ADJ; + else + new_correction.error = -ERTS_TCORR_ERR_SMALL_ADJ; + } + else { + set_new_correction = 0; + } + } + else if (cdata.curr.correction.error > 0) { + if (sdiff < 0) { + if (cdata.curr.correction.error == ERTS_TCORR_ERR_LARGE_ADJ + || -time_sup.r.o.adj.large_diff <= sdiff) + set_new_correction = 0; + else { + new_correction.error = ERTS_TCORR_ERR_LARGE_ADJ; + set_new_correction = 1; + } + } + else if (sdiff > time_sup.r.o.adj.small_diff) { + set_new_correction = 1; + if (sdiff > time_sup.r.o.adj.large_diff) + new_correction.error = -ERTS_TCORR_ERR_LARGE_ADJ; + else + new_correction.error = -ERTS_TCORR_ERR_SMALL_ADJ; + } + else { + set_new_correction = 1; + new_correction.error = 0; + } + } + else /* if (cdata.curr.correction.error < 0) */ { + if (0 < sdiff) { + if (cdata.curr.correction.error == -ERTS_TCORR_ERR_LARGE_ADJ + || sdiff <= time_sup.r.o.adj.large_diff) + set_new_correction = 0; + else { + new_correction.error = -ERTS_TCORR_ERR_LARGE_ADJ; + set_new_correction = 1; + } + set_new_correction = 0; + } + else if (sdiff < -time_sup.r.o.adj.small_diff) { + set_new_correction = 1; + if (sdiff < -time_sup.r.o.adj.large_diff) + new_correction.error = ERTS_TCORR_ERR_LARGE_ADJ; + else + new_correction.error = ERTS_TCORR_ERR_SMALL_ADJ; + } + else { + set_new_correction = 1; + new_correction.error = 0; + } } - if ((curr - hr_last_correction_check) / 1000 > 1000000) { - /* Check the correction need */ - SysHrTime tv_diff, diffdiff; - SysTimeval tmp; - int done = 0; - - sys_gettimeofday(&tmp); - tv_diff = ((SysHrTime) tmp.tv_sec) * 1000000 + tmp.tv_usec; - tv_diff -= ((SysHrTime) inittv.tv_sec) * 1000000 + inittv.tv_usec; - diffdiff = diff_time - tv_diff; - if (diffdiff > 10000) { - SysHrTime corr = (curr - hr_last_time) / 100; - if (corr / 1000 >= diffdiff) { - ++done; - hr_correction -= ((SysHrTime)diffdiff) * 1000; - } else { - hr_correction -= corr; +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + { + ErtsMonotonicDriftData *ddp = &time_sup.inf.c.parmon.cdata.drift; + int ix = ddp->ix; + ErtsMonotonicTime mtime_diff, old_os_mtime; + + old_os_mtime = ddp->intervals[ix].time.mon; + mtime_diff = os_mtime - old_os_mtime; + + if (mtime_diff >= ERTS_SEC_TO_MONOTONIC(10)) { + ErtsMonotonicTime drift_adj, drift_adj_diff, old_os_stime, + stime_diff, mtime_acc, stime_acc, avg_drift_adj; + + old_os_stime = ddp->intervals[ix].time.sys; + + mtime_acc = ddp->acc.mon; + stime_acc = ddp->acc.sys; + + avg_drift_adj = (((stime_acc - mtime_acc)*ERTS_MONOTONIC_TIME_UNIT) + / mtime_acc); + + mtime_diff = os_mtime - old_os_mtime; + stime_diff = os_stime - old_os_stime; + drift_adj = (((stime_diff - mtime_diff)*ERTS_MONOTONIC_TIME_UNIT) + / mtime_diff); + + ix++; + if (ix >= ERTS_DRIFT_INTERVALS) + ix = 0; + mtime_acc -= ddp->intervals[ix].diff.mon; + mtime_acc += mtime_diff; + stime_acc -= ddp->intervals[ix].diff.sys; + stime_acc += stime_diff; + + ddp->intervals[ix].diff.mon = mtime_diff; + ddp->intervals[ix].diff.sys = stime_diff; + ddp->intervals[ix].time.mon = os_mtime; + ddp->intervals[ix].time.sys = os_stime; + + ddp->ix = ix; + ddp->acc.mon = mtime_acc; + ddp->acc.sys = stime_acc; + + drift_adj_diff = avg_drift_adj - drift_adj; + if (drift_adj_diff < -ERTS_TIME_DRIFT_MAX_ADJ_DIFF + || ERTS_TIME_DRIFT_MAX_ADJ_DIFF < drift_adj_diff) { + ddp->dirty_counter = ERTS_DRIFT_INTERVALS; + begin_short_intervals = 1; } - diff_time = (curr + hr_correction - hr_init_time) / 1000; - } else if (diffdiff < -10000) { - SysHrTime corr = (curr - hr_last_time) / 100; - if (corr / 1000 >= -diffdiff) { - ++done; - hr_correction -= ((SysHrTime)diffdiff) * 1000; - } else { - hr_correction += corr; + else { + if (ddp->dirty_counter <= 0) { + drift_adj = ((stime_acc - mtime_acc) + *ERTS_MONOTONIC_TIME_UNIT) / mtime_acc; + } + if (ddp->dirty_counter >= 0) { + if (ddp->dirty_counter == 0) { + /* Force set new drift correction... */ + set_new_correction = 1; + } + ddp->dirty_counter--; + } + drift_adj_diff = drift_adj - new_correction.drift; + if (drift_adj_diff) { + if (drift_adj_diff > ERTS_TIME_DRIFT_MAX_ADJ_DIFF) + drift_adj_diff = ERTS_TIME_DRIFT_MAX_ADJ_DIFF; + else if (drift_adj_diff < -ERTS_TIME_DRIFT_MAX_ADJ_DIFF) + drift_adj_diff = -ERTS_TIME_DRIFT_MAX_ADJ_DIFF; + new_correction.drift += drift_adj_diff; + + if (drift_adj_diff < -ERTS_TIME_DRIFT_MIN_ADJ_DIFF + || ERTS_TIME_DRIFT_MIN_ADJ_DIFF < drift_adj_diff) { + set_new_correction = 1; + } + } } - diff_time = (curr + hr_correction - hr_init_time) / 1000; - } else { - ++done; } - if (done) { - hr_last_correction_check = curr; + } +#endif + + begin_short_intervals |= set_new_correction; + + if (begin_short_intervals) { + time_sup.inf.c.parmon.cdata.short_check_interval + = ERTS_INIT_SHORT_INTERVAL_COUNTER; + } + else if ((os_mtime - time_sup.inf.c.parmon.cdata.last_check + >= ERTS_SHORT_TIME_CORRECTION_CHECK - ERTS_MONOTONIC_TIME_UNIT) + && time_sup.inf.c.parmon.cdata.short_check_interval > 0) { + time_sup.inf.c.parmon.cdata.short_check_interval--; + } + time_sup.inf.c.parmon.cdata.last_check = os_mtime; + + if (new_correction.error == 0) + timeout = ERTS_MONOTONIC_TO_MSEC(ERTS_LONG_TIME_CORRECTION_CHECK); + else { + ErtsMonotonicTime ecorr = new_correction.error; + if (sdiff < 0) + sdiff = -1*sdiff; + if (ecorr < 0) + ecorr = -1*ecorr; + if (sdiff > ecorr*(ERTS_LONG_TIME_CORRECTION_CHECK/ERTS_TCORR_ERR_UNIT)) + timeout = ERTS_MONOTONIC_TO_MSEC(ERTS_LONG_TIME_CORRECTION_CHECK); + else { + timeout = ERTS_MONOTONIC_TO_MSEC((ERTS_TCORR_ERR_UNIT*sdiff)/ecorr); + if (timeout < 10) + timeout = 10; } } - tv->tv_sec += (int) (diff_time / ((SysHrTime) 1000000)); - tv->tv_usec += (int) (diff_time % ((SysHrTime) 1000000)); - if (tv->tv_usec >= 1000000) { - tv->tv_usec -= 1000000; - tv->tv_sec += 1; + + if (timeout > ERTS_MONOTONIC_TO_MSEC(ERTS_SHORT_TIME_CORRECTION_CHECK) + && time_sup.inf.c.parmon.cdata.short_check_interval) { + timeout = ERTS_MONOTONIC_TO_MSEC(ERTS_SHORT_TIME_CORRECTION_CHECK); + } + + ERTS_PRINT_CORRECTION; + + if (set_new_correction) { + erts_smp_rwmtx_rwlock(&time_sup.inf.c.parmon.rwmtx); + + os_mtime = erts_os_monotonic_time(); + + /* Save previous correction instance */ + time_sup.inf.c.parmon.cdata.prev = *cip; + + /* + * Current correction instance begin when + * OS monotonic time has increased one unit. + */ + os_mtime++; + + /* + * Erlang monotonic time corresponding to + * next OS monotonic time using previous + * correction. + */ + erl_mtime = calc_corrected_erl_mtime(os_mtime, cip, NULL); + + /* + * Save new current correction instance. + */ + time_sup.inf.c.parmon.cdata.curr.erl_mtime = erl_mtime; + time_sup.inf.c.parmon.cdata.curr.os_mtime = os_mtime; + time_sup.inf.c.parmon.cdata.curr.correction = new_correction; + + erts_smp_rwmtx_rwunlock(&time_sup.inf.c.parmon.rwmtx); } - hr_last_time = curr; + + erts_set_timer(&time_sup.inf.c.parmon.timer, + check_time_correction, + NULL, + NULL, + timeout); + +#undef ERTS_PRINT_CORRECTION } -#define correction (hr_correction/1000000) +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC -#else /* !HAVE_GETHRTIME */ -#if !defined(CORRECT_USING_TIMES) -#define init_tolerant_timeofday() -#define get_tolerant_timeofday(tvp) sys_gettimeofday(tvp) -#else +static void +init_check_time_correction(void *unused) +{ + ErtsMonotonicDriftData *ddp; + ErtsMonotonicTime old_mtime, old_stime, mtime, stime, mtime_diff, + stime_diff; + int ix; + + ddp = &time_sup.inf.c.parmon.cdata.drift; + ix = ddp->ix; + old_mtime = ddp->intervals[0].time.mon; + old_stime = ddp->intervals[0].time.sys; + + erts_os_times(&mtime, &stime); + + mtime_diff = mtime - old_mtime; + stime_diff = stime - old_stime; + if (100*stime_diff < 80*mtime_diff || 120*mtime_diff < 100*stime_diff ) { + /* Had a system time leap... pretend no drift... */ + stime_diff = mtime_diff; + } + + /* + * We use old time values in order to trigger + * a drift adjustment, and repeat this interval + * in all slots... + */ + for (ix = 0; ix < ERTS_DRIFT_INTERVALS; ix++) { + ddp->intervals[ix].diff.mon = mtime_diff; + ddp->intervals[ix].diff.sys = stime_diff; + ddp->intervals[ix].time.mon = old_mtime; + ddp->intervals[ix].time.sys = old_stime; + } -typedef Sint64 Milli; - -static clock_t init_ct; -static Sint64 ct_wrap; -static Milli init_tv_m; -static Milli correction_supress; -static Milli last_ct_diff; -static Milli last_cc; -static clock_t last_ct; - -/* sys_times() might need to be wrapped and the values shifted (right) - a bit to cope with newer linux (2.5.*) kernels, this has to be taken care - of dynamically to start with, a special version that uses - the times() return value as a high resolution timer can be made - to fully utilize the faster ticks, like on windows, but for now, we'll - settle with this silly workaround */ -#ifdef ERTS_WRAP_SYS_TIMES -#define KERNEL_TICKS() (sys_times_wrap() & \ - ((1UL << ((sizeof(clock_t) * 8) - 1)) - 1)) -#else -SysTimes dummy_tms; + ddp->acc.sys = stime_diff*ERTS_DRIFT_INTERVALS; + ddp->acc.mon = mtime_diff*ERTS_DRIFT_INTERVALS; + ddp->ix = 0; + ddp->dirty_counter = ERTS_DRIFT_INTERVALS; -#define KERNEL_TICKS() (sys_times(&dummy_tms) & \ - ((1UL << ((sizeof(clock_t) * 8) - 1)) - 1)) + check_time_correction(NULL); +} #endif -static void init_tolerant_timeofday(void) +static ErtsMonotonicTime +finalize_corrected_time_offset(ErtsSystemTime *stimep) { - last_ct = init_ct = KERNEL_TICKS(); - last_cc = 0; - init_tv_m = (((Milli) inittv.tv_sec) * 1000) + - (inittv.tv_usec / 1000); - ct_wrap = 0; - correction_supress = 0; -} + ErtsMonotonicTime os_mtime; + ErtsMonotonicCorrectionData cdata; + ErtsMonotonicCorrectionInstance *cip; + erts_smp_rwmtx_rlock(&time_sup.inf.c.parmon.rwmtx); + + erts_os_times(&os_mtime, stimep); + + cdata = time_sup.inf.c.parmon.cdata; + + erts_smp_rwmtx_runlock(&time_sup.inf.c.parmon.rwmtx); + + if (os_mtime < cdata.curr.os_mtime) + erl_exit(ERTS_ABORT_EXIT, + "OS monotonic time stepped backwards\n"); + cip = &cdata.curr; + + return calc_corrected_erl_mtime(os_mtime, cip, NULL); +} -static void get_tolerant_timeofday(SysTimeval *tvp) +static void +late_init_time_correction(void) { - clock_t current_ct; - SysTimeval current_tv; - Milli ct_diff; - Milli tv_diff; - Milli current_correction; - Milli act_correction; /* long shown to be too small */ - Milli max_adjust; - - if (erts_disable_tolerant_timeofday) { - sys_gettimeofday(tvp); - return; - } + if (time_sup.inf.c.finalized_offset) { -#ifdef ERTS_WRAP_SYS_TIMES -#define TICK_MS (1000 / SYS_CLK_TCK_WRAP) + erts_init_timer(&time_sup.inf.c.parmon.timer); + erts_set_timer(&time_sup.inf.c.parmon.timer, +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + init_check_time_correction, #else -#define TICK_MS (1000 / SYS_CLK_TCK) + check_time_correction, #endif - current_ct = KERNEL_TICKS(); - sys_gettimeofday(¤t_tv); - - /* I dont know if uptime can move some units backwards - on some systems, but I allow for small backward - jumps to avoid such problems if they exist...*/ - if (last_ct > 100 && current_ct < (last_ct - 100)) { - ct_wrap += ((Sint64) 1) << ((sizeof(clock_t) * 8) - 1); + NULL, + NULL, + ERTS_MONOTONIC_TO_MSEC(ERTS_SHORT_TIME_CORRECTION_CHECK)); } - last_ct = current_ct; - ct_diff = ((ct_wrap + current_ct) - init_ct) * TICK_MS; +} - /* - * We will adjust the time in milliseconds and we allow for 1% - * adjustments, but if this function is called more often then every 100 - * millisecond (which is obviously possible), we will never adjust, so - * we accumulate small times by setting last_ct_diff iff max_adjust > 0 - */ - if ((max_adjust = (ct_diff - last_ct_diff)/100) > 0) - last_ct_diff = ct_diff; +#endif /* ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT */ - tv_diff = ((((Milli) current_tv.tv_sec) * 1000) + - (current_tv.tv_usec / 1000)) - init_tv_m; +static ErtsMonotonicTime get_not_corrected_time(void) +{ + ErtsMonotonicTime stime, mtime; - current_correction = ((ct_diff - tv_diff) / TICK_MS) * TICK_MS; /* trunc */ + erts_smp_mtx_lock(&erts_get_time_mtx); - /* - * We allow the current_correction value to wobble a little, as it - * suffers from the low resolution of the kernel ticks. - * if it hasn't changed more than one tick in either direction, - * we will keep the old value. - */ - if ((last_cc > current_correction + TICK_MS) || - (last_cc < current_correction - TICK_MS)) { - last_cc = current_correction; - } else { - current_correction = last_cc; - } - - /* - * As time goes, we try to get the actual correction to 0, - * that is, make erlangs time correspond to the systems dito. - * The act correction is what we seem to need (current_correction) - * minus the correction suppression. The correction supression - * will change slowly (max 1% of elapsed time) but in millisecond steps. - */ - act_correction = current_correction - correction_supress; - if (max_adjust > 0) { - /* - * Here we slowly adjust erlangs time to correspond with the - * system time by changing the correction_supress variable. - * It can change max_adjust milliseconds which is 1% of elapsed time - */ - if (act_correction > 0) { - if (current_correction - correction_supress > max_adjust) { - correction_supress += max_adjust; - } else { - correction_supress = current_correction; - } - act_correction = current_correction - correction_supress; - } else if (act_correction < 0) { - if (correction_supress - current_correction > max_adjust) { - correction_supress -= max_adjust; - } else { - correction_supress = current_correction; + stime = erts_os_system_time(); + + mtime = stime - time_sup.inf.c.not_corrected_moffset; + + if (mtime >= time_sup.f.c.last_not_corrected_time) + time_sup.f.c.last_not_corrected_time = mtime; + else { + mtime = time_sup.f.c.last_not_corrected_time; + + if (time_sup.r.o.warp_mode == ERTS_MULTI_TIME_WARP_MODE) { + ErtsMonotonicTime new_offset = stime - mtime; + new_offset = ERTS_MONOTONIC_TO_USEC(new_offset); + new_offset = ERTS_USEC_TO_MONOTONIC(new_offset); + if (time_sup.inf.c.not_corrected_moffset != new_offset) { + time_sup.inf.c.not_corrected_moffset = new_offset; + set_time_offset(new_offset); + schedule_send_time_offset_changed_notifications(new_offset); } - act_correction = current_correction - correction_supress; } + } - /* - * The actual correction will correct the timeval so that system - * time warps gets smothed down. - */ - current_tv.tv_sec += act_correction / 1000; - current_tv.tv_usec += (act_correction % 1000) * 1000; - - if (current_tv.tv_usec >= 1000000) { - ++current_tv.tv_sec ; - current_tv.tv_usec -= 1000000; - } else if (current_tv.tv_usec < 0) { - --current_tv.tv_sec; - current_tv.tv_usec += 1000000; - } - *tvp = current_tv; -#undef TICK_MS + + ASSERT(stime == mtime + time_sup.inf.c.not_corrected_moffset); + + erts_smp_mtx_unlock(&erts_get_time_mtx); + + return mtime; } -#endif /* CORRECT_USING_TIMES */ -#endif /* !HAVE_GETHRTIME */ +int erts_check_time_adj_support(int time_correction, + ErtsTimeWarpMode time_warp_mode) +{ + if (!time_correction) + return 1; + + /* User wants time correction */ -/* -** Why this? Well, most platforms have a constant clock resolution of 1, -** we dont want the deliver_time/time_remaining routines to waste -** time dividing and multiplying by/with a variable that's always one. -** so the return value of sys_init_time is ignored on those platforms. -*/ - -#ifndef SYS_CLOCK_RESOLUTION -static int clock_resolution; -#define CLOCK_RESOLUTION clock_resolution +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + return !time_sup.r.o.os_monotonic_time_disable; #else -#define CLOCK_RESOLUTION SYS_CLOCK_RESOLUTION + return 0; #endif +} -/* -** The clock resolution should really be the resolution of the -** time function in use, which on most platforms -** is 1. On VxWorks the resolution should be -** the number of ticks per second (or 1, which would work nicely to). -** -** Setting lower resolutions is mostly interesting when timers are used -** instead of something like select. -*/ - -static SysTimeval last_delivered; - -static void init_erts_deliver_time(const SysTimeval *inittv) +int +erts_has_time_correction(void) { - /* We set the initial values for deliver_time here */ - last_delivered = *inittv; - last_delivered.tv_usec = 1000 * (last_delivered.tv_usec / 1000); - /* ms resolution */ + return time_sup.r.o.correction; } -static void do_erts_deliver_time(const SysTimeval *current) +void erts_init_sys_time_sup(void) { - SysTimeval cur_time; - erts_time_t elapsed; - - /* calculate and deliver appropriate number of ticks */ - cur_time = *current; - cur_time.tv_usec = 1000 * (cur_time.tv_usec / 1000); /* ms resolution */ - elapsed = (1000 * (cur_time.tv_sec - last_delivered.tv_sec) + - (cur_time.tv_usec - last_delivered.tv_usec) / 1000) / - CLOCK_RESOLUTION; + ErtsSysInitTimeResult sys_init_time_res + = ERTS_SYS_INIT_TIME_RESULT_INITER; - /* Sometimes the time jump backwards, - resulting in a negative elapsed time. We compensate for - this by simply pretend as if the time stood still. :) */ + sys_init_time(&sys_init_time_res); - if (elapsed > 0) { + erts_time_sup__.r.o.monotonic_time_unit + = sys_init_time_res.os_monotonic_time_unit; - ASSERT(elapsed < ((erts_time_t) ERTS_SHORT_TIME_T_MAX)); +#ifndef SYS_CLOCK_RESOLUTION + erts_time_sup__.r.o.clktck_resolution + = sys_init_time_res.sys_clock_resolution; + erts_time_sup__.r.o.clktck_resolution *= 1000; +#endif - erts_do_time_add((erts_short_time_t) elapsed); - last_delivered = cur_time; - } +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + time_sup.r.o.os_monotonic_time_disable + = !sys_init_time_res.have_os_monotonic_time; + time_sup.r.o.os_monotonic_time_func + = sys_init_time_res.os_monotonic_time_info.func; + time_sup.r.o.os_monotonic_time_clock_id + = sys_init_time_res.os_monotonic_time_info.clock_id; + time_sup.r.o.os_monotonic_time_locked + = sys_init_time_res.os_monotonic_time_info.locked_use; + time_sup.r.o.os_monotonic_time_resolution + = sys_init_time_res.os_monotonic_time_info.resolution; + time_sup.r.o.os_monotonic_time_extended + = sys_init_time_res.os_monotonic_time_info.extended; +#endif + time_sup.r.o.os_system_time_func + = sys_init_time_res.os_system_time_info.func; + time_sup.r.o.os_system_time_clock_id + = sys_init_time_res.os_system_time_info.clock_id; + time_sup.r.o.os_system_time_locked + = sys_init_time_res.os_system_time_info.locked_use; + time_sup.r.o.os_system_time_resolution + = sys_init_time_res.os_system_time_info.resolution; } int -erts_init_time_sup(void) +erts_init_time_sup(int time_correction, ErtsTimeWarpMode time_warp_mode) { + ErtsMonotonicTime resolution; +#if !ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + ErtsMonotonicTime abs_native_offset, native_offset; +#endif + + ASSERT(ERTS_MONOTONIC_TIME_MIN < ERTS_MONOTONIC_TIME_MAX); + erts_smp_mtx_init(&erts_timeofday_mtx, "timeofday"); + erts_smp_mtx_init(&erts_get_time_mtx, "get_time"); - init_approx_time(); + time_sup.r.o.correction = time_correction; + time_sup.r.o.warp_mode = time_warp_mode; + + if (time_warp_mode == ERTS_SINGLE_TIME_WARP_MODE) + time_sup.inf.c.finalized_offset = 0; + else + time_sup.inf.c.finalized_offset = ~0; + +#if !ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + +#ifdef ARCH_32 + time_sup.r.o.start = ((((ErtsMonotonicTime) 1) << 32)-1); + time_sup.r.o.start /= ERTS_MONOTONIC_TIME_UNIT; + time_sup.r.o.start *= ERTS_MONOTONIC_TIME_UNIT; + time_sup.r.o.start += ERTS_MONOTONIC_TIME_UNIT; + native_offset = time_sup.r.o.start - ERTS_MONOTONIC_TIME_UNIT; + native_offset = native_offset; +#else /* ARCH_64 */ + if (ERTS_MONOTONIC_TIME_UNIT <= 10*1000*1000) { + time_sup.r.o.start = 0; + native_offset = -ERTS_MONOTONIC_TIME_UNIT; + abs_native_offset = ERTS_MONOTONIC_TIME_UNIT; + } + else { + time_sup.r.o.start = ((ErtsMonotonicTime) MIN_SMALL); + time_sup.r.o.start /= ERTS_MONOTONIC_TIME_UNIT; + time_sup.r.o.start *= ERTS_MONOTONIC_TIME_UNIT; + native_offset = time_sup.r.o.start - ERTS_MONOTONIC_TIME_UNIT; + abs_native_offset = -1*native_offset; + } +#endif - last_emu_time.tv_sec = 0; - last_emu_time.tv_usec = 0; + time_sup.r.o.start_offset.native = (time_sup.r.o.start + - ERTS_MONOTONIC_TIME_UNIT); + time_sup.r.o.start_offset.nsec = (ErtsMonotonicTime) + erts_time_unit_conversion((Uint64) abs_native_offset, + (Uint32) ERTS_MONOTONIC_TIME_UNIT, + (Uint32) 1000*1000*1000); + time_sup.r.o.start_offset.usec = (ErtsMonotonicTime) + erts_time_unit_conversion((Uint64) abs_native_offset, + (Uint32) ERTS_MONOTONIC_TIME_UNIT, + (Uint32) 1000*1000); + time_sup.r.o.start_offset.msec = (ErtsMonotonicTime) + erts_time_unit_conversion((Uint64) abs_native_offset, + (Uint32) ERTS_MONOTONIC_TIME_UNIT, + (Uint32) 1000); + time_sup.r.o.start_offset.sec = (ErtsMonotonicTime) + erts_time_unit_conversion((Uint64) abs_native_offset, + (Uint32) ERTS_MONOTONIC_TIME_UNIT, + (Uint32) 1); + if (native_offset < 0) { + time_sup.r.o.start_offset.nsec *= -1; + time_sup.r.o.start_offset.usec *= -1; + time_sup.r.o.start_offset.msec *= -1; + time_sup.r.o.start_offset.sec *= -1; + } -#ifndef SYS_CLOCK_RESOLUTION - clock_resolution = sys_init_time(); -#else - (void) sys_init_time(); #endif - sys_gettimeofday(&inittv); + + resolution = time_sup.r.o.os_system_time_resolution; +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + if (resolution > time_sup.r.o.os_monotonic_time_resolution) + resolution = time_sup.r.o.os_monotonic_time_resolution; +#endif + + time_sup.r.o.adj.large_diff = erts_time_sup__.r.o.monotonic_time_unit; + time_sup.r.o.adj.large_diff *= 50; + time_sup.r.o.adj.large_diff /= resolution; + if (time_sup.r.o.adj.large_diff < ERTS_USEC_TO_MONOTONIC(500)) + time_sup.r.o.adj.large_diff = ERTS_USEC_TO_MONOTONIC(500); + time_sup.r.o.adj.small_diff = time_sup.r.o.adj.large_diff/10; + +#ifdef ERTS_TIME_CORRECTION_PRINT + fprintf(stderr, "start = %lld\n\r", (long long) ERTS_MONOTONIC_TIME_START); + fprintf(stderr, "native offset = %lld\n\r", (long long) ERTS_MONOTONIC_OFFSET_NATIVE); + fprintf(stderr, "nsec offset = %lld\n\r", (long long) ERTS_MONOTONIC_OFFSET_NSEC); + fprintf(stderr, "usec offset = %lld\n\r", (long long) ERTS_MONOTONIC_OFFSET_USEC); + fprintf(stderr, "msec offset = %lld\n\r", (long long) ERTS_MONOTONIC_OFFSET_MSEC); + fprintf(stderr, "sec offset = %lld\n\r", (long long) ERTS_MONOTONIC_OFFSET_SEC); + fprintf(stderr, "large diff = %lld usec\r\n", + (long long) ERTS_MONOTONIC_TO_USEC(time_sup.r.o.adj.large_diff)); + fprintf(stderr, "small diff = %lld usec\r\n", + (long long) ERTS_MONOTONIC_TO_USEC(time_sup.r.o.adj.small_diff)); +#endif + + if (ERTS_MONOTONIC_TIME_UNIT < ERTS_CLKTCK_RESOLUTION) + ERTS_INTERNAL_ERROR("Too small monotonic time time unit"); + +#ifndef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + time_sup.r.o.correction = 0; +#else + if (time_sup.r.o.os_monotonic_time_disable) + time_sup.r.o.correction = 0; + + if (time_sup.r.o.correction) { + ErtsMonotonicCorrectionData *cdatap; + erts_smp_rwmtx_opt_t rwmtx_opts = ERTS_SMP_RWMTX_OPT_DEFAULT_INITER; + ErtsMonotonicTime offset; + erts_os_times(&time_sup.inf.c.minit, + &time_sup.inf.c.sinit); + time_sup.r.o.moffset = -1*time_sup.inf.c.minit; + offset = time_sup.inf.c.sinit; + offset -= ERTS_MONOTONIC_TIME_UNIT; + init_time_offset(offset); + + rwmtx_opts.type = ERTS_SMP_RWMTX_TYPE_EXTREMELY_FREQUENT_READ; + rwmtx_opts.lived = ERTS_SMP_RWMTX_LONG_LIVED; + + erts_smp_rwmtx_init_opt(&time_sup.inf.c.parmon.rwmtx, + &rwmtx_opts, "get_corrected_time"); + + cdatap = &time_sup.inf.c.parmon.cdata; -#ifdef HAVE_GETHRTIME - sys_init_hrtime(); +#ifndef ERTS_HAVE_CORRECTED_OS_MONOTONIC + cdatap->drift.intervals[0].time.sys = time_sup.inf.c.sinit; + cdatap->drift.intervals[0].time.mon = time_sup.inf.c.minit; + cdatap->curr.correction.drift = 0; #endif - init_tolerant_timeofday(); + cdatap->curr.correction.error = 0; + cdatap->curr.erl_mtime = ERTS_MONOTONIC_TIME_UNIT; + cdatap->curr.os_mtime = time_sup.inf.c.minit; + cdatap->last_check = time_sup.inf.c.minit; + cdatap->short_check_interval = ERTS_INIT_SHORT_INTERVAL_COUNTER; + cdatap->prev = cdatap->curr; + + time_sup.r.o.get_time = get_corrected_time; + } + else +#endif + { + ErtsMonotonicTime stime, offset; + time_sup.r.o.get_time = get_not_corrected_time; + stime = time_sup.inf.c.sinit = erts_os_system_time(); + offset = stime - ERTS_MONOTONIC_TIME_UNIT; + time_sup.inf.c.not_corrected_moffset = offset; + init_time_offset(offset); + time_sup.f.c.last_not_corrected_time = 0; + } - init_erts_deliver_time(&inittv); - gtv = inittv; - then.tv_sec = then.tv_usec = 0; + prev_wall_clock_elapsed = 0; - erts_deliver_time(); + previous_now = ERTS_MONOTONIC_TO_USEC(get_time_offset()); - return CLOCK_RESOLUTION; +#ifdef DEBUG + time_sup_initialized = 1; +#endif + + return ERTS_CLKTCK_RESOLUTION/1000; } + +void +erts_late_init_time_sup(void) +{ +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + /* Timer wheel must be initialized */ + if (time_sup.r.o.get_time == get_corrected_time) + late_init_time_correction(); +#endif + erts_late_sys_init_time(); +} + +ErtsTimeWarpMode erts_time_warp_mode(void) +{ + return time_sup.r.o.warp_mode; +} + +ErtsTimeOffsetState erts_time_offset_state(void) +{ + switch (time_sup.r.o.warp_mode) { + case ERTS_NO_TIME_WARP_MODE: + return ERTS_TIME_OFFSET_FINAL; + case ERTS_SINGLE_TIME_WARP_MODE: + if (time_sup.inf.c.finalized_offset) + return ERTS_TIME_OFFSET_FINAL; + return ERTS_TIME_OFFSET_PRELIMINARY; + case ERTS_MULTI_TIME_WARP_MODE: + return ERTS_TIME_OFFSET_VOLATILE; + default: + ERTS_INTERNAL_ERROR("Invalid time warp mode"); + return ERTS_TIME_OFFSET_VOLATILE; + } +} + +/* + * erts_finalize_time_offset() will only change time offset + * the first time it is called when the emulator has been + * started in "single time warp" mode. Returns previous + * state: + * * ERTS_TIME_OFFSET_PRELIMINARY - Finalization performed + * * ERTS_TIME_OFFSET_FINAL - Already finialized; nothing changed + * * ERTS_TIME_OFFSET_VOLATILE - Not supported, either in + * * no correction mode (or multi time warp mode; not yet implemented). + */ + +ErtsTimeOffsetState +erts_finalize_time_offset(void) +{ + switch (time_sup.r.o.warp_mode) { + case ERTS_NO_TIME_WARP_MODE: + return ERTS_TIME_OFFSET_FINAL; + case ERTS_MULTI_TIME_WARP_MODE: + return ERTS_TIME_OFFSET_VOLATILE; + case ERTS_SINGLE_TIME_WARP_MODE: { + ErtsTimeOffsetState res = ERTS_TIME_OFFSET_FINAL; + + erts_smp_mtx_lock(&erts_get_time_mtx); + + if (!time_sup.inf.c.finalized_offset) { + ErtsMonotonicTime mtime, new_offset; + +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + if (!time_sup.r.o.correction) +#endif + { + ErtsMonotonicTime stime = erts_os_system_time(); + + mtime = stime - time_sup.inf.c.not_corrected_moffset; + + if (mtime >= time_sup.f.c.last_not_corrected_time) { + time_sup.f.c.last_not_corrected_time = mtime; + new_offset = time_sup.inf.c.not_corrected_moffset; + } + else { + mtime = time_sup.f.c.last_not_corrected_time; + + ASSERT(time_sup.inf.c.not_corrected_moffset != stime - mtime); + new_offset = stime - mtime; + time_sup.inf.c.not_corrected_moffset = new_offset; + } + + } +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + else { + ErtsSystemTime stime; + mtime = finalize_corrected_time_offset(&stime); + new_offset = stime - mtime; + } +#endif + new_offset = ERTS_MONOTONIC_TO_USEC(new_offset); + new_offset = ERTS_USEC_TO_MONOTONIC(new_offset); + + set_time_offset(new_offset); + schedule_send_time_offset_changed_notifications(new_offset); + + time_sup.inf.c.finalized_offset = ~0; + res = ERTS_TIME_OFFSET_PRELIMINARY; + } + + erts_smp_mtx_unlock(&erts_get_time_mtx); + +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + if (res == ERTS_TIME_OFFSET_PRELIMINARY + && time_sup.r.o.get_time == get_corrected_time) { + late_init_time_correction(); + } +#endif + + return res; + } + default: + ERTS_INTERNAL_ERROR("Invalid time warp mode"); + return ERTS_TIME_OFFSET_VOLATILE; + } +} + /* info functions */ void @@ -498,23 +1171,16 @@ elapsed_time_both(UWord *ms_user, UWord *ms_sys, void wall_clock_elapsed_time_both(UWord *ms_total, UWord *ms_diff) { - UWord prev_total; - SysTimeval tv; + ErtsMonotonicTime now, elapsed; erts_smp_mtx_lock(&erts_timeofday_mtx); - get_tolerant_timeofday(&tv); + now = time_sup.r.o.get_time(); - *ms_total = 1000 * (tv.tv_sec - inittv.tv_sec) + - (tv.tv_usec - inittv.tv_usec) / 1000; - - prev_total = 1000 * (gtv.tv_sec - inittv.tv_sec) + - (gtv.tv_usec - inittv.tv_usec) / 1000; - *ms_diff = *ms_total - prev_total; - gtv = tv; - - /* must sync the machine's idea of time here */ - do_erts_deliver_time(&tv); + elapsed = ERTS_MONOTONIC_TO_MSEC(now); + *ms_total = (UWord) elapsed; + *ms_diff = (UWord) (elapsed - prev_wall_clock_elapsed); + prev_wall_clock_elapsed = elapsed; erts_smp_mtx_unlock(&erts_timeofday_mtx); } @@ -890,146 +1556,555 @@ univ_to_local(Sint *year, Sint *month, Sint *day, return 0; } - /* get a timestamp */ void get_now(Uint* megasec, Uint* sec, Uint* microsec) { - SysTimeval now; + ErtsMonotonicTime now_megasec, now_sec, now, mtime, time_offset; + mtime = time_sup.r.o.get_time(); + time_offset = get_time_offset(); + now = ERTS_MONOTONIC_TO_USEC(mtime + time_offset); + erts_smp_mtx_lock(&erts_timeofday_mtx); - - get_tolerant_timeofday(&now); - do_erts_deliver_time(&now); - - /* Make sure time is later than last */ - if (then.tv_sec > now.tv_sec || - (then.tv_sec == now.tv_sec && then.tv_usec >= now.tv_usec)) { - now = then; - now.tv_usec++; - } - /* Check for carry from above + general reasonability */ - if (now.tv_usec >= 1000000) { - now.tv_usec = 0; - now.tv_sec++; - } - then = now; + + /* Make sure now time is later than last time */ + if (now <= previous_now) + now = previous_now + 1; + + previous_now = now; erts_smp_mtx_unlock(&erts_timeofday_mtx); - - *megasec = (Uint) (now.tv_sec / 1000000); - *sec = (Uint) (now.tv_sec % 1000000); - *microsec = (Uint) (now.tv_usec); - update_approx_time(&now); + now_megasec = now / ERTS_MONOTONIC_TIME_TERA; + now_sec = now / ERTS_MONOTONIC_TIME_MEGA; + *megasec = (Uint) now_megasec; + *sec = (Uint) (now_sec - now_megasec*ERTS_MONOTONIC_TIME_MEGA); + *microsec = (Uint) (now - now_sec*ERTS_MONOTONIC_TIME_MEGA); + + ASSERT(((ErtsMonotonicTime) *megasec)*ERTS_MONOTONIC_TIME_TERA + + ((ErtsMonotonicTime) *sec)*ERTS_MONOTONIC_TIME_MEGA + + ((ErtsMonotonicTime) *microsec) == now); +} + +ErtsMonotonicTime +erts_get_monotonic_time(void) +{ + return time_sup.r.o.get_time(); } void get_sys_now(Uint* megasec, Uint* sec, Uint* microsec) { - SysTimeval now; - - sys_gettimeofday(&now); - - *megasec = (Uint) (now.tv_sec / 1000000); - *sec = (Uint) (now.tv_sec % 1000000); - *microsec = (Uint) (now.tv_usec); + ErtsSystemTime stime = erts_os_system_time(); + ErtsSystemTime ms, s, us; + + us = ERTS_MONOTONIC_TO_USEC(stime); + s = us / (1000*1000); + ms = s / (1000*1000); - update_approx_time(&now); + *megasec = (Uint) ms; + *sec = (Uint) (s - ms*(1000*1000)); + *microsec = (Uint) (us - s*(1000*1000)); } +#ifdef HAVE_ERTS_NOW_CPU +void erts_get_now_cpu(Uint* megasec, Uint* sec, Uint* microsec) { + SysCpuTime t; + SysTimespec tp; + + sys_get_proc_cputime(t, tp); + *microsec = (Uint)(tp.tv_nsec / 1000); + t = (tp.tv_sec / 1000000); + *megasec = (Uint)(t % 1000000); + *sec = (Uint)(tp.tv_sec % 1000000); +} +#endif -/* deliver elapsed *ticks* to the machine - takes a pointer - to a struct timeval representing current time (to save - a gettimeofday() where possible) or NULL */ +#include "big.h" -void erts_deliver_time(void) { - SysTimeval now; - - erts_smp_mtx_lock(&erts_timeofday_mtx); - - get_tolerant_timeofday(&now); - do_erts_deliver_time(&now); - - erts_smp_mtx_unlock(&erts_timeofday_mtx); +void +erts_monitor_time_offset(Eterm id, Eterm ref) +{ + erts_smp_mtx_lock(&erts_get_time_mtx); + erts_add_monitor(&time_offset_monitors, MON_TIME_OFFSET, ref, id, NIL); + no_time_offset_monitors++; + erts_smp_mtx_unlock(&erts_get_time_mtx); +} - update_approx_time(&now); +int +erts_demonitor_time_offset(Eterm ref) +{ + int res; + ErtsMonitor *mon; + ASSERT(is_internal_ref(ref)); + erts_smp_mtx_lock(&erts_get_time_mtx); + mon = erts_remove_monitor(&time_offset_monitors, ref); + if (!mon) + res = 0; + else { + ASSERT(no_time_offset_monitors > 0); + no_time_offset_monitors--; + res = 1; + } + erts_smp_mtx_unlock(&erts_get_time_mtx); + if (res) + erts_destroy_monitor(mon); + return res; } -/* get *real* time (not ticks) remaining until next timeout - if there - isn't one, give a "long" time, that is guaranteed - to not cause overflow when we report elapsed time later on */ +typedef struct { + Eterm pid; + Eterm ref; + Eterm heap[REF_THING_SIZE]; +} ErtsTimeOffsetMonitorInfo; -void erts_time_remaining(SysTimeval *rem_time) +typedef struct { + Uint ix; + ErtsTimeOffsetMonitorInfo *to_mon_info; +} ErtsTimeOffsetMonitorContext; + +static void +save_time_offset_monitor(ErtsMonitor *mon, void *vcntxt) { - erts_time_t ticks; - SysTimeval cur_time; - erts_time_t elapsed; - - /* erts_next_time() returns no of ticks to next timeout or -1 if none */ - - ticks = (erts_time_t) erts_next_time(); - if (ticks == (erts_time_t) -1) { - /* timer queue empty */ - /* this will cause at most 100000000 ticks */ - rem_time->tv_sec = 100000; - rem_time->tv_usec = 0; - } else { - /* next timeout after ticks ticks */ - ticks *= CLOCK_RESOLUTION; - - erts_smp_mtx_lock(&erts_timeofday_mtx); - - get_tolerant_timeofday(&cur_time); - cur_time.tv_usec = 1000 * - (cur_time.tv_usec / 1000);/* ms resolution*/ - elapsed = 1000 * (cur_time.tv_sec - last_delivered.tv_sec) + - (cur_time.tv_usec - last_delivered.tv_usec) / 1000; - - erts_smp_mtx_unlock(&erts_timeofday_mtx); + ErtsTimeOffsetMonitorContext *cntxt; + Eterm *from_hp, *to_hp; + Uint mix; + int hix; + + cntxt = (ErtsTimeOffsetMonitorContext *) vcntxt; + mix = (cntxt->ix)++; + cntxt->to_mon_info[mix].pid = mon->pid; + to_hp = &cntxt->to_mon_info[mix].heap[0]; + + ASSERT(is_internal_ref(mon->ref)); + from_hp = internal_ref_val(mon->ref); + ASSERT(thing_arityval(*from_hp) + 1 == REF_THING_SIZE); + + for (hix = 0; hix < REF_THING_SIZE; hix++) + to_hp[hix] = from_hp[hix]; + + cntxt->to_mon_info[mix].ref + = make_internal_ref(&cntxt->to_mon_info[mix].heap[0]); + +} + +static void +send_time_offset_changed_notifications(void *new_offsetp) +{ + ErtsMonotonicTime new_offset; + ErtsTimeOffsetMonitorInfo *to_mon_info = NULL; /* Shut up faulty warning */ + Uint no_monitors; + char *tmp = NULL; + +#ifdef ARCH_64 + new_offset = (ErtsMonotonicTime) new_offsetp; +#else + new_offset = *((ErtsMonotonicTime *) new_offsetp); + erts_free(ERTS_ALC_T_NEW_TIME_OFFSET, new_offsetp); +#endif + new_offset -= ERTS_MONOTONIC_OFFSET_NATIVE; + + erts_smp_mtx_lock(&erts_get_time_mtx); + + no_monitors = no_time_offset_monitors; + if (no_monitors) { + ErtsTimeOffsetMonitorContext cntxt; + Uint alloc_sz; - if (ticks <= elapsed) { /* Ooops, better hurry */ - rem_time->tv_sec = rem_time->tv_usec = 0; - return; + /* Monitor info array size */ + alloc_sz = no_monitors*sizeof(ErtsTimeOffsetMonitorInfo); + /* + template max size */ + alloc_sz += 6*sizeof(Eterm); /* 5-tuple */ + alloc_sz += ERTS_MAX_SINT64_HEAP_SIZE*sizeof(Eterm); /* max offset size */ + tmp = erts_alloc(ERTS_ALC_T_TMP, alloc_sz); + + to_mon_info = (ErtsTimeOffsetMonitorInfo *) tmp; + cntxt.ix = 0; + cntxt.to_mon_info = to_mon_info; + + erts_doforall_monitors(time_offset_monitors, + save_time_offset_monitor, + &cntxt); + + ASSERT(cntxt.ix == no_monitors); + } + + erts_smp_mtx_unlock(&erts_get_time_mtx); + + if (no_monitors) { + Eterm *hp, *patch_refp, new_offset_term, message_template; + Uint mix, hsz; + + /* Make message template */ + + hp = (Eterm *) (tmp + no_monitors*sizeof(ErtsTimeOffsetMonitorInfo)); + + hsz = 6; /* 5-tuple */ + hsz += REF_THING_SIZE; + hsz += ERTS_SINT64_HEAP_SIZE(new_offset); + + if (IS_SSMALL(new_offset)) + new_offset_term = make_small(new_offset); + else + new_offset_term = erts_sint64_to_big(new_offset, &hp); + message_template = TUPLE5(hp, + am_CHANGE, + THE_NON_VALUE, /* Patch point for ref */ + am_time_offset, + am_clock_service, + new_offset_term); + patch_refp = &hp[2]; + + ASSERT(*patch_refp == THE_NON_VALUE); + + for (mix = 0; mix < no_monitors; mix++) { + Process *rp = erts_proc_lookup(to_mon_info[mix].pid); + if (rp) { + Eterm ref = to_mon_info[mix].ref; + ErtsProcLocks rp_locks = ERTS_PROC_LOCK_LINK; + erts_smp_proc_lock(rp, ERTS_PROC_LOCK_LINK); + if (erts_lookup_monitor(ERTS_P_MONITORS(rp), ref)) { + ErlHeapFragment *bp; + ErlOffHeap *ohp; + Eterm message; + + hp = erts_alloc_message_heap(hsz, &bp, &ohp, rp, &rp_locks); + *patch_refp = ref; + ASSERT(hsz == size_object(message_template)); + message = copy_struct(message_template, hsz, &hp, ohp); + erts_queue_message(rp, &rp_locks, bp, message, NIL +#ifdef USE_VM_PROBES + , NIL +#endif + ); + } + erts_smp_proc_unlock(rp, rp_locks); + } } - rem_time->tv_sec = (ticks - elapsed) / 1000; - rem_time->tv_usec = 1000 * ((ticks - elapsed) % 1000); + + erts_free(ERTS_ALC_T_TMP, tmp); } } -void erts_get_timeval(SysTimeval *tv) +static void +schedule_send_time_offset_changed_notifications(ErtsMonotonicTime new_offset) { - erts_smp_mtx_lock(&erts_timeofday_mtx); - get_tolerant_timeofday(tv); - erts_smp_mtx_unlock(&erts_timeofday_mtx); - update_approx_time(tv); +#ifdef ARCH_64 + void *new_offsetp = (void *) new_offset; + ASSERT(sizeof(void *) == sizeof(ErtsMonotonicTime)); +#else + void *new_offsetp = erts_alloc(ERTS_ALC_T_NEW_TIME_OFFSET, + sizeof(ErtsMonotonicTime)); + *((ErtsMonotonicTime *) new_offsetp) = new_offset; +#endif + erts_schedule_misc_aux_work(1, + send_time_offset_changed_notifications, + new_offsetp); } -erts_time_t -erts_get_time(void) +static ERTS_INLINE Eterm +make_time_val(Process *c_p, ErtsMonotonicTime time_val) { - SysTimeval sys_tv; - - erts_smp_mtx_lock(&erts_timeofday_mtx); - - get_tolerant_timeofday(&sys_tv); - - erts_smp_mtx_unlock(&erts_timeofday_mtx); + Sint64 val = (Sint64) time_val; + Eterm *hp; + Uint sz; - update_approx_time(&sys_tv); + if (IS_SSMALL(val)) + return make_small(val); - return sys_tv.tv_sec; + sz = ERTS_SINT64_HEAP_SIZE(val); + hp = HAlloc(c_p, sz); + return erts_sint64_to_big(val, &hp); } -#ifdef HAVE_ERTS_NOW_CPU -void erts_get_now_cpu(Uint* megasec, Uint* sec, Uint* microsec) { - SysCpuTime t; - SysTimespec tp; +Eterm +erts_get_monotonic_start_time(struct process *c_p) +{ + return make_time_val(c_p, ERTS_MONOTONIC_TIME_START); +} - sys_get_proc_cputime(t, tp); - *microsec = (Uint)(tp.tv_nsec / 1000); - t = (tp.tv_sec / 1000000); - *megasec = (Uint)(t % 1000000); - *sec = (Uint)(tp.tv_sec % 1000000); +static Eterm +bld_monotonic_time_source(Uint **hpp, Uint *szp, Sint64 os_mtime) +{ +#ifndef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + return NIL; +#else + int i = 0; + Eterm k[6]; + Eterm v[6]; + + if (time_sup.r.o.os_monotonic_time_disable) + return NIL; + + k[i] = erts_bld_atom(hpp, szp, "function"); + v[i++] = erts_bld_atom(hpp, szp, + time_sup.r.o.os_monotonic_time_func); + + if (time_sup.r.o.os_monotonic_time_clock_id) { + k[i] = erts_bld_atom(hpp, szp, "clock_id"); + v[i++] = erts_bld_atom(hpp, szp, + time_sup.r.o.os_monotonic_time_clock_id); + } + + k[i] = erts_bld_atom(hpp, szp, "resolution"); + v[i++] = erts_bld_uint64(hpp, szp, + time_sup.r.o.os_monotonic_time_resolution); + + k[i] = erts_bld_atom(hpp, szp, "extended"); + v[i++] = time_sup.r.o.os_monotonic_time_extended ? am_yes : am_no; + + k[i] = erts_bld_atom(hpp, szp, "parallel"); + v[i++] = time_sup.r.o.os_monotonic_time_locked ? am_no : am_yes; + + k[i] = erts_bld_atom(hpp, szp, "time"); + v[i++] = erts_bld_sint64(hpp, szp, os_mtime); + + return erts_bld_2tup_list(hpp, szp, (Sint) i, k, v); +#endif +} + +Eterm +erts_monotonic_time_source(struct process *c_p) +{ + Uint hsz = 0; + Eterm *hp = NULL; + Sint64 os_mtime = 0; +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + if (!time_sup.r.o.os_monotonic_time_disable) + os_mtime = (Sint64) erts_os_monotonic_time(); +#endif + + bld_monotonic_time_source(NULL, &hsz, os_mtime); + if (hsz) + hp = HAlloc(c_p, hsz); + return bld_monotonic_time_source(&hp, NULL, os_mtime); +} + +static Eterm +bld_system_time_source(Uint **hpp, Uint *szp, Sint64 os_stime) +{ + int i = 0; + Eterm k[5]; + Eterm v[5]; + + k[i] = erts_bld_atom(hpp, szp, "function"); + v[i++] = erts_bld_atom(hpp, szp, + time_sup.r.o.os_system_time_func); + + if (time_sup.r.o.os_system_time_clock_id) { + k[i] = erts_bld_atom(hpp, szp, "clock_id"); + v[i++] = erts_bld_atom(hpp, szp, + time_sup.r.o.os_system_time_clock_id); + } + + k[i] = erts_bld_atom(hpp, szp, "resolution"); + v[i++] = erts_bld_uint64(hpp, szp, + time_sup.r.o.os_system_time_resolution); + + k[i] = erts_bld_atom(hpp, szp, "parallel"); + v[i++] = am_yes; + + k[i] = erts_bld_atom(hpp, szp, "time"); + v[i++] = erts_bld_sint64(hpp, szp, os_stime); + + return erts_bld_2tup_list(hpp, szp, (Sint) i, k, v); } + +Eterm +erts_system_time_source(struct process *c_p) +{ + Uint hsz = 0; + Eterm *hp = NULL; + Sint64 os_stime = (Sint64) erts_os_system_time(); + + bld_system_time_source(NULL, &hsz, os_stime); + if (hsz) + hp = HAlloc(c_p, hsz); + return bld_system_time_source(&hp, NULL, os_stime); +} + + +#include "bif.h" + +static ERTS_INLINE Eterm +time_unit_conversion(Process *c_p, Eterm term, ErtsMonotonicTime val, ErtsMonotonicTime muloff) +{ + ErtsMonotonicTime result; + BIF_RETTYPE ret; + + if (val < 0) + goto trap_to_erlang_code; + + /* Convert to common user specified time units */ + switch (term) { + case am_seconds: + case make_small(1): + result = ERTS_MONOTONIC_TO_SEC(val) + muloff*ERTS_MONOTONIC_OFFSET_SEC; + ERTS_BIF_PREP_RET(ret, make_time_val(c_p, result)); + break; + case am_milli_seconds: + case make_small(1000): + result = ERTS_MONOTONIC_TO_MSEC(val) + muloff*ERTS_MONOTONIC_OFFSET_MSEC; + ERTS_BIF_PREP_RET(ret, make_time_val(c_p, result)); + break; + case am_micro_seconds: + case make_small(1000*1000): + result = ERTS_MONOTONIC_TO_USEC(val) + muloff*ERTS_MONOTONIC_OFFSET_USEC; + ERTS_BIF_PREP_RET(ret, make_time_val(c_p, result)); + break; +#ifdef ARCH_64 + case am_nano_seconds: + case make_small(1000*1000*1000): + result = ERTS_MONOTONIC_TO_NSEC(val) + muloff*ERTS_MONOTONIC_OFFSET_NSEC; + ERTS_BIF_PREP_RET(ret, make_time_val(c_p, result)); + break; +#endif + default: { + Eterm value, native_res; +#ifndef ARCH_64 + Sint user_res; + if (term == am_nano_seconds) + goto to_nano_seconds; + if (term_to_Sint(term, &user_res)) { + if (user_res == 1000*1000*1000) { + to_nano_seconds: + result = (ERTS_MONOTONIC_TO_NSEC(val) + + muloff*ERTS_MONOTONIC_OFFSET_NSEC); + ERTS_BIF_PREP_RET(ret, make_time_val(c_p, result)); + break; + } + if (user_res <= 0) + goto badarg; + } +#else + if (is_small(term)) { + if (signed_val(term) <= 0) + goto badarg; + } #endif + else if (is_big(term)) { + if (big_sign(term)) + goto badarg; + } + else { + badarg: + ERTS_BIF_PREP_ERROR(ret, c_p, BADARG); + break; + } + + trap_to_erlang_code: + /* Do it in erlang code instead; pass along values to use... */ + value = make_time_val(c_p, val + muloff*ERTS_MONOTONIC_OFFSET_NATIVE); + native_res = make_time_val(c_p, ERTS_MONOTONIC_TIME_UNIT); + + ERTS_BIF_PREP_TRAP3(ret, erts_convert_time_unit_trap, c_p, + value, native_res, term); + + break; + } + } + + return ret; +} + +/* Built in functions */ + +BIF_RETTYPE monotonic_time_0(BIF_ALIST_0) +{ + ErtsMonotonicTime mtime = time_sup.r.o.get_time(); + mtime += ERTS_MONOTONIC_OFFSET_NATIVE; + BIF_RET(make_time_val(BIF_P, mtime)); +} + +BIF_RETTYPE monotonic_time_1(BIF_ALIST_1) +{ + BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, time_sup.r.o.get_time(), 1)); +} + +BIF_RETTYPE system_time_0(BIF_ALIST_0) +{ + ErtsMonotonicTime mtime, offset; + mtime = time_sup.r.o.get_time(); + offset = get_time_offset(); + BIF_RET(make_time_val(BIF_P, mtime + offset)); +} + +BIF_RETTYPE system_time_1(BIF_ALIST_0) +{ + ErtsMonotonicTime mtime, offset; + mtime = time_sup.r.o.get_time(); + offset = get_time_offset(); + BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, mtime + offset, 0)); +} + +BIF_RETTYPE erts_internal_time_unit_0(BIF_ALIST_0) +{ + BIF_RET(make_time_val(BIF_P, ERTS_MONOTONIC_TIME_UNIT)); +} + +BIF_RETTYPE time_offset_0(BIF_ALIST_0) +{ + ErtsMonotonicTime time_offset = get_time_offset(); + time_offset -= ERTS_MONOTONIC_OFFSET_NATIVE; + BIF_RET(make_time_val(BIF_P, time_offset)); +} + +BIF_RETTYPE time_offset_1(BIF_ALIST_1) +{ + BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, get_time_offset(), -1)); +} + + +BIF_RETTYPE timestamp_0(BIF_ALIST_0) +{ + Eterm *hp, res; + ErtsMonotonicTime stime, mtime, all_sec, offset; + Uint mega_sec, sec, micro_sec; + + mtime = time_sup.r.o.get_time(); + offset = get_time_offset(); + stime = ERTS_MONOTONIC_TO_USEC(mtime + offset); + all_sec = stime / ERTS_MONOTONIC_TIME_MEGA; + mega_sec = (Uint) (stime / ERTS_MONOTONIC_TIME_TERA); + sec = (Uint) (all_sec - (((ErtsMonotonicTime) mega_sec) + * ERTS_MONOTONIC_TIME_MEGA)); + micro_sec = (Uint) (stime - all_sec*ERTS_MONOTONIC_TIME_MEGA); + + ASSERT(((ErtsMonotonicTime) mega_sec)*ERTS_MONOTONIC_TIME_TERA + + ((ErtsMonotonicTime) sec)*ERTS_MONOTONIC_TIME_MEGA + + micro_sec == stime); + + /* + * Mega seconds is the only value that potentially + * ever could be a bignum. However, that wont happen + * during at least the next 4 million years... + * + * (System time will also have wrapped in the + * 64-bit integer before we get there...) + */ + + ASSERT(IS_USMALL(0, mega_sec)); + ASSERT(IS_USMALL(0, sec)); + ASSERT(IS_USMALL(0, micro_sec)); + + hp = HAlloc(BIF_P, 4); + res = TUPLE3(hp, + make_small(mega_sec), + make_small(sec), + make_small(micro_sec)); + BIF_RET(res); +} + +BIF_RETTYPE os_system_time_0(BIF_ALIST_0) +{ + ErtsSystemTime stime = erts_os_system_time(); + BIF_RET(make_time_val(BIF_P, stime)); +} + +BIF_RETTYPE os_system_time_1(BIF_ALIST_0) +{ + ErtsSystemTime stime = erts_os_system_time(); + BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, stime, 0)); +} + diff --git a/erts/emulator/beam/erl_utils.h b/erts/emulator/beam/erl_utils.h index c32f8fd61c..7cb8972e29 100644 --- a/erts/emulator/beam/erl_utils.h +++ b/erts/emulator/beam/erl_utils.h @@ -113,12 +113,14 @@ void erts_silence_warn_unused_result(long unused); int erts_fit_in_bits_int64(Sint64); int erts_fit_in_bits_int32(Sint32); +int erts_fit_in_bits_uint(Uint); int erts_list_length(Eterm); int erts_is_builtin(Eterm, Eterm, int); Uint32 make_broken_hash(Eterm); Uint32 block_hash(byte *, unsigned, Uint32); Uint32 make_hash2(Eterm); Uint32 make_hash(Eterm); +Uint32 make_internal_hash(Eterm); void erts_save_emu_args(int argc, char **argv); Eterm erts_get_emu_args(struct process *c_p); diff --git a/erts/emulator/beam/erl_vm.h b/erts/emulator/beam/erl_vm.h index 78d98229d8..3a9fb1e07b 100644 --- a/erts/emulator/beam/erl_vm.h +++ b/erts/emulator/beam/erl_vm.h @@ -117,9 +117,9 @@ #if defined(DEBUG) || defined(CHECK_FOR_HOLES) #if HALFWORD_HEAP -# define ERTS_HOLE_MARKER (0xaf5e78ccU) +# define ERTS_HOLE_MARKER (0xdeadbeef) #else -# define ERTS_HOLE_MARKER (((0xaf5e78ccUL << 24) << 8) | 0xaf5e78ccUL) +# define ERTS_HOLE_MARKER (((0xdeadbeef << 24) << 8) | 0xdeadbeef) #endif #endif @@ -172,6 +172,7 @@ extern int H_MIN_SIZE; /* minimum (heap + stack) */ extern int BIN_VH_MIN_SIZE; /* minimum virtual (bin) heap */ extern int erts_atom_table_size;/* Atom table size */ +extern int erts_pd_initial_size;/* Initial Process dictionary table size */ #define ORIG_CREATION 0 #define INTERNAL_CREATION 255 diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c index 601cbe9d7d..2117dbec62 100644 --- a/erts/emulator/beam/external.c +++ b/erts/emulator/beam/external.c @@ -1182,7 +1182,8 @@ typedef struct { Eterm* hp_end; int remaining_n; char* remaining_bytes; - Eterm* maps_head; + Eterm* maps_list; + struct dec_term_hamt_placeholder* hamt_list; } B2TDecodeContext; typedef struct { @@ -1508,7 +1509,8 @@ static BIF_RETTYPE binary_to_term_int(Process* p, Uint32 flags, Eterm bin, Binar ctx->u.dc.hp_start = HAlloc(p, ctx->heap_size); ctx->u.dc.hp = ctx->u.dc.hp_start; ctx->u.dc.hp_end = ctx->u.dc.hp_start + ctx->heap_size; - ctx->u.dc.maps_head = NULL; + ctx->u.dc.maps_list = NULL; + ctx->u.dc.hamt_list = NULL; ctx->state = B2TDecode; /*fall through*/ case B2TDecode: @@ -2304,7 +2306,8 @@ dec_pid(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, Ete #define ENC_PATCH_FUN_SIZE ((Eterm) 2) #define ENC_BIN_COPY ((Eterm) 3) #define ENC_MAP_PAIR ((Eterm) 4) -#define ENC_LAST_ARRAY_ELEMENT ((Eterm) 5) +#define ENC_HASHMAP_NODE ((Eterm) 5) +#define ENC_LAST_ARRAY_ELEMENT ((Eterm) 6) static byte* enc_term(ErtsAtomCacheMap *acmp, Eterm obj, byte* ep, Uint32 dflags, @@ -2413,6 +2416,13 @@ enc_term_int(TTBEncodeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, byte* ep, WSTACK_PUSH2(s, ENC_TERM, *vptr); break; } + case ENC_HASHMAP_NODE: + if (is_list(obj)) { /* leaf node [K|V] */ + ptr = list_val(obj); + WSTACK_PUSH2(s, ENC_TERM, CDR(ptr)); + obj = CAR(ptr); + } + break; case ENC_LAST_ARRAY_ELEMENT: /* obj is the tuple */ { @@ -2594,23 +2604,54 @@ enc_term_int(TTBEncodeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, byte* ep, break; case MAP_DEF: - { - map_t *mp = (map_t*)map_val(obj); - Uint size = map_get_size(mp); + if (is_flatmap(obj)) { + flatmap_t *mp = (flatmap_t*)flatmap_val(obj); + Uint size = flatmap_get_size(mp); *ep++ = MAP_EXT; put_int32(size, ep); ep += 4; if (size > 0) { - Eterm *kptr = map_get_keys(mp); - Eterm *vptr = map_get_values(mp); + Eterm *kptr = flatmap_get_keys(mp); + Eterm *vptr = flatmap_get_values(mp); WSTACK_PUSH4(s, (UWord)kptr, (UWord)vptr, ENC_MAP_PAIR, size); } + } else { + Eterm hdr; + Uint node_sz; + ptr = boxed_val(obj); + hdr = *ptr; + ASSERT(is_header(hdr)); + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + *ep++ = MAP_EXT; + ptr++; + put_int32(*ptr, ep); ep += 4; + node_sz = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + *ep++ = MAP_EXT; + ptr++; + put_int32(*ptr, ep); ep += 4; + /*fall through*/ + case HAMT_SUBTAG_NODE_BITMAP: + node_sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(node_sz < 17); + break; + default: + erl_exit(1, "bad header\r\n"); + } + + ptr++; + WSTACK_RESERVE(s, node_sz*2); + while(node_sz--) { + WSTACK_FAST_PUSH(s, ENC_HASHMAP_NODE); + WSTACK_FAST_PUSH(s, *ptr++); + } } break; - case FLOAT_DEF: GET_DOUBLE(obj, f); if (dflags & DFLAG_NEW_FLOATS) { @@ -2892,9 +2933,19 @@ undo_offheap_in_area(ErlOffHeap* off_heap, Eterm* start, Eterm* end) #endif /* DEBUG */ } +struct dec_term_hamt_placeholder +{ + struct dec_term_hamt_placeholder* next; + Eterm* objp; /* write result here */ + Uint size; /* nr of leafs */ + Eterm leafs[1]; +}; + +#define DEC_TERM_HAMT_PLACEHOLDER_SIZE \ + (offsetof(struct dec_term_hamt_placeholder, leafs) / sizeof(Eterm)) /* Decode term from external format into *objp. -** On failure return NULL and (R13B04) *hpp will be unchanged. +** On failure return NULL and *hpp will be unchanged. */ static byte* dec_term(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, @@ -2904,7 +2955,8 @@ dec_term(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, int n; ErtsAtomEncoding char_enc; register Eterm* hp; /* Please don't take the address of hp */ - Eterm *maps_head; /* for validation of maps */ + Eterm *maps_list; /* for preprocessing of small maps */ + struct dec_term_hamt_placeholder* hamt_list; /* for preprocessing of big maps */ Eterm* next; SWord reds; @@ -2914,7 +2966,8 @@ dec_term(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, next = ctx->u.dc.next; ep = ctx->u.dc.ep; hpp = &ctx->u.dc.hp; - maps_head = ctx->u.dc.maps_head; + maps_list = ctx->u.dc.maps_list; + hamt_list = ctx->u.dc.hamt_list; if (ctx->state != B2TDecode) { int n_limit = reds; @@ -2995,7 +3048,8 @@ dec_term(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, reds = ERTS_SWORD_MAX; next = objp; *next = (Eterm) (UWord) NULL; - maps_head = NULL; + maps_list = NULL; + hamt_list = NULL; } hp = *hpp; @@ -3059,6 +3113,8 @@ dec_term(ErtsDistExternal *edep, Eterm** hpp, byte* ep, ErlOffHeap* off_heap, big = make_small(0); } else { big = bytes_to_big(first, n, neg, hp); + if (is_nil(big)) + goto error; if (is_big(big)) { hp += big_arity(big) + 1; } @@ -3397,6 +3453,7 @@ dec_term_atom_common: pb->size = n; pb->next = off_heap->first; off_heap->first = (struct erl_off_heap_header*)pb; + OH_OVERHEAD(off_heap, pb->size / sizeof(Eterm)); pb->val = dbin; pb->bytes = (byte*) dbin->orig_bytes; pb->flags = 0; @@ -3448,6 +3505,7 @@ dec_term_atom_common: pb->size = n; pb->next = off_heap->first; off_heap->first = (struct erl_off_heap_header*)pb; + OH_OVERHEAD(off_heap, pb->size / sizeof(Eterm)); pb->val = dbin; pb->bytes = (byte*) dbin->orig_bytes; pb->flags = 0; @@ -3529,46 +3587,67 @@ dec_term_atom_common: break; case MAP_EXT: { - map_t *mp; Uint32 size,n; Eterm *kptr,*vptr; Eterm keys; size = get_int32(ep); ep += 4; - keys = make_tuple(hp); - *hp++ = make_arityval(size); - hp += size; - kptr = hp - 1; - - mp = (map_t*)hp; - hp += MAP_HEADER_SIZE; - hp += size; - vptr = hp - 1; - - /* kptr, last word for keys - * vptr, last word for values - */ - - /* - * Use thing_word to link through decoded maps. - * The list of maps is for later validation. - */ - - mp->thing_word = (Eterm) COMPRESS_POINTER(maps_head); - maps_head = (Eterm *) mp; - - mp->size = size; - mp->keys = keys; - *objp = make_map(mp); - - for (n = size; n; n--) { - *vptr = (Eterm) COMPRESS_POINTER(next); - *kptr = (Eterm) COMPRESS_POINTER(vptr); - next = kptr; - vptr--; - kptr--; - } + if (size <= MAP_SMALL_MAP_LIMIT) { + flatmap_t *mp; + + keys = make_tuple(hp); + *hp++ = make_arityval(size); + hp += size; + kptr = hp - 1; + + mp = (flatmap_t*)hp; + hp += MAP_HEADER_FLATMAP_SZ; + hp += size; + vptr = hp - 1; + + /* kptr, last word for keys + * vptr, last word for values + */ + + /* + * Use thing_word to link through decoded maps. + * The list of maps is for later validation. + */ + + mp->thing_word = (Eterm) COMPRESS_POINTER(maps_list); + maps_list = (Eterm *) mp; + + mp->size = size; + mp->keys = keys; + *objp = make_flatmap(mp); + + for (n = size; n; n--) { + *vptr = (Eterm) COMPRESS_POINTER(next); + *kptr = (Eterm) COMPRESS_POINTER(vptr); + next = kptr; + vptr--; + kptr--; + } + } + else { /* Make hamt */ + struct dec_term_hamt_placeholder* holder = + (struct dec_term_hamt_placeholder*) hp; + + holder->next = hamt_list; + hamt_list = holder; + holder->objp = objp; + holder->size = size; + + hp += DEC_TERM_HAMT_PLACEHOLDER_SIZE; + + for (n = size; n; n--) { + CDR(hp) = (Eterm) COMPRESS_POINTER(next); + CAR(hp) = (Eterm) COMPRESS_POINTER(&CDR(hp)); + next = &CAR(hp); + hp += 2; + } + } } break; case NEW_FUN_EXT: @@ -3746,6 +3825,7 @@ dec_term_atom_common: hp += PROC_BIN_SIZE; pb->next = off_heap->first; off_heap->first = (struct erl_off_heap_header*)pb; + OH_OVERHEAD(off_heap, pb->size / sizeof(Eterm)); pb->flags = 0; *objp = make_binary(pb); break; @@ -3763,6 +3843,7 @@ dec_term_atom_common: hp += PROC_BIN_SIZE; pb->next = off_heap->first; off_heap->first = (struct erl_off_heap_header*)pb; + OH_OVERHEAD(off_heap, pb->size / sizeof(Eterm)); pb->flags = 0; sub = (ErlSubBin*)hp; @@ -3789,7 +3870,8 @@ dec_term_atom_common: ctx->u.dc.ep = ep; ctx->u.dc.next = next; ctx->u.dc.hp = hp; - ctx->u.dc.maps_head = maps_head; + ctx->u.dc.maps_list = maps_list; + ctx->u.dc.hamt_list = hamt_list; ctx->reds = 0; return NULL; } @@ -3804,12 +3886,40 @@ dec_term_atom_common: * - done here for when we know it is complete. */ - while (maps_head) { - next = (Eterm *)(EXPAND_POINTER(*maps_head)); - *maps_head = MAP_HEADER; - if (!erts_validate_and_sort_map((map_t*)maps_head)) + while (maps_list) { + next = (Eterm *)(EXPAND_POINTER(*maps_list)); + *maps_list = MAP_HEADER_FLATMAP; + if (!erts_validate_and_sort_flatmap((flatmap_t*)maps_list)) goto error; - maps_head = next; + maps_list = next; + } + + /* Iterate through all the hamts and build tree nodes. + */ + if (hamt_list) { + ErtsHeapFactory factory; + + factory.p = NULL; + factory.hp = hp; + /* We assume heap will suffice (see hashmap_over_estimated_heap_size) */ + + do { + struct dec_term_hamt_placeholder* hamt = hamt_list; + *hamt->objp = erts_hashmap_from_array(&factory, + hamt->leafs, + hamt->size, + 1); + if (is_non_value(*hamt->objp)) + goto error; + + hamt_list = hamt->next; + + /* Yes, we waste a couple of heap words per hamt + for the temporary placeholder */ + *(Eterm*)hamt = make_pos_bignum_header(DEC_TERM_HAMT_PLACEHOLDER_SIZE-1); + } while (hamt_list); + + hp = factory.hp; } if (ctx) { @@ -3994,8 +4104,9 @@ encode_size_struct_int(TTBSizeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, } for (i = 1; i <= arity; ++i) { if (is_list(ptr[i])) { - if ((m = is_string(obj)) && (m < MAX_STRING_LEN)) { + if ((m = is_string(ptr[i])) && (m < MAX_STRING_LEN)) { result += m + 2 + 1; + continue; } else { result += 5; } @@ -4006,43 +4117,99 @@ encode_size_struct_int(TTBSizeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, } break; case MAP_DEF: - { - map_t *mp = (map_t*)map_val(obj); - Uint size = map_get_size(mp); + if (is_flatmap(obj)) { + flatmap_t *mp = (flatmap_t*)flatmap_val(obj); + Uint size = flatmap_get_size(mp); Uint i; Eterm *ptr; result += 1 + 4; /* tag + 4 bytes size */ /* push values first */ - ptr = map_get_values(mp); - i = size; - while(i--) { + ptr = flatmap_get_values(mp); + for (i = size; i; i--, ptr++) { if (is_list(*ptr)) { if ((m = is_string(*ptr)) && (m < MAX_STRING_LEN)) { result += m + 2 + 1; + continue; } else { result += 5; } } ESTACK_PUSH(s,*ptr); - ++ptr; } - ptr = map_get_keys(mp); - i = size; - while(i--) { + ptr = flatmap_get_keys(mp); + for (i = size; i; i--, ptr++) { if (is_list(*ptr)) { if ((m = is_string(*ptr)) && (m < MAX_STRING_LEN)) { result += m + 2 + 1; + continue; } else { result += 5; } } ESTACK_PUSH(s,*ptr); - ++ptr; } goto outer_loop; + } else { + Eterm *ptr; + Eterm hdr; + Uint node_sz; + ptr = boxed_val(obj); + hdr = *ptr; + ASSERT(is_header(hdr)); + switch(hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + ptr++; + node_sz = 16; + result += 1 + 4; /* tag + 4 bytes size */ + break; + case HAMT_SUBTAG_HEAD_BITMAP: + ptr++; + result += 1 + 4; /* tag + 4 bytes size */ + /*fall through*/ + case HAMT_SUBTAG_NODE_BITMAP: + node_sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(node_sz < 17); + break; + default: + erl_exit(1, "bad header\r\n"); + } + + ptr++; + ESTACK_RESERVE(s, node_sz*2); + while(node_sz--) { + if (is_list(*ptr)) { + Eterm* leaf = list_val(*ptr); + if (is_not_list(CAR(leaf))) { + ESTACK_FAST_PUSH(s, CAR(leaf)); + } + else { + if ((m = is_string(CAR(leaf))) && (m < MAX_STRING_LEN)) { + result += m + 2 + 1; + } else { + result += 5; + ESTACK_FAST_PUSH(s, CAR(leaf)); + } + } + if (is_not_list(CDR(leaf))) { + ESTACK_FAST_PUSH(s, CDR(leaf)); + } + else { + if ((m = is_string(CDR(leaf))) && (m < MAX_STRING_LEN)) { + result += m + 2 + 1; + } else { + result += 5; + ESTACK_FAST_PUSH(s, CDR(leaf)); + } + } + } + else { + ESTACK_FAST_PUSH(s, *ptr); + } + ptr++; + } } break; case FLOAT_DEF: @@ -4147,6 +4314,8 @@ encode_size_struct_int(TTBSizeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, return 0; } + + static Sint decoded_size(byte *ep, byte* endp, int internal_tags, B2TContext* ctx) { @@ -4340,7 +4509,11 @@ init_done: n = get_int32(ep); ep += 4; ADDTERMS(2*n); - heap_size += 3 + n + 1 + n; + if (n <= MAP_SMALL_MAP_LIMIT) { + heap_size += 3 + n + 1 + n; + } else { + heap_size += hashmap_over_estimated_heap_size(n); + } break; case STRING_EXT: CHKSIZE(2); diff --git a/erts/emulator/beam/global.h b/erts/emulator/beam/global.h index 5330f389e0..634fe533d0 100644 --- a/erts/emulator/beam/global.h +++ b/erts/emulator/beam/global.h @@ -348,8 +348,6 @@ extern Uint display_items; /* no of items to display in traces etc */ extern int erts_backtrace_depth; extern erts_smp_atomic32_t erts_max_gen_gcs; -extern int erts_disable_tolerant_timeofday; - extern int bif_reductions; /* reductions + fcalls (when doing call_bif) */ extern int stackdump_on_exit; @@ -372,16 +370,17 @@ extern int stackdump_on_exit; * DESTROY_ESTACK(Stack) */ -typedef struct { +typedef struct ErtsEStack_ { Eterm* start; Eterm* sp; Eterm* end; + Eterm* edefault; ErtsAlcType_t alloc_type; }ErtsEStack; #define DEF_ESTACK_SIZE (16) -void erl_grow_estack(ErtsEStack*, Eterm* def_stack); +void erl_grow_estack(ErtsEStack*, Uint need); #define ESTK_CONCAT(a,b) a##b #define ESTK_DEF_STACK(s) ESTK_CONCAT(s,_default_estack) @@ -391,22 +390,23 @@ void erl_grow_estack(ErtsEStack*, Eterm* def_stack); ESTK_DEF_STACK(s), /* start */ \ ESTK_DEF_STACK(s), /* sp */ \ ESTK_DEF_STACK(s) + DEF_ESTACK_SIZE, /* end */ \ + ESTK_DEF_STACK(s), /* default */ \ ERTS_ALC_T_ESTACK /* alloc_type */ \ } #define ESTACK_CHANGE_ALLOCATOR(s,t) \ do { \ - if (s.start != ESTK_DEF_STACK(s)) { \ + if ((s).start != ESTK_DEF_STACK(s)) { \ erl_exit(1, "Internal error - trying to change allocator " \ "type of active estack\n"); \ } \ - s.alloc_type = (t); \ + (s).alloc_type = (t); \ } while (0) #define DESTROY_ESTACK(s) \ do { \ - if (s.start != ESTK_DEF_STACK(s)) { \ - erts_free(s.alloc_type, s.start); \ + if ((s).start != ESTK_DEF_STACK(s)) { \ + erts_free((s).alloc_type, (s).start); \ } \ } while(0) @@ -417,16 +417,17 @@ do { \ */ #define ESTACK_SAVE(s,dst)\ do {\ - if (s.start == ESTK_DEF_STACK(s)) {\ + if ((s).start == ESTK_DEF_STACK(s)) {\ UWord _wsz = ESTACK_COUNT(s);\ - (dst)->start = erts_alloc(s.alloc_type,\ + (dst)->start = erts_alloc((s).alloc_type,\ DEF_ESTACK_SIZE * sizeof(Eterm));\ - memcpy((dst)->start, s.start,_wsz*sizeof(Eterm));\ + memcpy((dst)->start, (s).start,_wsz*sizeof(Eterm));\ (dst)->sp = (dst)->start + _wsz;\ (dst)->end = (dst)->start + DEF_ESTACK_SIZE;\ - (dst)->alloc_type = s.alloc_type;\ + (dst)->edefault = NULL;\ + (dst)->alloc_type = (s).alloc_type;\ } else\ - *(dst) = s;\ + *(dst) = (s);\ } while (0) #define DESTROY_SAVED_ESTACK(estack)\ @@ -445,83 +446,114 @@ do {\ */ #define ESTACK_RESTORE(s, src) \ do { \ - ASSERT(s.start == ESTK_DEF_STACK(s)); \ - s = *(src); /* struct copy */ \ + ASSERT((s).start == ESTK_DEF_STACK(s)); \ + (s) = *(src); /* struct copy */ \ (src)->start = NULL; \ - ASSERT(s.sp >= s.start); \ - ASSERT(s.sp <= s.end); \ + ASSERT((s).sp >= (s).start); \ + ASSERT((s).sp <= (s).end); \ } while (0) -#define ESTACK_IS_STATIC(s) (s.start == ESTK_DEF_STACK(s))) +#define ESTACK_IS_STATIC(s) ((s).start == ESTK_DEF_STACK(s)) -#define ESTACK_PUSH(s, x) \ -do { \ - if (s.sp == s.end) { \ - erl_grow_estack(&s, ESTK_DEF_STACK(s)); \ - } \ - *s.sp++ = (x); \ +#define ESTACK_PUSH(s, x) \ +do { \ + if ((s).sp == (s).end) { \ + erl_grow_estack(&(s), 1); \ + } \ + *(s).sp++ = (x); \ } while(0) #define ESTACK_PUSH2(s, x, y) \ do { \ - if (s.sp > s.end - 2) { \ - erl_grow_estack(&s, ESTK_DEF_STACK(s)); \ + if ((s).sp > (s).end - 2) { \ + erl_grow_estack(&(s), 2); \ } \ - *s.sp++ = (x); \ - *s.sp++ = (y); \ + *(s).sp++ = (x); \ + *(s).sp++ = (y); \ } while(0) #define ESTACK_PUSH3(s, x, y, z) \ do { \ - if (s.sp > s.end - 3) { \ - erl_grow_estack(&s, ESTK_DEF_STACK(s)); \ + if ((s).sp > (s).end - 3) { \ + erl_grow_estack(&s, 3); \ } \ - *s.sp++ = (x); \ - *s.sp++ = (y); \ - *s.sp++ = (z); \ + *(s).sp++ = (x); \ + *(s).sp++ = (y); \ + *(s).sp++ = (z); \ } while(0) #define ESTACK_PUSH4(s, E1, E2, E3, E4) \ do { \ - if (s.sp > s.end - 4) { \ - erl_grow_estack(&s, ESTK_DEF_STACK(s)); \ + if ((s).sp > (s).end - 4) { \ + erl_grow_estack(&s, 4); \ } \ - *s.sp++ = (E1); \ - *s.sp++ = (E2); \ - *s.sp++ = (E3); \ - *s.sp++ = (E4); \ + *(s).sp++ = (E1); \ + *(s).sp++ = (E2); \ + *(s).sp++ = (E3); \ + *(s).sp++ = (E4); \ } while(0) -#define ESTACK_COUNT(s) (s.sp - s.start) -#define ESTACK_ISEMPTY(s) (s.sp == s.start) -#define ESTACK_POP(s) (*(--s.sp)) +#define ESTACK_RESERVE(s, push_cnt) \ +do { \ + if ((s).sp > (s).end - (push_cnt)) { \ + erl_grow_estack(&(s), (push_cnt)); \ + } \ +} while(0) + +/* Must be preceded by ESTACK_RESERVE */ +#define ESTACK_FAST_PUSH(s, x) \ +do { \ + ASSERT((s).sp < (s).end); \ + *s.sp++ = (x); \ +} while(0) + +#define ESTACK_COUNT(s) ((s).sp - (s).start) +#define ESTACK_ISEMPTY(s) ((s).sp == (s).start) +#define ESTACK_POP(s) (*(--(s).sp)) /* * WSTACK: same as ESTACK but with UWord instead of Eterm */ -typedef struct { +typedef struct ErtsWStack_ { UWord* wstart; UWord* wsp; UWord* wend; + UWord* wdefault; ErtsAlcType_t alloc_type; }ErtsWStack; #define DEF_WSTACK_SIZE (16) -void erl_grow_wstack(ErtsWStack*, UWord* def_stack); +void erl_grow_wstack(ErtsWStack*, Uint need); #define WSTK_CONCAT(a,b) a##b #define WSTK_DEF_STACK(s) WSTK_CONCAT(s,_default_wstack) -#define DECLARE_WSTACK(s) \ +#define WSTACK_DECLARE(s) \ UWord WSTK_DEF_STACK(s)[DEF_WSTACK_SIZE]; \ ErtsWStack s = { \ WSTK_DEF_STACK(s), /* wstart */ \ WSTK_DEF_STACK(s), /* wsp */ \ WSTK_DEF_STACK(s) + DEF_WSTACK_SIZE, /* wend */ \ + WSTK_DEF_STACK(s), /* wdflt */ \ ERTS_ALC_T_ESTACK /* alloc_type */ \ } +#define DECLARE_WSTACK WSTACK_DECLARE + +typedef struct ErtsDynamicWStack_ { + UWord default_stack[DEF_WSTACK_SIZE]; + ErtsWStack ws; +}ErtsDynamicWStack; + +#define WSTACK_INIT(dwsp, ALC_TYPE) \ +do { \ + (dwsp)->ws.wstart = (dwsp)->default_stack; \ + (dwsp)->ws.wsp = (dwsp)->default_stack; \ + (dwsp)->ws.wend = (dwsp)->default_stack + DEF_WSTACK_SIZE;\ + (dwsp)->ws.wdefault = (dwsp)->default_stack; \ + (dwsp)->ws.alloc_type = ALC_TYPE; \ +} while (0) #define WSTACK_CHANGE_ALLOCATOR(s,t) \ do { \ @@ -532,13 +564,20 @@ do { \ s.alloc_type = (t); \ } while (0) -#define DESTROY_WSTACK(s) \ +#define WSTACK_DESTROY(s) \ do { \ - if (s.wstart != WSTK_DEF_STACK(s)) { \ + if (s.wstart != s.wdefault) { \ erts_free(s.alloc_type, s.wstart); \ } \ } while(0) +#define DESTROY_WSTACK WSTACK_DESTROY +#define WSTACK_DEBUG(s) \ + do { \ + fprintf(stderr, "wstack size = %ld\r\n", s.wsp - s.wstart); \ + fprintf(stderr, "wstack wstart = %p\r\n", s.wstart); \ + fprintf(stderr, "wstack wsp = %p\r\n", s.wsp); \ + } while(0) /* * Do not free the stack after this, it may have pointers into what @@ -553,6 +592,7 @@ do {\ memcpy((dst)->wstart, s.wstart,_wsz*sizeof(UWord));\ (dst)->wsp = (dst)->wstart + _wsz;\ (dst)->wend = (dst)->wstart + DEF_WSTACK_SIZE;\ + (dst)->wdefault = NULL;\ (dst)->alloc_type = s.alloc_type;\ } else\ *(dst) = s;\ @@ -581,12 +621,12 @@ do { \ ASSERT(s.wsp <= s.wend); \ } while (0) -#define WSTACK_IS_STATIC(s) (s.wstart == WSTK_DEF_STACK(s))) +#define WSTACK_IS_STATIC(s) (s.wstart == WSTK_DEF_STACK(s)) #define WSTACK_PUSH(s, x) \ do { \ if (s.wsp == s.wend) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + erl_grow_wstack(&s, 1); \ } \ *s.wsp++ = (x); \ } while(0) @@ -594,7 +634,7 @@ do { \ #define WSTACK_PUSH2(s, x, y) \ do { \ if (s.wsp > s.wend - 2) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + erl_grow_wstack(&s, 2); \ } \ *s.wsp++ = (x); \ *s.wsp++ = (y); \ @@ -602,8 +642,8 @@ do { \ #define WSTACK_PUSH3(s, x, y, z) \ do { \ - if (s.wsp > s.wend - 3) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + if (s.wsp > s.wend - 3) { \ + erl_grow_wstack(&s, 3); \ } \ *s.wsp++ = (x); \ *s.wsp++ = (y); \ @@ -612,8 +652,8 @@ do { \ #define WSTACK_PUSH4(s, A1, A2, A3, A4) \ do { \ - if (s.wsp > s.wend - 4) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + if (s.wsp > s.wend - 4) { \ + erl_grow_wstack(&s, 4); \ } \ *s.wsp++ = (A1); \ *s.wsp++ = (A2); \ @@ -624,7 +664,7 @@ do { \ #define WSTACK_PUSH5(s, A1, A2, A3, A4, A5) \ do { \ if (s.wsp > s.wend - 5) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + erl_grow_wstack(&s, 5); \ } \ *s.wsp++ = (A1); \ *s.wsp++ = (A2); \ @@ -636,7 +676,7 @@ do { \ #define WSTACK_PUSH6(s, A1, A2, A3, A4, A5, A6) \ do { \ if (s.wsp > s.wend - 6) { \ - erl_grow_wstack(&s, WSTK_DEF_STACK(s)); \ + erl_grow_wstack(&s, 6); \ } \ *s.wsp++ = (A1); \ *s.wsp++ = (A2); \ @@ -646,9 +686,85 @@ do { \ *s.wsp++ = (A6); \ } while(0) +#define WSTACK_RESERVE(s, push_cnt) \ +do { \ + if (s.wsp > s.wend - (push_cnt)) { \ + erl_grow_wstack(&s, (push_cnt)); \ + } \ +} while(0) + +/* Must be preceded by WSTACK_RESERVE */ +#define WSTACK_FAST_PUSH(s, x) \ +do { \ + ASSERT(s.wsp < s.wend); \ + *s.wsp++ = (x); \ +} while(0) + #define WSTACK_COUNT(s) (s.wsp - s.wstart) #define WSTACK_ISEMPTY(s) (s.wsp == s.wstart) -#define WSTACK_POP(s) (*(--s.wsp)) +#define WSTACK_POP(s) ((ASSERT(s.wsp > s.wstart)),*(--s.wsp)) + +#define WSTACK_ROLLBACK(s, count) (ASSERT(WSTACK_COUNT(s) >= (count)), \ + s.wsp = s.wstart + (count)) + +/* PSTACK - Stack of any type. + * Usage: + * { + * #define PSTACK_TYPE MyType + * PSTACK_DECLARE(s,16); + * MyType *sp = PSTACK_PUSH(s); + * + * sp->x = .... + * sp->y = .... + * sp = PSTACK_PUSH(s); + * ... + * sp = PSTACK_POP(s); + * if (PSTACK_IS_EMPTY(s)) { + * // sp is invalid when stack is empty after pop + * } + * + * PSTACK_DESTROY(s); + * } + */ + + +typedef struct ErtsPStack_ { + byte* pstart; + byte* psp; + byte* pend; + ErtsAlcType_t alloc_type; +}ErtsPStack; + +void erl_grow_pstack(ErtsPStack* s, void* default_pstack, unsigned need_bytes); +#define PSTK_CONCAT(a,b) a##b +#define PSTK_DEF_STACK(s) PSTK_CONCAT(s,_default_pstack) + +#define PSTACK_DECLARE(s, DEF_PSTACK_SIZE) \ +PSTACK_TYPE PSTK_DEF_STACK(s)[DEF_PSTACK_SIZE]; \ +ErtsPStack s = { (byte*)PSTK_DEF_STACK(s), /* pstart */ \ + (byte*)(PSTK_DEF_STACK(s) - 1), /* psp */ \ + (byte*)(PSTK_DEF_STACK(s) + (DEF_PSTACK_SIZE)), /* pend */\ + ERTS_ALC_T_ESTACK /* alloc_type */ \ +} + +#define PSTACK_DESTROY(s) \ +do { \ + if (s.pstart != (byte*)PSTK_DEF_STACK(s)) { \ + erts_free(s.alloc_type, s.pstart); \ + } \ +} while(0) + +#define PSTACK_IS_EMPTY(s) (s.psp < s.pstart) + +#define PSTACK_TOP(s) (ASSERT(!PSTACK_IS_EMPTY(s)), (PSTACK_TYPE*)(s.psp)) + +#define PSTACK_PUSH(s) \ + (s.psp += sizeof(PSTACK_TYPE), \ + ((s.psp == s.pend) ? erl_grow_pstack(&s, PSTK_DEF_STACK(s), \ + sizeof(PSTACK_TYPE)) : (void)0), \ + ((PSTACK_TYPE*) s.psp)) + +#define PSTACK_POP(s) ((PSTACK_TYPE*) (s.psp -= sizeof(PSTACK_TYPE))) /* binary.c */ @@ -671,9 +787,6 @@ erts_bld_port_info(Eterm **hpp, void erts_bif_info_init(void); /* bif.c */ -Eterm erts_make_ref(Process *); -Eterm erts_make_ref_in_buffer(Eterm buffer[REF_THING_SIZE]); -void erts_make_ref_in_array(Uint32 ref[ERTS_MAX_REF_NUMBERS]); ERTS_GLB_INLINE Eterm erts_proc_store_ref(Process *c_p, Uint32 ref[ERTS_MAX_REF_NUMBERS]); @@ -1192,7 +1305,9 @@ erts_alloc_message_heap_state(Uint size, state = erts_smp_atomic32_read_acqb(&receiver->state); if (statep) *statep = state; - if (state & (ERTS_PSFLG_EXITING|ERTS_PSFLG_PENDING_EXIT)) + if (state & (ERTS_PSFLG_OFF_HEAP_MSGS + | ERTS_PSFLG_EXITING + | ERTS_PSFLG_PENDING_EXIT)) goto allocate_in_mbuf; #endif @@ -1212,7 +1327,9 @@ erts_alloc_message_heap_state(Uint size, state = erts_smp_atomic32_read_nob(&receiver->state); if (statep) *statep = state; - if ((state & (ERTS_PSFLG_EXITING|ERTS_PSFLG_PENDING_EXIT)) + if ((state & (ERTS_PSFLG_OFF_HEAP_MSGS + | ERTS_PSFLG_EXITING + | ERTS_PSFLG_PENDING_EXIT)) || (receiver->flags & F_DISABLE_GC) || HEAP_LIMIT(receiver) - HEAP_TOP(receiver) <= size) { /* diff --git a/erts/emulator/beam/io.c b/erts/emulator/beam/io.c index dc4c6fc350..1db3a9fba7 100644 --- a/erts/emulator/beam/io.c +++ b/erts/emulator/beam/io.c @@ -47,6 +47,7 @@ #include "external.h" #include "dtrace-wrapper.h" #include "erl_map.h" +#include "erl_bif_unique.h" extern ErlDrvEntry fd_driver_entry; #ifndef __OSE__ @@ -4085,6 +4086,9 @@ erts_port_control(Process* c_p, size, &resp_bufp, &resp_size); + + control_flags = prt->control_flags; + finalize_imm_drv_call(&try_call_state); if (tmp_alloced) erts_free(ERTS_ALC_T_TMP, bufp); @@ -4092,8 +4096,6 @@ erts_port_control(Process* c_p, return ERTS_PORT_OP_BADARG; } - control_flags = prt->control_flags; - hsz = port_control_result_size(control_flags, resp_bufp, &resp_size, @@ -4482,7 +4484,7 @@ make_port_info_term(Eterm **hpp_start, int len; int start; static Eterm item[] = ERTS_PORT_INFO_1_ITEMS; - static Eterm value[sizeof(item)/sizeof(item[0])]; + Eterm value[sizeof(item)/sizeof(item[0])]; start = 0; len = sizeof(item)/sizeof(item[0]); @@ -5349,7 +5351,11 @@ driver_deliver_term(Eterm to, ErlDrvTermData* data, int len) case ERL_DRV_MAP: { /* int */ ERTS_DDT_CHK_ENOUGH_ARGS(1); if ((int) ptr[0] < 0) ERTS_DDT_FAIL; - need += MAP_HEADER_SIZE + 1 + 2*ptr[0]; + if (ptr[0] > MAP_SMALL_MAP_LIMIT) { + need += hashmap_over_estimated_heap_size(ptr[0]); + } else { + need += MAP_HEADER_FLATMAP_SZ + 1 + 2*ptr[0]; + } depth -= 2*ptr[0]; if (depth < 0) ERTS_DDT_FAIL; ptr++; @@ -5593,31 +5599,52 @@ driver_deliver_term(Eterm to, ErlDrvTermData* data, int len) case ERL_DRV_MAP: { /* int */ int size = (int)ptr[0]; - Eterm* tp = hp; - Eterm* vp; - map_t *mp; - - *tp = make_arityval(size); - - hp += 1 + size; - mp = (map_t*)hp; - mp->thing_word = MAP_HEADER; - mp->size = size; - mp->keys = make_tuple(tp); - mess = make_map(mp); - - hp += MAP_HEADER_SIZE + size; /* advance "heap" pointer */ - - tp += size; /* point at last key */ - vp = hp - 1; /* point at last value */ - - while(size--) { - *vp-- = ESTACK_POP(stack); - *tp-- = ESTACK_POP(stack); - } - if (!erts_validate_and_sort_map(mp)) - ERTS_DDT_FAIL; - ptr++; + if (size > MAP_SMALL_MAP_LIMIT) { + int ix = 2*size; + ErtsHeapFactory factory; + Eterm* leafs = hp; + + hp += 2*size; + while(ix--) { *--hp = ESTACK_POP(stack); } + + hp += 2*size; + factory.p = NULL; + factory.hp = hp; + /* We assume heap will suffice (see hashmap_over_estimated_heap_size) */ + + mess = erts_hashmap_from_array(&factory, leafs, size, 1); + + if (is_non_value(mess)) + ERTS_DDT_FAIL; + + hp = factory.hp; + } else { + Eterm* tp = hp; + Eterm* vp; + flatmap_t *mp; + + *tp = make_arityval(size); + + hp += 1 + size; + mp = (flatmap_t*)hp; + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = size; + mp->keys = make_tuple(tp); + mess = make_flatmap(mp); + + hp += MAP_HEADER_FLATMAP_SZ + size; + + tp += size; /* point at last key */ + vp = hp - 1; /* point at last value */ + + while(size--) { + *vp-- = ESTACK_POP(stack); + *tp-- = ESTACK_POP(stack); + } + if (!erts_validate_and_sort_flatmap(mp)) + ERTS_DDT_FAIL; + } + ptr++; break; } diff --git a/erts/emulator/beam/ops.tab b/erts/emulator/beam/ops.tab index 68fcc177ae..d3649080dc 100644 --- a/erts/emulator/beam/ops.tab +++ b/erts/emulator/beam/ops.tab @@ -166,22 +166,26 @@ is_tuple Fail=f S | select_tuple_arity S=d Fail=f Size=u Rest=* => \ select_tuple_arity S=d Fail=f Size=u Rest=* => \ gen_select_tuple_arity(S, Fail, Size, Rest) -i_select_val r f I -i_select_val x f I -i_select_val y f I +i_select_val_bins r f I +i_select_val_bins x f I +i_select_val_bins y f I -i_select_val2 r f c f c f -i_select_val2 x f c f c f -i_select_val2 y f c f c f +i_select_val_lins r f I +i_select_val_lins x f I +i_select_val_lins y f I -i_select_tuple_arity2 r f A f A f -i_select_tuple_arity2 x f A f A f -i_select_tuple_arity2 y f A f A f +i_select_val2 r f c c f f +i_select_val2 x f c c f f +i_select_val2 y f c c f f i_select_tuple_arity r f I i_select_tuple_arity x f I i_select_tuple_arity y f I +i_select_tuple_arity2 r f A A f f +i_select_tuple_arity2 x f A A f f +i_select_tuple_arity2 y f A A f f + i_jump_on_val_zero r f I i_jump_on_val_zero x f I i_jump_on_val_zero y f I diff --git a/erts/emulator/beam/sys.h b/erts/emulator/beam/sys.h index 828f5b427a..251b39508f 100644 --- a/erts/emulator/beam/sys.h +++ b/erts/emulator/beam/sys.h @@ -21,6 +21,25 @@ #define __SYS_H__ +#if defined(DEBUG) || defined(ERTS_ENABLE_LOCK_CHECK) +# undef ERTS_CAN_INLINE +# define ERTS_CAN_INLINE 0 +# undef ERTS_INLINE +# define ERTS_INLINE +#endif + +#if ERTS_CAN_INLINE +#define ERTS_GLB_INLINE static ERTS_INLINE +#else +#define ERTS_GLB_INLINE +#endif + +#if ERTS_CAN_INLINE || defined(ERTS_DO_INCL_GLB_INLINE_FUNC_DEF) +# define ERTS_GLB_INLINE_INCL_FUNC_DEF 1 +#else +# define ERTS_GLB_INLINE_INCL_FUNC_DEF 0 +#endif + #if defined(VALGRIND) && !defined(NO_FPE_SIGNALS) # define NO_FPE_SIGNALS #endif @@ -132,24 +151,8 @@ typedef ERTS_SYS_FD_TYPE ErtsSysFdType; # endif #endif -#if defined(DEBUG) || defined(ERTS_ENABLE_LOCK_CHECK) -# undef ERTS_CAN_INLINE -# define ERTS_CAN_INLINE 0 -# undef ERTS_INLINE -# define ERTS_INLINE -#endif - -#if ERTS_CAN_INLINE -#define ERTS_GLB_INLINE static ERTS_INLINE -#else -#define ERTS_GLB_INLINE -#endif - -#if ERTS_CAN_INLINE || defined(ERTS_DO_INCL_GLB_INLINE_FUNC_DEF) -# define ERTS_GLB_INLINE_INCL_FUNC_DEF 1 -#else -# define ERTS_GLB_INLINE_INCL_FUNC_DEF 0 -#endif +#define ERTS_MK_VSN_INT(Major, Minor, Build) \ + ((((Major) & 0x3ff) << 20) | (((Minor) & 0x3ff) << 10) | ((Build) & 0x3ff)) #ifndef ERTS_EXIT_AFTER_DUMP # define ERTS_EXIT_AFTER_DUMP exit @@ -188,6 +191,16 @@ __decl_noreturn void __noreturn erl_assert_error(const char* expr, const char *f # define ASSERT(e) ((void) 1) #endif +/* ERTS_UNDEF can be used to silence false warnings about + * "variable may be used uninitialized" while keeping the variable + * marked as undefined by valgrind. + */ +#ifdef VALGRIND +# define ERTS_UNDEF(V,I) +#else +# define ERTS_UNDEF(V,I) V = I +#endif + /* * Compile time assert * (the actual compiler error msg can be a bit confusing) @@ -375,17 +388,45 @@ typedef Sint SWord; typedef UWord BeamInstr; #ifndef HAVE_INT64 -#if SIZEOF_LONG == 8 -#define HAVE_INT64 1 +# if SIZEOF_LONG == 8 +# define HAVE_INT64 1 typedef unsigned long Uint64; typedef long Sint64; -#elif SIZEOF_LONG_LONG == 8 -#define HAVE_INT64 1 +# ifdef ULONG_MAX +# define ERTS_UINT64_MAX ULONG_MAX +# endif +# ifdef LONG_MAX +# define ERTS_SINT64_MAX LONG_MAX +# endif +# ifdef LONG_MIN +# define ERTS_SINT64_MIN LONG_MIN +# endif +# elif SIZEOF_LONG_LONG == 8 +# define HAVE_INT64 1 typedef unsigned long long Uint64; typedef long long Sint64; -#else -#define HAVE_INT64 0 +# ifdef ULLONG_MAX +# define ERTS_UINT64_MAX ULLONG_MAX +# endif +# ifdef LLONG_MAX +# define ERTS_SINT64_MAX LLONG_MAX +# endif +# ifdef LLONG_MIN +# define ERTS_SINT64_MIN LLONG_MIN +# endif +# else +# error "No 64-bit integer type found" +# endif +#endif + +#ifndef ERTS_UINT64_MAX +# define ERTS_UINT64_MAX (~((Uint64) 0)) +#endif +#ifndef ERTS_SINT64_MAX +# define ERTS_SINT64_MAX ((Sint64) ((((Uint64) 1) << 63)-1)) #endif +#ifndef ERTS_SINT64_MIN +# define ERTS_SINT64_MIN (-1*(((Sint64) 1) << 63)) #endif #if SIZEOF_LONG == 4 @@ -662,14 +703,43 @@ extern char *erts_default_arg0; extern char os_type[]; -extern int sys_init_time(void); +typedef enum { + ERTS_NO_TIME_WARP_MODE, + ERTS_SINGLE_TIME_WARP_MODE, + ERTS_MULTI_TIME_WARP_MODE +} ErtsTimeWarpMode; + +typedef struct { + int have_os_monotonic_time; + ErtsMonotonicTime os_monotonic_time_unit; + ErtsMonotonicTime sys_clock_resolution; + struct { + Uint64 resolution; + char *func; + char *clock_id; + int locked_use; + int extended; + } os_monotonic_time_info; + struct { + Uint64 resolution; + char *func; + char *clock_id; + int locked_use; + } os_system_time_info; +} ErtsSysInitTimeResult; + +#define ERTS_SYS_INIT_TIME_RESULT_INITER \ + {0, (ErtsMonotonicTime) -1, (ErtsMonotonicTime) 1} + +extern void erts_init_sys_time_sup(void); +extern void sys_init_time(ErtsSysInitTimeResult *); +extern void erts_late_sys_init_time(void); extern void erts_deliver_time(void); extern void erts_time_remaining(SysTimeval *); -extern int erts_init_time_sup(void); +extern int erts_init_time_sup(int, ErtsTimeWarpMode); extern void erts_sys_init_float(void); extern void erts_thread_init_float(void); extern void erts_thread_disable_fpe(void); - ERTS_GLB_INLINE int erts_block_fpe(void); ERTS_GLB_INLINE void erts_unblock_fpe(int); @@ -716,7 +786,7 @@ extern char *erts_sys_ddll_error(int code); void erts_sys_schedule_interrupt(int set); #ifdef ERTS_SMP -void erts_sys_schedule_interrupt_timed(int set, erts_short_time_t msec); +void erts_sys_schedule_interrupt_timed(int, ErtsMonotonicTime); void erts_sys_main_thread(void); #endif @@ -755,6 +825,7 @@ int univ_to_local( int local_to_univ(Sint *year, Sint *month, Sint *day, Sint *hour, Sint *minute, Sint *second, int isdst); void get_now(Uint*, Uint*, Uint*); +ErtsMonotonicTime erts_get_monotonic_time(void); void get_sys_now(Uint*, Uint*, Uint*); void set_break_quit(void (*)(void), void (*)(void)); diff --git a/erts/emulator/beam/time.c b/erts/emulator/beam/time.c index 2fd8e0cf00..2bdda6c8af 100644 --- a/erts/emulator/beam/time.c +++ b/erts/emulator/beam/time.c @@ -83,7 +83,8 @@ #define ASSERT_NO_LOCKED_LOCKS #endif -static erts_smp_mtx_t tiw_lock; +#define ERTS_MONOTONIC_DAY ERTS_SEC_TO_MONOTONIC(60*60*24) +#define ERTS_CLKTCKS_DAY ERTS_MONOTONIC_TO_CLKTCKS(ERTS_MONOTONIC_DAY) /* BEGIN tiw_lock protected variables @@ -91,18 +92,12 @@ static erts_smp_mtx_t tiw_lock; ** The individual timer cells in tiw are also protected by the same mutex. */ +/* timing wheel size NEED to be a power of 2 */ #ifdef SMALL_MEMORY -#define TIW_SIZE 8192 +#define TIW_SIZE (1 << 13) #else -#define TIW_SIZE 65536 /* timing wheel size (should be a power of 2) */ +#define TIW_SIZE (1 << 20) #endif -static ErlTimer** tiw; /* the timing wheel, allocated in init_time() */ -static Uint tiw_pos; /* current position in wheel */ -static Uint tiw_nto; /* number of timeouts in wheel */ -static Uint tiw_min; -static ErlTimer *tiw_min_ptr; - -/* END tiw_lock protected variables */ /* Actual interval time chosen by sys_init_time() */ @@ -114,83 +109,135 @@ static int tiw_itime; /* Constant after init */ # define TIW_ITIME tiw_itime #endif -erts_smp_atomic32_t do_time; /* set at clock interrupt */ -static ERTS_INLINE erts_short_time_t do_time_read(void) +struct ErtsTimerWheel_ { + ErlTimer *w[TIW_SIZE]; + ErtsMonotonicTime pos; + Uint nto; + struct { + ErlTimer *head; + ErlTimer **tail; + Uint nto; + } at_once; + int true_next_timeout_time; + ErtsMonotonicTime next_timeout_time; + erts_atomic64_t next_timeout; + erts_smp_atomic32_t is_bumping; + erts_smp_mtx_t lock; +}; + +ErtsTimerWheel *erts_default_timer_wheel; /* managed by aux thread */ + +static ERTS_INLINE ErtsTimerWheel * +get_timer_wheel(ErlTimer *p) +{ + return (ErtsTimerWheel *) erts_smp_atomic_read_acqb(&p->wheel); +} + +static ERTS_INLINE void +set_timer_wheel(ErlTimer *p, ErtsTimerWheel *tiw) { - return erts_smp_atomic32_read_acqb(&do_time); + erts_smp_atomic_set_relb(&p->wheel, (erts_aint_t) tiw); } -static ERTS_INLINE erts_short_time_t do_time_update(void) +static ERTS_INLINE void +init_next_timeout(ErtsTimerWheel *tiw, + ErtsMonotonicTime time) { - return do_time_read(); + erts_atomic64_init_nob(&tiw->next_timeout, + (erts_aint64_t) time); } -static ERTS_INLINE void do_time_init(void) +static ERTS_INLINE void +set_next_timeout(ErtsTimerWheel *tiw, + ErtsMonotonicTime time, + int true_timeout) { - erts_smp_atomic32_init_nob(&do_time, 0); + tiw->true_next_timeout_time = true_timeout; + tiw->next_timeout_time = time; + erts_atomic64_set_relb(&tiw->next_timeout, + (erts_aint64_t) time); } /* get the time (in units of TIW_ITIME) to the next timeout, or -1 if there are no timeouts */ -static erts_short_time_t next_time_internal(void) /* PRE: tiw_lock taken by caller */ +static ERTS_INLINE ErtsMonotonicTime +find_next_timeout(ErtsTimerWheel *tiw, + ErtsMonotonicTime curr_time, + ErtsMonotonicTime max_search_time) { - int i, tm, nto; - Uint32 min; - ErlTimer* p; - erts_short_time_t dt; - - if (tiw_nto == 0) - return -1; /* no timeouts in wheel */ + int start_ix, tiw_pos_ix; + ErlTimer *p; + int true_min_timeout; + ErtsMonotonicTime min_timeout, min_timeout_pos, slot_timeout_pos, timeout_limit; + + ERTS_SMP_LC_ASSERT(erts_smp_lc_mtx_is_locked(&tiw->lock)); + + if (tiw->true_next_timeout_time) + return tiw->next_timeout_time; - if (tiw_min_ptr) { - min = tiw_min; - dt = do_time_read(); - return ((min >= dt) ? (min - dt) : 0); + /* We never set next timeout beyond timeout_limit */ + timeout_limit = curr_time + ERTS_MONOTONIC_DAY; + + if (tiw->nto == 0) { /* no timeouts in wheel */ + true_min_timeout = tiw->true_next_timeout_time = 0; + min_timeout_pos = ERTS_MONOTONIC_TO_CLKTCKS(timeout_limit); + goto found_next; } - - /* start going through wheel to find next timeout */ - tm = nto = 0; - min = (Uint32) -1; /* max Uint32 */ - i = tiw_pos; + + /* + * Don't want others entering trying to bump + * timers while we are checking... + */ + set_next_timeout(tiw, timeout_limit, 0); + + true_min_timeout = 1; + slot_timeout_pos = tiw->pos; + min_timeout_pos = ERTS_MONOTONIC_TO_CLKTCKS(curr_time + max_search_time); + + start_ix = tiw_pos_ix = (int) (tiw->pos & (TIW_SIZE-1)); + do { - p = tiw[i]; - while (p != NULL) { - nto++; - if (p->count == 0) { - /* found next timeout */ - dt = do_time_read(); - /* p->count is zero */ - tiw_min_ptr = p; - tiw_min = tm; - return ((tm >= dt) ? (tm - dt) : 0); - } else { - /* keep shortest time in 'min' */ - if (tm + p->count*TIW_SIZE < min) { - min = tm + p->count*TIW_SIZE; - tiw_min_ptr = p; - tiw_min = min; - } + slot_timeout_pos++; + if (slot_timeout_pos >= min_timeout_pos) { + true_min_timeout = 0; + break; + } + + p = tiw->w[tiw_pos_ix]; + + while (p) { + ErtsMonotonicTime timeout_pos; + ASSERT(p != p->next); + timeout_pos = p->timeout_pos; + if (min_timeout_pos > timeout_pos) { + min_timeout_pos = timeout_pos; + if (min_timeout_pos <= slot_timeout_pos) + goto found_next; } p = p->next; } - /* when we have found all timeouts the shortest time will be in min */ - if (nto == tiw_nto) break; - tm++; - i = (i + 1) % TIW_SIZE; - } while (i != tiw_pos); - dt = do_time_read(); - if (min <= (Uint32) dt) - return 0; - if ((min - (Uint32) dt) > (Uint32) ERTS_SHORT_TIME_T_MAX) - return ERTS_SHORT_TIME_T_MAX; - return (erts_short_time_t) (min - (Uint32) dt); + + tiw_pos_ix++; + if (tiw_pos_ix == TIW_SIZE) + tiw_pos_ix = 0; + } while (start_ix != tiw_pos_ix); + +found_next: + + min_timeout = ERTS_CLKTCKS_TO_MONOTONIC(min_timeout_pos); + if (min_timeout != tiw->next_timeout_time) + set_next_timeout(tiw, min_timeout, true_min_timeout); + + return min_timeout; } -static void remove_timer(ErlTimer *p) { +static void +remove_timer(ErtsTimerWheel *tiw, ErlTimer *p) +{ /* first */ if (!p->prev) { - tiw[p->slot] = p->next; + tiw->w[p->slot] = p->next; if(p->next) p->next->prev = NULL; } else { @@ -207,79 +254,164 @@ static void remove_timer(ErlTimer *p) { p->next = NULL; p->prev = NULL; - /* Make sure cancel callback isn't called */ - p->active = 0; - tiw_nto--; + + set_timer_wheel(p, NULL); + tiw->nto--; +} + +ErtsMonotonicTime +erts_check_next_timeout_time(ErtsTimerWheel *tiw, + ErtsMonotonicTime max_search_time) +{ + ErtsMonotonicTime next, curr; + + curr = erts_get_monotonic_time(); + + erts_smp_mtx_lock(&tiw->lock); + + next = find_next_timeout(tiw, curr, max_search_time); + + erts_smp_mtx_unlock(&tiw->lock); + + return next; } -/* Private export to erl_time_sup.c */ -erts_short_time_t erts_next_time(void) +#ifndef DEBUG +#define ERTS_DBG_CHK_SAFE_TO_SKIP_TO(TIW, TO) ((void) 0) +#else +#define ERTS_DBG_CHK_SAFE_TO_SKIP_TO(TIW, TO) debug_check_safe_to_skip_to((TIW), (TO)) +static void +debug_check_safe_to_skip_to(ErtsTimerWheel *tiw, ErtsMonotonicTime skip_to_pos) { - erts_short_time_t ret; + int slots, ix; + ErlTimer *tmr; + ErtsMonotonicTime tmp; + + ix = (int) (tiw->pos & (TIW_SIZE-1)); + tmp = skip_to_pos - tiw->pos; + ASSERT(tmp >= 0); + if (tmp < (ErtsMonotonicTime) TIW_SIZE) + slots = (int) tmp; + else + slots = TIW_SIZE; - erts_smp_mtx_lock(&tiw_lock); - (void)do_time_update(); - ret = next_time_internal(); - erts_smp_mtx_unlock(&tiw_lock); - return ret; + while (slots > 0) { + tmr = tiw->w[ix]; + while (tmr) { + ASSERT(tmr->timeout_pos > skip_to_pos); + tmr = tmr->next; + } + ix++; + if (ix == TIW_SIZE) + ix = 0; + slots--; + } } +#endif -static ERTS_INLINE void bump_timer_internal(erts_short_time_t dt) /* PRE: tiw_lock is write-locked */ +void +erts_bump_timers(ErtsTimerWheel *tiw, ErtsMonotonicTime curr_time) { - Uint keep_pos; - Uint count; - ErlTimer *p, **prev, *timeout_head, **timeout_tail; - Uint dtime = (Uint) dt; - - /* no need to bump the position if there aren't any timeouts */ - if (tiw_nto == 0) { - erts_smp_mtx_unlock(&tiw_lock); - return; + int tiw_pos_ix, slots; + ErlTimer *p, *timeout_head, **timeout_tail; + ErtsMonotonicTime bump_to, tmp_slots; + + if (erts_smp_atomic32_cmpxchg_nob(&tiw->is_bumping, 1, 0) != 0) + return; /* Another thread is currently bumping... */ + + bump_to = ERTS_MONOTONIC_TO_CLKTCKS(curr_time); + + erts_smp_mtx_lock(&tiw->lock); + + if (tiw->pos >= bump_to) { + timeout_head = NULL; + goto done; } - /* if do_time > TIW_SIZE we want to go around just once */ - count = (Uint)(dtime / TIW_SIZE) + 1; - keep_pos = (tiw_pos + dtime) % TIW_SIZE; - if (dtime > TIW_SIZE) dtime = TIW_SIZE; - - timeout_head = NULL; - timeout_tail = &timeout_head; - while (dtime > 0) { - /* this is to decrease the counters with the right amount */ - /* when dtime >= TIW_SIZE */ - if (tiw_pos == keep_pos) count--; - prev = &tiw[tiw_pos]; - while ((p = *prev) != NULL) { - ASSERT( p != p->next); - if (p->count < count) { /* we have a timeout */ - /* remove min time */ - if (tiw_min_ptr == p) { - tiw_min_ptr = NULL; - tiw_min = 0; - } + /* Don't want others here while we are bumping... */ + set_next_timeout(tiw, curr_time + ERTS_MONOTONIC_DAY, 0); + if (!tiw->at_once.head) { + timeout_head = NULL; + timeout_tail = &timeout_head; + } + else { + ASSERT(tiw->nto >= tiw->at_once.nto); + timeout_head = tiw->at_once.head; + timeout_tail = tiw->at_once.tail; + tiw->nto -= tiw->at_once.nto; + tiw->at_once.head = NULL; + tiw->at_once.tail = &tiw->at_once.head; + tiw->at_once.nto = 0; + } + + if (tiw->nto == 0) { + ERTS_DBG_CHK_SAFE_TO_SKIP_TO(tiw, bump_to); + tiw->pos = bump_to; + goto done; + } + + if (tiw->true_next_timeout_time) { + ErtsMonotonicTime skip_until_pos; + /* + * No need inspecting slots where we know no timeouts + * to trigger should reside. + */ + + skip_until_pos = ERTS_MONOTONIC_TO_CLKTCKS(tiw->next_timeout_time); + if (skip_until_pos > bump_to) + skip_until_pos = bump_to; + + ERTS_DBG_CHK_SAFE_TO_SKIP_TO(tiw, skip_until_pos); + ASSERT(skip_until_pos > tiw->pos); + + tiw->pos = skip_until_pos - 1; + } + + tiw_pos_ix = (int) ((tiw->pos+1) & (TIW_SIZE-1)); + tmp_slots = (bump_to - tiw->pos); + if (tmp_slots < (ErtsMonotonicTime) TIW_SIZE) + slots = (int) tmp_slots; + else + slots = TIW_SIZE; + + while (slots > 0) { + p = tiw->w[tiw_pos_ix]; + while (p) { + ErlTimer *next = p->next; + ASSERT(p != next); + if (p->timeout_pos <= bump_to) { /* we have a timeout */ /* Remove from list */ - remove_timer(p); + remove_timer(tiw, p); *timeout_tail = p; /* Insert in timeout queue */ timeout_tail = &p->next; } - else { - /* no timeout, just decrease counter */ - p->count -= count; - prev = &p->next; - } + p = next; } - tiw_pos = (tiw_pos + 1) % TIW_SIZE; - dtime--; + tiw_pos_ix++; + if (tiw_pos_ix == TIW_SIZE) + tiw_pos_ix = 0; + slots--; } - tiw_pos = keep_pos; - if (tiw_min_ptr) - tiw_min -= dt; - - erts_smp_mtx_unlock(&tiw_lock); + + ASSERT(tmp_slots >= (ErtsMonotonicTime) TIW_SIZE + || tiw_pos_ix == (int) ((bump_to+1) & (TIW_SIZE-1))); + + tiw->pos = bump_to; + + /* Search at most two seconds ahead... */ + (void) find_next_timeout(tiw, curr_time, ERTS_SEC_TO_MONOTONIC(2)); + +done: + + erts_smp_mtx_unlock(&tiw->lock); + erts_smp_atomic32_set_nob(&tiw->is_bumping, 0); + /* Call timedout timers callbacks */ while (timeout_head) { + ErlTimeoutProc timeout; + void *arg; p = timeout_head; timeout_head = p->next; /* Here comes hairy use of the timer fields! @@ -288,35 +420,69 @@ static ERTS_INLINE void bump_timer_internal(erts_short_time_t dt) /* PRE: tiw_lo * accesses any field until the ->timeout * callback is called. */ + ASSERT(p->timeout_pos <= bump_to); p->next = NULL; p->prev = NULL; p->slot = 0; - (*p->timeout)(p->arg); + timeout = p->timeout; + arg = p->arg; + (*timeout)(arg); } } -void erts_bump_timer(erts_short_time_t dt) /* dt is value from do_time */ +Uint +erts_timer_wheel_memory_size(void) +{ +#ifdef ERTS_SMP + return sizeof(ErtsTimerWheel)*(1 + erts_no_schedulers); +#else + return sizeof(ErtsTimerWheel); +#endif +} + +ErtsTimerWheel * +erts_create_timer_wheel(int no) { - erts_smp_mtx_lock(&tiw_lock); - bump_timer_internal(dt); + ErtsMonotonicTime mtime; + int i; + ErtsTimerWheel *tiw; + tiw = (ErtsTimerWheel *) erts_alloc(ERTS_ALC_T_TIMER_WHEEL, + sizeof(ErtsTimerWheel)); + for(i = 0; i < TIW_SIZE; i++) + tiw->w[i] = NULL; + + erts_smp_atomic32_init_nob(&tiw->is_bumping, 0); + erts_smp_mtx_init_x(&tiw->lock, "timer_wheel", make_small(no)); + + mtime = erts_get_monotonic_time(); + tiw->pos = ERTS_MONOTONIC_TO_CLKTCKS(mtime); + tiw->nto = 0; + tiw->at_once.head = NULL; + tiw->at_once.tail = &tiw->at_once.head; + tiw->at_once.nto = 0; + tiw->true_next_timeout_time = 0; + tiw->next_timeout_time = mtime + ERTS_MONOTONIC_DAY; + init_next_timeout(tiw, mtime + ERTS_MONOTONIC_DAY); + return tiw; } -Uint -erts_timer_wheel_memory_size(void) +ErtsNextTimeoutRef +erts_get_next_timeout_reference(ErtsTimerWheel *tiw) { - return (Uint) TIW_SIZE * sizeof(ErlTimer*); + return (ErtsNextTimeoutRef) &tiw->next_timeout; } + /* this routine links the time cells into a free list at the start and sets the time queue as empty */ void -erts_init_time(void) +erts_init_time(int time_correction, ErtsTimeWarpMode time_warp_mode) { - int i, itime; + int itime; /* system dependent init; must be done before do_time_init() if timer thread is enabled */ - itime = erts_init_time_sup(); + itime = erts_init_time_sup(time_correction, time_warp_mode); #ifdef TIW_ITIME_IS_CONSTANT if (itime != TIW_ITIME) { erl_exit(ERTS_ABORT_EXIT, "timer resolution mismatch %d != %d", itime, TIW_ITIME); @@ -325,117 +491,111 @@ erts_init_time(void) tiw_itime = itime; #endif - erts_smp_mtx_init(&tiw_lock, "timer_wheel"); - - tiw = (ErlTimer**) erts_alloc(ERTS_ALC_T_TIMER_WHEEL, - TIW_SIZE * sizeof(ErlTimer*)); - for(i = 0; i < TIW_SIZE; i++) - tiw[i] = NULL; - do_time_init(); - tiw_pos = tiw_nto = 0; - tiw_min_ptr = NULL; - tiw_min = 0; + erts_default_timer_wheel = erts_create_timer_wheel(0); } +void +erts_set_timer(ErlTimer *p, ErlTimeoutProc timeout, + ErlCancelProc cancel, void *arg, Uint to) +{ + ErtsMonotonicTime timeout_time, timeout_pos; + ErtsMonotonicTime curr_time; + ErtsTimerWheel *tiw; + ErtsSchedulerData *esdp; + + curr_time = erts_get_monotonic_time(); + esdp = erts_get_scheduler_data(); + if (esdp) + tiw = esdp->timer_wheel; + else + tiw = erts_default_timer_wheel; + erts_smp_mtx_lock(&tiw->lock); + if (get_timer_wheel(p)) + ERTS_INTERNAL_ERROR("Double set timer"); -/* -** Insert a process into the time queue, with a timeout 't' -*/ -static void -insert_timer(ErlTimer* p, Uint t) -{ - Uint tm; - Uint64 ticks; + p->timeout = timeout; + p->cancel = cancel; + p->arg = arg; - /* The current slot (tiw_pos) in timing wheel is the next slot to be - * be processed. Hence no extra time tick is needed. - * - * (x + y - 1)/y is precisely the "number of bins" formula. - */ - ticks = (t + (TIW_ITIME - 1)) / TIW_ITIME; + if (to == 0) { + timeout_pos = ERTS_MONOTONIC_TO_CLKTCKS(curr_time); + tiw->nto++; + tiw->at_once.nto++; + *tiw->at_once.tail = p; + tiw->at_once.tail = &p->next; + p->next = NULL; + p->timeout_pos = timeout_pos; + timeout_time = ERTS_CLKTCKS_TO_MONOTONIC(timeout_pos); + } + else { + int tm; + ErtsMonotonicTime ticks; - /* - * Ticks must be a Uint64, or the addition may overflow here, - * resulting in an incorrect value for p->count below. - */ - ticks += do_time_update(); /* Add backlog of unprocessed time */ - - /* calculate slot */ - tm = (ticks + tiw_pos) % TIW_SIZE; - p->slot = (Uint) tm; - p->count = (Uint) (ticks / TIW_SIZE); + ticks = ERTS_MSEC_TO_CLKTCKS(to); + timeout_pos = ERTS_MONOTONIC_TO_CLKTCKS(curr_time - 1) + 1 + ticks; + + /* calculate slot */ + tm = (int) (timeout_pos & (TIW_SIZE-1)); + p->slot = (Uint) tm; - /* insert at head of list at slot */ - p->next = tiw[tm]; - p->prev = NULL; - if (p->next != NULL) - p->next->prev = p; - tiw[tm] = p; + /* insert at head of list at slot */ + p->next = tiw->w[tm]; + p->prev = NULL; + if (p->next != NULL) + p->next->prev = p; + tiw->w[tm] = p; + tiw->nto++; - /* insert min time */ - if ((tiw_nto == 0) || ((tiw_min_ptr != NULL) && (ticks < tiw_min))) { - tiw_min = ticks; - tiw_min_ptr = p; - } - if ((tiw_min_ptr == p) && (ticks > tiw_min)) { - /* some other timer might be 'min' now */ - tiw_min = 0; - tiw_min_ptr = NULL; + timeout_time = ERTS_CLKTCKS_TO_MONOTONIC(timeout_pos); + p->timeout_pos = timeout_pos; + + ASSERT(ERTS_MSEC_TO_MONOTONIC(to) <= timeout_time - curr_time); + ASSERT(timeout_time - curr_time + < ERTS_MSEC_TO_MONOTONIC(to) + ERTS_CLKTCKS_TO_MONOTONIC(1)); } - tiw_nto++; -} + if (timeout_time < tiw->next_timeout_time) + set_next_timeout(tiw, timeout_time, 1); -void -erts_set_timer(ErlTimer* p, ErlTimeoutProc timeout, ErlCancelProc cancel, - void* arg, Uint t) -{ + set_timer_wheel(p, tiw); + + erts_smp_mtx_unlock(&tiw->lock); - erts_deliver_time(); - erts_smp_mtx_lock(&tiw_lock); - if (p->active) { /* XXX assert ? */ - erts_smp_mtx_unlock(&tiw_lock); - return; - } - p->timeout = timeout; - p->cancel = cancel; - p->arg = arg; - p->active = 1; - insert_timer(p, t); - erts_smp_mtx_unlock(&tiw_lock); #if defined(ERTS_SMP) - if (t <= (Uint) ERTS_SHORT_TIME_T_MAX) - erts_sys_schedule_interrupt_timed(1, (erts_short_time_t) t); + if (tiw == erts_default_timer_wheel) + erts_interupt_aux_thread_timed(timeout_time); #endif + } void -erts_cancel_timer(ErlTimer* p) +erts_cancel_timer(ErlTimer *p) { - erts_smp_mtx_lock(&tiw_lock); - if (!p->active) { /* allow repeated cancel (drivers) */ - erts_smp_mtx_unlock(&tiw_lock); + ErtsTimerWheel *tiw; + ErlCancelProc cancel; + void *arg = NULL; /* Shut up faulty warning... */ + + tiw = get_timer_wheel(p); + if (!tiw) return; - } + + erts_smp_mtx_lock(&tiw->lock); + if (tiw != get_timer_wheel(p)) + cancel = NULL; + else { + remove_timer(tiw, p); + p->slot = 0; - /* is it the 'min' timer, remove min */ - if (p == tiw_min_ptr) { - tiw_min_ptr = NULL; - tiw_min = 0; + cancel = p->cancel; + arg = p->arg; } + erts_smp_mtx_unlock(&tiw->lock); - remove_timer(p); - p->slot = p->count = 0; - - if (p->cancel != NULL) { - erts_smp_mtx_unlock(&tiw_lock); - (*p->cancel)(p->arg); - return; - } - erts_smp_mtx_unlock(&tiw_lock); + if (cancel) + (*cancel)(arg); } /* @@ -447,59 +607,58 @@ erts_cancel_timer(ErlTimer* p) Uint erts_time_left(ErlTimer *p) { - Uint left; - erts_short_time_t dt; + ErtsTimerWheel *tiw; + ErtsMonotonicTime current_time, timeout_time; - erts_smp_mtx_lock(&tiw_lock); - - if (!p->active) { - erts_smp_mtx_unlock(&tiw_lock); + tiw = get_timer_wheel(p); + if (!tiw) return 0; - } - if (p->slot < tiw_pos) - left = (p->count + 1) * TIW_SIZE + p->slot - tiw_pos; + erts_smp_mtx_lock(&tiw->lock); + if (tiw != get_timer_wheel(p)) + timeout_time = ERTS_MONOTONIC_TIME_MIN; else - left = p->count * TIW_SIZE + p->slot - tiw_pos; - dt = do_time_read(); - if (left < dt) - left = 0; - else - left -= dt; - - erts_smp_mtx_unlock(&tiw_lock); + timeout_time = ERTS_CLKTCKS_TO_MONOTONIC(p->timeout_pos); + erts_smp_mtx_unlock(&tiw->lock); - return (Uint) left * TIW_ITIME; + current_time = erts_get_monotonic_time(); + if (timeout_time <= current_time) + return 0; + return (Uint) ERTS_MONOTONIC_TO_MSEC(timeout_time - current_time); } #ifdef DEBUG void erts_p_slpq(void) { + ErtsTimerWheel *tiw = erts_default_timer_wheel; + ErtsMonotonicTime current_time = erts_get_monotonic_time(); int i; ErlTimer* p; - erts_smp_mtx_lock(&tiw_lock); + erts_smp_mtx_lock(&tiw->lock); /* print the whole wheel, starting at the current position */ - erts_printf("\ntiw_pos = %d tiw_nto %d\n", tiw_pos, tiw_nto); - i = tiw_pos; - if (tiw[i] != NULL) { + erts_printf("\ncurrent time = %bps tiw_pos = %d tiw_nto %d\n", + current_time, tiw->pos, tiw->nto); + i = tiw->pos; + if (tiw->w[i] != NULL) { erts_printf("%d:\n", i); - for(p = tiw[i]; p != NULL; p = p->next) { - erts_printf(" (count %d, slot %d)\n", - p->count, p->slot); + for(p = tiw->w[i]; p != NULL; p = p->next) { + erts_printf(" (timeout time %bps, slot %d)\n", + ERTS_CLKTCKS_TO_MONOTONIC(p->timeout_pos), + p->slot); } } - for(i = (i+1)%TIW_SIZE; i != tiw_pos; i = (i+1)%TIW_SIZE) { - if (tiw[i] != NULL) { + for(i = ((i+1) & (TIW_SIZE-1)); i != (tiw->pos & (TIW_SIZE-1)); i = ((i+1) & (TIW_SIZE-1))) { + if (tiw->w[i] != NULL) { erts_printf("%d:\n", i); - for(p = tiw[i]; p != NULL; p = p->next) { - erts_printf(" (count %d, slot %d)\n", - p->count, p->slot); + for(p = tiw->w[i]; p != NULL; p = p->next) { + erts_printf(" (timeout time %bps, slot %d)\n", + ERTS_CLKTCKS_TO_MONOTONIC(p->timeout_pos), p->slot); } } } - erts_smp_mtx_unlock(&tiw_lock); + erts_smp_mtx_unlock(&tiw->lock); } #endif /* DEBUG */ diff --git a/erts/emulator/beam/utils.c b/erts/emulator/beam/utils.c index b341c4d949..8fc8962e4f 100644 --- a/erts/emulator/beam/utils.c +++ b/erts/emulator/beam/utils.c @@ -49,6 +49,7 @@ #include "beam_bp.h" #include "erl_ptab.h" #include "erl_check_io.h" +#include "erl_bif_unique.h" #ifdef HIPE # include "hipe_mode_switch.h" #endif @@ -190,12 +191,18 @@ erts_set_hole_marker(Eterm* ptr, Uint sz) * Helper function for the ESTACK macros defined in global.h. */ void -erl_grow_estack(ErtsEStack* s, Eterm* default_estack) +erl_grow_estack(ErtsEStack* s, Uint need) { Uint old_size = (s->end - s->start); - Uint new_size = old_size * 2; + Uint new_size; Uint sp_offs = s->sp - s->start; - if (s->start != default_estack) { + + if (need < old_size) + new_size = 2*old_size; + else + new_size = ((need / old_size) + 2) * old_size; + + if (s->start != s->edefault) { s->start = erts_realloc(s->alloc_type, s->start, new_size*sizeof(Eterm)); } else { @@ -210,12 +217,18 @@ erl_grow_estack(ErtsEStack* s, Eterm* default_estack) * Helper function for the WSTACK macros defined in global.h. */ void -erl_grow_wstack(ErtsWStack* s, UWord* default_wstack) +erl_grow_wstack(ErtsWStack* s, Uint need) { Uint old_size = (s->wend - s->wstart); - Uint new_size = old_size * 2; + Uint new_size; Uint sp_offs = s->wsp - s->wstart; - if (s->wstart != default_wstack) { + + if (need < old_size) + new_size = 2 * old_size; + else + new_size = ((need / old_size) + 2) * old_size; + + if (s->wstart != s->wdefault) { s->wstart = erts_realloc(s->alloc_type, s->wstart, new_size*sizeof(UWord)); } else { @@ -227,6 +240,32 @@ erl_grow_wstack(ErtsWStack* s, UWord* default_wstack) s->wsp = s->wstart + sp_offs; } +/* + * Helper function for the PSTACK macros defined in global.h. + */ +void +erl_grow_pstack(ErtsPStack* s, void* default_pstack, unsigned need_bytes) +{ + Uint old_size = s->pend - s->pstart; + Uint new_size; + Uint sp_offs = s->psp - s->pstart; + + if (need_bytes < old_size) + new_size = 2 * old_size; + else + new_size = ((need_bytes / old_size) + 2) * old_size; + + if (s->pstart != default_pstack) { + s->pstart = erts_realloc(s->alloc_type, s->pstart, new_size); + } else { + byte* new_ptr = erts_alloc(s->alloc_type, new_size); + sys_memcpy(new_ptr, s->pstart, old_size); + s->pstart = new_ptr; + } + s->pend = s->pstart + new_size; + s->psp = s->pstart + sp_offs; +} + /* CTYPE macros */ #define LATIN1 @@ -314,6 +353,17 @@ int erts_fit_in_bits_int32(Sint32 value) return fit_in_bits((Sint64) (Uint32) value, 4); } +int erts_fit_in_bits_uint(Uint value) +{ +#if ERTS_SIZEOF_ETERM == 4 + return fit_in_bits((Sint64) (Uint32) value, 4); +#elif ERTS_SIZEOF_ETERM == 8 + return fit_in_bits(value, 5); +#else +# error "No way, Jose" +#endif +} + int erts_print(int to, void *arg, char *format, ...) { @@ -710,7 +760,7 @@ erts_bld_atom_2uint_3tup_list(Uint **hpp, Uint *szp, Sint length, ** If N < 0, Y = FUNNY_NUMBER4 else Y = FUNNY_NUMBER3. ** The hash value is Y*h(J) mod 2^32 where h(J) is calculated like ** h(0) = <initial hash> -** h(i) = h(i-i)*X + B(i-1) +** h(i) = h(i-1)*X + B(i-1) ** The above should hold regardless of internal representation. ** Pids are hashed like small numbers but with differrent constants, as are ** ports. @@ -792,10 +842,10 @@ Uint32 make_hash(Eterm term_arg) unsigned op; /* Must not collide with the real tag_val_def's: */ -#define MAKE_HASH_TUPLE_OP 0x11 -#define MAKE_HASH_TERM_ARRAY_OP 0x12 -#define MAKE_HASH_CDR_PRE_OP 0x13 -#define MAKE_HASH_CDR_POST_OP 0x14 +#define MAKE_HASH_TUPLE_OP (FIRST_VACANT_TAG_DEF) +#define MAKE_HASH_TERM_ARRAY_OP (FIRST_VACANT_TAG_DEF+1) +#define MAKE_HASH_CDR_PRE_OP (FIRST_VACANT_TAG_DEF+2) +#define MAKE_HASH_CDR_POST_OP (FIRST_VACANT_TAG_DEF+3) /* ** Convenience macro for calculating a bytewise hash on an unsigned 32 bit @@ -975,23 +1025,8 @@ tail_recur: break; } case MAP_DEF: - { - map_t *mp = (map_t *)map_val(term); - int size = map_get_size(mp); - Eterm *ks = map_get_keys(mp); - Eterm *vs = map_get_values(mp); - - /* Use a prime with size to remedy some of - * the {} and <<>> hash problems */ - hash = hash*FUNNY_NUMBER13 + FUNNY_NUMBER14 + size; - if (size == 0) - break; - - /* push values first */ - WSTACK_PUSH3(stack, (UWord)vs, (UWord) size, MAKE_HASH_TERM_ARRAY_OP); - WSTACK_PUSH3(stack, (UWord)ks, (UWord) size, MAKE_HASH_TERM_ARRAY_OP); - break; - } + hash = hash*FUNNY_NUMBER13 + FUNNY_NUMBER14 + make_hash2(term); + break; case TUPLE_DEF: { Eterm* ptr = tuple_val(term); @@ -1095,10 +1130,11 @@ Uint32 make_hash2(Eterm term) { Uint32 hash; - Uint32 hash_xor_keys = 0; - Uint32 hash_xor_values = 0; + Uint32 hash_xor_pairs; DeclareTmpHeapNoproc(tmp_big,2); + ERTS_UNDEF(hash_xor_pairs, 0); + /* (HCONST * {2, ..., 16}) mod 2^32 */ #define HCONST_2 0x3c6ef372UL #define HCONST_3 0xdaa66d2bUL @@ -1115,10 +1151,15 @@ make_hash2(Eterm term) #define HCONST_14 0xa708a81eUL #define HCONST_15 0x454021d7UL #define HCONST_16 0xe3779b90UL +#define HCONST_17 0x81af1549UL +#define HCONST_18 0x1fe68f02UL +#define HCONST_19 0xbe1e08bbUL +#define HCONST_20 0x5c558274UL +#define HCONST_21 0xfa8cfc2dUL #define HASH_MAP_TAIL (_make_header(1,_TAG_HEADER_REF)) -#define HASH_MAP_KEY (_make_header(2,_TAG_HEADER_REF)) -#define HASH_MAP_VAL (_make_header(3,_TAG_HEADER_REF)) +#define HASH_MAP_PAIR (_make_header(2,_TAG_HEADER_REF)) +#define HASH_CDR (_make_header(3,_TAG_HEADER_REF)) #define UINT32_HASH_2(Expr1, Expr2, AConst) \ do { \ @@ -1141,6 +1182,13 @@ make_hash2(Eterm term) } while(0) #define IS_SSMALL28(x) (((Uint) (((x) >> (28-1)) + 1)) < 2) + +#ifdef ARCH_64 +# define POINTER_HASH(Ptr, AConst) UINT32_HASH_2((Uint32)(UWord)(Ptr), (((UWord)(Ptr)) >> 32), AConst) +#else +# define POINTER_HASH(Ptr, AConst) UINT32_HASH(Ptr, AConst) +#endif + /* Optimization. Simple cases before declaration of estack. */ if (primary_tag(term) == TAG_PRIMARY_IMMED1) { switch (term & _TAG_IMMED1_MASK) { @@ -1195,9 +1243,9 @@ make_hash2(Eterm term) if (c > 0) UINT32_HASH(sh, HCONST_4); if (is_list(term)) { - term = *ptr; - tmp = *++ptr; - ESTACK_PUSH(s, tmp); + tmp = CDR(ptr); + ESTACK_PUSH(s, tmp); + term = CAR(ptr); } } break; @@ -1214,46 +1262,90 @@ make_hash2(Eterm term) UINT32_HASH(arity, HCONST_9); if (arity == 0) /* Empty tuple */ goto hash2_common; - for (i = arity; i >= 1; i--) { - tmp = elem[i]; - ESTACK_PUSH(s, tmp); - } - goto hash2_common; - } - break; - case MAP_SUBTAG: - { - map_t *mp = (map_t *)map_val(term); - int i; - int size = map_get_size(mp); - Eterm *ks = map_get_keys(mp); - Eterm *vs = map_get_values(mp); - UINT32_HASH(size, HCONST_16); - if (size == 0) { - goto hash2_common; - } - ESTACK_PUSH4(s, hash_xor_values, hash_xor_keys, hash, HASH_MAP_TAIL); - hash = 0; - hash_xor_keys = 0; - hash_xor_values = 0; - for (i = size - 1; i >= 0; i--) { - tmp = vs[i]; - ESTACK_PUSH2(s, HASH_MAP_VAL, tmp); + for (i = arity; ; i--) { + term = elem[i]; + if (i == 1) + break; + ESTACK_PUSH(s, term); } - /* We do not want to expose the tuple representation. - * Do not push the keys as a tuple. - */ - for (i = size - 1; i >= 0; i--) { - tmp = ks[i]; - ESTACK_PUSH2(s, HASH_MAP_KEY, tmp); - } - goto hash2_common; } break; + case MAP_SUBTAG: + { + Eterm* ptr = boxed_val(term) + 1; + Uint size; + int i; + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_FLATMAP: + { + flatmap_t *mp = (flatmap_t *)flatmap_val(term); + Eterm *ks = flatmap_get_keys(mp); + Eterm *vs = flatmap_get_values(mp); + size = flatmap_get_size(mp); + UINT32_HASH(size, HCONST_16); + if (size == 0) + goto hash2_common; + + /* We want a portable hash function that is *independent* of + * the order in which keys and values are encountered. + * We therefore calculate context independent hashes for all . + * key-value pairs and then xor them together. + */ + ESTACK_PUSH(s, hash_xor_pairs); + ESTACK_PUSH(s, hash); + ESTACK_PUSH(s, HASH_MAP_TAIL); + hash = 0; + hash_xor_pairs = 0; + for (i = size - 1; i >= 0; i--) { + ESTACK_PUSH(s, HASH_MAP_PAIR); + ESTACK_PUSH(s, vs[i]); + ESTACK_PUSH(s, ks[i]); + } + goto hash2_common; + } + + case HAMT_SUBTAG_HEAD_ARRAY: + case HAMT_SUBTAG_HEAD_BITMAP: + size = *ptr++; + UINT32_HASH(size, HCONST_16); + if (size == 0) + goto hash2_common; + ESTACK_PUSH(s, hash_xor_pairs); + ESTACK_PUSH(s, hash); + ESTACK_PUSH(s, HASH_MAP_TAIL); + hash = 0; + hash_xor_pairs = 0; + } + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + i = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + case HAMT_SUBTAG_NODE_BITMAP: + i = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + break; + default: + erl_exit(1, "bad header"); + } + while (i) { + if (is_list(*ptr)) { + Eterm* cons = list_val(*ptr); + ESTACK_PUSH(s, HASH_MAP_PAIR); + ESTACK_PUSH(s, CDR(cons)); + ESTACK_PUSH(s, CAR(cons)); + } + else { + ASSERT(is_boxed(*ptr)); + ESTACK_PUSH(s, *ptr); + } + i--; ptr++; + } + goto hash2_common; + } + break; case EXPORT_SUBTAG: { Export* ep = *((Export **) (export_val(term) + 1)); - UINT32_HASH_2 (ep->code[2], atom_tab(atom_val(ep->code[0]))->slot.bucket.hvalue, @@ -1268,7 +1360,6 @@ make_hash2(Eterm term) { ErlFunThing* funp = (ErlFunThing *) fun_val(term); Uint num_free = funp->num_free; - UINT32_HASH_2 (num_free, atom_tab(atom_val(funp->fe->module))->slot.bucket.hvalue, @@ -1349,7 +1440,8 @@ make_hash2(Eterm term) do { Uint t; Uint32 x, y; - t = i < n ? BIG_DIGIT(ptr, i++) : 0; + ASSERT(i < n); + t = BIG_DIGIT(ptr, i++); x = t & 0xffffffff; y = t >> 32; UINT32_HASH_2(x, y, con); @@ -1457,20 +1549,395 @@ make_hash2(Eterm term) switch (term) { case HASH_MAP_TAIL: { hash = (Uint32) ESTACK_POP(s); - UINT32_HASH(hash_xor_keys, HCONST_16); - UINT32_HASH(hash_xor_values, HCONST_16); - hash_xor_keys = (Uint32) ESTACK_POP(s); - hash_xor_values = (Uint32) ESTACK_POP(s); + UINT32_HASH(hash_xor_pairs, HCONST_19); + hash_xor_pairs = (Uint32) ESTACK_POP(s); goto hash2_common; } - case HASH_MAP_KEY: - hash_xor_keys ^= hash; + case HASH_MAP_PAIR: + hash_xor_pairs ^= hash; hash = 0; goto hash2_common; - case HASH_MAP_VAL: - hash_xor_values ^= hash; + default: + break; + } + } + } + } +} + +/* Term hash function for internal use. + * + * Limitation #1: Is not "portable" in any way between different VM instances. + * + * Limitation #2: The hash value is only valid as long as the term exists + * somewhere in the VM. Why? Because external pids, ports and refs are hashed + * by mixing the node *pointer* value. If a node disappears and later reappears + * with a new ErlNode struct, externals from that node will hash different than + * before. + * + * One IMPORTANT property must hold (for hamt). + * EVERY BIT of the term that is significant for equality (see EQ) + * MUST BE USED AS INPUT FOR THE HASH. Two different terms must always have a + * chance of hashing different when salted: hash([Salt|A]) vs hash([Salt|B]). + * + * This is why we can not use cached hash values for atoms for example. + * + */ + +#define CONST_HASH(AConst) \ +do { /* Lightweight mixing of constant (type info) */ \ + hash ^= AConst; \ + hash = (hash << 17) ^ (hash >> (32-17)); \ +} while (0) + +Uint32 +make_internal_hash(Eterm term) +{ + Uint32 hash; + Uint32 hash_xor_pairs; + + ERTS_UNDEF(hash_xor_pairs, 0); + + /* Optimization. Simple cases before declaration of estack. */ + if (primary_tag(term) == TAG_PRIMARY_IMMED1) { + hash = 0; + #if ERTS_SIZEOF_ETERM == 8 + UINT32_HASH_2((Uint32)term, (Uint32)(term >> 32), HCONST); + #elif ERTS_SIZEOF_ETERM == 4 + UINT32_HASH(term, HCONST); + #else + # error "No you don't" + #endif + return hash; + } + { + Eterm tmp; + DECLARE_ESTACK(s); + + UseTmpHeapNoproc(2); + hash = 0; + for (;;) { + switch (primary_tag(term)) { + case TAG_PRIMARY_LIST: + { + int c = 0; + Uint32 sh = 0; + Eterm* ptr = list_val(term); + while (is_byte(*ptr)) { + /* Optimization for strings. */ + sh = (sh << 8) + unsigned_val(*ptr); + if (c == 3) { + UINT32_HASH(sh, HCONST_4); + c = sh = 0; + } else { + c++; + } + term = CDR(ptr); + if (is_not_list(term)) + break; + ptr = list_val(term); + } + if (c > 0) + UINT32_HASH(sh, HCONST_4); + if (is_list(term)) { + tmp = CDR(ptr); + CONST_HASH(HCONST_17); /* Hash CAR in cons cell */ + ESTACK_PUSH(s, tmp); + if (is_not_list(tmp)) { + ESTACK_PUSH(s, HASH_CDR); + } + term = CAR(ptr); + } + } + break; + case TAG_PRIMARY_BOXED: + { + Eterm hdr = *boxed_val(term); + ASSERT(is_header(hdr)); + switch (hdr & _TAG_HEADER_MASK) { + case ARITYVAL_SUBTAG: + { + int i; + int arity = header_arity(hdr); + Eterm* elem = tuple_val(term); + UINT32_HASH(arity, HCONST_9); + if (arity == 0) /* Empty tuple */ + goto pop_next; + for (i = arity; ; i--) { + term = elem[i]; + if (i == 1) + break; + ESTACK_PUSH(s, term); + } + } + break; + + case MAP_SUBTAG: + { + Eterm* ptr = boxed_val(term) + 1; + Uint size; + int i; + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_FLATMAP: + { + flatmap_t *mp = (flatmap_t *)flatmap_val(term); + Eterm *ks = flatmap_get_keys(mp); + Eterm *vs = flatmap_get_values(mp); + size = flatmap_get_size(mp); + UINT32_HASH(size, HCONST_16); + if (size == 0) + goto pop_next; + + /* We want a hash function that is *independent* of + * the order in which keys and values are encountered. + * We therefore calculate context independent hashes for all . + * key-value pairs and then xor them together. + */ + ESTACK_PUSH(s, hash_xor_pairs); + ESTACK_PUSH(s, hash); + ESTACK_PUSH(s, HASH_MAP_TAIL); + hash = 0; + hash_xor_pairs = 0; + for (i = size - 1; i >= 0; i--) { + ESTACK_PUSH(s, HASH_MAP_PAIR); + ESTACK_PUSH(s, vs[i]); + ESTACK_PUSH(s, ks[i]); + } + goto pop_next; + } + case HAMT_SUBTAG_HEAD_ARRAY: + case HAMT_SUBTAG_HEAD_BITMAP: + size = *ptr++; + UINT32_HASH(size, HCONST_16); + if (size == 0) + goto pop_next; + ESTACK_PUSH(s, hash_xor_pairs); + ESTACK_PUSH(s, hash); + ESTACK_PUSH(s, HASH_MAP_TAIL); + hash = 0; + hash_xor_pairs = 0; + } + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + i = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + case HAMT_SUBTAG_NODE_BITMAP: + i = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + break; + default: + erl_exit(1, "bad header"); + } + while (i) { + if (is_list(*ptr)) { + Eterm* cons = list_val(*ptr); + ESTACK_PUSH(s, HASH_MAP_PAIR); + ESTACK_PUSH(s, CDR(cons)); + ESTACK_PUSH(s, CAR(cons)); + } + else { + ASSERT(is_boxed(*ptr)); + ESTACK_PUSH(s, *ptr); + } + i--; ptr++; + } + goto pop_next; + } + break; + case EXPORT_SUBTAG: + { + Export* ep = *((Export **) (export_val(term) + 1)); + /* Assumes Export entries never moves */ + POINTER_HASH(ep, HCONST_14); + goto pop_next; + } + + case FUN_SUBTAG: + { + ErlFunThing* funp = (ErlFunThing *) fun_val(term); + Uint num_free = funp->num_free; + UINT32_HASH_2(num_free, funp->fe->module, HCONST_20); + UINT32_HASH_2(funp->fe->old_index, funp->fe->old_uniq, HCONST_21); + if (num_free == 0) { + goto pop_next; + } else { + Eterm* bptr = funp->env + num_free - 1; + while (num_free-- > 1) { + term = *bptr--; + ESTACK_PUSH(s, term); + } + term = *bptr; + } + } + break; + case REFC_BINARY_SUBTAG: + case HEAP_BINARY_SUBTAG: + case SUB_BINARY_SUBTAG: + { + byte* bptr; + unsigned sz = binary_size(term); + Uint32 con = HCONST_13 + hash; + Uint bitoffs; + Uint bitsize; + + ERTS_GET_BINARY_BYTES(term, bptr, bitoffs, bitsize); + if (sz == 0 && bitsize == 0) { + hash = con; + } else { + if (bitoffs == 0) { + hash = block_hash(bptr, sz, con); + if (bitsize > 0) { + UINT32_HASH_2(bitsize, (bptr[sz] >> (8 - bitsize)), + HCONST_15); + } + } else { + byte* buf = (byte *) erts_alloc(ERTS_ALC_T_TMP, + sz + (bitsize != 0)); + erts_copy_bits(bptr, bitoffs, 1, buf, 0, 1, sz*8+bitsize); + hash = block_hash(buf, sz, con); + if (bitsize > 0) { + UINT32_HASH_2(bitsize, (buf[sz] >> (8 - bitsize)), + HCONST_15); + } + erts_free(ERTS_ALC_T_TMP, (void *) buf); + } + } + goto pop_next; + } + break; + case POS_BIG_SUBTAG: + case NEG_BIG_SUBTAG: + { + Eterm* ptr = big_val(term); + Uint i = 0; + Uint n = BIG_SIZE(ptr); + Uint32 con = BIG_SIGN(ptr) ? HCONST_10 : HCONST_11; +#if D_EXP == 16 + do { + Uint32 x, y; + x = i < n ? BIG_DIGIT(ptr, i++) : 0; + x += (Uint32)(i < n ? BIG_DIGIT(ptr, i++) : 0) << 16; + y = i < n ? BIG_DIGIT(ptr, i++) : 0; + y += (Uint32)(i < n ? BIG_DIGIT(ptr, i++) : 0) << 16; + UINT32_HASH_2(x, y, con); + } while (i < n); +#elif D_EXP == 32 + do { + Uint32 x, y; + x = i < n ? BIG_DIGIT(ptr, i++) : 0; + y = i < n ? BIG_DIGIT(ptr, i++) : 0; + UINT32_HASH_2(x, y, con); + } while (i < n); +#elif D_EXP == 64 + do { + Uint t; + Uint32 x, y; + ASSERT(i < n); + t = BIG_DIGIT(ptr, i++); + x = t & 0xffffffff; + y = t >> 32; + UINT32_HASH_2(x, y, con); + } while (i < n); +#else +#error "unsupported D_EXP size" +#endif + goto pop_next; + } + break; + case REF_SUBTAG: + UINT32_HASH(internal_ref_numbers(term)[0], HCONST_7); + ASSERT(internal_ref_no_of_numbers(term) == 3); + UINT32_HASH_2(internal_ref_numbers(term)[1], + internal_ref_numbers(term)[2], HCONST_8); + goto pop_next; + + case EXTERNAL_REF_SUBTAG: + { + ExternalThing* thing = external_thing_ptr(term); + + ASSERT(external_thing_ref_no_of_numbers(thing) == 3); + /* See limitation #2 */ + #ifdef ARCH_64 + POINTER_HASH(thing->node, HCONST_7); + UINT32_HASH(external_thing_ref_numbers(thing)[0], HCONST_7); + #else + UINT32_HASH_2(thing->node, + external_thing_ref_numbers(thing)[0], HCONST_7); + #endif + UINT32_HASH_2(external_thing_ref_numbers(thing)[1], + external_thing_ref_numbers(thing)[2], HCONST_8); + goto pop_next; + } + case EXTERNAL_PID_SUBTAG: { + ExternalThing* thing = external_thing_ptr(term); + /* See limitation #2 */ + #ifdef ARCH_64 + POINTER_HASH(thing->node, HCONST_5); + UINT32_HASH(thing->data.ui[0], HCONST_5); + #else + UINT32_HASH_2(thing->node, thing->data.ui[0], HCONST_5); + #endif + goto pop_next; + } + case EXTERNAL_PORT_SUBTAG: { + ExternalThing* thing = external_thing_ptr(term); + /* See limitation #2 */ + #ifdef ARCH_64 + POINTER_HASH(thing->node, HCONST_6); + UINT32_HASH(thing->data.ui[0], HCONST_6); + #else + UINT32_HASH_2(thing->node, thing->data.ui[0], HCONST_6); + #endif + goto pop_next; + } + case FLOAT_SUBTAG: + { + FloatDef ff; + GET_DOUBLE(term, ff); + UINT32_HASH_2(ff.fw[0], ff.fw[1], HCONST_12); + goto pop_next; + } + + default: + erl_exit(1, "Invalid tag in make_hash2(0x%X)\n", term); + } + } + break; + case TAG_PRIMARY_IMMED1: + #if ERTS_SIZEOF_ETERM == 8 + UINT32_HASH_2((Uint32)term, (Uint32)(term >> 32), HCONST); + #else + UINT32_HASH(term, HCONST); + #endif + goto pop_next; + + default: + erl_exit(1, "Invalid tag in make_hash2(0x%X)\n", term); + + pop_next: + if (ESTACK_ISEMPTY(s)) { + DESTROY_ESTACK(s); + UnUseTmpHeapNoproc(2); + return hash; + } + + term = ESTACK_POP(s); + + switch (term) { + case HASH_MAP_TAIL: { + hash = (Uint32) ESTACK_POP(s); + UINT32_HASH(hash_xor_pairs, HCONST_19); + hash_xor_pairs = (Uint32) ESTACK_POP(s); + goto pop_next; + } + case HASH_MAP_PAIR: + hash_xor_pairs ^= hash; hash = 0; - goto hash2_common; + goto pop_next; + + case HASH_CDR: + CONST_HASH(HCONST_18); /* Hash CDR i cons cell */ + goto pop_next; default: break; } @@ -1478,9 +1945,10 @@ make_hash2(Eterm term) } } +#undef CONST_HASH #undef HASH_MAP_TAIL -#undef HASH_MAP_KEY -#undef HASH_MAP_VAL +#undef HASH_MAP_PAIR +#undef HASH_CDR #undef UINT32_HASH_2 #undef UINT32_HASH @@ -1697,23 +2165,8 @@ tail_recur: break; case MAP_DEF: - { - map_t *mp = (map_t *)map_val(term); - int size = map_get_size(mp); - Eterm *ks = map_get_keys(mp); - Eterm *vs = map_get_values(mp); - - /* Use a prime with size to remedy some of - * the {} and <<>> hash problems */ - hash = hash*FUNNY_NUMBER13 + FUNNY_NUMBER14 + size; - if (size == 0) - break; - - /* push values first */ - WSTACK_PUSH3(stack, (UWord)vs, (UWord) size, MAKE_HASH_TERM_ARRAY_OP); - WSTACK_PUSH3(stack, (UWord)ks, (UWord) size, MAKE_HASH_TERM_ARRAY_OP); - break; - } + hash = hash*FUNNY_NUMBER13 + FUNNY_NUMBER14 + make_hash2(term); + break; case TUPLE_DEF: { Eterm* ptr = tuple_val(term); @@ -2118,22 +2571,6 @@ tailrecur_ne: ++bb; goto term_array; } - case MAP_SUBTAG: - { - aa = map_val_rel(a, a_base); - if (!is_boxed(b) || *boxed_val_rel(b,b_base) != *aa) - goto not_equal; - bb = map_val_rel(b,b_base); - sz = map_get_size((map_t*)aa); - - if (sz != map_get_size((map_t*)bb)) goto not_equal; - if (sz == 0) goto pop_next; - - aa += 2; - bb += 2; - sz += 1; /* increment for tuple-keys */ - goto term_array; - } case REFC_BINARY_SUBTAG: case HEAP_BINARY_SUBTAG: case SUB_BINARY_SUBTAG: @@ -2325,6 +2762,46 @@ tailrecur_ne: } break; /* not equal */ } + case MAP_SUBTAG: + if (is_flatmap_rel(a, a_base)) { + aa = flatmap_val_rel(a, a_base); + if (!is_boxed(b) || *boxed_val_rel(b,b_base) != *aa) + goto not_equal; + bb = flatmap_val_rel(b,b_base); + sz = flatmap_get_size((flatmap_t*)aa); + + if (sz != flatmap_get_size((flatmap_t*)bb)) goto not_equal; + if (sz == 0) goto pop_next; + + aa += 2; + bb += 2; + sz += 1; /* increment for tuple-keys */ + goto term_array; + + } else { + if (!is_boxed(b) || *boxed_val_rel(b,b_base) != hdr) + goto not_equal; + + aa = hashmap_val_rel(a, a_base) + 1; + bb = hashmap_val_rel(b, b_base) + 1; + switch (hdr & _HEADER_MAP_SUBTAG_MASK) { + case HAMT_SUBTAG_HEAD_ARRAY: + aa++; bb++; + sz = 16; + break; + case HAMT_SUBTAG_HEAD_BITMAP: + aa++; bb++; + case HAMT_SUBTAG_NODE_BITMAP: + sz = hashmap_bitcount(MAP_HEADER_VAL(hdr)); + ASSERT(sz > 0 && sz < 17); + break; + default: + erl_exit(1, "Unknown hashmap subsubtag\n"); + } + goto term_array; + } + default: + ASSERT(!"Unknown boxed subtab in EQ"); } break; } @@ -2448,7 +2925,18 @@ Sint erts_cmp_rel_opt(Eterm a, Eterm* a_base, Eterm b, Eterm* b_base, int exact) Sint erts_cmp(Eterm a, Eterm b, int exact) #endif { - DECLARE_WSTACK(stack); +#define PSTACK_TYPE struct erts_cmp_hashmap_state + struct erts_cmp_hashmap_state { + Sint wstack_rollback; + int was_exact; + Eterm *ap; + Eterm *bp; + Eterm min_key; + Sint cmp_res; /* result so far -1,0,+1 */ + }; + PSTACK_DECLARE(hmap_stack, 1); + WSTACK_DECLARE(stack); + WSTACK_DECLARE(b_stack); /* only used by hashmaps */ Eterm* aa; Eterm* bb; int i; @@ -2464,6 +2952,26 @@ Sint erts_cmp(Eterm a, Eterm b, int exact) Uint32 *anum; Uint32 *bnum; +/* The WSTACK contains naked Eterms and Operations marked with header-tags */ +#define OP_BITS 4 +#define OP_MASK 0xF +#define TERM_ARRAY_OP 0 +#define SWITCH_EXACT_OFF_OP 1 +#define HASHMAP_PHASE1_ARE_KEYS_EQUAL 2 +#define HASHMAP_PHASE1_IS_MIN_KEY 3 +#define HASHMAP_PHASE1_CMP_VALUES 4 +#define HASHMAP_PHASE2_ARE_KEYS_EQUAL 5 +#define HASHMAP_PHASE2_IS_MIN_KEY_A 6 +#define HASHMAP_PHASE2_IS_MIN_KEY_B 7 + + +#define OP_WORD(OP) (((OP) << _TAG_PRIMARY_SIZE) | TAG_PRIMARY_HEADER) +#define TERM_ARRAY_OP_WORD(SZ) OP_WORD(((SZ) << OP_BITS) | TERM_ARRAY_OP) + +#define GET_OP(WORD) (ASSERT(is_header(WORD)), ((WORD) >> _TAG_PRIMARY_SIZE) & OP_MASK) +#define GET_OP_ARG(WORD) (ASSERT(is_header(WORD)), ((WORD) >> (OP_BITS + _TAG_PRIMARY_SIZE))) + + #define RETURN_NEQ(cmp) { j=(cmp); ASSERT(j != 0); goto not_equal; } #define ON_CMP_GOTO(cmp) if ((j=(cmp)) == 0) goto pop_next; else goto not_equal @@ -2479,6 +2987,8 @@ Sint erts_cmp(Eterm a, Eterm b, int exact) } while (0) +bodyrecur: + j = 0; tailrecur: if (is_same(a,a_base,b,b_base)) { /* Equal values or pointers. */ goto pop_next; @@ -2605,25 +3115,96 @@ tailrecur_ne: ++aa; ++bb; goto term_array; - case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE) : - if (!is_map_rel(b,b_base)) { - a_tag = MAP_DEF; - goto mixed_types; - } - aa = (Eterm *)map_val_rel(a,a_base); - bb = (Eterm *)map_val_rel(b,b_base); - - i = map_get_size((map_t*)aa); - if (i != map_get_size((map_t*)bb)) { - RETURN_NEQ((int)(i - map_get_size((map_t*)bb))); - } - if (i == 0) { - goto pop_next; + case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE) : + { + struct erts_cmp_hashmap_state* sp; + if (is_flatmap_header(ahdr)) { + if (!is_flatmap_rel(b,b_base)) { + if (is_hashmap_rel(b,b_base)) { + aa = (Eterm *)flatmap_val_rel(a,a_base); + i = flatmap_get_size((flatmap_t*)aa) - hashmap_size_rel(b,b_base); + ASSERT(i != 0); + RETURN_NEQ(i); + } + a_tag = MAP_DEF; + goto mixed_types; + } + aa = (Eterm *)flatmap_val_rel(a,a_base); + bb = (Eterm *)flatmap_val_rel(b,b_base); + + i = flatmap_get_size((flatmap_t*)aa); + if (i != flatmap_get_size((flatmap_t*)bb)) { + RETURN_NEQ((int)(i - flatmap_get_size((flatmap_t*)bb))); + } + if (i == 0) { + goto pop_next; + } + aa += 2; + bb += 2; + if (exact) { + i += 1; /* increment for tuple-keys */ + goto term_array; + } + else { + /* Value array */ + WSTACK_PUSH3(stack,(UWord)(bb+1),(UWord)(aa+1),TERM_ARRAY_OP_WORD(i)); + /* Switch back from 'exact' key compare */ + WSTACK_PUSH(stack,OP_WORD(SWITCH_EXACT_OFF_OP)); + /* Now do 'exact' compare of key tuples */ + a = *aa; + b = *bb; + exact = 1; + goto bodyrecur; + } + } + if (!is_hashmap_rel(b,b_base)) { + if (is_flatmap_rel(b,b_base)) { + bb = (Eterm *)flatmap_val_rel(b,b_base); + i = hashmap_size_rel(a,a_base) - flatmap_get_size((flatmap_t*)bb); + ASSERT(i != 0); + RETURN_NEQ(i); + } + a_tag = MAP_DEF; + goto mixed_types; + } + i = hashmap_size_rel(a,a_base) - hashmap_size_rel(b,b_base); + if (i) { + RETURN_NEQ(i); + } + if (hashmap_size_rel(a,a_base) == 0) { + goto pop_next; + } + + /* Hashmap compare strategy: + Phase 1. While keys are identical + Do synchronous stepping through leafs of both trees in hash + order. Maintain value compare result of minimal key. + + Phase 2. If key diff was found in phase 1 + Ignore values from now on. + Continue iterate trees by always advancing the one + lagging behind hash-wise. Identical keys are skipped. + A minimal key can only be candidate as tie-breaker if we + have passed that hash value in the other tree (which means + the key did not exist in the other tree). + */ + + sp = PSTACK_PUSH(hmap_stack); + hashmap_iterator_init(&stack, a, 0); + hashmap_iterator_init(&b_stack, b, 0); + sp->ap = hashmap_iterator_next(&stack); + sp->bp = hashmap_iterator_next(&b_stack); + sp->cmp_res = 0; + ASSERT(sp->ap && sp->bp); + + a = CAR(sp->ap); + b = CAR(sp->bp); + sp->was_exact = exact; + exact = 1; + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE1_ARE_KEYS_EQUAL)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; } - aa += 2; - bb += 2; - i += 1; /* increment for tuple-keys */ - goto term_array; case (_TAG_HEADER_FLOAT >> _TAG_PRIMARY_SIZE): if (!is_float_rel(b,b_base)) { a_tag = FLOAT_DEF; @@ -2983,8 +3564,7 @@ term_array: /* arrays in 'aa' and 'bb', length in 'i' */ goto not_equal; } } else { - /* (ab)Use TAG_PRIMARY_HEADER to recognize a term_array */ - WSTACK_PUSH3(stack, i, (UWord)bb, (UWord)aa | TAG_PRIMARY_HEADER); + WSTACK_PUSH3(stack, (UWord)bb, (UWord)aa, TERM_ARRAY_OP_WORD(i)); goto tailrecur_ne; } } @@ -2996,22 +3576,179 @@ term_array: /* arrays in 'aa' and 'bb', length in 'i' */ pop_next: if (!WSTACK_ISEMPTY(stack)) { UWord something = WSTACK_POP(stack); - if (primary_tag((Eterm) something) == TAG_PRIMARY_HEADER) { /* a term_array */ - aa = (Eterm*) something; - bb = (Eterm*) WSTACK_POP(stack); - i = WSTACK_POP(stack); - goto term_array; + struct erts_cmp_hashmap_state* sp; + if (primary_tag((Eterm) something) == TAG_PRIMARY_HEADER) { /* an operation */ + switch (GET_OP(something)) { + case TERM_ARRAY_OP: + i = GET_OP_ARG(something); + aa = (Eterm*)WSTACK_POP(stack); + bb = (Eterm*) WSTACK_POP(stack); + goto term_array; + + case SWITCH_EXACT_OFF_OP: + /* Done with exact compare of map keys, switch back */ + ASSERT(exact); + exact = 0; + goto pop_next; + + case HASHMAP_PHASE1_ARE_KEYS_EQUAL: { + sp = PSTACK_TOP(hmap_stack); + if (j) { + /* Key diff found, enter phase 2 */ + if (hashmap_key_hash_cmp(sp->ap, sp->bp) < 0) { + sp->min_key = CAR(sp->ap); + sp->cmp_res = -1; + sp->ap = hashmap_iterator_next(&stack); + } + else { + sp->min_key = CAR(sp->bp); + sp->cmp_res = 1; + sp->bp = hashmap_iterator_next(&b_stack); + } + exact = 1; /* only exact key compares in phase 2 */ + goto case_HASHMAP_PHASE2_LOOP; + } + + /* No key diff found so far, compare values if min key */ + + if (sp->cmp_res) { + a = CAR(sp->ap); + b = sp->min_key; + exact = 1; + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE1_IS_MIN_KEY)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + } + /* no min key-value found yet */ + a = CDR(sp->ap); + b = CDR(sp->bp); + exact = sp->was_exact; + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE1_CMP_VALUES)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + } + case HASHMAP_PHASE1_IS_MIN_KEY: + sp = PSTACK_TOP(hmap_stack); + if (j < 0) { + a = CDR(sp->ap); + b = CDR(sp->bp); + exact = sp->was_exact; + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE1_CMP_VALUES)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + } + goto case_HASHMAP_PHASE1_LOOP; + + case HASHMAP_PHASE1_CMP_VALUES: + sp = PSTACK_TOP(hmap_stack); + if (j) { + sp->cmp_res = j; + sp->min_key = CAR(sp->ap); + } + case_HASHMAP_PHASE1_LOOP: + sp->ap = hashmap_iterator_next(&stack); + sp->bp = hashmap_iterator_next(&b_stack); + if (!sp->ap) { + /* end of maps with identical keys */ + ASSERT(!sp->bp); + j = sp->cmp_res; + exact = sp->was_exact; + (void) PSTACK_POP(hmap_stack); + ON_CMP_GOTO(j); + } + a = CAR(sp->ap); + b = CAR(sp->bp); + exact = 1; + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE1_ARE_KEYS_EQUAL)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + + case_HASHMAP_PHASE2_LOOP: + if (sp->ap && sp->bp) { + a = CAR(sp->ap); + b = CAR(sp->bp); + ASSERT(exact); + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE2_ARE_KEYS_EQUAL)); + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + } + goto case_HASHMAP_PHASE2_NEXT_STEP; + + case HASHMAP_PHASE2_ARE_KEYS_EQUAL: + sp = PSTACK_TOP(hmap_stack); + if (j == 0) { + /* keys are equal, skip them */ + sp->ap = hashmap_iterator_next(&stack); + sp->bp = hashmap_iterator_next(&b_stack); + goto case_HASHMAP_PHASE2_LOOP; + } + /* fall through */ + case_HASHMAP_PHASE2_NEXT_STEP: + if (sp->ap || sp->bp) { + if (hashmap_key_hash_cmp(sp->ap, sp->bp) < 0) { + ASSERT(sp->ap); + a = CAR(sp->ap); + b = sp->min_key; + ASSERT(exact); + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE2_IS_MIN_KEY_A)); + } + else { /* hash_cmp > 0 */ + ASSERT(sp->bp); + a = CAR(sp->bp); + b = sp->min_key; + ASSERT(exact); + WSTACK_PUSH(stack, OP_WORD(HASHMAP_PHASE2_IS_MIN_KEY_B)); + } + sp->wstack_rollback = WSTACK_COUNT(stack); + goto bodyrecur; + } + /* End of both maps */ + j = sp->cmp_res; + exact = sp->was_exact; + (void) PSTACK_POP(hmap_stack); + ON_CMP_GOTO(j); + + case HASHMAP_PHASE2_IS_MIN_KEY_A: + sp = PSTACK_TOP(hmap_stack); + if (j < 0) { + sp->min_key = CAR(sp->ap); + sp->cmp_res = -1; + } + sp->ap = hashmap_iterator_next(&stack); + goto case_HASHMAP_PHASE2_LOOP; + + case HASHMAP_PHASE2_IS_MIN_KEY_B: + sp = PSTACK_TOP(hmap_stack); + if (j < 0) { + sp->min_key = CAR(sp->bp); + sp->cmp_res = 1; + } + sp->bp = hashmap_iterator_next(&b_stack); + goto case_HASHMAP_PHASE2_LOOP; + + default: + ASSERT(!"Invalid cmp op"); + } /* switch */ } a = (Eterm) something; b = (Eterm) WSTACK_POP(stack); goto tailrecur; } - DESTROY_WSTACK(stack); + ASSERT(PSTACK_IS_EMPTY(hmap_stack)); + PSTACK_DESTROY(hmap_stack); + WSTACK_DESTROY(stack); + WSTACK_DESTROY(b_stack); return 0; not_equal: - DESTROY_WSTACK(stack); + if (!PSTACK_IS_EMPTY(hmap_stack)) { + WSTACK_ROLLBACK(stack, PSTACK_TOP(hmap_stack)->wstack_rollback); + goto pop_next; + } + PSTACK_DESTROY(hmap_stack); + WSTACK_DESTROY(stack); + WSTACK_DESTROY(b_stack); return j; #undef CMP_NODES @@ -3774,7 +4511,7 @@ erts_create_smp_ptimer(ErtsSmpPTimer **timer_ref, res->timer.timeout_func = timeout_func; res->timer.timer_ref = timer_ref; res->timer.id = id; - res->timer.tm.active = 0; /* MUST be initalized */ + erts_init_timer(&res->timer.tm); ASSERT(!*timer_ref); @@ -4351,8 +5088,8 @@ erts_smp_ensure_later_interval_acqb(erts_interval_t *icp, Uint64 ic) */ Uint64 erts_timestamp_millis(void) { -#ifdef HAVE_GETHRTIME - return (Uint64) (sys_gethrtime() / 1000000); +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + return ERTS_MONOTONIC_TO_MSEC(erts_os_monotonic_time()); #else Uint64 res; SysTimeval tv; diff --git a/erts/emulator/drivers/common/efile_drv.c b/erts/emulator/drivers/common/efile_drv.c index b62e9a0306..3b8e7acb6e 100644 --- a/erts/emulator/drivers/common/efile_drv.c +++ b/erts/emulator/drivers/common/efile_drv.c @@ -264,23 +264,6 @@ dt_private *get_dt_private(int); -#define GET_TIME(i, b) \ - (i).year = get_int32((b) + 0 * 4); \ - (i).month = get_int32((b) + 1 * 4); \ - (i).day = get_int32((b) + 2 * 4); \ - (i).hour = get_int32((b) + 3 * 4); \ - (i).minute = get_int32((b) + 4 * 4); \ - (i).second = get_int32((b) + 5 * 4) - -#define PUT_TIME(i, b) \ - put_int32((i).year, (b) + 0 * 4); \ - put_int32((i).month, (b) + 1 * 4); \ - put_int32((i).day, (b) + 2 * 4); \ - put_int32((i).hour, (b) + 3 * 4); \ - put_int32((i).minute,(b) + 4 * 4); \ - put_int32((i).second,(b) + 5 * 4) - - #if ALWAYS_READ_LINE_AHEAD #define DEFAULT_LINEBUF_SIZE 2048 #else diff --git a/erts/emulator/hipe/hipe_amd64_abi.txt b/erts/emulator/hipe/hipe_amd64_abi.txt index 8a34bfa67f..72aed13995 100644 --- a/erts/emulator/hipe/hipe_amd64_abi.txt +++ b/erts/emulator/hipe/hipe_amd64_abi.txt @@ -45,7 +45,7 @@ The first return value from a function is placed in %rax, the second (if any) is placed in %rdx. Notes: -- Currently, NR_ARG_REGS==0. +- Currently, NR_ARG_REGS == 4. - C BIFs expect P in C parameter register 1: %rdi. By making Erlang parameter registers 1-5 coincide with C parameter registers 2-6, our BIF wrappers can simply move P to %rdi without having to shift diff --git a/erts/emulator/hipe/hipe_amd64_asm.m4 b/erts/emulator/hipe/hipe_amd64_asm.m4 index b4b3c073ab..ca55d5bf3b 100644 --- a/erts/emulator/hipe/hipe_amd64_asm.m4 +++ b/erts/emulator/hipe/hipe_amd64_asm.m4 @@ -253,6 +253,10 @@ define(NBIF_ARG,`ifelse(eval($3 >= NR_ARG_REGS),0,`NBIF_REG_ARG($1,$3)',`NBIF_ST `/* #define NBIF_ARG_3_0 'NBIF_ARG(%rsi,3,0)` */' `/* #define NBIF_ARG_3_1 'NBIF_ARG(%rdx,3,1)` */' `/* #define NBIF_ARG_3_2 'NBIF_ARG(%rcx,3,2)` */' +`/* #define NBIF_ARG_4_0 'NBIF_ARG(%rsi,4,0)` */' +`/* #define NBIF_ARG_4_1 'NBIF_ARG(%rdx,4,1)` */' +`/* #define NBIF_ARG_4_2 'NBIF_ARG(%rcx,4,2)` */' +`/* #define NBIF_ARG_4_3 'NBIF_ARG(%r8,4,3)` */' `/* #define NBIF_ARG_5_0 'NBIF_ARG(%rsi,5,0)` */' `/* #define NBIF_ARG_5_1 'NBIF_ARG(%rdx,5,1)` */' `/* #define NBIF_ARG_5_2 'NBIF_ARG(%rcx,5,2)` */' @@ -277,6 +281,7 @@ define(NBIF_RET,`NBIF_RET_N(eval(RET_POP($1)))')dnl `/* #define NBIF_RET_1 'NBIF_RET(1)` */' `/* #define NBIF_RET_2 'NBIF_RET(2)` */' `/* #define NBIF_RET_3 'NBIF_RET(3)` */' +`/* #define NBIF_RET_4 'NBIF_RET(4)` */' `/* #define NBIF_RET_5 'NBIF_RET(5)` */' `#endif /* ASM */' diff --git a/erts/emulator/hipe/hipe_amd64_bifs.m4 b/erts/emulator/hipe/hipe_amd64_bifs.m4 index 7a4bb30447..7d94aa05b3 100644 --- a/erts/emulator/hipe/hipe_amd64_bifs.m4 +++ b/erts/emulator/hipe/hipe_amd64_bifs.m4 @@ -51,9 +51,10 @@ define(HANDLE_GOT_MBUF,` * standard_bif_interface_1(nbif_name, cbif_name) * standard_bif_interface_2(nbif_name, cbif_name) * standard_bif_interface_3(nbif_name, cbif_name) + * standard_bif_interface_4(nbif_name, cbif_name) * standard_bif_interface_0(nbif_name, cbif_name) * - * Generate native interface for a BIF with 0-3 parameters and + * Generate native interface for a BIF with 0-4 parameters and * standard failure mode. */ define(standard_bif_interface_1, @@ -154,6 +155,43 @@ ASYM($1): TYPE_FUNCTION(ASYM($1)) #endif') +define(standard_bif_interface_4, +` +#ifndef HAVE_$1 +#`define' HAVE_$1 + TEXT + .align 4 + GLOBAL(ASYM($1)) +ASYM($1): + /* set up the parameters */ + movq P, %rdi + NBIF_ARG(%rsi,4,0) + NBIF_ARG(%rdx,4,1) + NBIF_ARG(%rcx,4,2) + NBIF_ARG(%r8,4,3) + + /* make the call on the C stack */ + SWITCH_ERLANG_TO_C + pushq %r8 + pushq %rcx + pushq %rdx + pushq %rsi + movq %rsp, %rsi /* Eterm* BIF__ARGS */ + sub $(8), %rsp /* stack frame 16-byte alignment */ + CALL_BIF($2) + add $(4*8 + 8), %rsp + TEST_GOT_MBUF + SWITCH_C_TO_ERLANG + + /* throw exception if failure, otherwise return */ + TEST_GOT_EXN + jz nbif_4_simple_exception + NBIF_RET(4) + HANDLE_GOT_MBUF(4) + SET_SIZE(ASYM($1)) + TYPE_FUNCTION(ASYM($1)) +#endif') + define(standard_bif_interface_0, ` #ifndef HAVE_$1 diff --git a/erts/emulator/hipe/hipe_amd64_glue.S b/erts/emulator/hipe/hipe_amd64_glue.S index 955f7362b4..3cb0a2875b 100644 --- a/erts/emulator/hipe/hipe_amd64_glue.S +++ b/erts/emulator/hipe/hipe_amd64_glue.S @@ -321,6 +321,7 @@ ASYM(nbif_fail): GLOBAL(nbif_1_gc_after_bif) GLOBAL(nbif_2_gc_after_bif) GLOBAL(nbif_3_gc_after_bif) + GLOBAL(nbif_4_gc_after_bif) .align 4 nbif_0_gc_after_bif: xorl %edx, %edx @@ -336,6 +337,10 @@ nbif_2_gc_after_bif: .align 4 nbif_3_gc_after_bif: movl $3, %edx + jmp .gc_after_bif + .align 4 +nbif_4_gc_after_bif: + movl $4, %edx /*FALLTHROUGH*/ .align 4 .gc_after_bif: @@ -359,6 +364,7 @@ nbif_3_gc_after_bif: GLOBAL(nbif_1_simple_exception) GLOBAL(nbif_2_simple_exception) GLOBAL(nbif_3_simple_exception) + GLOBAL(nbif_4_simple_exception) .align 4 nbif_0_simple_exception: xorl %eax, %eax @@ -374,6 +380,10 @@ nbif_2_simple_exception: .align 4 nbif_3_simple_exception: movl $3, %eax + jmp .nbif_simple_exception + .align 4 +nbif_4_simple_exception: + movl $4, %eax /*FALLTHROUGH*/ .align 4 .nbif_simple_exception: diff --git a/erts/emulator/hipe/hipe_bif0.c b/erts/emulator/hipe/hipe_bif0.c index 9e5830f345..6c1de05a4c 100644 --- a/erts/emulator/hipe/hipe_bif0.c +++ b/erts/emulator/hipe/hipe_bif0.c @@ -1012,22 +1012,32 @@ static struct { * they create a new stub for the mfa, which forces locking. * XXX: Redesign apply et al to avoid those updates. */ - erts_smp_mtx_t lock; + erts_smp_rwmtx_t lock; } hipe_mfa_info_table; static inline void hipe_mfa_info_table_init_lock(void) { - erts_smp_mtx_init(&hipe_mfa_info_table.lock, "hipe_mfait_lock"); + erts_smp_rwmtx_init(&hipe_mfa_info_table.lock, "hipe_mfait_lock"); } -static inline void hipe_mfa_info_table_lock(void) +static inline void hipe_mfa_info_table_rlock(void) { - erts_smp_mtx_lock(&hipe_mfa_info_table.lock); + erts_smp_rwmtx_rlock(&hipe_mfa_info_table.lock); } -static inline void hipe_mfa_info_table_unlock(void) +static inline void hipe_mfa_info_table_runlock(void) { - erts_smp_mtx_unlock(&hipe_mfa_info_table.lock); + erts_smp_rwmtx_runlock(&hipe_mfa_info_table.lock); +} + +static inline void hipe_mfa_info_table_rwlock(void) +{ + erts_smp_rwmtx_rwlock(&hipe_mfa_info_table.lock); +} + +static inline void hipe_mfa_info_table_rwunlock(void) +{ + erts_smp_rwmtx_rwunlock(&hipe_mfa_info_table.lock); } #define HIPE_MFA_HASH(M,F,A) ((M) * (F) + (A)) @@ -1118,7 +1128,17 @@ static inline struct hipe_mfa_info *hipe_mfa_info_table_get_locked(Eterm m, Eter return NULL; } -static struct hipe_mfa_info *hipe_mfa_info_table_put_locked(Eterm m, Eterm f, unsigned int arity) +#if 0 /* XXX: unused */ +void *hipe_mfa_find_na(Eterm m, Eterm f, unsigned int arity) +{ + const struct hipe_mfa_info *p; + + p = hipe_mfa_info_table_get(m, f, arity); + return p ? p->address : NULL; +} +#endif + +static struct hipe_mfa_info *hipe_mfa_info_table_put_rwlocked(Eterm m, Eterm f, unsigned int arity) { unsigned long h; unsigned int i; @@ -1147,8 +1167,8 @@ static void hipe_mfa_set_na(Eterm m, Eterm f, unsigned int arity, void *address, { struct hipe_mfa_info *p; - hipe_mfa_info_table_lock(); - p = hipe_mfa_info_table_put_locked(m, f, arity); + hipe_mfa_info_table_rwlock(); + p = hipe_mfa_info_table_put_rwlocked(m, f, arity); #ifdef DEBUG_LINKER printf("%s: ", __FUNCTION__); print_mfa(m, f, arity); @@ -1157,7 +1177,7 @@ static void hipe_mfa_set_na(Eterm m, Eterm f, unsigned int arity, void *address, p->local_address = address; if (is_exported) p->remote_address = address; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); } #if defined(__powerpc__) || defined(__ppc__) || defined(__powerpc64__) || defined(__arm__) @@ -1166,10 +1186,10 @@ void *hipe_mfa_get_trampoline(Eterm m, Eterm f, unsigned int arity) struct hipe_mfa_info *p; void *trampoline; - hipe_mfa_info_table_lock(); - p = hipe_mfa_info_table_put_locked(m, f, arity); - trampoline = p->trampoline; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rlock(); + p = hipe_mfa_info_table_get_locked(m, f, arity); + trampoline = p ? p->trampoline : NULL; + hipe_mfa_info_table_runlock(); return trampoline; } @@ -1177,10 +1197,10 @@ void hipe_mfa_set_trampoline(Eterm m, Eterm f, unsigned int arity, void *trampol { struct hipe_mfa_info *p; - hipe_mfa_info_table_lock(); - p = hipe_mfa_info_table_put_locked(m, f, arity); + hipe_mfa_info_table_rwlock(); + p = hipe_mfa_info_table_put_rwlocked(m, f, arity); p->trampoline = trampoline; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); } #endif @@ -1211,7 +1231,7 @@ BIF_RETTYPE hipe_bifs_invalidate_funinfo_native_addresses_1(BIF_ALIST_1) struct mfa mfa; struct hipe_mfa_info *p; - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rwlock(); lst = BIF_ARG_1; while (is_list(lst)) { if (!term_to_mfa(CAR(list_val(lst)), &mfa)) @@ -1240,7 +1260,7 @@ BIF_RETTYPE hipe_bifs_invalidate_funinfo_native_addresses_1(BIF_ALIST_1) } } } - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); if (is_not_nil(lst)) BIF_ERROR(BIF_P, BADARG); BIF_RET(NIL); @@ -1254,8 +1274,8 @@ void hipe_mfa_save_orig_beam_op(Eterm mod, Eterm fun, unsigned int ari, Eterm *p orig_beam_op = pc[0]; if (orig_beam_op != BeamOpCode(op_hipe_trap_call_closure) && orig_beam_op != BeamOpCode(op_hipe_trap_call)) { - hipe_mfa_info_table_lock(); - p = hipe_mfa_info_table_put_locked(mod, fun, ari); + hipe_mfa_info_table_rwlock(); + p = hipe_mfa_info_table_put_rwlocked(mod, fun, ari); #ifdef DEBUG_LINKER printf("%s: ", __FUNCTION__); print_mfa(mod, fun, ari); @@ -1263,7 +1283,7 @@ void hipe_mfa_save_orig_beam_op(Eterm mod, Eterm fun, unsigned int ari, Eterm *p #endif p->beam_code = pc; p->orig_beam_op = orig_beam_op; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); } else { #ifdef DEBUG_LINKER printf("%s: ", __FUNCTION__); @@ -1287,7 +1307,7 @@ static void *hipe_make_stub(Eterm m, Eterm f, unsigned int arity, int is_remote) return StubAddress; } -static void *hipe_get_na_nofail_locked(Eterm m, Eterm f, unsigned int a, int is_remote) +static void *hipe_get_na_try_locked(Eterm m, Eterm f, unsigned int a, int is_remote, struct hipe_mfa_info **pp) { struct hipe_mfa_info *p; void *address; @@ -1305,22 +1325,53 @@ static void *hipe_get_na_nofail_locked(Eterm m, Eterm f, unsigned int a, int is_ address = p->remote_address; if (address) return address; - } else - p = hipe_mfa_info_table_put_locked(m, f, a); + } + /* Caller must take the slow path with the write lock held, but allow + it to avoid some work if it already holds the write lock. */ + if (pp) + *pp = p; + return NULL; +} + +static void *hipe_get_na_slow_rwlocked(Eterm m, Eterm f, unsigned int a, int is_remote, struct hipe_mfa_info *p) +{ + void *address; + + if (!p) + p = hipe_mfa_info_table_put_rwlocked(m, f, a); address = hipe_make_stub(m, f, a, is_remote); /* XXX: how to tell if a BEAM MFA is exported or not? */ p->remote_address = address; return address; } +static void *hipe_get_na_nofail_rwlocked(Eterm m, Eterm f, unsigned int a, int is_remote) +{ + struct hipe_mfa_info *p; + void *address; + + address = hipe_get_na_try_locked(m, f, a, is_remote, &p); + if (address) + return address; + + address = hipe_get_na_slow_rwlocked(m, f, a, is_remote, p); + return address; +} + static void *hipe_get_na_nofail(Eterm m, Eterm f, unsigned int a, int is_remote) { - void *p; + void *address; - hipe_mfa_info_table_lock(); - p = hipe_get_na_nofail_locked(m, f, a, is_remote); - hipe_mfa_info_table_unlock(); - return p; + hipe_mfa_info_table_rlock(); + address = hipe_get_na_try_locked(m, f, a, is_remote, NULL); + hipe_mfa_info_table_runlock(); + if (address) + return address; + + hipe_mfa_info_table_rwlock(); + address = hipe_get_na_slow_rwlocked(m, f, a, is_remote, NULL); + hipe_mfa_info_table_rwunlock(); + return address; } /* used for apply/3 in hipe_mode_switch */ @@ -1399,7 +1450,7 @@ int hipe_find_mfa_from_ra(const void *ra, Eterm *m, Eterm *f, unsigned int *a) /* Note about locking: the table is only updated from the loader, which runs with the rest of the system suspended. */ /* XXX: alas not true; see comment at hipe_mfa_info_table.lock */ - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rlock(); bucket = hipe_mfa_info_table.bucket; nrbuckets = 1 << hipe_mfa_info_table.log2size; mfa = NULL; @@ -1420,7 +1471,7 @@ int hipe_find_mfa_from_ra(const void *ra, Eterm *m, Eterm *f, unsigned int *a) *f = mfa->f; *a = mfa->a; } - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_runlock(); return mfa ? 1 : 0; } @@ -1497,9 +1548,9 @@ BIF_RETTYPE hipe_bifs_add_ref_2(BIF_ALIST_2) default: goto badarg; } - hipe_mfa_info_table_lock(); - callee_mfa = hipe_mfa_info_table_put_locked(callee.mod, callee.fun, callee.ari); - caller_mfa = hipe_mfa_info_table_put_locked(caller.mod, caller.fun, caller.ari); + hipe_mfa_info_table_rwlock(); + callee_mfa = hipe_mfa_info_table_put_rwlocked(callee.mod, callee.fun, callee.ari); + caller_mfa = hipe_mfa_info_table_put_rwlocked(caller.mod, caller.fun, caller.ari); refers_to = erts_alloc(ERTS_ALC_T_HIPE, sizeof(*refers_to)); refers_to->mfa = callee_mfa; @@ -1513,7 +1564,7 @@ BIF_RETTYPE hipe_bifs_add_ref_2(BIF_ALIST_2) ref->flags = flags; ref->next = callee_mfa->referred_from; callee_mfa->referred_from = ref; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); BIF_RET(NIL); @@ -1533,12 +1584,12 @@ BIF_RETTYPE hipe_bifs_mark_referred_from_1(BIF_ALIST_1) /* get_refs_from */ if (!term_to_mfa(BIF_ARG_1, &mfa)) BIF_ERROR(BIF_P, BADARG); - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rwlock(); p = hipe_mfa_info_table_get_locked(mfa.mod, mfa.fun, mfa.ari); if (p) for (ref = p->referred_from; ref != NULL; ref = ref->next) ref->flags |= REF_FLAG_PENDING_REDIRECT; - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); BIF_RET(NIL); } @@ -1552,7 +1603,7 @@ static void hipe_purge_all_refs(void) struct hipe_mfa_info **bucket; unsigned int i, nrbuckets; - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rwlock(); bucket = hipe_mfa_info_table.bucket; nrbuckets = 1 << hipe_mfa_info_table.log2size; @@ -1574,7 +1625,7 @@ static void hipe_purge_all_refs(void) erts_free(ERTS_ALC_T_HIPE, mfa); } } - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); } BIF_RETTYPE hipe_bifs_remove_refs_from_1(BIF_ALIST_1) @@ -1591,7 +1642,7 @@ BIF_RETTYPE hipe_bifs_remove_refs_from_1(BIF_ALIST_1) if (!term_to_mfa(BIF_ARG_1, &mfa)) BIF_ERROR(BIF_P, BADARG); - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rwlock(); caller_mfa = hipe_mfa_info_table_get_locked(mfa.mod, mfa.fun, mfa.ari); if (caller_mfa) { refers_to = caller_mfa->refers_to; @@ -1622,7 +1673,7 @@ BIF_RETTYPE hipe_bifs_remove_refs_from_1(BIF_ALIST_1) } caller_mfa->refers_to = NULL; } - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); BIF_RET(am_ok); } @@ -1641,7 +1692,7 @@ BIF_RETTYPE hipe_bifs_redirect_referred_from_1(BIF_ALIST_1) if (!term_to_mfa(BIF_ARG_1, &mfa)) BIF_ERROR(BIF_P, BADARG); - hipe_mfa_info_table_lock(); + hipe_mfa_info_table_rwlock(); p = hipe_mfa_info_table_get_locked(mfa.mod, mfa.fun, mfa.ari); if (p) { prev = &p->referred_from; @@ -1649,7 +1700,7 @@ BIF_RETTYPE hipe_bifs_redirect_referred_from_1(BIF_ALIST_1) while (ref) { if (ref->flags & REF_FLAG_PENDING_REDIRECT) { is_remote = ref->flags & REF_FLAG_IS_REMOTE; - new_address = hipe_get_na_nofail_locked(p->m, p->f, p->a, is_remote); + new_address = hipe_get_na_nofail_rwlocked(p->m, p->f, p->a, is_remote); if (ref->flags & REF_FLAG_IS_LOAD_MFA) res = hipe_patch_insn(ref->address, (Uint)new_address, am_load_mfa); else @@ -1672,7 +1723,7 @@ BIF_RETTYPE hipe_bifs_redirect_referred_from_1(BIF_ALIST_1) } } } - hipe_mfa_info_table_unlock(); + hipe_mfa_info_table_rwunlock(); BIF_RET(NIL); } diff --git a/erts/emulator/hipe/hipe_ppc_asm.m4 b/erts/emulator/hipe/hipe_ppc_asm.m4 index 4a1caa1543..e5a56de687 100644 --- a/erts/emulator/hipe/hipe_ppc_asm.m4 +++ b/erts/emulator/hipe/hipe_ppc_asm.m4 @@ -280,6 +280,10 @@ define(NBIF_ARG,`ifelse(eval($3 >= NR_ARG_REGS),0,`NBIF_REG_ARG($1,$3)',`NBIF_ST `/* #define NBIF_ARG_3_0 'NBIF_ARG(r3,3,0)` */' `/* #define NBIF_ARG_3_1 'NBIF_ARG(r3,3,1)` */' `/* #define NBIF_ARG_3_2 'NBIF_ARG(r3,3,2)` */' +`/* #define NBIF_ARG_4_0 'NBIF_ARG(r3,4,0)` */' +`/* #define NBIF_ARG_4_1 'NBIF_ARG(r3,4,1)` */' +`/* #define NBIF_ARG_4_2 'NBIF_ARG(r3,4,2)` */' +`/* #define NBIF_ARG_4_3 'NBIF_ARG(r3,4,3)` */' `/* #define NBIF_ARG_5_0 'NBIF_ARG(r3,5,0)` */' `/* #define NBIF_ARG_5_1 'NBIF_ARG(r3,5,1)` */' `/* #define NBIF_ARG_5_2 'NBIF_ARG(r3,5,2)` */' @@ -301,6 +305,7 @@ define(NBIF_RET,`NBIF_RET_N(eval(RET_POP($1)))')dnl `/* #define NBIF_RET_1 'NBIF_RET(1)` */' `/* #define NBIF_RET_2 'NBIF_RET(2)` */' `/* #define NBIF_RET_3 'NBIF_RET(3)` */' +`/* #define NBIF_RET_4 'NBIF_RET(4)` */' `/* #define NBIF_RET_5 'NBIF_RET(5)` */' dnl diff --git a/erts/emulator/hipe/hipe_ppc_bifs.m4 b/erts/emulator/hipe/hipe_ppc_bifs.m4 index f53b79b52e..b173b896b8 100644 --- a/erts/emulator/hipe/hipe_ppc_bifs.m4 +++ b/erts/emulator/hipe/hipe_ppc_bifs.m4 @@ -46,9 +46,10 @@ define(HANDLE_GOT_MBUF,` * standard_bif_interface_1(nbif_name, cbif_name) * standard_bif_interface_2(nbif_name, cbif_name) * standard_bif_interface_3(nbif_name, cbif_name) + * standard_bif_interface_4(nbif_name, cbif_name) * standard_bif_interface_0(nbif_name, cbif_name) * - * Generate native interface for a BIF with 0-3 parameters and + * Generate native interface for a BIF with 0-4 parameters and * standard failure mode. */ define(standard_bif_interface_1, @@ -144,6 +145,41 @@ ASYM($1): TYPE_FUNCTION(ASYM($1)) #endif') +define(standard_bif_interface_4, +` +#ifndef HAVE_$1 +#`define' HAVE_$1 + GLOBAL(ASYM($1)) +ASYM($1): + /* Set up C argument registers. */ + mr r3, P + NBIF_ARG(r4,4,0) + NBIF_ARG(r5,4,1) + NBIF_ARG(r6,4,2) + NBIF_ARG(r7,4,3) + + /* Save caller-save registers and call the C function. */ + SAVE_CONTEXT_BIF + STORE r4, P_ARG0(r3) /* Store BIF__ARGS in def_arg_reg[] */ + STORE r5, P_ARG1(r3) + STORE r6, P_ARG2(r3) + STORE r7, P_ARG3(r3) + addi r4, r3, P_ARG0 + CALL_BIF($2) + TEST_GOT_MBUF + + /* Restore registers. Check for exception. */ + CMPI r3, THE_NON_VALUE + RESTORE_CONTEXT_BIF + beq- 1f + NBIF_RET(4) +1: /* workaround for bc:s small offset operand */ + b CSYM(nbif_4_simple_exception) + HANDLE_GOT_MBUF(4) + SET_SIZE(ASYM($1)) + TYPE_FUNCTION(ASYM($1)) +#endif') + define(standard_bif_interface_0, ` #ifndef HAVE_$1 diff --git a/erts/emulator/hipe/hipe_ppc_glue.S b/erts/emulator/hipe/hipe_ppc_glue.S index c48fb150af..b07f4bc9c8 100644 --- a/erts/emulator/hipe/hipe_ppc_glue.S +++ b/erts/emulator/hipe/hipe_ppc_glue.S @@ -462,10 +462,12 @@ ASYM(nbif_fail): OPD(nbif_1_gc_after_bif) OPD(nbif_2_gc_after_bif) OPD(nbif_3_gc_after_bif) + OPD(nbif_4_gc_after_bif) GLOBAL(CSYM(nbif_0_gc_after_bif)) GLOBAL(CSYM(nbif_1_gc_after_bif)) GLOBAL(CSYM(nbif_2_gc_after_bif)) GLOBAL(CSYM(nbif_3_gc_after_bif)) + GLOBAL(CSYM(nbif_4_gc_after_bif)) CSYM(nbif_0_gc_after_bif): li r4, 0 b .gc_after_bif @@ -477,6 +479,9 @@ CSYM(nbif_2_gc_after_bif): b .gc_after_bif CSYM(nbif_3_gc_after_bif): li r4, 3 + b .gc_after_bif +CSYM(nbif_4_gc_after_bif): + li r4, 4 /*FALLTHROUGH*/ .gc_after_bif: stw r4, P_NARITY(P) /* Note: narity is a 32-bit field */ @@ -519,6 +524,11 @@ CSYM(nbif_2_simple_exception): GLOBAL(CSYM(nbif_3_simple_exception)) CSYM(nbif_3_simple_exception): li r4, 3 + b .nbif_simple_exception + OPD(nbif_3_simple_exception) + GLOBAL(CSYM(nbif_4_simple_exception)) +CSYM(nbif_4_simple_exception): + li r4, 4 /*FALLTHROUGH*/ .nbif_simple_exception: LOAD r3, P_FREASON(P) diff --git a/erts/emulator/hipe/hipe_sparc_asm.m4 b/erts/emulator/hipe/hipe_sparc_asm.m4 index c3c3bcb74a..8020104e40 100644 --- a/erts/emulator/hipe/hipe_sparc_asm.m4 +++ b/erts/emulator/hipe/hipe_sparc_asm.m4 @@ -176,6 +176,10 @@ define(NBIF_ARG,`ifelse(eval($3 >= NR_ARG_REGS),0,`NBIF_REG_ARG($1,$3)',`NBIF_ST `/* #define NBIF_ARG_3_0 'NBIF_ARG(r1,3,0)` */' `/* #define NBIF_ARG_3_1 'NBIF_ARG(r2,3,1)` */' `/* #define NBIF_ARG_3_2 'NBIF_ARG(r3,3,2)` */' +`/* #define NBIF_ARG_4_0 'NBIF_ARG(r1,4,0)` */' +`/* #define NBIF_ARG_4_1 'NBIF_ARG(r2,4,1)` */' +`/* #define NBIF_ARG_4_2 'NBIF_ARG(r3,4,2)` */' +`/* #define NBIF_ARG_4_3 'NBIF_ARG(r3,4,3)` */' `/* #define NBIF_ARG_5_0 'NBIF_ARG(r1,5,0)` */' `/* #define NBIF_ARG_5_1 'NBIF_ARG(r2,5,1)` */' `/* #define NBIF_ARG_5_2 'NBIF_ARG(r3,5,2)` */' @@ -200,6 +204,7 @@ define(NBIF_RET,`NBIF_RET_N(eval(RET_POP($1)))')dnl `/* #define NBIF_RET_1 'NBIF_RET(1)` */' `/* #define NBIF_RET_2 'NBIF_RET(2)` */' `/* #define NBIF_RET_3 'NBIF_RET(3)` */' +`/* #define NBIF_RET_4 'NBIF_RET(4)` */' `/* #define NBIF_RET_5 'NBIF_RET(5)` */' dnl diff --git a/erts/emulator/hipe/hipe_sparc_bifs.m4 b/erts/emulator/hipe/hipe_sparc_bifs.m4 index 2bfe3a4646..8dfb28c8e0 100644 --- a/erts/emulator/hipe/hipe_sparc_bifs.m4 +++ b/erts/emulator/hipe/hipe_sparc_bifs.m4 @@ -54,9 +54,10 @@ define(HANDLE_GOT_MBUF,` * standard_bif_interface_1(nbif_name, cbif_name) * standard_bif_interface_2(nbif_name, cbif_name) * standard_bif_interface_3(nbif_name, cbif_name) + * standard_bif_interface_3(nbif_name, cbif_name) * standard_bif_interface_0(nbif_name, cbif_name) * - * Generate native interface for a BIF with 0-3 parameters and + * Generate native interface for a BIF with 0-4 parameters and * standard failure mode. */ define(standard_bif_interface_1, @@ -146,6 +147,39 @@ $1: .type $1, #function #endif') +define(standard_bif_interface_4, +` +#ifndef HAVE_$1 +#`define' HAVE_$1 + .global $1 +$1: + /* Set up C argument registers. */ + mov P, %o0 + NBIF_ARG(%o1,4,0) + NBIF_ARG(%o2,4,1) + NBIF_ARG(%o3,4,2) + NBIF_ARG(%o4,4,3) + + /* Save caller-save registers and call the C function. */ + SAVE_CONTEXT_BIF + st %o1, [%o0+P_ARG0] ! Store BIF__ARGS in def_arg_reg + st %o2, [%o0+P_ARG1] + st %o3, [%o0+P_ARG2] + st %o4, [%o0+P_ARG3] + add %o0, P_ARG0, %o1 + CALL_BIF($2) + nop + TEST_GOT_MBUF + + /* Restore registers. Check for exception. */ + TEST_GOT_EXN(4) + RESTORE_CONTEXT_BIF + NBIF_RET(4) + HANDLE_GOT_MBUF(4) + .size $1, .-$1 + .type $1, #function +#endif') + define(standard_bif_interface_0, ` #ifndef HAVE_$1 diff --git a/erts/emulator/hipe/hipe_sparc_glue.S b/erts/emulator/hipe/hipe_sparc_glue.S index 6c8c841194..094a87fd58 100644 --- a/erts/emulator/hipe/hipe_sparc_glue.S +++ b/erts/emulator/hipe/hipe_sparc_glue.S @@ -316,6 +316,7 @@ nbif_fail: .global nbif_1_gc_after_bif .global nbif_2_gc_after_bif .global nbif_3_gc_after_bif + .global nbif_4_gc_after_bif nbif_0_gc_after_bif: ba .gc_after_bif mov 0, %o1 /* delay slot */ @@ -326,7 +327,10 @@ nbif_2_gc_after_bif: ba .gc_after_bif mov 2, %o1 /* delay slot */ nbif_3_gc_after_bif: - mov 3, %o1 + ba .gc_after_bif + mov 3, %o1 /* delay slot */ +nbif_4_gc_after_bif: + mov 4, %o1 /*FALLTHROUGH*/ .gc_after_bif: st %o1, [P+P_NARITY] @@ -364,7 +368,11 @@ nbif_2_simple_exception: mov 2, %o1 /* delay slot */ .global nbif_3_simple_exception nbif_3_simple_exception: - mov 3, %o1 + ba .nbif_simple_exception + mov 3, %o1 /* delay slot */ + .global nbif_4_simple_exception +nbif_4_simple_exception: + mov 4, %o1 /*FALLTHROUGH*/ .nbif_simple_exception: ld [P+P_FREASON], %o0 diff --git a/erts/emulator/hipe/hipe_x86_asm.m4 b/erts/emulator/hipe/hipe_x86_asm.m4 index 39c5cb1044..436feca506 100644 --- a/erts/emulator/hipe/hipe_x86_asm.m4 +++ b/erts/emulator/hipe/hipe_x86_asm.m4 @@ -212,6 +212,7 @@ define(NBIF_COPY_NSP,`ifelse(eval($1 > NR_ARG_REGS),0,,`movl %esp, TEMP_NSP')')d `/* #define NBIF_COPY_NSP_1 'NBIF_COPY_NSP(1)` */' `/* #define NBIF_COPY_NSP_2 'NBIF_COPY_NSP(2)` */' `/* #define NBIF_COPY_NSP_3 'NBIF_COPY_NSP(3)` */' +`/* #define NBIF_COPY_NSP_4 'NBIF_COPY_NSP(4)` */' `/* #define NBIF_COPY_NSP_5 'NBIF_COPY_NSP(5)` */' dnl @@ -235,6 +236,10 @@ define(NBIF_ARG_OPND,`ifelse(eval($2 >= NR_ARG_REGS),0,`ARG'$2,BASE_OFFSET(eval( `/* #define NBIF_ARG_OPND_3_0 'NBIF_ARG_OPND(3,0)` */' `/* #define NBIF_ARG_OPND_3_1 'NBIF_ARG_OPND(3,1)` */' `/* #define NBIF_ARG_OPND_3_2 'NBIF_ARG_OPND(3,2)` */' +`/* #define NBIF_ARG_OPND_4_0 'NBIF_ARG_OPND(4,0)` */' +`/* #define NBIF_ARG_OPND_4_1 'NBIF_ARG_OPND(4,1)` */' +`/* #define NBIF_ARG_OPND_4_2 'NBIF_ARG_OPND(4,2)` */' +`/* #define NBIF_ARG_OPND_4_3 'NBIF_ARG_OPND(4,3)` */' `/* #define NBIF_ARG_OPND_5_0 'NBIF_ARG_OPND(5,0)` */' `/* #define NBIF_ARG_OPND_5_1 'NBIF_ARG_OPND(5,1)` */' `/* #define NBIF_ARG_OPND_5_2 'NBIF_ARG_OPND(5,2)` */' @@ -274,6 +279,7 @@ define(NBIF_RET,`NBIF_RET_N(eval(RET_POP($1)))')dnl `/* #define NBIF_RET_1 'NBIF_RET(1)` */' `/* #define NBIF_RET_2 'NBIF_RET(2)` */' `/* #define NBIF_RET_3 'NBIF_RET(3)` */' +`/* #define NBIF_RET_4 'NBIF_RET(4)` */' `/* #define NBIF_RET_5 'NBIF_RET(5)` */' dnl diff --git a/erts/emulator/hipe/hipe_x86_bifs.m4 b/erts/emulator/hipe/hipe_x86_bifs.m4 index a0f16efa33..b0064ee628 100644 --- a/erts/emulator/hipe/hipe_x86_bifs.m4 +++ b/erts/emulator/hipe/hipe_x86_bifs.m4 @@ -48,6 +48,7 @@ define(HANDLE_GOT_MBUF,` * standard_bif_interface_1(nbif_name, cbif_name) * standard_bif_interface_2(nbif_name, cbif_name) * standard_bif_interface_3(nbif_name, cbif_name) + * standard_bif_interface_4(nbif_name, cbif_name) * standard_bif_interface_0(nbif_name, cbif_name) * * Generate native interface for a BIF with 0-3 parameters and @@ -158,6 +159,43 @@ ASYM($1): TYPE_FUNCTION(ASYM($1)) #endif') +define(standard_bif_interface_4, +` +#ifndef HAVE_$1 +#`define' HAVE_$1 + TEXT + .align 4 + GLOBAL(ASYM($1)) +ASYM($1): + /* copy native stack pointer */ + NBIF_COPY_NSP(4) + + /* switch to C stack */ + SWITCH_ERLANG_TO_C + + /* make the call on the C stack */ + NBIF_ARG_REG(0,P) + NBIF_ARG(2,4,0) + NBIF_ARG(3,4,1) + NBIF_ARG(4,4,2) + NBIF_ARG(5,4,3) + lea 8(%esp), %eax + NBIF_ARG_REG(1,%eax) /* BIF__ARGS */ + CALL_BIF($2) + TEST_GOT_MBUF + + /* switch to native stack */ + SWITCH_C_TO_ERLANG + + /* throw exception if failure, otherwise return */ + TEST_GOT_EXN + jz nbif_4_simple_exception + NBIF_RET(4) + HANDLE_GOT_MBUF(4) + SET_SIZE(ASYM($1)) + TYPE_FUNCTION(ASYM($1)) +#endif') + define(standard_bif_interface_0, ` #ifndef HAVE_$1 diff --git a/erts/emulator/hipe/hipe_x86_glue.S b/erts/emulator/hipe/hipe_x86_glue.S index 9d38eaaafd..f124e36a26 100644 --- a/erts/emulator/hipe/hipe_x86_glue.S +++ b/erts/emulator/hipe/hipe_x86_glue.S @@ -299,6 +299,7 @@ ASYM(nbif_fail): GLOBAL(nbif_1_gc_after_bif) GLOBAL(nbif_2_gc_after_bif) GLOBAL(nbif_3_gc_after_bif) + GLOBAL(nbif_4_gc_after_bif) .align 4 nbif_0_gc_after_bif: xorl %edx, %edx @@ -314,6 +315,10 @@ nbif_2_gc_after_bif: .align 4 nbif_3_gc_after_bif: movl $3, %edx + jmp .gc_after_bif + .align 4 +nbif_4_gc_after_bif: + movl $4, %edx /*FALLTHROUGH*/ .align 4 .gc_after_bif: @@ -337,6 +342,7 @@ nbif_3_gc_after_bif: GLOBAL(nbif_1_simple_exception) GLOBAL(nbif_2_simple_exception) GLOBAL(nbif_3_simple_exception) + GLOBAL(nbif_4_simple_exception) .align 4 nbif_0_simple_exception: xorl %eax, %eax @@ -352,6 +358,10 @@ nbif_2_simple_exception: .align 4 nbif_3_simple_exception: movl $3, %eax + jmp .nbif_simple_exception + .align 4 +nbif_4_simple_exception: + movl $4, %eax /*FALLTHROUGH*/ .align 4 .nbif_simple_exception: diff --git a/erts/emulator/sys/common/erl_check_io.c b/erts/emulator/sys/common/erl_check_io.c index 0051b45b31..7be17d20bb 100644 --- a/erts/emulator/sys/common/erl_check_io.c +++ b/erts/emulator/sys/common/erl_check_io.c @@ -1590,9 +1590,9 @@ ERTS_CIO_EXPORT(erts_check_io_interrupt)(int set) void ERTS_CIO_EXPORT(erts_check_io_interrupt_timed)(int set, - erts_short_time_t msec) + ErtsMonotonicTime timeout_time) { - ERTS_CIO_POLL_INTR_TMD(pollset.ps, set, msec); + ERTS_CIO_POLL_INTR_TMD(pollset.ps, set, timeout_time); } void @@ -1600,9 +1600,12 @@ ERTS_CIO_EXPORT(erts_check_io)(int do_wait) { ErtsPollResFd *pollres; int pollres_len; - SysTimeval wait_time; + ErtsMonotonicTime timeout_time; int poll_ret, i; erts_aint_t current_cio_time; + ErtsSchedulerData *esdp = erts_get_scheduler_data(); + + ASSERT(esdp); restart: @@ -1612,12 +1615,10 @@ ERTS_CIO_EXPORT(erts_check_io)(int do_wait) #endif /* Figure out timeout value */ - if (do_wait) { - erts_time_remaining(&wait_time); - } else { /* poll only */ - wait_time.tv_sec = 0; - wait_time.tv_usec = 0; - } + timeout_time = (do_wait + ? erts_check_next_timeout_time(esdp->timer_wheel, + ERTS_SEC_TO_MONOTONIC(10*60)) + : ERTS_POLL_NO_TIMEOUT /* poll only */); /* * No need for an atomic inc op when incrementing @@ -1640,14 +1641,12 @@ ERTS_CIO_EXPORT(erts_check_io)(int do_wait) erts_smp_atomic_set_nob(&pollset.in_poll_wait, 1); - poll_ret = ERTS_CIO_POLL_WAIT(pollset.ps, pollres, &pollres_len, &wait_time); + poll_ret = ERTS_CIO_POLL_WAIT(pollset.ps, pollres, &pollres_len, timeout_time); #ifdef ERTS_ENABLE_LOCK_CHECK erts_lc_check_exact(NULL, 0); /* No locks should be locked */ #endif - erts_deliver_time(); /* sync the machine's idea of time */ - #ifdef ERTS_BREAK_REQUESTED if (ERTS_BREAK_REQUESTED) erts_do_break_handling(); diff --git a/erts/emulator/sys/common/erl_check_io.h b/erts/emulator/sys/common/erl_check_io.h index d01297d55c..71355965aa 100644 --- a/erts/emulator/sys/common/erl_check_io.h +++ b/erts/emulator/sys/common/erl_check_io.h @@ -47,8 +47,8 @@ void erts_check_io_async_sig_interrupt_nkp(void); #endif void erts_check_io_interrupt_kp(int); void erts_check_io_interrupt_nkp(int); -void erts_check_io_interrupt_timed_kp(int, erts_short_time_t); -void erts_check_io_interrupt_timed_nkp(int, erts_short_time_t); +void erts_check_io_interrupt_timed_kp(int, ErtsMonotonicTime); +void erts_check_io_interrupt_timed_nkp(int, ErtsMonotonicTime); void erts_check_io_kp(int); void erts_check_io_nkp(int); void erts_init_check_io_kp(void); @@ -65,7 +65,7 @@ int erts_check_io_max_files(void); void erts_check_io_async_sig_interrupt(void); #endif void erts_check_io_interrupt(int); -void erts_check_io_interrupt_timed(int, erts_short_time_t); +void erts_check_io_interrupt_timed(int, ErtsMonotonicTime); void erts_check_io(int); void erts_init_check_io(void); diff --git a/erts/emulator/sys/common/erl_os_monotonic_time_extender.c b/erts/emulator/sys/common/erl_os_monotonic_time_extender.c new file mode 100644 index 0000000000..f3633b7267 --- /dev/null +++ b/erts/emulator/sys/common/erl_os_monotonic_time_extender.c @@ -0,0 +1,88 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2015. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#include "erl_os_monotonic_time_extender.h" + +#ifdef USE_THREADS + +static void *os_monotonic_time_extender(void *vstatep) +{ + ErtsOsMonotonicTimeExtendState *state = (ErtsOsMonotonicTimeExtendState *) vstatep; + long sleep_time = state->check_interval*1000; + Uint32 (*raw_os_mtime)(void) = state->raw_os_monotonic_time; + Uint32 last_msb = 0; + + while (1) { + Uint32 msb = (*raw_os_mtime)() & (((Uint32) 1) << 31); + + if (msb != last_msb) { + int ix = ((int) (last_msb >> 31)) & 1; + Uint32 xtnd = (Uint32) erts_atomic32_read_nob(&state->extend[ix]); + erts_atomic32_set_nob(&state->extend[ix], (erts_aint32_t) (xtnd + 1)); + last_msb = msb; + } + erts_milli_sleep(sleep_time); + } + + erl_exit(ERTS_ABORT_EXIT, "os_monotonic_time_extender thread terminating"); + return NULL; +} + +static erts_tid_t os_monotonic_extender_tid; +#endif + +void +erts_init_os_monotonic_time_extender(ErtsOsMonotonicTimeExtendState *statep, + Uint32 (*raw_os_monotonic_time)(void), + int check_seconds) +{ +#ifdef USE_THREADS + statep->raw_os_monotonic_time = raw_os_monotonic_time; + erts_atomic32_init_nob(&statep->extend[0], (erts_aint32_t) 0); + erts_atomic32_init_nob(&statep->extend[1], (erts_aint32_t) 0); + statep->check_interval = check_seconds; + +#else + statep->extend[0] = (Uint32) 0; + statep->extend[1] = (Uint32) 0; + statep->last_msb = (ErtsMonotonicTime) 0; +#endif +} + +void +erts_late_init_os_monotonic_time_extender(ErtsOsMonotonicTimeExtendState *statep) +{ +#ifdef USE_THREADS + erts_thr_opts_t thr_opts = ERTS_THR_OPTS_DEFAULT_INITER; + thr_opts.detached = 1; + thr_opts.suggested_stack_size = 4; + +#if 0 + thr_opts.name = "os_monotonic_time_extender"; +#endif + + erts_thr_create(&os_monotonic_extender_tid, + os_monotonic_time_extender, + (void*) statep, + &thr_opts); +#endif +} diff --git a/erts/emulator/sys/common/erl_os_monotonic_time_extender.h b/erts/emulator/sys/common/erl_os_monotonic_time_extender.h new file mode 100644 index 0000000000..0f9e7c86ae --- /dev/null +++ b/erts/emulator/sys/common/erl_os_monotonic_time_extender.h @@ -0,0 +1,65 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2015. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + +#ifndef ERL_OS_MONOTONIC_TIME_EXTENDER_H__ +#define ERL_OS_MONOTONIC_TIME_EXTENDER_H__ + +#include "sys.h" +#include "erl_threads.h" + +typedef struct { +#ifdef USE_THREADS + Uint32 (*raw_os_monotonic_time)(void); + erts_atomic32_t extend[2]; + int check_interval; +#else + Uint32 extend[2]; + ErtsMonotonicTime last_msb; +#endif +} ErtsOsMonotonicTimeExtendState; + +#ifdef USE_THREADS +# define ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(S, RT) ((void) 1) +# define ERTS_EXTEND_OS_MONOTONIC_TIME(S, RT) \ + ((((ErtsMonotonicTime) \ + erts_atomic32_read_nob(&((S)->extend[((int) ((RT) >> 31)) & 1]))) \ + << 32) \ + + (RT)) +#else +# define ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(S, RT) \ + do { \ + Uint32 msb__ = (RT) & (((Uint32) 1) << 31); \ + if (msb__ != (S)->last_msb) { \ + int ix__ = ((int) ((S)->last_msb >> 31)) & 1; \ + (S)->extend[ix__]++; \ + (S)->last_msb = msb; \ + } \ + } while (0) +# define ERTS_EXTEND_OS_MONOTONIC_TIME(S, RT) \ + ((((ErtsMonotonicTime) (S)->extend[((int) ((RT) >> 31)) & 1]) << 32) + (RT)) +#endif + +void +erts_init_os_monotonic_time_extender(ErtsOsMonotonicTimeExtendState *statep, + Uint32 (*raw_os_monotonic_time)(void), + int check_seconds); +void +erts_late_init_os_monotonic_time_extender(ErtsOsMonotonicTimeExtendState *statep); + +#endif diff --git a/erts/emulator/sys/common/erl_poll.c b/erts/emulator/sys/common/erl_poll.c index aa412a20c8..f4d4a85ca4 100644 --- a/erts/emulator/sys/common/erl_poll.c +++ b/erts/emulator/sys/common/erl_poll.c @@ -320,7 +320,7 @@ struct ErtsPollSet_ { #if defined(USE_THREADS) || ERTS_POLL_ASYNC_INTERRUPT_SUPPORT erts_atomic32_t wakeup_state; #endif - erts_smp_atomic32_t timeout; + erts_atomic64_t timeout_time; #ifdef ERTS_POLL_COUNT_AVOIDED_WAKEUPS erts_smp_atomic_t no_avoided_wakeups; erts_smp_atomic_t no_avoided_interrupts; @@ -384,6 +384,26 @@ static void check_poll_status(ErtsPollSet ps); static void print_misc_debug_info(void); #endif +static ERTS_INLINE void +init_timeout_time(ErtsPollSet ps) +{ + erts_atomic64_init_nob(&ps->timeout_time, + (erts_aint64_t) ERTS_MONOTONIC_TIME_MAX); +} + +static ERTS_INLINE void +set_timeout_time(ErtsPollSet ps, ErtsMonotonicTime time) +{ + erts_atomic64_set_relb(&ps->timeout_time, + (erts_aint64_t) time); +} + +static ERTS_INLINE ErtsMonotonicTime +get_timeout_time(ErtsPollSet ps) +{ + return (ErtsMonotonicTime) erts_atomic64_read_acqb(&ps->timeout_time); +} + #define ERTS_POLL_NOT_WOKEN 0 #define ERTS_POLL_WOKEN -1 #define ERTS_POLL_WOKEN_INTR 1 @@ -1993,44 +2013,153 @@ save_poll_result(ErtsPollSet ps, ErtsPollResFd pr[], int max_res, } } +static ERTS_INLINE ErtsMonotonicTime +get_timeout(ErtsPollSet ps, + int resolution, + ErtsMonotonicTime timeout_time) +{ + ErtsMonotonicTime timeout, save_timeout_time; + + if (timeout_time == ERTS_POLL_NO_TIMEOUT) { + save_timeout_time = ERTS_MONOTONIC_TIME_MIN; + timeout = 0; + } + else { + ErtsMonotonicTime diff_time, current_time; + current_time = erts_get_monotonic_time(); + diff_time = timeout_time - current_time; + if (diff_time <= 0) { + save_timeout_time = ERTS_MONOTONIC_TIME_MIN; + timeout = 0; + } + else { + save_timeout_time = current_time; + switch (resolution) { + case 1000: + /* Round up to nearest even milli second */ + timeout = ERTS_MONOTONIC_TO_MSEC(diff_time - 1) + 1; + if (timeout > (ErtsMonotonicTime) INT_MAX) + timeout = (ErtsMonotonicTime) INT_MAX; + save_timeout_time += ERTS_MSEC_TO_MONOTONIC(timeout); + break; + case 1000000: + /* Round up to nearest even micro second */ + timeout = ERTS_MONOTONIC_TO_USEC(diff_time - 1) + 1; + save_timeout_time += ERTS_USEC_TO_MONOTONIC(timeout); + break; + case 1000000000: + /* Round up to nearest even nano second */ + timeout = ERTS_MONOTONIC_TO_NSEC(diff_time - 1) + 1; + save_timeout_time += ERTS_NSEC_TO_MONOTONIC(timeout); + break; + default: + ERTS_INTERNAL_ERROR("Invalid resolution"); + timeout = 0; + save_timeout_time = 0; + break; + } + } + } + set_timeout_time(ps, save_timeout_time); + return timeout; +} + +#if ERTS_POLL_USE_SELECT + static ERTS_INLINE int -check_fd_events(ErtsPollSet ps, SysTimeval *tv, int max_res) +get_timeout_timeval(ErtsPollSet ps, + SysTimeval *tvp, + ErtsMonotonicTime timeout_time) +{ + ErtsMonotonicTime timeout = get_timeout(ps, + 1000*1000, + timeout_time); + + if (!timeout) { + tvp->tv_sec = 0; + tvp->tv_usec = 0; + + return 0; + } + else { + ErtsMonotonicTime sec = timeout/(1000*1000); + tvp->tv_sec = sec; + tvp->tv_usec = timeout - sec*(1000*1000); + + ASSERT(tvp->tv_sec >= 0); + ASSERT(tvp->tv_usec >= 0); + ASSERT(tvp->tv_usec < 1000*1000); + + return !0; + } + +} + +#endif + +#if ERTS_POLL_USE_KQUEUE + +static ERTS_INLINE int +get_timeout_timespec(ErtsPollSet ps, + struct timespec *tsp, + ErtsMonotonicTime timeout_time) +{ + ErtsMonotonicTime timeout = get_timeout(ps, + 1000*1000*1000, + timeout_time); + + if (!timeout) { + tsp->tv_sec = 0; + tsp->tv_nsec = 0; + return 0; + } + else { + ErtsMonotonicTime sec = timeout/(1000*1000*1000); + tsp->tv_sec = sec; + tsp->tv_nsec = timeout - sec*(1000*1000*1000); + + ASSERT(tsp->tv_sec >= 0); + ASSERT(tsp->tv_nsec >= 0); + ASSERT(tsp->tv_nsec < 1000*1000); + + return !0; + } +} + +#endif + +static ERTS_INLINE int +check_fd_events(ErtsPollSet ps, ErtsMonotonicTime timeout_time, int max_res) { int res; if (erts_smp_atomic_read_nob(&ps->no_of_user_fds) == 0 - && tv->tv_usec == 0 && tv->tv_sec == 0) { + && timeout_time == ERTS_POLL_NO_TIMEOUT) { /* Nothing to poll and zero timeout; done... */ return 0; } else { - long timeout = tv->tv_sec*1000 + tv->tv_usec/1000; - if (timeout > ERTS_AINT32_T_MAX) - timeout = ERTS_AINT32_T_MAX; - ASSERT(timeout >= 0); - erts_smp_atomic32_set_relb(&ps->timeout, (erts_aint32_t) timeout); + int timeout; #if ERTS_POLL_USE_FALLBACK if (!(ps->fallback_used = ERTS_POLL_NEED_FALLBACK(ps))) { #if ERTS_POLL_USE_EPOLL /* --- epoll ------------------------------- */ - if (timeout > INT_MAX) - timeout = INT_MAX; if (max_res > ps->res_events_len) grow_res_events(ps, max_res); + timeout = (int) get_timeout(ps, 1000, timeout_time); #ifdef ERTS_SMP if (timeout) erts_thr_progress_prepare_wait(NULL); #endif - res = epoll_wait(ps->kp_fd, ps->res_events, max_res, (int)timeout); + res = epoll_wait(ps->kp_fd, ps->res_events, max_res, timeout); #elif ERTS_POLL_USE_KQUEUE /* --- kqueue ------------------------------ */ struct timespec ts; if (max_res > ps->res_events_len) grow_res_events(ps, max_res); + timeout = get_timeout_timespec(ps, &ts, timeout_time); #ifdef ERTS_SMP if (timeout) erts_thr_progress_prepare_wait(NULL); #endif - ts.tv_sec = tv->tv_sec; - ts.tv_nsec = tv->tv_usec*1000; res = kevent(ps->kp_fd, NULL, 0, ps->res_events, max_res, &ts); #endif /* ----------------------------------------- */ } @@ -2049,8 +2178,7 @@ check_fd_events(ErtsPollSet ps, SysTimeval *tv, int max_res) #if ERTS_POLL_USE_WAKEUP_PIPE nfds++; /* Wakeup pipe */ #endif - if (timeout > INT_MAX) - timeout = INT_MAX; + timeout = (int) get_timeout(ps, 1000, timeout_time); poll_res.dp_nfds = nfds < max_res ? nfds : max_res; if (poll_res.dp_nfds > ps->res_events_len) grow_res_events(ps, poll_res.dp_nfds); @@ -2059,33 +2187,33 @@ check_fd_events(ErtsPollSet ps, SysTimeval *tv, int max_res) if (timeout) erts_thr_progress_prepare_wait(NULL); #endif - poll_res.dp_timeout = (int) timeout; + poll_res.dp_timeout = timeout; res = ioctl(ps->kp_fd, DP_POLL, &poll_res); #elif ERTS_POLL_USE_POLL /* --- poll -------------------------------- */ - if (timeout > INT_MAX) - timeout = INT_MAX; + timeout = (int) get_timeout(ps, 1000, timeout_time); #ifdef ERTS_SMP if (timeout) erts_thr_progress_prepare_wait(NULL); #endif - res = poll(ps->poll_fds, ps->no_poll_fds, (int) timeout); + res = poll(ps->poll_fds, ps->no_poll_fds, timeout); #elif ERTS_POLL_USE_SELECT /* --- select ------------------------------ */ - SysTimeval to = *tv; + SysTimeval to; + timeout = get_timeout_timeval(ps, &to, timeout_time); ERTS_FD_COPY(&ps->input_fds, &ps->res_input_fds); ERTS_FD_COPY(&ps->output_fds, &ps->res_output_fds); #ifdef ERTS_SMP - if (to.tv_sec || to.tv_usec) + if (timeout) erts_thr_progress_prepare_wait(NULL); #endif res = ERTS_SELECT(ps->max_fd + 1, - &ps->res_input_fds, - &ps->res_output_fds, - NULL, - &to); + &ps->res_input_fds, + &ps->res_output_fds, + NULL, + &to); #ifdef ERTS_SMP - if (to.tv_sec || to.tv_usec) + if (timeout) erts_thr_progress_finalize_wait(NULL); if (res < 0 && errno == EBADF @@ -2108,10 +2236,10 @@ check_fd_events(ErtsPollSet ps, SysTimeval *tv, int max_res) handle_update_requests(ps); ERTS_POLLSET_UNLOCK(ps); res = ERTS_SELECT(ps->max_fd + 1, - &ps->res_input_fds, - &ps->res_output_fds, - NULL, - &to); + &ps->res_input_fds, + &ps->res_output_fds, + NULL, + &to); if (res == 0) { errno = EAGAIN; res = -1; @@ -2133,15 +2261,14 @@ int ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet ps, ErtsPollResFd pr[], int *len, - SysTimeval *utvp) + ErtsMonotonicTime timeout_time) { + ErtsMonotonicTime to; int res, no_fds; int ebadf = 0; #ifdef ERTS_SMP int ps_locked = 0; #endif - SysTimeval *tvp; - SysTimeval itv; no_fds = *len; #ifdef ERTS_POLL_MAX_RES @@ -2151,13 +2278,9 @@ ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet ps, *len = 0; - ASSERT(utvp); - - tvp = utvp; - #ifdef ERTS_POLL_DEBUG_PRINT - erts_printf("Entering erts_poll_wait(), timeout=%d\n", - (int) tvp->tv_sec*1000 + tvp->tv_usec/1000); + erts_printf("Entering erts_poll_wait(), timeout_time=%bps\n", + timeout_time); #endif if (ERTS_POLLSET_SET_POLLED_CHK(ps)) { @@ -2166,12 +2289,9 @@ ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet ps, goto done; } - if (is_woken(ps)) { - /* Use zero timeout */ - itv.tv_sec = 0; - itv.tv_usec = 0; - tvp = &itv; - } + to = (is_woken(ps) + ? ERTS_POLL_NO_TIMEOUT /* Use zero timeout */ + : timeout_time); #if ERTS_POLL_USE_UPDATE_REQUESTS_QUEUE if (ERTS_POLLSET_HAVE_UPDATE_REQUESTS(ps)) { @@ -2181,7 +2301,7 @@ ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet ps, } #endif - res = check_fd_events(ps, tvp, no_fds); + res = check_fd_events(ps, to, no_fds); woke_up(ps); @@ -2224,7 +2344,7 @@ ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet ps, #endif done: - erts_smp_atomic32_set_relb(&ps->timeout, ERTS_AINT32_T_MAX); + set_timeout_time(ps, ERTS_MONOTONIC_TIME_MAX); #ifdef ERTS_POLL_DEBUG_PRINT erts_printf("Leaving %s = erts_poll_wait()\n", res == 0 ? "0" : erl_errno_id(res)); @@ -2268,13 +2388,14 @@ ERTS_POLL_EXPORT(erts_poll_async_sig_interrupt)(ErtsPollSet ps) void ERTS_POLL_EXPORT(erts_poll_interrupt_timed)(ErtsPollSet ps, int set, - erts_short_time_t msec) + ErtsMonotonicTime timeout_time) { #if ERTS_POLL_ASYNC_INTERRUPT_SUPPORT || defined(ERTS_SMP) if (!set) reset_wakeup_state(ps); else { - if (erts_smp_atomic32_read_acqb(&ps->timeout) > (erts_aint32_t) msec) + ErtsMonotonicTime max_wait_time = get_timeout_time(ps); + if (max_wait_time > timeout_time) wake_poller(ps, 1, 0); #ifdef ERTS_POLL_COUNT_AVOIDED_WAKEUPS else { @@ -2431,7 +2552,7 @@ ERTS_POLL_EXPORT(erts_poll_create_pollset)(void) ps->internal_fd_limit = kp_fd + 1; ps->kp_fd = kp_fd; #endif - erts_smp_atomic32_init_nob(&ps->timeout, ERTS_AINT32_T_MAX); + init_timeout_time(ps); #ifdef ERTS_POLL_COUNT_AVOIDED_WAKEUPS erts_smp_atomic_init_nob(&ps->no_avoided_wakeups, 0); erts_smp_atomic_init_nob(&ps->no_avoided_interrupts, 0); diff --git a/erts/emulator/sys/common/erl_poll.h b/erts/emulator/sys/common/erl_poll.h index 2f1c05f401..d02ed2396b 100644 --- a/erts/emulator/sys/common/erl_poll.h +++ b/erts/emulator/sys/common/erl_poll.h @@ -29,6 +29,8 @@ #include "sys.h" +#define ERTS_POLL_NO_TIMEOUT ERTS_MONOTONIC_TIME_MIN + #if 0 #define ERTS_POLL_COUNT_AVOIDED_WAKEUPS #endif @@ -241,7 +243,7 @@ void ERTS_POLL_EXPORT(erts_poll_interrupt)(ErtsPollSet, int); void ERTS_POLL_EXPORT(erts_poll_interrupt_timed)(ErtsPollSet, int, - erts_short_time_t); + ErtsMonotonicTime); ErtsPollEvents ERTS_POLL_EXPORT(erts_poll_control)(ErtsPollSet, ErtsSysFdType, ErtsPollEvents, @@ -254,7 +256,7 @@ void ERTS_POLL_EXPORT(erts_poll_controlv)(ErtsPollSet, int ERTS_POLL_EXPORT(erts_poll_wait)(ErtsPollSet, ErtsPollResFd [], int *, - SysTimeval *); + ErtsMonotonicTime); int ERTS_POLL_EXPORT(erts_poll_max_fds)(void); void ERTS_POLL_EXPORT(erts_poll_info)(ErtsPollSet, ErtsPollInfo *); diff --git a/erts/emulator/sys/ose/erl_poll.c b/erts/emulator/sys/ose/erl_poll.c index 7d2a3d1e0b..36ee2557e8 100644 --- a/erts/emulator/sys/ose/erl_poll.c +++ b/erts/emulator/sys/ose/erl_poll.c @@ -114,7 +114,7 @@ struct ErtsPollSet_ { Uint item_count; PROCESS interrupt; erts_atomic32_t wakeup_state; - erts_smp_atomic32_t timeout; + erts_atomic64_t timeout_time; #ifdef ERTS_SMP erts_smp_mtx_t mtx; #endif @@ -122,6 +122,26 @@ struct ErtsPollSet_ { static int max_fds = -1; +static ERTS_INLINE void +init_timeout_time(ErtsPollSet ps) +{ + erts_atomic64_init_nob(&ps->timeout_time, + (erts_aint64_t) ERTS_MONOTONIC_TIME_MAX); +} + +static ERTS_INLINE void +set_timeout_time(ErtsPollSet ps, ErtsMonotonicTime time) +{ + erts_atomic64_set_relb(&ps->timeout_time, + (erts_aint64_t) time); +} + +static ERTS_INLINE ErtsMonotonicTime +get_timeout_time(ErtsPollSet ps) +{ + return (ErtsMonotonicTime) erts_atomic64_read_acqb(&ps->timeout_time); +} + #define ERTS_POLL_NOT_WOKEN ((erts_aint32_t) (1 << 0)) #define ERTS_POLL_WOKEN_INTR ((erts_aint32_t) (1 << 1)) #define ERTS_POLL_WOKEN_TIMEDOUT ((erts_aint32_t) (1 << 2)) @@ -386,12 +406,14 @@ void erts_poll_interrupt(ErtsPollSet ps,int set) { } -void erts_poll_interrupt_timed(ErtsPollSet ps,int set,erts_short_time_t msec) { +void erts_poll_interrupt_timed(ErtsPollSet ps, + int set, + ErtsTimeoutTime timeout_time) { HARDTRACEF("erts_poll_interrupt_timed called!\n"); if (!set) reset_interrupt(ps); - else if (erts_smp_atomic32_read_acqb(&ps->timeout) > (erts_aint32_t) msec) + else if (get_timeout_time(ps) > timeout_time) set_interrupt(ps); } @@ -453,12 +475,14 @@ done: } int erts_poll_wait(ErtsPollSet ps, - ErtsPollResFd pr[], - int *len, - SysTimeval *utvp) { + ErtsPollResFd pr[], + int *len, + ErtsMonotonicTime timeout_time) +{ int res = ETIMEDOUT, no_fds, currid = 0; OSTIME timeout; union SIGNAL *sig; + ErtsMonotonicTime current_time, diff_time, timeout; // HARDTRACEF("%ux: In erts_poll_wait",ps); if (ps->interrupt == (PROCESS)0) ps->interrupt = current_process(); @@ -472,16 +496,29 @@ int erts_poll_wait(ErtsPollSet ps, *len = 0; - ASSERT(utvp); + /* erts_printf("Entering erts_poll_wait(), timeout_time=%bps\n", + timeout_time); */ - /* erts_printf("Entering erts_poll_wait(), timeout=%d\n", - (int) utvp->tv_sec*1000 + utvp->tv_usec/1000); */ - - timeout = utvp->tv_sec*1000 + utvp->tv_usec/1000; + if (timeout_time == ERTS_POLL_NO_TIMEOUT) { + no_timeout: + timeout = (OSTIME) 0; + save_timeout_time = ERTS_MONOTONIC_TIME_MIN; + } + else { + ErtsMonotonicTime current_time, diff_time; + current_time = erts_get_monotonic_time(); + diff_time = timeout_time - current_time; + if (diff_time <= 0) + goto no_timeout; + diff_time = (ERTS_MONOTONIC_TO_MSEC(diff_time - 1) + 1); + if (diff_time > INT_MAX) + diff_time = INT_MAX; + timeout = (OSTIME) diff_time; + save_timeout_time = current_time; + save_timeout_time += ERTS_MSEC_TO_MONOTONIC(diff_time); + } - if (timeout > ((time_t) ERTS_AINT32_T_MAX)) - timeout = ERTS_AINT32_T_MAX; - erts_smp_atomic32_set_relb(&ps->timeout, (erts_aint32_t) timeout); + set_timeout_time(ps, save_timeout_time); while (currid < no_fds) { if (timeout > 0) { @@ -627,7 +664,7 @@ int erts_poll_wait(ErtsPollSet ps, } erts_atomic32_set_nob(&ps->wakeup_state, ERTS_POLL_NOT_WOKEN); - erts_smp_atomic32_set_nob(&ps->timeout, ERTS_AINT32_T_MAX); + set_timeout_time(ps, ERTS_MONOTONIC_TIME_MAX); *len = currid; @@ -690,7 +727,7 @@ ErtsPollSet erts_poll_create_pollset(void) ps->info = NULL; ps->interrupt = (PROCESS)0; erts_atomic32_init_nob(&ps->wakeup_state, ERTS_POLL_NOT_WOKEN); - erts_smp_atomic32_init_nob(&ps->timeout, ERTS_AINT32_T_MAX); + init_timeout_time(ps); #ifdef ERTS_SMP erts_smp_mtx_init(&ps->mtx, "pollset"); #endif diff --git a/erts/emulator/sys/ose/sys.c b/erts/emulator/sys/ose/sys.c index 5b950a7dae..13a5b71496 100644 --- a/erts/emulator/sys/ose/sys.c +++ b/erts/emulator/sys/ose/sys.c @@ -298,9 +298,9 @@ erts_sys_schedule_interrupt(int set) #ifdef ERTS_SMP void -erts_sys_schedule_interrupt_timed(int set, erts_short_time_t msec) +erts_sys_schedule_interrupt_timed(int set, ErtsMonotonicTime timeout_time) { - ERTS_CHK_IO_INTR_TMD(set, msec); + ERTS_CHK_IO_INTR_TMD(set, timeout_time); } #endif diff --git a/erts/emulator/sys/unix/erl_unix_sys.h b/erts/emulator/sys/unix/erl_unix_sys.h index f0050db114..8fc5a3ca49 100644 --- a/erts/emulator/sys/unix/erl_unix_sys.h +++ b/erts/emulator/sys/unix/erl_unix_sys.h @@ -114,11 +114,6 @@ /* * Make sure that MAXPATHLEN is defined. */ -#ifdef GETHRTIME_WITH_CLOCK_GETTIME -#undef HAVE_GETHRTIME -#define HAVE_GETHRTIME 1 -#endif - #ifndef MAXPATHLEN # ifdef PATH_MAX # define MAXPATHLEN PATH_MAX @@ -160,33 +155,134 @@ typedef struct timeval SysTimeval; typedef struct tms SysTimes; -extern int erts_ticks_per_sec; - -#define SYS_CLK_TCK (erts_ticks_per_sec) +#define SYS_CLK_TCK (erts_sys_time_data__.r.o.ticks_per_sec) #define sys_times(Arg) times(Arg) -#define ERTS_WRAP_SYS_TIMES 1 -extern int erts_ticks_per_sec_wrap; -#define SYS_CLK_TCK_WRAP (erts_ticks_per_sec_wrap) -extern clock_t sys_times_wrap(void); +#if SIZEOF_LONG == 8 +typedef long ErtsMonotonicTime; +typedef long ErtsSysHrTime; +#elif SIZEOF_LONG_LONG == 8 +typedef long long ErtsMonotonicTime; +typedef long long ErtsSysHrTime; +#else +#error No signed 64-bit type found... +#endif + +typedef ErtsMonotonicTime ErtsSystemTime; + +#define ERTS_MONOTONIC_TIME_MIN (((ErtsMonotonicTime) 1) << 63) +#define ERTS_MONOTONIC_TIME_MAX (~ERTS_MONOTONIC_TIME_MIN) + +/* + * OS monotonic time and OS system time + */ + +#undef ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ -#ifdef HAVE_GETHRTIME -#ifdef GETHRTIME_WITH_CLOCK_GETTIME -typedef long long SysHrTime; +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) +# if defined(__linux__) +# define ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ 1 +# endif +#endif + +ErtsSystemTime erts_os_system_time(void); -extern SysHrTime sys_gethrtime(void); -#define sys_init_hrtime() /* Nothing */ +#undef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT +#undef ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT +#undef ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ +#undef ERTS_HAVE_CORRECTED_OS_MONOTONIC + +#if defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) +# define ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT 1 +# define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT (1000*1000*1000) +# if defined(__linux__) +# define ERTS_HAVE_CORRECTED_OS_MONOTONIC 1 +# define ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ 1 +# endif +#elif defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) +# define ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT 1 +# define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT (1000*1000*1000) +#elif defined(OS_MONOTONIC_TIME_USING_GETHRTIME) +# define ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT 1 +# define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT (1000*1000*1000) +#elif defined(OS_MONOTONIC_TIME_USING_TIMES) +# define ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT 1 +/* Time unit determined at runtime... */ +# define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT 0 +#else /* No OS monotonic available... */ +# define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT (1000*1000) +#endif -#else /* Real gethrtime (Solaris) */ +/* + * erts_sys_hrtime() is the highest resolution + * time function found. Time unit is nano-seconds. + * It may or may not be monotonic. + */ +ErtsSysHrTime erts_sys_hrtime(void); + +struct erts_sys_time_read_only_data__ { +#ifdef ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ + ErtsMonotonicTime (*os_monotonic_time)(void); +#endif +#ifdef ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ + void (*os_times)(ErtsMonotonicTime *, ErtsSystemTime *); +#endif + int ticks_per_sec; +}; -typedef hrtime_t SysHrTime; +typedef struct { + union { + struct erts_sys_time_read_only_data__ o; + char align__[(((sizeof(struct erts_sys_time_read_only_data__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } r; +} ErtsSysTimeData__; -#define sys_gethrtime() gethrtime() -#define sys_init_hrtime() /* Nothing */ +extern ErtsSysTimeData__ erts_sys_time_data__; -#endif /* GETHRTIME_WITH_CLOCK_GETTIME */ -#endif /* HAVE_GETHRTIME */ +#ifdef ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT + +#ifdef ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ +ERTS_GLB_INLINE +#endif +ErtsMonotonicTime erts_os_monotonic_time(void); + +#ifdef ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ +ERTS_GLB_INLINE +#endif +void erts_os_times(ErtsMonotonicTime *, ErtsSystemTime *); + +#if ERTS_GLB_INLINE_INCL_FUNC_DEF + +#ifdef ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ + +ERTS_GLB_INLINE ErtsMonotonicTime +erts_os_monotonic_time(void) +{ + return (*erts_sys_time_data__.r.o.os_monotonic_time)(); +} + +#endif /* ERTS_OS_MONOTONIC_INLINE_FUNC_PTR_CALL__ */ + +#ifdef ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ + +ERTS_GLB_INLINE void +erts_os_times(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + return (*erts_sys_time_data__.r.o.os_times)(mtimep, stimep); +} + +#endif /* ERTS_OS_TIMES_INLINE_FUNC_PTR_CALL__ */ + +#endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ + +#endif /* ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT */ + +/* + * + */ #if (defined(HAVE_GETHRVTIME) || defined(HAVE_CLOCK_GETTIME_CPU_TIME)) typedef long long SysCpuTime; diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index 0d9c743c0c..7d52650a70 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -308,7 +308,7 @@ struct { int (*event)(ErlDrvPort, ErlDrvEvent, ErlDrvEventData); void (*check_io_as_interrupt)(void); void (*check_io_interrupt)(int); - void (*check_io_interrupt_tmd)(int, erts_short_time_t); + void (*check_io_interrupt_tmd)(int, ErtsMonotonicTime); void (*check_io)(int); Uint (*size)(void); Eterm (*info)(void *); @@ -414,9 +414,9 @@ erts_sys_schedule_interrupt(int set) #ifdef ERTS_SMP void -erts_sys_schedule_interrupt_timed(int set, erts_short_time_t msec) +erts_sys_schedule_interrupt_timed(int set, ErtsMonotonicTime timeout_time) { - ERTS_CHK_IO_INTR_TMD(set, msec); + ERTS_CHK_IO_INTR_TMD(set, timeout_time); } #endif @@ -532,11 +532,14 @@ thr_create_prepare_child(void *vtcdp) void erts_sys_pre_init(void) { +#ifdef USE_THREADS + erts_thr_init_data_t eid = ERTS_THR_INIT_DATA_DEF_INITER; +#endif + erts_printf_add_cr_to_stdout = 1; erts_printf_add_cr_to_stderr = 1; + #ifdef USE_THREADS - { - erts_thr_init_data_t eid = ERTS_THR_INIT_DATA_DEF_INITER; eid.thread_create_child_func = thr_create_prepare_child; /* Before creation in parent */ @@ -559,6 +562,12 @@ erts_sys_pre_init(void) erts_lcnt_init(); #endif +#endif /* USE_THREADS */ + + erts_init_sys_time_sup(); + +#ifdef USE_THREADS + #if CHLDWTHR || defined(ERTS_SMP) erts_mtx_init(&chld_stat_mtx, "child_status"); #endif @@ -569,7 +578,7 @@ erts_sys_pre_init(void) erts_cnd_init(&chld_stat_cnd); children_alive = 0; #endif - } + #ifdef ERTS_SMP erts_smp_atomic32_init_nob(&erts_break_requested, 0); erts_smp_atomic32_init_nob(&erts_got_sigusr1, 0); @@ -582,7 +591,9 @@ erts_sys_pre_init(void) #if !CHLDWTHR && !defined(ERTS_SMP) children_died = 0; #endif + #endif /* USE_THREADS */ + erts_smp_atomic_init_nob(&sys_misc_mem_sz, 0); { @@ -1003,25 +1014,6 @@ static void unblock_signals(void) } -/************************** Time stuff **************************/ -#ifdef HAVE_GETHRTIME -#ifdef GETHRTIME_WITH_CLOCK_GETTIME - -SysHrTime sys_gethrtime(void) -{ - struct timespec ts; - long long result; - if (clock_gettime(CLOCK_MONOTONIC,&ts) != 0) { - erl_exit(1,"Fatal, could not get clock_monotonic value!, " - "errno = %d\n", errno); - } - result = ((long long) ts.tv_sec) * 1000000000LL + - ((long long) ts.tv_nsec); - return (SysHrTime) result; -} -#endif -#endif - /************************** OS info *******************************/ /* Used by erlang:info/1. */ diff --git a/erts/emulator/sys/unix/sys_time.c b/erts/emulator/sys/unix/sys_time.c index fcce54a2c4..d535457977 100644 --- a/erts/emulator/sys/unix/sys_time.c +++ b/erts/emulator/sys/unix/sys_time.c @@ -33,6 +33,21 @@ #include "sys.h" #include "global.h" +#include "erl_os_monotonic_time_extender.h" + +#undef ERTS_HAVE_ERTS_OS_TIMES_IMPL__ +#undef ERTS_HAVE_ERTS_SYS_HRTIME_IMPL__ + +#if defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) \ + || defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) +# include <mach/clock.h> +# include <mach/mach.h> +# ifdef HAVE_CLOCK_GET_ATTRIBUTES +# define ERTS_HAVE_MACH_CLOCK_GETRES +static Sint64 +mach_clock_getres(clock_id_t clkid, char *clkid_str); +# endif +#endif #ifdef NO_SYSCONF # define TICKS_PER_SEC() HZ @@ -53,9 +68,23 @@ /******************* Routines for time measurement *********************/ -int erts_ticks_per_sec = 0; /* Will be SYS_CLK_TCK in erl_unix_sys.h */ -int erts_ticks_per_sec_wrap = 0; /* Will be SYS_CLK_TCK_WRAP */ -static int ticks_bsr = 0; /* Shift wrapped tick value this much to the right */ +#undef ERTS_SYS_TIME_INTERNAL_STATE_WRITE_FREQ__ +#undef ERTS_SYS_TIME_INTERNAL_STATE_READ_ONLY__ +#undef ERTS_SYS_TIME_INTERNAL_STATE_READ_MOSTLY__ + +#if defined(OS_MONOTONIC_TIME_USING_TIMES) + +static Uint32 +get_tick_count(void) +{ + struct tms unused; + return (Uint32) times(&unused); +} + +#define ERTS_SYS_TIME_INTERNAL_STATE_READ_ONLY__ +#define ERTS_SYS_TIME_INTERNAL_STATE_READ_MOSTLY__ + +#endif /* * init timers, chose a tick length, and return it. @@ -63,37 +92,713 @@ static int ticks_bsr = 0; /* Shift wrapped tick value this much to the right */ * does almost everything. Other platforms have to * emulate Unix in this sense. */ -int sys_init_time(void) + +ErtsSysTimeData__ erts_sys_time_data__ erts_align_attribute(ERTS_CACHE_LINE_SIZE); + +#if defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) + +#define ERTS_SYS_TIME_INTERNAL_STATE_WRITE_FREQ__ + +static ErtsMonotonicTime clock_gettime_monotonic_raw(void); +static ErtsMonotonicTime clock_gettime_monotonic_verified(void); +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) +static void clock_gettime_times_raw(ErtsMonotonicTime *, ErtsSystemTime *); +static void clock_gettime_times_verified(ErtsMonotonicTime *, ErtsSystemTime *); +#endif + +#endif /* defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) */ + +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_READ_ONLY__ +struct sys_time_internal_state_read_only__ { +#if defined(OS_MONOTONIC_TIME_USING_TIMES) + int times_shift; +#endif +}; +#endif + +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_READ_MOSTLY__ +struct sys_time_internal_state_read_mostly__ { +#if defined(OS_MONOTONIC_TIME_USING_TIMES) + ErtsOsMonotonicTimeExtendState os_mtime_xtnd; +#endif +}; +#endif + +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_WRITE_FREQ__ +struct sys_time_internal_state_write_freq__ { + erts_smp_mtx_t mtx; +#if defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) + ErtsMonotonicTime last_delivered; +#endif +}; +#endif + +#if defined(ERTS_SYS_TIME_INTERNAL_STATE_READ_ONLY__) \ + || defined(ERTS_SYS_TIME_INTERNAL_STATE_WRITE_FREQ__) +static struct { +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_READ_ONLY__ + union { + struct sys_time_internal_state_read_only__ o; + char align__[(((sizeof(struct sys_time_internal_state_read_only__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } r; +#endif +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_READ_MOSTLY__ + union { + struct sys_time_internal_state_read_mostly__ m; + char align__[(((sizeof(struct sys_time_internal_state_read_mostly__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } wr; +#endif +#ifdef ERTS_SYS_TIME_INTERNAL_STATE_WRITE_FREQ__ + union { + struct sys_time_internal_state_write_freq__ f; + char align__[(((sizeof(struct sys_time_internal_state_write_freq__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } w; +#endif +} internal_state erts_align_attribute(ERTS_CACHE_LINE_SIZE); +#endif + +void +sys_init_time(ErtsSysInitTimeResult *init_resp) { +#if !defined(ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT) + + init_resp->have_os_monotonic_time = 0; + +#else /* defined(ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT) */ + + int major, minor, build, vsn; + + init_resp->os_monotonic_time_info.resolution = (Uint64) 1000*1000*1000; +#if defined(HAVE_CLOCK_GETRES) && defined(MONOTONIC_CLOCK_ID) + { + struct timespec ts; + if (clock_getres(MONOTONIC_CLOCK_ID, &ts) == 0) { + if (ts.tv_sec == 0 && ts.tv_nsec != 0) + init_resp->os_monotonic_time_info.resolution /= ts.tv_nsec; + else if (ts.tv_sec >= 1) + init_resp->os_monotonic_time_info.resolution = 1; + } + } +#elif defined(ERTS_HAVE_MACH_CLOCK_GETRES) && defined(MONOTONIC_CLOCK_ID) + init_resp->os_monotonic_time_info.resolution + = mach_clock_getres(MONOTONIC_CLOCK_ID, MONOTONIC_CLOCK_ID_STR); +#endif + +#ifdef MONOTONIC_CLOCK_ID_STR + init_resp->os_monotonic_time_info.clock_id = MONOTONIC_CLOCK_ID_STR; +#else + init_resp->os_monotonic_time_info.clock_id = NULL; +#endif + + init_resp->os_monotonic_time_info.locked_use = 0; + +#if defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) + init_resp->os_monotonic_time_info.func = "clock_gettime"; +#elif defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) + init_resp->os_monotonic_time_info.func = "clock_get_time"; +#elif defined(OS_MONOTONIC_TIME_USING_GETHRTIME) + init_resp->os_monotonic_time_info.func = "gethrtime"; +#elif defined(OS_MONOTONIC_TIME_USING_TIMES) + init_resp->os_monotonic_time_info.func = "times"; +#else +# error Unknown erts_os_monotonic_time() implementation +#endif + + init_resp->have_os_monotonic_time = 1; + + os_version(&major, &minor, &build); + + vsn = ERTS_MK_VSN_INT(major, minor, build); + + +#if defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) + if (vsn >= ERTS_MK_VSN_INT(2, 6, 33)) { + erts_sys_time_data__.r.o.os_monotonic_time = + clock_gettime_monotonic_raw; +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + erts_sys_time_data__.r.o.os_times = + clock_gettime_times_raw; +#endif + } + else { + /* + * Linux versions prior to 2.6.33 have a + * known bug that sometimes cause monotonic + * time to take small steps backwards. + */ + erts_sys_time_data__.r.o.os_monotonic_time = + clock_gettime_monotonic_verified; +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + erts_sys_time_data__.r.o.os_times = + clock_gettime_times_verified; +#endif + erts_smp_mtx_init(&internal_state.w.f.mtx, + "os_monotonic_time"); + internal_state.w.f.last_delivered + = clock_gettime_monotonic_raw(); + init_resp->os_monotonic_time_info.locked_use = 1; + } +#else /* !(defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME)) */ + { + char flavor[1024]; + + os_flavor(flavor, sizeof(flavor)); + + if (sys_strcmp(flavor, "sunos") == 0) { + /* + * Don't trust hrtime on multi processors + * on SunOS prior to SunOS 5.8 + */ + if (vsn < ERTS_MK_VSN_INT(5, 8, 0)) { +#if defined(HAVE_SYSCONF) && defined(_SC_NPROCESSORS_CONF) + if (sysconf(_SC_NPROCESSORS_CONF) > 1) +#endif + init_resp->have_os_monotonic_time = 0; + } + } + } +#endif /* !(defined(__linux__) && defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME)) */ + +#endif /* defined(ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT) */ + +#ifdef ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT + init_resp->os_monotonic_time_unit = ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT; +#endif + init_resp->sys_clock_resolution = SYS_CLOCK_RESOLUTION; + /* - * This (erts_ticks_per_sec) is only for times() (CLK_TCK), - * the resolution is always one millisecond.. + * This (erts_sys_time_data__.r.o.ticks_per_sec) is only for + * times() (CLK_TCK), the resolution is always one millisecond.. */ - if ((erts_ticks_per_sec = TICKS_PER_SEC()) < 0) - erl_exit(1, "Can't get clock ticks/sec\n"); - if (erts_ticks_per_sec >= 1000) { - /* Workaround for beta linux kernels, need to be done in runtime - to make erlang run on both 2.4 and 2.5 kernels. In the future, - the kernel ticks might as - well be used as a high res timer instead, but that's for when the - majority uses kernels with HZ == 1024 */ - ticks_bsr = 3; - } else { - ticks_bsr = 0; + if ((erts_sys_time_data__.r.o.ticks_per_sec = TICKS_PER_SEC()) < 0) + erl_exit(ERTS_ABORT_EXIT, "Can't get clock ticks/sec\n"); + +#if defined(OS_MONOTONIC_TIME_USING_TIMES) +#if ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT +# error Time unit is supposed to be determined at runtime... +#endif + { + ErtsMonotonicTime resolution = erts_sys_time_data__.r.o.ticks_per_sec; + ErtsMonotonicTime time_unit = resolution; + int shift = 0; + + while (time_unit < 1000*1000) { + time_unit <<= 1; + shift++; + } + + init_resp->os_monotonic_time_info.resolution = resolution; + init_resp->os_monotonic_time_unit = time_unit; + init_resp->os_monotonic_time_info.extended = 1; + internal_state.r.o.times_shift = shift; + + erts_init_os_monotonic_time_extender(&internal_state.wr.m.os_mtime_xtnd, + get_tick_count, + (1 << 29) / resolution); } - erts_ticks_per_sec_wrap = (erts_ticks_per_sec >> ticks_bsr); - return SYS_CLOCK_RESOLUTION; +#endif /* defined(OS_MONOTONIC_TIME_USING_TIMES) */ + +#ifdef WALL_CLOCK_ID_STR + init_resp->os_system_time_info.clock_id = WALL_CLOCK_ID_STR; +#else + init_resp->os_system_time_info.clock_id = NULL; +#endif + + init_resp->os_system_time_info.locked_use = 0; + init_resp->os_system_time_info.resolution = (Uint64) 1000*1000*1000; +#if defined(HAVE_CLOCK_GETRES) && defined(WALL_CLOCK_ID) + { + struct timespec ts; + if (clock_getres(WALL_CLOCK_ID, &ts) == 0) { + if (ts.tv_sec == 0 && ts.tv_nsec != 0) + init_resp->os_system_time_info.resolution /= ts.tv_nsec; + else if (ts.tv_sec >= 1) + init_resp->os_system_time_info.resolution = 1; + } + } +#elif defined(ERTS_HAVE_MACH_CLOCK_GETRES) && defined(WALL_CLOCK_ID) + init_resp->os_system_time_info.resolution + = mach_clock_getres(WALL_CLOCK_ID, WALL_CLOCK_ID_STR); +#endif + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + init_resp->os_system_time_info.func = "clock_gettime"; +#elif defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) + init_resp->os_system_time_info.func = "clock_get_time"; +#elif defined(OS_SYSTEM_TIME_GETTIMEOFDAY) + init_resp->os_system_time_info.func = "gettimeofday"; + init_resp->os_system_time_info.resolution = 1000*1000; + init_resp->os_system_time_info.clock_id = NULL; +#else +# error Missing erts_os_system_time() implementation +#endif + +} + +void +erts_late_sys_init_time(void) +{ +#if defined(OS_MONOTONIC_TIME_USING_TIMES) + erts_late_init_os_monotonic_time_extender(&internal_state.wr.m.os_mtime_xtnd); +#endif } -clock_t sys_times_wrap(void) +static ERTS_INLINE ErtsSystemTime +adj_stime_time_unit(ErtsSystemTime stime, Uint32 res) { - SysTimes dummy; - clock_t result = (sys_times(&dummy) >> ticks_bsr); - return result; + if (res == ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT) + return stime; + if (res == (Uint32) 1000*1000*1000 + && ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT == 1000*1000) + return stime/1000; + if (res == (Uint32) 1000*1000 + && ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT == 1000*1000*1000) + return stime*1000; + return ((ErtsSystemTime) + erts_time_unit_conversion(stime, + (Uint32) res, + (Uint32) ERTS_MONOTONIC_TIME_UNIT)); } +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * POSIX clock_gettime() * +\* */ +#if defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) \ + || defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +static ERTS_INLINE ErtsMonotonicTime +timespec2montime(struct timespec *ts) +{ + ErtsMonotonicTime time; + time = (ErtsMonotonicTime) ts->tv_sec; + time *= (ErtsMonotonicTime) 1000*1000*1000; + time += (ErtsMonotonicTime) ts->tv_nsec; + return time; +} + +static ERTS_INLINE ErtsMonotonicTime +posix_clock_gettime(clockid_t id, char *name) +{ + struct timespec ts; + + if (clock_gettime(id, &ts) != 0) { + int err = errno; + char *errstr = err ? strerror(err) : "unknown"; + erl_exit(ERTS_ABORT_EXIT, + "clock_gettime(%s, _) failed: %s (%d)\n", + name, errstr, err); + } + return timespec2montime(&ts); +} + +#endif /* defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) \ + || defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +ErtsSystemTime +erts_os_system_time(void) +{ + ErtsSystemTime stime; + + stime = (ErtsSystemTime) posix_clock_gettime(WALL_CLOCK_ID, + WALL_CLOCK_ID_STR); + return adj_stime_time_unit(stime, (Uint32) 1000*1000*1000); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +#if defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +#define ERTS_HAVE_ERTS_OS_TIMES_IMPL__ + +static ERTS_INLINE void +posix_clock_gettime_times(ErtsMonotonicTime *mtimep, + ErtsSystemTime *stimep) +{ + struct timespec mts, sts; + int mres, sres, merr, serr; + + mres = clock_gettime(MONOTONIC_CLOCK_ID, &mts); + merr = errno; + sres = clock_gettime(WALL_CLOCK_ID, &sts); + serr = errno; + + if (mres != 0) { + char *errstr = merr ? strerror(merr) : "unknown"; + erl_exit(ERTS_ABORT_EXIT, + "clock_gettime(%s, _) failed: %s (%d)\n", + MONOTONIC_CLOCK_ID_STR, errstr, merr); + } + if (sres != 0) { + char *errstr = serr ? strerror(serr) : "unknown"; + erl_exit(ERTS_ABORT_EXIT, + "clock_gettime(%s, _) failed: %s (%d)\n", + WALL_CLOCK_ID_STR, errstr, serr); + } + + *mtimep = timespec2montime(&mts); + *stimep = (ErtsSystemTime) timespec2montime(&sts); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +#if defined(__linux__) + +static ErtsMonotonicTime clock_gettime_monotonic_verified(void) +{ + ErtsMonotonicTime mtime = posix_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); + + erts_smp_mtx_lock(&internal_state.w.f.mtx); + if (mtime < internal_state.w.f.last_delivered) + mtime = internal_state.w.f.last_delivered; + else + internal_state.w.f.last_delivered = mtime; + erts_smp_mtx_unlock(&internal_state.w.f.mtx); + + return mtime; +} + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +static void clock_gettime_times_verified(ErtsMonotonicTime *mtimep, + ErtsSystemTime *stimep) +{ + posix_clock_gettime_times(mtimep, stimep); + + erts_smp_mtx_lock(&internal_state.w.f.mtx); + if (*mtimep < internal_state.w.f.last_delivered) + *mtimep = internal_state.w.f.last_delivered; + else + internal_state.w.f.last_delivered = *mtimep; + erts_smp_mtx_unlock(&internal_state.w.f.mtx); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +static ErtsMonotonicTime clock_gettime_monotonic_raw(void) +{ + return posix_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); +} + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +static void clock_gettime_times_raw(ErtsMonotonicTime *mtimep, + ErtsSystemTime *stimep) +{ + posix_clock_gettime_times(mtimep, stimep); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +#else /* !defined(__linux__) */ + +ErtsMonotonicTime erts_os_monotonic_time(void) +{ + return posix_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); +} + +#if defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) + +void erts_os_times(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + posix_clock_gettime_times(mtimep, stimep); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_CLOCK_GETTIME) */ + +#endif /* !defined(__linux__) */ + +#define ERTS_HAVE_ERTS_SYS_HRTIME_IMPL__ + +ErtsSysHrTime +erts_sys_hrtime(void) +{ + return (ErtsSysHrTime) posix_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); +} + +#endif /* defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) */ + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * MACH clock_get_time() * +\* */ + +#if defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) \ + || defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) + +#ifdef ERTS_HAVE_MACH_CLOCK_GETRES + +static Sint64 +mach_clock_getres(clock_id_t clkid, char *clkid_str) +{ + mach_port_t task; + host_name_port_t host; + natural_t attr[1]; + kern_return_t kret; + clock_serv_t clk_srv; + mach_msg_type_number_t cnt; + + host = mach_host_self(); + kret = host_get_clock_service(host, clkid, &clk_srv); + if (kret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "host_get_clock_service(_, %s, _) failed\n", + clkid_str); + } + + cnt = sizeof(attr); + kret = clock_get_attributes(clk_srv, CLOCK_GET_TIME_RES, (clock_attr_t) attr, &cnt); + if (kret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "clock_get_attributes(%s, _) failed\n", + clkid_str); + } + task = mach_task_self(); + mach_port_deallocate(task, host); + mach_port_deallocate(task, clk_srv); + + return (Sint64) attr[0]; +} + +#endif /* ERTS_HAVE_MACH_CLOCK_GETRES */ + +static ERTS_INLINE Sint64 +mach_clock_gettime(clock_id_t clkid, char *clkid_str) +{ + Sint64 time; + mach_port_t task; + host_name_port_t host; + kern_return_t kret; + clock_serv_t clk_srv; + mach_timespec_t time_spec; + + host = mach_host_self(); + kret = host_get_clock_service(host, clkid, &clk_srv); + if (kret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "host_get_clock_service(_, %s, _) failed\n", + clkid_str); + } + errno = 0; + kret = clock_get_time(clk_srv, &time_spec); + if (kret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "clock_get_time(%s, _) failed\n", + clkid_str); + } + task = mach_task_self(); + mach_port_deallocate(task, host); + mach_port_deallocate(task, clk_srv); + + time = (Sint64) time_spec.tv_sec; + time *= (Sint64) 1000*1000*1000; + time += (Sint64) time_spec.tv_nsec; + return time; +} + +#endif /* defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) \ + || defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) */ + +#if defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) + +#define ERTS_HAVE_ERTS_OS_TIMES_IMPL__ + +ErtsSystemTime +erts_os_system_time(void) +{ + ErtsSystemTime stime; + stime = (ErtsSystemTime) mach_clock_gettime(WALL_CLOCK_ID, + WALL_CLOCK_ID_STR); + return adj_stime_time_unit(stime, (Uint32) 1000*1000*1000); +} + +#endif /* defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) */ + +#if defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) + +ErtsMonotonicTime +erts_os_monotonic_time(void) +{ + return (ErtsMonotonicTime) mach_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); +} + +#define ERTS_HAVE_ERTS_SYS_HRTIME_IMPL__ + +ErtsSysHrTime +erts_sys_hrtime(void) +{ + return (ErtsMonotonicTime) mach_clock_gettime(MONOTONIC_CLOCK_ID, + MONOTONIC_CLOCK_ID_STR); +} + +#if defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) + +#define ERTS_HAVE_ERTS_OS_TIMES_IMPL__ + +void +erts_os_times(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + ErtsMonotonicTime mtime; + ErtsSystemTime stime; + mach_port_t task; + host_name_port_t host; + kern_return_t mkret, skret; + clock_serv_t mclk_srv, sclk_srv; + mach_timespec_t mon_time_spec, sys_time_spec; + + host = mach_host_self(); + mkret = host_get_clock_service(host, MONOTONIC_CLOCK_ID, &mclk_srv); + skret = host_get_clock_service(host, WALL_CLOCK_ID, &sclk_srv); + if (mkret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "host_get_clock_service(_, %s, _) failed\n", + MONOTONIC_CLOCK_ID); + } + if (skret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "host_get_clock_service(_, %s, _) failed\n", + WALL_CLOCK_ID); + } + mkret = clock_get_time(mclk_srv, &mon_time_spec); + skret = clock_get_time(sclk_srv, &sys_time_spec); + if (mkret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "clock_get_time(%s, _) failed\n", + MONOTONIC_CLOCK_ID); + } + if (skret != KERN_SUCCESS) { + erl_exit(ERTS_ABORT_EXIT, + "clock_get_time(%s, _) failed\n", + WALL_CLOCK_ID); + } + task = mach_task_self(); + mach_port_deallocate(task, host); + mach_port_deallocate(task, mclk_srv); + mach_port_deallocate(task, sclk_srv); + + mtime = (ErtsMonotonicTime) mon_time_spec.tv_sec; + mtime *= (ErtsMonotonicTime) 1000*1000*1000; + mtime += (ErtsMonotonicTime) mon_time_spec.tv_nsec; + stime = (ErtsSystemTime) sys_time_spec.tv_sec; + stime *= (ErtsSystemTime) 1000*1000*1000; + stime += (ErtsSystemTime) sys_time_spec.tv_nsec; + *mtimep = mtime; + *stimep = stime; +} + +#endif /* defined(OS_SYSTEM_TIME_USING_MACH_CLOCK_GET_TIME) */ + +#endif /* defined(OS_MONOTONIC_TIME_USING_MACH_CLOCK_GET_TIME) */ + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Solaris gethrtime() - OS monotonic time * +\* */ + +#if defined(OS_MONOTONIC_TIME_USING_GETHRTIME) + +ErtsMonotonicTime erts_os_monotonic_time(void) +{ + return (ErtsMonotonicTime) gethrtime(); +} + +#define ERTS_HAVE_ERTS_SYS_HRTIME_IMPL__ + +ErtsSysHrTime +erts_sys_hrtime(void) +{ + return (ErtsSysHrTime) gethrtime(); +} + +#endif /* defined(OS_MONOTONIC_TIME_USING_GETHRTIME) */ + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * gettimeofday() - OS system time * +\* */ + +#if defined(OS_SYSTEM_TIME_GETTIMEOFDAY) + +ErtsSystemTime +erts_os_system_time(void) +{ + ErtsSystemTime stime; + struct timeval tv; + + if (gettimeofday(&tv, NULL) != 0) { + int err = errno; + char *errstr = err ? strerror(err) : "unknown"; + erl_exit(ERTS_ABORT_EXIT, + "gettimeofday(_, NULL) failed: %s (%d)\n", + errstr, err); + } + + stime = (ErtsSystemTime) tv.tv_sec; + stime *= (ErtsSystemTime) 1000*1000; + stime += (ErtsSystemTime) tv.tv_usec; + + return adj_stime_time_unit(stime, (Uint32) 1000*1000); +} + +#endif /* defined(OS_SYSTEM_TIME_GETTIMEOFDAY) */ + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * times() - OS monotonic time * +\* */ + +#if defined(OS_MONOTONIC_TIME_USING_TIMES) + +ErtsMonotonicTime +erts_os_monotonic_time(void) +{ + Uint32 ticks = get_tick_count(); + ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + return ERTS_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks) << internal_state.r.o.times_shift; +} + +#endif + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Fallbacks * +\* */ + +#ifndef ERTS_HAVE_ERTS_SYS_HRTIME_IMPL__ + +ErtsSysHrTime +erts_sys_hrtime(void) +{ + return (ErtsSysHrTime) ERTS_MONOTONIC_TO_NSEC(erts_os_system_time()); +} + +#endif + +#if !defined(ERTS_HAVE_ERTS_OS_TIMES_IMPL__) \ + && defined(ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT) + +void +erts_os_times(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + *mtimep = erts_os_monotonic_time(); + *stimep = erts_os_system_time(); +} + +#endif +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ #ifdef HAVE_GETHRVTIME_PROCFS_IOCTL diff --git a/erts/emulator/sys/win32/erl_poll.c b/erts/emulator/sys/win32/erl_poll.c index 972170d465..5a62b00a68 100644 --- a/erts/emulator/sys/win32/erl_poll.c +++ b/erts/emulator/sys/win32/erl_poll.c @@ -285,7 +285,7 @@ struct ErtsPollSet_ { #ifdef ERTS_SMP erts_smp_mtx_t mtx; #endif - erts_smp_atomic32_t timeout; + erts_atomic64_t timeout_time; }; #ifdef ERTS_SMP @@ -363,6 +363,26 @@ do { \ wait_standby(PS); \ } while(0) +static ERTS_INLINE void +init_timeout_time(ErtsPollSet ps) +{ + erts_atomic64_init_nob(&ps->timeout_time, + (erts_aint64_t) ERTS_MONOTONIC_TIME_MAX); +} + +static ERTS_INLINE void +set_timeout_time(ErtsPollSet ps, ErtsMonotonicTime time) +{ + erts_atomic64_set_relb(&ps->timeout_time, + (erts_aint64_t) time); +} + +static ERTS_INLINE ErtsMonotonicTime +get_timeout_time(ErtsPollSet ps) +{ + return (ErtsMonotonicTime) erts_atomic64_read_acqb(&ps->timeout_time); +} + #define ERTS_POLL_NOT_WOKEN ((erts_aint32_t) 0) #define ERTS_POLL_WOKEN_IO_READY ((erts_aint32_t) 1) #define ERTS_POLL_WOKEN_INTR ((erts_aint32_t) 2) @@ -422,15 +442,29 @@ wakeup_cause(ErtsPollSet ps) } static ERTS_INLINE DWORD -poll_wait_timeout(ErtsPollSet ps, SysTimeval *tvp) +poll_wait_timeout(ErtsPollSet ps, ErtsMonotonicTime timeout_time) { - time_t timeout = tvp->tv_sec * 1000 + tvp->tv_usec / 1000; + ErtsMonotonicTime current_time, diff_time, timeout; - if (timeout <= 0) { + if (timeout_time == ERTS_POLL_NO_TIMEOUT) { + no_timeout: + set_timeout_time(ps, ERTS_MONOTONIC_TIME_MIN); woke_up(ps); return (DWORD) 0; } + current_time = erts_get_monotonic_time(); + diff_time = timeout_time - current_time; + if (diff_time <= 0) + goto no_timeout; + + /* Round up to nearest milli second */ + timeout = (ERTS_MONOTONIC_TO_MSEC(diff_time - 1) + 1); + if (timeout > INT_MAX) + timeout = INT_MAX; /* Also prevents DWORD overflow */ + + set_timeout_time(ps, current_time + ERTS_MSEC_TO_MONOTONIC(timeout)); + ResetEvent(ps->event_io_ready); /* * Since we don't know the internals of ResetEvent() we issue @@ -442,10 +476,6 @@ poll_wait_timeout(ErtsPollSet ps, SysTimeval *tvp) if (erts_atomic32_read_nob(&ps->wakeup_state) != ERTS_POLL_NOT_WOKEN) return (DWORD) 0; - if (timeout > ((time_t) ERTS_AINT32_T_MAX)) - timeout = ERTS_AINT32_T_MAX; /* Also prevents DWORD overflow */ - - erts_smp_atomic32_set_relb(&ps->timeout, (erts_aint32_t) timeout); return (DWORD) timeout; } @@ -1012,12 +1042,12 @@ void erts_poll_interrupt(ErtsPollSet ps, int set /* bool */) void erts_poll_interrupt_timed(ErtsPollSet ps, int set /* bool */, - erts_short_time_t msec) + ErtsMonotonicTime timeout_time) { - HARDTRACEF(("In erts_poll_interrupt_timed(%d,%ld)",set,msec)); + HARDTRACEF(("In erts_poll_interrupt_timed(%d,%ld)",set,timeout_time)); if (!set) reset_interrupt(ps); - else if (erts_smp_atomic32_read_acqb(&ps->timeout) > (erts_aint32_t) msec) + else if (get_timeout_time(ps) > timeout_time) set_interrupt(ps); HARDTRACEF(("Out erts_poll_interrupt_timed")); } @@ -1092,7 +1122,7 @@ void erts_poll_controlv(ErtsPollSet ps, int erts_poll_wait(ErtsPollSet ps, ErtsPollResFd pr[], int *len, - SysTimeval *tvp) + ErtsMonotonicTime timeout_time) { int no_fds; DWORD timeout; @@ -1149,7 +1179,7 @@ int erts_poll_wait(ErtsPollSet ps, no_fds = ERTS_POLL_MAX_RES; #endif - timeout = poll_wait_timeout(ps, tvp); + timeout = poll_wait_timeout(ps, timeout_time); /*HARDDEBUGF(("timeout = %ld",(long) timeout));*/ @@ -1242,7 +1272,7 @@ int erts_poll_wait(ErtsPollSet ps, erts_mtx_unlock(&w->mtx); } done: - erts_smp_atomic32_set_nob(&ps->timeout, ERTS_AINT32_T_MAX); + set_timeout_time(ps, ERTS_MONOTONIC_TIME_MAX); *len = num; ERTS_POLLSET_UNLOCK(ps); HARDTRACEF(("Out erts_poll_wait")); @@ -1326,7 +1356,7 @@ ErtsPollSet erts_poll_create_pollset(void) #ifdef ERTS_SMP erts_smp_mtx_init(&ps->mtx, "pollset"); #endif - erts_smp_atomic32_init_nob(&ps->timeout, ERTS_AINT32_T_MAX); + init_timeout_time(ps); HARDTRACEF(("Out erts_poll_create_pollset")); return ps; diff --git a/erts/emulator/sys/win32/erl_win_sys.h b/erts/emulator/sys/win32/erl_win_sys.h index fde32c8684..5181d6b584 100644 --- a/erts/emulator/sys/win32/erl_win_sys.h +++ b/erts/emulator/sys/win32/erl_win_sys.h @@ -120,9 +120,6 @@ /* * For erl_time_sup */ -#define HAVE_GETHRTIME - -#define sys_init_hrtime() /* Nothing */ #define SYS_CLK_TCK 1000 #define SYS_CLOCK_RESOLUTION 1 @@ -164,18 +161,81 @@ typedef struct { #if defined (__GNUC__) typedef unsigned long long Uint64; typedef long long Sint64; - -typedef long long SysHrTime; +# ifdef ULLONG_MAX +# define ERTS_UINT64_MAX ULLONG_MAX +# endif +# ifdef LLONG_MAX +# define ERTS_SINT64_MAX LLONG_MAX +# endif +# ifdef LLONG_MIN +# define ERTS_SINT64_MIN LLONG_MIN +# endif + +typedef long long ErtsMonotonicTime; +typedef long long ErtsSysHrTime; #else typedef ULONGLONG Uint64; typedef LONGLONG Sint64; -typedef LONGLONG SysHrTime; +typedef LONGLONG ErtsMonotonicTime; +typedef LONGLONG ErtsSysHrTime; #endif -extern int sys_init_time(void); +typedef ErtsMonotonicTime ErtsSystemTime; + +ErtsSystemTime erts_os_system_time(void); + +#define ERTS_MONOTONIC_TIME_MIN (((ErtsMonotonicTime) 1) << 63) +#define ERTS_MONOTONIC_TIME_MAX (~ERTS_MONOTONIC_TIME_MIN) + +#define ERTS_HAVE_OS_MONOTONIC_TIME_SUPPORT 1 +#define ERTS_COMPILE_TIME_MONOTONIC_TIME_UNIT 0 + +struct erts_sys_time_read_only_data__ { + ErtsMonotonicTime (*os_monotonic_time)(void); + void (*os_times)(ErtsMonotonicTime *, ErtsSystemTime*); + ErtsSysHrTime (*sys_hrtime)(void); +}; + +typedef struct { + union { + struct erts_sys_time_read_only_data__ o; + char align__[(((sizeof(struct erts_sys_time_read_only_data__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } r; +} ErtsSysTimeData__; + +extern ErtsSysTimeData__ erts_sys_time_data__; + +ERTS_GLB_INLINE ErtsMonotonicTime erts_os_monotonic_time(void); +ERTS_GLB_INLINE void erts_os_times(ErtsMonotonicTime *, + ErtsSystemTime *); +ERTS_GLB_INLINE ErtsSysHrTime erts_sys_hrtime(void); + +#if ERTS_GLB_INLINE_INCL_FUNC_DEF + +ERTS_GLB_INLINE ErtsMonotonicTime +erts_os_monotonic_time(void) +{ + return (*erts_sys_time_data__.r.o.os_monotonic_time)(); +} + +ERTS_GLB_INLINE void +erts_os_times(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + return (*erts_sys_time_data__.r.o.os_times)(mtimep, stimep); +} + +ERTS_GLB_INLINE ErtsSysHrTime +erts_sys_hrtime(void) +{ + return (*erts_sys_time_data__.r.o.sys_hrtime)(); +} + +#endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ + extern void sys_gettimeofday(SysTimeval *tv); -extern SysHrTime sys_gethrtime(void); extern clock_t sys_times(SysTimes *buffer); extern char *win_build_environment(char *); diff --git a/erts/emulator/sys/win32/sys.c b/erts/emulator/sys/win32/sys.c index 5d51659b4e..cf587af4ac 100644 --- a/erts/emulator/sys/win32/sys.c +++ b/erts/emulator/sys/win32/sys.c @@ -3172,25 +3172,31 @@ thr_create_prepare_child(void *vtcdp) void erts_sys_pre_init(void) { +#ifdef USE_THREADS + erts_thr_init_data_t eid = ERTS_THR_INIT_DATA_DEF_INITER; +#endif int_os_version.dwOSVersionInfoSize = sizeof(OSVERSIONINFO); GetVersionEx(&int_os_version); check_supported_os_version(); + #ifdef USE_THREADS - { - erts_thr_init_data_t eid = ERTS_THR_INIT_DATA_DEF_INITER; + eid.thread_create_child_func = thr_create_prepare_child; + /* Before creation in parent */ + eid.thread_create_prepare_func = thr_create_prepare; + /* After creation in parent */ + eid.thread_create_parent_func = thr_create_cleanup; - eid.thread_create_child_func = thr_create_prepare_child; - /* Before creation in parent */ - eid.thread_create_prepare_func = thr_create_prepare; - /* After creation in parent */ - eid.thread_create_parent_func = thr_create_cleanup, + erts_thr_init(&eid); +#endif - erts_thr_init(&eid); + erts_init_sys_time_sup(); + +#ifdef USE_THREADS #ifdef ERTS_ENABLE_LOCK_COUNT - erts_lcnt_init(); + erts_lcnt_init(); #endif - } #endif + erts_smp_atomic_init_nob(&sys_misc_mem_sz, 0); } @@ -3273,9 +3279,9 @@ erts_sys_schedule_interrupt(int set) #ifdef ERTS_SMP void -erts_sys_schedule_interrupt_timed(int set, erts_short_time_t msec) +erts_sys_schedule_interrupt_timed(int set, ErtsMonotonicTime timeout_time) { - erts_check_io_interrupt_timed(set, msec); + erts_check_io_interrupt_timed(set, timeout_time); } #endif diff --git a/erts/emulator/sys/win32/sys_time.c b/erts/emulator/sys/win32/sys_time.c index b84c8f85ce..b292d9279e 100644 --- a/erts/emulator/sys/win32/sys_time.c +++ b/erts/emulator/sys/win32/sys_time.c @@ -25,6 +25,8 @@ #endif #include "sys.h" #include "assert.h" +#include "erl_os_monotonic_time_extender.h" +#include "erl_time.h" #define LL_LITERAL(X) ERTS_I64_LITERAL(X) @@ -61,11 +63,6 @@ (epoch) = ((ull.QuadPart / TICKS_PER_SECOND) - EPOCH_JULIAN_DIFF); \ } while(0) -static SysHrTime wrap = 0; -static DWORD last_tick_count = 0; -static erts_smp_mtx_t wrap_lock; -static ULONGLONG (WINAPI *pGetTickCount64)(void) = NULL; - /* Getting timezone information is a heavy operation, so we want to do this only once */ @@ -76,27 +73,343 @@ static int days_in_month[2][13] = { {0,31,28,31,30,31,30,31,31,30,31,30,31}, {0,31,29,31,30,31,30,31,31,30,31,30,31}}; -int -sys_init_time(void) +#define ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT 10 + +/* + * erts_os_monotonic_time() + */ + +struct sys_time_internal_state_read_only__ { + ULONGLONG (WINAPI *pGetTickCount64)(void); + BOOL (WINAPI *pQueryPerformanceCounter)(LARGE_INTEGER *); + Sint32 pcf; + int using_get_tick_count_time_unit; +}; + +struct sys_time_internal_state_read_mostly__ { + ErtsOsMonotonicTimeExtendState os_mtime_xtnd; +}; + +struct sys_time_internal_state_write_freq__ { + erts_smp_mtx_t mtime_mtx; + ULONGLONG wrap; + ULONGLONG last_tick_count; +}; + +__declspec(align(ASSUMED_CACHE_LINE_SIZE)) struct { + union { + struct sys_time_internal_state_read_only__ o; + char align__[(((sizeof(struct sys_time_internal_state_read_only__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } r; + union { + struct sys_time_internal_state_read_mostly__ m; + char align__[(((sizeof(struct sys_time_internal_state_read_mostly__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } wr; + union { + struct sys_time_internal_state_write_freq__ f; + char align__[(((sizeof(struct sys_time_internal_state_write_freq__) - 1) + / ASSUMED_CACHE_LINE_SIZE) + 1) + * ASSUMED_CACHE_LINE_SIZE]; + } w; +} internal_state; + +__declspec(align(ASSUMED_CACHE_LINE_SIZE)) ErtsSysTimeData__ erts_sys_time_data__; + + +static ERTS_INLINE ErtsSystemTime +SystemTime2MilliSec(SYSTEMTIME *stp) +{ + ErtsSystemTime stime; + FILETIME ft; + ULARGE_INTEGER ull; + + SystemTimeToFileTime(stp, &ft); + FILETIME_TO_ULI(ull,ft); + /* now in 100 ns units */ + stime = (ErtsSystemTime) ull.QuadPart; + stime -= (((ErtsSystemTime) EPOCH_JULIAN_DIFF) + * ((ErtsSystemTime) (10*1000*1000))); + stime /= (ErtsSystemTime) (10*1000); /* ms */ + return stime; +} + +static ErtsMonotonicTime +os_monotonic_time_qpc(void) +{ + LARGE_INTEGER pc; + + if (!(*internal_state.r.o.pQueryPerformanceCounter)(&pc)) + erl_exit(ERTS_ABORT_EXIT, "QueryPerformanceCounter() failed\n"); + + return (ErtsMonotonicTime) pc.QuadPart; +} + +static void +os_times_qpc(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + LARGE_INTEGER pc; + SYSTEMTIME st; + ErtsSystemTime stime; + BOOL qpcr; + + qpcr = (*internal_state.r.o.pQueryPerformanceCounter)(&pc); + GetSystemTime(&st); + + if (!qpcr) + erl_exit(ERTS_ABORT_EXIT, "QueryPerformanceCounter() failed\n"); + + *mtimep = (ErtsMonotonicTime) pc.QuadPart; + + stime = SystemTime2MilliSec(&st); + + *stimep = ((ErtsSystemTime) + erts_time_unit_conversion((Uint64) stime, + (Uint32) 1000, + internal_state.r.o.pcf)); +} + +static Uint32 +get_tick_count(void) +{ + return (Uint32) GetTickCount(); +} + +static ErtsMonotonicTime +os_monotonic_time_gtc32(void) +{ + ErtsMonotonicTime mtime; + Uint32 ticks = (Uint32) GetTickCount(); + ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + mtime = ERTS_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + mtime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + return mtime; +} + +static void +os_times_gtc32(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + SYSTEMTIME st; + ErtsSystemTime stime, mtime; + Uint32 ticks; + + ticks = (Uint32) GetTickCount(); + GetSystemTime(&st); + + ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + mtime = ERTS_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + mtime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + *mtimep = mtime; + + stime = SystemTime2MilliSec(&st); + stime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + *stimep = stime; + +} + +static ErtsMonotonicTime +os_monotonic_time_gtc64(void) +{ + ULONGLONG ticks = (*internal_state.r.o.pGetTickCount64)(); + ErtsMonotonicTime mtime = (ErtsMonotonicTime) ticks; + return mtime << ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; +} + +static void +os_times_gtc64(ErtsMonotonicTime *mtimep, ErtsSystemTime *stimep) +{ + SYSTEMTIME st; + ErtsSystemTime stime, mtime; + ULONGLONG ticks; + + ticks = (*internal_state.r.o.pGetTickCount64)(); + GetSystemTime(&st); + + mtime = (ErtsMonotonicTime) ticks; + mtime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + *mtimep = mtime; + + stime = SystemTime2MilliSec(&st); + stime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + *stimep = stime; +} + +static ErtsSysHrTime +sys_hrtime_qpc(void) +{ + LARGE_INTEGER pc; + + if (!(*internal_state.r.o.pQueryPerformanceCounter)(&pc)) + erl_exit(ERTS_ABORT_EXIT, "QueryPerformanceCounter() failed\n"); + + ASSERT(pc.QuadPart > 0); + + return (ErtsSysHrTime) erts_time_unit_conversion((Uint64) pc.QuadPart, + internal_state.r.o.pcf, + (Uint32) 1000*1000*1000); +} + +static ErtsSysHrTime +sys_hrtime_gtc32(void) +{ + ErtsSysHrTime time; + Uint32 ticks = (Uint32) GetTickCount(); + ERTS_CHK_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + tick_count); + time = (ErtsSysHrTime) ERTS_EXTEND_OS_MONOTONIC_TIME(&internal_state.wr.m.os_mtime_xtnd, + ticks); + time *= (ErtsSysHrTime) (1000 * 1000); + return time; +} + +static ErtsSysHrTime +sys_hrtime_gtc64(void) +{ + ErtsSysHrTime time = (*internal_state.r.o.pGetTickCount64)(); + time *= (ErtsSysHrTime) (1000*1000); + return time; +} + +/* + * Init + */ + +void +sys_init_time(ErtsSysInitTimeResult *init_resp) { + ErtsMonotonicTime (*os_mtime_func)(void); + void (*os_times_func)(ErtsMonotonicTime *, ErtsSystemTime *); + ErtsSysHrTime (*sys_hrtime_func)(void) = NULL; + ErtsMonotonicTime time_unit; char kernel_dll_name[] = "kernel32"; HMODULE module; + init_resp->os_monotonic_time_info.clock_id = NULL; + module = GetModuleHandle(kernel_dll_name); - pGetTickCount64 = (module != NULL) ? - (ULONGLONG (WINAPI *)(void)) - GetProcAddress(module,"GetTickCount64") : - NULL; + if (!module) { + get_tick_count: + erts_smp_mtx_init(&internal_state.w.f.mtime_mtx, + "os_monotonic_time"); + internal_state.w.f.wrap = 0; + internal_state.w.f.last_tick_count = 0; + + init_resp->os_monotonic_time_info.func = "GetTickCount"; + init_resp->os_monotonic_time_info.locked_use = 0; + /* 10-16 ms resolution according to MicroSoft documentation */ + init_resp->os_monotonic_time_info.resolution = 100; /* 10 ms */ + time_unit = (ErtsMonotonicTime) 1000; + time_unit <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + internal_state.r.o.using_get_tick_count_time_unit = 1; + os_mtime_func = os_monotonic_time_gtc32; + os_times_func = os_times_gtc32; + init_resp->os_monotonic_time_info.extended = 1; + erts_init_os_monotonic_time_extender(&internal_state.wr.m.os_mtime_xtnd, + get_tick_count, + 60*60*24*7); /* Check once a week */ + if (!sys_hrtime_func) + sys_hrtime_func = sys_hrtime_gtc32; + } + else { + int major, minor, build; + + os_version(&major, &minor, &build); + + if (major < 6) { + + get_tick_count64: + + internal_state.r.o.pGetTickCount64 + = ((ULONGLONG (WINAPI *)(void)) + GetProcAddress(module, "GetTickCount64")); + if (!internal_state.r.o.pGetTickCount64) + goto get_tick_count; + + init_resp->os_monotonic_time_info.func = "GetTickCount64"; + init_resp->os_monotonic_time_info.locked_use = 0; + /* 10-16 ms resolution according to MicroSoft documentation */ + init_resp->os_monotonic_time_info.resolution = 100; /* 10 ms */ + time_unit = (ErtsMonotonicTime) 1000; + time_unit <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + internal_state.r.o.using_get_tick_count_time_unit = 1; + os_mtime_func = os_monotonic_time_gtc64; + os_times_func = os_times_gtc64; + if (!sys_hrtime_func) + sys_hrtime_func = sys_hrtime_gtc64; + } + else { /* Vista or newer... */ + + LARGE_INTEGER pf; + BOOL (WINAPI *QPF)(LARGE_INTEGER *); + + QPF = ((BOOL (WINAPI *)(LARGE_INTEGER *)) + GetProcAddress(module, "QueryPerformanceFrequency")); + if (!QPF) + goto get_tick_count64; + if (!(*QPF)(&pf)) + goto get_tick_count64; + + internal_state.r.o.pQueryPerformanceCounter + = ((BOOL (WINAPI *)(LARGE_INTEGER *)) + GetProcAddress(module, "QueryPerformanceCounter")); + if (!internal_state.r.o.pQueryPerformanceCounter) + goto get_tick_count64; + + if (pf.QuadPart < (((LONGLONG) 1) << 32)) { + internal_state.r.o.pcf = (Uint32) pf.QuadPart; + sys_hrtime_func = sys_hrtime_qpc; + } + + /* + * We only use QueryPerformanceCounter() for + * os-monotonic-time if its frequency is equal + * to, or larger than GHz in order to ensure + * that the user wont be able to observe faulty + * order between values retrieved on different threads. + */ + if (pf.QuadPart < (LONGLONG) 1000*1000*1000) + goto get_tick_count64; + + init_resp->os_monotonic_time_info.func = "QueryPerformanceCounter"; + init_resp->os_monotonic_time_info.locked_use = 0; + time_unit = (ErtsMonotonicTime) pf.QuadPart; + internal_state.r.o.using_get_tick_count_time_unit = 0; + init_resp->os_monotonic_time_info.resolution = time_unit; + os_mtime_func = os_monotonic_time_qpc; + os_times_func = os_times_qpc; + } + } + + erts_sys_time_data__.r.o.os_monotonic_time = os_mtime_func; + erts_sys_time_data__.r.o.os_times = os_times_func; + init_resp->os_monotonic_time_unit = time_unit; + init_resp->have_os_monotonic_time = 1; + init_resp->sys_clock_resolution = 1; + + init_resp->os_system_time_info.func = "GetSystemTime"; + init_resp->os_system_time_info.clock_id = NULL; + init_resp->os_system_time_info.resolution = 100; + init_resp->os_system_time_info.locked_use = 0; if(GetTimeZoneInformation(&static_tzi) && static_tzi.StandardDate.wMonth != 0 && static_tzi.DaylightDate.wMonth != 0) { have_static_tzi = 1; } +} - erts_smp_mtx_init(&wrap_lock, "sys_gethrtime"); - - return 1; +void +erts_late_sys_init_time(void) +{ + if (erts_sys_time_data__.r.o.os_monotonic_time == os_monotonic_time_gtc32) + erts_late_init_os_monotonic_time_extender(&internal_state.wr.m.os_mtime_xtnd); } /* Returns a switchtimes for DST as UTC filetimes given data from a @@ -377,41 +690,27 @@ sys_gettimeofday(SysTimeval *tv) EPOCH_JULIAN_DIFF); } -extern int erts_initialized; -SysHrTime -sys_gethrtime(void) +ErtsSystemTime +erts_os_system_time(void) { - if (pGetTickCount64 != NULL) { - return ((SysHrTime) pGetTickCount64()) * LL_LITERAL(1000000); - } else { - DWORD ticks; - SysHrTime res; - erts_smp_mtx_lock(&wrap_lock); - ticks = (SysHrTime) (GetTickCount() & 0x7FFFFFFF); - if (ticks < (SysHrTime) last_tick_count) { - /* Detect a race that should no longer be here... */ - if ((((SysHrTime) last_tick_count) - ((SysHrTime) ticks)) > 1000) { - wrap += LL_LITERAL(1) << 31; - } else { - /* - * XXX Debug: Violates locking order, remove all this, - * after testing! - */ - erts_dsprintf_buf_t *dsbufp = erts_create_logger_dsbuf(); - erts_dsprintf(dsbufp, "Did not wrap when last_tick %d " - "and tick %d", - last_tick_count, ticks); - erts_send_error_to_logger_nogl(dsbufp); - ticks = last_tick_count; - } - } - last_tick_count = ticks; - res = ((((LONGLONG) ticks) + wrap) * LL_LITERAL(1000000)); - erts_smp_mtx_unlock(&wrap_lock); - return res; + SYSTEMTIME st; + ErtsSystemTime stime; + + GetSystemTime(&st); + stime = SystemTime2MilliSec(&st); + + if (internal_state.r.o.using_get_tick_count_time_unit) { + stime <<= ERTS_GET_TICK_COUNT_TIME_UNIT_SHIFT; + return stime; } + + return ((ErtsSystemTime) + erts_time_unit_conversion((Uint64) stime, + (Uint32) 1000, + internal_state.r.o.pcf)); } + clock_t sys_times(SysTimes *buffer) { clock_t kernel_ticks = (GetTickCount() / diff --git a/erts/emulator/test/Makefile b/erts/emulator/test/Makefile index dfbe47786a..dd2e2cb504 100644 --- a/erts/emulator/test/Makefile +++ b/erts/emulator/test/Makefile @@ -108,6 +108,7 @@ MODULES= \ trace_call_time_SUITE \ scheduler_SUITE \ old_scheduler_SUITE \ + unique_SUITE \ z_SUITE \ old_mod \ long_timers_test \ diff --git a/erts/emulator/test/big_SUITE.erl b/erts/emulator/test/big_SUITE.erl index 413bd3bcae..3193d56e2a 100644 --- a/erts/emulator/test/big_SUITE.erl +++ b/erts/emulator/test/big_SUITE.erl @@ -23,7 +23,7 @@ init_per_group/2,end_per_group/2]). -export([t_div/1, eq_28/1, eq_32/1, eq_big/1, eq_math/1, big_literals/1, borders/1, negative/1, big_float_1/1, big_float_2/1, - shift_limit_1/1, powmod/1, system_limit/1, otp_6692/1]). + shift_limit_1/1, powmod/1, system_limit/1, toobig/1, otp_6692/1]). %% Internal exports. -export([eval/1]). @@ -40,7 +40,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [t_div, eq_28, eq_32, eq_big, eq_math, big_literals, borders, negative, {group, big_float}, shift_limit_1, - powmod, system_limit, otp_6692]. + powmod, system_limit, toobig, otp_6692]. groups() -> [{big_float, [], [big_float_1, big_float_2]}]. @@ -370,6 +370,16 @@ maxbig() -> id(I) -> I. +toobig(Config) when is_list(Config) -> + ?line {'EXIT',{{badmatch,_},_}} = (catch toobig()), + ok. + +toobig() -> + A = erlang:term_to_binary(lists:seq(1000000, 2200000)), + ASize = erlang:bit_size(A), + <<ANr:ASize>> = A, % should fail + ANr band ANr. + otp_6692(suite) -> []; otp_6692(doc) -> diff --git a/erts/emulator/test/long_timers_test.erl b/erts/emulator/test/long_timers_test.erl index 28a4fba9f6..f381332b51 100644 --- a/erts/emulator/test/long_timers_test.erl +++ b/erts/emulator/test/long_timers_test.erl @@ -28,7 +28,7 @@ -define(MAX_TIMEOUT, 60). % Minutes --define(MAX_LATE, 10*1000). % Milliseconds +-define(MAX_LATE_MS, 10*1000). % Milliseconds -define(REG_NAME, '___LONG___TIMERS___TEST___SERVER___'). -define(DRV_NAME, timer_driver). @@ -75,7 +75,7 @@ check_result() -> erlang:demonitor(Mon), receive {'DOWN', Mon, _, _, _} -> ok after 0 -> ok end, stop_node(Node), - check(TORs, (timer:now_diff(End, Start) div 1000) - ?MAX_LATE, ok) + check(TORs, ms((End - Start) - max_late()), ok) end. check([#timeout_rec{timeout = Timeout, @@ -83,7 +83,7 @@ check([#timeout_rec{timeout = Timeout, timeout_diff = undefined} | TORs], NeedRes, _Ok) when Timeout < NeedRes -> - io:format("~p timeout = ~p failed! No timeout.~n", + io:format("~p timeout = ~p ms failed! No timeout.~n", [Type, Timeout]), check(TORs, NeedRes, failed); check([#timeout_rec{timeout_diff = undefined} | TORs], @@ -95,7 +95,7 @@ check([#timeout_rec{timeout = Timeout, timeout_diff = {error, Reason}} | TORs], NeedRes, _Ok) -> - io:format("~p timeout = ~p failed! exit reason ~p~n", + io:format("~p timeout = ~p ms failed! exit reason ~p~n", [Type, Timeout, Reason]), check(TORs, NeedRes, failed); check([#timeout_rec{timeout = Timeout, @@ -103,43 +103,77 @@ check([#timeout_rec{timeout = Timeout, timeout_diff = TimeoutDiff} | TORs], NeedRes, Ok) -> - case (0 =< TimeoutDiff) and (TimeoutDiff =< ?MAX_LATE) of - true -> - io:format("~p timeout = ~p succeded! timeout diff = ~p.~n", - [Type, Timeout, TimeoutDiff]), - check(TORs, NeedRes, Ok); - false -> - io:format("~p timeout = ~p failed! timeout diff = ~p.~n", - [Type, Timeout, TimeoutDiff]), - check(TORs, NeedRes, failed) - end; + {NewOk, SuccessStr} = case ((0 =< TimeoutDiff) + andalso (TimeoutDiff =< max_late())) of + true -> {Ok, "succeeded"}; + false -> {failed, "FAILED"} + end, + io:format("~s timeout = ~s ms ~s! timeout diff = ~s.~n", + [type_str(Type), + time_str(Timeout), + SuccessStr, + time_str(TimeoutDiff, erlang:convert_time_unit(1, seconds, native))]), + check(TORs, NeedRes, NewOk); check([], _NeedRes, Ok) -> Ok. +type_str(receive_after) -> "receive ... after"; +type_str(bif_timer) -> "BIF timer"; +type_str(driver) -> "driver". + +time_str(Time, Unit) -> + lists:flatten([time_str(Time), " ", unit_str(Unit)]). + +time_str(Time) -> + lists:reverse(conv_time_str(lists:reverse(integer_to_list(Time)))). + +conv_time_str([X,Y,Z,C|Cs]) when C /= $- -> + [X,Y,Z,$`|conv_time_str([C|Cs])]; +conv_time_str(Cs) -> + Cs. + +unit_str(1) -> "s"; +unit_str(1000) -> "ms"; +unit_str(1000000) -> "us"; +unit_str(1000000000) -> "ns"; +unit_str(Res) when is_integer(Res) -> ["/ ", integer_to_list(Res), " s"]; +unit_str(Res) -> Res. + +to_diff(Timeout, Start, Stop) -> + %% 'Timeout' in milli seconds + %% 'Start', 'Stop', and result in native unit + (Stop - Start) - erlang:convert_time_unit(Timeout, milli_seconds, native). + +ms(Time) -> + erlang:convert_time_unit(Time, native, milli_seconds). + +max_late() -> + erlang:convert_time_unit(?MAX_LATE_MS, milli_seconds, native). + receive_after(Timeout) -> - Start = now(), + Start = erlang:monotonic_time(), receive {get_result, ?REG_NAME} -> ?REG_NAME ! #timeout_rec{pid = self(), type = receive_after, timeout = Timeout} after Timeout -> - Stop = now(), + Stop = erlang:monotonic_time(), receive {get_result, ?REG_NAME} -> - TimeoutDiff = ((timer:now_diff(Stop, Start) div 1000) - - Timeout), ?REG_NAME ! #timeout_rec{pid = self(), type = receive_after, timeout = Timeout, - timeout_diff = TimeoutDiff} + timeout_diff = to_diff(Timeout, + Start, + Stop)} end end. driver(Timeout) -> Port = open_port({spawn, ?DRV_NAME},[]), link(Port), - Start = now(), + Start = erlang:monotonic_time(), erlang:port_command(Port, <<?START_TIMER, Timeout:32>>), receive {get_result, ?REG_NAME} -> @@ -147,38 +181,38 @@ driver(Timeout) -> type = driver, timeout = Timeout}; {Port,{data,[?TIMER]}} -> - Stop = now(), + Stop = erlang:monotonic_time(), unlink(Port), true = erlang:port_close(Port), receive {get_result, ?REG_NAME} -> - TimeoutDiff = ((timer:now_diff(Stop, Start) div 1000) - - Timeout), ?REG_NAME ! #timeout_rec{pid = self(), type = driver, timeout = Timeout, - timeout_diff = TimeoutDiff} + timeout_diff = to_diff(Timeout, + Start, + Stop)} end end. bif_timer(Timeout) -> Tmr = erlang:start_timer(Timeout, self(), ok), - Start = now(), + Start = erlang:monotonic_time(), receive {get_result, ?REG_NAME} -> ?REG_NAME ! #timeout_rec{pid = self(), type = bif_timer, timeout = Timeout}; {timeout, Tmr, ok} -> - Stop = now(), + Stop = erlang:monotonic_time(), receive {get_result, ?REG_NAME} -> - TimeoutDiff = ((timer:now_diff(Stop, Start) div 1000) - - Timeout), ?REG_NAME ! #timeout_rec{pid = self(), type = bif_timer, timeout = Timeout, - timeout_diff = TimeoutDiff} + timeout_diff = to_diff(Timeout, + Start, + Stop)} end end. @@ -189,7 +223,7 @@ test(Starter, DrvDir, StartDone) -> register(?REG_NAME, self()), {group_leader, GL} = process_info(whereis(net_kernel),group_leader), group_leader(GL, self()), - Start = now(), + Start = erlang:monotonic_time(), TORs = lists:map(fun (Min) -> TO = Min*60*1000, [#timeout_rec{pid = spawn_opt( @@ -222,7 +256,7 @@ test(Starter, DrvDir, StartDone) -> test_loop(TORs, Start) -> receive {get_result, ?REG_NAME, Pid} -> - End = now(), + End = erlang:monotonic_time(), Pid ! {result, ?REG_NAME, get_test_results(TORs), Start, End}, erl_ddll:unload_driver(?DRV_NAME), erl_ddll:stop(), diff --git a/erts/emulator/test/map_SUITE.erl b/erts/emulator/test/map_SUITE.erl index 888ed8e272..228832ac0a 100644 --- a/erts/emulator/test/map_SUITE.erl +++ b/erts/emulator/test/map_SUITE.erl @@ -30,7 +30,7 @@ t_list_comprehension/1, t_map_sort_literals/1, t_map_equal/1, - %t_size/1, + t_map_compare/1, t_map_size/1, %% Specific Map BIFs @@ -51,6 +51,10 @@ t_erlang_hash/1, t_map_encode_decode/1, + %% non specific BIF related + t_bif_build_and_check/1, + t_bif_merge_and_check/1, + %% maps module not bifs t_maps_fold/1, t_maps_map/1, @@ -58,6 +62,8 @@ t_maps_without/1, %% misc + t_hashmap_balance/1, + t_erts_internal_order/1, t_pdict/1, t_ets/1, t_dets/1, @@ -66,6 +72,13 @@ -include_lib("stdlib/include/ms_transform.hrl"). +-define(CHECK(Cond,Term), + case (catch (Cond)) of + true -> true; + _ -> io:format("###### CHECK FAILED ######~nINPUT: ~p~n", [Term]), + exit(Term) + end). + suite() -> []. all() -> [ @@ -75,7 +88,7 @@ all() -> [ t_update_assoc,t_update_exact, t_guard_bifs, t_guard_sequence, t_guard_update, t_guard_receive,t_guard_fun, t_list_comprehension, - t_map_equal, + t_map_equal, t_map_compare, t_map_sort_literals, %% Specific Map BIFs @@ -90,12 +103,18 @@ all() -> [ t_erlang_hash, t_map_encode_decode, t_map_size, + %% non specific BIF related + t_bif_build_and_check, + t_bif_merge_and_check, + %% maps module t_maps_fold, t_maps_map, t_maps_size, t_maps_without, %% Other functions + t_hashmap_balance, + t_erts_internal_order, t_pdict, t_ets, t_tracing @@ -146,17 +165,6 @@ t_build_and_match_literals(Config) when is_list(Config) -> ok. -%% Tests size(Map). -%% not implemented, perhaps it shouldn't be either - -%t_size(Config) when is_list(Config) -> -% 0 = size(#{}), -% 1 = size(#{a=>1}), -% 1 = size(#{a=>#{a=>1}}), -% 2 = size(#{a=>1, b=>2}), -% 3 = size(#{a=>1, b=>2, b=>"3"}), -% ok. - t_map_size(Config) when is_list(Config) -> 0 = map_size(id(#{})), 1 = map_size(id(#{a=>1})), @@ -172,12 +180,23 @@ t_map_size(Config) when is_list(Config) -> true = map_is_size(M#{ "a" => 2}, 2), false = map_is_size(M#{ "c" => 2}, 2), + Ks = [build_key(fun(K) -> <<1,K:32,1>> end,I)||I<-lists:seq(1,100)], + ok = build_and_check_size(Ks,0,#{}), + %% Error cases. {'EXIT',{badarg,_}} = (catch map_size([])), {'EXIT',{badarg,_}} = (catch map_size(<<1,2,3>>)), {'EXIT',{badarg,_}} = (catch map_size(1)), ok. +build_and_check_size([K|Ks],N,M0) -> + N = map_size(M0), + M1 = M0#{ K => K }, + build_and_check_size(Ks,N + 1,M1); +build_and_check_size([],N,M) -> + N = map_size(M), + ok. + map_is_size(M,N) when map_size(M) =:= N -> true; map_is_size(_,_) -> false. @@ -432,7 +451,7 @@ t_map_sort_literals(Config) when is_list(Config) -> true = #{ c => 1, b => 1, a => 1 } < id(#{ b => 1, c => 1, d => 1}), true = #{ "a" => 1 } < id(#{ <<"a">> => 1}), false = #{ <<"a">> => 1 } < id(#{ "a" => 1}), - false = #{ 1 => 1 } < id(#{ 1.0 => 1}), + true = #{ 1 => 1 } < id(#{ 1.0 => 1}), false = #{ 1.0 => 1 } < id(#{ 1 => 1}), %% value order @@ -440,16 +459,47 @@ t_map_sort_literals(Config) when is_list(Config) -> false = #{ a => 2 } < id(#{ a => 1}), false = #{ a => 2, b => 1 } < id(#{ a => 1, b => 3}), true = #{ a => 1, b => 1 } < id(#{ a => 1, b => 3}), + false = #{ a => 1 } < id(#{ a => 1.0}), + false = #{ a => 1.0 } < id(#{ a => 1}), true = #{ "a" => "hi", b => 134 } == id(#{ b => 134,"a" => "hi"}), + %% large maps + + M = maps:from_list([{I,I}||I <- lists:seq(1,500)]), + + %% size order + true = M#{ a => 1, b => 2} < id(M#{ a => 1, b => 1, c => 1}), + true = M#{ b => 1, a => 1} < id(M#{ c => 1, a => 1, b => 1}), + false = M#{ c => 1, b => 1, a => 1} < id(M#{ c => 1, a => 1}), + + %% key order + true = M#{ a => 1 } < id(M#{ b => 1}), + false = M#{ b => 1 } < id(M#{ a => 1}), + true = M#{ a => 1, b => 1, c => 1 } < id(M#{ b => 1, c => 1, d => 1}), + true = M#{ b => 1, c => 1, d => 1 } > id(M#{ a => 1, b => 1, c => 1}), + true = M#{ c => 1, b => 1, a => 1 } < id(M#{ b => 1, c => 1, d => 1}), + true = M#{ "a" => 1 } < id(M#{ <<"a">> => 1}), + false = M#{ <<"a">> => 1 } < id(#{ "a" => 1}), + true = M#{ 1 => 1 } < id(maps:remove(1,M#{ 1.0 => 1})), + false = M#{ 1.0 => 1 } < id(M#{ 1 => 1}), + + %% value order + true = M#{ a => 1 } < id(M#{ a => 2}), + false = M#{ a => 2 } < id(M#{ a => 1}), + false = M#{ a => 2, b => 1 } < id(M#{ a => 1, b => 3}), + true = M#{ a => 1, b => 1 } < id(M#{ a => 1, b => 3}), + false = M#{ a => 1 } < id(M#{ a => 1.0}), + false = M#{ a => 1.0 } < id(M#{ a => 1}), + + true = M#{ "a" => "hi", b => 134 } == id(M#{ b => 134,"a" => "hi"}), + %% lists:sort SortVs = [#{"a"=>1},#{a=>2},#{1=>3},#{<<"a">>=>4}], [#{1:=ok},#{a:=ok},#{"a":=ok},#{<<"a">>:=ok}] = lists:sort([#{"a"=>ok},#{a=>ok},#{1=>ok},#{<<"a">>=>ok}]), [#{1:=3},#{a:=2},#{"a":=1},#{<<"a">>:=4}] = lists:sort(SortVs), [#{1:=3},#{a:=2},#{"a":=1},#{<<"a">>:=4}] = lists:sort(lists:reverse(SortVs)), - ok. t_map_equal(Config) when is_list(Config) -> @@ -469,27 +519,285 @@ t_map_equal(Config) when is_list(Config) -> true = id(#{ a => 1, b => 3, c => <<"wat">> }) =:= id(#{ a => 1, b => 3, c=><<"wat">>}), ok. + +t_map_compare(Config) when is_list(Config) -> + Seed = erlang:now(), + io:format("seed = ~p\n", [Seed]), + random:seed(Seed), + repeat(100, fun(_) -> float_int_compare() end, []), + repeat(100, fun(_) -> recursive_compare() end, []), + ok. + +float_int_compare() -> + Terms = numeric_keys(3), + %%io:format("Keys to use: ~p\n", [Terms]), + Pairs = lists:map(fun(K) -> list_to_tuple([{K,V} || V <- Terms]) end, Terms), + lists:foreach(fun(Size) -> + MapGen = fun() -> map_gen(list_to_tuple(Pairs), Size) end, + repeat(100, fun do_compare/1, [MapGen, MapGen]) + end, + lists:seq(1,length(Terms))), + ok. + +numeric_keys(N) -> + lists:foldl(fun(_,Acc) -> + Int = random:uniform(N*4) - N*2, + Float = float(Int), + [Int, Float, Float * 0.99, Float * 1.01 | Acc] + end, + [], + lists:seq(1,N)). + + +repeat(0, _, _) -> + ok; +repeat(N, Fun, Arg) -> + Fun(Arg), + repeat(N-1, Fun, Arg). + +copy_term(T) -> + Papa = self(), + P = spawn_link(fun() -> receive Msg -> Papa ! Msg end end), + P ! T, + receive R -> R end. + +do_compare([Gen1, Gen2]) -> + M1 = Gen1(), + M2 = Gen2(), + %%io:format("Maps to compare: ~p AND ~p\n", [M1, M2]), + C = (M1 < M2), + Erlang = maps_lessthan(M1, M2), + C = Erlang, + ?CHECK(M1==M1, M1), + + %% Change one key from int to float (or vice versa) and check compare + ML1 = maps:to_list(M1), + {K1,V1} = lists:nth(random:uniform(length(ML1)), ML1), + case K1 of + I when is_integer(I) -> + case maps:find(float(I),M1) of + error -> + M1f = maps:remove(I, maps:put(float(I), V1, M1)), + ?CHECK(M1f > M1, [M1f, M1]); + _ -> ok + end; + + F when is_float(F), round(F) == F -> + case maps:find(round(F),M1) of + error -> + M1i = maps:remove(F, maps:put(round(F), V1, M1)), + ?CHECK(M1i < M1, [M1i, M1]); + _ -> ok + end; + + _ -> ok % skip floats with decimals + end, + + ?CHECK(M2 == M2, [M2]). + + +maps_lessthan(M1, M2) -> + case {maps:size(M1),maps:size(M2)} of + {_S,_S} -> + {K1,V1} = lists:unzip(term_sort(maps:to_list(M1))), + {K2,V2} = lists:unzip(term_sort(maps:to_list(M2))), + + case erts_internal:cmp_term(K1,K2) of + -1 -> true; + 0 -> (V1 < V2); + 1 -> false + end; + + {S1, S2} -> + S1 < S2 + end. + +term_sort(L) -> + lists:sort(fun(A,B) -> erts_internal:cmp_term(A,B) =< 0 end, + L). + + +cmp(T1, T2, Exact) when is_tuple(T1) and is_tuple(T2) -> + case {size(T1),size(T2)} of + {_S,_S} -> cmp(tuple_to_list(T1), tuple_to_list(T2), Exact); + {S1,S2} when S1 < S2 -> -1; + {S1,S2} when S1 > S2 -> 1 + end; + +cmp([H1|T1], [H2|T2], Exact) -> + case cmp(H1,H2, Exact) of + 0 -> cmp(T1,T2, Exact); + C -> C + end; + +cmp(M1, M2, Exact) when is_map(M1) andalso is_map(M2) -> + cmp_maps(M1,M2,Exact); +cmp(M1, M2, Exact) -> + cmp_others(M1, M2, Exact). + +cmp_maps(M1, M2, Exact) -> + case {maps:size(M1),maps:size(M2)} of + {_S,_S} -> + {K1,V1} = lists:unzip(term_sort(maps:to_list(M1))), + {K2,V2} = lists:unzip(term_sort(maps:to_list(M2))), + + case cmp(K1, K2, true) of + 0 -> cmp(V1, V2, Exact); + C -> C + end; + + {S1,S2} when S1 < S2 -> -1; + {S1,S2} when S1 > S2 -> 1 + end. + +cmp_others(I, F, true) when is_integer(I), is_float(F) -> + -1; +cmp_others(F, I, true) when is_float(F), is_integer(I) -> + 1; +cmp_others(T1, T2, _) -> + case {T1<T2, T1==T2} of + {true,false} -> -1; + {false,true} -> 0; + {false,false} -> 1 + end. + +map_gen(Pairs, Size) -> + {_,L} = lists:foldl(fun(_, {Keys, Acc}) -> + KI = random:uniform(size(Keys)), + K = element(KI,Keys), + KV = element(random:uniform(size(K)), K), + {erlang:delete_element(KI,Keys), [KV | Acc]} + end, + {Pairs, []}, + lists:seq(1,Size)), + + maps:from_list(L). + + +recursive_compare() -> + Leafs = {atom, 17, 16.9, 17.1, [], self(), spawn(fun() -> ok end), make_ref(), make_ref()}, + {A, B} = term_gen_recursive(Leafs, 0, 0), + %%io:format("Recursive term A = ~p\n", [A]), + %%io:format("Recursive term B = ~p\n", [B]), + + ?CHECK({true,false} =:= case do_cmp(A, B, false) of + -1 -> {A<B, A>=B}; + 0 -> {A==B, A/=B}; + 1 -> {A>B, A=<B} + end, + {A,B}), + A2 = copy_term(A), + ?CHECK(A == A2, {A,A2}), + ?CHECK(0 =:= cmp(A, A2, false), {A,A2}), + + B2 = copy_term(B), + ?CHECK(B == B2, {B,B2}), + ?CHECK(0 =:= cmp(B, B2, false), {B,B2}), + ok. + +do_cmp(A, B, Exact) -> + C = cmp(A, B, Exact), + C. + +%% Generate two terms {A,B} that may only differ +%% at float vs integer types. +term_gen_recursive(Leafs, Flags, Depth) -> + MaxDepth = 10, + Rnd = case {Flags, Depth} of + {_, MaxDepth} -> % Only leafs + random:uniform(size(Leafs)) + 3; + {0, 0} -> % Only containers + random:uniform(3); + {0,_} -> % Anything + random:uniform(size(Leafs)+3) + end, + case Rnd of + 1 -> % Make map + Size = random:uniform(size(Leafs)), + lists:foldl(fun(_, {Acc1,Acc2}) -> + {K1,K2} = term_gen_recursive(Leafs, Flags, + Depth+1), + {V1,V2} = term_gen_recursive(Leafs, Flags, Depth+1), + %%ok = check_keys(K1,K2, 0), + {maps:put(K1,V1, Acc1), maps:put(K2,V2, Acc2)} + end, + {maps:new(), maps:new()}, + lists:seq(1,Size)); + 2 -> % Make cons + {Car1,Car2} = term_gen_recursive(Leafs, Flags, Depth+1), + {Cdr1,Cdr2} = term_gen_recursive(Leafs, Flags, Depth+1), + {[Car1 | Cdr1], [Car2 | Cdr2]}; + 3 -> % Make tuple + Size = random:uniform(size(Leafs)), + L = lists:map(fun(_) -> term_gen_recursive(Leafs, Flags, Depth+1) end, + lists:seq(1,Size)), + {L1, L2} = lists:unzip(L), + {list_to_tuple(L1), list_to_tuple(L2)}; + + N -> % Make leaf + case element(N-3, Leafs) of + I when is_integer(I) -> + case random:uniform(4) of + 1 -> {I, float(I)}; + 2 -> {float(I), I}; + _ -> {I,I} + end; + T -> {T,T} + end + end. + +check_keys(K1, K2, _) when K1 =:= K2 -> + case erlang:phash3(K1) =:= erlang:phash3(K2) of + true -> ok; + false -> + io:format("Same keys with different hash values !!!\nK1 = ~p\nK2 = ~p\n", [K1,K2]), + error + end; +check_keys(K1, K2, 0) -> + case {erlang:phash3(K1), erlang:phash3(K2)} of + {H,H} -> check_keys(K1, K2, 1); + {_,_} -> ok + end; +check_keys(K1, K2, L) when L < 10 -> + case {erlang:phash3([L|K1]), erlang:phash3([L|K2])} of + {H,H} -> check_keys(K1, K2, L+1); + {_,_} -> ok + end; +check_keys(K1, K2, L) -> + io:format("Same hash value at level ~p !!!\nK1 = ~p\nK2 = ~p\n", [L,K1,K2]), + error. + %% BIFs t_bif_map_get(Config) when is_list(Config) -> - + %% small map 1 = maps:get(a, #{ a=> 1}), 2 = maps:get(b, #{ a=> 1, b => 2}), "hi" = maps:get("hello", #{ a=>1, "hello" => "hi"}), "tuple hi" = maps:get({1,1.0}, #{ a=>a, {1,1.0} => "tuple hi"}), - M = id(#{ k1=>"v1", <<"k2">> => <<"v3">> }), - "v4" = maps:get(<<"k2">>, M#{ <<"k2">> => "v4" }), + M0 = id(#{ k1=>"v1", <<"k2">> => <<"v3">> }), + "v4" = maps:get(<<"k2">>, M0#{<<"k2">> => "v4"}), + + %% large map + M1 = maps:from_list([{I,I}||I<-lists:seq(1,100)] ++ + [{a,1},{b,2},{"hello","hi"},{{1,1.0},"tuple hi"}, + {k1,"v1"},{<<"k2">>,"v3"}]), + 1 = maps:get(a, M1), + 2 = maps:get(b, M1), + "hi" = maps:get("hello", M1), + "tuple hi" = maps:get({1,1.0}, M1), + "v3" = maps:get(<<"k2">>, M1), %% error case {'EXIT',{badarg, [{maps,get,_,_}|_]}} = (catch maps:get(a,[])), {'EXIT',{badarg, [{maps,get,_,_}|_]}} = (catch maps:get(a,<<>>)), {'EXIT',{bad_key,[{maps,get,_,_}|_]}} = (catch maps:get({1,1}, #{{1,1.0} => "tuple"})), {'EXIT',{bad_key,[{maps,get,_,_}|_]}} = (catch maps:get(a,#{})), - {'EXIT',{bad_key,[{maps,get,_,_}|_]}} = (catch maps:get(a,#{ b=>1, c=>2})), + {'EXIT',{bad_key,[{maps,get,_,_}|_]}} = (catch maps:get(a,#{b=>1, c=>2})), ok. t_bif_map_find(Config) when is_list(Config) -> - + %% small map {ok, 1} = maps:find(a, #{ a=> 1}), {ok, 2} = maps:find(b, #{ a=> 1, b => 2}), {ok, "int"} = maps:find(1, #{ 1 => "int"}), @@ -498,8 +806,18 @@ t_bif_map_find(Config) when is_list(Config) -> {ok, "hi"} = maps:find("hello", #{ a=>1, "hello" => "hi"}), {ok, "tuple hi"} = maps:find({1,1.0}, #{ a=>a, {1,1.0} => "tuple hi"}), - M = id(#{ k1=>"v1", <<"k2">> => <<"v3">> }), - {ok, "v4"} = maps:find(<<"k2">>, M#{ <<"k2">> => "v4" }), + M0 = id(#{ k1=>"v1", <<"k2">> => <<"v3">> }), + {ok, "v4"} = maps:find(<<"k2">>, M0#{ <<"k2">> => "v4" }), + + %% large map + M1 = maps:from_list([{I,I}||I<-lists:seq(1,100)] ++ + [{a,1},{b,2},{"hello","hi"},{{1,1.0},"tuple hi"}, + {k1,"v1"},{<<"k2">>,"v3"}]), + {ok, 1} = maps:find(a, M1), + {ok, 2} = maps:find(b, M1), + {ok, "hi"} = maps:find("hello", M1), + {ok, "tuple hi"} = maps:find({1,1.0}, M1), + {ok, "v3"} = maps:find(<<"k2">>, M1), %% error case error = maps:find(a,#{}), @@ -508,7 +826,6 @@ t_bif_map_find(Config) when is_list(Config) -> error = maps:find(1, #{ 1.0 => "float"}), error = maps:find({1.0,1}, #{ a=>a, {1,1.0} => "tuple hi"}), % reverse types in tuple key - {'EXIT',{badarg,[{maps,find,_,_}|_]}} = (catch maps:find(a,id([]))), {'EXIT',{badarg,[{maps,find,_,_}|_]}} = (catch maps:find(a,id(<<>>))), ok. @@ -542,12 +859,12 @@ t_bif_map_is_key(Config) when is_list(Config) -> t_bif_map_keys(Config) when is_list(Config) -> [] = maps:keys(#{}), - [1,2,3,4,5] = maps:keys(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e}), - [1,2,3,4,5] = maps:keys(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c}), + [1,2,3,4,5] = lists:sort(maps:keys(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e})), + [1,2,3,4,5] = lists:sort(maps:keys(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c})), % values in key order: [4,int,"hi",<<"key">>] M1 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, 4 => number}, - [4,int,"hi",<<"key">>] = maps:keys(M1), + [4,int,"hi",<<"key">>] = lists:sort(maps:keys(M1)), %% error case {'EXIT',{badarg,[{maps,keys,_,_}|_]}} = (catch maps:keys(1 bsl 65 + 3)), @@ -596,33 +913,33 @@ t_bif_map_put(Config) when is_list(Config) -> M1 = #{ "hi" := "hello"} = maps:put("hi", "hello", #{}), - ["hi"] = maps:keys(M1), - ["hello"] = maps:values(M1), + true = is_members(["hi"],maps:keys(M1)), + true = is_members(["hello"],maps:values(M1)), M2 = #{ int := 3 } = maps:put(int, 3, M1), - [int,"hi"] = maps:keys(M2), - [3,"hello"] = maps:values(M2), + true = is_members([int,"hi"],maps:keys(M2)), + true = is_members([3,"hello"],maps:values(M2)), M3 = #{ <<"key">> := <<"value">> } = maps:put(<<"key">>, <<"value">>, M2), - [int,"hi",<<"key">>] = maps:keys(M3), - [3,"hello",<<"value">>] = maps:values(M3), + true = is_members([int,"hi",<<"key">>],maps:keys(M3)), + true = is_members([3,"hello",<<"value">>],maps:values(M3)), M4 = #{ 18446744073709551629 := wat } = maps:put(18446744073709551629, wat, M3), - [18446744073709551629,int,"hi",<<"key">>] = maps:keys(M4), - [wat,3,"hello",<<"value">>] = maps:values(M4), + true = is_members([18446744073709551629,int,"hi",<<"key">>],maps:keys(M4)), + true = is_members([wat,3,"hello",<<"value">>],maps:values(M4)), M0 = #{ 4 := number } = M5 = maps:put(4, number, M4), - [4,18446744073709551629,int,"hi",<<"key">>] = maps:keys(M5), - [number,wat,3,"hello",<<"value">>] = maps:values(M5), + true = is_members([4,18446744073709551629,int,"hi",<<"key">>],maps:keys(M5)), + true = is_members([number,wat,3,"hello",<<"value">>],maps:values(M5)), M6 = #{ <<"key">> := <<"other value">> } = maps:put(<<"key">>, <<"other value">>, M5), - [4,18446744073709551629,int,"hi",<<"key">>] = maps:keys(M6), - [number,wat,3,"hello",<<"other value">>] = maps:values(M6), + true = is_members([4,18446744073709551629,int,"hi",<<"key">>],maps:keys(M6)), + true = is_members([number,wat,3,"hello",<<"other value">>],maps:values(M6)), %% error case {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,1 bsl 65 + 3)), @@ -630,7 +947,15 @@ t_bif_map_put(Config) when is_list(Config) -> {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,atom)), {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,[])), {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,<<>>)), - ok. + ok. + +is_members(Ks,Ls) when length(Ks) =/= length(Ls) -> false; +is_members(Ks,Ls) -> is_members_do(Ks,Ls). + +is_members_do([],[]) -> true; +is_members_do([],_) -> false; +is_members_do([K|Ks],Ls) -> + is_members_do(Ks, lists:delete(K,Ls)). t_bif_map_remove(Config) when is_list(Config) -> 0 = erlang:map_size(maps:remove(some_key, #{})), @@ -639,20 +964,20 @@ t_bif_map_remove(Config) when is_list(Config) -> 4 => number, 18446744073709551629 => wat}, M1 = maps:remove("hi", M0), - [4,18446744073709551629,int,<<"key">>] = maps:keys(M1), - [number,wat,3,<<"value">>] = maps:values(M1), + true = is_members([4,18446744073709551629,int,<<"key">>],maps:keys(M1)), + true = is_members([number,wat,3,<<"value">>],maps:values(M1)), M2 = maps:remove(int, M1), - [4,18446744073709551629,<<"key">>] = maps:keys(M2), - [number,wat,<<"value">>] = maps:values(M2), + true = is_members([4,18446744073709551629,<<"key">>],maps:keys(M2)), + true = is_members([number,wat,<<"value">>],maps:values(M2)), M3 = maps:remove(<<"key">>, M2), - [4,18446744073709551629] = maps:keys(M3), - [number,wat] = maps:values(M3), + true = is_members([4,18446744073709551629],maps:keys(M3)), + true = is_members([number,wat],maps:values(M3)), M4 = maps:remove(18446744073709551629, M3), - [4] = maps:keys(M4), - [number] = maps:values(M4), + true = is_members([4],maps:keys(M4)), + true = is_members([number],maps:values(M4)), M5 = maps:remove(4, M4), [] = maps:keys(M5), @@ -702,15 +1027,15 @@ t_bif_map_update(Config) when is_list(Config) -> t_bif_map_values(Config) when is_list(Config) -> [] = maps:values(#{}), + [1] = maps:values(#{a=>1}), - [a,b,c,d,e] = maps:values(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e}), - [a,b,c,d,e] = maps:values(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c}), + true = is_members([a,b,c,d,e],maps:values(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e})), + true = is_members([a,b,c,d,e],maps:values(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c})), - % values in key order: [4,int,"hi",<<"key">>] M1 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, 4 => number}, M2 = M1#{ "hi" => "hello2", <<"key">> => <<"value2">> }, - [number,3,"hello2",<<"value2">>] = maps:values(M2), - [number,3,"hello",<<"value">>] = maps:values(M1), + true = is_members([number,3,"hello2",<<"value2">>],maps:values(M2)), + true = is_members([number,3,"hello",<<"value">>],maps:values(M1)), %% error case {'EXIT',{badarg,[{maps,values,_,_}|_]}} = (catch maps:values(1 bsl 65 + 3)), @@ -730,61 +1055,61 @@ t_erlang_hash(Config) when is_list(Config) -> t_bif_erlang_phash2() -> 39679005 = erlang:phash2(#{}), - 78942764 = erlang:phash2(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 }), - 37338230 = erlang:phash2(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} }), - 14363616 = erlang:phash2(#{ 1 => a }), - 51612236 = erlang:phash2(#{ a => 1 }), + 33667975 = erlang:phash2(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 }), % 78942764 + 95332690 = erlang:phash2(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} }), % 37338230 + 108954384 = erlang:phash2(#{ 1 => a }), % 14363616 + 59617982 = erlang:phash2(#{ a => 1 }), % 51612236 - 37468437 = erlang:phash2(#{{} => <<>>}), - 44049159 = erlang:phash2(#{<<>> => {}}), + 42770201 = erlang:phash2(#{{} => <<>>}), % 37468437 + 71687700 = erlang:phash2(#{<<>> => {}}), % 44049159 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 118679416 = erlang:phash2(M0), - 51612236 = erlang:phash2(M1), - 118679416 = erlang:phash2(M2), + 70249457 = erlang:phash2(M0), % 118679416 + 59617982 = erlang:phash2(M1), % 51612236 + 70249457 = erlang:phash2(M2), % 118679416 ok. t_bif_erlang_phash() -> Sz = 1 bsl 32, - 268440612 = erlang:phash(#{},Sz), - 1196461908 = erlang:phash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), - 3944426064 = erlang:phash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), - 1394238263 = erlang:phash(#{ 1 => a },Sz), - 4066388227 = erlang:phash(#{ a => 1 },Sz), + 1113425985 = erlang:phash(#{},Sz), % 268440612 + 1510068139 = erlang:phash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), % 1196461908 + 3182345590 = erlang:phash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), % 3944426064 + 2927531828 = erlang:phash(#{ 1 => a },Sz), % 1394238263 + 1670235874 = erlang:phash(#{ a => 1 },Sz), % 4066388227 - 1578050717 = erlang:phash(#{{} => <<>>},Sz), - 1578050717 = erlang:phash(#{<<>> => {}},Sz), % yep, broken + 3935089469 = erlang:phash(#{{} => <<>>},Sz), % 1578050717 + 71692856 = erlang:phash(#{<<>> => {}},Sz), % 1578050717 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 3590546636 = erlang:phash(M0,Sz), - 4066388227 = erlang:phash(M1,Sz), - 3590546636 = erlang:phash(M2,Sz), + 2620391445 = erlang:phash(M0,Sz), % 3590546636 + 1670235874 = erlang:phash(M1,Sz), % 4066388227 + 2620391445 = erlang:phash(M2,Sz), % 3590546636 ok. t_bif_erlang_hash() -> Sz = 1 bsl 27 - 1, - 5158 = erlang:hash(#{},Sz), - 71555838 = erlang:hash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), - 5497225 = erlang:hash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), - 126071654 = erlang:hash(#{ 1 => a },Sz), - 126426236 = erlang:hash(#{ a => 1 },Sz), + 39684169 = erlang:hash(#{},Sz), % 5158 + 33673142 = erlang:hash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), % 71555838 + 95337869 = erlang:hash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), % 5497225 + 108959561 = erlang:hash(#{ 1 => a },Sz), % 126071654 + 59623150 = erlang:hash(#{ a => 1 },Sz), % 126426236 - 101655720 = erlang:hash(#{{} => <<>>},Sz), - 101655720 = erlang:hash(#{<<>> => {}},Sz), % yep, broken + 42775386 = erlang:hash(#{{} => <<>>},Sz), % 101655720 + 71692856 = erlang:hash(#{<<>> => {}},Sz), % 101655720 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 38260486 = erlang:hash(M0,Sz), - 126426236 = erlang:hash(M1,Sz), - 38260486 = erlang:hash(M2,Sz), + 70254632 = erlang:hash(M0,Sz), % 38260486 + 59623150 = erlang:hash(M1,Sz), % 126426236 + 70254632 = erlang:hash(M2,Sz), % 38260486 ok. @@ -818,14 +1143,35 @@ t_map_encode_decode(Config) when is_list(Config) -> %% literally #{ "hi" => "value", a=>33, b=>55 } in the internal order #{ a:=33, b:=55, "hi" := "value"} = erlang:binary_to_term(<<131,116,0,0,0,3, - 107,0,2,104,105, % "hi" :: list() + 107,0,2,104,105, % "hi" :: list() 107,0,5,118,97,108,117,101, % "value" :: list() - 100,0,1,97, % a :: atom() - 97,33, % 33 :: integer() - 100,0,1,98, % b :: atom() - 97,55 % 55 :: integer() + 100,0,1,97, % a :: atom() + 97,33, % 33 :: integer() + 100,0,1,98, % b :: atom() + 97,55 % 55 :: integer() >>), + %% Maps of different sizes + lists:foldl(fun(Key, M0) -> + M1 = M0#{Key => Key}, + case Key rem 17 of + 0 -> + M1 = binary_to_term(term_to_binary(M1)); + _ -> + ok + end, + M1 + end, + #{}, + lists:seq(1,10000)), + + %% many maps in same binary + MapList = lists:foldl(fun(K, [M|_]=Acc) -> [M#{K => K} | Acc] end, + [#{}], + lists:seq(1,100)), + MapList = binary_to_term(term_to_binary(MapList)), + MapListR = lists:reverse(MapList), + MapListR = binary_to_term(term_to_binary(MapListR)), %% error cases %% template: <<131,116,0,0,0,2,100,0,1,97,100,0,1,98,97,1,97,1>> @@ -856,39 +1202,42 @@ t_map_encode_decode(Config) when is_list(Config) -> map_encode_decode_and_match([{K,V}|Pairs], EncodedPairs, M0) -> M1 = maps:put(K,V,M0), B0 = erlang:term_to_binary(M1), - Ls = lists:sort(fun(A,B) -> erts_internal:cmp_term(A,B) < 0 end, [{K, erlang:term_to_binary(K), erlang:term_to_binary(V)}|EncodedPairs]), - %% sort Ks and Vs according to term spec, then match it - KVbins = lists:foldr(fun({_,Kbin,Vbin}, Acc) -> [Kbin,Vbin | Acc] end, [], Ls), - ok = match_encoded_map(B0, length(Ls), KVbins), + Ls = [{erlang:term_to_binary(K), erlang:term_to_binary(V)}|EncodedPairs], + ok = match_encoded_map(B0, length(Ls), Ls), %% decode and match it M1 = erlang:binary_to_term(B0), map_encode_decode_and_match(Pairs,Ls,M1); map_encode_decode_and_match([],_,_) -> ok. match_encoded_map(<<131,116,Size:32,Encoded/binary>>,Size,Items) -> - match_encoded_map(Encoded,Items); + match_encoded_map_stripped_size(Encoded,Items,Items); match_encoded_map(_,_,_) -> no_match_size. -match_encoded_map(<<>>,[]) -> ok; -match_encoded_map(Bin,[<<131,Item/binary>>|Items]) -> - Size = erlang:byte_size(Item), - <<EncodedTerm:Size/binary, Bin1/binary>> = Bin, - EncodedTerm = Item, %% Asssert - match_encoded_map(Bin1,Items). +match_encoded_map_stripped_size(<<>>,_,_) -> ok; +match_encoded_map_stripped_size(B0,[{<<131,K/binary>>,<<131,V/binary>>}|Items],Ls) -> + Ksz = byte_size(K), + Vsz = byte_size(V), + case B0 of + <<K:Ksz/binary,V:Vsz/binary,B1/binary>> -> + match_encoded_map_stripped_size(B1,Ls,Ls); + _ -> + match_encoded_map_stripped_size(B0,Items,Ls) + end; +match_encoded_map_stripped_size(_,[],_) -> fail. t_bif_map_to_list(Config) when is_list(Config) -> [] = maps:to_list(#{}), - [{a,1},{b,2}] = maps:to_list(#{a=>1,b=>2}), - [{a,1},{b,2},{c,3}] = maps:to_list(#{c=>3,a=>1,b=>2}), - [{a,1},{b,2},{g,3}] = maps:to_list(#{g=>3,a=>1,b=>2}), - [{a,1},{b,2},{g,3},{"c",4}] = maps:to_list(#{g=>3,a=>1,b=>2,"c"=>4}), - [{3,v2},{hi,v4},{{hi,3},v5},{"hi",v3},{<<"hi">>,v1}] = maps:to_list(#{ - <<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5}), + [{a,1},{b,2}] = lists:sort(maps:to_list(#{a=>1,b=>2})), + [{a,1},{b,2},{c,3}] = lists:sort(maps:to_list(#{c=>3,a=>1,b=>2})), + [{a,1},{b,2},{g,3}] = lists:sort(maps:to_list(#{g=>3,a=>1,b=>2})), + [{a,1},{b,2},{g,3},{"c",4}] = lists:sort(maps:to_list(#{g=>3,a=>1,b=>2,"c"=>4})), + [{3,v2},{hi,v4},{{hi,3},v5},{"hi",v3},{<<"hi">>,v1}] = + lists:sort(maps:to_list(#{<<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5})), - [{3,v7},{hi,v9},{{hi,3},v10},{"hi",v8},{<<"hi">>,v6}] = maps:to_list(#{ - <<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5, - <<"hi">>=>v6,3=>v7,"hi"=>v8,hi=>v9,{hi,3}=>v10}), + [{3,v7},{hi,v9},{{hi,3},v10},{"hi",v8},{<<"hi">>,v6}] = + lists:sort(maps:to_list(#{<<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5, + <<"hi">>=>v6,3=>v7,"hi"=>v8,hi=>v9,{hi,3}=>v10})), %% error cases {'EXIT', {badarg,_}} = (catch maps:to_list(id(a))), @@ -901,7 +1250,7 @@ t_bif_map_from_list(Config) when is_list(Config) -> A = maps:from_list([]), 0 = erlang:map_size(A), - #{a:=1,b:=2} = maps:from_list([{a,1},{b,2}]), + #{a:=1,b:=2} = maps:from_list([{a,1},{b,2}]), #{c:=3,a:=1,b:=2} = maps:from_list([{a,1},{b,2},{c,3}]), #{g:=3,a:=1,b:=2} = maps:from_list([{a,1},{b,2},{g,3}]), @@ -923,6 +1272,136 @@ t_bif_map_from_list(Config) when is_list(Config) -> {'EXIT', {badarg,_}} = (catch maps:from_list(id(42))), ok. +t_bif_build_and_check(Config) when is_list(Config) -> + ok = check_build_and_remove(750,[ + fun(K) -> [K,K] end, + fun(K) -> [float(K),K] end, + fun(K) -> K end, + fun(K) -> {1,K} end, + fun(K) -> {K} end, + fun(K) -> [K|K] end, + fun(K) -> [K,1,2,3,4] end, + fun(K) -> {K,atom} end, + fun(K) -> float(K) end, + fun(K) -> integer_to_list(K) end, + fun(K) -> list_to_atom(integer_to_list(K)) end, + fun(K) -> [K,{K,[K,{K,[K]}]}] end, + fun(K) -> <<K:32>> end + ]), + + ok. + +check_build_and_remove(_,[]) -> ok; +check_build_and_remove(N,[F|Fs]) -> + {M,Ks} = build_and_check(N, maps:new(), F, []), + ok = remove_and_check(Ks,M), + check_build_and_remove(N,Fs). + +build_and_check(0, M0, _, Ks) -> {M0, Ks}; +build_and_check(N, M0, F, Ks) -> + K = build_key(F,N), + M1 = maps:put(K,K,M0), + ok = check_keys_exist([K|Ks], M1), + M2 = maps:update(K,v,M1), + v = maps:get(K,M2), + build_and_check(N-1,M1,F,[K|Ks]). + +remove_and_check([],_) -> ok; +remove_and_check([K|Ks], M0) -> + K = maps:get(K,M0), + true = maps:is_key(K,M0), + M1 = maps:remove(K,M0), + false = maps:is_key(K,M1), + true = maps:is_key(K,M0), + ok = check_keys_exist(Ks,M1), + error = maps:find(K,M1), + remove_and_check(Ks, M1). + +build_key(F,N) when N rem 3 =:= 0 -> F(N); +build_key(F,N) when N rem 3 =:= 1 -> K = F(N), {K,K}; +build_key(F,N) when N rem 3 =:= 2 -> K = F(N), [K,K]. + +check_keys_exist([], _) -> ok; +check_keys_exist([K|Ks],M) -> + true = maps:is_key(K,M), + check_keys_exist(Ks,M). + +t_bif_merge_and_check(Config) when is_list(Config) -> + %% simple disjunct ones + %% make sure all keys are unique + Kss = [[a,b,c,d], + [1,2,3,4], + [], + ["hi"], + [e], + [build_key(fun(K) -> {small,K} end, I) || I <- lists:seq(1,32)], + lists:seq(5, 28), + lists:seq(29, 59), + [build_key(fun(K) -> integer_to_list(K) end, I) || I <- lists:seq(2000,10000)], + [build_key(fun(K) -> <<K:32>> end, I) || I <- lists:seq(1,80)], + [build_key(fun(K) -> {<<K:32>>} end, I) || I <- lists:seq(100,1000)]], + + + KsMs = build_keys_map_pairs(Kss), + Cs = [{CKs1,CM1,CKs2,CM2} || {CKs1,CM1} <- KsMs, {CKs2,CM2} <- KsMs], + ok = merge_and_check_combo(Cs), + + %% overlapping ones + + KVs1 = [{a,1},{b,2},{c,3}], + KVs2 = [{b,3},{c,4},{d,5}], + KVs = [{I,I} || I <- lists:seq(1,32)], + KVs3 = KVs1 ++ KVs, + KVs4 = KVs2 ++ KVs, + + M1 = maps:from_list(KVs1), + M2 = maps:from_list(KVs2), + M3 = maps:from_list(KVs3), + M4 = maps:from_list(KVs4), + + M12 = maps:merge(M1,M2), + ok = check_key_values(KVs2 ++ [{a,1}], M12), + M21 = maps:merge(M2,M1), + ok = check_key_values(KVs1 ++ [{d,5}], M21), + + M34 = maps:merge(M3,M4), + ok = check_key_values(KVs4 ++ [{a,1}], M34), + M43 = maps:merge(M4,M3), + ok = check_key_values(KVs3 ++ [{d,5}], M43), + + M14 = maps:merge(M1,M4), + ok = check_key_values(KVs4 ++ [{a,1}], M14), + M41 = maps:merge(M4,M1), + ok = check_key_values(KVs1 ++ [{d,5}] ++ KVs, M41), + + ok. + +check_key_values([],_) -> ok; +check_key_values([{K,V}|KVs],M) -> + V = maps:get(K,M), + check_key_values(KVs,M). + +merge_and_check_combo([]) -> ok; +merge_and_check_combo([{Ks1,M1,Ks2,M2}|Cs]) -> + M12 = maps:merge(M1,M2), + ok = check_keys_exist(Ks1 ++ Ks2, M12), + M21 = maps:merge(M2,M1), + ok = check_keys_exist(Ks1 ++ Ks2, M21), + + true = M12 =:= M21, + M12 = M21, + + merge_and_check_combo(Cs). + +build_keys_map_pairs([]) -> []; +build_keys_map_pairs([Ks|Kss]) -> + M = maps:from_list(keys_to_pairs(Ks)), + ok = check_keys_exist(Ks, M), + [{Ks,M}|build_keys_map_pairs(Kss)]. + +keys_to_pairs(Ks) -> [{K,K} || K <- Ks]. + + %% Maps module, not BIFs t_maps_fold(_Config) -> Vs = lists:seq(1,100), @@ -960,6 +1439,116 @@ t_maps_without(_Config) -> %% MISC + +%% Verify that the the number of nodes in hashmaps +%% of different types and sizes does not deviate too +%% much from the theoretical model. +t_hashmap_balance(_Config) -> + io:format("Integer keys\n", []), + hashmap_balance(fun(I) -> I end), + io:format("Float keys\n", []), + hashmap_balance(fun(I) -> float(I) end), + io:format("String keys\n", []), + hashmap_balance(fun(I) -> integer_to_list(I) end), + io:format("Binary (big) keys\n", []), + hashmap_balance(fun(I) -> <<I:16/big>> end), + io:format("Binary (little) keys\n", []), + hashmap_balance(fun(I) -> <<I:16/little>> end), + io:format("Atom keys\n", []), + erts_debug:set_internal_state(available_internal_state, true), + hashmap_balance(fun(I) -> erts_debug:get_internal_state({atom,I}) end), + erts_debug:set_internal_state(available_internal_state, false), + + ok. + +hashmap_balance(KeyFun) -> + F = fun(I, {M0,Max0}) -> + Key = KeyFun(I), + M1 = M0#{Key => Key}, + Max1 = case erts_internal:map_type(M1) of + hashmap -> + Nodes = hashmap_nodes(M1), + Avg = maps:size(M1) * 0.4, + StdDev = math:sqrt(maps:size(M1)) / 3, + SD_diff = abs(Nodes - Avg) / StdDev, + %%io:format("~p keys: ~p nodes avg=~p SD_diff=~p\n", + %% [maps:size(M1), Nodes, Avg, SD_diff]), + {MaxDiff0, _} = Max0, + case {Nodes > Avg, SD_diff > MaxDiff0} of + {true, true} -> {SD_diff, M1}; + _ -> Max0 + end; + + flatmap -> Max0 + end, + {M1, Max1} + end, + + {_,{MaxDiff,MaxMap}} = lists:foldl(F, + {#{}, {0, 0}}, + lists:seq(1,10000)), + io:format("Max std dev diff ~p for map of size ~p (nodes=~p, flatsize=~p)\n", + [MaxDiff, maps:size(MaxMap), hashmap_nodes(MaxMap), erts_debug:flat_size(MaxMap)]), + + true = (MaxDiff < 6), % The probability of this line failing is about 0.000000001 + % for a uniform hash. I've set the probability this "high" for now + % to detect flaws in our make_internal_hash. + % Hard limit is 15 (see hashmap_over_estimated_heap_size). + ok. + +hashmap_nodes(M) -> + Info = erts_debug:map_info(M), + lists:foldl(fun(Tpl,Acc) -> + case element(1,Tpl) of + bitmaps -> Acc + element(2,Tpl); + arrays -> Acc + element(2,Tpl); + _ -> Acc + end + end, + 0, + Info). + +t_erts_internal_order(_Config) when is_list(_Config) -> + + -1 = erts_internal:cmp_term(1,2), + 1 = erts_internal:cmp_term(2,1), + 0 = erts_internal:cmp_term(2,2), + + + -1 = erts_internal:cmp_term(1,a), + 1 = erts_internal:cmp_term(a,1), + 0 = erts_internal:cmp_term(a,a), + + -1 = erts_internal:cmp_term(1,1.0), + 1 = erts_internal:cmp_term(1.0,1), + 0 = erts_internal:cmp_term(1.0,1.0), + + -1 = erts_internal:cmp_term(1,1 bsl 65), + 1 = erts_internal:cmp_term(1 bsl 65,1), + 0 = erts_internal:cmp_term(1 bsl 65, 1 bsl 65), + + -1 = erts_internal:cmp_term(1 bsl 65,float(1)), + 1 = erts_internal:cmp_term(float(1),1 bsl 65), + -1 = erts_internal:cmp_term(1,float(1 bsl 65)), + 1 = erts_internal:cmp_term(float(1 bsl 65),1), + 0 = erts_internal:cmp_term(float(1 bsl 65), float(1 bsl 65)), + + %% reported errors + -1 = erts_internal:cmp_term(0,2147483648), + 0 = erts_internal:cmp_term(2147483648,2147483648), + 1 = erts_internal:cmp_term(2147483648,0), + + M = #{0 => 0,2147483648 => 0}, + true = M =:= binary_to_term(term_to_binary(M)), + + F1 = fun(_, _) -> 0 end, + F2 = fun(_, _) -> 1 end, + M0 = maps:from_list( [{-2147483649, 0}, {0,0}, {97, 0}, {false, 0}, {flower, 0}, {F1, 0}, {F2, 0}, {<<>>, 0}]), + M1 = maps:merge(M0, #{0 => 1}), + 8 = maps:size(M1), + 1 = maps:get(0,M1), + ok. + t_pdict(_Config) -> put(#{ a => b, b => a},#{ c => d}), diff --git a/erts/emulator/test/match_spec_SUITE.erl b/erts/emulator/test/match_spec_SUITE.erl index fc4a5028e1..d3c884689f 100644 --- a/erts/emulator/test/match_spec_SUITE.erl +++ b/erts/emulator/test/match_spec_SUITE.erl @@ -924,8 +924,51 @@ maps(Config) when is_list(Config) -> table), {ok,#{foo := 3},[],[]} = erlang:match_spec_test({}, [{{},[],[#{foo => {'+',1,2}}]}], table), + + {ok,"camembert",[],[]} = + erlang:match_spec_test(#{b => "camembert",c => "cabécou"}, + [{#{b => '$1',c => "cabécou"},[],['$1']}], table), + + {ok,#{a :="camembert",b := "hi"},[],[]} = + erlang:match_spec_test(#{<<"b">> =>"camembert","c"=>"cabécou", "wat"=>"hi", b=><<"other">>}, + [{#{<<"b">> => '$1',"wat" => '$2'},[],[#{a=>'$1',b=>'$2'}]}], + table), + %% large maps + + Ls0 = [{I,<<I:32>>}||I <- lists:seq(1,415)], + M0 = maps:from_list(Ls0), + M1 = #{a=>1,b=>2,c=>3,d=>4}, + + R1 = M0#{263 := #{ a=> 3 }}, + Ms1 = [{M1#{c:='$1'},[],[M0#{263 := #{a => '$1'}}]}], + + {ok,R1,[],[]} = erlang:match_spec_test(M1,Ms1,table), + + Ms2 = [{M0#{63:='$1', 19:='$2'},[],[M0#{19:='$1', 63:='$2'}]}], + R2 = M0#{63 := maps:get(19,M0), 19 := maps:get(63,M0) }, + {ok,R2,[],[]} = erlang:match_spec_test(M0,Ms2,table), + + ok = maps_check_loop(M1), + ok = maps_check_loop(M0), + M2 = maps:from_list([{integer_to_list(K),V} || {K,V} <- Ls0]), + ok = maps_check_loop(M2), ok. +maps_check_loop(M) -> + Ks = maps:keys(M), + maps_check_loop(M,M,M,M,Ks,lists:reverse(Ks),1). + +maps_check_loop(Orig,M0,MsM0,Rm0,[K|Ks],[Rk|Rks],Ix) -> + MsK = list_to_atom([$$]++integer_to_list(Ix)), + MsM1 = MsM0#{K := MsK}, + Rm1 = Rm0#{Rk := MsK}, + M1 = M0#{Rk := maps:get(K,MsM0)}, + Ms = [{MsM1,[],[Rm1]}], + {ok,M1,[],[]} = erlang:match_spec_test(Orig,Ms,table), + maps_check_loop(Orig,M1,MsM1,Rm1,Ks,Rks,Ix+1); +maps_check_loop(_,_,_,_,[],[],_) -> ok. + + empty_list(Config) when is_list(Config) -> Val=[{'$1',[], [{message,'$1'},{message,{caller}},{return_trace}]}], %% Did crash debug VM in faulty assert: diff --git a/erts/emulator/test/monitor_SUITE.erl b/erts/emulator/test/monitor_SUITE.erl index aec59867d8..07e2862b2a 100644 --- a/erts/emulator/test/monitor_SUITE.erl +++ b/erts/emulator/test/monitor_SUITE.erl @@ -26,7 +26,8 @@ case_1/1, case_1a/1, case_2/1, case_2a/1, mon_e_1/1, demon_e_1/1, demon_1/1, demon_2/1, demon_3/1, demonitor_flush/1, local_remove_monitor/1, remote_remove_monitor/1, mon_1/1, mon_2/1, - large_exit/1, list_cleanup/1, mixer/1, named_down/1, otp_5827/1]). + large_exit/1, list_cleanup/1, mixer/1, named_down/1, otp_5827/1, + monitor_time_offset/1]). -export([init_per_testcase/2, end_per_testcase/2]). @@ -38,7 +39,8 @@ all() -> [case_1, case_1a, case_2, case_2a, mon_e_1, demon_e_1, demon_1, mon_1, mon_2, demon_2, demon_3, demonitor_flush, {group, remove_monitor}, large_exit, - list_cleanup, mixer, named_down, otp_5827]. + list_cleanup, mixer, named_down, otp_5827, + monitor_time_offset]. groups() -> [{remove_monitor, [], @@ -59,7 +61,7 @@ end_per_group(_GroupName, Config) -> init_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> Dog=?t:timetrap(?t:minutes(15)), - [{watchdog, Dog}|Config]. + [{watchdog, Dog},{testcase, Func}|Config]. end_per_testcase(_Func, Config) -> Dog=?config(watchdog, Config), @@ -837,6 +839,89 @@ otp_5827(Config) when is_list(Config) -> ?line ?t:fail("erlang:monitor/2 hangs") end. +monitor_time_offset(Config) when is_list(Config) -> + {ok, Node} = start_node(Config, "+C single_time_warp"), + Me = self(), + PMs = lists:map(fun (_) -> + Pid = spawn(Node, + fun () -> + check_monitor_time_offset(Me) + end), + {Pid, erlang:monitor(process, Pid)} + end, + lists:seq(1, 100)), + lists:foreach(fun ({P, _M}) -> + P ! check_no_change_message + end, PMs), + lists:foreach(fun ({P, M}) -> + receive + {no_change_message_received, P} -> + ok; + {'DOWN', M, process, P, Reason} -> + ?t:fail(Reason) + end + end, PMs), + preliminary = rpc:call(Node, erlang, system_flag, [time_offset, finalize]), + lists:foreach(fun ({P, M}) -> + receive + {change_messages_received, P} -> + erlang:demonitor(M, [flush]); + {'DOWN', M, process, P, Reason} -> + ?t:fail(Reason) + end + end, PMs), + stop_node(Node), + ok. + +check_monitor_time_offset(Leader) -> + Mon1 = erlang:monitor(time_offset, clock_service), + Mon2 = erlang:monitor(time_offset, clock_service), + Mon3 = erlang:monitor(time_offset, clock_service), + Mon4 = erlang:monitor(time_offset, clock_service), + + erlang:demonitor(Mon2, [flush]), + + Mon5 = erlang:monitor(time_offset, clock_service), + Mon6 = erlang:monitor(time_offset, clock_service), + Mon7 = erlang:monitor(time_offset, clock_service), + + receive check_no_change_message -> ok end, + receive + {'CHANGE', _, time_offset, clock_service, _} -> + exit(unexpected_change_message_received) + after 0 -> + Leader ! {no_change_message_received, self()} + end, + receive after 100 -> ok end, + erlang:demonitor(Mon4, [flush]), + receive + {'CHANGE', Mon3, time_offset, clock_service, _} -> + ok + end, + receive + {'CHANGE', Mon6, time_offset, clock_service, _} -> + ok + end, + erlang:demonitor(Mon5, [flush]), + receive + {'CHANGE', Mon7, time_offset, clock_service, _} -> + ok + end, + receive + {'CHANGE', Mon1, time_offset, clock_service, _} -> + ok + end, + receive + {'CHANGE', _, time_offset, clock_service, _} -> + exit(unexpected_change_message_received) + after 1000 -> + ok + end, + Leader ! {change_messages_received, self()}. + +%% +%% ... +%% wait_for_m(_,_,0) -> exit(monitor_wait_timeout); @@ -959,3 +1044,25 @@ generate(_Fun, 0) -> []; generate(Fun, N) -> [Fun() | generate(Fun, N-1)]. + +start_node(Config) -> + start_node(Config, ""). + +start_node(Config, Args) -> + TestCase = ?config(testcase, Config), + PA = filename:dirname(code:which(?MODULE)), + ESTime = erlang:monotonic_time(1) + erlang:time_offset(1), + Unique = erlang:unique_integer([positive]), + Name = list_to_atom(atom_to_list(?MODULE) + ++ "-" + ++ atom_to_list(TestCase) + ++ "-" + ++ integer_to_list(ESTime) + ++ "-" + ++ integer_to_list(Unique)), + test_server:start_node(Name, + slave, + [{args, "-pa " ++ PA ++ " " ++ Args}]). + +stop_node(Node) -> + test_server:stop_node(Node). diff --git a/erts/emulator/test/nif_SUITE.erl b/erts/emulator/test/nif_SUITE.erl index 4560077a51..b0624fb8c1 100644 --- a/erts/emulator/test/nif_SUITE.erl +++ b/erts/emulator/test/nif_SUITE.erl @@ -451,7 +451,7 @@ maps(Config) when is_list(Config) -> M = maps_from_list_nif(Pairs), R = {RIs,Is} = sorted_list_from_maps_nif(M), io:format("Pairs: ~p~nMap: ~p~nReturned: ~p~n", [lists:sort(Pairs),M,R]), - Is = lists:sort(Pairs), + true = (lists:sort(Is) =:= lists:sort(Pairs)), Is = lists:reverse(RIs), #{} = maps_from_list_nif([]), diff --git a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c index 85544db2ab..5a3be84825 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c +++ b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c @@ -1717,8 +1717,9 @@ static ERL_NIF_TERM sorted_list_from_maps_nif(ErlNifEnv* env, int argc, const ER return enif_make_int(env, __LINE__); cnt = 0; + next_ret = 1; while(enif_map_iterator_get_pair(env,&iter_f,&key,&value)) { - if (cnt && !next_ret) + if (!next_ret) return enif_make_int(env, __LINE__); list_f = enif_make_list_cell(env, enif_make_tuple2(env, key, value), list_f); next_ret = enif_map_iterator_next(env,&iter_f); @@ -1731,8 +1732,9 @@ static ERL_NIF_TERM sorted_list_from_maps_nif(ErlNifEnv* env, int argc, const ER return enif_make_int(env, __LINE__); cnt = 0; + prev_ret = 1; while(enif_map_iterator_get_pair(env,&iter_b,&key,&value)) { - if (cnt && !prev_ret) + if (!prev_ret) return enif_make_int(env, __LINE__); /* Test that iter_f can step "backwards" */ @@ -1744,6 +1746,7 @@ static ERL_NIF_TERM sorted_list_from_maps_nif(ErlNifEnv* env, int argc, const ER list_b = enif_make_list_cell(env, enif_make_tuple2(env, key, value), list_b); prev_ret = enif_map_iterator_prev(env,&iter_b); + cnt++; } if (cnt) { diff --git a/erts/emulator/test/node_container_SUITE.erl b/erts/emulator/test/node_container_SUITE.erl index 3f9b339ed2..9c1839811a 100644 --- a/erts/emulator/test/node_container_SUITE.erl +++ b/erts/emulator/test/node_container_SUITE.erl @@ -686,6 +686,9 @@ timer_refc(doc) -> "as they should for data stored in bif timers."]; timer_refc(suite) -> []; timer_refc(Config) when is_list(Config) -> + {skipped, "Test needs to be UPDATED for new timer implementation"}. + +timer_refc_test(Config) when is_list(Config) -> ?line RNode = {get_nodename(), 1}, ?line RPid = mk_pid(RNode, 4711, 2), ?line RPort = mk_port(RNode, 4711), diff --git a/erts/emulator/test/port_bif_SUITE.erl b/erts/emulator/test/port_bif_SUITE.erl index f439867e9c..0c5b09d45a 100644 --- a/erts/emulator/test/port_bif_SUITE.erl +++ b/erts/emulator/test/port_bif_SUITE.erl @@ -24,7 +24,7 @@ init_per_group/2,end_per_group/2, command/1, command_e_1/1, command_e_2/1, command_e_3/1, command_e_4/1, port_info1/1, port_info2/1, - port_info_os_pid/1, + port_info_os_pid/1, port_info_race/1, connect/1, control/1, echo_to_busy/1]). -export([do_command_e_1/1, do_command_e_2/1, do_command_e_4/1]). @@ -42,7 +42,8 @@ all() -> groups() -> [{command_e, [], [command_e_1, command_e_2, command_e_3, command_e_4]}, - {port_info, [], [port_info1, port_info2, port_info_os_pid]}]. + {port_info, [], + [port_info1, port_info2, port_info_os_pid, port_info_race]}]. init_per_suite(Config) -> Config. @@ -254,6 +255,28 @@ do_port_info_os_pid() -> true = erlang:port_close(P), ok. +port_info_race(Config) when is_list(Config) -> + DataDir = ?config(data_dir, Config), + Program = filename:join(DataDir, "port_test"), + Top = self(), + P1 = open_port({spawn,Program}, [{packet,1}]), + P2 = open_port({spawn,Program}, [{packet,1}]), + Info1 = erlang:port_info(P1), + Info2 = erlang:port_info(P2), + F = fun Loop(Port, _, 0) -> + Top ! {ok,Port}; + Loop(Port, Info, N) -> + Info = erlang:port_info(Port), + Loop(Port, Info, N - 1) + end, + spawn_link(fun () -> F(P1, Info1, 1000) end), + spawn_link(fun () -> F(P2, Info2, 1000) end), + receive {ok,P1} -> ok end, + receive {ok,P2} -> ok end, + true = erlang:port_close(P1), + true = erlang:port_close(P2), + ok. + output_test(_, _, Input, Output) when Output > 16#1fffffff -> io:format("~p bytes received\n", [Input]); output_test(P, Bin, Input0, Output0) -> diff --git a/erts/emulator/test/time_SUITE.erl b/erts/emulator/test/time_SUITE.erl index a0a8a9c42c..43f7ac7f7c 100644 --- a/erts/emulator/test/time_SUITE.erl +++ b/erts/emulator/test/time_SUITE.erl @@ -34,7 +34,14 @@ bad_univ_to_local/1, bad_local_to_univ/1, univ_to_seconds/1, seconds_to_univ/1, consistency/1, - now_unique/1, now_update/1, timestamp/1]). + now_unique/1, now_update/1, timestamp/1, + time_warp_modes/1, + monotonic_time_monotonicity/1, + time_unit_conversion/1, + signed_time_unit_conversion/1, + erlang_timestamp/1]). + +-export([init_per_testcase/2, end_per_testcase/2]). -export([local_to_univ_utc/1]). @@ -56,6 +63,12 @@ -define(dst_timezone, 2). +init_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> + [{testcase, Func}|Config]. + +end_per_testcase(_Func, Config) -> + ok. + suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> @@ -63,7 +76,12 @@ all() -> bad_univ_to_local, bad_local_to_univ, univ_to_seconds, seconds_to_univ, consistency, - {group, now}, timestamp]. + {group, now}, timestamp, + time_warp_modes, + monotonic_time_monotonicity, + time_unit_conversion, + signed_time_unit_conversion, + erlang_timestamp]. groups() -> [{now, [], [now_unique, now_update]}]. @@ -420,6 +438,368 @@ now_update1(N) when N > 0 -> now_update1(0) -> ?line test_server:fail(). +time_warp_modes(Config) when is_list(Config) -> + %% All time warp modes always supported in + %% combination with no time correction... + check_time_warp_mode(Config, false, no_time_warp), + check_time_warp_mode(Config, false, single_time_warp), + check_time_warp_mode(Config, false, multi_time_warp), + + erts_debug:set_internal_state(available_internal_state, true), + try + case erts_debug:get_internal_state({check_time_config, + true, no_time_warp}) of + false -> ok; + true -> check_time_warp_mode(Config, true, no_time_warp) + end, + case erts_debug:get_internal_state({check_time_config, + true, single_time_warp}) of + false -> ok; + true -> check_time_warp_mode(Config, true, single_time_warp) + end, + case erts_debug:get_internal_state({check_time_config, + true, multi_time_warp}) of + false -> ok; + true -> check_time_warp_mode(Config, true, multi_time_warp) + end + after + erts_debug:set_internal_state(available_internal_state, false) + end. + +check_time_warp_mode(Config, TimeCorrection, TimeWarpMode) -> + io:format("~n~n~n***** Testing TimeCorrection=~p TimeWarpMode=~p *****~n", + [TimeCorrection, TimeWarpMode]), + Mon = erlang:monitor(time_offset, clock_service), + _ = erlang:time_offset(), + Start = erlang:monotonic_time(1000), + MonotonicityTimeout = 2000, + {ok, Node} = start_node(Config, + "+c " ++ atom_to_list(TimeCorrection) + ++ " +C " ++ atom_to_list(TimeWarpMode)), + StartTime = rpc:call(Node, erlang, system_info, [start_time]), + Me = self(), + MonotincityTestStarted = make_ref(), + MonotincityTestDone = make_ref(), + spawn_link(Node, + fun () -> + Me ! MonotincityTestStarted, + cmp_times(erlang:start_timer(MonotonicityTimeout, + self(), + timeout), + erlang:monotonic_time()), + Me ! MonotincityTestDone + end), + receive MonotincityTestStarted -> ok end, + check_time_offset(Node, TimeWarpMode), + TimeWarpMode = rpc:call(Node, erlang, system_info, [time_warp_mode]), + TimeCorrection = rpc:call(Node, erlang, system_info, [time_correction]), + receive MonotincityTestDone -> ok end, + MonotonicTime = rpc:call(Node, erlang, monotonic_time, []), + MonotonicTimeUnit = rpc:call(Node, + erlang, + convert_time_unit, + [1, seconds, native]), + UpMilliSeconds = erlang:convert_time_unit(MonotonicTime - StartTime, + MonotonicTimeUnit, + milli_seconds), + io:format("UpMilliSeconds=~p~n", [UpMilliSeconds]), + End = erlang:monotonic_time(milli_seconds), + stop_node(Node), + try + true = (UpMilliSeconds > (98*MonotonicityTimeout) div 100), + true = (UpMilliSeconds < (102*(End-Start)) div 100) + catch + error:_ -> + io:format("Uptime inconsistency", []), + case {TimeCorrection, erlang:system_info(time_correction)} of + {true, true} -> + ?t:fail(uptime_inconsistency); + {true, false} -> + _ = erlang:time_offset(), + receive + {'CHANGE', Mon, time_offset, clock_service, _} -> + ignore + after 1000 -> + ?t:fail(uptime_inconsistency) + end; + _ -> + ignore + end + end, + erlang:demonitor(Mon, [flush]), + ok. + +check_time_offset(Node, no_time_warp) -> + final = rpc:call(Node, erlang, system_info, [time_offset]), + final = rpc:call(Node, erlang, system_flag, [time_offset, finalize]), + final = rpc:call(Node, erlang, system_info, [time_offset]); +check_time_offset(Node, single_time_warp) -> + preliminary = rpc:call(Node, erlang, system_info, [time_offset]), + preliminary = rpc:call(Node, erlang, system_flag, [time_offset, finalize]), + final = rpc:call(Node, erlang, system_info, [time_offset]), + final = rpc:call(Node, erlang, system_flag, [time_offset, finalize]); +check_time_offset(Node, multi_time_warp) -> + volatile = rpc:call(Node, erlang, system_info, [time_offset]), + volatile = rpc:call(Node, erlang, system_flag, [time_offset, finalize]), + volatile = rpc:call(Node, erlang, system_info, [time_offset]). + +monotonic_time_monotonicity(Config) when is_list(Config) -> + Done = erlang:start_timer(10000,self(),timeout), + cmp_times(Done, erlang:monotonic_time()). + +cmp_times(Done, X0) -> + X1 = erlang:monotonic_time(), + X2 = erlang:monotonic_time(), + X3 = erlang:monotonic_time(), + X4 = erlang:monotonic_time(), + X5 = erlang:monotonic_time(), + true = (X0 =< X1), + true = (X1 =< X2), + true = (X2 =< X3), + true = (X3 =< X4), + true = (X4 =< X5), + receive + {timeout, Done, timeout} -> + ok + after 0 -> + cmp_times(Done, X5) + end. + +-define(CHK_RES_CONVS_TIMEOUT, 400). + +time_unit_conversion(Config) when is_list(Config) -> + Mon = erlang:monitor(time_offset, clock_service), + start_check_res_convs(Mon, 1000000000000), + start_check_res_convs(Mon, 2333333333333), + start_check_res_convs(Mon, 5732678356789), + erlang:demonitor(Mon, [flush]). + +start_check_res_convs(Mon, Res) -> + io:format("Checking ~p time_unit~n", [Res]), + check_res_convs(Mon, + erlang:start_timer(?CHK_RES_CONVS_TIMEOUT, + self(), + timeout), + Res). + + +check_res_convs(Mon, Done, Res) -> + receive + {timeout, Done, timeout} -> + case Res div 10 of + 0 -> + ok; + NewRes -> + start_check_res_convs(Mon, NewRes) + end + after 0 -> + do_check_res_convs(Mon, Done, Res) + end. + +do_check_res_convs(Mon, Done, Res) -> + TStart = erlang:monotonic_time(), + T = erlang:monotonic_time(Res), + TEnd = erlang:monotonic_time(), + TMin = erlang:convert_time_unit(TStart, native, Res), + TMax = erlang:convert_time_unit(TEnd, native, Res), + %io:format("~p =< ~p =< ~p~n", [TMin, T, TEnd]), + true = (TMin =< T), + true = (TMax >= T), + check_time_offset_res_conv(Mon, Res), + check_res_convs(Mon, Done, Res). + + +check_time_offset_res_conv(Mon, Res) -> + TORes = erlang:time_offset(Res), + TO = erlang:time_offset(), + case erlang:convert_time_unit(TO, native, Res) of + TORes -> + ok; + TORes2 -> + case check_time_offset_change(Mon, TO, 1000) of + {TO, false} -> + ?t:fail({time_unit_conversion_inconsistency, + TO, TORes, TORes2}); + {_NewTO, true} -> + ?t:format("time_offset changed", []), + check_time_offset_res_conv(Mon, Res) + end + end. + +signed_time_unit_conversion(Config) when is_list(Config) -> + chk_strc(1000000000, 1000000), + chk_strc(1000000000, 1000), + chk_strc(1000000000, 1), + chk_strc(1000000, 1000), + chk_strc(1000000, 1), + chk_strc(1000, 1), + chk_strc(4711, 17), + chk_strc(1 bsl 10, 1), + chk_strc(1 bsl 16, 10), + chk_strc(1 bsl 17, 1 bsl 8), + chk_strc((1 bsl 17) + 1, (1 bsl 8) - 1), + chk_strc(1 bsl 17, 11), + ok. + +chk_strc(Res0, Res1) -> + case (Res0 /= Res1) andalso (Res0 =< 1000000) andalso (Res1 =< 1000000) of + true -> + {FromRes, ToRes} = case Res0 > Res1 of + true -> {Res0, Res1}; + false -> {Res1, Res0} + end, + MinFromValuesPerToValue = FromRes div ToRes, + MaxFromValuesPerToValue = ((FromRes-1) div ToRes)+1, + io:format("~p -> ~p [~p, ~p]~n", + [FromRes, ToRes, + MinFromValuesPerToValue, MaxFromValuesPerToValue]), + chk_values_per_value(FromRes, ToRes, + -10*FromRes, 10*FromRes, + MinFromValuesPerToValue, + MaxFromValuesPerToValue, + undefined, MinFromValuesPerToValue); + _ -> + ok + end, + chk_random_values(Res0, Res1), + chk_random_values(Res1, Res0), + ok. + +chk_random_values(FR, TR) -> +% case (FR rem TR == 0) orelse (TR rem FR == 0) of +% true -> + io:format("rand values ~p -> ~p~n", [FR, TR]), + random:seed(268438039, 268440479, 268439161), + Values = lists:map(fun (_) -> random:uniform(1 bsl 65) - (1 bsl 64) end, + lists:seq(1, 100000)), + CheckFun = fun (V) -> + CV = erlang:convert_time_unit(V, FR, TR), + case {(FR*CV) div TR =< V, + (FR*(CV+1)) div TR >= V} of + {true, true} -> + ok; + Failure -> + ?t:fail({Failure, CV, V, FR, TR}) + end + end, + lists:foreach(CheckFun, Values).%; +% false -> ok +% end. + + +chk_values_per_value(_FromRes, _ToRes, + EndValue, EndValue, + MinFromValuesPerToValue, MaxFromValuesPerToValue, + _ToValue, FromValueCount) -> +% io:format("~p [~p]~n", [EndValue, FromValueCount]), + case ((MinFromValuesPerToValue =< FromValueCount) + andalso (FromValueCount =< MaxFromValuesPerToValue)) of + false -> + ?t:fail({MinFromValuesPerToValue, + FromValueCount, + MaxFromValuesPerToValue}); + true -> + ok + end; +chk_values_per_value(FromRes, ToRes, Value, EndValue, + MinFromValuesPerToValue, MaxFromValuesPerToValue, + ToValue, FromValueCount) -> + case erlang:convert_time_unit(Value, FromRes, ToRes) of + ToValue -> + chk_values_per_value(FromRes, ToRes, + Value+1, EndValue, + MinFromValuesPerToValue, + MaxFromValuesPerToValue, + ToValue, FromValueCount+1); + NewToValue -> + case ((MinFromValuesPerToValue =< FromValueCount) + andalso (FromValueCount =< MaxFromValuesPerToValue)) of + false -> + ?t:fail({MinFromValuesPerToValue, + FromValueCount, + MaxFromValuesPerToValue}); + true -> +% io:format("~p -> ~p [~p]~n", +% [Value, NewToValue, FromValueCount]), + chk_values_per_value(FromRes, ToRes, + Value+1, EndValue, + MinFromValuesPerToValue, + MaxFromValuesPerToValue, + NewToValue, 1) + end + end. + +erlang_timestamp(Config) when is_list(Config) -> + Mon = erlang:monitor(time_offset, clock_service), + {TO, _} = check_time_offset_change(Mon, + erlang:time_offset(), + 0), + Done = erlang:start_timer(10000,self(),timeout), + ok = check_erlang_timestamp(Done, Mon, TO). + +check_erlang_timestamp(Done, Mon, TO) -> + receive + {timeout, Done, timeout} -> + erlang:demonitor(Mon, [flush]), + ok + after 0 -> + do_check_erlang_timestamp(Done, Mon, TO) + end. + +do_check_erlang_timestamp(Done, Mon, TO) -> + MinMon = erlang:monotonic_time(), + {MegaSec, Sec, MicroSec} = erlang:timestamp(), + MaxMon = erlang:monotonic_time(), + TsMin = erlang:convert_time_unit(MinMon+TO, + native, + micro_seconds), + TsMax = erlang:convert_time_unit(MaxMon+TO, + native, + micro_seconds), + TsTime = (MegaSec*1000000+Sec)*1000000+MicroSec, + case (TsMin =< TsTime) andalso (TsTime =< TsMax) of + true -> + NewTO = case erlang:time_offset() of + TO -> + TO; + _ -> + check_time_offset_change(Mon, TO, 0) + end, + check_erlang_timestamp(Done, Mon, NewTO); + false -> + io:format("TsMin=~p TsTime=~p TsMax=~p~n", [TsMin, TsTime, TsMax]), + ?t:format("Detected inconsistency; " + "checking for time_offset change...", []), + case check_time_offset_change(Mon, TO, 1000) of + {TO, false} -> + ?t:fail(timestamp_inconsistency); + {NewTO, true} -> + ?t:format("time_offset changed", []), + check_erlang_timestamp(Done, Mon, NewTO) + end + end. + +check_time_offset_change(Mon, TO, Wait) -> + process_changed_time_offset(Mon, TO, false, Wait). + +process_changed_time_offset(Mon, TO, Changed, Wait) -> + receive + {'CHANGE', Mon, time_offset, clock_service, NewTO} -> + process_changed_time_offset(Mon, NewTO, true, Wait) + after Wait -> + case erlang:time_offset() of + TO -> + {TO, Changed}; + _OtherTO -> + receive + {'CHANGE', Mon, time_offset, clock_service, NewTO} -> + process_changed_time_offset(Mon, NewTO, true, Wait) + end + end + end. + + + %% Returns the test data: a list of {Utc, Local} tuples. test_data() -> @@ -554,4 +934,25 @@ bad_dates() -> {{1996, 4, 30}, {12, 0, -1}}, % Sec {{1996, 4, 30}, {12, 0, 60}}]. - + +start_node(Config) -> + start_node(Config, ""). + +start_node(Config, Args) -> + TestCase = ?config(testcase, Config), + PA = filename:dirname(code:which(?MODULE)), + ESTime = erlang:monotonic_time(1) + erlang:time_offset(1), + Unique = erlang:unique_integer([positive]), + Name = list_to_atom(atom_to_list(?MODULE) + ++ "-" + ++ atom_to_list(TestCase) + ++ "-" + ++ integer_to_list(ESTime) + ++ "-" + ++ integer_to_list(Unique)), + test_server:start_node(Name, + slave, + [{args, "-pa " ++ PA ++ " " ++ Args}]). + +stop_node(Node) -> + test_server:stop_node(Node). diff --git a/erts/emulator/test/timer_bif_SUITE.erl b/erts/emulator/test/timer_bif_SUITE.erl index c28224729d..56a1cef761 100644 --- a/erts/emulator/test/timer_bif_SUITE.erl +++ b/erts/emulator/test/timer_bif_SUITE.erl @@ -232,12 +232,16 @@ read_timer(Config) when is_list(Config) -> cleanup(doc) -> []; cleanup(suite) -> []; cleanup(Config) when is_list(Config) -> + {skipped, "Test needs to be UPDATED for new timer implementation"}. + +cleanup_test(Config) when is_list(Config) -> ?line Mem = mem(), %% Timer on dead process ?line P1 = spawn(fun () -> ok end), ?line wait_until(fun () -> process_is_cleaned_up(P1) end), ?line T1 = erlang:start_timer(10000, P1, "hej"), ?line T2 = erlang:send_after(10000, P1, "hej"), + receive after 1000 -> ok end, ?line Mem = mem(), ?line false = erlang:read_timer(T1), ?line false = erlang:read_timer(T2), @@ -250,6 +254,7 @@ cleanup(Config) when is_list(Config) -> ?line true = is_integer(erlang:read_timer(T3)), ?line true = is_integer(erlang:read_timer(T4)), ?line wait_until(fun () -> process_is_cleaned_up(P2) end), + receive after 1000 -> ok end, ?line false = erlang:read_timer(T3), ?line false = erlang:read_timer(T4), ?line Mem = mem(), @@ -418,6 +423,9 @@ evil_recv_timeouts(TOs, N, M) -> registered_process(doc) -> []; registered_process(suite) -> []; registered_process(Config) when is_list(Config) -> + {skipped, "Test needs to be UPDATED for new timer implementation"}. + +registered_process_test(Config) when is_list(Config) -> ?line Mem = mem(), %% Cancel ?line T1 = erlang:start_timer(500, ?MODULE, "hej"), @@ -455,10 +463,18 @@ registered_process(Config) when is_list(Config) -> ?line ok. mem() -> - AA = erlang:system_info(allocated_areas), - {value,{bif_timer,Mem}} = lists:keysearch(bif_timer, 1, AA), - Mem. - + TSrvs = erts_internal:get_bif_timer_servers(), + lists:foldl(fun (Tab, Sz) -> + case lists:member(ets:info(Tab, owner), TSrvs) of + true -> + ets:info(Tab, memory) + Sz; + false -> + Sz + end + end, + 0, + ets:all())*erlang:system_info({wordsize,external}). + process_is_cleaned_up(P) when is_pid(P) -> undefined == erts_debug:get_internal_state({process_status, P}). diff --git a/erts/emulator/test/unique_SUITE.erl b/erts/emulator/test/unique_SUITE.erl new file mode 100644 index 0000000000..5ad6e59272 --- /dev/null +++ b/erts/emulator/test/unique_SUITE.erl @@ -0,0 +1,390 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2014. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +-module(unique_SUITE). + +-export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, + init_per_group/2,end_per_group/2, + init_per_testcase/2,end_per_testcase/2]). +-export([unique_monotonic_integer_white_box/1, + unique_integer_white_box/1]). + +-include_lib("test_server/include/test_server.hrl"). + +%-define(P(V), V). +-define(P(V), print_ret_val(?FILE, ?LINE, V)). + +-define(PRINT(V), print_ret_val(?FILE, ?LINE, V)). + + +init_per_testcase(Case, Config) -> + ?line Dog=test_server:timetrap(test_server:minutes(2)), + [{watchdog, Dog}, {testcase, Case}|Config]. + +end_per_testcase(_, Config) -> + Dog=?config(watchdog, Config), + test_server:timetrap_cancel(Dog), + ok. + +suite() -> [{ct_hooks,[ts_install_cth]}]. + +all() -> + [unique_monotonic_integer_white_box, + unique_integer_white_box]. + +groups() -> + []. + +init_per_suite(Config) -> + erts_debug:set_internal_state(available_internal_state, true), + Config. + +end_per_suite(_Config) -> + erts_debug:set_internal_state(available_internal_state, false), + ok. + +init_per_group(_GroupName, Config) -> + Config. + +end_per_group(_GroupName, Config) -> + Config. + +%% +%% +%% Unique counter white box test case +%% +%% + +unique_monotonic_integer_white_box(Config) when is_list(Config) -> + {ok, Node} = start_node(Config), + TestServer = self(), + Success = make_ref(), + %% Run this in a separate node, so we don't mess up + %% the system when moving the strict monotonic counter + %% around in a non-strict monotonic way... + Test = spawn(Node, + fun () -> + unique_monotonic_integer_white_box_test(TestServer, Success) + end), + Mon = erlang:monitor(process, Test), + receive + {'DOWN', Mon, process, Test, Error} -> + ?t:fail(Error); + Success -> + ok + end, + erlang:demonitor(Mon, [flush]), + stop_node(Node), + ok. + +set_unique_monotonic_integer_state(MinCounter, NextValue) -> + true = erts_debug:set_internal_state(unique_monotonic_integer_state, + NextValue-MinCounter-1). + + + +unique_monotonic_integer_white_box_test(TestServer, Success) -> + erts_debug:set_internal_state(available_internal_state, true), + + WordSize = erlang:system_info({wordsize, internal}), + SmallBits = WordSize*8 - 4, + + MinSmall = -1*(1 bsl (SmallBits-1)), + MaxSmall = (1 bsl (SmallBits-1))-1, + %% Make sure we got small sizes correct... + 0 = erts_debug:size(MinSmall), + false = 0 =:= erts_debug:size(MinSmall-1), + 0 = erts_debug:size(MaxSmall), + false = 0 =:= erts_debug:size(MaxSmall+1), + + ?PRINT({min_small, MinSmall}), + ?PRINT({max_small, MaxSmall}), + + MinSint64 = -1*(1 bsl 63), + MaxSint64 = (1 bsl 63)-1, + + ?PRINT({min_Sint64, MinSint64}), + ?PRINT({max_Sint64, MaxSint64}), + + MinCounter = erts_debug:get_internal_state(min_unique_monotonic_integer), + MaxCounter = MinCounter + (1 bsl 64) - 1, + + ?PRINT({min_counter, MinCounter}), + ?PRINT({max_counter, MaxCounter}), + + case WordSize of + 4 -> + MinCounter = MinSint64; + 8 -> + MinCounter = MinSmall + end, + + StartState = erts_debug:get_internal_state(unique_monotonic_integer_state), + + %% Verify that we get expected results over all internal limits... + + case MinCounter < MinSmall of + false -> + 8 = WordSize, + ok; + true -> + 4 = WordSize, + ?PRINT(over_min_small), + set_unique_monotonic_integer_state(MinCounter, MinSmall-2), + true = (?P(erlang:unique_integer([monotonic])) == MinSmall - 2), + true = (?P(erlang:unique_integer([monotonic])) == MinSmall - 1), + true = (?P(erlang:unique_integer([monotonic])) == MinSmall), + true = (?P(erlang:unique_integer([monotonic])) == MinSmall + 1), + true = (?P(erlang:unique_integer([monotonic])) == MinSmall + 2), + garbage_collect(), + ok + end, + + ?PRINT(over_zero), %% Not really an interesting limit, but... + set_unique_monotonic_integer_state(MinCounter, -2), + true = (?P(erlang:unique_integer([monotonic])) == -2), + true = (?P(erlang:unique_integer([monotonic])) == -1), + true = (?P(erlang:unique_integer([monotonic])) == 0), + true = (?P(erlang:unique_integer([monotonic])) == 1), + true = (?P(erlang:unique_integer([monotonic])) == 2), + garbage_collect(), + + ?PRINT(over_max_small), + set_unique_monotonic_integer_state(MinCounter, MaxSmall-2), + true = (?P(erlang:unique_integer([monotonic])) == MaxSmall - 2), + true = (?P(erlang:unique_integer([monotonic])) == MaxSmall - 1), + true = (?P(erlang:unique_integer([monotonic])) == MaxSmall), + true = (?P(erlang:unique_integer([monotonic])) == MaxSmall + 1), + true = (?P(erlang:unique_integer([monotonic])) == MaxSmall + 2), + garbage_collect(), + + case MaxCounter > MaxSint64 of + false -> + 4 = WordSize, + ok; + true -> + 8 = WordSize, + ?PRINT(over_max_sint64), + set_unique_monotonic_integer_state(MinCounter, MaxSint64-2), + true = (?P(erlang:unique_integer([monotonic])) == MaxSint64 - 2), + true = (?P(erlang:unique_integer([monotonic])) == MaxSint64 - 1), + true = (?P(erlang:unique_integer([monotonic])) == MaxSint64), + true = (?P(erlang:unique_integer([monotonic])) == MaxSint64 + 1), + true = (?P(erlang:unique_integer([monotonic])) == MaxSint64 + 2), + garbage_collect() + end, + + ?PRINT(over_max_min_counter), + set_unique_monotonic_integer_state(MinCounter, if MaxCounter == MaxSint64 -> + MaxCounter-2; + true -> + MinCounter-3 + end), + true = (?P(erlang:unique_integer([monotonic])) == MaxCounter - 2), + true = (?P(erlang:unique_integer([monotonic])) == MaxCounter - 1), + true = (?P(erlang:unique_integer([monotonic])) == MaxCounter), + true = (?P(erlang:unique_integer([monotonic])) == MinCounter), + true = (?P(erlang:unique_integer([monotonic])) == MinCounter + 1), + true = (?P(erlang:unique_integer([monotonic])) == MinCounter + 2), + garbage_collect(), + + %% Restore initial state and hope we didn't mess it up for the + %% system... + true = erts_debug:set_internal_state(unique_monotonic_integer_state, + StartState), + + TestServer ! Success. + +%% +%% +%% Unique integer white box test case +%% +%% + +-record(uniqint_info, {min_int, + max_int, + max_small, + schedulers, + sched_bits}). + +unique_integer_white_box(Config) when is_list(Config) -> + UinqintInfo = init_uniqint_info(), + #uniqint_info{min_int = MinInt, + max_int = MaxInt, + max_small = MaxSmall} = UinqintInfo, + io:format("****************************************************~n", []), + io:format("*** Around MIN_UNIQ_INT ~p ***~n", [MinInt]), + io:format("****************************************************~n", []), + check_unique_integer_around(MinInt, UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around 0 ***~n", []), + io:format("****************************************************~n", []), + check_unique_integer_around(0, UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around MAX_SMALL ~p ***~n", [MaxSmall]), + io:format("****************************************************~n", []), + check_unique_integer_around(MaxSmall, UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around 2^64+MIN_UNIQ_INT ~p ***~n", [(1 bsl 64)+MinInt]), + io:format("****************************************************~n", []), + check_unique_integer_around((1 bsl 64)+MinInt, UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around 2^64 ~p~n", [(1 bsl 64)]), + io:format("****************************************************~n", []), + check_unique_integer_around((1 bsl 64), UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around 2^64-MIN_UNIQ_INT ~p ***~n", [(1 bsl 64)-MinInt]), + io:format("****************************************************~n", []), + check_unique_integer_around((1 bsl 64)-MinInt, UinqintInfo), + io:format("****************************************************~n", []), + io:format("*** Around MAX_UNIQ_INT ~p ***~n", [MaxInt]), + io:format("****************************************************~n", []), + check_unique_integer_around(MaxInt, UinqintInfo), + ok. + + +%%% Internal unique_integer_white_box/1 test case + +calc_sched_bits(NoScheds, Shift) when NoScheds < 1 bsl Shift -> + Shift; +calc_sched_bits(NoScheds, Shift) -> + calc_sched_bits(NoScheds, Shift+1). + +init_uniqint_info() -> + SmallBits = erlang:system_info({wordsize, internal})*8-4, + io:format("SmallBits=~p~n", [SmallBits]), + Schedulers = erlang:system_info(schedulers), + io:format("Schedulers=~p~n", [Schedulers]), + MinSmall = -1*(1 bsl (SmallBits-1)), + io:format("MinSmall=~p~n", [MinSmall]), + MaxSmall = (1 bsl (SmallBits-1))-1, + io:format("MaxSmall=~p~n", [MaxSmall]), + SchedBits = calc_sched_bits(Schedulers, 0), + io:format("SchedBits=~p~n", [SchedBits]), + MaxInt = ((((1 bsl 64) - 1) bsl SchedBits) bor Schedulers) + MinSmall, + io:format("MaxInt=~p~n", [MaxInt]), + #uniqint_info{min_int = MinSmall, + max_int = MaxInt, + max_small = MaxSmall, + schedulers = Schedulers, + sched_bits = SchedBits}. + +valid_uniqint(Int, #uniqint_info{min_int = MinInt} = UinqintInfo) when Int < MinInt -> + valid_uniqint(MinInt, UinqintInfo); +valid_uniqint(Int, #uniqint_info{min_int = MinInt, + sched_bits = SchedBits, + schedulers = Scheds}) -> + Int1 = Int - MinInt, + {Inc, ThreadNo} = case Int1 band ((1 bsl SchedBits) - 1) of + TN when TN > Scheds -> + {1, Scheds}; + TN -> + {0, TN} + end, + Counter = ((Int1 bsr SchedBits) + Inc) rem (1 bsl 64), + ((Counter bsl SchedBits) bor ThreadNo) + MinInt. + +smaller_valid_uniqint(Int, UinqintInfo) -> + Cand = Int-1, + case valid_uniqint(Cand, UinqintInfo) of + RI when RI < Int -> + RI; + _ -> + smaller_valid_uniqint(Cand, UinqintInfo) + end. + +int32_to_bigendian_list(Int) -> + 0 = Int bsr 32, + [(Int bsr 24) band 16#ff, + (Int bsr 16) band 16#ff, + (Int bsr 8) band 16#ff, + Int band 16#ff]. + +mk_uniqint(Int, #uniqint_info {min_int = MinInt, + sched_bits = SchedBits} = _UinqintInfo) -> + Int1 = Int - MinInt, + ThrId = Int1 band ((1 bsl SchedBits) - 1), + Value = (Int1 bsr SchedBits) band ((1 bsl 64) - 1), + 0 = Int1 bsr (SchedBits + 64), + NodeName = atom_to_list(node()), + Make = {make_unique_integer, ThrId, Value}, + %% erlang:display(Make), + Res = erts_debug:get_internal_state(Make), + %% erlang:display({uniq_int, Res}), + Res. + +check_uniqint(Int, UinqintInfo) -> + UniqInt = mk_uniqint(Int, UinqintInfo), + io:format("UniqInt=~p ", [UniqInt]), + case UniqInt =:= Int of + true -> + io:format("OK~n~n", []); + false -> + io:format("result UniqInt=~p FAILED~n", [UniqInt]), + exit(badres) + end. + +check_unique_integer_around(Int, #uniqint_info{min_int = MinInt, + max_int = MaxInt} = UinqintInfo) -> + {Start, End} = case {Int =< MinInt+100, Int >= MaxInt-100} of + {true, false} -> + {MinInt, MinInt+100}; + {false, false} -> + {smaller_valid_uniqint(Int-100, UinqintInfo), + valid_uniqint(Int+100, UinqintInfo)}; + {false, true} -> + {MaxInt-100, MaxInt} + end, + lists:foldl(fun (I, OldRefInt) -> + RefInt = valid_uniqint(I, UinqintInfo), + case OldRefInt =:= RefInt of + true -> + ok; + false -> + check_uniqint(RefInt, UinqintInfo) + end, + RefInt + end, + none, + lists:seq(Start, End)). + + +%% helpers + +print_ret_val(File, Line, Value) -> + io:format("~s:~p: ~p~n", [File, Line, Value]), + Value. + +start_node(Config) -> + start_node(Config, []). +start_node(Config, Opts) when is_list(Config), is_list(Opts) -> + ?line Pa = filename:dirname(code:which(?MODULE)), + ?line A = erlang:monotonic_time(1) + erlang:time_offset(1), + ?line B = erlang:unique_integer([positive]), + ?line Name = list_to_atom(atom_to_list(?MODULE) + ++ "-" + ++ atom_to_list(?config(testcase, Config)) + ++ "-" + ++ integer_to_list(A) + ++ "-" + ++ integer_to_list(B)), + ?line ?t:start_node(Name, slave, [{args, Opts++" -pa "++Pa}]). + +stop_node(Node) -> + ?t:stop_node(Node). diff --git a/erts/etc/common/erlexec.c b/erts/etc/common/erlexec.c index 5ebde8ca3c..23226909a7 100644 --- a/erts/etc/common/erlexec.c +++ b/erts/etc/common/erlexec.c @@ -143,6 +143,7 @@ static char *pluss_val_switches[] = { static char *plush_val_switches[] = { "ms", "mbs", + "pds", "", NULL }; @@ -807,6 +808,7 @@ int main(int argc, char **argv) case 'a': case 'A': case 'b': + case 'C': case 'e': case 'i': case 'n': @@ -880,6 +882,19 @@ int main(int argc, char **argv) } add_Eargs(argv[i]); break; + case 'c': + argv[i][0] = '-'; + if (argv[i][2] == '\0' && i+1 < argc) { + if (sys_strcmp(argv[i+1], "true") == 0 + || sys_strcmp(argv[i+1], "false") == 0) { + add_Eargs(argv[i]); + add_Eargs(argv[i+1]); + i++; + break; + } + } + add_Eargs(argv[i]); + break; case 'M': { int x; for (x = 0; plusM_au_allocs[x]; x++) @@ -1149,8 +1164,8 @@ usage_aux(void) #endif "] " "[-make] [-man [manopts] MANPAGE] [-x] [-emu_args] " - "[-args_file FILENAME] [+A THREADS] [+a SIZE] [+B[c|d|i]] [+c] " - "[+h HEAP_SIZE_OPTION] [+K BOOLEAN] " + "[-args_file FILENAME] [+A THREADS] [+a SIZE] [+B[c|d|i]] [+c [BOOLEAN]] " + "[+C MODE] [+h HEAP_SIZE_OPTION] [+K BOOLEAN] " "[+l] [+M<SUBSWITCH> <ARGUMENT>] [+P MAX_PROCS] [+Q MAX_PORTS] " "[+R COMPAT_REL] " "[+r] [+rg READER_GROUPS_LIMIT] [+s SCHEDULER_OPTION] " diff --git a/erts/etc/common/heart.c b/erts/etc/common/heart.c index 2830641802..a4f34e21d0 100644 --- a/erts/etc/common/heart.c +++ b/erts/etc/common/heart.c @@ -109,7 +109,7 @@ # include <sys/time.h> # include <unistd.h> # include <signal.h> -# if defined(CORRECT_USING_TIMES) +# if defined(OS_MONOTONIC_TIME_USING_TIMES) # include <sys/times.h> # include <limits.h> # endif @@ -1084,9 +1084,9 @@ time_t timestamp(time_t *res) return r; } -#elif defined(HAVE_GETHRTIME) || defined(GETHRTIME_WITH_CLOCK_GETTIME) +#elif defined(OS_MONOTONIC_TIME_USING_GETHRTIME) || defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) -#if defined(GETHRTIME_WITH_CLOCK_GETTIME) +#if defined(OS_MONOTONIC_TIME_USING_CLOCK_GETTIME) typedef long long SysHrTime; SysHrTime sys_gethrtime(void); @@ -1095,7 +1095,7 @@ SysHrTime sys_gethrtime(void) { struct timespec ts; long long result; - if (clock_gettime(CLOCK_MONOTONIC,&ts) != 0) { + if (clock_gettime(MONOTONIC_CLOCK_ID,&ts) != 0) { print_error("Fatal, could not get clock_monotonic value, terminating! " "errno = %d\n", errno); exit(1); @@ -1122,7 +1122,7 @@ time_t timestamp(time_t *res) return r; } -#elif defined(CORRECT_USING_TIMES) +#elif defined(OS_MONOTONIC_TIME_USING_TIMES) # ifdef NO_SYSCONF # include <sys/param.h> diff --git a/erts/etc/unix/cerl.src b/erts/etc/unix/cerl.src index aa51eabfc5..59cf29d381 100644 --- a/erts/etc/unix/cerl.src +++ b/erts/etc/unix/cerl.src @@ -381,7 +381,9 @@ elif [ "x$GDB" = "xegdb" ]; then # Set annotation level for gdb in emacs 22 and higher. Seems to # be working with level 1 for emacs 22 and level 3 for emacs 23... emacs_major=`$EMACS --version | head -1 | sed 's,^[^0-9]*\([0-9]*\).*,\1,g'` - if [ '!' -z "$emacs_major" -a $emacs_major -gt 22 ]; then + if [ '!' -z "$emacs_major" -a $emacs_major -gt 23 ]; then + GDBARGS="-i=mi " + elif [ '!' -z "$emacs_major" -a $emacs_major -gt 22 ]; then GDBARGS="--annotate=3 " elif [ '!' -z "$emacs_major" -a $emacs_major -gt 21 ]; then GDBARGS="--annotate=1 " diff --git a/erts/example/Makefile b/erts/example/Makefile index 6e1a88b4da..b637bee033 100644 --- a/erts/example/Makefile +++ b/erts/example/Makefile @@ -30,7 +30,7 @@ CFLAGS += $(OUR_C_FLAGS) CXXFLAGS += $(OUR_C_FLAGS) TARGETS = pg_sync.beam pg_async.beam pg_sync.so pg_async.so \ -next_perm.so next_perm.beam +next_perm.so next_perm.beam time_compat.beam all: $(TARGETS) diff --git a/erts/example/time_compat.erl b/erts/example/time_compat.erl new file mode 100644 index 0000000000..90b7fbcc80 --- /dev/null +++ b/erts/example/time_compat.erl @@ -0,0 +1,303 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2014. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +%% +%% If your code need to be able to execute on ERTS versions both +%% earlier and later than 7.0, the best approach is to use the new +%% time API introduced in ERTS 7.0 and implement a fallback +%% solution using the old primitives to be used on old ERTS +%% versions. This way your code can automatically take advantage +%% of the improvements in the API when available. This is an +%% example of how to implement such an API, but it can be used +%% as is if you want to. Just add this module to your project, +%% and call the API via this module instead of calling the +%% BIFs directly. +%% + +-module(time_compat). + +%% We don't want warnings about the use of erlang:now/0 in +%% this module. +-compile(nowarn_deprecated_function). +%% +%% We don't use +%% -compile({nowarn_deprecated_function, [{erlang, now, 0}]}). +%% since this will produce warnings when compiled on systems +%% where it has not yet been deprecated. +%% + +-export([monotonic_time/0, + monotonic_time/1, + erlang_system_time/0, + erlang_system_time/1, + os_system_time/0, + os_system_time/1, + time_offset/0, + time_offset/1, + convert_time_unit/3, + timestamp/0, + unique_integer/0, + unique_integer/1, + monitor/2, + system_info/1, + system_flag/2]). + +monotonic_time() -> + try + erlang:monotonic_time() + catch + error:undef -> + %% Use Erlang system time as monotonic time + erlang_system_time_fallback() + end. + +monotonic_time(Unit) -> + try + erlang:monotonic_time(Unit) + catch + error:badarg -> + erlang:error(badarg, [Unit]); + error:undef -> + %% Use Erlang system time as monotonic time + STime = erlang_system_time_fallback(), + try + convert_time_unit_fallback(STime, native, Unit) + catch + error:bad_time_unit -> erlang:error(badarg, [Unit]) + end + end. + +erlang_system_time() -> + try + erlang:system_time() + catch + error:undef -> + erlang_system_time_fallback() + end. + +erlang_system_time(Unit) -> + try + erlang:system_time(Unit) + catch + error:badarg -> + erlang:error(badarg, [Unit]); + error:undef -> + STime = erlang_system_time_fallback(), + try + convert_time_unit_fallback(STime, native, Unit) + catch + error:bad_time_unit -> erlang:error(badarg, [Unit]) + end + end. + +os_system_time() -> + try + os:system_time() + catch + error:undef -> + os_system_time_fallback() + end. + +os_system_time(Unit) -> + try + os:system_time(Unit) + catch + error:badarg -> + erlang:error(badarg, [Unit]); + error:undef -> + STime = os_system_time_fallback(), + try + convert_time_unit_fallback(STime, native, Unit) + catch + error:bad_time_unit -> erlang:error(badarg, [Unit]) + end + end. + +time_offset() -> + try + erlang:time_offset() + catch + error:undef -> + %% Erlang system time and Erlang monotonic + %% time are always aligned + 0 + end. + +time_offset(Unit) -> + try + erlang:time_offset(Unit) + catch + error:badarg -> + erlang:error(badarg, [Unit]); + error:undef -> + try + _ = integer_time_unit(Unit) + catch + error:bad_time_unit -> erlang:error(badarg, [Unit]) + end, + %% Erlang system time and Erlang monotonic + %% time are always aligned + 0 + end. + +convert_time_unit(Time, FromUnit, ToUnit) -> + try + erlang:convert_time_unit(Time, FromUnit, ToUnit) + catch + error:undef -> + try + convert_time_unit_fallback(Time, FromUnit, ToUnit) + catch + _:_ -> + erlang:error(badarg, [Time, FromUnit, ToUnit]) + end; + error:Error -> + erlang:error(Error, [Time, FromUnit, ToUnit]) + end. + +timestamp() -> + try + erlang:timestamp() + catch + error:undef -> + erlang:now() + end. + +unique_integer() -> + try + erlang:unique_integer() + catch + error:undef -> + {MS, S, US} = erlang:now(), + (MS*1000000+S)*1000000+US + end. + +unique_integer(Modifiers) -> + try + erlang:unique_integer(Modifiers) + catch + error:badarg -> + erlang:error(badarg, [Modifiers]); + error:undef -> + case is_valid_modifier_list(Modifiers) of + true -> + %% now() converted to an integer + %% fullfill the requirements of + %% all modifiers: unique, positive, + %% and monotonic... + {MS, S, US} = erlang:now(), + (MS*1000000+S)*1000000+US; + false -> + erlang:error(badarg, [Modifiers]) + end + end. + +monitor(Type, Item) -> + try + erlang:monitor(Type, Item) + catch + error:Error -> + case {Error, Type, Item} of + {badarg, time_offset, clock_service} -> + %% Time offset is final and will never change. + %% Return a dummy reference, there will never + %% be any need for 'CHANGE' messages... + make_ref(); + _ -> + erlang:error(Error, [Type, Item]) + end + end. + +system_info(Item) -> + try + erlang:system_info(Item) + catch + error:badarg -> + case Item of + time_correction -> + case erlang:system_info(tolerant_timeofday) of + enabled -> true; + disabled -> false + end; + time_warp_mode -> + no_time_warp; + time_offset -> + final; + NotSupArg when NotSupArg == os_monotonic_time_source; + NotSupArg == os_system_time_source; + NotSupArg == start_time -> + %% Cannot emulate this... + erlang:error(notsup, [NotSupArg]); + _ -> + erlang:error(badarg, [Item]) + end; + error:Error -> + erlang:error(Error, [Item]) + end. + +system_flag(Flag, Value) -> + try + erlang:system_flag(Flag, Value) + catch + error:Error -> + case {Error, Flag, Value} of + {badarg, time_offset, finalize} -> + %% Time offset is final + final; + _ -> + erlang:error(Error, [Flag, Value]) + end + end. + +%% +%% Internal functions +%% + +integer_time_unit(native) -> 1000*1000; +integer_time_unit(nano_seconds) -> 1000*1000*1000; +integer_time_unit(micro_seconds) -> 1000*1000; +integer_time_unit(milli_seconds) -> 1000; +integer_time_unit(seconds) -> 1; +integer_time_unit(I) when is_integer(I), I > 0 -> I; +integer_time_unit(BadRes) -> erlang:error(bad_time_unit, [BadRes]). + +erlang_system_time_fallback() -> + {MS, S, US} = erlang:now(), + (MS*1000000+S)*1000000+US. + +os_system_time_fallback() -> + {MS, S, US} = os:timestamp(), + (MS*1000000+S)*1000000+US. + +convert_time_unit_fallback(Time, FromUnit, ToUnit) -> + FU = integer_time_unit(FromUnit), + TU = integer_time_unit(ToUnit), + case Time < 0 of + true -> TU*Time - (FU - 1); + false -> TU*Time + end div FU. + +is_valid_modifier_list([positive|Ms]) -> + is_valid_modifier_list(Ms); +is_valid_modifier_list([monotonic|Ms]) -> + is_valid_modifier_list(Ms); +is_valid_modifier_list([]) -> + true; +is_valid_modifier_list(_) -> + false. diff --git a/erts/include/internal/ethr_internal.h b/erts/include/internal/ethr_internal.h index c9b1db5b46..65195145af 100644 --- a/erts/include/internal/ethr_internal.h +++ b/erts/include/internal/ethr_internal.h @@ -57,6 +57,33 @@ ETHR_PROTO_NORETURN__ ethr_abort__(void); int ethr_win_get_errno__(void); #endif +#ifdef ETHR_INCLUDE_MONOTONIC_CLOCK__ +#undef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME +#if defined(ETHR_HAVE_CLOCK_GETTIME_MONOTONIC) \ + || defined(ETHR_HAVE_MACH_CLOCK_GET_TIME) \ + || defined(ETHR_HAVE_GETHRTIME) +#ifdef ETHR_TIME_WITH_SYS_TIME +# include <time.h> +# include <sys/time.h> +#else +# ifdef ETHR_HAVE_SYS_TIME_H +# include <sys/time.h> +# else +# include <time.h> +# endif +#endif +#ifdef ETHR_HAVE_MACH_CLOCK_GET_TIME +#include <mach/clock.h> +#include <mach/mach.h> +#endif +#define ETHR_HAVE_ETHR_GET_MONOTONIC_TIME +ethr_sint64_t ethr_get_monotonic_time(void); +int ethr_get_monotonic_time_is_broken(void); +#endif +#endif /* ETHR_INCLUDE_MONOTONIC_CLOCK__ */ + +void ethr_init_event__(void); + /* implemented in lib_src/common/ethread_aux.c */ int ethr_init_common__(ethr_init_data *id); int ethr_late_init_common__(ethr_late_init_data *lid); diff --git a/erts/include/internal/ethread.h b/erts/include/internal/ethread.h index 0d9a4a4305..e598017ada 100644 --- a/erts/include/internal/ethread.h +++ b/erts/include/internal/ethread.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2004-2013. All Rights Reserved. + * Copyright Ericsson AB 2004-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -362,6 +362,9 @@ extern ethr_runtime_t ethr_runtime__; # include "sparc64/ethread.h" # endif # endif +# if ETHR_HAVE_GCC___ATOMIC_BUILTINS +# include "gcc/ethread.h" +# endif # include "libatomic_ops/ethread.h" # include "gcc/ethread.h" # endif diff --git a/erts/include/internal/ethread_header_config.h.in b/erts/include/internal/ethread_header_config.h.in index b36322490a..a9727568a2 100644 --- a/erts/include/internal/ethread_header_config.h.in +++ b/erts/include/internal/ethread_header_config.h.in @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2004-2011. All Rights Reserved. + * Copyright Ericsson AB 2004-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -84,32 +84,62 @@ /* Define if run in Sparc RMO, PSO, or TSO mode */ #undef ETHR_SPARC_RMO -/* Define if you have __sync_add_and_fetch() for 32-bit integers */ -#undef ETHR_HAVE___SYNC_ADD_AND_FETCH32 +/* Define as a boolean indicating whether you have a gcc compatible compiler + capable of generating the ARM DMB instruction, and are compiling for an ARM + processor with ARM DMB instruction support, or not */ +#undef ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION -/* Define if you have __sync_add_and_fetch() for 64-bit integers */ -#undef ETHR_HAVE___SYNC_ADD_AND_FETCH64 +/* Define as a bitmask corresponding to the word sizes that + __sync_synchronize() can handle on your system */ +#undef ETHR_HAVE___sync_synchronize -/* Define if you have __sync_fetch_and_and() for 32-bit integers */ -#undef ETHR_HAVE___SYNC_FETCH_AND_AND32 +/* Define as a bitmask corresponding to the word sizes that + __sync_add_and_fetch() can handle on your system */ +#undef ETHR_HAVE___sync_add_and_fetch -/* Define if you have __sync_fetch_and_and() for 64-bit integers */ -#undef ETHR_HAVE___SYNC_FETCH_AND_AND64 +/* Define as a bitmask corresponding to the word sizes that + __sync_fetch_and_and() can handle on your system */ +#undef ETHR_HAVE___sync_fetch_and_and -/* Define if you have __sync_fetch_and_or() for 32-bit integers */ -#undef ETHR_HAVE___SYNC_FETCH_AND_OR32 +/* Define as a bitmask corresponding to the word sizes that + __sync_fetch_and_or() can handle on your system */ +#undef ETHR_HAVE___sync_fetch_and_or -/* Define if you have __sync_fetch_and_or() for 64-bit integers */ -#undef ETHR_HAVE___SYNC_FETCH_AND_OR64 +/* Define as a bitmask corresponding to the word sizes that + __sync_val_compare_and_swap() can handle on your system */ +#undef ETHR_HAVE___sync_val_compare_and_swap -/* Define if you have __sync_val_compare_and_swap() for 32-bit integers */ -#undef ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP32 +/* Define as a boolean indicating whether you have a gcc __atomic builtins or + not */ +#undef ETHR_HAVE_GCC___ATOMIC_BUILTINS -/* Define if you have __sync_val_compare_and_swap() for 64-bit integers */ -#undef ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64 +/* Define as a boolean indicating whether you trust gcc's __atomic_* builtins + memory barrier implementations, or not */ +#undef ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS -/* Define if you have __sync_val_compare_and_swap() for 128-bit integers */ -#undef ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP128 +/* Define as a bitmask corresponding to the word sizes that __atomic_store_n() + can handle on your system */ +#undef ETHR_HAVE___atomic_store_n + +/* Define as a bitmask corresponding to the word sizes that __atomic_load_n() + can handle on your system */ +#undef ETHR_HAVE___atomic_load_n + +/* Define as a bitmask corresponding to the word sizes that + __atomic_add_fetch() can handle on your system */ +#undef ETHR_HAVE___atomic_add_fetch + +/* Define as a bitmask corresponding to the word sizes that + __atomic_fetch_and() can handle on your system */ +#undef ETHR_HAVE___atomic_fetch_and + +/* Define as a bitmask corresponding to the word sizes that + __atomic_fetch_or() can handle on your system */ +#undef ETHR_HAVE___atomic_fetch_or + +/* Define as a bitmask corresponding to the word sizes that + __atomic_compare_exchange_n() can handle on your system */ +#undef ETHR_HAVE___atomic_compare_exchange_n /* Define if you prefer gcc native ethread implementations */ #undef ETHR_PREFER_GCC_NATIVE_IMPLS @@ -237,3 +267,18 @@ /* Assumed cache-line size (in bytes) */ #undef ASSUMED_CACHE_LINE_SIZE + +/* Define if you have a clock_gettime() with a monotonic clock */ +#undef ETHR_HAVE_CLOCK_GETTIME_MONOTONIC + +/* Define if you have a monotonic gethrtime() */ +#undef ETHR_HAVE_GETHRTIME + +/* Define if you have a mach clock_get_time() with a monotonic clock */ +#undef ETHR_HAVE_MACH_CLOCK_GET_TIME + +/* Define to the monotonic clock id to use */ +#undef ETHR_MONOTONIC_CLOCK_ID + +/* Define if pthread_cond_timedwait() can be used with a monotonic clock */ +#undef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC diff --git a/erts/include/internal/ethread_inline.h b/erts/include/internal/ethread_inline.h index ffb756c84f..c09a67619a 100644 --- a/erts/include/internal/ethread_inline.h +++ b/erts/include/internal/ethread_inline.h @@ -20,6 +20,29 @@ #ifndef ETHREAD_INLINE_H__ #define ETHREAD_INLINE_H__ +#define ETHR_GCC_COMPILER_FALSE 0 /* Not a gcc compatible compiler */ +#define ETHR_GCC_COMPILER_TRUE 1 /* The GNU gcc compiler */ +/* Negative integers for gcc compatible compilers */ +#define ETHR_GCC_COMPILER_CLANG -1 /* The Clang gcc compatible compiler */ +#define ETHR_GCC_COMPILER_ICC -2 /* The Intel gcc compatible compiler */ +/* Line them up... */ + +/* + * Unfortunately there is no easy and certain way of + * detecting a true gcc compiler, since the compatible + * ones all define the same defines as the true gnu-gcc... + */ +#if !defined(__GNUC__) && !defined(__GNUG__) +# define ETHR_GCC_COMPILER ETHR_GCC_COMPILER_FALSE +#elif defined(__clang__) +# define ETHR_GCC_COMPILER ETHR_GCC_COMPILER_CLANG +#elif defined(__ICC) || defined(__INTEL_COMPILER) +# define ETHR_GCC_COMPILER ETHR_GCC_COMPILER_ICC +#else +/* Seems to be the true gnu-gcc... */ +# define ETHR_GCC_COMPILER ETHR_GCC_COMPILER_TRUE +#endif + #if !defined(__GNUC__) # define ETHR_AT_LEAST_GCC_VSN__(MAJ, MIN, PL) 0 #elif !defined(__GNUC_MINOR__) diff --git a/erts/include/internal/gcc/ethr_atomic.h b/erts/include/internal/gcc/ethr_atomic.h index f598f8537b..62eed78f76 100644 --- a/erts/include/internal/gcc/ethr_atomic.h +++ b/erts/include/internal/gcc/ethr_atomic.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2010-2011. All Rights Reserved. + * Copyright Ericsson AB 2010-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -18,78 +18,81 @@ */ /* - * Description: Native atomics ethread support using gcc's builtins + * Description: Native atomics ethread support using gcc's __atomic + * and __sync builtins * Author: Rickard Green + * + * Note: The C11 memory model implemented by gcc's __atomic + * builtins does not match the ethread API very well. + * + * Due to this we cannot use the __ATOMIC_SEQ_CST + * memory model. For more information see the comment + * in the begining of ethr_membar.h in this directory. */ #undef ETHR_INCLUDE_ATOMIC_IMPL__ -#if !defined(ETHR_GCC_ATOMIC32_H__) && defined(ETHR_ATOMIC_WANT_32BIT_IMPL__) -#define ETHR_GCC_ATOMIC32_H__ -#if defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP32) -# define ETHR_INCLUDE_ATOMIC_IMPL__ 4 -#endif +#if !defined(ETHR_GCC_ATOMIC_ATOMIC32_H__) \ + && defined(ETHR_ATOMIC_WANT_32BIT_IMPL__) \ + && ((ETHR_HAVE___sync_val_compare_and_swap & 4) \ + || (ETHR_HAVE___atomic_compare_exchange_n & 4)) + +#define ETHR_GCC_ATOMIC_ATOMIC32_H__ +#define ETHR_INCLUDE_ATOMIC_IMPL__ 4 #undef ETHR_ATOMIC_WANT_32BIT_IMPL__ -#elif !defined(ETHR_GCC_ATOMIC64_H__) && defined(ETHR_ATOMIC_WANT_64BIT_IMPL__) -#define ETHR_GCC_ATOMIC64_H__ -#if defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64) -# define ETHR_INCLUDE_ATOMIC_IMPL__ 8 -#endif -#undef ETHR_ATOMIC_WANT_64BIT_IMPL__ -#endif -#ifdef ETHR_INCLUDE_ATOMIC_IMPL__ +#elif !defined(ETHR_GCC_ATOMIC64_H__) \ + && defined(ETHR_ATOMIC_WANT_64BIT_IMPL__) \ + && ((ETHR_HAVE___sync_val_compare_and_swap & 8) \ + || (ETHR_HAVE___atomic_compare_exchange_n & 8)) -#ifndef ETHR_GCC_ATOMIC_COMMON__ -#define ETHR_GCC_ATOMIC_COMMON__ +#define ETHR_GCC_ATOMIC64_H__ +#define ETHR_INCLUDE_ATOMIC_IMPL__ 8 +#undef ETHR_ATOMIC_WANT_64BIT_IMPL__ -#define ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ 0 -#if defined(__i386__) || defined(__x86_64__) || defined(__sparc__) \ - || defined(__powerpc__) || defined(__ppc__) || defined(__mips__) -# undef ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ -# define ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ 1 #endif -#endif /* ETHR_GCC_ATOMIC_COMMON__ */ +#ifdef ETHR_INCLUDE_ATOMIC_IMPL__ #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 #define ETHR_HAVE_NATIVE_ATOMIC32 1 -#define ETHR_NATIVE_ATOMIC32_IMPL "gcc" #define ETHR_NATMC_FUNC__(X) ethr_native_atomic32_ ## X #define ETHR_ATMC_T__ ethr_native_atomic32_t #define ETHR_AINT_T__ ethr_sint32_t -#if defined(ETHR_HAVE___SYNC_ADD_AND_FETCH32) -# define ETHR_HAVE___SYNC_ADD_AND_FETCH -#endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_AND32) -# define ETHR_HAVE___SYNC_FETCH_AND_AND -#endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_OR32) -# define ETHR_HAVE___SYNC_FETCH_AND_OR +#if ((ETHR_HAVE___sync_val_compare_and_swap & 4) \ + && (ETHR_HAVE___atomic_compare_exchange_n & 4)) +# define ETHR_NATIVE_ATOMIC32_IMPL "gcc_atomic_and_sync_builtins" +#elif (ETHR_HAVE___atomic_compare_exchange_n & 4) +# define ETHR_NATIVE_ATOMIC32_IMPL "gcc_atomic_builtins" +#elif (ETHR_HAVE___sync_val_compare_and_swap & 4) +# define ETHR_NATIVE_ATOMIC32_IMPL "gcc_sync_builtins" +#else +# error "!?" #endif #elif ETHR_INCLUDE_ATOMIC_IMPL__ == 8 #define ETHR_HAVE_NATIVE_ATOMIC64 1 -#define ETHR_NATIVE_ATOMIC64_IMPL "gcc" #define ETHR_NATMC_FUNC__(X) ethr_native_atomic64_ ## X #define ETHR_ATMC_T__ ethr_native_atomic64_t #define ETHR_AINT_T__ ethr_sint64_t -#if defined(ETHR_HAVE___SYNC_ADD_AND_FETCH64) -# define ETHR_HAVE___SYNC_ADD_AND_FETCH -#endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_AND64) -# define ETHR_HAVE___SYNC_FETCH_AND_AND -#endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_OR64) -# define ETHR_HAVE___SYNC_FETCH_AND_OR +#if ((ETHR_HAVE___sync_val_compare_and_swap & 8) \ + && (ETHR_HAVE___atomic_compare_exchange_n & 8)) +# define ETHR_NATIVE_ATOMIC64_IMPL "gcc_atomic_and_sync_builtins" +#elif (ETHR_HAVE___atomic_compare_exchange_n & 8) +# define ETHR_NATIVE_ATOMIC64_IMPL "gcc_atomic_builtins" +#elif (ETHR_HAVE___sync_val_compare_and_swap & 8) +# define ETHR_NATIVE_ATOMIC64_IMPL "gcc_sync_builtins" +#else +# error "!?" #endif #else #error "Unsupported integer size" #endif +#undef ETHR_NATIVE_ATOMIC_IMPL__ + typedef struct { - volatile ETHR_AINT_T__ counter; + volatile ETHR_AINT_T__ value; } ETHR_ATMC_T__; - #if defined(ETHR_TRY_INLINE_FUNCS) || defined(ETHR_ATOMIC_IMPL__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 @@ -98,13 +101,19 @@ typedef struct { # define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_ADDR 1 #endif + static ETHR_INLINE ETHR_AINT_T__ * ETHR_NATMC_FUNC__(addr)(ETHR_ATMC_T__ *var) { - return (ETHR_AINT_T__ *) &var->counter; + return (ETHR_AINT_T__ *) &var->value; } -#if ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ +/* + * set() + */ +#if (ETHR_HAVE___atomic_store_n & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_SET 1 @@ -115,12 +124,109 @@ ETHR_NATMC_FUNC__(addr)(ETHR_ATMC_T__ *var) static ETHR_INLINE void ETHR_NATMC_FUNC__(set)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ value) { - var->counter = value; + __atomic_store_n(&var->value, value, __ATOMIC_RELAXED); } -#endif /* ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ */ +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ -#if ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ +#if (ETHR_GCC_RELB_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_SET_RELB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_SET_RELB 1 +#endif + +static ETHR_INLINE void +ETHR_NATMC_FUNC__(set_relb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ value) +{ + __atomic_store_n(&var->value, value, __ATOMIC_RELEASE); +} + +#endif /* ETHR_GCC_RELB_VERSIONS__ */ + +#elif (ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_SET 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_SET 1 +#endif + +static ETHR_INLINE void +ETHR_NATMC_FUNC__(set)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ value) +{ + var->value = value; +} + +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ + +#if (ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELB_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_SET_RELB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_SET_RELB 1 +#endif + +static ETHR_INLINE void +ETHR_NATMC_FUNC__(set_relb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ value) +{ + var->value = value; +} + +#endif /* ETHR_GCC_RELB_VERSIONS__ */ + +#endif /* ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ */ + +#endif /* ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ */ + +/* + * read() + */ + +#if (ETHR_HAVE___atomic_load_n & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_READ 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_READ 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(read)(ETHR_ATMC_T__ *var) +{ + return __atomic_load_n(&var->value, __ATOMIC_RELAXED); +} + +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ + +#if ((ETHR_GCC_ACQB_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) \ + & ~ETHR___atomic_load_ACQUIRE_barrier_bug) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_READ_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_READ_ACQB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(read_acqb)(ETHR_ATMC_T__ *var) +{ + return __atomic_load_n(&var->value, __ATOMIC_ACQUIRE); +} + +#endif /* ETHR_GCC_ACQB_VERSIONS__ */ + +#elif (ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_READ 1 @@ -131,12 +237,90 @@ ETHR_NATMC_FUNC__(set)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ value) static ETHR_INLINE ETHR_AINT_T__ ETHR_NATMC_FUNC__(read)(ETHR_ATMC_T__ *var) { - return var->counter; + return var->value; +} + +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ + +#if (ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_ACQB_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_READ_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_READ_ACQB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(read_acqb)(ETHR_ATMC_T__ *var) +{ + return var->value; +} + +#endif /* ETHR_GCC_ACQB_VERSIONS__ */ + +#endif /* ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ */ + +#endif /* ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ */ + +/* + * add_return() + */ +#if (ETHR_HAVE___atomic_add_fetch & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_ADD_RETURN 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_ADD_RETURN 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(add_return)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ incr) +{ + return __atomic_add_fetch(&var->value, incr, __ATOMIC_RELAXED); } -#endif /* ETHR_READ_AND_SET_WITHOUT_SYNC_OP__ */ +#endif /* ETHR_GCC_RELAXED_MOD_VERSIONS__ */ -#if defined(ETHR_HAVE___SYNC_ADD_AND_FETCH) +#if (ETHR_GCC_ACQB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_ADD_RETURN_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_ADD_RETURN_ACQB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(add_return_acqb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ incr) +{ + return __atomic_add_fetch(&var->value, incr, __ATOMIC_ACQUIRE); +} + +#endif /* ETHR_GCC_ACQB_MOD_VERSIONS__ */ + +#if (ETHR_GCC_RELB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_ADD_RETURN_RELB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_ADD_RETURN_RELB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(add_return_relb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ incr) +{ + return __atomic_add_fetch(&var->value, incr, __ATOMIC_RELEASE); +} + +#endif /* ETHR_GCC_RELB_MOD_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_add_fetch */ + +#if ((ETHR_HAVE___sync_add_and_fetch & ETHR_INCLUDE_ATOMIC_IMPL__) \ + & ETHR_GCC_MB_MOD_VERSIONS__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_ADD_RETURN_MB 1 @@ -147,12 +331,68 @@ ETHR_NATMC_FUNC__(read)(ETHR_ATMC_T__ *var) static ETHR_INLINE ETHR_AINT_T__ ETHR_NATMC_FUNC__(add_return_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ incr) { - return __sync_add_and_fetch(&var->counter, incr); + return __sync_add_and_fetch(&var->value, incr); +} + +#endif /* ETHR_HAVE___sync_add_and_fetch */ + +/* + * and_retold() + */ +#if (ETHR_HAVE___atomic_fetch_and & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_AND_RETOLD 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_AND_RETOLD 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(and_retold)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_and(&var->value, mask, __ATOMIC_RELAXED); +} + +#endif /* ETHR_GCC_RELAXED_MOD_VERSIONS__ */ + +#if (ETHR_GCC_ACQB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_AND_RETOLD_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_AND_RETOLD_ACQB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(and_retold_acqb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_and(&var->value, mask, __ATOMIC_ACQUIRE); } +#endif /* ETHR_GCC_ACQB_MOD_VERSIONS__ */ + +#if (ETHR_GCC_RELB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_AND_RETOLD_RELB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_AND_RETOLD_RELB 1 #endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_AND) +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(and_retold_relb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_and(&var->value, mask, __ATOMIC_RELEASE); +} + +#endif /* ETHR_GCC_RELB_MOD_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_fetch_and */ + +#if ((ETHR_HAVE___sync_fetch_and_and & ETHR_INCLUDE_ATOMIC_IMPL__) \ + & ETHR_GCC_MB_MOD_VERSIONS__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_AND_RETOLD_MB 1 @@ -163,12 +403,68 @@ ETHR_NATMC_FUNC__(add_return_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ incr) static ETHR_INLINE ETHR_AINT_T__ ETHR_NATMC_FUNC__(and_retold_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) { - return __sync_fetch_and_and(&var->counter, mask); + return __sync_fetch_and_and(&var->value, mask); +} + +#endif /* ETHR_HAVE___sync_fetch_and_and */ + +/* + * or_retold() + */ +#if (ETHR_HAVE___atomic_fetch_or & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_OR_RETOLD 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_OR_RETOLD 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(or_retold)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_or(&var->value, mask, __ATOMIC_RELAXED); +} + +#endif /* ETHR_GCC_RELAXED_MOD_VERSIONS__ */ + +#if (ETHR_GCC_ACQB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_OR_RETOLD_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_OR_RETOLD_ACQB 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(or_retold_acqb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_or(&var->value, mask, __ATOMIC_ACQUIRE); } +#endif /* ETHR_GCC_ACQB_MOD_VERSIONS__ */ + +#if (ETHR_GCC_RELB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_OR_RETOLD_RELB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_OR_RETOLD_RELB 1 #endif -#if defined(ETHR_HAVE___SYNC_FETCH_AND_OR) +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(or_retold_relb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) +{ + return __atomic_fetch_or(&var->value, mask, __ATOMIC_RELEASE); +} + +#endif /* ETHR_GCC_RELB_MOD_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_fetch_or */ + +#if ((ETHR_HAVE___sync_fetch_and_or & ETHR_INCLUDE_ATOMIC_IMPL__) \ + & ETHR_GCC_MB_MOD_VERSIONS__) #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_OR_RETOLD_MB 1 @@ -179,11 +475,73 @@ ETHR_NATMC_FUNC__(and_retold_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) static ETHR_INLINE ETHR_AINT_T__ ETHR_NATMC_FUNC__(or_retold_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ mask) { - return (ETHR_AINT_T__) __sync_fetch_and_or(&var->counter, mask); + return (ETHR_AINT_T__) __sync_fetch_and_or(&var->value, mask); +} + +#endif /* ETHR_HAVE___sync_fetch_and_or */ + +/* + * cmpxchg() + */ +#if (ETHR_HAVE___atomic_compare_exchange_n & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if (ETHR_GCC_RELAXED_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_CMPXCHG 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_CMPXCHG 1 +#endif + +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(cmpxchg)(ETHR_ATMC_T__ *var, + ETHR_AINT_T__ new, + ETHR_AINT_T__ exp) +{ + ETHR_AINT_T__ xchg = exp; + if (__atomic_compare_exchange_n(&var->value, + &xchg, + new, + 0, /* No spurious failures, please */ + __ATOMIC_RELAXED, + __ATOMIC_RELAXED)) + return exp; + return xchg; } +#endif /* ETHR_GCC_RELAXED_MOD_VERSIONS__ */ + +#if (ETHR_GCC_ACQB_MOD_VERSIONS__ & ETHR_INCLUDE_ATOMIC_IMPL__) + +#if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_CMPXCHG_ACQB 1 +#else +# define ETHR_HAVE_ETHR_NATIVE_ATOMIC64_CMPXCHG_ACQB 1 #endif +static ETHR_INLINE ETHR_AINT_T__ +ETHR_NATMC_FUNC__(cmpxchg_acqb)(ETHR_ATMC_T__ *var, + ETHR_AINT_T__ new, + ETHR_AINT_T__ exp) +{ + ETHR_AINT_T__ xchg = exp; + if (__atomic_compare_exchange_n(&var->value, + &xchg, + new, + 0, /* No spurious failures, please */ + __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE)) + return exp; + return xchg; +} + +#endif /* ETHR_GCC_ACQB_MOD_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_compare_exchange_n */ + +#if ((ETHR_HAVE___sync_val_compare_and_swap & ETHR_INCLUDE_ATOMIC_IMPL__) \ + & ETHR_GCC_MB_MOD_VERSIONS__) + #if ETHR_INCLUDE_ATOMIC_IMPL__ == 4 # define ETHR_HAVE_ETHR_NATIVE_ATOMIC32_CMPXCHG_MB 1 #else @@ -195,17 +553,16 @@ ETHR_NATMC_FUNC__(cmpxchg_mb)(ETHR_ATMC_T__ *var, ETHR_AINT_T__ new, ETHR_AINT_T__ old) { - return __sync_val_compare_and_swap(&var->counter, old, new); + return __sync_val_compare_and_swap(&var->value, old, new); } +#endif /* ETHR_HAVE___sync_val_compare_and_swap */ + #endif /* ETHR_TRY_INLINE_FUNCS */ #undef ETHR_NATMC_FUNC__ #undef ETHR_ATMC_T__ #undef ETHR_AINT_T__ #undef ETHR_AINT_SUFFIX__ -#undef ETHR_HAVE___SYNC_ADD_AND_FETCH -#undef ETHR_HAVE___SYNC_FETCH_AND_AND -#undef ETHR_HAVE___SYNC_FETCH_AND_OR #endif diff --git a/erts/include/internal/gcc/ethr_dw_atomic.h b/erts/include/internal/gcc/ethr_dw_atomic.h index 6736f9c547..c2c8f85b7b 100644 --- a/erts/include/internal/gcc/ethr_dw_atomic.h +++ b/erts/include/internal/gcc/ethr_dw_atomic.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2011. All Rights Reserved. + * Copyright Ericsson AB 2011-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -18,35 +18,39 @@ */ /* - * Description: Native double word atomics using gcc's builtins + * Description: Native double word atomics using gcc's __atomic + * and __sync builtins * Author: Rickard Green + * + * Note: The C11 memory model implemented by gcc's __atomic + * builtins does not match the ethread API very well. + * + * Due to this we cannot use the __ATOMIC_SEQ_CST + * memory model. For more information see the comment + * in the begining of ethr_membar.h in this directory. */ #undef ETHR_INCLUDE_DW_ATOMIC_IMPL__ -#ifndef ETHR_GCC_DW_ATOMIC_H__ -# define ETHR_GCC_DW_ATOMIC_H__ -# if ((ETHR_SIZEOF_PTR == 4 \ - && defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64)) \ - || (ETHR_SIZEOF_PTR == 8 \ - && defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP128) \ - && defined(ETHR_HAVE_INT128_T))) -# define ETHR_INCLUDE_DW_ATOMIC_IMPL__ -# endif +#if !defined(ETHR_GCC_ATOMIC_DW_ATOMIC_H__) \ + && ((ETHR_HAVE___sync_val_compare_and_swap & (2*ETHR_SIZEOF_PTR)) \ + || (ETHR_HAVE___atomic_compare_exchange_n & (2*ETHR_SIZEOF_PTR))) +# define ETHR_GCC_ATOMIC_DW_ATOMIC_H__ +# define ETHR_INCLUDE_DW_ATOMIC_IMPL__ #endif #ifdef ETHR_INCLUDE_DW_ATOMIC_IMPL__ # define ETHR_HAVE_NATIVE_SU_DW_ATOMIC -# define ETHR_NATIVE_DW_ATOMIC_IMPL "gcc" -# if defined(__i386__) || defined(__x86_64__) -/* - * If ETHR_RTCHK_USE_NATIVE_DW_ATOMIC_IMPL__ is defined, it will be used - * at runtime in order to determine if native or fallback implementation - * should be used. - */ -# define ETHR_RTCHK_USE_NATIVE_DW_ATOMIC_IMPL__ \ - ETHR_X86_RUNTIME_CONF_HAVE_DW_CMPXCHG__ -# endif +#if ((ETHR_HAVE___sync_val_compare_and_swap & (2*ETHR_SIZEOF_PTR)) \ + && (ETHR_HAVE___atomic_compare_exchange_n & (2*ETHR_SIZEOF_PTR))) +# define ETHR_NATIVE_DW_ATOMIC_IMPL "gcc_atomic_and_sync_builtins" +#elif (ETHR_HAVE___atomic_compare_exchange_n & (2*ETHR_SIZEOF_PTR)) +# define ETHR_NATIVE_DW_ATOMIC_IMPL "gcc_atomic_builtins" +#elif (ETHR_HAVE___sync_val_compare_and_swap & (2*ETHR_SIZEOF_PTR)) +# define ETHR_NATIVE_DW_ATOMIC_IMPL "gcc_sync_builtins" +#else +# error "!?" +#endif # if ETHR_SIZEOF_PTR == 4 # define ETHR_DW_NATMC_ALIGN_MASK__ 0x7 @@ -89,15 +93,138 @@ typedef union { # define ETHR_DW_DBG_ALIGNED__(PTR) # endif -#define ETHR_HAVE_ETHR_NATIVE_DW_ATOMIC_ADDR + +#define ETHR_HAVE_ETHR_NATIVE_DW_ATOMIC_ADDR 1 + static ETHR_INLINE ethr_sint_t * ethr_native_dw_atomic_addr(ethr_native_dw_atomic_t *var) { return (ethr_sint_t *) ETHR_DW_NATMC_MEM__(var); } +#if (ETHR_HAVE___atomic_store_n & (2*ETHR_SIZEOF_PTR)) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_SET 1 + +static ETHR_INLINE void +ethr_native_su_dw_atomic_set(ethr_native_dw_atomic_t *var, + ETHR_NATIVE_SU_DW_SINT_T value) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_DW_DBG_ALIGNED__(p); + __atomic_store_n(p, value, __ATOMIC_RELAXED); +} + +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ + +#if (ETHR_GCC_RELB_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) -#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_CMPXCHG_MB +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_SET_RELB 1 + +static ETHR_INLINE void +ethr_native_su_dw_atomic_set_relb(ethr_native_dw_atomic_t *var, + ETHR_NATIVE_SU_DW_SINT_T value) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_DW_DBG_ALIGNED__(p); + __atomic_store_n(p, value, __ATOMIC_RELEASE); +} + +#endif /* ETHR_GCC_RELB_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_store_n */ + +#if (ETHR_HAVE___atomic_load_n & (2*ETHR_SIZEOF_PTR)) + +#if (ETHR_GCC_RELAXED_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_READ 1 + +static ETHR_INLINE ETHR_NATIVE_SU_DW_SINT_T +ethr_native_su_dw_atomic_read(ethr_native_dw_atomic_t *var) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_DW_DBG_ALIGNED__(p); + return __atomic_load_n(p, __ATOMIC_RELAXED); +} + +#endif /* ETHR_GCC_RELAXED_VERSIONS__ */ + +#if ((ETHR_GCC_ACQB_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) \ + & ~ETHR___atomic_load_ACQUIRE_barrier_bug) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_READ_ACQB 1 + +static ETHR_INLINE ETHR_NATIVE_SU_DW_SINT_T +ethr_native_su_dw_atomic_read_acqb(ethr_native_dw_atomic_t *var) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_DW_DBG_ALIGNED__(p); + return __atomic_load_n(p, __ATOMIC_ACQUIRE); +} + +#endif /* ETHR_GCC_ACQB_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_load_n */ + +#if (ETHR_HAVE___atomic_compare_exchange_n & (2*ETHR_SIZEOF_PTR)) + +#if (ETHR_GCC_RELAXED_MOD_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_CMPXCHG 1 + +static ETHR_INLINE ETHR_NATIVE_SU_DW_SINT_T +ethr_native_su_dw_atomic_cmpxchg(ethr_native_dw_atomic_t *var, + ETHR_NATIVE_SU_DW_SINT_T new, + ETHR_NATIVE_SU_DW_SINT_T exp) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_NATIVE_SU_DW_SINT_T xchg = exp; + ETHR_DW_DBG_ALIGNED__(p); + if (__atomic_compare_exchange_n(p, + &xchg, + new, + 0, + __ATOMIC_RELAXED, + __ATOMIC_RELAXED)) + return exp; + return xchg; +} + +#endif /* ETHR_GCC_RELAXED_MOD_VERSIONS__ */ + +#if (ETHR_GCC_ACQB_MOD_VERSIONS__ & (2*ETHR_SIZEOF_PTR)) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_CMPXCHG_ACQB 1 + +static ETHR_INLINE ETHR_NATIVE_SU_DW_SINT_T +ethr_native_su_dw_atomic_cmpxchg_acqb(ethr_native_dw_atomic_t *var, + ETHR_NATIVE_SU_DW_SINT_T new, + ETHR_NATIVE_SU_DW_SINT_T exp) +{ + ethr_native_dw_ptr_t p = (ethr_native_dw_ptr_t) ETHR_DW_NATMC_MEM__(var); + ETHR_NATIVE_SU_DW_SINT_T xchg = exp; + ETHR_DW_DBG_ALIGNED__(p); + if (__atomic_compare_exchange_n(p, + &xchg, + new, + 0, + __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE)) + return exp; + return xchg; +} + +#endif /* ETHR_GCC_ACQB_MOD_VERSIONS__ */ + +#endif /* ETHR_HAVE___atomic_compare_exchange_n */ + +#if ((ETHR_HAVE___sync_val_compare_and_swap & (2*ETHR_SIZEOF_PTR)) \ + & ETHR_GCC_MB_MOD_VERSIONS__) + +#define ETHR_HAVE_ETHR_NATIVE_SU_DW_ATOMIC_CMPXCHG_MB 1 static ETHR_INLINE ETHR_NATIVE_SU_DW_SINT_T ethr_native_su_dw_atomic_cmpxchg_mb(ethr_native_dw_atomic_t *var, @@ -109,7 +236,8 @@ ethr_native_su_dw_atomic_cmpxchg_mb(ethr_native_dw_atomic_t *var, return __sync_val_compare_and_swap(p, old, new); } -#endif /* ETHR_TRY_INLINE_FUNCS */ +#endif /* ETHR_HAVE___sync_val_compare_and_swap */ -#endif /* ETHR_GCC_DW_ATOMIC_H__ */ +#endif /* ETHR_TRY_INLINE_FUNCS */ +#endif /* ETHR_INCLUDE_DW_ATOMIC_IMPL__ */ diff --git a/erts/include/internal/gcc/ethr_membar.h b/erts/include/internal/gcc/ethr_membar.h index 7d428fc68e..d2d36907f3 100644 --- a/erts/include/internal/gcc/ethr_membar.h +++ b/erts/include/internal/gcc/ethr_membar.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2011. All Rights Reserved. + * Copyright Ericsson AB 2011-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -18,56 +18,196 @@ */ /* - * Description: Memory barriers when using gcc's builtins + * Description: Memory barriers when using gcc's __atomic and + * __sync builtins * Author: Rickard Green + * + * Note: The C11 memory model implemented by gcc's __atomic + * builtins does not match the ethread API very well. + * + * A function with a barrier postfix in the ethread atomic + * API needs to ensure that all stores and loads are + * ordered around it according to the semantics of the + * barrier specified. + * + * The C11 aproch is different. The __atomic builtins + * API takes a memory model parameter. Assuming that all + * memory syncronizations using the involved atomic + * variables are made using this API, the synchronizations + * will adhere to the memory models used. That is, you do + * *not* know how loads and stores will be ordered around + * a specific __atomic operation in the general case. You + * only know the total effect of the combination of + * operations issued will adhere to the model. + * + * This limits how we can use the __atomic builtins. What + * we cannot use: + * + * 1. We cannot rely on __atomic_thread_fence() to issue + * any specific memory barriers at all. This regardless + * of memory model parameter passed. That is, we cannot + * use the __atomic_thread_fence() builtin at all. + * + * Why is this? If all __atomic builtins accessing + * memory issue memory barriers, __atomic_thread_fence() + * does not have to issue memory barriers. The + * implementation for the Itanium architecture is an + * example of this. Even using the __ATOMIC_RELAXED + * memory model all __atomic builtins accessing memory + * will issue memory barriers. Due to this no memory + * barriers at all will be issued by + * __atomic_thread_fence() using either one of the + * __ATOMIC_CONSUME, __ATOMIC_ACQUIRE, or + * __ATOMIC_RELEASE memory models. + * + * 2. We cannot rely on any __atomic builtin with the + * __ATOMIC_SEQ_CST memory model parameters to + * issue any specific memory barriers. That is, we + * cannot use these memory models at all. + * + * Why is this? Since all synchronizations is expected + * to be made using the __atomic builtins, memory + * barriers only have to be issued by some of them, + * and you do not know which ones wont issue memory + * barriers. + * + * One can easily be fooled into believing that when + * using the __ATOMIC_SEQ_CST memory model on all + * operations, all operations will issue full memory + * barriers. This is however not the case. The + * implementation for the x86_64 architecture is an + * example of this. Since all operations except loads + * issue full memory barriers, no memory barriers at + * all is issued by loads. This could also be + * implemented by issuing a full memory barrier on + * loads, but no barrier at all on stores. + * + * What can be used then? + * 1. All (legacy) __sync builtins implying full memory + * barriers issued. + * 2. All __atomic builtins using the __ATOMIC_RELAXED + * memory model can, of course, be used. This since + * no ordering guarantees at all are made. + * 3. All __atomic builtins accessing memory using the + * __ATOMIC_ACQUIRE and __ATOMIC_RELEASE memory + * models. This since an __atomic builtin memory + * access using the __ATOMIC_ACQUIRE must at least + * issue an aquire memory barrier and an __atomic + * builtin memory acess with the __ATOMIC_RELEASE + * memory model must at least issue a release memory + * barrier. Otherwise the two can not be paired. + * 4. All __atomic builtins accessing memory using the + * __ATOMIC_CONSUME builtin can be used for the same + * reason __ATOMIC_ACQUIRE can be used. The ethread + * atomic framework implementing the ethread API + * using native implementations does not expect the + * native implementations to produce versions with + * data dependent read barriers, so until the + * framework is changed we haven't got any use for + * for it. + * + * For some architectures we have our own memory barrier + * implementations. We prefer to use these since they + * should be as fine grained as possible. For other + * architectures we use the __sync_synchronize() builtin + * which issue a full memory barrier. For these + * architectures we have to assume that all loads and + * stores can be reordered without limitation. That is, + * unnecessary memory barriers will be issued if such + * reordering actually cannot occur. */ -#ifndef ETHR_GCC_MEMBAR_H__ -#define ETHR_GCC_MEMBAR_H__ +/* + * We prefer to use our own memory barrier implementation if + * such exist instead of using __sync_synchronize()... + */ +#if defined(__i386__) || defined(__x86_64__) +# include "../i386/ethr_membar.h" +#elif defined(__sparc__) +# include "../sparc32/ethr_membar.h" +#elif defined(__powerpc__) || defined(__ppc__) || defined(__powerpc64__) +# include "../ppc32/ethr_membar.h" +#elif !defined(ETHR_GCC_ATOMIC_MEMBAR_H__) \ + && (ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION \ + || ETHR_HAVE___sync_synchronize \ + || (ETHR_HAVE___sync_val_compare_and_swap & 12)) +#define ETHR_GCC_ATOMIC_MEMBAR_H__ #define ETHR_LoadLoad (1 << 0) #define ETHR_LoadStore (1 << 1) #define ETHR_StoreLoad (1 << 2) #define ETHR_StoreStore (1 << 3) +#define ETHR_COMPILER_BARRIER __asm__ __volatile__("" : : : "memory") + +#if ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION + +static __inline__ __attribute__((__always_inline__)) void +ethr_full_fence__(void) +{ + __asm__ __volatile__("dmb sy" : : : "memory"); +} + +static __inline__ __attribute__((__always_inline__)) void +ethr_store_fence__(void) +{ + __asm__ __volatile__("dmb st" : : : "memory"); +} + +#define ETHR_MEMBAR(B) \ + ETHR_CHOOSE_EXPR((B) == ETHR_StoreStore, ethr_store_fence__(), ethr_full_fence__()) + +#elif ETHR_HAVE___sync_synchronize + +static __inline__ __attribute__((__always_inline__)) void +ethr_full_fence__(void) +{ + /* + * The compiler barriers are here to fix missing clobbers + * in __sync_synchronize() when using buggy LLVM + * implementation of __sync_synchronize(). They + * do not introduce any unnecessary overhead when used + * here, so we use them for all systems. + */ + ETHR_COMPILER_BARRIER; + __sync_synchronize(); + ETHR_COMPILER_BARRIER; +} + +#else /* !ETHR_HAVE___sync_synchronize */ + /* - * According to the documentation __sync_synchronize() will - * issue a full memory barrier. However, __sync_synchronize() - * is known to erroneously be a noop on at least some - * platforms with some gcc versions. This has suposedly been - * fixed in some gcc version, but we don't know from which - * version. Therefore, we only use it when it has been - * verified to work. Otherwise we use the workaround - * below. + * Buggy __sync_synchronize(); call __sync_val_compare_and_swap() + * instead which imply a full memory barrier (and hope that one + * isn't buggy too). */ -#if defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP32) +#if (ETHR_HAVE___sync_val_compare_and_swap & 4) # define ETHR_MB_T__ ethr_sint32_t -#elif defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64) +#elif (ETHR_HAVE___sync_val_compare_and_swap & 8) # define ETHR_MB_T__ ethr_sint64_t -#else -# error "No __sync_val_compare_and_swap" #endif -#define ETHR_SYNC_SYNCHRONIZE_WORKAROUND__ \ -do { \ - volatile ETHR_MB_T__ x___ = 0; \ - (void) __sync_val_compare_and_swap(&x___, (ETHR_MB_T__) 0, (ETHR_MB_T__) 1); \ -} while (0) -#define ETHR_COMPILER_BARRIER __asm__ __volatile__("" : : : "memory") +static __inline__ __attribute__((__always_inline__)) void +ethr_full_fence__(void) +{ + volatile ETHR_MB_T__ x = 0; + (void) __sync_val_compare_and_swap(&x, (ETHR_MB_T__) 0, (ETHR_MB_T__) 1); +} -#if defined(__mips__) && ETHR_AT_LEAST_GCC_VSN__(4, 2, 0) -# define ETHR_MEMBAR(B) __sync_synchronize() -# define ETHR_READ_DEPEND_MEMORY_BARRIER __sync_synchronize() -#elif ((defined(__powerpc__) || defined(__ppc__)) \ - && ETHR_AT_LEAST_GCC_VSN__(4, 1, 2)) -# define ETHR_MEMBAR(B) __sync_synchronize() -#else /* Use workaround */ -# define ETHR_MEMBAR(B) \ - ETHR_SYNC_SYNCHRONIZE_WORKAROUND__ -# define ETHR_READ_DEPEND_MEMORY_BARRIER \ - ETHR_SYNC_SYNCHRONIZE_WORKAROUND__ +#endif /* !ETHR_HAVE___sync_synchronize */ + +#ifndef ETHR_MEMBAR +# define ETHR_MEMBAR(B) ethr_full_fence__() #endif +/* + * Define ETHR_READ_DEPEND_MEMORY_BARRIER for all architechtures + * not known to order data dependent loads + */ + +#if !defined(__ia64__) && !defined(__arm__) +# define ETHR_READ_DEPEND_MEMORY_BARRIER ETHR_MEMBAR(ETHR_LoadLoad) +#endif -#endif /* ETHR_GCC_MEMBAR_H__ */ +#endif /* ETHR_GCC_ATOMIC_MEMBAR_H__ */ diff --git a/erts/include/internal/gcc/ethread.h b/erts/include/internal/gcc/ethread.h index 365a3535cf..be3e1da90e 100644 --- a/erts/include/internal/gcc/ethread.h +++ b/erts/include/internal/gcc/ethread.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2010-2011. All Rights Reserved. + * Copyright Ericsson AB 2010-2015. All Rights Reserved. * * The contents of this file are subject to the Erlang Public License, * Version 1.1, (the "License"); you may not use this file except in @@ -18,20 +18,292 @@ */ /* - * Description: Native atomic ethread support when using gcc + * Description: Native atomic ethread support when using gcc's __atomic + * and __sync builtins * Author: Rickard Green */ -#ifndef ETHREAD_GCC_H__ -#define ETHREAD_GCC_H__ - -#if defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP32) \ - || defined(ETHR_HAVE___SYNC_VAL_COMPARE_AND_SWAP64) +#if !defined(ETHREAD_GCC_NATIVE_H__) && ETHR_GCC_COMPILER +#define ETHREAD_GCC_NATIVE_H__ #ifndef ETHR_MEMBAR # include "ethr_membar.h" #endif +#define ETHR_GCC_VERSIONS_MASK__ 28 + +#undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ +#undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ +#undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ +#undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ +#undef ETHR_GCC_RELAXED_VERSIONS__ +#undef ETHR_GCC_RELAXED_MOD_VERSIONS__ +#undef ETHR_GCC_ACQB_VERSIONS__ +#undef ETHR_GCC_ACQB_MOD_VERSIONS__ +#undef ETHR_GCC_RELB_VERSIONS__ +#undef ETHR_GCC_RELB_MOD_VERSIONS__ +#undef ETHR_GCC_MB_MOD_VERSIONS__ + +/* + * True GNU GCCs before version 4.8 do not emit a memory barrier + * after the load in the __atomic_load_n(_, __ATOMIC_ACQUIRE) + * case (which is needed on most architectures). + */ +#undef ETHR___atomic_load_ACQUIRE_barrier_bug +#if ETHR_GCC_COMPILER != ETHR_GCC_COMPILER_TRUE +/* + * A gcc compatible compiler. We have no information + * about the existence of this bug, but we assume + * that it is not impossible that it could have + * been "inherited". Therefore, until we are certain + * that the bug does not exist, we assume that it + * does. + */ +# define ETHR___atomic_load_ACQUIRE_barrier_bug ETHR_GCC_VERSIONS_MASK__ +#elif !ETHR_AT_LEAST_GCC_VSN__(4, 8, 0) +/* True gcc of version < 4.8, i.e., bug exist... */ +# define ETHR___atomic_load_ACQUIRE_barrier_bug ETHR_GCC_VERSIONS_MASK__ +#else /* True gcc of version >= 4.8 */ +/* + * Sizes less than or equal to word size have been fixed, + * but double word size has not been fixed. + */ +# if ETHR_SIZEOF_PTR == 8 +# define ETHR___atomic_load_ACQUIRE_barrier_bug \ + (~(8|4) & ETHR_GCC_VERSIONS_MASK__) +# elif ETHR_SIZEOF_PTR == 4 +# define ETHR___atomic_load_ACQUIRE_barrier_bug \ + (~4 & ETHR_GCC_VERSIONS_MASK__) +# else +# error word size not supported +# endif +#endif + +#define ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ 0 +#define ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ 0 +#define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ 0 +#define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ 0 +#define ETHR_GCC_RELAXED_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +#define ETHR_GCC_RELAXED_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ + +#if ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS +# define ETHR_GCC_ACQB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# define ETHR_GCC_ACQB_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# define ETHR_GCC_RELB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# define ETHR_GCC_RELB_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +#else +/* + * This is currently the default (on most platforms) since + * we've seen too many memory barrier bugs produced by gcc... + */ +# define ETHR_GCC_ACQB_VERSIONS__ 0 +# define ETHR_GCC_ACQB_MOD_VERSIONS__ 0 +# define ETHR_GCC_RELB_VERSIONS__ 0 +# define ETHR_GCC_RELB_MOD_VERSIONS__ 0 +#endif +/* + * In the general case we do not want any full barrier versions + * if we can implement more relaxed ones (using __atomic_* builtins). + * This since the implementations normally need extra memory barrier + * instructions to implement these. The x86/x86_64 implementations + * are an exception see below. + */ +#define ETHR_GCC_MB_MOD_VERSIONS__ \ + (ETHR_GCC_VERSIONS_MASK__ & ~ETHR_HAVE___atomic_compare_exchange_n) + +#if ETHR_SIZEOF_PTR == 8 +# define ETHR_GCC_VOLATILE_BIT_MASK__ 12 +#elif ETHR_SIZEOF_PTR == 4 +# define ETHR_GCC_VOLATILE_BIT_MASK__ 4 +#endif + +#if defined(__i386__) || defined(__x86_64__) || defined(__sparc__) \ + || defined(__powerpc__) || defined(__ppc__) || defined(__mips__) \ + || defined(__alpha__) || defined(__ia64__) + +/* + * Aligned volatile stores and loads of data smaller + * than or equal to word size are atomic... + */ +# undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ +# define ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ ETHR_GCC_VOLATILE_BIT_MASK__ +# undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ +# define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ ETHR_GCC_VOLATILE_BIT_MASK__ + +#elif defined(__arm__) + +/* volatile stores are problematic on some machines... */ +# undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ +# define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ ETHR_GCC_VOLATILE_BIT_MASK__ + +#endif + +#if defined(__ia64__) + +/* Volatile stores produce stores with release barriers. */ +# undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ +# define ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ ETHR_GCC_VOLATILE_BIT_MASK__ + +/* Volatile loads produce loads with acquire barrier. */ +# undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ +# define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ ETHR_GCC_VOLATILE_BIT_MASK__ + +/* + * We trust gcc to produce acquire/release barriers on itanium. + * Since all atomic ops also have at least acquire or release + * barriers (also when passed the relaxed memory model) it + * would be very inefficient not to use these as native + * barriers on Itanium. + */ +# undef ETHR_GCC_ACQB_VERSIONS__ +# define ETHR_GCC_ACQB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# undef ETHR_GCC_ACQB_MOD_VERSIONS__ +# define ETHR_GCC_ACQB_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# undef ETHR_GCC_RELB_VERSIONS__ +# define ETHR_GCC_RELB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ +# undef ETHR_GCC_RELB_MOD_VERSIONS__ +# define ETHR_GCC_RELB_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ + +/* + * Itanium is not effected by the load acquire + * bug since the barrier is part of the instruction + * on Itanium (ld.acq), and not a separate instruction + * as on most platforms. + */ +# undef ETHR___atomic_load_ACQUIRE_barrier_bug +# define ETHR___atomic_load_ACQUIRE_barrier_bug 0 + +/* + * No point exposing relaxed versions since they are + * implemended using either acquire or release + * barriers. + */ +# undef ETHR_GCC_RELAXED_VERSIONS__ +# define ETHR_GCC_RELAXED_VERSIONS__ 0 + +/* #endif defined(__ia64__) */ +#elif defined(__i386__) || defined(__x86_64__) + +/* + * Want full barrier versions of all modification + * operations since all of these are implemented + * using locked instructions implying full memory + * barriers. + */ +# undef ETHR_GCC_MB_MOD_VERSIONS__ +# define ETHR_GCC_MB_MOD_VERSIONS__ ETHR_HAVE___sync_val_compare_and_swap + +/* + * No point exposing acquire/release versions + * when we got full memory barrier versions + * of modification operations since all of these + * are implemented using locked instructions + * implying full memory barriers. + */ +# if ETHR_GCC_ACQB_MOD_VERSIONS__ +# undef ETHR_GCC_ACQB_MOD_VERSIONS__ +# define ETHR_GCC_ACQB_MOD_VERSIONS__ \ + (ETHR_GCC_VERSIONS_MASK__ & ~ETHR_HAVE___sync_val_compare_and_swap) +# endif +# if ETHR_GCC_RELB_MOD_VERSIONS__ +# undef ETHR_GCC_RELB_MOD_VERSIONS__ +# define ETHR_GCC_RELB_MOD_VERSIONS__ \ + (ETHR_GCC_VERSIONS_MASK__ & ~ETHR_HAVE___sync_val_compare_and_swap) +# endif + +# ifdef ETHR_X86_OUT_OF_ORDER + +/* See above... */ +# undef ETHR_GCC_RELAXED_MOD_VERSIONS__ +# define ETHR_GCC_RELAXED_MOD_VERSIONS__ 0 + +# else /* !ETHR_X86_OUT_OF_ORDER, i.e., we don't use any x86-OOO instructions... */ + +/* + * Not effected by the load acquire barrier bug, + * since no barrier at all is needed for a load + * acquire... + */ +# undef ETHR___atomic_load_ACQUIRE_barrier_bug +# define ETHR___atomic_load_ACQUIRE_barrier_bug 0 + +/* Stores imply release barriers semantics. */ +# undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ +# define ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ ETHR_GCC_VOLATILE_BIT_MASK__ + +/* Loads imply acquire barrier semantics. */ +# undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ +# define ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ ETHR_GCC_VOLATILE_BIT_MASK__ + +/* + * Trust load acquire and store release for sizes + * where volatile operation implies these barrier + * semantics since no barriers are needed. + */ +# if !ETHR_GCC_ACQB_VERSIONS__ +# undef ETHR_GCC_ACQB_VERSIONS__ +# define ETHR_GCC_ACQB_VERSIONS__ ETHR_GCC_VOLATILE_BIT_MASK__ +# endif +# if !ETHR_GCC_RELB_VERSIONS__ +# undef ETHR_GCC_RELB_VERSIONS__ +# define ETHR_GCC_RELB_VERSIONS__ ETHR_GCC_VOLATILE_BIT_MASK__ +# endif + +/* + * No point exposing relaxed versions at all since + * all mod operations are implemented with locked + * instructions implying full memory barriers and + * volatile store and load imply release and + * acquire barrier semantics. + */ +# undef ETHR_GCC_RELAXED_VERSIONS__ +# define ETHR_GCC_RELAXED_VERSIONS__ 0 + +# endif /* !ETHR_X86_OUT_OF_ORDER */ + +/* #endif defined(__i386__) || defined(__x86_64__) */ +#elif defined(__powerpc__) || defined(__ppc__) + +# if !defined(ETHR_PPC_HAVE_LWSYNC) +/* + * Release barriers are typically implemented using + * the lwsync instruction. We want our runtime + * configure test to determine if the lwsync + * instruction is available on the system or not + * before we use it. Therefore, do not implement any + * native ops using the __ATOMIC_RELEASE model. + */ +# undef ETHR_GCC_RELB_VERSIONS__ +# define ETHR_GCC_RELB_VERSIONS__ 0 +# if defined(ETHR_GCC_IMPLEMENT_ACQB_USING_LWSYNC) +/* + * Acquire barriers are usually implemented by other means + * than lwsync, but can be implemented using lwsync. Define + * ETHR_GCC_IMPLEMENT_ACQB_USING_LWSYNC if acquire barriers + * are implemented using lwsync. + */ +# undef ETHR_GCC_ACQB_VERSIONS__ +# define ETHR_GCC_ACQB_VERSIONS__ 0 +# endif +# endif + +#endif /* defined(__powerpc__) || defined(__ppc__) */ + +#if !ETHR_GCC_RELAXED_VERSIONS__ +# undef ETHR_GCC_RELAXED_MOD_VERSIONS__ +# define ETHR_GCC_RELAXED_MOD_VERSIONS__ 0 +#endif + +#if !ETHR_GCC_ACQB_VERSIONS__ +# undef ETHR_GCC_ACQB_MOD_VERSIONS__ +# define ETHR_GCC_ACQB_MOD_VERSIONS__ 0 +#endif + +#if !ETHR_GCC_RELB_VERSIONS__ +# undef ETHR_GCC_RELB_MOD_VERSIONS__ +# define ETHR_GCC_RELB_MOD_VERSIONS__ 0 +#endif + #if !defined(ETHR_HAVE_NATIVE_ATOMIC32) # define ETHR_ATOMIC_WANT_32BIT_IMPL__ # include "ethr_atomic.h" @@ -42,12 +314,51 @@ # include "ethr_atomic.h" #endif +#if defined(__x86_64__) +/* + * No instructions available for native implementation + * of these for dw-atomics... + */ +# undef ETHR_GCC_RELAXED_VERSIONS__ +# define ETHR_GCC_RELAXED_VERSIONS__ 0 +# undef ETHR_GCC_ACQB_VERSIONS__ +# define ETHR_GCC_ACQB_VERSIONS__ 0 +# undef ETHR_GCC_RELB_VERSIONS__ +# define ETHR_GCC_RELB_VERSIONS__ 0 +#endif + +#if !ETHR_GCC_RELAXED_VERSIONS__ +# undef ETHR_GCC_RELAXED_MOD_VERSIONS__ +# define ETHR_GCC_RELAXED_MOD_VERSIONS__ 0 +#endif + +#if !ETHR_GCC_ACQB_VERSIONS__ +# undef ETHR_GCC_ACQB_MOD_VERSIONS__ +# define ETHR_GCC_ACQB_MOD_VERSIONS__ 0 +#endif + +#if !ETHR_GCC_RELB_VERSIONS__ +# undef ETHR_GCC_RELB_MOD_VERSIONS__ +# define ETHR_GCC_RELB_MOD_VERSIONS__ 0 +#endif + #if (!defined(ETHR_HAVE_NATIVE_DW_ATOMIC) \ && !(ETHR_SIZEOF_PTR == 4 && defined(ETHR_HAVE_NATIVE_ATOMIC64)) \ && !(ETHR_SIZEOF_PTR == 8 && defined(ETHR_HAVE_NATIVE_ATOMIC128))) # include "ethr_dw_atomic.h" #endif -#endif +#undef ETHR___atomic_load_ACQUIRE_barrier_bug +#undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE__ +#undef ETHR_GCC_VOLATILE_STORE_IS_ATOMIC_STORE_RELB__ +#undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD__ +#undef ETHR_GCC_VOLATILE_LOAD_IS_ATOMIC_LOAD_ACQB__ +#undef ETHR_GCC_RELAXED_VERSIONS__ +#undef ETHR_GCC_RELB_VERSIONS__ +#undef ETHR_GCC_RELB_VERSIONS__ +#undef ETHR_GCC_RELAXED_MOD_VERSIONS__ +#undef ETHR_GCC_ACQB_MOD_VERSIONS__ +#undef ETHR_GCC_RELB_MOD_VERSIONS__ +#undef ETHR_GCC_MB_MOD_VERSIONS__ -#endif +#endif /* ETHREAD_GCC_NATIVE_H__ */ diff --git a/erts/include/internal/pthread/ethr_event.h b/erts/include/internal/pthread/ethr_event.h index d0a77990cc..f67bac858b 100644 --- a/erts/include/internal/pthread/ethr_event.h +++ b/erts/include/internal/pthread/ethr_event.h @@ -46,12 +46,12 @@ typedef struct { ethr_atomic32_t futex; } ethr_event; -#define ETHR_FUTEX__(FTX, OP, VAL) \ +#define ETHR_FUTEX__(FTX, OP, VAL, TIMEOUT) \ (-1 == syscall(__NR_futex, \ (void *) ethr_atomic32_addr((FTX)), \ (OP), \ (int) (VAL), \ - NULL, \ + (TIMEOUT), \ NULL, \ 0) \ ? errno : 0) @@ -64,7 +64,7 @@ ETHR_INLINE_FUNC_NAME_(ethr_event_set)(ethr_event *e) ethr_sint32_t val; val = ethr_atomic32_xchg_mb(&e->futex, ETHR_EVENT_ON__); if (val == ETHR_EVENT_OFF_WAITER__) { - int res = ETHR_FUTEX__(&e->futex, ETHR_FUTEX_WAKE__, 1); + int res = ETHR_FUTEX__(&e->futex, ETHR_FUTEX_WAKE__, 1, NULL); if (res != 0) ETHR_FATAL_ERROR__(res); } @@ -80,35 +80,58 @@ ETHR_INLINE_FUNC_NAME_(ethr_event_reset)(ethr_event *e) #endif #elif defined(ETHR_PTHREADS) -/* --- Posix mutex/cond implementation of events ---------------------------- */ +/* --- Posix mutex/cond pipe/select implementation of events ---------------- */ + typedef struct { ethr_atomic32_t state; pthread_mutex_t mtx; pthread_cond_t cnd; + int fd[2]; } ethr_event; -#define ETHR_EVENT_OFF_WAITER__ -1L -#define ETHR_EVENT_OFF__ 1L -#define ETHR_EVENT_ON__ 0L +#define ETHR_EVENT_OFF_WAITER_SELECT__ ((ethr_sint32_t) -2) +#define ETHR_EVENT_OFF_WAITER__ ((ethr_sint32_t) -1) +#define ETHR_EVENT_OFF__ ((ethr_sint32_t) 1) +#define ETHR_EVENT_ON__ ((ethr_sint32_t) 0) + +#define ETHR_EVENT_IS_WAITING__(VAL) ((VAL) < 0) #if defined(ETHR_TRY_INLINE_FUNCS) || defined(ETHR_EVENT_IMPL__) +#ifndef ETHR_HAVE_PTHREAD_TIMED_COND_MONOTONIC +#include <unistd.h> +#include <errno.h> +#endif + static void ETHR_INLINE ETHR_INLINE_FUNC_NAME_(ethr_event_set)(ethr_event *e) { ethr_sint32_t val; val = ethr_atomic32_xchg_mb(&e->state, ETHR_EVENT_ON__); - if (val == ETHR_EVENT_OFF_WAITER__) { - int res = pthread_mutex_lock(&e->mtx); - if (res != 0) - ETHR_FATAL_ERROR__(res); - res = pthread_cond_signal(&e->cnd); - if (res != 0) - ETHR_FATAL_ERROR__(res); - res = pthread_mutex_unlock(&e->mtx); - if (res != 0) - ETHR_FATAL_ERROR__(res); + if (ETHR_EVENT_IS_WAITING__(val)) { + int res; + if (val == ETHR_EVENT_OFF_WAITER_SELECT__) { + ssize_t wres; + int fd = e->fd[1]; + ETHR_ASSERT(fd >= 0); + do { + wres = write(fd, "!", 1); + } while (wres < 0 && errno == EINTR); + if (wres < 0 && errno != EAGAIN && errno != EWOULDBLOCK) + ETHR_FATAL_ERROR__(errno); + } + else { + res = pthread_mutex_lock(&e->mtx); + if (res != 0) + ETHR_FATAL_ERROR__(res); + res = pthread_cond_signal(&e->cnd); + if (res != 0) + ETHR_FATAL_ERROR__(res); + res = pthread_mutex_unlock(&e->mtx); + if (res != 0) + ETHR_FATAL_ERROR__(res); + } } } @@ -127,6 +150,8 @@ int ethr_event_init(ethr_event *e); int ethr_event_destroy(ethr_event *e); int ethr_event_wait(ethr_event *e); int ethr_event_swait(ethr_event *e, int spincount); +int ethr_event_twait(ethr_event *e, ethr_sint64_t timeout); +int ethr_event_stwait(ethr_event *e, int spincount, ethr_sint64_t timeout); #if !defined(ETHR_TRY_INLINE_FUNCS) || defined(ETHR_EVENT_IMPL__) void ethr_event_set(ethr_event *e); void ethr_event_reset(ethr_event *e); diff --git a/erts/include/internal/win/ethr_event.h b/erts/include/internal/win/ethr_event.h index 6363174a74..95e681983f 100644 --- a/erts/include/internal/win/ethr_event.h +++ b/erts/include/internal/win/ethr_event.h @@ -58,6 +58,8 @@ int ethr_event_init(ethr_event *e); int ethr_event_destroy(ethr_event *e); int ethr_event_wait(ethr_event *e); int ethr_event_swait(ethr_event *e, int spincount); +int ethr_event_twait(ethr_event *e, ethr_sint64_t timeout); +int ethr_event_stwait(ethr_event *e, int spincount, ethr_sint64_t timeout); #if !defined(ETHR_TRY_INLINE_FUNCS) || defined(ETHR_EVENT_IMPL__) void ethr_event_set(ethr_event *e); void ethr_event_reset(ethr_event *e); diff --git a/erts/lib_src/common/ethr_aux.c b/erts/lib_src/common/ethr_aux.c index b77f2178f2..1ba51882c3 100644 --- a/erts/lib_src/common/ethr_aux.c +++ b/erts/lib_src/common/ethr_aux.c @@ -148,6 +148,8 @@ ethr_init_common__(ethr_init_data *id) { int res; + ethr_init_event__(); + #if defined(ETHR_X86_RUNTIME_CONF__) x86_init(); #endif diff --git a/erts/lib_src/pthread/ethr_event.c b/erts/lib_src/pthread/ethr_event.c index 9434d60d0a..b35c599365 100644 --- a/erts/lib_src/pthread/ethr_event.c +++ b/erts/lib_src/pthread/ethr_event.c @@ -29,6 +29,9 @@ #endif #include "ethread.h" +#undef ETHR_INCLUDE_MONOTONIC_CLOCK__ +#define ETHR_INCLUDE_MONOTONIC_CLOCK__ +#include "ethr_internal.h" #if defined(ETHR_LINUX_FUTEX_IMPL__) /* --- Linux futex implementation of ethread events ------------------------- */ @@ -38,6 +41,12 @@ #define ETHR_YIELD_AFTER_BUSY_LOOPS 50 +void +ethr_init_event__(void) +{ + +} + int ethr_event_init(ethr_event *e) { @@ -52,21 +61,43 @@ ethr_event_destroy(ethr_event *e) } static ETHR_INLINE int -wait__(ethr_event *e, int spincount) +wait__(ethr_event *e, int spincount, ethr_sint64_t timeout) { unsigned sc = spincount; int res; ethr_sint32_t val; int until_yield = ETHR_YIELD_AFTER_BUSY_LOOPS; + ethr_sint64_t time = 0; /* SHUT UP annoying faulty warning... */ + struct timespec ts, *tsp; +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + ethr_sint64_t start = 0; /* SHUT UP annoying faulty warning... */ +#endif if (spincount < 0) ETHR_FATAL_ERROR__(EINVAL); + if (timeout < 0) { + tsp = NULL; + } + else { + tsp = &ts; + time = timeout; + if (spincount == 0) { + val = ethr_atomic32_read(&e->futex); + if (val == ETHR_EVENT_ON__) + goto return_event_on; + goto set_timeout; + } +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + start = ethr_get_monotonic_time(); +#endif + } + while (1) { while (1) { val = ethr_atomic32_read(&e->futex); if (val == ETHR_EVENT_ON__) - return 0; + goto return_event_on; if (sc == 0) break; sc--; @@ -79,44 +110,147 @@ wait__(ethr_event *e, int spincount) } } + if (timeout >= 0) { +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + time = timeout - (ethr_get_monotonic_time() - start); +#endif + set_timeout: + if (time <= 0) { + val = ethr_atomic32_read(&e->futex); + if (val == ETHR_EVENT_ON__) + goto return_event_on; + return ETIMEDOUT; + } + ts.tv_sec = time / (1000*1000*1000); + ts.tv_nsec = time % (1000*1000*1000); + } + if (val != ETHR_EVENT_OFF_WAITER__) { val = ethr_atomic32_cmpxchg(&e->futex, ETHR_EVENT_OFF_WAITER__, ETHR_EVENT_OFF__); if (val == ETHR_EVENT_ON__) - return 0; + goto return_event_on; ETHR_ASSERT(val == ETHR_EVENT_OFF__); } res = ETHR_FUTEX__(&e->futex, ETHR_FUTEX_WAIT__, - ETHR_EVENT_OFF_WAITER__); - if (res == EINTR) + ETHR_EVENT_OFF_WAITER__, + tsp); + switch (res) { + case EINTR: + case ETIMEDOUT: + return res; + case 0: + case EWOULDBLOCK: break; - if (res != 0 && res != EWOULDBLOCK) + default: ETHR_FATAL_ERROR__(res); + } } - return res; +return_event_on: + + ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); + + return 0; + } #elif defined(ETHR_PTHREADS) -/* --- Posix mutex/cond implementation of events ---------------------------- */ +/* --- Posix mutex/cond pipe/select implementation of events ---------------- */ + +#include <fcntl.h> +#include <sys/types.h> +#include <sys/uio.h> +#include <unistd.h> +#include <sys/select.h> +#include <errno.h> +#include <string.h> + +static void +setup_nonblocking_pipe(ethr_event *e) +{ + int flgs; + int res; + + res = pipe(e->fd); + if (res != 0) + ETHR_FATAL_ERROR__(errno); + + ETHR_ASSERT(e->fd[0] >= 0 && e->fd[1] >= 0); + + flgs = fcntl(e->fd[0], F_GETFL, 0); + fcntl(e->fd[0], F_SETFL, flgs | O_NONBLOCK); + flgs = fcntl(e->fd[1], F_GETFL, 0); + fcntl(e->fd[1], F_SETFL, flgs | O_NONBLOCK); + + ETHR_MEMBAR(ETHR_StoreStore); +} + +#define ETHR_EVENT_INVALID_FD__ -1 +#define ETHR_EVENT_COND_TIMEDWAIT__ -2 + +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC +static pthread_condattr_t monotonic_clock_cond_attr; +#endif +static pthread_condattr_t *monotonic_clock_cond_attr_p; + +#ifndef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME +# undef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC +#endif +#ifndef ETHR_MONOTONIC_CLOCK_ID +# undef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC +#endif + +void +ethr_init_event__(void) +{ + monotonic_clock_cond_attr_p = NULL; +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + if (!ethr_get_monotonic_time_is_broken() + && pthread_condattr_init(&monotonic_clock_cond_attr) == 0) { + if (pthread_condattr_setclock(&monotonic_clock_cond_attr, + ETHR_MONOTONIC_CLOCK_ID) == 0) + monotonic_clock_cond_attr_p = &monotonic_clock_cond_attr; + else + pthread_condattr_destroy(&monotonic_clock_cond_attr); + } +#endif +} int ethr_event_init(ethr_event *e) { int res; + ethr_atomic32_init(&e->state, ETHR_EVENT_OFF__); + res = pthread_mutex_init(&e->mtx, NULL); if (res != 0) return res; - res = pthread_cond_init(&e->cnd, NULL); + + res = pthread_cond_init(&e->cnd, monotonic_clock_cond_attr_p); if (res != 0) { pthread_mutex_destroy(&e->mtx); return res; } + +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + /* + * If ethr_get_monotonic_time() is broken we + * fall back on the pipe/select solution... + */ + if (monotonic_clock_cond_attr_p) { + e->fd[0] = e->fd[1] = ETHR_EVENT_COND_TIMEDWAIT__; + return 0; + } +#endif + + e->fd[0] = e->fd[1] = ETHR_EVENT_INVALID_FD__; + return 0; } @@ -124,6 +258,10 @@ int ethr_event_destroy(ethr_event *e) { int res; + if (e->fd[0] >= 0) { + close(e->fd[0]); + close(e->fd[1]); + } res = pthread_mutex_destroy(&e->mtx); if (res != 0) return res; @@ -134,12 +272,58 @@ ethr_event_destroy(ethr_event *e) } static ETHR_INLINE int -wait__(ethr_event *e, int spincount) +wait__(ethr_event *e, int spincount, ethr_sint64_t timeout) { int sc = spincount; ethr_sint32_t val; int res, ulres; int until_yield = ETHR_YIELD_AFTER_BUSY_LOOPS; + ethr_sint64_t time = 0; /* SHUT UP annoying faulty warning... */ +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + ethr_sint64_t start = 0; /* SHUT UP annoying faulty warning... */ +#endif +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + struct timespec cond_timeout; +#endif + + val = ethr_atomic32_read(&e->state); + if (val == ETHR_EVENT_ON__) + goto return_event_on; + + if (timeout < 0) { + if (spincount == 0) + goto set_event_off_waiter; + } + if (timeout == 0) + return ETIMEDOUT; + else { + time = timeout; + switch (e->fd[0]) { + case ETHR_EVENT_INVALID_FD__: +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + start = ethr_get_monotonic_time(); +#endif + setup_nonblocking_pipe(e); + break; +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + case ETHR_EVENT_COND_TIMEDWAIT__: + time += ethr_get_monotonic_time(); + cond_timeout.tv_sec = time / (1000*1000*1000); + cond_timeout.tv_nsec = time % (1000*1000*1000); + if (spincount == 0) + goto set_event_off_waiter; + break; +#endif + default: + /* Already initialized pipe... */ + if (spincount == 0) + goto set_select_timeout; +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + start = ethr_get_monotonic_time(); +#endif + break; + } + } if (spincount < 0) ETHR_FATAL_ERROR__(EINVAL); @@ -147,7 +331,8 @@ wait__(ethr_event *e, int spincount) while (1) { val = ethr_atomic32_read(&e->state); if (val == ETHR_EVENT_ON__) - return 0; + goto return_event_on; + if (sc == 0) break; sc--; @@ -160,40 +345,150 @@ wait__(ethr_event *e, int spincount) } } - if (val != ETHR_EVENT_OFF_WAITER__) { - val = ethr_atomic32_cmpxchg(&e->state, - ETHR_EVENT_OFF_WAITER__, - ETHR_EVENT_OFF__); - if (val == ETHR_EVENT_ON__) - return 0; - ETHR_ASSERT(val == ETHR_EVENT_OFF__); + if (timeout < 0 +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + || e->fd[0] == ETHR_EVENT_COND_TIMEDWAIT__ +#endif + ) { + + set_event_off_waiter: + + if (val != ETHR_EVENT_OFF_WAITER__) { + ethr_sint32_t act; + act = ethr_atomic32_cmpxchg(&e->state, + ETHR_EVENT_OFF_WAITER__, + val); + if (act == ETHR_EVENT_ON__) + goto return_event_on; + ETHR_ASSERT(act == val); + } + + res = pthread_mutex_lock(&e->mtx); + if (res != 0) + ETHR_FATAL_ERROR__(res); + + while (1) { + + val = ethr_atomic32_read(&e->state); + if (val == ETHR_EVENT_ON__) { + ETHR_ASSERT(res == 0); + ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); + break; + } + +#ifdef ETHR_HAVE_PTHREAD_COND_TIMEDWAIT_MONOTONIC + if (timeout > 0) { + res = pthread_cond_timedwait(&e->cnd, &e->mtx, &cond_timeout); + if (res == EINTR || res == ETIMEDOUT) + break; + } + else +#endif + { + res = pthread_cond_wait(&e->cnd, &e->mtx); + if (res == EINTR) + break; + } + if (res != 0) + ETHR_FATAL_ERROR__(res); + } + + ulres = pthread_mutex_unlock(&e->mtx); + if (ulres != 0) + ETHR_FATAL_ERROR__(ulres); + } + else { + int fd; + int sres; + ssize_t rres; + fd_set rset; + fd_set eset; + struct timeval select_timeout; + +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + time -= ethr_get_monotonic_time() - start; + if (time <= 0) + return ETIMEDOUT; +#endif - ETHR_ASSERT(val == ETHR_EVENT_OFF_WAITER__ - || val == ETHR_EVENT_OFF__); + set_select_timeout: - res = pthread_mutex_lock(&e->mtx); - if (res != 0) - ETHR_FATAL_ERROR__(res); + ETHR_ASSERT(time > 0); - while (1) { + /* + * timeout in nano-second, but we can only wait + * for micro-seconds... + */ + time = ((time - 1) / 1000) + 1; + + select_timeout.tv_sec = time / (1000*1000); + select_timeout.tv_usec = time % (1000*1000); + + ETHR_ASSERT(val != ETHR_EVENT_ON__); + + fd = e->fd[0]; + + /* Cleanup pipe... */ + do { + char buf[64]; + rres = read(fd, buf, sizeof(buf)); + } while (rres > 0 || (rres < 0 && errno == EINTR)); + if (rres < 0 && errno != EAGAIN && errno != EWOULDBLOCK) + ETHR_FATAL_ERROR__(errno); + + /* + * Need to verify that state is still off + * after cleaning the pipe... + */ + if (val == ETHR_EVENT_OFF_WAITER_SELECT__) { + val = ethr_atomic32_read(&e->state); + if (val == ETHR_EVENT_ON__) + goto return_event_on; + } + else { + ethr_sint32_t act; + act = ethr_atomic32_cmpxchg(&e->state, + ETHR_EVENT_OFF_WAITER_SELECT__, + val); + if (act == ETHR_EVENT_ON__) + goto return_event_on; + ETHR_ASSERT(act == val); + } + + FD_ZERO(&rset); + FD_SET(fd, &rset); + FD_ZERO(&eset); + FD_SET(fd, &eset); + + sres = select(fd + 1, &rset, NULL, &eset, &select_timeout); + if (sres == 0) + res = ETIMEDOUT; + else { + res = EINTR; + if (sres < 0 && errno != EINTR) + ETHR_FATAL_ERROR__(errno); + /* else: + * Event is *probably* set, but it can be a + * lingering writer. That is, it is important + * that we verify that it actually is set. If + * it isn't, return EINTR (spurious wakeup). + */ + } val = ethr_atomic32_read(&e->state); if (val == ETHR_EVENT_ON__) - break; + goto return_event_on; - res = pthread_cond_wait(&e->cnd, &e->mtx); - if (res == EINTR) - break; - if (res != 0) - ETHR_FATAL_ERROR__(res); } - ulres = pthread_mutex_unlock(&e->mtx); - if (ulres != 0) - ETHR_FATAL_ERROR__(ulres); + return res; + +return_event_on: + + ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); - return res; /* 0 || EINTR */ + return 0; } #else @@ -215,11 +510,23 @@ ethr_event_set(ethr_event *e) int ethr_event_wait(ethr_event *e) { - return wait__(e, 0); + return wait__(e, 0, -1); } int ethr_event_swait(ethr_event *e, int spincount) { - return wait__(e, spincount); + return wait__(e, spincount, -1); +} + +int +ethr_event_twait(ethr_event *e, ethr_sint64_t timeout) +{ + return wait__(e, 0, timeout); +} + +int +ethr_event_stwait(ethr_event *e, int spincount, ethr_sint64_t timeout) +{ + return wait__(e, spincount, timeout); } diff --git a/erts/lib_src/pthread/ethread.c b/erts/lib_src/pthread/ethread.c index 7cf38580c5..c0b1dad0b6 100644 --- a/erts/lib_src/pthread/ethread.c +++ b/erts/lib_src/pthread/ethread.c @@ -50,6 +50,8 @@ #define ETHREAD_IMPL__ #include "ethread.h" +#undef ETHR_INCLUDE_MONOTONIC_CLOCK__ +#define ETHR_INCLUDE_MONOTONIC_CLOCK__ #include "ethr_internal.h" #ifndef ETHR_HAVE_ETHREAD_DEFINES @@ -237,6 +239,10 @@ ethr_x86_cpuid__(int *eax, int *ebx, int *ecx, int *edx) #endif /* ETHR_X86_RUNTIME_CONF__ */ +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME +static void init_get_monotonic_time(void); +#endif + /* * -------------------------------------------------------------------------- * Exported functions @@ -259,6 +265,10 @@ ethr_init(ethr_init_data *id) goto error; #endif +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + init_get_monotonic_time(); +#endif + res = ethr_init_common__(id); if (res != 0) goto error; @@ -613,6 +623,144 @@ int ethr_kill(const ethr_tid tid, const int sig) #endif /* #if ETHR_HAVE_ETHR_SIG_FUNCS */ +#ifdef ETHR_HAVE_ETHR_GET_MONOTONIC_TIME + +static int broken_get_monotonic_time; + +#if defined(ETHR_HAVE_CLOCK_GETTIME_MONOTONIC) +# ifndef ETHR_MONOTONIC_CLOCK_ID +# error ETHR_MONOTONIC_CLOCK_ID should have been defined +# endif + +ethr_sint64_t +ethr_get_monotonic_time(void) +{ + ethr_sint64_t time; + struct timespec ts; + + if (broken_get_monotonic_time) + return (ethr_sint64_t) 0; + + if (0 != clock_gettime(ETHR_MONOTONIC_CLOCK_ID, &ts)) + ETHR_FATAL_ERROR__(errno); + + time = (ethr_sint64_t) ts.tv_sec; + time *= (ethr_sint64_t) 1000*1000*1000; + time += (ethr_sint64_t) ts.tv_nsec; + return time; +} + +#elif defined(ETHR_HAVE_MACH_CLOCK_GET_TIME) +# ifndef ETHR_MONOTONIC_CLOCK_ID +# error ETHR_MONOTONIC_CLOCK_ID should have been defined +# endif + +ethr_sint64_t +ethr_get_monotonic_time(void) +{ + ethr_sint64_t time; + kern_return_t res; + clock_serv_t clk_srv; + mach_timespec_t time_spec; + + if (broken_get_monotonic_time) + return (ethr_sint64_t) 0; + + errno = EFAULT; + host_get_clock_service(mach_host_self(), + ETHR_MONOTONIC_CLOCK_ID, + &clk_srv); + res = clock_get_time(clk_srv, &time_spec); + if (res != KERN_SUCCESS) + ETHR_FATAL_ERROR__(errno); + mach_port_deallocate(mach_task_self(), clk_srv); + + time = (ethr_sint64_t) time_spec.tv_sec; + time *= (ethr_sint64_t) 1000*1000*1000; + time += (ethr_sint64_t) time_spec.tv_nsec; + return time; +} + +#elif defined(ETHR_HAVE_GETHRTIME) + +ethr_sint64_t +ethr_get_monotonic_time(void) +{ + if (broken_get_monotonic_time) + return (ethr_sint64_t) 0; + return (ethr_sint64_t) gethrtime(); +} + +#else +#error missing monotonic clock +#endif + +int +ethr_get_monotonic_time_is_broken(void) +{ + return broken_get_monotonic_time; +} + +#include <string.h> +#include <ctype.h> +#include <sys/utsname.h> + +static void +init_get_monotonic_time(void) +{ + struct utsname uts; + int vsn[3]; + int i; + char *c; + + broken_get_monotonic_time = 0; + + (void) uname(&uts); + + for (c = uts.sysname; *c; c++) { + if (isupper((int) *c)) + *c = tolower((int) *c); + } + + c = uts.release; + for (i = 0; i < sizeof(vsn)/sizeof(int); i++) { + if (!isdigit((int) *c)) + vsn[i] = 0; + else { + char *c2 = c; + do { + c2++; + } while (isdigit((int) *c2)); + *c2 = '\0'; + vsn[i] = atoi(c); + c = c2; + c++; + } + } + + if (strcmp("linux", uts.sysname) == 0) { + if (vsn[0] < 2 + || (vsn[0] == 2 && vsn[1] < 6) + || (vsn[0] == 2 && vsn[1] == 6 && vsn[2] < 33)) { + broken_get_monotonic_time = 1; + } + } + else if (strcmp("sunos", uts.sysname) == 0) { + if ((vsn[0] < 5 + || (vsn[0] == 5 && vsn[1] < 8)) +#if defined(HAVE_SYSCONF) && defined(_SC_NPROCESSORS_CONF) + && sysconf(_SC_NPROCESSORS_CONF) > 1 +#endif + ) { + broken_get_monotonic_time = 1; + } + } + +} + + +#endif /* ETHR_HAVE_ETHR_GET_MONOTONIC_TIME */ + ETHR_IMPL_NORETURN__ ethr_abort__(void) { diff --git a/erts/lib_src/win/ethr_event.c b/erts/lib_src/win/ethr_event.c index bc2f635c26..a0d506356d 100644 --- a/erts/lib_src/win/ethr_event.c +++ b/erts/lib_src/win/ethr_event.c @@ -25,9 +25,16 @@ #define ETHR_EVENT_IMPL__ #include "ethread.h" +#include "ethr_internal.h" /* --- Windows implementation of thread events ------------------------------ */ +void +ethr_init_event__(void) +{ + +} + int ethr_event_init(ethr_event *e) { @@ -58,11 +65,29 @@ ethr_event_reset(ethr_event *e) } static ETHR_INLINE int -wait(ethr_event *e, int spincount) +wait(ethr_event *e, int spincount, ethr_sint64_t timeout) { - DWORD code; + DWORD code, tmo; int sc, res, until_yield = ETHR_YIELD_AFTER_BUSY_LOOPS; + if (timeout < 0) + tmo = INFINITE; + else if (timeout == 0) { + ethr_sint32_t state = ethr_atomic32_read(&e->state); + if (state == ETHR_EVENT_ON__) { + ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); + return 0; + } + return ETIMEDOUT; + } + else { + /* + * Timeout in nano-seconds, but we can only + * wait for milli-seconds... + */ + tmo = (DWORD) (timeout - 1) / (1000*1000) + 1; + } + if (spincount < 0) ETHR_FATAL_ERROR__(EINVAL); @@ -72,8 +97,10 @@ wait(ethr_event *e, int spincount) ethr_sint32_t state; while (1) { state = ethr_atomic32_read(&e->state); - if (state == ETHR_EVENT_ON__) + if (state == ETHR_EVENT_ON__) { + ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); return 0; + } if (sc == 0) break; sc--; @@ -95,7 +122,9 @@ wait(ethr_event *e, int spincount) ETHR_ASSERT(state == ETHR_EVENT_OFF__); } - code = WaitForSingleObject(e->handle, INFINITE); + code = WaitForSingleObject(e->handle, tmo); + if (code == WAIT_TIMEOUT) + return ETIMEDOUT; if (code != WAIT_OBJECT_0) ETHR_FATAL_ERROR__(ethr_win_get_errno__()); } @@ -105,11 +134,23 @@ wait(ethr_event *e, int spincount) int ethr_event_wait(ethr_event *e) { - return wait(e, 0); + return wait(e, 0, -1); } int ethr_event_swait(ethr_event *e, int spincount) { - return wait(e, spincount); + return wait(e, spincount, -1); +} + +int +ethr_event_twait(ethr_event *e, ethr_sint64_t timeout) +{ + return wait(e, 0, timeout); +} + +int +ethr_event_stwait(ethr_event *e, int spincount, ethr_sint64_t timeout) +{ + return wait(e, spincount, timeout); } diff --git a/erts/preloaded/ebin/erl_prim_loader.beam b/erts/preloaded/ebin/erl_prim_loader.beam Binary files differindex c8ec111e57..df768f9ed6 100644 --- a/erts/preloaded/ebin/erl_prim_loader.beam +++ b/erts/preloaded/ebin/erl_prim_loader.beam diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam Binary files differindex d0f9907709..3478a80dd4 100644 --- a/erts/preloaded/ebin/erlang.beam +++ b/erts/preloaded/ebin/erlang.beam diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam Binary files differindex ba45e4e011..9ed45b34bf 100644 --- a/erts/preloaded/ebin/erts_internal.beam +++ b/erts/preloaded/ebin/erts_internal.beam diff --git a/erts/preloaded/ebin/init.beam b/erts/preloaded/ebin/init.beam Binary files differindex 26f779500c..7361139cde 100644 --- a/erts/preloaded/ebin/init.beam +++ b/erts/preloaded/ebin/init.beam diff --git a/erts/preloaded/ebin/otp_ring0.beam b/erts/preloaded/ebin/otp_ring0.beam Binary files differindex 4d22d8bace..4af9d233b5 100644 --- a/erts/preloaded/ebin/otp_ring0.beam +++ b/erts/preloaded/ebin/otp_ring0.beam diff --git a/erts/preloaded/ebin/prim_eval.beam b/erts/preloaded/ebin/prim_eval.beam Binary files differindex efc8347b6e..7c0b49235e 100644 --- a/erts/preloaded/ebin/prim_eval.beam +++ b/erts/preloaded/ebin/prim_eval.beam diff --git a/erts/preloaded/ebin/prim_file.beam b/erts/preloaded/ebin/prim_file.beam Binary files differindex 6c49b5185e..00babefbb4 100644 --- a/erts/preloaded/ebin/prim_file.beam +++ b/erts/preloaded/ebin/prim_file.beam diff --git a/erts/preloaded/ebin/prim_inet.beam b/erts/preloaded/ebin/prim_inet.beam Binary files differindex f58ee4b4d5..6640a29c62 100644 --- a/erts/preloaded/ebin/prim_inet.beam +++ b/erts/preloaded/ebin/prim_inet.beam diff --git a/erts/preloaded/ebin/prim_zip.beam b/erts/preloaded/ebin/prim_zip.beam Binary files differindex 73be297bbb..3d6f1548d0 100644 --- a/erts/preloaded/ebin/prim_zip.beam +++ b/erts/preloaded/ebin/prim_zip.beam diff --git a/erts/preloaded/ebin/zlib.beam b/erts/preloaded/ebin/zlib.beam Binary files differindex 8783f189a4..3224546179 100644 --- a/erts/preloaded/ebin/zlib.beam +++ b/erts/preloaded/ebin/zlib.beam diff --git a/erts/preloaded/src/erlang.erl b/erts/preloaded/src/erlang.erl index 2e4331e15f..fd11c101bc 100644 --- a/erts/preloaded/src/erlang.erl +++ b/erts/preloaded/src/erlang.erl @@ -38,7 +38,6 @@ -export([integer_to_list/2]). -export([integer_to_binary/2]). --export([flush_monitor_message/2]). -export([set_cpu_topology/1, format_cpu_topology/1]). -export([await_proc_exit/3]). -export([memory/0, memory/1]). @@ -48,7 +47,7 @@ await_sched_wall_time_modifications/2, gather_gc_info_result/1]). --deprecated([hash/2]). +-deprecated([hash/2, now/0]). %% Get rid of autoimports of spawn to avoid clashes with ourselves. -compile({no_auto_import,[spawn_link/1]}). @@ -58,12 +57,21 @@ -compile({no_auto_import,[spawn_opt/5]}). -export_type([timestamp/0]). +-export_type([time_unit/0]). -type ext_binary() :: binary(). -type timestamp() :: {MegaSecs :: non_neg_integer(), Secs :: non_neg_integer(), MicroSecs :: non_neg_integer()}. +-type time_unit() :: + pos_integer() + | 'seconds' + | 'milli_seconds' + | 'micro_seconds' + | 'nano_seconds' + | 'native'. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Native code BIF stubs and their types %% (BIF's actually implemented in this module goes last in the file) @@ -81,7 +89,7 @@ -export([binary_to_list/3, binary_to_term/1, binary_to_term/2]). -export([bit_size/1, bitsize/1, bitstring_to_list/1]). -export([bump_reductions/1, byte_size/1, call_on_load_function/1]). --export([cancel_timer/1, check_old_code/1, check_process_code/2, +-export([cancel_timer/1, cancel_timer/2, check_old_code/1, check_process_code/2, check_process_code/3, crc32/1]). -export([crc32/2, crc32_combine/3, date/0, decode_packet/3]). -export([delete_element/2]). @@ -104,7 +112,8 @@ -export([list_to_bitstring/1, list_to_existing_atom/1, list_to_float/1]). -export([list_to_integer/1, list_to_integer/2]). -export([list_to_pid/1, list_to_tuple/1, loaded/0]). --export([localtime/0, make_ref/0, map_size/1, match_spec_test/3, md5/1, md5_final/1]). +-export([localtime/0, make_ref/0]). +-export([map_size/1, match_spec_test/3, md5/1, md5_final/1]). -export([md5_init/0, md5_update/2, module_loaded/1, monitor/2]). -export([monitor_node/2, monitor_node/3, nif_error/1, nif_error/2]). -export([node/0, node/1, now/0, phash/2, phash2/1, phash2/2]). @@ -112,9 +121,14 @@ -export([port_connect/2, port_control/3, port_get_data/1]). -export([port_set_data/2, port_to_list/1, ports/0]). -export([posixtime_to_universaltime/1, pre_loaded/0, prepare_loading/2]). +-export([monotonic_time/0, monotonic_time/1]). +-export([system_time/0, system_time/1]). +-export([convert_time_unit/3]). +-export([unique_integer/0, unique_integer/1]). +-export([time_offset/0, time_offset/1, timestamp/0]). -export([process_display/2]). -export([process_flag/3, process_info/1, processes/0, purge_module/1]). --export([put/2, raise/3, read_timer/1, ref_to_list/1, register/2]). +-export([put/2, raise/3, read_timer/1, read_timer/2, ref_to_list/1, register/2]). -export([registered/0, resume_process/1, round/1, self/0, send_after/3]). -export([seq_trace/2, seq_trace_print/1, seq_trace_print/2, setnode/2]). -export([setnode/3, size/1, spawn/3, spawn_link/3, split_binary/2]). @@ -413,8 +427,77 @@ call_on_load_function(_P1) -> -spec erlang:cancel_timer(TimerRef) -> Time | false when TimerRef :: reference(), Time :: non_neg_integer(). -cancel_timer(_TimerRef) -> - erlang:nif_error(undefined). +cancel_timer(TimerRef) -> + try + case erts_internal:access_bif_timer(TimerRef) of + undefined -> + false; + {BTR, TSrv} -> + Req = erlang:make_ref(), + TSrv ! {cancel_timeout, BTR, erlang:self(), + true, Req, TimerRef}, + receive + {cancel_timer, Req, Result} -> + Result + end + end + catch + _:_ -> erlang:error(badarg, [TimerRef]) + end. + +%% cancel_timer/2 +-spec erlang:cancel_timer(TimerRef, Options) -> Time | false | ok when + TimerRef :: reference(), + Option :: {async, boolean()} | {info, boolean()}, + Options :: [Option], + Time :: non_neg_integer(). +cancel_timer(TimerRef, Options) -> + try + {Async, Info} = get_cancel_timer_options(Options, false, true), + case erts_internal:access_bif_timer(TimerRef) of + undefined -> + case {Async, Info} of + {true, true} -> + erlang:self() ! {cancel_timer, TimerRef, false}, ok; + {false, true} -> + false; + _ -> + ok + end; + {BTR, TSrv} -> + case Async of + true -> + TSrv ! {cancel_timeout, BTR, erlang:self(), + Info, TimerRef, TimerRef}, + ok; + false -> + Req = erlang:make_ref(), + TSrv ! {cancel_timeout, BTR, erlang:self(), + true, Req, TimerRef}, + receive + {cancel_timer, Req, Result} -> + case Info of + true -> Result; + false -> ok + end + end + end + end + catch + _:_ -> erlang:error(badarg, [TimerRef, Options]) + end. + +get_cancel_timer_options([], Async, Info) -> + {Async, Info}; +get_cancel_timer_options([{async, Bool} | Opts], + _Async, Info) when Bool == true; + Bool == false -> + get_cancel_timer_options(Opts, Bool, Info); +get_cancel_timer_options([{info, Bool} | Opts], + Async, _Info) when Bool == true; + Bool == false -> + get_cancel_timer_options(Opts, Async, Bool). + %% check_old_code/1 -spec check_old_code(Module) -> boolean() when @@ -1190,13 +1273,18 @@ md5_update(_Context, _Data) -> module_loaded(_Module) -> erlang:nif_error(undefined). +-type registered_name() :: atom(). + +-type registered_process_identifier() :: registered_name() | {registered_name(), node()}. + +-type monitor_process_identifier() :: pid() | registered_process_identifier(). + %% monitor/2 --spec monitor(Type, Item) -> MonitorRef when - Type :: process, - Item :: pid() | RegName | {RegName, Node}, - RegName :: module(), - Node :: node(), +-spec monitor(process, monitor_process_identifier()) -> MonitorRef when + MonitorRef :: reference(); + (time_offset, clock_service) -> MonitorRef when MonitorRef :: reference(). + monitor(_Type, _Item) -> erlang:nif_error(undefined). @@ -1298,6 +1386,90 @@ ports() -> posixtime_to_universaltime(_P1) -> erlang:nif_error(undefined). +-spec erlang:unique_integer(ModifierList) -> integer() when + ModifierList :: [Modifier], + Modifier :: positive | monotonic. + +unique_integer(_ModifierList) -> + erlang:nif_error(undefined). + +-spec erlang:unique_integer() -> integer(). + +unique_integer() -> + erlang:nif_error(undefined). + +-spec erlang:monotonic_time() -> integer(). + +monotonic_time() -> + erlang:nif_error(undefined). + +-spec erlang:monotonic_time(Unit) -> integer() when + Unit :: time_unit(). + +monotonic_time(_Unit) -> + erlang:nif_error(undefined). + +-spec erlang:system_time() -> integer(). + +system_time() -> + erlang:nif_error(undefined). + +-spec erlang:system_time(Unit) -> integer() when + Unit :: time_unit(). + +system_time(_Unit) -> + erlang:nif_error(undefined). + +-spec erlang:convert_time_unit(Time, FromUnit, ToUnit) -> ConvertedTime when + Time :: integer(), + ConvertedTime :: integer(), + FromUnit :: time_unit(), + ToUnit :: time_unit(). + +convert_time_unit(Time, FromUnit, ToUnit) -> + try + FU = case FromUnit of + native -> erts_internal:time_unit(); + nano_seconds -> 1000*1000*1000; + micro_seconds -> 1000*1000; + milli_seconds -> 1000; + seconds -> 1; + _ when FromUnit > 0 -> FromUnit + end, + TU = case ToUnit of + native -> erts_internal:time_unit(); + nano_seconds -> 1000*1000*1000; + micro_seconds -> 1000*1000; + milli_seconds -> 1000; + seconds -> 1; + _ when ToUnit > 0 -> ToUnit + end, + case Time < 0 of + true -> TU*Time - (FU - 1); + false -> TU*Time + end div FU + catch + _ : _ -> + erlang:error(badarg, [Time, FromUnit, ToUnit]) + end. + +-spec erlang:time_offset() -> integer(). + +time_offset() -> + erlang:nif_error(undefined). + +-spec erlang:time_offset(Unit) -> integer() when + Unit :: time_unit(). + +time_offset(_Unit) -> + erlang:nif_error(undefined). + +-spec erlang:timestamp() -> Timestamp when + Timestamp :: timestamp(). + +timestamp() -> + erlang:nif_error(undefined). + %% prepare_loading/2 -spec erlang:prepare_loading(Module, Code) -> PreparedCode | {error, Reason} when Module :: module(), @@ -1365,8 +1537,53 @@ raise(_Class, _Reason, _Stacktrace) -> %% read_timer/1 -spec erlang:read_timer(TimerRef) -> non_neg_integer() | false when TimerRef :: reference(). -read_timer(_TimerRef) -> - erlang:nif_error(undefined). + +read_timer(TimerRef) -> + read_timer(TimerRef, []). + +%% read_timer/2 +-spec erlang:read_timer(TimerRef, Options) -> non_neg_integer() | false | ok when + TimerRef :: reference(), + Option :: {async, boolean()}, + Options :: [Option]. + +read_timer(TimerRef, Options) -> + try + Async = get_read_timer_options(Options, false), + case erts_internal:access_bif_timer(TimerRef) of + undefined -> + case Async of + true -> + erlang:self() ! {read_timer, TimerRef, false}, + ok; + false -> + false + end; + {BTR, TSrv} -> + case Async of + true -> + TSrv ! {read_timeout, BTR, erlang:self(), + TimerRef, TimerRef}, + ok; + false -> + Req = erlang:make_ref(), + TSrv ! {read_timeout, BTR, erlang:self(), + Req, TimerRef}, + receive + {read_timer, Req, Result} -> + Result + end + end + end + catch + _:_ -> erlang:error(badarg, [TimerRef]) + end. + +get_read_timer_options([], Async) -> + Async; +get_read_timer_options([{async, Bool} | Opts], + _Async) when Bool == true; Bool == false -> + get_read_timer_options(Opts, Bool). %% ref_to_list/1 -spec erlang:ref_to_list(Ref) -> string() when @@ -1412,8 +1629,36 @@ self() -> Dest :: pid() | atom(), Msg :: term(), TimerRef :: reference(). -send_after(_Time, _Dest, _Msg) -> - erlang:nif_error(undefined). + +send_after(0, Dest, Msg) -> + try + true = ((erlang:is_pid(Dest) + andalso erlang:node(Dest) == erlang:node()) + orelse (erlang:is_atom(Dest) + andalso Dest /= undefined)), + try Dest ! Msg catch _:_ -> ok end, + erlang:make_ref() + catch + _:_ -> + erlang:error(badarg, [0, Dest, Msg]) + end; +send_after(Time, Dest, Msg) -> + Now = erlang:monotonic_time(), + try + true = ((erlang:is_pid(Dest) + andalso erlang:node(Dest) == erlang:node()) + orelse (erlang:is_atom(Dest) + andalso Dest /= undefined)), + true = Time > 0, + true = Time < (1 bsl 32), % Maybe lift this restriction... + TO = Now + (erts_internal:time_unit()*Time) div 1000, + {BTR, TSrv, TRef} = erts_internal:create_bif_timer(), + TSrv ! {set_timeout, BTR, Dest, TO, TRef, Msg}, + TRef + catch + _:_ -> + erlang:error(badarg, [Time, Dest, Msg]) + end. %% seq_trace/2 -spec erlang:seq_trace(P1, P2) -> seq_trace_info_returns() | {term(), term(), term(), term(), term()} when @@ -1486,8 +1731,37 @@ split_binary(_Bin, _Pos) -> Dest :: pid() | atom(), Msg :: term(), TimerRef :: reference(). -start_timer(_Time, _Dest, _Msg) -> - erlang:nif_error(undefined). +start_timer(0, Dest, Msg) -> + try + true = ((erlang:is_pid(Dest) + andalso erlang:node(Dest) == erlang:node()) + orelse (erlang:is_atom(Dest) + andalso Dest /= undefined)), + TimerRef = erlang:make_ref(), + _ = try Dest ! {timeout, TimerRef, Msg} catch _:_ -> ok end, + TimerRef + catch + _:_ -> + erlang:error(badarg, [0, Dest, Msg]) + end; +start_timer(Time, Dest, Msg) -> + Now = erlang:monotonic_time(), + try + true = ((erlang:is_pid(Dest) + andalso erlang:node(Dest) == erlang:node()) + orelse (erlang:is_atom(Dest) + andalso Dest /= undefined)), + true = Time > 0, + true = Time < (1 bsl 32), % Maybe lift this restriction... + TO = Now + (erts_internal:time_unit()*Time) div 1000, + {BTR, TSrv, TimerRef} = erts_internal:create_bif_timer(), + TSrv ! {set_timeout, BTR, Dest, TO, TimerRef, + {timeout, TimerRef, Msg}}, + TimerRef + catch + _:_ -> + erlang:error(badarg, [Time, Dest, Msg]) + end. %% suspend_process/2 -spec erlang:suspend_process(Suspendee, OptList) -> boolean() when @@ -2124,6 +2398,8 @@ subtract(_,_) -> (trace_control_word, TCW) -> OldTCW when TCW :: non_neg_integer(), OldTCW :: non_neg_integer(); + (time_offset, finalize) -> OldState when + OldState :: preliminary | final | volatile; %% These are deliberately not documented (internal_cpu_topology, term()) -> term(); (sequential_tracer, pid() | port() | false) -> pid() | port() | false; @@ -2260,6 +2536,8 @@ tuple_to_list(_Tuple) -> (multi_scheduling_blockers) -> [PID :: pid()]; (nif_version) -> string(); (otp_release) -> string(); + (os_monotonic_time_source) -> [{atom(),term()}]; + (os_system_time_source) -> [{atom(),term()}]; (port_count) -> non_neg_integer(); (port_limit) -> pos_integer(); (process_count) -> pos_integer(); @@ -2277,10 +2555,14 @@ tuple_to_list(_Tuple) -> (scheduler_id) -> SchedulerId :: pos_integer(); (schedulers | schedulers_online) -> pos_integer(); (smp_support) -> boolean(); + (start_time) -> integer(); (system_version) -> string(); (system_architecture) -> string(); (threads) -> boolean(); (thread_pool_size) -> non_neg_integer(); + (time_correction) -> true | false; + (time_offset) -> preliminary | final | volatile; + (time_warp_mode) -> no_time_warp | single_time_warp | multi_time_warp; (tolerant_timeofday) -> enabled | disabled; (trace_control_word) -> non_neg_integer(); (update_cpu_info) -> changed | unchanged; @@ -3047,16 +3329,6 @@ integer_to_binary(I0, Base, R0) -> true -> integer_to_binary(I1, Base, R1) end. -%% erlang:flush_monitor_message/2 is for internal use only! -%% -%% erlang:demonitor(Ref, [flush]) traps to -%% erlang:flush_monitor_message(Ref, Res) when -%% it needs to flush a monitor message. -flush_monitor_message(Ref, Res) when erlang:is_reference(Ref), - erlang:is_atom(Res) -> - receive {_, Ref, _, _, _} -> ok after 0 -> ok end, - Res. - -record(cpu, {node = -1, processor = -1, processor_node = -1, diff --git a/erts/preloaded/src/erts_internal.erl b/erts/preloaded/src/erts_internal.erl index 2c5bd82cf0..e489001532 100644 --- a/erts/preloaded/src/erts_internal.erl +++ b/erts/preloaded/src/erts_internal.erl @@ -30,7 +30,7 @@ -export([await_port_send_result/3]). -export([cmp_term/2]). --export([map_to_tuple_keys/1]). +-export([map_to_tuple_keys/1, map_type/1, map_hashmap_children/1]). -export([port_command/3, port_connect/2, port_close/1, port_control/3, port_call/3, port_info/1, port_info/2]). @@ -38,6 +38,18 @@ -export([check_process_code/2]). +-export([flush_monitor_messages/3]). + +-export([time_unit/0]). + +-export([bif_timer_server/2]). + +-export([get_bif_timer_servers/0, create_bif_timer/0, access_bif_timer/1]). + +-export([monitor_process/2]). + +-export([is_system_process/1]). + %% %% Await result of send to port %% @@ -178,3 +190,289 @@ cmp_term(_A,_B) -> map_to_tuple_keys(_M) -> erlang:nif_error(undefined). + +%% return the internal map type +-spec map_type(M) -> Type when + M :: map(), + Type :: 'flatmap' | 'hashmap' | 'hashmap_node'. + +map_type(_M) -> + erlang:nif_error(undefined). + +%% return the internal hashmap sub-nodes from +%% a hashmap node +-spec map_hashmap_children(M) -> Children when + M :: map(), %% hashmap node + Children :: [map() | nonempty_improper_list(term(),term())]. + +map_hashmap_children(_M) -> + erlang:nif_error(undefined). + +-spec erts_internal:flush_monitor_messages(Ref, Multi, Res) -> term() when + Ref :: reference(), + Multi :: boolean(), + Res :: term(). + +%% erlang:demonitor(Ref, [flush]) traps to +%% erts_internal:flush_monitor_messages(Ref, Res) when +%% it needs to flush monitor messages. +flush_monitor_messages(Ref, Multi, Res) when is_reference(Ref) -> + receive + {_, Ref, _, _, _} -> + case Multi of + false -> + Res; + _ -> + flush_monitor_messages(Ref, Multi, Res) + end + after 0 -> + Res + end. + +-spec erts_internal:time_unit() -> pos_integer(). + +time_unit() -> + erlang:nif_error(undefined). + +-spec erts_internal:get_bif_timer_servers() -> Pids when + Pid :: pid(), + Pids :: [Pid]. + +get_bif_timer_servers() -> + erlang:nif_error(undefined). + +-spec erts_internal:create_bif_timer() -> Res when + Res :: {reference(), pid(), reference()}. + +create_bif_timer() -> + erlang:nif_error(undefined). + +-spec erts_internal:access_bif_timer(Ref) -> Res when + Ref :: reference(), + Res :: {reference(), pid()} | 'undefined'. + +access_bif_timer(_Ref) -> + erlang:nif_error(undefined). + +-spec erts_internal:monitor_process(Pid, Ref) -> boolean() when + Pid :: pid(), + Ref :: reference(). + +monitor_process(_Pid, _Ref) -> + erlang:nif_error(undefined). + +-spec erts_internal:is_system_process(Pid) -> boolean() when + Pid :: pid(). + +is_system_process(_Pid) -> + erlang:nif_error(undefined). + +%% +%% BIF timer servers +%% + +-record(tsrv_state, {rtab, + ttab, + btr, + unit, + next}). + +bif_timer_server(N, BTR) -> + try + tsrv_loop(tsrv_init_static_state(N, BTR), infinity) + catch + Type:Reason -> + erlang:display({'BIF_timer_server', + {Type, Reason}, + erlang:get_stacktrace()}), + exit(Reason) + end. + +tsrv_init_static_state(N, BTR) -> + process_flag(trap_exit, true), + NList = integer_to_list(N), + RTabName = list_to_atom("BIF_timer_reference_table_" ++ NList), + TTabName = list_to_atom("BIF_timer_time_table_" ++ NList), + #tsrv_state{rtab = ets:new(RTabName, + [set, private, {keypos, 2}]), + ttab = ets:new(TTabName, + [ordered_set, private, {keypos, 1}]), + btr = BTR, + unit = erts_internal:time_unit(), + next = infinity}. + + +tsrv_loop(#tsrv_state{unit = Unit} = StaticState, Nxt) -> + CallTime = erlang:monotonic_time(), + %% 'infinity' is greater than all integers... + NewNxt = case CallTime >= Nxt of + true -> + tsrv_handle_timeout(CallTime, StaticState); + false -> + TMO = try + (1000*(Nxt - CallTime - 1)) div Unit + 1 + catch + error:badarith when Nxt == infinity -> infinity + end, + receive + Msg -> + tsrv_handle_msg(Msg, StaticState, Nxt) + after TMO -> + Nxt + end + end, + tsrv_loop(StaticState, NewNxt). + +tsrv_handle_msg({set_timeout, BTR, Proc, Time, TRef, Msg}, + #tsrv_state{rtab = RTab, + ttab = TTab, + btr = BTR}, + Nxt) when erlang:is_integer(Time) -> + RcvTime = erlang:monotonic_time(), + case Time =< RcvTime of + true -> + try Proc ! Msg catch _:_ -> ok end, + Nxt; + false -> + Ins = case erlang:is_atom(Proc) of + true -> + true; + false -> + try + erts_internal:monitor_process(Proc, TRef) + catch + _:_ -> false + end + end, + case Ins of + false -> + Nxt; + true -> + TKey = {Time, TRef}, + true = ets:insert(RTab, TKey), + true = ets:insert(TTab, {TKey, Proc, Msg}), + case Time < Nxt of + true -> Time; + false -> Nxt + end + end + end; +tsrv_handle_msg({cancel_timeout, BTR, From, Reply, Req, TRef}, + #tsrv_state{rtab = RTab, + ttab = TTab, + unit = Unit, + btr = BTR}, + Nxt) -> + case ets:lookup(RTab, TRef) of + [] -> + case Reply of + false -> + ok; + _ -> + _ = try From ! {cancel_timer, Req, false} catch _:_ -> ok end + end, + Nxt; + [{Time, TRef} = TKey] -> + ets:delete(RTab, TRef), + ets:delete(TTab, TKey), + erlang:demonitor(TRef), + case Reply of + false -> + ok; + _ -> + RcvTime = erlang:monotonic_time(), + RT = case Time =< RcvTime of + true -> + 0; + false -> + ((1000*(Time - RcvTime)) div Unit) + end, + _ = try From ! {cancel_timer, Req, RT} catch _:_ -> ok end + end, + case Time =:= Nxt of + false -> + Nxt; + true -> + case ets:first(TTab) of + '$end_of_table' -> infinity; + {NextTime, _TRef} -> NextTime + end + end + end; +tsrv_handle_msg({read_timeout, BTR, From, Req, TRef}, + #tsrv_state{rtab = RTab, + unit = Unit, + btr = BTR}, + Nxt) -> + case ets:lookup(RTab, TRef) of + [] -> + _ = try From ! {read_timer, Req, false} catch _:_ -> ok end; + [{Time, TRef}] -> + RcvTime = erlang:monotonic_time(), + RT = case Time =< RcvTime of + true -> 0; + false -> (1000*(Time - RcvTime)) div Unit + end, + _ = try From ! {read_timer, Req, RT} catch _:_ -> ok end + end, + Nxt; +tsrv_handle_msg({'DOWN', TRef, process, _, _}, + #tsrv_state{rtab = RTab, + ttab = TTab}, + Nxt) -> + case ets:lookup(RTab, TRef) of + [] -> + Nxt; + [{Time, TRef} = TKey] -> + ets:delete(RTab, TRef), + ets:delete(TTab, TKey), + case Time =:= Nxt of + false -> + Nxt; + true -> + case ets:first(TTab) of + '$end_of_table' -> infinity; + {NextTime, _} -> NextTime + end + end + end; +tsrv_handle_msg({cancel_all_timeouts, BTR, From, Ref}, + #tsrv_state{rtab = RTab, + ttab = TTab, + btr = BTR}, + _Nxt) -> + tsrv_delete_monitor_objects(RTab), + ets:delete_all_objects(TTab), + try From ! {canceled_all_timeouts, Ref} catch _:_ -> ok end, + infinity; +tsrv_handle_msg(_GarbageMsg, _StaticState, Nxt) -> + Nxt. + +tsrv_delete_monitor_objects(RTab) -> + case ets:first(RTab) of + '$end_of_table' -> + ok; + TRef -> + erlang:demonitor(TRef), + ets:delete(RTab, TRef), + tsrv_delete_monitor_objects(RTab) + end. + +tsrv_handle_timeout(CallTime, #tsrv_state{rtab = RTab, + ttab = TTab} = S) -> + case ets:first(TTab) of + '$end_of_table' -> + infinity; + {Time, _TRef} when Time > CallTime -> + Time; + {_Time, TRef} = TKey -> + [{TKey, Proc, Msg}] = ets:lookup(TTab, TKey), + case erlang:is_pid(Proc) of + false -> ok; + _ -> erlang:demonitor(TRef) + end, + ets:delete(TTab, TKey), + ets:delete(RTab, TRef), + _ = try Proc ! Msg catch _:_ -> ok end, + tsrv_handle_timeout(CallTime, S) + end. diff --git a/erts/preloaded/src/init.erl b/erts/preloaded/src/init.erl index e95e11b3e6..48c5c37717 100644 --- a/erts/preloaded/src/init.erl +++ b/erts/preloaded/src/init.erl @@ -522,6 +522,7 @@ shutdown_pids(Heart,BootPid,State) -> Timer = shutdown_timer(State#state.flags), catch shutdown(State#state.kernel,BootPid,Timer,State), kill_all_pids(Heart), % Even the shutdown timer. + cancel_all_bif_timeouts(), kill_all_ports(Heart), flush_timout(Timer). @@ -580,6 +581,30 @@ resend([ExitMsg|Exits]) -> resend(_) -> ok. + +cancel_all_bif_timeouts() -> + TSrvs = erts_internal:get_bif_timer_servers(), + Ref = make_ref(), + {BTR, _TSrv} = erts_internal:access_bif_timer(Ref), %% Cheat... + request_cancel_all_bif_timeouts(Ref, BTR, TSrvs), + wait_response_cancel_all_bif_timeouts(Ref, BTR, TSrvs), + ok. + +request_cancel_all_bif_timeouts(_Ref, _BTR, []) -> + ok; +request_cancel_all_bif_timeouts(Ref, BTR, [TSrv|TSrvs]) -> + TSrv ! {cancel_all_timeouts, BTR, self(), {Ref, TSrv}}, + request_cancel_all_bif_timeouts(Ref, BTR, TSrvs). + +wait_response_cancel_all_bif_timeouts(_Ref, _BTR, []) -> + ok; +wait_response_cancel_all_bif_timeouts(Ref, BTR, [TSrv|TSrvs]) -> + receive + {canceled_all_timeouts, {Ref, TSrv}} -> + wait_response_cancel_all_bif_timeouts(Ref, BTR, TSrvs) + end. + + %% %% Kill all existing pids in the system (except init and heart). kill_all_pids(Heart) -> @@ -591,12 +616,9 @@ kill_all_pids(Heart) -> kill_all_pids(Heart) % Continue until all are really killed. end. -%% All except zombies. -alive_processes() -> - [P || P <- processes(), erlang:is_process_alive(P)]. - +%% All except system processes. get_pids(Heart) -> - Pids = alive_processes(), + Pids = [P || P <- processes(), not erts_internal:is_system_process(P)], delete(Heart,self(),Pids). delete(Heart,Init,[Heart|Pids]) -> delete(Heart,Init,Pids); diff --git a/erts/preloaded/src/zlib.erl b/erts/preloaded/src/zlib.erl index 5ebc67dcaa..5d9f90ec58 100644 --- a/erts/preloaded/src/zlib.erl +++ b/erts/preloaded/src/zlib.erl @@ -31,7 +31,7 @@ compress/1,uncompress/1,zip/1,unzip/1, gzip/1,gunzip/1]). --export_type([zstream/0]). +-export_type([zstream/0, zlevel/0, zwindowbits/0, zmemlevel/0, zstrategy/0]). %% flush argument encoding -define(Z_NO_FLUSH, 0). @@ -126,7 +126,7 @@ -type zlevel() :: 'none' | 'default' | 'best_compression' | 'best_speed' | 0..9. -type zmethod() :: 'deflated'. --type zwindowbits() :: -15..-9 | 9..47. +-type zwindowbits() :: -15..-8 | 8..47. -type zmemlevel() :: 1..9. -type zstrategy() :: 'default' | 'filtered' | 'huffman_only' | 'rle'. @@ -531,8 +531,8 @@ arg_method(_) -> erlang:error(badarg). -spec arg_bitsz(zwindowbits()) -> zwindowbits(). arg_bitsz(Bits) when is_integer(Bits) andalso - ((8 < Bits andalso Bits < 48) orelse - (-15 =< Bits andalso Bits < -8)) -> + ((8 =< Bits andalso Bits < 48) orelse + (-15 =< Bits andalso Bits =< -8)) -> Bits; arg_bitsz(_) -> erlang:error(badarg). diff --git a/erts/test/otp_SUITE.erl b/erts/test/otp_SUITE.erl index 385353f046..c416e031c2 100644 --- a/erts/test/otp_SUITE.erl +++ b/erts/test/otp_SUITE.erl @@ -23,7 +23,7 @@ init_per_suite/1,end_per_suite/1]). -export([undefined_functions/1,deprecated_not_in_obsolete/1, obsolete_but_not_deprecated/1,call_to_deprecated/1, - call_to_size_1/1,strong_components/1, + call_to_size_1/1,call_to_now_0/1,strong_components/1, erl_file_encoding/1,xml_file_encoding/1,runtime_dependencies/1]). -include_lib("test_server/include/test_server.hrl"). @@ -35,7 +35,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [undefined_functions, deprecated_not_in_obsolete, obsolete_but_not_deprecated, call_to_deprecated, - call_to_size_1, strong_components, + call_to_size_1, call_to_now_0, strong_components, erl_file_encoding, xml_file_encoding, runtime_dependencies]. @@ -95,8 +95,7 @@ undefined_functions(Config) when is_list(Config) -> Undef5 = dialyzer_filter(Undef4), Undef6 = wx_filter(Undef5), Undef7 = gs_filter(Undef6), - Undef8 = diameter_filter(Undef7), - Undef = ssh_filter(Undef8), + Undef = diameter_filter(Undef7), case Undef of [] -> ok; @@ -220,13 +219,11 @@ gs_filter(Undef) -> end. diameter_filter(Undef) -> - %% Filter away function calls that are catched for OTP 18 time API - filter(fun({{diameter_lib,_,_},{erlang,convert_time_resolution,3}}) -> + %% Filter away function calls that are catched. + filter(fun({{diameter_lib,_,_},{erlang,convert_time_unit,3}}) -> false; ({{diameter_lib,_,_},{erlang,monotonic_time,0}}) -> false; - ({{diameter_lib,_,_},{erlang,time_resolution,0}}) -> - false; ({{diameter_lib,_,_},{erlang,unique_integer,0}}) -> false; ({{diameter_lib,_,_},{erlang,time_offset,0}}) -> @@ -234,13 +231,6 @@ diameter_filter(Undef) -> (_) -> true end, Undef). -ssh_filter(Undef) -> - %% Filter away function calls that are catched for OTP 18 time API - filter(fun({{ssh_info,_,_},{erlang,timestamp,0}}) -> - false; - (_) -> true - end, Undef). - deprecated_not_in_obsolete(Config) when is_list(Config) -> ?line Server = ?config(xref_server, Config), ?line {ok,DeprecatedFunctions} = xref:q(Server, "DF"), @@ -297,48 +287,58 @@ call_to_deprecated(Config) when is_list(Config) -> {comment,integer_to_list(length(DeprecatedCalls))++" calls to deprecated functions"}. call_to_size_1(Config) when is_list(Config) -> - Server = ?config(xref_server, Config), - %% Applications that do not call erlang:size/1: Apps = [asn1,compiler,debugger,kernel,observer,parsetools, runtime_tools,stdlib,tools,webtool], + not_recommended_calls(Config, Apps, {erlang,size,1}). + +call_to_now_0(Config) when is_list(Config) -> + %% Applications that do not call erlang:now/1: + Apps = [asn1,common_test,compiler,debugger,dialyzer, + gs,kernel,mnesia,observer,parsetools,reltool, + runtime_tools,sasl,stdlib,syntax_tools, + test_server,tools,webtool], + not_recommended_calls(Config, Apps, {erlang,now,0}). + +not_recommended_calls(Config, Apps, MFA) -> + Server = ?config(xref_server, Config), - Fs = [{erlang,size,1}], + Fs = [MFA], Q1 = io_lib:format("E || ~p : Fun", [Fs]), - ?line {ok,AllCallsToSize1} = xref:q(Server, lists:flatten(Q1)), + {ok,AllCallsToMFA} = xref:q(Server, lists:flatten(Q1)), Q2 = io_lib:format("E | ~p : App || ~p : Fun", [Apps,Fs]), - ?line {ok,CallsToSize1} = xref:q(Server, lists:flatten(Q2)), + {ok,CallsToMFA} = xref:q(Server, lists:flatten(Q2)), - case CallsToSize1 of + case CallsToMFA of [] -> ok; _ -> - io:format("These calls cause an error:~n"), + io:format("These calls are not allowed:\n"), foreach(fun ({MFA1,MFA2}) -> - io:format("~s calls soon to be deprecated ~s", + io:format("~s calls non-recommended ~s", [format_mfa(MFA1),format_mfa(MFA2)]) - end, CallsToSize1) + end, CallsToMFA) end, - %% Enumerate calls to erlang:size/1 from other applications than - %% the ones known not to call erlang:size/1: - case AllCallsToSize1--CallsToSize1 of + %% Enumerate calls to MFA from other applications than + %% the ones known not to call MFA: + case AllCallsToMFA--CallsToMFA of [] -> ok; Calls -> - io:format("~n~nThese calls do not cause an error (yet):~n"), + io:format("~n~nThese calls are allowed for now:\n"), foreach(fun ({MFA1,MFA2}) -> - io:format("~s calls soon to be deprecated ~s", + io:format("~s calls non-recommended ~s", [format_mfa(MFA1),format_mfa(MFA2)]) end, Calls) end, - case CallsToSize1 of + case CallsToMFA of [] -> ok; _ -> - ?line ?t:fail({length(CallsToSize1),calls_to_size_1}) + ?t:fail({length(CallsToMFA),calls_to_size_1}) end. strong_components(Config) when is_list(Config) -> diff --git a/lib/asn1/src/asn1.app.src b/lib/asn1/src/asn1.app.src index 02cbba0f10..1f8805ff5e 100644 --- a/lib/asn1/src/asn1.app.src +++ b/lib/asn1/src/asn1.app.src @@ -11,5 +11,5 @@ ]}, {env, []}, {applications, [kernel, stdlib]}, - {runtime_dependencies, ["stdlib-2.0","kernel-3.0","erts-6.0"]} + {runtime_dependencies, ["stdlib-2.0","kernel-3.0","erts-7.0"]} ]}. diff --git a/lib/asn1/src/asn1ct_gen.erl b/lib/asn1/src/asn1ct_gen.erl index d3c1f34821..0e41aa1a7a 100644 --- a/lib/asn1/src/asn1ct_gen.erl +++ b/lib/asn1/src/asn1ct_gen.erl @@ -1233,15 +1233,23 @@ gen_record(TorPtype,Name,Type,Num) when is_record(Type,type) -> emit({"}).",nl,nl}), Tr ++ ExtensionList2; {Rootl1,Extl,Rootl2} -> + case Rootl1 =/= [] andalso Extl++Rootl2 =/= [] of + true -> emit([com]); + false -> ok + end, case Rootl1 of - [] -> true; - _ -> emit([",",nl]) + [_|_] -> emit([nl]); + [] -> ok end, emit(["%% with extensions",nl]), gen_record2(Name,'SEQUENCE',Extl,"",ext), + case Extl =/= [] andalso Rootl2 =/= [] of + true -> emit([com]); + false -> ok + end, case Extl of - [_H|_] when Rootl2 /= [] -> emit([",",nl]); - _ -> ok + [_|_] -> emit([nl]); + [] -> ok end, emit(["%% end of extensions",nl]), gen_record2(Name,'SEQUENCE',Rootl2,"",noext), diff --git a/lib/asn1/src/asn1ct_imm.erl b/lib/asn1/src/asn1ct_imm.erl index 91820e08de..5bf69e9294 100644 --- a/lib/asn1/src/asn1ct_imm.erl +++ b/lib/asn1/src/asn1ct_imm.erl @@ -1922,16 +1922,7 @@ enc_opt(nil, St) -> enc_opt({seq,H0,T0}, St0) -> {H,St1} = enc_opt(H0, St0), {T,St} = enc_opt(T0, St1), - case {H,T} of - {none,_} -> - {T,St}; - {{list,Imm,Data}, - {seq,{call,per,complete,[Data],_},_}} -> - %% Get rid of any explicit 'align' added by per_enc_open_type/2. - {{seq,{list,remove_trailing_align(Imm),Data},T},St}; - {_,_} -> - {{seq,H,T},St} - end; + {enc_opt_seq(H, T),St}; enc_opt({set,_,_}=Imm, St) -> {Imm,St#ost{t=undefined}}; enc_opt({sub,Src0,Int,Dst}, St0) -> @@ -1965,6 +1956,28 @@ remove_trailing_align({seq,H,T}) -> {seq,H,remove_trailing_align(T)}; remove_trailing_align(Imm) -> Imm. +enc_opt_seq(none, T) -> + T; +enc_opt_seq({list,Imm,Data}, {seq,{call,per,complete,[Data],_},_}=T) -> + %% Get rid of any explicit 'align' added by per_enc_open_type/2. + {seq,{list,remove_trailing_align(Imm),Data},T}; +enc_opt_seq({call,_,_,_,{var,_}=Dst}=H, T) -> + case is_var_unused(Dst, T) of + false -> {seq,H,T}; + true -> T + end; +enc_opt_seq(H, T) -> + {seq,H,T}. + +is_var_unused(_, align) -> + true; +is_var_unused(V, {call,_,_,Args}) -> + not lists:member(V, Args); +is_var_unused(V, {cons,H,T}) -> + is_var_unused(V, H) andalso is_var_unused(V, T); +is_var_unused(_, _) -> + false. + bit_size_propagate(Bin, Type, St) -> case t_range(Type) of any -> diff --git a/lib/asn1/src/asn1ct_value.erl b/lib/asn1/src/asn1ct_value.erl index 221cd991a7..c5901d5489 100644 --- a/lib/asn1/src/asn1ct_value.erl +++ b/lib/asn1/src/asn1ct_value.erl @@ -352,8 +352,9 @@ random_unnamed_bit_string(M, C) -> %% end. random(Upper) -> - {A1,A2,A3} = erlang:now(), - _ = random:seed(A1, A2, A3), + _ = random:seed(erlang:phash2([erlang:node()]), + erlang:monotonic_time(), + erlang:unique_integer()), random:uniform(Upper). size_random(C) -> diff --git a/lib/asn1/test/asn1_SUITE_data/DoubleEllipses.asn b/lib/asn1/test/asn1_SUITE_data/DoubleEllipses.asn index a96425cbea..846c3e7569 100644 --- a/lib/asn1/test/asn1_SUITE_data/DoubleEllipses.asn +++ b/lib/asn1/test/asn1_SUITE_data/DoubleEllipses.asn @@ -59,6 +59,18 @@ SeqAltV2 ::= SEQUENCE g INTEGER } +SeqDoubleEmpty1 ::= SEQUENCE { + ..., + ... +} + +SeqDoubleEmpty2 ::= SEQUENCE { + a BOOLEAN, + b INTEGER OPTIONAL, + ..., + ... +} + Set ::= SET { a INTEGER, ..., diff --git a/lib/asn1/test/testDoubleEllipses.erl b/lib/asn1/test/testDoubleEllipses.erl index 3caa166ae0..bd6831bf1e 100644 --- a/lib/asn1/test/testDoubleEllipses.erl +++ b/lib/asn1/test/testDoubleEllipses.erl @@ -58,6 +58,14 @@ main(_Rules) -> #'SetAltV2'{a=10,d=12, b = <<2#1010:4>>, e=true,h="PS",i=13,c=false,f=14,g=16}), + + roundtrip('SeqDoubleEmpty1', + {'SeqDoubleEmpty1'}), + roundtrip('SeqDoubleEmpty2', + {'SeqDoubleEmpty2',true,42}), + roundtrip('SeqDoubleEmpty2', + {'SeqDoubleEmpty2',true,asn1_NOVALUE}), + ok. roundtrip(T, V) -> diff --git a/lib/asn1/test/testPrimStrings.erl b/lib/asn1/test/testPrimStrings.erl index 155d6f6ff5..43835728e8 100644 --- a/lib/asn1/test/testPrimStrings.erl +++ b/lib/asn1/test/testPrimStrings.erl @@ -232,7 +232,8 @@ bit_string(Rules, Opts) -> end. random_bits(N) -> - Seed = integer_to_list(erlang:phash2(erlang:now())), + Seed0 = {erlang:monotonic_time(),erlang:unique_integer()}, + Seed = integer_to_list(erlang:phash2(Seed0)), random_bits(<<>>, N, Seed). random_bits(Bin, N, Seed) -> diff --git a/lib/common_test/doc/src/event_handler_chapter.xml b/lib/common_test/doc/src/event_handler_chapter.xml index 45f01c12ec..f39f391818 100644 --- a/lib/common_test/doc/src/event_handler_chapter.xml +++ b/lib/common_test/doc/src/event_handler_chapter.xml @@ -59,6 +59,15 @@ Event handlers plugged into this manager will receive the events from all the test nodes as well as information from the CT Master server itself.</p> + + <p>User specific event handlers may be plugged into a Common Test event + manager, either by telling Common Test to install them before the test + run (see below), or by adding the handlers dynamically during the test + run by means of + <c>gen_event:add_handler/3</c> or <c>gen_event:add_sup_handler/3</c>. + In the latter scenario, the reference of the Common Test event manager is + required. To get it, call <c>ct:get_event_mgr_ref/0</c> or (on the CT + Master node) <c>ct_master:get_event_mgr_ref/0</c>.</p> </section> <section> <marker id="usage"></marker> diff --git a/lib/common_test/src/ct.erl b/lib/common_test/src/ct.erl index 85afdc7834..9d8fce2789 100644 --- a/lib/common_test/src/ct.erl +++ b/lib/common_test/src/ct.erl @@ -52,6 +52,7 @@ -module(ct). -include("ct.hrl"). +-include("ct_util.hrl"). %% Command line user interface for running tests -export([install/1, run/1, run/2, run/3, @@ -77,6 +78,7 @@ %% Other interface functions -export([get_status/0, abort_current_testcase/1, + get_event_mgr_ref/0, encrypt_config_file/2, encrypt_config_file/3, decrypt_config_file/2, decrypt_config_file/3]). @@ -1005,6 +1007,18 @@ abort_current_testcase(Reason) -> test_server_ctrl:abort_current_testcase(Reason). %%%----------------------------------------------------------------- +%%% @spec get_event_mgr_ref() -> EvMgrRef +%%% EvMgrRef = atom() +%%% +%%% @doc <p>Call this function in order to get a reference to the +%%% CT event manager. The reference can be used to e.g. add +%%% a user specific event handler while tests are running. +%%% Example: +%%% <c>gen_event:add_handler(ct:get_event_mgr_ref(), my_ev_h, [])</c></p> +get_event_mgr_ref() -> + ?CT_EVMGR_REF. + +%%%----------------------------------------------------------------- %%% @spec encrypt_config_file(SrcFileName, EncryptFileName) -> %%% ok | {error,Reason} %%% SrcFileName = string() diff --git a/lib/common_test/src/ct_config.erl b/lib/common_test/src/ct_config.erl index 5c80a299f8..4b92ca6f8f 100644 --- a/lib/common_test/src/ct_config.erl +++ b/lib/common_test/src/ct_config.erl @@ -693,8 +693,7 @@ make_crypto_key(String) -> {[K1,K2,K3],IVec}. random_bytes(N) -> - {A,B,C} = now(), - random:seed(A, B, C), + random:seed(os:timestamp()), random_bytes_1(N, []). random_bytes_1(0, Acc) -> Acc; diff --git a/lib/common_test/src/ct_conn_log_h.erl b/lib/common_test/src/ct_conn_log_h.erl index cff02a46d9..2d15035cd8 100644 --- a/lib/common_test/src/ct_conn_log_h.erl +++ b/lib/common_test/src/ct_conn_log_h.erl @@ -34,6 +34,8 @@ -define(WIDTH,80). +-define(now, os:timestamp()). + %%%----------------------------------------------------------------- %%% Callbacks init({GL,ConnLogs}) -> @@ -72,14 +74,14 @@ handle_event({_Type, GL, _Msg}, State) when node(GL) /= node() -> handle_event({_Type,GL,{Pid,{ct_connection,Mod,Action,ConnName},Report}}, State) -> Info = conn_info(Pid,#conn_log{name=ConnName,action=Action,module=Mod}), - write_report(now(),Info,Report,GL,State), + write_report(?now,Info,Report,GL,State), {ok, State}; handle_event({_Type,GL,{Pid,Info=#conn_log{},Report}}, State) -> - write_report(now(),conn_info(Pid,Info),Report,GL,State), + write_report(?now,conn_info(Pid,Info),Report,GL,State), {ok, State}; handle_event({error_report,GL,{Pid,_,[{ct_connection,ConnName}|R]}}, State) -> %% Error reports from connection - write_error(now(),conn_info(Pid,#conn_log{name=ConnName}),R,GL,State), + write_error(?now,conn_info(Pid,#conn_log{name=ConnName}),R,GL,State), {ok, State}; handle_event(_What, State) -> {ok, State}. diff --git a/lib/common_test/src/ct_framework.erl b/lib/common_test/src/ct_framework.erl index ec525784ec..498950c9d1 100644 --- a/lib/common_test/src/ct_framework.erl +++ b/lib/common_test/src/ct_framework.erl @@ -686,18 +686,21 @@ end_tc(Mod,Func,TCPid,Result,Args,Return) -> undefined -> %% send sync notification so that event handlers may print %% in the log file before it gets closed - ct_event:sync_notify(#event{name=tc_done, - node=node(), - data={Mod,FuncSpec, - tag_cth(FinalNotify)}}), + Event = #event{name=tc_done, + node=node(), + data={Mod,FuncSpec, + tag(FinalNotify)}}, + ct_event:sync_notify(Event), Result1; Fun -> %% send sync notification so that event handlers may print %% in the log file before it gets closed - ct_event:sync_notify(#event{name=tc_done, - node=node(), - data={Mod,FuncSpec, - tag(FinalNotify)}}), + Event = #event{name=tc_done, + node=node(), + data={Mod,FuncSpec, + tag({'$test_server_framework_test', + FinalNotify})}}, + ct_event:sync_notify(Event), Fun(end_tc, Return) end, @@ -770,44 +773,37 @@ end_tc(Mod,Func,TCPid,Result,Args,Return) -> %% {error,Reason} | {skip,Reason} | {timetrap_timeout,TVal} | %% {testcase_aborted,Reason} | testcase_aborted_or_killed | -%% {'EXIT',Reason} | Other (ignored return value, e.g. 'ok') -tag({STag,Reason}) when STag == skip; STag == skipped -> - case Reason of - {failed,{_,init_per_testcase,_}} -> {auto_skipped,Reason}; - _ -> {skipped,Reason} - end; -tag({auto_skip,Reason}) -> - {auto_skipped,Reason}; -tag(E = {ETag,_}) when ETag == error; ETag == 'EXIT'; - ETag == timetrap_timeout; - ETag == testcase_aborted -> - {failed,E}; -tag(E = testcase_aborted_or_killed) -> - {failed,E}; -tag(Other) -> - Other. - -tag_cth({skipped,Reason={failed,{_,init_per_testcase,_}}}) -> +%% {'EXIT',Reason} | {fail,Reason} | {failed,Reason} | +%% {user_timetrap_error,Reason} | +%% Other (ignored return value, e.g. 'ok') +tag({'$test_server_framework_test',Result}) -> + case tag(Result) of + ok -> Result; + Failure -> Failure + end; +tag({skipped,Reason={failed,{_,init_per_testcase,_}}}) -> {auto_skipped,Reason}; -tag_cth({STag,Reason}) when STag == skip; STag == skipped -> +tag({STag,Reason}) when STag == skip; STag == skipped -> case Reason of {failed,{_,init_per_testcase,_}} -> {auto_skipped,Reason}; _ -> {skipped,Reason} end; -tag_cth({auto_skip,Reason}) -> +tag({auto_skip,Reason}) -> {auto_skipped,Reason}; -tag_cth({fail,Reason}) -> +tag({fail,Reason}) -> {failed,{error,Reason}}; -tag_cth(E = {ETag,_}) when ETag == error; ETag == 'EXIT'; +tag(Failed = {failed,_Reason}) -> + Failed; +tag(E = {ETag,_}) when ETag == error; ETag == 'EXIT'; ETag == timetrap_timeout; ETag == testcase_aborted -> {failed,E}; -tag_cth(E = testcase_aborted_or_killed) -> +tag(E = testcase_aborted_or_killed) -> {failed,E}; -tag_cth(List) when is_list(List) -> - ok; -tag_cth(Other) -> - Other. +tag(UserTimetrap = {user_timetrap_error,_Reason}) -> + UserTimetrap; +tag(_Other) -> + ok. %%%----------------------------------------------------------------- %%% @spec error_notification(Mod,Func,Args,Error) -> ok @@ -841,6 +837,8 @@ error_notification(Mod,Func,_Args,{Error,Loc}) -> io_lib:format("{test_case_failed,~p}", [Reason]); Result -> Result end; + {'EXIT',_Reason} = EXIT -> + io_lib:format("~P", [EXIT,5]); {Spec,_Reason} when is_atom(Spec) -> io_lib:format("~w", [Spec]); Other -> diff --git a/lib/common_test/src/ct_logs.erl b/lib/common_test/src/ct_logs.erl index 23332ad268..12749a8cc4 100644 --- a/lib/common_test/src/ct_logs.erl +++ b/lib/common_test/src/ct_logs.erl @@ -73,6 +73,8 @@ -define(abs(Name), filename:absname(Name)). +-define(now, os:timestamp()). + -record(log_cache, {version, all_runs = [], tests = []}). @@ -311,7 +313,7 @@ unregister_groupleader(Pid) -> %%% data to log (as in <code>io:format(Format,Args)</code>).</p> log(Heading,Format,Args) -> cast({log,sync,self(),group_leader(),ct_internal,?MAX_IMPORTANCE, - [{int_header(),[log_timestamp(now()),Heading]}, + [{int_header(),[log_timestamp(?now),Heading]}, {Format,Args}, {int_footer(),[]}]}), ok. @@ -333,7 +335,7 @@ log(Heading,Format,Args) -> %%% @see end_log/0 start_log(Heading) -> cast({log,sync,self(),group_leader(),ct_internal,?MAX_IMPORTANCE, - [{int_header(),[log_timestamp(now()),Heading]}]}), + [{int_header(),[log_timestamp(?now),Heading]}]}), ok. %%%----------------------------------------------------------------- @@ -491,11 +493,11 @@ tc_print(Category,Importance,Format,Args) -> get_heading(default) -> io_lib:format("\n-----------------------------" "-----------------------\n~s\n", - [log_timestamp(now())]); + [log_timestamp(?now)]); get_heading(Category) -> io_lib:format("\n-----------------------------" "-----------------------\n~s ~w\n", - [log_timestamp(now()),Category]). + [log_timestamp(?now),Category]). %%%----------------------------------------------------------------- @@ -553,13 +555,13 @@ div_header(Class) -> div_header(Class,"User"). div_header(Class,Printer) -> "\n<div class=\"" ++ atom_to_list(Class) ++ "\"><b>*** " ++ Printer ++ - " " ++ log_timestamp(now()) ++ " ***</b>". + " " ++ log_timestamp(?now) ++ " ***</b>". div_footer() -> "</div>". maybe_log_timestamp() -> - {MS,S,US} = now(), + {MS,S,US} = ?now, case get(log_timestamp) of {MS,S,_} -> ok; @@ -686,7 +688,7 @@ logger(Parent, Mode, Verbosity) -> make_last_run_index(Time), CtLogFd = open_ctlog(?misc_io_log), io:format(CtLogFd,int_header()++int_footer(), - [log_timestamp(now()),"Common Test Logger started"]), + [log_timestamp(?now),"Common Test Logger started"]), Parent ! {started,self(),{Time,filename:absname("")}}, set_evmgr_gl(CtLogFd), @@ -835,7 +837,7 @@ logger_loop(State) -> stop -> io:format(State#logger_state.ct_log_fd, int_header()++int_footer(), - [log_timestamp(now()),"Common Test Logger finished"]), + [log_timestamp(?now),"Common Test Logger finished"]), close_ctlog(State#logger_state.ct_log_fd), ok end. @@ -1905,6 +1907,17 @@ sort_all_runs(Dirs) -> {Date1,HH1,MM1,SS1} > {Date2,HH2,MM2,SS2} end, Dirs). +sort_ct_runs(Dirs) -> + %% Directory naming: <Prefix>.NodeName.Date_Time[/...] + %% Sort on Date_Time string: "YYYY-MM-DD_HH.MM.SS" + lists:sort(fun(Dir1,Dir2) -> + [_Prefix,_Node1,DateHH1,MM1,SS1] = + string:tokens(filename:dirname(Dir1),[$.]), + [_Prefix,_Node2,DateHH2,MM2,SS2] = + string:tokens(filename:dirname(Dir2),[$.]), + {DateHH1,MM1,SS1} =< {DateHH2,MM2,SS2} + end, Dirs). + dir_diff_all_runs(Dirs, LogCache) -> case LogCache#log_cache.all_runs of [] -> @@ -2217,7 +2230,8 @@ make_all_suites_index(When) when is_atom(When) -> end end, - LogDirs = filelib:wildcard(logdir_prefix()++".*/*"++?logdir_ext), + Wildcard = logdir_prefix()++".*/*"++?logdir_ext, + LogDirs = sort_ct_runs(filelib:wildcard(Wildcard)), LogCacheInfo = get_cache_data(UseCache), diff --git a/lib/common_test/src/ct_master.erl b/lib/common_test/src/ct_master.erl index b42ff73846..2cdb259899 100644 --- a/lib/common_test/src/ct_master.erl +++ b/lib/common_test/src/ct_master.erl @@ -25,6 +25,7 @@ -export([run/1,run/3,run/4]). -export([run_on_node/2,run_on_node/3]). -export([run_test/1,run_test/2]). +-export([get_event_mgr_ref/0]). -export([basic_html/1]). -export([abort/0,abort/1,progress/0]). @@ -292,6 +293,18 @@ progress() -> call(progress). %%%----------------------------------------------------------------- +%%% @spec get_event_mgr_ref() -> MasterEvMgrRef +%%% MasterEvMgrRef = atom() +%%% +%%% @doc <p>Call this function in order to get a reference to the +%%% CT master event manager. The reference can be used to e.g. +%%% add a user specific event handler while tests are running. +%%% Example: +%%% <c>gen_event:add_handler(ct_master:get_event_mgr_ref(), my_ev_h, [])</c></p> +get_event_mgr_ref() -> + ?CT_MEVMGR_REF. + +%%%----------------------------------------------------------------- %%% @spec basic_html(Bool) -> ok %%% Bool = true | false %%% diff --git a/lib/common_test/src/ct_master_logs.erl b/lib/common_test/src/ct_master_logs.erl index 5393097f57..384c1f6863 100644 --- a/lib/common_test/src/ct_master_logs.erl +++ b/lib/common_test/src/ct_master_logs.erl @@ -37,6 +37,8 @@ -define(details_file_name,"details.info"). -define(table_color,"lightblue"). +-define(now, os:timestamp()). + %%%-------------------------------------------------------------------- %%% API %%%-------------------------------------------------------------------- @@ -54,7 +56,7 @@ start(LogDir,Nodes) -> end. log(Heading,Format,Args) -> - cast({log,self(),[{int_header(),[log_timestamp(now()),Heading]}, + cast({log,self(),[{int_header(),[log_timestamp(?now),Heading]}, {Format,Args}, {int_footer(),[]}]}), ok. @@ -132,7 +134,7 @@ init(Parent,LogDir,Nodes) -> atom_to_list(N) ++ " " end,Nodes)), - io:format(CtLogFd,int_header(),[log_timestamp(now()),"Test Nodes\n"]), + io:format(CtLogFd,int_header(),[log_timestamp(?now),"Test Nodes\n"]), io:format(CtLogFd,"~ts\n",[NodeStr]), io:put_chars(CtLogFd,[int_footer(),"\n"]), @@ -189,7 +191,7 @@ loop(State) -> make_all_runs_index(State#state.logdir), io:format(State#state.log_fd, int_header()++int_footer(), - [log_timestamp(now()),"Finished!"]), + [log_timestamp(?now),"Finished!"]), close_ct_master_log(State#state.log_fd), close_nodedir_index(State#state.nodedir_ix_fd), ok diff --git a/lib/common_test/src/ct_netconfc.erl b/lib/common_test/src/ct_netconfc.erl index 14ee55703f..af82f2dcbf 100644 --- a/lib/common_test/src/ct_netconfc.erl +++ b/lib/common_test/src/ct_netconfc.erl @@ -794,8 +794,9 @@ action(Client,Action) -> Client :: client(), Action :: simple_xml(), Timeout :: timeout(), - Result :: {ok,[simple_xml()]} | {error,error_reason()}. -%% @doc Execute an action. + Result :: ok | {ok,[simple_xml()]} | {error,error_reason()}. +%% @doc Execute an action. If the return type is void, <c>ok</c> will +%% be returned instead of <c>{ok,[simple_xml()]}</c>. %% %% @end %%---------------------------------------------------------------------- @@ -1606,6 +1607,9 @@ decode_ok(Other) -> decode_data([{Tag,Attrs,Content}]) -> case get_local_name_atom(Tag) of + ok -> + %% when action has return type void + ok; data -> %% Since content of data has nothing from the netconf %% namespace, we remove the parent's xmlns attribute here diff --git a/lib/common_test/src/ct_telnet.erl b/lib/common_test/src/ct_telnet.erl index 4e03bf8630..d906a267a1 100644 --- a/lib/common_test/src/ct_telnet.erl +++ b/lib/common_test/src/ct_telnet.erl @@ -29,7 +29,9 @@ %% Command timeout = 10 sec (time to wait for a command to return) %% Max no of reconnection attempts = 3 %% Reconnection interval = 5 sek (time to wait in between reconnection attempts) -%% Keep alive = true (will send NOP to the server every 10 sec if connection is idle)</pre> +%% Keep alive = true (will send NOP to the server every 10 sec if connection is idle) +%% Polling limit = 0 (max number of times to poll to get a remaining string terminated) +%% Polling interval = 1 sec (sleep time between polls)</pre> %% <p>These parameters can be altered by the user with the following %% configuration term:</p> %% <pre> @@ -37,7 +39,9 @@ %% {command_timeout,Millisec}, %% {reconnection_attempts,N}, %% {reconnection_interval,Millisec}, -%% {keep_alive,Bool}]}.</pre> +%% {keep_alive,Bool}, +%% {poll_limit,N}, +%% {poll_interval,Millisec}]}.</pre> %% <p><code>Millisec = integer(), N = integer()</code></p> %% <p>Enter the <code>telnet_settings</code> term in a configuration %% file included in the test and ct_telnet will retrieve the information @@ -156,6 +160,8 @@ -define(RECONN_TIMEOUT,5000). -define(DEFAULT_TIMEOUT,10000). -define(DEFAULT_PORT,23). +-define(POLL_LIMIT,0). +-define(POLL_INTERVAL,1000). -include("ct_util.hrl"). @@ -169,6 +175,8 @@ type, target_mod, keep_alive, + poll_limit=?POLL_LIMIT, + poll_interval=?POLL_INTERVAL, extra, conn_to=?DEFAULT_TIMEOUT, com_to=?DEFAULT_TIMEOUT, @@ -379,8 +387,15 @@ cmdf(Connection,CmdFormat,Args,Opts) when is_list(Args) -> %%% Connection = ct_telnet:connection() %%% Data = [string()] %%% Reason = term() -%%% @doc Get all data which has been received by the telnet client -%%% since last command was sent. +%%% @doc Get all data that has been received by the telnet client +%%% since the last command was sent. Note that only newline terminated +%%% strings are returned. If the last string received has not yet +%%% been terminated, the connection may be polled automatically until +%%% the string is complete. The polling feature is controlled +%%% by the `poll_limit' and `poll_interval' config values and is +%%% by default disabled (meaning the function will immediately +%%% return all complete strings received and save a remaining +%%% non-terminated string for a later `get_data' call). get_data(Connection) -> case get_handle(Connection) of {ok,Pid} -> @@ -596,9 +611,12 @@ init(Name,{Ip,Port,Type},{TargetMod,KeepAlive,Extra}) -> "Reconnection attempts: ~p\n" "Reconnection interval: ~p\n" "Connection timeout: ~p\n" - "Keep alive: ~w", + "Keep alive: ~w\n" + "Poll limit: ~w\n" + "Poll interval: ~w", [Ip,Port,S1#state.com_to,S1#state.reconns, - S1#state.reconn_int,S1#state.conn_to,KeepAlive]), + S1#state.reconn_int,S1#state.conn_to,KeepAlive, + S1#state.poll_limit,S1#state.poll_interval]), {ok,TelnPid,S1}; {'EXIT',Reason} -> {error,Reason}; @@ -619,6 +637,10 @@ set_telnet_defaults([{reconnection_interval,RInt}|Ss],S) -> set_telnet_defaults(Ss,S#state{reconn_int=RInt}); set_telnet_defaults([{keep_alive,_}|Ss],S) -> set_telnet_defaults(Ss,S); +set_telnet_defaults([{poll_limit,PL}|Ss],S) -> + set_telnet_defaults(Ss,S#state{poll_limit=PL}); +set_telnet_defaults([{poll_interval,PI}|Ss],S) -> + set_telnet_defaults(Ss,S#state{poll_interval=PI}); set_telnet_defaults([Unknown|Ss],S) -> force_log(S,error, "Bad element in telnet_settings: ~p",[Unknown]), @@ -706,10 +728,8 @@ handle_msg({send,Cmd,Opts},State) -> handle_msg(get_data,State) -> start_gen_log(heading(get_data,State#state.name)), log(State,cmd,"Reading data...",[]), - {ok,Data,Buffer} = teln_get_all_data(State#state.teln_pid, - State#state.prx, - State#state.buffer, - [],[]), + {ok,Data,Buffer} = teln_get_all_data(State,State#state.buffer,[],[], + State#state.poll_limit), log(State,recv,"Return: ~p",[{ok,Data}]), end_gen_log(), {{ok,Data},State#state{buffer=Buffer}}; @@ -944,16 +964,25 @@ teln_cmd(Pid,Cmd,Prx,Newline,Timeout) -> ct_telnet_client:send_data(Pid,Cmd,Newline), teln_receive_until_prompt(Pid,Prx,Timeout). -teln_get_all_data(Pid,Prx,Data,Acc,LastLine) -> +teln_get_all_data(State=#state{teln_pid=Pid,prx=Prx},Data,Acc,LastLine,Polls) -> case check_for_prompt(Prx,LastLine++Data) of {prompt,Lines,_PromptType,Rest} -> - teln_get_all_data(Pid,Prx,Rest,[Lines|Acc],[]); + teln_get_all_data(State,Rest,[Lines|Acc],[],State#state.poll_limit); {noprompt,Lines,LastLine1} -> case ct_telnet_client:get_data(Pid) of + {ok,[]} when LastLine1 /= [], Polls > 0 -> + %% No more data from server but the last string is not + %% a complete line (maybe because of a slow connection), + timer:sleep(State#state.poll_interval), + NewPolls = if Polls == infinity -> infinity; + true -> Polls-1 + end, + teln_get_all_data(State,[],[Lines|Acc],LastLine1,NewPolls); {ok,[]} -> {ok,lists:reverse(lists:append([Lines|Acc])),LastLine1}; {ok,Data1} -> - teln_get_all_data(Pid,Prx,Data1,[Lines|Acc],LastLine1) + teln_get_all_data(State,Data1,[Lines|Acc],LastLine1, + State#state.poll_limit) end end. @@ -1106,12 +1135,18 @@ repeat_expect(Name,Pid,Data,Pattern,Acc,EO) -> teln_expect1(Name,Pid,Data,Pattern,Acc,EO=#eo{idle_timeout=IdleTO, total_timeout=TotalTO}) -> - ExpectFun = case EO#eo.seq of + %% TotalTO is a float value in this loop (unless it's 'infinity'), + %% but an integer value will be passed to the other functions + EOMod = if TotalTO /= infinity -> EO#eo{total_timeout=trunc(TotalTO)}; + true -> EO + end, + + ExpectFun = case EOMod#eo.seq of true -> fun() -> - seq_expect(Name,Pid,Data,Pattern,Acc,EO) + seq_expect(Name,Pid,Data,Pattern,Acc,EOMod) end; false -> fun() -> - one_expect(Name,Pid,Data,Pattern,EO) + one_expect(Name,Pid,Data,Pattern,EOMod) end end, case ExpectFun() of @@ -1121,9 +1156,14 @@ teln_expect1(Name,Pid,Data,Pattern,Acc,EO=#eo{idle_timeout=IdleTO, {halt,Why,Rest}; NotFinished -> %% Get more data - Fun = fun() -> get_data1(EO#eo.teln_pid) end, - BreakAfter = if TotalTO < IdleTO -> TotalTO; true -> IdleTO end, - case timer:tc(ct_gen_conn, do_within_time, [Fun, BreakAfter]) of + Fun = fun() -> get_data1(EOMod#eo.teln_pid) end, + BreakAfter = if TotalTO < IdleTO -> + %% use the integer value + EOMod#eo.total_timeout; + true -> + IdleTO + end, + case timer:tc(ct_gen_conn, do_within_time, [Fun,BreakAfter]) of {_,{error,Reason}} -> %% A timeout will occur when the telnet connection %% is idle for EO#eo.idle_timeout milliseconds. @@ -1132,13 +1172,15 @@ teln_expect1(Name,Pid,Data,Pattern,Acc,EO=#eo{idle_timeout=IdleTO, case NotFinished of {nomatch,Rest} -> %% One expect - teln_expect1(Name,Pid,Rest++Data1,Pattern,[],EO); + teln_expect1(Name,Pid,Rest++Data1, + Pattern,[],EOMod); {continue,Patterns1,Acc1,Rest} -> %% Sequence - teln_expect1(Name,Pid,Rest++Data1,Patterns1,Acc1,EO) + teln_expect1(Name,Pid,Rest++Data1, + Patterns1,Acc1,EOMod) end; {Elapsed,{ok,Data1}} -> - TVal = trunc(TotalTO - (Elapsed/1000)), + TVal = TotalTO - (Elapsed/1000), if TVal =< 0 -> {error,timeout}; true -> @@ -1146,10 +1188,12 @@ teln_expect1(Name,Pid,Data,Pattern,Acc,EO=#eo{idle_timeout=IdleTO, case NotFinished of {nomatch,Rest} -> %% One expect - teln_expect1(Name,Pid,Rest++Data1,Pattern,[],EO1); + teln_expect1(Name,Pid,Rest++Data1, + Pattern,[],EO1); {continue,Patterns1,Acc1,Rest} -> %% Sequence - teln_expect1(Name,Pid,Rest++Data1,Patterns1,Acc1,EO1) + teln_expect1(Name,Pid,Rest++Data1, + Patterns1,Acc1,EO1) end end end @@ -1430,8 +1474,10 @@ check_for_prompt(Prx,Data) -> split_lines(String) -> split_lines(String,[],[]). -split_lines([$\n|Rest],Line,Lines) -> +split_lines([$\n|Rest],Line,Lines) when Line /= [] -> split_lines(Rest,[],[lists:reverse(Line)|Lines]); +split_lines([$\n|Rest],[],Lines) -> + split_lines(Rest,[],Lines); split_lines([$\r|Rest],Line,Lines) -> split_lines(Rest,Line,Lines); split_lines([0|Rest],Line,Lines) -> diff --git a/lib/common_test/src/ct_telnet_client.erl b/lib/common_test/src/ct_telnet_client.erl index 36d33522a3..b0734d8d65 100644 --- a/lib/common_test/src/ct_telnet_client.erl +++ b/lib/common_test/src/ct_telnet_client.erl @@ -32,7 +32,7 @@ -module(ct_telnet_client). -%% -define(debug, true). +%%-define(debug, true). -export([open/2, open/3, open/4, open/5, close/1]). -export([send_data/2, send_data/3, get_data/1]). @@ -111,7 +111,6 @@ get_data(Pid) -> {ok,Data} end. - %%%----------------------------------------------------------------- %%% Internal functions init(Parent, Server, Port, Timeout, KeepAlive, ConnName) -> @@ -146,7 +145,7 @@ loop(State, Sock, Acc) -> ok end; {tcp,_,Msg0} -> - dbg("tcp msg: ~tp~n",[Msg0]), + dbg("rcv tcp msg: ~tp~n",[Msg0]), Msg = check_msg(Sock,Msg0,[]), loop(State, Sock, [Msg | Acc]); {send_data,Data} -> @@ -180,6 +179,7 @@ loop(State, Sock, Acc) -> NewState = case State of #state{keep_alive = true, get_data = 0} -> + dbg("sending NOP\n",[]), if Acc == [] -> send([?IAC,?NOP], Sock, State#state.conn_name); true -> ok @@ -225,15 +225,17 @@ loop(State, Sock, Acc) -> gen_tcp:close(Sock), Pid ! closed after wait(State#state.keep_alive,?IDLE_TIMEOUT) -> + dbg("idle timeout\n",[]), Data = lists:reverse(lists:append(Acc)), case Data of [] -> + dbg("sending NOP\n",[]), send([?IAC,?NOP], Sock, State#state.conn_name), loop(State, Sock, Acc); _ when State#state.log_pos == length(Data)+1 -> loop(State, Sock, Acc); _ -> - dbg("Idle timeout, printing ~tp\n",[Data]), + dbg("idle timeout, printing ~tp\n",[Data]), Len = length(Data), ct_telnet:log(State#state.conn_name, general_io, "~ts", @@ -391,7 +393,7 @@ cmd_dbg(Prefix,Cmd) -> end. timestamp() -> - {MS,S,US} = now(), + {MS,S,US} = os:timestamp(), {{Year,Month,Day}, {Hour,Min,Sec}} = calendar:now_to_local_time({MS,S,US}), MilliSec = trunc(US/1000), diff --git a/lib/common_test/src/cth_surefire.erl b/lib/common_test/src/cth_surefire.erl index bb12171ea7..3deaefe0e9 100644 --- a/lib/common_test/src/cth_surefire.erl +++ b/lib/common_test/src/cth_surefire.erl @@ -59,6 +59,8 @@ -define(default_report,"junit_report.xml"). -define(suite_log,"suite.log.html"). +-define(now, os:timestamp()). + %% Number of dirs from log root to testcase log file. %% ct_run.<node>.<timestamp>/<test_name>/run.<timestamp>/<tc_log>.html -define(log_depth,3). @@ -77,11 +79,11 @@ init(Path, Opts) -> axis = proplists:get_value(axis,Opts,[]), properties = proplists:get_value(properties,Opts,[]), url_base = proplists:get_value(url_base,Opts), - timer = now() }. + timer = ?now }. pre_init_per_suite(Suite,SkipOrFail,State) when is_tuple(SkipOrFail) -> {SkipOrFail, init_tc(State#state{curr_suite = Suite, - curr_suite_ts = now()}, + curr_suite_ts = ?now}, SkipOrFail) }; pre_init_per_suite(Suite,Config,#state{ test_cases = [] } = State) -> TcLog = proplists:get_value(tc_logfile,Config), @@ -96,7 +98,7 @@ pre_init_per_suite(Suite,Config,#state{ test_cases = [] } = State) -> end, {Config, init_tc(State#state{ filepath = Path, curr_suite = Suite, - curr_suite_ts = now(), + curr_suite_ts = ?now, curr_log_dir = CurrLogDir}, Config) }; pre_init_per_suite(Suite,Config,State) -> @@ -169,9 +171,9 @@ do_tc_skip(Res, State) -> State#state{ test_cases = [NewTC | tl(TCs)]}. init_tc(State, Config) when is_list(Config) == false -> - State#state{ timer = now(), tc_log = "" }; + State#state{ timer = ?now, tc_log = "" }; init_tc(State, Config) -> - State#state{ timer = now(), + State#state{ timer = ?now, tc_log = proplists:get_value(tc_logfile, Config, [])}. end_tc(Func, Config, Res, State) when is_atom(Func) -> @@ -194,7 +196,7 @@ end_tc(Name, _Config, _Res, State = #state{ curr_suite = Suite, ClassName = atom_to_list(Suite), PGroup = string:join([ atom_to_list(Group)|| Group <- lists:reverse(Groups)],"."), - TimeTakes = io_lib:format("~f",[timer:now_diff(now(),TS) / 1000000]), + TimeTakes = io_lib:format("~f",[timer:now_diff(?now,TS) / 1000000]), State#state{ test_cases = [#testcase{ log = Log, url = Url, timestamp = now_to_string(TS), @@ -209,7 +211,7 @@ close_suite(#state{ test_cases = [] } = State) -> State; close_suite(#state{ test_cases = TCs, url_base = UrlBase } = State) -> {Total,Fail,Skip} = count_tcs(TCs,0,0,0), - TimeTaken = timer:now_diff(now(),State#state.curr_suite_ts) / 1000000, + TimeTaken = timer:now_diff(?now,State#state.curr_suite_ts) / 1000000, SuiteLog = filename:join(State#state.curr_log_dir,?suite_log), SuiteUrl = make_url(UrlBase,SuiteLog), Suite = #testsuite{ name = atom_to_list(State#state.curr_suite), diff --git a/lib/common_test/test/ct_config_SUITE_data/config/test/config_dynamic_SUITE.erl b/lib/common_test/test/ct_config_SUITE_data/config/test/config_dynamic_SUITE.erl index 8ee12a2e4d..ef1fd63905 100644 --- a/lib/common_test/test/ct_config_SUITE_data/config/test/config_dynamic_SUITE.erl +++ b/lib/common_test/test/ct_config_SUITE_data/config/test/config_dynamic_SUITE.erl @@ -35,7 +35,7 @@ %% which will return the list with the following variables: %% localtime = the erlang:localtime() result in list [{date, Date}, {time, Time}] %% node = erlang:node() - can be compared in the testcase -%% now = erlang:now() - easier to compare than localtime() +%% now = os:timestamp() - easier to compare than localtime() %% config_server_pid - pid of the config server, should NOT change! %% config_server_vsn - .19 %% config_server_iteration - a number of iteration config_server's loop done @@ -73,7 +73,7 @@ test_get_known_variable(_)-> test_localtime_update(_)-> Seconds = 5, LT1 = ct:get_config(localtime), - timer:sleep(Seconds*1000), + ct:sleep(Seconds*1000), LT2 = ct:reload_config(localtime), case is_diff_ok(LT1, LT2, Seconds) of {false, Actual, Exp}-> diff --git a/lib/common_test/test/ct_config_SUITE_data/config/test/config_server.erl b/lib/common_test/test/ct_config_SUITE_data/config/test/config_server.erl index 8463fea645..e65d6584b1 100644 --- a/lib/common_test/test/ct_config_SUITE_data/config/test/config_server.erl +++ b/lib/common_test/test/ct_config_SUITE_data/config/test/config_server.erl @@ -73,7 +73,7 @@ loop(Iteration)-> [{localtime, [{date, D}, {time, T}]}, {node, erlang:node()}, {config_server_iteration, Iteration}, - {now, erlang:now()}, + {now, os:timestamp()}, {config_server_pid, self()}, {config_server_vsn, ?vsn}], Config2 = if Iteration rem 2 == 0-> diff --git a/lib/common_test/test/ct_error_SUITE.erl b/lib/common_test/test/ct_error_SUITE.erl index ecf231529a..8464225284 100644 --- a/lib/common_test/test/ct_error_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE.erl @@ -1466,7 +1466,8 @@ test_events(misc_errors) -> {failed,{error,{suite_failed,this_is_expected}}}}}, {?eh,test_stats,{0,5,{0,0}}}, {?eh,tc_start,{misc_error_1_SUITE,killed_by_signal_1}}, - {?eh,tc_done,{misc_error_1_SUITE,killed_by_signal_1,i_die_now}}, + {?eh,tc_done,{misc_error_1_SUITE,killed_by_signal_1, + {failed,{'EXIT',i_die_now}}}}, {?eh,test_stats,{0,6,{0,0}}}, {?eh,tc_start,{misc_error_1_SUITE,killed_by_signal_2}}, {?eh,tc_done,{misc_error_1_SUITE,killed_by_signal_2, diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_12_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_12_SUITE.erl index 806d3caf72..0ff8659269 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_12_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_12_SUITE.erl @@ -26,10 +26,10 @@ init_per_testcase(_, Config) -> Config. end_per_testcase(tc2, _Config) -> - timer:sleep(2000), + ct:sleep(2000), exit(this_should_not_be_printed); end_per_testcase(tc4, _Config) -> - timer:sleep(2000), + ct:sleep(2000), exit(this_should_not_be_printed); end_per_testcase(_, _) -> ok. @@ -42,7 +42,7 @@ tc1() -> put('$test_server_framework_test', fun(init_tc, _Default) -> ct:pal("init_tc(~p): Night time...",[self()]), - timer:sleep(2000), + ct:sleep(2000), ct:pal("init_tc(~p): Day time!",[self()]), exit(this_should_not_be_printed); (_, Default) -> Default @@ -67,7 +67,7 @@ tc3(_) -> put('$test_server_framework_test', fun(end_tc, _Default) -> ct:pal("end_tc(~p): Night time...",[self()]), - timer:sleep(1000), + ct:sleep(1000), ct:pal("end_tc(~p): Day time!",[self()]); (_, Default) -> Default end), @@ -78,7 +78,7 @@ tc4() -> put('$test_server_framework_test', fun(end_tc, _Default) -> ct:pal("end_tc(~p): Night time...",[self()]), - timer:sleep(1000), + ct:sleep(1000), ct:pal("end_tc(~p): Day time!",[self()]); (_, Default) -> Default end), diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_13_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_13_SUITE.erl index c8a3c1d15e..cfc0babb68 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_13_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_13_SUITE.erl @@ -26,7 +26,7 @@ init_per_suite() -> put('$test_server_framework_test', fun(end_tc, _Default) -> ct:pal("end_tc(~p): Night time...",[self()]), - timer:sleep(1000), + ct:sleep(1000), ct:pal("end_tc(~p): Day time!",[self()]); (_, Default) -> Default end), diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_14_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_14_SUITE.erl index 960d0f61b0..54b09e78c6 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_14_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_14_SUITE.erl @@ -29,7 +29,7 @@ end_per_suite() -> put('$test_server_framework_test', fun(end_tc, _Default) -> ct:pal("end_tc(~p): Night time...",[self()]), - timer:sleep(1000), + ct:sleep(1000), ct:pal("end_tc(~p): Day time!",[self()]); (_, Default) -> Default end), diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_3_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_3_SUITE.erl index 08c57887ef..0d93e46501 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_3_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_3_SUITE.erl @@ -36,7 +36,7 @@ suite() -> %% Reason = term() %%-------------------------------------------------------------------- init_per_suite(Config) -> - timer:sleep(5000), + ct:sleep(5000), exit(shouldnt_happen). % Config. diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_7_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_7_SUITE.erl index 9cd5b6ad29..d95f3b235b 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_7_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_7_SUITE.erl @@ -43,7 +43,7 @@ init_per_suite(Config) -> %% Config0 = Config1 = [tuple()] %%-------------------------------------------------------------------- end_per_suite(Config) -> - timer:sleep(5000), + ct:sleep(5000), ok. %%-------------------------------------------------------------------- diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_8_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_8_SUITE.erl index 25993833d7..d8f0c48034 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_8_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/cfg_error_8_SUITE.erl @@ -57,7 +57,7 @@ init_per_group(g1, Config) -> Config; init_per_group(g2, Config) -> ct:comment("init_per_group(g2) timeout"), - timer:sleep(5000), + ct:sleep(5000), Config; init_per_group(g3, _Config) -> badmatch = 42; @@ -80,7 +80,7 @@ end_per_group(g11, _Config) -> ok; end_per_group(g12, _Config) -> ct:comment("end_per_group(g6) timeout"), - timer:sleep(5000), + ct:sleep(5000), ok; end_per_group(_GroupName, _Config) -> ok. diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_1_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_1_SUITE.erl index a98382965f..1451a4119e 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_1_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_1_SUITE.erl @@ -111,7 +111,7 @@ end_per_testcase1(tc2, Config) -> ct:pal("end_per_testcase(tc2): ~p", [Config]), tc2 = ?config(tc, Config), {failed,timetrap_timeout} = ?config(tc_status, Config), - timer:sleep(2000); + ct:sleep(2000); end_per_testcase1(tc3, Config) -> ct:pal("end_per_testcase(tc3): ~p", [Config]), @@ -123,7 +123,7 @@ end_per_testcase1(tc4, Config) -> ct:pal("end_per_testcase(tc4): ~p", [Config]), tc4 = ?config(tc, Config), {failed,{testcase_aborted,testing_end_conf}} = ?config(tc_status, Config), - timer:sleep(2000); + ct:sleep(2000); end_per_testcase1(tc5, Config) -> ct:pal("end_per_testcase(tc5): ~p", [Config]), @@ -182,29 +182,29 @@ all() -> [tc1, tc2, tc3, tc4, tc5, tc6, tc7, tc8, tc9]. tc1(_) -> - timer:sleep(2000), + ct:sleep(2000), ok. tc2(_) -> - timer:sleep(2000). + ct:sleep(2000). tc3(_) -> spawn(ct, abort_current_testcase, [testing_end_conf]), - timer:sleep(2000), + ct:sleep(2000), ok. tc4(_) -> spawn(ct, abort_current_testcase, [testing_end_conf]), - timer:sleep(2000), + ct:sleep(2000), ok. tc5(_) -> - timer:sleep(2000), + ct:sleep(2000), ok. tc6(_) -> spawn(ct, abort_current_testcase, [testing_end_conf]), - timer:sleep(2000). + ct:sleep(2000). tc7(_) -> sleep(2000), @@ -220,5 +220,5 @@ tc9(_) -> %%%----------------------------------------------------------------- sleep(T) -> - timer:sleep(T), + ct:sleep(T), ok. diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_2_SUITE.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_2_SUITE.erl index a77d06815e..d926fc55a4 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_2_SUITE.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_2_SUITE.erl @@ -141,12 +141,13 @@ tc3() -> [{timetrap,{seconds,2}}]. tc3(_) -> - T0 = now(), + T0 = erlang:monotonic_time(), ct:timetrap(infinity), N = list_to_integer(ct:get_config(multiply)), ct:comment(io_lib:format("Sleeping for ~w sec...", [4*N])), ct:sleep(4000), - Diff = timer:now_diff(now(), T0), + T1 = erlang:monotonic_time(), + Diff = erlang:convert_time_unit(T1-T0, native, micro_seconds), if ((Diff < (N*4000000)) or (Diff > (N*4500000))) -> exit(not_expected); true -> diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_helper.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_helper.erl index 1389acca11..a9ea0be847 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_helper.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/timetrap_helper.erl @@ -3,5 +3,5 @@ -export([sleep/1]). sleep(T) -> - timer:sleep(T), + ct:sleep(T), ok. diff --git a/lib/common_test/test/ct_error_SUITE_data/error/test/verify_config.erl b/lib/common_test/test/ct_error_SUITE_data/error/test/verify_config.erl index 446dd8bfdf..d5b3e0035a 100644 --- a/lib/common_test/test/ct_error_SUITE_data/error/test/verify_config.erl +++ b/lib/common_test/test/ct_error_SUITE_data/error/test/verify_config.erl @@ -81,7 +81,7 @@ init(Id, Opts) -> -spec id(Opts :: proplists:proplist()) -> Id :: term(). id(Opts) -> - now(). + os:timestamp(). %% @doc Called before init_per_suite is called. Note that this callback is %% only called if the CTH is added before init_per_suite is run (eg. in a test diff --git a/lib/common_test/test/ct_event_handler_SUITE.erl b/lib/common_test/test/ct_event_handler_SUITE.erl index 30a5e650fe..b759424e46 100644 --- a/lib/common_test/test/ct_event_handler_SUITE.erl +++ b/lib/common_test/test/ct_event_handler_SUITE.erl @@ -29,6 +29,7 @@ -compile(export_all). -include_lib("common_test/include/ct.hrl"). +-include_lib("common_test/src/ct_util.hrl"). %-include_lib("common_test/include/ct_event.hrl"). @@ -59,7 +60,7 @@ end_per_testcase(TestCase, Config) -> suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> - [start_stop, results]. + [start_stop, results, event_mgrs]. groups() -> []. @@ -156,21 +157,28 @@ results(Config) when is_list(Config) -> TestEvents = [{eh_A,start_logging,{'DEF','RUNDIR'}}, {eh_A,test_start,{'DEF',{'START_TIME','LOGDIR'}}}, - {eh_A,start_info,{1,1,4}}, + {eh_A,start_info,{1,1,5}}, {eh_A,tc_start,{eh_11_SUITE,init_per_suite}}, {eh_A,tc_done,{eh_11_SUITE,init_per_suite,ok}}, - {eh_A,tc_start,{eh_11_SUITE,tc1}}, - {eh_A,tc_done,{eh_11_SUITE,tc1,ok}}, - {eh_A,test_stats,{1,0,{0,0}}}, - {eh_A,tc_start,{eh_11_SUITE,tc2}}, - {eh_A,tc_done,{eh_11_SUITE,tc2,{skipped,"Skip"}}}, - {eh_A,test_stats,{1,0,{1,0}}}, - {eh_A,tc_start,{eh_11_SUITE,tc3}}, - {eh_A,tc_done,{eh_11_SUITE,tc3,{skipped,"Skipped"}}}, - {eh_A,test_stats,{1,0,{2,0}}}, - {eh_A,tc_start,{eh_11_SUITE,tc4}}, - {eh_A,tc_done,{eh_11_SUITE,tc4,{failed,{error,'Failing'}}}}, - {eh_A,test_stats,{1,1,{2,0}}}, + [{eh_A,tc_start,{eh_11_SUITE,{init_per_group,g1,[]}}}, + {eh_A,tc_done,{eh_11_SUITE,{init_per_group,g1,[]},ok}}, + {eh_A,tc_start,{eh_11_SUITE,tc1}}, + {eh_A,tc_done,{eh_11_SUITE,tc1,ok}}, + {eh_A,test_stats,{1,0,{0,0}}}, + {eh_A,tc_start,{eh_11_SUITE,tc2}}, + {eh_A,tc_done,{eh_11_SUITE,tc2,ok}}, + {eh_A,test_stats,{2,0,{0,0}}}, + {eh_A,tc_start,{eh_11_SUITE,tc3}}, + {eh_A,tc_done,{eh_11_SUITE,tc3,{skipped,"Skip"}}}, + {eh_A,test_stats,{2,0,{1,0}}}, + {eh_A,tc_start,{eh_11_SUITE,tc4}}, + {eh_A,tc_done,{eh_11_SUITE,tc4,{skipped,"Skipped"}}}, + {eh_A,test_stats,{2,0,{2,0}}}, + {eh_A,tc_start,{eh_11_SUITE,tc5}}, + {eh_A,tc_done,{eh_11_SUITE,tc5,{failed,{error,'Failing'}}}}, + {eh_A,test_stats,{2,1,{2,0}}}, + {eh_A,tc_start,{eh_11_SUITE,{end_per_group,g1,[]}}}, + {eh_A,tc_done,{eh_11_SUITE,{end_per_group,g1,[]},ok}}], {eh_A,tc_start,{eh_11_SUITE,end_per_suite}}, {eh_A,tc_done,{eh_11_SUITE,end_per_suite,ok}}, {eh_A,test_done,{'DEF','STOP_TIME'}}, @@ -179,5 +187,10 @@ results(Config) when is_list(Config) -> ok = ct_test_support:verify_events(TestEvents++TestEvents, Events, Config). +event_mgrs(_) -> + ?CT_EVMGR_REF = ct:get_event_mgr_ref(), + ?CT_MEVMGR_REF = ct_master:get_event_mgr_ref(). + + %%%----------------------------------------------------------------- %%% HELP FUNCTIONS diff --git a/lib/common_test/test/ct_event_handler_SUITE_data/event_handling_1/test/eh_11_SUITE.erl b/lib/common_test/test/ct_event_handler_SUITE_data/event_handling_1/test/eh_11_SUITE.erl index a52fe96f30..14ea12d579 100644 --- a/lib/common_test/test/ct_event_handler_SUITE_data/event_handling_1/test/eh_11_SUITE.erl +++ b/lib/common_test/test/ct_event_handler_SUITE_data/event_handling_1/test/eh_11_SUITE.erl @@ -32,100 +32,36 @@ %% COMMON TEST CALLBACK FUNCTIONS %%-------------------------------------------------------------------- -%%-------------------------------------------------------------------- -%% Function: suite() -> Info -%% -%% Info = [tuple()] -%% List of key/value pairs. -%% -%% Description: Returns list of tuples to set default properties -%% for the suite. -%% -%% Note: The suite/0 function is only meant to be used to return -%% default data values, not perform any other operations. -%%-------------------------------------------------------------------- suite() -> [ {timetrap,{seconds,10}} ]. -%%-------------------------------------------------------------------- -%% Function: init_per_suite(Config0) -> -%% Config1 | {skip,Reason} | {skip_and_save,Reason,Config1} -%% -%% Config0 = Config1 = [tuple()] -%% A list of key/value pairs, holding the test case configuration. -%% Reason = term() -%% The reason for skipping the suite. -%% -%% Description: Initialization before the suite. -%% -%% Note: This function is free to add any key/value pairs to the Config -%% variable, but should NOT alter/remove any existing entries. -%%-------------------------------------------------------------------- init_per_suite(Config) -> Config. -%%-------------------------------------------------------------------- -%% Function: end_per_suite(Config0) -> void() | {save_config,Config1} -%% -%% Config0 = Config1 = [tuple()] -%% A list of key/value pairs, holding the test case configuration. -%% -%% Description: Cleanup after the suite. -%%-------------------------------------------------------------------- end_per_suite(_Config) -> - ok. + %% should report ok as result to event handler + done. + +init_per_group(_GroupName, Config) -> + Config. + +end_per_group(_GroupName, Config) -> + %% should report ok as result to event handler + void. -%%-------------------------------------------------------------------- -%% Function: init_per_testcase(TestCase, Config0) -> -%% Config1 | {skip,Reason} | {skip_and_save,Reason,Config1} -%% -%% TestCase = atom() -%% Name of the test case that is about to run. -%% Config0 = Config1 = [tuple()] -%% A list of key/value pairs, holding the test case configuration. -%% Reason = term() -%% The reason for skipping the test case. -%% -%% Description: Initialization before each test case. -%% -%% Note: This function is free to add any key/value pairs to the Config -%% variable, but should NOT alter/remove any existing entries. -%%-------------------------------------------------------------------- init_per_testcase(_TestCase, Config) -> Config. -%%-------------------------------------------------------------------- -%% Function: end_per_testcase(TestCase, Config0) -> -%% void() | {save_config,Config1} -%% -%% TestCase = atom() -%% Name of the test case that is finished. -%% Config0 = Config1 = [tuple()] -%% A list of key/value pairs, holding the test case configuration. -%% -%% Description: Cleanup after each test case. -%%-------------------------------------------------------------------- end_per_testcase(_TestCase, _Config) -> - ok. + true. -%%-------------------------------------------------------------------- -%% Function: all() -> TestCases | {skip,Reason} -%% -%% TestCases = [TestCase | {sequence,SeqName}] -%% TestCase = atom() -%% Name of a test case. -%% SeqName = atom() -%% Name of a test case sequence. -%% Reason = term() -%% The reason for skipping all test cases. -%% -%% Description: Returns the list of test cases that are to be executed. -%%-------------------------------------------------------------------- -all() -> - [tc1, tc2, tc3, tc4]. +groups() -> + [{g1, [], [tc1, tc2, tc3, tc4, tc5]}]. +all() -> + [{group,g1}]. %%-------------------------------------------------------------------- %% TEST CASES @@ -134,11 +70,15 @@ all() -> tc1(_Config) -> ok. -tc2(_Config) -> - {skip,"Skip"}. +tc2(_Config) -> + %% should report ok as result to event handler + 42. tc3(_Config) -> + {skip,"Skip"}. + +tc4(_Config) -> {skipped,"Skipped"}. -tc4(_Config) -> +tc5(_Config) -> exit('Failing'). diff --git a/lib/common_test/test/ct_gen_conn_SUITE_data/conn_SUITE.erl b/lib/common_test/test/ct_gen_conn_SUITE_data/conn_SUITE.erl index 1344878675..96dd80e4e8 100644 --- a/lib/common_test/test/ct_gen_conn_SUITE_data/conn_SUITE.erl +++ b/lib/common_test/test/ct_gen_conn_SUITE_data/conn_SUITE.erl @@ -73,23 +73,23 @@ handles_to_multi_conn_pids(_Config) -> {true,true} = {is_process_alive(Handle3),is_process_alive(ConnPid3)}, ok = proto:close(Handle1), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle1),is_process_alive(ConnPid1)}, {true,true} = {is_process_alive(Handle2),is_process_alive(ConnPid2)}, ok = proto:kill_conn_proc(Handle2), - timer:sleep(100), + ct:sleep(100), {true,false} = {is_process_alive(Handle2),is_process_alive(ConnPid2)}, ConnPid2x = ct_gen_conn:get_conn_pid(Handle2), true = is_process_alive(ConnPid2x), ok = proto:close(Handle2), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle2),is_process_alive(ConnPid2x)}, application:set_env(ct_test, reconnect, false), ok = proto:kill_conn_proc(Handle3), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle3),is_process_alive(ConnPid3)}, ok. @@ -116,23 +116,23 @@ handles_to_single_conn_pids(_Config) -> ct:pal("CONNS = ~n~p", [Conns]), ok = proto:close(Handle1), - timer:sleep(100), + ct:sleep(100), {false,true} = {is_process_alive(Handle1),is_process_alive(ConnPid)}, ok = proto:kill_conn_proc(Handle2), - timer:sleep(100), + ct:sleep(100), NewConnPid = ct_gen_conn:get_conn_pid(Handle2), NewConnPid = ct_gen_conn:get_conn_pid(Handle3), true = is_process_alive(Handle2), true = is_process_alive(Handle3), ok = proto:close(Handle2), - timer:sleep(100), + ct:sleep(100), {false,true} = {is_process_alive(Handle2),is_process_alive(NewConnPid)}, application:set_env(ct_test, reconnect, false), ok = proto:kill_conn_proc(Handle3), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle3),is_process_alive(NewConnPid)}, ok. @@ -158,29 +158,29 @@ names_to_multi_conn_pids(_Config) -> Handle1 = proto:open(mconn1), ok = proto:close(mconn1), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle1),is_process_alive(ConnPid1)}, ok = proto:kill_conn_proc(Handle2), - timer:sleep(100), + ct:sleep(100), Handle2 = proto:open(mconn2), % should've been reconnected already {true,false} = {is_process_alive(Handle2),is_process_alive(ConnPid2)}, ConnPid2x = ct_gen_conn:get_conn_pid(Handle2), true = is_process_alive(ConnPid2x), ok = proto:close(mconn2), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle2),is_process_alive(ConnPid2x)}, Handle2y = proto:open(mconn2), ConnPid2y = ct_gen_conn:get_conn_pid(Handle2y), {true,true} = {is_process_alive(Handle2y),is_process_alive(ConnPid2y)}, ok = proto:close(mconn2), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle2y),is_process_alive(ConnPid2y)}, application:set_env(ct_test, reconnect, false), ok = proto:kill_conn_proc(Handle3), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle3),is_process_alive(ConnPid3)}, ok. @@ -211,11 +211,11 @@ names_to_single_conn_pids(_Config) -> ct:pal("CONNS on ~p = ~n~p", [ConnPid,Conns]), ok = proto:close(sconn1), - timer:sleep(100), + ct:sleep(100), {false,true} = {is_process_alive(Handle1),is_process_alive(ConnPid)}, ok = proto:kill_conn_proc(Handle2), - timer:sleep(100), + ct:sleep(100), {true,false} = {is_process_alive(Handle2),is_process_alive(ConnPid)}, Handle2 = proto:open(sconn2), % should've been reconnected already NewConnPid = ct_gen_conn:get_conn_pid(Handle2), @@ -227,12 +227,12 @@ names_to_single_conn_pids(_Config) -> ct:pal("CONNS on ~p = ~n~p", [NewConnPid,Conns1]), ok = proto:close(sconn2), - timer:sleep(100), + ct:sleep(100), {false,true} = {is_process_alive(Handle2),is_process_alive(NewConnPid)}, application:set_env(ct_test, reconnect, false), ok = proto:kill_conn_proc(Handle3), - timer:sleep(100), + ct:sleep(100), {false,false} = {is_process_alive(Handle3),is_process_alive(NewConnPid)}, ok. diff --git a/lib/common_test/test/ct_group_leader_SUITE_data/group_leader_SUITE.erl b/lib/common_test/test/ct_group_leader_SUITE_data/group_leader_SUITE.erl index 804f722081..bfdc78639e 100644 --- a/lib/common_test/test/ct_group_leader_SUITE_data/group_leader_SUITE.erl +++ b/lib/common_test/test/ct_group_leader_SUITE_data/group_leader_SUITE.erl @@ -258,14 +258,14 @@ gen_io(Label, N, Acc) -> %% (via ct logging functions) from an external process which has a %% different group leader than the test cases. unexp1(Config) -> - timer:sleep(1000), + ct:sleep(1000), gen_unexp_io(), - timer:sleep(1000), + ct:sleep(1000), check_unexp_io(Config), ok. unexp2(_) -> - timer:sleep(2000), + ct:sleep(2000), ok. gen_unexp_io() -> diff --git a/lib/common_test/test/ct_groups_test_1_SUITE.erl b/lib/common_test/test/ct_groups_test_1_SUITE.erl index e520a72227..d5de949554 100644 --- a/lib/common_test/test/ct_groups_test_1_SUITE.erl +++ b/lib/common_test/test/ct_groups_test_1_SUITE.erl @@ -302,7 +302,7 @@ test_events(groups_suite_1) -> {?eh,tc_done,{groups_11_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_11_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]}]; @@ -410,7 +410,7 @@ test_events(groups_suite_2) -> {?eh,tc_done,{groups_12_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_12_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_12_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_12_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]}]; @@ -505,7 +505,7 @@ test_events(groups_suites_1) -> {?eh,tc_done,{groups_11_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_11_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {?eh,tc_start,{groups_12_SUITE,init_per_suite}}, {?eh,tc_done,{groups_12_SUITE,init_per_suite,ok}}, @@ -596,7 +596,7 @@ test_events(groups_suites_1) -> {?eh,tc_done,{groups_12_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_12_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_12_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_12_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]}]; @@ -691,7 +691,7 @@ test_events(groups_dir_1) -> {?eh,tc_done,{groups_11_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_11_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {?eh,tc_start,{groups_12_SUITE,init_per_suite}}, {?eh,tc_done,{groups_12_SUITE,init_per_suite,ok}}, @@ -782,7 +782,7 @@ test_events(groups_dir_1) -> {?eh,tc_done,{groups_12_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_12_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_12_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_12_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]}]; @@ -878,7 +878,7 @@ test_events(groups_dirs_1) -> {?eh,tc_done,{groups_11_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_11_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {?eh,tc_start,{groups_12_SUITE,init_per_suite}}, {?eh,tc_done,{groups_12_SUITE,init_per_suite,ok}}, @@ -969,7 +969,7 @@ test_events(groups_dirs_1) -> {?eh,tc_done,{groups_12_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_12_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_12_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_12_SUITE,end_per_suite,ok}}, {?eh,tc_start,{groups_21_SUITE,init_per_suite}}, {?eh,tc_done,{groups_21_SUITE,init_per_suite,ok}}, @@ -1089,7 +1089,7 @@ test_events(groups_dirs_1) -> {groups_21_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_21_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_21_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_21_SUITE,end_per_suite,ok}}, {?eh,tc_start,{groups_22_SUITE,init_per_suite}}, {?eh,tc_done,{groups_22_SUITE,init_per_suite,ok}}, @@ -1223,6 +1223,6 @@ test_events(groups_dirs_1) -> {groups_22_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_22_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_22_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_22_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]}]. diff --git a/lib/common_test/test/ct_groups_test_1_SUITE_data/groups_1/test/groups_12_SUITE.erl b/lib/common_test/test/ct_groups_test_1_SUITE_data/groups_1/test/groups_12_SUITE.erl index ec90ef95d1..6f49f9a957 100644 --- a/lib/common_test/test/ct_groups_test_1_SUITE_data/groups_1/test/groups_12_SUITE.erl +++ b/lib/common_test/test/ct_groups_test_1_SUITE_data/groups_1/test/groups_12_SUITE.erl @@ -278,7 +278,7 @@ testcase_5a(Config) -> %% increase chance the done event will come %% during execution of subgroup (could be %% tricky to handle) - timer:sleep(3), + ct:sleep(3), ok. testcase_5b() -> []. diff --git a/lib/common_test/test/ct_groups_test_2_SUITE.erl b/lib/common_test/test/ct_groups_test_2_SUITE.erl index 8b0de98709..f41395e028 100644 --- a/lib/common_test/test/ct_groups_test_2_SUITE.erl +++ b/lib/common_test/test/ct_groups_test_2_SUITE.erl @@ -302,7 +302,7 @@ test_events(empty_group) -> {?eh,tc_done, {groups_22_SUITE,{end_per_group,test_group_8,[]},ok}}], {?eh,tc_start,{groups_22_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_22_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_22_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]. diff --git a/lib/common_test/test/ct_groups_test_2_SUITE_data/groups_2/groups_22_SUITE.erl b/lib/common_test/test/ct_groups_test_2_SUITE_data/groups_2/groups_22_SUITE.erl index 154c676d7e..80bb5ba69b 100644 --- a/lib/common_test/test/ct_groups_test_2_SUITE_data/groups_2/groups_22_SUITE.erl +++ b/lib/common_test/test/ct_groups_test_2_SUITE_data/groups_2/groups_22_SUITE.erl @@ -293,7 +293,7 @@ testcase_5a(Config) -> %% increase chance the done event will come %% during execution of subgroup (could be %% tricky to handle) - timer:sleep(3), + ct:sleep(3), ok. testcase_5b() -> []. diff --git a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/ct_update_config_SUITE.erl b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/ct_update_config_SUITE.erl index 3c1f5669e8..f8c8725602 100644 --- a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/ct_update_config_SUITE.erl +++ b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/ct_update_config_SUITE.erl @@ -26,21 +26,23 @@ -include("ct.hrl").
+-define(now, os:timestamp()).
+
%% Test server callback functions
init_per_suite(Config) ->
- [{init_per_suite,now()}|Config].
+ [{init_per_suite,?now}|Config].
end_per_suite(_Config) ->
ok.
init_per_testcase(_TestCase, Config) ->
- [{init_per_testcase,now()}|Config].
+ [{init_per_testcase,?now}|Config].
end_per_testcase(_TestCase, _Config) ->
ok.
init_per_group(GroupName, Config) ->
- [{init_per_group,now()}|Config].
+ [{init_per_group,?now}|Config].
end_per_group(GroupName, Config) ->
ok.
diff --git a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/cth_log_SUITE.erl b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/cth_log_SUITE.erl index 18dd07e87e..80ce248418 100644 --- a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/cth_log_SUITE.erl +++ b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/cth_log_SUITE.erl @@ -50,7 +50,7 @@ init_per_suite(Config) -> end_per_suite(Config) -> Gen = proplists:get_value(gen, Config), exit(Gen, kill), - timer:sleep(100), + ct:sleep(100), ok. %%-------------------------------------------------------------------- diff --git a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/empty_cth.erl b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/empty_cth.erl index 77783fccf5..5f8eae1f70 100644 --- a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/empty_cth.erl +++ b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/empty_cth.erl @@ -87,7 +87,7 @@ id(Opts) -> gen_event:notify(?CT_EVMGR_REF, #event{ name = cth, node = node(), data = {?MODULE, id, [Opts]}}), ct:log("~w:id called", [?MODULE]), - now(). + os:timestamp(). %% @doc Called before init_per_suite is called. Note that this callback is %% only called if the CTH is added before init_per_suite is run (eg. in a test diff --git a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/update_config_cth.erl b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/update_config_cth.erl index 2ee0d7da9c..55a1b9a130 100644 --- a/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/update_config_cth.erl +++ b/lib/common_test/test/ct_hooks_SUITE_data/cth/tests/update_config_cth.erl @@ -24,6 +24,7 @@ -include_lib("common_test/src/ct_util.hrl").
-include_lib("common_test/include/ct_event.hrl").
+-define(now, os:timestamp()).
%% CT Hooks
-compile(export_all).
@@ -33,44 +34,44 @@ init(Id, Opts) -> pre_init_per_suite(Suite, Config, State) ->
empty_cth:pre_init_per_suite(Suite,Config,State),
- {[{pre_init_per_suite,now()}|Config],State}.
+ {[{pre_init_per_suite,?now}|Config],State}.
post_init_per_suite(Suite,Config,Return,State) ->
empty_cth:post_init_per_suite(Suite,Config,Return,State),
- {[{post_init_per_suite,now()}|Return],State}.
+ {[{post_init_per_suite,?now}|Return],State}.
pre_end_per_suite(Suite,Config,State) ->
empty_cth:pre_end_per_suite(Suite,Config,State),
- {[{pre_end_per_suite,now()}|Config],State}.
+ {[{pre_end_per_suite,?now}|Config],State}.
post_end_per_suite(Suite,Config,Return,State) ->
empty_cth:post_end_per_suite(Suite,Config,Return,State),
- NewConfig = [{post_end_per_suite,now()}|Config],
+ NewConfig = [{post_end_per_suite,?now}|Config],
{NewConfig,NewConfig}.
pre_init_per_group(Group,Config,State) ->
empty_cth:pre_init_per_group(Group,Config,State),
- {[{pre_init_per_group,now()}|Config],State}.
+ {[{pre_init_per_group,?now}|Config],State}.
post_init_per_group(Group,Config,Return,State) ->
empty_cth:post_init_per_group(Group,Config,Return,State),
- {[{post_init_per_group,now()}|Return],State}.
+ {[{post_init_per_group,?now}|Return],State}.
pre_end_per_group(Group,Config,State) ->
empty_cth:pre_end_per_group(Group,Config,State),
- {[{pre_end_per_group,now()}|Config],State}.
+ {[{pre_end_per_group,?now}|Config],State}.
post_end_per_group(Group,Config,Return,State) ->
empty_cth:post_end_per_group(Group,Config,Return,State),
- {[{post_end_per_group,now()}|Config],State}.
+ {[{post_end_per_group,?now}|Config],State}.
pre_init_per_testcase(TC,Config,State) ->
empty_cth:pre_init_per_testcase(TC,Config,State),
- {[{pre_init_per_testcase,now()}|Config],State}.
+ {[{pre_init_per_testcase,?now}|Config],State}.
post_end_per_testcase(TC,Config,Return,State) ->
empty_cth:post_end_per_testcase(TC,Config,Return,State),
- {[{post_end_per_testcase,now()}|Config],State}.
+ {[{post_end_per_testcase,?now}|Config],State}.
on_tc_fail(TC, Reason, State) ->
empty_cth:on_tc_fail(TC,Reason,State).
diff --git a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl index 332e54d1a7..e7bbdc28a5 100644 --- a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl +++ b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl @@ -204,7 +204,7 @@ hello_required_exists(Config) -> ?NS:expect_do_reply('close-session',close,ok), ?ok = ct_netconfc:close_session(my_named_connection), - timer:sleep(500), + ct:sleep(500), %% Then check that it can be used again after the first is closed {ok,_Client2} = open_configured_success(my_named_connection,DataDir), @@ -486,8 +486,15 @@ action(Config) -> DataDir = ?config(data_dir,Config), {ok,Client} = open_success(DataDir), Data = [{myactionreturn,[{xmlns,"myns"}],["value"]}], - ?NS:expect_reply(action,{data,Data}), - {ok,Data} = ct_netconfc:action(Client,{myaction,[{xmlns,"myns"}],[]}), + %% test either to receive {data,Data} or {ok,Data}, + %% both need to be handled + {Reply,RetVal} = case element(3, now()) rem 2 of + 0 -> {{data,Data},{ok,Data}}; + 1 -> {{ok,Data},ok} + end, + ct:log("Client will receive {~w,Data}", [element(1,Reply)]), + ?NS:expect_reply(action,Reply), + RetVal = ct_netconfc:action(Client,{myaction,[{xmlns,"myns"}],[]}), ?NS:expect_do_reply('close-session',close,ok), ?ok = ct_netconfc:close_session(Client), ok. @@ -654,10 +661,10 @@ receive_chunked_data(Config) -> %% Spawn a process which will wait a bit for the client to send %% the request (below), then order the server to the chunks of the %% rpc-reply one by one. - spawn(fun() -> timer:sleep(500),?NS:hupp(send,Part1), - timer:sleep(100),?NS:hupp(send,Part2), - timer:sleep(100),?NS:hupp(send,Part3), - timer:sleep(100),?NS:hupp(send,Part4) + spawn(fun() -> ct:sleep(500),?NS:hupp(send,Part1), + ct:sleep(100),?NS:hupp(send,Part2), + ct:sleep(100),?NS:hupp(send,Part3), + ct:sleep(100),?NS:hupp(send,Part4) end), %% Order server to expect a get - then the process above will make @@ -702,8 +709,8 @@ timeout_receive_chunked_data(Config) -> %% Spawn a process which will wait a bit for the client to send %% the request (below), then order the server to the chunks of the %% rpc-reply one by one. - spawn(fun() -> timer:sleep(500),?NS:hupp(send,Part1), - timer:sleep(100),?NS:hupp(send,Part2) + spawn(fun() -> ct:sleep(500),?NS:hupp(send,Part1), + ct:sleep(100),?NS:hupp(send,Part2) end), %% Order server to expect a get - then the process above will make @@ -748,9 +755,9 @@ close_while_waiting_for_chunked_data(Config) -> %% Spawn a process which will wait a bit for the client to send %% the request (below), then order the server to the chunks of the %% rpc-reply one by one. - spawn(fun() -> timer:sleep(500),?NS:hupp(send,Part1), - timer:sleep(100),?NS:hupp(send,Part2), - timer:sleep(100),?NS:hupp(kill) + spawn(fun() -> ct:sleep(500),?NS:hupp(send,Part1), + ct:sleep(100),?NS:hupp(send,Part2), + ct:sleep(100),?NS:hupp(kill) end), %% Order server to expect a get - then the process above will make @@ -766,7 +773,7 @@ connection_crash(Config) -> %% Test that if the test survives killing the connection %% process. Earlier this caused ct_util_server to terminate, and %% this aborting the complete test run. - spawn(fun() -> timer:sleep(500),exit(Client,kill) end), + spawn(fun() -> ct:sleep(500),exit(Client,kill) end), ?NS:expect(get), {error,{closed,killed}}=ct_netconfc:get(Client,{server,[{xmlns,"myns"}],[]}), ok. diff --git a/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl b/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl index f7c7b891bb..e4bc396536 100644 --- a/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl +++ b/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl @@ -351,7 +351,7 @@ check_expected(SessionId,ConnRef,Msg) -> do(ConnRef, Do), reply(ConnRef,Reply); error -> - timer:sleep(1000), + ct:sleep(1000), exit({error,{got_unexpected,SessionId,Msg,ets:tab2list(ns_tab)}}) end. @@ -548,8 +548,13 @@ make_msg({hello,SessionId,Stuff}) -> SessionIdXml/binary,"</hello>">>); make_msg(ok) -> xml(rpc_reply("<ok/>")); + +make_msg({ok,Data}) -> + xml(rpc_reply(from_simple({ok,Data}))); + make_msg({data,Data}) -> xml(rpc_reply(from_simple({data,Data}))); + make_msg(event) -> xml(<<"<notification xmlns=\"",?NETCONF_NOTIF_NAMESPACE,"\">" "<eventTime>2012-06-14T14:50:54+02:00</eventTime>" diff --git a/lib/common_test/test/ct_pre_post_test_io_SUITE.erl b/lib/common_test/test/ct_pre_post_test_io_SUITE.erl index 5de1ecc2bd..1e6018f442 100644 --- a/lib/common_test/test/ct_pre_post_test_io_SUITE.erl +++ b/lib/common_test/test/ct_pre_post_test_io_SUITE.erl @@ -91,27 +91,27 @@ pre_post_io(Config) -> spawn(fun() -> ct:pal("CONTROLLER: Started!", []), %% --- test run 1 --- - timer:sleep(3000), + ct:sleep(3000), ct:pal("CONTROLLER: Handle remote events = true", []), ok = ct_test_support:ct_rpc({cth_log_redirect, handle_remote_events, [true]}, Config), - timer:sleep(2000), + ct:sleep(2000), ct:pal("CONTROLLER: Proceeding with test run #1!", []), ok = ct_test_support:ct_rpc({cth_ctrl,proceed,[]}, Config), - timer:sleep(6000), + ct:sleep(6000), ct:pal("CONTROLLER: Proceeding with shutdown #1!", []), ok = ct_test_support:ct_rpc({cth_ctrl,proceed,[]}, Config), %% --- test run 2 --- - timer:sleep(3000), + ct:sleep(3000), ct:pal("CONTROLLER: Handle remote events = true", []), ok = ct_test_support:ct_rpc({cth_log_redirect, handle_remote_events, [true]}, Config), - timer:sleep(2000), + ct:sleep(2000), ct:pal("CONTROLLER: Proceeding with test run #2!", []), ok = ct_test_support:ct_rpc({cth_ctrl,proceed,[]}, Config), - timer:sleep(6000), + ct:sleep(6000), ct:pal("CONTROLLER: Proceeding with shutdown #2!", []), ok = ct_test_support:ct_rpc({cth_ctrl,proceed,[]}, Config) end), diff --git a/lib/common_test/test/ct_repeat_1_SUITE.erl b/lib/common_test/test/ct_repeat_1_SUITE.erl index e37aeb196c..50e07608f6 100644 --- a/lib/common_test/test/ct_repeat_1_SUITE.erl +++ b/lib/common_test/test/ct_repeat_1_SUITE.erl @@ -220,8 +220,7 @@ test_events(repeat_cs_and_grs) -> {?eh,test_stats,{1,1,{0,0}}}, [{?eh,tc_done,{repeat_1_SUITE,{init_per_group,gr_fail_result,[]},ok}}, {?eh,test_stats,{2,1,{0,0}}}, - {?eh,tc_done,{repeat_1_SUITE,{end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {?eh,tc_done,{repeat_1_SUITE,{end_per_group,gr_fail_result,[]},ok}}], {?eh,test_stats,{3,1,{0,0}}}, [{?eh,tc_done,{repeat_1_SUITE,{init_per_group,gr_fail_init,[]}, {failed,{error,fails_on_purpose}}}}, @@ -242,8 +241,7 @@ test_events(repeat_cs_and_grs) -> {?eh,test_stats,{5,2,{0,1}}}, [{?eh,tc_done,{repeat_1_SUITE,{init_per_group,gr_fail_result,[]},ok}}, {?eh,test_stats,{6,2,{0,1}}}, - {?eh,tc_done,{repeat_1_SUITE,{end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {?eh,tc_done,{repeat_1_SUITE,{end_per_group,gr_fail_result,[]},ok}}], {?eh,test_stats,{7,2,{0,1}}}, [{?eh,tc_done,{repeat_1_SUITE,{init_per_group,gr_fail_init,[]}, {failed,{error,fails_on_purpose}}}}, @@ -289,8 +287,7 @@ test_events(repeat_seq) -> {init_per_group,gr_fail_result,[]},ok}}, {?eh,test_stats,{4,2,{0,2}}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result,[]},ok}}], {?eh,tc_auto_skip,{repeat_1_SUITE,{tc_ok_2,repeat_seq_2}, {group_result,gr_fail_result,failed}}}, {?eh,test_stats,{4,2,{0,3}}}, @@ -402,8 +399,7 @@ test_events(repeat_gr_until_any_ok) -> [{?eh,tc_done,{repeat_1_SUITE, {init_per_group,gr_fail_result,[]},ok}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE,tc_fail_1, {failed,{error,{{badmatch,2},'_'}}}}}, {?eh,test_stats,{1,1,{0,0}}}, @@ -418,8 +414,7 @@ test_events(repeat_gr_until_any_ok) -> [{?eh,tc_done,{repeat_1_SUITE, {init_per_group,gr_fail_result_then_ok,[]},ok}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result_then_ok,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result_then_ok,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE, {end_per_group,repeat_gr_until_any_ok_1, [{repeat_until_any_ok,3}]},ok}}], @@ -441,8 +436,7 @@ test_events(repeat_gr_until_any_ok) -> {init_per_group,repeat_gr_until_any_ok_2, [{repeat_until_any_ok,3}]},ok}}, [{?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE,tc_fail_1, {failed,{error,{{badmatch,2},'_'}}}}}, {?eh,test_stats,{5,5,{0,2}}}, @@ -675,8 +669,7 @@ test_events(repeat_gr_until_any_fail) -> {repeat_1_SUITE,{end_per_group,gr_ok_then_fail_result,[]}}}, {?eh,tc_done, {repeat_1_SUITE, - {end_per_group,gr_ok_then_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_ok_then_fail_result,[]},ok}}], {?eh,tc_start,{repeat_1_SUITE,tc_ok_2}}, {?eh,tc_done,{repeat_1_SUITE,tc_ok_2,ok}}, {?eh,test_stats,{8,0,{0,0}}}, @@ -938,8 +931,7 @@ test_events(repeat_gr_until_all_ok) -> {?eh,tc_done,{repeat_1_SUITE,tc_ok_1,ok}}, {?eh,test_stats,{3,1,{0,0}}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result_then_ok,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result_then_ok,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE, {end_per_group,repeat_gr_until_all_ok_1, [{repeat_until_all_ok,3}]},ok}}], @@ -1113,8 +1105,7 @@ test_events(repeat_gr_until_all_fail) -> gr_ok_then_fail_result,[]},ok}}, {?eh,test_stats,{3,3,{0,2}}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_ok_then_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_ok_then_fail_result,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE, {end_per_group,repeat_gr_until_all_fail_1, [{repeat_until_all_fail,2}]},ok}}], @@ -1148,8 +1139,7 @@ test_events(repeat_gr_until_all_fail) -> {init_per_group,repeat_gr_until_all_fail_3, [{repeat_until_all_fail,3}]},ok}}, [{?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE,tc_ok_then_fail_1,ok}}, {?eh,test_stats,{6,5,{0,3}}}, {?eh,tc_done,{repeat_1_SUITE, @@ -1159,8 +1149,7 @@ test_events(repeat_gr_until_all_fail) -> {init_per_group,repeat_gr_until_all_fail_3, [{repeat_until_all_fail,2}]},ok}}, [{?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_fail_result,[]},ok}}], {?eh,tc_done,{repeat_1_SUITE,tc_ok_then_fail_1, {failed,{error,failing_this_time}}}}, {?eh,test_stats,{7,6,{0,3}}}, @@ -1263,8 +1252,7 @@ test_events(repeat_seq_until_any_fail) -> {init_per_group,repeat_seq_until_any_fail_4, [{repeat_until_any_fail,2},sequence]},ok}}, [{?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_ok_then_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_ok_then_fail_result,[]},ok}}], {?eh,tc_auto_skip,{repeat_1_SUITE,{tc_ok_1,gr_ok_1}, {group_result,gr_ok_then_fail_result,failed}}}, {?eh,test_stats,{19,1,{0,3}}}, @@ -1473,8 +1461,7 @@ test_events(repeat_shuffled_seq_until_any_fail) -> [{?eh,tc_start,{repeat_1_SUITE, {end_per_group,gr_ok_then_fail_result,[]}}}, {?eh,tc_done,{repeat_1_SUITE, - {end_per_group,gr_ok_then_fail_result,[]}, - {return_group_result,failed}}}], + {end_per_group,gr_ok_then_fail_result,[]},ok}}], {?eh,tc_start,{repeat_1_SUITE, {end_per_group,repeat_shuffled_seq_until_any_fail_4, [{shuffle,repeated},{repeat_until_any_fail,2},sequence]}}}, diff --git a/lib/common_test/test/ct_repeat_testrun_SUITE_data/a_test/r1_SUITE.erl b/lib/common_test/test/ct_repeat_testrun_SUITE_data/a_test/r1_SUITE.erl index 3fd5943691..3d7049a9c4 100644 --- a/lib/common_test/test/ct_repeat_testrun_SUITE_data/a_test/r1_SUITE.erl +++ b/lib/common_test/test/ct_repeat_testrun_SUITE_data/a_test/r1_SUITE.erl @@ -68,7 +68,7 @@ end_per_testcase(_Case, Config) -> %%%----------------------------------------------------------------- %%% Test cases tc1(_Config) -> - timer:sleep(10000), + ct:sleep(10000), ok. tc2(_Config) -> diff --git a/lib/common_test/test/ct_repeat_testrun_SUITE_data/b_test/r2_SUITE.erl b/lib/common_test/test/ct_repeat_testrun_SUITE_data/b_test/r2_SUITE.erl index dc9abc2863..e4f6e7dcc1 100644 --- a/lib/common_test/test/ct_repeat_testrun_SUITE_data/b_test/r2_SUITE.erl +++ b/lib/common_test/test/ct_repeat_testrun_SUITE_data/b_test/r2_SUITE.erl @@ -68,7 +68,7 @@ end_per_testcase(_Case, Config) -> %%%----------------------------------------------------------------- %%% Test cases tc1(_Config) -> - %% timer:sleep(3000), + %% ct:sleep(3000), ok. tc2(_Config) -> diff --git a/lib/common_test/test/ct_sequence_1_SUITE.erl b/lib/common_test/test/ct_sequence_1_SUITE.erl index 5a775a1117..4055cd789e 100644 --- a/lib/common_test/test/ct_sequence_1_SUITE.erl +++ b/lib/common_test/test/ct_sequence_1_SUITE.erl @@ -182,8 +182,7 @@ test_events(subgroup_return_fail) -> {?eh,test_stats,{0,1,{0,0}}}, {?eh,tc_start, {subgroups_1_SUITE,{end_per_group,return_fail,[]}}}, - {?eh,tc_done,{subgroups_1_SUITE,{end_per_group,return_fail,[]}, - {return_group_result,failed}}}], + {?eh,tc_done,{subgroups_1_SUITE,{end_per_group,return_fail,[]},ok}}], {?eh,tc_auto_skip, {subgroups_1_SUITE,{ok_tc,ok_group}, {group_result,return_fail,failed}}}, @@ -191,8 +190,7 @@ test_events(subgroup_return_fail) -> {?eh,tc_start, {subgroups_1_SUITE,{end_per_group,subgroup_return_fail,[sequence]}}}, {?eh,tc_done, - {subgroups_1_SUITE,{end_per_group,subgroup_return_fail,[sequence]}, - {return_group_result,failed}}}], + {subgroups_1_SUITE,{end_per_group,subgroup_return_fail,[sequence]},ok}}], {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -221,8 +219,7 @@ test_events(subgroup_init_fail) -> {?eh,test_stats,{0,0,{0,2}}}, {?eh,tc_start,{subgroups_1_SUITE,{end_per_group,subgroup_init_fail,[sequence]}}}, {?eh,tc_done,{subgroups_1_SUITE, - {end_per_group,subgroup_init_fail,[sequence]}, - {return_group_result,failed}}}], + {end_per_group,subgroup_init_fail,[sequence]},ok}}], {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -245,8 +242,7 @@ test_events(subgroup_after_failed_case) -> {?eh,tc_start,{subgroups_1_SUITE, {end_per_group,subgroup_after_failed_case,[sequence]}}}, {?eh,tc_done,{subgroups_1_SUITE, - {end_per_group,subgroup_after_failed_case,[sequence]}, - {return_group_result,failed}}}], + {end_per_group,subgroup_after_failed_case,[sequence]},ok}}], {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -266,16 +262,14 @@ test_events(case_after_subgroup_return_fail) -> {?eh,tc_done,{subgroups_1_SUITE,failing_tc,{failed,{error,{{badmatch,3},'_'}}}}}, {?eh,test_stats,{0,1,{0,0}}}, {?eh,tc_start,{subgroups_1_SUITE,{end_per_group,return_fail,[]}}}, - {?eh,tc_done,{subgroups_1_SUITE,{end_per_group,return_fail,[]}, - {return_group_result,failed}}}], + {?eh,tc_done,{subgroups_1_SUITE,{end_per_group,return_fail,[]},ok}}], {?eh,tc_auto_skip,{subgroups_1_SUITE,{ok_tc,case_after_subgroup_return_fail}, {group_result,return_fail,failed}}}, {?eh,test_stats,{0,1,{0,1}}}, {?eh,tc_start,{subgroups_1_SUITE, {end_per_group,case_after_subgroup_return_fail,[sequence]}}}, {?eh,tc_done,{subgroups_1_SUITE, - {end_per_group,case_after_subgroup_return_fail,[sequence]}, - {return_group_result,failed}}}], + {end_per_group,case_after_subgroup_return_fail,[sequence]},ok}}], {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -310,8 +304,7 @@ test_events(case_after_subgroup_fail_init) -> {?eh,tc_start,{subgroups_1_SUITE, {end_per_group,case_after_subgroup_fail_init,[sequence]}}}, {?eh,tc_done,{subgroups_1_SUITE, - {end_per_group,case_after_subgroup_fail_init,[sequence]}, - {return_group_result,failed}}}], + {end_per_group,case_after_subgroup_fail_init,[sequence]},ok}}], {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]. diff --git a/lib/common_test/test/ct_shell_SUITE.erl b/lib/common_test/test/ct_shell_SUITE.erl index 4b8c43d800..70c0ab8127 100644 --- a/lib/common_test/test/ct_shell_SUITE.erl +++ b/lib/common_test/test/ct_shell_SUITE.erl @@ -93,7 +93,7 @@ start_interactive(Config) -> test_server:format(Level, "ct_util_server not stopped on ~p yet, waiting 5 s...~n", [CTNode]), - timer:sleep(5000), + ct:sleep(5000), undefined = rpc:call(CTNode, erlang, whereis, [ct_util_server]) end, Events = ct_test_support:get_events(ERPid, Config), diff --git a/lib/common_test/test/ct_skip_SUITE_data/skip/test/auto_skip_4_SUITE.erl b/lib/common_test/test/ct_skip_SUITE_data/skip/test/auto_skip_4_SUITE.erl index 825846cd55..89e202a404 100644 --- a/lib/common_test/test/ct_skip_SUITE_data/skip/test/auto_skip_4_SUITE.erl +++ b/lib/common_test/test/ct_skip_SUITE_data/skip/test/auto_skip_4_SUITE.erl @@ -72,7 +72,7 @@ end_per_group(_GroupName, _Config) -> %% Reason = term() %%-------------------------------------------------------------------- init_per_testcase(tc1, Config) -> - timer:sleep(5000), + ct:sleep(5000), Config; init_per_testcase(_TestCase, Config) -> Config. diff --git a/lib/common_test/test/ct_smoke_test_SUITE.erl b/lib/common_test/test/ct_smoke_test_SUITE.erl index 49b38361e2..6077946c33 100644 --- a/lib/common_test/test/ct_smoke_test_SUITE.erl +++ b/lib/common_test/test/ct_smoke_test_SUITE.erl @@ -480,7 +480,7 @@ events(Test) when Test == dir1 ; Test == dir2 ; {Suite,tc4,{skipped,"Skipping this one"}}}, {?eh,test_stats,{7,0,{1,0}}}, {?eh,tc_start,{Suite,end_per_suite}}, - {?eh,tc_done,{Suite,end_per_suite,ips_data}}, + {?eh,tc_done,{Suite,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -517,7 +517,7 @@ events(Test) when Test == dir1_2 ; Test == suite11_21 -> {happy_11_SUITE,tc4,{skipped,"Skipping this one"}}}, {?eh,test_stats,{7,0,{1,0}}}, {?eh,tc_start,{happy_11_SUITE,end_per_suite}}, - {?eh,tc_done,{happy_11_SUITE,end_per_suite,ips_data}}, + {?eh,tc_done,{happy_11_SUITE,end_per_suite,ok}}, {?eh,tc_start,{happy_21_SUITE,init_per_suite}}, {?eh,tc_done,{happy_21_SUITE,init_per_suite,ok}}, {?eh,tc_start,{happy_21_SUITE,tc1}}, @@ -546,7 +546,7 @@ events(Test) when Test == dir1_2 ; Test == suite11_21 -> {happy_21_SUITE,tc4,{skipped,"Skipping this one"}}}, {?eh,test_stats,{14,0,{2,0}}}, {?eh,tc_start,{happy_21_SUITE,end_per_suite}}, - {?eh,tc_done,{happy_21_SUITE,end_per_suite,ips_data}}, + {?eh,tc_done,{happy_21_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -563,7 +563,7 @@ events(Test) when Test == tc111 ; Test == tc211 -> {?eh,tc_done,{Suite,tc1,ok}}, {?eh,test_stats,{1,0,{0,0}}}, {?eh,tc_start,{Suite,end_per_suite}}, - {?eh,tc_done,{Suite,end_per_suite,ips_data}}, + {?eh,tc_done,{Suite,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]; @@ -582,7 +582,7 @@ events(tc111_112) -> {?eh,tc_done,{happy_11_SUITE,tc2,ok}}, {?eh,test_stats,{2,0,{0,0}}}, {?eh,tc_start,{happy_11_SUITE,end_per_suite}}, - {?eh,tc_done,{happy_11_SUITE,end_per_suite,ips_data}}, + {?eh,tc_done,{happy_11_SUITE,end_per_suite,ok}}, {?eh,test_done,{'DEF','STOP_TIME'}}, {?eh,stop_logging,[]} ]. diff --git a/lib/common_test/test/ct_snmp_SUITE_data/snmp_SUITE.erl b/lib/common_test/test/ct_snmp_SUITE_data/snmp_SUITE.erl index e20832e1e7..07f7bf02e4 100644 --- a/lib/common_test/test/ct_snmp_SUITE_data/snmp_SUITE.erl +++ b/lib/common_test/test/ct_snmp_SUITE_data/snmp_SUITE.erl @@ -117,12 +117,12 @@ break(_Config) -> start_stop(Config) -> ok = ct_snmp:start(Config,snmp1,snmp_app1), - timer:sleep(1000), + ct:sleep(1000), {snmp,_,_} = lists:keyfind(snmp,1,application:which_applications()), [_|_] = filelib:wildcard("*/*.conf",?config(priv_dir,Config)), ok = ct_snmp:stop(Config), - timer:sleep(1000), + ct:sleep(1000), false = lists:keyfind(snmp,1,application:which_applications()), [] = filelib:wildcard("*/*.conf",?config(priv_dir,Config)), ok. diff --git a/lib/common_test/test/ct_telnet_SUITE.erl b/lib/common_test/test/ct_telnet_SUITE.erl index 84e69c2b54..62cb821ede 100644 --- a/lib/common_test/test/ct_telnet_SUITE.erl +++ b/lib/common_test/test/ct_telnet_SUITE.erl @@ -203,7 +203,9 @@ telnet_config(_, LogType) -> {command_timeout,10000}, {reconnection_attempts,0}, {reconnection_interval,0}, - {keep_alive,true}]} | + {keep_alive,true}, + {poll_limit,10}, + {poll_interval,1000}]} | if LogType == legacy -> [{ct_conn_log,[]}]; true -> diff --git a/lib/common_test/test/ct_telnet_SUITE_data/ct_telnet_own_server_SUITE.erl b/lib/common_test/test/ct_telnet_SUITE_data/ct_telnet_own_server_SUITE.erl index bd5d76266a..1d3f5918d2 100644 --- a/lib/common_test/test/ct_telnet_SUITE_data/ct_telnet_own_server_SUITE.erl +++ b/lib/common_test/test/ct_telnet_SUITE_data/ct_telnet_own_server_SUITE.erl @@ -283,14 +283,14 @@ large_string(_) -> %% yield the same result as the single request case. ok = ct_telnet:send(Handle, "echo_sep "++BigString), - timer:sleep(1000), + ct:sleep(1000), {ok,Data1} = ct_telnet:get_data(Handle), ct:log("[GET DATA #1] Received ~w chars: ~s", [length(lists:flatten(Data1)),Data1]), VerifyStr = [C || C <- lists:flatten(Data1), C/=$ , C/=$\r, C/=$\n, C/=$>], ok = ct_telnet:send(Handle, "echo_sep "++BigString), - timer:sleep(50), + ct:sleep(50), {ok,Data2} = ct_telnet:get_data(Handle), ct:log("[GET DATA #2] Received ~w chars: ~s", [length(lists:flatten(Data2)),Data2]), VerifyStr = [C || C <- lists:flatten(Data2), C/=$ , C/=$\r, C/=$\n, C/=$>], @@ -316,7 +316,7 @@ server_speaks(_) -> "echo_no_prompt This is the second message"), %% Let ct_telnet_client get an idle timeout. This should print the %% two messages to the log. Note that the buffers are not flushed here! - timer:sleep(15000), + ct:sleep(15000), ok = ct_telnet_client:send_data(Backdoor, "echo_no_prompt This is the third message"), {ok,_} = ct_telnet:expect(Handle, ["first.*second.*third"], @@ -326,7 +326,7 @@ server_speaks(_) -> ok = ct_telnet_client:send_data(Backdoor, "echo_no_prompt This is the fourth message"), %% give the server time to respond - timer:sleep(2000), + ct:sleep(2000), %% closing the connection should print last message in log ok = ct_telnet:close(Handle), ok. @@ -338,11 +338,11 @@ server_disconnects(_) -> ok = ct_telnet:send(Handle, "disconnect_after 1500"), %% wait until the get_data operation (triggered by send/2) times out %% before sending the msg - timer:sleep(500), + ct:sleep(500), ok = ct_telnet:send(Handle, "echo_no_prompt This is the message"), %% when the server closes the connection, the last message should be %% printed in the log - timer:sleep(3000), + ct:sleep(3000), _ = ct_telnet:close(Handle), ok. diff --git a/lib/common_test/test/ct_test_server_if_1_SUITE_data/test_server_if/test/ts_if_1_SUITE.erl b/lib/common_test/test/ct_test_server_if_1_SUITE_data/test_server_if/test/ts_if_1_SUITE.erl index 06fa6ac638..d30feb0bd2 100644 --- a/lib/common_test/test/ct_test_server_if_1_SUITE_data/test_server_if/test/ts_if_1_SUITE.erl +++ b/lib/common_test/test/ct_test_server_if_1_SUITE_data/test_server_if/test/ts_if_1_SUITE.erl @@ -76,7 +76,7 @@ end_per_group(_GroupName, _Config) -> %% Reason = term() %%-------------------------------------------------------------------- init_per_testcase(tc1, Config) -> - timer:sleep(5000), + ct:sleep(5000), Config; init_per_testcase(tc8, _Config) -> {skip,"tc8 skipped"}; @@ -92,7 +92,7 @@ init_per_testcase(_TestCase, Config) -> %% Config0 = Config1 = [tuple()] %%-------------------------------------------------------------------- end_per_testcase(tc2, Config) -> - timer:sleep(5000); + ct:sleep(5000); end_per_testcase(tc12, Config) -> ct:comment("end_per_testcase(tc12) called!"), ct:pal("end_per_testcase(tc12) called!", []), @@ -146,7 +146,7 @@ tc2(_) -> timeout_in_end_per_testcase. tc3(_) -> - timer:sleep(5000). + ct:sleep(5000). tc4(_) -> exit(failed_on_purpose). @@ -186,7 +186,7 @@ gtc2(_) -> tc12(_) -> F = fun() -> ct:abort_current_testcase('stopping tc12') end, spawn(F), - timer:sleep(1000), + ct:sleep(1000), exit(should_have_been_aborted). tc13(_) -> diff --git a/lib/common_test/test/ct_test_support.erl b/lib/common_test/test/ct_test_support.erl index 2c1f98d63b..ffef8187f3 100644 --- a/lib/common_test/test/ct_test_support.erl +++ b/lib/common_test/test/ct_test_support.erl @@ -51,12 +51,12 @@ init_per_suite(Config) -> init_per_suite(Config, 50). init_per_suite(Config, Level) -> + ScaleFactor = test_server:timetrap_scale_factor(), case os:type() of {win32, _} -> %% Extend timeout to 1 hour for windows as starting node %% can take a long time there - test_server:timetrap( 60*60*1000 * - test_server:timetrap_scale_factor()); + test_server:timetrap( 60*60*1000 * ScaleFactor ); _ -> ok end, @@ -67,6 +67,16 @@ init_per_suite(Config, Level) -> _ -> ok end, + + {Mult,Scale} = test_server_ctrl:get_timetrap_parameters(), + test_server:format(Level, "Timetrap multiplier: ~w~n", [Mult]), + if Scale == true -> + test_server:format(Level, "Timetrap scale factor: ~w~n", + [ScaleFactor]); + true -> + ok + end, + start_slave(Config, Level). start_slave(Config, Level) -> @@ -277,10 +287,13 @@ run_ct_run_test(Opts,Config) -> Level = proplists:get_value(trace_level, Config), test_server:format(Level, "~n[RUN #1] Calling ct:run_test(~p) on ~p~n", [Opts, CTNode]), - T0 = now(), + + T0 = erlang:monotonic_time(), CtRunTestResult = rpc:call(CTNode, ct, run_test, [Opts]), + T1 = erlang:monotonic_time(), + Elapsed = erlang:convert_time_unit(T1-T0, native, milli_seconds), test_server:format(Level, "~n[RUN #1] Got return value ~p after ~p ms~n", - [CtRunTestResult,trunc(timer:now_diff(now(), T0)/1000)]), + [CtRunTestResult,Elapsed]), case rpc:call(CTNode, erlang, whereis, [ct_util_server]) of undefined -> ok; @@ -303,10 +316,12 @@ run_ct_script_start(Opts, Config) -> [common_test, run_test_start_opts, Opts1]), test_server:format(Level, "[RUN #2] Calling ct_run:script_start() on ~p~n", [CTNode]), - T0 = now(), + T0 = erlang:monotonic_time(), ExitStatus = rpc:call(CTNode, ct_run, script_start, []), + T1 = erlang:monotonic_time(), + Elapsed = erlang:convert_time_unit(T1-T0, native, milli_seconds), test_server:format(Level, "[RUN #2] Got exit status value ~p after ~p ms~n", - [ExitStatus,trunc(timer:now_diff(now(), T0)/1000)]), + [ExitStatus,Elapsed]), ExitStatus. check_result({_Ok,Failed,{_UserSkipped,_AutoSkipped}},1,_Opts) @@ -398,7 +413,7 @@ ct_rpc({M,F,A}, Config) -> %%%----------------------------------------------------------------- %%% random_error/1 random_error(Config) when is_list(Config) -> - random:seed(now()), + random:seed(os:timestamp()), Gen = fun(0,_) -> ok; (N,Fun) -> Fun(N-1, Fun) end, Gen(random:uniform(100), Gen), diff --git a/lib/common_test/test/ct_testspec_1_SUITE.erl b/lib/common_test/test/ct_testspec_1_SUITE.erl index c2670316b6..bc19283a47 100644 --- a/lib/common_test/test/ct_testspec_1_SUITE.erl +++ b/lib/common_test/test/ct_testspec_1_SUITE.erl @@ -795,7 +795,7 @@ test_events(skip_all_groups) -> {?eh,test_stats,{0,0,{12,0}}}, {?eh,tc_user_skip,{groups_11_SUITE,{end_per_group,test_group_4},"SKIPPED!"}}, {?eh,tc_start,{groups_11_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {negative,{?eh,tc_start,'_'},{?eh,stop_logging,'_'}} ]; @@ -840,7 +840,7 @@ test_events(skip_group) -> {?eh,test_stats,{2,0,{6,0}}}, {?eh,tc_user_skip,{groups_11_SUITE,{end_per_group,test_group_2}, "SKIPPED!"}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {negative,{?eh,tc_start,'_'},{?eh,stop_logging,'_'}} ]; @@ -876,7 +876,7 @@ test_events(skip_group_all_testcases) -> {?eh,test_stats,{0,0,{4,0}}}, {?eh,tc_user_skip,{groups_11_SUITE,{end_per_group,test_group_1b}, "SKIPPED!"}}, - {?eh,tc_done,{groups_11_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_11_SUITE,end_per_suite,ok}}, {negative,{?eh,tc_start,'_'},{?eh,stop_logging,'_'}} ]; @@ -1065,7 +1065,7 @@ test_events(skip_subgroup) -> {?eh,tc_done,{groups_12_SUITE,{end_per_group,test_group_4,[]},ok}}], {?eh,tc_start,{groups_12_SUITE,end_per_suite}}, - {?eh,tc_done,{groups_12_SUITE,end_per_suite,init}}, + {?eh,tc_done,{groups_12_SUITE,end_per_suite,ok}}, {negative,{?eh,tc_start,'_'},{?eh,stop_logging,'_'}} ]; diff --git a/lib/common_test/test/ct_testspec_1_SUITE_data/groups_1/groups_12_SUITE.erl b/lib/common_test/test/ct_testspec_1_SUITE_data/groups_1/groups_12_SUITE.erl index 69c06f9b83..e5de5df1a8 100644 --- a/lib/common_test/test/ct_testspec_1_SUITE_data/groups_1/groups_12_SUITE.erl +++ b/lib/common_test/test/ct_testspec_1_SUITE_data/groups_1/groups_12_SUITE.erl @@ -285,7 +285,7 @@ testcase_5a(Config) -> %% increase chance the done event will come %% during execution of subgroup (could be %% tricky to handle) - timer:sleep(3), + ct:sleep(3), ok. testcase_5b() -> []. diff --git a/lib/common_test/test/ct_testspec_1_SUITE_data/groups_2/groups_22_SUITE.erl b/lib/common_test/test/ct_testspec_1_SUITE_data/groups_2/groups_22_SUITE.erl index cd517876df..dcd361d658 100644 --- a/lib/common_test/test/ct_testspec_1_SUITE_data/groups_2/groups_22_SUITE.erl +++ b/lib/common_test/test/ct_testspec_1_SUITE_data/groups_2/groups_22_SUITE.erl @@ -278,7 +278,7 @@ testcase_5a(Config) -> %% increase chance the done event will come %% during execution of subgroup (could be %% tricky to handle) - timer:sleep(3), + ct:sleep(3), ok. testcase_5b() -> []. diff --git a/lib/common_test/test/telnet_server.erl b/lib/common_test/test/telnet_server.erl index d25ee62d38..e073f0bfa4 100644 --- a/lib/common_test/test/telnet_server.erl +++ b/lib/common_test/test/telnet_server.erl @@ -59,7 +59,7 @@ init(Opts) -> accept(State), ok = gen_tcp:close(LSock), dbg("telnet_server closed the listen socket ~p\n", [LSock]), - timer:sleep(1000), + ct:sleep(1000), ok. listen(0, _Port, _Opts) -> @@ -68,7 +68,7 @@ listen(Retries, Port, Opts) -> case gen_tcp:listen(Port, Opts) of {error,eaddrinuse} -> dbg("Listen port not released, trying again..."), - timer:sleep(5000), + ct:sleep(5000), listen(Retries-1, Port, Opts); Ok = {ok,_LSock} -> Ok; @@ -220,7 +220,7 @@ do_handle_data("echo_sep " ++ Data,State) -> Msgs = string:tokens(Data," "), lists:foreach(fun(Msg) -> send(Msg,State), - timer:sleep(10) + ct:sleep(10) end, Msgs), send("\r\n> ",State), {ok,State}; @@ -284,15 +284,15 @@ send(Data,State) -> send_loop(T,Data,State) -> dbg("Server sending ~p in loop for ~w ms...~n",[Data,T]), - send_loop(now(),T,Data,State). + send_loop(os:timestamp(),T,Data,State). send_loop(T0,T,Data,State) -> - ElapsedMS = trunc(timer:now_diff(now(),T0)/1000), + ElapsedMS = trunc(timer:now_diff(os:timestamp(),T0)/1000), if ElapsedMS >= T -> ok; true -> send(Data,State), - timer:sleep(500), + ct:sleep(500), send_loop(T0,T,Data,State) end. @@ -314,7 +314,7 @@ dbg(_F,_A) -> io:format("[telnet_server, ~s]\n" ++ _F,[TS|_A]). timestamp() -> - {MS,S,US} = now(), + {MS,S,US} = os:timestamp(), {{Year,Month,Day}, {Hour,Min,Sec}} = calendar:now_to_local_time({MS,S,US}), MilliSec = trunc(US/1000), diff --git a/lib/compiler/src/Makefile b/lib/compiler/src/Makefile index 2032392821..7c4cebdc28 100644 --- a/lib/compiler/src/Makefile +++ b/lib/compiler/src/Makefile @@ -159,6 +159,10 @@ $(EBIN)/beam_asm.beam: $(ESRC)/beam_asm.erl $(EGEN)/beam_opcodes.hrl $(EBIN)/cerl_inline.beam: $(ESRC)/cerl_inline.erl $(V_ERLC) $(ERL_COMPILE_FLAGS) +nowarn_shadow_vars -o$(EBIN) $< +# Inlining core_parse is slow and has no benefit. +$(EBIN)/core_parse.beam: $(EGEN)/core_parse.erl + $(V_ERLC) $(subst +inline,,$(ERL_COMPILE_FLAGS)) -o$(EBIN) $< + # ---------------------------------------------------- # Release Target # ---------------------------------------------------- diff --git a/lib/compiler/src/beam_block.erl b/lib/compiler/src/beam_block.erl index 92f09e400c..5216f39296 100644 --- a/lib/compiler/src/beam_block.erl +++ b/lib/compiler/src/beam_block.erl @@ -252,13 +252,6 @@ combine_alloc({_,Ns,Nh1,Init}, {_,nostack,Nh2,[]}) -> %% opt([Instruction]) -> [Instruction] %% Optimize the instruction stream inside a basic block. -opt([{set,[Dst],As,{bif,Bif,Fail}}=I1, - {set,[Dst],[Dst],{bif,'not',Fail}}=I2|Is]) -> - %% Get rid of the 'not' if the operation can be inverted. - case inverse_comp_op(Bif) of - none -> [I1,I2|opt(Is)]; - RevBif -> [{set,[Dst],As,{bif,RevBif,Fail}}|opt(Is)] - end; opt([{set,[X],[X],move}|Is]) -> opt(Is); opt([{set,_,_,{line,_}}=Line1, {set,[D1],[{integer,Idx1},Reg],{bif,element,{f,0}}}=I1, @@ -428,18 +421,6 @@ x_live([{x,N}|Rs], Regs) -> x_live(Rs, Regs bor (1 bsl N)); x_live([_|Rs], Regs) -> x_live(Rs, Regs); x_live([], Regs) -> Regs. -%% inverse_comp_op(Op) -> none|RevOp - -inverse_comp_op('=:=') -> '=/='; -inverse_comp_op('=/=') -> '=:='; -inverse_comp_op('==') -> '/='; -inverse_comp_op('/=') -> '=='; -inverse_comp_op('>') -> '=<'; -inverse_comp_op('<') -> '>='; -inverse_comp_op('>=') -> '<'; -inverse_comp_op('=<') -> '>'; -inverse_comp_op(_) -> none. - %%% %%% Evaluation of constant bit fields. %%% diff --git a/lib/compiler/src/beam_bool.erl b/lib/compiler/src/beam_bool.erl index a452d30b61..5ed9c16d61 100644 --- a/lib/compiler/src/beam_bool.erl +++ b/lib/compiler/src/beam_bool.erl @@ -787,6 +787,9 @@ is_not_used(R, Is, Label, #st{ll=Ll}) -> initialized_regs(Is) -> initialized_regs(Is, ordsets:new()). +initialized_regs([{set,Dst,_Src,{alloc,Live,_}}|_], Regs0) -> + Regs = add_init_regs(free_vars_regs(Live), Regs0), + add_init_regs(Dst, Regs); initialized_regs([{set,Dst,Src,_}|Is], Regs) -> initialized_regs(Is, add_init_regs(Dst, add_init_regs(Src, Regs))); initialized_regs([{test,_,_,Src}|Is], Regs) -> diff --git a/lib/compiler/src/beam_dead.erl b/lib/compiler/src/beam_dead.erl index 7cd07dc3be..f4515ba2a7 100644 --- a/lib/compiler/src/beam_dead.erl +++ b/lib/compiler/src/beam_dead.erl @@ -98,6 +98,12 @@ move_move_into_block([], Acc) -> reverse(Acc). forward(Is, Lc) -> forward(Is, gb_trees:empty(), Lc, []). +forward([{move,_,_}=Move|[{label,L}|_]=Is], D, Lc, Acc) -> + %% move/2 followed by jump/1 is optimized by backward/3. + forward([Move,{jump,{f,L}}|Is], D, Lc, Acc); +forward([{bif,_,_,_,_}=Bif|[{label,L}|_]=Is], D, Lc, Acc) -> + %% bif/4 followed by jump/1 is optimized by backward/3. + forward([Bif,{jump,{f,L}}|Is], D, Lc, Acc); forward([{block,[]}|Is], D, Lc, Acc) -> %% Empty blocks can prevent optimizations. forward(Is, D, Lc, Acc); @@ -124,6 +130,8 @@ forward([{label,Lbl}=LblI|[{move,Lit,Dst}|Is1]=Is0], D, Lc, Acc) -> _ -> Is0 %Keep move instruction. end, forward(Is, D, Lc, [LblI|Acc]); +forward([{test,is_eq_exact,_,[Same,Same]}|Is], D, Lc, Acc) -> + forward(Is, D, Lc, Acc); forward([{test,is_eq_exact,_,[Dst,Src]}=I, {block,[{set,[Dst],[Src],move}|Bl]}|Is], D, Lc, Acc) -> forward([I,{block,Bl}|Is], D, Lc, Acc); @@ -234,10 +242,8 @@ backward([{select,select_val,Reg,{f,Fail0},List0}|Is], D, Acc) -> Fail = shortcut_bs_test(Fail1, Is, D), Sel = {select,select_val,Reg,{f,Fail},List}, backward(Is, D, [Sel|Acc]); -backward([{jump,{f,To0}},{move,Src0,Reg}|Is], D, Acc) -> - To1 = shortcut_select_label(To0, Reg, Src0, D), - {To,Src} = shortcut_boolean_label(To1, Reg, Src0, D), - Move = {move,Src,Reg}, +backward([{jump,{f,To0}},{move,Src,Reg}=Move|Is], D, Acc) -> + To = shortcut_select_label(To0, Reg, Src, D), Jump = {jump,{f,To}}, case beam_utils:is_killed_at(Reg, To, D) of false -> backward([Move|Is], D, [Jump|Acc]); @@ -330,16 +336,6 @@ shortcut_label(To0, D) -> shortcut_select_label(To, Reg, Lit, D) -> shortcut_rel_op(To, is_ne_exact, [Reg,Lit], D). -shortcut_boolean_label(To0, Reg, {atom,Bool0}=Lit, D) when is_boolean(Bool0) -> - case beam_utils:code_at(To0, D) of - [{line,_},{bif,'not',_,[Reg],Reg},{jump,{f,To}}|_] -> - Bool = {atom,not Bool0}, - {shortcut_select_label(To, Reg, Bool, D),Bool}; - _ -> - {To0,Lit} - end; -shortcut_boolean_label(To, _, Bool, _) -> {To,Bool}. - %% Replace a comparison operator with a test instruction and a jump. %% For example, if we have this code: %% diff --git a/lib/compiler/src/beam_peep.erl b/lib/compiler/src/beam_peep.erl index 97a8c7ba70..5abacc8d5d 100644 --- a/lib/compiler/src/beam_peep.erl +++ b/lib/compiler/src/beam_peep.erl @@ -108,14 +108,14 @@ peep([{test,Op,_,Ops}=I|Is], SeenTests0, Acc) -> %% has succeeded. peep(Is, gb_sets:empty(), [I|Acc]); true -> - Test = {Op,Ops}, - case gb_sets:is_element(Test, SeenTests0) of + case is_test_redundant(Op, Ops, SeenTests0) of true -> - %% This test has already succeeded and + %% This test or a similar test has already succeeded and %% is therefore redundant. peep(Is, SeenTests0, Acc); false -> %% Remember that we have seen this test. + Test = {Op,Ops}, SeenTests = gb_sets:insert(Test, SeenTests0), peep(Is, SeenTests, [I|Acc]) end @@ -136,6 +136,15 @@ peep([I|Is], _, Acc) -> peep(Is, gb_sets:empty(), [I|Acc]); peep([], _, Acc) -> reverse(Acc). +is_test_redundant(Op, Ops, Seen) -> + gb_sets:is_element({Op,Ops}, Seen) orelse + is_test_redundant_1(Op, Ops, Seen). + +is_test_redundant_1(is_boolean, [R], Seen) -> + gb_sets:is_element({is_eq_exact,[R,{atom,false}]}, Seen) orelse + gb_sets:is_element({is_eq_exact,[R,{atom,true}]}, Seen); +is_test_redundant_1(_, _, _) -> false. + kill_seen(Dst, Seen0) -> gb_sets:from_ordset(kill_seen_1(gb_sets:to_list(Seen0), Dst)). diff --git a/lib/compiler/src/cerl.erl b/lib/compiler/src/cerl.erl index 3d4b9ee0c6..ea960abc1a 100644 --- a/lib/compiler/src/cerl.erl +++ b/lib/compiler/src/cerl.erl @@ -138,7 +138,8 @@ ]). -export_type([c_binary/0, c_bitstr/0, c_call/0, c_clause/0, c_cons/0, c_fun/0, - c_literal/0, c_map/0, c_map_pair/0, c_module/0, c_tuple/0, + c_let/0, c_literal/0, c_map/0, c_map_pair/0, + c_module/0, c_tuple/0, c_values/0, c_var/0, cerl/0, var_name/0]). -include("core_parse.hrl"). @@ -255,7 +256,7 @@ %% @see c_primop/2 %% @see c_receive/1 %% @see c_seq/2 -%% @see c_try/3 +%% @see c_try/5 %% @see c_tuple/1 %% @see c_values/1 %% @see c_var/1 @@ -1455,7 +1456,7 @@ is_proper_list(_) -> %% X4]</code>. %% %% @see c_cons/2 -%% @see c_nil/1 +%% @see c_nil/0 %% @see is_c_list/1 %% @see list_length/1 %% @see make_list/2 @@ -1486,7 +1487,7 @@ abstract_list([]) -> %% efficient.</p> %% %% @see c_cons/2 -%% @see c_nil/1 +%% @see c_nil/0 %% @see is_c_list/1 %% @see list_elements/1 @@ -1991,7 +1992,7 @@ update_c_fname(Node, Atom, Arity) -> %% %% @see c_fname/2 %% @see c_var/1 -%% @see c_var_name/1 +%% @see var_name/1 -spec is_c_fname(cerl()) -> boolean(). @@ -3669,7 +3670,7 @@ c_try(Expr, Vs, Body, Evs, Handler) -> %% @spec ann_c_try(As::[term()], Expression::cerl(), %% Variables::[cerl()], Body::cerl(), %% EVars::[cerl()], Handler::cerl()) -> cerl() -%% @see c_try/3 +%% @see c_try/5 -spec ann_c_try([term()], cerl(), [cerl()], cerl(), [cerl()], cerl()) -> c_try(). @@ -3682,7 +3683,7 @@ ann_c_try(As, Expr, Vs, Body, Evs, Handler) -> %% @spec update_c_try(Old::cerl(), Expression::cerl(), %% Variables::[cerl()], Body::cerl(), %% EVars::[cerl()], Handler::cerl()) -> cerl() -%% @see c_try/3 +%% @see c_try/5 -spec update_c_try(c_try(), cerl(), [cerl()], cerl(), [cerl()], cerl()) -> c_try(). @@ -3697,7 +3698,7 @@ update_c_try(Node, Expr, Vs, Body, Evs, Handler) -> %% @doc Returns <code>true</code> if <code>Node</code> is an abstract %% try-expression, otherwise <code>false</code>. %% -%% @see c_try/3 +%% @see c_try/5 -spec is_c_try(cerl()) -> boolean(). @@ -3711,7 +3712,7 @@ is_c_try(_) -> %% %% @doc Returns the expression subtree of an abstract try-expression. %% -%% @see c_try/3 +%% @see c_try/5 -spec try_arg(c_try()) -> cerl(). @@ -3724,7 +3725,7 @@ try_arg(Node) -> %% @doc Returns the list of success variable subtrees of an abstract %% try-expression. %% -%% @see c_try/3 +%% @see c_try/5 -spec try_vars(c_try()) -> [cerl()]. @@ -3736,7 +3737,7 @@ try_vars(Node) -> %% %% @doc Returns the success body subtree of an abstract try-expression. %% -%% @see c_try/3 +%% @see c_try/5 -spec try_body(c_try()) -> cerl(). @@ -3749,7 +3750,7 @@ try_body(Node) -> %% @doc Returns the list of exception variable subtrees of an abstract %% try-expression. %% -%% @see c_try/3 +%% @see c_try/5 -spec try_evars(c_try()) -> [cerl()]. @@ -3762,7 +3763,7 @@ try_evars(Node) -> %% @doc Returns the exception body subtree of an abstract %% try-expression. %% -%% @see c_try/3 +%% @see c_try/5 -spec try_handler(c_try()) -> cerl(). @@ -3784,7 +3785,7 @@ try_handler(Node) -> %% @see update_c_catch/2 %% @see is_c_catch/1 %% @see catch_body/1 -%% @see c_try/3 +%% @see c_try/5 -spec c_catch(cerl()) -> c_catch(). diff --git a/lib/compiler/src/cerl_trees.erl b/lib/compiler/src/cerl_trees.erl index b93da8e97f..f1bf0e02e7 100644 --- a/lib/compiler/src/cerl_trees.erl +++ b/lib/compiler/src/cerl_trees.erl @@ -19,7 +19,7 @@ %% @doc Basic functions on Core Erlang abstract syntax trees. %% %% <p>Syntax trees are defined in the module <a -%% href=""><code>cerl</code></a>.</p> +%% href="cerl"><code>cerl</code></a>.</p> %% %% @type cerl() = cerl:cerl() diff --git a/lib/compiler/src/compile.erl b/lib/compiler/src/compile.erl index f347438509..c45c9a1a29 100644 --- a/lib/compiler/src/compile.erl +++ b/lib/compiler/src/compile.erl @@ -1295,8 +1295,9 @@ encrypt({des3_cbc=Type,Key,IVec,BlockSize}, Bin0) -> list_to_binary([0,length(TypeString),TypeString,Bin]). random_bytes(N) -> - {A,B,C} = now(), - _ = random:seed(A, B, C), + _ = random:seed(erlang:time_offset(), + erlang:monotonic_time(), + erlang:unique_integer()), random_bytes_1(N, []). random_bytes_1(0, Acc) -> Acc; diff --git a/lib/compiler/src/compiler.app.src b/lib/compiler/src/compiler.app.src index fbaa7a96fe..2a40c1c379 100644 --- a/lib/compiler/src/compiler.app.src +++ b/lib/compiler/src/compiler.app.src @@ -69,5 +69,5 @@ {registered, []}, {applications, [kernel, stdlib]}, {env, []}, - {runtime_dependencies, ["stdlib-2.0","kernel-3.0","hipe-3.10.3","erts-6.0", + {runtime_dependencies, ["stdlib-2.0","kernel-3.0","hipe-3.10.3","erts-7.0", "crypto-3.3"]}]}. diff --git a/lib/compiler/src/sys_core_fold.erl b/lib/compiler/src/sys_core_fold.erl index ea1959d0f8..0d020578f5 100644 --- a/lib/compiler/src/sys_core_fold.erl +++ b/lib/compiler/src/sys_core_fold.erl @@ -96,7 +96,7 @@ t=[], %Types in_guard=false}). %In guard or not. --type type_info() :: cerl:cerl() | 'bool'. +-type type_info() :: cerl:cerl() | 'bool' | 'integer'. -type yes_no_maybe() :: 'yes' | 'no' | 'maybe'. -type sub() :: #sub{}. @@ -297,7 +297,8 @@ expr(#c_seq{arg=Arg0,body=B0}=Seq0, Ctxt, Sub) -> false -> Seq0#c_seq{arg=Arg,body=B1} end end; -expr(#c_let{}=Let, Ctxt, Sub) -> +expr(#c_let{}=Let0, Ctxt, Sub) -> + Let = opt_case_in_let(Let0), case simplify_let(Let, Sub) of impossible -> %% The argument for the let is "simple", i.e. has no @@ -829,16 +830,16 @@ eval_rel_op(Call, '=:=', [Term,#c_literal{val=true}], Sub) -> maybe -> Call; no -> #c_literal{val=false} end; -eval_rel_op(Call, '==', Ops, _Sub) -> - case is_exact_eq_ok(Ops) of +eval_rel_op(Call, '==', Ops, Sub) -> + case is_exact_eq_ok(Ops, Sub) of true -> Name = #c_literal{anno=cerl:get_ann(Call),val='=:='}, Call#c_call{name=Name}; false -> Call end; -eval_rel_op(Call, '/=', Ops, _Sub) -> - case is_exact_eq_ok(Ops) of +eval_rel_op(Call, '/=', Ops, Sub) -> + case is_exact_eq_ok(Ops, Sub) of true -> Name = #c_literal{anno=cerl:get_ann(Call),val='=/='}, Call#c_call{name=Name}; @@ -847,11 +848,17 @@ eval_rel_op(Call, '/=', Ops, _Sub) -> end; eval_rel_op(Call, _, _, _) -> Call. -is_exact_eq_ok([#c_literal{val=Lit}|_]) -> +is_exact_eq_ok([A,B]=L, Sub) -> + case is_int_type(A, Sub) =:= yes andalso is_int_type(B, Sub) =:= yes of + true -> true; + false -> is_exact_eq_ok_1(L) + end. + +is_exact_eq_ok_1([#c_literal{val=Lit}|_]) -> is_non_numeric(Lit); -is_exact_eq_ok([_|T]) -> - is_exact_eq_ok(T); -is_exact_eq_ok([]) -> false. +is_exact_eq_ok_1([_|T]) -> + is_exact_eq_ok_1(T); +is_exact_eq_ok_1([]) -> false. is_non_numeric([H|T]) -> is_non_numeric(H) andalso is_non_numeric(T); @@ -963,7 +970,7 @@ eval_element(Call, #c_literal{val=Pos}, Tuple, Types) 1 =< Pos, Pos =< length(Es) -> El = lists:nth(Pos, Es), try - pat_to_expr(El) + cerl:set_ann(pat_to_expr(El), [compiler_generated]) catch throw:impossible -> Call @@ -1008,28 +1015,32 @@ eval_is_record(Call, _, _, _, _) -> Call. %% eval_setelement(Call, Pos, Tuple, NewVal) -> Core. %% Evaluates setelement/3 if position Pos is an integer -%% the shape of the tuple Tuple is known. +%% and the shape of the tuple Tuple is known. %% -eval_setelement(Call, Pos, Tuple, NewVal) -> - try - eval_setelement_1(Pos, Tuple, NewVal) - catch - error:_ -> - Call - end. - -eval_setelement_1(#c_literal{val=Pos}, #c_tuple{anno=A,es=Es}, NewVal) - when is_integer(Pos) -> - ann_c_tuple(A, eval_setelement_2(Pos, Es, NewVal)); -eval_setelement_1(#c_literal{val=Pos}, #c_literal{anno=A,val=Es0}, NewVal) +eval_setelement(Call, #c_literal{val=Pos}, Tuple, NewVal) when is_integer(Pos) -> - Es = [#c_literal{anno=A,val=E} || E <- tuple_to_list(Es0)], - ann_c_tuple(A, eval_setelement_2(Pos, Es, NewVal)). + case cerl:is_data(Tuple) of + false -> + Call; + true -> + Es0 = case cerl:is_c_tuple(Tuple) of + false -> []; + true -> cerl:tuple_es(Tuple) + end, + if + 1 =< Pos, Pos =< length(Es0) -> + Es = eval_setelement_1(Pos, Es0, NewVal), + cerl:update_c_tuple(Tuple, Es); + true -> + eval_failure(Call, badarg) + end + end; +eval_setelement(Call, _, _, _) -> Call. -eval_setelement_2(1, [_|T], NewVal) -> +eval_setelement_1(1, [_|T], NewVal) -> [NewVal|T]; -eval_setelement_2(Pos, [H|T], NewVal) when Pos > 1 -> - [H|eval_setelement_2(Pos-1, T, NewVal)]. +eval_setelement_1(Pos, [H|T], NewVal) when Pos > 1 -> + [H|eval_setelement_1(Pos-1, T, NewVal)]. %% eval_failure(Call, Reason) -> Core. %% Warn for a call that will fail and replace the call with @@ -1955,46 +1966,125 @@ letify(Bs, Body) -> cerl:ann_c_let(Ann, [V], Val, B) end, Body, Bs). -%% opt_case_in_let(LetExpr) -> LetExpr' +%% opt_not_in_let(Let) -> Cerl +%% Try to optimize away a 'not' operator in a 'let'. -opt_case_in_let(#c_let{vars=Vs,arg=Arg,body=B}=Let, Sub) -> - opt_case_in_let_0(Vs, Arg, B, Let, Sub). +-spec opt_not_in_let(cerl:c_let()) -> cerl:cerl(). -opt_case_in_let_0([#c_var{name=V}], Arg, - #c_case{arg=#c_var{name=V},clauses=Cs}=Case, Let, Sub) -> - case opt_case_in_let_1(V, Arg, Cs) of - impossible -> - case is_simple_case_arg(Arg) andalso - not core_lib:is_var_used(V, Case#c_case{arg=#c_literal{val=nil}}) of - true -> - expr(opt_bool_case(Case#c_case{arg=Arg,clauses=Cs}), sub_new(Sub)); - false -> - Let +opt_not_in_let(#c_let{vars=[_]=Vs0,arg=Arg0,body=Body0}=Let) -> + case opt_not_in_let(Vs0, Arg0, Body0) of + {[],#c_values{es=[]},Body} -> + Body; + {Vs,Arg,Body} -> + Let#c_let{vars=Vs,arg=Arg,body=Body} + end; +opt_not_in_let(Let) -> Let. + +%% opt_not_in_let(Vs, Arg, Body) -> {Vs',Arg',Body'} +%% Try to optimize away a 'not' operator in a 'let'. + +-spec opt_not_in_let([cerl:c_var()], cerl:cerl(), cerl:cerl()) -> + {[cerl:c_var()],cerl:cerl(),cerl:cerl()}. + +opt_not_in_let([#c_var{name=V}]=Vs0, Arg0, Body0) -> + case cerl:type(Body0) of + call -> + %% let <V> = Expr in not V ==> + %% let <> = <> in notExpr + case opt_not_in_let_1(V, Body0, Arg0) of + no -> + {Vs0,Arg0,Body0}; + {yes,Body} -> + {[],#c_values{es=[]},Body} end; - Expr -> Expr + 'let' -> + %% let <V> = Expr in let <Var> = not V in Body ==> + %% let <Var> = notExpr in Body + %% V must not be used in Body. + LetArg = cerl:let_arg(Body0), + case opt_not_in_let_1(V, LetArg, Arg0) of + no -> + {Vs0,Arg0,Body0}; + {yes,Arg} -> + LetBody = cerl:let_body(Body0), + case core_lib:is_var_used(V, LetBody) of + true -> + {Vs0,Arg0,Body0}; + false -> + LetVars = cerl:let_vars(Body0), + {LetVars,Arg,LetBody} + end + end; + _ -> + {Vs0,Arg0,Body0} end; -opt_case_in_let_0(_, _, _, Let, _) -> Let. - -opt_case_in_let_1(V, Arg, Cs) -> - try - opt_case_in_let_2(V, Arg, Cs) - catch - _:_ -> impossible +opt_not_in_let(Vs, Arg, Body) -> + {Vs,Arg,Body}. + +opt_not_in_let_1(V, Call, Body) -> + case Call of + #c_call{module=#c_literal{val=erlang}, + name=#c_literal{val='not'}, + args=[#c_var{name=V}]} -> + opt_not_in_let_2(Body); + _ -> + no end. -opt_case_in_let_2(V, Arg0, - [#c_clause{pats=[#c_tuple{es=Es}], - guard=#c_literal{val=true},body=B}|_]) -> +opt_not_in_let_2(#c_case{clauses=Cs0}=Case) -> + Vars = make_vars([], 1), + Body = #c_call{module=#c_literal{val=erlang}, + name=#c_literal{val='not'}, + args=Vars}, + Cs = [begin + Let = #c_let{vars=Vars,arg=B,body=Body}, + C#c_clause{body=opt_not_in_let(Let)} + end || #c_clause{body=B}=C <- Cs0], + {yes,Case#c_case{clauses=Cs}}; +opt_not_in_let_2(#c_call{}=Call0) -> + invert_call(Call0); +opt_not_in_let_2(_) -> no. + +invert_call(#c_call{module=#c_literal{val=erlang}, + name=#c_literal{val=Name0}, + args=[_,_]}=Call) -> + case inverse_rel_op(Name0) of + no -> no; + Name -> {yes,Call#c_call{name=#c_literal{val=Name}}} + end; +invert_call(#c_call{}) -> no. + +%% inverse_rel_op(Op) -> no | RevOp + +inverse_rel_op('=:=') -> '=/='; +inverse_rel_op('=/=') -> '=:='; +inverse_rel_op('==') -> '/='; +inverse_rel_op('/=') -> '=='; +inverse_rel_op('>') -> '=<'; +inverse_rel_op('<') -> '>='; +inverse_rel_op('>=') -> '<'; +inverse_rel_op('=<') -> '>'; +inverse_rel_op(_) -> no. - %% In {V1,V2,...} = case E of P -> ... {Val1,Val2,...}; ... end. - %% avoid building tuples, by converting tuples to multiple values. - %% (The optimisation is not done if the built tuple is used or returned.) - true = all(fun (#c_var{}) -> true; - (_) -> false end, Es), %Only variables in tuple - false = core_lib:is_var_used(V, B), %Built tuple must not be used. - Arg1 = tuple_to_values(Arg0, length(Es)), %Might fail. - #c_let{vars=Es,arg=Arg1,body=B}. +%% opt_bool_case_in_let(LetExpr, Sub) -> Core + +opt_bool_case_in_let(#c_let{vars=Vs,arg=Arg,body=B}=Let, Sub) -> + opt_case_in_let_1(Vs, Arg, B, Let, Sub). + +opt_case_in_let_1([#c_var{name=V}], Arg, + #c_case{arg=#c_var{name=V}}=Case0, Let, Sub) -> + case is_simple_case_arg(Arg) of + true -> + Case = opt_bool_case(Case0#c_case{arg=Arg}), + case core_lib:is_var_used(V, Case) of + false -> expr(Case, sub_new(Sub)); + true -> Let + end; + false -> + Let + end; +opt_case_in_let_1(_, _, _, Let, _) -> Let. %% is_simple_case_arg(Expr) -> true|false %% Determine whether the Expr is simple enough to be worth @@ -2036,7 +2126,7 @@ is_bool_expr(#c_clause{body=B}, Sub) -> is_bool_expr(B, Sub); is_bool_expr(#c_let{vars=[V],arg=Arg,body=B}, Sub0) -> Sub = case is_bool_expr(Arg, Sub0) of - true -> update_types(V, [#c_literal{val=true}], Sub0); + true -> update_types(V, [bool], Sub0); false -> Sub0 end, is_bool_expr(B, Sub); @@ -2122,38 +2212,6 @@ is_safe_bool_expr_list([C|Cs], Sub, BoolVars) -> end; is_safe_bool_expr_list([], _, _) -> true. -%% tuple_to_values(Expr, TupleArity) -> Expr' -%% Convert tuples in return position of arity TupleArity to values. -%% Throws an exception for constructs that are not handled. - -tuple_to_values(#c_tuple{es=Es}, Arity) when length(Es) =:= Arity -> - core_lib:make_values(Es); -tuple_to_values(#c_literal{val=Tuple}=Lit, Arity) when tuple_size(Tuple) =:= Arity -> - Es = [Lit#c_literal{val=E} || E <- tuple_to_list(Tuple)], - core_lib:make_values(Es); -tuple_to_values(#c_case{clauses=Cs0}=Case, Arity) -> - Cs1 = [tuple_to_values(E, Arity) || E <- Cs0], - Case#c_case{clauses=Cs1}; -tuple_to_values(#c_seq{body=B0}=Seq, Arity) -> - Seq#c_seq{body=tuple_to_values(B0, Arity)}; -tuple_to_values(#c_let{body=B0}=Let, Arity) -> - Let#c_let{body=tuple_to_values(B0, Arity)}; -tuple_to_values(#c_receive{clauses=Cs0,timeout=Timeout,action=A0}=Rec, Arity) -> - Cs = [tuple_to_values(E, Arity) || E <- Cs0], - A = case Timeout of - #c_literal{val=infinity} -> A0; - _ -> tuple_to_values(A0, Arity) - end, - Rec#c_receive{clauses=Cs,action=A}; -tuple_to_values(#c_clause{body=B0}=Clause, Arity) -> - B = tuple_to_values(B0, Arity), - Clause#c_clause{body=B}; -tuple_to_values(Expr, _) -> - case will_fail(Expr) of - true -> Expr; - false -> erlang:error({not_handled,Expr}) - end. - %% simplify_let(Let, Sub) -> Expr | impossible %% If the argument part of an let contains a complex expression, such %% as a let or a sequence, move the original let body into the complex @@ -2180,7 +2238,7 @@ move_let_into_expr(#c_let{vars=InnerVs0,body=InnerBody0}=Inner, Arg = body(Arg0, Sub0), ScopeSub0 = sub_subst_scope(Sub0#sub{t=[]}), {OuterVs,ScopeSub} = pattern_list(OuterVs0, ScopeSub0), - + OuterBody = body(OuterBody0, ScopeSub), {InnerVs,Sub} = pattern_list(InnerVs0, Sub0), @@ -2258,50 +2316,179 @@ move_let_into_expr(_Let, _Expr, _Sub) -> impossible. is_failing_clause(#c_clause{body=B}) -> will_fail(B). +%% opt_case_in_let(Let) -> Let' +%% Try to avoid building tuples that are immediately matched. +%% A common pattern is: +%% +%% {V1,V2,...} = case E of P -> ... {Val1,Val2,...}; ... end +%% +%% In Core Erlang the pattern would look like this: +%% +%% let <V> = case E of +%% ... -> ... {Val1,Val2} +%% ... +%% end, +%% in case V of +%% {A,B} -> ... <use A and B> ... +%% end +%% +%% Rewrite this to: +%% +%% let <V1,V2> = case E of +%% ... -> ... <Val1,Val2> +%% ... +%% end, +%% in +%% let <V> = {V1,V2} +%% in case V of +%% {A,B} -> ... <use A and B> ... +%% end +%% +%% Note that the second 'case' is unchanged. The other optimizations +%% in this module will eliminate the building of the tuple and +%% rewrite the second case to: +%% +%% case <V1,V2> of +%% <A,B> -> ... <use A and B> ... +%% end +%% + +opt_case_in_let(#c_let{vars=Vs,arg=Arg0,body=B}=Let0) -> + case matches_data(Vs, B) of + {yes,TypeSig} -> + case delay_build(Arg0, TypeSig) of + no -> + Let0; + {yes,Vars,Arg,Data} -> + InnerLet = Let0#c_let{arg=Data}, + Let0#c_let{vars=Vars,arg=Arg,body=InnerLet} + end; + no -> + Let0 + end. + +matches_data([#c_var{name=V}], #c_case{arg=#c_var{name=V}, + clauses=[#c_clause{pats=[P]}|_]}) -> + case cerl:is_data(P) of + false -> + no; + true -> + case cerl:data_type(P) of + {atomic,_} -> + no; + Type -> + {yes,{Type,cerl:data_arity(P)}} + end + end; +matches_data(_, _) -> no. + +delay_build(Core, TypeSig) -> + case cerl:is_data(Core) of + true -> no; + false -> delay_build_1(Core, TypeSig) + end. + +delay_build_1(Core0, TypeSig) -> + try delay_build_expr(Core0, TypeSig) of + Core -> + {Type,Arity} = TypeSig, + Vars = make_vars([], Arity), + Data = cerl:ann_make_data([compiler_generated], Type, Vars), + {yes,Vars,Core,Data} + catch + throw:impossible -> + no + end. + +delay_build_cs([#c_clause{body=B0}=C0|Cs], TypeSig) -> + B = delay_build_expr(B0, TypeSig), + C = C0#c_clause{body=B}, + [C|delay_build_cs(Cs, TypeSig)]; +delay_build_cs([], _) -> []. + +delay_build_expr(Core, {Type,Arity}=TypeSig) -> + case cerl:is_data(Core) of + false -> + delay_build_expr_1(Core, TypeSig); + true -> + case {cerl:data_type(Core),cerl:data_arity(Core)} of + {Type,Arity} -> + core_lib:make_values(cerl:data_es(Core)); + {_,_} -> + throw(impossible) + end + end. + +delay_build_expr_1(#c_case{clauses=Cs0}=Case, TypeSig) -> + Cs = delay_build_cs(Cs0, TypeSig), + Case#c_case{clauses=Cs}; +delay_build_expr_1(#c_let{body=B0}=Let, TypeSig) -> + B = delay_build_expr(B0, TypeSig), + Let#c_let{body=B}; +delay_build_expr_1(#c_receive{clauses=Cs0, + timeout=Timeout, + action=A0}=Rec, TypeSig) -> + Cs = delay_build_cs(Cs0, TypeSig), + A = case Timeout of + #c_literal{val=infinity} -> A0; + _ -> delay_build_expr(A0, TypeSig) + end, + Rec#c_receive{clauses=Cs,action=A}; +delay_build_expr_1(#c_seq{body=B0}=Seq, TypeSig) -> + B = delay_build_expr(B0, TypeSig), + Seq#c_seq{body=B}; +delay_build_expr_1(Core, _TypeSig) -> + case will_fail(Core) of + true -> Core; + false -> throw(impossible) + end. + %% opt_simple_let(#c_let{}, Context, Sub) -> CoreTerm %% Optimize a let construct that does not contain any lets in %% in its argument. -opt_simple_let(#c_let{arg=Arg0}=Let, Ctxt, Sub0) -> - Arg = body(Arg0, value, Sub0), %This is a body +opt_simple_let(Let0, Ctxt, Sub) -> + case opt_not_in_let(Let0) of + #c_let{}=Let -> + opt_simple_let_0(Let, Ctxt, Sub); + Expr -> + expr(Expr, Ctxt, Sub) + end. + +opt_simple_let_0(#c_let{arg=Arg0}=Let, Ctxt, Sub) -> + Arg = body(Arg0, value, Sub), %This is a body case will_fail(Arg) of true -> Arg; - false -> opt_simple_let_1(Let, Arg, Ctxt, Sub0) + false -> opt_simple_let_1(Let, Arg, Ctxt, Sub) end. opt_simple_let_1(#c_let{vars=Vs0,body=B0}=Let, Arg0, Ctxt, Sub0) -> %% Optimise let and add new substitutions. - {Vs,Args,Sub1} = let_substs(Vs0, Arg0, Sub0), - BodySub = case {Vs,Args} of - {[V],[A]} -> - case is_bool_expr(A, Sub0) of - true -> - update_types(V, [#c_literal{val=true}], Sub1); - false -> - Sub1 - end; - {_,_} -> Sub1 - end, - B = body(B0, Ctxt, BodySub), - Arg = core_lib:make_values(Args), - opt_simple_let_2(Let, Vs, Arg, B, Ctxt, Sub1). - -opt_simple_let_2(Let0, Vs0, Arg0, Body, Ctxt, Sub) -> + {Vs1,Args,Sub1} = let_substs(Vs0, Arg0, Sub0), + BodySub = update_let_types(Vs1, Args, Sub1), + B1 = body(B0, Ctxt, BodySub), + Arg1 = core_lib:make_values(Args), + {Vs,Arg,B} = opt_not_in_let(Vs1, Arg1, B1), + opt_simple_let_2(Let, Vs, Arg, B, B0, Ctxt, Sub1). + +opt_simple_let_2(Let0, Vs0, Arg0, Body, PrevBody, Ctxt, Sub) -> case {Vs0,Arg0,Body} of - {[#c_var{name=N1}],Arg,#c_var{name=N2}} -> + {[#c_var{name=N1}],Arg1,#c_var{name=N2}} -> case N1 =:= N2 of true -> %% let <Var> = Arg in <Var> ==> Arg - Arg; + Arg1; false -> %% let <Var> = Arg in <OtherVar> ==> seq Arg OtherVar + Arg = maybe_suppress_warnings(Arg1, Vs0, PrevBody, Ctxt), expr(#c_seq{arg=Arg,body=Body}, Ctxt, sub_new_preserve_types(Sub)) end; {[],#c_values{es=[]},_} -> %% No variables left. Body; - {_,Arg,#c_literal{}} -> + {Vs,Arg1,#c_literal{}} -> + Arg = maybe_suppress_warnings(Arg1, Vs, PrevBody, Ctxt), E = case Ctxt of effect -> %% Throw away the literal body. @@ -2313,22 +2500,50 @@ opt_simple_let_2(Let0, Vs0, Arg0, Body, Ctxt, Sub) -> #c_seq{arg=Arg,body=Body} end, expr(E, Ctxt, sub_new_preserve_types(Sub)); - {Vs,Arg,Body} -> + {Vs,Arg1,Body} -> %% If none of the variables are used in the body, we can %% rewrite the let to a sequence: %% let <Var> = Arg in BodyWithoutVar ==> %% seq Arg BodyWithoutVar case is_any_var_used(Vs, Body) of false -> + Arg = maybe_suppress_warnings(Arg1, Vs, PrevBody, Ctxt), expr(#c_seq{arg=Arg,body=Body}, Ctxt, sub_new_preserve_types(Sub)); true -> - Let1 = Let0#c_let{vars=Vs,arg=Arg,body=Body}, - Let2 = opt_case_in_let(Let1, Sub), + Let1 = Let0#c_let{vars=Vs,arg=Arg1,body=Body}, + Let2 = opt_bool_case_in_let(Let1, Sub), opt_case_in_let_arg(Let2, Ctxt, Sub) end end. +%% maybe_suppress_warnings(Arg, [#c_var{}], PreviousBody, Context) -> Arg' +%% Try to suppress false warnings when a variable is not used. +%% For instance, we don't expect a warning for useless building in: +%% +%% R = #r{}, %No warning expected. +%% R#r.f %Optimization would remove the reference to R. +%% +%% To avoid false warnings, we will check whether the variables were +%% referenced in the original unoptimized code. If they were, we will +%% consider the warning false and suppress it. + +maybe_suppress_warnings(Arg, _, _, effect) -> + %% Don't suppress any warnings in effect context. + Arg; +maybe_suppress_warnings(Arg, Vs, PrevBody, value) -> + case suppress_warning(Arg) of + true -> + Arg; %Already suppressed. + false -> + case is_any_var_used(Vs, PrevBody) of + true -> + cerl:set_ann(Arg, [compiler_generated]); + false -> + Arg + end + end. + move_case_into_arg(#c_case{arg=#c_let{vars=OuterVars0,arg=OuterArg, body=InnerArg0}=Outer, clauses=InnerClauses}=Inner, Sub) -> @@ -2416,7 +2631,7 @@ move_case_into_arg(_, _) -> %% <> when 'true' -> %% let <Var> = Literal2 in LetBody %% end -%% +%% %% In the worst case, the size of the code could increase. %% In practice, though, substituting the literals into %% LetBody and doing constant folding will decrease the code @@ -2490,6 +2705,7 @@ is_boolean_type(Var, Sub) -> is_int_type(Var, Sub) -> case get_type(Var, Sub) of none -> maybe; + integer -> yes; C -> yes_no(cerl:is_c_int(C)) end. @@ -2504,8 +2720,58 @@ is_tuple_type(Var, Sub) -> yes_no(true) -> yes; yes_no(false) -> no. +%%% +%%% Update type information. +%%% + +update_let_types(Vs, Args, Sub) when is_list(Args) -> + update_let_types_1(Vs, Args, Sub); +update_let_types(_Vs, _Arg, Sub) -> + %% The argument is a complex expression (such as a 'case') + %% that returns multiple values. + Sub. + +update_let_types_1([#c_var{}=V|Vs], [A|As], Sub0) -> + Sub = update_types_from_expr(V, A, Sub0), + update_let_types_1(Vs, As, Sub); +update_let_types_1([], [], Sub) -> Sub. + +update_types_from_expr(V, Expr, Sub) -> + Type = extract_type(Expr, Sub), + update_types(V, [Type], Sub). + +extract_type(#c_call{module=#c_literal{val=erlang}, + name=#c_literal{val=Name}, + args=Args}=Call, Sub) -> + case returns_integer(Name, Args) of + true -> integer; + false -> extract_type_1(Call, Sub) + end; +extract_type(Expr, Sub) -> + extract_type_1(Expr, Sub). + +extract_type_1(Expr, Sub) -> + case is_bool_expr(Expr, Sub) of + false -> Expr; + true -> bool + end. + +returns_integer(bit_size, [_]) -> true; +returns_integer('bsl', [_,_]) -> true; +returns_integer('bsr', [_,_]) -> true; +returns_integer(byte_size, [_]) -> true; +returns_integer(length, [_]) -> true; +returns_integer('rem', [_,_]) -> true; +returns_integer(size, [_]) -> true; +returns_integer(tuple_size, [_]) -> true; +returns_integer(trunc, [_]) -> true; +returns_integer(_, _) -> false. + %% update_types(Expr, Pattern, Sub) -> Sub' %% Update the type database. + +-spec update_types(cerl:cerl(), [type_info()], sub()) -> sub(). + update_types(Expr, Pat, #sub{t=Tdb0}=Sub) -> Tdb = update_types_1(Expr, Pat, Tdb0), Sub#sub{t=Tdb}. @@ -2525,6 +2791,8 @@ update_types_2(V, [#c_tuple{}=P], Types) -> orddict:store(V, P, Types); update_types_2(V, [#c_literal{val=Bool}], Types) when is_boolean(Bool) -> orddict:store(V, bool, Types); +update_types_2(V, [Type], Types) when is_atom(Type) -> + orddict:store(V, Type, Types); update_types_2(_, _, Types) -> Types. %% kill_types(V, Tdb) -> Tdb' @@ -2791,7 +3059,7 @@ bsm_ensure_no_partition_after([#c_clause{pats=Ps}|Cs], Pos) -> bsm_problem(P, bin_partition) end; bsm_ensure_no_partition_after([], _) -> ok. - + bsm_could_match_binary(#c_alias{pat=P}) -> bsm_could_match_binary(P); bsm_could_match_binary(#c_cons{}) -> false; bsm_could_match_binary(#c_tuple{}) -> false; diff --git a/lib/compiler/src/v3_codegen.erl b/lib/compiler/src/v3_codegen.erl index cbe50b93b0..7eec9dd62b 100644 --- a/lib/compiler/src/v3_codegen.erl +++ b/lib/compiler/src/v3_codegen.erl @@ -69,10 +69,8 @@ stk=[], %Stack table res=[]}). %Reserved regs: [{reserved,I,V}] -module({Mod,Exp,Attr,Forms}, Options) -> - put(?MODULE, Options), +module({Mod,Exp,Attr,Forms}, _Options) -> {Fs,St} = functions(Forms, {atom,Mod}), - erase(?MODULE), {ok,{Mod,Exp,Attr,Fs,St#cg.lcount}}. functions(Forms, AtomMod) -> @@ -924,7 +922,7 @@ select_extract_tuple(Src, Vs, I, Vdb, Bef, St) -> select_map(Scs, V, Tf, Vf, Bef, St0) -> Reg = fetch_var(V, Bef), {Is,Aft,St1} = - match_fmf(fun(#l{ke={val_clause,{map,_,Es},B},i=I,vdb=Vdb}, Fail, St1) -> + match_fmf(fun(#l{ke={val_clause,{map,exact,_,Es},B},i=I,vdb=Vdb}, Fail, St1) -> select_map_val(V, Es, B, Fail, I, Vdb, Bef, St1) end, Vf, St0, Scs), {[{test,is_map,{f,Tf},[Reg]}|Is],Aft,St1}. diff --git a/lib/compiler/src/v3_core.erl b/lib/compiler/src/v3_core.erl index 3c19a209c0..c954d21e59 100644 --- a/lib/compiler/src/v3_core.erl +++ b/lib/compiler/src/v3_core.erl @@ -78,7 +78,7 @@ splitwith/2,keyfind/3,sort/1,foreach/2,droplast/1,last/1]). -import(ordsets, [add_element/2,del_element/2,is_element/2, union/1,union/2,intersection/2,subtract/2]). --import(cerl, [ann_c_cons/3,ann_c_cons_skel/3,ann_c_tuple/2,c_tuple/1, +-import(cerl, [ann_c_cons/3,ann_c_tuple/2,c_tuple/1, ann_c_map/3]). -include("core_parse.hrl"). @@ -1660,48 +1660,55 @@ pat_segment({bin_element,_,Val,Size,[Type,{unit,Unit}|Flags]}, St) -> %% pat_alias(CorePat, CorePat) -> AliasPat. %% Normalise aliases. Trap bad aliases by throwing 'nomatch'. -pat_alias(#c_var{name=V1}, P2) -> #c_alias{var=#c_var{name=V1},pat=P2}; -pat_alias(P1, #c_var{name=V2}) -> #c_alias{var=#c_var{name=V2},pat=P1}; - -%% alias cons -pat_alias(#c_cons{}=Cons, #c_literal{anno=A,val=[H|T]}=S) -> - pat_alias(Cons, ann_c_cons_skel(A, #c_literal{anno=A,val=H}, - S#c_literal{val=T})); -pat_alias(#c_literal{anno=A,val=[H|T]}=S, #c_cons{}=Cons) -> - pat_alias(ann_c_cons_skel(A, #c_literal{anno=A,val=H}, - S#c_literal{val=T}), Cons); -pat_alias(#c_cons{anno=Anno,hd=H1,tl=T1}, #c_cons{hd=H2,tl=T2}) -> - ann_c_cons(Anno, pat_alias(H1, H2), pat_alias(T1, T2)); - -%% alias tuples -pat_alias(#c_tuple{anno=Anno,es=Es1}, #c_literal{val=T}) when is_tuple(T) -> - Es2 = [#c_literal{val=E} || E <- tuple_to_list(T)], - ann_c_tuple(Anno, pat_alias_list(Es1, Es2)); -pat_alias(#c_literal{anno=Anno,val=T}, #c_tuple{es=Es2}) when is_tuple(T) -> - Es1 = [#c_literal{val=E} || E <- tuple_to_list(T)], - ann_c_tuple(Anno, pat_alias_list(Es1, Es2)); -pat_alias(#c_tuple{anno=Anno,es=Es1}, #c_tuple{es=Es2}) -> - ann_c_tuple(Anno, pat_alias_list(Es1, Es2)); - -%% alias maps -%% There are no literals in maps patterns (patterns are always abstract) -pat_alias(#c_map{es=Es1}=M,#c_map{es=Es2}) -> - M#c_map{es=pat_alias_map_pairs(Es1++Es2)}; - -pat_alias(#c_alias{var=V1,pat=P1}, - #c_alias{var=V2,pat=P2}) -> - if V1 =:= V2 -> #c_alias{var=V1,pat=pat_alias(P1, P2)}; - true -> #c_alias{var=V1,pat=#c_alias{var=V2,pat=pat_alias(P1, P2)}} +pat_alias(#c_var{name=V1}=P, #c_var{name=V1}) -> P; +pat_alias(#c_var{name=V1}=Var, + #c_alias{var=#c_var{name=V2},pat=Pat}=Alias) -> + if + V1 =:= V2 -> + Alias; + true -> + Alias#c_alias{pat=pat_alias(Var, Pat)} + end; +pat_alias(#c_var{}=P1, P2) -> #c_alias{var=P1,pat=P2}; + +pat_alias(#c_alias{var=#c_var{name=V1}}=Alias, #c_var{name=V1}) -> + Alias; +pat_alias(#c_alias{var=#c_var{name=V1}=Var1,pat=P1}, + #c_alias{var=#c_var{name=V2}=Var2,pat=P2}) -> + Pat = pat_alias(P1, P2), + if + V1 =:= V2 -> + #c_alias{var=Var1,pat=Pat}; + true -> + pat_alias(Var1, pat_alias(Var2, Pat)) end; -pat_alias(#c_alias{var=V1,pat=P1}, P2) -> - #c_alias{var=V1,pat=pat_alias(P1, P2)}; -pat_alias(P1, #c_alias{var=V2,pat=P2}) -> - #c_alias{var=V2,pat=pat_alias(P1, P2)}; +pat_alias(#c_alias{var=#c_var{}=Var,pat=P1}, P2) -> + #c_alias{var=Var,pat=pat_alias(P1, P2)}; + +pat_alias(#c_map{es=Es1}=M, #c_map{es=Es2}) -> + M#c_map{es=pat_alias_map_pairs(Es1 ++ Es2)}; + +pat_alias(P1, #c_var{}=Var) -> + #c_alias{var=Var,pat=P1}; +pat_alias(P1, #c_alias{pat=P2}=Alias) -> + Alias#c_alias{pat=pat_alias(P1, P2)}; + pat_alias(P1, P2) -> - case {set_anno(P1, []),set_anno(P2, [])} of - {P,P} -> P; + %% Aliases between binaries are not allowed, so the only + %% legal patterns that remain are data patterns. + case cerl:is_data(P1) andalso cerl:is_data(P2) of + false -> throw(nomatch); + true -> ok + end, + Type = cerl:data_type(P1), + case cerl:data_type(P2) of + Type -> ok; _ -> throw(nomatch) - end. + end, + Es1 = cerl:data_es(P1), + Es2 = cerl:data_es(P2), + Es = pat_alias_list(Es1, Es2), + cerl:make_data(Type, Es). %% pat_alias_list([A1], [A2]) -> [A]. diff --git a/lib/compiler/src/v3_life.erl b/lib/compiler/src/v3_life.erl index cd4b5fd674..75bd188479 100644 --- a/lib/compiler/src/v3_life.erl +++ b/lib/compiler/src/v3_life.erl @@ -270,7 +270,7 @@ match(#k_select{anno=A,var=V,types=Kts}, Ls0, I, Ctxt, Vdb0) -> end, Vdb1 = use_vars(union(A#k.us, Ls1), I, Vdb0), Ts = [type_clause(Tc, Ls1, I+1, Ctxt, Vdb1) || Tc <- Kts], - #l{ke={select,literal2(V, Ctxt),Ts},i=I,vdb=Vdb1,a=Anno}; + #l{ke={select,literal(V, Ctxt),Ts},i=I,vdb=Vdb1,a=Anno}; match(#k_guard{anno=A,clauses=Kcs}, Ls, I, Ctxt, Vdb0) -> Vdb1 = use_vars(union(A#k.us, Ls), I, Vdb0), Cs = [guard_clause(G, Ls, I+1, Ctxt, Vdb1) || G <- Kcs], @@ -297,7 +297,7 @@ val_clause(#k_val_clause{anno=A,val=V,body=Kb}, Ls0, I, Ctxt0, Vdb0) -> _ -> Ctxt0 end, B = match(Kb, Ls1, I+1, Ctxt, Vdb1), - #l{ke={val_clause,literal2(V, Ctxt),B},i=I,vdb=use_vars(Bus, I+1, Vdb1),a=A#k.a}. + #l{ke={val_clause,literal(V, Ctxt),B},i=I,vdb=use_vars(Bus, I+1, Vdb1),a=A#k.a}. guard_clause(#k_guard_clause{anno=A,guard=Kg,body=Kb}, Ls, I, Ctxt, Vdb0) -> Vdb1 = use_vars(union(A#k.us, Ls), I+2, Vdb0), @@ -350,6 +350,7 @@ atomic_list(Ks) -> [atomic(K) || K <- Ks]. %% literal_list([Klit]) -> [Lit]. literal(#k_var{name=N}, _) -> {var,N}; +literal(#k_literal{val=I}, _) -> {literal,I}; literal(#k_int{val=I}, _) -> {integer,I}; literal(#k_float{val=F}, _) -> {float,F}; literal(#k_atom{val=N}, _) -> {atom,N}; @@ -358,58 +359,29 @@ literal(#k_nil{}, _) -> nil; literal(#k_cons{hd=H,tl=T}, Ctxt) -> {cons,[literal(H, Ctxt),literal(T, Ctxt)]}; literal(#k_binary{segs=V}, Ctxt) -> - {binary,literal(V, Ctxt)}; + {binary,literal(V, Ctxt)}; +literal(#k_bin_seg{size=S,unit=U,type=T,flags=Fs,seg=Seg,next=[]}, Ctxt) -> + %% Only occurs in patterns. + {bin_seg,Ctxt,literal(S, Ctxt),U,T,Fs,[literal(Seg, Ctxt)]}; literal(#k_bin_seg{size=S,unit=U,type=T,flags=Fs,seg=Seg,next=N}, Ctxt) -> {bin_seg,Ctxt,literal(S, Ctxt),U,T,Fs, [literal(Seg, Ctxt),literal(N, Ctxt)]}; +literal(#k_bin_int{size=S,unit=U,flags=Fs,val=Int,next=N}, Ctxt) -> + %% Only occurs in patterns. + {bin_int,Ctxt,literal(S, Ctxt),U,Fs,Int, + [literal(N, Ctxt)]}; literal(#k_bin_end{}, Ctxt) -> {bin_end,Ctxt}; literal(#k_tuple{es=Es}, Ctxt) -> {tuple,literal_list(Es, Ctxt)}; -literal(#k_map{op=Op,var=Var,es=Es}, Ctxt) -> - {map,Op,literal(Var, Ctxt),literal_list(Es, Ctxt)}; +literal(#k_map{op=Op,var=Var,es=Es0}, Ctxt) -> + {map,Op,literal(Var, Ctxt),literal_list(Es0, Ctxt)}; literal(#k_map_pair{key=K,val=V}, Ctxt) -> - {map_pair,literal(K, Ctxt),literal(V, Ctxt)}; -literal(#k_literal{val=V}, _Ctxt) -> - {literal,V}. + {map_pair,literal(K, Ctxt),literal(V, Ctxt)}. literal_list(Ks, Ctxt) -> [literal(K, Ctxt) || K <- Ks]. -literal2(#k_var{name=N}, _) -> {var,N}; -literal2(#k_literal{val=I}, _) -> {literal,I}; -literal2(#k_int{val=I}, _) -> {integer,I}; -literal2(#k_float{val=F}, _) -> {float,F}; -literal2(#k_atom{val=N}, _) -> {atom,N}; -%%literal2(#k_char{val=C}, _) -> {char,C}; -literal2(#k_nil{}, _) -> nil; -literal2(#k_cons{hd=H,tl=T}, Ctxt) -> - {cons,[literal2(H, Ctxt),literal2(T, Ctxt)]}; -literal2(#k_binary{segs=V}, Ctxt) -> - {binary,literal2(V, Ctxt)}; -literal2(#k_bin_seg{size=S,unit=U,type=T,flags=Fs,seg=Seg,next=[]}, Ctxt) -> - {bin_seg,Ctxt,literal2(S, Ctxt),U,T,Fs,[literal2(Seg, Ctxt)]}; -literal2(#k_bin_seg{size=S,unit=U,type=T,flags=Fs,seg=Seg,next=N}, Ctxt) -> - {bin_seg,Ctxt,literal2(S, Ctxt),U,T,Fs, - [literal2(Seg, Ctxt),literal2(N, Ctxt)]}; -literal2(#k_bin_int{size=S,unit=U,flags=Fs,val=Int,next=N}, Ctxt) -> - {bin_int,Ctxt,literal2(S, Ctxt),U,Fs,Int, - [literal2(N, Ctxt)]}; -literal2(#k_bin_end{}, Ctxt) -> - {bin_end,Ctxt}; -literal2(#k_tuple{es=Es}, Ctxt) -> - {tuple,literal_list2(Es, Ctxt)}; -literal2(#k_map{op=Op,es=Es}, Ctxt) -> - {map,Op,literal_list2(Es, Ctxt)}; -literal2(#k_map_pair{key=K,val=V}, Ctxt) -> - {map_pair,literal2(K, Ctxt),literal2(V, Ctxt)}. - -literal_list2(Ks, Ctxt) -> - [literal2(K, Ctxt) || K <- Ks]. - -%% literal_bin(#k_bin_seg{size=S,unit=U,type=T,flags=Fs,seg=Seg,next=N}) -> -%% {bin_seg,literal(S),U,T,Fs,[literal(Seg),literal(N)]} - %% is_gc_bif(Name, Arity) -> true|false %% Determines whether the BIF Name/Arity might do a GC. diff --git a/lib/compiler/test/andor_SUITE.erl b/lib/compiler/test/andor_SUITE.erl index 3199440d84..4d7f444c4f 100644 --- a/lib/compiler/test/andor_SUITE.erl +++ b/lib/compiler/test/andor_SUITE.erl @@ -370,6 +370,11 @@ combined(Config) when is_list(Config) -> ?line true = ?COMB(false, blurf, true), ?line true = ?COMB(true, true, blurf), + false = simple_comb(false, false), + false = simple_comb(false, true), + false = simple_comb(true, false), + true = simple_comb(true, true), + ok. -undef(COMB). @@ -396,6 +401,13 @@ comb(A, B, C) -> end, id(Res). +simple_comb(A, B) -> + %% Use Res twice, to ensure that a careless optimization of 'not' + %% doesn't leave Res as a free variable. + Res = A andalso B, + _ = id(not Res), + Res. + %% Test that a boolean expression in a case expression is properly %% optimized (in particular, that the error behaviour is correct). in_case(Config) when is_list(Config) -> diff --git a/lib/compiler/test/core_fold_SUITE.erl b/lib/compiler/test/core_fold_SUITE.erl index 512aada203..bc82eaf5aa 100644 --- a/lib/compiler/test/core_fold_SUITE.erl +++ b/lib/compiler/test/core_fold_SUITE.erl @@ -88,6 +88,7 @@ t_element(Config) when is_list(Config) -> {_,_,_}=Tup -> ?line {'EXIT',{badarg,_}} = (catch element(4, Tup)) end, + {'EXIT',{badarg,_}} = (catch element(1, tuple_size(Tuple))), ok. @@ -106,6 +107,7 @@ setelement(Config) when is_list(Config) -> ?line error = setelement_crash_2({a,b,c,d,e,f}, <<42>>), {'EXIT',{badarg,_}} = (catch setelement(1, not_a_tuple, New)), + {'EXIT',{badarg,_}} = (catch setelement(3, {a,b}, New)), ok. diff --git a/lib/compiler/test/map_SUITE.erl b/lib/compiler/test/map_SUITE.erl index cfa8262701..1c9574c2a2 100644 --- a/lib/compiler/test/map_SUITE.erl +++ b/lib/compiler/test/map_SUITE.erl @@ -543,7 +543,7 @@ t_map_sort_literals(Config) when is_list(Config) -> true = id(#{ c => 1, b => 1, a => 1 }) < id(#{ b => 1, c => 1, d => 1}), true = id(#{ "a" => 1 }) < id(#{ <<"a">> => 1}), false = id(#{ <<"a">> => 1 }) < id(#{ "a" => 1}), - false = id(#{ 1 => 1 }) < id(#{ 1.0 => 1}), + true = id(#{ 1 => 1 }) < id(#{ 1.0 => 1}), false = id(#{ 1.0 => 1 }) < id(#{ 1 => 1}), %% value order diff --git a/lib/compiler/test/match_SUITE.erl b/lib/compiler/test/match_SUITE.erl index 7522ee985f..9aec0b3d4e 100644 --- a/lib/compiler/test/match_SUITE.erl +++ b/lib/compiler/test/match_SUITE.erl @@ -141,6 +141,13 @@ aliases(Config) when is_list(Config) -> ?line {a,b} = list_alias2([a,b]), ?line {a,b} = list_alias3([a,b]), + %% Non-matching aliases. + none = mixed_aliases(<<42>>), + none = mixed_aliases([b]), + none = mixed_aliases([d]), + none = mixed_aliases({a,42}), + none = mixed_aliases(42), + ok. str_alias(V) -> @@ -244,6 +251,12 @@ list_alias2([X,Y]=[a,b]) -> list_alias3([X,b]=[a,Y]) -> {X,Y}. +mixed_aliases(<<X:8>> = x) -> {a,X}; +mixed_aliases([b] = <<X:8>>) -> {b,X}; +mixed_aliases(<<X:8>> = {a,X}) -> {c,X}; +mixed_aliases([X] = <<X:8>>) -> {d,X}; +mixed_aliases(_) -> none. + %% OTP-7018. match_in_call(Config) when is_list(Config) -> diff --git a/lib/compiler/test/warnings_SUITE.erl b/lib/compiler/test/warnings_SUITE.erl index dcd3910926..d0b7c71be8 100644 --- a/lib/compiler/test/warnings_SUITE.erl +++ b/lib/compiler/test/warnings_SUITE.erl @@ -39,7 +39,7 @@ guard/1,bad_arith/1,bool_cases/1,bad_apply/1, files/1,effect/1,bin_opt_info/1,bin_construction/1, comprehensions/1,maps/1,redundant_boolean_clauses/1, - latin1_fallback/1,underscore/1]). + latin1_fallback/1,underscore/1,no_warnings/1]). % Default timetrap timeout (set in init_per_testcase). -define(default_timeout, ?t:minutes(2)). @@ -65,7 +65,7 @@ groups() -> bad_arith,bool_cases,bad_apply,files,effect, bin_opt_info,bin_construction,comprehensions,maps, redundant_boolean_clauses,latin1_fallback, - underscore]}]. + underscore,no_warnings]}]. init_per_suite(Config) -> Config. @@ -281,7 +281,6 @@ bad_arith(Config) when is_list(Config) -> {3,sys_core_fold,{eval_failure,badarith}}, {9,sys_core_fold,nomatch_guard}, {9,sys_core_fold,{eval_failure,badarith}}, - {9,sys_core_fold,{no_effect,{erlang,is_integer,1}}}, {10,sys_core_fold,nomatch_guard}, {10,sys_core_fold,{eval_failure,badarith}}, {15,sys_core_fold,{eval_failure,badarith}} @@ -719,6 +718,27 @@ underscore(Config) when is_list(Config) -> ok. +no_warnings(Config) when is_list(Config) -> + Ts = [{no_warnings, + <<"-record(r, {s=ordsets:new(),a,b}). + + a() -> + R = #r{}, %No warning expected. + {R#r.a,R#r.b}. + + b(X) -> + T = true, + Var = [X], %No warning expected. + case T of + false -> Var; + true -> [] + end. + ">>, + [], + []}], + run(Config, Ts), + ok. + %%% %%% End of test cases. %%% diff --git a/lib/debugger/test/map_SUITE.erl b/lib/debugger/test/map_SUITE.erl index e525484a8e..b114d29f44 100644 --- a/lib/debugger/test/map_SUITE.erl +++ b/lib/debugger/test/map_SUITE.erl @@ -494,7 +494,7 @@ t_map_sort_literals(Config) when is_list(Config) -> true = #{ c => 1, b => 1, a => 1 } < id(#{ b => 1, c => 1, d => 1}), true = #{ "a" => 1 } < id(#{ <<"a">> => 1}), false = #{ <<"a">> => 1 } < id(#{ "a" => 1}), - false = #{ 1 => 1 } < id(#{ 1.0 => 1}), + true = #{ 1 => 1 } < id(#{ 1.0 => 1}), false = #{ 1.0 => 1 } < id(#{ 1 => 1}), %% value order @@ -587,12 +587,11 @@ t_bif_map_is_key(Config) when is_list(Config) -> t_bif_map_keys(Config) when is_list(Config) -> [] = maps:keys(#{}), - [1,2,3,4,5] = maps:keys(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e}), - [1,2,3,4,5] = maps:keys(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c}), + [1,2,3,4,5] = lists:sort(maps:keys(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e})), + [1,2,3,4,5] = lists:sort(maps:keys(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c})), - % values in key order: [4,int,"hi",<<"key">>] M1 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, 4 => number}, - [4,int,"hi",<<"key">>] = maps:keys(M1), + [4,int,"hi",<<"key">>] = lists:sort(maps:keys(M1)), %% error case {'EXIT',{badarg,[{maps,keys,_,_}|_]}} = (catch maps:keys(1 bsl 65 + 3)), @@ -634,40 +633,39 @@ t_bif_map_merge(Config) when is_list(Config) -> ok. - t_bif_map_put(Config) when is_list(Config) -> M0 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, 4 => number, 18446744073709551629 => wat}, M1 = #{ "hi" := "hello"} = maps:put("hi", "hello", #{}), - ["hi"] = maps:keys(M1), - ["hello"] = maps:values(M1), + true = is_members(["hi"],maps:keys(M1)), + true = is_members(["hello"],maps:values(M1)), M2 = #{ int := 3 } = maps:put(int, 3, M1), - [int,"hi"] = maps:keys(M2), - [3,"hello"] = maps:values(M2), + true = is_members([int,"hi"],maps:keys(M2)), + true = is_members([3,"hello"],maps:values(M2)), M3 = #{ <<"key">> := <<"value">> } = maps:put(<<"key">>, <<"value">>, M2), - [int,"hi",<<"key">>] = maps:keys(M3), - [3,"hello",<<"value">>] = maps:values(M3), + true = is_members([int,"hi",<<"key">>],maps:keys(M3)), + true = is_members([3,"hello",<<"value">>],maps:values(M3)), M4 = #{ 18446744073709551629 := wat } = maps:put(18446744073709551629, wat, M3), - [18446744073709551629,int,"hi",<<"key">>] = maps:keys(M4), - [wat,3,"hello",<<"value">>] = maps:values(M4), + true = is_members([18446744073709551629,int,"hi",<<"key">>],maps:keys(M4)), + true = is_members([wat,3,"hello",<<"value">>],maps:values(M4)), M0 = #{ 4 := number } = M5 = maps:put(4, number, M4), - [4,18446744073709551629,int,"hi",<<"key">>] = maps:keys(M5), - [number,wat,3,"hello",<<"value">>] = maps:values(M5), + true = is_members([4,18446744073709551629,int,"hi",<<"key">>],maps:keys(M5)), + true = is_members([number,wat,3,"hello",<<"value">>],maps:values(M5)), M6 = #{ <<"key">> := <<"other value">> } = maps:put(<<"key">>, <<"other value">>, M5), - [4,18446744073709551629,int,"hi",<<"key">>] = maps:keys(M6), - [number,wat,3,"hello",<<"other value">>] = maps:values(M6), + true = is_members([4,18446744073709551629,int,"hi",<<"key">>],maps:keys(M6)), + true = is_members([number,wat,3,"hello",<<"other value">>],maps:values(M6)), %% error case {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,1 bsl 65 + 3)), @@ -675,46 +673,16 @@ t_bif_map_put(Config) when is_list(Config) -> {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,atom)), {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,[])), {'EXIT',{badarg,[{maps,put,_,_}|_]}} = (catch maps:put(1,a,<<>>)), - ok. - -t_bif_map_remove(Config) when is_list(Config) -> - 0 = erlang:map_size(maps:remove(some_key, #{})), - - M0 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, - 4 => number, 18446744073709551629 => wat}, - - M1 = maps:remove("hi", M0), - [4,18446744073709551629,int,<<"key">>] = maps:keys(M1), - [number,wat,3,<<"value">>] = maps:values(M1), - - M2 = maps:remove(int, M1), - [4,18446744073709551629,<<"key">>] = maps:keys(M2), - [number,wat,<<"value">>] = maps:values(M2), - - M3 = maps:remove(<<"key">>, M2), - [4,18446744073709551629] = maps:keys(M3), - [number,wat] = maps:values(M3), - - M4 = maps:remove(18446744073709551629, M3), - [4] = maps:keys(M4), - [number] = maps:values(M4), - - M5 = maps:remove(4, M4), - [] = maps:keys(M5), - [] = maps:values(M5), + ok. - M0 = maps:remove(5,M0), - M0 = maps:remove("hi there",M0), +is_members(Ks,Ls) when length(Ks) =/= length(Ls) -> false; +is_members(Ks,Ls) -> is_members_do(Ks,Ls). - #{ "hi" := "hello", int := 3, 4 := number} = maps:remove(18446744073709551629,maps:remove(<<"key">>,M0)), +is_members_do([],[]) -> true; +is_members_do([],_) -> false; +is_members_do([K|Ks],Ls) -> + is_members_do(Ks, lists:delete(K,Ls)). - %% error case - {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,1 bsl 65 + 3)), - {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(1,154)), - {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,atom)), - {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(1,[])), - {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,<<>>)), - ok. t_bif_map_update(Config) when is_list(Config) -> M0 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, @@ -742,20 +710,57 @@ t_bif_map_update(Config) when is_list(Config) -> ok. +t_bif_map_remove(Config) when is_list(Config) -> + 0 = erlang:map_size(maps:remove(some_key, #{})), + + M0 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, + 4 => number, 18446744073709551629 => wat}, + + M1 = maps:remove("hi", M0), + true = is_members([4,18446744073709551629,int,<<"key">>],maps:keys(M1)), + true = is_members([number,wat,3,<<"value">>],maps:values(M1)), + + M2 = maps:remove(int, M1), + true = is_members([4,18446744073709551629,<<"key">>],maps:keys(M2)), + true = is_members([number,wat,<<"value">>],maps:values(M2)), + + M3 = maps:remove(<<"key">>, M2), + true = is_members([4,18446744073709551629],maps:keys(M3)), + true = is_members([number,wat],maps:values(M3)), + M4 = maps:remove(18446744073709551629, M3), + true = is_members([4],maps:keys(M4)), + true = is_members([number],maps:values(M4)), + + M5 = maps:remove(4, M4), + [] = maps:keys(M5), + [] = maps:values(M5), + + M0 = maps:remove(5,M0), + M0 = maps:remove("hi there",M0), + + #{ "hi" := "hello", int := 3, 4 := number} = maps:remove(18446744073709551629,maps:remove(<<"key">>,M0)), + + %% error case + {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,1 bsl 65 + 3)), + {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(1,154)), + {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,atom)), + {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(1,[])), + {'EXIT',{badarg,[{maps,remove,_,_}|_]}} = (catch maps:remove(a,<<>>)), + ok. t_bif_map_values(Config) when is_list(Config) -> [] = maps:values(#{}), + [1] = maps:values(#{a=>1}), - [a,b,c,d,e] = maps:values(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e}), - [a,b,c,d,e] = maps:values(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c}), + true = is_members([a,b,c,d,e],maps:values(#{ 1 => a, 2 => b, 3 => c, 4 => d, 5 => e})), + true = is_members([a,b,c,d,e],maps:values(#{ 4 => d, 5 => e, 1 => a, 2 => b, 3 => c})), - % values in key order: [4,int,"hi",<<"key">>] M1 = #{ "hi" => "hello", int => 3, <<"key">> => <<"value">>, 4 => number}, M2 = M1#{ "hi" => "hello2", <<"key">> => <<"value2">> }, - [number,3,"hello2",<<"value2">>] = maps:values(M2), - [number,3,"hello",<<"value">>] = maps:values(M1), + true = is_members([number,3,"hello2",<<"value2">>],maps:values(M2)), + true = is_members([number,3,"hello",<<"value">>],maps:values(M1)), %% error case {'EXIT',{badarg,[{maps,values,_,_}|_]}} = (catch maps:values(1 bsl 65 + 3)), @@ -764,75 +769,74 @@ t_bif_map_values(Config) when is_list(Config) -> {'EXIT',{badarg,[{maps,values,_,_}|_]}} = (catch maps:values(<<>>)), ok. + + t_erlang_hash(Config) when is_list(Config) -> ok = t_bif_erlang_phash2(), ok = t_bif_erlang_phash(), ok = t_bif_erlang_hash(), - ok. t_bif_erlang_phash2() -> - 39679005 = erlang:phash2(#{}), - 78942764 = erlang:phash2(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 }), - 37338230 = erlang:phash2(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} }), - 14363616 = erlang:phash2(#{ 1 => a }), - 51612236 = erlang:phash2(#{ a => 1 }), + 33667975 = erlang:phash2(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 }), % 78942764 + 95332690 = erlang:phash2(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} }), % 37338230 + 108954384 = erlang:phash2(#{ 1 => a }), % 14363616 + 59617982 = erlang:phash2(#{ a => 1 }), % 51612236 - 37468437 = erlang:phash2(#{{} => <<>>}), - 44049159 = erlang:phash2(#{<<>> => {}}), + 42770201 = erlang:phash2(#{{} => <<>>}), % 37468437 + 71687700 = erlang:phash2(#{<<>> => {}}), % 44049159 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 118679416 = erlang:phash2(M0), - 51612236 = erlang:phash2(M1), - 118679416 = erlang:phash2(M2), + 70249457 = erlang:phash2(M0), % 118679416 + 59617982 = erlang:phash2(M1), % 51612236 + 70249457 = erlang:phash2(M2), % 118679416 ok. t_bif_erlang_phash() -> Sz = 1 bsl 32, - 268440612 = erlang:phash(#{},Sz), - 1196461908 = erlang:phash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), - 3944426064 = erlang:phash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), - 1394238263 = erlang:phash(#{ 1 => a },Sz), - 4066388227 = erlang:phash(#{ a => 1 },Sz), + 1113425985 = erlang:phash(#{},Sz), % 268440612 + 1510068139 = erlang:phash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), % 1196461908 + 3182345590 = erlang:phash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), % 3944426064 + 2927531828 = erlang:phash(#{ 1 => a },Sz), % 1394238263 + 1670235874 = erlang:phash(#{ a => 1 },Sz), % 4066388227 - 1578050717 = erlang:phash(#{{} => <<>>},Sz), - 1578050717 = erlang:phash(#{<<>> => {}},Sz), % yep, broken + 3935089469 = erlang:phash(#{{} => <<>>},Sz), % 1578050717 + 71692856 = erlang:phash(#{<<>> => {}},Sz), % 1578050717 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 3590546636 = erlang:phash(M0,Sz), - 4066388227 = erlang:phash(M1,Sz), - 3590546636 = erlang:phash(M2,Sz), + 2620391445 = erlang:phash(M0,Sz), % 3590546636 + 1670235874 = erlang:phash(M1,Sz), % 4066388227 + 2620391445 = erlang:phash(M2,Sz), % 3590546636 ok. t_bif_erlang_hash() -> Sz = 1 bsl 27 - 1, - 5158 = erlang:hash(#{},Sz), - 71555838 = erlang:hash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), - 5497225 = erlang:hash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), - 126071654 = erlang:hash(#{ 1 => a },Sz), - 126426236 = erlang:hash(#{ a => 1 },Sz), + 39684169 = erlang:hash(#{},Sz), % 5158 + 33673142 = erlang:hash(#{ a => 1, "a" => 2, <<"a">> => 3, {a,b} => 4 },Sz), % 71555838 + 95337869 = erlang:hash(#{ 1 => a, 2 => "a", 3 => <<"a">>, 4 => {a,b} },Sz), % 5497225 + 108959561 = erlang:hash(#{ 1 => a },Sz), % 126071654 + 59623150 = erlang:hash(#{ a => 1 },Sz), % 126426236 - 101655720 = erlang:hash(#{{} => <<>>},Sz), - 101655720 = erlang:hash(#{<<>> => {}},Sz), % yep, broken + 42775386 = erlang:hash(#{{} => <<>>},Sz), % 101655720 + 71692856 = erlang:hash(#{<<>> => {}},Sz), % 101655720 M0 = #{ a => 1, "key" => <<"value">> }, M1 = maps:remove("key",M0), M2 = M1#{ "key" => <<"value">> }, - 38260486 = erlang:hash(M0,Sz), - 126426236 = erlang:hash(M1,Sz), - 38260486 = erlang:hash(M2,Sz), + 70254632 = erlang:hash(M0,Sz), % 38260486 + 59623150 = erlang:hash(M1,Sz), % 126426236 + 70254632 = erlang:hash(M2,Sz), % 38260486 ok. - t_map_encode_decode(Config) when is_list(Config) -> <<131,116,0,0,0,0>> = erlang:term_to_binary(#{}), Pairs = [ @@ -895,39 +899,42 @@ t_map_encode_decode(Config) when is_list(Config) -> map_encode_decode_and_match([{K,V}|Pairs], EncodedPairs, M0) -> M1 = maps:put(K,V,M0), B0 = erlang:term_to_binary(M1), - Ls = lists:sort(fun(A,B) -> erts_internal:cmp_term(A,B) < 0 end, [{K, erlang:term_to_binary(K), erlang:term_to_binary(V)}|EncodedPairs]), - %% sort Ks and Vs according to term spec, then match it - KVbins = lists:foldr(fun({_,Kbin,Vbin}, Acc) -> [Kbin,Vbin | Acc] end, [], Ls), - ok = match_encoded_map(B0, length(Ls), KVbins), + Ls = [{erlang:term_to_binary(K), erlang:term_to_binary(V)}|EncodedPairs], + ok = match_encoded_map(B0, length(Ls), Ls), %% decode and match it M1 = erlang:binary_to_term(B0), map_encode_decode_and_match(Pairs,Ls,M1); map_encode_decode_and_match([],_,_) -> ok. match_encoded_map(<<131,116,Size:32,Encoded/binary>>,Size,Items) -> - match_encoded_map(Encoded,Items); + match_encoded_map_stripped_size(Encoded,Items,Items); match_encoded_map(_,_,_) -> no_match_size. -match_encoded_map(<<>>,[]) -> ok; -match_encoded_map(Bin,[<<131,Item/binary>>|Items]) -> - Size = erlang:byte_size(Item), - <<EncodedTerm:Size/binary, Bin1/binary>> = Bin, - EncodedTerm = Item, %% Asssert - match_encoded_map(Bin1,Items). +match_encoded_map_stripped_size(<<>>,_,_) -> ok; +match_encoded_map_stripped_size(B0,[{<<131,K/binary>>,<<131,V/binary>>}|Items],Ls) -> + Ksz = byte_size(K), + Vsz = byte_size(V), + case B0 of + <<K:Ksz/binary,V:Vsz/binary,B1/binary>> -> + match_encoded_map_stripped_size(B1,Ls,Ls); + _ -> + match_encoded_map_stripped_size(B0,Items,Ls) + end; +match_encoded_map_stripped_size(_,[],_) -> fail. t_bif_map_to_list(Config) when is_list(Config) -> [] = maps:to_list(#{}), - [{a,1},{b,2}] = maps:to_list(#{a=>1,b=>2}), - [{a,1},{b,2},{c,3}] = maps:to_list(#{c=>3,a=>1,b=>2}), - [{a,1},{b,2},{g,3}] = maps:to_list(#{g=>3,a=>1,b=>2}), - [{a,1},{b,2},{g,3},{"c",4}] = maps:to_list(#{g=>3,a=>1,b=>2,"c"=>4}), - [{3,v2},{hi,v4},{{hi,3},v5},{"hi",v3},{<<"hi">>,v1}] = maps:to_list(#{ - <<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5}), - - [{3,v7},{hi,v9},{{hi,3},v10},{"hi",v8},{<<"hi">>,v6}] = maps:to_list(#{ - <<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5, - <<"hi">>=>v6,3=>v7,"hi"=>v8,hi=>v9,{hi,3}=>v10}), + [{a,1},{b,2}] = lists:sort(maps:to_list(#{a=>1,b=>2})), + [{a,1},{b,2},{c,3}] = lists:sort(maps:to_list(#{c=>3,a=>1,b=>2})), + [{a,1},{b,2},{g,3}] = lists:sort(maps:to_list(#{g=>3,a=>1,b=>2})), + [{a,1},{b,2},{g,3},{"c",4}] = lists:sort(maps:to_list(#{g=>3,a=>1,b=>2,"c"=>4})), + [{3,v2},{hi,v4},{{hi,3},v5},{"hi",v3},{<<"hi">>,v1}] = + lists:sort(maps:to_list(#{<<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5})), + + [{3,v7},{hi,v9},{{hi,3},v10},{"hi",v8},{<<"hi">>,v6}] = + lists:sort(maps:to_list(#{<<"hi">>=>v1,3=>v2,"hi"=>v3,hi=>v4,{hi,3}=>v5, + <<"hi">>=>v6,3=>v7,"hi"=>v8,hi=>v9,{hi,3}=>v10})), %% error cases {'EXIT', {badarg,_}} = (catch maps:to_list(id(a))), diff --git a/lib/dialyzer/src/dialyzer.app.src b/lib/dialyzer/src/dialyzer.app.src index 1756800c4f..7b2e1d4a9d 100644 --- a/lib/dialyzer/src/dialyzer.app.src +++ b/lib/dialyzer/src/dialyzer.app.src @@ -46,5 +46,5 @@ {applications, [compiler, gs, hipe, kernel, stdlib, wx]}, {env, []}, {runtime_dependencies, ["wx-1.2","syntax_tools-1.6.14","stdlib-2.0", - "kernel-3.0","hipe-3.10.3","erts-6.0", + "kernel-3.0","hipe-3.10.3","erts-7.0", "compiler-5.0"]}]}. diff --git a/lib/dialyzer/src/dialyzer_dataflow.erl b/lib/dialyzer/src/dialyzer_dataflow.erl index 336b4641d4..48d1ab9ebc 100644 --- a/lib/dialyzer/src/dialyzer_dataflow.erl +++ b/lib/dialyzer/src/dialyzer_dataflow.erl @@ -3519,6 +3519,7 @@ find_terminals(Tree) -> 'let' -> find_terminals(cerl:let_body(Tree)); letrec -> find_terminals(cerl:letrec_body(Tree)); literal -> {false, true}; + map -> {false, true}; primop -> {false, false}; %% match_fail, etc. are not explicit exits. 'receive' -> Timeout = cerl:receive_timeout(Tree), diff --git a/lib/dialyzer/src/dialyzer_timing.erl b/lib/dialyzer/src/dialyzer_timing.erl index b1a4bdc07c..759d49abc8 100644 --- a/lib/dialyzer/src/dialyzer_timing.erl +++ b/lib/dialyzer/src/dialyzer_timing.erl @@ -38,7 +38,7 @@ init(Active) -> case Active of true -> io:format("\n"), - spawn_link(fun() -> loop(now(), 0, "") end); + spawn_link(fun() -> loop(erlang:monotonic_time(), 0, "") end); debug -> io:format("\n"), spawn_link(fun() -> debug_loop("") end); @@ -105,14 +105,14 @@ debug_loop(Phase) -> start_stamp(none, _) -> ok; start_stamp(Pid, Msg) -> - Pid ! {stamp, Msg, now()}, + Pid ! {stamp, Msg, erlang:monotonic_time()}, ok. -spec end_stamp(timing_server()) -> ok. end_stamp(none) -> ok; end_stamp(Pid) -> - Pid ! {stamp, now()}, + Pid ! {stamp, erlang:monotonic_time()}, ok. -spec send_size_info(timing_server(), integer(), string()) -> ok. @@ -126,8 +126,8 @@ send_size_info(Pid, Size, Unit) -> stop(none) -> ok; stop(Pid) -> - Pid ! {self(), stop, now()}, + Pid ! {self(), stop, erlang:monotonic_time()}, receive ok -> ok end. diff(T2, T1) -> - timer:now_diff(T2,T1) / 1000000. + (T2-T1) / erlang:convert_time_unit(1, seconds, native). diff --git a/lib/diameter/doc/src/diameter.xml b/lib/diameter/doc/src/diameter.xml index 638c1c4c2b..37e67d8630 100644 --- a/lib/diameter/doc/src/diameter.xml +++ b/lib/diameter/doc/src/diameter.xml @@ -303,7 +303,7 @@ Defaults to <c>none</c>.</p> <tag><c>{timeout, &dict_Unsigned32;}</c></tag> <item> <p> -The number of milliseconds after which the request should +Number of milliseconds after which the request should timeout. Defaults to 5000.</p> </item> @@ -674,7 +674,7 @@ connection establishment.</p> <tag><c>{'CEA', Result, Caps, Pkt}</c></tag> <item> <pre> -Result = integer() | atom() | {capabilities_cb, CB, ResultCode|discard} +Result = ResultCode | atom() | {capabilities_cb, CB, ResultCode|discard} Caps = #diameter_caps{} Pkt = #diameter_packet{} ResultCode = integer() @@ -742,7 +742,7 @@ info fields of forms other than the above.</p> <tag><c>service_name() = term()</c></tag> <item> <p> -The name of a service as passed to &start_service; and with which the +Name of a service as passed to &start_service; and with which the service is identified. There can be at most one service with a given name on a given node. Note that &make_ref; @@ -754,7 +754,7 @@ can be used to generate a service name that is somewhat unique.</p> <tag><c>service_opt()</c></tag> <item> <p> -An option passed to &start_service;. +Option passed to &start_service;. Can be any <c>&capability;</c> as well as the following.</p> <taglist> @@ -762,7 +762,7 @@ Can be any <c>&capability;</c> as well as the following.</p> <tag><c>{application, [&application_opt;]}</c></tag> <item> <p> -Defines a Diameter application supported by the service.</p> +A Diameter application supported by the service.</p> <p> A service must configure one tuple for each Diameter @@ -790,7 +790,7 @@ be matched by corresponding &capability; configuration, of | evaluable()}</c></tag> <item> <p> -Specifies the degree to which the service allows multiple transport +The degree to which the service allows multiple transport connections to the same peer, as identified by its Origin-Host at capabilities exchange.</p> @@ -816,7 +816,7 @@ Defaults to <c>nodes</c>.</p> <tag><c>{sequence, {H,N} | &evaluable;}</c></tag> <item> <p> -Specifies a constant value <c>H</c> for the topmost <c>32-N</c> bits of +A constant value <c>H</c> for the topmost <c>32-N</c> bits of of 32-bit End-to-End and Hop-by-Hop Identifiers generated by the service, either explicitly or as a return value of a function to be evaluated at &start_service;. @@ -851,7 +851,7 @@ outgoing requests.</p> <tag><c>{share_peers, boolean() | [node()] | evaluable()}</c></tag> <item> <p> -Specifies nodes to which peer connections established on the local +Nodes to which peer connections established on the local Erlang node are communicated. Shared peers become available in the remote candidates list passed to &app_pick_peer; callbacks on remote nodes whose services are @@ -890,7 +890,7 @@ of a single Diameter node across multiple Erlang nodes.</p> <tag><c>{spawn_opt, [term()]}</c></tag> <item> <p> -An options list passed to &spawn_opt; when spawning a process for an +Options list passed to &spawn_opt; when spawning a process for an incoming Diameter request, unless the transport in question specifies another value. Options <c>monitor</c> and <c>link</c> are ignored.</p> @@ -899,10 +899,34 @@ Options <c>monitor</c> and <c>link</c> are ignored.</p> Defaults to the empty list.</p> </item> +<marker id="string_decode"/> +<tag><c>{string_decode, boolean()}</c></tag> +<item> +<p> +Whether or not to decode AVPs of type &dict_OctetString; and its +derived types &dict_DiameterIdentity;, &dict_DiameterURI;, +&dict_IPFilterRule;, &dict_QoSFilterRule;, and &dict_UTF8String;. +If <c>true</c> then AVPs of these types are decoded to string(). +If <c>false</c> then values are retained as binary().</p> + +<p> +Defaults to <c>true</c>.</p> + +<warning> +<p> +This option should be set to <c>false</c> +since a sufficiently malicious peer can otherwise cause large amounts +of memory to be consumed when decoded Diameter messages are passed +between processes. +The default value is for backwards compatibility.</p> +</warning> + +</item> + <tag><c>{use_shared_peers, boolean() | [node()] | evaluable()}</c></tag> <item> <p> -Specifies nodes from which communicated peers are made available in +Nodes from which communicated peers are made available in the remote candidates list of &app_pick_peer; callbacks.</p> <p> @@ -942,7 +966,7 @@ each node from which requests are sent.</p> <tag><c>transport_opt()</c></tag> <item> <p> -An option passed to &add_transport;. +Option passed to &add_transport;. Has one of the following types.</p> <taglist> @@ -950,8 +974,7 @@ Has one of the following types.</p> <tag><c>{applications, [&application_alias;]}</c></tag> <item> <p> -The list of Diameter applications to which the transport should be -restricted. +Diameter applications to which the transport should be restricted. Defaults to all applications configured on the service in question. Applications not configured on the service in question are ignored.</p> @@ -984,7 +1007,7 @@ TLS is desired over TCP as implemented by &man_tcp;.</p> <tag><c>{capabilities_cb, &evaluable;}</c></tag> <item> <p> -A callback invoked upon reception of CER/CEA during capabilities +Callback invoked upon reception of CER/CEA during capabilities exchange in order to ask whether or not the connection should be accepted. Applied to the <c>&transport_ref;</c> and @@ -1115,7 +1138,7 @@ Defaults to <c>rebooting</c> for <c>Reason=service|application</c> and <p> Number of milliseconds after which the transport process is terminated if DPA has not been received. -Defaults to 1000.</p> +Defaults to the value of &dpa_timeout;.</p> </item> </taglist> </item> @@ -1152,6 +1175,29 @@ configured them.</p> Defaults to a single callback returning <c>dpr</c>.</p> </item> +<marker id="dpa_timeout"/> +<tag><c>{dpa_timeout, &dict_Unsigned32;}</c></tag> +<item> +<p> +Number of milliseconds after which a transport connection is +terminated following an outgoing DPR if DPA is not received.</p> + +<p> +Defaults to 1000.</p> +</item> + +<marker id="dpr_timeout"/> +<tag><c>{dpr_timeout, &dict_Unsigned32;}</c></tag> +<item> +<p> +Number of milliseconds after which a transport connection is +terminated following an incoming DPR if the peer does not close the +connection.</p> + +<p> +Defaults to 5000.</p> +</item> + <marker id="length_errors"/> <tag><c>{length_errors, exit|handle|discard}</c></tag> <item> @@ -1207,7 +1253,7 @@ the same peer.</p> <tag><c>{spawn_opt, [term()]}</c></tag> <item> <p> -Options list passed to &spawn_opt; when spawning a process for an +Options passed to &spawn_opt; when spawning a process for an incoming Diameter request. Options <c>monitor</c> and <c>link</c> are ignored.</p> diff --git a/lib/diameter/doc/src/diameter_dict.xml b/lib/diameter/doc/src/diameter_dict.xml index 810a146b88..9db9bcffde 100644 --- a/lib/diameter/doc/src/diameter_dict.xml +++ b/lib/diameter/doc/src/diameter_dict.xml @@ -529,6 +529,11 @@ answer record and passed to a &app_handle_request; callback upon reception of an incoming request.</p> <p> +In cases in which there is a choice between list() and binary() types +for OctetString() and derived types, the representation is determined +by the value of &mod_string_decode;.</p> + +<p> <em>Basic AVP Data Formats</em></p> <marker id="OctetString"/> @@ -541,7 +546,7 @@ callback upon reception of an incoming request.</p> <marker id="Grouped"/> <pre> -OctetString() = [0..255] +OctetString() = string() | binary() Integer32() = -2147483647..2147483647 Integer64() = -9223372036854775807..9223372036854775807 Unsigned32() = 0..4294967295 @@ -603,7 +608,7 @@ and <c>{{2104,2,26},{9,42,23}}</c> (both inclusive) can be encoded.</p> <marker id="UTF8String"/> <pre> -UTF8String() = [integer()] +UTF8String() = [integer()] | binary() </pre> <p> diff --git a/lib/diameter/doc/src/seealso.ent b/lib/diameter/doc/src/seealso.ent index 44541afb9b..4e205ffad7 100644 --- a/lib/diameter/doc/src/seealso.ent +++ b/lib/diameter/doc/src/seealso.ent @@ -4,7 +4,7 @@ %CopyrightBegin% -Copyright Ericsson AB 2012-2014. All Rights Reserved. +Copyright Ericsson AB 2012-2015. All Rights Reserved. The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in @@ -64,11 +64,14 @@ significant. <!ENTITY capabilities_cb '<seealso marker="#capabilities_cb">capabilities_cb</seealso>'> <!ENTITY capx_timeout '<seealso marker="#capx_timeout">capx_timeout</seealso>'> <!ENTITY disconnect_cb '<seealso marker="#disconnect_cb">disconnect_cb</seealso>'> +<!ENTITY dpa_timeout '<seealso marker="#dpa_timeout">dpa_timeout</seealso>'> <!ENTITY transport_config '<seealso marker="#transport_config">transport_config</seealso>'> <!ENTITY transport_module '<seealso marker="#transport_module">transport_module</seealso>'> <!ENTITY connect_timer '<seealso marker="#connect_timer">connect_timer</seealso>'> <!ENTITY watchdog_timer '<seealso marker="#watchdog_timer">watchdog_timer</seealso>'> +<!ENTITY mod_string_decode '<seealso marker="diameter#service_opt">diameter:service_opt()</seealso> <seealso marker="diameter#string_decode">string_decode</seealso>'> + <!-- diameter_app --> <!ENTITY app_handle_answer '<seealso marker="diameter_app#Mod:handle_answer-4">handle_answer/4</seealso>'> @@ -102,6 +105,9 @@ significant. <!ENTITY dict_Address '<seealso marker="diameter_dict#DATA_TYPES">Address()</seealso>'> <!ENTITY dict_DiameterIdentity '<seealso marker="diameter_dict#DATA_TYPES">DiameterIdentity()</seealso>'> +<!ENTITY dict_DiameterURI '<seealso marker="diameter_dict#DATA_TYPES">DiameterURI()</seealso>'> +<!ENTITY dict_IPFilterRule '<seealso marker="diameter_dict#DATA_TYPES">IPFilterRule()</seealso>'> +<!ENTITY dict_QoSFilterRule '<seealso marker="diameter_dict#DATA_TYPES">QoSFilterRule()</seealso>'> <!ENTITY dict_Grouped '<seealso marker="diameter_dict#DATA_TYPES">Grouped()</seealso>'> <!ENTITY dict_OctetString '<seealso marker="diameter_dict#DATA_TYPES">OctetString()</seealso>'> <!ENTITY dict_Time '<seealso marker="diameter_dict#DATA_TYPES">Time()</seealso>'> diff --git a/lib/diameter/examples/code/client.erl b/lib/diameter/examples/code/client.erl index be5b4cbba5..844c9cdbdd 100644 --- a/lib/diameter/examples/code/client.erl +++ b/lib/diameter/examples/code/client.erl @@ -41,6 +41,7 @@ -include_lib("diameter/include/diameter_gen_base_rfc6733.hrl"). -export([start/1, %% start a service + start/2, %% connect/2, %% add a connecting transport call/1, %% send using the record encoding cast/1, %% send using the list encoding and detached @@ -68,6 +69,7 @@ {'Vendor-Id', 0}, {'Product-Name', "Client"}, {'Auth-Application-Id', [0]}, + {string_decode, false}, {application, [{alias, common}, {dictionary, diameter_gen_base_rfc6733}, {module, client_cb}]}]). @@ -76,11 +78,23 @@ start(Name) when is_atom(Name) -> - node:start(Name, ?SERVICE(Name)). + start(Name, []); + +start(Opts) + when is_list(Opts) -> + start(?DEF_SVC_NAME, Opts). + +%% start/0 start() -> start(?DEF_SVC_NAME). +%% start/2 + +start(Name, Opts) -> + node:start(Name, Opts ++ [T || {K,_} = T <- ?SERVICE(Name), + false == lists:keymember(K, 1, Opts)]). + %% connect/2 connect(Name, T) -> diff --git a/lib/diameter/examples/code/relay.erl b/lib/diameter/examples/code/relay.erl index 0aa3cd06d3..7bc46dc68d 100644 --- a/lib/diameter/examples/code/relay.erl +++ b/lib/diameter/examples/code/relay.erl @@ -32,6 +32,7 @@ -module(relay). -export([start/1, + start/2, listen/2, connect/2, stop/1]). @@ -49,6 +50,7 @@ {'Vendor-Id', 193}, {'Product-Name', "RelayAgent"}, {'Auth-Application-Id', [16#FFFFFFFF]}, + {string_decode, false}, {application, [{alias, relay}, {dictionary, diameter_relay}, {module, relay_cb}]}]). @@ -57,11 +59,19 @@ start(Name) when is_atom(Name) -> - node:start(Name, ?SERVICE(Name)). + start(Name, []). + +%% start/1 start() -> start(?DEF_SVC_NAME). +%% start/2 + +start(Name, Opts) -> + node:start(Name, Opts ++ [T || {K,_} = T <- ?SERVICE(Name), + false == lists:keymember(K, 1, Opts)]). + %% listen/2 listen(Name, T) -> diff --git a/lib/diameter/examples/code/server.erl b/lib/diameter/examples/code/server.erl index 8c91e68895..f32cec594c 100644 --- a/lib/diameter/examples/code/server.erl +++ b/lib/diameter/examples/code/server.erl @@ -35,6 +35,7 @@ -module(server). -export([start/1, %% start a service + start/2, %% listen/2, %% add a listening transport stop/1]). %% stop a service @@ -53,6 +54,8 @@ {'Vendor-Id', 193}, {'Product-Name', "Server"}, {'Auth-Application-Id', [0]}, + {restrict_connections, false}, + {string_decode, false}, {application, [{alias, common}, {dictionary, diameter_gen_base_rfc6733}, {module, server_cb}]}]). @@ -61,11 +64,23 @@ start(Name) when is_atom(Name) -> - node:start(Name, ?SERVICE(Name)). + start(Name, []); + +start(Opts) + when is_list(Opts) -> + start(?DEF_SVC_NAME, Opts). + +%% start/0 start() -> start(?DEF_SVC_NAME). +%% start/2 + +start(Name, Opts) -> + node:start(Name, Opts ++ [T || {K,_} = T <- ?SERVICE(Name), + false == lists:keymember(K, 1, Opts)]). + %% listen/2 listen(Name, T) -> diff --git a/lib/diameter/src/base/diameter.erl b/lib/diameter/src/base/diameter.erl index 1bbdf6e34d..67dfc7bdbf 100644 --- a/lib/diameter/src/base/diameter.erl +++ b/lib/diameter/src/base/diameter.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2014. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -306,6 +306,7 @@ call(SvcName, App, Message) -> | {restrict_connections, restriction()} | {sequence, sequence() | evaluable()} | {share_peers, remotes()} + | {string_decode, boolean()} | {use_shared_peers, remotes()} | {spawn_opt, list()}. @@ -343,6 +344,8 @@ call(SvcName, App, Message) -> | {capabilities_cb, evaluable()} | {capx_timeout, 'Unsigned32'()} | {disconnect_cb, evaluable()} + | {dpr_timeout, 'Unsigned32'()} + | {dpa_timeout, 'Unsigned32'()} | {length_errors, exit | handle | discard} | {connect_timer, 'Unsigned32'()} | {watchdog_timer, 'Unsigned32'() | {module(), atom(), list()}} diff --git a/lib/diameter/src/base/diameter_capx.erl b/lib/diameter/src/base/diameter_capx.erl index 93548ecafd..7dc61f229f 100644 --- a/lib/diameter/src/base/diameter_capx.erl +++ b/lib/diameter/src/base/diameter_capx.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2013. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -50,7 +50,8 @@ -export([build_CER/2, recv_CER/3, recv_CEA/3, - make_caps/2]). + make_caps/2, + binary_caps/1]). -include_lib("diameter/include/diameter.hrl"). -include("diameter_internal.hrl"). @@ -115,7 +116,8 @@ mk_caps(Caps0, Opts) -> -define(SC(K,F), set_cap({K, Val}, {Caps, #diameter_caps{F = false} = C}) -> - {Caps#diameter_caps{F = cap(K, Val)}, C#diameter_caps{F = true}}). + {Caps#diameter_caps{F = cap(K, copy(Val))}, + C#diameter_caps{F = true}}). ?SC('Origin-Host', origin_host); ?SC('Origin-Realm', origin_realm); @@ -375,10 +377,10 @@ capx_to_caps(CEX, Dict) -> 'Firmware-Revision', 'AVP'], CEX), - #diameter_caps{origin_host = OH, - origin_realm = OR, + #diameter_caps{origin_host = copy(OH), + origin_realm = copy(OR), vendor_id = VId, - product_name = PN, + product_name = copy(PN), origin_state_id = OSI, host_ip_address = IP, supported_vendor_id = SV, @@ -389,6 +391,32 @@ capx_to_caps(CEX, Dict) -> firmware_revision = FR, avp = X}. +%% Copy binaries to avoid retaining a reference to a large binary +%% containing AVPs we aren't interested in. +copy(B) + when is_binary(B) -> + binary:copy(B); + +copy(T) -> + T. + +%% binary_caps/1 +%% +%% Encode stringish capabilities with {string_decode, false}. + +binary_caps(Caps) -> + lists:foldl(fun bcaps/2, Caps, [#diameter_caps.origin_host, + #diameter_caps.origin_realm, + #diameter_caps.product_name]). + +bcaps(N, Caps) -> + case element(N, Caps) of + undefined -> + Caps; + V -> + setelement(N, Caps, iolist_to_binary(V)) + end. + %% --------------------------------------------------------------------------- %% --------------------------------------------------------------------------- diff --git a/lib/diameter/src/base/diameter_codec.erl b/lib/diameter/src/base/diameter_codec.erl index b4ecb63961..15a4c5e86f 100644 --- a/lib/diameter/src/base/diameter_codec.erl +++ b/lib/diameter/src/base/diameter_codec.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2014. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -22,6 +22,8 @@ -export([encode/2, decode/2, decode/3, + setopts/1, + getopt/1, collect_avps/1, decode_header/1, sequence_numbers/1, @@ -59,6 +61,50 @@ %% +-+-+-+-+-+-+-+-+-+-+-+-+- %%% --------------------------------------------------------------------------- +%%% # setopts/1 +%%% # getopt/1 +%%% --------------------------------------------------------------------------- + +%% These functions are a compromise in the same vein as the use of the +%% process dictionary in diameter_gen.hrl in generated codec modules. +%% Instead of rewriting the entire dictionary generation to pass +%% encode/decode options around, the calling process sets them by +%% calling setopts/1. At current, the only option is whether or not to +%% decode binaries as strings, which is used by diameter_types. + +setopts(Opts) + when is_list(Opts) -> + lists:foreach(fun setopt/1, Opts). + +%% Decode stringish types to string()? The default true is for +%% backwards compatibility. +setopt({string_decode = K, false = B}) -> + setopt(K, B); + +%% Regard anything but the generated RFC 3588 dictionary as modern. +%% This affects the interpretation of defaults during the decode +%% of values of type DiameterURI, this having changed from RFC 3588. +%% (So much for backwards compatibility.) +setopt({common_dictionary, diameter_gen_base_rfc3588}) -> + setopt(rfc, 3588); + +setopt(_) -> + ok. + +setopt(Key, Value) -> + put({diameter, Key}, Value). + +getopt(Key) -> + case get({diameter, Key}) of + undefined when Key == string_decode -> + true; + undefined when Key == rfc -> + 6733; + V -> + V + end. + +%%% --------------------------------------------------------------------------- %%% # encode/2 %%% --------------------------------------------------------------------------- @@ -90,7 +136,7 @@ encode(Mod, Msg) -> msg = Msg}). e(_, #diameter_packet{msg = [#diameter_header{} = Hdr | As]} = Pkt) -> - try encode_avps(As) of + try encode_avps(reorder(As)) of Avps -> Length = size(Avps) + 20, @@ -183,26 +229,50 @@ values(Avps) -> %% Message as a list of #diameter_avp{} ... encode_avps(_, _, [#diameter_avp{} | _] = Avps) -> - encode_avps(reorder(Avps, [], Avps)); + encode_avps(reorder(Avps)); %% ... or as a tuple list or record. encode_avps(Mod, MsgName, Values) -> Mod:encode_avps(MsgName, Values). %% reorder/1 +%% +%% Reorder AVPs for the relay case using the index field of +%% diameter_avp records. Decode populates this field in collect_avps +%% and presents AVPs in reverse order. A relay then sends the reversed +%% list with a Route-Record AVP prepended. The goal here is just to do +%% lists:reverse/1 in Grouped AVPs and the outer list, but only in the +%% case there are indexed AVPs at all, so as not to reverse lists that +%% have been explicilty sent (unindexed, in the desired order) as a +%% diameter_avp list. The effect is the same as lists:keysort/2, but +%% only on the cases we expect, not a general sort. + +reorder(Avps) -> + case reorder(Avps, []) of + false -> + Avps; + Sorted -> + Sorted + end. -reorder([#diameter_avp{index = 0} | _] = Avps, Acc, _) -> +%% reorder/3 + +%% In case someone has reversed the list already. (Not likely.) +reorder([#diameter_avp{index = 0} | _] = Avps, Acc) -> Avps ++ Acc; -reorder([#diameter_avp{index = N} = A | Avps], Acc, _) +%% Assume indexed AVPs are in reverse order. +reorder([#diameter_avp{index = N} = A | Avps], Acc) when is_integer(N) -> lists:reverse(Avps, [A | Acc]); -reorder([H | T], Acc, Avps) -> - reorder(T, [H | Acc], Avps); +%% An unindexed AVP. +reorder([H | T], Acc) -> + reorder(T, [H | Acc]); -reorder([], Acc, _) -> - Acc. +%% No indexed members. +reorder([], _) -> + false. %% encode_avps/1 diff --git a/lib/diameter/src/base/diameter_config.erl b/lib/diameter/src/base/diameter_config.erl index c0a4f7df69..0d0304bf33 100644 --- a/lib/diameter/src/base/diameter_config.erl +++ b/lib/diameter/src/base/diameter_config.erl @@ -532,7 +532,10 @@ opt({applications, As}) -> opt({capabilities, Os}) -> is_list(Os) andalso ok == encode_CER(Os); -opt({capx_timeout, Tmo}) -> +opt({K, Tmo}) + when K == capx_timeout; + K == dpr_timeout; + K == dpa_timeout -> ?IS_UINT32(Tmo); opt({length_errors, T}) -> @@ -642,13 +645,23 @@ make_config(SvcName, Opts) -> {false, monitor}, {?NOMASK, sequence}, {nodes, restrict_connections}, + {true, string_decode}, {[], spawn_opt}]), + D = proplists:get_value(string_decode, SvcOpts, true), + #service{name = SvcName, rec = #diameter_service{applications = Apps, - capabilities = Caps}, + capabilities = binary_caps(Caps, D)}, options = SvcOpts}. +binary_caps(Caps, true) -> + Caps; +binary_caps(Caps, false) -> + diameter_capx:binary_caps(Caps). + +%% make_opts/2 + make_opts(Opts, Defs) -> Known = [{K, get_opt(K, Opts, D)} || {D,K} <- Defs], Unknown = Opts -- Known, @@ -667,7 +680,8 @@ opt(K, false = B) opt(K, true = B) when K == share_peers; - K == use_shared_peers -> + K == use_shared_peers; + K == string_decode -> B; opt(restrict_connections, T) diff --git a/lib/diameter/src/base/diameter_lib.erl b/lib/diameter/src/base/diameter_lib.erl index d0d730f47c..51d203d722 100644 --- a/lib/diameter/src/base/diameter_lib.erl +++ b/lib/diameter/src/base/diameter_lib.erl @@ -19,6 +19,7 @@ -module(diameter_lib). -compile({no_auto_import, [now/0]}). +-compile({nowarn_deprecated_function, [{erlang, now, 0}]}). -export([info_report/2, error_report/2, @@ -129,12 +130,13 @@ timestamp({_,_,_} = T) -> %% erlang:now() T; timestamp(MonoT) -> %% monotonic time - MicroSecs = erlang:convert_time_resolution(MonoT + erlang:time_offset(), - erlang:time_resolution(), - 1000000), + MicroSecs = monotonic_to_microseconds(MonoT + erlang:time_offset()), Secs = MicroSecs div 1000000, {Secs div 1000000, Secs rem 1000000, MicroSecs rem 1000000}. +monotonic_to_microseconds(MonoT) -> + erlang:convert_time_unit(MonoT, native, micro_seconds). + %% --------------------------------------------------------------------------- %% # now_diff/1 %% --------------------------------------------------------------------------- @@ -164,9 +166,7 @@ micro_diff({_,_,_} = T0) -> timer:now_diff(erlang:now(), T0); micro_diff(T0) -> %% monotonic time - erlang:convert_time_resolution(erlang:monotonic_time() - T0, - erlang:time_resolution(), - 1000000). + monotonic_to_microseconds(erlang:monotonic_time() - T0). %% --------------------------------------------------------------------------- %% # micro_diff/2 @@ -178,9 +178,7 @@ micro_diff(T0) -> %% monotonic time micro_diff(T1, T0) when is_integer(T1), is_integer(T0) -> %% monotonic time - erlang:convert_time_resolution(T1 - T0, - erlang:time_resolution(), - 1000000); + monotonic_to_microseconds(T1 - T0); micro_diff(T1, T0) -> %% at least one erlang:now() timer:now_diff(timestamp(T1), timestamp(T0)). diff --git a/lib/diameter/src/base/diameter_peer.erl b/lib/diameter/src/base/diameter_peer.erl index ea326dd03e..89b63c8a92 100644 --- a/lib/diameter/src/base/diameter_peer.erl +++ b/lib/diameter/src/base/diameter_peer.erl @@ -232,12 +232,22 @@ recv(Pid, Pkt) -> %% # send/2 %% --------------------------------------------------------------------------- -send(Pid, #diameter_packet{transport_data = undefined, - bin = Bin}) -> - send(Pid, Bin); +send(Pid, Msg) -> + ifc_send(Pid, {send, strip(Msg)}). -send(Pid, Pkt) -> - ifc_send(Pid, {send, Pkt}). +%% Send only binary when possible. +strip(#diameter_packet{transport_data = undefined, + bin = Bin}) -> + Bin; + +%% Strip potentially large message terms. +strip(#diameter_packet{transport_data = T, + bin = Bin}) -> + #diameter_packet{transport_data = T, + bin = Bin}; + +strip(Msg) -> + Msg. %% --------------------------------------------------------------------------- %% # close/1 @@ -326,7 +336,6 @@ code_change(_OldVsn, State, _Extra) -> {ok, State}. %% --------------------------------------------------------- -%% INTERNAL FUNCTIONS %% --------------------------------------------------------- %% ifc_send/2 diff --git a/lib/diameter/src/base/diameter_peer_fsm.erl b/lib/diameter/src/base/diameter_peer_fsm.erl index ee6e7dd89e..aac2685514 100644 --- a/lib/diameter/src/base/diameter_peer_fsm.erl +++ b/lib/diameter/src/base/diameter_peer_fsm.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2014. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -63,6 +63,8 @@ %% Keys in process dictionary. -define(CB_KEY, cb). %% capabilities callback -define(DPR_KEY, dpr). %% disconnect callback +-define(DPA_KEY, dpa). %% timeout for incoming DPA, or shutdown after + %% outgoing DPA -define(REF_KEY, ref). %% transport_ref() -define(Q_KEY, q). %% transport start queue -define(START_KEY, start). %% start of connected transport @@ -82,18 +84,26 @@ N == ?GOAWAY; N == goaway; N == ?BUSY; N == busy). -%% RFC 3588: +%% RFC 6733: %% %% Timeout An application-defined timer has expired while waiting %% for some event. %% --define(EVENT_TIMEOUT, 10000). + %% Default timeout for reception of CER/CEA. +-define(CAPX_TIMEOUT, 10000). -%% Default timeout for DPA in response to DPR. A bit short but the -%% timeout used to be hardcoded. (So it could be worse.) +%% Default timeout for DPA to be received in response to an outgoing +%% DPR. A bit short but the timeout used to be hardcoded. (So it could +%% be worse.) -define(DPA_TIMEOUT, 1000). +%% Default timeout for the connection to be closed by the peer +%% following an outgoing DPA in response to an incoming DPR. It's the +%% recipient of DPA that should close the connection according to the +%% RFC. +-define(DPR_TIMEOUT, 5000). + -type uint32() :: diameter:'Unsigned32'(). -record(state, @@ -107,8 +117,14 @@ transport :: pid(), %% transport process dictionary :: module(), %% common dictionary service :: #diameter_service{}, - dpr = false :: false | {uint32(), uint32()}, - %% | hop by hop and end to end identifiers + dpr = false :: false + | true %% DPR received, DPA sent + | {uint32(), uint32()} %% set in old code + | {boolean(), uint32(), uint32()}, + %% hop by hop and end to end identifiers in + %% outgoing DPR; boolean says whether or not + %% the request was sent explicitly with + %% diameter:call/4. length_errors :: exit | handle | discard}). %% There are non-3588 states possible as a consequence of 5.6.1 of the @@ -138,7 +154,8 @@ %% # start/3 %% --------------------------------------------------------------------------- --spec start(T, [Opt], {diameter:sequence(), +-spec start(T, [Opt], {[diameter:service_opt()] + | diameter:sequence(), %% from old code [node()], module(), #diameter_service{}}) @@ -177,18 +194,25 @@ init(T) -> proc_lib:init_ack({ok, self()}), gen_server:enter_loop(?MODULE, [], i(T)). -i({Ack, WPid, {M, Ref} = T, Opts, {Mask, Nodes, Dict0, Svc}}) -> +i({Ack, WPid, T, Opts, {{_,_} = Mask, Nodes, Dict0, Svc}}) -> %% from old code + i({Ack, WPid, T, Opts, {[{sequence, Mask}], Nodes, Dict0, Svc}}); + +i({Ack, WPid, {M, Ref} = T, Opts, {SvcOpts, Nodes, Dict0, Svc}}) -> erlang:monitor(process, WPid), wait(Ack, WPid), diameter_stats:reg(Ref), + diameter_codec:setopts([{common_dictionary, Dict0} | SvcOpts]), + {_,_} = Mask = proplists:get_value(sequence, SvcOpts), {[Cs,Ds], Rest} = proplists:split(Opts, [capabilities_cb, disconnect_cb]), putr(?CB_KEY, {Ref, [F || {_,F} <- Cs]}), putr(?DPR_KEY, [F || {_, F} <- Ds]), putr(?REF_KEY, Ref), putr(?SEQUENCE_KEY, Mask), putr(?RESTRICT_KEY, Nodes), + putr(?DPA_KEY, {proplists:get_value(dpr_timeout, Opts, ?DPR_TIMEOUT), + proplists:get_value(dpa_timeout, Opts, ?DPA_TIMEOUT)}), - Tmo = proplists:get_value(capx_timeout, Opts, ?EVENT_TIMEOUT), + Tmo = proplists:get_value(capx_timeout, Opts, ?CAPX_TIMEOUT), OnLengthErr = proplists:get_value(length_errors, Opts, exit), {TPid, Addrs} = start_transport(T, Rest, Svc), @@ -212,9 +236,12 @@ wait(Ref, Pid) -> Ref -> ok; {'DOWN', _, process, Pid, _} = D -> - exit({shutdown, D}) + x(D) end. +x(T) -> + exit({shutdown, T}). + start_transport(T, Opts, #diameter_service{capabilities = LCaps} = Svc) -> Addrs0 = LCaps#diameter_caps.host_ip_address, start_transport(Addrs0, {T, Opts, Svc}). @@ -226,7 +253,7 @@ start_transport(Addrs0, T) -> q_next(TPid, Addrs0, Tmo, Data), {TPid, Addrs}; {error, No} -> - exit({shutdown, {no_connection, No}}) + x({no_connection, No}) end. svc(#diameter_service{capabilities = LCaps0} = Svc, Addrs) -> @@ -333,6 +360,9 @@ eraser(Key) -> %% transition/2 +transition(T, #state{dpr = {Hid, Eid}} = S) -> %% DPR sent from old code + transition(T, S#state{dpr = {false, Hid, Eid}}); + %% Connection to peer. transition({diameter, {TPid, connected, Remote}}, #state{transport = TPid, @@ -397,9 +427,8 @@ transition({timeout, _}, _) -> ok; %% Outgoing message. -transition({send, Msg}, #state{transport = TPid}) -> - send(TPid, Msg), - ok; +transition({send, Msg}, S) -> + outgoing(Msg, S); %% Request for graceful shutdown at remove_transport, stop_service of %% application shutdown. @@ -408,7 +437,8 @@ transition({shutdown, Pid, Reason}, #state{parent = Pid, dpr = false} = S) -> transition({shutdown, Pid, _}, #state{parent = Pid}) -> ok; -%% DPA reception has timed out. +%% DPA reception has timed out, or peer has not closed the connection +%% as a result of outgoing DPA. transition(dpa_timeout, _) -> stop; @@ -516,12 +546,9 @@ encode(Rec, Dict) -> recv(#diameter_packet{header = #diameter_header{} = Hdr} = Pkt, - #state{parent = Pid, - dictionary = Dict0} + #state{dictionary = Dict0} = S) -> - Name = diameter_codec:msg_name(Dict0, Hdr), - Pid ! {recv, self(), Name, Pkt}, - rcv(Name, Pkt, S); + recv1(diameter_codec:msg_name(Dict0, Hdr), Pkt, S); recv(#diameter_packet{header = undefined, bin = Bin} @@ -532,6 +559,41 @@ recv(#diameter_packet{header = undefined, recv(Bin, S) -> recv(#diameter_packet{bin = Bin}, S). +%% recv1/3 + +%% Incoming request after outgoing DPR: discard. Don't discard DPR, so +%% both ends don't do so when sending simultaneously. +recv1(Name, + #diameter_packet{header = #diameter_header{is_request = true} = H}, + #state{dpr = {_,_,_}}) + when Name /= 'DPR' -> + invalid(false, recv_after_outgoing_dpr, H); + +%% Incoming request after incoming DPR: discard. +recv1(_, + #diameter_packet{header = #diameter_header{is_request = true} = H}, + #state{dpr = true}) -> + invalid(false, recv_after_incoming_dpr, H); + +%% DPA with identifier mismatch, or in response to a DPR initiated by +%% the service. +recv1('DPA' = N, + #diameter_packet{header = #diameter_header{hop_by_hop_id = Hid, + end_to_end_id = Eid}} + = Pkt, + #state{dpr = {X,H,E}} + = S) + when H /= Hid; + E /= Eid; + not X -> + rcv(N, Pkt, S); + +%% Any other message with a header and no length errors: send to the +%% parent. +recv1(Name, Pkt, #state{parent = Pid} = S) -> + Pid ! {recv, self(), Name, Pkt}, + rcv(Name, Pkt, S). + %% recv/3 recv(#diameter_header{length = Len} @@ -588,7 +650,7 @@ rcv(Name, _, #state{state = PS}) rcv('DPR' = N, Pkt, S) -> handle_request(N, Pkt, S); -%% DPA in response to DPR and with the expected identifiers. +%% DPA in response to DPR, with the expected identifiers. rcv('DPA' = N, #diameter_packet{header = #diameter_header{end_to_end_id = Eid, hop_by_hop_id = Hid} @@ -596,14 +658,21 @@ rcv('DPA' = N, = Pkt, #state{dictionary = Dict0, transport = TPid, - dpr = {Hid, Eid}}) -> + dpr = {X, Hid, Eid}}) -> ?LOG(recv, N), - incr(recv, H, Dict0), - incr_rc(recv, diameter_codec:decode(Dict0, Pkt), Dict0), + X orelse begin + %% Only count DPA in response to a DPR sent by the + %% service: explicit DPR is counted in the same way + %% as other explicitly sent requests. + incr(recv, H, Dict0), + incr_rc(recv, diameter_codec:decode(Dict0, Pkt), Dict0) + end, diameter_peer:close(TPid), {stop, N}; -%% Ignore anything else, an unsolicited DPA in particular. +%% Ignore anything else, an unsolicited DPA in particular. Note that +%% dpa_timeout deals with the case in which the peer sends the wrong +%% identifiers in DPA. rcv(N, #diameter_packet{header = H}, _) when N == 'CER'; N == 'CEA'; @@ -637,9 +706,61 @@ incr_error(Dir, Pkt, Dict0) -> %% Msg here could be a #diameter_packet or a binary depending on who's %% sending. In particular, the watchdog will send DWR as a binary %% while messages coming from clients will be in a #diameter_packet. + send(Pid, Msg) -> diameter_peer:send(Pid, Msg). +%% outgoing/2 + +%% Explicit DPR. +outgoing(#diameter_packet{header = #diameter_header{application_id = 0, + cmd_code = 282, + is_request = true} + = H} + = Pkt, + #state{dpr = T, + parent = Pid} + = S) -> + if T == false -> + inform_dpr(Pid), + send_dpr(true, Pkt, dpa_timeout(), S); + T == true -> + invalid(false, dpr_after_dpa, H); %% DPA sent: discard + true -> + invalid(false, dpr_after_dpr, H) %% DPR sent: discard + end; + +%% Explict CER or DWR: discard. These are sent by us. +outgoing(#diameter_packet{header = #diameter_header{application_id = 0, + cmd_code = C, + is_request = true} + = H}, + _) + when 257 == C; %% CER + 280 == C -> %% DWR + invalid(false, invalid_request, H); + +%% DPR not sent: send. +outgoing(Msg, #state{transport = TPid, dpr = false}) -> + send(TPid, Msg), + ok; + +%% Outgoing answer: send. +outgoing(#diameter_packet{header = #diameter_header{is_request = false}} + = Pkt, + #state{transport = TPid}) -> + send(TPid, Pkt), + ok; + +%% Outgoing request: discard. +outgoing(Msg, #state{dpr = {_,_,_}}) -> + invalid(false, send_after_dpr, header(Msg)). + +header(#diameter_packet{header = H}) -> + H; +header(Bin) -> %% DWR + diameter_codec:decode_header(Bin). + %% handle_request/3 %% %% Incoming CER or DPR. @@ -699,6 +820,8 @@ build_answer('CER', = Pkt, #state{dictionary = Dict0} = S) -> + diameter_codec:setopts([{string_decode, false}]), + {SupportedApps, RCaps, CEA} = recv_CER(CER, S), [RC, IS] = Dict0:'#get-'(['Result-Code', 'Inband-Security-Id'], CEA), @@ -731,7 +854,7 @@ build_answer(Type, errors = Es} = Pkt, S) -> - {RC, FailedAVP} = result_code(H, Es), + {RC, FailedAVP} = result_code(Type, H, Es), {answer(Type, RC, FailedAVP, S), post(Type, RC, Pkt, S)}. inband_security([]) -> @@ -748,8 +871,16 @@ cea(CEA, RC, Dict0) -> post('CER' = T, RC, Pkt, S) -> {T, caps(S), {RC, Pkt}}; -post('DPR' = T, _, _, #state{parent = Pid}) -> - [fun(S) -> Pid ! {T, self()}, S end]. +post('DPR', _, _, #state{parent = Pid}) -> + [fun(S) -> dpr_timer(), inform_dpr(Pid), dpr(S) end]. + +dpr(#state{dpr = false} = S) -> %% not awaiting DPA + S#state{dpr = true}; %% DPR received +dpr(S) -> %% DPR already sent or received + S. + +inform_dpr(Pid) -> + Pid ! {'DPR', self()}. %% tell watchdog to die with us rejected({capabilities_cb, _F, Reason}, T, S) -> rejected(Reason, T, S); @@ -798,6 +929,19 @@ set(['answer-message' | _] = Ans, FailedAvp) -> set([_|_] = Ans, FailedAvp) -> Ans ++ FailedAvp. +%% result_code/3 + +%% Be lenient with errors in DPR since there's no reason to be +%% otherwise. Rejecting may cause the peer to missinterpret the error +%% as meaning that the connection should not be closed, which may well +%% lead to more problems than any errors in the DPR. + +result_code('DPR', _, _) -> + {2001, []}; + +result_code('CER', H, Es) -> + result_code(H, Es). + %% result_code/2 result_code(#diameter_header{is_error = true}, _) -> @@ -886,6 +1030,8 @@ handle_CEA(#diameter_packet{header = H} = DPkt = diameter_codec:decode(Dict0, Pkt), + diameter_codec:setopts([{string_decode, false}]), + RC = result_code(incr_rc(recv, DPkt, Dict0)), {SApps, IS, RCaps} = recv_CEA(DPkt, S), @@ -1026,7 +1172,7 @@ close(Reason) -> %% dpr/2 %% -%% The RFC isn't clear on whether DPR should be send in a non-Open +%% The RFC isn't clear on whether DPR should be sent in a non-Open %% state. The Peer State Machine transitions it documents aren't %% exhaustive (no Stop in Wait-I-CEA for example) so assume it's up to %% the implementation and transition to Closed (ie. die) if we haven't @@ -1042,7 +1188,7 @@ dpr(Reason, #state{state = 'Open', Peer = {self(), Caps}, dpr(CBs, [Reason, Ref, Peer], S); -%% Connection is open, DPR already sent. +%% Connection is open, DPR already sent or received. dpr(_, #state{state = 'Open'}) -> ok; @@ -1073,10 +1219,9 @@ dpr([CB|Rest], [Reason | _] = Args, S) -> dpr([], [Reason | _], S) -> send_dpr(Reason, [], S). --record(opts, {cause, timeout = ?DPA_TIMEOUT}). +-record(opts, {cause, timeout}). -send_dpr(Reason, Opts, #state{transport = TPid, - dictionary = Dict, +send_dpr(Reason, Opts, #state{dictionary = Dict, service = #diameter_service{capabilities = Caps}} = S) -> #opts{cause = Cause, timeout = Tmo} @@ -1085,24 +1230,37 @@ send_dpr(Reason, Opts, #state{transport = TPid, transport -> ?GOAWAY; _ -> ?REBOOT end, - timeout = ?DPA_TIMEOUT}, + timeout = dpa_timeout()}, Opts), #diameter_caps{origin_host = {OH, _}, origin_realm = {OR, _}} = Caps, - #diameter_packet{header = #diameter_header{end_to_end_id = Eid, - hop_by_hop_id = Hid}} - = Pkt - = encode(['DPR', {'Origin-Host', OH}, + Pkt = encode(['DPR', {'Origin-Host', OH}, {'Origin-Realm', OR}, {'Disconnect-Cause', Cause}], Dict), - incr(send, Pkt, Dict), + send_dpr(false, Pkt, Tmo, S). + +%% send_dpr/4 + +send_dpr(X, + #diameter_packet{header = #diameter_header{end_to_end_id = Eid, + hop_by_hop_id = Hid}} + = Pkt, + Tmo, + #state{transport = TPid, + dictionary = Dict} + = S) -> + %% Only count DPR sent by the service: explicit DPR is counted in + %% the same way as other explicitly sent requests. + X orelse incr(send, Pkt, Dict), send(TPid, Pkt), dpa_timer(Tmo), ?LOG(send, 'DPR'), - S#state{dpr = {Hid, Eid}}. + S#state{dpr = {X, Hid, Eid}}. + +%% opt/2 opt({timeout, Tmo}, Rec) when ?IS_TIMEOUT(Tmo) -> @@ -1125,6 +1283,27 @@ cause(N) -> dpa_timer(Tmo) -> erlang:send_after(Tmo, self(), dpa_timeout). +dpa_timeout() -> + dpa_timeout(getr(?DPA_KEY)). + +dpa_timeout({_, Tmo}) -> + Tmo; +dpa_timeout(undefined) -> %% set in old code + ?DPA_TIMEOUT; +dpa_timeout(Tmo) -> %% ditto + Tmo. + +dpr_timer() -> + dpa_timer(dpr_timeout()). + +dpr_timeout() -> + dpr_timeout(getr(?DPA_KEY)). + +dpr_timeout({Tmo, _}) -> + Tmo; +dpr_timeout(_) -> %% set in old code + ?DPR_TIMEOUT. + %% register_everywhere/1 %% %% Register a term and ensure it's not registered elsewhere. Note that diff --git a/lib/diameter/src/base/diameter_service.erl b/lib/diameter/src/base/diameter_service.erl index 04401a3d87..a01bcdd4e7 100644 --- a/lib/diameter/src/base/diameter_service.erl +++ b/lib/diameter/src/base/diameter_service.erl @@ -130,7 +130,8 @@ :: [{sequence, diameter:sequence()} %% sequence mask | {share_peers, diameter:remotes()} %% broadcast to | {use_shared_peers, diameter:remotes()} %% use from - | {restrict_connections, diameter:restriction()}]}). + | {restrict_connections, diameter:restriction()} + | {string_decode, boolean()}]}). %% shared_peers reflects the peers broadcast from remote nodes. %% Record representing an RFC 3539 watchdog process implemented by @@ -261,16 +262,22 @@ whois(SvcName) -> %% --------------------------------------------------------------------------- -spec pick_peer(SvcName, AppOrAlias, Opts) - -> {{TPid, Caps, App}, Mask} - | false - | {error, term()} + -> {{TPid, Caps, App}, Mask, SvcOpts} + | false %% no selection + | {error, no_service} when SvcName :: diameter:service_name(), - AppOrAlias :: {alias, diameter:app_alias()} | #diameter_app{}, - Opts :: tuple(), + AppOrAlias :: #diameter_app{} + | {alias, diameter:app_alias()}, + Opts :: {fun((Dict :: module()) -> [term()]), + diameter:peer_filter(), + Xtra :: list()}, TPid :: pid(), Caps :: #diameter_caps{}, App :: #diameter_app{}, - Mask :: diameter:sequence(). + Mask :: diameter:sequence(), + SvcOpts :: [diameter:service_opt()]. +%% Extract Mask in the returned tuple so that diameter_traffic doesn't +%% need to know about the ordering of SvcOpts used here. pick_peer(SvcName, App, Opts) -> pick(lookup_state(SvcName), App, Opts). @@ -287,10 +294,10 @@ pick(#state{service = #diameter_service{applications = Apps}} Opts) -> %% initial call from diameter:call/4 pick(S, find_outgoing_app(Alias, Apps), Opts); -pick(_, false, _) -> - false; +pick(_, false = No, _) -> + No; -pick(#state{options = [{_, Mask} | _]} +pick(#state{options = [{_, Mask} | SvcOpts]} = S, #diameter_app{module = ModX, dictionary = Dict} = App0, @@ -299,7 +306,7 @@ pick(#state{options = [{_, Mask} | _]} [_,_] = RealmAndHost = diameter_lib:eval([DestF, Dict]), case pick_peer(App, RealmAndHost, Filter, S) of {TPid, Caps} -> - {{TPid, Caps, App}, Mask}; + {{TPid, Caps, App}, Mask, SvcOpts}; false = No -> No end. @@ -690,7 +697,8 @@ service_options(Opts) -> {restrict_connections, proplists:get_value(restrict_connections, Opts, ?RESTRICT)}, - {spawn_opt, proplists:get_value(spawn_opt, Opts, [])}]. + {spawn_opt, proplists:get_value(spawn_opt, Opts, [])}, + {string_decode, proplists:get_value(string_decode, Opts, true)}]. %% The order of options is significant since we match against the list. mref(false = No) -> @@ -802,10 +810,13 @@ start(Ref, Type, Opts, N, #state{watchdogT = WatchdogT, when Type == connect; Type == accept -> #diameter_service{applications = Apps} - = Svc + = Svc1 = merge_service(Opts, Svc0), - {_,_} = Mask = proplists:get_value(sequence, SvcOpts), - RecvData = diameter_traffic:make_recvdata([SvcName, PeerT, Apps, Mask]), + Svc = binary_caps(Svc1, proplists:get_value(string_decode, SvcOpts, true)), + RecvData = diameter_traffic:make_recvdata([SvcName, + PeerT, + Apps, + SvcOpts]), T = {{spawn_opts([Opts, SvcOpts]), RecvData}, Opts, SvcOpts, Svc}, Rec = #watchdog{type = Type, ref = Ref, @@ -816,8 +827,13 @@ start(Ref, Type, Opts, N, #state{watchdogT = WatchdogT, [], N). +binary_caps(Svc, true) -> + Svc; +binary_caps(#diameter_service{capabilities = Caps} = Svc, false) -> + Svc#diameter_service{capabilities = diameter_capx:binary_caps(Caps)}. + wd(Type, Ref, T, WatchdogT, Rec) -> - Pid = wd(Type, Ref, T), + Pid = start_watchdog(Type, Ref, T), insert(WatchdogT, Rec#watchdog{pid = Pid}), Pid. @@ -831,7 +847,7 @@ spawn_opts(Optss) -> T /= link, T /= monitor]. -wd(Type, Ref, T) -> +start_watchdog(Type, Ref, T) -> {_MRef, Pid} = diameter_watchdog:start({Type, Ref}, T), Pid. @@ -852,7 +868,7 @@ ms({applications, As}, #diameter_service{applications = Apps} = S) %% The fact that all capabilities can be configured on the transports %% means that the service doesn't necessarily represent a single -%% locally implemented Diameter peer as identified by Origin-Host: a +%% locally implemented Diameter node as identified by Origin-Host: a %% transport can configure its own Origin-Host. This means that the %% service little more than a placeholder for default capabilities %% plus a list of applications that individual transports can choose diff --git a/lib/diameter/src/base/diameter_traffic.erl b/lib/diameter/src/base/diameter_traffic.erl index 0b503338a6..784f9ca08f 100644 --- a/lib/diameter/src/base/diameter_traffic.erl +++ b/lib/diameter/src/base/diameter_traffic.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2013-2014. All Rights Reserved. +%% Copyright Ericsson AB 2013-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -77,7 +77,8 @@ {peerT :: ets:tid(), service_name :: diameter:service_name(), apps :: [#diameter_app{}], - sequence :: diameter:sequence()}). + sequence :: diameter:sequence(), + codec :: list()}). %% Record stored in diameter_request for each outgoing request. -record(request, @@ -92,11 +93,16 @@ %% # make_recvdata/1 %% --------------------------------------------------------------------------- -make_recvdata([SvcName, PeerT, Apps, Mask | _]) -> +make_recvdata([SvcName, PeerT, Apps, {_,_} = Mask | _]) -> %% from old code + make_recvdata([SvcName, PeerT, Apps, [{sequence, Mask}]]); + +make_recvdata([SvcName, PeerT, Apps, SvcOpts | _]) -> + {_,_} = Mask = proplists:get_value(sequence, SvcOpts), #recvdata{service_name = SvcName, peerT = PeerT, apps = Apps, - sequence = Mask}. + sequence = Mask, + codec = [T || {K,_} = T <- SvcOpts, K == string_decode]}. %% --------------------------------------------------------------------------- %% peer_up/1 @@ -270,8 +276,11 @@ recv_request(TPid, #diameter_packet{header = #diameter_header{application_id = Id}} = Pkt, Dict0, - #recvdata{peerT = PeerT, apps = Apps} + #recvdata{peerT = PeerT, + apps = Apps, + codec = Opts} = RecvData) -> + diameter_codec:setopts([{common_dictionary, Dict0} | Opts]), send_A(recv_R(diameter_service:find_incoming_app(PeerT, TPid, Id, Apps), TPid, Pkt, @@ -279,7 +288,13 @@ recv_request(TPid, RecvData), TPid, Dict0, - RecvData). + RecvData); + +recv_request(TPid, Pkt, Dict0, RecvData) -> %% from old code + recv_request(TPid, + Pkt, + Dict0, + #recvdata{} = erlang:append_element(RecvData, [])). %% recv_R/5 @@ -596,7 +611,7 @@ resend(false, Route = #diameter_avp{data = {Dict0, 'Route-Record', OH}}, Seq = diameter_session:sequence(Mask), Hdr = Hdr0#diameter_header{hop_by_hop_id = Seq}, - Msg = [Hdr, Route | Avps], + Msg = [Hdr, Route | Avps], %% reordered at encode resend(send_request(SvcName, App, Msg, Opts), Caps, Dict0, Pkt). %% The incoming request is relayed with the addition of a %% Route-Record. Note the requirement on the return from call/4 below, @@ -1225,10 +1240,9 @@ answer_rc(_, _, Sent) -> send_R(SvcName, AppOrAlias, Msg, Opts, Caller) -> case pick_peer(SvcName, AppOrAlias, Msg, Opts) of - {{_,_,_} = Transport, Mask} -> + {Transport, Mask, SvcOpts} -> + diameter_codec:setopts(SvcOpts), send_request(Transport, Mask, Msg, Opts, Caller, SvcName); - false -> - {error, no_connection}; {error, _} = No -> No end. @@ -1290,6 +1304,8 @@ send_request({TPid, Caps, App} SvcName, []). +%% send_R/7 + send_R({send, Msg}, Pkt, Transport, Opts, Caller, SvcName, Fs) -> send_R(make_request_packet(Msg, Pkt), Transport, @@ -1550,7 +1566,9 @@ a(Hdr, SvcName, discard) -> %% timer value is ignored. This means that an answer could be accepted %% from a peer after timeout in the case of failover. -retransmit({{_,_,App} = Transport, _Mask}, Req, Opts, SvcName, Timeout) -> +%% retransmit/5 + +retransmit({{_,_,App} = Transport, _, _}, Req, Opts, SvcName, Timeout) -> try retransmit(Transport, Req, SvcName, Timeout) of T -> recv_A(Timeout, SvcName, App, Opts, T) catch @@ -1571,17 +1589,26 @@ pick_peer(SvcName, pick_peer(SvcName, App, Msg, Opts#options{extra = []}); pick_peer(_, _, undefined, _) -> - false; + {error, no_connection}; pick_peer(SvcName, AppOrAlias, Msg, #options{filter = Filter, extra = Xtra}) -> - diameter_service:pick_peer(SvcName, - AppOrAlias, - {fun(D) -> get_destination(D, Msg) end, - Filter, - Xtra}). + pick(diameter_service:pick_peer(SvcName, + AppOrAlias, + {fun(D) -> get_destination(D, Msg) end, + Filter, + Xtra})). + +pick({{_,_,_} = Transport, Mask}) -> %% from old code; dialyzer complains + {Transport, Mask, []}; %% about this + +pick(false) -> + {error, no_connection}; + +pick(T) -> + T. %% handle_error/4 @@ -1679,8 +1706,14 @@ recv(TPid, Pid, TRef, Ref) -> %% send/2 -send(Pid, Pkt) -> - Pid ! {send, Pkt}. +send(Pid, Pkt) -> %% Strip potentially large message terms. + #diameter_packet{header = H, + bin = Bin, + transport_data = T} + = Pkt, + Pid ! {send, #diameter_packet{header = H, + bin = Bin, + transport_data = T}}. %% retransmit/4 diff --git a/lib/diameter/src/base/diameter_types.erl b/lib/diameter/src/base/diameter_types.erl index 442d90c98b..fe7613541c 100644 --- a/lib/diameter/src/base/diameter_types.erl +++ b/lib/diameter/src/base/diameter_types.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2013. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -90,7 +90,12 @@ 'OctetString'(decode, Bin) when is_binary(Bin) -> - binary_to_list(Bin); + case diameter_codec:getopt(string_decode) of + true -> + binary_to_list(Bin); + _ -> + Bin + end; 'OctetString'(decode, B) -> ?INVALID_LENGTH(B); @@ -298,21 +303,29 @@ 'OctetString'(M, lists:duplicate(0,7)); 'DiameterURI'(encode, #diameter_uri{type = Type, - fqdn = D, - port = P, + fqdn = DN, + port = PN, transport = T, - protocol = Prot} - = U) -> - S = lists:append([atom_to_list(Type), "://", D, - ":", integer_to_list(P), + protocol = P}) + when (Type == 'aaa' orelse Type == 'aaas'), + is_integer(PN), + 0 =< PN, + (T == tcp orelse T == sctp orelse T == udp), + (P == diameter orelse P == radius orelse P == 'tacacs+'), + (P /= diameter orelse T /= udp) -> + iolist_to_binary([atom_to_list(Type), "://", DN, + ":", integer_to_list(PN), ";transport=", atom_to_list(T), - ";protocol=", atom_to_list(Prot)]), - U = scan_uri(S), %% assert - list_to_binary(S); + ";protocol=", atom_to_list(P)]); +%% Don't omit defaults since they're dependent on whether RFC 3588 or +%% 6733 is being followed. For one, we don't know this at encode; for +%% two (more importantly), we don't know how the peer will interpret +%% defaults, so it's best to be explicit. Interpret defaults on decode +%% since there's no choice. 'DiameterURI'(encode, Str) -> Bin = iolist_to_binary(Str), - #diameter_uri{} = scan_uri(Bin), %% type check + #diameter_uri{} = scan_uri(Bin), %% assert Bin. %% -------------------- @@ -321,7 +334,6 @@ 'IPFilterRule'(encode = M, zero) -> 'OctetString'(M, lists:duplicate(0,33)); -%% TODO: parse grammar. 'IPFilterRule'(M, X) -> 'OctetString'(M, X). @@ -331,7 +343,6 @@ 'QoSFilterRule'(encode = M, zero = X) -> 'IPFilterRule'(M, X); -%% TODO: parse grammar. 'QoSFilterRule'(M, X) -> 'OctetString'(M, X). @@ -339,7 +350,13 @@ 'UTF8String'(decode, Bin) when is_binary(Bin) -> - tl([0|_] = unicode:characters_to_list([0, Bin])); %% assert list return + case diameter_codec:getopt(string_decode) of + true -> + %% assert list return + tl([0|_] = unicode:characters_to_list([0, Bin])); + false -> + <<_/binary>> = unicode:characters_to_binary(Bin) + end; 'UTF8String'(decode, B) -> ?INVALID_LENGTH(B); @@ -507,55 +524,87 @@ msb(false) -> ?TIME_2036. %% %% aaa-protocol = ( "diameter" / "radius" / "tacacs+" ) -scan_uri(Bin) - when is_binary(Bin) -> - scan_uri(binary_to_list(Bin)); -scan_uri("aaa://" ++ Rest) -> - scan_fqdn(Rest, #diameter_uri{type = aaa}); -scan_uri("aaas://" ++ Rest) -> - scan_fqdn(Rest, #diameter_uri{type = aaas}). - -scan_fqdn(S, U) -> - {[_|_] = F, Rest} = lists:splitwith(fun is_fqdn/1, S), - scan_opt_port(Rest, U#diameter_uri{fqdn = F}). - -scan_opt_port(":" ++ S, U) -> - {[_|_] = P, Rest} = lists:splitwith(fun is_digit/1, S), - scan_opt_transport(Rest, U#diameter_uri{port = list_to_integer(P)}); -scan_opt_port(S, U) -> - scan_opt_transport(S, U). - -scan_opt_transport(";transport=" ++ S, U) -> - {P, Rest} = transport(S), - scan_opt_protocol(Rest, U#diameter_uri{transport = P}); -scan_opt_transport(S, U) -> - scan_opt_protocol(S, U). - -scan_opt_protocol(";protocol=" ++ S, U) -> - {P, ""} = protocol(S), - U#diameter_uri{protocol = P}; -scan_opt_protocol("", U) -> - U. - -transport("tcp" ++ S) -> - {tcp, S}; -transport("sctp" ++ S) -> - {sctp, S}; -transport("udp" ++ S) -> - {udp, S}. - -protocol("diameter" ++ S) -> - {diameter, S}; -protocol("radius" ++ S) -> - {radius, S}; -protocol("tacacs+" ++ S) -> - {'tacacs+', S}. - -is_fqdn(C) -> - is_digit(C) orelse is_alpha(C) orelse C == $. orelse C == $-. - -is_alpha(C) -> - ($a =< C andalso C =< $z) orelse ($A =< C andalso C =< $Z). - -is_digit(C) -> - $0 =< C andalso C =< $9. +%% RFC 6733, 4.3.1, changes the defaults: +%% +%% "aaa://" FQDN [ port ] [ transport ] [ protocol ] +%% +%% ; No transport security +%% +%% "aaas://" FQDN [ port ] [ transport ] [ protocol ] +%% +%% ; Transport security used +%% +%% FQDN = < Fully Qualified Domain Name > +%% +%% port = ":" 1*DIGIT +%% +%% ; One of the ports used to listen for +%% ; incoming connections. +%% ; If absent, the default Diameter port +%% ; (3868) is assumed if no transport +%% ; security is used and port 5658 when +%% ; transport security (TLS/TCP and DTLS/SCTP) +%% ; is used. +%% +%% transport = ";transport=" transport-protocol +%% +%% ; One of the transports used to listen +%% ; for incoming connections. If absent, +%% ; the default protocol is assumed to be TCP. +%% ; UDP MUST NOT be used when the aaa-protocol +%% ; field is set to diameter. +%% +%% transport-protocol = ( "tcp" / "sctp" / "udp" ) +%% +%% protocol = ";protocol=" aaa-protocol +%% +%% ; If absent, the default AAA protocol +%% ; is Diameter. +%% +%% aaa-protocol = ( "diameter" / "radius" / "tacacs+" ) + +scan_uri(Bin) -> + RE = "^(aaas?)://" + "([-a-zA-Z0-9.]+)" + "(:([0-9]+))?" + "(;transport=(tcp|sctp|udp))?" + "(;protocol=(diameter|radius|tacacs\\+))?$", + {match, [A, DN, PN, T, P]} = re:run(Bin, + RE, + [{capture, [1,2,4,6,8], binary}]), + Type = to_atom(A), + {PN0, T0} = defaults(diameter_codec:getopt(rfc), Type), + #diameter_uri{type = Type, + fqdn = from_bin(DN), + port = to_int(PN, PN0), + transport = to_atom(T, T0), + protocol = to_atom(P, diameter)}. + +%% Choose defaults based on the RFC, since 6733 has changed them. +defaults(3588, _) -> + {3868, sctp}; +defaults(6733, aaa) -> + {3868, tcp}; +defaults(6733, aaas) -> + {5658, tcp}. + +from_bin(B) -> + case diameter_codec:getopt(string_decode) of + true -> + binary_to_list(B); + false -> + B + end. + +to_int(<<>>, N) -> + N; +to_int(B, _) -> + binary_to_integer(B). + +to_atom(<<>>, A) -> + A; +to_atom(B, _) -> + to_atom(B). + +to_atom(B) -> + binary_to_atom(B, latin1). diff --git a/lib/diameter/src/base/diameter_watchdog.erl b/lib/diameter/src/base/diameter_watchdog.erl index 67715906e8..de9c4bca33 100644 --- a/lib/diameter/src/base/diameter_watchdog.erl +++ b/lib/diameter/src/base/diameter_watchdog.erl @@ -124,14 +124,16 @@ i({Ack, T, Pid, {RecvData, wait(Ack, Pid), {_, Seed} = diameter_lib:seed(), random:seed(Seed), - putr(restart, {T, Opts, Svc}), %% save seeing it in trace - putr(dwr, dwr(Caps)), %% + putr(restart, {T, Opts, Svc, SvcOpts}), %% save seeing it in trace + putr(dwr, dwr(Caps)), %% {_,_} = Mask = proplists:get_value(sequence, SvcOpts), Restrict = proplists:get_value(restrict_connections, SvcOpts), Nodes = restrict_nodes(Restrict), Dict0 = common_dictionary(Apps), + diameter_codec:setopts([{common_dictionary, Dict0}, + {string_decode, false}]), #watchdog{parent = Pid, - transport = start(T, Opts, Mask, Nodes, Dict0, Svc), + transport = start(T, Opts, SvcOpts, Nodes, Dict0, Svc), tw = proplists:get_value(watchdog_timer, Opts, ?DEFAULT_TW_INIT), @@ -166,11 +168,11 @@ config({okay, N}, Rec) when ?IS_NATURAL(N) -> Rec#config{okay = N}. -%% start/5 +%% start/6 -start(T, Opts, Mask, Nodes, Dict0, Svc) -> +start(T, Opts, SvcOpts, Nodes, Dict0, Svc) -> {_MRef, Pid} - = diameter_peer_fsm:start(T, Opts, {Mask, Nodes, Dict0, Svc}), + = diameter_peer_fsm:start(T, Opts, {SvcOpts, Nodes, Dict0, Svc}), Pid. %% common_dictionary/1 @@ -320,7 +322,7 @@ code_change(_, State, _) -> %% expiry; or another watchdog is saying the same after reestablishing %% a connection previously had by this one. transition(close, #watchdog{}) -> - {{accept, _}, _, _} = getr(restart), %% assert + {accept, _} = role(), %% assert stop; %% Service is asking for the peer to be taken down gracefully. @@ -333,8 +335,9 @@ transition({shutdown = T, Pid, Reason}, #watchdog{parent = Pid, send(TPid, {T, self(), Reason}), S#watchdog{shutdown = true}; -%% Transport is telling us that DPA has been sent in response to DPR: -%% its death should lead to ours. +%% Transport is telling us that DPA has been sent in response to DPR, +%% or that DPR has been explicitly sent: transport death should lead +%% to ours. transition({'DPR', TPid}, #watchdog{transport = TPid} = S) -> S#watchdog{shutdown = true}; @@ -369,7 +372,7 @@ transition({open, TPid, Hosts, _} = Open, restrict = {_,R}, config = #config{suspect = OS}} = S) -> - case okay(getr(restart), Hosts, R) of + case okay(role(), Hosts, R) of okay -> set_watchdog(S#watchdog{status = okay, num_dwa = OS}); @@ -423,7 +426,7 @@ transition({'DOWN', _, process, TPid, _Reason} = D, = S0) -> S = S0#watchdog{pending = false, transport = undefined}, - {{M,_}, _, _} = getr(restart), + {M,_} = role(), %% Close an accepting watchdog immediately if there's no %% restriction on the number of connections to the same peer: the @@ -490,7 +493,7 @@ encode(dwa, Dict0, #diameter_packet{header = H, transport_data = TD} %% okay/3 -okay({{accept, Ref}, _, _}, Hosts, Restrict) -> +okay({accept, Ref}, Hosts, Restrict) -> T = {?MODULE, connection, Ref, Hosts}, diameter_reg:add(T), if Restrict -> @@ -501,7 +504,7 @@ okay({{accept, Ref}, _, _}, Hosts, Restrict) -> %% Register before matching so that at least one of two registering %% processes will match the other. -okay({{connect, _}, _, _}, _, _) -> +okay({connect, _}, _, _) -> okay. %% okay/2 @@ -516,6 +519,11 @@ okay(C) -> [_|_] = [send(P, close) || {_,P} <- C, self() /= P], reopen. +%% role/0 + +role() -> + element(1, getr(restart)). + %% set_watchdog/1 set_watchdog(#watchdog{tw = TwInit, @@ -550,7 +558,7 @@ send_watchdog(#watchdog{pending = false, ?LOG(send, 'DWR'), S#watchdog{pending = true}. -%% Dont' count encode errors since we don't expect any on DWR/DWA. +%% Don't count encode errors since we don't expect any on DWR/DWA. %% recv/3 @@ -572,11 +580,18 @@ rcv('DWR', Pkt, #watchdog{transport = TPid, DPkt = diameter_codec:decode(Dict0, Pkt), diameter_traffic:incr(recv, DPkt, TPid, Dict0), diameter_traffic:incr_error(recv, DPkt, TPid, Dict0), - EPkt = encode(dwa, Dict0, Pkt), + #diameter_packet{header = H, + transport_data = T, + bin = Bin} + = EPkt + = encode(dwa, Dict0, Pkt), diameter_traffic:incr(send, EPkt, TPid, Dict0), diameter_traffic:incr_rc(send, EPkt, TPid, Dict0), - send(TPid, {send, EPkt}), + %% Strip potentially large message terms. + send(TPid, {send, #diameter_packet{header = H, + transport_data = T, + bin = Bin}}), ?LOG(send, 'DWA'); rcv('DWA', Pkt, #watchdog{transport = TPid, @@ -591,9 +606,10 @@ rcv('DWA', Pkt, #watchdog{transport = TPid, rcv(N, _, _) when N == 'CER'; N == 'CEA'; - N == 'DPR'; - N == 'DPA' -> + N == 'DPR' -> false; +%% DPR can be sent explicitly with diameter:call/4. Only the +%% corresponding DPAs arrive here. rcv(_, Pkt, #watchdog{transport = TPid, dictionary = Dict0, @@ -794,26 +810,28 @@ restart(S) -> %% reconnect has won race with timeout %% state down rather then initial when receiving notification of an %% open connection. -restart({{connect, _} = T, Opts, Svc}, +restart({T, Opts, Svc}, S) -> %% put in old code + restart({T, Opts, Svc, []}, S); + +restart({{connect, _} = T, Opts, Svc, SvcOpts}, #watchdog{parent = Pid, - sequence = Mask, restrict = {R,_}, dictionary = Dict0} = S) -> send(Pid, {reconnect, self()}), Nodes = restrict_nodes(R), - S#watchdog{transport = start(T, Opts, Mask, Nodes, Dict0, Svc), + S#watchdog{transport = start(T, Opts, SvcOpts, Nodes, Dict0, Svc), restrict = {R, lists:member(node(), Nodes)}}; %% No restriction on the number of connections to the same peer: just %% die. Note that a state machine never enters state REOPEN in this %% case. -restart({{accept, _}, _, _}, #watchdog{restrict = {_, false}}) -> +restart({{accept, _}, _, _, _}, #watchdog{restrict = {_, false}}) -> stop; %% 'DOWN' was in old code: 'close' was not sent %% Otherwise hang around until told to die, either by the service or %% by another watchdog. -restart({{accept, _}, _, _}, S) -> +restart({{accept, _}, _, _, _}, S) -> S. %% Don't currently use Opts/Svc in the accept case. diff --git a/lib/diameter/src/diameter.appup.src b/lib/diameter/src/diameter.appup.src index 881d25b5fb..a54eb24031 100644 --- a/lib/diameter/src/diameter.appup.src +++ b/lib/diameter/src/diameter.appup.src @@ -2,7 +2,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2014. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -59,7 +59,32 @@ {"1.7.1", [{load_module, diameter_traffic}, %% 17.3 {load_module, diameter_watchdog}, {load_module, diameter_peer_fsm}, - {load_module, diameter_service}]} + {load_module, diameter_service}]}, + {"1.8", [{load_module, diameter_lib}, %% 17.4 + {load_module, diameter_peer}, + {load_module, diameter_reg}, + {load_module, diameter_session}, + {load_module, diameter_stats}, + {load_module, diameter_sync}, + {load_module, diameter_capx}, + {load_module, diameter_codec}, + {load_module, diameter_types}, + {load_module, diameter_traffic}, + {load_module, diameter_service}, + {load_module, diameter_peer_fsm}, + {load_module, diameter_watchdog}, + {load_module, diameter_tcp}, + {load_module, diameter_sctp}, + {load_module, diameter_config}, + {load_module, diameter}, + {load_module, diameter_gen_base_rfc6733}, + {load_module, diameter_gen_acct_rfc6733}, + {load_module, diameter_gen_base_rfc3588}, + {load_module, diameter_gen_base_accounting}, + {load_module, diameter_gen_relay}, + {update, diameter_transport_sup, supervisor}, + {update, diameter_service_sup, supervisor}, + {update, diameter_sup, supervisor}]} ], [ {"0.9", [{restart_application, diameter}]}, @@ -101,6 +126,31 @@ {"1.7.1", [{load_module, diameter_service}, {load_module, diameter_peer_fsm}, {load_module, diameter_watchdog}, - {load_module, diameter_traffic}]} + {load_module, diameter_traffic}]}, + {"1.8", [{update, diameter_sup, supervisor}, + {update, diameter_service_sup, supervisor}, + {update, diameter_transport_sup, supervisor}, + {load_module, diameter_gen_relay}, + {load_module, diameter_gen_base_accounting}, + {load_module, diameter_gen_base_rfc3588}, + {load_module, diameter_gen_acct_rfc6733}, + {load_module, diameter_gen_base_rfc6733}, + {load_module, diameter}, + {load_module, diameter_config}, + {load_module, diameter_sctp}, + {load_module, diameter_tcp}, + {load_module, diameter_watchdog}, + {load_module, diameter_peer_fsm}, + {load_module, diameter_service}, + {load_module, diameter_traffic}, + {load_module, diameter_types}, + {load_module, diameter_codec}, + {load_module, diameter_capx}, + {load_module, diameter_sync}, + {load_module, diameter_stats}, + {load_module, diameter_session}, + {load_module, diameter_reg}, + {load_module, diameter_peer}, + {load_module, diameter_lib}]} ] }. diff --git a/lib/diameter/test/diameter_app_SUITE.erl b/lib/diameter/test/diameter_app_SUITE.erl index cf34c762e1..6975e83830 100644 --- a/lib/diameter/test/diameter_app_SUITE.erl +++ b/lib/diameter/test/diameter_app_SUITE.erl @@ -226,15 +226,18 @@ ignored({FromMod,_,_}, {ToMod,_,_} = To, Rel)-> %% New time api in OTP 18. time_api() -> - [{erlang, F, A} || {F,A} <- [{convert_time_resolution,3}, + [{erlang, F, A} || {F,A} <- [{convert_time_unit,3}, {monotonic_time,0}, {monotonic_time,1}, + {system_time,0}, + {system_time,1}, {time_offset,0}, {time_offset,1}, - {time_resolution,0}, {timestamp,0}, {unique_integer,0}, - {unique_integer,1}]]. + {unique_integer,1}]] + ++ [{os, system_time, 0}, + {os, system_time, 1}]. release() -> Rel = erlang:system_info(otp_release), diff --git a/lib/diameter/test/diameter_codec_test.erl b/lib/diameter/test/diameter_codec_test.erl index 472755c62a..854b71ba93 100644 --- a/lib/diameter/test/diameter_codec_test.erl +++ b/lib/diameter/test/diameter_codec_test.erl @@ -356,8 +356,15 @@ values('DiameterURI') -> Tr <- ["" | [";transport=" ++ X || X <- ["tcp", "sctp", "udp"]]], Pr <- ["" | [";protocol=" ++ X - || X <- ["diameter","radius","tacacs+"]]]], - []}; + || X <- ["diameter","radius","tacacs+"]]], + Tr /= ";transport=udp" + orelse (Pr /= ";protocol=diameter" andalso Pr /= "")], + ["aaa://diameter.se;transport=udp;protocol=diameter", + "aaa://diameter.se;transport=udp", + "aaa://:3868", + "aaax://diameter.se", + "aaa://diameter.se;transport=tcpx", + "aaa://diameter.se;transport=tcp;protocol=diameter "]}; values(T) when T == 'IPFilterRule'; diff --git a/lib/diameter/test/diameter_config_SUITE.erl b/lib/diameter/test/diameter_config_SUITE.erl index ad5b3f9420..77f7aace1b 100644 --- a/lib/diameter/test/diameter_config_SUITE.erl +++ b/lib/diameter/test/diameter_config_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2013. All Rights Reserved. +%% Copyright Ericsson AB 2013-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -82,6 +82,9 @@ [false], [[node(), node()]]], [[x]]}, + {string_decode, + [[true], [false]], + [[0], [x]]}, {invalid_option, %% invalid service options are rejected [], [[x], @@ -157,6 +160,12 @@ {length_errors, [[exit], [handle], [discard]], [[x]]}, + {dpr_timeout, + [[0], [3000], [16#FFFFFFFF]], + [[infinity], [-1], [1 bsl 32], [x]]}, + {dpa_timeout, + [[0], [3000], [16#FFFFFFFF]], + [[infinity], [-1], [1 bsl 32], [x]]}, {connect_timer, [[3000]], [[infinity]]}, @@ -171,6 +180,9 @@ [[{suspect, 2}]]], [[x], [[{open, 0}]]]}, + {pool_size, + [[1], [100]], + [[0], [infinity], [-1], [x]]}, {private, [[x]], []}, diff --git a/lib/diameter/test/diameter_dpr_SUITE.erl b/lib/diameter/test/diameter_dpr_SUITE.erl index f3f16b06e0..81178e2bda 100644 --- a/lib/diameter/test/diameter_dpr_SUITE.erl +++ b/lib/diameter/test/diameter_dpr_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2012-2014. All Rights Reserved. +%% Copyright Ericsson AB 2012-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -32,6 +32,7 @@ %% testcases -export([start/1, connect/1, + send_dpr/1, remove_transport/1, stop_service/1, check/1, @@ -41,6 +42,7 @@ -export([disconnect/5]). -include("diameter.hrl"). +-include("diameter_gen_base_rfc6733.hrl"). %% =========================================================================== @@ -51,9 +53,6 @@ -define(CLIENT, "CLIENT"). -define(SERVER, "SERVER"). --define(DICT_COMMON, ?DIAMETER_DICT_COMMON). --define(APP_ID, ?DICT_COMMON:id()). - %% Config for diameter:start_service/2. -define(SERVICE(Host), [{'Origin-Host', Host}, @@ -61,9 +60,10 @@ {'Host-IP-Address', [?ADDR]}, {'Vendor-Id', hd(Host)}, %% match this in disconnect/5 {'Product-Name', "OTP/diameter"}, - {'Acct-Application-Id', [?APP_ID]}, + {'Acct-Application-Id', [0]}, {restrict_connections, false}, - {application, [{dictionary, ?DICT_COMMON}, + {application, [{dictionary, diameter_gen_base_rfc6733}, + {alias, common}, {module, #diameter_callback{_ = false}}]}]). %% Disconnect reasons that diameter passes as the first argument of a @@ -74,10 +74,12 @@ -define(CAUSES, [0, rebooting, 1, busy, 2, goaway]). %% Establish one client connection for each element of this list, -%% configured with disconnect/5 as disconnect_cb and returning the -%% specified value. +%% configured with disconnect/5, disconnect_cb returning the specified +%% value. -define(RETURNS, - [[close, {dpr, [{cause, invalid}]}], [ignore, close], []] + [[close, {dpr, [{cause, invalid}]}], + [ignore, close], + []] ++ [[{dpr, [{timeout, 5000}, {cause, T}]}] || T <- ?CAUSES]). %% =========================================================================== @@ -86,7 +88,7 @@ suite() -> [{timetrap, {seconds, 60}}]. all() -> - [{group, R} || R <- ?REASONS]. + [start, send_dpr, stop | [{group, R} || R <- ?REASONS]]. %% The group determines how transports are terminated: by remove_transport, %% stop_service or application stop. @@ -111,6 +113,22 @@ start(_Config) -> ok = diameter:start_service(?SERVER, ?SERVICE(?SERVER)), ok = diameter:start_service(?CLIENT, ?SERVICE(?CLIENT)). +send_dpr(_Config) -> + LRef = ?util:listen(?SERVER, tcp), + Ref = ?util:connect(?CLIENT, tcp, LRef, [{dpa_timeout, 10000}]), + #diameter_base_DPA{'Result-Code' = 2001} + = diameter:call(?CLIENT, + common, + ['DPR', {'Origin-Host', "CLIENT.erlang.org"}, + {'Origin-Realm', "erlang.org"}, + {'Disconnect-Cause', 0}]), + ok = receive %% endure the transport dies on DPA + #diameter_event{service = ?CLIENT, info = {down, Ref, _, _}} -> + ok + after 5000 -> + erlang:process_info(self(), messages) + end. + connect(Config) -> Pid = spawn(fun init/0), %% process for disconnect_cb to bang Grp = group(Config), diff --git a/lib/diameter/test/diameter_traffic_SUITE.erl b/lib/diameter/test/diameter_traffic_SUITE.erl index 9822b95301..10c58ab6e7 100644 --- a/lib/diameter/test/diameter_traffic_SUITE.erl +++ b/lib/diameter/test/diameter_traffic_SUITE.erl @@ -122,8 +122,6 @@ -define(ADDR, {127,0,0,1}). --define(CLIENT, "CLIENT"). --define(SERVER, "SERVER"). -define(REALM, "erlang.org"). -define(HOST(Host, Realm), Host ++ [$.|Realm]). @@ -141,11 +139,19 @@ %% Which common dictionary to use in the clients. -define(RFCS, [rfc3588, rfc6733]). +%% Whether to decode stringish Diameter types to strings, or leave +%% them as binary. +-define(STRING_DECODES, [true, false]). + -record(group, - {client_encoding, + {client_service, + client_encoding, client_dict0, + client_strings, + server_service, server_encoding, - server_container}). + server_container, + server_strings}). %% Not really what we should be setting unless the message is sent in %% the common application but diameter doesn't care. @@ -166,7 +172,7 @@ ?answer_message(_, ResultCode)). %% Config for diameter:start_service/2. --define(SERVICE(Name), +-define(SERVICE(Name, Decode), [{'Origin-Host', Name ++ "." ++ ?REALM}, {'Origin-Realm', ?REALM}, {'Host-IP-Address', [?ADDR]}, @@ -175,6 +181,7 @@ {'Auth-Application-Id', [?DIAMETER_APP_ID_COMMON]}, {'Acct-Application-Id', [?DIAMETER_APP_ID_ACCOUNTING]}, {restrict_connections, false}, + {string_decode, Decode}, {spawn_opt, [{min_heap_size, 5000}]} | [{application, [{dictionary, D}, {module, ?MODULE}, @@ -227,28 +234,53 @@ suite() -> [{timetrap, {seconds, 60}}]. all() -> - [start, start_services, add_transports, result_codes] - ++ [{group, ?util:name([R,D,A,C]), P} || R <- ?ENCODINGS, - D <- ?RFCS, - A <- ?ENCODINGS, - C <- ?CONTAINERS, - P <- [[], [parallel]]] - ++ [outstanding, remove_transports, empty, stop_services, stop]. + [start, result_codes, {group, traffic}, outstanding, empty, stop]. groups() -> Ts = tc(), - [{?util:name([R,D,A,C]), [], Ts} || R <- ?ENCODINGS, - D <- ?RFCS, - A <- ?ENCODINGS, - C <- ?CONTAINERS]. + [{?util:name([R,D,A,C]), [parallel], Ts} || R <- ?ENCODINGS, + D <- ?RFCS, + A <- ?ENCODINGS, + C <- ?CONTAINERS] + ++ + [{?util:name([R,D,A,C,SD,CD]), + [], + [start_services, + add_transports, + result_codes, + {group, ?util:name([R,D,A,C])}, + remove_transports, + stop_services]} + || R <- ?ENCODINGS, + D <- ?RFCS, + A <- ?ENCODINGS, + C <- ?CONTAINERS, + SD <- ?STRING_DECODES, + CD <- ?STRING_DECODES] + ++ + [{traffic, [parallel], [{group, ?util:name([R,D,A,C,SD,CD])} + || R <- ?ENCODINGS, + D <- ?RFCS, + A <- ?ENCODINGS, + C <- ?CONTAINERS, + SD <- ?STRING_DECODES, + CD <- ?STRING_DECODES]}]. init_per_group(Name, Config) -> - [R,D,A,C] = ?util:name(Name), - G = #group{client_encoding = R, - client_dict0 = dict0(D), - server_encoding = A, - server_container = C}, - [{group, G} | Config]. + case ?util:name(Name) of + [R,D,A,C,SD,CD] -> + G = #group{client_service = [$C|?util:unique_string()], + client_encoding = R, + client_dict0 = dict0(D), + client_strings = CD, + server_service = [$S|?util:unique_string()], + server_encoding = A, + server_container = C, + server_strings = SD}, + [{group, G} | Config]; + _ -> + Config + end. end_per_group(_, _) -> ok. @@ -319,18 +351,26 @@ tc() -> start(_Config) -> ok = diameter:start(). -start_services(_Config) -> - ok = diameter:start_service(?SERVER, ?SERVICE(?SERVER)), - ok = diameter:start_service(?CLIENT, [{sequence, ?CLIENT_MASK} - | ?SERVICE(?CLIENT)]). +start_services(Config) -> + #group{client_service = CN, + client_strings = CD, + server_service = SN, + server_strings = SD} + = group(Config), + ok = diameter:start_service(SN, ?SERVICE(SN, SD)), + ok = diameter:start_service(CN, [{sequence, ?CLIENT_MASK} + | ?SERVICE(CN, CD)]). add_transports(Config) -> - LRef = ?util:listen(?SERVER, + #group{client_service = CN, + server_service = SN} + = group(Config), + LRef = ?util:listen(SN, tcp, [{capabilities_cb, fun capx/2}, {spawn_opt, [{min_heap_size, 8096}]}, {applications, apps(rfc3588)}]), - Cs = [?util:connect(?CLIENT, + Cs = [?util:connect(CN, tcp, LRef, [{id, Id}, @@ -354,12 +394,18 @@ outstanding(_Config) -> is_atom(element(1,T))]. remove_transports(Config) -> + #group{client_service = CN, + server_service = SN} + = group(Config), [LRef | Cs] = ?util:read_priv(Config, "transport"), - [?util:disconnect(?CLIENT, C, ?SERVER, LRef) || C <- Cs]. + [?util:disconnect(CN, C, SN, LRef) || C <- Cs]. -stop_services(_Config) -> - ok = diameter:stop_service(?CLIENT), - ok = diameter:stop_service(?SERVER). +stop_services(Config) -> + #group{client_service = CN, + server_service = SN} + = group(Config), + ok = diameter:stop_service(CN), + ok = diameter:stop_service(SN). %% Ensure even transports have been removed from request table. empty(_Config) -> @@ -439,8 +485,9 @@ send_arbitrary(Config) -> ['ASA', _SessionId, {'Result-Code', ?SUCCESS} | Avps] = call(Config, Req), {'AVP', [#diameter_avp{name = 'Product-Name', - value = "XXX"}]} - = lists:last(Avps). + value = V}]} + = lists:last(Avps), + "XXX" = string(V, Config). %% Send an unknown AVP (to some client) and check that it comes back. send_unknown(Config) -> @@ -594,9 +641,11 @@ send_nopeer(Config) -> {error, no_connection} = call(Config, Req, [{extra, [?EXTRA]}]). %% Send something on an unconfigured application. -send_noapp(_Config) -> +send_noapp(Config) -> + #group{client_service = CN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}], - {error, no_connection} = diameter:call(?CLIENT, unknown_alias, Req). + {error, no_connection} = diameter:call(CN, unknown_alias, Req). %% Send something that's discarded by prepare_request. send_discard(Config) -> @@ -608,8 +657,10 @@ send_any_1(Config) -> Req = ['STR', {'Termination-Cause', ?LOGOUT}], {error, no_connection} = call(Config, Req, [{filter, {any, []}}]). send_any_2(Config) -> + #group{server_service = SN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}, - {'Destination-Host', [?HOST(?SERVER, "unknown.org")]}], + {'Destination-Host', [?HOST(SN, "unknown.org")]}], ?answer_message(?UNABLE_TO_DELIVER) = call(Config, Req, [{filter, {any, [host, realm]}}]). @@ -621,8 +672,10 @@ send_all_1(Config) -> = call(Config, Req, [{filter, {all, [{host, any}, {realm, Realm}]}}]). send_all_2(Config) -> + #group{server_service = SN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}, - {'Destination-Host', [?HOST(?SERVER, "unknown.org")]}], + {'Destination-Host', [?HOST(SN, "unknown.org")]}], {error, no_connection} = call(Config, Req, [{filter, {all, [host, realm]}}]). @@ -655,8 +708,10 @@ send_encode_error(Config) -> %% Send with filtering and expect success. send_destination_1(Config) -> + #group{server_service = SN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}, - {'Destination-Host', [?HOST(?SERVER, ?REALM)]}], + {'Destination-Host', [?HOST(SN, ?REALM)]}], ['STA', _SessionId, {'Result-Code', ?SUCCESS} | _] = call(Config, Req, [{filter, {all, [host, realm]}}]). send_destination_2(Config) -> @@ -672,8 +727,10 @@ send_destination_3(Config) -> {error, no_connection} = call(Config, Req, [{filter, {all, [host, realm]}}]). send_destination_4(Config) -> + #group{server_service = SN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}, - {'Destination-Host', [?HOST(?SERVER, "unknown.org")]}], + {'Destination-Host', [?HOST(SN, "unknown.org")]}], {error, no_connection} = call(Config, Req, [{filter, {all, [host, realm]}}]). @@ -685,8 +742,10 @@ send_destination_5(Config) -> ?answer_message(?REALM_NOT_SERVED) = call(Config, Req). send_destination_6(Config) -> + #group{server_service = SN} + = group(Config), Req = ['STR', {'Termination-Cause', ?LOGOUT}, - {'Destination-Host', [?HOST(?SERVER, "unknown.org")]}], + {'Destination-Host', [?HOST(SN, "unknown.org")]}], ?answer_message(?UNABLE_TO_DELIVER) = call(Config, Req). @@ -748,16 +807,31 @@ send_anything(Config) -> %% =========================================================================== +group(Config) -> + #group{} = proplists:get_value(group, Config). + +string(V, Config) -> + #group{client_strings = B} = group(Config), + decode(V,B). + +decode(S, true) + when is_list(S) -> + S; +decode(B, false) + when is_binary(B) -> + binary_to_list(B). + call(Config, Req) -> call(Config, Req, []). call(Config, Req, Opts) -> Name = proplists:get_value(testcase, Config), - #group{client_encoding = ReqEncoding, + #group{client_service = CN, + client_encoding = ReqEncoding, client_dict0 = Dict0} = Group - = proplists:get_value(group, Config), - diameter:call(?CLIENT, + = group(Config), + diameter:call(CN, dict(Req, Dict0), msg(Req, ReqEncoding, Dict0), [{extra, [{Name, Group}, diameter_lib:now()]} | Opts]). @@ -844,35 +918,38 @@ peer_down(_SvcName, _Peer, State) -> %% pick_peer/6-7 -pick_peer(Peers, _, ?CLIENT, _State, {Name, Group}, _) +pick_peer(Peers, _, [$C|_], _State, {Name, Group}, _) when Name /= send_detach -> find(Group, Peers). -pick_peer(_Peers, _, ?CLIENT, _State, {send_nopeer, _}, _, ?EXTRA) -> +pick_peer(_Peers, _, [$C|_], _State, {send_nopeer, _}, _, ?EXTRA) -> false; -pick_peer(Peers, _, ?CLIENT, _State, {send_detach, Group}, _, {_,_}) -> +pick_peer(Peers, _, [$C|_], _State, {send_detach, Group}, _, {_,_}) -> find(Group, Peers). -find(#group{server_encoding = A, server_container = C}, Peers) -> +find(#group{client_service = CN, + server_encoding = A, + server_container = C}, + Peers) -> Id = {A,C}, - [P] = [P || P <- Peers, id(Id, P)], + [P] = [P || P <- Peers, id(Id, P, CN)], {ok, P}. -id(Id, {Pid, _Caps}) -> +id(Id, {Pid, _Caps}, SvcName) -> [{ref, _}, {type, _}, {options, Opts} | _] - = diameter:service_info(?CLIENT, Pid), + = diameter:service_info(SvcName, Pid), lists:member({id, Id}, Opts). %% prepare_request/5-6 -prepare_request(_Pkt, ?CLIENT, {_Ref, _Caps}, {send_discard, _}, _) -> +prepare_request(_Pkt, [$C|_], {_Ref, _Caps}, {send_discard, _}, _) -> {discard, unprepared}; -prepare_request(Pkt, ?CLIENT, {_Ref, Caps}, {Name, Group}, _) -> +prepare_request(Pkt, [$C|_], {_Ref, Caps}, {Name, Group}, _) -> {send, prepare(Pkt, Caps, Name, Group)}. -prepare_request(Pkt, ?CLIENT, {_Ref, Caps}, {send_detach, Group}, _, _) -> +prepare_request(Pkt, [$C|_], {_Ref, Caps}, {send_detach, Group}, _, _) -> {eval_packet, {send, prepare(Pkt, Caps, Group)}, [fun log/2, detach]}. log(#diameter_packet{bin = Bin} = P, T) @@ -1043,10 +1120,10 @@ prepare_retransmit(_Pkt, false, _Peer, _Name, _Group) -> %% handle_answer/6-7 -handle_answer(Pkt, Req, ?CLIENT, Peer, {Name, Group}, _) -> +handle_answer(Pkt, Req, [$C|_], Peer, {Name, Group}, _) -> answer(Pkt, Req, Peer, Name, Group). -handle_answer(Pkt, Req, ?CLIENT, Peer, {send_detach = Name, Group}, _, X) -> +handle_answer(Pkt, Req, [$C|_], Peer, {send_detach = Name, Group}, _, X) -> {Pid, Ref} = X, Pid ! {Ref, answer(Pkt, Req, Peer, Name, Group)}. @@ -1075,13 +1152,13 @@ app(Req, _, Dict0) -> %% handle_error/6 -handle_error(timeout = Reason, _Req, ?CLIENT, _Peer, _, Time) -> +handle_error(timeout = Reason, _Req, [$C|_], _Peer, _, Time) -> Now = diameter_lib:now(), {Reason, {diameter_lib:timestamp(Time), diameter_lib:timestamp(Now), diameter_lib:micro_diff(Now, Time)}}; -handle_error(Reason, _Req, ?CLIENT, _Peer, _, _Time) -> +handle_error(Reason, _Req, [$C|_], _Peer, _, _Time) -> {error, Reason}. %% handle_request/3 @@ -1089,7 +1166,7 @@ handle_error(Reason, _Req, ?CLIENT, _Peer, _, _Time) -> %% Note that diameter will set Result-Code and Failed-AVPs if %% #diameter_packet.errors is non-null. -handle_request(#diameter_packet{header = H, msg = M}, ?SERVER, {_Ref, Caps}) -> +handle_request(#diameter_packet{header = H, msg = M}, _, {_Ref, Caps}) -> #diameter_header{end_to_end_id = EI, hop_by_hop_id = HI} = H, diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index 587ae08b3d..c00bac26bb 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -1,8 +1,6 @@ -#-*-makefile-*- ; force emacs to enter makefile-mode - # %CopyrightBegin% # -# Copyright Ericsson AB 2010-2014. All Rights Reserved. +# Copyright Ericsson AB 2010-2015. All Rights Reserved. # # The contents of this file are subject to the Erlang Public License, # Version 1.1, (the "License"); you may not use this file except in @@ -18,5 +16,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 1.8 +DIAMETER_VSN = 1.9 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/edoc/doc/overview.edoc b/lib/edoc/doc/overview.edoc index 0ced8cab32..3639bb43a5 100644 --- a/lib/edoc/doc/overview.edoc +++ b/lib/edoc/doc/overview.edoc @@ -78,7 +78,7 @@ The following are the main functions for running EDoc: typical Erlang application.</li> <li>{@link edoc:files/2}: Creates documentation for a specified set of source files.</li> - <li>{@link edoc:run/3}: General interface function; the common + <li>{@link edoc:run/2}: General interface function; the common back-end for the above functions. Options are documented here.</li> </ul> diff --git a/lib/edoc/src/edoc.erl b/lib/edoc/src/edoc.erl index 78915e8943..88e7ab5346 100644 --- a/lib/edoc/src/edoc.erl +++ b/lib/edoc/src/edoc.erl @@ -313,7 +313,7 @@ opt_negations() -> %% INHERIT-OPTIONS: target_dir_info/5 %% INHERIT-OPTIONS: edoc_lib:find_sources/2 %% INHERIT-OPTIONS: edoc_lib:run_doclet/2 -%% INHERIT-OPTIONS: edoc_lib:get_doc_env/4 +%% INHERIT-OPTIONS: edoc_lib:get_doc_env/3 run(Files, Opts0) -> Opts = expand_opts(Opts0), @@ -421,7 +421,7 @@ toc(Dir, Opts) -> %% INHERIT-OPTIONS: init_context/1 %% INHERIT-OPTIONS: edoc_lib:run_doclet/2 -%% INHERIT-OPTIONS: edoc_lib:get_doc_env/4 +%% INHERIT-OPTIONS: edoc_lib:get_doc_env/3 toc(Dir, Paths, Opts0) -> Opts = expand_opts(Opts0 ++ [{dir, Dir}]), @@ -769,7 +769,7 @@ get_doc(File) -> %% </dl> %% %% See {@link read_source/2}, {@link read_comments/2} and {@link -%% edoc_lib:get_doc_env/4} for further options. +%% edoc_lib:get_doc_env/3} for further options. %% %% @see get_doc/3 %% @see run/2 @@ -778,7 +778,7 @@ get_doc(File) -> %% @see layout/2 %% INHERIT-OPTIONS: get_doc/3 -%% INHERIT-OPTIONS: edoc_lib:get_doc_env/4 +%% INHERIT-OPTIONS: edoc_lib:get_doc_env/3 get_doc(File, Opts) -> Env = edoc_lib:get_doc_env(Opts), @@ -790,7 +790,7 @@ get_doc(File, Opts) -> %% %% @doc Like {@link get_doc/2}, but for a given environment %% parameter. `Env' is an environment created by {@link -%% edoc_lib:get_doc_env/4}. +%% edoc_lib:get_doc_env/3}. %% INHERIT-OPTIONS: read_source/2, read_comments/2, edoc_extract:source/5 %% DEFER-OPTIONS: get_doc/2 diff --git a/lib/edoc/src/edoc_extract.erl b/lib/edoc/src/edoc_extract.erl index 6d34de3a85..758750083d 100644 --- a/lib/edoc/src/edoc_extract.erl +++ b/lib/edoc/src/edoc_extract.erl @@ -91,7 +91,7 @@ source(Forms, Comments, File, Env, Opts) -> %% type `form_list', or a list of syntax trees representing %% "program forms" (cf. {@link edoc:read_source/2}. %% `Env' is an environment created by {@link -%% edoc_lib:get_doc_env/4}. The `File' argument is used for +%% edoc_lib:get_doc_env/3}. The `File' argument is used for %% error reporting and output file name generation only. %% %% See {@link edoc:get_doc/2} for descriptions of the `def', @@ -222,7 +222,7 @@ add_macro_defs(Defs0, Opts, Env) -> %% %% @doc Reads a text file and returns the list of tags in the file. Any %% lines of text before the first tag are ignored. `Env' is an -%% environment created by {@link edoc_lib:get_doc_env/4}. Upon error, +%% environment created by {@link edoc_lib:get_doc_env/3}. Upon error, %% `Reason' is an atom returned from the call to {@link %% //kernel/file:read_file/1} or the atom 'invalid_unicode'. %% @@ -252,7 +252,7 @@ file(File, Context, Env, Opts) -> %% %% @doc Returns the list of tags in the text. Any lines of text before %% the first tag are ignored. `Env' is an environment created by {@link -%% edoc_lib:get_doc_env/4}. +%% edoc_lib:get_doc_env/3}. %% %% See {@link source/4} for a description of the `def' option. diff --git a/lib/edoc/src/edoc_lib.erl b/lib/edoc/src/edoc_lib.erl index 813fcf2476..c248964dc4 100644 --- a/lib/edoc/src/edoc_lib.erl +++ b/lib/edoc/src/edoc_lib.erl @@ -888,7 +888,7 @@ find_doc_dirs([]) -> %% implies that we use the default app-path. %% NEW-OPTIONS: doc_path -%% DEFER-OPTIONS: get_doc_env/4 +%% DEFER-OPTIONS: get_doc_env/3 get_doc_links(App, Modules, Opts) -> Path = proplists:append_values(doc_path, Opts) ++ find_doc_dirs(), @@ -924,7 +924,7 @@ add_new(K, V, D) -> end. %% @spec (Options::proplist()) -> edoc_env() -%% @equiv get_doc_env([], [], [], Opts) +%% @equiv get_doc_env([], [], Opts) %% @private get_doc_env(Opts) -> @@ -940,7 +940,7 @@ get_doc_env(Opts) -> %% generating references. The data representation is not documented. %% %% @doc Creates an environment data structure used by parts of EDoc for -%% generating references, etc. See {@link edoc:run/3} for a description +%% generating references, etc. See {@link edoc:run/2} for a description %% of the options `file_suffix', `app_default' and `doc_path'. %% %% @see edoc_extract:source/4 @@ -948,7 +948,7 @@ get_doc_env(Opts) -> %% NEW-OPTIONS: file_suffix, app_default %% INHERIT-OPTIONS: get_doc_links/4 -%% DEFER-OPTIONS: edoc:run/3 +%% DEFER-OPTIONS: edoc:run/2 get_doc_env(App, Modules, Opts) -> Suffix = proplists:get_value(file_suffix, Opts, @@ -967,10 +967,10 @@ get_doc_env(App, Modules, Opts) -> %% --------------------------------------------------------------------- %% Plug-in modules -%% @doc See {@link edoc:run/3} for a description of the `doclet' option. +%% @doc See {@link edoc:run/2} for a description of the `doclet' option. %% NEW-OPTIONS: doclet -%% DEFER-OPTIONS: edoc:run/3 +%% DEFER-OPTIONS: edoc:run/2 %% @private run_doclet(Fun, Opts) -> diff --git a/lib/eldap/doc/src/eldap.xml b/lib/eldap/doc/src/eldap.xml index b68115cd82..ed35ee3a9c 100644 --- a/lib/eldap/doc/src/eldap.xml +++ b/lib/eldap/doc/src/eldap.xml @@ -103,7 +103,7 @@ filter() See present/1, substrings/2, <type> <v>Handle = handle()</v> <v>Options = ssl:ssl_options()</v> - <v>Timeout = inifinity | positive_integer()</v> + <v>Timeout = infinity | positive_integer()</v> </type> <desc> <p>Upgrade the connection associated with <c>Handle</c> to a tls connection if possible.</p> diff --git a/lib/eldap/vsn.mk b/lib/eldap/vsn.mk index 432ba2e742..adca41ed63 100644 --- a/lib/eldap/vsn.mk +++ b/lib/eldap/vsn.mk @@ -1 +1 @@ -ELDAP_VSN = 1.1 +ELDAP_VSN = 1.1.1 diff --git a/lib/erl_docgen/priv/css/otp_doc.css b/lib/erl_docgen/priv/css/otp_doc.css index c56de378f4..2aae87a759 100644 --- a/lib/erl_docgen/priv/css/otp_doc.css +++ b/lib/erl_docgen/priv/css/otp_doc.css @@ -66,7 +66,7 @@ a:visited { color: blue; text-decoration: none } span.bold_code { font-family: Courier, monospace; font-weight: bold } span.code { font-family: Courier, monospace; font-weight: normal } -.note, .warning { +.note, .warning, .do, .dont { border: solid black 1px; margin: 1em 3em; } @@ -96,6 +96,32 @@ span.code { font-family: Courier, monospace; font-weight: normal } font-size: 90%; padding: 5px 10px; } +.do .label { + background: #30d42a; + color: white; + font-weight: bold; + padding: 5px 10px; +} +.do .content { + background: #eafeea; + color: black; + line-height: 120%; + font-size: 90%; + padding: 5px 10px; +} +.dont .label { + background: #C00; + color: white; + font-weight: bold; + padding: 5px 10px; +} +.dont .content { + background: #FFF0F0; + color: black; + line-height: 120%; + font-size: 90%; + padding: 5px 10px; +} .example { background-color:#eeeeff; padding: 0px 10px; diff --git a/lib/erl_docgen/priv/dtd/application.dtd b/lib/erl_docgen/priv/dtd/application.dtd index 8a1e8832ec..fcadaced72 100644 --- a/lib/erl_docgen/priv/dtd/application.dtd +++ b/lib/erl_docgen/priv/dtd/application.dtd @@ -24,6 +24,6 @@ %common.header; <!ELEMENT application (header,description?,include+) > -<!ELEMENT description (%block;|quote|br|marker|warning|note)* > +<!ELEMENT description (%block;|quote|br|marker|warning|note|dont|do)* > <!ELEMENT include EMPTY > <!ATTLIST include file CDATA #REQUIRED> diff --git a/lib/erl_docgen/priv/dtd/book.dtd b/lib/erl_docgen/priv/dtd/book.dtd index bb89a6d255..ef723a9eed 100644 --- a/lib/erl_docgen/priv/dtd/book.dtd +++ b/lib/erl_docgen/priv/dtd/book.dtd @@ -38,7 +38,7 @@ <!ELEMENT pagetext (#PCDATA) > <!ELEMENT preamble (contents?,preface?) > -<!ELEMENT preface (title?,(%block;|quote|br|marker|warning|note|table)*) > +<!ELEMENT preface (title?,(%block;|quote|br|marker|warning|note|dont|do|table)*) > <!ELEMENT insidecover (#PCDATA|br|theheader|vfill|vspace|tt|bold| include)* > @@ -67,7 +67,7 @@ <!ELEMENT onepart (title?,description?,include+) > <!ATTLIST onepart lift (yes|no) "no" > -<!ELEMENT description (%block;|quote|br|marker|warning|note)* > +<!ELEMENT description (%block;|quote|br|marker|warning|note|dont|do)* > <!ELEMENT include EMPTY > <!ATTLIST include file CDATA #REQUIRED> diff --git a/lib/erl_docgen/priv/dtd/chapter.dtd b/lib/erl_docgen/priv/dtd/chapter.dtd index eb2c96b04f..4beff6cc54 100644 --- a/lib/erl_docgen/priv/dtd/chapter.dtd +++ b/lib/erl_docgen/priv/dtd/chapter.dtd @@ -29,8 +29,8 @@ <!-- Structure --> -<!ELEMENT chapter (header,(%block;|quote|warning|note|br| +<!ELEMENT chapter (header,(%block;|quote|warning|note|dont|do|br| image|marker|table)*,section+) > <!ELEMENT section (marker*,title, - (%block;|quote|warning|note|br|image|marker| + (%block;|quote|warning|note|dont|do|br|image|marker| table|section)*) > diff --git a/lib/erl_docgen/priv/dtd/common.dtd b/lib/erl_docgen/priv/dtd/common.dtd index f999ef8ea4..92d814e0f1 100644 --- a/lib/erl_docgen/priv/dtd/common.dtd +++ b/lib/erl_docgen/priv/dtd/common.dtd @@ -34,6 +34,8 @@ <!ELEMENT quote (p)* > <!ELEMENT warning (%block;|quote|br|marker)* > <!ELEMENT note (%block;|quote|br|marker)* > +<!ELEMENT dont (%block;|quote|br|marker)* > +<!ELEMENT do (%block;|quote|br|marker)* > <!ELEMENT c (#PCDATA) > <!ELEMENT em (#PCDATA|c)* > diff --git a/lib/erl_docgen/priv/dtd/common.refs.dtd b/lib/erl_docgen/priv/dtd/common.refs.dtd index 93592607df..a08b9e89d4 100644 --- a/lib/erl_docgen/priv/dtd/common.refs.dtd +++ b/lib/erl_docgen/priv/dtd/common.refs.dtd @@ -24,7 +24,7 @@ <!ENTITY % common.header SYSTEM "common.header.dtd" > %common.header; -<!ELEMENT description (%block;|quote|br|marker|warning|note)* > +<!ELEMENT description (%block;|quote|br|marker|warning|note|dont|do)* > <!ELEMENT funcs (func)+ > <!ELEMENT func (name+,type_desc*,fsummary,type?,desc?) > <!-- ELEMENT name is defined in each ref dtd --> @@ -34,12 +34,12 @@ name_i CDATA #IMPLIED> <!ELEMENT v (#PCDATA) > <!ELEMENT d (#PCDATA|c|em)* > -<!ELEMENT desc (%block;|quote|br|marker|warning|note|anno)* > +<!ELEMENT desc (%block;|quote|br|marker|warning|note|dont|do|anno)* > <!ELEMENT authors (aname,email)+ > <!ELEMENT aname (#PCDATA) > <!ELEMENT email (#PCDATA) > <!ELEMENT section (marker*,title,(%block;|quote|br|marker| - warning|note)*) > + warning|note|dont|do)*) > <!ELEMENT datatypes (datatype)+ > <!ELEMENT datatype (name+,desc?) > <!ELEMENT type_desc (#PCDATA) > diff --git a/lib/erl_docgen/priv/dtd/part.dtd b/lib/erl_docgen/priv/dtd/part.dtd index 3f97199042..79f68c415d 100644 --- a/lib/erl_docgen/priv/dtd/part.dtd +++ b/lib/erl_docgen/priv/dtd/part.dtd @@ -24,6 +24,6 @@ %common.header; <!ELEMENT part (header,description?,include+) > -<!ELEMENT description (%block;|quote|br|marker|warning|note)* > +<!ELEMENT description (%block;|quote|br|marker|warning|note|dont|do)* > <!ELEMENT include EMPTY > <!ATTLIST include file CDATA #REQUIRED> diff --git a/lib/erl_docgen/priv/dtd/report.dtd b/lib/erl_docgen/priv/dtd/report.dtd index 3d07e6e5a7..eb463f8867 100644 --- a/lib/erl_docgen/priv/dtd/report.dtd +++ b/lib/erl_docgen/priv/dtd/report.dtd @@ -48,7 +48,7 @@ <!ELEMENT file (#PCDATA) > <!ELEMENT section (marker*,title, - (%block;|quote|warning|note|br|image|marker| + (%block;|quote|warning|note|dont|do|br|image|marker| table|section)*) > <!ELEMENT p (%inline;|index)* > <!ELEMENT pre (#PCDATA|seealso|url|input)* > @@ -58,6 +58,8 @@ <!ELEMENT quote (p)* > <!ELEMENT warning (%block;|quote|br|image|marker|table)* > <!ELEMENT note (%block;|quote|br|image|marker|table)* > +<!ELEMENT dont (%block;|quote|br|image|marker|table)* > +<!ELEMENT do (%block;|quote|br|image|marker|table)* > <!ELEMENT i (#PCDATA|b|c|em)* > <!ELEMENT b (#PCDATA|i|c|em)* > <!ELEMENT c (#PCDATA) > diff --git a/lib/erl_docgen/priv/xsl/db_html.xsl b/lib/erl_docgen/priv/xsl/db_html.xsl index ab5f24c406..3b390f48fb 100644 --- a/lib/erl_docgen/priv/xsl/db_html.xsl +++ b/lib/erl_docgen/priv/xsl/db_html.xsl @@ -952,6 +952,36 @@ </div> </xsl:template> + <!-- Do --> + <xsl:template match="do"> + <xsl:param name="chapnum"/> + <div class="do"> + <div class="label">Do</div> + <div class="content"> + <p> + <xsl:apply-templates> + <xsl:with-param name="chapnum" select="$chapnum"/> + </xsl:apply-templates> + </p> + </div> + </div> + </xsl:template> + + <!-- Dont --> + <xsl:template match="dont"> + <xsl:param name="chapnum"/> + <div class="dont"> + <div class="label">Don't</div> + <div class="content"> + <p> + <xsl:apply-templates> + <xsl:with-param name="chapnum" select="$chapnum"/> + </xsl:apply-templates> + </p> + </div> + </div> + </xsl:template> + <!-- Paragraph --> <xsl:template match="p"> <p> diff --git a/lib/erl_docgen/priv/xsl/db_man.xsl b/lib/erl_docgen/priv/xsl/db_man.xsl index 3bcdd11c35..0caaba560f 100644 --- a/lib/erl_docgen/priv/xsl/db_man.xsl +++ b/lib/erl_docgen/priv/xsl/db_man.xsl @@ -543,7 +543,29 @@ <xsl:text> </xsl:text> </xsl:template> - <xsl:template match="warning/p | note/p"> + <!-- Do --> + <xsl:template match="do"> + <xsl:text> .LP </xsl:text> + <xsl:text> .RS -4</xsl:text> + <xsl:text> .B </xsl:text> + <xsl:text>Do:</xsl:text> + <xsl:text> .RE</xsl:text> + <xsl:apply-templates/> + <xsl:text> </xsl:text> + </xsl:template> + + <!-- Dont --> + <xsl:template match="dont"> + <xsl:text> .LP </xsl:text> + <xsl:text> .RS -4</xsl:text> + <xsl:text> .B </xsl:text> + <xsl:text>Dont:</xsl:text> + <xsl:text> .RE</xsl:text> + <xsl:apply-templates/> + <xsl:text> </xsl:text> + </xsl:template> + + <xsl:template match="warning/p | note/p | dont/p | do/p"> <xsl:variable name="content"> <xsl:text> </xsl:text> <xsl:apply-templates/> diff --git a/lib/erl_docgen/priv/xsl/db_pdf.xsl b/lib/erl_docgen/priv/xsl/db_pdf.xsl index c15b16eb5b..8e7ffddefa 100644 --- a/lib/erl_docgen/priv/xsl/db_pdf.xsl +++ b/lib/erl_docgen/priv/xsl/db_pdf.xsl @@ -1138,6 +1138,31 @@ </fo:block> </xsl:template> + <!-- Do --> + <xsl:template match="do"> + <xsl:param name="partnum"/> + <fo:block xsl:use-attribute-sets="do"> + <fo:block xsl:use-attribute-sets="note-warning-title"> + <xsl:text>Do:</xsl:text> + </fo:block> + <xsl:apply-templates> + <xsl:with-param name="partnum" select="$partnum"/> + </xsl:apply-templates> + </fo:block> + </xsl:template> + + <!-- Dont --> + <xsl:template match="dont"> + <xsl:param name="partnum"/> + <fo:block xsl:use-attribute-sets="dont"> + <fo:block xsl:use-attribute-sets="note-warning-title"> + <xsl:text>Don't:</xsl:text> + </fo:block> + <xsl:apply-templates> + <xsl:with-param name="partnum" select="$partnum"/> + </xsl:apply-templates> + </fo:block> + </xsl:template> <!-- Paragraph --> <xsl:template match="p"> diff --git a/lib/erl_docgen/priv/xsl/db_pdf_params.xsl b/lib/erl_docgen/priv/xsl/db_pdf_params.xsl index 2e3b22acf4..c2d9fb4320 100644 --- a/lib/erl_docgen/priv/xsl/db_pdf_params.xsl +++ b/lib/erl_docgen/priv/xsl/db_pdf_params.xsl @@ -289,6 +289,33 @@ <xsl:attribute name="keep-together.within-page">always</xsl:attribute> </xsl:attribute-set> +<xsl:attribute-set name="do"> + <xsl:attribute name="background-color">#d0fed0</xsl:attribute> + <xsl:attribute name="space-after">1em</xsl:attribute> + <xsl:attribute name="space-before">2em</xsl:attribute> + <xsl:attribute name="text-align">justify</xsl:attribute> + <xsl:attribute name="padding-before">1em</xsl:attribute> + <xsl:attribute name="padding-after">0.3em</xsl:attribute> + <xsl:attribute name="padding-left">0.5em</xsl:attribute> + <xsl:attribute name="padding-right">0.5em</xsl:attribute> + <xsl:attribute name="margin-left">0.5em</xsl:attribute> + <xsl:attribute name="margin-right">0.5em</xsl:attribute> + <xsl:attribute name="keep-together.within-page">always</xsl:attribute> + </xsl:attribute-set> + +<xsl:attribute-set name="dont"> + <xsl:attribute name="background-color">#ffd6d6</xsl:attribute> + <xsl:attribute name="space-after">1em</xsl:attribute> + <xsl:attribute name="space-before">2em</xsl:attribute> + <xsl:attribute name="text-align">justify</xsl:attribute> + <xsl:attribute name="padding-before">1em</xsl:attribute> + <xsl:attribute name="padding-after">0.3em</xsl:attribute> + <xsl:attribute name="padding-left">0.5em</xsl:attribute> + <xsl:attribute name="padding-right">0.5em</xsl:attribute> + <xsl:attribute name="margin-left">0.5em</xsl:attribute> + <xsl:attribute name="margin-right">0.5em</xsl:attribute> + <xsl:attribute name="keep-together.within-page">always</xsl:attribute> + </xsl:attribute-set> <xsl:attribute-set name="note-warning-title"> <xsl:attribute name="font-size">1.33em</xsl:attribute> diff --git a/lib/hipe/main/hipe.app.src b/lib/hipe/main/hipe.app.src index e81212d4dc..22ea71b4e6 100644 --- a/lib/hipe/main/hipe.app.src +++ b/lib/hipe/main/hipe.app.src @@ -224,4 +224,4 @@ {applications, [kernel,stdlib]}, {env, []}, {runtime_dependencies, ["syntax_tools-1.6.14","stdlib-2.0","kernel-3.0", - "erts-6.0","compiler-5.0"]}]}. + "erts-7.0","compiler-5.0"]}]}. diff --git a/lib/hipe/test/maps_SUITE_data/maps_map_sort_literals.erl b/lib/hipe/test/maps_SUITE_data/maps_map_sort_literals.erl index 31abf15d49..ccacbfe5c8 100644 --- a/lib/hipe/test/maps_SUITE_data/maps_map_sort_literals.erl +++ b/lib/hipe/test/maps_SUITE_data/maps_map_sort_literals.erl @@ -10,23 +10,25 @@ test() -> false = #{ c => 1, b => 1, a => 1} < id(#{ c => 1, a => 1}), %% key order - true = id(#{ a => 1 }) < id(#{ b => 1}), - false = id(#{ b => 1 }) < id(#{ a => 1}), - true = id(#{ a => 1, b => 1, c => 1 }) < id(#{ b => 1, c => 1, d => 1}), - true = id(#{ b => 1, c => 1, d => 1 }) > id(#{ a => 1, b => 1, c => 1}), - true = id(#{ c => 1, b => 1, a => 1 }) < id(#{ b => 1, c => 1, d => 1}), - true = id(#{ "a" => 1 }) < id(#{ <<"a">> => 1}), - false = id(#{ <<"a">> => 1 }) < id(#{ "a" => 1}), - false = id(#{ 1 => 1 }) < id(#{ 1.0 => 1}), - false = id(#{ 1.0 => 1 }) < id(#{ 1 => 1}), + true = #{ a => 1 } < id(#{ b => 1}), + false = #{ b => 1 } < id(#{ a => 1}), + true = #{ a => 1, b => 1, c => 1 } < id(#{ b => 1, c => 1, d => 1}), + true = #{ b => 1, c => 1, d => 1 } > id(#{ a => 1, b => 1, c => 1}), + true = #{ c => 1, b => 1, a => 1 } < id(#{ b => 1, c => 1, d => 1}), + true = #{ "a" => 1 } < id(#{ <<"a">> => 1}), + false = #{ <<"a">> => 1 } < id(#{ "a" => 1}), + true = #{ 1 => 1 } < id(#{ 1.0 => 1}), + false = #{ 1.0 => 1 } < id(#{ 1 => 1}), %% value order - true = id(#{ a => 1 }) < id(#{ a => 2}), - false = id(#{ a => 2 }) < id(#{ a => 1}), - false = id(#{ a => 2, b => 1 }) < id(#{ a => 1, b => 3}), - true = id(#{ a => 1, b => 1 }) < id(#{ a => 1, b => 3}), + true = #{ a => 1 } < id(#{ a => 2}), + false = #{ a => 2 } < id(#{ a => 1}), + false = #{ a => 2, b => 1 } < id(#{ a => 1, b => 3}), + true = #{ a => 1, b => 1 } < id(#{ a => 1, b => 3}), + false = #{ a => 1 } < id(#{ a => 1.0}), + false = #{ a => 1.0 } < id(#{ a => 1}), - true = id(#{ "a" => "hi", b => 134 }) == id(#{ b => 134,"a" => "hi"}), + true = #{ "a" => "hi", b => 134 } == id(#{ b => 134,"a" => "hi"}), %% lists:sort @@ -34,7 +36,6 @@ test() -> [#{1:=ok},#{a:=ok},#{"a":=ok},#{<<"a">>:=ok}] = lists:sort([#{"a"=>ok},#{a=>ok},#{1=>ok},#{<<"a">>=>ok}]), [#{1:=3},#{a:=2},#{"a":=1},#{<<"a">>:=4}] = lists:sort(SortVs), [#{1:=3},#{a:=2},#{"a":=1},#{<<"a">>:=4}] = lists:sort(lists:reverse(SortVs)), - ok. %% Use this function to avoid compile-time evaluation of an expression. diff --git a/lib/hipe/tools/hipe_timer.erl b/lib/hipe/tools/hipe_timer.erl index 03cc358f17..5f44bc066d 100644 --- a/lib/hipe/tools/hipe_timer.erl +++ b/lib/hipe/tools/hipe_timer.erl @@ -46,27 +46,27 @@ tr(F) -> {R,{WT-EWT,(RT-ERT)/1000}}. empty_time() -> - {WT1,WT2,WT3} = erlang:now(), + WTA = erlang:monotonic_time(), {A,_} = erlang:statistics(runtime), - {WT12,WT22,WT32} = erlang:now(), + WTB = erlang:monotonic_time(), {B,_} = erlang:statistics(runtime), - {(WT12-WT1)*1000000+(WT22-WT2)+(WT32-WT3)/1000000,B-A}. + {(WTB-WTA)/erlang:convert_time_unit(1, seconds, native),B-A}. time(F) -> - {WT1,WT2,WT3} = erlang:now(), + WTA = erlang:monotonic_time(), {A,_} = erlang:statistics(runtime), F(), - {WT12,WT22,WT32} = erlang:now(), + WTB = erlang:monotonic_time(), {B,_} = erlang:statistics(runtime), - {(WT12-WT1)*1000000+(WT22-WT2)+(WT32-WT3)/1000000,B-A}. + {(WTB-WTA)/erlang:convert_time_unit(1, seconds, native),B-A}. timer(F) -> - {WT1,WT2,WT3} = erlang:now(), + WTA = erlang:monotonic_time(), {A,_} = erlang:statistics(runtime), R = F(), - {WT12,WT22,WT32} = erlang:now(), + WTB = erlang:monotonic_time(), {B,_} = erlang:statistics(runtime), - {R,{(WT12-WT1)*1000000+(WT22-WT2)+(WT32-WT3)/1000000,B-A}}. + {R,{(WTB-WTA)/erlang:convert_time_unit(1, seconds, native),B-A}}. advanced(_Fun, I) when I < 2 -> false; advanced(Fun, Iterations) -> diff --git a/lib/inets/examples/httpd_load_test/hdlt_random_html.erl b/lib/inets/examples/httpd_load_test/hdlt_random_html.erl index e3a572c61f..59073a3d23 100644 --- a/lib/inets/examples/httpd_load_test/hdlt_random_html.erl +++ b/lib/inets/examples/httpd_load_test/hdlt_random_html.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010. All Rights Reserved. +%% Copyright Ericsson AB 2010-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -48,7 +48,10 @@ stop() -> ". content(WorkSim, SzSim) -> - {A, B, C} = now(), + {A, B, C} = {erlang:phash2([node()]), + inets_time_compat:monotonic_time(), + inets_time_compat:unique_integer()}, + random:seed(A, B, C), lists:sort([random:uniform(X) || X <- lists:seq(1, WorkSim)]), lists:flatten(lists:duplicate(SzSim, "Dummy data ")). diff --git a/lib/inets/src/ftp/ftp.erl b/lib/inets/src/ftp/ftp.erl index 8e51b1be5a..7eebe8d5bf 100644 --- a/lib/inets/src/ftp/ftp.erl +++ b/lib/inets/src/ftp/ftp.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1997-2013. All Rights Reserved. +%% Copyright Ericsson AB 1997-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -2176,16 +2176,16 @@ handle_caller(#state{caller = {transfer_data, {Cmd, Bin, RemoteFile}}} = %% Connect to FTP server at Host (default is TCP port 21) %% in order to establish a control connection. setup_ctrl_connection(Host, Port, Timeout, State) -> - MsTime = millisec_time(), + MsTime = inets_time_compat:monotonic_time(), case connect(Host, Port, Timeout, State) of {ok, IpFam, CSock} -> NewState = State#state{csock = {tcp, CSock}, ipfamily = IpFam}, activate_ctrl_connection(NewState), - case Timeout - (millisec_time() - MsTime) of + case Timeout - inets_lib:millisec_passed(MsTime) of Timeout2 when (Timeout2 >= 0) -> {ok, NewState#state{caller = open}, Timeout2}; _ -> - %% Oups: Simulate timeout + %% Oups: Simulate timeout {ok, NewState#state{caller = open}, 0} end; Error -> @@ -2501,10 +2501,6 @@ progress_report(Report, #state{progress = ProgressPid}) -> ftp_progress:report(ProgressPid, Report). -millisec_time() -> - {A,B,C} = erlang:now(), - A*1000000000+B*1000+(C div 1000). - peername({tcp, Socket}) -> inet:peername(Socket); peername({ssl, Socket}) -> ssl:peername(Socket). diff --git a/lib/inets/src/http_client/httpc_cookie.erl b/lib/inets/src/http_client/httpc_cookie.erl index ed306a84f5..35778d3ed5 100644 --- a/lib/inets/src/http_client/httpc_cookie.erl +++ b/lib/inets/src/http_client/httpc_cookie.erl @@ -115,8 +115,8 @@ maybe_dets_close(Db) -> %%-------------------------------------------------------------------- -%% Func: insert(CookieDb) -> ok -%% Purpose: Close the cookie db +%% Func: insert(CookieDb, Cookie) -> ok +%% Purpose: insert cookies into the cookie db %%-------------------------------------------------------------------- %% If no persistent cookie database is defined we diff --git a/lib/inets/src/http_client/httpc_handler.erl b/lib/inets/src/http_client/httpc_handler.erl index 7f7328f1d9..8f2f11ce8e 100644 --- a/lib/inets/src/http_client/httpc_handler.erl +++ b/lib/inets/src/http_client/httpc_handler.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2002-2014. All Rights Reserved. +%% Copyright Ericsson AB 2002-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -1122,7 +1122,7 @@ handle_http_body(Body, #state{headers = Headers, handle_response(State#state{headers = NewHeaders, body = NewBody}); _ -> - {NewBody2, NewRequest} = + {NewBody2, _NewRequest} = stream(NewBody, Request, Code), handle_response(State#state{headers = NewHeaders, body = NewBody2}) diff --git a/lib/inets/src/inets_app/Makefile b/lib/inets/src/inets_app/Makefile index 22426eee79..926585f198 100644 --- a/lib/inets/src/inets_app/Makefile +++ b/lib/inets/src/inets_app/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2005-2012. All Rights Reserved. +# Copyright Ericsson AB 2005-2015. All Rights Reserved. # # The contents of this file are subject to the Erlang Public License, # Version 1.1, (the "License"); you may not use this file except in @@ -47,7 +47,9 @@ MODULES = \ inets_app \ inets_sup \ inets_regexp \ - inets_trace + inets_trace \ + inets_lib \ + inets_time_compat INTERNAL_HRL_FILES = inets_internal.hrl EXTERNAL_HRL_FILES = ../../include/httpd.hrl \ diff --git a/lib/inets/src/inets_app/inets.app.src b/lib/inets/src/inets_app/inets.app.src index 9eae962d03..7207672b7f 100644 --- a/lib/inets/src/inets_app/inets.app.src +++ b/lib/inets/src/inets_app/inets.app.src @@ -1,7 +1,7 @@ %% This is an -*- erlang -*- file. %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1997-2014. All Rights Reserved. +%% Copyright Ericsson AB 1997-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -26,7 +26,9 @@ inets_app, inets_service, inets_regexp, - inets_trace, + inets_trace, + inets_lib, + inets_time_compat, %% FTP ftp, diff --git a/lib/inets/src/inets_app/inets_lib.erl b/lib/inets/src/inets_app/inets_lib.erl new file mode 100644 index 0000000000..e79959f678 --- /dev/null +++ b/lib/inets/src/inets_app/inets_lib.erl @@ -0,0 +1,49 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2015-2015. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +-module(inets_lib). + +-export([millisec_passed/1, formated_timestamp/0, format_timestamp/1]). + + + +%% Help function, elapsed milliseconds since T0 +millisec_passed({_,_,_} = T0 ) -> + %% OTP 17 and earlier + timer:now_diff(inets_time_compat:monotonic_time(), T0) div 1000; + +millisec_passed(T0) -> + %% OTP 18 + erlang:convert_time_unit(erlang:monotonic_time() - T0, + native, + micro_seconds) div 1000. + +%% Return formated time stamp (e.g. 2015:03:16 10:05:23 1234) +formated_timestamp() -> + format_timestamp( os:timestamp() ). + +%% Return formated time stamp (e.g. 2015:03:16 10:05:23 1234) +format_timestamp({_N1, _N2, N3} = Tme) -> + {Date, Time} = calendar:now_to_datetime(Tme), + {YYYY,MM,DD} = Date, + {Hour,Min,Sec} = Time, + FormatDate = + io_lib:format("~.4w:~.2.0w:~.2.0w ~.2.0w:~.2.0w:~.2.0w 4~w", + [YYYY,MM,DD,Hour,Min,Sec,round(N3/1000)]), + lists:flatten(FormatDate). diff --git a/lib/inets/src/inets_app/inets_time_compat.erl b/lib/inets/src/inets_app/inets_time_compat.erl new file mode 100644 index 0000000000..d6297d9caf --- /dev/null +++ b/lib/inets/src/inets_app/inets_time_compat.erl @@ -0,0 +1,71 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2015-2015. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +%% This module is created to be able to execute on ERTS versions both +%% earlier and later than 7.0. + +-module(inets_time_compat). + +%% We don't want warnings about the use of erlang:now/0 in +%% this module. +-compile(nowarn_deprecated_function). + +-export([monotonic_time/0, + timestamp/0, + unique_integer/0, + unique_integer/1]). + +monotonic_time() -> + try + erlang:monotonic_time() + catch + error:undef -> + %% Use Erlang system time as monotonic time + erlang_system_time_fallback() + end. + +timestamp() -> + try + erlang:timestamp() + catch + error:undef -> + erlang:now() + end. + +unique_integer() -> + try + erlang:unique_integer() + catch + error:undef -> + erlang_system_time_fallback() + end. + +unique_integer(Modifiers) -> + try + erlang:unique_integer(Modifiers) + catch + error:badarg -> + erlang:error(badarg, [Modifiers]); + error:undef -> + erlang_system_time_fallback() + end. + +erlang_system_time_fallback() -> + {MS, S, US} = erlang:now(), + (MS*1000000+S)*1000000+US. diff --git a/lib/inets/src/inets_app/inets_trace.erl b/lib/inets/src/inets_app/inets_trace.erl index 8911f65897..cb6d6d8bdb 100644 --- a/lib/inets/src/inets_app/inets_trace.erl +++ b/lib/inets/src/inets_app/inets_trace.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2011-2012. All Rights Reserved. +%% Copyright Ericsson AB 2011-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -228,21 +228,24 @@ handle_trace({trace_ts, _Who, call, [_Sev, "stop trace", stop_trace, [stop_trace]]}, Timestamp}, {_, standard_io} = Fd) -> - (catch io:format(standard_io, "stop trace at ~s~n", [format_timestamp(Timestamp)])), + (catch io:format(standard_io, "stop trace at ~s~n", + [inets_lib:format_timestamp(Timestamp)])), Fd; handle_trace({trace_ts, _Who, call, {?MODULE, report_event, [_Sev, "stop trace", stop_trace, [stop_trace]]}, Timestamp}, standard_io = Fd) -> - (catch io:format(Fd, "stop trace at ~s~n", [format_timestamp(Timestamp)])), + (catch io:format(Fd, "stop trace at ~s~n", + [inets_lib:format_timestamp(Timestamp)])), Fd; handle_trace({trace_ts, _Who, call, {?MODULE, report_event, [_Sev, "stop trace", stop_trace, [stop_trace]]}, Timestamp}, {_Service, Fd}) -> - (catch io:format(Fd, "stop trace at ~s~n", [format_timestamp(Timestamp)])), + (catch io:format(Fd, "stop trace at ~s~n", + [inets_lib:format_timestamp(Timestamp)])), (catch file:close(Fd)), closed_file; handle_trace({trace_ts, _Who, call, @@ -250,7 +253,8 @@ handle_trace({trace_ts, _Who, call, [_Sev, "stop trace", stop_trace, [stop_trace]]}, Timestamp}, Fd) -> - (catch io:format(Fd, "stop trace at ~s~n", [format_timestamp(Timestamp)])), + (catch io:format(Fd, "stop trace at ~s~n", + [inets_lib:format_timestamp(Timestamp)])), (catch file:close(Fd)), closed_file; handle_trace({trace_ts, Who, call, @@ -280,7 +284,7 @@ print_inets_trace(Fd, Sev, Timestamp, Who, Label, Service, Content) -> do_print_inets_trace(Fd, Sev, Timestamp, Who, Label, Service, Content). do_print_inets_trace(Fd, Sev, Timestamp, Who, Label, Service, Content) -> - Ts = format_timestamp(Timestamp), + Ts = inets_lib:format_timestamp(Timestamp), io:format(Fd, "[inets ~w trace ~w ~w ~s] ~s " "~n Content: ~p" "~n", @@ -307,7 +311,7 @@ do_print_trace(Fd, {trace, Who, What, Where, Extra}) -> "~n", [Who, What, Where, Extra]); do_print_trace(Fd, {trace_ts, Who, What, Where, When}) -> - Ts = format_timestamp(When), + Ts = inets_lib:format_timestamp(When), io:format(Fd, "[trace ~s]" "~n Who: ~p" "~n What: ~p" @@ -315,7 +319,7 @@ do_print_trace(Fd, {trace_ts, Who, What, Where, When}) -> "~n", [Ts, Who, What, Where]); do_print_trace(Fd, {trace_ts, Who, What, Where, Extra, When}) -> - Ts = format_timestamp(When), + Ts = inets_lib:format_timestamp(When), io:format(Fd, "[trace ~s]" "~n Who: ~p" "~n What: ~p" @@ -330,7 +334,7 @@ do_print_trace(Fd, {seq_trace, What, Where}) -> "~n", [What, Where]); do_print_trace(Fd, {seq_trace, What, Where, When}) -> - Ts = format_timestamp(When), + Ts = inets_lib:format_timestamp(When), io:format(Fd, "[seq trace ~s]" "~n What: ~p" "~n Where: ~p" @@ -345,13 +349,3 @@ do_print_trace(Fd, Trace) -> "~n", [Trace]). -format_timestamp({_N1, _N2, N3} = Now) -> - {Date, Time} = calendar:now_to_datetime(Now), - {YYYY,MM,DD} = Date, - {Hour,Min,Sec} = Time, - FormatDate = - io_lib:format("~.4w:~.2.0w:~.2.0w ~.2.0w:~.2.0w:~.2.0w 4~w", - [YYYY,MM,DD,Hour,Min,Sec,round(N3/1000)]), - lists:flatten(FormatDate). - - diff --git a/lib/inets/src/tftp/tftp_logger.erl b/lib/inets/src/tftp/tftp_logger.erl index 0c3620e665..231a705371 100644 --- a/lib/inets/src/tftp/tftp_logger.erl +++ b/lib/inets/src/tftp/tftp_logger.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2009. All Rights Reserved. +%% Copyright Ericsson AB 2008-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -84,8 +84,8 @@ info_msg(Format, Data) -> %%------------------------------------------------------------------- add_timestamp(Format, Data) -> - Now = {_MegaSecs, _Secs, _MicroSecs} = erlang:now(), - {{_Y, _Mo, _D}, {H, Mi, S}} = calendar:now_to_universal_time(Now), + Time = inets_time_compat:timestamp(), + {{_Y, _Mo, _D}, {H, Mi, S}} = calendar:now_to_universal_time(Time), %% {"~p-~s-~sT~s:~s:~sZ,~6.6.0w tftp: " ++ Format ++ "\n", %% [Y, t(Mo), t(D), t(H), t(Mi), t(S), MicroSecs | Data]}. {"~s:~s:~s tftp: " ++ Format, [t(H), t(Mi), t(S) | Data]}. diff --git a/lib/inets/src/tftp/tftp_sup.erl b/lib/inets/src/tftp/tftp_sup.erl index 1cafcc1069..7a0dcffc90 100644 --- a/lib/inets/src/tftp/tftp_sup.erl +++ b/lib/inets/src/tftp/tftp_sup.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2005-2009. All Rights Reserved. +%% Copyright Ericsson AB 2005-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -93,7 +93,7 @@ unique_name(Options) -> {value, {_, Port}} when is_integer(Port), Port > 0 -> {tftpd, Port}; _ -> - {tftpd, erlang:now()} + {tftpd, inets_time_compat:unique_integer([positive])} end. default_kill_after() -> diff --git a/lib/inets/test/ftp_suite_lib.erl b/lib/inets/test/ftp_suite_lib.erl index daee1bdcdc..b637832101 100644 --- a/lib/inets/test/ftp_suite_lib.erl +++ b/lib/inets/test/ftp_suite_lib.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2005-2013. All Rights Reserved. +%% Copyright Ericsson AB 2005-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -1352,9 +1352,9 @@ do_delete(Pid, Config) -> ok. do_mkdir(Pid) -> - {A, B, C} = erlang:now(), - NewDir = "nisse_" ++ integer_to_list(A) ++ "_" ++ - integer_to_list(B) ++ "_" ++ integer_to_list(C), + NewDir = "earl_" ++ + integer_to_list(inets_time_compat:unique_integer([positive])), + ok = ftp:cd(Pid, "incoming"), {ok, CurrDir} = ftp:pwd(Pid), {error, efnamena} = ftp:mkdir(Pid, NewDir++"\r\nCWD ."), diff --git a/lib/inets/test/httpc_SUITE.erl b/lib/inets/test/httpc_SUITE.erl index 0e89e831fb..0dfc65e8f7 100644 --- a/lib/inets/test/httpc_SUITE.erl +++ b/lib/inets/test/httpc_SUITE.erl @@ -1933,12 +1933,13 @@ run_clients(NumClients, ServerPort, SeqNumServer) -> wait4clients([], _Timeout) -> ok; wait4clients(Clients, Timeout) when Timeout > 0 -> - Time = now_ms(), + Time = inets_time_compat:monotonic_time(), + receive {'DOWN', _MRef, process, Pid, normal} -> {value, {Id, _, _}} = lists:keysearch(Pid, 2, Clients), NewClients = lists:keydelete(Id, 1, Clients), - wait4clients(NewClients, Timeout - (now_ms() - Time)); + wait4clients(NewClients, Timeout - inets_lib:millisec_passed(Time)); {'DOWN', _MRef, process, Pid, Reason} -> {value, {Id, _, _}} = lists:keysearch(Pid, 2, Clients), ct:fail({bad_client_termination, Id, Reason}) @@ -2031,14 +2032,10 @@ parse_connection_type(Request) -> "keep-alive" -> keep_alive end. -%% Time in milli seconds -now_ms() -> - {A,B,C} = erlang:now(), - A*1000000000+B*1000+(C div 1000). - set_random_seed() -> - {_, _, Micros} = now(), - A = erlang:phash2([make_ref(), self(), Micros]), + Unique = inets_time_compat:unique_integer(), + + A = erlang:phash2([make_ref(), self(), Unique]), random:seed(A, A, A). diff --git a/lib/inets/test/httpd_time_test.erl b/lib/inets/test/httpd_time_test.erl index 0bb457f9b9..7dd61a5517 100644 --- a/lib/inets/test/httpd_time_test.erl +++ b/lib/inets/test/httpd_time_test.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2001-2011. All Rights Reserved. +%% Copyright Ericsson AB 2001-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -116,13 +116,14 @@ main(N, SocketType, Host, Port, Time) loop(Pollers, Timeout) -> d("loop -> entry when" "~n Timeout: ~p", [Timeout]), - Start = t(), + Start = inets_time_compat:monotonic_time(), + receive {'EXIT', Pid, {poller_stat_failure, SocketType, Host, Port, Time, Reason}} -> case is_poller(Pid, Pollers) of true -> error_msg("received unexpected exit from poller ~p~n" - "befor completion of test " + "before completion of test " "after ~p micro sec" "~n SocketType: ~p" "~n Host: ~p" @@ -133,7 +134,7 @@ loop(Pollers, Timeout) -> false -> error_msg("received unexpected ~p from ~p" "befor completion of test", [Reason, Pid]), - loop(Pollers, to(Timeout, Start)) + loop(Pollers, Timeout - inets_lib:millisec_passed(Start)) end; {poller_stat_failure, Pid, {SocketType, Host, Port, Time, Reason}} -> @@ -412,35 +413,6 @@ validate(ExpStatusCode, _SocketType, _Socket, Response) -> end. -trash_the_rest(Socket, N) -> - receive - {ssl, Socket, Trash} -> - trash_the_rest(Socket, add(N,sz(Trash))); - {ssl_closed, Socket} -> - N; - {ssl_error, Socket, Error} -> - exit({connection_error, Error}); - - {tcp, Socket, Trash} -> - trash_the_rest(Socket, add(N,sz(Trash))); - {tcp_closed, Socket} -> - N; - {tcp_error, Socket, Error} -> - exit({connection_error, Error}) - - after 10000 -> - exit({connection_timed_out, N}) - end. - - -add(N1,N2) when is_integer(N1) andalso is_integer(N2) -> - N1 + N2; -add(N1,_) when is_integer(N1) -> - N1; -add(_,N2) when is_integer(N2) -> - N2. - - sz(L) when is_list(L) -> length(lists:flatten(L)); sz(B) when is_binary(B) -> @@ -505,17 +477,6 @@ status_to_message(Code) -> io_lib:format("Unknown status code: ~p",[Code]). %% ---------------------------------------------------------------- -to(To, Start) -> - To - (t() - Start). - -%% Time in milli seconds -t() -> - {A,B,C} = erlang:now(), - A*1000000000+B*1000+(C div 1000). - - -%% ---------------------------------------------------------------- - % close(Socket) -> diff --git a/lib/inets/test/inets_SUITE.erl b/lib/inets/test/inets_SUITE.erl index 6510c70d08..a07dc79c02 100644 --- a/lib/inets/test/inets_SUITE.erl +++ b/lib/inets/test/inets_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1997-2013. All Rights Reserved. +%% Copyright Ericsson AB 1997-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -549,25 +549,12 @@ tsf(Reason) -> tsp(F) -> tsp(F, []). tsp(F, A) -> - Timestamp = formated_timestamp(), + Timestamp = inets_lib:formated_timestamp(), test_server:format("** ~s ** ~p ~p:" ++ F ++ "~n", [Timestamp, self(), ?MODULE | A]). i(F) -> i(F, []). i(F, A) -> - Timestamp = formated_timestamp(), + Timestamp = inets_lib:formated_timestamp(), io:format("*** ~s ~w:" ++ F ++ "~n", [Timestamp, ?MODULE | A]). - -formated_timestamp() -> - format_timestamp( os:timestamp() ). - -format_timestamp({_N1, _N2, N3} = Now) -> - {Date, Time} = calendar:now_to_datetime(Now), - {YYYY,MM,DD} = Date, - {Hour,Min,Sec} = Time, - FormatDate = - io_lib:format("~.4w:~.2.0w:~.2.0w ~.2.0w:~.2.0w:~.2.0w 4~w", - [YYYY,MM,DD,Hour,Min,Sec,round(N3/1000)]), - lists:flatten(FormatDate). - diff --git a/lib/inets/test/inets_app_test.erl b/lib/inets/test/inets_app_test.erl index eabfa69f7c..22d6e25c87 100644 --- a/lib/inets/test/inets_app_test.erl +++ b/lib/inets/test/inets_app_test.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2002-2012. All Rights Reserved. +%% Copyright Ericsson AB 2002-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -32,19 +32,6 @@ %% Test server callbacks -init_per_testcase(undef_funcs, Config) -> - NewConfig = lists:keydelete(watchdog, 1, Config), - Dog = test_server:timetrap(inets_test_lib:minutes(10)), - - %% We need to check if there is a point to run this test. - %% On some platforms, crypto will not build, which in turn - %% causes ssl to not build (at this time, this will - %% change in the future). - %% So, we first check if we can start crypto, and if not, - %% we skip this test case! - ?ENSURE_STARTED(crypto), - - [{watchdog, Dog}| NewConfig]; init_per_testcase(_, Config) -> Config. @@ -54,7 +41,7 @@ end_per_testcase(_Case, Config) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% all() -> - [fields, modules, exportall, app_depend, undef_funcs]. + [fields, modules, exportall, app_depend]. groups() -> []. @@ -244,56 +231,6 @@ check_apps([App|Apps]) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -undef_funcs(suite) -> - []; -undef_funcs(doc) -> - []; -undef_funcs(Config) when is_list(Config) -> - App = inets, - AppFile = key1search(app_file, Config), - Mods = key1search(modules, AppFile), - Root = code:root_dir(), - LibDir = code:lib_dir(App), - EbinDir = filename:join([LibDir,"ebin"]), - XRefTestName = undef_funcs_make_name(App, xref_test_name), - {ok, XRef} = xref:start(XRefTestName), - ok = xref:set_default(XRef, - [{verbose,false},{warnings,false}]), - XRefName = undef_funcs_make_name(App, xref_name), - {ok, XRefName} = xref:add_release(XRef, Root, {name, XRefName}), - {ok, App} = xref:replace_application(XRef, App, EbinDir), - {ok, Undefs} = xref:analyze(XRef, undefined_function_calls), - xref:stop(XRef), - analyze_undefined_function_calls(Undefs, Mods, []). - -analyze_undefined_function_calls([], _, []) -> - ok; -analyze_undefined_function_calls([], _, AppUndefs) -> - exit({suite_failed, {undefined_function_calls, AppUndefs}}); -analyze_undefined_function_calls([{{Mod, _F, _A}, _C} = AppUndef|Undefs], - AppModules, AppUndefs) -> - %% Check that this module is our's - case lists:member(Mod,AppModules) of - true -> - {Calling,Called} = AppUndef, - {Mod1,Func1,Ar1} = Calling, - {Mod2,Func2,Ar2} = Called, - io:format("undefined function call: " - "~n ~w:~w/~w calls ~w:~w/~w~n", - [Mod1,Func1,Ar1,Mod2,Func2,Ar2]), - analyze_undefined_function_calls(Undefs, AppModules, - [AppUndef|AppUndefs]); - false -> - io:format("dropping ~p~n", [Mod]), - analyze_undefined_function_calls(Undefs, AppModules, AppUndefs) - end. - -%% This function is used simply to avoid cut-and-paste errors later... -undef_funcs_make_name(App, PostFix) -> - list_to_atom(atom_to_list(App) ++ "_" ++ atom_to_list(PostFix)). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - fail(Reason) -> exit({suite_failed, Reason}). diff --git a/lib/inets/test/inets_test_lib.erl b/lib/inets/test/inets_test_lib.erl index 4be9d9c8b3..7485971d3e 100644 --- a/lib/inets/test/inets_test_lib.erl +++ b/lib/inets/test/inets_test_lib.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2001-2013. All Rights Reserved. +%% Copyright Ericsson AB 2001-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -498,13 +498,6 @@ close(essl,Socket) -> close(ip_comm,Socket) -> catch gen_tcp:close(Socket). -millis() -> - erlang:now(). - -millis_diff(A,B) -> - T1 = (element(1,A)*1000000) + element(2,A) + (element(3,A)/1000000), - T2 = (element(1,B)*1000000) + element(2,B) + (element(3,B)/1000000), - T1 - T2. hours(N) -> trunc(N * 1000 * 60 * 60). minutes(N) -> trunc(N * 1000 * 60). @@ -546,7 +539,7 @@ flush() -> tsp(F) -> tsp(F, []). tsp(F, A) -> - Timestamp = formated_timestamp(), + Timestamp = inets_lib:formated_timestamp(), ct:pal("*** ~s ~p ~p " ++ F ++ "~n", [Timestamp, node(), self() | A]). @@ -559,18 +552,6 @@ tss(Time) -> timestamp() -> http_util:timestamp(). -formated_timestamp() -> - format_timestamp( os:timestamp() ). - -format_timestamp({_N1, _N2, N3} = Now) -> - {Date, Time} = calendar:now_to_datetime(Now), - {YYYY,MM,DD} = Date, - {Hour,Min,Sec} = Time, - FormatDate = - io_lib:format("~.4w:~.2.0w:~.2.0w ~.2.0w:~.2.0w:~.2.0w 4~w", - [YYYY,MM,DD,Hour,Min,Sec,round(N3/1000)]), - lists:flatten(FormatDate). - start_apps(Apps) -> lists:foreach(fun(App) -> application:stop(App), diff --git a/lib/kernel/doc/src/os.xml b/lib/kernel/doc/src/os.xml index 8b85f24455..b9dbede0d3 100644 --- a/lib/kernel/doc/src/os.xml +++ b/lib/kernel/doc/src/os.xml @@ -142,14 +142,49 @@ DirOut = os:cmd("dir"), % on Win32 platform</code> </desc> </func> <func> + <name name="system_time" arity="0"/> + <fsummary>Current OS system time</fsummary> + <desc> + <p>Returns current + <seealso marker="erts:time_correction#OS_System_Time">OS system time</seealso> + in <c>native</c> + <seealso marker="erts:erlang#type_time_unit">time unit</seealso>.</p> + + <note><p>This time is <em>not</em> a monotonically increasing time.</p></note> + </desc> + </func> + <func> + <name name="system_time" arity="1"/> + <fsummary>Current OS system time</fsummary> + <desc> + <p>Returns current + <seealso marker="erts:time_correction#OS_System_Time">OS system time</seealso> + converted into the <c><anno>Unit</anno></c> passed as argument.</p> + + <p>Calling <c>os:system_time(<anno>Unit</anno>)</c> is equivalent to: + <seealso marker="erts:erlang#convert_time_unit/3"><c>erlang:convert_time_unit</c></seealso><c>(</c><seealso marker="#system_time/0"><c>os:system_time()</c></seealso><c>, + native, <anno>Unit</anno>)</c>.</p> + + <note><p>This time is <em>not</em> a monotonically increasing time.</p></note> + </desc> + </func> + <func> <name name="timestamp" arity="0"/> <type_desc variable="Timestamp">Timestamp = {MegaSecs, Secs, MicroSecs}</type_desc> - <fsummary>Returna a timestamp from the OS in the erlang:now/0 format</fsummary> + <fsummary>Current OS system time on the erlang:timestamp/0 format</fsummary> <desc> - <p>Returns a tuple in the same format as <seealso marker="erts:erlang#now/0">erlang:now/0</seealso>. The difference is that this function returns what the operating system thinks (a.k.a. the wall clock time) without any attempts at time correction. The result of two different calls to this function is <em>not</em> guaranteed to be different.</p> - <p>The most obvious use for this function is logging. The tuple can be used together with the function <seealso marker="stdlib:calendar#now_to_universal_time/1">calendar:now_to_universal_time/1</seealso> -or <seealso marker="stdlib:calendar#now_to_local_time/1">calendar:now_to_local_time/1</seealso> to get calendar time. Using the calendar time together with the <c>MicroSecs</c> part of the return tuple from this function allows you to log timestamps in high resolution and consistent with the time in the rest of the operating system.</p> - <p>Example of code formatting a string in the format "DD Mon YYYY HH:MM:SS.mmmmmm", where DD is the day of month, Mon is the textual month name, YYYY is the year, HH:MM:SS is the time and mmmmmm is the microseconds in six positions:</p> + <p>Returns current + <seealso marker="erts:time_correction#OS_System_Time">OS system time</seealso> + in the same format as <seealso marker="erts:erlang#timestamp/0">erlang:timestamp/0</seealso>. + The tuple can be used together with the function + <seealso marker="stdlib:calendar#now_to_universal_time/1">calendar:now_to_universal_time/1</seealso> + or <seealso marker="stdlib:calendar#now_to_local_time/1">calendar:now_to_local_time/1</seealso> to + get calendar time. Using the calendar time together with the <c>MicroSecs</c> part of the return + tuple from this function allows you to log timestamps in high resolution and consistent with the + time in the rest of the operating system.</p> + <p>Example of code formatting a string in the format "DD Mon YYYY HH:MM:SS.mmmmmm", where + DD is the day of month, Mon is the textual month name, YYYY is the year, HH:MM:SS is the time and + mmmmmm is the microseconds in six positions:</p> <code> -module(print_time). -export([format_utc_timestamp/0]). @@ -168,6 +203,9 @@ format_utc_timestamp() -> 1> <input>io:format("~s~n",[print_time:format_utc_timestamp()]).</input> 29 Apr 2009 9:55:30.051711 </pre> + <p>OS system time can also be retreived by + <c><seealso marker="#system_time/0"><c>os:system_time/0</c></seealso></c>, + and <seealso marker="#system_time/1"><c>os:system_time/1</c></seealso>.</p> </desc> </func> <func> diff --git a/lib/kernel/src/auth.erl b/lib/kernel/src/auth.erl index eda35147d3..dbc486bee1 100644 --- a/lib/kernel/src/auth.erl +++ b/lib/kernel/src/auth.erl @@ -370,8 +370,8 @@ check_cookie1([], Result) -> %% Creates a new, random cookie. create_cookie(Name) -> - {_, S1, S2} = now(), - Seed = S2*10000+S1, + Seed = abs(erlang:monotonic_time() + bxor erlang:unique_integer()), Cookie = random_cookie(20, Seed, []), case file:open(Name, [write, raw]) of {ok, File} -> diff --git a/lib/kernel/src/dist_util.erl b/lib/kernel/src/dist_util.erl index b127fe2e33..6b510bd0c3 100644 --- a/lib/kernel/src/dist_util.erl +++ b/lib/kernel/src/dist_util.erl @@ -298,7 +298,7 @@ shutdown(_Module, _Line, _Data, Reason) -> exit(Reason). %% Use this line to debug connection. %% Set net_kernel verbose = 1 as well. -%% exit({Reason, ?MODULE, _Line, _Data, erlang:now()}). +%% exit({Reason, ?MODULE, _Line, _Data, erlang:timestamp()}). flush_down() -> @@ -373,7 +373,9 @@ gen_digest(Challenge, Cookie) when is_integer(Challenge), is_atom(Cookie) -> %% gen_challenge() returns a "random" number %% --------------------------------------------------------------- gen_challenge() -> - {A,B,C} = erlang:now(), + A = erlang:phash2([erlang:node()]), + B = erlang:monotonic_time(), + C = erlang:unique_integer(), {D,_} = erlang:statistics(reductions), {E,_} = erlang:statistics(runtime), {F,_} = erlang:statistics(wall_clock), diff --git a/lib/kernel/src/erl_distribution.erl b/lib/kernel/src/erl_distribution.erl index 25ad34357a..3c4429129e 100644 --- a/lib/kernel/src/erl_distribution.erl +++ b/lib/kernel/src/erl_distribution.erl @@ -22,7 +22,6 @@ -export([start_link/0,start_link/1,init/1,start/1,stop/0]). -%-define(DBG,io:format("~p:~p~n",[?MODULE,?LINE])). -define(DBG,erlang:display([?MODULE,?LINE])). start_link() -> diff --git a/lib/kernel/src/erts_debug.erl b/lib/kernel/src/erts_debug.erl index ef605d0bfe..17bee06b5e 100644 --- a/lib/kernel/src/erts_debug.erl +++ b/lib/kernel/src/erts_debug.erl @@ -33,7 +33,7 @@ -export([breakpoint/2, disassemble/1, display/1, dist_ext_to_term/2, dump_monitors/1, dump_links/1, flat_size/1, get_internal_state/1, instructions/0, lock_counters/1, - same/2, set_internal_state/2]). + map_info/1, same/2, set_internal_state/2]). -spec breakpoint(MFA, Flag) -> non_neg_integer() when MFA :: {Module :: module(), @@ -164,8 +164,10 @@ set_internal_state(_, _) -> -spec size(term()) -> non_neg_integer(). +-record(s, {seen, maps}). + size(Term) -> - {Sum,_} = size(Term, gb_trees:empty(), 0), + {Sum,_} = size(Term, #s{seen=gb_trees:empty(),maps=[]}, 0), Sum. size([H|T]=Term, Seen0, Sum0) -> @@ -209,10 +211,24 @@ tuple_size(I, Sz, Tuple, Seen0, Sum0) -> tuple_size(I+1, Sz, Tuple, Seen, Sum). map_size(Map,Seen0,Sum0) -> - Kt = erts_internal:map_to_tuple_keys(Map), - Vs = maps:values(Map), - {Sum1,Seen1} = size(Kt,Seen0,Sum0), - fold_size(Vs,Seen1,Sum1+length(Vs)+3). + %% Danger: + %% The internal nodes from erts_internal:map_hashmap_children/1 + %% is not allowed to leak anywhere. They are only allowed in + %% containers (cons cells and tuples, not maps), in gc and + %% in erts_debug:same/2 + case erts_internal:map_type(Map) of + flatmap -> + Kt = erts_internal:map_to_tuple_keys(Map), + Vs = maps:values(Map), + {Sum1,Seen1} = size(Kt,Seen0,Sum0), + fold_size(Vs,Seen1,Sum1+length(Vs)+3); + hashmap -> + Cs = erts_internal:map_hashmap_children(Map), + fold_size(Cs,Seen0,Sum0+length(Cs)+2); + hashmap_node -> + Cs = erts_internal:map_hashmap_children(Map), + fold_size(Cs,Seen0,Sum0+length(Cs)+1) + end. fun_size(Fun, Seen, Sum) -> case erlang:fun_info(Fun, type) of @@ -229,13 +245,18 @@ fold_size([H|T], Seen0, Sum0) -> fold_size(T, Seen, Sum); fold_size([], Seen, Sum) -> {Sum,Seen}. -remember_term(Term, Seen) -> - case gb_trees:lookup(Term, Seen) of - none -> gb_trees:insert(Term, [Term], Seen); +remember_term(Term, #s{maps=Ms}=S) when is_map(Term) -> + case is_term_seen(Term, Ms) of + false -> S#s{maps=[Term|Ms]}; + true -> seen + end; +remember_term(Term, #s{seen=T}=S) -> + case gb_trees:lookup(Term,T) of + none -> S#s{seen=gb_trees:insert(Term,[Term],T)}; {value,Terms} -> case is_term_seen(Term, Terms) of - false -> gb_trees:update(Term, [Term|Terms], Seen); - true -> seen + false -> S#s{seen=gb_trees:update(Term,[Term|Terms],T)}; + true -> seen end end. @@ -313,3 +334,9 @@ cont_dis(File, {Addr,Str,MFA}, MFA) -> io:put_chars(File, binary_to_list(Str)), cont_dis(File, erts_debug:disassemble(Addr), MFA); cont_dis(_, {_,_,_}, _) -> ok. + +-spec map_info(Map) -> list() when + Map :: map(). + +map_info(_) -> + erlang:nif_error(undef). diff --git a/lib/kernel/src/global.erl b/lib/kernel/src/global.erl index 0a4edea452..6c36d417a2 100644 --- a/lib/kernel/src/global.erl +++ b/lib/kernel/src/global.erl @@ -881,11 +881,12 @@ handle_info({nodeup, Node}, S0) when S0#state.connect_all -> false -> resend_pre_connect(Node), - %% now() is used as a tag to separate different synch sessions + %% erlang:unique_integer([monotonic]) is used as a tag to + %% separate different synch sessions %% from each others. Global could be confused at bursty nodeups %% because it couldn't separate the messages between the different %% synch sessions started by a nodeup. - MyTag = now(), + MyTag = erlang:unique_integer([monotonic]), put({sync_tag_my, Node}, MyTag), ?trace({sending_nodeup_to_locker, {node,Node},{mytag,MyTag}}), S1#state.the_locker ! {nodeup, Node, MyTag}, @@ -1772,8 +1773,8 @@ update_locker_known(Upd, S) -> S#multi{known = Known, the_boss = TheBoss}. random_element(L) -> - {A,B,C} = now(), - E = (A+B+C) rem length(L), + E = abs(erlang:monotonic_time() + bxor erlang:unique_integer()) rem length(L), lists:nth(E+1, L). exclude_known(Others, Known) -> @@ -2072,9 +2073,10 @@ random_sleep(Times) -> end, case get(random_seed) of undefined -> - {A1, A2, A3} = now(), - _ = random:seed(A1, A2, A3 + erlang:phash(node(), 100000)), - ok; + _ = random:seed(erlang:phash2([erlang:node()]), + erlang:monotonic_time(), + erlang:unique_integer()), + ok; _ -> ok end, %% First time 1/4 seconds, then doubling each time up to 8 seconds max. @@ -2106,7 +2108,7 @@ trace_message(S, M, X) -> S#state{trace = [trace_message(M, X) | S#state.trace]}. trace_message(M, X) -> - {node(), now(), M, nodes(), X}. + {node(), erlang:timestamp(), M, nodes(), X}. %%----------------------------------------------------------------- %% Each sync process corresponds to one call to sync. Each such diff --git a/lib/kernel/src/inet_db.erl b/lib/kernel/src/inet_db.erl index 2ebdc0f554..abe207295f 100644 --- a/lib/kernel/src/inet_db.erl +++ b/lib/kernel/src/inet_db.erl @@ -1372,8 +1372,7 @@ cache_rr(_Db, Cache, RR) -> ets:insert(Cache, RR). times() -> - {Mega,Secs,_} = erlang:now(), - Mega*1000000 + Secs. + erlang:monotonic_time(1). %% lookup and remove old entries diff --git a/lib/kernel/src/inet_res.erl b/lib/kernel/src/inet_res.erl index 6037da1d22..410128a16a 100644 --- a/lib/kernel/src/inet_res.erl +++ b/lib/kernel/src/inet_res.erl @@ -715,10 +715,10 @@ udp_send(#sock{inet=I}, {A,B,C,D}=IP, Port, Buffer) udp_recv(#sock{inet6=I}, {A,B,C,D,E,F,G,H}=IP, Port, Timeout, Decode) when ?ip6(A,B,C,D,E,F,G,H), ?port(Port) -> - do_udp_recv(I, IP, Port, Timeout, Decode, erlang:now(), Timeout); + do_udp_recv(I, IP, Port, Timeout, Decode, time_now(), Timeout); udp_recv(#sock{inet=I}, {A,B,C,D}=IP, Port, Timeout, Decode) when ?ip(A,B,C,D), ?port(Port) -> - do_udp_recv(I, IP, Port, Timeout, Decode, erlang:now(), Timeout). + do_udp_recv(I, IP, Port, Timeout, Decode, time_now(), Timeout). do_udp_recv(_I, _IP, _Port, 0, _Decode, _Start, _T) -> timeout; @@ -742,7 +742,7 @@ do_udp_recv(I, IP, Port, Timeout, Decode, Start, T) -> NewTimeout = erlang:max(0, Timeout - 50), do_udp_recv(I, IP, Port, NewTimeout, Decode, Start, T); false -> - Now = erlang:now(), + Now = time_now(), NewT = erlang:max(0, Timeout - now_ms(Now, Start)), do_udp_recv(I, IP, Port, Timeout, Decode, Start, NewT); Result -> @@ -1057,5 +1057,9 @@ dns_msg(Msg) -> end. -compile({inline, [now_ms/2]}). -now_ms({Meg1,Sec1,Mic1}, {Meg0,Sec0,Mic0}) -> - ((Meg1-Meg0)*1000000 + (Sec1-Sec0))*1000 + ((Mic1-Mic0) div 1000). +now_ms(Int1, Int0) -> + Int1 - Int0. + +-compile({inline, [time_now/0]}). +time_now() -> + erlang:monotonic_time(1000). diff --git a/lib/kernel/src/kernel.app.src b/lib/kernel/src/kernel.app.src index 9f6c0f4624..0cb10791d7 100644 --- a/lib/kernel/src/kernel.app.src +++ b/lib/kernel/src/kernel.app.src @@ -115,6 +115,6 @@ {applications, []}, {env, [{error_logger, tty}]}, {mod, {kernel, []}}, - {runtime_dependencies, ["erts-6.1.2", "stdlib-2.0", "sasl-2.4"]} + {runtime_dependencies, ["erts-7.0", "stdlib-2.0", "sasl-2.4"]} ] }. diff --git a/lib/kernel/src/os.erl b/lib/kernel/src/os.erl index 7468a06f3c..3647545777 100644 --- a/lib/kernel/src/os.erl +++ b/lib/kernel/src/os.erl @@ -26,7 +26,8 @@ %%% BIFs --export([getenv/0, getenv/1, getenv/2, getpid/0, putenv/2, timestamp/0, unsetenv/1]). +-export([getenv/0, getenv/1, getenv/2, getpid/0, putenv/2, system_time/0, system_time/1, + timestamp/0, unsetenv/1]). -spec getenv() -> [string()]. @@ -65,6 +66,17 @@ getpid() -> putenv(_, _) -> erlang:nif_error(undef). +-spec system_time() -> integer(). + +system_time() -> + erlang:nif_error(undef). + +-spec system_time(Unit) -> integer() when + Unit :: erlang:time_unit(). + +system_time(_Unit) -> + erlang:nif_error(undef). + -spec timestamp() -> Timestamp when Timestamp :: erlang:timestamp(). diff --git a/lib/kernel/src/pg2.erl b/lib/kernel/src/pg2.erl index b562d4ffd2..70d7a75671 100644 --- a/lib/kernel/src/pg2.erl +++ b/lib/kernel/src/pg2.erl @@ -140,19 +140,22 @@ get_closest_pid(Name) -> [Pid] -> Pid; [] -> - {_,_,X} = erlang:now(), case get_members(Name) of [] -> {error, {no_process, Name}}; Members -> - lists:nth((X rem length(Members))+1, Members) + random_element(Members) end; Members when is_list(Members) -> - {_,_,X} = erlang:now(), - lists:nth((X rem length(Members))+1, Members); + random_element(Members); Else -> Else end. +random_element(List) -> + X = abs(erlang:monotonic_time() + bxor erlang:unique_integer()), + lists:nth((X rem length(List)) + 1, List). + %%% %%% Callback functions from gen_server %%% diff --git a/lib/kernel/test/inet_SUITE.erl b/lib/kernel/test/inet_SUITE.erl index 849013ac79..44a32fc1ec 100644 --- a/lib/kernel/test/inet_SUITE.erl +++ b/lib/kernel/test/inet_SUITE.erl @@ -88,10 +88,30 @@ init_per_group(_GroupName, Config) -> end_per_group(_GroupName, Config) -> Config. +init_per_testcase(lookup_bad_search_option, Config) -> + Db = inet_db, + Key = res_lookup, + %% The bad option can not enter through inet_db:set_lookup/1, + %% but through e.g .inetrc. + Prev = ets:lookup(Db, Key), + ets:delete(Db, Key), + ets:insert(Db, {Key,[lookup_bad_search_option]}), + ?t:format("Misconfigured resolver lookup order", []), + Dog = test_server:timetrap(test_server:seconds(60)), + [{Key,Prev},{watchdog,Dog}|Config]; init_per_testcase(_Func, Config) -> Dog = test_server:timetrap(test_server:seconds(60)), [{watchdog,Dog}|Config]. +end_per_testcase(lookup_bad_search_option, Config) -> + Dog = ?config(watchdog, Config), + test_server:timetrap_cancel(Dog), + Db = inet_db, + Key = res_lookup, + Prev = ?config(Key, Config), + ets:delete(Db, Key), + ets:insert(Db, Prev), + ?t:format("Restored resolver lookup order", []); end_per_testcase(_Func, Config) -> Dog = ?config(watchdog, Config), test_server:timetrap_cancel(Dog). @@ -915,10 +935,8 @@ lookup_bad_search_option(suite) -> lookup_bad_search_option(doc) -> ["Test lookup with erroneously configured lookup option (OTP-12133)"]; lookup_bad_search_option(Config) when is_list(Config) -> - Db = inet_db, - %% The bad option can not enter through inet_db:set_lookup/1, - %% but through e.g .inetrc. - ets:insert(Db, {res_lookup,[lookup_bad_search_option]}), + %% Manipulation of resolver config is done in init_per_testcase + %% and end_per_testcase to ensure cleanup. {ok,Hostname} = inet:gethostname(), {ok,_Hent} = inet:gethostbyname(Hostname), % Will hang loop for this bug ok. diff --git a/lib/kernel/test/zlib_SUITE.erl b/lib/kernel/test/zlib_SUITE.erl index e99151284f..41c19fce51 100644 --- a/lib/kernel/test/zlib_SUITE.erl +++ b/lib/kernel/test/zlib_SUITE.erl @@ -146,8 +146,6 @@ api_deflateInit(Config) when is_list(Config) -> ?m(?BARG, zlib:deflateInit(Z1,default,deflated,-20,8,default)), ?m(?BARG, zlib:deflateInit(Z1,default,deflated,-7,8,default)), ?m(?BARG, zlib:deflateInit(Z1,default,deflated,7,8,default)), - ?m(?BARG, zlib:deflateInit(Z1,default,deflated,-8,8,default)), - ?m(?BARG, zlib:deflateInit(Z1,default,deflated,8,8,default)), ?m(?BARG, zlib:deflateInit(Z1,default,deflated,-15,0,default)), ?m(?BARG, zlib:deflateInit(Z1,default,deflated,-15,10,default)), @@ -169,7 +167,7 @@ api_deflateInit(Config) when is_list(Config) -> ?m(ok, zlib:deflateInit(Z12,default,deflated,-Wbits,8,default)), ?m(ok,zlib:close(Z11)), ?m(ok,zlib:close(Z12)) - end, lists:seq(9, 15)), + end, lists:seq(8, 15)), lists:foreach(fun(MemLevel) -> ?line Z = zlib:open(), @@ -277,7 +275,7 @@ api_inflateInit(Config) when is_list(Config) -> ?m(ok, zlib:inflateInit(Z12,-Wbits)), ?m(ok,zlib:close(Z11)), ?m(ok,zlib:close(Z12)) - end, lists:seq(9,15)), + end, lists:seq(8,15)), ?m(?BARG, zlib:inflateInit(gurka, -15)), ?m(?BARG, zlib:inflateInit(Z1, 7)), ?m(?BARG, zlib:inflateInit(Z1, -7)), diff --git a/lib/mnesia/src/mnesia.app.src b/lib/mnesia/src/mnesia.app.src index e755864792..c78a7cba1e 100644 --- a/lib/mnesia/src/mnesia.app.src +++ b/lib/mnesia/src/mnesia.app.src @@ -48,6 +48,6 @@ ]}, {applications, [kernel, stdlib]}, {mod, {mnesia_sup, []}}, - {runtime_dependencies, ["stdlib-2.0","kernel-3.0","erts-6.0"]}]}. + {runtime_dependencies, ["stdlib-2.0","kernel-3.0","erts-7.0"]}]}. diff --git a/lib/mnesia/src/mnesia.erl b/lib/mnesia/src/mnesia.erl index 8f14831ad3..792b6c4a73 100644 --- a/lib/mnesia/src/mnesia.erl +++ b/lib/mnesia/src/mnesia.erl @@ -2171,7 +2171,7 @@ system_info2(version) -> Version; false -> %% Ensure that it does not match - {mnesia_not_loaded, node(), now()} + {mnesia_not_loaded, node(), erlang:timestamp()} end; Version -> Version diff --git a/lib/mnesia/src/mnesia.hrl b/lib/mnesia/src/mnesia.hrl index c8010d5466..9a58ea581a 100644 --- a/lib/mnesia/src/mnesia.hrl +++ b/lib/mnesia/src/mnesia.hrl @@ -53,7 +53,9 @@ up_stores = [], %% list of upper layer stores for nested trans level = 1}). %% transaction level --define(unique_cookie, {erlang:now(), node()}). +-define(unique_cookie, {{erlang:monotonic_time() + erlang:time_offset(), + erlang:unique_integer(),1}, + node()}). -record(cstruct, {name, % Atom type = set, % set | bag diff --git a/lib/mnesia/src/mnesia_bup.erl b/lib/mnesia/src/mnesia_bup.erl index 3b084e7371..71e610bd63 100644 --- a/lib/mnesia/src/mnesia_bup.erl +++ b/lib/mnesia/src/mnesia_bup.erl @@ -368,7 +368,7 @@ create_schema(_Ns, Reason) -> {error, Reason}. mk_str() -> - Now = [integer_to_list(I) || I <- tuple_to_list(now())], + Now = integer_to_list(erlang:unique_integer([positive])), lists:concat([node()] ++ Now ++ ".TMP"). make_initial_backup(Ns, Opaque, Mod) -> diff --git a/lib/mnesia/src/mnesia_checkpoint.erl b/lib/mnesia/src/mnesia_checkpoint.erl index 173e3be2f5..442e68a13d 100644 --- a/lib/mnesia/src/mnesia_checkpoint.erl +++ b/lib/mnesia/src/mnesia_checkpoint.erl @@ -68,12 +68,12 @@ -import(mnesia_lib, [add/2, del/2, set/2, unset/1]). -import(mnesia_lib, [dbg_out/2]). --record(checkpoint_args, {name = {now(), node()}, +-record(checkpoint_args, {name = {erlang:unique_integer([positive]), node()}, allow_remote = true, ram_overrides_dump = false, nodes = [], node = node(), - now = now(), + now, %% unused cookie = ?unique_cookie, min = [], max = [], diff --git a/lib/mnesia/src/mnesia_event.erl b/lib/mnesia/src/mnesia_event.erl index 67ec9d7399..8a4be88e9a 100644 --- a/lib/mnesia/src/mnesia_event.erl +++ b/lib/mnesia/src/mnesia_event.erl @@ -235,8 +235,7 @@ report_fatal(Format, Args, BinaryCore, CoreDumped) -> end. core_file(CoreDir,BinaryCore,Format,Args) -> - %% Integers = tuple_to_list(date()) ++ tuple_to_list(time()), - Integers = tuple_to_list(now()), + Integers = tuple_to_list(erlang:timestamp()), Fun = fun(I) when I < 10 -> ["_0",I]; (I) -> ["_",I] end, diff --git a/lib/mnesia/src/mnesia_lib.erl b/lib/mnesia/src/mnesia_lib.erl index a32c69c59e..fc41cbedcb 100644 --- a/lib/mnesia/src/mnesia_lib.erl +++ b/lib/mnesia/src/mnesia_lib.erl @@ -930,8 +930,9 @@ random_time(Retries, _Counter0) -> case get(random_seed) of undefined -> - {X, Y, Z} = erlang:now(), %% time() - _ = random:seed(X, Y, Z), + _ = random:seed(erlang:unique_integer(), + erlang:monotonic_time(), + erlang:unique_integer()), Time = Dup + random:uniform(MaxIntv), %% dbg_out("---random_test rs ~w max ~w val ~w---~n", [Retries, MaxIntv, Time]), Time; diff --git a/lib/mnesia/src/mnesia_log.erl b/lib/mnesia/src/mnesia_log.erl index d2fd04a60b..6206a8bdb9 100644 --- a/lib/mnesia/src/mnesia_log.erl +++ b/lib/mnesia/src/mnesia_log.erl @@ -200,7 +200,7 @@ log_header(Kind, Version) -> log_kind=Kind, mnesia_version=mnesia:system_info(version), node=node(), - now=now()}. + now=erlang:timestamp()}. version() -> "4.3". diff --git a/lib/mnesia/src/mnesia_schema.erl b/lib/mnesia/src/mnesia_schema.erl index 6e43052fb0..52180cfcf5 100644 --- a/lib/mnesia/src/mnesia_schema.erl +++ b/lib/mnesia/src/mnesia_schema.erl @@ -262,7 +262,7 @@ incr_version(Cs) -> [] -> {Major + 1, 0}; % All replicas are active _ -> {Major, Minor + 1} % Some replicas are inactive end, - Cs#cstruct{version = {V, {node(), now()}}}. + Cs#cstruct{version = {V, {node(), erlang:timestamp()}}}. %% Returns table name insert_cstruct(Tid, Cs, KeepWhereabouts) -> diff --git a/lib/mnesia/test/mnesia_trans_access_test.erl b/lib/mnesia/test/mnesia_trans_access_test.erl index 237984978e..f906670296 100644 --- a/lib/mnesia/test/mnesia_trans_access_test.erl +++ b/lib/mnesia/test/mnesia_trans_access_test.erl @@ -930,20 +930,20 @@ index_update_bag(Config)when is_list(Config) -> [IPos] = mnesia_lib:val({Tab,index}), ITab = mnesia_lib:val({index_test,{index, IPos}}), io:format("~n Index ~p @ ~p => ~p ~n~n",[IPos,ITab, ets:tab2list(ITab)]), - ?match([{2,1},{2,2},{12,1}], ets:tab2list(ITab)), + ?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:write(Rec5) end)), {atomic, R60} = mnesia:transaction(fun() -> mnesia:index_read(Tab, 2, ValPos) end), ?match([Rec1,Rec5,Rec2], lists:sort(R60)), - ?match([{2,1},{2,2},{12,1}], ets:tab2list(ITab)), + ?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:delete_object(Rec3) end)), {atomic, R61} = mnesia:transaction(fun() -> mnesia:index_read(Tab, 2, ValPos) end), ?match([Rec1,Rec5,Rec2], lists:sort(R61)), {atomic, R62} = mnesia:transaction(fun() -> mnesia:index_read(Tab,12, ValPos) end), ?match([], lists:sort(R62)), - ?match([{2,1},{2,2}], ets:tab2list(ITab)), + ?match([{2,1},{2,2}], lists:keysort(1,ets:tab2list(ITab))), %% reset for rest of testcase ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:write(Rec3) end)), diff --git a/lib/observer/src/observer.app.src b/lib/observer/src/observer.app.src index ea7950ae52..e293990d64 100644 --- a/lib/observer/src/observer.app.src +++ b/lib/observer/src/observer.app.src @@ -64,6 +64,6 @@ {env, []}, {runtime_dependencies, ["wx-1.2","stdlib-2.0","runtime_tools-1.8.14", "kernel-3.0","inets-5.10","et-1.5", - "erts-6.0"]}]}. + "erts-7.0"]}]}. diff --git a/lib/observer/src/observer_html_lib.erl b/lib/observer/src/observer_html_lib.erl index 53197078cf..df0bc05312 100644 --- a/lib/observer/src/observer_html_lib.erl +++ b/lib/observer/src/observer_html_lib.erl @@ -155,7 +155,7 @@ all_or_expand(_Tab,Term,Str,false) href_proc_port(lists:flatten(Str)); all_or_expand(Tab,Term,Preview,true) when not is_binary(Term) -> - Key = {Key1,Key2,Key3} = now(), + Key = {Key1,Key2,Key3} = {erlang:unique_integer([positive]),1,2}, ets:insert(Tab,{Key,Term}), [href_proc_port(lists:flatten(Preview), false), $\n, href("TARGET=\"expanded\"", diff --git a/lib/observer/src/ttb.erl b/lib/observer/src/ttb.erl index 61fd6d1787..a2db40aa2f 100644 --- a/lib/observer/src/ttb.erl +++ b/lib/observer/src/ttb.erl @@ -849,7 +849,7 @@ get_nodes() -> receive {?MODULE,Nodes} -> Nodes end. ts() -> - {{Y,M,D},{H,Min,S}} = calendar:now_to_local_time(now()), + {{Y,M,D},{H,Min,S}} = calendar:now_to_local_time(erlang:timestamp()), io_lib:format("-~4.4.0w~2.2.0w~2.2.0w-~2.2.0w~2.2.0w~2.2.0w", [Y,M,D,H,Min,S]). diff --git a/lib/os_mon/src/cpu_sup.erl b/lib/os_mon/src/cpu_sup.erl index 1f088ecbde..66e7973e7e 100644 --- a/lib/os_mon/src/cpu_sup.erl +++ b/lib/os_mon/src/cpu_sup.erl @@ -221,7 +221,7 @@ get_uint32_measurement(Request, #internal{port = P, os_type = {unix, sunos}}) -> port_server_call(P, Request); get_uint32_measurement(Request, #internal{os_type = {unix, linux}}) -> {ok,F} = file:open("/proc/loadavg",[read,raw]), - {ok,D} = file:read(F,24), + {ok,D} = file:read_line(F), ok = file:close(F), {ok,[Load1,Load5,Load15,_PRun,PTotal],_} = io_lib:fread("~f ~f ~f ~d/~d", D), case Request of diff --git a/lib/public_key/src/public_key.erl b/lib/public_key/src/public_key.erl index a0a87e5351..e8ff965982 100644 --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -114,7 +114,7 @@ pem_encode(PemEntries) when is_list(PemEntries) -> iolist_to_binary(pubkey_pem:encode(PemEntries)). %%-------------------------------------------------------------------- --spec pem_entry_decode(pem_entry(), [string()]) -> term(). +-spec pem_entry_decode(pem_entry(), string()) -> term(). % %% Description: Decodes a pem entry. pem_decode/1 returns a list of %% pem entries. @@ -146,14 +146,16 @@ pem_entry_decode({Asn1Type, CryptDer, {Cipher, #'PBES2-params'{}}} = PemEntry, pem_entry_decode({Asn1Type, CryptDer, {Cipher, {#'PBEParameter'{},_}}} = PemEntry, Password) when is_atom(Asn1Type) andalso is_binary(CryptDer) andalso - is_list(Cipher) -> + is_list(Cipher) andalso + is_list(Password) -> do_pem_entry_decode(PemEntry, Password); pem_entry_decode({Asn1Type, CryptDer, {Cipher, Salt}} = PemEntry, Password) when is_atom(Asn1Type) andalso is_binary(CryptDer) andalso is_list(Cipher) andalso is_binary(Salt) andalso - ((erlang:byte_size(Salt) == 8) or (erlang:byte_size(Salt) == 16)) -> + ((erlang:byte_size(Salt) == 8) or (erlang:byte_size(Salt) == 16)) andalso + is_list(Password) -> do_pem_entry_decode(PemEntry, Password). @@ -626,8 +628,12 @@ pkix_is_fixed_dh_cert(Cert) when is_binary(Cert) -> % %% Description: Returns the issuer id. %%-------------------------------------------------------------------- -pkix_issuer_id(Cert, Signed)-> - pkix_issuer_id(Cert, Signed, decode). +pkix_issuer_id(#'OTPCertificate'{} = OtpCert, Signed) when (Signed == self) or + (Signed == other) -> + pubkey_cert:issuer_id(OtpCert, Signed); +pkix_issuer_id(Cert, Signed) when is_binary(Cert) -> + OtpCert = pkix_decode_cert(Cert, otp), + pkix_issuer_id(OtpCert, Signed). %%-------------------------------------------------------------------- -spec pkix_crl_issuer(CRL::binary()| #'CertificateList'{}) -> @@ -990,17 +996,3 @@ ec_key({PubKey, PrivateKey}, Params) -> parameters = Params, publicKey = {0, PubKey}}. -pkix_issuer_id(#'OTPCertificate'{} = OtpCert, Signed, decode) when (Signed == self) or - (Signed == other) -> - pubkey_cert:issuer_id(OtpCert, Signed); -pkix_issuer_id(#'OTPCertificate'{} = OtpCert, Signed, encode) when (Signed == self) or - (Signed == other) -> - case pubkey_cert:issuer_id(OtpCert, Signed) of - {ok, {Serial, Issuer}} -> - {ok, {Serial, pubkey_cert_records:transform(Issuer, encode)}}; - Error -> - Error - end; -pkix_issuer_id(Cert, Signed, Decode) when is_binary(Cert) -> - OtpCert = pkix_decode_cert(Cert, otp), - pkix_issuer_id(OtpCert, Signed, Decode). diff --git a/lib/reltool/src/reltool.app.src b/lib/reltool/src/reltool.app.src index 65fcf4aae5..579d2c0d1b 100644 --- a/lib/reltool/src/reltool.app.src +++ b/lib/reltool/src/reltool.app.src @@ -36,5 +36,5 @@ {applications, [stdlib, kernel]}, {env, []}, {runtime_dependencies, ["wx-1.2","tools-2.6.14","stdlib-2.0","sasl-2.4", - "kernel-3.0","erts-6.0"]} + "kernel-3.0","erts-7.0"]} ]}. diff --git a/lib/reltool/src/reltool_fgraph_win.erl b/lib/reltool/src/reltool_fgraph_win.erl index 66bc2b5ab3..d9e6f6d427 100644 --- a/lib/reltool/src/reltool_fgraph_win.erl +++ b/lib/reltool/src/reltool_fgraph_win.erl @@ -252,10 +252,10 @@ ticker_init(Pid) -> ticker_loop(Pid, Time) -> receive after Time -> Pid ! {self(), redraw}, - T0 = now(), + T0 = erlang:monotonic_time(), receive {Pid, ok} -> ok end, - T1 = now(), - D = timer:now_diff(T1, T0)/1000, + T1 = erlang:monotonic_time(), + D = erlang:convert_time_unit(T1-T0, native, milli_seconds), case round(40 - D) of Ms when Ms < 0 -> %io:format("ticker: wait is 0 ms [fg ~7s ms] [fps ~7s]~n", diff --git a/lib/runtime_tools/src/observer_backend.erl b/lib/runtime_tools/src/observer_backend.erl index fea0854042..fe814ceda4 100644 --- a/lib/runtime_tools/src/observer_backend.erl +++ b/lib/runtime_tools/src/observer_backend.erl @@ -248,7 +248,7 @@ etop_collect(Collector) -> SchedulerWallTime = erlang:statistics(scheduler_wall_time), ProcInfo = etop_collect(processes(), []), - Collector ! {self(),#etop_info{now = now(), + Collector ! {self(),#etop_info{now = erlang:timestamp(), n_procs = length(ProcInfo), run_queue = erlang:statistics(run_queue), runtime = SchedulerWallTime, diff --git a/lib/runtime_tools/src/percept_profile.erl b/lib/runtime_tools/src/percept_profile.erl index cdc7a0fca1..dfadb21aa8 100644 --- a/lib/runtime_tools/src/percept_profile.erl +++ b/lib/runtime_tools/src/percept_profile.erl @@ -119,7 +119,7 @@ stop() -> undefined -> {error, not_started}; Port -> - erlang:port_command(Port, erlang:term_to_binary({profile_stop, erlang:now()})), + erlang:port_command(Port, erlang:term_to_binary({profile_stop, erlang:timestamp()})), %% trace delivered? erlang:port_close(Port), ok @@ -139,7 +139,7 @@ profile_to_file(Filename, Opts) -> erlang:system_flag(multi_scheduling, block), Port = (dbg:trace_port(file, Filename))(), % Send start time - erlang:port_command(Port, erlang:term_to_binary({profile_start, erlang:now()})), + erlang:port_command(Port, erlang:term_to_binary({profile_start, erlang:timestamp()})), erlang:system_flag(multi_scheduling, unblock), %% Register Port diff --git a/lib/runtime_tools/src/runtime_tools.app.src b/lib/runtime_tools/src/runtime_tools.app.src index 0a70802c08..32ea9e564b 100644 --- a/lib/runtime_tools/src/runtime_tools.app.src +++ b/lib/runtime_tools/src/runtime_tools.app.src @@ -27,6 +27,6 @@ {env, []}, {mod, {runtime_tools, []}}, {runtime_dependencies, ["stdlib-2.0","mnesia-4.12","kernel-3.0", - "erts-6.0"]}]}. + "erts-7.0"]}]}. diff --git a/lib/ssh/examples/Makefile b/lib/ssh/examples/Makefile index de019f75b5..9280c42076 100644 --- a/lib/ssh/examples/Makefile +++ b/lib/ssh/examples/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2005-2012. All Rights Reserved. +# Copyright Ericsson AB 2005-2015. All Rights Reserved. # # The contents of this file are subject to the Erlang Public License, # Version 1.1, (the "License"); you may not use this file except in @@ -38,7 +38,8 @@ RELSYSDIR = $(RELEASE_PATH)/lib/ssh-$(VSN) MODULES = \ - ssh_sample_cli + ssh_sample_cli \ + ssh_device.erl ERL_FILES= $(MODULES:=.erl) diff --git a/lib/ssh/examples/ssh_device.erl b/lib/ssh/examples/ssh_device.erl new file mode 100644 index 0000000000..f6be812915 --- /dev/null +++ b/lib/ssh/examples/ssh_device.erl @@ -0,0 +1,62 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2005-2015. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +-module(ssh_device). + +%% api +-export([ssh_device/5]). + +%%% I wrote this because of i think a fully ssh client sample will be easy to start the ssh module better than +%%% go though each function file. +ssh_device(Host, Port, User, Pass, Cmd) -> + ssh:start(), + case ssh:connect(Host, Port, + [{user, User}, {password, Pass}, + {silently_accept_hosts, true}, {quiet_mode, true}]) + of + {ok, Conn} -> + {ok, ChannelId} = ssh_connection:session_channel(Conn, + infinity), + ssh_connection:exec(Conn, ChannelId, Cmd, infinity), + Init_rep = <<>>, + wait_for_response(Conn, Host, Init_rep), + ssh:close(Conn); + {error, nxdomain} -> + {error,nxdomain} + end. + +%%-------------------------------------------------------------------- +%%% Internal application API +%%-------------------------------------------------------------------- +wait_for_response(Conn, Host, Acc) -> + receive + {ssh_cm, Conn, Msg} -> + case Msg of + {closed, _ChannelId} -> + {ok,Acc}; + {data, _, _, A} -> + Acc2 = <<Acc/binary, A/binary>>, + wait_for_response(Conn, Host, Acc2); + _ -> + wait_for_response(Conn, Host, Acc) + end + after + 5000 -> + {error,timeout} + end. diff --git a/lib/ssh/src/ssh_info.erl b/lib/ssh/src/ssh_info.erl index e5a8666af0..eae9ded5c6 100644 --- a/lib/ssh/src/ssh_info.erl +++ b/lib/ssh/src/ssh_info.erl @@ -179,14 +179,7 @@ line(Len, Char) -> datetime() -> - %% Adapt to new OTP 18 erlang time API and be back-compatible - TimeStamp = try - erlang:timestamp() - catch - error:undef -> - erlang:now() - end, - {{YYYY,MM,DD}, {H,M,S}} = calendar:now_to_universal_time(TimeStamp), + {{YYYY,MM,DD}, {H,M,S}} = calendar:now_to_universal_time(erlang:timestamp()), lists:flatten(io_lib:format('~4w-~2..0w-~2..0w ~2..0w:~2..0w:~2..0w UTC',[YYYY,MM,DD, H,M,S])). diff --git a/lib/ssh/test/ssh_basic_SUITE.erl b/lib/ssh/test/ssh_basic_SUITE.erl index b449012ffc..4c5498dc0e 100644 --- a/lib/ssh/test/ssh_basic_SUITE.erl +++ b/lib/ssh/test/ssh_basic_SUITE.erl @@ -715,14 +715,7 @@ ssh_connect_arg4_timeout(_Config) -> %% try to connect with a timeout, but "supervise" it Client = spawn(fun() -> - %% Adapt to OTP 18 erlang time API and be back-compatible - T0 = try - erlang:monotonic_time() - catch - error:undef -> - %% Use Erlang system time as monotonic time - erlang:now() - end, + T0 = erlang:monotonic_time(), Rc = ssh:connect("localhost",Port,[],Timeout), ct:log("Client ssh:connect got ~p",[Rc]), Parent ! {done,self(),Rc,T0} @@ -731,7 +724,7 @@ ssh_connect_arg4_timeout(_Config) -> %% Wait for client reaction on the connection try: receive {done, Client, {error,timeout}, T0} -> - Msp = ms_passed(T0), + Msp = ms_passed(T0), exit(Server,hasta_la_vista___baby), Low = 0.9*Timeout, High = 1.1*Timeout, @@ -755,17 +748,12 @@ ssh_connect_arg4_timeout(_Config) -> {fail, "Didn't timeout"} end. - %% Help function, elapsed milliseconds since T0 -ms_passed({_,_,_} = T0 ) -> - %% OTP 17 and earlier - timer:now_diff(erlang:now(), T0)/1000; - ms_passed(T0) -> %% OTP 18 - erlang:convert_time_resolution(erlang:monotonic_time() - T0, - erlang:time_resolution(), - 1000000)/1000. + erlang:convert_time_unit(erlang:monotonic_time() - T0, + native, + micro_seconds) / 1000. %%-------------------------------------------------------------------- ssh_connect_negtimeout_parallel(Config) -> ssh_connect_negtimeout(Config,true). diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index c8cac3e852..bfebe2c60b 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,5 +1,5 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 3.1 +SSH_VSN = 3.1.1 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index c9b02d44ec..47b0dbc206 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -89,12 +89,14 @@ |{dh, der_encoded()} | {dhfile, path()} | {ciphers, ciphers()} | {user_lookup_fun, {fun(), term()}}, {psk_identity, string()}, {srp_identity, {string(), string()}} | {ssl_imp, ssl_imp()} | {reuse_sessions, boolean()} | {reuse_session, fun()} + {alpn_advertised_protocols, [binary()]} | + {alpn_preferred_protocols, [binary()]} | {next_protocols_advertised, [binary()]} | {client_preferred_next_protocols, {client | server, [binary()]} | {client | server, [binary()], binary()}} | {log_alert, boolean()} | {server_name_indication, hostname() | disable} </c></p> - <p><c>transportoption() = {cb_info, {CallbackModule::atom(), DataTag::atom(), ClosedTag::atom(), ErrTag:atom()}} + <p><c>transportoption() = {cb_info, {CallbackModule :: atom(), DataTag :: atom(), ClosedTag :: atom(), ErrTag:atom()}} - defaults to {gen_tcp, tcp, tcp_closed, tcp_error}. Can be used to customize the transport layer. The callback module must implement a reliable transport protocol and behave as gen_tcp and in addition have functions corresponding to @@ -303,20 +305,20 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom() | {revo </taglist> </item> - <tag>{crl_check, boolean() | peer | best_effort )</tag> + <tag>{crl_check, boolean() | peer | best_effort }</tag> <item> Perform CRL (Certificate Revocation List) verification <seealso marker="public_key:public_key#pkix_crl_validate-3"> - public_key:pkix_crls_validate/3</seealso>, during the + (public_key:pkix_crls_validate/3)</seealso> on all the certificates during the path validation <seealso - marker="public_key:public_key#pkix_path_validation-3">public_key:pkix_path_validation/3 </seealso> - invokation on all the certificates in the peer certificate chain. Defaults to - false. - + marker="public_key:public_key#pkix_path_validation-3">(public_key:pkix_path_validation/3) + </seealso> + of the certificate chain. Defaults to false. + <p><c>peer</c> - check is only performed on the peer certificate.</p> - <p><c>best_effort</c> - if certificate revokation status can not be determined + <p><c>best_effort</c> - if certificate revocation status can not be determined it will be accepted as valid.</p> <p>The CA certificates specified for the connection will be used to @@ -326,7 +328,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom() | {revo <seealso marker="ssl:ssl_crl_cache_api">ssl_crl_cache_api(3)</seealso>.</p> </item> - <tag>{crl_cache, {Module::atom, {DbHandle::internal | term(), Args::list()}}</tag> + <tag>{crl_cache, {Module :: atom(), {DbHandle :: internal | term(), Args :: list()}}}</tag> <item> <p>Module defaults to ssl_crl_cache with <c> DbHandle </c> internal and an empty argument list. The following arguments may be specified for the internal cache.</p> @@ -425,7 +427,20 @@ fun(srp, Username :: string(), UserState :: term()) -> certificates are used during server authentication and when building the client certificate chain. </item> - + + <tag>{alpn_advertised_protocols, [binary()]}</tag> + <item> + <p>The list of protocols supported by the client to be sent to the + server to be used for an Application-Layer Protocol Negotiation (ALPN). + If the server supports ALPN then it will choose a protocol from this + list; otherwise it will fail the connection with a "no_application_protocol" + alert. A server that does not support ALPN will ignore this value.</p> + + <p>The list of protocols must not contain an empty binary.</p> + + <p>The negotiated protocol can be retrieved using the <c>negotiated_protocol/1</c> function.</p> + </item> + <tag>{client_preferred_next_protocols, {Precedence :: server | client, ClientPrefs :: [binary()]}}</tag> <tag>{client_preferred_next_protocols, {Precedence :: server | client, ClientPrefs :: [binary()], Default :: binary()}}</tag> <item> @@ -545,12 +560,25 @@ fun(srp, Username :: string(), UserState :: term()) -> and CipherSuite is of type ciphersuite(). </item> + <tag>{alpn_preferred_protocols, [binary()]}</tag> + <item> + <p>Indicates the server will try to perform Application-Layer + Protocol Negotiation (ALPN).</p> + + <p>The list of protocols is in order of preference. The protocol + negotiated will be the first in the list that matches one of the + protocols advertised by the client. If no protocol matches, the + server will fail the connection with a "no_application_protocol" alert.</p> + + <p>The negotiated protocol can be retrieved using the <c>negotiated_protocol/1</c> function.</p> + </item> + <tag>{next_protocols_advertised, Protocols :: [binary()]}</tag> <item>The list of protocols to send to the client if the client indicates it supports the Next Protocol extension. The client may select a protocol that is not on this list. The list of protocols must not contain an empty binary. If the server negotiates a Next Protocol it can be accessed - using <c>negotiated_next_protocol/1</c> method. + using <c>negotiated_protocol/1</c> function. </item> <tag>{psk_identity, string()}</tag> @@ -1018,15 +1046,15 @@ fun(srp, Username :: string(), UserState :: term()) -> </desc> </func> <func> - <name>negotiated_next_protocol(Socket) -> {ok, Protocol} | {error, next_protocol_not_negotiated}</name> - <fsummary>Returns the Next Protocol negotiated.</fsummary> + <name>negotiated_protocol(Socket) -> {ok, Protocol} | {error, protocol_not_negotiated}</name> + <fsummary>Returns the protocol negotiated through ALPN or NPN extensions.</fsummary> <type> <v>Socket = sslsocket()</v> <v>Protocol = binary()</v> </type> <desc> <p> - Returns the Next Protocol negotiated. + Returns the protocol negotiated through ALPN or NPN extensions. </p> </desc> </func> diff --git a/lib/ssl/doc/src/ssl_crl_cache.xml b/lib/ssl/doc/src/ssl_crl_cache.xml index 1ed76d3e2a..b291c7b633 100644 --- a/lib/ssl/doc/src/ssl_crl_cache.xml +++ b/lib/ssl/doc/src/ssl_crl_cache.xml @@ -29,7 +29,7 @@ <p> Implements an internal CRL (Certificate Revocation List) cache. In addition to implementing the <seealso - marker="ssl_cache_crl_api"> ssl_cache_crl_api</seealso> + marker="ssl_cache_crl_api"> ssl_cache_crl_api</seealso> behaviour the following functions are available. </p> </description> diff --git a/lib/ssl/doc/src/ssl_crl_cache_api.xml b/lib/ssl/doc/src/ssl_crl_cache_api.xml index 24365c9f59..3f518496be 100644 --- a/lib/ssl/doc/src/ssl_crl_cache_api.xml +++ b/lib/ssl/doc/src/ssl_crl_cache_api.xml @@ -27,13 +27,15 @@ <modulesummary>API for a SSL/TLS CRL (Certificate Revocation List) cache.</modulesummary> <description> <p> - When SSL/TLS performs certificate path validation according to - <url href="http://www.ietf.org/rfc/rfc5280.txt">RFC 5280 </url> it should - also perform CRL validation checks. To enable the CRL checks the application - needs access to CRLs. A database of CRLs can be set up in many different ways. - This module provides an API to integrate an arbitrary CRL cache with the erlang - ssl application. It is also used by the application itself to provide a simple - default implementation of a CRL cache. + When SSL/TLS performs certificate path validation according to + <url href="http://www.ietf.org/rfc/rfc5280.txt">RFC 5280 </url> + it should also perform CRL validation checks. To enable the CRL + checks the application needs access to CRLs. A database of CRLs + can be set up in many different ways. This module provides the + behavior of the API needed to integrate an arbitrary CRL cache + with the erlang ssl application. It is also used by the + application itself to provide a simple default implementation of + a CRL cache. </p> </description> diff --git a/lib/ssl/src/dtls_connection.erl b/lib/ssl/src/dtls_connection.erl index 508983ddac..610e2c4e41 100644 --- a/lib/ssl/src/dtls_connection.erl +++ b/lib/ssl/src/dtls_connection.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2013-2014. All Rights Reserved. +%% Copyright Ericsson AB 2013-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -146,7 +146,7 @@ init([Role, Host, Port, Socket, {SSLOpts0, _} = Options, User, CbInfo]) -> Handshake = ssl_handshake:init_handshake_history(), TimeStamp = calendar:datetime_to_gregorian_seconds({date(), time()}), try ssl_config:init(SSLOpts0, Role) of - {ok, Ref, CertDbHandle, FileRefHandle, CacheHandle, OwnCert, Key, DHParams} -> + {ok, Ref, CertDbHandle, FileRefHandle, CacheHandle, CRLDbInfo, OwnCert, Key, DHParams} -> Session = State0#state.session, State = State0#state{ tls_handshake_history = Handshake, @@ -155,6 +155,7 @@ init([Role, Host, Port, Socket, {SSLOpts0, _} = Options, User, CbInfo]) -> file_ref_db = FileRefHandle, cert_db_ref = Ref, cert_db = CertDbHandle, + crl_db = CRLDbInfo, session_cache = CacheHandle, private_key = Key, diffie_hellman_params = DHParams}, @@ -227,9 +228,9 @@ hello(Hello, case dtls_handshake:hello(Hello, SslOptions, ConnectionStates0, Renegotiation) of #alert{} = Alert -> handle_own_alert(Alert, ReqVersion, hello, State); - {Version, NewId, ConnectionStates, NextProtocol} -> + {Version, NewId, ConnectionStates, ProtoExt, Protocol} -> ssl_connection:handle_session(Hello, - Version, NewId, ConnectionStates, NextProtocol, State) + Version, NewId, ConnectionStates, ProtoExt, Protocol, State) end; hello(Msg, State) -> diff --git a/lib/ssl/src/dtls_handshake.erl b/lib/ssl/src/dtls_handshake.erl index 31d525b295..30381df050 100644 --- a/lib/ssl/src/dtls_handshake.erl +++ b/lib/ssl/src/dtls_handshake.erl @@ -181,8 +181,8 @@ handle_server_hello_extensions(Version, SessionId, Random, CipherSuite, SslOpt, ConnectionStates0, Renegotiation) of #alert{} = Alert -> Alert; - {ConnectionStates, Protocol} -> - {Version, SessionId, ConnectionStates, Protocol} + {ConnectionStates, ProtoExt, Protocol} -> + {Version, SessionId, ConnectionStates, ProtoExt, Protocol} end. dtls_fragment(Mss, MsgType, Len, MsgSeq, Bin, Offset, Acc) diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index 623fa92121..6461f64c1c 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -38,10 +38,12 @@ %% SSL/TLS protocol handling -export([cipher_suites/0, cipher_suites/1, suite_definition/1, connection_info/1, versions/0, session_info/1, format_error/1, - renegotiate/1, prf/5, negotiated_next_protocol/1]). + renegotiate/1, prf/5, negotiated_protocol/1, negotiated_next_protocol/1]). %% Misc -export([random_bytes/1]). +-deprecated({negotiated_next_protocol, 1, next_major_release}). + -include("ssl_api.hrl"). -include("ssl_internal.hrl"). -include("ssl_record.hrl"). @@ -330,13 +332,27 @@ suite_definition(S) -> {KeyExchange, Cipher, Hash}. %%-------------------------------------------------------------------- +-spec negotiated_protocol(#sslsocket{}) -> {ok, binary()} | {error, reason()}. +%% +%% Description: Returns the protocol that has been negotiated. If no +%% protocol has been negotiated will return {error, protocol_not_negotiated} +%%-------------------------------------------------------------------- +negotiated_protocol(#sslsocket{pid = Pid}) -> + ssl_connection:negotiated_protocol(Pid). + +%%-------------------------------------------------------------------- -spec negotiated_next_protocol(#sslsocket{}) -> {ok, binary()} | {error, reason()}. %% %% Description: Returns the next protocol that has been negotiated. If no %% protocol has been negotiated will return {error, next_protocol_not_negotiated} %%-------------------------------------------------------------------- -negotiated_next_protocol(#sslsocket{pid = Pid}) -> - ssl_connection:negotiated_next_protocol(Pid). +negotiated_next_protocol(Socket) -> + case negotiated_protocol(Socket) of + {error, protocol_not_negotiated} -> + {error, next_protocol_not_negotiated}; + Res -> + Res + end. %%-------------------------------------------------------------------- -spec cipher_suites(erlang | openssl | all) -> [ssl_cipher:erl_cipher_suite()] | @@ -644,6 +660,10 @@ handle_options(Opts0) -> renegotiate_at = handle_option(renegotiate_at, Opts, ?DEFAULT_RENEGOTIATE_AT), hibernate_after = handle_option(hibernate_after, Opts, undefined), erl_dist = handle_option(erl_dist, Opts, false), + alpn_advertised_protocols = + handle_option(alpn_advertised_protocols, Opts, undefined), + alpn_preferred_protocols = + handle_option(alpn_preferred_protocols, Opts, undefined), next_protocols_advertised = handle_option(next_protocols_advertised, Opts, undefined), next_protocol_selector = @@ -667,7 +687,8 @@ handle_options(Opts0) -> user_lookup_fun, psk_identity, srp_identity, ciphers, reuse_session, reuse_sessions, ssl_imp, cb_info, renegotiate_at, secure_renegotiate, hibernate_after, - erl_dist, next_protocols_advertised, + erl_dist, alpn_advertised_protocols, + alpn_preferred_protocols, next_protocols_advertised, client_preferred_next_protocols, log_alert, server_name_indication, honor_cipher_order, padding_check, crl_check, crl_cache, fallback], @@ -803,6 +824,20 @@ validate_option(hibernate_after, Value) when is_integer(Value), Value >= 0 -> Value; validate_option(erl_dist,Value) when is_boolean(Value) -> Value; +validate_option(Opt, Value) + when Opt =:= alpn_advertised_protocols orelse Opt =:= alpn_preferred_protocols, + is_list(Value) -> + case tls_record:highest_protocol_version([]) of + {3,0} -> + throw({error, {options, {not_supported_in_sslv3, {Opt, Value}}}}); + _ -> + validate_binary_list(Opt, Value), + Value + end; +validate_option(Opt, Value) + when Opt =:= alpn_advertised_protocols orelse Opt =:= alpn_preferred_protocols, + Value =:= undefined -> + undefined; validate_option(client_preferred_next_protocols = Opt, {Precedence, PreferredProtocols} = Value) when is_list(PreferredProtocols) -> case tls_record:highest_protocol_version([]) of @@ -1131,6 +1166,10 @@ new_ssl_options([{secure_renegotiate, Value} | Rest], #ssl_options{} = Opts, Rec new_ssl_options(Rest, Opts#ssl_options{secure_renegotiate = validate_option(secure_renegotiate, Value)}, RecordCB); new_ssl_options([{hibernate_after, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> new_ssl_options(Rest, Opts#ssl_options{hibernate_after = validate_option(hibernate_after, Value)}, RecordCB); +new_ssl_options([{alpn_advertised_protocols, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> + new_ssl_options(Rest, Opts#ssl_options{alpn_advertised_protocols = validate_option(alpn_advertised_protocols, Value)}, RecordCB); +new_ssl_options([{alpn_preferred_protocols, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> + new_ssl_options(Rest, Opts#ssl_options{alpn_preferred_protocols = validate_option(alpn_preferred_protocols, Value)}, RecordCB); new_ssl_options([{next_protocols_advertised, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> new_ssl_options(Rest, Opts#ssl_options{next_protocols_advertised = validate_option(next_protocols_advertised, Value)}, RecordCB); new_ssl_options([{client_preferred_next_protocols, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> diff --git a/lib/ssl/src/ssl_alert.erl b/lib/ssl/src/ssl_alert.erl index 9e372f739a..c46facb75d 100644 --- a/lib/ssl/src/ssl_alert.erl +++ b/lib/ssl/src/ssl_alert.erl @@ -163,5 +163,7 @@ description_txt(?UNKNOWN_PSK_IDENTITY) -> "unknown psk identity"; description_txt(?INAPPROPRIATE_FALLBACK) -> "inappropriate fallback"; +description_txt(?NO_APPLICATION_PROTOCOL) -> + "no application protocol"; description_txt(Enum) -> lists:flatten(io_lib:format("unsupported/unknown alert: ~p", [Enum])). diff --git a/lib/ssl/src/ssl_alert.hrl b/lib/ssl/src/ssl_alert.hrl index a3619e4a35..70b7523975 100644 --- a/lib/ssl/src/ssl_alert.hrl +++ b/lib/ssl/src/ssl_alert.hrl @@ -69,6 +69,8 @@ %% bad_certificate_hash_value(114), %% RFC 4366 %% unknown_psk_identity(115), +%% RFC 7301 +%% no_application_protocol(120), %% (255) %% } AlertDescription; @@ -103,6 +105,7 @@ -define(BAD_CERTIFICATE_STATUS_RESPONSE, 113). -define(BAD_CERTIFICATE_HASH_VALUE, 114). -define(UNKNOWN_PSK_IDENTITY, 115). +-define(NO_APPLICATION_PROTOCOL, 120). -define(ALERT_REC(Level,Desc), #alert{level=Level,description=Desc,where={?FILE, ?LINE}}). diff --git a/lib/ssl/src/ssl_api.hrl b/lib/ssl/src/ssl_api.hrl index 22185ff60a..78127eeafa 100644 --- a/lib/ssl/src/ssl_api.hrl +++ b/lib/ssl/src/ssl_api.hrl @@ -49,6 +49,8 @@ {srp_identity, {string(), string()}} | {ciphers, ciphers()} | {ssl_imp, ssl_imp()} | {reuse_sessions, boolean()} | {reuse_session, fun()} | {hibernate_after, integer()|undefined} | + {alpn_advertised_protocols, [binary()]} | + {alpn_preferred_protocols, [binary()]} | {next_protocols_advertised, list(binary())} | {client_preferred_next_protocols, binary(), client | server, list(binary())}. diff --git a/lib/ssl/src/ssl_certificate.erl b/lib/ssl/src/ssl_certificate.erl index 764bd82de0..34e4a8b447 100644 --- a/lib/ssl/src/ssl_certificate.erl +++ b/lib/ssl/src/ssl_certificate.erl @@ -84,7 +84,7 @@ trusted_cert_and_path(CertChain, CertDbHandle, CertDbRef, PartialChainHandler) - end. %%-------------------------------------------------------------------- --spec certificate_chain(undefined | binary(), db_handle(), certdb_ref()) -> +-spec certificate_chain(undefined | binary() | #'OTPCertificate'{} , db_handle(), certdb_ref()) -> {error, no_cert} | {ok, #'OTPCertificate'{} | undefined, [der_cert()]}. %% %% Description: Return the certificate chain to send to peer. diff --git a/lib/ssl/src/ssl_connection.erl b/lib/ssl/src/ssl_connection.erl index 08d0145aa7..4a839872a6 100644 --- a/lib/ssl/src/ssl_connection.erl +++ b/lib/ssl/src/ssl_connection.erl @@ -42,10 +42,10 @@ %% User Events -export([send/2, recv/3, close/1, shutdown/2, new_user/2, get_opts/2, set_opts/2, info/1, session_info/1, - peer_certificate/1, renegotiation/1, negotiated_next_protocol/1, prf/5 + peer_certificate/1, renegotiation/1, negotiated_protocol/1, prf/5 ]). --export([handle_session/6]). +-export([handle_session/7]). %% SSL FSM state functions -export([hello/3, abbreviated/3, certify/3, cipher/3, connection/3]). @@ -191,12 +191,12 @@ new_user(ConnectionPid, User) -> sync_send_all_state_event(ConnectionPid, {new_user, User}). %%-------------------------------------------------------------------- --spec negotiated_next_protocol(pid()) -> {ok, binary()} | {error, reason()}. +-spec negotiated_protocol(pid()) -> {ok, binary()} | {error, reason()}. %% %% Description: Returns the negotiated protocol %%-------------------------------------------------------------------- -negotiated_next_protocol(ConnectionPid) -> - sync_send_all_state_event(ConnectionPid, negotiated_next_protocol). +negotiated_protocol(ConnectionPid) -> + sync_send_all_state_event(ConnectionPid, negotiated_protocol). %%-------------------------------------------------------------------- -spec get_opts(pid(), list()) -> {ok, list()} | {error, reason()}. @@ -258,27 +258,26 @@ prf(ConnectionPid, Secret, Label, Seed, WantedLength) -> handle_session(#server_hello{cipher_suite = CipherSuite, compression_method = Compression}, - Version, NewId, ConnectionStates, NextProtocol, + Version, NewId, ConnectionStates, ProtoExt, Protocol0, #state{session = #session{session_id = OldId}, - negotiated_version = ReqVersion} = State0) -> + negotiated_version = ReqVersion, + negotiated_protocol = CurrentProtocol} = State0) -> {KeyAlgorithm, _, _, _} = ssl_cipher:suite_definition(CipherSuite), PremasterSecret = make_premaster_secret(ReqVersion, KeyAlgorithm), - - NewNextProtocol = case NextProtocol of - undefined -> - State0#state.next_protocol; - _ -> - NextProtocol - end, - + + {ExpectNPN, Protocol} = case Protocol0 of + undefined -> {false, CurrentProtocol}; + _ -> {ProtoExt =:= npn, Protocol0} + end, + State = State0#state{key_algorithm = KeyAlgorithm, negotiated_version = Version, connection_states = ConnectionStates, premaster_secret = PremasterSecret, - expecting_next_protocol_negotiation = NextProtocol =/= undefined, - next_protocol = NewNextProtocol}, + expecting_next_protocol_negotiation = ExpectNPN, + negotiated_protocol = Protocol}, case ssl_session:is_new(OldId, NewId) of true -> @@ -371,7 +370,7 @@ abbreviated(#finished{verify_data = Data} = Finished, abbreviated(#next_protocol{selected_protocol = SelectedProtocol}, #state{role = server, expecting_next_protocol_negotiation = true} = State0, Connection) -> - {Record, State} = Connection:next_record(State0#state{next_protocol = SelectedProtocol}), + {Record, State} = Connection:next_record(State0#state{negotiated_protocol = SelectedProtocol}), Connection:next_state(abbreviated, abbreviated, Record, State#state{expecting_next_protocol_negotiation = false}); abbreviated(timeout, State, _) -> @@ -593,7 +592,7 @@ cipher(#certificate_verify{signature = Signature, hashsign_algorithm = CertHashS %% client must send a next protocol message if we are expecting it cipher(#finished{}, #state{role = server, expecting_next_protocol_negotiation = true, - next_protocol = undefined, negotiated_version = Version} = State0, + negotiated_protocol = undefined, negotiated_version = Version} = State0, Connection) -> Connection:handle_own_alert(?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE), Version, cipher, State0); @@ -623,7 +622,7 @@ cipher(#finished{verify_data = Data} = Finished, cipher(#next_protocol{selected_protocol = SelectedProtocol}, #state{role = server, expecting_next_protocol_negotiation = true, expecting_finished = true} = State0, Connection) -> - {Record, State} = Connection:next_record(State0#state{next_protocol = SelectedProtocol}), + {Record, State} = Connection:next_record(State0#state{negotiated_protocol = SelectedProtocol}), Connection:next_state(cipher, cipher, Record, State#state{expecting_next_protocol_negotiation = false}); cipher(timeout, State, _) -> @@ -759,10 +758,10 @@ handle_sync_event({get_opts, OptTags}, _From, StateName, socket_options = SockOpts} = State) -> OptsReply = get_socket_opts(Transport, Socket, OptTags, SockOpts, []), {reply, OptsReply, StateName, State, get_timeout(State)}; -handle_sync_event(negotiated_next_protocol, _From, StateName, #state{next_protocol = undefined} = State) -> - {reply, {error, next_protocol_not_negotiated}, StateName, State, get_timeout(State)}; -handle_sync_event(negotiated_next_protocol, _From, StateName, #state{next_protocol = NextProtocol} = State) -> - {reply, {ok, NextProtocol}, StateName, State, get_timeout(State)}; +handle_sync_event(negotiated_protocol, _From, StateName, #state{negotiated_protocol = undefined} = State) -> + {reply, {error, protocol_not_negotiated}, StateName, State, get_timeout(State)}; +handle_sync_event(negotiated_protocol, _From, StateName, #state{negotiated_protocol = SelectedProtocol} = State) -> + {reply, {ok, SelectedProtocol}, StateName, State, get_timeout(State)}; handle_sync_event({set_opts, Opts0}, _From, StateName0, #state{socket_options = Opts1, protocol_cb = Connection, @@ -1484,11 +1483,11 @@ finalize_handshake(State0, StateName, Connection) -> next_protocol(#state{role = server} = State, _) -> State; -next_protocol(#state{next_protocol = undefined} = State, _) -> +next_protocol(#state{negotiated_protocol = undefined} = State, _) -> State; next_protocol(#state{expecting_next_protocol_negotiation = false} = State, _) -> State; -next_protocol(#state{next_protocol = NextProtocol} = State0, Connection) -> +next_protocol(#state{negotiated_protocol = NextProtocol} = State0, Connection) -> NextProtocolMessage = ssl_handshake:next_protocol(NextProtocol), Connection:send_handshake(NextProtocolMessage, State0). diff --git a/lib/ssl/src/ssl_connection.hrl b/lib/ssl/src/ssl_connection.hrl index ac3b26e4bf..e569d706af 100644 --- a/lib/ssl/src/ssl_connection.hrl +++ b/lib/ssl/src/ssl_connection.hrl @@ -78,7 +78,7 @@ allow_renegotiate = true ::boolean(), expecting_next_protocol_negotiation = false ::boolean(), expecting_finished = false ::boolean(), - next_protocol = undefined :: undefined | binary(), + negotiated_protocol = undefined :: undefined | binary(), client_ecc, % {Curves, PointFmt} tracker :: pid() %% Tracker process for listen socket }). diff --git a/lib/ssl/src/ssl_crl.erl b/lib/ssl/src/ssl_crl.erl index b8761f0601..1a08d3c80a 100644 --- a/lib/ssl/src/ssl_crl.erl +++ b/lib/ssl/src/ssl_crl.erl @@ -73,8 +73,6 @@ verify_crl_issuer(CRL, ErlCertCandidate, Issuer, NotIssuer) -> true -> throw({ok, ErlCertCandidate}); false -> - NotIssuer; - _ -> NotIssuer end; _ -> diff --git a/lib/ssl/src/ssl_crl_cache.erl b/lib/ssl/src/ssl_crl_cache.erl index b2bdb19979..b9d6a61c3b 100644 --- a/lib/ssl/src/ssl_crl_cache.erl +++ b/lib/ssl/src/ssl_crl_cache.erl @@ -34,7 +34,7 @@ %% Cache callback API %%==================================================================== -lookup(#'DistributionPoint'{distributionPoint={fullName, Names}}, +lookup(#'DistributionPoint'{distributionPoint = {fullName, Names}}, CRLDbInfo) -> get_crls(Names, CRLDbInfo); lookup(_,_) -> @@ -48,8 +48,8 @@ select(Issuer, {{_Cache, Mapping},_}) -> CRLs end. -fresh_crl(DistributionPoint, CRL) -> - case get_crls(DistributionPoint, undefined) of +fresh_crl(#'DistributionPoint'{distributionPoint = {fullName, Names}}, CRL) -> + case get_crls(Names, undefined) of not_available -> CRL; [NewCRL] -> diff --git a/lib/ssl/src/ssl_crl_cache_api.erl b/lib/ssl/src/ssl_crl_cache_api.erl index 0915ba12e5..79db65104b 100644 --- a/lib/ssl/src/ssl_crl_cache_api.erl +++ b/lib/ssl/src/ssl_crl_cache_api.erl @@ -25,6 +25,6 @@ -type db_handle() :: term(). --callback lookup(#'DistributionPoint'{}, db_handle()) -> not_available | [public_key:der_encode()]. --callback select(term(), db_handle()) -> [public_key:der_encode()]. --callback fresh_crl(#'DistributionPoint'{}, public_key:der_encode()) -> public_key:der_encode(). +-callback lookup(#'DistributionPoint'{}, db_handle()) -> not_available | [public_key:der_encoded()]. +-callback select(term(), db_handle()) -> [public_key:der_encoded()]. +-callback fresh_crl(#'DistributionPoint'{}, public_key:der_encoded()) -> public_key:der_encoded(). diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 6cab8eb7a1..493e5a87d9 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -136,6 +136,7 @@ client_hello_extensions(Host, Version, CipherSuites, SslOpts, ConnectionStates, hash_signs = advertised_hash_signs(Version), ec_point_formats = EcPointFormats, elliptic_curves = EllipticCurves, + alpn = encode_alpn(SslOpts#ssl_options.alpn_advertised_protocols, Renegotiation), next_protocol_negotiation = encode_client_protocol_negotiation(SslOpts#ssl_options.next_protocol_selector, Renegotiation), @@ -764,6 +765,11 @@ encode_hello_extensions([], Acc) -> Size = byte_size(Acc), <<?UINT16(Size), Acc/binary>>; +encode_hello_extensions([#alpn{extension_data = ExtensionData} | Rest], Acc) -> + Len = byte_size(ExtensionData), + ExtLen = Len + 2, + encode_hello_extensions(Rest, <<?UINT16(?ALPN_EXT), ?UINT16(ExtLen), ?UINT16(Len), + ExtensionData/binary, Acc/binary>>); encode_hello_extensions([#next_protocol_negotiation{extension_data = ExtensionData} | Rest], Acc) -> Len = byte_size(ExtensionData), encode_hello_extensions(Rest, <<?UINT16(?NEXTPROTONEG_EXT), ?UINT16(Len), @@ -862,6 +868,25 @@ decode_client_key(ClientKey, Type, Version) -> decode_server_key(ServerKey, Type, Version) -> dec_server_key(ServerKey, key_exchange_alg(Type), Version). +%% +%% Description: Encode and decode functions for ALPN extension data. +%%-------------------------------------------------------------------- + +%% While the RFC opens the door to allow ALPN during renegotiation, in practice +%% this does not work and it is recommended to ignore any ALPN extension during +%% renegotiation, as done here. +encode_alpn(_, true) -> + undefined; +encode_alpn(undefined, _) -> + undefined; +encode_alpn(Protocols, _) -> + #alpn{extension_data = lists:foldl(fun encode_protocol/2, <<>>, Protocols)}. + +decode_alpn(undefined) -> + undefined; +decode_alpn(#alpn{extension_data=Data}) -> + decode_protocols(Data, []). + encode_client_protocol_negotiation(undefined, _) -> undefined; encode_client_protocol_negotiation(_, false) -> @@ -1124,8 +1149,10 @@ handle_client_hello_extensions(RecordCB, Random, ClientCipherSuites, #hello_extensions{renegotiation_info = Info, srp = SRP, ec_point_formats = ECCFormat, + alpn = ALPN, next_protocol_negotiation = NextProtocolNegotiation}, Version, - #ssl_options{secure_renegotiate = SecureRenegotation} = Opts, + #ssl_options{secure_renegotiate = SecureRenegotation, + alpn_preferred_protocols = ALPNPreferredProtocols} = Opts, #session{cipher_suite = NegotiatedCipherSuite, compression_method = Compression} = Session0, ConnectionStates0, Renegotiation) -> @@ -1134,19 +1161,34 @@ handle_client_hello_extensions(RecordCB, Random, ClientCipherSuites, Random, NegotiatedCipherSuite, ClientCipherSuites, Compression, ConnectionStates0, Renegotiation, SecureRenegotation), - ProtocolsToAdvertise = handle_next_protocol_extension(NextProtocolNegotiation, Renegotiation, Opts), - + ServerHelloExtensions = #hello_extensions{ renegotiation_info = renegotiation_info(RecordCB, server, ConnectionStates, Renegotiation), - ec_point_formats = server_ecc_extension(Version, ECCFormat), - next_protocol_negotiation = - encode_protocols_advertised_on_server(ProtocolsToAdvertise) + ec_point_formats = server_ecc_extension(Version, ECCFormat) }, - {Session, ConnectionStates, ServerHelloExtensions}. + + %% If we receive an ALPN extension and have ALPN configured for this connection, + %% we handle it. Otherwise we check for the NPN extension. + if + ALPN =/= undefined, ALPNPreferredProtocols =/= undefined -> + case handle_alpn_extension(ALPNPreferredProtocols, decode_alpn(ALPN)) of + #alert{} = Alert -> + Alert; + Protocol -> + {Session, ConnectionStates, Protocol, + ServerHelloExtensions#hello_extensions{alpn=encode_alpn([Protocol], Renegotiation)}} + end; + true -> + ProtocolsToAdvertise = handle_next_protocol_extension(NextProtocolNegotiation, Renegotiation, Opts), + {Session, ConnectionStates, undefined, + ServerHelloExtensions#hello_extensions{next_protocol_negotiation= + encode_protocols_advertised_on_server(ProtocolsToAdvertise)}} + end. handle_server_hello_extensions(RecordCB, Random, CipherSuite, Compression, #hello_extensions{renegotiation_info = Info, + alpn = ALPN, next_protocol_negotiation = NextProtocolNegotiation}, Version, #ssl_options{secure_renegotiate = SecureRenegotation, next_protocol_selector = NextProtoSelector}, @@ -1155,11 +1197,23 @@ handle_server_hello_extensions(RecordCB, Random, CipherSuite, Compression, CipherSuite, undefined, Compression, ConnectionStates0, Renegotiation, SecureRenegotation), - case handle_next_protocol(NextProtocolNegotiation, NextProtoSelector, Renegotiation) of - #alert{} = Alert -> - Alert; - Protocol -> - {ConnectionStates, Protocol} + + %% If we receive an ALPN extension then this is the protocol selected, + %% otherwise handle the NPN extension. + case decode_alpn(ALPN) of + %% ServerHello contains exactly one protocol: the one selected. + %% We also ignore the ALPN extension during renegotiation (see encode_alpn/2). + [Protocol] when not Renegotiation -> + {ConnectionStates, alpn, Protocol}; + undefined -> + case handle_next_protocol(NextProtocolNegotiation, NextProtoSelector, Renegotiation) of + #alert{} = Alert -> + Alert; + Protocol -> + {ConnectionStates, npn, Protocol} + end; + _ -> %% {error, _Reason} or a list of 0/2+ protocols. + ?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE) end. select_version(RecordCB, ClientVersion, Versions) -> @@ -1267,10 +1321,11 @@ hello_extensions_list(#hello_extensions{renegotiation_info = RenegotiationInfo, hash_signs = HashSigns, ec_point_formats = EcPointFormats, elliptic_curves = EllipticCurves, + alpn = ALPN, next_protocol_negotiation = NextProtocolNegotiation, sni = Sni}) -> [Ext || Ext <- [RenegotiationInfo, SRP, HashSigns, - EcPointFormats, EllipticCurves, NextProtocolNegotiation, Sni], Ext =/= undefined]. + EcPointFormats, EllipticCurves, ALPN, NextProtocolNegotiation, Sni], Ext =/= undefined]. srp_user(#ssl_options{srp_identity = {UserName, _}}) -> #srp{username = UserName}; @@ -1708,6 +1763,10 @@ dec_server_key_signature(_, _, _) -> dec_hello_extensions(<<>>, Acc) -> Acc; +dec_hello_extensions(<<?UINT16(?ALPN_EXT), ?UINT16(ExtLen), ?UINT16(Len), ExtensionData:Len/binary, Rest/binary>>, Acc) + when Len + 2 =:= ExtLen -> + ALPN = #alpn{extension_data = ExtensionData}, + dec_hello_extensions(Rest, Acc#hello_extensions{alpn = ALPN}); dec_hello_extensions(<<?UINT16(?NEXTPROTONEG_EXT), ?UINT16(Len), ExtensionData:Len/binary, Rest/binary>>, Acc) -> NextP = #next_protocol_negotiation{extension_data = ExtensionData}, dec_hello_extensions(Rest, Acc#hello_extensions{next_protocol_negotiation = NextP}); @@ -1788,18 +1847,19 @@ dec_sni(<<?BYTE(_), ?UINT16(Len), _:Len, Rest/binary>>) -> dec_sni(Rest); dec_sni(_) -> undefined. decode_next_protocols({next_protocol_negotiation, Protocols}) -> - decode_next_protocols(Protocols, []). -decode_next_protocols(<<>>, Acc) -> + decode_protocols(Protocols, []). + +decode_protocols(<<>>, Acc) -> lists:reverse(Acc); -decode_next_protocols(<<?BYTE(Len), Protocol:Len/binary, Rest/binary>>, Acc) -> +decode_protocols(<<?BYTE(Len), Protocol:Len/binary, Rest/binary>>, Acc) -> case Len of 0 -> - {error, invalid_next_protocols}; + {error, invalid_protocols}; _ -> - decode_next_protocols(Rest, [Protocol|Acc]) + decode_protocols(Rest, [Protocol|Acc]) end; -decode_next_protocols(_Bytes, _Acc) -> - {error, invalid_next_protocols}. +decode_protocols(_Bytes, _Acc) -> + {error, invalid_protocols}. %% encode/decode stream of certificate data to/from list of certificate data certs_to_list(ASN1Certs) -> @@ -1853,6 +1913,17 @@ key_exchange_alg(_) -> %%-------------Extension handling -------------------------------- +%% Receive protocols, choose one from the list, return it. +handle_alpn_extension(_, {error, _Reason}) -> + ?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE); +handle_alpn_extension([], _) -> + ?ALERT_REC(?FATAL, ?NO_APPLICATION_PROTOCOL); +handle_alpn_extension([ServerProtocol|Tail], ClientProtocols) -> + case lists:member(ServerProtocol, ClientProtocols) of + true -> ServerProtocol; + false -> handle_alpn_extension(Tail, ClientProtocols) + end. + handle_next_protocol(undefined, _NextProtocolSelector, _Renegotiating) -> undefined; @@ -1998,12 +2069,12 @@ crl_check(OtpCert, Check, CertDbHandle, CertDbRef, {Callback, CRLDbHandle}, _) - case dps_and_crls(OtpCert, Callback, CRLDbHandle, same_issuer) of [] -> valid; %% No relevant CRL existed - Dps -> - crl_check_same_issuer(OtpCert, Check, Dps, Options) + DpsAndCRls -> + crl_check_same_issuer(OtpCert, Check, DpsAndCRls, Options) end; - Dps -> %% This DP list may be empty if relevant CRLs existed + DpsAndCRLs -> %% This DP list may be empty if relevant CRLs existed %% but could not be retrived, will result in {bad_cert, revocation_status_undetermined} - case public_key:pkix_crls_validate(OtpCert, Dps, Options) of + case public_key:pkix_crls_validate(OtpCert, DpsAndCRLs, Options) of {bad_cert, revocation_status_undetermined} -> crl_check_same_issuer(OtpCert, Check, dps_and_crls(OtpCert, Callback, CRLDbHandle, same_issuer), Options); diff --git a/lib/ssl/src/ssl_handshake.hrl b/lib/ssl/src/ssl_handshake.hrl index 80284faef0..91f674a6fc 100644 --- a/lib/ssl/src/ssl_handshake.hrl +++ b/lib/ssl/src/ssl_handshake.hrl @@ -95,6 +95,7 @@ -record(hello_extensions, { renegotiation_info, hash_signs, % supported combinations of hashes/signature algos + alpn, next_protocol_negotiation = undefined, % [binary()] srp, ec_point_formats, @@ -301,6 +302,14 @@ }). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Application-Layer Protocol Negotiation RFC 7301 +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +-define(ALPN_EXT, 16). + +-record(alpn, {extension_data}). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Next Protocol Negotiation %% (http://tools.ietf.org/html/draft-agl-tls-nextprotoneg-02) %% (http://technotes.googlecode.com/git/nextprotoneg.html) diff --git a/lib/ssl/src/ssl_internal.hrl b/lib/ssl/src/ssl_internal.hrl index 8df79f9e8c..90f8b8a412 100644 --- a/lib/ssl/src/ssl_internal.hrl +++ b/lib/ssl/src/ssl_internal.hrl @@ -116,16 +116,18 @@ hibernate_after :: boolean(), %% This option should only be set to true by inet_tls_dist erl_dist = false :: boolean(), - next_protocols_advertised = undefined, %% [binary()], + alpn_advertised_protocols = undefined :: [binary()] | undefined , + alpn_preferred_protocols = undefined :: [binary()] | undefined, + next_protocols_advertised = undefined :: [binary()] | undefined, next_protocol_selector = undefined, %% fun([binary()]) -> binary()) log_alert :: boolean(), server_name_indication = undefined, %% Should the server prefer its own cipher order over the one provided by %% the client? - honor_cipher_order = false, - padding_check = true, - fallback = false, - crl_check, + honor_cipher_order = false :: boolean(), + padding_check = true :: boolean(), + fallback = false :: boolean(), + crl_check :: boolean() | peer | best_effort, crl_cache }). diff --git a/lib/ssl/src/ssl_manager.erl b/lib/ssl/src/ssl_manager.erl index 9c4b2a8bad..396013825e 100644 --- a/lib/ssl/src/ssl_manager.erl +++ b/lib/ssl/src/ssl_manager.erl @@ -101,8 +101,10 @@ start_link_dist(Opts) -> gen_server:start_link({local, DistMangerName}, ?MODULE, [DistMangerName, Opts], []). %%-------------------------------------------------------------------- --spec connection_init(binary()| {der, list()}, client | server, {Cb :: atom(), Handle:: term()}) -> - {ok, certdb_ref(), db_handle(), db_handle(), db_handle(), db_handle()}. +-spec connection_init(binary()| {der, list()}, client | server, + {Cb :: atom(), Handle:: term()}) -> + {ok, certdb_ref(), db_handle(), db_handle(), + db_handle(), db_handle(), CRLInfo::term()}. %% %% Description: Do necessary initializations for a new connection. %%-------------------------------------------------------------------- diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl index 77d3aa7889..0577222980 100644 --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -188,19 +188,27 @@ hello(Hello = #client_hello{client_version = ClientVersion, renegotiation = {Renegotiation, _}, session_cache = Cache, session_cache_cb = CacheCb, + negotiated_protocol = CurrentProtocol, ssl_options = SslOpts}) -> case tls_handshake:hello(Hello, SslOpts, {Port, Session0, Cache, CacheCb, ConnectionStates0, Cert}, Renegotiation) of + #alert{} = Alert -> + handle_own_alert(Alert, ClientVersion, hello, State); {Version, {Type, Session}, - ConnectionStates, ServerHelloExt} -> + ConnectionStates, Protocol0, ServerHelloExt} -> + + Protocol = case Protocol0 of + undefined -> CurrentProtocol; + _ -> Protocol0 + end, + HashSign = ssl_handshake:select_hashsign(HashSigns, Cert, Version), ssl_connection:hello({common_client_hello, Type, ServerHelloExt, HashSign}, State#state{connection_states = ConnectionStates, negotiated_version = Version, session = Session, - client_ecc = {EllipticCurves, EcPointFormats}}, ?MODULE); - #alert{} = Alert -> - handle_own_alert(Alert, ClientVersion, hello, State) + client_ecc = {EllipticCurves, EcPointFormats}, + negotiated_protocol = Protocol}, ?MODULE) end; hello(Hello, #state{connection_states = ConnectionStates0, @@ -211,9 +219,9 @@ hello(Hello, case tls_handshake:hello(Hello, SslOptions, ConnectionStates0, Renegotiation) of #alert{} = Alert -> handle_own_alert(Alert, ReqVersion, hello, State); - {Version, NewId, ConnectionStates, NextProtocol} -> + {Version, NewId, ConnectionStates, ProtoExt, Protocol} -> ssl_connection:handle_session(Hello, - Version, NewId, ConnectionStates, NextProtocol, State) + Version, NewId, ConnectionStates, ProtoExt, Protocol, State) end; hello(Msg, State) -> diff --git a/lib/ssl/src/tls_handshake.erl b/lib/ssl/src/tls_handshake.erl index b0b6d5a8e3..d936310991 100644 --- a/lib/ssl/src/tls_handshake.erl +++ b/lib/ssl/src/tls_handshake.erl @@ -78,12 +78,14 @@ client_hello(Host, Port, ConnectionStates, %%-------------------------------------------------------------------- -spec hello(#server_hello{} | #client_hello{}, #ssl_options{}, #connection_states{} | {inet:port_number(), #session{}, db_handle(), - atom(), #connection_states{}, binary() | undefined}, + atom(), #connection_states{}, + binary() | undefined}, boolean()) -> - {tls_record:tls_version(), session_id(), #connection_states{}, binary() | undefined}| - {tls_record:tls_version(), {resumed | new, #session{}}, #connection_states{}, - [binary()] | undefined, - [ssl_handshake:oid()] | undefined, [ssl_handshake:oid()] | undefined} | + {tls_record:tls_version(), session_id(), + #connection_states{}, alpn | npn, binary() | undefined}| + {tls_record:tls_version(), {resumed | new, #session{}}, + #connection_states{}, binary() | undefined, + #hello_extensions{}} | #alert{}. %% %% Description: Handles a recieved hello message @@ -246,8 +248,10 @@ handle_client_hello_extensions(Version, Type, Random, CipherSuites, try ssl_handshake:handle_client_hello_extensions(tls_record, Random, CipherSuites, HelloExt, Version, SslOpts, Session0, ConnectionStates0, Renegotiation) of - {Session, ConnectionStates, ServerHelloExt} -> - {Version, {Type, Session}, ConnectionStates, ServerHelloExt} + #alert{} = Alert -> + Alert; + {Session, ConnectionStates, Protocol, ServerHelloExt} -> + {Version, {Type, Session}, ConnectionStates, Protocol, ServerHelloExt} catch throw:Alert -> Alert end. @@ -260,7 +264,7 @@ handle_server_hello_extensions(Version, SessionId, Random, CipherSuite, SslOpt, ConnectionStates0, Renegotiation) of #alert{} = Alert -> Alert; - {ConnectionStates, Protocol} -> - {Version, SessionId, ConnectionStates, Protocol} + {ConnectionStates, ProtoExt, Protocol} -> + {Version, SessionId, ConnectionStates, ProtoExt, Protocol} end. diff --git a/lib/ssl/test/Makefile b/lib/ssl/test/Makefile index 09cc5981e7..8c45a788a4 100644 --- a/lib/ssl/test/Makefile +++ b/lib/ssl/test/Makefile @@ -36,6 +36,7 @@ VSN=$(GS_VSN) MODULES = \ ssl_test_lib \ + ssl_alpn_handshake_SUITE \ ssl_basic_SUITE \ ssl_bench_SUITE \ ssl_cipher_SUITE \ diff --git a/lib/ssl/test/ssl_alpn_handshake_SUITE.erl b/lib/ssl/test/ssl_alpn_handshake_SUITE.erl new file mode 100644 index 0000000000..ccd70fa605 --- /dev/null +++ b/lib/ssl/test/ssl_alpn_handshake_SUITE.erl @@ -0,0 +1,414 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2008-2015. All Rights Reserved. +%% +%% The contents of this file are subject to the Erlang Public License, +%% Version 1.1, (the "License"); you may not use this file except in +%% compliance with the License. You should have received a copy of the +%% Erlang Public License along with this software. If not, it can be +%% retrieved online at http://www.erlang.org/. +%% +%% Software distributed under the License is distributed on an "AS IS" +%% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +%% the License for the specific language governing rights and limitations +%% under the License. +%% +%% %CopyrightEnd% +%% + +%% +-module(ssl_alpn_handshake_SUITE). + +%% Note: This directive should only be used in test suites. +-compile(export_all). +-include_lib("common_test/include/ct.hrl"). + +-define(SLEEP, 500). + +%%-------------------------------------------------------------------- +%% Common Test interface functions ----------------------------------- +%%-------------------------------------------------------------------- + +suite() -> [{ct_hooks,[ts_install_cth]}]. + +all() -> + [{group, 'tlsv1.2'}, + {group, 'tlsv1.1'}, + {group, 'tlsv1'}, + {group, 'sslv3'}]. + +groups() -> + [ + {'tlsv1.2', [], alpn_tests()}, + {'tlsv1.1', [], alpn_tests()}, + {'tlsv1', [], alpn_tests()}, + {'sslv3', [], alpn_not_supported()} + ]. + +alpn_tests() -> + [empty_protocols_are_not_allowed, + protocols_must_be_a_binary_list, + empty_client, + empty_server, + empty_client_empty_server, + no_matching_protocol, + client_alpn_and_server_alpn, + client_alpn_and_server_no_support, + client_no_support_and_server_alpn, + client_alpn_npn_and_server_alpn, + client_alpn_npn_and_server_alpn_npn, + client_alpn_and_server_alpn_npn, + client_renegotiate, + session_reused + ]. + +alpn_not_supported() -> + [alpn_not_supported_client, + alpn_not_supported_server + ]. + +init_per_suite(Config) -> + catch crypto:stop(), + try crypto:start() of + ok -> + ssl:start(), + Result = + (catch make_certs:all(?config(data_dir, Config), + ?config(priv_dir, Config))), + ct:log("Make certs ~p~n", [Result]), + ssl_test_lib:cert_options(Config) + catch _:_ -> + {skip, "Crypto did not start"} + end. + +end_per_suite(_Config) -> + ssl:stop(), + application:unload(ssl), + application:stop(crypto). + + +init_per_group(GroupName, Config) -> + case ssl_test_lib:is_tls_version(GroupName) of + true -> + case ssl_test_lib:sufficient_crypto_support(GroupName) of + true -> + ssl_test_lib:init_tls_version(GroupName), + Config; + false -> + {skip, "Missing crypto support"} + end; + _ -> + ssl:start(), + Config + end. + +end_per_group(_GroupName, Config) -> + Config. + +%%-------------------------------------------------------------------- +%% Test Cases -------------------------------------------------------- +%%-------------------------------------------------------------------- + +empty_protocols_are_not_allowed(Config) when is_list(Config) -> + {error, {options, {alpn_preferred_protocols, {invalid_protocol, <<>>}}}} + = (catch ssl:listen(9443, + [{alpn_preferred_protocols, [<<"foo/1">>, <<"">>]}])), + {error, {options, {alpn_advertised_protocols, {invalid_protocol, <<>>}}}} + = (catch ssl:connect({127,0,0,1}, 9443, + [{alpn_advertised_protocols, [<<"foo/1">>, <<"">>]}])). + +%-------------------------------------------------------------------------------- + +protocols_must_be_a_binary_list(Config) when is_list(Config) -> + Option1 = {alpn_preferred_protocols, hello}, + {error, {options, Option1}} = (catch ssl:listen(9443, [Option1])), + Option2 = {alpn_preferred_protocols, [<<"foo/1">>, hello]}, + {error, {options, {alpn_preferred_protocols, {invalid_protocol, hello}}}} + = (catch ssl:listen(9443, [Option2])), + Option3 = {alpn_advertised_protocols, hello}, + {error, {options, Option3}} = (catch ssl:connect({127,0,0,1}, 9443, [Option3])), + Option4 = {alpn_advertised_protocols, [<<"foo/1">>, hello]}, + {error, {options, {alpn_advertised_protocols, {invalid_protocol, hello}}}} + = (catch ssl:connect({127,0,0,1}, 9443, [Option4])). + +%-------------------------------------------------------------------------------- + +empty_client(Config) when is_list(Config) -> + run_failing_handshake(Config, + [{alpn_advertised_protocols, []}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"spdy/3">>, <<"http/2">>]}], + {connect_failed,{tls_alert,"no application protocol"}}). + +%-------------------------------------------------------------------------------- + +empty_server(Config) when is_list(Config) -> + run_failing_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}], + [{alpn_preferred_protocols, []}], + {connect_failed,{tls_alert,"no application protocol"}}). + +%-------------------------------------------------------------------------------- + +empty_client_empty_server(Config) when is_list(Config) -> + run_failing_handshake(Config, + [{alpn_advertised_protocols, []}], + [{alpn_preferred_protocols, []}], + {connect_failed,{tls_alert,"no application protocol"}}). + +%-------------------------------------------------------------------------------- + +no_matching_protocol(Config) when is_list(Config) -> + run_failing_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"spdy/3">>, <<"http/2">>]}], + {connect_failed,{tls_alert,"no application protocol"}}). + +%-------------------------------------------------------------------------------- + +client_alpn_and_server_alpn(Config) when is_list(Config) -> + run_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}], + {ok, <<"http/1.1">>}). + +%-------------------------------------------------------------------------------- + +client_alpn_and_server_no_support(Config) when is_list(Config) -> + run_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}], + [], + {error, protocol_not_negotiated}). + +%-------------------------------------------------------------------------------- + +client_no_support_and_server_alpn(Config) when is_list(Config) -> + run_handshake(Config, + [], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}], + {error, protocol_not_negotiated}). + +%-------------------------------------------------------------------------------- + +client_alpn_npn_and_server_alpn(Config) when is_list(Config) -> + run_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}, + {client_preferred_next_protocols, {client, [<<"spdy/2">>], <<"spdy/3">>}}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}], + {ok, <<"http/1.1">>}). + +%-------------------------------------------------------------------------------- + +client_alpn_npn_and_server_alpn_npn(Config) when is_list(Config) -> + run_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}, + {client_preferred_next_protocols, {client, [<<"spdy/2">>], <<"spdy/3">>}}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}, + {next_protocols_advertised, [<<"spdy/2">>, <<"http/1.0">>]}], + {ok, <<"http/1.1">>}). + +%-------------------------------------------------------------------------------- + +client_alpn_and_server_alpn_npn(Config) when is_list(Config) -> + run_handshake(Config, + [{alpn_advertised_protocols, [<<"http/1.0">>, <<"http/1.1">>]}], + [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}, + {next_protocols_advertised, [<<"spdy/2">>, <<"http/1.0">>]}], + {ok, <<"http/1.1">>}). + +%-------------------------------------------------------------------------------- + +client_renegotiate(Config) when is_list(Config) -> + Data = "hello world", + + ClientOpts0 = ?config(client_opts, Config), + ClientOpts = [{alpn_advertised_protocols, [<<"http/1.0">>]}] ++ ClientOpts0, + ServerOpts0 = ?config(server_opts, Config), + ServerOpts = [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}] ++ ServerOpts0, + ExpectedProtocol = {ok, <<"http/1.0">>}, + + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, ssl_receive_and_assert_alpn, [ExpectedProtocol, Data]}}, + {options, ServerOpts}]), + + Port = ssl_test_lib:inet_port(Server), + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, assert_alpn_and_renegotiate_and_send_data, [ExpectedProtocol, Data]}}, + {options, ClientOpts}]), + + ssl_test_lib:check_result(Server, ok, Client, ok). + +%-------------------------------------------------------------------------------- + +session_reused(Config) when is_list(Config)-> + ClientOpts0 = ?config(client_opts, Config), + ClientOpts = [{alpn_advertised_protocols, [<<"http/1.0">>]}] ++ ClientOpts0, + ServerOpts0 = ?config(server_opts, Config), + ServerOpts = [{alpn_preferred_protocols, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}] ++ ServerOpts0, + + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, session_info_result, []}}, + {options, ServerOpts}]), + + Port = ssl_test_lib:inet_port(Server), + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {ssl_test_lib, no_result_msg, []}}, + {options, ClientOpts}]), + + SessionInfo = + receive + {Server, Info} -> + Info + end, + + Server ! {listen, {mfa, {ssl_test_lib, no_result, []}}}, + + %% Make sure session is registered + ct:sleep(?SLEEP), + + Client1 = + ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, session_info_result, []}}, + {from, self()}, {options, ClientOpts}]), + + receive + {Client1, SessionInfo} -> + ok; + {Client1, Other} -> + ct:fail(Other) + end, + + ssl_test_lib:close(Server), + ssl_test_lib:close(Client), + ssl_test_lib:close(Client1). + +%-------------------------------------------------------------------------------- + +alpn_not_supported_client(Config) when is_list(Config) -> + ClientOpts0 = ?config(client_opts, Config), + PrefProtocols = {client_preferred_next_protocols, + {client, [<<"http/1.0">>], <<"http/1.1">>}}, + ClientOpts = [PrefProtocols] ++ ClientOpts0, + {ClientNode, _ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Client = ssl_test_lib:start_client_error([{node, ClientNode}, + {port, 8888}, {host, Hostname}, + {from, self()}, {options, ClientOpts}]), + + ssl_test_lib:check_result(Client, {error, + {options, + {not_supported_in_sslv3, PrefProtocols}}}). + +%-------------------------------------------------------------------------------- + +alpn_not_supported_server(Config) when is_list(Config)-> + ServerOpts0 = ?config(server_opts, Config), + AdvProtocols = {next_protocols_advertised, [<<"spdy/2">>, <<"http/1.1">>, <<"http/1.0">>]}, + ServerOpts = [AdvProtocols] ++ ServerOpts0, + + {error, {options, {not_supported_in_sslv3, AdvProtocols}}} = ssl:listen(0, ServerOpts). + +%%-------------------------------------------------------------------- +%% Internal functions ------------------------------------------------ +%%-------------------------------------------------------------------- + +run_failing_handshake(Config, ClientExtraOpts, ServerExtraOpts, ExpectedResult) -> + ClientOpts = ClientExtraOpts ++ ?config(client_opts, Config), + ServerOpts = ServerExtraOpts ++ ?config(server_opts, Config), + + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, placeholder, []}}, + {options, ServerOpts}]), + + Port = ssl_test_lib:inet_port(Server), + ExpectedResult + = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, placeholder, []}}, + {options, ClientOpts}]). + +run_handshake(Config, ClientExtraOpts, ServerExtraOpts, ExpectedProtocol) -> + Data = "hello world", + + ClientOpts0 = ?config(client_opts, Config), + ClientOpts = ClientExtraOpts ++ ClientOpts0, + ServerOpts0 = ?config(server_opts, Config), + ServerOpts = ServerExtraOpts ++ ServerOpts0, + + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, ssl_receive_and_assert_alpn, [ExpectedProtocol, Data]}}, + {options, ServerOpts}]), + + Port = ssl_test_lib:inet_port(Server), + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, ssl_send_and_assert_alpn, [ExpectedProtocol, Data]}}, + {options, ClientOpts}]), + + ssl_test_lib:check_result(Server, ok, Client, ok). + +assert_alpn(Socket, Protocol) -> + ct:log("Negotiated Protocol ~p, Expecting: ~p ~n", + [ssl:negotiated_protocol(Socket), Protocol]), + Protocol = ssl:negotiated_protocol(Socket). + +assert_alpn_and_renegotiate_and_send_data(Socket, Protocol, Data) -> + assert_alpn(Socket, Protocol), + ct:log("Renegotiating ~n", []), + ok = ssl:renegotiate(Socket), + ssl:send(Socket, Data), + assert_alpn(Socket, Protocol), + ok. + +ssl_send_and_assert_alpn(Socket, Protocol, Data) -> + assert_alpn(Socket, Protocol), + ssl_send(Socket, Data). + +ssl_receive_and_assert_alpn(Socket, Protocol, Data) -> + assert_alpn(Socket, Protocol), + ssl_receive(Socket, Data). + +ssl_send(Socket, Data) -> + ct:log("Connection info: ~p~n", + [ssl:connection_info(Socket)]), + ssl:send(Socket, Data). + +ssl_receive(Socket, Data) -> + ssl_receive(Socket, Data, []). + +ssl_receive(Socket, Data, Buffer) -> + ct:log("Connection info: ~p~n", + [ssl:connection_info(Socket)]), + receive + {ssl, Socket, MoreData} -> + ct:log("Received ~p~n",[MoreData]), + NewBuffer = Buffer ++ MoreData, + case NewBuffer of + Data -> + ssl:send(Socket, "Got it"), + ok; + _ -> + ssl_receive(Socket, Data, NewBuffer) + end; + Other -> + ct:fail({unexpected_message, Other}) + after 4000 -> + ct:fail({did_not_get, Data}) + end. + +connection_info_result(Socket) -> + ssl:connection_info(Socket). diff --git a/lib/ssl/test/ssl_npn_handshake_SUITE.erl b/lib/ssl/test/ssl_npn_handshake_SUITE.erl index 30c0a67a36..326f907e66 100644 --- a/lib/ssl/test/ssl_npn_handshake_SUITE.erl +++ b/lib/ssl/test/ssl_npn_handshake_SUITE.erl @@ -172,7 +172,7 @@ no_client_negotiate_but_server_supports_npn(Config) when is_list(Config) -> run_npn_handshake(Config, [], [{next_protocols_advertised, [<<"spdy/1">>, <<"http/1.1">>, <<"http/1.0">>]}], - {error, next_protocol_not_negotiated}). + {error, protocol_not_negotiated}). %-------------------------------------------------------------------------------- @@ -180,7 +180,7 @@ client_negotiate_server_does_not_support(Config) when is_list(Config) -> run_npn_handshake(Config, [{client_preferred_next_protocols, {client, [<<"spdy/2">>], <<"http/1.1">>}}], [], - {error, next_protocol_not_negotiated}). + {error, protocol_not_negotiated}). %-------------------------------------------------------------------------------- renegotiate_from_client_after_npn_handshake(Config) when is_list(Config) -> @@ -311,8 +311,8 @@ run_npn_handshake(Config, ClientExtraOpts, ServerExtraOpts, ExpectedProtocol) -> assert_npn(Socket, Protocol) -> ct:log("Negotiated Protocol ~p, Expecting: ~p ~n", - [ssl:negotiated_next_protocol(Socket), Protocol]), - Protocol = ssl:negotiated_next_protocol(Socket). + [ssl:negotiated_protocol(Socket), Protocol]), + Protocol = ssl:negotiated_protocol(Socket). assert_npn_and_renegotiate_and_send_data(Socket, Protocol, Data) -> assert_npn(Socket, Protocol), diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index 7d0546210c..d19e3b7fdb 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -1090,6 +1090,8 @@ cipher_restriction(Config0) -> check_sane_openssl_version(Version) -> case {Version, os:cmd("openssl version")} of + {_, "OpenSSL 1.0.2" ++ _} -> + true; {_, "OpenSSL 1.0.1" ++ _} -> true; {'tlsv1.2', "OpenSSL 1.0" ++ _} -> diff --git a/lib/ssl/test/ssl_to_openssl_SUITE.erl b/lib/ssl/test/ssl_to_openssl_SUITE.erl index 942c446ec4..27ee07ffc6 100644 --- a/lib/ssl/test/ssl_to_openssl_SUITE.erl +++ b/lib/ssl/test/ssl_to_openssl_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2014. All Rights Reserved. +%% Copyright Ericsson AB 2008-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -50,9 +50,9 @@ all() -> groups() -> [{basic, [], basic_tests()}, - {'tlsv1.2', [], all_versions_tests() ++ npn_tests()}, - {'tlsv1.1', [], all_versions_tests() ++ npn_tests()}, - {'tlsv1', [], all_versions_tests()++ npn_tests()}, + {'tlsv1.2', [], all_versions_tests() ++ alpn_tests() ++ npn_tests()}, + {'tlsv1.1', [], all_versions_tests() ++ alpn_tests() ++ npn_tests()}, + {'tlsv1', [], all_versions_tests()++ alpn_tests() ++ npn_tests()}, {'sslv3', [], all_versions_tests()}]. basic_tests() -> @@ -79,6 +79,18 @@ all_versions_tests() -> expired_session, ssl2_erlang_server_openssl_client]. +alpn_tests() -> + [erlang_client_alpn_openssl_server_alpn, + erlang_server_alpn_openssl_client_alpn, + erlang_client_alpn_openssl_server, + erlang_client_openssl_server_alpn, + erlang_server_alpn_openssl_client, + erlang_server_openssl_client_alpn, + erlang_client_alpn_openssl_server_alpn_renegotiate, + erlang_server_alpn_openssl_client_alpn_renegotiate, + erlang_client_alpn_npn_openssl_server_alpn_npn, + erlang_server_alpn_npn_openssl_client_alpn_npn]. + npn_tests() -> [erlang_client_openssl_server_npn, erlang_server_openssl_client_npn, @@ -161,6 +173,36 @@ special_init(ssl2_erlang_server_openssl_client, Config) -> check_sane_openssl_sslv2(Config); special_init(TestCase, Config) + when TestCase == erlang_client_alpn_openssl_server_alpn; + TestCase == erlang_server_alpn_openssl_client_alpn; + TestCase == erlang_client_alpn_openssl_server; + TestCase == erlang_client_openssl_server_alpn; + TestCase == erlang_server_alpn_openssl_client; + TestCase == erlang_server_openssl_client_alpn -> + check_openssl_alpn_support(Config); + +special_init(TestCase, Config) + when TestCase == erlang_client_alpn_openssl_server_alpn_renegotiate; + TestCase == erlang_server_alpn_openssl_client_alpn_renegotiate -> + {ok, Version} = application:get_env(ssl, protocol_version), + case check_sane_openssl_renegotaite(Config, Version) of + {skip, _} = Skip -> + Skip; + _ -> + check_openssl_alpn_support(Config) + end; + +special_init(TestCase, Config) + when TestCase == erlang_client_alpn_npn_openssl_server_alpn_npn; + TestCase == erlang_server_alpn_npn_openssl_client_alpn_npn -> + case check_openssl_alpn_support(Config) of + {skip, _} = Skip -> + Skip; + _ -> + check_openssl_npn_support(Config) + end; + +special_init(TestCase, Config) when TestCase == erlang_client_openssl_server_npn; TestCase == erlang_server_openssl_client_npn; TestCase == erlang_server_openssl_client_npn_only_server; @@ -179,6 +221,7 @@ special_init(TestCase, Config) _ -> check_openssl_npn_support(Config) end; + special_init(_, Config) -> Config. @@ -248,7 +291,7 @@ basic_erlang_server_openssl_client(Config) when is_list(Config) -> Port = ssl_test_lib:inet_port(Server), Cmd = "openssl s_client -port " ++ integer_to_list(Port) ++ - " -host localhost" ++ workaround_openssl_s_clinent(), + " -host localhost" ++ workaround_openssl_s_client(), ct:log("openssl cmd: ~p~n", [Cmd]), @@ -924,6 +967,128 @@ ssl2_erlang_server_openssl_client(Config) when is_list(Config) -> process_flag(trap_exit, false). %%-------------------------------------------------------------------- + +erlang_client_alpn_openssl_server_alpn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_client_and_openssl_server_for_alpn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------- + +erlang_server_alpn_openssl_client_alpn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_server_and_openssl_client_for_alpn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------------- + +erlang_client_alpn_openssl_server(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_client_and_openssl_server_with_opts(Config, + [{alpn_advertised_protocols, [<<"spdy/2">>]}], + "", + Data, fun(Server, OpensslPort) -> + true = port_command(OpensslPort, Data), + ssl_test_lib:check_result(Server, ok) + end), + ok. + +%%-------------------------------------------------------------------------- + +erlang_client_openssl_server_alpn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_client_and_openssl_server_with_opts(Config, + [], + "-alpn spdy/2", + Data, fun(Server, OpensslPort) -> + true = port_command(OpensslPort, Data), + ssl_test_lib:check_result(Server, ok) + end), + ok. + +%%-------------------------------------------------------------------------- + +erlang_server_alpn_openssl_client(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_server_and_openssl_client_with_opts(Config, + [{alpn_advertised_protocols, [<<"spdy/2">>]}], + "", + Data, fun(Server, OpensslPort) -> + true = port_command(OpensslPort, Data), + ssl_test_lib:check_result(Server, ok) + end), + ok. + +%%-------------------------------------------------------------------------- + +erlang_server_openssl_client_alpn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_server_and_openssl_client_with_opts(Config, + [], + "-alpn spdy/2", + Data, fun(Server, OpensslPort) -> + true = port_command(OpensslPort, Data), + ssl_test_lib:check_result(Server, ok) + end), + ok. + +%%-------------------------------------------------------------------- + +erlang_client_alpn_openssl_server_alpn_renegotiate(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_client_and_openssl_server_for_alpn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, ?OPENSSL_RENEGOTIATE), + ct:sleep(?SLEEP), + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------- + +erlang_server_alpn_openssl_client_alpn_renegotiate(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_server_and_openssl_client_for_alpn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, ?OPENSSL_RENEGOTIATE), + ct:sleep(?SLEEP), + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------- + +erlang_client_alpn_npn_openssl_server_alpn_npn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_client_and_openssl_server_for_alpn_npn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------- + +erlang_server_alpn_npn_openssl_client_alpn_npn(Config) when is_list(Config) -> + Data = "From openssl to erlang", + start_erlang_server_and_openssl_client_for_alpn_npn_negotiation(Config, Data, fun(Client, OpensslPort) -> + true = port_command(OpensslPort, Data), + + ssl_test_lib:check_result(Client, ok) + end), + ok. + +%%-------------------------------------------------------------------- erlang_client_openssl_server_npn() -> [{doc,"Test erlang client with openssl server doing npn negotiation"}]. @@ -1139,6 +1304,142 @@ start_erlang_client_and_openssl_server_with_opts(Config, ErlangClientOpts, Opens ssl_test_lib:close(Client), process_flag(trap_exit, false). +start_erlang_client_and_openssl_server_for_alpn_negotiation(Config, Data, Callback) -> + process_flag(trap_exit, true), + ServerOpts = ?config(server_opts, Config), + ClientOpts0 = ?config(client_opts, Config), + ClientOpts = [{alpn_advertised_protocols, [<<"spdy/2">>]} | ClientOpts0], + + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + Data = "From openssl to erlang", + + Port = ssl_test_lib:inet_port(node()), + CertFile = proplists:get_value(certfile, ServerOpts), + KeyFile = proplists:get_value(keyfile, ServerOpts), + Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), + + Cmd = "openssl s_server -msg -alpn http/1.1,spdy/2 -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ + " -cert " ++ CertFile ++ " -key " ++ KeyFile, + + ct:log("openssl cmd: ~p~n", [Cmd]), + + OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + + ssl_test_lib:wait_for_openssl_server(), + + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, + erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, + {options, ClientOpts}]), + + Callback(Client, OpensslPort), + + %% Clean close down! Server needs to be closed first !! + ssl_test_lib:close_port(OpensslPort), + + ssl_test_lib:close(Client), + process_flag(trap_exit, false). + +start_erlang_server_and_openssl_client_for_alpn_negotiation(Config, Data, Callback) -> + process_flag(trap_exit, true), + ServerOpts0 = ?config(server_opts, Config), + ServerOpts = [{alpn_preferred_protocols, [<<"spdy/2">>]} | ServerOpts0], + + {_, ServerNode, _} = ssl_test_lib:run_where(Config), + + + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, + {options, ServerOpts}]), + Port = ssl_test_lib:inet_port(Server), + Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), + Cmd = "openssl s_client -alpn http/1.0,spdy/2 -msg -port " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ + " -host localhost", + + ct:log("openssl cmd: ~p~n", [Cmd]), + + OpenSslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + + Callback(Server, OpenSslPort), + + ssl_test_lib:close(Server), + + ssl_test_lib:close_port(OpenSslPort), + process_flag(trap_exit, false). + +start_erlang_client_and_openssl_server_for_alpn_npn_negotiation(Config, Data, Callback) -> + process_flag(trap_exit, true), + ServerOpts = ?config(server_opts, Config), + ClientOpts0 = ?config(client_opts, Config), + ClientOpts = [{alpn_advertised_protocols, [<<"spdy/2">>]}, + {client_preferred_next_protocols, {client, [<<"spdy/3">>, <<"http/1.1">>]}} | ClientOpts0], + + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + Data = "From openssl to erlang", + + Port = ssl_test_lib:inet_port(node()), + CertFile = proplists:get_value(certfile, ServerOpts), + KeyFile = proplists:get_value(keyfile, ServerOpts), + Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), + + Cmd = "openssl s_server -msg -alpn http/1.1,spdy/2 -nextprotoneg spdy/3 -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ + " -cert " ++ CertFile ++ " -key " ++ KeyFile, + + ct:log("openssl cmd: ~p~n", [Cmd]), + + OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + + ssl_test_lib:wait_for_openssl_server(), + + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, + erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, + {options, ClientOpts}]), + + Callback(Client, OpensslPort), + + %% Clean close down! Server needs to be closed first !! + ssl_test_lib:close_port(OpensslPort), + + ssl_test_lib:close(Client), + process_flag(trap_exit, false). + +start_erlang_server_and_openssl_client_for_alpn_npn_negotiation(Config, Data, Callback) -> + process_flag(trap_exit, true), + ServerOpts0 = ?config(server_opts, Config), + ServerOpts = [{alpn_preferred_protocols, [<<"spdy/2">>]}, + {next_protocols_advertised, [<<"spdy/3">>, <<"http/1.1">>]} | ServerOpts0], + + {_, ServerNode, _} = ssl_test_lib:run_where(Config), + + + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, + {options, ServerOpts}]), + Port = ssl_test_lib:inet_port(Server), + Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), + Cmd = "openssl s_client -alpn http/1.1,spdy/2 -nextprotoneg spdy/3 -msg -port " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ + " -host localhost", + + ct:log("openssl cmd: ~p~n", [Cmd]), + + OpenSslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + + Callback(Server, OpenSslPort), + + ssl_test_lib:close(Server), + + ssl_test_lib:close_port(OpenSslPort), + process_flag(trap_exit, false). + start_erlang_client_and_openssl_server_for_npn_negotiation(Config, Data, Callback) -> process_flag(trap_exit, true), ServerOpts = ?config(server_opts, Config), @@ -1167,7 +1468,7 @@ start_erlang_client_and_openssl_server_for_npn_negotiation(Config, Data, Callbac {host, Hostname}, {from, self()}, {mfa, {?MODULE, - erlang_ssl_receive_and_assert_npn, [<<"spdy/2">>, Data]}}, + erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, {options, ClientOpts}]), Callback(Client, OpensslPort), @@ -1188,7 +1489,7 @@ start_erlang_server_and_openssl_client_for_npn_negotiation(Config, Data, Callbac Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, {from, self()}, - {mfa, {?MODULE, erlang_ssl_receive_and_assert_npn, [<<"spdy/2">>, Data]}}, + {mfa, {?MODULE, erlang_ssl_receive_and_assert_negotiated_protocol, [<<"spdy/2">>, Data]}}, {options, ServerOpts}]), Port = ssl_test_lib:inet_port(Server), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), @@ -1236,10 +1537,10 @@ start_erlang_server_and_openssl_client_with_opts(Config, ErlangServerOpts, OpenS process_flag(trap_exit, false). -erlang_ssl_receive_and_assert_npn(Socket, Protocol, Data) -> - {ok, Protocol} = ssl:negotiated_next_protocol(Socket), +erlang_ssl_receive_and_assert_negotiated_protocol(Socket, Protocol, Data) -> + {ok, Protocol} = ssl:negotiated_protocol(Socket), erlang_ssl_receive(Socket, Data), - {ok, Protocol} = ssl:negotiated_next_protocol(Socket), + {ok, Protocol} = ssl:negotiated_protocol(Socket), ok. erlang_ssl_receive(Socket, Data) -> @@ -1297,6 +1598,15 @@ check_openssl_npn_support(Config) -> Config end. +check_openssl_alpn_support(Config) -> + HelpText = os:cmd("openssl s_client --help"), + case string:str(HelpText, "alpn") of + 0 -> + {skip, "Openssl not compiled with alpn support"}; + _ -> + Config + end. + check_sane_openssl_renegotaite(Config, Version) when Version == 'tlsv1.1'; Version == 'tlsv1.2' -> case os:cmd("openssl version") of @@ -1348,7 +1658,7 @@ supports_sslv2(Port) -> true end. -workaround_openssl_s_clinent() -> +workaround_openssl_s_client() -> %% http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683159 %% https://bugs.archlinux.org/task/33919 %% Bug seems to manifests it self if TLS version is not @@ -1362,6 +1672,8 @@ workaround_openssl_s_clinent() -> " -no_tls1_2 "; "OpenSSL 1.0.1f" ++ _ -> " -no_tls1_2 "; - _ -> + "OpenSSL 1.0.1l" ++ _ -> + " -cipher AES256-SHA"; + _ -> "" end. diff --git a/lib/stdlib/doc/src/calendar.xml b/lib/stdlib/doc/src/calendar.xml index e32a639b81..d8193a9ec2 100644 --- a/lib/stdlib/doc/src/calendar.xml +++ b/lib/stdlib/doc/src/calendar.xml @@ -270,7 +270,8 @@ <fsummary>Convert now to local date and time</fsummary> <desc> <p>This function returns local date and time converted from - the return value from <c>erlang:now()</c>.</p> + the return value from + <seealso marker="erts:erlang#timestamp/0"><c>erlang:timestamp/0</c></seealso>.</p> </desc> </func> <func> @@ -279,7 +280,8 @@ <fsummary>Convert now to date and time</fsummary> <desc> <p>This function returns Universal Coordinated Time (UTC) - converted from the return value from <c>erlang:now()</c>.</p> + converted from the return value from + <seealso marker="erts:erlang#timestamp/0"><c>erlang:timestamp/0</c></seealso>.</p> </desc> </func> <func> diff --git a/lib/stdlib/doc/src/ets.xml b/lib/stdlib/doc/src/ets.xml index 902a921fbf..6b9524ef63 100644 --- a/lib/stdlib/doc/src/ets.xml +++ b/lib/stdlib/doc/src/ets.xml @@ -1618,14 +1618,18 @@ true</pre> </func> <func> <name name="update_counter" arity="3" clause_i="1"/> + <name name="update_counter" arity="4" clause_i="1"/> <name name="update_counter" arity="3" clause_i="2"/> + <name name="update_counter" arity="4" clause_i="2"/> <name name="update_counter" arity="3" clause_i="3"/> + <name name="update_counter" arity="4" clause_i="3"/> <type variable="Tab"/> <type variable="Key"/> <type variable="UpdateOp" name_i="1"/> <type variable="Pos" name_i="1"/> <type variable="Threshold" name_i="1"/> <type variable="SetValue" name_i="1"/> + <type variable="Default"/> <fsummary>Update a counter object in an ETS table.</fsummary> <desc> <p>This function provides an efficient way to update one or more @@ -1667,12 +1671,22 @@ true</pre> <seealso marker="#lookup/2">lookup/2</seealso> and <seealso marker="#new/2">new/2</seealso> for details on the difference).</p> + <p>If a default object <c><anno>Default</anno></c> is given, it is used + as the object to be updated if the key is missing from the table. The + value in place of the key is ignored and replaced by the proper key + value. The return value is as if the default object had not been used, + that is a single updated element or a list of them.</p> <p>The function will fail with reason <c>badarg</c> if:</p> <list type="bulleted"> <item>the table is not of type <c>set</c> or <c>ordered_set</c>,</item> - <item>no object with the right key exists,</item> + <item>no object with the right key exists and no default object were + supplied,</item> <item>the object has the wrong arity,</item> + <item>the default object arity is smaller than + <c><![CDATA[<keypos>]]></c></item> + <item>any field from the default object being updated is not an + integer</item> <item>the element to update is not an integer,</item> <item>the element to update is also the key, or,</item> <item>any of <c><anno>Pos</anno></c>, <c><anno>Incr</anno></c>, <c><anno>Threshold</anno></c> or diff --git a/lib/stdlib/doc/src/file_sorter.xml b/lib/stdlib/doc/src/file_sorter.xml index 16572df3c5..c069333c29 100644 --- a/lib/stdlib/doc/src/file_sorter.xml +++ b/lib/stdlib/doc/src/file_sorter.xml @@ -105,9 +105,9 @@ <c>file:get_cwd()</c> is used instead. The names of temporary files are derived from the Erlang nodename (<c>node()</c>), the process identifier of the current Erlang - emulator (<c>os:getpid()</c>), and a timestamp - (<c>erlang:now()</c>); a typical name would be - <c>fs_mynode@myhost_1763_1043_337000_266005.17</c>, where + emulator (<c>os:getpid()</c>), and a unique integer + (<c>erlang:unique_integer([positive])</c>); a typical name would be + <c>fs_mynode@myhost_1763_4711.17</c>, where <c>17</c> is a sequence number. Existing files will be overwritten. Temporary files are deleted unless some uncaught EXIT signal occurs. diff --git a/lib/stdlib/doc/src/proc_lib.xml b/lib/stdlib/doc/src/proc_lib.xml index f27a974242..9a0ff85038 100644 --- a/lib/stdlib/doc/src/proc_lib.xml +++ b/lib/stdlib/doc/src/proc_lib.xml @@ -173,7 +173,7 @@ <name name="init_ack" arity="2"/> <fsummary>Used by a process when it has started.</fsummary> <desc> - <p>This function must used by a process that has been started by + <p>This function must be used by a process that has been started by a <seealso marker="#start/3">start[_link]/3,4,5</seealso> function. It tells <c><anno>Parent</anno></c> that the process has initialized itself, has started, or has failed to initialize diff --git a/lib/stdlib/doc/src/random.xml b/lib/stdlib/doc/src/random.xml index e001058e19..2cc621ffc3 100644 --- a/lib/stdlib/doc/src/random.xml +++ b/lib/stdlib/doc/src/random.xml @@ -70,12 +70,11 @@ <desc> <p>Seeds random number generation with integer values in the process dictionary, and returns the old state.</p> - <p>One way of obtaining a seed is to use the BIF <c>now/0</c>:</p> + <p>One easy way of obtaining a unique value to seed with is to:</p> <code type="none"> - ... - {A1,A2,A3} = now(), - random:seed(A1, A2, A3), - ...</code> + random:seed(<seealso marker="erts:erlang#phash2/1">erlang:phash2</seealso>([<seealso marker="erts:erlang#node/0">node()</seealso>]), + <seealso marker="erts:erlang#monotonic_time/0">erlang:monotonic_time()</seealso>, + <seealso marker="erts:erlang#unique_integer/0">erlang:unique_integer()</seealso>)</code> </desc> </func> <func> diff --git a/lib/stdlib/doc/src/timer.xml b/lib/stdlib/doc/src/timer.xml index eca9a72d36..791a29689e 100644 --- a/lib/stdlib/doc/src/timer.xml +++ b/lib/stdlib/doc/src/timer.xml @@ -217,12 +217,14 @@ </func> <func> <name name="now_diff" arity="2"/> - <fsummary>Calculate time difference between <c>now/0</c>timestamps</fsummary> + <fsummary>Calculate time difference between timestamps</fsummary> <type_desc variable="Tdiff">In microseconds</type_desc> <desc> <p>Calculates the time difference <c><anno>Tdiff</anno> = <anno>T2</anno> - <anno>T1</anno></c> in - <em>microseconds</em>, where <c><anno>T1</anno></c> and <c><anno>T2</anno></c> probably - are timestamp tuples returned from <c>erlang:now/0</c>.</p> + <em>microseconds</em>, where <c><anno>T1</anno></c> and <c><anno>T2</anno></c> + are timestamp tuples on the same format as returned from + <seealso marker="erts:erlang#timestamp/0"><c>erlang:timestamp/0</c></seealso>, + or <seealso marker="kernel:os#timestamp/0"><c>os:timestamp/0</c></seealso>.</p> </desc> </func> <func> @@ -234,7 +236,7 @@ </func> <func> <name name="minutes" arity="1"/> - <fsummary>Converts <c>Minutes</c>to <c>Milliseconds</c>.</fsummary> + <fsummary>Converts <c>Minutes</c> to <c>Milliseconds</c>.</fsummary> <desc> <p>Return the number of milliseconds in <c><anno>Minutes</anno></c>.</p> </desc> diff --git a/lib/stdlib/doc/src/zip.xml b/lib/stdlib/doc/src/zip.xml index 48b376743d..d201e81a79 100644 --- a/lib/stdlib/doc/src/zip.xml +++ b/lib/stdlib/doc/src/zip.xml @@ -135,6 +135,12 @@ <p>These options are described in <seealso marker="#zip_options">create/3</seealso>.</p> </desc> </datatype> + <datatype> + <name name="handle"/> + <desc> + <p>As returned by <seealso marker="#zip_open/2">zip_open/2</seealso>.</p> + </desc> + </datatype> </datatypes> <funcs> <func> @@ -430,6 +436,8 @@ means that subsequently reading files from the archive will be faster than unzipping files one at a time with <c>unzip</c>.</p> <p>The archive must be closed with <c>zip_close/1</c>.</p> + <p>The <c><anno>ZipHandle</anno></c> will be closed if the + process which originally opened the archive dies.</p> </desc> </func> <func> diff --git a/lib/stdlib/src/calendar.erl b/lib/stdlib/src/calendar.erl index 0320e0cd0e..d08001c933 100644 --- a/lib/stdlib/src/calendar.erl +++ b/lib/stdlib/src/calendar.erl @@ -299,7 +299,7 @@ local_time_to_universal_time_dst(DateTime) -> %% now_to_universal_time(Now) %% now_to_datetime(Now) %% -%% Convert from now() to UTC. +%% Convert from erlang:timestamp() to UTC. %% %% Args: Now = now(); now() = {MegaSec, Sec, MilliSec}, MegaSec = Sec %% = MilliSec = integer() diff --git a/lib/stdlib/src/dets.erl b/lib/stdlib/src/dets.erl index a4bd45ea19..5d365ac962 100644 --- a/lib/stdlib/src/dets.erl +++ b/lib/stdlib/src/dets.erl @@ -1963,7 +1963,7 @@ do_safe_fixtable(Head, Pid, true) -> case Head#head.fixed of false -> link(Pid), - Fixed = {erlang:now(), [{Pid, 1}]}, + Fixed = {utime_now(), [{Pid, 1}]}, Ftab = dets_utils:get_freelists(Head), Head#head{fixed = Fixed, freelists = {Ftab, Ftab}}; {TimeStamp, Counters} -> @@ -3088,14 +3088,14 @@ update_cache(Head, ToAdd) -> {Head1, Found, []}; Cache#cache.wrtime =:= undefined -> %% Empty cache. Schedule a delayed write. - Now = now(), Me = self(), + Now = time_now(), Me = self(), Call = ?DETS_CALL(Me, {delayed_write, Now}), erlang:send_after(Cache#cache.delay, Me, Call), {Head1#head{cache = NewCache#cache{wrtime = Now}}, Found, []}; Size0 =:= 0 -> %% Empty cache that has been written after the %% currently scheduled delayed write. - {Head1#head{cache = NewCache#cache{wrtime = now()}}, Found, []}; + {Head1#head{cache = NewCache#cache{wrtime = time_now()}}, Found, []}; true -> %% Cache is not empty, delayed write has been scheduled. {Head1, Found, []} @@ -3158,11 +3158,7 @@ delayed_write(Head, WrTime) -> Head#head{cache = NewCache}; true -> %% Yes, schedule a new delayed write. - {MS1,S1,M1} = WrTime, - {MS2,S2,M2} = LastWrTime, - WrT = M1+1000000*(S1+1000000*MS1), - LastWrT = M2+1000000*(S2+1000000*MS2), - When = round((LastWrT - WrT)/1000), Me = self(), + When = round((LastWrTime - WrTime)/1000), Me = self(), Call = ?DETS_CALL(Me, {delayed_write, LastWrTime}), erlang:send_after(When, Me, Call), Head @@ -3274,6 +3270,16 @@ err(Error) -> Error end. +-compile({inline, [time_now/0]}). +time_now() -> + erlang:monotonic_time(1000000). + +-compile({inline, [utime_now/0]}). +utime_now() -> + Time = time_now(), + UniqueCounter = erlang:unique_integer([monotonic]), + {Time, UniqueCounter}. + %%%%%%%%%%%%%%%%% DEBUG functions %%%%%%%%%%%%%%%% file_info(FileName) -> diff --git a/lib/stdlib/src/dets_utils.erl b/lib/stdlib/src/dets_utils.erl index 6c176ad513..26e22dbd5b 100644 --- a/lib/stdlib/src/dets_utils.erl +++ b/lib/stdlib/src/dets_utils.erl @@ -447,7 +447,7 @@ reset_cache(C) -> WrTime =:= undefined -> WrTime; true -> - now() + erlang:monotonic_time(1000000) end, PK = family(C#cache.cache), NewC = C#cache{cache = [], csize = 0, inserts = 0, wrtime = NewWrTime}, diff --git a/lib/stdlib/src/erl_tar.erl b/lib/stdlib/src/erl_tar.erl index caa3276d09..72bd54fa29 100644 --- a/lib/stdlib/src/erl_tar.erl +++ b/lib/stdlib/src/erl_tar.erl @@ -300,7 +300,7 @@ format_error(Term) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% add1(TarFile, Bin, NameInArchive, Opts) when is_binary(Bin) -> - Now = calendar:now_to_local_time(now()), + Now = calendar:now_to_local_time(erlang:timestamp()), Info = #file_info{size = byte_size(Bin), type = regular, access = read_write, diff --git a/lib/stdlib/src/escript.erl b/lib/stdlib/src/escript.erl index 6bd0eb8a22..90e1f3a8d6 100644 --- a/lib/stdlib/src/escript.erl +++ b/lib/stdlib/src/escript.erl @@ -663,7 +663,8 @@ check_source(S, CheckOnly) -> end. pre_def_macros(File) -> - {MegaSecs, Secs, MicroSecs} = erlang:now(), + {MegaSecs, Secs, MicroSecs} = erlang:timestamp(), + Unique = erlang:unique_integer([positive]), Replace = fun(Char) -> case Char of $\. -> $\_; @@ -675,8 +676,9 @@ pre_def_macros(File) -> CleanBase ++ "__" ++ "escript__" ++ integer_to_list(MegaSecs) ++ "__" ++ - integer_to_list(Secs) ++ "__" ++ - integer_to_list(MicroSecs), + integer_to_list(Secs) ++ "__" ++ + integer_to_list(MicroSecs) ++ "__" ++ + integer_to_list(Unique), Module = list_to_atom(ModuleStr), PreDefMacros = [{'MODULE', Module, redefine}, {'MODULE_STRING', ModuleStr, redefine}], diff --git a/lib/stdlib/src/ets.erl b/lib/stdlib/src/ets.erl index 09c8924650..1df069755d 100644 --- a/lib/stdlib/src/ets.erl +++ b/lib/stdlib/src/ets.erl @@ -72,7 +72,7 @@ select_count/2, select_delete/2, select_reverse/1, select_reverse/2, select_reverse/3, setopts/2, slot/2, take/2, - update_counter/3, update_element/3]). + update_counter/3, update_counter/4, update_element/3]). -spec all() -> [Tab] when Tab :: tab(). @@ -439,6 +439,38 @@ take(_, _) -> update_counter(_, _, _) -> erlang:nif_error(undef). +-spec update_counter(Tab, Key, UpdateOp, Default) -> Result when + Tab :: tab(), + Key :: term(), + UpdateOp :: {Pos, Incr} + | {Pos, Incr, Threshold, SetValue}, + Pos :: integer(), + Incr :: integer(), + Threshold :: integer(), + SetValue :: integer(), + Result :: integer(), + Default :: tuple(); + (Tab, Key, [UpdateOp], Default) -> [Result] when + Tab :: tab(), + Key :: term(), + UpdateOp :: {Pos, Incr} + | {Pos, Incr, Threshold, SetValue}, + Pos :: integer(), + Incr :: integer(), + Threshold :: integer(), + SetValue :: integer(), + Result :: integer(), + Default :: tuple(); + (Tab, Key, Incr, Default) -> Result when + Tab :: tab(), + Key :: term(), + Incr :: integer(), + Result :: integer(), + Default :: tuple(). + +update_counter(_, _, _, _) -> + erlang:nif_error(undef). + -spec update_element(Tab, Key, ElementSpec :: {Pos, Value}) -> boolean() when Tab :: tab(), Key :: term(), diff --git a/lib/stdlib/src/file_sorter.erl b/lib/stdlib/src/file_sorter.erl index 687d72b4bd..61b489513a 100644 --- a/lib/stdlib/src/file_sorter.erl +++ b/lib/stdlib/src/file_sorter.erl @@ -1425,8 +1425,8 @@ tmp_prefix1(Dir, TmpDirOpt) -> U = "_", Node = node(), Pid = os:getpid(), - {MSecs,Secs,MySecs} = now(), - F = lists:concat(["fs_",Node,U,Pid,U,MSecs,U,Secs,U,MySecs,"."]), + Unique = erlang:unique_integer([positive]), + F = lists:concat(["fs_",Node,U,Pid,U,Unique,"."]), TmpDir = case TmpDirOpt of default -> Dir; diff --git a/lib/stdlib/src/io_lib.erl b/lib/stdlib/src/io_lib.erl index e90cda0533..3378d668a5 100644 --- a/lib/stdlib/src/io_lib.erl +++ b/lib/stdlib/src/io_lib.erl @@ -287,6 +287,8 @@ write([H|T], D) -> end; write(F, _D) when is_function(F) -> erlang:fun_to_list(F); +write(Term, D) when is_map(Term) -> + write_map(Term, D); write(T, D) when is_tuple(T) -> if D =:= 1 -> "{...}"; @@ -295,9 +297,7 @@ write(T, D) when is_tuple(T) -> [write(element(1, T), D-1)| write_tail(tl(tuple_to_list(T)), D-1, $,)], $}] - end; -%write(Term, D) when is_map(Term) -> write_map(Term, D); -write(Term, D) -> write_map(Term, D). + end. %% write_tail(List, Depth, CharacterBeforeDots) %% Test the terminating case first as this looks better with depth. diff --git a/lib/stdlib/src/otp_internal.erl b/lib/stdlib/src/otp_internal.erl index 4a338798d0..540c1cac9c 100644 --- a/lib/stdlib/src/otp_internal.erl +++ b/lib/stdlib/src/otp_internal.erl @@ -59,6 +59,11 @@ obsolete_1(erl_eval, arg_list, 3) -> obsolete_1(erlang, hash, 2) -> {deprecated, {erlang, phash2, 2}}; +obsolete_1(erlang, now, 0) -> + {deprecated, + "Deprecated BIF. See the \"Time and Time Correction in Erlang\" " + "chapter of the ERTS User's Guide for more information."}; + obsolete_1(calendar, local_time_to_universal_time, 1) -> {deprecated, {calendar, local_time_to_universal_time_dst, 1}}; @@ -590,6 +595,8 @@ obsolete_1(core_lib, is_literal_list, 1) -> " instead"}; obsolete_1(core_lib, literal_value, 1) -> {deprecated,{core_lib,concrete,1}}; +obsolete_1(ssl, negotiated_next_protocol, 1) -> + {deprecated,{ssl,negotiated_protocol}}; obsolete_1(_, _, _) -> no. diff --git a/lib/stdlib/src/qlc.erl b/lib/stdlib/src/qlc.erl index 002032d48d..5b19ee6190 100644 --- a/lib/stdlib/src/qlc.erl +++ b/lib/stdlib/src/qlc.erl @@ -2764,8 +2764,8 @@ tmp_filename(TmpDirOpt) -> U = "_", Node = node(), Pid = os:getpid(), - {MSecs,Secs,MySecs} = erlang:now(), - F = lists:concat([?MODULE,U,Node,U,Pid,U,MSecs,U,Secs,U,MySecs]), + Unique = erlang:unique_integer(), + F = lists:concat([?MODULE,U,Node,U,Pid,U,Unique]), TmpDir = case TmpDirOpt of "" -> {ok, CurDir} = file:get_cwd(), diff --git a/lib/stdlib/src/random.erl b/lib/stdlib/src/random.erl index d7b51a151c..cf84f8cecf 100644 --- a/lib/stdlib/src/random.erl +++ b/lib/stdlib/src/random.erl @@ -57,11 +57,17 @@ seed() -> %% seed({A1, A2, A3}) %% Seed random number generation --spec seed({A1, A2, A3}) -> 'undefined' | ran() when +-spec seed(SValue) -> 'undefined' | ran() when + SValue :: {A1, A2, A3} | integer(), A1 :: integer(), A2 :: integer(), A3 :: integer(). +seed(Int) when is_integer(Int) -> + A1 = (Int bsr 16) band 16#fffffff, + A2 = Int band 16#ffffff, + A3 = (Int bsr 36) bor (A2 bsr 16), + seed(A1, A2, A3); seed({A1, A2, A3}) -> seed(A1, A2, A3). diff --git a/lib/stdlib/src/stdlib.app.src b/lib/stdlib/src/stdlib.app.src index f134c75869..a435d683a5 100644 --- a/lib/stdlib/src/stdlib.app.src +++ b/lib/stdlib/src/stdlib.app.src @@ -102,7 +102,7 @@ dets]}, {applications, [kernel]}, {env, []}, - {runtime_dependencies, ["sasl-2.4","kernel-3.0.2","erts-6.2","crypto-3.3", + {runtime_dependencies, ["sasl-2.4","kernel-3.0.2","erts-7.0","crypto-3.3", "compiler-5.0"]} ]}. diff --git a/lib/stdlib/src/supervisor.erl b/lib/stdlib/src/supervisor.erl index 658c00dc77..7c0cd8b26a 100644 --- a/lib/stdlib/src/supervisor.erl +++ b/lib/stdlib/src/supervisor.erl @@ -1383,7 +1383,7 @@ add_restart(State) -> I = State#state.intensity, P = State#state.period, R = State#state.restarts, - Now = erlang:now(), + Now = erlang:monotonic_time(1), R1 = add_restart([Now|R], Now, P), State1 = State#state{restarts = R1}, case length(R1) of @@ -1404,26 +1404,13 @@ add_restart([], _, _) -> []. inPeriod(Time, Now, Period) -> - case difference(Time, Now) of + case Time - Now of T when T > Period -> false; _ -> true end. -%% -%% Time = {MegaSecs, Secs, MicroSecs} (NOTE: MicroSecs is ignored) -%% Calculate the time elapsed in seconds between two timestamps. -%% If MegaSecs is equal just subtract Secs. -%% Else calculate the Mega difference and add the Secs difference, -%% note that Secs difference can be negative, e.g. -%% {827, 999999, 676} diff {828, 1, 653753} == > 2 secs. -%% -difference({TimeM, TimeS, _}, {CurM, CurS, _}) when CurM > TimeM -> - ((CurM - TimeM) * 1000000) + (CurS - TimeS); -difference({_, TimeS, _}, {_, CurS, _}) -> - CurS - TimeS. - %%% ------------------------------------------------------ %%% Error and progress reporting. %%% ------------------------------------------------------ diff --git a/lib/stdlib/src/timer.erl b/lib/stdlib/src/timer.erl index 72a2dd9616..19d803345e 100644 --- a/lib/stdlib/src/timer.erl +++ b/lib/stdlib/src/timer.erl @@ -437,10 +437,8 @@ positive(X) -> %% %% system_time() -> time in microseconds %% -system_time() -> - {M,S,U} = erlang:now(), - 1000000 * (M*1000000 + S) + U. - +system_time() -> + erlang:monotonic_time(1000000). send([Pid, Msg]) -> Pid ! Msg. diff --git a/lib/stdlib/src/zip.erl b/lib/stdlib/src/zip.erl index b768c6d0b9..44e75ff15b 100644 --- a/lib/stdlib/src/zip.erl +++ b/lib/stdlib/src/zip.erl @@ -214,7 +214,9 @@ -type zip_comment() :: #zip_comment{}. -type zip_file() :: #zip_file{}. --export_type([create_option/0, filename/0]). +-opaque handle() :: pid(). + +-export_type([create_option/0, filename/0, handle/0]). %% Open a zip archive with options %% @@ -500,7 +502,7 @@ do_list_dir(F, Options) -> -spec(t(Archive) -> ok when Archive :: file:name() | binary() | ZipHandle, - ZipHandle :: pid()). + ZipHandle :: handle()). t(F) when is_pid(F) -> zip_t(F); t(F) when is_record(F, openzip) -> openzip_t(F); @@ -524,7 +526,7 @@ do_t(F, RawPrint) -> -spec(tt(Archive) -> ok when Archive :: file:name() | binary() | ZipHandle, - ZipHandle :: pid()). + ZipHandle :: handle()). tt(F) when is_pid(F) -> zip_tt(F); tt(F) when is_record(F, openzip) -> openzip_tt(F); @@ -1114,15 +1116,19 @@ local_file_header_from_info_method_name(#file_info{mtime = MTime}, file_name_length = length(Name), extra_field_length = 0}. +server_init(Parent) -> + %% we want to know if our parent dies + process_flag(trap_exit, true), + server_loop(Parent, not_open). %% small, simple, stupid zip-archive server -server_loop(OpenZip) -> +server_loop(Parent, OpenZip) -> receive {From, {open, Archive, Options}} -> case openzip_open(Archive, Options) of {ok, NewOpenZip} -> From ! {self(), {ok, self()}}, - server_loop(NewOpenZip); + server_loop(Parent, NewOpenZip); Error -> From ! {self(), Error} end; @@ -1130,43 +1136,47 @@ server_loop(OpenZip) -> From ! {self(), openzip_close(OpenZip)}; {From, get} -> From ! {self(), openzip_get(OpenZip)}, - server_loop(OpenZip); + server_loop(Parent, OpenZip); {From, {get, FileName}} -> From ! {self(), openzip_get(FileName, OpenZip)}, - server_loop(OpenZip); + server_loop(Parent, OpenZip); {From, list_dir} -> From ! {self(), openzip_list_dir(OpenZip)}, - server_loop(OpenZip); + server_loop(Parent, OpenZip); {From, {list_dir, Opts}} -> From ! {self(), openzip_list_dir(OpenZip, Opts)}, - server_loop(OpenZip); + server_loop(Parent, OpenZip); {From, get_state} -> From ! {self(), OpenZip}, - server_loop(OpenZip); + server_loop(Parent, OpenZip); + {'EXIT', Parent, Reason} -> + openzip_close(OpenZip), + exit({parent_died, Reason}); _ -> {error, bad_msg} end. -spec(zip_open(Archive) -> {ok, ZipHandle} | {error, Reason} when Archive :: file:name() | binary(), - ZipHandle :: pid(), + ZipHandle :: handle(), Reason :: term()). zip_open(Archive) -> zip_open(Archive, []). -spec(zip_open(Archive, Options) -> {ok, ZipHandle} | {error, Reason} when Archive :: file:name() | binary(), - ZipHandle :: pid(), + ZipHandle :: handle(), Options :: [Option], Option :: cooked | memory | {cwd, CWD :: file:filename()}, Reason :: term()). zip_open(Archive, Options) -> - Pid = spawn(fun() -> server_loop(not_open) end), - request(self(), Pid, {open, Archive, Options}). + Self = self(), + Pid = spawn_link(fun() -> server_init(Self) end), + request(Self, Pid, {open, Archive, Options}). -spec(zip_get(ZipHandle) -> {ok, [Result]} | {error, Reason} when - ZipHandle :: pid(), + ZipHandle :: handle(), Result :: file:name() | {file:name(), binary()}, Reason :: term()). @@ -1174,14 +1184,14 @@ zip_get(Pid) when is_pid(Pid) -> request(self(), Pid, get). -spec(zip_close(ZipHandle) -> ok | {error, einval} when - ZipHandle :: pid()). + ZipHandle :: handle()). zip_close(Pid) when is_pid(Pid) -> request(self(), Pid, close). -spec(zip_get(FileName, ZipHandle) -> {ok, Result} | {error, Reason} when FileName :: file:name(), - ZipHandle :: pid(), + ZipHandle :: handle(), Result :: file:name() | {file:name(), binary()}, Reason :: term()). @@ -1190,7 +1200,7 @@ zip_get(FileName, Pid) when is_pid(Pid) -> -spec(zip_list_dir(ZipHandle) -> {ok, Result} | {error, Reason} when Result :: [zip_comment() | zip_file()], - ZipHandle :: pid(), + ZipHandle :: handle(), Reason :: term()). zip_list_dir(Pid) when is_pid(Pid) -> diff --git a/lib/stdlib/test/ets_SUITE.erl b/lib/stdlib/test/ets_SUITE.erl index 2674f6886f..9f552b5a6b 100644 --- a/lib/stdlib/test/ets_SUITE.erl +++ b/lib/stdlib/test/ets_SUITE.erl @@ -47,6 +47,7 @@ -export([ordered/1, ordered_match/1, interface_equality/1, fixtable_next/1, fixtable_insert/1, rename/1, rename_unnamed/1, evil_rename/1, update_element/1, update_counter/1, evil_update_counter/1, partly_bound/1, match_heavy/1]). +-export([update_counter_with_default/1]). -export([member/1]). -export([memory/1]). -export([select_fail/1]). @@ -99,7 +100,7 @@ misc1_do/1, safe_fixtable_do/1, info_do/1, dups_do/1, heavy_lookup_do/1, heavy_lookup_element_do/1, member_do/1, otp_5340_do/1, otp_7665_do/1, meta_wb_do/1, do_heavy_concurrent/1, tab2file2_do/2, exit_large_table_owner_do/2, - types_do/1, sleeper/0, memory_do/1, + types_do/1, sleeper/0, memory_do/1, update_counter_with_default_do/1, ms_tracee_dummy/1, ms_tracee_dummy/2, ms_tracee_dummy/3, ms_tracee_dummy/4 ]). @@ -136,7 +137,8 @@ all() -> {group, heavy}, ordered, ordered_match, interface_equality, fixtable_next, fixtable_insert, rename, rename_unnamed, evil_rename, update_element, - update_counter, evil_update_counter, partly_bound, + update_counter, evil_update_counter, + update_counter_with_default, partly_bound, match_heavy, {group, fold}, member, t_delete_object, t_init_table, t_whitebox, t_delete_all_objects, t_insert_list, t_test_ms, t_select_delete, t_ets_dets, @@ -1761,6 +1763,14 @@ update_counter_do(Opts) -> OrdSet = ets_new(ordered_set,[ordered_set | Opts]), update_counter_for(Set), update_counter_for(OrdSet), + ets:delete_all_objects(Set), + ets:delete_all_objects(OrdSet), + ets:safe_fixtable(Set, true), + ets:safe_fixtable(OrdSet, true), + update_counter_for(Set), + update_counter_for(OrdSet), + ets:safe_fixtable(Set, false), + ets:safe_fixtable(OrdSet, false), ets:delete(Set), ets:delete(OrdSet), update_counter_neg(Opts). @@ -1780,10 +1790,14 @@ update_counter_for(T) -> ?line {NewObj, Ret} = uc_mimic(Obj,Arg3), ArgHash = erlang:phash2({T,a,Arg3}), %%io:format("update_counter(~p, ~p, ~p) expecting ~p\n",[T,a,Arg3,Ret]), + [DefaultObj] = ets:lookup(T, a), ?line Ret = ets:update_counter(T,a,Arg3), + Ret = ets:update_counter(T, b, Arg3, DefaultObj), % Use other key ?line ArgHash = erlang:phash2({T,a,Arg3}), %%io:format("NewObj=~p~n ",[NewObj]), ?line [NewObj] = ets:lookup(T,a), + true = ets:lookup(T, b) =:= [setelement(1, NewObj, b)], + ets:delete(T, b), Myself(NewObj,Times-1,Arg3,Myself) end, @@ -2008,6 +2022,44 @@ evil_counter_1(Iter, T) -> ets:update_counter(T, dracula, 1), evil_counter_1(Iter-1, T). +update_counter_with_default(Config) when is_list(Config) -> + repeat_for_opts(update_counter_with_default_do). + +update_counter_with_default_do(Opts) -> + T1 = ets_new(a, [set | Opts]), + %% Insert default object. + 3 = ets:update_counter(T1, foo, 2, {beaufort,1}), + %% Increment. + 5 = ets:update_counter(T1, foo, 2, {cabecou,1}), + %% Increment with list. + [9] = ets:update_counter(T1, foo, [{2,4}], {camembert,1}), + %% Same with non-immediate key. + 3 = ets:update_counter(T1, {foo,bar}, 2, {{chaource,chevrotin},1}), + 5 = ets:update_counter(T1, {foo,bar}, 2, {{cantal,comté},1}), + [9] = ets:update_counter(T1, {foo,bar}, [{2,4}], {{emmental,de,savoie},1}), + %% Same with ordered set. + T2 = ets_new(b, [ordered_set | Opts]), + 3 = ets:update_counter(T2, foo, 2, {maroilles,1}), + 5 = ets:update_counter(T2, foo, 2, {mimolette,1}), + [9] = ets:update_counter(T2, foo, [{2,4}], {morbier,1}), + 3 = ets:update_counter(T2, {foo,bar}, 2, {{laguiole},1}), + 5 = ets:update_counter(T2, {foo,bar}, 2, {{saint,nectaire},1}), + [9] = ets:update_counter(T2, {foo,bar}, [{2,4}], {{rocamadour},1}), + %% Arithmetically-equal keys. + 3 = ets:update_counter(T2, 1.0, 2, {1,1}), + 5 = ets:update_counter(T2, 1, 2, {1,1}), + 7 = ets:update_counter(T2, 1, 2, {1.0,1}), + %% Same with reversed type difference. + 3 = ets:update_counter(T2, 2, 2, {2.0,1}), + 5 = ets:update_counter(T2, 2.0, 2, {2.0,1}), + 7 = ets:update_counter(T2, 2.0, 2, {2,1}), + %% bar is not an integer. + {'EXIT',{badarg,_}} = (catch ets:update_counter(T1, qux, 3, {saint,félicien})), + %% No third element in default value. + {'EXIT',{badarg,_}} = (catch ets:update_counter(T1, qux, [{3,1}], {roquefort,1})), + + ok. + fixtable_next(doc) -> ["Check that a first-next sequence always works on a fixed table"]; fixtable_next(suite) -> @@ -3779,6 +3831,7 @@ match_object_do(Opts) -> ?line ets:insert(Tab,{{one,5},5}), ?line ets:insert(Tab,{{two,4},4}), ?line ets:insert(Tab,{{two,5},6}), + ?line ets:insert(Tab, {#{camembert=>cabécou},7}), ?line case ets:match_object(Tab, {{one, '_'}, '$0'}) of [{{one,5},5},{{one,4},4}] -> ok; [{{one,4},4},{{one,5},5}] -> ok; @@ -3799,6 +3852,10 @@ match_object_do(Opts) -> [{{two,4},4},{{two,5},6}] -> ok; _ -> ?t:fail("ets:match_object() returned something funny.") end, + % Check that maps are inspected for variables. + [{#{camembert:=cabécou},7}] = + ets:match_object(Tab, {#{camembert=>'_'},7}), + {'EXIT',{badarg,_}} = (catch ets:match_object(Tab, {#{'$1'=>'_'},7})), % Check that unsucessful match returns an empty list. ?line [] = ets:match_object(Tab, {{three,'$0'}, '$92'}), % Check that '$0' equals '_'. diff --git a/lib/stdlib/test/qlc_SUITE.erl b/lib/stdlib/test/qlc_SUITE.erl index 4173a40d14..0b7b96da8e 100644 --- a/lib/stdlib/test/qlc_SUITE.erl +++ b/lib/stdlib/test/qlc_SUITE.erl @@ -3418,7 +3418,8 @@ lookup2(Config) when is_list(Config) -> end, [{1},{2}])">> ], - ?line run(Config, Ts), + + ok = run(Config, Ts), TsR = [ %% is_record/2,3: @@ -3456,7 +3457,8 @@ lookup2(Config) when is_list(Config) -> end, [{keypos,1}], [#r{}])">> ], - ?line run(Config, <<"-record(r, {a}).\n">>, TsR), + + ok = run(Config, <<"-record(r, {a}).\n">>, TsR), Ts2 = [ <<"etsc(fun(E) -> @@ -3566,7 +3568,6 @@ lookup2(Config) when is_list(Config) -> [{1,2},{2,2}] = qlc:e(Q), [2] = lookup_keys(Q) end, [{keypos,1}], [{1},{2},{3}])">>, - <<"%% Matchspec only. No cache. etsc(fun(E) -> Q = qlc:q([{X,Y} || @@ -3578,7 +3579,7 @@ lookup2(Config) when is_list(Config) -> {generate,_, {table,{ets,_,[_,[{traverse,_}]]}}}],[]} = i(Q), - [{1,2},{1,3},{2,2},{2,3}] = qlc:e(Q), + [{1,2},{1,3},{2,2},{2,3}] = lists:sort(qlc:e(Q)), false = lookup_keys(Q) end, [{keypos,1}], [{1},{2},{3}])">>, <<"%% Matchspec only. Cache @@ -3592,7 +3593,7 @@ lookup2(Config) when is_list(Config) -> {generate,_,{qlc,_, [{generate,_,{table,{ets,_,[_,[{traverse,_}]]}}}], [{cache,ets}]}}],[]} = i(Q), - [{1,2},{1,3},{2,2},{2,3}] = qlc:e(Q), + [{1,2},{1,3},{2,2},{2,3}] = lists:sort(qlc:e(Q)), false = lookup_keys(Q) end, [{keypos,1}], [{1},{2},{3}])">>, <<"%% An empty list. Always unique and cached. @@ -3645,7 +3646,7 @@ lookup2(Config) when is_list(Config) -> ], - ?line run(Config, Ts2), + ok = run(Config, Ts2), LTs = [ <<"etsc(fun(E) -> @@ -3677,7 +3678,8 @@ lookup2(Config) when is_list(Config) -> end, [{1,a},{2,b}])">> ], - ?line run(Config, LTs), + + ok = run(Config, LTs), ok. diff --git a/lib/stdlib/test/zip_SUITE.erl b/lib/stdlib/test/zip_SUITE.erl index a57641ef62..d168a9d9bc 100644 --- a/lib/stdlib/test/zip_SUITE.erl +++ b/lib/stdlib/test/zip_SUITE.erl @@ -23,7 +23,7 @@ bad_zip/1, unzip_from_binary/1, unzip_to_binary/1, zip_to_binary/1, unzip_options/1, zip_options/1, list_dir_options/1, aliases/1, - openzip_api/1, zip_api/1, unzip_jar/1, + openzip_api/1, zip_api/1, open_leak/1, unzip_jar/1, compress_control/1, foldl/1]). @@ -38,7 +38,7 @@ all() -> [borderline, atomic, bad_zip, unzip_from_binary, unzip_to_binary, zip_to_binary, unzip_options, zip_options, list_dir_options, aliases, openzip_api, - zip_api, unzip_jar, compress_control, foldl]. + zip_api, open_leak, unzip_jar, compress_control, foldl]. groups() -> []. @@ -318,8 +318,46 @@ zip_api(Config) when is_list(Config) -> %% Clean up. delete_files([Names]), + ok. + +open_leak(doc) -> + ["Test that zip doesn't leak processes and ports where the " + "controlling process dies without closing an zip opened with " + "zip:zip_open/1."]; +open_leak(suite) -> []; +open_leak(Config) when is_list(Config) -> + %% Create a zip archive + Zip = "zip.zip", + {ok, Zip} = zip:zip(Zip, [], []), + + %% Open archive in a another process that dies immediately. + ZipSrv = spawn_zip(Zip, [memory]), + + %% Expect the ZipSrv process to die soon after. + true = spawned_zip_dead(ZipSrv), + + %% Clean up. + delete_files([Zip]), + ok. +spawn_zip(Zip, Options) -> + Self = self(), + spawn(fun() -> Self ! zip:zip_open(Zip, Options) end), + receive + {ok, ZipSrv} -> + ZipSrv + end. + +spawned_zip_dead(ZipSrv) -> + Ref = monitor(process, ZipSrv), + receive + {'DOWN', Ref, _, ZipSrv, _} -> + true + after 1000 -> + false + end. + unzip_options(doc) -> ["Test options for unzip, only cwd and file_list currently"]; unzip_options(suite) -> diff --git a/lib/test_server/src/test_server.app.src b/lib/test_server/src/test_server.app.src index 173f7075db..5538e8b851 100644 --- a/lib/test_server/src/test_server.app.src +++ b/lib/test_server/src/test_server.app.src @@ -34,5 +34,5 @@ {env, []}, {runtime_dependencies, ["tools-2.6.14","stdlib-2.0","runtime_tools-1.8.14", "observer-2.0","kernel-3.0","inets-5.10", - "syntax_tools-1.6.16","erts-6.0"]}]}. + "syntax_tools-1.6.16","erts-7.0"]}]}. diff --git a/lib/test_server/src/test_server.erl b/lib/test_server/src/test_server.erl index 0e685a2d8a..acd2e0bff2 100644 --- a/lib/test_server/src/test_server.erl +++ b/lib/test_server/src/test_server.erl @@ -404,15 +404,6 @@ run_test_case_apply({CaseNum,Mod,Func,Args,Name, }). run_test_case_apply(Mod, Func, Args, Name, RunInit, TimetrapData) -> - {ok,Cwd} = file:get_cwd(), - Args2Print = case Args of - [Args1] when is_list(Args1) -> - lists:keydelete(tc_group_result, 1, Args1); - _ -> - Args - end, - print(minor, "Test case started with:\n~w:~w(~tp)\n", [Mod,Func,Args2Print]), - print(minor, "Current directory is ~tp\n", [Cwd]), print_timestamp(minor,"Started at "), print(minor, "", [], internal_raw), TCCallback = get(test_server_testcase_callback), @@ -687,7 +678,7 @@ handle_tc_exit(Reason, #st{status=tc,config=Config0,mf={Mod,Func},pid=Pid}=St) Msg = {E,AbortReason}, {Msg,Loc0,Msg}; Other -> - {Other,unknown,Other} + {{'EXIT',Other},unknown,Other} end, Timeout = end_conf_timeout(Reason, St), Config = [{tc_status,{failed,F}}|Config0], @@ -701,7 +692,7 @@ handle_tc_exit(Reason, #st{config=Config,mf={Mod,Func0},pid=Pid, {testcase_aborted=E,AbortReason,Loc0} -> {{E,AbortReason},Loc0}; Other -> - {Other,St#st.last_known_loc} + {{'EXIT',Other},St#st.last_known_loc} end, Func = case Status of init_per_testcase=F -> {F,Func0}; @@ -738,18 +729,16 @@ do_call_end_conf(Starter,Mod,Func,Data,Conf,TVal) -> EndConfApply = fun() -> timetrap(TVal), - case catch apply(Mod, - end_per_testcase, - [Func,Conf]) of - {'EXIT',Why} -> + try apply(Mod,end_per_testcase,[Func,Conf]) of + _ -> ok + catch + _:Why -> timer:sleep(1), group_leader() ! {printout,12, "WARNING! " "~w:end_per_testcase(~w, ~p)" " crashed!\n\tReason: ~p\n", - [Mod,Func,Conf,Why]}; - _ -> - ok + [Mod,Func,Conf,Why]} end, Supervisor ! {self(),end_conf} end, @@ -778,13 +767,11 @@ spawn_fw_call(Mod,{init_per_testcase,Func},CurrConf,Pid, Skip = {skip,{failed,{Mod,init_per_testcase,Why}}}, %% if init_per_testcase fails, the test case %% should be skipped - case catch do_end_tc_call(Mod,Func, - {Pid,Skip,[CurrConf]}, - Why) of - {'EXIT',FwEndTCErr} -> - exit({fw_notify_done,end_tc,FwEndTCErr}); - _ -> - ok + try do_end_tc_call(Mod,Func, {Pid,Skip,[CurrConf]}, Why) of + _ -> ok + catch + _:FwEndTCErr -> + exit({fw_notify_done,end_tc,FwEndTCErr}) end, %% finished, report back SendTo ! {self(),fw_notify_done, @@ -812,12 +799,12 @@ spawn_fw_call(Mod,{end_per_testcase,Func},EndConf,Pid, " failed!\n\tReason: timetrap timeout" " after ~w ms!\n", [Mod,Func,EndConf,TVal]}, FailLoc = proplists:get_value(tc_fail_loc, EndConf), - case catch do_end_tc_call(Mod,Func, + try do_end_tc_call(Mod,Func, {Pid,Report,[EndConf]}, Why) of - {'EXIT',FwEndTCErr} -> - exit({fw_notify_done,end_tc,FwEndTCErr}); - _ -> - ok + _ -> ok + catch + _:FwEndTCErr -> + exit({fw_notify_done,end_tc,FwEndTCErr}) end, Warn = "<font color=\"red\">" "WARNING: end_per_testcase timed out!</font>", @@ -853,21 +840,21 @@ spawn_fw_call(Mod,Func,CurrConf,Pid,Error,Loc,SendTo) -> end, FwCall = fun() -> - case catch fw_error_notify(Mod,Func1,[], - Error,Loc) of - {'EXIT',FwErrorNotifyErr} -> + try fw_error_notify(Mod,Func1,[], + Error,Loc) of + _ -> ok + catch + _:FwErrorNotifyErr -> exit({fw_notify_done,error_notification, - FwErrorNotifyErr}); - _ -> - ok + FwErrorNotifyErr}) end, Conf = [{tc_status,{failed,Error}}|CurrConf], - case catch do_end_tc_call(Mod,Func1, - {Pid,Error,[Conf]},Error) of - {'EXIT',FwEndTCErr} -> - exit({fw_notify_done,end_tc,FwEndTCErr}); - _ -> - ok + try do_end_tc_call(Mod,Func1, + {Pid,Error,[Conf]},Error) of + _ -> ok + catch + _:FwEndTCErr -> + exit({fw_notify_done,end_tc,FwEndTCErr}) end, %% finished, report back SendTo ! {self(),fw_notify_done,{died,Error,Loc,[],undefined}} @@ -1358,8 +1345,8 @@ fw_error_notify(Mod, Func, Args, Error, Loc) -> %% Just like io:format, except that depending on the Detail value, the output %% is directed to console, major and/or minor log files. -print(Detail,Format,Args) -> - test_server_ctrl:print(Detail, Format, Args). +%% print(Detail,Format,Args) -> +%% test_server_ctrl:print(Detail, Format, Args). print(Detail,Format,Args,Printer) -> test_server_ctrl:print(Detail, Format, Args, Printer). @@ -1395,7 +1382,7 @@ lookup_config(Key,Config) -> %% timer:tc/3 ts_tc(M, F, A) -> - Before = erlang:now(), + Before = erlang:monotonic_time(), Result = try apply(M, F, A) catch @@ -1415,12 +1402,8 @@ ts_tc(M, F, A) -> {'EXIT',Reason} end end, - After = erlang:now(), - Elapsed = - (element(1,After)*1000000000000 - +element(2,After)*1000000+element(3,After)) - - (element(1,Before)*1000000000000 - +element(2,Before)*1000000+element(3,Before)), + After = erlang:monotonic_time(), + Elapsed = erlang:convert_time_unit(After-Before, native, micro_seconds), {Elapsed, Result}. set_loc(Stk) -> @@ -1839,7 +1822,7 @@ time_ms_check(Other) -> time_ms_apply(Func, TCPid, MultAndScale) -> {_,GL} = process_info(TCPid, group_leader), WhoAmI = self(), % either TC or IO server - T0 = now(), + T0 = os:timestamp(), UserTTSup = spawn(fun() -> user_timetrap_supervisor(Func, WhoAmI, TCPid, @@ -1872,7 +1855,7 @@ user_timetrap_supervisor(Func, Spawner, TCPid, GL, T0, MultAndScale) -> receive {UserTT,Result} -> demonitor(MonRef, [flush]), - Elapsed = trunc(timer:now_diff(now(), T0) / 1000), + Elapsed = trunc(timer:now_diff(os:timestamp(), T0) / 1000), try time_ms_check(Result) of TimeVal -> %% this is the new timetrap value to set (return value @@ -2395,9 +2378,8 @@ is_release_available(Release) -> %% run_on_shielded_node(Fun, CArgs) when is_function(Fun), is_list(CArgs) -> - {A,B,C} = now(), - Name = "shielded_node-" ++ integer_to_list(A) ++ "-" ++ integer_to_list(B) - ++ "-" ++ integer_to_list(C), + Nr = erlang:unique_integer([positive]), + Name = "shielded_node-" ++ integer_to_list(Nr), Node = case start_node(Name, slave, [{args, "-hidden " ++ CArgs}]) of {ok, N} -> N; Err -> fail({failed_to_start_shielded_node, Err}) @@ -2456,9 +2438,8 @@ is_cover(Name) -> %% A filename of the form <Stem><Number> is generated, and the %% function checks that that file doesn't already exist. temp_name(Stem) -> - {A,B,C} = erlang:now(), - RandomNum = A bxor B bxor C, - RandomName = Stem ++ integer_to_list(RandomNum), + Num = erlang:unique_integer([positive]), + RandomName = Stem ++ integer_to_list(Num), {ok,Files} = file:list_dir(filename:dirname(Stem)), case lists:member(RandomName,Files) of true -> diff --git a/lib/test_server/src/test_server_ctrl.erl b/lib/test_server/src/test_server_ctrl.erl index 488f38d05d..349b033c89 100644 --- a/lib/test_server/src/test_server_ctrl.erl +++ b/lib/test_server/src/test_server_ctrl.erl @@ -99,7 +99,7 @@ -define(last_link, "last_link"). -define(last_test, "last_test"). -define(html_ext, ".html"). --define(now, erlang:now()). +-define(now, os:timestamp()). -define(void_fun, fun() -> ok end). -define(mod_result(X), if X == skip -> skipped; @@ -1204,19 +1204,14 @@ init_tester(Mod, Func, Args, Dir, Name, {_,_,MinLev}=Levels, report_severe_error(Reason) -> test_server_sup:framework_call(report, [severe_error,Reason]). -%% timer:tc/3 -ts_tc(M, F, A) -> - Before = ?now, - Val = (catch apply(M, F, A)), - After = ?now, - Elapsed = elapsed_time(Before, After), - {Elapsed,Val}. - -elapsed_time(Before, After) -> - (element(1,After)*1000000000000 + - element(2,After)*1000000 + element(3,After)) - - (element(1,Before)*1000000000000 + - element(2,Before)*1000000 + element(3,Before)). +ts_tc(M,F,A) -> + Before = erlang:monotonic_time(), + Result = (catch apply(M, F, A)), + After = erlang:monotonic_time(), + Elapsed = erlang:convert_time_unit(After-Before, + native, + micro_seconds), + {Elapsed, Result}. start_extra_tools(ExtraTools) -> start_extra_tools(ExtraTools, []). @@ -1808,20 +1803,32 @@ start_minor_log_file1(Mod, Func, LogDir, AbsName, MFA) -> put(test_server_minor_footer, Footer), io:put_chars(Fd, Header), + io:put_chars(Fd, "<a name=\"top\"></a>"), + io:put_chars(Fd, "<pre>\n"), + SrcListing = downcase(atom_to_list(Mod)) ++ ?src_listing_ext, + + {Info,Arity} = + if Func == init_per_suite; Func == end_per_suite -> + {"Config function: ", 1}; + Func == init_per_group; Func == end_per_group -> + {"Config function: ", 2}; + true -> + {"Test case: ", 1} + end, + case {filelib:is_file(filename:join(LogDir, SrcListing)), lists:member(no_src, get(test_server_logopts))} of {true,false} -> - print(Lev, "<a href=\"~ts#~ts\">source code for ~w:~w/1</a>\n", + print(Lev, Info ++ "<a href=\"~ts#~ts\">~w:~w/~w</a> " + "(click for source code)\n", [uri_encode(SrcListing), uri_encode(atom_to_list(Func)++"-1",utf8), - Mod,Func]); + Mod,Func,Arity]); _ -> - ok + print(Lev, Info ++ "~w:~w/~w\n", [Mod,Func,Arity]) end, - io:put_chars(Fd, "<pre>\n"), - AbsName. stop_minor_log_file() -> @@ -2478,7 +2485,7 @@ run_test_cases_loop([{conf,Ref,Props,{Mod,Func}}|_Cases]=Cs0, file:set_cwd(filename:dirname(get(test_server_dir))), After = ?now, Before = get(test_server_parallel_start_time), - Elapsed = elapsed_time(Before, After)/1000000, + Elapsed = timer:now_diff(After, Before)/1000000, put(test_server_total_time, Elapsed), {false,tl(Mode0),undefined,Elapsed, update_status(Ref, OkSkipFail, Status)}; @@ -2487,7 +2494,7 @@ run_test_cases_loop([{conf,Ref,Props,{Mod,Func}}|_Cases]=Cs0, %% parallel group (io buffering is active) OkSkipFail = wait_for_cases(Ref), queue_test_case_io(Ref, self(), 0, Mod, Func), - Elapsed = elapsed_time(conf_start(Ref, Mode0),?now)/1000000, + Elapsed = timer:now_diff(?now, conf_start(Ref, Mode0))/1000000, case CurrIOHandler of {Ref,_} -> %% current_io_handler was set by start conf of this @@ -2504,12 +2511,12 @@ run_test_cases_loop([{conf,Ref,Props,{Mod,Func}}|_Cases]=Cs0, %% this is an end conf for a non-parallel group that's not %% nested under a parallel group, so no need to buffer io {false,tl(Mode0),undefined, - elapsed_time(conf_start(Ref, Mode0),?now)/1000000, Status}; + timer:now_diff(?now, conf_start(Ref, Mode0))/1000000, Status}; {Ref,_} -> %% this is an end conf for a non-parallel group nested under %% a parallel group (io buffering is active) queue_test_case_io(Ref, self(), 0, Mod, Func), - Elapsed = elapsed_time(conf_start(Ref, Mode0),?now)/1000000, + Elapsed = timer:now_diff(?now, conf_start(Ref, Mode0))/1000000, case CurrIOHandler of {Ref,_} -> %% current_io_handler was set by start conf of this @@ -2564,7 +2571,7 @@ run_test_cases_loop([{conf,Ref,Props,{Mod,Func}}|_Cases]=Cs0, %% 1. check the TS_RANDOM_SEED env variable %% 2. check random_seed in process state %% 3. use value provided with shuffle option - %% 4. use now() values for seed + %% 4. use timestamp() values for seed case os:getenv("TS_RANDOM_SEED") of Undef when Undef == false ; Undef == "undefined" -> case get(test_server_random_seed) of @@ -3076,13 +3083,11 @@ print_conf_time(ConfTime) -> print(major, "=group_time ~.3fs", [ConfTime]), print(minor, "~n=== Total execution time of group: ~.3fs~n", [ConfTime]). -print_props(_, []) -> +print_props([]) -> ok; -print_props(true, Props) -> +print_props(Props) -> print(major, "=group_props ~p", [Props]), - print(minor, "Group properties: ~p~n", [Props]); -print_props(_, _) -> - ok. + print(minor, "Group properties: ~p~n", [Props]). %% repeat N times: {repeat,N} %% repeat N times or until all successful: {repeat_until_all_ok,N} @@ -3687,7 +3692,6 @@ run_test_case1(Ref, Num, Mod, Func, Args, RunInit, print(major, "=case ~w:~w", [Mod, Func]), MinorName = start_minor_log_file(Mod, Func, self() /= Main), - print(minor, "<a name=\"top\"></a>", [], internal_raw), MinorBase = filename:basename(MinorName), print(major, "=logfile ~ts", [filename:basename(MinorName)]), @@ -3701,8 +3705,8 @@ run_test_case1(Ref, Num, Mod, Func, Args, RunInit, RunDir = filename:dirname(MinorName), Ext = if Num == 0 -> - {_,S,Us} = now(), - lists:flatten(io_lib:format(".~w.~w", [S,Us])); + Nr = erlang:unique_integer([positive]), + lists:flatten(io_lib:format(".~w", [Nr])); true -> lists:flatten(io_lib:format(".~w", [Num])) end, @@ -3720,7 +3724,20 @@ run_test_case1(Ref, Num, Mod, Func, Args, RunInit, [tc_start,{{Mod,{Func,GrName}}, MinorName}]), - print_props((RunInit==skip_init), get_props(Mode)), + {ok,Cwd} = file:get_cwd(), + Args2Print = if is_list(UpdatedArgs) -> + lists:keydelete(tc_group_result, 1, UpdatedArgs); + true -> + UpdatedArgs + end, + if RunInit == skip_init -> + print_props(get_props(Mode)); + true -> + ok + end, + print(minor, "Config value:\n\n ~tp\n", [Args2Print]), + print(minor, "Current directory is ~tp\n", [Cwd]), + GrNameStr = case GrName of undefined -> ""; Name -> cast_to_list(Name) diff --git a/lib/test_server/src/test_server_node.erl b/lib/test_server/src/test_server_node.erl index acd47788db..9d87eca07e 100644 --- a/lib/test_server/src/test_server_node.erl +++ b/lib/test_server/src/test_server_node.erl @@ -618,9 +618,8 @@ do_quote_progname([Prog,Arg|Args]) -> end. random_element(L) -> - {A,B,C} = now(), - E = lists:sum([A,B,C]) rem length(L), - lists:nth(E+1, L). + random:seed(os:timestamp()), + lists:nth(random:uniform(length(L)), L). find_release(latest) -> "/usr/local/otp/releases/latest/bin/erl"; diff --git a/lib/test_server/src/test_server_sup.erl b/lib/test_server/src/test_server_sup.erl index 96e369a138..7a1f7803eb 100644 --- a/lib/test_server/src/test_server_sup.erl +++ b/lib/test_server/src/test_server_sup.erl @@ -121,14 +121,8 @@ messages_get(Msgs) -> end. timecall(M, F, A) -> - Befor = erlang:now(), - Val = apply(M, F, A), - After = erlang:now(), - Elapsed = - (element(1,After)*1000000+element(2,After)+element(3,After)/1000000)- - (element(1,Befor)*1000000+element(2,Befor)+element(3,Befor)/1000000), - {Elapsed, Val}. - + {Elapsed, Val} = timer:tc(M, F, A), + {Elapsed / 1000000, Val}. call_crash(Time,Crash,M,F,A) -> @@ -874,9 +868,8 @@ unique_name() -> util_loop(State) -> receive {From,unique_name} -> - {_,S,Us} = now(), - Ms = trunc(Us/1000), - Name = lists:flatten(io_lib:format("~w.~w", [S,Ms])), + Nr = erlang:unique_integer([positive]), + Name = integer_to_list(Nr), if Name == State#util_state.latest_name -> timer:sleep(1), self() ! {From,unique_name}, diff --git a/lib/test_server/src/ts_install.erl b/lib/test_server/src/ts_install.erl index a41409f2fd..594e619fbc 100644 --- a/lib/test_server/src/ts_install.erl +++ b/lib/test_server/src/ts_install.erl @@ -18,7 +18,6 @@ %% -module(ts_install). - -export([install/2, platform_id/1]). -include("ts.hrl"). @@ -112,6 +111,12 @@ get_vars([], name, [], Result) -> get_vars(_, _, _, _) -> {error, fatal_bad_conf_vars}. +config_flags() -> + case os:getenv("CONFIG_FLAGS") of + false -> []; + CF -> string:tokens(CF, " \t\n") + end. + unix_autoconf(XConf) -> Configure = filename:absname("configure"), Flags = proplists:get_value(crossflags,XConf,[]), @@ -122,22 +127,70 @@ unix_autoconf(XConf) -> erlang:system_info(threads) /= false], Debug = [" --enable-debug-mode" || string:str(erlang:system_info(system_version),"debug") > 0], - MXX_Build = [Y || Y <- string:tokens(os:getenv("CONFIG_FLAGS", ""), " \t\n"), + MXX_Build = [Y || Y <- config_flags(), Y == "--enable-m64-build" orelse Y == "--enable-m32-build"], Args = Host ++ Build ++ Threads ++ Debug ++ " " ++ MXX_Build, case filelib:is_file(Configure) of true -> OSXEnv = macosx_cflags(), + UnQuotedEnv = assign_vars(unquote(Env++OSXEnv)), io:format("Running ~s~nEnv: ~p~n", - [lists:flatten(Configure ++ Args),Env++OSXEnv]), + [lists:flatten(Configure ++ Args),UnQuotedEnv]), Port = open_port({spawn, lists:flatten(["\"",Configure,"\"",Args])}, - [stream, eof, {env,Env++OSXEnv}]), + [stream, eof, {env,UnQuotedEnv}]), ts_lib:print_data(Port); false -> {error, no_configure_script} end. +unquote([{Var,Val}|T]) -> + [{Var,unquote(Val)}|unquote(T)]; +unquote([]) -> + []; +unquote("\""++Rest) -> + lists:reverse(tl(lists:reverse(Rest))); +unquote(String) -> + String. + +assign_vars([]) -> + []; +assign_vars([{VAR,FlagsStr} | VARs]) -> + [{VAR,assign_vars(FlagsStr)} | assign_vars(VARs)]; +assign_vars(FlagsStr) -> + Flags = [assign_all_vars(Str,[]) || Str <- string:tokens(FlagsStr, [$ ])], + string:strip(lists:flatten(lists:map(fun(Flag) -> + Flag ++ " " + end, Flags)), right). + +assign_all_vars([$$ | Rest], FlagSoFar) -> + {VarName,Rest1} = get_var_name(Rest, []), + assign_all_vars(Rest1, FlagSoFar ++ assign_var(VarName)); +assign_all_vars([Char | Rest], FlagSoFar) -> + assign_all_vars(Rest, FlagSoFar ++ [Char]); +assign_all_vars([], Flag) -> + Flag. + +get_var_name([Ch | Rest] = Str, VarR) -> + case valid_char(Ch) of + true -> get_var_name(Rest, [Ch | VarR]); + false -> {lists:reverse(VarR),Str} + end; +get_var_name([], VarR) -> + {lists:reverse(VarR),[]}. + +assign_var(VarName) -> + case os:getenv(VarName) of + false -> ""; + Val -> Val + end. + +valid_char(Ch) when Ch >= $a, Ch =< $z -> true; +valid_char(Ch) when Ch >= $A, Ch =< $Z -> true; +valid_char(Ch) when Ch >= $0, Ch =< $9 -> true; +valid_char($_) -> true; +valid_char(_) -> false. + get_xcomp_flag(Flag, Flags) -> get_xcomp_flag(Flag, Flag, Flags). get_xcomp_flag(Flag, Tag, Flags) -> @@ -228,7 +281,7 @@ add_vars(Vars0, Opts0) -> {Opts, [{longnames, LongNames}, {platform_id, PlatformId}, {platform_filename, PlatformFilename}, - {rsh_name, os:getenv("ERL_RSH", "rsh")}, + {rsh_name, get_rsh_name()}, {platform_label, PlatformLabel}, {ts_net_dir, Mounted}, {erl_flags, []}, @@ -249,6 +302,12 @@ get_testcase_callback() -> end end. +get_rsh_name() -> + case os:getenv("ERL_RSH") of + false -> "rsh"; + Str -> Str + end. + platform_id(Vars) -> {Id,_,_,_} = platform(Vars), Id. diff --git a/lib/test_server/src/ts_install_cth.erl b/lib/test_server/src/ts_install_cth.erl index 7746bbed6f..3dfa6174fe 100644 --- a/lib/test_server/src/ts_install_cth.erl +++ b/lib/test_server/src/ts_install_cth.erl @@ -238,12 +238,9 @@ generate_nodenames2(0, _Hosts, Acc) -> Acc; generate_nodenames2(N, Hosts, Acc) -> Host=lists:nth((N rem (length(Hosts)))+1, Hosts), - Name=list_to_atom(temp_nodename("nod", []) ++ "@" ++ Host), + Name=list_to_atom(temp_nodename("nod") ++ "@" ++ Host), generate_nodenames2(N-1, Hosts, [Name|Acc]). -temp_nodename([], Acc) -> - lists:flatten(Acc); -temp_nodename([Chr|Base], Acc) -> - {A,B,C} = erlang:now(), - New = [Chr | integer_to_list(Chr bxor A bxor B+A bxor C+B)], - temp_nodename(Base, [New|Acc]). +temp_nodename(Base) -> + Num = erlang:unique_integer([positive]), + Base ++ integer_to_list(Num). diff --git a/lib/tools/emacs/erlang.el b/lib/tools/emacs/erlang.el index c56759ebb9..0c003bab39 100644 --- a/lib/tools/emacs/erlang.el +++ b/lib/tools/emacs/erlang.el @@ -4743,6 +4743,23 @@ for a tag on the form `module:tag'." ;;; `module:tag'. +(when (and (fboundp 'etags-tags-completion-table) + (fboundp 'tags-lazy-completion-table)) ; Emacs 23.1+ + (if (fboundp 'advice-add) + ;; Emacs 24.4+ + (advice-add 'etags-tags-completion-table :around + (lambda (oldfun) + (if (eq find-tag-default-function 'erlang-find-tag-for-completion) + (erlang-etags-tags-completion-table) + (funcall oldfun))) + (list :name 'erlang-replace-tags-table)) + ;; Emacs 23.1-24.3 + (defadvice etags-tags-completion-table (around erlang-replace-tags-table activate) + (if (eq find-tag-default-function 'erlang-find-tag-for-completion) + (setq ad-return-value (erlang-etags-tags-completion-table)) + ad-do-it)))) + + (defun erlang-complete-tag () "Perform tags completion on the text around point. Completes to the set of names listed in the current tags table. @@ -4754,7 +4771,17 @@ about Erlang modules." (require 'etags) (error nil)) (cond ((and erlang-tags-installed - (fboundp 'complete-tag)) ; Emacs 19 + (fboundp 'etags-tags-completion-table) + (fboundp 'tags-lazy-completion-table)) ; Emacs 23.1+ + ;; This depends on the advice called erlang-replace-tags-table + ;; above. It is not enough to let-bind + ;; tags-completion-table-function since that will not override + ;; the buffer-local value in the TAGS buffer. + (let ((find-tag-default-function 'erlang-find-tag-for-completion)) + (complete-tag))) + ((and erlang-tags-installed + (fboundp 'complete-tag) + (fboundp 'tags-complete-tag)) ; Emacs 19 (let ((orig-tags-complete-tag (symbol-function 'tags-complete-tag))) (fset 'tags-complete-tag (symbol-function 'erlang-tags-complete-tag)) @@ -4769,6 +4796,15 @@ about Erlang modules." (error "This version of Emacs can't complete tags")))) +(defun erlang-find-tag-for-completion () + (let ((start (save-excursion + (skip-chars-backward "[:word:][:digit:]_:'") + (point)))) + (unless (eq start (point)) + (buffer-substring-no-properties start (point))))) + + + ;; Based on `tags-complete-tag', but this one uses ;; `erlang-tags-completion-table' instead of `tags-completion-table'. ;; @@ -4816,7 +4852,12 @@ about Erlang modules." ;; the only format supported by Emacs, so far.) (defun erlang-etags-tags-completion-table () (let ((table (make-vector 511 0)) - (file nil)) + (file nil) + (progress-reporter + (when (fboundp 'make-progress-reporter) + (make-progress-reporter + (format "Making erlang tags completion table for %s..." buffer-file-name) + (point-min) (point-max))))) (save-excursion (goto-char (point-min)) ;; This monster regexp matches an etags tag line. @@ -4828,31 +4869,33 @@ about Erlang modules." ;; \6 is the line to start searching at; ;; \7 is the char to start searching at. (while (progn - (while (and - (eq (following-char) ?\f) - (looking-at "\f\n\\([^,\n]*\\),.*\n")) - (setq file (buffer-substring - (match-beginning 1) (match-end 1))) - (goto-char (match-end 0))) - (re-search-forward - "\ + (while (and + (eq (following-char) ?\f) + (looking-at "\f\n\\([^,\n]*\\),.*\n")) + (setq file (buffer-substring + (match-beginning 1) (match-end 1))) + (goto-char (match-end 0))) + (re-search-forward + "\ ^\\(\\([^\177]+[^-a-zA-Z0-9_$\177]+\\)?\\([-a-zA-Z0-9_$?:]+\\)\ \[^-a-zA-Z0-9_$?:\177]*\\)\177\\(\\([^\n\001]+\\)\001\\)?\ \\([0-9]+\\)?,\\([0-9]+\\)?\n" - nil t)) - (let ((tag (if (match-beginning 5) - ;; There is an explicit tag name. - (buffer-substring (match-beginning 5) (match-end 5)) - ;; No explicit tag name. Best guess. - (buffer-substring (match-beginning 3) (match-end 3)))) - (module (and file - (erlang-get-module-from-file-name file)))) - (intern tag table) - (if (stringp module) - (progn - (intern (concat module ":" tag) table) - ;; Only the first one will be stored in the table. - (intern (concat module ":") table)))))) + nil t)) + (let ((tag (if (match-beginning 5) + ;; There is an explicit tag name. + (buffer-substring (match-beginning 5) (match-end 5)) + ;; No explicit tag name. Best guess. + (buffer-substring (match-beginning 3) (match-end 3)))) + (module (and file + (erlang-get-module-from-file-name file)))) + (intern tag table) + (when (stringp module) + (intern (concat module ":" tag) table) + ;; Only the first ones will be stored in the table. + (intern (concat module ":") table) + (intern (concat module ":module_info") table)) + (when progress-reporter + (progress-reporter-update progress-reporter (point)))))) table)) ;;; diff --git a/lib/tools/src/cover_web.erl b/lib/tools/src/cover_web.erl index 69f2f3b1aa..75bb45c659 100644 --- a/lib/tools/src/cover_web.erl +++ b/lib/tools/src/cover_web.erl @@ -734,7 +734,7 @@ generate_filename(Prefix) -> filename:join(Cwd,Prefix ++ "_" ++ ts() ++ ".coverdata"). ts() -> - {{Y,M,D},{H,Min,S}} = calendar:now_to_local_time(now()), + {{Y,M,D},{H,Min,S}} = calendar:now_to_local_time(erlang:timestamp()), io_lib:format("~4.4.0w~2.2.0w~2.2.0w-~2.2.0w~2.2.0w~2.2.0w", [Y,M,D,H,Min,S]). diff --git a/lib/tools/src/eprof.erl b/lib/tools/src/eprof.erl index bfbbefb473..a3fef91e61 100644 --- a/lib/tools/src/eprof.erl +++ b/lib/tools/src/eprof.erl @@ -187,7 +187,7 @@ handle_call({profile_start, Rootset, Pattern, {M,F,A}, Opts}, From, #state{fd = case set_process_trace(true, [Pid|Rootset], Topts) of true -> ok = set_pattern_trace(true, Pattern), - T0 = now(), + T0 = erlang:timestamp(), ok = execute_profiling(Pid), {noreply, #state{ profiling = true, @@ -211,7 +211,7 @@ handle_call({profile_start, Rootset, Pattern, undefined, Opts}, From, #state{ fd case set_process_trace(true, Rootset, Topts) of true -> - T0 = now(), + T0 = erlang:timestamp(), ok = set_pattern_trace(true, Pattern), {reply, profiling, #state{ profiling = true, diff --git a/lib/tools/src/tags.erl b/lib/tools/src/tags.erl index e3cc51cdb2..e25db2eb1b 100644 --- a/lib/tools/src/tags.erl +++ b/lib/tools/src/tags.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2013. All Rights Reserved. +%% Copyright Ericsson AB 1996-2015. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -297,15 +297,16 @@ word_char(_) -> false. %% Check the options `outfile' and `outdir'. open_out(Options) -> + Opts = [write, {encoding, unicode}], case lists:keysearch(outfile, 1, Options) of {value, {outfile, File}} -> - file:open(File, [write]); + file:open(File, Opts); _ -> case lists:keysearch(outdir, 1, Options) of {value, {outdir, Dir}} -> - file:open(filename:join(Dir, "TAGS"), [write]); + file:open(filename:join(Dir, "TAGS"), Opts); _ -> - file:open("TAGS", [write]) + file:open("TAGS", Opts) end end. diff --git a/lib/tools/src/tools.app.src b/lib/tools/src/tools.app.src index ec5b6f3a82..a4e5d85f92 100644 --- a/lib/tools/src/tools.app.src +++ b/lib/tools/src/tools.app.src @@ -41,7 +41,7 @@ ] }, {runtime_dependencies, ["webtool-0.8.10","stdlib-2.0","runtime_tools-1.8.14", - "kernel-3.0","inets-5.10","erts-6.0", + "kernel-3.0","inets-5.10","erts-7.0", "compiler-5.0"]} ] }. diff --git a/otp_patch_apply b/otp_patch_apply new file mode 100755 index 0000000000..947aa1e6ee --- /dev/null +++ b/otp_patch_apply @@ -0,0 +1,480 @@ +#!/bin/sh +# +# %CopyrightBegin% +# +# Copyright Ericsson AB 2014. All Rights Reserved. +# +# The contents of this file are subject to the Erlang Public License, +# Version 1.1, (the "License"); you may not use this file except in +# compliance with the License. You should have received a copy of the +# Erlang Public License along with this software. If not, it can be +# retrieved online at http://www.erlang.org/. +# +# Software distributed under the License is distributed on an "AS IS" +# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See +# the License for the specific language governing rights and limitations +# under the License. +# +# %CopyrightEnd% +# + +version="1.0.1" + +force= +lib_path= +orig_dir= +sdir= +idir="/broken/path/here" +cleanup=no +install_docs=yes + +invalid_src="does not seem to be a valid OTP source tree" +not_built="Source in has not been built" +doc_not_built="Documentation has not been built. Either build the +documentation and re-run 'otp_patch_apply', or re-run 'otp_patch_apply' +with the '-n' switch." + +print_usage() +{ + cat <<EOF +otp_patch_apply -s <Dir> -i <Dir> [-l <Dir>] [-c] [-f] [-h] [-n] [-v] \\ + <App1> [... <AppN>] + + -s <Dir> -- OTP source directory that contain build results. + -i <Dir> -- OTP installation directory to patch. + -l <Dir> -- Alternative OTP source library directory path(s) containing + build results of OTP applications. Multiple paths should be + colon separated. + -c -- Cleanup (remove) old versions of applications patched + in the installation. + -f -- Force patch of application(s) even though dependencies are + not fullfilled. + -h -- Print this help then exit. + -n -- Do not install documentation. + -v -- Print version then exit. + <AppX> -- Application to patch. + +Environment Variable: + ERL_LIBS -- Alternative OTP source library directory path(s) containing + build results of OTP applications. Multiple paths should be + colon separated. + +NOTE: + * Complete build environment is required while running otp_patch_apply. + * Before applying a patch you need to build all of OTP in the source + directory. + * All source directories identified by -s and -l should contain build + results of OTP applications. + +Version: $version + +EOF + +} + +error() +{ + echo "ERROR:" "$@" 1>&2 + exit 1 +} + +usage_error() +{ + echo "ERROR:" "$@" 1>&2 + echo "" 1>&2 + print_usage 1>&2 + exit 1 +} + +usage() +{ + print_usage + exit 0 +} + +alt_lib_path() +{ + app=$1 + save_ifs=$IFS + IFS=: + + cd "$orig_dir" || error "Cannot change directory to $orig_dir" + + for lib in $lib_path; do + # Want absolute path + case "$lib" in + /*) + ;; + *) + cd "$lib" || error "Cannot change directory to $lib" + lib=`pwd` + cd "$orig_dir" || error "Cannot change directory to $orig_dir" + esac + if [ -d "$lib/$app" ]; then + echo "$lib/$app" + IFS=$save_ifs + return 0 + fi + done + + IFS=$save_ifs + + return 1 +} + +prog_in_mod_path() +{ + chk_path="/bin:$PATH" + PROG=$1 + save_ifs=$IFS + IFS=: + if [ "X$TARGET" = "Xwin32" ]; then + for p in $chk_path; do + if [ -f "$p/$PROG.exe" ]; then + IFS=$save_ifs + echo "$p/$PROG.exe" + return 0 + fi + done + else + for p in $chk_path; do + if [ -x "$p/$PROG" ]; then + IFS=$save_ifs + echo "$p/$PROG" + return 0 + fi + done + fi + IFS=$save_ifs + return 1 +} + +find_prog() +{ + prog_in_mod_path "$1" + if [ $? -ne 0 ]; then + echo "$1" + fi + return 0 +} + +# Parse arguments + +while [ $# -gt 0 ]; do + case "$1" in + "-s") + shift + if [ ! $# -gt 0 ]; then + usage_error "Missing OTP source directory" + fi + sdir="$1";; + "-i") + shift + if [ ! $# -gt 0 ]; then + usage_error "Missing OTP install directory" + fi + idir="$1";; + "-l") + shift + if [ ! $# -gt 0 ]; then + usage_error "Missing OTP library directory" + fi + if [ "x$lib_path" = "x" ]; then + lib_path="$1" + else + lib_path="$lib_path:$1" + fi;; + "-f") + force="-force";; + "-c") + cleanup=yes;; + "-h") + usage;; + "-n") + install_docs=no;; + "-v") + echo "otp_patch_apply version $version" + exit 0;; + *) + app="$1" + applications="$applications $app";; + esac + shift +done + +# Check that we got mandatory arguments +test "x$sdir" != "x" || usage_error "Missing OTP source directory" +test "x$idir" != "x" || usage_error "Missing OTP install directory" +test "x$applications" != "x" || usage_error "Missing applications" + +orig_dir=`pwd` + +# Check that the source directory seems sane +cd "$sdir" 2>/dev/null || error "Cannot change directory to $sdir" + +# Want absolute path +case "$sdir" in + /*) ;; + *) sdir=`pwd`;; +esac + +export ERL_TOP="$sdir" +test -f "$sdir/otp_build" || error "$ERL_TOP" $invalid_src +test -f "$sdir/OTP_VERSION" || error "$ERL_TOP" $invalid_src +test -f "$sdir/otp_versions.table" || error "$ERL_TOP" $invalid_src +test -f "$sdir/erts/autoconf/config.guess" || error "$ERL_TOP" $invalid_src +test -f "$sdir/make/verify_runtime_dependencies" || error "$ERL_TOP" $invalid_src +test -x "$sdir/bootstrap/bin/erl" || error $not_built +test -x "$sdir/bootstrap/bin/erlc" || error $not_built +test -x "$sdir/bootstrap/bin/escript" || error $not_built +test -f "$sdir/make/otp_built" || error $not_built + +if [ $install_docs = yes ]; then + test -f "$sdir/make/otp_doc_built" || usage_error $doc_not_built +fi + +otp_rel=`sed 's|\([0-9]*\).*|\1|' < $ERL_TOP/OTP_VERSION` || \ + error "Failed to read $ERL_TOP/OTP_VERSION" + +case "$otp_rel" in + 1[7-9]|[2-9][0-9]) ;; # ok; release 17-99 + *) error "Invalid OTP release: $otp_rel";; +esac + +export PATH="$ERL_TOP/bootstrap/bin:$PATH" +erlc="$ERL_TOP/bootstrap/bin/erlc" +erl="$ERL_TOP/bootstrap/bin/erl" + +erl_otp_rel=`$erl -noshell -noinput -eval "io:format(\"~s~n\", [erlang:system_info(otp_release)]), erlang:halt(0)"` || \ + error "Failed to execute: $erl" + +test "$otp_rel" = "$erl_otp_rel" || error "Inconsistent source: $sdir" + +app_dirs= +for app in $applications; do + case "$app" in + "erts") + dir="$ERL_TOP/erts";; + *) + dir="$ERL_TOP/lib/$app";; + esac + if [ ! -d "$dir" ]; then + dir=`alt_lib_path "$app"` + if [ $? -ne 0 ]; then + error "Application missing in source: $app" + fi + fi + app_dirs="$app_dirs $dir" +done + +cd "$orig_dir" 2>/dev/null || error "Cannot change directory to $orig_dir" + +# Check that the install directory seems sane +cd "$idir" 2>/dev/null || error "Cannot change directory to $idir" + +# Want absolute path +case "$idir" in + /*) ;; + *) idir=`pwd`;; +esac + +test -d "$idir/releases/$otp_rel" || \ + error "No OTP-$otp_rel installation present in $idir" + +cd "$ERL_TOP" 2>/dev/null || error "Cannot change directory to $ERL_TOP" + +# Some tools we use +rm=`find_prog rm` +rmdir=`find_prog rmdir` +cp=`find_prog cp` +mv=`find_prog mv` +mkdir=`find_prog mkdir` + +# Setup build stuff +if [ "x$TARGET" = "x" ]; then + TARGET=`$ERL_TOP/erts/autoconf/config.guess` +fi +BUILDSYS=$TARGET +if [ -z "$MAKE" ]; then + case $TARGET in + win32) + MAKE=make;; + *) + prog_in_mod_path gmake >/dev/null + if [ $? -eq 0 ]; then + MAKE=gmake + else + MAKE=make + fi;; + esac +fi +if [ X`$MAKE is_cross_configured` = Xyes ]; then + TARGET=`$MAKE target_configured` +elif [ "x$OVERRIDE_TARGET" != "x" -a "x$OVERRIDE_TARGET" != "xwin32" ]; then + TARGET=$OVERRIDE_TARGET +fi + +# Check for cleanup +inst_app_vers="$idir/releases/$otp_rel/installed_application_versions" +rm_app_vers= +if [ $cleanup = yes ]; then + $mv "$inst_app_vers" "${inst_app_vers}.save" || \ + error "Failed to save $inst_app_vers" + for app in $applications; do + tmp=`grep "$app-*" "${inst_app_vers}.save"` + rm_app_vers="$rm_app_vers $tmp" + done + $cp "${inst_app_vers}.save" "$inst_app_vers" + for rm_app_ver in $rm_app_vers; do + $cp "$inst_app_vers" "${inst_app_vers}.tmp" + grep -v $rm_app_ver "${inst_app_vers}.tmp" > "$inst_app_vers" + done + $rm -f "${inst_app_vers}.tmp" +fi + +# Verify runtime dependencies +$ERL_TOP/make/verify_runtime_dependencies -release "$otp_rel" \ + -source "$ERL_TOP" -target "$idir" $force $applications || { + test ! -f "${inst_app_vers}.save" || \ + $mv "${inst_app_vers}.save" "$inst_app_vers" + exit 1 +} + +# Update OTP_VERSION in installation +otp_version=`cat "$idir/releases/$otp_rel/OTP_VERSION"` || { + test ! -f "${inst_app_vers}.save" || \ + $mv "${inst_app_vers}.save" "$inst_app_vers" + error "Not able to read $idir/releases/$otp_rel/OTP_VERSION" +} + +{ + echo "$otp_version" | sed "s|^\([^\*]*\)\**|\1\*\*|g" > \ + "$idir/releases/$otp_rel/OTP_VERSION" +} 2>/dev/null || { + test ! -f "${inst_app_vers}.save" || \ + $mv "${inst_app_vers}.save" "$inst_app_vers" + error "Not able to update $idir/releases/$otp_rel/OTP_VERSION" +} + +# Do actual cleanup +if [ "x$rm_app_vers" != "x" ]; then + for app_ver in $rm_app_vers; do + case x"$app_ver" in + x) + ;; + xerts-*) + $rm -rf "$idir/$app_ver" ;; + x*) + $rm -rf "$idir/lib/$app_ver" ;; + esac + done + $rm -f "${inst_app_vers}.save" +fi + +# Install application from built source +for app_dir in $app_dirs; do + (cd "$app_dir" && \ + $MAKE MAKE="$MAKE" TARGET=$TARGET RELEASE_ROOT="$idir" \ + RELEASE_PATH="$idir" TESTROOT="$idir" release) || exit 1 +done + +if [ $install_docs = yes ]; then +# Documentation have been built and should be installed + + for app_dir in $app_dirs; do + (cd "$app_dir" && \ + $MAKE MAKE="$MAKE" RELEASE_ROOT="$idir" RELEASE_PATH="$idir" \ + TESTROOT="$idir" release_docs) || exit 1 + done + + (cd "$sdir/system/doc/top" && $MAKE clean) + + (cd "$sdir/system/doc/top" && \ + $MAKE MAKE="$MAKE" RELEASE_ROOT="$idir" RELEASE_PATH="$idir" \ + TESTROOT="$idir" release_docs) || exit 1 + + echo "" + echo "*" + echo "* NOTE! In order to update pre-formatted man pages you" + echo "* need to run the 'Install' script located in:" + echo "* $idir" + echo "*" +fi + +# If erts, kernel, stdlib or sasl is included, find versions +for app in $applications; do + case "$app" in + erts) + erts_vsn=`grep '^VSN' erts/vsn.mk | sed "s|^VSN.*=[^0-9]*\([0-9].*\)$|\1|g"` + update_rel=true;; + kernel) + kernel_vsn=`sed "s|^KERNEL_VSN[^=]*=[^0-9]*\([0-9].*\)$|\1|g" lib/kernel/vsn.mk` + update_rel=true;; + stdlib) + stdlib_vsn=`sed "s|^STDLIB_VSN[^=]*=[^0-9]*\([0-9].*\)$|\1|g" lib/stdlib/vsn.mk` + update_rel=true;; + sasl) + sasl_vsn=`sed "s|^SASL_VSN[^=]*=[^0-9]*\([0-9].*\)$|\1|g" lib/sasl/vsn.mk` + update_rel=true;; + *) + ;; + esac +done + +# and find the old versions for those not included +if [ "X$update_rel" != "X" ]; then + if [ "X$erts_vsn" = "X" ]; then + erts_vsns=`ls -d "$idir"/erts-* | sed "s|$idir/erts-\([0-9\.].*\)|\1|g"` + erts_vsn=`echo "$erts_vsns" | sort -t '.' -g | tail -n 1` + fi + if [ "X$kernel_vsn" = "X" ]; then + kernel_vsns=`ls -d "$idir"/lib/kernel-* | sed "s|$idir/lib/kernel-\([0-9\.].*\)|\1|g"` + kernel_vsn=`echo "$kernel_vsns" | sort -t '.' -g | tail -n 1` + fi + if [ "X$stdlib_vsn" = "X" ]; then + stdlib_vsns=`ls -d "$idir"/lib/stdlib-* | sed "s|$idir/lib/stdlib-\([0-9\.].*\)|\1|g"` + stdlib_vsn=`echo "$stdlib_vsns" | sort -t '.' -g | tail -n 1` + fi + if [ "X$sasl_vsn" = "X" ]; then + sasl_vsns=`ls -d "$idir"/lib/sasl-* | sed "s|$idir/lib/sasl-\([0-9\.].*\)|\1|g"` + sasl_vsn=`echo "$sasl_vsns" | sort -t '.' -g | tail -n 1` + fi + + # Generate .rel, .script and .boot - to tmp dir + start_clean="{release, {\"Erlang/OTP\",\"$otp_rel\"}, {erts, \"$erts_vsn\"}, [{kernel,\"$kernel_vsn\"}, {stdlib,\"$stdlib_vsn\"}]}." + start_sasl="{release, {\"Erlang/OTP\",\"$otp_rel\"}, {erts, \"$erts_vsn\"}, [{kernel,\"$kernel_vsn\"}, {stdlib,\"$stdlib_vsn\"}, {sasl,\"$sasl_vsn\"}]}." + + tmp_dir="$idir/tmp"; + if [ ! -d "$tmp_dir" ]; then + $mkdir "$tmp_dir" + fi + echo "$start_sasl" > "$tmp_dir/start_sasl.rel" + echo "$start_clean" > "$tmp_dir/start_clean.rel" + echo "$start_clean" > "$tmp_dir/no_dot_erlang.rel" + + $erlc -I"$idir"/lib/*/ebin -o"$tmp_dir" "$tmp_dir/start_sasl.rel" || exit 1 + $erlc -I"$idir"/lib/*/ebin -o"$tmp_dir" +no_warn_sasl "$tmp_dir/start_clean.rel" || exit 1 + $erlc -I"$idir"/lib/*/ebin -o"$tmp_dir" +no_warn_sasl +no_dot_erlang "$tmp_dir/no_dot_erlang.rel" || exit 1 + + # Generate RELEASES file + "$erl" -noinput +B -eval "release_handler:create_RELEASES(\"%ERL_ROOT%\", \"$tmp_dir\", \"$tmp_dir/start_sasl.rel\", []), halt()" || exit 1 + + # If all good so far, move generated files into target area + $mv "$tmp_dir/RELEASES" "$idir/releases/RELEASES.src" + $mv "$tmp_dir"/* "$idir/releases/$otp_rel" + $rmdir "$tmp_dir" + + # Remove old start scripts (forces a new run of Install) + $rm -f "$idir"/releases/RELEASES + $rm -f "$idir"/bin/*.script + $rm -f "$idir"/bin/*.boot + $rm -f "$idir"/bin/erl + + echo "" + echo "*" + echo "* NOTE! In order to get a runnable OTP system again you" + echo "* need to run the 'Install' script located in:" + echo "* $idir" + echo "*" +fi + diff --git a/system/doc/design_principles/applications.xml b/system/doc/design_principles/applications.xml index 7b030115df..9d3a204999 100644 --- a/system/doc/design_principles/applications.xml +++ b/system/doc/design_principles/applications.xml @@ -29,55 +29,63 @@ <file>applications.xml</file> </header> <marker id="appl"></marker> - <p>This chapter should be read in conjunction with <c>app(4)</c> and - <c>application(3)</c>.</p> + <p>This section is to be read with the <c>app(4)</c> and + <c>application(3)</c> manual pages in Kernel.</p> <section> <title>Application Concept</title> - <p>When we have written code implementing some specific - functionality, we might want to make the code into an - <em>application</em>, that is a component that can be started and - stopped as a unit, and which can be re-used in other systems as - well.</p> - <p>To do this, we create an - <seealso marker="#callback_module">application callback module</seealso>, where we describe how the application should - be started and stopped.</p> + <p>When you have written code implementing some specific functionality + you might want to make the code into an <em>application</em>, + that is, a component that can be started and stopped as a unit, + and which can also be reused in other systems.</p> + <p>To do this, create an + <seealso marker="#callback_module">application callback module</seealso>, + and describe how the application is to be started and stopped.</p> <p>Then, an <em>application specification</em> is needed, which is - put in an <seealso marker="#appl_res_file">application resource file</seealso>. Among other things, we specify which - modules the application consists of and the name of the callback - module.</p> - <p>If we use <c>systools</c>, the Erlang/OTP tools for packaging code + put in an + <seealso marker="#appl_res_file">application resource file</seealso>. + Among other things, this file specifies which modules the application + consists of and the name of the callback module.</p> + <p>If you use <c>systools</c>, the Erlang/OTP tools for packaging code (see <seealso marker="release_structure">Releases</seealso>), - the code for each application is placed in a separate directory - following a pre-defined <seealso marker="#app_dir">directory structure</seealso>.</p> + the code for each application is placed in a + separate directory following a pre-defined + <seealso marker="#app_dir">directory structure</seealso>.</p> </section> <section> <marker id="callback_module"></marker> <title>Application Callback Module</title> - <p>How to start and stop the code for the application, i.e. + <p>How to start and stop the code for the application, that is, the supervision tree, is described by two callback functions:</p> <code type="none"> start(StartType, StartArgs) -> {ok, Pid} | {ok, Pid, State} -stop(State)</code> - <p><c>start</c> is called when starting the application and should - create the supervision tree by starting the top supervisor. - It is expected to return the pid of the top supervisor and an - optional term <c>State</c>, which defaults to []. This term is - passed as-is to <c>stop</c>.</p> - <p><c>StartType</c> is usually the atom <c>normal</c>. It has other +stop(State) + </code> + <list type="bulleted"> + <item><c>start</c> is called when starting the application and is to + create the supervision tree by starting the top supervisor. It is + expected to return the pid of the top supervisor and an optional + term, <c>State</c>, which defaults to <c>[]</c>. This term is passed + as is to <c>stop</c>.</item> + <item><c>StartType</c> is usually the atom <c>normal</c>. It has other values only in the case of a takeover or failover, see - <seealso marker="distributed_applications">Distributed Applications</seealso>. <c>StartArgs</c> is defined by the key - <c>mod</c> in the <seealso marker="#appl_res_file">application resource file</seealso> file.</p> - <p><c>stop/1</c> is called <em>after</em> the application has been - stopped and should do any necessary cleaning up. Note that - the actual stopping of the application, that is the shutdown of - the supervision tree, is handled automatically as described in - <seealso marker="#stopping">Starting and Stopping Applications</seealso>.</p> + <seealso marker="distributed_applications">Distributed Applications</seealso>. + </item> + <item><c>StartArgs</c> is defined by the key <c>mod</c> in the + <seealso marker="#appl_res_file">application + resource file</seealso>.</item> + <item><c>stop/1</c> is called <em>after</em> the application has been + stopped and is to do any necessary cleaning up. The actual stopping of + the application, that is, the shutdown of the supervision tree, is + handled automatically as described in + <seealso marker="#stopping">Starting and Stopping Applications</seealso>. + </item> + </list> <marker id="ch_app"></marker> <p>Example of an application callback module for packaging the supervision tree from - the <seealso marker="sup_princ#ex">Supervisor</seealso> chapter:</p> + <seealso marker="sup_princ#ex">Supervisor Behaviour</seealso>:</p> <code type="none"> -module(ch_app). -behaviour(application). @@ -89,44 +97,48 @@ start(_Type, _Args) -> stop(_State) -> ok.</code> - <p>A library application, which can not be started or stopped, - does not need any application callback module.</p> + <p>A library application that cannot be started or stopped, does not + need any application callback module.</p> </section> <section> <marker id="appl_res_file"></marker> <title>Application Resource File</title> - <p>To define an application, we create an <em>application specification</em> which is put in an <em>application resource file</em>, or in short <c>.app</c> file:</p> + <p>To define an application, an <em>application specification</em> is + created, which is put in an <em>application resource file</em>, or in + short an <c>.app</c> file:</p> <code type="none"> {application, Application, [Opt1,...,OptN]}.</code> - <p><c>Application</c>, an atom, is the name of the application. - The file must be named <c>Application.app</c>.</p> - <p>Each <c>Opt</c> is a tuple <c>{Key, Value}</c> which define a +<list> + <item><c>Application</c>, an atom, is the name of the application. + The file must be named <c>Application.app</c>.</item> + <item>Each <c>Opt</c> is a tuple <c>{Key,Value}</c>, which define a certain property of the application. All keys are optional. - Default values are used for any omitted keys.</p> + Default values are used for any omitted keys.</item> +</list> <p>The contents of a minimal <c>.app</c> file for a library - application <c>libapp</c> looks like this:</p> + application <c>libapp</c> looks as follows:</p> <code type="none"> {application, libapp, []}.</code> <p>The contents of a minimal <c>.app</c> file <c>ch_app.app</c> for - a supervision tree application like <c>ch_app</c> looks like this:</p> + a supervision tree application like <c>ch_app</c> looks as follows:</p> <code type="none"> {application, ch_app, [{mod, {ch_app,[]}}]}.</code> - <p>The key <c>mod</c> defines the callback module and start - argument of the application, in this case <c>ch_app</c> and - [], respectively. This means that</p> + <p>The key <c>mod</c> defines the callback module and start argument of + the application, in this case <c>ch_app</c> and <c>[]</c>, respectively. + This means that the following is called when the application is to be + started:</p> <code type="none"> ch_app:start(normal, [])</code> - <p>will be called when the application should be started and</p> + <p>The following is called when the application is stopped.</p> <code type="none"> ch_app:stop([])</code> - <p>will be called when the application has been stopped.</p> <p>When using <c>systools</c>, the Erlang/OTP tools for packaging - code (see <seealso marker="release_structure">Releases</seealso>), - the keys <c>description</c>, <c>vsn</c>, <c>modules</c>, - <c>registered</c> and <c>applications</c> should also be - specified:</p> + code (see Section + <seealso marker="release_structure">Releases</seealso>), the keys + <c>description</c>, <c>vsn</c>, <c>modules</c>, <c>registered</c>, + and <c>applications</c> are also to be specified:</p> <code type="none"> {application, ch_app, [{description, "Channel allocator"}, @@ -136,67 +148,54 @@ ch_app:stop([])</code> {applications, [kernel, stdlib, sasl]}, {mod, {ch_app,[]}} ]}.</code> - <taglist> - <tag><c>description</c></tag> - <item>A short description, a string. Defaults to "".</item> - <tag><c>vsn</c></tag> - <item>Version number, a string. Defaults to "".</item> - <tag><c>modules</c></tag> - <item>All modules <em>introduced</em> by this application. - <c>systools</c> uses this list when generating boot scripts and - tar files. A module must be defined in one and only one - application. Defaults to [].</item> - <tag><c>registered</c></tag> - <item>All names of registered processes in the application. - <c>systools</c> uses this list to detect name clashes - between applications. Defaults to [].</item> - <tag><c>applications</c></tag> - <item>All applications which must be started before this - application is started. <c>systools</c> uses this list to - generate correct boot scripts. Defaults to [], but note that - all applications have dependencies to at least <c>kernel</c> - and <c>stdlib</c>.</item> - </taglist> - <note><p>The syntax and contents of of the application resource file - are described in detail in the<seealso marker="kernel:app"> - Application resource file reference</seealso>.</p></note> + <list> + <item><c>description</c> - A short description, a string. Defaults to + "".</item> + <item><c>vsn</c> - Version number, a string. Defaults to "".</item> + <item><c>modules</c> - All modules <em>introduced</em> by this + application. <c>systools</c> uses this list when generating boot scripts + and tar files. A module must be defined in only one application. + Defaults to <c>[]</c>.</item> + <item><c>registered</c> - All names of registered processes in the + application. <c>systools</c> uses this list to detect name clashes + between applications. Defaults to <c>[]</c>.</item> + <item><c>applications</c> - All applications that must be + started before this application is started. <c>systools</c> uses this + list to generate correct boot scripts. Defaults to <c>[]</c>. Notice + that all applications have dependencies to at least Kernel + and STDLIB.</item> + </list> + <note><p>For details about the syntax and contents of the application + resource file, see the <seealso marker="kernel:app">app</seealso> + manual page in Kernel.</p></note> </section> <section> <marker id="app_dir"></marker> <title>Directory Structure</title> <p>When packaging code using <c>systools</c>, the code for each - application is placed in a separate directory + application is placed in a separate directory, <c>lib/Application-Vsn</c>, where <c>Vsn</c> is the version number.</p> - <p>This may be useful to know, even if <c>systools</c> is not used, - since Erlang/OTP itself is packaged according to the OTP principles + <p>This can be useful to know, even if <c>systools</c> is not used, + since Erlang/OTP is packaged according to the OTP principles and thus comes with this directory structure. The code server - (see <c>code(3)</c>) will automatically use code from - the directory with the highest version number, if there are - more than one version of an application present.</p> - <p>The application directory structure can of course be used in - the development environment as well. The version number may then + (see the <c>code(3)</c> manual page in Kernel) automatically + uses code from + the directory with the highest version number, if more than one + version of an application is present.</p> + <p>The application directory structure can also be used in the + development environment. The version number can then be omitted from the name.</p> - <p>The application directory have the following sub-directories:</p> + <p>The application directory has the following sub-directories:</p> <list type="bulleted"> - <item><c>src</c></item> - <item><c>ebin</c></item> - <item><c>priv</c></item> - <item><c>include</c></item> + <item><c>src</c> - Contains the Erlang source code.</item> + <item><c>ebin</c> - Contains the Erlang object code, the + <c>beam</c> files. The <c>.app</c> file is also placed here.</item> + <item><c>priv</c> - Used for application specific files. For + example, C executables are placed here. The function + <c>code:priv_dir/1</c> is to be used to access this directory.</item> + <item><c>include</c> - Used for include files.</item> </list> - <taglist> - <tag><c>src</c></tag> - <item>Contains the Erlang source code.</item> - <tag><c>ebin</c></tag> - <item>Contains the Erlang object code, the <c>beam</c> files. - The <c>.app</c> file is also placed here.</item> - <tag><c>priv</c></tag> - <item>Used for application specific files. For example, C - executables are placed here. The function <c>code:priv_dir/1</c> - should be used to access this directory.</item> - <tag><c>include</c></tag> - <item>Used for include files.</item> - </taglist> </section> <section> @@ -207,17 +206,17 @@ ch_app:stop([])</code> processes is the <em>application controller</em> process, registered as <c>application_controller</c>.</p> <p>All operations on applications are coordinated by the application - controller. It is interfaced through the functions in - the module <c>application</c>, see <c>application(3)</c>. - In particular, applications can be loaded, unloaded, started and - stopped.</p> + controller. It is interacted through the functions in + the module <c>application</c>, see the <c>application(3)</c> + manual page in Kernel. In particular, applications can be + loaded, unloaded, started, and stopped.</p> </section> <section> <title>Loading and Unloading Applications</title> <p>Before an application can be started, it must be <em>loaded</em>. The application controller reads and stores the information from - the <c>.app</c> file.</p> + the <c>.app</c> file:</p> <pre> 1> <input>application:load(ch_app).</input> ok @@ -236,7 +235,7 @@ ok {stdlib,"ERTS CXC 138 10","1.11.4.3"}]</pre> <note> <p>Loading/unloading an application does not load/unload the code - used by the application. Code loading is done the usual way.</p> + used by the application. Code loading is done the usual way.</p> </note> </section> @@ -252,13 +251,14 @@ ok {stdlib,"ERTS CXC 138 10","1.11.4.3"}, {ch_app,"Channel allocator","1"}]</pre> <p>If the application is not already loaded, the application - controller will first load it using <c>application:load/1</c>. It - will check the value of the <c>applications</c> key, to ensure - that all applications that should be started before this + controller first loads it using <c>application:load/1</c>. It + checks the value of the <c>applications</c> key, to ensure + that all applications that are to be started before this application are running.</p> <marker id="application_master"></marker> - <p>The application controller then creates an <em>application master</em> for the application. The application master is - the group leader of all the processes in the application. + <p>The application controller then creates an + <em>application master</em> for the application. The application master + is the group leader of all the processes in the application. The application master starts the application by calling the application callback function <c>start/2</c> in the module, and with the start argument, defined by the <c>mod</c> key in @@ -268,8 +268,8 @@ ok 7> <input>application:stop(ch_app).</input> ok</pre> <p>The application master stops the application by telling the top - supervisor to shutdown. The top supervisor tells all its child - processes to shutdown etc. and the entire tree is terminated in + supervisor to shut down. The top supervisor tells all its child + processes to shut down, and so on; the entire tree is terminated in reversed start order. The application master then calls the application callback function <c>stop/1</c> in the module defined by the <c>mod</c> key.</p> @@ -277,8 +277,10 @@ ok</pre> <section> <title>Configuring an Application</title> - <p>An application can be configured using <em>configuration parameters</em>. These are a list of <c>{Par, Val}</c> tuples - specified by a key <c>env</c> in the <c>.app</c> file.</p> + <p>An application can be configured using + <em>configuration parameters</em>. These are a list of + <c>{Par,Val}</c> tuples + specified by a key <c>env</c> in the <c>.app</c> file:</p> <code type="none"> {application, ch_app, [{description, "Channel allocator"}, @@ -289,11 +291,12 @@ ok</pre> {mod, {ch_app,[]}}, {env, [{file, "/usr/local/log"}]} ]}.</code> - <p><c>Par</c> should be an atom, <c>Val</c> is any term. + <p><c>Par</c> is to be an atom. <c>Val</c> is any term. The application can retrieve the value of a configuration parameter by calling <c>application:get_env(App, Par)</c> or a - number of similar functions, see <c>application(3)</c>.</p> - <p>Example:</p> + number of similar functions, see the <c>application(3)</c> + manual page in Kernel.</p> + <p><em>Example:</em></p> <pre> % <input>erl</input> Erlang (BEAM) emulator version 5.2.3.6 [hipe] [threads:0] @@ -304,20 +307,21 @@ ok 2> <input>application:get_env(ch_app, file).</input> {ok,"/usr/local/log"}</pre> <p>The values in the <c>.app</c> file can be overridden by values - in a <em>system configuration file</em>. This is a file which + in a <em>system configuration file</em>. This is a file that contains configuration parameters for relevant applications:</p> <code type="none"> [{Application1, [{Par11,Val11},...]}, ..., {ApplicationN, [{ParN1,ValN1},...]}].</code> - <p>The system configuration should be called <c>Name.config</c> and - Erlang should be started with the command line argument - <c>-config Name</c>. See <c>config(4)</c> for more information.</p> - <p>Example: A file <c>test.config</c> is created with the following - contents:</p> + <p>The system configuration is to be called <c>Name.config</c> and + Erlang is to be started with the command-line argument + <c>-config Name</c>. For details, see the <c>config(4)</c> + manual page in Kernel.</p> + <p><em>Example:</em></p> + <p>A file <c>test.config</c> is created with the following contents:</p> <code type="none"> [{ch_app, [{file, "testlog"}]}].</code> - <p>The value of <c>file</c> will override the value of <c>file</c> + <p>The value of <c>file</c> overrides the value of <c>file</c> as defined in the <c>.app</c> file:</p> <pre> % <input>erl -config test</input> @@ -330,14 +334,14 @@ ok {ok,"testlog"}</pre> <p>If <seealso marker="release_handling#sys">release handling</seealso> - is used, exactly one system configuration file should be used and - that file should be called <c>sys.config</c></p> - <p>The values in the <c>.app</c> file, as well as the values in a - system configuration file, can be overridden directly from + is used, exactly one system configuration file is to be used and + that file is to be called <c>sys.config</c>.</p> + <p>The values in the <c>.app</c> file and the values in a + system configuration file can be overridden directly from the command line:</p> <pre> % <input>erl -ApplName Par1 Val1 ... ParN ValN</input></pre> - <p>Example:</p> + <p><em>Example:</em></p> <pre> % <input>erl -ch_app file '"testlog"'</input> Erlang (BEAM) emulator version 5.2.3.6 [hipe] [threads:0] @@ -368,10 +372,10 @@ application:start(Application, Type)</code> <item>If a temporary application terminates, this is reported but no other applications are terminated.</item> </list> - <p>It is always possible to stop an application explicitly by + <p>An application can always be stopped explicitly by calling <c>application:stop/1</c>. Regardless of the mode, no - other applications will be affected.</p> - <p>Note that transient mode is of little practical use, since when + other applications are affected.</p> + <p>The transient mode is of little practical use, since when a supervision tree terminates, the reason is set to <c>shutdown</c>, not <c>normal</c>.</p> </section> diff --git a/system/doc/design_principles/appup_cookbook.xml b/system/doc/design_principles/appup_cookbook.xml index 22c48db855..63adea8a5c 100644 --- a/system/doc/design_principles/appup_cookbook.xml +++ b/system/doc/design_principles/appup_cookbook.xml @@ -28,15 +28,15 @@ <rev></rev> <file>appup_cookbook.xml</file> </header> - <p>This chapter contains examples of <c>.appup</c> files for - typical cases of upgrades/downgrades done in run-time.</p> + <marker id="appup cookbook"></marker> + <p>This section includes examples of <c>.appup</c> files for + typical cases of upgrades/downgrades done in runtime.</p> <section> <title>Changing a Functional Module</title> - <p>When a change has been made to a functional module, for example + <p>When a functional module has been changed, for example, if a new function has been added or a bug has been corrected, - simple code replacement is sufficient.</p> - <p>Example:</p> + simple code replacement is sufficient, for example:</p> <code type="none"> {"2", [{"1", [{load_module, m}]}], @@ -46,29 +46,31 @@ <section> <title>Changing a Residence Module</title> - <p>In a system implemented according to the OTP Design Principles, + <p>In a system implemented according to the OTP design principles, all processes, except system processes and special processes, reside in one of the behaviours <c>supervisor</c>, - <c>gen_server</c>, <c>gen_fsm</c> or <c>gen_event</c>. These + <c>gen_server</c>, <c>gen_fsm</c>, or <c>gen_event</c>. These belong to the STDLIB application and upgrading/downgrading normally requires an emulator restart.</p> - <p>OTP thus provides no support for changing residence modules - except in the case of <seealso marker="#spec">special processes</seealso>.</p> + <p>OTP thus provides no support for changing residence modules except + in the case of <seealso marker="#spec">special processes</seealso>.</p> </section> <section> <title>Changing a Callback Module</title> - <p>A callback module is a functional module, and for code + <p>A callback module is a functional module, and for code extensions simple code replacement is sufficient.</p> - <p>Example: When adding a function to <c>ch3</c> as described in - the example in <seealso marker="release_handling#appup">Release Handling</seealso>, <c>ch_app.appup</c> looks as follows:</p> + <p><em>Example:</em> When adding a function to <c>ch3</c>, + as described in the example in + <seealso marker="release_handling#appup">Release Handling</seealso>, + <c>ch_app.appup</c> looks as follows:</p> <code type="none"> {"2", [{"1", [{load_module, ch3}]}], [{"1", [{load_module, ch3}]}] }.</code> <p>OTP also supports changing the internal state of behaviour - processes, see <seealso marker="#int_state">Changing Internal State</seealso> below.</p> + processes, see <seealso marker="#int_state">Changing Internal State</seealso>.</p> </section> <section> @@ -77,26 +79,28 @@ <p>In this case, simple code replacement is not sufficient. The process must explicitly transform its state using the callback function <c>code_change</c> before switching to the new version - of the callback module. Thus synchronized code replacement is + of the callback module. Thus, synchronized code replacement is used.</p> - <p>Example: Consider the gen_server <c>ch3</c> from the chapter - about the <seealso marker="gen_server_concepts#ex">gen_server behaviour</seealso>. The internal state is a term <c>Chs</c> - representing the available channels. Assume we want add a counter - <c>N</c> which keeps track of the number of <c>alloc</c> requests - so far. This means we need to change the format to + <p><em>Example:</em> Consider <c>gen_server</c> <c>ch3</c> from + <seealso marker="gen_server_concepts#ex">gen_server Behaviour</seealso>. + The internal state is a term <c>Chs</c> + representing the available channels. Assume you want to add a counter + <c>N</c>, which keeps track of the number of <c>alloc</c> requests + so far. This means that the format must be changed to <c>{Chs,N}</c>.</p> - <p>The <c>.appup</c> file could look as follows:</p> + <p>The <c>.appup</c> file can look as follows:</p> <code type="none"> {"2", [{"1", [{update, ch3, {advanced, []}}]}], [{"1", [{update, ch3, {advanced, []}}]}] }.</code> <p>The third element of the <c>update</c> instruction is a tuple - <c>{advanced,Extra}</c> which says that the affected processes - should do a state transformation before loading the new version + <c>{advanced,Extra}</c>, which says that the affected processes + are to do a state transformation before loading the new version of the module. This is done by the processes calling the callback - function <c>code_change</c> (see <c>gen_server(3)</c>). The term - <c>Extra</c>, in this case [], is passed as-is to the function:</p> + function <c>code_change</c> (see the <c>gen_server(3)</c> manual + page in STDLIB). The term <c>Extra</c>, in this case + <c>[]</c>, is passed as is to the function:</p> <marker id="code_change"></marker> <code type="none"> -module(ch3). @@ -107,40 +111,41 @@ code_change({down, _Vsn}, {Chs, N}, _Extra) -> {ok, Chs}; code_change(_Vsn, Chs, _Extra) -> {ok, {Chs, 0}}.</code> - <p>The first argument is <c>{down,Vsn}</c> in case of a downgrade, - or <c>Vsn</c> in case of an upgrade. The term <c>Vsn</c> is - fetched from the 'original' version of the module, i.e. - the version we are upgrading from, or downgrading to.</p> + <p>The first argument is <c>{down,Vsn}</c> if there is a downgrade, + or <c>Vsn</c> if there is a upgrade. The term <c>Vsn</c> is + fetched from the 'original' version of the module, that is, + the version you are upgrading from, or downgrading to.</p> <p>The version is defined by the module attribute <c>vsn</c>, if any. There is no such attribute in <c>ch3</c>, so in this case - the version is the checksum (a huge integer) of the BEAM file, an - uninteresting value which is ignored.</p> - <p>(The other callback functions of <c>ch3</c> need to be modified - as well and perhaps a new interface function added, this is not - shown here).</p> + the version is the checksum (a huge integer) of the beam file, an + uninteresting value, which is ignored.</p> + <p>The other callback functions of <c>ch3</c> must also be modified + and perhaps a new interface function must be added, but this is not + shown here.</p> </section> <section> <title>Module Dependencies</title> - <p>Assume we extend a module by adding a new interface function, as - in the example in <seealso marker="release_handling#appup">Release Handling</seealso>, where a function <c>available/0</c> is - added to <c>ch3</c>.</p> - <p>If we also add a call to this function, say in the module - <c>m1</c>, a run-time error could occur during release upgrade if + <p>Assume that a module is extended by adding an interface function, + as in the example in + <seealso marker="release_handling#appup">Release Handling</seealso>, + where a function <c>available/0</c> is added to <c>ch3</c>.</p> + <p>If a call is added to this function, say in module + <c>m1</c>, a runtime error could can occur during release upgrade if the new version of <c>m1</c> is loaded first and calls <c>ch3:available/0</c> before the new version of <c>ch3</c> is loaded.</p> - <p>Thus, <c>ch3</c> must be loaded before <c>m1</c> is, in - the upgrade case, and vice versa in the downgrade case. We say - that <c>m1</c> <em>is dependent on</em> <c>ch3</c>. In a release - handling instruction, this is expressed by the element - <c>DepMods</c>:</p> + <p>Thus, <c>ch3</c> must be loaded before <c>m1</c>, in + the upgrade case, and conversely in the downgrade case. + <c>m1</c> is said to be <em>dependent on</em> <c>ch3</c>. In a release + handling instruction, this is expressed by the + <c>DepMods</c> element:</p> <code type="none"> {load_module, Module, DepMods} {update, Module, {advanced, Extra}, DepMods}</code> <p><c>DepMods</c> is a list of modules, on which <c>Module</c> is dependent.</p> - <p>Example: The module <c>m1</c> in the application <c>myapp</c> is + <p><em>Example:</em> The module <c>m1</c> in application <c>myapp</c> is dependent on <c>ch3</c> when upgrading from "1" to "2", or downgrading from "2" to "1":</p> <code type="none"> @@ -157,8 +162,8 @@ ch_app.appup: [{"1", [{load_module, ch3}]}], [{"1", [{load_module, ch3}]}] }.</code> - <p>If <c>m1</c> and <c>ch3</c> had belonged to the same application, - the <c>.appup</c> file could have looked like this:</p> + <p>If instead <c>m1</c> and <c>ch3</c> belong to the same application, + the <c>.appup</c> file can look as follows:</p> <code type="none"> {"2", [{"1", @@ -168,48 +173,48 @@ ch_app.appup: [{load_module, ch3}, {load_module, m1, [ch3]}]}] }.</code> - <p>Note that it is <c>m1</c> that is dependent on <c>ch3</c> also + <p><c>m1</c> is dependent on <c>ch3</c> also when downgrading. <c>systools</c> knows the difference between - up- and downgrading and will generate a correct <c>relup</c>, - where <c>ch3</c> is loaded before <c>m1</c> when upgrading but + up- and downgrading and generates a correct <c>relup</c>, + where <c>ch3</c> is loaded before <c>m1</c> when upgrading, but <c>m1</c> is loaded before <c>ch3</c> when downgrading.</p> </section> <section> <marker id="spec"></marker> - <title>Changing Code For a Special Process</title> + <title>Changing Code for a Special Process</title> <p>In this case, simple code replacement is not sufficient. When a new version of a residence module for a special process is loaded, the process must make a fully qualified call to - its loop function to switch to the new code. Thus synchronized + its loop function to switch to the new code. Thus, synchronized code replacement must be used.</p> <note> <p>The name(s) of the user-defined residence module(s) must be listed in the <c>Modules</c> part of the child specification - for the special process, in order for the release handler to + for the special process. Otherwise the release handler cannot find the process.</p> </note> - <p>Example. Consider the example <c>ch4</c> from the chapter about + <p><em>Example:</em> Consider the example <c>ch4</c> in <seealso marker="spec_proc#ex">sys and proc_lib</seealso>. - When started by a supervisor, the child specification could look - like this:</p> + When started by a supervisor, the child specification can look + as follows:</p> <code type="none"> {ch4, {ch4, start_link, []}, permanent, brutal_kill, worker, [ch4]}</code> <p>If <c>ch4</c> is part of the application <c>sp_app</c> and a new - version of the module should be loaded when upgrading from - version "1" to "2" of this application, <c>sp_app.appup</c> could - look like this:</p> + version of the module is to be loaded when upgrading from + version "1" to "2" of this application, <c>sp_app.appup</c> can + look as follows:</p> <code type="none"> {"2", [{"1", [{update, ch4, {advanced, []}}]}], [{"1", [{update, ch4, {advanced, []}}]}] }.</code> <p>The <c>update</c> instruction must contain the tuple - <c>{advanced,Extra}</c>. The instruction will make the special + <c>{advanced,Extra}</c>. The instruction makes the special process call the callback function <c>system_code_change/4</c>, a function the user must implement. The term <c>Extra</c>, in this - case [], is passed as-is to <c>system_code_change/4</c>:</p> + case <c>[]</c>, is passed as is to <c>system_code_change/4</c>:</p> <code type="none"> -module(ch4). ... @@ -218,39 +223,43 @@ ch_app.appup: system_code_change(Chs, _Module, _OldVsn, _Extra) -> {ok, Chs}.</code> - <p>The first argument is the internal state <c>State</c> passed from - the function <c>sys:handle_system_msg(Request, From, Parent, Module, Deb, State)</c>, called by the special process when - a system message is received. In <c>ch4</c>, the internal state is - the set of available channels <c>Chs</c>.</p> - <p>The second argument is the name of the module (<c>ch4</c>).</p> - <p>The third argument is <c>Vsn</c> or <c>{down,Vsn}</c> as - described for - <seealso marker="#code_change">gen_server:code_change/3</seealso>.</p> + <list type="bulleted"> + <item>The first argument is the internal state <c>State</c>, + passed from function + <c>sys:handle_system_msg(Request, From, Parent, Module, Deb, State)</c>, + and called by the special process when a system message is received. + In <c>ch4</c>, the internal state is the set of available channels + <c>Chs</c>.</item> + <item>The second argument is the name of the module + (<c>ch4</c>).</item> + <item>The third argument is <c>Vsn</c> or <c>{down,Vsn}</c>, as + described for <c>gen_server:code_change/3</c> in + <seealso marker="#code_change">Changing Internal State</seealso>.</item> + </list> <p>In this case, all arguments but the first are ignored and the function simply returns the internal state again. This is - enough if the code only has been extended. If we had wanted to - change the internal state (similar to the example in + enough if the code only has been extended. If instead the + internal state is changed (similar to the example in <seealso marker="#int_state">Changing Internal State</seealso>), - it would have been done in this function and - <c>{ok,Chs2}</c> returned.</p> + this is done in this function and <c>{ok,Chs2}</c> returned.</p> </section> <section> <marker id="sup"></marker> <title>Changing a Supervisor</title> <p>The supervisor behaviour supports changing the internal state, - i.e. changing restart strategy and maximum restart intensity - properties, as well as changing existing child specifications.</p> - <p>Adding and deleting child processes are also possible, but not + that is, changing the restart strategy and maximum restart frequency + properties, as well as changing the existing child specifications.</p> + <p>Child processes can be added or deleted, but this is not handled automatically. Instructions must be given by in the <c>.appup</c> file.</p> <section> <title>Changing Properties</title> - <p>Since the supervisor should change its internal state, + <p>Since the supervisor is to change its internal state, synchronized code replacement is required. However, a special <c>update</c> instruction must be used.</p> - <p>The new version of the callback module must be loaded first + <p>First, the new version of the callback module must be loaded, both in the case of upgrade and downgrade. Then the new return value of <c>init/1</c> can be checked and the internal state be changed accordingly.</p> @@ -258,10 +267,11 @@ system_code_change(Chs, _Module, _OldVsn, _Extra) -> supervisors:</p> <code type="none"> {update, Module, supervisor}</code> - <p>Example: Assume we want to change the restart strategy of - <c>ch_sup</c> from the <seealso marker="sup_princ#ex">Supervisor Behaviour</seealso> chapter from one_for_one to one_for_all. - We change the callback function <c>init/1</c> in - <c>ch_sup.erl</c>:</p> + <p><em>Example:</em> To change the restart strategy of + <c>ch_sup</c> (from + <seealso marker="sup_princ#ex">Supervisor Behaviour</seealso>) + from <c>one_for_one</c> to <c>one_for_all</c>, change the callback + function <c>init/1</c> in <c>ch_sup.erl</c>:</p> <code type="none"> -module(ch_sup). ... @@ -280,7 +290,7 @@ init(_Args) -> <title>Changing Child Specifications</title> <p>The instruction, and thus the <c>.appup</c> file, when changing an existing child specification, is the same as when - changing properties as described above:</p> + changing properties as described earlier:</p> <code type="none"> {"2", [{"1", [{update, ch_sup, supervisor}]}], @@ -288,25 +298,25 @@ init(_Args) -> }.</code> <p>The changes do not affect existing child processes. For example, changing the start function only specifies how - the child process should be restarted, if needed later on.</p> - <p>Note that the id of the child specification cannot be changed.</p> - <p>Note also that changing the <c>Modules</c> field of the child - specification may affect the release handling process itself, + the child process is to be restarted, if needed later on.</p> + <p>The id of the child specification cannot be changed.</p> + <p>Changing the <c>Modules</c> field of the child + specification can affect the release handling process itself, as this field is used to identify which processes are affected when doing a synchronized code replacement.</p> </section> <marker id="sup_add"></marker> <section> - <title>Adding And Deleting Child Processes</title> - <p>As stated above, changing child specifications does not affect + <title>Adding and Deleting Child Processes</title> + <p>As stated earlier, changing child specifications does not affect existing child processes. New child specifications are - automatically added, but not deleted. Also, child processes are - not automatically started or terminated. Instead, this must be - done explicitly using <c>apply</c> instructions.</p> - <p>Example: Assume we want to add a new child process <c>m1</c> to - <c>ch_sup</c> when upgrading <c>ch_app</c> from "1" to "2". - This means <c>m1</c> should be deleted when downgrading from + automatically added, but not deleted. Child processes are + not automatically started or terminated, this must be + done using <c>apply</c> instructions.</p> + <p><em>Example:</em> Assume a new child process <c>m1</c> is to be + added to <c>ch_sup</c> when upgrading <c>ch_app</c> from "1" to "2". + This means <c>m1</c> is to be deleted when downgrading from "2" to "1":</p> <code type="none"> {"2", @@ -320,13 +330,13 @@ init(_Args) -> {update, ch_sup, supervisor} ]}] }.</code> - <p>Note that the order of the instructions is important.</p> - <p>Note also that the supervisor must be registered as + <p>The order of the instructions is important.</p> + <p>The supervisor must be registered as <c>ch_sup</c> for the script to work. If the supervisor is not registered, it cannot be accessed directly from the script. Instead a help function that finds the pid of the supervisor - and calls <c>supervisor:restart_child</c> etc. must be written, - and it is this function that should be called from the script + and calls <c>supervisor:restart_child</c>, and so on, must be + written. This function is then to be called from the script using the <c>apply</c> instruction.</p> <p>If the module <c>m1</c> is introduced in version "2" of <c>ch_app</c>, it must also be loaded when upgrading and @@ -345,18 +355,18 @@ init(_Args) -> {delete_module, m1} ]}] }.</code> - <p>Note again that the order of the instructions is important. - When upgrading, <c>m1</c> must be loaded and the supervisor's + <p>As stated earlier, the order of the instructions is important. + When upgrading, <c>m1</c> must be loaded, and the supervisor child specification changed, before the new child process can be started. When downgrading, the child process must be - terminated before child specification is changed and the module + terminated before the child specification is changed and the module is deleted.</p> </section> </section> <section> <title>Adding or Deleting a Module</title> - <p>Example: A new functional module <c>m</c> is added to + <p><em>Example:</em> A new functional module <c>m</c> is added to <c>ch_app</c>:</p> <code type="none"> {"2", @@ -367,15 +377,16 @@ init(_Args) -> <section> <title>Starting or Terminating a Process</title> <p>In a system structured according to the OTP design principles, - any process would be a child process belonging to a supervisor, - see <seealso marker="#sup_add">Adding and Deleting Child Processes</seealso> above.</p> + any process would be a child process belonging to a supervisor, see + <seealso marker="#sup_add">Adding and Deleting Child Processes</seealso> + in Changing a Supervisor.</p> </section> <section> <title>Adding or Removing an Application</title> <p>When adding or removing an application, no <c>.appup</c> file is needed. When generating <c>relup</c>, the <c>.rel</c> files - are compared and <c>add_application</c> and + are compared and the <c>add_application</c> and <c>remove_application</c> instructions are added automatically.</p> </section> @@ -383,11 +394,11 @@ init(_Args) -> <title>Restarting an Application</title> <p>Restarting an application is useful when a change is too complicated to be made without restarting the processes, for - example if the supervisor hierarchy has been restructured.</p> - <p>Example: When adding a new child <c>m1</c> to <c>ch_sup</c>, as - in the <seealso marker="#sup_add">example above</seealso>, an - alternative to updating the supervisor is to restart the entire - application:</p> + example, if the supervisor hierarchy has been restructured.</p> + <p><em>Example:</em> When adding a child <c>m1</c> to <c>ch_sup</c>, as in + <seealso marker="#sup_add">Adding and Deleting Child Processes</seealso> + in Changing a Supervisor, an alternative to updating + the supervisor is to restart the entire application:</p> <code type="none"> {"2", [{"1", [{restart_application, ch_app}]}], @@ -400,7 +411,7 @@ init(_Args) -> <title>Changing an Application Specification</title> <p>When installing a release, the application specifications are automatically updated before evaluating the <c>relup</c> script. - Hence, no instructions are needed in the <c>.appup</c> file:</p> + Thus, no instructions are needed in the <c>.appup</c> file:</p> <pre> {"2", [{"1", []}], @@ -412,28 +423,29 @@ init(_Args) -> <title>Changing Application Configuration</title> <p>Changing an application configuration by updating the <c>env</c> key in the <c>.app</c> file is an instance of changing an - application specification, <seealso marker="#app_spec">see above</seealso>.</p> + application specification, see the previous section.</p> <p>Alternatively, application configuration parameters can be added or updated in <c>sys.config</c>.</p> </section> <section> <title>Changing Included Applications</title> - <p>The release handling instructions for adding, removing and + <p>The release handling instructions for adding, removing, and restarting applications apply to primary applications only. There are no corresponding instructions for included applications. However, since an included application is really a supervision tree with a topmost supervisor, started as a child process to a supervisor in the including application, a <c>relup</c> file can be manually created.</p> - <p>Example: Assume we have a release containing an application - <c>prim_app</c> which have a supervisor <c>prim_sup</c> in its + <p><em>Example:</em> Assume there is a release containing an application + <c>prim_app</c>, which have a supervisor <c>prim_sup</c> in its supervision tree.</p> - <p>In a new version of the release, our example application - <c>ch_app</c> should be included in <c>prim_app</c>. That is, - its topmost supervisor <c>ch_sup</c> should be started as a child + <p>In a new version of the release, the application <c>ch_app</c> + is to be included in <c>prim_app</c>. That is, + its topmost supervisor <c>ch_sup</c> is to be started as a child process to <c>prim_sup</c>.</p> - <p>1) Edit the code for <c>prim_sup</c>:</p> + <p>The workflow is as follows:</p> + <p><em>Step 1)</em> Edit the code for <c>prim_sup</c>:</p> <code type="none"> init(...) -> {ok, {...supervisor flags..., @@ -441,7 +453,7 @@ init(...) -> {ch_sup, {ch_sup,start_link,[]}, permanent,infinity,supervisor,[ch_sup]}, ...]}}.</code> - <p>2) Edit the <c>.app</c> file for <c>prim_app</c>:</p> + <p><em>Step 2)</em> Edit the <c>.app</c> file for <c>prim_app</c>:</p> <code type="none"> {application, prim_app, [..., @@ -450,27 +462,29 @@ init(...) -> {included_applications, [ch_app]}, ... ]}.</code> - <p>3) Create a new <c>.rel</c> file, including <c>ch_app</c>:</p> + <p><em>Step 3)</em> Create a new <c>.rel</c> file, including + <c>ch_app</c>:</p> <code type="none"> {release, ..., [..., {prim_app, "2"}, {ch_app, "1"}]}.</code> + <p>The included application can be started in two ways. + This is described in the next two sections.</p> <section> <title>Application Restart</title> - <p>4a) One way to start the included application is to restart - the entire <c>prim_app</c> application. Normally, we would then - use the <c>restart_application</c> instruction in - the <c>.appup</c> file for <c>prim_app</c>.</p> - <p>However, if we did this and then generated a <c>relup</c> file, - not only would it contain instructions for restarting (i.e. + <p><em>Step 4a)</em> One way to start the included application is to + restart the entire <c>prim_app</c> application. Normally, the + <c>restart_application</c> instruction in the <c>.appup</c> file + for <c>prim_app</c> would be used.</p> + <p>However, if this is done and a <c>relup</c> file is generated, + not only would it contain instructions for restarting (that is, removing and adding) <c>prim_app</c>, it would also contain instructions for starting <c>ch_app</c> (and stopping it, in - the case of downgrade). This is due to the fact that - <c>ch_app</c> is included in the new <c>.rel</c> file, but not - in the old one.</p> + the case of downgrade). This is because <c>ch_app</c> is included + in the new <c>.rel</c> file, but not in the old one.</p> <p>Instead, a correct <c>relup</c> file can be created manually, either from scratch or by editing the generated version. The instructions for starting/stopping <c>ch_app</c> are @@ -512,7 +526,8 @@ init(...) -> <section> <title>Supervisor Change</title> - <p>4b) Another way to start the included application (or stop it + <p><em>Step 4b)</em> Another way to start the included + application (or stop it in the case of downgrade) is by combining instructions for adding and removing child processes to/from <c>prim_sup</c> with instructions for loading/unloading all <c>ch_app</c> code and @@ -521,7 +536,7 @@ init(...) -> scratch or by editing a generated version. Load all code for <c>ch_app</c> first, and also load the application specification, before <c>prim_sup</c> is updated. When - downgrading, <c>prim_sup</c> should be updated first, before + downgrading, <c>prim_sup</c> is to updated first, before the code for <c>ch_app</c> and its application specification are unloaded.</p> <code type="none"> @@ -560,10 +575,10 @@ init(...) -> <section> <title>Changing Non-Erlang Code</title> <p>Changing code for a program written in another programming - language than Erlang, for example a port program, is very - application dependent and OTP provides no special support for it.</p> - <p>Example, changing code for a port program: Assume that - the Erlang process controlling the port is a gen_server + language than Erlang, for example, a port program, is + application-dependent and OTP provides no special support for it.</p> + <p><em>Example:</em> When changing code for a port program, assume that + the Erlang process controlling the port is a <c>gen_server</c> <c>portc</c> and that the port is opened in the callback function <c>init/1</c>:</p> <code type="none"> @@ -573,10 +588,11 @@ init(...) -> Port = open_port({spawn,PortPrg}, [...]), ..., {ok, #state{port=Port, ...}}.</code> - <p>If the port program should be updated, we can extend the code for - the gen_server with a <c>code_change</c> function which closes - the old port and opens a new port. (If necessary, the gen_server - may first request data that needs to be saved from the port + <p>If the port program is to be updated, the code for the + <c>gen_server</c> can be extended with a <c>code_change</c> function, + which closes the old port and opens a new port. + (If necessary, the <c>gen_server</c> can + first request data that must be saved from the port program and pass this data to the new port):</p> <code type="none"> code_change(_OldVsn, State, port) -> @@ -595,8 +611,8 @@ code_change(_OldVsn, State, port) -> [{"1", [{update, portc, {advanced,port}}]}], [{"1", [{update, portc, {advanced,port}}]}] ].</code> - <p>Make sure the <c>priv</c> directory where the C program is - located is included in the new release package:</p> + <p>Ensure that the <c>priv</c> directory, where the C program is + located, is included in the new release package:</p> <pre> 1> <input>systools:make_tar("my_release", [{dirs,[priv]}]).</input> ...</pre> @@ -604,28 +620,29 @@ code_change(_OldVsn, State, port) -> <section> <title>Emulator Restart and Upgrade</title> - <p>There are two upgrade instructions that will restart the emulator:</p> - <taglist> - <tag><c>restart_new_emulator</c></tag> - <item>Intended for when erts, kernel, stdlib or sasl is - upgraded. It is automatically added when the relup file is - generated by <c>systools:make_relup/3,4</c>. It is executed - before all other upgrade instructions. See - <seealso marker="release_handling#restart_new_emulator_instr">Release - Handling</seealso> for more information about this - instruction.</item> - <tag><c>restart_emulator</c></tag> - <item>Used when a restart of the emulator is required after all - other upgrade instructions are executed. See - <seealso marker="release_handling#restart_emulator_instr">Release - Handling</seealso> for more information about this - instruction.</item> - </taglist> - + <p>Two upgrade instructions restart the emulator:</p> + <list type="bulleted"> + <item><p><c>restart_new_emulator</c></p> + <p>Intended when ERTS, Kernel, STDLIB, or + SASL is upgraded. It is automatically added when the + <c>relup</c> file is generated by <c>systools:make_relup/3,4</c>. + It is executed before all other upgrade instructions. + For more information about this instruction, see + restart_new_emulator (Low-Level) in + <seealso marker="release_handling#restart_new_emulator_instr">Release Handling Instructions</seealso>. + </p></item> + <item><p><c>restart_emulator</c></p> + <p>Used when a restart of the emulator is required after all + other upgrade instructions are executed. + For more information about this instruction, see + restart_emulator (Low-Level) in + <seealso marker="release_handling#restart_emulator_instr">Release Handling Instructions</seealso>. + </p></item> + </list> <p>If an emulator restart is necessary and no upgrade instructions - are needed, i.e. if the restart itself is enough for the - upgraded applications to start running the new versions, a very - simple <c>.relup</c> file can be created manually:</p> + are needed, that is, if the restart itself is enough for the + upgraded applications to start running the new versions, a + simple <c>relup</c> file can be created manually:</p> <code type="none"> {"B", [{"A", @@ -637,26 +654,27 @@ code_change(_OldVsn, State, port) -> }.</code> <p>In this case, the release handler framework with automatic packing and unpacking of release packages, automatic path - updates etc. can be used without having to specify <c>.appup</c> - files.</p> + updates, and so on, can be used without having to specify + <c>.appup</c> files.</p> </section> <section> - <title>Emulator Upgrade from pre OTP R15</title> + <title>Emulator Upgrade From Pre OTP R15</title> <p>From OTP R15, an emulator upgrade is performed by restarting the emulator with new versions of the core applications - (<c>kernel</c>, <c>stdlib</c> and <c>sasl</c>) before loading code + (Kernel, STDLIB, and SASL) before loading code and running upgrade instruction for other applications. For this - to work, the release to upgrade from must includes OTP R15 or - later. For the case where the release to upgrade from includes an - earlier emulator version, <c>systools:make_relup</c> will create a + to work, the release to upgrade from must include OTP R15 or + later.</p> + <p>For the case where the release to upgrade from includes an + earlier emulator version, <c>systools:make_relup</c> creates a backwards compatible relup file. This means that all upgrade - instructions will be executed before the emulator is - restarted. The new application code will therefore be loaded into + instructions are executed before the emulator is + restarted. The new application code is therefore loaded into the old emulator. If the new code is compiled with the new - emulator, there might be cases where the beam format has changed - and beam files can not be loaded. To overcome this problem, the - new code should be compiled with the old emulator.</p> + emulator, there can be cases where the beam format has changed + and beam files cannot be loaded. To overcome this problem, compile + the new code with the old emulator.</p> </section> </chapter> diff --git a/system/doc/design_principles/des_princ.xml b/system/doc/design_principles/des_princ.xml index e8f289b905..77c61eafb0 100644 --- a/system/doc/design_principles/des_princ.xml +++ b/system/doc/design_principles/des_princ.xml @@ -28,50 +28,52 @@ <rev></rev> <file>des_princ.xml</file> </header> - <p>The <em>OTP Design Principles</em> is a set of principles for how - to structure Erlang code in terms of processes, modules and - directories.</p> + <marker id="otp design principles"></marker> + <p>The <em>OTP design principles</em> define how to + structure Erlang code in terms of processes, modules, + and directories.</p> <section> <title>Supervision Trees</title> <p>A basic concept in Erlang/OTP is the <em>supervision tree</em>. This is a process structuring model based on the idea of - <em>workers</em> and <em>supervisors</em>.</p> + <em>workers</em> and <em>supervisors</em>:</p> <list type="bulleted"> - <item>Workers are processes which perform computations, that is, + <item>Workers are processes that perform computations, that is, they do the actual work.</item> - <item>Supervisors are processes which monitor the behaviour of + <item>Supervisors are processes that monitor the behaviour of workers. A supervisor can restart a worker if something goes wrong.</item> <item>The supervision tree is a hierarchical arrangement of - code into supervisors and workers, making it possible to + code into supervisors and workers, which makes it possible to design and program fault-tolerant software.</item> </list> + <p>In the following figure, square boxes represents supervisors and + circles represent workers:</p> <marker id="sup6"></marker> <image file="../design_principles/sup6.gif"> <icaption>Supervision Tree</icaption> </image> - <p>In the figure above, square boxes represents supervisors and - circles represent workers.</p> </section> <section> <title>Behaviours</title> <p>In a supervision tree, many of the processes have similar structures, they follow similar patterns. For example, - the supervisors are very similar in structure. The only difference - between them is which child processes they supervise. Also, many - of the workers are servers in a server-client relation, finite - state machines, or event handlers such as error loggers.</p> + the supervisors are similar in structure. The only difference + between them is which child processes they supervise. Many + of the workers are servers in a server-client relation, + finite-state machines, or event handlers such as error loggers.</p> <p><em>Behaviours</em> are formalizations of these common patterns. The idea is to divide the code for a process in a generic part - (a behaviour module) and a specific part (a <em>callback module</em>).</p> + (a behaviour module) and a specific part (a + <em>callback module</em>).</p> <p>The behaviour module is part of Erlang/OTP. To implement a process such as a supervisor, the user only has to implement - the callback module which should export a pre-defined set of + the callback module which is to export a pre-defined set of functions, the <em>callback functions</em>.</p> - <p>An example to illustrate how code can be divided into a generic - and a specific part: Consider the following code (written in + <p>The following example illustrate how code can be divided into a + generic and a specific part. Consider the following code (written in plain Erlang) for a simple server, which keeps track of a number of "channels". Other processes can allocate and free the channels by calling the functions <c>alloc/0</c> and <c>free/1</c>, @@ -149,7 +151,7 @@ loop(Mod, State) -> State2 = Mod:handle_cast(Req, State), loop(Mod, State2) end.</code> - <p>and a callback module <c>ch2.erl</c>:</p> + <p>And a callback module <c>ch2.erl</c>:</p> <code type="none"> -module(ch2). -export([start/0]). @@ -173,27 +175,27 @@ handle_call(alloc, Chs) -> handle_cast({free, Ch}, Chs) -> free(Ch, Chs). % => Chs2</code> - <p>Note the following:</p> + <p>Notice the following:</p> <list type="bulleted"> - <item>The code in <c>server</c> can be re-used to build many + <item>The code in <c>server</c> can be reused to build many different servers.</item> - <item>The name of the server, in this example the atom - <c>ch2</c>, is hidden from the users of the client functions. - This means the name can be changed without affecting them.</item> + <item>The server name, in this example the atom + <c>ch2</c>, is hidden from the users of the client functions. This + means that the name can be changed without affecting them.</item> <item>The protcol (messages sent to and received from the server) - is hidden as well. This is good programming practice and allows - us to change the protocol without making changes to code using + is also hidden. This is good programming practice and allows + one to change the protocol without changing the code using the interface functions.</item> - <item>We can extend the functionality of <c>server</c>, without + <item>The functionality of <c>server</c> can be extended without having to change <c>ch2</c> or any other callback module.</item> </list> - <p>(In <c>ch1.erl</c> and <c>ch2.erl</c> above, the implementation - of <c>channels/0</c>, <c>alloc/1</c> and <c>free/2</c> has been + <p>In <c>ch1.erl</c> and <c>ch2.erl</c> above, the implementation + of <c>channels/0</c>, <c>alloc/1</c>, and <c>free/2</c> has been intentionally left out, as it is not relevant to the example. For completeness, one way to write these functions are given - below. Note that this is an example only, a realistic + below. This is an example only, a realistic implementation must be able to handle situations like running out - of channels to allocate etc.)</p> + of channels to allocate, and so on.</p> <code type="none"> channels() -> {_Allocated = [], _Free = lists:seq(1,100)}. @@ -208,30 +210,30 @@ free(Ch, {Alloc, Free} = Channels) -> false -> Channels end. </code> - <p>Code written without making use of behaviours may be more - efficient, but the increased efficiency will be at the expense of + <p>Code written without using behaviours can be more + efficient, but the increased efficiency is at the expense of generality. The ability to manage all applications in the system - in a consistent manner is very important.</p> + in a consistent manner is important.</p> <p>Using behaviours also makes it easier to read and understand - code written by other programmers. Ad hoc programming structures, + code written by other programmers. Improvised programming structures, while possibly more efficient, are always more difficult to understand.</p> - <p>The module <c>server</c> corresponds, greatly simplified, + <p>The <c>server</c> module corresponds, greatly simplified, to the Erlang/OTP behaviour <c>gen_server</c>.</p> <p>The standard Erlang/OTP behaviours are:</p> - <taglist> - <tag><seealso marker="gen_server_concepts">gen_server</seealso></tag> - <item>For implementing the server of a client-server relation.</item> - <tag><seealso marker="fsm">gen_fsm</seealso></tag> - <item>For implementing finite state machines.</item> - <tag><seealso marker="events">gen_event</seealso></tag> - <item>For implementing event handling functionality.</item> - <tag><seealso marker="sup_princ">supervisor</seealso></tag> - <item>For implementing a supervisor in a supervision tree.</item> - </taglist> + <list type="bulleted"> + <item><p><seealso marker="gen_server_concepts">gen_server</seealso></p> + <p>For implementing the server of a client-server relation</p></item> + <item><p><seealso marker="fsm">gen_fsm</seealso></p> + <p>For implementing finite-state machines</p></item> + <item><p><seealso marker="events">gen_event</seealso></p> + <p>For implementing event handling functionality</p></item> + <item><p><seealso marker="sup_princ">supervisor</seealso></p> + <p>For implementing a supervisor in a supervision tree</p></item> + </list> <p>The compiler understands the module attribute <c>-behaviour(Behaviour)</c> and issues warnings about - missing callback functions. Example:</p> + missing callback functions, for example:</p> <code type="none"> -module(chs3). -behaviour(gen_server). @@ -248,13 +250,17 @@ free(Ch, {Alloc, Free} = Channels) -> some specific functionality. Components are with Erlang/OTP terminology called <em>applications</em>. Examples of Erlang/OTP applications are Mnesia, which has everything needed for - programming database services, and Debugger which is used to - debug Erlang programs. The minimal system based on Erlang/OTP - consists of the applications Kernel and STDLIB.</p> + programming database services, and Debugger, which is used + to debug Erlang programs. The minimal system based on Erlang/OTP + consists of the following two applications:</p> + <list type="bulleted"> + <item>Kernel - Functionality necessary to run Erlang</item> + <item>STDLIB - Erlang standard libraries</item> + </list> <p>The application concept applies both to program structure (processes) and directory structure (modules).</p> - <p>The simplest kind of application does not have any processes, - but consists of a collection of functional modules. Such an + <p>The simplest applications do not have any processes, + but consist of a collection of functional modules. Such an application is called a <em>library application</em>. An example of a library application is STDLIB.</p> <p>An application with processes is easiest implemented as a @@ -266,12 +272,11 @@ free(Ch, {Alloc, Free} = Channels) -> <section> <title>Releases</title> <p>A <em>release</em> is a complete system made out from a subset of - the Erlang/OTP applications and a set of user-specific - applications.</p> + Erlang/OTP applications and a set of user-specific applications.</p> <p>How to program releases is described in <seealso marker="release_structure">Releases</seealso>.</p> <p>How to install a release in a target environment is described - in the chapter about Target Systems in System Principles.</p> + in the section about target systems in Section 2 System Principles.</p> </section> <section> diff --git a/system/doc/design_principles/distributed_applications.xml b/system/doc/design_principles/distributed_applications.xml index 4d4ba3136e..f40a24fdf5 100644 --- a/system/doc/design_principles/distributed_applications.xml +++ b/system/doc/design_principles/distributed_applications.xml @@ -28,71 +28,73 @@ <rev></rev> <file>distributed_applications.xml</file> </header> + <marker id="distributed appl"></marker> <section> - <title>Definition</title> - <p>In a distributed system with several Erlang nodes, there may be - a need to control applications in a distributed manner. If + <title>Introduction</title> + <p>In a distributed system with several Erlang nodes, it can be + necessary to control applications in a distributed manner. If the node, where a certain application is running, goes down, - the application should be restarted at another node.</p> + the application is to be restarted at another node.</p> <p>Such an application is called a <em>distributed application</em>. - Note that it is the control of the application which is - distributed, all applications can of course be distributed in - the sense that they, for example, use services on other nodes.</p> - <p>Because a distributed application may move between nodes, some + Notice that it is the control of the application that is distributed. + All applications can be distributed in the sense that they, + for example, use services on other nodes.</p> + <p>Since a distributed application can move between nodes, some addressing mechanism is required to ensure that it can be addressed by other applications, regardless on which node it currently executes. This issue is not addressed here, but the - Kernel modules <c>global</c> or <c>pg2</c> can be - used for this purpose.</p> + <c>global</c> or <c>pg2</c> modules in Kernel + can be used for this purpose.</p> </section> <section> <title>Specifying Distributed Applications</title> <p>Distributed applications are controlled by both the application - controller and a distributed application controller process, - <c>dist_ac</c>. Both these processes are part of the <c>kernel</c> - application. Therefore, distributed applications are specified by - configuring the <c>kernel</c> application, using the following - configuration parameter (see also <c>kernel(6)</c>):</p> - <taglist> - <tag><c>distributed = [{Application, [Timeout,] NodeDesc}]</c></tag> - <item> - <p>Specifies where the application <c>Application = atom()</c> - may execute. <c>NodeDesc = [Node | {Node,...,Node}]</c> is - a list of node names in priority order. The order between - nodes in a tuple is undefined.</p> - <p><c>Timeout = integer()</c> specifies how many milliseconds to - wait before restarting the application at another node. - Defaults to 0.</p> - </item> - </taglist> + controller and a distributed application controller process, + <c>dist_ac</c>. Both these processes are part of the Kernel + application. Distributed applications are thus specified by + configuring the Kernel application, using the following + configuration parameter (see also <c>kernel(6)</c>):</p> + <p><c>distributed = [{Application, [Timeout,] NodeDesc}]</c></p> + <list type="bulleted"> + <item>Specifies where the application <c>Application = atom()</c> + can execute.</item> + <item>><c>NodeDesc = [Node | {Node,...,Node}]</c> is a list of + node names in priority order. The order between nodes in a tuple + is undefined.</item> + <item><c>Timeout = integer()</c> specifies how many milliseconds + to wait before restarting the application at another node. It + defaults to 0.</item> + </list> <p>For distribution of application control to work properly, - the nodes where a distributed application may run must contact + the nodes where a distributed application can run must contact each other and negotiate where to start the application. This is - done using the following <c>kernel</c> configuration parameters:</p> - <taglist> - <tag><c>sync_nodes_mandatory = [Node]</c></tag> - <item>Specifies which other nodes must be started (within - the timeout specified by <c>sync_nodes_timeout</c>.</item> - <tag><c>sync_nodes_optional = [Node]</c></tag> - <item>Specifies which other nodes can be started (within - the timeout specified by <c>sync_nodes_timeout</c>.</item> - <tag><c>sync_nodes_timeout = integer() | infinity</c></tag> - <item>Specifies how many milliseconds to wait for the other nodes - to start.</item> - </taglist> - <p>When started, the node will wait for all nodes specified by + done using the following configuration parameters in + Kernel:</p> + <list type="bulleted"> + <item><c>sync_nodes_mandatory = [Node]</c> - Specifies which + other nodes must be started (within the time-out specified by + <c>sync_nodes_timeout</c>).</item> + <item><c>sync_nodes_optional = [Node]</c> - Specifies which + other nodes can be started (within the time-out specified by + <c>sync_nodes_timeout</c>).</item> + <item><c>sync_nodes_timeout = integer() | infinity</c> - + Specifies how many milliseconds to wait for the other nodes to + start.</item> + </list> + <p>When started, the node waits for all nodes specified by <c>sync_nodes_mandatory</c> and <c>sync_nodes_optional</c> to - come up. When all nodes have come up, or when all mandatory nodes - have come up and the time specified by <c>sync_nodes_timeout</c> - has elapsed, all applications will be started. If not all - mandatory nodes have come up, the node will terminate.</p> - <p>Example: An application <c>myapp</c> should run at the node - <c>cp1@cave</c>. If this node goes down, <c>myapp</c> should + come up. When all nodes are up, or when all mandatory nodes + are up and the time specified by <c>sync_nodes_timeout</c> + has elapsed, all applications start. If not all + mandatory nodes are up, the node terminates.</p> + <p><em>Example:</em></p> + <p>An application <c>myapp</c> is to run at the node + <c>cp1@cave</c>. If this node goes down, <c>myapp</c> is to be restarted at <c>cp2@cave</c> or <c>cp3@cave</c>. A system - configuration file <c>cp1.config</c> for <c>cp1@cave</c> could - look like:</p> + configuration file <c>cp1.config</c> for <c>cp1@cave</c> can + look as follows:</p> <code type="none"> [{kernel, [{distributed, [{myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}]}, @@ -103,13 +105,13 @@ ].</code> <p>The system configuration files for <c>cp2@cave</c> and <c>cp3@cave</c> are identical, except for the list of mandatory - nodes which should be <c>[cp1@cave, cp3@cave]</c> for + nodes, which is to be <c>[cp1@cave, cp3@cave]</c> for <c>cp2@cave</c> and <c>[cp1@cave, cp2@cave]</c> for <c>cp3@cave</c>.</p> <note> <p>All involved nodes must have the same value for - <c>distributed</c> and <c>sync_nodes_timeout</c>, or - the behaviour of the system is undefined.</p> + <c>distributed</c> and <c>sync_nodes_timeout</c>. + Otherwise the system behaviour is undefined.</p> </note> </section> @@ -117,28 +119,29 @@ <title>Starting and Stopping Distributed Applications</title> <p>When all involved (mandatory) nodes have been started, the distributed application can be started by calling - <c>application:start(Application)</c> at <em>all of these nodes.</em></p> - <p>It is of course also possible to use a boot script (see - <seealso marker="release_structure">Releases</seealso>) which - automatically starts the application.</p> - <p>The application will be started at the first node, specified - by the <c>distributed</c> configuration parameter, which is up - and running. The application is started as usual. That is, an - application master is created and calls the application callback - function:</p> + <c>application:start(Application)</c> at <em>all of these + nodes.</em></p> + <p>A boot script (see + <seealso marker="release_structure">Releases</seealso>) + can be used that automatically starts the application.</p> + <p>The application is started at the first operational node that + is listed in the list of nodes in the <c>distributed</c> + configuration parameter. The application is started as usual. + That is, an application master is created and calls the + application callback function:</p> <code type="none"> Module:start(normal, StartArgs)</code> - <p>Example: Continuing the example from the previous section, - the three nodes are started, specifying the system configuration - file:</p> + <p>Example:</p> + <p>Continuing the example from the previous section, the three nodes + are started, specifying the system configuration file:</p> <pre> > <input>erl -sname cp1 -config cp1</input> > <input>erl -sname cp2 -config cp2</input> > <input>erl -sname cp3 -config cp3</input></pre> - <p>When all nodes are up and running, <c>myapp</c> can be started. + <p>When all nodes are operational, <c>myapp</c> can be started. This is achieved by calling <c>application:start(myapp)</c> at all three nodes. It is then started at <c>cp1</c>, as shown in - the figure below.</p> + the following figure:</p> <marker id="dist1"></marker> <image file="../design_principles/dist1.gif"> <icaption>Application myapp - Situation 1</icaption> @@ -150,31 +153,33 @@ Module:start(normal, StartArgs)</code> <section> <title>Failover</title> <p>If the node where the application is running goes down, - the application is restarted (after the specified timeout) at - the first node, specified by the <c>distributed</c> configuration - parameter, which is up and running. This is called a + the application is restarted (after the specified time-out) at + the first operational node that is listed in the list of nodes + in the <c>distributed</c> configuration parameter. This is called a <em>failover</em>.</p> <p>The application is started the normal way at the new node, that is, by the application master calling:</p> <code type="none"> Module:start(normal, StartArgs)</code> - <p>Exception: If the application has the <c>start_phases</c> key - defined (see <seealso marker="included_applications">Included Applications</seealso>), then the application is instead started - by calling:</p> + <p>An exception is if the application has the <c>start_phases</c> + key defined + (see <seealso marker="included_applications">Included Applications</seealso>). + The application is then instead started by calling:</p> <code type="none"> Module:start({failover, Node}, StartArgs)</code> - <p>where <c>Node</c> is the terminated node.</p> - <p>Example: If <c>cp1</c> goes down, the system checks which one of + <p>Here <c>Node</c> is the terminated node.</p> + <p><em>Example:</em></p> + <p> If <c>cp1</c> goes down, the system checks which one of the other nodes, <c>cp2</c> or <c>cp3</c>, has the least number of running applications, but waits for 5 seconds for <c>cp1</c> to restart. If <c>cp1</c> does not restart and <c>cp2</c> runs fewer - applications than <c>cp3,</c> then <c>myapp</c> is restarted on + applications than <c>cp3</c>, <c>myapp</c> is restarted on <c>cp2</c>.</p> <marker id="dist2"></marker> <image file="../design_principles/dist2.gif"> <icaption>Application myapp - Situation 2</icaption> </image> - <p>Suppose now that <c>cp2</c> goes down as well and does not + <p>Suppose now that <c>cp2</c> goes also down and does not restart within 5 seconds. <c>myapp</c> is now restarted on <c>cp3</c>.</p> <marker id="dist3"></marker> @@ -186,28 +191,29 @@ Module:start({failover, Node}, StartArgs)</code> <section> <title>Takeover</title> <p>If a node is started, which has higher priority according - to <c>distributed</c>, than the node where a distributed - application is currently running, the application will be - restarted at the new node and stopped at the old node. This is + to <c>distributed</c> than the node where a distributed + application is running, the application is restarted at the + new node and stopped at the old node. This is called a <em>takeover</em>.</p> <p>The application is started by the application master calling:</p> <code type="none"> Module:start({takeover, Node}, StartArgs)</code> - <p>where <c>Node</c> is the old node.</p> - <p>Example: If <c>myapp</c> is running at <c>cp3</c>, and if - <c>cp2</c> now restarts, it will not restart <c>myapp</c>, - because the order between nodes <c>cp2</c> and <c>cp3</c> is + <p>Here <c>Node</c> is the old node.</p> + <p><em>Example: </em></p> + <p>If <c>myapp</c> is running at <c>cp3</c>, and if + <c>cp2</c> now restarts, it does not restart <c>myapp</c>, + as the order between the <c>cp2</c> and <c>cp3</c> nodes is undefined.</p> <marker id="dist4"></marker> <image file="../design_principles/dist4.gif"> <icaption>Application myapp - Situation 4</icaption> </image> - <p>However, if <c>cp1</c> restarts as well, the function + <p>However, if <c>cp1</c> also restarts, the function <c>application:takeover/2</c> moves <c>myapp</c> to <c>cp1</c>, - because <c>cp1</c> has a higher priority than <c>cp3</c> for this - application. In this case, - <c>Module:start({takeover, cp3@cave}, StartArgs)</c> is executed - at <c>cp1</c> to start the application.</p> + as <c>cp1</c> has a higher priority than <c>cp3</c> for this + application. In this case, + <c>Module:start({takeover, cp3@cave}, StartArgs)</c> is + executed at <c>cp1</c> to start the application.</p> <marker id="dist5"></marker> <image file="../design_principles/dist5.gif"> <icaption>Application myapp - Situation 5</icaption> diff --git a/system/doc/design_principles/events.xml b/system/doc/design_principles/events.xml index 529e12c216..6e5afb939e 100644 --- a/system/doc/design_principles/events.xml +++ b/system/doc/design_principles/events.xml @@ -21,7 +21,7 @@ </legalnotice> - <title>Gen_Event Behaviour</title> + <title>gen_event Behaviour</title> <prepared></prepared> <docno></docno> <date></date> @@ -29,35 +29,36 @@ <file>events.xml</file> </header> <marker id="gen_event"></marker> - <p>This chapter should be read in conjunction with - <c>gen_event(3)</c>, where all interface functions and callback - functions are described in detail.</p> + <p>This section is to be read with the <c>gen_event(3)</c> manual + page in STDLIB, where all interface functions and callback + functions are described in detail.</p> <section> <title>Event Handling Principles</title> <p>In OTP, an <em>event manager</em> is a named object to which - events can be sent. An <em>event</em> could be, for example, - an error, an alarm or some information that should be logged.</p> - <p>In the event manager, zero, one or several <em>event handlers</em> are installed. When the event manager is notified - about an event, the event will be processed by all the installed + events can be sent. An <em>event</em> can be, for example, + an error, an alarm, or some information that is to be logged.</p> + <p>In the event manager, zero, one, or many <em>event handlers</em> + are installed. When the event manager is notified + about an event, the event is processed by all the installed event handlers. For example, an event manager for handling errors - can by default have a handler installed which writes error + can by default have a handler installed, which writes error messages to the terminal. If the error messages during a certain - period should be saved to a file as well, the user adds another - event handler which does this. When logging to file is no longer - necessary, this event handler is deleted.</p> + period is to be saved to a file as well, the user adds another + event handler that does this. When logging to the file is no + longer necessary, this event handler is deleted.</p> <p>An event manager is implemented as a process and each event handler is implemented as a callback module.</p> <p>The event manager essentially maintains a list of <c>{Module, State}</c> pairs, where each <c>Module</c> is an - event handler, and <c>State</c> the internal state of that event - handler.</p> + event handler, and <c>State</c> is the internal state of that + event handler.</p> </section> <section> <title>Example</title> <p>The callback module for the event handler writing error messages - to the terminal could look like:</p> + to the terminal can look as follows:</p> <code type="none"> -module(terminal_logger). -behaviour(gen_event). @@ -74,7 +75,7 @@ handle_event(ErrorMsg, State) -> terminate(_Args, _State) -> ok.</code> <p>The callback module for the event handler writing error messages - to a file could look like:</p> + to a file can look as follows:</p> <code type="none"> -module(file_logger). -behaviour(gen_event). @@ -98,29 +99,28 @@ terminate(_Args, Fd) -> <marker id="mgr"></marker> <title>Starting an Event Manager</title> <p>To start an event manager for handling errors, as described in - the example above, call the following function:</p> + the previous example, call the following function:</p> <code type="none"> gen_event:start_link({local, error_man})</code> <p>This function spawns and links to a new process, an event manager.</p> - <p>The argument, <c>{local, error_man}</c> specifies the name. In - this case, the event manager will be locally registered as - <c>error_man</c>.</p> + <p>The argument, <c>{local, error_man}</c> specifies the name. The + event manager is then locally registered as <c>error_man</c>.</p> <p>If the name is omitted, the event manager is not registered. - Instead its pid must be used. The name could also be given + Instead its pid must be used. The name can also be given as <c>{global, Name}</c>, in which case the event manager is registered using <c>global:register_name/2</c>.</p> <p><c>gen_event:start_link</c> must be used if the event manager is - part of a supervision tree, i.e. is started by a supervisor. - There is another function <c>gen_event:start</c> to start a - stand-alone event manager, i.e. an event manager which is not + part of a supervision tree, that is, started by a supervisor. + There is another function, <c>gen_event:start</c>, to start a + standalone event manager, that is, an event manager that is not part of a supervision tree.</p> </section> <section> <title>Adding an Event Handler</title> - <p>Here is an example using the shell on how to start an event - manager and add an event handler to it:</p> + <p>The following example shows how to start an event manager and + add an event handler to it by using the shell:</p> <pre> 1> <input>gen_event:start({local, error_man}).</input> {ok,<0.31.0>} @@ -128,16 +128,16 @@ gen_event:start_link({local, error_man})</code> ok</pre> <p>This function sends a message to the event manager registered as <c>error_man</c>, telling it to add the event handler - <c>terminal_logger</c>. The event manager will call the callback - function <c>terminal_logger:init([])</c>, where the argument [] - is the third argument to <c>add_handler</c>. <c>init</c> is - expected to return <c>{ok, State}</c>, where <c>State</c> is + <c>terminal_logger</c>. The event manager calls the callback + function <c>terminal_logger:init([])</c>, where the argument + <c>[]</c> is the third argument to <c>add_handler</c>. <c>init</c> + is expected to return <c>{ok, State}</c>, where <c>State</c> is the internal state of the event handler.</p> <code type="none"> init(_Args) -> {ok, []}.</code> <p>Here, <c>init</c> does not need any input data and ignores its - argument. Also, for <c>terminal_logger</c> the internal state is + argument. For <c>terminal_logger</c>, the internal state is not used. For <c>file_logger</c>, the internal state is used to save the open file descriptor.</p> <code type="none"> @@ -147,7 +147,7 @@ init(File) -> </section> <section> - <title>Notifying About Events</title> + <title>Notifying about Events</title> <pre> 3> <input>gen_event:notify(error_man, no_reply).</input> ***Error*** no_reply @@ -158,7 +158,7 @@ ok</pre> When the event is received, the event manager calls <c>handle_event(Event, State)</c> for each installed event handler, in the same order as they were added. The function is - expected to return a tuple <c>{ok, State1}</c>, where + expected to return a tuple <c>{ok,State1}</c>, where <c>State1</c> is a new value for the state of the event handler.</p> <p>In <c>terminal_logger</c>:</p> <code type="none"> @@ -179,17 +179,17 @@ handle_event(ErrorMsg, Fd) -> ok</pre> <p>This function sends a message to the event manager registered as <c>error_man</c>, telling it to delete the event handler - <c>terminal_logger</c>. The event manager will call the callback + <c>terminal_logger</c>. The event manager calls the callback function <c>terminal_logger:terminate([], State)</c>, where - the argument [] is the third argument to <c>delete_handler</c>. - <c>terminate</c> should be the opposite of <c>init</c> and do any + the argument <c>[]</c> is the third argument to <c>delete_handler</c>. + <c>terminate</c> is to be the opposite of <c>init</c> and do any necessary cleaning up. Its return value is ignored.</p> <p>For <c>terminal_logger</c>, no cleaning up is necessary:</p> <code type="none"> terminate(_Args, _State) -> ok.</code> <p>For <c>file_logger</c>, the file descriptor opened in <c>init</c> - needs to be closed:</p> + must be closed:</p> <code type="none"> terminate(_Args, Fd) -> file:close(Fd).</code> @@ -197,20 +197,22 @@ terminate(_Args, Fd) -> <section> <title>Stopping</title> - <p>When an event manager is stopped, it will give each of + <p>When an event manager is stopped, it gives each of the installed event handlers the chance to clean up by calling <c>terminate/2</c>, the same way as when deleting a handler.</p> <section> <title>In a Supervision Tree</title> <p>If the event manager is part of a supervision tree, no stop - function is needed. The event manager will automatically be + function is needed. The event manager is automatically terminated by its supervisor. Exactly how this is done is - defined by a <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> set in the supervisor.</p> + defined by a + <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> + set in the supervisor.</p> </section> <section> - <title>Stand-Alone Event Managers</title> + <title>Standalone Event Managers</title> <p>An event manager can also be stopped by calling:</p> <pre> > <input>gen_event:stop(error_man).</input> @@ -219,16 +221,17 @@ ok</pre> </section> <section> <title>Handling Other Messages</title> - <p>If the gen_event should be able to receive other messages than - events, the callback function <c>handle_info(Info, StateName, StateData)</c> - must be implemented to handle them. Examples of - other messages are exit messages, if the gen_event is linked to + <p>If the <c>gen_event</c> is to be able to receive other messages + than events, the callback function + <c>handle_info(Info, StateName, StateData)</c> + must be implemented to handle them. Examples of other + messages are exit messages, if the <c>gen_event</c> is linked to other processes (than the supervisor) and trapping exit signals.</p> <code type="none"> handle_info({'EXIT', Pid, Reason}, State) -> ..code to handle exits here.. {ok, NewState}.</code> - <p>The code_change method also has to be implemented.</p> + <p>The <c>code_change</c> method must also be implemented.</p> <code type="none"> code_change(OldVsn, State, Extra) -> ..code to convert state (and more) during code change diff --git a/system/doc/design_principles/fsm.xml b/system/doc/design_principles/fsm.xml index 9dce159dca..ef961f5fad 100644 --- a/system/doc/design_principles/fsm.xml +++ b/system/doc/design_principles/fsm.xml @@ -21,32 +21,33 @@ </legalnotice> - <title>Gen_Fsm Behaviour</title> + <title>gen_fsm Behaviour</title> <prepared></prepared> <docno></docno> <date></date> <rev></rev> <file>fsm.xml</file> </header> - <p>This chapter should be read in conjunction with <c>gen_fsm(3)</c>, - where all interface functions and callback functions are described - in detail.</p> + <marker id="gen_fsm behaviour"></marker> + <p>This section is to be read with the <c>gen_fsm(3)</c> manual page + in STDLIB, where all interface functions and callback + functions are described in detail.</p> <section> - <title>Finite State Machines</title> - <p>A finite state machine, FSM, can be described as a set of + <title>Finite-State Machines</title> + <p>A Finite-State Machine (FSM) can be described as a set of relations of the form:</p> <pre> State(S) x Event(E) -> Actions(A), State(S')</pre> <p>These relations are interpreted as meaning:</p> <quote> - <p>If we are in state <c>S</c> and the event <c>E</c> occurs, we - should perform the actions <c>A</c> and make a transition to - the state <c>S'</c>.</p> + <p>If we are in state <c>S</c> and event <c>E</c> occurs, we + are to perform actions <c>A</c> and make a transition to + state <c>S'</c>.</p> </quote> <p>For an FSM implemented using the <c>gen_fsm</c> behaviour, the state transition rules are written as a number of Erlang - functions which conform to the following convention:</p> + functions, which conform to the following convention:</p> <pre> StateName(Event, StateData) -> .. code for actions here ... @@ -55,16 +56,16 @@ StateName(Event, StateData) -> <section> <title>Example</title> - <p>A door with a code lock could be viewed as an FSM. Initially, + <p>A door with a code lock can be viewed as an FSM. Initially, the door is locked. Anytime someone presses a button, this generates an event. Depending on what buttons have been pressed - before, the sequence so far may be correct, incomplete or wrong.</p> - <p>If it is correct, the door is unlocked for 30 seconds (30000 ms). + before, the sequence so far can be correct, incomplete, or wrong.</p> + <p>If it is correct, the door is unlocked for 30 seconds (30,000 ms). If it is incomplete, we wait for another button to be pressed. If it is is wrong, we start all over, waiting for a new button sequence.</p> <p>Implementing the code lock FSM using <c>gen_fsm</c> results in - this callback module:</p> + the following callback module:</p> <marker id="ex"></marker> <code type="none"><![CDATA[ -module(code_lock). @@ -101,85 +102,88 @@ open(timeout, State) -> </section> <section> - <title>Starting a Gen_Fsm</title> - <p>In the example in the previous section, the gen_fsm is started by - calling <c>code_lock:start_link(Code)</c>:</p> + <title>Starting gen_fsm</title> + <p>In the example in the previous section, the <c>gen_fsm</c> is + started by calling <c>code_lock:start_link(Code)</c>:</p> <code type="none"> start_link(Code) -> gen_fsm:start_link({local, code_lock}, code_lock, lists:reverse(Code), []). </code> - <p><c>start_link</c> calls the function <c>gen_fsm:start_link/4</c>. - This function spawns and links to a new process, a gen_fsm.</p> + <p><c>start_link</c> calls the function <c>gen_fsm:start_link/4</c>, + which spawns and links to a new process, a <c>gen_fsm</c>.</p> <list type="bulleted"> <item> - <p>The first argument <c>{local, code_lock}</c> specifies - the name. In this case, the gen_fsm will be locally registered - as <c>code_lock</c>.</p> - <p>If the name is omitted, the gen_fsm is not registered. - Instead its pid must be used. The name could also be given as - <c>{global, Name}</c>, in which case the gen_fsm is registered - using <c>global:register_name/2</c>.</p> + <p>The first argument, <c>{local, code_lock}</c>, specifies + the name. In this case, the <c>gen_fsm</c> is locally + registered as <c>code_lock</c>.</p> + <p>If the name is omitted, the <c>gen_fsm</c> is not registered. + Instead its pid must be used. The name can also be given + as <c>{global, Name}</c>, in which case the <c>gen_fsm</c> is + registered using <c>global:register_name/2</c>.</p> </item> <item> <p>The second argument, <c>code_lock</c>, is the name of - the callback module, that is the module where the callback + the callback module, that is, the module where the callback functions are located.</p> - <p>In this case, the interface functions (<c>start_link</c> and - <c>button</c>) are located in the same module as the callback - functions (<c>init</c>, <c>locked</c> and <c>open</c>). This + <p>The interface functions (<c>start_link</c> and <c>button</c>) + are then located in the same module as the callback + functions (<c>init</c>, <c>locked</c>, and <c>open</c>). This is normally good programming practice, to have the code corresponding to one process contained in one module.</p> </item> <item> - <p>The third argument, <c>Code</c>, is a list of digits which is passed - reversed to the callback function <c>init</c>. Here, <c>init</c> + <p>The third argument, <c>Code</c>, is a list of digits that + which is passed reversed to the callback function <c>init</c>. + Here, <c>init</c> gets the correct code for the lock as indata.</p> </item> <item> - <p>The fourth argument, [], is a list of options. See - <c>gen_fsm(3)</c> for available options.</p> + <p>The fourth argument, <c>[]</c>, is a list of options. See + the <c>gen_fsm(3)</c> manual page for available options.</p> </item> </list> - <p>If name registration succeeds, the new gen_fsm process calls + <p>If name registration succeeds, the new <c>gen_fsm</c> process calls the callback function <c>code_lock:init(Code)</c>. This function is expected to return <c>{ok, StateName, StateData}</c>, where - <c>StateName</c> is the name of the initial state of the gen_fsm. - In this case <c>locked</c>, assuming the door is locked to begin - with. <c>StateData</c> is the internal state of the gen_fsm. (For - gen_fsms, the internal state is often referred to 'state data' to + <c>StateName</c> is the name of the initial state of the + <c>gen_fsm</c>. In this case <c>locked</c>, assuming the door is + locked to begin with. <c>StateData</c> is the internal state of + the <c>gen_fsm</c>. (For <c>gen_fsm</c>, the internal state is + often referred to 'state data' to distinguish it from the state as in states of a state machine.) In this case, the state data is the button sequence so far (empty to begin with) and the correct code of the lock.</p> <code type="none"> init(Code) -> {ok, locked, {[], Code}}.</code> - <p>Note that <c>gen_fsm:start_link</c> is synchronous. It does not - return until the gen_fsm has been initialized and is ready to + <p><c>gen_fsm:start_link</c> is synchronous. It does not return until + the <c>gen_fsm</c> has been initialized and is ready to receive notifications.</p> - <p><c>gen_fsm:start_link</c> must be used if the gen_fsm is part of - a supervision tree, i.e. is started by a supervisor. There is - another function <c>gen_fsm:start</c> to start a stand-alone - gen_fsm, i.e. a gen_fsm which is not part of a supervision tree.</p> + <p><c>gen_fsm:start_link</c> must be used if the <c>gen_fsm</c> is + part of a supervision tree, that is, started by a supervisor. There + is another function, <c>gen_fsm:start</c>, to start a standalone + <c>gen_fsm</c>, that is, a <c>gen_fsm</c> that is not part of a + supervision tree.</p> </section> <section> - <title>Notifying About Events</title> + <title>Notifying about Events</title> <p>The function notifying the code lock about a button event is implemented using <c>gen_fsm:send_event/2</c>:</p> <code type="none"> button(Digit) -> gen_fsm:send_event(code_lock, {button, Digit}).</code> - <p><c>code_lock</c> is the name of the gen_fsm and must agree with - the name used to start it. <c>{button, Digit}</c> is the actual - event.</p> - <p>The event is made into a message and sent to the gen_fsm. When - the event is received, the gen_fsm calls - <c>StateName(Event, StateData)</c> which is expected to return a - tuple <c>{next_state, StateName1, StateData1}</c>. + <p><c>code_lock</c> is the name of the <c>gen_fsm</c> and must + agree with the name used to start it. + <c>{button, Digit}</c> is the actual event.</p> + <p>The event is made into a message and sent to the <c>gen_fsm</c>. + When the event is received, the <c>gen_fsm</c> calls + <c>StateName(Event, StateData)</c>, which is expected to return a + tuple <c>{next_state,StateName1,StateData1}</c>. <c>StateName</c> is the name of the current state and <c>StateName1</c> is the name of the next state to go to. <c>StateData1</c> is a new value for the state data of - the gen_fsm.</p> + the <c>gen_fsm</c>.</p> <code type="none"><![CDATA[ locked({button, Digit}, {SoFar, Code}) -> case [Digit|SoFar] of @@ -198,20 +202,21 @@ open(timeout, State) -> <p>If the door is locked and a button is pressed, the complete button sequence so far is compared with the correct code for the lock and, depending on the result, the door is either unlocked - and the gen_fsm goes to state <c>open</c>, or the door remains in - state <c>locked</c>.</p> + and the <c>gen_fsm</c> goes to state <c>open</c>, or the door + remains in state <c>locked</c>.</p> </section> <section> - <title>Timeouts</title> + <title>Time-Outs</title> <p>When a correct code has been given, the door is unlocked and the following tuple is returned from <c>locked/2</c>:</p> <code type="none"> {next_state, open, {[], Code}, 30000};</code> - <p>30000 is a timeout value in milliseconds. After 30000 ms, i.e. - 30 seconds, a timeout occurs. Then <c>StateName(timeout, StateData)</c> is called. In this case, the timeout occurs when - the door has been in state <c>open</c> for 30 seconds. After that - the door is locked again:</p> + <p>30,000 is a time-out value in milliseconds. After this time, + that is, 30 seconds, a time-out occurs. Then, + <c>StateName(timeout, StateData)</c> is called. The time-out + then occurs when the door has been in state <c>open</c> for 30 + seconds. After that the door is locked again:</p> <code type="none"> open(timeout, State) -> do_lock(), @@ -220,7 +225,7 @@ open(timeout, State) -> <section> <title>All State Events</title> - <p>Sometimes an event can arrive at any state of the gen_fsm. + <p>Sometimes an event can arrive at any state of the <c>gen_fsm</c>. Instead of sending the message with <c>gen_fsm:send_event/2</c> and writing one clause handling the event for each state function, the message can be sent with <c>gen_fsm:send_all_state_event/2</c> @@ -245,15 +250,16 @@ handle_event(stop, _StateName, StateData) -> <section> <title>In a Supervision Tree</title> - <p>If the gen_fsm is part of a supervision tree, no stop function - is needed. The gen_fsm will automatically be terminated by its - supervisor. Exactly how this is done is defined by a - <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> + <p>If the <c>gen_fsm</c> is part of a supervision tree, no stop + function is needed. The <c>gen_fsm</c> is automatically + terminated by its supervisor. Exactly how this is done is + defined by a + <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> set in the supervisor.</p> <p>If it is necessary to clean up before termination, the shutdown - strategy must be a timeout value and the gen_fsm must be set to - trap exit signals in the <c>init</c> function. When ordered - to shutdown, the gen_fsm will then call the callback function + strategy must be a time-out value and the <c>gen_fsm</c> must be + set to trap exit signals in the <c>init</c> function. When ordered + to shutdown, the <c>gen_fsm</c> then calls the callback function <c>terminate(shutdown, StateName, StateData)</c>:</p> <code type="none"> init(Args) -> @@ -270,9 +276,9 @@ terminate(shutdown, StateName, StateData) -> </section> <section> - <title>Stand-Alone Gen_Fsms</title> - <p>If the gen_fsm is not part of a supervision tree, a stop - function may be useful, for example:</p> + <title>Standalone gen_fsm</title> + <p>If the <c>gen_fsm</c> is not part of a supervision tree, a stop + function can be useful, for example:</p> <code type="none"> ... -export([stop/0]). @@ -290,26 +296,28 @@ handle_event(stop, _StateName, StateData) -> terminate(normal, _StateName, _StateData) -> ok.</code> <p>The callback function handling the <c>stop</c> event returns a - tuple <c>{stop,normal,StateData1}</c>, where <c>normal</c> + tuple, <c>{stop,normal,StateData1}</c>, where <c>normal</c> specifies that it is a normal termination and <c>StateData1</c> - is a new value for the state data of the gen_fsm. This will - cause the gen_fsm to call + is a new value for the state data of the <c>gen_fsm</c>. This + causes the <c>gen_fsm</c> to call <c>terminate(normal,StateName,StateData1)</c> and then - terminate gracefully:</p> + it terminates gracefully:</p> </section> </section> <section> <title>Handling Other Messages</title> - <p>If the gen_fsm should be able to receive other messages than - events, the callback function <c>handle_info(Info, StateName, StateData)</c> must be implemented to handle them. Examples of - other messages are exit messages, if the gen_fsm is linked to + <p>If the <c>gen_fsm</c> is to be able to receive other messages + than events, the callback function + <c>handle_info(Info, StateName, StateData)</c> must be implemented + to handle them. Examples of + other messages are exit messages, if the <c>gen_fsm</c> is linked to other processes (than the supervisor) and trapping exit signals.</p> <code type="none"> handle_info({'EXIT', Pid, Reason}, StateName, StateData) -> ..code to handle exits here.. {next_state, StateName1, StateData1}.</code> - <p>The code_change method also has to be implemented.</p> + <p>The code_change method must also be implemented.</p> <code type="none"> code_change(OldVsn, StateName, StateData, Extra) -> ..code to convert state (and more) during code change diff --git a/system/doc/design_principles/gen_server_concepts.xml b/system/doc/design_principles/gen_server_concepts.xml index d24d87aa03..d721845c6d 100644 --- a/system/doc/design_principles/gen_server_concepts.xml +++ b/system/doc/design_principles/gen_server_concepts.xml @@ -21,7 +21,7 @@ </legalnotice> - <title>Gen_Server Behaviour</title> + <title>gen_server Behaviour</title> <prepared></prepared> <docno></docno> <date></date> @@ -29,16 +29,16 @@ <file>gen_server_concepts.xml</file> </header> <marker id="gen_server"></marker> - <p>This chapter should be read in conjunction with - <seealso marker="stdlib:gen_server">gen_server(3)</seealso>, - where all interface functions and callback - functions are described in detail.</p> + <p>This section is to be read with the + <seealso marker="stdlib:gen_server">gen_server(3)</seealso> + manual page in <c>stdblib</c>, where all interface functions and + callback functions are described in detail.</p> <section> <title>Client-Server Principles</title> <p>The client-server model is characterized by a central server and an arbitrary number of clients. The client-server model is - generally used for resource management operations, where several + used for resource management operations, where several different clients want to share a common resource. The server is responsible for managing this resource.</p> <marker id="clientserver"></marker> @@ -49,9 +49,10 @@ <section> <title>Example</title> - <p>An example of a simple server written in plain Erlang was - given in <seealso marker="des_princ#ch1">Overview</seealso>. - The server can be re-implemented using <c>gen_server</c>, + <p>An example of a simple server written in plain Erlang is + provided in + <seealso marker="des_princ#ch1">Overview</seealso>. + The server can be reimplemented using <c>gen_server</c>, resulting in this callback module:</p> <marker id="ex"></marker> <code type="none"> @@ -86,61 +87,60 @@ handle_cast({free, Ch}, Chs) -> <section> <title>Starting a Gen_Server</title> - <p>In the example in the previous section, the gen_server is started - by calling <c>ch3:start_link()</c>:</p> + <p>In the example in the previous section, <c>gen_server</c> is + started by calling <c>ch3:start_link()</c>:</p> <code type="none"> start_link() -> gen_server:start_link({local, ch3}, ch3, [], []) => {ok, Pid}</code> - <p><c>start_link</c> calls the function - <c>gen_server:start_link/4</c>. This function spawns and links to - a new process, a gen_server.</p> + <p><c>start_link</c> calls function <c>gen_server:start_link/4</c>. + This function spawns and links to a new process, a + <c>gen_server</c>.</p> <list type="bulleted"> <item> - <p>The first argument <c>{local, ch3}</c> specifies the name. In - this case, the gen_server will be locally registered as - <c>ch3</c>.</p> - <p>If the name is omitted, the gen_server is not registered. - Instead its pid must be used. The name could also be given - as <c>{global, Name}</c>, in which case the gen_server is + <p>The first argument, <c>{local, ch3}</c>, specifies the name. + The gen_server is then locally registered as <c>ch3</c>.</p> + <p>If the name is omitted, the <c>gen_server</c> is not registered. + Instead its pid must be used. The name can also be given + as <c>{global, Name}</c>, in which case the <c>gen_server</c> is registered using <c>global:register_name/2</c>.</p> </item> <item> <p>The second argument, <c>ch3</c>, is the name of the callback - module, that is the module where the callback functions are + module, that is, the module where the callback functions are located.</p> - <p>In this case, the interface functions (<c>start_link</c>, - <c>alloc</c> and <c>free</c>) are located in the same module - as the callback functions (<c>init</c>, <c>handle_call</c> and + <p>The interface functions (<c>start_link</c>, <c>alloc</c>, + and <c>free</c>) are then located in the same module + as the callback functions (<c>init</c>, <c>handle_call</c>, and <c>handle_cast</c>). This is normally good programming practice, to have the code corresponding to one process contained in one module.</p> </item> <item> - <p>The third argument, [], is a term which is passed as-is to - the callback function <c>init</c>. Here, <c>init</c> does not + <p>The third argument, <c>[]</c>, is a term that is passed as is + to the callback function <c>init</c>. Here, <c>init</c> does not need any indata and ignores the argument.</p> </item> <item> - <p>The fourth argument, [], is a list of options. See - <c>gen_server(3)</c> for available options.</p> + <p>The fourth argument, <c>[]</c>, is a list of options. See the + <c>gen_server(3)</c> manual page for available options.</p> </item> </list> - <p>If name registration succeeds, the new gen_server process calls - the callback function <c>ch3:init([])</c>. <c>init</c> is expected - to return <c>{ok, State}</c>, where <c>State</c> is the internal - state of the gen_server. In this case, the state is the available - channels.</p> + <p>If name registration succeeds, the new <c>gen_server</c> process + calls the callback function <c>ch3:init([])</c>. <c>init</c> is + expected to return <c>{ok, State}</c>, where <c>State</c> is the + internal state of the <c>gen_server</c>. In this case, the state + is the available channels.</p> <code type="none"> init(_Args) -> {ok, channels()}.</code> - <p>Note that <c>gen_server:start_link</c> is synchronous. It does - not return until the gen_server has been initialized and is ready + <p><c>gen_server:start_link</c> is synchronous. It does not return + until the <c>gen_server</c> has been initialized and is ready to receive requests.</p> - <p><c>gen_server:start_link</c> must be used if the gen_server is - part of a supervision tree, i.e. is started by a supervisor. - There is another function <c>gen_server:start</c> to start a - stand-alone gen_server, i.e. a gen_server which is not part of a - supervision tree.</p> + <p><c>gen_server:start_link</c> must be used if the <c>gen_server</c> + is part of a supervision tree, that is, started by a supervisor. + There is another function, <c>gen_server:start</c>, to start a + standalone <c>gen_server</c>, that is, a <c>gen_server</c> that + is not part of a supervision tree.</p> </section> <section> @@ -150,14 +150,17 @@ init(_Args) -> <code type="none"> alloc() -> gen_server:call(ch3, alloc).</code> - <p><c>ch3</c> is the name of the gen_server and must agree with - the name used to start it. <c>alloc</c> is the actual request.</p> - <p>The request is made into a message and sent to the gen_server. - When the request is received, the gen_server calls - <c>handle_call(Request, From, State)</c> which is expected to - return a tuple <c>{reply, Reply, State1}</c>. <c>Reply</c> is - the reply which should be sent back to the client, and - <c>State1</c> is a new value for the state of the gen_server.</p> + <p><c>ch3</c> is the name of the <c>gen_server</c> and must agree + with the name used to start it. <c>alloc</c> is the actual + request.</p> + <p>The request is made into a message and sent to the + <c>gen_server</c>. When the request is received, the + <c>gen_server</c> calls + <c>handle_call(Request, From, State)</c>, which is expected to + return a tuple <c>{reply,Reply,State1}</c>. <c>Reply</c> is + the reply that is to be sent back to the client, and + <c>State1</c> is a new value for the state of the + <c>gen_server</c>.</p> <code type="none"> handle_call(alloc, _From, Chs) -> {Ch, Chs2} = alloc(Chs), @@ -166,8 +169,8 @@ handle_call(alloc, _From, Chs) -> the new state is the set of remaining available channels <c>Chs2</c>.</p> <p>Thus, the call <c>ch3:alloc()</c> returns the allocated channel - <c>Ch</c> and the gen_server then waits for new requests, now - with an updated list of available channels.</p> + <c>Ch</c> and the <c>gen_server</c> then waits for new requests, + now with an updated list of available channels.</p> </section> <section> @@ -177,20 +180,21 @@ handle_call(alloc, _From, Chs) -> <code type="none"> free(Ch) -> gen_server:cast(ch3, {free, Ch}).</code> - <p><c>ch3</c> is the name of the gen_server. <c>{free, Ch}</c> is - the actual request.</p> - <p>The request is made into a message and sent to the gen_server. + <p><c>ch3</c> is the name of the <c>gen_server</c>. + <c>{free, Ch}</c> is the actual request.</p> + <p>The request is made into a message and sent to the + <c>gen_server</c>. <c>cast</c>, and thus <c>free</c>, then returns <c>ok</c>.</p> - <p>When the request is received, the gen_server calls - <c>handle_cast(Request, State)</c> which is expected to - return a tuple <c>{noreply, State1}</c>. <c>State1</c> is a new - value for the state of the gen_server.</p> + <p>When the request is received, the <c>gen_server</c> calls + <c>handle_cast(Request, State)</c>, which is expected to + return a tuple <c>{noreply,State1}</c>. <c>State1</c> is a new + value for the state of the <c>gen_server</c>.</p> <code type="none"> handle_cast({free, Ch}, Chs) -> Chs2 = free(Ch, Chs), {noreply, Chs2}.</code> <p>In this case, the new state is the updated list of available - channels <c>Chs2</c>. The gen_server is now ready for new + channels <c>Chs2</c>. The <c>gen_server</c> is now ready for new requests.</p> </section> @@ -199,15 +203,17 @@ handle_cast({free, Ch}, Chs) -> <section> <title>In a Supervision Tree</title> - <p>If the gen_server is part of a supervision tree, no stop - function is needed. The gen_server will automatically be + <p>If the <c>gen_server</c> is part of a supervision tree, no stop + function is needed. The <c>gen_server</c> is automatically terminated by its supervisor. Exactly how this is done is - defined by a <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> set in the supervisor.</p> + defined by a + <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> + set in the supervisor.</p> <p>If it is necessary to clean up before termination, the shutdown - strategy must be a timeout value and the gen_server must be set - to trap exit signals in the <c>init</c> function. When ordered - to shutdown, the gen_server will then call the callback function - <c>terminate(shutdown, State)</c>:</p> + strategy must be a time-out value and the <c>gen_server</c> must + be set to trap exit signals in function <c>init</c>. When ordered + to shutdown, the <c>gen_server</c> then calls the callback + function <c>terminate(shutdown, State)</c>:</p> <code type="none"> init(Args) -> ..., @@ -223,9 +229,9 @@ terminate(shutdown, State) -> </section> <section> - <title>Stand-Alone Gen_Servers</title> - <p>If the gen_server is not part of a supervision tree, a stop - function may be useful, for example:</p> + <title>Standalone Gen_Servers</title> + <p>If the <c>gen_server</c> is not part of a supervision tree, a + stop function can be useful, for example:</p> <code type="none"> ... export([stop/0]). @@ -245,26 +251,26 @@ handle_cast({free, Ch}, State) -> terminate(normal, State) -> ok.</code> <p>The callback function handling the <c>stop</c> request returns - a tuple <c>{stop, normal, State1}</c>, where <c>normal</c> + a tuple <c>{stop,normal,State1}</c>, where <c>normal</c> specifies that it is a normal termination and <c>State1</c> is - a new value for the state of the gen_server. This will cause - the gen_server to call <c>terminate(normal,State1)</c> and then - terminate gracefully.</p> + a new value for the state of the <c>gen_server</c>. This causes + the <c>gen_server</c> to call <c>terminate(normal, State1)</c> + and then it terminates gracefully.</p> </section> </section> <section> <title>Handling Other Messages</title> - <p>If the gen_server should be able to receive other messages than - requests, the callback function <c>handle_info(Info, State)</c> + <p>If the <c>gen_server</c> is to be able to receive other messages + than requests, the callback function <c>handle_info(Info, State)</c> must be implemented to handle them. Examples of other messages - are exit messages, if the gen_server is linked to other processes - (than the supervisor) and trapping exit signals.</p> + are exit messages, if the <c>gen_server</c> is linked to other + processes (than the supervisor) and trapping exit signals.</p> <code type="none"> handle_info({'EXIT', Pid, Reason}, State) -> ..code to handle exits here.. {noreply, State1}.</code> - <p>The code_change method also has to be implemented.</p> + <p>The <c>code_change</c> method must also be implemented.</p> <code type="none"> code_change(OldVsn, State, Extra) -> ..code to convert state (and more) during code change diff --git a/system/doc/design_principles/included_applications.xml b/system/doc/design_principles/included_applications.xml index 3aa43fd595..7f139edf76 100644 --- a/system/doc/design_principles/included_applications.xml +++ b/system/doc/design_principles/included_applications.xml @@ -28,35 +28,36 @@ <rev></rev> <file>included_applications.xml</file> </header> + <marker id="included appl"></marker> <section> - <title>Definition</title> + <title>Introduction</title> <p>An application can <em>include</em> other applications. An <em>included application</em> has its own application directory and <c>.app</c> file, but it is started as part of the supervisor tree of another application.</p> <p>An application can only be included by one other application.</p> <p>An included application can include other applications.</p> - <p>An application which is not included by any other application is + <p>An application that is not included by any other application is called a <em>primary application</em>.</p> <marker id="inclappls"></marker> <image file="../design_principles/inclappls.gif"> - <icaption>Primary Application and Included Applications.</icaption> + <icaption>Primary Application and Included Applications</icaption> </image> - <p>The application controller will automatically load any included - applications when loading a primary application, but not start + <p>The application controller automatically loads any included + applications when loading a primary application, but does not start them. Instead, the top supervisor of the included application must be started by a supervisor in the including application.</p> <p>This means that when running, an included application is in fact - part of the primary application and a process in an included - application will consider itself belonging to the primary + part of the primary application, and a process in an included + application considers itself belonging to the primary application.</p> </section> <section> <title>Specifying Included Applications</title> <p>Which applications to include is defined by - the <c>included_applications</c> key in the <c>.app</c> file.</p> + the <c>included_applications</c> key in the <c>.app</c> file:</p> <pre> {application, prim_app, [{description, "Tree application"}, @@ -71,7 +72,7 @@ </section> <section> - <title>Synchronizing Processes During Startup</title> + <title>Synchronizing Processes during Startup</title> <p>The supervisor tree of an included application is started as part of the supervisor tree of the including application. If there is a need for synchronization between processes in @@ -79,12 +80,12 @@ by using <em>start phases</em>.</p> <p>Start phases are defined by the <c>start_phases</c> key in the <c>.app</c> file as a list of tuples <c>{Phase,PhaseArgs}</c>, - where <c>Phase</c> is an atom and <c>PhaseArgs</c> is a term. - Also, the value of the <c>mod</c> key of the including application + where <c>Phase</c> is an atom and <c>PhaseArgs</c> is a term.</p> + <p>The value of the <c>mod</c> key of the including application must be set to <c>{application_starter,[Module,StartArgs]}</c>, - where <c>Module</c> as usual is the application callback module - and <c>StartArgs</c> a term provided as argument to the callback - function <c>Module:start/2</c>.</p> + where <c>Module</c> as usual is the application callback module. + <c>StartArgs</c> is a term provided as argument to the callback + function <c>Module:start/2</c>:</p> <code type="none"> {application, prim_app, [{description, "Tree application"}, @@ -108,36 +109,38 @@ {mod, {incl_app_cb,[]}} ]}.</code> <p>When starting a primary application with included applications, - the primary application is started the normal way: - The application controller creates an application master for - the application, and the application master calls + the primary application is started the normal way, that is:</p> + <list type="bulleted"> + <item>The application controller creates an application master for + the application</item> + <item>The application master calls <c>Module:start(normal, StartArgs)</c> to start the top - supervisor.</p> + supervisor.</item> + </list> <p>Then, for the primary application and each included application in top-down, left-to-right order, the application master calls <c>Module:start_phase(Phase, Type, PhaseArgs)</c> for each phase - defined for for the primary application, in that order. - Note that if a phase is not defined for an included application, + defined for the primary application, in that order. If a phase + is not defined for an included application, the function is not called for this phase and application.</p> <p>The following requirements apply to the <c>.app</c> file for an included application:</p> <list type="bulleted"> - <item>The <c>{mod, {Module,StartArgs}}</c> option must be - included. This option is used to find the callback module - <c>Module</c> of the application. <c>StartArgs</c> is ignored, - as <c>Module:start/2</c> is called only for the primary - application.</item> + <item>The <c>{mod, {Module,StartArgs}}</c> option must be included. + This option is used to find the callback module <c>Module</c> of the + application. <c>StartArgs</c> is ignored, as <c>Module:start/2</c> + is called only for the primary application.</item> <item>If the included application itself contains included - applications, instead the option - <c>{mod, {application_starter, [Module,StartArgs]}}</c> must be - included.</item> + applications, instead the + <c>{mod, {application_starter, [Module,StartArgs]}}</c> option + must be included.</item> <item>The <c>{start_phases, [{Phase,PhaseArgs}]}</c> option must - be included, and the set of specified phases must be a subset - of the set of phases specified for the primary application.</item> + be included, and the set of specified phases must be a subset + of the set of phases specified for the primary application.</item> </list> <p>When starting <c>prim_app</c> as defined above, the application - controller will call the following callback functions, before - <c>application:start(prim_app)</c> returns a value:</p> + controller calls the following callback functions before + <c>application:start(prim_app)</c> returns a value:</p> <code type="none"> application:start(prim_app) => prim_app_cb:start(normal, []) diff --git a/system/doc/design_principles/release_handling.xml b/system/doc/design_principles/release_handling.xml index 9d1e2c8669..eeb71125b6 100644 --- a/system/doc/design_principles/release_handling.xml +++ b/system/doc/design_principles/release_handling.xml @@ -28,128 +28,126 @@ <rev></rev> <file>release_handling.xml</file> </header> + <marker id="release handling"></marker> <section> <title>Release Handling Principles</title> <p>An important feature of the Erlang programming language is - the ability to change module code in run-time, <em>code replacement</em>, as described in <em>Erlang Reference Manual</em>.</p> + the ability to change module code in runtime, + <em>code replacement</em>, as described in the Erlang + Reference Manual.</p> <p>Based on this feature, the OTP application SASL provides a framework for upgrading and downgrading between different - versions of an entire release in run-time. This is what we call + versions of an entire release in runtime. This is called <em>release handling</em>.</p> - <p>The framework consists of off-line support (<c>systools</c>) for - generating scripts and building release packages, and on-line - support (<c>release_handler</c>) for unpacking and installing - release packages.</p> - <p>Note that the minimal system based on Erlang/OTP, enabling - release handling, thus consists of Kernel, STDLIB and SASL.</p> - <list type="ordered"> - <item> - <p>A release is created as described in the previous chapter - <seealso marker="release_structure">Releases</seealso>. - The release is transferred to and installed at target - environment. Refer to <em>System Principles</em> for - information of how to install the first target system.</p> - </item> - <item> - <p>Modifications, for example error corrections, are made to - the code in the development environment.</p> - </item> - <item> - <p>At some point it is time to make a new version of release. - The relevant <c>.app</c> files are updated and a new - <c>.rel</c> file is written.</p> - </item> - <item> - <p>For each modified application, an - <seealso marker="#appup">application upgrade file</seealso>, - <c>.appup</c>, is created. In this file, it is described how - to upgrade and/or downgrade between the old and new version of - the application.</p> - </item> - <item> - <p>Based on the <c>.appup</c> files, a - <seealso marker="#relup">release upgrade file</seealso> called - <c>relup</c>, is created. This file describes how to upgrade - and/or downgrade between the old and new version of - the entire release.</p> - </item> - <item> - <p>A new release package is made and transferred to - the target system.</p> - </item> - <item> - <p>The new release package is unpacked using the release - handler.</p> - </item> - <item> - <p>The new version of the release is installed, also using - the release handler. This is done by evaluating - the instructions in <c>relup</c>. Modules may be added, - deleted or re-loaded, applications may be started, stopped or - re-started etc. In some cases, it is even necessary to restart - the entire emulator.</p> - <p>If the installation fails, the system may be rebooted. - The old release version is then automatically used.</p> - </item> - <item> - <p>If the installation succeeds, the new version is made - the default version, which should now be used in case of a - system reboot.</p> - </item> - </list> - <p>The next chapter, <seealso marker="appup_cookbook">Appup Cookbook</seealso>, contains examples of <c>.appup</c> files - for typical cases of upgrades/downgrades that are normally easy - to handle in run-time. However, there are a many aspects that can - make release handling complicated. To name a few examples:</p> + <p>The framework consists of:</p> <list type="bulleted"> - <item> - <p>Complicated or circular dependencies can make it difficult - or even impossible to decide in which order things must be - done without risking run-time errors during an upgrade or - downgrade. Dependencies may be:</p> - <list type="bulleted"> - <item>between nodes,</item> - <item>between processes, and</item> - <item>between modules.</item> - </list> - </item> - <item> - <p>During release handling, non-affected processes continue - normal execution. This may lead to timeouts or other problems. - For example, new processes created in the time window between - suspending processes using a certain module and loading a new - version of this module, may execute old code.</p> - </item> + <item>Offline support - <c>systools</c> for generating scripts + and building release packages</item> + <item>Online support - <c>release_handler</c> for unpacking and + installing release packages</item> </list> - <p>It is therefore recommended that code is changed in as small + <p>The minimal system based on Erlang/OTP, enabling release handling, + thus consists of the Kernel, STDLIB, and SASL + applications.</p> + + <section> + <title>Release Handling Workflow</title> + <p><em>Step 1</em>) A release is created as described in + <seealso marker="release_structure">Releases</seealso>.</p> + <p><em>Step 2</em>) The release is transferred to and installed at + target environment. For information of how to install the first + target system, see System Principles.</p> + <p><em>Step 3</em>) Modifications, for example, error corrections, + are made to the code in the development environment.</p> + <p><em>Step 4</em>) At some point, it is time to make a new version + of release. The relevant <c>.app</c> files are updated and a new + <c>.rel</c> file is written.</p> + <p><em>Step 5</em>) For each modified application, an + <seealso marker="#appup">application upgrade file</seealso>, + <c>.appup</c>, is created. In this file, it is described how to + upgrade and/or downgrade between the old and new version of the + application.</p> + <p><em>Step 6</em>) Based on the <c>.appup</c> files, a + <seealso marker="#relup">release upgrade file</seealso> called + <c>relup</c>, is created. This file describes how to upgrade and/or + downgrade between the old and new version of the entire release.</p> + <p><em>Step 7</em>) A new release package is made and transferred to + the target system.</p> + <p><em>Step 8</em>) The new release package is unpacked using the + release handler.</p> + <p><em>Step 9</em>) The new version of the release is installed, + also using the release handler. This is done by evaluating the + instructions in <c>relup</c>. Modules can be added, deleted, or + reloaded, applications can be started, stopped, or restarted, and so + on. In some cases, it is even necessary to restart the entire + emulator.</p> + <list type="bulleted"> + <item>If the installation fails, the system can be rebooted. + The old release version is then automatically used.</item> + <item>If the installation succeeds, the new version is made + the default version, which is to now be used if there is a + system reboot.</item> + </list> + </section> + + <section> + <title>Release Handling Aspects</title> + <p><seealso marker="appup_cookbook">Appup Cookbook</seealso>, + contains examples of <c>.appup</c> files + for typical cases of upgrades/downgrades that are normally easy to + handle in runtime. However, many aspects can make release handling + complicated, for example:</p> + <list type="bulleted"> + <item> + <p>Complicated or circular dependencies can make it difficult + or even impossible to decide in which order things must be + done without risking runtime errors during an upgrade or + downgrade. Dependencies can be:</p> + <list type="bulleted"> + <item>Between nodes</item> + <item>Between processes</item> + <item>Between modules</item> + </list> + </item> + <item> + <p>During release handling, non-affected processes continue + normal execution. This can lead to time-outs or other problems. + For example, new processes created in the time window between + suspending processes using a certain module, and loading a new + version of this module, can execute old code.</p> + </item> + </list> + <p>It is thus recommended that code is changed in as small steps as possible, and always kept backwards compatible.</p> + </section> </section> <section> <marker id="req"></marker> <title>Requirements</title> - <p>For release handling to work properly, the runtime system needs - to have knowledge about which release it is currently running. It - must also be able to change (in run-time) which boot script and - system configuration file should be used if the system is - rebooted, for example by <c>heart</c> after a failure. - Therefore, Erlang must be started as an embedded system, see - <em>Embedded System</em> for information on how to do this.</p> + <p>For release handling to work properly, the runtime system must + have knowledge about which release it is running. It + must also be able to change (in runtime) which boot script and + system configuration file to use if the system is + rebooted, for example, by <c>heart</c> after a failure. + Thus, Erlang must be started as an embedded system; for + information on how to do this, see Embedded System.</p> <p>For system reboots to work properly, it is also required that - the system is started with heart beat monitoring, see - <c>erl(1)</c> and <c>heart(3)</c>.</p> + the system is started with heartbeat monitoring, see the + <c>erl(1)</c> manual page in ERTS and the <c>heart(3)</c> + manual page in Kernel</p> <p>Other requirements:</p> <list type="bulleted"> <item> <p>The boot script included in a release package must be generated from the same <c>.rel</c> file as the release package itself.</p> - <p>Information about applications are fetched from the script + <p>Information about applications is fetched from the script when an upgrade or downgrade is performed.</p> </item> <item> - <p>The system must be configured using one and only one system + <p>The system must be configured using only one system configuration file, called <c>sys.config</c>.</p> <p>If found, this file is automatically included when a release package is created.</p> @@ -165,13 +163,13 @@ <section> <title>Distributed Systems</title> - <p>If the system consists of several Erlang nodes, each node may use + <p>If the system consists of several Erlang nodes, each node can use its own version of the release. The release handler is a locally registered process and must be called at each node where an - upgrade or downgrade is required. There is a release handling - instruction that can be used to synchronize the release handler - processes at a number of nodes: <c>sync_nodes</c>. See - <c>appup(4)</c>.</p> + upgrade or downgrade is required. A release handling + instruction, <c>sync_nodes</c>, can be used to synchronize the + release handler processes at a number of nodes, see the + <c>appup(4)</c> manual page in SASL.</p> </section> <section> @@ -183,31 +181,26 @@ instructions. To make it easier for the user, there are also a number of <em>high-level</em> instructions, which are translated to low-level instructions by <c>systools:make_relup</c>.</p> - <p>Here, some of the most frequently used instructions are - described. The complete list of instructions can be found in - <c>appup(4)</c>.</p> + <p>Some of the most frequently used instructions are described in + this section. The complete list of instructions is included in the + <c>appup(4)</c> manual page in SASL.</p> <p>First, some definitions:</p> - <taglist> - <tag><em>Residence module</em></tag> - <item> - <p>The module where a process has its tail-recursive loop - function(s). If the tail-recursive loop functions are - implemented in several modules, all those modules are residence - modules for the process.</p> - </item> - <tag><em>Functional module</em></tag> - <item> - <p>A module which is not a residence module for any process.</p> - </item> - </taglist> - <p>Note that for a process implemented using an OTP behaviour, - the behaviour module is the residence module for that process. - The callback module is a functional module.</p> + <list type="bulleted"> + <item><em>Residence module</em> - The module where a process has + its tail-recursive loop function(s). If these functions are + implemented in several modules, all those modules are residence + modules for the process.</item> + <item><em>Functional module</em> - A module that is not a + residence module for any process.</item> + </list> + <p>For a process implemented using an OTP behaviour, the behaviour + module is the residence module for that process. + The callback module is a functional module.</p> <section> <title>load_module</title> <p>If a simple extension has been made to a functional module, it - is sufficient to simply load the new version of the module into + is sufficient to load the new version of the module into the system, and remove the old version. This is called <em>simple code replacement</em> and for this the following instruction is used:</p> @@ -217,44 +210,50 @@ <section> <title>update</title> - <p>If a more complex change has been made, for example a change - to the format of the internal state of a gen_server, simple code - replacement is not sufficient. Instead it is necessary to - suspend the processes using the module (to avoid that they try - to handle any requests before the code replacement is - completed), ask them to transform the internal state format and - switch to the new version of the module, remove the old version - and last, resume the processes. This is called <em>synchronized code replacement</em> and for this the following instructions - are used:</p> + <p>If a more complex change has been made, for example, a change + to the format of the internal state of a <c>gen_server</c>, simple + code replacement is not sufficient. Instead, it is necessary to:</p> + <list type="bulleted"> + <item>Suspend the processes using the module (to avoid that + they try to handle any requests before the code replacement is + completed).</item> + <item>Ask them to transform the internal state format and + switch to the new version of the module.</item> + <item>Remove the old version.</item> + <item>Resume the processes.</item> + </list> + <p>This is called <em>synchronized code replacement</em> and for + this the following instructions are used:</p> <code type="none"> {update, Module, {advanced, Extra}} {update, Module, supervisor}</code> <p><c>update</c> with argument <c>{advanced,Extra}</c> is used when changing the internal state of a behaviour as described - above. It will cause behaviour processes to call the callback + above. It causes behaviour processes to call the callback function <c>code_change</c>, passing the term <c>Extra</c> and - some other information as arguments. See the man pages for + some other information as arguments. See the manual pages for the respective behaviours and - <seealso marker="appup_cookbook#int_state">Appup Cookbook</seealso>.</p> + <seealso marker="appup_cookbook#int_state">Appup Cookbook</seealso>.</p> <p><c>update</c> with argument <c>supervisor</c> is used when changing the start specification of a supervisor. See - <seealso marker="appup_cookbook#sup">Appup Cookbook</seealso>.</p> + <seealso marker="appup_cookbook#sup">Appup Cookbook</seealso>.</p> <p>When a module is to be updated, the release handler finds which processes that are <em>using</em> the module by traversing the supervision tree of each running application and checking all the child specifications:</p> <code type="none"> {Id, StartFunc, Restart, Shutdown, Type, Modules}</code> - <p>A process is using a module if the name is listed in + <p>A process uses a module if the name is listed in <c>Modules</c> in the child specification for the process.</p> <p>If <c>Modules=dynamic</c>, which is the case for event managers, the event manager process informs the release handler - about the list of currently installed event handlers (gen_fsm) - and it is checked if the module name is in this list instead.</p> + about the list of currently installed event handlers + (<c>gen_fsm</c>), and it is checked if the module name is in + this list instead.</p> <p>The release handler suspends, asks for code change, and resumes processes by calling the functions - <c>sys:suspend/1,2</c>, <c>sys:change_code/4,5</c> and - <c>sys:resume/1,2</c> respectively.</p> + <c>sys:suspend/1,2</c>, <c>sys:change_code/4,5</c>, and + <c>sys:resume/1,2</c>, respectively.</p> </section> <section> @@ -263,39 +262,39 @@ used:</p> <code type="none"> {add_module, Module}</code> - <p>The instruction loads the module and is absolutely necessary + <p>The instruction loads the module and is necessary when running Erlang in embedded mode. It is not strictly required when running Erlang in interactive (default) mode, since the code server then automatically searches for and loads unloaded modules.</p> - <p>The opposite of <c>add_module</c> is <c>delete_module</c> which + <p>The opposite of <c>add_module</c> is <c>delete_module</c>, which unloads a module:</p> <code type="none"> {delete_module, Module}</code> - <p>Note that any process, in any application, with <c>Module</c> + <p>Any process, in any application, with <c>Module</c> as residence module, is killed when the instruction is - evaluated. The user should therefore ensure that all such + evaluated. The user must therefore ensure that all such processes are terminated before deleting the module, to avoid - a possible situation with failing supervisor restarts.</p> + a situation with failing supervisor restarts.</p> </section> <section> <title>Application Instructions</title> - <p>Instruction for adding an application:</p> + <p>The following is the instruction for adding an application:</p> <code type="none"> {add_application, Application}</code> <p>Adding an application means that the modules defined by the <c>modules</c> key in the <c>.app</c> file are loaded using - a number of <c>add_module</c> instructions, then the application + a number of <c>add_module</c> instructions, and then the application is started.</p> - <p>Instruction for removing an application:</p> + <p>The following is the instruction for removing an application:</p> <code type="none"> {remove_application, Application}</code> <p>Removing an application means that the application is stopped, the modules are unloaded using a number of <c>delete_module</c> - instructions and then the application specification is unloaded + instructions, and then the application specification is unloaded from the application controller.</p> - <p>Instruction for restarting an application:</p> + <p>The following is the instruction for restarting an application:</p> <code type="none"> {restart_application, Application}</code> <p>Restarting an application means that the application is stopped @@ -305,46 +304,48 @@ </section> <section> - <title>apply (low-level)</title> + <title>apply (Low-Level)</title> <p>To call an arbitrary function from the release handler, the following instruction is used:</p> <code type="none"> {apply, {M, F, A}}</code> - <p>The release handler will evaluate <c>apply(M, F, A)</c>.</p> + <p>The release handler evalutes <c>apply(M, F, A)</c>.</p> </section> <section> <marker id="restart_new_emulator_instr"></marker> - <title>restart_new_emulator (low-level)</title> + <title>restart_new_emulator (Low-Level)</title> <p>This instruction is used when changing to a new emulator - version, or when any of the core applications kernel, stdlib - or sasl is upgraded. If a system reboot is needed for some - other reason, the <c>restart_emulator</c> instruction should - be used instead.</p> - <p>Requires that the system is started with heart beat - monitoring, see <c>erl(1)</c> and <c>heart(3)</c>.</p> - <p>The <c>restart_new_emulator</c> instruction shall always be - the very first instruction in a relup. If the relup is - generated by <c>systools:make_relup/3,4</c> this is + version, or when any of the core applications Kernel, + STDLIB, or SASL is upgraded. If a system reboot + is needed for another reason, the <c>restart_emulator</c> + instruction is to be used instead.</p> + <p>This instruction requires that the system is started with + heartbeat monitoring, see the <c>erl(1)</c> manual page in + ERTS and the <c>heart(3)</c> manual page in Kernel.</p> + <p>The <c>restart_new_emulator</c> instruction must always be + the first instruction in a relup. If the relup is + generated by <c>systools:make_relup/3,4</c>, this is automatically ensured.</p> <p>When the release handler encounters the instruction, it first generates a temporary boot file, which starts the new versions of the emulator and the core applications, and the old version of all other applications. Then it shuts down - the current emulator by calling <c>init:reboot()</c>, see - <c>init(3)</c>. All processes are terminated gracefully and - the system is rebooted by the heart program, using the + the current emulator by calling <c>init:reboot()</c>, see the + <c>init(3)</c> manual page in Kernel. + All processes are terminated gracefully and + the system is rebooted by the <c>heart</c> program, using the temporary boot file. After the reboot, the rest of the relup instructions are executed. This is done as a part of the temporary boot script.</p> <warning> - <p>Since this mechanism causes the new versions of the - emulator and core applications to run with the old version of - other applications during startup, extra care must be taken to + <p>This mechanism causes the new versions of the emulator and + core applications to run with the old version of other + applications during startup. Thus, take extra care to avoid incompatibility. Incompatible changes in the core - applications may in some situations be necessary. If possible, + applications can in some situations be necessary. If possible, such changes are preceded by deprecation over two major - releases before the actual change. To make sure your + releases before the actual change. To ensure the application is not crashed by an incompatible change, always remove any call to deprecated functions as soon as possible.</p> @@ -352,35 +353,36 @@ <p>An info report is written when the upgrade is completed. To programmatically find out if the upgrade is complete, call <c>release_handler:which_releases(current)</c> and check - if it returns the expected (i.e. the new) release.</p> + if it returns the expected (that is, the new) release.</p> <p>The new release version must be made permanent when the new - emulator is up and running. Otherwise, the old version will be - used in case of a new system reboot.</p> - <p>On UNIX, the release handler tells the heart program which - command to use to reboot the system. Note that the environment - variable <c>HEART_COMMAND</c>, normally used by the heart - program, in this case is ignored. The command instead defaults - to <c>$ROOT/bin/start</c>. Another command can be set - by using the SASL configuration parameter <c>start_prg</c>, see - <c>sasl(6)</c>.</p> + emulator is operational. Otherwise, the old version will be + used if there is a new system reboot.</p> + <p>On UNIX, the release handler tells the <c>heart</c> program + which command to use to reboot the system. The environment + variable <c>HEART_COMMAND</c>, normally used by the <c>heart</c> + program, is ignored in this case. The command instead defaults + to <c>$ROOT/bin/start</c>. Another command can be set by using + the SASL configuration parameter <c>start_prg</c>, see + the <c>sasl(6)</c> manual page.</p> </section> <section> <marker id="restart_emulator_instr"></marker> - <title>restart_emulator (low-level)</title> - <p>This instruction is not related to upgrades of erts or any of - the core applications. It can be used by any application to + <title>restart_emulator (Low-Level)</title> + <p>This instruction is not related to upgrades of ERTS or any + of the core applications. It can be used by any application to force a restart of the emulator after all upgrade instructions are executed.</p> - <p>There can only be one <c>restart_emulator</c> instruction in - a relup script, and it shall always be placed at the end. If - the relup is generated by <c>systools:make_relup/3,4</c> this + <p>A relup script can only have one <c>restart_emulator</c> + instruction and it must always be placed at the end. If + the relup is generated by <c>systools:make_relup/3,4</c>, this is automatically ensured.</p> <p>When the release handler encounters the instruction, it shuts - down the emulator by calling <c>init:reboot()</c>, see - <c>init(3)</c>. All processes are terminated gracefully and - the system can then be rebooted by the heart program using the - new release version. No more upgrade instruction will be + down the emulator by calling <c>init:reboot()</c>, see the + <c>init(3)</c> manual page in Kernel. + All processes are terminated gracefully and the system + can then be rebooted by the <c>heart</c> program using the + new release version. No more upgrade instruction is executed after the restart.</p> </section> </section> @@ -389,10 +391,11 @@ <marker id="appup"></marker> <title>Application Upgrade File</title> <p>To define how to upgrade/downgrade between the current version - and previous versions of an application, we create an - <em>application upgrade file</em>, or in short <c>.appup</c> file. - The file should be called <c>Application.appup</c>, where - <c>Application</c> is the name of the application:</p> + and previous versions of an application, an + <em>application upgrade file</em>, or in short an <c>.appup</c> + file is created. + The file is to be called <c>Application.appup</c>, where + <c>Application</c> is the application name:</p> <code type="none"> {Vsn, [{UpFromVsn1, InstructionsU1}, @@ -401,24 +404,27 @@ [{DownToVsn1, InstructionsD1}, ..., {DownToVsnK, InstructionsDK}]}.</code> - <p><c>Vsn</c>, a string, is the current version of the application, - as defined in the <c>.app</c> file. Each <c>UpFromVsn</c> - is a previous version of the application to upgrade from, and each - <c>DownToVsn</c> is a previous version of the application to - downgrade to. Each <c>Instructions</c> is a list of release - handling instructions.</p> - <p>The syntax and contents of the <c>appup</c> file are described - in detail in <c>appup(4)</c>.</p> - <p>In the chapter <seealso marker="appup_cookbook">Appup Cookbook</seealso>, examples of <c>.appup</c> files for typical - upgrade/downgrade cases are given.</p> - <p>Example: Consider the release <c>ch_rel-1</c> from - the <seealso marker="release_structure#ch_rel">Releases</seealso> - chapter. Assume we want to add a function <c>available/0</c> to - the server <c>ch3</c> which returns the number of available - channels:</p> - <p>(Hint: When trying out the example, make the changes in a copy of - the original directory, so that the first versions are still - available.)</p> + <list type="bulleted"> + <item><c>Vsn</c>, a string, is the current version of the application, + as defined in the <c>.app</c> file.</item> + <item>Each <c>UpFromVsn</c> is a previous version of the application + to upgrade from.</item> + <item>Each <c>DownToVsn</c> is a previous version of the application + to downgrade to.</item> + <item>Each <c>Instructions</c> is a list of release handling + instructions.</item> + </list> + <p>For information about the syntax and contents of the <c>.appup</c> + file, see the <c>appup(4)</c> manual page in SASL.</p> + <p><seealso marker="appup_cookbook">Appup Cookbook</seealso> + includes examples of <c>.appup</c> files for typical upgrade/downgrade + cases.</p> + <p><em>Example:</em> Consider the release <c>ch_rel-1</c> from + <seealso marker="release_structure#ch_rel">Releases</seealso>. + Assume you want to add a function <c>available/0</c> to server + <c>ch3</c>, which returns the number of available channels (when + trying out the example, change in a copy of the original + directory, so that the first versions are still available):</p> <code type="none"> -module(ch3). -behaviour(gen_server). @@ -465,9 +471,9 @@ handle_cast({free, Ch}, Chs) -> {mod, {ch_app,[]}} ]}.</code> <p>To upgrade <c>ch_app</c> from <c>"1"</c> to <c>"2"</c> (and - to downgrade from <c>"2"</c> to <c>"1"</c>), we simply need to + to downgrade from <c>"2"</c> to <c>"1"</c>), you only need to load the new (old) version of the <c>ch3</c> callback module. - We create the application upgrade file <c>ch_app.appup</c> in + Create the application upgrade file <c>ch_app.appup</c> in the <c>ebin</c> directory:</p> <code type="none"> {"2", @@ -480,25 +486,25 @@ handle_cast({free, Ch}, Chs) -> <marker id="relup"></marker> <title>Release Upgrade File</title> <p>To define how to upgrade/downgrade between the new version and - previous versions of a release, we create a <em>release upgrade file</em>, or in short <c>relup</c> file.</p> + previous versions of a release, a <em>release upgrade file</em>, + or in short <c>relup</c> file, is to be created.</p> <p>This file does not need to be created manually, it can be generated by <c>systools:make_relup/3,4</c>. The relevant versions - of the <c>.rel</c> file, <c>.app</c> files and <c>.appup</c> files - are used as input. It is deducted which applications should be - added and deleted, and which applications that need to be upgraded - and/or downgraded. The instructions for this is fetched from + of the <c>.rel</c> file, <c>.app</c> files, and <c>.appup</c> files + are used as input. It is deducted which applications are to be + added and deleted, and which applications that must be upgraded + and/or downgraded. The instructions for this are fetched from the <c>.appup</c> files and transformed into a single list of low-level instructions in the right order.</p> <p>If the <c>relup</c> file is relatively simple, it can be created - manually. Remember that it should only contain low-level - instructions.</p> - <p>The syntax and contents of the release upgrade file are - described in detail in <c>relup(4)</c>.</p> - <p>Example, continued from the previous section. We have a new - version "2" of <c>ch_app</c> and an <c>.appup</c> file. We also - need a new version of the <c>.rel</c> file. This time the file is - called <c>ch_rel-2.rel</c> and the release version string is - changed changed from "A" to "B":</p> + manually. It it only to contain low-level instructions.</p> + <p>For details about the syntax and contents of the release upgrade + file, see the <c>relup(4)</c> manual page in SASL.</p> + <p><em>Example, continued from the previous section:</em> You have a + new version "2" of <c>ch_app</c> and an <c>.appup</c> file. A new + version of the <c>.rel</c> file is also needed. This time the file + is called <c>ch_rel-2.rel</c> and the release version string is + changed from "A" to "B":</p> <code type="none"> {release, {"ch_rel", "B"}, @@ -512,13 +518,13 @@ handle_cast({free, Ch}, Chs) -> <pre> 1> <input>systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"]).</input> ok</pre> - <p>This will generate a <c>relup</c> file with instructions for + <p>This generates a <c>relup</c> file with instructions for how to upgrade from version "A" ("ch_rel-1") to version "B" ("ch_rel-2") and how to downgrade from version "B" to version "A".</p> - <p>Note that both the old and new versions of the <c>.app</c> and - <c>.rel</c> files must be in the code path, as well as - the <c>.appup</c> and (new) <c>.beam</c> files. It is possible - to extend the code path by using the option <c>path</c>:</p> + <p>Both the old and new versions of the <c>.app</c> and + <c>.rel</c> files must be in the code path, as well as the + <c>.appup</c> and (new) <c>.beam</c> files. The code path can be + extended by using the option <c>path</c>:</p> <pre> 1> <input>systools:make_relup("ch_rel-2", ["ch_rel-1"], ["ch_rel-1"],</input> <input>[{path,["../ch_rel-1",</input> @@ -529,31 +535,34 @@ ok</pre> <section> <marker id="rel_handler"></marker> <title>Installing a Release</title> - <p>When we have made a new version of a release, a release package + <p>When you have made a new version of a release, a release package can be created with this new version and transferred to the target environment.</p> - <p>To install the new version of the release in run-time, + <p>To install the new version of the release in runtime, the <em>release handler</em> is used. This is a process belonging - to the SASL application, that handles unpacking, installation, - and removal of release packages. It is interfaced through - the module <c>release_handler</c>, which is described in detail in - <c>release_handler(3)</c>.</p> - <p>Assuming there is a target system up and running with + to the SASL application, which handles unpacking, installation, + and removal of release packages. It is communicated through + the <c>release_handler</c> module. For details, see the + <c>release_handler(3)</c> manual page in SASL.</p> + <p>Assuming there is an operational target system with installation root directory <c>$ROOT</c>, the release package with - the new version of the release should be copied to + the new version of the release is to be copied to <c>$ROOT/releases</c>.</p> - <p>The first action is to <em>unpack</em> the release package, - the files are then extracted from the package:</p> + <p>First, <em>unpack</em> the release package. + The files are then extracted from the package:</p> <code type="none"> release_handler:unpack_release(ReleaseName) => {ok, Vsn}</code> - <p><c>ReleaseName</c> is the name of the release package except - the <c>.tar.gz</c> extension. <c>Vsn</c> is the version of - the unpacked release, as defined in its <c>.rel</c> file.</p> - <p>A directory <c>$ROOT/lib/releases/Vsn</c> will be created, where + <list type="bulleted"> + <item><c>ReleaseName</c> is the name of the release package except + the <c>.tar.gz</c> extension.</item> + <item><c>Vsn</c> is the version of the unpacked release, as + defined in its <c>.rel</c> file.</item> + </list> + <p>A directory <c>$ROOT/lib/releases/Vsn</c> is created, where the <c>.rel</c> file, the boot script <c>start.boot</c>, - the system configuration file <c>sys.config</c> and <c>relup</c> + the system configuration file <c>sys.config</c>, and <c>relup</c> are placed. For applications with new version numbers, - the application directories will be placed under <c>$ROOT/lib</c>. + the application directories are placed under <c>$ROOT/lib</c>. Unchanged applications are not affected.</p> <p>An unpacked release can be <em>installed</em>. The release handler then evaluates the instructions in <c>relup</c>, step by @@ -563,11 +572,11 @@ release_handler:install_release(Vsn) => {ok, FromVsn, []}</code> <p>If an error occurs during the installation, the system is rebooted using the old version of the release. If installation succeeds, the system is afterwards using the new version of - the release, but should anything happen and the system is - rebooted, it would start using the previous version again. To be - made the default version, the newly installed release must be made - <em>permanent</em>, which means the previous version becomes - <em>old</em>:</p> + the release, but if anything happens and the system is + rebooted, it starts using the previous version again.</p> + <p>To be made the default version, the newly installed release + must be made <em>permanent</em>, which means the previous + version becomes <em>old</em>:</p> <code type="none"> release_handler:make_permanent(Vsn) => ok</code> <p>The system keeps information about which versions are old and @@ -579,41 +588,44 @@ release_handler:make_permanent(Vsn) => ok</code> release_handler:install_release(FromVsn) => {ok, Vsn, []}</code> <p>An installed, but not permanent, release can be <em>removed</em>. Information about the release is then deleted from - <c>$ROOT/releases/RELEASES</c> and the release specific code, - that is the new application directories and + <c>$ROOT/releases/RELEASES</c> and the release-specific code, + that is, the new application directories and the <c>$ROOT/releases/Vsn</c> directory, are removed.</p> <code type="none"> release_handler:remove_release(Vsn) => ok</code> - <p>Example, continued from the previous sections:</p> - <p>1) Create a target system as described in <em>System Principles</em> of the first version <c>"A"</c> of <c>ch_rel</c> - from - the <seealso marker="release_structure#ch_rel">Releases</seealso> - chapter. This time <c>sys.config</c> must be included in - the release package. If no configuration is needed, the file - should contain the empty list:</p> - <code type="none"> + + <section> + <title>Example (continued from the previous sections)</title> + <p><em>Step 1)</em> Create a target system as described in + System Principles of the first version <c>"A"</c> + of <c>ch_rel</c> from + <seealso marker="release_structure#ch_rel">Releases</seealso>. + This time <c>sys.config</c> must be included in the release package. + If no configuration is needed, the file is to contain the empty + list:</p> + <code type="none"> [].</code> - <p>2) Start the system as a simple target system. Note that in - reality, it should be started as an embedded system. However, - using <c>erl</c> with the correct boot script and config file is - enough for illustration purposes:</p> - <pre> + <p><em>Step 2)</em> Start the system as a simple target system. In + reality, it is to be started as an embedded system. However, using + <c>erl</c> with the correct boot script and config file is enough for + illustration purposes:</p> + <pre> % <input>cd $ROOT</input> % <input>bin/erl -boot $ROOT/releases/A/start -config $ROOT/releases/A/sys</input> ...</pre> <p><c>$ROOT</c> is the installation directory of the target system.</p> - <p>3) In another Erlang shell, generate start scripts and create a - release package for the new version <c>"B"</c>. Remember to - include (a possible updated) <c>sys.config</c> and - the <c>relup</c> file, see <seealso marker="#relup">Release Upgrade File</seealso> above.</p> - <pre> + <p><em>Step 3)</em> In another Erlang shell, generate start scripts and + create a release package for the new version <c>"B"</c>. Remember to + include (a possible updated) <c>sys.config</c> and the <c>relup</c> file, + see <seealso marker="#relup">Release Upgrade File</seealso>.</p> + <pre> 1> <input>systools:make_script("ch_rel-2").</input> ok 2> <input>systools:make_tar("ch_rel-2").</input> ok</pre> - <p>The new release package now contains version "2" of <c>ch_app</c> - and the <c>relup</c> file as well:</p> - <code type="none"> + <p>The new release package now also contains version "2" of <c>ch_app</c> + and the <c>relup</c> file:</p> + <code type="none"> % tar tf ch_rel-2.tar lib/kernel-2.9/ebin/kernel.app lib/kernel-2.9/ebin/application.beam @@ -633,28 +645,30 @@ releases/B/relup releases/B/sys.config releases/B/ch_rel-2.rel releases/ch_rel-2.rel</code> - <p>4) Copy the release package <c>ch_rel-2.tar.gz</c> to - the <c>$ROOT/releases</c> directory.</p> - <p>5) In the running target system, unpack the release package:</p> - <pre> + <p><em>Step 4)</em> Copy the release package <c>ch_rel-2.tar.gz</c> + to the <c>$ROOT/releases</c> directory.</p> + <p><em>Step 5)</em> In the running target system, unpack the release + package:</p> + <pre> 1> <input>release_handler:unpack_release("ch_rel-2").</input> {ok,"B"}</pre> <p>The new application version <c>ch_app-2</c> is installed under <c>$ROOT/lib</c> next to <c>ch_app-1</c>. The <c>kernel</c>, - <c>stdlib</c> and <c>sasl</c> directories are not affected, as + <c>stdlib</c>, and <c>sasl</c> directories are not affected, as they have not changed.</p> <p>Under <c>$ROOT/releases</c>, a new directory <c>B</c> is created, containing <c>ch_rel-2.rel</c>, <c>start.boot</c>, - <c>sys.config</c> and <c>relup</c>.</p> - <p>6) Check if the function <c>ch3:available/0</c> is available:</p> - <pre> + <c>sys.config</c>, and <c>relup</c>.</p> + <p><em>Step 6)</em> Check if the function <c>ch3:available/0</c> is + available:</p> + <pre> 2> <input>ch3:available().</input> ** exception error: undefined function ch3:available/0</pre> - <p>7) Install the new release. The instructions in + <p><em>Step 7)</em> Install the new release. The instructions in <c>$ROOT/releases/B/relup</c> are executed one by one, resulting in the new version of <c>ch3</c> being loaded. The function <c>ch3:available/0</c> is now available:</p> - <pre> + <pre> 3> <input>release_handler:install_release("B").</input> {ok,"A",[]} 4> <input>ch3:available().</input> @@ -663,15 +677,16 @@ releases/ch_rel-2.rel</code> ".../lib/ch_app-2/ebin/ch3.beam" 6> <input>code:which(ch_sup).</input> ".../lib/ch_app-1/ebin/ch_sup.beam"</pre> - <p>Note that processes in <c>ch_app</c> for which code have not - been updated, for example the supervisor, are still evaluating + <p>Processes in <c>ch_app</c> for which code have not + been updated, for example, the supervisor, are still evaluating code from <c>ch_app-1</c>.</p> - <p>8) If the target system is now rebooted, it will use version "A" - again. The "B" version must be made permanent, in order to be + <p><em>Step 8)</em> If the target system is now rebooted, it uses + version "A" again. The "B" version must be made permanent, to be used when the system is rebooted.</p> - <pre> + <pre> 7> <input>release_handler:make_permanent("B").</input> ok</pre> + </section> </section> <section> @@ -681,37 +696,40 @@ ok</pre> specifications are automatically updated for all loaded applications.</p> <note> - <p>The information about the new application specifications are + <p>The information about the new application specifications is fetched from the boot script included in the release package. - It is therefore important that the boot script is generated from + Thus, it is important that the boot script is generated from the same <c>.rel</c> file as is used to build the release package itself.</p> </note> <p>Specifically, the application configuration parameters are automatically updated according to (in increasing priority order):</p> - <list type="ordered"> + <list type="bulleted"> <item>The data in the boot script, fetched from the new application resource file <c>App.app</c></item> <item>The new <c>sys.config</c></item> - <item>Command line arguments <c>-App Par Val</c></item> + <item>Command-line arguments <c>-App Par Val</c></item> </list> <p>This means that parameter values set in the other system - configuration files, as well as values set using - <c>application:set_env/3</c>, are disregarded.</p> + configuration files and values set using + <c>application:set_env/3</c> are disregarded.</p> <p>When an installed release is made permanent, the system process <c>init</c> is set to point out the new <c>sys.config</c>.</p> - <p>After the installation, the application controller will compare + <p>After the installation, the application controller compares the old and new configuration parameters for all running applications and call the callback function:</p> <code type="none"> Module:config_change(Changed, New, Removed)</code> - <p><c>Module</c> is the application callback module as defined by - the <c>mod</c> key in the <c>.app</c> file. <c>Changed</c> and - <c>New</c> are lists of <c>{Par,Val}</c> for all changed and - added configuration parameters, respectively. <c>Removed</c> is - a list of all parameters <c>Par</c> that have been removed.</p> - <p>The function is optional and may be omitted when implementing an + <list type="bulleted"> + <item><c>Module</c> is the application callback module as defined + by the <c>mod</c> key in the <c>.app</c> file.</item> + <item><c>Changed</c> and <c>New</c> are lists of <c>{Par,Val}</c> for + all changed and added configuration parameters, respectively.</item> + <item><c>Removed</c> is a list of all parameters <c>Par</c> that have + been removed.</item> + </list> + <p>The function is optional and can be omitted when implementing an application callback module.</p> </section> </chapter> diff --git a/system/doc/design_principles/release_structure.xml b/system/doc/design_principles/release_structure.xml index cec33f42e3..aa04f5e6a3 100644 --- a/system/doc/design_principles/release_structure.xml +++ b/system/doc/design_principles/release_structure.xml @@ -28,21 +28,23 @@ <rev></rev> <file>release_structure.xml</file> </header> - <p>This chapter should be read in conjuction with <c>rel(4)</c>, - <c>systools(3)</c> and <c>script(4)</c>.</p> + <marker id="releases section"></marker> + <p>This section is to be read with the <c>rel(4)</c>, <c>systools(3)</c>, + and <c>script(4)</c> manual pages in SASL.</p> <section> <title>Release Concept</title> - <p>When we have written one or more applications, we might want to - create a complete system consisting of these applications and a + <p>When you have written one or more applications, you might want + to create a complete system with these applications and a subset of the Erlang/OTP applications. This is called a <em>release</em>.</p> - <p>To do this, we create a <seealso marker="#res_file">release resource file</seealso> which defines which applications - are included in the release.</p> + <p>To do this, create a + <seealso marker="#res_file">release resource file</seealso> that + defines which applications are included in the release.</p> <p>The release resource file is used to generate <seealso marker="#boot">boot scripts</seealso> and <seealso marker="#pack">release packages</seealso>. A system - which is transfered to and installed at another site is called a + that is transferred to and installed at another site is called a <em>target system</em>. How to use a release package to create a target system is described in System Principles.</p> </section> @@ -50,29 +52,30 @@ <section> <marker id="res_file"></marker> <title>Release Resource File</title> - <p>To define a release, we create a <em>release resource file</em>, - or in short <c>.rel</c> file, where we specify the name and - version of the release, which ERTS version it is based on, and - which applications it consists of:</p> + <p>To define a release, create a <em>release resource file</em>, + or in short a <c>.rel</c> file. In the file, specify the name and + version of the release, which ERTS version it is based on, + and which applications it consists of:</p> <code type="none"> {release, {Name,Vsn}, {erts, EVsn}, [{Application1, AppVsn1}, ... {ApplicationN, AppVsnN}]}.</code> + <p><c>Name</c>, <c>Vsn</c>, <c>EVsn</c>, and <c>AppVsn</c> are + strings.</p> <p>The file must be named <c>Rel.rel</c>, where <c>Rel</c> is a unique name.</p> - <p><c>Name</c>, <c>Vsn</c> and <c>EVsn</c> are strings.</p> - <p>Each <c>Application</c> (atom) and <c>AppVsn</c> (string) is + <p>Each <c>Application</c> (atom) and <c>AppVsn</c> is the name and version of an application included in the release. - Note that the minimal release based on Erlang/OTP consists of - the <c>kernel</c> and <c>stdlib</c> applications, so these + The minimal release based on Erlang/OTP consists of + the Kernel and STDLIB applications, so these applications must be included in the list.</p> <p>If the release is to be upgraded, it must also include - the <c>sasl</c> application.</p> + the SASL application.</p> <marker id="ch_rel"></marker> - <p>Example: We want to make a release of <c>ch_app</c> from - the <seealso marker="applications#ch_app">Applications</seealso> - chapter. It has the following <c>.app</c> file:</p> + <p><em>Example: </em> A release of <c>ch_app</c> from + <seealso marker="applications#ch_app">Applications</seealso> + has the following <c>.app</c> file:</p> <code type="none"> {application, ch_app, [{description, "Channel allocator"}, @@ -83,8 +86,8 @@ {mod, {ch_app,[]}} ]}.</code> <p>The <c>.rel</c> file must also contain <c>kernel</c>, - <c>stdlib</c> and <c>sasl</c>, since these applications are - required by <c>ch_app</c>. We call the file <c>ch_rel-1.rel</c>:</p> + <c>stdlib</c>, and <c>sasl</c>, as these applications are required + by <c>ch_app</c>. The file is called <c>ch_rel-1.rel</c>:</p> <code type="none"> {release, {"ch_rel", "A"}, @@ -99,24 +102,28 @@ <section> <marker id="boot"></marker> <title>Generating Boot Scripts</title> - <p>There are tools in the SASL module <c>systools</c> available to - build and check releases. The functions read the <c>.rel</c> and + <p><c>systools</c> in the SASL application includes tools to + build and check releases. The functions read the <c>rel</c> and <c>.app</c> files and performs syntax and dependency checks. - The function <c>systools:make_script/1,2</c> is used to generate - a boot script (see System Principles).</p> + The <c>systools:make_script/1,2</c> function is used to generate + a boot script (see System Principles):</p> <pre> 1> <input>systools:make_script("ch_rel-1", [local]).</input> ok</pre> - <p>This creates a boot script, both the readable version - <c>ch_rel-1.script</c> and the binary version used by the runtime - system, <c>ch_rel-1.boot</c>. <c>"ch_rel-1"</c> is the name of - the <c>.rel</c> file, minus the extension. <c>local</c> is an - option that means that the directories where the applications are - found are used in the boot script, instead of <c>$ROOT/lib</c>. - (<c>$ROOT</c> is the root directory of the installed release.) - This is a useful way to test a generated boot script locally.</p> + <p>This creates a boot script, both the readable version, + <c>ch_rel-1.script</c>, and the binary version, <c>ch_rel-1.boot</c>, + used by the runtime system.</p> + <list type="bulleted"> + <item><c>"ch_rel-1"</c> is the name of the <c>.rel</c> file, + minus the extension.</item> + <item><c>local</c> is an option that means that the directories + where the applications are found are used in the boot script, + instead of <c>$ROOT/lib</c> (<c>$ROOT</c> is the root directory + of the installed release).</item> + </list> + <p> This is a useful way to test a generated boot script locally.</p> <p>When starting Erlang/OTP using the boot script, all applications - from the <c>.rel</c> file are automatically loaded and started:</p> + from the <c>.rel</c> file are automatically loaded and started:</p> <pre> % <input>erl -boot ch_rel-1</input> Erlang (BEAM) emulator version 5.3 @@ -147,18 +154,24 @@ Eshell V5.3 (abort with ^G) <section> <marker id="pack"></marker> <title>Creating a Release Package</title> - <p>There is a function <c>systools:make_tar/1,2</c> which takes - a <c>.rel</c> file as input and creates a zipped tar-file with - the code for the specified applications, a <em>release package</em>.</p> + <p>The <c>systools:make_tar/1,2</c> function takes a <c>.rel</c> file + as input and creates a zipped tar file with the code for the specified + applications, a <em>release package</em>:</p> <pre> 1> <input>systools:make_script("ch_rel-1").</input> ok 2> <input>systools:make_tar("ch_rel-1").</input> ok</pre> - <p>The release package by default contains the <c>.app</c> files and - object code for all applications, structured according to - the <seealso marker="applications#app_dir">application directory structure</seealso>, the binary boot script renamed to - <c>start.boot</c>, and the <c>.rel</c> file.</p> + <p>The release package by default contains:</p> + <list type="bulleted"> + <item>The <c>.app</c> files</item> + <item>The <c>.rel</c> file</item> + <item>The object code for all applications, structured according + to the + <seealso marker="applications#app_dir">application directory + structure</seealso></item> + <item>The binary boot script renamed to <c>start.boot</c></item> + </list> <pre> % <input>tar tf ch_rel-1.tar</input> lib/kernel-2.9/ebin/kernel.app @@ -177,40 +190,39 @@ lib/ch_app-1/ebin/ch3.beam releases/A/start.boot releases/A/ch_rel-1.rel releases/ch_rel-1.rel</pre> - <p>Note that a new boot script was generated, without + <p>A new boot script was generated, without the <c>local</c> option set, before the release package was made. In the release package, all application directories are placed - under <c>lib</c>. Also, we do not know where the release package - will be installed, so we do not want any hardcoded absolute paths - in the boot script here.</p> + under <c>lib</c>. You do not know where the release package + will be installed, so no hard-coded absolute paths are allowed.</p> <p>The release resource file <c>mysystem.rel</c> is duplicated in the tar file. Originally, this file was only stored in - the <c>releases</c> directory in order to make it possible for + the <c>releases</c> directory to make it possible for the <c>release_handler</c> to extract this file separately. After unpacking the tar file, <c>release_handler</c> would automatically copy the file to <c>releases/FIRST</c>. However, sometimes the tar file is - unpacked without involving the <c>release_handler</c> (e.g. when - unpacking the first target system) and therefore the file is now - instead duplicated in the tar file so no manual copying is - necessary.</p> + unpacked without involving the <c>release_handler</c> (for + example, when unpacking the first target system) and the file + is therefore now instead duplicated in the tar file so no manual + copying is necessary.</p> <p>If a <c>relup</c> file and/or a system configuration file called - <c>sys.config</c> is found, these files are included in - the release package as well. See + <c>sys.config</c> is found, these files are also included in + the release package. See <seealso marker="release_handling#req">Release Handling</seealso>.</p> <p>Options can be set to make the release package include source code and the ERTS binary as well.</p> - <p>Refer to System Principles for how to install the first target - system, using a release package, and to - <seealso marker="release_handling">Release Handling</seealso> for - how to install a new release package in an existing system.</p> + <p>For information on how to install the first target system, using + a release package, see System Principles. For information + on how to install a new release package in an existing system, see + <seealso marker="release_handling">Release Handling</seealso>.</p> </section> <section> <marker id="reldir"></marker> <title>Directory Structure</title> - <p>Directory structure for the code installed by the release handler - from a release package:</p> + <p>The directory structure for the code installed by the release handler + from a release package is as follows:</p> <code type="none"> $ROOT/lib/App1-AVsn1/ebin /priv @@ -222,24 +234,18 @@ $ROOT/lib/App1-AVsn1/ebin /erts-EVsn/bin /releases/Vsn /bin</code> - <taglist> - <tag><c>lib</c></tag> - <item>Application directories.</item> - <tag><c>erts-EVsn/bin</c></tag> - <item>Erlang runtime system executables.</item> - <tag><c>releases/Vsn</c></tag> - <item><c>.rel</c> file and boot script <c>start.boot</c>. <br></br> - - If present in the release package, <br></br> -<c>relup</c> and/or <c>sys.config</c>.</item> - <tag><c>bin</c></tag> - <item>Top level Erlang runtime system executables.</item> - </taglist> - <p>Applications are not required to be located under the - <c>$ROOT/lib</c> directory. Accordingly, several installation - directories may exist which contain different parts of a - system. For example, the previous example could be extended as - follows:</p> + <list type="bulleted"> + <item><c>lib</c> - Application directories</item> + <item><c>erts-EVsn/bin</c> - Erlang runtime system executables</item> + <item><c>releases/Vsn</c> - <c>.rel</c> file and boot script + <c>start.boot</c>; if present in the release package, <c>relup</c> + and/or <c>sys.config</c></item> + <item><c>bin</c> - Top-level Erlang runtime system executables</item> + </list> + <p>Applications are not required to be located under directory + <c>$ROOT/lib</c>. Several installation directories, which contain + different parts of a system, can thus exist. + For example, the previous example can be extended as follows:</p> <pre> $SECOND_ROOT/.../SApp1-SAVsn1/ebin /priv @@ -256,24 +262,24 @@ $THIRD_ROOT/TApp1-TAVsn1/ebin ... /TAppN-TAVsnN/ebin /priv</pre> - <p>The <c>$SECOND_ROOT</c> and <c>$THIRD_ROOT</c> are introduced as + <p><c>$SECOND_ROOT</c> and <c>$THIRD_ROOT</c> are introduced as <c>variables</c> in the call to the <c>systools:make_script/2</c> function.</p> <section> <title>Disk-Less and/or Read-Only Clients</title> - <p>If a complete system consists of some disk-less and/or - read-only client nodes, a <c>clients</c> directory should be - added to the <c>$ROOT</c> directory. By a read-only node we - mean a node with a read-only file system.</p> - <p>The <c>clients</c> directory should have one sub-directory + <p>If a complete system consists of disk-less and/or + read-only client nodes, a <c>clients</c> directory is to be + added to the <c>$ROOT</c> directory. A read-only node is + a node with a read-only file system.</p> + <p>The <c>clients</c> directory is to have one subdirectory per supported client node. The name of each client directory - should be the name of the corresponding client node. As a - minimum, each client directory should contain the <c>bin</c> and - <c>releases</c> sub-directories. These directories are used to + is to be the name of the corresponding client node. As a + minimum, each client directory is to contain the <c>bin</c> and + <c>releases</c> subdirectories. These directories are used to store information about installed releases and to appoint the - current release to the client. Accordingly, the <c>$ROOT</c> - directory contains the following:</p> + current release to the client. The <c>$ROOT</c> + directory thus contains the following:</p> <code type="none"> $ROOT/... /clients/ClientName1/bin @@ -283,14 +289,14 @@ $ROOT/... ... /ClientNameN/bin /releases/Vsn</code> - <p>This structure should be used if all clients are running + <p>This structure is to be used if all clients are running the same type of Erlang machine. If there are clients running different types of Erlang machines, or on different operating - systems, the <c>clients</c> directory could be divided into one - sub-directory per type of Erlang machine. Alternatively, you - can set up one <c>$ROOT</c> per type of machine. For each + systems, the <c>clients</c> directory can be divided into one + subdirectory per type of Erlang machine. Alternatively, one + <c>$ROOT</c> can be set up per type of machine. For each type, some of the directories specified for the <c>$ROOT</c> - directory should be included:</p> + directory are to be included:</p> <code type="none"> $ROOT/... /clients/Type1/lib diff --git a/system/doc/design_principles/spec_proc.xml b/system/doc/design_principles/spec_proc.xml index e849388a38..aceb5ba99e 100644 --- a/system/doc/design_principles/spec_proc.xml +++ b/system/doc/design_principles/spec_proc.xml @@ -21,30 +21,31 @@ </legalnotice> - <title>Sys and Proc_Lib</title> + <title>sys and proc_lib</title> <prepared></prepared> <docno></docno> <date></date> <rev></rev> <file>spec_proc.xml</file> </header> - <p>The module <c>sys</c> contains functions for simple debugging of - processes implemented using behaviours.</p> - <p>There are also functions that, together with functions in - the module <c>proc_lib</c>, can be used to implement a - <em>special process</em>, a process which comply to the OTP design - principles without making use of a standard behaviour. They can - also be used to implement user defined (non-standard) behaviours.</p> - <p>Both <c>sys</c> and <c>proc_lib</c> belong to the STDLIB - application.</p> + <marker id="sys and proc_lib"></marker> + <p>The <c>sys</c> module has functions for simple debugging of + processes implemented using behaviours. It also has functions that, + together with functions in the <c>proc_lib</c> module, can be used + to implement a <em>special process</em> that complies to the OTP + design principles without using a standard behaviour. These + functions can also be used to implement user-defined (non-standard) + behaviours.</p> + <p>Both <c>sys</c> and <c>proc_lib</c> belong to the STDLIB + application.</p> <section> <title>Simple Debugging</title> - <p>The module <c>sys</c> contains some functions for simple debugging - of processes implemented using behaviours. We use the - <c>code_lock</c> example from - the <seealso marker="fsm#ex">gen_fsm</seealso> chapter to - illustrate this:</p> + <p>The <c>sys</c> module has functions for simple debugging of + processes implemented using behaviours. The <c>code_lock</c> + example from + <seealso marker="fsm#ex">gen_fsm Behaviour</seealso> + is used to illustrate this:</p> <pre> % <input>erl</input> Erlang (BEAM) emulator version 5.2.3.6 [hipe] [threads:0] @@ -102,16 +103,18 @@ ok <section> <title>Special Processes</title> - <p>This section describes how to write a process which comply to - the OTP design principles, without making use of a standard - behaviour. Such a process should:</p> + <p>This section describes how to write a process that complies to + the OTP design principles, without using a standard behaviour. + Such a process is to:</p> <list type="bulleted"> - <item>be started in a way that makes the process fit into a - supervision tree,</item> - <item>support the <c>sys</c> <seealso marker="#debug">debug facilities</seealso>, and</item> - <item>take care of <seealso marker="#msg">system messages</seealso>.</item> + <item>Be started in a way that makes the process fit into a + supervision tree</item> + <item>Support the <c>sys</c> + <seealso marker="#debug">debug facilities</seealso></item> + <item>Take care of + <seealso marker="#msg">system messages</seealso>.</item> </list> - <p>System messages are messages with special meaning, used in + <p>System messages are messages with a special meaning, used in the supervision tree. Typical system messages are requests for trace output, and requests to suspend or resume process execution (used during release handling). Processes implemented using @@ -120,9 +123,9 @@ ok <section> <title>Example</title> <p>The simple server from - the <seealso marker="des_princ#ch1">Overview</seealso> chapter, - implemented using <c>sys</c> and <c>proc_lib</c> so it fits into - a supervision tree:</p> + <seealso marker="des_princ#ch1">Overview</seealso>, + implemented using <c>sys</c> and <c>proc_lib</c> so it fits into a + supervision tree:</p> <marker id="ex"></marker> <pre> -module(ch4). @@ -190,8 +193,8 @@ system_replace_state(StateFun, Chs) -> write_debug(Dev, Event, Name) -> io:format(Dev, "~p event = ~p~n", [Name, Event]).</pre> - <p>Example on how the simple debugging functions in <c>sys</c> can - be used for <c>ch4</c> as well:</p> + <p>Example on how the simple debugging functions in the <c>sys</c> + module can also be used for <c>ch4</c>:</p> <pre> % <input>erl</input> Erlang (BEAM) emulator version 5.2.3.6 [hipe] [threads:0] @@ -230,31 +233,32 @@ ok <section> <title>Starting the Process</title> - <p>A function in the <c>proc_lib</c> module should be used to - start the process. There are several possible functions, for - example <c>spawn_link/3,4</c> for asynchronous start and + <p>A function in the <c>proc_lib</c> module is to be used to + start the process. Several functions are available, for + example, <c>spawn_link/3,4</c> for asynchronous start and <c>start_link/3,4,5</c> for synchronous start.</p> - <p>A process started using one of these functions will store - information that is needed for a process in a supervision tree, - for example about the ancestors and initial call.</p> - <p>Also, if the process terminates with another reason than - <c>normal</c> or <c>shutdown</c>, a crash report (see SASL - User's Guide) is generated.</p> - <p>In the example, synchronous start is used. The process is - started by calling <c>ch4:start_link()</c>:</p> + <p>A process started using one of these functions stores + information (for example, about the ancestors and initial call) + that is needed for a process in a supervision tree.</p> + <p>If the process terminates with another reason than + <c>normal</c> or <c>shutdown</c>, a crash report is generated. + For more information about the crash report, see the SASL + User's Guide.</p> + <p>In the example, synchronous start is used. The process + starts by calling <c>ch4:start_link()</c>:</p> <code type="none"> start_link() -> proc_lib:start_link(ch4, init, [self()]).</code> <p><c>ch4:start_link</c> calls the function <c>proc_lib:start_link</c>. This function takes a module name, - a function name and an argument list as arguments and spawns + a function name, and an argument list as arguments, spawns, and links to a new process. The new process starts by executing - the given function, in this case <c>ch4:init(Pid)</c>, where - <c>Pid</c> is the pid (<c>self()</c>) of the first process, that - is the parent process.</p> - <p>In <c>init</c>, all initialization including name registration - is done. The new process must also acknowledge that it has been - started to the parent:</p> + the given function, here <c>ch4:init(Pid)</c>, where + <c>Pid</c> is the pid (<c>self()</c>) of the first process, + which is the parent process.</p> + <p>All initialization, including name registration, is done in + <c>init</c>. The new process must also acknowledge that it has + been started to the parent:</p> <code type="none"> init(Parent) -> ... @@ -267,8 +271,8 @@ init(Parent) -> <section> <marker id="debug"></marker> <title>Debugging</title> - <p>To support the debug facilites in <c>sys</c>, we need a - <em>debug structure</em>, a term <c>Deb</c> which is + <p>To support the debug facilites in <c>sys</c>, a + <em>debug structure</em> is needed. The <c>Deb</c> term is initialized using <c>sys:debug_options/1</c>:</p> <code type="none"> init(Parent) -> @@ -278,50 +282,41 @@ init(Parent) -> loop(Chs, Parent, Deb).</code> <p><c>sys:debug_options/1</c> takes a list of options as argument. Here the list is empty, which means no debugging is enabled - initially. See <c>sys(3)</c> for information about possible - options.</p> - <p>Then for each <em>system event</em> that we want to be logged - or traced, the following function should be called.</p> + initially. For information about the possible options, see the + <c>sys(3)</c> manual page in STDLIB.</p> + <p>Then, for each <em>system event</em> to be logged + or traced, the following function is to be called.</p> <code type="none"> sys:handle_debug(Deb, Func, Info, Event) => Deb1</code> +<p>Here:</p> <list type="bulleted"> - <item> - <p><c>Deb</c> is the debug structure.</p> - </item> - <item> - <p><c>Func</c> is a fun specifying - a (user defined) function used to format + <item><c>Deb</c> is the debug structure.</item> + <item><c>Func</c> is a fun specifying + a (user-defined) function used to format trace output. For each system event, the format function is - called as <c>Func(Dev, Event, Info)</c>, where:</p> + called as <c>Func(Dev, Event, Info)</c>, where: <list type="bulleted"> - <item> - <p><c>Dev</c> is the IO device to which the output should - be printed. See <c>io(3)</c>.</p> - </item> - <item> - <p><c>Event</c> and <c>Info</c> are passed as-is from - <c>handle_debug</c>.</p> - </item> + <item><c>Dev</c> is the I/O device to which the output is to + be printed. See the <c>io(3)</c> manual page in + STDLIB.</item> + <item><c>Event</c> and <c>Info</c> are passed as is from + <c>handle_debug</c>.</item> </list> </item> - <item> - <p><c>Info</c> is used to pass additional information to - <c>Func</c>, it can be any term and is passed as-is.</p> - </item> - <item> - <p><c>Event</c> is the system event. It is up to the user to - define what a system event is and how it should be - represented, but typically at least incoming and outgoing + <item><c>Info</c> is used to pass more information to + <c>Func</c>. It can be any term and is passed as is.</item> + <item><c>Event</c> is the system event. It is up to the user to + define what a system event is and how it is to be + represented. Typically at least incoming and outgoing messages are considered system events and represented by the tuples <c>{in,Msg[,From]}</c> and <c>{out,Msg,To}</c>, - respectively.</p> - </item> + respectively.</item> </list> <p><c>handle_debug</c> returns an updated debug structure <c>Deb1</c>.</p> <p>In the example, <c>handle_debug</c> is called for each incoming and outgoing message. The format function <c>Func</c> is - the function <c>ch4:write_debug/3</c> which prints the message + the function <c>ch4:write_debug/3</c>, which prints the message using <c>io:format/3</c>.</p> <code type="none"> loop(Chs, Parent, Deb) -> @@ -354,22 +349,22 @@ write_debug(Dev, Event, Name) -> {system, From, Request}</code> <p>The content and meaning of these messages do not need to be interpreted by the process. Instead the following function - should be called:</p> + is to be called:</p> <code type="none"> sys:handle_system_msg(Request, From, Parent, Module, Deb, State)</code> - <p>This function does not return. It will handle the system - message and then call:</p> + <p>This function does not return. It handles the system + message and then either calls the following if process execution is + to continue:</p> <code type="none"> Module:system_continue(Parent, Deb, State)</code> - <p>if process execution should continue, or:</p> + <p>Or calls the following if the process is to terminate:</p> <code type="none"> Module:system_terminate(Reason, Parent, Deb, State)</code> - <p>if the process should terminate. Note that a process in a - supervision tree is expected to terminate with the same reason as - its parent.</p> + <p>A process in a supervision tree is expected to terminate with + the same reason as its parent.</p> <list type="bulleted"> - <item><c>Request</c> and <c>From</c> should be passed as-is from - the system message to the call to <c>handle_system_msg</c>.</item> + <item><c>Request</c> and <c>From</c> are to be passed as is from + the system message to the call to <c>handle_system_msg</c>.</item> <item><c>Parent</c> is the pid of the parent.</item> <item><c>Module</c> is the name of the module.</item> <item><c>Deb</c> is the debug structure.</item> @@ -377,10 +372,12 @@ Module:system_terminate(Reason, Parent, Deb, State)</code> is passed to <c>system_continue</c>/<c>system_terminate</c>/ <c>system_get_state</c>/<c>system_replace_state</c>.</item> </list> - <p>If the process should return its state <c>handle_system_msg</c> will call:</p> + <p>If the process is to return its state, <c>handle_system_msg</c> + calls:</p> <code type="none"> Module:system_get_state(State)</code> - <p>or if the process should replace its state using the fun <c>StateFun</c>:</p> + <p>If the process is to replace its state using the fun <c>StateFun</c>, + <c>handle_system_msg</c> calls:</p> <code type="none"> Module:system_replace_state(StateFun, State)</code> <p>In the example:</p> @@ -407,9 +404,9 @@ system_replace_state(StateFun, Chs) -> NChs = StateFun(Chs), {ok, NChs, NChs}. </code> - <p>If the special process is set to trap exits, note that if - the parent process terminates, the expected behavior is to - terminate with the same reason:</p> + <p>If the special process is set to trap exits and if the parent + process terminates, the expected behavior is to terminate with + the same reason:</p> <code type="none"> init(...) -> ..., @@ -431,28 +428,23 @@ loop(...) -> <section> <title>User-Defined Behaviours</title> - <p><marker id="behaviours"/>To implement a user-defined behaviour, - write code similar to code for a special process but calling - functions in a callback module for handling specific tasks.</p> - <p>If it is desired that the compiler should warn for missing - callback functions, as it does for the OTP behaviours, add - <c>-callback</c> attributes in the behaviour module to describe - the expected callback functions:</p> - + write code similar to + code for a special process, but call functions in a callback + module for handling specific tasks.</p> + <p>If the compiler is to warn for missing callback functions, as it + does for the OTP behaviours, add <c>-callback</c> attributes in the + behaviour module to describe the expected callbacks:</p> <code type="none"> -callback Name1(Arg1_1, Arg1_2, ..., Arg1_N1) -> Res1. -callback Name2(Arg2_1, Arg2_2, ..., Arg2_N2) -> Res2. ... -callback NameM(ArgM_1, ArgM_2, ..., ArgM_NM) -> ResM.</code> - - <p>where each <c>Name</c> is the name of a callback function and - <c>Arg</c> and <c>Res</c> are types as described in - Specifications for functions in <seealso - marker="../reference_manual/typespec">Types and Function - Specifications</seealso>. The whole syntax of the - <c>-spec</c> attribute is supported by <c>-callback</c> - attribute.</p> + <p><c>NameX</c> are the names of the expected callbacks. + <c>ArgX_Y</c> and <c>ResX</c> are types as they are described in + <seealso marker="../reference_manual/typespec">Types and + Function Specifications</seealso>. The whole syntax of the <c>-spec</c> + attribute is supported by the <c>-callback</c> attribute.</p> <p>Callback functions that are optional for the user of the behaviour to implement are specified by use of the <c>-optional_callbacks</c> attribute:</p> @@ -487,10 +479,10 @@ behaviour_info(callbacks) -> generated by the compiler using the <c>-callback</c> attributes.</p> <p>When the compiler encounters the module attribute - <c>-behaviour(Behaviour).</c> in a module <c>Mod</c>, it will - call <c>Behaviour:behaviour_info(callbacks)</c> and compare the + <c>-behaviour(Behaviour).</c> in a module <c>Mod</c>, it + calls <c>Behaviour:behaviour_info(callbacks)</c> and compares the result with the set of functions actually exported from - <c>Mod</c>, and issue a warning if any callback function is + <c>Mod</c>, and issues a warning if any callback function is missing.</p> <p>Example:</p> <code type="none"> diff --git a/system/doc/design_principles/sup_princ.xml b/system/doc/design_principles/sup_princ.xml index 3d7b53e339..9583ca5c55 100644 --- a/system/doc/design_principles/sup_princ.xml +++ b/system/doc/design_principles/sup_princ.xml @@ -28,15 +28,16 @@ <rev></rev> <file>sup_princ.xml</file> </header> - <p>This section should be read in conjunction with - <seealso marker="stdlib:supervisor">supervisor(3)</seealso>, where - all details about the supervisor behaviour are described.</p> + <p>This section should be read with the + <seealso marker="stdlib:supervisor">supervisor(3)</seealso> manual page + in STDLIB, where all details about the supervisor + behaviour is given.</p> <section> <title>Supervision Principles</title> <p>A supervisor is responsible for starting, stopping, and monitoring its child processes. The basic idea of a supervisor is - that it shall keep its child processes alive by restarting them + that it is to keep its child processes alive by restarting them when necessary.</p> <p>Which child processes to start and monitor is specified by a list of <seealso marker="#spec">child specifications</seealso>. @@ -47,8 +48,8 @@ <section> <title>Example</title> <p>The callback module for a supervisor starting the server from - the <seealso marker="gen_server_concepts#ex">gen_server chapter</seealso> - could look like this:</p> + <seealso marker="gen_server_concepts#ex">gen_server Behaviour</seealso> + can look as follows:</p> <marker id="ex"></marker> <code type="none"> -module(ch_sup). @@ -79,6 +80,7 @@ init(_Args) -> <section> <title>Supervisor Flags</title> + <marker id="flags"/> <p>This is the type definition for the supervisor flags:</p> <code type="none"><![CDATA[ sup_flags() = #{strategy => strategy(), % optional @@ -136,9 +138,9 @@ SupFlags = #{strategy => Strategy, ...}</code> <section> <title>rest_for_one</title> - <p>If a child process terminates, the 'rest' of the child - processes -- i.e. the child processes after the terminated - process in start order -- are terminated. Then the terminated + <p>If a child process terminates, the rest of the child + processes (that is, the child processes after the terminated + process in start order) are terminated. Then the terminated child process and the rest of the child processes are restarted.</p> </section> @@ -162,7 +164,7 @@ SupFlags = #{intensity => MaxR, period => MaxT, ...}</code> <p>If more than <c>MaxR</c> number of restarts occur in the last <c>MaxT</c> seconds, the supervisor terminates all the child processes and then itself.</p> - <p>When the supervisor terminates, the next higher level + <p>When the supervisor terminates, then the next higher-level supervisor takes some action. It either restarts the terminated supervisor or terminates itself.</p> <p>The intention of the restart mechanism is to prevent a situation @@ -176,14 +178,14 @@ SupFlags = #{intensity => MaxR, period => MaxT, ...}</code> <section> <marker id="spec"></marker> <title>Child Specification</title> - <p>This is the type definition for a child specification:</p> + <p>The type definition for a child specification is as follows:</p> <code type="none"><![CDATA[ child_spec() = #{id => child_id(), % mandatory start => mfargs(), % mandatory restart => restart(), % optional shutdown => shutdown(), % optional type => worker(), % optional - modules => modules()} % optional</pre> + modules => modules()} % optional child_id() = term() mfargs() = {M :: module(), F :: atom(), A :: [term()]} modules() = [module()] | dynamic @@ -195,7 +197,7 @@ child_spec() = #{id => child_id(), % mandatory <p><c>id</c> is used to identify the child specification internally by the supervisor.</p> <p>The <c>id</c> key is mandatory.</p> - <p>Note that this identifier on occations has been called + <p>Note that this identifier occasionally has been called "name". As far as possible, the terms "identifier" or "id" are now used but in order to keep backwards compatibility, some occurences of "name" can still be found, for example @@ -205,24 +207,28 @@ child_spec() = #{id => child_id(), % mandatory <p><c>start</c> defines the function call used to start the child process. It is a module-function-arguments tuple used as <c>apply(M, F, A)</c>.</p> - <p>It should be (or result in) a call to - <c>supervisor:start_link</c>, <c>gen_server:start_link</c>, - <c>gen_fsm:start_link</c>, or <c>gen_event:start_link</c>. - (Or a function compliant with these functions, see - <c>supervisor(3)</c> for details.</p> + <p>It is to be (or result in) a call to any of the following:</p> + <list type="bulleted"> + <item><c>supervisor:start_link</c></item> + <item><c>gen_server:start_link</c></item> + <item><c>gen_fsm:start_link</c></item> + <item><c>gen_event:start_link</c></item> + <item>A function compliant with these functions. For details, + see the <c>supervisor(3)</c> manual page.</item> + </list> <p>The <c>start</c> key is mandatory.</p> </item> <item> - <p><c>restart</c> defines when a terminated child process shall + <p><c>restart</c> defines when a terminated child process is to be restarted.</p> <list type="bulleted"> <item>A <c>permanent</c> child process is always restarted.</item> <item>A <c>temporary</c> child process is never restarted - (not even when the supervisor's restart strategy - is <c>rest_for_one</c> or <c>one_for_all</c> and a sibling's + (not even when the supervisor restart strategy + is <c>rest_for_one</c> or <c>one_for_all</c> and a sibling death causes the temporary process to be terminated).</item> <item>A <c>transient</c> child process is restarted only if it - terminates abnormally, i.e. with another exit reason than + terminates abnormally, that is, with another exit reason than <c>normal</c>, <c>shutdown</c>, or <c>{shutdown,Term}</c>.</item> </list> <p>The <c>restart</c> key is optional. If it is not given, the @@ -230,27 +236,27 @@ child_spec() = #{id => child_id(), % mandatory </item> <item> <marker id="shutdown"></marker> - <p><c>shutdown</c> defines how a child process shall be + <p><c>shutdown</c> defines how a child process is to be terminated.</p> <list type="bulleted"> - <item><c>brutal_kill</c> means the child process is + <item><c>brutal_kill</c> means that the child process is unconditionally terminated using <c>exit(Child, kill)</c>.</item> - <item>An integer timeout value means that the supervisor tells + <item>An integer time-out value means that the supervisor tells the child process to terminate by calling <c>exit(Child, shutdown)</c> and then waits for an exit signal back. If no exit signal is received within the specified time, the child process is unconditionally terminated using <c>exit(Child, kill)</c>.</item> - <item>If the child process is another supervisor, it should be + <item>If the child process is another supervisor, it is to be set to <c>infinity</c> to give the subtree enough time to shut down. It is also allowed to set it to <c>infinity</c>, - if the child process is a worker.</item> + if the child process is a worker. See the warning below:</item> </list> <warning> <p>Be careful when setting the shutdown time to <c>infinity</c> when the child process is a worker. Because, in this situation, the termination of the supervision tree depends on the - child process, it must be implemented in a safe way and its cleanup + child process; it must be implemented in a safe way and its cleanup procedure must always return.</p> </warning> <p>The <c>shutdown</c> key is optional. If it is not given, @@ -266,7 +272,7 @@ child_spec() = #{id => child_id(), % mandatory default value <c>worker</c> will be used.</p> </item> <item> - <p><c>modules</c> should be a list with one element + <p><c>modules</c> are to be a list with one element <c>[Module]</c>, where <c>Module</c> is the name of the callback module, if the child process is a supervisor, gen_server or gen_fsm. If the child process is a gen_event, @@ -279,8 +285,8 @@ child_spec() = #{id => child_id(), % mandatory child's start <c>{M,F,A}</c>.</p> </item> </list> - <p>Example: The child specification to start the server <c>ch3</c> - in the example above looks like:</p> + <p><em>Example:</em> The child specification to start the server + <c>ch3</c> in the previous example look as follows:</p> <code type="none"> #{id => ch3, start => {ch3, start_link, []}, @@ -301,11 +307,11 @@ child_spec() = #{id => child_id(), % mandatory start => {gen_event, start_link, [{local, error_man}]}, modules => dynamic}</code> <p>Both server and event manager are registered processes which - can be expected to be accessible at all times, thus they are + can be expected to be always accessible. Thus they are specified to be <c>permanent</c>.</p> <p><c>ch3</c> does not need to do any cleaning up before - termination, thus no shutdown time is needed but - <c>brutal_kill</c> should be sufficient. <c>error_man</c> may + termination. Thus, no shutdown time is needed, but + <c>brutal_kill</c> is sufficient. <c>error_man</c> can need some time for the event handlers to clean up, thus the shutdown time is set to 5000 ms (which is the default value).</p> @@ -320,19 +326,20 @@ child_spec() = #{id => child_id(), % mandatory <section> <marker id="super_tree"></marker> <title>Starting a Supervisor</title> - <p>In the example above, the supervisor is started by calling + <p>In the previous example, the supervisor is started by calling <c>ch_sup:start_link()</c>:</p> <code type="none"> start_link() -> supervisor:start_link(ch_sup, []).</code> - <p><c>ch_sup:start_link</c> calls the function - <c>supervisor:start_link/2</c>. This function spawns and links to - a new process, a supervisor.</p> + <p><c>ch_sup:start_link</c> calls function + <c>supervisor:start_link/2</c>, which spawns and links to a new + process, a supervisor.</p> <list type="bulleted"> <item>The first argument, <c>ch_sup</c>, is the name of - the callback module, that is the module where the <c>init</c> + the callback module, that is, the module where the <c>init</c> callback function is located.</item> - <item>The second argument, [], is a term which is passed as-is to + <item>The second argument, <c>[]</c>, is a term that is passed + as is to the callback function <c>init</c>. Here, <c>init</c> does not need any indata and ignores the argument.</item> </list> @@ -351,26 +358,27 @@ init(_Args) -> shutdown => brutal_kill}], {ok, {SupFlags, ChildSpecs}}.</code> <p>The supervisor then starts all its child processes according to - the given child specifications. In this case there, is one child - process, <c>ch3</c>.</p> - <p>Note that <c>supervisor:start_link</c> is synchronous. It does + the child specifications in the start specification. In this case + there is one child process, <c>ch3</c>.</p> + <p><c>supervisor:start_link</c> is synchronous. It does not return until all child processes have been started.</p> </section> <section> <title>Adding a Child Process</title> - <p>In addition to the static supervision tree, we can also add - dynamic child processes to an existing supervisor with - the following call:</p> + <p>In addition to the static supervision tree, dynamic child + processes can be added to an existing supervisor with the following + call:</p> <code type="none"> supervisor:start_child(Sup, ChildSpec)</code> <p><c>Sup</c> is the pid, or name, of the supervisor. - <c>ChildSpec</c> is a <seealso marker="#spec">child specification</seealso>.</p> + <c>ChildSpec</c> is a + <seealso marker="#spec">child specification</seealso>.</p> <p>Child processes added using <c>start_child/2</c> behave in - the same manner as the other child processes, with the following - important exception: If a supervisor dies and is re-created, then - all child processes which were dynamically added to the supervisor - will be lost.</p> + the same way as the other child processes, with the an important + exception: if a supervisor dies and is recreated, then + all child processes that were dynamically added to the supervisor + are lost.</p> </section> <section> @@ -393,11 +401,12 @@ supervisor:delete_child(Sup, Id)</code> <marker id="simple"/> <section> - <title>Simple-One-For-One Supervisors</title> + <title>Simplified one_for_one Supervisors</title> <p>A supervisor with restart strategy <c>simple_one_for_one</c> is - a simplified one_for_one supervisor, where all child processes are - dynamically added instances of the same child specification.</p> - <p>Example of a callback module for a simple_one_for_one supervisor:</p> + a simplified <c>one_for_one</c> supervisor, where all child + processes are dynamically added instances of the same process.</p> + <p>The following is an example of a callback module for a + <c>simple_one_for_one</c> supervisor:</p> <code type="none"> -module(simple_sup). -behaviour(supervisor). @@ -416,12 +425,12 @@ init(_Args) -> start => {call, start_link, []}, shutdown => brutal_kill}], {ok, {SupFlags, ChildSpecs}}.</code> - <p>When started, the supervisor will not start any child processes. + <p>When started, the supervisor does not start any child processes. Instead, all child processes are added dynamically by calling:</p> <code type="none"> supervisor:start_child(Sup, List)</code> <p><c>Sup</c> is the pid, or name, of the supervisor. - <c>List</c> is an arbitrary list of terms which will be added to + <c>List</c> is an arbitrary list of terms, which are added to the list of arguments specified in the child specification. If the start function is specified as <c>{M, F, A}</c>, the child process is started by calling @@ -429,17 +438,17 @@ supervisor:start_child(Sup, List)</code> <p>For example, adding a child to <c>simple_sup</c> above:</p> <code type="none"> supervisor:start_child(Pid, [id1])</code> - <p>results in the child process being started by calling + <p>The result is that the child process is started by calling <c>apply(call, start_link, []++[id1])</c>, or actually:</p> <code type="none"> call:start_link(id1)</code> - <p>A child under a <c>simple_one_for_one</c> supervisor can be terminated - with</p> + <p>A child under a <c>simple_one_for_one</c> supervisor can be + terminated with the following:</p> <code type="none"> supervisor:terminate_child(Sup, Pid)</code> - <p>where <c>Sup</c> is the pid, or name, of the supervisor and + <p><c>Sup</c> is the pid, or name, of the supervisor and <c>Pid</c> is the pid of the child.</p> - <p>Because a <c>simple_one_for_one</c> supervisor could have many + <p>Because a <c>simple_one_for_one</c> supervisor can have many children, it shuts them all down asynchronously. This means that the children will do their cleanup in parallel and therefore the order in which they are stopped is not defined.</p> @@ -447,11 +456,11 @@ supervisor:terminate_child(Sup, Pid)</code> <section> <title>Stopping</title> - <p>Since the supervisor is part of a supervision tree, it will - automatically be terminated by its supervisor. When asked to - shutdown, it will terminate all child processes in reversed start + <p>Since the supervisor is part of a supervision tree, it is + automatically terminated by its supervisor. When asked to + shut down, it terminates all child processes in reversed start order according to the respective shutdown specifications, and - then terminate itself.</p> + then terminates itself.</p> </section> </chapter> diff --git a/system/doc/efficiency_guide/advanced.xml b/system/doc/efficiency_guide/advanced.xml index 51f1b2612c..df2ab0f811 100644 --- a/system/doc/efficiency_guide/advanced.xml +++ b/system/doc/efficiency_guide/advanced.xml @@ -18,7 +18,6 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> <title>Advanced</title> @@ -31,175 +30,245 @@ <section> <title>Memory</title> - <p>A good start when programming efficiently is to have knowledge about + <p>A good start when programming efficiently is to know how much memory different data types and operations require. It is implementation-dependent how much memory the Erlang data types and - other items consume, but here are some figures for the - erts-5.2 system (OTP release R9B). (There have been no significant - changes in R13.)</p> + other items consume, but the following table shows some figures for + the <c>erts-5.2</c> system in R9B. There have been no significant + changes in R13.</p> - <p>The unit of measurement is memory words. There exists both a 32-bit - and a 64-bit implementation, and a word is therefore, 4 bytes or + <p>The unit of measurement is memory words. There exists both a + 32-bit and a 64-bit implementation. A word is therefore 4 bytes or 8 bytes, respectively.</p> <table> <row> - <cell align="center" valign="middle">Data type</cell> - <cell align="center" valign="middle">Memory size</cell> + <cell><em>Data Type</em></cell> + <cell><em>Memory Size</em></cell> </row> <row> - <cell align="left" valign="middle">Small integer</cell> - <cell align="left" valign="middle">1 word<br></br> -On 32-bit architectures: -134217729 < i < 134217728 (28 bits)<br></br> -On 64-bit architectures: -576460752303423489 < i < 576460752303423488 (60 bits)</cell> + <cell>Small integer</cell> + <cell>1 word.<br></br> + On 32-bit architectures: -134217729 < i < 134217728 + (28 bits).<br></br> + On 64-bit architectures: -576460752303423489 < i < + 576460752303423488 (60 bits).</cell> </row> <row> - <cell align="left" valign="middle">Big integer</cell> - <cell align="left" valign="middle">3..N words</cell> + <cell>Large integer</cell> + <cell>3..N words.</cell> </row> <row> - <cell align="left" valign="middle">Atom</cell> - <cell align="left" valign="middle">1 word. Note: an atom refers into - an atom table which also consumes memory. + <cell>Atom</cell> + <cell>1 word.<br></br> + An atom refers into an atom table, which also consumes memory. The atom text is stored once for each unique atom in this table. The atom table is <em>not</em> garbage-collected.</cell> </row> <row> - <cell align="left" valign="middle">Float</cell> - <cell align="left" valign="middle">On 32-bit architectures: 4 words <br></br> -On 64-bit architectures: 3 words</cell> + <cell>Float</cell> + <cell>On 32-bit architectures: 4 words.<br></br> + On 64-bit architectures: 3 words.</cell> </row> <row> - <cell align="left" valign="middle">Binary</cell> - <cell align="left" valign="middle">3..6 + data (can be shared)</cell> + <cell>Binary</cell> + <cell>3..6 words + data (can be shared).</cell> </row> <row> - <cell align="left" valign="middle">List</cell> - <cell align="left" valign="middle">1 word + 1 word per element + the size of each element</cell> + <cell>List</cell> + <cell>1 word + 1 word per element + the size of each element.</cell> </row> <row> - <cell align="left" valign="middle">String (is the same as a list of integers)</cell> - <cell align="left" valign="middle">1 word + 2 words per character</cell> + <cell>String (is the same as a list of integers)</cell> + <cell>1 word + 2 words per character.</cell> </row> <row> - <cell align="left" valign="middle">Tuple</cell> - <cell align="left" valign="middle">2 words + the size of each element</cell> + <cell>Tuple</cell> + <cell>2 words + the size of each element.</cell> </row> <row> - <cell align="left" valign="middle">Pid</cell> - <cell align="left" valign="middle">1 word for a process identifier from the current local node, and 5 words for a process identifier from another node. Note: a process identifier refers into a process table and a node table which also consumes memory.</cell> + <cell>Pid</cell> + <cell>1 word for a process identifier from the current local node + + 5 words for a process identifier from another node.<br></br> + A process identifier refers into a process table and a node table, + which also consumes memory.</cell> </row> <row> - <cell align="left" valign="middle">Port</cell> - <cell align="left" valign="middle">1 word for a port identifier from the current local node, and 5 words for a port identifier from another node. Note: a port identifier refers into a port table and a node table which also consumes memory.</cell> + <cell>Port</cell> + <cell>1 word for a port identifier from the current local node + + 5 words for a port identifier from another node.<br></br> + A port identifier refers into a port table and a node table, + which also consumes memory.</cell> </row> <row> - <cell align="left" valign="middle">Reference</cell> - <cell align="left" valign="middle">On 32-bit architectures: 5 words for a reference from the current local node, and 7 words for a reference from another node. <br></br> -On 64-bit architectures: 4 words for a reference from the current local node, and 6 words for a reference from another node. Note: a reference refers into a node table which also consumes memory.</cell> + <cell>Reference</cell> + <cell>On 32-bit architectures: 5 words for a reference from the + current local node + 7 words for a reference from another + node.<br></br> + On 64-bit architectures: 4 words for a reference from the current + local node + 6 words for a reference from another node.<br></br> + A reference refers into a node table, which also consumes + memory.</cell> </row> <row> - <cell align="left" valign="middle">Fun</cell> - <cell align="left" valign="middle">9..13 words + size of environment. Note: a fun refers into a fun table which also consumes memory.</cell> + <cell>Fun</cell> + <cell>9..13 words + the size of environment.<br></br> + A fun refers into a fun table, which also consumes memory.</cell> </row> <row> - <cell align="left" valign="middle">Ets table</cell> - <cell align="left" valign="middle">Initially 768 words + the size of each element (6 words + size of Erlang data). The table will grow when necessary.</cell> + <cell>Ets table</cell> + <cell>Initially 768 words + the size of each element (6 words + + the size of Erlang data). The table grows when necessary.</cell> </row> <row> - <cell align="left" valign="middle">Erlang process</cell> - <cell align="left" valign="middle">327 words when spawned including a heap of 233 words.</cell> + <cell>Erlang process</cell> + <cell>327 words when spawned, including a heap of 233 words.</cell> </row> - <tcaption>Memory size of different data types</tcaption> + <tcaption>Memory Size of Different Data Types</tcaption> </table> </section> <section> - <title>System limits</title> - <p>The Erlang language specification puts no limits on number of processes, - length of atoms etc., but for performance and memory saving reasons, - there will always be limits in a practical implementation of the Erlang - language and execution environment.</p> - <taglist> - <tag><em>Processes</em></tag> - <item> - <p>The maximum number of simultaneously alive Erlang processes is - by default 32768. This limit can be configured at startup, - for more information see the - <seealso marker="erts:erl#max_processes"><c>+P</c></seealso> - command line flag of - <seealso marker="erts:erl"><c>erl(1)</c></seealso>.</p> - </item> - <tag><em>Distributed nodes</em></tag> - <item> - <taglist> - <tag>Known nodes</tag> - <item> - <p>A remote node Y has to be known to node X if there exist - any pids, ports, references, or funs (Erlang data types) from Y - on X, or if X and Y are connected. The maximum number of remote - nodes simultaneously/ever known to a node is limited by the - <seealso marker="#atoms">maximum number of atoms</seealso> - available for node names. All data concerning remote nodes, - except for the node name atom, are garbage-collected.</p> - </item> - <tag>Connected nodes</tag> - <item>The maximum number of simultaneously connected nodes is limited by - either the maximum number of simultaneously known remote nodes, - <seealso marker="#ports">the maximum number of (Erlang) ports</seealso> - available, or - <seealso marker="#files_sockets">the maximum number of sockets</seealso> - available.</item> - </taglist> - </item> - <tag><em>Characters in an atom</em></tag> - <item>255</item> - <tag><em>Atoms </em></tag> - <item> <marker id="atoms"></marker> - By default, the maximum number of atoms is 1048576. - This limit can be raised or lowered using the <c>+t</c> option.</item> - <tag><em>Ets-tables</em></tag> - <item>The default is 1400, can be changed with the environment variable <c>ERL_MAX_ETS_TABLES</c>.</item> - <tag><em>Elements in a tuple</em></tag> - <item>The maximum number of elements in a tuple is 67108863 (26 bit unsigned integer). Other factors - such as the available memory can of course make it hard to create a tuple of that size. </item> - <tag><em>Size of binary</em></tag> - <item>In the 32-bit implementation of Erlang, 536870911 bytes is the - largest binary that can be constructed or matched using the bit syntax. - (In the 64-bit implementation, the maximum size is 2305843009213693951 bytes.) - If the limit is exceeded, bit syntax construction will fail with a - <c>system_limit</c> exception, while any attempt to match a binary that is - too large will fail. - This limit is enforced starting with the R11B-4 release; in earlier releases, - operations on too large binaries would in general either fail or give incorrect - results. - In future releases of Erlang/OTP, other operations that create binaries (such as - <c>list_to_binary/1</c>) will probably also enforce the same limit.</item> - <tag><em>Total amount of data allocated by an Erlang node</em></tag> - <item>The Erlang runtime system can use the complete 32 (or 64) bit address space, - but the operating system often limits a single process to use less than that.</item> - <tag><em>Length of a node name</em></tag> - <item>An Erlang node name has the form host@shortname or host@longname. The node name is - used as an atom within the system so the maximum size of 255 holds for the node name too.</item> - <tag><em>Open ports</em></tag> - <item> - <marker id="ports"></marker> - <p>The maximum number of simultaneously open Erlang ports is - often by default 16384. This limit can be configured at startup, - for more information see the - <seealso marker="erts:erl#max_ports"><c>+Q</c></seealso> - command line flag of - <seealso marker="erts:erl"><c>erl(1)</c></seealso>.</p> - </item> - <tag><em>Open files, and sockets</em></tag> - <item> <marker id="files_sockets"></marker> + <title>System Limits</title> + <p>The Erlang language specification puts no limits on the number of + processes, length of atoms, and so on. However, for performance and + memory saving reasons, there will always be limits in a practical + implementation of the Erlang language and execution environment.</p> - The maximum number of simultaneously open files and sockets - depend on - <seealso marker="#ports">the maximum number of Erlang ports</seealso> - available, and operating system specific settings and limits.</item> - <tag><em>Number of arguments to a function or fun</em></tag> - <item>255</item> - </taglist> + <table> + <row> + <cell>Processes</cell> + <cell>The maximum number of simultaneously alive Erlang processes + is by default 32,768. This limit can be configured at startup. + For more information, see the + <seealso marker="erts:erl#max_processes"><c>+P</c></seealso> + command-line flag in the + <seealso marker="erts:erl"><c>erl(1)</c></seealso> + manual page in <c>erts</c>.</cell> + </row> + <row> + <cell>Known nodes</cell> + <cell>A remote node Y must be known to node X if there exists + any pids, ports, references, or funs (Erlang data types) from Y + on X, or if X and Y are connected. The maximum number of remote + nodes simultaneously/ever known to a node is limited by the + <seealso marker="#atoms">maximum number of atoms</seealso> + available for node names. All data concerning remote nodes, + except for the node name atom, are garbage-collected.</cell> + </row> + <row> + <cell>Connected nodes</cell> + <cell>The maximum number of simultaneously connected nodes is + limited by either the maximum number of simultaneously known + remote nodes, + <seealso marker="#ports">the maximum number of (Erlang) ports</seealso> + available, or + <seealso marker="#files_sockets">the maximum number of sockets</seealso> + available.</cell> + </row> + <row> + <cell>Characters in an atom</cell> + <cell>255.</cell> + </row> + <row> + <cell><marker id="atoms"></marker>Atoms</cell> + <cell>By default, the maximum number of atoms is 1,048,576. This + limit can be raised or lowered using the <c>+t</c> option.</cell> + </row> + <row> + <cell>Ets tables</cell> + <cell>Default is 1400. It can be changed with the environment + variable <c>ERL_MAX_ETS_TABLES</c>.</cell> + </row> + <row> + <cell>Elements in a tuple</cell> + <cell>The maximum number of elements in a tuple is 67,108,863 + (26-bit unsigned integer). Clearly, other factors such as the + available memory can make it difficult to create a tuple of + that size.</cell> + </row> + <row> + <cell>Size of binary</cell> + <cell>In the 32-bit implementation of Erlang, 536,870,911 + bytes is the largest binary that can be constructed or matched + using the bit syntax. In the 64-bit implementation, the maximum + size is 2,305,843,009,213,693,951 bytes. If the limit is + exceeded, bit syntax construction fails with a + <c>system_limit</c> exception, while any attempt to match a + binary that is too large fails. This limit is enforced starting + in R11B-4.<br></br> + In earlier Erlang/OTP releases, operations on too large + binaries in general either fail or give incorrect results. In + future releases, other operations that create binaries (such as + <c>list_to_binary/1</c>) will probably also enforce the same + limit.</cell> + </row> + <row> + <cell>Total amount of data allocated by an Erlang node</cell> + <cell>The Erlang runtime system can use the complete 32-bit + (or 64-bit) address space, but the operating system often + limits a single process to use less than that.</cell> + </row> + <row> + <cell>Length of a node name</cell> + <cell>An Erlang node name has the form host@shortname or + host@longname. The node name is used as an atom within + the system, so the maximum size of 255 holds also for the + node name.</cell> + </row> + <row> + <cell><marker id="ports"></marker>Open ports</cell> + <cell>The maximum number of simultaneously open Erlang ports is + often by default 16,384. This limit can be configured at startup. + For more information, see the + <seealso marker="erts:erl#max_ports"><c>+Q</c></seealso> + command-line flag in the + <seealso marker="erts:erl"><c>erl(1)</c></seealso> manual page + in <c>erts</c>.</cell> + </row> + <row> + <cell><marker id="files_sockets"></marker>Open files and + sockets</cell> + <cell>The maximum number of simultaneously open files and + sockets depends on + <seealso marker="#ports">the maximum number of Erlang ports</seealso> + available, as well as on operating system-specific settings + and limits.</cell> + </row> + <row> + <cell>Number of arguments to a function or fun</cell> + <cell>255</cell> + </row> + <row> + <cell><marker id="unique_references"/>Unique References on a Runtime System Instance</cell> + <cell>Each scheduler thread has its own set of references, and all + other threads have a shared set of references. Each set of references + consist of <c>2⁶⁴ - 1</c> unique references. That is the total + amount of unique references that can be produced on a runtime + system instance is <c>(NoSchedulers + 1) * (2⁶⁴ - 1)</c>. If a + scheduler thread create a new reference each nano second, + references will at earliest be reused after more than 584 years. + That is, for the foreseeable future they are unique enough.</cell> + </row> + <row> + <cell><marker id="unique_integers"/>Unique Integers on a Runtime System Instance</cell> + <cell>There are two types of unique integers both created using the + <seealso marker="erts:erlang#unique_integer/1">erlang:unique_integer()</seealso> + BIF. Unique integers created: + <taglist> + <tag>with the <c>monotonic</c> modifier</tag> + <item>consist of a set of <c>2⁶⁴ - 1</c> unique integers.</item> + <tag>without the <c>monotonic</c> modifier</tag> + <item>consist of a set of <c>2⁶⁴ - 1</c> unique integers per scheduler + thread and a set of <c>2⁶⁴ - 1</c> unique integers shared by + other threads. That is the total amount of unique integers without + the <c>monotonic</c> modifier is <c>(NoSchedulers + 1) * (2⁶⁴ - 1)</c></item> + </taglist> + If a unique integer is created each nano second, unique integers + will at earliest be reused after more than 584 years. That is, for + the foreseeable future they are unique enough.</cell> + </row> + <tcaption>System Limits</tcaption> + </table> </section> </chapter> diff --git a/system/doc/efficiency_guide/binaryhandling.xml b/system/doc/efficiency_guide/binaryhandling.xml index 4ba1378059..0ac1a7ee32 100644 --- a/system/doc/efficiency_guide/binaryhandling.xml +++ b/system/doc/efficiency_guide/binaryhandling.xml @@ -23,7 +23,7 @@ The Initial Developer of the Original Code is Ericsson AB. </legalnotice> - <title>Constructing and matching binaries</title> + <title>Constructing and Matching Binaries</title> <prepared>Bjorn Gustavsson</prepared> <docno></docno> <date>2007-10-12</date> @@ -31,10 +31,10 @@ <file>binaryhandling.xml</file> </header> - <p>In R12B, the most natural way to write binary construction and matching is now + <p>In R12B, the most natural way to construct and match binaries is significantly faster than in earlier releases.</p> - <p>To construct at binary, you can simply write</p> + <p>To construct a binary, you can simply write as follows:</p> <p><em>DO</em> (in R12B) / <em>REALLY DO NOT</em> (in earlier releases)</p> <code type="erl"><![CDATA[ @@ -46,13 +46,12 @@ my_list_to_binary([H|T], Acc) -> my_list_to_binary([], Acc) -> Acc.]]></code> - <p>In releases before R12B, <c>Acc</c> would be copied in every iteration. - In R12B, <c>Acc</c> will be copied only in the first iteration and extra - space will be allocated at the end of the copied binary. In the next iteration, - <c>H</c> will be written in to the extra space. When the extra space runs out, - the binary will be reallocated with more extra space.</p> - - <p>The extra space allocated (or reallocated) will be twice the size of the + <p>In releases before R12B, <c>Acc</c> is copied in every iteration. + In R12B, <c>Acc</c> is copied only in the first iteration and extra + space is allocated at the end of the copied binary. In the next iteration, + <c>H</c> is written into the extra space. When the extra space runs out, + the binary is reallocated with more extra space. The extra space allocated + (or reallocated) is twice the size of the existing binary data, or 256, whichever is larger.</p> <p>The most natural way to match binaries is now the fastest:</p> @@ -64,55 +63,79 @@ my_binary_to_list(<<H,T/binary>>) -> my_binary_to_list(<<>>) -> [].]]></code> <section> - <title>How binaries are implemented</title> + <title>How Binaries are Implemented</title> <p>Internally, binaries and bitstrings are implemented in the same way. - In this section, we will call them <em>binaries</em> since that is what + In this section, they are called <em>binaries</em> because that is what they are called in the emulator source code.</p> - <p>There are four types of binary objects internally. Two of them are - containers for binary data and two of them are merely references to - a part of a binary.</p> - - <p>The binary containers are called <em>refc binaries</em> - (short for <em>reference-counted binaries</em>) and <em>heap binaries</em>.</p> + <p>Four types of binary objects are available internally:</p> + <list type="bulleted"> + <item><p>Two are containers for binary data and are called:</p> + <list type="bulleted"> + <item><em>Refc binaries</em> (short for + <em>reference-counted binaries</em>)</item> + <item><em>Heap binaries</em></item> + </list></item> + <item><p>Two are merely references to a part of a binary and + are called:</p> + <list type="bulleted"> + <item><em>sub binaries</em></item> + <item><em>match contexts</em></item> + </list></item> + </list> - <p><marker id="refc_binary"></marker><em>Refc binaries</em> - consist of two parts: an object stored on - the process heap, called a <em>ProcBin</em>, and the binary object itself - stored outside all process heaps.</p> + <section> + <marker id="refc_binary"></marker> + <title>Refc Binaries</title> + <p>Refc binaries consist of two parts:</p> + <list type="bulleted"> + <item>An object stored on the process heap, called a + <em>ProcBin</em></item> + <item>The binary object itself, stored outside all process + heaps</item> + </list> <p>The binary object can be referenced by any number of ProcBins from any - number of processes; the object contains a reference counter to keep track + number of processes. The object contains a reference counter to keep track of the number of references, so that it can be removed when the last reference disappears.</p> <p>All ProcBin objects in a process are part of a linked list, so that the garbage collector can keep track of them and decrement the reference counters in the binary when a ProcBin disappears.</p> + </section> - <p><marker id="heap_binary"></marker><em>Heap binaries</em> are small binaries, - up to 64 bytes, that are stored directly on the process heap. - They will be copied when the process - is garbage collected and when they are sent as a message. They don't + <section> + <marker id="heap_binary"></marker> + <title>Heap Binaries</title> + <p>Heap binaries are small binaries, up to 64 bytes, and are stored + directly on the process heap. They are copied when the process is + garbage-collected and when they are sent as a message. They do not require any special handling by the garbage collector.</p> + </section> - <p>There are two types of reference objects that can reference part of - a refc binary or heap binary. They are called <em>sub binaries</em> and - <em>match contexts</em>.</p> + <section> + <title>Sub Binaries</title> + <p>The reference objects <em>sub binaries</em> and + <em>match contexts</em> can reference part of + a refc binary or heap binary.</p> <p><marker id="sub_binary"></marker>A <em>sub binary</em> is created by <c>split_binary/2</c> and when a binary is matched out in a binary pattern. A sub binary is a reference - into a part of another binary (refc or heap binary, never into a another + into a part of another binary (refc or heap binary, but never into another sub binary). Therefore, matching out a binary is relatively cheap because the actual binary data is never copied.</p> + </section> - <p><marker id="match_context"></marker>A <em>match context</em> is - similar to a sub binary, but is optimized - for binary matching; for instance, it contains a direct pointer to the binary - data. For each field that is matched out of a binary, the position in the - match context will be incremented.</p> + <section> + <title>Match Context</title> + <marker id="match_context"></marker> + <p>A <em>match context</em> is similar to a sub binary, but is + optimized for binary matching. For example, it contains a direct + pointer to the binary data. For each field that is matched out of + a binary, the position in the match context is incremented.</p> <p>In R11B, a match context was only used during a binary matching operation.</p> @@ -122,27 +145,28 @@ my_binary_to_list(<<>>) -> [].]]></code> context and discard the sub binary. Instead of creating a sub binary, the match context is kept.</p> - <p>The compiler can only do this optimization if it can know for sure + <p>The compiler can only do this optimization if it knows that the match context will not be shared. If it would be shared, the functional properties (also called referential transparency) of Erlang would break.</p> + </section> </section> <section> - <title>Constructing binaries</title> - - <p>In R12B, appending to a binary or bitstring</p> + <title>Constructing Binaries</title> + <p>In R12B, appending to a binary or bitstring + is specially optimized by the <em>runtime system</em>:</p> <code type="erl"><![CDATA[ <<Binary/binary, ...>> <<Binary/bitstring, ...>>]]></code> - <p>is specially optimized by the <em>run-time system</em>. - Because the run-time system handles the optimization (instead of + <p>As the runtime system handles the optimization (instead of the compiler), there are very few circumstances in which the optimization - will not work.</p> + does not work.</p> - <p>To explain how it works, we will go through this code</p> + <p>To explain how it works, let us examine the following code line + by line:</p> <code type="erl"><![CDATA[ Bin0 = <<0>>, %% 1 @@ -152,81 +176,81 @@ Bin3 = <<Bin2/binary,7,8,9>>, %% 4 Bin4 = <<Bin1/binary,17>>, %% 5 !!! {Bin4,Bin3} %% 6]]></code> - <p>line by line.</p> - - <p>The first line (marked with the <c>%% 1</c> comment), assigns + <list type="bulleted"> + <item>Line 1 (marked with the <c>%% 1</c> comment), assigns a <seealso marker="#heap_binary">heap binary</seealso> to - the variable <c>Bin0</c>.</p> + the <c>Bin0</c> variable.</item> - <p>The second line is an append operation. Since <c>Bin0</c> + <item>Line 2 is an append operation. As <c>Bin0</c> has not been involved in an append operation, a new <seealso marker="#refc_binary">refc binary</seealso> - will be created and the contents of <c>Bin0</c> will be copied - into it. The <em>ProcBin</em> part of the refc binary will have + is created and the contents of <c>Bin0</c> is copied + into it. The <em>ProcBin</em> part of the refc binary has its size set to the size of the data stored in the binary, while - the binary object will have extra space allocated. - The size of the binary object will be either twice the + the binary object has extra space allocated. + The size of the binary object is either twice the size of <c>Bin0</c> or 256, whichever is larger. In this case - it will be 256.</p> + it is 256.</item> - <p>It gets more interesting in the third line. + <item>Line 3 is more interesting. <c>Bin1</c> <em>has</em> been used in an append operation, - and it has 255 bytes of unused storage at the end, so the three new bytes - will be stored there.</p> + and it has 255 bytes of unused storage at the end, so the 3 new + bytes are stored there.</item> - <p>Same thing in the fourth line. There are 252 bytes left, - so there is no problem storing another three bytes.</p> + <item>Line 4. The same applies here. There are 252 bytes left, + so there is no problem storing another 3 bytes.</item> - <p>But in the fifth line something <em>interesting</em> happens. - Note that we don't append to the previous result in <c>Bin3</c>, - but to <c>Bin1</c>. We expect that <c>Bin4</c> will be assigned - the value <c><<0,1,2,3,17>></c>. We also expect that + <item>Line 5. Here, something <em>interesting</em> happens. Notice + that the result is not appended to the previous result in <c>Bin3</c>, + but to <c>Bin1</c>. It is expected that <c>Bin4</c> will be assigned + the value <c><<0,1,2,3,17>></c>. It is also expected that <c>Bin3</c> will retain its value (<c><<0,1,2,3,4,5,6,7,8,9>></c>). - Clearly, the run-time system cannot write the byte <c>17</c> into the binary, + Clearly, the runtime system cannot write byte <c>17</c> into the binary, because that would change the value of <c>Bin3</c> to - <c><<0,1,2,3,4,17,6,7,8,9>></c>.</p> - - <p>What will happen?</p> + <c><<0,1,2,3,4,17,6,7,8,9>></c>.</item> + </list> - <p>The run-time system will see that <c>Bin1</c> is the result + <p>The runtime system sees that <c>Bin1</c> is the result from a previous append operation (not from the latest append operation), - so it will <em>copy</em> the contents of <c>Bin1</c> to a new binary - and reserve extra storage and so on. (We will not explain here how the - run-time system can know that it is not allowed to write into <c>Bin1</c>; + so it <em>copies</em> the contents of <c>Bin1</c> to a new binary, + reserve extra storage, and so on. (Here is not explained how the + runtime system can know that it is not allowed to write into <c>Bin1</c>; it is left as an exercise to the curious reader to figure out how it is done by reading the emulator sources, primarily <c>erl_bits.c</c>.)</p> <section> - <title>Circumstances that force copying</title> + <title>Circumstances That Force Copying</title> <p>The optimization of the binary append operation requires that there is a <em>single</em> ProcBin and a <em>single reference</em> to the ProcBin for the binary. The reason is that the binary object can be - moved (reallocated) during an append operation, and when that happens + moved (reallocated) during an append operation, and when that happens, the pointer in the ProcBin must be updated. If there would be more than one ProcBin pointing to the binary object, it would not be possible to find and update all of them.</p> - <p>Therefore, certain operations on a binary will mark it so that + <p>Therefore, certain operations on a binary mark it so that any future append operation will be forced to copy the binary. In most cases, the binary object will be shrunk at the same time to reclaim the extra space allocated for growing.</p> - <p>When appending to a binary</p> + <p>When appending to a binary as follows, only the binary returned + from the latest append operation will support further cheap append + operations:</p> <code type="erl"><![CDATA[ Bin = <<Bin0,...>>]]></code> - <p>only the binary returned from the latest append operation will - support further cheap append operations. In the code fragment above, + <p>In the code fragment in the beginning of this section, appending to <c>Bin</c> will be cheap, while appending to <c>Bin0</c> will force the creation of a new binary and copying of the contents of <c>Bin0</c>.</p> <p>If a binary is sent as a message to a process or port, the binary will be shrunk and any further append operation will copy the binary - data into a new binary. For instance, in the following code fragment</p> + data into a new binary. For example, in the following code fragment + <c>Bin1</c> will be copied in the third line:</p> <code type="erl"><![CDATA[ Bin1 = <<Bin0,...>>, @@ -234,12 +258,12 @@ PortOrPid ! Bin1, Bin = <<Bin1,...>> %% Bin1 will be COPIED ]]></code> - <p><c>Bin1</c> will be copied in the third line.</p> - - <p>The same thing happens if you insert a binary into an <em>ets</em> - table or send it to a port using <c>erlang:port_command/2</c> or pass it to + <p>The same happens if you insert a binary into an Ets + table, send it to a port using <c>erlang:port_command/2</c>, or + pass it to <seealso marker="erts:erl_nif#enif_inspect_binary">enif_inspect_binary</seealso> in a NIF.</p> + <p>Matching a binary will also cause it to shrink and the next append operation will copy the binary data:</p> @@ -249,22 +273,23 @@ Bin1 = <<Bin0,...>>, Bin = <<Bin1,...>> %% Bin1 will be COPIED ]]></code> - <p>The reason is that a <seealso marker="#match_context">match context</seealso> + <p>The reason is that a + <seealso marker="#match_context">match context</seealso> contains a direct pointer to the binary data.</p> - <p>If a process simply keeps binaries (either in "loop data" or in the process - dictionary), the garbage collector may eventually shrink the binaries. - If only one such binary is kept, it will not be shrunk. If the process later - appends to a binary that has been shrunk, the binary object will be reallocated - to make place for the data to be appended.</p> + <p>If a process simply keeps binaries (either in "loop data" or in the + process + dictionary), the garbage collector can eventually shrink the binaries. + If only one such binary is kept, it will not be shrunk. If the process + later appends to a binary that has been shrunk, the binary object will + be reallocated to make place for the data to be appended.</p> </section> - </section> <section> - <title>Matching binaries</title> + <title>Matching Binaries</title> - <p>We will revisit the example shown earlier</p> + <p>Let us revisit the example in the beginning of the previous section:</p> <p><em>DO</em> (in R12B)</p> <code type="erl"><![CDATA[ @@ -272,36 +297,35 @@ my_binary_to_list(<<H,T/binary>>) -> [H|my_binary_to_list(T)]; my_binary_to_list(<<>>) -> [].]]></code> - <p>too see what is happening under the hood.</p> - - <p>The very first time <c>my_binary_to_list/1</c> is called, + <p>The first time <c>my_binary_to_list/1</c> is called, a <seealso marker="#match_context">match context</seealso> - will be created. The match context will point to the first - byte of the binary. One byte will be matched out and the match context - will be updated to point to the second byte in the binary.</p> + is created. The match context points to the first + byte of the binary. 1 byte is matched out and the match context + is updated to point to the second byte in the binary.</p> - <p>In R11B, at this point a <seealso marker="#sub_binary">sub binary</seealso> + <p>In R11B, at this point a + <seealso marker="#sub_binary">sub binary</seealso> would be created. In R12B, the compiler sees that there is no point in creating a sub binary, because there will soon be a call to a function (in this case, - to <c>my_binary_to_list/1</c> itself) that will immediately + to <c>my_binary_to_list/1</c> itself) that immediately will create a new match context and discard the sub binary.</p> - <p>Therefore, in R12B, <c>my_binary_to_list/1</c> will call itself + <p>Therefore, in R12B, <c>my_binary_to_list/1</c> calls itself with the match context instead of with a sub binary. The instruction - that initializes the matching operation will basically do nothing + that initializes the matching operation basically does nothing when it sees that it was passed a match context instead of a binary.</p> <p>When the end of the binary is reached and the second clause matches, the match context will simply be discarded (removed in the next - garbage collection, since there is no longer any reference to it).</p> + garbage collection, as there is no longer any reference to it).</p> <p>To summarize, <c>my_binary_to_list/1</c> in R12B only needs to create <em>one</em> match context and no sub binaries. In R11B, if the binary contains <em>N</em> bytes, <em>N+1</em> match contexts and <em>N</em> - sub binaries will be created.</p> + sub binaries are created.</p> - <p>In R11B, the fastest way to match binaries is:</p> + <p>In R11B, the fastest way to match binaries is as follows:</p> <p><em>DO NOT</em> (in R12B)</p> <code type="erl"><![CDATA[ @@ -317,13 +341,14 @@ my_complicated_binary_to_list(Bin, Skip) -> end.]]></code> <p>This function cleverly avoids building sub binaries, but it cannot - avoid building a match context in each recursion step. Therefore, in both R11B and R12B, + avoid building a match context in each recursion step. + Therefore, in both R11B and R12B, <c>my_complicated_binary_to_list/1</c> builds <em>N+1</em> match - contexts. (In a future release, the compiler might be able to generate code - that reuses the match context, but don't hold your breath.)</p> + contexts. (In a future Erlang/OTP release, the compiler might be able + to generate code that reuses the match context.)</p> - <p>Returning to <c>my_binary_to_list/1</c>, note that the match context was - discarded when the entire binary had been traversed. What happens if + <p>Returning to <c>my_binary_to_list/1</c>, notice that the match context + was discarded when the entire binary had been traversed. What happens if the iteration stops before it has reached the end of the binary? Will the optimization still work?</p> @@ -336,29 +361,23 @@ after_zero(<<>>) -> <<>>. ]]></code> - <p>Yes, it will. The compiler will remove the building of the sub binary in the - second clause</p> + <p>Yes, it will. The compiler will remove the building of the sub binary in + the second clause:</p> <code type="erl"><![CDATA[ -. -. -. +... after_zero(<<_,T/binary>>) -> after_zero(T); -. -. -.]]></code> +...]]></code> - <p>but will generate code that builds a sub binary in the first clause</p> + <p>But it will generate code that builds a sub binary in the first clause:</p> <code type="erl"><![CDATA[ after_zero(<<0,T/binary>>) -> T; -. -. -.]]></code> +...]]></code> - <p>Therefore, <c>after_zero/1</c> will build one match context and one sub binary + <p>Therefore, <c>after_zero/1</c> builds one match context and one sub binary (assuming it is passed a binary that contains a zero byte).</p> <p>Code like the following will also be optimized:</p> @@ -371,12 +390,14 @@ all_but_zeroes_to_list(<<0,T/binary>>, Acc, Remaining) -> all_but_zeroes_to_list(<<Byte,T/binary>>, Acc, Remaining) -> all_but_zeroes_to_list(T, [Byte|Acc], Remaining-1).]]></code> - <p>The compiler will remove building of sub binaries in the second and third clauses, - and it will add an instruction to the first clause that will convert <c>Buffer</c> - from a match context to a sub binary (or do nothing if <c>Buffer</c> already is a binary).</p> + <p>The compiler removes building of sub binaries in the second and third + clauses, and it adds an instruction to the first clause that converts + <c>Buffer</c> from a match context to a sub binary (or do nothing if + <c>Buffer</c> is a binary already).</p> - <p>Before you begin to think that the compiler can optimize any binary patterns, - here is a function that the compiler (currently, at least) is not able to optimize:</p> + <p>Before you begin to think that the compiler can optimize any binary + patterns, the following function cannot be optimized by the compiler + (currently, at least):</p> <code type="erl"><![CDATA[ non_opt_eq([H|T1], <<H,T2/binary>>) -> @@ -386,43 +407,43 @@ non_opt_eq([_|_], <<_,_/binary>>) -> non_opt_eq([], <<>>) -> true.]]></code> - <p>It was briefly mentioned earlier that the compiler can only delay creation of - sub binaries if it can be sure that the binary will not be shared. In this case, - the compiler cannot be sure.</p> + <p>It was mentioned earlier that the compiler can only delay creation of + sub binaries if it knows that the binary will not be shared. In this case, + the compiler cannot know.</p> - <p>We will soon show how to rewrite <c>non_opt_eq/2</c> so that the delayed sub binary - optimization can be applied, and more importantly, we will show how you can find out - whether your code can be optimized.</p> + <p>Soon it is shown how to rewrite <c>non_opt_eq/2</c> so that the delayed + sub binary optimization can be applied, and more importantly, it is shown + how you can find out whether your code can be optimized.</p> <section> - <title>The bin_opt_info option</title> + <title>Option bin_opt_info</title> <p>Use the <c>bin_opt_info</c> option to have the compiler print a lot of - information about binary optimizations. It can be given either to the compiler or - <c>erlc</c></p> + information about binary optimizations. It can be given either to the + compiler or <c>erlc</c>:</p> <code type="erl"><![CDATA[ erlc +bin_opt_info Mod.erl]]></code> - <p>or passed via an environment variable</p> + <p>or passed through an environment variable:</p> <code type="erl"><![CDATA[ export ERL_COMPILER_OPTIONS=bin_opt_info]]></code> - <p>Note that the <c>bin_opt_info</c> is not meant to be a permanent option added - to your <c>Makefile</c>s, because it is not possible to eliminate all messages that - it generates. Therefore, passing the option through the environment is in most cases - the most practical approach.</p> + <p>Notice that the <c>bin_opt_info</c> is not meant to be a permanent + option added to your <c>Makefile</c>s, because all messages that it + generates cannot be eliminated. Therefore, passing the option through + the environment is in most cases the most practical approach.</p> - <p>The warnings will look like this:</p> + <p>The warnings look as follows:</p> <code type="erl"><![CDATA[ ./efficiency_guide.erl:60: Warning: NOT OPTIMIZED: sub binary is used or returned ./efficiency_guide.erl:62: Warning: OPTIMIZED: creation of sub binary delayed]]></code> - <p>To make it clearer exactly what code the warnings refer to, - in the examples that follow, the warnings are inserted as comments - after the clause they refer to:</p> + <p>To make it clearer exactly what code the warnings refer to, the + warnings in the following examples are inserted as comments + after the clause they refer to, for example:</p> <code type="erl"><![CDATA[ after_zero(<<0,T/binary>>) -> @@ -434,12 +455,12 @@ after_zero(<<_,T/binary>>) -> after_zero(<<>>) -> <<>>.]]></code> - <p>The warning for the first clause tells us that it is not possible to - delay the creation of a sub binary, because it will be returned. - The warning for the second clause tells us that a sub binary will not be + <p>The warning for the first clause says that the creation of a sub + binary cannot be delayed, because it will be returned. + The warning for the second clause says that a sub binary will not be created (yet).</p> - <p>It is time to revisit the earlier example of the code that could not + <p>Let us revisit the earlier example of the code that could not be optimized and find out why:</p> <code type="erl"><![CDATA[ @@ -456,16 +477,16 @@ non_opt_eq([_|_], <<_,_/binary>>) -> non_opt_eq([], <<>>) -> true.]]></code> - <p>The compiler emitted two warnings. The <c>INFO</c> warning refers to the function - <c>non_opt_eq/2</c> as a callee, indicating that any functions that call <c>non_opt_eq/2</c> - will not be able to make delayed sub binary optimization. - There is also a suggestion to change argument order. - The second warning (that happens to refer to the same line) refers to the construction of - the sub binary itself.</p> + <p>The compiler emitted two warnings. The <c>INFO</c> warning refers + to the function <c>non_opt_eq/2</c> as a callee, indicating that any + function that call <c>non_opt_eq/2</c> cannot make delayed sub binary + optimization. There is also a suggestion to change argument order. + The second warning (that happens to refer to the same line) refers to + the construction of the sub binary itself.</p> - <p>We will soon show another example that should make the distinction between <c>INFO</c> - and <c>NOT OPTIMIZED</c> warnings somewhat clearer, but first we will heed the suggestion - to change argument order:</p> + <p>Soon another example will show the difference between the + <c>INFO</c> and <c>NOT OPTIMIZED</c> warnings somewhat clearer, but + let us first follow the suggestion to change argument order:</p> <code type="erl"><![CDATA[ opt_eq(<<H,T1/binary>>, [H|T2]) -> @@ -485,15 +506,13 @@ match_body([0|_], <<H,_/binary>>) -> %% sub binary optimization; %% SUGGEST changing argument order done; -. -. -.]]></code> +...]]></code> <p>The warning means that <em>if</em> there is a call to <c>match_body/2</c> (from another clause in <c>match_body/2</c> or another function), the - delayed sub binary optimization will not be possible. There will be additional - warnings for any place where a sub binary is matched out at the end of and - passed as the second argument to <c>match_body/2</c>. For instance:</p> + delayed sub binary optimization will not be possible. More warnings will + occur for any place where a sub binary is matched out at the end of and + passed as the second argument to <c>match_body/2</c>, for example:</p> <code type="erl"><![CDATA[ match_head(List, <<_:10,Data/binary>>) -> @@ -504,10 +523,10 @@ match_head(List, <<_:10,Data/binary>>) -> </section> <section> - <title>Unused variables</title> + <title>Unused Variables</title> - <p>The compiler itself figures out if a variable is unused. The same - code is generated for each of the following functions</p> + <p>The compiler figures out if a variable is unused. The same + code is generated for each of the following functions:</p> <code type="erl"><![CDATA[ count1(<<_,T/binary>>, Count) -> count1(T, Count+1); @@ -519,11 +538,9 @@ count2(<<>>, Count) -> Count. count3(<<_H,T/binary>>, Count) -> count3(T, Count+1); count3(<<>>, Count) -> Count.]]></code> - <p>In each iteration, the first 8 bits in the binary will be skipped, not matched out.</p> - + <p>In each iteration, the first 8 bits in the binary will be skipped, + not matched out.</p> </section> - </section> - </chapter> diff --git a/system/doc/efficiency_guide/commoncaveats.xml b/system/doc/efficiency_guide/commoncaveats.xml index 551b0a03e6..71991d342f 100644 --- a/system/doc/efficiency_guide/commoncaveats.xml +++ b/system/doc/efficiency_guide/commoncaveats.xml @@ -18,7 +18,6 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> <title>Common Caveats</title> @@ -29,49 +28,50 @@ <file>commoncaveats.xml</file> </header> - <p>Here we list a few modules and BIFs to watch out for, and not only + <p>This section lists a few modules and BIFs to watch out for, not only from a performance point of view.</p> <section> - <title>The timer module</title> + <title>Timer Module</title> <p>Creating timers using <seealso marker="erts:erlang#erlang:send_after/3">erlang:send_after/3</seealso> - and <seealso marker="erts:erlang#erlang:start_timer/3">erlang:start_timer/3</seealso> + and + <seealso marker="erts:erlang#erlang:start_timer/3">erlang:start_timer/3</seealso> +, is much more efficient than using the timers provided by the - <seealso marker="stdlib:timer">timer</seealso> module. The - <c>timer</c> module uses a separate process to manage the timers, - and that process can easily become overloaded if many processes + <seealso marker="stdlib:timer">timer</seealso> module in STDLIB. + The <c>timer</c> module uses a separate process to manage the timers. + That process can easily become overloaded if many processes create and cancel timers frequently (especially when using the SMP emulator).</p> - <p>The functions in the <c>timer</c> module that do not manage timers (such as - <c>timer:tc/3</c> or <c>timer:sleep/1</c>), do not call the timer-server process - and are therefore harmless.</p> + <p>The functions in the <c>timer</c> module that do not manage timers + (such as <c>timer:tc/3</c> or <c>timer:sleep/1</c>), do not call the + timer-server process and are therefore harmless.</p> </section> <section> <title>list_to_atom/1</title> - <p>Atoms are not garbage-collected. Once an atom is created, it will never - be removed. The emulator will terminate if the limit for the number - of atoms (1048576 by default) is reached.</p> + <p>Atoms are not garbage-collected. Once an atom is created, it is never + removed. The emulator terminates if the limit for the number + of atoms (1,048,576 by default) is reached.</p> - <p>Therefore, converting arbitrary input strings to atoms could be - dangerous in a system that will run continuously. - If only certain well-defined atoms are allowed as input, you can use + <p>Therefore, converting arbitrary input strings to atoms can be + dangerous in a system that runs continuously. + If only certain well-defined atoms are allowed as input, <seealso marker="erts:erlang#list_to_existing_atom/1">list_to_existing_atom/1</seealso> + can be used to to guard against a denial-of-service attack. (All atoms that are allowed - must have been created earlier, for instance by simply using all of them + must have been created earlier, for example, by simply using all of them in a module and loading that module.)</p> <p>Using <c>list_to_atom/1</c> to construct an atom that is passed to - <c>apply/3</c> like this</p> - + <c>apply/3</c> as follows, is quite expensive and not recommended + in time-critical code:</p> <code type="erl"> -apply(list_to_atom("some_prefix"++Var), foo, Args)</code> - - <p>is quite expensive and is not recommended in time-critical code.</p> +apply(list_to_atom("some_prefix"++Var), foo, Args)</code> </section> <section> @@ -81,25 +81,25 @@ apply(list_to_atom("some_prefix"++Var), foo, Args)</code> length of the list, as opposed to <c>tuple_size/1</c>, <c>byte_size/1</c>, and <c>bit_size/1</c>, which all execute in constant time.</p> - <p>Normally you don't have to worry about the speed of <c>length/1</c>, - because it is efficiently implemented in C. In time critical-code, though, - you might want to avoid it if the input list could potentially be very long.</p> + <p>Normally, there is no need to worry about the speed of <c>length/1</c>, + because it is efficiently implemented in C. In time-critical code, + you might want to avoid it if the input list could potentially be very + long.</p> <p>Some uses of <c>length/1</c> can be replaced by matching. - For instance, this code</p> - + For example, the following code:</p> <code type="erl"> foo(L) when length(L) >= 3 -> ...</code> - <p>can be rewritten to</p> + <p>can be rewritten to:</p> <code type="erl"> foo([_,_,_|_]=L) -> ...</code> - <p>(One slight difference is that <c>length(L)</c> will fail if the <c>L</c> - is an improper list, while the pattern in the second code fragment will - accept an improper list.)</p> + <p>One slight difference is that <c>length(L)</c> fails if <c>L</c> + is an improper list, while the pattern in the second code fragment + accepts an improper list.</p> </section> <section> @@ -107,50 +107,49 @@ foo([_,_,_|_]=L) -> <p><seealso marker="erts:erlang#setelement/3">setelement/3</seealso> copies the tuple it modifies. Therefore, updating a tuple in a loop - using <c>setelement/3</c> will create a new copy of the tuple every time.</p> + using <c>setelement/3</c> creates a new copy of the tuple every time.</p> <p>There is one exception to the rule that the tuple is copied. If the compiler clearly can see that destructively updating the tuple would - give exactly the same result as if the tuple was copied, the call to - <c>setelement/3</c> will be replaced with a special destructive setelement - instruction. In the following code sequence</p> - + give the same result as if the tuple was copied, the call to + <c>setelement/3</c> is replaced with a special destructive <c>setelement</c> + instruction. In the following code sequence, the first <c>setelement/3</c> + call copies the tuple and modifies the ninth element:</p> <code type="erl"> multiple_setelement(T0) -> T1 = setelement(9, T0, bar), T2 = setelement(7, T1, foobar), setelement(5, T2, new_value).</code> - <p>the first <c>setelement/3</c> call will copy the tuple and modify the - ninth element. The two following <c>setelement/3</c> calls will modify + <p>The two following <c>setelement/3</c> calls modify the tuple in place.</p> - <p>For the optimization to be applied, <em>all</em> of the followings conditions + <p>For the optimization to be applied, <em>all</em> the followings conditions must be true:</p> <list type="bulleted"> <item>The indices must be integer literals, not variables or expressions.</item> <item>The indices must be given in descending order.</item> - <item>There must be no calls to other function in between the calls to + <item>There must be no calls to another function in between the calls to <c>setelement/3</c>.</item> <item>The tuple returned from one <c>setelement/3</c> call must only be used in the subsequent call to <c>setelement/3</c>.</item> </list> - <p>If it is not possible to structure the code as in the <c>multiple_setelement/1</c> + <p>If the code cannot be structured as in the <c>multiple_setelement/1</c> example, the best way to modify multiple elements in a large tuple is to - convert the tuple to a list, modify the list, and convert the list back to + convert the tuple to a list, modify the list, and convert it back to a tuple.</p> </section> <section> <title>size/1</title> - <p><c>size/1</c> returns the size for both tuples and binary.</p> + <p><c>size/1</c> returns the size for both tuples and binaries.</p> - <p>Using the new BIFs <c>tuple_size/1</c> and <c>byte_size/1</c> introduced - in R12B gives the compiler and run-time system more opportunities for - optimization. A further advantage is that the new BIFs could help Dialyzer + <p>Using the new BIFs <c>tuple_size/1</c> and <c>byte_size/1</c>, introduced + in R12B, gives the compiler and the runtime system more opportunities for + optimization. Another advantage is that the new BIFs can help Dialyzer to find more bugs in your program.</p> </section> @@ -159,22 +158,21 @@ multiple_setelement(T0) -> <p>It is usually more efficient to split a binary using matching instead of calling the <c>split_binary/2</c> function. Furthermore, mixing bit syntax matching and <c>split_binary/2</c> - may prevent some optimizations of bit syntax matching.</p> + can prevent some optimizations of bit syntax matching.</p> <p><em>DO</em></p> <code type="none"><![CDATA[ <<Bin1:Num/binary,Bin2/binary>> = Bin,]]></code> <p><em>DO NOT</em></p> <code type="none"> - {Bin1,Bin2} = split_binary(Bin, Num) - </code> + {Bin1,Bin2} = split_binary(Bin, Num)</code> </section> <section> - <title>The '--' operator</title> - <p>Note that the '<c>--</c>' operator has a complexity - proportional to the product of the length of its operands, - meaning that it will be very slow if both of its operands + <title>Operator "--"</title> + <p>The "<c>--</c>" operator has a complexity + proportional to the product of the length of its operands. + This means that the operator is very slow if both of its operands are long lists:</p> <p><em>DO NOT</em></p> @@ -182,42 +180,39 @@ multiple_setelement(T0) -> HugeList1 -- HugeList2]]></code> <p>Instead use the <seealso marker="stdlib:ordsets">ordsets</seealso> - module:</p> + module in STDLIB:</p> <p><em>DO</em></p> <code type="none"> HugeSet1 = ordsets:from_list(HugeList1), HugeSet2 = ordsets:from_list(HugeList2), - ordsets:subtract(HugeSet1, HugeSet2) - </code> + ordsets:subtract(HugeSet1, HugeSet2)</code> - <p>Obviously, that code will not work if the original order + <p>Obviously, that code does not work if the original order of the list is important. If the order of the list must be - preserved, do like this:</p> + preserved, do as follows:</p> <p><em>DO</em></p> <code type="none"><![CDATA[ Set = gb_sets:from_list(HugeList2), [E || E <- HugeList1, not gb_sets:is_element(E, Set)]]]></code> - <p>Subtle note 1: This code behaves differently from '<c>--</c>' - if the lists contain duplicate elements. (One occurrence - of an element in HugeList2 will remove <em>all</em> + <note><p>This code behaves differently from "<c>--</c>" + if the lists contain duplicate elements (one occurrence + of an element in HugeList2 removes <em>all</em> occurrences in HugeList1.)</p> + <p>Also, this code compares lists elements using the + "<c>==</c>" operator, while "<c>--</c>" uses the "<c>=:=</c>" operator. + If that difference is important, <c>sets</c> can be used instead of + <c>gb_sets</c>, but <c>sets:from_list/1</c> is much + slower than <c>gb_sets:from_list/1</c> for long lists.</p></note> - <p>Subtle note 2: This code compares lists elements using the - '<c>==</c>' operator, while '<c>--</c>' uses the '<c>=:=</c>'. If - that difference is important, <c>sets</c> can be used instead of - <c>gb_sets</c>, but note that <c>sets:from_list/1</c> is much - slower than <c>gb_sets:from_list/1</c> for long lists.</p> - - <p>Using the '<c>--</c>' operator to delete an element + <p>Using the "<c>--</c>" operator to delete an element from a list is not a performance problem:</p> <p><em>OK</em></p> <code type="none"> - HugeList1 -- [Element] - </code> + HugeList1 -- [Element]</code> </section> diff --git a/system/doc/efficiency_guide/drivers.xml b/system/doc/efficiency_guide/drivers.xml index dfc49bdf21..33d6333e7d 100644 --- a/system/doc/efficiency_guide/drivers.xml +++ b/system/doc/efficiency_guide/drivers.xml @@ -18,7 +18,6 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> <title>Drivers</title> @@ -29,26 +28,26 @@ <file>drivers.xml</file> </header> - <p>This chapter provides a (very) brief overview on how to write efficient - drivers. It is assumed that you already have a good understanding of + <p>This section provides a brief overview on how to write efficient drivers.</p> + <p>It is assumed that you have a good understanding of drivers.</p> <section> - <title>Drivers and concurrency</title> + <title>Drivers and Concurrency</title> - <p>The run-time system will always take a lock before running + <p>The runtime system always takes a lock before running any code in a driver.</p> - <p>By default, that lock will be at the driver level, meaning that + <p>By default, that lock is at the driver level, that is, if several ports have been opened to the same driver, only code for one port at the same time can be running.</p> - <p>A driver can be configured to instead have one lock for each port.</p> + <p>A driver can be configured to have one lock for each port instead.</p> - <p>If a driver is used in a functional way (i.e. it holds no state, + <p>If a driver is used in a functional way (that is, holds no state, but only does some heavy calculation and returns a result), several - ports with registered names can be opened beforehand and the port to - be used can be chosen based on the scheduler ID like this:</p> + ports with registered names can be opened beforehand, and the port to + be used can be chosen based on the scheduler ID as follows:</p> <code type="none"> -define(PORT_NAMES(), @@ -67,82 +66,82 @@ client_port() -> </section> <section> - <title>Avoiding copying of binaries when calling a driver</title> + <title>Avoiding Copying Binaries When Calling a Driver</title> <p>There are basically two ways to avoid copying a binary that is - sent to a driver.</p> - - <p>If the <c>Data</c> argument for - <seealso marker="erts:erlang#port_control/3">port_control/3</seealso> - is a binary, the driver will be passed a pointer to the contents of - the binary and the binary will not be copied. - If the <c>Data</c> argument is an iolist (list of binaries and lists), - all binaries in the iolist will be copied.</p> - - <p>Therefore, if you want to send both a pre-existing binary and some - additional data to a driver without copying the binary, you must call - <c>port_control/3</c> twice; once with the binary and once with the - additional data. However, that will only work if there is only one - process communicating with the port (because otherwise another process - could call the driver in-between the calls).</p> - - <p>Another way to avoid copying binaries is to implement an <c>outputv</c> - callback (instead of an <c>output</c> callback) in the driver. - If a driver has an <c>outputv</c> callback, refc binaries passed - in an iolist in the <c>Data</c> argument for - <seealso marker="erts:erlang#port_command/2">port_command/2</seealso> - will be passed as references to the driver.</p> + sent to a driver:</p> + + <list type="bulleted"> + <item><p>If the <c>Data</c> argument for + <seealso marker="erts:erlang#port_control/3">port_control/3</seealso> + is a binary, the driver will be passed a pointer to the contents of + the binary and the binary will not be copied. If the <c>Data</c> + argument is an iolist (list of binaries and lists), all binaries in + the iolist will be copied.</p> + + <p>Therefore, if you want to send both a pre-existing binary and some + extra data to a driver without copying the binary, you must call + <c>port_control/3</c> twice; once with the binary and once with the + extra data. However, that will only work if there is only one + process communicating with the port (because otherwise another process + can call the driver in-between the calls).</p></item> + + <item><p>Implement an <c>outputv</c> callback (instead of an + <c>output</c> callback) in the driver. If a driver has an + <c>outputv</c> callback, refc binaries passed in an iolist + in the <c>Data</c> argument for + <seealso marker="erts:erlang#port_command/2">port_command/2</seealso> + will be passed as references to the driver.</p></item> + </list> </section> <section> - <title>Returning small binaries from a driver</title> + <title>Returning Small Binaries from a Driver</title> - <p>The run-time system can represent binaries up to 64 bytes as - heap binaries. They will always be copied when sent in a messages, - but they will require less memory if they are not sent to another + <p>The runtime system can represent binaries up to 64 bytes as + heap binaries. They are always copied when sent in messages, + but they require less memory if they are not sent to another process and garbage collection is cheaper.</p> - <p>If you know that the binaries you return are always small, - you should use driver API calls that do not require a pre-allocated - binary, for instance + <p>If you know that the binaries you return are always small, you + are advised to use driver API calls that do not require a pre-allocated + binary, for example, <seealso marker="erts:erl_driver#driver_output">driver_output()</seealso> - or - <seealso marker="erts:erl_driver#erl_drv_output_term">erl_drv_output_term()</seealso> + or + <seealso marker="erts:erl_driver#erl_drv_output_term">erl_drv_output_term()</seealso>, using the <c>ERL_DRV_BUF2BINARY</c> format, - to allow the run-time to construct a heap binary.</p> + to allow the runtime to construct a heap binary.</p> </section> <section> - <title>Returning big binaries without copying from a driver</title> + <title>Returning Large Binaries without Copying from a Driver</title> - <p>To avoid copying data when a big binary is sent or returned from + <p>To avoid copying data when a large binary is sent or returned from the driver to an Erlang process, the driver must first allocate the binary and then send it to an Erlang process in some way.</p> - <p>Use <seealso marker="erts:erl_driver#driver_alloc_binary">driver_alloc_binary()</seealso> to allocate a binary.</p> + <p>Use + <seealso marker="erts:erl_driver#driver_alloc_binary">driver_alloc_binary()</seealso> + to allocate a binary.</p> <p>There are several ways to send a binary created with - <c>driver_alloc_binary()</c>.</p> + <c>driver_alloc_binary()</c>:</p> <list type="bulleted"> - <item><p>From the <c>control</c> callback, a binary can be returned provided - that - <seealso marker="erts:erl_driver#set_port_control_flags">set_port_control_flags()</seealso> - has been called with the flag value <c>PORT_CONTROL_FLAG_BINARY</c>.</p> - </item> + <item>From the <c>control</c> callback, a binary can be returned if + <seealso marker="erts:erl_driver#set_port_control_flags">set_port_control_flags()</seealso> + has been called with the flag value <c>PORT_CONTROL_FLAG_BINARY</c>.</item> - <item><p>A single binary can be sent with - <seealso marker="erts:erl_driver#driver_output_binary">driver_output_binary()</seealso>.</p></item> + <item>A single binary can be sent with + <seealso marker="erts:erl_driver#driver_output_binary">driver_output_binary()</seealso>.</item> - <item><p>Using + <item>Using <seealso marker="erts:erl_driver#erl_drv_output_term">erl_drv_output_term()</seealso> or <seealso marker="erts:erl_driver#erl_drv_send_term">erl_drv_send_term()</seealso>, - a binary can be included in an Erlang term.</p> - </item> + a binary can be included in an Erlang term.</item> </list> </section> - </chapter> diff --git a/system/doc/efficiency_guide/functions.xml b/system/doc/efficiency_guide/functions.xml index ec1a45eaa9..bd23c9d90d 100644 --- a/system/doc/efficiency_guide/functions.xml +++ b/system/doc/efficiency_guide/functions.xml @@ -18,7 +18,6 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> <title>Functions</title> @@ -30,17 +29,18 @@ </header> <section> - <title>Pattern matching</title> - <p>Pattern matching in function head and in <c>case</c> and <c>receive</c> - clauses are optimized by the compiler. With a few exceptions, there is nothing - to gain by rearranging clauses.</p> + <title>Pattern Matching</title> + <p>Pattern matching in function head as well as in <c>case</c> and + <c>receive</c> clauses are optimized by the compiler. With a few + exceptions, there is nothing to gain by rearranging clauses.</p> <p>One exception is pattern matching of binaries. The compiler - will not rearrange clauses that match binaries. Placing the - clause that matches against the empty binary <em>last</em> will usually - be slightly faster than placing it <em>first</em>.</p> + does not rearrange clauses that match binaries. Placing the + clause that matches against the empty binary <em>last</em> is usually + slightly faster than placing it <em>first</em>.</p> - <p>Here is a rather contrived example to show another exception:</p> + <p>The following is a rather unnatural example to show another + exception:</p> <p><em>DO NOT</em></p> <code type="erl"> @@ -53,27 +53,30 @@ atom_map1(five) -> 5; atom_map1(six) -> 6.</code> <p>The problem is the clause with the variable <c>Int</c>. - Since a variable can match anything, including the atoms - <c>four</c>, <c>five</c>, and <c>six</c> that the following clauses - also will match, the compiler must generate sub-optimal code that will - execute as follows:</p> + As a variable can match anything, including the atoms + <c>four</c>, <c>five</c>, and <c>six</c>, which the following clauses + also match, the compiler must generate suboptimal code that + executes as follows:</p> - <p>First the input value is compared to <c>one</c>, <c>two</c>, and + <list type="bulleted"> + <item>First, the input value is compared to <c>one</c>, <c>two</c>, and <c>three</c> (using a single instruction that does a binary search; thus, quite efficient even if there are many values) to select which - one of the first three clauses to execute (if any).</p> + one of the first three clauses to execute (if any).</item> + + <item>>If none of the first three clauses match, the fourth clause + match as a variable always matches.</item> - <p>If none of the first three clauses matched, the fourth clause - will match since a variable always matches. If the guard test - <c>is_integer(Int)</c> succeeds, the fourth clause will be - executed.</p> + <item>If the guard test <c>is_integer(Int)</c> succeeds, the fourth + clause is executed.</item> - <p>If the guard test failed, the input value is compared to + <item>If the guard test fails, the input value is compared to <c>four</c>, <c>five</c>, and <c>six</c>, and the appropriate clause - is selected. (There will be a <c>function_clause</c> exception if - none of the values matched.)</p> + is selected. (There is a <c>function_clause</c> exception if none of + the values matched.)</item> + </list> - <p>Rewriting to either</p> + <p>Rewriting to either:</p> <p><em>DO</em></p> <code type="erl"><![CDATA[ @@ -85,7 +88,7 @@ atom_map2(five) -> 5; atom_map2(six) -> 6; atom_map2(Int) when is_integer(Int) -> Int.]]></code> - <p>or</p> + <p>or:</p> <p><em>DO</em></p> <code type="erl"><![CDATA[ @@ -97,9 +100,9 @@ atom_map3(four) -> 4; atom_map3(five) -> 5; atom_map3(six) -> 6.]]></code> - <p>will give slightly more efficient matching code.</p> + <p>gives slightly more efficient matching code.</p> - <p>Here is a less contrived example:</p> + <p>Another example:</p> <p><em>DO NOT</em></p> <code type="erl"><![CDATA[ @@ -116,7 +119,8 @@ map_pairs1(Map, [X|Xs], [Y|Ys]) -> match anything, the compiler is not allowed to rearrange the clauses, but must generate code that matches them in the order written.</p> - <p>If the function is rewritten like this</p> + <p>If the function is rewritten as follows, the compiler is free to + rearrange the clauses:</p> <p><em>DO</em></p> <code type="erl"><![CDATA[ @@ -127,8 +131,7 @@ map_pairs2(_Map, [_|_]=Xs, [] ) -> map_pairs2(Map, [X|Xs], [Y|Ys]) -> [Map(X, Y)|map_pairs2(Map, Xs, Ys)].]]></code> - <p>the compiler is free to rearrange the clauses. It will generate code - similar to this</p> + <p>The compiler will generate code similar to this:</p> <p><em>DO NOT (already done by the compiler)</em></p> <code type="erl"><![CDATA[ @@ -145,31 +148,35 @@ explicit_map_pairs(Map, Xs0, Ys0) -> Ys0 end.]]></code> - <p>which should be slightly faster for presumably the most common case + <p>This is slightly faster for probably the most common case that the input lists are not empty or very short. - (Another advantage is that Dialyzer is able to deduce a better type - for the variable <c>Xs</c>.)</p> + (Another advantage is that Dialyzer can deduce a better type + for the <c>Xs</c> variable.)</p> </section> <section> - <title>Function Calls </title> + <title>Function Calls</title> - <p>Here is an intentionally rough guide to the relative costs of - different kinds of calls. It is based on benchmark figures run on + <p>This is an intentionally rough guide to the relative costs of + different calls. It is based on benchmark figures run on Solaris/Sparc:</p> <list type="bulleted"> <item>Calls to local or external functions (<c>foo()</c>, <c>m:foo()</c>) - are the fastest kind of calls.</item> + are the fastest calls.</item> + <item>Calling or applying a fun (<c>Fun()</c>, <c>apply(Fun, [])</c>) - is about <em>three times</em> as expensive as calling a local function.</item> + is about <em>three times</em> as expensive as calling a local + function.</item> + <item>Applying an exported function (<c>Mod:Name()</c>, - <c>apply(Mod, Name, [])</c>) is about twice as expensive as calling a fun, - or about <em>six times</em> as expensive as calling a local function.</item> + <c>apply(Mod, Name, [])</c>) is about twice as expensive as calling + a fun or about <em>six times</em> as expensive as calling a local + function.</item> </list> <section> - <title>Notes and implementation details</title> + <title>Notes and Implementation Details</title> <p>Calling and applying a fun does not involve any hash-table lookup. A fun contains an (indirect) pointer to the function that implements @@ -178,42 +185,44 @@ explicit_map_pairs(Map, Xs0, Ys0) -> <warning><p><em>Tuples are not fun(s)</em>. A "tuple fun", <c>{Module,Function}</c>, is not a fun. The cost for calling a "tuple fun" is similar to that - of <c>apply/3</c> or worse. Using "tuple funs" is <em>strongly discouraged</em>, - as they may not be supported in a future release, - and because there exists a superior alternative since the R10B - release, namely the <c>fun Module:Function/Arity</c> syntax.</p></warning> + of <c>apply/3</c> or worse. + Using "tuple funs" is <em>strongly discouraged</em>, + as they might not be supported in a future Erlang/OTP release, + and because there exists a superior alternative from R10B, + namely the <c>fun Module:Function/Arity</c> syntax.</p></warning> <p><c>apply/3</c> must look up the code for the function to execute - in a hash table. Therefore, it will always be slower than a + in a hash table. It is therefore always slower than a direct call or a fun call.</p> <p>It no longer matters (from a performance point of view) - whether you write</p> + whether you write:</p> <code type="erl"> Module:Function(Arg1, Arg2)</code> - <p>or</p> + <p>or:</p> <code type="erl"> apply(Module, Function, [Arg1,Arg2])</code> - <p>(The compiler internally rewrites the latter code into the former.)</p> + <p>The compiler internally rewrites the latter code into the + former.</p> - <p>The following code</p> + <p>The following code is slightly slower because the shape of the + list of arguments is unknown at compile time.</p> <code type="erl"> apply(Module, Function, Arguments)</code> - <p>is slightly slower because the shape of the list of arguments - is not known at compile time.</p> </section> </section> <section> - <title>Memory usage in recursion</title> - <p>When writing recursive functions it is preferable to make them - tail-recursive so that they can execute in constant memory space.</p> + <title>Memory Usage in Recursion</title> + <p>When writing recursive functions, it is preferable to make them + tail-recursive so that they can execute in constant memory space:</p> + <p><em>DO</em></p> <code type="none"> list_length(List) -> @@ -224,13 +233,14 @@ list_length([], AccLen) -> list_length([_|Tail], AccLen) -> list_length(Tail, AccLen + 1). % Tail-recursive</code> + <p><em>DO NOT</em></p> + <code type="none"> list_length([]) -> 0. % Base case list_length([_ | Tail]) -> list_length(Tail) + 1. % Not tail-recursive</code> </section> - </chapter> diff --git a/system/doc/efficiency_guide/introduction.xml b/system/doc/efficiency_guide/introduction.xml index 9726d3ad11..a8360f1cdd 100644 --- a/system/doc/efficiency_guide/introduction.xml +++ b/system/doc/efficiency_guide/introduction.xml @@ -18,7 +18,6 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> <title>Introduction</title> @@ -32,38 +31,39 @@ <section> <title>Purpose</title> - <quote><p>Premature optimization is the root of all evil. -- D.E. Knuth</p></quote> + <quote><p>"Premature optimization is the root of all evil" + (D.E. Knuth)</p></quote> - <p>Efficient code can be well-structured and clean code, based on + <p>Efficient code can be well-structured and clean, based on a sound overall architecture and sound algorithms. Efficient code can be highly implementation-code that bypasses documented interfaces and takes advantage of obscure quirks in the current implementation.</p> - <p>Ideally, your code should only contain the first kind of efficient - code. If that turns out to be too slow, you should profile the application + <p>Ideally, your code only contains the first type of efficient + code. If that turns out to be too slow, profile the application to find out where the performance bottlenecks are and optimize only the - bottlenecks. Other code should stay as clean as possible.</p> + bottlenecks. Let other code stay as clean as possible.</p> - <p>Fortunately, compiler and run-time optimizations introduced in - R12B makes it easier to write code that is both clean and - efficient. For instance, the ugly workarounds needed in R11B and earlier + <p>Fortunately, compiler and runtime optimizations introduced in + Erlang/OTP R12B makes it easier to write code that is both clean and + efficient. For example, the ugly workarounds needed in R11B and earlier releases to get the most speed out of binary pattern matching are no longer necessary. In fact, the ugly code is slower than the clean code (because the clean code has become faster, not because the uglier code has become slower).</p> - <p>This Efficiency Guide cannot really learn you how to write efficient + <p>This Efficiency Guide cannot really teach you how to write efficient code. It can give you a few pointers about what to avoid and what to use, and some understanding of how certain language features are implemented. - We have generally not included general tips about optimization that will - work in any language, such as moving common calculations out of loops.</p> + This guide does not include general tips about optimization that + works in any language, such as moving common calculations out of loops.</p> </section> <section> <title>Prerequisites</title> - <p>It is assumed that the reader is familiar with the Erlang - programming language and concepts of OTP.</p> + <p>It is assumed that you are familiar with the Erlang programming + language and the OTP concepts.</p> </section> </chapter> diff --git a/system/doc/efficiency_guide/listhandling.xml b/system/doc/efficiency_guide/listhandling.xml index 9112738b18..b950f55ad1 100644 --- a/system/doc/efficiency_guide/listhandling.xml +++ b/system/doc/efficiency_guide/listhandling.xml @@ -18,10 +18,9 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> - <title>List handling</title> + <title>List Handling</title> <prepared>Bjorn Gustavsson</prepared> <docno></docno> <date>2007-11-16</date> @@ -30,19 +29,18 @@ </header> <section> - <title>Creating a list</title> + <title>Creating a List</title> - <p>Lists can only be built starting from the end and attaching - list elements at the beginning. If you use the <c>++</c> operator - like this</p> + <p>Lists can only be built starting from the end and attaching list + elements at the beginning. If you use the "<c>++</c>" operator as + follows, a new list is created that is a copy of the elements in + <c>List1</c>, followed by <c>List2</c>:</p> <code type="erl"> List1 ++ List2</code> - <p>you will create a new list which is copy of the elements in <c>List1</c>, - followed by <c>List2</c>. Looking at how <c>lists:append/1</c> or <c>++</c> would be - implemented in plain Erlang, it can be seen clearly that the first list - is copied:</p> + <p>Looking at how <c>lists:append/1</c> or <c>++</c> would be + implemented in plain Erlang, clearly the first list is copied:</p> <code type="erl"> append([H|T], Tail) -> @@ -50,12 +48,12 @@ append([H|T], Tail) -> append([], Tail) -> Tail.</code> - <p>So the important thing when recursing and building a list is to - make sure that you attach the new elements to the beginning of the list, - so that you build <em>a</em> list, and not hundreds or thousands of - copies of the growing result list.</p> + <p>When recursing and building a list, it is important to ensure + that you attach the new elements to the beginning of the list. In + this way, you will build <em>one</em> list, not hundreds or thousands + of copies of the growing result list.</p> - <p>Let us first look at how it should not be done:</p> + <p>Let us first see how it is not to be done:</p> <p><em>DO NOT</em></p> <code type="erl"><![CDATA[ @@ -67,11 +65,11 @@ bad_fib(0, _Current, _Next, Fibs) -> bad_fib(N, Current, Next, Fibs) -> bad_fib(N - 1, Next, Current + Next, Fibs ++ [Current]).]]></code> - <p>Here we are not a building a list; in each iteration step we - create a new list that is one element longer than the new previous list.</p> + <p>Here more than one list is built. In each iteration step a new list + is created that is one element longer than the new previous list.</p> - <p>To avoid copying the result in each iteration, we must build the list in - reverse order and reverse the list when we are done:</p> + <p>To avoid copying the result in each iteration, build the list in + reverse order and reverse the list when you are done:</p> <p><em>DO</em></p> <code type="erl"><![CDATA[ @@ -86,49 +84,45 @@ tail_recursive_fib(N, Current, Next, Fibs) -> </section> <section> - <title>List comprehensions</title> + <title>List Comprehensions</title> <p>Lists comprehensions still have a reputation for being slow. They used to be implemented using funs, which used to be slow.</p> - <p>In recent Erlang/OTP releases (including R12B), a list comprehension</p> + <p>In recent Erlang/OTP releases (including R12B), a list comprehension:</p> <code type="erl"><![CDATA[ [Expr(E) || E <- List]]]></code> - <p>is basically translated to a local function</p> + <p>is basically translated to a local function:</p> <code type="erl"> 'lc^0'([E|Tail], Expr) -> [Expr(E)|'lc^0'(Tail, Expr)]; 'lc^0'([], _Expr) -> [].</code> - <p>In R12B, if the result of the list comprehension will <em>obviously</em> not be used, - a list will not be constructed. For instance, in this code</p> + <p>In R12B, if the result of the list comprehension will <em>obviously</em> + not be used, a list will not be constructed. For example, in this code:</p> <code type="erl"><![CDATA[ [io:put_chars(E) || E <- List], ok.]]></code> - <p>or in this code</p> + <p>or in this code:</p> <code type="erl"><![CDATA[ -. -. -. +... case Var of ... -> [io:put_chars(E) || E <- List]; ... -> end, some_function(...), -. -. -.]]></code> +...]]></code> - <p>the value is neither assigned to a variable, nor passed to another function, - nor returned, so there is no need to construct a list and the compiler will simplify - the code for the list comprehension to</p> + <p>the value is not assigned to a variable, not passed to another function, + and not returned. This means that there is no need to construct a list and + the compiler will simplify the code for the list comprehension to:</p> <code type="erl"> 'lc^0'([E|Tail], Expr) -> @@ -139,14 +133,15 @@ some_function(...), </section> <section> - <title>Deep and flat lists</title> + <title>Deep and Flat Lists</title> <p><seealso marker="stdlib:lists#flatten/1">lists:flatten/1</seealso> - builds an entirely new list. Therefore, it is expensive, and even - <em>more</em> expensive than the <c>++</c> (which copies its left argument, - but not its right argument).</p> + builds an entirely new list. It is therefore expensive, and even + <em>more</em> expensive than the <c>++</c> operator (which copies its + left argument, but not its right argument).</p> - <p>In the following situations, you can easily avoid calling <c>lists:flatten/1</c>:</p> + <p>In the following situations, you can easily avoid calling + <c>lists:flatten/1</c>:</p> <list type="bulleted"> <item>When sending data to a port. Ports understand deep lists @@ -155,16 +150,19 @@ some_function(...), <item>When calling BIFs that accept deep lists, such as <seealso marker="erts:erlang#list_to_binary/1">list_to_binary/1</seealso> or <seealso marker="erts:erlang#iolist_to_binary/1">iolist_to_binary/1</seealso>.</item> - <item>When you know that your list is only one level deep, you can can use + <item>When you know that your list is only one level deep, you can use <seealso marker="stdlib:lists#append/1">lists:append/1</seealso>.</item> </list> - <p><em>Port example</em></p> + <section> + <title>Port Example</title> + <p><em>DO</em></p> <pre> ... port_command(Port, DeepList) ...</pre> + <p><em>DO NOT</em></p> <pre> ... @@ -180,7 +178,7 @@ some_function(...), port_command(Port, TerminatedStr) ...</pre> - <p>Instead do like this:</p> + <p>Instead:</p> <p><em>DO</em></p> <pre> @@ -188,47 +186,53 @@ some_function(...), TerminatedStr = [String, 0], % String="foo" => [[$f, $o, $o], 0] port_command(Port, TerminatedStr) ...</pre> + </section> + + <section> + <title>Append Example</title> - <p><em>Append example</em></p> <p><em>DO</em></p> <pre> > lists:append([[1], [2], [3]]). [1,2,3] ></pre> + <p><em>DO NOT</em></p> <pre> > lists:flatten([[1], [2], [3]]). [1,2,3] ></pre> + </section> </section> <section> - <title>Why you should not worry about recursive lists functions</title> + <title>Recursive List Functions</title> - <p>In the performance myth chapter, the following myth was exposed: - <seealso marker="myths#tail_recursive">Tail-recursive functions - are MUCH faster than recursive functions</seealso>.</p> + <p>In Section 7.2, the following myth was exposed: + <seealso marker="myths#tail_recursive">Tail-Recursive Functions + are Much Faster Than Recursive Functions</seealso>.</p> <p>To summarize, in R12B there is usually not much difference between a body-recursive list function and tail-recursive function that reverses the list at the end. Therefore, concentrate on writing beautiful code - and forget about the performance of your list functions. In the time-critical - parts of your code (and only there), <em>measure</em> before rewriting - your code.</p> - - <p><em>Important note</em>: This section talks about lists functions that - <em>construct</em> lists. A tail-recursive function that does not construct - a list runs in constant space, while the corresponding body-recursive - function uses stack space proportional to the length of the list. - For instance, a function that sums a list of integers, should <em>not</em> be - written like this</p> + and forget about the performance of your list functions. In the + time-critical parts of your code (and only there), <em>measure</em> + before rewriting your code.</p> + + <note><p>This section is about list functions that <em>construct</em> + lists. A tail-recursive function that does not construct a list runs + in constant space, while the corresponding body-recursive function + uses stack space proportional to the length of the list.</p></note> + + <p>For example, a function that sums a list of integers, is + <em>not</em> to be written as follows:</p> <p><em>DO NOT</em></p> <code type="erl"> recursive_sum([H|T]) -> H+recursive_sum(T); recursive_sum([]) -> 0.</code> - <p>but like this</p> + <p>Instead:</p> <p><em>DO</em></p> <code type="erl"> diff --git a/system/doc/efficiency_guide/myths.xml b/system/doc/efficiency_guide/myths.xml index b1108dbab2..70d2dae88e 100644 --- a/system/doc/efficiency_guide/myths.xml +++ b/system/doc/efficiency_guide/myths.xml @@ -31,47 +31,48 @@ <file>myths.xml</file> </header> + <marker id="myths"></marker> <p>Some truths seem to live on well beyond their best-before date, - perhaps because "information" spreads more rapidly from person-to-person - faster than a single release note that notes, for instance, that funs + perhaps because "information" spreads faster from person-to-person + than a single release note that says, for example, that funs have become faster.</p> - <p>Here we try to kill the old truths (or semi-truths) that have + <p>This section tries to kill the old truths (or semi-truths) that have become myths.</p> <section> - <title>Myth: Funs are slow</title> - <p>Yes, funs used to be slow. Very slow. Slower than <c>apply/3</c>. + <title>Myth: Funs are Slow</title> + <p>Funs used to be very slow, slower than <c>apply/3</c>. Originally, funs were implemented using nothing more than compiler trickery, ordinary tuples, <c>apply/3</c>, and a great deal of ingenuity.</p> - <p>But that is ancient history. Funs was given its own data type - in the R6B release and was further optimized in the R7B release. - Now the cost for a fun call falls roughly between the cost for a call to - local function and <c>apply/3</c>.</p> + <p>But that is history. Funs was given its own data type + in R6B and was further optimized in R7B. + Now the cost for a fun call falls roughly between the cost for a call + to a local function and <c>apply/3</c>.</p> </section> <section> - <title>Myth: List comprehensions are slow</title> + <title>Myth: List Comprehensions are Slow</title> <p>List comprehensions used to be implemented using funs, and in the - bad old days funs were really slow.</p> + old days funs were indeed slow.</p> - <p>Nowadays the compiler rewrites list comprehensions into an ordinary - recursive function. Of course, using a tail-recursive function with + <p>Nowadays, the compiler rewrites list comprehensions into an ordinary + recursive function. Using a tail-recursive function with a reverse at the end would be still faster. Or would it? That leads us to the next myth.</p> </section> <section> - <title>Myth: Tail-recursive functions are MUCH faster - than recursive functions</title> + <title>Myth: Tail-Recursive Functions are Much Faster + Than Recursive Functions</title> <p><marker id="tail_recursive"></marker>According to the myth, recursive functions leave references - to dead terms on the stack and the garbage collector will have to - copy all those dead terms, while tail-recursive functions immediately + to dead terms on the stack and the garbage collector has to copy + all those dead terms, while tail-recursive functions immediately discard those terms.</p> <p>That used to be true before R7B. In R7B, the compiler started @@ -79,48 +80,47 @@ be used with an empty list, so that the garbage collector would not keep dead values any longer than necessary.</p> - <p>Even after that optimization, a tail-recursive function would - still most of the time be faster than a body-recursive function. Why?</p> + <p>Even after that optimization, a tail-recursive function is + still most of the times faster than a body-recursive function. Why?</p> <p>It has to do with how many words of stack that are used in each - recursive call. In most cases, a recursive function would use more words + recursive call. In most cases, a recursive function uses more words on the stack for each recursion than the number of words a tail-recursive - would allocate on the heap. Since more memory is used, the garbage - collector will be invoked more frequently, and it will have more work traversing + would allocate on the heap. As more memory is used, the garbage + collector is invoked more frequently, and it has more work traversing the stack.</p> - <p>In R12B and later releases, there is an optimization that will + <p>In R12B and later releases, there is an optimization that in many cases reduces the number of words used on the stack in - body-recursive calls, so that a body-recursive list function and + body-recursive calls. A body-recursive list function and a tail-recursive function that calls <seealso - marker="stdlib:lists#reverse/1">lists:reverse/1</seealso> at the - end will use exactly the same amount of memory. + marker="stdlib:lists#reverse/1">lists:reverse/1</seealso> at + the end will use the same amount of memory. <c>lists:map/2</c>, <c>lists:filter/2</c>, list comprehensions, and many other recursive functions now use the same amount of space as their tail-recursive equivalents.</p> - <p>So which is faster?</p> + <p>So, which is faster? + It depends. On Solaris/Sparc, the body-recursive function seems to + be slightly faster, even for lists with a lot of elements. On the x86 + architecture, tail-recursion was up to about 30% faster.</p> - <p>It depends. On Solaris/Sparc, the body-recursive function seems to - be slightly faster, even for lists with very many elements. On the x86 - architecture, tail-recursion was up to about 30 percent faster.</p> - - <p>So the choice is now mostly a matter of taste. If you really do need + <p>So, the choice is now mostly a matter of taste. If you really do need the utmost speed, you must <em>measure</em>. You can no longer be - absolutely sure that the tail-recursive list function will be the fastest - in all circumstances.</p> + sure that the tail-recursive list function always is the fastest.</p> - <p>Note: A tail-recursive function that does not need to reverse the - list at the end is, of course, faster than a body-recursive function, + <note><p>A tail-recursive function that does not need to reverse the + list at the end is faster than a body-recursive function, as are tail-recursive functions that do not construct any terms at all - (for instance, a function that sums all integers in a list).</p> + (for example, a function that sums all integers in a list).</p></note> </section> <section> - <title>Myth: '++' is always bad</title> + <title>Myth: Operator "++" is Always Bad</title> - <p>The <c>++</c> operator has, somewhat undeservedly, got a very bad reputation. - It probably has something to do with code like</p> + <p>The <c>++</c> operator has, somewhat undeservedly, got a bad reputation. + It probably has something to do with code like the following, + which is the most inefficient way there is to reverse a list:</p> <p><em>DO NOT</em></p> <code type="erl"> @@ -129,12 +129,10 @@ naive_reverse([H|T]) -> naive_reverse([]) -> [].</code> - <p>which is the most inefficient way there is to reverse a list. - Since the <c>++</c> operator copies its left operand, the result - will be copied again and again and again... leading to quadratic - complexity.</p> + <p>As the <c>++</c> operator copies its left operand, the result + is copied repeatedly, leading to quadratic complexity.</p> - <p>On the other hand, using <c>++</c> like this</p> + <p>But using <c>++</c> as follows is not bad:</p> <p><em>OK</em></p> <code type="erl"> @@ -143,11 +141,11 @@ naive_but_ok_reverse([H|T], Acc) -> naive_but_ok_reverse([], Acc) -> Acc.</code> - <p>is not bad. Each list element will only be copied once. + <p>Each list element is copied only once. The growing result <c>Acc</c> is the right operand - for the <c>++</c> operator, and it will <em>not</em> be copied.</p> + for the <c>++</c> operator, and it is <em>not</em> copied.</p> - <p>Of course, experienced Erlang programmers would actually write</p> + <p>Experienced Erlang programmers would write as follows:</p> <p><em>DO</em></p> <code type="erl"> @@ -156,32 +154,34 @@ vanilla_reverse([H|T], Acc) -> vanilla_reverse([], Acc) -> Acc.</code> - <p>which is slightly more efficient because you don't build a - list element only to directly copy it. (Or it would be more efficient - if the the compiler did not automatically rewrite <c>[H]++Acc</c> + <p>This is slightly more efficient because here you do not build a + list element only to copy it directly. (Or it would be more efficient + if the compiler did not automatically rewrite <c>[H]++Acc</c> to <c>[H|Acc]</c>.)</p> </section> <section> - <title>Myth: Strings are slow</title> - - <p>Actually, string handling could be slow if done improperly. - In Erlang, you'll have to think a little more about how the strings - are used and choose an appropriate representation and use - the <seealso marker="stdlib:re">re</seealso> module instead of the obsolete - <c>regexp</c> module if you are going to use regular expressions.</p> + <title>Myth: Strings are Slow</title> + + <p>String handling can be slow if done improperly. + In Erlang, you need to think a little more about how the strings + are used and choose an appropriate representation. If you + use regular expressions, use the + <seealso marker="stdlib:re">re</seealso> module in STDLIB + instead of the obsolete <c>regexp</c> module.</p> </section> <section> - <title>Myth: Repairing a Dets file is very slow</title> + <title>Myth: Repairing a Dets File is Very Slow</title> <p>The repair time is still proportional to the number of records - in the file, but Dets repairs used to be much, much slower in the past. + in the file, but Dets repairs used to be much slower in the past. Dets has been massively rewritten and improved.</p> </section> <section> - <title>Myth: BEAM is a stack-based byte-code virtual machine (and therefore slow)</title> + <title>Myth: BEAM is a Stack-Based Byte-Code Virtual Machine + (and Therefore Slow)</title> <p>BEAM is a register-based virtual machine. It has 1024 virtual registers that are used for holding temporary values and for passing arguments when @@ -193,11 +193,11 @@ vanilla_reverse([], Acc) -> </section> <section> - <title>Myth: Use '_' to speed up your program when a variable is not used</title> + <title>Myth: Use "_" to Speed Up Your Program When a Variable + is Not Used</title> - <p>That was once true, but since R6B the BEAM compiler is quite capable of seeing itself + <p>That was once true, but from R6B the BEAM compiler can see that a variable is not used.</p> </section> - </chapter> diff --git a/system/doc/efficiency_guide/processes.xml b/system/doc/efficiency_guide/processes.xml index 86951e2dcc..3bdc314235 100644 --- a/system/doc/efficiency_guide/processes.xml +++ b/system/doc/efficiency_guide/processes.xml @@ -30,15 +30,15 @@ </header> <section> - <title>Creation of an Erlang process</title> + <title>Creating an Erlang Process</title> - <p>An Erlang process is lightweight compared to operating - systems threads and processes.</p> + <p>An Erlang process is lightweight compared to threads and + processes in operating systems.</p> <p>A newly spawned Erlang process uses 309 words of memory in the non-SMP emulator without HiPE support. (SMP support - and HiPE support will both add to this size.) The size can - be found out like this:</p> + and HiPE support both add to this size.) The size can + be found as follows:</p> <pre> Erlang (BEAM) emulator version 5.6 [async-threads:0] [kernel-poll:false] @@ -51,11 +51,11 @@ Eshell V5.6 (abort with ^G) 3> <input>Bytes div erlang:system_info(wordsize).</input> 309</pre> - <p>The size includes 233 words for the heap area (which includes the stack). - The garbage collector will increase the heap as needed.</p> + <p>The size includes 233 words for the heap area (which includes the + stack). The garbage collector increases the heap as needed.</p> <p>The main (outer) loop for a process <em>must</em> be tail-recursive. - If not, the stack will grow until the process terminates.</p> + Otherwise, the stack grows until the process terminates.</p> <p><em>DO NOT</em></p> <code type="erl"> @@ -74,7 +74,7 @@ loop() -> <p>The call to <c>io:format/2</c> will never be executed, but a return address will still be pushed to the stack each time <c>loop/0</c> is called recursively. The correct tail-recursive - version of the function looks like this:</p> + version of the function looks as follows:</p> <p><em>DO</em></p> <code type="erl"> @@ -90,92 +90,98 @@ loop() -> end.</code> <section> - <title>Initial heap size</title> + <title>Initial Heap Size</title> <p>The default initial heap size of 233 words is quite conservative - in order to support Erlang systems with hundreds of thousands or - even millions of processes. The garbage collector will grow and - shrink the heap as needed.</p> + to support Erlang systems with hundreds of thousands or + even millions of processes. The garbage collector grows and + shrinks the heap as needed.</p> <p>In a system that use comparatively few processes, performance - <em>might</em> be improved by increasing the minimum heap size using either - the <c>+h</c> option for + <em>might</em> be improved by increasing the minimum heap size + using either the <c>+h</c> option for <seealso marker="erts:erl">erl</seealso> or on a process-per-process basis using the <c>min_heap_size</c> option for <seealso marker="erts:erlang#spawn_opt/4">spawn_opt/4</seealso>.</p> - <p>The gain is twofold: Firstly, although the garbage collector will - grow the heap, it will grow it step by step, which will be more - costly than directly establishing a larger heap when the process - is spawned. Secondly, the garbage collector may also shrink the - heap if it is much larger than the amount of data stored on it; - setting the minimum heap size will prevent that.</p> - - <warning><p>The emulator will probably use more memory, and because garbage - collections occur less frequently, huge binaries could be + <p>The gain is twofold:</p> + <list type="bulleted"> + <item>Although the garbage collector grows the heap, it grows it + step-by-step, which is more costly than directly establishing a + larger heap when the process is spawned.</item> + <item>The garbage collector can also shrink the heap if it is + much larger than the amount of data stored on it; + setting the minimum heap size prevents that.</item> + </list> + + <warning><p>The emulator probably uses more memory, and because garbage + collections occur less frequently, huge binaries can be kept much longer.</p></warning> <p>In systems with many processes, computation tasks that run - for a short time could be spawned off into a new process with - a higher minimum heap size. When the process is done, it will - send the result of the computation to another process and terminate. - If the minimum heap size is calculated properly, the process may not - have to do any garbage collections at all. - <em>This optimization should not be attempted + for a short time can be spawned off into a new process with + a higher minimum heap size. When the process is done, it sends + the result of the computation to another process and terminates. + If the minimum heap size is calculated properly, the process might + not have to do any garbage collections at all. + <em>This optimization is not to be attempted without proper measurements.</em></p> </section> - </section> <section> - <title>Process messages</title> + <title>Process Messages</title> - <p>All data in messages between Erlang processes is copied, with - the exception of + <p>All data in messages between Erlang processes is copied, + except for <seealso marker="binaryhandling#refc_binary">refc binaries</seealso> on the same Erlang node.</p> <p>When a message is sent to a process on another Erlang node, - it will first be encoded to the Erlang External Format before - being sent via an TCP/IP socket. The receiving Erlang node decodes - the message and distributes it to the right process.</p> + it is first encoded to the Erlang External Format before + being sent through a TCP/IP socket. The receiving Erlang node decodes + the message and distributes it to the correct process.</p> <section> - <title>The constant pool</title> + <title>Constant Pool</title> <p>Constant Erlang terms (also called <em>literals</em>) are now kept in constant pools; each loaded module has its own pool. - The following function</p> + The following function does no longer build the tuple every time + it is called (only to have it discarded the next time the garbage + collector was run), but the tuple is located in the module's + constant pool:</p> <p><em>DO</em> (in R12B and later)</p> <code type="erl"> days_in_month(M) -> - element(M, {31,28,31,30,31,30,31,31,30,31,30,31}).</code> - - <p>will no longer build the tuple every time it is called (only - to have it discarded the next time the garbage collector was run), but - the tuple will be located in the module's constant pool.</p> + element(M, {31,28,31,30,31,30,31,31,30,31,30,31}).</code> <p>But if a constant is sent to another process (or stored in - an ETS table), it will be <em>copied</em>. - The reason is that the run-time system must be able - to keep track of all references to constants in order to properly - unload code containing constants. (When the code is unloaded, - the constants will be copied to the heap of the processes that refer + an Ets table), it is <em>copied</em>. + The reason is that the runtime system must be able + to keep track of all references to constants to unload code + containing constants properly. (When the code is unloaded, + the constants are copied to the heap of the processes that refer to them.) The copying of constants might be eliminated in a future - release.</p> + Erlang/OTP release.</p> </section> <section> - <title>Loss of sharing</title> + <title>Loss of Sharing</title> - <p>Shared sub-terms are <em>not</em> preserved when a term is sent - to another process, passed as the initial process arguments in - the <c>spawn</c> call, or stored in an ETS table. - That is an optimization. Most applications do not send messages - with shared sub-terms.</p> + <p>Shared subterms are <em>not</em> preserved in the following + cases:</p> + <list type="bulleted"> + <item>When a term is sent to another process</item> + <item>When a term is passed as the initial process arguments in + the <c>spawn</c> call</item> + <item>When a term is stored in an Ets table</item> + </list> + <p>That is an optimization. Most applications do not send messages + with shared subterms.</p> - <p>Here is an example of how a shared sub-term can be created:</p> + <p>The following example shows how a shared subterm can be created:</p> <code type="erl"> kilo_byte() -> @@ -186,32 +192,32 @@ kilo_byte(0, Acc) -> kilo_byte(N, Acc) -> kilo_byte(N-1, [Acc|Acc]).</code> - <p><c>kilo_byte/0</c> creates a deep list. If we call - <c>list_to_binary/1</c>, we can convert the deep list to a binary - of 1024 bytes:</p> + <p><c>kilo_byte/1</c> creates a deep list. + If <c>list_to_binary/1</c> is called, the deep list can be + converted to a binary of 1024 bytes:</p> <pre> 1> <input>byte_size(list_to_binary(efficiency_guide:kilo_byte())).</input> 1024</pre> - <p>Using the <c>erts_debug:size/1</c> BIF we can see that the + <p>Using the <c>erts_debug:size/1</c> BIF, it can be seen that the deep list only requires 22 words of heap space:</p> <pre> 2> <input>erts_debug:size(efficiency_guide:kilo_byte()).</input> 22</pre> - <p>Using the <c>erts_debug:flat_size/1</c> BIF, we can calculate - the size of the deep list if sharing is ignored. It will be + <p>Using the <c>erts_debug:flat_size/1</c> BIF, the size of the + deep list can be calculated if sharing is ignored. It becomes the size of the list when it has been sent to another process - or stored in an ETS table:</p> + or stored in an Ets table:</p> <pre> 3> <input>erts_debug:flat_size(efficiency_guide:kilo_byte()).</input> 4094</pre> - <p>We can verify that sharing will be lost if we insert the - data into an ETS table:</p> + <p>It can be verified that sharing will be lost if the data is + inserted into an Ets table:</p> <pre> 4> <input>T = ets:new(tab, []).</input> @@ -223,21 +229,21 @@ true 7> <input>erts_debug:flat_size(element(2, hd(ets:lookup(T, key)))).</input> 4094</pre> - <p>When the data has passed through an ETS table, + <p>When the data has passed through an Ets table, <c>erts_debug:size/1</c> and <c>erts_debug:flat_size/1</c> return the same value. Sharing has been lost.</p> - <p>In a future release of Erlang/OTP, we might implement a - way to (optionally) preserve sharing. We have no plans to make - preserving of sharing the default behaviour, since that would + <p>In a future Erlang/OTP release, it might be implemented a + way to (optionally) preserve sharing. There are no plans to make + preserving of sharing the default behaviour, as that would penalize the vast majority of Erlang applications.</p> </section> </section> <section> - <title>The SMP emulator</title> + <title>SMP Emulator</title> - <p>The SMP emulator (introduced in R11B) will take advantage of a + <p>The SMP emulator (introduced in R11B) takes advantage of a multi-core or multi-CPU computer by running several Erlang scheduler threads (typically, the same as the number of cores). Each scheduler thread schedules Erlang processes in the same way as the Erlang scheduler @@ -247,11 +253,11 @@ true <em>must have more than one runnable Erlang process</em> most of the time. Otherwise, the Erlang emulator can still only run one Erlang process at the time, but you must still pay the overhead for locking. Although - we try to reduce the locking overhead as much as possible, it will never - become exactly zero.</p> + Erlang/OTP tries to reduce the locking overhead as much as possible, + it will never become exactly zero.</p> - <p>Benchmarks that may seem to be concurrent are often sequential. - The estone benchmark, for instance, is entirely sequential. So is also + <p>Benchmarks that appear to be concurrent are often sequential. + The estone benchmark, for example, is entirely sequential. So is the most common implementation of the "ring benchmark"; usually one process is active, while the others wait in a <c>receive</c> statement.</p> @@ -259,6 +265,5 @@ true can be used to profile your application to see how much potential (or lack thereof) it has for concurrency.</p> </section> - </chapter> diff --git a/system/doc/efficiency_guide/profiling.xml b/system/doc/efficiency_guide/profiling.xml index b93c884270..5df12eefe0 100644 --- a/system/doc/efficiency_guide/profiling.xml +++ b/system/doc/efficiency_guide/profiling.xml @@ -30,190 +30,197 @@ </header> <section> - <title>Do not guess about performance - profile</title> + <title>Do Not Guess About Performance - Profile</title> <p>Even experienced software developers often guess wrong about where - the performance bottlenecks are in their programs.</p> - - <p>Therefore, profile your program to see where the performance + the performance bottlenecks are in their programs. Therefore, profile + your program to see where the performance bottlenecks are and concentrate on optimizing them.</p> - <p>Erlang/OTP contains several tools to help finding bottlenecks.</p> + <p>Erlang/OTP contains several tools to help finding bottlenecks:</p> + + <list type="bulleted"> + <item><c>fprof</c> provides the most detailed information about + where the program time is spent, but it significantly slows down the + program it profiles.</item> - <p><c>fprof</c> provide the most detailed information - about where the time is spent, but it significantly slows down the - program it profiles.</p> + <item><p><c>eprof</c> provides time information of each function + used in the program. No call graph is produced, but <c>eprof</c> has + considerable less impact on the program it profiles.</p> + <p>If the program is too large to be profiled by <c>fprof</c> or + <c>eprof</c>, the <c>cover</c> and <c>cprof</c> tools can be used + to locate code parts that are to be more thoroughly profiled using + <c>fprof</c> or <c>eprof</c>.</p></item> - <p><c>eprof</c> provides time information of each function used - in the program. No callgraph is produced but <c>eprof</c> has - considerable less impact on the program profiled.</p> + <item><c>cover</c> provides execution counts per line per + process, with less overhead than <c>fprof</c>. Execution counts + can, with some caution, be used to locate potential performance + bottlenecks.</item> - <p>If the program is too big to be profiled by <c>fprof</c> or <c>eprof</c>, - <c>cover</c> and <c>cprof</c> could be used to locate parts of the - code that should be more thoroughly profiled using <c>fprof</c> or - <c>eprof</c>.</p> + <item><c>cprof</c> is the most lightweight tool, but it only + provides execution counts on a function basis (for all processes, + not per process).</item> + </list> - <p><c>cover</c> provides execution counts per line per process, - with less overhead than <c>fprof</c>. Execution counts can - with some caution be used to locate potential performance bottlenecks. - The most lightweight tool is <c>cprof</c>, but it only provides execution - counts on a function basis (for all processes, not per process).</p> + <p>The tools are further described in + <seealso marker="#profiling_tools">Tools</seealso>.</p> </section> <section> - <title>Big systems</title> - <p>If you have a big system it might be interesting to run profiling + <title>Large Systems</title> + <p>For a large system, it can be interesting to run profiling on a simulated and limited scenario to start with. But bottlenecks - have a tendency to only appear or cause problems when - there are many things going on at the same time, and when there - are many nodes involved. Therefore it is desirable to also run + have a tendency to appear or cause problems only when + many things are going on at the same time, and when + many nodes are involved. Therefore, it is also desirable to run profiling in a system test plant on a real target system.</p> - <p>When your system is big you do not want to run the profiling - tools on the whole system. You want to concentrate on processes - and modules that you know are central and stand for a big part of the - execution.</p> + + <p>For a large system, you do not want to run the profiling + tools on the whole system. Instead you want to concentrate on + central processes and modules, which contribute for a big part + of the execution.</p> </section> <section> - <title>What to look for</title> - <p>When analyzing the result file from the profiling activity - you should look for functions that are called many + <title>What to Look For</title> + <p>When analyzing the result file from the profiling activity, + look for functions that are called many times and have a long "own" execution time (time excluding calls - to other functions). Functions that just are called very - many times can also be interesting, as even small things can add - up to quite a bit if they are repeated often. Then you need to - ask yourself what can I do to reduce this time. Appropriate - types of questions to ask yourself are: </p> + to other functions). Functions that are called a lot of + times can also be interesting, as even small things can add + up to quite a bit if repeated often. Also + ask yourself what you can do to reduce this time. The following + are appropriate types of questions to ask yourself:</p> + <list type="bulleted"> - <item>Can I reduce the number of times the function is called?</item> - <item>Are there tests that can be run less often if I change - the order of tests?</item> - <item>Are there redundant tests that can be removed? </item> - <item>Is there some expression calculated giving the same result - each time? </item> - <item>Are there other ways of doing this that are equivalent and + <item>Is it possible to reduce the number of times the function + is called?</item> + <item>Can any test be run less often if the order of tests is + changed?</item> + <item>Can any redundant tests be removed?</item> + <item>Does any calculated expression give the same result + each time?</item> + <item>Are there other ways to do this that are equivalent and more efficient?</item> - <item>Can I use another internal data representation to make - things more efficient? </item> + <item>Can another internal data representation be used to make + things more efficient?</item> </list> - <p>These questions are not always trivial to answer. You might - need to do some benchmarks to back up your theory, to avoid - making things slower if your theory is wrong. See <seealso marker="#benchmark">benchmarking</seealso>.</p> + + <p>These questions are not always trivial to answer. Some + benchmarks might be needed to back up your theory and to avoid + making things slower if your theory is wrong. For details, see + <seealso marker="#benchmark">Benchmarking</seealso>.</p> </section> <section> <title>Tools</title> - + <marker id="profiling_tools"></marker> <section> <title>fprof</title> - <p> - <c>fprof</c> measures the execution time for each function, - both own time i.e how much time a function has used for its - own execution, and accumulated time i.e. including called - functions. The values are displayed per process. You also get - to know how many times each function has been - called. <c>fprof</c> is based on trace to file in order to - minimize runtime performance impact. Using fprof is just a - matter of calling a few library functions, see - <seealso marker="tools:fprof">fprof</seealso> - manual page under the application tools.<c> fprof</c> was introduced in - version R8 of Erlang/OTP. - </p> + <p><c>fprof</c> measures the execution time for each function, + both own time, that is, how much time a function has used for its + own execution, and accumulated time, that is, including called + functions. The values are displayed per process. You also get + to know how many times each function has been called.</p> + + <p><c>fprof</c> is based on trace to file to minimize runtime + performance impact. Using <c>fprof</c> is just a matter of + calling a few library functions, see the + <seealso marker="tools:fprof">fprof</seealso> manual page in + <c>tools</c> .<c>fprof</c> was introduced in R8.</p> </section> - <section> - <title>eprof</title> - <p> - <c>eprof</c> is based on the Erlang trace_info BIFs. Eprof shows how much time has been used by - each process, and in which function calls this time has been - spent. Time is shown as percentage of total time and absolute time. - See <seealso marker="tools:eprof">eprof</seealso> for - additional information. - </p> - </section> + <section> + <title>eprof</title> + <p><c>eprof</c> is based on the Erlang <c>trace_info</c> BIFs. + <c>eprof</c> shows how much time has been used by each process, + and in which function calls this time has been spent. Time is + shown as percentage of total time and absolute time. For more + information, see the <seealso marker="tools:eprof">eprof</seealso> + manual page in <c>tools</c>.</p> + </section> <section> <title>cover</title> - <p> - <c>cover</c>'s primary use is coverage analysis to verify - test cases, making sure all relevant code is covered. - <c>cover</c> counts how many times each executable line of - code is executed when a program is run. This is done on a per - module basis. Of course this information can be used to - determine what code is run very frequently and could therefore - be subject for optimization. Using cover is just a matter of - calling a few library functions, see - <seealso marker="tools:cover">cover</seealso> - manual page under the application tools.</p> + <p>The primary use of <c>cover</c> is coverage analysis to verify + test cases, making sure that all relevant code is covered. + <c>cover</c> counts how many times each executable line of code + is executed when a program is run, on a per module basis.</p> + <p>Clearly, this information can be used to determine what + code is run very frequently and can therefore be subject for + optimization. Using <c>cover</c> is just a matter of calling a + few library functions, see the + <seealso marker="tools:cover">cover</seealso> manual page in + <c>tools</c>.</p> </section> <section> <title>cprof</title> <p><c>cprof</c> is something in between <c>fprof</c> and - <c>cover</c> regarding features. It counts how many times each - function is called when the program is run, on a per module - basis. <c>cprof</c> has a low performance degradation effect (versus - <c>fprof</c>) and does not need to recompile - any modules to profile (versus <c>cover</c>). - See <seealso marker="tools:cprof">cprof</seealso> manual page for additional - information. - </p> + <c>cover</c> regarding features. It counts how many times each + function is called when the program is run, on a per module + basis. <c>cprof</c> has a low performance degradation effect + (compared with <c>fprof</c>) and does not need to recompile + any modules to profile (compared with <c>cover</c>). + For more information, see the + <seealso marker="tools:cprof">cprof</seealso> manual page in + <c>tools</c>.</p> </section> <section> - <title>Tool summarization</title> + <title>Tool Summary</title> <table> <row> - <cell align="center" valign="middle">Tool</cell> - <cell align="center" valign="middle">Results</cell> - <cell align="center" valign="middle">Size of result</cell> - <cell align="center" valign="middle">Effects on program execution time</cell> - <cell align="center" valign="middle">Records number of calls</cell> - <cell align="center" valign="middle">Records Execution time</cell> - <cell align="center" valign="middle">Records called by</cell> - <cell align="center" valign="middle">Records garbage collection</cell> + <cell><em>Tool</em></cell> + <cell><em>Results</em></cell> + <cell><em>Size of Result</em></cell> + <cell><em>Effects on Program Execution Time</em></cell> + <cell><em>Records Number of Calls</em></cell> + <cell><em>Records Execution Time</em></cell> + <cell><em>Records Called by</em></cell> + <cell><em>Records Garbage Collection</em></cell> </row> <row> - <cell align="left" valign="middle"><c>fprof </c></cell> - <cell align="left" valign="middle">per process to screen/file </cell> - <cell align="left" valign="middle">large </cell> - <cell align="left" valign="middle">significant slowdown </cell> - <cell align="left" valign="middle">yes </cell> - <cell align="left" valign="middle">total and own</cell> - <cell align="left" valign="middle">yes </cell> - <cell align="left" valign="middle">yes </cell> + <cell><c>fprof</c></cell> + <cell>Per process to screen/file</cell> + <cell>Large</cell> + <cell>Significant slowdown</cell> + <cell>Yes</cell> + <cell>Total and own</cell> + <cell>Yes</cell> + <cell>Yes</cell> </row> <row> - <cell align="left" valign="middle"><c>eprof </c></cell> - <cell align="left" valign="middle">per process/function to screen/file </cell> - <cell align="left" valign="middle">medium </cell> - <cell align="left" valign="middle">small slowdown </cell> - <cell align="left" valign="middle">yes </cell> - <cell align="left" valign="middle">only total </cell> - <cell align="left" valign="middle">no </cell> - <cell align="left" valign="middle">no </cell> + <cell><c>eprof</c></cell> + <cell>Per process/function to screen/file</cell> + <cell>Medium</cell> + <cell>Small slowdown</cell> + <cell>Yes</cell> + <cell>Only total</cell> + <cell>No</cell> + <cell>No</cell> </row> <row> - <cell align="left" valign="middle"><c>cover </c></cell> - <cell align="left" valign="middle">per module to screen/file</cell> - <cell align="left" valign="middle">small </cell> - <cell align="left" valign="middle">moderate slowdown</cell> - <cell align="left" valign="middle">yes, per line </cell> - <cell align="left" valign="middle">no </cell> - <cell align="left" valign="middle">no </cell> - <cell align="left" valign="middle">no </cell> + <cell><c>cover</c></cell> + <cell>Per module to screen/file</cell> + <cell>Small</cell> + <cell>Moderate slowdown</cell> + <cell>Yes, per line</cell> + <cell>No</cell> + <cell>No</cell> + <cell>No</cell> </row> <row> - <cell align="left" valign="middle"><c>cprof </c></cell> - <cell align="left" valign="middle">per module to caller</cell> - <cell align="left" valign="middle">small </cell> - <cell align="left" valign="middle">small slowdown </cell> - <cell align="left" valign="middle">yes </cell> - <cell align="left" valign="middle">no </cell> - <cell align="left" valign="middle">no </cell> - <cell align="left" valign="middle">no </cell> + <cell><c>cprof</c></cell> + <cell>Per module to caller</cell> + <cell>Small</cell> + <cell>Small slowdown</cell> + <cell>Yes</cell> + <cell>No</cell> + <cell>No</cell> + <cell>No</cell> </row> - <tcaption></tcaption> + <tcaption>Tool Summary</tcaption> </table> </section> </section> @@ -226,49 +233,51 @@ implementation of a given algorithm or function is the fastest. Benchmarking is far from an exact science. Today's operating systems generally run background tasks that are difficult to turn off. - Caches and multiple CPU cores doesn't make it any easier. - It would be best to run Unix-computers in single-user mode when + Caches and multiple CPU cores does not facilitate benchmarking. + It would be best to run UNIX computers in single-user mode when benchmarking, but that is inconvenient to say the least for casual testing.</p> <p>Benchmarks can measure wall-clock time or CPU time.</p> - <p><seealso marker="stdlib:timer#tc/3">timer:tc/3</seealso> measures + <list type="bulleted"> + <item><seealso marker="stdlib:timer#tc/3">timer:tc/3</seealso> measures wall-clock time. The advantage with wall-clock time is that I/O, - swapping, and other activities in the operating-system kernel are + swapping, and other activities in the operating system kernel are included in the measurements. The disadvantage is that the - the measurements will vary wildly. Usually it is best to run the - benchmark several times and note the shortest time - that time should + measurements vary a lot. Usually it is best to run the + benchmark several times and note the shortest time, which is to be the minimum time that is possible to achieve under the best of - circumstances.</p> + circumstances.</item> - <p><seealso marker="erts:erlang#statistics/1">statistics/1</seealso> - with the argument <c>runtime</c> measures CPU time spent in the Erlang - virtual machine. The advantage is that the results are more + <item><seealso marker="erts:erlang#statistics/1">statistics/1</seealso> + with argument <c>runtime</c> measures CPU time spent in the Erlang + virtual machine. The advantage with CPU time is that the results are more consistent from run to run. The disadvantage is that the time spent in the operating system kernel (such as swapping and I/O) - are not included. Therefore, measuring CPU time is misleading if - any I/O (file or socket) is involved.</p> + is not included. Therefore, measuring CPU time is misleading if + any I/O (file or socket) is involved.</item> + </list> <p>It is probably a good idea to do both wall-clock measurements and CPU time measurements.</p> - <p>Some additional advice:</p> + <p>Some final advice:</p> <list type="bulleted"> - <item>The granularity of both types of measurement could be quite - high so you should make sure that each individual measurement + <item>The granularity of both measurement types can be high. + Therefore, ensure that each individual measurement lasts for at least several seconds.</item> - <item>To make the test fair, each new test run should run in its own, + <item>To make the test fair, each new test run is to run in its own, newly created Erlang process. Otherwise, if all tests run in the - same process, the later tests would start out with larger heap sizes - and therefore probably do less garbage collections. You could - also consider restarting the Erlang emulator between each test.</item> + same process, the later tests start out with larger heap sizes + and therefore probably do fewer garbage collections. + Also consider restarting the Erlang emulator between each test.</item> <item>Do not assume that the fastest implementation of a given algorithm - on computer architecture X also is the fastest on computer architecture Y.</item> - + on computer architecture X is also the fastest on computer architecture + Y.</item> </list> </section> </chapter> diff --git a/system/doc/efficiency_guide/tablesDatabases.xml b/system/doc/efficiency_guide/tablesDatabases.xml index 94c921fa1c..215c2afa1f 100644 --- a/system/doc/efficiency_guide/tablesDatabases.xml +++ b/system/doc/efficiency_guide/tablesDatabases.xml @@ -18,10 +18,9 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - </legalnotice> - <title>Tables and databases</title> + <title>Tables and Databases</title> <prepared>Ingela Anderton</prepared> <docno></docno> <date>2001-08-07</date> @@ -30,46 +29,45 @@ </header> <section> - <title>Ets, Dets and Mnesia</title> + <title>Ets, Dets, and Mnesia</title> <p>Every example using Ets has a corresponding example in - Mnesia. In general all Ets examples also apply to Dets tables.</p> + Mnesia. In general, all Ets examples also apply to Dets tables.</p> <section> - <title>Select/Match operations</title> - <p>Select/Match operations on Ets and Mnesia tables can become + <title>Select/Match Operations</title> + <p>Select/match operations on Ets and Mnesia tables can become very expensive operations. They usually need to scan the complete - table. You should try to structure your - data so that you minimize the need for select/match - operations. However, if you really need a select/match operation, - it will still be more efficient than using <c>tab2list</c>. - Examples of this and also of ways to avoid select/match will be provided in - some of the following sections. The functions - <c>ets:select/2</c> and <c>mnesia:select/3</c> should be preferred over - <c>ets:match/2</c>,<c>ets:match_object/2</c>, and <c>mnesia:match_object/3</c>.</p> - <note> - <p>There are exceptions when the complete table is not - scanned, for instance if part of the key is bound when searching an - <c>ordered_set</c> table, or if it is a Mnesia - table and there is a secondary index on the field that is - selected/matched. If the key is fully bound there will, of course, be - no point in doing a select/match, unless you have a bag table and - you are only interested in a sub-set of the elements with - the specific key.</p> - </note> - <p>When creating a record to be used in a select/match operation you - want most of the fields to have the value '_'. The easiest and fastest way - to do that is as follows:</p> + table. Try to structure the data to minimize the need for select/match + operations. However, if you require a select/match operation, + it is still more efficient than using <c>tab2list</c>. + Examples of this and of how to avoid select/match are provided in + the following sections. The functions + <c>ets:select/2</c> and <c>mnesia:select/3</c> are to be preferred + over <c>ets:match/2</c>, <c>ets:match_object/2</c>, and + <c>mnesia:match_object/3</c>.</p> + <p>In some circumstances, the select/match operations do not need + to scan the complete table. + For example, if part of the key is bound when searching an + <c>ordered_set</c> table, or if it is a Mnesia + table and there is a secondary index on the field that is + selected/matched. If the key is fully bound, there is + no point in doing a select/match, unless you have a bag table + and are only interested in a subset of the elements with + the specific key.</p> + <p>When creating a record to be used in a select/match operation, you + want most of the fields to have the value "_". The easiest and + fastest way to do that is as follows:</p> <pre> #person{age = 42, _ = '_'}. </pre> </section> <section> - <title>Deleting an element</title> - <p>The delete operation is considered + <title>Deleting an Element</title> + <p>The <c>delete</c> operation is considered successful if the element was not present in the table. Hence all attempts to check that the element is present in the Ets/Mnesia table before deletion are unnecessary. Here follows - an example for Ets tables.</p> + an example for Ets tables:</p> <p><em>DO</em></p> <pre> ... @@ -88,14 +86,16 @@ end, </section> <section> - <title>Data fetching</title> - <p>Do not fetch data that you already have! Consider that you - have a module that handles the abstract data type Person. You - export the interface function <c>print_person/1</c> that uses the internal functions - <c>print_name/1</c>, <c>print_age/1</c>, <c>print_occupation/1</c>.</p> + <title>Fetching Data</title> + <p>Do not fetch data that you already have.</p> + <p>Consider that you have a module that handles the abstract data + type <c>Person</c>. You export the interface function + <c>print_person/1</c>, which uses the internal functions + <c>print_name/1</c>, <c>print_age/1</c>, and + <c>print_occupation/1</c>.</p> <note> - <p>If the functions <c>print_name/1</c> and so on, had been interface - functions the matter comes in to a whole new light, as you + <p>If the function <c>print_name/1</c>, and so on, had been interface + functions, the situation would have been different, as you do not want the user of the interface to know about the internal data representation. </p> </note> @@ -136,7 +136,7 @@ print_person(PersonId) -> io:format("No person with ID = ~p~n", [PersonID]) end. -%%% Internal functions +%%% Internal functionss print_name(PersonID) -> [Person] = ets:lookup(person, PersonId), io:format("No person ~p~n", [Person#person.name]). @@ -151,31 +151,31 @@ print_occupation(PersonID) -> </section> <section> - <title>Non-persistent data storage </title> + <title>Non-Persistent Database Storage</title> <p>For non-persistent database storage, prefer Ets tables over - Mnesia local_content tables. Even the Mnesia <c>dirty_write</c> + Mnesia <c>local_content</c> tables. Even the Mnesia <c>dirty_write</c> operations carry a fixed overhead compared to Ets writes. Mnesia must check if the table is replicated or has indices, this involves at least one Ets lookup for each - <c>dirty_write</c>. Thus, Ets writes will always be faster than + <c>dirty_write</c>. Thus, Ets writes is always faster than Mnesia writes.</p> </section> <section> <title>tab2list</title> - <p>Assume we have an Ets-table, which uses <c>idno</c> as key, - and contains:</p> + <p>Assuming an Ets table that uses <c>idno</c> as key + and contains the following:</p> <pre> [#person{idno = 1, name = "Adam", age = 31, occupation = "mailman"}, #person{idno = 2, name = "Bryan", age = 31, occupation = "cashier"}, #person{idno = 3, name = "Bryan", age = 35, occupation = "banker"}, #person{idno = 4, name = "Carl", age = 25, occupation = "mailman"}]</pre> - <p>If we <em>must</em> return all data stored in the Ets-table we - can use <c>ets:tab2list/1</c>. However, usually we are only + <p>If you <em>must</em> return all data stored in the Ets table, you + can use <c>ets:tab2list/1</c>. However, usually you are only interested in a subset of the information in which case - <c>ets:tab2list/1</c> is expensive. If we only want to extract - one field from each record, e.g., the age of every person, we - should use:</p> + <c>ets:tab2list/1</c> is expensive. If you only want to extract + one field from each record, for example, the age of every person, + then:</p> <p><em>DO</em></p> <pre> ... @@ -192,8 +192,8 @@ ets:select(Tab,[{ #person{idno='_', TabList = ets:tab2list(Tab), lists:map(fun(X) -> X#person.age end, TabList), ...</pre> - <p>If we are only interested in the age of all persons named - Bryan, we should:</p> + <p>If you are only interested in the age of all persons named + "Bryan", then:</p> <p><em>DO</em></p> <pre> ... @@ -224,8 +224,8 @@ BryanList = lists:filter(fun(X) -> X#person.name == "Bryan" end, TabList), lists:map(fun(X) -> X#person.age end, BryanList), ...</pre> - <p>If we need all information stored in the Ets table about - persons named Bryan we should:</p> + <p>If you need all information stored in the Ets table about + persons named "Bryan", then:</p> <p><em>DO</em></p> <pre> ... @@ -243,60 +243,60 @@ lists:filter(fun(X) -> X#person.name == "Bryan" end, TabList), </section> <section> - <title>Ordered_set tables</title> - <p>If the data in the table should be accessed so that the order + <title>Ordered_set Tables</title> + <p>If the data in the table is to be accessed so that the order of the keys in the table is significant, the table type - <c>ordered_set</c> could be used instead of the more usual + <c>ordered_set</c> can be used instead of the more usual <c>set</c> table type. An <c>ordered_set</c> is always - traversed in Erlang term order with regard to the key field - so that return values from functions such as <c>select</c>, + traversed in Erlang term order regarding the key field + so that the return values from functions such as <c>select</c>, <c>match_object</c>, and <c>foldl</c> are ordered by the key values. Traversing an <c>ordered_set</c> with the <c>first</c> and <c>next</c> operations also returns the keys ordered.</p> <note> <p>An <c>ordered_set</c> only guarantees that - objects are processed in <em>key</em> order. Results from functions as - <c>ets:select/2</c> appear in the <em>key</em> order even if + objects are processed in <em>key</em> order. + Results from functions such as + <c>ets:select/2</c> appear in <em>key</em> order even if the key is not included in the result.</p> </note> </section> </section> <section> - <title>Ets specific</title> + <title>Ets-Specific</title> <section> - <title>Utilizing the keys of the Ets table</title> - <p>An Ets table is a single key table (either a hash table or a - tree ordered by the key) and should be used as one. In other + <title>Using Keys of Ets Table</title> + <p>An Ets table is a single-key table (either a hash table or a + tree ordered by the key) and is to be used as one. In other words, use the key to look up things whenever possible. A - lookup by a known key in a set Ets table is constant and for a - ordered_set Ets table it is O(logN). A key lookup is always + lookup by a known key in a <c>set</c> Ets table is constant and for + an <c>ordered_set</c> Ets table it is O(logN). A key lookup is always preferable to a call where the whole table has to be - scanned. In the examples above, the field <c>idno</c> is the + scanned. In the previous examples, the field <c>idno</c> is the key of the table and all lookups where only the name is known - will result in a complete scan of the (possibly large) table + result in a complete scan of the (possibly large) table for a matching result.</p> <p>A simple solution would be to use the <c>name</c> field as the key instead of the <c>idno</c> field, but that would cause - problems if the names were not unique. A more general solution - would be to create a second table with <c>name</c> as key and - <c>idno</c> as data, i.e. to index (invert) the table with regards - to the <c>name</c> field. The second table would of course have to be - kept consistent with the master table. Mnesia could do this - for you, but a home brew index table could be very efficient + problems if the names were not unique. A more general solution would + be to create a second table with <c>name</c> as key and + <c>idno</c> as data, that is, to index (invert) the table regarding + the <c>name</c> field. Clearly, the second table would have to be + kept consistent with the master table. Mnesia can do this + for you, but a home brew index table can be very efficient compared to the overhead involved in using Mnesia.</p> <p>An index table for the table in the previous examples would - have to be a bag (as keys would appear more than once) and could + have to be a bag (as keys would appear more than once) and can have the following contents:</p> <pre> - [#index_entry{name="Adam", idno=1}, #index_entry{name="Bryan", idno=2}, #index_entry{name="Bryan", idno=3}, #index_entry{name="Carl", idno=4}]</pre> - <p>Given this index table a lookup of the <c>age</c> fields for - all persons named "Bryan" could be done like this:</p> + <p>Given this index table, a lookup of the <c>age</c> fields for + all persons named "Bryan" can be done as follows:</p> <pre> ... MatchingIDs = ets:lookup(IndexTable,"Bryan"), @@ -306,30 +306,31 @@ lists:map(fun(#index_entry{idno = ID}) -> end, MatchingIDs), ...</pre> - <p>Note that the code above never uses <c>ets:match/2</c> but - instead utilizes the <c>ets:lookup/2</c> call. The + <p>Notice that this code never uses <c>ets:match/2</c> but + instead uses the <c>ets:lookup/2</c> call. The <c>lists:map/2</c> call is only used to traverse the <c>idno</c>s - matching the name "Bryan" in the table; therefore the number of lookups + matching the name "Bryan" in the table; thus the number of lookups in the master table is minimized.</p> <p>Keeping an index table introduces some overhead when - inserting records in the table, therefore the number of operations - gained from the table has to be weighted against the number of - operations inserting objects in the table. However, note that the gain when - the key can be used to lookup elements is significant.</p> + inserting records in the table. The number of operations gained + from the table must therefore be compared against the number of + operations inserting objects in the table. However, notice that the + gain is significant when the key can be used to lookup elements.</p> </section> </section> <section> - <title>Mnesia specific</title> + <title>Mnesia-Specific</title> <section> - <title>Secondary index</title> + <title>Secondary Index</title> <p>If you frequently do a lookup on a field that is not the - key of the table, you will lose performance using - "mnesia:select/match_object" as this function will traverse the - whole table. You may create a secondary index instead and + key of the table, you lose performance using + "mnesia:select/match_object" as this function traverses the + whole table. You can create a secondary index instead and use "mnesia:index_read" to get faster access, however this - will require more memory. Example:</p> + requires more memory.</p> + <p><em>Example</em></p> <pre> -record(person, {idno, name, age, occupation}). ... @@ -347,14 +348,15 @@ PersonsAge42 = <section> <title>Transactions </title> - <p>Transactions is a way to guarantee that the distributed + <p>Using transactions is a way to guarantee that the distributed Mnesia database remains consistent, even when many different - processes update it in parallel. However if you have - real time requirements it is recommended to use dirty - operations instead of transactions. When using the dirty - operations you lose the consistency guarantee, this is usually + processes update it in parallel. However, if you have + real-time requirements it is recommended to use <c>dirty</c> + operations instead of transactions. When using <c>dirty</c> + operations, you lose the consistency guarantee; this is usually solved by only letting one process update the table. Other - processes have to send update requests to that process.</p> + processes must send update requests to that process.</p> + <p><em>Example</em></p> <pre> ... % Using transaction diff --git a/system/doc/embedded/embedded_nt.xml b/system/doc/embedded/embedded_nt.xml index 530e3663e4..2e3b32eb84 100644 --- a/system/doc/embedded/embedded_nt.xml +++ b/system/doc/embedded/embedded_nt.xml @@ -31,54 +31,47 @@ <rev>PA2</rev> <file>embedded_nt.xml</file> </header> - <p>This chapter describes the OS specific parts of OTP which relate - to Windows NT. - </p> + <marker id="windows nt"></marker> + <p>This section describes the operating system-specific parts of OTP + that relate to Windows NT.</p> + <p>A normal installation of Windows NT 4.0, with Service Pack 4 or + later, is required for an embedded Windows NT running OTP.</p> <section> - <title>Introduction</title> - <p>A normal installation of NT 4.0, with service pack 4 or later, - is required for an embedded Windows NT running OTP.</p> + <title>Memory Use</title> + <p>RAM memory of 96 MB is recommended to run OTP on Windows NT. + A system with less than 64 MB of RAM is not recommended.</p> </section> <section> - <title>Memory Usage</title> - <p>RAM memory of 96 MBytes is recommended to run OTP on NT. - A system with less than 64 Mbytes of RAM is not recommended.</p> + <title>Disk Space Use</title> + <p>A minimum Windows NT installation with networking needs 250 MB, + and an extra 130 MB for the swap file.</p> </section> <section> - <title>Disk Space Usage</title> - <p>A minimum NT installation with networking needs 250 MB, and - an additional 130 MB for the swap file. </p> - </section> - - <section> - <title>Installation</title> - <p>Normal NT installation is performed. No additional application - programs are needed, such as Internet explorer or web server. Networking - with TCP/IP is required. <br></br> - - Service pack 4 or later must be installed.</p> + <title>Installing an Embedded System</title> + <p>Normal Windows NT installation is performed. No additional + application programs are needed, such as Internet Explorer or + web server. Networking with TCP/IP is required.</p> + <p>Service Pack 4 or later must be installed.</p> <section> <title>Hardware Watchdog</title> - <p>For Windows NT running on standard PCs with ISA and/or PCI bus - there is a possibility to install an extension card with a hardware - watchdog. - </p> - <p>See also the <c>heart(3)</c> reference manual page in - <em>Kernel</em>. - </p> + <p>For Windows NT running on standard PCs with ISA and/or PCI bus, + an extension card with a hardware watchdog can be installed.</p> + <p>For more information, see the <c>heart(3)</c> manual page in + <c>kernel</c>.</p> </section> </section> <section> <title>Starting Erlang</title> - <p>On an embedded system, the <c>erlsrv</c> module should be used, - to install the erlang process as a Windows system service. - This service can start - after NT has booted. See documentation for <c>erlsrv</c>.</p> + <p>On an embedded system, the <c>erlsrv</c> module is to be used + to install the Erlang process as a Windows system service. + This service can start after Windows NT has booted.</p> + <p>For more information, see the <c>erlsrv</c> manual page + in <c>erts</c>.</p> </section> </chapter> diff --git a/system/doc/embedded/embedded_solaris.xml b/system/doc/embedded/embedded_solaris.xml index cab3437725..1861436a8e 100644 --- a/system/doc/embedded/embedded_solaris.xml +++ b/system/doc/embedded/embedded_solaris.xml @@ -31,125 +31,97 @@ <rev>B</rev> <file>embedded_solaris.xml</file> </header> - <p>This chapter describes the OS specific parts of OTP which relate - to Solaris. - </p> + <marker id="embedded solaris"></marker> + + <p>This section describes the operating system-specific parts + of OTP that relate to Solaris.</p> <section> - <title>Memory Usage</title> - <p>Solaris takes about 17 Mbyte of RAM on a system with 64 Mbyte of - total RAM. This leaves about 47 Mbyte for the applications. If - the system utilizes swapping, these figures cannot be improved + <title>Memory Use</title> + <p>Solaris takes about 17 MB of RAM on a system with 64 MB of + total RAM. This leaves about 47 MB for the applications. If + the system uses swapping, these figures cannot be improved because unnecessary daemon processes are swapped out. However, if swapping is disabled, or if the swap space is of limited resource in the system, it becomes necessary to kill off - unnecessary daemon processes. - </p> + unnecessary daemon processes.</p> </section> <section> - <title>Disk Space Usage</title> + <title>Disk Space Use</title> <p>The disk space required by Solaris can be minimized by using the - Core User support installation. It requires about 80 Mbyte of + Core User support installation. It requires about 80 MB of disk space. This installs only the minimum software required to - boot and run Solaris. The disk space can be further reduced by + boot and run Solaris. The disk space can be further reduced by deleting unnecessary individual files. However, unless disk space is a critical resource the effort required and the risks - involved may not be justified.</p> + involved cannot be justified.</p> </section> <section> - <title>Installation</title> + <title>Installing an Embedded System</title> <p>This section is about installing an embedded system. - The following topics are considered, - </p> + The following topics are considered: + </p> <list type="bulleted"> - <item> - <p>Creation of user and installation directory,</p> - </item> - <item> - <p>Installation of embedded system,</p> - </item> - <item> - <p>Configuration for automatic start at reboot,</p> - </item> - <item> - <p>Making a hardware watchdog available,</p> - </item> - <item> - <p>Changing permission for reboot,</p> - </item> - <item> - <p>Patches,</p> - </item> - <item> - <p>Configuration of the OS_Mon application.</p> - </item> + <item>Creating user and installation directory</item> + <item>Installing an embedded system</item> + <item>Configuring automatic start at boot</item> + <item>Making a hardware watchdog available</item> + <item>Changing permission for reboot</item> + <item>Setting TERM environment variable</item> + <item>Adding patches</item> + <item>Installing module os_sup in application os_mon</item> </list> - <p>Several of the procedures described below require expert - knowledge of the Solaris 2 operating system. For most of them - super user privilege is needed. - </p> + <p>Several of the procedures in this section require expert + knowledge of the Solaris operating system. For most of them + super user privilege is needed.</p> <section> - <title>Creation of User and Installation Directory</title> - <p>It is recommended that the Embedded Environment is run by an - ordinary user, i.e. a user who does not have super user - privileges. - </p> - <p>Throughout this section we assume that the user name is - <c>otpuser</c>, and that the home directory of that user is, - </p> + <title>Creating User and Installation Directory</title> + <p>It is recommended that the embedded environment is run by an + ordinary user, that is, a user who does not have super user + privileges.</p> + <p>In this section, it is assumed that the username is + <c>otpuser</c> and that the home directory of that user is:</p> <pre> /export/home/otpuser</pre> - <p>Furthermore, we assume that in the home directory of + <p>It is also assumed that in the home directory of <c>otpuser</c>, there is a directory named <c>otp</c>, the - full path of which is, - </p> + full path of which is:</p> <pre> /export/home/otpuser/otp</pre> <p>This directory is the <em>installation directory</em> of the - Embedded Environment. - </p> + embedded environment.</p> </section> <section> - <title>Installation of an Embedded System</title> - <p>The procedure for installation of an embedded system does - not differ from that of an ordinary system (see the - <em>Installation Guide</em>), - except for the following: - </p> + <title>Installing an Embedded System</title> + <p>The procedure for installing an embedded system + is the same as for an ordinary system (see + Installation Guide), except for the following:</p> <list type="bulleted"> - <item> - <p>the (compressed) tape archive file should be - extracted in the installation directory as defined above, - and,</p> - </item> - <item> - <p>there is no need to link the start script to a - standard directory like <c>/usr/local/bin</c>.</p> - </item> + <item>The (compressed) tape archive file is to be extracted in + the installation directory defined above.</item> + <item>It is not needed to link the start script to a standard + directory like <c>/usr/local/bin</c>.</item> </list> </section> <section> - <title>Configuration for Automatic Start at Boot</title> - <p>A true embedded system has to start when the system - boots. This section accounts for the necessary configurations - needed to achieve that. - </p> - <p>The embedded system and all the applications will start - automatically if the script file shown below is added to the - <c>/etc/rc3.d</c> directory. The file must be owned and - readable by <c>root</c>, and its name cannot be arbitrarily - assigned. The following name is recommended, - </p> + <title>Configuring Automatic Start at Boot</title> + <p>A true embedded system must start when the system boots. + This section accounts for the necessary configurations + needed to achieve that.</p> + <p>The embedded system and all the applications start + automatically if the script file shown below is added to + directory <c>/etc/rc3.d</c>. The file must be owned and + readable by <c>root</c>. Its name cannot be arbitrarily + assigned; the following name is recommended:</p> <pre> S75otp.system</pre> - <p>For further details on initialization (and termination) - scripts, and naming thereof, see the Solaris documentation. - </p> + <p>For more details on initialization (and termination) + scripts, and naming thereof, see the Solaris documentation.</p> <pre> #!/bin/sh # @@ -187,386 +159,333 @@ case "$1" in echo "Usage: $0 { start | stop }" ;; esac</pre> - <p>The file <c>/export/home/otpuser/otp/bin/start</c> referred to - in the above script, is precisely the script <c>start</c> - described in the section <em>Starting Erlang</em> below. The + <p>File <c>/export/home/otpuser/otp/bin/start</c> referred to + in the above script is precisely the <c>start</c> script + described in <em>Starting Erlang</em>. The script variable <c>OTP_ROOT</c> in that <c>start</c> script - corresponds to the example path - </p> + corresponds to the following example path used in this + section:</p> <pre> /export/home/otpuser/otp</pre> - <p>used in this section. The <c>start</c> script should be edited - accordingly. - </p> - <p>Use of the <c>killproc</c> procedure in the above script could - be combined with a call to <c>erl_call</c>, e.g. - </p> + <p>The <c>start</c> script is to be edited accordingly.</p> + <p>Use of the <c>killproc</c> procedure in the above script can + be combined with a call to <c>erl_call</c>, for example:</p> <pre> $SOME_PATH/erl_call -n Node init stop</pre> - <p>In order to take Erlang down gracefully see the - <c>erl_call(1)</c> reference manual page for further details - on the use of <c>erl_call</c>. That however requires that - Erlang runs as a distributed node which is not always the - case. - </p> - <p>The <c>killproc</c> procedure should not be removed: the + <p>To take Erlang down gracefully, see the <c>erl_call(1)</c> + manual page in <c>erl_interface</c> for details on the use + of <c>erl_call</c>. However, + that requires that Erlang runs as a distributed node, which is + not always the case.</p> + <p>The <c>killproc</c> procedure is not to be removed. The purpose is here to move from run level 3 (multi-user mode with networking resources) to run level 2 (multi-user mode without - such resources), in which Erlang should not run. - </p> + such resources), in which Erlang is not to run.</p> </section> <section> - <title>Hardware Watchdog</title> + <title>Making Hardware Watchdog Available</title> <p>For Solaris running on VME boards from Force Computers, - there is a possibility to activate the onboard hardware - watchdog, provided a VME bus driver is added to the operating - system (see also <em>Installation Problems</em> below). - </p> - <p>See also the <c>heart(3)</c> reference manual page in - <em>Kernel</em>. - </p> + the onboard hardware watchdog can be activated, + provided a VME bus driver is added to the operating system + (see also Installation Problems).</p> + <p>See also the <c>heart(3)</c> manual page in <c>kernel</c>.</p> </section> <section> <title>Changing Permissions for Reboot</title> <p>If the <c>HEART_COMMAND</c> environment variable is to be set - in the <c>start</c> script in the section, <em>Starting Erlang</em>, and if the value shall be set to the - path of the Solaris <c>reboot</c> command, i.e. - </p> + in the <c>start</c> script in + <em>Starting Erlang</em>, and if the value is to be set to the + path of the Solaris <c>reboot</c> command, that is:</p> <pre> HEART_COMMAND=/usr/sbin/reboot</pre> - <p>the ownership and file permissions for <c>/usr/sbin/reboot</c> - must be changed as follows, - </p> + <p>then the ownership and file permissions for + <c>/usr/sbin/reboot</c> must be changed as follows:</p> <pre> chown 0 /usr/sbin/reboot chmod 4755 /usr/sbin/reboot</pre> - <p>See also the <c>heart(3)</c> reference manual page in - <em>Kernel</em>. - </p> + <p>See also the <c>heart(3)</c> manual page in <c>kernel</c>.</p> </section> <section> - <title>The TERM Environment Variable</title> - <p>When the Erlang runtime system is automatically started from the - <c>S75otp.system</c> script the <c>TERM</c> environment - variable has to be set. The following is a minimal setting, - </p> + <title>Setting TERM Environment Variable</title> + <p>When the Erlang runtime system is automatically started from + the <c>S75otp.system</c> script, the <c>TERM</c> environment + variable must be set. The following is a minimal setting:</p> <pre> TERM=sun</pre> - <p>which should be added to the <c>start</c> script described in - the section. - </p> + <p>This is to be added to the <c>start</c> script.</p> </section> <section> - <title>Patches</title> + <title>Adding Patches</title> <p>For proper functioning of flushing file system data to disk on - Solaris 2.5.1, the version specific patch with number - 103640-02 must be added to the operating system. There may be - other patches needed, see the release README file - <c><![CDATA[<ERL_INSTALL_DIR>/README]]></c>. - </p> + Solaris 2.5.1, the version-specific patch with number + 103640-02 must be added to the operating system. Other + patches might be needed, see the release README file + <c><![CDATA[<ERL_INSTALL_DIR>/README]]></c>.</p> </section> <section> - <title>Installation of Module os_sup in Application OS_Mon</title> + <title>Installing Module os_sup in Application os_mon</title> <p>The following four installation procedures require super user - privilege. - </p> - - <section> - <title>Installation</title> - <list type="ordered"> - <item> - <p><em>Make a copy the Solaris standard configuration file for syslogd.</em></p> - <list type="bulleted"> - <item> - <p>Make a copy the Solaris standard configuration - file for syslogd. This file is usually named - <c>syslog.conf</c> and found in the <c>/etc</c> - directory.</p> - </item> - <item> - <p>The file name of the copy must be - <c>syslog.conf.ORIG</c> but the directory location - is optional. Usually it is <c>/etc</c>. - </p> - <p>A simple way to do this is to issue the command</p> - <code type="none"> + privilege:</p> + + <section> + <title>Installation</title> + <list type="bulleted"> + <item><em>Make a copy of the Solaris standard configuration + file for <c>syslogd</c>:</em> + <list type="bulleted"> + <item>Make a copy of the Solaris standard configuration + file for <c>syslogd</c>. This file is usually named + <c>syslog.conf</c> and found in directory <c>/etc</c>.</item> + <item>The filename of the copy must be <c>syslog.conf.ORIG</c>. + The directory location is optional; usually it is <c>/etc</c>. + A simple way to do this is to issue the following command: + <code type="none"> cp /etc/syslog.conf /etc/syslog.conf.ORIG</code> </item> - </list> - </item> - <item> - <p><em>Make an Erlang specific configuration file for syslogd.</em></p> - <list type="bulleted"> - <item> - <p>Make an edited copy of the back-up copy previously - made.</p> - </item> - <item> - <p>The file name must be <c>syslog.conf.OTP</c> and the - path must be the same as the back-up copy.</p> - </item> - <item> - <p>The format of the configuration file is found in the - man page for <c>syslog.conf(5)</c>, by issuing the - command <c>man syslog.conf</c>.</p> - </item> - <item> - <p>Usually a line is added which should state:</p> - <list type="bulleted"> - <item> - <p>which types of information that will be - supervised by Erlang,</p> - </item> - <item> - <p>the name of the file (actually a named pipe) - that should receive the information.</p> - </item> - </list> - </item> - <item> - <p>If e.g. only information originating from the - unix-kernel should be supervised, the line should - begin with <c>kern.LEVEL</c> (for the possible - values of <c>LEVEL</c> see <c>syslog.conf(5)</c>).</p> - </item> - <item> - <p>After at least one tab-character, the line added - should contain the full name of the named pipe where - syslogd writes its information. The path must be the - same as for the <c>syslog.conf.ORIG</c> and - <c>syslog.conf.OTP</c> files. The file name must be - <c>syslog.otp</c>.</p> - </item> - <item> - <p>If the directory for the <c>syslog.conf.ORIG</c> and - <c>syslog.conf.OTP</c> files is <c>/etc</c> the line - in <c>syslog.conf.OTP</c> will look like:</p> - <code type="none"> + </list> + </item> + <item><em>Make an Erlang-specific configuration file for + <c>syslogd</c>:</em> + <list type="bulleted"> + <item>Make an edited copy of the backup copy previously + made.</item> + <item>The filename must be <c>syslog.conf.OTP</c>. The + path must be the same as the backup copy.</item> + <item>The format of the configuration file is found in the + <c>syslog.conf(5)</c> manual page, by issuing the command + <c>man syslog.conf</c>.</item> + <item>Usually a line is added that is to state: + <list type="bulleted"> + <item>Which types of information that is to be + supervised by Erlang</item> + <item>The name of the file (actually a named pipe) that + is to receive the information</item> + </list> + </item> + <item>If, for example, only information originating from + the UNIX kernel is to be supervised, the line is to + begin with <c>kern.LEVEL</c>. For the possible + values of <c>LEVEL</c>, see <c>syslog.conf(5)</c>.</item> + <item>After at least one tab-character, the line added is to + contain the full name of the named pipe where <c>syslogd</c> + writes its information. The path must be the same as for the + files <c>syslog.conf.ORIG</c> and <c>syslog.conf.OTP</c>. + The filename must be <c>syslog.otp</c>.</item> + <item>If the directory for the files <c>syslog.conf.ORIG</c> + and <c>syslog.conf.OTP</c> is <c>/etc</c>, the line in + <c>syslog.conf.OTP</c> is as follows: + <code type="none"> kern.LEVEL /etc/syslog.otp</code> - </item> - </list> - </item> - <item> - <p><em>Check the file privileges of the configuration files.</em></p> - <list type="bulleted"> - <item> - <p>The configuration files should have <c>rw-r--r--</c> - file privileges and be owned by root.</p> - </item> - <item> - <p>A simple way to do this is to issue the commands</p> - <code type="none"> + </item> + </list> + </item> + <item><em>Check the file privileges of the configuration + files:</em> + <list type="bulleted"> + <item>The configuration files is to have <c>rw-r--r--</c> + file privileges and be owned by root.</item> + <item>A simple way to do this is to issue these commands: + <code type="none"> chmod 644 /etc/syslog.conf chmod 644 /etc/syslog.conf.ORIG chmod 644 /etc/syslog.conf.OTP</code> - </item> - <item> - <p><em>Note:</em> If the <c>syslog.conf.ORIG</c> and - <c>syslog.conf.OTP</c> files are not in the - <c>/etc</c> directory, the file path in the second - and third command must be modified.</p> - </item> - </list> - </item> - <item> - <p><em>Modify file privileges and ownership of the mod_syslog utility.</em></p> - <list type="bulleted"> - <item> - <p>The file privileges and ownership of the - <c>mod_syslog</c> utility must be modified.</p> - </item> - <item> - <p>The full name of the binary executable file is - derived from the position of the <c>os_mon</c> - application if the file system by adding - <c>/priv/bin/mod_syslog</c>. The generic full name - of the binary executable file is thus</p> - <code type="none"><![CDATA[ + </item> + <item>Notice that if the files <c>syslog.conf.ORIG</c> and + <c>syslog.conf.OTP</c> are not in directory <c>/etc</c>, + the file path in the second and third command must be + modified.</item> + </list> + </item> + <item><em>Modify file privileges and ownership of the + <c>mod_syslog</c> utility:</em> + <list type="bulleted"> + <item>The file privileges and ownership of the + <c>mod_syslog</c> utility must be modified.</item> + <item><p>The full name of the binary executable file is + derived from the position of application <c>os_mon</c> + in the file system by adding + <c>/priv/bin/mod_syslog</c>. The generic full name + of the binary executable file is thus:</p> + <code type="none"><![CDATA[ <OTP_ROOT>/lib/os_mon-<REV>/priv/bin/mod_syslog]]></code> - <p><em>Example:</em> If the path to the otp-root is - <c>/usr/otp</c>, thus the path to the <c>os_mon</c> - application is <c>/usr/otp/lib/os_mon-1.0</c> - (assuming revision 1.0) and the full name of the - binary executable file is - <c>/usr/otp/lib/os_mon-1.0/priv/bin/mod_syslog</c>.</p> - </item> - <item> - <p>The binary executable file must be owned by root, - have <c>rwsr-xr-x</c> file privileges, in particular - the setuid bit of user must be set. - </p> - </item> - <item> - <p>A simple way to do this is to issue the commands</p> - <code type="none"><![CDATA[ + <p><em>Example:</em> If the path to <c>otp-root</c> is + <c>/usr/otp</c>, then the path to the <c>os_mon</c> + application is <c>/usr/otp/lib/os_mon-1.0</c> + (assuming revision 1.0) and the full name of the + binary executable file is + <c>/usr/otp/lib/os_mon-1.0/priv/bin/mod_syslog</c>.</p> + </item> + <item>The binary executable file must be owned by root, + have <c>rwsr-xr-x</c> file privileges, in particular + the <c>setuid</c> bit of the user must be set.</item> + <item><p>A simple way to do this is to issue the following + commands:</p> + <code type="none"><![CDATA[ cd <OTP_ROOT>/lib/os_mon-<REV>/priv/bin/mod_syslog chmod 4755 mod_syslog chown root mod_syslog]]></code> - </item> - </list> - </item> - </list> - </section> - - <section> - <title>Testing the Application Configuration File</title> - <p>The following procedure does not require root privilege. - </p> - <list type="bulleted"> - <item> - <p>Ensure that the configuration parameters for the - <c>os_sup</c> module in the <c>os_mon</c> application - are correct.</p> - </item> - <item> - <p>Browse the application configuration file (do - <em>not</em> edit it). The full name of the application - configuration file is derived from the position of the - OS_Mon application if the file system by adding - <c>/ebin/os_mon.app</c>. - </p> - <p>The generic full name of the file is thus</p> - <code type="none"><![CDATA[ + </item> + </list> + </item> + </list> + </section> + + <section> + <title>Testing the Application Configuration File</title> + <p>The following procedure does not require root privilege:</p> + <list type="bulleted"> + <item>Ensure that the configuration parameters for the + <c>os_sup</c> module in the <c>os_mon</c> application + are correct.</item> + <item><p>Browse the application configuration file (do + <em>not</em> edit it). The full name of the application + configuration file is derived from the position of the + <c>os_mon</c> application in the file system by adding + <c>/ebin/os_mon.app</c>.</p> + <p>The generic full name of the file is thus:</p> + <code type="none"><![CDATA[ <OTP_ROOT>/lib/os_mon-<REV>/ebin/os_mon.app.]]></code> - <p><em>Example:</em> If the path to the otp-root is - <c>/usr/otp</c>, thus the path to the <c>os_mon</c> - application is <c>/usr/otp/lib/os_mon-1.0 </c> (assuming - revision 1.0) and the full name of the binary executable - file is <c>/usr/otp/lib/os_mon-1.0/ebin/os_mon.app</c>.</p> - </item> - <item> - <p>Ensure that the following configuration parameters are - bound to the correct values.</p> - </item> + <p><em>Example:</em> If the path to <c>otp-root</c> is + <c>/usr/otp</c>, then the path to the <c>os_mon</c> application + is <c>/usr/otp/lib/os_mon-1.0 </c> (assuming revision 1.0) and + the full name of the binary executable file is + <c>/usr/otp/lib/os_mon-1.0/ebin/os_mon.app</c>.</p> + </item> + <item>Ensure that the following configuration parameters have + correct values:</item> </list> - <table> + + <table> <row> <cell align="left" valign="top"><em>Parameter</em></cell> <cell align="left" valign="top"><em>Function</em></cell> <cell align="left" valign="top"><em>Standard value</em></cell> </row> <row> - <cell align="left" valign="middle">start_os_sup</cell> - <cell align="left" valign="middle">Specifies if os_sup will be started or not.</cell> - <cell align="left" valign="middle"><c>true</c>for the first instance on the hardware; <c>false</c>for the other instances.</cell> + <cell align="left" valign="middle"><c>start_os_sup</c></cell> + <cell align="left" valign="middle">Specifies if <c>os_sup</c> + is to be started or not.</cell> + <cell align="left" valign="middle"><c>true</c> for the + first instance on the hardware; <c>false</c> for the + other instances</cell> </row> <row> - <cell align="left" valign="middle">os_sup_own</cell> - <cell align="left" valign="middle">The directory for (1)the back-up copy, (2) the Erlang specific configuration file for syslogd.</cell> + <cell align="left" valign="middle"><c>os_sup_own</c></cell> + <cell align="left" valign="middle">The directory for + (1) back-up copy and (2) Erlang-specific configuration + file for <c>syslogd</c></cell> <cell align="left" valign="middle"><c>"/etc"</c></cell> </row> <row> - <cell align="left" valign="middle">os_sup_syslogconf</cell> - <cell align="left" valign="middle">The full name for the Solaris standard configuration file for syslogd </cell> + <cell align="left" valign="middle"><c>os_sup_syslogconf</c></cell> + <cell align="left" valign="middle">The full name for the + Solaris standard configuration file for <c>syslogd</c></cell> <cell align="left" valign="middle"><c>"/etc/syslog.conf"</c></cell> </row> <row> - <cell align="left" valign="middle">error_tag</cell> - <cell align="left" valign="middle">The tag for the messages that are sent to the error logger in the Erlang runtime system.</cell> + <cell align="left" valign="middle"><c>error_tag</c></cell> + <cell align="left" valign="middle">The tag for the + messages that are sent to the error logger in the Erlang + runtime system</cell> <cell align="left" valign="middle"><c>std_error</c></cell> </row> <tcaption>Configuration Parameters</tcaption> </table> - <p>If the values listed in the <c>os_mon.app</c> do not suit - your needs, you should <c>not</c> edit that file. Instead - you should <em>override</em> values in a <em>system configuration file</em>, the full pathname of which is given - on the command line to <c>erl</c>. - </p> - <p><em>Example:</em> The following is an example of the - contents of an application configuration file.</p> - <p></p> - <pre> + + <p>If the values listed in <c>os_mon.app</c> do not suit + your needs, do <em>not</em> edit that file. Instead + <em>override</em> the values in a <em>system configuration + file</em>, the full pathname of which is given + on the command line to <c>erl</c>.</p> + <p><em>Example:</em> Contents of an application configuration + file:</p> + <pre> [{os_mon, [{start_os_sup, true}, {os_sup_own, "/etc"}, {os_sup_syslogconf, "/etc/syslog.conf"}, {os_sup_errortag, std_error}]}].</pre> - </section> - - <section> - <title>Related Documents</title> - <p>See also the <c>os_mon(3)</c>, <c>application(3)</c> and - <c>erl(1)</c> reference manual pages.</p> - </section> </section> <section> - <title>Installation Problems</title> - <p>The hardware watchdog timer which is controlled by the - <c>heart</c> port program requires the <c>FORCEvme</c> - package, which contains the VME bus driver, to be - installed. This driver, however, may clash with the Sun - <c>mcp</c> driver and cause the system to completely refuse to - boot. To cure this problem, the following lines should be - added to <c>/etc/system</c>: - </p> + <title>Related Documents</title> + <p>See the <c>os_mon(3)</c> application, + the <c>application(3)</c> manual page in <c>kernel</c>, + and the <c>erl(1)</c> manual page in <c>erts</c>.</p> + </section> + </section> + + <section> + <title>Installation Problems</title> + <p>The hardware watchdog timer, which is controlled by the + <c>heart</c> port program, requires package <c>FORCEvme</c>, + which contains the VME bus driver, to be + installed. However, this driver can clash with the Sun + <c>mcp</c> driver and cause the system to refuse to + boot. To cure this problem, the following lines are + to be added to <c>/etc/system</c>:</p> <list type="bulleted"> <item><c>exclude: drv/mcp</c></item> <item><c>exclude: drv/mcpzsa</c></item> <item><c>exclude: drv/mcpp</c></item> </list> <warning> - <p>It is recommended that these lines be added to avoid the - clash described, which may make it completely impossible to - boot the system.</p> + <p>It is recommended to add these lines to avoid a clash. + The clash can make it impossible to boot the system.</p> </warning> </section> </section> <section> <title>Starting Erlang</title> - <p>This section describes how an embedded system is started. There - are four programs involved, and they all normally reside in the - directory <c><![CDATA[<ERL_INSTALL_DIR>/bin]]></c>. The only exception is - the program <c>start</c>, which may be located anywhere, and - also is the only program that must be modified by the user. - </p> - <p>In an embedded system there usually is no interactive shell. - However, it is possible for an operator to attach to the Erlang - system by giving the command <c>to_erl</c>. He is then - connected to the Erlang shell, and may give ordinary Erlang - commands. All interaction with the system through this shell is - logged in a special directory. - </p> - <p>Basically, the procedure is as follows. The program - <c>start</c> is called when the machine is started. It calls - <c>run_erl</c>, which sets things up so the operator can attach - to the system. It calls <c>start_erl</c> which calls the - correct version of <c>erlexec</c> (which is located in - <c><![CDATA[<ERL_INSTALL_DIR>/erts-EVsn/bin]]></c>) with the correct - <c>boot</c> and <c>config</c> files. - </p> + <p>This section describes how an embedded system is started. Four + programs are involved and they normally reside in the directory + <c><![CDATA[<ERL_INSTALL_DIR>/bin]]></c>. The only exception is + the <c>start</c> program, which can be located anywhere, and + is also the only program that must be modified by the user.</p> + <p>In an embedded system, there is usually no interactive shell. + However, an operator can attach to the Erlang + system by command <c>to_erl</c>. The operator is then + connected to the Erlang shell and can give ordinary Erlang + commands. All interaction with the system through this shell is + logged in a special directory.</p> + <p>Basically, the procedure is as follows:</p> + <list type="bulleted"> + <item>The <c>start</c> program is called when the machine + is started.</item> + <item>It calls <c>run_erl</c>, which sets up things so the + operator can attach to the system.</item> + <item>It calls <c>start_erl</c>, which calls the correct + version of <c>erlexec</c> (which is located in + <c><![CDATA[<ERL_INSTALL_DIR>/erts-EVsn/bin]]></c>) with the + correct <c>boot</c> and <c>config</c> files.</item> + </list> </section> <section> <title>Programs</title> - <section> <title>start</title> - <p>This program is called when the machine is started. It may - be modified or re-written to suit a special system. By + <p>This program is called when the machine is started. It can + be modified or rewritten to suit a special system. By default, it must be called <c>start</c> and reside in - <c><![CDATA[<ERL_INSTALL_DIR>/bin]]></c>. Another start program can be - used, by using the configuration parameter <c>start_prg</c> in - the application <c>sasl</c>.</p> + <c><![CDATA[<ERL_INSTALL_DIR>/bin]]></c>. Another start + program can be used, by using configuration parameter + <c>start_prg</c> in application <c>sasl</c>.</p> <p>The start program must call <c>run_erl</c> as shown below. - It must also take an optional parameter which defaults to - <c><![CDATA[<ERL_INSTALL_DIR>/releases/start_erl.data]]></c>. - </p> - <p>This program should set static parameters and environment + It must also take an optional parameter, which defaults to + <c><![CDATA[<ERL_INSTALL_DIR>/releases/start_erl.data]]></c>.</p> + <p>This program is to set static parameters and environment variables such as <c>-sname Name</c> and <c>HEART_COMMAND</c> - to reboot the machine. - </p> - <p>The <c><![CDATA[<RELDIR>]]></c> directory is where new release packets - are installed, and where the release handler keeps information - about releases. See <c>release_handler(3)</c> in the - application <c>sasl</c> for further information. - </p> + to reboot the machine.</p> + <p>The <c><![CDATA[<RELDIR>]]></c> directory is where new release + packets are installed, and where the release handler keeps + information about releases. For more information, see the + <c>release_handler(3)</c> manual page in <c>sasl</c>.</p> <p>The following script illustrates the default behaviour of the - program. - </p> + program:</p> <code type="none"><![CDATA[ #!/bin/sh # Usage: start [DataFile] @@ -583,10 +502,9 @@ START_ERL_DATA=${1:-$RELDIR/start_erl.data} $ROOTDIR/bin/run_erl /tmp/ $ROOTDIR/log "exec $ROOTDIR/bin/start_erl \ $ROOTDIR $RELDIR $START_ERL_DATA" > /dev/null 2>&1 &]]></code> <p>The following script illustrates a modification where the node - is given the name <c>cp1</c>, and the environment variables + is given the name <c>cp1</c>, and where the environment variables <c>HEART_COMMAND</c> and <c>TERM</c> have been added to the - above script. - </p> + previous script:</p> <code type="none"><![CDATA[ #!/bin/sh # Usage: start [DataFile] @@ -606,11 +524,10 @@ START_ERL_DATA=${1:-$RELDIR/start_erl.data} $ROOTDIR/bin/run_erl /tmp/ $ROOTDIR/log "exec $ROOTDIR/bin/start_erl \ $ROOTDIR $RELDIR $START_ERL_DATA -heart -sname cp1" > /dev/null 2>&1 &]]></code> - <p>If a diskless and/or read-only client node is about to start the - <c>start_erl.data</c> file is located in the client directory at - the master node. Thus, the <c>START_ERL_DATA</c> line should look - like: - </p> + <p>If a diskless and/or read-only client node is about to start, + file <c>start_erl.data</c> is located in the client directory at + the master node. Thus, the <c>START_ERL_DATA</c> line is to look + like:</p> <code type="none"> CLIENTDIR=$ROOTDIR/clients/clientname START_ERL_DATA=${1:-$CLIENTDIR/bin/start_erl.data}</code> @@ -620,22 +537,24 @@ START_ERL_DATA=${1:-$CLIENTDIR/bin/start_erl.data}</code> <title>run_erl</title> <p>This program is used to start the emulator, but you will not be connected to the shell. <c>to_erl</c> is used to connect to - the Erlang shell. - </p> + the Erlang shell.</p> <code type="none"> Usage: run_erl pipe_dir/ log_dir "exec command [parameters ...]"</code> - <p>Where <c>pipe_dir/</c> should be <c>/tmp/</c> (<c>to_erl</c> - uses this name by default) and <c>log_dir</c> is where the log - files are written. <c>command [parameters]</c> is executed, - and everything written to stdin and stdout is logged in the - <c>log_dir</c>. - </p> - <p>In the <c>log_dir</c>, log files are written. Each logfile - has a name of the form: <c>erlang.log.N</c> where N is a - generation number, ranging from 1 to 5. Each logfile holds up - to 100kB text. As time goes by the following logfiles will be - found in the logfile directory</p> - <code type="none"> +<p>Here:</p> + <list type="bulleted"> + <item><c>pipe_dir/</c> is to be <c>/tmp/</c> (<c>to_erl</c> + uses this name by default).</item> + <item><c>log_dir</c> is where the log files are written.</item> + <item><c>command [parameters]</c> is executed.</item> + <item>Everything written to <c>stdin</c> and <c>stdout</c> + is logged in <c>log_dir</c>.</item> + </list> + <p>Log files are written in <c>log_dir</c>. Each log file + has a name of the form <c>erlang.log.N</c>, where N is a + generation number, ranging from 1 to 5. Each log file holds up + to 100 kB text. As time goes by, the following log files are + found in the log file directory:</p> + <code type="none"> erlang.log.1 erlang.log.1, erlang.log.2 erlang.log.1, erlang.log.2, erlang.log.3 @@ -643,48 +562,40 @@ erlang.log.1, erlang.log.2, erlang.log.3, erlang.log.4 erlang.log.2, erlang.log.3, erlang.log.4, erlang.log.5 erlang.log.3, erlang.log.4, erlang.log.5, erlang.log.1 ...</code> - <p>with the most recent logfile being the right most in each row - of the above list. That is, the most recent file is the one - with the highest number, or if there are already four files, - the one before the skip. - </p> - <p>When a logfile is opened (for appending or created) a time - stamp is written to the file. If nothing has been written to + <p>The most recent log file is the rightmost in each row. That + is, the most recent file is the one with the highest number, or + if there are already four files, the one before the skip.</p> + <p>When a log file is opened (for appending or created), a time + stamp is written to the file. If nothing has been written to the log files for 15 minutes, a record is inserted that says - that we're still alive. - </p> + that we are still alive.</p> </section> <section> <title>to_erl</title> <p>This program is used to attach to a running Erlang runtime - system, started with <c>run_erl</c>. - </p> + system, started with <c>run_erl</c>.</p> <code type="none"> Usage: to_erl [pipe_name | pipe_dir]</code> - <p>Where <c>pipe_name</c> defaults to <c>/tmp/erlang.pipe.N</c>. - </p> + <p>Here <c>pipe_name</c> defaults to <c>/tmp/erlang.pipe.N</c>.</p> <p>To disconnect from the shell without exiting the Erlang - system, type <c>Ctrl-D</c>. - </p> + system, type <c>Ctrl-D</c>.</p> </section> <section> <title>start_erl</title> <p>This program starts the Erlang emulator with parameters - <c>-boot</c> and <c>-config</c> set. It reads data about - where these files are located from a file called - <c>start_erl.data</c> which is located in the <c><![CDATA[<RELDIR>]]></c>. - Each new release introduces a new data file. This file is - automatically generated by the release handler in Erlang. - </p> - <p>The following script illustrates the behaviour of the - program. - </p> + <c>-boot</c> and <c>-config</c> set. It reads data about + where these files are located from a file named + <c>start_erl.data</c>, which is located in + <c><![CDATA[<RELDIR>]]></c>. + Each new release introduces a new data file. This file is + automatically generated by the release handler in Erlang.</p> + <p>The following script illustrates the behaviour of the program:</p> <code type="none"> #!/bin/sh # -# This program is called by run_erl. It starts +# This program is called by run_erl. It starts # the Erlang emulator and sets -boot and -config parameters. # It should only be used at an embedded target system. # @@ -710,22 +621,23 @@ export PROGNAME export RELDIR exec $BINDIR/erlexec -boot $RELDIR/$VSN/start -config $RELDIR/$VSN/sys $*</code> + <p>If a diskless and/or read-only client node with the <c>sasl</c> configuration parameter <c>static_emulator</c> set - to <c>true</c> is about to start the <c>-boot</c> and - <c>-config</c> flags must be changed. As such a client cannot - read a new <c>start_erl.data</c> file (the file is not - possible to change dynamically) the boot and config files are + to <c>true</c> is about to start, the <c>-boot</c> and + <c>-config</c> flags must be changed.</p> + <p>As such a client cannot + read a new <c>start_erl.data</c> file (the file cannot + be changed dynamically). The boot and config files are always fetched from the same place (but with new contents if - a new release has been installed). The <c>release_handler</c> - copies this files to the <c>bin</c> directory in the client + a new release has been installed).</p> + <p>The <c>release_handler</c> + copies these files to the <c>bin</c> directory in the client directory at the master nodes whenever a new release is made - permanent. - </p> - <p>Assuming the same <c>CLIENTDIR</c> as above the last line - should look like: - </p> - <code type="none"> + permanent.</p> + <p>Assuming the same <c>CLIENTDIR</c> as above, the last line + is to look like:</p> + <code type="none"> exec $BINDIR/erlexec -boot $CLIENTDIR/bin/start \ -config $CLIENTDIR/bin/sys $*</code> </section> diff --git a/system/doc/embedded/part.xml b/system/doc/embedded/part.xml index e4366bd2c2..f3b44bf494 100644 --- a/system/doc/embedded/part.xml +++ b/system/doc/embedded/part.xml @@ -31,17 +31,17 @@ <rev>C</rev> <file></file> </header> + <description> - <p>This manual describes the issues that are specific + <marker id="embedded systems user guide"></marker> + <p>This section describes the issues that are specific for running Erlang on an embedded system. It describes the differences in installing and starting - Erlang compared to how it is done for a non-embedded system. - </p> - <p>Note that this is a supplementary document. You still need to - read the Installation Guide. - </p> - <p>There is also target architecture specific information in - the top level README file of the Erlang distribution.</p> + Erlang compared to how it is done for a non-embedded system.</p> + <note><p>This is a supplementary section. You also need to + read Section 1 Installation Guide.</p></note> + <p>There is also target architecture-specific information in + the top-level README file of the Erlang distribution.</p> </description> <xi:include href="embedded_solaris.xml"/> <xi:include href="embedded_nt.xml"/> diff --git a/system/doc/getting_started/conc_prog.xml b/system/doc/getting_started/conc_prog.xml index 2b64826a93..0dd9efb363 100644 --- a/system/doc/getting_started/conc_prog.xml +++ b/system/doc/getting_started/conc_prog.xml @@ -29,25 +29,26 @@ <file>conc_prog.xml</file> </header> + <marker id="Distributed Programming"></marker> <section> <title>Processes</title> <p>One of the main reasons for using Erlang instead of other functional languages is Erlang's ability to handle concurrency - and distributed programming. By concurrency we mean programs - which can handle several threads of execution at the same time. - For example, modern operating systems would allow you to use a - word processor, a spreadsheet, a mail client and a print job all - running at the same time. Of course each processor (CPU) in + and distributed programming. By concurrency is meant programs + that can handle several threads of execution at the same time. + For example, modern operating systems allow you to use a + word processor, a spreadsheet, a mail client, and a print job all + running at the same time. Each processor (CPU) in the system is probably only handling one thread (or job) at a - time, but it swaps between the jobs a such a rate that it gives + time, but it swaps between the jobs at such a rate that it gives the illusion of running them all at the same time. It is easy to - create parallel threads of execution in an Erlang program and it - is easy to allow these threads to communicate with each other. In - Erlang we call each thread of execution a <em>process</em>.</p> + create parallel threads of execution in an Erlang program and + to allow these threads to communicate with each other. In + Erlang, each thread of execution is called a <em>process</em>.</p> <p>(Aside: the term "process" is usually used when the threads of execution share no data with each other and the term "thread" when they share data in some way. Threads of execution in Erlang - share no data, that's why we call them processes).</p> + share no data, that is why they are called processes).</p> <p>The Erlang BIF <c>spawn</c> is used to create a new process: <c>spawn(Module, Exported_Function, List of Arguments)</c>. Consider the following module:</p> @@ -73,14 +74,14 @@ hello hello hello done</pre> - <p>We can see that function <c>say_something</c> writes its first - argument the number of times specified by second argument. Now - look at the function <c>start</c>. It starts two Erlang processes, - one which writes "hello" three times and one which writes - "goodbye" three times. Both of these processes use the function - <c>say_something</c>. Note that a function used in this way by - <c>spawn</c> to start a process must be exported from the module - (i.e. in the <c>-export</c> at the start of the module).</p> + <p>As shown, the function <c>say_something</c> writes its first + argument the number of times specified by second argument. + The function <c>start</c> starts two Erlang processes, + one that writes "hello" three times and one that writes + "goodbye" three times. Both processes use the function + <c>say_something</c>. Notice that a function used in this way by + <c>spawn</c>, to start a process, must be exported from the module + (that is, in the <c>-export</c> at the start of the module).</p> <pre> 9> <input>tut14:start().</input> hello @@ -90,19 +91,19 @@ hello goodbye hello goodbye</pre> - <p>Notice that it didn't write "hello" three times and then - "goodbye" three times, but the first process wrote a "hello", + <p>Notice that it did not write "hello" three times and then + "goodbye" three times. Instead, the first process wrote a "hello", the second a "goodbye", the first another "hello" and so forth. But where did the <0.63.0> come from? The return value of a - function is of course the return value of the last "thing" in - the function. The last thing in the function <c>start</c> is:</p> + function is the return value of the last "thing" in + the function. The last thing in the function <c>start</c> is</p> <code type="none"> spawn(tut14, say_something, [goodbye, 3]).</code> <p><c>spawn</c> returns a <em>process identifier</em>, or <em>pid</em>, which uniquely identifies the process. So <0.63.0> - is the pid of the <c>spawn</c> function call above. We will see - how to use pids in the next example.</p> - <p>Note as well that we have used ~p instead of ~w in + is the pid of the <c>spawn</c> function call above. + The next example shows how to use pids.</p> + <p>Notice also that ~p is used instead of ~w in <c>io:format</c>. To quote the manual: "~p Writes the data with standard syntax in the same way as ~w, but breaks terms whose printed representation is longer than one line into many lines @@ -112,8 +113,8 @@ spawn(tut14, say_something, [goodbye, 3]).</code> <section> <title>Message Passing</title> - <p>In the following example we create two processes which send - messages to each other a number of times.</p> + <p>In the following example two processes are created and + they send messages to each other a number of times.</p> <code type="none"> -module(tut15). @@ -157,13 +158,13 @@ Pong received ping Ping received pong ping finished Pong finished</pre> - <p>The function <c>start</c> first creates a process, let's call it - "pong":</p> + <p>The function <c>start</c> first creates a process, + let us call it "pong":</p> <code type="none"> Pong_PID = spawn(tut15, pong, [])</code> <p>This process executes <c>tut15:pong()</c>. <c>Pong_PID</c> is the process identity of the "pong" process. The function - <c>start</c> now creates another process "ping".</p> + <c>start</c> now creates another process "ping":</p> <code type="none"> spawn(tut15, ping, [3, Pong_PID]),</code> <p>This process executes:</p> @@ -181,7 +182,7 @@ receive pong() end.</code> <p>The <c>receive</c> construct is used to allow processes to wait - for messages from other processes. It has the format:</p> + for messages from other processes. It has the following format:</p> <code type="none"> receive pattern1 -> @@ -192,35 +193,37 @@ receive patternN actionsN end.</code> - <p>Note: no ";" before the <c>end</c>.</p> + <p>Notice there is no ";" before the <c>end</c>.</p> <p>Messages between Erlang processes are simply valid Erlang terms. - I.e. they can be lists, tuples, integers, atoms, pids etc.</p> + That is, they can be lists, tuples, integers, atoms, pids, + and so on.</p> <p>Each process has its own input queue for messages it receives. New messages received are put at the end of the queue. When a process executes a <c>receive</c>, the first message in the queue - is matched against the first pattern in the <c>receive</c>, if + is matched against the first pattern in the <c>receive</c>. If this matches, the message is removed from the queue and - the actions corresponding to the the pattern are executed.</p> + the actions corresponding to the pattern are executed.</p> <p>However, if the first pattern does not match, the second pattern - is tested, if this matches the message is removed from the queue + is tested. If this matches, the message is removed from the queue and the actions corresponding to the second pattern are executed. - If the second pattern does not match the third is tried and so on - until there are no more pattern to test. If there are no more - patterns to test, the first message is kept in the queue and we - try the second message instead. If this matches any pattern, + If the second pattern does not match, the third is tried and so on + until there are no more patterns to test. If there are no more + patterns to test, the first message is kept in the queue and + the second message is tried instead. If this matches any pattern, the appropriate actions are executed and the second message is removed from the queue (keeping the first message and any other - messages in the queue). If the second message does not match we - try the third message and so on until we reach the end of - the queue. If we reach the end of the queue, the process blocks + messages in the queue). If the second message does not match, + the third message is tried, and so on, until the end of + the queue is reached. If the end of the queue is reached, + the process blocks (stops execution) and waits until a new message is received and this procedure is repeated.</p> - <p>Of course the Erlang implementation is "clever" and minimizes + <p>The Erlang implementation is "clever" and minimizes the number of times each message is tested against the patterns in each <c>receive</c>.</p> <p>Now back to the ping pong example.</p> <p>"Pong" is waiting for messages. If the atom <c>finished</c> is - received, "pong" writes "Pong finished" to the output and as it + received, "pong" writes "Pong finished" to the output and, as it has nothing more to do, terminates. If it receives a message with the format:</p> <code type="none"> @@ -229,20 +232,20 @@ end.</code> <c>pong</c> to the process "ping":</p> <code type="none"> Ping_PID ! pong</code> - <p>Note how the operator "!" is used to send messages. The syntax + <p>Notice how the operator "!" is used to send messages. The syntax of "!" is:</p> <code type="none"> Pid ! Message</code> - <p>I.e. <c>Message</c> (any Erlang term) is sent to the process + <p>That is, <c>Message</c> (any Erlang term) is sent to the process with identity <c>Pid</c>.</p> <p>After sending the message <c>pong</c> to the process "ping", "pong" calls the <c>pong</c> function again, which causes it to - get back to the <c>receive</c> again and wait for another message. - Now let's look at the process "ping". Recall that it was started + get back to the <c>receive</c> again and wait for another message.</p> + <p>Now let us look at the process "ping". Recall that it was started by executing:</p> <code type="none"> tut15:ping(3, Pong_PID)</code> - <p>Looking at the function <c>ping/2</c> we see that the second + <p>Looking at the function <c>ping/2</c>, the second clause of <c>ping/2</c> is executed since the value of the first argument is 3 (not 0) (first clause head is <c>ping(0,Pong_PID)</c>, second clause head is @@ -250,9 +253,9 @@ tut15:ping(3, Pong_PID)</code> <p>The second clause sends a message to "pong":</p> <code type="none"> Pong_PID ! {ping, self()},</code> - <p><c>self()</c> returns the pid of the process which executes + <p><c>self()</c> returns the pid of the process that executes <c>self()</c>, in this case the pid of "ping". (Recall the code - for "pong", this will land up in the variable <c>Ping_PID</c> in + for "pong", this lands up in the variable <c>Ping_PID</c> in the <c>receive</c> previously explained.)</p> <p>"Ping" now waits for a reply from "pong":</p> <code type="none"> @@ -260,37 +263,37 @@ receive pong -> io:format("Ping received pong~n", []) end,</code> - <p>and writes "Ping received pong" when this reply arrives, after + <p>It writes "Ping received pong" when this reply arrives, after which "ping" calls the <c>ping</c> function again.</p> <code type="none"> ping(N - 1, Pong_PID)</code> <p><c>N-1</c> causes the first argument to be decremented until it becomes 0. When this occurs, the first clause of <c>ping/2</c> - will be executed:</p> + is executed:</p> <code type="none"> ping(0, Pong_PID) -> Pong_PID ! finished, io:format("ping finished~n", []);</code> <p>The atom <c>finished</c> is sent to "pong" (causing it to terminate as described above) and "ping finished" is written to - the output. "Ping" then itself terminates as it has nothing left + the output. "Ping" then terminates as it has nothing left to do.</p> </section> <section> <title>Registered Process Names</title> - <p>In the above example, we first created "pong" so as to be able - to give the identity of "pong" when we started "ping". I.e. in - some way "ping" must be able to know the identity of "pong" in - order to be able to send a message to it. Sometimes processes - which need to know each others identities are started completely + <p>In the above example, "pong" was first created to be able + to give the identity of "pong" when "ping" was started. That is, in + some way "ping" must be able to know the identity of "pong" to be + able to send a message to it. Sometimes processes + which need to know each other's identities are started independently of each other. Erlang thus provides a mechanism for processes to be given names so that these names can be used as identities instead of pids. This is done by using the <c>register</c> BIF:</p> <code type="none"> register(some_atom, Pid)</code> - <p>We will now re-write the ping pong example using this and giving + <p>Let us now rewrite the ping pong example using this and give the name <c>pong</c> to the "pong" process:</p> <code type="none"> -module(tut16). @@ -335,52 +338,57 @@ Pong received ping Ping received pong ping finished Pong finished</pre> - <p>In the <c>start/0</c> function,</p> + <p>Here the <c>start/0</c> function,</p> <code type="none"> register(pong, spawn(tut16, pong, [])),</code> <p>both spawns the "pong" process and gives it the name <c>pong</c>. - In the "ping" process we can now send messages to <c>pong</c> by:</p> + In the "ping" process, messages can be sent to <c>pong</c> by:</p> <code type="none"> pong ! {ping, self()},</code> - <p>so that <c>ping/2</c> now becomes <c>ping/1</c> as we don't have - to use the argument <c>Pong_PID</c>.</p> + <p><c>ping/2</c> now becomes <c>ping/1</c> as + the argument <c>Pong_PID</c> is not needed.</p> </section> <section> <title>Distributed Programming</title> - <p>Now let's re-write the ping pong program with "ping" and "pong" - on different computers. Before we do this, there are a few things - we need to set up to get this to work. The distributed Erlang + <p>Let us rewrite the ping pong program with "ping" and "pong" + on different computers. First a few things + are needed to set up to get this to work. The distributed Erlang implementation provides a basic security mechanism to prevent unauthorized access to an Erlang system on another computer. Erlang systems which talk to each other must have the same <em>magic cookie</em>. The easiest way to achieve this is by having a file called <c>.erlang.cookie</c> in your home - directory on all machines which on which you are going to run - Erlang systems communicating with each other (on Windows systems - the home directory is the directory where pointed to by the $HOME - environment variable - you may need to set this. On Linux or Unix - you can safely ignore this and simply create a file called - <c>.erlang.cookie</c> in the directory you get to after executing - the command <c>cd</c> without any argument). - The <c>.erlang.cookie</c> file should contain one line with - the same atom. For example, on Linux or Unix in the OS shell:</p> + directory on all machines on which you are going to run + Erlang systems communicating with each other: + </p> + <list type="bulleted"> + <item>On Windows systems the home directory is the directory + pointed out by the environment variable $HOME - you may need + to set this.</item> + <item> On Linux or UNIX + you can safely ignore this and simply create a file called + <c>.erlang.cookie</c> in the directory you get to after executing + the command <c>cd</c> without any argument.</item> + </list> + <p>The <c>.erlang.cookie</c> file is to contain a line with + the same atom. For example, on Linux or UNIX, in the OS shell:</p> <pre> $ <input>cd</input> $ <input>cat > .erlang.cookie</input> this_is_very_secret $ <input>chmod 400 .erlang.cookie</input></pre> - <p>The <c>chmod</c> above make the <c>.erlang.cookie</c> file + <p>The <c>chmod</c> above makes the <c>.erlang.cookie</c> file accessible only by the owner of the file. This is a requirement.</p> - <p>When you start an Erlang system which is going to talk to other - Erlang systems, you must give it a name, e.g.: </p> + <p>When you start an Erlang system that is going to talk to other + Erlang systems, you must give it a name, for example:</p> <pre> $ <input>erl -sname my_name</input></pre> <p>We will see more details of this later. If you want to experiment with distributed Erlang, but you only have one computer to work on, you can start two separate Erlang systems on the same computer but give them different names. Each Erlang - system running on a computer is called an Erlang node.</p> + system running on a computer is called an <em>Erlang node</em>.</p> <p>(Note: <c>erl -sname</c> assumes that all nodes are in the same IP domain and we can use only the first component of the IP address, if we want to use nodes in different domains we use @@ -420,10 +428,10 @@ start_pong() -> start_ping(Pong_Node) -> spawn(tut17, ping, [3, Pong_Node]).</code> - <p>Let us assume we have two computers called gollum and kosken. We - will start a node on kosken called ping and then a node on gollum + <p>Let us assume there are two computers called gollum and kosken. + First a node is started on kosken, called ping, and then a node on gollum, called pong.</p> - <p>On kosken (on a Linux/Unix system):</p> + <p>On kosken (on a Linux/UNIX system):</p> <pre> kosken> <input>erl -sname ping</input> Erlang (BEAM) emulator version 5.2.3.7 [hipe] [threads:0] @@ -437,12 +445,12 @@ Erlang (BEAM) emulator version 5.2.3.7 [hipe] [threads:0] Eshell V5.2.3.7 (abort with ^G) (pong@gollum)1></pre> - <p>Now we start the "pong" process on gollum:</p> + <p>Now the "pong" process on gollum is started:</p> <pre> (pong@gollum)1> <input>tut17:start_pong().</input> true</pre> - <p>and start the "ping" process on kosken (from the code above you - will see that a parameter of the <c>start_ping</c> function is + <p>And the "ping" process on kosken is started (from the code above you + can see that a parameter of the <c>start_ping</c> function is the node name of the Erlang system where "pong" is running):</p> <pre> (ping@kosken)1> <input>tut17:start_ping(pong@gollum).</input> @@ -451,8 +459,7 @@ Ping received pong Ping received pong Ping received pong ping finished</pre> - <p>Here we see that the ping pong program has run, on the "pong" - side we see:</p> + <p>As shown, the ping pong program has run. On the "pong" side:</p> <pre> (pong@gollum)2> Pong received ping @@ -460,28 +467,28 @@ Pong received ping Pong received ping Pong finished (pong@gollum)2></pre> - <p>Looking at the <c>tut17</c> code we see that the <c>pong</c> - function itself is unchanged, the lines:</p> + <p>Looking at the <c>tut17</c> code, you see that the <c>pong</c> + function itself is unchanged, the following lines work in the same way + irrespective of on which node the "ping" process is executes:</p> <code type="none"> {ping, Ping_PID} -> io:format("Pong received ping~n", []), Ping_PID ! pong,</code> - <p>work in the same way irrespective of on which node the "ping" - process is executing. Thus Erlang pids contain information about - where the process executes so if you know the pid of a process, - the "!" operator can be used to send it a message if the process - is on the same node or on a different node.</p> - <p>A difference is how we send messages to a registered process on + <p>Thus, Erlang pids contain information about + where the process executes. So if you know the pid of a process, + the "!" operator can be used to send it a message disregarding + if the process is on the same node or on a different node.</p> + <p>A difference is how messages are sent to a registered process on another node:</p> <code type="none"> {pong, Pong_Node} ! {ping, self()},</code> - <p>We use a tuple <c>{registered_name,node_name}</c> instead of + <p>A tuple <c>{registered_name,node_name}</c> is used instead of just the <c>registered_name</c>.</p> - <p>In the previous example, we started "ping" and "pong" from + <p>In the previous example, "ping" and "pong" were started from the shells of two separate Erlang nodes. <c>spawn</c> can also be - used to start processes in other nodes. The next example is - the ping pong program, yet again, but this time we will start - "ping" in another node:</p> + used to start processes in other nodes.</p> + <p>The next example is the ping pong program, yet again, + but this time "ping" is started in another node:</p> <code type="none"> -module(tut18). @@ -513,7 +520,7 @@ start(Ping_Node) -> register(pong, spawn(tut18, pong, [])), spawn(Ping_Node, tut18, ping, [3, node()]).</code> <p>Assuming an Erlang system called ping (but not the "ping" - process) has already been started on kosken, then on gollum we do:</p> + process) has already been started on kosken, then on gollum this is done:</p> <pre> (pong@gollum)1> <input>tut18:start(ping@kosken).</input> <3934.39.0> @@ -525,39 +532,40 @@ Pong received ping Ping received pong Pong finished ping finished</pre> - <p>Notice we get all the output on gollum. This is because the io + <p>Notice that all the output is received on gollum. This is because + the I/O system finds out where the process is spawned from and sends all output there.</p> </section> <section> <title>A Larger Example</title> - <p>Now for a larger example. We will make an extremely simple - "messenger". The messenger is a program which allows users to log + <p>Now for a larger example with a simple + "messenger". The messenger is a program that allows users to log in on different nodes and send simple messages to each other.</p> - <p>Before we start, let's note the following:</p> + <p>Before starting, notice the following:</p> <list type="bulleted"> <item> - <p>This example will just show the message passing logic - no - attempt at all has been made to provide a nice graphical user - interface. This can, of course, also be done in Erlang - but - that's another tutorial.</p> + <p>This example only shows the message passing logic - no + attempt has been made to provide a nice graphical user + interface, although this can also be done in Erlang.</p> </item> <item> - <p>This sort of problem can be solved more easily if you use - the facilities in OTP, which will also provide methods for - updating code on the fly etc. But again, that's another - tutorial.</p> + <p>This sort of problem can be solved easier by use of + the facilities in OTP, which also provide methods for + updating code on the fly and so on (see + <seealso marker="doc/design_principles:des_princ#otp design principles"> + OTP Design Principles</seealso>).</p> </item> <item> - <p>The first program we write will contain some inadequacies - regarding the handling of nodes which disappear. We will correct - these in a later version of the program.</p> + <p>The first program contains some inadequacies + regarding handling of nodes which disappear. + These are corrected in a later version of the program.</p> </item> </list> - <p>We will set up the messenger by allowing "clients" to connect to - a central server and say who and where they are. I.e. a user - won't need to know the name of the Erlang node where another user + <p>The messenger is set up by allowing "clients" to connect to + a central server and say who and where they are. That is, a user + does not need to know the name of the Erlang node where another user is located to send a message.</p> <p>File <c>messenger.erl</c>:</p> <marker id="ex"></marker> @@ -728,19 +736,19 @@ await_result() -> {messenger, What} -> % Normal response io:format("~p~n", [What]) end.</code> - <p>To use this program you need to:</p> + <p>To use this program, you need to:</p> <list type="bulleted"> - <item>configure the <c>server_node()</c> function</item> - <item>copy the compiled code (<c>messenger.beam</c>) to - the directory on each computer where you start Erlang.</item> + <item>Configure the <c>server_node()</c> function.</item> + <item>Copy the compiled code (<c>messenger.beam</c>) to + the directory on each computer where you start Erlang.</item> </list> - <p>In the following example of use of this program I have started - nodes on four different computers, but if you don't have that - many machines available on your network you could start up + <p>In the following example using this program, + nodes are started on four different computers. If you do not have that + many machines available on your network, you can start several nodes on the same machine.</p> - <p>We start up four Erlang nodes: messenger@super, c1@bilbo, + <p>Four Erlang nodes are started up: messenger@super, c1@bilbo, c2@kosken, c3@gollum.</p> - <p>First we start up a the server at messenger@super:</p> + <p>First the server at messenger@super is started up:</p> <pre> (messenger@super)1> <input>messenger:start_server().</input> true</pre> @@ -754,7 +762,7 @@ logged_on</pre> (c2@kosken)1> <input>messenger:logon(james).</input> true logged_on</pre> - <p>and Fred logs on at c3@gollum:</p> + <p>And Fred logs on at c3@gollum:</p> <pre> (c3@gollum)1> <input>messenger:logon(fred).</input> true @@ -764,7 +772,7 @@ logged_on</pre> (c1@bilbo)2> <input>messenger:message(fred, "hello").</input> ok sent</pre> - <p>And Fred receives the message and sends a message to Peter and + <p>Fred receives the message and sends a message to Peter and logs off:</p> <pre> Message from peter: "hello" @@ -779,27 +787,28 @@ logoff</pre> ok receiver_not_found</pre> <p>But this fails as Fred has already logged off.</p> - <p>First let's look at some of the new concepts we have introduced.</p> + <p>First let us look at some of the new concepts that have + been introduced.</p> <p>There are two versions of the <c>server_transfer</c> function: one with four arguments (<c>server_transfer/4</c>) and one with five (<c>server_transfer/5</c>). These are regarded by Erlang as two separate functions.</p> - <p>Note how we write the <c>server</c> function so that it calls - itself, via <c>server(User_List)</c>, and thus creates a loop. + <p>Notice how to write the <c>server</c> function so that it calls + itself, through <c>server(User_List)</c>, and thus creates a loop. The Erlang compiler is "clever" and optimizes the code so that this really is a sort of loop and not a proper function call. But - this only works if there is no code after the call, otherwise - the compiler will expect the call to return and make a proper + this only works if there is no code after the call. Otherwise, + the compiler expects the call to return and make a proper function call. This would result in the process getting bigger and bigger for every loop.</p> - <p>We use functions from the <c>lists</c> module. This is a very + <p>Functions in the <c>lists</c> module are used. This is a very useful module and a study of the manual page is recommended (<c>erl -man lists</c>). <c>lists:keymember(Key,Position,Lists)</c> looks through a list of tuples and looks at <c>Position</c> in each tuple to see if it is the same as <c>Key</c>. The first element is position 1. If it finds a tuple where the element at <c>Position</c> is the same as - Key, it returns <c>true</c>, otherwise <c>false</c>.</p> + <c>Key</c>, it returns <c>true</c>, otherwise <c>false</c>.</p> <pre> 3> <input>lists:keymember(a, 2, [{x,y,z},{b,b,b},{b,a,c},{q,r,s}]).</input> true @@ -812,82 +821,83 @@ false</pre> [{x,y,z},{b,b,b},{q,r,s}]</pre> <p><c>lists:keysearch</c> is like <c>lists:keymember</c>, but it returns <c>{value,Tuple_Found}</c> or the atom <c>false</c>.</p> - <p>There are a lot more very useful functions in the <c>lists</c> + <p>There are many very useful functions in the <c>lists</c> module.</p> - <p>An Erlang process will (conceptually) run until it does a + <p>An Erlang process (conceptually) runs until it does a <c>receive</c> and there is no message which it wants to receive - in the message queue. I say "conceptually" because the Erlang + in the message queue. "conceptually" is used here because the Erlang system shares the CPU time between the active processes in the system.</p> <p>A process terminates when there is nothing more for it to do, - i.e. the last function it calls simply returns and doesn't call + that is, the last function it calls simply returns and does not call another function. Another way for a process to terminate is for it to call <c>exit/1</c>. The argument to <c>exit/1</c> has a - special meaning which we will look at later. In this example we - will do <c>exit(normal)</c> which has the same effect as a + special meaning, which is discussed later. In this example, + <c>exit(normal)</c> is done, which has the same effect as a process running out of functions to call.</p> <p>The BIF <c>whereis(RegisteredName)</c> checks if a registered - process of name <c>RegisteredName</c> exists and return the pid - of the process if it does exist or the atom <c>undefined</c> if - it does not.</p> - <p>You should by now be able to understand most of the code above - so I'll just go through one case: a message is sent from one user - to another.</p> + process of name <c>RegisteredName</c> exists. If it exists, the pid of + that process is returned. If it does not exist, the atom + <c>undefined</c> is returned.</p> + <p>You should by now be able to understand most of the code in the + messenger-module. Let us study one case in detail: a message is + sent from one user to another.</p> <p>The first user "sends" the message in the example above by:</p> <code type="none"> messenger:message(fred, "hello")</code> <p>After testing that the client process exists:</p> <code type="none"> whereis(mess_client) </code> - <p>and a message is sent to <c>mess_client</c>:</p> + <p>And a message is sent to <c>mess_client</c>:</p> <code type="none"> mess_client ! {message_to, fred, "hello"}</code> <p>The client sends the message to the server by:</p> <code type="none"> {messenger, messenger@super} ! {self(), message_to, fred, "hello"},</code> - <p>and waits for a reply from the server.</p> + <p>And waits for a reply from the server.</p> <p>The server receives this message and calls:</p> <code type="none"> server_transfer(From, fred, "hello", User_List),</code> - <p>which checks that the pid <c>From</c> is in the <c>User_List</c>:</p> + <p>This checks that the pid <c>From</c> is in the <c>User_List</c>:</p> <code type="none"> lists:keysearch(From, 1, User_List) </code> - <p>If <c>keysearch</c> returns the atom <c>false</c>, some sort of + <p>If <c>keysearch</c> returns the atom <c>false</c>, some error has occurred and the server sends back the message:</p> <code type="none"> From ! {messenger, stop, you_are_not_logged_on}</code> - <p>which is received by the client which in turn does + <p>This is received by the client, which in turn does <c>exit(normal)</c> and terminates. If <c>keysearch</c> returns - <c>{value,{From,Name}}</c> we know that the user is logged on and - is his name (peter) is in variable <c>Name</c>. We now call:</p> + <c>{value,{From,Name}}</c> it is certain that the user is logged on and + that his name (peter) is in variable <c>Name</c>.</p> + <p>Let us now call:</p> <code type="none"> server_transfer(From, peter, fred, "hello", User_List)</code> - <p>Note that as this is <c>server_transfer/5</c> it is not the same - as the previous function <c>server_transfer/4</c>. We do another - <c>keysearch</c> on <c>User_List</c> to find the pid of the client - corresponding to fred:</p> + <p>Notice that as this is <c>server_transfer/5</c>, it is not the same + as the previous function <c>server_transfer/4</c>. Another + <c>keysearch</c> is done on <c>User_List</c> to find the pid of + the client corresponding to fred:</p> <code type="none"> lists:keysearch(fred, 2, User_List)</code> - <p>This time we use argument 2 which is the second element in - the tuple. If this returns the atom <c>false</c> we know that - fred is not logged on and we send the message:</p> + <p>This time argument 2 is used, which is the second element in + the tuple. If this returns the atom <c>false</c>, + fred is not logged on and the following message is sent:</p> <code type="none"> From ! {messenger, receiver_not_found};</code> - <p>which is received by the client, if <c>keysearch</c> returns:</p> + <p>This is received by the client.</p> + <p> If <c>keysearch</c> returns:</p> <code type="none"> {value, {ToPid, fred}}</code> - <p>we send the message:</p> + <p>The following message is sent to fred's client:</p> <code type="none"> ToPid ! {message_from, peter, "hello"}, </code> - <p>to fred's client and the message:</p> + <p>The following message is sent to peter's client:</p> <code type="none"> From ! {messenger, sent} </code> - <p>to peter's client.</p> <p>Fred's client receives the message and prints it:</p> <code type="none"> {message_from, peter, "hello"} -> io:format("Message from ~p: ~p~n", [peter, "hello"])</code> - <p>and peter's client receives the message in + <p>Peter's client receives the message in the <c>await_result</c> function.</p> </section> </chapter> diff --git a/system/doc/getting_started/intro.xml b/system/doc/getting_started/intro.xml index e8d568bcaf..f9a56e4322 100644 --- a/system/doc/getting_started/intro.xml +++ b/system/doc/getting_started/intro.xml @@ -18,7 +18,7 @@ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. - + </legalnotice> <title>Introduction</title> @@ -28,38 +28,47 @@ <rev></rev> <file>intro.xml</file> </header> + <marker id="getting started"></marker> + + <p>This section is a quick start tutorial to get you started with Erlang. + Everything in this section is true, but only part of the truth. For example, + only the simplest form of the syntax is shown, not all esoteric forms. + Also, parts that are greatly simplified are indicated with *manual*. + This means that a lot more information on the subject is to be found in + the Erlang book or in + <seealso marker="doc/reference_manual:introduction#erlang ref manual"> + Erlang Reference Manual</seealso>.</p> <section> - <title>Introduction</title> - <p>This is a "kick start" tutorial to get you started with Erlang. - Everything here is true, but only part of the truth. For example, - I'll only tell you the simplest form of the syntax, not all - esoteric forms. Where I've greatly oversimplified things I'll - write *manual* which means there is lots more information to be - found in the Erlang book or in the <em>Erlang Reference Manual</em>.</p> - <p>I also assume that this isn't the first time you have touched a - computer and you have a basic idea about how they are programmed. - Don't worry, I won't assume you're a wizard programmer.</p> + <title>Prerequisites</title> + + <p>The reader of this section is assumed to be familiar with the following:</p> + <list type="bulleted"> + <item>Computers in general</item> + <item>Basics on how computers are programmed</item> + </list> + </section> <section> - <title>Things Left Out</title> - <p>In particular the following has been omitted:</p> + <title>Omitted Topics</title> + + <p>The following topics are not treated in this section:</p> <list type="bulleted"> - <item>References</item> - <item>Local error handling (catch/throw)</item> - <item>Single direction links (monitor)</item> - <item>Handling of binary data (binaries / bit syntax)</item> - <item>List comprehensions</item> - <item>How to communicate with the outside world and/or software - written in other languages (ports). There is however a separate - tutorial for this, <em>Interoperability Tutorial</em></item> - <item>Very few of the Erlang libraries have been touched on (for - example file handling)</item> - <item>OTP has been totally skipped and in consequence the Mnesia - database has been skipped.</item> - <item>Hash tables for Erlang terms (ETS)</item> - <item>Changing code in running systems</item> + <item>References.</item> + <item>Local error handling (catch/throw).</item> + <item>Single direction links (monitor).</item> + <item>Handling of binary data (binaries / bit syntax).</item> + <item>List comprehensions.</item> + <item>How to communicate with the outside world and software + written in other languages (ports); + this is described in + <seealso marker="doc/tutorial:introduction#interoperability tutorial"> + Interoperability Tutorial</seealso>.</item> + <item>Erlang libraries (for example, file handling).</item> + <item>OTP and (in consequence) the Mnesia database.</item> + <item>Hash tables for Erlang terms (ETS).</item> + <item>Changing code in running systems.</item> </list> </section> </chapter> diff --git a/system/doc/getting_started/records_macros.xml b/system/doc/getting_started/records_macros.xml index 73c8ce5c8d..bec751fea2 100644 --- a/system/doc/getting_started/records_macros.xml +++ b/system/doc/getting_started/records_macros.xml @@ -29,27 +29,32 @@ <file>record_macros.xml</file> </header> <p>Larger programs are usually written as a collection of files with - a well defined interface between the various parts.</p> + a well-defined interface between the various parts.</p> <section> <title>The Larger Example Divided into Several Files</title> - <p>To illustrate this, we will divide the messenger example from - the previous chapter into five files.</p> - <taglist> - <tag><c>mess_config.hrl</c></tag> - <item>header file for configuration data</item> - <tag><c>mess_interface.hrl</c></tag> - <item>interface definitions between the client and the messenger</item> - <tag><c>user_interface.erl</c></tag> - <item>functions for the user interface</item> - <tag><c>mess_client.erl</c></tag> - <item>functions for the client side of the messenger</item> - <tag><c>mess_server.erl</c></tag> - <item>functions for the server side of the messenger</item> - </taglist> - <p>While doing this we will also clean up the message passing - interface between the shell, the client and the server and define - it using <em>records</em>, we will also introduce <em>macros</em>.</p> + <p>To illustrate this, the messenger example from + the previous section is divided into the following five files:</p> + <list type="bulleted"> + <item> + <p><c>mess_config.hrl</c></p> + <p>Header file for configuration data</p></item> + <item> + <p><c>mess_interface.hrl</c></p> + <p>Interface definitions between the client and the messenger</p></item> + <item> + <p><c>user_interface.erl</c></p> + <p>Functions for the user interface</p></item> + <item> + <p><c>mess_client.erl</c></p> + <p>Functions for the client side of the messenger</p></item> + <item> + <p><c>mess_server.erl</c></p> + <p>Functions for the server side of the messenger</p></item> + </list> + <p>While doing this, the message passing interface between the shell, + the client, and the server is cleaned up and is defined + using <em>records</em>. Also, <em>macros</em> are introduced:</p> <code type="none"> %%%----FILE mess_config.hrl---- @@ -244,14 +249,14 @@ server_transfer(From, Name, To, Message, User_List) -> <section> <title>Header Files</title> - <p>You will see some files above with extension <c>.hrl</c>. These - are header files which are included in the <c>.erl</c> files by:</p> + <p>As shown above, some files have extension <c>.hrl</c>. These + are header files that are included in the <c>.erl</c> files by:</p> <code type="none"> -include("File_Name").</code> <p>for example:</p> <code type="none"> -include("mess_interface.hrl").</code> - <p>In our case above the file is fetched from the same directory as + <p>In the case above the file is fetched from the same directory as all the other files in the messenger example. (*manual*).</p> <p>.hrl files can contain any valid Erlang code but are most often used for record and macro definitions.</p> @@ -265,64 +270,63 @@ server_transfer(From, Name, To, Message, User_List) -> <p>For example:</p> <code type="none"> -record(message_to,{to_name, message}).</code> - <p>This is exactly equivalent to:</p> + <p>This is equivalent to:</p> <code type="none"> {message_to, To_Name, Message}</code> - <p>Creating record, is best illustrated by an example:</p> + <p>Creating a record is best illustrated by an example:</p> <code type="none"> #message_to{message="hello", to_name=fred)</code> - <p>This will create:</p> + <p>This creates:</p> <code type="none"> {message_to, fred, "hello"}</code> - <p>Note that you don't have to worry about the order you assign + <p>Notice that you do not have to worry about the order you assign values to the various parts of the records when you create it. The advantage of using records is that by placing their definitions in header files you can conveniently define - interfaces which are easy to change. For example, if you want to - add a new field to the record, you will only have to change + interfaces that are easy to change. For example, if you want to + add a new field to the record, you only have to change the code where the new field is used and not at every place the record is referred to. If you leave out a field when creating - a record, it will get the value of the atom undefined. (*manual*)</p> + a record, it gets the value of the atom <c>undefined</c>. (*manual*)</p> <p>Pattern matching with records is very similar to creating records. For example, inside a <c>case</c> or <c>receive</c>:</p> <code type="none"> #message_to{to_name=ToName, message=Message} -></code> - <p>is the same as:</p> + <p>This is the same as:</p> <code type="none"> {message_to, ToName, Message}</code> </section> <section> <title>Macros</title> - <p>The other thing we have added to the messenger is a macro. + <p>Another thing that has been added to the messenger is a macro. The file <c>mess_config.hrl</c> contains the definition:</p> <code type="none"> %%% Configure the location of the server node, -define(server_node, messenger@super).</code> - <p>We include this file in mess_server.erl:</p> + <p>This file is included in <c>mess_server.erl</c>:</p> <code type="none"> -include("mess_config.hrl").</code> <p>Every occurrence of <c>?server_node</c> in <c>mess_server.erl</c> - will now be replaced by <c>messenger@super</c>.</p> - <p>The other place a macro is used is when we spawn the server - process:</p> + is now replaced by <c>messenger@super</c>.</p> + <p>A macro is also used when spawning the server process:</p> <code type="none"> spawn(?MODULE, server, [])</code> - <p>This is a standard macro (i.e. defined by the system, not - the user). <c>?MODULE</c> is always replaced by the name of - current module (i.e. the <c>-module</c> definition near the start + <p>This is a standard macro (that is, defined by the system, not by + the user). <c>?MODULE</c> is always replaced by the name of the + current module (that is, the <c>-module</c> definition near the start of the file). There are more advanced ways of using macros with, - for example parameters (*manual*).</p> + for example, parameters (*manual*).</p> <p>The three Erlang (<c>.erl</c>) files in the messenger example are individually compiled into object code file (<c>.beam</c>). The Erlang system loads and links these files into the system - when they are referred to during execution of the code. In our - case we simply have put them in the same directory which is our - current working directory (i.e. the place we have done "cd" to). + when they are referred to during execution of the code. In this + case, they are simply put in our current working directory + (that is, the place you have done "cd" to). There are ways of putting the <c>.beam</c> files in other directories.</p> <p>In the messenger example, no assumptions have been made about - what the message being sent is. It could be any valid Erlang term.</p> + what the message being sent is. It can be any valid Erlang term.</p> </section> </chapter> diff --git a/system/doc/getting_started/robustness.xml b/system/doc/getting_started/robustness.xml index e8fb81d5e8..82fe0cbc4f 100644 --- a/system/doc/getting_started/robustness.xml +++ b/system/doc/getting_started/robustness.xml @@ -28,27 +28,27 @@ <rev></rev> <file>robustness.xml</file> </header> - <p>There are several things which are wrong with - the <seealso marker="conc_prog#ex">messenger example</seealso> from - the previous chapter. For example, if a node where a user is logged - on goes down without doing a log off, the user will remain in - the server's <c>User_List</c> but the client will disappear thus - making it impossible for the user to log on again as the server - thinks the user already logged on.</p> + <p>Several things are wrong with the messenger example in + <seealso marker="conc_prog#ex">A Larger Example</seealso>. + For example, if a node where a user is logged + on goes down without doing a logoff, the user remains in + the server's <c>User_List</c>, but the client disappears. This + makes it impossible for the user to log on again as the server + thinks the user already is logged on.</p> <p>Or what happens if the server goes down in the middle of sending a - message leaving the sending client hanging for ever in + message, leaving the sending client hanging forever in the <c>await_result</c> function?</p> <section> - <title>Timeouts</title> - <p>Before improving the messenger program we will look into some + <title>Time-outs</title> + <p>Before improving the messenger program, let us look at some general principles, using the ping pong program as an example. Recall that when "ping" finishes, it tells "pong" that it has done so by sending the atom <c>finished</c> as a message to "pong" - so that "pong" could also finish. Another way to let "pong" - finish, is to make "pong" exit if it does not receive a message - from ping within a certain time, this can be done by adding a - <em>timeout</em> to <c>pong</c> as shown in the following example:</p> + so that "pong" can also finish. Another way to let "pong" + finish is to make "pong" exit if it does not receive a message + from ping within a certain time. This can be done by adding a + <em>time-out</em> to <c>pong</c> as shown in the following example:</p> <code type="none"> -module(tut19). @@ -80,9 +80,9 @@ start_pong() -> start_ping(Pong_Node) -> spawn(tut19, ping, [3, Pong_Node]).</code> - <p>After we have compiled this and copied the <c>tut19.beam</c> - file to the necessary directories:</p> - <p>On (pong@kosken):</p> + <p>After this is compiled and the file <c>tut19.beam</c> + is copied to the necessary directories, the following is seen + on (pong@kosken): </p> <pre> (pong@kosken)1> <input>tut19:start_pong().</input> true @@ -90,7 +90,7 @@ Pong received ping Pong received ping Pong received ping Pong timed out</pre> - <p>On (ping@gollum):</p> + <p>And the following is seen on (ping@gollum):</p> <pre> (ping@gollum)1> <input>tut19:start_ping(pong@kosken).</input> <0.36.0> @@ -98,7 +98,7 @@ Ping received pong Ping received pong Ping received pong ping finished </pre> - <p>(The timeout is set in:</p> + <p>The time-out is set in:</p> <code type="none"> pong() -> receive @@ -109,35 +109,36 @@ pong() -> after 5000 -> io:format("Pong timed out~n", []) end.</code> - <p>We start the timeout (<c>after 5000</c>) when we enter - <c>receive</c>. The timeout is canceled if <c>{ping,Ping_PID}</c> + <p>The time-out (<c>after 5000</c>) is started when + <c>receive</c> is entered. + The time-out is canceled if <c>{ping,Ping_PID}</c> is received. If <c>{ping,Ping_PID}</c> is not received, - the actions following the timeout will be done after 5000 + the actions following the time-out are done after 5000 milliseconds. <c>after</c> must be last in the <c>receive</c>, - i.e. preceded by all other message reception specifications in - the <c>receive</c>. Of course we could also call a function which - returned an integer for the timeout:</p> + that is, preceded by all other message reception specifications in + the <c>receive</c>. It is also possible to call a function that + returned an integer for the time-out:</p> <code type="none"> after pong_timeout() -></code> - <p>In general, there are better ways than using timeouts to - supervise parts of a distributed Erlang system. Timeouts are - usually appropriate to supervise external events, for example if + <p>In general, there are better ways than using time-outs to + supervise parts of a distributed Erlang system. Time-outs are + usually appropriate to supervise external events, for example, if you have expected a message from some external system within a - specified time. For example, we could use a timeout to log a user - out of the messenger system if they have not accessed it, for - example, in ten minutes.</p> + specified time. For example, a time-out can be used to log a user + out of the messenger system if they have not accessed it for, + say, ten minutes.</p> </section> <section> <title>Error Handling</title> - <p>Before we go into details of the supervision and error handling - in an Erlang system, we need see how Erlang processes terminate, + <p>Before going into details of the supervision and error handling + in an Erlang system, let us see how Erlang processes terminate, or in Erlang terminology, <em>exit</em>.</p> <p>A process which executes <c>exit(normal)</c> or simply runs out of things to do has a <em>normal</em> exit.</p> - <p>A process which encounters a runtime error (e.g. divide by zero, - bad match, trying to call a function which doesn't exist etc) - exits with an error, i.e. has an <em>abnormal</em> exit. A + <p>A process which encounters a runtime error (for example, divide by zero, + bad match, trying to call a function that does not exist and so on) + exits with an error, that is, has an <em>abnormal</em> exit. A process which executes <seealso marker="erts:erlang#exit/1">exit(Reason)</seealso> where <c>Reason</c> is any Erlang term except the atom @@ -151,18 +152,23 @@ after pong_timeout() -></code> links to.</p> <p>The signal carries information about the pid it was sent from and the exit reason.</p> - <p>The default behaviour of a process which receives a normal exit + <p>The default behaviour of a process that receives a normal exit is to ignore the signal.</p> - <p>The default behaviour in the two other cases (i.e. abnormal exit) - above is to bypass all messages to the receiving process and to - kill it and to propagate the same error signal to the killed - process' links. In this way you can connect all processes in a - transaction together using links and if one of the processes - exits abnormally, all the processes in the transaction will be - killed. As we often want to create a process and link to it at + <p>The default behaviour in the two other cases (that is, abnormal exit) + above is to:</p> + <list type="bulleted"> + <item>Bypass all messages to the receiving process.</item> + <item>Kill the receiving process.</item> + <item>Propagate the same error signal to the links of the + killed process.</item> + </list> + <p>In this way you can connect all processes in a + transaction together using links. If one of the processes + exits abnormally, all the processes in the transaction are + killed. As it is often wanted to create a process and link to it at the same time, there is a special BIF, <seealso marker="erts:erlang#spawn_link/1">spawn_link</seealso> - which does the same as <c>spawn</c>, but also creates a link to + that does the same as <c>spawn</c>, but also creates a link to the spawned process.</p> <p>Now an example of the ping pong example using links to terminate "pong":</p> @@ -208,13 +214,13 @@ Pong received ping Ping received pong</pre> <p>This is a slight modification of the ping pong program where both processes are spawned from the same <c>start/1</c> function, - where the "ping" process can be spawned on a separate node. Note + and the "ping" process can be spawned on a separate node. Notice the use of the <c>link</c> BIF. "Ping" calls - <c>exit(ping)</c> when it finishes and this will cause an exit - signal to be sent to "pong" which will also terminate.</p> + <c>exit(ping)</c> when it finishes and this causes an exit + signal to be sent to "pong", which also terminates.</p> <p>It is possible to modify the default behaviour of a process so that it does not get killed when it receives abnormal exit - signals, but all signals will be turned into normal messages with + signals. Instead, all signals are turned into normal messages on the format <c>{'EXIT',FromPID,Reason}</c> and added to the end of the receiving process' message queue. This behaviour is set by:</p> <code type="none"> @@ -223,8 +229,8 @@ process_flag(trap_exit, true)</code> <seealso marker="erts:erlang#process_flag/2">erlang(3)</seealso>. Changing the default behaviour of a process in this way is usually not done in standard user programs, but is left to - the supervisory programs in OTP (but that's another tutorial). - However we will modify the ping pong program to illustrate exit + the supervisory programs in OTP. + However, the ping pong program is modified to illustrate exit trapping.</p> <code type="none"> -module(tut21). @@ -277,7 +283,7 @@ pong exiting, got {'EXIT',<3820.39.0>,ping}</pre> <section> <title>The Larger Example with Robustness Added</title> - <p>Now we return to the messenger program and add changes which + <p>Let us return to the messenger program and add changes to make it more robust:</p> <code type="none"> %%% Message passing utility. @@ -449,35 +455,34 @@ await_result() -> io:format("No response from server~n", []), exit(timeout) end.</code> - <p>We have added the following changes:</p> + <p>The following changes are added:</p> <p>The messenger server traps exits. If it receives an exit signal, - <c>{'EXIT',From,Reason}</c> this means that a client process has - terminated or is unreachable because:</p> + <c>{'EXIT',From,Reason}</c>, this means that a client process has + terminated or is unreachable for one of the following reasons:</p> <list type="bulleted"> - <item>the user has logged off (we have removed the "logoff" - message),</item> - <item>the network connection to the client is broken,</item> - <item>the node on which the client process resides has gone down, - or</item> - <item>the client processes has done some illegal operation.</item> + <item>The user has logged off (the "logoff" + message is removed).</item> + <item>The network connection to the client is broken.</item> + <item>The node on which the client process resides has gone down.</item> + <item>The client processes has done some illegal operation.</item> </list> - <p>If we receive an exit signal as above, we delete the tuple, - <c>{From,Name}</c> from the servers <c>User_List</c> using + <p>If an exit signal is received as above, the tuple + <c>{From,Name}</c> is deleted from the servers <c>User_List</c> using the <c>server_logoff</c> function. If the node on which the server runs goes down, an exit signal (automatically generated by - the system), will be sent to all of the client processes: + the system) is sent to all of the client processes: <c>{'EXIT',MessengerPID,noconnection}</c> causing all the client processes to terminate.</p> - <p>We have also introduced a timeout of five seconds in - the <c>await_result</c> function. I.e. if the server does not + <p>Also, a time-out of five seconds has been introduced in + the <c>await_result</c> function. That is, if the server does not reply within five seconds (5000 ms), the client terminates. This - is really only needed in the logon sequence before the client and + is only needed in the logon sequence before the client and the server are linked.</p> - <p>An interesting case is if the client was to terminate before + <p>An interesting case is if the client terminates before the server links to it. This is taken care of because linking to a non-existent process causes an exit signal, - <c>{'EXIT',From,noproc}</c>, to be automatically generated as if - the process terminated immediately after the link operation.</p> + <c>{'EXIT',From,noproc}</c>, to be automatically generated. This is + as if the process terminated immediately after the link operation.</p> </section> </chapter> diff --git a/system/doc/getting_started/seq_prog.xml b/system/doc/getting_started/seq_prog.xml index 699b9487ed..5d96aed8d4 100644 --- a/system/doc/getting_started/seq_prog.xml +++ b/system/doc/getting_started/seq_prog.xml @@ -31,11 +31,15 @@ <section> <title>The Erlang Shell</title> - <p>Most operating systems have a command interpreter or shell- Unix - and Linux have many, while Windows has the Command Prompt. Erlang has - its own shell where you can directly write bits of Erlang code - and evaluate (run) them to see what happens (see - <seealso marker="stdlib:shell">shell(3)</seealso>). Start + <p> + Most operating systems have a command interpreter or shell, UNIX + and Linux have many, Windows has the command prompt. Erlang has + its own shell where bits of Erlang code can be written directly, + and be evaluated to see what happens + (see the <seealso marker="stdlib:shell">shell(3)</seealso> + manual page in STDLIB). + </p> + <p>Start the Erlang shell (in Linux or UNIX) by starting a shell or command interpreter in your operating system and typing <c>erl</c>. You will see something like this.</p> @@ -45,41 +49,39 @@ Erlang R15B (erts-5.9.1) [source] [smp:8:8] [rq:8] [async-threads:0] [hipe] [ker Eshell V5.9.1 (abort with ^G) 1></pre> - <p>Now type in "2 + 5." as shown below.</p> + <p>Type "2 + 5." in the shell and then press Enter (carriage return). + Notice that you tell the shell you are done entering code by finishing + with a full stop "." and a carriage return.</p> <pre> 1> <input>2 + 5.</input> 7 2></pre> - <p>In Windows, the shell is started by double-clicking on the Erlang - shell icon.</p> - <p>You'll notice that the Erlang shell has numbered the lines that - can be entered, (as 1> 2>) and that it has correctly told you - that 2 + 5 is 7! Also notice that you have to tell it you are - done entering code by finishing with a full stop "." and a - carriage return. If you make mistakes writing things in the shell, - you can delete things by using the backspace key as in most - shells. There are many more editing commands in the shell - (See the chapter <seealso marker="erts:tty">"tty - A command line interface"</seealso> in ERTS User's Guide).</p> - <p>(Note: you will find a lot of line numbers given by the shell - out of sequence in this tutorial as it was written and the code - tested in several sessions.)</p> - <p>Now let's try a more complex calculation.</p> + <p>As shown, the Erlang shell numbers the lines that + can be entered, (as 1> 2>) and that it correctly says + that 2 + 5 is 7. If you make writing mistakes in the shell, + you can delete with the backspace key, as in most shells. + There are many more editing commands in the shell + (see <seealso marker="erts:tty">tty - A command line interface</seealso> in ERTS User's Guide).</p> + <p>(Notice that many line numbers given by the shell in the + following examples are out of sequence. This is because this + tutorial was written and code-tested in separate sessions).</p> + <p>Here is a bit more complex calculation:</p> <pre> 2> <input>(42 + 77) * 66 / 3.</input> 2618.0</pre> - <p>Here you can see the use of brackets and the multiplication - operator "*" and division operator "/", just as in normal - arithmetic (see the chapter - <seealso marker="doc/reference_manual:expressions">"Arithmetic Expressions"</seealso> in the Erlang Reference Manual).</p> - <p>To shutdown the Erlang system and the Erlang shell type - Control-C. You will see the following output:</p> + <p>Notice the use of brackets, the multiplication operator "*", + and the division operator "/", as in normal arithmetic (see + <seealso marker="doc/reference_manual:expressions">Expressions</seealso>).</p> + <p>Press Control-C to shut down the Erlang system and the Erlang + shell.</p> + <p>The following output is shown:</p> <pre> BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution <input>a</input> %</pre> <p>Type "a" to leave the Erlang system.</p> - <p>Another way to shutdown the Erlang system is by entering + <p>Another way to shut down the Erlang system is by entering <c>halt()</c>:</p> <pre> 3> <input>halt().</input> @@ -88,67 +90,70 @@ BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded <section> <title>Modules and Functions</title> - <p>A programming language isn't much use if you can just run code + <p>A programming language is not much use if you only can run code from the shell. So here is a small Erlang program. Enter it into - a file called <c>tut.erl</c> (the file name <c>tut.erl</c> is - important, also make sure that it is in the same directory as - the one where you started <c>erl</c>) using a suitable - text editor. If you are lucky your editor will have an Erlang - mode which will make it easier for you to enter and format your - code nicely (see the chapter - <seealso marker="tools:erlang_mode_chapter">"The Erlang mode for Emacs"</seealso> in Tools User's Guide), but you can manage - perfectly well without. Here's the code to enter:</p> + a file named <c>tut.erl</c> using a suitable + text editor. The file name <c>tut.erl</c> is important, and also + that it is in the same directory as the one where you started + <c>erl</c>). If you are lucky your editor has an Erlang mode + that makes it easier for you to enter and format your code + nicely (see <seealso + marker="tools:erlang_mode_chapter">The Erlang mode for + Emacs</seealso> in Tools User's Guide), but you can manage + perfectly well without. Here is the code to enter:</p> <code type="none"> -module(tut). -export([double/1]). double(X) -> 2 * X.</code> - <p>It's not hard to guess that this "program" doubles the value of - numbers. I'll get back to the first two lines later. Let's compile - the program. This can be done in your Erlang shell as shown below:</p> + <p>It is not hard to guess that this program doubles the value of + numbers. The first two lines of the code are described later. + Let us compile the program. This can be done in an Erlang shell + as follows, where <c>c</c> means compile:</p> <pre> 3> <input>c(tut).</input> {ok,tut}</pre> - <p>The <c>{ok,tut}</c> tells you that the compilation was OK. If it - said "error" instead, you have made some mistake in the text you - entered and there will also be error messages to give you some - idea as to what has gone wrong so you can change what you have - written and try again.</p> - <p>Now let's run the program.</p> + <p>The <c>{ok,tut}</c> means that the compilation is OK. If it + says "error" it means that there is some mistake in the text + that you entered. Additional error messages gives an idea to + what is wrong so you can modify the text and then try to compile + the program again.</p> + <p>Now run the program:</p> <pre> 4> <input>tut:double(10).</input> 20</pre> - <p>As expected double of 10 is 20.</p> - <p>Now let's get back to the first two lines. Erlang programs are - written in files. Each file contains what we call an Erlang - <em>module</em>. The first line of code in the module tells us - the name of the module (see the chapter - <seealso marker="doc/reference_manual:modules">"Modules"</seealso> - in the Erlang Reference Manual).</p> + <p>As expected, double of 10 is 20.</p> + <p>Now let us get back to the first two lines of the code. Erlang + programs are + written in files. Each file contains an Erlang + <em>module</em>. The first line of code in the module is + the module name (see + <seealso marker="doc/reference_manual:modules">Modules</seealso>):</p> <code type="none"> -module(tut).</code> - <p>This tells us that the module is called <em>tut</em>. Note - the "." at the end of the line. The files which are used to store + <p>Thus, the module is called <em>tut</em>. Notice + the full stop "." at the end of the line. The files which are + used to store the module must have the same name as the module but with - the extension ".erl". In our case the file name is <c>tut.erl</c>. - When we use a function in another module, we use the syntax, - <c>module_name:function_name(arguments)</c>. So</p> + the extension ".erl". In this case the file name is <c>tut.erl</c>. + When using a function in another module, the syntax + <c>module_name:function_name(arguments)</c> is used. So the + following means call function <c>double</c> in module <c>tut</c> + with argument "10".</p> <pre> 4> <input>tut:double(10).</input></pre> - <p>means call function <c>double</c> in module <c>tut</c> with - argument "10".</p> - <p>The second line:</p> + <p>The second line says that the module <c>tut</c> contains a + function called <c>double</c>, which takes one argument + (<c>X</c> in our example):</p> <code type="none"> -export([double/1]).</code> - <p>says that the module <c>tut</c> contains a function called - <c>double</c> which takes one argument (<c>X</c> in our example) - and that this function can be called from outside the module - <c>tut</c>. More about this later. Again note the "." at the end - of the line.</p> - <p>Now for a more complicated example, the factorial of a number - (e.g. factorial of 4 is 4 * 3 * 2 * 1). Enter the following code - in a file called <c>tut1.erl</c>.</p> + <p>The second line also says that this function can be called from + outside the module <c>tut</c>. More about this later. Again, + notice the "." at the end of the line.</p> + <p>Now for a more complicated example, the factorial of a number. + For example, the factorial of 4 is 4 * 3 * 2 * 1, which equals 24.</p> + <p>Enter the following code in a file named <c>tut1.erl</c>:</p> <code type="none"> -module(tut1). -export([fac/1]). @@ -157,30 +162,34 @@ fac(1) -> 1; fac(N) -> N * fac(N - 1).</code> - <p>Compile the file</p> - <pre> -5> <input>c(tut1).</input> -{ok,tut1}</pre> - <p>And now calculate the factorial of 4.</p> - <pre> -6> <input>tut1:fac(4).</input> -24</pre> - <p>The first part:</p> + <p>So this is a module, called <c>tut1</c> that contains a + function called <c>fac></c>, which takes one argument, + <c>N</c>.</p> + <p>The first part says that the factorial of 1 is 1.:</p> <code type="none"> fac(1) -> 1;</code> - <p>says that the factorial of 1 is 1. Note that we end this part - with a ";" which indicates that there is more of this function to - come. The second part:</p> + <p>Notice that this part ends with a semicolon ";" that indicates + that there is more of the function <c>fac></c> to come.</p> + <p>The second part says that the factorial of N is N multiplied + by the factorial of N - 1:</p> <code type="none"> fac(N) -> N * fac(N - 1).</code> - <p>says that the factorial of N is N multiplied by the factorial of - N - 1. Note that this part ends with a "." saying that there are + <p>Notice that this part ends with a "." saying that there are no more parts of this function.</p> - <p>A function can have many arguments. Let's expand the module - <c>tut1</c> with the rather stupid function to multiply two - numbers:</p> + <p>Compile the file:</p> + <pre> +5> <input>c(tut1).</input> +{ok,tut1}</pre> + <p>And now calculate the factorial of 4.</p> + <pre> +6> <input>tut1:fac(4).</input> +24</pre> + <p>Here the function <c>fac></c> in module <c>tut1</c> is called + with argument <c>4</c>.</p> + <p>A function can have many arguments. Let us expand the module + <c>tut1</c> with the function to multiply two numbers:</p> <code type="none"> -module(tut1). -export([fac/1, mult/2]). @@ -192,36 +201,36 @@ fac(N) -> mult(X, Y) -> X * Y.</code> - <p>Note that we have also had to expand the <c>-export</c> line + <p>Notice that it is also required to expand the <c>-export</c> line with the information that there is another function <c>mult</c> with two arguments.</p> <p>Compile:</p> <pre> 7> <input>c(tut1).</input> {ok,tut1}</pre> - <p>and try it out:</p> + <p>Try out the new function <c>mult</c>:</p> <pre> 8> <input>tut1:mult(3,4).</input> 12</pre> - <p>In the example above the numbers are integers and the arguments - in the functions in the code, <c>N</c>, <c>X</c>, <c>Y</c> are + <p>In this example the numbers are integers and the arguments + in the functions in the code <c>N</c>, <c>X</c>, and <c>Y</c> are called variables. Variables must start with a capital letter - (see the chapter - <seealso marker="doc/reference_manual:expressions">"Variables"</seealso> - in the Erlang Reference Manual). Examples of variables could be - <c>Number</c>, <c>ShoeSize</c>, <c>Age</c> etc.</p> + (see + <seealso marker="doc/reference_manual:expressions">Variables</seealso>). + Examples of variables are + <c>Number</c>, <c>ShoeSize</c>, and <c>Age</c>.</p> </section> <section> <title>Atoms</title> - <p>Atoms are another data type in Erlang. Atoms start with a small - letter ((see the chapter - <seealso marker="doc/reference_manual:data_types">"Atom"</seealso> - in the Erlang Reference Manual)), for example: <c>charles</c>, - <c>centimeter</c>, <c>inch</c>. Atoms are simply names, nothing - else. They are not like variables which can have a value.</p> - <p>Enter the next program (file: <c>tut2.erl</c>) which could be - useful for converting from inches to centimeters and vice versa:</p> + <p>Atom is another data type in Erlang. Atoms start with a small + letter (see + <seealso marker="doc/reference_manual:data_types">Atom</seealso>), + for example, <c>charles</c>, + <c>centimeter</c>, and <c>inch</c>. Atoms are simply names, nothing + else. They are not like variables, which can have a value.</p> + <p>Enter the next program in a file named <c>tut2.erl</c>). It can be + useful for converting from inches to centimeters and conversely:</p> <code type="none"> -module(tut2). -export([convert/2]). @@ -231,27 +240,30 @@ convert(M, inch) -> convert(N, centimeter) -> N * 2.54.</code> - <p>Compile and test:</p> + <p>Compile:</p> <pre> 9> <input>c(tut2).</input> {ok,tut2} +</pre> + <p>Test:</p> + <pre> 10> <input>tut2:convert(3, inch).</input> 1.1811023622047243 11> <input>tut2:convert(7, centimeter).</input> 17.78</pre> - <p>Notice that I have introduced decimals (floating point numbers) - without any explanation, but I guess you can cope with that.</p> - <p>See what happens if I enter something other than centimeter or - inch in the convert function:</p> + <p>Notice the introduction of decimals (floating point numbers) + without any explanation. Hopefully you can cope with that.</p> + <p>Let us see what happens if something other than <c>centimeter</c> or + <c>inch</c> is entered in the <c>convert</c> function:</p> <pre> 12> <input>tut2:convert(3, miles).</input> ** exception error: no function clause matching tut2:convert(3,miles) (tut2.erl, line 4)</pre> <p>The two parts of the <c>convert</c> function are called its - clauses. Here we see that "miles" is not part of either of - the clauses. The Erlang system can't <em>match</em> either of - the clauses so we get an error message <c>function_clause</c>. - The shell formats the error message nicely, but the error tuple - is saved in the shell's history list and can be output by the shell + clauses. As shown, <c>miles</c> is not part of either of + the clauses. The Erlang system cannot <em>match</em> either of + the clauses so an error message <c>function_clause</c> is returned. + The shell formats the error message nicely, but the error tuple + is saved in the shell's history list and can be output by the shell command <c>v/1</c>:</p> <pre> 13> <input>v(12).</input> @@ -271,14 +283,15 @@ convert(N, centimeter) -> Consider:</p> <code type="none"> tut2:convert(3, inch).</code> - <p>Does this mean that 3 is in inches? Or that 3 is in centimeters - and we want to convert it to inches? So Erlang has a way to group - things together to make things more understandable. We call these - <em>tuples</em>. Tuples are surrounded by "{" and "}".</p> - <p>So we can write <c>{inch,3}</c> to denote 3 inches and - <c>{centimeter,5}</c> to denote 5 centimeters. Now let's write a - new program which converts centimeters to inches and vice versa. - (file <c>tut3.erl</c>).</p> + <p>Does this mean that 3 is in inches? Or does it mean that 3 is + in centimeters + and is to be converted to inches? Erlang has a way to group + things together to make things more understandable. These are called + <em>tuples</em> and are surrounded by curly brackets, "{" and "}".</p> + <p>So, <c>{inch,3}</c> denotes 3 inches and + <c>{centimeter,5}</c> denotes 5 centimeters. Now let us write a + new program that converts centimeters to inches and conversely. + Enter the following code in a file called <c>tut3.erl</c>):</p> <code type="none"> -module(tut3). -export([convert_length/1]). @@ -295,47 +308,48 @@ convert_length({inch, Y}) -> {centimeter,12.7} 16> <input>tut3:convert_length(tut3:convert_length({inch, 5})).</input> {inch,5.0}</pre> - <p>Note on line 16 we convert 5 inches to centimeters and back - again and reassuringly get back to the original value. I.e + <p>Notice on line 16 that 5 inches is converted to centimeters and back + again and reassuringly get back to the original value. That is, the argument to a function can be the result of another function. - Pause for a moment and consider how line 16 (above) works. - The argument we have given the function <c>{inch,5}</c> is first - matched against the first head clause of <c>convert_length</c> - i.e. <c>convert_length({centimeter,X})</c> where it can be seen + Consider how line 16 (above) works. + The argument given to the function <c>{inch,5}</c> is first + matched against the first head clause of <c>convert_length</c>, + that is, <c>convert_length({centimeter,X})</c>. It can be seen that <c>{centimeter,X}</c> does not match <c>{inch,5}</c> - (the head is the bit before the "->"). This having failed, we try - the head of the next clause i.e. <c>convert_length({inch,Y})</c>, - this matches and <c>Y</c> get the value 5.</p> - <p>We have shown tuples with two parts above, but tuples can have - as many parts as we want and contain any valid Erlang + (the head is the bit before the "->"). This having failed, + let us try + the head of the next clause that is, <c>convert_length({inch,Y})</c>. + This matches, and <c>Y</c> gets the value 5.</p> + <p>Tuples can have more than two parts, in fact + as many parts as you want, and contain any valid Erlang <em>term</em>. For example, to represent the temperature of - various cities of the world we could write:</p> + various cities of the world:</p> <code type="none"> {moscow, {c, -10}} {cape_town, {f, 70}} {paris, {f, 28}}</code> - <p>Tuples have a fixed number of things in them. We call each thing - in a tuple an element. So in the tuple <c>{moscow,{c,-10}}</c>, - element 1 is <c>moscow</c> and element 2 is <c>{c,-10}</c>. I - have chosen <c>c</c> meaning Centigrade (or Celsius) and <c>f</c> - meaning Fahrenheit.</p> + <p>Tuples have a fixed number of items in them. Each item in a + tuple is called an <em>element</em>. In the tuple + <c>{moscow,{c,-10}}</c>, element 1 is <c>moscow</c> and element + 2 is <c>{c,-10}</c>. Here <c>c</c> represents Celsius and + <c>f</c> Fahrenheit.</p> </section> <section> <title>Lists</title> - <p>Whereas tuples group things together, we also want to be able to - represent lists of things. Lists in Erlang are surrounded by "[" - and "]". For example, a list of the temperatures of various cities - in the world could be:</p> + <p>Whereas tuples group things together, it is also needed to + represent lists of things. Lists in Erlang are surrounded by + square brackets, "[" and "]". For example, a list of the + temperatures of various cities in the world can be:</p> <code type="none"> [{moscow, {c, -10}}, {cape_town, {f, 70}}, {stockholm, {c, -4}}, {paris, {f, 28}}, {london, {f, 36}}]</code> - <p>Note that this list was so long that it didn't fit on one line. - This doesn't matter, Erlang allows line breaks at all "sensible - places" but not, for example, in the middle of atoms, integers - etc.</p> - <p>A very useful way of looking at parts of lists, is by using "|". - This is best explained by an example using the shell.</p> + <p>Notice that this list was so long that it did not fit on one line. + This does not matter, Erlang allows line breaks at all "sensible + places" but not, for example, in the middle of atoms, integers, + and others.</p> + <p>A useful way of looking at parts of lists, is by using "|". + This is best explained by an example using the shell:</p> <pre> 17> <input>[First |TheRest] = [1,2,3,4,5].</input> [1,2,3,4,5] @@ -343,9 +357,9 @@ convert_length({inch, Y}) -> 1 19> <input>TheRest.</input> [2,3,4,5]</pre> - <p>We use | to separate the first elements of the list from - the rest of the list. (<c>First</c> has got value 1 and - <c>TheRest</c> value [2,3,4,5].)</p> + <p>To separate the first elements of the list from the rest of the + list, <c>|</c> is used. <c>First</c> has got value 1 and + <c>TheRest</c> has got the value [2,3,4,5].</p> <p>Another example:</p> <pre> 20> <input>[E1, E2 | R] = [1,2,3,4,5,6,7].</input> @@ -356,10 +370,10 @@ convert_length({inch, Y}) -> 2 23> <input>R.</input> [3,4,5,6,7]</pre> - <p>Here we see the use of | to get the first two elements from - the list. Of course if we try to get more elements from the list - than there are elements in the list we will get an error. Note - also the special case of the list with no elements [].</p> + <p>Here you see the use of <c>|</c> to get the first two elements from + the list. If you try to get more elements from the list + than there are elements in the list, an error is returned. Notice + also the special case of the list with no elements, []:</p> <pre> 24> <input>[A, B | C] = [1, 2].</input> [1,2] @@ -369,13 +383,13 @@ convert_length({inch, Y}) -> 2 27> <input>C.</input> []</pre> - <p>In all the examples above, I have been using new variable names, - not reusing the old ones: <c>First</c>, <c>TheRest</c>, <c>E1</c>, - <c>E2</c>, <c>R</c>, <c>A</c>, <c>B</c>, <c>C</c>. The reason + <p>In the previous examples, new variable names are used, instead of + reusing the old ones: <c>First</c>, <c>TheRest</c>, <c>E1</c>, + <c>E2</c>, <c>R</c>, <c>A</c>, <c>B</c>, and <c>C</c>. The reason for this is that a variable can only be given a value once in its - context (scope). I'll get back to this later, it isn't so - peculiar as it sounds!</p> - <p>The following example shows how we find the length of a list:</p> + context (scope). More about this later.</p> + <p>The following example shows how to find the length of a list. + Enter the following code in a file named <c>tut4.erl</c>):</p> <code type="none"> -module(tut4). @@ -385,7 +399,7 @@ list_length([]) -> 0; list_length([First | Rest]) -> 1 + list_length(Rest).</code> - <p>Compile (file <c>tut4.erl</c>) and test:</p> + <p>Compile and test:</p> <pre> 28> <input>c(tut4).</input> {ok,tut4} @@ -404,14 +418,14 @@ list_length([First | Rest]) -> <c>Rest</c>.</p> <p>(Advanced readers only: This is not tail recursive, there is a better way to write this function.)</p> - <p>In general we can say we use tuples where we would use "records" - or "structs" in other languages and we use lists when we want to - represent things which have varying sizes, (i.e. where we would - use linked lists in other languages).</p> - <p>Erlang does not have a string data type, instead strings can be - represented by lists of ASCII characters. So the list - <c>[97,98,99]</c> is equivalent to "abc". The Erlang shell is - "clever" and guesses the what sort of list we mean and outputs it + <p>In general, tuples are used where "records" + or "structs" are used in other languages. Also, lists are used when + representing things with varying sizes, that is, where + linked lists are used in other languages.</p> + <p>Erlang does not have a string data type. Instead, strings can be + represented by lists of Unicode characters. This implies for example that + the list <c>[97,98,99]</c> is equivalent to "abc". The Erlang shell is + "clever" and guesses what list you mean and outputs it in what it thinks is the most appropriate form, for example:</p> <pre> 30> <input>[97,98,99].</input> @@ -420,16 +434,17 @@ list_length([First | Rest]) -> <section> <title>Maps</title> - <p>Maps are a set of key to value associations. These associations - are encapsulated with "#{" and "}". To create an association from - "key" to value 42, we write:</p> + <p>Maps are a set of key to value associations. These associations + are encapsulated with "#{" and "}". To create an association + from "key" to value 42:</p> <code type="none"> > #{ "key" => 42 }. #{"key" => 42}</code> - <p>We will jump straight into the deep end with an example using some - interesting features.</p> - <p>The following example shows how we calculate alpha blending using - maps to reference color and alpha channels:</p> + <p>Let us jump straight into the deep end with an example using some + interesting features.</p> + <p>The following example shows how to calculate alpha blending + using maps to reference color and alpha channels. Enter the code + in a file named <c>color.erl</c>):</p> <code type="none"> -module(color). @@ -468,7 +483,7 @@ green(#{green := SV, alpha := SA}, #{green := DV, alpha := DA}) -> SV*SA + DV*DA*(1.0 - SA). blue(#{blue := SV, alpha := SA}, #{blue := DV, alpha := DA}) -> SV*SA + DV*DA*(1.0 - SA).</code> - <p>Compile (file <c>color.erl</c>) and test:</p> + <p>Compile and test:</p> <pre> > <input>c(color).</input> {ok,color} @@ -484,50 +499,48 @@ blue(#{blue := SV, alpha := SA}, #{blue := DV, alpha := DA}) -> <p>This example warrants some explanation:</p> <code type="none"> -define(is_channel(V), (is_float(V) andalso V >= 0.0 andalso V =< 1.0)).</code> - <p> - First we define a macro <c>is_channel</c> to help with our guard tests. - This is only here for convenience and to reduce syntax cluttering. - - You can read more about <seealso marker="doc/reference_manual:macros">Macros</seealso> - in the Erlang Reference Manual. - </p> + <p>First a macro <c>is_channel</c> is defined to help with the + guard tests. This is only here for convenience and to reduce + syntax cluttering. For more information about macros, see + <seealso marker="doc/reference_manual:macros"> + The Preprocessor</seealso>. + </p> <code type="none"> new(R,G,B,A) when ?is_channel(R), ?is_channel(G), ?is_channel(B), ?is_channel(A) -> #{red => R, green => G, blue => B, alpha => A}.</code> - <p> - The function <c>new/4</c> creates a new map term with and lets the keys - <c>red</c>, <c>green</c>, <c>blue</c> and <c>alpha</c> be associated - with an initial value. In this case we only allow for float - values between and including 0.0 and 1.0 as ensured by the <c>?is_channel/1</c> macro - for each argument. Only the <c>=></c> operator is allowed when creating a new map. + <p>The function <c>new/4</c> creates a new map term and lets the keys + <c>red</c>, <c>green</c>, <c>blue</c>, and <c>alpha</c> be + associated with an initial value. In this case, only float + values between and including 0.0 and 1.0 are allowed, as ensured + by the <c>?is_channel/1</c> macro for each argument. Only the + <c>=></c> operator is allowed when creating a new map. + </p> + <p>By calling <c>blend/2</c> on any color term created by + <c>new/4</c>, the resulting color can be calculated as + determined by the two map terms. + </p> + <p>The first thing <c>blend/2</c> does is to calculate the + resulting alpha channel: </p> - <p> - By calling <c>blend/2</c> on any color term created by <c>new/4</c> we can calculate - the resulting color as determined by the two maps terms. - </p> - <p> - The first thing <c>blend/2</c> does is to calculate the resulting alpha channel. - </p> <code type="none"> alpha(#{alpha := SA}, #{alpha := DA}) -> SA + DA*(1.0 - SA).</code> - <p> - We fetch the value associated with key <c>alpha</c> for both arguments using - the <c>:=</c> operator. Any other keys - in the map are ignored, only the key <c>alpha</c> is required and checked for. - </p> - <p>This is also the case for functions <c>red/2</c>, <c>blue/2</c> and <c>green/2</c>.</p> + <p>The value associated with key <c>alpha</c> is fetched for both + arguments using the <c>:=</c> operator. The other keys in the + map are ignored, only the key <c>alpha</c> is required and + checked for. + </p> + <p>This is also the case for functions <c>red/2</c>, + <c>blue/2</c>, and <c>green/2</c>.</p> <code type="none"> red(#{red := SV, alpha := SA}, #{red := DV, alpha := DA}) -> SV*SA + DV*DA*(1.0 - SA).</code> - <p> - The difference here is that we check for two keys in each map argument. The other keys - are ignored. - </p> - <p> - Finally we return the resulting color in <c>blend/3</c>. - </p> + <p>The difference here is that a check is made for two keys in + each map argument. The other keys are ignored. + </p> + <p>Finally, let us return the resulting color in <c>blend/3</c>: + </p> <code type="none"> blend(Src,Dst,Alpha) when Alpha > 0.0 -> Dst#{ @@ -536,20 +549,20 @@ blend(Src,Dst,Alpha) when Alpha > 0.0 -> blue := blue(Src,Dst) / Alpha, alpha := Alpha };</code> - <p> - We update the <c>Dst</c> map with new channel values. The syntax for updating an existing key with a new value is done with <c>:=</c> operator. - </p> + <p>The <c>Dst</c> map is updated with new channel values. The + syntax for updating an existing key with a new value is with the + <c>:=</c> operator. + </p> </section> <section> <title>Standard Modules and Manual Pages</title> - <p>Erlang has a lot of standard modules to help you do things. For - example, the module <c>io</c> contains a lot of functions to help - you do formatted input/output. To look up information about - standard modules, the command <c>erl -man</c> can be used at - the operating shell or command prompt (i.e. at the same place as - that where you started <c>erl</c>). Try the operating system - shell command:</p> + <p>Erlang has many standard modules to help you do things. For + example, the module <c>io</c> contains many functions that help + in doing formatted input/output. To look up information about + standard modules, the command <c>erl -man</c> can be used at the + operating shell or command prompt (the same place as you started + <c>erl</c>). Try the operating system shell command:</p> <pre> % <input>erl -man io</input> ERLANG MODULE DEFINITION io(3) @@ -561,21 +574,21 @@ DESCRIPTION This module provides an interface to standard Erlang IO servers. The output functions all return ok if they are suc- ...</pre> - <p>If this doesn't work on your system, the documentation is - included as HTML in the Erlang/OTP release, or you can read + <p>If this does not work on your system, the documentation is + included as HTML in the Erlang/OTP release. You can also read the documentation as HTML or download it as PDF from either of the sites www.erlang.se (commercial Erlang) or www.erlang.org - (open source), for example for release R9B:</p> + (open source). For example, for Erlang/OTP release R9B:</p> <code type="none"> http://www.erlang.org/doc/r9b/doc/index.html</code> </section> <section> <title>Writing Output to a Terminal</title> - <p>It's nice to be able to do formatted output in these example, so + <p>It is nice to be able to do formatted output in examples, so the next example shows a simple way to use the <c>io:format</c> - function. Of course, just like all other exported functions, you - can test the <c>io:format</c> function in the shell:</p> + function. Like all other exported functions, you can test the + <c>io:format</c> function in the shell:</p> <pre> 31> <input>io:format("hello world~n", []).</input> hello world @@ -589,28 +602,28 @@ ok 34> <input>io:format("this outputs two Erlang terms: ~w ~w~n", [hello, world]).</input> this outputs two Erlang terms: hello world ok</pre> - <p>The function <c>format/2</c> (i.e. <c>format</c> with two + <p>The function <c>format/2</c> (that is, <c>format</c> with two arguments) takes two lists. The first one is nearly always a list - written between " ". This list is printed out as it stands, + written between " ". This list is printed out as it is, except that each ~w is replaced by a term taken in order from the second list. Each ~n is replaced by a new line. The <c>io:format/2</c> function itself returns the atom <c>ok</c> if everything goes as planned. Like other functions in Erlang, it crashes if an error occurs. This is not a fault in Erlang, it is a deliberate policy. Erlang has sophisticated mechanisms to - handle errors which we will show later. As an exercise, try to - make <c>io:format</c> crash, it shouldn't be difficult. But + handle errors which are shown later. As an exercise, try to + make <c>io:format</c> crash, it should not be difficult. But notice that although <c>io:format</c> crashes, the Erlang shell itself does not crash.</p> </section> <section> <title>A Larger Example</title> - <p>Now for a larger example to consolidate what we have learnt so - far. Assume we have a list of temperature readings from a number - of cities in the world. Some of them are in Celsius (Centigrade) - and some in Fahrenheit (as in the previous list). First let's - convert them all to Celsius, then let's print out the data neatly.</p> + <p>Now for a larger example to consolidate what you have learnt so + far. Assume that you have a list of temperature readings from a number + of cities in the world. Some of them are in Celsius + and some in Fahrenheit (as in the previous list). First let us + convert them all to Celsius, then let us print the data neatly.</p> <code type="none"> %% This module is in file tut5.erl @@ -642,50 +655,50 @@ stockholm -4 c paris -2.2222222222222223 c london 2.2222222222222223 c ok</pre> - <p>Before we look at how this program works, notice that we have - added a few comments to the code. A comment starts with a % - character and goes on to the end of the line. Note as well that + <p>Before looking at how this program works, notice that + a few comments are added to the code. A comment starts with a + %-character and goes on to the end of the line. Notice also that the <c>-export([format_temps/1]).</c> line only includes - the function <c>format_temps/1</c>, the other functions are - <em>local</em> functions, i.e. they are not visible from outside + the function <c>format_temps/1</c>. The other functions are + <em>local</em> functions, that is, they are not visible from outside the module <c>tut5</c>.</p> - <p>Note as well that when testing the program from the shell, I had - to spread the input over two lines as the line was too long.</p> - <p>When we call <c>format_temps</c> the first time, <c>City</c> + <p>Notice also that when testing the program from the shell, + the input is spread over two lines as the line was too long.</p> + <p>When <c>format_temps</c> is called the first time, <c>City</c> gets the value <c>{moscow,{c,-10}}</c> and <c>Rest</c> is - the rest of the list. So we call the function - <c>print_temp(convert_to_celsius({moscow,{c,-10}}))</c>.</p> - <p>Here we see a function call as + the rest of the list. So the function + <c>print_temp(convert_to_celsius({moscow,{c,-10}}))</c> is called.</p> + <p>Here is a function call as <c>convert_to_celsius({moscow,{c,-10}})</c> as the argument to - the function <c>print_temp</c>. When we <em>nest</em> function - calls like this we execute (evaluate) them from the inside out. - I.e. we first evaluate <c>convert_to_celsius({moscow,{c,-10}})</c> + the function <c>print_temp</c>. When function calls are <em>nested</em> + like this, they execute (evaluate) from the inside out. + That is, first <c>convert_to_celsius({moscow,{c,-10}})</c> is evaluated, which gives the value <c>{moscow,{c,-10}}</c> as the temperature - is already in Celsius and then we evaluate - <c>print_temp({moscow,{c,-10}})</c>. The function - <c>convert_to_celsius</c> works in a similar way to + is already in Celsius. Then <c>print_temp({moscow,{c,-10}})</c> + is evaluated. + The function <c>convert_to_celsius</c> works in a similar way to the <c>convert_length</c> function in the previous example.</p> <p><c>print_temp</c> simply calls <c>io:format</c> in a similar way - to what has been described above. Note that ~-15w says to print + to what has been described above. Notice that ~-15w says to print the "term" with a field length (width) of 15 and left justify it. - (<seealso marker="stdlib:io#fwrite/1">io(3)</seealso>).</p> - <p>Now we call <c>format_temps(Rest)</c> with the rest of the list + (see the <seealso marker="stdlib:io#fwrite/1">io(3)</seealso>) manual page in STDLIB.</p> + <p>Now <c>format_temps(Rest)</c> is called with the rest of the list as an argument. This way of doing things is similar to the loop - constructs in other languages. (Yes, this is recursion, but don't + constructs in other languages. (Yes, this is recursion, but do not let that worry you.) So the same <c>format_temps</c> function is called again, this time <c>City</c> gets the value - <c>{cape_town,{f,70}}</c> and we repeat the same procedure as - before. We go on doing this until the list becomes empty, i.e. [], + <c>{cape_town,{f,70}}</c> and the same procedure is repeated as + before. This is done until the list becomes empty, that is [], which causes the first clause <c>format_temps([])</c> to match. This simply returns (results in) the atom <c>ok</c>, so the program ends.</p> </section> <section> - <title>Matching, Guards and Scope of Variables</title> - <p>It could be useful to find the maximum and minimum temperature + <title>Matching, Guards, and Scope of Variables</title> + <p>It can be useful to find the maximum and minimum temperature in lists like this. Before extending the program to do this, - let's look at functions for finding the maximum value of + let us look at functions for finding the maximum value of the elements in a list:</p> <code type="none"> -module(tut6). @@ -705,53 +718,57 @@ list_max([Head|Rest], Result_so_far) -> {ok,tut6} 38> <input>tut6:list_max([1,2,3,4,5,7,4,3,2,1]).</input> 7</pre> - <p>First note that we have two functions here with the same name - <c>list_max</c>. However each of these takes a different number + <p>First notice that two functions have the same name, + <c>list_max</c>. However, each of these takes a different number of arguments (parameters). In Erlang these are regarded as - completely different functions. Where we need to distinguish - between these functions we write <c>name/arity</c>, where - <c>name</c> is the name of the function and <c>arity</c> is + completely different functions. Where you need to distinguish + between these functions, you write Name/Arity, where + Name is the function name and Arity is the number of arguments, in this case <c>list_max/1</c> and <c>list_max/2</c>.</p> - <p>This is an example where we walk through a list "carrying" a - value with us, in this case <c>Result_so_far</c>. + <p>In this example you walk through a list "carrying" a + value, in this case <c>Result_so_far</c>. <c>list_max/1</c> simply assumes that the max value of the list is the head of the list and calls <c>list_max/2</c> with the rest - of the list and the value of the head of the list, in the above - this would be <c>list_max([2,3,4,5,7,4,3,2,1],1)</c>. If we tried + of the list and the value of the head of the list. In the above + this would be <c>list_max([2,3,4,5,7,4,3,2,1],1)</c>. If you tried to use <c>list_max/1</c> with an empty list or tried to use it - with something which isn't a list at all, we would cause an error. - Note that the Erlang philosophy is not to handle errors of this + with something that is not a list at all, you would cause an error. + Notice that the Erlang philosophy is not to handle errors of this type in the function they occur, but to do so elsewhere. More about this later.</p> - <p>In <c>list_max/2</c> we walk down the list and use <c>Head</c> + <p>In <c>list_max/2</c>, you walk down the list and use <c>Head</c> instead of <c>Result_so_far</c> when <c>Head</c> > - <c>Result_so_far</c>. <c>when</c> is a special word we use before - the -> in the function to say that we should only use this part - of the function if the test which follows is true. We call tests - of this type a <em>guard</em>. If the guard isn't true (we say - the guard fails), we try the next part of the function. In this - case if <c>Head</c> isn't greater than <c>Result_so_far</c> then - it must be smaller or equal to is, so we don't need a guard on - the next part of the function.</p> - <p>Some useful operators in guards are, < less than, > - greater than, == equal, >= greater or equal, =< less or - equal, /= not equal. (See the chapter - <seealso marker="doc/reference_manual:expressions">"Guard Sequences"</seealso> in the Erlang Reference Manual.)</p> - <p>To change the above program to one which works out the minimum - value of the element in a list, all we would need to do is to + <c>Result_so_far</c>. <c>when</c> is a special word used before + the -> in the function to say that you only use this part + of the function if the test that follows is true. A test + of this type is called <em>guard</em>. If the guard is false (that is, + the guard fails), the next part of the function is tried. In this + case, if <c>Head</c> is not greater than <c>Result_so_far</c>, then + it must be smaller or equal to it. This means that a guard on + the next part of the function is not needed.</p> + <p>Some useful operators in guards are: + </p><list type="bulleted"><item>< less than</item> + <item>> greater than</item> + <item>== equal</item> + <item>>= greater or equal</item> + <item>=< less or equal</item> + <item>/= not equal</item></list> + <p>(see <seealso marker="doc/reference_manual:expressions">Guard Sequences</seealso>).</p> + <p>To change the above program to one that works out the minimum + value of the element in a list, you only need to write < instead of >. (But it would be wise to change - the name of the function to <c>list_min</c> :-).)</p> - <p>Remember that I mentioned earlier that a variable could only be - given a value once in its scope? In the above we see, for example, - that <c>Result_so_far</c> has been given several values. This is - OK since every time we call <c>list_max/2</c> we create a new - scope and one can regard the <c>Result_so_far</c> as a completely + the name of the function to <c>list_min</c>.)</p> + <p>Earlier it was mentioned that a variable can only be + given a value once in its scope. In the above you see + that <c>Result_so_far</c> is given several values. This is + OK since every time you call <c>list_max/2</c> you create a new + scope and one can regard <c>Result_so_far</c> as a different variable in each scope.</p> <p>Another way of creating and giving a variable a value is by using - the match operator = . So if I write <c>M = 5</c>, a variable - called <c>M</c> will be created and given the value 5. If, in - the same scope I then write <c>M = 6</c>, I'll get an error. Try + the match operator = . So if you write <c>M = 5</c>, a variable + called <c>M</c> is created with the value 5. If, in + the same scope, you then write <c>M = 6</c>, an error is returned. Try this out in the shell:</p> <pre> 39> <input>M = 5.</input> @@ -771,21 +788,21 @@ list_max([Head|Rest], Result_so_far) -> paris 45> <input>Y.</input> {f,28}</pre> - <p>Here we see that <c>X</c> gets the value <c>paris</c> and + <p>Here <c>X</c> gets the value <c>paris</c> and <c>Y</c><c>{f,28}</c>.</p> - <p>Of course if we try to do the same again with another city, we - get an error:</p> + <p>If you try to do the same again with another city, + an error is returned:</p> <pre> 46> <input>{X, Y} = {london, {f, 36}}.</input> ** exception error: no match of right hand side value {london,{f,36}}</pre> <p>Variables can also be used to improve the readability of - programs, for example, in the <c>list_max/2</c> function above, - we could write:</p> + programs. For example, in function <c>list_max/2</c> above, + you can write:</p> <code type="none"> list_max([Head|Rest], Result_so_far) when Head > Result_so_far -> New_result_far = Head, list_max(Rest, New_result_far);</code> - <p>which is possibly a little clearer.</p> + <p>This is possibly a little clearer.</p> </section> <section> @@ -824,9 +841,9 @@ reverse([], Reversed_List) -> {ok,tut8} 53> <input>tut8:reverse([1,2,3]).</input> [3,2,1]</pre> - <p>Consider how <c>Reversed_List</c> is built. It starts as [], we - then successively take off the heads of the list to be reversed - and add them to the the <c>Reversed_List</c>, as shown in + <p>Consider how <c>Reversed_List</c> is built. It starts as [], + then successively the heads are taken off of the list to be reversed + and added to the the <c>Reversed_List</c>, as shown in the following:</p> <code type="none"> reverse([1|2,3], []) => @@ -840,14 +857,15 @@ reverse([3|[]], [2,1]) => reverse([], [3,2,1]) => [3,2,1]</code> - <p>The module <c>lists</c> contains a lot of functions for - manipulating lists, for example for reversing them, so before you - write a list manipulating function it is a good idea to check - that one isn't already written for you. (see - <seealso marker="stdlib:lists">lists(3)</seealso>).</p> - <p>Now let's get back to the cities and temperatures, but take a more - structured approach this time. First let's convert the whole list - to Celsius as follows and test the function:</p> + <p>The module <c>lists</c> contains many functions for + manipulating lists, for example, for reversing them. So before + writing a list-manipulating function it is a good idea to check + if one not already is written for you + (see the <seealso marker="stdlib:lists">lists(3)</seealso> + manual page in STDLIB).</p> + <p>Now let us get back to the cities and temperatures, but take a more + structured approach this time. First let us convert the whole list + to Celsius as follows:</p> <code type="none"> -module(tut7). -export([format_temps/1]). @@ -864,6 +882,7 @@ convert_list_to_c([City | Rest]) -> convert_list_to_c([]) -> [].</code> + <p>Test the function:</p> <pre> 54> <input>c(tut7).</input> {ok, tut7}. @@ -874,26 +893,26 @@ convert_list_to_c([]) -> {stockholm,{c,-4}}, {paris,{c,-2.2222222222222223}}, {london,{c,2.2222222222222223}}]</pre> - <p>Looking at this bit by bit:</p> + <p>Explanation:</p> <code type="none"> format_temps(List_of_cities) -> convert_list_to_c(List_of_cities).</code> - <p>Here we see that <c>format_temps/1</c> calls + <p>Here <c>format_temps/1</c> calls <c>convert_list_to_c/1</c>. <c>convert_list_to_c/1</c> takes off the head of the <c>List_of_cities</c>, converts it to Celsius if needed. The | operator is used to add the (maybe) converted to the converted rest of the list:</p> <code type="none"> [Converted_City | convert_list_to_c(Rest)];</code> - <p>or</p> + <p>or:</p> <code type="none"> [City | convert_list_to_c(Rest)];</code> - <p>We go on doing this until we get to the end of the list (i.e. - the list is empty):</p> + <p>This is done until the end of the list is reached, that is, + the list is empty:</p> <code type="none"> convert_list_to_c([]) -> [].</code> - <p>Now we have converted the list, we add a function to print it:</p> + <p>Now when the list is converted, a function to print it is added:</p> <code type="none"> -module(tut7). -export([format_temps/1]). @@ -928,12 +947,12 @@ stockholm -4 c paris -2.2222222222222223 c london 2.2222222222222223 c ok</pre> - <p>We now have to add a function to find the cities with - the maximum and minimum temperatures. The program below isn't - the most efficient way of doing this as we walk through the list + <p>Now a function has to be added to find the cities with + the maximum and minimum temperatures. The following program is not + the most efficient way of doing this as you walk through the list of cities four times. But it is better to first strive for clarity and correctness and to make programs efficient only if - really needed.</p> + needed.</p> <code type="none"><![CDATA[ -module(tut7). -export([format_temps/1]). @@ -1003,8 +1022,8 @@ ok</pre> <section> <title>If and Case</title> <p>The function <c>find_max_and_min</c> works out the maximum and - minimum temperature. We have introduced a new construct here - <c>if</c>. If works as follows:</p> + minimum temperature. A new construct, <c>if</c>, is introduced here. + If works as follows:</p> <code type="none"> if Condition 1 -> @@ -1016,14 +1035,15 @@ if Condition 4 -> Action 4 end</code> - <p>Note there is no ";" before <c>end</c>! Conditions are the same - as guards, tests which succeed or fail. Erlang starts at the top - until it finds a condition which succeeds and then it evaluates + <p>Notice that there is no ";" before <c>end</c>. Conditions do + the same as guards, that is, tests that succeed or fail. Erlang + starts at the top + and tests until it finds a condition that succeeds. Then it evaluates (performs) the action following the condition and ignores all - other conditions and action before the <c>end</c>. If no - condition matches, there will be a run-time failure. A condition - which always is succeeds is the atom, <c>true</c> and this is - often used last in an <c>if</c> meaning do the action following + other conditions and actions before the <c>end</c>. If no + condition matches, a run-time failure occurs. A condition + that always succeeds is the atom <c>true</c>. This is + often used last in an <c>if</c>, meaning, do the action following the <c>true</c> if all other conditions have failed.</p> <p>The following is a short program to show the workings of <c>if</c>.</p> @@ -1039,10 +1059,10 @@ test_if(A, B) -> B == 6 -> io:format("B == 6~n", []), b_equals_6; - A == 2, B == 3 -> %i.e. A equals 2 and B equals 3 + A == 2, B == 3 -> %That is A equals 2 and B equals 3 io:format("A == 2, B == 3~n", []), a_equals_2_b_equals_3; - A == 1 ; B == 7 -> %i.e. A equals 1 or B equals 7 + A == 1 ; B == 7 -> %That is A equals 1 or B equals 7 io:format("A == 1 ; B == 7~n", []), a_equals_1_or_b_equals_7 end.</code> @@ -1068,19 +1088,19 @@ a_equals_1_or_b_equals_7 66> <input>tut9:test_if(33, 33).</input> ** exception error: no true branch found when evaluating an if expression in function tut9:test_if/2 (tut9.erl, line 5)</pre> - <p>Notice that <c>tut9:test_if(33,33)</c> did not cause any - condition to succeed so we got the run time error - <c>if_clause</c>, here nicely formatted by the shell. See the chapter - <seealso marker="doc/reference_manual:expressions">"Guard Sequences"</seealso> in the Erlang Reference Manual for details - of the many guard tests available. <c>case</c> is another - construct in Erlang. Recall that we wrote the - <c>convert_length</c> function as:</p> + <p>Notice that <c>tut9:test_if(33,33)</c> does not cause any + condition to succeed. This leads to the run time error + <c>if_clause</c>, here nicely formatted by the shell. See + <seealso marker="doc/reference_manual:expressions">Guard Sequences</seealso> + for details of the many guard tests available.</p> + <p><c>case</c> is another construct in Erlang. Recall that the + <c>convert_length</c> function was written as:</p> <code type="none"> convert_length({centimeter, X}) -> {inch, X / 2.54}; convert_length({inch, Y}) -> {centimeter, Y * 2.54}.</code> - <p>We could also write the same program as:</p> + <p>The same program can also be written as:</p> <code type="none"> -module(tut10). -export([convert_length/1]). @@ -1099,12 +1119,13 @@ convert_length(Length) -> {centimeter,15.24} 69> <input>tut10:convert_length({centimeter, 2.5}).</input> {inch,0.984251968503937}</pre> - <p>Notice that both <c>case</c> and <c>if</c> have <em>return values</em>, i.e. in the above example <c>case</c> returned + <p>Both <c>case</c> and <c>if</c> have <em>return values</em>, that is, + in the above example <c>case</c> returned either <c>{inch,X/2.54}</c> or <c>{centimeter,Y*2.54}</c>. The behaviour of <c>case</c> can also be modified by using guards. - An example should hopefully clarify this. The following example - tells us the length of a month, given the year. We need to know - the year of course, since February has 29 days in a leap year.</p> + The following example clarifies this. It + tells us the length of a month, given the year. + The year must be known, since February has 29 days in a leap year.</p> <code type="none"> -module(tut11). -export([month_length/2]). @@ -1150,57 +1171,58 @@ month_length(Year, Month) -> </section> <section> - <title>Built In Functions (BIFs)</title> - <p>Built in functions (BIFs) are functions which for some reason are - built in to the Erlang virtual machine. BIFs often implement - functionality that is impossible to implement in Erlang or is too - inefficient to implement in Erlang. Some BIFs can be called - by use of the function name only, but they by default belong - to the erlang module. So for example, the call to the BIF <c>trunc</c> + <title>Built-In Functions (BIFs)</title> + <p>BIFs are functions that for some reason are + built-in to the Erlang virtual machine. BIFs often implement + functionality that is impossible or is too + inefficient to implement in Erlang. Some BIFs can be called + using the function name only but they are by default belonging + to the <c>erlang</c> module. For example, the call to the + BIF <c>trunc</c> below is equivalent to a call to <c>erlang:trunc</c>.</p> - <p>As you can see, we first find out if a year is leap or not. If a - year is divisible by 400, it is a leap year. To find this out we - first divide the year by 400 and use the built in function - <c>trunc</c> (more later) to cut off any decimals. We then - multiply by 400 again and see if we get back the same value. For - example, year 2004:</p> + <p>As shown, first it is checked if a year is leap. If a + year is divisible by 400, it is a leap year. To determine this, + first divide the year by 400 and use the BIF + <c>trunc</c> (more about this later) to cut off any decimals. Then + multiply by 400 again and see if the same value is returned again. + For example, year 2004:</p> <code type="none"> 2004 / 400 = 5.01 trunc(5.01) = 5 5 * 400 = 2000</code> - <p>and we can see that we got back 2000 which is not the same as - 2004, so 2004 isn't divisible by 400. Year 2000:</p> + <p>2000 is not the same as 2004, so 2004 is not divisible by 400. + Year 2000:</p> <code type="none"> 2000 / 400 = 5.0 trunc(5.0) = 5 5 * 400 = 2000</code> - <p>so we have a leap year. The next two tests, which check if the year is - divisible by 100 or 4, are done in the same way. The first - <c>if</c> returns <c>leap</c> or <c>not_leap</c> which ends up - in the variable <c>Leap</c>. We use this variable in the guard - for <c>feb</c> in the following <c>case</c> which tells us how + <p>That is, a leap year. The next two <c>trunc</c>-tests evaluate + if the year is divisible by 100 or 4 in the same way. The first + <c>if</c> returns <c>leap</c> or <c>not_leap</c>, which lands up + in the variable <c>Leap</c>. This variable is used in the guard + for <c>feb</c> in the following <c>case</c> that tells us how long the month is.</p> - <p>This example showed the use of <c>trunc</c>. An easier way would - be to use the Erlang operator <c>rem</c>, which gives the remainder - after division. For example:</p> + <p>This example showed the use of <c>trunc</c>. It is easier + to use the Erlang operator <c>rem</c> that gives the remainder + after division, for example:</p> <pre> 74> <input>2004 rem 400.</input> 4</pre> - <p>so instead of writing:</p> + <p>So instead of writing:</p> <code type="none"> trunc(Year / 400) * 400 == Year -> leap;</code> - <p>we could write:</p> + <p>it can be written:</p> <code type="none"> Year rem 400 == 0 -> leap;</code> - <p>There are many other built in functions (BIF) such as - <c>trunc</c>. Only a few built in functions can be used in guards, + <p>There are many other BIFs such as + <c>trunc</c>. Only a few BIFs can be used in guards, and you cannot use functions you have defined yourself in guards. - (see the chapter - <seealso marker="doc/reference_manual:expressions">"Guard Sequences"</seealso> in the Erlang Reference Manual) (Aside for - advanced readers: This is to ensure that guards don't have side - effects.) Let's play with a few of these functions in the shell:</p> + (see + <seealso marker="doc/reference_manual:expressions">Guard Sequences</seealso>) + (For advanced readers: This is to ensure that guards do not have side + effects.) Let us play with a few of these functions in the shell:</p> <pre> 75> <input>trunc(5.6).</input> 5 @@ -1218,7 +1240,7 @@ false true 82> <input>is_tuple([paris, {c, 30}]).</input> false</pre> - <p>All the above can be used in guards. Now for some which can't be + <p>All of these can be used in guards. Now for some BIFs that cannot be used in guards:</p> <pre> 83> <input>atom_to_list(hello).</input> @@ -1227,22 +1249,22 @@ false</pre> goodbye 85> <input>integer_to_list(22).</input> "22"</pre> - <p>The 3 BIFs above do conversions which would be difficult (or + <p>These three BIFs do conversions that would be difficult (or impossible) to do in Erlang.</p> </section> <section> - <title>Higher Order Functions (Funs)</title> + <title>Higher-Order Functions (Funs)</title> <p>Erlang, like most modern functional programming languages, has - higher order functions. We start with an example using the shell:</p> + higher-order functions. Here is an example using the shell:</p> <pre> 86> <input>Xf = fun(X) -> X * 2 end.</input> #Fun<erl_eval.5.123085357> 87> <input>Xf(5).</input> 10</pre> - <p>What we have done here is to define a function which doubles - the value of number and assign this function to a variable. Thus - <c>Xf(5)</c> returned the value 10. Two useful functions when + <p>Here is defined a function that doubles + the value of a number and assigned this function to a variable. Thus + <c>Xf(5)</c> returns value 10. Two useful functions when working with lists are <c>foreach</c> and <c>map</c>, which are defined as follows:</p> <code type="none"> @@ -1258,17 +1280,16 @@ map(Fun, []) -> [].</code> <p>These two functions are provided in the standard module <c>lists</c>. <c>foreach</c> takes a list and applies a fun to - every element in the list, <c>map</c> creates a new list by + every element in the list. <c>map</c> creates a new list by applying a fun to every element in a list. Going back to - the shell, we start by using <c>map</c> and a fun to add 3 to + the shell, <c>map</c> is used and a fun to add 3 to every element of a list:</p> <pre> 88> <input>Add_3 = fun(X) -> X + 3 end.</input> #Fun<erl_eval.5.123085357> 89> <input>lists:map(Add_3, [1,2,3]).</input> [4,5,6]</pre> - <p>Now let's print out the temperatures in a list of cities (yet - again):</p> + <p>Let us (again) print the temperatures in a list of cities:</p> <pre> 90> <input>Print_City = fun({City, {X, Temp}}) -> io:format("~-15w ~w ~w~n",</input> <input>[City, X, Temp]) end.</input> @@ -1281,7 +1302,7 @@ stockholm c -4 paris f 28 london f 36 ok</pre> - <p>We will now define a fun which can be used to go through a list + <p>Let us now define a fun that can be used to go through a list of cities and temperatures and transform them all to Celsius.</p> <code type="none"> -module(tut13). @@ -1303,21 +1324,21 @@ convert_list_to_c(List) -> {stockholm,{c,-4}}, {paris,{c,-2}}, {london,{c,2}}]</pre> - <p>The <c>convert_to_c</c> function is the same as before, but we - use it as a fun:</p> + <p>The <c>convert_to_c</c> function is the same as before, but here + it is used as a fun:</p> <code type="none"> lists:map(fun convert_to_c/1, List)</code> - <p>When we use a function defined elsewhere as a fun we can refer - to it as <c>Function/Arity</c> (remember that <c>Arity</c> = - number of arguments). So in the <c>map</c> call we write - <c>lists:map(fun convert_to_c/1, List)</c>. As you can see + <p>When a function defined elsewhere is used as a fun, it can be referred + to as <c>Function/Arity</c> (remember that <c>Arity</c> = + number of arguments). So in the <c>map</c>-call + <c>lists:map(fun convert_to_c/1, List)</c> is written. As shown, <c>convert_list_to_c</c> becomes much shorter and easier to understand.</p> <p>The standard module <c>lists</c> also contains a function <c>sort(Fun, List)</c> where <c>Fun</c> is a fun with two - arguments. This fun should return <c>true</c> if the the first + arguments. This fun returns <c>true</c> if the first argument is less than the second argument, or else <c>false</c>. - We add sorting to the <c>convert_list_to_c</c>:</p> + Sorting is added to the <c>convert_list_to_c</c>:</p> <code type="none"><![CDATA[ -module(tut13). @@ -1342,13 +1363,13 @@ convert_list_to_c(List) -> {paris,{c,-2}}, {london,{c,2}}, {cape_town,{c,21}}]</pre> - <p>In <c>sort</c> we use the fun:</p> + <p>In <c>sort</c> the fun is used:</p> <code type="none"><![CDATA[ fun({_, {c, Temp1}}, {_, {c, Temp2}}) -> Temp1 < Temp2 end,]]></code> - <p>Here we introduce the concept of an <em>anonymous variable</em> - "_". This is simply shorthand for a variable which is going to - get a value, but we will ignore the value. This can be used - anywhere suitable, not just in fun's. <c><![CDATA[Temp1 < Temp2]]></c> + <p>Here the concept of an <em>anonymous variable</em> + "_" is introduced. This is simply shorthand for a variable that + gets a value, but the value is ignored. This can be used + anywhere suitable, not just in funs. <c><![CDATA[Temp1 < Temp2]]></c> returns <c>true</c> if <c>Temp1</c> is less than <c>Temp2</c>.</p> </section> </chapter> diff --git a/system/doc/installation_guide/Makefile b/system/doc/installation_guide/Makefile index 83210bd21f..a4ef6c9d7c 100644 --- a/system/doc/installation_guide/Makefile +++ b/system/doc/installation_guide/Makefile @@ -58,7 +58,8 @@ XML_FILES = \ GENERATED_XML_FILES = \ INSTALL.xml \ INSTALL-CROSS.xml \ - INSTALL-WIN32.xml + INSTALL-WIN32.xml \ + OTP-PATCH-APPLY.xml # ---------------------------------------------------- @@ -73,7 +74,8 @@ REDIRECT_HTML_DIR = $(HTMLDIR)/source REDIRECT_HTML_FILES = \ $(REDIRECT_HTML_DIR)/INSTALL.html \ $(REDIRECT_HTML_DIR)/INSTALL-CROSS.html \ - $(REDIRECT_HTML_DIR)/INSTALL-WIN32.html + $(REDIRECT_HTML_DIR)/INSTALL-WIN32.html \ + $(REDIRECT_HTML_DIR)/OTP-PATCH-APPLY.html # ---------------------------------------------------- # FLAGS diff --git a/system/doc/installation_guide/install-binary.xml b/system/doc/installation_guide/install-binary.xml index af7dab6e44..ead4b19323 100644 --- a/system/doc/installation_guide/install-binary.xml +++ b/system/doc/installation_guide/install-binary.xml @@ -33,25 +33,23 @@ </header> <section> <title>Windows</title> + <p>The system is delivered as a Windows Installer executable. + Get it from http://www.erlang.org/download.html</p> <section> - <title>Introduction</title> - <p>The system is delivered as a Windows Installer executable. - Get it from our <url href="http://www.erlang.org/download.html">download page</url>.</p> - </section> - - <section> - <title>Installation</title> - <p>The installation procedure is is automated. Double-click the + <title>Installing</title> + <p>The installation procedure is automated. Double-click the <c>.exe</c> file icon and follow the instructions.</p> </section> + <section> - <title>Verification</title> + <title>Verifying</title> <list type="bulleted"> <item> <p>Start Erlang/OTP by double-clicking on the Erlang shortcut icon on the desktop.</p> - <p>Expect a command line window to pop up with an output looking something like this:</p> + <p>Expect a command-line window to pop up with an output looking + something like this:</p> <pre> Erlang/OTP 17 [erts-6.0] [64-bit] [smp:2:2] @@ -59,12 +57,12 @@ 1></pre> </item> <item> - <p>Exit by entering the command <c>halt()</c>,</p> + <p>Exit by entering the command <c>halt()</c>.</p> <pre> 2> <input>halt().</input></pre> - <p>which will close the Erlang/OTP shell. </p> + <p>This closes the Erlang/OTP shell.</p> </item> </list> - </section> + </section> </section> </chapter> diff --git a/system/doc/installation_guide/part.xml b/system/doc/installation_guide/part.xml index 02bf98db7c..5b1b3833cd 100644 --- a/system/doc/installation_guide/part.xml +++ b/system/doc/installation_guide/part.xml @@ -29,10 +29,12 @@ <file>part.xml</file> </header> <description> - <p>How to install Erlang/OTP on UNIX or Windows.</p> + <marker id="installation guide"></marker> + <p>This section describes how to install Erlang/OTP on UNIX and Windows.</p> </description> <xi:include href="install-binary.xml"/> <xi:include href="INSTALL.xml"/> <xi:include href="INSTALL-CROSS.xml"/> <xi:include href="INSTALL-WIN32.xml"/> -</part>
\ No newline at end of file + <xi:include href="OTP-PATCH-APPLY.xml"/> +</part> diff --git a/system/doc/installation_guide/xmlfiles.mk b/system/doc/installation_guide/xmlfiles.mk index c443334cd7..a18c82bc25 100644 --- a/system/doc/installation_guide/xmlfiles.mk +++ b/system/doc/installation_guide/xmlfiles.mk @@ -20,4 +20,5 @@ INST_GUIDE_CHAPTER_FILES = \ install-binary.xml \ INSTALL.xml \ INSTALL-CROSS.xml \ - INSTALL-WIN32.xml + INSTALL-WIN32.xml \ + OTP-PATCH-APPLY.xml diff --git a/system/doc/oam/oam_intro.xml b/system/doc/oam/oam_intro.xml index f4f990393e..de4867ca16 100644 --- a/system/doc/oam/oam_intro.xml +++ b/system/doc/oam/oam_intro.xml @@ -28,241 +28,224 @@ <rev>A</rev> <file>oam_intro.xml</file> </header> - <p>The operation and maintenance support in OTP consists of a - generic model for management subsystems in OTP, and some - components to be used in these subsystems. This document - describes the model. - </p> - <p>The main idea in the model is that it is management protocol - independent. Thus, it is not tied to any specific management - protocol. An API is defined which can be used to write - adaptations for specific management protocols. - </p> - <p>Each OAM component in OTP is implemented as one sub application, - which can be included in a management application for the system. - Note that such a complete management application is not in the - scope of this generic functionality. Examples illustrating how such an - application can be built are included however. - </p> + <marker id="oam principles"></marker> + <p>The Operation and Maintenance (OAM) support in OTP consists of a + generic model for management subsystems in OTP, and some components + to be used in these subsystems. This section describes the model.</p> + + <p>The main idea in the model is that it is not tied to any specific + management protocol. An Application Programming Interface (API) is + defined, which can be used to write adaptations for specific + management protocols.</p> + + <p>Each OAM component in OTP is implemented as one sub-application, which + can be included in a management application for the system. Notice that + such a complete management application is not in the scope of this + generic functionality. However, this section includes examples + illustrating how such an application can be built.</p> <section> <title>Terminology</title> - <p>The protocol independent architectural model on the network - level is the well-known <term id="Manager-Agent model"><termdef>Client-Server model for management operations</termdef></term>. This model is based on the client-server - principle, where the manager (client) sends <term id="requests"><termdef>A request is sent from a manager to an agent when it accesses management information.</termdef></term>to the - agent (server), the agent sends <term id="replies"><termdef>A reply is sent from the agent as a response to a request from a manager.</termdef></term>back to the manager. There are two main - differences to the normal client-server model. First, there are - usually a few managers that communicate with many agents; and - second, the agent may spontaneously send <term id="notifications"><termdef>A notification is sent spontaneously from an agent to a manager, e.g. an alarm.</termdef></term>to the - manager. The picture below illustrates the idea.</p> + <p>The protocol-independent architectural model on the network level + is the well-known client-server model for management operations. This + model is based on the client-server principle, where the manager + (client) sends a request from a manager to an agent (server) when it + accesses management information. The agent sends a reply back to the + manager. There are two main differences to the normal + client-server model:</p> + <list type="bulleted"> + <item><p>Usually a few managers communicate with many agents.</p></item> + <item><p>The agent can spontaneously send a notification, for example, + an alarm, to the manager.</p></item> + </list> + <p>The following picture illustrates the idea:</p> + <image file="../oam/terminology.gif"> <icaption>Terminology</icaption> </image> - <p>The manager is often referred to as the <term id="NMS"></term>, to - emphasize that it usually is realized as a program that presents - data to an operator. - </p> - <p>The agent is an entity that executes within a <term id="NE"></term>. - In OTP, the network element may be a distributed system, meaning - that the distributed system is managed as one entity. Of - course, the agent may be configured to be able to run on one of - several nodes, making it a distributed OTP application. - </p> - <p>The management information is defined in an <term id="MIB"></term>. - It is a formal definition of which information the agent makes - available to the manager. The manager accesses the MIB through - a management protocol, such as SNMP, CMIP, HTTP or CORBA. Each - of these protocols have their own MIB definition language. In - SNMP, it is a subset of ASN.1, in CMIP it is GDMO, in HTTP it is - implicit, and using CORBA, it is IDL. Usually, the entities - defined in the MIB are called <term id="MO"></term>, although these - objects do not have to be objects in the OO way,for example, a simple - scalar variable defined in an MIB is called a Managed Object. - The Managed Objects are logical objects, not necessarily with a - one-to-one mapping to the resources. - </p> + + <p>The manager is often referred to as the <em>Network Management + System (NMS)</em>, to emphasize that it usually is realized as a + program that presents data to an operator.</p> + + <p>The agent is an entity that executes within a <em>Network + Element (NE)</em>. In OTP, the NE can be a distributed system, + meaning that the distributed system is managed as one entity. + Of course, the agent can be configured to be able to run on one + of several nodes, making it a distributed OTP application.</p> + + <p>The management information is defined in a <em>Management + Information Base (MIB)</em>. It is a formal definition of which + information the agent makes available to the manager. The + manager accesses the MIB through a management protocol, such + as SNMP, CMIP, HTTP, or CORBA. Each protocol has its own MIB + definition language. In SNMP, it is a subset of ASN.1, in CMIP + it is GDMO, in HTTP it is implicit, and using CORBA, it is IDL.</p> + + <p>Usually, the entities defined in the MIB are + called <em>Managed Objects (MOs)</em>, although they do not + have to be objects in the object-oriented way. For example, + a simple scalar variable defined in a MIB is called an MO. The + MOs are logical objects, not necessarily with a one-to-one + mapping to the resources.</p> </section> <section> <title>Model</title> - <p>In this section, the generic protocol independent model for use - within an OTP based network element is presented. This model is - used by all operation and maintenance components, and may be - used by the applications. The advantage of the model is that it - clearly separates the resources from the management protocol. - The resources do not need to be aware of which management - protocol is used to manage the system. This makes it possible - to manage the same resources with different protocols. - </p> - <p>The different entities involved in this model are the <term id="agent"></term>which terminates the management protocol, and the - <term id="resources"></term>which is to be managed, i.e. the actual - application entities. The resources should in general have no - knowledge of the management protocol used, and the agent should - have no knowledge of the managed resources. This implies that - some sort of translation mechanism must be used, to translate - the management operations to operations on the resources. This - translation mechanism is usually called - <em>instrumentation</em>, and the function that implements it is - called <term id="instrumentation function"></term>. The - instrumentation functions are written for each combination of - management protocol and resource to be managed. For example, if - an application is to be managed by SNMP and HTTP, two sets of - instrumentation functions are defined; one that maps SNMP - requests to the resources, and one that e.g. generates an HTML - page for some resources. - </p> - <p>When a manager makes a request to the agent, we have the - following picture:</p> + <p>This section presents the generic protocol-independent model + for use within an OTP-based NE. This model is used by + all OAM components and can be used by the applications. The + advantage of the model is that it clearly separates the + resources from the management protocol. The resources do not + need to be aware of which management protocol is used to manage + the system. The same resources can therefore be managed with + different protocols.</p> + + <p>The entities involved in this model are the agent, which + terminates the management protocol, and the resources, which + is to be managed, that is, the actual application entities. + The resources should in general have no knowledge of the + management protocol used, and the agent should have no + knowledge of the managed resources. This implies that a + translation mechanism is needed, to translate the management + operations to operations on the resources. This translation + mechanism is usually called <em>instrumentation</em> and the + function that implements it is called <em>instrumentation + function</em>. The instrumentation functions are written for + each combination of management protocol and resource to be + managed. For example, if an application is to be managed by + SNMP and HTTP, two sets of instrumentation functions are + defined; one that maps SNMP requests to the resources, and + one that, for example, generates an HTML page for some + resources.</p> + + <p>When a manager makes a request to the agent, the following + illustrates the situation:</p> + <image file="../oam/snmp_model_1.gif"> - <icaption>Request to an agent by a manager</icaption> + <icaption>Request to An Agent by a Manager</icaption> </image> - <p>Note that the mapping between instrumentation function and - resource is not necessarily 1-1. It is also possible to write - one instrumentation function for each resource, and use that - function from different protocols. - </p> - <p>The agent receives a request and maps this request to calls to - one or several instrumentation functions. The instrumentation - functions perform operations on the resources to implement the - semantics associated with the managed object. - </p> - <p>For example, a system that is managed with SNMP and HTTP may be - structured in the following way:</p> + + <p>The mapping between an instrumentation function and a + resource is not necessarily 1-1. It is also possible to write + one instrumentation function for each resource, and use that + function from different protocols.</p> + + <p>The agent receives a request and maps it to calls to one or + more instrumentation functions. These functions perform + operations on the resources to implement the semantics + associated with the MO.</p> + + <p>For example, a system that is managed with SNMP and HTTP + can be structured as follows:</p> + <image file="../oam/snmp_model_2.gif"> - <icaption>Structure of a system managed with SNMP and HTTP</icaption> + <icaption>Structure of a System Managed with SNMP and HTTP</icaption> </image> - <p>The resources may send notifications to the manager as well. - Examples of notifications are events and alarms. There is a - need for the resource to generate protocol independent - notifications. The following picture illustrates how this is - achieved:</p> + + <p>The resources can send notifications to the manager as well. + Examples of notifications are events and alarms. The resource + needs to generate protocol-independent notifications. + The following picture illustrates how this is achieved:</p> + <image file="../oam/snmp_model_3.gif"> - <icaption>Notification handling</icaption> + <icaption>Notification Handling</icaption> </image> - <p>The main idea is that the resource sends the notfications as - Erlang terms to a dedicated <c>gen_event</c> process. Into this - process, handlers for the different management protocols are - installed. When an event is received by this process, it is - forwarded to each installed handler. The handlers are - responsible for translating the event into a notification to be - sent over the management protocol. For example, a handler for - SNMP would translate each event into an SNMP trap. - </p> + + <p>The main idea is that the resource sends the notifications as + Erlang terms to a dedicated <c>gen_event</c> process. Into this + process, handlers for the different management protocols are + installed. When an event is received by this process, it is + forwarded to each installed handler. The handlers are + responsible for translating the event into a notification to be + sent over the management protocol. For example, a handler for + SNMP translates each event into an SNMP trap.</p> </section> <section> - <title>SNMP based OAM</title> - <p>For all OAM components, SNMP adaptations are provided. Other - adaptations may be defined in the future. - </p> + <title>SNMP-Based OAM</title> + <p>For all OAM components, SNMP adaptations are provided. Other + adaptations might be defined in the future.</p> + <p>The OAM components, and some other OTP applications, define - SNMP MIBs. All these MIBs are written in SNMPv2 SMI syntax, as - defined in RFC1902. For convenience we also deliver the SNMPv1 - SMI equivalent. All MIBs are designed to be v1/v2 compatible, - i.e. the v2 MIBs do not use any construct not available in v1. - </p> + SNMP MIBs. These MIBs are written in SNMPv2 SMI syntax, as + defined in RFC 1902. For convenience we also deliver the SNMPv1 + SMI equivalent. All MIBs are designed to be v1/v2 compatible, + that is, the v2 MIBs do not use any construct not available in + v1.</p> <section> - <title>MIB structure</title> - <p>The top-level OTP MIB is called <c>OTP-REG</c>, and it is - included in the <c>sasl</c> application. All other OTP mibs - import some objects from this MIB. - </p> - <p>Each MIB is contained in one application. The MIB text files - are stored under <c><![CDATA[mibs/<MIB>.mib]]></c> in the application - directory. The generated <c>.hrl</c> files with constant - declarations are stored under <c><![CDATA[include/<MIB>.hrl]]></c>, and - the compiled MIBs are stored under - <c><![CDATA[priv/mibs/<MIB>.bin]]></c>. For example, the <c>OTP-MIB</c> - is included in the <c>sasl</c> application: - </p> + <title>MIB Structure</title> + <p>The top-level OTP MIB is called <c>OTP-REG</c> and it is + included in the <c>sasl</c> application. All other OTP MIBs + import some objects from this MIB.</p> + + <p>Each MIB is contained in one application. The MIB text + files are stored under <c><![CDATA[mibs/<MIB>.mib]]></c> in + the application directory. The generated <c>.hrl</c> files + with constant declarations are stored under + <c><![CDATA[include/<MIB>.hrl]]></c>, and the compiled MIBs + are stored under <c><![CDATA[priv/mibs/<MIB>.bin]]></c>. + For example, the <c>OTP-MIB</c> is included in the + <c>sasl</c> application:</p> + <code type="none"> sasl-1.3/mibs/OTP-MIB.mib - include/OTP-MIB.hrl - priv/mibs/OTP-MIB.bin</code> - <p>An application that needs to IMPORT this mib into another - MIB, should use the <c>il</c> option to the snmp mib compiler: - </p> +include/OTP-MIB.hrl +priv/mibs/OTP-MIB.bin</code> + + <p>An application that needs to import this MIB into another + MIB is to use the <c>il</c> option to the SNMP MIB compiler:</p> + <code type="none"> snmp:c("MY-MIB", [{il, ["sasl/priv/mibs"]}]).</code> + <p>If the application needs to include the generated - <c>.hrl</c> file, it should use the <c>-include_lib</c> - directive to the Erlang compiler. - </p> + <c>.hrl</c> file, it is to use the <c>-include_lib</c> + directive to the Erlang compiler:</p> + <code type="none"> -module(my_mib). - -include_lib("sasl/include/OTP-MIB.hrl").</code> - <p>The following MIBs are defined in the OTP system: - </p> - <taglist> - <tag>OTP-REG (sasl)</tag> - <item> - <p>This MIB contains the top-level OTP registration - objects, used by all other MIBs. - </p> - </item> - <tag>OTP-TC (sasl)</tag> - <item> - <p>This MIB contains the general Textual Conventions, - which can be used by any other MIB. - </p> - </item> - <tag>OTP-MIB (sasl)</tag> - <item> - <p>This MIB contains objects for instrumentation of the - Erlang nodes, the Erlang machines and the applications in - the system. - </p> - </item> - <tag>OTP-OS-MON-MIB (os_mon)</tag> - <item> - <p>This MIB contains objects for instrumentation of disk, - memory and cpu usage of the nodes in the system. - </p> - </item> - <tag>OTP-SNMPEA-MIB (snmp)</tag> - <item> - <p>This MIB contains objects for instrumentation and - control of the extensible snmp agent itself. Note that - the agent also implements the standard SNMPv2-MIB (or v1 - part of MIB-II, if SNMPv1 is used). - </p> - </item> - <tag>OTP-EVA-MIB (eva)</tag> - <item> - <p>This MIB contains objects for instrumentation and - control of the events and alarms in the system. - </p> - </item> - <tag>OTP-LOG-MIB (eva)</tag> - <item> - <p>This MIB contains objects for instrumentation and - control of the logs and FTP transfer of logs. - </p> - </item> - <tag>OTP-EVA-LOG-MIB (eva)</tag> - <item> - <p>This MIB contains objects for instrumentation and - control of the events and alarm logs in the system. - </p> - </item> - <tag>OTP-SNMPEA-LOG-MIB (eva)</tag> - <item> - <p>This MIB contains objects for instrumentation and - control of the snmp audit trail log in the system. - </p> - </item> - </taglist> + + <p>The following MIBs are defined in the OTP system:</p> + <list type="bulleted"> + <item><p><c>OTP-REG)</c> (in <c>sasl</c>) contains the top-level + OTP registration objects, used by all other MIBs.</p></item> + <item><p><c>OTP-TC</c> (in <c>sasl</c>) contains the general + Textual Conventions, which can be used by any other MIB.</p></item> + <item><p><c>OTP-MIB</c> (in <c>sasl</c>) contains objects for + instrumentation of the Erlang nodes, the Erlang machines, + and the applications in the system.</p></item> + <item><p><c>OTP-OS-MON-MIB</c> (in <c>oc_mon</c>) contains + objects for instrumentation of disk, memory, and CPU use + of the nodes in the system.</p></item> + <item><p><c>OTP-SNMPEA-MIB</c> (in <c>snmp</c>) + contains objects for instrumentation and control of the extensible + SNMP agent itself. The agent also implements the standard SNMPv2-MIB + (or v1 part of MIB-II, if SNMPv1 is used).</p></item> + <item><p><c>OTP-EVA-MIB</c> (in <c>eva</c>) contains objects + for instrumentation and control of the events and alarms in + the system.</p></item> + <item><p><c>OTP-LOG-MIB</c> (in <c>eva</c>) contains objects + for instrumentation and control of the logs and FTP transfer of + logs.</p></item> + <item><p><c>OTP-EVA-LOG-MIB</c> (in <c>eva</c>) contains objects + for instrumentation and control of the events and alarm logs + in the system.</p></item> + <item><p><c>OTP-SNMPEA-LOG-MIB</c> (in <c>eva</c>) contains + objects for instrumentation and control of the SNMP audit + trail log in the system.</p></item> + </list> + <p>The different applications use different strategies for - loading the MIBs into the agent. Some MIB implementations are - code-only, while others need a server. One way, used by the - code-only mib implementations, is for the user to call a - function such as <c>otp_mib:init(Agent)</c> to load the MIB, - and <c>otp_mib:stop(Agent)</c> to unload the MIB. See the - application manual page for each application for a description - of how to load each MIB. - </p> + loading the MIBs into the agent. Some MIB implementations are + code-only, while others need a server. One way, used by the + code-only MIB implementations, is for the user to call a + function such as <c>otp_mib:init(Agent)</c> to load the MIB, + and <c>otp_mib:stop(Agent)</c> to unload the MIB. See the + manual page for each application for a description of how + to load each MIB.</p> </section> </section> </chapter> diff --git a/system/doc/programming_examples/bit_syntax.xml b/system/doc/programming_examples/bit_syntax.xml index fb321c1ba9..f97f171a69 100644 --- a/system/doc/programming_examples/bit_syntax.xml +++ b/system/doc/programming_examples/bit_syntax.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -31,62 +31,64 @@ <section> <title>Introduction</title> - <p>In Erlang a Bin is used for constructing binaries and matching + <p>In Erlang, a Bin is used for constructing binaries and matching binary patterns. A Bin is written with the following syntax:</p> <code type="none"><![CDATA[ <<E1, E2, ... En>>]]></code> - <p>A Bin is a low-level sequence of bits or bytes. The purpose of a Bin is - to be able to, from a high level, construct a binary,</p> + <p>A Bin is a low-level sequence of bits or bytes. + The purpose of a Bin is to enable construction of binaries:</p> <code type="none"><![CDATA[ Bin = <<E1, E2, ... En>>]]></code> - <p>in which case all elements must be bound, or to match a binary,</p> + <p>All elements must be bound. Or match a binary:</p> <code type="none"><![CDATA[ <<E1, E2, ... En>> = Bin ]]></code> - <p>where <c>Bin</c> is bound, and where the elements are bound or + <p>Here, <c>Bin</c> is bound and the elements are bound or unbound, as in any match.</p> - <p>In R12B, a Bin need not consist of a whole number of bytes.</p> + <p>Since Erlang R12B, a Bin does not need to consist of a whole number of bytes.</p> <p>A <em>bitstring</em> is a sequence of zero or more bits, where - the number of bits doesn't need to be divisible by 8. If the number + the number of bits does not need to be divisible by 8. If the number of bits is divisible by 8, the bitstring is also a binary.</p> <p>Each element specifies a certain <em>segment</em> of the bitstring. A segment is a set of contiguous bits of the binary (not necessarily on a byte boundary). The first element specifies the initial segment, the second element specifies the following - segment etc.</p> - <p>The following examples illustrate how binaries are constructed + segment, and so on.</p> + <p>The following examples illustrate how binaries are constructed, or matched, and how elements and tails are specified.</p> <section> <title>Examples</title> - <p><em>Example 1: </em>A binary can be constructed from a set of + <p><em>Example 1:</em> A binary can be constructed from a set of constants or a string literal:</p> <code type="none"><![CDATA[ Bin11 = <<1, 17, 42>>, Bin12 = <<"abc">>]]></code> - <p>yields binaries of size 3; <c>binary_to_list(Bin11)</c> - evaluates to <c>[1, 17, 42]</c>, and - <c>binary_to_list(Bin12)</c> evaluates to <c>[97, 98, 99]</c>.</p> - <p><em>Example 2: </em>Similarly, a binary can be constructed + <p>This gives two binaries of size 3, with the following evaluations:</p> + <list type="bulleted"> + <item><c>binary_to_list(Bin11)</c> evaluates to <c>[1, 17, 42]</c>.</item> + <item><c>binary_to_list(Bin12)</c> evaluates to <c>[97, 98, 99]</c>.</item> + </list> + <p><em>Example 2:</em>Similarly, a binary can be constructed from a set of bound variables:</p> <code type="none"><![CDATA[ A = 1, B = 17, C = 42, Bin2 = <<A, B, C:16>>]]></code> - <p>yields a binary of size 4, and <c>binary_to_list(Bin2)</c> - evaluates to <c>[1, 17, 00, 42]</c> too. Here we used a - <em>size expression</em> for the variable <c>C</c> in order to + <p>This gives a binary of size 4. + Here, a <em>size expression</em> is used for the variable <c>C</c> to specify a 16-bits segment of <c>Bin2</c>.</p> - <p><em>Example 3: </em>A Bin can also be used for matching: if + <p><c>binary_to_list(Bin2)</c> evaluates to <c>[1, 17, 00, 42]</c>.</p> + <p><em>Example 3:</em> A Bin can also be used for matching. <c>D</c>, <c>E</c>, and <c>F</c> are unbound variables, and - <c>Bin2</c> is bound as in the former example,</p> + <c>Bin2</c> is bound, as in Example 2:</p> <code type="none"><![CDATA[ <<D:16, E, F/binary>> = Bin2]]></code> - <p>yields <c>D = 273</c>, <c>E = 00</c>, and F binds to a binary + <p>This gives <c>D = 273</c>, <c>E = 00</c>, and F binds to a binary of size 1: <c>binary_to_list(F) = [42]</c>.</p> <p><em>Example 4:</em> The following is a more elaborate example - of matching, where <c>Dgram</c> is bound to the consecutive - bytes of an IP datagram of IP protocol version 4, and where we - want to extract the header and the data of the datagram:</p> + of matching. Here, <c>Dgram</c> is bound to the consecutive + bytes of an IP datagram of IP protocol version 4. The ambition is + to extract the header and the data of the datagram:</p> <code type="none"><![CDATA[ -define(IP_VERSION, 4). -define(IP_MIN_HDR_LEN, 5). @@ -102,53 +104,59 @@ case Dgram of <<Opts:OptsLen/binary,Data/binary>> = RestDgram, ... end.]]></code> - <p>Here the segment corresponding to the <c>Opts</c> variable - has a <em>type modifier</em> specifying that <c>Opts</c> should + <p>Here, the segment corresponding to the <c>Opts</c> variable + has a <em>type modifier</em>, specifying that <c>Opts</c> is to bind to a binary. All other variables have the default type equal to unsigned integer.</p> - <p>An IP datagram header is of variable length, and its length - - measured in the number of 32-bit words - is given in - the segment corresponding to <c>HLen</c>, the minimum value of - which is 5. It is the segment corresponding to <c>Opts</c> - that is variable: if <c>HLen</c> is equal to 5, <c>Opts</c> - will be an empty binary.</p> + <p>An IP datagram header is of variable length. This length is + measured in the number of 32-bit words and is given in + the segment corresponding to <c>HLen</c>. The minimum value of + <c>HLen</c> is 5. It is the segment corresponding to <c>Opts</c> + that is variable, so if <c>HLen</c> is equal to 5, <c>Opts</c> + becomes an empty binary.</p> <p>The tail variables <c>RestDgram</c> and <c>Data</c> bind to - binaries, as all tail variables do. Both may bind to empty + binaries, as all tail variables do. Both can bind to empty binaries.</p> - <p>If the first 4-bits segment of <c>Dgram</c> is not equal to - 4, or if <c>HLen</c> is less than 5, or if the size of - <c>Dgram</c> is less than <c>4*HLen</c>, the match of - <c>Dgram</c> fails.</p> + <p>The match of <c>Dgram</c> fails if one of the following occurs:</p> + <list type="bulleted"> + <item>The first 4-bits segment of <c>Dgram</c> is not equal to 4.</item> + <item><c>HLen</c> is less than 5.</item> + <item>The size of <c>Dgram</c> is less than <c>4*HLen</c>.</item> + </list> </section> </section> <section> - <title>A Lexical Note</title> - <p>Note that "<c><![CDATA[B=<<1>>]]></c>" will be interpreted as + <title>Lexical Note</title> + <p>Notice that "<c><![CDATA[B=<<1>>]]></c>" will be interpreted as "<c><![CDATA[B =< <1>>]]></c>", which is a syntax error. - The correct way to write the expression is - "<c><![CDATA[B = <<1>>]]></c>".</p> + The correct way to write the expression is: + <c><![CDATA[B = <<1>>]]></c>.</p> </section> <section> <title>Segments</title> <p>Each segment has the following general syntax:</p> <p><c>Value:Size/TypeSpecifierList</c></p> - <p>Both the <c>Size</c> and the <c>TypeSpecifier</c> or both may be - omitted; thus the following variations are allowed:</p> - <p><c>Value</c></p> - <p><c>Value:Size</c></p> - <p><c>Value/TypeSpecifierList</c></p> - <p>Default values will be used for missing specifications. - The default values are described in the section + <p>The <c>Size</c> or the <c>TypeSpecifier</c>, or both, can be + omitted. Thus, the following variants are allowed:</p> + <list type="bulleted"> + <item><c>Value</c></item> + <item><c>Value:Size</c></item> + <item><c>Value/TypeSpecifierList</c></item> + </list> + <p>Default values are used when specifications are missing. + The default values are described in <seealso marker="#Defaults">Defaults</seealso>.</p> - <p>Used in binary construction, the <c>Value</c> part is any - expression. Used in binary matching, the <c>Value</c> part must - be a literal or variable. You can read more about - the <c>Value</c> part in the section about constructing - binaries and matching binaries.</p> + <p>The <c>Value</c> part is any expression, when used in binary construction. + Used in binary matching, the <c>Value</c> part must + be a literal or a variable. For more information about + the <c>Value</c> part, see + <seealso marker="#Constructing Binaries and Bitstrings">Constructing Binaries and Bitstrings</seealso> + and + <seealso marker="#Matching Binaries">Matching Binaries</seealso>.</p> <p>The <c>Size</c> part of the segment multiplied by the unit in - the <c>TypeSpecifierList</c> (described below) gives the number + <c>TypeSpecifierList</c> (described later) gives the number of bits for the segment. In construction, <c>Size</c> is any expression that evaluates to an integer. In matching, <c>Size</c> must be a constant expression or a variable.</p> @@ -160,22 +168,22 @@ end.]]></code> <c>binary</c>.</item> <tag>Signedness</tag> <item>The signedness specification can be either <c>signed</c> - or <c>unsigned</c>. Note that signedness only matters for + or <c>unsigned</c>. Notice that signedness only matters for matching.</item> <tag>Endianness</tag> <item>The endianness specification can be either <c>big</c>, <c>little</c>, or <c>native</c>. Native-endian means that - the endian will be resolved at load time to be either + the endian is resolved at load time, to be either big-endian or little-endian, depending on what is "native" for the CPU that the Erlang machine is run on.</item> <tag>Unit</tag> <item>The unit size is given as <c>unit:IntegerLiteral</c>. - The allowed range is 1-256. It will be multiplied by + The allowed range is 1-256. It is multiplied by the <c>Size</c> specifier to give the effective size of - the segment. In R12B, the unit size specifies the alignment - for binary segments without size (examples will follow).</item> + the segment. Since Erlang R12B, the unit size specifies the alignment + for binary segments without size.</item> </taglist> - <p>Example:</p> + <p><em>Example:</em></p> <code type="none"> X:4/little-signed-integer-unit:8</code> <p>This element has a total size of 4*8 = 32 bits, and it contains @@ -184,13 +192,14 @@ X:4/little-signed-integer-unit:8</code> <section> <title>Defaults</title> - <p><marker id="Defaults"></marker>The default type for a segment is integer. The default + <p><marker id="Defaults"></marker>The default type for + a segment is integer. The default type does not depend on the value, even if the value is a - literal. For instance, the default type in '<c><![CDATA[<<3.14>>]]></c>' is + literal. For example, the default type in <c><![CDATA[<<3.14>>]]></c> is integer, not float.</p> <p>The default <c>Size</c> depends on the type. For integer it is 8. For float it is 64. For binary it is all of the binary. In - matching, this default value is only valid for the very last + matching, this default value is only valid for the last element. All other binary elements in matching must have a size specification.</p> <p>The default unit depends on the the type. For <c>integer</c>, @@ -201,61 +210,60 @@ X:4/little-signed-integer-unit:8</code> <section> <title>Constructing Binaries and Bitstrings</title> + <marker id="Constructing Binaries and Bitstrings"></marker> <p>This section describes the rules for constructing binaries using the bit syntax. Unlike when constructing lists or tuples, the construction of a binary can fail with a <c>badarg</c> exception.</p> <p>There can be zero or more segments in a binary to be - constructed. The expression '<c><![CDATA[<<>>]]></c>' constructs a zero + constructed. The expression <c><![CDATA[<<>>]]></c> constructs a zero length binary.</p> <p>Each segment in a binary can consist of zero or more bits. There are no alignment rules for individual segments of type <c>integer</c> and <c>float</c>. For binaries and bitstrings without size, the unit specifies the alignment. Since the default alignment for the <c>binary</c> type is 8, the size of a binary - segment must be a multiple of 8 bits (i.e. only whole bytes). - Example:</p> + segment must be a multiple of 8 bits, that is, only whole bytes.</p> + <p><em>Example:</em></p> <code type="none"><![CDATA[ <<Bin/binary,Bitstring/bitstring>>]]></code> <p>The variable <c>Bin</c> must contain a whole number of bytes, because the <c>binary</c> type defaults to <c>unit:8</c>. - A <c>badarg</c> exception will be generated if <c>Bin</c> would - consist of (for instance) 17 bits.</p> + A <c>badarg</c> exception is generated if <c>Bin</c> + consist of, for example, 17 bits.</p> - <p>On the other hand, the variable <c>Bitstring</c> may consist of - any number of bits, for instance 0, 1, 8, 11, 17, 42, and so on, - because the default <c>unit</c> for bitstrings is 1.</p> + <p>The <c>Bitstring</c> variable can consist of + any number of bits, for example, 0, 1, 8, 11, 17, 42, and so on. + This is because the default <c>unit</c> for bitstrings is 1.</p> - <warning><p>For clarity, it is recommended not to change the unit - size for binaries, but to use <c>binary</c> when you need byte - alignment, and <c>bitstring</c> when you need bit alignment.</p></warning> + <p>For clarity, it is recommended not to change the unit + size for binaries. Instead, use <c>binary</c> when you need byte alignment + and <c>bitstring</c> when you need bit alignment.</p> - <p>The following example</p> + <p>The following example successfully constructs a bitstring of 7 bits, + provided that all of X and Y are integers:</p> <code type="none"><![CDATA[ <<X:1,Y:6>>]]></code> - <p>will successfully construct a bitstring of 7 bits. - (Provided that all of X and Y are integers.)</p> - <p>As noted earlier, segments have the following general syntax:</p> + <p>As mentioned earlier, segments have the following general syntax:</p> <p><c>Value:Size/TypeSpecifierList</c></p> <p>When constructing binaries, <c>Value</c> and <c>Size</c> can be any Erlang expression. However, for syntactical reasons, both <c>Value</c> and <c>Size</c> must be enclosed in parenthesis if the expression consists of anything more than a single literal - or variable. The following gives a compiler syntax error:</p> + or a variable. The following gives a compiler syntax error:</p> <code type="none"><![CDATA[ <<X+1:8>>]]></code> - <p>This expression must be rewritten to</p> + <p>This expression must be rewritten into the following, + to be accepted by the compiler:</p> <code type="none"><![CDATA[ <<(X+1):8>>]]></code> - <p>in order to be accepted by the compiler.</p> <section> <title>Including Literal Strings</title> - <p>As syntactic sugar, an literal string may be written instead - of a element.</p> + <p>A literal string can be written instead of an element:</p> <code type="none"><![CDATA[ <<"hello">>]]></code> - <p>which is syntactic sugar for</p> + <p>This is syntactic sugar for the following:</p> <code type="none"><![CDATA[ <<$h,$e,$l,$l,$o>>]]></code> </section> @@ -263,29 +271,30 @@ X:4/little-signed-integer-unit:8</code> <section> <title>Matching Binaries</title> - <p>This section describes the rules for matching binaries using + <marker id="Matching Binaries"></marker> + <p>This section describes the rules for matching binaries, using the bit syntax.</p> <p>There can be zero or more segments in a binary pattern. - A binary pattern can occur in every place patterns are allowed, - also inside other patterns. Binary patterns cannot be nested.</p> - <p>The pattern '<c><![CDATA[<<>>]]></c>' matches a zero length binary.</p> - <p>Each segment in a binary can consist of zero or more bits.</p> - <p>A segment of type <c>binary</c> must have a size evenly - divisible by 8 (or divisible by the unit size, if the unit size has been changed).</p> - <p>A segment of type <c>bitstring</c> has no restrictions on the size.</p> - <p>As noted earlier, segments have the following general syntax:</p> + A binary pattern can occur wherever patterns are allowed, + including inside other patterns. Binary patterns cannot be nested. + The pattern <c><![CDATA[<<>>]]></c> matches a zero length binary.</p> + <p>Each segment in a binary can consist of zero or more bits. + A segment of type <c>binary</c> must have a size evenly divisible by 8 + (or divisible by the unit size, if the unit size has been changed). + A segment of type <c>bitstring</c> has no restrictions on the size.</p> + <p>As mentioned earlier, segments have the following general syntax:</p> <p><c>Value:Size/TypeSpecifierList</c></p> - <p>When matching <c>Value</c> value must be either a variable or - an integer or floating point literal. Expressions are not + <p>When matching <c>Value</c>, value must be either a variable or + an integer, or a floating point literal. Expressions are not allowed.</p> <p><c>Size</c> must be an integer literal, or a previously bound - variable. Note that the following is not allowed:</p> + variable. The following is not allowed:</p> <code type="none"><![CDATA[ foo(N, <<X:N,T/binary>>) -> {X,T}.]]></code> <p>The two occurrences of <c>N</c> are not related. The compiler will complain that the <c>N</c> in the size field is unbound.</p> - <p>The correct way to write this example is like this:</p> + <p>The correct way to write this example is as follows:</p> <code type="none"><![CDATA[ foo(N, Bin) -> <<X:N,T/binary>> = Bin, @@ -303,14 +312,14 @@ foo(<<A:8,Rest/binary>>) ->]]></code> without size:</p> <code type="none"><![CDATA[ foo(<<A:8,Rest/bitstring>>) ->]]></code> - <p>There is no restriction on the number of bits in the tail.</p> + <p>There are no restrictions on the number of bits in the tail.</p> </section> </section> <section> <title>Appending to a Binary</title> - <p>In R12B, the following function for creating a binary out of - a list of triples of integers is now efficient:</p> + <p>Since Erlang R12B, the following function for creating a binary out of + a list of triples of integers is efficient:</p> <code type="none"><![CDATA[ triples_to_bin(T) -> triples_to_bin(T, <<>>). @@ -321,7 +330,9 @@ triples_to_bin([], Acc) -> Acc.]]></code> <p>In previous releases, this function was highly inefficient, because the binary constructed so far (<c>Acc</c>) was copied in each recursion step. - That is no longer the case. See the Efficiency Guide for more information.</p> + That is no longer the case. For more information, see + <seealso marker="doc/efficiency_guide:introduction"> + Efficiency Guide</seealso>.</p> </section> </chapter> diff --git a/system/doc/programming_examples/funs.xmlsrc b/system/doc/programming_examples/funs.xmlsrc index 7bfac9db8c..d4c32bc854 100644 --- a/system/doc/programming_examples/funs.xmlsrc +++ b/system/doc/programming_examples/funs.xmlsrc @@ -30,146 +30,138 @@ </header> <section> - <title>Example 1 - map</title> - <p>If we want to double every element in a list, we could write a - function named <c>double</c>:</p> + <title>map</title> + <p>The following function, <c>double</c>, doubles every element in a list:</p> <code type="none"> double([H|T]) -> [2*H|double(T)]; double([]) -> [].</code> - <p>This function obviously doubles the argument entered as input - as follows:</p> + <p>Hence, the argument entered as input is doubled as follows:</p> <pre> > <input>double([1,2,3,4]).</input> [2,4,6,8]</pre> - <p>We now add the function <c>add_one</c>, which adds one to every + <p>The following function, <c>add_one</c>, adds one to every element in a list:</p> <code type="none"> add_one([H|T]) -> [H+1|add_one(T)]; add_one([]) -> [].</code> - <p>These functions, <c>double</c> and <c>add_one</c>, have a very - similar structure. We can exploit this fact and write a function - <c>map</c> which expresses this similarity:</p> + <p>The functions <c>double</c> and <c>add_one</c> have a + similar structure. This can be used by writing a function + <c>map</c> that expresses this similarity:</p> <codeinclude file="funs1.erl" tag="%1" type="erl"></codeinclude> - <p>We can now express the functions <c>double</c> and - <c>add_one</c> in terms of <c>map</c> as follows:</p> + <p>The functions <c>double</c> and <c>add_one</c> can now be expressed + in terms of <c>map</c> as follows:</p> <code type="none"> double(L) -> map(fun(X) -> 2*X end, L). add_one(L) -> map(fun(X) -> 1 + X end, L).</code> - <p><c>map(F, List)</c> is a function which takes a function - <c>F</c> and a list <c>L</c> as arguments and returns the new - list which is obtained by applying <c>F</c> to each of + <p><c>map(F, List)</c> is a function that takes a function + <c>F</c> and a list <c>L</c> as arguments and returns a new + list, obtained by applying <c>F</c> to each of the elements in <c>L</c>.</p> <p>The process of abstracting out the common features of a number - of different programs is called procedural abstraction. - Procedural abstraction can be used in order to write several - different functions which have a similar structure, but differ - only in some minor detail. This is done as follows:</p> + of different programs is called <em>procedural abstraction</em>. + Procedural abstraction can be used to write several + different functions that have a similar structure, but differ + in some minor detail. This is done as follows:</p> <list type="ordered"> - <item>write one function which represents the common features of - these functions</item> - <item>parameterize the difference in terms of functions which + <item><em>Step 1.</em> Write one function that represents the common features of + these functions.</item> + <item><em>Step 2.</em> Parameterize the difference in terms of functions that are passed as arguments to the common function.</item> </list> </section> <section> - <title>Example 2 - foreach</title> - <p>This example illustrates procedural abstraction. Initially, we - show the following two examples written as conventional - functions:</p> - <list type="ordered"> - <item>all elements of a list are printed onto a stream</item> - <item>a message is broadcast to a list of processes.</item> - </list> + <title>foreach</title> + <p>This section illustrates procedural abstraction. Initially, + the following two examples are written as conventional + functions.</p> + <p>This function prints all elements of a list onto a stream:</p> <code type="none"> print_list(Stream, [H|T]) -> io:format(Stream, "~p~n", [H]), print_list(Stream, T); print_list(Stream, []) -> true.</code> + <p>This function broadcasts a message to a list of processes:</p> <code type="none"> broadcast(Msg, [Pid|Pids]) -> Pid ! Msg, broadcast(Msg, Pids); broadcast(_, []) -> true.</code> - <p>Both these functions have a very similar structure. They both - iterate over a list doing something to each element in the list. - The "something" has to be carried round as an extra argument to - the function which does this.</p> + <p>These two functions have a similar structure. They both + iterate over a list and do something to each element in the list. + The "something" is passed on as an extra argument to + the function that does this.</p> <p>The function <c>foreach</c> expresses this similarity:</p> <codeinclude file="funs1.erl" tag="%2" type="erl"></codeinclude> - <p>Using <c>foreach</c>, <c>print_list</c> becomes:</p> + <p>Using the function <c>foreach</c>, the function <c>print_list</c> becomes:</p> <code type="none"> foreach(fun(H) -> io:format(S, "~p~n",[H]) end, L)</code> - <p><c>broadcast</c> becomes:</p> + <p>Using the function <c>foreach</c>, the function <c>broadcast</c> becomes:</p> <code type="none"> foreach(fun(Pid) -> Pid ! M end, L)</code> <p><c>foreach</c> is evaluated for its side-effect and not its value. <c>foreach(Fun ,L)</c> calls <c>Fun(X)</c> for each element <c>X</c> in <c>L</c> and the processing occurs in - the order in which the elements were defined in <c>L</c>. + the order that the elements were defined in <c>L</c>. <c>map</c> does not define the order in which its elements are processed.</p> </section> <section> - <title>The Syntax of Funs</title> - <p>Funs are written with the syntax:</p> + <title>Syntax of Funs</title> + <p>Funs are written with the following syntax:</p> <code type="none"> F = fun (Arg1, Arg2, ... ArgN) -> ... end</code> <p>This creates an anonymous function of <c>N</c> arguments and binds it to the variable <c>F</c>.</p> - <p>If we have already written a function in the same module and - wish to pass this function as an argument, we can use - the following syntax:</p> + <p>Another function, <c>FunctionName</c>, written in the same module, + can be passed as an argument, using the following syntax:</p> <code type="none"> F = fun FunctionName/Arity</code> - <p>With this form of function reference, the function which is + <p>With this form of function reference, the function that is referred to does not need to be exported from the module.</p> - <p>We can also refer to a function defined in a different module + <p>It is also possible to refer to a function defined in a different module, with the following syntax:</p> <code type="none"> -F = {Module, FunctionName}</code> +F = fun Module:FunctionName/Arity</code> <p>In this case, the function must be exported from the module in question.</p> - <p>The follow program illustrates the different ways of creating + <p>The following program illustrates the different ways of creating funs:</p> <codeinclude file="fun_test.erl" tag="%1" type="erl"></codeinclude> - <p>We can evaluate the fun <c>F</c> with the syntax:</p> + <p>The fun <c>F</c> can be evaluated with the following syntax:</p> <code type="none"> F(Arg1, Arg2, ..., Argn)</code> <p>To check whether a term is a fun, use the test - <c>is_function/1</c> in a guard. Example:</p> + <c>is_function/1</c> in a guard.</p> + <p><em>Example:</em></p> <code type="none"> f(F, Args) when is_function(F) -> apply(F, Args); f(N, _) when is_integer(N) -> N.</code> - <p>Funs are a distinct type. The BIFs erlang:fun_info/1,2 can + <p>Funs are a distinct type. The BIFs <c>erlang:fun_info/1,2</c> can be used to retrieve information about a fun, and the BIF - erlang:fun_to_list/1 returns a textual representation of a fun. - The check_process_code/2 BIF returns true if the process + <c>erlang:fun_to_list/1</c> returns a textual representation of a fun. + The <c>check_process_code/2</c> BIF returns <c>true</c> if the process contains funs that depend on the old version of a module.</p> - <note> - <p>In OTP R5 and earlier releases, funs were represented using - tuples.</p> - </note> </section> <section> <title>Variable Bindings Within a Fun</title> - <p>The scope rules for variables which occur in funs are as + <p>The scope rules for variables that occur in funs are as follows:</p> <list type="bulleted"> - <item>All variables which occur in the head of a fun are assumed + <item>All variables that occur in the head of a fun are assumed to be "fresh" variables.</item> - <item>Variables which are defined before the fun, and which + <item>Variables that are defined before the fun, and that occur in function calls or guard tests within the fun, have the values they had outside the fun.</item> - <item>No variables may be exported from a fun.</item> + <item>Variables cannot be exported from a fun.</item> </list> <p>The following examples illustrate these rules:</p> <code type="none"> @@ -177,12 +169,13 @@ print_list(File, List) -> {ok, Stream} = file:open(File, write), foreach(fun(X) -> io:format(Stream,"~p~n",[X]) end, List), file:close(Stream).</code> - <p>In the above example, the variable <c>X</c> which is defined in - the head of the fun is a new variable. The value of the variable - <c>Stream</c> which is used within within the fun gets its value + <p>Here, the variable <c>X</c>, defined in + the head of the fun, is a new variable. The variable + <c>Stream</c>, which is used within the fun, gets its value from the <c>file:open</c> line.</p> - <p>Since any variable which occurs in the head of a fun is - considered a new variable it would be equally valid to write:</p> + <p>As any variable that occurs in the head of a fun is + considered a new variable, it is equally valid to write + as follows:</p> <code type="none"> print_list(File, List) -> {ok, Stream} = file:open(File, write), @@ -190,21 +183,21 @@ print_list(File, List) -> io:format(Stream,"~p~n",[File]) end, List), file:close(Stream).</code> - <p>In this example, <c>File</c> is used as the new variable - instead of <c>X</c>. This is rather silly since code in the body - of the fun cannot refer to the variable <c>File</c> which is - defined outside the fun. Compiling this example will yield - the diagnostic:</p> + <p>Here, <c>File</c> is used as the new variable + instead of <c>X</c>. This is not so wise because code in the fun + body cannot refer to the variable <c>File</c>, which is + defined outside of the fun. Compiling this example gives + the following diagnostic:</p> <code type="none"> ./FileName.erl:Line: Warning: variable 'File' - shadowed in 'lambda head'</code> - <p>This reminds us that the variable <c>File</c> which is defined - inside the fun collides with the variable <c>File</c> which is + shadowed in 'fun'</code> + <p>This indicates that the variable <c>File</c>, which is defined + inside the fun, collides with the variable <c>File</c>, which is defined outside the fun.</p> <p>The rules for importing variables into a fun has the consequence - that certain pattern matching operations have to be moved into + that certain pattern matching operations must be moved into guard expressions and cannot be written in the head of the fun. - For example, we might write the following code if we intend + For example, you might write the following code if you intend the first clause of <c>F</c> to be evaluated when the value of its argument is <c>Y</c>:</p> <code type="none"> @@ -216,7 +209,7 @@ f(...) -> ... end, ...) ...</code> - <p>instead of</p> + <p>instead of writng the following code:</p> <code type="none"> f(...) -> Y = ... @@ -229,35 +222,37 @@ f(...) -> </section> <section> - <title>Funs and the Module Lists</title> + <title>Funs and Module Lists</title> <p>The following examples show a dialogue with the Erlang shell. All the higher order functions discussed are exported from the module <c>lists</c>.</p> <section> <title>map</title> + <p><c>map</c> takes a function of one argument and a list of terms:</p> <codeinclude file="funs1.erl" tag="%1" type="erl"></codeinclude> - <p><c>map</c> takes a function of one argument and a list of - terms. It returns the list obtained by applying the function + <p>It returns the list obtained by applying the function to every argument in the list.</p> + <p>When a new fun is defined in the shell, the value of the fun + is printed as <c><![CDATA[Fun#<erl_eval>]]></c>:</p> <pre> > <input>Double = fun(X) -> 2 * X end.</input> #Fun<erl_eval.6.72228031> > <input>lists:map(Double, [1,2,3,4,5]).</input> [2,4,6,8,10]</pre> - <p>When a new fun is defined in the shell, the value of the Fun - is printed as <c><![CDATA[Fun#<erl_eval>]]></c>.</p> + </section> <section> <title>any</title> - <codeinclude file="funs1.erl" tag="%4" type="erl"></codeinclude> <p><c>any</c> takes a predicate <c>P</c> of one argument and a - list of terms. A predicate is a function which returns - <c>true</c> or <c>false</c>. <c>any</c> is true if there is a - term <c>X</c> in the list such that <c>P(X)</c> is <c>true</c>.</p> - <p>We define a predicate <c>Big(X)</c> which is <c>true</c> if - its argument is greater that 10.</p> + list of terms:</p> + <codeinclude file="funs1.erl" tag="%4" type="erl"></codeinclude> + <p>A predicate is a function that returns <c>true</c> or <c>false</c>. + <c>any</c> is <c>true</c> if there is a term <c>X</c> in the list such that + <c>P(X)</c> is <c>true</c>.</p> + <p>A predicate <c>Big(X)</c> is defined, which is <c>true</c> if + its argument is greater that 10:</p> <pre> > <input>Big = fun(X) -> if X > 10 -> true; true -> false end end.</input> #Fun<erl_eval.6.72228031> @@ -269,9 +264,10 @@ true</pre> <section> <title>all</title> + <p><c>all</c> has the same arguments as <c>any</c>:</p> <codeinclude file="funs1.erl" tag="%3" type="erl"></codeinclude> - <p><c>all</c> has the same arguments as <c>any</c>. It is true - if the predicate applied to all elements in the list is true.</p> + <p>It is <c>true</c> + if the predicate applied to all elements in the list is <c>true</c>.</p> <pre> > <input>lists:all(Big, [1,2,3,4,12,6]).</input> false @@ -281,11 +277,12 @@ true</pre> <section> <title>foreach</title> - <codeinclude file="funs1.erl" tag="%2" type="erl"></codeinclude> <p><c>foreach</c> takes a function of one argument and a list of - terms. The function is applied to each argument in the list. - <c>foreach</c> returns <c>ok</c>. It is used for its - side-effect only.</p> + terms:</p> + <codeinclude file="funs1.erl" tag="%2" type="erl"></codeinclude> + <p>The function is applied to each argument in the list. + <c>foreach</c> returns <c>ok</c>. It is only used for its + side-effect:</p> <pre> > <input>lists:foreach(fun(X) -> io:format("~w~n",[X]) end, [1,2,3,4]).</input> 1 @@ -297,15 +294,16 @@ ok</pre> <section> <title>foldl</title> - <codeinclude file="funs1.erl" tag="%8" type="erl"></codeinclude> <p><c>foldl</c> takes a function of two arguments, an - accumulator and a list. The function is called with two + accumulator and a list:</p> + <codeinclude file="funs1.erl" tag="%8" type="erl"></codeinclude> + <p>The function is called with two arguments. The first argument is the successive elements in - the list, the second argument is the accumulator. The function - must return a new accumulator which is used the next time + the list. The second argument is the accumulator. The function + must return a new accumulator, which is used the next time the function is called.</p> - <p>If we have a list of lists <c>L = ["I","like","Erlang"]</c>, - then we can sum the lengths of all the strings in <c>L</c> as + <p>If you have a list of lists <c>L = ["I","like","Erlang"]</c>, + then you can sum the lengths of all the strings in <c>L</c> as follows:</p> <pre> > <input>L = ["I","like","Erlang"].</input> @@ -325,11 +323,11 @@ end</code> <section> <title>mapfoldl</title> + <p><c>mapfoldl</c> simultaneously maps and folds over a list:</p> <codeinclude file="funs1.erl" tag="%10" type="erl"></codeinclude> - <p><c>mapfoldl</c> simultaneously maps and folds over a list. - The following example shows how to change all letters in - <c>L</c> to upper case and count them.</p> - <p>First upcase:</p> + <p>The following example shows how to change all letters in + <c>L</c> to upper case and then count them.</p> + <p>First the change to upper case:</p> <pre> > <input>Upcase = fun(X) when $a =< X, X =< $z -> X + $A - $a;</input> <input>(X) -> X</input> @@ -344,7 +342,7 @@ end</code> "ERLANG" > <input>lists:map(Upcase_word, L).</input> ["I","LIKE","ERLANG"]</pre> - <p>Now we can do the fold and the map at the same time:</p> + <p>Now, the fold and the map can be done at the same time:</p> <pre> > <input>lists:mapfoldl(fun(Word, Sum) -></input> <input>{Upcase_word(Word), Sum + length(Word)}</input> @@ -354,23 +352,24 @@ end</code> <section> <title>filter</title> - <codeinclude file="funs1.erl" tag="%9" type="erl"></codeinclude> <p><c>filter</c> takes a predicate of one argument and a list - and returns all element in the list which satisfy - the predicate.</p> + and returns all elements in the list that satisfy + the predicate:</p> + <codeinclude file="funs1.erl" tag="%9" type="erl"></codeinclude> <pre> > <input>lists:filter(Big, [500,12,2,45,6,7]).</input> [500,12,45]</pre> - <p>When we combine maps and filters we can write very succinct - code. For example, suppose we want to define a set difference - function. We want to define <c>diff(L1, L2)</c> to be - the difference between the lists <c>L1</c> and <c>L2</c>. - This is the list of all elements in L1 which are not contained - in L2. This code can be written as follows:</p> + <p>Combining maps and filters enables writing of very succinct + code. For example, to define a set difference + function <c>diff(L1, L2)</c> to be + the difference between the lists <c>L1</c> and <c>L2</c>, + the code can be written as follows:</p> <code type="none"> diff(L1, L2) -> filter(fun(X) -> not member(X, L2) end, L1).</code> - <p>The AND intersection of the list <c>L1</c> and <c>L2</c> is + <p>This gives the list of all elements in L1 that are not contained + in L2.</p> + <p> The AND intersection of the list <c>L1</c> and <c>L2</c> is also easily defined:</p> <code type="none"> intersection(L1,L2) -> filter(fun(X) -> member(X,L1) end, L2).</code> @@ -378,9 +377,9 @@ intersection(L1,L2) -> filter(fun(X) -> member(X,L1) end, L2).</code> <section> <title>takewhile</title> - <codeinclude file="funs1.erl" tag="%5" type="erl"></codeinclude> <p><c>takewhile(P, L)</c> takes elements <c>X</c> from a list - <c>L</c> as long as the predicate <c>P(X)</c> is true.</p> + <c>L</c> as long as the predicate <c>P(X)</c> is true:</p> + <codeinclude file="funs1.erl" tag="%5" type="erl"></codeinclude> <pre> > <input>lists:takewhile(Big, [200,500,45,5,3,45,6]).</input> [200,500,45]</pre> @@ -388,8 +387,8 @@ intersection(L1,L2) -> filter(fun(X) -> member(X,L1) end, L2).</code> <section> <title>dropwhile</title> + <p><c>dropwhile</c> is the complement of <c>takewhile</c>:</p> <codeinclude file="funs1.erl" tag="%6" type="erl"></codeinclude> - <p><c>dropwhile</c> is the complement of <c>takewhile</c>.</p> <pre> > <input>lists:dropwhile(Big, [200,500,45,5,3,45,6]).</input> [5,3,45,6]</pre> @@ -397,10 +396,10 @@ intersection(L1,L2) -> filter(fun(X) -> member(X,L1) end, L2).</code> <section> <title>splitwith</title> - <codeinclude file="funs1.erl" tag="%7" type="erl"></codeinclude> <p><c>splitwith(P, L)</c> splits the list <c>L</c> into the two - sub-lists <c>{L1, L2}</c>, where <c>L = takewhile(P, L)</c> - and <c>L2 = dropwhile(P, L)</c>.</p> + sublists <c>{L1, L2}</c>, where <c>L = takewhile(P, L)</c> + and <c>L2 = dropwhile(P, L)</c>:</p> + <codeinclude file="funs1.erl" tag="%7" type="erl"></codeinclude> <pre> > <input>lists:splitwith(Big, [200,500,45,5,3,45,6]).</input> {[200,500,45],[5,3,45,6]}</pre> @@ -408,17 +407,17 @@ intersection(L1,L2) -> filter(fun(X) -> member(X,L1) end, L2).</code> </section> <section> - <title>Funs Which Return Funs</title> - <p>So far, this section has only described functions which take - funs as arguments. It is also possible to write more powerful - functions which themselves return funs. The following examples - illustrate these type of functions.</p> + <title>Funs Returning Funs</title> + <p>So far, only functions that take + funs as arguments have been described. More powerful + functions, that themselves return funs, can also be written. The following + examples illustrate these type of functions.</p> <section> <title>Simple Higher Order Functions</title> - <p><c>Adder(X)</c> is a function which, given <c>X</c>, returns + <p><c>Adder(X)</c> is a function that given <c>X</c>, returns a new function <c>G</c> such that <c>G(K)</c> returns - <c>K + X</c>.</p> + <c>K + X</c>:</p> <pre> > <input>Adder = fun(X) -> fun(Y) -> X + Y end end.</input> #Fun<erl_eval.6.72228031> @@ -438,7 +437,7 @@ ints_from(N) -> fun() -> [N|ints_from(N+1)] end.</code> - <p>Then we can proceed as follows:</p> + <p>Then proceed as follows:</p> <pre> > <input>XX = lazy:ints_from(1).</input> #Fun<lazy.0.29874839> @@ -450,7 +449,7 @@ ints_from(N) -> #Fun<lazy.0.29874839> > <input>hd(Y()).</input> 2</pre> - <p>etc. - this is an example of "lazy embedding".</p> + <p>And so on. This is an example of "lazy embedding".</p> </section> <section> @@ -459,17 +458,21 @@ ints_from(N) -> <pre> Parser(Toks) -> {ok, Tree, Toks1} | fail</pre> <p><c>Toks</c> is the list of tokens to be parsed. A successful - parse returns <c>{ok, Tree, Toks1}</c>, where <c>Tree</c> is a - parse tree and <c>Toks1</c> is a tail of <c>Tree</c> which - contains symbols encountered after the structure which was - correctly parsed. Otherwise <c>fail</c> is returned.</p> - <p>The example which follows illustrates a simple, functional - parser which parses the grammar:</p> + parse returns <c>{ok, Tree, Toks1}</c>.</p> + <list type="bulleted"> + <item><c>Tree</c> is a parse tree.</item> + <item><c>Toks1</c> is a tail of <c>Tree</c> that + contains symbols encountered after the structure that was + correctly parsed.</item> + </list> + <p>An unsuccessful parse returns <c>fail</c>.</p> + <p>The following example illustrates a simple, functional + parser that parses the grammar:</p> <pre> (a | b) & (c | d)</pre> <p>The following code defines a function <c>pconst(X)</c> in - the module <c>funparse</c>, which returns a fun which parses a - list of tokens.</p> + the module <c>funparse</c>, which returns a fun that parses a + list of tokens:</p> <codeinclude file="funparse.erl" tag="%14" type="erl"></codeinclude> <p>This function can be used as follows:</p> <pre> @@ -479,17 +482,18 @@ Parser(Toks) -> {ok, Tree, Toks1} | fail</pre> {ok,{const,a},[b,c]} > <input>P1([x,y,z]).</input> fail</pre> - <p>Next, we define the two higher order functions <c>pand</c> - and <c>por</c> which combine primitive parsers to produce more - complex parsers. Firstly <c>pand</c>:</p> + <p>Next, the two higher order functions <c>pand</c> + and <c>por</c> are defined. They combine primitive parsers to produce more + complex parsers.</p> + <p>First <c>pand</c>:</p> <codeinclude file="funparse.erl" tag="%16" type="erl"></codeinclude> <p>Given a parser <c>P1</c> for grammar <c>G1</c>, and a parser <c>P2</c> for grammar <c>G2</c>, <c>pand(P1, P2)</c> returns a - parser for the grammar which consists of sequences of tokens - which satisfy <c>G1</c> followed by sequences of tokens which + parser for the grammar, which consists of sequences of tokens + that satisfy <c>G1</c>, followed by sequences of tokens that satisfy <c>G2</c>.</p> <p><c>por(P1, P2)</c> returns a parser for the language - described by the grammar <c>G1</c> or <c>G2</c>.</p> + described by the grammar <c>G1</c> or <c>G2</c>:</p> <codeinclude file="funparse.erl" tag="%15" type="erl"></codeinclude> <p>The original problem was to parse the grammar <c><![CDATA[(a | b) & (c | d)]]></c>. The following code addresses this @@ -497,7 +501,7 @@ fail</pre> <codeinclude file="funparse.erl" tag="%13" type="erl"></codeinclude> <p>The following code adds a parser interface to the grammar:</p> <codeinclude file="funparse.erl" tag="%12" type="erl"></codeinclude> - <p>We can test this parser as follows:</p> + <p>The parser can be tested as follows:</p> <pre> > <input>funparse:parse([a,c]).</input> {ok,{'and',{'or',1,{const,a}},{'or',1,{const,c}}}} diff --git a/system/doc/programming_examples/list_comprehensions.xml b/system/doc/programming_examples/list_comprehensions.xml index d6c8a66e13..5b33b14dea 100644 --- a/system/doc/programming_examples/list_comprehensions.xml +++ b/system/doc/programming_examples/list_comprehensions.xml @@ -31,18 +31,15 @@ <section> <title>Simple Examples</title> - <p>We start with a simple example:</p> + <p>This section starts with a simple example, showing a generator and a filter:</p> <pre> > <input>[X || X <- [1,2,a,3,4,b,5,6], X > 3].</input> [a,4,b,5,6]</pre> - <p>This should be read as follows:</p> - <quote> - <p>The list of X such that X is taken from the list + <p>This is read as follows: The list of X such that X is taken from the list <c>[1,2,a,...]</c> and X is greater than 3.</p> - </quote> <p>The notation <c><![CDATA[X <- [1,2,a,...]]]></c> is a generator and the expression <c>X > 3</c> is a filter.</p> - <p>An additional filter can be added in order to restrict + <p>An additional filter, <c>integer(X)</c>, can be added to restrict the result to integers:</p> <pre> > <input>[X || X <- [1,2,a,3,4,b,5,6], integer(X), X > 3].</input> @@ -56,7 +53,7 @@ <section> <title>Quick Sort</title> - <p>The well known quick sort routine can be written as follows:</p> + <p>The well-known quick sort routine can be written as follows:</p> <code type="none"><![CDATA[ sort([Pivot|T]) -> sort([ X || X <- T, X < Pivot]) ++ @@ -64,15 +61,20 @@ sort([Pivot|T]) -> sort([ X || X <- T, X >= Pivot]); sort([]) -> [].]]></code> <p>The expression <c><![CDATA[[X || X <- T, X < Pivot]]]></c> is the list of - all elements in <c>T</c>, which are less than <c>Pivot</c>.</p> + all elements in <c>T</c> that are less than <c>Pivot</c>.</p> <p><c><![CDATA[[X || X <- T, X >= Pivot]]]></c> is the list of all elements in - <c>T</c>, which are greater or equal to <c>Pivot</c>.</p> - <p>To sort a list, we isolate the first element in the list and - split the list into two sub-lists. The first sub-list contains - all elements which are smaller than the first element in - the list, the second contains all elements which are greater - than or equal to the first element in the list. We then sort - the sub-lists and combine the results.</p> + <c>T</c> that are greater than or equal to <c>Pivot</c>.</p> + <p>A list sorted as follows:</p> + <list type="bulleted"> + <item>The first element in the list is isolated + and the list is split into two sublists.</item> + <item>The first sublist contains + all elements that are smaller than the first element in + the list.</item> + <item>The second sublist contains all elements that are greater + than, or equal to, the first element in the list.</item> + <item>Then the sublists are sorted and the results are combined.</item> + </list> </section> <section> @@ -82,10 +84,10 @@ sort([]) -> [].]]></code> <code type="none"><![CDATA[ perms([]) -> [[]]; perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])].]]></code> - <p>We take take <c>H</c> from <c>L</c> in all possible ways. + <p>This takes <c>H</c> from <c>L</c> in all possible ways. The result is the set of all lists <c>[H|T]</c>, where <c>T</c> - is the set of all possible permutations of <c>L</c> with - <c>H</c> removed.</p> + is the set of all possible permutations of <c>L</c>, with + <c>H</c> removed:</p> <pre> > <input>perms([b,u,g]).</input> [[b,u,g],[b,g,u],[u,b,g],[u,g,b],[g,b,u],[g,u,b]]</pre> @@ -97,7 +99,7 @@ perms(L) -> [[H|T] || H <- L, T <- perms(L--[H])].]]></code> that <c>A**2 + B**2 = C**2</c>.</p> <p>The function <c>pyth(N)</c> generates a list of all integers <c>{A,B,C}</c> such that <c>A**2 + B**2 = C**2</c> and where - the sum of the sides is equal to or less than <c>N</c>.</p> + the sum of the sides is equal to, or less than, <c>N</c>:</p> <code type="none"><![CDATA[ pyth(N) -> [ {A,B,C} || @@ -140,7 +142,7 @@ pyth1(N) -> </section> <section> - <title>Simplifications with List Comprehensions</title> + <title>Simplifications With List Comprehensions</title> <p>As an example, list comprehensions can be used to simplify some of the functions in <c>lists.erl</c>:</p> <code type="none"><![CDATA[ @@ -151,45 +153,47 @@ filter(Pred, L) -> [X || X <- L, Pred(X)].]]></code> <section> <title>Variable Bindings in List Comprehensions</title> - <p>The scope rules for variables which occur in list + <p>The scope rules for variables that occur in list comprehensions are as follows:</p> <list type="bulleted"> - <item>all variables which occur in a generator pattern are - assumed to be "fresh" variables</item> - <item>any variables which are defined before the list - comprehension and which are used in filters have the values - they had before the list comprehension</item> - <item>no variables may be exported from a list comprehension.</item> + <item>All variables that occur in a generator pattern are + assumed to be "fresh" variables.</item> + <item>Any variables that are defined before the list + comprehension, and that are used in filters, have the values + they had before the list comprehension.</item> + <item>Variables cannot be exported from a list comprehension.</item> </list> - <p>As an example of these rules, suppose we want to write + <p>As an example of these rules, suppose you want to write the function <c>select</c>, which selects certain elements from - a list of tuples. We might write + a list of tuples. Suppose you write <c><![CDATA[select(X, L) -> [Y || {X, Y} <- L].]]></c> with the intention - of extracting all tuples from <c>L</c> where the first item is + of extracting all tuples from <c>L</c>, where the first item is <c>X</c>.</p> - <p>Compiling this yields the following diagnostic:</p> + <p>Compiling this gives the following diagnostic:</p> <code type="none"> ./FileName.erl:Line: Warning: variable 'X' shadowed in generate</code> - <p>This diagnostic warns us that the variable <c>X</c> in - the pattern is not the same variable as the variable <c>X</c> - which occurs in the function head.</p> - <p>Evaluating <c>select</c> yields the following result:</p> + <p>This diagnostic warns that the variable <c>X</c> in + the pattern is not the same as the variable <c>X</c> + that occurs in the function head.</p> + <p>Evaluating <c>select</c> gives the following result:</p> <pre> > <input>select(b,[{a,1},{b,2},{c,3},{b,7}]).</input> [1,2,3,7]</pre> - <p>This result is not what we wanted. To achieve the desired - effect we must write <c>select</c> as follows:</p> + <p>This is not the wanted result. To achieve the desired + effect, <c>select</c> must be written as follows:</p> <code type="none"><![CDATA[ select(X, L) -> [Y || {X1, Y} <- L, X == X1].]]></code> <p>The generator now contains unbound variables and the test has - been moved into the filter. This now works as expected:</p> + been moved into the filter.</p> + <p>This now works as expected:</p> <pre> > <input>select(b,[{a,1},{b,2},{c,3},{b,7}]).</input> [2,7]</pre> - <p>One consequence of the rules for importing variables into a + <p>A consequence of the rules for importing variables into a list comprehensions is that certain pattern matching operations - have to be moved into the filters and cannot be written directly - in the generators. To illustrate this, do not write as follows:</p> + must be moved into the filters and cannot be written directly + in the generators.</p> + <p>To illustrate this, do <em>not</em> write as follows:</p> <code type="none"><![CDATA[ f(...) -> Y = ... diff --git a/system/doc/programming_examples/part.xml b/system/doc/programming_examples/part.xml index 0bec9b4cf5..9329717ce4 100644 --- a/system/doc/programming_examples/part.xml +++ b/system/doc/programming_examples/part.xml @@ -28,8 +28,9 @@ <rev></rev> </header> <description> - <p>This chapter contains examples on using records, funs, list - comprehensions and the bit syntax.</p> + <marker id="programming examples"></marker> + <p>This section contains examples on using records, funs, list + comprehensions, and the bit syntax.</p> </description> <xi:include href="records.xml"/> <xi:include href="funs.xml"/> diff --git a/system/doc/programming_examples/records.xml b/system/doc/programming_examples/records.xml index 58cf136a0b..ffcc05e758 100644 --- a/system/doc/programming_examples/records.xml +++ b/system/doc/programming_examples/records.xml @@ -30,37 +30,39 @@ </header> <section> - <title>Records vs Tuples</title> - <p>The main advantage of using records instead of tuples is that + <title>Records and Tuples</title> + <p>The main advantage of using records rather than tuples is that fields in a record are accessed by name, whereas fields in a tuple are accessed by position. To illustrate these differences, - suppose that we want to represent a person with the tuple + suppose that you want to represent a person with the tuple <c>{Name, Address, Phone}</c>.</p> - <p>We must remember that the <c>Name</c> field is the first - element of the tuple, the <c>Address</c> field is the second - element, and so on, in order to write functions which manipulate - this data. For example, to extract data from a variable <c>P</c> - which contains such a tuple we might write the following code - and then use pattern matching to extract the relevant fields.</p> + <p>To write functions that manipulate this data, remember the following:</p> + <list type="bulleted"> + <item>The <c>Name</c> field is the first element of the tuple.</item> + <item>The <c>Address</c> field is the second element.</item> + <item>The <c>Phone</c> field is the third element.</item> + </list> + <p>For example, to extract data from a variable <c>P</c> + that contains such a tuple, you can write the following code + and then use pattern matching to extract the relevant fields:</p> <code type="none"> Name = element(1, P), Address = element(2, P), ...</code> - <p>Code like this is difficult to read and understand and errors - occur if we get the numbering of the elements in the tuple wrong. - If we change the data representation by re-ordering the fields, - or by adding or removing a field, then all references to - the person tuple, wherever they occur, must be checked and - possibly modified.</p> - <p>Records allow us to refer to the fields by name and not - position. We use a record instead of a tuple to store the data. - If we write a record definition of the type shown below, we can - then refer to the fields of the record by name.</p> + <p>Such code is difficult to read and understand, and errors + occur if the numbering of the elements in the tuple is wrong. + If the data representation of the fields is changed, by re-ordering, + adding, or removing fields, all references to + the person tuple must be checked and possibly modified.</p> + <p>Records allow references to the fields by name, instead of by + position. In the following example, a record instead of a tuple + is used to store the data:</p> <code type="none"> -record(person, {name, phone, address}).</code> - <p>For example, if <c>P</c> is now a variable whose value is a - <c>person</c> record, we can code as follows in order to access - the name and address fields of the records.</p> + <p>This enables references to the fields of the record by name. + For example, if <c>P</c> is a variable whose value is a + <c>person</c> record, the following code access + the name and address fields of the records:</p> <code type="none"> Name = P#person.name, Address = P#person.address, @@ -72,24 +74,25 @@ Address = P#person.address, <section> <title>Defining a Record</title> - <p>This definition of a person will be used in many of - the examples which follow. It contains three fields, <c>name</c>, - <c>phone</c> and <c>address</c>. The default values for + <p>This following definition of a <c>person</c> is used in several + examples in this section. Three fields are included, <c>name</c>, + <c>phone</c>, and <c>address</c>. The default values for <c>name</c> and <c>phone</c> is "" and [], respectively. The default value for <c>address</c> is the atom <c>undefined</c>, since no default value is supplied for this field:</p> <pre> -record(person, {name = "", phone = [], address}).</pre> - <p>We have to define the record in the shell in order to be able - use the record syntax in the examples:</p> + <p>The record must be defined in the shell to enable + use of the record syntax in the examples:</p> <pre> > <input>rd(person, {name = "", phone = [], address}).</input> person</pre> - <p>This is due to the fact that record definitions are available - at compile time only, not at runtime. See <c>shell(3)</c> for - details on records in the shell. - </p> + <p>This is because record definitions are only available + at compile time, not at runtime. For details on records + in the shell, see the + <seealso marker="stdlib:shell">shell(3)</seealso> + manual page in <c>stdlib</c>.</p> </section> <section> @@ -98,12 +101,12 @@ person</pre> <pre> > <input>#person{phone=[0,8,2,3,4,3,1,2], name="Robert"}.</input> #person{name = "Robert",phone = [0,8,2,3,4,3,1,2],address = undefined}</pre> - <p>Since the <c>address</c> field was omitted, its default value + <p>As the <c>address</c> field was omitted, its default value is used.</p> - <p>There is a new feature introduced in Erlang 5.1/OTP R8B, - with which you can set a value to all fields in a record, - overriding the defaults in the record specification. The special - field <c>_</c>, means "all fields not explicitly specified".</p> + <p>From Erlang 5.1/OTP R8B, a value to all + fields in a record can be set with the special field <c>_</c>. + <c>_</c> means "all fields not explicitly specified".</p> + <p><em>Example:</em></p> <pre> > <input>#person{name = "Jakob", _ = '_'}.</input> #person{name = "Jakob",phone = '_',address = '_'}</pre> @@ -114,6 +117,7 @@ person</pre> <section> <title>Accessing a Record Field</title> + <p>The following example shows how to access a record field:</p> <pre> > <input>P = #person{name = "Joe", phone = [0,8,2,3,4,3,1,2]}.</input> #person{name = "Joe",phone = [0,8,2,3,4,3,1,2],address = undefined} @@ -123,6 +127,7 @@ person</pre> <section> <title>Updating a Record</title> + <p>The following example shows how to update a record:</p> <pre> > <input>P1 = #person{name="Joe", phone=[1,2,3], address="A street"}.</input> #person{name = "Joe",phone = [1,2,3],address = "A street"} @@ -133,7 +138,7 @@ person</pre> <section> <title>Type Testing</title> <p>The following example shows that the guard succeeds if - <c>P</c> is record of type <c>person</c>.</p> + <c>P</c> is record of type <c>person</c>:</p> <pre> foo(P) when is_record(P, person) -> a_person; foo(_) -> not_a_person.</pre> @@ -141,7 +146,7 @@ foo(_) -> not_a_person.</pre> <section> <title>Pattern Matching</title> - <p>Matching can be used in combination with records as shown in + <p>Matching can be used in combination with records, as shown in the following example:</p> <pre> > <input>P3 = #person{name="Joe", phone=[0,0,7], address="A street"}.</input> @@ -163,7 +168,7 @@ find_phone([], Name) -> <section> <title>Nested Records</title> - <p>The value of a field in a record might be an instance of a + <p>The value of a field in a record can be an instance of a record. Retrieval of nested data can be done stepwise, or in a single step, as shown in the following example:</p> <pre> @@ -173,11 +178,12 @@ find_phone([], Name) -> demo() -> P = #person{name= #name{first="Robert",last="Virding"}, phone=123}, First = (P#person.name)#name.first.</pre> - <p>In this example, <c>demo()</c> evaluates to <c>"Robert"</c>.</p> + <p>Here, <c>demo()</c> evaluates to <c>"Robert"</c>.</p> </section> <section> - <title>Example</title> + <title>A Longer Example</title> + <p>Comments are embedded in the following example:</p> <pre> %% File: person.hrl diff --git a/system/doc/reference_manual/character_set.xml b/system/doc/reference_manual/character_set.xml index b09b484582..d6989373bf 100644 --- a/system/doc/reference_manual/character_set.xml +++ b/system/doc/reference_manual/character_set.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2014</year><year>2014</year> + <year>2014</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -31,7 +31,7 @@ <section> <title>Character Set</title> - <p>In Erlang 4.8/OTP R5A the syntax of Erlang tokens was extended to + <p>Since Erlang 4.8/OTP R5A, the syntax of Erlang tokens is extended to allow the use of the full ISO-8859-1 (Latin-1) character set. This is noticeable in the following ways:</p> <list type="bulleted"> @@ -98,7 +98,7 @@ <cell align="center" valign="middle">ø - ÿ</cell> <cell align="left" valign="middle">Lowercase letters</cell> </row> - <tcaption>Character Classes.</tcaption> + <tcaption>Character Classes</tcaption> </table> <p>In Erlang/OTP R16B the syntax of Erlang tokens was extended to handle Unicode. The support is limited to @@ -111,13 +111,13 @@ </section> <section> <title>Source File Encoding</title> - <p>The Erlang source file <marker - id="encoding">encoding</marker> is selected by a + <marker id="encoding"></marker> + <p>The Erlang source file <c>encoding</c> is selected by a comment in one of the first two lines of the source file. The first string that matches the regular expression <c>coding\s*[:=]\s*([-a-zA-Z0-9])+</c> selects the encoding. If - the matching string is not a valid encoding it is ignored. The - valid encodings are <c>Latin-1</c> and <c>UTF-8</c> where the + the matching string is an invalid encoding, it is ignored. The + valid encodings are <c>Latin-1</c> and <c>UTF-8</c>, where the case of the characters can be chosen freely.</p> <p>The following example selects UTF-8 as default encoding:</p> <pre> @@ -127,7 +127,7 @@ %% For this file we have chosen encoding = Latin-1</pre> <pre> %% -*- coding: latin-1 -*-</pre> - <p>The default encoding for Erlang source files was changed from - Latin-1 to UTF-8 in Erlang OTP 17.0.</p> + <p>The default encoding for Erlang source files is changed from + Latin-1 to UTF-8 since Erlang/OTP 17.0.</p> </section> </chapter> diff --git a/system/doc/reference_manual/code_loading.xml b/system/doc/reference_manual/code_loading.xml index b5b5704df5..48ec32d6df 100644 --- a/system/doc/reference_manual/code_loading.xml +++ b/system/doc/reference_manual/code_loading.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2014</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -29,35 +29,39 @@ <file>code_loading.xml</file> </header> <p>How code is compiled and loaded is not a language issue, but - is system dependent. This chapter describes compilation and - code loading in Erlang/OTP with pointers to relevant parts of + is system-dependent. This section describes compilation and + code loading in Erlang/OTP with references to relevant parts of the documentation.</p> <section> <title>Compilation</title> <p>Erlang programs must be <em>compiled</em> to object code. - The compiler can generate a new file which contains the object - code. The current abstract machine which runs the object code is + The compiler can generate a new file that contains the object + code. The current abstract machine, which runs the object code, is called BEAM, therefore the object files get the suffix <c>.beam</c>. The compiler can also generate a binary which can be loaded directly.</p> - <p>The compiler is located in the Kernel module <c>compile</c>, see - <c>compile(3)</c>.</p> + <p>The compiler is located in the module <c>compile</c> (see the + <seealso marker="compiler:compile">compile(3)</seealso> manual page in + Compiler).</p> <pre> compile:file(Module) compile:file(Module, Options)</pre> <p>The Erlang shell understands the command <c>c(Module)</c> which both compiles and loads <c>Module</c>.</p> - <p>There is also a module <c>make</c> which provides a set of - functions similar to the UNIX type Make functions, see - <c>make(3)</c>.</p> - <p>The compiler can also be accessed from the OS prompt, see - <c>erl(1)</c>.</p> + <p>There is also a module <c>make</c>, which provides a set of + functions similar to the UNIX type Make functions, see the + <seealso marker="tools:make">make(3)</seealso> + manual page in Tools.</p> + <p>The compiler can also be accessed from the OS prompt, see the + <seealso marker="erts:erl">erl(1)</seealso> manual page in ERTS.</p> <pre> % erl -compile <input>Module1</input>...<input>ModuleN</input> % erl -make</pre> <p>The <c>erlc</c> program provides an even better way to compile - modules from the shell, see <c>erlc(1)</c>. It understands a + modules from the shell, see the + <seealso marker="erts:erlc">erlc(1)</seealso> manual page in ERTS. + It understands a number of flags that can be used to define macros, add search paths for include files, and more.</p> <pre> @@ -68,13 +72,17 @@ compile:file(Module, Options)</pre> <marker id="loading"></marker> <title>Code Loading</title> <p>The object code must be <em>loaded</em> into the Erlang runtime - system. This is handled by the <em>code server</em>, see - <c>code(3)</c>.</p> - <p>The code server loads code according to a code loading strategy + system. This is handled by the <em>code server</em>, see the + <seealso marker="kernel:code">code(3)</seealso> + manual page in Kernel.</p> + <p>The code server loads code according to a code loading strategy, which is either <em>interactive</em> (default) or - <em>embedded</em>. In interactive mode, code are searched for in + <em>embedded</em>. In interactive mode, code is searched for in a <em>code path</em> and loaded when first referenced. In - embedded mode, code is loaded at start-up according to a <em>boot script</em>. This is described in <em>System Principles</em>.</p> + embedded mode, code is loaded at start-up according to a + <em>boot script</em>. This is described in + <seealso marker="doc/system_principles:system_principles#code_loading"> + System Principles </seealso>.</p> </section> <section> @@ -86,16 +94,17 @@ compile:file(Module, Options)</pre> the system for the first time, the code becomes 'current'. If then a new instance of the module is loaded, the code of the previous instance becomes 'old' and the new instance becomes 'current'.</p> - <p>Both old and current code is valid, and may be evaluated + <p>Both old and current code is valid, and can be evaluated concurrently. Fully qualified function calls always refer to - current code. Old code may still be evaluated because of processes + current code. Old code can still be evaluated because of processes lingering in the old code.</p> - <p>If a third instance of the module is loaded, the code server will - remove (purge) the old code and any processes lingering in it will - be terminated. Then the third instance becomes 'current' and + <p>If a third instance of the module is loaded, the code server + removes (purges) the old code and any processes lingering in it is + terminated. Then the third instance becomes 'current' and the previously current code becomes 'old'.</p> <p>To change from old code to current code, a process must make a - fully qualified function call. Example:</p> + fully qualified function call.</p> + <p><em>Example:</em></p> <pre> -module(m). -export([loop/0]). @@ -109,60 +118,62 @@ loop() -> loop() end.</pre> <p>To make the process change code, send the message - <c>code_switch</c> to it. The process then will make a fully - qualified call to <c>m:loop()</c> and change to current code. - Note that <c>m:loop/0</c> must be exported.</p> - <p>For code replacement of funs to work, the syntax - <c>fun Module:FunctionName/Arity</c> should be used.</p> + <c>code_switch</c> to it. The process then makes a fully + qualified call to <c>m:loop()</c> and changes to current code. + Notice that <c>m:loop/0</c> must be exported.</p> + <p>For code replacement of funs to work, use the syntax + <c>fun Module:FunctionName/Arity</c>.</p> </section> <section> <marker id="on_load"></marker> - <title>Running a function when a module is loaded</title> + <title>Running a Function When a Module is Loaded</title> <warning> - <p>The <c>on_load</c> feature should be considered experimental - as there are a number of known weak points in current semantics - which therefore might also change in future releases:</p> + <p>The <c>on_load</c> feature is to be considered experimental + as there are a number of known weak points in current semantics, + which therefore might change in future Erlang/OTP releases:</p> <list> - <item><p>Doing external call in on_load to the module itself + <item><p>Doing external call in <c>on_load</c> to the module itself leads to deadlock.</p></item> <item><p>At module upgrade, other processes calling the module - get suspended waiting for on_load to finish. This can be very bad + get suspended waiting for <c>on_load</c> to finish. This can be very bad for applications with demands on realtime characteristics.</p></item> - <item><p>At module upgrade, no rollback is done if the on_load function fails. - The system will be left in a bad limbo state without any working + <item><p>At module upgrade, no rollback is done if the + <c>on_load</c> function fails. + The system is left in a bad limbo state without any working and reachable instance of the module.</p></item> </list> - <p>The problems with module upgrade described above could be fixed in future - releases by changing the behaviour to not make the module reachable until - after the on_load function has successfully returned.</p> + <p>The problems with module upgrade described above can be fixed in future + Erlang/OTP releases by changing the behaviour to not make the module reachable until + after the <c>on_load</c> function has successfully returned.</p> </warning> - <p>The <c>-on_load()</c> directive names a function that should - be run automatically when a module a loaded. Its syntax is:</p> + <p>The <c>-on_load()</c> directive names a function that is to + be run automatically when a module is loaded.</p> + <p>Its syntax is as follows:</p> <pre> -on_load(Name/0).</pre> - <p>It is not necessary to export the function. It will be called in a - freshly spawned process (which will be terminated as soon as the function + <p>It is not necessary to export the function. It is called in a + freshly spawned process (which terminates as soon as the function returns). The function must return <c>ok</c> if the module is to - be remained loaded and become callable, or any other value if the module - is to be unloaded. Generating an exception will also cause the + remain loaded and become callable, or any other value if the module + is to be unloaded. Generating an exception also causes the module to be unloaded. If the return value is not an atom, - a warning error report will be sent to the error logger.</p> + a warning error report is sent to the error logger.</p> <p>A process that calls any function in a module whose <c>on_load</c> - function has not yet returned will be suspended until the <c>on_load</c> + function has not yet returned, is suspended until the <c>on_load</c> function has returned.</p> - <p>In embedded mode, all modules will be loaded first and then - will all on_load functions be called. The system will be - terminated unless all of the on_load functions return + <p>In embedded mode, first all modules are loaded. + Then all <c>on_load</c> functions are called. The system is + terminated unless all of the <c>on_load</c> functions return <c>ok</c></p>. - <p>Example:</p> + <p><em>Example:</em></p> <pre> -module(m). @@ -174,7 +185,7 @@ load_my_nifs() -> erlang:load_nif(NifPath, Info).</pre> <p>If the call to <c>erlang:load_nif/2</c> fails, the module - will be unloaded and there will be warning report sent to + is unloaded and a warning report is sent to the error loader.</p> </section> diff --git a/system/doc/reference_manual/data_types.xml b/system/doc/reference_manual/data_types.xml index ad92143179..6226fa2f31 100644 --- a/system/doc/reference_manual/data_types.xml +++ b/system/doc/reference_manual/data_types.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,12 +28,12 @@ <rev></rev> <file>data_types.xml</file> </header> + <p>Erlang provides a number of data types, which are listed in + this section.</p> <section> <title>Terms</title> - <p>Erlang provides a number of data types which are listed in this - chapter. A piece of data of any data type is called a - <em>term</em>.</p> + <p>A piece of data of any data type is called a <em>term</em>.</p> </section> <section> @@ -48,13 +48,13 @@ <em><c>char</c></em>.</item> <item><em><c>base</c></em><c>#</c><em><c>value</c></em> <br></br> - Integer with the base <em><c>base</c></em>, which must be an + Integer with the base <em><c>base</c></em>, that must be an integer in the range 2..36. <br></br> In Erlang 5.2/OTP R9B and earlier versions, the allowed range is 2..16.</item> </list> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>42.</input> 42 @@ -76,11 +76,11 @@ <section> <title>Atom</title> - <p>An atom is a literal, a constant with name. An atom should be + <p>An atom is a literal, a constant with name. An atom is to be enclosed in single quotes (') if it does not begin with a lower-case letter or if it contains other characters than alphanumeric characters, underscore (_), or @.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> hello phone_number @@ -91,11 +91,11 @@ phone_number <section> <title>Bit Strings and Binaries</title> <p>A bit string is used to store an area of untyped memory.</p> - <p>Bit Strings are expressed using the + <p>Bit strings are expressed using the <seealso marker="expressions#bit_syntax">bit syntax</seealso>.</p> - <p>Bit Strings which consists of a number of bits which is evenly - divisible by eight are called Binaries</p> - <p>Examples:</p> + <p>Bit strings that consist of a number of bits that are evenly + divisible by eight, are called <em>binaries</em></p> + <p><em>Examples:</em></p> <pre> 1> <input><<10,20>>.</input> <<10,20>> @@ -103,12 +103,14 @@ phone_number <<"ABC">> 1> <input><<1:1,0:1>>.</input> <<2:2>></pre> - <p>More examples can be found in Programming Examples.</p> + <p>For more examples, + see <seealso marker="doc/programming_examples:bit_syntax"> + Programming Examples</seealso>.</p> </section> <section> <title>Reference</title> - <p>A reference is a term which is unique in an Erlang runtime + <p>A reference is a term that is unique in an Erlang runtime system, created by calling <c>make_ref/0</c>.</p> </section> @@ -117,34 +119,42 @@ phone_number <p>A fun is a functional object. Funs make it possible to create an anonymous function and pass the function itself -- not its name -- as argument to other functions.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> 1> <input>Fun1 = fun (X) -> X+1 end.</input> #Fun<erl_eval.6.39074546> 2> <input>Fun1(2).</input> 3</pre> - <p>Read more about funs in <seealso marker="expressions#funs">Fun Expressions</seealso>. More examples can be found in Programming - Examples.</p> + <p>Read more about funs in <seealso marker="expressions#funs"> + Fun Expressions</seealso>. For more examples, see + <seealso marker="doc/programming_examples:funs"> + Programming Examples</seealso>.</p> </section> <section> <title>Port Identifier</title> - <p>A port identifier identifies an Erlang port. <c>open_port/2</c>, - which is used to create ports, will return a value of this type.</p> + <p>A port identifier identifies an Erlang port.</p> + <p><c>open_port/2</c>, which is used to create ports, returns + a value of this data type.</p> <p>Read more about ports in <seealso marker="ports">Ports and Port Drivers</seealso>.</p> </section> <section> <title>Pid</title> - <p>A process identifier, pid, identifies a process. - <c>spawn/1,2,3,4</c>, <c>spawn_link/1,2,3,4</c> and - <c>spawn_opt/4</c>, which are used to create processes, return - values of this type. Example:</p> + <p>A process identifier, pid, identifies a process.</p> + <p>The following BIFs, which are used to create processes, return + values of this data type:</p> + <list type="bulleted"> + <item><c>spawn/1,2,3,4</c></item> + <item><c>spawn_link/1,2,3,4</c></item> + <item><c>spawn_opt/4</c></item> + </list> + <p><em>Example:</em></p> <pre> 1> <input>spawn(m, f, []).</input> <0.51.0></pre> - <p>The BIF <c>self()</c> returns the pid of the calling process. - Example:</p> + <p>In the following example, the BIF <c>self()</c> returns + the pid of the calling process:</p> <pre> -module(m). -export([loop/0]). @@ -167,14 +177,14 @@ who_are_you</pre> <section> <title>Tuple</title> - <p>Compound data type with a fixed number of terms:</p> + <p>A tuple is a compound data type with a fixed number of terms:</p> <pre> {Term1,...,TermN}</pre> <p>Each term <c>Term</c> in the tuple is called an <em>element</em>. The number of elements is said to be the <em>size</em> of the tuple.</p> <p>There exists a number of BIFs to manipulate tuples.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>P = {adam,24,{july,29}}.</input> {adam,24,{july,29}} @@ -192,7 +202,8 @@ adam <section> <title>Map</title> - <p>Compound data type with a variable number of key-value associations:</p> + <p>A map is a compound data type with a variable number of + key-value associations:</p> <pre> #{Key1=>Value1,...,KeyN=>ValueN}</pre> <p>Each key-value association in the map is called an @@ -200,7 +211,7 @@ adam called <em>elements</em>. The number of association pairs is said to be the <em>size</em> of the map.</p> <p>There exists a number of BIFs to manipulate maps.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>M1 = #{name=>adam,age=>24,date=>{july,29}}.</input> #{age => 24,date => {july,29},name => adam} @@ -215,16 +226,18 @@ adam 6> <input>map_size(#{}).</input> 0</pre> <p>A collection of maps processing functions can be found in - the STDLIB module <seealso marker="stdlib:maps"><c>maps</c></seealso>.</p> - <p>Read more about <seealso marker="expressions#map_expressions">Maps</seealso>.</p> + <seealso marker="stdlib:maps"><c>maps</c></seealso> manual page + in STDLIB.</p> + <p>Read more about maps in <seealso marker="expressions#map_expressions"> + Map Expressions</seealso>.</p> <note> - <p>Maps are considered experimental during OTP 17.</p> + <p>Maps are considered to be experimental during Erlang/OTP R17.</p> </note> </section> <section> <title>List</title> - <p>Compound data type with a variable number of terms.</p> + <p>A list is a compound data type with a variable number of terms.</p> <pre> [Term1,...,TermN]</pre> <p>Each term <c>Term</c> in the list is called an @@ -232,20 +245,21 @@ adam the <em>length</em> of the list.</p> <p>Formally, a list is either the empty list <c>[]</c> or consists of a <em>head</em> (first element) and a <em>tail</em> - (remainder of the list) which is also a list. The latter can + (remainder of the list). + The <em>tail</em> is also a list. The latter can be expressed as <c>[H|T]</c>. The notation - <c>[Term1,...,TermN]</c> above is actually shorthand for + <c>[Term1,...,TermN]</c> above is equivalent with the list <c>[Term1|[...|[TermN|[]]]]</c>.</p> - <p>Example: <br></br> -<c>[]</c> is a list, thus <br></br> + <p><em>Example:</em></p> +<p><c>[]</c> is a list, thus <br></br> <c>[c|[]]</c> is a list, thus <br></br> <c>[b|[c|[]]]</c> is a list, thus <br></br> -<c>[a|[b|[c|[]]]]</c> is a list, or in short <c>[a,b,c]</c>.</p> - <p></p> +<c>[a|[b|[c|[]]]]</c> is a list, or in short <c>[a,b,c]</c></p> + <p>A list where the tail is a list is sometimes called a <em>proper list</em>. It is allowed to have a list where the tail is not a - list, for example <c>[a|b]</c>. However, this type of list is of + list, for example, <c>[a|b]</c>. However, this type of list is of little practical use.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>L1 = [a,2,{c,4}].</input> [a,2,{c,4}] @@ -262,18 +276,19 @@ a 7> <input>length([]).</input> 0</pre> <p>A collection of list processing functions can be found in - the STDLIB module <c>lists</c>.</p> + the <seealso marker="stdlib:lists">lists</seealso> manual + page in STDLIB.</p> </section> <section> <title>String</title> <p>Strings are enclosed in double quotes ("), but is not a - data type in Erlang. Instead a string <c>"hello"</c> is - shorthand for the list <c>[$h,$e,$l,$l,$o]</c>, that is + data type in Erlang. Instead, a string <c>"hello"</c> is + shorthand for the list <c>[$h,$e,$l,$l,$o]</c>, that is, <c>[104,101,108,108,111]</c>.</p> <p>Two adjacent string literals are concatenated into one. This is - done at compile-time and does not incur any runtime overhead. - Example:</p> + done in the compilation, thus, does not incur any runtime overhead.</p> + <p><em>Example:</em></p> <pre> "string" "42"</pre> <p>is equivalent to</p> @@ -285,12 +300,13 @@ a <title>Record</title> <p>A record is a data structure for storing a fixed number of elements. It has named fields and is similar to a struct in C. - However, record is not a true data type. Instead record + However, a record is not a true data type. Instead, record expressions are translated to tuple expressions during compilation. Therefore, record expressions are not understood by - the shell unless special actions are taken. See <c>shell(3)</c> - for details.</p> - <p>Examples:</p> + the shell unless special actions are taken. For details, see the + <seealso marker="stdlib:shell">shell(3)</seealso> manual + page in STDLIB).</p> + <p><em>Examples:</em></p> <pre> -module(person). -export([new/2]). @@ -304,14 +320,15 @@ new(Name, Age) -> {person,ernie,44}</pre> <p>Read more about records in <seealso marker="records">Records</seealso>. More examples can be - found in Programming Examples.</p> + found in <seealso marker="doc/programming_examples:records"> + Programming Examples</seealso>.</p> </section> <section> <title>Boolean</title> <p>There is no Boolean data type in Erlang. Instead the atoms <c>true</c> and <c>false</c> are used to denote Boolean values.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>2 =< 3</input>. true @@ -330,76 +347,80 @@ true</pre> </row> <row> <cell align="left" valign="middle">\b</cell> - <cell align="left" valign="middle">backspace</cell> + <cell align="left" valign="middle">Backspace</cell> </row> <row> <cell align="left" valign="middle">\d</cell> - <cell align="left" valign="middle">delete</cell> + <cell align="left" valign="middle">Delete</cell> </row> <row> <cell align="left" valign="middle">\e</cell> - <cell align="left" valign="middle">escape</cell> + <cell align="left" valign="middle">Escape</cell> </row> <row> <cell align="left" valign="middle">\f</cell> - <cell align="left" valign="middle">form feed</cell> + <cell align="left" valign="middle">Form feed</cell> </row> <row> <cell align="left" valign="middle">\n</cell> - <cell align="left" valign="middle">newline</cell> + <cell align="left" valign="middle">Newline</cell> </row> <row> <cell align="left" valign="middle">\r</cell> - <cell align="left" valign="middle">carriage return</cell> + <cell align="left" valign="middle">Carriage return</cell> </row> <row> <cell align="left" valign="middle">\s</cell> - <cell align="left" valign="middle">space</cell> + <cell align="left" valign="middle">Space</cell> </row> <row> <cell align="left" valign="middle">\t</cell> - <cell align="left" valign="middle">tab</cell> + <cell align="left" valign="middle">Tab</cell> </row> <row> <cell align="left" valign="middle">\v</cell> - <cell align="left" valign="middle">vertical tab</cell> + <cell align="left" valign="middle">Vertical tab</cell> </row> <row> <cell align="left" valign="middle">\XYZ, \YZ, \Z</cell> - <cell align="left" valign="middle">character with octal representation XYZ, YZ or Z</cell> + <cell align="left" valign="middle">Character with octal + representation XYZ, YZ or Z</cell> </row> <row> <cell align="left" valign="middle">\xXY</cell> - <cell align="left" valign="middle">character with hexadecimal representation XY</cell> + <cell align="left" valign="middle">Character with hexadecimal + representation XY</cell> </row> <row> <cell align="left" valign="middle">\x{X...}</cell> - <cell align="left" valign="middle">character with hexadecimal representation; X... is one or more hexadecimal characters</cell> + <cell align="left" valign="middle">Character with hexadecimal + representation; X... is one or more hexadecimal characters</cell> </row> <row> <cell align="left" valign="middle">\^a...\^z <br></br> \^A...\^Z</cell> - <cell align="left" valign="middle">control A to control Z</cell> + <cell align="left" valign="middle">Control A to control Z</cell> </row> <row> <cell align="left" valign="middle">\'</cell> - <cell align="left" valign="middle">single quote</cell> + <cell align="left" valign="middle">Single quote</cell> </row> <row> <cell align="left" valign="middle">\"</cell> - <cell align="left" valign="middle">double quote</cell> + <cell align="left" valign="middle">Double quote</cell> </row> <row> <cell align="left" valign="middle">\\</cell> - <cell align="left" valign="middle">backslash</cell> + <cell align="left" valign="middle">Backslash</cell> </row> - <tcaption>Recognized Escape Sequences.</tcaption> + <tcaption>Recognized Escape Sequences</tcaption> </table> </section> <section> <title>Type Conversions</title> - <p>There are a number of BIFs for type conversions. Examples:</p> + <p>There are a number of BIFs for type conversions.</p> + <p><em>Examples:</em></p> <pre> 1> <input>atom_to_list(hello).</input> "hello" diff --git a/system/doc/reference_manual/distributed.xml b/system/doc/reference_manual/distributed.xml index 88f98bc106..fb83e356f9 100644 --- a/system/doc/reference_manual/distributed.xml +++ b/system/doc/reference_manual/distributed.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -36,22 +36,24 @@ runtime system is called a <em>node</em>. Message passing between processes at different nodes, as well as links and monitors, are transparent when pids are used. Registered names, however, are - local to each node. This means the node must be specified as well - when sending messages etc. using registered names.</p> + local to each node. This means that the node must be specified as well + when sending messages, and so on, using registered names.</p> <p>The distribution mechanism is implemented using TCP/IP sockets. - How to implement an alternative carrier is described in <em>ERTS User's Guide</em>.</p> + How to implement an alternative carrier is described in the + <seealso marker="erts:alt_dist">ERTS User's Guide</seealso>.</p> </section> <section> <title>Nodes</title> - <p>A <em>node</em> is an executing Erlang runtime system which has - been given a name, using the command line flag <c>-name</c> + <p>A <em>node</em> is an executing Erlang runtime system that has + been given a name, using the command-line flag <c>-name</c> (long names) or <c>-sname</c> (short names).</p> - <p>The format of the node name is an atom <c>name@host</c> where - <c>name</c> is the name given by the user and <c>host</c> is + <p>The format of the node name is an atom <c>name@host</c>. + <c>name</c> is the name given by the user. <c>host</c> is the full host name if long names are used, or the first part of the host name if short names are used. <c>node()</c> returns - the name of the node. Example:</p> + the name of the node.</p> + <p><em>Example:</em></p> <pre> % <input>erl -name dilbert</input> ([email protected])1> <input>node().</input> @@ -69,16 +71,16 @@ dilbert@uab</pre> <section> <title>Node Connections</title> <p>The nodes in a distributed Erlang system are loosely connected. - The first time the name of another node is used, for example if + The first time the name of another node is used, for example, if <c>spawn(Node,M,F,A)</c> or <c>net_adm:ping(Node)</c> is called, - a connection attempt to that node will be made.</p> + a connection attempt to that node is made.</p> <p>Connections are by default transitive. If a node A connects to - node B, and node B has a connection to node C, then node A will - also try to connect to node C. This feature can be turned off by - using the command line flag <c>-connect_all false</c>, see - <c>erl(1)</c>.</p> + node B, and node B has a connection to node C, then node A + also tries to connect to node C. This feature can be turned off by + using the command-line flag <c>-connect_all false</c>, see the + <seealso marker="erts:erl">erl(1)</seealso> manual page in ERTS.</p> <p>If a node goes down, all connections to that node are removed. - Calling <c>erlang:disconnect_node(Node)</c> will force disconnection + Calling <c>erlang:disconnect_node(Node)</c> forces disconnection of a node.</p> <p>The list of (visible) nodes currently connected to is returned by <c>nodes()</c>.</p> @@ -89,23 +91,24 @@ dilbert@uab</pre> <p>The Erlang Port Mapper Daemon <em>epmd</em> is automatically started at every host where an Erlang node is started. It is responsible for mapping the symbolic node names to machine - addresses. See <c>epmd(1)</c>.</p> + addresses. See the + <seealso marker="erts:epmd">epmd(1)</seealso> manual page in ERTS.</p> </section> <section> <title>Hidden Nodes</title> <p>In a distributed Erlang system, it is sometimes useful to connect to a node without also connecting to all other nodes. - An example could be some kind of O&M functionality used to - inspect the status of a system without disturbing it. For this - purpose, a <em>hidden node</em> may be used.</p> - <p>A hidden node is a node started with the command line flag + An example is some kind of O&M functionality used to + inspect the status of a system, without disturbing it. For this + purpose, a <em>hidden node</em> can be used.</p> + <p>A hidden node is a node started with the command-line flag <c>-hidden</c>. Connections between hidden nodes and other nodes are not transitive, they must be set up explicitly. Also, hidden nodes does not show up in the list of nodes returned by <c>nodes()</c>. Instead, <c>nodes(hidden)</c> or <c>nodes(connected)</c> must be used. This means, for example, - that the hidden node will not be added to the set of nodes that + that the hidden node is not added to the set of nodes that <c>global</c> is keeping track of.</p> <p>This feature was added in Erlang 5.0/OTP R7.</p> </section> @@ -114,9 +117,11 @@ dilbert@uab</pre> <title>C Nodes</title> <p>A <em>C node</em> is a C program written to act as a hidden node in a distributed Erlang system. The library <em>Erl_Interface</em> - contains functions for this purpose. Refer to the documentation - for Erl_Interface and <em>Interoperability Tutorial</em> for more - information about C nodes.</p> + contains functions for this purpose. For more information about + C nodes, see the <seealso marker="erl_interface:ei_users_guide"> + Erl_Interface</seealso> application and + <seealso marker="doc/tutorial:introduction#interoperability tutorial"> + Interoperability Tutorial.</seealso>.</p> </section> <section> @@ -125,7 +130,7 @@ dilbert@uab</pre> with each other. In a network of different Erlang nodes, it is built into the system at the lowest possible level. Each node has its own <em>magic cookie</em>, which is an Erlang atom.</p> - <p>When a nodes tries to connect to another node, the magic cookies + <p>When a node tries to connect to another node, the magic cookies are compared. If they do not match, the connected node rejects the connection.</p> <p>At start-up, a node has a random atom assigned as its magic @@ -141,8 +146,8 @@ dilbert@uab</pre> the local node assume that all other nodes have the same cookie <c>Cookie</c>.</p> <p>Thus, groups of users with identical cookie files get Erlang - nodes which can communicate freely and without interference from - the magic cookie system. Users who want run nodes on separate + nodes that can communicate freely and without interference from + the magic cookie system. Users who want to run nodes on separate file systems must make certain that their cookie files are identical on the different file systems.</p> <p>For a node <c>Node1</c> with magic cookie <c>Cookie</c> to be @@ -154,19 +159,25 @@ dilbert@uab</pre> <p>The default when a connection is established between two nodes, is to immediately connect all other visible nodes as well. This way, there is always a fully connected network. If there are - nodes with different cookies, this method might be inappropriate - and the command line flag <c>-connect_all false</c> must be set, - see <seealso marker="erts:erl">erl(1)</seealso>.</p> + nodes with different cookies, this method can be inappropriate + and the command-line flag <c>-connect_all false</c> must be set, + see the <seealso marker="erts:erl">erl(1)</seealso> + manual page in ERTS.</p> <p>The magic cookie of the local node is retrieved by calling <c>erlang:get_cookie()</c>.</p> </section> <section> <title>Distribution BIFs</title> - <p>Some useful BIFs for distributed programming, see - <c>erlang(3)</c> for more information:</p> + <p>Some useful BIFs for distributed programming + (for more information, see the <seealso marker="erts:erlang"> + erlang(3)</seealso> manual page in ERTS:</p> <table> <row> + <cell align="left" valign="middle"><em>BIF</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>erlang:disconnect_node(Node)</c></cell> <cell align="left" valign="middle">Forces the disconnection of a node.</cell> </row> @@ -180,7 +191,9 @@ dilbert@uab</pre> </row> <row> <cell align="left" valign="middle"><c>monitor_node(Node, true|false)</c></cell> - <cell align="left" valign="middle">Monitor the status of <c>Node</c>. A message<c>{nodedown, Node}</c> is received if the connection to it is lost.</cell> + <cell align="left" valign="middle">Monitors the status of + <c>Node</c>. A message<c>{nodedown, Node}</c> is received + if the connection to it is lost.</cell> </row> <row> <cell align="left" valign="middle"><c>node()</c></cell> @@ -196,11 +209,16 @@ dilbert@uab</pre> </row> <row> <cell align="left" valign="middle"><c>nodes(Arg)</c></cell> - <cell align="left" valign="middle">Depending on <c>Arg</c>, this function can return a list not only of visible nodes, but also hidden nodes and previously known nodes, etc.</cell> + <cell align="left" valign="middle">Depending on <c>Arg</c>, + this function can return a list not only of visible nodes, + but also hidden nodes and previously known nodes, and so on.</cell> </row> <row> <cell align="left" valign="middle"><c>erlang:set_cookie(Node, Cookie)</c></cell> - <cell align="left" valign="middle">Sets the magic cookie used when connecting to <c>Node</c>. If <c>Node</c> is the current node, <c>Cookie</c> will be used when connecting to all new nodes.</cell> + <cell align="left" valign="middle">Sets the magic cookie used + when connecting to <c>Node</c>. If <c>Node</c> is the + current node, <c>Cookie</c> is used when connecting to + all new nodes.</cell> </row> <row> <cell align="left" valign="middle"><c>spawn[_link|_opt](Node, Fun)</c></cell> @@ -210,18 +228,24 @@ dilbert@uab</pre> <cell align="left" valign="middle"><c>spawn[_link|opt](Node, Module, FunctionName, Args)</c></cell> <cell align="left" valign="middle">Creates a process at a remote node.</cell> </row> - <tcaption>Distribution BIFs.</tcaption> + <tcaption>Distribution BIFs</tcaption> </table> </section> <section> - <title>Distribution Command Line Flags</title> - <p>Examples of command line flags used for distributed programming, - see <c>erl(1)</c> for more information:</p> + <title>Distribution Command-Line Flags</title> + <p>Examples of command-line flags used for distributed programming + (for more information, see the <seealso marker="erts:erl">erl(1) + </seealso> manual page in ERTS:</p> <table> <row> + <cell align="left" valign="middle"><em>Command-Line Flag</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>-connect_all false</c></cell> - <cell align="left" valign="middle">Only explicit connection set-ups will be used.</cell> + <cell align="left" valign="middle">Only explicit connection + set-ups are used.</cell> </row> <row> <cell align="left" valign="middle"><c>-hidden</c></cell> @@ -239,16 +263,20 @@ dilbert@uab</pre> <cell align="left" valign="middle"><c>-sname Name</c></cell> <cell align="left" valign="middle">Makes a runtime system into a node, using short node names.</cell> </row> - <tcaption>Distribution Command Line Flags.</tcaption> + <tcaption>Distribution Command-Line Flags</tcaption> </table> </section> <section> <title>Distribution Modules</title> <p>Examples of modules useful for distributed programming:</p> - <p>In Kernel:</p> + <p>In the Kernel application:</p> <table> <row> + <cell align="left" valign="middle"><em>Module</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>global</c></cell> <cell align="left" valign="middle">A global name registration facility.</cell> </row> @@ -266,9 +294,13 @@ dilbert@uab</pre> </row> <tcaption>Kernel Modules Useful For Distribution.</tcaption> </table> - <p>In STDLIB:</p> + <p>In the STDLIB application:</p> <table> <row> + <cell align="left" valign="middle"><em>Module</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>slave</c></cell> <cell align="left" valign="middle">Start and control of slave nodes.</cell> </row> diff --git a/system/doc/reference_manual/errors.xml b/system/doc/reference_manual/errors.xml index dde6e68f4a..66ecf6aa94 100644 --- a/system/doc/reference_manual/errors.xml +++ b/system/doc/reference_manual/errors.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -38,12 +38,12 @@ <item>Run-time errors</item> <item>Generated errors</item> </list> - <p>A compile-time error, for example a syntax error, should not + <p>A compile-time error, for example a syntax error, does not cause much trouble as it is caught by the compiler.</p> <p>A logical error is when a program does not behave as intended, - but does not crash. An example could be that nothing happens when + but does not crash. An example is that nothing happens when a button in a graphical user interface is clicked.</p> - <p>A run-time error is when a crash occurs. An example could be + <p>A run-time error is when a crash occurs. An example is when an operator is applied to arguments of the wrong type. The Erlang programming language has built-in features for handling of run-time errors.</p> @@ -54,23 +54,23 @@ of class <c>error</c>. </p> <p>A generated error is when the code itself calls - <c>exit/1</c> or <c>throw/1</c>. Note that emulated run-time + <c>exit/1</c> or <c>throw/1</c>. Notice that emulated run-time errors are not denoted as generated errors here. </p> <p>Generated errors are exceptions of classes <c>exit</c> and <c>throw</c>. </p> <p>When a run-time error or generated error occurs in Erlang, - execution for the process which evaluated + execution for the process that evaluated the erroneous expression is stopped. This is referred to as a <em>failure</em>, that execution or evaluation <em>fails</em>, or that the process <em>fails</em>, - <em>terminates</em> or <em>exits</em>. Note that a process may + <em>terminates</em>, or <em>exits</em>. Notice that a process can terminate/exit for other reasons than a failure.</p> - <p>A process that terminates will emit an <em>exit signal</em> with + <p>A process that terminates emits an <em>exit signal</em> with an <em>exit reason</em> that says something about which error - has occurred. Normally, some information about the error will - be printed to the terminal.</p> + has occurred. Normally, some information about the error is + printed to the terminal.</p> </section> <section> @@ -78,10 +78,12 @@ <p>Exceptions are run-time errors or generated errors and are of three different classes, with different origins. The <seealso marker="expressions#try">try</seealso> expression - (appeared in Erlang 5.4/OTP-R10B) + (new in Erlang 5.4/OTP R10B) can distinguish between the different classes, whereas the <seealso marker="expressions#catch">catch</seealso> - expression can not. They are described in the Expressions chapter.</p> + expression cannot. They are described in + <seealso marker="expressions">Expressions + </seealso>.</p> <table> <row> <cell align="left" valign="middle"><em>Class</em></cell> @@ -89,7 +91,9 @@ </row> <row> <cell align="left" valign="middle"><c>error</c></cell> - <cell align="left" valign="middle">Run-time error for example <c>1+a</c>, or the process called <c>erlang:error/1,2</c> (appeared in Erlang 5.4/OTP-R10B)</cell> + <cell align="left" valign="middle">Run-time error, + for example, <c>1+a</c>, or the process called + <c>erlang:error/1,2</c> (new in Erlang 5.4/OTP R10B)</cell> </row> <row> <cell align="left" valign="middle"><c>exit</c></cell> @@ -102,11 +106,11 @@ <tcaption>Exception Classes.</tcaption> </table> <p>An exception consists of its class, an exit reason - (the <seealso marker="#exit_reasons">Exit Reason</seealso>), - and a stack trace (that aids in finding the code location of + (see <seealso marker="#exit_reasons">Exit Reason</seealso>), + and a stack trace (which aids in finding the code location of the exception).</p> <p>The stack trace can be retrieved using - <c>erlang:get_stacktrace/0</c> (new in Erlang 5.4/OTP-R10B) + <c>erlang:get_stacktrace/0</c> (new in Erlang 5.4/OTP R10B) from within a <c>try</c> expression, and is returned for exceptions of class <c>error</c> from a <c>catch</c> expression.</p> <p>An exception of class <c>error</c> is also known as a run-time @@ -114,38 +118,38 @@ </section> <section> - <title>Handling of Run-Time Errors in Erlang</title> + <title>Handling of Run-time Errors in Erlang</title> <section> <title>Error Handling Within Processes</title> <p>It is possible to prevent run-time errors and other exceptions from causing the process to terminate by using <c>catch</c> or - <c>try</c>, see the Expressions chapter about - <seealso marker="expressions#catch">Catch</seealso> - and <seealso marker="expressions#try">Try</seealso>.</p> + <c>try</c>, see <seealso marker="expressions"> + Expressions</seealso> about + <seealso marker="expressions#catch">catch</seealso> + and <seealso marker="expressions#try">try</seealso>.</p> </section> <section> <title>Error Handling Between Processes</title> <p>Processes can monitor other processes and detect process terminations, see - the <seealso marker="processes#errors">Processes</seealso> - chapter.</p> + <seealso marker="processes#errors">Processes</seealso>.</p> </section> </section> <section> <marker id="exit_reasons"></marker> <title>Exit Reasons</title> - <p>When a run-time error occurs, - that is an exception of class <c>error</c>, - the exit reason is a tuple <c>{Reason,Stack}</c>. + <p>When a run-time error occurs, + that is an exception of class <c>error</c>. + The exit reason is a tuple <c>{Reason,Stack}</c>, where <c>Reason</c> is a term indicating the type of error:</p> <table> <row> <cell align="left" valign="middle"><em>Reason</em></cell> - <cell align="left" valign="middle"><em>Type of error</em></cell> + <cell align="left" valign="middle"><em>Type of Error</em></cell> </row> <row> <cell align="left" valign="middle"><c>badarg</c></cell> @@ -181,7 +185,7 @@ </row> <row> <cell align="left" valign="middle"><c>{badfun,F}</c></cell> - <cell align="left" valign="middle">There is something wrong with a fun <c>F</c>.</cell> + <cell align="left" valign="middle">Something is wrong with a fun <c>F</c>.</cell> </row> <row> <cell align="left" valign="middle"><c>{badarity,F}</c></cell> @@ -201,14 +205,17 @@ </row> <row> <cell align="left" valign="middle"><c>system_limit</c></cell> - <cell align="left" valign="middle">A system limit has been reached. See Efficiency Guide for information about system limits.</cell> + <cell align="left" valign="middle">A system limit has been reached. + See <seealso marker="doc/efficiency_guide:advanced"> + Efficiency Guide</seealso> for information about system limits. + </cell> </row> - <tcaption>Exit Reasons.</tcaption> + <tcaption>Exit Reasons</tcaption> </table> <p><c>Stack</c> is the stack of function calls being evaluated when the error occurred, given as a list of tuples <c>{Module,Name,Arity}</c> with the most recent function call - first. The most recent function call tuple may in some + first. The most recent function call tuple can in some cases be <c>{Module,Name,[Arg]}</c>.</p> </section> </chapter> diff --git a/system/doc/reference_manual/expressions.xml b/system/doc/reference_manual/expressions.xml index 62a344ad58..fd3cfabd3d 100644 --- a/system/doc/reference_manual/expressions.xml +++ b/system/doc/reference_manual/expressions.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,13 +28,17 @@ <rev></rev> <file>expressions.xml</file> </header> - <p>In this chapter, all valid Erlang expressions are listed. + <p>In this section, all valid Erlang expressions are listed. When writing Erlang programs, it is also allowed to use macro- and record expressions. However, these expressions are expanded during compilation and are in that sense not true Erlang expressions. Macro- and record expressions are covered in - separate chapters: <seealso marker="macros">Macros</seealso> and - <seealso marker="records">Records</seealso>.</p> + separate sections: + </p> + <list type="bulleted"> + <item><p><seealso marker="macros">Preprocessor</seealso></p></item> + <item><p><seealso marker="records">Records</seealso></p></item> + </list> <section> <title>Expression Evaluation</title> @@ -48,15 +52,15 @@ Expr1 + Expr2</code> performed.</p> <p>Many of the operators can only be applied to arguments of a certain type. For example, arithmetic operators can only be - applied to numbers. An argument of the wrong type will cause - a <c>badarg</c> run-time error.</p> + applied to numbers. An argument of the wrong type causes + a <c>badarg</c> runtime error.</p> </section> <section> <marker id="term"></marker> <title>Terms</title> <p>The simplest form of expression is a term, that is an integer, - float, atom, string, list, map or tuple. + float, atom, string, list, map, or tuple. The return value is the term itself.</p> </section> @@ -65,9 +69,10 @@ Expr1 + Expr2</code> <p>A variable is an expression. If a variable is bound to a value, the return value is this value. Unbound variables are only allowed in patterns.</p> - <p>Variables start with an uppercase letter or underscore (_) - and may contain alphanumeric characters, underscore and @. - Examples:</p> + <p>Variables start with an uppercase letter or underscore (_). + Variables can contain alphanumeric characters, underscore and <c>@</c>. + </p> + <p><em>Examples:</em></p> <pre> X Name1 @@ -77,18 +82,20 @@ _ _Height</pre> <p>Variables are bound to values using <seealso marker="patterns">pattern matching</seealso>. Erlang - uses <em>single assignment</em>, a variable can only be bound + uses <em>single assignment</em>, that is, a variable can only be bound once.</p> <p>The <em>anonymous variable</em> is denoted by underscore (_) and can be used when a variable is required but its value can be - ignored. Example:</p> + ignored.</p> + <p><em>Example:</em></p> <pre> [H|_] = [1,2,3]</pre> - <p>Variables starting with underscore (_), for example + <p>Variables starting with underscore (_), for example, <c>_Height</c>, are normal variables, not anonymous. They are - however ignored by the compiler in the sense that they will not - generate any warnings for unused variables. Example: The following - code</p> + however ignored by the compiler in the sense that they do not + generate any warnings for unused variables.</p> + <p><em>Example:</em></p> + <p>The following code:</p> <pre> member(_, []) -> [].</pre> @@ -96,36 +103,37 @@ member(_, []) -> <pre> member(Elem, []) -> [].</pre> - <p>This will however cause a warning for an unused variable + <p>This causes a warning for an unused variable, <c>Elem</c>, if the code is compiled with the flag <c>warn_unused_vars</c> set. Instead, the code can be rewritten to:</p> <pre> member(_Elem, []) -> [].</pre> - <p>Note that since variables starting with an underscore are - not anonymous, this will match:</p> + <p>Notice that since variables starting with an underscore are + not anonymous, this matches:</p> <pre> {_,_} = {1,2}</pre> - <p>But this will fail:</p> + <p>But this fails:</p> <pre> {_N,_N} = {1,2}</pre> <p>The scope for a variable is its function clause. Variables bound in a branch of an <c>if</c>, <c>case</c>, or <c>receive</c> expression must be bound in all branches - to have a value outside the expression, otherwise they - will be regarded as 'unsafe' outside the expression.</p> + to have a value outside the expression. Otherwise they + are regarded as 'unsafe' outside the expression.</p> <p>For the <c>try</c> expression introduced in - Erlang 5.4/OTP-R10B, variable scoping is limited so that + Erlang 5.4/OTP R10B, variable scoping is limited so that variables bound in the expression are always 'unsafe' outside - the expression. This will be improved.</p> + the expression. This is to be improved.</p> </section> <section> <marker id="pattern"></marker> <title>Patterns</title> - <p>A pattern has the same structure as a term but may contain - unbound variables. Example:</p> + <p>A pattern has the same structure as a term but can contain + unbound variables.</p> + <p><em>Example:</em></p> <pre> Name1 [H|T] @@ -136,13 +144,13 @@ Name1 <section> <title>Match Operator = in Patterns</title> <p>If <c>Pattern1</c> and <c>Pattern2</c> are valid patterns, - then the following is also a valid pattern:</p> + the following is also a valid pattern:</p> <pre> Pattern1 = Pattern2</pre> <p>When matched against a term, both <c>Pattern1</c> and - <c>Pattern2</c> will be matched against the term. The idea - behind this feature is to avoid reconstruction of terms. - Example:</p> + <c>Pattern2</c> are matched against the term. The idea + behind this feature is to avoid reconstruction of terms.</p> + <p><em>Example:</em></p> <pre> f({connect,From,To,Number,Options}, To) -> Signal = {connect,From,To,Number,Options}, @@ -163,16 +171,20 @@ f(Signal, To) -> <pre> f("prefix" ++ Str) -> ...</pre> <p>This is syntactic sugar for the equivalent, but harder to - read</p> + read:</p> <pre> f([$p,$r,$e,$f,$i,$x | Str]) -> ...</pre> </section> <section> <title>Expressions in Patterns</title> - <p>An arithmetic expression can be used within a pattern, if - it uses only numeric or bitwise operators, and if its value - can be evaluated to a constant at compile-time. Example:</p> + <p>An arithmetic expression can be used within a pattern if + it meets both of the following two conditions:</p> + <list type="bulleted"> + <item>It uses only numeric or bitwise operators.</item> + <item>Its value can be evaluated to a constant when complied.</item> + </list> + <p><em>Example:</em></p> <pre> case {Value, Result} of {?THRESHOLD+1, ok} -> ...</pre> @@ -182,21 +194,21 @@ case {Value, Result} of <section> <title>Match</title> + <p>The following matches <c>Expr1</c>, a pattern, against + <c>Expr2</c>:</p> <pre> Expr1 = Expr2</pre> - <p>Matches <c>Expr1</c>, a pattern, against <c>Expr2</c>. - If the matching succeeds, any unbound variable in the pattern + <p>If the matching succeeds, any unbound variable in the pattern becomes bound and the value of <c>Expr2</c> is returned.</p> - <p>If the matching fails, a <c>badmatch</c> run-time error will - occur.</p> - <p>Examples:</p> + <p>If the matching fails, a <c>badmatch</c> run-time error occurs.</p> + <p><em>Examples:</em></p> <pre> 1> <input>{A, B} = {answer, 42}.</input> {answer,42} 2> <input>A.</input> answer 3> <input>{C, D} = [1, 2].</input> -** exception error: no match of right hand side value [1,2]</pre> +** exception error: no match of right-hand side value [1,2]</pre> </section> <section> @@ -210,27 +222,28 @@ ExprM:ExprF(Expr1,...,ExprN)</pre> <c>ExprF</c> must be an atom or an expression that evaluates to an atom. The function is said to be called by using the <em>fully qualified function name</em>. This is often referred - to as a <em>remote</em> or <em>external function call</em>. - Example:</p> + to as a <em>remote</em> or <em>external function call</em>.</p> + <p><em>Example:</em></p> <code type="none"> lists:keysearch(Name, 1, List)</code> <p>In the second form of function calls, <c>ExprF(Expr1,...,ExprN)</c>, <c>ExprF</c> must be an atom or evaluate to a fun.</p> - <p>If <c>ExprF</c> is an atom the function is said to be called by + <p>If <c>ExprF</c> is an atom, the function is said to be called by using the <em>implicitly qualified function name</em>. If the function <c>ExprF</c> is locally defined, it is called. - Alternatively if <c>ExprF</c> is explicitly imported from module - <c>M</c>, <c>M:ExprF(Expr1,...,ExprN)</c> is called. If + Alternatively, if <c>ExprF</c> is explicitly imported from the + <c>M</c> module, <c>M:ExprF(Expr1,...,ExprN)</c> is called. If <c>ExprF</c> is neither declared locally nor explicitly imported, <c>ExprF</c> must be the name of an automatically - imported BIF. Examples:</p> + imported BIF. </p> + <p><em>Examples:</em></p> <code type="none"> handle(Msg, State) spawn(m, init, [])</code> - <p>Examples where ExprF is a fun:</p> + <p><em>Examples</em> where <c>ExprF</c> is a fun:</p> <code type="none"> Fun1 = fun(X) -> X+1 end Fun1(3) @@ -239,16 +252,15 @@ Fun1(3) fun lists:append/2([1,2], [3,4]) => [1,2,3,4]</code> - <p>Note that when calling a local function, there is a difference - between using the implicitly or fully qualified function name, as - the latter always refers to the latest version of the module. See - <seealso marker="code_loading">Compilation and Code Loading</seealso>.</p> - - <p>See also the chapter about - <seealso marker="functions#eval">Function Evaluation</seealso>.</p> + <p>Notice that when calling a local function, there is a difference + between using the implicitly or fully qualified function name. + The latter always refers to the latest version of the module. + See <seealso marker="code_loading">Compilation and Code Loading + </seealso> and <seealso marker="functions#eval"> + Function Evaluation</seealso>.</p> <section> - <title>Local Function Names Clashing With Auto-imported BIFs</title> + <title>Local Function Names Clashing With Auto-Imported BIFs</title> <p>If a local function has the same name as an auto-imported BIF, the semantics is that implicitly qualified function calls are directed to the locally defined function, not to the BIF. To avoid @@ -260,9 +272,9 @@ fun lists:append/2([1,2], [3,4]) <warning><p>Before OTP R14A (ERTS version 5.8), an implicitly qualified function call to a function having the same name as an auto-imported BIF always resulted in the BIF being called. In - newer versions of the compiler the local function is instead - called. The change is there to avoid that future additions to the - set of auto-imported BIFs does not silently change the behavior + newer versions of the compiler, the local function is called instead. + This is to avoid that future additions to the + set of auto-imported BIFs do not silently change the behavior of old code.</p> <p>However, to avoid that old (pre R14) code changed its @@ -272,8 +284,8 @@ fun lists:append/2([1,2], [3,4]) 5.8) and have an implicitly qualified call to that function in your code, you either need to explicitly remove the auto-import using a compiler directive, or replace the call with a fully - qualified function call, otherwise you will get a compilation - error. See example below:</p> </warning> + qualified function call. Otherwise you get a compilation + error. See the following example:</p> </warning> <code type="none"> -export([length/1,f/1]). @@ -290,9 +302,10 @@ f(X) when erlang:length(X) > 3 -> %% Calls erlang:length/1, long.</code> <p>The same logic applies to explicitly imported functions from - other modules as to locally defined functions. To both import a + other modules, as to locally defined functions. + It is not allowed to both import a function from another module and have the function declared in the - module at the same time is not allowed.</p> + module at the same time:</p> <code type="none"> -export([f/1]). @@ -310,10 +323,10 @@ f(X) -> length(X). %% mod:length/1 is called</code> - <p>For auto-imported BIFs added to Erlang in release R14A and thereafter, + <p>For auto-imported BIFs added in Erlang/OTP R14A and thereafter, overriding the name with a local function or explicit import is always allowed. However, if the <c>-compile({no_auto_import,[F/A])</c> - directive is not used, the compiler will issue a warning whenever + directive is not used, the compiler issues a warning whenever the function is called in the module using the implicitly qualified function name.</p> </section> @@ -330,15 +343,16 @@ if BodyN end</pre> <p>The branches of an <c>if</c>-expression are scanned sequentially - until a guard sequence <c>GuardSeq</c> which evaluates to true is + until a guard sequence <c>GuardSeq</c> that evaluates to true is found. Then the corresponding <c>Body</c> (sequence of expressions separated by ',') is evaluated.</p> <p>The return value of <c>Body</c> is the return value of the <c>if</c> expression.</p> - <p>If no guard sequence is true, an <c>if_clause</c> run-time error - will occur. If necessary, the guard expression <c>true</c> can be + <p>If no guard sequence is evaluated as true, + an <c>if_clause</c> run-time error + occurs. If necessary, the guard expression <c>true</c> can be used in the last branch, as that guard sequence is always true.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> is_greater_than(X, Y) -> if @@ -367,8 +381,8 @@ end</pre> <p>The return value of <c>Body</c> is the return value of the <c>case</c> expression.</p> <p>If there is no matching pattern with a true guard sequence, - a <c>case_clause</c> run-time error will occur.</p> - <p>Example:</p> + a <c>case_clause</c> run-time error occurs.</p> + <p><em>Example:</em></p> <pre> is_valid_signal(Signal) -> case Signal of @@ -389,15 +403,15 @@ Expr1 ! Expr2</pre> <p>Sends the value of <c>Expr2</c> as a message to the process specified by <c>Expr1</c>. The value of <c>Expr2</c> is also the return value of the expression.</p> - <p><c>Expr1</c> must evaluate to a pid, a registered name (atom) or - a tuple <c>{Name,Node}</c>, where <c>Name</c> is an atom and - <c>Node</c> a node name, also an atom.</p> + <p><c>Expr1</c> must evaluate to a pid, a registered name (atom), or + a tuple <c>{Name,Node}</c>. <c>Name</c> is an atom and + <c>Node</c> is a node name, also an atom.</p> <list type="bulleted"> <item>If <c>Expr1</c> evaluates to a name, but this name is not - registered, a <c>badarg</c> run-time error will occur.</item> + registered, a <c>badarg</c> run-time error occurs.</item> <item>Sending a message to a pid never fails, even if the pid identifies a non-existing process.</item> - <item>Distributed message sending, that is if <c>Expr1</c> + <item>Distributed message sending, that is, if <c>Expr1</c> evaluates to a tuple <c>{Name,Node}</c> (or a pid located at another node), also never fails.</item> </list> @@ -420,14 +434,14 @@ end</pre> the second, and so on. If a match succeeds and the optional guard sequence <c>GuardSeq</c> is true, the corresponding <c>Body</c> is evaluated. The matching message is consumed, that - is removed from the mailbox, while any other messages in + is, removed from the mailbox, while any other messages in the mailbox remain unchanged.</p> <p>The return value of <c>Body</c> is the return value of the <c>receive</c> expression.</p> - <p><c>receive</c> never fails. Execution is suspended, possibly - indefinitely, until a message arrives that does match one of + <p><c>receive</c> never fails. The execution is suspended, possibly + indefinitely, until a message arrives that matches one of the patterns and with a true guard sequence. </p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> wait_for_onhook() -> receive @@ -438,7 +452,7 @@ wait_for_onhook() -> B ! {busy, self()}, wait_for_onhook() end.</pre> - <p>It is possible to augment the <c>receive</c> expression with a + <p>The <c>receive</c> expression can be augmented with a timeout:</p> <pre> receive @@ -451,14 +465,14 @@ after ExprT -> BodyT end</pre> - <p><c>ExprT</c> should evaluate to an integer. The highest allowed - value is 16#ffffffff, that is, the value must fit in 32 bits. + <p><c>ExprT</c> is to evaluate to an integer. The highest allowed + value is 16#FFFFFFFF, that is, the value must fit in 32 bits. <c>receive..after</c> works exactly as <c>receive</c>, except that if no matching message has arrived within <c>ExprT</c> - milliseconds, then <c>BodyT</c> is evaluated instead and its - return value becomes the return value of the <c>receive..after</c> - expression.</p> - <p>Example:</p> + milliseconds, then <c>BodyT</c> is evaluated instead. The + return value of <c>BodyT</c> then becomes the return value + of the <c>receive..after</c> expression.</p> + <p><em>Example:</em></p> <pre> wait_for_onhook() -> receive @@ -481,10 +495,10 @@ after ExprT -> BodyT end</pre> - <p>This construction will not consume any messages, only suspend - execution in the process for <c>ExprT</c> milliseconds and can be + <p>This construction does not consume any messages, only suspends + execution in the process for <c>ExprT</c> milliseconds. This can be used to implement simple timers.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> timer() -> spawn(m, timer, [self()]). @@ -498,12 +512,12 @@ timer(Pid) -> <p>There are two special cases for the timeout value <c>ExprT</c>:</p> <taglist> <tag><c>infinity</c></tag> - <item>The process should wait indefinitely for a matching message - -- this is the same as not using a timeout. Can be - useful for timeout values that are calculated at run-time.</item> + <item>The process is to wait indefinitely for a matching message; + this is the same as not using a timeout. This can be + useful for timeout values that are calculated at runtime.</item> <tag>0</tag> <item>If there is no matching message in the mailbox, the timeout - will occur immediately.</item> + occurs immediately.</item> </taglist> </section> @@ -518,39 +532,39 @@ Expr1 <input>op</input> Expr2</pre> </row> <row> <cell align="left" valign="middle">==</cell> - <cell align="left" valign="middle">equal to</cell> + <cell align="left" valign="middle">Equal to</cell> </row> <row> <cell align="left" valign="middle">/=</cell> - <cell align="left" valign="middle">not equal to</cell> + <cell align="left" valign="middle">Not equal to</cell> </row> <row> <cell align="left" valign="middle">=<</cell> - <cell align="left" valign="middle">less than or equal to</cell> + <cell align="left" valign="middle">Less than or equal to</cell> </row> <row> <cell align="left" valign="middle"><</cell> - <cell align="left" valign="middle">less than</cell> + <cell align="left" valign="middle">Less than</cell> </row> <row> <cell align="left" valign="middle">>=</cell> - <cell align="left" valign="middle">greater than or equal to</cell> + <cell align="left" valign="middle">Greater than or equal to</cell> </row> <row> <cell align="left" valign="middle">></cell> - <cell align="left" valign="middle">greater than</cell> + <cell align="left" valign="middle">Greater than</cell> </row> <row> <cell align="left" valign="middle">=:=</cell> - <cell align="left" valign="middle">exactly equal to</cell> + <cell align="left" valign="middle">Exactly equal to</cell> </row> <row> <cell align="left" valign="middle">=/=</cell> - <cell align="left" valign="middle">exactly not equal to</cell> + <cell align="left" valign="middle">Exactly not equal to</cell> </row> <tcaption>Term Comparison Operators.</tcaption> </table> - <p>The arguments may be of different data types. The following + <p>The arguments can be of different data types. The following order is defined:</p> <pre> number < atom < reference < fun < port < pid < tuple < list < bit string</pre> @@ -558,17 +572,18 @@ number < atom < reference < fun < port < pid < tuple < list size, two tuples with the same size are compared element by element.</p> <p>When comparing an integer to a float, the term with the lesser - precision will be converted into the other term's type, unless the - operator is one of =:= or =/=. A float is more precise than + precision is converted into the type of the other term, unless the + operator is one of <c>=:=</c> or <c>=/=</c>. A float is more precise than an integer until all significant figures of the float are to the left of the decimal point. This happens when the float is larger/smaller than +/-9007199254740992.0. The conversion strategy is changed depending on the size of the float because otherwise comparison of large floats and integers would lose their transitivity.</p> - <p>Returns the Boolean value of the expression, <c>true</c> or - <c>false</c>.</p> - <p>Examples:</p> + <p>Term comparison operators return the Boolean value of the + expression, <c>true</c> or <c>false</c>.</p> + + <p><em>Examples:</em></p> <pre> 1> <input>1==1.0.</input> true @@ -585,19 +600,19 @@ false</pre> Expr1 <input>op</input> Expr2</pre> <table> <row> - <cell align="left" valign="middle"><em>op</em></cell> + <cell align="left" valign="middle"><em>Operator</em></cell> <cell align="left" valign="middle"><em>Description</em></cell> - <cell align="left" valign="middle"><em>Argument type</em></cell> + <cell align="left" valign="middle"><em>Argument Type</em></cell> </row> <row> <cell align="left" valign="middle">+</cell> - <cell align="left" valign="middle">unary +</cell> - <cell align="left" valign="middle">number</cell> + <cell align="left" valign="middle">Unary +</cell> + <cell align="left" valign="middle">Number</cell> </row> <row> <cell align="left" valign="middle">-</cell> - <cell align="left" valign="middle">unary -</cell> - <cell align="left" valign="middle">number</cell> + <cell align="left" valign="middle">Unary -</cell> + <cell align="left" valign="middle">Number</cell> </row> <row> <cell align="left" valign="middle">+</cell> @@ -607,62 +622,62 @@ Expr1 <input>op</input> Expr2</pre> <row> <cell align="left" valign="middle">-</cell> <cell align="left" valign="middle"> </cell> - <cell align="left" valign="middle">number</cell> + <cell align="left" valign="middle">Number</cell> </row> <row> <cell align="left" valign="middle">*</cell> <cell align="left" valign="middle"> </cell> - <cell align="left" valign="middle">number</cell> + <cell align="left" valign="middle">Number</cell> </row> <row> <cell align="left" valign="middle">/</cell> - <cell align="left" valign="middle">floating point division</cell> - <cell align="left" valign="middle">number</cell> + <cell align="left" valign="middle">Floating point division</cell> + <cell align="left" valign="middle">Number</cell> </row> <row> <cell align="left" valign="middle">bnot</cell> - <cell align="left" valign="middle">unary bitwise not</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Unary bitwise NOT</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">div</cell> - <cell align="left" valign="middle">integer division</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Integer division</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">rem</cell> - <cell align="left" valign="middle">integer remainder of X/Y</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Integer remainder of X/Y</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">band</cell> - <cell align="left" valign="middle">bitwise and</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Bitwise AND</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">bor</cell> - <cell align="left" valign="middle">bitwise or</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Bitwise OR</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">bxor</cell> - <cell align="left" valign="middle">arithmetic bitwise xor</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Arithmetic bitwise XOR</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">bsl</cell> - <cell align="left" valign="middle">arithmetic bitshift left</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Arithmetic bitshift left</cell> + <cell align="left" valign="middle">Integer</cell> </row> <row> <cell align="left" valign="middle">bsr</cell> - <cell align="left" valign="middle">bitshift right</cell> - <cell align="left" valign="middle">integer</cell> + <cell align="left" valign="middle">Bitshift right</cell> + <cell align="left" valign="middle">Integer</cell> </row> <tcaption>Arithmetic Operators.</tcaption> </table> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>+1.</input> 1 @@ -697,28 +712,28 @@ Expr1 <input>op</input> Expr2</pre> Expr1 <input>op</input> Expr2</pre> <table> <row> - <cell align="left" valign="middle"><em>op</em></cell> + <cell align="left" valign="middle"><em>Operator</em></cell> <cell align="left" valign="middle"><em>Description</em></cell> </row> <row> <cell align="left" valign="middle">not</cell> - <cell align="left" valign="middle">unary logical not</cell> + <cell align="left" valign="middle">Unary logical NOT</cell> </row> <row> <cell align="left" valign="middle">and</cell> - <cell align="left" valign="middle">logical and</cell> + <cell align="left" valign="middle">Logical AND</cell> </row> <row> <cell align="left" valign="middle">or</cell> - <cell align="left" valign="middle">logical or</cell> + <cell align="left" valign="middle">Logical OR</cell> </row> <row> <cell align="left" valign="middle">xor</cell> - <cell align="left" valign="middle">logical xor</cell> + <cell align="left" valign="middle">Logical XOR</cell> </row> <tcaption>Logical Operators.</tcaption> </table> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>not true.</input> false @@ -737,28 +752,37 @@ true <pre> Expr1 orelse Expr2 Expr1 andalso Expr2</pre> - <p>Expressions where <c>Expr2</c> is evaluated only if - necessary. That is, <c>Expr2</c> is evaluated only if <c>Expr1</c> - evaluates to <c>false</c> in an <c>orelse</c> expression, or only - if <c>Expr1</c> evaluates to <c>true</c> in an <c>andalso</c> - expression. Returns either the value of <c>Expr1</c> (that is, + <p><c>Expr2</c> is evaluated only if + necessary. That is, <c>Expr2</c> is evaluated only if:</p> + <list type="bulleted"> + <item><p><c>Expr1</c> evaluates to <c>false</c> in an + <c>orelse</c> expression.</p> + </item> + </list> + <p>or</p> + <list type="bulleted"> + <item><p><c>Expr1</c> evaluates to <c>true</c> in an + <c>andalso</c> expression.</p> + </item> + </list> + <p>Returns either the value of <c>Expr1</c> (that is, <c>true</c> or <c>false</c>) or the value of <c>Expr2</c> - (if <c>Expr2</c> was evaluated).</p> + (if <c>Expr2</c> is evaluated).</p> - <p>Example 1:</p> + <p><em>Example 1:</em></p> <pre> case A >= -1.0 andalso math:sqrt(A+1) > B of</pre> - <p>This will work even if <c>A</c> is less than <c>-1.0</c>, + <p>This works even if <c>A</c> is less than <c>-1.0</c>, since in that case, <c>math:sqrt/1</c> is never evaluated.</p> - <p>Example 2:</p> + <p><em>Example 2:</em></p> <pre> OnlyOne = is_atom(L) orelse (is_list(L) andalso length(L) == 1),</pre> - <p>From R13A, <c>Expr2</c> is no longer required to evaluate to a - boolean value. As a consequence, <c>andalso</c> and <c>orelse</c> + <p>From Erlang/OTP R13A, <c>Expr2</c> is no longer required to evaluate to a + Boolean value. As a consequence, <c>andalso</c> and <c>orelse</c> are now tail-recursive. For instance, the following function is - tail-recursive in R13A and later:</p> + tail-recursive in Erlang/OTP R13A and later:</p> <pre> all(Pred, [Hd|Tail]) -> @@ -774,11 +798,11 @@ Expr1 ++ Expr2 Expr1 -- Expr2</pre> <p>The list concatenation operator <c>++</c> appends its second argument to its first and returns the resulting list.</p> - <p>The list subtraction operator <c>--</c> produces a list which - is a copy of the first argument, subjected to the following - procedure: for each element in the second argument, the first + <p>The list subtraction operator <c>--</c> produces a list that + is a copy of the first argument. The procedure is a follows: + for each element in the second argument, the first occurrence of this element (if any) is removed.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> 1> <input>[1,2,3]++[4,5].</input> [1,2,3,4,5] @@ -786,8 +810,8 @@ Expr1 -- Expr2</pre> [3,1,2]</pre> <warning><p>The complexity of <c>A -- B</c> is - proportional to <c>length(A)*length(B)</c>, meaning that it - will be very slow if both <c>A</c> and <c>B</c> are + proportional to <c>length(A)*length(B)</c>. That is, it + becomes very slow if both <c>A</c> and <c>B</c> are long lists.</p></warning> </section> @@ -802,7 +826,7 @@ Expr1 -- Expr2</pre> </p> <code>#{ K => V }</code> <p> - New maps may include multiple associations at construction by listing every + New maps can include multiple associations at construction by listing every association: </p> <code>#{ K1 => V1, .., Kn => Vn }</code> @@ -816,11 +840,11 @@ Expr1 -- Expr2</pre> </p> <p> Keys and values are separated by the <c>=></c> arrow and associations are - separated by <c>,</c>. + separated by a comma <c>,</c>. </p> <p> - Examples: + <em>Examples:</em> </p> <code> M0 = #{}, % empty map @@ -829,14 +853,14 @@ M2 = #{1 => 2, b => b}, % multiple associations with literals M3 = #{k => {A,B}}, % single association with variables M4 = #{{"w", 1} => f()}. % compound key associated with an evaluated expression</code> <p> - where, <c>A</c> and <c>B</c> are any expressions and <c>M0</c> through <c>M4</c> + Here, <c>A</c> and <c>B</c> are any expressions and <c>M0</c> through <c>M4</c> are the resulting map terms. </p> <p> - If two matching keys are declared, the latter key will take precedence. + If two matching keys are declared, the latter key takes precedence. </p> <p> - Example: + <em>Example:</em> </p> <pre> @@ -846,54 +870,57 @@ M4 = #{{"w", 1} => f()}. % compound key associated with an evaluated expression #{1 => b, 1.0 => a} </pre> <p> - The order in which the expressions constructing the keys and their - associated values are evaluated is not defined. The syntactic order of + The order in which the expressions constructing the keys (and their + associated values) are evaluated is not defined. The syntactic order of the key-value pairs in the construction is of no relevance, except in - the above mentioned case of two matching keys. + the recently mentioned case of two matching keys. </p> </section> <section> <title>Updating Maps</title> <p> - Updating a map has similar syntax as constructing it. + Updating a map has a similar syntax as constructing it. </p> <p> - An expression defining the map to be updated is put in front of the expression - defining the keys to be updated and their respective values. + An expression defining the map to be updated, is put in front of the expression + defining the keys to be updated and their respective values: </p> <code>M#{ K => V }</code> <p> - where <c>M</c> is a term of type map and <c>K</c> and <c>V</c> are any expression. + Here <c>M</c> is a term of type map and <c>K</c> and <c>V</c> are any expression. </p> <p> If key <c>K</c> does not match any existing key in the map, a new association - will be created from key <c>K</c> to value <c>V</c>. If key <c>K</c> matches - an existing key in map <c>M</c> its associated value will be replaced by the - new value <c>V</c>. In both cases the evaluated map expression will return a new map. + is created from key <c>K</c> to value <c>V</c>. + </p> + <p> If key <c>K</c> matches an existing key in map <c>M</c>, + its associated value + is replaced by the new value <c>V</c>. In both cases, the evaluated map expression + returns a new map. </p> <p> - If <c>M</c> is not of type map an exception of type <c>badmap</c> is thrown. + If <c>M</c> is not of type map, an exception of type <c>badmap</c> is thrown. </p> <p> - To only update an existing value, the following syntax is used, + To only update an existing value, the following syntax is used: </p> <code>M#{ K := V } </code> <p> - where <c>M</c> is an term of type map, <c>V</c> is an expression and <c>K</c> - is an expression which evaluates to an existing key in <c>M</c>. + Here <c>M</c> is a term of type map, <c>V</c> is an expression and <c>K</c> + is an expression that evaluates to an existing key in <c>M</c>. </p> <p> - If key <c>K</c> does not match any existing keys in map <c>M</c> an exception - of type <c>badarg</c> will be triggered at runtime. If a matching key <c>K</c> - is present in map <c>M</c> its associated value will be replaced by the new - value <c>V</c> and the evaluated map expression returns a new map. + If key <c>K</c> does not match any existing keys in map <c>M</c>, an exception + of type <c>badarg</c> is triggered at runtime. If a matching key <c>K</c> + is present in map <c>M</c>, its associated value is replaced by the new + value <c>V</c>, and the evaluated map expression returns a new map. </p> <p> - If <c>M</c> is not of type map an exception of type <c>badmap</c> is thrown. + If <c>M</c> is not of type map, an exception of type <c>badmap</c> is thrown. </p> <p> - Examples: + <em>Examples:</em> </p> <code> M0 = #{}, @@ -902,10 +929,10 @@ M2 = M1#{a => 1, b => 2}, M3 = M2#{"function" => fun() -> f() end}, M4 = M3#{a := 2, b := 3}. % 'a' and 'b' was added in `M1` and `M2`.</code> <p> - where <c>M0</c> is any map. It follows that <c>M1 .. M4</c> are maps as well. + Here <c>M0</c> is any map. It follows that <c>M1 .. M4</c> are maps as well. </p> <p> - More Examples: + More <em>Examples:</em> </p> <pre> 1> <input>M = #{1 => a}.</input> @@ -921,83 +948,84 @@ M4 = M3#{a := 2, b := 3}. % 'a' and 'b' was added in `M1` and `M2`.</code> As in construction, the order in which the key and value expressions are evaluated is not defined. The syntactic order of the key-value pairs in the update is of no - relevance, except in the case where two keys match, in which - case the latter value is used. + relevance, except in the case where two keys match. + In that case, the latter value is used. </p> </section> <section> <title>Maps in Patterns</title> <p> - Matching of key-value associations from maps is done in the following way: + Matching of key-value associations from maps is done as follows: </p> <code>#{ K := V } = M</code> <p> - where <c>M</c> is any map. The key <c>K</c> has to be an expression with bound - variables or a literals, and <c>V</c> can be any pattern with either bound or + Here <c>M</c> is any map. The key <c>K</c> must be an expression with bound + variables or literals. <c>V</c> can be any pattern with either bound or unbound variables. </p> <p> - If the variable <c>V</c> is unbound, it will be bound to the value associated - with the key <c>K</c>, which has to exist in the map <c>M</c>. If the variable - <c>V</c> is bound, it has to match the value associated with <c>K</c> in <c>M</c>. + If the variable <c>V</c> is unbound, it becomes bound to the value associated + with the key <c>K</c>, which must exist in the map <c>M</c>. If the variable + <c>V</c> is bound, it must match the value associated with <c>K</c> in <c>M</c>. </p> - <p> Example: </p> -<code> + <p><em>Example:</em></p> +<pre> 1> <input>M = #{"tuple" => {1,2}}.</input> #{"tuple" => {1,2}} 2> <input>#{"tuple" := {1,B}} = M.</input> #{"tuple" => {1,2}} 3> <input>B.</input> -2.</code> +2.</pre> <p> - This will bind variable <c>B</c> to integer <c>2</c>. + This binds variable <c>B</c> to integer <c>2</c>. </p> <p> - Similarly, multiple values from the map may be matched: + Similarly, multiple values from the map can be matched: </p> <code>#{ K1 := V1, .., Kn := Vn } = M</code> <p> - where keys <c>K1 .. Kn</c> are any expressions with literals or bound variables. If all - keys exist in map <c>M</c> all variables in <c>V1 .. Vn</c> will be matched to the + Here keys <c>K1 .. Kn</c> are any expressions with literals or bound variables. If all + keys exist in map <c>M</c>, all variables in <c>V1 .. Vn</c> is matched to the associated values of their respective keys. </p> <p> - If the matching conditions are not met, the match will fail, either with + If the matching conditions are not met, the match fails, either with: </p> <list> - <item> - a <c>badmatch</c> exception, if used in the context of the matching operator - as in the example, + <item><p>A <c>badmatch</c> exception.</p> + <p>This is if it is used in the context of the matching operator + as in the example.</p> </item> - <item> - or resulting in the next clause being tested in function heads and - case expressions. + <item><p>Or resulting in the next clause being tested in function heads and + case expressions.</p> </item> </list> <p> Matching in maps only allows for <c>:=</c> as delimiters of associations. + </p> + <p> The order in which keys are declared in matching has no relevance. </p> <p> - Duplicate keys are allowed in matching and will match each pattern associated - to the keys. + Duplicate keys are allowed in matching and match each pattern associated + to the keys: </p> <code>#{ K := V1, K := V2 } = M</code> <p> - Matching an expression against an empty map literal will match its type but - no variables will be bound: + Matching an expression against an empty map literal, matches its type but + no variables are bound: </p> <code>#{} = Expr</code> <p> - This expression will match if the expression <c>Expr</c> is of type map, otherwise - it will fail with an exception <c>badmatch</c>. + This expression matches if the expression <c>Expr</c> is of type map, otherwise + it fails with an exception <c>badmatch</c>. </p> <section> - <title>Matching syntax: Example with literals in function heads</title> + <title>Matching Syntax</title> <p> - Matching of literals as keys are allowed in function heads. + Matching of literals as keys are allowed in function heads: </p> <code> %% only start if not_started @@ -1014,17 +1042,19 @@ handle_call(change, From, #{ state := start } = S) -> <section> <title>Maps in Guards</title> <p> - Maps are allowed in guards as long as all sub-expressions are valid guard expressions. + Maps are allowed in guards as long as all subexpressions are valid guard expressions. </p> <p> - Two guard BIFs handles maps: + Two guard BIFs handle maps: </p> <list> <item> <seealso marker="erts:erlang#is_map/1">is_map/1</seealso> + in the <c>erlang</c> module </item> <item> <seealso marker="erts:erlang#map_size/1">map_size/1</seealso> + in the <c>erlang</c> module </item> </list> </section> @@ -1044,29 +1074,34 @@ Ei = Value | Value/TypeSpecifierList | Value:Size/TypeSpecifierList</pre> <p>Used in a bit string construction, <c>Value</c> is an expression - which should evaluate to an integer, float or bit string. If the - expression is something else than a single literal or variable, it - should be enclosed in parenthesis.</p> + that is to evaluate to an integer, float, or bit string. If the + expression is not a single literal or variable, it + is to be enclosed in parenthesis.</p> <p>Used in a bit string matching, <c>Value</c> must be a variable, - or an integer, float or string.</p> + or an integer, float, or string.</p> - <p>Note that, for example, using a string literal as in + <p>Notice that, for example, using a string literal as in <c><![CDATA[<<"abc">>]]></c> is syntactic sugar for <c><![CDATA[<<$a,$b,$c>>]]></c>.</p> <p>Used in a bit string construction, <c>Size</c> is an expression - which should evaluate to an integer.</p> + that is to evaluate to an integer.</p> - <p>Used in a bit string matching, <c>Size</c> must be an integer or a + <p>Used in a bit string matching, <c>Size</c> must be an integer, or a variable bound to an integer.</p> <p>The value of <c>Size</c> specifies the size of the segment in units (see below). The default value depends on the type (see - below). For <c>integer</c> it is 8, for - <c>float</c> it is 64, for <c>binary</c> and <c>bitstring</c> it is - the whole binary or bit string. In matching, this default value is only - valid for the very last element. All other bit string or binary + below):</p> + <list type="bulleted"> + <item>For <c>integer</c> it is 8.</item> + <item>For <c>float</c> it is 64.</item> + <item>For <c>binary</c> and <c>bitstring</c> it is + the whole binary or bit string.</item> + </list> + <p>In matching, this default value is only + valid for the last element. All other bit string or binary elements in the matching must have a size specification.</p> <p>For the <c>utf8</c>, <c>utf16</c>, and <c>utf32</c> types, @@ -1090,7 +1125,7 @@ Ei = Value | The default is <c>unsigned</c>.</item> <tag><c>Endianness</c>= <c>big</c> | <c>little</c> | <c>native</c></tag> - <item>Native-endian means that the endianness will be resolved at load + <item>Native-endian means that the endianness is resolved at load time to be either big-endian or little-endian, depending on what is native for the CPU that the Erlang machine is run on. Endianness only matters when the Type is either <c>integer</c>, @@ -1099,7 +1134,7 @@ Ei = Value | <tag><c>Unit</c>= <c>unit:IntegerLiteral</c></tag> <item>The allowed range is 1..256. Defaults to 1 for <c>integer</c>, - <c>float</c> and <c>bitstring</c>, and to 8 for <c>binary</c>. + <c>float</c>, and <c>bitstring</c>, and to 8 for <c>binary</c>. No unit specifier must be given for the types <c>utf8</c>, <c>utf16</c>, and <c>utf32</c>. </item> @@ -1110,8 +1145,8 @@ Ei = Value | <note><p>When constructing binaries, if the size <c>N</c> of an integer segment is too small to contain the given integer, the most significant - bits of the integer will be silently discarded and only the <c>N</c> least - significant bits will be put into the binary.</p></note> + bits of the integer are silently discarded and only the <c>N</c> least + significant bits are put into the binary.</p></note> <p>The types <c>utf8</c>, <c>utf16</c>, and <c>utf32</c> specifies encoding/decoding of the <em>Unicode Transformation Format</em>s UTF-8, UTF-16, @@ -1120,39 +1155,39 @@ Ei = Value | <p>When constructing a segment of a <c>utf</c> type, <c>Value</c> must be an integer in the range 0..16#D7FF or 16#E000....16#10FFFF. Construction - will fail with a <c>badarg</c> exception if <c>Value</c> is + fails with a <c>badarg</c> exception if <c>Value</c> is outside the allowed ranges. The size of the resulting binary - segment depends on the type and/or <c>Value</c>. For <c>utf8</c>, - <c>Value</c> will be encoded in 1 through 4 bytes. For - <c>utf16</c>, <c>Value</c> will be encoded in 2 or 4 - bytes. Finally, for <c>utf32</c>, <c>Value</c> will always be - encoded in 4 bytes.</p> + segment depends on the type or <c>Value</c>, or both:</p> + <list type="bulleted"> + <item>For <c>utf8</c>, <c>Value</c> is encoded in 1-4 bytes.</item> + <item>For <c>utf16</c>, <c>Value</c> is encoded in 2 or 4 bytes.</item> + <item>For <c>utf32</c>, <c>Value</c> is always be encoded in 4 bytes.</item> + </list> - <p>When constructing, a literal string may be given followed + <p>When constructing, a literal string can be given followed by one of the UTF types, for example: <c><![CDATA[<<"abc"/utf8>>]]></c> - which is syntatic sugar for + which is syntactic sugar for <c><![CDATA[<<$a/utf8,$b/utf8,$c/utf8>>]]></c>.</p> - <p>A successful match of a segment of a <c>utf</c> type results + <p>A successful match of a segment of a <c>utf</c> type, results in an integer in the range 0..16#D7FF or 16#E000..16#10FFFF. - The match will fail if returned value - would fall outside those ranges.</p> + The match fails if the returned value falls outside those ranges.</p> - <p>A segment of type <c>utf8</c> will match 1 to 4 bytes in the binary, + <p>A segment of type <c>utf8</c> matches 1-4 bytes in the binary, if the binary at the match position contains a valid UTF-8 sequence. (See RFC-3629 or the Unicode standard.)</p> - <p>A segment of type <c>utf16</c> may match 2 or 4 bytes in the binary. - The match will fail if the binary at the match position does not contain + <p>A segment of type <c>utf16</c> can match 2 or 4 bytes in the binary. + The match fails if the binary at the match position does not contain a legal UTF-16 encoding of a Unicode code point. (See RFC-2781 or the Unicode standard.)</p> - <p>A segment of type <c>utf32</c> may match 4 bytes in the binary in the - same way as an <c>integer</c> segment matching 32 bits. - The match will fail if the resulting integer is outside the legal ranges + <p>A segment of type <c>utf32</c> can match 4 bytes in the binary in the + same way as an <c>integer</c> segment matches 32 bits. + The match fails if the resulting integer is outside the legal ranges mentioned above.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>Bin1 = <<1,17,42>>.</input> <<1,17,42>> @@ -1181,11 +1216,13 @@ Ei = Value | 13> <input><<1024/utf8>>.</input> <<208,128>> </pre> - <p>Note that bit string patterns cannot be nested.</p> - <p>Note also that "<c><![CDATA[B=<<1>>]]></c>" is interpreted as + <p>Notice that bit string patterns cannot be nested.</p> + <p>Notice also that "<c><![CDATA[B=<<1>>]]></c>" is interpreted as "<c><![CDATA[B =<<1>>]]></c>" which is a syntax error. The correct way is to write a space after '=': "<c><![CDATA[B= <<1>>]]></c>.</p> - <p>More examples can be found in <em>Programming Examples</em>.</p> + <p>More examples are provided in + <seealso marker="doc/programming_examples:bit_syntax"> + Programming Examples</seealso>.</p> </section> <section> @@ -1200,16 +1237,16 @@ fun BodyK end</pre> <p>A fun expression begins with the keyword <c>fun</c> and ends - with the keyword <c>end</c>. Between them should be a function + with the keyword <c>end</c>. Between them is to be a function declaration, similar to a <seealso marker="functions#syntax">regular function declaration</seealso>, - except that the function name is optional and should be a variable if + except that the function name is optional and is to be a variable, if any.</p> <p>Variables in a fun head shadow the function name and both shadow - variables in the function clause surrounding the fun expression, and - variables bound in a fun body are local to the fun body.</p> + variables in the function clause surrounding the fun expression. + Variables bound in a fun body are local to the fun body.</p> <p>The return value of the expression is the resulting fun.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>Fun1 = fun (X) -> X+1 end.</input> #Fun<erl_eval.6.39074546> @@ -1232,15 +1269,17 @@ fun Module:Name/Arity</pre> syntactic sugar for:</p> <pre> fun (Arg1,...,ArgN) -> Name(Arg1,...,ArgN) end</pre> - <p>In <c>Module:Name/Arity</c>, <c>Module</c> and <c>Name</c> are atoms - and <c>Arity</c> is an integer. Starting from the R15 release, - <c>Module</c>, <c>Name</c>, and <c>Arity</c> may also be variables. - A fun defined in this way will refer to the function <c>Name</c> + <p>In <c>Module:Name/Arity</c>, <c>Module</c>, and <c>Name</c> are atoms + and <c>Arity</c> is an integer. Starting from Erlang/OTP R15, + <c>Module</c>, <c>Name</c>, and <c>Arity</c> can also be variables. + A fun defined in this way refers to the function <c>Name</c> with arity <c>Arity</c> in the <em>latest</em> version of module - <c>Module</c>. A fun defined in this way will not be dependent on - the code for module in which it is defined. + <c>Module</c>. A fun defined in this way is not dependent on + the code for the module in which it is defined. </p> - <p>More examples can be found in <em>Programming Examples</em>.</p> + <p>More examples are provided in + <seealso marker="doc/programming_examples:funs"> + Programming Examples</seealso>.</p> </section> <section> @@ -1250,23 +1289,26 @@ fun (Arg1,...,ArgN) -> Name(Arg1,...,ArgN) end</pre> catch Expr</code> <p>Returns the value of <c>Expr</c> unless an exception occurs during the evaluation. In that case, the exception is - caught. For exceptions of class <c>error</c>, - that is run-time errors: <c>{'EXIT',{Reason,Stack}}</c> - is returned. For exceptions of class <c>exit</c>, that is - the code called <c>exit(Term)</c>: <c>{'EXIT',Term}</c> is returned. - For exceptions of class <c>throw</c>, that is - the code called <c>throw(Term)</c>: <c>Term</c> is returned.</p> + caught.</p> + <p>For exceptions of class <c>error</c>, that is, + run-time errors, + <c>{'EXIT',{Reason,Stack}}</c> is returned.</p> + <p>For exceptions of class <c>exit</c>, that is, + the code called <c>exit(Term)</c>, + <c>{'EXIT',Term}</c> is returned.</p> + <p>For exceptions of class <c>throw</c>, that is + the code called <c>throw(Term)</c>, + <c>Term</c> is returned.</p> <p><c>Reason</c> depends on the type of error that occurred, and <c>Stack</c> is the stack of recent function calls, see - <seealso marker="errors#exit_reasons">Errors and Error Handling</seealso>.</p> - <p>Examples:</p> - <p></p> + <seealso marker="errors#exit_reasons">Exit Reasons</seealso>.</p> + <p><em>Examples:</em></p> <pre> 1> <input>catch 1+2.</input> 3 2> <input>catch 1+a.</input> {'EXIT',{badarith,[...]}}</pre> - <p>Note that <c>catch</c> has low precedence and catch + <p>Notice that <c>catch</c> has low precedence and catch subexpressions often needs to be enclosed in a block expression or in parenthesis:</p> <pre> @@ -1275,13 +1317,14 @@ catch Expr</code> 4> <input>A = (catch 1+2).</input> 3</pre> <p>The BIF <c>throw(Any)</c> can be used for non-local return from - a function. It must be evaluated within a <c>catch</c>, which will - return the value <c>Any</c>. Example:</p> + a function. It must be evaluated within a <c>catch</c>, which + returns the value <c>Any</c>.</p> + <p><em>Example:</em></p> <pre> 5> <input>catch throw(hello).</input> hello</pre> <p>If <c>throw/1</c> is not evaluated within a catch, a - <c>nocatch</c> run-time error will occur.</p> + <c>nocatch</c> run-time error occurs.</p> </section> <section> @@ -1297,14 +1340,17 @@ catch end</code> <p>This is an enhancement of <seealso marker="#catch">catch</seealso> that appeared in - Erlang 5.4/OTP-R10B. It gives the possibility do distinguish - between different exception classes, and to choose to handle only - the desired ones, passing the others on to an enclosing - <c>try</c> or <c>catch</c> or to default error handling.</p> - <p>Note that although the keyword <c>catch</c> is used in + Erlang 5.4/OTP R10B. It gives the possibility to:</p> + <list type="bulleted"> + <item>Distinguish between different exception classes.</item> + <item>Choose to handle only the desired ones.</item> + <item>Passing the others on to an enclosing + <c>try</c> or <c>catch</c>, or to default error handling.</item> + </list> + <p>Notice that although the keyword <c>catch</c> is used in the <c>try</c> expression, there is not a <c>catch</c> expression within the <c>try</c> expression.</p> - <p>Returns the value of <c>Exprs</c> (a sequence of expressions + <p>It returns the value of <c>Exprs</c> (a sequence of expressions <c>Expr1, ..., ExprN</c>) unless an exception occurs during the evaluation. In that case the exception is caught and the patterns <c>ExceptionPattern</c> with the right exception @@ -1318,7 +1364,7 @@ end</code> <c>Class</c> with a true guard sequence, the exception is passed on as if <c>Exprs</c> had not been enclosed in a <c>try</c> expression.</p> - <p>If an exception occurs during evaluation of <c>ExceptionBody</c> + <p>If an exception occurs during evaluation of <c>ExceptionBody</c>, it is not caught.</p> <p>The <c>try</c> expression can have an <c>of</c> section: @@ -1341,7 +1387,7 @@ end</code> the patterns <c>Pattern</c> are sequentially matched against the result in the same way as for a <seealso marker="#case">case</seealso> expression, except that if - the matching fails, a <c>try_clause</c> run-time error will occur.</p> + the matching fails, a <c>try_clause</c> run-time error occurs.</p> <p>An exception occurring during the evaluation of <c>Body</c> is not caught.</p> <p>The <c>try</c> expression can also be augmented with an @@ -1364,7 +1410,7 @@ after AfterBody end</code> <p><c>AfterBody</c> is evaluated after either <c>Body</c> or - <c>ExceptionBody</c> no matter which one. The evaluated value of + <c>ExceptionBody</c>, no matter which one. The evaluated value of <c>AfterBody</c> is lost; the return value of the <c>try</c> expression is the same with an <c>after</c> section as without.</p> <p>Even if an exception occurs during evaluation of <c>Body</c> or @@ -1373,13 +1419,13 @@ end</code> evaluated, so the exception from the <c>try</c> expression is the same with an <c>after</c> section as without.</p> <p>If an exception occurs during evaluation of <c>AfterBody</c> - itself it is not caught, so if <c>AfterBody</c> is evaluated after - an exception in <c>Exprs</c>, <c>Body</c> or <c>ExceptionBody</c>, + itself, it is not caught. So if <c>AfterBody</c> is evaluated after + an exception in <c>Exprs</c>, <c>Body</c>, or <c>ExceptionBody</c>, that exception is lost and masked by the exception in <c>AfterBody</c>.</p> - <p>The <c>of</c>, <c>catch</c> and <c>after</c> sections are all + <p>The <c>of</c>, <c>catch</c>, and <c>after</c> sections are all optional, as long as there is at least a <c>catch</c> or an - <c>after</c> section, so the following are valid <c>try</c> + <c>after</c> section. So the following are valid <c>try</c> expressions:</p> <code type="none"> try Exprs of @@ -1398,9 +1444,9 @@ after end try Exprs after AfterBody end</code> - <p>Example of using <c>after</c>, this code will close the file + <p>Next is an example of using <c>after</c>. This closes the file, even in the event of exceptions in <c>file:read/2</c> or in - <c>binary_to_term/1</c>, and exceptions will be the same as + <c>binary_to_term/1</c>. The exceptions are the same as without the <c>try</c>...<c>after</c>...<c>end</c> expression:</p> <code type="none"> termize_file(Name) -> @@ -1411,7 +1457,7 @@ termize_file(Name) -> after file:close(F) end.</code> - <p>Example: Using <c>try</c> to emulate <c>catch Expr</c>.</p> + <p>Next is an example of using <c>try</c> to emulate <c>catch Expr</c>:</p> <code type="none"> try Expr catch @@ -1427,7 +1473,7 @@ end</code> (Expr)</pre> <p>Parenthesized expressions are useful to override <seealso marker="#prec">operator precedences</seealso>, - for example in arithmetic expressions:</p> + for example, in arithmetic expressions:</p> <pre> 1> <input>1 + 2 * 3.</input> 7 @@ -1451,7 +1497,7 @@ end</pre> <section> <marker id="lcs"></marker> <title>List Comprehensions</title> - <p>List comprehensions are a feature of many modern functional + <p>List comprehensions is a feature of many modern functional programming languages. Subject to certain rules, they provide a succinct notation for generating elements in a list.</p> <p>List comprehensions are analogous to set comprehensions in @@ -1461,32 +1507,34 @@ end</pre> <p>List comprehensions are written with the following syntax:</p> <pre> [Expr || Qualifier1,...,QualifierN]</pre> - <p><c>Expr</c> is an arbitrary expression, and each + <p>Here, <c>Expr</c> is an arbitrary expression, and each <c>Qualifier</c> is either a generator or a filter.</p> <list type="bulleted"> <item>A <em>generator</em> is written as: <br></br> <c><![CDATA[Pattern <- ListExpr]]></c>. <br></br> -<c>ListExpr</c> must be an expression which evaluates to a +<c>ListExpr</c> must be an expression, which evaluates to a list of terms.</item> <item>A <em>bit string generator</em> is written as: <br></br> <c><![CDATA[BitstringPattern <= BitStringExpr]]></c>. <br></br> -<c>BitStringExpr</c> must be an expression which evaluates to a +<c>BitStringExpr</c> must be an expression, which evaluates to a bitstring.</item> - <item>A <em>filter</em> is an expression which evaluates to + <item>A <em>filter</em> is an expression, which evaluates to <c>true</c> or <c>false</c>.</item> </list> - <p>The variables in the generator patterns shadow variables in the function - clause surrounding the list comprehensions.</p> <p>A list comprehension + <p>The variables in the generator patterns, shadow variables in the function + clause, surrounding the list comprehensions.</p> <p>A list comprehension returns a list, where the elements are the result of evaluating <c>Expr</c> for each combination of generator list elements and bit string generator - elements for which all filters are true.</p> <p></p> <p>Example:</p> + elements, for which all filters are true.</p> + <p><em>Example:</em></p> <pre> 1> <input>[X*2 || X <- [1,2,3]].</input> [2,4,6]</pre> - <p>More examples can be found in <em>Programming Examples</em>.</p> - + <p>More examples are provoded in + <seealso marker="doc/programming_examples:list_comprehensions"> + Programming Examples.</seealso></p> </section> @@ -1500,34 +1548,35 @@ end</pre> the following syntax:</p> <pre> << BitString || Qualifier1,...,QualifierN >></pre> - <p><c>BitString</c> is a bit string expression, and each + <p>Here, <c>BitString</c> is a bit string expression and each <c>Qualifier</c> is either a generator, a bit string generator or a filter.</p> <list type="bulleted"> <item>A <em>generator</em> is written as: <br></br> <c><![CDATA[Pattern <- ListExpr]]></c>. <br></br> - <c>ListExpr</c> must be an expression which evaluates to a + <c>ListExpr</c> must be an expression that evaluates to a list of terms.</item> <item>A <em>bit string generator</em> is written as: <br></br> <c><![CDATA[BitstringPattern <= BitStringExpr]]></c>. <br></br> -<c>BitStringExpr</c> must be an expression which evaluates to a +<c>BitStringExpr</c> must be an expression that evaluates to a bitstring.</item> - <item>A <em>filter</em> is an expression which evaluates to + <item>A <em>filter</em> is an expression that evaluates to <c>true</c> or <c>false</c>.</item> </list> - <p>The variables in the generator patterns shadow variables in - the function clause surrounding the bit string comprehensions.</p> + <p>The variables in the generator patterns, shadow variables in + the function clause, surrounding the bit string comprehensions.</p> <p>A bit string comprehension returns a bit string, which is created by concatenating the results of evaluating <c>BitString</c> - for each combination of bit string generator elements for which all + for each combination of bit string generator elements, for which all filters are true.</p> - <p></p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> -1> <input><< << (X*2) >> || +1> <input><< << (X*2) >> || <<X>> <= << 1,2,3 >> >>.</input> <<2,4,6>></pre> - <p>More examples can be found in <em>Programming Examples</em>.</p> + <p>More examples are provided in + <seealso marker="doc/programming_examples:bit_syntax"> + Programming Examples.</seealso></p> </section> <section> @@ -1536,27 +1585,27 @@ end</pre> <p>A <em>guard sequence</em> is a sequence of guards, separated by semicolon (;). The guard sequence is true if at least one of - the guards is true. (The remaining guards, if any, will not be - evaluated.)<br></br> -<c>Guard1;...;GuardK</c></p> + the guards is true. (The remaining guards, if any, are not + evaluated.)</p> + <p><c>Guard1;...;GuardK</c></p> <p>A <em>guard</em> is a sequence of guard expressions, separated by comma (,). The guard is true if all guard expressions - evaluate to <c>true</c>.<br></br> -<c>GuardExpr1,...,GuardExprN</c></p> + evaluate to <c>true</c>.</p> + <p><c>GuardExpr1,...,GuardExprN</c></p> <p>The set of valid <em>guard expressions</em> (sometimes called guard tests) is a subset of the set of valid Erlang expressions. The reason for restricting the set of valid expressions is that evaluation of a guard expression must be guaranteed to be free - of side effects. Valid guard expressions are:</p> + of side effects. Valid guard expressions are the following:</p> <list type="bulleted"> - <item>the atom <c>true</c>,</item> - <item>other constants (terms and bound variables), all regarded - as false,</item> - <item>calls to the BIFs specified below,</item> - <item>term comparisons,</item> - <item>arithmetic expressions,</item> - <item>boolean expressions, and</item> - <item>short-circuit expressions (<c>andalso</c>/<c>orelse</c>).</item> + <item>The atom <c>true</c></item> + <item>Other constants (terms and bound variables), all regarded + as false</item> + <item>Calls to the BIFs specified in table <c>Type Test BIFs</c></item> + <item>Term comparisons</item> + <item>Arithmetic expressions</item> + <item>Boolean expressions</item> + <item>Short-circuit expressions (<c>andalso</c>/<c>orelse</c>)</item> </list> <table> <row> @@ -1610,13 +1659,13 @@ end</pre> <row> <cell align="left" valign="middle"><c>is_tuple/1</c></cell> </row> - <tcaption>Type Test BIFs.</tcaption> + <tcaption>Type Test BIFs</tcaption> </table> - <p>Note that most type test BIFs have older equivalents, without + <p>Notice that most type test BIFs have older equivalents, without the <c>is_</c> prefix. These old BIFs are retained for backwards - compatibility only and should not be used in new code. They are + compatibility only and are not to be used in new code. They are also only allowed at top level. For example, they are not allowed - in boolean expressions in guards.</p> + in Boolean expressions in guards.</p> <table> <row> <cell align="left" valign="middle"><c>abs(Number)</c></cell> @@ -1666,14 +1715,14 @@ end</pre> <row> <cell align="left" valign="middle"><c>tuple_size(Tuple)</c></cell> </row> - <tcaption>Other BIFs Allowed in Guard Expressions.</tcaption> + <tcaption>Other BIFs Allowed in Guard Expressions</tcaption> </table> - <p>If an arithmetic expression, a boolean expression, a + <p>If an arithmetic expression, a Boolean expression, a short-circuit expression, or a call to a guard BIF fails (because of invalid arguments), the entire guard fails. If the guard was part of a guard sequence, the next guard in the sequence (that is, - the guard following the next semicolon) will be evaluated.</p> + the guard following the next semicolon) is evaluated.</p> </section> @@ -1726,12 +1775,13 @@ end</pre> <cell align="left" valign="middle">catch</cell> <cell align="left" valign="middle"> </cell> </row> - <tcaption>Operator Precedence.</tcaption> + <tcaption>Operator Precedence</tcaption> </table> <p>When evaluating an expression, the operator with the highest priority is evaluated first. Operators with the same priority - are evaluated according to their associativity. Example: - The left associative arithmetic operators are evaluated left to + are evaluated according to their associativity.</p> + <p><em>Example:</em></p> + <p>The left associative arithmetic operators are evaluated left to right:</p> <pre> <input>6 + 5 * 4 - 3 / 2</input> evaluates to diff --git a/system/doc/reference_manual/functions.xml b/system/doc/reference_manual/functions.xml index 9498ef1402..8cf4da1b8b 100644 --- a/system/doc/reference_manual/functions.xml +++ b/system/doc/reference_manual/functions.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -38,7 +38,7 @@ clause body, separated by <c>-></c>.</p> <p>A clause <em>head</em> consists of the function name, an argument list, and an optional guard sequence - beginning with the keyword <c>when</c>.</p> + beginning with the keyword <c>when</c>:</p> <pre> Name(Pattern11,...,Pattern1N) [when GuardSeq1] -> Body1; @@ -48,9 +48,9 @@ Name(PatternK1,...,PatternKN) [when GuardSeqK] -> <p>The function name is an atom. Each argument is a pattern.</p> <p>The number of arguments <c>N</c> is the <em>arity</em> of the function. A function is uniquely defined by the module name, - function name and arity. That is, two functions with the same + function name, and arity. That is, two functions with the same name and in the same module, but with different arities are two - completely different functions.</p> + different functions.</p> <p>A function named <c>f</c> in the module <c>m</c> and with arity <c>N</c> is often denoted as <c>m:f/N</c>.</p> <p>A clause <em>body</em> consists of a sequence of expressions @@ -60,8 +60,8 @@ Expr1, ..., ExprN</pre> <p>Valid Erlang expressions and guard sequences are described in - <seealso marker="expressions">Erlang Expressions</seealso>.</p> - <p>Example:</p> + <seealso marker="expressions">Expressions</seealso>.</p> + <p><em>Example:</em></p> <pre> fact(N) when N>0 -> % first clause head N * fact(N-1); % first clause body @@ -75,23 +75,23 @@ fact(0) -> % second clause head <title>Function Evaluation</title> <p>When a function <c>m:f/N</c> is called, first the code for the function is located. If the function cannot be found, an - <c>undef</c> run-time error will occur. Note that the function + <c>undef</c> runtime error occurs. Notice that the function must be exported to be visible outside the module it is defined in.</p> <p>If the function is found, the function clauses are scanned - sequentially until a clause is found that fulfills the following - two conditions:</p> + sequentially until a clause is found that fulfills both of + the following two conditions:</p> <list type="ordered"> - <item>the patterns in the clause head can be successfully - matched against the given arguments, and</item> - <item>the guard sequence, if any, is true.</item> + <item>The patterns in the clause head can be successfully + matched against the given arguments.</item> + <item>The guard sequence, if any, is true.</item> </list> <p>If such a clause cannot be found, a <c>function_clause</c> - run-time error will occur.</p> + runtime error occurs.</p> <p>If such a clause is found, the corresponding clause body is evaluated. That is, the expressions in the body are evaluated sequentially and the value of the last expression is returned.</p> - <p>Example: Consider the function <c>fact</c>:</p> + <p>Consider the function <c>fact</c>:</p> <pre> -module(m). -export([fact/1]). @@ -100,17 +100,17 @@ fact(N) when N>0 -> N * fact(N-1); fact(0) -> 1.</pre> - <p>Assume we want to calculate factorial for 1:</p> + <p>Assume that you want to calculate the factorial for 1:</p> <pre> 1> <input>m:fact(1).</input></pre> <p>Evaluation starts at the first clause. The pattern <c>N</c> is - matched against the argument 1. The matching succeeds and - the guard (<c>N>0</c>) is true, thus <c>N</c> is bound to 1 and + matched against argument 1. The matching succeeds and + the guard (<c>N>0</c>) is true, thus <c>N</c> is bound to 1, and the corresponding body is evaluated:</p> <pre> <input>N * fact(N-1)</input> => (N is bound to 1) <input>1 * fact(0)</input></pre> - <p>Now <c>fact(0)</c> is called and the function clauses are + <p>Now, <c>fact(0)</c> is called, and the function clauses are scanned sequentially again. First, the pattern <c>N</c> is matched against 0. The matching succeeds, but the guard (<c>N>0</c>) is false. Second, the pattern 0 is matched against @@ -121,48 +121,51 @@ fact(0) -> <input>1</input></pre> <p>Evaluation has succeed and <c>m:fact(1)</c> returns 1.</p> <p>If <c>m:fact/1</c> is called with a negative number as - argument, no clause head will match. A <c>function_clause</c> - run-time error will occur.</p> + argument, no clause head matches. A <c>function_clause</c> + runtime error occurs.</p> </section> <section> <title>Tail recursion</title> <p>If the last expression of a function body is a function call, - a <em>tail recursive</em> call is done so that no system - resources for example call stack are consumed. This means - that an infinite loop can be done if it uses tail recursive + a <em>tail recursive</em> call is done. + This is to ensure that no system + resources, for example, call stack, are consumed. This means + that an infinite loop can be done if it uses tail-recursive calls.</p> - <p>Example:</p> + <p><em>Example:</em></p> <pre> loop(N) -> io:format("~w~n", [N]), loop(N+1).</pre> - <p>As a counter-example see the factorial example above - that is not tail recursive since a multiplication is done + <p>The earlier factorial example can act as a counter-example. + It is not tail-recursive, since a multiplication is done on the result of the recursive call to <c>fact(N-1)</c>.</p> </section> <section> - <title>Built-In Functions, BIFs</title> - <p><em>Built-in functions</em>, BIFs, are implemented in C code in - the runtime system and do things that are difficult or impossible - to implement in Erlang. Most of the built-in functions belong - to the module <c>erlang</c> but there are also built-in functions + <title>Built-In Functions (BIFs)</title> + <p>BIFs are implemented in C code in + the runtime system. BIFs do things that are difficult or impossible + to implement in Erlang. Most of the BIFs belong + to the module <c>erlang</c> but there are also BIFs belonging to a few other modules, for example <c>lists</c> and <c>ets</c>.</p> - <p>The most commonly used BIFs belonging to <c>erlang</c> are - <em>auto-imported</em>, they do not need to be prefixed with - the module name. Which BIFs are auto-imported is specified in - <c>erlang(3)</c>. For example, standard type conversion BIFs like + <p>The most commonly used BIFs belonging to <c>erlang(3)</c> are + <em>auto-imported</em>. They do not need to be prefixed with + the module name. Which BIFs that are auto-imported is specified in the + <seealso marker="erts:erlang">erlang(3)</seealso> module in ERTS. + For example, standard-type conversion BIFs like <c>atom_to_list</c> and BIFs allowed in guards can be called - without specifying the module name. Examples:</p> + without specifying the module name.</p> + <p><em>Examples:</em></p> <pre> 1> <input>tuple_size({a,b,c}).</input> 3 2> <input>atom_to_list('Erlang').</input> "Erlang"</pre> - <p>Note that normally it is the set of auto-imported built-in - functions that is referred to when talking about 'BIFs'.</p> + <p>Notice that it is normally the set of auto-imported BIFs + that are referred to when talking about 'BIFs'.</p> </section> </chapter> diff --git a/system/doc/reference_manual/introduction.xml b/system/doc/reference_manual/introduction.xml index 36bec17825..ee8b82e60f 100644 --- a/system/doc/reference_manual/introduction.xml +++ b/system/doc/reference_manual/introduction.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2014</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,20 +28,38 @@ <rev></rev> <file>introduction.xml</file> </header> + <marker id="erlang ref manual"></marker> + + <p>This section is the Erlang reference manual. It describes the + Erlang programming language. </p> <section> <title>Purpose</title> - <p>This reference manual describes the Erlang programming - language. The focus is on the language itself, not - the implementation. The language constructs are described in - text and with examples rather than formally specified, with - the intention to make the manual more readable. - The manual is not intended as a tutorial.</p> - <p>Information about this implementation of Erlang can be found, for - example, in <em>System Principles</em> (starting and stopping, - boot scripts, code loading, error logging, creating target - systems), <em>Efficiency Guide</em> (memory consumption, system - limits) and <em>ERTS User's Guide</em> (crash dumps, drivers).</p> + <p>The focus of the Erlang reference manual is on the language itself, + not the implementation of it. The language constructs are described in + text and with examples rather than formally specified. This is + to make the manual more readable. + The Erlang reference manual is not intended as a tutorial.</p> + <p>Information about implementation of Erlang can, for example, be found, + in the following:</p> + <list type="bulleted"> + <item><p><seealso marker="doc/system_principles:system_principles"> + System Principles</seealso></p> + <p>Starting and stopping, boot scripts, code loading, + <seealso marker="doc/system_principles:error_logging"> + error logging</seealso>, + <seealso marker="doc/system_principles:create_target"> + creating target systems</seealso></p> + </item> + <item><p><seealso marker="doc/efficiency_guide:advanced"> + Efficiency Guide</seealso></p> + <p>Memory consumption, system limits</p> + </item> + <item><p>ERTS User's Guide</p> + <p><seealso marker="erts:crash_dump">Crash dumps</seealso>, + <seealso marker="erts:driver">drivers</seealso></p> + </item> + </list> </section> <section> @@ -53,13 +71,13 @@ <section> <title>Document Conventions</title> - <p>In the document, the following terminology is used:</p> + <p>In this section, the following terminology is used:</p> <list type="bulleted"> <item>A <em>sequence</em> is one or more items. For example, a clause body consists of a sequence of expressions. This means that there must be at least one expression.</item> <item>A <em>list</em> is any number of items. For example, - an argument list can consist of zero, one or more arguments.</item> + an argument list can consist of zero, one, or more arguments.</item> </list> <p>If a feature has been added recently, in Erlang 5.0/OTP R7 or later, this is mentioned in the text.</p> @@ -68,15 +86,16 @@ <section> <title>Complete List of BIFs</title> <p>For a complete list of BIFs, their arguments and return values, - refer to <c>erlang(3)</c>.</p> + see <seealso marker="erts:erlang#process_flag/2">erlang(3)</seealso> + manual page in ERTS.</p> </section> <section> <title>Reserved Words</title> <p>The following are reserved words in Erlang:</p> - <p>after and andalso band begin bnot bor bsl bsr bxor case catch + <p><c>after and andalso band begin bnot bor bsl bsr bxor case catch cond div end fun if let not of or orelse receive rem try - when xor</p> + when xor</c></p> </section> </chapter> diff --git a/system/doc/reference_manual/macros.xml b/system/doc/reference_manual/macros.xml index 9fd0b0f287..01994aae5e 100644 --- a/system/doc/reference_manual/macros.xml +++ b/system/doc/reference_manual/macros.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -21,7 +21,7 @@ </legalnotice> - <title>The Preprocessor</title> + <title>Preprocessor</title> <prepared></prepared> <docno></docno> <date></date> @@ -31,17 +31,17 @@ <section> <title>File Inclusion</title> - <p>A file can be included in the following way:</p> + <p>A file can be included as follows:</p> <pre> -include(File). -include_lib(File).</pre> - <p><c>File</c>, a string, should point out a file. The contents of - this file are included as-is, at the position of the directive.</p> + <p><c>File</c>, a string, is to point out a file. The contents of + this file are included as is, at the position of the directive.</p> <p>Include files are typically used for record and macro definitions that are shared by several modules. It is - recommended that the file name extension <c>.hrl</c> be used - for include files.</p> - <p><c>File</c> may start with a path component <c>$VAR</c>, for + recommended to use the file name extension <c>.hrl</c> for + include files.</p> + <p><c>File</c> can start with a path component <c>$VAR</c>, for some string <c>VAR</c>. If that is the case, the value of the environment variable <c>VAR</c> as returned by <c>os:getenv(VAR)</c> is substituted for <c>$VAR</c>. If @@ -49,21 +49,29 @@ as is.</p> <p>If the filename <c>File</c> is absolute (possibly after variable substitution), the include file with that name is - included. Otherwise, the specified file is searched for in - the current working directory, in the same directory as - the module being compiled, and in the directories given by - the <c>include</c> option, in that order. - See <c>erlc(1)</c> and <c>compile(3)</c> for details.</p> - <p>Examples:</p> + included. Otherwise, the specified file is searched for + in the following directories, and in this order:</p> + <list type="ordered"> + <item>The current working directory</item> + <item>The directory where the module is being compiled</item> + <item>The directories given by the <c>include</c> option</item> + </list> + <p>For details, see the + <seealso marker="erts:erlc">erlc(1)</seealso> manual page + in ERTS and + <seealso marker="compiler:compile">compile(3)</seealso> + manual page in Compiler.</p> + <p><em>Examples:</em></p> <pre> -include("my_records.hrl"). -include("incdir/my_records.hrl"). -include("/home/user/proj/my_records.hrl"). -include("$PROJ_ROOT/my_records.hrl").</pre> - <p><c>include_lib</c> is similar to <c>include</c>, but should not + <p><c>include_lib</c> is similar to <c>include</c>, but is not to point out an absolute file. Instead, the first path component (possibly after variable substitution) is assumed to be - the name of an application. Example:</p> + the name of an application.</p> + <p><em>Example:</em></p> <pre> -include_lib("kernel/include/file.hrl").</pre> <p>The code server uses <c>code:lib_dir(kernel)</c> to find @@ -74,7 +82,7 @@ <section> <title>Defining and Using Macros</title> - <p>A macro is defined the following way:</p> + <p>A macro is defined as follows:</p> <code type="none"> -define(Const, Replacement). -define(Func(Var1,...,VarN), Replacement).</code> @@ -83,33 +91,34 @@ come before any usage of the macro.</p> <p>If a macro is used in several modules, it is recommended that the macro definition is placed in an include file.</p> - <p>A macro is used the following way:</p> + <p>A macro is used as follows:</p> <code type="none"> ?Const ?Func(Arg1,...,ArgN)</code> <p>Macros are expanded during compilation. A simple macro - <c>?Const</c> will be replaced with <c>Replacement</c>. - Example:</p> + <c>?Const</c> is replaced with <c>Replacement</c>.</p> + <p><em>Example:</em></p> <code type="none"> -define(TIMEOUT, 200). ... call(Request) -> server:call(refserver, Request, ?TIMEOUT).</code> - <p>This will be expanded to:</p> + <p>This is expanded to:</p> <code type="none"> call(Request) -> server:call(refserver, Request, 200).</code> - <p>A macro <c>?Func(Arg1,...,ArgN)</c> will be replaced with + <p>A macro <c>?Func(Arg1,...,ArgN)</c> is replaced with <c>Replacement</c>, where all occurrences of a variable <c>Var</c> from the macro definition are replaced with the corresponding - argument <c>Arg</c>. Example:</p> + argument <c>Arg</c>.</p> + <p><em>Example:</em></p> <code type="none"> -define(MACRO1(X, Y), {a, X, b, Y}). ... bar(X) -> ?MACRO1(a, b), ?MACRO1(X, 123)</code> - <p>This will be expanded to:</p> + <p>This is expanded to:</p> <code type="none"> bar(X) -> {a,a,b,b}, @@ -154,7 +163,7 @@ bar(X) -> -define(F0(), c). -define(F1(A), A). -define(C, m:f).</code> - <p>the following will not work:</p> + <p>the following does not work:</p> <code type="none"> f0() -> ?F0. % No, an empty list of arguments expected. @@ -165,7 +174,7 @@ f1(A) -> <code> f() -> ?C().</code> - <p>will expand to</p> + <p>is expanded to</p> <code> f() -> m:f().</code> @@ -185,7 +194,7 @@ f() -> defined.</item> <tag><c>-else.</c></tag> <item>Only allowed after an <c>ifdef</c> or <c>ifndef</c> - directive. If that condition was false, the lines following + directive. If that condition is false, the lines following <c>else</c> are evaluated instead.</item> <tag><c>-endif.</c></tag> <item>Specifies the end of an <c>ifdef</c> or <c>ifndef</c> @@ -194,7 +203,7 @@ f() -> <note> <p>The macro directives cannot be used inside functions.</p> </note> - <p>Example:</p> + <p><em>Example:</em></p> <code type="none"> -module(m). ... @@ -206,7 +215,7 @@ f() -> -endif. ...</code> - <p>When trace output is desired, <c>debug</c> should be defined + <p>When trace output is desired, <c>debug</c> is to be defined when the module <c>m</c> is compiled:</p> <pre> % <input>erlc -Ddebug m.erl</input> @@ -215,18 +224,18 @@ or 1> <input>c(m, {d, debug}).</input> {ok,m}</pre> - <p><c>?LOG(Arg)</c> will then expand to a call to <c>io:format/2</c> + <p><c>?LOG(Arg)</c> is then expanded to a call to <c>io:format/2</c> and provide the user with some simple trace output.</p> </section> <section> <title>Stringifying Macro Arguments</title> <p>The construction <c>??Arg</c>, where <c>Arg</c> is a macro - argument, will be expanded to a string containing the tokens of + argument, is expanded to a string containing the tokens of the argument. This is similar to the <c>#arg</c> stringifying construction in C.</p> <p>The feature was added in Erlang 5.0/OTP R7.</p> - <p>Example:</p> + <p><em>Example:</em></p> <code type="none"> -define(TESTCALL(Call), io:format("Call ~s: ~w~n", [??Call, Call])). @@ -236,7 +245,7 @@ or <code type="none"> io:format("Call ~s: ~w~n",["myfunction ( 1 , 2 )",myfunction(1,2)]), io:format("Call ~s: ~w~n",["you : function ( 2 , 1 )",you:function(2,1)]).</code> - <p>That is, a trace output with both the function called and + <p>That is, a trace output, with both the function called and the resulting value.</p> </section> </chapter> diff --git a/system/doc/reference_manual/modules.xml b/system/doc/reference_manual/modules.xml index 5cb0c11371..39c739a146 100644 --- a/system/doc/reference_manual/modules.xml +++ b/system/doc/reference_manual/modules.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2014</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -33,7 +33,8 @@ <title>Module Syntax</title> <p>Erlang code is divided into <em>modules</em>. A module consists of a sequence of attributes and function declarations, each - terminated by period (.). Example:</p> + terminated by period (.).</p> + <p><em>Example:</em></p> <pre> -module(m). % module attribute -export([fact/1]). % module attribute @@ -42,50 +43,52 @@ fact(N) when N>0 -> % beginning of function declaration N * fact(N-1); % | fact(0) -> % | 1. % end of function declaration</pre> - <p>See the <seealso marker="functions">Functions</seealso> chapter - for a description of function declarations.</p> + <p>For a description of function declarations, see + <seealso marker="functions">Function Declaration Syntax</seealso>.</p> </section> <section> <title>Module Attributes</title> <p>A <em>module attribute</em> defines a certain property of a - module. A module attribute consists of a tag and a value.</p> + module.</p> + <p>A module attribute consists of a tag and a value:</p> <pre> -Tag(Value).</pre> <p><c>Tag</c> must be an atom, while <c>Value</c> must be a literal term. As a convenience in user-defined attributes, if the literal term <c>Value</c> has the syntax <c>Name/Arity</c> (where <c>Name</c> is an atom and <c>Arity</c> a positive integer), - the term <c>Name/Arity</c> will be translated to <c>{Name,Arity}</c>.</p> + the term <c>Name/Arity</c> is translated to <c>{Name,Arity}</c>.</p> <p>Any module attribute can be specified. The attributes are stored in the compiled code and can be retrieved by calling - <c>Module:module_info(attributes)</c> or by using - <seealso marker="stdlib:beam_lib#chunks/2">beam_lib(3)</seealso>.</p> + <c>Module:module_info(attributes)</c>, or by using the module + <seealso marker="stdlib:beam_lib#chunks/2">beam_lib(3)</seealso> + in STDLIB.</p> - <p>There are several module attributes with predefined meanings, - some of which have arity two, but user-defined module + <p>Several module attributes have predefined meanings. + Some of them have arity two, but user-defined module attributes must have arity one.</p> <section> <title>Pre-Defined Module Attributes</title> - <p>Pre-defined module attributes should be placed before any + <p>Pre-defined module attributes is to be placed before any function declaration.</p> <taglist> <tag><c>-module(Module).</c></tag> <item> <p>Module declaration, defining the name of the module. - The name <c>Module</c>, an atom, should be the same as - the file name minus the extension <c>erl</c>. Otherwise - <seealso marker="code_loading#loading">code loading</seealso> will + The name <c>Module</c>, an atom, is to be same as + the file name minus the extension <c>.erl</c>. Otherwise + <seealso marker="code_loading#loading">code loading</seealso> does not work as intended.</p> - <p>This attribute should be specified first and is the only - attribute which is mandatory.</p> + <p>This attribute is to be specified first and is the only + mandatory attribute.</p> </item> <tag><c>-export(Functions).</c></tag> <item> - <p>Exported functions. Specifies which of the functions - defined within the module that are visible outside + <p>Exported functions. Specifies which of the functions, + defined within the module, that are visible from outside the module.</p> <p><c>Functions</c> is a list <c>[Name1/Arity1, ..., NameN/ArityN]</c>, where each @@ -93,32 +96,37 @@ fact(0) -> % | </item> <tag><c>-import(Module,Functions).</c></tag> <item> - <p>Imported functions. Imported functions can be called - the same way as local functions, that is without any module + <p>Imported functions. Can be called + the same way as local functions, that is, without any module prefix.</p> <p><c>Module</c>, an atom, specifies which module to import functions from. <c>Functions</c> is a list similar as for - <c>export</c> above.</p> + <c>export</c>.</p> </item> <tag><c>-compile(Options).</c></tag> <item> - <p>Compiler options. <c>Options</c>, which is a single option - or a list of options, will be added to the option list when - compiling the module. See <c>compile(3)</c>.</p> + <p>Compiler options. <c>Options</c> is a single option + or a list of options. + This attribute is added to the option list when + compiling the module. See the <seealso marker="compiler:compile"> + compile(3)</seealso> manual page in Compiler.</p> </item> <tag><c>-vsn(Vsn).</c></tag> <item> <p>Module version. <c>Vsn</c> is any literal term and can be - retrieved using <c>beam_lib:version/1</c>, see - <seealso marker="stdlib:beam_lib#version/1">beam_lib(3)</seealso>.</p> + retrieved using <c>beam_lib:version/1</c>, see the + <seealso marker="stdlib:beam_lib#version/1">beam_lib(3)</seealso> + manual page in STDLIB.</p> <p>If this attribute is not specified, the version defaults to the MD5 checksum of the module.</p> </item> <tag><c>-on_load(Function).</c></tag> <item> - <p>Names a function that should be run automatically when a - module a loaded. See <seealso marker="code_loading#on_load"> - code loading</seealso> for more information.</p> + <p>This attribute names a function that is to be run + automatically when a + module is loaded. For more information, see + <seealso marker="code_loading#on_load"> + Running a Function When a Module is Loaded</seealso>.</p> </item> </taglist> </section> @@ -130,9 +138,14 @@ fact(0) -> % | <pre> -behaviour(Behaviour).</pre> <p>The atom <c>Behaviour</c> gives the name of the behaviour, - which can be a user defined behaviour or one of the OTP - standard behaviours <c>gen_server</c>, <c>gen_fsm</c>, - <c>gen_event</c> or <c>supervisor</c>.</p> + which can be a user-defined behaviour or one of the following OTP + standard behaviours:</p> + <list type="bulleted"> + <item><c>gen_server</c></item> + <item><c>gen_fsm</c></item> + <item><c>gen_event</c></item> + <item><c>supervisor</c></item> + </list> <p>The spelling <c>behavior</c> is also accepted.</p> <p>The callback functions of the module can be specified either directly by the exported function <c>behaviour_info/1</c>:</p> @@ -142,7 +155,7 @@ behaviour_info(callbacks) -> Callbacks.</pre> function:</p> <pre> -callback Name(Arguments) -> Result.</pre> - <p>where <c>Arguments</c> is a list of zero or more arguments. + <p>Here, <c>Arguments</c> is a list of zero or more arguments. The <c>-callback</c> attribute is to be preferred since the extra type information can be used by tools to produce documentation or find discrepancies.</p> @@ -153,7 +166,7 @@ behaviour_info(callbacks) -> Callbacks.</pre> <section> <title>Record Definitions</title> - <p>The same syntax as for module attributes is used by + <p>The same syntax as for module attributes is used for record definitions:</p> <pre> -record(Record,Fields).</pre> @@ -163,7 +176,7 @@ behaviour_info(callbacks) -> Callbacks.</pre> </section> <section> - <title>The Preprocessor</title> + <title>Preprocessor</title> <p>The same syntax as for module attributes is used by the preprocessor, which supports file inclusion, macros, and conditional compilation:</p> @@ -171,7 +184,7 @@ behaviour_info(callbacks) -> Callbacks.</pre> -include("SomeFile.hrl"). -define(Macro,Replacement).</pre> - <p>Read more in <seealso marker="macros">The Preprocessor</seealso>.</p> + <p>Read more in <seealso marker="macros">Preprocessor</seealso>.</p> </section> <section> @@ -180,17 +193,17 @@ behaviour_info(callbacks) -> Callbacks.</pre> changing the pre-defined macros <c>?FILE</c> and <c>?LINE</c>:</p> <pre> -file(File, Line).</pre> - <p>This attribute is used by tools such as Yecc to inform the - compiler that the source program was generated by another tool - and indicates the correspondence of source files to lines of - the original user-written file from which the source program - was produced.</p> + <p>This attribute is used by tools, such as Yecc, to inform the + compiler that the source program is generated by another tool. + It also indicates the correspondence of source files to lines of + the original user-written file, from which the source program + is produced.</p> </section> <section> <title>Types and function specifications</title> <p>A similar syntax as for module attributes is used for - specifying types and function specifications. + specifying types and function specifications: </p> <pre> -type my_type() :: atom() | integer(). @@ -200,32 +213,36 @@ behaviour_info(callbacks) -> Callbacks.</pre> <p> The description is based on <url href="http://www.erlang.org/eeps/eep-0008.html">EEP8 - - Types and function specifications</url> - which will not be further updated. + Types and function specifications</url>, + which is not to be further updated. </p> </section> </section> <section> <title>Comments</title> - <p>Comments may be placed anywhere in a module except within strings - and quoted atoms. The comment begins with the character "%", + <p>Comments can be placed anywhere in a module except within strings + and quoted atoms. A comment begins with the character "%", continues up to, but does not include the next end-of-line, and - has no effect. Note that the terminating end-of-line has + has no effect. Notice that the terminating end-of-line has the effect of white space.</p> </section> <section> - <title>The module_info/0 and module_info/1 functions</title> + <title>module_info/0 and module_info/1 functions</title> <p>The compiler automatically inserts the two special, exported - functions into each module: <c>Module:module_info/0</c> and - <c>Module:module_info/1</c>. These functions can be called to - retrieve information about the module.</p> + functions into each module:</p> + <list type="bulleted"> + <item><c>Module:module_info/0</c></item> + <item><c>Module:module_info/1</c></item> + </list> + <p>These functions can be called to retrieve information + about the module.</p> <section> <title>module_info/0</title> - <p>The <c>module_info/0</c> function in each module returns + <p>The <c>module_info/0</c> function in each module, returns a list of <c>{Key,Value}</c> tuples with information about the module. Currently, the list contain tuples with the following <c>Key</c>s: <c>module</c>, <c>attributes</c>, <c>compile</c>, @@ -235,7 +252,7 @@ behaviour_info(callbacks) -> Callbacks.</pre> <section> <title>module_info/1</title> - <p>The call <c>module_info(Key)</c>, where key is an atom, + <p>The call <c>module_info(Key)</c>, where <c>Key</c> is an atom, returns a single piece of information about the module.</p> <p>The following values are allowed for <c>Key</c>:</p> @@ -243,44 +260,46 @@ behaviour_info(callbacks) -> Callbacks.</pre> <taglist> <tag><c>module</c></tag> <item> - <p>Return an atom representing the module name.</p> + <p>Returns an atom representing the module name.</p> </item> <tag><c>attributes</c></tag> <item> - <p>Return a list of <c>{AttributeName,ValueList}</c> tuples, + <p>Returns a list of <c>{AttributeName,ValueList}</c> tuples, where <c>AttributeName</c> is the name of an attribute, - and <c>ValueList</c> is a list of values. Note: a given - attribute may occur more than once in the list with different + and <c>ValueList</c> is a list of values. Notice that a given + attribute can occur more than once in the list with different values if the attribute occurs more than once in the module.</p> - <p>The list of attributes will be empty if - the module has been stripped with - <seealso marker="stdlib:beam_lib#strip/1">beam_lib(3)</seealso>.</p> + <p>The list of attributes becomes empty if + the module is stripped with the + <seealso marker="stdlib:beam_lib#strip/1">beam_lib(3)</seealso> + module (in STDLIB).</p> </item> <tag><c>compile</c></tag> <item> - <p>Return a list of tuples containing information about - how the module was compiled. This list will be empty if - the module has been stripped with - <seealso marker="stdlib:beam_lib#strip/1">beam_lib(3)</seealso>.</p> + <p>Returns a list of tuples with information about + how the module was compiled. This list is empty if + the module has been stripped with the + <seealso marker="stdlib:beam_lib#strip/1">beam_lib(3)</seealso> + module (in STDLIB).</p> </item> <tag><c>md5</c></tag> <item> - <p>Return a binary representing the MD5 checksum of the module.</p> + <p>Returns a binary representing the MD5 checksum of the module.</p> </item> <tag><c>exports</c></tag> <item> - <p>Return a list of <c>{Name,Arity}</c> tuples with + <p>Returns a list of <c>{Name,Arity}</c> tuples with all exported functions in the module.</p> </item> <tag><c>functions</c></tag> <item> - <p>Return a list of <c>{Name,Arity}</c> tuples with + <p>Returns a list of <c>{Name,Arity}</c> tuples with all functions in the module.</p> </item> </taglist> diff --git a/system/doc/reference_manual/patterns.xml b/system/doc/reference_manual/patterns.xml index 1611002fa1..2163583636 100644 --- a/system/doc/reference_manual/patterns.xml +++ b/system/doc/reference_manual/patterns.xml @@ -40,7 +40,7 @@ <seealso marker="expressions#term">term</seealso>. If the matching succeeds, any unbound variables in the pattern become bound. If the matching fails, a run-time error occurs.</p> - <p>Examples:</p> + <p><em>Examples:</em></p> <pre> 1> <input>X.</input> ** 1: variable 'X' is unbound ** diff --git a/system/doc/reference_manual/ports.xml b/system/doc/reference_manual/ports.xml index 621af10624..e5dc99641b 100644 --- a/system/doc/reference_manual/ports.xml +++ b/system/doc/reference_manual/ports.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2004</year><year>2013</year> + <year>2004</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,9 +28,12 @@ <rev></rev> <file>ports.xml</file> </header> - <p>Examples of how to use ports and port drivers can be found in - <em>Interoperability Tutorial</em>. The BIFs mentioned are as usual - documented in <c>erlang(3)</c>.</p> + <p>Examples of how to use ports and port drivers are provided in + <seealso marker="doc/tutorial:introduction#interoperability tutorial"> + Interoperability Tutorial</seealso>. + For information about the BIFs mentioned, see the + <seealso marker="erts:erlang">erlang(3)</seealso> manual + page in ERTS.</p> <section> <title>Ports</title> @@ -39,29 +42,34 @@ provide a byte-oriented interface to an external program. When a port has been created, Erlang can communicate with it by sending and receiving lists of bytes, including binaries.</p> - <p>The Erlang process which creates a port is said to be + <p>The Erlang process creating a port is said to be the <em>port owner</em>, or the <em>connected process</em> of - the port. All communication to and from the port should go via - the port owner. If the port owner terminates, so will the port + the port. All communication to and from the port must go through + the port owner. If the port owner terminates, so does the port (and the external program, if it is written correctly).</p> <p>The external program resides in another OS process. By default, - it should read from standard input (file descriptor 0) and write + it reads from standard input (file descriptor 0) and writes to standard output (file descriptor 1). The external program - should terminate when the port is closed.</p> + is to terminate when the port is closed.</p> </section> <section> <title>Port Drivers</title> - <p>It is also possible to write a driver in C according to certain + <p>It is possible to write a driver in C according to certain principles and dynamically link it to the Erlang runtime system. The linked-in driver looks like a port from the Erlang programmer's point of view and is called a <em>port driver</em>.</p> <warning> - <p>An erroneous port driver will cause the entire Erlang runtime + <p>An erroneous port driver causes the entire Erlang runtime system to leak memory, hang or crash.</p> </warning> - <p>Port drivers are documented in <c>erl_driver(4)</c>, - <c>driver_entry(1)</c> and <c>erl_ddll(3)</c>.</p> + <p>For information about port drivers, see the + <seealso marker="erts:erl_driver">erl_driver(4)</seealso> + manual page in ERTS, + <seealso marker="erts:driver_entry">driver_entry(1)</seealso> + manual page in ERTS, and + <seealso marker="kernel:erl_ddll">erl_ddll(3)</seealso> + manual page in Kernel.</p> </section> <section> @@ -70,53 +78,74 @@ <table> <row> <cell align="left" valign="middle"><c>open_port(PortName, PortSettings</c></cell> - <cell align="left" valign="middle">Returns a port identifier <c>Port</c>as the result of opening a new Erlang port. Messages can be sent to and received from a port identifier, just like a pid. Port identifiers can also be linked to or registered under a name using <c>link/1</c>and <c>register/2</c>.</cell> + <cell align="left" valign="middle">Returns a port identifier + <c>Port</c> as the result of opening a new Erlang port. + Messages can be sent to, and received from, a port identifier, + just like a pid. Port identifiers can also be linked to + using <c>link/1</c>, or registered under a name using + <c>register/2</c>.</cell> </row> - <tcaption>Port Creation BIF.</tcaption> + <tcaption>Port Creation BIF</tcaption> </table> <p><c>PortName</c> is usually a tuple <c>{spawn,Command}</c>, where the string <c>Command</c> is the name of the external program. - The external program runs outside the Erlang workspace unless a - port driver with the name <c>Command</c> is found. If found, that - driver is started.</p> + The external program runs outside the Erlang workspace, unless a + port driver with the name <c>Command</c> is found. If <c>Command</c> + is found, that driver is started.</p> <p><c>PortSettings</c> is a list of settings (options) for the port. - The list typically contains at least a tuple <c>{packet,N}</c> + The list typically contains at least a tuple <c>{packet,N}</c>, which specifies that data sent between the port and the external program are preceded by an N-byte length indicator. Valid values - for N are 1, 2 or 4. If binaries should be used instead of lists + for N are 1, 2, or 4. If binaries are to be used instead of lists of bytes, the option <c>binary</c> must be included.</p> <p>The port owner <c>Pid</c> can communicate with the port <c>Port</c> by sending and receiving messages. (In fact, any process can send the messages to the port, but the port owner must be identified in the message).</p> - <p>As of OTP-R16 messages sent to ports are delivered truly + <p>As of Erlang/OTP R16, messages sent to ports are delivered truly asynchronously. The underlying implementation previously delivered messages to ports synchronously. Message passing has - however always been documented as an asynchronous operation, so - this should not be an issue for an Erlang program communicating - with ports, unless false assumptions about ports has been made.</p> - <p>Below, <c>Data</c> must be an I/O list. An I/O list is a binary - or a (possibly deep) list of binaries or integers in the range - 0..255.</p> + however always been documented as an asynchronous operation. Hence, + this is not to be an issue for an Erlang program communicating + with ports, unless false assumptions about ports have been made.</p> + <p>In the following tables of examples, <c>Data</c> must be an I/O list. + An I/O list is a binary or a (possibly deep) list of binaries + or integers in the range 0..255:</p> <table> <row> + <cell align="left" valign="middle"><em>Message</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>{Pid,{command,Data}}</c></cell> - <cell align="left" valign="middle">Sends <c>Data</c>to the port.</cell> + <cell align="left" valign="middle">Sends <c>Data</c> to the port.</cell> </row> <row> <cell align="left" valign="middle"><c>{Pid,close}</c></cell> - <cell align="left" valign="middle">Closes the port. Unless the port is already closed, the port replies with <c>{Port,closed}</c>when all buffers have been flushed and the port really closes.</cell> + <cell align="left" valign="middle">Closes the port. Unless the + port is already closed, the port replies with + <c>{Port,closed}</c> when all buffers have been flushed + and the port really closes.</cell> </row> <row> <cell align="left" valign="middle"><c>{Pid,{connect,NewPid}}</c></cell> - <cell align="left" valign="middle">Sets the port owner of <c>Port</c>to <c>NewPid</c>. Unless the port is already closed, the port replies with<c>{Port,connected}</c>to the old port owner. Note that the old port owner is still linked to the port, but the new port owner is not.</cell> + <cell align="left" valign="middle">Sets the port owner of + <c>Port</c>to <c>NewPid</c>. Unless the port is already closed, + the port replies with<c>{Port,connected}</c> to the old + port owner. Note that the old port owner is still linked + to the port, but the new port owner is not.</cell> </row> - <tcaption>Messages Sent To a Port.</tcaption> + <tcaption>Messages Sent To a Port</tcaption> </table> + <p></p> <table> <row> + <cell align="left" valign="middle"><em>Message</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>{Port,{data,Data}}</c></cell> - <cell align="left" valign="middle"><c>Data</c>is received from the external program.</cell> + <cell align="left" valign="middle"><c>Data</c> is received from the external program.</cell> </row> <row> <cell align="left" valign="middle"><c>{Port,closed}</c></cell> @@ -124,20 +153,24 @@ </row> <row> <cell align="left" valign="middle"><c>{Port,connected}</c></cell> - <cell align="left" valign="middle">Reply to <c>Port ! {Pid,{connect,NewPid}}</c></cell> + <cell align="left" valign="middle">Reply to <c>Port ! {Pid,{connect,NewPid}}</c>.</cell> </row> <row> <cell align="left" valign="middle"><c>{'EXIT',Port,Reason}</c></cell> <cell align="left" valign="middle">If the port has terminated for some reason.</cell> </row> - <tcaption>Messages Received From a Port.</tcaption> + <tcaption>Messages Received From a Port</tcaption> </table> <p>Instead of sending and receiving messages, there are also a - number of BIFs that can be used.</p> + number of BIFs that can be used:</p> <table> <row> + <cell align="left" valign="middle"><em>Port BIF</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>port_command(Port,Data)</c></cell> - <cell align="left" valign="middle">Sends <c>Data</c>to the port.</cell> + <cell align="left" valign="middle">Sends <c>Data</c> to the port.</cell> </row> <row> <cell align="left" valign="middle"><c>port_close(Port)</c></cell> @@ -145,7 +178,10 @@ </row> <row> <cell align="left" valign="middle"><c>port_connect(Port,NewPid)</c></cell> - <cell align="left" valign="middle">Sets the port owner of <c>Port</c>to <c>NewPid</c>. The old port owner <c>Pid</c>stays linked to the port and have to call <c>unlink(Port)</c>if this is not desired.</cell> + <cell align="left" valign="middle">Sets the port owner of + <c>Port</c>to <c>NewPid</c>. The old port owner <c>Pid</c> + stays linked to the port and must call <c>unlink(Port)</c> + if this is not desired.</cell> </row> <row> <cell align="left" valign="middle"><c>erlang:port_info(Port,Item)</c></cell> @@ -155,9 +191,9 @@ <cell align="left" valign="middle"><c>erlang:ports()</c></cell> <cell align="left" valign="middle">Returns a list of all ports on the current node.</cell> </row> - <tcaption>Port BIFs.</tcaption> + <tcaption>Port BIFs</tcaption> </table> - <p>There are some additional BIFs that only apply to port drivers: + <p>Some additional BIFs that apply to port drivers: <c>port_control/3</c> and <c>erlang:port_call/3</c>.</p> </section> </chapter> diff --git a/system/doc/reference_manual/processes.xml b/system/doc/reference_manual/processes.xml index 95ae0672ec..d8474c163c 100644 --- a/system/doc/reference_manual/processes.xml +++ b/system/doc/reference_manual/processes.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -32,8 +32,8 @@ <section> <title>Processes</title> <p>Erlang is designed for massive concurrency. Erlang processes are - light-weight (grow and shrink dynamically) with small memory - footprint, fast to create and terminate and the scheduling + lightweight (grow and shrink dynamically) with small memory + footprint, fast to create and terminate, and the scheduling overhead is low.</p> </section> @@ -46,10 +46,10 @@ spawn(Module, Name, Args) -> pid() Args = [Arg1,...,ArgN] ArgI = term()</pre> <p><c>spawn</c> creates a new process and returns the pid.</p> - <p>The new process will start executing in - <c>Module:Name(Arg1,...,ArgN)</c> where the arguments is + <p>The new process starts executing in + <c>Module:Name(Arg1,...,ArgN)</c> where the arguments are the elements of the (possible empty) <c>Args</c> argument list.</p> - <p>There exist a number of other <c>spawn</c> BIFs, for example + <p>There exist a number of other <c>spawn</c> BIFs, for example, <c>spawn/4</c> for spawning a process at another node.</p> </section> @@ -60,18 +60,25 @@ spawn(Module, Name, Args) -> pid() atom and is automatically unregistered if the process terminates:</p> <table> <row> + <cell align="left" valign="middle"><em>BIF</em></cell> + <cell align="left" valign="middle"><em>Description</em></cell> + </row> + <row> <cell align="left" valign="middle"><c>register(Name, Pid)</c></cell> <cell align="left" valign="middle">Associates the name <c>Name</c>, an atom, with the process <c>Pid</c>.</cell> </row> <row> <cell align="left" valign="middle"><c>registered()</c></cell> - <cell align="left" valign="middle">Returns a list of names which have been registered using<c>register/2</c>.</cell> + <cell align="left" valign="middle">Returns a list of names that + have been registered using <c>register/2</c>.</cell> </row> <row> <cell align="left" valign="middle"><c>whereis(Name)</c></cell> - <cell align="left" valign="middle">Returns the pid registered under <c>Name</c>, or<c>undefined</c>if the name is not registered.</cell> + <cell align="left" valign="middle">Returns the pid registered + under <c>Name</c>, or <c>undefined </c>if the name is not + registered.</cell> </row> - <tcaption>Name Registration BIFs.</tcaption> + <tcaption>Name Registration BIFs</tcaption> </table> </section> @@ -79,22 +86,27 @@ spawn(Module, Name, Args) -> pid() <marker id="term"></marker> <title>Process Termination</title> <p>When a process terminates, it always terminates with an - <em>exit reason</em>. The reason may be any term.</p> + <em>exit reason</em>. The reason can be any term.</p> <p>A process is said to terminate <em>normally</em>, if the exit reason is the atom <c>normal</c>. A process with no more code to execute terminates normally.</p> - <p>A process terminates with exit reason <c>{Reason,Stack}</c> + <p>A process terminates with an exit reason <c>{Reason,Stack}</c> when a run-time error occurs. See - <seealso marker="errors#exit_reasons">Error and Error Handling</seealso>.</p> - <p>A process can terminate itself by calling one of the BIFs - <c>exit(Reason)</c>, - <c>erlang:error(Reason)</c>, <c>erlang:error(Reason, Args)</c>, - <c>erlang:fault(Reason)</c> or <c>erlang:fault(Reason, Args)</c>. - The process then terminates with reason <c>Reason</c> for + <seealso marker="errors#exit_reasons">Exit Reasons</seealso>.</p> + <p>A process can terminate itself by calling one of the + following BIFs:</p> + <list type="bulleted"> + <item><c>exit(Reason)</c></item> + <item><c>erlang:error(Reason)</c></item> + <item><c>erlang:error(Reason, Args)</c></item> + <item><c>erlang:fault(Reason)</c></item> + <item><c>erlang:fault(Reason, Args)</c></item> + </list> + <p>The process then terminates with reason <c>Reason</c> for <c>exit/1</c> or <c>{Reason,Stack} for the others</c>.</p> - <p>A process may also be terminated if it receives an exit signal + <p>A process can also be terminated if it receives an exit signal with another exit reason than <c>normal</c>, see - <seealso marker="#errors">Error Handling</seealso> below.</p> + <seealso marker="#errors">Error Handling</seealso>.</p> </section> <section> @@ -113,35 +125,39 @@ spawn(Module, Name, Args) -> pid() <title>Links</title> <p>Two processes can be <em>linked</em> to each other. A link between two processes <c>Pid1</c> and <c>Pid2</c> is created - by <c>Pid1</c> calling the BIF <c>link(Pid2)</c> (or vice versa). + by <c>Pid1</c> calling the BIF <c>link(Pid2)</c> (or conversely). There also exist a number of <c>spawn_link</c> BIFs, which spawn and link to a process in one operation.</p> <p>Links are bidirectional and there can only be one link between two processes. Repeated calls to <c>link(Pid)</c> have no effect.</p> <p>A link can be removed by calling the BIF <c>unlink(Pid)</c>.</p> <p>Links are used to monitor the behaviour of other processes, see - <seealso marker="#errors">Error Handling</seealso> below.</p> + <seealso marker="#errors">Error Handling</seealso>.</p> </section> <section> <marker id="errors"></marker> <title>Error Handling</title> <p>Erlang has a built-in feature for error handling between - processes. Terminating processes will emit exit signals to all - linked processes, which may terminate as well or handle the exit + processes. Terminating processes emit exit signals to all + linked processes, which can terminate as well or handle the exit in some way. This feature can be used to build hierarchical program structures where some processes are supervising other - processes, for example restarting them if they terminate + processes, for example, restarting them if they terminate abnormally.</p> - <p>Refer to OTP Design Principles for more information about - OTP supervision trees, which uses this feature.</p> + <p>See <seealso marker= + "doc/design_principles:des_princ#otp design principles"> + OTP Design Principles</seealso> for more information about + OTP supervision trees, which use this feature.</p> <section> <title>Emitting Exit Signals</title> - <p>When a process terminates, it will terminate with an <em>exit reason</em> as explained in <seealso marker="#term">Process Termination</seealso> above. This exit reason is emitted in + <p>When a process terminates, it terminates with an + <em>exit reason</em> as explained in <seealso marker="#term"> + Process Termination</seealso>. This exit reason is emitted in an <em>exit signal</em> to all linked processes.</p> <p>A process can also call the function <c>exit(Pid,Reason)</c>. - This will result in an exit signal with exit reason + This results in an exit signal with exit reason <c>Reason</c> being emitted to <c>Pid</c>, but does not affect the calling process.</p> </section> @@ -156,14 +172,14 @@ spawn(Module, Name, Args) -> pid() <p>A process can be set to trap exit signals by calling:</p> <pre> process_flag(trap_exit, true)</pre> - <p>When a process is trapping exits, it will not terminate when + <p>When a process is trapping exits, it does not terminate when an exit signal is received. Instead, the signal is transformed - into a message <c>{'EXIT',FromPid,Reason}</c> which is put into - the mailbox of the process just like a regular message.</p> + into a message <c>{'EXIT',FromPid,Reason}</c>, which is put into + the mailbox of the process, just like a regular message.</p> <p>An exception to the above is if the exit reason is <c>kill</c>, - that is if <c>exit(Pid,kill)</c> has been called. This will - unconditionally terminate the process, regardless of if it is - trapping exit signals or not.</p> + that is if <c>exit(Pid,kill)</c> has been called. This + unconditionally terminates the process, regardless of if it is + trapping exit signals.</p> </section> </section> @@ -180,12 +196,12 @@ process_flag(trap_exit, true)</pre> <p>If <c>Pid2</c> does not exist, the 'DOWN' message is sent immediately with <c>Reason</c> set to <c>noproc</c>.</p> <p>Monitors are unidirectional. Repeated calls to - <c>erlang:monitor(process, Pid)</c> will create several, - independent monitors and each one will send a 'DOWN' message when + <c>erlang:monitor(process, Pid)</c> creates several + independent monitors, and each one sends a 'DOWN' message when <c>Pid</c> terminates.</p> <p>A monitor can be removed by calling <c>erlang:demonitor(Ref)</c>.</p> - <p>It is possible to create monitors for processes with registered + <p>Monitors can be created for processes with registered names, also at other nodes.</p> </section> diff --git a/system/doc/reference_manual/records.xml b/system/doc/reference_manual/records.xml index 04766531df..72c56b693d 100644 --- a/system/doc/reference_manual/records.xml +++ b/system/doc/reference_manual/records.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2013</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -32,16 +32,19 @@ elements. It has named fields and is similar to a struct in C. Record expressions are translated to tuple expressions during compilation. Therefore, record expressions are not understood by - the shell unless special actions are taken. See <c>shell(3)</c> - for details.</p> - <p>More record examples can be found in <em>Programming Examples</em>.</p> + the shell unless special actions are taken. For details, see the + <seealso marker="stdlib:shell">shell(3)</seealso> + manual page in STDLIB.</p> + <p>More examples are provided in + <seealso marker="doc/programming_examples:records"> + Programming Examples</seealso>.</p> <section> <title>Defining Records</title> <p>A record definition consists of the name of the record, followed by the field names of the record. Record and field names must be atoms. Each field can be given an optional default value. - If no default value is supplied, <c>undefined</c> will be used.</p> + If no default value is supplied, <c>undefined</c> is used.</p> <pre> -record(Name, {Field1 [= Value1], ... @@ -60,17 +63,18 @@ the corresponding expression <c>ExprI</c>:</p> <pre> #Name{Field1=Expr1,...,FieldK=ExprK}</pre> - <p>The fields may be in any order, not necessarily the same order as + <p>The fields can be in any order, not necessarily the same order as in the record definition, and fields can be omitted. Omitted - fields will get their respective default value instead.</p> - <p>If several fields should be assigned the same value, + fields get their respective default value instead.</p> + <p>If several fields are to be assigned the same value, the following construction can be used:</p> <pre> #Name{Field1=Expr1,...,FieldK=ExprK, _=ExprL}</pre> - <p>Omitted fields will then get the value of evaluating <c>ExprL</c> + <p>Omitted fields then get the value of evaluating <c>ExprL</c> instead of their default values. This feature was added in Erlang 5.1/OTP R8 and is primarily intended to be used to create - patterns for ETS and Mnesia match functions. Example:</p> + patterns for ETS and Mnesia match functions.</p> + <p><em>Example:</em></p> <pre> -record(person, {name, phone, address}). @@ -84,13 +88,13 @@ lookup(Name, Tab) -> <title>Accessing Record Fields</title> <pre> Expr#Name.Field</pre> - <p>Returns the value of the specified field. <c>Expr</c> should + <p>Returns the value of the specified field. <c>Expr</c> is to evaluate to a <c>Name</c> record.</p> <p>The following expression returns the position of the specified field in the tuple representation of the record:</p> <pre> #Name.Field</pre> - <p>Example:</p> + <p><em>Example:</em></p> <pre> -record(person, {name, phone, address}). @@ -104,8 +108,8 @@ lookup(Name, List) -> <title>Updating Records</title> <pre> Expr#Name{Field1=Expr1,...,FieldK=ExprK}</pre> - <p><c>Expr</c> should evaluate to a <c>Name</c> record. Returns a - copy of this record, with the value of each specified field + <p><c>Expr</c> is to evaluate to a <c>Name</c> record. A + copy of this record is returned, with the value of each specified field <c>FieldI</c> changed to the value of evaluating the corresponding expression <c>ExprI</c>. All other fields retain their old values.</p> @@ -116,17 +120,17 @@ Expr#Name{Field1=Expr1,...,FieldK=ExprK}</pre> <title>Records in Guards</title> <p>Since record expressions are expanded to tuple expressions, creating records and accessing record fields are allowed in - guards. However all subexpressions, for example for field - initiations, must of course be valid guard expressions as well. - Examples:</p> + guards. However all subexpressions, for example, for field + initiations, must be valid guard expressions as well.</p> + <p><em>Examples:</em></p> <code type="none"> handle(Msg, State) when Msg==#msg{to=void, no=3} -> ... handle(Msg, State) when State#state.running==true -> ...</code> - <p>There is also a type test BIF <c>is_record(Term, RecordTag)</c>. - Example:</p> + <p>There is also a type test BIF <c>is_record(Term, RecordTag)</c>.</p> + <p><em>Example:</em></p> <pre> is_person(P) when is_record(P, person) -> true; @@ -136,18 +140,18 @@ is_person(_P) -> <section> <title>Records in Patterns</title> - <p>A pattern that will match a certain record is created the same + <p>A pattern that matches a certain record is created in the same way as a record is created:</p> <pre> #Name{Field1=Expr1,...,FieldK=ExprK}</pre> - <p>In this case, one or more of <c>Expr1</c>...<c>ExprK</c> may be + <p>In this case, one or more of <c>Expr1</c>...<c>ExprK</c> can be unbound variables.</p> </section> <section> - <title>Nested records</title> - <p>Beginning with R14 parentheses when accessing or updating nested - records can be omitted. Assuming we have the following record + <title>Nested Records</title> + <p>Beginning with Erlang/OTP R14, parentheses when accessing or updating nested + records can be omitted. Assume the following record definitions:</p> <pre> -record(nrec0, {name = "nested0"}). @@ -156,12 +160,12 @@ is_person(_P) -> N2 = #nrec2{}, </pre> - <p>Before R14 you would have needed to use parentheses as following:</p> + <p>Before R14, parentheses were needed as follows:</p> <pre> "nested0" = ((N2#nrec2.nrec1)#nrec1.nrec0)#nrec0.name, N0n = ((N2#nrec2.nrec1)#nrec1.nrec0)#nrec0{name = "nested0a"}, </pre> - <p>Since R14 you can also write:</p> + <p>Since R14, the following can also be written:</p> <pre> "nested0" = N2#nrec2.nrec1#nrec1.nrec0#nrec0.name, N0n = N2#nrec2.nrec1#nrec1.nrec0#nrec0{name = "nested0a"},</pre> @@ -170,23 +174,23 @@ N0n = N2#nrec2.nrec1#nrec1.nrec0#nrec0{name = "nested0a"},</pre> <section> <title>Internal Representation of Records</title> <p>Record expressions are translated to tuple expressions during - compilation. A record defined as</p> + compilation. A record defined as:</p> <pre> -record(Name, {Field1,...,FieldN}).</pre> - <p>is internally represented by the tuple</p> + <p>is internally represented by the tuple:</p> <pre> {Name,Value1,...,ValueN}</pre> - <p>where each <c>ValueI</c> is the default value for <c>FieldI</c>.</p> + <p>Here each <c>ValueI</c> is the default value for <c>FieldI</c>.</p> <p>To each module using records, a pseudo function is added during compilation to obtain information about records:</p> <pre> record_info(fields, Record) -> [Field] record_info(size, Record) -> Size</pre> - <p><c>Size</c> is the size of the tuple representation, that is + <p><c>Size</c> is the size of the tuple representation, that is, one more than the number of fields.</p> <p>In addition, <c>#Record.Name</c> returns the index in the tuple - representation of <c>Name</c> of the record <c>Record</c>. - <c>Name</c> must be an atom.</p> + representation of <c>Name</c> of the record <c>Record</c>.</p> + <p><c>Name</c> must be an atom.</p> </section> </chapter> diff --git a/system/doc/reference_manual/typespec.xml b/system/doc/reference_manual/typespec.xml index d1584d2b98..53ef93b60f 100644 --- a/system/doc/reference_manual/typespec.xml +++ b/system/doc/reference_manual/typespec.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2003</year><year>2014</year> + <year>2003</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -34,43 +34,46 @@ <p> Erlang is a dynamically typed language. Still, it comes with a notation for declaring sets of Erlang terms to form a particular - type, effectively forming a specific sub-type of the set of all + type. This effectively forms specific subtypes of the set of all Erlang terms. </p> <p> Subsequently, these types can be used to specify types of record fields - and the argument and return types of functions. - </p> - <p> - Type information can be used to document function interfaces, - provide more information for bug detection tools such as <c>Dialyzer</c>, - and can be exploited by documentation tools such as <c>Edoc</c> for - generating program documentation of various forms. - It is expected that the type language described in this document will - supersede and replace the purely comment-based <c>@type</c> and - <c>@spec</c> declarations used by <c>Edoc</c>. - </p> + and also the argument and return types of functions. + </p> + <p> + Type information can be used for the following:</p> + <list type="bulleted"> + <item>To document function interfaces</item> + <item>To provide more information for bug detection tools, + such as <c>Dialyzer</c></item> + <item>To be exploited by documentation tools, such as EDoc, for + generating program documentation of various forms</item> + </list> + <p>It is expected that the type language described in this section + supersedes and replaces the purely comment-based <c>@type</c> and + <c>@spec</c> declarations used by EDoc.</p> </section> <section> <marker id="syntax"></marker> <title>Types and their Syntax</title> <p> Types describe sets of Erlang terms. - Types consist and are built from a set of predefined types - (e.g. <c>integer()</c>, <c>atom()</c>, <c>pid()</c>, ...) - described below. - Predefined types represent a typically infinite set of Erlang terms which + Types consist of, and are built from, a set of predefined types, + for example, <c>integer()</c>, <c>atom()</c>, and <c>pid()</c>. + Predefined types represent a typically infinite set of Erlang terms that belong to this type. For example, the type <c>atom()</c> stands for the set of all Erlang atoms. </p> <p> - For integers and atoms, we allow for singleton types (e.g. the integers - <c>-1</c> and <c>42</c> or the atoms <c>'foo'</c> and <c>'bar'</c>). + For integers and atoms, it is allowed for singleton types; for example, + the integers + <c>-1</c> and <c>42</c>, or the atoms <c>'foo'</c> and <c>'bar'</c>). All other types are built using unions of either predefined types or singleton types. In a type union between a type and one - of its sub-types the sub-type is absorbed by the super-type and - the union is subsequently treated as if the sub-type was not a + of its subtypes, the subtype is absorbed by the supertype. Thus, + the union is then treated as if the subtype was not a constituent of the union. For example, the type union: </p> <pre> atom() | 'bar' | integer() | 42</pre> @@ -79,13 +82,13 @@ </p> <pre> atom() | integer()</pre> <p> - Because of sub-type relations that exist between types, types - form a lattice where the topmost element, <c>any()</c>, denotes + Because of subtype relations that exist between types, types + form a lattice where the top-most element, <c>any()</c>, denotes the set of all Erlang terms and the bottom-most element, <c>none()</c>, denotes the empty set of terms. </p> <p> - The set of predefined types and the syntax for types is given below: + The set of predefined types and the syntax for types follows: </p> <pre><![CDATA[ Type :: any() %% The top type, the set of all Erlang terms @@ -103,7 +106,7 @@ | Map | Tuple | Union - | UserDefined %% described in Section 7.3 + | UserDefined %% described in Type Declarations of User-Defined Types Atom :: atom() | Erlang_Atom %% 'foo', 'bar', ... @@ -146,22 +149,22 @@ <p> The general form of bitstrings is <c><<_:M, _:_*N>></c>, where <c>M</c> and <c>N</c> are positive integers. It denotes a - bitstring that is <c>M + (k*N)</c> bits long (i.e., a bitstring that + bitstring that is <c>M + (k*N)</c> bits long (that is, a bitstring that starts with <c>M</c> bits and continues with <c>k</c> segments of <c>N</c> bits each, where <c>k</c> is also a positive integer). The notations <c><<_:_*N>></c>, <c><<_:M>></c>, and <c><<>></c> are convenient shorthands for the cases - that <c>M</c>, <c>N</c>, or both, respectively, are zero. + that <c>M</c> or <c>N</c>, or both, are zero. </p> <p> Because lists are commonly used, they have shorthand type notations. The types <c>list(T)</c> and <c>nonempty_list(T)</c> have the shorthands <c>[T]</c> and <c>[T,...]</c>, respectively. - The only difference between the two shorthands is that <c>[T]</c> may be an - empty list but <c>[T,...]</c> may not. + The only difference between the two shorthands is that <c>[T]</c> can be an + empty list but <c>[T,...]</c> cannot. </p> <p> - Notice that the shorthand for <c>list()</c>, i.e. the list of + Notice that the shorthand for <c>list()</c>, that is, the list of elements of unknown type, is <c>[_]</c> (or <c>[any()]</c>), not <c>[]</c>. The notation <c>[]</c> specifies the singleton type for the empty list. </p> @@ -172,7 +175,7 @@ </p> <table> <row> - <cell><b>Built-in type</b></cell><cell><b>Defined as</b></cell> + <cell><em>Built-in type</em></cell><cell><em>Defined as</em></cell> </row> <row> <cell><c>term()</c></cell><cell><c>any()</c></cell> @@ -223,7 +226,7 @@ <cell><c>module()</c></cell><cell><c>atom()</c></cell> </row> <row> - <cell><c>mfa()</c></cell><cell><c>{atom(),atom(),arity()}</c></cell> + <cell><c>mfa()</c></cell><cell><c>{module(),atom(),arity()}</c></cell> </row> <row> <cell><c>arity()</c></cell><cell><c>0..255</c></cell> @@ -237,6 +240,7 @@ <row> <cell><c>no_return()</c></cell><cell><c>none()</c></cell> </row> + <tcaption>Built-in types, predefined aliases</tcaption> </table> <p> In addition, the following three built-in types exist and can be @@ -245,7 +249,8 @@ </p> <table> <row> - <cell><b>Built-in type</b></cell><cell><b>Could be thought defined by the syntax</b></cell> + <cell><em>Built-in type</em></cell><cell><em> + Can be thought defined by the syntax</em></cell> </row> <row> <cell><c>non_neg_integer()</c></cell><cell><c>0..</c></cell> @@ -256,6 +261,7 @@ <row> <cell><c>neg_integer()</c></cell><cell><c>..-1</c></cell> </row> + <tcaption>Additional built-in types</tcaption> </table> <p> @@ -278,109 +284,118 @@ define the set of Erlang terms one would expect. </p> <p> - Also for convenience, we allow for record notation to be used. - Records are just shorthands for the corresponding tuples. + Also for convenience, record notation is allowed to be used. + Records are shorthands for the corresponding tuples: </p> <pre> Record :: #Erlang_Atom{} | #Erlang_Atom{Fields}</pre> <p> - Records have been extended to possibly contain type information. - This is described in the sub-section <seealso marker="#typeinrecords">"Type information in record declarations"</seealso> below. + Records are extended to possibly contain type information. + This is described in <seealso marker="#typeinrecords"> + Type Information in Record Declarations</seealso>. </p> <note> - <p>Map types, both <c>map()</c> and <c>#{ ... }</c>, are considered experimental during OTP 17.</p> - <p>No type information of maps pairs, only the containing map types, are used by Dialyzer in OTP 17.</p> + <p>Map types, both <c>map()</c> and <c>#{...}</c>, + are considered experimental during OTP 17.</p> + <p>No type information of maps pairs, only the containing map types, + are used by Dialyzer in OTP 17.</p> </note> </section> - + <section> - <title>Type declarations of user-defined types</title> + <title>Type Declarations of User-Defined Types</title> <p> As seen, the basic syntax of a type is an atom followed by closed - parentheses. New types are declared using '-type' and '-opaque' + parentheses. New types are declared using <c>-type</c> and <c>-opaque</c> compiler attributes as in the following: </p> <pre> -type my_struct_type() :: Type. -opaque my_opaq_type() :: Type.</pre> <p> - where the type name is an atom (<c>'my_struct_type'</c> in the above) - followed by parentheses. Type is a type as defined in the + The type name is the atom <c>my_struct_type</c>, + followed by parentheses. <c>Type</c> is a type as defined in the previous section. - A current restriction is that Type can contain only predefined types, - or user-defined types which are either module-local (i.e., with a - definition that is present in the code of the module) or are remote - types (i.e., types defined in and exported by other modules; see below). - For module-local types, the restriction that their definition + A current restriction is that <c>Type</c> can contain + only predefined types, + or user-defined types which are either of the following: + </p> + <list type="bulleted"> + <item>Module-local type, that is, with a + definition that is present in the code of the module</item> + <item>Remote type, that is, type defined in, and exported by, + other modules; more about this soon.</item> + </list> + <p>For module-local types, the restriction that their definition exists in the module is enforced by the compiler and results in a - compilation error. (A similar restriction currently exists for records.) - </p> + compilation error. (A similar restriction currently exists for records.) </p> <p> Type declarations can also be parameterized by including type variables between the parentheses. The syntax of type variables is the same as - Erlang variables (starts with an upper case letter). - Naturally, these variables can - and should - appear on the RHS of the - definition. A concrete example appears below: + Erlang variables, that is, starts with an upper-case letter. + Naturally, these variables can - and is to - appear on the RHS of the + definition. A concrete example follows: </p> <pre> -type orddict(Key, Val) :: [{Key, Val}].</pre> <p> - A module can export some types in order to declare that other modules + A module can export some types to declare that other modules are allowed to refer to them as <em>remote types</em>. - This declaration has the following form: + This declaration has the following form:</p> <pre> -export_type([T1/A1, ..., Tk/Ak]).</pre> - where the Ti's are atoms (the name of the type) and the Ai's are their - arguments. An example is given below: + <p>Here the Ti's are atoms (the name of the type) and the Ai's are their + arguments</p> + <p><em>Example:</em></p> <pre> -export_type([my_struct_type/0, orddict/2]).</pre> - Assuming that these types are exported from module <c>'mod'</c> then - one can refer to them from other modules using remote type expressions - like those below: + <p>Assuming that these types are exported from module <c>'mod'</c>, + you can refer to them from other modules using remote type expressions + like the following:</p> <pre> mod:my_struct_type() mod:orddict(atom(), term())</pre> - One is not allowed to refer to types which are not declared as exported. + <p>It is not allowed to refer to types that are not declared as exported. </p> <p> Types declared as <c>opaque</c> represent sets of terms whose - structure is not supposed to be visible in any way outside of - their defining module (i.e., only the module defining them is - allowed to depend on their term structure). Consequently, such + structure is not supposed to be visible from outside of + their defining module. That is, only the module defining them + is allowed to depend on their term structure. Consequently, such types do not make much sense as module local - module local - types are not accessible by other modules anyway - and should - always be exported. + types are not accessible by other modules anyway - and is + always to be exported. </p> </section> - - <marker id="typeinrecords"/> + <section> - <title>Type information in record declarations</title> + <marker id="typeinrecords"/> + <title>Type Information in Record Declarations</title> <p> - The types of record fields can be specified in the declaration of the - record. The syntax for this is: + The types of record fields can be specified in the declaration of the + record. The syntax for this is as follows: </p> <pre> -record(rec, {field1 :: Type1, field2, field3 :: Type3}).</pre> <p> For fields without type annotations, their type defaults to any(). - I.e., the above is a shorthand for: + That is, the previous example is a shorthand for the following: </p> <pre> -record(rec, {field1 :: Type1, field2 :: any(), field3 :: Type3}).</pre> <p> In the presence of initial values for fields, - the type must be declared after the initialization as in the following: + the type must be declared after the initialization, as follows: </p> <pre> -record(rec, {field1 = [] :: Type1, field2, field3 = 42 :: Type3}).</pre> <p> - Naturally, the initial values for fields should be compatible - with (i.e. a member of) the corresponding types. - This is checked by the compiler and results in a compilation error - if a violation is detected. For fields without initial values, - the singleton type <c>'undefined'</c> is added to all declared types. + The initial values for fields are to be compatible + with (that is, a member of) the corresponding types. + This is checked by the compiler and results in a compilation error + if a violation is detected. For fields without initial values, + the singleton type <c>'undefined'</c> is added to all declared types. In other words, the following two record declarations have identical effects: </p> @@ -398,13 +413,13 @@ </p> <p> Any record, containing type information or not, once defined, - can be used as a type using the syntax: + can be used as a type using the following syntax: </p> <pre> #rec{}</pre> <p> In addition, the record fields can be further specified when using - a record type by adding type information about the field in - the following manner: + a record type by adding type information about the field + as follows: </p> <pre> #rec{some_field :: Type}</pre> <p> @@ -414,16 +429,16 @@ </section> <section> - <title>Specifications for functions</title> + <title>Specifications for Functions</title> <p> A specification (or contract) for a function is given using the new - compiler attribute <c>'-spec'</c>. The general format is as follows: + compiler attribute <c>-spec</c>. The general format is as follows: </p> <pre> -spec Module:Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> <p> - The arity of the function has to match the number of arguments, - or else a compilation error occurs. + The arity of the function must match the number of arguments, + else a compilation error occurs. </p> <p> This form can also be used in header files (.hrl) to declare type @@ -432,7 +447,7 @@ explicitly) import these functions. </p> <p> - For most uses within a given module, the following shorthand suffices: + Within a given module, the following shorthand suffice in most cases: </p> <pre> -spec Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> @@ -450,8 +465,8 @@ ; (T4, T5) -> T6.</pre> <p> A current restriction, which currently results in a warning - (OBS: not an error) by the compiler, is that the domains of - the argument types cannot be overlapping. + (not an error) by the compiler, is that the domains of + the argument types cannot overlap. For example, the following specification results in a warning: </p> <pre> @@ -466,41 +481,43 @@ <pre> -spec id(X) -> X.</pre> <p> - However, note that the above specification does not restrict the input - and output type in any way. - We can constrain these types by guard-like subtype constraints + Notice that the above specification does not restrict the input + and output type in any way. + These types can be constrained by guard-like subtype constraints and provide bounded quantification: </p> <pre> -spec id(X) -> X when X :: tuple().</pre> <p> Currently, the <c>::</c> constraint (read as <c>is_subtype</c>) is - the only guard constraint which can be used in the <c>'when'</c> + the only guard constraint that can be used in the <c>'when'</c> part of a <c>'-spec'</c> attribute. </p> <note> <p> - The above function specification, using multiple occurrences of - the same type variable, provides more type information than the - function specification below where the type variables are missing: + The above function specification uses multiple occurrences of + the same type variable. That provides more type information than the + following function specification, where the type variables are missing: </p> <pre> -spec id(tuple()) -> tuple().</pre> <p> The latter specification says that the function takes some tuple - and returns some tuple, while the one with the <c>X</c> type + and returns some tuple. The specification with the <c>X</c> type variable specifies that the function takes a tuple and returns <em>the same</em> tuple. </p> <p> - However, it's up to the tools that process the specs to choose - whether to take this extra information into account or ignore it. + However, it is up to the tools that process the specificationss + to choose whether to take this extra information into account + or not. </p> </note> <p> - The scope of an <c>::</c> constraint is the - <c>(...) -> RetType</c> - specification after which it appears. To avoid confusion, - we suggest that different variables are used in different - constituents of an overloaded contract as in the example below: + The scope of a <c>::</c> constraint is the + <c>(...) -> RetType</c> + specification after which it appears. To avoid confusion, + it is suggested that different variables are used in different + constituents of an overloaded contract, as shown in the + following example: </p> <pre> -spec foo({X, integer()}) -> X when X :: atom() @@ -511,19 +528,20 @@ </p> <pre> -spec id(X) -> X when is_subtype(X, tuple()).</pre> <p> - but its use is discouraged. It will be taken out in a future + but its use is discouraged. It will be removed in a future Erlang/OTP release. </p> </note> <p> Some functions in Erlang are not meant to return; either because they define servers or because they are used to - throw exceptions as the function below: + throw exceptions, as in the following function: </p> <pre> my_error(Err) -> erlang:throw({error, Err}).</pre> <p> - For such functions we recommend the use of the special <c>no_return()</c> - type for their "return", via a contract of the form: + For such functions, it is recommended to use the special + <c>no_return()</c> type for their "return", through a contract + of the following form: </p> <pre> -spec my_error(term()) -> no_return().</pre> </section> diff --git a/system/doc/system_principles/create_target.xmlsrc b/system/doc/system_principles/create_target.xmlsrc index a8ee2d1245..7c566229ac 100644 --- a/system/doc/system_principles/create_target.xmlsrc +++ b/system/doc/system_principles/create_target.xmlsrc @@ -31,55 +31,54 @@ <rev>A</rev> <file>create_target.xml</file> </header> + <marker id="creating upgrading target system"></marker> - <section> - <title>Introduction</title> - <p>When creating a system using Erlang/OTP, the most simple way is - to install Erlang/OTP somewhere, install the application specific + <p>When creating a system using Erlang/OTP, the simplest way is + to install Erlang/OTP somewhere, install the application-specific code somewhere else, and then start the Erlang runtime system, - making sure the code path includes the application specific code.</p> - <p>Often it is not desirable to use an Erlang/OTP system as is. A - developer may create new Erlang/OTP compliant applications for a + making sure the code path includes the application-specific code.</p> + <p>It is often not desirable to use an Erlang/OTP system as is. A + developer can create new Erlang/OTP-compliant applications for a particular purpose, and several original Erlang/OTP applications - may be irrelevant for the purpose in question. Thus, there is a + can be irrelevant for the purpose in question. Thus, there is a need to be able to create a new system based on a given - Erlang/OTP system, where dispensable applications are removed, - and a set of new applications are included. Documentation and + Erlang/OTP system, where dispensable applications are removed + and new applications are included. Documentation and source code is irrelevant and is therefore not included in the new system.</p> - <p>This chapter is about creating such a system, which we call a + <p>This chapter is about creating such a system, which is called a <em>target system</em>.</p> - <p>In the following sections we consider creating target systems with - different requirements of functionality:</p> + <p>The following sections deal with target systems + with different requirements of functionality:</p> <list type="bulleted"> - <item>a <em>basic target system</em> that can be started by - calling the ordinary <c>erl</c> script, </item> - <item>a <em>simple target system</em> where also code - replacement in run-time can be performed, and</item> - <item>an <em>embedded target system</em> where there is also + <item>A <em>basic target system</em> that can be started by + calling the ordinary <c>erl</c> script.</item> + <item>A <em>simple target system</em> where also code + replacement in runtime can be performed.</item> + <item>An <em>embedded target system</em> where there is also support for logging output from the system to file for later inspection, and where the system can be started automatically - at boot time. </item> + at boot time.</item> </list> - <p>We only consider the case when Erlang/OTP is running on a UNIX - system.</p> - <p>In the <c>sasl</c> application there is an example Erlang - module <c>target_system.erl</c> that contains functions for - creating and installing a target system. This module is used in - the examples below, and the source code of the module is listed - at the end of this chapter.</p> - </section> + <p>Here is only considered the case when Erlang/OTP is running on a + UNIX system.</p> + <p>The <c>sasl</c> application includes the example Erlang + module <c>target_system.erl</c>, which contains functions for + creating and installing a target system. This module is used in + the following examples. The source code of the module is listed + in <seealso marker="#listing of target system"> + Listing of target_system.erl</seealso></p> <section> <marker id="create"/> <title>Creating a Target System</title> <p>It is assumed that you have a working Erlang/OTP system structured - according to the OTP Design Principles.</p> - <p><em>Step 1.</em> First create a <c>.rel</c> file (see <seealso - marker="sasl:rel">rel(4)</seealso>) that specifies the <c>erts</c> - version and lists all applications that should be included in the - new basic target system. An example is the following - <c>mysystem.rel</c> file:</p> + according to the OTP design principles.</p> + <p><em>Step 1.</em> Create a <c>.rel</c> file (see the + <seealso marker="sasl:rel">rel(4)</seealso> manual page in + SASL), which specifies the ERTS version and lists + all applications that are to be included in the new basic target + system. An example is the following <c>mysystem.rel</c> file:</p> <code type="none"> %% mysystem.rel {release, @@ -91,23 +90,23 @@ {pea, "1.0"}]}.</code> <p>The listed applications are not only original Erlang/OTP applications but possibly also new applications that you have - written yourself (here exemplified by the application - <c>pea</c>). </p> - <p><em>Step 2.</em> From the directory where the <c>mysystem.rel</c> - file reside, start the Erlang/OTP system:</p> + written (here exemplified by the application Pea (<c>pea</c>)).</p> + <p><em>Step 2.</em> Start Erlang/OTP from the directory where + the <c>mysystem.rel</c> file resides:</p> <pre> os> <input>erl -pa /home/user/target_system/myapps/pea-1.0/ebin</input></pre> - <p>where also the path to the <c>pea-1.0</c> ebin directory is - provided. </p> - <p><em>Step 3.</em> Now create the target system: </p> + <p>Here also the path to the <c>pea-1.0</c> ebin directory is + provided.</p> + <p><em>Step 3.</em> Create the target system:</p> <pre> 1> <input>target_system:create("mysystem").</input></pre> - <p>The <c>target_system:create/1</c> function does the following:</p> + <p>The function <c>target_system:create/1</c> performs the + following:</p> <list type="ordered"> - <item>Reads the <c>mysystem.rel</c> file, and creates a new file - <c>plain.rel</c> which is identical to former, except that it - only lists the <c>kernel</c> and <c>stdlib</c> applications. </item> - <item>From the <c>mysystem.rel</c> and <c>plain.rel</c> files + <item>Reads the file <c>mysystem.rel</c> and creates a new file + <c>plain.rel</c> that is identical to the former, except that it + only lists the Kernel and STDLIB applications.</item> + <item>From the files <c>mysystem.rel</c> and <c>plain.rel</c> creates the files <c>mysystem.script</c>, <c>mysystem.boot</c>, <c>plain.script</c>, and <c>plain.boot</c> through a call to @@ -124,26 +123,26 @@ releases/mysystem.rel lib/kernel-2.16.4/ lib/stdlib-1.19.4/ lib/sasl-2.3.4/ -lib/pea-1.0/ </code> +lib/pea-1.0/</code> <p>The file <c>releases/FIRST/start.boot</c> is a copy of our <c>mysystem.boot</c></p> <p>The release resource file <c>mysystem.rel</c> is duplicated in the tar file. Originally, this file was only stored in - the <c>releases</c> directory in order to make it possible + the <c>releases</c> directory to make it possible for the <c>release_handler</c> to extract this file separately. After unpacking the tar file, <c>release_handler</c> would automatically copy the file to <c>releases/FIRST</c>. However, sometimes the tar file is unpacked without involving - the <c>release_handler</c> (e.g. when unpacking the first - target system) and therefore the file is now instead + the <c>release_handler</c> (for example, when unpacking the + first target system). The file is therefore now instead duplicated in the tar file so no manual copying is - necessary.</p> + needed.</p> </item> - <item>Creates the temporary directory <c>tmp</c> and extracts the tar file - <c>mysystem.tar.gz</c> into that directory. </item> - <item>Deletes the <c>erl</c> and <c>start</c> files from - <c>tmp/erts-5.10.4/bin</c>. These files will be created again from + <item>Creates the temporary directory <c>tmp</c> and extracts + the tar file <c>mysystem.tar.gz</c> into that directory.</item> + <item>Deletes the files <c>erl</c> and <c>start</c> from + <c>tmp/erts-5.10.4/bin</c>. These files are created again from source when installing the release.</item> <item>Creates the directory <c>tmp/bin</c>.</item> <item>Copies the previously created file <c>plain.boot</c> to @@ -151,31 +150,31 @@ lib/pea-1.0/ </code> <item>Copies the files <c>epmd</c>, <c>run_erl</c>, and <c>to_erl</c> from the directory <c>tmp/erts-5.10.4/bin</c> to the directory <c>tmp/bin</c>.</item> - <item>Creates the directory <c>tmp/log</c>, which will be used + <item>Creates the directory <c>tmp/log</c>, which is used if the system is started as embedded with the <c>bin/start</c> script.</item> <item>Creates the file <c>tmp/releases/start_erl.data</c> with the contents "5.10.4 FIRST". This file is to be passed as data - file to the <c>start_erl</c> script. - </item> + file to the <c>start_erl</c> script.</item> <item>Recreates the file <c>mysystem.tar.gz</c> from the directories - in the directory <c>tmp</c>, and removes <c>tmp</c>.</item> + in the directory <c>tmp</c> and removes <c>tmp</c>.</item> </list> </section> <section> <title>Installing a Target System</title> <p><em>Step 4.</em> Install the created target system in a - suitable directory. </p> + suitable directory.</p> <pre> 2> <input>target_system:install("mysystem", "/usr/local/erl-target").</input></pre> - <p>The function <c>target_system:install/2</c> does the following: + <p>The function <c>target_system:install/2</c> performs the following: </p> <list type="ordered"> <item>Extracts the tar file <c>mysystem.tar.gz</c> into the target directory <c>/usr/local/erl-target</c>.</item> - <item>In the target directory reads the file <c>releases/start_erl.data</c> - in order to find the Erlang runtime system version ("5.10.4").</item> + <item>In the target directory reads the file + <c>releases/start_erl.data</c> to find the Erlang runtime system + version ("5.10.4").</item> <item>Substitutes <c>%FINAL_ROOTDIR%</c> and <c>%EMU%</c> for <c>/usr/local/erl-target</c> and <c>beam</c>, respectively, in the files <c>erl.src</c>, <c>start.src</c>, and @@ -184,97 +183,102 @@ lib/pea-1.0/ </code> <c>start</c>, and <c>run_erl</c> in the target <c>bin</c> directory.</item> <item>Finally the target <c>releases/RELEASES</c> file is created - from data in the <c>releases/mysystem.rel</c> file.</item> + from data in the file <c>releases/mysystem.rel</c>.</item> </list> </section> <section> <marker id="start"/> <title>Starting a Target System</title> - <p>Now we have a target system that can be started in various ways.</p> - <p>We start it as a <em>basic target system</em> by invoking</p> + <p>Now we have a target system that can be started in various ways. + We start it as a <em>basic target system</em> by invoking:</p> <pre> os> <input>/usr/local/erl-target/bin/erl</input></pre> - <p>where only the <c>kernel</c> and <c>stdlib</c> applications are - started, i.e. the system is started as an ordinary development - system. There are only two files needed for all this to work: - <c>bin/erl</c> file (obtained from <c>erts-5.10.4/bin/erl.src</c>) - and the <c>bin/start.boot</c> file (a copy of <c>plain.boot</c>).</p> + <p>Here only the Kernel and STDLIB applications are + started, that is, the system is started as an ordinary development + system. Only two files are needed for all this to work:</p> + <list type="ordered"> + <item><c>bin/erl</c> (obtained from + <c>erts-5.10.4/bin/erl.src</c>)</item> + <item><c>bin/start.boot</c> (a copy of + <c>plain.boot</c>)</item> + </list> <p>We can also start a distributed system (requires <c>bin/epmd</c>).</p> <p>To start all applications specified in the original - <c>mysystem.rel</c> file, use the <c>-boot</c> flag as follows:</p> + <c>mysystem.rel</c> file, use flag <c>-boot</c> as follows:</p> <pre> os> <input>/usr/local/erl-target/bin/erl -boot /usr/local/erl-target/releases/FIRST/start</input></pre> - <p>We start a <em>simple target system</em> as above. The only difference - is that also the file <c>releases/RELEASES</c> is present for - code replacement in run-time to work.</p> - <p>To start an <em>embedded target system</em> the shell script - <c>bin/start</c> is used. That shell script calls + <p>We start a <em>simple target system</em> as above. The only + difference is that also the file <c>releases/RELEASES</c> is + present for code replacement in runtime to work.</p> + <p>To start an <em>embedded target system</em>, the shell script + <c>bin/start</c> is used. The script calls <c>bin/run_erl</c>, which in turn calls <c>bin/start_erl</c> (roughly, <c>start_erl</c> is an embedded variant of - <c>erl</c>). </p> + <c>erl</c>).</p> <p>The shell script <c>start</c>, which is generated from erts-5.10.4/bin/start.src during installation, is only an - example. You should edit it to suite your needs. Typically it is + example. Edit it to suite your needs. Typically it is executed when the UNIX system boots.</p> <p><c>run_erl</c> is a wrapper that provides logging of output from - the run-time system to file. It also provides a simple mechanism + the runtime system to file. It also provides a simple mechanism for attaching to the Erlang shell (<c>to_erl</c>).</p> - <p><c>start_erl</c> requires the root directory - (<c>"/usr/local/erl-target"</c>), the releases directory - (<c>"/usr/local/erl-target/releases"</c>), and the location of - the <c>start_erl.data</c> file. It reads the run-time system - version (<c>"5.10.4"</c>) and release version (<c>"FIRST"</c>) from - the <c>start_erl.data</c> file, starts the run-time system of the - version found, and provides <c>-boot</c> flag specifying the boot - file of the release version found - (<c>"releases/FIRST/start.boot"</c>).</p> - <p><c>start_erl</c> also assumes that there is <c>sys.config</c> in - release version directory (<c>"releases/FIRST/sys.config"</c>). That - is the topic of the next section (see below).</p> - <p>The <c>start_erl</c> shell script should normally not be + <p><c>start_erl</c> requires:</p> + <list type="ordered"> + <item>The root directory (<c>"/usr/local/erl-target"</c>)</item> + <item>The releases directory + (<c>"/usr/local/erl-target/releases"</c></item> + <item>The location of the file <c>start_erl.data</c></item> + </list> + <p>It performs the following:</p> + <list type="ordered"> + <item>Reads the runtime system version (<c>"5.10.4"</c>) and + release version (<c>"FIRST"</c>) from the file + <c>start_erl.data</c>.</item> + <item>Starts the runtime system of the version found.</item> + <item>Provides the flag <c>-boot</c> specifying the boot + file of the release version found + (<c>"releases/FIRST/start.boot"</c>).</item> + </list> + <p><c>start_erl</c> also assumes that there is <c>sys.config</c> + in the release version directory (<c>"releases/FIRST/sys.config"</c>). + That is the topic of the next section.</p> + <p>The <c>start_erl</c> shell script is normally not to be altered by the user.</p> </section> <section> <title>System Configuration Parameters</title> - <p>As was pointed out above <c>start_erl</c> requires a - <c>sys.config</c> in the release version directory - (<c>"releases/FIRST/sys.config"</c>). If there is no such a - file, the system start will fail. Hence such a file has to - be added as well.</p> - <p></p> - <p>If you have system configuration data that are neither file - location dependent nor site dependent, it may be convenient to - create the <c>sys.config</c> early, so that it becomes a part of + <p>As was mentioned in the previous section, <c>start_erl</c> + requires a <c>sys.config</c> in the release version directory + (<c>"releases/FIRST/sys.config"</c>). If there is no such + file, the system start fails. Such a file must therefore + also be added.</p> + <p>If you have system configuration data that is neither + file-location-dependent nor site-dependent, it can be convenient + to create <c>sys.config</c> early, so it becomes part of the target system tar file created by - <c>target_system:create/1</c>. In fact, if you create, in the - current directory, not only the <c>mysystem.rel</c> file, but - also a <c>sys.config</c> file, that latter file will be tacitly + <c>target_system:create/1</c>. In fact, if you in the + current directory create not only the file <c>mysystem.rel</c>, + but also file <c>sys.config</c>, the latter file is tacitly put in the appropriate directory.</p> </section> <section> - <title>Differences from the Install Script</title> - <p>The above <c>install/2</c> procedure differs somewhat from that + <title>Differences From the Install Script</title> + <p>The previous <c>install/2</c> procedure differs somewhat from that of the ordinary <c>Install</c> shell script. In fact, <c>create/1</c> makes the release package as complete as possible, and leave to the - <c>install/2</c> procedure to finish by only considering location - dependent files.</p> + <c>install/2</c> procedure to finish by only considering + location-dependent files.</p> </section> <section> <title>Creating the Next Version</title> - - <p> - In this example the <c>pea</c> application has been changed, and - so are <c>erts</c>, <c>kernel</c>, <c>stdlib</c> and - <c>sasl</c>. - </p> - - <p> - <em>Step 1.</em> Create the <c>.rel</c> file: - </p> + <p>In this example the Pea application has been changed, and + so are the applications ERTS, Kernel, STDLIB + and SASL.</p> + <p><em>Step 1.</em> Create the file <c>.rel</c>:</p> <code type="none"> %% mysystem2.rel {release, @@ -284,65 +288,49 @@ os> <input>/usr/local/erl-target/bin/erl -boot /usr/local/erl-target/releases/FI {stdlib, "2.0"}, {sasl, "2.4"}, {pea, "2.0"}]}.</code> - <p> - <em>Step 2.</em> Create the application upgrade file (see - <seealso marker="sasl:appup">appup(4)</seealso>) for <c>pea</c>, - for example: - </p> + <p><em>Step 2.</em> Create the application upgrade file (see the + <seealso marker="sasl:appup">appup(4)</seealso> manual page in + SASL) for Pea, for example:</p> <code type="none"> %% pea.appup {"2.0", [{"1.0",[{load_module,pea_lib}]}], [{"1.0",[{load_module,pea_lib}]}]}.</code> - <p> - <em>Step 3.</em> From the directory where the - <c>mysystem2.rel</c> file reside, start the Erlang/OTP system: - </p> + <p><em>Step 3.</em> From the directory where the file + <c>mysystem2.rel</c> resides, start the Erlang/OTP system, + giving the path to the new version of Pea:</p> <pre> os> <input>erl -pa /home/user/target_system/myapps/pea-2.0/ebin</input></pre> - <p>giving the path to the new version of <c>pea</c>. </p> - - <p> - <em>Step 4.</em> Create the release upgrade file (see <seealso - marker="sasl:relup">relup(4)</seealso>): - </p> + <p><em>Step 4.</em> Create the release upgrade file (see the + <seealso marker="sasl:relup">relup(4)</seealso> manual page in + SASL):</p> <pre> -1> <input>systools:make_relup("mysystem2",["mysystem"],["mysystem"],[{path,["/home/user/target_system/myapps/pea-1.0/ebin","/my/old/erlang/lib/*/ebin"]}]).</input></pre> - <p> - where <c>"mysystem"</c> is the base release and - <c>"mysystem2"</c> is the release to upgrade to. - </p> - <p> - Note that the <c>path</c> option is used for pointing out the +1> <input>systools:make_relup("mysystem2",["mysystem"],["mysystem"], + [{path,["/home/user/target_system/myapps/pea-1.0/ebin", + "/my/old/erlang/lib/*/ebin"]}]).</input></pre> + <p>Here <c>"mysystem"</c> is the base release and + <c>"mysystem2"</c> is the release to upgrade to.</p> + <p>The <c>path</c> option is used for pointing out the old version of all applications. (The new versions are already - in the code path - assuming of course that the erlang node on + in the code path - assuming of course that the Erlang node on which this is executed is running the correct version of - Erlang/OTP.) - </p> - <p> - <em>Step 5.</em> Create the new release: - </p> + Erlang/OTP.)</p> + <p><em>Step 5.</em> Create the new release:</p> <pre> 2> <input>target_system:create("mysystem2").</input></pre> - <p> - Given that the <c>relup</c> file generated in step 4 above is - now located in the current directory, it will automatically be - included in the release package. - </p> + <p>Given that the file <c>relup</c> generated in Step 4 is + now located in the current directory, it is automatically + included in the release package.</p> </section> <section> <title>Upgrading the Target System</title> - <p> - This part is done on the target node, and for this example we + <p>This part is done on the target node, and for this example we want the node to be running as an embedded system with the - <c>-heart</c> option, allowing automatic restart of the - node. See <seealso marker="#start">Starting a Target - System</seealso> above for more information. - </p> - <p> - We add <c>-heart</c> to <c>bin/start</c>: - </p> + <c>-heart</c> option, allowing automatic restart of the node. + For more information, see <seealso marker="#start"> + Starting a Target System</seealso>.</p> + <p>We add <c>-heart</c> to <c>bin/start</c>:</p> <code type="none"> #!/bin/sh ROOTDIR=/usr/local/erl-target/ @@ -354,36 +342,27 @@ fi START_ERL_DATA=${1:-$RELDIR/start_erl.data} -$ROOTDIR/bin/run_erl -daemon /tmp/ $ROOTDIR/log "exec $ROOTDIR/bin/start_erl $ROOTDIR $RELDIR $START_ERL_DATA -heart</code> - <p> - And we use the simplest possible <c>sys.config</c>, which we - store in <c>releases/FIRST</c>: - </p> +$ROOTDIR/bin/run_erl -daemon /tmp/ $ROOTDIR/log "exec $ROOTDIR/bin/start_erl $ROOTDIR\ +$RELDIR $START_ERL_DATA -heart</code> + <p>We use the simplest possible <c>sys.config</c>, which we + store in <c>releases/FIRST</c>:</p> <code type="none"> %% sys.config [].</code> - <p> - Finally, in order to prepare the upgrade, we need to put the new + <p>Finally, to prepare the upgrade, we must put the new release package in the <c>releases</c> directory of the first - target system: - </p> + target system:</p> <pre> os> <input>cp mysystem2.tar.gz /usr/local/erl-target/releases</input></pre> - <p> - And assuming that the node has been started like this: - </p> + <p>Assuming that the node has been started as follows:</p> <pre> os> <input>/usr/local/erl-target/bin/start</input></pre> - <p> - it can be accessed like this: - </p> + <p>It can be accessed as follows:</p> <pre> os> <input>/usr/local/erl-target/bin/to_erl /tmp/erlang.pipe.1</input></pre> - <p> - Also note that logs can be found in + <p>Logs can be found in <c>/usr/local/erl-target/log</c>. This directory is specified as - an argument to <c>run_erl</c>in the start script listed above. - </p> + an argument to <c>run_erl</c>in the start script listed above.</p> <p> <em>Step 1.</em> Unpack the release: </p> @@ -402,18 +381,19 @@ heart: Tue Apr 1 12:15:11 2014: Executed "/usr/local/erl-target/bin/start /usr/ The above return value and output after the call to <c>release_handler:install_release/1</c> means that the <c>release_handler</c> has restarted the node by using - <c>heart</c>. This will always be done when the upgrade involves - a change of <c>erts</c>, <c>kernel</c>, <c>stdlib</c> or - <c>sasl</c>. See <seealso marker="upgrade">Upgrade when - Erlang/OTP has Changed</seealso> for more infomation about this. + <c>heart</c>. This is always done when the upgrade involves + a change of the applications ERTS, Kernel, + STDLIB, or SASL. For more information, see + <seealso marker="upgrade"> + Upgrade when Erlang/OTP has Changed</seealso>. </p> <p> - The node will be accessible via a new pipe: + The node is accessible through a new pipe: </p> <pre> os> <input>/usr/local/erl-target/bin/to_erl /tmp/erlang.pipe.2</input></pre> <p> - Let's see which releases we have in our system: + Check which releases there are in the system: </p> <pre> 1> <input>release_handler:which_releases().</input> @@ -426,7 +406,7 @@ os> <input>/usr/local/erl-target/bin/to_erl /tmp/erlang.pipe.2</input></pre> <p> Our new release, "SECOND", is now the current release, but we can also see that our "FIRST" release is still permanent. This - means that if the node would be restarted at this point, it + means that if the node would be restarted now, it would come up running the "FIRST" release again. </p> <p> @@ -434,11 +414,9 @@ os> <input>/usr/local/erl-target/bin/to_erl /tmp/erlang.pipe.2</input></pre> </p> <pre> 2> <input>release_handler:make_permanent("SECOND").</input></pre> - <p> - Now look at the releases again: + Check the releases again: </p> - <pre> 3> <input>release_handler:which_releases().</input> <output>[{"MYSYSTEM","SECOND", @@ -447,19 +425,16 @@ os> <input>/usr/local/erl-target/bin/to_erl /tmp/erlang.pipe.2</input></pre> {"MYSYSTEM","FIRST", ["kernel-2.16.4","stdlib-1.19.4","sasl-2.3.4","pea-1.0"], old}]</output></pre> - <p> - Here we see that the new release version is <c>permanent</c>, so - it would be safe to restart the node. - </p> - + We see that the new release version is <c>permanent</c>, so + it would be safe to restart the node.</p> </section> <section> + <marker id="listing of target system"/> <title>Listing of target_system.erl</title> <p>This module can also be found in the <c>examples</c> directory - of the <c>sasl</c> application.</p> + of the SASL application.</p> <codeinclude file="../../../lib/sasl/examples/src/target_system.erl" tag="%module" type="erl"></codeinclude> - </section> </chapter> diff --git a/system/doc/system_principles/error_logging.xml b/system/doc/system_principles/error_logging.xml index 80d5211323..3a82f4e0e0 100644 --- a/system/doc/system_principles/error_logging.xml +++ b/system/doc/system_principles/error_logging.xml @@ -28,41 +28,43 @@ <rev></rev> <file>error_logging.xml</file> </header> + <marker id="error logging"></marker> <section> <title>Error Information From the Runtime System</title> <p>Error information from the runtime system, that is, information - about a process terminating due to an uncaught error exception, + about a process terminating because of an uncaught error exception, is by default written to terminal (tty):</p> <code type="none"><![CDATA[ =ERROR REPORT==== 9-Dec-2003::13:25:02 === Error in process <0.27.0> with exit value: {{badmatch,[1,2,3]},[{m,f,1},{shell,eval_loop,2}]}]]></code> <p>The error information is handled by the <em>error logger</em>, a system process registered as <c>error_logger</c>. This process - receives all error messages from the Erlang runtime system and - also from the standard behaviours and different Erlang/OTP + receives all error messages from the Erlang runtime system as + well as from the standard behaviours and different Erlang/OTP applications.</p> - <p>The exit reasons (such as <c>badarg</c> above) used by + <p>The exit reasons (such as <c>badarg</c>) used by the runtime system are described in - <seealso marker="doc/reference_manual:errors#exit_reasons">Errors and Error Handling</seealso> - in the Erlang Reference Manual.</p> - <p>The process <c>error_logger</c> and its user interface (with - the same name) are described in - <seealso marker="kernel:error_logger">error_logger(3)</seealso>. - It is possible to configure the system so that error information - is written to file instead/as well as tty. Also, it is possible - for user defined applications to send and format error - information using <c>error_logger</c>.</p> + <seealso marker="doc/reference_manual:errors#exit_reasons"> + Errors and Error Handling</seealso>.</p> + <p>For information about the process <c>error_logger</c> and its user + interface (with the same name), see the + <seealso marker="kernel:error_logger">error_logger(3)</seealso> + manual page in Kernel. The system can be configured so that + error information + is written to file or to tty, or both. In addition, user-defined + applications can send and format error information using + <c>error_logger</c>.</p> </section> <section> <title>SASL Error Logging</title> - <p>The standard behaviors (<c>supervisor</c>, <c>gen_server</c>, - etc.) sends progress and error information to <c>error_logger</c>. - If the SASL application is started, this information is written - to tty as well. See + <p>The standard behaviours (<c>supervisor</c>, <c>gen_server</c>, + and so on) send progress and error information to <c>error_logger</c>. + If the SASL application is started, this information is + written to tty as well. For more information, see <seealso marker="sasl:error_logging">SASL Error Logging</seealso> - in the SASL User's Guide for further information.</p> + in the SASL User's Guide.</p> <pre> % <input>erl -boot start_sasl</input> Erlang (BEAM) emulator version 5.4.13 [hipe] [threads:0] [kernel-poll] diff --git a/system/doc/system_principles/system_principles.xml b/system/doc/system_principles/system_principles.xml index 79ed86cd9f..70bb3cd441 100644 --- a/system/doc/system_principles/system_principles.xml +++ b/system/doc/system_principles/system_principles.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>1996</year><year>2014</year> + <year>1996</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,35 +28,41 @@ <rev></rev> <file>system_principles.xml</file> </header> + <marker id="system principles"></marker> <section> <title>Starting the System</title> - <p>An Erlang runtime system is started with the command <c>erl</c>:</p> + <p>An Erlang runtime system is started with command <c>erl</c>:</p> <pre> % <input>erl</input> Erlang/OTP 17 [erts-6.0] [hipe] [smp:8:8] Eshell V6.0 (abort with ^G) 1> </pre> - <p><c>erl</c> understands a number of command line arguments, see - <c>erl(1)</c>. A number of them are also described in this chapter.</p> - <p>Application programs can access the values of the command line - arguments by calling one of the functions - <c>init:get_argument(Key)</c>, or <c>init:get_arguments()</c>. - See <c>init(3)</c>.</p> + <p><c>erl</c> understands a number of command-line arguments, see + the <seealso marker="erts:erl">erl(1)</seealso> manual page in + ERTS. Some of them are also described in this chapter.</p> + <p>Application programs can access the values of the command-line + arguments by calling the function <c>init:get_argument(Key)</c> + or <c>init:get_arguments()</c>. See the + <seealso marker="erts:init">init(3)</seealso> manual page in + ERTS.</p> </section> <section> <title>Restarting and Stopping the System</title> - <p>The runtime system can be halted by calling <c>halt/0,1</c>. - See <c>erlang(3)</c>.</p> - <p>The module <c>init</c> contains function for restarting, - rebooting and stopping the runtime system. See <c>init(3)</c>.</p> + <p>The runtime system is halted by calling <c>halt/0,1</c>. For + details, see the <seealso marker="erts:erlang">erlang(3)</seealso> + manual page in ERTS.</p> + <p>The module <c>init</c> contains functions for restarting, + rebooting, and stopping the runtime system:</p> <pre> init:restart() init:reboot() init:stop()</pre> - <p>Also, the runtime system will terminate if the Erlang shell is + <p>For details, see the <seealso marker="erts:init">init(3)</seealso> + manual page in ERTS.</p> + <p>The runtime system terminates if the Erlang shell is terminated.</p> </section> @@ -69,14 +75,15 @@ init:stop()</pre> <p>A boot script file has the extension <c>.script</c>. The runtime system uses a binary version of the script. This <em>binary boot script</em> file has the extension <c>.boot</c>.</p> - <p>Which boot script to use is specified by the command line flag - <c>-boot</c>. The extension <c>.boot</c> should be omitted. - Example, using the boot script <c>start_all.boot</c>:</p> + <p>Which boot script to use is specified by the command-line flag + <c>-boot</c>. The extension <c>.boot</c> is to be omitted. + For example, using the boot script <c>start_all.boot</c>:</p> <pre> % <input>erl -boot start_all</input></pre> <p>If no boot script is specified, it defaults to - <c>ROOT/bin/start</c>, see Default Boot Scripts below.</p> - <p>The command line flag <c>-init_debug</c> makes the <c>init</c> + <c>ROOT/bin/start</c>, see <seealso marker="#default_boot_scripts"> + Default Boot Scripts</seealso>.</p> + <p>The command-line flag <c>-init_debug</c> makes the <c>init</c> process write some debug information while interpreting the boot script:</p> <pre> @@ -87,59 +94,55 @@ init:stop()</pre> {start,heart} {start,error_logger} ...</pre> - <p>See <c>script(4)</c> for a detailed description of the syntax - and contents of the boot script.</p> + <p>For a detailed description of the syntax and contents of the + boot script, see the <c>script(4)</c> manual page in SASL.</p> <section> + <marker id="default_boot_scripts"></marker> <title>Default Boot Scripts</title> - <p>Erlang/OTP comes with two boot scripts:</p> - <taglist> - <tag><c>start_clean.boot</c></tag> - <item> - <p>Loads the code for and starts the applications Kernel and - STDLIB.</p> - </item> - <tag><c>start_sasl.boot</c></tag> - <item> - <p>Loads the code for and starts the applications Kernel, - STDLIB and SASL.</p> - </item> - <tag><c>no_dot_erlang.boot</c></tag> - <item> - <p>Loads the code for and starts the applications Kernel and - STDLIB, skips loading the <c>.erlang</c> file. - Useful for scripts and other tools that should be behave the - same regardless of user preferences. - </p> - </item> - </taglist> + <p>Erlang/OTP comes with these boot scripts:</p> + <list type="bulleted"> + <item><c>start_clean.boot</c> - Loads the code for and starts + the applications Kernel and STDLIB.</item> + <item><c>start_sasl.boot</c> - Loads the code for and starts + the applications Kernel, STDLIB, and + SASL).</item> + <item><c>no_dot_erlang.boot</c> - Loads the code for and + starts the applications Kernel and STDLIB. + Skips loading the file <c>.erlang</c>. Useful for scripts and + other tools that are to behave the same irrespective of user + preferences.</item> + </list> <p>Which of <c>start_clean</c> and <c>start_sasl</c> to use as default is decided by the user when installing Erlang/OTP using <c>Install</c>. The user is asked "Do you want to use a minimal system startup instead of the SASL startup". If the answer is yes, then <c>start_clean</c> is used, otherwise - <c>start_sasl</c> is used. A copy of the selected boot script - is made, named <c>start.boot</c> and placed in - the <c>ROOT/bin</c> directory.</p> + <c>start_sasl</c> is used. A copy of the selected boot script is + made, named <c>start.boot</c> and placed in directory + <c>ROOT/bin</c>.</p> </section> <section> <title>User-Defined Boot Scripts</title> <p>It is sometimes useful or necessary to create a user-defined boot script. This is true especially when running Erlang in - embedded mode, see <seealso marker="#code_loading">Code Loading Strategy</seealso>.</p> - <p>It is possible to write a boot script manually. - The recommended way to create a boot script, however, is to - generate the boot script from a release resource file - <c>Name.rel</c>, using the function + embedded mode, see <seealso marker="#code_loading"> + Code Loading Strategy</seealso>.</p> + <p>A boot script can be written manually. However, it is + recommended to create a boot script by generating it from a + release resource file <c>Name.rel</c>, using the function <c>systools:make_script/1,2</c>. This requires that the source code is structured as applications according to the OTP design principles. (The program does not have to be started in terms of - OTP applications but can be plain Erlang).</p> - <p>Read more about <c>.rel</c> files in OTP Design Principles and - <c>rel(4)</c>.</p> + OTP applications, but can be plain Erlang).</p> + <p>For more information about <c>.rel</c> files, see + <seealso marker="doc/design_principles:release_handling"> + OTP Design Principles</seealso> and the + <seealso marker="sasl:rel">rel(4)</seealso> manual page in + SASL.</p> <p>The binary boot script file <c>Name.boot</c> is generated from - the boot script file <c>Name.script</c> using the function + the boot script file <c>Name.script</c>, using the function <c>systools:script2boot(File)</c>.</p> </section> </section> @@ -148,16 +151,17 @@ init:stop()</pre> <marker id="code_loading"></marker> <title>Code Loading Strategy</title> <p>The runtime system can be started in either <em>embedded</em> or - <em>interactive</em> mode. Which one is decided by the command - line flag <c>-mode</c>.</p> + <em>interactive</em> mode. Which one is decided by the + command-line flag <c>-mode</c>.</p> <pre> % <input>erl -mode embedded</input></pre> <p>Default mode is <c>interactive</c>.</p> + <p>The mode properties are as follows:</p> <list type="bulleted"> - <item>In embedded mode, all code is loaded during system start-up + <item>In embedded mode, all code is loaded during system startup according to the boot script. (Code can also be loaded later - by explicitly ordering the code server to do so).</item> - <item>In interactive mode, code is dynamically loaded when first + by explicitly ordering the code server to do so.)</item> + <item>In interactive mode, the code is dynamically loaded when first referenced. When a call to a function in a module is made, and the module is not loaded, the code server searches the code path and loads the module into the system.</item> @@ -165,21 +169,21 @@ init:stop()</pre> <p>Initially, the code path consists of the current working directory and all object code directories under <c>ROOT/lib</c>, where <c>ROOT</c> is the installation directory - of Erlang/OTP. Directories can be named <c>Name[-Vsn]</c> and - the code server, by default, chooses the directory with + of Erlang/OTP. Directories can be named <c>Name[-Vsn]</c>. The + code server, by default, chooses the directory with the highest version number among those which have the same <c>Name</c>. The <c>-Vsn</c> suffix is optional. If an <c>ebin</c> directory exists under the <c>Name[-Vsn]</c> - directory, it is this directory which is added to the code path.</p> - <p>The code path can be extended by using the command line flags - <c>-pa Directories</c> and <c>-pz Directories</c>. These will add - <c>Directories</c> to the head or end of the code path, - respectively. Example</p> + directory, this directory is added to the code path.</p> + <p>The code path can be extended by using the command-line flags + <c>-pa Directories</c> and <c>-pz Directories</c>. These add + <c>Directories</c> to the head or the end of the code path, + respectively. Example:</p> <pre> % <input>erl -pa /home/arne/mycode</input></pre> <p>The code server module <c>code</c> contains a number of - functions for modifying and checking the search path, see - <c>code(3)</c>.</p> + functions for modifying and checking the search path, see the + <c>code(3)</c> manual page in Kernel.</p> </section> <section> @@ -192,49 +196,65 @@ init:stop()</pre> <cell align="left" valign="middle"><em>Documented in</em></cell> </row> <row> - <cell align="left" valign="middle">module</cell> + <cell align="left" valign="middle">Module</cell> <cell align="left" valign="middle"><c>.erl</c></cell> - <cell align="left" valign="middle">Erlang Reference Manual</cell> + <cell align="left" valign="middle"> + <seealso marker="doc/reference_manual:modules"> + Erlang Reference Manual</seealso></cell> </row> <row> - <cell align="left" valign="middle">include file</cell> + <cell align="left" valign="middle">Include file</cell> <cell align="left" valign="middle"><c>.hrl</c></cell> - <cell align="left" valign="middle">Erlang Reference Manual</cell> + <cell align="left" valign="middle"> + <seealso marker="doc/reference_manual:modules"> + Erlang Reference Manual</seealso></cell> </row> <row> - <cell align="left" valign="middle">release resource file</cell> + <cell align="left" valign="middle">Release resource file</cell> <cell align="left" valign="middle"><c>.rel</c></cell> - <cell align="left" valign="middle"><c>rel(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="sasl:rel">rel(4)</seealso> + manual page in SASL</cell> </row> <row> - <cell align="left" valign="middle">application resource file</cell> + <cell align="left" valign="middle">Application resource file</cell> <cell align="left" valign="middle"><c>.app</c></cell> - <cell align="left" valign="middle"><c>app(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="kernel:app">app(4)</seealso> + manual page in Kernel</cell> </row> <row> - <cell align="left" valign="middle">boot script</cell> + <cell align="left" valign="middle">Boot script</cell> <cell align="left" valign="middle"><c>.script</c></cell> - <cell align="left" valign="middle"><c>script(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="sasl:script">script(4)</seealso> + manual page in SASL</cell> </row> <row> - <cell align="left" valign="middle">binary boot script</cell> + <cell align="left" valign="middle">Binary boot script</cell> <cell align="left" valign="middle"><c>.boot</c></cell> <cell align="left" valign="middle">-</cell> </row> <row> - <cell align="left" valign="middle">configuration file</cell> + <cell align="left" valign="middle">Configuration file</cell> <cell align="left" valign="middle"><c>.config</c></cell> - <cell align="left" valign="middle"><c>config(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="kernel:config">config(4)</seealso> + manual page in Kernel</cell> </row> <row> - <cell align="left" valign="middle">application upgrade file</cell> + <cell align="left" valign="middle">Application upgrade file</cell> <cell align="left" valign="middle"><c>.appup</c></cell> - <cell align="left" valign="middle"><c>appup(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="sasl:appup">appup(4)</seealso> + manual page in SASL</cell> </row> <row> - <cell align="left" valign="middle">release upgrade file</cell> + <cell align="left" valign="middle">Release upgrade file</cell> <cell align="left" valign="middle"><c>relup</c></cell> - <cell align="left" valign="middle"><c>relup(4)</c></cell> + <cell align="left" valign="middle"> + <seealso marker="sasl:relup">relup(4)</seealso> + manual page in SASL</cell> </row> <tcaption>File Types</tcaption> </table> diff --git a/system/doc/system_principles/upgrade.xml b/system/doc/system_principles/upgrade.xml index 68e48da0b8..83e8128f94 100644 --- a/system/doc/system_principles/upgrade.xml +++ b/system/doc/system_principles/upgrade.xml @@ -31,88 +31,72 @@ <rev></rev> <file>upgrade.xml</file> </header> + <marker id="upgrade section"></marker> <section> <title>Introduction</title> - <p> - As of Erlang/OTP 17, most applications deliver a valid - application upgrade (<c>appup</c>) file. In earlier releases, a + <marker id="upgrade"></marker> + <p>As of Erlang/OTP 17, most applications deliver a valid + application upgrade file (<c>appup</c>). In earlier releases, a majority of the applications in Erlang/OTP did not support - upgrade at all. Many of the applications use the + upgrade. Many of the applications use the <c>restart_application</c> instruction. These are applications for which it is not crucial to support real soft upgrade, for - instance tools and library applications. The + example, tools and library applications. The <c>restart_application</c> instruction ensures that all modules in the application are reloaded and - thereby running the new code. - </p> + thereby running the new code.</p> </section> <section> - <title>Upgrade of core applications</title> - <p> - The core applications ERTS, Kernel, STDLIB + <title>Upgrade of Core Applications</title> + <p>The core applications ERTS, Kernel, STDLIB, and SASL never allow real soft upgrade, but require the Erlang emulator to be restarted. This is indicated to the <c>release_handler</c> by the upgrade instruction - <c>restart_new_emulator</c>. This instruction will always be the - very first instruction executed, and it will restart the + <c>restart_new_emulator</c>. This instruction is always the + very first instruction executed, and it restarts the emulator with the new versions of the above mentioned core - applications and the old versions of all other - applications. When the node is back up all other upgrade instructions are + applications and the old versions of all other applications. + When the node is back up, all other upgrade instructions are executed, making sure each application is finally running its - new version. - </p> - - <p> - It might seem strange to do a two-step upgrade instead of + new version.</p> + <p>It might seem strange to do a two-step upgrade instead of just restarting the emulator with the new version of all applications. The reason for this design decision is to allow - <c>code_change</c> functions to have side effects, for example changing - data on disk. It also makes sure that the upgrade mechanism for - non-core applications does not differ depending on whether or not - core applications are changed at the same time. - </p> - - <p> - If, however, the more brutal variant is preferred, it is - possible to handwrite the release upgrade file using only the + <c>code_change</c> functions to have side effects, for example, + changing data on disk. It also guarantees that the upgrade + mechanism for non-core applications does not differ depending + on whether or not core applications are changed at the same time.</p> + <p>If, however, the more brutal variant is preferred, the + the release upgrade file can be handwritten using only the single upgrade instruction <c>restart_emulator</c>. This - instruction, in contrast to <c>restart_new_emulator</c>, will - cause the emulator to restart with the new versions of - <em>all</em> applications. - </p> - - <p> - <em>Note</em> that if other instructions are included before + instruction, in contrast to <c>restart_new_emulator</c>, + causes the emulator to restart with the new versions of + <em>all</em> applications.</p> + <p><em>Note:</em> If other instructions are included before <c>restart_emulator</c> in the handwritten <c>relup</c> file, - they will be executed in the old emulator. This is a big risk + they are executed in the old emulator. This is a big risk since there is no guarantee that new beam code can be loaded into the old emulator. Adding instructions after <c>restart_emulator</c> has no effect as the - <c>release_handler</c> will not do any attempt at executing - them. - </p> - - <p> - See <seealso marker="sasl:relup">relup(4)</seealso> for - information about the release upgrade file, and <seealso - marker="sasl:appup">appup(4)</seealso> for further information - about upgrade instructions. - </p> + <c>release_handler</c> will not execute them.</p> + <p>For information about the release upgrade file, see the + <seealso marker="sasl:relup">relup(4)</seealso> manual page + in SASL. + For more information about upgrade instructions, see the + <seealso marker="sasl:appup">appup(4)</seealso> manual page + in SASL.</p> </section> <section> - <title>Applications that still do not allow code upgrade</title> - <p> - A few applications, for instance HiPE do not support - upgrade at all. This is indicated by an application upgrade file - containing only <c>{Vsn,[],[]}</c>. Any attempt at creating a release - upgrade file with such input will fail. - The only way to force an upgrade involving applications like this is to - handwrite the <c>relup</c> file, preferably as described above - with only the <c>restart_emulator</c> instruction. - </p> - + <title>Applications that Still do Not Allow Code Upgrade</title> + <p>A few applications, such as HiPE, do not support upgrade. + This is indicated by an application upgrade file containing only + <c>{Vsn,[],[]}</c>. Any attempt at creating a release upgrade file + with such input fails. The only way to force an upgrade involving + applications like this is to + handwrite the file <c>relup</c>, preferably as described above + with only the <c>restart_emulator</c> instruction.</p> </section> </chapter> diff --git a/system/doc/system_principles/versions.xml b/system/doc/system_principles/versions.xml index ff042f4a3b..a0f13774ce 100644 --- a/system/doc/system_principles/versions.xml +++ b/system/doc/system_principles/versions.xml @@ -31,93 +31,100 @@ <rev></rev> <file>versions.xml</file> </header> - <section><title>OTP Version</title> + <marker id="versions section"></marker> + + <section> + <title>OTP Version</title> <p>As of OTP release 17, the OTP release number corresponds to the major part of the OTP version. The OTP version as a concept was - introduced in OTP 17. The <seealso marker="#version_scheme">version - scheme</seealso> used is described in more detail below.</p> - + introduced in OTP 17. The version scheme used is described in detail in + <seealso marker="#version_scheme">Version Scheme</seealso>.</p> <p>OTP of a specific version is a set of applications of specific versions. The application versions identified by an OTP version corresponds to application versions that have been tested together - by the Erlang/OTP team at Ericsson AB. An OTP system can however be + by the Erlang/OTP team at Ericsson AB. An OTP system can, however, be put together with applications from different OTP versions. Such a combination of application versions has not been tested by the Erlang/OTP team. It is therefore <em>always preferred to use OTP applications from one single OTP version</em>.</p> - <p>Release candidates have an <c>-rc<N></c> - suffix. The suffix <c>-rc0</c> will be used during development up to + suffix. The suffix <c>-rc0</c> is used during development up to the first release candidate.</p> - <section><title>Retrieving Current OTP Version</title> - <p>In an OTP source code tree, the OTP version can be read from - the text file <c><OTP source root>/OTP_VERSION</c>. The - absolute path to the file can be constructed by calling - <c>filename:join([<seealso marker="kernel:code#root_dir/0">code:root_dir()</seealso>, "OTP_VERSION"])</c>.</p> - <p>In an installed OTP development system, the OTP version can be read - from the text file <c><OTP installation root>/releases/<OTP release number>/OTP_VERSION</c>. - The absolute path to the file can by constructed by calling - <c>filename:join([<seealso marker="kernel:code#root_dir/0">code:root_dir()</seealso>, "releases", <seealso marker="erts:erlang#system_info_otp_release">erlang:system_info(otp_release)</seealso>, "OTP_VERSION"]).</c></p> - <p>If the version read from the <c>OTP_VERSION</c> file in a - development system has a <c>**</c> suffix, the system has been - patched using the <c>otp_patch_apply</c> tool available to - licensed customers. In this case, the system consists of application - versions from multiple OTP versions. The version preceding the <c>**</c> - suffix corresponds to the OTP version of the base system that - has been patched. Note that if a development system is updated by - other means than <c>otp_patch_apply</c>, the <c>OTP_VERSION</c> file - may identify an incorrect OTP version.</p> - - <p>No <c>OTP_VERSION</c> file will be placed in a - <seealso marker="create_target">target system</seealso> created - by OTP tools. This since one easily can create a target system - where it is hard to even determine the base OTP version. You may, - however, place such a file there yourself if you know the OTP - version.</p> + <section> + <title>Retrieving Current OTP Version</title> + <p>In an OTP source code tree, the OTP version can be read from + the text file <c><OTP source root>/OTP_VERSION</c>. The + absolute path to the file can be constructed by calling + <c>filename:join([<seealso marker="kernel:code#root_dir/0">code:root_dir()</seealso>, "OTP_VERSION"])</c>.</p> + <p>In an installed OTP development system, the OTP version can be read + from the text file <c><OTP installation root>/releases/<OTP release number>/OTP_VERSION</c>. + The absolute path to the file can by constructed by calling + <c>filename:join([<seealso marker="kernel:code#root_dir/0">code:root_dir()</seealso>, "releases", <seealso marker="erts:erlang#system_info_otp_release"> erlang:system_info(otp_release)</seealso>, "OTP_VERSION"]).</c></p> + <p>If the version read from the <c>OTP_VERSION</c> file in a + development system has a <c>**</c> suffix, the system has been + patched using the + <seealso marker="../installation_guide/OTP-PATCH-APPLY"><c>otp_patch_apply</c></seealso> + tool. In this case, the system consists of application + versions from multiple OTP versions. The version preceding the <c>**</c> + suffix corresponds to the OTP version of the base system that + has been patched. Notice that if a development system is updated by + other means than <c>otp_patch_apply</c>, the file <c>OTP_VERSION</c> + can identify an incorrect OTP version.</p> + <p>No <c>OTP_VERSION</c> file is placed in a + <seealso marker="create_target">target system</seealso> created + by OTP tools. This since one easily can create a target system + where it is hard to even determine the base OTP version. You can, + however, place such a file there if you know the OTP version.</p> </section> - <section><title>OTP Versions Table</title> - <p>The text file <c><OTP source root>/otp_versions.table</c> that - is part of the source code contains information about all OTP versions - from OTP 17.0 up to current OTP version. Each line contains information - about application versions that are part of a specific OTP version, and - is on the format:</p> -<pre> -<OtpVersion> : <ChangedAppVersions> # <UnchangedAppVersions> : -</pre> - <p><c><OtpVersion></c> is on the format <c>OTP-<VSN></c>, i.e., - the same as the git tag used to identify the source. - <c><ChangedAppVersions></c> and <c><UnchangedAppVersions></c> - are space separated lists of application versions on the - format <c><application>-<vsn></c>. - <c><ChangedAppVersions></c> corresponds to changed applications - with new version numbers in this OTP version, and - <c><UnchangedAppVersions></c> corresponds to unchanged application - versions in this OTP version. Both of them might be empty, although - not at the same time. If <ChangedAppVersions> is empty, no changes - has been made that change the build result of any application. This could - for example be a pure bug fix of the build system. The order of lines - is undefined. All white space characters in this file are either space - (character 32) or line-break (character 10).</p> - <p>Using ordinary UNIX tools like <c>sed</c> and <c>grep</c> one - can easily find answers to various questions like:</p> - <taglist> - <tag>Which OTP versions are <c>kernel-3.0</c> part of?</tag> - <item><p><c> $ grep ' kernel-3\.0 ' otp_versions.table</c></p></item> - <tag>In which OTP version was <c>kernel-3.0</c> introduced?</tag> - <item><p><c> $ sed 's/#.*//;/ kernel-3\.0 /!d' otp_versions.table</c></p></item> - </taglist> - <p>The above commands give a bit more information than the exact answers, - but adequate information when manually searching for answers to these - questions.</p> - <warning><p>The format of the <c>otp_versions.table</c> might be subject - to changes during the OTP 17 release.</p></warning> - </section> + <section> + <title>OTP Versions Table</title> + <p>The text file <c><OTP source root>/otp_versions.table</c>, + which is part of the source code, contains information about all + OTP versions from OTP 17.0 up to the current OTP version. Each line + contains information about application versions that are part of a + specific OTP version, and has the following format:</p> + <pre> +<OtpVersion> : <ChangedAppVersions> # <UnchangedAppVersions> :</pre> + <p><c><OtpVersion></c> has the format <c>OTP-<VSN></c>, + that is, the same as the git tag used to identify the source.</p> + <p><c><ChangedAppVersions></c> and + <c><UnchangedAppVersions></c> are space-separated lists of + application versions and has the format + <c><application>-<vsn></c>.</p> + <list type="bulleted"> + <item><c><ChangedAppVersions></c> corresponds to changed + applications with new version numbers in this OTP version.</item> + <item><c><UnchangedAppVersions></c> corresponds to unchanged + application versions in this OTP version.</item> + </list> + <p>Both of them can be empty, but not at the same time. + If <c><ChangedAppVersions></c> is empty, no changes have + been made that change the build result of any application. This could, + for example, be a pure bug fix of the build system. The order of lines + is undefined. All white-space characters in this file are either space + (character 32) or line-break (character 10).</p> + <p>By using ordinary UNIX tools like <c>sed</c> and <c>grep</c> one + can easily find answers to various questions like:</p> + <list type="bulleted"> + <item><p>Which OTP versions are <c>kernel-3.0</c> part of?</p> + <p><c>$ grep ' kernel-3\.0 ' otp_versions.table</c> </p></item> + <item><p>In which OTP version was <c>kernel-3.0</c> introduced?</p> + <p><c>$ sed 's/#.*//;/ kernel-3\.0 /!d' otp_versions.table</c> + </p></item> + </list> + <p>The above commands give a bit more information than the exact + answers, but adequate information when manually searching for answers + to these questions.</p> + <warning><p>The format of the <c>otp_versions.table</c> might be + subject to changes during the OTP 17 release.</p></warning> </section> + </section> - <section><title>Application Version</title> - <p>As of OTP 17.0 application versions will use the same + <section> + <title>Application Version</title> + <p>As of OTP 17.0 application versions use the same <seealso marker="#version_scheme">version scheme</seealso> as the OTP version. Application versions part of a release candidate will however not have an <c>-rc<N></c> suffix as the OTP version. @@ -125,88 +132,88 @@ necessarily imply a major increment of the OTP version. This depends on whether the major change in the application is considered as a major change for OTP as a whole or not.</p> - </section> + </section> + <section> + <title>Version Scheme</title> <marker id="version_scheme"/> - <section><title>Version Scheme</title> - <note>Note that the version scheme was changed as of OTP 17.0. This implies + <note><p>The version scheme was changed as of OTP 17.0. This implies that application versions used prior to OTP 17.0 do not adhere to this version scheme. <seealso marker="#otp_17_0_app_versions">A list of - application versions used in OTP 17.0</seealso> can be found - at the end of this document.</note> - - <p>In the normal case, a version will be constructed as - <c><Major>.<Minor>.<Patch></c> where <c><Major></c> - is the most significant part. However, more dot separated parts than - this may exist. The dot separated parts consists of non-negative integers. - If all parts less significant than <c><Minor></c> equals <c>0</c>, - they are omitted. The three normal parts - <c><Major>.<Minor>.<Patch></c> will be changed as + application versions used in OTP 17.0</seealso> is included at the + end of this section</p></note> + <p>In the normal case, a version is constructed as + <c><Major>.<Minor>.<Patch></c>, + where <c><Major></c> is the most significant part.</p> + <p>However, more dot-separated parts than this can exist. + The dot-separated parts consist of non-negative integers. If + all parts less significant than <c><Minor></c> equals + <c>0</c>, they are omitted. The three normal parts + <c><Major>.<Minor>.<Patch></c> are changed as follows:</p> - <taglist> - <tag><c><Major></c></tag><item>Increased when major changes, - including incompatibilities, have been made.</item> - <tag><c><Minor></c></tag><item>Increased when new functionality - has been added.</item> - <tag><c><Patch></c></tag><item>Increased when pure bug fixes - have been made.</item> - </taglist> - <p>When a part in the version number is increased, all less significant + <list type="bulleted"> + <item><c><Major></c> - Increases when major changes, + including incompatibilities, are made.</item> + <item><c><Minor></c> - Increases when new + functionality is added.</item> + <item><c><Patch></c> - Increases when pure bug fixes + are made.</item> + </list> + <p>When a part in the version number increases, all less significant parts are set to <c>0</c>.</p> - <p>An application version or an OTP version identifies source code - versions. That is, it does not imply anything about how the application + versions. That is, it implies nothing about how the application or OTP has been built.</p> - <section><title>Order of Versions</title> - <p>Version numbers in general are only partially ordered. However, - normal version numbers (with three parts) as of OTP 17.0 have a total - or linear order. This applies both to normal OTP versions and - normal application versions.</p> - - <p>When comparing two version numbers that have an order, one - compare each part as ordinary integers from the most - significant part towards less significant parts. The order is - defined by the first parts of the same significance that - differ. An OTP version with a larger version include all - changes that that are part of a smaller OTP version. The same - goes for application versions.</p> - - <p>In the general case, versions may have more than three parts. In - this case the versions are only partially ordered. Note that such - versions are only used in exceptional cases. When an extra - part (out of the normal three parts) is added to a version number, - a new branch of versions is made. The new branch has a linear - order against the base version. However, versions on different - branches have no order. Since they have no order, we - only know that they all include what is included in their - closest common ancestor. When branching multiple times from the - same base version, <c>0</c> parts are added between the base - version and the least significant <c>1</c> part until a unique - version is found. Versions that have an order can be compared - as described in the paragraph above.</p> - - <p>An example of branched versions: The version <c>6.0.2.1</c> - is a branched version from the base version <c>6.0.2</c>. - Versions on the form <c>6.0.2.<X></c> can be compared - with normal versions smaller than or equal to <c>6.0.2</c>, - and other versions on the form <c>6.0.2.<X></c>. The - version <c>6.0.2.1</c> will include all changes in - <c>6.0.2</c>. However, <c>6.0.3</c> will most likely - <em>not</em> include all changes in <c>6.0.2.1</c> (note that - these versions have no order). A second branched version from the base - version <c>6.0.2</c> will be version <c>6.0.2.0.1</c>, and a - third branched version will be <c>6.0.2.0.0.1</c>.</p> - </section> + <section> + <title>Order of Versions</title> + <p>Version numbers in general are only partially ordered. However, + normal version numbers (with three parts) as of OTP 17.0 have a total + or linear order. This applies both to normal OTP versions and + normal application versions.</p> + <p>When comparing two version numbers that have an order, one + compare each part as ordinary integers from the most + significant part to less significant parts. The order is + defined by the first parts of the same significance that + differ. An OTP version with a larger version includes all + changes that are part of a smaller OTP version. The same + goes for application versions.</p> + <p>In general, versions can have more than three parts. + The versions are then only partially ordered. Such + versions are only used in exceptional cases. When an extra + part (out of the normal three parts) is added to a version number, + a new branch of versions is made. The new branch has a linear + order against the base version. However, versions on different + branches have no order, and therefore one can only conclude + that they all include what is included in their + closest common ancestor. When branching multiple times from the + same base version, <c>0</c> parts are added between the base + version and the least significant <c>1</c> part until a unique + version is found. Versions that have an order can be compared + as described in the previous paragraph.</p> + <p>An example of branched versions: The version <c>6.0.2.1</c> + is a branched version from the base version <c>6.0.2</c>. + Versions on the form <c>6.0.2.<X></c> can be compared + with normal versions smaller than or equal to <c>6.0.2</c>, + and other versions on the form <c>6.0.2.<X></c>. The + version <c>6.0.2.1</c> will include all changes in + <c>6.0.2</c>. However, <c>6.0.3</c> will most likely + <em>not</em> include all changes in <c>6.0.2.1</c> (note that + these versions have no order). A second branched version from the base + version <c>6.0.2</c> will be version <c>6.0.2.0.1</c>, and a + third branched version will be <c>6.0.2.0.0.1</c>.</p> </section> + </section> + <section> + <title>OTP 17.0 Application Versions</title> <marker id="otp_17_0_app_versions"/> - <section><title>OTP 17.0 Application Versions</title> - <p>The following application versions were part of OTP 17.0. If - the normal part of an applications version number compares - as smaller than the corresponding application version in this list, + <p>The following list details the application versions that + were part of OTP 17.0. If + the normal part of an application version number compares + as smaller than the corresponding application version in the list, the version number does not adhere to the version scheme introduced - in OTP 17.0 and should be considered as not having an order against + in OTP 17.0 and is to be considered as not having an order against versions used as of OTP 17.0.</p> <list> <item><c>asn1-3.0</c></item> @@ -262,6 +269,6 @@ <item><c>wx-1.2</c></item> <item><c>xmerl-1.3.7</c></item> </list> - </section> + </section> </chapter> diff --git a/system/doc/tutorial/c_port.xmlsrc b/system/doc/tutorial/c_port.xmlsrc index 8579da8520..0631293237 100644 --- a/system/doc/tutorial/c_port.xmlsrc +++ b/system/doc/tutorial/c_port.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,16 +28,34 @@ <rev></rev> <file>c_port.xml</file> </header> - <p>This is an example of how to solve the <seealso marker="example">example problem</seealso> by using a port.</p> + <p>This section outlines an example of how to solve the example + problem in the <seealso marker="example">previous section</seealso> + by using a port.</p> + <p>The scenario is illustrated in the following figure:</p> <image file="../tutorial/port.gif"> - <icaption>Port Communication.</icaption> + <icaption>Port Communication</icaption> </image> <section> <title>Erlang Program</title> - <p>First of all communication between Erlang and C must be established by creating the port. The Erlang process which creates a port is said to be <em>the connected process</em> of the port. All communication to and from the port should go via the connected process. If the connected process terminates, so will the port (and the external program, if it is written correctly).</p> - <p>The port is created using the BIF <c>open_port/2</c> with <c>{spawn,ExtPrg}</c> as the first argument. The string <c>ExtPrg</c> is the name of the external program, including any command line arguments. The second argument is a list of options, in this case only <c>{packet,2}</c>. This option says that a two byte length indicator will be used to simplify the communication between C and Erlang. Adding the length indicator will be done automatically by the Erlang port, but must be done explicitly in the external C program.</p> - <p>The process is also set to trap exits which makes it possible to detect if the external program fails.</p> + <p>All communication between Erlang and C must be established by + creating the port. The Erlang process that creates a port is + said to be <em>the connected process</em> of the port. All + communication to and from the port must go through the connected + process. If the connected process terminates, the port also + terminates (and the external program, if it is written + properly).</p> + <p>The port is created using the BIF <c>open_port/2</c> with + <c>{spawn,ExtPrg}</c> as the first argument. The string + <c>ExtPrg</c> is the name of the external program, including any + command line arguments. The second argument is a list of + options, in this case only <c>{packet,2}</c>. This option says + that a 2 byte length indicator is to be used to simplify the + communication between C and Erlang. The Erlang port + automatically adds the length indicator, but this must be done + explicitly in the external C program.</p> + <p>The process is also set to trap exits, which enables detection + of failure of the external program:</p> <pre> -module(complex1). -export([start/1, init/1]). @@ -50,7 +68,9 @@ init(ExtPrg) -> process_flag(trap_exit, true), Port = open_port({spawn, ExtPrg}, [{packet, 2}]), loop(Port).</pre> - <p>Now it is possible to implement <c>complex1:foo/1</c> and <c>complex1:bar/1</c>. They both send a message to the <c>complex</c> process and receive the reply.</p> + <p>Now <c>complex1:foo/1</c> and <c>complex1:bar/1</c> can be + implemented. Both send a message to the <c>complex</c> process + and receive the following replies:</p> <pre> foo(X) -> call_port({foo, X}). @@ -63,7 +83,14 @@ call_port(Msg) -> {complex, Result} -> Result end.</pre> - <p>The <c>complex</c> process encodes the message into a sequence of bytes, sends it to the port, waits for a reply, decodes the reply and sends it back to the caller.</p> + <p>The <c>complex</c> process does the following:</p> + <list type="bulleted"> + <item>Encodes the message into a sequence of bytes.</item> + <item>Sends it to the port.</item> + <item>Waits for a reply.</item> + <item>Decodes the reply.</item> + <item>Sends it back to the caller:</item> + </list> <pre> loop(Port) -> receive @@ -75,37 +102,52 @@ loop(Port) -> end, loop(Port) end.</pre> - <p>Assuming that both the arguments and the results from the C functions will be less than 256, a very simple encoding/decoding scheme is employed where <c>foo</c> is represented by the byte 1, <c>bar</c> is represented by 2, and the argument/result is represented by a single byte as well.</p> + <p>Assuming that both the arguments and the results from the C + functions are less than 256, a simple encoding/decoding scheme + is employed. In this scheme, <c>foo</c> is represented by byte + 1, <c>bar</c> is represented by 2, and the argument/result is + represented by a single byte as well:</p> <pre> encode({foo, X}) -> [1, X]; encode({bar, Y}) -> [2, Y]. - + decode([Int]) -> Int.</pre> - <p>The resulting Erlang program, including functionality for stopping the port and detecting port failures is shown below. + <p>The resulting Erlang program, including functionality for + stopping the port and detecting port failures, is as follows: </p> <codeinclude file="complex1.erl" type="erl"/> </section> <section> <title>C Program</title> - <p>On the C side, it is necessary to write functions for receiving and sending - data with two byte length indicators from/to Erlang. By default, the C program - should read from standard input (file descriptor 0) and write to standard output - (file descriptor 1). Examples of such functions, <c>read_cmd/1</c> and - <c>write_cmd/2</c>, are shown below.</p> + <p>On the C side, it is necessary to write functions for receiving + and sending data with 2 byte length indicators from/to Erlang. + By default, the C program is to read from standard input (file + descriptor 0) and write to standard output (file descriptor 1). + Examples of such functions, <c>read_cmd/1</c> and + <c>write_cmd/2</c>, follows:</p> <codeinclude file="erl_comm.c" type="erl"/> - <p>Note that <c>stdin</c> and <c>stdout</c> are for buffered input/output and should not be used for the communication with Erlang!</p> - <p>In the <c>main</c> function, the C program should listen for a message from Erlang and, according to the selected encoding/decoding scheme, use the first byte to determine which function to call and the second byte as argument to the function. The result of calling the function should then be sent back to Erlang.</p> + <p>Notice that <c>stdin</c> and <c>stdout</c> are for buffered + input/output and must <em>not</em> be used for the communication + with Erlang.</p> + <p>In the <c>main</c> function, the C program is to listen for a + message from Erlang and, according to the selected + encoding/decoding scheme, use the first byte to determine which + function to call and the second byte as argument to the + function. The result of calling the function is then to be sent + back to Erlang:</p> <codeinclude file="port.c" tag="" type="none"></codeinclude> - <p>Note that the C program is in a <c>while</c>-loop checking for the return value of <c>read_cmd/1</c>. The reason for this is that the C program must detect when the port gets closed and terminate.</p> + <p>Notice that the C program is in a <c>while</c>-loop, checking + for the return value of <c>read_cmd/1</c>. This is because the C + program must detect when the port closes and terminates.</p> </section> <section> <title>Running the Example</title> - <p>1. Compile the C code.</p> + <p><em>Step 1.</em> Compile the C code:</p> <pre> unix> <input>gcc -o extprg complex.c erl_comm.c port.c</input></pre> - <p>2. Start Erlang and compile the Erlang code.</p> + <p><em>Step 2.</em> Start Erlang and compile the Erlang code:</p> <pre> unix> <input>erl</input> Erlang (BEAM) emulator version 4.9.1.2 @@ -113,7 +155,7 @@ Erlang (BEAM) emulator version 4.9.1.2 Eshell V4.9.1.2 (abort with ^G) 1> <input>c(complex1).</input> {ok,complex1}</pre> - <p>3. Run the example.</p> + <p><em>Step 3.</em> Run the example:</p> <pre> 2> <input>complex1:start("extprg").</input> <0.34.0> diff --git a/system/doc/tutorial/c_portdriver.xmlsrc b/system/doc/tutorial/c_portdriver.xmlsrc index 2fd6fb0aac..61187703a4 100644 --- a/system/doc/tutorial/c_portdriver.xmlsrc +++ b/system/doc/tutorial/c_portdriver.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -21,46 +21,45 @@ </legalnotice> - <title>Port drivers</title> + <title>Port Drivers</title> <prepared></prepared> <docno></docno> <date></date> <rev></rev> <file>c_portdriver.xml</file> </header> - <p>This is an example of how to solve the <seealso marker="example">example problem</seealso> by using a linked in port driver.</p> + <p>This section outlines an example of how to solve the example problem + in <seealso marker="example">Problem Example</seealso> + by using a linked-in port driver.</p> + <p>A port driver is a linked-in driver that is accessible as a port + from an Erlang program. It is a shared library (SO in UNIX, DLL in + Windows), with special entry points. The Erlang runtime system + calls these entry points when the driver is started and when data + is sent to the port. The port driver can also send data to + Erlang.</p> + <p>As a port driver is dynamically linked into the emulator process, + this is the fastest way of calling C-code from Erlang. Calling + functions in the port driver requires no context switches. But it + is also the least safe way, because a crash in the port driver + brings the emulator down too.</p> + <p>The scenario is illustrated in the following figure:</p> <image file="../tutorial/port_driver.gif"> - <icaption>Port Driver Communication.</icaption> + <icaption>Port Driver Communication</icaption> </image> <section> - <title>Port Drivers</title> - <p>A port driver is a linked in driver that is accessible as a - port from an Erlang program. It is a shared library (SO in Unix, - DLL in Windows), with special entry points. The Erlang runtime - calls these entry points, when the driver is started and when - data is sent to the port. The port driver can also send data to - Erlang.</p> - <p>Since a port driver is dynamically linked into the emulator - process, this is the fastest way of calling C-code from Erlang. - Calling functions in the port driver requires no context - switches. But it is also the least safe, because a crash in the - port driver brings the emulator down too.</p> - </section> - - <section> <title>Erlang Program</title> - <p>Just as with a port program, the port communicates with a Erlang + <p>Like a port program, the port communicates with an Erlang process. All communication goes through one Erlang process that is the <em>connected process</em> of the port driver. Terminating this process closes the port driver.</p> <p>Before the port is created, the driver must be loaded. This is done with the function <c>erl_dll:load_driver/1</c>, with the name of the shared library as argument.</p> - <p>The port is then created using the BIF <c>open_port/2</c> with + <p>The port is then created using the BIF <c>open_port/2</c>, with the tuple <c>{spawn, DriverName}</c> as the first argument. The string <c>SharedLib</c> is the name of the port driver. The second - argument is a list of options, none in this case.</p> + argument is a list of options, none in this case:</p> <pre> -module(complex5). -export([start/1, init/1]). @@ -77,9 +76,9 @@ init(SharedLib) -> register(complex, self()), Port = open_port({spawn, SharedLib}, []), loop(Port).</pre> - <p>Now it is possible to implement <c>complex5:foo/1</c> and - <c>complex5:bar/1</c>. They both send a message to the - <c>complex</c> process and receive the reply.</p> + <p>Now <c>complex5:foo/1</c> and <c>complex5:bar/1</c> + can be implemented. Both send a message to the + <c>complex</c> process and receive the following reply:</p> <pre> foo(X) -> call_port({foo, X}). @@ -92,10 +91,14 @@ call_port(Msg) -> {complex, Result} -> Result end.</pre> - <p>The <c>complex</c> process encodes the message into a sequence - of bytes, sends it to the port, waits for a reply, decodes the - reply and sends it back to the caller. - </p> + <p>The <c>complex</c> process performs the following:</p> + <list type="bulleted"> + <item>Encodes the message into a sequence of bytes.</item> + <item>Sends it to the port.</item> + <item>Waits for a reply.</item> + <item>Decodes the reply.</item> + <item>Sends it back to the caller:</item> + </list> <pre> loop(Port) -> receive @@ -108,59 +111,58 @@ loop(Port) -> loop(Port) end.</pre> <p>Assuming that both the arguments and the results from the C - functions will be less than 256, a very simple encoding/decoding - scheme is employed where <c>foo</c> is represented by the byte + functions are less than 256, a simple encoding/decoding scheme + is employed. In this scheme, <c>foo</c> is represented by byte 1, <c>bar</c> is represented by 2, and the argument/result is - represented by a single byte as well. - </p> + represented by a single byte as well:</p> <pre> encode({foo, X}) -> [1, X]; encode({bar, Y}) -> [2, Y]. - + decode([Int]) -> Int.</pre> - <p>The resulting Erlang program, including functionality for - stopping the port and detecting port failures is shown below.</p> + <p>The resulting Erlang program, including functions for stopping + the port and detecting port failures, is as follows:</p> <codeinclude file="complex5.erl" type="erl"/> </section> <section> <title>C Driver</title> <p>The C driver is a module that is compiled and linked into a - shared library. It uses a driver structure, and includes the + shared library. It uses a driver structure and includes the header file <c>erl_driver.h</c>.</p> <p>The driver structure is filled with the driver name and function pointers. It is returned from the special entry point, declared with the macro <c><![CDATA[DRIVER_INIT(<driver_name>)]]></c>.</p> - <p>The functions for receiving and sending data, are combined into + <p>The functions for receiving and sending data are combined into a function, pointed out by the driver structure. The data sent - into the port is given as arguments, and the data the port - sends back is sent with the C-function <c>driver_output</c>.</p> - <p>Since the driver is a shared module, not a program, no main - function should be present. All function pointers are not used - in our example, and the corresponding fields in the + into the port is given as arguments, and the replied data is sent + with the C-function <c>driver_output</c>.</p> + <p>As the driver is a shared module, not a program, no main + function is present. All function pointers are not used + in this example, and the corresponding fields in the <c>driver_entry</c> structure are set to NULL.</p> - <p>All functions in the driver, takes a handle (returned from - <c>start</c>), that is just passed along by the erlang + <p>All functions in the driver takes a handle (returned from + <c>start</c>) that is just passed along by the Erlang process. This must in some way refer to the port driver instance.</p> - <p>The example_drv_start, is the only function that is called with - a handle to the port instance, so we must save this. It is - customary to use a allocated driver-defined structure for this - one, and pass a pointer back as a reference.</p> - <p>It is not a good idea to use a global variable; since the port - driver can be spawned by multiple Erlang processes, this - driver-structure should be instantiated multiple times. + <p>The <c>example_drv_start</c>, is the only function that is called with + a handle to the port instance, so this must be saved. It is + customary to use an allocated driver-defined structure for this + one, and to pass a pointer back as a reference.</p> + <p>It is not a good idea to use a global variable as the port + driver can be spawned by multiple Erlang processes. This + driver-structure is to be instantiated multiple times: </p> <codeinclude file="port_driver.c" tag="" type="none"></codeinclude> </section> <section> <title>Running the Example</title> - <p>1. Compile the C code.</p> + <p><em>Step 1.</em> Compile the C code:</p> <pre> unix> <input>gcc -o exampledrv -fpic -shared complex.c port_driver.c</input> windows> <input>cl -LD -MD -Fe exampledrv.dll complex.c port_driver.c</input></pre> - <p>2. Start Erlang and compile the Erlang code.</p> + <p><em>Step 2.</em> Start Erlang and compile the Erlang code:</p> <pre> > <input>erl</input> Erlang (BEAM) emulator version 5.1 @@ -168,7 +170,7 @@ Erlang (BEAM) emulator version 5.1 Eshell V5.1 (abort with ^G) 1> <input>c(complex5).</input> {ok,complex5}</pre> - <p>3. Run the example.</p> + <p><em>Step 3.</em> Run the example:</p> <pre> 2> <input>complex5:start("example_drv").</input> <0.34.0> diff --git a/system/doc/tutorial/cnode.xmlsrc b/system/doc/tutorial/cnode.xmlsrc index 293406160f..bcdd1298de 100644 --- a/system/doc/tutorial/cnode.xmlsrc +++ b/system/doc/tutorial/cnode.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,19 +28,39 @@ <rev></rev> <file>cnode.xml</file> </header> - <p>This is an example of how to solve the <seealso marker="example">example problem</seealso> by using a C node. Note that a C node would not typically be used for solving a simple problem like this, a port would suffice.</p> + <p>This section outlines an example of how to solve the example + problem in <seealso marker="example">Problem Example</seealso> + by using a C node. Notice that a C node is not typically + used for solving simple problems like this, a port is + sufficient.</p> <section> <title>Erlang Program</title> - <p>From Erlang's point of view, the C node is treated like a normal Erlang node. Therefore, calling the functions <c>foo</c> and <c>bar</c> only involves sending a message to the C node asking for the function to be called, and receiving the result. Sending a message requires a recipient; a process which can be defined using either a pid or a tuple consisting of a registered name and a node name. In this case a tuple is the only alternative as no pid is known.</p> + <p>From Erlang's point of view, the C node is treated like a + normal Erlang node. Thus, calling the functions <c>foo</c> and + <c>bar</c> only involves sending a message to the C node asking + for the function to be called, and receiving the result. Sending + a message requires a recipient, that is, a process that can be + defined using either a pid or a tuple, consisting of a + registered name and a node name. In this case, a tuple is the + only alternative as no pid is known:</p> <pre> {RegName, Node} ! Msg</pre> - <p>The node name <c>Node</c> should be the name of the C node. If short node names are used, the plain name of the node will be <c>cN</c> where <c>N</c> is an integer. If long node names are used, there is no such restriction. An example of a C node name using short node names is thus <c>c1@idril</c>, an example using long node names is <c>[email protected]</c>.</p> - <p>The registered name <c>RegName</c> could be any atom. The name can be ignored by the C code, or it could be used for example to distinguish between different types of messages. Below is an example of what the Erlang code could look like when using short node names. + <p>The node name <c>Node</c> is to be the name of the C node. If + short node names are used, the plain name of the node is + <c>cN</c>, where <c>N</c> is an integer. If long node names are + used, there is no such restriction. An example of a C node name + using short node names is thus <c>c1@idril</c>, an example using + long node names is <c>[email protected]</c>.</p> + <p>The registered name, <c>RegName</c>, can be any atom. The name + can be ignored by the C code, or, for example, be used to + distinguish between different types of messages. An example of + Erlang code using short node names follows: </p> <codeinclude file="complex3.erl" tag="" type="erl"></codeinclude> <p> - When using long node names the code is slightly different as shown in the following example: + When using long node names, the code is slightly different as + shown in the following example: </p> <codeinclude file="complex4.erl" tag="" type="erl"></codeinclude> @@ -50,39 +70,77 @@ <title>C Program</title> <section> - <title>Setting Up the Communication</title> - <p>Before calling any other Erl_Interface function, the memory handling must be initiated.</p> + <title>Setting Up Communication</title> + <p>Before calling any other function in Erl_Interface, the + memory handling must be initiated:</p> <pre> erl_init(NULL, 0);</pre> - <p>Now the C node can be initiated. If short node names are used, this is done by calling <c>erl_connect_init()</c>.</p> + <p>Now the C node can be initiated. If short node names are + used, this is done by calling <c>erl_connect_init()</c>:</p> <pre> erl_connect_init(1, "secretcookie", 0);</pre> - <p>The first argument is the integer which is used to construct the node name. In the example the plain node name will be <c>c1</c>. <br></br> - - The second argument is a string defining the magic cookie. <br></br> - - The third argument is an integer which is used to identify a particular instance of a C node.</p> - <p>If long node node names are used, initiation is done by calling <c>erl_connect_xinit()</c>.</p> + <p>Here:</p> + <list type="bulleted"> + <item>The first argument is the integer used to construct the node name. + <p>In the example, the plain node name is <c>c1</c>.</p></item> + <item>The second argument is a string defining the magic cookie.</item> + <item>The third argument is an integer that is used to identify + a particular instance of a C node.</item> + </list> + <p>If long node node names are used, initiation is done by + calling <c>erl_connect_xinit()</c>:</p> <pre> erl_connect_xinit("idril", "cnode", "[email protected]", &addr, "secretcookie", 0);</pre> - <p>The first three arguments are the host name, the plain node name, and the full node name. The fourth argument is a pointer to an <c>in_addr</c> struct with the IP address of the host, and the fifth and sixth arguments are the magic cookie and instance number.</p> - <p>The C node can act as a server or a client when setting up the communication Erlang-C. If it acts as a client, it connects to an Erlang node by calling <c>erl_connect()</c>, which will return an open file descriptor at success.</p> + <p>Here:</p> + <list type="bulleted"> + <item>The first argument is the host name.</item> + <item>The second argument is the plain node name.</item> + <item>The third argument is the full node name.</item> + <item>The fourth argument is a pointer to an <c>in_addr</c> + struct with the IP address of the host.</item> + <item>The fifth argument is the magic cookie.</item> + <item>The sixth argument is the instance number.</item> + </list> + <p>The C node can act as a server or a client when setting up + the Erlang-C communication. If it acts as a client, it + connects to an Erlang node by calling <c>erl_connect()</c>, + which returns an open file descriptor at success:</p> <pre> fd = erl_connect("e1@idril");</pre> - <p>If the C node acts as a server, it must first create a socket (call <c>bind()</c> and <c>listen()</c>) listening to a certain port number <c>port</c>. It then publishes its name and port number with <c>epmd</c> (the Erlang port mapper daemon, see the man page for <c>epmd</c>).</p> + <p>If the C node acts as a server, it must first create a socket + (call <c>bind()</c> and <c>listen()</c>) listening to a + certain port number <c>port</c>. It then publishes its name + and port number with <c>epmd</c>, the Erlang port mapper + daemon. For details, see the <seealso + marker="erts:epmd">epmd</seealso> manual page in ERTS:</p> <pre> erl_publish(port);</pre> - <p>Now the C node server can accept connections from Erlang nodes.</p> + <p>Now the C node server can accept connections from Erlang nodes:</p> <pre> fd = erl_accept(listen, &conn);</pre> - <p>The second argument to <c>erl_accept</c> is a struct <c>ErlConnect</c> that will contain useful information when a connection has been established; for example, the name of the Erlang node.</p> + <p>The second argument to <c>erl_accept</c> is a struct + <c>ErlConnect</c> which contains useful information when a + connection has been established, for example, the name of the + Erlang node.</p> </section> <section> <title>Sending and Receiving Messages</title> - <p>The C node can receive a message from Erlang by calling <c>erl_receive msg()</c>. This function reads data from the open file descriptor <c>fd</c> into a buffer and puts the result in an <c>ErlMessage</c> struct <c>emsg</c>. <c>ErlMessage</c> has a field <c>type</c> defining which kind of data was received. In this case the type of interest is <c>ERL_REG_SEND</c> which indicates that Erlang sent a message to a registered process at the C node. The actual message, an <c>ETERM</c>, will be in the <c>msg</c> field.</p> - <p>It is also necessary to take care of the types <c>ERL_ERROR</c> (an error occurred) and <c>ERL_TICK</c> (alive check from other node, should be ignored). Other possible types indicate process events such as link/unlink and exit.</p> + <p>The C node can receive a message from Erlang by calling + <c>erl_receive msg()</c>. This function reads data from the + open file descriptor <c>fd</c> into a buffer and puts the + result in an <c>ErlMessage</c> struct <c>emsg</c>. + <c>ErlMessage</c> has a field <c>type</c> defining what kind + of data is received. In this case, the type of interest is + <c>ERL_REG_SEND</c> which indicates that Erlang sent a message + to a registered process at the C node. The actual message, an + <c>ETERM</c>, is in the <c>msg</c> field.</p> + <p>It is also necessary to take care of the types + <c>ERL_ERROR</c> (an error occurred) and <c>ERL_TICK</c> + (alive check from other node, is to be ignored). Other + possible types indicate process events such as link, unlink, + and exit:</p> <pre> while (loop) { @@ -93,7 +151,16 @@ fd = erl_accept(listen, &conn);</pre> loop = 0; /* exit while loop */ } else { if (emsg.type == ERL_REG_SEND) {</pre> - <p>Since the message is an <c>ETERM</c> struct, Erl_Interface functions can be used to manipulate it. In this case, the message will be a 3-tuple (because that was how the Erlang code was written, see above). The second element will be the pid of the caller and the third element will be the tuple <c>{Function,Arg}</c> determining which function to call with which argument. The result of calling the function is made into an <c>ETERM</c> struct as well and sent back to Erlang using <c>erl_send()</c>, which takes the open file descriptor, a pid and a term as arguments.</p> + <p>As the message is an <c>ETERM</c> struct, Erl_Interface + functions can be used to manipulate it. In this case, the + message becomes a 3-tuple, because that is how the Erlang code + is written. The second element will be the pid of the caller + and the third element will be the tuple <c>{Function,Arg}</c> + determining which function to call, and with which argument. + The result of calling the function is made into an + <c>ETERM</c> struct as well and sent back to Erlang using + <c>erl_send()</c>, which takes the open file descriptor, a + pid, and a term as arguments:</p> <pre> fromp = erl_element(2, emsg.msg); tuplep = erl_element(3, emsg.msg); @@ -108,29 +175,30 @@ fd = erl_accept(listen, &conn);</pre> resp = erl_format("{cnode, ~i}", res); erl_send(fd, fromp, resp);</pre> - <p>Finally, the memory allocated by the <c>ETERM</c> creating functions (including <c>erl_receive_msg()</c> must be freed.</p> + <p>Finally, the memory allocated by the <c>ETERM</c> creating + functions (including <c>erl_receive_msg()</c> must be + freed:</p> <pre> erl_free_term(emsg.from); erl_free_term(emsg.msg); erl_free_term(fromp); erl_free_term(tuplep); erl_free_term(fnp); erl_free_term(argp); erl_free_term(resp);</pre> - <p>The resulting C programs can be found in looks like the following examples. First a C node server using short node names.</p> + <p>The following examples show the resulting C programs. + First a C node server using short node names:</p> <codeinclude file="cnode_s.c" type="c"/> - <p>Below follows a C node server using long node names.</p> + <p>A C node server using long node names:</p> <codeinclude file="cnode_s2.c" type="c"/> - <p>And finally we have the code for the C node client.</p> + <p>Finally, the code for the C node client:</p> <codeinclude file="cnode_c.c" type="c"/> </section> </section> <section> <title>Running the Example</title> - <p>1. Compile the C code, providing the paths to the Erl_Interface include files and libraries, and to the <c>socket</c> and <c>nsl</c> libraries.</p> - <p>In R5B and later versions of OTP, the <c>include</c> and <c>lib</c> directories are situated under <c>OTPROOT/lib/erl_interface-VSN</c>, where <c>OTPROOT</c> is the root directory of the OTP installation (<c>/usr/local/otp</c> in the example above) and <c>VSN</c> is the version of the <c>erl_interface</c> application (3.2.1 in the example above). <br></br> - - In R4B and earlier versions of OTP, <c>include</c> and <c>lib</c> are situated under <c>OTPROOT/usr</c>.</p> + <p><em>Step 1.</em> Compile the C code. This provides the paths to + the Erl_Interface include files and libraries, and to the + <c>socket</c> and <c>nsl</c> libraries:</p> <pre> - > <input>gcc -o cserver \\ </input> <input>-I/usr/local/otp/lib/erl_interface-3.2.1/include \\ </input> <input>-L/usr/local/otp/lib/erl_interface-3.2.1/lib \\ </input> @@ -148,11 +216,29 @@ unix> <input>gcc -o cclient \\ </input> <input>-L/usr/local/otp/lib/erl_interface-3.2.1/lib \\ </input> <input>complex.c cnode_c.c \\ </input> <input>-lerl_interface -lei -lsocket -lnsl</input></pre> - <p>2. Compile the Erlang code.</p> + <p>In Erlang/OTP R5B and later versions of OTP, the + <c>include</c> and <c>lib</c> directories are situated under + <c>OTPROOT/lib/erl_interface-VSN</c>, where <c>OTPROOT</c> is + the root directory of the OTP installation + (<c>/usr/local/otp</c> in the recent example) and <c>VSN</c> is + the version of the Erl_Interface application (3.2.1 in the + recent example).</p> + <p>In R4B and earlier versions of OTP, <c>include</c> and + <c>lib</c> are situated under <c>OTPROOT/usr</c>.</p> + <p><em>Step 2.</em> Compile the Erlang code:</p> <pre> unix> <input>erl -compile complex3 complex4</input></pre> - <p>3. Run the C node server example with short node names.</p> - <p>Start the C program <c>cserver</c> and Erlang in different windows. <c>cserver</c> takes a port number as argument and must be started before trying to call the Erlang functions. The Erlang node should be given the short name <c>e1</c> and must be set to use the same magic cookie as the C node, <c>secretcookie</c>.</p> + <p><em>Step 3.</em> Run the C node server example with short node names.</p> + <p>Do as follows:</p> + <list type="bulleted"> + <item>Start the C program <c>cserver</c> and Erlang in + different windows.</item> + <item><c>cserver</c> takes a port number as argument and must + be started before trying to call the Erlang functions.</item> + <item>The Erlang node is to be given the short name <c>e1</c> + and must be set to use the same magic cookie as the C node, + <c>secretcookie</c>:</item> + </list> <pre> unix> <input>cserver 3456</input> @@ -164,7 +250,9 @@ Eshell V4.9.1.2 (abort with ^G) 4 (e1@idril)2> <input>complex3:bar(5).</input> 10</pre> - <p>4. Run the C node client example. Terminate <c>cserver</c> but not Erlang and start <c>cclient</c>. The Erlang node must be started before the C node client is.</p> + <p><em>Step 4.</em> Run the C node client example. Terminate + <c>cserver</c>, but not Erlang, and start <c>cclient</c>. The + Erlang node must be started before the C node client:</p> <pre> unix> <input>cclient</input> @@ -172,7 +260,7 @@ unix> <input>cclient</input> 4 (e1@idril)4> <input>complex3:bar(5).</input> 10</pre> - <p>5. Run the C node server, long node names, example.</p> + <p><em>Step 5.</em> Run the C node server example with long node names:</p> <pre> unix> <input>cserver2 3456</input> diff --git a/system/doc/tutorial/erl_interface.xmlsrc b/system/doc/tutorial/erl_interface.xmlsrc index 0c4c5a99c2..5751a945d6 100644 --- a/system/doc/tutorial/erl_interface.xmlsrc +++ b/system/doc/tutorial/erl_interface.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,14 +28,29 @@ <rev></rev> <file>erl_interface.xml</file> </header> - <p>This is an example of how to solve the <seealso marker="example">example problem</seealso> by using a port and <c>erl_interface</c>. It is necessary to read the <seealso marker="c_port">port example</seealso> before reading this chapter.</p> + + <p>This section outlines an example of how to solve the example + problem in <seealso marker="example">Problem Example</seealso> by + using a port and Erl_Interface. It is necessary to read the port + example in <seealso marker="c_port">Ports</seealso> before reading + this section.</p> <section> <title>Erlang Program</title> - <p>The example below shows an Erlang program communicating with a C program over a plain port with home made encoding.</p> - <codeinclude file="complex1.erl" type="erl"/> - <p>Compared to the Erlang module - above used for the plain port, there are two differences when using Erl_Interface on the C side: Since Erl_Interface operates on the Erlang external term format the port must be set to use binaries and, instead of inventing an encoding/decoding scheme, the BIFs <c>term_to_binary/1</c> and <c>binary_to_term/1</c> should be used. That is:</p> + <p>The following example shows an Erlang program communicating + with a C program over a plain port with home made encoding:</p> + <codeinclude file="complex1.erl" type="erl"/> + <p>There are two differences when using Erl_Interface on the C + side compared to the example in <seealso marker="c_port"> + Ports</seealso>, using only the plain port:</p> + <list type="bulleted"> + <item>As Erl_Interface operates on the Erlang external term format, + the port must be set to use binaries.</item> + <item>Instead of inventing an encoding/decoding scheme, the + <c>term_to_binary/1</c> and <c>binary_to_term/1</c> BIFs are to + be used.</item> + </list> + <p>That is:</p> <pre> open_port({spawn, ExtPrg}, [{packet, 2}])</pre> <p>is replaced with:</p> @@ -55,69 +70,110 @@ receive {Port, {data, Data}} -> Caller ! {complex, binary_to_term(Data)} end</pre> - <p>The resulting Erlang program is shown below.</p> + <p>The resulting Erlang program is as follows:</p> <codeinclude file="complex2.erl" type="erl"/> - <p>Note that calling <c>complex2:foo/1</c> and <c>complex2:bar/1</c> will result in the tuple <c>{foo,X}</c> or <c>{bar,Y}</c> being sent to the <c>complex</c> process, which will code them as binaries and send them to the port. This means that the C program must be able to handle these two tuples.</p> + <p>Notice that calling <c>complex2:foo/1</c> and + <c>complex2:bar/1</c> results in the tuple <c>{foo,X}</c> or + <c>{bar,Y}</c> being sent to the <c>complex</c> process, which + codes them as binaries and sends them to the port. This means + that the C program must be able to handle these two tuples.</p> </section> <section> <title>C Program</title> - <p>The example below shows a C program communicating with an Erlang program over a plain port with home made encoding.</p> + <p>The following example shows a C program communicating with an + Erlang program over a plain port with home made encoding:</p> <codeinclude file="port.c" type="c"/> - <p>Compared to the C program above - used for the plain port the <c>while</c>-loop must be rewritten. Messages coming from the port will be on the Erlang external term format. They should be converted into an <c>ETERM</c> struct, a C struct similar to an Erlang term. The result of calling <c>foo()</c> or <c>bar()</c> must be converted to the Erlang external term format before being sent back to the port. But before calling any other <c>erl_interface</c> function, the memory handling must be initiated.</p> + <p>Compared to the C program in <seealso marker="c_port"> + Ports</seealso>, using only the plain port, the + <c>while</c>-loop must be rewritten. Messages coming from the + port is on the Erlang external term format. They must be + converted into an <c>ETERM</c> struct, which is a C struct + similar to an Erlang term. The result of calling <c>foo()</c> or + <c>bar()</c> must be converted to the Erlang external term + format before being sent back to the port. But before calling + any other Erl_Interface function, the memory handling must be + initiated:</p> <pre> erl_init(NULL, 0);</pre> - <p>For reading from and writing to the port the functions <c>read_cmd()</c> and <c>write_cmd()</c> from the erl_comm.c example below - can still be used. + <p>The following functions, <c>read_cmd()</c> and + <c>write_cmd()</c>, from the <c>erl_comm.c</c> example in + <seealso marker="c_port">Ports</seealso> can still be + used for reading from and writing to the port: </p> <codeinclude file="erl_comm.c" type="c"/> - <p>The function <c>erl_decode()</c> from <c>erl_marshal</c> will convert the binary into an <c>ETERM</c> struct.</p> + <p>The function <c>erl_decode()</c> from <c>erl_marshal</c> + converts the binary into an <c>ETERM</c> struct:</p> <pre> int main() { ETERM *tuplep; while (read_cmd(buf) > 0) { tuplep = erl_decode(buf);</pre> - <p>In this case <c>tuplep</c> now points to an <c>ETERM</c> struct representing a tuple with two elements; the function name (atom) and the argument (integer). By using the function <c>erl_element()</c> from <c>erl_eterm</c> it is possible to extract these elements, which also must be declared as pointers to an <c>ETERM</c> struct.</p> + <p>Here, <c>tuplep</c> points to an <c>ETERM</c> struct + representing a tuple with two elements; the function name (atom) + and the argument (integer). Using the function + <c>erl_element()</c> from <c>erl_eterm</c>, these elements can + be extracted, but they must also be declared as pointers to an + <c>ETERM</c> struct:</p> <pre> fnp = erl_element(1, tuplep); argp = erl_element(2, tuplep);</pre> - <p>The macros <c>ERL_ATOM_PTR</c> and <c>ERL_INT_VALUE</c> from <c>erl_eterm</c> can be used to obtain the actual values of the atom and the integer. The atom value is represented as a string. By comparing this value with the strings "foo" and "bar" it can be decided which function to call.</p> + <p>The macros <c>ERL_ATOM_PTR</c> and <c>ERL_INT_VALUE</c> from + <c>erl_eterm</c> can be used to obtain the actual values of the + atom and the integer. The atom value is represented as a string. + By comparing this value with the strings "foo" and "bar", it can + be decided which function to call:</p> <pre> if (strncmp(ERL_ATOM_PTR(fnp), "foo", 3) == 0) { res = foo(ERL_INT_VALUE(argp)); } else if (strncmp(ERL_ATOM_PTR(fnp), "bar", 3) == 0) { res = bar(ERL_INT_VALUE(argp)); }</pre> - <p>Now an <c>ETERM</c> struct representing the integer result can be constructed using the function <c>erl_mk_int()</c> from <c>erl_eterm</c>. It is also possible to use the function <c>erl_format()</c> from the module <c>erl_format</c>.</p> + <p>Now an <c>ETERM</c> struct that represents the integer result + can be constructed using the function <c>erl_mk_int()</c> from + <c>erl_eterm</c>. The function + <c>erl_format()</c> from the module <c>erl_format</c> can also + be used:</p> <pre> intp = erl_mk_int(res);</pre> - <p>The resulting <c>ETERM</c> struct is converted into the Erlang external term format using the function <c>erl_encode()</c> from <c>erl_marshal</c> and sent to Erlang using <c>write_cmd()</c>.</p> + <p>The resulting <c>ETERM</c> struct is converted into the Erlang + external term format using the function <c>erl_encode()</c> from + <c>erl_marshal</c> and sent to Erlang using + <c>write_cmd()</c>:</p> <pre> erl_encode(intp, buf); write_cmd(buf, erl_eterm_len(intp));</pre> - <p>Last, the memory allocated by the <c>ETERM</c> creating functions must be freed.</p> + <p>Finally, the memory allocated by the <c>ETERM</c> creating + functions must be freed:</p> <pre> erl_free_compound(tuplep); erl_free_term(fnp); erl_free_term(argp); erl_free_term(intp);</pre> - <p>The resulting C program is shown below:</p> + <p>The resulting C program is as follows:</p> <codeinclude file="ei.c" type="c"/> </section> <section> <title>Running the Example</title> - <p>1. Compile the C code, providing the paths to the include files <c>erl_interface.h</c> and <c>ei.h</c>, and to the libraries <c>erl_interface</c> and <c>ei</c>.</p> + <p><em>Step 1.</em> Compile the C code. This provides the paths to + the include files <c>erl_interface.h</c> and <c>ei.h</c>, and + also to the libraries <c>erl_interface</c> and <c>ei</c>:</p> <pre> unix> <input>gcc -o extprg -I/usr/local/otp/lib/erl_interface-3.2.1/include \\ </input> <input> -L/usr/local/otp/lib/erl_interface-3.2.1/lib \\ </input> <input> complex.c erl_comm.c ei.c -lerl_interface -lei</input></pre> - <p>In R5B and later versions of OTP, the <c>include</c> and <c>lib</c> directories are situated under <c>OTPROOT/lib/erl_interface-VSN</c>, where <c>OTPROOT</c> is the root directory of the OTP installation (<c>/usr/local/otp</c> in the example above) and <c>VSN</c> is the version of the <c>erl_interface</c> application (3.2.1 in the example above). <br></br> - - In R4B and earlier versions of OTP, <c>include</c> and <c>lib</c> are situated under <c>OTPROOT/usr</c>.</p> - <p>2. Start Erlang and compile the Erlang code.</p> + <p>In Erlang/OTP R5B and later versions of OTP, the <c>include</c> + and <c>lib</c> directories are situated under + <c>OTPROOT/lib/erl_interface-VSN</c>, where <c>OTPROOT</c> is + the root directory of the OTP installation + (<c>/usr/local/otp</c> in the recent example) and <c>VSN</c> is + the version of the Erl_interface application (3.2.1 in the + recent example).</p> + <p>In R4B and earlier versions of OTP, <c>include</c> and <c>lib</c> + are situated under <c>OTPROOT/usr</c>.</p> + <p><em>Step 2.</em> Start Erlang and compile the Erlang code:</p> <pre> unix> <input>erl</input> Erlang (BEAM) emulator version 4.9.1.2 @@ -125,7 +181,7 @@ Erlang (BEAM) emulator version 4.9.1.2 Eshell V4.9.1.2 (abort with ^G) 1> <input>c(complex2).</input> {ok,complex2}</pre> - <p>3. Run the example.</p> + <p><em>Step 3.</em> Run the example:</p> <pre> 2> <input>complex2:start("extprg").</input> <0.34.0> diff --git a/system/doc/tutorial/example.xmlsrc b/system/doc/tutorial/example.xmlsrc index f87eb217e9..e205ca189e 100644 --- a/system/doc/tutorial/example.xmlsrc +++ b/system/doc/tutorial/example.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -31,16 +31,25 @@ <section> <title>Description</title> - <p>A common interoperability situation is when there exists a piece of code solving some complex problem, and we would like to incorporate this piece of code in our Erlang program. Suppose for example we have the following C functions that we would like to be able to call from Erlang.</p> - <codeinclude file="complex.c" tag="" type="none"></codeinclude> - <p>(For the sake of keeping the example as simple as possible, the functions are not very complicated in this case).</p> - <p>Preferably we would like to able to call <c>foo</c> and <c>bar</c> without having to bother about them actually being C functions.</p> + <p>A common interoperability situation is when you want to incorporate + a piece of code, solving a complex problem, in your Erlang + program. Suppose for example, that you have the following C + functions that you would like to call from Erlang:</p> + <codeinclude file="complex.c" tag="" type="none"></codeinclude> + <p>The functions are deliberately kept as simple as possible, for + readability reasons.</p> + <p>From an Erlang perspektive, it is preferable to be able to call + <c>foo</c> and <c>bar</c> without having to bother about that + they are C functions:</p> <pre> % Erlang code ... Res = complex:foo(X), ...</pre> - <p>The communication with C is hidden in the implementation of <c>complex.erl</c>. In the following chapters it is shown how this module can be implemented using the different interoperability mechanisms.</p> + <p>Here, the communication with C is hidden in the implementation + of <c>complex.erl</c>. + In the following sections, it is shown how this module can be + implemented using the different interoperability mechanisms.</p> </section> </chapter> diff --git a/system/doc/tutorial/introduction.xml b/system/doc/tutorial/introduction.xml index ed86a00f76..dcf462e311 100644 --- a/system/doc/tutorial/introduction.xml +++ b/system/doc/tutorial/introduction.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,18 +28,34 @@ <rev></rev> <file>introduction.xml</file> </header> + <marker id="interoperability tutorial"></marker> + <p>This section informs on interoperability, that is, information + exchange, between Erlang and other programming languages. The + included examples mainly treat interoperability between Erlang and + C.</p> <section> <title>Purpose</title> - <p>The purpose of this tutorial is to give the reader an orientation of the different interoperability mechanisms that can be used when integrating a program written in Erlang with a program written in another programming language, from the Erlang programmer's point of view.</p> + <p>The purpose of this tutorial is to describe different + interoperability mechanisms that can be used when integrating a + program written in Erlang with a program written in another + programming language, from the Erlang programmer's + perspective.</p> </section> <section> <title>Prerequisites</title> - <p>It is assumed that the reader is a skilled Erlang programmer, familiar with concepts such as Erlang data types, processes, messages and error handling.</p> - <p>To illustrate the interoperability principles C programs running in a UNIX environment have been used. It is assumed that the reader has enough knowledge to be able to apply these principles to the relevant programming languages and platforms.</p> + <p>It is assumed that you are a skilled Erlang programmer, + familiar with concepts such as Erlang data types, processes, + messages, and error handling.</p> + <p>To illustrate the interoperability principles, C programs + running in a UNIX environment have been used. It is assumed that + you have enough knowledge to apply these principles to the + relevant programming languages and platforms.</p> <note> - <p>For the sake of readability, the example code has been kept as simple as possible. It does not include functionality such as error handling, which might be vital in a real-life system.</p> + <p>For readability, the example code is kept as simple as + possible. For example, it does not include error handling, + which might be vital in a real-life system.</p> </note> </section> </chapter> diff --git a/system/doc/tutorial/nif.xmlsrc b/system/doc/tutorial/nif.xmlsrc index 8ddad60f74..c79370e8c8 100644 --- a/system/doc/tutorial/nif.xmlsrc +++ b/system/doc/tutorial/nif.xmlsrc @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -28,92 +28,105 @@ <rev></rev> <file>nif.xml</file> </header> - <p>This is an example of how to solve the <seealso marker="example">example problem</seealso> - by using NIFs. NIFs were introduced in R13B03 as an experimental - feature. It is a simpler and more efficient way of calling C-code - than using port drivers. NIFs are most suitable for synchronous functions like - <c>foo</c> and <c>bar</c> in the example, that does some - relatively short calculations without side effects and return the result.</p> - <section> - <title>NIFs</title> - <p>A NIF (Native Implemented Function) is a function that is - implemented in C instead of Erlang. NIFs appear as any other functions to - the callers. They belong to a module and are called like any other Erlang - functions. The NIFs of a module are compiled and linked into a dynamic - loadable shared library (SO in Unix, DLL in Windows). The NIF library must - be loaded in runtime by the Erlang code of the module.</p> - <p>Since a NIF library is dynamically linked into the emulator - process, this is the fastest way of calling C-code from Erlang (alongside - port drivers). Calling NIFs requires no context switches. But it is also - the least safe, because a crash in a NIF will bring the emulator down - too.</p> - </section> + <p>This section outlines an example of how to solve the example + problem in <seealso marker="example">Problem Example</seealso> + by using Native Implemented Functions (NIFs).</p> + <p>NIFs were introduced in Erlang/OTP R13B03 as an experimental + feature. It is a simpler and more efficient way of calling C-code + than using port drivers. NIFs are most suitable for synchronous + functions, such as <c>foo</c> and <c>bar</c> in the example, that + do some relatively short calculations without side effects and + return the result.</p> + <p>A NIF is a function that is implemented in C instead of Erlang. + NIFs appear as any other functions to the callers. They belong to + a module and are called like any other Erlang functions. The NIFs + of a module are compiled and linked into a dynamic loadable, + shared library (SO in UNIX, DLL in Windows). The NIF library must + be loaded in runtime by the Erlang code of the module.</p> + <p>As a NIF library is dynamically linked into the emulator process, + this is the fastest way of calling C-code from Erlang (alongside + port drivers). Calling NIFs requires no context switches. But it + is also the least safe, because a crash in a NIF brings the + emulator down too.</p> <section> <title>Erlang Program</title> - <p>Even if all functions of a module will be NIFs, you still need an Erlang - module for two reasons. First, the NIF library must be explicitly loaded - by Erlang code in the same module. Second, all NIFs of a module must have - an Erlang implementation as well. Normally these are minimal stub - implementations that throw an exception. But it can also be used as - fallback implementations for functions that do not have native - implemenations on some architectures.</p> - <p>NIF libraries are loaded by calling <c>erlang:load_nif/2</c>, with the - name of the shared library as argument. The second argument can be any - term that will be passed on to the library and used for - initialization.</p> + <p>Even if all functions of a module are NIFs, an Erlang + module is still needed for two reasons:</p> + <list type="bulleted"> + <item>The NIF library must be explicitly loaded by + Erlang code in the same module.</item> + <item>All NIFs of a module must have an Erlang implementation + as well.</item> + </list> + <p>Normally these are minimal stub implementations that throw an + exception. But they can also be used as fallback implementations + for functions that do not have native implemenations on some + architectures.</p> + <p>NIF libraries are loaded by calling <c>erlang:load_nif/2</c>, + with the name of the shared library as argument. The second + argument can be any term that will be passed on to the library + and used for initialization:</p> <codeinclude file="complex6.erl" tag="" type="none"></codeinclude> - <p>We use the directive <c>on_load</c> to get function <c>init</c> to be - automatically called when the module is loaded. If <c>init</c> - returns anything other than <c>ok</c>, such when the loading of - the NIF library fails in this example, the module will be - unloaded and calls to functions within it will fail.</p> - <p>Loading the NIF library will override the stub implementations + <p>Here, the directive <c>on_load</c> is used to get function + <c>init</c> to be automatically called when the module is + loaded. If <c>init</c> returns anything other than <c>ok</c>, + such when the loading of the NIF library fails in this example, + the module is unloaded and calls to functions within it, + fail.</p> + <p>Loading the NIF library overrides the stub implementations and cause calls to <c>foo</c> and <c>bar</c> to be dispatched to the NIF implementations instead.</p> </section> <section> - <title>NIF library code</title> + <title>NIF Library Code</title> <p>The NIFs of the module are compiled and linked into a shared library. Each NIF is implemented as a normal C function. The macro <c>ERL_NIF_INIT</c> together with an array of structures defines the names, - arity and function pointers of all the NIFs in the module. The header - file <c>erl_nif.h</c> must be included. Since the library is a shared - module, not a program, no main function should be present.</p> + arity, and function pointers of all the NIFs in the module. The header + file <c>erl_nif.h</c> must be included. As the library is a shared + module, not a program, no main function is to be present.</p> <p>The function arguments passed to a NIF appears in an array <c>argv</c>, - with <c>argc</c> as the length of the array and thus the arity of the + with <c>argc</c> as the length of the array, and thus the arity of the function. The Nth argument of the function can be accessed as <c>argv[N-1]</c>. NIFs also take an environment argument that serves as an opaque handle that is needed to be passed on to most API functions. The environment contains information about - the calling Erlang process.</p> + the calling Erlang process:</p> <codeinclude file="complex6_nif.c" tag="" type="none"></codeinclude> - <p>The first argument to <c>ERL_NIF_INIT</c> must be the name of the + <p>Here,<c>ERL_NIF_INIT</c> has the following arguments:</p> + <list type="bulleted"> + <item><p>The first argument must be the name of the Erlang module as a C-identifier. It will be stringified by the - macro. The second argument is the array of <c>ErlNifFunc</c> - structures containing name, arity and function pointer of - each NIF. The other arguments are pointers to callback functions - that can be used to initialize the library. We do not use them - in this simple example so we set them all to <c>NULL</c>.</p> + macro.</p> + </item> + <item>The second argument is the array of <c>ErlNifFunc</c> + structures containing name, arity, and function pointer of + each NIF.</item> + <item>The remaining arguments are pointers to callback functions + that can be used to initialize the library. They are not used + in this simple example, hence they are all set to <c>NULL</c>.</item> + </list> <p>Function arguments and return values are represented as values - of type <c>ERL_NIF_TERM</c>. We use functions like <c>enif_get_int</c> - and <c>enif_make_int</c> to convert between Erlang term and C-type. - If the function argument <c>argv[0]</c> is not an integer then - <c>enif_get_int</c> will return false, in which case we return + of type <c>ERL_NIF_TERM</c>. Here, functions like <c>enif_get_int</c> + and <c>enif_make_int</c> are used to convert between Erlang term + and C-type. + If the function argument <c>argv[0]</c> is not an integer, + <c>enif_get_int</c> returns false, in which case it returns by throwing a <c>badarg</c>-exception with <c>enif_make_badarg</c>.</p> </section> <section> <title>Running the Example</title> - <p>1. Compile the C code.</p> + <p><em>Step 1.</em> Compile the C code:</p> <pre> unix> <input>gcc -o complex6_nif.so -fpic -shared complex.c complex6_nif.c</input> windows> <input>cl -LD -MD -Fe complex6_nif.dll complex.c complex6_nif.c</input></pre> - <p>2. Start Erlang and compile the Erlang code.</p> + <p><em>Step 2:</em> Start Erlang and compile the Erlang code:</p> <pre> > <input>erl</input> Erlang R13B04 (erts-5.7.5) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] @@ -121,7 +134,7 @@ Erlang R13B04 (erts-5.7.5) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-p Eshell V5.7.5 (abort with ^G) 1> <input>c(complex6).</input> {ok,complex6}</pre> - <p>3. Run the example.</p> + <p><em>Step 3:</em> Run the example:</p> <pre> 3> <input>complex6:foo(3).</input> 4 diff --git a/system/doc/tutorial/overview.xml b/system/doc/tutorial/overview.xml index 1fe1aad22b..3814a135b4 100644 --- a/system/doc/tutorial/overview.xml +++ b/system/doc/tutorial/overview.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>2000</year><year>2013</year> + <year>2000</year><year>2015</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -31,35 +31,90 @@ <section> <title>Built-In Mechanisms</title> - <p>There are two interoperability mechanisms built into the Erlang runtime system. One is <em>distributed Erlang</em> and the other one is <em>ports</em>. A variation of ports is <em>linked-in drivers</em>.</p> + <p>Two interoperability mechanisms are built into the Erlang + runtime system, <em>distributed Erlang</em> and <em>ports</em>. + A variation of ports is <em>linked-in drivers</em>.</p> <marker id="dist"></marker> <section> <title>Distributed Erlang</title> - <p>An Erlang runtime system is made into a distributed Erlang node by giving it a name. A distributed Erlang node can connect to and monitor other nodes, it is also possible to spawn processes at other nodes. Message passing and error handling between processes at different nodes are transparent. There exists a number of useful <c>stdlib</c> modules intended for use in a distributed Erlang system; for example, <c>global</c> which provides global name registration. The distribution mechanism is implemented using TCP/IP sockets.</p> - <p><em>When to use:</em> Distributed Erlang is primarily used for communication Erlang-Erlang. It can also be used for communication between Erlang and C, if the C program is implemented as a <seealso marker="#cnode">C node</seealso>, see below.</p> - <p><em>Where to read more:</em> Distributed Erlang and some distributed programming techniques are described in the Erlang book. <br></br> - - In the Erlang/OTP documentation there is a chapter about distributed Erlang in "Getting Started" (User's Guide). <br></br> - - Relevant man pages are <c>erlang</c> (describes the BIFs) and <c>global</c>, <c>net_adm</c>, <c>pg2</c>, <c>rpc</c>, <c>pool</c> and <c>slave</c>.</p> + <p>An Erlang runtime system is made a distributed Erlang node by + giving it a name. A distributed Erlang node can connect to, + and monitor, other nodes. It can also spawn processes at other + nodes. Message passing and error handling between processes at + different nodes are transparent. A number of useful STDLIB + modules are available in a distributed Erlang system. For + example, <c>global</c>, which provides global name + registration. The distribution mechanism is implemented using + TCP/IP sockets.</p> + <p><em>When to use:</em> Distributed Erlang is primarily used + for Erlang-Erlang communication. It can also be used for + communication between Erlang and C, if the C program is + implemented as a C node, see + <seealso marker="#cnode">C and Java Libraries</seealso>.</p> + <p><em>Where to read more:</em> Distributed Erlang and some distributed + programming techniques are described in the Erlang book.</p> + <p>For more information, see <seealso + marker="doc/getting_started:conc_prog#Distributed Programming"> + Distributed Programming.</seealso></p> + <p>Relevant manual pages are the following:</p> + <list type="bulleted"> + <item><seealso marker="erts:erlang">erlang</seealso> manual page in ERTS + (describes the BIFs)</item> + <item><seealso marker="kernel:global">global</seealso> manual page in Kernel</item> + <item><seealso marker="kernel:net_adm">net_adm</seealso> manual page in Kernel</item> + <item><seealso marker="kernel:pg2">pg2</seealso> manual page in Kernel</item> + <item><seealso marker="kernel:rpc">rpc</seealso> manual page in Kernel</item> + <item><seealso marker="stdlib:pool">pool</seealso> manual page in STDLIB</item> + <item><seealso marker="stdlib:slave">slave</seealso> manual page in STDLIB</item> + </list> </section> <section> <title>Ports and Linked-In Drivers</title> - <p>Ports provide the basic mechanism for communication with the external world, from Erlang's point of view. They provide a byte-oriented interface to an external program. When a port has been created, Erlang can communicate with it by sending and receiving lists of bytes (not Erlang terms). This means that the programmer may have to invent a suitable encoding and decoding scheme.</p> - <p>The actual implementation of the port mechanism depends on the platform. In the Unix case, pipes are used and the external program should as default read from standard input and write to standard output. Theoretically, the external program could be written in any programming language as long as it can handle the interprocess communication mechanism with which the port is implemented.</p> - <p>The external program resides in another OS process than the Erlang runtime system. In some cases this is not acceptable, consider for example drivers with very hard time requirements. It is therefore possible to write a program in C according to certain principles and dynamically link it to the Erlang runtime system, this is called a linked-in driver.</p> - <p><em>When to use:</em> Being the basic mechanism, ports can be used for all kinds of interoperability situations where the Erlang program and the other program runs on the same machine. Programming is fairly straight-forward. <br></br> - - Linked-in drivers involves writing certain call-back functions in C. Very good skills are required as the code is linked to the Erlang runtime system.</p> + <p>Ports provide the basic mechanism for communication with the + external world, from Erlang's point of view. The ports provide + a byte-oriented interface to an external program. When a port + is created, Erlang can communicate with it by sending and + receiving lists of bytes (not Erlang terms). This means that + the programmer might have to invent a suitable encoding and + decoding scheme.</p> + <p>The implementation of the port mechanism depends on the + platform. For UNIX, pipes are used and the external program is + assumed to read from standard input and write to standard + output. The external program can be written in any programming + language as long as it can handle the interprocess + communication mechanism with which the port is + implemented.</p> + <p>The external program resides in another OS process than the + Erlang runtime system. In some cases this is not acceptable. + Consider, for example, drivers with very hard time + requirements. It is therefore possible to write a program in C + according to certain principles, and dynamically link it to + the Erlang runtime system. This is called a <em>linked-in + driver</em>.</p> + <p><em>When to use:</em> Ports can be used for all kinds of + interoperability situations where the Erlang program and the + other program runs on the same machine. Programming is fairly + straight-forward.</p> + <p>Linked-in drivers involves writing certain call-back + functions in C. This requires very good skills as the code is + linked to the Erlang runtime system.</p> <warning> - <p>An erroneous linked-in driver will cause the entire Erlang runtime system to leak memory, hang or crash.</p> + <p>A faulty linked-in driver causes the entire Erlang runtime + system to leak memory, hang, or crash.</p> </warning> - <p><em>Where to read more:</em> Ports are described in the "Miscellaneous Items" chapter of the Erlang book. Linked-in drivers are described in Appendix E. <br></br> - - The BIF <c>open_port/2</c> is documented in the man page for <c>erlang</c>. For linked-in drivers, the programmer needs to read the information in the man page for <c>erl_ddll</c>.</p> - <p><em>Examples:</em><seealso marker="c_port">Port example</seealso>.</p> + <p><em>Where to read more:</em> Ports are described in section + "Miscellaneous Items" of the Erlang book. Linked-in drivers + are described in Appendix E.</p> + <p>The BIF <c>open_port/2</c> is documented in the + <seealso marker="erts:erlang">erlang</seealso> manual page in + ERTS.</p> + <p>For linked-in drivers, the programmer needs to read the + <seealso marker="kernel:erl_ddll">erl_ddll</seealso> manual + page in Kernel.</p> + <p><em>Examples:</em> Port example in <seealso marker="c_port"> + Ports</seealso>.</p> </section> </section> @@ -68,64 +123,152 @@ <section> <title>Erl_Interface</title> - <p>Very often the program at the other side of a port is a C program. To help the C programmer a library called Erl_Interface has been developed. It consists of five parts:</p> + <p>The program at the other side of a port is often a C program. + To help the C programmer, the Erl_Interface library + has been developed, including the following five parts:</p> <list type="bulleted"> - <item><c>erl_marshal</c>, <c>erl_eterm</c>, <c>erl_format</c>, <c>erl_malloc</c> Handling of the Erlang external term format.</item> - <item><c>erl_connect</c> Communication with distributed Erlang, see <seealso marker="#cnode">C nodes</seealso> below.</item> - <item><c>erl_error</c> Error print routines.</item> - <item><c>erl_global</c> Access globally registered names.</item> - <item><c>Registry</c> Store and backup of key-value pairs.</item> + <item> + <c>erl_marshal</c>, <c>erl_eterm</c>, <c>erl_format</c>, and + <c>erl_malloc</c>: Handling of the Erlang external term format</item> + <item> + <c>erl_connect</c>: + Communication with distributed Erlang, see <seealso + marker="#cnode">C nodes</seealso> below</item> + <item> + <c>erl_error</c>: + Error print routines</item> + <item> + <c>erl_global</c>: + Access globally registered names</item> + <item> + <c>Registry</c>: + Store and backup of key-value pairs</item> </list> - <p>The Erlang external term format is a representation of an Erlang term as a sequence of bytes, a binary. Conversion between the two representations is done using BIFs.</p> + <p>The Erlang external term format is a representation of an + Erlang term as a sequence of bytes, that is, a binary. + Conversion between the two representations is done using the + following BIFs:</p> <pre> Binary = term_to_binary(Term) Term = binary_to_term(Binary)</pre> - <p>A port can be set to use binaries instead of lists of bytes. It is then not necessary to invent any encoding/decoding scheme. Erl_Interface functions are used for unpacking the binary and convert it into a struct similar to an Erlang term. Such a struct can be manipulated in different ways and be converted to the Erlang external format and sent to Erlang.</p> + <p>A port can be set to use binaries instead of lists of bytes. + It is then not necessary to invent any encoding/decoding + scheme. Erl_Interface functions are used for unpacking the + binary and convert it into a struct similar to an Erlang term. + Such a struct can be manipulated in different ways, be + converted to the Erlang external format, and sent to + Erlang.</p> <p><em>When to use:</em> In C code, in conjunction with Erlang binaries.</p> - <p><em>Where to read more:</em> Read about the Erl_Interface User's Guide; Command Reference and Library Reference. In R5B and earlier versions the information can be found under the Kernel application.</p> - </section> - <p><em>Examples:</em><seealso marker="erl_interface">erl_interface example</seealso>.</p> + <p><em>Where to read more:</em> See the Erlang Interface User's + Guide, Command Reference, and Library Reference. In Erlang/OTP + R5B, and earlier versions, the information is part of the + Kernel application.</p> </section> + <p><em>Examples:</em> Erl_Interface example in + <seealso marker="erl_interface">Erl_Interface</seealso>.</p> <marker id="cnode"></marker> <section> <title>C Nodes</title> - <p>A C program which uses the Erl_Interface functions for setting up a connection to and communicating with a distributed Erlang node is called a <em>C node</em>, or a <em>hidden node</em>. The main advantage with a C node is that the communication from the Erlang programmer's point of view is extremely easy, since the C program behaves as a distributed Erlang node.</p> - <p><em>When to use:</em> C nodes can typically be used on device processors (as opposed to control processors) where C is a better choice than Erlang due to memory limitations and/or application characteristics.</p> - <p><em>Where to read more:</em> In the <c>erl_connect</c> part of the Erl_Interface documentation, see above. The programmer also needs to be familiar with TCP/IP sockets, see <seealso marker="#sockets">below</seealso>, and distributed Erlang, see <seealso marker="#dist">above</seealso>.</p> - <p><em>Examples:</em><seealso marker="cnode">C node example</seealso>.</p> + <p>A C program that uses the Erl_Interface functions for setting + up a connection to, and communicating with, a distributed + Erlang node is called a <em>C node</em>, or a <em>hidden + node</em>. The main advantage with a C node is that the + communication from the Erlang programmer's perspective is + extremely easy, as the C program behaves as a distributed + Erlang node.</p> + <p><em>When to use:</em> C nodes can typically be used on device + processors (as opposed to control processors) where C is a + better choice than Erlang due to memory limitations or + application characteristics, or both.</p> + <p><em>Where to read more:</em> See the <c>erl_connect</c> part + of the Erl_Interface documentation. The programmer also needs + to be familiar with TCP/IP sockets, see Sockets in <seealso + marker="#sockets">Standard + Protocols</seealso> and Distributed Erlang in <seealso + marker="#dist">Built-In Mechanisms</seealso>.</p> + <p><em>Example:</em> C node example in <seealso marker="cnode"> + C Nodes</seealso>.</p> </section> <section> <title>Jinterface</title> - <p>In Erlang/OTP R6B, a library similar to Erl_Interface for Java was added called <em>jinterface</em>.</p> + <p>In Erlang/OTP R6B, a library similar to Erl_Interface for + Java was added called <em>jinterface</em>. It provides a tool + for Java programs to communicate with Erlang nodes.</p> </section> </section> <section> <title>Standard Protocols</title> - <p>Sometimes communication between an Erlang program and another program using a standard protocol is desirable. Erlang/OTP currently supports TCP/IP and UDP <em>sockets</em>, SNMP, HTTP and IIOP (CORBA). Using one of the latter three requires good knowledge about the protocol and is not covered by this tutorial. Please refer to the documentation for the SNMP, Inets and Orber applications, respectively.</p> + <p>Sometimes communication between an Erlang program and another + program using a standard protocol is desirable. Erlang/OTP + currently supports TCP/IP and UDP <em>sockets</em>: as + follows:</p> + <list type="bulleted"> + <item>SNMP</item> + <item>HTTP</item> + <item>IIOP (CORBA)</item> + </list> + <p>Using one of the latter three requires good knowledge about the + protocol and is not covered by this tutorial. See the SNMP, + Inets, and Orber applications, respectively.</p> <marker id="sockets"></marker> <section> <title>Sockets</title> - <p>Simply put, connection-oriented socket communication (TCP/IP) consists of an initiator socket ("server") started at a certain host with a certain port number. A connector socket ("client") aware of the initiator's host name and port number can connect to it and data can be sent between them. Connection-less socket communication (UDP) consists of an initiator socket at a certain host with a certain port number and a connector socket sending data to it. For a detailed description of the socket concept, please refer to a suitable book about network programming. A suggestion is <em>UNIX Network Programming, Volume 1: Networking APIs - Sockets and XTI</em> by W. Richard Stevens, ISBN: 013490012X.</p> - <p>In Erlang/OTP, access to TCP/IP and UDP sockets is provided by the - Kernel modules <c>gen_tcp</c> and <c>gen_udp</c>. Both are easy to - use and do not require any deeper knowledge about the socket concept.</p> - <p><em>When to use:</em> For programs running on the same or on another machine than the Erlang program.</p> - <p><em>Where to read more:</em> The man pages for <c>gen_tcp</c> and <c>gen_udp</c>.</p> + <p>Simply put, connection-oriented socket communication (TCP/IP) + consists of an initiator socket ("server") started at a + certain host with a certain port number. A connector socket + ("client"), which is aware of the initiator host name and port + number, can connect to it and data can be sent between + them.</p> + <p>Connection-less socket communication (UDP) consists of an + initiator socket at a certain host with a certain port number + and a connector socket sending data to it.</p> + <p>For a detailed description of the socket concept, refer to a + suitable book about network programming. A suggestion is + <em>UNIX Network Programming, Volume 1: Networking APIs - + Sockets and XTI</em> by W. Richard Stevens, ISBN: + 013490012X.</p> + <p>In Erlang/OTP, access to TCP/IP and UDP sockets is provided + by the modules <c>gen_tcp</c> and <c>gen_udp</c> in + Kernel. Both are easy to use and do not require + detailed knowledge about the socket concept.</p> + <p><em>When to use:</em> For programs running on the same or on + another machine than the Erlang program.</p> + <p><em>Where to read more:</em> See the <seealso + marker="kernel:gen_tcp">gen_tcp</seealso> and the <seealso + marker="kernel:gen_udp">gen_udp</seealso> manual pages in + Kernel.</p> </section> </section> <section> <title>IC</title> - <p>IC (IDL Compiler) is an interface generator which given an IDL interface specification automatically generates stub code in Erlang, C or Java. Please refer to the IC User's Guide and IC Reference Manual.</p> + <p>IC (Erlang IDL Compiler) is an interface generator that, given + an IDL interface specification, automatically generates stub + code in Erlang, C, or Java. See the IC User's Guide and IC + Reference Manual.</p> + <p>For details, see the <seealso marker="ic:ic">ic</seealso> + manual page in IC.</p> </section> <section> <title>Old Applications</title> - <p>There are two old applications of interest when talking about interoperability: <em>IG</em> which was removed in Erlang/OTP R6B and <em>Jive</em> which was removed in Erlang/OTP R7B. Both applications have been replaced by IC and are mentioned here for reference only.</p> - <p>IG (Interface Generator) automatically generated code for port or socket communication between an Erlang program and a C program, given a C header file with certain keywords. Jive provided a simple interface between an Erlang program and a Java program.</p> + <p>Two old applications are of interest regarding + interoperability. Both have been replaced by IC and are + mentioned here for reference only:</p> + <list type="bulleted"> + <item><p>IG - Removed from Erlang/OTP R6B.</p> + <p>IG (Interface Generator) automatically generated code for + port or socket communication between an Erlang program and a + C program, given a C header file with certain keywords.</p> + </item> + <item><p>Jive - Removed from Erlang/OTP R7B.</p> + <p>Jive provided a simple interface between an Erlang program + and a Java program.</p> + </item> + </list> </section> </chapter> diff --git a/xcomp/erl-xcomp-arm-linux.conf b/xcomp/erl-xcomp-arm-linux.conf index 76912d25e0..6656c1a1aa 100644 --- a/xcomp/erl-xcomp-arm-linux.conf +++ b/xcomp/erl-xcomp-arm-linux.conf @@ -74,7 +74,8 @@ erl_xcomp_configure_flags="--disable-hipe" CC="arm-wrs-linux-gnueabi-gcc --sysroot=$ARM_SYSROOT" # * `CFLAGS' - C compiler flags. -#CFLAGS="-O@OPT_LEVEL@ -DSMALL_MEMORY" +CFLAGS="-O2 -DSMALL_MEMORY --sysroot=$ARM_SYSROOT -Wall -g" + # * `STATIC_CFLAGS' - Static C compiler flags. #STATIC_CFLAGS= @@ -87,19 +88,19 @@ CC="arm-wrs-linux-gnueabi-gcc --sysroot=$ARM_SYSROOT" CPP="arm-wrs-linux-gnueabi-cpp --sysroot=$ARM_SYSROOT" # * `CPPFLAGS' - C pre-processor flags. -#CPPFLAGS="--sysroot=$ARM_SYSROOT" +CPPFLAGS="--sysroot=$ARM_SYSROOT" # * `CXX' - C++ compiler. CXX="arm-wrs-linux-gnueabi-c++ --sysroot=$ARM_SYSROOT" # * `CXXFLAGS' - C++ compiler flags. -#CXXFLAGS= +CXXFLAGS="--sysroot=$ARM_SYSROOT" # * `LD' - Linker. -#LD= +LD="arm-wrs-linux-gnueabi-gcc" # * `LDFLAGS' - Linker flags. -#LDFLAGS= +LDFLAGS="--sysroot=$ARM_SYSROOT" # * `LIBS' - Libraries. #LIBS= @@ -109,14 +110,14 @@ CXX="arm-wrs-linux-gnueabi-c++ --sysroot=$ARM_SYSROOT" ## *NOTE*! Either set all or none of the `DED_LD*' variables. # * `DED_LD' - Linker for Dynamically loaded Erlang Drivers. -#DED_LD= +DED_LD="arm-wrs-linux-gnueabi-gcc" # * `DED_LDFLAGS' - Linker flags to use with `DED_LD'. -#DED_LDFLAGS= +DED_LDFLAGS="--sysroot=$ARM_SYSROOT -shared -Wl,-Bsymbolic" # * `DED_LD_FLAG_RUNTIME_LIBRARY_PATH' - This flag should set runtime library # search path for shared libraries when linking with `DED_LD'. -#DED_LD_FLAG_RUNTIME_LIBRARY_PATH= +DED_LD_FLAG_RUNTIME_LIBRARY_PATH="-Wl,-R" ## -- Large File Support -- |