diff options
245 files changed, 10934 insertions, 6202 deletions
diff --git a/.gitignore b/.gitignore index 3dcfa79f4d..455d3e5f55 100644 --- a/.gitignore +++ b/.gitignore @@ -360,15 +360,8 @@ JAVADOC-GENERATED # system -/system/doc/pdf/*.pdf -/system/doc/pdf/*.fo -/system/doc/html/*.html -/system/doc/html/*.eix -/system/doc/html/js -/system/doc/html/*/*.html -/system/doc/html/*/*.gif -/system/doc/html/*/*.jpg -/system/doc/html/*/*.eix +/system/doc/pdf +/system/doc/html /system/doc/top/PR.template /system/doc/top/erlresolvelinks.js /system/doc/programming_examples/funs.xml diff --git a/OTP_VERSION b/OTP_VERSION index 0b602e3cc8..6ea9a3bd47 100644 --- a/OTP_VERSION +++ b/OTP_VERSION @@ -1 +1 @@ -19.0-rc0 +19.0 diff --git a/bootstrap/lib/compiler/ebin/beam_bool.beam b/bootstrap/lib/compiler/ebin/beam_bool.beam Binary files differindex a793db8bed..4a302a8709 100644 --- a/bootstrap/lib/compiler/ebin/beam_bool.beam +++ b/bootstrap/lib/compiler/ebin/beam_bool.beam diff --git a/bootstrap/lib/compiler/ebin/beam_split.beam b/bootstrap/lib/compiler/ebin/beam_split.beam Binary files differindex 3a02e2b17c..4a63468419 100644 --- a/bootstrap/lib/compiler/ebin/beam_split.beam +++ b/bootstrap/lib/compiler/ebin/beam_split.beam diff --git a/bootstrap/lib/compiler/ebin/beam_utils.beam b/bootstrap/lib/compiler/ebin/beam_utils.beam Binary files differindex 661cd1c60d..664140540c 100644 --- a/bootstrap/lib/compiler/ebin/beam_utils.beam +++ b/bootstrap/lib/compiler/ebin/beam_utils.beam diff --git a/bootstrap/lib/compiler/ebin/cerl.beam b/bootstrap/lib/compiler/ebin/cerl.beam Binary files differindex 82bd89c17a..dea6cb4ee2 100644 --- a/bootstrap/lib/compiler/ebin/cerl.beam +++ b/bootstrap/lib/compiler/ebin/cerl.beam diff --git a/bootstrap/lib/compiler/ebin/cerl_clauses.beam b/bootstrap/lib/compiler/ebin/cerl_clauses.beam Binary files differindex 95dd3f3cbb..cc4e3a92a3 100644 --- a/bootstrap/lib/compiler/ebin/cerl_clauses.beam +++ b/bootstrap/lib/compiler/ebin/cerl_clauses.beam diff --git a/bootstrap/lib/compiler/ebin/cerl_inline.beam b/bootstrap/lib/compiler/ebin/cerl_inline.beam Binary files differindex ce1f13db6a..dcfb32d866 100644 --- a/bootstrap/lib/compiler/ebin/cerl_inline.beam +++ b/bootstrap/lib/compiler/ebin/cerl_inline.beam diff --git a/bootstrap/lib/compiler/ebin/core_parse.beam b/bootstrap/lib/compiler/ebin/core_parse.beam Binary files differindex baf2c9d70f..920dbc922f 100644 --- a/bootstrap/lib/compiler/ebin/core_parse.beam +++ b/bootstrap/lib/compiler/ebin/core_parse.beam diff --git a/bootstrap/lib/compiler/ebin/core_pp.beam b/bootstrap/lib/compiler/ebin/core_pp.beam Binary files differindex 6047596cb6..e274211c6c 100644 --- a/bootstrap/lib/compiler/ebin/core_pp.beam +++ b/bootstrap/lib/compiler/ebin/core_pp.beam diff --git a/bootstrap/lib/compiler/ebin/rec_env.beam b/bootstrap/lib/compiler/ebin/rec_env.beam Binary files differindex 9388b5d4e1..a7c78175c3 100644 --- a/bootstrap/lib/compiler/ebin/rec_env.beam +++ b/bootstrap/lib/compiler/ebin/rec_env.beam diff --git a/bootstrap/lib/compiler/ebin/sys_core_fold.beam b/bootstrap/lib/compiler/ebin/sys_core_fold.beam Binary files differindex 088d69fa2a..7d01ed81d5 100644 --- a/bootstrap/lib/compiler/ebin/sys_core_fold.beam +++ b/bootstrap/lib/compiler/ebin/sys_core_fold.beam diff --git a/bootstrap/lib/compiler/ebin/sys_pre_attributes.beam b/bootstrap/lib/compiler/ebin/sys_pre_attributes.beam Binary files differindex 934a539ad9..58dff1b796 100644 --- a/bootstrap/lib/compiler/ebin/sys_pre_attributes.beam +++ b/bootstrap/lib/compiler/ebin/sys_pre_attributes.beam diff --git a/bootstrap/lib/compiler/ebin/v3_codegen.beam b/bootstrap/lib/compiler/ebin/v3_codegen.beam Binary files differindex 0d76bb21e2..0923d4e678 100644 --- a/bootstrap/lib/compiler/ebin/v3_codegen.beam +++ b/bootstrap/lib/compiler/ebin/v3_codegen.beam diff --git a/bootstrap/lib/compiler/ebin/v3_core.beam b/bootstrap/lib/compiler/ebin/v3_core.beam Binary files differindex 3b3b8f4208..7be50a757f 100644 --- a/bootstrap/lib/compiler/ebin/v3_core.beam +++ b/bootstrap/lib/compiler/ebin/v3_core.beam diff --git a/bootstrap/lib/compiler/ebin/v3_kernel.beam b/bootstrap/lib/compiler/ebin/v3_kernel.beam Binary files differindex 5e3ae80d14..47eaf3b5e8 100644 --- a/bootstrap/lib/compiler/ebin/v3_kernel.beam +++ b/bootstrap/lib/compiler/ebin/v3_kernel.beam diff --git a/bootstrap/lib/kernel/ebin/disk_log.beam b/bootstrap/lib/kernel/ebin/disk_log.beam Binary files differindex 998d4bca2f..5aa8bf7dc1 100644 --- a/bootstrap/lib/kernel/ebin/disk_log.beam +++ b/bootstrap/lib/kernel/ebin/disk_log.beam diff --git a/bootstrap/lib/kernel/ebin/disk_log_1.beam b/bootstrap/lib/kernel/ebin/disk_log_1.beam Binary files differindex fc1c4781f1..cb627c10a5 100644 --- a/bootstrap/lib/kernel/ebin/disk_log_1.beam +++ b/bootstrap/lib/kernel/ebin/disk_log_1.beam diff --git a/bootstrap/lib/kernel/ebin/file.beam b/bootstrap/lib/kernel/ebin/file.beam Binary files differindex c82dada559..28d72dfafc 100644 --- a/bootstrap/lib/kernel/ebin/file.beam +++ b/bootstrap/lib/kernel/ebin/file.beam diff --git a/bootstrap/lib/kernel/ebin/global.beam b/bootstrap/lib/kernel/ebin/global.beam Binary files differindex b461d30baa..465dab3a52 100644 --- a/bootstrap/lib/kernel/ebin/global.beam +++ b/bootstrap/lib/kernel/ebin/global.beam diff --git a/bootstrap/lib/kernel/ebin/group.beam b/bootstrap/lib/kernel/ebin/group.beam Binary files differindex 0511d1fa09..38eb0e7a6c 100644 --- a/bootstrap/lib/kernel/ebin/group.beam +++ b/bootstrap/lib/kernel/ebin/group.beam diff --git a/bootstrap/lib/kernel/ebin/inet.beam b/bootstrap/lib/kernel/ebin/inet.beam Binary files differindex 60469432b9..8cfdf389ad 100644 --- a/bootstrap/lib/kernel/ebin/inet.beam +++ b/bootstrap/lib/kernel/ebin/inet.beam diff --git a/bootstrap/lib/kernel/ebin/inet_db.beam b/bootstrap/lib/kernel/ebin/inet_db.beam Binary files differindex 557852c917..30ab40c93f 100644 --- a/bootstrap/lib/kernel/ebin/inet_db.beam +++ b/bootstrap/lib/kernel/ebin/inet_db.beam diff --git a/bootstrap/lib/kernel/ebin/inet_parse.beam b/bootstrap/lib/kernel/ebin/inet_parse.beam Binary files differindex 30f05d5c09..c5c171aa54 100644 --- a/bootstrap/lib/kernel/ebin/inet_parse.beam +++ b/bootstrap/lib/kernel/ebin/inet_parse.beam diff --git a/bootstrap/lib/kernel/ebin/kernel_config.beam b/bootstrap/lib/kernel/ebin/kernel_config.beam Binary files differindex 8691aff64a..8379f77e94 100644 --- a/bootstrap/lib/kernel/ebin/kernel_config.beam +++ b/bootstrap/lib/kernel/ebin/kernel_config.beam diff --git a/bootstrap/lib/stdlib/ebin/dets.beam b/bootstrap/lib/stdlib/ebin/dets.beam Binary files differindex 754663b531..0dfaca7a48 100644 --- a/bootstrap/lib/stdlib/ebin/dets.beam +++ b/bootstrap/lib/stdlib/ebin/dets.beam diff --git a/bootstrap/lib/stdlib/ebin/dets_v9.beam b/bootstrap/lib/stdlib/ebin/dets_v9.beam Binary files differindex 3e02f4b0be..f381b9b624 100644 --- a/bootstrap/lib/stdlib/ebin/dets_v9.beam +++ b/bootstrap/lib/stdlib/ebin/dets_v9.beam diff --git a/bootstrap/lib/stdlib/ebin/digraph_utils.beam b/bootstrap/lib/stdlib/ebin/digraph_utils.beam Binary files differindex 66777eeb8f..80e91be663 100644 --- a/bootstrap/lib/stdlib/ebin/digraph_utils.beam +++ b/bootstrap/lib/stdlib/ebin/digraph_utils.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_eval.beam b/bootstrap/lib/stdlib/ebin/erl_eval.beam Binary files differindex 8793d70a39..d5bd46593c 100644 --- a/bootstrap/lib/stdlib/ebin/erl_eval.beam +++ b/bootstrap/lib/stdlib/ebin/erl_eval.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_expand_records.beam b/bootstrap/lib/stdlib/ebin/erl_expand_records.beam Binary files differindex f6c0d0a732..e8ce1fa0da 100644 --- a/bootstrap/lib/stdlib/ebin/erl_expand_records.beam +++ b/bootstrap/lib/stdlib/ebin/erl_expand_records.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_lint.beam b/bootstrap/lib/stdlib/ebin/erl_lint.beam Binary files differindex aeea1e2fcd..ebe80a32a8 100644 --- a/bootstrap/lib/stdlib/ebin/erl_lint.beam +++ b/bootstrap/lib/stdlib/ebin/erl_lint.beam diff --git a/bootstrap/lib/stdlib/ebin/erl_pp.beam b/bootstrap/lib/stdlib/ebin/erl_pp.beam Binary files differindex 9841362f9d..b3343c9ccb 100644 --- a/bootstrap/lib/stdlib/ebin/erl_pp.beam +++ b/bootstrap/lib/stdlib/ebin/erl_pp.beam diff --git a/bootstrap/lib/stdlib/ebin/file_sorter.beam b/bootstrap/lib/stdlib/ebin/file_sorter.beam Binary files differindex ac4f1a36f1..03d9020bb5 100644 --- a/bootstrap/lib/stdlib/ebin/file_sorter.beam +++ b/bootstrap/lib/stdlib/ebin/file_sorter.beam diff --git a/bootstrap/lib/stdlib/ebin/filelib.beam b/bootstrap/lib/stdlib/ebin/filelib.beam Binary files differindex a9783ed990..904edee66f 100644 --- a/bootstrap/lib/stdlib/ebin/filelib.beam +++ b/bootstrap/lib/stdlib/ebin/filelib.beam diff --git a/bootstrap/lib/stdlib/ebin/filename.beam b/bootstrap/lib/stdlib/ebin/filename.beam Binary files differindex b6248fe460..c129ac4f97 100644 --- a/bootstrap/lib/stdlib/ebin/filename.beam +++ b/bootstrap/lib/stdlib/ebin/filename.beam diff --git a/bootstrap/lib/stdlib/ebin/gen_event.beam b/bootstrap/lib/stdlib/ebin/gen_event.beam Binary files differindex 2486647f46..b53ac26280 100644 --- a/bootstrap/lib/stdlib/ebin/gen_event.beam +++ b/bootstrap/lib/stdlib/ebin/gen_event.beam diff --git a/bootstrap/lib/stdlib/ebin/gen_fsm.beam b/bootstrap/lib/stdlib/ebin/gen_fsm.beam Binary files differindex dbf38090af..595574585b 100644 --- a/bootstrap/lib/stdlib/ebin/gen_fsm.beam +++ b/bootstrap/lib/stdlib/ebin/gen_fsm.beam diff --git a/bootstrap/lib/stdlib/ebin/gen_server.beam b/bootstrap/lib/stdlib/ebin/gen_server.beam Binary files differindex e7026a2a9a..3097207512 100644 --- a/bootstrap/lib/stdlib/ebin/gen_server.beam +++ b/bootstrap/lib/stdlib/ebin/gen_server.beam diff --git a/bootstrap/lib/stdlib/ebin/gen_statem.beam b/bootstrap/lib/stdlib/ebin/gen_statem.beam Binary files differindex 5a3fd2a1a5..3dc104f1c3 100644 --- a/bootstrap/lib/stdlib/ebin/gen_statem.beam +++ b/bootstrap/lib/stdlib/ebin/gen_statem.beam diff --git a/bootstrap/lib/stdlib/ebin/lib.beam b/bootstrap/lib/stdlib/ebin/lib.beam Binary files differindex 541ea59421..06ddba221a 100644 --- a/bootstrap/lib/stdlib/ebin/lib.beam +++ b/bootstrap/lib/stdlib/ebin/lib.beam diff --git a/bootstrap/lib/stdlib/ebin/lists.beam b/bootstrap/lib/stdlib/ebin/lists.beam Binary files differindex 422a9bd214..22705c8a9a 100644 --- a/bootstrap/lib/stdlib/ebin/lists.beam +++ b/bootstrap/lib/stdlib/ebin/lists.beam diff --git a/bootstrap/lib/stdlib/ebin/otp_internal.beam b/bootstrap/lib/stdlib/ebin/otp_internal.beam Binary files differindex e1ebd34afd..ce8fc8a5c5 100644 --- a/bootstrap/lib/stdlib/ebin/otp_internal.beam +++ b/bootstrap/lib/stdlib/ebin/otp_internal.beam diff --git a/bootstrap/lib/stdlib/ebin/qlc.beam b/bootstrap/lib/stdlib/ebin/qlc.beam Binary files differindex f0ed1e961b..fa3240810d 100644 --- a/bootstrap/lib/stdlib/ebin/qlc.beam +++ b/bootstrap/lib/stdlib/ebin/qlc.beam diff --git a/bootstrap/lib/stdlib/ebin/qlc_pt.beam b/bootstrap/lib/stdlib/ebin/qlc_pt.beam Binary files differindex 17c698de60..4a065ec57f 100644 --- a/bootstrap/lib/stdlib/ebin/qlc_pt.beam +++ b/bootstrap/lib/stdlib/ebin/qlc_pt.beam diff --git a/bootstrap/lib/stdlib/ebin/queue.beam b/bootstrap/lib/stdlib/ebin/queue.beam Binary files differindex 4365c5518d..a63a321330 100644 --- a/bootstrap/lib/stdlib/ebin/queue.beam +++ b/bootstrap/lib/stdlib/ebin/queue.beam diff --git a/bootstrap/lib/stdlib/ebin/shell.beam b/bootstrap/lib/stdlib/ebin/shell.beam Binary files differindex 9fa5727aa5..b0c98f4b98 100644 --- a/bootstrap/lib/stdlib/ebin/shell.beam +++ b/bootstrap/lib/stdlib/ebin/shell.beam diff --git a/bootstrap/lib/stdlib/ebin/sofs.beam b/bootstrap/lib/stdlib/ebin/sofs.beam Binary files differindex 01f7810764..5099d4ecc0 100644 --- a/bootstrap/lib/stdlib/ebin/sofs.beam +++ b/bootstrap/lib/stdlib/ebin/sofs.beam diff --git a/erts/configure.in b/erts/configure.in index 11c428ced7..4a63381eb7 100644 --- a/erts/configure.in +++ b/erts/configure.in @@ -62,9 +62,6 @@ if test x"${ERL_TOP}/erts" != x"$srcdir"; then fi erl_top=${ERL_TOP} -# Remove old configuration information -/bin/rm -f "$ERL_TOP/erts/CONF_INFO" - # echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # echo X # echo "X srcdir = $srcdir" @@ -102,7 +99,7 @@ ERL_XCOMP_SYSROOT_INIT AC_ISC_POSIX -AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in include/$host/erl_native_features_config.h:include/erl_native_features_config.h.in) +AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in) dnl ---------------------------------------------------------------------- dnl Optional features. dnl ---------------------------------------------------------------------- @@ -140,7 +137,7 @@ AS_HELP_STRING([--enable-dirty-schedulers], [enable dirty scheduler support]), [ case "$enableval" in no) enable_dirty_schedulers=no ;; *) enable_dirty_schedulers=yes ;; - esac ], enable_dirty_schedulers=no) + esac ], enable_dirty_schedulers=default) AC_ARG_ENABLE(smp-support, AS_HELP_STRING([--enable-smp-support], [enable smp support]) @@ -729,8 +726,27 @@ esac AC_SUBST(LIBCARBON) -dnl Check if we should/can build a sharing-preserving emulator +_search_path=/bin:/usr/bin:/usr/local/bin:$PATH +AC_PATH_PROG(RM, rm, false, $_search_path) +if test "$ac_cv_path_RM" = false; then + AC_MSG_ERROR([No 'rm' command found]) +fi + +AC_PATH_PROG(MKDIR, mkdir, false, $_search_path) +if test "$ac_cv_path_MKDIR" = false; then + AC_MSG_ERROR([No 'mkdir' command found]) +fi + +_search_path= + + +# Remove old configuration information. +# Next line should be placed after AC_PATH_PROG(RM, ...), but before +# first output to CONN_INFO. So this is just the right place. +$RM -f "$ERL_TOP/erts/CONF_INFO" + +dnl Check if we should/can build a sharing-preserving emulator AC_MSG_CHECKING(if we are building a sharing-preserving emulator) if test "$enable_sharing_preserving" = "yes"; then AC_DEFINE(SHCOPY, [1], @@ -760,27 +776,13 @@ if test "$ac_cv_prog_AR" = false; then AC_MSG_ERROR([No 'ar' command found in PATH]) fi -_search_path=/bin:/usr/bin:/usr/local/bin:$PATH - -AC_PATH_PROG(RM, rm, false, $_search_path) -if test "$ac_cv_path_RM" = false; then - AC_MSG_ERROR([No 'rm' command found]) -fi - -AC_PATH_PROG(MKDIR, mkdir, false, $_search_path) -if test "$ac_cv_path_MKDIR" = false; then - AC_MSG_ERROR([No 'mkdir' command found]) -fi - -_search_path= - # # Get programs needed for building the documentation # ## Delete previous failed configure results if test -f doc/CONF_INFO; then - rm doc/CONF_INFO + $RM doc/CONF_INFO fi AC_CHECK_PROGS(XSLTPROC, xsltproc) @@ -1004,6 +1006,23 @@ case $ERTS_BUILD_SMP_EMU in ;; esac +AC_MSG_CHECKING(whether dirty schedulers should be enabled) +case $ERTS_BUILD_SMP_EMU-$enable_dirty_schedulers in + yes-yes) + DIRTY_SCHEDULER_SUPPORT=yes;; + yes-default) + ## Maybe yes for OTP 19... + DIRTY_SCHEDULER_SUPPORT=no;; + no-default) + DIRTY_SCHEDULER_SUPPORT=no;; + no-yes) + AC_MSG_ERROR([No smp emulator will be built, but dirty schedulers requested]);; + *) + DIRTY_SCHEDULER_SUPPORT=no;; +esac +AC_MSG_RESULT($DIRTY_SCHEDULER_SUPPORT) +AC_SUBST(DIRTY_SCHEDULER_SUPPORT) + if test $ERTS_BUILD_SMP_EMU = yes; then if test $found_threads = no; then @@ -1206,14 +1225,6 @@ esac if test $emu_threads != yes; then enable_lock_check=no enable_lock_count=no - AC_MSG_CHECKING(whether dirty schedulers should be enabled) - if test "x$enable_dirty_schedulers" != "xno"; then - AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support]) - AC_DEFINE(ERL_DRV_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support]) - AC_MSG_RESULT(yes) - else - AC_MSG_RESULT(no) - fi else # Threads enabled for emulator EMU_THR_LIB_NAME=$ETHR_LIB_NAME @@ -1233,17 +1244,6 @@ else EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_ENABLE_LOCK_COUNT" fi - AC_MSG_CHECKING(whether dirty schedulers should be enabled) - if test "x$enable_dirty_schedulers" != "xno"; then - EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_DIRTY_SCHEDULERS" - AC_DEFINE(ERTS_DIRTY_SCHEDULERS, 1, [Define if the emulator supports dirty schedulers]) - AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support]) - AC_DEFINE(ERL_DRV_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support]) - AC_MSG_RESULT(yes) - else - AC_MSG_RESULT(no) - fi - case $host_os in linux*) AC_MSG_CHECKING([whether dlopen() needs to be called before first call to dlerror()]) @@ -3823,7 +3823,7 @@ if test "$enable_dtrace_test" = "yes" ; then [$RM -f $DTRACE_2STEP_TEST dtrace -G $DTRACE_CPP $DTRACE_BITS_FLAG -Iemulator/beam -o $DTRACE_2STEP_TEST -s emulator/beam/erlang_dtrace.d conftest.$OBJEXT 2>&AS_MESSAGE_LOG_FD if test -f $DTRACE_2STEP_TEST; then - rm $DTRACE_2STEP_TEST + $RM $DTRACE_2STEP_TEST DTRACE_ENABLED_2STEP=yes fi], []) @@ -4030,7 +4030,7 @@ ssl_done=yes # Default only one run # Remove all SKIP files from previous runs for a in ssl crypto ssh; do - /bin/rm -f $ERL_TOP/lib/$a/SKIP + $RM -f $ERL_TOP/lib/$a/SKIP done SSL_DYNAMIC_ONLY=$enable_dynamic_ssl @@ -4627,7 +4627,7 @@ need_java="jinterface ic/java_src" # Remove all SKIP files from previous runs for a in $need_java ; do - /bin/rm -f $ERL_TOP/lib/$a/SKIP + $RM -f $ERL_TOP/lib/$a/SKIP done if test "X$with_javac" = "Xno"; then @@ -4678,7 +4678,7 @@ dnl this deliberately does not believe that 'gcc' is a C++ compiler AC_CHECK_TOOLS(CXX, [$CCC c++ g++ CC cxx cc++ cl], false) # Remove SKIP file from previous run -/bin/rm -f $ERL_TOP/lib/orber/SKIP +$RM -f $ERL_TOP/lib/orber/SKIP if test "$CXX" = false; then echo "No C++ compiler found" > $ERL_TOP/lib/orber/SKIP diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index c499fc8081..1bbde7f1e0 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -627,6 +627,34 @@ <p>Sets the default binary virtual heap size of processes to the size <c><![CDATA[Size]]></c>.</p> </item> + <marker id="+hmax"/> + <tag><c><![CDATA[+hmax Size]]></c></tag> + <item> + <p>Sets the default maximum heap size of processes to the size + <c><![CDATA[Size]]></c>. If <c>+hmax</c> is not given, the default is <c>0</c> + which means that no maximum heap size is used. + For more information, see the documentation of + <seealso marker="erlang#process_flag_max_heap_size"> + <c>process_flag(max_heap_size, MaxHeapSize)</c></seealso>.</p> + </item> + <marker id="+hmaxel"/> + <tag><c><![CDATA[+hmaxel true|false]]></c></tag> + <item> + <p>Sets whether to send an error logger message for processes that reach + the maximum heap size or not. If <c>+hmaxel</c> is not given, the default is <c>true</c>. + For more information, see the documentation of + <seealso marker="erlang#process_flag_max_heap_size"> + <c>process_flag(max_heap_size, MaxHeapSize)</c></seealso>.</p> + </item> + <marker id="+hmaxk"/> + <tag><c><![CDATA[+hmaxk true|false]]></c></tag> + <item> + <p>Sets whether to kill processes that reach the maximum heap size or not. If + <c>+hmaxk</c> is not given, the default is <c>true</c>. For more information, + see the documentation of + <seealso marker="erlang#process_flag_max_heap_size"> + <c>process_flag(max_heap_size, MaxHeapSize)</c></seealso>.</p> + </item> <tag><c><![CDATA[+hpds Size]]></c></tag> <item> <p>Sets the initial process dictionary size of processes to the size diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml index 7546f7ef81..33a4fee182 100644 --- a/erts/doc/src/erl_nif.xml +++ b/erts/doc/src/erl_nif.xml @@ -138,29 +138,6 @@ ok automatically unloaded when the module code that it belongs to is purged by the code server.</p> - <p><marker id="lengthy_work"/> - As mentioned in the <seealso marker="#WARNING">warning</seealso> text at - the beginning of this document it is of vital importance that a native function - return relatively quickly. It is hard to give an exact maximum amount - of time that a native function is allowed to work, but as a rule of thumb - a well-behaving native function should return to its caller before a - millisecond has passed. This can be achieved using different approaches. - If you have full control over the code to execute in the native - function, the best approach is to divide the work into multiple chunks of - work and call the native function multiple times, either directly from Erlang code - or by having a native function schedule a future NIF call via the - <seealso marker="#enif_schedule_nif"> enif_schedule_nif</seealso> function. Function - <seealso marker="#enif_consume_timeslice">enif_consume_timeslice</seealso> can be - used to help with such work division. In some cases, however, this might not - be possible, e.g. when calling third-party libraries. Then you typically want - to dispatch the work to another thread, return - from the native function, and wait for the result. The thread can send - the result back to the calling thread using message passing. Information - about thread primitives can be found below. If you have built your system - with <em>the currently experimental</em> support for dirty schedulers, - you may want to try out this functionality by dispatching the work to a - <seealso marker="#dirty_nifs">dirty NIF</seealso>, - which does not have the same duration restriction as a normal NIF.</p> </description> <section> <title>FUNCTIONALITY</title> @@ -328,38 +305,161 @@ ok </list></p> </item> - <tag>Long-running NIFs</tag> - <item><p><marker id="dirty_nifs"/>Native functions - <seealso marker="#lengthy_work"> - must normally run quickly</seealso>, as explained earlier in this document. They - generally should execute for no more than a millisecond. But not all native functions - can execute so quickly; for example, functions that encrypt large blocks of data or - perform lengthy file system operations can often run for tens of seconds or more.</p> - <p>If the functionality of a long-running NIF can be split so that its work can be - achieved through a series of shorter NIF calls, the application can either make that series - of NIF calls from the Erlang level, or it can call a NIF that first performs a chunk of the - work, then invokes the <seealso marker="#enif_schedule_nif">enif_schedule_nif</seealso> - function to schedule another NIF call to perform the next chunk. The final call scheduled - in this manner can then return the overall result. Breaking up a long-running function in - this manner enables the VM to regain control between calls to the NIFs, thereby avoiding - degraded responsiveness, scheduler load balancing problems, and other strange behaviours.</p> - <p>A NIF that cannot be split and cannot execute in a millisecond or less is called a "dirty NIF" - because it performs work that the Erlang runtime cannot handle cleanly. - <em>Note that the dirty NIF functionality described here is experimental</em> and that you have to - enable support for dirty schedulers when building OTP in order to try the functionality out. - Applications that make use of such functions must indicate to the runtime that the functions are - dirty so they can be handled specially. To schedule a dirty NIF for execution, the - appropriate flags value can be set for the NIF in its <seealso marker="#ErlNifFunc">ErlNifFunc</seealso> - entry, or the application can call <seealso marker="#enif_schedule_nif">enif_schedule_nif</seealso>, - passing to it a pointer to the dirty NIF to be executed and indicating with the <c>flags</c> - argument whether it expects the operation to be CPU-bound or I/O-bound.</p> - <note><p>Dirty NIF support is available only when the emulator is configured with dirty - schedulers enabled. This feature is currently disabled by default. To determine whether - the dirty NIF API is available, native code can check to see if the C preprocessor macro - <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined. Also, if the Erlang runtime was built - without threading support, dirty schedulers are disabled. To check at runtime for the presence - of dirty scheduler threads, code can use the <seealso marker="#enif_system_info"><c> - enif_system_info()</c></seealso> API function.</p></note> + <tag><marker id="lengthy_work"/>Long-running NIFs</tag> + + <item><p> + As mentioned in the <seealso marker="#WARNING">warning</seealso> text at + the beginning of this document it is of <em>vital importance</em> that a + native function return relatively quickly. It is hard to give an exact + maximum amount of time that a native function is allowed to work, but as a + rule of thumb a well-behaving native function should return to its caller + before a millisecond has passed. This can be achieved using different + approaches. If you have full control over the code to execute in the + native function, the best approach is to divide the work into multiple + chunks of work and call the native function multiple times. In some + cases this might however not always be possible, e.g. when calling + third-party libraries.</p> + + <p>The + <seealso marker="#enif_consume_timeslice">enif_consume_timeslice()</seealso> + function can be used to inform the runtime system about the lenght of the + NIF call. It should typically always be used unless the NIF executes very + quickly.</p> + + <p>If the NIF call is too lenghty one needs to handle this in one of the + following ways in order to avoid degraded responsiveness, scheduler load + balancing problems, and other strange behaviours:</p> + + <taglist> + <tag>Yielding NIF</tag> + <item> + <p> + If the functionality of a long-running NIF can be split so that + its work can be achieved through a series of shorter NIF calls, + the application can either make that series of NIF calls from the + Erlang level, or it can call a NIF that first performs a chunk of + the work, then invokes the + <seealso marker="#enif_schedule_nif">enif_schedule_nif</seealso> + function to schedule another NIF call to perform the next chunk. + The final call scheduled in this manner can then return the + overall result. Breaking up a long-running function in + this manner enables the VM to regain control between calls to the + NIFs. + </p> + <p> + This approach is always preferred over the other alternatives + described below. This both from a performance perspective and + a system characteristics perspective. + </p> + </item> + + <tag>Threaded NIF</tag> + <item> + <p> + This is accomplished by dispatching the work to another thread + managed by the NIF library, return from the NIF, and wait for the + result. The thread can send the result back to the Erlang + process using <seealso marker="#enif_send">enif_send</seealso>. + Information about thread primitives can be found below. + </p> + </item> + + <tag><marker id="dirty_nifs"/>Dirty NIF</tag> + <item> + + <note> + <p> + <em>The dirty NIF functionality described here + is experimental</em>. Dirty NIF support is available only when + the emulator is configured with dirty schedulers enabled. This + feature is currently disabled by default. The Erlang runtime + without SMP support do not support dirty schedulers even when + the dirty scheduler support has been enabled. To check at + runtime for the presence of dirty scheduler threads, code can + use the + <seealso marker="#enif_system_info"><c>enif_system_info()</c></seealso> + API function. + </p> + </note> + + <p> + A NIF that cannot be split and cannot execute in a millisecond or + less is called a "dirty NIF" because it performs work that the + Erlang runtime cannot handle cleanly. Applications that make use + of such functions must indicate to the runtime that the functions + are dirty so they can be handled specially. To schedule a dirty + NIF for execution, the appropriate flags value can be set for the + NIF in its <seealso marker="#ErlNifFunc"><c>ErlNifFunc</c></seealso> + entry, or the application can call + <seealso marker="#enif_schedule_nif"><c>enif_schedule_nif</c></seealso>, + passing to it a pointer to the dirty NIF to be executed and + indicating with the <c>flags</c> argument whether it expects the + operation to be CPU-bound or I/O-bound. A dirty NIF executing + on a dirty scheduler does not have the same duration restriction + as a normal NIF. + </p> + + <p> + While a process is executing a dirty NIF some operations that + communicate with it may take a very long time to complete. + Suspend, or garbage collection of a process executing a dirty + NIF cannot be done until the dirty NIF has returned, so other + processes waiting for such operations to complete might have to + wait for a very long time. Blocking multi scheduling, i.e., + calling + <seealso marker="erlang#system_flag_multi_scheduling"><c>erlang:system_flag(multi_scheduling, + block)</c></seealso>, might also take a very long time to + complete. This since all ongoing dirty operations on all + dirty schedulers need to complete before the the block + operation can complete. + </p> + + <p> + A lot of operations communicating with a process executing a + dirty NIF can, however, complete while it is executing the + dirty NIF. For example, retreiving information about it via + <c>process_info()</c>, setting its group leader, + register/unregister its name, etc. + </p> + + <p> + Termination of a process executing a dirty NIF can only be + completed up to a certain point while it is executing the + dirty NIF. All Erlang resources such as registered names, + ETS tables, etc will be released. All links and monitors + will be triggered. The actual execution of the NIF will + however <em>not</em> be stopped. The NIF can safely contiue + execution, allocate heap memory, etc, but it is of course better + to stop executing as soon as possible. The NIF can check + whether current process is alive or not using + <seealso marker="#enif_is_current_process_alive"><c>enif_is_current_process_alive</c></seealso>. + Communication using + <seealso marker="#enif_send"><c>enif_send</c></seealso>, + and <seealso marker="#enif_port_command"><c>enif_port_command</c></seealso> + will also be dropped when the sending process is not alive. + Deallocation of certain internal resources such as process + heap, and process control block will be delayed until the + dirty NIF has completed. + </p> + + <p>Currently known issues that are planned to be fixed:</p> + <list> + <item> + <p> + Since purging of a module currently might need to garbage + collect a process in order to determine if it has + references to the module, a process executing a dirty + NIF might delay purging for a very long time. Delaying + a purge operatin implies delaying <em>all</em> code + loding operations which might cause severe problems for + the system as a whole. + </p> + </item> + </list> + + </item> + </taglist> + </item> </taglist> </section> @@ -508,6 +608,10 @@ typedef struct { CPU-bound, its <c>flags</c> field should be set to <c>ERL_NIF_DIRTY_JOB_CPU_BOUND</c>, or for I/O-bound jobs, <c>ERL_NIF_DIRTY_JOB_IO_BOUND</c>.</p> + <note><p>If one of the + <c>ERL_NIF_DIRTY_JOB_*_BOUND</c> flags is set, and the runtime + system has no support for dirty schedulers, the runtime system + will refuse to load the NIF library.</p></note> </item> <tag><marker id="ErlNifBinary"/>ErlNifBinary</tag> <item> @@ -963,6 +1067,13 @@ typedef enum { <fsummary>Determine if a term is a binary</fsummary> <desc><p>Return true if <c>term</c> is a binary</p></desc> </func> + <func><name><ret>int</ret><nametext>enif_is_current_process_alive(ErlNifEnv* env)</nametext></name> + <fsummary>Determine if currently executing process is alive or not.</fsummary> + <desc><p>Return true if currently executing process is currently alive; otherwise + false.</p> + <p>This function can only be used from a NIF-calling thread, and with an + environment corresponding to currently executing processes.</p></desc> + </func> <func><name><ret>int</ret><nametext>enif_is_empty_list(ErlNifEnv* env, ERL_NIF_TERM term)</nametext></name> <fsummary>Determine if a term is an empty list</fsummary> <desc><p>Return true if <c>term</c> is an empty list.</p></desc> @@ -993,15 +1104,10 @@ typedef enum { <func><name><ret>int</ret><nametext>enif_is_on_dirty_scheduler(ErlNifEnv* env)</nametext></name> <fsummary>Check to see if executing on a dirty scheduler thread</fsummary> <desc> - <p>Check to see if the current NIF is executing on a dirty scheduler thread. If the - emulator is built with threading support, calling <c>enif_is_on_dirty_scheduler</c> - from within a dirty NIF returns true. It returns false when the calling NIF is a regular - NIF running on a normal scheduler thread, or when the emulator is built without threading - support.</p> - <note><p>This function is available only when the emulator is configured with dirty - schedulers enabled. This feature is currently disabled by default. To determine whether - the dirty NIF API is available, native code can check to see if the C preprocessor macro - <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note> + <p>Check to see if the current NIF is executing on a dirty scheduler thread. If + executing on a dirty scheduler thread true returned; otherwise false.</p> + <p>This function can only be used from a NIF-calling thread, and with an + environment corresponding to currently executing processes.</p> </desc> </func> <func><name><ret>int</ret><nametext>enif_is_pid(ErlNifEnv* env, ERL_NIF_TERM term)</nametext></name> @@ -1015,7 +1121,8 @@ typedef enum { <func><name><ret>int</ret><nametext>enif_is_port_alive(ErlNifEnv* env, ErlNifPort *port_id)</nametext></name> <fsummary>Determine if a local port is alive or not.</fsummary> <desc><p>Return true if <c>port_id</c> is currently alive.</p> - <p>This function can only be used in a from a NIF-calling thread.</p></desc> + <p>This function is only thread-safe when the emulator with SMP support is used. + It can only be used in a non-SMP emulator from a NIF-calling thread.</p></desc> </func> <func><name><ret>int</ret><nametext>enif_is_process_alive(ErlNifEnv* env, ErlNifPid *pid)</nametext></name> <fsummary>Determine if a local process is alive or not.</fsummary> @@ -1483,9 +1590,7 @@ enif_map_iterator_destroy(env, &iter); <fsummary>Send a port_command to to_port</fsummary> <desc> <p>This function works the same as <seealso marker="erlang#port_command-2">erlang:port_command/2</seealso> - except that it is always completely asynchronous. This call may return false - if it detects that the port is already dead, otherwise it will return true. - </p> + except that it is always completely asynchronous.</p> <taglist> <tag><c>env</c></tag> <item>The environment of the calling process. May not be NULL.</item> @@ -1504,7 +1609,10 @@ enif_map_iterator_destroy(env, &iter); calls to <c>enif_alloc_env</c>, <c>enif_make_copy</c>, <c>enif_port_command</c> and <c>enif_free_env</c> into one call. This optimization is only usefull when a majority of the terms are to be copied from <c>env</c> to the <c>msg_env</c>.</p> - <p>The call may return false if it detects that the command failed for some reason. Otherwise true is returned.</p> + <p>This function return true if the command was successfully sent; otherwise, + false. The call may return false if it detects that the command failed for some + reason. For example, <c>*to_port</c> does not refer to a local port, if currently + executing process, i.e. the sender, is not alive, or if <c>msg</c> is invalid.</p> <p>See also: <seealso marker="#enif_get_local_port"><c>enif_get_local_port</c></seealso>.</p> </desc> </func> @@ -1635,7 +1743,9 @@ enif_map_iterator_destroy(env, &iter); <tag><c>msg</c></tag> <item>The message term to send.</item> </taglist> - <p>Return true on success, or false if <c>*to_pid</c> does not refer to an alive local process.</p> + <p>Return true if the message was successfully sent; otherwise, false. The send + operation will fail if <c>*to_pid</c> does not refer to an alive local process, + or if currently executing process, i.e. the sender, is not alive.</p> <p>The message environment <c>msg_env</c> with all its terms (including <c>msg</c>) will be invalidated by a successful call to <c>enif_send</c>. The environment should either be freed with <seealso marker="#enif_free_env">enif_free_env</seealso> @@ -1653,6 +1763,15 @@ enif_map_iterator_destroy(env, &iter); <desc><p>Get the byte size of a resource object <c>obj</c> obtained by <seealso marker="#enif_alloc_resource">enif_alloc_resource</seealso>.</p></desc> </func> + + <func><name><ret>int</ret><nametext>enif_snprintf(char *str, size_t size, const char *format, ...)</nametext></name> + <fsummary>Format strings and Erlang terms</fsummary> + <desc> + <p>Similar to <c>snprintf</c> but this format string also accepts <c>"%T"</c> which formats Erlang terms. + </p> + </desc> + </func> + <func> <name><ret>void</ret><nametext>enif_system_info(ErlNifSysInfo *sys_info_ptr, size_t size)</nametext></name> <fsummary>Get information about the Erlang runtime system</fsummary> diff --git a/erts/doc/src/erl_tracer.xml b/erts/doc/src/erl_tracer.xml index 2075b962d8..7fd5512b33 100644 --- a/erts/doc/src/erl_tracer.xml +++ b/erts/doc/src/erl_tracer.xml @@ -90,7 +90,7 @@ <item>If not set to <c>true</c>, the tracer has been requested to include the output of a match specification that was run.</item> <tag><c>scheduler_id</c></tag> - <item>Set to a number of the scheduler id is to be included by the tracer. + <item>Set to a number if the scheduler id is to be included by the tracer. Otherwise it is set to <c>undefined</c>.</item> </taglist> </p> @@ -155,14 +155,13 @@ overhead associated with tracing. If <c>trace</c> is returned the necessary trace data will be created and the trace call-back of the tracer will be called. If <c>discard</c> is returned, this trace call - will be discarded and no call to trace will be done. If - <c>remove</c> is returned, the VM will attempt to remove this tracer - from the tracee, together with any trace flags set on the tracee. + will be discarded and no call to trace will be done. </p> <p><c>trace_status</c> is a special type of <c>TraceTag</c> which is used to check if the tracer should still be active. It is called in multiple scenarios, but most significantly it is used when tracing is started - using this tracer.</p> + using this tracer. If <c>remove</c> is returned when the <c>trace_status</c> + is checked, the tracer will be removed from the tracee.</p> <p>This function may be called multiple times per tracepoint, so it is important that it is both fast and side effect free.</p> </desc> @@ -617,7 +616,7 @@ static int upgrade(ErlNifEnv* env, void** priv_data, void** old_priv_data, } /* - * argv[0]: Trace Tag + * argv[0]: TraceTag * argv[1]: TracerState * argv[2]: Tracee */ @@ -626,8 +625,11 @@ static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) ErlNifPid to_pid; if (enif_get_local_pid(env, argv[1], &to_pid)) if (!enif_is_process_alive(env, &to_pid)) - /* tracer is dead so we should remove this tracepoint */ - return enif_make_atom(env, "remove"); + if (enif_is_identical(enif_make_atom(env, "trace_status"), argv[0])) + /* tracer is dead so we should remove this tracepoint */ + return enif_make_atom(env, "remove"); + else + return enif_make_atom(env, "discard"); /* Only generate trace for when tracer != tracee */ if (enif_is_identical(argv[1], argv[2])) @@ -645,7 +647,7 @@ static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) } /* - * argv[0]: Trace Tag, should only be 'send' + * argv[0]: TraceTag, should only be 'send' * argv[1]: TracerState, process to send {argv[2], argv[4]} to * argv[2]: Tracee * argv[3]: Message, ignored diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 3276bc34b0..e0723127f2 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -4322,6 +4322,7 @@ os_prompt% </pre> </desc> </func> + <marker id="process_flag_min_heap_size"/> <func> <name name="process_flag" arity="2" clause_i="3"/> <fsummary>Sets process flag <c>min_heap_size</c> for the calling process.</fsummary> @@ -4340,9 +4341,95 @@ os_prompt% </pre> <p>Returns the old value of the flag.</p> </desc> </func> - <marker id="process_flag_message_queue_data"/> + <marker id="process_flag_max_heap_size"/> <func> <name name="process_flag" arity="2" clause_i="5"/> + <type name="max_heap_size"/> + <fsummary>Sets process flag <c>max_heap_size</c> for the calling process.</fsummary> + <desc> + <p> + This flag sets the maximum heap size for the calling process. + If <c><anno>MaxHeapSize</anno></c> is an integer, the system default + values for <c>kill</c> and <c>error_logger</c> are used. + <taglist> + <tag><c>size</c></tag> + <item> + <p> + The maximum size in words of the process. If set to zero, the + heap size limit is disabled. Badarg will be thrown if the value is + smaller than + <seealso marker="#process_flag_min_heap_size"><c>min_heap_size</c></seealso>. + The size check is only done when a garbage collection is triggered. + </p> + <p> + <c>size</c> is the entire heap of the process when garbage collection + is triggered, this includes all generational heaps, the process stack, + any <seealso marker="#process_flag_message_queue_data"> + messages that are considered to be part of the heap</seealso> and any + extra memory that the garbage collector needs during collection. + </p> + <p> + <c>size</c> is the same as can be retrieved using + <seealso marker="#process_info_total_heap_size"> + <c>erlang:process_info(Pid, total_heap_size)</c></seealso>, + or by adding <c>heap_block_size</c>, <c>old_heap_block_size</c> + and <c>mbuf_size</c> from <seealso marker="#process_info_garbage_collection_info"> + <c>erlang:process_info(Pid, garbage_collection_info)</c></seealso>. + </p> + </item> + <tag><c>kill</c></tag> + <item> + <p> + When set to <c>true</c> the runtime system will send an + untrappable exit signal with reason <c>kill</c> to the process + if the maximum heap size is reached. The garbage collection + that triggered the <c>kill</c> will not be completed, instead the + process will exit as soon as is possible. When set to <c>false</c> + no exit signal will be sent to the process, instead it will + continue executing. + </p> + <p> + If <c>kill</c> is not defined in the map + the system default will be used. The default system default + is <c>true</c>. It can be changed by either the erl + <seealso marker="erl#+hmaxk">+hmaxk</seealso> option, + or <seealso marker="#system_flag_max_heap_size"><c> + erlang:system_flag(max_heap_size, MaxHeapSize)</c></seealso>. + </p> + </item> + <tag><c>error_logger</c></tag> + <item> + <p> + When set to <c>true</c> the runtime system will send a + message to the current <seealso marker="kernel:error_logger"><c>error_logger</c></seealso> + containing details about the process when the maximum + heap size is reached. One <c>error_logger</c> report will + be sent each time the limit is reached. + </p> + <p> + If <c>error_logger</c> is not defined in the map the system + default will be used. The default system default is <c>true</c>. + It can be changed by either the erl <seealso marker="erl#+hmaxel">+hmaxel</seealso> + option, or <seealso marker="#system_flag_max_heap_size"><c> + erlang:system_flag(max_heap_size, MaxHeapSize)</c></seealso>. + </p> + </item> + <p> + The heap size of a process is quite hard to predict, especially the + amount of memory that is used during the garbage collection. When + contemplating using this option, it is recommended to first run + it in production with <c>kill</c> set to <c>false</c> and inspect + the <c>error_logger</c> reports to see what the normal peak sizes + of the processes in the system is and then tune the value + accordingly. + </p> + </taglist> + </p> + </desc> + </func> + <marker id="process_flag_message_queue_data"/> + <func> + <name name="process_flag" arity="2" clause_i="6"/> <fsummary>Set process flag <c>message_queue_data</c> for the calling process</fsummary> <type name="message_queue_data"/> <desc> @@ -4392,7 +4479,7 @@ os_prompt% </pre> </desc> </func> <func> - <name name="process_flag" arity="2" clause_i="6"/> + <name name="process_flag" arity="2" clause_i="7"/> <fsummary>Sets process flag <c>priority</c> for the calling process.</fsummary> <type name="priority_level"/> <desc> @@ -4466,7 +4553,7 @@ os_prompt% </pre> </func> <func> - <name name="process_flag" arity="2" clause_i="7"/> + <name name="process_flag" arity="2" clause_i="8"/> <fsummary>Sets process flag <c>save_calls</c> for the calling process.</fsummary> <desc> <p><c><anno>N</anno></c> must be an integer in the interval 0..10000. @@ -4497,7 +4584,7 @@ os_prompt% </pre> </func> <func> - <name name="process_flag" arity="2" clause_i="8"/> + <name name="process_flag" arity="2" clause_i="9"/> <fsummary>Sets process flag <c>sensitive</c> for the calling process.</fsummary> <desc> <p>Sets or clears flag <c>sensitive</c> for the current process. @@ -4551,6 +4638,7 @@ os_prompt% </pre> <type name="process_info_result_item"/> <type name="priority_level"/> <type name="stack_item"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <desc> <p>Returns a list containing <c><anno>InfoTuple</anno></c>s with @@ -4604,6 +4692,7 @@ os_prompt% </pre> <type name="process_info_result_item"/> <type name="stack_item"/> <type name="priority_level"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <desc> <p>Returns information about the process identified by @@ -4696,6 +4785,7 @@ os_prompt% </pre> The content of <c><anno>GCInfo</anno></c> can be changed without prior notice.</p> </item> + <marker id="process_info_garbage_collection_info"/> <tag><c>{garbage_collection_info, <anno>GCInfo</anno>}</c></tag> <item> <p><c><anno>GCInfo</anno></c> is a list containing miscellaneous @@ -4875,6 +4965,7 @@ os_prompt% </pre> total suspend count on <c><anno>Suspendee</anno></c>, only the parts contributed by <c><anno>Pid</anno></c>.</p> </item> + <marker id="process_info_total_heap_size"/> <tag><c>{total_heap_size, <anno>Size</anno>}</c></tag> <item> <p><c><anno>Size</anno></c> is the total size, in words, of all heap @@ -5569,6 +5660,7 @@ true</pre> <name name="spawn_opt" arity="2"/> <fsummary>Creates a new process with a fun as entry point.</fsummary> <type name="priority_level"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <type name="spawn_opt_option" /> <desc> @@ -5586,6 +5678,7 @@ true</pre> <name name="spawn_opt" arity="3"/> <fsummary>Creates a new process with a fun as entry point on a given node.</fsummary> <type name="priority_level"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <type name="spawn_opt_option" /> <desc> @@ -5602,6 +5695,7 @@ true</pre> <name name="spawn_opt" arity="4"/> <fsummary>Creates a new process with a function as entry point.</fsummary> <type name="priority_level"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <type name="spawn_opt_option" /> <desc> @@ -5705,6 +5799,16 @@ true</pre> fine-tuning an application and to measure the execution time with various <c><anno>VSize</anno></c> values.</p> </item> + <tag><c>{max_heap_size, <anno>Size</anno>}</c></tag> + <item> + <p>Sets the <c>max_heap_size</c> process flag. The default + <c>max_heap_size</c> is determined by the + <seealso marker="erl#+hmax"><c>+hmax</c></seealso> <c>erl</c> + command line argument. For more information, see the + documentation of + <seealso marker="#process_flag_max_heap_size"><c>process_flag(max_heap_size, + <anno>Size</anno>)</c></seealso>.</p> + </item> <tag><c>{message_queue_data, <anno>MQD</anno>}</c></tag> <item> <p>Sets the state of the <c>message_queue_data</c> process @@ -5725,6 +5829,7 @@ true</pre> <name name="spawn_opt" arity="5"/> <fsummary>Creates a new process with a function as entry point on a given node.</fsummary> <type name="priority_level"/> + <type name="max_heap_size" /> <type name="message_queue_data" /> <type name="spawn_opt_option" /> <desc> @@ -6506,8 +6611,25 @@ ok </desc> </func> + <marker id="system_flag_max_heap_size"></marker> <func> <name name="system_flag" arity="2" clause_i="8"/> + <type name="max_heap_size"/> + <fsummary>Sets system flag <c>max_heap_size</c></fsummary> + <desc> + <p> + Sets the default maximum heap size settings for processes. + The size is given in words. The new <c>max_heap_size</c> + effects only processes spawned efter the change has been made. + <c>max_heap_size</c> can be set for individual processes using + <seealso marker="#spawn_opt/4">spawn_opt/N</seealso> or + <seealso marker="#process_flag_message_queue_data">process_flag/2</seealso>.</p> + <p>Returns the old value of the flag.</p> + </desc> + </func> + + <func> + <name name="system_flag" arity="2" clause_i="9"/> <fsummary>Sets system flag <c>multi_scheduling</c>.</fsummary> <desc> <p><marker id="system_flag_multi_scheduling"></marker> @@ -6557,7 +6679,7 @@ ok </func> <func> - <name name="system_flag" arity="2" clause_i="9"/> + <name name="system_flag" arity="2" clause_i="10"/> <fsummary>Sets system flag <c>scheduler_bind_type</c>.</fsummary> <type name="scheduler_bind_type"/> <desc> @@ -6675,7 +6797,7 @@ ok </func> <func> - <name name="system_flag" arity="2" clause_i="10"/> + <name name="system_flag" arity="2" clause_i="11"/> <fsummary>Sets system flag <c>scheduler_wall_time</c>.</fsummary> <desc><p><marker id="system_flag_scheduler_wall_time"></marker> Turns on or off scheduler wall time measurements.</p> @@ -6685,7 +6807,7 @@ ok </func> <func> - <name name="system_flag" arity="2" clause_i="11"/> + <name name="system_flag" arity="2" clause_i="12"/> <fsummary>Sets system flag <c>schedulers_online</c>.</fsummary> <desc> <p><marker id="system_flag_schedulers_online"></marker> @@ -6710,7 +6832,7 @@ ok </func> <func> - <name name="system_flag" arity="2" clause_i="12"/> + <name name="system_flag" arity="2" clause_i="13"/> <fsummary>Sets system flag <c>trace_control_word</c>.</fsummary> <desc> <p>Sets the value of the node trace control word to @@ -6724,7 +6846,7 @@ ok </func> <func> - <name name="system_flag" arity="2" clause_i="12"/> + <name name="system_flag" arity="2" clause_i="14"/> <fsummary>Finalize the Time Offset</fsummary> <desc> <p><marker id="system_flag_time_offset"></marker> @@ -6841,11 +6963,7 @@ ok As from <c>ERTS</c> 5.6.1, the return value is a list of <c>{instance, InstanceNo, InstanceInfo}</c> tuples, where <c>InstanceInfo</c> contains information about - a specific instance of the allocator. As from - <c>ERTS</c> 5.10.4, the returned list when calling - <c>erlang:system_info({allocator, mseg_alloc})</c> also - includes an <c>{erts_mmap, _}</c> tuple as one element - in the list. If <c><anno>Alloc</anno></c> is not a + a specific instance of the allocator. If <c><anno>Alloc</anno></c> is not a recognized allocator, <c>undefined</c> is returned. If <c><anno>Alloc</anno></c> is disabled, <c>false</c> is returned.</p> @@ -6857,7 +6975,13 @@ ok briefly documented.</p> <p>The recognized allocators are listed in <seealso marker="erts:erts_alloc">erts_alloc(3)</seealso>. - After reading the <c>erts_alloc(3)</c> documentation, + Information about super carriers can be obtained from + <c>ERTS</c> 8.0 with <c>{allocator, erts_mmap}</c> or from + <c>ERTS</c> 5.10.4, the returned list when calling with + <c>{allocator, mseg_alloc}</c> also includes an + <c>{erts_mmap, _}</c> tuple as one element in the list.</p> + + <p>After reading the <c>erts_alloc(3)</c> documentation, the returned information more or less speaks for itself, but it can be worth explaining some things. Call counts are presented by two @@ -6989,6 +7113,81 @@ ok </func> <func> + <name name="system_info" arity="1" clause_i="27"/> + <name name="system_info" arity="1" clause_i="28"/> + <name name="system_info" arity="1" clause_i="36"/> + <name name="system_info" arity="1" clause_i="37"/> + <name name="system_info" arity="1" clause_i="38"/> + <name name="system_info" arity="1" clause_i="39"/> + <type name="message_queue_data"/> + <type name="max_heap_size"/> + <fsummary>Information about the default process heap settings.</fsummary> + <desc> + <taglist> + <tag><c>fullsweep_after</c></tag> + <item> + <p>Returns <c>{fullsweep_after, integer() >= 0}</c>, which is + the <c>fullsweep_after</c> garbage collection setting used + by default. For more information, see + <c>garbage_collection</c> described in the following.</p> + </item> + <tag><c>garbage_collection</c></tag> + <item> + <p>Returns a list describing the default garbage collection + settings. A process spawned on the local node by a + <c>spawn</c> or <c>spawn_link</c> uses these + garbage collection settings. The default settings can be + changed by using + <seealso marker="#system_flag/2">system_flag/2</seealso>. + <seealso marker="#spawn_opt/4">spawn_opt/4</seealso> + can spawn a process that does not use the default + settings.</p> + </item> + <tag><c>max_heap_size</c></tag> + <item> + <p>Returns <c>{max_heap_size, <anno>MaxHeapSize</anno>}</c>, + where <c><anno>MaxHeapSize</anno></c> is the current + system-wide max heap size settings for spawned processes. + This setting can be set using the <c>erl</c> command line + flags <seealso marker="erl#+hmax"><c>+hmax</c></seealso>, + <seealso marker="erl#+hmaxk"><c>+hmaxk</c></seealso> and + <seealso marker="erl#+hmaxel"><c>+hmaxel</c></seealso>. It can + also be changed at run-time using + <seealso marker="#system_flag_max_heap_size"> + <c>erlang:system_flag(max_heap_size, MaxHeapSize)</c></seealso>. + For more details about the <c>max_heap_size</c> process flag + see <seealso marker="#process_flag_max_heap_size"> + <c>process_flag(max_heap_size, MaxHeapSize)</c></seealso>. + </p> + </item> + <tag><c>min_heap_size</c></tag> + <item> + <p>Returns <c>{min_heap_size, <anno>MinHeapSize</anno>}</c>, + where <c><anno>MinHeapSize</anno></c> is the current + system-wide minimum heap size for spawned processes.</p> + </item> + <tag><marker id="system_info_message_queue_data"><c>message_queue_data</c></marker></tag> + <item> + <p>Returns the default value of the <c>message_queue_data</c> + process flag which is either <c>off_heap</c>, <c>on_heap</c>, or <c>mixed</c>. + This default is set by the <c>erl</c> command line argument + <seealso marker="erl#+hmqd"><c>+hmqd</c></seealso>. For more information on the + <c>message_queue_data</c> process flag, see documentation of + <seealso marker="#process_flag_message_queue_data"><c>process_flag(message_queue_data, + MQD)</c></seealso>.</p> + </item> + <tag><c>min_bin_vheap_size</c></tag> + <item> + <p>Returns <c>{min_bin_vheap_size, + <anno>MinBinVHeapSize</anno>}</c>, where + <c><anno>MinBinVHeapSize</anno></c> is the current system-wide + minimum binary virtual heap size for spawned processes.</p> + </item> + </taglist> + </desc> + </func> + + <func> <name name="system_info" arity="1" clause_i="6"/> <name name="system_info" arity="1" clause_i="7"/> <name name="system_info" arity="1" clause_i="8"/> @@ -7008,8 +7207,6 @@ ok <name name="system_info" arity="1" clause_i="24"/> <name name="system_info" arity="1" clause_i="25"/> <name name="system_info" arity="1" clause_i="26"/> - <name name="system_info" arity="1" clause_i="27"/> - <name name="system_info" arity="1" clause_i="28"/> <name name="system_info" arity="1" clause_i="29"/> <name name="system_info" arity="1" clause_i="30"/> <name name="system_info" arity="1" clause_i="31"/> @@ -7017,10 +7214,6 @@ ok <name name="system_info" arity="1" clause_i="33"/> <name name="system_info" arity="1" clause_i="34"/> <name name="system_info" arity="1" clause_i="35"/> - <name name="system_info" arity="1" clause_i="36"/> - <name name="system_info" arity="1" clause_i="37"/> - <name name="system_info" arity="1" clause_i="38"/> - <name name="system_info" arity="1" clause_i="39"/> <name name="system_info" arity="1" clause_i="40"/> <name name="system_info" arity="1" clause_i="41"/> <name name="system_info" arity="1" clause_i="42"/> @@ -7050,6 +7243,7 @@ ok <name name="system_info" arity="1" clause_i="66"/> <name name="system_info" arity="1" clause_i="67"/> <name name="system_info" arity="1" clause_i="68"/> + <name name="system_info" arity="1" clause_i="69"/> <fsummary>Information about the system.</fsummary> <desc> <p>Returns various information about the current system @@ -7286,25 +7480,6 @@ ok <c>ERL_MAX_ETS_TABLES</c> before starting the Erlang runtime system.</p> </item> - <tag><c>fullsweep_after</c></tag> - <item> - <p>Returns <c>{fullsweep_after, integer() >= 0}</c>, which is - the <c>fullsweep_after</c> garbage collection setting used - by default. For more information, see - <c>garbage_collection</c> described in the following.</p> - </item> - <tag><c>garbage_collection</c></tag> - <item> - <p>Returns a list describing the default garbage collection - settings. A process spawned on the local node by a - <c>spawn</c> or <c>spawn_link</c> uses these - garbage collection settings. The default settings can be - changed by using - <seealso marker="#system_flag/2">system_flag/2</seealso>. - <seealso marker="#spawn_opt/4">spawn_opt/4</seealso> - can spawn a process that does not use the default - settings.</p> - </item> <tag><c>heap_sizes</c></tag> <item> <p>Returns a list of integers representing valid heap sizes @@ -7379,29 +7554,6 @@ ok <item> <p>Returns a string containing the Erlang machine name.</p> </item> - <tag><c>min_heap_size</c></tag> - <item> - <p>Returns <c>{min_heap_size, <anno>MinHeapSize</anno>}</c>, - where <c><anno>MinHeapSize</anno></c> is the current - system-wide minimum heap size for spawned processes.</p> - </item> - <tag><marker id="system_info_message_queue_data"><c>message_queue_data</c></marker></tag> - <item> - <p>Returns the default value of the <c>message_queue_data</c> - process flag which is either <c>off_heap</c>, <c>on_heap</c>, or <c>mixed</c>. - This default is set by the <c>erl</c> command line argument - <seealso marker="erl#+hmqd"><c>+hmqd</c></seealso>. For more information on the - <c>message_queue_data</c> process flag, see documentation of - <seealso marker="#process_flag_message_queue_data"><c>process_flag(message_queue_data, - MQD)</c></seealso>.</p> - </item> - <tag><c>min_bin_vheap_size</c></tag> - <item> - <p>Returns <c>{min_bin_vheap_size, - <anno>MinBinVHeapSize</anno>}</c>, where - <c><anno>MinBinVHeapSize</anno></c> is the current system-wide - minimum binary virtual heap size for spawned processes.</p> - </item> <tag><c>modified_timing_level</c></tag> <item> <p>Returns the modified timing-level (an integer) if @@ -8026,12 +8178,13 @@ ok <c>GcPid</c> and <c>Info</c> are the same as for <c>long_gc</c> earlier, except that the tuple tagged with <c>timeout</c> is not present.</p> - <p>As of <c>ERTS</c> 5.6, the monitor message is sent - if the sum of the sizes of all memory blocks allocated - for all heap generations is equal to or higher than <c>Size</c>. - Previously the monitor message was sent if the memory block - allocated for the youngest generation was equal to or higher - than <c>Size</c>.</p> + <p>The monitor message is sent if the sum of the sizes of + all memory blocks allocated for all heap generations after + a garbage collection is equal to or higher than <c>Size</c>.</p> + <p>When a process is killed by <seealso marker="#process_flag_max_heap_size"> + <c>max_heap_size</c></seealso>, it is killed before the + garbage collection is complete and thus no large heap message + will be sent.</p> </item> <tag><c>busy_port</c></tag> <item> @@ -8558,7 +8711,9 @@ timestamp() -> <tag><c>garbage_collection</c></tag> <item> <p>Traces garbage collections of processes.</p> - <p>Message tags: <c><seealso marker="#trace_3_trace_messages_gc_minor_start">gc_minor_start</seealso></c> and <c><seealso marker="#trace_3_trace_messages_gc_minor_end">gc_minor_end</seealso></c>.</p> + <p>Message tags: <c><seealso marker="#trace_3_trace_messages_gc_minor_start">gc_minor_start</seealso></c>, + <c><seealso marker="#trace_3_trace_messages_gc_max_heap_size">gc_max_heap_size</seealso></c> and + <c><seealso marker="#trace_3_trace_messages_gc_minor_end">gc_minor_end</seealso></c>.</p> </item> <tag><c>timestamp</c></tag> <item> @@ -8927,6 +9082,19 @@ timestamp() -> <p>All sizes are in words.</p> </item> <tag> + <marker id="trace_3_trace_messages_gc_max_heap_size"></marker> + <c>{trace, Pid, gc_max_heap_size, Info}</c> + </tag> + <item> + <p> + Sent when the <seealso marker="#process_flag_max_heap_size"><c>max_heap_size</c></seealso> + is reached during garbage collection. <c>Info</c> contains the + same kind of list as in message <c>gc_start</c>, + but the sizes reflect the sizes that triggered max_heap_size to + be reached. + </p> + </item> + <tag> <marker id="trace_3_trace_messages_gc_minor_end"></marker> <c>{trace, Pid, gc_minor_end, Info}</c> </tag> @@ -9034,16 +9202,16 @@ timestamp() -> <type name="trace_info_flag"/> <type name="trace_match_spec"/> <desc> - <p>Returns trace information about a port, process or function.</p> - <p>To get information about a port or process, - <c><anno>PidPortOrFunc</anno></c> is to + <p>Returns trace information about a port, process, function or event.</p> + <p><em>To get information about a port or process</em>, + <c><anno>PidPortFuncEvent</anno></c> is to be a process identifier (pid), port identifier or one of the atoms <c>new</c>, <c>new_processes</c>, <c>new_ports</c>. The atom <c>new</c> or <c>new_processes</c> means that the default trace state for processes to be created is returned. The atom <c>new_ports</c> means that the default trace state for ports to be created is returned. </p> - <p>The following <c>Item</c>s are valid:</p> + <p>The following <c>Item</c>s are valid for ports and processes:</p> <taglist> <tag><c>flags</c></tag> <item> @@ -9067,12 +9235,15 @@ timestamp() -> value is <c>[]</c>.</p> </item> </taglist> - <p>To get information about a function, <c><anno>PidPortOrFunc</anno></c> is to + <p><em>To get information about a function</em>, <c><anno>PidPortFuncEvent</anno></c> is to be the three-element tuple <c>{Module, Function, Arity}</c> or the atom <c>on_load</c>. No wild cards are allowed. Returns <c>undefined</c> if the function does not exist, or - <c>false</c> if the function is not traced.</p> - <p>The following <c>Item</c>s are valid::</p> + <c>false</c> if the function is not traced. If <c><anno>PidPortFuncEvent</anno></c> + is <c>on_load</c>, the information returned refers to + the default value for code that will be loaded.</p> + + <p>The following <c>Item</c>s are valid for functions:</p> <taglist> <tag><c>traced</c></tag> <item> @@ -9131,39 +9302,177 @@ timestamp() -> is active for this function.</p> </item> </taglist> + <p><em>To get information about an event</em>, <c><anno>PidPortFuncEvent</anno></c> is to + be one of the atoms <c>send</c> or <c>'receive'</c>.</p> + <p>The only valid <c>Item</c> for events is:</p> + <taglist> + <tag><c>match_spec</c></tag> + <item> + <p>Returns the match specification for this event, if it + has one, or <c>true</c> if no match specification has been + set.</p> + </item> + </taglist> <p>The return value is <c>{<anno>Item</anno>, Value}</c>, where <c>Value</c> is the requested information as described earlier. If a pid for a dead process was given, or the name of a non-existing function, <c>Value</c> is <c>undefined</c>.</p> - <p>If <c><anno>PidPortOrFunc</anno></c> is <c>on_load</c>, the information - returned refers to the default value for code that will be - loaded.</p> </desc> </func> <func> <name name="trace_pattern" arity="2" clause_i="1"/> - <fsummary>Sets trace patterns for global call tracing.</fsummary> + <fsummary>Sets trace patterns for call, send or 'receive' tracing.</fsummary> <type name="trace_pattern_mfa"/> <type name="trace_match_spec"/> <desc> <p>The same as - <seealso marker="#trace_pattern/3">erlang:trace_pattern(MFA, MatchSpec, [])</seealso>, + <seealso marker="#trace_pattern/3">erlang:trace_pattern(Event, MatchSpec, [])</seealso>, retained for backward compatibility.</p> </desc> </func> <func> - <name name="trace_pattern" arity="3"/> + <name name="trace_pattern" arity="3" clause_i="1"/> + <fsummary>Sets trace pattern for message sending.</fsummary> + <type name="trace_match_spec"/> + <desc> + <p>Sets trace pattern for <em>message sending</em>. + Must be combined with + <seealso marker="#trace/3">erlang:trace/3</seealso> + to set the <c>send</c> trace flag for one or more processes. + By default all messages, sent from <c>send</c> traced processes, + are traced. Use <c>erlang:trace_pattern/3</c> to limit + traced send events based on the message content, the sender + and/or the receiver.</p> + <p>Argument <c><anno>MatchSpec</anno></c> can take the + following forms:</p> + <taglist> + <tag><c><anno>MatchSpecList</anno></c></tag> + <item> + <p>A list of match specifications. The matching is done + on the list <c>[Receiver, Msg]</c>. <c>Receiver</c> + is the process or port identity of the receiver and + <c>Msg</c> is the message term. The pid of the sending + process can be accessed with the guard function + <c>self/0</c>. An empty list is the same as <c>true</c>. + See the users guide section + <seealso marker="erts:match_spec">Match Specifications in Erlang</seealso> + for more information.</p> + </item> + <tag><c>true</c></tag> + <item> + <p>Enables tracing for all sent messages (from <c>send</c> + traced processes). Any match specification is + removed. <em>This is the default</em>.</p> + </item> + <tag><c>false</c></tag> + <item> + <p>Disables tracing for all sent messages. + Any match specification is removed.</p> + </item> + </taglist> + <p>Argument <c><anno>FlagList</anno></c> must be <c>[]</c> + for send tracing.</p> + <p>The return value is always <c>1</c>.</p> + <p>Example; only trace messages to a specific process <c>Pid</c>:</p> + <pre> +> <input>erlang:trace_pattern(send, [{[Pid, '_'],[],[]}], []).</input> +1</pre> + <p>Only trace messages matching <c>{reply, _}</c>:</p> + <pre> +> <input>erlang:trace_pattern(send, [{['_', {reply,'_'}],[],[]}], []).</input> +1</pre> + <p>Only trace messages sent to the sender itself:</p> + <pre> +> <input>erlang:trace_pattern(send, [{['$1', '_'],[{'=:=','$1',{self}}],[]}], []).</input> +1</pre> + <p>Only trace messages sent to other nodes:</p> + <pre> +> <input>erlang:trace_pattern(send, [{['$1', '_'],[{'=/=',{node,'$1'},{node}}],[]}], []).</input> +1</pre> + <note><p>A match specification for <c>send</c> trace can use + all guard and body functions except <c>caller</c>.</p></note> + </desc> + </func> + + <func> + <name name="trace_pattern" arity="3" clause_i="2"/> + <fsummary>Sets trace pattern for tracing of message receiving.</fsummary> + <type name="trace_match_spec"/> + <desc> + <p></p> + <p>Sets trace pattern for <em>message receiving</em>. + Must be combined with + <seealso marker="#trace/3">erlang:trace/3</seealso> + to set the <c>'receive'</c> trace flag for one or more processes. + By default all messages, received by <c>'receive'</c> traced processes, + are traced. Use <c>erlang:trace_pattern/3</c> to limit + traced receive events based on the message content, the sender + and/or the receiver.</p> + <p>Argument <c><anno>MatchSpec</anno></c> can take the + following forms:</p> + <taglist> + <tag><c><anno>MatchSpecList</anno></c></tag> + <item> + <p>A list of match specifications. The matching is done + on the list <c>[Node, Sender, Msg]</c>. <c>Node</c> + is the node name of the sender. <c>Sender</c> is the + process or port identity of the sender, or the atom + <c>undefined</c> if the sender is not known (which may + be the case for remote senders). <c>Msg</c> is the + message term. The pid of the receiving process can be + accessed with the guard function <c>self/0</c>. An empty + list is the same as <c>true</c>. See the users guide section + <seealso marker="erts:match_spec">Match Specifications in Erlang</seealso> + for more information.</p> + </item> + <tag><c>true</c></tag> + <item> + <p>Enables tracing for all received messages (to <c>'receive'</c> + traced processes). Any match specification is + removed. <em>This is the default</em>.</p> + </item> + <tag><c>false</c></tag> + <item> + <p>Disables tracing for all received messages. + Any match specification is removed.</p> + </item> + </taglist> + <p>Argument <c><anno>FlagList</anno></c> must be <c>[]</c> + for receive tracing.</p> + <p>The return value is always <c>1</c>.</p> + <p>Example; only trace messages from a specific process <c>Pid</c>:</p> + <pre> +> <input>erlang:trace_pattern('receive', [{['_',Pid, '_'],[],[]}], []).</input> +1</pre> + <p>Only trace messages matching <c>{reply, _}</c>:</p> + <pre> +> <input>erlang:trace_pattern('receive', [{['_','_', {reply,'_'}],[],[]}], []).</input> +1</pre> + <p>Only trace messages from other nodes:</p> + <pre> +> <input>erlang:trace_pattern('receive', [{['$1', '_', '_'],[{'=/=','$1',{node}}],[]}], []).</input> +1</pre> + <note><p>A match specification for <c>'receive'</c> trace can + use all guard and body functions except <c>caller, + is_seq_trace, get_seq_token, set_seq_token, enable_trace, + disable_trace, trace, silent</c> and <c>process_dump</c>.</p></note> + </desc> + </func> + + <func> + <name name="trace_pattern" arity="3" clause_i="3"/> <fsummary>Sets trace patterns for tracing of function calls.</fsummary> <type name="trace_pattern_mfa"/> <type name="trace_match_spec"/> <type name="trace_pattern_flag"/> <desc> - <p>Enables or disables call tracing for - one or more functions. Must be combined with + <p>Enables or disables <em>call tracing</em> for one or more functions. + Must be combined with <seealso marker="#trace/3">erlang:trace/3</seealso> - to set the <c>call</c> trace flag for one or more processes.</p> + to set the <c>call</c> trace flag + for one or more processes.</p> <p>Conceptually, call tracing works as follows. Inside the Erlang Virtual Machine, a set of processes and a set of functions are to be traced. If a traced process @@ -9218,7 +9527,8 @@ timestamp() -> </item> <tag><c>true</c></tag> <item> - <p>Enables tracing for the matching functions.</p> + <p>Enables tracing for the matching functions. + Any match specification is removed.</p> </item> <tag><c><anno>MatchSpecList</anno></c></tag> <item> diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index fb486c917f..2212aed5e0 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -50,6 +50,8 @@ LDFLAGS=@LDFLAGS@ ARFLAGS=rc OMIT_OMIT_FP=no +DIRTY_SCHEDULER_SUPPORT=@DIRTY_SCHEDULER_SUPPORT@ + ifeq ($(TYPE),debug) PURIFY = TYPEMARKER = .debug @@ -174,6 +176,10 @@ FLAVOR_MARKER=.smp FLAVOR_FLAGS=-DERTS_SMP ENABLE_ALLOC_TYPE_VARS += smp nofrag M4FLAGS += -DERTS_SMP=1 +ifeq ($(DIRTY_SCHEDULER_SUPPORT),yes) +THR_DEFS += -DERTS_DIRTY_SCHEDULERS +endif + else # If flavor isn't one of the above, it *is* plain flavor... @@ -182,7 +188,6 @@ FLAVOR_MARKER= FLAVOR_FLAGS= ENABLE_ALLOC_TYPE_VARS += nofrag M4FLAGS += - endif TF_MARKER=$(TYPEMARKER)$(FLAVOR_MARKER) diff --git a/erts/emulator/beam/atom.names b/erts/emulator/beam/atom.names index 3022c0a99a..8f65e71531 100644 --- a/erts/emulator/beam/atom.names +++ b/erts/emulator/beam/atom.names @@ -275,6 +275,7 @@ atom garbage_collection_info atom gc_end atom gc_major_end atom gc_major_start +atom gc_max_heap_size atom gc_minor_end atom gc_minor_start atom gc_start @@ -366,6 +367,7 @@ atom match_spec atom match_spec_result atom max atom maximum +atom max_heap_size atom max_tables max_processes atom mbuf_size atom md5 diff --git a/erts/emulator/beam/beam_bp.c b/erts/emulator/beam/beam_bp.c index 2ee98ed7b5..8489897d3a 100644 --- a/erts/emulator/beam/beam_bp.c +++ b/erts/emulator/beam/beam_bp.c @@ -82,7 +82,7 @@ erts_smp_atomic32_t erts_staging_bp_index; static ERTS_INLINE ErtsMonotonicTime get_mtime(Process *c_p) { - return erts_get_monotonic_time(ERTS_PROC_GET_SCHDATA(c_p)); + return erts_get_monotonic_time(erts_proc_sched_data(c_p)); } /* ************************************************************************* @@ -248,7 +248,10 @@ erts_bp_match_export(BpFunctions* f, Eterm mfa[3], int specified) void erts_bp_free_matched_functions(BpFunctions* f) { - Free(f->matching); + if (f->matching) { + Free(f->matching); + } + else ASSERT(f->matched == 0); } void @@ -652,8 +655,7 @@ erts_generic_breakpoint(Process* c_p, BeamInstr* I, Eterm* reg) erts_smp_atomic_inc_nob(&bp->count->acount); } - if (bp_flags & ERTS_BPF_TIME_TRACE_ACTIVE - && ERTS_TRACER_PROC_IS_ENABLED(c_p)) { + if (bp_flags & ERTS_BPF_TIME_TRACE_ACTIVE) { Eterm w; erts_trace_time_call(c_p, I, bp->time); w = (BeamInstr) *c_p->cp; @@ -750,8 +752,7 @@ erts_bif_trace(int bif_index, Process* p, Eterm* args, BeamInstr* I) } } if (bp_flags & ERTS_BPF_TIME_TRACE_ACTIVE && - IS_TRACED_FL(p, F_TRACE_CALLS) && - ERTS_TRACER_PROC_IS_ENABLED(p)) { + IS_TRACED_FL(p, F_TRACE_CALLS)) { BeamInstr *pc = (BeamInstr *)ep->code+3; erts_trace_time_call(p, pc, bp->time); } @@ -973,7 +974,8 @@ erts_trace_time_call(Process* c_p, BeamInstr* I, BpDataTime* bdt) BpDataTime *pbdt = NULL; ASSERT(c_p); - ASSERT(erts_smp_atomic32_read_acqb(&c_p->state) & ERTS_PSFLG_RUNNING); + ASSERT(erts_smp_atomic32_read_acqb(&c_p->state) & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING)); /* get previous timestamp and breakpoint * from the process psd */ @@ -1050,7 +1052,8 @@ erts_trace_time_return(Process *p, BeamInstr *pc) BpDataTime *pbdt = NULL; ASSERT(p); - ASSERT(erts_smp_atomic32_read_acqb(&p->state) & ERTS_PSFLG_RUNNING); + ASSERT(erts_smp_atomic32_read_acqb(&p->state) & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING)); /* get previous timestamp and breakpoint * from the process psd */ @@ -1432,7 +1435,7 @@ set_function_break(BeamInstr *pc, Binary *match_spec, Uint break_flags, g = (GenericBp *) pc[-4]; if (g == 0) { int i; - if (count_op == erts_break_reset || count_op == erts_break_stop) { + if (count_op == ERTS_BREAK_RESTART || count_op == ERTS_BREAK_PAUSE) { /* Do not insert a new breakpoint */ return; } @@ -1456,7 +1459,7 @@ set_function_break(BeamInstr *pc, Binary *match_spec, Uint break_flags, MatchSetUnref(bp->meta_ms); bp_meta_unref(bp->meta_tracer); } else if (common & ERTS_BPF_COUNT) { - if (count_op == erts_break_stop) { + if (count_op == ERTS_BREAK_PAUSE) { bp->flags &= ~ERTS_BPF_COUNT_ACTIVE; } else { bp->flags |= ERTS_BPF_COUNT_ACTIVE; @@ -1468,7 +1471,7 @@ set_function_break(BeamInstr *pc, Binary *match_spec, Uint break_flags, BpDataTime* bdt = bp->time; Uint i = 0; - if (count_op == erts_break_stop) { + if (count_op == ERTS_BREAK_PAUSE) { bp->flags &= ~ERTS_BPF_TIME_TRACE_ACTIVE; } else { bp->flags |= ERTS_BPF_TIME_TRACE_ACTIVE; diff --git a/erts/emulator/beam/beam_bp.h b/erts/emulator/beam/beam_bp.h index 08641b86d6..541af77211 100644 --- a/erts/emulator/beam/beam_bp.h +++ b/erts/emulator/beam/beam_bp.h @@ -80,16 +80,16 @@ typedef struct generic_bp { #define ERTS_BP_CALL_TIME_SCHEDULE_EXITING (2) #ifdef ERTS_SMP -#define bp_sched2ix_proc(p) ((p)->scheduler_data->no - 1) +#define bp_sched2ix_proc(p) (erts_proc_sched_data(p)->no - 1) #else #define bp_sched2ix_proc(p) (0) #endif enum erts_break_op{ - erts_break_nop = 0, /* Must be false */ - erts_break_set = !0, /* Must be true */ - erts_break_reset, - erts_break_stop + ERTS_BREAK_NOP = 0, /* Must be false */ + ERTS_BREAK_SET = !0, /* Must be true */ + ERTS_BREAK_RESTART, + ERTS_BREAK_PAUSE }; typedef Uint32 ErtsBpIndex; diff --git a/erts/emulator/beam/beam_emu.c b/erts/emulator/beam/beam_emu.c index 59112e1e43..f8f2e29c95 100644 --- a/erts/emulator/beam/beam_emu.c +++ b/erts/emulator/beam/beam_emu.c @@ -64,18 +64,21 @@ # ifdef ERTS_SMP # define PROCESS_MAIN_CHK_LOCKS(P) \ do { \ - if ((P)) { \ + if ((P)) \ erts_proc_lc_chk_only_proc_main((P)); \ - } \ - else \ - erts_lc_check_exact(NULL, 0); \ - ERTS_SMP_LC_ASSERT(!erts_thr_progress_is_blocking()); \ + ERTS_SMP_LC_ASSERT(!erts_thr_progress_is_blocking()); \ +} while (0) +# define ERTS_SMP_REQ_PROC_MAIN_LOCK(P) \ +do { \ + if ((P)) \ + erts_proc_lc_require_lock((P), ERTS_PROC_LOCK_MAIN, \ + __FILE__, __LINE__); \ +} while (0) +# define ERTS_SMP_UNREQ_PROC_MAIN_LOCK(P) \ +do { \ + if ((P)) \ + erts_proc_lc_unrequire_lock((P), ERTS_PROC_LOCK_MAIN); \ } while (0) -# define ERTS_SMP_REQ_PROC_MAIN_LOCK(P) \ - if ((P)) erts_proc_lc_require_lock((P), ERTS_PROC_LOCK_MAIN,\ - __FILE__, __LINE__) -# define ERTS_SMP_UNREQ_PROC_MAIN_LOCK(P) \ - if ((P)) erts_proc_lc_unrequire_lock((P), ERTS_PROC_LOCK_MAIN) # else # define ERTS_SMP_REQ_PROC_MAIN_LOCK(P) # define ERTS_SMP_UNREQ_PROC_MAIN_LOCK(P) @@ -1202,12 +1205,12 @@ init_emulator(void) do { \ if (ERTS_PROC_GET_SAVED_CALLS_BUF((P))) { \ ASSERT(FC <= 0); \ - ASSERT(ERTS_PROC_GET_SCHDATA(c_p)->virtual_reds \ + ASSERT(erts_proc_sched_data(c_p)->virtual_reds \ <= 0 - (FC)); \ } \ else { \ ASSERT(FC <= CONTEXT_REDS); \ - ASSERT(ERTS_PROC_GET_SCHDATA(c_p)->virtual_reds \ + ASSERT(erts_proc_sched_data(c_p)->virtual_reds \ <= CONTEXT_REDS - (FC)); \ } \ } while (0) @@ -1321,8 +1324,8 @@ void process_main(void) if (start_time != 0) { Sint64 diff = erts_timestamp_millis() - start_time; if (diff > 0 && (Uint) diff > erts_system_monitor_long_schedule -#ifdef ERTS_DIRTY_SCHEDULERS - && !ERTS_SCHEDULER_IS_DIRTY(c_p->scheduler_data) +#if defined(ERTS_SMP) && defined(ERTS_DIRTY_SCHEDULERS) + && !ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(c_p)) #endif ) { BeamInstr *inptr = find_function_from_pc(start_time_i); @@ -1351,8 +1354,8 @@ void process_main(void) start_time_i = c_p->i; } - reg = ERTS_PROC_GET_SCHDATA(c_p)->x_reg_array; - freg = ERTS_PROC_GET_SCHDATA(c_p)->f_reg_array; + reg = erts_proc_sched_data(c_p)->x_reg_array; + freg = erts_proc_sched_data(c_p)->f_reg_array; ERL_BITS_RELOAD_STATEP(c_p); { int reds; @@ -1704,6 +1707,14 @@ void process_main(void) BeamInstr *next; Eterm result; + if (!(FCALLS > 0 || FCALLS > neg_o_reds)) { + /* If we have run out of reductions, we do a context + switch before calling the bif */ + c_p->arity = 2; + c_p->current = NULL; + goto context_switch3; + } + PRE_BIF_SWAPOUT(c_p); c_p->fcalls = FCALLS - 1; result = erl_send(c_p, r(0), x(1)); @@ -2200,7 +2211,7 @@ void process_main(void) PreFetch(0, next); if (IS_TRACED_FL(c_p, F_TRACE_RECEIVE)) { - trace_receive(c_p, am_timeout); + trace_receive(c_p, am_clock_service, am_timeout, NULL); } if (ERTS_PROC_GET_SAVED_CALLS_BUF(c_p)) { save_calls(c_p, &exp_timeout); @@ -2810,6 +2821,15 @@ do { \ BeamInstr *next; ErlHeapFragment *live_hf_end; + + if (!((FCALLS - 1) > 0 || (FCALLS-1) > neg_o_reds)) { + /* If we have run out of reductions, we do a context + switch before calling the bif */ + c_p->arity = ((Export *)Arg(0))->code[2]; + c_p->current = ((Export *)Arg(0))->code; + goto context_switch3; + } + if (ERTS_MSACC_IS_ENABLED_CACHED_X()) { if (GET_BIF_MODULE(Arg(0)) == am_ets) { ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_ETS); @@ -3338,10 +3358,19 @@ do { \ context_switch2: /* Entry for fun calls. */ c_p->current = I-3; /* Pointer to Mod, Func, Arity */ + context_switch3: + { Eterm* argp; int i; + if (erts_smp_atomic32_read_nob(&c_p->state) & ERTS_PSFLG_EXITING) { + c_p->i = beam_exit; + c_p->arity = 0; + c_p->current = NULL; + goto do_schedule; + } + /* * Make sure that there is enough room for the argument registers to be saved. */ @@ -3509,6 +3538,12 @@ do { \ BifFunction vbf; ErlHeapFragment *live_hf_end; + if (!((FCALLS - 1) > 0 || (FCALLS - 1) > neg_o_reds)) { + /* If we have run out of reductions, we do a context + switch before calling the nif */ + goto context_switch; + } + ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_NIF); DTRACE_NIF_ENTRY(c_p, (Eterm)I[-3], (Eterm)I[-2], (Uint)I[-1]); @@ -3524,18 +3559,27 @@ do { \ typedef Eterm NifF(struct enif_environment_t*, int argc, Eterm argv[]); NifF* fp = vbf = (NifF*) I[1]; struct enif_environment_t env; +#ifdef ERTS_DIRTY_SCHEDULERS + if (!c_p->scheduler_data) + live_hf_end = ERTS_INVALID_HFRAG_PTR; /* On dirty scheduler */ + else +#endif + live_hf_end = c_p->mbuf; erts_pre_nif(&env, c_p, (struct erl_module_nif*)I[2], NULL); - live_hf_end = c_p->mbuf; nif_bif_result = (*fp)(&env, bif_nif_arity, reg); if (env.exception_thrown) nif_bif_result = THE_NON_VALUE; erts_post_nif(&env); - } - ASSERT(!ERTS_PROC_IS_EXITING(c_p) || is_non_value(nif_bif_result)); - PROCESS_MAIN_CHK_LOCKS(c_p); - ERTS_VERIFY_UNUSED_TEMP_ALLOC(c_p); - ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_EMULATOR); + PROCESS_MAIN_CHK_LOCKS(c_p); + ERTS_VERIFY_UNUSED_TEMP_ALLOC(c_p); + ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_EMULATOR); + if (env.exiting) { + ERTS_SMP_REQ_PROC_MAIN_LOCK(c_p); + goto do_schedule; + } + ASSERT(!ERTS_PROC_IS_EXITING(c_p)); + } DTRACE_NIF_RETURN(c_p, (Eterm)I[-3], (Eterm)I[-2], (Uint)I[-1]); goto apply_bif_or_nif_epilogue; @@ -3551,6 +3595,13 @@ do { \ * code[3]: &&apply_bif * code[4]: Function pointer to BIF function */ + + if (!((FCALLS - 1) > 0 || (FCALLS - 1) > neg_o_reds)) { + /* If we have run out of reductions, we do a context + switch before calling the bif */ + goto context_switch; + } + if (ERTS_MSACC_IS_ENABLED_CACHED_X()) { if ((Eterm)I[-3] == am_ets) { ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_ETS); @@ -4887,8 +4938,8 @@ do { \ #ifdef DEBUG pid = c_p->common.id; /* may have switched process... */ #endif - reg = ERTS_PROC_GET_SCHDATA(c_p)->x_reg_array; - freg = ERTS_PROC_GET_SCHDATA(c_p)->f_reg_array; + reg = erts_proc_sched_data(c_p)->x_reg_array; + freg = erts_proc_sched_data(c_p)->f_reg_array; ERL_BITS_RELOAD_STATEP(c_p); /* XXX: this abuse of def_arg_reg[] is horrid! */ neg_o_reds = -c_p->def_arg_reg[4]; diff --git a/erts/emulator/beam/bif.c b/erts/emulator/beam/bif.c index 9d6dfefa3c..2a3bd4afe5 100644 --- a/erts/emulator/beam/bif.c +++ b/erts/emulator/beam/bif.c @@ -44,6 +44,7 @@ #include "erl_ptab.h" #include "erl_bits.h" #include "erl_bif_unique.h" +#include "erl_map.h" #include "erl_msacc.h" Export *erts_await_result; @@ -611,7 +612,7 @@ erts_queue_monitor_message(Process *p, ref_copy = copy_struct(ref, ref_size, &hp, ohp); tup = TUPLE5(hp, am_DOWN, ref_copy, type, item_copy, reason_copy); - erts_queue_message(p, p_locksp, msgp, tup); + erts_queue_message(p, *p_locksp, msgp, tup, am_system); } static BIF_RETTYPE @@ -880,6 +881,8 @@ BIF_RETTYPE spawn_opt_1(BIF_ALIST_1) so.flags = erts_default_spo_flags|SPO_USE_ARGS; so.min_heap_size = H_MIN_SIZE; so.min_vheap_size = BIN_VH_MIN_SIZE; + so.max_heap_size = H_MAX_SIZE; + so.max_heap_flags = H_MAX_FLAGS; so.priority = PRIORITY_NORMAL; so.max_gen_gcs = (Uint16) erts_smp_atomic32_read_nob(&erts_max_gen_gcs); so.scheduler = 0; @@ -937,6 +940,9 @@ BIF_RETTYPE spawn_opt_1(BIF_ALIST_1) } else { so.min_heap_size = erts_next_heap_size(min_heap_size, 0); } + } else if (arg == am_max_heap_size) { + if (!erts_max_heap_size(val, &so.max_heap_size, &so.max_heap_flags)) + goto error; } else if (arg == am_min_bin_vheap_size && is_small(val)) { Sint min_vheap_size = signed_val(val); if (min_vheap_size < 0) { @@ -970,6 +976,10 @@ BIF_RETTYPE spawn_opt_1(BIF_ALIST_1) goto error; } + if (so.max_heap_size != 0 && so.max_heap_size < so.min_heap_size) { + goto error; + } + /* * Spawn the process. */ @@ -1686,7 +1696,7 @@ BIF_RETTYPE process_flag_2(BIF_ALIST_2) ERTS_PSFLG_BOUND); } - curr = ERTS_GET_SCHEDULER_DATA_FROM_PROC(BIF_P)->run_queue; + curr = erts_proc_sched_data(BIF_P)->run_queue; old = (ERTS_PSFLG_BOUND & state) ? curr : NULL; ASSERT(!old || old == curr); @@ -1731,6 +1741,23 @@ BIF_RETTYPE process_flag_2(BIF_ALIST_2) } BIF_RET(old_value); } + else if (BIF_ARG_1 == am_max_heap_size) { + Eterm *hp; + Uint sz = 0, max_heap_size, max_heap_flags; + + if (!erts_max_heap_size(BIF_ARG_2, &max_heap_size, &max_heap_flags)) + goto error; + + if ((max_heap_size < MIN_HEAP_SIZE(BIF_P) && max_heap_size != 0)) + goto error; + + erts_max_heap_size_map(MAX_HEAP_SIZE_GET(BIF_P), MAX_HEAP_SIZE_FLAGS_GET(BIF_P), NULL, &sz); + hp = HAlloc(BIF_P, sz); + old_value = erts_max_heap_size_map(MAX_HEAP_SIZE_GET(BIF_P), MAX_HEAP_SIZE_FLAGS_GET(BIF_P), &hp, NULL); + MAX_HEAP_SIZE_SET(BIF_P, max_heap_size); + MAX_HEAP_SIZE_FLAGS_SET(BIF_P, max_heap_flags); + BIF_RET(old_value); + } else if (BIF_ARG_1 == am_message_queue_data) { old_value = erts_change_message_queue_management(BIF_P, BIF_ARG_2); if (is_non_value(old_value)) @@ -4198,8 +4225,28 @@ BIF_RETTYPE group_leader_2(BIF_ALIST_2) else { locks &= ~ERTS_PROC_LOCK_STATUS; erts_smp_proc_unlock(new_member, ERTS_PROC_LOCK_STATUS); - new_member->group_leader = STORE_NC_IN_PROC(new_member, - BIF_ARG_1); + if (erts_smp_atomic32_read_nob(&new_member->state) + & !(ERTS_PSFLG_DIRTY_RUNNING|ERTS_PSFLG_DIRTY_RUNNING_SYS)) { + new_member->group_leader = STORE_NC_IN_PROC(new_member, + BIF_ARG_1); + } + else { + ErlHeapFragment *bp; + Eterm *hp; + /* + * Other process executing on a dirty scheduler, + * so we are not allowed to write to its heap. + * Store in heap fragment. + */ + + bp = new_message_buffer(NC_HEAP_SIZE(BIF_ARG_1)); + hp = bp->mem; + new_member->group_leader = STORE_NC(&hp, + &new_member->off_heap, + BIF_ARG_1); + bp->next = new_member->mbuf; + new_member->mbuf = bp; + } } } @@ -4350,6 +4397,31 @@ BIF_RETTYPE system_flag_2(BIF_ALIST_2) erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_MAIN); BIF_RET(make_small(oval)); + } else if (BIF_ARG_1 == am_max_heap_size) { + + Eterm *hp, old_value; + Uint sz = 0, max_heap_size, max_heap_flags; + + if (!erts_max_heap_size(BIF_ARG_2, &max_heap_size, &max_heap_flags)) + goto error; + + if (max_heap_size < H_MIN_SIZE && max_heap_size != 0) + goto error; + + erts_max_heap_size_map(H_MAX_SIZE, H_MAX_FLAGS, NULL, &sz); + hp = HAlloc(BIF_P, sz); + old_value = erts_max_heap_size_map(H_MAX_SIZE, H_MAX_FLAGS, &hp, NULL); + + erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCK_MAIN); + erts_smp_thr_progress_block(); + + H_MAX_SIZE = max_heap_size; + H_MAX_FLAGS = max_heap_flags; + + erts_smp_thr_progress_unblock(); + erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_MAIN); + + BIF_RET(old_value); } else if (BIF_ARG_1 == am_display_items) { int oval = display_items; if (!is_small(BIF_ARG_2) || (n = signed_val(BIF_ARG_2)) < 0) { diff --git a/erts/emulator/beam/bif.h b/erts/emulator/beam/bif.h index 5d751dd67d..2203182a0d 100644 --- a/erts/emulator/beam/bif.h +++ b/erts/emulator/beam/bif.h @@ -59,12 +59,12 @@ extern Export *erts_convert_time_unit_trap; do { \ if (!ERTS_PROC_GET_SAVED_CALLS_BUF((p))) { \ if ((fcalls) > 0) \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds += (fcalls); \ + erts_proc_sched_data((p))->virtual_reds += (fcalls); \ (fcalls) = 0; \ } \ else { \ if ((fcalls) > -CONTEXT_REDS) \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds \ + erts_proc_sched_data((p))->virtual_reds \ += ((fcalls) - (-CONTEXT_REDS)); \ (fcalls) = -CONTEXT_REDS; \ } \ @@ -91,22 +91,22 @@ do { \ if (!ERTS_PROC_GET_SAVED_CALLS_BUF((p))) { \ if ((p)->fcalls >= reds) { \ (p)->fcalls -= reds; \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds += reds; \ + erts_proc_sched_data((p))->virtual_reds += reds; \ } \ else { \ if ((p)->fcalls > 0) \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds += (p)->fcalls;\ + erts_proc_sched_data((p))->virtual_reds += (p)->fcalls; \ (p)->fcalls = 0; \ } \ } \ else { \ if ((p)->fcalls >= reds - CONTEXT_REDS) { \ (p)->fcalls -= reds; \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds += reds; \ + erts_proc_sched_data((p))->virtual_reds += reds; \ } \ else { \ if ((p)->fcalls > -CONTEXT_REDS) \ - ERTS_PROC_GET_SCHDATA((p))->virtual_reds \ + erts_proc_sched_data((p))->virtual_reds \ += (p)->fcalls - (-CONTEXT_REDS); \ (p)->fcalls = -CONTEXT_REDS; \ } \ @@ -118,14 +118,14 @@ do { \ if (ERTS_PROC_GET_SAVED_CALLS_BUF((P))) { \ int nreds__ = ((int)(Reds)) - CONTEXT_REDS; \ if ((FCalls) > nreds__) { \ - ERTS_PROC_GET_SCHDATA((P))->virtual_reds \ + erts_proc_sched_data((P))->virtual_reds \ += (FCalls) - nreds__; \ (FCalls) = nreds__; \ } \ } \ else { \ if ((FCalls) > (Reds)) { \ - ERTS_PROC_GET_SCHDATA((P))->virtual_reds \ + erts_proc_sched_data((P))->virtual_reds \ += (FCalls) - (Reds); \ (FCalls) = (Reds); \ } \ @@ -165,7 +165,7 @@ do { \ #define ERTS_BIF_ERROR_TRAPPED1(Proc, Reason, Bif, A0) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -174,7 +174,7 @@ do { \ #define ERTS_BIF_ERROR_TRAPPED2(Proc, Reason, Bif, A0, A1) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -184,7 +184,7 @@ do { \ #define ERTS_BIF_ERROR_TRAPPED3(Proc, Reason, Bif, A0, A1, A2) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -208,7 +208,7 @@ do { \ #define ERTS_BIF_PREP_ERROR_TRAPPED1(Ret, Proc, Reason, Bif, A0) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -217,7 +217,7 @@ do { \ #define ERTS_BIF_PREP_ERROR_TRAPPED2(Ret, Proc, Reason, Bif, A0, A1) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -227,7 +227,7 @@ do { \ #define ERTS_BIF_PREP_ERROR_TRAPPED3(Ret, Proc, Reason, Bif, A0, A1, A2) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->freason = (Reason); \ (Proc)->current = (Bif)->code; \ reg[0] = (Eterm) (A0); \ @@ -246,7 +246,7 @@ do { \ #define ERTS_BIF_PREP_TRAP1(Ret, Trap, Proc, A0) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->arity = 1; \ reg[0] = (Eterm) (A0); \ (Proc)->i = (BeamInstr*) ((Trap)->addressv[erts_active_code_ix()]); \ @@ -256,7 +256,7 @@ do { \ #define ERTS_BIF_PREP_TRAP2(Ret, Trap, Proc, A0, A1) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->arity = 2; \ reg[0] = (Eterm) (A0); \ reg[1] = (Eterm) (A1); \ @@ -267,7 +267,7 @@ do { \ #define ERTS_BIF_PREP_TRAP3(Ret, Trap, Proc, A0, A1, A2) \ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->arity = 3; \ reg[0] = (Eterm) (A0); \ reg[1] = (Eterm) (A1); \ @@ -279,7 +279,7 @@ do { \ #define ERTS_BIF_PREP_TRAP3_NO_RET(Trap, Proc, A0, A1, A2)\ do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((Proc))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((Proc))->x_reg_array; \ (Proc)->arity = 3; \ reg[0] = (Eterm) (A0); \ reg[1] = (Eterm) (A1); \ @@ -296,7 +296,7 @@ do { \ } while(0) #define BIF_TRAP1(Trap_, p, A0) do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((p))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((p))->x_reg_array; \ (p)->arity = 1; \ reg[0] = (A0); \ (p)->i = (BeamInstr*) ((Trap_)->addressv[erts_active_code_ix()]); \ @@ -305,7 +305,7 @@ do { \ } while(0) #define BIF_TRAP2(Trap_, p, A0, A1) do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((p))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((p))->x_reg_array; \ (p)->arity = 2; \ reg[0] = (A0); \ reg[1] = (A1); \ @@ -315,7 +315,7 @@ do { \ } while(0) #define BIF_TRAP3(Trap_, p, A0, A1, A2) do { \ - Eterm* reg = ERTS_PROC_GET_SCHDATA((p))->x_reg_array; \ + Eterm* reg = erts_proc_sched_data((p))->x_reg_array; \ (p)->arity = 3; \ reg[0] = (A0); \ reg[1] = (A1); \ diff --git a/erts/emulator/beam/break.c b/erts/emulator/beam/break.c index d02c6828f9..3c19e82b66 100644 --- a/erts/emulator/beam/break.c +++ b/erts/emulator/beam/break.c @@ -118,7 +118,9 @@ process_killer(void) | ERTS_PSFLG_ACTIVE_SYS | ERTS_PSFLG_IN_RUNQ | ERTS_PSFLG_RUNNING - | ERTS_PSFLG_RUNNING_SYS)) { + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)) { erts_printf("Can only kill WAITING processes this way\n"); } else { @@ -214,7 +216,8 @@ print_process_info(int to, void *to_arg, Process *p) if (state & ERTS_PSFLG_GC) { garbing = 1; running = 1; - } else if (state & ERTS_PSFLG_RUNNING) + } else if (state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING)) running = 1; /* diff --git a/erts/emulator/beam/dist.c b/erts/emulator/beam/dist.c index 277e0668a2..09c83f1117 100644 --- a/erts/emulator/beam/dist.c +++ b/erts/emulator/beam/dist.c @@ -397,7 +397,7 @@ static void doit_node_link_net_exits(ErtsLink *lnk, void *vnecp) msgp = erts_alloc_message_heap(rp, &rp_locks, 3, &hp, &ohp); tup = TUPLE2(hp, am_nodedown, name); - erts_queue_message(rp, &rp_locks, msgp, tup); + erts_queue_message(rp, rp_locks, msgp, tup, am_system); } erts_smp_proc_unlock(rp, rp_locks); } @@ -1456,7 +1456,7 @@ int erts_net_message(Port *prt, token = copy_struct(token, token_size, &hp, ohp); } - erts_queue_dist_message(rp, &locks, ede_copy, token); + erts_queue_dist_message(rp, locks, ede_copy, token, from); if (locks) erts_smp_proc_unlock(rp, locks); } @@ -1505,7 +1505,7 @@ int erts_net_message(Port *prt, token = copy_struct(token, token_size, &hp, ohp); } - erts_queue_dist_message(rp, &locks, ede_copy, token); + erts_queue_dist_message(rp, locks, ede_copy, token, tuple[2]); if (locks) erts_smp_proc_unlock(rp, locks); } @@ -3317,7 +3317,7 @@ send_nodes_mon_msg(Process *rp, } ASSERT(hend == hp); - erts_queue_message(rp, rp_locksp, mp, msg); + erts_queue_message(rp, *rp_locksp, mp, msg, am_system); } static void diff --git a/erts/emulator/beam/erl_alloc.c b/erts/emulator/beam/erl_alloc.c index d04977b9ae..c367d4162c 100644 --- a/erts/emulator/beam/erl_alloc.c +++ b/erts/emulator/beam/erl_alloc.c @@ -137,6 +137,11 @@ static ErtsAllocatorState_t exec_alloc_state; #endif static ErtsAllocatorState_t test_alloc_state; +#define ERTS_ALC_INFO_A_ALLOC_UTIL (ERTS_ALC_A_MAX + 1) +#define ERTS_ALC_INFO_A_MSEG_ALLOC (ERTS_ALC_A_MAX + 2) +#define ERTS_ALC_INFO_A_ERTS_MMAP (ERTS_ALC_A_MAX + 3) +#define ERTS_ALC_INFO_A_MAX ERTS_ALC_INFO_A_ERTS_MMAP + typedef struct { erts_smp_atomic32_t refc; int only_sz; @@ -145,13 +150,9 @@ typedef struct { Process *proc; Eterm ref; Eterm ref_heap[REF_THING_SIZE]; - int allocs[ERTS_ALC_A_MAX-ERTS_ALC_A_MIN+1+2]; + int allocs[ERTS_ALC_INFO_A_MAX - ERTS_ALC_A_MIN + 1 + 1]; } ErtsAllocInfoReq; -#define ERTS_ALC_INFO_A_ALLOC_UTIL (ERTS_ALC_A_MAX + 1) -#define ERTS_ALC_INFO_A_MSEG_ALLOC (ERTS_ALC_A_MAX + 2) -#define ERTS_ALC_INFO_A_MAX ERTS_ALC_INFO_A_MSEG_ALLOC - ERTS_SCHED_PREF_QUICK_ALLOC_IMPL(aireq, ErtsAllocInfoReq, 5, @@ -2840,10 +2841,18 @@ erts_allocator_info(int to, void *arg) int i; for (i = 0; i <= max; i++) { erts_print(to, arg, "=allocator:mseg_alloc[%d]\n", i); - erts_mseg_info(i, &to, arg, 0, NULL, NULL); + erts_mseg_info(i, &to, arg, 0, 0, NULL, NULL); } - erts_print(to, arg, "=allocator:mseg_alloc.erts_mmap\n"); + erts_print(to, arg, "=allocator:erts_mmap.default_mmap\n"); erts_mmap_info(&erts_dflt_mmapper, &to, arg, NULL, NULL, &emis); +#if defined(ARCH_64) && defined(ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION) + erts_print(to, arg, "=allocator:erts_mmap.literal_mmap\n"); + erts_mmap_info(&erts_literal_mmapper, &to, arg, NULL, NULL, &emis); +#endif +#ifdef ERTS_ALC_A_EXEC + erts_print(to, arg, "=allocator:erts_mmap.exec_mmap\n"); + erts_mmap_info(&erts_exec_mmapper, &to, arg, NULL, NULL, &emis); +#endif } #endif @@ -2954,6 +2963,11 @@ erts_allocator_options(void *proc) atoms[length] = am_atom_put("alloc_util", 10); terms[length++] = erts_alcu_au_info_options(NULL, NULL, hpp, szp); +#if HAVE_ERTS_MMAP + atoms[length] = ERTS_MAKE_AM("erts_mmap"); + terms[length++] = erts_mmap_info_options(&erts_dflt_mmapper, NULL, NULL, + NULL, hpp, szp); +#endif { Eterm o[3], v[3]; o[0] = am_atom_put("m", 1); @@ -2990,7 +3004,12 @@ erts_allocator_options(void *proc) #if ERTS_HAVE_ERTS_SYS_ALIGNED_ALLOC terms[length++] = am_atom_put("sys_aligned_alloc", 17); #endif - +#if defined(ARCH_64) && defined(ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION) + terms[length++] = ERTS_MAKE_AM("literal_mmap"); +#endif +#ifdef ERTS_ALC_A_EXEC + terms[length++] = ERTS_MAKE_AM("exec_mmap"); +#endif features = length ? erts_bld_list(hpp, szp, length, terms) : NIL; #if defined(__GLIBC__) @@ -3075,7 +3094,15 @@ reply_alloc_info(void *vair) Uint sz, *szp; ErlOffHeap *ohp = NULL; ErtsMessage *mp = NULL; - struct erts_mmap_info_struct emis; +#if HAVE_ERTS_MMAP + struct erts_mmap_info_struct mmap_info_dflt; +# if defined(ARCH_64) && defined(ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION) + struct erts_mmap_info_struct mmap_info_literal; +# endif +# ifdef ERTS_ALC_A_EXEC + struct erts_mmap_info_struct mmap_info_exec; +# endif +#endif int i; Eterm (*info_func)(Allctr_t *, int, @@ -3181,26 +3208,53 @@ reply_alloc_info(void *vair) make_small(0), ainfo); break; + case ERTS_ALC_INFO_A_ERTS_MMAP: + alloc_atom = erts_bld_atom(hpp, szp, "erts_mmap"); +#if HAVE_ERTS_MMAP + ainfo = (air->only_sz ? NIL : + erts_mmap_info(&erts_dflt_mmapper, NULL, NULL, + hpp, szp, &mmap_info_dflt)); + ainfo = erts_bld_tuple3(hpp, szp, + alloc_atom, + erts_bld_atom(hpp,szp,"default_mmap"), + ainfo); +# if defined(ARCH_64) && defined(ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION) + ai_list = erts_bld_cons(hpp, szp, + ainfo, ai_list); + ainfo = (air->only_sz ? NIL : + erts_mmap_info(&erts_literal_mmapper, NULL, NULL, + hpp, szp, &mmap_info_literal)); + ainfo = erts_bld_tuple3(hpp, szp, + alloc_atom, + erts_bld_atom(hpp,szp,"literal_mmap"), + ainfo); +# endif +# ifdef ERTS_ALC_A_EXEC + ai_list = erts_bld_cons(hpp, szp, + ainfo, ai_list); + ainfo = (air->only_sz ? NIL : + erts_mmap_info(&erts_exec_mmapper, NULL, NULL, + hpp, szp, &mmap_info_exec)); + ainfo = erts_bld_tuple3(hpp, szp, + alloc_atom, + erts_bld_atom(hpp,szp,"exec_mmap"), + ainfo); +# endif +#else /* !HAVE_ERTS_MMAP */ + ainfo = erts_bld_tuple2(hpp, szp, alloc_atom, + am_false); +#endif + break; case ERTS_ALC_INFO_A_MSEG_ALLOC: alloc_atom = erts_bld_atom(hpp, szp, "mseg_alloc"); #if HAVE_ERTS_MSEG - ainfo = (air->only_sz - ? NIL - : erts_mseg_info(0, NULL, NULL, hpp != NULL, - hpp, szp)); + ainfo = erts_mseg_info(0, NULL, NULL, hpp != NULL, + air->only_sz, hpp, szp); ainfo = erts_bld_tuple3(hpp, szp, alloc_atom, make_small(0), ainfo); - ai_list = erts_bld_cons(hpp, szp, - ainfo, ai_list); - ainfo = (air->only_sz ? NIL : - erts_mmap_info(&erts_dflt_mmapper, NULL, NULL, hpp, szp, &emis)); - ainfo = erts_bld_tuple3(hpp, szp, - alloc_atom, - erts_bld_atom(hpp,szp,"erts_mmap"), - ainfo); #else ainfo = erts_bld_tuple2(hpp, szp, alloc_atom, am_false); @@ -3232,15 +3286,14 @@ reply_alloc_info(void *vair) } switch (ai) { case ERTS_ALC_A_SYSTEM: - case ERTS_ALC_INFO_A_ALLOC_UTIL: + case ERTS_ALC_INFO_A_ALLOC_UTIL: + case ERTS_ALC_INFO_A_ERTS_MMAP: break; case ERTS_ALC_INFO_A_MSEG_ALLOC: #if HAVE_ERTS_MSEG && defined(ERTS_SMP) alloc_atom = erts_bld_atom(hpp, szp, "mseg_alloc"); - ainfo = (air->only_sz - ? NIL - : erts_mseg_info(sched_id, NULL, NULL, - hpp != NULL, hpp, szp)); + ainfo = erts_mseg_info(sched_id, NULL, NULL, + hpp != NULL, air->only_sz, hpp, szp); ainfo = erts_bld_tuple(hpp, szp, 3, alloc_atom, make_small(sched_id), @@ -3286,7 +3339,7 @@ reply_alloc_info(void *vair) if (hp != hp_end) erts_shrink_message_heap(&mp, rp, hp_start, hp, hp_end, &msg, 1); - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); if (air->req_sched == sched_id) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -3306,7 +3359,7 @@ erts_request_alloc_info(struct process *c_p, int internal) { ErtsAllocInfoReq *air = aireq_alloc(); - Eterm req_ai[ERTS_ALC_A_MAX+1+2] = {0}; + Eterm req_ai[ERTS_ALC_INFO_A_MAX+1] = {0}; Eterm alist; Eterm *hp; int airix = 0, ai; @@ -3342,6 +3395,10 @@ erts_request_alloc_info(struct process *c_p, ai = ERTS_ALC_INFO_A_MSEG_ALLOC; goto save_alloc; } + if (erts_is_atom_str("erts_mmap", alloc, 0)) { + ai = ERTS_ALC_INFO_A_ERTS_MMAP; + goto save_alloc; + } if (erts_is_atom_str("alloc_util", alloc, 0)) { ai = ERTS_ALC_INFO_A_ALLOC_UTIL; save_alloc: diff --git a/erts/emulator/beam/erl_alloc.types b/erts/emulator/beam/erl_alloc.types index ba216c7eb4..227fedfb69 100644 --- a/erts/emulator/beam/erl_alloc.types +++ b/erts/emulator/beam/erl_alloc.types @@ -278,6 +278,7 @@ type IOB_REQ SHORT_LIVED SYSTEM io_bytes_request type TRACER_NIF LONG_LIVED SYSTEM tracer_nif type TRACE_MSG_QUEUE SHORT_LIVED SYSTEM trace_message_queue type SCHED_ASYNC_JOB SHORT_LIVED SYSTEM async_calls +type DIRTY_START STANDARD PROCESSES dirty_start +if threads_no_smp # Need thread safe allocs, but std_alloc and fix_alloc are not; diff --git a/erts/emulator/beam/erl_bif_ddll.c b/erts/emulator/beam/erl_bif_ddll.c index 1e2db38442..ef77201544 100644 --- a/erts/emulator/beam/erl_bif_ddll.c +++ b/erts/emulator/beam/erl_bif_ddll.c @@ -1737,7 +1737,7 @@ static void notify_proc(Process *proc, Eterm ref, Eterm driver_name, Eterm type, hp += REF_THING_SIZE; mess = TUPLE5(hp,type,r,am_driver,driver_name,tag); } - erts_queue_message(proc, &rp_locks, mp, mess); + erts_queue_message(proc, rp_locks, mp, mess, am_system); erts_smp_proc_unlock(proc, rp_locks); ERTS_SMP_CHK_NO_PROC_LOCKS; } diff --git a/erts/emulator/beam/erl_bif_info.c b/erts/emulator/beam/erl_bif_info.c index 99fe847ba2..2e195db0ee 100644 --- a/erts/emulator/beam/erl_bif_info.c +++ b/erts/emulator/beam/erl_bif_info.c @@ -45,6 +45,7 @@ #include "erl_async.h" #include "erl_thr_progress.h" #include "erl_bif_unique.h" +#include "erl_map.h" #define ERTS_PTAB_WANT_DEBUG_FUNCS__ #include "erl_ptab.h" #ifdef HIPE @@ -594,6 +595,7 @@ static Eterm pi_args[] = { am_suspending, am_min_heap_size, am_min_bin_vheap_size, + am_max_heap_size, am_current_location, am_current_stacktrace, am_message_queue_data, @@ -643,10 +645,11 @@ pi_arg2ix(Eterm arg) case am_suspending: return 26; case am_min_heap_size: return 27; case am_min_bin_vheap_size: return 28; - case am_current_location: return 29; - case am_current_stacktrace: return 30; - case am_message_queue_data: return 31; - case am_garbage_collection_info: return 32; + case am_max_heap_size: return 29; + case am_current_location: return 30; + case am_current_stacktrace: return 31; + case am_message_queue_data: return 32; + case am_garbage_collection_info: return 33; default: return -1; } } @@ -1107,7 +1110,7 @@ process_info_aux(Process *BIF_P, break; case am_status: - res = erts_process_status(BIF_P, ERTS_PROC_LOCK_MAIN, rp, rpid); + res = erts_process_status(rp, rpid); ASSERT(res != am_undefined); hp = HAlloc(BIF_P, 3); break; @@ -1348,6 +1351,18 @@ process_info_aux(Process *BIF_P, break; } + case am_max_heap_size: { + Uint hsz = 3; + (void) erts_max_heap_size_map(MAX_HEAP_SIZE_GET(rp), + MAX_HEAP_SIZE_FLAGS_GET(rp), + NULL, &hsz); + hp = HAlloc(BIF_P, hsz); + res = erts_max_heap_size_map(MAX_HEAP_SIZE_GET(rp), + MAX_HEAP_SIZE_FLAGS_GET(rp), + &hp, NULL); + break; + } + case am_total_heap_size: { ErtsMessage *mp; Uint total_heap_size; @@ -1391,8 +1406,12 @@ process_info_aux(Process *BIF_P, case am_garbage_collection: { DECL_AM(minor_gcs); Eterm t; + Uint map_sz = 0; + + erts_max_heap_size_map(MAX_HEAP_SIZE_GET(rp), MAX_HEAP_SIZE_FLAGS_GET(rp), NULL, &map_sz); - hp = HAlloc(BIF_P, 3+2 + 3+2 + 3+2 + 3+2 + 3); /* last "3" is for outside tuple */ + hp = HAlloc(BIF_P, 3+2 + 3+2 + 3+2 + 3+2 + 3+2 + map_sz + 3); + /* last "3" is for outside tuple */ t = TUPLE2(hp, AM_minor_gcs, make_small(GEN_GCS(rp))); hp += 3; res = CONS(hp, t, NIL); hp += 2; @@ -1403,6 +1422,11 @@ process_info_aux(Process *BIF_P, res = CONS(hp, t, res); hp += 2; t = TUPLE2(hp, am_min_bin_vheap_size, make_small(MIN_VHEAP_SIZE(rp))); hp += 3; res = CONS(hp, t, res); hp += 2; + + t = erts_max_heap_size_map(MAX_HEAP_SIZE_GET(rp), MAX_HEAP_SIZE_FLAGS_GET(rp), &hp, NULL); + + t = TUPLE2(hp, am_max_heap_size, t); hp += 3; + res = CONS(hp, t, res); hp += 2; break; } @@ -1412,12 +1436,12 @@ process_info_aux(Process *BIF_P, if (rp == BIF_P) { sz += ERTS_PROCESS_GC_INFO_MAX_SIZE; } else { - erts_process_gc_info(rp, &sz, NULL); + erts_process_gc_info(rp, &sz, NULL, 0, 0); sz += 3; } hp = HAlloc(BIF_P, sz); - res = erts_process_gc_info(rp, &actual_sz, &hp); + res = erts_process_gc_info(rp, &actual_sz, &hp, 0, 0); /* We may have some extra space, fill with 0 tuples */ if (actual_sz <= sz - 3) { @@ -2035,12 +2059,8 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) Uint arity = *tp++; return info_1_tuple(BIF_P, tp, arityval(arity)); } else if (BIF_ARG_1 == am_scheduler_id) { -#ifdef ERTS_SMP - ASSERT(BIF_P->scheduler_data); - BIF_RET(make_small(BIF_P->scheduler_data->no)); -#else - BIF_RET(make_small(1)); -#endif + ErtsSchedulerData *esdp = erts_proc_sched_data(BIF_P); + BIF_RET(make_small(esdp->no)); } else if (BIF_ARG_1 == am_compat_rel) { ASSERT(erts_compat_rel > 0); BIF_RET(make_small(erts_compat_rel)); @@ -2173,7 +2193,7 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) } else if (BIF_ARG_1 == am_garbage_collection){ Uint val = (Uint) erts_smp_atomic32_read_nob(&erts_max_gen_gcs); Eterm tup; - hp = HAlloc(BIF_P, 3+2 + 3+2 + 3+2); + hp = HAlloc(BIF_P, 3+2 + 3+2 + 3+2 + 3+2); tup = TUPLE2(hp, am_fullsweep_after, make_small(val)); hp += 3; res = CONS(hp, tup, NIL); hp += 2; @@ -2184,6 +2204,9 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) tup = TUPLE2(hp, am_min_bin_vheap_size, make_small(BIN_VH_MIN_SIZE)); hp += 3; res = CONS(hp, tup, res); hp += 2; + tup = TUPLE2(hp, am_max_heap_size, make_small(H_MAX_SIZE)); hp += 3; + res = CONS(hp, tup, res); hp += 2; + BIF_RET(res); } else if (BIF_ARG_1 == am_fullsweep_after){ Uint val = (Uint) erts_smp_atomic32_read_nob(&erts_max_gen_gcs); @@ -2194,6 +2217,12 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) hp = HAlloc(BIF_P, 3); res = TUPLE2(hp, am_min_heap_size,make_small(H_MIN_SIZE)); BIF_RET(res); + } else if (BIF_ARG_1 == am_max_heap_size) { + Uint sz = 0; + erts_max_heap_size_map(H_MAX_SIZE, H_MAX_FLAGS, NULL, &sz); + hp = HAlloc(BIF_P, sz); + res = erts_max_heap_size_map(H_MAX_SIZE, H_MAX_FLAGS, &hp, NULL); + BIF_RET(res); } else if (BIF_ARG_1 == am_min_bin_vheap_size) { hp = HAlloc(BIF_P, 3); res = TUPLE2(hp, am_min_bin_vheap_size,make_small(BIN_VH_MIN_SIZE)); @@ -3555,7 +3584,7 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1) BIF_RET(res); } else if (ERTS_IS_ATOM_STR("mmap", BIF_ARG_1)) { - BIF_RET(erts_mmap_debug_info(&erts_dflt_mmapper, BIF_P)); + BIF_RET(erts_mmap_debug_info(BIF_P)); } else if (ERTS_IS_ATOM_STR("unique_monotonic_integer_state", BIF_ARG_1)) { BIF_RET(erts_debug_get_unique_monotonic_integer_state(BIF_P)); @@ -3589,10 +3618,7 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1) /* Used by timer process_SUITE, timer_bif_SUITE, and node_container_SUITE (emulator) */ if (is_internal_pid(tp[2])) { - BIF_RET(erts_process_status(BIF_P, - ERTS_PROC_LOCK_MAIN, - NULL, - tp[2])); + BIF_RET(erts_process_status(NULL, tp[2])); } } else if (ERTS_IS_ATOM_STR("link_list", tp[1])) { diff --git a/erts/emulator/beam/erl_bif_trace.c b/erts/emulator/beam/erl_bif_trace.c index ff2018aa27..66e5146da0 100644 --- a/erts/emulator/beam/erl_bif_trace.c +++ b/erts/emulator/beam/erl_bif_trace.c @@ -68,6 +68,9 @@ static struct { /* Protected by code write permission */ static Eterm trace_pattern(Process* p, Eterm MFA, Eterm Pattern, Eterm flaglist); +static int +erts_set_tracing_event_pattern(Eterm event, Binary*, int on); + #ifdef ERTS_SMP static void smp_bp_finisher(void* arg); #endif @@ -78,6 +81,8 @@ static void new_seq_trace_token(Process* p); /* help func for seq_trace_2*/ static Eterm trace_info_pid(Process* p, Eterm pid_spec, Eterm key); static Eterm trace_info_func(Process* p, Eterm pid_spec, Eterm key); static Eterm trace_info_on_load(Process* p, Eterm key); +static Eterm trace_info_event(Process* p, Eterm event, Eterm key); + static void reset_bif_trace(void); static void setup_bif_trace(void); @@ -85,14 +90,26 @@ static void install_exp_breakpoints(BpFunctions* f); static void uninstall_exp_breakpoints(BpFunctions* f); static void clean_export_entries(BpFunctions* f); +ErtsTracingEvent erts_send_tracing[ERTS_NUM_BP_IX]; +ErtsTracingEvent erts_receive_tracing[ERTS_NUM_BP_IX]; + void erts_bif_trace_init(void) { + int i; + erts_default_trace_pattern_is_on = 0; erts_default_match_spec = NULL; erts_default_meta_match_spec = NULL; erts_default_trace_pattern_flags = erts_trace_pattern_flags_off; erts_default_meta_tracer = erts_tracer_nil; + + for (i=0; i<ERTS_NUM_BP_IX; i++) { + erts_send_tracing[i].on = 1; + erts_send_tracing[i].match_spec = NULL; + erts_receive_tracing[i].on = 1; + erts_receive_tracing[i].match_spec = NULL; + } } /* @@ -137,15 +154,18 @@ trace_pattern(Process* p, Eterm MFA, Eterm Pattern, Eterm flaglist) on = 1; } else if (Pattern == am_restart) { match_prog_set = NULL; - on = erts_break_reset; + on = ERTS_BREAK_RESTART; } else if (Pattern == am_pause) { match_prog_set = NULL; - on = erts_break_stop; - } else if ((match_prog_set = erts_match_set_compile(p, Pattern)) != NULL) { - MatchSetRef(match_prog_set); - on = 1; - } else{ - goto error; + on = ERTS_BREAK_PAUSE; + } else { + match_prog_set = erts_match_set_compile(p, Pattern, MFA); + if (match_prog_set) { + MatchSetRef(match_prog_set); + on = 1; + } else{ + goto error; + } } is_global = 0; @@ -314,6 +334,11 @@ trace_pattern(Process* p, Eterm MFA, Eterm Pattern, Eterm flaglist) matches = erts_set_trace_pattern(p, mfa, specified, match_prog_set, match_prog_set, on, flags, meta_tracer, 0); + } else if (is_atom(MFA)) { + if (is_global || flags.breakpoint || on > ERTS_BREAK_SET) { + goto error; + } + matches = erts_set_tracing_event_pattern(MFA, match_prog_set, on); } error: @@ -487,8 +512,7 @@ start_trace(Process *c_p, ErtsTracer tracer, && !ERTS_TRACER_COMPARE(ERTS_TRACER(port), tracer)) { /* This tracee is already being traced, and not by the * tracer to be */ - if (erts_is_tracer_proc_enabled(c_p, ERTS_PROC_LOCKS_ALL, - common, am_trace_status)) { + if (erts_is_tracer_enabled(tracer, common)) { /* The tracer is still in use */ return 1; } @@ -791,6 +815,8 @@ Eterm trace_info_2(BIF_ALIST_2) if (What == am_on_load) { res = trace_info_on_load(p, Key); + } else if (What == am_send || What == am_receive) { + res = trace_info_event(p, What, Key); } else if (is_atom(What) || is_pid(What) || is_port(What)) { res = trace_info_pid(p, What, Key); } else if (is_tuple(What)) { @@ -829,7 +855,7 @@ trace_info_pid(Process* p, Eterm pid_spec, Eterm key) return am_undefined; if (!ERTS_TRACER_IS_NIL(ERTS_TRACER(tracee))) - erts_is_tracer_proc_enabled(NULL, 0, &tracee->common, am_trace_status); + erts_is_tracer_proc_enabled(NULL, 0, &tracee->common); tracer = erts_tracer_to_term(p, ERTS_TRACER(tracee)); trace_flags = ERTS_TRACE_FLAGS(tracee); @@ -837,22 +863,24 @@ trace_info_pid(Process* p, Eterm pid_spec, Eterm key) erts_port_release(tracee); } else if (is_internal_pid(pid_spec)) { - Process *tracee; - tracee = erts_pid2proc(p, ERTS_PROC_LOCK_MAIN, - pid_spec, ERTS_PROC_LOCK_MAIN); + Process *tracee = erts_pid2proc_not_running(p, ERTS_PROC_LOCK_MAIN, + pid_spec, ERTS_PROC_LOCK_MAIN); + + if (tracee == ERTS_PROC_LOCK_BUSY) + ERTS_BIF_YIELD2(bif_export[BIF_trace_info_2], p, pid_spec, key); if (!tracee) return am_undefined; if (!ERTS_TRACER_IS_NIL(ERTS_TRACER(tracee))) erts_is_tracer_proc_enabled(tracee, ERTS_PROC_LOCK_MAIN, - &tracee->common, am_trace_status); + &tracee->common); tracer = erts_tracer_to_term(p, ERTS_TRACER(tracee)); trace_flags = ERTS_TRACE_FLAGS(tracee); - if (tracee != p) - erts_smp_proc_unlock(tracee, ERTS_PROC_LOCK_MAIN); + if (tracee != p) + erts_smp_proc_unlock(tracee, ERTS_PROC_LOCK_MAIN); } else if (is_external_pid(pid_spec) && external_pid_dist_entry(pid_spec) == erts_this_dist_entry) { return am_undefined; @@ -1280,6 +1308,42 @@ trace_info_on_load(Process* p, Eterm key) } } +static Eterm +trace_info_event(Process* p, Eterm event, Eterm key) +{ + ErtsTracingEvent* te; + Eterm retval; + Eterm* hp; + + switch (event) { + case am_send: te = erts_send_tracing; break; + case am_receive: te = erts_receive_tracing; break; + default: + goto error; + } + + if (key != am_match_spec) + goto error; + + te = &te[erts_active_bp_ix()]; + + if (te->on) { + if (!te->match_spec) + retval = am_true; + else + retval = copy_object(MatchSetGetSource(te->match_spec), p); + } + else + retval = am_false; + + hp = HAlloc(p, 3); + return TUPLE2(hp, key, retval); + + error: + BIF_ERROR(p, BADARG); +} + + #undef FUNC_TRACE_NOEXIST #undef FUNC_TRACE_UNTRACED #undef FUNC_TRACE_GLOBAL_TRACE @@ -1307,7 +1371,7 @@ erts_set_trace_pattern(Process*p, Eterm* mfa, int specified, for (i = 0; i < n; i++) { BeamInstr* pc = fp[i].pc; - Export* ep = (Export *)(((char *)(pc-3)) - offsetof(Export, code)); + Export* ep = ErtsContainerStruct(pc, Export, code[3]); if (on && !flags.breakpoint) { /* Turn on global call tracing */ @@ -1468,12 +1532,57 @@ erts_set_trace_pattern(Process*p, Eterm* mfa, int specified, } int +erts_set_tracing_event_pattern(Eterm event, Binary* match_spec, int on) +{ + ErtsBpIndex ix = erts_staging_bp_ix(); + ErtsTracingEvent* st; + + switch (event) { + case am_send: st = &erts_send_tracing[ix]; break; + case am_receive: st = &erts_receive_tracing[ix]; break; + default: return -1; + } + + MatchSetUnref(st->match_spec); + + st->on = on; + st->match_spec = match_spec; + MatchSetRef(match_spec); + + finish_bp.current = 1; /* prepare phase not needed for event trace */ + finish_bp.install = on; + finish_bp.e.matched = 0; + finish_bp.e.matching = NULL; + finish_bp.f.matched = 0; + finish_bp.f.matching = NULL; + +#ifndef ERTS_SMP + while (erts_finish_breakpointing()) { + /* Empty loop body */ + } +#endif + return 1; +} + +static void +consolidate_event_tracing(ErtsTracingEvent te[]) +{ + ErtsTracingEvent* src = &te[erts_active_bp_ix()]; + ErtsTracingEvent* dst = &te[erts_staging_bp_ix()]; + + MatchSetUnref(dst->match_spec); + dst->on = src->on; + dst->match_spec = src->match_spec; + MatchSetRef(dst->match_spec); +} + +int erts_finish_breakpointing(void) { ERTS_SMP_LC_ASSERT(erts_has_code_write_permission()); /* - * Memory barriers will be issued for all processes *before* + * Memory barriers will be issued for all schedulers *before* * each of the stages below. (Unless the other schedulers * are blocked, in which case memory barriers will be issued * when they are awaken.) @@ -1542,6 +1651,8 @@ erts_finish_breakpointing(void) erts_consolidate_bp_data(&finish_bp.f, 1); erts_bp_free_matched_functions(&finish_bp.e); erts_bp_free_matched_functions(&finish_bp.f); + consolidate_event_tracing(erts_send_tracing); + consolidate_event_tracing(erts_receive_tracing); return 0; default: ASSERT(0); @@ -1556,11 +1667,10 @@ install_exp_breakpoints(BpFunctions* f) BpFunction* fp = f->matching; Uint ne = f->matched; Uint i; - Uint offset = offsetof(Export, code) + 3*sizeof(BeamInstr); for (i = 0; i < ne; i++) { BeamInstr* pc = fp[i].pc; - Export* ep = (Export *) (((char *)pc)-offset); + Export* ep = ErtsContainerStruct(pc, Export, code[3]); ep->addressv[code_ix] = pc; } @@ -1573,11 +1683,10 @@ uninstall_exp_breakpoints(BpFunctions* f) BpFunction* fp = f->matching; Uint ne = f->matched; Uint i; - Uint offset = offsetof(Export, code) + 3*sizeof(BeamInstr); for (i = 0; i < ne; i++) { BeamInstr* pc = fp[i].pc; - Export* ep = (Export *) (((char *)pc)-offset); + Export* ep = ErtsContainerStruct(pc, Export, code[3]); if (ep->addressv[code_ix] != pc) { continue; @@ -1594,11 +1703,10 @@ clean_export_entries(BpFunctions* f) BpFunction* fp = f->matching; Uint ne = f->matched; Uint i; - Uint offset = offsetof(Export, code) + 3*sizeof(BeamInstr); for (i = 0; i < ne; i++) { BeamInstr* pc = fp[i].pc; - Export* ep = (Export *) (((char *)pc)-offset); + Export* ep = ErtsContainerStruct(pc, Export, code[3]); if (ep->addressv[code_ix] == pc) { continue; @@ -2277,7 +2385,7 @@ reply_trace_delivered_all(void *vtdarp) #ifdef ERTS_SMP erts_send_sys_msg_proc(rp->common.id, rp->common.id, msg, bp); #else - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); #endif erts_free(ERTS_ALC_T_MISC_AUX_WORK, vtdarp); diff --git a/erts/emulator/beam/erl_bif_unique.c b/erts/emulator/beam/erl_bif_unique.c index 1e57e9fa53..7c70217d8d 100644 --- a/erts/emulator/beam/erl_bif_unique.c +++ b/erts/emulator/beam/erl_bif_unique.c @@ -257,7 +257,7 @@ static ERTS_INLINE Eterm unique_integer_bif(Process *c_p, int positive) Uint hsz; Eterm *hp; - esdp = ERTS_PROC_GET_SCHDATA(c_p); + esdp = erts_proc_sched_data(c_p); thr_id = (Uint64) esdp->thr_id; unique = esdp->unique++; bld_unique_integer_term(NULL, &hsz, thr_id, unique, positive); @@ -515,7 +515,7 @@ BIF_RETTYPE make_ref_0(BIF_ALIST_0) hp = HAlloc(BIF_P, REF_THING_SIZE); - res = erts_sched_make_ref_in_buffer(ERTS_PROC_GET_SCHDATA(BIF_P), hp); + res = erts_sched_make_ref_in_buffer(erts_proc_sched_data(BIF_P), hp); BIF_RET(res); } diff --git a/erts/emulator/beam/erl_bits.h b/erts/emulator/beam/erl_bits.h index 1c2a090f07..4bd5b24157 100644 --- a/erts/emulator/beam/erl_bits.h +++ b/erts/emulator/beam/erl_bits.h @@ -83,8 +83,8 @@ typedef struct erl_bin_match_struct{ #ifdef ERTS_SMP /* the state resides in the current process' scheduler data */ #define ERL_BITS_DECLARE_STATEP struct erl_bits_state *EBS -#define ERL_BITS_RELOAD_STATEP(P) do{EBS = &(P)->scheduler_data->erl_bits_state;}while(0) -#define ERL_BITS_DEFINE_STATEP(P) struct erl_bits_state *EBS = &(P)->scheduler_data->erl_bits_state +#define ERL_BITS_RELOAD_STATEP(P) do{EBS = &erts_proc_sched_data((P))->erl_bits_state;}while(0) +#define ERL_BITS_DEFINE_STATEP(P) struct erl_bits_state *EBS = &erts_proc_sched_data((P))->erl_bits_state #else /* reentrant API but with a hidden single global state, for testing only */ extern struct erl_bits_state ErlBitsState_; diff --git a/erts/emulator/beam/erl_db.c b/erts/emulator/beam/erl_db.c index 615d23402b..bad34211a5 100644 --- a/erts/emulator/beam/erl_db.c +++ b/erts/emulator/beam/erl_db.c @@ -2833,7 +2833,8 @@ BIF_RETTYPE ets_match_spec_run_r_3(BIF_ALIST_3) BIF_TRAP3(bif_export[BIF_ets_match_spec_run_r_3], BIF_P,lst,BIF_ARG_2,ret); } - res = db_prog_match(BIF_P, mp, CAR(list_val(lst)), NULL, 0, + res = db_prog_match(BIF_P, BIF_P, + mp, CAR(list_val(lst)), NULL, 0, ERTS_PAM_COPY_RESULT, &dummy); if (is_value(res)) { hp = HAlloc(BIF_P, 2); @@ -3461,7 +3462,7 @@ static void fix_table_locked(Process* p, DbTable* tb) fix = tb->common.fixations; if (fix == NULL) { tb->common.time.monotonic - = erts_get_monotonic_time(ERTS_PROC_GET_SCHDATA(p)); + = erts_get_monotonic_time(erts_proc_sched_data(p)); tb->common.time.offset = erts_get_time_offset(); } else { diff --git a/erts/emulator/beam/erl_db_util.c b/erts/emulator/beam/erl_db_util.c index 76b96637ae..6732b708a8 100644 --- a/erts/emulator/beam/erl_db_util.c +++ b/erts/emulator/beam/erl_db_util.c @@ -123,6 +123,9 @@ do { \ #define TermWords(t) (((t) / (sizeof(UWord)/sizeof(Eterm))) + !!((t) % (sizeof(UWord)/sizeof(Eterm)))) +#define add_dmc_err(EINFO, STR, VAR, TERM, SEV) \ + vadd_dmc_err(EINFO, SEV, VAR, STR, TERM) + static ERTS_INLINE Process * get_proc(Process *cp, Uint32 cp_locks, Eterm id, Uint32 id_locks) @@ -171,7 +174,8 @@ set_match_trace(Process *tracee_p, Eterm fail_term, ErtsTracer tracer, ERTS_PROC_LOCKS_ALL == erts_proc_lc_my_proc_locks(tracee_p) || erts_thr_progress_is_blocking()); - if (ERTS_TRACER_IS_NIL(tracer) || erts_is_tracer_enabled(tracee_p, tracer)) + if (ERTS_TRACER_IS_NIL(tracer) + || erts_is_tracer_enabled(tracer, &tracee_p->common)) return set_tracee_flags(tracee_p, tracer, d_flags, e_flags); return fail_term; } @@ -411,17 +415,27 @@ get_match_pseudo_process(Process *c_p, Uint heap_size) { ErtsMatchPseudoProcess *mpsp; #ifdef ERTS_SMP - mpsp = (ErtsMatchPseudoProcess *) c_p->scheduler_data->match_pseudo_process; - if (mpsp) + ErtsSchedulerData *esdp; + + esdp = c_p ? c_p->scheduler_data : erts_get_scheduler_data(); + + mpsp = esdp ? esdp->match_pseudo_process : + (ErtsMatchPseudoProcess*) erts_smp_tsd_get(match_pseudo_process_key); + + if (mpsp) { + ASSERT(mpsp == erts_smp_tsd_get(match_pseudo_process_key)); + ASSERT(mpsp->process.scheduler_data == esdp); cleanup_match_pseudo_process(mpsp, 0); + } else { ASSERT(erts_smp_tsd_get(match_pseudo_process_key) == NULL); mpsp = create_match_pseudo_process(); - c_p->scheduler_data->match_pseudo_process = (void *) mpsp; + if (esdp) { + esdp->match_pseudo_process = (void *) mpsp; + } + mpsp->process.scheduler_data = esdp; erts_smp_tsd_set(match_pseudo_process_key, (void *) mpsp); } - ASSERT(mpsp == erts_smp_tsd_get(match_pseudo_process_key)); - mpsp->process.scheduler_data = c_p->scheduler_data; #else mpsp = match_pseudo_process; cleanup_match_pseudo_process(mpsp, 0); @@ -889,11 +903,7 @@ void db_match_dis(Binary *prog); #define TRACE /* Nothing */ #define FENCE_PATTERN_SIZE 0 #endif -static void add_dmc_err(DMCErrInfo *err_info, - char *str, - int variable, - Eterm term, - DMCErrorSeverity severity); +static void vadd_dmc_err(DMCErrInfo*, DMCErrorSeverity, int var, const char *str, ...); static Eterm dpm_array_to_list(Process *psp, Eterm *arr, int arity); @@ -989,12 +999,20 @@ Eterm erts_match_set_get_source(Binary *mpsp) } /* This one is for the tracing */ -Binary *erts_match_set_compile(Process *p, Eterm matchexpr) { +Binary *erts_match_set_compile(Process *p, Eterm matchexpr, Eterm MFA) { Binary *bin; Uint sz; Eterm *hp; + Uint flags; + + switch (MFA) { + case am_receive: flags = DCOMP_TRACE; break; + case am_send: flags = DCOMP_TRACE | DCOMP_ALLOW_TRACE_OPS; break; + default: + flags = DCOMP_TRACE | DCOMP_CALL_TRACE | DCOMP_ALLOW_TRACE_OPS; + } - bin = db_match_set_compile(p, matchexpr, DCOMP_TRACE); + bin = db_match_set_compile(p, matchexpr, flags); if (bin != NULL) { MatchProg *prog = Binary2MatchProg(bin); sz = size_object(matchexpr); @@ -1124,8 +1142,8 @@ Eterm db_match_set_lint(Process *p, Eterm matchexpr, Uint flags) int i; if (!is_list(matchexpr)) { - add_dmc_err(err_info, "Match programs are not in a list.", - -1, 0UL, dmcError); + add_dmc_err(err_info, "Match programs are not in a list.", + -1, 0UL, dmcError); goto done; } num_heads = 0; @@ -1133,9 +1151,8 @@ Eterm db_match_set_lint(Process *p, Eterm matchexpr, Uint flags) ++num_heads; if (l != NIL) { /* proper list... */ - add_dmc_err(err_info, "Match programs are not in a proper " - "list.", - -1, 0UL, dmcError); + add_dmc_err(err_info, "Match programs are not in a proper list.", + -1, 0UL, dmcError); goto done; } @@ -1202,30 +1219,37 @@ done: return ret; } -Eterm erts_match_set_run(Process *p, Binary *mpsp, - Eterm *args, int num_args, - enum erts_pam_run_flags in_flags, - Uint32 *return_flags) +/* Returns + * am_false if no match or + * if {message,false} has been called, + * am_true if {message,_} has NOT been called or + * if {message,true} has been called, + * Msg if {message,Msg} has been called. + * + * If return value is_not_immed + * then erts_match_set_release_result_trace() must be called to release it. + */ +Eterm erts_match_set_run_trace(Process *c_p, + Process *self, + Binary *mpsp, + Eterm *args, int num_args, + enum erts_pam_run_flags in_flags, + Uint32 *return_flags) { Eterm ret; - ret = db_prog_match(p, mpsp, NIL, args, num_args, + ret = db_prog_match(c_p, self, mpsp, NIL, args, num_args, in_flags, return_flags); -#if defined(HARDDEBUG) - if (is_non_value(ret)) { - erts_fprintf(stderr, "Failed\n"); - } else { - erts_fprintf(stderr, "Returning : %T\n", ret); + + ASSERT(!(is_non_value(ret) && *return_flags)); + + if (is_non_value(ret) || ret == am_false) { + erts_match_set_release_result(c_p); + return am_false; } -#endif + if (is_immed(ret)) + erts_match_set_release_result(c_p); return ret; - /* Returns - * THE_NON_VALUE if no match - * am_false if {message,false} has been called, - * am_true if {message,_} has not been called or - * if {message,true} has been called, - * Msg if {message,Msg} has been called. - */ } static Eterm erts_match_set_run_ets(Process *p, Binary *mpsp, @@ -1234,7 +1258,8 @@ static Eterm erts_match_set_run_ets(Process *p, Binary *mpsp, { Eterm ret; - ret = db_prog_match(p, mpsp, args, NULL, num_args, + ret = db_prog_match(p, p, + mpsp, args, NULL, num_args, ERTS_PAM_COPY_RESULT, return_flags); #if defined(HARDDEBUG) @@ -1730,7 +1755,9 @@ static Eterm dpm_array_to_list(Process *psp, Eterm *arr, int arity) ** the parameter 'arity' is only used if 'term' is actually an array, ** i.e. 'DCOMP_TRACE' was specified */ -Eterm db_prog_match(Process *c_p, Binary *bprog, +Eterm db_prog_match(Process *c_p, + Process *self, + Binary *bprog, Eterm term, Eterm *termp, int arity, @@ -1743,10 +1770,10 @@ Eterm db_prog_match(Process *c_p, Binary *bprog, Eterm *esp; MatchVariable* variables; BeamInstr *cp; - UWord *pc = prog->text; + const UWord *pc = prog->text; Eterm *ehp; Eterm ret; - Uint n = 0; /* To avoid warning. */ + Uint n; int i; unsigned do_catch; ErtsMatchPseudoProcess *mpsp; @@ -1758,46 +1785,28 @@ Eterm db_prog_match(Process *c_p, Binary *bprog, Eterm (*bif)(Process*, ...); Eterm bif_args[3]; int fail_label; - int atomic_trace; #ifdef DMC_DEBUG Uint *heap_fence; Uint *stack_fence; Uint save_op; #endif /* DMC_DEBUG */ + ERTS_UNDEF(n,0); + ERTS_UNDEF(current_scheduled,NULL); + + ASSERT(c_p || !(in_flags & ERTS_PAM_COPY_RESULT)); + mpsp = get_match_pseudo_process(c_p, prog->heap_size); psp = &mpsp->process; /* We need to lure the scheduler into believing in the pseudo process, because of floating point exceptions. Do *after* mpsp is set!!! */ - esdp = ERTS_GET_SCHEDULER_DATA_FROM_PROC(c_p); - ASSERT(esdp != NULL); - current_scheduled = esdp->current_process; + esdp = erts_get_scheduler_data(); + if (esdp) + current_scheduled = esdp->current_process; /* SMP: psp->scheduler_data is set by get_match_pseudo_process */ - atomic_trace = 0; -#define BEGIN_ATOMIC_TRACE(p) \ - do { \ - if (! atomic_trace) { \ - erts_refc_inc(&bprog->refc, 2); \ - erts_smp_proc_unlock((p), ERTS_PROC_LOCK_MAIN); \ - erts_smp_thr_progress_block(); \ - atomic_trace = !0; \ - } \ - } while (0) -#define END_ATOMIC_TRACE(p) \ - do { \ - if (atomic_trace) { \ - erts_smp_thr_progress_unblock(); \ - erts_smp_proc_lock((p), ERTS_PROC_LOCK_MAIN); \ - if (erts_refc_dectest(&bprog->refc, 0) == 0) {\ - erts_bin_free(bprog); \ - } \ - atomic_trace = 0; \ - } \ - } while (0) - #ifdef DMC_DEBUG save_op = 0; heap_fence = (Eterm*)((char*) mpsp->u.heap + prog->stack_offset) - 1; @@ -2256,7 +2265,7 @@ restart: pc += n; break; case matchSelf: - *esp++ = c_p->common.id; + *esp++ = self->common.id; break; case matchWaste: --esp; @@ -2266,6 +2275,7 @@ restart: break; case matchProcessDump: { erts_dsprintf_buf_t *dsbufp = erts_create_tmp_dsbuf(0); + ASSERT(c_p == self); print_process_info(ERTS_PRINT_DSBUF, (void *) dsbufp, c_p); *esp++ = new_binary(build_proc, (byte *)dsbufp->str, dsbufp->str_len); @@ -2284,14 +2294,16 @@ restart: *return_flags |= MATCH_SET_EXCEPTION_TRACE; *esp++ = am_true; break; - case matchIsSeqTrace: + case matchIsSeqTrace: + ASSERT(c_p == self); if (have_seqtrace(SEQ_TRACE_TOKEN(c_p))) *esp++ = am_true; else *esp++ = am_false; break; case matchSetSeqToken: - t = erts_seq_trace(c_p, esp[-1], esp[-2], 0); + ASSERT(c_p == self); + t = erts_seq_trace(c_p, esp[-1], esp[-2], 0); if (is_non_value(t)) { esp[-2] = FAIL_TERM; } else { @@ -2299,7 +2311,8 @@ restart: } --esp; break; - case matchSetSeqTokenFake: + case matchSetSeqTokenFake: + ASSERT(c_p == self); t = seq_trace_fake(c_p, esp[-1]); if (is_non_value(t)) { esp[-2] = FAIL_TERM; @@ -2308,7 +2321,8 @@ restart: } --esp; break; - case matchGetSeqToken: + case matchGetSeqToken: + ASSERT(c_p == self); if (have_no_seqtrace(SEQ_TRACE_TOKEN(c_p))) *esp++ = NIL; else { @@ -2332,49 +2346,62 @@ restart: ASSERT(is_immed(ehp[5])); } break; - case matchEnableTrace: + case matchEnableTrace: + ASSERT(c_p == self); if ( (n = erts_trace_flag2bit(esp[-1]))) { - BEGIN_ATOMIC_TRACE(c_p); + erts_smp_proc_lock(c_p, ERTS_PROC_LOCKS_ALL_MINOR); set_tracee_flags(c_p, ERTS_TRACER(c_p), 0, n); + erts_smp_proc_unlock(c_p, ERTS_PROC_LOCKS_ALL_MINOR); esp[-1] = am_true; } else { esp[-1] = FAIL_TERM; } break; - case matchEnableTrace2: + case matchEnableTrace2: + ASSERT(c_p == self); n = erts_trace_flag2bit((--esp)[-1]); esp[-1] = FAIL_TERM; if (n) { - BEGIN_ATOMIC_TRACE(c_p); - if ( (tmpp = get_proc(c_p, 0, esp[0], 0))) { + if ( (tmpp = get_proc(c_p, ERTS_PROC_LOCK_MAIN, esp[0], ERTS_PROC_LOCKS_ALL))) { /* Always take over the tracer of the current process */ set_tracee_flags(tmpp, ERTS_TRACER(c_p), 0, n); - esp[-1] = am_true; + if (tmpp == c_p) + erts_smp_proc_unlock(tmpp, ERTS_PROC_LOCKS_ALL_MINOR); + else + erts_smp_proc_unlock(tmpp, ERTS_PROC_LOCKS_ALL); + esp[-1] = am_true; } } break; - case matchDisableTrace: + case matchDisableTrace: + ASSERT(c_p == self); if ( (n = erts_trace_flag2bit(esp[-1]))) { - BEGIN_ATOMIC_TRACE(c_p); + erts_smp_proc_lock(c_p, ERTS_PROC_LOCKS_ALL_MINOR); set_tracee_flags(c_p, ERTS_TRACER(c_p), n, 0); + erts_smp_proc_unlock(c_p, ERTS_PROC_LOCKS_ALL_MINOR); esp[-1] = am_true; } else { esp[-1] = FAIL_TERM; } break; - case matchDisableTrace2: + case matchDisableTrace2: + ASSERT(c_p == self); n = erts_trace_flag2bit((--esp)[-1]); esp[-1] = FAIL_TERM; if (n) { - BEGIN_ATOMIC_TRACE(c_p); - if ( (tmpp = get_proc(c_p, 0, esp[0], 0))) { + if ( (tmpp = get_proc(c_p, ERTS_PROC_LOCK_MAIN, esp[0], ERTS_PROC_LOCKS_ALL))) { /* Always take over the tracer of the current process */ set_tracee_flags(tmpp, ERTS_TRACER(c_p), n, 0); - esp[-1] = am_true; + if (tmpp == c_p) + erts_smp_proc_unlock(tmpp, ERTS_PROC_LOCKS_ALL_MINOR); + else + erts_smp_proc_unlock(tmpp, ERTS_PROC_LOCKS_ALL); + esp[-1] = am_true; } } break; - case matchCaller: + case matchCaller: + ASSERT(c_p == self); if (!(c_p->cp) || !(cp = find_function_from_pc(c_p->cp))) { *esp++ = am_undefined; } else { @@ -2386,7 +2413,8 @@ restart: ehp[3] = make_small((Uint) cp[2]); } break; - case matchSilent: + case matchSilent: + ASSERT(c_p == self); --esp; if (in_flags & ERTS_PAM_IGNORE_TRACE_SILENT) break; @@ -2401,7 +2429,8 @@ restart: erts_smp_proc_unlock(c_p, ERTS_PROC_LOCKS_ALL_MINOR); } break; - case matchTrace2: + case matchTrace2: + ASSERT(c_p == self); { /* disable enable */ Uint d_flags = 0, e_flags = 0; /* process trace flags */ @@ -2433,7 +2462,8 @@ restart: ERTS_TRACER_CLEAR(&tracer); } break; - case matchTrace3: + case matchTrace3: + ASSERT(c_p == self); { /* disable enable */ Uint d_flags = 0, e_flags = 0; /* process trace flags */ @@ -2506,15 +2536,12 @@ success: } #endif - esdp->current_process = current_scheduled; - - END_ATOMIC_TRACE(c_p); + if (esdp) + esdp->current_process = current_scheduled; return ret; #undef FAIL #undef FAIL_TERM -#undef BEGIN_ATOMIC_TRACE -#undef END_ATOMIC_TRACE } @@ -3259,20 +3286,20 @@ int erts_db_is_compiled_ms(Eterm term) ** Utility to add an error */ -static void add_dmc_err(DMCErrInfo *err_info, - char *str, - int variable, - Eterm term, - DMCErrorSeverity severity) +static void vadd_dmc_err(DMCErrInfo *err_info, + DMCErrorSeverity severity, + int variable, + const char *str, + ...) { + DMCError *e; + va_list args; + va_start(args, str); + + /* Linked in in reverse order, to ease the formatting */ - DMCError *e = erts_alloc(ERTS_ALC_T_DB_DMC_ERROR, sizeof(DMCError)); - if (term != 0UL) { - erts_snprintf(e->error_string, DMC_ERR_STR_LEN, str, term); - } else { - strncpy(e->error_string, str, DMC_ERR_STR_LEN); - e->error_string[DMC_ERR_STR_LEN] ='\0'; - } + e = erts_alloc(ERTS_ALC_T_DB_DMC_ERROR, sizeof(DMCError)); + erts_vsnprintf(e->error_string, DMC_ERR_STR_LEN, str, args); e->variable = variable; e->severity = severity; e->next = err_info->first; @@ -3282,8 +3309,11 @@ static void add_dmc_err(DMCErrInfo *err_info, err_info->first = e; if (severity >= dmcError) err_info->error_added = 1; + + va_end(args); } + /* ** Handle one term in the match expression (not the guard) */ @@ -3482,24 +3512,21 @@ static void do_emit_constant(DMCContext *context, DMC_STACK_TYPE(UWord) *text, context->stack_need = context->stack_used; } -#define RETURN_ERROR_X(String, X, Y, ContextP, ConstantF) \ -do { \ -if ((ContextP)->err_info != NULL) { \ - (ConstantF) = 0; \ - add_dmc_err((ContextP)->err_info, String, X, Y, dmcError); \ - return retOk; \ -} else \ - return retFail; \ -} while(0) +#define RETURN_ERROR_X(VAR, ContextP, ConstantF, String, ARG) \ + (((ContextP)->err_info != NULL) \ + ? ((ConstantF) = 0, \ + vadd_dmc_err((ContextP)->err_info, dmcError, VAR, String, ARG), \ + retOk) \ + : retFail) #define RETURN_ERROR(String, ContextP, ConstantF) \ - RETURN_ERROR_X(String, -1, 0UL, ContextP, ConstantF) + return RETURN_ERROR_X(-1, ContextP, ConstantF, String, 0) #define RETURN_VAR_ERROR(String, N, ContextP, ConstantF) \ - RETURN_ERROR_X(String, N, 0UL, ContextP, ConstantF) + return RETURN_ERROR_X(N, ContextP, ConstantF, String, 0) #define RETURN_TERM_ERROR(String, T, ContextP, ConstantF) \ - RETURN_ERROR_X(String, -1, T, ContextP, ConstantF) + return RETURN_ERROR_X(-1, ContextP, ConstantF, String, T) #define WARNING(String, ContextP) \ add_dmc_err((ContextP)->err_info, String, -1, 0UL, dmcWarning) @@ -3765,7 +3792,7 @@ static DMCRet dmc_variable(DMCContext *context, Uint n = db_is_variable(t); if (n >= heap->vars_used || !heap->vars[n].is_bound) { - RETURN_VAR_ERROR("Variable $%d is unbound.", n, context, *constant); + RETURN_VAR_ERROR("Variable $%%d is unbound.", n, context, *constant); } dmc_add_pushv_variant(context, heap, text, n); @@ -4097,7 +4124,30 @@ static DMCRet dmc_exception_trace(DMCContext *context, return retOk; } - +static int check_trace(const char* op, + DMCContext *context, + int *constant, + int need_cflags, + int allow_in_guard, + DMCRet* retp) +{ + if (!(context->cflags & DCOMP_TRACE)) { + *retp = RETURN_ERROR_X(-1, context, *constant, "Special form '%s' " + "used in wrong dialect.", op); + return 0; + } + if ((context->cflags & need_cflags) != need_cflags) { + *retp = RETURN_ERROR_X(-1, context, *constant, "Special form '%s' " + "not allow for this trace event.", op); + return 0; + } + if (context->is_guard && !allow_in_guard) { + *retp = RETURN_ERROR_X(-1, context, *constant, "Special form '%s' " + "called in guard context.", op); + return 0; + } + return 1; +} static DMCRet dmc_is_seq_trace(DMCContext *context, DMCHeap *heap, @@ -4107,12 +4157,11 @@ static DMCRet dmc_is_seq_trace(DMCContext *context, { Eterm *p = tuple_val(t); Uint a = arityval(*p); + DMCRet ret; - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'is_seq_trace' used in wrong dialect.", - context, - *constant); - } + if (!check_trace("is_seq_trace", context, constant, DCOMP_ALLOW_TRACE_OPS, 1, &ret)) + return ret; + if (a != 1) { RETURN_TERM_ERROR("Special form 'is_seq_trace' called with " "arguments in %T.", t, context, *constant); @@ -4136,16 +4185,8 @@ static DMCRet dmc_set_seq_token(DMCContext *context, DMCRet ret; int c; - - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'set_seq_token' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'set_seq_token' called in " - "guard context.", context, *constant); - } + if (!check_trace("set_seq_trace", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; if (a != 3) { RETURN_TERM_ERROR("Special form 'set_seq_token' called with wrong " @@ -4182,16 +4223,11 @@ static DMCRet dmc_get_seq_token(DMCContext *context, { Eterm *p = tuple_val(t); Uint a = arityval(*p); + DMCRet ret; + + if (!check_trace("get_seq_token", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'get_seq_token' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'get_seq_token' called in " - "guard context.", context, *constant); - } if (a != 1) { RETURN_TERM_ERROR("Special form 'get_seq_token' called with " "arguments in %T.", t, context, @@ -4255,16 +4291,10 @@ static DMCRet dmc_process_dump(DMCContext *context, { Eterm *p = tuple_val(t); Uint a = arityval(*p); - - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'process_dump' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'process_dump' called in " - "guard context.", context, *constant); - } + DMCRet ret; + + if (!check_trace("process_dump", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; if (a != 1) { RETURN_TERM_ERROR("Special form 'process_dump' called with " @@ -4288,17 +4318,8 @@ static DMCRet dmc_enable_trace(DMCContext *context, DMCRet ret; int c; - - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'enable_trace' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'enable_trace' called in guard context.", - context, - *constant); - } + if (!check_trace("enable_trace", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; switch (a) { case 2: @@ -4347,18 +4368,9 @@ static DMCRet dmc_disable_trace(DMCContext *context, Uint a = arityval(*p); DMCRet ret; int c; - - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'disable_trace' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'disable_trace' called in guard context.", - context, - *constant); - } + if (!check_trace("disable_trace", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; switch (a) { case 2: @@ -4408,17 +4420,8 @@ static DMCRet dmc_trace(DMCContext *context, DMCRet ret; int c; - - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'trace' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'trace' called in guard context.", - context, - *constant); - } + if (!check_trace("trace", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; switch (a) { case 3: @@ -4479,16 +4482,11 @@ static DMCRet dmc_caller(DMCContext *context, { Eterm *p = tuple_val(t); Uint a = arityval(*p); + DMCRet ret; - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'caller' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'caller' called in " - "guard context.", context, *constant); - } + if (!check_trace("caller", context, constant, + (DCOMP_CALL_TRACE|DCOMP_ALLOW_TRACE_OPS), 0, &ret)) + return ret; if (a != 1) { RETURN_TERM_ERROR("Special form 'caller' called with " @@ -4514,15 +4512,8 @@ static DMCRet dmc_silent(DMCContext *context, DMCRet ret; int c; - if (!(context->cflags & DCOMP_TRACE)) { - RETURN_ERROR("Special form 'silent' used in wrong dialect.", - context, - *constant); - } - if (context->is_guard) { - RETURN_ERROR("Special form 'silent' called in " - "guard context.", context, *constant); - } + if (!check_trace("silent", context, constant, DCOMP_ALLOW_TRACE_OPS, 0, &ret)) + return ret; if (a != 2) { RETURN_TERM_ERROR("Special form 'silent' called with wrong " @@ -5062,11 +5053,14 @@ static Eterm match_spec_test(Process *p, Eterm against, Eterm spec, int trace) return THE_NON_VALUE; } if (trace) { - lint_res = db_match_set_lint(p, spec, DCOMP_TRACE | DCOMP_FAKE_DESTRUCTIVE); - mps = db_match_set_compile(p, spec, DCOMP_TRACE | DCOMP_FAKE_DESTRUCTIVE); + const Uint cflags = (DCOMP_TRACE | DCOMP_FAKE_DESTRUCTIVE | + DCOMP_CALL_TRACE | DCOMP_ALLOW_TRACE_OPS); + lint_res = db_match_set_lint(p, spec, cflags); + mps = db_match_set_compile(p, spec, cflags); } else { - lint_res = db_match_set_lint(p, spec, DCOMP_TABLE | DCOMP_FAKE_DESTRUCTIVE); - mps = db_match_set_compile(p, spec, DCOMP_TABLE | DCOMP_FAKE_DESTRUCTIVE); + const Uint cflags = (DCOMP_TABLE | DCOMP_FAKE_DESTRUCTIVE); + lint_res = db_match_set_lint(p, spec, cflags); + mps = db_match_set_compile(p, spec, cflags); } if (mps == NULL) { @@ -5099,7 +5093,8 @@ static Eterm match_spec_test(Process *p, Eterm against, Eterm spec, int trace) } save_cp = p->cp; p->cp = NULL; - res = erts_match_set_run(p, mps, arr, n, + res = erts_match_set_run_trace(p, p, + mps, arr, n, ERTS_PAM_COPY_RESULT|ERTS_PAM_IGNORE_TRACE_SILENT, &ret_flags); p->cp = save_cp; @@ -5182,7 +5177,8 @@ Eterm db_match_dbterm(DbTableCommon* tb, Process* c_p, Binary* bprog, obj = db_alloc_tmp_uncompressed(tb, obj); } - res = db_prog_match(c_p, bprog, make_tuple(obj->tpl), NULL, 0, + res = db_prog_match(c_p, c_p, + bprog, make_tuple(obj->tpl), NULL, 0, ERTS_PAM_COPY_RESULT|ERTS_PAM_CONTIGUOUS_TUPLE, &dummy); if (is_value(res) && hpp!=NULL) { diff --git a/erts/emulator/beam/erl_db_util.h b/erts/emulator/beam/erl_db_util.h index 5d7946a525..60f7067d70 100644 --- a/erts/emulator/beam/erl_db_util.h +++ b/erts/emulator/beam/erl_db_util.h @@ -425,6 +425,11 @@ typedef struct dmc_err_info { #define DCOMP_FAKE_DESTRUCTIVE ((Uint) 8) /* When this is active, no setting of trace control words or seq_trace tokens will be done. */ +/* Allow lock seizing operations on the tracee and 3rd party processes */ +#define DCOMP_ALLOW_TRACE_OPS ((Uint) 0x10) + +/* This is call trace */ +#define DCOMP_CALL_TRACE ((Uint) 0x20) Binary *db_match_compile(Eterm *matchexpr, Eterm *guards, Eterm *body, int num_matches, @@ -435,7 +440,8 @@ Binary *db_match_compile(Eterm *matchexpr, Eterm *guards, Eterm db_match_dbterm(DbTableCommon* tb, Process* c_p, Binary* bprog, int all, DbTerm* obj, Eterm** hpp, Uint extra); -Eterm db_prog_match(Process *p, Binary *prog, Eterm term, +Eterm db_prog_match(Process *p, Process *self, + Binary *prog, Eterm term, Eterm *termp, int arity, enum erts_pam_run_flags in_flags, Uint32 *return_flags /* Zeroed on enter */); diff --git a/erts/emulator/beam/erl_drv_nif.h b/erts/emulator/beam/erl_drv_nif.h index 2700b62854..6ec5fbb895 100644 --- a/erts/emulator/beam/erl_drv_nif.h +++ b/erts/emulator/beam/erl_drv_nif.h @@ -43,12 +43,11 @@ typedef struct { int suggested_stack_size; } ErlDrvThreadOpts; -#if defined(ERL_DRV_DIRTY_SCHEDULER_SUPPORT) || defined(ERL_NIF_DIRTY_SCHEDULER_SUPPORT) + typedef enum { - ERL_DRV_DIRTY_JOB_CPU_BOUND = 1, - ERL_DRV_DIRTY_JOB_IO_BOUND = 2 -} ErlDrvDirtyJobFlags; -#endif + ERL_DIRTY_JOB_CPU_BOUND = 1, + ERL_DIRTY_JOB_IO_BOUND = 2 +} ErlDirtyJobFlags; #ifdef SIZEOF_CHAR # define SIZEOF_CHAR_SAVED__ SIZEOF_CHAR diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index e21fba6bf1..d740b2baec 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -41,6 +41,7 @@ #endif #include "dtrace-wrapper.h" #include "erl_bif_unique.h" +#include "dist.h" #define ERTS_INACT_WR_PB_LEAVE_MUCH_LIMIT 1 #define ERTS_INACT_WR_PB_LEAVE_MUCH_PERCENTAGE 20 @@ -146,7 +147,8 @@ static void offset_rootset(Process *p, Sint offs, char* area, Uint area_size, static void offset_off_heap(Process* p, Sint offs, char* area, Uint area_size); static void offset_mqueue(Process *p, Sint offs, char* area, Uint area_size); static void move_msgq_to_heap(Process *p); - +static int reached_max_heap_size(Process *p, Uint total_heap_size, + Uint extra_heap_size, Uint extra_old_heap_size); static void init_gc_info(ErtsGCInfo *gcip); #ifdef HARDDEBUG @@ -389,7 +391,7 @@ erts_gc_after_bif_call_lhf(Process* p, ErlHeapFragment *live_hf_end, if (p->freason == TRAP) { #if HIPE if (regs == NULL) { - regs = ERTS_PROC_GET_SCHDATA(p)->x_reg_array; + regs = erts_proc_sched_data(p)->x_reg_array; } #endif cost = garbage_collect(p, live_hf_end, 0, regs, p->arity, p->fcalls); @@ -404,6 +406,7 @@ erts_gc_after_bif_call_lhf(Process* p, ErlHeapFragment *live_hf_end, result = val[0]; } BUMP_REDS(p, cost); + return result; } @@ -507,14 +510,14 @@ delay_garbage_collection(Process *p, ErlHeapFragment *live_hf_end, int need, int /* Make sure that we do a proper GC as soon as possible... */ p->flags |= F_FORCE_GC; reds_left = ERTS_REDS_LEFT(p, fcalls); - ASSERT(CONTEXT_REDS - reds_left >= ERTS_PROC_GET_SCHDATA(p)->virtual_reds); + ASSERT(CONTEXT_REDS - reds_left >= erts_proc_sched_data(p)->virtual_reds); if (reds_left > ERTS_ABANDON_HEAP_COST) { int vreds = reds_left - ERTS_ABANDON_HEAP_COST; - ERTS_PROC_GET_SCHDATA((p))->virtual_reds += vreds; + erts_proc_sched_data((p))->virtual_reds += vreds; } - ASSERT(CONTEXT_REDS >= ERTS_PROC_GET_SCHDATA(p)->virtual_reds); + ASSERT(CONTEXT_REDS >= erts_proc_sched_data(p)->virtual_reds); return reds_left; } @@ -577,18 +580,22 @@ garbage_collect(Process* p, ErlHeapFragment *live_hf_end, int need, Eterm* objv, int nobj, int fcalls) { Uint reclaimed_now = 0; + Eterm gc_trace_end_tag; int reds; ErtsMonotonicTime start_time = 0; /* Shut up faulty warning... */ ErtsSchedulerData *esdp; + erts_aint32_t state; ERTS_MSACC_PUSH_STATE_M(); #ifdef USE_VM_PROBES DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); #endif ASSERT(CONTEXT_REDS - ERTS_REDS_LEFT(p, fcalls) - >= ERTS_PROC_GET_SCHDATA(p)->virtual_reds); + >= erts_proc_sched_data(p)->virtual_reds); - if (p->flags & (F_DISABLE_GC|F_DELAY_GC)) + state = erts_smp_atomic32_read_nob(&p->state); + + if (p->flags & (F_DISABLE_GC|F_DELAY_GC) || state & ERTS_PSFLG_EXITING) return delay_garbage_collection(p, live_hf_end, need, fcalls); if (p->abandoned_heap) @@ -623,28 +630,28 @@ garbage_collect(Process* p, ErlHeapFragment *live_hf_end, if (GEN_GCS(p) < MAX_GEN_GCS(p) && !(FLAGS(p) & F_NEED_FULLSWEEP)) { if (IS_TRACED_FL(p, F_TRACE_GC)) { - trace_gc(p, am_gc_minor_start, need); + trace_gc(p, am_gc_minor_start, need, THE_NON_VALUE); } DTRACE2(gc_minor_start, pidbuf, need); reds = minor_collection(p, live_hf_end, need, objv, nobj, &reclaimed_now); DTRACE2(gc_minor_end, pidbuf, reclaimed_now); - if (IS_TRACED_FL(p, F_TRACE_GC)) { - trace_gc(p, am_gc_minor_end, reclaimed_now); - } - if (reds < 0) + if (reds == -1) { + if (IS_TRACED_FL(p, F_TRACE_GC)) { + trace_gc(p, am_gc_minor_end, reclaimed_now, THE_NON_VALUE); + } goto do_major_collection; + } + gc_trace_end_tag = am_gc_minor_end; } else { do_major_collection: ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_GC_FULL); if (IS_TRACED_FL(p, F_TRACE_GC)) { - trace_gc(p, am_gc_major_start, need); + trace_gc(p, am_gc_major_start, need, THE_NON_VALUE); } DTRACE2(gc_major_start, pidbuf, need); reds = major_collection(p, live_hf_end, need, objv, nobj, &reclaimed_now); DTRACE2(gc_major_end, pidbuf, reclaimed_now); - if (IS_TRACED_FL(p, F_TRACE_GC)) { - trace_gc(p, am_gc_major_end, reclaimed_now); - } + gc_trace_end_tag = am_gc_major_end; ERTS_MSACC_SET_STATE_CACHED_M_X(ERTS_MSACC_STATE_GC); } @@ -658,8 +665,33 @@ do_major_collection: ErtsGcQuickSanityCheck(p); + /* Max heap size has been reached and the process was configured + to be killed, so we kill it and set it in a delayed garbage + collecting state. There should be no gc_end trace or + long_gc/large_gc triggers when this happens as process was + killed before a GC could be done. */ + if (reds == -2) { + ErtsProcLocks locks = ERTS_PROC_LOCKS_ALL; + + erts_smp_proc_lock(p, ERTS_PROC_LOCKS_ALL_MINOR); + erts_send_exit_signal(p, p->common.id, p, &locks, + am_kill, NIL, NULL, 0); + erts_smp_proc_unlock(p, locks & ERTS_PROC_LOCKS_ALL_MINOR); + + /* erts_send_exit_signal looks for ERTS_PSFLG_GC, so + we have to remove it after the signal is sent */ + erts_smp_atomic32_read_band_nob(&p->state, ~ERTS_PSFLG_GC); + + /* We have to make sure that we have space for need on the heap */ + return delay_garbage_collection(p, live_hf_end, need, fcalls); + } + erts_smp_atomic32_read_band_nob(&p->state, ~ERTS_PSFLG_GC); + if (IS_TRACED_FL(p, F_TRACE_GC)) { + trace_gc(p, gc_trace_end_tag, reclaimed_now, THE_NON_VALUE); + } + if (erts_system_monitor_long_gc != 0) { ErtsMonotonicTime end_time; Uint gc_time; @@ -716,7 +748,7 @@ erts_garbage_collect_nobump(Process* p, int need, Eterm* objv, int nobj, int fca int reds_left = ERTS_REDS_LEFT(p, fcalls); if (reds > reds_left) reds = reds_left; - ASSERT(CONTEXT_REDS - (reds_left - reds) >= ERTS_PROC_GET_SCHDATA(p)->virtual_reds); + ASSERT(CONTEXT_REDS - (reds_left - reds) >= erts_proc_sched_data(p)->virtual_reds); return reds; } @@ -726,7 +758,7 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj) int reds = garbage_collect(p, ERTS_INVALID_HFRAG_PTR, need, objv, nobj, p->fcalls); BUMP_REDS(p, reds); ASSERT(CONTEXT_REDS - ERTS_BIF_REDS_LEFT(p) - >= ERTS_PROC_GET_SCHDATA(p)->virtual_reds); + >= erts_proc_sched_data(p)->virtual_reds); } /* @@ -761,6 +793,7 @@ erts_garbage_collect_hibernate(Process* p) heap_size = p->heap_sz + (p->old_htop - p->old_heap) + p->mbuf_sz; + heap = (Eterm*) ERTS_HEAP_ALLOC(ERTS_ALC_T_TMP_HEAP, sizeof(Eterm)*heap_size); htop = heap; @@ -1031,6 +1064,34 @@ minor_collection(Process* p, ErlHeapFragment *live_hf_end, Uint size_before = young_gen_usage(p); /* + * Check if we have gone past the max heap size limit + */ + + if (MAX_HEAP_SIZE_GET(p)) { + Uint heap_size = size_before, + /* Note that we also count the un-allocated area + in between the stack and heap */ + stack_size = HEAP_END(p) - HEAP_TOP(p), + extra_heap_size, + extra_old_heap_size = 0; + + /* Add potential old heap size */ + if (OLD_HEAP(p) == NULL && mature_size != 0) { + extra_old_heap_size = erts_next_heap_size(size_before, 1); + heap_size += extra_old_heap_size; + } else if (OLD_HEAP(p)) + heap_size += OLD_HEND(p) - OLD_HEAP(p); + + /* Add potential new young heap size */ + extra_heap_size = next_heap_size(p, stack_size + size_before, 0); + heap_size += extra_heap_size; + + if (heap_size > MAX_HEAP_SIZE_GET(p)) + if (reached_max_heap_size(p, heap_size, extra_heap_size, extra_old_heap_size)) + return -2; + } + + /* * Allocate an old heap if we don't have one and if we'll need one. */ @@ -1139,6 +1200,16 @@ minor_collection(Process* p, ErlHeapFragment *live_hf_end, ASSERT(HEAP_SIZE(p) == next_heap_size(p, HEAP_SIZE(p), 0)); ASSERT(MBUF(p) == NULL); + /* The heap usage during GC should be larger than what we end up + after a GC, even if we grow it. If this assertion is not true + we have to check size in grow_new_heap and potentially kill the + process from there */ + ASSERT(!MAX_HEAP_SIZE_GET(p) || + !(MAX_HEAP_SIZE_FLAGS_GET(p) & MAX_HEAP_SIZE_KILL) || + MAX_HEAP_SIZE_GET(p) > (young_gen_usage(p) + + (OLD_HEND(p) - OLD_HEAP(p)) + + (HEAP_END(p) - HEAP_TOP(p)))); + return gc_cost(size_after, adjust_size); } @@ -1457,6 +1528,25 @@ major_collection(Process* p, ErlHeapFragment *live_hf_end, if (new_sz == HEAP_SIZE(p) && FLAGS(p) & F_HEAP_GROW) { new_sz = next_heap_size(p, HEAP_SIZE(p), 1); } + + + if (MAX_HEAP_SIZE_GET(p)) { + Uint heap_size = size_before; + + /* Add unused space in old heap */ + heap_size += OLD_HEND(p) - OLD_HTOP(p); + + /* Add stack + unused space in young heap */ + heap_size += HEAP_END(p) - HEAP_TOP(p); + + /* Add size of new young heap */ + heap_size += new_sz; + + if (MAX_HEAP_SIZE_GET(p) < heap_size) + if (reached_max_heap_size(p, heap_size, new_sz, 0)) + return -2; + } + FLAGS(p) &= ~(F_HEAP_GROW|F_NEED_FULLSWEEP); n_htop = n_heap = (Eterm *) ERTS_HEAP_ALLOC(ERTS_ALC_T_HEAP, sizeof(Eterm)*new_sz); @@ -2938,7 +3028,7 @@ reply_gc_info(void *vgcirp) hpp = &hp; } - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); if (gcirp->req_sched == esdp->no) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -2955,7 +3045,7 @@ reply_gc_info(void *vgcirp) Eterm erts_gc_info_request(Process *c_p) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); Eterm ref; ErtsGCInfoReq *gcirp; Eterm *hp; @@ -2986,7 +3076,9 @@ erts_gc_info_request(Process *c_p) } Eterm -erts_process_gc_info(Process *p, Uint *sizep, Eterm **hpp) +erts_process_gc_info(Process *p, Uint *sizep, Eterm **hpp, + Uint extra_heap_block, + Uint extra_old_heap_block_size) { ERTS_DECL_AM(bin_vheap_size); ERTS_DECL_AM(bin_vheap_block_size); @@ -3009,8 +3101,9 @@ erts_process_gc_info(Process *p, Uint *sizep, Eterm **hpp) AM_bin_old_vheap_block_size }; UWord values[] = { - OLD_HEAP(p) ? OLD_HEND(p) - OLD_HEAP(p) : 0, - HEAP_SIZE(p), + OLD_HEAP(p) ? OLD_HEND(p) - OLD_HEAP(p) + extra_old_heap_block_size + : extra_old_heap_block_size, + HEAP_SIZE(p) + extra_heap_block, MBUF_SIZE(p), HIGH_WATER(p) - HEAP_START(p), STACK_START(p) - p->stop, @@ -3056,6 +3149,130 @@ erts_process_gc_info(Process *p, Uint *sizep, Eterm **hpp) return res; } +static int +reached_max_heap_size(Process *p, Uint total_heap_size, + Uint extra_heap_size, Uint extra_old_heap_size) +{ + Uint max_heap_flags = MAX_HEAP_SIZE_FLAGS_GET(p); + if (IS_TRACED_FL(p, F_TRACE_GC) || + max_heap_flags & MAX_HEAP_SIZE_LOG) { + Eterm msg; + Uint size = 0; + Eterm *o_hp , *hp; + erts_process_gc_info(p, &size, NULL, extra_heap_size, + extra_old_heap_size); + o_hp = hp = erts_alloc(ERTS_ALC_T_TMP, size * sizeof(Eterm)); + msg = erts_process_gc_info(p, NULL, &hp, extra_heap_size, + extra_old_heap_size); + + if (max_heap_flags & MAX_HEAP_SIZE_LOG) { + int alive = erts_is_alive; + erts_dsprintf_buf_t *dsbufp = erts_create_logger_dsbuf(); + Eterm *o_hp, *hp, args = NIL; + + /* Build the format message */ + erts_dsprintf(dsbufp, " Process: ~p "); + if (alive) + erts_dsprintf(dsbufp, "on node ~p"); + erts_dsprintf(dsbufp, "~n Context: maximum heap size reached~n"); + erts_dsprintf(dsbufp, " Max Heap Size: ~p~n"); + erts_dsprintf(dsbufp, " Total Heap Size: ~p~n"); + erts_dsprintf(dsbufp, " Kill: ~p~n"); + erts_dsprintf(dsbufp, " Error Logger: ~p~n"); + erts_dsprintf(dsbufp, " GC Info: ~p~n"); + + /* Build the args in reverse order */ + o_hp = hp = erts_alloc(ERTS_ALC_T_TMP, 2*(alive ? 7 : 6) * sizeof(Eterm)); + args = CONS(hp, msg, args); hp += 2; + args = CONS(hp, am_true, args); hp += 2; + args = CONS(hp, (max_heap_flags & MAX_HEAP_SIZE_KILL ? am_true : am_false), args); hp += 2; + args = CONS(hp, make_small(total_heap_size), args); hp += 2; + args = CONS(hp, make_small(MAX_HEAP_SIZE_GET(p)), args); hp += 2; + if (alive) { + args = CONS(hp, erts_this_node->sysname, args); hp += 2; + } + args = CONS(hp, p->common.id, args); hp += 2; + + erts_send_error_term_to_logger(p->group_leader, dsbufp, args); + erts_free(ERTS_ALC_T_TMP, o_hp); + } + + if (IS_TRACED_FL(p, F_TRACE_GC)) + trace_gc(p, am_gc_max_heap_size, 0, msg); + + erts_free(ERTS_ALC_T_TMP, o_hp); + } + /* returns true if we should kill the process */ + return max_heap_flags & MAX_HEAP_SIZE_KILL; +} + +Eterm +erts_max_heap_size_map(Sint max_heap_size, Uint max_heap_flags, + Eterm **hpp, Uint *sz) +{ + if (!hpp) { + *sz += (2*3 + 1 + MAP_HEADER_FLATMAP_SZ); + return THE_NON_VALUE; + } else { + Eterm *hp = *hpp; + Eterm keys = TUPLE3(hp, am_error_logger, am_kill, am_size); + flatmap_t *mp; + hp += 4; + mp = (flatmap_t*) hp; + mp->thing_word = MAP_HEADER_FLATMAP; + mp->size = 3; + mp->keys = keys; + hp += MAP_HEADER_FLATMAP_SZ; + *hp++ = max_heap_flags & MAX_HEAP_SIZE_LOG ? am_true : am_false; + *hp++ = max_heap_flags & MAX_HEAP_SIZE_KILL ? am_true : am_false; + *hp++ = make_small(max_heap_size); + *hpp = hp; + return make_flatmap(mp); + } +} + +int +erts_max_heap_size(Eterm arg, Uint *max_heap_size, Uint *max_heap_flags) +{ + Sint sz; + *max_heap_flags = H_MAX_FLAGS; + if (is_small(arg)) { + sz = signed_val(arg); + *max_heap_flags = H_MAX_FLAGS; + } else if (is_map(arg)) { + const Eterm *size = erts_maps_get(am_size, arg); + const Eterm *kill = erts_maps_get(am_kill, arg); + const Eterm *log = erts_maps_get(am_error_logger, arg); + if (size && is_small(*size)) { + sz = signed_val(*size); + } else { + /* size is mandatory */ + return 0; + } + if (kill) { + if (*kill == am_true) + *max_heap_flags |= MAX_HEAP_SIZE_KILL; + else if (*kill == am_false) + *max_heap_flags &= ~MAX_HEAP_SIZE_KILL; + else + return 0; + } + if (log) { + if (*log == am_true) + *max_heap_flags |= MAX_HEAP_SIZE_LOG; + else if (*log == am_false) + *max_heap_flags &= ~MAX_HEAP_SIZE_LOG; + else + return 0; + } + } else + return 0; + if (sz < 0) + return 0; + *max_heap_size = sz; + return 1; +} + #if defined(DEBUG) || defined(ERTS_OFFHEAP_DEBUG) static int diff --git a/erts/emulator/beam/erl_gc.h b/erts/emulator/beam/erl_gc.h index 8a6ff99990..54ea9ca3c0 100644 --- a/erts/emulator/beam/erl_gc.h +++ b/erts/emulator/beam/erl_gc.h @@ -135,7 +135,7 @@ typedef struct { #define ERTS_PROCESS_GC_INFO_MAX_TERMS (11) /* number of elements in process_gc_info*/ #define ERTS_PROCESS_GC_INFO_MAX_SIZE \ (ERTS_PROCESS_GC_INFO_MAX_TERMS * (2/*cons*/ + 3/*2-tuple*/ + BIG_UINT_HEAP_SIZE)) -Eterm erts_process_gc_info(struct process*, Uint *, Eterm **); +Eterm erts_process_gc_info(struct process*, Uint *, Eterm **, Uint, Uint); void erts_gc_info(ErtsGCInfo *gcip); void erts_init_gc(void); @@ -155,5 +155,7 @@ void erts_offset_off_heap(struct erl_off_heap*, Sint, Eterm*, Eterm*); void erts_offset_heap_ptr(Eterm*, Uint, Sint, Eterm*, Eterm*); void erts_offset_heap(Eterm*, Uint, Sint, Eterm*, Eterm*); void erts_free_heap_frags(struct process* p); +Eterm erts_max_heap_size_map(Sint, Uint, Eterm **, Uint *); +int erts_max_heap_size(Eterm, Uint *, Uint *); #endif /* __ERL_GC_H__ */ diff --git a/erts/emulator/beam/erl_hl_timer.c b/erts/emulator/beam/erl_hl_timer.c index 8e201d5711..ebeff51aac 100644 --- a/erts/emulator/beam/erl_hl_timer.c +++ b/erts/emulator/beam/erl_hl_timer.c @@ -1247,8 +1247,8 @@ hlt_bif_timer_timeout(ErtsHLTimer *tmr, Uint32 roflgs) if (!ERTS_PROC_IS_EXITING(proc)) { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = tmr->btm.bp; - erts_queue_message(proc, &proc_locks, mp, - tmr->btm.message); + erts_queue_message(proc, proc_locks, mp, + tmr->btm.message, am_clock_service); erts_smp_proc_unlock(proc, ERTS_PROC_LOCKS_MSG_SEND); queued_message = 1; proc_locks &= ~ERTS_PROC_LOCKS_MSG_SEND; @@ -1766,7 +1766,7 @@ setup_bif_timer(Process *c_p, ErtsMonotonicTime timeout_pos, if (is_not_internal_pid(rcvr) && is_not_atom(rcvr)) goto badarg; - esdp = ERTS_PROC_GET_SCHDATA(c_p); + esdp = erts_proc_sched_data(c_p); hp = HAlloc(c_p, REF_THING_SIZE); ref = erts_sched_make_ref_in_buffer(esdp, hp); @@ -1871,7 +1871,7 @@ access_sched_local_btm(Process *c_p, Eterm pid, if (!c_p) esdp = erts_get_scheduler_data(); else { - esdp = ERTS_PROC_GET_SCHDATA(c_p); + esdp = erts_proc_sched_data(c_p); ERTS_HLT_ASSERT(esdp == erts_get_scheduler_data()); } @@ -1980,7 +1980,7 @@ access_sched_local_btm(Process *c_p, Eterm pid, ERTS_HLT_ASSERT(hp + (async ? 4 : 3) == hp_end); - erts_queue_message(proc, &proc_locks, mp, msg); + erts_queue_message(proc, proc_locks, mp, msg, am_clock_service); if (c_p) proc_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -2111,7 +2111,7 @@ try_access_sched_remote_btm(ErtsSchedulerData *esdp, msg = TUPLE3(hp, tag, tref, res); - erts_queue_message(c_p, &proc_locks, mp, msg); + erts_queue_message(c_p, proc_locks, mp, msg, am_clock_service); proc_locks &= ~ERTS_PROC_LOCK_MAIN; if (proc_locks) @@ -2138,7 +2138,7 @@ access_bif_timer(Process *c_p, Eterm tref, int cancel, int async, int info) goto no_timer; } - esdp = ERTS_PROC_GET_SCHDATA(c_p); + esdp = erts_proc_sched_data(c_p); trefn = internal_ref_numbers(tref); sid = erts_get_ref_numbers_thr_id(trefn); @@ -2363,7 +2363,7 @@ typedef struct { int erts_cancel_bif_timers(Process *p, ErtsBifTimers *btm, void **vyspp) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(p); + ErtsSchedulerData *esdp = erts_proc_sched_data(p); ErtsBifTimerYieldState ys = {btm, {ERTS_RBT_YIELD_STAT_INITER}}; ErtsBifTimerYieldState *ysp; int res; @@ -2409,7 +2409,7 @@ detach_bif_timer(ErtsHLTimer *tmr, void *vesdp) int erts_detach_accessor_bif_timers(Process *p, ErtsBifTimers *btm, void **vyspp) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(p); + ErtsSchedulerData *esdp = erts_proc_sched_data(p); ErtsBifTimerYieldState ys = {btm, {ERTS_RBT_YIELD_STAT_INITER}}; ErtsBifTimerYieldState *ysp; int res; @@ -2516,7 +2516,7 @@ BIF_RETTYPE send_after_3(BIF_ALIST_3) ErtsMonotonicTime timeout_pos; int short_time, tres; - tres = parse_timeout_pos(ERTS_PROC_GET_SCHDATA(BIF_P), BIF_ARG_1, NULL, + tres = parse_timeout_pos(erts_proc_sched_data(BIF_P), BIF_ARG_1, NULL, 0, &timeout_pos, &short_time); if (tres != 0) BIF_ERROR(BIF_P, BADARG); @@ -2534,7 +2534,7 @@ BIF_RETTYPE send_after_4(BIF_ALIST_4) if (!parse_bif_timer_options(BIF_ARG_4, NULL, NULL, &abs, &accessor)) BIF_ERROR(BIF_P, BADARG); - tres = parse_timeout_pos(ERTS_PROC_GET_SCHDATA(BIF_P), BIF_ARG_1, NULL, + tres = parse_timeout_pos(erts_proc_sched_data(BIF_P), BIF_ARG_1, NULL, abs, &timeout_pos, &short_time); if (tres != 0) BIF_ERROR(BIF_P, BADARG); @@ -2548,7 +2548,7 @@ BIF_RETTYPE start_timer_3(BIF_ALIST_3) ErtsMonotonicTime timeout_pos; int short_time, tres; - tres = parse_timeout_pos(ERTS_PROC_GET_SCHDATA(BIF_P), BIF_ARG_1, NULL, + tres = parse_timeout_pos(erts_proc_sched_data(BIF_P), BIF_ARG_1, NULL, 0, &timeout_pos, &short_time); if (tres != 0) BIF_ERROR(BIF_P, BADARG); @@ -2566,7 +2566,7 @@ BIF_RETTYPE start_timer_4(BIF_ALIST_4) if (!parse_bif_timer_options(BIF_ARG_4, NULL, NULL, &abs, &accessor)) BIF_ERROR(BIF_P, BADARG); - tres = parse_timeout_pos(ERTS_PROC_GET_SCHDATA(BIF_P), BIF_ARG_1, NULL, + tres = parse_timeout_pos(erts_proc_sched_data(BIF_P), BIF_ARG_1, NULL, abs, &timeout_pos, &short_time); if (tres != 0) BIF_ERROR(BIF_P, BADARG); @@ -2720,7 +2720,7 @@ set_proc_timer_common(Process *c_p, ErtsSchedulerData *esdp, Sint64 tmo, int erts_set_proc_timer_term(Process *c_p, Eterm etmo) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); ErtsMonotonicTime tmo, timeout_pos; int short_time, tres; @@ -2742,7 +2742,7 @@ erts_set_proc_timer_term(Process *c_p, Eterm etmo) void erts_set_proc_timer_uword(Process *c_p, UWord tmo) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); ERTS_HLT_ASSERT(erts_smp_atomic_read_nob(&c_p->common.timer) == ERTS_PTMR_NONE); @@ -2776,7 +2776,7 @@ erts_cancel_proc_timer(Process *c_p) erts_smp_atomic_set_nob(&c_p->common.timer, ERTS_PTMR_NONE); return; } - continue_cancel_ptimer(ERTS_PROC_GET_SCHDATA(c_p), + continue_cancel_ptimer(erts_proc_sched_data(c_p), (ErtsTimer *) tval); } diff --git a/erts/emulator/beam/erl_init.c b/erts/emulator/beam/erl_init.c index fcd2739ac3..0649fb68de 100644 --- a/erts/emulator/beam/erl_init.c +++ b/erts/emulator/beam/erl_init.c @@ -164,6 +164,8 @@ int erts_use_sender_punish; Uint display_items; /* no of items to display in traces etc */ int H_MIN_SIZE; /* The minimum heap grain */ int BIN_VH_MIN_SIZE; /* The minimum binary virtual*/ +int H_MAX_SIZE; /* The maximum heap size */ +int H_MAX_FLAGS; /* The maximum heap flags */ Uint32 erts_debug_flags; /* Debug flags. */ int erts_backtrace_depth; /* How many functions to show in a backtrace @@ -576,6 +578,10 @@ void erts_usage(void) H_DEFAULT_SIZE); erts_fprintf(stderr, "-hmbs size set minimum binary virtual heap size in words (default %d)\n", VH_DEFAULT_SIZE); + erts_fprintf(stderr, "-hmax size set maximum heap size in words (default %d)\n", + H_DEFAULT_MAX_SIZE); + erts_fprintf(stderr, "-hmaxk bool enable or disable kill at max heap size (default true)\n"); + erts_fprintf(stderr, "-hmaxel bool enable or disable error_logger report at max heap size (default true)\n"); erts_fprintf(stderr, "-hpds size initial process dictionary size (default %d)\n", erts_pd_initial_size); erts_fprintf(stderr, "-hmqd val set default message queue data flag for processes,\n"); @@ -759,6 +765,8 @@ early_init(int *argc, char **argv) /* erts_async_thread_suggested_stack_size = ERTS_ASYNC_THREAD_MIN_STACK_SIZE; H_MIN_SIZE = H_DEFAULT_SIZE; BIN_VH_MIN_SIZE = VH_DEFAULT_SIZE; + H_MAX_SIZE = H_DEFAULT_MAX_SIZE; + H_MAX_FLAGS = MAX_HEAP_SIZE_KILL|MAX_HEAP_SIZE_LOG; erts_initialized = 0; @@ -1484,10 +1492,13 @@ erl_start(int argc, char **argv) char *sub_param = argv[i]+2; /* set default heap size * - * h|ms - min_heap_size - * h|mbs - min_bin_vheap_size - * h|pds - erts_pd_initial_size - * h|mqd - message_queue_data + * h|ms - min_heap_size + * h|mbs - min_bin_vheap_size + * h|pds - erts_pd_initial_size + * h|mqd - message_queue_data + * h|max - max_heap_size + * h|maxk - max_heap_kill + * h|maxel - max_heap_error_logger * */ if (has_prefix("mbs", sub_param)) { @@ -1530,6 +1541,41 @@ erl_start(int argc, char **argv) "Invalid message_queue_data flag: %s\n", arg); erts_usage(); } + } else if (has_prefix("maxk", sub_param)) { + arg = get_arg(sub_param+4, argv[i+1], &i); + if (strcmp(arg,"true") == 0) { + H_MAX_FLAGS |= MAX_HEAP_SIZE_KILL; + } else if (strcmp(arg,"false") == 0) { + H_MAX_FLAGS &= ~MAX_HEAP_SIZE_KILL; + } else { + erts_fprintf(stderr, "bad max heap kill %s\n", arg); + erts_usage(); + } + VERBOSE(DEBUG_SYSTEM, ("using max heap kill %d\n", H_MAX_FLAGS)); + } else if (has_prefix("maxel", sub_param)) { + arg = get_arg(sub_param+5, argv[i+1], &i); + if (strcmp(arg,"true") == 0) { + H_MAX_FLAGS |= MAX_HEAP_SIZE_LOG; + } else if (strcmp(arg,"false") == 0) { + H_MAX_FLAGS &= ~MAX_HEAP_SIZE_LOG; + } else { + erts_fprintf(stderr, "bad max heap error logger %s\n", arg); + erts_usage(); + } + VERBOSE(DEBUG_SYSTEM, ("using max heap log %d\n", H_MAX_FLAGS)); + } else if (has_prefix("max", sub_param)) { + arg = get_arg(sub_param+3, argv[i+1], &i); + if ((H_MAX_SIZE = atoi(arg)) < 0) { + erts_fprintf(stderr, "bad max heap size %s\n", arg); + erts_usage(); + } + if (H_MAX_SIZE < H_MIN_SIZE && H_MAX_SIZE) { + erts_fprintf(stderr, "max heap size (%s) is not allowed to be " + "smaller than min heap size (%d)\n", + arg, H_MIN_SIZE); + erts_usage(); + } + VERBOSE(DEBUG_SYSTEM, ("using max heap size %d\n", H_MAX_SIZE)); } else { /* backward compatibility */ arg = get_arg(argv[i]+2, argv[i+1], &i); diff --git a/erts/emulator/beam/erl_message.c b/erts/emulator/beam/erl_message.c index 9beff52835..579f6e427d 100644 --- a/erts/emulator/beam/erl_message.c +++ b/erts/emulator/beam/erl_message.c @@ -32,6 +32,7 @@ #include "erl_process.h" #include "erl_binary.h" #include "dtrace-wrapper.h" +#include "beam_bp.h" ERTS_SCHED_PREF_QUICK_ALLOC_IMPL(message_ref, ErtsMessageRef, @@ -251,9 +252,10 @@ erts_realloc_shrink_message(ErtsMessage *mp, Uint sz, Eterm *brefs, Uint brefs_s void erts_queue_dist_message(Process *rcvr, - ErtsProcLocks *rcvr_locks, + ErtsProcLocks rcvr_locks, ErtsDistExternal *dist_ext, - Eterm token) + Eterm token, + Eterm from) { ErtsMessage* mp; #ifdef USE_VM_PROBES @@ -265,7 +267,7 @@ erts_queue_dist_message(Process *rcvr, erts_aint_t state; #endif - ERTS_SMP_LC_ASSERT(*rcvr_locks == erts_proc_lc_my_proc_locks(rcvr)); + ERTS_SMP_LC_ASSERT(rcvr_locks == erts_proc_lc_my_proc_locks(rcvr)); mp = erts_alloc_message(0, NULL); mp->data.dist_ext = dist_ext; @@ -280,10 +282,10 @@ erts_queue_dist_message(Process *rcvr, ERL_MESSAGE_TOKEN(mp) = token; #ifdef ERTS_SMP - if (!(*rcvr_locks & ERTS_PROC_LOCK_MSGQ)) { + if (!(rcvr_locks & ERTS_PROC_LOCK_MSGQ)) { if (erts_smp_proc_trylock(rcvr, ERTS_PROC_LOCK_MSGQ) == EBUSY) { ErtsProcLocks need_locks = ERTS_PROC_LOCK_MSGQ; - if (*rcvr_locks & ERTS_PROC_LOCK_STATUS) { + if (rcvr_locks & ERTS_PROC_LOCK_STATUS) { erts_smp_proc_unlock(rcvr, ERTS_PROC_LOCK_STATUS); need_locks |= ERTS_PROC_LOCK_STATUS; } @@ -293,7 +295,7 @@ erts_queue_dist_message(Process *rcvr, state = erts_smp_atomic32_read_acqb(&rcvr->state); if (state & (ERTS_PSFLG_PENDING_EXIT|ERTS_PSFLG_EXITING)) { - if (!(*rcvr_locks & ERTS_PROC_LOCK_MSGQ)) + if (!(rcvr_locks & ERTS_PROC_LOCK_MSGQ)) erts_smp_proc_unlock(rcvr, ERTS_PROC_LOCK_MSGQ); /* Drop message if receiver is exiting or has a pending exit ... */ erts_cleanup_messages(mp); @@ -301,10 +303,13 @@ erts_queue_dist_message(Process *rcvr, else #endif if (IS_TRACED_FL(rcvr, F_TRACE_RECEIVE)) { + if (from == am_Empty) + from = dist_ext->dep->sysname; + /* Ahh... need to decode it in order to trace it... */ - if (!(*rcvr_locks & ERTS_PROC_LOCK_MSGQ)) + if (!(rcvr_locks & ERTS_PROC_LOCK_MSGQ)) erts_smp_proc_unlock(rcvr, ERTS_PROC_LOCK_MSGQ); - if (!erts_decode_dist_message(rcvr, *rcvr_locks, mp, 0)) + if (!erts_decode_dist_message(rcvr, rcvr_locks, mp, 0)) erts_free_message(mp); else { Eterm msg = ERL_MESSAGE_TERM(mp); @@ -324,7 +329,7 @@ erts_queue_dist_message(Process *rcvr, tok_label, tok_lastcnt, tok_serial); } #endif - erts_queue_message(rcvr, rcvr_locks, mp, msg); + erts_queue_message(rcvr, rcvr_locks, mp, msg, from); } } else { @@ -351,12 +356,12 @@ erts_queue_dist_message(Process *rcvr, LINK_MESSAGE(rcvr, mp, &mp->next, 1); - if (!(*rcvr_locks & ERTS_PROC_LOCK_MSGQ)) + if (!(rcvr_locks & ERTS_PROC_LOCK_MSGQ)) erts_smp_proc_unlock(rcvr, ERTS_PROC_LOCK_MSGQ); erts_proc_notify_new_message(rcvr, #ifdef ERTS_SMP - *rcvr_locks + rcvr_locks #else 0 #endif @@ -366,14 +371,15 @@ erts_queue_dist_message(Process *rcvr, /* Add messages last in message queue */ static Sint -queue_messages(Process *c_p, - Process* receiver, +queue_messages(Process* receiver, erts_aint32_t *receiver_state, - ErtsProcLocks *receiver_locks, + ErtsProcLocks receiver_locks, ErtsMessage* first, ErtsMessage** last, - Uint len) + Uint len, + Eterm from) { + ErtsTracingEvent* te; Sint res; int locked_msgq = 0; erts_aint32_t state; @@ -386,12 +392,12 @@ queue_messages(Process *c_p, #ifdef ERTS_SMP #ifdef ERTS_ENABLE_LOCK_CHECK ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(receiver) < ERTS_PROC_LOCK_MSGQ || - *receiver_locks == erts_proc_lc_my_proc_locks(receiver)); + receiver_locks == erts_proc_lc_my_proc_locks(receiver)); #endif - if (!(*receiver_locks & ERTS_PROC_LOCK_MSGQ)) { + if (!(receiver_locks & ERTS_PROC_LOCK_MSGQ)) { if (erts_smp_proc_trylock(receiver, ERTS_PROC_LOCK_MSGQ) == EBUSY) { - ErtsProcLocks need_locks = ERTS_PROC_LOCK_MSGQ; + ErtsProcLocks need_locks; if (receiver_state) state = *receiver_state; @@ -400,10 +406,11 @@ queue_messages(Process *c_p, if (state & (ERTS_PSFLG_EXITING|ERTS_PSFLG_PENDING_EXIT)) goto exiting; - if (*receiver_locks & ERTS_PROC_LOCK_STATUS) { - erts_smp_proc_unlock(receiver, ERTS_PROC_LOCK_STATUS); - need_locks |= ERTS_PROC_LOCK_STATUS; + need_locks = receiver_locks & (ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); + if (need_locks) { + erts_smp_proc_unlock(receiver, need_locks); } + need_locks |= ERTS_PROC_LOCK_MSGQ; erts_smp_proc_lock(receiver, need_locks); } locked_msgq = 1; @@ -426,7 +433,7 @@ queue_messages(Process *c_p, res = receiver->msg.len; #ifdef ERTS_SMP - if (*receiver_locks & ERTS_PROC_LOCK_MAIN) { + if (receiver_locks & ERTS_PROC_LOCK_MAIN) { /* * We move 'in queue' to 'private queue' and place * message at the end of 'private queue' in order @@ -445,7 +452,10 @@ queue_messages(Process *c_p, LINK_MESSAGE(receiver, first, last, len); } - if (IS_TRACED_FL(receiver, F_TRACE_RECEIVE)) { + if (IS_TRACED_FL(receiver, F_TRACE_RECEIVE) + && (te = &erts_receive_tracing[erts_active_bp_ix()], + te->on)) { + ErtsMessage *msg = first; #ifdef USE_VM_PROBES @@ -468,52 +478,50 @@ queue_messages(Process *c_p, tok_label, tok_lastcnt, tok_serial); } #endif - while (msg) { - trace_receive(receiver, ERL_MESSAGE_TERM(msg)); + trace_receive(receiver, from, ERL_MESSAGE_TERM(msg), te); msg = msg->next; } } - - if (locked_msgq) + if (locked_msgq) { erts_smp_proc_unlock(receiver, ERTS_PROC_LOCK_MSGQ); + } #ifdef ERTS_SMP - erts_proc_notify_new_message(receiver, *receiver_locks); + erts_proc_notify_new_message(receiver, receiver_locks); #else erts_proc_notify_new_message(receiver, 0); - ERTS_HOLE_CHECK(receiver); #endif return res; } static Sint -queue_message(Process *c_p, - Process* receiver, +queue_message(Process* receiver, erts_aint32_t *receiver_state, - ErtsProcLocks *receiver_locks, - ErtsMessage* mp, Eterm msg) + ErtsProcLocks receiver_locks, + ErtsMessage* mp, Eterm msg, Eterm from) { ERL_MESSAGE_TERM(mp) = msg; - return queue_messages(c_p, receiver, receiver_state, receiver_locks, - mp, &mp->next, 1 ); + return queue_messages(receiver, receiver_state, receiver_locks, + mp, &mp->next, 1, from); } Sint -erts_queue_message(Process* receiver, ErtsProcLocks *receiver_locks, - ErtsMessage* mp, Eterm msg) +erts_queue_message(Process* receiver, ErtsProcLocks receiver_locks, + ErtsMessage* mp, Eterm msg, Eterm from) { - return queue_message(NULL, receiver, NULL, receiver_locks, mp, msg); + return queue_message(receiver, NULL, receiver_locks, mp, msg, from); } Sint -erts_queue_messages(Process* receiver, ErtsProcLocks *receiver_locks, - ErtsMessage* first, ErtsMessage** last, Uint len) +erts_queue_messages(Process* receiver, ErtsProcLocks receiver_locks, + ErtsMessage* first, ErtsMessage** last, Uint len, + Eterm from) { - return queue_messages(NULL, receiver, NULL, receiver_locks, - first, last, len); + return queue_messages(receiver, NULL, receiver_locks, + first, last, len, from); } void @@ -592,7 +600,9 @@ erts_try_alloc_message_on_heap(Process *pp, ASSERT(!(*psp & ERTS_PSFLG_OFF_HEAP_MSGQ)); - if ( + if ((*psp) & ERTS_PSFLGS_VOLATILE_HEAP) + goto in_message_fragment; + else if ( #if defined(ERTS_SMP) *plp & ERTS_PROC_LOCK_MAIN #else @@ -602,7 +612,7 @@ erts_try_alloc_message_on_heap(Process *pp, #ifdef ERTS_SMP try_on_heap: #endif - if ((*psp & (ERTS_PSFLG_EXITING|ERTS_PSFLG_PENDING_EXIT)) + if (((*psp) & ERTS_PSFLGS_VOLATILE_HEAP) || (pp->flags & F_DISABLE_GC) || HEAP_LIMIT(pp) - HEAP_TOP(pp) <= sz) { /* @@ -830,11 +840,11 @@ erts_send_message(Process* sender, #ifdef USE_VM_PROBES ERL_MESSAGE_DT_UTAG(mp) = utag; #endif - res = queue_message(sender, - receiver, + res = queue_message(receiver, &receiver_state, - receiver_locks, - mp, message); + *receiver_locks, + mp, message, + sender->common.id); BM_SWAP_TIMER(send,system); @@ -891,7 +901,7 @@ erts_deliver_exit_message(Eterm from, Process *to, ErtsProcLocks *to_locksp, seq_trace_output(token, save, SEQ_TRACE_SEND, to->common.id, NULL); temptoken = copy_struct(token, sz_token, &hp, ohp); ERL_MESSAGE_TOKEN(mp) = temptoken; - erts_queue_message(to, to_locksp, mp, save); + erts_queue_message(to, *to_locksp, mp, save, am_system); } else { sz_from = IS_CONST(from) ? 0 : size_object(from); #ifdef SHCOPY_SEND @@ -913,7 +923,7 @@ erts_deliver_exit_message(Eterm from, Process *to, ErtsProcLocks *to_locksp, ? from : copy_struct(from, sz_from, &hp, ohp)); save = TUPLE3(hp, am_EXIT, from_copy, mess); - erts_queue_message(to, to_locksp, mp, save); + erts_queue_message(to, *to_locksp, mp, save, am_system); } } diff --git a/erts/emulator/beam/erl_message.h b/erts/emulator/beam/erl_message.h index 608cf552a2..851ac37fda 100644 --- a/erts/emulator/beam/erl_message.h +++ b/erts/emulator/beam/erl_message.h @@ -295,10 +295,10 @@ ErlHeapFragment* new_message_buffer(Uint); ErlHeapFragment* erts_resize_message_buffer(ErlHeapFragment *, Uint, Eterm *, Uint); void free_message_buffer(ErlHeapFragment *); -void erts_queue_dist_message(Process*, ErtsProcLocks*, ErtsDistExternal *, Eterm); -Sint erts_queue_message(Process*, ErtsProcLocks*,ErtsMessage*, Eterm); -Sint erts_queue_messages(Process*, ErtsProcLocks*, - ErtsMessage*, ErtsMessage**, Uint); +void erts_queue_dist_message(Process*, ErtsProcLocks, ErtsDistExternal *, Eterm, Eterm); +Sint erts_queue_message(Process*, ErtsProcLocks,ErtsMessage*, Eterm, Eterm); +Sint erts_queue_messages(Process*, ErtsProcLocks, + ErtsMessage*, ErtsMessage**, Uint, Eterm); void erts_deliver_exit_message(Eterm, Process*, ErtsProcLocks *, Eterm, Eterm); Sint erts_send_message(Process*, Process*, ErtsProcLocks*, Eterm, unsigned); void erts_link_mbuf_to_proc(Process *proc, ErlHeapFragment *bp); diff --git a/erts/emulator/beam/erl_msacc.c b/erts/emulator/beam/erl_msacc.c index d0f305900a..544bc8b983 100644 --- a/erts/emulator/beam/erl_msacc.c +++ b/erts/emulator/beam/erl_msacc.c @@ -257,7 +257,7 @@ static void send_reply(ErtsMsAcc *msacc, ErtsMSAccReq *msaccrp) { if (msacc->unmanaged) erts_mtx_unlock(&msacc->mtx); - erts_queue_message(rp, &rp_locks, msgp, msg); + erts_queue_message(rp, rp_locks, msgp, msg, am_system); if (esdp && msaccrp->req_sched == esdp->no) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -338,7 +338,7 @@ erts_msacc_request(Process *c_p, int action, Eterm *threads) { #ifdef ERTS_ENABLE_MSACC ErtsMsAcc *msacc = ERTS_MSACC_TSD_GET(); - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); Eterm ref; ErtsMSAccReq *msaccrp; Eterm *hp; diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index dc83a780a2..159dc66ad5 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -20,6 +20,23 @@ /* Erlang Native InterFace */ +/* + * Environment contains a pointer to currently executing process. + * In the dirty case this pointer do however not point to the + * actual process structure of the executing process, but instead + * a "shadow process structure". This in order to be able to handle + * heap allocation without the need to acquire the main lock on + * the process. + * + * The dirty process is allowed to allocate on the heap without + * the main lock, i.e., incrementing htop, but is not allowed to + * modify mbuf, offheap, etc without the main lock. The dirty + * process moves mbuf list and offheap list of the shadow process + * structure into the real structure when the dirty nif call + * completes. + */ + + #ifdef HAVE_CONFIG_H # include "config.h" #endif @@ -81,6 +98,43 @@ void dtrace_nifenv_str(ErlNifEnv *, char *); #define MIN_HEAP_FRAG_SZ 200 static Eterm* alloc_heap_heavy(ErlNifEnv* env, unsigned need, Eterm* hp); +static ERTS_INLINE int +is_scheduler(void) +{ + ErtsSchedulerData *esdp = erts_get_scheduler_data(); + if (!esdp) + return 0; + if (ERTS_SCHEDULER_IS_DIRTY(esdp)) + return -1; + return 1; +} + +static ERTS_INLINE void +execution_state(ErlNifEnv *env, Process **c_pp, int *schedp) +{ + if (schedp) + *schedp = is_scheduler(); + if (c_pp) { + if (!env || env->proc->common.id == ERTS_INVALID_PID) + *c_pp = NULL; + else { + Process *c_p = env->proc; + + if (!(c_p->static_flags & ERTS_STC_FLG_SHADOW_PROC)) + ASSERT(is_scheduler() > 0); + else { + c_p = env->proc->next; + ASSERT(is_scheduler() < 0); + ASSERT(c_p && env->proc->common.id == c_p->common.id); + } + + *c_pp = c_p; + + ASSERT(!(c_p->static_flags & ERTS_STC_FLG_SHADOW_PROC)); + } + } +} + static ERTS_INLINE Eterm* alloc_heap(ErlNifEnv* env, unsigned need) { Eterm* hp = env->hp; @@ -124,6 +178,9 @@ static ERTS_INLINE void ensure_heap(ErlNifEnv* env, unsigned may_need) void erts_pre_nif(ErlNifEnv* env, Process* p, struct erl_module_nif* mod_nif, Process* tracee) { +#ifdef ERTS_DIRTY_SCHEDULERS + ErtsSchedulerData *esdp; +#endif env->mod_nif = mod_nif; env->proc = p; env->hp = HEAP_TOP(p); @@ -133,6 +190,61 @@ void erts_pre_nif(ErlNifEnv* env, Process* p, struct erl_module_nif* mod_nif, env->tmp_obj_list = NULL; env->exception_thrown = 0; env->tracee = tracee; + + ASSERT(p->common.id != ERTS_INVALID_PID); + +#ifdef ERTS_DIRTY_SCHEDULERS + esdp = erts_get_scheduler_data(); + ASSERT(esdp); + + if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { +#ifdef DEBUG + erts_aint32_t state = erts_smp_atomic32_read_nob(&p->state); + + ASSERT(p->scheduler_data == esdp); + ASSERT((state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS)) + && !(state & (ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS))); +#endif + + } + else { + Process *sproc; +#ifdef DEBUG + erts_aint32_t state = erts_smp_atomic32_read_nob(&p->state); + + ASSERT(!p->scheduler_data); + ASSERT((state & ERTS_PSFLG_DIRTY_RUNNING) + && !(state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS))); +#endif + + sproc = esdp->dirty_shadow_process; + ASSERT(sproc); + ASSERT(sproc->static_flags & ERTS_STC_FLG_SHADOW_PROC); + ASSERT(erts_smp_atomic32_read_nob(&sproc->state) + == (ERTS_PSFLG_ACTIVE + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_PROXY)); + + sproc->next = p; + sproc->common.id = p->common.id; + sproc->htop = p->htop; + sproc->stop = p->stop; + sproc->hend = p->hend; + sproc->heap = p->heap; + sproc->abandoned_heap = p->abandoned_heap; + sproc->heap_sz = p->heap_sz; + sproc->high_water = p->high_water; + sproc->old_hend = p->old_hend; + sproc->old_htop = p->old_htop; + sproc->old_heap = p->old_heap; + sproc->mbuf = NULL; + sproc->mbuf_sz = 0; + ERTS_INIT_OFF_HEAP(&sproc->off_heap); + env->proc = sproc; + } +#endif } /* Temporary object header, auto-deallocated when NIF returns @@ -157,18 +269,75 @@ static ERTS_INLINE void free_tmp_objs(ErlNifEnv* env) void erts_post_nif(ErlNifEnv* env) { erts_unblock_fpe(env->fpe_was_unmasked); - if (env->heap_frag == NULL) { - ASSERT(env->hp_end == HEAP_LIMIT(env->proc)); - ASSERT(env->hp >= HEAP_TOP(env->proc)); - ASSERT(env->hp <= HEAP_LIMIT(env->proc)); - HEAP_TOP(env->proc) = env->hp; + +#ifdef ERTS_DIRTY_SCHEDULERS + if (!(env->proc->static_flags & ERTS_STC_FLG_SHADOW_PROC)) +#endif + { + ASSERT(is_scheduler() > 0); + if (env->heap_frag == NULL) { + ASSERT(env->hp_end == HEAP_LIMIT(env->proc)); + ASSERT(env->hp >= HEAP_TOP(env->proc)); + ASSERT(env->hp <= HEAP_LIMIT(env->proc)); + HEAP_TOP(env->proc) = env->hp; + } + else { + ASSERT(env->hp_end != HEAP_LIMIT(env->proc)); + ASSERT(env->hp_end - env->hp <= env->heap_frag->alloc_size); + env->heap_frag->used_size = env->hp - env->heap_frag->mem; + ASSERT(env->heap_frag->used_size <= env->heap_frag->alloc_size); + } + env->exiting = ERTS_PROC_IS_EXITING(env->proc); } - else { - ASSERT(env->hp_end != HEAP_LIMIT(env->proc)); - ASSERT(env->hp_end - env->hp <= env->heap_frag->alloc_size); - env->heap_frag->used_size = env->hp - env->heap_frag->mem; - ASSERT(env->heap_frag->used_size <= env->heap_frag->alloc_size); +#ifdef ERTS_DIRTY_SCHEDULERS + else { /* Dirty nif call using shadow process struct */ + Process *c_p = env->proc->next; + + ASSERT(is_scheduler() < 0); + ASSERT(env->proc->common.id == c_p->common.id); + + if (!env->heap_frag) { + ASSERT(env->hp_end == HEAP_LIMIT(c_p)); + ASSERT(env->hp >= HEAP_TOP(c_p)); + ASSERT(env->hp <= HEAP_LIMIT(c_p)); + HEAP_TOP(c_p) = env->hp; + } + else { + ASSERT(env->hp_end != HEAP_LIMIT(c_p)); + ASSERT(env->hp_end - env->hp <= env->heap_frag->alloc_size); + + HEAP_TOP(c_p) = HEAP_TOP(env->proc); + env->heap_frag->used_size = env->hp - env->heap_frag->mem; + + ASSERT(env->heap_frag->used_size <= env->heap_frag->alloc_size); + + if (c_p->mbuf) { + ErlHeapFragment *bp; + for (bp = env->proc->mbuf; bp->next; bp = bp->next) + ; + bp->next = c_p->mbuf; + } + + c_p->mbuf = env->proc->mbuf; + c_p->mbuf_sz += env->proc->mbuf_sz; + + } + + if (!c_p->off_heap.first) + c_p->off_heap.first = env->proc->off_heap.first; + else if (env->proc->off_heap.first) { + struct erl_off_heap_header *ohhp; + for (ohhp = env->proc->off_heap.first; ohhp->next; ohhp = ohhp->next) + ; + ohhp->next = c_p->off_heap.first; + c_p->off_heap.first = env->proc->off_heap.first; + } + c_p->off_heap.overhead += env->proc->off_heap.overhead; + + env->exiting = ERTS_PROC_IS_EXITING(c_p); + BUMP_ALL_REDS(c_p); } +#endif free_tmp_objs(env); } @@ -366,7 +535,7 @@ int erts_flush_trace_messages(Process *c_p, ErtsProcLocks c_p_locks) rp_locks = 0; if (rp->common.id == c_p->common.id) rp_locks = c_p_locks; - erts_queue_messages(rp, &rp_locks, first, last, len); + erts_queue_messages(rp, rp_locks, first, last, len, c_p->common.id); if (rp->common.id == c_p->common.id) rp_locks &= ~c_p_locks; if (rp_locks) @@ -400,44 +569,44 @@ error: #endif -int -enif_send(ErlNifEnv* env, const ErlNifPid* to_pid, - ErlNifEnv* msg_env, ERL_NIF_TERM msg) +int enif_send(ErlNifEnv* env, const ErlNifPid* to_pid, + ErlNifEnv* msg_env, ERL_NIF_TERM msg) { struct enif_msg_environment_t* menv = (struct enif_msg_environment_t*)msg_env; - ErtsProcLocks rp_locks = 0, lc_locks = 0, c_p_locks = ERTS_PROC_LOCK_MAIN; + ErtsProcLocks rp_locks = 0; +#ifdef ERTS_SMP + ErtsProcLocks lc_locks = 0; +#endif Process* rp; Process* c_p; ErtsMessage *mp; Eterm receiver = to_pid->pid; - int flush_me = 0; - ErtsSchedulerData *esdp = erts_get_scheduler_data(); - int scheduler = esdp ? esdp->no : 0; + int scheduler; - if (env != NULL) { - c_p = env->proc; - if (receiver == c_p->common.id) { - rp_locks = c_p_locks; - flush_me = 1; - } + execution_state(env, &c_p, &scheduler); + +#ifndef ERTS_SMP + if (!scheduler) { + erts_exit(ERTS_ABORT_EXIT, + "enif_send: called from non-scheduler thread on non-SMP VM"); + return 0; + } +#endif + + if (scheduler > 0) { /* Normal scheduler */ + rp = erts_proc_lookup(receiver); + if (c_p == rp) + rp_locks = ERTS_PROC_LOCK_MAIN; } else { -#ifdef ERTS_SMP - c_p = NULL; -#else - erts_exit(ERTS_ABORT_EXIT,"enif_send: env==NULL on non-SMP VM"); -#endif + if (c_p && ERTS_PROC_IS_EXITING(c_p)) + return 0; + rp = erts_pid2proc_opt(c_p, 0, receiver, rp_locks, + ERTS_P2P_FLG_INC_REFC); } - - rp = (scheduler - ? erts_proc_lookup(receiver) - : erts_pid2proc_opt(c_p, ERTS_PROC_LOCK_MAIN, - receiver, rp_locks, ERTS_P2P_FLG_INC_REFC)); - - if (rp == NULL) { - ASSERT(env == NULL || receiver != c_p->common.id); + if (rp == NULL) return 0; - } + if (menv) { flush_env(msg_env); mp = erts_alloc_message(0, NULL); @@ -462,22 +631,12 @@ enif_send(ErlNifEnv* env, const ErlNifPid* to_pid, ERL_MESSAGE_TERM(mp) = msg; - if (flush_me) { - flush_env(env); /* Needed for ERTS_HOLE_CHECK */ - } - if (!env || !env->tracee) { if (c_p && IS_TRACED_FL(c_p, F_TRACE_SEND)) trace_send(c_p, receiver, msg); - -#ifndef ERTS_SMP } -#endif - - erts_queue_message(rp, &rp_locks, mp, msg); #ifdef ERTS_SMP - } else { /* This clause is taken when the nif is called in the context of a traced process. We do not know which locks we have @@ -502,8 +661,6 @@ enif_send(ErlNifEnv* env, const ErlNifPid* to_pid, #ifdef ERTS_ENABLE_LOCK_CHECK lc_locks = erts_proc_lc_my_proc_locks(rp); rp_locks |= lc_locks; - if (receiver == c_p->common.id) - c_p_locks |= lc_locks; #endif if (ERTS_FORCE_ENIF_SEND_DELAY() || msgq || rp_locks & ERTS_PROC_LOCK_MSGQ || @@ -532,24 +689,28 @@ enif_send(ErlNifEnv* env, const ErlNifPid* to_pid, msgq->last = &mp->next; erts_smp_proc_unlock(t_p, ERTS_PROC_LOCK_TRACE); } + goto done; } else { erts_smp_proc_unlock(t_p, ERTS_PROC_LOCK_TRACE); rp_locks &= ~ERTS_PROC_LOCK_TRACE; rp_locks |= ERTS_PROC_LOCK_MSGQ; - erts_queue_message(rp, &rp_locks, mp, msg); } } -#endif +#endif /* ERTS_SMP */ + erts_queue_message(rp, rp_locks, mp, msg, + c_p ? c_p->common.id : am_undefined); + +#ifdef ERTS_SMP +done: if (c_p == rp) rp_locks &= ~ERTS_PROC_LOCK_MAIN; if (rp_locks & ~lc_locks) erts_smp_proc_unlock(rp, rp_locks & ~lc_locks); - if (!scheduler) +#endif + if (scheduler <= 0) erts_proc_dec_refc(rp); - if (flush_me) { - cache_env(env); - } + return 1; } @@ -557,26 +718,52 @@ int enif_port_command(ErlNifEnv *env, const ErlNifPort* to_port, ErlNifEnv *msg_env, ERL_NIF_TERM msg) { - - ErtsSchedulerData *esdp = erts_get_scheduler_data(); - int scheduler = esdp ? esdp->no : 0; + int iflags = (erts_port_synchronous_ops + ? ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP + : ERTS_PORT_SFLGS_INVALID_LOOKUP); + int scheduler; + Process *c_p; Port *prt; + int res; - if (scheduler == 0 || !env) - return 0; + if (!env) + erts_exit(ERTS_ABORT_EXIT, "enif_port_command: env == NULL"); + + execution_state(env, &c_p, &scheduler); - prt = erts_port_lookup(to_port->port_id, - (erts_port_synchronous_ops - ? ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP - : ERTS_PORT_SFLGS_INVALID_LOOKUP)); + if (!c_p) + c_p = env->proc; + + if (scheduler > 0) + prt = erts_port_lookup(to_port->port_id, iflags); +#ifdef ERTS_DIRTY_SCHEDULERS + else if (scheduler < 0) { + if (ERTS_PROC_IS_EXITING(c_p)) + return 0; + prt = erts_thr_port_lookup(to_port->port_id, iflags); + } +#endif + else { + erts_exit(ERTS_ABORT_EXIT, "enif_port_command: " + "called from non-scheduler thread"); + } if (!prt) - return 0; + res = 0; + else { - if (IS_TRACED_FL(prt, F_TRACE_RECEIVE)) - trace_port_receive(prt, env->proc->common.id, am_command, msg); + if (IS_TRACED_FL(prt, F_TRACE_RECEIVE)) + trace_port_receive(prt, c_p->common.id, am_command, msg); - return erts_port_output_async(prt, env->proc->common.id, msg); + res = erts_port_output_async(prt, c_p->common.id, msg); + } + +#ifdef ERTS_DIRTY_SCHEDULERS + if (scheduler < 0) + erts_port_dec_refc(prt); +#endif + + return res; } ERL_NIF_TERM enif_make_copy(ErlNifEnv* dst_env, ERL_NIF_TERM src_term) @@ -1038,15 +1225,21 @@ Eterm enif_make_badarg(ErlNifEnv* env) Eterm enif_raise_exception(ErlNifEnv* env, ERL_NIF_TERM reason) { + Process *c_p; + + execution_state(env, &c_p, NULL); + env->exception_thrown = 1; - env->proc->fvalue = reason; - BIF_ERROR(env->proc, EXC_ERROR); + c_p->fvalue = reason; + BIF_ERROR(c_p, EXC_ERROR); } int enif_has_pending_exception(ErlNifEnv* env, ERL_NIF_TERM* reason) { if (env->exception_thrown && reason != NULL) { - *reason = env->proc->fvalue; + Process *c_p; + execution_state(env, &c_p, NULL); + *reason = c_p->fvalue; } return env->exception_thrown; } @@ -1440,56 +1633,71 @@ int enif_make_reverse_list(ErlNifEnv* env, ERL_NIF_TERM term, ERL_NIF_TERM *list return 1; } +int enif_is_current_process_alive(ErlNifEnv* env) +{ + Process *c_p; + int scheduler; + + execution_state(env, &c_p, &scheduler); + + if (!c_p) + erts_exit(ERTS_ABORT_EXIT, + "enif_is_current_process_alive: " + "Invalid environment"); + + if (!scheduler) + erts_exit(ERTS_ABORT_EXIT, "enif_is_current_process_alive: " + "called from non-scheduler thread"); + + return !ERTS_PROC_IS_EXITING(c_p); +} + int enif_is_process_alive(ErlNifEnv* env, ErlNifPid *proc) { - ErtsProcLocks rp_locks = 0; /* We don't need any locks, - just to check if it is alive */ - Eterm target = proc->pid; - Process* rp; - Process* c_p; - int scheduler = erts_get_scheduler_id() != 0; + int scheduler; - if (env != NULL) { - c_p = env->proc; - if (target == c_p->common.id) { - /* We are alive! */ - return 1; - } - } + execution_state(env, NULL, &scheduler); + + if (scheduler > 0) + return !!erts_proc_lookup(proc->pid); else { #ifdef ERTS_SMP - c_p = NULL; + Process* rp = erts_pid2proc_opt(NULL, 0, proc->pid, 0, + ERTS_P2P_FLG_INC_REFC); + if (rp) + erts_proc_dec_refc(rp); + return !!rp; #else - erts_exit(ERTS_ABORT_EXIT,"enif_is_process_alive: " - "env==NULL on non-SMP VM"); -#endif - } - - rp = (scheduler - ? erts_proc_lookup(target) - : erts_pid2proc_opt(c_p, ERTS_PROC_LOCK_MAIN, - target, rp_locks, ERTS_P2P_FLG_INC_REFC)); - if (rp == NULL) { - ASSERT(env == NULL || target != c_p->common.id); + erts_exit(ERTS_ABORT_EXIT, "enif_is_process_alive: " + "called from non-scheduler thread"); return 0; - } else { - if (!scheduler) - erts_proc_dec_refc(rp); - return 1; +#endif } } int enif_is_port_alive(ErlNifEnv *env, ErlNifPort *port) { - /* only allowed if called from scheduler */ - if (erts_get_scheduler_id() == 0) - erts_exit(ERTS_ABORT_EXIT,"enif_is_port_alive: called from non-scheduler"); + int scheduler; + Uint32 iflags = (erts_port_synchronous_ops + ? ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP + : ERTS_PORT_SFLGS_INVALID_LOOKUP); + + execution_state(env, NULL, &scheduler); - return erts_port_lookup( - port->port_id, - (erts_port_synchronous_ops - ? ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP - : ERTS_PORT_SFLGS_INVALID_LOOKUP)) != NULL; + if (scheduler > 0) + return !!erts_port_lookup(port->port_id, iflags); + else { +#ifdef ERTS_SMP + Port *prt = erts_thr_port_lookup(port->port_id, iflags); + if (prt) + erts_port_dec_refc(prt); + return !!prt; +#else + erts_exit(ERTS_ABORT_EXIT, "enif_is_port_alive: " + "called from non-scheduler thread"); + return 0; +#endif + } } ERL_NIF_TERM @@ -1602,6 +1810,16 @@ int enif_fprintf(void* filep, const char* format, ...) return ret; } +int enif_snprintf(char *buffer, size_t size, const char* format, ...) +{ + int ret; + va_list arglist; + va_start(arglist, format); + ret = erts_vsnprintf(buffer, size, format, arglist); + va_end(arglist); + return ret; +} + /*********************************************************** ** Memory managed (GC'ed) "resource" objects ** ***********************************************************/ @@ -1956,16 +2174,19 @@ void* enif_dlsym(void* handle, const char* symbol, int enif_consume_timeslice(ErlNifEnv* env, int percent) { + Process *proc; Sint reds; + execution_state(env, &proc, NULL); + ASSERT(is_proc_bound(env) && percent >= 1 && percent <= 100); if (percent < 1) percent = 1; else if (percent > 100) percent = 100; reds = ((CONTEXT_REDS+99) / 100) * percent; ASSERT(reds > 0 && reds <= CONTEXT_REDS); - BUMP_REDS(env->proc, reds); - return ERTS_BIF_REDS_LEFT(env->proc) == 0; + BUMP_REDS(proc, reds); + return ERTS_BIF_REDS_LEFT(proc) == 0; } /* @@ -2060,10 +2281,19 @@ static ERL_NIF_TERM init_nif_sched_data(ErlNifEnv* env, NativeFunPtr direct_fp, NativeFunPtr indirect_fp, int need_save, int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; - Eterm* reg = ERTS_PROC_GET_SCHDATA(proc)->x_reg_array; + Process* proc; + Eterm* reg; NifExport* ep; - int i; + int i, scheduler; + + execution_state(env, &proc, &scheduler); + + ASSERT(scheduler); + + ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(proc) + & ERTS_PROC_LOCK_MAIN); + + reg = erts_proc_sched_data(proc)->x_reg_array; ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); if (!ep) @@ -2075,12 +2305,13 @@ init_nif_sched_data(ErlNifEnv* env, NativeFunPtr direct_fp, NativeFunPtr indirec } if (env->exception_thrown) { ep->exception_thrown = 1; - ep->rootset[0] = env->proc->fvalue; + ep->rootset[0] = proc->fvalue; } else { ep->exception_thrown = 0; ep->rootset[0] = NIL; } - ERTS_VBUMP_ALL_REDS(proc); + if (scheduler > 0) + ERTS_VBUMP_ALL_REDS(proc); for (i = 0; i < argc; i++) { if (need_save) ep->rootset[i+1] = reg[i]; @@ -2112,7 +2343,12 @@ static void restore_nif_mfa(Process* proc, NifExport* ep, int exception) { int i; - Eterm* reg = ERTS_PROC_GET_SCHDATA(proc)->x_reg_array; + Eterm* reg = erts_proc_sched_data(proc)->x_reg_array; + + ERTS_SMP_LC_ASSERT(!(proc->static_flags + & ERTS_STC_FLG_SHADOW_PROC)); + ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(proc) + & ERTS_PROC_LOCK_MAIN); proc->current[0] = ep->saved_mfa[0]; proc->current[1] = ep->saved_mfa[1]; @@ -2137,11 +2373,13 @@ restore_nif_mfa(Process* proc, NifExport* ep, int exception) static ERL_NIF_TERM dirty_nif_finalizer(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; + Process* proc; NifExport* ep; + execution_state(env, &proc, NULL); + ASSERT(argc == 1); - ASSERT(!ERTS_SCHEDULER_IS_DIRTY(env->proc->scheduler_data)); + ASSERT(!ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(proc))); ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); ASSERT(ep); ASSERT(!ep->exception_thrown); @@ -2156,10 +2394,12 @@ dirty_nif_finalizer(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) static ERL_NIF_TERM dirty_nif_exception(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; + Process* proc; NifExport* ep; - ASSERT(!ERTS_SCHEDULER_IS_DIRTY(env->proc->scheduler_data)); + execution_state(env, &proc, NULL); + + ASSERT(!ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(proc))); ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); ASSERT(ep); ASSERT(ep->exception_thrown); @@ -2176,23 +2416,32 @@ dirty_nif_exception(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) static ERL_NIF_TERM execute_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; - NativeFunPtr fp = (NativeFunPtr) proc->current[6]; + Process* proc; + NativeFunPtr fp; NifExport* ep; ERL_NIF_TERM result; - ASSERT(ERTS_SCHEDULER_IS_DIRTY(env->proc->scheduler_data)); + execution_state(env, &proc, NULL); + + fp = (NativeFunPtr) proc->current[6]; + + ASSERT(ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(proc))); /* * Set ep->fp to NULL before the native call so we know later whether it scheduled another NIF for execution */ ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); - ASSERT(ep); + ASSERT(ep && fp); ep->fp = NULL; erts_smp_atomic32_read_band_mb(&proc->state, ~(ERTS_PSFLG_DIRTY_CPU_PROC | ERTS_PSFLG_DIRTY_IO_PROC)); + + erts_smp_proc_unlock(proc, ERTS_PROC_LOCK_MAIN); + result = (*fp)(env, argc, argv); + erts_smp_proc_lock(proc, ERTS_PROC_LOCK_MAIN); + if (erts_refc_dectest(&env->mod_nif->rt_dtor_cnt, 0) == 0 && env->mod_nif->mod == NULL) close_lib(env->mod_nif); /* @@ -2229,29 +2478,49 @@ execute_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) static ERTS_INLINE ERL_NIF_TERM schedule_dirty_nif(ErlNifEnv* env, int flags, int argc, const ERL_NIF_TERM argv[]) { - erts_aint32_t state, n, a; - Process* proc = env->proc; - NativeFunPtr fp = (NativeFunPtr) proc->current[6]; + ERL_NIF_TERM result; + erts_aint32_t act, dirty_flag; + Process* proc; + NativeFunPtr fp; NifExport* ep; - int need_save; + int need_save, scheduler; + + execution_state(env, &proc, &scheduler); + if (scheduler <= 0) { + ASSERT(scheduler < 0); + erts_smp_proc_lock(proc, ERTS_PROC_LOCK_MAIN); + } + + fp = (NativeFunPtr) proc->current[6]; + + ASSERT(fp); ASSERT(flags==ERL_NIF_DIRTY_JOB_IO_BOUND || flags==ERL_NIF_DIRTY_JOB_CPU_BOUND); - a = erts_smp_atomic32_read_acqb(&proc->state); - while (1) { - n = state = a; + if (flags == ERL_NIF_DIRTY_JOB_CPU_BOUND) + dirty_flag = ERTS_PSFLG_DIRTY_CPU_PROC; + else + dirty_flag = ERTS_PSFLG_DIRTY_IO_PROC; + + act = erts_smp_atomic32_read_bor_nob(&proc->state, dirty_flag); + if (!(act & (ERTS_PSFLG_DIRTY_CPU_PROC|ERTS_PSFLG_DIRTY_IO_PROC))) + erts_refc_inc(&env->mod_nif->rt_dtor_cnt, 1); + else if ((act & (ERTS_PSFLG_DIRTY_CPU_PROC + | ERTS_PSFLG_DIRTY_IO_PROC)) & ~dirty_flag) { + /* clear other flag... */ if (flags == ERL_NIF_DIRTY_JOB_CPU_BOUND) - n |= ERTS_PSFLG_DIRTY_CPU_PROC; + dirty_flag = ERTS_PSFLG_DIRTY_IO_PROC; else - n |= ERTS_PSFLG_DIRTY_IO_PROC; - a = erts_smp_atomic32_cmpxchg_mb(&proc->state, n, state); - if (a == state) - break; + dirty_flag = ERTS_PSFLG_DIRTY_CPU_PROC; + erts_smp_atomic32_read_band_nob(&proc->state, ~dirty_flag); } - erts_refc_inc(&env->mod_nif->rt_dtor_cnt, 1); + ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); need_save = (ep == NULL || is_non_value(ep->saved_mfa[0])); - return init_nif_sched_data(env, execute_dirty_nif, fp, need_save, argc, argv); + result = init_nif_sched_data(env, execute_dirty_nif, fp, need_save, argc, argv); + if (scheduler <= 0) + erts_smp_proc_unlock(proc, ERTS_PROC_LOCK_MAIN); + return result; } static ERL_NIF_TERM @@ -2276,11 +2545,14 @@ schedule_dirty_cpu_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) static ERL_NIF_TERM execute_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; - NativeFunPtr fp = (NativeFunPtr) proc->current[6]; + Process* proc; + NativeFunPtr fp; NifExport* ep; ERL_NIF_TERM result; + execution_state(env, &proc, NULL); + fp = (NativeFunPtr) proc->current[6]; + ASSERT(!env->exception_thrown); ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); ASSERT(ep); @@ -2303,10 +2575,10 @@ enif_schedule_nif(ErlNifEnv* env, const char* fun_name, int flags, ERL_NIF_TERM (*fp)(ErlNifEnv*, int, const ERL_NIF_TERM[]), int argc, const ERL_NIF_TERM argv[]) { - Process* proc = env->proc; + Process* proc; NifExport* ep; ERL_NIF_TERM fun_name_atom, result; - int need_save; + int need_save, scheduler; if (argc > MAX_ARG) return enif_make_badarg(env); @@ -2314,6 +2586,13 @@ enif_schedule_nif(ErlNifEnv* env, const char* fun_name, int flags, if (enif_is_exception(env, fun_name_atom)) return fun_name_atom; + execution_state(env, &proc, &scheduler); + if (scheduler <= 0) { + if (scheduler == 0) + enif_make_badarg(env); + erts_smp_proc_lock(proc, ERTS_PROC_LOCK_MAIN); + } + ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); need_save = (ep == NULL || is_non_value(ep->saved_mfa[0])); @@ -2325,12 +2604,15 @@ enif_schedule_nif(ErlNifEnv* env, const char* fun_name, int flags, sched_fun = schedule_dirty_io_nif; else if (chkflgs == ERL_NIF_DIRTY_JOB_CPU_BOUND) sched_fun = schedule_dirty_cpu_nif; - else - return enif_make_badarg(env); + else { + result = enif_make_badarg(env); + goto done; + } result = init_nif_sched_data(env, sched_fun, fp, need_save, argc, argv); #else - return enif_make_badarg(env); + result = enif_make_badarg(env); #endif + goto done; } else result = init_nif_sched_data(env, execute_nif, fp, need_save, argc, argv); @@ -2338,18 +2620,28 @@ enif_schedule_nif(ErlNifEnv* env, const char* fun_name, int flags, ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); ASSERT(ep); ep->exp.code[1] = (BeamInstr) fun_name_atom; + +done: + if (scheduler < 0) + erts_smp_proc_unlock(proc, ERTS_PROC_LOCK_MAIN); + return result; } -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT - int enif_is_on_dirty_scheduler(ErlNifEnv* env) { - return ERTS_SCHEDULER_IS_DIRTY(env->proc->scheduler_data); -} + int scheduler; + Process *c_p; -#endif /* ERL_NIF_DIRTY_SCHEDULER_SUPPORT */ + execution_state(env, &c_p, &scheduler); + + if (!c_p || !scheduler) + erts_exit(ERTS_ABORT_EXIT, "enif_is_on_dirty_scheduler: " + "Invalid env"); + + return scheduler < 0; +} /* Maps */ @@ -3050,16 +3342,16 @@ BIF_RETTYPE load_nif_2(BIF_ALIST_2) (BeamInstr) BeamOp(op_i_generic_breakpoint)); g->orig_instr = (BeamInstr) BeamOp(op_call_nif); } +#ifdef ERTS_DIRTY_SCHEDULERS if ((entry->major > 2 || (entry->major == 2 && entry->minor >= 7)) && (entry->options & ERL_NIF_DIRTY_NIF_OPTION) && f->flags) { -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT code_ptr[5+3] = (BeamInstr) f->fptr; code_ptr[5+1] = (f->flags == ERL_NIF_DIRTY_JOB_IO_BOUND) ? (BeamInstr) schedule_dirty_io_nif : (BeamInstr) schedule_dirty_cpu_nif; -#endif } else +#endif code_ptr[5+1] = (BeamInstr) f->fptr; code_ptr[5+2] = (BeamInstr) lib; f = next_func(entry, &incr, f); diff --git a/erts/emulator/beam/erl_nif.h b/erts/emulator/beam/erl_nif.h index 3964f7f679..da7a754757 100644 --- a/erts/emulator/beam/erl_nif.h +++ b/erts/emulator/beam/erl_nif.h @@ -28,7 +28,6 @@ # include "config.h" #endif -#include "erl_native_features_config.h" #include "erl_drv_nif.h" /* Version history: @@ -50,9 +49,10 @@ ** 2.8: 18.0 add enif_has_pending_exception ** 2.9: 18.2 enif_getenv ** 2.10: Time API +** 2.11: 19.0 enif_snprintf */ #define ERL_NIF_MAJOR_VERSION 2 -#define ERL_NIF_MINOR_VERSION 10 +#define ERL_NIF_MINOR_VERSION 11 /* * The emulator will refuse to load a nif-lib with a major version @@ -167,13 +167,11 @@ typedef int ErlNifTSDKey; typedef ErlDrvThreadOpts ErlNifThreadOpts; -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT typedef enum { - ERL_NIF_DIRTY_JOB_CPU_BOUND = ERL_DRV_DIRTY_JOB_CPU_BOUND, - ERL_NIF_DIRTY_JOB_IO_BOUND = ERL_DRV_DIRTY_JOB_IO_BOUND + ERL_NIF_DIRTY_JOB_CPU_BOUND = ERL_DIRTY_JOB_CPU_BOUND, + ERL_NIF_DIRTY_JOB_IO_BOUND = ERL_DIRTY_JOB_IO_BOUND }ErlNifDirtyTaskFlags; -#endif typedef struct /* All fields all internal and may change */ { @@ -257,11 +255,7 @@ extern TWinDynNifCallbacks WinDynNifCallbacks; # define ERL_NIF_INIT_DECL(MODNAME) ERL_NIF_INIT_EXPORT ErlNifEntry* nif_init(ERL_NIF_INIT_ARGS) #endif -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT -# define ERL_NIF_ENTRY_OPTIONS ERL_NIF_DIRTY_NIF_OPTION -#else -# define ERL_NIF_ENTRY_OPTIONS 0 -#endif +#define ERL_NIF_ENTRY_OPTIONS ERL_NIF_DIRTY_NIF_OPTION #ifdef __cplusplus } diff --git a/erts/emulator/beam/erl_nif_api_funcs.h b/erts/emulator/beam/erl_nif_api_funcs.h index a5acd86551..b211ab4b16 100644 --- a/erts/emulator/beam/erl_nif_api_funcs.h +++ b/erts/emulator/beam/erl_nif_api_funcs.h @@ -166,26 +166,19 @@ ERL_NIF_API_FUNC_DECL(ErlNifTime, enif_convert_time_unit, (ErlNifTime, ErlNifTim ERL_NIF_API_FUNC_DECL(ERL_NIF_TERM, enif_now_time, (ErlNifEnv *env)); ERL_NIF_API_FUNC_DECL(ERL_NIF_TERM, enif_cpu_time, (ErlNifEnv *env)); ERL_NIF_API_FUNC_DECL(ERL_NIF_TERM, enif_make_unique_integer, (ErlNifEnv *env, ErlNifUniqueInteger properties)); +ERL_NIF_API_FUNC_DECL(int, enif_is_current_process_alive, (ErlNifEnv *env)); ERL_NIF_API_FUNC_DECL(int, enif_is_process_alive, (ErlNifEnv *env, ErlNifPid *pid)); ERL_NIF_API_FUNC_DECL(int, enif_is_port_alive, (ErlNifEnv *env, ErlNifPort *port_id)); ERL_NIF_API_FUNC_DECL(int, enif_get_local_port, (ErlNifEnv* env, ERL_NIF_TERM, ErlNifPort* port_id)); ERL_NIF_API_FUNC_DECL(int, enif_term_to_binary, (ErlNifEnv *env, ERL_NIF_TERM term, ErlNifBinary *bin)); ERL_NIF_API_FUNC_DECL(size_t, enif_binary_to_term, (ErlNifEnv *env, const unsigned char* data, size_t sz, ERL_NIF_TERM *term, unsigned int opts)); ERL_NIF_API_FUNC_DECL(int, enif_port_command, (ErlNifEnv *env, const ErlNifPort* to_port, ErlNifEnv *msg_env, ERL_NIF_TERM msg)); +ERL_NIF_API_FUNC_DECL(int,enif_is_on_dirty_scheduler,(ErlNifEnv*)); +ERL_NIF_API_FUNC_DECL(int,enif_snprintf,(char * buffer, size_t size, const char *format, ...)); /* ** ADD NEW ENTRIES HERE (before this comment) !!! */ - - -/* - * Conditional EXPERIMENTAL stuff always last. - * Must be moved up and made unconditional to support binary backward - * compatibility on Windows. - */ -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT -ERL_NIF_API_FUNC_DECL(int,enif_is_on_dirty_scheduler,(ErlNifEnv*)); -#endif #endif /* ERL_NIF_API_FUNC_DECL */ /* @@ -330,12 +323,15 @@ ERL_NIF_API_FUNC_DECL(int,enif_is_on_dirty_scheduler,(ErlNifEnv*)); # define enif_now_time ERL_NIF_API_FUNC_MACRO(enif_now_time) # define enif_cpu_time ERL_NIF_API_FUNC_MACRO(enif_cpu_time) # define enif_make_unique_integer ERL_NIF_API_FUNC_MACRO(enif_make_unique_integer) +# define enif_is_current_process_alive ERL_NIF_API_FUNC_MACRO(enif_is_current_process_alive) # define enif_is_process_alive ERL_NIF_API_FUNC_MACRO(enif_is_process_alive) # define enif_is_port_alive ERL_NIF_API_FUNC_MACRO(enif_is_port_alive) # define enif_get_local_port ERL_NIF_API_FUNC_MACRO(enif_get_local_port) # define enif_term_to_binary ERL_NIF_API_FUNC_MACRO(enif_term_to_binary) # define enif_binary_to_term ERL_NIF_API_FUNC_MACRO(enif_binary_to_term) # define enif_port_command ERL_NIF_API_FUNC_MACRO(enif_port_command) +# define enif_is_on_dirty_scheduler ERL_NIF_API_FUNC_MACRO(enif_is_on_dirty_scheduler) +# define enif_snprintf ERL_NIF_API_FUNC_MACRO(enif_snprintf) /* ** ADD NEW ENTRIES HERE (before this comment) @@ -346,9 +342,6 @@ ERL_NIF_API_FUNC_DECL(int,enif_is_on_dirty_scheduler,(ErlNifEnv*)); * Must be moved up and made unconditional to support binary backward * compatibility on Windows. */ -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT -# define enif_is_on_dirty_scheduler ERL_NIF_API_FUNC_MACRO(enif_is_on_dirty_scheduler) -#endif #endif /* ERL_NIF_API_FUNC_MACRO */ diff --git a/erts/emulator/beam/erl_port.h b/erts/emulator/beam/erl_port.h index c2588e718d..f0075ca2b9 100644 --- a/erts/emulator/beam/erl_port.h +++ b/erts/emulator/beam/erl_port.h @@ -487,6 +487,7 @@ ERTS_GLB_INLINE Port*erts_id2port(Eterm id); ERTS_GLB_INLINE Port *erts_id2port_sflgs(Eterm, Process *, ErtsProcLocks, Uint32); ERTS_GLB_INLINE void erts_port_release(Port *); #ifdef ERTS_SMP +ERTS_GLB_INLINE Port *erts_thr_port_lookup(Eterm id, Uint32 invalid_sflgs); ERTS_GLB_INLINE Port *erts_thr_id2port_sflgs(Eterm id, Uint32 invalid_sflgs); ERTS_GLB_INLINE void erts_thr_port_release(Port *prt); #endif @@ -626,6 +627,44 @@ erts_port_release(Port *prt) } #ifdef ERTS_SMP +/* + * erts_thr_id2port_sflgs() and erts_port_dec_refc(prt) can + * be used by unmanaged threads in the SMP case. + */ +ERTS_GLB_INLINE Port * +erts_thr_port_lookup(Eterm id, Uint32 invalid_sflgs) +{ + Port *prt; + ErtsThrPrgrDelayHandle dhndl; + + if (is_not_internal_port(id)) + return NULL; + + dhndl = erts_thr_progress_unmanaged_delay(); + + prt = (Port *) erts_ptab_pix2intptr_ddrb(&erts_port, + internal_port_index(id)); + + if (!prt || prt->common.id != id) { + erts_thr_progress_unmanaged_continue(dhndl); + return NULL; + } + else { + erts_aint32_t state; + erts_port_inc_refc(prt); + + if (dhndl != ERTS_THR_PRGR_DHANDLE_MANAGED) + erts_thr_progress_unmanaged_continue(dhndl); + + state = erts_atomic32_read_acqb(&prt->state); + if (state & invalid_sflgs) { + erts_port_dec_refc(prt); + return NULL; + } + + return prt; + } +} /* * erts_thr_id2port_sflgs() and erts_thr_port_release() can diff --git a/erts/emulator/beam/erl_process.c b/erts/emulator/beam/erl_process.c index bc32f3f167..a853ec585b 100644 --- a/erts/emulator/beam/erl_process.c +++ b/erts/emulator/beam/erl_process.c @@ -409,6 +409,10 @@ ErtsAlignedSchedulerData *erts_aligned_scheduler_data; #ifdef ERTS_DIRTY_SCHEDULERS ErtsAlignedSchedulerData *erts_aligned_dirty_cpu_scheduler_data; ErtsAlignedSchedulerData *erts_aligned_dirty_io_scheduler_data; +typedef union { + Process dsp; + char align[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(Process))]; +} ErtsAlignedDirtyShadowProcess; #endif typedef union { @@ -589,6 +593,7 @@ dbg_chk_aux_work_val(erts_aint32_t value) valid |= ERTS_SSI_AUX_WORK_CNCLD_TMRS; valid |= ERTS_SSI_AUX_WORK_CNCLD_TMRS_THR_PRGR; valid |= ERTS_SSI_AUX_WORK_THR_PRGR_LATER_OP; + valid |= ERTS_SSI_AUX_WORK_PENDING_EXITERS; #endif #if HAVE_ERTS_MSEG valid |= ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK; @@ -611,7 +616,7 @@ dbg_chk_aux_work_val(erts_aint32_t value) #endif #ifdef ERTS_SMP -static void handle_pending_exiters(ErtsProcList *); +static void do_handle_pending_exiters(ErtsProcList *); static void wake_scheduler(ErtsRunQueue *rq); #endif @@ -679,6 +684,8 @@ erts_pre_init_process(void) = "MISC_THR_PRGR"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_MISC_IX] = "MISC"; + erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_PENDING_EXITERS_IX] + = "PENDING_EXITERS"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_SET_TMO_IX] = "SET_TMO"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK_IX] @@ -1139,7 +1146,7 @@ reply_sched_wall_time(void *vswtrp) hpp = &hp; } - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); if (swtrp->req_sched == esdp->no) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -1156,7 +1163,7 @@ reply_sched_wall_time(void *vswtrp) Eterm erts_sched_wall_time_request(Process *c_p, int set, int enable) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); Eterm ref; ErtsSchedWallTimeReq *swtrp; Eterm *hp; @@ -1218,7 +1225,7 @@ reply_system_check(void *vscrp) hpp = &hp; msg = STORE_NC(hpp, ohp, scrp->ref); - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); if (scrp->req_sched == esdp->no) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -1234,7 +1241,7 @@ reply_system_check(void *vscrp) Eterm erts_system_check_request(Process *c_p) { - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); Eterm ref; ErtsSystemCheckReq *scrp; Eterm *hp; @@ -2336,6 +2343,30 @@ handle_mseg_cache_check(ErtsAuxWorkData *awdp, erts_aint32_t aux_work, int waiti #endif +#ifdef ERTS_SMP + +static ERTS_INLINE erts_aint32_t +handle_pending_exiters(ErtsAuxWorkData *awdp, erts_aint32_t aux_work, int waiting) +{ + ErtsProcList *pnd_xtrs; + ErtsRunQueue *rq; + + rq = awdp->esdp->run_queue; + unset_aux_work_flags(awdp->ssi, ERTS_SSI_AUX_WORK_PENDING_EXITERS); + + erts_smp_runq_lock(rq); + pnd_xtrs = rq->procs.pending_exiters; + rq->procs.pending_exiters = NULL; + erts_smp_runq_unlock(rq); + + if (erts_proclist_fetch(&pnd_xtrs, NULL)) + do_handle_pending_exiters(pnd_xtrs); + + return aux_work & ~ERTS_SSI_AUX_WORK_PENDING_EXITERS; +} + +#endif + static ERTS_INLINE erts_aint32_t handle_setup_aux_work_timer(ErtsAuxWorkData *awdp, erts_aint32_t aux_work, int waiting) { @@ -2427,6 +2458,10 @@ handle_aux_work(ErtsAuxWorkData *awdp, erts_aint32_t orig_aux_work, int waiting) HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_MISC, handle_misc_aux_work); +#ifdef ERTS_SMP + HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_PENDING_EXITERS, + handle_pending_exiters); +#endif HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_SET_TMO, handle_setup_aux_work_timer); @@ -3979,6 +4014,33 @@ schedule_bound_processes(ErtsRunQueue *rq, } } +#ifdef ERTS_DIRTY_SCHEDULERS + +static ERTS_INLINE void +clear_proc_dirty_queue_bit(Process *p, ErtsRunQueue *rq, int prio_bit) +{ +#ifdef DEBUG + erts_aint32_t old; +#endif + erts_aint32_t qb = prio_bit; + if (rq == ERTS_DIRTY_CPU_RUNQ) + qb <<= ERTS_PDSFLGS_IN_CPU_PRQ_MASK_OFFSET; + else { + ASSERT(rq == ERTS_DIRTY_IO_RUNQ); + qb <<= ERTS_PDSFLGS_IN_IO_PRQ_MASK_OFFSET; + } +#ifdef DEBUG + old = (int) +#else + (void) +#endif + erts_smp_atomic32_read_band_mb(&p->dirty_state, ~qb); + ASSERT(old & qb); +} + +#endif /* ERTS_DIRTY_SCHEDULERS */ + + static void evacuate_run_queue(ErtsRunQueue *rq, ErtsStuckBoundProcesses *sbpp) @@ -4141,29 +4203,8 @@ evacuate_run_queue(ErtsRunQueue *rq, } #ifdef ERTS_DIRTY_SCHEDULERS - - if (ERTS_RUNQ_IX_IS_DIRTY(rq->ix)) { - erts_aint32_t dqbit = qbit; -#ifdef DEBUG - erts_aint32_t old_dqbit; -#endif - - if (rq == ERTS_DIRTY_CPU_RUNQ) - dqbit <<= ERTS_PDSFLGS_IN_CPU_PRQ_MASK_OFFSET; - else { - ASSERT(rq == ERTS_DIRTY_IO_RUNQ); - dqbit <<= ERTS_PDSFLGS_IN_IO_PRQ_MASK_OFFSET; - } - -#ifdef DEBUG - old_dqbit = (int) -#else - (void) -#endif - erts_smp_atomic32_read_band_mb(&real_proc->dirty_state, - ~dqbit); - ASSERT(old_dqbit & dqbit); - } + if (ERTS_RUNQ_IX_IS_DIRTY(rq->ix)) + clear_proc_dirty_queue_bit(real_proc, rq, qbit); #endif if (ERTS_PSFLG_BOUND & real_state) { @@ -5653,7 +5694,8 @@ static void init_scheduler_data(ErtsSchedulerData* esdp, int num, ErtsSchedulerSleepInfo* ssi, ErtsRunQueue* runq, - char** daww_ptr, size_t daww_sz) + char** daww_ptr, size_t daww_sz, + Process *shadow_proc) { esdp->timer_wheel = NULL; #ifdef ERTS_SMP @@ -5677,6 +5719,15 @@ init_scheduler_data(ErtsSchedulerData* esdp, int num, esdp->no = (Uint) num; ERTS_DIRTY_SCHEDULER_NO(esdp) = 0; } + esdp->dirty_shadow_process = shadow_proc; + if (shadow_proc) { + erts_init_empty_process(shadow_proc); + erts_smp_atomic32_init_nob(&shadow_proc->state, + (ERTS_PSFLG_ACTIVE + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_PROXY)); + shadow_proc->static_flags = ERTS_STC_FLG_SHADOW_PROC; + } #else esdp->no = (Uint) num; #endif @@ -5928,31 +5979,41 @@ erts_init_scheduling(int no_schedulers, int no_schedulers_online for (ix = 0; ix < n; ix++) { ErtsSchedulerData *esdp = ERTS_SCHEDULER_IX(ix); init_scheduler_data(esdp, ix+1, ERTS_SCHED_SLEEP_INFO_IX(ix), - ERTS_RUNQ_IX(ix), &daww_ptr, daww_sz); + ERTS_RUNQ_IX(ix), &daww_ptr, daww_sz, + NULL); } #ifdef ERTS_DIRTY_SCHEDULERS -#ifdef ERTS_SMP - erts_aligned_dirty_cpu_scheduler_data = - erts_alloc_permanent_cache_aligned( - ERTS_ALC_T_SCHDLR_DATA, - no_dirty_cpu_schedulers*sizeof(ErtsAlignedSchedulerData)); - for (ix = 0; ix < no_dirty_cpu_schedulers; ix++) { - ErtsSchedulerData *esdp = ERTS_DIRTY_CPU_SCHEDULER_IX(ix); - init_scheduler_data(esdp, ix+1, ERTS_DIRTY_CPU_SCHED_SLEEP_INFO_IX(ix), - ERTS_DIRTY_CPU_RUNQ, NULL, 0); - } - erts_aligned_dirty_io_scheduler_data = - erts_alloc_permanent_cache_aligned( - ERTS_ALC_T_SCHDLR_DATA, - no_dirty_io_schedulers*sizeof(ErtsAlignedSchedulerData)); - for (ix = 0; ix < no_dirty_io_schedulers; ix++) { - ErtsSchedulerData *esdp = ERTS_DIRTY_IO_SCHEDULER_IX(ix); - init_scheduler_data(esdp, ix+1, ERTS_DIRTY_IO_SCHED_SLEEP_INFO_IX(ix), - ERTS_DIRTY_IO_RUNQ, NULL, 0); + { + int dirty_scheds = no_dirty_cpu_schedulers + no_dirty_io_schedulers; + int adspix = 0; + ErtsAlignedDirtyShadowProcess *adsp = + erts_alloc_permanent_cache_aligned( + ERTS_ALC_T_SCHDLR_DATA, + dirty_scheds * sizeof(ErtsAlignedDirtyShadowProcess)); + + erts_aligned_dirty_cpu_scheduler_data = + erts_alloc_permanent_cache_aligned( + ERTS_ALC_T_SCHDLR_DATA, + dirty_scheds * sizeof(ErtsAlignedSchedulerData)); + + erts_aligned_dirty_io_scheduler_data = + &erts_aligned_dirty_cpu_scheduler_data[no_dirty_cpu_schedulers]; + + for (ix = 0; ix < no_dirty_cpu_schedulers; ix++) { + ErtsSchedulerData *esdp = ERTS_DIRTY_CPU_SCHEDULER_IX(ix); + init_scheduler_data(esdp, ix+1, ERTS_DIRTY_CPU_SCHED_SLEEP_INFO_IX(ix), + ERTS_DIRTY_CPU_RUNQ, NULL, 0, + &adsp[adspix++].dsp); + } + for (ix = 0; ix < no_dirty_io_schedulers; ix++) { + ErtsSchedulerData *esdp = ERTS_DIRTY_IO_SCHEDULER_IX(ix); + init_scheduler_data(esdp, ix+1, ERTS_DIRTY_IO_SCHED_SLEEP_INFO_IX(ix), + ERTS_DIRTY_IO_RUNQ, NULL, 0, + &adsp[adspix++].dsp); + } } #endif -#endif init_misc_aux_work(); init_swtreq_alloc(); @@ -6167,7 +6228,7 @@ check_dirty_enqueue_in_prio_queue(Process *c_p, erts_aint32_t dact, max_qbit; /* Termination should be done on an ordinary scheduler */ - if (actual & ERTS_PSFLG_EXITING) { + if ((*newp) & ERTS_PSFLG_EXITING) { *newp &= ~ERTS_PSFLGS_DIRTY_WORK; return ERTS_ENQUEUE_NORMAL_QUEUE; } @@ -6176,7 +6237,7 @@ check_dirty_enqueue_in_prio_queue(Process *c_p, * If we have system tasks, we enqueue on ordinary run-queue * and take care of those system tasks first. */ - if (actual & ERTS_PSFLG_ACTIVE_SYS) + if ((*newp) & ERTS_PSFLG_ACTIVE_SYS) return ERTS_ENQUEUE_NORMAL_QUEUE; dact = erts_smp_atomic32_read_mb(&c_p->dirty_state); @@ -6356,23 +6417,29 @@ select_enqueue_run_queue(int enqueue, int enq_prio, Process *p, erts_aint32_t st * schedule_out_process() return with c_rq locked. */ static ERTS_INLINE int -schedule_out_process(ErtsRunQueue *c_rq, erts_aint32_t state, Process *p, Process *proxy) +schedule_out_process(ErtsRunQueue *c_rq, erts_aint32_t state, Process *p, + Process *proxy, int is_normal_sched) { - erts_aint32_t a, e, n, enq_prio = -1; + erts_aint32_t a, e, n, enq_prio = -1, running_flgs; int enqueue; /* < 0 -> use proxy */ ErtsRunQueue* runq; + if (is_normal_sched) + running_flgs = ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS; + else + running_flgs = ERTS_PSFLG_DIRTY_RUNNING|ERTS_PSFLG_DIRTY_RUNNING_SYS; + a = state; while (1) { n = e = a; - ASSERT(a & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS)); + ASSERT(a & running_flgs); enqueue = ERTS_ENQUEUE_NOT; - n &= ~(ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS); - if (a & (ERTS_PSFLG_ACTIVE_SYS|ERTS_PSFLG_DIRTY_ACTIVE_SYS) + n &= ~running_flgs; + if ((a & (ERTS_PSFLG_ACTIVE_SYS|ERTS_PSFLG_DIRTY_ACTIVE_SYS)) || (a & (ERTS_PSFLG_ACTIVE|ERTS_PSFLG_SUSPENDED)) == ERTS_PSFLG_ACTIVE) { enqueue = check_enqueue_in_prio_queue(p, &enq_prio, &n, a); } @@ -6485,8 +6552,9 @@ change_proc_schedule_state(Process *p, ErtsProcLocks locks) { /* - * NOTE: ERTS_PSFLG_RUNNING, ERTS_PSFLG_RUNNING_SYS and - * ERTS_PSFLG_ACTIVE_SYS are not allowed to be + * NOTE: ERTS_PSFLG_RUNNING, ERTS_PSFLG_RUNNING_SYS, + * ERTS_PSFLG_DIRTY_RUNNING, ERTS_PSFLG_DIRTY_RUNNING_SYS + * and ERTS_PSFLG_ACTIVE_SYS are not allowed to be * altered by this function! */ erts_aint32_t a = *statep, n; @@ -6500,9 +6568,13 @@ change_proc_schedule_state(Process *p, ASSERT(!(a & ERTS_PSFLG_PROXY)); ASSERT((clear_state_flags & (ERTS_PSFLG_RUNNING | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS | ERTS_PSFLG_ACTIVE_SYS)) == 0); ASSERT((set_state_flags & (ERTS_PSFLG_RUNNING | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS | ERTS_PSFLG_ACTIVE_SYS)) == 0); if (lock_status) @@ -6526,8 +6598,16 @@ change_proc_schedule_state(Process *p, if ((n & (ERTS_PSFLG_SUSPENDED | ERTS_PSFLG_RUNNING | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS | ERTS_PSFLG_IN_RUNQ - | ERTS_PSFLG_ACTIVE)) == ERTS_PSFLG_ACTIVE) { + | ERTS_PSFLG_ACTIVE)) == ERTS_PSFLG_ACTIVE +#ifdef ERTS_DIRTY_SCHEDULERS + || (n & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_EXITING)) == ERTS_PSFLG_EXITING +#endif + ) { /* * Active and seemingly need to be enqueued, but * process may be in a run queue via proxy, need @@ -6551,7 +6631,9 @@ change_proc_schedule_state(Process *p, | ERTS_PSFLG_ACTIVE)) == ERTS_PSFLG_ACTIVE) && (!(a & (ERTS_PSFLG_ACTIVE_SYS | ERTS_PSFLG_RUNNING - | ERTS_PSFLG_RUNNING_SYS) + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS) && (!(a & ERTS_PSFLG_ACTIVE) || (a & ERTS_PSFLG_SUSPENDED))))) { /* We activated a prevously inactive process */ @@ -6693,7 +6775,10 @@ schedule_process_sys_task(Process *p, erts_aint32_t prio, ErtsProcSysTask *st) enqueue = ERTS_ENQUEUE_NOT; n |= ERTS_PSFLG_ACTIVE_SYS; - if (!(a & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS))) + if (!(a & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS))) enqueue = check_enqueue_in_prio_queue(p, &enq_prio, &n, a); a = erts_smp_atomic32_cmpxchg_mb(&p->state, n, e); if (a == e) @@ -6706,7 +6791,9 @@ schedule_process_sys_task(Process *p, erts_aint32_t prio, ErtsProcSysTask *st) if (!(a & (ERTS_PSFLG_ACTIVE_SYS | ERTS_PSFLG_RUNNING - | ERTS_PSFLG_RUNNING_SYS)) + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)) && (!(a & ERTS_PSFLG_ACTIVE) || (a & ERTS_PSFLG_SUSPENDED))) { /* We activated a prevously inactive process */ profile_runnable_proc(p, am_active); @@ -6746,11 +6833,16 @@ suspend_process(Process *c_p, Process *p) if (c_p == p) { state = erts_smp_atomic32_read_bor_relb(&p->state, ERTS_PSFLG_SUSPENDED); - ASSERT(state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS)); + ASSERT(state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)); suspended = (state & ERTS_PSFLG_SUSPENDED) ? -1: 1; } else { - while (!(state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_EXITING))) { + while (!(state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_EXITING))) { erts_aint32_t n, e; n = e = state; @@ -6776,8 +6868,11 @@ suspend_process(Process *c_p, Process *p) if ((state & (ERTS_PSFLG_ACTIVE | ERTS_PSFLG_ACTIVE_SYS + | ERTS_PSFLG_DIRTY_ACTIVE_SYS | ERTS_PSFLG_RUNNING | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS | ERTS_PSFLG_SUSPENDED)) == ERTS_PSFLG_ACTIVE) { /* We made process inactive */ profile_runnable_proc(p, am_inactive); @@ -7759,8 +7854,10 @@ erts_block_multi_scheduling(Process *p, ErtsProcLocks plocks, int on, int normal plp = proclist_create(p); erts_proclist_store_last(&msbp->blckrs, plp); p->flags |= have_blckd_flg; - ASSERT(schdlr_sspnd.active == ERTS_SCHDLR_SSPND_MAKE_NSCHEDS_VAL(1, 0, 0)); - ASSERT(p->scheduler_data->no == 1); + ASSERT(normal + ? 1 == schdlr_sspnd_get_nscheds(&schdlr_sspnd.active, ERTS_SCHED_NORMAL) + : schdlr_sspnd.active == ERTS_SCHDLR_SSPND_MAKE_NSCHEDS_VAL(1, 0, 0)); + ASSERT(erts_proc_sched_data(p)->no == 1); if (schdlr_sspnd.msb.ongoing) res = ERTS_SCHDLR_SSPND_DONE_MSCHED_BLOCKED; else @@ -7780,7 +7877,7 @@ erts_block_multi_scheduling(Process *p, ErtsProcLocks plocks, int on, int normal if (schdlr_sspnd.active == ERTS_SCHDLR_SSPND_MAKE_NSCHEDS_VAL(1, 0, 0) || (normal && schdlr_sspnd_get_nscheds(&schdlr_sspnd.active, ERTS_SCHED_NORMAL) == 1)) { - ASSERT(p->scheduler_data->no == 1); + ASSERT(erts_proc_sched_data(p)->no == 1); plp = proclist_create(p); erts_proclist_store_last(&msbp->blckrs, plp); if (schdlr_sspnd.msb.ongoing) @@ -7830,7 +7927,7 @@ erts_block_multi_scheduling(Process *p, ErtsProcLocks plocks, int on, int normal else res = ERTS_SCHDLR_SSPND_YIELD_DONE_NMSCHED_BLOCKED; } - ASSERT(p->scheduler_data); + ASSERT(erts_proc_sched_data(p)); } } else if (!msbp->ongoing) { @@ -8420,9 +8517,23 @@ pid2proc_not_running(Process *c_p, ErtsProcLocks c_p_locks, if (!suspend_process(c_p, rp)) { /* Other process running */ - ASSERT(ERTS_PSFLG_RUNNING + ASSERT((ERTS_PSFLG_RUNNING | ERTS_PSFLG_DIRTY_RUNNING) & erts_smp_atomic32_read_nob(&rp->state)); +#ifdef ERTS_DIRTY_SCHEDULERS + if (!suspend + && (erts_smp_atomic32_read_nob(&rp->state) + & ERTS_PSFLG_DIRTY_RUNNING)) { + ErtsProcLocks need_locks = pid_locks & ~ERTS_PROC_LOCK_STATUS; + if (need_locks && erts_smp_proc_trylock(rp, need_locks) == EBUSY) { + erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_STATUS); + rp = erts_pid2proc(c_p, c_p_locks|ERTS_PROC_LOCK_STATUS, + pid, pid_locks|ERTS_PROC_LOCK_STATUS); + } + goto done; + } +#endif + running: /* @@ -8447,7 +8558,7 @@ pid2proc_not_running(Process *c_p, ErtsProcLocks c_p_locks, else { ErtsProcLocks need_locks = pid_locks & ~ERTS_PROC_LOCK_STATUS; if (need_locks && erts_smp_proc_trylock(rp, need_locks) == EBUSY) { - if (ERTS_PSFLG_RUNNING_SYS + if ((ERTS_PSFLG_RUNNING_SYS|ERTS_PSFLG_DIRTY_RUNNING_SYS) & erts_smp_atomic32_read_nob(&rp->state)) { /* Executing system task... */ resume_process(rp, ERTS_PROC_LOCK_STATUS); @@ -8474,7 +8585,7 @@ pid2proc_not_running(Process *c_p, ErtsProcLocks c_p_locks, * from being selected for normal execution regardless * of locks held or not held on it... */ - ASSERT(!(ERTS_PSFLG_RUNNING + ASSERT(!((ERTS_PSFLG_RUNNING|ERTS_PSFLG_DIRTY_RUNNING_SYS) & erts_smp_atomic32_read_nob(&rp->state))); if (!suspend) @@ -9015,28 +9126,43 @@ erts_run_queues_len(Uint *qlen, int atomic_queues_read, int incl_active_sched) } Eterm -erts_process_status(Process *c_p, ErtsProcLocks c_p_locks, - Process *rp, Eterm rpid) +erts_process_state2status(erts_aint32_t state) +{ + if (state & ERTS_PSFLG_FREE) + return am_free; + + if (state & ERTS_PSFLG_EXITING) + return am_exiting; + + if (state & ERTS_PSFLG_GC) + return am_garbage_collecting; + + if (state & ERTS_PSFLG_SUSPENDED) + return am_suspended; + + if (state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)) + return am_running; + + if (state & (ERTS_PSFLG_ACTIVE + | ERTS_PSFLG_ACTIVE_SYS + | ERTS_PSFLG_DIRTY_ACTIVE_SYS)) + return am_runnable; + + return am_waiting; +} + +Eterm +erts_process_status(Process *rp, Eterm rpid) { Eterm res = am_undefined; Process *p = rp ? rp : erts_proc_lookup_raw(rpid); if (p) { erts_aint32_t state = erts_smp_atomic32_read_acqb(&p->state); - if (state & ERTS_PSFLG_FREE) - res = am_free; - else if (state & ERTS_PSFLG_EXITING) - res = am_exiting; - else if (state & ERTS_PSFLG_GC) - res = am_garbage_collecting; - else if (state & ERTS_PSFLG_SUSPENDED) - res = am_suspended; - else if (state & ERTS_PSFLG_RUNNING) - res = am_running; - else if (state & ERTS_PSFLG_ACTIVE) - res = am_runnable; - else - res = am_waiting; + res = erts_process_state2status(state); } #ifdef ERTS_SMP else { @@ -9251,7 +9377,76 @@ scheduler_gc_proc(Process *c_p, int reds_left) return reds; } +static ERTS_INLINE void +clean_dirty_start(Process *p) +{ +#if defined(ERTS_DIRTY_SCHEDULERS) && !defined(ARCH_64) + void *ptr = ERTS_PROC_SET_DIRTY_CPU_START(p, NULL); + if (ptr) + erts_free(ERTS_ALC_T_DIRTY_START, ptr); +#endif +} +static ERTS_INLINE void +save_dirty_start(ErtsSchedulerData *esdp, Process *c_p) +{ +#ifdef ERTS_DIRTY_SCHEDULERS + if (ERTS_RUNQ_IS_DIRTY_CPU_RUNQ(esdp->run_queue)) { + ErtsMonotonicTime time = erts_get_monotonic_time(esdp); +#ifdef ARCH_64 + ERTS_PROC_SET_DIRTY_CPU_START(c_p, (void *) time); +#else + ErtsMonotonicTime *stimep; + + stimep = (ErtsMonotonicTime *) ERTS_PROC_GET_DIRTY_CPU_START(c_p); + if (!stimep) { + stimep = erts_alloc(ERTS_ALC_T_DIRTY_START, + sizeof(ErtsMonotonicTime)); + ERTS_PROC_SET_DIRTY_CPU_START(c_p, (void *) stimep); + } + *stimep = time; +#endif + } +#endif +} + +static ERTS_INLINE int +get_dirty_reds(ErtsSchedulerData *esdp, Process *c_p) +{ + +#ifndef ERTS_DIRTY_SCHEDULERS + return -1; +#else + ErtsMonotonicTime stime, time; + + if (!ERTS_RUNQ_IS_DIRTY_CPU_RUNQ(esdp->run_queue)) + return 1; + +#ifdef ARCH_64 + stime = (ErtsMonotonicTime) ERTS_PROC_GET_DIRTY_CPU_START(c_p); +#else + { + ErtsMonotonicTime *stimep; + stimep = (ErtsMonotonicTime *) ERTS_PROC_GET_DIRTY_CPU_START(c_p); + ASSERT(stimep); + stime = *stimep; + } +#endif + + time = erts_get_monotonic_time(esdp); + + ASSERT(stime && stime < time); + + time -= stime; + time = ERTS_MONOTONIC_TO_USEC(time); + time *= 2; + + if (time > INT_MAX) + return INT_MAX; + return (int) time; +#endif + +} /* * schedule() is called from BEAM (process_main()) or HiPE @@ -9283,6 +9478,7 @@ Process *schedule(Process *p, int calls) int reds; Uint32 flags; erts_aint32_t state = 0; /* Supress warning... */ + int is_normal_sched; ERTS_MSACC_DECLARE_CACHE(); @@ -9312,25 +9508,44 @@ Process *schedule(Process *p, int calls) */ if (!p) { /* NULL in the very first schedule() call */ esdp = erts_get_scheduler_data(); + is_normal_sched = !ERTS_SCHEDULER_IS_DIRTY(esdp); rq = erts_get_runq_current(esdp); ASSERT(esdp); fcalls = (int) erts_smp_atomic32_read_acqb(&function_calls); actual_reds = reds = 0; erts_smp_runq_lock(rq); } else { - sched_out_proc: - #ifdef ERTS_SMP - ERTS_SMP_CHK_HAVE_ONLY_MAIN_PROC_LOCK(p); +#ifdef ERTS_DIRTY_SCHEDULERS + esdp = p->scheduler_data; + is_normal_sched = esdp != NULL; + if (is_normal_sched) + ASSERT(!ERTS_SCHEDULER_IS_DIRTY(esdp)); + else { + esdp = erts_get_scheduler_data(); + ASSERT(ERTS_SCHEDULER_IS_DIRTY(esdp)); + } +#else esdp = p->scheduler_data; + is_normal_sched = 1; +#endif ASSERT(esdp->current_process == p || esdp->free_process == p); #else esdp = erts_scheduler_data; ASSERT(esdp->current_process == p); + is_normal_sched = 1; #endif - reds = actual_reds = calls - esdp->virtual_reds; + sched_out_proc: + + ERTS_SMP_CHK_HAVE_ONLY_MAIN_PROC_LOCK(p); + + if (is_normal_sched) + reds = actual_reds = calls - esdp->virtual_reds; + else + reds = actual_reds = get_dirty_reds(esdp, p); + ASSERT(actual_reds >= 0); if (reds < ERTS_PROC_MIN_CONTEXT_SWITCH_REDS_COST) reds = ERTS_PROC_MIN_CONTEXT_SWITCH_REDS_COST; @@ -9377,7 +9592,7 @@ Process *schedule(Process *p, int calls) state = erts_smp_atomic32_read_nob(&p->state); #ifdef ERTS_SMP - if (state & ERTS_PSFLG_PENDING_EXIT) + if (is_normal_sched && (state & ERTS_PSFLG_PENDING_EXIT)) erts_handle_pending_exit(p, (ERTS_PROC_LOCK_MAIN | ERTS_PROC_LOCK_STATUS)); if (p->pending_suspenders) @@ -9387,7 +9602,8 @@ Process *schedule(Process *p, int calls) esdp->reductions += reds; - schedule_out_process(rq, state, p, proxy_p); /* Returns with rq locked! */ + /* schedule_out_process() returns with rq locked! */ + schedule_out_process(rq, state, p, proxy_p, is_normal_sched); proxy_p = NULL; ERTS_PROC_REDUCTIONS_EXECUTED(esdp, rq, @@ -9405,13 +9621,20 @@ Process *schedule(Process *p, int calls) ERTS_MSACC_SET_STATE_CACHED_M(ERTS_MSACC_STATE_OTHER); if (state & ERTS_PSFLG_FREE) { + if (!is_normal_sched) { + ASSERT(p->flags & F_DELAYED_DEL_PROC); + erts_proc_dec_refc(p); + } + else { #ifdef ERTS_SMP - ASSERT(esdp->free_process == p); - esdp->free_process = NULL; + ASSERT(esdp->free_process == p); + esdp->free_process = NULL; #else - erts_proc_dec_refc(p); + erts_proc_dec_refc(p); #endif + } } + #ifdef ERTS_SMP ASSERT(!esdp->free_process); #endif @@ -9419,7 +9642,7 @@ Process *schedule(Process *p, int calls) ERTS_SMP_CHK_NO_PROC_LOCKS; - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { + if (is_normal_sched) { if (esdp->check_time_reds >= ERTS_CHECK_TIME_REDS) (void) erts_get_monotonic_time(esdp); @@ -9433,23 +9656,15 @@ Process *schedule(Process *p, int calls) } - ERTS_SMP_LC_ASSERT(ERTS_SCHEDULER_IS_DIRTY(esdp) - || !erts_thr_progress_is_blocking()); + ERTS_SMP_LC_ASSERT(!is_normal_sched || !erts_thr_progress_is_blocking()); check_activities_to_run: { + erts_aint32_t psflg_running, psflg_running_sys; #ifdef ERTS_SMP ErtsMigrationPaths *mps; ErtsMigrationPath *mp; - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { - ErtsProcList *pnd_xtrs = rq->procs.pending_exiters; - if (erts_proclist_fetch(&pnd_xtrs, NULL)) { - rq->procs.pending_exiters = NULL; - erts_smp_runq_unlock(rq); - handle_pending_exiters(pnd_xtrs); - erts_smp_runq_lock(rq); - } - + if (is_normal_sched) { if (rq->check_balance_reds <= 0) check_balance(rq); @@ -9466,32 +9681,35 @@ Process *schedule(Process *p, int calls) continue_check_activities_to_run: flags = ERTS_RUNQ_FLGS_GET_NOB(rq); continue_check_activities_to_run_known_flags: - ASSERT(ERTS_SCHEDULER_IS_DIRTY(esdp) - || flags & ERTS_RUNQ_FLG_NONEMPTY); + ASSERT(!is_normal_sched || (flags & ERTS_RUNQ_FLG_NONEMPTY)); - if (flags & (ERTS_RUNQ_FLG_CHK_CPU_BIND|ERTS_RUNQ_FLG_SUSPENDED)) { - if (flags & ERTS_RUNQ_FLG_SUSPENDED) { - (void) ERTS_RUNQ_FLGS_UNSET_NOB(rq, ERTS_RUNQ_FLG_EXEC); + if (!is_normal_sched) { + if (erts_smp_atomic32_read_acqb(&esdp->ssi->flags) + & ERTS_SSI_FLG_SUSPENDED) { suspend_scheduler(esdp); - flags = ERTS_RUNQ_FLGS_SET_NOB(rq, ERTS_RUNQ_FLG_EXEC); - flags |= ERTS_RUNQ_FLG_EXEC; - } - if (flags & ERTS_RUNQ_FLG_CHK_CPU_BIND) { - flags = ERTS_RUNQ_FLGS_UNSET(rq, ERTS_RUNQ_FLG_CHK_CPU_BIND); - flags &= ~ ERTS_RUNQ_FLG_CHK_CPU_BIND; - erts_sched_check_cpu_bind(esdp); } } -#ifdef ERTS_DIRTY_SCHEDULERS - else if (ERTS_SCHEDULER_IS_DIRTY(esdp) - && (erts_smp_atomic32_read_acqb(&esdp->ssi->flags) - & ERTS_SSI_FLG_SUSPENDED)) - suspend_scheduler(esdp); -#endif - - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { + else { erts_aint32_t aux_work; - int leader_update = erts_thr_progress_update(esdp); + int leader_update; + + ASSERT(is_normal_sched); + + if (flags & (ERTS_RUNQ_FLG_CHK_CPU_BIND|ERTS_RUNQ_FLG_SUSPENDED)) { + if (flags & ERTS_RUNQ_FLG_SUSPENDED) { + (void) ERTS_RUNQ_FLGS_UNSET_NOB(rq, ERTS_RUNQ_FLG_EXEC); + suspend_scheduler(esdp); + flags = ERTS_RUNQ_FLGS_SET_NOB(rq, ERTS_RUNQ_FLG_EXEC); + flags |= ERTS_RUNQ_FLG_EXEC; + } + if (flags & ERTS_RUNQ_FLG_CHK_CPU_BIND) { + flags = ERTS_RUNQ_FLGS_UNSET(rq, ERTS_RUNQ_FLG_CHK_CPU_BIND); + flags &= ~ERTS_RUNQ_FLG_CHK_CPU_BIND; + erts_sched_check_cpu_bind(esdp); + } + } + + leader_update = erts_thr_progress_update(esdp); aux_work = erts_atomic32_read_acqb(&esdp->ssi->aux_work); if (aux_work | leader_update) { erts_smp_runq_unlock(rq); @@ -9517,19 +9735,13 @@ Process *schedule(Process *p, int calls) flags = ERTS_RUNQ_FLGS_GET_NOB(rq); -#ifdef ERTS_DIRTY_SCHEDULERS - if (ERTS_RUNQ_IX_IS_DIRTY(rq->ix) && rq->halt_in_progress) { - /* - * TODO: if halt in progress, need to put the dirty scheduler - * to sleep somewhere around here to prevent it from picking up - * new work - */ + if (!is_normal_sched && rq->halt_in_progress) { + /* Wait for emulator to terminate... */ + while (1) + erts_milli_sleep(1000*1000); } - else -#endif - - if ((!(flags & ERTS_RUNQ_FLGS_QMASK) && !rq->misc.start) - || (rq->halt_in_progress && ERTS_EMPTY_RUNQ_PORTS(rq))) { + else if ((!(flags & ERTS_RUNQ_FLGS_QMASK) && !rq->misc.start) + || (rq->halt_in_progress && ERTS_EMPTY_RUNQ_PORTS(rq))) { /* Prepare for scheduler wait */ #ifdef ERTS_SMP ERTS_SMP_LC_ASSERT(erts_smp_lc_runq_is_locked(rq)); @@ -9543,7 +9755,7 @@ Process *schedule(Process *p, int calls) if (flags & ERTS_RUNQ_FLG_INACTIVE) empty_runq(rq); else { - if (!ERTS_RUNQ_IX_IS_DIRTY(rq->ix) && try_steal_task(rq)) + if (is_normal_sched && try_steal_task(rq)) goto continue_check_activities_to_run; empty_runq(rq); @@ -9572,9 +9784,9 @@ Process *schedule(Process *p, int calls) goto check_activities_to_run; } - else if (!ERTS_SCHEDULER_IS_DIRTY(esdp) && - (fcalls > input_reductions && - prepare_for_sys_schedule(!0))) { + else if (is_normal_sched + && (fcalls > input_reductions + && prepare_for_sys_schedule(!0))) { ErtsMonotonicTime current_time; /* * Schedule system-level activities. @@ -9688,11 +9900,17 @@ Process *schedule(Process *p, int calls) ASSERT(p); /* Wrong qmask in rq->flags? */ - if (ERTS_SCHEDULER_IS_DIRTY(esdp)) - psflg_band_mask = ~((erts_aint32_t) 0); - else + if (is_normal_sched) { + psflg_running = ERTS_PSFLG_RUNNING; + psflg_running_sys = ERTS_PSFLG_RUNNING_SYS; psflg_band_mask = ~(((erts_aint32_t) 1) << (ERTS_PSFLGS_GET_PRQ_PRIO(state) + ERTS_PSFLGS_IN_PRQ_MASK_OFFSET)); + } + else { + psflg_running = ERTS_PSFLG_DIRTY_RUNNING; + psflg_running_sys = ERTS_PSFLG_DIRTY_RUNNING_SYS; + psflg_band_mask = ~((erts_aint32_t) 0); + } if (!(state & ERTS_PSFLG_PROXY)) psflg_band_mask &= ~ERTS_PSFLG_IN_RUNQ; @@ -9707,34 +9925,53 @@ Process *schedule(Process *p, int calls) state = erts_smp_atomic32_read_nob(&p->state); } +#ifdef ERTS_DIRTY_SCHEDULERS + if (!is_normal_sched) + clear_proc_dirty_queue_bit(p, rq, qbit); +#endif + while (1) { - erts_aint32_t exp, new, tmp; - tmp = new = exp = state; + erts_aint32_t exp, new; + int run_process; + new = exp = state; new &= psflg_band_mask; - if (!(state & (ERTS_PSFLG_RUNNING - | ERTS_PSFLG_RUNNING_SYS))) { - tmp = state & (ERTS_PSFLG_SUSPENDED - | ERTS_PSFLG_PENDING_EXIT - | ERTS_PSFLG_ACTIVE_SYS - | ERTS_PSFLG_DIRTY_ACTIVE_SYS); - if (tmp != ERTS_PSFLG_SUSPENDED) { - if (state & (ERTS_PSFLG_ACTIVE_SYS - | ERTS_PSFLG_DIRTY_ACTIVE_SYS)) - new |= ERTS_PSFLG_RUNNING_SYS; - else - new |= ERTS_PSFLG_RUNNING; - } + /* + * Run process if not already running (or free) + * or exiting and not running on a normal + * scheduler, and not suspended (and not in a + * state where suspend should be ignored). + */ + run_process = (((!(state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS + | ERTS_PSFLG_FREE))) +#ifdef ERTS_DIRTY_SCHEDULERS + | (((state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_FREE + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_EXITING)) + == ERTS_PSFLG_EXITING) + & (!!is_normal_sched)) +#endif + ) + & ((state & (ERTS_PSFLG_SUSPENDED + | ERTS_PSFLG_EXITING + | ERTS_PSFLG_FREE + | ERTS_PSFLG_PENDING_EXIT + | ERTS_PSFLG_ACTIVE_SYS + | ERTS_PSFLG_DIRTY_ACTIVE_SYS)) + != ERTS_PSFLG_SUSPENDED)); + if (run_process) { + if (state & (ERTS_PSFLG_ACTIVE_SYS + | ERTS_PSFLG_DIRTY_ACTIVE_SYS)) + new |= psflg_running_sys; + else + new |= psflg_running; } state = erts_smp_atomic32_cmpxchg_relb(&p->state, new, exp); if (state == exp) { - if ((state & (ERTS_PSFLG_RUNNING - | ERTS_PSFLG_RUNNING_SYS - | ERTS_PSFLG_FREE)) - || ((state & (ERTS_PSFLG_SUSPENDED - | ERTS_PSFLG_PENDING_EXIT - | ERTS_PSFLG_ACTIVE_SYS - | ERTS_PSFLG_DIRTY_ACTIVE_SYS)) - == ERTS_PSFLG_SUSPENDED)) { + if (!run_process) { if (proxy_p) { free_proxy_proc(proxy_p); proxy_p = NULL; @@ -9761,34 +9998,13 @@ Process *schedule(Process *p, int calls) erts_smp_runq_unlock(rq); -#ifdef ERTS_DIRTY_SCHEDULERS - if (ERTS_SCHEDULER_IS_DIRTY(esdp)) { -#ifdef DEBUG - int old_dqbit; -#endif - int dqbit = qbit; - - if (rq == ERTS_DIRTY_CPU_RUNQ) - dqbit <<= ERTS_PDSFLGS_IN_CPU_PRQ_MASK_OFFSET; - else { - ASSERT(rq == ERTS_DIRTY_IO_RUNQ); - dqbit <<= ERTS_PDSFLGS_IN_IO_PRQ_MASK_OFFSET; - } - -#ifdef DEBUG - old_dqbit = (int) -#else - (void) -#endif - erts_smp_atomic32_read_band_mb(&p->dirty_state, ~dqbit); - ASSERT(old_dqbit & dqbit); - } -#endif /* ERTS_DIRTY_SCHEDULERS */ - #endif /* ERTS_SMP */ } + if (!is_normal_sched) + save_dirty_start(esdp, p); + #ifdef ERTS_SMP if (flags & ERTS_RUNQ_FLG_PROTECTED) @@ -9805,9 +10021,7 @@ Process *schedule(Process *p, int calls) UWord old = ERTS_PROC_SCHED_ID(p, (UWord) esdp->no); int migrated = old && old != esdp->no; -#ifdef ERTS_DIRTY_SCHEDULERS - ASSERT(!ERTS_SCHEDULER_IS_DIRTY(esdp)); -#endif + ASSERT(is_normal_sched); prio = (int) ERTS_PSFLGS_GET_USR_PRIO(state); @@ -9821,20 +10035,21 @@ Process *schedule(Process *p, int calls) erts_smp_spin_unlock(&erts_sched_stat.lock); } - ASSERT(!p->scheduler_data); - p->scheduler_data = esdp; - state = erts_smp_atomic32_read_nob(&p->state); -#ifdef ERTS_DIRTY_SCHEDULERS - if (!ERTS_SCHEDULER_IS_DIRTY(esdp)) { - if (!!(state & ERTS_PSFLGS_DIRTY_WORK) - & !(state & ERTS_PSFLG_ACTIVE_SYS)) { + ASSERT(!p->scheduler_data); +#ifndef ERTS_DIRTY_SCHEDULERS + p->scheduler_data = esdp; +#else /* ERTS_DIRTY_SCHEDULERS */ + if (is_normal_sched) { + if ((!!(state & ERTS_PSFLGS_DIRTY_WORK)) + & (!(state & ERTS_PSFLG_ACTIVE_SYS))) { /* Migrate to dirty scheduler... */ sunlock_sched_out_proc: erts_smp_proc_unlock(p, ERTS_PROC_LOCK_STATUS); goto sched_out_proc; } + p->scheduler_data = esdp; } else { if (state & (ERTS_PSFLG_ACTIVE_SYS @@ -9870,7 +10085,10 @@ Process *schedule(Process *p, int calls) erts_smp_proc_unlock(p, ERTS_PROC_LOCK_STATUS); - if (IS_TRACED(p)) { + /* Clear tracer if it has been removed */ + if (IS_TRACED(p) && erts_is_tracer_proc_enabled( + p, ERTS_PROC_LOCK_MAIN, &p->common)) { + if (state & ERTS_PSFLG_EXITING) { if (ARE_TRACE_FLAGS_ON(p, F_TRACE_SCHED_EXIT)) trace_sched(p, ERTS_PROC_LOCK_MAIN, am_in_exiting); @@ -9885,13 +10103,8 @@ Process *schedule(Process *p, int calls) } } - -#ifdef ERTS_SMP - /* Clears tracer if it has been removed */ - (void)ERTS_TRACER_PROC_IS_ENABLED(p); -#endif - - if (state & ERTS_PSFLG_RUNNING_SYS) { + if (state & (ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING_SYS)) { /* * GC is normally never delayed when a process * is scheduled out, but might be when executing @@ -9905,7 +10118,7 @@ Process *schedule(Process *p, int calls) reds -= cost; if (reds <= 0 #ifdef ERTS_DIRTY_SCHEDULERS - || ERTS_SCHEDULER_IS_DIRTY(esdp) + || !is_normal_sched || (state & ERTS_PSFLGS_DIRTY_WORK) #endif ) { @@ -9913,8 +10126,8 @@ Process *schedule(Process *p, int calls) } } - ASSERT(state & ERTS_PSFLG_RUNNING_SYS); - ASSERT(!(state & ERTS_PSFLG_RUNNING)); + ASSERT(state & psflg_running_sys); + ASSERT(!(state & psflg_running)); while (1) { erts_aint32_t n, e; @@ -9926,8 +10139,8 @@ Process *schedule(Process *p, int calls) } n = e = state; - n &= ~ERTS_PSFLG_RUNNING_SYS; - n |= ERTS_PSFLG_RUNNING; + n &= ~psflg_running_sys; + n |= psflg_running; state = erts_smp_atomic32_cmpxchg_mb(&p->state, n, e); if (state == e) { @@ -9935,8 +10148,8 @@ Process *schedule(Process *p, int calls) break; } - ASSERT(state & ERTS_PSFLG_RUNNING_SYS); - ASSERT(!(state & ERTS_PSFLG_RUNNING)); + ASSERT(state & psflg_running_sys); + ASSERT(!(state & psflg_running)); } } @@ -10024,7 +10237,7 @@ notify_sys_task_executed(Process *c_p, ErtsProcSysTask *st, Eterm st_result) ASSERT(hp_start + hsz == hp); #endif - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, c_p->common.id); if (c_p == rp) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -10525,7 +10738,10 @@ save_gc_task(Process *c_p, ErtsProcSysTask *st, int prio) } state = erts_smp_atomic32_read_nob(&c_p->state); - ASSERT((ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS) & state); + ASSERT((ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS) & state); while (!(state & ERTS_PSFLG_DELAYED_SYS) || prio < ERTS_PSFLGS_GET_ACT_PRIO(state)) { @@ -10850,6 +11066,8 @@ erts_get_exact_total_reductions(Process *c_p, Uint *redsp, Uint *diffp) erts_smp_proc_lock(c_p, ERTS_PROC_LOCK_MAIN); } +static void delete_process(Process* p); + void erts_free_proc(Process *p) { @@ -10858,6 +11076,8 @@ erts_free_proc(Process *p) #endif ASSERT(erts_smp_atomic32_read_nob(&p->state) & ERTS_PSFLG_FREE); ASSERT(0 == erts_proc_read_refc(p)); + if (p->flags & F_DELAYED_DEL_PROC) + delete_process(p); erts_free(ERTS_ALC_T_PROC, (void *) p); } @@ -11021,9 +11241,13 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). p->min_heap_size = so->min_heap_size; p->min_vheap_size = so->min_vheap_size; p->max_gen_gcs = so->max_gen_gcs; + MAX_HEAP_SIZE_SET(p, so->max_heap_size); + MAX_HEAP_SIZE_FLAGS_SET(p, so->max_heap_flags); } else { p->min_heap_size = H_MIN_SIZE; p->min_vheap_size = BIN_VH_MIN_SIZE; + MAX_HEAP_SIZE_SET(p, H_MAX_SIZE); + MAX_HEAP_SIZE_FLAGS_SET(p, H_MAX_FLAGS); p->max_gen_gcs = (Uint16) erts_smp_atomic32_read_nob(&erts_max_gen_gcs); } p->schedule_count = 0; @@ -11486,18 +11710,36 @@ erts_cleanup_empty_process(Process* p) #endif } -/* - * p must be the currently executing process. - */ static void delete_process(Process* p) { Eterm *heap; ErtsPSD *psd; + struct saved_calls *scb; + process_breakpoint_time_t *pbt; + void *nif_export; + VERBOSE(DEBUG_PROCESSES, ("Removing process: %T\n",p->common.id)); VERBOSE(DEBUG_SHCOPY, ("[pid=%T] delete process: %p %p %p %p\n", p->common.id, HEAP_START(p), HEAP_END(p), OLD_HEAP(p), OLD_HEND(p))); + scb = ERTS_PROC_SET_SAVED_CALLS_BUF(p, NULL); + + if (scb) { + p->fcalls += CONTEXT_REDS; /* Reduction counting depends on this... */ + erts_free(ERTS_ALC_T_CALLS_BUF, (void *) scb); + } + + pbt = ERTS_PROC_SET_CALL_TIME(p, NULL); + if (pbt) + erts_free(ERTS_ALC_T_BPD, (void *) pbt); + + nif_export = ERTS_PROC_SET_NIF_TRAP_EXPORT(p, NULL); + if (nif_export) + erts_destroy_nif_export(nif_export); + + clean_dirty_start(p); + /* Cleanup psd */ psd = (ErtsPSD *) erts_smp_atomic_read_nob(&p->psd); @@ -11600,7 +11842,8 @@ set_proc_exiting(Process *p, p->i = (BeamInstr *) beam_exit; #ifndef ERTS_SMP - if (state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS)) { + if (state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS) + && !(state & ERTS_PSFLG_GC)) { /* * I non smp case: * @@ -11629,7 +11872,10 @@ set_proc_self_exiting(Process *c_p) ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(c_p) == ERTS_PROC_LOCKS_ALL); state = erts_smp_atomic32_read_nob(&c_p->state); - ASSERT(state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS)); + ASSERT(state & (ERTS_PSFLG_RUNNING + |ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)); #ifdef DEBUG enqueue = @@ -11679,51 +11925,73 @@ erts_handle_pending_exit(Process *c_p, ErtsProcLocks locks) erts_smp_proc_unlock(c_p, xlocks); } +static void save_pending_exiter(Process *p, ErtsProcList *plp); + static void -handle_pending_exiters(ErtsProcList *pnd_xtrs) +do_handle_pending_exiters(ErtsProcList *pnd_xtrs) { /* 'list' is expected to have been fetched (i.e. not a ring anymore) */ ErtsProcList *plp = pnd_xtrs; while (plp) { - ErtsProcList *free_plp; - Process *p = erts_pid2proc(NULL, 0, plp->pid, ERTS_PROC_LOCKS_ALL); + ErtsProcList *next_plp = plp->next; + Process *p = erts_proc_lookup(plp->pid); if (p) { - if (erts_proclist_same(plp, p)) { - erts_aint32_t state = erts_smp_atomic32_read_acqb(&p->state); - if (!(state & (ERTS_PSFLG_RUNNING|ERTS_PSFLG_RUNNING_SYS))) { - ASSERT(state & ERTS_PSFLG_PENDING_EXIT); - erts_handle_pending_exit(p, ERTS_PROC_LOCKS_ALL); + erts_aint32_t state; + /* + * If the process is running on a normal scheduler, the + * pending exit will soon be detected and handled by the + * scheduler running the process (at schedule in/out). + */ + if (erts_smp_proc_trylock(p, ERTS_PROC_LOCKS_ALL) != EBUSY) { + if (erts_proclist_same(plp, p)) { + state = erts_smp_atomic32_read_acqb(&p->state); + if (!(state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_EXITING))) { + ASSERT(state & ERTS_PSFLG_PENDING_EXIT); + erts_handle_pending_exit(p, ERTS_PROC_LOCKS_ALL); + } + } + erts_smp_proc_unlock(p, ERTS_PROC_LOCKS_ALL); + } + else { + erts_smp_proc_lock(p, ERTS_PROC_LOCK_STATUS); + if (erts_proclist_same(plp, p)) { + state = erts_smp_atomic32_read_acqb(&p->state); + if (!(state & (ERTS_PSFLG_RUNNING + | ERTS_PSFLG_RUNNING_SYS + | ERTS_PSFLG_EXITING))) { + /* + * Save process and try to acquire all + * locks at a later time... + */ + save_pending_exiter(p, plp); + plp = NULL; + } } + erts_smp_proc_unlock(p, ERTS_PROC_LOCK_STATUS); } - erts_smp_proc_unlock(p, ERTS_PROC_LOCKS_ALL); } - free_plp = plp; - plp = plp->next; - proclist_destroy(free_plp); + if (plp) + proclist_destroy(plp); + plp = next_plp; } } static void -save_pending_exiter(Process *p) +save_pending_exiter(Process *p, ErtsProcList *plp) { - ErtsProcList *plp; + ErtsSchedulerSleepInfo *ssi; ErtsRunQueue *rq; - ErtsSchedulerData *esdp = erts_get_scheduler_data(); ERTS_SMP_LC_ASSERT(ERTS_PROC_LOCK_STATUS & erts_proc_lc_my_proc_locks(p)); - if (!esdp) - rq = RUNQ_READ_RQ(&p->run_queue); - else - rq = esdp->run_queue; - -#ifdef ERTS_DIRTY_SCHEDULERS - if (ERTS_RUNQ_IX_IS_DIRTY(rq->ix)) - rq = ERTS_RUNQ_IX(0); /* Handle on ordinary scheduler */ -#endif + rq = RUNQ_READ_RQ(&p->run_queue); + ASSERT(rq && !ERTS_RUNQ_IX_IS_DIRTY(rq->ix)); - plp = proclist_create(p); + if (!plp) + plp = proclist_create(p); erts_smp_runq_lock(rq); @@ -11731,9 +11999,11 @@ save_pending_exiter(Process *p) non_empty_runq(rq); + ssi = rq->scheduler->ssi; + erts_smp_runq_unlock(rq); - wake_scheduler(rq); + set_aux_work_flags_wakeup_nob(ssi, ERTS_SSI_AUX_WORK_PENDING_EXITERS); } #endif @@ -11766,7 +12036,7 @@ send_exit_message(Process *to, ErtsProcLocks *to_locksp, mp = erts_alloc_message_heap(to, to_locksp, term_size, &hp, &ohp); mess = copy_struct(exit_term, term_size, &hp, ohp); #endif - erts_queue_message(to, to_locksp, mp, mess); + erts_queue_message(to, *to_locksp, mp, mess, am_system); } else { Eterm temp_token; Uint sz_token; @@ -11787,7 +12057,7 @@ send_exit_message(Process *to, ErtsProcLocks *to_locksp, seq_trace_output(token, mess, SEQ_TRACE_SEND, to->common.id, to); temp_token = copy_struct(token, sz_token, &hp, ohp); ERL_MESSAGE_TOKEN(mp) = temp_token; - erts_queue_message(to, to_locksp, mp, mess); + erts_queue_message(to, *to_locksp, mp, mess, am_system); } } @@ -11933,7 +12203,7 @@ send_exit_signal(Process *c_p, /* current process if and only if (need_locks && erts_smp_proc_trylock(rp, need_locks) == EBUSY) { /* ... but we havn't got all locks on it ... */ - save_pending_exiter(rp); + save_pending_exiter(rp, NULL); /* * The pending exit will be discovered when next * process is scheduled in @@ -12401,10 +12671,8 @@ erts_continue_exit_process(Process *p) ErtsProcLocks curr_locks = ERTS_PROC_LOCK_MAIN; Eterm reason = p->fvalue; DistEntry *dep; - struct saved_calls *scb; - process_breakpoint_time_t *pbt; erts_aint32_t state; - void *nif_export; + int delay_del_proc = 0; #ifdef DEBUG int yield_allowed = 1; @@ -12547,7 +12815,7 @@ erts_continue_exit_process(Process *p) { /* Do *not* use erts_get_runq_proc() */ ErtsRunQueue *rq; - rq = erts_get_runq_current(ERTS_GET_SCHEDULER_DATA_FROM_PROC(p)); + rq = erts_get_runq_current(erts_proc_sched_data(p)); erts_smp_runq_lock(rq); @@ -12593,16 +12861,24 @@ erts_continue_exit_process(Process *p) break; } +#ifdef ERTS_DIRTY_SCHEDULERS + if (a & (ERTS_PSFLG_DIRTY_RUNNING + | ERTS_PSFLG_DIRTY_RUNNING_SYS)) { + p->flags |= F_DELAYED_DEL_PROC; + delay_del_proc = 1; + /* + * The dirty scheduler will also decrease + * refc when done... + */ + erts_proc_inc_refc(p); + } +#endif + if (refc_inced && !(n & ERTS_PSFLG_IN_RUNQ)) erts_proc_dec_refc(p); } dep = (p->flags & F_DISTRIBUTION) ? erts_this_dist_entry : NULL; - scb = ERTS_PROC_SET_SAVED_CALLS_BUF(p, NULL); - if (scb) - p->fcalls += CONTEXT_REDS; /* Reduction counting depends on this... */ - pbt = ERTS_PROC_SET_CALL_TIME(p, NULL); - nif_export = ERTS_PROC_SET_NIF_TRAP_EXPORT(p, NULL); erts_smp_proc_unlock(p, ERTS_PROC_LOCKS_ALL); #ifdef BM_COUNTERS @@ -12642,22 +12918,14 @@ erts_continue_exit_process(Process *p) have none here */ } - if (scb) - erts_free(ERTS_ALC_T_CALLS_BUF, (void *) scb); - - if (pbt) - erts_free(ERTS_ALC_T_BPD, (void *) pbt); - - if (nif_export) - erts_destroy_nif_export(nif_export); - #ifdef ERTS_SMP erts_flush_trace_messages(p, 0); #endif ERTS_TRACER_CLEAR(&ERTS_TRACER(p)); - delete_process(p); + if (!delay_del_proc) + delete_process(p); #ifdef ERTS_SMP erts_smp_proc_lock(p, ERTS_PROC_LOCK_MAIN); @@ -12687,6 +12955,7 @@ erts_continue_exit_process(Process *p) ERTS_SMP_LC_ASSERT(ERTS_PROC_LOCK_MAIN == erts_proc_lc_my_proc_locks(p)); + BUMP_ALL_REDS(p); } /* @@ -13006,11 +13275,13 @@ void erts_halt(int code) int erts_dbg_check_halloc_lock(Process *p) { + ErtsSchedulerData *esdp; if (ERTS_PROC_LOCK_MAIN & erts_proc_lc_my_proc_locks(p)) return 1; if (p->common.id == ERTS_INVALID_PID) return 1; - if (p->scheduler_data && p == p->scheduler_data->match_pseudo_process) + esdp = erts_proc_sched_data(p); + if (esdp && p == esdp->match_pseudo_process) return 1; if (erts_thr_progress_is_blocking()) return 1; diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index 59da9c1779..2801947613 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -304,6 +304,7 @@ typedef enum { ERTS_SSI_AUX_WORK_ASYNC_READY_CLEAN_IX, ERTS_SSI_AUX_WORK_MISC_THR_PRGR_IX, ERTS_SSI_AUX_WORK_MISC_IX, + ERTS_SSI_AUX_WORK_PENDING_EXITERS_IX, ERTS_SSI_AUX_WORK_SET_TMO_IX, ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK_IX, ERTS_SSI_AUX_WORK_REAP_PORTS_IX, @@ -336,6 +337,8 @@ typedef enum { (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_MISC_THR_PRGR_IX) #define ERTS_SSI_AUX_WORK_MISC \ (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_MISC_IX) +#define ERTS_SSI_AUX_WORK_PENDING_EXITERS \ + (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_PENDING_EXITERS_IX) #define ERTS_SSI_AUX_WORK_SET_TMO \ (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_SET_TMO_IX) #define ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK \ @@ -645,6 +648,7 @@ struct ErtsSchedulerData_ { Uint no; /* Scheduler number for normal schedulers */ #ifdef ERTS_DIRTY_SCHEDULERS ErtsDirtySchedId dirty_no; /* Scheduler number for dirty schedulers */ + Process *dirty_shadow_process; #endif Port *current_port; ErtsRunQueue *run_queue; @@ -805,14 +809,26 @@ erts_smp_reset_max_len(ErtsRunQueue *rq, ErtsRunQueueInfo *rqi) #define ERTS_PSD_CALL_TIME_BP 3 #define ERTS_PSD_DELAYED_GC_TASK_QS 4 #define ERTS_PSD_NIF_TRAP_EXPORT 5 -#ifdef HIPE #define ERTS_PSD_SUSPENDED_SAVED_CALLS_BUF 6 -#endif - -#ifdef HIPE -#define ERTS_PSD_SIZE 7 -#else -#define ERTS_PSD_SIZE 6 +#define ERTS_PSD_DIRTY_CPU_START 7 + +#define ERTS_PSD_SIZE 8 + +#if !defined(HIPE) && !defined(ERTS_DIRTY_SCHEDULERS) +# undef ERTS_PSD_SUSPENDED_SAVED_CALLS_BUF +# undef ERTS_PSD_DIRTY_CPU_START +# undef ERTS_PSD_SIZE +# define ERTS_PSD_SIZE 6 +#elif !defined(HIPE) +# undef ERTS_PSD_SUSPENDED_SAVED_CALLS_BUF +# undef ERTS_PSD_DIRTY_CPU_START +# undef ERTS_PSD_SIZE +# define ERTS_PSD_DIRTY_CPU_START 6 +# define ERTS_PSD_SIZE 7 +#elif !defined(ERTS_DIRTY_SCHEDULERS) +# undef ERTS_PSD_DIRTY_CPU_START +# undef ERTS_PSD_SIZE +# define ERTS_PSD_SIZE 7 #endif typedef struct { @@ -918,6 +934,15 @@ struct ErtsPendingSuspend_ { # define BIN_OLD_VHEAP_SZ(p) (p)->bin_old_vheap_sz # define BIN_OLD_VHEAP(p) (p)->bin_old_vheap +# define MAX_HEAP_SIZE_GET(p) ((p)->max_heap_size >> 2) +# define MAX_HEAP_SIZE_SET(p, sz) ((p)->max_heap_size = ((sz) << 2) | \ + MAX_HEAP_SIZE_FLAGS_GET(p)) +# define MAX_HEAP_SIZE_FLAGS_GET(p) ((p)->max_heap_size & 0x3) +# define MAX_HEAP_SIZE_FLAGS_SET(p, flags) ((p)->max_heap_size = flags | \ + ((p)->max_heap_size & ~0x3)) +# define MAX_HEAP_SIZE_KILL 1 +# define MAX_HEAP_SIZE_LOG 2 + struct process { ErtsPTabElementCommon common; /* *Need* to be first in struct */ @@ -935,6 +960,7 @@ struct process { Uint heap_sz; /* Size of heap in words */ Uint min_heap_size; /* Minimum size of heap (in words). */ Uint min_vheap_size; /* Minimum size of virtual heap (in words). */ + Uint max_heap_size; /* Maximum size of heap (in words). */ #if !defined(NO_FPE_SIGNALS) || defined(HIPE) volatile unsigned long fp_exception; @@ -1183,7 +1209,10 @@ void erts_check_for_holes(Process* p); #define ERTS_PSFLG_DIRTY_CPU_PROC ERTS_PSFLG_BIT(20) #define ERTS_PSFLG_DIRTY_IO_PROC ERTS_PSFLG_BIT(21) #define ERTS_PSFLG_DIRTY_ACTIVE_SYS ERTS_PSFLG_BIT(22) -#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 22) +#define ERTS_PSFLG_DIRTY_RUNNING ERTS_PSFLG_BIT(23) +#define ERTS_PSFLG_DIRTY_RUNNING_SYS ERTS_PSFLG_BIT(24) + +#define ERTS_PSFLG_MAX (ERTS_PSFLGS_ZERO_BIT_OFFSET + 24) #define ERTS_PSFLGS_DIRTY_WORK (ERTS_PSFLG_DIRTY_CPU_PROC \ | ERTS_PSFLG_DIRTY_IO_PROC \ @@ -1194,6 +1223,11 @@ void erts_check_for_holes(Process* p); | ERTS_PSFLG_IN_PRQ_NORMAL \ | ERTS_PSFLG_IN_PRQ_LOW) +#define ERTS_PSFLGS_VOLATILE_HEAP (ERTS_PSFLG_EXITING \ + | ERTS_PSFLG_PENDING_EXIT \ + | ERTS_PSFLG_DIRTY_RUNNING \ + | ERTS_PSFLG_DIRTY_RUNNING_SYS) + #define ERTS_PSFLGS_GET_ACT_PRIO(PSFLGS) \ (((PSFLGS) >> ERTS_PSFLGS_ACT_PRIO_OFFSET) & ERTS_PSFLGS_PRIO_MASK) #define ERTS_PSFLGS_GET_USR_PRIO(PSFLGS) \ @@ -1235,6 +1269,7 @@ void erts_check_for_holes(Process* p); * Static flags that do not change after process creation. */ #define ERTS_STC_FLG_SYSTEM_PROC (((Uint32) 1) << 0) +#define ERTS_STC_FLG_SHADOW_PROC (((Uint32) 1) << 1) /* The sequential tracing token is a tuple of size 5: * @@ -1285,6 +1320,8 @@ typedef struct { Uint min_vheap_size; /* Minimum virtual heap size */ int priority; /* Priority for process. */ Uint16 max_gen_gcs; /* Maximum number of gen GCs before fullsweep. */ + Uint max_heap_size; /* Maximum heap size in words */ + Uint max_heap_flags; /* Maximum heap flags (kill | log) */ int scheduler; } ErlSpawnOpts; @@ -1363,6 +1400,7 @@ extern int erts_system_profile_ts_type; #define F_SCHDLR_ONLN_WAITQ (1 << 17) /* Process enqueued waiting to change schedulers online */ #define F_HAVE_BLCKD_NMSCHED (1 << 18) /* Process has blocked normal multi-scheduling */ #define F_HIPE_MODE (1 << 19) +#define F_DELAYED_DEL_PROC (1 << 20) /* Delay delete process (dirty proc exit case) */ /* * F_DISABLE_GC and F_DELAY_GC are similar. Both will prevent @@ -1783,7 +1821,8 @@ erts_aint32_t erts_set_aux_work_timeout(int, erts_aint32_t, int); void erts_sched_notify_check_cpu_bind(void); Uint erts_active_schedulers(void); void erts_init_process(int, int, int); -Eterm erts_process_status(Process *, ErtsProcLocks, Process *, Eterm); +Eterm erts_process_state2status(erts_aint32_t); +Eterm erts_process_status(Process *, Eterm); Uint erts_run_queues_len(Uint *, int, int); void erts_add_to_runq(Process *); Eterm erts_bound_schedulers_term(Process *c_p); @@ -1860,19 +1899,11 @@ int erts_debug_wait_completed(Process *c_p, int flags); Uint erts_process_memory(Process *c_p, int incl_msg_inq); -#ifdef ERTS_SMP -# define ERTS_GET_SCHEDULER_DATA_FROM_PROC(PROC) ((PROC)->scheduler_data) -# define ERTS_PROC_GET_SCHDATA(PROC) ((PROC)->scheduler_data) -#else -# define ERTS_GET_SCHEDULER_DATA_FROM_PROC(PROC) (erts_scheduler_data) -# define ERTS_PROC_GET_SCHDATA(PROC) (erts_scheduler_data) -#endif - #ifdef ERTS_DO_VERIFY_UNUSED_TEMP_ALLOC # define ERTS_VERIFY_UNUSED_TEMP_ALLOC(P) \ do { \ ErtsSchedulerData *esdp__ = ((P) \ - ? ERTS_PROC_GET_SCHDATA((Process *) (P)) \ + ? erts_proc_sched_data((Process *) (P)) \ : erts_get_scheduler_data()); \ if (esdp__ && !ERTS_SCHEDULER_IS_DIRTY(esdp__)) \ esdp__->verify_unused_temp_alloc( \ @@ -1965,12 +1996,15 @@ erts_psd_set(Process *p, int ix, void *data) ErtsPSD *psd; #if defined(ERTS_SMP) && defined(ERTS_ENABLE_LOCK_CHECK) ErtsProcLocks locks = erts_proc_lc_my_proc_locks(p); - if (ERTS_LC_PSD_ANY_LOCK == erts_psd_required_locks[ix].set_locks) - ERTS_SMP_LC_ASSERT(locks || erts_thr_progress_is_blocking()); - else { - locks &= erts_psd_required_locks[ix].set_locks; - ERTS_SMP_LC_ASSERT(erts_psd_required_locks[ix].set_locks == locks - || erts_thr_progress_is_blocking()); + erts_aint32_t state = state = erts_smp_atomic32_read_nob(&p->state); + if (!(state & ERTS_PSFLG_FREE)) { + if (ERTS_LC_PSD_ANY_LOCK == erts_psd_required_locks[ix].set_locks) + ERTS_SMP_LC_ASSERT(locks || erts_thr_progress_is_blocking()); + else { + locks &= erts_psd_required_locks[ix].set_locks; + ERTS_SMP_LC_ASSERT(erts_psd_required_locks[ix].set_locks == locks + || erts_thr_progress_is_blocking()); + } } #endif psd = (ErtsPSD *) erts_smp_atomic_read_nob(&p->psd); @@ -2027,6 +2061,13 @@ erts_psd_set(Process *p, int ix, void *data) ((struct saved_calls *) erts_psd_set((P), ERTS_PSD_SUSPENDED_SAVED_CALLS_BUF, (void *) (SCB))) #endif +#ifdef ERTS_DIRTY_SCHEDULERS +#define ERTS_PROC_GET_DIRTY_CPU_START(P) \ + ((void *) erts_psd_get((P), ERTS_PSD_DIRTY_CPU_START)) +#define ERTS_PROC_SET_DIRTY_CPU_START(P, DCS) \ + ((void *) erts_psd_set((P), ERTS_PSD_DIRTY_CPU_START, (void *) (DCS))) +#endif + ERTS_GLB_INLINE Eterm erts_proc_get_error_handler(Process *p); ERTS_GLB_INLINE Eterm erts_proc_set_error_handler(Process *p, Eterm handler); @@ -2169,6 +2210,7 @@ erts_check_emigration_need(ErtsRunQueue *c_rq, int prio) #endif +ERTS_GLB_INLINE ErtsSchedulerData *erts_proc_sched_data(Process *c_p); ERTS_GLB_INLINE int erts_is_scheduler_bound(ErtsSchedulerData *esdp); ERTS_GLB_INLINE Process *erts_get_current_process(void); ERTS_GLB_INLINE Eterm erts_get_current_pid(void); @@ -2202,6 +2244,31 @@ ERTS_GLB_INLINE void erts_shrink_message_heap(ErtsMessage **msgpp, Process *pp, #if ERTS_GLB_INLINE_INCL_FUNC_DEF ERTS_GLB_INLINE +ErtsSchedulerData *erts_proc_sched_data(Process *c_p) +{ + ErtsSchedulerData *esdp; + ASSERT(c_p); +#if !defined(ERTS_SMP) + esdp = erts_get_scheduler_data(); +#else + esdp = c_p->scheduler_data; +# if defined(ERTS_DIRTY_SCHEDULERS) + if (esdp) { + ASSERT(esdp == erts_get_scheduler_data()); + ASSERT(!ERTS_SCHEDULER_IS_DIRTY(esdp)); + } + else { + esdp = erts_get_scheduler_data(); + ASSERT(esdp); + ASSERT(ERTS_SCHEDULER_IS_DIRTY(esdp)); + } +# endif +#endif + ASSERT(esdp); + return esdp; +} + +ERTS_GLB_INLINE int erts_is_scheduler_bound(ErtsSchedulerData *esdp) { if (!esdp) @@ -2416,7 +2483,7 @@ ERTS_GLB_INLINE ErtsAtomCacheMap * erts_get_atom_cache_map(Process *c_p) { ErtsSchedulerData *esdp = (c_p - ? ERTS_PROC_GET_SCHDATA(c_p) + ? erts_proc_sched_data(c_p) : erts_get_scheduler_data()); ASSERT(esdp); return &esdp->atom_cache_map; diff --git a/erts/emulator/beam/erl_process_dump.c b/erts/emulator/beam/erl_process_dump.c index fa76773cac..eeaa9a569c 100644 --- a/erts/emulator/beam/erl_process_dump.c +++ b/erts/emulator/beam/erl_process_dump.c @@ -568,23 +568,21 @@ dump_externally(int to, void *to_arg, Eterm term) } } -void erts_dump_process_state(int to, void *to_arg, erts_aint32_t psflg) { - if (psflg & ERTS_PSFLG_FREE) - erts_print(to, to_arg, "Non Existing\n"); /* Should never happen */ - else if (psflg & ERTS_PSFLG_EXITING) - erts_print(to, to_arg, "Exiting\n"); - else if (psflg & ERTS_PSFLG_GC) { - erts_print(to, to_arg, "Garbing\n"); - } - else if (psflg & ERTS_PSFLG_SUSPENDED) - erts_print(to, to_arg, "Suspended\n"); - else if (psflg & ERTS_PSFLG_RUNNING) { - erts_print(to, to_arg, "Running\n"); +void erts_dump_process_state(int to, void *to_arg, erts_aint32_t psflg) +{ + char *s; + switch (erts_process_state2status(psflg)) { + case am_free: s = "Non Existing"; break; /* Should never happen */ + case am_exiting: s = "Exiting"; break; + case am_garbage_collecting: s = "Garbing"; break; + case am_suspended: s = "Suspended"; break; + case am_running: s = "Running"; break; + case am_runnable: s = "Scheduled"; break; + case am_waiting: s = "Waiting"; break; + default: s = "Undefined"; break; /* Should never happen */ } - else if (psflg & ERTS_PSFLG_ACTIVE) - erts_print(to, to_arg, "Scheduled\n"); - else - erts_print(to, to_arg, "Waiting\n"); + + erts_print(to, to_arg, "%s\n", s); } void @@ -668,6 +666,10 @@ erts_dump_extended_process_state(int to, void *to_arg, erts_aint32_t psflg) { erts_print(to, to_arg, "DIRTY_IO_PROC"); break; case ERTS_PSFLG_DIRTY_ACTIVE_SYS: erts_print(to, to_arg, "DIRTY_ACTIVE_SYS"); break; + case ERTS_PSFLG_DIRTY_RUNNING: + erts_print(to, to_arg, "DIRTY_RUNNING"); break; + case ERTS_PSFLG_DIRTY_RUNNING_SYS: + erts_print(to, to_arg, "DIRTY_RUNNING_SYS"); break; default: erts_print(to, to_arg, "UNKNOWN(%d)", chk); break; } diff --git a/erts/emulator/beam/erl_time_sup.c b/erts/emulator/beam/erl_time_sup.c index 346404fe2a..9e37106b88 100644 --- a/erts/emulator/beam/erl_time_sup.c +++ b/erts/emulator/beam/erl_time_sup.c @@ -1966,7 +1966,7 @@ send_time_offset_changed_notifications(void *new_offsetp) *patch_refp = ref; ASSERT(hsz == size_object(message_template)); message = copy_struct(message_template, hsz, &hp, ohp); - erts_queue_message(rp, &rp_locks, mp, message); + erts_queue_message(rp, rp_locks, mp, message, am_clock_service); } erts_smp_proc_unlock(rp, rp_locks); } @@ -2348,7 +2348,7 @@ erts_napi_convert_time_unit(ErtsMonotonicTime val, int from, int to) BIF_RETTYPE monotonic_time_0(BIF_ALIST_0) { ErtsMonotonicTime mtime = time_sup.r.o.get_time(); - update_last_mtime(ERTS_PROC_GET_SCHDATA(BIF_P), mtime); + update_last_mtime(erts_proc_sched_data(BIF_P), mtime); mtime += ERTS_MONOTONIC_OFFSET_NATIVE; BIF_RET(make_time_val(BIF_P, mtime)); } @@ -2356,7 +2356,7 @@ BIF_RETTYPE monotonic_time_0(BIF_ALIST_0) BIF_RETTYPE monotonic_time_1(BIF_ALIST_1) { ErtsMonotonicTime mtime = time_sup.r.o.get_time(); - update_last_mtime(ERTS_PROC_GET_SCHDATA(BIF_P), mtime); + update_last_mtime(erts_proc_sched_data(BIF_P), mtime); BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, mtime, 1)); } @@ -2365,7 +2365,7 @@ BIF_RETTYPE system_time_0(BIF_ALIST_0) ErtsMonotonicTime mtime, offset; mtime = time_sup.r.o.get_time(); offset = get_time_offset(); - update_last_mtime(ERTS_PROC_GET_SCHDATA(BIF_P), mtime); + update_last_mtime(erts_proc_sched_data(BIF_P), mtime); BIF_RET(make_time_val(BIF_P, mtime + offset)); } @@ -2374,7 +2374,7 @@ BIF_RETTYPE system_time_1(BIF_ALIST_0) ErtsMonotonicTime mtime, offset; mtime = time_sup.r.o.get_time(); offset = get_time_offset(); - update_last_mtime(ERTS_PROC_GET_SCHDATA(BIF_P), mtime); + update_last_mtime(erts_proc_sched_data(BIF_P), mtime); BIF_RET(time_unit_conversion(BIF_P, BIF_ARG_1, mtime + offset, 0)); } @@ -2404,7 +2404,7 @@ BIF_RETTYPE timestamp_0(BIF_ALIST_0) mtime = time_sup.r.o.get_time(); offset = get_time_offset(); - update_last_mtime(ERTS_PROC_GET_SCHDATA(BIF_P), mtime); + update_last_mtime(erts_proc_sched_data(BIF_P), mtime); make_timestamp_value(&mega_sec, &sec, µ_sec, mtime, offset); diff --git a/erts/emulator/beam/erl_trace.c b/erts/emulator/beam/erl_trace.c index 436b4aca21..ca001fc156 100644 --- a/erts/emulator/beam/erl_trace.c +++ b/erts/emulator/beam/erl_trace.c @@ -238,7 +238,6 @@ write_timestamp(ErtsTraceTimeStamp *tsp, Eterm **hpp) } #ifdef ERTS_SMP -#define PATCH_TS_SIZE(p) patch_ts_size(TFLGS_TS_TYPE(p)) static ERTS_INLINE Uint patch_ts_size(int ts_type) @@ -258,7 +257,7 @@ patch_ts_size(int ts_type) return 0; } } -#endif +#endif /* ERTS_SMP */ /* * Write a timestamp. The timestamp MUST be the last @@ -394,9 +393,10 @@ static ERTS_INLINE int send_to_tracer_nif(Process *c_p, ErtsPTabElementCommon *t_p, Eterm t_p_id, ErtsTracerNif *tnif, enum ErtsTracerOpt topt, - Eterm tag, Eterm msg, Eterm extra); + Eterm tag, Eterm msg, Eterm extra, + Eterm pam_result); static ERTS_INLINE Eterm -call_enabled_tracer(Process *c_p, const ErtsTracer tracer, +call_enabled_tracer(const ErtsTracer tracer, ErtsTracerNif **tnif_ref, enum ErtsTracerOpt topt, Eterm tag, Eterm t_p_id); @@ -459,8 +459,7 @@ erts_set_system_seq_tracer(Process *c_p, ErtsProcLocks c_p_locks, ErtsTracer new if (!ERTS_TRACER_IS_NIL(new)) { Eterm nif_result = call_enabled_tracer( - NULL, new, NULL, - TRACE_FUN_ENABLED, am_trace_status, am_undefined); + new, NULL, TRACE_FUN_ENABLED, am_trace_status, am_undefined); switch (nif_result) { case am_trace: break; default: @@ -492,7 +491,7 @@ erts_get_system_seq_tracer(void) erts_smp_rwmtx_runlock(&sys_trace_rwmtx); if (st != erts_tracer_nil && - call_enabled_tracer(NULL, st, NULL, TRACE_FUN_ENABLED, + call_enabled_tracer(st, NULL, TRACE_FUN_ENABLED, am_trace_status, am_undefined) == am_remove) { erts_set_system_seq_tracer(NULL, 0, erts_tracer_nil); st = erts_tracer_nil; @@ -513,7 +512,7 @@ get_default_tracing(Uint *flagsp, ErtsTracer *tracerp, *default_trace_flags &= ~TRACEE_FLAGS; } else { Eterm nif_res; - nif_res = call_enabled_tracer(NULL, *default_tracer, + nif_res = call_enabled_tracer(*default_tracer, NULL, TRACE_FUN_ENABLED, am_trace_status, am_undefined); switch (nif_res) { @@ -739,7 +738,7 @@ profile_send(Eterm from, Eterm message) { else msg = copy_struct(message, sz, &hp, &mp->hfrag.off_heap); - erts_queue_message(profile_p, NULL, mp, msg); + erts_queue_message(profile_p, 0, mp, msg, from); } } @@ -787,7 +786,7 @@ trace_sched_aux(Process *p, ErtsProcLocks locks, Eterm what) } send_to_tracer_nif(p, &p->common, p->common.id, tnif, TRACE_FUN_T_SCHED_PROC, - what, tmp, THE_NON_VALUE); + what, tmp, THE_NON_VALUE, am_true); } /* Send {trace_ts, Pid, What, {Mod, Func, Arity}, Timestamp} @@ -812,9 +811,32 @@ trace_send(Process *p, Eterm to, Eterm msg) { Eterm operation = am_send; ErtsTracerNif *tnif = NULL; + ErtsTracingEvent* te; + Eterm pam_result; ASSERT(ARE_TRACE_FLAGS_ON(p, F_TRACE_SEND)); + te = &erts_send_tracing[erts_active_bp_ix()]; + if (!te->on) { + return; + } + if (te->match_spec) { + Eterm args[2]; + Uint32 return_flags; + args[0] = to; + args[1] = msg; + pam_result = erts_match_set_run_trace(p, p, + te->match_spec, args, 2, + ERTS_PAM_TMP_RESULT, &return_flags); + if (pam_result == am_false) + return; + if (ERTS_TRACE_FLAGS(p) & F_TRACE_SILENT) { + erts_match_set_release_result_trace(p, pam_result); + return; + } + } else + pam_result = am_true; + if (is_internal_pid(to)) { if (!erts_proc_lookup(to)) goto send_to_non_existing_process; @@ -826,23 +848,63 @@ trace_send(Process *p, Eterm to, Eterm msg) } if (is_tracer_enabled(p, ERTS_PROC_LOCK_MAIN, &p->common, &tnif, - TRACE_FUN_E_SEND, operation)) + TRACE_FUN_E_SEND, operation)) { send_to_tracer_nif(p, &p->common, p->common.id, tnif, TRACE_FUN_T_SEND, - operation, msg, to); + operation, msg, to, pam_result); + } + erts_match_set_release_result_trace(p, pam_result); } /* Send {trace_ts, Pid, receive, Msg, Timestamp} * or {trace, Pid, receive, Msg} */ void -trace_receive(Process *c_p, Eterm msg) +trace_receive(Process* receiver, + Eterm from, + Eterm msg, ErtsTracingEvent* te) { ErtsTracerNif *tnif = NULL; - if (is_tracer_enabled(NULL, 0, &c_p->common, &tnif, - TRACE_FUN_E_RECEIVE, am_receive)) - send_to_tracer_nif(NULL, &c_p->common, c_p->common.id, + Eterm pam_result; + + if (!te) { + te = &erts_receive_tracing[erts_active_bp_ix()]; + if (!te->on) + return; + } + else ASSERT(te->on); + + if (te->match_spec) { + Eterm args[3]; + Uint32 return_flags; + if (is_pid(from)) { + args[0] = pid_node_name(from); + args[1] = from; + } + else { + ASSERT(is_atom(from)); + args[0] = from; /* node name or other atom (e.g 'system') */ + args[1] = am_undefined; + } + args[2] = msg; + pam_result = erts_match_set_run_trace(NULL, receiver, + te->match_spec, args, 3, + ERTS_PAM_TMP_RESULT, &return_flags); + if (pam_result == am_false) + return; + if (ERTS_TRACE_FLAGS(receiver) & F_TRACE_SILENT) { + erts_match_set_release_result_trace(NULL, pam_result); + return; + } + } else + pam_result = am_true; + + if (is_tracer_enabled(NULL, 0, &receiver->common, &tnif, + TRACE_FUN_E_RECEIVE, am_receive)) { + send_to_tracer_nif(NULL, &receiver->common, receiver->common.id, tnif, TRACE_FUN_T_RECEIVE, - am_receive, msg, THE_NON_VALUE); + am_receive, msg, THE_NON_VALUE, pam_result); + } + erts_match_set_release_result_trace(NULL, pam_result); } int @@ -852,8 +914,8 @@ seq_trace_update_send(Process *p) ASSERT((is_tuple(SEQ_TRACE_TOKEN(p)) || is_nil(SEQ_TRACE_TOKEN(p)))); if (have_no_seqtrace(SEQ_TRACE_TOKEN(p)) || (seq_tracer != NIL && - call_enabled_tracer(NULL, seq_tracer, NULL, - TRACE_FUN_ENABLED, am_trace_status, + call_enabled_tracer(seq_tracer, NULL, + TRACE_FUN_ENABLED, am_seq_trace, p ? p->common.id : am_undefined) != am_trace) #ifdef USE_VM_PROBES || (SEQ_TRACE_TOKEN(p) == am_have_dt_utag) @@ -900,9 +962,9 @@ seq_trace_output_generic(Eterm token, Eterm msg, Uint type, ASSERT(is_tuple(token) || is_nil(token)); if (token == NIL || (process && ERTS_TRACE_FLAGS(process) & F_SENSITIVE) || ERTS_TRACER_IS_NIL(seq_tracer) || - call_enabled_tracer(NULL, seq_tracer, + call_enabled_tracer(seq_tracer, NULL, TRACE_FUN_ENABLED, - am_trace_status, + am_seq_trace, process ? process->common.id : am_undefined) != am_trace) { return; } @@ -964,7 +1026,7 @@ erts_trace_return_to(Process *p, BeamInstr *pc) } send_to_tracer_nif(p, &p->common, p->common.id, NULL, TRACE_FUN_T_CALL, - am_return_to, mfa, THE_NON_VALUE); + am_return_to, mfa, THE_NON_VALUE, am_true); } @@ -1123,7 +1185,11 @@ erts_call_trace(Process* p, BeamInstr mfa[3], Binary *match_spec, * use process flags */ tracee_flags = &ERTS_TRACE_FLAGS(p); + /* Is is not ideal at all to call this check twice, + it should be optimized so that only one call is made. */ if (!is_tracer_enabled(p, ERTS_PROC_LOCK_MAIN, &p->common, &tnif, + TRACE_FUN_ENABLED, am_trace_status) + || !is_tracer_enabled(p, ERTS_PROC_LOCK_MAIN, &p->common, &tnif, TRACE_FUN_E_CALL, am_call)) { return 0; } @@ -1139,13 +1205,21 @@ erts_call_trace(Process* p, BeamInstr mfa[3], Binary *match_spec, } meta_flags = F_TRACE_CALLS | F_NOW_TS; tracee_flags = &meta_flags; - switch (call_enabled_tracer(p, *tracer, - &tnif, TRACE_FUN_T_CALL, - am_call, p->common.id)) { + switch (call_enabled_tracer(*tracer, + &tnif, TRACE_FUN_ENABLED, + am_trace_status, p->common.id)) { default: case am_remove: *tracer = erts_tracer_nil; case am_discard: return 0; - case am_trace: break; + case am_trace: + switch (call_enabled_tracer(*tracer, + &tnif, TRACE_FUN_T_CALL, + am_call, p->common.id)) { + default: + case am_discard: return 0; + case am_trace: break; + } + break; } } @@ -1202,20 +1276,14 @@ erts_call_trace(Process* p, BeamInstr mfa[3], Binary *match_spec, may remove it, and we still want to generate a trace message */ erts_tracer_update(&pre_ms_tracer, *tracer); tracer = &pre_ms_tracer; - pam_result = erts_match_set_run(p, match_spec, args, arity, - ERTS_PAM_TMP_RESULT, &return_flags); - if (is_non_value(pam_result)) { - erts_match_set_release_result(p); - UnUseTmpHeap(ERL_SUB_BIN_SIZE,p); - ERTS_TRACER_CLEAR(&pre_ms_tracer); - return 0; - } + pam_result = erts_match_set_run_trace(p, p, + match_spec, args, arity, + ERTS_PAM_TMP_RESULT, &return_flags); } if (tracee_flags == &meta_flags) { /* Meta trace */ if (pam_result == am_false) { - erts_match_set_release_result(p); UnUseTmpHeap(ERL_SUB_BIN_SIZE,p); ERTS_TRACER_CLEAR(&pre_ms_tracer); return return_flags; @@ -1223,13 +1291,12 @@ erts_call_trace(Process* p, BeamInstr mfa[3], Binary *match_spec, } else { /* Non-meta trace */ if (*tracee_flags & F_TRACE_SILENT) { - erts_match_set_release_result(p); + erts_match_set_release_result_trace(p, pam_result); UnUseTmpHeap(ERL_SUB_BIN_SIZE,p); ERTS_TRACER_CLEAR(&pre_ms_tracer); return 0; } if (pam_result == am_false) { - erts_match_set_release_result(p); UnUseTmpHeap(ERL_SUB_BIN_SIZE,p); ERTS_TRACER_CLEAR(&pre_ms_tracer); return return_flags; @@ -1265,11 +1332,14 @@ erts_call_trace(Process* p, BeamInstr mfa[3], Binary *match_spec, * Build the trace tuple and send it to the port. */ send_to_tracer_nif_raw(p, NULL, *tracer, *tracee_flags, p->common.id, - tnif, TRACE_FUN_T_CALL, am_call, mfa_tuple, THE_NON_VALUE, pam_result); - erts_match_set_release_result(p); + tnif, TRACE_FUN_T_CALL, am_call, mfa_tuple, + THE_NON_VALUE, pam_result); - if (match_spec && tracer == &pre_ms_tracer) - ERTS_TRACER_CLEAR(&pre_ms_tracer); + if (match_spec) { + erts_match_set_release_result_trace(p, pam_result); + if (tracer == &pre_ms_tracer) + ERTS_TRACER_CLEAR(&pre_ms_tracer); + } return return_flags; } @@ -1287,10 +1357,10 @@ trace_proc(Process *c_p, ErtsProcLocks c_p_locks, Process *t_p, Eterm what, Eterm data) { ErtsTracerNif *tnif = NULL; - if (is_tracer_enabled(c_p, c_p_locks, &t_p->common, &tnif, + if (is_tracer_enabled(NULL, 0, &t_p->common, &tnif, TRACE_FUN_E_PROCS, what)) - send_to_tracer_nif(c_p, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_PROCS, - what, data, THE_NON_VALUE); + send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_PROCS, + what, data, THE_NON_VALUE, am_true); } @@ -1306,17 +1376,16 @@ trace_proc_spawn(Process *p, Eterm what, Eterm pid, Eterm mod, Eterm func, Eterm args) { ErtsTracerNif *tnif = NULL; - if (is_tracer_enabled(p, ERTS_PROC_LOCKS_ALL & - ~(ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE), - &p->common, &tnif, TRACE_FUN_E_PROCS, what)) { + if (is_tracer_enabled(NULL, 0, + &p->common, &tnif, TRACE_FUN_E_PROCS, what)) { Eterm mfa; Eterm* hp; hp = HAlloc(p, 4); mfa = TUPLE3(hp, mod, func, args); hp += 4; - send_to_tracer_nif(p, &p->common, p->common.id, tnif, TRACE_FUN_T_PROCS, - what, pid, mfa); + send_to_tracer_nif(NULL, &p->common, p->common.id, tnif, TRACE_FUN_T_PROCS, + what, pid, mfa, am_true); } } @@ -1350,25 +1419,28 @@ void save_calls(Process *p, Export *e) * are all small (atomic) integers. */ void -trace_gc(Process *p, Eterm what, Uint size) +trace_gc(Process *p, Eterm what, Uint size, Eterm msg) { ErtsTracerNif *tnif = NULL; Eterm* hp; - Eterm msg = NIL; Uint sz = 0; Eterm tup; - if (is_tracer_enabled(p, ERTS_PROC_LOCK_MAIN, &p->common, &tnif, TRACE_FUN_E_GC, what)) { + if (is_tracer_enabled(p, ERTS_PROC_LOCK_MAIN, &p->common, &tnif, + TRACE_FUN_E_GC, what)) { - (void) erts_process_gc_info(p, &sz, NULL); - hp = HAlloc(p, sz + 3 + 2); + if (is_non_value(msg)) { - msg = erts_process_gc_info(p, NULL, &hp); - tup = TUPLE2(hp, am_wordsize, make_small(size)); hp += 3; - msg = CONS(hp, tup, msg); hp += 2; + (void) erts_process_gc_info(p, &sz, NULL, 0, 0); + hp = HAlloc(p, sz + 3 + 2); + + msg = erts_process_gc_info(p, NULL, &hp, 0, 0); + tup = TUPLE2(hp, am_wordsize, make_small(size)); hp += 3; + msg = CONS(hp, tup, msg); hp += 2; + } send_to_tracer_nif(p, &p->common, p->common.id, tnif, TRACE_FUN_T_GC, - what, msg, am_undefined); + what, msg, THE_NON_VALUE, am_true); } } @@ -1431,7 +1503,7 @@ monitor_long_schedule_proc(Process *p, BeamInstr *in_fp, BeamInstr *out_fp, Uint { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(monitor_p, NULL, mp, msg); + erts_queue_message(monitor_p, 0, mp, msg, am_system); } #endif } @@ -1496,7 +1568,7 @@ monitor_long_schedule_port(Port *pp, ErtsPortTaskType type, Uint time) { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(monitor_p, NULL, mp, msg); + erts_queue_message(monitor_p, 0, mp, msg, am_system); } #endif } @@ -1571,7 +1643,7 @@ monitor_long_gc(Process *p, Uint time) { { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(monitor_p, NULL, mp, msg); + erts_queue_message(monitor_p, 0, mp, msg, am_system); } #endif } @@ -1646,7 +1718,7 @@ monitor_large_heap(Process *p) { { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(monitor_p, NULL, mp, msg); + erts_queue_message(monitor_p, 0, mp, msg, am_system); } #endif } @@ -1678,7 +1750,7 @@ monitor_generic(Process *p, Eterm type, Eterm spec) { { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(monitor_p, NULL, mp, msg); + erts_queue_message(monitor_p, 0, mp, msg, am_system); } #endif @@ -1748,7 +1820,7 @@ trace_port_open(Port *p, Eterm calling_pid, Eterm drv_name) { ERTS_SMP_CHK_NO_PROC_LOCKS; if (is_tracer_enabled(NULL, 0, &p->common, &tnif, TRACE_FUN_E_PORTS, am_open)) send_to_tracer_nif(NULL, &p->common, p->common.id, tnif, TRACE_FUN_T_PORTS, - am_open, calling_pid, drv_name); + am_open, calling_pid, drv_name, am_true); } /* Sends trace message: @@ -1767,7 +1839,7 @@ trace_port(Port *t_p, Eterm what, Eterm data) { ERTS_SMP_CHK_NO_PROC_LOCKS; if (is_tracer_enabled(NULL, 0, &t_p->common, &tnif, TRACE_FUN_E_PORTS, what)) send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_PORTS, - what, data, THE_NON_VALUE); + what, data, THE_NON_VALUE, am_true); } @@ -1907,7 +1979,7 @@ trace_port_receive(Port *t_p, Eterm caller, Eterm what, ...) ASSERT(hp <= (local_heap + LOCAL_HEAP_SIZE) || orig_hp); send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_RECEIVE, - am_receive, data, THE_NON_VALUE); + am_receive, data, THE_NON_VALUE, am_true); if (bptr && erts_refc_dectest(&bptr->refc, 1) == 0) erts_bin_free(bptr); @@ -1930,7 +2002,7 @@ trace_port_send(Port *t_p, Eterm receiver, Eterm msg, int exists) ERTS_SMP_CHK_NO_PROC_LOCKS; if (is_tracer_enabled(NULL, 0, &t_p->common, &tnif, TRACE_FUN_E_SEND, op)) send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_SEND, - op, msg, receiver); + op, msg, receiver, am_true); } void trace_port_send_binary(Port *t_p, Eterm to, Eterm what, char *bin, Sint sz) @@ -1959,7 +2031,7 @@ void trace_port_send_binary(Port *t_p, Eterm to, Eterm what, char *bin, Sint sz) hp += 3; send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_SEND, - am_send, msg, to); + am_send, msg, to, am_true); if (bptr && erts_refc_dectest(&bptr->refc, 1) == 0) erts_bin_free(bptr); @@ -1990,7 +2062,7 @@ trace_sched_ports_where(Port *t_p, Eterm what, Eterm where) { if (is_tracer_enabled(NULL, 0, &t_p->common, &tnif, TRACE_FUN_E_SCHED_PORT, what)) send_to_tracer_nif(NULL, &t_p->common, t_p->common.id, tnif, TRACE_FUN_T_SCHED_PORT, - what, where, THE_NON_VALUE); + what, where, THE_NON_VALUE, am_true); } /* Port profiling */ @@ -2430,7 +2502,7 @@ sys_msg_dispatcher_func(void *unused) queue_proc_msg: mp = erts_alloc_message(0, NULL); mp->data.heap_frag = smqp->bp; - erts_queue_message(proc,&proc_locks,mp,smqp->msg); + erts_queue_message(proc,proc_locks,mp,smqp->msg,am_system); #ifdef DEBUG_PRINTOUTS erts_fprintf(stderr, "delivered\n"); #endif @@ -2813,7 +2885,7 @@ send_to_tracer_nif_raw(Process *c_p, Process *tracee, static ERTS_INLINE int send_to_tracer_nif(Process *c_p, ErtsPTabElementCommon *t_p, Eterm t_p_id, ErtsTracerNif *tnif, enum ErtsTracerOpt topt, - Eterm tag, Eterm msg, Eterm extra) + Eterm tag, Eterm msg, Eterm extra, Eterm pam_result) { #if defined(ERTS_SMP) && defined(ERTS_ENABLE_LOCK_CHECK) if (c_p) { @@ -2833,27 +2905,32 @@ send_to_tracer_nif(Process *c_p, ErtsPTabElementCommon *t_p, is_internal_pid(t_p->id) ? (Process*)t_p : NULL, t_p->tracer, t_p->trace_flags, t_p_id, tnif, topt, tag, msg, extra, - am_true); + pam_result); } static ERTS_INLINE Eterm -call_enabled_tracer(Process *c_p, const ErtsTracer tracer, +call_enabled_tracer(const ErtsTracer tracer, ErtsTracerNif **tnif_ret, enum ErtsTracerOpt topt, Eterm tag, Eterm t_p_id) { ErtsTracerNif *tnif = lookup_tracer_nif(tracer); if (tnif) { - Eterm argv[] = {tag, ERTS_TRACER_STATE(tracer), t_p_id}; + Eterm argv[] = {tag, ERTS_TRACER_STATE(tracer), t_p_id}, + ret; topt = (tnif->tracers[topt].cb) ? topt : TRACE_FUN_ENABLED; ASSERT(topt < NIF_TRACER_TYPES); ASSERT(tnif->tracers[topt].cb != NULL); if (tnif_ret) *tnif_ret = tnif; - return erts_nif_call_function(c_p, NULL, tnif->nif_mod, - tnif->tracers[topt].cb, - tnif->tracers[topt].arity, - argv); + ret = erts_nif_call_function(NULL, NULL, tnif->nif_mod, + tnif->tracers[topt].cb, + tnif->tracers[topt].arity, + argv); + if (tag == am_trace_status && ret != am_remove) + return am_trace; + ASSERT(tag == am_trace_status || ret != am_remove); + return ret; } - return am_remove; + return tag == am_trace_status ? am_remove : am_discard; } static int @@ -2878,12 +2955,12 @@ is_tracer_enabled(Process* c_p, ErtsProcLocks c_p_locks, } #endif - nif_result = call_enabled_tracer(c_p, t_p->tracer, tnif_ret, topt, tag, t_p->id); + nif_result = call_enabled_tracer(t_p->tracer, tnif_ret, topt, tag, t_p->id); switch (nif_result) { case am_discard: return 0; case am_trace: return 1; case THE_NON_VALUE: - case am_remove: break; + case am_remove: ASSERT(tag == am_trace_status); break; default: /* only am_remove should be returned, but if something else is returned we fall-through @@ -2914,19 +2991,14 @@ is_tracer_enabled(Process* c_p, ErtsProcLocks c_p_locks, return 0; } -int erts_is_tracer_proc_enabled(Process* c_p, ErtsProcLocks c_p_locks, - ErtsPTabElementCommon *t_p, Eterm type) -{ - return is_tracer_enabled(c_p, c_p_locks, t_p, NULL, TRACE_FUN_ENABLED, am_trace_status); -} - -int erts_is_tracer_enabled(Process *c_p, const ErtsTracer tracer) +int erts_is_tracer_enabled(const ErtsTracer tracer, ErtsPTabElementCommon *t_p) { ErtsTracerNif *tnif = lookup_tracer_nif(tracer); if (tnif) { - Eterm nif_result = call_enabled_tracer(c_p, tracer, &tnif, - TRACE_FUN_ENABLED, am_trace_status, - c_p->common.id); + Eterm nif_result = call_enabled_tracer(tracer, &tnif, + TRACE_FUN_ENABLED, + am_trace_status, + t_p->id); switch (nif_result) { case am_discard: case am_trace: return 1; @@ -2937,6 +3009,20 @@ int erts_is_tracer_enabled(Process *c_p, const ErtsTracer tracer) return 0; } +int erts_is_tracer_proc_enabled(Process* c_p, ErtsProcLocks c_p_locks, + ErtsPTabElementCommon *t_p) +{ + return is_tracer_enabled(c_p, c_p_locks, t_p, NULL, TRACE_FUN_ENABLED, + am_trace_status); +} + +int erts_is_tracer_proc_enabled_send(Process* c_p, ErtsProcLocks c_p_locks, + ErtsPTabElementCommon *t_p) +{ + return is_tracer_enabled(c_p, c_p_locks, t_p, NULL, TRACE_FUN_T_SEND, am_send); +} + + void erts_tracer_replace(ErtsPTabElementCommon *t_p, const ErtsTracer tracer) { #if defined(ERTS_SMP) && defined(ERTS_ENABLE_LOCK_CHECK) diff --git a/erts/emulator/beam/erl_trace.h b/erts/emulator/beam/erl_trace.h index 9a007e62ec..0095d4386b 100644 --- a/erts/emulator/beam/erl_trace.h +++ b/erts/emulator/beam/erl_trace.h @@ -56,6 +56,15 @@ struct binary; +typedef struct +{ + int on; + struct binary* match_spec; +} ErtsTracingEvent; + +extern ErtsTracingEvent erts_send_tracing[]; +extern ErtsTracingEvent erts_receive_tracing[]; + /* erl_bif_trace.c */ Eterm erl_seq_trace_info(Process *p, Eterm arg1); void erts_system_monitor_clear(Process *c_p); @@ -91,7 +100,7 @@ void erts_send_sys_msg_proc(Eterm, Eterm, Eterm, ErlHeapFragment *); #endif void trace_send(Process*, Eterm, Eterm); -void trace_receive(Process *, Eterm); +void trace_receive(Process*, Eterm, Eterm, ErtsTracingEvent*); Uint32 erts_call_trace(Process *p, BeamInstr mfa[], struct binary *match_spec, Eterm* args, int local, ErtsTracer *tracer); void erts_trace_return(Process* p, BeamInstr* fi, Eterm retval, @@ -103,7 +112,7 @@ void trace_sched(Process*, ErtsProcLocks, Eterm); void trace_proc(Process*, ErtsProcLocks, Process*, Eterm, Eterm); void trace_proc_spawn(Process*, Eterm what, Eterm pid, Eterm mod, Eterm func, Eterm args); void save_calls(Process *p, Export *); -void trace_gc(Process *p, Eterm what, Uint size); +void trace_gc(Process *p, Eterm what, Uint size, Eterm msg); /* port tracing */ void trace_virtual_sched(Process*, ErtsProcLocks, Eterm); void trace_sched_ports(Port *pp, Eterm); @@ -184,8 +193,10 @@ int erts_finish_breakpointing(void); /* Nif tracer functions */ int erts_is_tracer_proc_enabled(Process *c_p, ErtsProcLocks c_p_locks, - ErtsPTabElementCommon *t_p, Eterm type); -int erts_is_tracer_enabled(Process *c_p, const ErtsTracer tracer); + ErtsPTabElementCommon *t_p); +int erts_is_tracer_proc_enabled_send(Process* c_p, ErtsProcLocks c_p_locks, + ErtsPTabElementCommon *t_p); +int erts_is_tracer_enabled(const ErtsTracer tracer, ErtsPTabElementCommon *t_p); Eterm erts_tracer_to_term(Process *p, ErtsTracer tracer); ErtsTracer erts_term_to_tracer(Eterm prefix, Eterm term); void erts_tracer_replace(ErtsPTabElementCommon *t_p, @@ -215,9 +226,4 @@ ERTS_DECLARE_DUMMY(erts_tracer_nil) = NIL; #define ERTS_TRACER_FROM_ETERM(termp) \ ((ErtsTracer*)(termp)) -#define ERTS_TRACER_PROC_IS_ENABLED(PROC) \ - (!ERTS_TRACER_IS_NIL(ERTS_TRACER(PROC)) \ - && erts_is_tracer_proc_enabled(PROC, ERTS_PROC_LOCK_MAIN, \ - &(PROC)->common, am_trace_status)) - #endif /* ERL_TRACE_H__ */ diff --git a/erts/emulator/beam/erl_vm.h b/erts/emulator/beam/erl_vm.h index f3c54de214..f97716d030 100644 --- a/erts/emulator/beam/erl_vm.h +++ b/erts/emulator/beam/erl_vm.h @@ -50,6 +50,7 @@ #define H_DEFAULT_SIZE 233 /* default (heap + stack) min size */ #define VH_DEFAULT_SIZE 32768 /* default virtual (bin) heap min size (words) */ +#define H_DEFAULT_MAX_SIZE 0 /* default max heap size is off */ #define CP_SIZE 1 @@ -160,6 +161,8 @@ extern int num_instructions; /* Number of instruction in opc[]. */ extern int H_MIN_SIZE; /* minimum (heap + stack) */ extern int BIN_VH_MIN_SIZE; /* minimum virtual (bin) heap */ +extern int H_MAX_SIZE; /* maximum (heap + stack) */ +extern int H_MAX_FLAGS; /* maximum heap flags */ extern int erts_atom_table_size;/* Atom table size */ extern int erts_pd_initial_size;/* Initial Process dictionary table size */ diff --git a/erts/emulator/beam/global.h b/erts/emulator/beam/global.h index 49eec44053..1abcc6cbf4 100644 --- a/erts/emulator/beam/global.h +++ b/erts/emulator/beam/global.h @@ -57,6 +57,7 @@ struct enif_environment_t /* ErlNifEnv */ struct enif_tmp_obj_t* tmp_obj_list; int exception_thrown; /* boolean */ Process *tracee; + int exiting; /* boolean (dirty nifs might return in exiting state) */ }; extern void erts_pre_nif(struct enif_environment_t*, Process*, struct erl_module_nif*, Process* tracee); @@ -1483,9 +1484,19 @@ do { \ #define MatchSetGetSource(MPSP) erts_match_set_get_source(MPSP) -extern Binary *erts_match_set_compile(Process *p, Eterm matchexpr); +extern Binary *erts_match_set_compile(Process *p, Eterm matchexpr, Eterm MFA); Eterm erts_match_set_lint(Process *p, Eterm matchexpr); extern void erts_match_set_release_result(Process* p); +ERTS_GLB_INLINE void erts_match_set_release_result_trace(Process* p, Eterm); + +#if ERTS_GLB_INLINE_INCL_FUNC_DEF +ERTS_GLB_INLINE +void erts_match_set_release_result_trace(Process* p, Eterm pam_result) +{ + if (is_not_immed(pam_result)) + erts_match_set_release_result(p); +} +#endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ enum erts_pam_run_flags { ERTS_PAM_TMP_RESULT=1, @@ -1493,10 +1504,12 @@ enum erts_pam_run_flags { ERTS_PAM_CONTIGUOUS_TUPLE=4, ERTS_PAM_IGNORE_TRACE_SILENT=8 }; -extern Eterm erts_match_set_run(Process *p, Binary *mpsp, - Eterm *args, int num_args, - enum erts_pam_run_flags in_flags, - Uint32 *return_flags); +extern Eterm erts_match_set_run_trace(Process *p, + Process *self, + Binary *mpsp, + Eterm *args, int num_args, + enum erts_pam_run_flags in_flags, + Uint32 *return_flags); extern Eterm erts_match_set_get_source(Binary *mpsp); extern void erts_match_prog_foreach_offheap(Binary *b, void (*)(ErlOffHeap *, void *), diff --git a/erts/emulator/beam/io.c b/erts/emulator/beam/io.c index 5c2595c69d..0377f6cb5e 100644 --- a/erts/emulator/beam/io.c +++ b/erts/emulator/beam/io.c @@ -1432,10 +1432,11 @@ finalize_force_imm_drv_call(ErtsTryImmDrvCallState *sp) static ERTS_INLINE void queue_port_sched_op_reply(Process *rp, - ErtsProcLocks *rp_locksp, + ErtsProcLocks rp_locks, ErtsHeapFactory* factory, Uint32 *ref_num, - Eterm msg) + Eterm msg, + Port* prt) { Eterm* hp = erts_produce_heap(factory, ERTS_QUEUE_PORT_SCHED_OP_REPLY_SIZE, 0); Eterm ref; @@ -1448,11 +1449,12 @@ queue_port_sched_op_reply(Process *rp, erts_factory_trim_and_close(factory, &msg, 1); - erts_queue_message(rp, rp_locksp, factory->message, msg); + erts_queue_message(rp, rp_locks, factory->message, msg, + prt ? prt->common.id : am_undefined); } static void -port_sched_op_reply(Eterm to, Uint32 *ref_num, Eterm msg) +port_sched_op_reply(Eterm to, Uint32 *ref_num, Eterm msg, Port* prt) { Process *rp = erts_proc_lookup_raw(to); if (rp) { @@ -1478,10 +1480,11 @@ port_sched_op_reply(Eterm to, Uint32 *ref_num, Eterm msg) factory.off_heap)); queue_port_sched_op_reply(rp, - &rp_locks, + rp_locks, &factory, ref_num, - msg_copy); + msg_copy, + prt); if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); @@ -1651,7 +1654,7 @@ port_badsig(Port *prt, erts_aint32_t state, int op, state, sigdp->flags & ERTS_P2P_SIG_DATA_FLG_BAD_OUTPUT); if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg, prt); return ERTS_PORT_REDS_BADSIG; } /* port_badsig */ /* bad_port_signal() will @@ -1820,7 +1823,7 @@ port_sig_outputv(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *s } if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, reply); + port_sched_op_reply(sigdp->caller, sigdp->ref, reply, prt); cleanup_scheduled_outputv(sigdp->u.outputv.evp, sigdp->u.outputv.cbinp); @@ -1928,7 +1931,7 @@ port_sig_output(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *si } if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, reply); + port_sched_op_reply(sigdp->caller, sigdp->ref, reply, prt); cleanup_scheduled_output(sigdp->u.output.bufp); @@ -2509,7 +2512,7 @@ erts_port_output(Process *c_p, sigdp->flags &= ~ERTS_P2P_SIG_DATA_FLG_NOSUSPEND; else if (async_nosuspend) { ErtsSchedulerData *esdp = (c_p - ? ERTS_PROC_GET_SCHDATA(c_p) + ? erts_proc_sched_data(c_p) : erts_get_scheduler_data()); ASSERT(esdp); ns_pthp = &esdp->nosuspend_port_task_handle; @@ -2636,7 +2639,7 @@ port_sig_exit(Port *prt, if (sigdp->u.exit.bp) free_message_buffer(sigdp->u.exit.bp); if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, msg); + port_sched_op_reply(sigdp->caller, sigdp->ref, msg, prt); return ERTS_PORT_REDS_EXIT; } @@ -2829,7 +2832,7 @@ port_sig_connect(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *s msg = am_true; } if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, msg); + port_sched_op_reply(sigdp->caller, sigdp->ref, msg, prt); return ERTS_PORT_REDS_CONNECT; } @@ -2912,7 +2915,7 @@ port_sig_unlink(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *si if (op == ERTS_PROC2PORT_SIG_EXEC) port_unlink(prt, sigdp->u.unlink.from); if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, am_true); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_true, prt); return ERTS_PORT_REDS_UNLINK; } @@ -3007,7 +3010,7 @@ port_sig_link(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *sigd port_link_failure(sigdp->u.link.port, sigdp->u.link.to); } if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) - port_sched_op_reply(sigdp->caller, sigdp->ref, am_true); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_true, prt); return ERTS_PORT_REDS_LINK; } @@ -3064,7 +3067,8 @@ init_ack_send_reply(Port *port, Eterm resp) } port_sched_op_reply(port->async_open_port->to, port->async_open_port->ref, - resp); + resp, + port); erts_free(ERTS_ALC_T_PRTSD, port->async_open_port); port->async_open_port = NULL; @@ -3461,7 +3465,7 @@ deliver_result(Port *prt, Eterm sender, Eterm pid, Eterm res) sz_res + 3, &hp, &ohp); res = copy_struct(res, sz_res, &hp, ohp); tuple = TUPLE2(hp, sender, res); - erts_queue_message(rp, &rp_locks, mp, tuple); + erts_queue_message(rp, rp_locks, mp, tuple, sender); if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); @@ -3562,7 +3566,7 @@ static void deliver_read_message(Port* prt, erts_aint32_t state, Eterm to, trace_port_send(prt, to, tuple, 1); ERL_MESSAGE_TOKEN(mp) = am_undefined; - erts_queue_message(rp, &rp_locks, mp, tuple); + erts_queue_message(rp, rp_locks, mp, tuple, prt->common.id); if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); if (!scheduler) @@ -3734,7 +3738,7 @@ deliver_vec_message(Port* prt, /* Port */ trace_port_send(prt, to, tuple, 1); ERL_MESSAGE_TOKEN(mp) = am_undefined; - erts_queue_message(rp, &rp_locks, mp, tuple); + erts_queue_message(rp, rp_locks, mp, tuple, prt->common.id); erts_smp_proc_unlock(rp, rp_locks); if (!scheduler) erts_proc_dec_refc(rp); @@ -4388,10 +4392,11 @@ port_sig_control(Port *prt, &factory.hp, factory.off_heap); queue_port_sched_op_reply(rp, - &rp_locks, + rp_locks, &factory, sigdp->ref, - msg); + msg, + prt); if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); @@ -4402,7 +4407,7 @@ port_sig_control(Port *prt, /* failure */ if (sigdp->caller != ERTS_INVALID_PID) - port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg, prt); done: @@ -4739,10 +4744,11 @@ port_sig_call(Port *prt, msg = TUPLE2(hp, am_ok, msg); queue_port_sched_op_reply(rp, - &rp_locks, + rp_locks, &factory, sigdp->ref, - msg); + msg, + prt); if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); @@ -4754,7 +4760,7 @@ port_sig_call(Port *prt, } } - port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg, prt); done: @@ -4969,7 +4975,7 @@ port_sig_info(Port *prt, { ASSERT(sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY); if (op != ERTS_PROC2PORT_SIG_EXEC) - port_sched_op_reply(sigdp->caller, sigdp->ref, am_undefined); + port_sched_op_reply(sigdp->caller, sigdp->ref, am_undefined, prt); else { Eterm *hp, *hp_start; Uint hsz; @@ -4995,10 +5001,11 @@ port_sig_info(Port *prt, mp->data.heap_frag = bp; erts_factory_selfcontained_message_init(&factory, mp, hp); queue_port_sched_op_reply(rp, - &rp_locks, + rp_locks, &factory, sigdp->ref, - value); + value, + prt); } if (rp_locks) erts_smp_proc_unlock(rp, rp_locks); @@ -5115,7 +5122,7 @@ reply_io_bytes(void *vreq) msg = TUPLE4(hp, ref, make_small(sched_id), ein, eout); - erts_queue_message(rp, &rp_locks, mp, msg); + erts_queue_message(rp, rp_locks, mp, msg, am_system); if (req->sched_id == sched_id) rp_locks &= ~ERTS_PROC_LOCK_MAIN; @@ -5133,7 +5140,7 @@ erts_request_io_bytes(Process *c_p) Uint *hp; Eterm ref; Uint32 *refn; - ErtsSchedulerData *esdp = ERTS_PROC_GET_SCHDATA(c_p); + ErtsSchedulerData *esdp = erts_proc_sched_data(c_p); ErtsIOBytesReq *req = erts_alloc(ERTS_ALC_T_IOB_REQ, sizeof(ErtsIOBytesReq)); @@ -5613,7 +5620,7 @@ void driver_report_exit(ErlDrvPort ix, int status) trace_port_send(prt, pid, tuple, 1); ERL_MESSAGE_TOKEN(mp) = am_undefined; - erts_queue_message(rp, &rp_locks, mp, tuple); + erts_queue_message(rp, rp_locks, mp, tuple, prt->common.id); erts_smp_proc_unlock(rp, rp_locks); if (!scheduler) @@ -5934,7 +5941,7 @@ driver_deliver_term(Port *prt, Eterm to, ErlDrvTermData* data, int len) if (!rp) { if (!prt || !IS_TRACED_FL(prt, F_TRACE_SEND)) goto done; - if (!erts_is_tracer_proc_enabled(NULL, 0, &prt->common, am_send)) + if (!erts_is_tracer_proc_enabled_send(NULL, 0, &prt->common)) goto done; res = -2; @@ -6217,15 +6224,20 @@ driver_deliver_term(Port *prt, Eterm to, ErlDrvTermData* data, int len) done: if (res > 0) { + Eterm from = am_undefined; mess = ESTACK_POP(stack); /* get resulting value */ erts_factory_trim_and_close(&factory, &mess, 1); - if (prt && IS_TRACED_FL(prt, F_TRACE_SEND)) - trace_port_send(prt, to, mess, 1); + if (prt) { + if (IS_TRACED_FL(prt, F_TRACE_SEND)) { + trace_port_send(prt, to, mess, 1); + } + from = prt->common.id; + } /* send message */ ERL_MESSAGE_TOKEN(factory.message) = am_undefined; - erts_queue_message(rp, &rp_locks, factory.message, mess); + erts_queue_message(rp, rp_locks, factory.message, mess, from); } else if (res == -2) { /* this clause only happens when we were requested to @@ -7869,11 +7881,13 @@ driver_system_info(ErlDrvSysInfo *sip, size_t si_size) * (driver version 3.1, NIF version 2.7) */ if (si_size >= ERL_DRV_SYS_INFO_SIZE(dirty_scheduler_support)) { -#if defined(ERL_NIF_DIRTY_SCHEDULER_SUPPORT) && defined(USE_THREADS) - sip->dirty_scheduler_support = 1; + sip->dirty_scheduler_support = +#ifdef ERTS_DIRTY_SCHEDULERS + 1 #else - sip->dirty_scheduler_support = 0; + 0 #endif + ; } } diff --git a/erts/emulator/beam/sys.h b/erts/emulator/beam/sys.h index 748fba15c7..f303d4f167 100644 --- a/erts/emulator/beam/sys.h +++ b/erts/emulator/beam/sys.h @@ -34,6 +34,10 @@ (((__GNUC__ << 24) | (__GNUC_MINOR__ << 12) | __GNUC_PATCHLEVEL__) >= (((MAJ) << 24) | ((MIN) << 12) | (PL))) #endif +#if defined(ERTS_DIRTY_SCHEDULERS) && !defined(ERTS_SMP) +# error "Dirty schedulers not supported without smp support" +#endif + #ifdef ERTS_INLINE # ifndef ERTS_CAN_INLINE # define ERTS_CAN_INLINE 1 @@ -92,6 +96,9 @@ #define ErtsInArea(ptr,start,nbytes) \ ((UWord)((char*)(ptr) - (char*)(start)) < (nbytes)) +#define ErtsContainerStruct(ptr, type, member) \ + (type *)((char *)(1 ? (ptr) : &((type *)0)->member) - offsetof(type, member)) + #if defined (__WIN32__) # include "erl_win_sys.h" #else diff --git a/erts/emulator/beam/utils.c b/erts/emulator/beam/utils.c index 68006e7ef3..cedc88e5fe 100644 --- a/erts/emulator/beam/utils.c +++ b/erts/emulator/beam/utils.c @@ -2263,7 +2263,7 @@ static void do_send_logger_message(Eterm *hp, ErlOffHeap *ohp, ErlHeapFragment * { ErtsMessage *mp = erts_alloc_message(0, NULL); mp->data.heap_frag = bp; - erts_queue_message(p, NULL /* only used for smp build */, mp, message); + erts_queue_message(p, 0, mp, message, am_system); } #endif } diff --git a/erts/emulator/hipe/hipe_bif0.c b/erts/emulator/hipe/hipe_bif0.c index 48cfafb8c5..4063cbf306 100644 --- a/erts/emulator/hipe/hipe_bif0.c +++ b/erts/emulator/hipe/hipe_bif0.c @@ -533,13 +533,13 @@ BIF_RETTYPE hipe_bifs_merge_term_1(BIF_ALIST_1) BIF_RET(val); } -struct mfa { +struct mfa_t { Eterm mod; Eterm fun; Uint ari; }; -static int term_to_mfa(Eterm term, struct mfa *mfa) +static int term_to_mfa(Eterm term, struct mfa_t *mfa) { Eterm mod, fun, a; Uint ari; @@ -597,7 +597,7 @@ static Uint *hipe_find_emu_address(Eterm mod, Eterm name, unsigned int arity) Uint *hipe_bifs_find_pc_from_mfa(Eterm term) { - struct mfa mfa; + struct mfa_t mfa; if (!term_to_mfa(term, &mfa)) return NULL; @@ -617,7 +617,7 @@ BIF_RETTYPE hipe_bifs_set_native_address_3(BIF_ALIST_3) Eterm *pc; void *address; int is_closure; - struct mfa mfa; + struct mfa_t mfa; switch (BIF_ARG_3) { case am_false: @@ -1225,7 +1225,7 @@ void hipe_mfa_set_trampoline(Eterm m, Eterm f, unsigned int arity, void *trampol BIF_RETTYPE hipe_bifs_set_funinfo_native_address_3(BIF_ALIST_3) { - struct mfa mfa; + struct mfa_t mfa; void *address; int is_exported; @@ -1247,7 +1247,7 @@ BIF_RETTYPE hipe_bifs_set_funinfo_native_address_3(BIF_ALIST_3) BIF_RETTYPE hipe_bifs_invalidate_funinfo_native_addresses_1(BIF_ALIST_1) { Eterm lst; - struct mfa mfa; + struct mfa_t mfa; struct hipe_mfa_info *p; hipe_mfa_info_table_rwlock(); @@ -1416,7 +1416,7 @@ BIF_RETTYPE hipe_find_na_or_make_stub(BIF_ALIST_3) BIF_RETTYPE hipe_bifs_find_na_or_make_stub_2(BIF_ALIST_2) { - struct mfa mfa; + struct mfa_t mfa; void *address; int is_remote; @@ -1518,9 +1518,9 @@ struct ref { */ BIF_RETTYPE hipe_bifs_add_ref_2(BIF_ALIST_2) { - struct mfa callee; + struct mfa_t callee; Eterm *tuple; - struct mfa caller; + struct mfa_t caller; void *address; void *trampoline; unsigned int flags; @@ -1597,7 +1597,7 @@ BIF_RETTYPE hipe_bifs_add_ref_2(BIF_ALIST_2) */ BIF_RETTYPE hipe_bifs_mark_referred_from_1(BIF_ALIST_1) /* get_refs_from */ { - struct mfa mfa; + struct mfa_t mfa; const struct hipe_mfa_info *p; struct ref *ref; @@ -1649,7 +1649,7 @@ static void hipe_purge_all_refs(void) BIF_RETTYPE hipe_bifs_remove_refs_from_1(BIF_ALIST_1) { - struct mfa mfa; + struct mfa_t mfa; struct hipe_mfa_info *caller_mfa, *callee_mfa; struct hipe_mfa_info_list *refers_to, *tmp_refers_to; struct ref **prev, *ref; @@ -1703,7 +1703,7 @@ BIF_RETTYPE hipe_bifs_remove_refs_from_1(BIF_ALIST_1) */ BIF_RETTYPE hipe_bifs_redirect_referred_from_1(BIF_ALIST_1) { - struct mfa mfa; + struct mfa_t mfa; struct hipe_mfa_info *p; struct ref **prev, *ref; int is_remote, res; diff --git a/erts/emulator/hipe/hipe_mode_switch.c b/erts/emulator/hipe/hipe_mode_switch.c index 9ad44b25ac..f532d3151f 100644 --- a/erts/emulator/hipe/hipe_mode_switch.c +++ b/erts/emulator/hipe/hipe_mode_switch.c @@ -59,6 +59,7 @@ * TODO: check PCB consistency at native BIF calls */ int hipe_modeswitch_debug = 0; +extern BeamInstr beam_exit[]; #define HIPE_DEBUG 0 @@ -509,6 +510,10 @@ Process *hipe_mode_switch(Process *p, unsigned cmd, Eterm reg[]) if (scb) ERTS_PROC_SET_SAVED_CALLS_BUF(p, scb); + /* The process may have died while it was executing, + if so we return out from native code to the interpreter */ + if (erts_smp_atomic32_read_nob(&p->state) & ERTS_PSFLG_EXITING) + p->i = beam_exit; #ifdef DEBUG ASSERT(p->debug_reds_in == reds_in); #endif diff --git a/erts/emulator/hipe/hipe_native_bif.c b/erts/emulator/hipe/hipe_native_bif.c index 00b572029a..9c03b3811c 100644 --- a/erts/emulator/hipe/hipe_native_bif.c +++ b/erts/emulator/hipe/hipe_native_bif.c @@ -601,7 +601,7 @@ void hipe_clear_timeout(Process *c_p) } #endif if (IS_TRACED_FL(c_p, F_TRACE_RECEIVE)) { - trace_receive(c_p, am_timeout); + trace_receive(c_p, am_clock_service, am_timeout, NULL); } c_p->flags &= ~F_TIMO; JOIN_MESSAGE(c_p); diff --git a/erts/emulator/nifs/common/erl_tracer_nif.c b/erts/emulator/nifs/common/erl_tracer_nif.c index 8a9a1bf16c..6dddc80607 100644 --- a/erts/emulator/nifs/common/erl_tracer_nif.c +++ b/erts/emulator/nifs/common/erl_tracer_nif.c @@ -70,6 +70,7 @@ ERL_NIF_INIT(erl_tracer, nif_funcs, load, NULL, upgrade, unload) ATOM_DECL(strict_monotonic); \ ATOM_DECL(timestamp); \ ATOM_DECL(trace); \ + ATOM_DECL(trace_status); \ ATOM_DECL(trace_ts); \ ATOM_DECL(true); \ ATOM_DECL(gc_minor_start); \ @@ -118,19 +119,22 @@ static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { ErlNifPid to_pid; ErlNifPort to_port; + ERL_NIF_TERM ret = enif_is_identical(argv[0], atom_trace_status) ? + atom_remove : atom_discard; + ASSERT(argc == 3); if (enif_get_local_pid(env, argv[1], &to_pid)) { if (!enif_is_process_alive(env, &to_pid)) /* tracer is dead so we should remove this trace point */ - return atom_remove; + return ret; } else if (enif_get_local_port(env, argv[1], &to_port)) { if (!enif_is_port_alive(env, &to_port)) /* tracer is dead so we should remove this trace point */ - return atom_remove; + return ret; } else { /* The state was not a pid or a port */ - return atom_remove; + return ret; } /* Only generate trace for when tracer != tracee */ diff --git a/erts/emulator/sys/common/erl_mmap.c b/erts/emulator/sys/common/erl_mmap.c index a7e710070b..53009a1481 100644 --- a/erts/emulator/sys/common/erl_mmap.c +++ b/erts/emulator/sys/common/erl_mmap.c @@ -28,6 +28,8 @@ #include "erl_mmap.h" #include <stddef.h> +#if HAVE_ERTS_MMAP + /* #define ERTS_MMAP_OP_RINGBUF_SZ 100 */ #if defined(DEBUG) || 0 @@ -2099,6 +2101,7 @@ int erts_mmap_in_supercarrier(ErtsMemMapper* mm, void *ptr) } static struct { + Eterm options; Eterm total; Eterm total_sa; Eterm total_sua; @@ -2132,6 +2135,7 @@ static void init_atoms(void) erts_mtx_lock(&am.init_mutex); if (!am.is_initialized) { + AM_INIT(options); AM_INIT(total); AM_INIT(total_sa); AM_INIT(total_sua); @@ -2387,9 +2391,9 @@ Eterm erts_mmap_info(ErtsMemMapper* mm, Eterm seg_tags[] = { am.used, am.max, am.allocated, am.reserved, am.used_sa, am.used_sua }; Eterm group[2]; Eterm group_tags[] = { am.sizes, am.free_segs }; - Eterm list[2]; - Eterm list_tags[2]; /* { am.supercarrier, am.os } */ - int lix; + Eterm list[3]; + Eterm list_tags[3]; /* { am.options, am.supercarrier, am.os } */ + int lix = 0; Eterm res = THE_NON_VALUE; if (!hpp) { @@ -2412,6 +2416,12 @@ Eterm erts_mmap_info(ErtsMemMapper* mm, erts_smp_mtx_unlock(&mm->mtx); } + list[lix] = erts_mmap_info_options(mm, "option ", print_to_p, print_to_arg, + hpp, szp); + list_tags[lix] = am.options; + lix++; + + if (print_to_p) { int to = *print_to_p; void *arg = print_to_arg; @@ -2441,7 +2451,6 @@ Eterm erts_mmap_info(ErtsMemMapper* mm, init_atoms(); } - lix = 0; if (mm->supercarrier) { group[0] = erts_bld_atom_uword_2tup_list(hpp, szp, sizeof(size_tags)/sizeof(Eterm), @@ -2506,9 +2515,13 @@ Eterm erts_mmap_info_options(ErtsMemMapper* mm, return res; } +#endif /* HAVE_ERTS_MMAP */ -Eterm erts_mmap_debug_info(ErtsMemMapper* mm, Process* p) +Eterm erts_mmap_debug_info(Process* p) { +#if HAVE_ERTS_MMAP + ErtsMemMapper* mm = &erts_dflt_mmapper; + if (mm->supercarrier) { ERTS_DECL_AM(sabot); ERTS_DECL_AM(satop); @@ -2546,9 +2559,8 @@ Eterm erts_mmap_debug_info(ErtsMemMapper* mm, Process* p) HRelease(p, hp_end, hp); return list; } - else { - return am_undefined; - } +#endif + return am_undefined; } diff --git a/erts/emulator/sys/common/erl_mmap.h b/erts/emulator/sys/common/erl_mmap.h index b3c45ba116..7ac61a82c1 100644 --- a/erts/emulator/sys/common/erl_mmap.h +++ b/erts/emulator/sys/common/erl_mmap.h @@ -26,9 +26,36 @@ #define ERTS_MMAP_SUPERALIGNED_BITS (18) /* Affects hard limits for sbct and lmbcs documented in erts_alloc.xml */ -#define ERTS_MMAPFLG_OS_ONLY (((Uint32) 1) << 0) -#define ERTS_MMAPFLG_SUPERCARRIER_ONLY (((Uint32) 1) << 1) -#define ERTS_MMAPFLG_SUPERALIGNED (((Uint32) 1) << 2) +#ifndef HAVE_MMAP +# define HAVE_MMAP 0 +#endif +#ifndef HAVE_MREMAP +# define HAVE_MREMAP 0 +#endif +#if HAVE_MMAP +# define ERTS_HAVE_OS_MMAP 1 +# define ERTS_HAVE_GENUINE_OS_MMAP 1 +# if HAVE_MREMAP +# define ERTS_HAVE_OS_MREMAP 1 +# endif +# if defined(MAP_FIXED) && defined(MAP_NORESERVE) +# define ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION 1 +# endif +#endif + +#ifndef HAVE_VIRTUALALLOC +# define HAVE_VIRTUALALLOC 0 +#endif +#if HAVE_VIRTUALALLOC +# define ERTS_HAVE_OS_MMAP 1 +#endif + +#ifdef ERTS_HAVE_GENUINE_OS_MMAP +# define HAVE_ERTS_MMAP 1 +#else +# define HAVE_ERTS_MMAP 0 +#endif + extern UWord erts_page_inv_mask; @@ -60,26 +87,6 @@ typedef struct { #define ERTS_MMAP_INIT_HIPE_EXEC_INITER \ {{NULL, NULL}, {NULL, NULL}, ERTS_HIPE_EXEC_VIRTUAL_AREA_SIZE, 1, (1 << 10), 0} -typedef struct ErtsMemMapper_ ErtsMemMapper; - -void *erts_mmap(ErtsMemMapper*, Uint32 flags, UWord *sizep); -void erts_munmap(ErtsMemMapper*, Uint32 flags, void *ptr, UWord size); -void *erts_mremap(ErtsMemMapper*, Uint32 flags, void *ptr, UWord old_size, UWord *sizep); -int erts_mmap_in_supercarrier(ErtsMemMapper*, void *ptr); -void erts_mmap_init(ErtsMemMapper*, ErtsMMapInit*, int executable); -struct erts_mmap_info_struct -{ - UWord sizes[6]; - UWord segs[6]; - UWord os_used; -}; -Eterm erts_mmap_info(ErtsMemMapper*, int *print_to_p, void *print_to_arg, - Eterm** hpp, Uint* szp, struct erts_mmap_info_struct*); -Eterm erts_mmap_info_options(ErtsMemMapper*, - char *prefix, int *print_to_p, void *print_to_arg, - Uint **hpp, Uint *szp); -struct process; -Eterm erts_mmap_debug_info(ErtsMemMapper*, struct process*); #define ERTS_SUPERALIGNED_SIZE \ (1 << ERTS_MMAP_SUPERALIGNED_BITS) @@ -107,29 +114,34 @@ Eterm erts_mmap_debug_info(ErtsMemMapper*, struct process*); #define ERTS_PAGEALIGNED_SIZE \ (ERTS_INV_PAGEALIGNED_MASK + 1) -#ifndef HAVE_MMAP -# define HAVE_MMAP 0 -#endif -#ifndef HAVE_MREMAP -# define HAVE_MREMAP 0 -#endif -#if HAVE_MMAP -# define ERTS_HAVE_OS_MMAP 1 -# define ERTS_HAVE_GENUINE_OS_MMAP 1 -# if HAVE_MREMAP -# define ERTS_HAVE_OS_MREMAP 1 -# endif -# if defined(MAP_FIXED) && defined(MAP_NORESERVE) -# define ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION 1 -# endif -#endif +struct process; +Eterm erts_mmap_debug_info(struct process*); + +#if HAVE_ERTS_MMAP + +typedef struct ErtsMemMapper_ ErtsMemMapper; + +#define ERTS_MMAPFLG_OS_ONLY (((Uint32) 1) << 0) +#define ERTS_MMAPFLG_SUPERCARRIER_ONLY (((Uint32) 1) << 1) +#define ERTS_MMAPFLG_SUPERALIGNED (((Uint32) 1) << 2) + +void *erts_mmap(ErtsMemMapper*, Uint32 flags, UWord *sizep); +void erts_munmap(ErtsMemMapper*, Uint32 flags, void *ptr, UWord size); +void *erts_mremap(ErtsMemMapper*, Uint32 flags, void *ptr, UWord old_size, UWord *sizep); +int erts_mmap_in_supercarrier(ErtsMemMapper*, void *ptr); +void erts_mmap_init(ErtsMemMapper*, ErtsMMapInit*, int executable); +struct erts_mmap_info_struct +{ + UWord sizes[6]; + UWord segs[6]; + UWord os_used; +}; +Eterm erts_mmap_info(ErtsMemMapper*, int *print_to_p, void *print_to_arg, + Eterm** hpp, Uint* szp, struct erts_mmap_info_struct*); +Eterm erts_mmap_info_options(ErtsMemMapper*, + char *prefix, int *print_to_p, void *print_to_arg, + Uint **hpp, Uint *szp); -#ifndef HAVE_VIRTUALALLOC -# define HAVE_VIRTUALALLOC 0 -#endif -#if HAVE_VIRTUALALLOC -# define ERTS_HAVE_OS_MMAP 1 -#endif #ifdef ERTS_WANT_MEM_MAPPERS # include "erl_alloc_types.h" @@ -154,4 +166,6 @@ void hard_dbg_remove_mseg(void* seg, UWord sz); # define HARD_DBG_REMOVE_MSEG(SEG,SZ) #endif +#endif /* HAVE_ERTS_MMAP */ + #endif /* ERL_MMAP_H__ */ diff --git a/erts/emulator/sys/common/erl_mseg.c b/erts/emulator/sys/common/erl_mseg.c index 43e75e2573..f3306a888c 100644 --- a/erts/emulator/sys/common/erl_mseg.c +++ b/erts/emulator/sys/common/erl_mseg.c @@ -996,10 +996,7 @@ info_options(ErtsMsegAllctr_t *ma, Uint **hpp, Uint *szp) { - Eterm res; - - res = erts_mmap_info_options(&erts_dflt_mmapper, - prefix, print_to_p, print_to_arg, hpp, szp); + Eterm res = NIL; if (print_to_p) { int to = *print_to_p; @@ -1110,7 +1107,7 @@ info_calls(ErtsMsegAllctr_t *ma, int *print_to_p, void *print_to_arg, Uint **hpp static Eterm info_status(ErtsMsegAllctr_t *ma, int *print_to_p, void *print_to_arg, - int begin_new_max_period, Uint **hpp, Uint *szp) + int begin_new_max_period, int only_sz, Uint **hpp, Uint *szp) { Eterm res = THE_NON_VALUE; @@ -1123,38 +1120,41 @@ info_status(ErtsMsegAllctr_t *ma, int *print_to_p, void *print_to_arg, int to = *print_to_p; void *arg = print_to_arg; - erts_print(to, arg, "cached_segments: %beu\n", ma->cache_size); - erts_print(to, arg, "cache_hits: %beu\n", ma->cache_hits); - erts_print(to, arg, "segments: %beu %beu %beu\n", - ma->segments.current.no, ma->segments.max.no, ma->segments.max_ever.no); - erts_print(to, arg, "segments_size: %beu %beu %beu\n", + if (!only_sz) { + erts_print(to, arg, "cached_segments: %beu\n", ma->cache_size); + erts_print(to, arg, "cache_hits: %beu\n", ma->cache_hits); + erts_print(to, arg, "segments: %beu %beu %beu\n", + ma->segments.current.no, ma->segments.max.no, ma->segments.max_ever.no); + erts_print(to, arg, "segments_watermark: %beu\n", + ma->segments.current.watermark); + } + erts_print(to, arg, "segments_size: %beu %beu %beu\n", ma->segments.current.sz, ma->segments.max.sz, ma->segments.max_ever.sz); - erts_print(to, arg, "segments_watermark: %beu\n", - ma->segments.current.watermark); } if (hpp || szp) { res = NIL; - add_2tup(hpp, szp, &res, - am.segments_watermark, - bld_unstable_uint(hpp, szp, ma->segments.current.watermark)); - add_4tup(hpp, szp, &res, - am.segments_size, - bld_unstable_uint(hpp, szp, ma->segments.current.sz), - bld_unstable_uint(hpp, szp, ma->segments.max.sz), - bld_unstable_uint(hpp, szp, ma->segments.max_ever.sz)); - add_4tup(hpp, szp, &res, - am.segments, - bld_unstable_uint(hpp, szp, ma->segments.current.no), - bld_unstable_uint(hpp, szp, ma->segments.max.no), - bld_unstable_uint(hpp, szp, ma->segments.max_ever.no)); - add_2tup(hpp, szp, &res, - am.cache_hits, - bld_unstable_uint(hpp, szp, ma->cache_hits)); - add_2tup(hpp, szp, &res, - am.cached_segments, - bld_unstable_uint(hpp, szp, ma->cache_size)); - + add_4tup(hpp, szp, &res, + am.segments_size, + bld_unstable_uint(hpp, szp, ma->segments.current.sz), + bld_unstable_uint(hpp, szp, ma->segments.max.sz), + bld_unstable_uint(hpp, szp, ma->segments.max_ever.sz)); + if (!only_sz) { + add_2tup(hpp, szp, &res, + am.segments_watermark, + bld_unstable_uint(hpp, szp, ma->segments.current.watermark)); + add_4tup(hpp, szp, &res, + am.segments, + bld_unstable_uint(hpp, szp, ma->segments.current.no), + bld_unstable_uint(hpp, szp, ma->segments.max.no), + bld_unstable_uint(hpp, szp, ma->segments.max_ever.no)); + add_2tup(hpp, szp, &res, + am.cache_hits, + bld_unstable_uint(hpp, szp, ma->cache_hits)); + add_2tup(hpp, szp, &res, + am.cached_segments, + bld_unstable_uint(hpp, szp, ma->cache_size)); + } } if (begin_new_max_period) { @@ -1166,26 +1166,31 @@ info_status(ErtsMsegAllctr_t *ma, int *print_to_p, void *print_to_arg, } static Eterm info_memkind(ErtsMsegAllctr_t *ma, int *print_to_p, void *print_to_arg, - int begin_max_per, Uint **hpp, Uint *szp) + int begin_max_per, int only_sz, Uint **hpp, Uint *szp) { Eterm res = THE_NON_VALUE; Eterm atoms[3]; Eterm values[3]; - if (print_to_p) { - erts_print(*print_to_p, print_to_arg, "memory kind: %s\n", "all memory"); - } - if (hpp || szp) { - atoms[0] = am.name; - atoms[1] = am.status; - atoms[2] = am.calls; - values[0] = erts_bld_string(hpp, szp, "all memory"); + if (!only_sz) { + if (print_to_p) { + erts_print(*print_to_p, print_to_arg, "memory kind: %s\n", "all memory"); + } + if (hpp || szp) { + atoms[0] = am.name; + atoms[1] = am.status; + atoms[2] = am.calls; + values[0] = erts_bld_string(hpp, szp, "all memory"); + } } - values[1] = info_status(ma, print_to_p, print_to_arg, begin_max_per, hpp, szp); - values[2] = info_calls(ma, print_to_p, print_to_arg, hpp, szp); + res = info_status(ma, print_to_p, print_to_arg, begin_max_per, only_sz, hpp, szp); + if (!only_sz) { + values[1] = res; + values[2] = info_calls(ma, print_to_p, print_to_arg, hpp, szp); - if (hpp || szp) - res = bld_2tup_list(hpp, szp, 3, atoms, values); + if (hpp || szp) + res = bld_2tup_list(hpp, szp, 3, atoms, values); + } return res; } @@ -1229,6 +1234,7 @@ erts_mseg_info(int ix, int *print_to_p, void *print_to_arg, int begin_max_per, + int only_sz, Uint **hpp, Uint *szp) { @@ -1239,24 +1245,29 @@ erts_mseg_info(int ix, Uint n = 0; if (hpp || szp) { - - if (!atoms_initialized) - init_atoms(ma); - - atoms[0] = am.version; - atoms[1] = am.options; - atoms[2] = am.memkind; - atoms[3] = am.memkind; + if (!atoms_initialized) + init_atoms(ma); + } + if (!only_sz) { + if (hpp || szp) { + atoms[0] = am.version; + atoms[1] = am.options; + atoms[2] = am.memkind; + } + values[n++] = info_version(ma, print_to_p, print_to_arg, hpp, szp); + values[n++] = info_options(ma, "option ", print_to_p, print_to_arg, hpp, szp); } - values[n++] = info_version(ma, print_to_p, print_to_arg, hpp, szp); - values[n++] = info_options(ma, "option ", print_to_p, print_to_arg, hpp, szp); ERTS_MSEG_LOCK(ma); ERTS_DBG_MA_CHK_THR_ACCESS(ma); - values[n++] = info_memkind(ma, print_to_p, print_to_arg, begin_max_per, hpp, szp); - if (hpp || szp) - res = bld_2tup_list(hpp, szp, n, atoms, values); + res = info_memkind(ma, print_to_p, print_to_arg, begin_max_per, only_sz, hpp, szp); + + if (!only_sz) { + values[n++] = res; + if (hpp || szp) + res = bld_2tup_list(hpp, szp, n, atoms, values); + } ERTS_MSEG_UNLOCK(ma); diff --git a/erts/emulator/sys/common/erl_mseg.h b/erts/emulator/sys/common/erl_mseg.h index 192e5767e4..a43b409e94 100644 --- a/erts/emulator/sys/common/erl_mseg.h +++ b/erts/emulator/sys/common/erl_mseg.h @@ -99,7 +99,7 @@ void erts_mseg_init(ErtsMsegInit_t *init); void erts_mseg_late_init(void); /* Have to be called after all allocators, threads and timers have been initialized. */ Eterm erts_mseg_info_options(int, int *, void*, Uint **, Uint *); -Eterm erts_mseg_info(int, int *, void*, int, Uint **, Uint *); +Eterm erts_mseg_info(int, int *, void*, int, int, Uint **, Uint *); #endif /* #if HAVE_ERTS_MSEG */ diff --git a/erts/emulator/test/Makefile b/erts/emulator/test/Makefile index de395dfb97..b580211eff 100644 --- a/erts/emulator/test/Makefile +++ b/erts/emulator/test/Makefile @@ -53,6 +53,7 @@ MODULES= \ crypto_SUITE \ ddll_SUITE \ decode_packet_SUITE \ + dirty_nif_SUITE \ distribution_SUITE \ driver_SUITE \ efile_SUITE \ diff --git a/erts/emulator/test/alloc_SUITE.erl b/erts/emulator/test/alloc_SUITE.erl index 5fb560d1ec..1f7b499dcb 100644 --- a/erts/emulator/test/alloc_SUITE.erl +++ b/erts/emulator/test/alloc_SUITE.erl @@ -102,8 +102,8 @@ erts_mmap_do(Config, SCO, SCRPM, SCRFSD) -> Self = self(), Ref = make_ref(), F = fun() -> - SI = erlang:system_info({allocator,mseg_alloc}), - {erts_mmap,EM} = lists:keyfind(erts_mmap, 1, SI), + SI = erlang:system_info({allocator,erts_mmap}), + {default_mmap,EM} = lists:keyfind(default_mmap, 1, SI), {supercarrier,SC} = lists:keyfind(supercarrier, 1, EM), {sizes,Sizes} = lists:keyfind(sizes, 1, SC), {free_segs,Segs} = lists:keyfind(free_segs,1,SC), diff --git a/erts/emulator/test/dirty_nif_SUITE.erl b/erts/emulator/test/dirty_nif_SUITE.erl new file mode 100644 index 0000000000..c3afbc0803 --- /dev/null +++ b/erts/emulator/test/dirty_nif_SUITE.erl @@ -0,0 +1,327 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2010-2014. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +-module(dirty_nif_SUITE). + +%%-define(line_trace,true). +-define(CHECK(Exp,Got), check(Exp,Got,?LINE)). +%%-define(CHECK(Exp,Got), Exp = Got). + +-include_lib("common_test/include/ct.hrl"). + +-export([all/0, suite/0, + init_per_suite/1, end_per_suite/1, + init_per_testcase/2, end_per_testcase/2, + dirty_nif/1, dirty_nif_send/1, + dirty_nif_exception/1, call_dirty_nif_exception/1, + dirty_scheduler_exit/1, dirty_call_while_terminated/1, + dirty_heap_access/1]). + +-define(nif_stub,nif_stub_error(?LINE)). + +suite() -> [{ct_hooks,[ts_install_cth]}]. + +all() -> + [dirty_nif, + dirty_nif_send, + dirty_nif_exception, + dirty_scheduler_exit, + dirty_call_while_terminated, + dirty_heap_access]. + +init_per_suite(Config) -> + try erlang:system_info(dirty_cpu_schedulers) of + N when is_integer(N), N > 0 -> + case lib_loaded() of + false -> + ok = erlang:load_nif( + filename:join(?config(data_dir, Config), + "dirty_nif_SUITE"), []); + true -> + ok + end, + Config + catch _:_ -> + {skipped, "No dirty scheduler support"} + end. + +end_per_suite(_Config) -> + ok. + +init_per_testcase(Case, Config) -> + [{testcase, Case} | Config]. + +end_per_testcase(_Case, _Config) -> + ok. + +dirty_nif(Config) when is_list(Config) -> + Val1 = 42, + Val2 = "Erlang", + Val3 = list_to_binary([Val2, 0]), + {Val1, Val2, Val3} = call_dirty_nif(Val1, Val2, Val3), + LargeArray = lists:duplicate(1000, ok), + LargeArray = call_dirty_nif_zero_args(), + ok. + +dirty_nif_send(Config) when is_list(Config) -> + Parent = self(), + Pid = spawn_link(fun() -> + Self = self(), + {ok, Self} = receive_any(), + Parent ! {ok, Self} + end), + {ok, Pid} = send_from_dirty_nif(Pid), + {ok, Pid} = receive_any(), + ok. + +dirty_nif_exception(Config) when is_list(Config) -> + try + %% this checks that the expected exception occurs when the + %% dirty NIF returns the result of enif_make_badarg + %% directly + call_dirty_nif_exception(1), + ct:fail(expected_badarg) + catch + error:badarg -> + [{?MODULE,call_dirty_nif_exception,[1],_}|_] = + erlang:get_stacktrace(), + ok + end, + try + %% this checks that the expected exception occurs when the + %% dirty NIF calls enif_make_badarg at some point but then + %% returns a value that isn't an exception + call_dirty_nif_exception(0), + ct:fail(expected_badarg) + catch + error:badarg -> + [{?MODULE,call_dirty_nif_exception,[0],_}|_] = + erlang:get_stacktrace(), + ok + end, + %% this checks that a dirty NIF can raise various terms as + %% exceptions + ok = nif_raise_exceptions(call_dirty_nif_exception). + +nif_raise_exceptions(NifFunc) -> + ExcTerms = [{error, test}, "a string", <<"a binary">>, + 42, [1,2,3,4,5], [{p,1},{p,2},{p,3}]], + lists:foldl(fun(Term, ok) -> + try + erlang:apply(?MODULE,NifFunc,[Term]), + ct:fail({expected,Term}) + catch + error:Term -> + [{?MODULE,NifFunc,[Term],_}|_] = erlang:get_stacktrace(), + ok + end + end, ok, ExcTerms). + +dirty_scheduler_exit(Config) when is_list(Config) -> + {ok, Node} = start_node(Config, "+SDio 1"), + Path = proplists:get_value(data_dir, Config), + NifLib = filename:join(Path, atom_to_list(?MODULE)), + [ok] = mcall(Node, + [fun() -> + ok = erlang:load_nif(NifLib, []), + Start = erlang:monotonic_time(milli_seconds), + ok = test_dirty_scheduler_exit(), + End = erlang:monotonic_time(milli_seconds), + io:format("Time=~p ms~n", [End-Start]), + ok + end]), + stop_node(Node), + ok. + +test_dirty_scheduler_exit() -> + process_flag(trap_exit,true), + test_dse(10,[]). +test_dse(0,Pids) -> + timer:sleep(100), + kill_dse(Pids,[]); +test_dse(N,Pids) -> + Pid = spawn_link(fun dirty_sleeper/0), + test_dse(N-1,[Pid|Pids]). + +kill_dse([],Killed) -> + wait_dse(Killed); +kill_dse([Pid|Pids],AlreadyKilled) -> + exit(Pid,kill), + kill_dse(Pids,[Pid|AlreadyKilled]). + +wait_dse([]) -> + ok; +wait_dse([Pid|Pids]) -> + receive + {'EXIT',Pid,Reason} -> + killed = Reason + end, + wait_dse(Pids). + +dirty_call_while_terminated(Config) when is_list(Config) -> + Me = self(), + Bin = list_to_binary(lists:duplicate(4711, $r)), + {value, {BinAddr, 4711, 1}} = lists:keysearch(4711, 2, + element(2, + process_info(self(), + binary))), + {Dirty, DM} = spawn_opt(fun () -> + dirty_call_while_terminated_nif(Me), + blipp:blupp(Bin) + end, + [monitor,link]), + receive {dirty_alive, Pid} -> ok end, + {value, {BinAddr, 4711, 2}} = lists:keysearch(4711, 2, + element(2, + process_info(self(), + binary))), + Reason = die_dirty_process, + OT = process_flag(trap_exit, true), + exit(Dirty, Reason), + receive + {'DOWN', DM, process, Dirty, R0} -> + R0 = Reason + end, + receive + {'EXIT', Dirty, R1} -> + R1 = Reason + end, + undefined = process_info(Dirty), + undefined = process_info(Dirty, status), + false = erlang:is_process_alive(Dirty), + false = lists:member(Dirty, processes()), + %% Binary still refered by Dirty process not yet cleaned up + %% since the dirty nif has not yet returned... + {value, {BinAddr, 4711, 2}} = lists:keysearch(4711, 2, + element(2, + process_info(self(), + binary))), + receive after 2000 -> ok end, + receive + Msg -> + ct:fail({unexpected_message, Msg}) + after + 0 -> + ok + end, + {value, {BinAddr, 4711, 1}} = lists:keysearch(4711, 2, + element(2, + process_info(self(), + binary))), + process_flag(trap_exit, OT), + ok. + +dirty_heap_access(Config) when is_list(Config) -> + {ok, Node} = start_node(Config), + Me = self(), + RGL = rpc:call(Node,erlang,whereis,[init]), + Ref = rpc:call(Node,erlang,make_ref,[]), + Dirty = spawn_link(fun () -> + Res = dirty_heap_access_nif(Ref), + garbage_collect(), + Me ! {self(), Res}, + receive after infinity -> ok end + end), + {N, R} = access_dirty_heap(Dirty, RGL, 0, 0), + receive + {Pid, Res} -> + 1000 = length(Res), + lists:foreach(fun (X) -> Ref = X end, Res) + end, + unlink(Dirty), + exit(Dirty, kill), + stop_node(Node), + {comment, integer_to_list(N) ++ " GL change loops; " + ++ integer_to_list(R) ++ " while running dirty"}. + +access_dirty_heap(Dirty, RGL, N, R) -> + case process_info(Dirty, status) of + {status, waiting} -> + {N, R}; + {status, Status} -> + {group_leader, GL} = process_info(Dirty, group_leader), + true = group_leader(RGL, Dirty), + {group_leader, RGL} = process_info(Dirty, group_leader), + true = group_leader(GL, Dirty), + {group_leader, GL} = process_info(Dirty, group_leader), + access_dirty_heap(Dirty, RGL, N+1, case Status of + running -> + R+1; + _ -> + R + end) + end. + +%% +%% Internal... +%% + +receive_any() -> + receive M -> M end. + +start_node(Config) -> + start_node(Config, ""). + +start_node(Config, Args) when is_list(Config) -> + Pa = filename:dirname(code:which(?MODULE)), + Name = list_to_atom(atom_to_list(?MODULE) + ++ "-" + ++ atom_to_list(proplists:get_value(testcase, Config)) + ++ "-" + ++ integer_to_list(erlang:system_time(seconds)) + ++ "-" + ++ integer_to_list(erlang:unique_integer([positive]))), + test_server:start_node(Name, slave, [{args, "-pa "++Pa++" "++Args}]). + +stop_node(Node) -> + test_server:stop_node(Node). + +mcall(Node, Funs) -> + Parent = self(), + Refs = lists:map(fun (Fun) -> + Ref = make_ref(), + spawn_link(Node, + fun () -> + Res = Fun(), + unlink(Parent), + Parent ! {Ref, Res} + end), + Ref + end, Funs), + lists:map(fun (Ref) -> + receive + {Ref, Res} -> + Res + end + end, Refs). + +%% The NIFs: +lib_loaded() -> false. +call_nif_schedule(_,_) -> ?nif_stub. +call_dirty_nif(_,_,_) -> ?nif_stub. +send_from_dirty_nif(_) -> ?nif_stub. +call_dirty_nif_exception(_) -> ?nif_stub. +call_dirty_nif_zero_args() -> ?nif_stub. +dirty_call_while_terminated_nif(_) -> ?nif_stub. +dirty_sleeper() -> ?nif_stub. +dirty_heap_access_nif(_) -> ?nif_stub. + +nif_stub_error(Line) -> + exit({nif_not_loaded,module,?MODULE,line,Line}). diff --git a/erts/emulator/test/dirty_nif_SUITE_data/Makefile.src b/erts/emulator/test/dirty_nif_SUITE_data/Makefile.src new file mode 100644 index 0000000000..e9301753b0 --- /dev/null +++ b/erts/emulator/test/dirty_nif_SUITE_data/Makefile.src @@ -0,0 +1,6 @@ + +NIF_LIBS = dirty_nif_SUITE@dll@ + +all: $(NIF_LIBS) + +@SHLIB_RULES@ diff --git a/erts/emulator/test/dirty_nif_SUITE_data/dirty_nif_SUITE.c b/erts/emulator/test/dirty_nif_SUITE_data/dirty_nif_SUITE.c new file mode 100644 index 0000000000..2013c88167 --- /dev/null +++ b/erts/emulator/test/dirty_nif_SUITE_data/dirty_nif_SUITE.c @@ -0,0 +1,223 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2009-2014. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * %CopyrightEnd% + */ +#include "erl_nif.h" +#include <assert.h> +#ifndef __WIN32__ +#include <unistd.h> +#endif + +static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info) +{ + return 0; +} + +static ERL_NIF_TERM lib_loaded(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + return enif_make_atom(env, "true"); +} + +static int have_dirty_schedulers(void) +{ + ErlNifSysInfo si; + enif_system_info(&si, sizeof(si)); + return si.dirty_scheduler_support; +} + +static ERL_NIF_TERM dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + int n; + char s[10]; + ErlNifBinary b; + if (have_dirty_schedulers()) { + assert(enif_is_on_dirty_scheduler(env)); + } + assert(argc == 3); + enif_get_int(env, argv[0], &n); + enif_get_string(env, argv[1], s, sizeof s, ERL_NIF_LATIN1); + enif_inspect_binary(env, argv[2], &b); + return enif_make_tuple3(env, + enif_make_int(env, n), + enif_make_string(env, s, ERL_NIF_LATIN1), + enif_make_binary(env, &b)); +} + +static ERL_NIF_TERM call_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + int n; + char s[10]; + ErlNifBinary b; + assert(!enif_is_on_dirty_scheduler(env)); + if (argc != 3) + return enif_make_badarg(env); + if (have_dirty_schedulers()) { + if (enif_get_int(env, argv[0], &n) && + enif_get_string(env, argv[1], s, sizeof s, ERL_NIF_LATIN1) && + enif_inspect_binary(env, argv[2], &b)) + return enif_schedule_nif(env, "call_dirty_nif", ERL_NIF_DIRTY_JOB_CPU_BOUND, dirty_nif, argc, argv); + else + return enif_make_badarg(env); + } else { + return dirty_nif(env, argc, argv); + } +} + +static ERL_NIF_TERM send_from_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + ERL_NIF_TERM result; + ErlNifPid pid; + ErlNifEnv* menv; + int res; + + if (!enif_get_local_pid(env, argv[0], &pid)) + return enif_make_badarg(env); + result = enif_make_tuple2(env, enif_make_atom(env, "ok"), enif_make_pid(env, &pid)); + menv = enif_alloc_env(); + res = enif_send(env, &pid, menv, result); + enif_free_env(menv); + if (!res) + return enif_make_badarg(env); + else + return result; +} + +static ERL_NIF_TERM call_dirty_nif_exception(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + switch (argc) { + case 1: { + int arg; + if (enif_get_int(env, argv[0], &arg) && arg < 2) { + ERL_NIF_TERM args[255]; + int i; + args[0] = argv[0]; + for (i = 1; i < 255; i++) + args[i] = enif_make_int(env, i); + return enif_schedule_nif(env, "call_dirty_nif_exception", ERL_NIF_DIRTY_JOB_CPU_BOUND, + call_dirty_nif_exception, 255, args); + } else { + return enif_raise_exception(env, argv[0]); + } + } + case 2: { + int return_badarg_directly; + enif_get_int(env, argv[0], &return_badarg_directly); + assert(return_badarg_directly == 1 || return_badarg_directly == 0); + if (return_badarg_directly) + return enif_make_badarg(env); + else { + /* ignore return value */ enif_make_badarg(env); + return enif_make_atom(env, "ok"); + } + } + default: + return enif_schedule_nif(env, "call_dirty_nif_exception", ERL_NIF_DIRTY_JOB_CPU_BOUND, + call_dirty_nif_exception, argc-1, argv); + } +} + +static ERL_NIF_TERM call_dirty_nif_zero_args(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + int i; + ERL_NIF_TERM result[1000]; + ERL_NIF_TERM ok = enif_make_atom(env, "ok"); + assert(argc == 0); + for (i = 0; i < sizeof(result)/sizeof(*result); i++) { + result[i] = ok; + } + return enif_make_list_from_array(env, result, i); +} + +static ERL_NIF_TERM +dirty_sleeper(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + assert(enif_is_on_dirty_scheduler(env)); +#ifdef __WIN32__ + Sleep(6000); +#else + sleep(6); +#endif + return enif_make_atom(env, "ok"); +} + +static ERL_NIF_TERM dirty_call_while_terminated_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + ErlNifPid self; + ERL_NIF_TERM result, self_term; + ErlNifPid to; + ErlNifEnv* menv; + int res; + + if (!enif_get_local_pid(env, argv[0], &to)) + return enif_make_badarg(env); + + if (!enif_self(env, &self)) + return enif_make_badarg(env); + + self_term = enif_make_pid(env, &self); + + result = enif_make_tuple2(env, enif_make_atom(env, "dirty_alive"), self_term); + menv = enif_alloc_env(); + res = enif_send(env, &to, menv, result); + enif_free_env(menv); + if (!res) + return enif_make_badarg(env); + + /* Wait until we have been killed */ + while (enif_is_process_alive(env, &self)) + ; + + result = enif_make_tuple2(env, enif_make_atom(env, "dirty_dead"), self_term); + menv = enif_alloc_env(); + res = enif_send(env, &to, menv, result); + enif_free_env(menv); + +#ifdef __WIN32__ + Sleep(1000); +#else + sleep(1); +#endif + + return enif_make_atom(env, "ok"); +} + +static ERL_NIF_TERM dirty_heap_access_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + ERL_NIF_TERM res = enif_make_list(env, 0); + int i; + assert(enif_is_on_dirty_scheduler(env)); + for (i = 0; i < 1000; i++) + res = enif_make_list_cell(env, enif_make_copy(env, argv[0]), res); + + return res; +} + + +static ErlNifFunc nif_funcs[] = +{ + {"lib_loaded", 0, lib_loaded}, + {"call_dirty_nif", 3, call_dirty_nif}, + {"send_from_dirty_nif", 1, send_from_dirty_nif, ERL_NIF_DIRTY_JOB_CPU_BOUND}, + {"call_dirty_nif_exception", 1, call_dirty_nif_exception, ERL_NIF_DIRTY_JOB_IO_BOUND}, + {"call_dirty_nif_zero_args", 0, call_dirty_nif_zero_args, ERL_NIF_DIRTY_JOB_CPU_BOUND}, + {"dirty_sleeper", 0, dirty_sleeper, ERL_NIF_DIRTY_JOB_IO_BOUND}, + {"dirty_call_while_terminated_nif", 1, dirty_call_while_terminated_nif, ERL_NIF_DIRTY_JOB_CPU_BOUND}, + {"dirty_heap_access_nif", 1, dirty_heap_access_nif, ERL_NIF_DIRTY_JOB_CPU_BOUND} +}; + +ERL_NIF_INIT(dirty_nif_SUITE,nif_funcs,load,NULL,NULL,NULL) diff --git a/erts/emulator/test/gc_SUITE.erl b/erts/emulator/test/gc_SUITE.erl index 79c229a34d..8a600b7d9f 100644 --- a/erts/emulator/test/gc_SUITE.erl +++ b/erts/emulator/test/gc_SUITE.erl @@ -25,13 +25,13 @@ -include_lib("common_test/include/ct.hrl"). -export([all/0, suite/0]). --export([grow_heap/1, grow_stack/1, grow_stack_heap/1]). +-export([grow_heap/1, grow_stack/1, grow_stack_heap/1, max_heap_size/1]). suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> - [grow_heap, grow_stack, grow_stack_heap]. + [grow_heap, grow_stack, grow_stack_heap, max_heap_size]. %% Produce a growing list of elements, @@ -163,3 +163,30 @@ show_heap(String) -> {stack_size, SSize}=process_info(self(), stack_size), io:format("Heap/Stack "++String++"~p/~p", [HSize, SSize]). +%% Test that doing a remote GC that triggers the max heap size +%% kills the process. +max_heap_size(_Config) -> + + Pid = spawn_opt(fun long_receive/0,[{max_heap_size, 1024}, + {message_queue_data, on_heap}]), + [Pid ! lists:duplicate(I,I) || I <- lists:seq(1,100)], + Ref = erlang:monitor(process, Pid), + + %% Force messages to be viewed as part of heap + erlang:process_info(Pid, messages), + + %% Do the GC that triggers max heap + erlang:garbage_collect(Pid), + + %% Verify that max heap was triggered + receive + {'DOWN', Ref, process, Pid, killed} -> ok + after 5000 -> + ct:fail({process_did_not_die, Pid, erlang:process_info(Pid)}) + end. + +long_receive() -> + receive + after 10000 -> + ok + end. diff --git a/erts/emulator/test/nif_SUITE.erl b/erts/emulator/test/nif_SUITE.erl index a7767132ee..a0e9f1bad6 100644 --- a/erts/emulator/test/nif_SUITE.erl +++ b/erts/emulator/test/nif_SUITE.erl @@ -38,15 +38,15 @@ is_checks/1, get_length/1, make_atom/1, make_string/1, reverse_list_test/1, otp_9828/1, - otp_9668/1, consume_timeslice/1, dirty_nif/1, dirty_nif_send/1, - dirty_nif_exception/1, call_dirty_nif_exception/1, nif_schedule/1, + otp_9668/1, consume_timeslice/1, nif_schedule/1, nif_exception/1, call_nif_exception/1, nif_nan_and_inf/1, nif_atom_too_long/1, nif_monotonic_time/1, nif_time_offset/1, nif_convert_time_unit/1, nif_now_time/1, nif_cpu_time/1, nif_unique_integer/1, nif_is_process_alive/1, nif_is_port_alive/1, nif_term_to_binary/1, nif_binary_to_term/1, - nif_port_command/1 + nif_port_command/1, + nif_snprintf/1 ]). -export([many_args_100/100]). @@ -75,14 +75,13 @@ all() -> make_string,reverse_list_test, otp_9828, otp_9668, consume_timeslice, - nif_schedule, dirty_nif, dirty_nif_send, dirty_nif_exception, - nif_exception, nif_nan_and_inf, nif_atom_too_long, + nif_schedule, nif_exception, nif_nan_and_inf, nif_atom_too_long, nif_monotonic_time, nif_time_offset, nif_convert_time_unit, nif_now_time, nif_cpu_time, nif_unique_integer, nif_is_process_alive, nif_is_port_alive, nif_term_to_binary, nif_binary_to_term, - nif_port_command - ]. + nif_port_command, + nif_snprintf]. init_per_testcase(_Case, Config) -> Config. @@ -1541,76 +1540,6 @@ nif_schedule(Config) when is_list(Config) -> end, ok. -dirty_nif(Config) when is_list(Config) -> - try erlang:system_info(dirty_cpu_schedulers) of - N when is_integer(N) -> - ensure_lib_loaded(Config), - Val1 = 42, - Val2 = "Erlang", - Val3 = list_to_binary([Val2, 0]), - {Val1, Val2, Val3} = call_dirty_nif(Val1, Val2, Val3), - LargeArray = lists:duplicate(1000, ok), - LargeArray = call_dirty_nif_zero_args(), - ok - catch - error:badarg -> - {skipped,"No dirty scheduler support"} - end. - -dirty_nif_send(Config) when is_list(Config) -> - try erlang:system_info(dirty_cpu_schedulers) of - N when is_integer(N) -> - ensure_lib_loaded(Config), - Parent = self(), - Pid = spawn_link(fun() -> - Self = self(), - {ok, Self} = receive_any(), - Parent ! {ok, Self} - end), - {ok, Pid} = send_from_dirty_nif(Pid), - {ok, Pid} = receive_any(), - ok - catch - error:badarg -> - {skipped,"No dirty scheduler support"} - end. - -dirty_nif_exception(Config) when is_list(Config) -> - try erlang:system_info(dirty_cpu_schedulers) of - N when is_integer(N) -> - ensure_lib_loaded(Config), - try - %% this checks that the expected exception occurs when the - %% dirty NIF returns the result of enif_make_badarg - %% directly - call_dirty_nif_exception(1), - ct:fail(expected_badarg) - catch - error:badarg -> - [{?MODULE,call_dirty_nif_exception,[1],_}|_] = - erlang:get_stacktrace(), - ok - end, - try - %% this checks that the expected exception occurs when the - %% dirty NIF calls enif_make_badarg at some point but then - %% returns a value that isn't an exception - call_dirty_nif_exception(0), - ct:fail(expected_badarg) - catch - error:badarg -> - [{?MODULE,call_dirty_nif_exception,[0],_}|_] = - erlang:get_stacktrace(), - ok - end, - %% this checks that a dirty NIF can raise various terms as - %% exceptions - ok = nif_raise_exceptions(call_dirty_nif_exception) - catch - error:badarg -> - {skipped,"No dirty scheduler support"} - end. - nif_exception(Config) when is_list(Config) -> ensure_lib_loaded(Config), try @@ -2024,9 +1953,18 @@ nif_port_command(Config) -> port_close(Port), {'EXIT', {badarg, _}} = (catch port_command_nif(Port, "hello\n")), + ok. +nif_snprintf(Config) -> + ensure_lib_loaded(Config), + <<"ok",0>> = format_term_nif(3,ok), + <<"o",0>> = format_term_nif(2,ok), + <<"\"hello world\"",0>> = format_term_nif(14,"hello world"), + <<"{{hello,world,-33},3.14",_/binary>> = format_term_nif(50,{{hello,world, -33}, 3.14, self()}), + <<"{{hello,world,-33},",0>> = format_term_nif(20,{{hello,world, -33}, 3.14, self()}), ok. + %% The NIFs: lib_version() -> undefined. call_history() -> ?nif_stub. @@ -2078,10 +2016,6 @@ otp_9668_nif(_) -> ?nif_stub. otp_9828_nif(_) -> ?nif_stub. consume_timeslice_nif(_,_) -> ?nif_stub. call_nif_schedule(_,_) -> ?nif_stub. -call_dirty_nif(_,_,_) -> ?nif_stub. -send_from_dirty_nif(_) -> ?nif_stub. -call_dirty_nif_exception(_) -> ?nif_stub. -call_dirty_nif_zero_args() -> ?nif_stub. call_nif_exception(_) -> ?nif_stub. call_nif_nan_or_inf(_) -> ?nif_stub. call_nif_atom_too_long(_) -> ?nif_stub. @@ -2091,6 +2025,7 @@ is_port_alive_nif(_) -> ?nif_stub. term_to_binary_nif(_, _) -> ?nif_stub. binary_to_term_nif(_, _, _) -> ?nif_stub. port_command_nif(_, _) -> ?nif_stub. +format_term_nif(_,_) -> ?nif_stub. %% maps is_map_nif(_) -> ?nif_stub. diff --git a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c index 11e5dab58e..13846244d4 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c +++ b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c @@ -23,6 +23,9 @@ #include <string.h> #include <assert.h> #include <limits.h> +#ifndef __WIN32__ +#include <unistd.h> +#endif #include "nif_mod.h" @@ -1574,120 +1577,6 @@ static ERL_NIF_TERM call_nif_schedule(ErlNifEnv* env, int argc, const ERL_NIF_TE return result; } -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT - -static int have_dirty_schedulers(void) -{ - ErlNifSysInfo si; - enif_system_info(&si, sizeof(si)); - return si.dirty_scheduler_support; -} - -static ERL_NIF_TERM dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ - int n; - char s[10]; - ErlNifBinary b; - ERL_NIF_TERM result; - if (have_dirty_schedulers()) { - assert(enif_is_on_dirty_scheduler(env)); - } - assert(argc == 3); - enif_get_int(env, argv[0], &n); - enif_get_string(env, argv[1], s, sizeof s, ERL_NIF_LATIN1); - enif_inspect_binary(env, argv[2], &b); - return enif_make_tuple3(env, - enif_make_int(env, n), - enif_make_string(env, s, ERL_NIF_LATIN1), - enif_make_binary(env, &b)); -} - -static ERL_NIF_TERM call_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ - int n; - char s[10]; - ErlNifBinary b; - assert(!enif_is_on_dirty_scheduler(env)); - if (argc != 3) - return enif_make_badarg(env); - if (have_dirty_schedulers()) { - if (enif_get_int(env, argv[0], &n) && - enif_get_string(env, argv[1], s, sizeof s, ERL_NIF_LATIN1) && - enif_inspect_binary(env, argv[2], &b)) - return enif_schedule_nif(env, "call_dirty_nif", ERL_NIF_DIRTY_JOB_CPU_BOUND, dirty_nif, argc, argv); - else - return enif_make_badarg(env); - } else { - return dirty_nif(env, argc, argv); - } -} - -static ERL_NIF_TERM send_from_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ - ERL_NIF_TERM result; - ErlNifPid pid; - ErlNifEnv* menv; - int res; - - if (!enif_get_local_pid(env, argv[0], &pid)) - return enif_make_badarg(env); - result = enif_make_tuple2(env, enif_make_atom(env, "ok"), enif_make_pid(env, &pid)); - menv = enif_alloc_env(); - res = enif_send(env, &pid, menv, result); - enif_free_env(menv); - if (!res) - return enif_make_badarg(env); - else - return result; -} - -static ERL_NIF_TERM call_dirty_nif_exception(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ - switch (argc) { - case 1: { - int arg; - if (enif_get_int(env, argv[0], &arg) && arg < 2) { - ERL_NIF_TERM args[255]; - int i; - args[0] = argv[0]; - for (i = 1; i < 255; i++) - args[i] = enif_make_int(env, i); - return enif_schedule_nif(env, "call_dirty_nif_exception", ERL_NIF_DIRTY_JOB_CPU_BOUND, - call_dirty_nif_exception, 255, args); - } else { - return enif_raise_exception(env, argv[0]); - } - } - case 2: { - int return_badarg_directly; - enif_get_int(env, argv[0], &return_badarg_directly); - assert(return_badarg_directly == 1 || return_badarg_directly == 0); - if (return_badarg_directly) - return enif_make_badarg(env); - else { - /* ignore return value */ enif_make_badarg(env); - return enif_make_atom(env, "ok"); - } - } - default: - return enif_schedule_nif(env, "call_dirty_nif_exception", ERL_NIF_DIRTY_JOB_CPU_BOUND, - call_dirty_nif_exception, argc-1, argv); - } -} - -static ERL_NIF_TERM call_dirty_nif_zero_args(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ - int i; - ERL_NIF_TERM result[1000]; - ERL_NIF_TERM ok = enif_make_atom(env, "ok"); - assert(argc == 0); - for (i = 0; i < sizeof(result)/sizeof(*result); i++) { - result[i] = ok; - } - return enif_make_list_from_array(env, result, i); -} -#endif - /* * If argv[0] is the integer 0, call enif_make_badarg, but don't return its * return value. Instead, return ok. Result should still be a badarg @@ -2109,6 +1998,23 @@ static ERL_NIF_TERM port_command(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar return atom_true; } +static ERL_NIF_TERM format_term(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + ErlNifBinary obin; + unsigned int size; + + if (!enif_get_uint(env, argv[0], &size)) + return enif_make_badarg(env); + if (!enif_alloc_binary(size,&obin)) + return enif_make_badarg(env); + + if (enif_snprintf((char*)obin.data, (size_t)size, "%T", argv[1]) < 0) + return atom_false; + + return enif_make_binary(env,&obin); +} + + static ErlNifFunc nif_funcs[] = { {"lib_version", 0, lib_version}, @@ -2162,12 +2068,6 @@ static ErlNifFunc nif_funcs[] = {"otp_9828_nif", 1, otp_9828_nif}, {"consume_timeslice_nif", 2, consume_timeslice_nif}, {"call_nif_schedule", 2, call_nif_schedule}, -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT - {"call_dirty_nif", 3, call_dirty_nif}, - {"send_from_dirty_nif", 1, send_from_dirty_nif, ERL_NIF_DIRTY_JOB_CPU_BOUND}, - {"call_dirty_nif_exception", 1, call_dirty_nif_exception, ERL_NIF_DIRTY_JOB_IO_BOUND}, - {"call_dirty_nif_zero_args", 0, call_dirty_nif_zero_args, ERL_NIF_DIRTY_JOB_CPU_BOUND}, -#endif {"call_nif_exception", 1, call_nif_exception}, {"call_nif_nan_or_inf", 1, call_nif_nan_or_inf}, {"call_nif_atom_too_long", 1, call_nif_atom_too_long}, @@ -2190,7 +2090,8 @@ static ErlNifFunc nif_funcs[] = {"is_port_alive_nif", 1, is_port_alive}, {"term_to_binary_nif", 2, term_to_binary}, {"binary_to_term_nif", 3, binary_to_term}, - {"port_command_nif", 2, port_command} + {"port_command_nif", 2, port_command}, + {"format_term_nif", 2, format_term} }; ERL_NIF_INIT(nif_SUITE,nif_funcs,load,reload,upgrade,unload) diff --git a/erts/emulator/test/process_SUITE.erl b/erts/emulator/test/process_SUITE.erl index 61a68f9759..eaa4026a8a 100644 --- a/erts/emulator/test/process_SUITE.erl +++ b/erts/emulator/test/process_SUITE.erl @@ -46,7 +46,7 @@ process_status_exiting/1, otp_4725/1, bad_register/1, garbage_collect/1, otp_6237/1, process_info_messages/1, process_flag_badarg/1, process_flag_heap_size/1, - spawn_opt_heap_size/1, + spawn_opt_heap_size/1, spawn_opt_max_heap_size/1, processes_large_tab/1, processes_default_tab/1, processes_small_tab/1, processes_this_tab/1, processes_apply_trap/1, processes_last_call_trap/1, processes_gc_trap/1, @@ -60,7 +60,7 @@ system_task_on_suspended/1, gc_request_when_gc_disabled/1, gc_request_blast_when_gc_disabled/1]). --export([prio_server/2, prio_client/2]). +-export([prio_server/2, prio_client/2, init/1, handle_event/2]). -export([init_per_testcase/2, end_per_testcase/2]). @@ -84,7 +84,8 @@ all() -> bump_reductions, low_prio, yield, yield2, otp_4725, bad_register, garbage_collect, process_info_messages, process_flag_badarg, process_flag_heap_size, - spawn_opt_heap_size, otp_6237, {group, processes_bif}, + spawn_opt_heap_size, spawn_opt_max_heap_size, otp_6237, + {group, processes_bif}, {group, otp_7738}, garb_other_running, {group, system_task}]. @@ -425,6 +426,8 @@ t_process_info(Config) when is_list(Config) -> {status, running} = process_info(self(), status), {min_heap_size, 233} = process_info(self(), min_heap_size), {min_bin_vheap_size,46422} = process_info(self(), min_bin_vheap_size), + {max_heap_size, #{ size := 0, kill := true, error_logger := true}} = + process_info(self(), max_heap_size), {current_function,{?MODULE,t_process_info,1}} = process_info(self(), current_function), {current_function,{?MODULE,t_process_info,1}} = @@ -564,6 +567,8 @@ process_info_other_msg(Config) when is_list(Config) -> {min_heap_size, 233} = process_info(Pid, min_heap_size), {min_bin_vheap_size, 46422} = process_info(Pid, min_bin_vheap_size), + {max_heap_size, #{ size := 0, kill := true, error_logger := true}} = + process_info(self(), max_heap_size), Pid ! stop, ok. @@ -914,10 +919,14 @@ process_info_garbage_collection(_Config) -> Parent = self(), Pid = spawn_link( fun() -> + %% We set mqd to off_heap and send an tuple + %% to process in order to force mbuf_size + %% to be used + process_flag(message_queue_data, off_heap), receive go -> ok end, (fun F(0) -> Parent ! deep, - receive ok -> ok end, + receive {ok,_} -> ok end, []; F(N) -> timer:sleep(1), @@ -926,31 +935,52 @@ process_info_garbage_collection(_Config) -> Parent ! shallow, receive done -> ok end end), - {garbage_collection_info, Before} = - erlang:process_info(Pid, garbage_collection_info), + [{garbage_collection_info, Before},{total_heap_size, THSBefore}] = + erlang:process_info(Pid, [garbage_collection_info, total_heap_size]), Pid ! go, receive deep -> ok end, - {_, Deep} = erlang:process_info(Pid, garbage_collection_info), - Pid ! ok, receive shallow -> ok end, - {_, After} = erlang:process_info(Pid, garbage_collection_info), + [{_, Deep},{_,THSDeep}] = + erlang:process_info(Pid, [garbage_collection_info, total_heap_size]), + Pid ! {ok, make_ref()}, receive shallow -> ok end, + [{_, After},{_, THSAfter}] = + erlang:process_info(Pid, [garbage_collection_info, total_heap_size]), Pid ! done, %% Do some general checks to see if everything seems to be roughly correct ct:log("Before: ~p",[Before]), ct:log("Deep: ~p",[Deep]), ct:log("After: ~p",[After]), + ct:log("Before THS: ~p",[THSBefore]), + ct:log("Deep THS: ~p",[THSDeep]), + ct:log("After THS: ~p",[THSAfter]), %% Check stack_size - true = proplists:get_value(stack_size, Before) < proplists:get_value(stack_size, Deep), - true = proplists:get_value(stack_size, After) < proplists:get_value(stack_size, Deep), + true = gv(stack_size, Before) < gv(stack_size, Deep), + true = gv(stack_size, After) < gv(stack_size, Deep), %% Check used heap size - true = proplists:get_value(heap_size, Before) + proplists:get_value(old_heap_size, Before) - < proplists:get_value(heap_size, Deep) + proplists:get_value(old_heap_size, Deep), - true = proplists:get_value(heap_size, Before) + proplists:get_value(old_heap_size, Before) - < proplists:get_value(heap_size, After) + proplists:get_value(old_heap_size, After), + true = gv(heap_size, Before) + gv(old_heap_size, Before) + < gv(heap_size, Deep) + gv(old_heap_size, Deep), + true = gv(heap_size, Before) + gv(old_heap_size, Before) + < gv(heap_size, After) + gv(old_heap_size, After), + + %% Check that total_heap_size == heap_block_size + old_heap_block_size + mbuf_size + THSBefore = gv(heap_block_size, Before) + + gv(old_heap_block_size, Before) + + gv(mbuf_size, Before), + + THSDeep = gv(heap_block_size, Deep) + + gv(old_heap_block_size, Deep) + + gv(mbuf_size, Deep), + + THSAfter = gv(heap_block_size, After) + + gv(old_heap_block_size, After) + + gv(mbuf_size, After), ok. +gv(Key,List) -> + proplists:get_value(Key,List). + %% Tests erlang:bump_reductions/1. bump_reductions(Config) when is_list(Config) -> erlang:garbage_collect(), @@ -1323,6 +1353,28 @@ process_flag_badarg(Config) when is_list(Config) -> chk_badarg(fun () -> process_flag(min_heap_size, gurka) end), chk_badarg(fun () -> process_flag(min_bin_vheap_size, gurka) end), chk_badarg(fun () -> process_flag(min_bin_vheap_size, -1) end), + + chk_badarg(fun () -> process_flag(max_heap_size, gurka) end), + chk_badarg(fun () -> process_flag(max_heap_size, -1) end), + chk_badarg(fun () -> + {_,Min} = process_info(self(), min_heap_size), + process_flag(max_heap_size, Min - 1) + end), + chk_badarg(fun () -> + {_,Min} = process_info(self(), min_heap_size), + process_flag(max_heap_size, #{size => Min - 1}) + end), + chk_badarg(fun () -> process_flag(max_heap_size, #{}) end), + chk_badarg(fun () -> process_flag(max_heap_size, #{ kill => true }) end), + chk_badarg(fun () -> process_flag(max_heap_size, #{ size => 233, + kill => gurka }) end), + chk_badarg(fun () -> process_flag(max_heap_size, #{ size => 233, + error_logger => gurka }) end), + chk_badarg(fun () -> process_flag(max_heap_size, #{ size => 233, + kill => true, + error_logger => gurka }) end), + chk_badarg(fun () -> process_flag(max_heap_size, #{ size => 1 bsl 64 }) end), + chk_badarg(fun () -> process_flag(priority, 4711) end), chk_badarg(fun () -> process_flag(save_calls, hmmm) end), P= spawn_link(fun () -> receive die -> ok end end), @@ -1923,6 +1975,110 @@ spawn_opt_heap_size(Config) when is_list(Config) -> Pid ! stop, ok. +spawn_opt_max_heap_size(_Config) -> + + error_logger:add_report_handler(?MODULE, self()), + + %% Test that numerical limit works + max_heap_size_test(1024, 1024, true, true), + + %% Test that map limit works + max_heap_size_test(#{ size => 1024 }, 1024, true, true), + + %% Test that no kill is sent + max_heap_size_test(#{ size => 1024, kill => false }, 1024, false, true), + + %% Test that no error_logger report is sent + max_heap_size_test(#{ size => 1024, error_logger => false }, 1024, true, false), + + %% Test that system_flag works + erlang:system_flag(max_heap_size, #{ size => 0, kill => false, + error_logger => true}), + max_heap_size_test(#{ size => 1024 }, 1024, false, true), + max_heap_size_test(#{ size => 1024, kill => true }, 1024, true, true), + + erlang:system_flag(max_heap_size, #{ size => 0, kill => true, + error_logger => false}), + max_heap_size_test(#{ size => 1024 }, 1024, true, false), + max_heap_size_test(#{ size => 1024, error_logger => true }, 1024, true, true), + + erlang:system_flag(max_heap_size, #{ size => 1 bsl 20, kill => true, + error_logger => true}), + max_heap_size_test(#{ }, 1 bsl 20, true, true), + + erlang:system_flag(max_heap_size, #{ size => 0, kill => true, + error_logger => true}), + + %% Test that ordinary case works as expected again + max_heap_size_test(1024, 1024, true, true), + + ok. + +max_heap_size_test(Option, Size, Kill, ErrorLogger) + when map_size(Option) == 0 -> + max_heap_size_test([], Size, Kill, ErrorLogger); +max_heap_size_test(Option, Size, Kill, ErrorLogger) + when is_map(Option); is_integer(Option) -> + max_heap_size_test([{max_heap_size, Option}], Size, Kill, ErrorLogger); +max_heap_size_test(Option, Size, Kill, ErrorLogger) -> + OomFun = fun F() -> timer:sleep(5),[lists:seq(1,1000)|F()] end, + Pid = spawn_opt(OomFun, Option), + {max_heap_size, MHSz} = erlang:process_info(Pid, max_heap_size), + ct:log("Default: ~p~nOption: ~p~nProc: ~p~n", + [erlang:system_info(max_heap_size), Option, MHSz]), + + #{ size := Size} = MHSz, + + Ref = erlang:monitor(process, Pid), + if Kill -> + receive + {'DOWN', Ref, process, Pid, killed} -> + ok + end; + true -> + ok + end, + if ErrorLogger -> + receive + {error, _, {emulator, _, [Pid|_]}} -> + ok + end; + true -> + ok + end, + if not Kill -> + exit(Pid, die), + receive + {'DOWN', Ref, process, Pid, die} -> + ok + end, + flush(); + true -> + ok + end, + receive + M -> + ct:fail({unexpected_message, M}) + after 10 -> + ok + end. + +flush() -> + receive + _M -> + flush() + after 1000 -> + ok + end. + +%% error_logger report handler proxy +init(Pid) -> + {ok, Pid}. + +handle_event(Event, Pid) -> + Pid ! Event, + {ok, Pid}. + processes_term_proc_list(Config) when is_list(Config) -> Tester = self(), as_expected = processes_term_proc_list_test(false), diff --git a/erts/emulator/test/scheduler_SUITE.erl b/erts/emulator/test/scheduler_SUITE.erl index 6b49b68ec8..f18d79d770 100644 --- a/erts/emulator/test/scheduler_SUITE.erl +++ b/erts/emulator/test/scheduler_SUITE.erl @@ -57,7 +57,6 @@ scheduler_suspend_basic/1, scheduler_suspend/1, dirty_scheduler_threads/1, - dirty_scheduler_exit/1, reader_groups/1]). suite() -> @@ -72,7 +71,7 @@ all() -> bound_process, {group, scheduler_bind}, scheduler_threads, scheduler_suspend_basic, scheduler_suspend, - dirty_scheduler_threads, dirty_scheduler_exit, + dirty_scheduler_threads, reader_groups]. groups() -> @@ -1162,53 +1161,6 @@ get_dsstate(Config, Cmd) -> stop_node(Node), {DSCPU, DSCPUOnln, DSIO}. -dirty_scheduler_exit(Config) when is_list(Config) -> - try - erlang:system_info(dirty_cpu_schedulers), - dirty_scheduler_exit_test(Config) - catch - error:badarg -> - {skipped, "No dirty scheduler support"} - end. - -dirty_scheduler_exit_test(Config) -> - {ok, Node} = start_node(Config, "+SDio 1"), - [ok] = mcall(Node, - [fun() -> - Path = proplists:get_value(data_dir, Config), - Lib = atom_to_list(?MODULE), - ok = erlang:load_nif(filename:join(Path,Lib), []), - ok = test_dirty_scheduler_exit() - end]), - stop_node(Node), - ok. - -test_dirty_scheduler_exit() -> - process_flag(trap_exit,true), - test_dse(10,[]). -test_dse(0,Pids) -> - timer:sleep(100), - kill_dse(Pids,[]); -test_dse(N,Pids) -> - Pid = spawn_link(fun dirty_sleeper/0), - test_dse(N-1,[Pid|Pids]). -kill_dse([],Killed) -> - wait_dse(Killed); -kill_dse([Pid|Pids],AlreadyKilled) -> - exit(Pid,kill), - kill_dse(Pids,[Pid|AlreadyKilled]). -wait_dse([]) -> - ok; -wait_dse([Pid|Pids]) -> - receive - {'EXIT',Pid,killed} -> - ok - end, - wait_dse(Pids). - -dirty_sleeper() -> - erlang:nif_error({error,?MODULE}). - scheduler_suspend_basic(Config) when is_list(Config) -> case erlang:system_info(multi_scheduling) of disabled -> diff --git a/erts/emulator/test/scheduler_SUITE_data/Makefile.src b/erts/emulator/test/scheduler_SUITE_data/Makefile.src deleted file mode 100644 index 859112cf19..0000000000 --- a/erts/emulator/test/scheduler_SUITE_data/Makefile.src +++ /dev/null @@ -1,8 +0,0 @@ - -SCHEDULER_LIBS = scheduler_SUITE@dll@ - -all: $(SCHEDULER_LIBS) - -@SHLIB_RULES@ - -$(SCHEDULER_LIBS): scheduler_SUITE.c diff --git a/erts/emulator/test/scheduler_SUITE_data/scheduler_SUITE.c b/erts/emulator/test/scheduler_SUITE_data/scheduler_SUITE.c deleted file mode 100644 index ab4863337f..0000000000 --- a/erts/emulator/test/scheduler_SUITE_data/scheduler_SUITE.c +++ /dev/null @@ -1,37 +0,0 @@ -#ifndef __WIN32__ -#include <unistd.h> -#endif -#include "erl_nif.h" - -static int -load(ErlNifEnv* env, void** priv, ERL_NIF_TERM info) -{ - ErlNifSysInfo sys_info; - enif_system_info(&sys_info, sizeof(ErlNifSysInfo)); - if (!sys_info.smp_support || !sys_info.dirty_scheduler_support) - return 1; - return 0; -} - -static ERL_NIF_TERM -dirty_sleeper(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) -{ -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT -#ifdef __WIN32__ - Sleep(3000); -#else - sleep(3); -#endif -#endif - return enif_make_atom(env, "ok"); -} - -static ErlNifFunc funcs[] = { -#ifdef ERL_NIF_DIRTY_SCHEDULER_SUPPORT - {"dirty_sleeper", 0, dirty_sleeper, ERL_NIF_DIRTY_JOB_IO_BOUND} -#else - {"dirty_sleeper", 0, dirty_sleeper, 0} -#endif -}; - -ERL_NIF_INIT(scheduler_SUITE, funcs, &load, NULL, NULL, NULL); diff --git a/erts/emulator/test/trace_SUITE.erl b/erts/emulator/test/trace_SUITE.erl index 29e043dd5c..da6a6bdea4 100644 --- a/erts/emulator/test/trace_SUITE.erl +++ b/erts/emulator/test/trace_SUITE.erl @@ -84,7 +84,6 @@ cpu_timestamp(Config) when is_list(Config) -> receive_trace(Config) when is_list(Config) -> Receiver = fun_spawn(fun receiver/0), - process_flag(trap_exit, true), %% Trace the process; make sure that we receive the trace messages. 1 = erlang:trace(Receiver, true, ['receive']), @@ -96,15 +95,116 @@ receive_trace(Config) when is_list(Config) -> {trace, Receiver, 'receive', Hello2} = receive_first_trace(), receive_nothing(), + %% Test 'receive' with matchspec + F1 = fun ({Pat, IsMatching}) -> + set_trace_pattern('receive', Pat, []), + Receiver ! Hello, + case IsMatching of + true -> + {trace, Receiver, 'receive', Hello} = receive_first_trace(); + false -> + ok + end, + receive_nothing() + end, + From = self(), + Node = node(), + lists:foreach(F1, [{no, true}, + {[{[Node, undefined,"Unexpected"],[],[]}], false}, + {[{[Node, From,'_'],[],[]}], true}, + {[{[Node, '$1','_'],[{'=/=','$1',From}],[]}], false}, + {[{['$1', '_','_'],[{'=:=','$1',Node}],[]}], true}, + {false, false}, + {true, true}]), + + %% Remote messages + OtherName = atom_to_list(?MODULE)++"_receive_trace", + {ok, OtherNode} = start_node(OtherName), + RemoteProc = spawn_link(OtherNode, ?MODULE, process, [self()]), + io:format("RemoteProc = ~p ~n", [RemoteProc]), + + RemoteProc ! {send_please, Receiver, Hello}, + {trace, Receiver, 'receive', Hello} = receive_first_trace(), + RemoteProc ! {send_please, Receiver, 99}, + {trace, Receiver, 'receive', 99} = receive_first_trace(), + + %% Remote with matchspec + F2 = fun (To, {Pat, IsMatching}) -> + set_trace_pattern('receive', Pat, []), + RemoteProc ! {send_please, To, Hello}, + case IsMatching of + true -> + {trace, Receiver, 'receive', Hello} = receive_first_trace(); + false -> + ok + end, + receive_nothing() + end, + F2(Receiver, {no, true}), + F2(Receiver, {[{[OtherNode, undefined,"Unexpected"],[],[]}], false}), + F2(Receiver, {[{[OtherNode, RemoteProc,'_'],[],[]}, + {[OtherNode, undefined,'_'],[],[]}], true}), + F2(Receiver, {[{[OtherNode, '$1','_'], + [{'orelse',{'=:=','$1',undefined},{'=/=',{node,'$1'},{node}}}], + []}], true}), + F2(Receiver, {[{['$1', '_','_'], [{'=:=','$1',OtherNode}], []}], true}), + F2(Receiver, {false, false}), + F2(Receiver, {true, true}), + + %% Remote to named with matchspec + Name = trace_SUITE_receiver, + register(Name, Receiver), + NN = {Name, node()}, + F2(NN, {no, true}), + F2(NN, {[{[OtherNode, undefined,"Unexpected"],[],[]}], false}), + F2(NN, {[{[OtherNode, RemoteProc,'_'],[],[]}, + {[OtherNode, undefined,'_'],[],[]}], true}), + F2(NN, {[{[OtherNode, '$1','_'], + [{'orelse',{'=:=','$1',undefined},{'=/=',{node,'$1'},{node}}}], + []}], true}), + F2(NN, {[{['$1', '_','_'], [{'==','$1',OtherNode}], []}], true}), + F2(NN, {false, false}), + F2(NN, {true, true}), + + unlink(RemoteProc), + true = stop_node(OtherNode), + + %% Timeout + Receiver ! {set_timeout, 10}, + {trace, Receiver, 'receive', {set_timeout, 10}} = receive_first_trace(), + {trace, Receiver, 'receive', timeout} = receive_first_trace(), + erlang:trace_pattern('receive', [{[clock_service,undefined,timeout], [], []}], []), + Receiver ! {set_timeout, 7}, + {trace, Receiver, 'receive', timeout} = receive_first_trace(), + erlang:trace_pattern('receive', true, []), + %% Another process should not be able to trace Receiver. + process_flag(trap_exit, true), Intruder = fun_spawn(fun() -> erlang:trace(Receiver, true, ['receive']) end), {'EXIT', Intruder, {badarg, _}} = receive_first(), %% Untrace the process; we should not receive anything. - 1 = erlang:trace(Receiver, false, ['receive']), - Receiver ! {hello, there}, - Receiver ! any_garbage, - receive_nothing(), + ?line 1 = erlang:trace(Receiver, false, ['receive']), + ?line Receiver ! {hello, there}, + ?line Receiver ! any_garbage, + ?line receive_nothing(), + + %% Verify restrictions in matchspec for 'receive' + F3 = fun (Pat) -> {'EXIT', {badarg,_}} = (catch erlang:trace_pattern('receive', Pat, [])) end, + WC = ['_','_','_'], + F3([{WC,[],[{message, {process_dump}}]}]), + F3([{WC,[{is_seq_trace}],[]}]), + F3([{WC,[],[{set_seq_token,label,4711}]}]), + F3([{WC,[],[{get_seq_token}]}]), + F3([{WC,[],[{enable_trace,call}]}]), + F3([{WC,[],[{enable_trace,self(),call}]}]), + F3([{WC,[],[{disable_trace,call}]}]), + F3([{WC,[],[{disable_trace,self(),call}]}]), + F3([{WC,[],[{trace,[call],[]}]}]), + F3([{WC,[],[{trace,self(),[],[call]}]}]), + F3([{WC,[],[{caller}]}]), + F3([{WC,[],[{silent,true}]}]), + ok. %% Tests that receive of a message always happens before a call with @@ -257,30 +357,111 @@ send_trace(Config) when is_list(Config) -> %% Check that a message sent to another process is traced. 1 = erlang:trace(Sender, true, [send]), + F1 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, Receiver, to_receiver}, + {trace, Sender, send, to_receiver, Receiver} = receive_first_trace(), + receive_nothing() + end, + lists:foreach(F1, [no, + [{[Receiver,to_receiver],[],[]}], + [{['_','_'],[],[]}], + [{['$1','_'],[{is_pid,'$1'}],[]}], + [{['_','$1'],[{is_atom,'$1'}],[]}], + true]), + + %% Test {message, Msg} + F1m = fun ({Pat, Msg}) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, Receiver, to_receiver}, + {trace, Sender, send, to_receiver, Receiver, Msg} = receive_first_trace(), + receive_nothing() + end, + lists:foreach(F1m, [{[{['_','_'],[],[{message, 4711}]}], 4711}, + {[{['_','_'],[],[{message, "4711"}]}], "4711"} + ]), + + %% Test {message, {process_dump}} + set_trace_pattern(send, [{['_','_'],[],[{message, {process_dump}}]}], []), Sender ! {send_please, Receiver, to_receiver}, - {trace, Sender, send, to_receiver, Receiver} = receive_first_trace(), + {trace, Sender, send, to_receiver, Receiver, ProcDump} = receive_first_trace(), + true = is_binary(ProcDump), receive_nothing(), + %% Same test with false match spec + F2 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, Receiver, to_receiver}, + receive_nothing() + end, + lists:foreach(F2, [[{[Sender,to_receiver],[],[]}], + [{[Receiver,nomatch],[],[]}], + [{['$1','_'],[{is_atom,'$1'}],[]}], + [{['_','$1'],[{is_pid,'$1'}],[]}], + false, + [{['_','_'],[],[{message,false}]}], + [{['_','_'],[],[{silent,true}]}]]), + erlang:trace_pattern(send, true, []), + erlang:trace(Sender, false, [silent]), + %% Check that a message sent to another registered process is traced. register(?MODULE,Receiver), - Sender ! {send_please, ?MODULE, to_receiver}, - {trace, Sender, send, to_receiver, ?MODULE} = receive_first_trace(), - receive_nothing(), + F3 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, ?MODULE, to_receiver}, + {trace, Sender, send, to_receiver, ?MODULE} = receive_first_trace(), + receive_nothing() + end, + lists:foreach(F3, [no, + [{[?MODULE,to_receiver],[],[]}], + [{['_','_'],[],[]}], + [{['$1','_'],[{is_atom,'$1'}],[]}], + [{['_','$1'],[{is_atom,'$1'}],[]}], + true]), + %% Again with false match spec + F4 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, ?MODULE, to_receiver}, + receive_nothing() + end, + lists:foreach(F4, [[{[nomatch,to_receiver],[],[]}], + [{[?MODULE,nomatch],[],[]}], + [{['$1','_'],[{is_pid,'$1'}],[]}], + [{['_','$1'],[{is_pid,'$1'}],[]}], + [{['_','_'],[],[{message,false}]}], + [{['_','_'],[],[{silent,true}]}] + ]), unregister(?MODULE), + erlang:trace_pattern(send, true, []), + erlang:trace(Sender, false, [silent]), %% Check that a message sent to this process is traced. - Sender ! {send_please, self(), to_myself}, - receive to_myself -> ok end, - Self = self(), - {trace, Sender, send, to_myself, Self} = receive_first_trace(), - receive_nothing(), + F5 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, self(), to_myself}, + receive to_myself -> ok end, + Self = self(), + {trace, Sender, send, to_myself, Self} = receive_first_trace(), + receive_nothing() + end, + lists:foreach(F5, [no, + [{[self(),to_myself],[],[]}], + [{['_','_'],[],[]}], + true]), %% Check that a message sent to dead process is traced. {Pid,Ref} = spawn_monitor(fun() -> ok end), receive {'DOWN',Ref,_,_,_} -> ok end, - Sender ! {send_please, Pid, to_dead}, - {trace, Sender, send_to_non_existing_process, to_dead, Pid} = receive_first_trace(), - receive_nothing(), + F6 = fun (Pat) -> + set_trace_pattern(send, Pat, []), + Sender ! {send_please, Pid, to_dead}, + {trace, Sender, send_to_non_existing_process, to_dead, Pid} = receive_first_trace(), + receive_nothing() + end, + lists:foreach(F6, [no, + [{[Pid,to_dead],[],[]}], + [{['_','_'],[],[]}], + true]), %% Check that a message sent to unknown registrated process is traced. BadargSender = fun_spawn(fun sender/0), @@ -301,8 +482,26 @@ send_trace(Config) when is_list(Config) -> Sender ! {send_please, self(), to_myself_again}, receive to_myself_again -> ok end, receive_nothing(), + + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [global])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [local])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [meta])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [{meta,self()}])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [call_count])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, true, [call_time])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, restart, [])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, pause, [])), + {'EXIT',{badarg,_}} = (catch erlang:trace_pattern(send, [{['_','_'],[],[{caller}]}], [])), + + %% Done. ok. +set_trace_pattern(_, no, _) -> 0; +set_trace_pattern(MFA, Pat, Flg) -> + R = erlang:trace_pattern(MFA, Pat, Flg), + {match_spec, Pat} = erlang:trace_info(MFA, match_spec), + R. + %% Test trace(Pid, How, [procs]). procs_trace(Config) when is_list(Config) -> Name = list_to_atom(atom_to_list(?MODULE)++"_procs_trace"), @@ -438,6 +637,7 @@ dist_procs_trace(Config) when is_list(Config) -> Proc2 ! {unlink_please, Proc1}, {trace, Proc1, getting_unlinked, Proc2} = receive_first_trace(), receive_nothing(), + %% %% exit (with registered name, due to link) Name = list_to_atom(OtherName), @@ -1516,8 +1716,8 @@ receive_nothing() -> receive Any -> ct:fail({unexpected_message, Any}) - after 200 -> - ok + after 100 -> + ok end. @@ -1592,9 +1792,14 @@ sender() -> %% Just consumes messages from its message queue. receiver() -> - receive - _Any -> receiver() - end. + receiver(infinity). + +receiver(Timeout) -> + receiver(receive + {set_timeout, NewTimeout} -> NewTimeout; + _Any -> Timeout + after Timeout -> infinity %% reset + end). %% Works as long as it receives CPU time. Will always be RUNNABLE. diff --git a/erts/emulator/test/trace_bif_SUITE.erl b/erts/emulator/test/trace_bif_SUITE.erl index 8c3ffccc45..491b37ae46 100644 --- a/erts/emulator/test/trace_bif_SUITE.erl +++ b/erts/emulator/test/trace_bif_SUITE.erl @@ -232,6 +232,7 @@ do_trace_bif_return(TsType, TsFlags) -> {?MODULE, bif_process,0}}, Ts11, TsType), check_ts(TsType, Ts12, make_ts(TsType)), + erlang:trace_pattern({erlang,'_','_'}, false, [local]), ok. diff --git a/erts/emulator/test/tracer_SUITE.erl b/erts/emulator/test/tracer_SUITE.erl index de44d6656a..20fb7e475e 100644 --- a/erts/emulator/test/tracer_SUITE.erl +++ b/erts/emulator/test/tracer_SUITE.erl @@ -44,6 +44,8 @@ groups() -> gc_start, gc_end]}]. init_per_suite(Config) -> + erlang:trace_pattern({'_','_','_'}, false, [local]), + erlang:trace_pattern({'_','_','_'}, false, []), purge(), Config. @@ -119,23 +121,24 @@ unload(_Config) -> end, 1 = erlang:trace(Pid, true, [{tracer, tracer_test, - {#{ call => trace}, self(), []}}, + {#{ call => trace }, self(), []}}, call]), 1 = erlang:trace_pattern({?MODULE, all, 0}, [], []), Tc(1), - receive _ -> ok after 0 -> ct:fail(timeout) end, + receive _M -> ok after 0 -> ct:fail(timeout) end, + receive M0 -> ct:fail({unexpected_message0, M0}) after 0 -> ok end, code:purge(tracer_test), code:delete(tracer_test), Tc(1), - receive M1 -> ct:fail({unexpected_message, M1}) after 0 -> ok end, + receive M1 -> ct:fail({unexpected_message1, M1}) after 0 -> ok end, code:purge(tracer_test), Tc(1), - receive M2 -> ct:fail({unexpected_message, M2}) after 0 -> ok end, + receive M2 -> ct:fail({unexpected_message2, M2}) after 0 -> ok end, ok. @@ -167,13 +170,19 @@ reload(_Config) -> false = code:purge(tracer_test), true = code:delete(tracer_test), - false = code:purge(tracer_test) + false = code:purge(tracer_test), + timer:sleep(10) end || _ <- lists:seq(1,15)], ok. reload_loop() -> ?MODULE:all(), + ?MODULE:all(), + ?MODULE:all(), + ?MODULE:all(), + ?MODULE:all(), + timer:sleep(1), reload_loop(). invalid_tracers(_Config) -> @@ -214,7 +223,7 @@ send(_Config) -> Expect = fun(Pid, State, EOpts) -> receive Msg -> - {Pid, send, State, Pid, ok, Self, Opts} = Msg, + {send, State, Pid, ok, Self, Opts} = Msg, check_opts(EOpts, Opts) end end, @@ -230,7 +239,7 @@ recv(_Config) -> Expect = fun(Pid, State, EOpts) -> receive Msg -> - {undefined, 'receive', State, Pid, ok, undefined, Opts} = Msg, + {'receive', State, Pid, ok, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, @@ -247,14 +256,14 @@ spawn(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, spawn, State, Pid, NewPid, + {spawn, State, Pid, NewPid, {lists,seq,[1,10]}, Opts} = Msg, check_opts(EOpts, Opts), true = is_pid(NewPid) andalso NewPid /= Pid end end, - test(spawn, procs, Tc, Expect, true). + test(spawn, procs, Tc, Expect, false). exit(_Config) -> Tc = fun(Pid) -> @@ -265,7 +274,7 @@ exit(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, exit, State, Pid, normal, undefined, Opts} = Msg, + {exit, State, Pid, normal, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, @@ -286,13 +295,13 @@ link(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, link, State, Pid, NewPid, undefined, Opts} = Msg, + {link, State, Pid, NewPid, undefined, Opts} = Msg, check_opts(EOpts, Opts), true = is_pid(NewPid) andalso NewPid /= Pid end end, - test(link, procs, Tc, Expect, true). + test(link, procs, Tc, Expect, false). unlink(_Config) -> @@ -309,13 +318,13 @@ unlink(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, unlink, State, Pid, NewPid, undefined, Opts} = Msg, + {unlink, State, Pid, NewPid, undefined, Opts} = Msg, check_opts(EOpts, Opts), true = is_pid(NewPid) andalso NewPid /= Pid end end, - test(unlink, procs, Tc, Expect, true). + test(unlink, procs, Tc, Expect, false). getting_linked(_Config) -> @@ -331,7 +340,7 @@ getting_linked(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {NewPid, getting_linked, State, Pid, NewPid, undefined, Opts} = Msg, + {getting_linked, State, Pid, NewPid, undefined, Opts} = Msg, check_opts(EOpts, Opts), true = is_pid(NewPid) andalso NewPid /= Pid end @@ -355,7 +364,7 @@ getting_unlinked(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {NewPid, getting_unlinked, State, Pid, NewPid, undefined, Opts} = Msg, + {getting_unlinked, State, Pid, NewPid, undefined, Opts} = Msg, check_opts(EOpts, Opts), true = is_pid(NewPid) andalso NewPid /= Pid end @@ -377,12 +386,12 @@ register(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, register, State, Pid, ?MODULE, undefined, Opts} = Msg, + {register, State, Pid, ?MODULE, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, - test(register, procs, Tc, Expect, true). + test(register, procs, Tc, Expect, false). unregister(_Config) -> @@ -398,18 +407,18 @@ unregister(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, unregister, State, Pid, ?MODULE, undefined, Opts} = Msg, + {unregister, State, Pid, ?MODULE, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, - test(unregister, procs, Tc, Expect, true). + test(unregister, procs, Tc, Expect, false). in(_Config) -> Tc = fun(Pid) -> Self = self(), - Pid ! fun() -> receive after 1 -> Self ! ok end end, + Pid ! fun() -> receive after 10 -> Self ! ok end end, receive ok -> ok end end, @@ -418,7 +427,7 @@ in(_Config) -> N = (fun F(N) -> receive Msg -> - {Pid, in, State, Pid, _, + {in, State, Pid, _, undefined, Opts} = Msg, check_opts(EOpts, Opts), F(N+1) @@ -428,7 +437,7 @@ in(_Config) -> true = N > 0 end, - test(in, running, Tc, Expect, true). + test(in, running, Tc, Expect, false). out(_Config) -> Tc = fun(Pid) -> @@ -443,7 +452,7 @@ out(_Config) -> N = (fun F(N) -> receive Msg -> - {Pid, out, State, Pid, _, + {out, State, Pid, _, undefined, Opts} = Msg, check_opts(EOpts, Opts), F(N+1) @@ -453,7 +462,7 @@ out(_Config) -> true = N > 0 end, - test(out, running, Tc, Expect, true, true). + test(out, running, Tc, Expect, false, true). gc_start(_Config) -> @@ -468,12 +477,12 @@ gc_start(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, gc_major_start, State, Pid, _, undefined, Opts} = Msg, + {gc_major_start, State, Pid, _, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, - test(gc_major_start, garbage_collection, Tc, Expect, true). + test(gc_major_start, garbage_collection, Tc, Expect, false). gc_end(_Config) -> @@ -488,20 +497,20 @@ gc_end(_Config) -> fun(Pid, State, EOpts) -> receive Msg -> - {Pid, gc_major_end, State, Pid, _, undefined, Opts} = Msg, + {gc_major_end, State, Pid, _, undefined, Opts} = Msg, check_opts(EOpts, Opts) end end, - test(gc_major_end, garbage_collection, Tc, Expect, true). + test(gc_major_end, garbage_collection, Tc, Expect, false). test(Event, Tc, Expect) -> - test(Event, Tc, Expect, true). + test(Event, Tc, Expect, false). test(Event, Tc, Expect, Removes) -> test(Event, Event, Tc, Expect, Removes). test(Event, TraceFlag, Tc, Expect, Removes) -> test(Event, TraceFlag, Tc, Expect, Removes, false). -test(Event, TraceFlag, Tc, Expect, Removes, Dies) -> +test(Event, TraceFlag, Tc, Expect, _Removes, Dies) -> ComplexState = {fun() -> ok end, <<0:(128*8)>>}, Opts = #{ timestamp => undefined, @@ -557,25 +566,6 @@ test(Event, TraceFlag, Tc, Expect, Removes, Dies) -> ok end, - %% Test that remove works - Pid3 = start_tracee(), - State3 = {#{ Event => remove }, self(), ComplexState}, - 1 = erlang:trace(Pid3, true, [TraceFlag, {tracer, tracer_test, State3}]), - Tc(Pid3), - ok = trace_delivered(Pid3), - receive M3 -> ct:fail({unexpected, M3}) after 0 -> ok end, - if not Dies -> - if Removes -> - {flags, []} = erlang:trace_info(Pid3, flags), - {tracer, []} = erlang:trace_info(Pid3, tracer); - true -> - {flags, [TraceFlag]} = erlang:trace_info(Pid3, flags), - {tracer, {tracer_test, State3}} = erlang:trace_info(Pid3, tracer) - end, - erlang:trace(Pid3, false, [TraceFlag]); - true -> - ok - end, ok. check_opts(#{ scheduler_id := number } = E, #{ scheduler_id := N } = O) diff --git a/erts/emulator/test/tracer_SUITE_data/tracer_test.c b/erts/emulator/test/tracer_SUITE_data/tracer_test.c index 8b4be1345d..908f35da9c 100644 --- a/erts/emulator/test/tracer_SUITE_data/tracer_test.c +++ b/erts/emulator/test/tracer_SUITE_data/tracer_test.c @@ -104,16 +104,10 @@ static ERL_NIF_TERM trace(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) enif_get_tuple(env, argv[1], &state_arity, &state_tuple); - tuple = enif_alloc(sizeof(ERL_NIF_TERM)*(argc+1)); - memcpy(tuple+1,argv,sizeof(ERL_NIF_TERM)*argc); + tuple = enif_alloc(sizeof(ERL_NIF_TERM)*(argc)); + memcpy(tuple,argv,sizeof(ERL_NIF_TERM)*argc); - if (enif_self(env, &self)) { - tuple[0] = enif_make_pid(env, &self); - } else { - tuple[0] = enif_make_atom(env, "undefined"); - } - - msg = enif_make_tuple_from_array(env, tuple, argc + 1); + msg = enif_make_tuple_from_array(env, tuple, argc); enif_get_local_pid(env, state_tuple[1], &to); enif_send(env, &to, NULL, msg); enif_free(tuple); diff --git a/erts/epmd/test/epmd_SUITE.erl b/erts/epmd/test/epmd_SUITE.erl index 763984267a..0f0a5acde7 100644 --- a/erts/epmd/test/epmd_SUITE.erl +++ b/erts/epmd/test/epmd_SUITE.erl @@ -23,64 +23,58 @@ % Timeout for test cases (rather long to work on slow machines) --define(SHORT_TEST_TIMEOUT, ?t:seconds(30)). % Default --define(MEDIUM_TEST_TIMEOUT, ?t:minutes(3)). --define(LONG_TEST_TIMEOUT, ?t:minutes(10)). +-define(MEDIUM_TEST_TIMEOUT, {minutes,3}). +-define(LONG_TEST_TIMEOUT, {minutes,10}). % Delay inserted into code -define(SHORT_PAUSE, 100). --define(MEDIUM_PAUSE, ?t:seconds(1)). --define(LONG_PAUSE, ?t:seconds(5)). +-define(MEDIUM_PAUSE, 1000). +-define(LONG_PAUSE, 5000). % Information about nodes -record(node_info, {port, node_type, prot, lvsn, hvsn, node_name, extra}). % Test server specific exports --export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, - init_per_group/2,end_per_group/2, - init_per_testcase/2, end_per_testcase/2]). - --export( - [ - register_name/1, - register_name_ipv6/1, - register_names_1/1, - register_names_2/1, - register_duplicate_name/1, - unicode_name/1, - long_unicode_name/1, - get_port_nr/1, - slow_get_port_nr/1, - unregister_others_name_1/1, - unregister_others_name_2/1, - register_overflow/1, - name_with_null_inside/1, - name_null_terminated/1, - stupid_names_req/1, - - no_data/1, - one_byte/1, - two_bytes/1, - partial_packet/1, - zero_length/1, - too_large/1, - alive_req_too_small_1/1, - alive_req_too_small_2/1, - alive_req_too_large/1, - - returns_valid_empty_extra/1, - returns_valid_populated_extra_with_nulls/1, - - names_stdout/1, - - buffer_overrun_1/1, - buffer_overrun_2/1, - no_nonlocal_register/1, - no_nonlocal_kill/1, - no_live_killing/1, - - socket_reset_before_alive2_reply_is_written/1 - ]). +-export([all/0, suite/0, groups/0, init_per_testcase/2, end_per_testcase/2]). + +-export([register_name/1, + register_name_ipv6/1, + register_names_1/1, + register_names_2/1, + register_duplicate_name/1, + unicode_name/1, + long_unicode_name/1, + get_port_nr/1, + slow_get_port_nr/1, + unregister_others_name_1/1, + unregister_others_name_2/1, + register_overflow/1, + name_with_null_inside/1, + name_null_terminated/1, + stupid_names_req/1, + + no_data/1, + one_byte/1, + two_bytes/1, + partial_packet/1, + zero_length/1, + too_large/1, + alive_req_too_small_1/1, + alive_req_too_small_2/1, + alive_req_too_large/1, + + returns_valid_empty_extra/1, + returns_valid_populated_extra_with_nulls/1, + + names_stdout/1, + + buffer_overrun_1/1, + buffer_overrun_2/1, + no_nonlocal_register/1, + no_nonlocal_kill/1, + no_live_killing/1, + + socket_reset_before_alive2_reply_is_written/1]). % Port we use for testing @@ -88,7 +82,7 @@ -define(EPMDARGS,"-packet_timeout 1"). -define(DUMMY_PORT, 1000). % Port number to register - % not in real use. +% not in real use. % Timeouts etc inside test cases. Time is in milliseconds. -define(CONN_RETRY, 4). % Times to retry connecting @@ -111,7 +105,9 @@ %% all/1 %% -suite() -> [{ct_hooks,[ts_install_cth]}]. +suite() -> + [{ct_hooks,[ts_install_cth]}, + {timetrap, ?MEDIUM_TEST_TIMEOUT}]. all() -> [register_name, register_name_ipv6, @@ -134,105 +130,71 @@ groups() -> [{buffer_overrun, [], [buffer_overrun_1, buffer_overrun_2]}]. -init_per_suite(Config) -> - Config. - -end_per_suite(_Config) -> - ok. - -init_per_group(_GroupName, Config) -> - Config. - -end_per_group(_GroupName, Config) -> - Config. - - %% %% Run before and after each test case %% init_per_testcase(_Func, Config) -> - Dog = test_server:timetrap(?MEDIUM_TEST_TIMEOUT), cleanup(), - [{watchdog, Dog} | Config]. + Config. -end_per_testcase(_Func, Config) -> +end_per_testcase(_Func, _Config) -> cleanup(), - Dog = ?config(watchdog, Config), - catch test_server:timetrap_cancel(Dog), % We may have canceled already ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -register_name(doc) -> - ["Register a name"]; -register_name(suite) -> - []; +%% Register a name register_name(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node("foobar"), - ?line ok = close(Sock), % Unregister + ok = epmdrun(), + {ok,Sock} = register_node("foobar"), + ok = close(Sock), % Unregister ok. -register_name_ipv6(doc) -> - ["Register a name over IPv6"]; -register_name_ipv6(suite) -> - []; +%% Register a name over IPv6 register_name_ipv6(Config) when is_list(Config) -> % Test if the host has an IPv6 loopback address Res = gen_tcp:listen(0, [inet6, {ip, {0,0,0,0,0,0,0,1}}]), case Res of - {ok,LSock} -> - gen_tcp:close(LSock), - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node6("foobar6"), - ?line ok = close(Sock), % Unregister - ok; - _Error -> - {skip, "Host does not have an IPv6 loopback address"} + {ok,LSock} -> + gen_tcp:close(LSock), + ok = epmdrun(), + {ok,Sock} = register_node6("foobar6"), + ok = close(Sock), % Unregister + ok; + _Error -> + {skip, "Host does not have an IPv6 loopback address"} end. -register_names_1(doc) -> - ["Register and unregister two nodes"]; -register_names_1(suite) -> - []; +%% Register and unregister two nodes register_names_1(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock1} = register_node("foobar"), - ?line {ok,Sock2} = register_node("foozap"), - ?line ok = close(Sock1), % Unregister - ?line ok = close(Sock2), % Unregister + ok = epmdrun(), + {ok,Sock1} = register_node("foobar"), + {ok,Sock2} = register_node("foozap"), + ok = close(Sock1), % Unregister + ok = close(Sock2), % Unregister ok. -register_names_2(doc) -> - ["Register and unregister two nodes"]; -register_names_2(suite) -> - []; +%% Register and unregister two nodes register_names_2(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock1} = register_node("foobar"), - ?line {ok,Sock2} = register_node("foozap"), - ?line ok = close(Sock2), % Unregister - ?line ok = close(Sock1), % Unregister + ok = epmdrun(), + {ok,Sock1} = register_node("foobar"), + {ok,Sock2} = register_node("foozap"), + ok = close(Sock2), % Unregister + ok = close(Sock1), % Unregister ok. -register_duplicate_name(doc) -> - ["Two nodes with the same name"]; -register_duplicate_name(suite) -> - []; +%% Two nodes with the same name register_duplicate_name(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node("foobar"), - ?line error = register_node("foobar"), - ?line ok = close(Sock), % Unregister + ok = epmdrun(), + {ok,Sock} = register_node("foobar"), + error = register_node("foobar"), + ok = close(Sock), % Unregister ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -unicode_name(doc) -> - ["Check that we can register and lookup a unicode name"]; -unicode_name(suite) -> - []; +%% Check that we can register and lookup a unicode name unicode_name(Config) when is_list(Config) -> ok = epmdrun(), NodeName = [16#1f608], @@ -244,10 +206,7 @@ unicode_name(Config) when is_list(Config) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -long_unicode_name(doc) -> - ["Check that we can register and lookup a long unicode name"]; -long_unicode_name(suite) -> - []; +%% Check that we can register and lookup a long unicode name long_unicode_name(Config) when is_list(Config) -> ok = epmdrun(), BaseChar = 16#1f600, @@ -273,101 +232,85 @@ register_node_v2(Port, NodeType, Prot, HVsn, LVsn, Name, Extra) -> register_node_v2(Addr, Port, NodeType, Prot, HVsn, LVsn, Name, Extra) -> Req = alive2_req(Port, NodeType, Prot, HVsn, LVsn, Name, Extra), case send_req(Req, Addr) of - {ok,Sock} -> - case recv(Sock,4) of - {ok, [?EPMD_ALIVE2_RESP,_Res=0,_C0,_C1]} -> - {ok,Sock}; - Other -> - test_server:format("recv on sock ~w: ~p~n", - [Sock,Other]), - error - end; - error -> - error + {ok,Sock} -> + case recv(Sock,4) of + {ok, [?EPMD_ALIVE2_RESP,_Res=0,_C0,_C1]} -> + {ok,Sock}; + Other -> + io:format("recv on sock ~w: ~p~n", [Sock,Other]), + error + end; + error -> + error end. % Internal function to fetch information about a node port_please_v2(Name) -> case send_req([?EPMD_PORT_PLEASE2_REQ, - binary_to_list(unicode:characters_to_binary(Name))]) of - {ok,Sock} -> - case recv_until_sock_closes(Sock) of - {ok, Resp} -> - parse_port2_resp(Resp); - Other -> - test_server:format("recv on sock ~w: ~p~n", - [Sock,Other]), - error - end; - error -> - error + binary_to_list(unicode:characters_to_binary(Name))]) of + {ok,Sock} -> + case recv_until_sock_closes(Sock) of + {ok, Resp} -> + parse_port2_resp(Resp); + Other -> + io:format("recv on sock ~w: ~p~n", [Sock,Other]), + error + end; + error -> + error end. parse_port2_resp(Resp) -> case list_to_binary(Resp) of - <<?EPMD_PORT2_RESP,Res,Port:16,NodeType,Prot,HVsn:16,LVsn:16, - NLen:16,NodeName:NLen/binary, - ELen:16,Extra:ELen/binary>> when Res =:= 0 -> - {ok, #node_info{port=Port,node_type=NodeType,prot=Prot, - hvsn=HVsn,lvsn=LVsn, - node_name=unicode:characters_to_list(NodeName), - extra=binary_to_list(Extra)}}; - _Other -> - test_server:format("invalid port2 resp: ~p~n", - [Resp]), - error + <<?EPMD_PORT2_RESP,Res,Port:16,NodeType,Prot,HVsn:16,LVsn:16, + NLen:16,NodeName:NLen/binary, + ELen:16,Extra:ELen/binary>> when Res =:= 0 -> + {ok, #node_info{port=Port,node_type=NodeType,prot=Prot, + hvsn=HVsn,lvsn=LVsn, + node_name=unicode:characters_to_list(NodeName), + extra=binary_to_list(Extra)}}; + _Other -> + io:format("invalid port2 resp: ~p~n", [Resp]), + error end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -name_with_null_inside(doc) -> - ["Register a name with a null char in it"]; -name_with_null_inside(suite) -> - []; +%% Register a name with a null char in it name_with_null_inside(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line error = register_node("foo\000bar"), + ok = epmdrun(), + error = register_node("foo\000bar"), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -name_null_terminated(doc) -> - ["Register a name with terminating null byte"]; -name_null_terminated(suite) -> - []; +%% Register a name with terminating null byte name_null_terminated(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line error = register_node("foobar\000"), + ok = epmdrun(), + error = register_node("foobar\000"), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -stupid_names_req(doc) -> - ["Read names from epmd in a stupid way"]; -stupid_names_req(suite) -> - []; +%% Read names from epmd in a stupid way stupid_names_req(Config) when is_list(Config) -> - Dog = ?config(watchdog, Config), - test_server:timetrap_cancel(Dog), - LongDog = test_server:timetrap(?MEDIUM_TEST_TIMEOUT), - ?line ok = epmdrun(), - ?line [FirstConn | Conn] = register_many(1, ?REG_REPEAT_LIM, "foo"), - ?line unregister_many([FirstConn]), + ok = epmdrun(), + [FirstConn | Conn] = register_many(1, ?REG_REPEAT_LIM, "foo"), + unregister_many([FirstConn]), sleep(?MEDIUM_PAUSE), - ?line ok = check_names(Conn), - ?line ok = unregister_many(Conn), - test_server:timetrap_cancel(LongDog), + ok = check_names(Conn), + ok = unregister_many(Conn), ok. check_names(Conn) -> - ?line {ok,Sock} = connect_active(), - ?line {ok,Reply} = do_get_names(Sock), - ?line SortConn = lists:sort(Conn), - ?line SortReply = lists:sort(Reply), - ?line ok = check_names_cmp(SortConn, SortReply), + {ok,Sock} = connect_active(), + {ok,Reply} = do_get_names(Sock), + SortConn = lists:sort(Conn), + SortReply = lists:sort(Reply), + ok = check_names_cmp(SortConn, SortReply), ok. - + % Compare if the result was the same as was registered @@ -381,43 +324,43 @@ check_names_cmp([{Name,Port,_Sock} | Conn], [{Name,Port} | Reply]) -> -define(int16(X), [(X bsr 8) band 16#ff, X band 16#ff]). -define(u32(X1,X2,X3,X4), - (((X1) bsl 24) bor ((X2) bsl 16) bor ((X3) bsl 8) bor X4)). + (((X1) bsl 24) bor ((X2) bsl 16) bor ((X3) bsl 8) bor X4)). do_get_names(Socket) -> inet_tcp:send(Socket, [?int16(1),?EPMD_NAMES_REQ]), receive - {tcp, Socket, [P0,P1,P2,P3 | T]} -> - EpmdPort = ?u32(P0,P1,P2,P3), - if EpmdPort == ?PORT -> - names_loop(Socket, T, []); - true -> - close(Socket), - {error, address} - end; - {tcp_closed, Socket} -> - {ok, []} + {tcp, Socket, [P0,P1,P2,P3 | T]} -> + EpmdPort = ?u32(P0,P1,P2,P3), + if EpmdPort == ?PORT -> + names_loop(Socket, T, []); + true -> + close(Socket), + {error, address} + end; + {tcp_closed, Socket} -> + {ok, []} end. names_loop(Socket, Acc, Ps) -> receive - {tcp, Socket, Bytes} -> - {NAcc, NPs} = scan_names(Acc ++ Bytes, Ps), - names_loop(Socket, NAcc, NPs); - {tcp_closed, Socket} -> - {_, NPs} = scan_names(Acc, Ps), % Really needed? - {ok, NPs} + {tcp, Socket, Bytes} -> + {NAcc, NPs} = scan_names(Acc ++ Bytes, Ps), + names_loop(Socket, NAcc, NPs); + {tcp_closed, Socket} -> + {_, NPs} = scan_names(Acc, Ps), % Really needed? + {ok, NPs} end. scan_names(Buf, Ps) -> case scan_line(Buf, []) of - {Line, NBuf} -> - case parse_line(Line) of - {ok, Entry} -> - scan_names(NBuf, [Entry | Ps]); - error -> - scan_names(NBuf, Ps) - end; - [] -> {Buf, Ps} + {Line, NBuf} -> + case parse_line(Line) of + {ok, Entry} -> + scan_names(NBuf, [Entry | Ps]); + error -> + scan_names(NBuf, Ps) + end; + [] -> {Buf, Ps} end. scan_line([$\n | Buf], Line) -> {lists:reverse(Line), Buf}; @@ -426,16 +369,16 @@ scan_line([], _) -> []. parse_line([$n,$a,$m,$e,$ | Buf0]) -> case parse_name(Buf0, []) of - {Name, Buf1} -> - case Buf1 of - [$a,$t,$ ,$p,$o,$r,$t,$ | Buf2] -> - case catch list_to_integer(Buf2) of - {'EXIT', _} -> error; - Port -> {ok, {Name, Port}} - end; - _ -> error - end; - error -> error + {Name, Buf1} -> + case Buf1 of + [$a,$t,$ ,$p,$o,$r,$t,$ | Buf2] -> + case catch list_to_integer(Buf2) of + {'EXIT', _} -> error; + Port -> {ok, {Name, Port}} + end; + _ -> error + end; + error -> error end; parse_line(_) -> error. @@ -447,17 +390,11 @@ parse_name([], _Name) -> error. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -get_port_nr(doc) -> - ["Register a name on a port and ask about port nr"]; -get_port_nr(suite) -> - []; +%% Register a name on a port and ask about port nr get_port_nr(Config) when is_list(Config) -> port_request([?EPMD_PORT_PLEASE2_REQ,"foo"]). -slow_get_port_nr(doc) -> - ["Register with slow write and ask about port nr"]; -slow_get_port_nr(suite) -> - []; +%% Register with slow write and ask about port nr slow_get_port_nr(Config) when is_list(Config) -> port_request([?EPMD_PORT_PLEASE2_REQ,d,$f,d,$o,d,$o]). @@ -465,142 +402,127 @@ slow_get_port_nr(Config) when is_list(Config) -> % Internal function used above port_request(M) -> - ?line ok = epmdrun(), + ok = epmdrun(), Port = 1042, - ?line {ok,RSock} = register_node("foo", Port), - ?line {ok,Sock} = connect(), - ?line ok = send(Sock,[size16(M),M]), - ?line case recv_until_sock_closes(Sock) of - {ok, Resp} -> - ?line close(RSock), - ?line {ok,Rec} = parse_port2_resp(Resp), - ?line Port = Rec#node_info.port, - ok; - Other -> - ?line close(RSock), - ?line test_server:format("recv on sock ~w: ~p~n", - [Sock,Other]), - ?line throw({error,Other}) - end, + {ok,RSock} = register_node("foo", Port), + {ok,Sock} = connect(), + ok = send(Sock,[size16(M),M]), + case recv_until_sock_closes(Sock) of + {ok, Resp} -> + close(RSock), + {ok,Rec} = parse_port2_resp(Resp), + Port = Rec#node_info.port, + ok; + Other -> + close(RSock), + io:format("recv on sock ~w: ~p~n", [Sock,Other]), + throw({error,Other}) + end, ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -unregister_others_name_1(doc) -> - ["Unregister name of other node"]; -unregister_others_name_1(suite) -> - []; +%% Unregister name of other node unregister_others_name_1(Config) when is_list(Config) -> - ?line ok = epmdrun("-relaxed_command_check"), - ?line {ok,RSock} = register_node("foo"), - ?line {ok,Sock} = connect(), + ok = epmdrun("-relaxed_command_check"), + {ok,RSock} = register_node("foo"), + {ok,Sock} = connect(), M = [?EPMD_STOP_REQ,"foo"], - ?line ok = send(Sock,[size16(M),M]), + ok = send(Sock,[size16(M),M]), R = "STOPPED", - ?line {ok,R} = recv(Sock,length(R)), - ?line ok = close(RSock), + {ok,R} = recv(Sock,length(R)), + ok = close(RSock), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -unregister_others_name_2(doc) -> - ["Unregister name of other node"]; -unregister_others_name_2(suite) -> - []; +%% Unregister name of other node unregister_others_name_2(Config) when is_list(Config) -> - ?line ok = epmdrun("-relaxed_command_check"), - ?line {ok,Sock} = connect(), + ok = epmdrun("-relaxed_command_check"), + {ok,Sock} = connect(), M = [?EPMD_STOP_REQ,"xxx42"], - ?line ok = send(Sock,[size16(M),M]), + ok = send(Sock,[size16(M),M]), R = "NOEXIST", - ?line {ok,R} = recv(Sock,length(R)), + {ok,R} = recv(Sock,length(R)), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -register_overflow(doc) -> - ["Register too many, clean and redo 10 times"]; -register_overflow(suite) -> - []; +%% Register too many, clean and redo 10 times register_overflow(Config) when is_list(Config) -> - Dog = ?config(watchdog, Config), - test_server:timetrap_cancel(Dog), - LongDog = test_server:timetrap(?LONG_TEST_TIMEOUT), - ?line ok = epmdrun(), - ?line Conn = register_many(1, ?REG_REPEAT_LIM, "foo"), + ct:timetrap(?LONG_TEST_TIMEOUT), + ok = epmdrun(), + Conn = register_many(1, ?REG_REPEAT_LIM, "foo"), Count = length(Conn), - ?line ok = unregister_many(Conn), + ok = unregister_many(Conn), sleep(?MEDIUM_PAUSE), - test_server:format("Limit was ~w names, now reg/unreg all 10 times~n", - [Count]), - ?line ok = register_repeat(Count), + io:format("Limit was ~w names, now reg/unreg all 10 times~n", [Count]), + ok = register_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = rregister_repeat(Count), + ok = rregister_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = register_repeat(Count), + ok = register_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = rregister_repeat(Count), + ok = rregister_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = register_repeat(Count), + ok = register_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = rregister_repeat(Count), + ok = rregister_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = register_repeat(Count), + ok = register_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = rregister_repeat(Count), + ok = rregister_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = register_repeat(Count), + ok = register_repeat(Count), sleep(?MEDIUM_PAUSE), - ?line ok = rregister_repeat(Count), - test_server:timetrap_cancel(LongDog), + ok = rregister_repeat(Count), ok. register_repeat(Count) -> Conn = register_many(1, ?REG_REPEAT_LIM, "foo"), ok = unregister_many(Conn), if - length(Conn) == Count -> - ok; - true -> - error + length(Conn) == Count -> + ok; + true -> + error end. rregister_repeat(Count) -> Conn = register_many(1, ?REG_REPEAT_LIM, "foo"), ok = unregister_many(lists:reverse(Conn)), if - length(Conn) == Count -> - ok; - true -> - error + length(Conn) == Count -> + ok; + true -> + error end. % Return count of successful registrations register_many(I, N, _Prefix) when I > N -> - test_server:format("Done with all ~n", []), + io:format("Done with all ~n", []), []; register_many(I, N, Prefix) -> Name = gen_name(Prefix, I), Port = ?DUMMY_PORT + I, % Just make it up case register_node(Name, Port) of - {ok,Sock} -> - [{Name,Port,Sock} | register_many(I + 1, N, Prefix)]; - Any -> - test_server:format("Can't register: ~w of 1..~w ~w~n", - [Name,N,Any]), - [] + {ok,Sock} -> + [{Name,Port,Sock} | register_many(I + 1, N, Prefix)]; + Any -> + test_server:format("Can't register: ~w of 1..~w ~w~n", [Name,N,Any]), + [] end. unregister_many([]) -> ok; unregister_many([{Name,_Port,Sock} | Socks]) -> case close(Sock) of - ok -> - unregister_many(Socks); - Any -> - test_server:format("Can't unregister: ~w reason ~w~n", [Name,Any]), - error + ok -> + unregister_many(Socks); + Any -> + test_server:format("Can't unregister: ~w reason ~w~n", [Name,Any]), + error end. gen_name(Str,Int) -> @@ -608,246 +530,203 @@ gen_name(Str,Int) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -no_data(doc) -> - ["Open but send no data"]; -no_data(suite) -> - []; +%% Open but send no data no_data(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), + ok = epmdrun(), + {ok,Sock} = connect(), sleep(?LONG_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -one_byte(doc) -> - ["Send one byte only"]; -one_byte(suite) -> - []; +%% Send one byte only one_byte(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), - ?line ok = send(Sock,[0]), + ok = epmdrun(), + {ok,Sock} = connect(), + ok = send(Sock,[0]), sleep(?LONG_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -two_bytes(doc) -> - ["Send packet size only"]; -two_bytes(suite) -> - []; +%% Send packet size only two_bytes(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), - ?line ok = send(Sock,[put16(3)]), + ok = epmdrun(), + {ok,Sock} = connect(), + ok = send(Sock,[put16(3)]), sleep(?LONG_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -partial_packet(doc) -> - ["Got only part of a packet"]; -partial_packet(suite) -> - []; +%% Got only part of a packet partial_packet(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), - ?line ok = send(Sock,[put16(100),"only a few bytes"]), + ok = epmdrun(), + {ok,Sock} = connect(), + ok = send(Sock,[put16(100),"only a few bytes"]), sleep(?LONG_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -zero_length(doc) -> - ["Invalid zero packet size"]; -zero_length(suite) -> - []; +%% Invalid zero packet size zero_length(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), - ?line ok = send(Sock,[0,0,0,0,0,0,0,0,0,0]), + ok = epmdrun(), + {ok,Sock} = connect(), + ok = send(Sock,[0,0,0,0,0,0,0,0,0,0]), sleep(?MEDIUM_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -too_large(doc) -> - ["Invalid large packet"]; -too_large(suite) -> - []; +%% Invalid large packet too_large(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), + ok = epmdrun(), + {ok,Sock} = connect(), Size = 63000, M = lists:duplicate(Size, $z), - ?line ok = send(Sock,[put16(Size),M]), + ok = send(Sock,[put16(Size),M]), sleep(?MEDIUM_PAUSE), % With such a large packet, even the writes can fail as the % daemon closes before everything is delivered -> econnaborted case recv(Sock,1) of - closed -> ok; - {error,econnaborted} -> ok; - Other -> exit({unexpected,Other}) + closed -> ok; + {error,econnaborted} -> ok; + Other -> exit({unexpected,Other}) end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -alive_req_too_small_1(doc) -> - ["Try to register but not enough data"]; -alive_req_too_small_1(suite) -> - []; +%% Try to register but not enough data alive_req_too_small_1(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), + ok = epmdrun(), + {ok,Sock} = connect(), M = [?EPMD_ALIVE2_REQ, put16(?DUMMY_PORT),$M,0, put16(5), - put16(5),put16(0)], - ?line ok = send(Sock, [size16(M), M]), + put16(5),put16(0)], + ok = send(Sock, [size16(M), M]), sleep(?MEDIUM_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -alive_req_too_small_2(doc) -> - ["Try to register but not enough data"]; -alive_req_too_small_2(suite) -> - []; +%% Try to register but not enough data alive_req_too_small_2(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), + ok = epmdrun(), + {ok,Sock} = connect(), M = [?EPMD_ALIVE2_REQ, put16(?DUMMY_PORT),$M,0, put16(5), - put16(5)], - ?line ok = send(Sock, [size16(M), M]), + put16(5)], + ok = send(Sock, [size16(M), M]), sleep(?MEDIUM_PAUSE), - ?line closed = recv(Sock,1), + closed = recv(Sock,1), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -alive_req_too_large(doc) -> - ["Try to register but node name too large"]; -alive_req_too_large(suite) -> - []; +%% Try to register but node name too large alive_req_too_large(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = connect(), - L = [ - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - ], + ok = epmdrun(), + {ok,Sock} = connect(), + L = ["aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"], S = length(lists:flatten(L)), M = [?EPMD_ALIVE2_REQ, put16(?DUMMY_PORT),$M,0, put16(5), - put16(5), put16(S),L,put16(0)], - ?line ok = send(Sock, [size16(M), M]), + put16(5), put16(S),L,put16(0)], + ok = send(Sock, [size16(M), M]), sleep(?MEDIUM_PAUSE), - ?line {ok,[?EPMD_ALIVE2_RESP,1]} = recv(Sock,2), + {ok,[?EPMD_ALIVE2_RESP,1]} = recv(Sock,2), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -returns_valid_empty_extra(doc) -> - ["Check that an empty extra is prefixed by a two byte length"]; -returns_valid_empty_extra(suite) -> - []; +%% Check that an empty extra is prefixed by a two byte length returns_valid_empty_extra(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node_v2(4711, 72, 0, 5, 5, "foo", []), - ?line {ok,#node_info{extra=[]}} = port_please_v2("foo"), - ?line ok = close(Sock), + ok = epmdrun(), + {ok,Sock} = register_node_v2(4711, 72, 0, 5, 5, "foo", []), + {ok,#node_info{extra=[]}} = port_please_v2("foo"), + ok = close(Sock), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -returns_valid_populated_extra_with_nulls(doc) -> - ["Check a populated extra with embedded null characters"]; -returns_valid_populated_extra_with_nulls(suite) -> - []; +%% Check a populated extra with embedded null characters returns_valid_populated_extra_with_nulls(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node_v2(4711, 72, 0, 5, 5, "foo", "ABC\000\000"), - ?line {ok,#node_info{extra="ABC\000\000"}} = port_please_v2("foo"), - ?line ok = close(Sock), + ok = epmdrun(), + {ok,Sock} = register_node_v2(4711, 72, 0, 5, 5, "foo", "ABC\000\000"), + {ok,#node_info{extra="ABC\000\000"}} = port_please_v2("foo"), + ok = close(Sock), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -names_stdout(doc) -> - ["Test that epmd -names prints registered nodes to stdout"]; -names_stdout(suite) -> - []; +%% Test that epmd -names prints registered nodes to stdout names_stdout(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,Sock} = register_node("foobar"), - ?line ok = epmdrun("-names"), - ?line {ok, Data} = receive {_Port, {data, D}} -> {ok, D} - after 10000 -> {error, timeout} - end, - ?line {match,_} = re:run(Data, "^epmd: up and running", [multiline]), - ?line {match,_} = re:run(Data, "^name foobar at port", [multiline]), - ?line ok = close(Sock), + ok = epmdrun(), + {ok,Sock} = register_node("foobar"), + ok = epmdrun("-names"), + {ok, Data} = receive {_Port, {data, D}} -> {ok, D} + after 10000 -> {error, timeout} + end, + {match,_} = re:run(Data, "^epmd: up and running", [multiline]), + {match,_} = re:run(Data, "^name foobar at port", [multiline]), + ok = close(Sock), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -buffer_overrun_1(suite) -> - []; -buffer_overrun_1(doc) -> - ["Test security vulnerability in fake extra lengths in alive2_req"]; +%% Test security vulnerability in fake extra lengths in alive2_req buffer_overrun_1(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line true = alltrue([hostile(N) || N <- lists:seq(1,10000)]), + ok = epmdrun(), + true = alltrue([hostile(N) || N <- lists:seq(1,10000)]), ok. -buffer_overrun_2(suite) -> - []; -buffer_overrun_2(doc) -> - ["Test security vulnerability in fake extra lengths in alive2_req"]; + +%% Test security vulnerability in fake extra lengths in alive2_req buffer_overrun_2(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line [false | Rest] = [hostile2(N) || N <- lists:seq(255*4,10000)], - ?line true = alltrue(Rest), + ok = epmdrun(), + [false | Rest] = [hostile2(N) || N <- lists:seq(255*4,10000)], + true = alltrue(Rest), ok. hostile(N) -> try - Bin= <<$x:8,4747:16,$M:8,0:8,5:16,5:16,5:16,"gurka",N:16>>, - S = size(Bin), - {ok,E}=connect_sturdy(), - gen_tcp:send(E,[<<S:16>>,Bin]), - closed = recv(E,1), - gen_tcp:close(E), - true + Bin= <<$x:8,4747:16,$M:8,0:8,5:16,5:16,5:16,"gurka",N:16>>, + S = size(Bin), + {ok,E}=connect_sturdy(), + gen_tcp:send(E,[<<S:16>>,Bin]), + closed = recv(E,1), + gen_tcp:close(E), + true catch - _:_ -> - false + _:_ -> + false end. hostile2(N) -> try - B2 = list_to_binary(lists:duplicate(N,255)), - Bin= <<$x:8,4747:16,$M:8,0:8,5:16,5:16,5:16,"gurka",N:16,B2/binary>>, - S = size(Bin), - {ok,E}=connect_sturdy(), - gen_tcp:send(E,[<<S:16>>,Bin]), - Z = recv(E,2), - gen_tcp:close(E), - (Z =:= closed) or (Z =:= {ok, [$y,1]}) + B2 = list_to_binary(lists:duplicate(N,255)), + Bin= <<$x:8,4747:16,$M:8,0:8,5:16,5:16,5:16,"gurka",N:16,B2/binary>>, + S = size(Bin), + {ok,E}=connect_sturdy(), + gen_tcp:send(E,[<<S:16>>,Bin]), + Z = recv(E,2), + gen_tcp:close(E), + (Z =:= closed) or (Z =:= {ok, [$y,1]}) catch - _A:_B -> - false + _A:_B -> + false end. alltrue([]) -> @@ -857,134 +736,124 @@ alltrue([true|T]) -> alltrue([_|_]) -> false. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -no_nonlocal_register(suite) -> - []; -no_nonlocal_register(doc) -> - ["Ensure that we cannot register throug a nonlocal connection"]; + +%% Ensure that we cannot register throug a nonlocal connection no_nonlocal_register(Config) when is_list(Config) -> - ?line case {os:find_executable("ssh"),ct:get_config(ssh_proxy_host)} of - {SSH,Name} when is_list(Name), is_list(SSH) -> - do_no_nonlocal_register(Config,Name); - {false,_} -> - {skip, "No ssh command found to create proxy"}; - _ -> - {skip, "No ssh_proxy_host configured in ts.config"} - end. + case {os:find_executable("ssh"),ct:get_config(ssh_proxy_host)} of + {SSH,Name} when is_list(Name), is_list(SSH) -> + do_no_nonlocal_register(Config,Name); + {false,_} -> + {skip, "No ssh command found to create proxy"}; + _ -> + {skip, "No ssh_proxy_host configured in ts.config"} + end. do_no_nonlocal_register(Config,SSHHost) when is_list(Config) -> - ?line ok = epmdrun(), - ?line ProxyPort = proxy_port(), - ?line ok = ssh_proxy(SSHHost,ProxyPort), + ok = epmdrun(), + ProxyPort = proxy_port(), + ok = ssh_proxy(SSHHost,ProxyPort), Res = try - ?line Name = "gurka_" - %++ - %integer_to_list(A1)++"_"++ - %integer_to_list(A2)++"_"++ - %integer_to_list(A3)++"_"++ - %integer_to_list(A4) - , - ?line Bname = list_to_binary(Name), - ?line NameS = byte_size(Bname), - ?line Bin= <<$x:8,4747:16,$M:8,0:8,5:16, - 5:16,NameS:16,Bname/binary, - 0:16>>, - ?line S = size(Bin), - ?line {ok, E} = connect("localhost",ProxyPort,passive), - ?line gen_tcp:send(E,[<<S:16>>,Bin]), - ?line closed = recv(E,1), - ?line gen_tcp:close(E), - true - catch - _:_ -> - false - end, + Name = "gurka_" + %++ + %integer_to_list(A1)++"_"++ + %integer_to_list(A2)++"_"++ + %integer_to_list(A3)++"_"++ + %integer_to_list(A4) + , + Bname = list_to_binary(Name), + NameS = byte_size(Bname), + Bin= <<$x:8,4747:16,$M:8,0:8,5:16, + 5:16,NameS:16,Bname/binary, + 0:16>>, + S = size(Bin), + {ok, E} = connect("localhost",ProxyPort,passive), + gen_tcp:send(E,[<<S:16>>,Bin]), + closed = recv(E,1), + gen_tcp:close(E), + true + catch + _:_ -> + false + end, %erlang:display(Res), true = Res, ok. -no_nonlocal_kill(suite) -> - []; -no_nonlocal_kill(doc) -> - ["Ensure that we cannot kill through nonlocal connection"]; +%% Ensure that we cannot kill through nonlocal connection no_nonlocal_kill(Config) when is_list(Config) -> - ?line case {os:find_executable("ssh"),ct:get_config(ssh_proxy_host)} of - {SSH,Name} when is_list(Name), is_list(SSH) -> - do_no_nonlocal_kill(Config,Name); - {false,_} -> - {skip, "No ssh command found to create proxy"}; - _ -> - {skip, "No ssh_proxy_host configured in ts.config"} - end. + case {os:find_executable("ssh"),ct:get_config(ssh_proxy_host)} of + {SSH,Name} when is_list(Name), is_list(SSH) -> + do_no_nonlocal_kill(Config,Name); + {false,_} -> + {skip, "No ssh command found to create proxy"}; + _ -> + {skip, "No ssh_proxy_host configured in ts.config"} + end. do_no_nonlocal_kill(Config,SSHHost) when is_list(Config) -> - ?line ok = epmdrun(), - ?line ProxyPort = proxy_port(), - ?line ok = ssh_proxy(SSHHost,ProxyPort), + ok = epmdrun(), + ProxyPort = proxy_port(), + ok = ssh_proxy(SSHHost,ProxyPort), Res = try - {ok, E} = connect("localhost",ProxyPort,passive), - M = [?EPMD_KILL_REQ], - send(E, [size16(M), M]), - closed = recv(E,2), - gen_tcp:close(E), - sleep(?MEDIUM_PAUSE), - {ok, E2} = connect("localhost",ProxyPort,passive), - gen_tcp:close(E2), - true - catch - _:_ -> - false - end, + {ok, E} = connect("localhost",ProxyPort,passive), + M = [?EPMD_KILL_REQ], + send(E, [size16(M), M]), + closed = recv(E,2), + gen_tcp:close(E), + sleep(?MEDIUM_PAUSE), + {ok, E2} = connect("localhost",ProxyPort,passive), + gen_tcp:close(E2), + true + catch + _:_ -> + false + end, %erlang:display(Res), true = Res, ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -no_live_killing(doc) -> - ["Dont allow killing with live nodes or any unregistering w/o -relaxed_command_check"]; -no_live_killing(suite) -> - []; + +%% Dont allow killing with live nodes or any unregistering w/o -relaxed_command_check no_live_killing(Config) when is_list(Config) -> - ?line ok = epmdrun(), - ?line {ok,RSock} = register_node("foo"), - ?line {ok,Sock} = connect(), - ?line M = [?EPMD_KILL_REQ], - ?line ok = send(Sock,[size16(M),M]), - ?line {ok,"NO"} = recv(Sock,2), - ?line close(Sock), - ?line {ok,Sock2} = connect(), - ?line M2 = [?EPMD_STOP_REQ,"foo"], - ?line ok = send(Sock2,[size16(M2),M2]), - ?line closed = recv(Sock2,1), - ?line close(Sock2), - ?line close(RSock), - ?line sleep(?MEDIUM_PAUSE), - ?line {ok,Sock3} = connect(), - ?line M3 = [?EPMD_KILL_REQ], - ?line ok = send(Sock3,[size16(M3),M3]), - ?line {ok,"OK"} = recv(Sock3,2), - ?line close(Sock3), + ok = epmdrun(), + {ok,RSock} = register_node("foo"), + {ok,Sock} = connect(), + M = [?EPMD_KILL_REQ], + ok = send(Sock,[size16(M),M]), + {ok,"NO"} = recv(Sock,2), + close(Sock), + {ok,Sock2} = connect(), + M2 = [?EPMD_STOP_REQ,"foo"], + ok = send(Sock2,[size16(M2),M2]), + closed = recv(Sock2,1), + close(Sock2), + close(RSock), + sleep(?MEDIUM_PAUSE), + {ok,Sock3} = connect(), + M3 = [?EPMD_KILL_REQ], + ok = send(Sock3,[size16(M3),M3]), + {ok,"OK"} = recv(Sock3,2), + close(Sock3), ok. -socket_reset_before_alive2_reply_is_written(doc) -> - ["Check for regression - don't make zombie from node which " - "sends TCP RST at wrong time"]; -socket_reset_before_alive2_reply_is_written(suite) -> - []; +%% Check for regression - don't make zombie from node which +%% sends TCP RST at wrong time socket_reset_before_alive2_reply_is_written(Config) when is_list(Config) -> %% - delay_write for easier triggering of race condition %% - relaxed_command_check for gracefull shutdown of epmd even if there %% is stuck node. - ?line ok = epmdrun("-delay_write 1 -relaxed_command_check"), + ok = epmdrun("-delay_write 1 -relaxed_command_check"), %% We can't use send_req/1 directly as we want to do inet:setopts/2 %% on our socket. - ?line {ok, Sock} = connect(), + {ok, Sock} = connect(), %% Issuing close/1 on such socket will result in immediate RST packet. - ?line ok = inet:setopts(Sock, [{linger, {true, 0}}]), + ok = inet:setopts(Sock, [{linger, {true, 0}}]), Req = alive2_req(4711, 77, 0, 5, 5, "test", []), - ?line ok = send(Sock, [size16(Req), Req]), + ok = send(Sock, [size16(Req), Req]), timer:sleep(500), %% Wait for the first 1/2 of delay_write before closing - ?line ok = close(Sock), + ok = close(Sock), timer:sleep(500 + ?SHORT_PAUSE), %% Wait for the other 1/2 of delay_write @@ -992,11 +861,11 @@ socket_reset_before_alive2_reply_is_written(Config) when is_list(Config) -> %% Should be removed when this is issue is fixed there. timer:sleep(1000), - ?line {ok, SockForNames} = connect_active(), + {ok, SockForNames} = connect_active(), %% And there should be no stuck nodes - ?line {ok, []} = do_get_names(SockForNames), - ?line ok = close(SockForNames), + {ok, []} = do_get_names(SockForNames), + ok = close(SockForNames), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1005,14 +874,14 @@ socket_reset_before_alive2_reply_is_written(Config) when is_list(Config) -> cleanup() -> sleep(?MEDIUM_PAUSE), case connect() of - {ok,Sock} -> - M = [?EPMD_KILL_REQ], - send(Sock, [size16(M), M]), - recv(Sock,length("OK")), - close(Sock), - sleep(?MEDIUM_PAUSE); - _ -> - true + {ok,Sock} -> + M = [?EPMD_KILL_REQ], + send(Sock, [size16(M), M]), + recv(Sock,length("OK")), + close(Sock), + sleep(?MEDIUM_PAUSE); + _ -> + true end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1022,11 +891,11 @@ proxy_port() -> ?PORT+1. ssh_proxy(SSHHost,ProxyPort) -> - ?line Host = lists:nth(2,string:tokens(atom_to_list(node()),"@")), + Host = lists:nth(2,string:tokens(atom_to_list(node()),"@")), % Requires proxy to be a unix host with the command 'read' accessible - ?line osrun("ssh -L "++integer_to_list(ProxyPort)++":"++Host++":" - ++integer_to_list(?PORT)++" "++SSHHost++" read"). - + osrun("ssh -L "++integer_to_list(ProxyPort)++":"++Host++":" + ++integer_to_list(?PORT)++" "++SSHHost++" read"). + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1036,21 +905,21 @@ epmdrun() -> epmdrun([]). epmdrun(Args) -> case os:find_executable(epmd) of - false -> - {error, {could_not_find_epmd_in_path}}; - Path -> - epmdrun(Path,Args) + false -> + {error, {could_not_find_epmd_in_path}}; + Path -> + epmdrun(Path,Args) end. epmdrun(Epmd,Args0) -> - %% test_server:format("epmdrun() => Epmd = ~p",[Epmd]), + %% test_server:format("epmdrun() => Epmd = ~p",[Epmd]), Args = case Args0 of - [] -> - []; - O -> - " "++O - end, - osrun("\"" ++ Epmd ++ "\"" ++ " " ?EPMDARGS " -port " ++ integer_to_list(?PORT) ++ Args). + [] -> + []; + O -> + " "++O + end, + osrun("\"" ++ Epmd ++ "\"" ++ " " ?EPMDARGS " -port " ++ integer_to_list(?PORT) ++ Args). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1059,7 +928,7 @@ epmdrun(Epmd,Args0) -> osrun(Cmd) -> _ = open_port({spawn, Cmd}, []), ok. - + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Wrappers of TCP functions @@ -1084,16 +953,16 @@ connect(Addr, Port, Mode) -> connect(Addr, Port, Mode, ?CONN_SLEEP, ?CONN_RETRY). connect(Addr, Port, Mode, Sleep, Retry) -> case connect_repeat(Addr, Retry, Port, Mode, Sleep) of - {ok,Sock} -> - {ok,Sock}; - {error,timeout} -> - timeout; - {error,Reason} -> - test_server:format("connect: error: ~w~n",[Reason]), - error; - Any -> - test_server:format("connect: unknown message: ~w~n",[Any]), - exit(1) + {ok,Sock} -> + {ok,Sock}; + {error,timeout} -> + timeout; + {error,Reason} -> + test_server:format("connect: error: ~w~n",[Reason]), + error; + Any -> + test_server:format("connect: unknown message: ~w~n",[Any]), + exit(1) end. @@ -1104,33 +973,33 @@ connect_repeat(Addr, 1, Port, Mode, _Sleep) -> connect_mode(Addr,Port, Mode); connect_repeat(Addr,Retry, Port, Mode, Sleep) -> case connect_mode(Addr,Port, Mode) of - {ok,Sock} -> - {ok,Sock}; - {error,Reason} -> - test_server:format("connect: error: ~w~n",[Reason]), - timer:sleep(Sleep), - connect_repeat(Addr, Retry - 1, Port, Mode, Sleep); - Any -> - test_server:format("connect: unknown message: ~w~n",[Any]), - exit(1) + {ok,Sock} -> + {ok,Sock}; + {error,Reason} -> + test_server:format("connect: error: ~w~n",[Reason]), + timer:sleep(Sleep), + connect_repeat(Addr, Retry - 1, Port, Mode, Sleep); + Any -> + test_server:format("connect: unknown message: ~w~n",[Any]), + exit(1) end. connect_mode(Addr,Port, active) -> gen_tcp:connect(Addr, Port, [{packet, 0}], ?CONN_TIMEOUT); connect_mode(Addr, Port, passive) -> gen_tcp:connect(Addr, Port, [{packet, 0}, {active, false}], - ?CONN_TIMEOUT). + ?CONN_TIMEOUT). close(Sock) -> case gen_tcp:close(Sock) of - {error,_} -> - error; - ok -> - ok; - Any -> - test_server:format("unknown message: ~w~n",[Any]), - exit(1) + {error,_} -> + error; + ok -> + ok; + Any -> + test_server:format("unknown message: ~w~n",[Any]), + exit(1) end. recv(Sock, Len) -> @@ -1138,19 +1007,19 @@ recv(Sock, Len) -> recv(Sock, Len, Timeout) -> case gen_tcp:recv(Sock, Len, Timeout) of - {ok,[]} -> % Should not be the case - recv(Sock, 1, 1); % any longer - {ok,Data} -> - {ok,Data}; - {error,timeout} -> - timeout; - {error,closed} -> - closed; - {error,_}=Error -> - Error; - Any -> - test_server:format("unknown message: ~w~n",[Any]), - exit(1) + {ok,[]} -> % Should not be the case + recv(Sock, 1, 1); % any longer + {ok,Data} -> + {ok,Data}; + {error,timeout} -> + timeout; + {error,closed} -> + closed; + {error,_}=Error -> + Error; + Any -> + test_server:format("unknown message: ~w~n",[Any]), + exit(1) end. %% Send data to socket. The list can be non flat and contain @@ -1159,12 +1028,12 @@ recv(Sock, Len, Timeout) -> send(Sock, SendSpec) -> case send(SendSpec, [], Sock) of - {ok,[]} -> - ok; - {ok,RevBytes} -> - send_direct(Sock, lists:reverse(RevBytes)); - Any -> - Any + {ok,[]} -> + ok; + {ok,RevBytes} -> + send_direct(Sock, lists:reverse(RevBytes)); + Any -> + Any end. @@ -1179,54 +1048,54 @@ send([Byte | Spec], RevBytes, Sock) when is_integer(Byte) -> send(Spec, [Byte | RevBytes], Sock); send([List | Spec], RevBytes, Sock) when is_list(List) -> case send(List, RevBytes, Sock) of - {ok,Left} -> - send(Spec, Left, Sock); - Other -> - Other + {ok,Left} -> + send(Spec, Left, Sock); + Other -> + Other end; send([d | Spec], RevBytes, Sock) -> send([{d,1000} | Spec], RevBytes, Sock); send([{d,S} | Spec], RevBytes, Sock) -> case send_direct(Sock, lists:reverse(RevBytes)) of - ok -> - timer:sleep(S), - send(Spec, [], Sock); - Any -> - Any + ok -> + timer:sleep(S), + send(Spec, [], Sock); + Any -> + Any end. %%%% send_direct(Sock, Bytes) -> case gen_tcp:send(Sock, Bytes) of - ok -> - ok; - {error, closed} -> - closed; - {error, _Reason} -> - error; - Any -> - test_server:format("unknown message: ~w~n",[Any]), - Any + ok -> + ok; + {error, closed} -> + closed; + {error, _Reason} -> + error; + Any -> + test_server:format("unknown message: ~w~n",[Any]), + Any end. send_req(Req) -> send_req(Req, "localhost"). send_req(Req, Addr) -> case connect(Addr) of - {ok,Sock} -> - case send(Sock, [size16(Req), Req]) of - ok -> - {ok,Sock}; - Other -> - test_server:format("Failed to send ~w on sock ~w: ~w~n", - [Req,Sock,Other]), - error - end; - Other -> - test_server:format("Connect failed when sending ~w: ~p~n", - [Req, Other]), - error + {ok,Sock} -> + case send(Sock, [size16(Req), Req]) of + ok -> + {ok,Sock}; + Other -> + test_server:format("Failed to send ~w on sock ~w: ~w~n", + [Req,Sock,Other]), + error + end; + Other -> + test_server:format("Connect failed when sending ~w: ~p~n", + [Req, Other]), + error end. recv_until_sock_closes(Sock) -> @@ -1234,12 +1103,12 @@ recv_until_sock_closes(Sock) -> recv_until_sock_closes_2(Sock,AccData) -> case recv(Sock,0) of - {ok,Data} -> - recv_until_sock_closes_2(Sock,AccData++Data); - closed -> - {ok,AccData}; - Other -> - Other + {ok,Data} -> + recv_until_sock_closes_2(Sock,AccData++Data); + closed -> + {ok,AccData}; + Other -> + Other end. sleep(MilliSeconds) -> @@ -1261,7 +1130,7 @@ flat_count([H|T], N) when is_list(H) -> flat_count([_|T], N) -> flat_count(T, N); flat_count([], N) -> N. - + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/erts/etc/common/erlexec.c b/erts/etc/common/erlexec.c index 5a5021b003..42da05b1f7 100644 --- a/erts/etc/common/erlexec.c +++ b/erts/etc/common/erlexec.c @@ -150,6 +150,9 @@ static char *plush_val_switches[] = { "ms", "mbs", "pds", + "max", + "maxk", + "maxel", "mqd", "", NULL @@ -842,7 +845,6 @@ int main(int argc, char **argv) if (argv[i][3] != '\0') goto the_default; } -#ifdef ERTS_DIRTY_SCHEDULERS else if (argv[i][2] == 'D') { char* type = argv[i]+3; if (strncmp(type, "cpu", 3) != 0 && @@ -854,7 +856,6 @@ int main(int argc, char **argv) (argv[i][3] == 'i' && argv[i][5] != '\0')) goto the_default; } -#endif else if (argv[i][2] != '\0') goto the_default; if (i+1 >= argc) diff --git a/erts/include/erl_native_features_config.h.in b/erts/include/erl_native_features_config.h.in deleted file mode 100644 index 59a5dde09e..0000000000 --- a/erts/include/erl_native_features_config.h.in +++ /dev/null @@ -1,22 +0,0 @@ -/* - * %CopyrightBegin% - * - * Copyright Ericsson AB 2004-2016. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - * %CopyrightEnd% - */ - -/* Dirty scheduler support */ -#undef ERL_NIF_DIRTY_SCHEDULER_SUPPORT diff --git a/erts/lib_src/Makefile.in b/erts/lib_src/Makefile.in index 6e2f236bdf..601f3917a8 100644 --- a/erts/lib_src/Makefile.in +++ b/erts/lib_src/Makefile.in @@ -465,7 +465,6 @@ RELSYSDIR = $(RELEASE_PATH)/erts-$(VSN) RELEASE_INCLUDES= \ $(ERTS_INCL)/erl_memory_trace_parser.h \ $(ERTS_INCL)/$(TARGET)/erl_int_sizes_config.h \ - $(ERTS_INCL)/$(TARGET)/erl_native_features_config.h \ $(ERTS_INCL)/erl_fixed_size_int_types.h RELEASE_LIBS=$(ERTS_LIBS) diff --git a/erts/preloaded/ebin/erl_prim_loader.beam b/erts/preloaded/ebin/erl_prim_loader.beam Binary files differindex 3e82b8e20e..de2693472a 100644 --- a/erts/preloaded/ebin/erl_prim_loader.beam +++ b/erts/preloaded/ebin/erl_prim_loader.beam diff --git a/erts/preloaded/ebin/erl_tracer.beam b/erts/preloaded/ebin/erl_tracer.beam Binary files differindex 67d96d0f0d..69804540c9 100644 --- a/erts/preloaded/ebin/erl_tracer.beam +++ b/erts/preloaded/ebin/erl_tracer.beam diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam Binary files differindex 321e4e7e1b..8379bf1768 100644 --- a/erts/preloaded/ebin/erlang.beam +++ b/erts/preloaded/ebin/erlang.beam diff --git a/erts/preloaded/ebin/erts_code_purger.beam b/erts/preloaded/ebin/erts_code_purger.beam Binary files differindex 0c667663fd..b1da0aa861 100644 --- a/erts/preloaded/ebin/erts_code_purger.beam +++ b/erts/preloaded/ebin/erts_code_purger.beam diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam Binary files differindex ed2e328f5b..cc4f3dbdaf 100644 --- a/erts/preloaded/ebin/erts_internal.beam +++ b/erts/preloaded/ebin/erts_internal.beam diff --git a/erts/preloaded/ebin/init.beam b/erts/preloaded/ebin/init.beam Binary files differindex 23ebd2cad9..7b5797e90a 100644 --- a/erts/preloaded/ebin/init.beam +++ b/erts/preloaded/ebin/init.beam diff --git a/erts/preloaded/ebin/otp_ring0.beam b/erts/preloaded/ebin/otp_ring0.beam Binary files differindex b63fa24848..f893b9c181 100644 --- a/erts/preloaded/ebin/otp_ring0.beam +++ b/erts/preloaded/ebin/otp_ring0.beam diff --git a/erts/preloaded/ebin/prim_eval.beam b/erts/preloaded/ebin/prim_eval.beam Binary files differindex 7c8fdc82c0..e4ce601c03 100644 --- a/erts/preloaded/ebin/prim_eval.beam +++ b/erts/preloaded/ebin/prim_eval.beam diff --git a/erts/preloaded/ebin/prim_file.beam b/erts/preloaded/ebin/prim_file.beam Binary files differindex d4559bc9ed..babba3081f 100644 --- a/erts/preloaded/ebin/prim_file.beam +++ b/erts/preloaded/ebin/prim_file.beam diff --git a/erts/preloaded/ebin/prim_inet.beam b/erts/preloaded/ebin/prim_inet.beam Binary files differindex 5a0678632e..0521060e34 100644 --- a/erts/preloaded/ebin/prim_inet.beam +++ b/erts/preloaded/ebin/prim_inet.beam diff --git a/erts/preloaded/ebin/prim_zip.beam b/erts/preloaded/ebin/prim_zip.beam Binary files differindex 3d32dd8449..e1faca7d96 100644 --- a/erts/preloaded/ebin/prim_zip.beam +++ b/erts/preloaded/ebin/prim_zip.beam diff --git a/erts/preloaded/ebin/zlib.beam b/erts/preloaded/ebin/zlib.beam Binary files differindex ced9f9c952..5d3cbc1b36 100644 --- a/erts/preloaded/ebin/zlib.beam +++ b/erts/preloaded/ebin/zlib.beam diff --git a/erts/preloaded/src/erl_tracer.erl b/erts/preloaded/src/erl_tracer.erl index de1e9ca01e..fe15812535 100644 --- a/erts/preloaded/src/erl_tracer.erl +++ b/erts/preloaded/src/erl_tracer.erl @@ -41,10 +41,14 @@ on_load() -> %%% NIF placeholders %%% --spec enabled(Tag :: trace_tag() | seq_trace | trace_status, +-spec enabled(Tag :: trace_status, TracerState :: tracer_state(), Tracee :: tracee()) -> - trace | discard | remove. + trace | remove; + (Tag :: trace_tag() | seq_trace, + TracerState :: tracer_state(), + Tracee :: tracee()) -> + trace | discard. enabled(_, _, _) -> erlang:nif_error(nif_not_loaded). diff --git a/erts/preloaded/src/erlang.erl b/erts/preloaded/src/erlang.erl index 20a64e81b4..3d152c4e92 100644 --- a/erts/preloaded/src/erlang.erl +++ b/erts/preloaded/src/erlang.erl @@ -1749,16 +1749,16 @@ trace_delivered(_Tracee) -> erlang:nif_error(undefined). %% trace_info/2 --spec erlang:trace_info(PidPortOrFunc, Item) -> Res when - PidPortOrFunc :: pid() | port() | new | new_processes | new_ports - | {Module, Function, Arity} | on_load, +-spec erlang:trace_info(PidPortFuncEvent, Item) -> Res when + PidPortFuncEvent :: pid() | port() | new | new_processes | new_ports + | {Module, Function, Arity} | on_load | send | 'receive', Module :: module(), Function :: atom(), Arity :: arity(), Item :: flags | tracer | traced | match_spec | meta | meta_match_spec | call_count | call_time | all, Res :: trace_info_return(). -trace_info(_PidPortOrFunc, _Item) -> +trace_info(_PidPortFuncEvent, _Item) -> erlang:nif_error(undefined). %% trunc/1 @@ -2073,6 +2073,9 @@ open_port(PortName, PortSettings) -> (min_bin_vheap_size, MinBinVHeapSize) -> OldMinBinVHeapSize when MinBinVHeapSize :: non_neg_integer(), OldMinBinVHeapSize :: non_neg_integer(); + (max_heap_size, MaxHeapSize) -> OldMaxHeapSize when + MaxHeapSize :: max_heap_size(), + OldMaxHeapSize :: max_heap_size(); (message_queue_data, MQD) -> OldMQD when MQD :: message_queue_data(), OldMQD :: message_queue_data(); @@ -2154,6 +2157,7 @@ process_flag(_Flag, _Value) -> {messages, MessageQueue :: [term()]} | {min_heap_size, MinHeapSize :: non_neg_integer()} | {min_bin_vheap_size, MinBinVHeapSize :: non_neg_integer()} | + {max_heap_size, MaxHeapSize :: max_heap_size()} | {monitored_by, Pids :: [pid()]} | {monitors, Monitors :: [{process, Pid :: pid() | @@ -2238,6 +2242,7 @@ setelement(_Index, _Tuple1, _Value) -> | {priority, Level :: priority_level()} | {fullsweep_after, Number :: non_neg_integer()} | {min_heap_size, Size :: non_neg_integer()} + | {max_heap_size, Size :: max_heap_size()} | {min_bin_vheap_size, VSize :: non_neg_integer()}. spawn_opt(_Tuple) -> erlang:nif_error(undefined). @@ -2330,6 +2335,9 @@ subtract(_,_) -> OldMinBinVHeapSize when MinBinVHeapSize :: non_neg_integer(), OldMinBinVHeapSize :: non_neg_integer(); + (max_heap_size, MaxHeapSize) -> OldMaxHeapSize when + MaxHeapSize :: max_heap_size(), + OldMaxHeapSize :: max_heap_size(); (multi_scheduling, BlockState) -> OldBlockState when BlockState :: block | unblock | block_normal | unblock_normal, OldBlockState :: blocked | disabled | enabled; @@ -2381,7 +2389,7 @@ tl(_List) -> [{[term()] | '_' ,[term()],[term()]}]. -spec erlang:trace_pattern(MFA, MatchSpec) -> non_neg_integer() when - MFA :: trace_pattern_mfa(), + MFA :: trace_pattern_mfa() | send | 'receive', MatchSpec :: (MatchSpecList :: trace_match_spec()) | boolean() | restart @@ -2403,7 +2411,13 @@ trace_pattern(MFA, MatchSpec) -> call_count | call_time. --spec erlang:trace_pattern(MFA, MatchSpec, FlagList) -> non_neg_integer() when +-spec erlang:trace_pattern(send, MatchSpec, []) -> non_neg_integer() when + MatchSpec :: (MatchSpecList :: trace_match_spec()) + | boolean(); + ('receive', MatchSpec, []) -> non_neg_integer() when + MatchSpec :: (MatchSpecList :: trace_match_spec()) + | boolean(); + (MFA, MatchSpec, FlagList) -> non_neg_integer() when MFA :: trace_pattern_mfa(), MatchSpec :: (MatchSpecList :: trace_match_spec()) | boolean() @@ -2505,6 +2519,7 @@ tuple_to_list(_Tuple) -> logical_processors_available | logical_processors_online) -> unknown | pos_integer(); (machine) -> string(); + (max_heap_size) -> {max_heap_size, MaxHeapSize :: max_heap_size()}; (message_queue_data) -> message_queue_data(); (min_heap_size) -> {min_heap_size, MinHeapSize :: pos_integer()}; (min_bin_vheap_size) -> {min_bin_vheap_size, @@ -2642,6 +2657,13 @@ spawn_monitor(M, F, A) -> erlang:error(badarg, [M,F,A]). +-type max_heap_size() :: + Size :: non_neg_integer() + %% TODO change size => to := when -type maps support is finalized + | #{ size => non_neg_integer(), + kill => boolean(), + error_logger => boolean() }. + -type spawn_opt_option() :: link | monitor @@ -2649,6 +2671,7 @@ spawn_monitor(M, F, A) -> | {fullsweep_after, Number :: non_neg_integer()} | {min_heap_size, Size :: non_neg_integer()} | {min_bin_vheap_size, VSize :: non_neg_integer()} + | {max_heap_size, Size :: max_heap_size()} | {message_queue_data, MQD :: message_queue_data()}. -spec spawn_opt(Fun, Options) -> pid() | {pid(), reference()} when diff --git a/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl b/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl index 8f4cf2b9ec..325bea5879 100644 --- a/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl +++ b/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl @@ -500,9 +500,7 @@ gen_decode_sof(Erules,TypeName,_InnerTypeName,D) when is_record(D,type) -> Atom when is_atom(Atom) -> Atom; _ -> TypeNameSuffix end, - ObjFun = false, - gen_dec_line(Erules,TypeName,ContName,[],Cont,mandatory,ObjFun), - %% gen_dec_line_sof(Erules,Typename,ContName,Cont,ObjFun), + gen_dec_line(Erules,TypeName,ContName,[],Cont,mandatory), emit([" || ",{curr,v}," <- ",{curr,tlv},"].",nl,nl,nl]). @@ -869,7 +867,7 @@ gen_dec_choice_cases(Erules,TopType, [H|T]) -> (?ASN1CT_GEN_BER:decode_class(T1class) bsl 10) + T1number,",_} -> ",nl]), emit([indent(8),"{",{asis,Cname},", "]), - gen_dec_line(Erules,TopType,Cname,[],Type,Prop,false), + gen_dec_line(Erules,TopType,Cname,[],Type,Prop), emit(["};",nl,nl]), Fun(Tail,Fun); ([],_) -> @@ -896,7 +894,7 @@ gen_dec_choice_cases(Erules,TopType, [H|T]) -> (?ASN1CT_GEN_BER:decode_class(FirstT#tag.class) bsl 10) + FirstT#tag.number,", ",{curr,v},"} -> ",nl]), emit([indent(8),"{",{asis,Cname},", "]), - gen_dec_line(Erules,TopType,Cname,[],Type#type{tag=RestT},Prop,false), + gen_dec_line(Erules,TopType,Cname,[],Type#type{tag=RestT},Prop), emit(["};",nl,nl]) end, gen_dec_choice_cases(Erules,TopType, T). @@ -1060,8 +1058,14 @@ gen_optormand_case({'DEFAULT',DefaultValue}, Erules, _TopType, end, emit([indent(9),"_ ->",nl,indent(12)]) end. - +%% Use for SEQUENCE OF and CHOICE. +gen_dec_line(Erules,TopType,Cname,CTags,Type,OptOrMand) -> + %% The matching on the next line is an assertion. + {[],[]} = gen_dec_line(Erules,TopType,Cname,CTags,Type,OptOrMand,false), + ok. + +%% Use for SEQUENCE. gen_dec_line(Erules,TopType,Cname,CTags,Type,OptOrMand,DecObjInf) -> BytesVar = asn1ct_gen:mk_var(asn1ct_name:curr(v)), Tag = diff --git a/lib/common_test/doc/src/ct_netconfc.xml b/lib/common_test/doc/src/ct_netconfc.xml index 9d1ff25a3a..e6930b30d5 100644 --- a/lib/common_test/doc/src/ct_netconfc.xml +++ b/lib/common_test/doc/src/ct_netconfc.xml @@ -201,7 +201,8 @@ information, see module <seealso marker="ct"><c>ct</c></seealso>.</p> </item> - <tag><c>host() = <seealso marker="kernel:inet#type-hostname"><c>inet:hostname()</c></seealso> | <seealso marker="kernel:inet#type-ip_address"><c>inet:ip_address()</c></seealso></c></tag> + <tag><c>host() = </c><seealso marker="kernel:inet#type-hostname"><c>inet:hostname()</c></seealso> + <c> | </c><seealso marker="kernel:inet#type-ip_address"><c>inet:ip_address()</c></seealso></tag> <item><marker id="type-host"/></item> <tag><c>key_or_name() = server_id() | target_name()</c></tag> @@ -218,7 +219,9 @@ <tag><c>notification_content() = [event_time() | simple_xml()]</c></tag> <item><marker id="type-notification_content"/> </item> - <tag><c>option() = {ssh, host()} | {port, <seealso marker="kernel:inet#type-port_number"><c>inet:port_number()</c></seealso>} | {timeout, timeout()} | SshConnectOption</c></tag> + <tag><c>option() = {ssh, host()} | {port, </c> + <seealso marker="kernel:inet#type-port_number"><c>inet:port_number()</c></seealso> + <c>} | {timeout, timeout()} | SshConnectOption</c></tag> <item><marker id="type-option"/> <p><c>SshConnectOption</c> is any valid option to diff --git a/lib/common_test/src/ct_logs.erl b/lib/common_test/src/ct_logs.erl index 29b38e9748..e6d683c8a9 100644 --- a/lib/common_test/src/ct_logs.erl +++ b/lib/common_test/src/ct_logs.erl @@ -269,7 +269,7 @@ cast(Msg) -> %%% <p>This function is called by ct_framework:init_tc/3</p> init_tc(RefreshLog) -> call({init_tc,self(),group_leader(),RefreshLog}), - io:format(["$tc_html",xhtml("", "<br />")]), + tc_io_format(group_leader(), xhtml("", "<br />"), []), ok. %%%----------------------------------------------------------------- @@ -800,7 +800,8 @@ logger_loop(State) -> %% make sure no IO for this test case from the %% CT logger gets rejected test_server:permit_io(GL, self()), - print_style(GL,GL,State#logger_state.stylesheet), + IoFormat = fun tc_io_format/3, + print_style(GL, IoFormat, State#logger_state.stylesheet), set_evmgr_gl(GL), TCGLs = add_tc_gl(TCPid,GL,State), if not RefreshLog -> @@ -931,7 +932,7 @@ print_to_log(sync, FromPid, Category, TCGL, Content, EscChars, State) -> if FromPid /= TCGL -> IoFun = create_io_fun(FromPid, CtLogFd, EscChars), IoList = lists:foldl(IoFun, [], Content), - try io:format(TCGL,["$tc_html","~ts"], [IoList]) of + try tc_io_format(TCGL, "~ts", [IoList]) of ok -> ok catch _:_ -> @@ -962,7 +963,7 @@ print_to_log(async, FromPid, Category, TCGL, Content, EscChars, State) -> case erlang:is_process_alive(TCGL) of true -> - try io:format(TCGL, ["$tc_html","~ts"], + try tc_io_format(TCGL, "~ts", [lists:foldl(IoFun,[],Content)]) of _ -> ok catch @@ -1113,27 +1114,25 @@ open_ctlog(MiscIoName) -> "View I/O logged after the test run</a></li>\n</ul>\n", [MiscIoName,MiscIoName]), - print_style(Fd,group_leader(),undefined), + print_style(Fd, fun io:format/3, undefined), io:format(Fd, xhtml("<br><h2>Progress Log</h2>\n<pre>\n", "<br />\n<h4>PROGRESS LOG</h4>\n<pre>\n"), []), Fd. -print_style(Fd,GL,undefined) -> +print_style(Fd, IoFormat, undefined) -> case basic_html() of true -> Style = "<style>\n div.ct_internal { background:lightgrey; color:black; }\n div.default { background:lightgreen; color:black; }\n </style>\n", - if Fd == GL -> io:format(["$tc_html",Style], []); - true -> io:format(Fd, Style, []) - end; + IoFormat(Fd, Style, []); _ -> ok end; -print_style(Fd,GL,StyleSheet) -> +print_style(Fd, IoFormat, StyleSheet) -> case file:read_file(StyleSheet) of {ok,Bin} -> Str = b2s(Bin,encoding(StyleSheet)), @@ -1146,36 +1145,55 @@ print_style(Fd,GL,StyleSheet) -> N1 -> N1 end, if (Pos0 == 0) and (Pos1 /= 0) -> - print_style_error(Fd,GL,StyleSheet,missing_style_start_tag); + print_style_error(Fd, IoFormat, + StyleSheet, missing_style_start_tag); (Pos0 /= 0) and (Pos1 == 0) -> - print_style_error(Fd,GL,StyleSheet,missing_style_end_tag); + print_style_error(Fd, IoFormat, + StyleSheet,missing_style_end_tag); Pos0 /= 0 -> Style = string:sub_string(Str,Pos0,Pos1+7), - if Fd == GL -> io:format(Fd,["$tc_html","~ts\n"],[Style]); - true -> io:format(Fd,"~ts\n",[Style]) - end; + IoFormat(Fd,"~ts\n",[Style]); Pos0 == 0 -> - if Fd == GL -> io:format(Fd,["$tc_html","<style>\n~ts</style>\n"],[Str]); - true -> io:format(Fd,"<style>\n~ts</style>\n",[Str]) - end + IoFormat(Fd,"<style>\n~ts</style>\n",[Str]) end; {error,Reason} -> - print_style_error(Fd,GL,StyleSheet,Reason) + print_style_error(Fd,IoFormat,StyleSheet,Reason) end. -print_style_error(Fd,GL,StyleSheet,Reason) -> +print_style_error(Fd, IoFormat, StyleSheet, Reason) -> IO = io_lib:format("\n<!-- Failed to load stylesheet ~ts: ~p -->\n", [StyleSheet,Reason]), - if Fd == GL -> io:format(Fd,["$tc_html",IO],[]); - true -> io:format(Fd,IO,[]) - end, - print_style(Fd,GL,undefined). + IoFormat(Fd, IO, []), + print_style(Fd, IoFormat, undefined). close_ctlog(Fd) -> io:format(Fd, "\n</pre>\n", []), io:format(Fd, [xhtml("<br><br>\n", "<br /><br />\n") | footer()], []), file:close(Fd). +%%%----------------------------------------------------------------- +%%% tc_io_format/3 +%%% Tell common_test's IO server (group leader) not to escape +%%% HTML characters. + +-spec tc_io_format(io:device(), io:format(), [term()]) -> 'ok'. + +tc_io_format(Fd, Format0, Args) -> + %% We know that the specially wrapped format string is handled + %% by our IO server, but Dialyzer does not and would tell us + %% that the call to io:format/3 would fail. Therefore, we must + %% fool dialyzer. + + Format = case cloaked_true() of + true -> ["$tc_html",Format0]; + false -> Format0 %Never happens. + end, + io:format(Fd, Format, Args). + +%% Return 'true', but let dialyzer think that a boolean is returned. +cloaked_true() -> + is_process_alive(self()). + %%%----------------------------------------------------------------- %%% Make an index page for the last run diff --git a/lib/compiler/src/beam_bool.erl b/lib/compiler/src/beam_bool.erl index f9a08f8718..359fdb6d3c 100644 --- a/lib/compiler/src/beam_bool.erl +++ b/lib/compiler/src/beam_bool.erl @@ -238,9 +238,9 @@ extend_block(BlAcc0, Fail, [{block,Is0}|OldAcc]) -> end; extend_block(BlAcc, _, OldAcc) -> {BlAcc,OldAcc}. -extend_block_1([{set,[_],_,{bif,_,{f,Fail}}}=I|Is], Fail, Acc) -> +extend_block_1([{set,[{x,_}],_,{bif,_,{f,Fail}}}=I|Is], Fail, Acc) -> extend_block_1(Is, Fail, [I|Acc]); -extend_block_1([{set,[_],As,{bif,Bif,_}}=I|Is]=Is0, Fail, Acc) -> +extend_block_1([{set,[{x,_}],As,{bif,Bif,_}}=I|Is]=Is0, Fail, Acc) -> case safe_bool_op(Bif, length(As)) of false -> {Acc,reverse(Is0)}; true -> extend_block_1(Is, Fail, [I|Acc]) diff --git a/lib/compiler/src/beam_utils.erl b/lib/compiler/src/beam_utils.erl index 37f89dd677..47703b4aa3 100644 --- a/lib/compiler/src/beam_utils.erl +++ b/lib/compiler/src/beam_utils.erl @@ -25,9 +25,8 @@ is_not_used/3,is_not_used_at/3, empty_label_index/0,index_label/3,index_labels/1, code_at/2,bif_to_test/3,is_pure_test/1, - live_opt/1,delete_live_annos/1,combine_heap_needs/2]). - --export([join_even/2,split_even/1]). + live_opt/1,delete_live_annos/1,combine_heap_needs/2, + join_even/2,split_even/1]). -import(lists, [member/2,sort/1,reverse/1,splitwith/2]). @@ -67,8 +66,7 @@ is_killed(R, Is, D) -> St = #live{bl=check_killed_block_fun(),lbl=D,res=gb_trees:empty()}, case check_liveness(R, Is, St) of {killed,_} -> true; - {used,_} -> false; - {unknown,_} -> false + {used,_} -> false end. %% is_killed_at(Reg, Lbl, State) -> true|false @@ -78,8 +76,7 @@ is_killed_at(R, Lbl, D) when is_integer(Lbl) -> St0 = #live{bl=check_killed_block_fun(),lbl=D,res=gb_trees:empty()}, case check_liveness_at(R, Lbl, St0) of {killed,_} -> true; - {used,_} -> false; - {unknown,_} -> false + {used,_} -> false end. %% is_not_used(Register, [Instruction], State) -> true|false @@ -93,8 +90,7 @@ is_not_used(R, Is, D) -> St = #live{bl=fun check_used_block/3,lbl=D,res=gb_trees:empty()}, case check_liveness(R, Is, St) of {killed,_} -> true; - {used,_} -> false; - {unknown,_} -> false + {used,_} -> false end. %% is_not_used(Register, [Instruction], State) -> true|false @@ -108,8 +104,7 @@ is_not_used_at(R, Lbl, D) -> St = #live{bl=fun check_used_block/3,lbl=D,res=gb_trees:empty()}, case check_liveness_at(R, Lbl, St) of {killed,_} -> true; - {used,_} -> false; - {unknown,_} -> false + {used,_} -> false end. %% index_labels(FunctionIs) -> State @@ -137,10 +132,7 @@ index_label(Lbl, Is0, Acc) -> %% Retrieve the code at the given label. code_at(L, Ll) -> - case gb_trees:lookup(L, Ll) of - {value,Code} -> Code; - none -> none - end. + gb_trees:get(L, Ll). %% bif_to_test(Bif, [Op], Fail) -> {test,Test,Fail,[Op]} %% Convert a BIF to a test. Fail if not possible. @@ -164,10 +156,10 @@ bif_to_test('=<', [A,B], Fail) -> {test,is_ge,Fail,[B,A]}; bif_to_test('>', [A,B], Fail) -> {test,is_lt,Fail,[B,A]}; bif_to_test('<', [_,_]=Ops, Fail) -> {test,is_lt,Fail,Ops}; bif_to_test('>=', [_,_]=Ops, Fail) -> {test,is_ge,Fail,Ops}; -bif_to_test('==', [A,[]], Fail) -> {test,is_nil,Fail,[A]}; +bif_to_test('==', [A,nil], Fail) -> {test,is_nil,Fail,[A]}; bif_to_test('==', [_,_]=Ops, Fail) -> {test,is_eq,Fail,Ops}; bif_to_test('/=', [_,_]=Ops, Fail) -> {test,is_ne,Fail,Ops}; -bif_to_test('=:=', [A,[]], Fail) -> {test,is_nil,Fail,[A]}; +bif_to_test('=:=', [A,nil], Fail) -> {test,is_nil,Fail,[A]}; bif_to_test('=:=', [_,_]=Ops, Fail) -> {test,is_eq_exact,Fail,Ops}; bif_to_test('=/=', [_,_]=Ops, Fail) -> {test,is_ne_exact,Fail,Ops}; bif_to_test(is_record, [_,_,_]=Ops, Fail) -> {test,is_record,Fail,Ops}. @@ -235,21 +227,28 @@ combine_heap_needs(Words, {alloc,Alloc}) when is_integer(Words) -> combine_heap_needs(H1, H2) when is_integer(H1), is_integer(H2) -> H1+H2. +%% split_even/1 +%% [1,2,3,4,5,6] -> {[1,3,5],[2,4,6]} + +split_even(Rs) -> split_even(Rs, [], []). + +%% join_even/1 +%% {[1,3,5],[2,4,6]} -> [1,2,3,4,5,6] + +join_even([], []) -> []; +join_even([S|Ss], [D|Ds]) -> [S,D|join_even(Ss, Ds)]. + %%% %%% Local functions. %%% -%% check_liveness(Reg, [Instruction], {State,BlockCheckFun}) -> -%% {killed | used | unknown,UpdateState} -%% Finds out how Reg is used in the instruction sequence. Returns one of: -%% killed - Reg is assigned a new value or killed by an allocation instruction -%% used - Reg is used (or possibly referenced by an allocation instruction) -%% unknown - not possible to determine (perhaps because of an instruction -%% that we don't recognize) +%% check_liveness(Reg, [Instruction], #live{}) -> +%% {killed | used, #live{}} +%% Find out whether Reg is used or killed in instruction sequence. +%% 'killed' means that Reg is assigned a new value or killed by an +%% allocation instruction. 'used' means that Reg is used in some way. -check_liveness(R, [{set,_,_,_}=I|_], St) -> - erlang:error(only_allowed_in_blocks, [R,I,St]); check_liveness(R, [{block,Blk}|Is], #live{bl=BlockCheck}=St0) -> case BlockCheck(R, Blk, St0) of {transparent,St} -> check_liveness(R, Is, St); @@ -461,8 +460,9 @@ check_liveness(R, [{loop_rec,{f,_},{x,0}}|_], St) -> {x,_} -> {killed,St}; _ -> - %% y register. Rarely happens. Be very conversative. - {unknown,St} + %% y register. Rarely happens. Be very conversative and + %% assume it's used. + {used,St} end; check_liveness(R, [{loop_rec_end,{f,Fail}}|_], St) -> check_liveness_at(R, Fail, St); @@ -493,7 +493,8 @@ check_liveness(R, [{put_map,{f,_},_,Src,_D,Live,{list,_}}|_], St0) -> {x,_} -> {killed,St0}; {y,_} -> - {unknown,St0} + %% Conservatively mark it as used. + {used,St0} end; check_liveness(R, [{test_heap,N,Live}|Is], St) -> I = {block,[{set,[],[],{alloc,Live,{nozero,nostack,N,[]}}}]}, @@ -505,12 +506,8 @@ check_liveness(R, [{get_list,S,D1,D2}|Is], St) -> I = {block,[{set,[D1,D2],[S],get_list}]}, check_liveness(R, [I|Is], St); check_liveness(_R, Is, St) when is_list(Is) -> -%% case Is of -%% [I|_] -> -%% io:format("~p ~p\n", [_R,I]); -%% _ -> ok -%% end, - {unknown,St}. + %% Not implemented. Conservatively assume that the register is used. + {used,St}. check_liveness_everywhere(R, [{f,Lbl}|T], St0) -> case check_liveness_at(R, Lbl, St0) of @@ -529,7 +526,7 @@ check_liveness_at(R, Lbl, #live{lbl=Ll,res=ResMemorized}=St0) -> none -> {Res,St} = case gb_trees:lookup(Lbl, Ll) of {value,Is} -> check_liveness(R, Is, St0); - none -> {unknown,St0} + none -> {used,St0} end, {Res,St#live{res=gb_trees:insert(Lbl, Res, St#live.res)}} end. @@ -594,8 +591,10 @@ check_killed_block(_, []) -> transparent. %% killed - Reg is assigned a new value or killed by an allocation instruction %% transparent - Reg is neither used nor killed %% used - Reg is explicitly used by an instruction -%% -%% (Unknown instructions will cause an exception.) +%% +%% '%live' annotations are not allowed. +%% +%% (Unknown instructions will cause an exception.) check_used_block({x,X}=R, [{set,Ds,Ss,{alloc,Live,Op}}|Is], St) -> if @@ -604,11 +603,6 @@ check_used_block({x,X}=R, [{set,Ds,Ss,{alloc,Live,Op}}|Is], St) -> end; check_used_block(R, [{set,Ds,Ss,Op}|Is], St) -> check_used_block_1(R, Ss, Ds, Op, Is, St); -check_used_block(R, [{'%live',Live,_}|Is], St) -> - case R of - {x,X} when X >= Live -> {killed,St}; - _ -> check_used_block(R, Is, St) - end; check_used_block(_, [], St) -> {transparent,St}. check_used_block_1(R, Ss, Ds, Op, Is, St0) -> @@ -639,8 +633,7 @@ is_reg_used_at_1(_, 0, St) -> is_reg_used_at_1(R, Lbl, St0) -> case check_liveness_at(R, Lbl, St0) of {killed,St} -> {false,St}; - {used,St} -> {true,St}; - {unknown,St} -> {true,St} + {used,St} -> {true,St} end. index_labels_1([{label,Lbl}|Is0], Acc) -> @@ -756,11 +749,6 @@ live_opt([{select,_,Src,Fail,List}=I|Is], Regs0, D, Acc) -> Regs1 = x_live([Src], Regs0), Regs = live_join_labels([Fail|List], D, Regs1), live_opt(Is, Regs, D, [I|Acc]); -live_opt([{'try',_,_}=I|Is], Regs, D, Acc) -> - %% If an exeption happens, all x registers will be killed. - %% Therefore, we should only base liveness of the code inside - %% the try. - live_opt(Is, Regs, D, [I|Acc]); live_opt([{try_case,_}=I|Is], _, D, Acc) -> live_opt(Is, live_call(1), D, [I|Acc]); live_opt([{loop_rec,_Fail,_Dst}=I|Is], _, D, Acc) -> @@ -860,14 +848,7 @@ x_live([], Regs) -> Regs. is_live(X, Regs) -> ((Regs bsr X) band 1) =:= 1. -%% split_even/1 -%% [1,2,3,4,5,6] -> {[1,3,5],[2,4,6]} -split_even(Rs) -> split_even(Rs,[],[]). -split_even([],Ss,Ds) -> {reverse(Ss),reverse(Ds)}; -split_even([S,D|Rs],Ss,Ds) -> - split_even(Rs,[S|Ss],[D|Ds]). - -%% join_even/1 -%% {[1,3,5],[2,4,6]} -> [1,2,3,4,5,6] -join_even([],[]) -> []; -join_even([S|Ss],[D|Ds]) -> [S,D|join_even(Ss,Ds)]. +split_even([], Ss, Ds) -> + {reverse(Ss),reverse(Ds)}; +split_even([S,D|Rs], Ss, Ds) -> + split_even(Rs, [S|Ss], [D|Ds]). diff --git a/lib/compiler/src/core_pp.erl b/lib/compiler/src/core_pp.erl index f34a5c034f..67209d06be 100644 --- a/lib/compiler/src/core_pp.erl +++ b/lib/compiler/src/core_pp.erl @@ -33,8 +33,7 @@ %% Prettyprint-formats (naively) an abstract Core Erlang syntax %% tree. --record(ctxt, {class = term :: 'clause' | 'def' | 'expr' | 'term', - indent = 0 :: integer(), +-record(ctxt, {indent = 0 :: integer(), item_indent = 2 :: integer(), body_indent = 4 :: integer(), line = 0 :: integer(), @@ -132,14 +131,11 @@ format_1(#c_literal{anno=A,val=Bitstring}, Ctxt) when is_bitstring(Bitstring) -> format_1(#c_binary{anno=A,segments=Segs}, Ctxt); format_1(#c_literal{anno=A,val=M},Ctxt) when is_map(M) -> Pairs = maps:to_list(M), - Op = case Ctxt of - #ctxt{ class = clause } -> exact; - _ -> assoc - end, - Cpairs = [#c_map_pair{op=#c_literal{val=Op}, + Op = #c_literal{val=assoc}, + Cpairs = [#c_map_pair{op=Op, key=#c_literal{val=K}, val=#c_literal{val=V}} || {K,V} <- Pairs], - format_1(#c_map{anno=A,arg=#c_literal{val=#{}},es=Cpairs},Ctxt); + format_1(#c_map{anno=A,arg=#c_literal{val=#{}},es=Cpairs},Ctxt); format_1(#c_var{name={I,A}}, _) -> [core_atom(I),$/,integer_to_list(A)]; format_1(#c_var{name=V}, _) -> @@ -340,35 +336,30 @@ format_1(#c_module{name=N,exports=Es,attrs=As,defs=Ds}, Ctxt) -> [Mod," [", format_vseq(Es, "", ",", - add_indent(set_class(Ctxt, term), width(Mod, Ctxt)+2), + add_indent(Ctxt, width(Mod, Ctxt)+2), fun format/2), "]", nl_indent(Ctxt), " attributes [", format_vseq(As, "", ",", - add_indent(set_class(Ctxt, def), 16), + add_indent(Ctxt, 16), fun format_def/2), "]", nl_indent(Ctxt), format_funcs(Ds, Ctxt), nl_indent(Ctxt) | "end" - ]; -format_1(Type, _) -> - ["** Unsupported type: ", - io_lib:write(Type) - | " **" ]. format_funcs(Fs, Ctxt) -> format_vseq(Fs, "", "", - set_class(Ctxt, def), + Ctxt, fun format_def/2). format_def({N,V}, Ctxt0) -> - Ctxt1 = add_indent(set_class(Ctxt0, expr), Ctxt0#ctxt.body_indent), + Ctxt1 = add_indent(Ctxt0, Ctxt0#ctxt.body_indent), [format(N, Ctxt0), " =", nl_indent(Ctxt1) @@ -392,8 +383,7 @@ do_format_bitstr(#c_bitstr{val=V,size=S,unit=U,type=T,flags=Fs}, Ctxt0) -> ["#<", Val, ">(", format_hseq(Vs,",", Ctxt2, fun format/2), $)]. format_clauses(Cs, Ctxt) -> - format_vseq(Cs, "", "", set_class(Ctxt, clause), - fun format_clause/2). + format_vseq(Cs, "", "", Ctxt, fun format_clause/2). format_clause(Node, Ctxt) -> maybe_anno(Node, fun format_clause_1/2, Ctxt). @@ -405,15 +395,13 @@ format_clause_1(#c_clause{pats=Ps,guard=G,body=B}, Ctxt) -> case is_trivial_guard(G) of true -> [" when ", - format_guard(G, add_indent(set_class(Ctxt, expr), - width(Ptxt, Ctxt) + 6))]; + format_guard(G, add_indent(Ctxt, width(Ptxt, Ctxt) + 6))]; false -> [nl_indent(Ctxt2), "when ", - format_guard(G, add_indent(set_class(Ctxt2, expr), 2))] + format_guard(G, add_indent(Ctxt2, 2))] end++ " ->", - nl_indent(Ctxt2) - | format(B, set_class(Ctxt2, expr)) + nl_indent(Ctxt2) | format(B, Ctxt2) ]. is_trivial_guard(#c_literal{val=Val}) when is_atom(Val) -> true; @@ -467,7 +455,7 @@ format_list_tail(Tail, Ctxt) -> format_map_pair(Op, K, V, Ctxt0) -> Ctxt1 = add_indent(Ctxt0, 1), - Txt = format(K, set_class(Ctxt1, expr)), + Txt = format(K, Ctxt1), Ctxt2 = add_indent(Ctxt0, width(Txt, Ctxt1)), [Txt,Op,format(V, Ctxt2)]. @@ -490,6 +478,7 @@ spaces(5) -> " "; spaces(6) -> " "; spaces(7) -> " ". +%% Undo indentation done by nl_indent/1. unindent(T, Ctxt) -> unindent(T, Ctxt#ctxt.indent, []). @@ -505,18 +494,11 @@ unindent([$\t|T], N, C) -> unindent([spaces(Tab - N)|T], 0, C) end; unindent([L|T], N, C) when is_list(L) -> - unindent(L, N, [T|C]); -unindent([H|T], _, C) -> - [H|[T|C]]; -unindent([], N, [H|T]) -> - unindent(H, N, T); -unindent([], _, []) -> []. + unindent(L, N, [T|C]). width(Txt, Ctxt) -> - try width(Txt, 0, Ctxt, []) - catch error:_ -> exit({bad_text,Txt}) - end. + width(Txt, 0, Ctxt, []). width([$\t|T], A, Ctxt, C) -> width(T, A + ?TAB_WIDTH, Ctxt, C); @@ -533,14 +515,9 @@ width([], A, _, []) -> A. add_indent(Ctxt, Dx) -> Ctxt#ctxt{indent = Ctxt#ctxt.indent + Dx}. -set_class(Ctxt, Class) -> - Ctxt#ctxt{class = Class}. - core_atom(A) -> io_lib:write_string(atom_to_list(A), $'). -is_simple_term(#c_values{es=Es}) -> - length(Es) < 3 andalso lists:all(fun is_simple_term/1, Es); is_simple_term(#c_tuple{es=Es}) -> length(Es) < 4 andalso lists:all(fun is_simple_term/1, Es); is_simple_term(#c_var{}) -> true; diff --git a/lib/compiler/src/sys_core_fold.erl b/lib/compiler/src/sys_core_fold.erl index b5b8d8a8ec..dbc27db377 100644 --- a/lib/compiler/src/sys_core_fold.erl +++ b/lib/compiler/src/sys_core_fold.erl @@ -374,10 +374,21 @@ expr(#c_receive{clauses=Cs0,timeout=T0,action=A0}=Recv, Ctxt, Sub) -> T1 = expr(T0, value, Sub), A1 = body(A0, Ctxt, Sub), Recv#c_receive{clauses=Cs1,timeout=T1,action=A1}; -expr(#c_apply{op=Op0,args=As0}=App, _, Sub) -> +expr(#c_apply{anno=Anno,op=Op0,args=As0}=App, _, Sub) -> Op1 = expr(Op0, value, Sub), As1 = expr_list(As0, value, Sub), - App#c_apply{op=Op1,args=As1}; + case Op1 of + #c_var{} -> + App#c_apply{op=Op1,args=As1}; + _ -> + add_warning(App, invalid_call), + Err = #c_call{anno=Anno, + module=#c_literal{val=erlang}, + name=#c_literal{val=error}, + args=[#c_tuple{es=[#c_literal{val='badfun'}, + Op1]}]}, + make_effect_seq(As1++[Err], Sub) + end; expr(#c_call{module=M0,name=N0}=Call0, Ctxt, Sub) -> M1 = expr(M0, value, Sub), N1 = expr(N0, value, Sub), @@ -3395,6 +3406,8 @@ format_error({no_effect,{erlang,F,A}}) -> format_error(result_ignored) -> "the result of the expression is ignored " "(suppress the warning by assigning the expression to the _ variable)"; +format_error(invalid_call) -> + "invalid function call"; format_error(useless_building) -> "a term is constructed, but never used"; format_error(bin_opt_alias) -> diff --git a/lib/compiler/test/Makefile b/lib/compiler/test/Makefile index 85118502e3..203a50db55 100644 --- a/lib/compiler/test/Makefile +++ b/lib/compiler/test/Makefile @@ -9,6 +9,7 @@ MODULES= \ andor_SUITE \ apply_SUITE \ beam_block_SUITE \ + beam_bool_SUITE \ beam_validator_SUITE \ beam_disasm_SUITE \ beam_except_SUITE \ @@ -46,6 +47,7 @@ NO_OPT= \ andor \ apply \ beam_block \ + beam_bool \ beam_except \ beam_reorder \ beam_type \ @@ -70,6 +72,7 @@ INLINE= \ andor \ apply \ beam_block \ + beam_bool \ beam_utils \ bs_bincomp \ bs_bit_binaries \ diff --git a/lib/compiler/test/andor_SUITE.erl b/lib/compiler/test/andor_SUITE.erl index f6e7b09010..05c087104d 100644 --- a/lib/compiler/test/andor_SUITE.erl +++ b/lib/compiler/test/andor_SUITE.erl @@ -22,8 +22,7 @@ -export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_group/2,end_per_group/2, t_case/1,t_and_or/1,t_andalso/1,t_orelse/1,inside/1,overlap/1, - combined/1,in_case/1,before_and_inside_if/1, - slow_compilation/1]). + combined/1,in_case/1,slow_compilation/1]). -include_lib("common_test/include/ct.hrl"). @@ -36,7 +35,7 @@ all() -> groups() -> [{p,[parallel], [t_case,t_and_or,t_andalso,t_orelse,inside,overlap, - combined,in_case,before_and_inside_if]}]. + combined,in_case,slow_compilation]}]. init_per_suite(Config) -> Config. @@ -450,64 +449,6 @@ in_case_1_guard(LenUp, LenDw, LenN, Rotation, Count) -> false -> loop end. -before_and_inside_if(Config) when is_list(Config) -> - no = before_and_inside_if([a], [b], delete), - no = before_and_inside_if([a], [b], x), - no = before_and_inside_if([a], [], delete), - no = before_and_inside_if([a], [], x), - no = before_and_inside_if([], [], delete), - yes = before_and_inside_if([], [], x), - yes = before_and_inside_if([], [b], delete), - yes = before_and_inside_if([], [b], x), - - {ch1,ch2} = before_and_inside_if_2([a], [b], blah), - {ch1,ch2} = before_and_inside_if_2([a], [b], xx), - {ch1,ch2} = before_and_inside_if_2([a], [], blah), - {ch1,ch2} = before_and_inside_if_2([a], [], xx), - {no,no} = before_and_inside_if_2([], [b], blah), - {no,no} = before_and_inside_if_2([], [b], xx), - {ch1,no} = before_and_inside_if_2([], [], blah), - {no,ch2} = before_and_inside_if_2([], [], xx), - ok. - -%% Thanks to Simon Cornish and Kostis Sagonas. -%% Used to crash beam_bool. -before_and_inside_if(XDo1, XDo2, Do3) -> - Do1 = (XDo1 =/= []), - Do2 = (XDo2 =/= []), - if - %% This expression occurs in a try/catch (protected) - %% block, which cannot refer to variables outside of - %% the block that are boolean expressions. - Do1 =:= true; - Do1 =:= false, Do2 =:= false, Do3 =:= delete -> - no; - true -> - yes - end. - -%% Thanks to Simon Cornish. -%% Used to generate code that would not set {y,0} on -%% all paths before its use (and therefore fail -%% validation by the beam_validator). -before_and_inside_if_2(XDo1, XDo2, Do3) -> - Do1 = (XDo1 =/= []), - Do2 = (XDo2 =/= []), - CH1 = if Do1 == true; - Do1 == false,Do2==false,Do3 == blah -> - ch1; - true -> - no - end, - CH2 = if Do1 == true; - Do1 == false,Do2==false,Do3 == xx -> - ch2; - true -> - no - end, - {CH1,CH2}. - - -record(state, {stack = []}). slow_compilation(_) -> diff --git a/lib/compiler/test/beam_block_SUITE.erl b/lib/compiler/test/beam_block_SUITE.erl index 81f8d10687..d343e26737 100644 --- a/lib/compiler/test/beam_block_SUITE.erl +++ b/lib/compiler/test/beam_block_SUITE.erl @@ -35,7 +35,8 @@ all() -> groups() -> [{p,[parallel], - [get_map_elements + [get_map_elements, + otp_7345 ]}]. init_per_suite(Config) -> diff --git a/lib/compiler/test/beam_bool_SUITE.erl b/lib/compiler/test/beam_bool_SUITE.erl new file mode 100644 index 0000000000..84d634e5ca --- /dev/null +++ b/lib/compiler/test/beam_bool_SUITE.erl @@ -0,0 +1,160 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2015-2016. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% +-module(beam_bool_SUITE). + +-export([all/0,suite/0,groups/0,init_per_suite/1,end_per_suite/1, + init_per_group/2,end_per_group/2, + before_and_inside_if/1, + scotland/1,y_registers/1]). + +suite() -> + [{ct_hooks,[ts_install_cth]}]. + +all() -> + test_lib:recompile(?MODULE), + [{group,p}]. + +groups() -> + [{p,[parallel], + [before_and_inside_if, + scotland, + y_registers + ]}]. + +init_per_suite(Config) -> + Config. + +end_per_suite(_Config) -> + ok. + +init_per_group(_GroupName, Config) -> + Config. + +end_per_group(_GroupName, Config) -> + Config. + +before_and_inside_if(_Config) -> + no = before_and_inside_if([a], [b], delete), + no = before_and_inside_if([a], [b], x), + no = before_and_inside_if([a], [], delete), + no = before_and_inside_if([a], [], x), + no = before_and_inside_if([], [], delete), + yes = before_and_inside_if([], [], x), + yes = before_and_inside_if([], [b], delete), + yes = before_and_inside_if([], [b], x), + + {ch1,ch2} = before_and_inside_if_2([a], [b], blah), + {ch1,ch2} = before_and_inside_if_2([a], [b], xx), + {ch1,ch2} = before_and_inside_if_2([a], [], blah), + {ch1,ch2} = before_and_inside_if_2([a], [], xx), + {no,no} = before_and_inside_if_2([], [b], blah), + {no,no} = before_and_inside_if_2([], [b], xx), + {ch1,no} = before_and_inside_if_2([], [], blah), + {no,ch2} = before_and_inside_if_2([], [], xx), + ok. + +%% Thanks to Simon Cornish and Kostis Sagonas. +%% Used to crash beam_bool. +before_and_inside_if(XDo1, XDo2, Do3) -> + Do1 = (XDo1 =/= []), + Do2 = (XDo2 =/= []), + if + %% This expression occurs in a try/catch (protected) + %% block, which cannot refer to variables outside of + %% the block that are boolean expressions. + Do1 =:= true; + Do1 =:= false, Do2 =:= false, Do3 =:= delete -> + no; + true -> + yes + end. + +%% Thanks to Simon Cornish. +%% Used to generate code that would not set {y,0} on +%% all paths before its use (and therefore fail +%% validation by the beam_validator). +before_and_inside_if_2(XDo1, XDo2, Do3) -> + Do1 = (XDo1 =/= []), + Do2 = (XDo2 =/= []), + CH1 = if Do1 == true; + Do1 == false,Do2==false,Do3 == blah -> + ch1; + true -> + no + end, + CH2 = if Do1 == true; + Do1 == false,Do2==false,Do3 == xx -> + ch2; + true -> + no + end, + {CH1,CH2}. + + +%% beam_bool would remove the initialization of {y,0}. +%% (Thanks to Thomas Arts and QuickCheck.) + +scotland(_Config) -> + million = do_scotland(placed), + {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(false)), + {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(true)), + {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(echo)), + ok. + +do_scotland(Echo) -> + found(case Echo of + Echo when true; Echo, Echo, Echo -> + Echo; + echo -> + [] + end, + Echo = placed). + +found(_, _) -> million. + + +%% ERL-143: beam_bool could not handle Y registers as a destination. +y_registers(_Config) -> + {'EXIT',{badarith,[_|_]}} = (catch baker(valentine)), + {'EXIT',{badarith,[_|_]}} = (catch baker(clementine)), + + {not_ok,true} = potter([]), + {ok,false} = potter([{encoding,any}]), + + ok. + +%% Thanks to Quickcheck. +baker(Baker) -> + (valentine == Baker) + + case Baker of + Baker when Baker; Baker -> + Baker; + Baker -> + [] + end. + +%% Thanks to Jose Valim. +potter(Modes) -> + Raw = lists:keyfind(encoding, 1, Modes) == false, + Final = case Raw of + X when X == false; X == nil -> ok; + _ -> not_ok + end, + {Final,Raw}. diff --git a/lib/compiler/test/compile_SUITE.erl b/lib/compiler/test/compile_SUITE.erl index e634f0fcc2..a15efc2a00 100644 --- a/lib/compiler/test/compile_SUITE.erl +++ b/lib/compiler/test/compile_SUITE.erl @@ -693,8 +693,7 @@ core(Config) when is_list(Config) -> Outdir = filename:join(PrivDir, "core"), ok = file:make_dir(Outdir), - Wc = filename:join(filename:dirname(code:which(?MODULE)), "*.beam"), - TestBeams = filelib:wildcard(Wc), + TestBeams = get_unique_beam_files(), Abstr = [begin {ok,{Mod,[{abstract_code, {raw_abstract_v1,Abstr}}]}} = beam_lib:chunks(Beam, [abstract_code]), @@ -755,8 +754,7 @@ core_roundtrip(Config) -> Outdir = filename:join(PrivDir, atom_to_list(?FUNCTION_NAME)), ok = file:make_dir(Outdir), - Wc = filename:join(filename:dirname(code:which(?MODULE)), "*.beam"), - TestBeams = filelib:wildcard(Wc), + TestBeams = get_unique_beam_files(), test_lib:p_run(fun(F) -> do_core_roundtrip(F, Outdir) end, TestBeams). do_core_roundtrip(Beam, Outdir) -> @@ -781,8 +779,13 @@ do_core_roundtrip_1(Mod, Abstr, Outdir) -> %% Primarily, test that annotations are accepted for all %% constructs. Secondarily, smoke test cerl_trees:label/1. - {Core,_} = cerl_trees:label(Core0), - do_core_roundtrip_2(Mod, Core, Outdir). + {Core1,_} = cerl_trees:label(Core0), + do_core_roundtrip_2(Mod, Core1, Outdir), + + %% Run the inliner to force generation of variables + %% with numeric names. + {ok,Mod,Core2} = compile:forms(Abstr, [inline,to_core]), + do_core_roundtrip_2(Mod, Core2, Outdir). do_core_roundtrip_2(M, Core0, Outdir) -> CoreFile = filename:join(Outdir, atom_to_list(M)++".core"), @@ -870,8 +873,7 @@ asm(Config) when is_list(Config) -> Outdir = filename:join(PrivDir, "asm"), ok = file:make_dir(Outdir), - Wc = filename:join(filename:dirname(code:which(?MODULE)), "*.beam"), - TestBeams = filelib:wildcard(Wc), + TestBeams = get_unique_beam_files(), Res = test_lib:p_run(fun(F) -> do_asm(F, Outdir) end, TestBeams), Res. @@ -947,8 +949,7 @@ dialyzer(Config) -> %% Test that warnings contain filenames and line numbers. warnings(_Config) -> - TestDir = filename:dirname(code:which(?MODULE)), - Files = filelib:wildcard(filename:join(TestDir, "*.erl")), + Files = get_unique_files(".erl"), test_lib:p_run(fun do_warnings/1, Files). do_warnings(F) -> @@ -1102,3 +1103,14 @@ compile_and_verify(Name, Target, Opts) -> beam_lib:chunks(Target, [compile_info]), {options,BeamOpts} = lists:keyfind(options, 1, CInfo), Opts = BeamOpts. + +get_unique_beam_files() -> + get_unique_files(".beam"). + +get_unique_files(Ext) -> + Wc = filename:join(filename:dirname(code:which(?MODULE)), "*"++Ext), + [F || F <- filelib:wildcard(Wc), not is_cloned(F, Ext)]. + +is_cloned(File, Ext) -> + Mod = list_to_atom(filename:basename(File, Ext)), + test_lib:is_cloned_mod(Mod). diff --git a/lib/compiler/test/fun_SUITE.erl b/lib/compiler/test/fun_SUITE.erl index 17ff8601d9..16474adf5b 100644 --- a/lib/compiler/test/fun_SUITE.erl +++ b/lib/compiler/test/fun_SUITE.erl @@ -22,7 +22,7 @@ -export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_group/2,end_per_group/2, test1/1,overwritten_fun/1,otp_7202/1,bif_fun/1, - external/1,eep37/1,eep37_dup/1,badarity/1]). + external/1,eep37/1,eep37_dup/1,badarity/1,badfun/1]). %% Internal exports. -export([call_me/1,dup1/0,dup2/0]). @@ -33,10 +33,12 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> test_lib:recompile(?MODULE), - [test1,overwritten_fun,otp_7202,bif_fun,external,eep37,eep37_dup,badarity]. + [{group,p}]. -groups() -> - []. +groups() -> + [{p,[parallel], + [test1,overwritten_fun,otp_7202,bif_fun,external,eep37, + eep37_dup,badarity,badfun]}]. init_per_suite(Config) -> Config. @@ -221,5 +223,25 @@ badarity(Config) when is_list(Config) -> {'EXIT',{{badarity,{_,[]}},_}} = (catch (fun badarity/1)()), ok. +badfun(_Config) -> + X = not_a_fun, + expect_badfun(42, catch 42()), + expect_badfun(42.0, catch 42.0(1)), + expect_badfun(X, catch X()), + expect_badfun(X, catch X(1)), + Len = length(atom_to_list(X)), + expect_badfun(Len, catch begin length(atom_to_list(X)) end(1)), + + expect_badfun(42, catch 42(put(?FUNCTION_NAME, yes))), + yes = erase(?FUNCTION_NAME), + + expect_badfun(X, catch X(put(?FUNCTION_NAME, of_course))), + of_course = erase(?FUNCTION_NAME), + + ok. + +expect_badfun(Term, Exit) -> + {'EXIT',{{badfun,Term},_}} = Exit. + id(I) -> I. diff --git a/lib/compiler/test/guard_SUITE.erl b/lib/compiler/test/guard_SUITE.erl index 13d274d98a..83298e546e 100644 --- a/lib/compiler/test/guard_SUITE.erl +++ b/lib/compiler/test/guard_SUITE.erl @@ -34,7 +34,7 @@ tricky/1,rel_ops/1,rel_op_combinations/1,literal_type_tests/1, basic_andalso_orelse/1,traverse_dcd/1, check_qlc_hrl/1,andalso_semi/1,t_tuple_size/1,binary_part/1, - bad_constants/1,bad_guards/1,scotland/1, + bad_constants/1,bad_guards/1, guard_in_catch/1]). suite() -> [{ct_hooks,[ts_install_cth]}]. @@ -54,7 +54,7 @@ groups() -> rel_ops,rel_op_combinations, literal_type_tests,basic_andalso_orelse,traverse_dcd, check_qlc_hrl,andalso_semi,t_tuple_size,binary_part, - bad_constants,bad_guards,scotland,guard_in_catch]}]. + bad_constants,bad_guards,guard_in_catch]}]. init_per_suite(Config) -> Config. @@ -1852,27 +1852,6 @@ bad_guards_2(M, [_]) when M#{a := 0, b => 0}, map_size(M) -> bad_guards_3(M, [_]) when is_map(M) andalso M#{a := 0, b => 0}, length(M) -> ok. -%% beam_bool would remove the initialization of {y,0}. -%% (Thanks to Thomas Arts and QuickCheck.) - -scotland(_Config) -> - million = do_scotland(placed), - {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(false)), - {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(true)), - {'EXIT',{{badmatch,placed},_}} = (catch do_scotland(echo)), - ok. - -do_scotland(Echo) -> - found(case Echo of - Echo when true; Echo, Echo, Echo -> - Echo; - echo -> - [] - end, - Echo = placed). - -found(_, _) -> million. - %% Building maps in a guard in a 'catch' would crash v3_codegen. guard_in_catch(_Config) -> diff --git a/lib/compiler/test/test_lib.erl b/lib/compiler/test/test_lib.erl index d141d86021..d5b79e2357 100644 --- a/lib/compiler/test/test_lib.erl +++ b/lib/compiler/test/test_lib.erl @@ -22,7 +22,7 @@ -include_lib("common_test/include/ct.hrl"). -compile({no_auto_import,[binary_part/2]}). -export([id/1,recompile/1,parallel/0,uniq/0,opt_opts/1,get_data_dir/1, - smoke_disasm/1,p_run/2,binary_part/2]). + is_cloned_mod/1,smoke_disasm/1,p_run/2,binary_part/2]). id(I) -> I. @@ -91,6 +91,17 @@ get_data_dir(Config) -> Data = re:replace(Data1, "_post_opt_SUITE", "_SUITE", Opts), re:replace(Data, "_inline_SUITE", "_SUITE", Opts). +is_cloned_mod(Mod) -> + is_cloned_mod_1(atom_to_list(Mod)). + +%% Test whether Mod is a cloned module. + +is_cloned_mod_1("no_opt_SUITE") -> true; +is_cloned_mod_1("post_opt_SUITE") -> true; +is_cloned_mod_1("inline_SUITE") -> true; +is_cloned_mod_1([_|T]) -> is_cloned_mod_1(T); +is_cloned_mod_1([]) -> false. + %% p_run(fun(Data) -> ok|error, List) -> ok %% Will fail the test case if there were any errors. @@ -106,8 +117,9 @@ p_run(Test, List) -> %% slightly faster than using 3. Using more than %% 4 would not buy us much and could actually be %% slower. - max(S, 4) + min(S, 4) end, + io:format("p_run: ~p parallel processes\n", [N]), p_run_loop(Test, List, N, [], 0, 0). p_run_loop(_, [], _, [], Errors, Ws) -> diff --git a/lib/compiler/test/trycatch_SUITE.erl b/lib/compiler/test/trycatch_SUITE.erl index f7ad78cb8d..a591d6cc93 100644 --- a/lib/compiler/test/trycatch_SUITE.erl +++ b/lib/compiler/test/trycatch_SUITE.erl @@ -1034,6 +1034,9 @@ grab_bag(_Config) -> end end, + %% Unnecessary catch. + 22 = (catch 22), + ok. diff --git a/lib/dialyzer/src/dialyzer_contracts.erl b/lib/dialyzer/src/dialyzer_contracts.erl index 116260778c..d1ffa07706 100644 --- a/lib/dialyzer/src/dialyzer_contracts.erl +++ b/lib/dialyzer/src/dialyzer_contracts.erl @@ -479,7 +479,8 @@ initialize_constraints([], _MFA, _RecDict, _ExpTypes, _AllRecords, Acc) -> initialize_constraints([Constr|Rest], MFA, RecDict, ExpTypes, AllRecords, Acc) -> case Constr of {type, _, constraint, [{atom, _, is_subtype}, [Type1, Type2]]} -> - T1 = final_form(Type1, ExpTypes, MFA, AllRecords, maps:new()), + VarTable = erl_types:var_table__new(), + T1 = final_form(Type1, ExpTypes, MFA, AllRecords, VarTable), Entry = {T1, Type2}, initialize_constraints(Rest, MFA, RecDict, ExpTypes, AllRecords, [Entry|Acc]); {type, _, constraint, [{atom,_,Name}, List]} -> @@ -489,8 +490,9 @@ initialize_constraints([Constr|Rest], MFA, RecDict, ExpTypes, AllRecords, Acc) - end. constraints_fixpoint(Constrs, MFA, RecDict, ExpTypes, AllRecords) -> + VarTable = erl_types:var_table__new(), VarDict = - constraints_to_dict(Constrs, MFA, RecDict, ExpTypes, AllRecords, maps:new()), + constraints_to_dict(Constrs, MFA, RecDict, ExpTypes, AllRecords, VarTable), constraints_fixpoint(VarDict, MFA, Constrs, RecDict, ExpTypes, AllRecords). constraints_fixpoint(OldVarDict, MFA, Constrs, RecDict, ExpTypes, AllRecords) -> @@ -498,11 +500,11 @@ constraints_fixpoint(OldVarDict, MFA, Constrs, RecDict, ExpTypes, AllRecords) -> constraints_to_dict(Constrs, MFA, RecDict, ExpTypes, AllRecords, OldVarDict), case NewVarDict of OldVarDict -> - DictFold = + Fun = fun(Key, Value, Acc) -> [{subtype, erl_types:t_var(Key), Value}|Acc] end, - FinalConstrs = maps:fold(DictFold, [], NewVarDict), + FinalConstrs = maps:fold(Fun, [], NewVarDict), {FinalConstrs, NewVarDict}; _Other -> constraints_fixpoint(NewVarDict, MFA, Constrs, RecDict, ExpTypes, AllRecords) @@ -512,12 +514,12 @@ final_form(Form, ExpTypes, MFA, AllRecords, VarDict) -> from_form_with_check(Form, ExpTypes, MFA, AllRecords, VarDict). from_form_with_check(Form, ExpTypes, MFA, AllRecords) -> - from_form_with_check(Form, ExpTypes, MFA, AllRecords, maps:new()). + VarTable = erl_types:var_table__new(), + from_form_with_check(Form, ExpTypes, MFA, AllRecords, VarTable). from_form_with_check(Form, ExpTypes, MFA, AllRecords, VarDict) -> Site = {spec, MFA}, - erl_types:t_check_record_fields(Form, ExpTypes, Site, AllRecords, - VarDict), + erl_types:t_check_record_fields(Form, ExpTypes, Site, AllRecords, VarDict), erl_types:t_from_form(Form, ExpTypes, Site, AllRecords, VarDict). constraints_to_dict(Constrs, MFA, RecDict, ExpTypes, AllRecords, VarDict) -> diff --git a/lib/dialyzer/src/dialyzer_dep.erl b/lib/dialyzer/src/dialyzer_dep.erl index 6678037bc0..273c05c54c 100644 --- a/lib/dialyzer/src/dialyzer_dep.erl +++ b/lib/dialyzer/src/dialyzer_dep.erl @@ -59,8 +59,14 @@ %% separately. %% --spec analyze(cerl:c_module()) -> - {dict:dict(), ordsets:ordset('external' | label()), dict:dict(), dict:dict()}. +-type dep_ordset() :: ordsets:ordset(label() | 'external'). + +-type deps() :: dict:dict(label() | 'external' | 'top', dep_ordset()). +-type esc() :: dep_ordset(). +-type calls() :: dict:dict(label(), ordsets:ordset(label())). +-type letrecs() :: dict:dict(label(), label()). + +-spec analyze(cerl:c_module()) -> {deps(), esc(), calls(), letrecs()}. analyze(Tree) -> %% io:format("Handling ~w\n", [cerl:atom_val(cerl:module_name(Tree))]), @@ -79,22 +85,26 @@ traverse(Tree, Out, State, CurrentFun) -> apply -> Op = cerl:apply_op(Tree), Args = cerl:apply_args(Tree), - %% Op is always a variable and should not be marked as escaping - %% based on its use. case var =:= cerl:type(Op) of - false -> erlang:error({apply_op_not_a_variable, cerl:type(Op)}); - true -> ok - end, - OpFuns = case map__lookup(cerl_trees:get_label(Op), Out) of - none -> output(none); - {value, OF} -> OF - end, - {ArgFuns, State2} = traverse_list(Args, Out, State, CurrentFun), - State3 = state__add_esc(merge_outs(ArgFuns), State2), - State4 = state__add_deps(CurrentFun, OpFuns, State3), - State5 = state__store_callsite(cerl_trees:get_label(Tree), - OpFuns, length(Args), State4), - {output(set__singleton(external)), State5}; + false -> + %% We have discovered an error here, but we ignore it and let + %% later passes handle it; we do not modify the dependencies. + %% erlang:error({apply_op_not_a_variable, cerl:type(Op)}); + {output(none), State}; + true -> + %% Op is a variable and should not be marked as escaping + %% based on its use. + OpFuns = case map__lookup(cerl_trees:get_label(Op), Out) of + none -> output(none); + {value, OF} -> OF + end, + {ArgFuns, State2} = traverse_list(Args, Out, State, CurrentFun), + State3 = state__add_esc(merge_outs(ArgFuns), State2), + State4 = state__add_deps(CurrentFun, OpFuns, State3), + State5 = state__store_callsite(cerl_trees:get_label(Tree), + OpFuns, length(Args), State4), + {output(set__singleton(external)), State5} + end; binary -> {output(none), State}; 'case' -> @@ -481,11 +491,11 @@ all_vars(Tree, AccIn) -> -type local_set() :: 'none' | #set{}. --record(state, {deps :: dict:dict(), +-record(state, {deps :: deps(), esc :: local_set(), - call :: dict:dict(), - arities :: dict:dict(), - letrecs :: dict:dict()}). + calls :: calls(), + arities :: dict:dict(label() | 'top', arity()), + letrecs :: letrecs()}). state__new(Tree) -> Exports = set__from_list([X || X <- cerl:module_exports(Tree)]), @@ -503,7 +513,7 @@ state__new(Tree) -> %% init the escaping function labels to exported + called from on_load InitEsc = set__from_list(OnLoadLs ++ ExpLs), Arities = cerl_trees:fold(fun find_arities/2, dict:new(), Tree), - #state{deps = map__new(), esc = InitEsc, call = map__new(), + #state{deps = map__new(), esc = InitEsc, calls = map__new(), arities = Arities, letrecs = map__new()}. find_arities(Tree, AccMap) -> @@ -518,7 +528,7 @@ find_arities(Tree, AccMap) -> state__add_deps(_From, #output{content = none}, State) -> State; -state__add_deps(From, #output{type = single, content=To}, +state__add_deps(From, #output{type = single, content = To}, #state{deps = Map} = State) -> %% io:format("Adding deps from ~w to ~w\n", [From, set__to_ordsets(To)]), State#state{deps = map__add(From, To, Map)}. @@ -544,16 +554,16 @@ state__esc(#state{esc = Esc}) -> state__store_callsite(_From, #output{content = none}, _CallArity, State) -> State; state__store_callsite(From, To, CallArity, - #state{call = Calls, arities = Arities} = State) -> + #state{calls = Calls, arities = Arities} = State) -> Filter = fun(external) -> true; (Fun) -> CallArity =:= dict:fetch(Fun, Arities) end, case filter_outs(To, Filter) of #output{content = none} -> State; - To1 -> State#state{call = map__store(From, To1, Calls)} + To1 -> State#state{calls = map__store(From, To1, Calls)} end. -state__calls(#state{call = Calls}) -> +state__calls(#state{calls = Calls}) -> Calls. %%------------------------------------------------------------ diff --git a/lib/dialyzer/vsn.mk b/lib/dialyzer/vsn.mk index 77ea9d0413..077fe01e85 100644 --- a/lib/dialyzer/vsn.mk +++ b/lib/dialyzer/vsn.mk @@ -1 +1 @@ -DIALYZER_VSN = 2.10 +DIALYZER_VSN = 3.0 diff --git a/lib/diameter/doc/src/diameter.xml b/lib/diameter/doc/src/diameter.xml index 66daf6bb0b..d68a78ed6d 100644 --- a/lib/diameter/doc/src/diameter.xml +++ b/lib/diameter/doc/src/diameter.xml @@ -1764,11 +1764,13 @@ An example return value with for a client service with Origin-Host {send_pend,0}]}]}, {statistics,[{{{0,258,0},recv},3}, {{{0,258,1},send},3}, + {{{0,258,0},recv,{'Result-Code',2001}},3}, {{{0,257,0},recv},1}, {{{0,257,1},send},1}, - {{{0,258,0},recv,{'Result-Code',2001}},3}, + {{{0,257,0},recv,{'Result-Code',2001}},1}, {{{0,280,1},recv},2}, - {{{0,280,0},send},2}]}]] + {{{0,280,0},send},2}, + {{{0,280,0},send,{'Result-Code',2001}},2}]}]] </pre> <p> @@ -1844,13 +1846,16 @@ connection might look as follows.</p> [{watchdog,{<0.72.0>,{1346,171491,998404},initial}}]]}, {statistics,[{{{0,280,0},recv},7}, {{{0,280,1},send},7}, - {{{0,258,0},send,{'Result-Code',2001}},3}, + {{{0,280,0},recv,{'Result-Code',2001}},7}, {{{0,258,1},recv},3}, {{{0,258,0},send},3}, + {{{0,258,0},send,{'Result-Code',2001}},3}, {{{0,280,1},recv},5}, {{{0,280,0},send},5}, + {{{0,280,0},send,{'Result-Code',2001}},5}, {{{0,257,1},recv},1}, - {{{0,257,0},send},1}]}]] + {{{0,257,0},send},1}, + {{{0,257,0},send,{'Result-Code',2001}},1}]}]] </pre> <p> @@ -1918,13 +1923,16 @@ A return value for the server above might look as follows.</p> {send_pend,0}]}]}, {statistics,[{{{0,280,0},recv},62}, {{{0,280,1},send},62}, - {{{0,258,0},send,{'Result-Code',2001}},3}, + {{{0,280,0},recv,{'Result-Code',2001}},62}, {{{0,258,1},recv},3}, {{{0,258,0},send},3}, + {{{0,258,0},send,{'Result-Code',2001}},3}, {{{0,280,1},recv},66}, {{{0,280,0},send},66}, + {{{0,280,0},send,{'Result-Code',2001}},66}, {{{0,257,1},recv},1}, - {{{0,257,0},send},1}]}]] + {{{0,257,0},send},1}, + {{{0,257,0},send,{'Result-Code',2001}},1}]}]] </pre> <p> diff --git a/lib/diameter/src/base/diameter.erl b/lib/diameter/src/base/diameter.erl index de88f6befd..e8f2f63f86 100644 --- a/lib/diameter/src/base/diameter.erl +++ b/lib/diameter/src/base/diameter.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2015. All Rights Reserved. +%% Copyright Ericsson AB 2010-2016. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -36,6 +36,8 @@ %% Information. -export([services/0, + peer_info/1, + peer_find/1, service_info/2]). %% Start/stop the application. In a "real" application this should @@ -53,6 +55,7 @@ service_name/0, capability/0, peer_filter/0, + peer_ref/0, service_opt/0, application_opt/0, app_module/0, @@ -147,6 +150,27 @@ service_info(SvcName, Option) -> diameter_service:info(SvcName, Option). %% --------------------------------------------------------------------------- +%% peer_info/2 +%% --------------------------------------------------------------------------- + +-spec peer_info(peer_ref()) + -> [tuple()]. + +peer_info(PeerRef) -> + diameter_service:peer_info(PeerRef). + +%% --------------------------------------------------------------------------- +%% peer_find/1 +%% --------------------------------------------------------------------------- + +-spec peer_find(peer_ref() | pid()) + -> {peer_ref(), pid()} + | false. + +peer_find(Pid) -> + diameter_peer_fsm:find(Pid). + +%% --------------------------------------------------------------------------- %% add_transport/3 %% --------------------------------------------------------------------------- @@ -280,6 +304,9 @@ call(SvcName, App, Message) -> | {all, [peer_filter()]} | {any, [peer_filter()]}. +-opaque peer_ref() + :: pid(). + -type evaluable() :: {module(), atom(), list()} | fun() diff --git a/lib/diameter/src/base/diameter_peer_fsm.erl b/lib/diameter/src/base/diameter_peer_fsm.erl index fb874013a3..996e75a8d3 100644 --- a/lib/diameter/src/base/diameter_peer_fsm.erl +++ b/lib/diameter/src/base/diameter_peer_fsm.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2015. All Rights Reserved. +%% Copyright Ericsson AB 2010-2016. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -32,6 +32,9 @@ -export([start/3, result_code/2]). +%% Interface towards diameter. +-export([find/1]). + %% gen_server callbacks -export([init/1, handle_call/3, @@ -185,6 +188,25 @@ start_link(T) -> infinity, diameter_lib:spawn_opts(server, [])). +%% find/1 +%% +%% Identify both pids of a peer_fsm/transport pair. + +find(Pid) -> + findl([{?MODULE, '_', Pid}, {?MODULE, Pid, '_'}]). + +findl([]) -> + false; + +findl([Pat | Rest]) -> + try + [{{_, Pid, TPid}, Pid}] = diameter_reg:match(Pat), + {Pid, TPid} + catch + error:_ -> + findl(Rest) + end. + %% --------------------------------------------------------------------------- %% --------------------------------------------------------------------------- @@ -215,6 +237,8 @@ i({Ack, WPid, {M, Ref} = T, Opts, {SvcOpts, Nodes, Dict0, Svc}}) -> {TPid, Addrs} = start_transport(T, Rest, Svc), + diameter_reg:add({?MODULE, self(), TPid}), %% lets pairs be discovered + #state{state = {'Wait-Conn-Ack', Tmo}, parent = WPid, transport = TPid, @@ -416,8 +440,8 @@ transition({connection_timeout, _}, _) -> ok; %% Incoming message from the transport. -transition({diameter, {recv, Pkt}}, S) -> - recv(Pkt, S); +transition({diameter, {recv, MsgT}}, S) -> + incoming(MsgT, S); %% Timeout when still in the same state ... transition({timeout = T, PS}, #state{state = PS}) -> @@ -543,6 +567,28 @@ encode(Rec, Dict) -> diameter_codec:encode(Dict, #diameter_packet{header = Hdr, msg = Rec}). +%% incoming/2 + +incoming({Msg, NPid}, S) -> + try recv(Msg, S) of + T -> + NPid ! {diameter, discard}, + T + catch + {?MODULE, Name, Pkt} -> + S#state.parent ! {recv, self(), Name, {Pkt, NPid}}, + rcv(Name, Pkt, S) + end; + +incoming(Msg, S) -> + try + recv(Msg, S) + catch + {?MODULE, Name, Pkt} -> + S#state.parent ! {recv, self(), Name, Pkt}, + rcv(Name, Pkt, S) + end. + %% recv/2 recv(#diameter_packet{header = #diameter_header{} = Hdr} @@ -597,9 +643,8 @@ recv1('DPA' = N, %% Any other message with a header and no length errors: send to the %% parent. -recv1(Name, Pkt, #state{parent = Pid} = S) -> - Pid ! {recv, self(), Name, Pkt}, - rcv(Name, Pkt, S). +recv1(Name, Pkt, #state{}) -> + throw({?MODULE, Name, Pkt}). %% recv/3 diff --git a/lib/diameter/src/base/diameter_service.erl b/lib/diameter/src/base/diameter_service.erl index efa4cc9108..cfb5cb5b82 100644 --- a/lib/diameter/src/base/diameter_service.erl +++ b/lib/diameter/src/base/diameter_service.erl @@ -32,6 +32,7 @@ -export([subscribe/1, unsubscribe/1, services/0, + peer_info/1, info/2]). %% towards diameter_config @@ -218,6 +219,29 @@ lookup_state(SvcName) -> end. %% --------------------------------------------------------------------------- +%% # peer_info/2 +%% --------------------------------------------------------------------------- + +%% An extended version of info_peer/1 for peer_info/1. +peer_info(Pid) -> + try + {_, PD} = process_info(Pid, dictionary), + {_, T} = lists:keyfind({diameter_peer_fsm, start}, 1, PD), + {TPid, {{Type, Ref}, TMod, Cfg}} = T, + {_, TD} = process_info(TPid, dictionary), + {_, Data} = lists:keyfind({TMod, info}, 1, TD), + [{ref, Ref}, + {type, Type}, + {owner, TPid}, + {module, TMod}, + {config, Cfg} + | try TMod:info(Data) catch _:_ -> [] end] + catch + error:_ -> + [] + end. + +%% --------------------------------------------------------------------------- %% # subscribe/1 %% # unsubscribe/1 %% --------------------------------------------------------------------------- diff --git a/lib/diameter/src/base/diameter_traffic.erl b/lib/diameter/src/base/diameter_traffic.erl index c169d3fc2c..2112941d5e 100644 --- a/lib/diameter/src/base/diameter_traffic.erl +++ b/lib/diameter/src/base/diameter_traffic.erl @@ -230,7 +230,15 @@ pending(TPids) -> %% used to come through the service process but this avoids that %% becoming a bottleneck. -receive_message(TPid, Pkt, Dict0, RecvData) +receive_message(TPid, {Pkt, NPid}, Dict0, RecvData) -> + NPid ! {diameter, incoming(TPid, Pkt, Dict0, RecvData)}; + +receive_message(TPid, Pkt, Dict0, RecvData) -> + incoming(TPid, Pkt, Dict0, RecvData). + +%% incoming/4 + +incoming(TPid, Pkt, Dict0, RecvData) when is_pid(TPid) -> #diameter_packet{header = #diameter_header{is_request = R}} = Pkt, recv(R, @@ -244,11 +252,18 @@ receive_message(TPid, Pkt, Dict0, RecvData) %% Incoming request ... recv(true, false, TPid, Pkt, Dict0, T) -> - spawn_request(TPid, Pkt, Dict0, T); + try + {request, spawn_request(TPid, Pkt, Dict0, T)} + catch + error: system_limit = E -> %% discard + ?LOG(error, E), + discard + end; %% ... answer to known request ... recv(false, #request{ref = Ref, handler = Pid} = Req, _, Pkt, Dict0, _) -> - Pid ! {answer, Ref, Req, Dict0, Pkt}; + Pid ! {answer, Ref, Req, Dict0, Pkt}, + {answer, Pid}; %% Note that failover could have happened prior to this message being %% received and triggering failback. That is, both a failover message @@ -263,7 +278,7 @@ recv(false, #request{ref = Ref, handler = Pid} = Req, _, Pkt, Dict0, _) -> recv(false, false, TPid, Pkt, _, _) -> ?LOG(discarded, Pkt#diameter_packet.header), incr(TPid, {{unknown, 0}, recv, discarded}), - ok. + discard. %% spawn_request/4 @@ -273,12 +288,7 @@ spawn_request(TPid, Pkt, Dict0, RecvData) -> spawn_request(TPid, Pkt, Dict0, ?DEFAULT_SPAWN_OPTS, RecvData). spawn_request(TPid, Pkt, Dict0, Opts, RecvData) -> - try - spawn_opt(fun() -> recv_request(TPid, Pkt, Dict0, RecvData) end, Opts) - catch - error: system_limit = E -> %% discard - ?LOG(error, E) - end. + spawn_opt(fun() -> recv_request(TPid, Pkt, Dict0, RecvData) end, Opts). %% --------------------------------------------------------------------------- %% recv_request/4 diff --git a/lib/diameter/src/base/diameter_watchdog.erl b/lib/diameter/src/base/diameter_watchdog.erl index ea8b2fdb0e..3fd87b223e 100644 --- a/lib/diameter/src/base/diameter_watchdog.erl +++ b/lib/diameter/src/base/diameter_watchdog.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2015. All Rights Reserved. +%% Copyright Ericsson AB 2010-2016. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -449,8 +449,14 @@ transition({'DOWN', _, process, TPid, _Reason} = D, end; %% Incoming message. -transition({recv, TPid, Name, Pkt}, #watchdog{transport = TPid} = S) -> - recv(Name, Pkt, S); +transition({recv, TPid, Name, PktT}, #watchdog{transport = TPid} = S) -> + try + incoming(Name, PktT, S) + catch + #watchdog{dictionary = Dict0, receive_data = T} = NS -> + diameter_traffic:receive_message(TPid, PktT, Dict0, T), + NS + end; %% Current watchdog has timed out. transition({timeout, TRef, tw}, #watchdog{tref = TRef} = S) -> @@ -578,22 +584,32 @@ send_watchdog(#watchdog{pending = false, %% Don't count encode errors since we don't expect any on DWR/DWA. +%% incoming/3 + +incoming(Name, {Pkt, NPid}, S) -> + NS = recv(Name, Pkt, S), + NPid ! {diameter, discard}, + NS; + +incoming(Name, Pkt, S) -> + recv(Name, Pkt, S). + %% recv/3 recv(Name, Pkt, S) -> - try rcv(Name, S) of + try rcv(Name, Pkt, rcv(Name, S)) of #watchdog{} = NS -> - rcv(Name, Pkt, S), - NS + throw(NS) catch - {?MODULE, throwaway, #watchdog{} = NS} -> + #watchdog{} = NS -> %% throwaway NS end. %% rcv/3 rcv('DWR', Pkt, #watchdog{transport = TPid, - dictionary = Dict0}) -> + dictionary = Dict0} + = S) -> ?LOG(recv, 'DWR'), DPkt = diameter_codec:decode(Dict0, Pkt), diameter_traffic:incr(recv, DPkt, TPid, Dict0), @@ -610,32 +626,30 @@ rcv('DWR', Pkt, #watchdog{transport = TPid, send(TPid, {send, #diameter_packet{header = H, transport_data = T, bin = Bin}}), - ?LOG(send, 'DWA'); + ?LOG(send, 'DWA'), + throw(S); rcv('DWA', Pkt, #watchdog{transport = TPid, - dictionary = Dict0}) -> + dictionary = Dict0} + = S) -> ?LOG(recv, 'DWA'), diameter_traffic:incr(recv, Pkt, TPid, Dict0), diameter_traffic:incr_rc(recv, diameter_codec:decode(Dict0, Pkt), TPid, - Dict0); + Dict0), + throw(S); -rcv(N, _, _) +rcv(N, _, S) when N == 'CER'; N == 'CEA'; N == 'DPR' -> - false; + throw(S); %% DPR can be sent explicitly with diameter:call/4. Only the %% corresponding DPAs arrive here. -rcv(_, Pkt, #watchdog{transport = TPid, - dictionary = Dict0, - receive_data = T}) -> - diameter_traffic:receive_message(TPid, Pkt, Dict0, T). - -throwaway(S) -> - throw({?MODULE, throwaway, S}). +rcv(_, _, S)-> + S. %% rcv/2 %% @@ -652,20 +666,20 @@ throwaway(S) -> %% INITIAL Receive non-DWA Throwaway() INITIAL rcv('DWA', #watchdog{status = initial} = S) -> - throwaway(S#watchdog{pending = false}); + throw(S#watchdog{pending = false}); rcv(_, #watchdog{status = initial} = S) -> - throwaway(S); + throw(S); %% DOWN Receive DWA Pending = FALSE %% Throwaway() DOWN %% DOWN Receive non-DWA Throwaway() DOWN rcv('DWA', #watchdog{status = down} = S) -> - throwaway(S#watchdog{pending = false}); + throw(S#watchdog{pending = false}); rcv(_, #watchdog{status = down} = S) -> - throwaway(S); + throw(S); %% OKAY Receive DWA Pending = FALSE %% SetWatchdog() OKAY @@ -721,7 +735,7 @@ rcv('DWR', #watchdog{status = reopen} = S) -> S; %% ensure DWA: the RFC isn't explicit about answering rcv(_, #watchdog{status = reopen} = S) -> - throwaway(S). + throw(S). %% timeout/1 %% diff --git a/lib/diameter/src/diameter.appup.src b/lib/diameter/src/diameter.appup.src index 12b09889d1..618d5a7f10 100644 --- a/lib/diameter/src/diameter.appup.src +++ b/lib/diameter/src/diameter.appup.src @@ -47,10 +47,9 @@ {"1.9.2.2", [{restart_application, diameter}]}, %% 17.5.6.7 {"1.9.2.3", [{restart_application, diameter}]}, %% 17.5.6.8 {"1.10", [{restart_application, diameter}]}, %% 18.0 - {"1.11", [{load_module, diameter_traffic}, %% 18.1 - {update, diameter_service, {advanced, []}}]}, - {"1.11.1", [{load_module, diameter_traffic}, %% 18.2 - {update, diameter_service, {advanced, []}}]} + {"1.11", [{restart_application, diameter}]}, %% 18.1 + {"1.11.1", [{restart_application, diameter}]}, %% 18.2 + {"1.11.2", [{restart_application, diameter}]} %% 18.3 ], [ {"0.9", [{restart_application, diameter}]}, @@ -80,6 +79,7 @@ {"1.9.2.3", [{restart_application, diameter}]}, {"1.10", [{restart_application, diameter}]}, {"1.11", [{restart_application, diameter}]}, - {"1.11.1", [{restart_application, diameter}]} + {"1.11.1", [{restart_application, diameter}]}, + {"1.11.2", [{restart_application, diameter}]} ] }. diff --git a/lib/diameter/src/transport/diameter_tcp.erl b/lib/diameter/src/transport/diameter_tcp.erl index c79d85820b..6a5e5fe89d 100644 --- a/lib/diameter/src/transport/diameter_tcp.erl +++ b/lib/diameter/src/transport/diameter_tcp.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2015. All Rights Reserved. +%% Copyright Ericsson AB 2010-2016. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -19,6 +19,7 @@ %% -module(diameter_tcp). +-dialyzer({no_fail_call, throttle/2}). -behaviour(gen_server). @@ -102,7 +103,8 @@ | gen_tcp:listen_option(). -type option() :: {port, non_neg_integer()} - | {fragment_timer, 0..16#FFFFFFFF}. + | {fragment_timer, 0..16#FFFFFFFF} + | {throttle_cb, diameter:evaluable()}. %% Accepting/connecting transport process state. -record(transport, @@ -110,10 +112,13 @@ parent :: pid(), %% of process that started us module :: module(), %% gen_tcp-like module frag = <<>> :: frag(), %% message fragment - ssl :: boolean() | [term()], %% ssl options + ssl :: [term()] | boolean(), %% ssl options, ssl or not timeout :: infinity | 0..16#FFFFFFFF, %% fragment timeout tref = false :: false | reference(), %% fragment timer reference - flush = false :: boolean()}). %% flush fragment at timeout? + flush = false :: boolean(), %% flush fragment at timeout? + throttle_cb :: false | diameter:evaluable(), %% ask to receive + throttled :: boolean() | binary()}). %% stopped receiving? + %% The usual transport using gen_tcp can be replaced by anything %% sufficiently gen_tcp-like by passing a 'module' option as the first %% (for simplicity) transport option. The transport_module diameter_etcp @@ -198,22 +203,27 @@ i({T, Ref, Mod, Pid, Opts, Addrs}) %% that does nothing but kill us with the parent until call %% returns. {ok, MPid} = diameter_tcp_sup:start_child(#monitor{parent = Pid}), - {SslOpts, Rest0} = ssl(Opts), - {OwnOpts, Rest} = own(Rest0), + {[SO|TO], Rest} = proplists:split(Opts, [ssl_options, + fragment_timer, + throttle_cb]), + SslOpts = ssl_opts(SO), + OwnOpts = lists:append(TO), Tmo = proplists:get_value(fragment_timer, OwnOpts, ?DEFAULT_FRAGMENT_TIMEOUT), ?IS_TIMEOUT(Tmo) orelse ?ERROR({fragment_timer, Tmo}), + Throttle = proplists:get_value(throttle_cb, OwnOpts, false), Sock = init(T, Ref, Mod, Pid, SslOpts, Rest, Addrs), MPid ! {stop, self()}, %% tell the monitor to die M = if SslOpts -> ssl; true -> Mod end, - setopts(M, Sock), putr(?REF_KEY, Ref), - #transport{parent = Pid, - module = M, - socket = Sock, - ssl = SslOpts, - timeout = Tmo}; + throttle(#transport{parent = Pid, + module = M, + socket = Sock, + ssl = SslOpts, + timeout = Tmo, + throttle_cb = Throttle, + throttled = false /= Throttle}); %% Put the reference in the process dictionary since we now use it %% advertise the ssl socket after TLS upgrade. @@ -246,14 +256,6 @@ laddr([], Mod, Sock) -> laddr([{ip, Addr}], _, _) -> Addr. -own(Opts) -> - {[Own], Rest} = proplists:split(Opts, [fragment_timer]), - {Own, Rest}. - -ssl(Opts) -> - {[SslOpts], Rest} = proplists:split(Opts, [ssl_options]), - {ssl_opts(SslOpts), Rest}. - ssl_opts([]) -> false; ssl_opts([{ssl_options, true}]) -> @@ -261,8 +263,8 @@ ssl_opts([{ssl_options, true}]) -> ssl_opts([{ssl_options, Opts}]) when is_list(Opts) -> Opts; -ssl_opts(L) -> - ?ERROR({ssl_options, L}). +ssl_opts(T) -> + ?ERROR({ssl_options, T}). %% init/7 @@ -393,7 +395,7 @@ get_port(Ps) -> gen_opts(LAddrOpt, Opts) -> {L,_} = proplists:split(Opts, [binary, packet, active]), [[],[],[]] == L orelse ?ERROR({reserved_options, Opts}), - [binary, {packet, 0}, {active, once}] ++ LAddrOpt ++ Opts. + [binary, {packet, 0}, {active, false}] ++ LAddrOpt ++ Opts. %% --------------------------------------------------------------------------- %% # ports/1 @@ -536,53 +538,37 @@ t(T,S) -> S; #transport{} = NS -> NS; - {stop, Reason} -> - x(Reason); stop -> x(T) end. %% transition/2 -%% Initial incoming message when we might need to upgrade to TLS: -%% don't request another message until we know. - -transition({tcp, Sock, Bin}, #transport{socket = Sock, - parent = Pid, - frag = Head, - module = M, - ssl = Opts} - = S) - when is_list(Opts) -> - case rcv(Head, Bin) of - {Msg, B} when is_binary(Msg) -> - diameter_peer:recv(Pid, Msg), - S#transport{frag = B}; - Frag -> - setopts(M, Sock), - start_fragment_timer(S#transport{frag = Frag}) - end; - %% Incoming message. transition({P, Sock, Bin}, #transport{socket = Sock, - module = M, - ssl = B} + ssl = B, + throttled = T} = S) - when P == tcp, not B; - P == ssl, B -> - setopts(M, Sock), - start_fragment_timer(recv(Bin, S)); + when P == ssl, true == B; + P == tcp -> + false = T, %% assert + recv(Bin, S); + +%% Make a new throttling callback after a timeout. +transition(throttle, #transport{throttled = false}) -> + ok; +transition(throttle, S) -> + throttle(S); %% Capabilties exchange has decided on whether or not to run over TLS. transition({diameter, {tls, Ref, Type, B}}, #transport{parent = Pid} = S) -> - #transport{socket = Sock, - module = M} + true = is_boolean(B), %% assert + #transport{} = NS = tls_handshake(Type, B, S), Pid ! {diameter, {tls, Ref}}, - setopts(M, Sock), - start_fragment_timer(NS#transport{ssl = B}); + throttle(NS#transport{ssl = B}); transition({C, Sock}, #transport{socket = Sock, ssl = B}) @@ -598,14 +584,8 @@ transition({E, Sock, _Reason} = T, #transport{socket = Sock, ?ERROR({T,S}); %% Outgoing message. -transition({diameter, {send, Bin}}, #transport{socket = Sock, - module = M}) -> - case send(M, Sock, Bin) of - ok -> - ok; - {error, Reason} -> - {stop, {send, Reason}} - end; +transition({diameter, {send, Bin}}, S) -> + send(Bin, S); %% Request to close the transport connection. transition({diameter, {close, Pid}}, #transport{parent = Pid, @@ -672,16 +652,25 @@ tls(accept, Sock, Opts) -> %% Reassemble fragmented messages and extract multiple message sent %% using Nagle. -recv(Bin, #transport{parent = Pid, frag = Head} = S) -> +%% Receive packets until a full message is received, +recv(Bin, #transport{frag = Head, throttled = false} = S) -> case rcv(Head, Bin) of - {Msg, B} when is_binary(Msg) -> - diameter_peer:recv(Pid, Msg), - recv(B, S#transport{frag = <<>>}); + {Msg, B} -> + throttle(S#transport{frag = B, throttled = Msg}); Frag -> - S#transport{frag = Frag, - flush = false} + setopts(S), + start_fragment_timer(S#transport{frag = Frag, + flush = false}) end. +%% recv/1 + +recv(#transport{throttled = false} = S) -> + recv(<<>>, S); + +recv(#transport{} = S) -> + S. + %% rcv/2 %% No previous fragment. @@ -765,8 +754,10 @@ bin(Bin) %% since all messages with length problems are discarded this should %% also eventually lead to watchdog failover. -%% No fragment to flush. -flush(#transport{frag = <<>>} = S) -> +%% No fragment to flush or not receiving messages. +flush(#transport{frag = Frag, throttled = B} = S) + when Frag == <<>>; + B /= false -> S; %% Messages have been received since last timer expiry. @@ -807,6 +798,17 @@ accept(Mod, LSock) -> connect(Mod, Host, Port, Opts) -> Mod:connect(Host, Port, Opts). +%% send/2 + +send(Bin, #transport{socket = Sock, + module = M}) -> + case send(M, Sock, Bin) of + ok -> + ok; + {error, Reason} -> + x({send, Reason}) + end. + %% send/3 send(gen_tcp, Sock, Bin) -> @@ -825,6 +827,11 @@ setopts(ssl, Sock, Opts) -> setopts(M, Sock, Opts) -> M:setopts(Sock, Opts). +%% setopts/1 + +setopts(#transport{socket = Sock, module = M}) -> + setopts(M, Sock). + %% setopts/2 setopts(M, Sock) -> @@ -833,6 +840,110 @@ setopts(M, Sock) -> X -> x({setopts, M, Sock, X}) %% possibly on peer disconnect end. +%% throttle/1 + +%% Still collecting packets for a complete message: keep receiving. +throttle(#transport{throttled = false} = S) -> + recv(S); + +%% Decide whether to receive another, or whether to accept a message +%% that's been received. +throttle(#transport{throttle_cb = F, throttled = T} = S) -> + Res = cb(F, T), + + try throttle(Res, S) of + #transport{ssl = SB} = NS when is_boolean(SB) -> + throttle(defrag(NS)); + #transport{throttled = Msg} = NS when is_binary(Msg) -> + %% Initial incoming message when we might need to upgrade + %% to TLS: wait for reception of a tls tuple. + defrag(NS) + catch + #transport{} = NS -> + recv(NS) + end. + +%% cb/2 + +cb(false, _) -> + ok; + +cb(F, B) -> + diameter_lib:eval([F, true /= B andalso B]). + +%% throttle/2 + +%% Callback says to receive another message. +throttle(ok, #transport{throttled = true} = S) -> + throw(S#transport{throttled = false}); + +%% Callback says to accept a received message. +throttle(ok, #transport{parent = Pid, throttled = Msg} = S) + when is_binary(Msg) -> + diameter_peer:recv(Pid, Msg), + S; + +throttle({ok = T, F}, S) -> + throttle(T, S#transport{throttle_cb = F}); + +%% Callback says to accept a received message and acknowledged the +%% returned pid with a {request, Pid} message if a request pid is +%% spawned, a discard message otherwise. The latter does not mean that +%% the message was necessarily discarded: it could have been an +%% answer. +throttle(NPid, #transport{parent = Pid, throttled = Msg} = S) + when is_pid(NPid), is_binary(Msg) -> + diameter_peer:recv(Pid, {Msg, NPid}), + S; + +throttle({NPid, F}, #transport{throttled = Msg} = S) + when is_pid(NPid), is_binary(Msg) -> + throttle(NPid, S#transport{throttle_cb = F}); + +%% Callback to accept a received message says to discard it. +throttle(discard, #transport{throttled = Msg} = S) + when is_binary(Msg) -> + S; + +throttle({discard = T, F}, #transport{throttled = Msg} = S) + when is_binary(Msg) -> + throttle(T, S#transport{throttle_cb = F}); + +%% Callback to accept a received message says to answer it with the +%% supplied binary. +throttle(Bin, #transport{throttled = Msg} = S) + when is_binary(Bin), is_binary(Msg) -> + send(Bin, S), + S; + +throttle({Bin, F}, #transport{throttled = Msg} = S) + when is_binary(Bin), is_binary(Msg) -> + throttle(Bin, S#transport{throttle_cb = F}); + +%% Callback says to ask again in the specified number of milliseconds. +throttle({timeout, Tmo}, S) -> + erlang:send_after(Tmo, self(), throttle), + throw(S); + +throttle({timeout = T, Tmo, F}, S) -> + throttle({T, Tmo}, S#transport{throttle_cb = F}); + +throttle(T, #transport{throttle_cb = F}) -> + ?ERROR({invalid_return, T, F}). + +%% defrag/1 +%% +%% Try to extract another message from packets already read before +%% another throttling callback. + +defrag(#transport{frag = Head} = S) -> + case rcv(Head, <<>>) of + {Msg, B} -> + S#transport{throttled = Msg, frag = B}; + _ -> + S#transport{throttled = true} + end. + %% portnr/2 portnr(gen_tcp, Sock) -> diff --git a/lib/diameter/test/diameter_tls_SUITE.erl b/lib/diameter/test/diameter_tls_SUITE.erl index 3784bf20a6..1ad897dcd2 100644 --- a/lib/diameter/test/diameter_tls_SUITE.erl +++ b/lib/diameter/test/diameter_tls_SUITE.erl @@ -327,10 +327,10 @@ make_cert(Dir, Keyfile, Certfile) -> "-subj /C=SE/ST=./L=Stockholm/CN=www.erlang.org"]), %% Hope for the best and only check that files are written. - [{_, _, {ok,_}},{_, _, {ok,_}}] - = [{P,O,T} || {P,C} <- [{KP,KC}, {CP,CC}], - O <- [os:cmd(C)], - T <- [file:read_file_info(P)]], + KR = os:cmd(KC), + {_, {ok, _}} = {KR, file:read_file_info(KP)}, + CR = os:cmd(CC), + {_, {ok, _}} = {CR, file:read_file_info(CP)}, {KP,CP}. diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index 836def8447..cb750c69a3 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -1,6 +1,6 @@ # %CopyrightBegin% # -# Copyright Ericsson AB 2010-2015. All Rights Reserved. +# Copyright Ericsson AB 2010-2016. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -17,5 +17,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 1.11.2 +DIAMETER_VSN = 1.12 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/eldap/src/eldap.erl b/lib/eldap/src/eldap.erl index dc236f8a44..625309271b 100644 --- a/lib/eldap/src/eldap.erl +++ b/lib/eldap/src/eldap.erl @@ -564,7 +564,12 @@ loop(Cpid, Data) -> ?MODULE:loop(Cpid, NewData); {_From, close} -> - {no_reply,_NewData} = do_unbind(Data), + % Ignore tcp error if connection is already closed. + try do_unbind(Data) of + {no_reply,_NewData} -> ok + catch + throw:{gen_tcp_error, _TcpErr} -> ok + end, unlink(Cpid), exit(closed); diff --git a/lib/eldap/test/eldap_basic_SUITE.erl b/lib/eldap/test/eldap_basic_SUITE.erl index 9a23f74e22..536e891a1e 100644 --- a/lib/eldap/test/eldap_basic_SUITE.erl +++ b/lib/eldap/test/eldap_basic_SUITE.erl @@ -94,7 +94,8 @@ connection_tests() -> client_side_start_tls_timeout, client_side_bind_timeout, client_side_add_timeout, - client_side_search_timeout + client_side_search_timeout, + close_after_tcp_error ]. @@ -312,6 +313,30 @@ tcp_connection(Config) -> end. %%%---------------------------------------------------------------- + +close_after_tcp_error(Config) -> + Host = proplists:get_value(listen_host, Config), + Port = proplists:get_value(listen_port, Config), + Opts = proplists:get_value(tcp_connect_opts, Config), + T = 1000, + case eldap:open([Host], [{timeout,T},{port,Port}|Opts]) of + {ok,H} -> + Sl = proplists:get_value(listen_socket, Config), + gen_tcp:close(Sl), + {error,{gen_tcp_error,closed}} = eldap:simple_bind(H, anon, anon), + ok = eldap:close(H), + wait_for_close(H); + Other -> ct:fail("eldap:open failed: ~p",[Other]) + end. + +wait_for_close(H) -> + case erlang:is_process_alive(H) of + true -> timer:sleep(100), + wait_for_close(H); + false -> ok + end. + +%%%---------------------------------------------------------------- ssl_connection(Config) -> Host = proplists:get_value(listen_host, Config), Port = proplists:get_value(ssl_listen_port, Config), diff --git a/lib/et/doc/src/et_desc.xmlsrc b/lib/et/doc/src/et_desc.xmlsrc index 96a4a9df31..29e0ab1fe3 100644 --- a/lib/et/doc/src/et_desc.xmlsrc +++ b/lib/et/doc/src/et_desc.xmlsrc @@ -55,13 +55,12 @@ <p></p> <code type="none"><![CDATA[ - % erl -pa et/examples - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] +% erl -pa et/examples +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] - Eshell V5.7.4 (abort with ^G) - 1> {ok, Viewer} = et_viewer:start([]). - {ok,<0.40.0>}]]></code> +Eshell V5.7.4 (abort with ^G) +1> {ok, Viewer} = et_viewer:start([]). +{ok,<0.40.0>}]]></code> <p>A <c>Viewer</c> gets trace <c>Events</c> from its <c>Collector</c> by polling it regularly for more <c>Events</c> to @@ -69,40 +68,38 @@ <c>Collector</c> with <c>et_collector:report_event/6</c>:</p> <code type="none"><![CDATA[ - 2> Collector = et_viewer:get_collector_pid(Viewer). - <0.39.0> - 3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, start_outer, - 3> "Start outer transaction"), - 3> et_collector:report_event(Collector, 40, mnesia_tm, my_shell, new_tid, - 3> "New transaction id is 4711"), - 3> et_collector:report_event(Collector, 20, my_shell, mnesia_locker, try_write_lock, - 3> "Acquire write lock for {my_tab, key}"), - 3> et_collector:report_event(Collector, 10, mnesia_locker, my_shell, granted, - 3> "You got the write lock for {my_tab, key}"), - 3> et_collector:report_event(Collector, 60, my_shell, do_commit, - 3> "Perform transaction commit"), - 3> et_collector:report_event(Collector, 40, my_shell, mnesia_locker, release_tid, - 3> "Release all locks for transaction 4711"), - 3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, delete_transaction, - 3> "End of outer transaction"), - 3> et_collector:report_event(Collector, 20, my_shell, end_outer, - 3> "Transaction returned {atomic, ok}"). - {ok,{table_handle,<0.39.0>,16402,trace_ts, - #Fun<et_collector.0.62831470>}}]]></code> +2> Collector = et_viewer:get_collector_pid(Viewer). +<0.39.0> +3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, start_outer, +3> "Start outer transaction"), +3> et_collector:report_event(Collector, 40, mnesia_tm, my_shell, new_tid, +3> "New transaction id is 4711"), +3> et_collector:report_event(Collector, 20, my_shell, mnesia_locker, try_write_lock, +3> "Acquire write lock for {my_tab, key}"), +3> et_collector:report_event(Collector, 10, mnesia_locker, my_shell, granted, +3> "You got the write lock for {my_tab, key}"), +3> et_collector:report_event(Collector, 60, my_shell, do_commit, +3> "Perform transaction commit"), +3> et_collector:report_event(Collector, 40, my_shell, mnesia_locker, release_tid, +3> "Release all locks for transaction 4711"), +3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, delete_transaction, +3> "End of outer transaction"), +3> et_collector:report_event(Collector, 20, my_shell, end_outer, +3> "Transaction returned {atomic, ok}"). +{ok,{table_handle,<0.39.0>,16402,trace_ts, + #Fun<et_collector.0.62831470>}}]]></code> <p>This actually is a simulation of the process <c>Events</c> caused by a <c>Mnesia</c> transaction that writes a record in a local table:</p> <code type="none"><![CDATA[ - mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).]]></code> +mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).]]></code> <p>At this stage when we have a couple of <c>Events</c>, it is time to show how it looks like in the graphical interface of <c>et_viewer</c>:</p> - <p></p> - <image file="sim_trans.png"> <icaption>A simulated Mnesia transaction which writes one record</icaption> </image> @@ -144,11 +141,11 @@ <p></p> <code type="none"><![CDATA[ - filter(TraceData) -> false | true | {true, NewEvent} +filter(TraceData) -> false | true | {true, NewEvent} - TraceData = Event | erlang_trace_data() - Event = #event{} - NewEvent = #event{}]]></code> +TraceData = Event | erlang_trace_data() +Event = #event{} +NewEvent = #event{}]]></code> <p>The interface of the filter function is the same as the the filter functions for the good old <c>lists:filtermap/2</c>. If the filter @@ -204,10 +201,10 @@ <p></p> <code type="none"><![CDATA[ - 4> Fun = fun(E) -> et_demo:mgr_actors(E) end. - #Fun<erl_eval.6.13229925> - 5> et_collector:dict_insert(Collector, {filter, mgr_actors}, Fun). - ok]]></code> +4> Fun = fun(E) -> et_demo:mgr_actors(E) end. +#Fun<erl_eval.6.13229925> +5> et_collector:dict_insert(Collector, {filter, mgr_actors}, Fun). +ok]]></code> <p>you will see that the <c>Filter</c> menu in all viewers have got a new entry called <c>mgr_actors</c>. Select it, and a new @@ -228,21 +225,16 @@ <c>Contents Viewer</c> window to pop up, showing the <c>Event</c> in the <c>mgr_actors</c> view:</p> - <p></p> - <image file="sim_trans_contents_viewer_mgr_actors.png"> - <icaption>The trace <c>Event</c> in the mgr_actors view</icaption> + <icaption>The trace Event in the mgr_actors view</icaption> </image> <p>Select the <c>all</c> entry in the <c>Filters</c> menu and a new <c>Contents Viewer window</c> will pop up showing the same trace <c>Event</c> in the collectors view:</p> - <p></p> - <image file="sim_trans_contents_viewer_collector.png"> - <icaption>The same trace <c>Event</c> in the collectors - view</icaption> + <icaption>The same trace Event in the collectors view</icaption> </image> </section> @@ -311,7 +303,7 @@ <c>et_collector</c> or <c>et_viewer</c> in order to activate the global tracing. There is no restriction on how many concurrent (anonymous) collectors you can have, but you can only have one - <b>global</b> <c>Collector</c> as its name is registered in + <em>global</em> <c>Collector</c> as its name is registered in <c>global</c>.</p> <p>In order to further simplify the tracing, you can make use of diff --git a/lib/et/doc/src/et_examples.xmlsrc b/lib/et/doc/src/et_examples.xmlsrc index 6f143a397e..f4d94f7cb0 100644 --- a/lib/et/doc/src/et_examples.xmlsrc +++ b/lib/et/doc/src/et_examples.xmlsrc @@ -55,34 +55,32 @@ <p></p> <code type="none"><![CDATA[ - mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).]]></code> +mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).]]></code> <p>And the viewer window will look like:</p> <p></p> <code type="none"><![CDATA[ - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] - Eshell V5.7.4 (abort with ^G) - 1> {ok, Viewer} = et_viewer:start([]). - {ok,<0.40.0>;} - 2> et_demo:sim_trans(). - {ok,{table_handle,<0.45.0>,24596,trace_ts, - #Fun<et_collector.0.62831470>}}]]></code> +Eshell V5.7.4 (abort with ^G) +1> {ok, Viewer} = et_viewer:start([]). +{ok,<0.40.0>;} +2> et_demo:sim_trans(). +{ok,{table_handle,<0.45.0>,24596,trace_ts, + #Fun<et_collector.0.62831470>}}]]></code> <p></p> <image file="sim_trans.png"> - <icaption>A simulated <c>Mnesia</c> transaction which writes one - record</icaption> + <icaption>A simulated Mnesia transaction which writes one record</icaption> </image> </section> <section> - <title>Some convenient functions used in the <c>Mnesia</c> transaction + <title>Some convenient functions used in the Mnesia transaction example</title> <p>The <c>module_as_actor</c> filter converts the <c>Event @@ -173,21 +171,19 @@ <p></p> <code type="none"><![CDATA[ - erl -pa ../examples - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] +erl -pa ../examples +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] + [async-threads:0] [kernel-poll:false] - Eshell V5.7.4 (abort with ^G) - 1> et_demo:live_trans(). - {atomic,ok}]]></code> +Eshell V5.7.4 (abort with ^G) +1> et_demo:live_trans(). +{atomic,ok}]]></code> <p>Please, explore the different filters in order to see how the traced transaction can be seen from different point of views:</p> - <p></p> - <image file="live_trans.png"> - <icaption>A real <c>Mnesia</c> transaction which writes one record</icaption> + <icaption>A real Mnesia transaction which writes one record</icaption> </image> </section> @@ -215,20 +211,20 @@ <p></p> <code type="none"><![CDATA[ - -module(megaco_filter). - -export([start/0]). - - start() -> - Options = - [{event_order, event_ts}, - {scale, 3}, - {max_actors, infinity}, - {trace_pattern, {megaco, max}}, - {trace_global, true}, - {dict_insert, {filter, megaco_filter}, fun filter/1}, - {active_filter, megaco_filter}, - {title, "Megaco tracer - Erlang/OTP"}], - et_viewer:start(Options).]]></code> +-module(megaco_filter). +-export([start/0]). + +start() -> + Options = + [{event_order, event_ts}, + {scale, 3}, + {max_actors, infinity}, + {trace_pattern, {megaco, max}}, + {trace_global, true}, + {dict_insert, {filter, megaco_filter}, fun filter/1}, + {active_filter, megaco_filter}, + {title, "Megaco tracer - Erlang/OTP"}], + et_viewer:start(Options).]]></code> <p>First we start an Erlang node with a global <c>Collector</c> and its <c>Viewer</c>.</p> @@ -236,13 +232,12 @@ <p></p> <code type="none"><![CDATA[ - erl -sname observer - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] +erl -sname observer +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] - Eshell V5.7.4 (abort with ^G) - (observer@falco)1> megaco_filter:start(). - {ok,<0.48.0>}]]></code> +Eshell V5.7.4 (abort with ^G) +(observer@falco)1> megaco_filter:start(). +{ok,<0.48.0>}]]></code> <p>Secondly we start another Erlang node which we connect the observer node, before we start the application that we want to @@ -253,28 +248,27 @@ <p></p> <code type="none"><![CDATA[ - erl -sname mgc -pa ../../megaco/examples/simple - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] - - Eshell V5.7.4 (abort with ^G) - (mgc@falco)1> net:ping(observer@falco). - pong - (mgc@falco)2> megaco:start(). - ok - (mgc@falco)3> megaco_simple_mgc:start(). - {ok,[{ok,2944, - {megaco_receive_handle,{deviceName,"controller"}, - megaco_pretty_text_encoder,[],megaco_tcp,dynamic}}, - {ok,2944, - {megaco_receive_handle,{deviceName,"controller"}, - megaco_pretty_text_encoder,[],megaco_udp,dynamic}}, - {ok,2945, - {megaco_receive_handle,{deviceName,"controller"}, - megaco_binary_encoder,[],megaco_tcp,dynamic}}, - {ok,2945, - {megaco_receive_handle,{deviceName,"controller"}, - megaco_binary_encoder,[],megaco_udp,dynamic}}]}]]></code> +erl -sname mgc -pa ../../megaco/examples/simple +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] + +Eshell V5.7.4 (abort with ^G) +(mgc@falco)1> net:ping(observer@falco). +pong +(mgc@falco)2> megaco:start(). +ok +(mgc@falco)3> megaco_simple_mgc:start(). +{ok,[{ok,2944, + {megaco_receive_handle,{deviceName,"controller"}, + megaco_pretty_text_encoder,[],megaco_tcp,dynamic}}, + {ok,2944, + {megaco_receive_handle,{deviceName,"controller"}, + megaco_pretty_text_encoder,[],megaco_udp,dynamic}}, + {ok,2945, + {megaco_receive_handle,{deviceName,"controller"}, + megaco_binary_encoder,[],megaco_tcp,dynamic}}, + {ok,2945, + {megaco_receive_handle,{deviceName,"controller"}, + megaco_binary_encoder,[],megaco_udp,dynamic}}]}]]></code> <p>And finally we start an Erlang node for the Media Gateways and connect to the observer node. Each Media Gateway connects to the @@ -288,94 +282,87 @@ <p></p> <code type="none"><![CDATA[ - Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] - - Eshell V5.7.4 (abort with ^G) - (mg@falco)1> net:ping(observer@falco). - pong - (mg@falco)2> megaco_simple_mg:start(). - [{{deviceName,"gateway_tt"}, - {error,{start_user,megaco_not_started}}}, - {{deviceName,"gateway_tb"}, - {error,{start_user,megaco_not_started}}}, - {{deviceName,"gateway_ut"}, - {error,{start_user,megaco_not_started}}}, - {{deviceName,"gateway_ub"}, - {error,{start_user,megaco_not_started}}}] - (mg@falco)3> megaco:start(). - ok - (mg@falco)4> megaco_simple_mg:start(). - [{{deviceName,"gateway_tt"}, - {1, - {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, - [{serviceChangeReply, - {'ServiceChangeReply', - [{megaco_term_id,false,["root"]}], - {serviceChangeResParms, - {'ServiceChangeResParm', - {deviceName,"controller"}, - asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, - asn1_NOVALUE}}}}]}]}}}, - {{deviceName,"gateway_tb"}, - {1, - {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, - [{serviceChangeReply, - {'ServiceChangeReply', - [{megaco_term_id,false,["root"]}], - {serviceChangeResParms, - {'ServiceChangeResParm', - {deviceName,"controller"}, - asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, - asn1_NOVALUE}}}}]}]}}}, - {{deviceName,"gateway_ut"}, - {1, - {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, - [{serviceChangeReply, - {'ServiceChangeReply', - [{megaco_term_id,false,["root"]}], - {serviceChangeResParms, - {'ServiceChangeResParm', - {deviceName,"controller"}, - asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, - asn1_NOVALUE}}}}]}]}}}, - {{deviceName,"gateway_ub"}, - {1, - {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, - [{serviceChangeReply, - {'ServiceChangeReply', - [{megaco_term_id,false,["root"]}], - {serviceChangeResParms, - {'ServiceChangeResParm', - {deviceName,"controller"}, - asn1_NOVALUE,asn1_NOVALUE, - asn1_NOVALUE,...}}}}]}]}}}]]]></code> +Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false] + +Eshell V5.7.4 (abort with ^G) +(mg@falco)1> net:ping(observer@falco). +pong +(mg@falco)2> megaco_simple_mg:start(). +[{{deviceName,"gateway_tt"}, + {error,{start_user,megaco_not_started}}}, + {{deviceName,"gateway_tb"}, + {error,{start_user,megaco_not_started}}}, + {{deviceName,"gateway_ut"}, + {error,{start_user,megaco_not_started}}}, + {{deviceName,"gateway_ub"}, + {error,{start_user,megaco_not_started}}}] +(mg@falco)3> megaco:start(). +ok +(mg@falco)4> megaco_simple_mg:start(). +[{{deviceName,"gateway_tt"}, + {1, + {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, + [{serviceChangeReply, + {'ServiceChangeReply', + [{megaco_term_id,false,["root"]}], + {serviceChangeResParms, + {'ServiceChangeResParm', + {deviceName,"controller"}, + asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, + asn1_NOVALUE}}}}]}]}}}, + {{deviceName,"gateway_tb"}, + {1, + {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, + [{serviceChangeReply, + {'ServiceChangeReply', + [{megaco_term_id,false,["root"]}], + {serviceChangeResParms, + {'ServiceChangeResParm', + {deviceName,"controller"}, + asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, + asn1_NOVALUE}}}}]}]}}}, + {{deviceName,"gateway_ut"}, + {1, + {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, + [{serviceChangeReply, + {'ServiceChangeReply', + [{megaco_term_id,false,["root"]}], + {serviceChangeResParms, + {'ServiceChangeResParm', + {deviceName,"controller"}, + asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE, + asn1_NOVALUE}}}}]}]}}}, + {{deviceName,"gateway_ub"}, + {1, + {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE, + [{serviceChangeReply, + {'ServiceChangeReply', + [{megaco_term_id,false,["root"]}], + {serviceChangeResParms, + {'ServiceChangeResParm', + {deviceName,"controller"}, + asn1_NOVALUE,asn1_NOVALUE, + asn1_NOVALUE,...}}}}]}]}}}]]]></code> <p>The <c>Megaco</c> adopted viewer looks like this, when we have clicked - on the <b>[gateway_tt]</b> actor name in order to only display the events + on the <em>[gateway_tt]</em> actor name in order to only display the events regarding that actor:</p> - <p></p> - <image file="megaco_tracer.png"> <icaption>The viewer adopted for Megaco</icaption> </image> <p>A pretty printed <c>Megaco</c> message looks like this:</p> - <p></p> - <image file="megaco_filter.png"> - <icaption>A textual <c>Megaco</c> message</icaption> + <icaption>A textual Megaco message</icaption> </image> <p>And the corresponding internal form for the same <c>Megaco</c> message looks like this:</p> - <p></p> - <image file="megaco_collector.png"> - <icaption>The internal form of a <c>Megaco</c> message</icaption> + <icaption>The internal form of a Megaco message</icaption> </image> </section> diff --git a/lib/et/doc/src/et_tutorial.xmlsrc b/lib/et/doc/src/et_tutorial.xmlsrc index b5d1b815be..b6e1ca141c 100644 --- a/lib/et/doc/src/et_tutorial.xmlsrc +++ b/lib/et/doc/src/et_tutorial.xmlsrc @@ -75,12 +75,10 @@ <codeinclude file="../../examples/et_display_demo.erl" tag="%module" type="erl"></codeinclude> <p>When you run the <c>et_display_demo:test().</c> function in the - example above, the <c>Viewer</c> window will look like this:</p>. - - <p></p> + example above, the <c>Viewer</c> window will look like this:</p> <image file="coffee_order.png"> - <icaption>Screenshot of the <c>Viewer</c> window</icaption> + <icaption>Screenshot of the Viewer window</icaption> </image> </section> @@ -262,14 +260,11 @@ </list> <p>When you run the <c>et_trace_demo:test()</c> function above, the - <c>Viewer</c> window will look like this screenshot:</p>. + <c>Viewer</c> window will look like this screenshot:</p> - <p></p> - <image file="coffee_order.png"> - <icaption>Screenshot of the <c>Viewer</c> window</icaption> + <icaption>Screenshot of the Viewer window</icaption> </image> </section> - </chapter> diff --git a/lib/hipe/cerl/erl_types.erl b/lib/hipe/cerl/erl_types.erl index 84addcc105..b037a4360c 100644 --- a/lib/hipe/cerl/erl_types.erl +++ b/lib/hipe/cerl/erl_types.erl @@ -220,6 +220,7 @@ is_opaque_type/2, is_erl_type/1, atom_to_string/1, + var_table__new/0, map_pairwise_merge/3 ]). @@ -236,7 +237,7 @@ -export([t_is_identifier/1]). -endif. --export_type([erl_type/0, opaques/0, type_table/0]). +-export_type([erl_type/0, opaques/0, type_table/0, var_table/0]). %%-define(DEBUG, true). @@ -380,7 +381,7 @@ -type type_table() :: dict:dict(record_key() | type_key(), record_value() | type_value()). --type var_table() :: #{atom() => erl_type()}. +-opaque var_table() :: #{atom() => erl_type()}. %%----------------------------------------------------------------------------- %% Unions @@ -5464,6 +5465,17 @@ family(L) -> sofs:to_external(F). %%============================================================================= +%% +%% Interface functions for abstract data types defined in this module +%% +%%============================================================================= + +-spec var_table__new() -> var_table(). + +var_table__new() -> + maps:new(). + +%%============================================================================= %% Consistency-testing function(s) below %%============================================================================= diff --git a/lib/inets/src/http_server/httpd_response.erl b/lib/inets/src/http_server/httpd_response.erl index c0b5f09faf..1374b7e85e 100644 --- a/lib/inets/src/http_server/httpd_response.erl +++ b/lib/inets/src/http_server/httpd_response.erl @@ -390,8 +390,7 @@ send_response_old(#mod{socket_type = Type, send_header(ModData, StatusCode, [{content_length, content_length(NewResponse)}]), httpd_socket:deliver(Type, Sock, NewResponse); - - {error, _Reason} -> + _ -> send_status(ModData, 500, "Internal Server Error") end. diff --git a/lib/inets/src/http_server/mod_htaccess.erl b/lib/inets/src/http_server/mod_htaccess.erl index 2bc0d79218..7b742bba24 100644 --- a/lib/inets/src/http_server/mod_htaccess.erl +++ b/lib/inets/src/http_server/mod_htaccess.erl @@ -412,8 +412,8 @@ getAuthenticatingDataFromHeader(Info)-> case httpd_util:split(UnCodedString,":",2) of {ok,[User,PassWord]}-> {user,User,PassWord}; - {error,Error}-> - {error,Error} + Other -> + {error, Other} end end; BadCredentials -> diff --git a/lib/inets/src/inets_app/Makefile b/lib/inets/src/inets_app/Makefile index a294381f67..1d870c14e8 100644 --- a/lib/inets/src/inets_app/Makefile +++ b/lib/inets/src/inets_app/Makefile @@ -49,8 +49,7 @@ MODULES = \ inets_sup \ inets_trace \ inets_lib \ - inets_time_compat \ - inets_regexp + inets_time_compat INTERNAL_HRL_FILES = inets_internal.hrl EXTERNAL_HRL_FILES = ../../include/httpd.hrl \ diff --git a/lib/inets/src/inets_app/inets.app.src b/lib/inets/src/inets_app/inets.app.src index abf8c0cdea..5706a335d7 100644 --- a/lib/inets/src/inets_app/inets.app.src +++ b/lib/inets/src/inets_app/inets.app.src @@ -29,7 +29,6 @@ inets_trace, inets_lib, inets_time_compat, - inets_regexp, %% FTP ftp, diff --git a/lib/inets/src/inets_app/inets_regexp.erl b/lib/inets/src/inets_app/inets_regexp.erl deleted file mode 100644 index fc1608bc5a..0000000000 --- a/lib/inets/src/inets_app/inets_regexp.erl +++ /dev/null @@ -1,414 +0,0 @@ -%% -%% %CopyrightBegin% -%% -%% Copyright Ericsson AB 2009. All Rights Reserved. -%% -%% Licensed under the Apache License, Version 2.0 (the "License"); -%% you may not use this file except in compliance with the License. -%% You may obtain a copy of the License at -%% -%% http://www.apache.org/licenses/LICENSE-2.0 -%% -%% Unless required by applicable law or agreed to in writing, software -%% distributed under the License is distributed on an "AS IS" BASIS, -%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -%% See the License for the specific language governing permissions and -%% limitations under the License. -%% -%% %CopyrightEnd% -%% - --module(inets_regexp). - --export([parse/1, match/2, first_match/2, split/2, sub/3, gsub/3]). - - -%%%========================================================================= -%%% API -%%%========================================================================= - -%% parse(RegExp) -> {ok, RE} | {error, E}. -%% Parse the regexp described in the string RegExp. - -parse(S) -> - case (catch reg(S)) of - {R, []} -> - {ok, R}; - {_R, [C|_]} -> - {error, {illegal, [C]}}; - {error, E} -> - {error, E} - end. - - -%% Find the longest match of RegExp in String. - -match(S, RegExp) when is_list(RegExp) -> - case parse(RegExp) of - {ok,RE} -> match(S, RE); - {error,E} -> {error,E} - end; -match(S, RE) -> - case match(RE, S, 1, 0, -1) of - {Start,Len} when Len >= 0 -> - {match, Start, Len}; - {_Start,_Len} -> - nomatch - end. - -%% Find the first match of RegExp in String. - -first_match(S, RegExp) when is_list(RegExp) -> - case parse(RegExp) of - {ok, RE} -> - first_match(S, RE); - {error, E} -> - {error, E} - end; -first_match(S, RE) -> - case first_match(RE, S, 1) of - {Start,Len} when Len >= 0 -> - {match, Start,Len}; - nomatch -> - nomatch - end. - -first_match(RE, S, St) when S =/= [] -> - case re_apply(S, St, RE) of - {match, P, _Rest} -> - {St, P-St}; - nomatch -> - first_match(RE, tl(S), St+1) - end; -first_match(_RE, [], _St) -> - nomatch. - - -match(RE, S, St, Pos, L) -> - case first_match(RE, S, St) of - {St1, L1} -> - Nst = St1 + 1, - if L1 > L -> - match(RE, lists:nthtail(Nst-St, S), Nst, St1, L1); - true -> - match(RE, lists:nthtail(Nst-St, S), Nst, Pos, L) - end; - nomatch -> - {Pos, L} - end. - - -%% Split a string into substrings where the RegExp describes the -%% field seperator. The RegExp " " is specially treated. - -split(String, " ") -> %This is really special - {ok, RE} = parse("[ \t]+"), - case split_apply(String, RE, true) of - [[]|Ss] -> - {ok,Ss}; - Ss -> - {ok,Ss} - end; -split(String, RegExp) when is_list(RegExp) -> - case parse(RegExp) of - {ok, RE} -> - {ok, split_apply(String, RE, false)}; - {error, E} -> - {error,E} - end; -split(String, RE) -> - {ok, split_apply(String, RE, false)}. - - -%% Substitute the first match of the regular expression RegExp -%% with the string Replace in String. Accept pre-parsed regular -%% expressions. - -sub(String, RegExp, Rep) when is_list(RegExp) -> - case parse(RegExp) of - {ok, RE} -> - sub(String, RE, Rep); - {error, E} -> - {error, E} - end; -sub(String, RE, Rep) -> - Ss = sub_match(String, RE, 1), - {ok, sub_repl(Ss, Rep, String, 1), length(Ss)}. - - -%% Substitute every match of the regular expression RegExp with -%% the string New in String. Accept pre-parsed regular expressions. - -gsub(String, RegExp, Rep) when is_list(RegExp) -> - case parse(RegExp) of - {ok, RE} -> - gsub(String, RE, Rep); - {error, E} -> - {error, E} - end; -gsub(String, RE, Rep) -> - Ss = matches(String, RE, 1), - {ok, sub_repl(Ss, Rep, String, 1), length(Ss)}. - - -%%%======================================================================== -%%% Internal functions -%%%======================================================================== - -%% This is the regular expression grammar used. It is equivalent to the -%% one used in AWK, except that we allow ^ $ to be used anywhere and fail -%% in the matching. -%% -%% reg -> reg1 : '$1'. -%% reg1 -> reg1 "|" reg2 : {'or','$1','$2'}. -%% reg1 -> reg2 : '$1'. -%% reg2 -> reg2 reg3 : {concat,'$1','$2'}. -%% reg2 -> reg3 : '$1'. -%% reg3 -> reg3 "*" : {kclosure,'$1'}. -%% reg3 -> reg3 "+" : {pclosure,'$1'}. -%% reg3 -> reg3 "?" : {optional,'$1'}. -%% reg3 -> reg4 : '$1'. -%% reg4 -> "(" reg ")" : '$2'. -%% reg4 -> "\\" char : '$2'. -%% reg4 -> "^" : bos. -%% reg4 -> "$" : eos. -%% reg4 -> "." : char. -%% reg4 -> "[" class "]" : {char_class,char_class('$2')} -%% reg4 -> "[" "^" class "]" : {comp_class,char_class('$3')} -%% reg4 -> "\"" chars "\"" : char_string('$2') -%% reg4 -> char : '$1'. -%% reg4 -> empty : epsilon. -%% The grammar of the current regular expressions. The actual parser -%% is a recursive descent implementation of the grammar. - -reg(S) -> reg1(S). - -%% reg1 -> reg2 reg1' -%% reg1' -> "|" reg2 -%% reg1' -> empty - -reg1(S0) -> - {L,S1} = reg2(S0), - reg1p(S1, L). - -reg1p([$||S0], L) -> - {R,S1} = reg2(S0), - reg1p(S1, {'or',L,R}); -reg1p(S, L) -> {L,S}. - -%% reg2 -> reg3 reg2' -%% reg2' -> reg3 -%% reg2' -> empty - -reg2(S0) -> - {L,S1} = reg3(S0), - reg2p(S1, L). - -reg2p([C|S0], L) when (C =/= $|) andalso (C =/= $)) -> - {R,S1} = reg3([C|S0]), - reg2p(S1, {concat,L,R}); -reg2p(S, L) -> {L,S}. - -%% reg3 -> reg4 reg3' -%% reg3' -> "*" reg3' -%% reg3' -> "+" reg3' -%% reg3' -> "?" reg3' -%% reg3' -> empty - -reg3(S0) -> - {L,S1} = reg4(S0), - reg3p(S1, L). - -reg3p([$*|S], L) -> reg3p(S, {kclosure,L}); -reg3p([$+|S], L) -> reg3p(S, {pclosure,L}); -reg3p([$?|S], L) -> reg3p(S, {optional,L}); -reg3p(S, L) -> {L,S}. - -reg4([$(|S0]) -> - case reg(S0) of - {R,[$)|S1]} -> {R,S1}; - {_R,_S} -> throw({error,{unterminated,"("}}) - end; -reg4([$\\,O1,O2,O3|S]) - when ((O1 >= $0) andalso - (O1 =< $7) andalso - (O2 >= $0) andalso - (O2 =< $7) andalso - (O3 >= $0) andalso - (O3 =< $7)) -> - {(O1*8 + O2)*8 + O3 - 73*$0,S}; -reg4([$\\,C|S]) -> - {escape_char(C),S}; -reg4([$\\]) -> - throw({error, {unterminated,"\\"}}); -reg4([$^|S]) -> - {bos,S}; -reg4([$$|S]) -> - {eos,S}; -reg4([$.|S]) -> - {{comp_class,"\n"},S}; -reg4("[^" ++ S0) -> - case char_class(S0) of - {Cc,[$]|S1]} -> {{comp_class,Cc},S1}; - {_Cc,_S} -> throw({error,{unterminated,"["}}) - end; -reg4([$[|S0]) -> - case char_class(S0) of - {Cc,[$]|S1]} -> {{char_class,Cc},S1}; - {_Cc,_S1} -> throw({error,{unterminated,"["}}) - end; -reg4([C|S]) - when (C =/= $*) andalso (C =/= $+) andalso (C =/= $?) andalso (C =/= $]) -> - {C, S}; -reg4([C|_S]) -> - throw({error,{illegal,[C]}}); -reg4([]) -> - {epsilon,[]}. - -escape_char($n) -> $\n; %\n = LF -escape_char($r) -> $\r; %\r = CR -escape_char($t) -> $\t; %\t = TAB -escape_char($v) -> $\v; %\v = VT -escape_char($b) -> $\b; %\b = BS -escape_char($f) -> $\f; %\f = FF -escape_char($e) -> $\e; %\e = ESC -escape_char($s) -> $\s; %\s = SPACE -escape_char($d) -> $\d; %\d = DEL -escape_char(C) -> C. - -char_class([$]|S]) -> char_class(S, [$]]); -char_class(S) -> char_class(S, []). - -char($\\, [O1,O2,O3|S]) when - O1 >= $0, O1 =< $7, O2 >= $0, O2 =< $7, O3 >= $0, O3 =< $7 -> - {(O1*8 + O2)*8 + O3 - 73*$0,S}; -char($\\, [C|S]) -> {escape_char(C),S}; -char(C, S) -> {C,S}. - -char_class([C1|S0], Cc) when C1 =/= $] -> - case char(C1, S0) of - {Cf,[$-,C2|S1]} when C2 =/= $] -> - case char(C2, S1) of - {Cl,S2} when Cf < Cl -> char_class(S2, [{Cf,Cl}|Cc]); - {Cl,_S2} -> throw({error,{char_class,[Cf,$-,Cl]}}) - end; - {C,S1} -> char_class(S1, [C|Cc]) - end; -char_class(S, Cc) -> {Cc,S}. - - -%% re_apply(String, StartPos, RegExp) -> re_app_res(). -%% -%% Apply the (parse of the) regular expression RegExp to String. If -%% there is a match return the position of the remaining string and -%% the string if else return 'nomatch'. BestMatch specifies if we want -%% the longest match, or just a match. -%% -%% StartPos should be the real start position as it is used to decide -%% if we ae at the beginning of the string. -%% -%% Pass two functions to re_apply_or so it can decide, on the basis -%% of BestMatch, whether to just any take any match or try both to -%% find the longest. This is slower but saves duplicatng code. - -re_apply(S, St, RE) -> re_apply(RE, [], S, St). - -re_apply(epsilon, More, S, P) -> %This always matches - re_apply_more(More, S, P); -re_apply({'or',RE1,RE2}, More, S, P) -> - re_apply_or(re_apply(RE1, More, S, P), - re_apply(RE2, More, S, P)); -re_apply({concat,RE1,RE2}, More, S0, P) -> - re_apply(RE1, [RE2|More], S0, P); -re_apply({kclosure,CE}, More, S, P) -> - %% Be careful with the recursion, explicitly do one call before - %% looping. - re_apply_or(re_apply_more(More, S, P), - re_apply(CE, [{kclosure,CE}|More], S, P)); -re_apply({pclosure,CE}, More, S, P) -> - re_apply(CE, [{kclosure,CE}|More], S, P); -re_apply({optional,CE}, More, S, P) -> - re_apply_or(re_apply_more(More, S, P), - re_apply(CE, More, S, P)); -re_apply(bos, More, S, 1) -> re_apply_more(More, S, 1); -re_apply(eos, More, [$\n|S], P) -> re_apply_more(More, S, P); -re_apply(eos, More, [], P) -> re_apply_more(More, [], P); -re_apply({char_class,Cc}, More, [C|S], P) -> - case in_char_class(C, Cc) of - true -> re_apply_more(More, S, P+1); - false -> nomatch - end; -re_apply({comp_class,Cc}, More, [C|S], P) -> - case in_char_class(C, Cc) of - true -> nomatch; - false -> re_apply_more(More, S, P+1) - end; -re_apply(C, More, [C|S], P) when is_integer(C) -> - re_apply_more(More, S, P+1); -re_apply(_RE, _More, _S, _P) -> nomatch. - -%% re_apply_more([RegExp], String, Length) -> re_app_res(). - -re_apply_more([RE|More], S, P) -> re_apply(RE, More, S, P); -re_apply_more([], S, P) -> {match,P,S}. - -%% in_char_class(Char, Class) -> bool(). - -in_char_class(C, [{C1,C2}|_Cc]) when C >= C1, C =< C2 -> true; -in_char_class(C, [C|_Cc]) -> true; -in_char_class(C, [_|Cc]) -> in_char_class(C, Cc); -in_char_class(_C, []) -> false. - -%% re_apply_or(Match1, Match2) -> re_app_res(). -%% If we want the best match then choose the longest match, else just -%% choose one by trying sequentially. - -re_apply_or({match,P1,S1}, {match,P2,_S2}) when P1 >= P2 -> {match,P1,S1}; -re_apply_or({match,_P1,_S1}, {match,P2,S2}) -> {match,P2,S2}; -re_apply_or(nomatch, R2) -> R2; -re_apply_or(R1, nomatch) -> R1. - - -matches(S, RE, St) -> - case first_match(RE, S, St) of - {St1,0} -> - [{St1,0}|matches(string:substr(S, St1+2-St), RE, St1+1)]; - {St1,L1} -> - [{St1,L1}|matches(string:substr(S, St1+L1+1-St), RE, St1+L1)]; - nomatch -> - [] - end. - -sub_match(S, RE, St) -> - case first_match(RE, S, St) of - {St1,L1} -> [{St1,L1}]; - nomatch -> [] - end. - -sub_repl([{St,L}|Ss], Rep, S, Pos) -> - Rs = sub_repl(Ss, Rep, S, St+L), - string:substr(S, Pos, St-Pos) ++ - sub_repl(Rep, string:substr(S, St, L), Rs); -sub_repl([], _Rep, S, Pos) -> - string:substr(S, Pos). - -sub_repl([$&|Rep], M, Rest) -> M ++ sub_repl(Rep, M, Rest); -sub_repl("\\&" ++ Rep, M, Rest) -> [$&|sub_repl(Rep, M, Rest)]; -sub_repl([C|Rep], M, Rest) -> [C|sub_repl(Rep, M, Rest)]; -sub_repl([], _M, Rest) -> Rest. - -split_apply(S, RE, Trim) -> split_apply(S, 1, RE, Trim, []). - -split_apply([], _P, _RE, true, []) -> - []; -split_apply([], _P, _RE, _T, Sub) -> - [lists:reverse(Sub)]; -split_apply(S, P, RE, T, Sub) -> - case re_apply(S, P, RE) of - {match,P,_Rest} -> - split_apply(tl(S), P+1, RE, T, [hd(S)|Sub]); - {match,P1,Rest} -> - [lists:reverse(Sub)|split_apply(Rest, P1, RE, T, [])]; - nomatch -> - split_apply(tl(S), P+1, RE, T, [hd(S)|Sub]) - end. diff --git a/lib/inets/vsn.mk b/lib/inets/vsn.mk index ef985f7c4e..543e0d44fd 100644 --- a/lib/inets/vsn.mk +++ b/lib/inets/vsn.mk @@ -18,6 +18,7 @@ # # %CopyrightEnd% +APPLICATION = inets INETS_VSN = 6.3 PRE_VSN = APP_VSN = "$(APPLICATION)-$(INETS_VSN)$(PRE_VSN)" diff --git a/lib/mnesia/src/Makefile b/lib/mnesia/src/Makefile index ae24bc72de..08a00e6aba 100644 --- a/lib/mnesia/src/Makefile +++ b/lib/mnesia/src/Makefile @@ -43,6 +43,7 @@ RELSYSDIR = $(RELEASE_PATH)/lib/mnesia-$(VSN) # ---------------------------------------------------- MODULES= \ mnesia \ + mnesia_backend_type \ mnesia_backup \ mnesia_bup \ mnesia_checkpoint \ @@ -50,6 +51,7 @@ MODULES= \ mnesia_controller \ mnesia_dumper\ mnesia_event \ + mnesia_ext_sup \ mnesia_frag \ mnesia_frag_hash \ mnesia_frag_old_hash \ diff --git a/lib/mnesia/src/mnesia.app.src b/lib/mnesia/src/mnesia.app.src index c78a7cba1e..006ad4bac1 100644 --- a/lib/mnesia/src/mnesia.app.src +++ b/lib/mnesia/src/mnesia.app.src @@ -3,6 +3,7 @@ {vsn, "%VSN%"}, {modules, [ mnesia, + mnesia_backend_type, mnesia_backup, mnesia_bup, mnesia_checkpoint, @@ -10,6 +11,7 @@ mnesia_controller, mnesia_dumper, mnesia_event, + mnesia_ext_sup, mnesia_frag, mnesia_frag_hash, mnesia_frag_old_hash, diff --git a/lib/mnesia/src/mnesia.erl b/lib/mnesia/src/mnesia.erl index e3d9bb1484..9586adbf93 100644 --- a/lib/mnesia/src/mnesia.erl +++ b/lib/mnesia/src/mnesia.erl @@ -82,7 +82,8 @@ system_info/0, % Not for public use %% Database mgt - create_schema/1, delete_schema/1, + create_schema/1, create_schema/2, delete_schema/1, + add_backend_type/2, backup/1, backup/2, traverse_backup/4, traverse_backup/6, install_fallback/1, install_fallback/2, uninstall_fallback/0, uninstall_fallback/1, @@ -197,6 +198,9 @@ e_has_var(X, Pos) -> %% Start and stop start() -> + start([]). + +start_() -> {Time , Res} = timer:tc(application, start, [?APPLICATION, temporary]), Secs = Time div 1000000, @@ -232,7 +236,7 @@ patched_start([{Env, Val} | Tail]) when is_atom(Env) -> patched_start([Head | _]) -> {error, {bad_type, Head}}; patched_start([]) -> - start(). + start_(). stop() -> case application:stop(?APPLICATION) of @@ -297,6 +301,7 @@ ms() -> %% Keep these last in the list, so %% mnesia_sup kills these last + mnesia_ext_sup, mnesia_monitor, mnesia_event ]. @@ -556,22 +561,16 @@ write(_Tid, _Ts, Tab, Val, LockKind) -> abort({bad_type, Tab, Val, LockKind}). write_to_store(Tab, Store, Oid, Val) -> - case ?catch_val({Tab, record_validation}) of - {RecName, Arity, Type} - when tuple_size(Val) == Arity, RecName == element(1, Val) -> - case Type of - bag -> - ?ets_insert(Store, {Oid, Val, write}); - _ -> - ?ets_delete(Store, Oid), - ?ets_insert(Store, {Oid, Val, write}) - end, - ok; - {'EXIT', _} -> - abort({no_exists, Tab}); - _ -> - abort({bad_type, Val}) - end. + {_, _, Type} = mnesia_lib:validate_record(Tab, Val), + Oid = {Tab, element(2, Val)}, + case Type of + bag -> + ?ets_insert(Store, {Oid, Val, write}); + _ -> + ?ets_delete(Store, Oid), + ?ets_insert(Store, {Oid, Val, write}) + end, + ok. delete({Tab, Key}) -> delete(Tab, Key, write); @@ -1548,16 +1547,9 @@ dirty_write(Tab, Val) -> do_dirty_write(SyncMode, Tab, Val) when is_atom(Tab), Tab /= schema, is_tuple(Val), tuple_size(Val) > 2 -> - case ?catch_val({Tab, record_validation}) of - {RecName, Arity, _Type} - when tuple_size(Val) == Arity, RecName == element(1, Val) -> - Oid = {Tab, element(2, Val)}, - mnesia_tm:dirty(SyncMode, {Oid, Val, write}); - {'EXIT', _} -> - abort({no_exists, Tab}); - _ -> - abort({bad_type, Val}) - end; + {_, _, _} = mnesia_lib:validate_record(Tab, Val), + Oid = {Tab, element(2, Val)}, + mnesia_tm:dirty(SyncMode, {Oid, Val, write}); do_dirty_write(_SyncMode, Tab, Val) -> abort({bad_type, Tab, Val}). @@ -1609,8 +1601,8 @@ dirty_update_counter(Tab, Key, Incr) -> do_dirty_update_counter(SyncMode, Tab, Key, Incr) when is_atom(Tab), Tab /= schema, is_integer(Incr) -> - case ?catch_val({Tab, record_validation}) of - {RecName, 3, set} -> + case mnesia_lib:validate_key(Tab, Key) of + {RecName, 3, Type} when Type == set; Type == ordered_set -> Oid = {Tab, Key}, mnesia_tm:dirty(SyncMode, {Oid, {RecName, Incr}, update_counter}); _ -> @@ -1910,6 +1902,8 @@ raw_table_info(Tab, Item) -> info_reply(?ets_info(Tab, Item), Tab, Item); disc_only_copies -> info_reply(dets:info(Tab, Item), Tab, Item); + {ext, Alias, Mod} -> + info_reply(catch Mod:info(Alias, Tab, Item), Tab, Item); unknown -> bad_info_reply(Tab, Item) end @@ -2022,15 +2016,26 @@ display_tab_info() -> MasterTabs = mnesia_recover:get_master_node_tables(), io:format("master node tables = ~p~n", [lists:sort(MasterTabs)]), + case get_backend_types() of + [] -> ok; + Ts -> list_backend_types(Ts, "backend types = ") + end, + + case get_index_plugins() of + [] -> ok; + Ps -> list_index_plugins(Ps, "index plugins = ") + end, + Tabs = system_info(tables), - {Unknown, Ram, Disc, DiscOnly} = - lists:foldl(fun storage_count/2, {[], [], [], []}, Tabs), + {Unknown, Ram, Disc, DiscOnly, Ext} = + lists:foldl(fun storage_count/2, {[], [], [], [], []}, Tabs), io:format("remote = ~p~n", [lists:sort(Unknown)]), io:format("ram_copies = ~p~n", [lists:sort(Ram)]), io:format("disc_copies = ~p~n", [lists:sort(Disc)]), io:format("disc_only_copies = ~p~n", [lists:sort(DiscOnly)]), + [io:format("~-19s= ~p~n", [atom_to_list(A), Ts]) || {A,Ts} <- Ext], Rfoldl = fun(T, Acc) -> Rpat = @@ -2038,7 +2043,7 @@ display_tab_info() -> read_only -> lists:sort([{A, read_only} || A <- val({T, active_replicas})]); read_write -> - table_info(T, where_to_commit) + [fix_wtc(W) || W <- table_info(T, where_to_commit)] end, case lists:keysearch(Rpat, 1, Acc) of {value, {_Rpat, Rtabs}} -> @@ -2051,12 +2056,60 @@ display_tab_info() -> Rdisp = fun({Rpat, Rtabs}) -> io:format("~p = ~p~n", [Rpat, Rtabs]) end, lists:foreach(Rdisp, lists:sort(Repl)). -storage_count(T, {U, R, D, DO}) -> +get_backend_types() -> + case ?catch_val({schema, user_property, mnesia_backend_types}) of + {'EXIT', _} -> + []; + {mnesia_backend_types, Ts} -> + lists:sort(Ts) + end. + +get_index_plugins() -> + case ?catch_val({schema, user_property, mnesia_index_plugins}) of + {'EXIT', _} -> + []; + {mnesia_index_plugins, Ps} -> + lists:sort(Ps) + end. + + +list_backend_types([{A,M} | T] = Ts, Legend) -> + Indent = [$\s || _ <- Legend], + W = integer_to_list( + lists:foldl(fun({Alias,_}, Wa) -> + erlang:max(Wa, length(atom_to_list(Alias))) + end, 0, Ts)), + io:fwrite(Legend ++ "~-" ++ W ++ "s - ~s~n", + [atom_to_list(A), atom_to_list(M)]), + [io:fwrite(Indent ++ "~-" ++ W ++ "s - ~s~n", + [atom_to_list(A1), atom_to_list(M1)]) + || {A1,M1} <- T]. + +list_index_plugins([{N,M,F} | T] = Ps, Legend) -> + Indent = [$\s || _ <- Legend], + W = integer_to_list( + lists:foldl(fun({N1,_,_}, Wa) -> + erlang:max(Wa, length(pp_ix_name(N1))) + end, 0, Ps)), + io:fwrite(Legend ++ "~-" ++ W ++ "s - ~s:~s~n", + [pp_ix_name(N), atom_to_list(M), atom_to_list(F)]), + [io:fwrite(Indent ++ "~-" ++ W ++ "s - ~s:~s~n", + [pp_ix_name(N1), atom_to_list(M1), atom_to_list(F1)]) + || {N1,M1,F1} <- T]. + +pp_ix_name(N) -> + lists:flatten(io_lib:fwrite("~w", [N])). + +fix_wtc({N, {ext,A,_}}) -> {N, A}; +fix_wtc({N,A}) when is_atom(A) -> {N, A}. + +storage_count(T, {U, R, D, DO, Ext}) -> case table_info(T, storage_type) of - unknown -> {[T | U], R, D, DO}; - ram_copies -> {U, [T | R], D, DO}; - disc_copies -> {U, R, [T | D], DO}; - disc_only_copies -> {U, R, D, [T | DO]} + unknown -> {[T | U], R, D, DO, Ext}; + ram_copies -> {U, [T | R], D, DO, Ext}; + disc_copies -> {U, R, [T | D], DO, Ext}; + disc_only_copies -> {U, R, D, [T | DO], Ext}; + {ext, A, _} -> {U, R, D, DO, orddict:append(A, T, Ext)} end. system_info(Item) -> @@ -2071,9 +2124,10 @@ system_info2(all) -> system_info2(db_nodes) -> DiscNs = ?catch_val({schema, disc_copies}), RamNs = ?catch_val({schema, ram_copies}), + ExtNs = ?catch_val({schema, external_copies}), if - is_list(DiscNs), is_list(RamNs) -> - DiscNs ++ RamNs; + is_list(DiscNs), is_list(RamNs), is_list(ExtNs) -> + DiscNs ++ RamNs ++ ExtNs; true -> case mnesia_schema:read_nodes() of {ok, Nodes} -> Nodes; @@ -2177,6 +2231,7 @@ system_info2(access_module) -> mnesia_monitor:get_env(access_module); system_info2(auto_repair) -> mnesia_monitor:get_env(auto_repair); system_info2(is_running) -> mnesia_lib:is_running(); system_info2(backup_module) -> mnesia_monitor:get_env(backup_module); +system_info2(backend_types) -> mnesia_schema:backend_types(); system_info2(event_module) -> mnesia_monitor:get_env(event_module); system_info2(debug) -> mnesia_monitor:get_env(debug); system_info2(dump_log_load_regulation) -> mnesia_monitor:get_env(dump_log_load_regulation); @@ -2213,6 +2268,7 @@ system_info_items(yes) -> [ access_module, auto_repair, + backend_types, backup_module, checkpoints, db_nodes, @@ -2306,11 +2362,17 @@ load_mnesia_or_abort() -> %% Database mgt create_schema(Ns) -> - mnesia_bup:create_schema(Ns). + create_schema(Ns, []). + +create_schema(Ns, Properties) -> + mnesia_bup:create_schema(Ns, Properties). delete_schema(Ns) -> mnesia_schema:delete_schema(Ns). +add_backend_type(Alias, Module) -> + mnesia_schema:add_backend_type(Alias, Module). + backup(Opaque) -> mnesia_log:backup(Opaque). diff --git a/lib/mnesia/src/mnesia.hrl b/lib/mnesia/src/mnesia.hrl index c58b4cfccd..0716dd87c8 100644 --- a/lib/mnesia/src/mnesia.hrl +++ b/lib/mnesia/src/mnesia.hrl @@ -68,6 +68,7 @@ ram_copies = [], % [Node] disc_copies = [], % [Node] disc_only_copies = [], % [Node] + external_copies = [], % [{{Alias,Mod},[Node]}] load_order = 0, % Integer access_mode = read_write, % read_write | read_only majority = false, % true | false @@ -103,7 +104,7 @@ ram_copies = [], disc_copies = [], disc_only_copies = [], - snmp = [], + ext = [], schema_ops = [] }). diff --git a/lib/mnesia/src/mnesia_backend_type.erl b/lib/mnesia/src/mnesia_backend_type.erl new file mode 100644 index 0000000000..4791b82a01 --- /dev/null +++ b/lib/mnesia/src/mnesia_backend_type.erl @@ -0,0 +1,115 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 1996-2015. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +%% +%% Behaviour definition for mnesia backend types. +%% + +%%%header_doc_include + +-module(mnesia_backend_type). + +-export([behaviour_info/1]). + +%%%header_doc_include + +%%%impl_doc_include + +%% Note that mnesia considers all callbacks mandatory!! +%% +behaviour_info(callbacks) -> + [ + {add_aliases, 1}, % (Aliases) -> ok + {check_definition, 4}, % (TypeAlias, Tab, Nodes, Properties) -> ok + {close_table, 2}, % (TypeAlias, Tab) -> ok + {create_table, 3}, % (TypeAlias, Tab, Properties) -> ok + {delete, 3}, % (TypeAlias, Tab, Key) -> true | ok + {delete_table, 2}, % (TypeAlias, Tab) -> ok + {first, 2}, % (TypeAlias, Tab) -> Key::Term | '$end_of_table' + {fixtable, 3}, % (TypeAlias, Tab, Bool) -> ok | true + {last, 2}, % (TypeAlias, Tab) -> Key::Term | '$end_of_table' + {index_is_consistent,3}, % (TypeAlias, IxTag, Bool) -> ok + {init_backend, 0}, % () -> ok + {info, 3}, % (TypeAlias, Tab, Item) -> Term + {insert, 3}, % (TypeAlias, Tab, Object) -> ok + {lookup, 3}, % (TypeAlias, Tab, Key) -> [Objects] + {is_index_consistent,2}, % (TypeAlias, IxTag) -> Bool + {load_table, 4}, % (TypeAlias, Tab, Reason, CsList) -> ok + {match_delete, 3}, % (TypeAlias, Tab, Pattern) -> ok + {next, 3}, % (TypeAlias, Tab, Key) -> Key::Term | '$end_of_table' + {prev, 3}, % (TypeAlias, Tab, Key) -> Key::Term | '$end_of_table' + {receiver_first_message, 4}, % (Sender, FirstMsg, Alias, Tab) -> {Size, State} + {receive_data, 5}, % (Data, Alias, Name, Sender, State) -> {more, State} | {{more, Msg}, State} + {receive_done, 4}, % (Alias, Tab, Sender, State) -> ok + {real_suffixes, 0}, % () -> [FileSuffix] + {remove_aliases, 1}, % (Aliases) -> ok + {repair_continuation, 2}, % (Continuation, MatchSpec) -> Continuation + {select, 1}, % (Continuation) -> {[Match], Continuation'} | '$end_of_table' + {select, 3}, % (TypeAlias, Tab, Pattern) -> {[Match], Continuation'} | '$end_of_table' + {select, 4}, % (TypeAlias, Tab, MatchSpec, Limit) {[Match], Continuation'} | '$end_of_table' + {sender_init, 4}, % (TypeAlias, Tab, LoadReason, Pid) -> + % {standard, Init(), Chunk()} | {Init(), Chunk()} + {semantics, 2}, % (TypeAlias, storage | types | index_fun | index_types) -> + % ram_copies | disc_copies, set | ordered_set | bag, fun(), ordered | bag + {slot, 3}, % (TypeAlias, Tab, Pos) -> '$end_of_table' | Objects | {error, Reason} + {sync_close_table, 2}, % (TypeAlias, Tab) -> ok + {tmp_suffixes, 0}, % () -> [FileSuffix] + {update_counter, 4}, % (TypeAlias, Tab, Counter, Val) -> NewVal + {validate_key, 6}, % (TypeAlias, Tab, RecName, Arity, Type, Key) -> {RecName, Arity, Type} + {validate_record, 6} % (TypeAlias, Tab, RecName, Arity, Type, Obj) -> {RecName, Arity, Type} + ]. + +%%%impl_doc_include + +%% -type tab() :: atom(). +%% -type alias() :: atom(). +%% -type rec_name() :: atom(). +%% -type type() :: set | bag | ordered_set. +%% -type proplist() :: [{atom(), any()}]. +%% -type key() :: any(). +%% -type db_object() :: tuple(). + +%% -type matchspec() :: ets:match_spec(). +%% -type limit() :: integer() | infinity. + +%% -type cont_fun() :: any(). +%% -type cont() :: '$end_of_table' | cont_fun(). + +%% -callback check_definition(alias(), tab(), [node()], proplist()) -> ok. +%% -callback create_table(alias(), tab(), proplist()) -> tab(). +%% -callback load_table(alias(), tab(), any()) -> ok. +%% -callback delete_table(alias(), tab()) -> ok. +%% -callback first(alias(), tab()) -> key(). +%% -callback last(alias(), tab()) -> key(). +%% -callback next(alias(), tab(), key()) -> key(). +%% -callback prev(alias(), tab(), key()) -> key(). +%% -callback insert(alias(), tab(), db_object()) -> ok. +%% -callback lookup(alias(), tab(), key()) -> [db_object()]. +%% -callback delete(alias(), tab(), key()) -> ok. +%% -callback update_counter(alias(), tab(), key(), integer()) -> integer(). +%% -callback select(cont()) -> {list(), cont()}. +%% -callback select(alias(), tab(), matchspec()) -> list() | '$end_of_table'. +%% -callback select(alias(), tab(), matchspec(), limit()) -> {list(), cont()}. +%% -callback slot(alias(), tab(), integer()) -> key(). +%% -callback validate_key(alias(), tab(), rec_name(), arity(), type(), key()) -> +%% {rec_name(), arity(), type()}. +%% -callback validate_record(alias(),tab(),rec_name(),arity(),type(),db_obj()) -> +%% {rec_name(), arity(), type()}. +%% -callback repair_continuation(cont(), matchspec()) -> cont(). diff --git a/lib/mnesia/src/mnesia_bup.erl b/lib/mnesia/src/mnesia_bup.erl index 8b1143a352..3e55deb958 100644 --- a/lib/mnesia/src/mnesia_bup.erl +++ b/lib/mnesia/src/mnesia_bup.erl @@ -28,6 +28,7 @@ fallback_exists/0, tm_fallback_start/1, create_schema/1, + create_schema/2, install_fallback/1, install_fallback/2, uninstall_fallback/0, @@ -46,7 +47,7 @@ uninstall_fallback_master/2, local_uninstall_fallback/2, do_traverse_backup/7, - trav_apply/4 + trav_apply/5 ]). -include("mnesia.hrl"). @@ -81,7 +82,8 @@ iterate(Mod, Fun, Opaque, Acc) -> R = #restore{bup_module = Mod, bup_data = Opaque}, try read_schema_section(R) of {R2, {Header, Schema, Rest}} -> - try iter(R2, Header, Schema, Fun, Acc, Rest) of + Ext = get_ext_types(Schema), + try iter(R2, Header, Schema, Ext, Fun, Acc, Rest) of {ok, R3, Res} -> close_read(R3), {ok, Res} @@ -96,17 +98,32 @@ iterate(Mod, Fun, Opaque, Acc) -> Err end. -iter(R, Header, Schema, Fun, Acc, []) -> +get_ext_types(Schema) -> + try + List = lookup_schema(schema, Schema), + case lists:keyfind(user_properties, 1, List) of + {_, Props} -> + proplists:get_value( + mnesia_backend_types, Props, []); + false -> + [] + end + catch + throw:{error, {"Cannot lookup",_}} -> + [] + end. + +iter(R, Header, Schema, Ext, Fun, Acc, []) -> case safe_apply(R, read, [R#restore.bup_data]) of {R2, []} -> - Res = Fun([], Header, Schema, Acc), + Res = Fun([], Header, Schema, Ext, Acc), {ok, R2, Res}; {R2, BupItems} -> - iter(R2, Header, Schema, Fun, Acc, BupItems) + iter(R2, Header, Schema, Ext, Fun, Acc, BupItems) end; -iter(R, Header, Schema, Fun, Acc, BupItems) -> - Acc2 = Fun(BupItems, Header, Schema, Acc), - iter(R, Header, Schema, Fun, Acc2, []). +iter(R, Header, Schema, Ext, Fun, Acc, BupItems) -> + Acc2 = Fun(BupItems, Header, Schema, Ext, Acc), + iter(R, Header, Schema, Ext, Fun, Acc2, []). -spec safe_apply(#restore{}, atom(), list()) -> tuple(). safe_apply(R, write, [_, Items]) when Items =:= [] -> @@ -262,7 +279,7 @@ convert_0_1(Schema) -> Cs = mnesia_schema:list2cs(List), convert_0_1(Schema2, [], Cs); false -> - List = mnesia_schema:get_initial_schema(disc_copies, [node()]), + List = mnesia_schema:get_initial_schema(disc_copies, [node()], []), Cs = mnesia_schema:list2cs(List), convert_0_1(Schema, [], Cs) end. @@ -310,16 +327,19 @@ schema2bup({schema, Tab, TableDef}) -> %% Create schema on the given nodes %% Requires that old schemas has been deleted %% Returns ok | {error, Reason} -create_schema([]) -> - create_schema([node()]); -create_schema(Ns) when is_list(Ns) -> +create_schema(Nodes) -> + create_schema(Nodes, []). + +create_schema([], Props) -> + create_schema([node()], Props); +create_schema(Ns, Props) when is_list(Ns), is_list(Props) -> case is_set(Ns) of true -> - create_schema(Ns, mnesia_schema:ensure_no_schema(Ns)); + create_schema(Ns, mnesia_schema:ensure_no_schema(Ns), Props); false -> {error, {combine_error, Ns}} end; -create_schema(Ns) -> +create_schema(Ns, _Props) -> {error, {badarg, Ns}}. is_set(List) when is_list(List) -> @@ -327,7 +347,7 @@ is_set(List) when is_list(List) -> is_set(_) -> false. -create_schema(Ns, ok) -> +create_schema(Ns, ok, Props) -> %% Ensure that we access the intended Mnesia %% directory. This function may not be called %% during startup since it will cause the @@ -346,7 +366,7 @@ create_schema(Ns, ok) -> Str = mk_str(), File = mnesia_lib:dir(Str), file:delete(File), - try make_initial_backup(Ns, File, Mod) of + try make_initial_backup(Ns, File, Mod, Props) of {ok, _Res} -> case do_install_fallback(File, Mod) of ok -> @@ -363,9 +383,9 @@ create_schema(Ns, ok) -> {error, Reason} -> {error, Reason} end; -create_schema(_Ns, {error, Reason}) -> +create_schema(_Ns, {error, Reason}, _) -> {error, Reason}; -create_schema(_Ns, Reason) -> +create_schema(_Ns, Reason, _) -> {error, Reason}. mk_str() -> @@ -373,7 +393,10 @@ mk_str() -> lists:concat([node()] ++ Now ++ ".TMP"). make_initial_backup(Ns, Opaque, Mod) -> - Orig = mnesia_schema:get_initial_schema(disc_copies, Ns), + make_initial_backup(Ns, Opaque, Mod, []). + +make_initial_backup(Ns, Opaque, Mod, Props) -> + Orig = mnesia_schema:get_initial_schema(disc_copies, Ns, Props), Modded = proplists:delete(storage_properties, proplists:delete(majority, Orig)), Schema = [{schema, schema, Modded}], O2 = do_apply(Mod, open_write, [Opaque], Opaque), @@ -486,15 +509,15 @@ install_fallback_master(ClientPid, FA) -> State = {start, FA}, Opaque = FA#fallback_args.opaque, Mod = FA#fallback_args.module, - Res = iterate(Mod, fun restore_recs/4, Opaque, State), + Res = iterate(Mod, fun restore_recs/5, Opaque, State), unlink(ClientPid), ClientPid ! {self(), Res}, exit(shutdown). -restore_recs(_, _, _, stop) -> +restore_recs(_, _, _, _, stop) -> throw({error, "restore_recs already stopped"}); -restore_recs(Recs, Header, Schema, {start, FA}) -> +restore_recs(Recs, Header, Schema, Ext, {start, FA}) -> %% No records in backup Schema2 = convert_schema(Header#log_header.log_version, Schema), CreateList = lookup_schema(schema, Schema2), @@ -505,19 +528,19 @@ restore_recs(Recs, Header, Schema, {start, FA}) -> Args = [self(), FA], Pids = [spawn_link(N, ?MODULE, fallback_receiver, Args) || N <- Ns], send_fallback(Pids, {start, Header, Schema2}), - Res = restore_recs(Recs, Header, Schema2, Pids), + Res = restore_recs(Recs, Header, Schema2, Ext, Pids), global:del_lock({{mnesia_table_lock, schema}, self()}, Ns), Res catch _:Reason -> throw({error, {"Bad schema in restore_recs", Reason}}) end; -restore_recs([], _Header, _Schema, Pids) -> +restore_recs([], _Header, _Schema, _Ext, Pids) -> send_fallback(Pids, swap), send_fallback(Pids, stop), stop; -restore_recs(Recs, _, _, Pids) -> +restore_recs(Recs, _, _, _, Pids) -> send_fallback(Pids, {records, Recs}), Pids. @@ -716,7 +739,7 @@ do_fallback_start(true, false) -> BupFile = fallback_bup(), Mod = mnesia_backup, LocalTabs = ?ets_new_table(mnesia_local_tables, [set, public, {keypos, 2}]), - case iterate(Mod, fun restore_tables/4, BupFile, {start, LocalTabs}) of + case iterate(Mod, fun restore_tables/5, BupFile, {start, LocalTabs}) of {ok, _Res} -> ?SAFE(dets:close(schema)), TmpSchema = mnesia_lib:tab2tmp(schema), @@ -737,23 +760,24 @@ do_fallback_start(true, false) -> {error, {"Cannot start from fallback", Reason}} end. -restore_tables(All=[Rec | Recs], Header, Schema, State={local, LocalTabs, LT}) -> +restore_tables(All=[Rec | Recs], Header, Schema, Ext, + State={local, LocalTabs, LT}) -> Tab = element(1, Rec), if Tab =:= LT#local_tab.name -> Key = element(2, Rec), (LT#local_tab.add)(Tab, Key, Rec, LT), - restore_tables(Recs, Header, Schema, State); + restore_tables(Recs, Header, Schema, Ext, State); true -> NewState = {new, LocalTabs}, - restore_tables(All, Header, Schema, NewState) + restore_tables(All, Header, Schema, Ext, NewState) end; -restore_tables(All=[Rec | Recs], Header, Schema, {new, LocalTabs}) -> +restore_tables(All=[Rec | Recs], Header, Schema, Ext, {new, LocalTabs}) -> Tab = element(1, Rec), case ?ets_lookup(LocalTabs, Tab) of [] -> State = {not_local, LocalTabs, Tab}, - restore_tables(Recs, Header, Schema, State); + restore_tables(Recs, Header, Schema, Ext, State); [LT] when is_record(LT, local_tab) -> State = {local, LocalTabs, LT}, case LT#local_tab.opened of @@ -762,38 +786,39 @@ restore_tables(All=[Rec | Recs], Header, Schema, {new, LocalTabs}) -> (LT#local_tab.open)(Tab, LT), ?ets_insert(LocalTabs,LT#local_tab{opened=true}) end, - restore_tables(All, Header, Schema, State) + restore_tables(All, Header, Schema, Ext, State) end; -restore_tables(All=[Rec | Recs], Header, Schema, S = {not_local, LocalTabs, PrevTab}) -> +restore_tables(All=[Rec | Recs], Header, Schema, Ext, + S = {not_local, LocalTabs, PrevTab}) -> Tab = element(1, Rec), if Tab =:= PrevTab -> - restore_tables(Recs, Header, Schema, S); + restore_tables(Recs, Header, Schema, Ext, S); true -> State = {new, LocalTabs}, - restore_tables(All, Header, Schema, State) + restore_tables(All, Header, Schema, Ext, State) end; -restore_tables(Recs, Header, Schema, {start, LocalTabs}) -> +restore_tables(Recs, Header, Schema, Ext, {start, LocalTabs}) -> Dir = mnesia_lib:dir(), OldDir = filename:join([Dir, "OLD_DIR"]), mnesia_schema:purge_dir(OldDir, []), mnesia_schema:purge_dir(Dir, [fallback_name()]), - init_dat_files(Schema, LocalTabs), + init_dat_files(Schema, Ext, LocalTabs), State = {new, LocalTabs}, - restore_tables(Recs, Header, Schema, State); -restore_tables([], _Header, _Schema, State) -> + restore_tables(Recs, Header, Schema, Ext, State); +restore_tables([], _Header, _Schema, _Ext, State) -> State. %% Creates all neccessary dat files and inserts %% the table definitions in the schema table %% %% Returns a list of local_tab tuples for all local tables -init_dat_files(Schema, LocalTabs) -> +init_dat_files(Schema, Ext, LocalTabs) -> TmpFile = mnesia_lib:tab2tmp(schema), Args = [{file, TmpFile}, {keypos, 2}, {type, set}], case dets:open_file(schema, Args) of % Assume schema lock {ok, _} -> - create_dat_files(Schema, LocalTabs), + create_dat_files(Schema, Ext, LocalTabs), ok = dets:close(schema), LocalTab = #local_tab{name = schema, storage_type = disc_copies, @@ -808,10 +833,10 @@ init_dat_files(Schema, LocalTabs) -> throw({error, {"Cannot open file", schema, Args, Reason}}) end. -create_dat_files([{schema, schema, TabDef} | Tail], LocalTabs) -> +create_dat_files([{schema, schema, TabDef} | Tail], Ext, LocalTabs) -> ok = dets:insert(schema, {schema, schema, TabDef}), - create_dat_files(Tail, LocalTabs); -create_dat_files([{schema, Tab, TabDef} | Tail], LocalTabs) -> + create_dat_files(Tail, Ext, LocalTabs); +create_dat_files([{schema, Tab, TabDef} | Tail], Ext, LocalTabs) -> TmpFile = mnesia_lib:tab2tmp(Tab), DatFile = mnesia_lib:tab2dat(Tab), DclFile = mnesia_lib:tab2dcl(Tab), @@ -824,56 +849,21 @@ create_dat_files([{schema, Tab, TabDef} | Tail], LocalTabs) -> mnesia_lib:dets_sync_close(Tab), file:delete(TmpFile), - Cs = mnesia_schema:list2cs(TabDef), + Cs = mnesia_schema:list2cs(TabDef, Ext), ok = dets:insert(schema, {schema, Tab, TabDef}), RecName = Cs#cstruct.record_name, Storage = mnesia_lib:cs_to_storage_type(node(), Cs), + delete_ext(Storage, Tab), + Semantics = mnesia_lib:semantics(Storage, storage), if - Storage =:= unknown -> + Semantics =:= undefined -> ok = dets:delete(schema, {schema, Tab}), - create_dat_files(Tail, LocalTabs); - Storage =:= disc_only_copies -> - Args = [{file, TmpFile}, {keypos, 2}, - {type, mnesia_lib:disk_type(Tab, Cs#cstruct.type)}], - Open = fun(T, LT) when T =:= LT#local_tab.name -> - case mnesia_lib:dets_sync_open(T, Args) of - {ok, _} -> - ok; - {error, Reason} -> - throw({error, {"Cannot open file", T, Args, Reason}}) - end - end, - Add = fun(T, Key, Rec, LT) when T =:= LT#local_tab.name -> - case Rec of - {_T, Key} -> - ok = dets:delete(T, Key); - (Rec) when T =:= RecName -> - ok = dets:insert(Tab, Rec); - (Rec) -> - Rec2 = setelement(1, Rec, RecName), - ok = dets:insert(T, Rec2) - end - end, - Close = fun(T, LT) when T =:= LT#local_tab.name -> - mnesia_lib:dets_sync_close(T) - end, - Swap = fun(T, LT) when T =:= LT#local_tab.name -> - Expunge(), - case LT#local_tab.opened of - true -> - Close(T,LT); - false -> - Open(T,LT), - Close(T,LT) - end, - case file:rename(TmpFile, DatFile) of - ok -> - ok; - {error, Reason} -> - mnesia_lib:fatal("Cannot rename file ~p -> ~p: ~p~n", - [TmpFile, DatFile, Reason]) - end - end, + create_dat_files(Tail, Ext, LocalTabs); + Semantics =:= disc_only_copies -> + Open = disc_only_open_fun(Storage, Cs), + Add = disc_only_add_fun(Storage, Cs), + Close = disc_only_close_fun(Storage), + Swap = disc_only_swap_fun(Storage, Expunge, Open, Close), LocalTab = #local_tab{name = Tab, storage_type = Storage, open = Open, @@ -883,8 +873,8 @@ create_dat_files([{schema, Tab, TabDef} | Tail], LocalTabs) -> record_name = RecName, opened = false}, ?ets_insert(LocalTabs, LocalTab), - create_dat_files(Tail, LocalTabs); - Storage =:= ram_copies; Storage =:= disc_copies -> + create_dat_files(Tail, Ext, LocalTabs); + Semantics =:= ram_copies; Storage =:= disc_copies -> Open = fun(T, LT) when T =:= LT#local_tab.name -> mnesia_log:open_log({?MODULE, T}, mnesia_log:dcl_log_header(), @@ -945,18 +935,97 @@ create_dat_files([{schema, Tab, TabDef} | Tail], LocalTabs) -> opened = false }, ?ets_insert(LocalTabs, LocalTab), - create_dat_files(Tail, LocalTabs) + create_dat_files(Tail, Ext, LocalTabs); + true -> + error({unknown_semantics, [{semantics, Semantics}, + {tabdef, TabDef}, + {ext, Ext}]}) end; -create_dat_files([{schema, Tab} | Tail], LocalTabs) -> +create_dat_files([{schema, Tab} | Tail], Ext, LocalTabs) -> ?ets_delete(LocalTabs, Tab), ok = dets:delete(schema, {schema, Tab}), TmpFile = mnesia_lib:tab2tmp(Tab), mnesia_lib:dets_sync_close(Tab), file:delete(TmpFile), - create_dat_files(Tail, LocalTabs); -create_dat_files([], _LocalTabs) -> + create_dat_files(Tail, Ext, LocalTabs); +create_dat_files([], _Ext, _LocalTabs) -> ok. +delete_ext({ext, Alias, Mod}, Tab) -> + Mod:close_table(Alias, Tab), + Mod:delete_table(Alias, Tab), + ok; +delete_ext(_, _) -> + ok. + + +disc_only_open_fun(disc_only_copies, #cstruct{name = Tab} =Cs) -> + TmpFile = mnesia_lib:tab2tmp(Tab), + Args = [{file, TmpFile}, {keypos, 2}, + {type, mnesia_lib:disk_type(Tab, Cs#cstruct.type)}], + fun(T, LT) when T =:= LT#local_tab.name -> + case mnesia_lib:dets_sync_open(T, Args) of + {ok, _} -> + ok; + {error, Reason} -> + throw({error, {"Cannot open file", T, Args, Reason}}) + end + end; +disc_only_open_fun({ext,Alias,Mod}, Cs) -> + fun(T, LT) when T =:= LT#local_tab.name -> + ok = Mod:load_table(Alias, T, restore, mnesia_schema:cs2list(Cs)) + end. + +disc_only_add_fun(Storage, #cstruct{name = Tab, + record_name = RecName}) -> + fun(T, Key, Rec, #local_tab{name = T}) when T =:= Tab-> + case Rec of + {_T, Key} -> + ok = mnesia_lib:db_erase(Storage, T, Key); + (Rec) when T =:= RecName -> + ok = mnesia_lib:db_put(Storage, T, Rec); + (Rec) -> + ok = mnesia_lib:db_put(Storage, T, + setelement(1, Rec, RecName)) + end + end. + +disc_only_close_fun(disc_only_copies) -> + fun(T, LT) when T =:= LT#local_tab.name -> + mnesia_lib:dets_sync_close(T) + end; +disc_only_close_fun({ext, Alias, Mod}) -> + fun(T, _LT) -> + Mod:sync_close_table(Alias, T) + end. + + +disc_only_swap_fun(disc_only_copies, Expunge, Open, Close) -> + fun(T, LT) when T =:= LT#local_tab.name -> + TmpFile = mnesia_lib:tab2tmp(T), + DatFile = mnesia_lib:tab2dat(T), + Expunge(), + case LT#local_tab.opened of + true -> + Close(T,LT); + false -> + Open(T,LT), + Close(T,LT) + end, + case file:rename(TmpFile, DatFile) of + ok -> + ok; + {error, Reason} -> + mnesia_lib:fatal("Cannot rename file ~p -> ~p: ~p~n", + [TmpFile, DatFile, Reason]) + end + end; +disc_only_swap_fun({ext, _Alias, _Mod}, _Expunge, _Open, Close) -> + fun(T, #local_tab{name = T} = LT) -> + Close(T, LT) + end. + + uninstall_fallback() -> uninstall_fallback([{scope, global}]). @@ -1133,7 +1202,7 @@ do_traverse_backup(ClientPid, Source, SourceMod, Target, TargetMod, Fun, Acc) -> end, A = {start, Fun, Acc, TargetMod, Iter}, Res = - case iterate(SourceMod, fun trav_apply/4, Source, A) of + case iterate(SourceMod, fun trav_apply/5, Source, A) of {ok, {iter, _, Acc2, _, Iter2}} when TargetMod =/= read_only -> try do_apply(TargetMod, commit_write, [Iter2], Iter2), @@ -1152,7 +1221,7 @@ do_traverse_backup(ClientPid, Source, SourceMod, Target, TargetMod, Fun, Acc) -> unlink(ClientPid), ClientPid ! {iter_done, self(), Res}. -trav_apply(Recs, _Header, _Schema, {iter, Fun, Acc, Mod, Iter}) -> +trav_apply(Recs, _Header, _Schema, _Ext, {iter, Fun, Acc, Mod, Iter}) -> {NewRecs, Acc2} = filter_foldl(Fun, Acc, Recs), if Mod =/= read_only, NewRecs =/= [] -> @@ -1161,7 +1230,7 @@ trav_apply(Recs, _Header, _Schema, {iter, Fun, Acc, Mod, Iter}) -> true -> {iter, Fun, Acc2, Mod, Iter} end; -trav_apply(Recs, Header, Schema, {start, Fun, Acc, Mod, Iter}) -> +trav_apply(Recs, Header, Schema, Ext, {start, Fun, Acc, Mod, Iter}) -> Iter2 = if Mod =/= read_only -> @@ -1169,8 +1238,9 @@ trav_apply(Recs, Header, Schema, {start, Fun, Acc, Mod, Iter}) -> true -> Iter end, - TravAcc = trav_apply(Schema, Header, Schema, {iter, Fun, Acc, Mod, Iter2}), - trav_apply(Recs, Header, Schema, TravAcc). + TravAcc = trav_apply(Schema, Header, Schema, Ext, + {iter, Fun, Acc, Mod, Iter2}), + trav_apply(Recs, Header, Schema, Ext, TravAcc). filter_foldl(Fun, Acc, [Head|Tail]) -> case Fun(Head, Acc) of diff --git a/lib/mnesia/src/mnesia_checkpoint.erl b/lib/mnesia/src/mnesia_checkpoint.erl index 33b28a2f29..9eb939e8d3 100644 --- a/lib/mnesia/src/mnesia_checkpoint.erl +++ b/lib/mnesia/src/mnesia_checkpoint.erl @@ -675,6 +675,16 @@ tab2retainer({Tab, Name}) -> FlatName = lists:flatten(io_lib:write(Name)), mnesia_lib:dir(lists:concat([?MODULE, "_", Tab, "_", FlatName, ".RET"])). +retainer_create(_Cp, R, Tab, Name, Ext = {ext, Alias, Mod}) -> + T = {Tab, retainer, Name}, + P = mnesia_schema:cs2list(val({Tab, cstruct})), + Mod:delete_table(Alias, T), + ok = Mod:create_table(Alias, T, P), + Cs = val({Tab, cstruct}), + Mod:load_table(Alias, T, {retainer, create_table}, + mnesia_schema:cs2list(Cs)), + dbg_out("Checkpoint retainer created ~p ~p~n", [Name, Tab]), + R#retainer{store = {Ext, T}, really_retain = true}; retainer_create(_Cp, R, Tab, Name, disc_only_copies) -> Fname = tab2retainer({Tab, Name}), file:delete(Fname), @@ -734,15 +744,23 @@ traverse_dcd({Cont, Recs}, Log, Fun) -> %% trashed data?? traverse_dcd(eof, _Log, _Fun) -> ok. +retainer_get({{ext, Alias, Mod}, Store}, Key) -> + Mod:lookup(Alias, Store, Key); retainer_get({ets, Store}, Key) -> ?ets_lookup(Store, Key); retainer_get({dets, Store}, Key) -> dets:lookup(Store, Key). +retainer_put({{ext, Alias, Mod}, Store}, Val) -> + Mod:insert(Alias, Store, Val); retainer_put({ets, Store}, Val) -> ?ets_insert(Store, Val); retainer_put({dets, Store}, Val) -> dets:insert(Store, Val). +retainer_first({{ext, Alias, Mod}, Store}) -> + Mod:first(Alias, Store); retainer_first({ets, Store}) -> ?ets_first(Store); retainer_first({dets, Store}) -> dets:first(Store). +retainer_next({{ext, Alias, Mod}, Store}, Key) -> + Mod:next(Alias, Store, Key); retainer_next({ets, Store}, Key) -> ?ets_next(Store, Key); retainer_next({dets, Store}, Key) -> dets:next(Store, Key). @@ -761,11 +779,16 @@ retainer_next({dets, Store}, Key) -> dets:next(Store, Key). retainer_fixtable(Tab, Bool) when is_atom(Tab) -> mnesia_lib:db_fixtable(val({Tab, storage_type}), Tab, Bool); +retainer_fixtable({Ext = {ext, _, _}, Tab}, Bool) -> + mnesia_lib:db_fixtable(Ext, Tab, Bool); retainer_fixtable({ets, Tab}, Bool) -> mnesia_lib:db_fixtable(ram_copies, Tab, Bool); retainer_fixtable({dets, Tab}, Bool) -> mnesia_lib:db_fixtable(disc_only_copies, Tab, Bool). +retainer_delete({{ext, Alias, Mod}, Store}) -> + Mod:close_table(Alias, Store), + Mod:delete_table(Alias, Store); retainer_delete({ets, Store}) -> ?ets_delete_table(Store); retainer_delete({dets, Store}) -> diff --git a/lib/mnesia/src/mnesia_controller.erl b/lib/mnesia/src/mnesia_controller.erl index 58cc66e18b..4791e2e290 100644 --- a/lib/mnesia/src/mnesia_controller.erl +++ b/lib/mnesia/src/mnesia_controller.erl @@ -385,6 +385,8 @@ force_load_table(Tab) when is_atom(Tab), Tab /= schema -> do_force_load_table(Tab); disc_only_copies -> do_force_load_table(Tab); + {ext, _, _} -> + do_force_load_table(Tab); unknown -> set({Tab, load_by_force}, true), cast({force_load_updated, Tab}), @@ -1533,8 +1535,8 @@ update_whereabouts(Tab, Node, State) -> Storage == unknown -> %% No own copy, continue to read remotely add_active_replica(Tab, Node), - NodeST = mnesia_lib:storage_type_at_node(Node, Tab), - ReadST = mnesia_lib:storage_type_at_node(Read, Tab), + NodeST = mnesia_lib:semantics(mnesia_lib:storage_type_at_node(Node, Tab), storage), + ReadST = mnesia_lib:semantics(mnesia_lib:storage_type_at_node(Read, Tab), storage), if %% Avoid reading from disc_only_copies NodeST == disc_only_copies -> ignore; @@ -1588,6 +1590,7 @@ last_consistent_replica(Tab, Downs) -> Ram = Cs#cstruct.ram_copies, Disc = Cs#cstruct.disc_copies, DiscOnly = Cs#cstruct.disc_only_copies, + Ext = Cs#cstruct.external_copies, BetterCopies0 = mnesia_lib:remote_copy_holders(Cs) -- Downs, BetterCopies = BetterCopies0 -- Ram, AccessMode = Cs#cstruct.access_mode, @@ -1620,7 +1623,7 @@ last_consistent_replica(Tab, Downs) -> false; Storage == ram_copies -> if - Disc == [], DiscOnly == [] -> + Disc == [], DiscOnly == [], Ext == [] -> %% Nobody has copy on disc {true, {Tab, ram_only}}; true -> @@ -1865,6 +1868,11 @@ info([Tab | Tail]) -> dets:info(Tab, size), dets:info(Tab, file_size), "bytes on disc"); + {ext, Alias, Mod} -> + info_format(Tab, + Mod:info(Alias, Tab, size), + Mod:info(Alias, Tab, memory), + "words of mem"); _ -> info_format(Tab, ?ets_info(Tab, size), diff --git a/lib/mnesia/src/mnesia_dumper.erl b/lib/mnesia/src/mnesia_dumper.erl index 7418ab7436..eb02a585a6 100644 --- a/lib/mnesia/src/mnesia_dumper.erl +++ b/lib/mnesia/src/mnesia_dumper.erl @@ -38,6 +38,8 @@ needs_dump_ets/1, raw_dump_table/2, raw_named_dump_table/2, + dump_to_logfile/2, + load_from_logfile/3, start_regulator/0, opt_dump_log/1, update/3, @@ -227,11 +229,11 @@ insert_rec(Rec, InPlace, InitBy, LogV) when is_record(Rec, commit) -> D = Rec#commit.decision, case mnesia_recover:wait_for_decision(D, InitBy) of {Tid, committed} -> - do_insert_rec(Tid, Rec, InPlace, InitBy, LogV); + do_insert_rec(Tid, mnesia_tm:new_cr_format(Rec), InPlace, InitBy, LogV); {Tid, aborted} -> case InitBy of startup -> - mnesia_schema:undo_prepare_commit(Tid, Rec); + mnesia_schema:undo_prepare_commit(Tid, mnesia_tm:new_cr_format(Rec)); _ -> ok end @@ -271,15 +273,30 @@ do_insert_rec(Tid, Rec, InPlace, InitBy, LogV) -> end end, D = Rec#commit.disc_copies, + ExtOps = commit_ext(Rec), insert_ops(Tid, disc_copies, D, InPlace, InitBy, LogV), + [insert_ops(Tid, Ext, Ops, InPlace, InitBy, LogV) || + {Ext, Ops} <- ExtOps, + storage_semantics(Ext) == disc_copies], case InitBy of startup -> DO = Rec#commit.disc_only_copies, - insert_ops(Tid, disc_only_copies, DO, InPlace, InitBy, LogV); + insert_ops(Tid, disc_only_copies, DO, InPlace, InitBy, LogV), + [insert_ops(Tid, Ext, Ops, InPlace, InitBy, LogV) || + {Ext, Ops} <- ExtOps, storage_semantics(Ext) == disc_only_copies]; _ -> ignore end. +commit_ext(#commit{ext = []}) -> []; +commit_ext(#commit{ext = Ext}) -> + case lists:keyfind(ext_copies, 1, Ext) of + {_, C} -> + lists:foldl(fun({Ext0, Op}, D) -> + orddict:append(Ext0, Op, D) + end, orddict:new(), C); + false -> [] + end. update(_Tid, [], _DumperMode) -> dumped; @@ -330,14 +347,15 @@ insert_ops(Tid, Storage, [Op | Ops], InPlace, InitBy, Ver) when Ver < "4.3" -> %% Normal ops disc_insert(_Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy) -> - case open_files(Tab, Storage, InPlace, InitBy) of + Semantics = storage_semantics(Storage), + case open_files(Tab, Semantics, Storage, InPlace, InitBy) of true -> - case Storage of + case Semantics of disc_copies when Tab /= schema -> mnesia_log:append({?MODULE,Tab}, {{Tab, Key}, Val, Op}), ok; _ -> - dets_insert(Op,Tab,Key,Val) + dets_insert(Op,Tab,Key,Val,Storage) end; false -> ignore @@ -349,34 +367,37 @@ disc_insert(_Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy) -> %% Otherwise we will get a double increment. %% This is perfect but update_counter is a dirty op. -dets_insert(Op,Tab,Key,Val) -> +dets_insert(Op,Tab,Key,Val, Storage0) -> + Storage = if Tab == schema -> disc_only_copies; + true -> Storage0 + end, case Op of write -> dets_updated(Tab,Key), - ok = dets:insert(Tab, Val); + ok = mnesia_lib:db_put(Storage, Tab, Val); delete -> dets_updated(Tab,Key), - ok = dets:delete(Tab, Key); + ok = mnesia_lib:db_erase(Storage, Tab, Key); update_counter -> case dets_incr_counter(Tab,Key) of true -> {RecName, Incr} = Val, - try _ = dets:update_counter(Tab, Key, Incr) + try _ = mnesia_lib:db_update_counter(Storage, Tab, Key, Incr) catch error:_ when Incr < 0 -> Zero = {RecName, Key, 0}, - ok = dets:insert(Tab, Zero); + ok = mnesia_lib:db_put(Storage, Tab, Zero); error:_ -> Init = {RecName, Key, Incr}, - ok = dets:insert(Tab, Init) + ok = mnesia_lib:db_put(Storage, Tab, Init) end; false -> ok end; delete_object -> dets_updated(Tab,Key), - ok = dets:delete_object(Tab, Val); + mnesia_lib:db_match_erase(Storage, Tab, Val); clear_table -> dets_cleared(Tab), - ok = dets:delete_all_objects(Tab) + ok = mnesia_lib:db_match_erase(Storage, Tab, '_') end. dets_updated(Tab,Key) -> @@ -431,23 +452,29 @@ insert(_Tid, _Storage, _Tab, _Key, [], _Op, _InPlace, _InitBy) -> ok; insert(Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy) -> + Semantics = storage_semantics(Storage), Item = {{Tab, Key}, Val, Op}, case InitBy of startup -> disc_insert(Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy); - _ when Storage == ram_copies -> + _ when Semantics == ram_copies -> mnesia_tm:do_update_op(Tid, Storage, Item), Snmp = mnesia_tm:prepare_snmp(Tab, Key, [Item]), mnesia_tm:do_snmp(Tid, Snmp); - _ when Storage == disc_copies -> + _ when Semantics == disc_copies -> disc_insert(Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy), mnesia_tm:do_update_op(Tid, Storage, Item), Snmp = mnesia_tm:prepare_snmp(Tab, Key, [Item]), mnesia_tm:do_snmp(Tid, Snmp); - _ when Storage == disc_only_copies -> + _ when Semantics == disc_only_copies -> + mnesia_tm:do_update_op(Tid, Storage, Item), + Snmp = mnesia_tm:prepare_snmp(Tab, Key, [Item]), + mnesia_tm:do_snmp(Tid, Snmp); + + _ when element(1, Storage) == ext -> mnesia_tm:do_update_op(Tid, Storage, Item), Snmp = mnesia_tm:prepare_snmp(Tab, Key, [Item]), mnesia_tm:do_snmp(Tid, Snmp); @@ -456,6 +483,9 @@ insert(Tid, Storage, Tab, Key, Val, Op, InPlace, InitBy) -> ignore end. +disc_delete_table(Tab, {ext, Alias, Mod}) -> + Mod:close_table(Alias, Tab), + Mod:delete_table(Alias, Tab); disc_delete_table(Tab, Storage) -> case mnesia_monitor:use_dir() of true -> @@ -485,11 +515,14 @@ disc_delete_table(Tab, Storage) -> ok end. -disc_delete_indecies(_Tab, _Cs, Storage) when Storage /= disc_only_copies -> - ok; -disc_delete_indecies(Tab, Cs, disc_only_copies) -> - Indecies = Cs#cstruct.index, - mnesia_index:del_transient(Tab, Indecies, disc_only_copies). +disc_delete_indecies(Tab, Cs, Storage) -> + case storage_semantics(Storage) of + disc_only_copies -> + Indecies = Cs#cstruct.index, + mnesia_index:del_transient(Tab, Indecies, Storage); + _ -> + ok + end. insert_op(Tid, Storage, {{Tab, Key}, Val, Op}, InPlace, InitBy) -> %% Propagate to disc only @@ -515,6 +548,8 @@ insert_op(Tid, _, {op, change_table_copy_type, N, FromS, ToS, TabDef}, InPlace, Cs = mnesia_schema:list2cs(TabDef), Val = mnesia_schema:insert_cstruct(Tid, Cs, true), % Update ram only {schema, Tab, _} = Val, + FromSem = storage_semantics(FromS), + ToSem = storage_semantics(ToS), case lists:member(N, val({current, db_nodes})) of true when InitBy /= startup -> mnesia_controller:add_active_replica(Tab, N, Cs); @@ -527,62 +562,118 @@ insert_op(Tid, _, {op, change_table_copy_type, N, FromS, ToS, TabDef}, InPlace, Dat = mnesia_lib:tab2dat(Tab), Dcd = mnesia_lib:tab2dcd(Tab), Dcl = mnesia_lib:tab2dcl(Tab), + Logtmp = mnesia_lib:tab2logtmp(Tab), case {FromS, ToS} of - {ram_copies, disc_copies} when Tab == schema -> - ok = ensure_rename(Dmp, Dat); - {ram_copies, disc_copies} -> - file:delete(Dcl), - ok = ensure_rename(Dmp, Dcd); - {disc_copies, ram_copies} when Tab == schema -> - mnesia_lib:set(use_dir, false), - mnesia_monitor:unsafe_close_dets(Tab), - ok = file:delete(Dat); - {disc_copies, ram_copies} -> - _ = file:delete(Dcl), - _ = file:delete(Dcd), - ok; - {ram_copies, disc_only_copies} -> - ok = ensure_rename(Dmp, Dat), - true = open_files(Tab, disc_only_copies, InPlace, InitBy), - %% ram_delete_table must be done before init_indecies, - %% it uses info which is reset in init_indecies, - %% it doesn't matter, because init_indecies don't use - %% the ram replica of the table when creating the disc - %% index; Could be improved :) - mnesia_schema:ram_delete_table(Tab, FromS), - PosList = Cs#cstruct.index, - mnesia_index:init_indecies(Tab, disc_only_copies, PosList); - {disc_only_copies, ram_copies} -> - mnesia_monitor:unsafe_close_dets(Tab), - disc_delete_indecies(Tab, Cs, disc_only_copies), - case InitBy of - startup -> - ignore; + {{ext,_FromAlias,_FromMod},{ext,ToAlias,ToMod}} -> + disc_delete_table(Tab, FromS), + ok = ToMod:delete_table(ToAlias, Tab), + ok = mnesia_monitor:unsafe_create_external(Tab, ToAlias, ToMod, TabDef), + ok = ToMod:load_table(ToAlias, Tab, {dumper,change_table_copy_type}, TabDef), + ok = load_from_logfile(ToS, Tab, Logtmp), + file:delete(Logtmp), + restore_indexes(Tab, ToS, Cs); + + {_,{ext,ToAlias,ToMod}} -> + case FromSem of + ram_copies -> + mnesia_schema:ram_delete_table(Tab, FromS); _ -> - mnesia_controller:get_disc_copy(Tab), - ok + if FromSem == disc_copies -> + mnesia_schema:ram_delete_table( + Tab, FromS); + true -> ok + end, + disc_delete_table(Tab, FromS) end, - disc_delete_table(Tab, disc_only_copies); - {disc_copies, disc_only_copies} -> - ok = ensure_rename(Dmp, Dat), - true = open_files(Tab, disc_only_copies, InPlace, InitBy), - mnesia_schema:ram_delete_table(Tab, FromS), - PosList = Cs#cstruct.index, - mnesia_index:init_indecies(Tab, disc_only_copies, PosList), - _ = file:delete(Dcl), - _ = file:delete(Dcd), - ok; - {disc_only_copies, disc_copies} -> - mnesia_monitor:unsafe_close_dets(Tab), - disc_delete_indecies(Tab, Cs, disc_only_copies), - case InitBy of - startup -> - ignore; - _ -> - mnesia_log:ets2dcd(Tab), - mnesia_controller:get_disc_copy(Tab), - disc_delete_table(Tab, disc_only_copies) - end + + ok = ToMod:delete_table(ToAlias, Tab), + ok = mnesia_monitor:unsafe_create_external(Tab, ToAlias, ToMod, TabDef), + ok = ToMod:load_table(ToAlias, Tab, {dumper,change_table_copy_type}, TabDef), + ok = load_from_logfile(ToS, Tab, Logtmp), + file:delete(Logtmp), + restore_indexes(Tab, ToS, Cs); + + {{ext,_FromAlias,_FromMod} = FromS, ToS} -> + disc_delete_table(Tab, FromS), + case ToS of + ram_copies -> + change_disc_to_ram( + Tab, Cs, FromS, ToS, Logtmp, InitBy); + disc_copies -> + Args = [{keypos, 2}, public, named_table, + Cs#cstruct.type], + mnesia_monitor:mktab(Tab, Args), + ok = load_from_logfile(ToS, Tab, Logtmp), + file:delete(Logtmp); + disc_only_copies -> + %% ok = ensure_rename(Dmp, Dat), + true = open_files(Tab, ToS, InPlace, InitBy), + ok = load_from_logfile(ToS, Tab, Logtmp), + file:delete(Logtmp) + end, + restore_indexes(Tab, ToS, Cs); + + _NoneAreExt -> + + case {FromSem, ToSem} of + {ram_copies, disc_copies} when Tab == schema -> + ok = ensure_rename(Dmp, Dat); + {ram_copies, disc_copies} -> + file:delete(Dcl), + ok = ensure_rename(Dmp, Dcd); + {disc_copies, ram_copies} when Tab == schema -> + mnesia_lib:set(use_dir, false), + mnesia_monitor:unsafe_close_dets(Tab), + ok = file:delete(Dat); + {disc_copies, ram_copies} -> + _ = file:delete(Dcl), + _ = file:delete(Dcd), + ok; + {ram_copies, disc_only_copies} -> + ok = ensure_rename(Dmp, Dat), + true = open_files(Tab, ToS, InPlace, InitBy), + %% ram_delete_table must be done before + %% init_indecies, it uses info which is reset + %% in init_indecies, it doesn't matter, because + %% init_indecies don't use the ram replica of + %% the table when creating the disc index; + %% Could be improved :) + mnesia_schema:ram_delete_table(Tab, FromS), + restore_indexes(Tab, ToS, Cs); + {disc_only_copies, ram_copies} when FromS == disc_only_copies -> + mnesia_monitor:unsafe_close_dets(Tab), + disc_delete_indecies(Tab, Cs, FromS), + case InitBy of + startup -> + ignore; + _ -> + mnesia_controller:get_disc_copy(Tab), + ok + end, + disc_delete_table(Tab, FromS); + {disc_only_copies, ram_copies} when element(1, FromS) == ext -> + change_disc_to_ram( + Tab, Cs, FromS, ToS, Logtmp, InitBy); + {disc_copies, disc_only_copies} -> + ok = ensure_rename(Dmp, Dat), + true = open_files(Tab, ToS, InPlace, InitBy), + mnesia_schema:ram_delete_table(Tab, FromS), + restore_indexes(Tab, ToS, Cs), + _ = file:delete(Dcl), + _ = file:delete(Dcd), + ok; + {disc_only_copies, disc_copies} -> + mnesia_monitor:unsafe_close_dets(Tab), + disc_delete_indecies(Tab, Cs, disc_only_copies), + case InitBy of + startup -> + ignore; + _ -> + mnesia_log:ets2dcd(Tab), + mnesia_controller:get_disc_copy(Tab), + disc_delete_table(Tab, disc_only_copies) + end + end end; true -> ignore @@ -607,6 +698,9 @@ insert_op(Tid, _, {op, restore_recreate, TabDef}, InPlace, InitBy) -> Tab = Cs#cstruct.name, Type = Cs#cstruct.type, Storage = mnesia_lib:cs_to_storage_type(node(), Cs), + Semantics = if Storage==unknown -> unknown; + true -> storage_semantics(Storage) + end, %% Delete all possibly existing files and tables disc_delete_table(Tab, Storage), disc_delete_indecies(Tab, Cs, Storage), @@ -625,13 +719,13 @@ insert_op(Tid, _, {op, restore_recreate, TabDef}, InPlace, InitBy) -> %% And create new ones.. if - (InitBy == startup) or (Storage == unknown) -> + (InitBy == startup) or (Semantics == unknown) -> ignore; - Storage == ram_copies -> + Semantics == ram_copies -> EtsProps = proplists:get_value(ets, StorageProps, []), Args = [{keypos, 2}, public, named_table, Type | EtsProps], mnesia_monitor:mktab(Tab, Args); - Storage == disc_copies -> + Semantics == disc_copies -> EtsProps = proplists:get_value(ets, StorageProps, []), Args = [{keypos, 2}, public, named_table, Type | EtsProps], mnesia_monitor:mktab(Tab, Args), @@ -640,7 +734,7 @@ insert_op(Tid, _, {op, restore_recreate, TabDef}, InPlace, InitBy) -> {repair, false}, {mode, read_write}], {ok, Log} = mnesia_monitor:open_log(FArg), mnesia_monitor:unsafe_close_log(Log); - Storage == disc_only_copies -> + Storage == disc_only_copies -> % note: Storage, not Semantics File = mnesia_lib:tab2dat(Tab), file:delete(File), DetsProps = proplists:get_value(dets, StorageProps, []), @@ -649,7 +743,10 @@ insert_op(Tid, _, {op, restore_recreate, TabDef}, InPlace, InitBy) -> {keypos, 2}, {repair, mnesia_monitor:get_env(auto_repair)} | DetsProps ], - mnesia_monitor:open_dets(Tab, Args) + mnesia_monitor:open_dets(Tab, Args); + element(1,Storage) == ext -> + {ext, Alias, Mod} = Storage, + Mod:create_table(Alias, Tab, []) end, insert_op(Tid, ignore, {op, create_table, TabDef}, InPlace, InitBy); @@ -659,9 +756,10 @@ insert_op(Tid, _, {op, create_table, TabDef}, InPlace, InitBy) -> Tab = Cs#cstruct.name, Storage = mnesia_lib:cs_to_storage_type(node(), Cs), StorageProps = Cs#cstruct.storage_properties, + Semantics = storage_semantics(Storage), case InitBy of startup -> - case Storage of + case Semantics of unknown -> ignore; ram_copies -> @@ -679,20 +777,10 @@ insert_op(Tid, _, {op, create_table, TabDef}, InPlace, InitBy) -> read_write), mnesia_log:unsafe_close_log(temp) end; - _ -> + disc_only_copies -> DetsProps = proplists:get_value(dets, StorageProps, []), - Args = [{file, mnesia_lib:tab2dat(Tab)}, - {type, mnesia_lib:disk_type(Tab, Cs#cstruct.type)}, - {keypos, 2}, - {repair, mnesia_monitor:get_env(auto_repair)} - | DetsProps ], - case mnesia_monitor:open_dets(Tab, Args) of - {ok, _} -> - mnesia_monitor:unsafe_close_dets(Tab); - {error, Error} -> - exit({"Failed to create dets table", Error}) - end + try_create_disc_only_copy(Storage, Tab, Cs, DetsProps) end; _ -> Copies = mnesia_lib:copy_holders(Cs), @@ -715,7 +803,7 @@ insert_op(Tid, _, {op, create_table, TabDef}, InPlace, InitBy) -> false -> mnesia_lib:set({Tab, where_to_read}, node()) end, - case Storage of + case Semantics of ram_copies -> ignore; _ -> @@ -838,7 +926,7 @@ insert_op(Tid, _, {op, del_table_copy, Storage, Node, TabDef}, InPlace, InitBy) insert_cstruct(Tid, Cs, true, InPlace, InitBy); Tab /= schema -> mnesia_controller:del_active_replica(Tab, Node), - mnesia_lib:del({Tab, Storage}, Node), + mnesia_lib:del({Tab, storage_alias(Storage)}, Node), if Node == node() -> case Cs#cstruct.local_content of @@ -896,9 +984,10 @@ insert_op(Tid, _, {op, add_index, Pos, TabDef}, InPlace, InitBy) -> Cs = mnesia_schema:list2cs(TabDef), Tab = insert_cstruct(Tid, Cs, true, InPlace, InitBy), Storage = mnesia_lib:cs_to_storage_type(node(), Cs), + Semantics = storage_semantics(Storage), case InitBy of - startup when Storage == disc_only_copies -> - true = open_files(Tab, Storage, InPlace, InitBy), + startup when Semantics == disc_only_copies -> + true = open_files(Tab, Semantics, Storage, InPlace, InitBy), mnesia_index:init_indecies(Tab, Storage, [Pos]); startup -> ignore; @@ -919,6 +1008,8 @@ insert_op(Tid, _, {op, del_index, Pos, TabDef}, InPlace, InitBy) -> mnesia_index:del_index_table(Tab, Storage, Pos); startup -> ignore; + _ when element(1, Storage) == ext -> + mnesia_index:del_index_table(Tab, Storage, Pos); _ -> mnesia_index:del_index_table(Tab, Storage, Pos) end, @@ -958,19 +1049,68 @@ insert_op(Tid, _, {op, change_table_frag, _Change, TabDef}, InPlace, InitBy) -> Cs = mnesia_schema:list2cs(TabDef), insert_cstruct(Tid, Cs, true, InPlace, InitBy). -open_files(Tab, Storage, UpdateInPlace, InitBy) - when Storage /= unknown, Storage /= ram_copies -> + +storage_semantics({ext, Alias, Mod}) -> + Mod:semantics(Alias, storage); +storage_semantics(Storage) when is_atom(Storage) -> + Storage. + +storage_alias({ext, Alias, _}) -> + Alias; +storage_alias(Storage) when is_atom(Storage) -> + Storage. + +change_disc_to_ram(Tab, Cs, FromS, ToS, Logtmp, InitBy) -> + disc_delete_indecies(Tab, Cs, FromS), + case InitBy of + startup -> + ignore; + _ -> + %% ram table will already have been created + Tab = ets:info(Tab, name), %% assertion + load_from_logfile(ToS, Tab, Logtmp), + PosList = Cs#cstruct.index, + mnesia_index:init_indecies(Tab, ToS, PosList) + end, + disc_delete_table(Tab, FromS). + + +try_create_disc_only_copy({ext,Alias,Mod}, Tab, Cs, _) -> + Mod:create_table(Alias, Tab, mnesia_schema:cs2list(Cs)); +try_create_disc_only_copy(disc_only_copies, Tab, Cs, DetsProps) -> + Args = [{file, mnesia_lib:tab2dat(Tab)}, + {type, mnesia_lib:disk_type(Tab, Cs#cstruct.type)}, + {keypos, 2}, + {repair, mnesia_monitor:get_env(auto_repair)} + | DetsProps], + case mnesia_monitor:open_dets(Tab, Args) of + {ok, _} -> + mnesia_monitor:unsafe_close_dets(Tab); + {error, Error} -> + exit({"Failed to create dets table", Error}) + end. + +restore_indexes(Tab, ToS, Cs) -> + PosList = Cs#cstruct.index, + mnesia_index:init_indecies(Tab, ToS, PosList). + + +open_files(Tab, Storage, UpdateInPlace, InitBy) -> + open_files(Tab, storage_semantics(Storage), Storage, UpdateInPlace, InitBy). + +open_files(Tab, Semantics, Storage, UpdateInPlace, InitBy) + when Storage /= unknown, Semantics /= ram_copies -> case get({?MODULE, Tab}) of undefined -> case ?catch_val({Tab, setorbag}) of {'EXIT', _} -> false; Type -> - case Storage of - disc_copies when Tab /= schema -> + Cs = val({Tab, cstruct}), + if Semantics == disc_copies, Tab /= schema -> Bool = open_disc_copies(Tab, InitBy), Bool; - _ -> + Storage == disc_only_copies; Tab == schema -> Props = val({Tab, storage_properties}), DetsProps = proplists:get_value(dets, Props, []), Fname = prepare_open(Tab, UpdateInPlace), @@ -981,6 +1121,12 @@ open_files(Tab, Storage, UpdateInPlace, InitBy) | DetsProps], {ok, _} = mnesia_monitor:open_dets(Tab, Args), put({?MODULE, Tab}, {opened_dumper, dat}), + true; + element(1, Storage) == ext -> + {ext, Alias, Mod} = Storage, + Mod:load_table(Alias, Tab, InitBy, + mnesia_schema:cs2list(Cs)), + put({?MODULE, Tab}, {opened_dumper, ext}), true end end; @@ -989,7 +1135,7 @@ open_files(Tab, Storage, UpdateInPlace, InitBy) {opened_dumper, _} -> true end; -open_files(_Tab, _Storage, _UpdateInPlace, _InitBy) -> +open_files(_Tab, _Semantics, _Storage, _UpdateInPlace, _InitBy) -> false. open_disc_copies(Tab, InitBy) -> @@ -1076,12 +1222,13 @@ close_files(InPlace, Outcome, InitBy, [{{?MODULE, Tab}, already_dumped} | Tail]) close_files(InPlace, Outcome, InitBy, Tail); close_files(InPlace, Outcome, InitBy, [{{?MODULE, Tab}, {opened_dumper, Type}} | Tail]) -> erase({?MODULE, Tab}), - case val({Tab, storage_type}) of + Storage = val({Tab, storage_type}), + case storage_semantics(Storage) of disc_only_copies when InitBy /= startup -> ignore; - disc_copies when Tab /= schema -> + disc_copies when Storage /= unknown, Tab /= schema -> mnesia_log:close_log({?MODULE,Tab}); - Storage -> + _ -> do_close(InPlace, Outcome, Tab, Type, Storage) end, close_files(InPlace, Outcome, InitBy, Tail); @@ -1093,11 +1240,16 @@ close_files(_, _, _InitBy, []) -> %% If storage is unknown during close clean up files, this can happen if timing %% is right and dirty_write conflicts with schema operations. +do_close(_, _, Tab, ext, {ext,Alias,Mod}) -> + Mod:close_table(Alias, Tab); do_close(_, _, Tab, dcl, unknown) -> mnesia_log:close_log({?MODULE,Tab}), file:delete(mnesia_lib:tab2dcl(Tab)); do_close(_, _, Tab, dcl, _) -> %% To be safe, can it happen? mnesia_log:close_log({?MODULE,Tab}); +do_close(_, _, _Tab, ext, unknown) -> + %% Not sure what to do here, but let's not crash + ok; do_close(InPlace, Outcome, Tab, dat, Storage) -> mnesia_monitor:close_dets(Tab), @@ -1148,7 +1300,8 @@ temp_set_master_nodes() -> Tabs = val({schema, local_tables}), Masters = [{Tab, (val({Tab, disc_copies}) ++ val({Tab, ram_copies}) ++ - val({Tab, disc_only_copies})) -- [node()]} + val({Tab, disc_only_copies}) ++ + external_copies(Tab)) -- [node()]} || Tab <- Tabs], %% UseDir = false since we don't want to remember these %% masternodes and we are running (really soon anyway) since we want this @@ -1156,6 +1309,13 @@ temp_set_master_nodes() -> mnesia_recover:log_master_nodes(Masters, false, yes), ok. +external_copies(Tab) -> + case ?catch_val({Tab, external_copies}) of + {'EXIT',_} -> []; + Ext -> + lists:concat([Ns || {_, Ns} <- Ext]) + end. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Raw dump of table. Dumper must have unique access to the ets table. @@ -1207,6 +1367,59 @@ raw_named_dump_table(Tab, Ftype) -> raw_dump_table(DetsRef, EtsRef) -> dets:from_ets(DetsRef, EtsRef). +dump_to_logfile(Storage, Tab) -> + case mnesia_monitor:use_dir() of + true -> + Logtmp = mnesia_lib:tab2logtmp(Tab), + file:delete(Logtmp), + case disk_log:open([{name, make_ref()}, + {file, Logtmp}, + {repair, false}, + {linkto, self()}]) of + {ok, Fd} -> + mnesia_lib:db_fixtable(Storage, Tab, true), + try do_dump_to_logfile(Storage, Tab, Fd) + after + mnesia_lib:db_fixtable(Storage, Tab, false) + end; + {error, _} = Error -> + Error + end; + false -> + {error, {has_no_disc, node()}} + end. + +do_dump_to_logfile(Storage, Tab, Fd) -> + Pat = [{'_',[],['$_']}], + log_terms(mnesia_lib:db_select_init(Storage, Tab, Pat, 100), Storage, Tab, Pat, Fd). + +log_terms({Objs, Cont}, Storage, Tab, Pat, Fd) -> + ok = disk_log:alog_terms(Fd, Objs), + log_terms(mnesia_lib:db_select_cont(Storage, Cont, '_'), Storage, Tab, Pat, Fd); +log_terms('$end_of_table', _, _, _, Fd) -> + disk_log:close(Fd). + +load_from_logfile(Storage, Tab, F) -> + case disk_log:open([{name, make_ref()}, + {file, F}, + {repair, true}, + {linkto, self()}]) of + {ok, Fd} -> + chunk_from_log(disk_log:chunk(Fd, start), Fd, Storage, Tab); + {repaired, Fd, _, _} -> + chunk_from_log(disk_log:chunk(Fd, start), Fd, Storage, Tab); + {error, _} = E -> + E + end. + +chunk_from_log({Cont, Terms}, Fd, Storage, Tab) -> + _ = [mnesia_lib:db_put(Storage, Tab, T) || T <- Terms], + chunk_from_log(disk_log:chunk(Fd, Cont), Fd, Storage, Tab); +chunk_from_log(eof, _, _, _) -> + ok. + + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Load regulator %% diff --git a/lib/mnesia/src/mnesia_ext_sup.erl b/lib/mnesia/src/mnesia_ext_sup.erl new file mode 100644 index 0000000000..3e6c72161c --- /dev/null +++ b/lib/mnesia/src/mnesia_ext_sup.erl @@ -0,0 +1,59 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 1996-2015. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +%% +%% This module implements a supervisor for external (plug-in) processes + +-module(mnesia_ext_sup). +-behaviour(supervisor). + +-export([start/0, + init/1, + start_proc/4, start_proc/5, + stop_proc/1]). + +-define(SHUTDOWN, 120000). % 2 minutes + +start() -> + supervisor:start_link({local, ?MODULE}, ?MODULE, []). + +start_proc(Name, M, F, A) -> + start_proc(Name, M, F, A, []). + +start_proc(Name, M, F, A, Opts) -> + [Restart, Shutdown, Type, Modules] = + [proplists:get_value(K, Opts, Default) + || {K, Default} <- [{restart, transient}, + {shutdown, ?SHUTDOWN}, + {type, worker}, + {modules, [M]}]], + case supervisor:start_child( + ?MODULE, {Name, {M,F,A}, Restart, Shutdown, Type, Modules}) of + {error, already_present} -> + supervisor:restart_child(?MODULE, Name); + Other -> + Other + end. + +stop_proc(Name) -> + supervisor:terminate_child(?MODULE, Name). + +init(_) -> + {ok, {{one_for_all, 10, 60}, []}}. diff --git a/lib/mnesia/src/mnesia_frag.erl b/lib/mnesia/src/mnesia_frag.erl index c4de603460..c6e812b36d 100644 --- a/lib/mnesia/src/mnesia_frag.erl +++ b/lib/mnesia/src/mnesia_frag.erl @@ -183,6 +183,8 @@ table_info2(ActivityId, Opaque, Tab, Frag, Item) -> length(val({Tab, disc_copies})); n_disc_only_copies -> length(val({Tab, disc_only_copies})); + n_external_copies -> + length(val({Tab, external_copies})); frag_names -> frag_names(Tab); @@ -498,13 +500,13 @@ expand_cstruct(Cs, Mode) -> %% Verify keys ValidKeys = [foreign_key, n_fragments, node_pool, n_ram_copies, n_disc_copies, n_disc_only_copies, - hash_module, hash_state], + n_external_copies, hash_module, hash_state], Keys = mnesia_schema:check_keys(Tab, Props, ValidKeys), mnesia_schema:check_duplicates(Tab, Keys), %% Pick fragmentation props ForeignKey = mnesia_schema:pick(Tab, foreign_key, Props, undefined), - {ForeignKey2, N, Pool, DefaultNR, DefaultND, DefaultNDO} = + {ForeignKey2, N, Pool, DefaultNR, DefaultND, DefaultNDO, DefaultNExt} = pick_props(Tab, Cs, ForeignKey), %% Verify node_pool @@ -518,6 +520,7 @@ expand_cstruct(Cs, Mode) -> NR = mnesia_schema:pick(Tab, n_ram_copies, Props, 0), ND = mnesia_schema:pick(Tab, n_disc_copies, Props, 0), NDO = mnesia_schema:pick(Tab, n_disc_only_copies, Props, 0), + NExt = mnesia_schema:pick(Tab, n_external_copies, Props, 0), PosInt = fun(I) when is_integer(I), I >= 0 -> true; (_I) -> false @@ -528,6 +531,8 @@ expand_cstruct(Cs, Mode) -> {bad_type, Tab, {n_disc_copies, ND}}), mnesia_schema:verify(true, PosInt(NDO), {bad_type, Tab, {n_disc_only_copies, NDO}}), + mnesia_schema:verify(true, PosInt(NExt), + {bad_type, Tab, {n_external_copies, NDO}}), %% Verify n_fragments Cs2 = verify_n_fragments(N, Cs, Mode), @@ -542,13 +547,13 @@ expand_cstruct(Cs, Mode) -> hash_module = HashMod, hash_state = HashState2}, if - NR == 0, ND == 0, NDO == 0 -> - do_expand_cstruct(Cs2, FH, N, Pool, DefaultNR, DefaultND, DefaultNDO, Mode); + NR == 0, ND == 0, NDO == 0, NExt == 0 -> + do_expand_cstruct(Cs2, FH, N, Pool, DefaultNR, DefaultND, DefaultNDO, DefaultNExt, Mode); true -> - do_expand_cstruct(Cs2, FH, N, Pool, NR, ND, NDO, Mode) + do_expand_cstruct(Cs2, FH, N, Pool, NR, ND, NDO, NExt, Mode) end. -do_expand_cstruct(Cs, FH, N, Pool, NR, ND, NDO, Mode) -> +do_expand_cstruct(Cs, FH, N, Pool, NR, ND, NDO, NExt, Mode) -> Tab = Cs#cstruct.name, LC = Cs#cstruct.local_content, @@ -562,14 +567,15 @@ do_expand_cstruct(Cs, FH, N, Pool, NR, ND, NDO, Mode) -> %% Add empty fragments CommonProps = [{base_table, Tab}], Cs2 = Cs#cstruct{frag_properties = lists:sort(CommonProps)}, - expand_frag_cstructs(N, NR, ND, NDO, Cs2, Pool, Pool, FH, Mode). + expand_frag_cstructs(N, NR, ND, NDO, NExt, Cs2, Pool, Pool, FH, Mode). verify_n_fragments(N, Cs, Mode) when is_integer(N), N >= 1 -> case Mode of create -> Cs#cstruct{ram_copies = [], disc_copies = [], - disc_only_copies = []}; + disc_only_copies = [], + external_copies = []}; activate -> Reason = {combine_error, Cs#cstruct.name, {n_fragments, N}}, mnesia_schema:verify(1, N, Reason), @@ -607,7 +613,8 @@ pick_props(Tab, Cs, {ForeignTab, Attr}) -> DefaultNR = length(val({ForeignTab, ram_copies})), DefaultND = length(val({ForeignTab, disc_copies})), DefaultNDO = length(val({ForeignTab, disc_only_copies})), - {Key, N, Pool, DefaultNR, DefaultND, DefaultNDO}; + DefaultNExt = length(val({ForeignTab, external_copies})), + {Key, N, Pool, DefaultNR, DefaultND, DefaultNDO, DefaultNExt}; pick_props(Tab, Cs, undefined) -> Props = Cs#cstruct.frag_properties, DefaultN = 1, @@ -617,22 +624,23 @@ pick_props(Tab, Cs, undefined) -> DefaultNR = 1, DefaultND = 0, DefaultNDO = 0, - {undefined, N, Pool, DefaultNR, DefaultND, DefaultNDO}; + DefaultNExt = 0, + {undefined, N, Pool, DefaultNR, DefaultND, DefaultNDO, DefaultNExt}; pick_props(Tab, _Cs, BadKey) -> mnesia:abort({bad_type, Tab, {foreign_key, BadKey}}). -expand_frag_cstructs(N, NR, ND, NDO, CommonCs, Dist, Pool, FH, Mode) +expand_frag_cstructs(N, NR, ND, NDO, NExt, CommonCs, Dist, Pool, FH, Mode) when N > 1, Mode == create -> Frag = n_to_frag_name(CommonCs#cstruct.name, N), Cs = CommonCs#cstruct{name = Frag}, - {Cs2, RevModDist, RestDist} = set_frag_nodes(NR, ND, NDO, Cs, Dist, []), + {Cs2, RevModDist, RestDist} = set_frag_nodes(NR, ND, NDO, NExt, Cs, Dist, []), ModDist = lists:reverse(RevModDist), Dist2 = rearrange_dist(Cs, ModDist, RestDist, Pool), %% Adjusts backwards, but it doesn't matter. {FH2, _FromFrags, _AdditionalWriteFrags} = adjust_before_split(FH), - CsList = expand_frag_cstructs(N - 1, NR, ND, NDO, CommonCs, Dist2, Pool, FH2, Mode), + CsList = expand_frag_cstructs(N - 1, NR, ND, NDO, NExt, CommonCs, Dist2, Pool, FH2, Mode), [Cs2 | CsList]; -expand_frag_cstructs(1, NR, ND, NDO, CommonCs, Dist, Pool, FH, Mode) -> +expand_frag_cstructs(1, NR, ND, NDO, NExt, CommonCs, Dist, Pool, FH, Mode) -> BaseProps = CommonCs#cstruct.frag_properties ++ [{foreign_key, FH#frag_state.foreign_key}, {hash_module, FH#frag_state.hash_module}, @@ -645,25 +653,29 @@ expand_frag_cstructs(1, NR, ND, NDO, CommonCs, Dist, Pool, FH, Mode) -> activate -> [BaseCs]; create -> - {BaseCs2, _, _} = set_frag_nodes(NR, ND, NDO, BaseCs, Dist, []), + {BaseCs2, _, _} = set_frag_nodes(NR, ND, NDO, NExt, BaseCs, Dist, []), [BaseCs2] end. -set_frag_nodes(NR, ND, NDO, Cs, [Head | Tail], Acc) when NR > 0 -> +set_frag_nodes(NR, ND, NDO, NExt, Cs, [Head | Tail], Acc) when NR > 0 -> Pos = #cstruct.ram_copies, {Cs2, Head2} = set_frag_node(Cs, Pos, Head), - set_frag_nodes(NR - 1, ND, NDO, Cs2, Tail, [Head2 | Acc]); -set_frag_nodes(NR, ND, NDO, Cs, [Head | Tail], Acc) when ND > 0 -> + set_frag_nodes(NR - 1, ND, NDO, NExt, Cs2, Tail, [Head2 | Acc]); +set_frag_nodes(NR, ND, NDO, NExt, Cs, [Head | Tail], Acc) when ND > 0 -> Pos = #cstruct.disc_copies, {Cs2, Head2} = set_frag_node(Cs, Pos, Head), - set_frag_nodes(NR, ND - 1, NDO, Cs2, Tail, [Head2 | Acc]); -set_frag_nodes(NR, ND, NDO, Cs, [Head | Tail], Acc) when NDO > 0 -> + set_frag_nodes(NR, ND - 1, NDO, NExt, Cs2, Tail, [Head2 | Acc]); +set_frag_nodes(NR, ND, NDO, NExt, Cs, [Head | Tail], Acc) when NDO > 0 -> Pos = #cstruct.disc_only_copies, {Cs2, Head2} = set_frag_node(Cs, Pos, Head), - set_frag_nodes(NR, ND, NDO - 1, Cs2, Tail, [Head2 | Acc]); -set_frag_nodes(0, 0, 0, Cs, RestDist, ModDist) -> + set_frag_nodes(NR, ND, NDO - 1, NExt, Cs2, Tail, [Head2 | Acc]); +set_frag_nodes(NR, ND, NDO, NExt, Cs, [Head | Tail], Acc) when NExt > 0 -> + Pos = #cstruct.external_copies, + {Cs2, Head2} = set_frag_node(Cs, Pos, Head), + set_frag_nodes(NR, ND, NDO, NExt - 1, Cs2, Tail, [Head2 | Acc]); +set_frag_nodes(0, 0, 0, 0, Cs, RestDist, ModDist) -> {Cs, ModDist, RestDist}; -set_frag_nodes(_, _, _, Cs, [], _) -> +set_frag_nodes(_, _, _, _, Cs, [], _) -> mnesia:abort({combine_error, Cs#cstruct.name, "Too few nodes in node_pool"}). set_frag_node(Cs, Pos, Head) -> @@ -840,13 +852,15 @@ make_add_frag(Tab, SortedNs) -> NR = length(Cs#cstruct.ram_copies), ND = length(Cs#cstruct.disc_copies), NDO = length(Cs#cstruct.disc_only_copies), + NExt = length(Cs#cstruct.external_copies), NewCs = Cs#cstruct{name = NewFrag, frag_properties = [{base_table, Tab}], ram_copies = [], disc_copies = [], - disc_only_copies = []}, + disc_only_copies = [], + external_copies = []}, - {NewCs2, _, _} = set_frag_nodes(NR, ND, NDO, NewCs, SortedNs, []), + {NewCs2, _, _} = set_frag_nodes(NR, ND, NDO, NExt, NewCs, SortedNs, []), [NewOp] = mnesia_schema:make_create_table(NewCs2), SplitOps = split(Tab, FH2, FromIndecies, FragNames, []), @@ -1319,7 +1333,8 @@ count_frag([Frag | Frags], Dist) -> Dist2 = incr_nodes(val({Frag, ram_copies}), Dist), Dist3 = incr_nodes(val({Frag, disc_copies}), Dist2), Dist4 = incr_nodes(val({Frag, disc_only_copies}), Dist3), - count_frag(Frags, Dist4); + Dist5 = incr_nodes(val({Frag, external_copies}), Dist4), + count_frag(Frags, Dist5); count_frag([], Dist) -> Dist. diff --git a/lib/mnesia/src/mnesia_index.erl b/lib/mnesia/src/mnesia_index.erl index be05cfbea3..73d170d1fa 100644 --- a/lib/mnesia/src/mnesia_index.erl +++ b/lib/mnesia/src/mnesia_index.erl @@ -23,8 +23,8 @@ -module(mnesia_index). -export([read/5, - add_index/5, - delete_index/3, + add_index/6, + delete_index/4, del_object_index/5, clear_index/4, dirty_match_object/3, @@ -39,19 +39,21 @@ get_index_table/3, tab2filename/2, - tab2tmp_filename/2, init_index/2, init_indecies/3, del_transient/2, del_transient/3, - del_index_table/3]). + del_index_table/3, + + index_info/2, + ext_index_instances/1]). -import(mnesia_lib, [val/1, verbose/2]). -include("mnesia.hrl"). -record(index, {setorbag, pos_list}). -%% read an object list throuh its index table +%% read an object list through its index table %% we assume that table Tab has index on attribute number Pos read(Tid, Store, Tab, IxKey, Pos) -> @@ -64,69 +66,103 @@ read(Tid, Store, Tab, IxKey, Pos) -> ResList end. -add_index(Index, Tab, Key, Obj, Old) -> - add_index2(Index#index.pos_list, Index#index.setorbag, Tab, Key, Obj, Old). - -add_index2([{Pos, Ixt} |Tail], bag, Tab, K, Obj, OldRecs) -> - db_put(Ixt, {element(Pos, Obj), K}), - add_index2(Tail, bag, Tab, K, Obj, OldRecs); -add_index2([{Pos, Ixt} |Tail], Type, Tab, K, Obj, OldRecs0) -> +ext_index_instances(Tab) -> + #index{pos_list = PosL} = val({Tab, index_info}), + lists:foldr( + fun({_, {{ext,Alias,Mod}, Tag}}, Acc) -> + [{Alias, Mod, Tag}|Acc]; + (_, Acc) -> + Acc + end, [], PosL). + + +add_index(#index{pos_list = PosL, setorbag = SorB}, + Storage, Tab, Key, Obj, Old) -> + add_index2(PosL, SorB, Storage, Tab, Key, Obj, Old). + +add_index2([{{Pos,Type}, Ixt} |Tail], bag, Storage, Tab, K, Obj, OldRecs) -> + ValsF = index_vals_f(Storage, Tab, Pos), + Vals = ValsF(Obj), + put_index_vals(Type, Ixt, Vals, K), + add_index2(Tail, bag, Storage, Tab, K, Obj, OldRecs); +add_index2([{{Pos, Type}, Ixt} |Tail], SorB, Storage, Tab, K, Obj, OldRecs0) -> %% Remove old tuples in index if Tab is updated + ValsF = index_vals_f(Storage, Tab, Pos), NewVals = ValsF(Obj), OldRecs1 = case OldRecs0 of - undefined -> mnesia_lib:db_get(Tab, K); + undefined -> mnesia_lib:db_get(Storage, Tab, K); _ -> OldRecs0 end, - IdxVal = element(Pos, Obj), - case [Old || Old <- OldRecs1, element(Pos, Old) =/= IdxVal] of - [] when OldRecs1 =:= [] -> %% Write - db_put(Ixt, {element(Pos, Obj), K}), - add_index2(Tail, Type, Tab, K, Obj, OldRecs0); + IdxVal = ValsF(Obj), + case [Old || Old <- OldRecs1, ValsF(Old) =/= IdxVal] of + [] when OldRecs1 =:= [] -> % Write + put_index_vals(Type, Ixt, NewVals, K), + add_index2(Tail, SorB, Storage, Tab, K, Obj, OldRecs1); [] -> %% when OldRecs1 =/= [] Update without modifying index field - add_index2(Tail, Type, Tab, K, Obj, OldRecs0); + add_index2(Tail, SorB, Storage, Tab, K, Obj, OldRecs1); OldRecs -> %% Update - db_put(Ixt, {element(Pos, Obj), K}), - del_ixes(Ixt, OldRecs, Pos, K), - add_index2(Tail, Type, Tab, K, Obj, OldRecs0) + put_index_vals(Type, Ixt, NewVals, K), + [del_ixes(Type, Ixt, ValsF, OldObj, K) || OldObj <- OldRecs], + add_index2(Tail, SorB, Storage, Tab, K, Obj, OldRecs1) end; -add_index2([], _, _Tab, _K, _Obj, _) -> ok. +add_index2([], _, _, _Tab, _K, _Obj, _) -> ok. + +delete_index(Index, Storage, Tab, K) -> + delete_index2(Index#index.pos_list, Storage, Tab, K). -delete_index(Index, Tab, K) -> - delete_index2(Index#index.pos_list, Tab, K). +delete_index2([{{Pos, Type}, Ixt} | Tail], Storage, Tab, K) -> + DelObjs = mnesia_lib:db_get(Storage, Tab, K), + ValsF = index_vals_f(Storage, Tab, Pos), + [del_ixes(Type, Ixt, ValsF, Obj, K) || Obj <- DelObjs], + delete_index2(Tail, Storage, Tab, K); +delete_index2([], _Storage, _Tab, _K) -> ok. -delete_index2([{Pos, Ixt} | Tail], Tab, K) -> - DelObjs = mnesia_lib:db_get(Tab, K), - del_ixes(Ixt, DelObjs, Pos, K), - delete_index2(Tail, Tab, K); -delete_index2([], _Tab, _K) -> ok. +put_index_vals(ordered, Ixt, Vals, K) -> + [db_put(Ixt, {{V, K}}) || V <- Vals]; +put_index_vals(bag, Ixt, Vals, K) -> + [db_put(Ixt, {V, K}) || V <- Vals]. -del_ixes(_Ixt, [], _Pos, _L) -> ok; -del_ixes(Ixt, [Obj | Tail], Pos, Key) -> - db_match_erase(Ixt, {element(Pos, Obj), Key}), - del_ixes(Ixt, Tail, Pos, Key). +del_ixes(bag, Ixt, ValsF, Obj, Key) -> + Vals = ValsF(Obj), + [db_match_erase(Ixt, {V, Key}) || V <- Vals]; +del_ixes(ordered, Ixt, ValsF, Obj, Key) -> + Vals = ValsF(Obj), + [db_erase(Ixt, {V,Key}) || V <- Vals]. -del_object_index(Index, Tab, K, Obj, Old) -> - del_object_index2(Index#index.pos_list, Index#index.setorbag, Tab, K, Obj, Old). +del_object_index(#index{pos_list = PosL, setorbag = SorB}, Storage, Tab, K, Obj) -> + del_object_index2(PosL, SorB, Storage, Tab, K, Obj). -del_object_index2([], _, _Tab, _K, _Obj, _Old) -> ok; -del_object_index2([{Pos, Ixt} | Tail], SoB, Tab, K, Obj, Old) -> +del_object_index2([], _, _Storage, _Tab, _K, _Obj) -> ok; +del_object_index2([{{Pos, Type}, Ixt} | Tail], SoB, Storage, Tab, K, Obj) -> + ValsF = index_vals_f(Storage, Tab, Pos), case SoB of bag -> - del_object_bag(Tab, K, Obj, Pos, Ixt, Old); + del_object_bag(Type, ValsF, Tab, K, Obj, Ixt); _ -> %% If set remove the tuple in index table - del_ixes(Ixt, [Obj], Pos, K) + del_ixes(Type, Ixt, ValsF, Obj, K) end, - del_object_index2(Tail, SoB, Tab, K, Obj, Old). - -del_object_bag(Tab, Key, Obj, Pos, Ixt, undefined) -> - IxKey = element(Pos, Obj), - Old = [X || X <- mnesia_lib:db_get(Tab, Key), element(Pos, X) =:= IxKey], - del_object_bag(Tab, Key, Obj, Pos, Ixt, Old); -%% If Tab type is bag we need remove index identifier if the object being -%% deleted was the last one -del_object_bag(_Tab, Key, Obj, Pos, Ixt, Old) when Old =:= [Obj] -> - del_ixes(Ixt, [Obj], Pos, Key); -del_object_bag(_Tab, _Key, _Obj, _Pos, _Ixt, _Old) -> ok. + del_object_index2(Tail, SoB, Storage, Tab, K, Obj). + +del_object_bag(Type, ValsF, Tab, Key, Obj, Ixt) -> + IxKeys = ValsF(Obj), + Found = [{X, ValsF(X)} || X <- mnesia_lib:db_get(Tab, Key)], + del_object_bag_(IxKeys, Found, Type, Tab, Key, Obj, Ixt). + +del_object_bag_([IxK|IxKs], Found, Type, Tab, Key, Obj, Ixt) -> + case [X || {X, Ixes} <- Found, lists:member(IxK, Ixes)] of + [Old] when Old =:= Obj -> + case Type of + bag -> + db_match_erase(Ixt, {IxK, Key}); + ordered -> + db_erase(Ixt, {{IxK, Key}}) + end; + _ -> + ok + end, + del_object_bag_(IxKs, Found, Type, Tab, Key, Obj, Ixt); +del_object_bag_([], _, _, _, _, _, _) -> + ok. clear_index(Index, Tab, K, Obj) -> clear_index2(Index#index.pos_list, Tab, K, Obj). @@ -138,8 +174,9 @@ clear_index2([{_Pos, Ixt} | Tail], Tab, K, Obj) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -dirty_match_object(Tab, Pat, Pos) -> +dirty_match_object(Tab, Pat, Pos) when is_integer(Pos) -> %% Assume that we are on the node where the replica is + %% Cannot use index plugins here, as they don't map to match patterns case element(2, Pat) of '_' -> IxKey = element(Pos, Pat), @@ -161,7 +198,7 @@ realkeys(Tab, Pos, IxKey) -> Index = get_index_table(Tab, Pos), db_get(Index, IxKey). % a list on the form [{IxKey, RealKey1} , .... -dirty_select(Tab, Spec, Pos) -> +dirty_select(Tab, Spec, Pos) when is_integer(Pos) -> %% Assume that we are on the node where the replica is %% Returns the records without applying the match spec %% The actual filtering is handled by the caller @@ -171,59 +208,80 @@ dirty_select(Tab, Spec, Pos) -> lists:append([mnesia_lib:db_get(StorageType, Tab, Key) || {_,Key} <- RealKeys]). dirty_read(Tab, IxKey, Pos) -> - ResList = mnesia:dirty_rpc(Tab, ?MODULE, dirty_read2, - [Tab, IxKey, Pos]), - case val({Tab, setorbag}) of - bag -> - %% Remove all tuples which don't include Ixkey - mnesia_lib:key_search_all(IxKey, Pos, ResList); - _ -> - ResList - end. + mnesia:dirty_rpc(Tab, ?MODULE, dirty_read2, + [Tab, IxKey, Pos]). dirty_read2(Tab, IxKey, Pos) -> - Ix = get_index_table(Tab, Pos), - Keys = db_match(Ix, {IxKey, '$1'}), - r_keys(Keys, Tab, []). - -r_keys([[H]|T],Tab,Ack) -> - V = mnesia_lib:db_get(Tab, H), - r_keys(T, Tab, V ++ Ack); -r_keys([], _, Ack) -> - Ack. + #index{pos_list = PosL} = val({Tab, index_info}), + Storage = val({Tab, storage_type}), + {Type, Ixt} = pick_index(PosL, Tab, Pos), + Pat = case Type of + ordered -> [{{{IxKey, '$1'}}, [], ['$1']}]; + bag -> [{{IxKey, '$1'}, [], ['$1']}] + end, + Keys = db_select(Ixt, Pat), + ValsF = index_vals_f(Storage, Tab, Pos), + lists:reverse( + lists:foldl( + fun(K, Acc) -> + lists:foldl( + fun(Obj, Acc1) -> + case lists:member(IxKey, ValsF(Obj)) of + true -> [Obj|Acc1]; + false -> Acc1 + end + end, Acc, mnesia_lib:db_get(Storage, Tab, K)) + end, [], Keys)). + +pick_index([{{{Pfx,_},IxType}, Ixt}|_], _Tab, {_} = Pfx) -> + {IxType, Ixt}; +pick_index([{{Pos,IxType}, Ixt}|_], _Tab, Pos) -> + {IxType, Ixt}; +pick_index([_|T], Tab, Pos) -> + pick_index(T, Tab, Pos); +pick_index([], Tab, Pos) -> + mnesia:abort({no_exist, Tab, {index, Pos}}). + %%%%%%% Creation, Init and deletion routines for index tables %% We can have several indexes on the same table %% this can be a fairly costly operation if table is *very* large -tab2filename(Tab, Pos) -> +tab2filename(Tab, {A}) when is_atom(A) -> + mnesia_lib:dir(Tab) ++ "_-" ++ atom_to_list(A) ++ "-.DAT"; +tab2filename(Tab, T) when is_tuple(T) -> + tab2filename(Tab, element(1, T)); +tab2filename(Tab, Pos) when is_integer(Pos) -> mnesia_lib:dir(Tab) ++ "_" ++ integer_to_list(Pos) ++ ".DAT". -tab2tmp_filename(Tab, Pos) -> - mnesia_lib:dir(Tab) ++ "_" ++ integer_to_list(Pos) ++ ".TMP". - init_index(Tab, Storage) -> - PosList = val({Tab, index}), + Cs = val({Tab, cstruct}), + PosList = Cs#cstruct.index, init_indecies(Tab, Storage, PosList). init_indecies(Tab, Storage, PosList) -> case Storage of unknown -> ignore; + {ext, Alias, Mod} -> + init_ext_index(Tab, Storage, Alias, Mod, PosList); disc_only_copies -> - init_disc_index(Tab, PosList); + init_disc_index(Tab, Storage, PosList); ram_copies -> - make_ram_index(Tab, PosList); + make_ram_index(Tab, Storage, PosList); disc_copies -> - make_ram_index(Tab, PosList) + make_ram_index(Tab, Storage, PosList) end. %% works for both ram and disc indexes del_index_table(_, unknown, _) -> ignore; -del_index_table(Tab, Storage, Pos) -> +del_index_table(Tab, Storage, {_} = Pos) -> + delete_transient_index(Tab, Pos, Storage), + mnesia_lib:del({Tab, index}, Pos); +del_index_table(Tab, Storage, Pos) when is_integer(Pos) -> delete_transient_index(Tab, Pos, Storage), mnesia_lib:del({Tab, index}, Pos). @@ -236,13 +294,24 @@ del_transient(Tab, [Pos | Tail], Storage) -> delete_transient_index(Tab, Pos, Storage), del_transient(Tab, Tail, Storage). +delete_transient_index(Tab, Pos, {ext, Alias, Mod}) -> + PosInfo = case Pos of + _ when is_integer(Pos) -> + Cs = val({Tab, cstruct}), + lists:keyfind(Pos, 1, Cs#cstruct.index); + {P, T} -> {P, T} + end, + Tag = {Tab, index, PosInfo}, + Mod:close_table(Alias, Tag), + Mod:delete_table(Alias, Tag), + del_index_info(Tab, Pos), + mnesia_lib:unset({Tab, {index, Pos}}); delete_transient_index(Tab, Pos, disc_only_copies) -> Tag = {Tab, index, Pos}, mnesia_monitor:unsafe_close_dets(Tag), _ = file:delete(tab2filename(Tab, Pos)), del_index_info(Tab, Pos), %% Uses val(..) mnesia_lib:unset({Tab, {index, Pos}}); - delete_transient_index(Tab, Pos, _Storage) -> Ixt = val({Tab, {index, Pos}}), ?ets_delete_table(Ixt), @@ -252,11 +321,12 @@ delete_transient_index(Tab, Pos, _Storage) -> %%%%% misc functions for the index create/init/delete functions above %% assuming that the file exists. -init_disc_index(_Tab, []) -> +init_disc_index(_Tab, _Storage, []) -> done; -init_disc_index(Tab, [Pos | Tail]) when is_integer(Pos) -> +init_disc_index(Tab, disc_only_copies, [{Pos,_Pref} | Tail]) -> + PosInfo = {Pos, bag}, Fn = tab2filename(Tab, Pos), - IxTag = {Tab, index, Pos}, + IxTag = {Tab, index, PosInfo}, _ = file:delete(Fn), Args = [{file, Fn}, {keypos, 1}, {type, bag}], mnesia_monitor:open_dets(IxTag, Args), @@ -269,16 +339,59 @@ init_disc_index(Tab, [Pos | Tail]) when is_integer(Pos) -> mnesia_lib:db_fixtable(Storage, Tab, true), ok = dets:init_table(IxTag, create_fun(Init, Tab, Pos)), mnesia_lib:db_fixtable(Storage, Tab, false), - mnesia_lib:set({Tab, {index, Pos}}, IxTag), - add_index_info(Tab, val({Tab, setorbag}), {Pos, {dets, IxTag}}), - init_disc_index(Tab, Tail). + mnesia_lib:set({Tab, {index, PosInfo}}, IxTag), + add_index_info(Tab, val({Tab, setorbag}), {PosInfo, {dets, IxTag}}), + init_disc_index(Tab, Storage, Tail). + +init_ext_index(_, _, _, _, []) -> + done; +init_ext_index(Tab, Storage, Alias, Mod, [{Pos,Type} | Tail]) -> + PosInfo = {Pos, Type}, + IxTag = {Tab, index, PosInfo}, + CS = val({Tab, cstruct}), + CsList = mnesia_schema:cs2list(CS), + _Res = mnesia_monitor:unsafe_create_external(IxTag, Alias, Mod, CsList), + Mod:load_table(Alias, IxTag, init_index, CsList), + case Mod:is_index_consistent(Alias, IxTag) of + false -> + Mod:index_is_consistent(Alias, IxTag, false), + Mod:match_delete(Alias, IxTag, '_'), + IxValsF = index_vals_f(Storage, Tab, Pos), + IxObjF = case Type of + bag -> fun(IxVal, Key) -> {IxVal, Key} end; + ordered -> fun(IxVal, Key) -> {{IxVal, Key}} end + end, + mnesia_lib:db_fixtable(Storage, Tab, true), + mnesia_lib:db_foldl( + Storage, + fun(Rec, Acc) -> + Key = element(2, Rec), + lists:foreach( + fun(V) -> + IxObj = IxObjF(V, Key), + Mod:insert(Alias, IxTag, IxObj) + end, IxValsF(Rec)), + Acc + end, ok, Tab, + [{'_', [], ['$_']}], 100), + Mod:index_is_consistent(Alias, IxTag, true); + true -> + ignore + end, + + mnesia_lib:set({Tab, {index, PosInfo}}, IxTag), + + add_index_info(Tab, val({Tab, setorbag}), {PosInfo, {Storage, IxTag}}), + init_ext_index(Tab, Storage, Alias, Mod, Tail). create_fun(Cont, Tab, Pos) -> + IxF = index_vals_f(disc_only_copies, Tab, Pos), fun(read) -> Data = case Cont of {start, KeysPerChunk} -> - mnesia_lib:db_init_chunk(disc_only_copies, Tab, KeysPerChunk); + mnesia_lib:db_init_chunk( + disc_only_copies, Tab, KeysPerChunk); '$end_of_table' -> '$end_of_table'; _Else -> @@ -288,56 +401,82 @@ create_fun(Cont, Tab, Pos) -> '$end_of_table' -> end_of_input; {Recs, Next} -> - IdxElems = [{element(Pos, Obj), element(2, Obj)} || Obj <- Recs], + IdxElems = lists:flatmap( + fun(Obj) -> + PrimK = element(2, Obj), + [{V, PrimK} || V <- IxF(Obj)] + end, Recs), {IdxElems, create_fun(Next, Tab, Pos)} end; (close) -> ok end. -make_ram_index(_, []) -> +make_ram_index(_, _, []) -> done; -make_ram_index(Tab, [Pos | Tail]) -> - add_ram_index(Tab, Pos), - make_ram_index(Tab, Tail). +make_ram_index(Tab, Storage, [Pos | Tail]) -> + add_ram_index(Tab, Storage, Pos), + make_ram_index(Tab, Storage, Tail). -add_ram_index(Tab, Pos) when is_integer(Pos) -> - verbose("Creating index for ~w ~n", [Tab]), +add_ram_index(Tab, Storage, {Pos, _Pref}) -> + Type = ordered, + verbose("Creating index for ~w ~p ~p~n", [Tab, Pos, Type]), SetOrBag = val({Tab, setorbag}), - IndexType = case SetOrBag of - set -> duplicate_bag; - ordered_set -> duplicate_bag; - bag -> bag - end, - Index = mnesia_monitor:mktab(mnesia_index, [IndexType, public]), + IxValsF = index_vals_f(Storage, Tab, Pos), + IxFun = fun(Val, Key) -> {{Val, Key}} end, + Index = mnesia_monitor:mktab(mnesia_index, [ordered_set, public]), Insert = fun(Rec, _Acc) -> - true = ?ets_insert(Index, {element(Pos, Rec), element(2, Rec)}) + PrimK = element(2, Rec), + true = ?ets_insert( + Index, [IxFun(V, PrimK) + || V <- IxValsF(Rec)]) end, mnesia_lib:db_fixtable(ram_copies, Tab, true), - true = ets:foldl(Insert, true, Tab), + true = mnesia_lib:db_foldl(Storage, Insert, true, Tab), mnesia_lib:db_fixtable(ram_copies, Tab, false), mnesia_lib:set({Tab, {index, Pos}}, Index), - add_index_info(Tab, SetOrBag, {Pos, {ram, Index}}); -add_ram_index(_Tab, snmp) -> + add_index_info(Tab, SetOrBag, {{Pos, Type}, {ram, Index}}); +add_ram_index(_Tab, _, snmp) -> ok. -add_index_info(Tab, Type, IxElem) -> +index_info(SetOrBag, PosList) -> + IxPlugins = mnesia_schema:index_plugins(), + ExpPosList = lists:map( + fun({{P,Type},Ixt} = PI) -> + case P of + {_} = IxN -> + {_, M, F} = + lists:keyfind(IxN, 1, IxPlugins), + {{{IxN,M,F}, Type}, Ixt}; + _ -> + PI + end + end, PosList), + #index{setorbag = SetOrBag, pos_list = ExpPosList}. + +add_index_info(Tab, SetOrBag, IxElem) -> Commit = val({Tab, commit_work}), case lists:keysearch(index, 1, Commit) of false -> - Index = #index{setorbag = Type, - pos_list = [IxElem]}, - %% Check later if mnesia_tm is sensative about the order + IndexInfo = index_info(SetOrBag, [IxElem]), + %% Check later if mnesia_tm is sensitive about the order + mnesia_lib:set({Tab, index_info}, IndexInfo), + mnesia_lib:set({Tab, index}, index_positions(IndexInfo)), mnesia_lib:set({Tab, commit_work}, - mnesia_lib:sort_commit([Index | Commit])); + mnesia_lib:sort_commit([IndexInfo | Commit])); {value, Old} -> %% We could check for consistency here Index = Old#index{pos_list = [IxElem | Old#index.pos_list]}, + mnesia_lib:set({Tab, index_info}, Index), + mnesia_lib:set({Tab, index}, index_positions(Index)), NewC = lists:keyreplace(index, 1, Commit, Index), mnesia_lib:set({Tab, commit_work}, mnesia_lib:sort_commit(NewC)) end. +index_positions(#index{pos_list = PL}) -> + [P || {{P,_},_} <- PL]. + del_index_info(Tab, Pos) -> Commit = val({Tab, commit_work}), case lists:keysearch(index, 1, Commit) of @@ -345,13 +484,21 @@ del_index_info(Tab, Pos) -> %% Something is wrong ignore skip; {value, Old} -> - case lists:keydelete(Pos, 1, Old#index.pos_list) of + case lists:filter(fun({P,_}) -> + element(1,P)=/=Pos + end, + Old#index.pos_list) of [] -> + IndexInfo = index_info(Old#index.setorbag,[]), + mnesia_lib:set({Tab, index_info}, IndexInfo), + mnesia_lib:set({Tab, index}, index_positions(IndexInfo)), NewC = lists:keydelete(index, 1, Commit), mnesia_lib:set({Tab, commit_work}, mnesia_lib:sort_commit(NewC)); New -> Index = Old#index{pos_list = New}, + mnesia_lib:set({Tab, index_info}, Index), + mnesia_lib:set({Tab, index}, index_positions(Index)), NewC = lists:keyreplace(index, 1, Commit, Index), mnesia_lib:set({Tab, commit_work}, mnesia_lib:sort_commit(NewC)) @@ -360,33 +507,69 @@ del_index_info(Tab, Pos) -> db_put({ram, Ixt}, V) -> true = ?ets_insert(Ixt, V); +db_put({{ext, _, _} = Ext, Ixt}, V) -> + mnesia_lib:db_put(Ext, Ixt, V); db_put({dets, Ixt}, V) -> ok = dets:insert(Ixt, V). -db_get({ram, Ixt}, K) -> - ?ets_lookup(Ixt, K); +db_get({ram, _}=Ixt, IxKey) -> + Pat = [{{{IxKey, '$1'}}, [], [{{IxKey,'$1'}}]}], + db_select(Ixt, Pat); +db_get({{ext,_,_} = _Storage, {_,_,{_,Type}}} = Ixt, IxKey) -> + Pat = case Type of + ordered -> [{{{IxKey, '$1'}}, [], [{{IxKey,'$1'}}]}]; + bag -> [{{IxKey, '_'}, [], ['$_']}] + end, + db_select(Ixt, Pat); db_get({dets, Ixt}, K) -> dets:lookup(Ixt, K). +db_erase({ram, Ixt}, K) -> + ?ets_delete(Ixt, K); +db_erase({{ext,_,_} = Ext, Ixt}, K) -> + mnesia_lib:db_erase(Ext, Ixt, K); +db_erase({dets, Ixt}, K) -> + dets:delete(Ixt, K). + db_match_erase({ram, Ixt}, Pat) -> true = ?ets_match_delete(Ixt, Pat); +db_match_erase({{ext,_,_} = Ext, Ixt}, Pat) -> + mnesia_lib:db_match_erase(Ext, Ixt, Pat); db_match_erase({dets, Ixt}, Pat) -> ok = dets:match_delete(Ixt, Pat). -db_match({ram, Ixt}, Pat) -> - ?ets_match(Ixt, Pat); -db_match({dets, Ixt}, Pat) -> - dets:match(Ixt, Pat). +db_select({ram, Ixt}, Pat) -> + ets:select(Ixt, Pat); +db_select({{ext,_,_} = Ext, Ixt}, Pat) -> + mnesia_lib:db_select(Ext, Ixt, Pat); +db_select({dets, Ixt}, Pat) -> + dets:select(Ixt, Pat). + get_index_table(Tab, Pos) -> get_index_table(Tab, val({Tab, storage_type}), Pos). -get_index_table(Tab, ram_copies, Pos) -> - {ram, val({Tab, {index, Pos}})}; -get_index_table(Tab, disc_copies, Pos) -> - {ram, val({Tab, {index, Pos}})}; -get_index_table(Tab, disc_only_copies, Pos) -> - {dets, val({Tab, {index, Pos}})}; -get_index_table(_Tab, unknown, _Pos) -> - unknown. - +get_index_table(Tab, _Storage, Pos) -> + #index{pos_list = PosL} = val({Tab, index_info}), + {_IxType, Ixt} = pick_index(PosL, Tab, Pos), + Ixt. + +index_vals_f(Storage, Tab, {_} = Pos) -> + index_vals_f(Storage, Tab, + lists:keyfind(Pos, 1, mnesia_schema:index_plugins())); +index_vals_f(_Storage, Tab, {Pos,M,F}) -> + fun(Obj) -> + M:F(Tab, Pos, Obj) + end; +index_vals_f(Storage, Tab, Pos) when is_integer(Pos) -> + case mnesia_lib:semantics(Storage, index_fun) of + undefined -> + fun(Obj) -> + [element(Pos, Obj)] + end; + F when is_function(F, 4) -> + {ext, Alias, _Mod} = Storage, + fun(Obj) -> + F(Alias, Tab, Pos, Obj) + end + end. diff --git a/lib/mnesia/src/mnesia_lib.erl b/lib/mnesia/src/mnesia_lib.erl index a5eded4f0f..10e232c800 100644 --- a/lib/mnesia/src/mnesia_lib.erl +++ b/lib/mnesia/src/mnesia_lib.erl @@ -54,6 +54,7 @@ db_erase_tab/2, db_first/1, db_first/2, + db_foldl/3, db_foldl/4, db_foldl/6, db_last/1, db_last/2, db_fixtable/3, @@ -135,6 +136,7 @@ set_local_content_whereabouts/1, set_remote_where_to_read/1, set_remote_where_to_read/2, + semantics/2, show/1, show/2, sort_commit/1, @@ -144,6 +146,7 @@ tab2tmp/1, tab2dcd/1, tab2dcl/1, + tab2logtmp/1, to_list/1, union/2, uniq/1, @@ -151,6 +154,8 @@ unset/1, %% update_counter/2, val/1, + validate_key/2, + validate_record/2, vcore/0, vcore/1, verbose/2, @@ -319,15 +324,41 @@ tab2dcd(Tab) -> %% Disc copies data tab2dcl(Tab) -> %% Disc copies log dir(lists:concat([Tab, ".DCL"])). +tab2logtmp(Tab) -> %% Disc copies log + dir(lists:concat([Tab, ".LOGTMP"])). + storage_type_at_node(Node, Tab) -> search_key(Node, [{disc_copies, val({Tab, disc_copies})}, {ram_copies, val({Tab, ram_copies})}, - {disc_only_copies, val({Tab, disc_only_copies})}]). + {disc_only_copies, val({Tab, disc_only_copies})}| + wrap_external(val({Tab, external_copies}))]). cs_to_storage_type(Node, Cs) -> search_key(Node, [{disc_copies, Cs#cstruct.disc_copies}, {ram_copies, Cs#cstruct.ram_copies}, - {disc_only_copies, Cs#cstruct.disc_only_copies}]). + {disc_only_copies, Cs#cstruct.disc_only_copies} | + wrap_external(Cs#cstruct.external_copies)]). + +-define(native(T), T==ram_copies; T==disc_copies; T==disc_only_copies). + +semantics({ext,Alias,Mod}, Item) -> + Mod:semantics(Alias, Item); +semantics({Alias,Mod}, Item) -> + Mod:semantics(Alias, Item); +semantics(Type, storage) when ?native(Type) -> + Type; +semantics(Type, types) when ?native(Type) -> + [set, ordered_set, bag]; +semantics(disc_only_copies, index_types) -> + [bag]; +semantics(Type, index_types) when ?native(Type) -> + [bag, ordered]; +semantics(_, _) -> + undefined. + + +wrap_external(L) -> + [{{ext,Alias,Mod},Ns} || {{Alias,Mod},Ns} <- L]. schema_cs_to_storage_type(Node, Cs) -> case cs_to_storage_type(Node, Cs) of @@ -335,7 +366,6 @@ schema_cs_to_storage_type(Node, Cs) -> Other -> Other end. - search_key(Key, [{Val, List} | Tail]) -> case lists:member(Key, List) of true -> Val; @@ -344,6 +374,33 @@ search_key(Key, [{Val, List} | Tail]) -> search_key(_Key, []) -> unknown. +validate_key(Tab, Key) -> + case ?catch_val({Tab, record_validation}) of + {RecName, Arity, Type} -> + {RecName, Arity, Type}; + {RecName, Arity, Type, Alias, Mod} -> + %% external type + Mod:validate_key(Alias, Tab, RecName, Arity, Type, Key); + {'EXIT', _} -> + mnesia:abort({no_exists, Tab}) + end. + + +validate_record(Tab, Obj) -> + case ?catch_val({Tab, record_validation}) of + {RecName, Arity, Type} + when tuple_size(Obj) == Arity, RecName == element(1, Obj) -> + {RecName, Arity, Type}; + {RecName, Arity, Type, Alias, Mod} + when tuple_size(Obj) == Arity, RecName == element(1, Obj) -> + %% external type + Mod:validate_record(Alias, Tab, RecName, Arity, Type, Obj); + {'EXIT', _} -> + mnesia:abort({no_exists, Tab}); + _ -> + mnesia:abort({bad_type, Obj}) + end. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% ops, we've got some global variables here :-) @@ -410,9 +467,9 @@ pr_other(Var) -> no -> {node_not_running, node()}; _ -> {no_exists, Var} end, - verbose("~p (~p) val(mnesia_gvar, ~w) -> ~p ~n", + verbose("~p (~p) val(mnesia_gvar, ~w) -> ~p ~p ~n", [self(), process_info(self(), registered_name), - Var, Why]), + Var, Why, erlang:get_stacktrace()]), mnesia:abort(Why). %% Some functions for list valued variables @@ -549,10 +606,16 @@ read_counter(Name) -> ?ets_lookup_element(mnesia_stats, Name, 2). cs_to_nodes(Cs) -> + ext_nodes(Cs#cstruct.external_copies) ++ Cs#cstruct.disc_only_copies ++ Cs#cstruct.disc_copies ++ Cs#cstruct.ram_copies. +ext_nodes(Ext) -> + lists:flatmap(fun({_, Ns}) -> + Ns + end, Ext). + overload_types() -> [mnesia_tm, mnesia_dump_log]. @@ -748,7 +811,7 @@ view(File) -> true -> view(File, dat); false -> - case suffix([".LOG", ".BUP", ".ETS"], File) of + case suffix([".LOG", ".BUP", ".ETS", ".LOGTMP"], File) of true -> view(File, log); false -> @@ -1044,18 +1107,24 @@ db_get(Tab, Key) -> db_get(val({Tab, storage_type}), Tab, Key). db_get(ram_copies, Tab, Key) -> ?ets_lookup(Tab, Key); db_get(disc_copies, Tab, Key) -> ?ets_lookup(Tab, Key); -db_get(disc_only_copies, Tab, Key) -> dets:lookup(Tab, Key). +db_get(disc_only_copies, Tab, Key) -> dets:lookup(Tab, Key); +db_get({ext, Alias, Mod}, Tab, Key) -> + Mod:lookup(Alias, Tab, Key). db_init_chunk(Tab) -> db_init_chunk(val({Tab, storage_type}), Tab, 1000). db_init_chunk(Tab, N) -> db_init_chunk(val({Tab, storage_type}), Tab, N). +db_init_chunk({ext, Alias, Mod}, Tab, N) -> + Mod:select(Alias, Tab, [{'_', [], ['$_']}], N); db_init_chunk(disc_only_copies, Tab, N) -> dets:select(Tab, [{'_', [], ['$_']}], N); db_init_chunk(_, Tab, N) -> ets:select(Tab, [{'_', [], ['$_']}], N). +db_chunk({ext, _Alias, Mod}, State) -> + Mod:select(State); db_chunk(disc_only_copies, State) -> dets:select(State); db_chunk(_, State) -> @@ -1066,7 +1135,9 @@ db_put(Tab, Val) -> db_put(ram_copies, Tab, Val) -> ?ets_insert(Tab, Val), ok; db_put(disc_copies, Tab, Val) -> ?ets_insert(Tab, Val), ok; -db_put(disc_only_copies, Tab, Val) -> dets:insert(Tab, Val). +db_put(disc_only_copies, Tab, Val) -> dets:insert(Tab, Val); +db_put({ext, Alias, Mod}, Tab, Val) -> + Mod:insert(Alias, Tab, Val). db_match_object(Tab, Pat) -> db_match_object(val({Tab, storage_type}), Tab, Pat). @@ -1075,12 +1146,36 @@ db_match_object(Storage, Tab, Pat) -> try case Storage of disc_only_copies -> dets:match_object(Tab, Pat); + {ext, Alias, Mod} -> Mod:select(Alias, Tab, [{Pat, [], ['$_']}]); _ -> ets:match_object(Tab, Pat) end after db_fixtable(Storage, Tab, false) end. +db_foldl(Fun, Acc, Tab) -> + db_foldl(val({Tab, storage_type}), Fun, Acc, Tab). + +db_foldl(Storage, Fun, Acc, Tab) -> + Limit = mnesia_monitor:get_env(fold_chunk_size), + db_foldl(Storage, Fun, Acc, Tab, [{'_', [], ['$_']}], Limit). + +db_foldl(ram_copies, Fun, Acc, Tab, Pat, Limit) -> + mnesia_lib:db_fixtable(ram_copies, Tab, true), + try select_foldl(db_select_init(ram_copies, Tab, Pat, Limit), + Fun, Acc, ram_copies) + after + mnesia_lib:db_fixtable(ram_copies, Tab, false) + end; +db_foldl(Storage, Fun, Acc, Tab, Pat, Limit) -> + select_foldl(mnesia_lib:db_select_init(Storage, Tab, Pat, Limit), Fun, Acc, Storage). + +select_foldl({Objs, Cont}, Fun, Acc, Storage) -> + select_foldl(mnesia_lib:db_select_cont(Storage, Cont, []), + Fun, lists:foldl(Fun, Acc, Objs), Storage); +select_foldl('$end_of_table', _, Acc, _) -> + Acc. + db_select(Tab, Pat) -> db_select(val({Tab, storage_type}), Tab, Pat). @@ -1089,17 +1184,33 @@ db_select(Storage, Tab, Pat) -> try case Storage of disc_only_copies -> dets:select(Tab, Pat); + {ext, Alias, Mod} -> Mod:select(Alias, Tab, Pat); _ -> ets:select(Tab, Pat) end after db_fixtable(Storage, Tab, false) end. +db_select_init({ext, Alias, Mod}, Tab, Pat, Limit) -> + case Mod:select(Alias, Tab, Pat, Limit) of + {Matches, Continuation} when is_list(Matches) -> + {Matches, {Alias, Continuation}}; + R -> + R + end; db_select_init(disc_only_copies, Tab, Pat, Limit) -> dets:select(Tab, Pat, Limit); db_select_init(_, Tab, Pat, Limit) -> ets:select(Tab, Pat, Limit). +db_select_cont({ext, Alias, Mod}, Cont0, Ms) -> + Cont = Mod:repair_continuation(Cont0, Ms), + case Mod:select(Cont) of + {Matches, Continuation} when is_list(Matches) -> + {Matches, {Alias, Continuation}}; + R -> + R + end; db_select_cont(disc_only_copies, Cont0, Ms) -> Cont = dets:repair_continuation(Cont0, Ms), dets:select(Cont); @@ -1116,13 +1227,18 @@ db_fixtable(disc_copies, Tab, Bool) -> db_fixtable(dets, Tab, Bool) -> dets:safe_fixtable(Tab, Bool); db_fixtable(disc_only_copies, Tab, Bool) -> - dets:safe_fixtable(Tab, Bool). + dets:safe_fixtable(Tab, Bool); +db_fixtable({ext, Alias, Mod}, Tab, Bool) -> + Mod:fixtable(Alias, Tab, Bool). db_erase(Tab, Key) -> db_erase(val({Tab, storage_type}), Tab, Key). db_erase(ram_copies, Tab, Key) -> ?ets_delete(Tab, Key), ok; db_erase(disc_copies, Tab, Key) -> ?ets_delete(Tab, Key), ok; -db_erase(disc_only_copies, Tab, Key) -> dets:delete(Tab, Key). +db_erase(disc_only_copies, Tab, Key) -> dets:delete(Tab, Key); +db_erase({ext, Alias, Mod}, Tab, Key) -> + Mod:delete(Alias, Tab, Key), + ok. db_match_erase(Tab, '_') -> db_delete_all(val({Tab, storage_type}),Tab); @@ -1130,7 +1246,10 @@ db_match_erase(Tab, Pat) -> db_match_erase(val({Tab, storage_type}), Tab, Pat). db_match_erase(ram_copies, Tab, Pat) -> ?ets_match_delete(Tab, Pat), ok; db_match_erase(disc_copies, Tab, Pat) -> ?ets_match_delete(Tab, Pat), ok; -db_match_erase(disc_only_copies, Tab, Pat) -> dets:match_delete(Tab, Pat). +db_match_erase(disc_only_copies, Tab, Pat) -> dets:match_delete(Tab, Pat); +db_match_erase({ext, Alias, Mod}, Tab, Pat) -> + Mod:match_delete(Alias, Tab, Pat), + ok. db_delete_all(ram_copies, Tab) -> ets:delete_all_objects(Tab); db_delete_all(disc_copies, Tab) -> ets:delete_all_objects(Tab); @@ -1140,31 +1259,41 @@ db_first(Tab) -> db_first(val({Tab, storage_type}), Tab). db_first(ram_copies, Tab) -> ?ets_first(Tab); db_first(disc_copies, Tab) -> ?ets_first(Tab); -db_first(disc_only_copies, Tab) -> dets:first(Tab). +db_first(disc_only_copies, Tab) -> dets:first(Tab); +db_first({ext, Alias, Mod}, Tab) -> + Mod:first(Alias, Tab). db_next_key(Tab, Key) -> db_next_key(val({Tab, storage_type}), Tab, Key). db_next_key(ram_copies, Tab, Key) -> ?ets_next(Tab, Key); db_next_key(disc_copies, Tab, Key) -> ?ets_next(Tab, Key); -db_next_key(disc_only_copies, Tab, Key) -> dets:next(Tab, Key). +db_next_key(disc_only_copies, Tab, Key) -> dets:next(Tab, Key); +db_next_key({ext, Alias, Mod}, Tab, Key) -> + Mod:next(Alias, Tab, Key). db_last(Tab) -> db_last(val({Tab, storage_type}), Tab). db_last(ram_copies, Tab) -> ?ets_last(Tab); db_last(disc_copies, Tab) -> ?ets_last(Tab); -db_last(disc_only_copies, Tab) -> dets:first(Tab). %% Dets don't have order +db_last(disc_only_copies, Tab) -> dets:first(Tab); %% Dets don't have order +db_last({ext, Alias, Mod}, Tab) -> + Mod:last(Alias, Tab). db_prev_key(Tab, Key) -> db_prev_key(val({Tab, storage_type}), Tab, Key). db_prev_key(ram_copies, Tab, Key) -> ?ets_prev(Tab, Key); db_prev_key(disc_copies, Tab, Key) -> ?ets_prev(Tab, Key); -db_prev_key(disc_only_copies, Tab, Key) -> dets:next(Tab, Key). %% Dets don't have order +db_prev_key(disc_only_copies, Tab, Key) -> dets:next(Tab, Key); %% Dets don't have order +db_prev_key({ext, Alias, Mod}, Tab, Key) -> + Mod:prev(Alias, Tab, Key). db_slot(Tab, Pos) -> db_slot(val({Tab, storage_type}), Tab, Pos). db_slot(ram_copies, Tab, Pos) -> ?ets_slot(Tab, Pos); db_slot(disc_copies, Tab, Pos) -> ?ets_slot(Tab, Pos); -db_slot(disc_only_copies, Tab, Pos) -> dets:slot(Tab, Pos). +db_slot(disc_only_copies, Tab, Pos) -> dets:slot(Tab, Pos); +db_slot({ext, Alias, Mod}, Tab, Pos) -> + Mod:slot(Alias, Tab, Pos). db_update_counter(Tab, C, Val) -> db_update_counter(val({Tab, storage_type}), Tab, C, Val). @@ -1173,13 +1302,16 @@ db_update_counter(ram_copies, Tab, C, Val) -> db_update_counter(disc_copies, Tab, C, Val) -> ?ets_update_counter(Tab, C, Val); db_update_counter(disc_only_copies, Tab, C, Val) -> - dets:update_counter(Tab, C, Val). + dets:update_counter(Tab, C, Val); +db_update_counter({ext, Alias, Mod}, Tab, C, Val) -> + Mod:update_counter(Alias, Tab, C, Val). db_erase_tab(Tab) -> db_erase_tab(val({Tab, storage_type}), Tab). db_erase_tab(ram_copies, Tab) -> ?ets_delete_table(Tab); db_erase_tab(disc_copies, Tab) -> ?ets_delete_table(Tab); -db_erase_tab(disc_only_copies, _Tab) -> ignore. +db_erase_tab(disc_only_copies, _Tab) -> ignore; +db_erase_tab({ext, _Alias, _Mod}, _Tab) -> ignore. %% assuming that Tab is a valid ets-table dets_to_ets(Tabname, Tab, File, Type, Rep, Lock) -> diff --git a/lib/mnesia/src/mnesia_loader.erl b/lib/mnesia/src/mnesia_loader.erl index bde5585139..71e5829c87 100644 --- a/lib/mnesia/src/mnesia_loader.erl +++ b/lib/mnesia/src/mnesia_loader.erl @@ -147,6 +147,19 @@ do_get_disc_copy2(Tab, Reason, Storage, Type) when Storage == disc_only_copies - {error, Error} -> {not_loaded, {"Failed to create dets table", Error}} end + end; + +do_get_disc_copy2(Tab, Reason, Storage = {ext, Alias, Mod}, _Type) -> + Cs = val({Tab, cstruct}), + case mnesia_monitor:unsafe_create_external(Tab, Alias, Mod, Cs) of + ok -> + ok = ext_load_table(Mod, Alias, Tab, Reason), + mnesia_index:init_index(Tab, Storage), + set({Tab, load_node}, node()), + set({Tab, load_reason}, Reason), + {loaded, ok}; + Other -> + {not_loaded, Other} end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -259,7 +272,7 @@ init_receiver(Node, Tab,Storage,Cs,Reason) -> true = lists:member(Node, Active), {SenderPid, TabSize, DetsData} = start_remote_sender(Node,Tab,Storage), - Init = table_init_fun(SenderPid), + Init = table_init_fun(SenderPid, Storage), Args = [self(),Tab,Storage,Cs,SenderPid, TabSize,DetsData,Init], Pid = spawn_link(?MODULE, spawned_receiver, Args), @@ -289,7 +302,12 @@ start_remote_sender(Node,Tab,Storage) -> mnesia_controller:start_remote_sender(Node, Tab, self(), Storage), put(mnesia_table_sender_node, {Tab, Node}), receive - {SenderPid, {first, TabSize}} -> + {SenderPid, {first, _} = Msg} + when is_pid(SenderPid), element(1, Storage) == ext -> + {ext, Alias, Mod} = Storage, + {Sz, Data} = Mod:receiver_first_message(SenderPid, Msg, Alias, Tab), + {SenderPid, Sz, Data}; + {SenderPid, {first, TabSize}} =_M1 -> {SenderPid, TabSize, false}; {SenderPid, {first, TabSize, DetsData}} -> {SenderPid, TabSize, DetsData}; @@ -299,16 +317,18 @@ start_remote_sender(Node,Tab,Storage) -> down(Tab, Storage) end. -table_init_fun(SenderPid) -> +table_init_fun(SenderPid, Storage) -> fun(read) -> Receiver = self(), SenderPid ! {Receiver, more}, - get_data(SenderPid, Receiver) + get_data(SenderPid, Receiver, Storage); + (close) -> + ok end. %% Add_table_copy get's it's own locks. start_receiver(Tab,Storage,Cs,SenderPid,TabSize,DetsData,{dumper,{add_table_copy,_}}) -> - Init = table_init_fun(SenderPid), + Init = table_init_fun(SenderPid, Storage), case do_init_table(Tab,Storage,Cs,SenderPid,TabSize,DetsData,self(), Init) of Err = {error, _} -> SenderPid ! {copier_done, node()}, @@ -391,7 +411,15 @@ create_table(Tab, TabSize, Storage, Cs) -> {Storage, Tab}; Else -> Else - end + end; + element(1, Storage) == ext -> + {_, Alias, Mod} = Storage, + case mnesia_monitor:unsafe_create_external(Tab, Alias, Mod, Cs) of + ok -> + {Storage, Tab}; + Else -> + Else + end end. tab_receiver(Node, Tab, Storage, Cs, OrigTabRec) -> @@ -409,21 +437,25 @@ tab_receiver(Node, Tab, Storage, Cs, OrigTabRec) -> tab_receiver(Node, Tab, Storage, Cs, OrigTabRec) end. -make_table_fun(Pid, TabRec) -> +make_table_fun(Pid, TabRec, Storage) -> fun(close) -> ok; + ({read, Msg}) -> + Pid ! {TabRec, Msg}, + get_data(Pid, TabRec, Storage); (read) -> - get_data(Pid, TabRec) + get_data(Pid, TabRec, Storage) end. -get_data(Pid, TabRec) -> +get_data(Pid, TabRec, Storage) -> receive {Pid, {more_z, CompressedRecs}} when is_binary(CompressedRecs) -> - Pid ! {TabRec, more}, - {zlib_uncompress(CompressedRecs), make_table_fun(Pid,TabRec)}; + maybe_reply(Pid, {TabRec, more}, Storage), + {zlib_uncompress(CompressedRecs), + make_table_fun(Pid, TabRec, Storage)}; {Pid, {more, Recs}} -> - Pid ! {TabRec, more}, - {Recs, make_table_fun(Pid,TabRec)}; + maybe_reply(Pid, {TabRec, more}, Storage), + {Recs, make_table_fun(Pid, TabRec, Storage)}; {Pid, no_more} -> end_of_input; {copier_done, Node} -> @@ -431,13 +463,48 @@ get_data(Pid, TabRec) -> Node -> {copier_done, Node}; _ -> - get_data(Pid, TabRec) + get_data(Pid, TabRec, Storage) end; {'EXIT', Pid, Reason} -> handle_exit(Pid, Reason), - get_data(Pid, TabRec) + get_data(Pid, TabRec, Storage) end. +maybe_reply(_, _, {ext, _, _}) -> + ignore; +maybe_reply(Pid, Msg, _) -> + Pid ! Msg. + +ext_init_table(Alias, Mod, Tab, Fun, State, Sender) -> + ok = ext_load_table(Mod, Alias, Tab, {net_load, node(Sender)}), + ext_init_table(read, Alias, Mod, Tab, Fun, State, Sender). + +ext_load_table(Mod, Alias, Tab, Reason) -> + CS = val({Tab, cstruct}), + Mod:load_table(Alias, Tab, Reason, mnesia_schema:cs2list(CS)). + + +ext_init_table(Action, Alias, Mod, Tab, Fun, State, Sender) -> + case Fun(Action) of + {copier_done, Node} -> + verbose("Receiver of table ~p crashed on ~p (more)~n", [Tab, Node]), + down(Tab, {ext,Alias,Mod}); + {Data, NewFun} -> + case Mod:receive_data(Data, Alias, Tab, Sender, State) of + {more, NewState} -> + ext_init_table({read, more}, Alias, Mod, + Tab, NewFun, NewState, Sender); + {{more,Msg}, NewState} -> + ext_init_table({read, Msg}, Alias, Mod, + Tab, NewFun, NewState, Sender) + end; + end_of_input -> + Mod:receive_done(Alias, Tab, Sender, State), + ok = Fun(close) + end. + +init_table(Tab, {ext,Alias,Mod}, Fun, State, Sender) -> + ext_init_table(Alias, Mod, Tab, Fun, State, Sender); init_table(Tab, disc_only_copies, Fun, DetsInfo,Sender) -> ErtsVer = erlang:system_info(version), case DetsInfo of @@ -567,7 +634,10 @@ handle_last({ram_copies, Tab}, _Type, DatBin) -> ok; false -> ok - end. + end; + +handle_last(_Storage, _Type, nobin) -> + ok. down(Tab, Storage) -> case Storage of @@ -578,7 +648,10 @@ down(Tab, Storage) -> disc_only_copies -> TmpFile = mnesia_lib:tab2tmp(Tab), mnesia_lib:dets_sync_close(Tab), - file:delete(TmpFile) + file:delete(TmpFile); + {ext, Alias, Mod} -> + catch Mod:close_table(Alias, Tab), + catch Mod:delete_table(Alias, Tab) end, mnesia_checkpoint:tm_del_copy(Tab, node()), mnesia_controller:sync_del_table_copy_whereabouts(Tab, node()), @@ -603,21 +676,35 @@ db_erase({ram_copies, Tab}, Key) -> db_erase({disc_copies, Tab}, Key) -> true = ?ets_delete(Tab, Key); db_erase({disc_only_copies, Tab}, Key) -> - ok = dets:delete(Tab, Key). + ok = dets:delete(Tab, Key); +db_erase({{ext, Alias, Mod}, Tab}, Key) -> + ok = Mod:delete(Alias, Tab, Key). db_match_erase({ram_copies, Tab} , Pat) -> true = ?ets_match_delete(Tab, Pat); db_match_erase({disc_copies, Tab} , Pat) -> true = ?ets_match_delete(Tab, Pat); db_match_erase({disc_only_copies, Tab}, Pat) -> - ok = dets:match_delete(Tab, Pat). + ok = dets:match_delete(Tab, Pat); +db_match_erase({{ext, Alias, Mod}, Tab}, Pat) -> + % "ets style" is to return true + % "dets style" is to return N | { error, Reason } + % or sometimes ok (?) + % be nice and accept both + case Mod:match_delete(Alias, Tab, Pat) of + N when is_integer (N) -> ok; + true -> ok; + ok -> ok + end. db_put({ram_copies, Tab}, Val) -> true = ?ets_insert(Tab, Val); db_put({disc_copies, Tab}, Val) -> true = ?ets_insert(Tab, Val); db_put({disc_only_copies, Tab}, Val) -> - ok = dets:insert(Tab, Val). + ok = dets:insert(Tab, Val); +db_put({{ext, Alias, Mod}, Tab}, Val) -> + ok = Mod:insert(Alias, Tab, Val). %% This code executes at the remote site where the data is %% executes in a special copier process. @@ -636,47 +723,62 @@ send_table(Pid, Tab, RemoteS) -> unknown -> {error, {no_exists, Tab}}; Storage -> - %% Send first - TabSize = mnesia:table_info(Tab, size), - KeysPerTransfer = calc_nokeys(Storage, Tab), - ChunkData = dets:info(Tab, bchunk_format), - - UseDetsChunk = - Storage == RemoteS andalso - Storage == disc_only_copies andalso - ChunkData /= undefined, - if - UseDetsChunk == true -> - DetsInfo = erlang:system_info(version), - Pid ! {self(), {first, TabSize, {DetsInfo, ChunkData}}}; - true -> - Pid ! {self(), {first, TabSize}} - end, + do_send_table(Pid, Tab, Storage, RemoteS) + end. - %% Debug info - put(mnesia_table_sender, {Tab, node(Pid), Pid}), - {Init, Chunk} = reader_funcs(UseDetsChunk, Tab, Storage, KeysPerTransfer), - - SendIt = fun() -> - NeedLock = need_lock(Tab), - {atomic, ok} = prepare_copy(Pid, Tab, Storage, NeedLock), - send_more(Pid, 1, Chunk, Init(), Tab), - finish_copy(Pid, Tab, Storage, RemoteS, NeedLock) - end, - - try SendIt() of - {_, receiver_died} -> ok; - {atomic, no_more} -> ok - catch - throw:receiver_died -> - cleanup_tab_copier(Pid, Storage, Tab), - ok; - error:Reason -> %% Prepare failed - cleanup_tab_copier(Pid, Storage, Tab), - {error, {tab_copier, Tab, {Reason, erlang:get_stacktrace()}}} - after - unlink(whereis(mnesia_tm)) - end +do_send_table(Pid, Tab, Storage, RemoteS) -> + {Init, Chunk} = + case Storage of + {ext, Alias, Mod} -> + case Mod:sender_init(Alias, Tab, RemoteS, Pid) of + {standard, I, C} -> + Pid ! {self(), {first, Mod:info(Alias, Tab, size)}}, + {I, C}; + {_, _} = Res -> + Res + end; + Storage -> + %% Send first + TabSize = mnesia:table_info(Tab, size), + KeysPerTransfer = calc_nokeys(Storage, Tab), + ChunkData = dets:info(Tab, bchunk_format), + + UseDetsChunk = + Storage == RemoteS andalso + Storage == disc_only_copies andalso + ChunkData /= undefined, + if + UseDetsChunk == true -> + DetsInfo = erlang:system_info(version), + Pid ! {self(), {first, TabSize, {DetsInfo, ChunkData}}}; + true -> + Pid ! {self(), {first, TabSize}} + end, + {_I, _C} = + reader_funcs(UseDetsChunk, Tab, Storage, KeysPerTransfer) + end, + %% Debug info + put(mnesia_table_sender, {Tab, node(Pid), Pid}), + + SendIt = fun() -> + NeedLock = need_lock(Tab), + {atomic, ok} = prepare_copy(Pid, Tab, Storage, NeedLock), + send_more(Pid, 1, Chunk, Init(), Tab, Storage), + finish_copy(Pid, Tab, Storage, RemoteS, NeedLock) + end, + + try SendIt() of + {_, receiver_died} -> ok; + {atomic, no_more} -> ok + catch + throw:receiver_died -> + cleanup_tab_copier(Pid, Storage, Tab), + ok; + error:Reason -> %% Prepare failed + cleanup_tab_copier(Pid, Storage, Tab), + {error, {tab_copier, Tab, {Reason, erlang:get_stacktrace()}}} + after + unlink(whereis(mnesia_tm)) end. prepare_copy(Pid, Tab, Storage, NeedLock) -> @@ -726,20 +828,32 @@ update_where_to_write([H|T], Tab, AddNode) -> [{update_where_to_write, [add, Tab, AddNode], self()}]), update_where_to_write(T, Tab, AddNode). -send_more(Pid, N, Chunk, DataState, Tab) -> +send_more(Pid, N, Chunk, DataState, Tab, Storage) -> receive {NewPid, more} -> case send_packet(N - 1, NewPid, Chunk, DataState) of New when is_integer(New) -> New - 1; NewData -> - send_more(NewPid, ?MAX_NOPACKETS, Chunk, NewData, Tab) + send_more(NewPid, ?MAX_NOPACKETS, Chunk, NewData, + Tab, Storage) + end; + {NewPid, {more, Msg}} when element(1, Storage) == ext -> + {ext, Alias, Mod} = Storage, + {NewChunk, NewState} = + Mod:sender_handle_info(Msg, Alias, Tab, NewPid, DataState), + case send_packet(N - 1, NewPid, NewChunk, NewState) of + New when is_integer(New) -> + New -1; + NewData -> + send_more(NewPid, N, NewChunk, NewData, Tab, + Storage) end; {_NewPid, {old_protocol, Tab}} -> Storage = val({Tab, storage_type}), {Init, NewChunk} = reader_funcs(false, Tab, Storage, calc_nokeys(Storage, Tab)), - send_more(Pid, 1, NewChunk, Init(), Tab); + send_more(Pid, 1, NewChunk, Init(), Tab, Storage); {copier_done, Node} when Node == node(Pid)-> verbose("Receiver of table ~p crashed on ~p (more)~n", [Tab, Node]), diff --git a/lib/mnesia/src/mnesia_log.erl b/lib/mnesia/src/mnesia_log.erl index 36135418c8..9536effd42 100644 --- a/lib/mnesia/src/mnesia_log.erl +++ b/lib/mnesia/src/mnesia_log.erl @@ -224,17 +224,12 @@ sappend(Log, Term) -> ok = disk_log:log(Log, Term). %% Write commit records to the latest_log -log(C) when C#commit.disc_copies == [], - C#commit.disc_only_copies == [], - C#commit.schema_ops == [] -> - ignore; log(C) -> - case mnesia_monitor:use_dir() of + case need_log(C) andalso mnesia_monitor:use_dir() of true -> if is_record(C, commit) -> - C2 = C#commit{ram_copies = [], snmp = []}, - append(latest_log, C2); + append(latest_log, strip_snmp(C)); true -> %% Either a commit record as binary %% or some decision related info @@ -247,17 +242,12 @@ log(C) -> %% Synced -slog(C) when C#commit.disc_copies == [], - C#commit.disc_only_copies == [], - C#commit.schema_ops == [] -> - ignore; slog(C) -> - case mnesia_monitor:use_dir() of + case need_log(C) andalso mnesia_monitor:use_dir() of true -> if is_record(C, commit) -> - C2 = C#commit{ram_copies = [], snmp = []}, - sappend(latest_log, C2); + sappend(latest_log, strip_snmp(C)); true -> %% Either a commit record as binary %% or some decision related info @@ -268,6 +258,13 @@ slog(C) -> ignore end. +need_log(#commit{disc_copies=[], disc_only_copies=[], schema_ops=[], ext=Ext}) -> + lists:keymember(ext_copies, 1, Ext); +need_log(_) -> true. + +strip_snmp(#commit{ext=[]}=CR) -> CR; +strip_snmp(#commit{ext=Ext}=CR) -> + CR#commit{ext=lists:keydelete(snmp, 1, Ext)}. %% Stuff related to the file LOG diff --git a/lib/mnesia/src/mnesia_monitor.erl b/lib/mnesia/src/mnesia_monitor.erl index 4069341700..ab78c9b13e 100644 --- a/lib/mnesia/src/mnesia_monitor.erl +++ b/lib/mnesia/src/mnesia_monitor.erl @@ -32,6 +32,7 @@ init/0, mktab/2, unsafe_mktab/2, + unsafe_create_external/4, mnesia_down/2, needs_protocol_conversion/1, negotiate_protocol/1, @@ -82,9 +83,9 @@ going_down = [], tm_started = false, early_connects = [], connecting, mq = [], remote_node_status = []}). --define(current_protocol_version, {8,2}). +-define(current_protocol_version, {8,3}). --define(previous_protocol_version, {8,1}). +-define(previous_protocol_version, {8,2}). start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, @@ -129,6 +130,8 @@ close_log(Name) -> unsafe_close_log(Name) -> unsafe_call({unsafe_close_log, Name}). +unsafe_create_external(Tab, Alias, Mod, Cs) -> + unsafe_call({unsafe_create_external, Tab, Alias, Mod, Cs}). disconnect(Node) -> cast({disconnect, Node}). @@ -193,7 +196,7 @@ protocol_version() -> %% A sorted list of acceptable protocols the %% preferred protocols are first in the list acceptable_protocol_versions() -> - [protocol_version(), ?previous_protocol_version]. + [protocol_version(), ?previous_protocol_version, {8,1}]. needs_protocol_conversion(Node) -> case {?catch_val({protocol, Node}), protocol_version()} of @@ -405,6 +408,14 @@ handle_call({unsafe_close_log, Name}, _From, State) -> _ = disk_log:close(Name), {reply, ok, State}; +handle_call({unsafe_create_external, Tab, Alias, Mod, Cs}, _From, State) -> + case catch Mod:create_table(Alias, Tab, mnesia_schema:cs2list(Cs)) of + {'EXIT', ExitReason} -> + {reply, {error, ExitReason}, State}; + Reply -> + {reply, Reply, State} + end; + handle_call({negotiate_protocol, Mon, _Version, _Protocols}, _From, State) when State#state.tm_started == false -> State2 = State#state{early_connects = [node(Mon) | State#state.early_connects]}, @@ -658,6 +669,7 @@ get_env(E) -> env() -> [ access_module, + allow_index_on_key, auto_repair, backup_module, debug, @@ -671,19 +683,23 @@ env() -> extra_db_nodes, ignore_fallback_at_startup, fallback_error_function, + fold_chunk_size, max_wait_for_decision, schema_location, core_dir, pid_sort_order, no_table_loaders, dc_dump_limit, - send_compressed + send_compressed, + schema ]. default_env(access_module) -> mnesia; default_env(auto_repair) -> true; +default_env(allow_index_on_key) -> + false; default_env(backup_module) -> mnesia_backup; default_env(debug) -> @@ -709,6 +725,8 @@ default_env(ignore_fallback_at_startup) -> false; default_env(fallback_error_function) -> {mnesia, lkill}; +default_env(fold_chunk_size) -> + 100; default_env(max_wait_for_decision) -> infinity; default_env(schema_location) -> @@ -722,7 +740,9 @@ default_env(no_table_loaders) -> default_env(dc_dump_limit) -> 4; default_env(send_compressed) -> - 0. + 0; +default_env(schema) -> + []. check_type(Env, Val) -> try do_check_type(Env, Val) @@ -730,6 +750,7 @@ check_type(Env, Val) -> end. do_check_type(access_module, A) when is_atom(A) -> A; +do_check_type(allow_index_on_key, B) -> bool(B); do_check_type(auto_repair, B) -> bool(B); do_check_type(backup_module, B) when is_atom(B) -> B; do_check_type(debug, debug) -> debug; @@ -753,6 +774,8 @@ do_check_type(extra_db_nodes, L) when is_list(L) -> (A) when is_atom(A) -> true end, lists:filter(Fun, L); +do_check_type(fold_chunk_size, I) when is_integer(I), I > 0; + I =:= infinity -> I; do_check_type(max_wait_for_decision, infinity) -> infinity; do_check_type(max_wait_for_decision, I) when is_integer(I), I > 0 -> I; do_check_type(schema_location, M) -> media(M); @@ -766,7 +789,8 @@ do_check_type(pid_sort_order, "standard") -> standard; do_check_type(pid_sort_order, _) -> false; do_check_type(no_table_loaders, N) when is_integer(N), N > 0 -> N; do_check_type(dc_dump_limit,N) when is_number(N), N > 0 -> N; -do_check_type(send_compressed, L) when is_integer(L), L >= 0, L =< 9 -> L. +do_check_type(send_compressed, L) when is_integer(L), L >= 0, L =< 9 -> L; +do_check_type(schema, L) when is_list(L) -> L. bool(true) -> true; bool(false) -> false. diff --git a/lib/mnesia/src/mnesia_schema.erl b/lib/mnesia/src/mnesia_schema.erl index 782493fb4f..0e4017e4c3 100644 --- a/lib/mnesia/src/mnesia_schema.erl +++ b/lib/mnesia/src/mnesia_schema.erl @@ -29,6 +29,16 @@ -module(mnesia_schema). -export([ + add_backend_type/2, + do_add_backend_type/2, + delete_backend_type/1, + do_delete_backend_type/1, + backend_types/0, + add_index_plugin/3, + do_add_index_plugin/3, + delete_index_plugin/1, + do_delete_index_plugin/1, + index_plugins/0, add_snmp/2, add_table_copy/3, add_table_index/2, @@ -55,14 +65,16 @@ dump_tables/1, ensure_no_schema/1, get_create_list/1, - get_initial_schema/2, + get_initial_schema/3, get_table_properties/1, info/0, info/1, init/1, + init_backends/0, insert_cstruct/3, is_remote_member/1, list2cs/1, + list2cs/2, lock_schema/0, merge_schema/0, merge_schema/1, @@ -110,7 +122,8 @@ do_delete_table/1, do_read_table_property/2, do_delete_table_property/2, - do_write_table_property/2]). + do_write_table_property/2, + do_change_table_copy_type/3]). -include("mnesia.hrl"). -include_lib("kernel/include/file.hrl"). @@ -138,12 +151,30 @@ init(IgnoreFallback) -> verbose("Schema initiated from: ~p~n", [Source]), set({schema, tables}, []), set({schema, local_tables}, []), + do_set_schema(schema), Tabs = set_schema(?ets_first(schema)), lists:foreach(fun(Tab) -> clear_whereabouts(Tab) end, Tabs), set({schema, where_to_read}, node()), set({schema, load_node}, node()), set({schema, load_reason}, initial), - mnesia_controller:add_active_replica(schema, node()). + mnesia_controller:add_active_replica(schema, node()), + init_backends(). + + +init_backends() -> + Backends = lists:foldl(fun({Alias, Mod}, Acc) -> + orddict:append(Mod, Alias, Acc) + end, orddict:new(), get_ext_types()), + [init_backend(Mod, Aliases) || {Mod, Aliases} <- Backends], + ok. + +init_backend(Mod, [_|_] = Aliases) -> + case Mod:init_backend() of + ok -> + Mod:add_aliases(Aliases); + Error -> + mnesia:abort({backend_init_error, Error}) + end. exit_on_error({error, Reason}) -> exit(Reason); @@ -180,6 +211,7 @@ do_set_schema(Tab, Cs) -> set({Tab, ram_copies}, Cs#cstruct.ram_copies), set({Tab, disc_copies}, Cs#cstruct.disc_copies), set({Tab, disc_only_copies}, Cs#cstruct.disc_only_copies), + set({Tab, external_copies}, Cs#cstruct.external_copies), set({Tab, load_order}, Cs#cstruct.load_order), set({Tab, access_mode}, Cs#cstruct.access_mode), set({Tab, majority}, Cs#cstruct.majority), @@ -195,15 +227,21 @@ do_set_schema(Tab, Cs) -> set({Tab, arity}, Arity), RecName = Cs#cstruct.record_name, set({Tab, record_name}, RecName), - set({Tab, record_validation}, {RecName, Arity, Type}), set({Tab, wild_pattern}, wild(RecName, Arity)), - set({Tab, index}, Cs#cstruct.index), + set({Tab, index}, [P || {P,_} <- Cs#cstruct.index]), + case Cs#cstruct.index of + [] -> + set({Tab, index_info}, mnesia_index:index_info(Type, [])); + _ -> + ignore + end, %% create actual index tabs later set({Tab, cookie}, Cs#cstruct.cookie), set({Tab, version}, Cs#cstruct.version), set({Tab, cstruct}, Cs), Storage = mnesia_lib:schema_cs_to_storage_type(node(), Cs), set({Tab, storage_type}, Storage), + set_record_validation(Tab, Storage, RecName, Arity, Type), mnesia_lib:add({schema, tables}, Tab), Ns = mnesia_lib:cs_to_nodes(Cs), case lists:member(node(), Ns) of @@ -213,7 +251,24 @@ do_set_schema(Tab, Cs) -> mnesia_lib:add({schema, local_tables}, Tab); false -> ignore - end. + end, + set_ext_types(Tab, get_ext_types(), Cs#cstruct.external_copies). + +set_record_validation(Tab, {ext,Alias,Mod}, RecName, Arity, Type) -> + set({Tab, record_validation}, {RecName, Arity, Type, Alias, Mod}); +set_record_validation(Tab, _, RecName, Arity, Type) -> + set({Tab, record_validation}, {RecName, Arity, Type}). + +set_ext_types(Tab, ExtTypes, ExtCopies) -> + lists:foreach( + fun({Type, _} = Key) -> + Nodes = case lists:keyfind(Key, 1, ExtCopies) of + {_, Ns} -> Ns; + false -> [] + end, + set({Tab, Type}, Nodes) + end, ExtTypes). + wild(RecName, Arity) -> Wp0 = list_to_tuple(lists:duplicate(Arity, '_')), @@ -525,9 +580,14 @@ do_read_disc_schema(Fname, Keep) -> Res. get_initial_schema(SchemaStorage, Nodes) -> + get_initial_schema(SchemaStorage, Nodes, []). + +get_initial_schema(SchemaStorage, Nodes, Properties) -> % + UserProps = initial_schema_properties(Properties), Cs = #cstruct{name = schema, record_name = schema, - attributes = [table, cstruct]}, + attributes = [table, cstruct], + user_properties = UserProps}, Cs2 = case SchemaStorage of ram_copies -> Cs#cstruct{ram_copies = Nodes}; @@ -535,6 +595,35 @@ get_initial_schema(SchemaStorage, Nodes) -> end, cs2list(Cs2). +initial_schema_properties(Props0) -> + DefaultProps = remove_duplicates(mnesia_monitor:get_env(schema)), + Props = lists:foldl( + fun({K,V}, Acc) -> + lists:keystore(K, 1, Acc, {K,V}) + end, DefaultProps, remove_duplicates(Props0)), + initial_schema_properties_(Props). + +initial_schema_properties_([{backend_types, Types}|Props]) -> + lists:foreach(fun({Name, Module}) -> + verify_backend_type(Name, Module) + end, Types), + [{mnesia_backend_types, Types}|initial_schema_properties_(Props)]; +initial_schema_properties_([{index_plugins, Plugins}|Props]) -> + lists:foreach(fun({Name, Module, Function}) -> + verify_index_plugin(Name, Module, Function) + end, Plugins), + [{mnesia_index_plugins, Plugins}|initial_schema_properties_(Props)]; +initial_schema_properties_([P|_Props]) -> + mnesia:abort({bad_schema_property, P}); +initial_schema_properties_([]) -> + []. + +remove_duplicates([{K,_} = H|T]) -> + [H | remove_duplicates([X || {K1,_} = X <- T, + K1 =/= K])]; +remove_duplicates([]) -> + []. + read_cstructs_from_disc() -> %% Assumptions: %% - local schema lock in global @@ -551,8 +640,9 @@ read_cstructs_from_disc() -> {type, set}], case dets:open_file(make_ref(), Args) of {ok, Tab} -> + ExtTypes = get_ext_types_disc(), Fun = fun({_, _, List}) -> - {continue, list2cs(List)} + {continue, list2cs(List, ExtTypes)} end, Cstructs = dets:traverse(Tab, Fun), dets:close(Tab), @@ -632,7 +722,7 @@ do_insert_schema_ops(_Store, []) -> api_list2cs(List) when is_list(List) -> Name = pick(unknown, name, List, must), - Keys = check_keys(Name, List, record_info(fields, cstruct)), + Keys = check_keys(Name, List), check_duplicates(Name, Keys), list2cs(List); api_list2cs(Other) -> @@ -647,7 +737,15 @@ cs2list(Cs) when is_record(Cs, cstruct) -> cs2list(CreateList) when is_list(CreateList) -> CreateList; -%% since 4.6 +cs2list(Cs) when element(1, Cs) == cstruct, tuple_size(Cs) == 20 -> + Tags = [name,type, + ram_copies,disc_copies,disc_only_copies,external_copies, + load_order,access_mode,majority,index,snmp,local_content, + record_name,attributes, + user_properties,frag_properties,storage_properties, + cookie,version], + rec2list(Tags, Tags, 2, Cs); +%% since vsn-4.6 (protocol 8.2 or older) cs2list(Cs) when element(1, Cs) == cstruct, tuple_size(Cs) == 19 -> Tags = [name,type,ram_copies,disc_copies,disc_only_copies, load_order,access_mode,majority,index,snmp,local_content, @@ -657,8 +755,38 @@ cs2list(Cs) when element(1, Cs) == cstruct, tuple_size(Cs) == 19 -> rec2list(Tags, Tags, 2, Cs). cs2list(false, Cs) -> - cs2list(Cs). + cs2list(Cs); +cs2list({8,3}, Cs) -> + cs2list(Cs); +cs2list({8,Minor}, Cs) when Minor =:= 2; Minor =:= 1 -> + Orig = record_info(fields, cstruct), + Tags = [name,type,ram_copies,disc_copies,disc_only_copies, + load_order,access_mode,majority,index,snmp,local_content, + record_name,attributes, + user_properties,frag_properties,storage_properties, + cookie,version], + CsList = rec2list(Tags, Orig, 2, Cs), + case proplists:get_value(index, CsList, []) of + [] -> CsList; + NewFormat -> + OldFormat = [Pos || {Pos, _Pref} <- NewFormat], + lists:keyreplace(index, 1, CsList, {index, OldFormat}) + end. +rec2list([index | Tags], [index|Orig], Pos, Rec) -> + Val = element(Pos, Rec), + [{index, lists:map( + fun({_, _Type}=P) -> P; + (P) when is_integer(P); is_atom(P) -> {P, ordered} + end, Val)} | rec2list(Tags, Orig, Pos + 1, Rec)]; +rec2list([external_copies | Tags], Orig0, Pos, Rec) -> + Orig = case Orig0 of + [external_copies|Rest] -> Rest; + _ -> Orig0 + end, + Val = element(Pos, Rec), + [{Alias, Ns} || {{Alias,_}, Ns} <- Val] + ++ rec2list(Tags, Orig, Pos+1, Rec); rec2list([Tag | Tags], [Tag | Orig], Pos, Rec) -> Val = element(Pos, Rec), [{Tag, Val} | rec2list(Tags, Orig, Pos + 1, Rec)]; @@ -667,17 +795,33 @@ rec2list([], _, _Pos, _Rec) -> rec2list(Tags, [_|Orig], Pos, Rec) -> rec2list(Tags, Orig, Pos+1, Rec). -normalize_cs(Cstructs, _Node) -> - Cstructs. +normalize_cs(Cstructs, Node) -> + %% backward-compatibility hack; normalize before returning + case need_old_cstructs([Node]) of + false -> + Cstructs; + Version -> + %% some other format + [convert_cs(Version, Cs) || Cs <- Cstructs] + end. + +convert_cs(Version, Cs) -> + Fields = [Value || {_, Value} <- cs2list(Version, Cs)], + list_to_tuple([cstruct|Fields]). + +list2cs(List) -> + list2cs(List, get_ext_types()). -list2cs(List) when is_list(List) -> +list2cs(List, ExtTypes) when is_list(List) -> Name = pick(unknown, name, List, must), Type = pick(Name, type, List, set), Rc0 = pick(Name, ram_copies, List, []), Dc = pick(Name, disc_copies, List, []), Doc = pick(Name, disc_only_copies, List, []), - Rc = case {Rc0, Dc, Doc} of - {[], [], []} -> [node()]; + + Ext = pick_external_copies(List, ExtTypes), + Rc = case {Rc0, Dc, Doc, Ext} of + {[], [], [], []} -> [node()]; _ -> Rc0 end, LC = pick(Name, local_content, List, false), @@ -695,8 +839,6 @@ list2cs(List) when is_list(List) -> Ix = pick(Name, index, List, []), verify({alt, [nil, list]}, mnesia_lib:etype(Ix), {bad_type, Name, {index, [Ix]}}), - Ix2 = [attr_to_pos(I, Attrs) || I <- Ix], - Frag = pick(Name, frag_properties, List, []), verify({alt, [nil, list]}, mnesia_lib:etype(Frag), {badarg, Name, {frag_properties, Frag}}), @@ -723,24 +865,48 @@ list2cs(List) when is_list(List) -> DetsOpts = proplists:get_value(dets, BEProps, []), is_list(DetsOpts) orelse mnesia:abort({badarg, Name, {dets, DetsOpts}}), [CheckProp(Prop, BadDetsOpts) || Prop <- DetsOpts], - #cstruct{name = Name, - ram_copies = Rc, - disc_copies = Dc, - disc_only_copies = Doc, - type = Type, - index = Ix2, - snmp = Snmp, - load_order = LoadOrder, - access_mode = AccessMode, - majority = Majority, - local_content = LC, - record_name = RecName, - attributes = Attrs, - user_properties = lists:sort(UserProps), - frag_properties = lists:sort(Frag), - storage_properties = lists:sort(BEProps), - cookie = Cookie, - version = Version}. + + case lists:keymember(mnesia, 1, application:which_applications()) of + true -> + Keys = check_keys(Name, List), + check_duplicates(Name, Keys); + false -> + %% check_keys/2 cannot be executed when mnesia is not + %% running, due to it not being possible to read what ext + %% backends are loaded. + %%% this doesn't work - disabled for now: + %%%Keys = check_keys(Name, List, record_info(fields, cstruct)), + %%%check_duplicates(Name, Keys) + ignore + end, + + Cs0 = #cstruct{name = Name, + ram_copies = Rc, + disc_copies = Dc, + disc_only_copies = Doc, + external_copies = Ext, + type = Type, + index = Ix, + snmp = Snmp, + load_order = LoadOrder, + access_mode = AccessMode, + majority = Majority, + local_content = LC, + record_name = RecName, + attributes = Attrs, + user_properties = lists:sort(UserProps), + frag_properties = lists:sort(Frag), + storage_properties = lists:sort(BEProps), + cookie = Cookie, + version = Version}, + case Ix of + [] -> Cs0; + [_|_] -> + Ix2 = expand_index_attrs(Cs0), + Cs0#cstruct{index = Ix2} + end; +list2cs(Other, _ExtTypes) -> + mnesia:abort({badarg, Other}). pick(Tab, Key, List, Default) -> case lists:keysearch(Key, 1, List) of @@ -754,6 +920,79 @@ pick(Tab, Key, List, Default) -> mnesia:abort({bad_type, Tab, BadArg}) end. +pick_external_copies(_List, []) -> + []; +pick_external_copies(List, ExtTypes) -> + lists:foldr( + fun({K, Val}, Acc) -> + case lists:keyfind(K, 1, ExtTypes) of + false -> + Acc; + {_, Mod} -> + [{{K,Mod}, Val}|Acc] + end + end, [], List). + +expand_storage_type(S) when S==ram_copies; + S==disc_copies; + S==disc_only_copies -> + S; +expand_storage_type(S) -> + case lists:keyfind(S, 1, get_ext_types()) of + false -> + mnesia:abort({bad_type, {storage_type, S}}); + {Alias, Mod} -> + {ext, Alias, Mod} + end. + +get_ext_types() -> + get_schema_user_property(mnesia_backend_types). + +get_index_plugins() -> + get_schema_user_property(mnesia_index_plugins). + +get_schema_user_property(Key) -> + Tab = schema, + %% Must work reliably both within transactions and outside of transactions + Res = case get(mnesia_activity_state) of + undefined -> + dirty_read_table_property(Tab, Key); + _ -> + do_read_table_property(Tab, Key) + end, + case Res of + undefined -> + []; + {_, Types} -> + Types + end. + +get_ext_types_disc() -> + try get_ext_types_disc_() + catch + error:_ ->[] + end. + +get_ext_types_disc_() -> + case mnesia_schema:remote_read_schema() of + {ok, _, Prop} -> + K1 = user_properties, + case lists:keyfind(K1, 1, Prop) of + {K1, UserProp} -> + K2 = mnesia_backend_types, + case lists:keyfind(K2, 1, UserProp) of + {K2, Types} -> + Types; + _ -> + [] + end; + _ -> + [] + end; + _ -> + [] + end. + %% Convert attribute name to integer if neccessary attr_tab_to_pos(_Tab, Pos) when is_integer(Pos) -> Pos; @@ -761,6 +1000,7 @@ attr_tab_to_pos(Tab, Attr) -> attr_to_pos(Attr, val({Tab, attributes})). %% Convert attribute name to integer if neccessary +attr_to_pos({_} = P, _) -> P; attr_to_pos(Pos, _Attrs) when is_integer(Pos) -> Pos; attr_to_pos(Attr, Attrs) when is_atom(Attr) -> @@ -775,8 +1015,18 @@ attr_to_pos(Attr, [_ | Attrs], Pos) -> attr_to_pos(Attr, _, _) -> mnesia:abort({bad_type, Attr}). +check_keys(Tab, Attrs) -> + Types = [T || {T,_} <- get_ext_types()], + check_keys(Tab, Attrs, Types ++ record_info(fields, cstruct)). + check_keys(Tab, [{Key, _Val} | Tail], Items) -> - case lists:member(Key, Items) of + Key1 = if + is_tuple(Key) -> + element(1, Key); + true -> + Key + end, + case lists:member(Key1, Items) of true -> [Key | check_keys(Tab, Tail, Items)]; false -> mnesia:abort({badarg, Tab, Key}) end; @@ -800,7 +1050,92 @@ has_duplicates([]) -> false. %% This is the only place where we check the validity of data -verify_cstruct(Cs) when is_record(Cs, cstruct) -> + +verify_cstruct(#cstruct{} = Cs) -> + assert_correct_cstruct(Cs), + Cs1 = verify_external_copies( + Cs#cstruct{index = expand_index_attrs(Cs)}), + assert_correct_cstruct(Cs1), + Cs1. + +expand_index_attrs(#cstruct{index = Ix, attributes = Attrs, + name = Tab} = Cs) -> + Prefered = prefered_index_types(Cs), + expand_index_attrs(Ix, Tab, Attrs, Prefered). + +expand_index_attrs(Ix, Tab, Attrs, Prefered) -> + lists:map(fun(P) when is_integer(P); is_atom(P) -> + {attr_to_pos(P, Attrs), Prefered}; + ({A} = P) when is_atom(A) -> + {P, Prefered}; + ({P, Type}) -> + {attr_to_pos(P, Attrs), Type}; + (_Other) -> + mnesia:abort({bad_type, Tab, {index, Ix}}) + end, Ix). + +prefered_index_types(#cstruct{external_copies = Ext}) -> + ExtTypes = [mnesia_lib:semantics(S, index_types) || + {S,Ns} <- Ext, Ns =/= []], + case intersect_types(ExtTypes) of + [] -> ordered; + [Pref|_] -> Pref + end. + +intersect_types([]) -> + []; +intersect_types([S1, S2|Rest]) -> + intersect_types([S1 -- (S1 -- S2)|Rest]); +intersect_types([S]) -> + S. + +verify_external_copies(#cstruct{external_copies = []} = Cs) -> + Cs; +verify_external_copies(#cstruct{name = Tab, external_copies = EC} = Cs) -> + Bad = {bad_type, Tab, {external_copies, EC}}, + AllECNodes = lists:concat([Ns || {_, Ns} <- EC, + is_list(Ns)]), + verify(true, length(lists:usort(AllECNodes)) == length(AllECNodes), Bad), + CsL = cs2list(Cs), + CsL1 = lists:foldl( + fun({{Alias, Mod}, Ns} = _X, CsLx) -> + BadTab = fun(Why) -> + {Why, Tab, {{ext, Alias, Mod},Ns}} + end, + verify(atom, mnesia_lib:etype(Mod), BadTab), + verify(true, fun() -> + lists:all(fun is_atom/1, Ns) + end, BadTab), + check_semantics(Mod, Alias, BadTab, Cs), + try Mod:check_definition(Alias, Tab, Ns, CsLx) of + ok -> + CsLx; + {ok, CsLx1} -> + CsLx1; + {error, Reason} -> + mnesia:abort(BadTab(Reason)) + catch + error:E -> + mnesia:abort(BadTab(E)) + end; + (_, CsLx) -> + CsLx + end, CsL, EC), + list2cs(CsL1). + +check_semantics(Mod, Alias, BadTab, #cstruct{type = Type}) -> + Ext = {ext, Alias, Mod}, + case lists:member(mnesia_lib:semantics(Ext, storage), [ram_copies, disc_copies, + disc_only_copies]) of + false -> mnesia:abort(BadTab(invalid_storage)); + true -> ok + end, + case lists:member(Type, mnesia_lib:semantics(Ext, types)) of + false -> mnesia:abort(BadTab(bad_type)); + true -> ok + end. + +assert_correct_cstruct(Cs) when is_record(Cs, cstruct) -> verify_nodes(Cs), Tab = Cs#cstruct.name, @@ -841,22 +1176,30 @@ verify_cstruct(Cs) when is_record(Cs, cstruct) -> Attrs), Index = Cs#cstruct.index, + verify({alt, [nil, list]}, mnesia_lib:etype(Index), {bad_type, Tab, {index, Index}}), + IxPlugins = get_index_plugins(), + AllowIndexOnKey = check_if_allow_index_on_key(), IxFun = - fun(Pos) -> - verify(true, fun() -> - if - is_integer(Pos), - Pos > 2, - Pos =< Arity -> - true; - true -> false - end - end, - {bad_type, Tab, {index, [Pos]}}) - end, + fun(Pos) -> + verify( + true, fun() -> + I = index_pos(Pos), + case Pos of + {_, T} -> + (T==bag orelse T==ordered) + andalso good_ix_pos( + I, AllowIndexOnKey, + Arity, IxPlugins); + _ -> + good_ix_pos(Pos, AllowIndexOnKey, + Arity, IxPlugins) + end + end, + {bad_type, Tab, {index, [Pos]}}) + end, lists:foreach(IxFun, Index), LC = Cs#cstruct.local_content, @@ -880,7 +1223,9 @@ verify_cstruct(Cs) when is_record(Cs, cstruct) -> {badarg, Tab, {snmp, Snmp}}), CheckProp = fun(Prop) when is_tuple(Prop), size(Prop) >= 1 -> ok; - (Prop) -> mnesia:abort({bad_type, Tab, {user_properties, [Prop]}}) + (Prop) -> + mnesia:abort({bad_type, Tab, + {user_properties, [Prop]}}) end, lists:foreach(CheckProp, Cs#cstruct.user_properties), @@ -900,17 +1245,45 @@ verify_cstruct(Cs) when is_record(Cs, cstruct) -> mnesia:abort({bad_type, Tab, {version, Version}}) end. +good_ix_pos({_} = P, _, _, Plugins) -> + lists:keymember(P, 1, Plugins); +good_ix_pos(I, true, Arity, _) when is_integer(I) -> + I >= 0 andalso I =< Arity; +good_ix_pos(I, false, Arity, _) when is_integer(I) -> + I > 2 andalso I =< Arity; +good_ix_pos(_, _, _, _) -> + false. + + +check_if_allow_index_on_key() -> + case mnesia_monitor:get_env(allow_index_on_key) of + true -> + true; + _ -> + false + end. + verify_nodes(Cs) -> Tab = Cs#cstruct.name, Ram = Cs#cstruct.ram_copies, Disc = Cs#cstruct.disc_copies, DiscOnly = Cs#cstruct.disc_only_copies, + Ext = lists:append([Ns || {_,Ns} <- Cs#cstruct.external_copies]), LoadOrder = Cs#cstruct.load_order, verify({alt, [nil, list]}, mnesia_lib:etype(Ram), {bad_type, Tab, {ram_copies, Ram}}), verify({alt, [nil, list]}, mnesia_lib:etype(Disc), {bad_type, Tab, {disc_copies, Disc}}), + lists:foreach( + fun({BE, Ns}) -> + verify({alt, [nil, list]}, mnesia_lib:etype(Ns), + {bad_type, Tab, {BE, Ns}}), + lists:foreach(fun(N) -> + verify(atom, mnesia_lib:etype(N), + {bad_type, Tab, {BE, Ns}}) + end, Ns) + end, Cs#cstruct.external_copies), case Tab of schema -> verify([], DiscOnly, {bad_type, Tab, {disc_only_copies, DiscOnly}}); @@ -922,12 +1295,15 @@ verify_nodes(Cs) -> verify(integer, mnesia_lib:etype(LoadOrder), {bad_type, Tab, {load_order, LoadOrder}}), - Nodes = Ram ++ Disc ++ DiscOnly, + Nodes = Ram ++ Disc ++ DiscOnly ++ Ext, verify(list, mnesia_lib:etype(Nodes), {combine_error, Tab, - [{ram_copies, []}, {disc_copies, []}, {disc_only_copies, []}]}), + [{ram_copies, []}, {disc_copies, []}, + {disc_only_copies, []}, {external_copies, []}]}), verify(false, has_duplicates(Nodes), {combine_error, Tab, Nodes}), - AtomCheck = fun(N) -> verify(atom, mnesia_lib:etype(N), {bad_type, Tab, N}) end, + AtomCheck = fun(N) -> + verify(atom, mnesia_lib:etype(N), {bad_type, Tab, N}) + end, lists:foreach(AtomCheck, Nodes). verify(Expected, Fun, Error) when is_function(Fun) -> @@ -1010,28 +1386,194 @@ check_active([{badrpc, Reason} | _Replies], Expl, Tab) -> check_active([], _Expl, _Tab) -> ok. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Function for definining an external backend type + +add_backend_type(Name, Module) -> + case schema_transaction(fun() -> do_add_backend_type(Name, Module) end) of + {atomic, NeedsInit} -> + case NeedsInit of + true -> + Module:init_backend(); + false -> + ignore + end, + Module:add_aliases([Name]), + {atomic, ok}; + Other -> + Other + end. + +do_add_backend_type(Name, Module) -> + verify_backend_type(Name, Module), + Types = case do_read_table_property(schema, mnesia_backend_types) of + undefined -> + []; + {_, Ts} -> + case lists:keymember(Name, 1, Ts) of + true -> + mnesia:abort({backend_type_already_exists, Name}); + false -> + Ts + end + end, + ModuleRegistered = lists:keymember(Module, 2, Types), + do_write_table_property(schema, {mnesia_backend_types, + [{Name, Module}|Types]}), + ModuleRegistered. + +delete_backend_type(Name) -> + schema_transaction(fun() -> do_delete_backend_type(Name) end). + +do_delete_backend_type(Name) -> + case do_read_table_property(schema, mnesia_backend_types) of + undefined -> + []; + {_, Ts} -> + case lists:keyfind(Name, 1, Ts) of + {_, Mod} -> + case using_backend_type(Name, Mod) of + [_|_] = Tabs -> + mnesia:abort({backend_in_use, {Name, Tabs}}); + [] -> + do_write_table_property( + schema, {mnesia_backend_types, + lists:keydelete(Name, 1, Ts)}) + end; + false -> + mnesia:abort({no_such_backend, Name}) + end + end. + +using_backend_type(Name, Mod) -> + Ext = ets:select(mnesia_gvar, + [{ {{'$1',external_copies},'$2'}, [], [{{'$1','$2'}}] }]), + Entry = {Name, Mod}, + [T || {T,C} <- Ext, + lists:keymember(Entry, 1, C)]. + +verify_backend_type(Name, Module) -> + case legal_backend_name(Name) of + false -> + mnesia:abort({bad_type, {backend_type,Name,Module}}); + true -> + ok + end, + ExpectedExports = mnesia_backend_type:behaviour_info(callbacks), + Exports = try Module:module_info(exports) + catch + error:_ -> + mnesia:abort({undef_backend, Module}) + end, + case ExpectedExports -- Exports of + [] -> + ok; + _Other -> + io:fwrite(user, "Missing backend_type exports: ~p~n", [_Other]), + mnesia:abort({bad_type, {backend_type,Name,Module}}) + end. + +legal_backend_name(Name) -> + is_atom(Name) andalso + (not lists:member(Name, record_info(fields, cstruct))). + +%% Used e.g. by mnesia:system_info(backend_types). +backend_types() -> + [ram_copies, disc_copies, disc_only_copies | + [T || {T,_} <- get_ext_types()]]. + +add_index_plugin(Name, Module, Function) -> + schema_transaction( + fun() -> do_add_index_plugin(Name, Module, Function) end). + +do_add_index_plugin(Name, Module, Function) -> + verify_index_plugin(Name, Module, Function), + Plugins = case do_read_table_property(schema, mnesia_index_plugins) of + undefined -> + []; + {_, Ps} -> + case lists:keymember(Name, 1, Ps) of + true -> + mnesia:abort({index_plugin_already_exists, Name}); + false -> + Ps + end + end, + do_write_table_property(schema, {mnesia_index_plugins, + [{Name, Module, Function}|Plugins]}). + +delete_index_plugin(P) -> + schema_transaction( + fun() -> do_delete_index_plugin(P) end). + +do_delete_index_plugin({A} = P) when is_atom(A) -> + Plugins = get_index_plugins(), + case lists:keyfind(P, 1, Plugins) of + false -> + mnesia:abort({no_exists, {index_plugin, P}}); + _Found -> + case ets:select(mnesia_gvar, + [{ {{'$1',{index,{P,'_'}}},'_'},[],['$1']}, + { {{'$1',{index,P}},'_'},[],['$1']}], 1) of + {[_], _} -> + mnesia:abort({plugin_in_use, P}); + '$end_of_table' -> + do_write_table_property( + schema, {mnesia_index_plugins, + lists:keydelete(P, 1, Plugins)}) + end + end. + +verify_index_plugin({A} = Name, Module, Function) + when is_atom(A), is_atom(Module), is_atom(Function) -> + case code:ensure_loaded(Module) of + {error, nofile} -> + mnesia:abort({bad_type, {index_plugin,Name,Module,Function}}); + {module,_} -> + %% Index plugins are called as Module:Function(Tab, Pos, Obj) + case erlang:function_exported(Module, Function, 3) of + true -> + ok; + false -> + mnesia:abort( + {bad_type, {index_plugin,Name,Module,Function}}) + end + end; +verify_index_plugin(Name, Module, Function) -> + mnesia:abort({bad_type, {index_plugin,Name,Module,Function}}). + + +%% Used e.g. by mnesia:system_info(backend_types). +index_plugins() -> + get_index_plugins(). + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Here's the real interface function to create a table -create_table(TabDef) -> - schema_transaction(fun() -> do_multi_create_table(TabDef) end). +create_table([_|_] = TabDef) -> + schema_transaction(fun() -> do_multi_create_table(TabDef) end); +create_table(Arg) -> {aborted, {badarg, Arg}}. %% And the corresponding do routines .... do_multi_create_table(TabDef) -> get_tid_ts_and_lock(schema, write), ensure_writable(schema), + do_create_table(TabDef), + ok. + +do_create_table(TabDef) when is_list(TabDef) -> Cs = api_list2cs(TabDef), case Cs#cstruct.frag_properties of [] -> - do_create_table(Cs); + do_create_table_1(Cs); _Props -> CsList = mnesia_frag:expand_cstruct(Cs), - lists:foreach(fun do_create_table/1, CsList) - end, - ok. + lists:foreach(fun do_create_table_1/1, CsList) + end. -do_create_table(Cs) -> +do_create_table_1(Cs) -> {_Mod, _Tid, Ts} = get_tid_ts_and_lock(schema, none), Store = Ts#tidstore.store, do_insert_schema_ops(Store, make_create_table(Cs)). @@ -1041,14 +1583,9 @@ make_create_table(Cs) -> verify(false, check_if_exists(Tab), {already_exists, Tab}), unsafe_make_create_table(Cs). -% unsafe_do_create_table(Cs) -> -% {_Mod, Tid, Ts} = get_tid_ts_and_lock(schema, none), -% Store = Ts#tidstore.store, -% do_insert_schema_ops(Store, unsafe_make_create_table(Cs)). - -unsafe_make_create_table(Cs) -> +unsafe_make_create_table(Cs0) -> {_Mod, Tid, Ts} = get_tid_ts_and_lock(schema, none), - verify_cstruct(Cs), + Cs = verify_cstruct(Cs0), Tab = Cs#cstruct.name, %% Check that we have all disc replica nodes running @@ -1196,8 +1733,7 @@ make_add_table_copy(Tab, Node, Storage) -> Cs = incr_version(val({Tab, cstruct})), Ns = mnesia_lib:cs_to_nodes(Cs), verify(false, lists:member(Node, Ns), {already_exists, Tab, Node}), - Cs2 = new_cs(Cs, Node, Storage, add), - verify_cstruct(Cs2), + Cs2 = verify_cstruct(new_cs(Cs, Node, Storage, add)), %% Check storage and if node is running IsRunning = lists:member(Node, val({current, db_nodes})), @@ -1245,14 +1781,14 @@ make_del_table_copy(Tab, Node) -> _ when Tab == schema -> %% ensure_active(Cs2), ensure_not_active(Tab, Node), - verify_cstruct(Cs2), + Cs3 = verify_cstruct(Cs2), Ops = remove_node_from_tabs(val({schema, tables}), Node), - [{op, del_table_copy, ram_copies, Node, vsn_cs2list(Cs2)} | Ops]; + [{op, del_table_copy, ram_copies, Node, vsn_cs2list(Cs3)} | Ops]; _ -> ensure_active(Cs), - verify_cstruct(Cs2), - get_tid_ts_and_lock(Tab, write), - [{op, del_table_copy, Storage, Node, vsn_cs2list(Cs2)}] + Cs3 = verify_cstruct(Cs2), + get_tid_ts_and_lock(Tab, write), + [{op, del_table_copy, Storage, Node, vsn_cs2list(Cs3)}] end. remove_node_from_tabs([], _Node) -> @@ -1278,9 +1814,9 @@ remove_node_from_tabs([Tab|Rest], Node) -> [{op, delete_table, vsn_cs2list(Cs)} | remove_node_from_tabs(Rest, Node)]; _Ns -> - verify_cstruct(Cs2), + Cs3 = verify_cstruct(Cs2), get_tid_ts_and_lock(Tab, write), - [{op, del_table_copy, ram_copies, Node, vsn_cs2list(Cs2)}| + [{op, del_table_copy, ram_copies, Node, vsn_cs2list(Cs3)}| remove_node_from_tabs(Rest, Node)] end end. @@ -1298,8 +1834,34 @@ new_cs(Cs, Node, disc_copies, del) -> new_cs(Cs, Node, disc_only_copies, del) -> Cs#cstruct{disc_only_copies = lists:delete(Node , Cs#cstruct.disc_only_copies)}; -new_cs(Cs, _Node, Storage, _Op) -> - mnesia:abort({badarg, Cs#cstruct.name, Storage}). +new_cs(#cstruct{external_copies = ExtCps} = Cs, Node, Storage0, Op) -> + Storage = case Storage0 of + {ext, Alias, _} -> Alias; + Alias -> Alias + end, + ExtTypes = get_ext_types(), + case lists:keyfind(Storage, 1, ExtTypes) of + false -> + mnesia:abort({badarg, Cs#cstruct.name, Storage}); + {_, Mod} -> + Key = {Storage, Mod}, + case {lists:keymember(Key, 1, ExtCps), Op} of + {false, del} -> + Cs; + {false, add} -> + Cs#cstruct{external_copies = [{Key, [Node]}|ExtCps]}; + {true, _} -> + F = fun({K, Ns}) when K == Key -> + case Op of + del -> {K, lists:delete(Node, Ns)}; + add -> {K, opt_add(Node, Ns)} + end; + (X) -> + X + end, + Cs#cstruct{external_copies = lists:map(F, ExtCps)} + end + end. opt_add(N, L) -> [N | lists:delete(N, L)]. @@ -1335,8 +1897,7 @@ make_move_table(Tab, FromNode, ToNode) -> verify(true, lists:member(ToNode, Running), {not_active, schema, ToNode}), Cs2 = new_cs(Cs, ToNode, Storage, add), - Cs3 = new_cs(Cs2, FromNode, Storage, del), - verify_cstruct(Cs3), + Cs3 = verify_cstruct(new_cs(Cs2, FromNode, Storage, del)), [{op, add_table_copy, Storage, ToNode, vsn_cs2list(Cs2)}, {op, sync_trans}, {op, del_table_copy, Storage, FromNode, vsn_cs2list(Cs3)}]. @@ -1363,9 +1924,11 @@ make_change_table_copy_type(Tab, Node, ToS) -> Cs = incr_version(val({Tab, cstruct})), FromS = mnesia_lib:storage_type_at_node(Node, Tab), - case compare_storage_type(false, FromS, ToS) of + ToSExp = expand_storage_type(ToS), + + case compare_storage_type(false, FromS, ToSExp) of {same, _} -> - mnesia:abort({already_exists, Tab, Node, ToS}); + mnesia:abort({already_exists, Tab, Node, ToSExp}); {diff, _} -> ignore; incompatible -> @@ -1373,10 +1936,8 @@ make_change_table_copy_type(Tab, Node, ToS) -> end, Cs2 = new_cs(Cs, Node, FromS, del), - Cs3 = new_cs(Cs2, Node, ToS, add), - verify_cstruct(Cs3), - - [{op, change_table_copy_type, Node, FromS, ToS, vsn_cs2list(Cs3)}]. + Cs3 = verify_cstruct(new_cs(Cs2, Node, ToS, add)), + [{op, change_table_copy_type, Node, FromS, ToSExp, vsn_cs2list(Cs3)}]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% change index functions .... @@ -1398,11 +1959,12 @@ make_add_table_index(Tab, Pos) -> Cs = incr_version(val({Tab, cstruct})), ensure_active(Cs), Ix = Cs#cstruct.index, - verify(false, lists:member(Pos, Ix), {already_exists, Tab, Pos}), + verify(false, lists:keymember(index_pos(Pos), 1, Ix), + {already_exists, Tab, Pos}), Ix2 = lists:sort([Pos | Ix]), - Cs2 = Cs#cstruct{index = Ix2}, - verify_cstruct(Cs2), - [{op, add_index, Pos, vsn_cs2list(Cs2)}]. + Cs2 = verify_cstruct(Cs#cstruct{index = Ix2}), + NewPosInfo = lists:keyfind(Pos, 1, Cs2#cstruct.index), + [{op, add_index, NewPosInfo, vsn_cs2list(Cs2)}]. del_table_index(Tab, Pos) -> schema_transaction(fun() -> do_del_table_index(Tab, Pos) end). @@ -1420,9 +1982,8 @@ make_del_table_index(Tab, Pos) -> Cs = incr_version(val({Tab, cstruct})), ensure_active(Cs), Ix = Cs#cstruct.index, - verify(true, lists:member(Pos, Ix), {no_exists, Tab, Pos}), - Cs2 = Cs#cstruct{index = lists:delete(Pos, Ix)}, - verify_cstruct(Cs2), + verify(true, lists:keymember(Pos, 1, Ix), {no_exists, Tab, Pos}), + Cs2 = verify_cstruct(Cs#cstruct{index = lists:keydelete(Pos, 1, Ix)}), [{op, del_index, Pos, vsn_cs2list(Cs2)}]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1444,8 +2005,7 @@ make_add_snmp(Tab, Ustruct) -> verify([], Cs#cstruct.snmp, {already_exists, Tab, snmp}), Error = {badarg, Tab, snmp, Ustruct}, verify(true, mnesia_snmp_hook:check_ustruct(Ustruct), Error), - Cs2 = Cs#cstruct{snmp = Ustruct}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{snmp = Ustruct}), [{op, add_snmp, Ustruct, vsn_cs2list(Cs2)}]. del_snmp(Tab) -> @@ -1462,8 +2022,7 @@ make_del_snmp(Tab) -> ensure_writable(schema), Cs = incr_version(val({Tab, cstruct})), ensure_active(Cs), - Cs2 = Cs#cstruct{snmp = []}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{snmp = []}), [{op, del_snmp, vsn_cs2list(Cs2)}]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1492,19 +2051,21 @@ make_transform(Tab, Fun, NewAttrs, NewRecName) -> Cs = incr_version(val({Tab, cstruct})), ensure_active(Cs), ensure_writable(Tab), - case mnesia_lib:val({Tab, index}) of + case Cs#cstruct.index of [] -> - Cs2 = Cs#cstruct{attributes = NewAttrs, record_name = NewRecName}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct( + Cs#cstruct{attributes = NewAttrs, + record_name = NewRecName}), [{op, transform, Fun, vsn_cs2list(Cs2)}]; PosList -> - DelIdx = fun(Pos, Ncs) -> + DelIdx = fun({Pos,_}, Ncs) -> Ix = Ncs#cstruct.index, - Ncs1 = Ncs#cstruct{index = lists:delete(Pos, Ix)}, + Ix2 = lists:keydelete(Pos, 1, Ix), + Ncs1 = Ncs#cstruct{index = Ix2}, Op = {op, del_index, Pos, vsn_cs2list(Ncs1)}, {Op, Ncs1} end, - AddIdx = fun(Pos, Ncs) -> + AddIdx = fun({_,_} = Pos, Ncs) -> Ix = Ncs#cstruct.index, Ix2 = lists:sort([Pos | Ix]), Ncs1 = Ncs#cstruct{index = Ix2}, @@ -1514,10 +2075,16 @@ make_transform(Tab, Fun, NewAttrs, NewRecName) -> {DelOps, Cs1} = lists:mapfoldl(DelIdx, Cs, PosList), Cs2 = Cs1#cstruct{attributes = NewAttrs, record_name = NewRecName}, {AddOps, Cs3} = lists:mapfoldl(AddIdx, Cs2, PosList), - verify_cstruct(Cs3), - lists:flatten([DelOps, {op, transform, Fun, vsn_cs2list(Cs2)}, AddOps]) + _ = verify_cstruct(Cs3), % just a sanity check + lists:flatten([DelOps, {op, transform, Fun, vsn_cs2list(Cs2)}, + AddOps]) end. +index_pos({Pos,_}) -> Pos; +index_pos(Pos) when is_integer(Pos) -> Pos; +index_pos({P} = Pos) when is_atom(P) -> Pos. + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% @@ -1537,8 +2104,7 @@ make_change_table_access_mode(Tab, Mode) -> ensure_active(Cs), OldMode = Cs#cstruct.access_mode, verify(false, OldMode == Mode, {already_exists, Tab, Mode}), - Cs2 = Cs#cstruct{access_mode = Mode}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{access_mode = Mode}), [{op, change_table_access_mode, vsn_cs2list(Cs2), OldMode, Mode}]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1558,8 +2124,7 @@ make_change_table_load_order(Tab, LoadOrder) -> Cs = incr_version(val({Tab, cstruct})), ensure_active(Cs), OldLoadOrder = Cs#cstruct.load_order, - Cs2 = Cs#cstruct{load_order = LoadOrder}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{load_order = LoadOrder}), [{op, change_table_load_order, vsn_cs2list(Cs2), OldLoadOrder, LoadOrder}]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1605,6 +2170,7 @@ write_table_property(Tab, Prop) when is_tuple(Prop), size(Prop) >= 1 -> schema_transaction(fun() -> do_write_table_property(Tab, Prop) end); write_table_property(Tab, Prop) -> {aborted, {bad_type, Tab, Prop}}. + do_write_table_property(Tab, Prop) -> TidTs = get_tid_ts_and_lock(schema, write), {_, _, Ts} = TidTs, @@ -1636,8 +2202,7 @@ make_write_table_properties(Tab, [Prop | Props], Cs) -> PropKey = element(1, Prop), DelProps = lists:keydelete(PropKey, 1, OldProps), MergedProps = lists:merge(DelProps, [Prop]), - Cs2 = Cs#cstruct{user_properties = MergedProps}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{user_properties = MergedProps}), [{op, write_property, vsn_cs2list(Cs2), Prop} | make_write_table_properties(Tab, Props, Cs2)]; make_write_table_properties(_Tab, [], _Cs) -> @@ -1682,22 +2247,38 @@ do_read_table_property(Tab, Key) -> {_, _, Ts} = TidTs, Store = Ts#tidstore.store, Props = ets:foldl( - fun({op, create_table, [{name, T}|Opts]}, _Acc) - when T==Tab -> + fun({op, announce_im_running,_,Opts,_,_}, _Acc) when Tab==schema -> + find_props(Opts); + ({op, create_table, [{name, T}|Opts]}, _Acc) + when T==Tab -> find_props(Opts); ({op, Op, [{name,T}|Opts], _Prop}, _Acc) - when T==Tab, Op==write_property; Op==delete_property -> + when T==Tab, Op==write_property; + T==Tab, Op==delete_property -> find_props(Opts); ({op, delete_table, [{name,T}|_]}, _Acc) when T==Tab -> []; (_Other, Acc) -> Acc - end, [], Store), - case lists:keysearch(Key, 1, Props) of - {value, Property} -> - Property; - false -> + end, undefined, Store), + case Props of + undefined -> + get_tid_ts_and_lock(Tab, read), + dirty_read_table_property(Tab, Key); + _ when is_list(Props) -> + case lists:keyfind(Key, 1, Props) of + false -> + undefined; + Other -> + Other + end + end. + +dirty_read_table_property(Tab, Key) -> + try ets:lookup_element(mnesia_gvar, {Tab,user_property,Key}, 2) + catch + error:_ -> undefined end. @@ -1757,8 +2338,7 @@ make_delete_table_properties(Tab, PropKeys) -> make_delete_table_properties(Tab, [PropKey | PropKeys], Cs) -> OldProps = Cs#cstruct.user_properties, Props = lists:keydelete(PropKey, 1, OldProps), - Cs2 = Cs#cstruct{user_properties = Props}, - verify_cstruct(Cs2), + Cs2 = verify_cstruct(Cs#cstruct{user_properties = Props}), [{op, delete_property, vsn_cs2list(Cs2), PropKey} | make_delete_table_properties(Tab, PropKeys, Cs2)]; make_delete_table_properties(_Tab, [], _Cs) -> @@ -1899,6 +2479,11 @@ prepare_op(Tid, {op, create_table, TabDef}, _WaitFor) -> create_disc_only_table(Tab,Cs), insert_cstruct(Tid, Cs, false), {true, optional}; + {ext, Alias, Mod} -> + mnesia_lib:set({Tab, create_table},true), + create_external_table(Alias, Tab, Mod, Cs), + insert_cstruct(Tid, Cs, false), + {true, optional}; unknown -> %% No replica on this node mnesia_lib:set({Tab, create_table},true), insert_cstruct(Tid, Cs, false), @@ -1967,6 +2552,12 @@ prepare_op(_Tid, {op, change_table_copy_type, N, FromS, ToS, TabDef}, _WaitFor) NotActive = mnesia_lib:not_active_here(Tab), + if Tab =/= schema -> + check_if_disc_required(FromS, ToS); + true -> + ok + end, + if NotActive == true -> mnesia:abort({not_active, Tab, node()}); @@ -2006,6 +2597,15 @@ prepare_op(_Tid, {op, change_table_copy_type, N, FromS, ToS, TabDef}, _WaitFor) mnesia:abort({combine_error, Tab, ToS}) end; + element(1,FromS) == ext; element(1,ToS) == ext -> + if ToS == ram_copies -> + create_ram_table(Tab, Cs); + true -> + ok + end, + mnesia_dumper:dump_to_logfile(FromS, Tab), + mnesia_checkpoint:tm_change_table_copy_type(Tab, FromS, ToS); + FromS == ram_copies -> case mnesia_monitor:use_dir() of true -> @@ -2021,7 +2621,9 @@ prepare_op(_Tid, {op, change_table_copy_type, N, FromS, ToS, TabDef}, _WaitFor) disc_only_copies -> mnesia_dumper:raw_named_dump_table(Tab, dmp) end, - mnesia_checkpoint:tm_change_table_copy_type(Tab, FromS, ToS) + mnesia_checkpoint:tm_change_table_copy_type(Tab, + FromS, + ToS) end; false -> mnesia:abort({has_no_disc, node()}) @@ -2029,6 +2631,7 @@ prepare_op(_Tid, {op, change_table_copy_type, N, FromS, ToS, TabDef}, _WaitFor) FromS == disc_copies, ToS == disc_only_copies -> mnesia_dumper:raw_named_dump_table(Tab, dmp); + FromS == disc_only_copies -> Type = Cs#cstruct.type, create_ram_table(Tab, Cs), @@ -2040,6 +2643,7 @@ prepare_op(_Tid, {op, change_table_copy_type, N, FromS, ToS, TabDef}, _WaitFor) Err = "Failed to copy disc data to ram", mnesia:abort({system_limit, Tab, {Err,Reason}}) end; + true -> ignore end, @@ -2103,6 +2707,8 @@ prepare_op(_Tid, {op, transform, Fun, TabDef}, _WaitFor) -> {true, Objs, mandatory} catch _:Reason -> mnesia_lib:db_fixtable(Storage, Tab, false), + mnesia_lib:important("Transform function failed: '~p' in '~p'", + [Reason, erlang:get_stacktrace()]), exit({"Bad transform function", Tab, Fun, node(), Reason}) end end; @@ -2119,6 +2725,22 @@ prepare_op(_Tid, {op, merge_schema, TabDef}, _WaitFor) -> prepare_op(_Tid, _Op, _WaitFor) -> {true, optional}. +check_if_disc_required(FromS, ToS) -> + FromSem = mnesia_lib:semantics(FromS, storage), + ToSem = mnesia_lib:semantics(ToS, storage), + case {FromSem, ToSem} of + {ram_copies, _} when ToSem == disc_copies; + ToSem == disc_only_copies -> + case mnesia_monitor:use_dir() of + true -> + ok; + false -> + mnesia:abort({has_no_disc, node()}) + end; + _ -> + ok + end. + create_ram_table(Tab, #cstruct{type=Type, storage_properties=Props}) -> EtsOpts = proplists:get_value(ets, Props, []), Args = [{keypos, 2}, public, named_table, Type | EtsOpts], @@ -2160,6 +2782,14 @@ create_disc_only_table(Tab, #cstruct{type=Type, storage_properties=Props}) -> mnesia:abort({system_limit, Tab, {Err,Reason}}) end. +create_external_table(Alias, Tab, Mod, Cs) -> + case mnesia_monitor:unsafe_create_external(Tab, Alias, Mod, Cs) of + ok -> + ok; + {error,Reason} -> + Err = "Failed to create external table", + mnesia:abort({system_limit, Tab, {Err,Reason}}) + end. receive_sync([], Pids) -> Pids; @@ -2294,7 +2924,10 @@ undo_prepare_op(Tid, {op, create_table, TabDef}) -> mnesia_monitor:unsafe_close_dets(Tab), Dat = mnesia_lib:tab2dat(Tab), %% disc_delete_table(Tab, Storage), - file:delete(Dat) + file:delete(Dat); + {ext, Alias, Mod} -> + Mod:close_table(Alias, Tab), + Mod:delete_table(Alias, Tab) end; undo_prepare_op(Tid, {op, add_table_copy, Storage, Node, TabDef}) -> @@ -2394,6 +3027,8 @@ ram_delete_table(Tab, Storage) -> case Storage of unknown -> ignore; + {ext, _, _} -> + ignore; disc_only_copies -> ignore; _Else -> @@ -2455,13 +3090,27 @@ purge_known_files([File | Tail], KeepFiles, Dir, Suffixes) -> ignore; true -> AbsFile = filename:join([Dir, File]), - file:delete(AbsFile) - end + delete_recursive(AbsFile) + end end, purge_known_files(Tail, KeepFiles, Dir, Suffixes); purge_known_files([], _KeepFiles, _Dir, _Suffixes) -> ok. +%% Removes a directory or file recursively +delete_recursive(Path) -> + case filelib:is_dir(Path) of + true -> + {ok, Names} = file:list_dir(Path), + lists:foreach(fun(Name) -> + delete_recursive(filename:join(Path, Name)) + end, + Names), + file:del_dir(Path); + false -> + file:delete(Path) + end. + has_known_suffix(_File, _Suffixes, true) -> true; has_known_suffix(File, [Suffix | Tail], false) -> @@ -2469,11 +3118,33 @@ has_known_suffix(File, [Suffix | Tail], false) -> has_known_suffix(_File, [], Bool) -> Bool. -known_suffixes() -> real_suffixes() ++ tmp_suffixes(). +known_suffixes() -> known_suffixes(get_ext_types_disc()). + +known_suffixes(Ext) -> real_suffixes(Ext) ++ tmp_suffixes(Ext). -real_suffixes() -> [".DAT", ".LOG", ".BUP", ".DCL", ".DCD"]. +real_suffixes(Ext) -> [".DAT", ".LOG", ".BUP", ".DCL", ".DCD"] ++ ext_real_suffixes(Ext). -tmp_suffixes() -> [".TMP", ".BUPTMP", ".RET", ".DMP"]. +tmp_suffixes() -> tmp_suffixes(get_ext_types_disc()). + +tmp_suffixes(Ext) -> [".TMP", ".BUPTMP", ".RET", ".DMP", "."] ++ ext_tmp_suffixes(Ext). + +ext_real_suffixes(Ext) -> + try lists:foldl(fun(Mod, Acc) -> Acc++Mod:real_suffixes() end, [], + [M || {_,M} <- Ext]) + catch + error:E -> + verbose("Cant find real ext suffixes (~p)~n", [E]), + [] + end. + +ext_tmp_suffixes(Ext) -> + try lists:foldl(fun(Mod, Acc) -> Acc++Mod:tmp_suffixes() end, [], + [M || {_,M} <- Ext]) + catch + error:E -> + verbose("Cant find tmp ext suffixes (~p)~n", [E]), + [] + end. info() -> Tabs = lists:sort(val({schema, tables})), @@ -2591,12 +3262,12 @@ do_restore(R, BupSchema) -> arrange_restore(R, Fun, Recs) -> R2 = R#r{insert_op = Fun, recs = Recs}, - case mnesia_bup:iterate(R#r.module, fun restore_items/4, R#r.opaque, R2) of + case mnesia_bup:iterate(R#r.module, fun restore_items/5, R#r.opaque, R2) of {ok, R3} -> R3#r.recs; {error, Reason} -> mnesia:abort(Reason) end. -restore_items([Rec | Recs], Header, Schema, R) -> +restore_items([Rec | Recs], Header, Schema, Ext, R) -> Tab = element(1, Rec), case lists:keysearch(Tab, 1, R#r.tables) of {value, {Tab, Where0, Snmp, RecName}} -> @@ -2609,13 +3280,13 @@ restore_items([Rec | Recs], Header, Schema, R) -> {Rest, NRecs} = restore_tab_items([Rec | Recs], Tab, RecName, Where, Snmp, R#r.recs, R#r.insert_op), - restore_items(Rest, Header, Schema, R#r{recs = NRecs}); + restore_items(Rest, Header, Schema, Ext, R#r{recs = NRecs}); false -> Rest = skip_tab_items(Recs, Tab), - restore_items(Rest, Header, Schema, R) + restore_items(Rest, Header, Schema, Ext, R) end; -restore_items([], _Header, _Schema, R) -> +restore_items([], _Header, _Schema, _Ext, R) -> R. restore_func(Tab, R) -> @@ -2629,8 +3300,14 @@ restore_func(Tab, R) -> where_to_commit(Tab, CsList) -> Ram = [{N, ram_copies} || N <- pick(Tab, ram_copies, CsList, [])], Disc = [{N, disc_copies} || N <- pick(Tab, disc_copies, CsList, [])], - DiscO = [{N, disc_only_copies} || N <- pick(Tab, disc_only_copies, CsList, [])], - Ram ++ Disc ++ DiscO. + DiscO = [{N, disc_only_copies} || + N <- pick(Tab, disc_only_copies, CsList, [])], + ExtNodes = [{Alias, Mod, pick(Tab, Alias, CsList, [])} || + {Alias, Mod} <- get_ext_types()], + Ext = lists:foldl(fun({Alias, Mod, Ns}, Acc) -> + [{N, {ext, Alias, Mod}} || N <- Ns] ++ Acc + end, [], ExtNodes), + Ram ++ Disc ++ DiscO ++ Ext. %% Changes of the Meta info of schema itself is not allowed restore_schema([{schema, schema, _List} | Schema], R) -> @@ -2709,7 +3386,13 @@ make_dump_tables([schema | _Tabs]) -> make_dump_tables([Tab | Tabs]) -> get_tid_ts_and_lock(Tab, read), TabDef = get_create_list(Tab), - DiscResident = val({Tab, disc_copies}) ++ val({Tab, disc_only_copies}), + DiscResident = + val({Tab, disc_copies}) ++ + val({Tab, disc_only_copies}) ++ + lists:concat([Ns || {{A,M},Ns} <- val({Tab, external_copies}), + lists:member( + mnesia_lib:semantics({ext,A,M},storage), + [disc_copies, disc_only_copies])]), verify([], DiscResident, {"Only allowed on ram_copies", Tab, DiscResident}), [{op, dump_table, unknown, TabDef} | make_dump_tables(Tabs)]; @@ -2721,7 +3404,9 @@ merge_schema() -> schema_transaction(fun() -> do_merge_schema([]) end). merge_schema(UserFun) -> - schema_transaction(fun() -> UserFun(fun(Arg) -> do_merge_schema(Arg) end) end). + schema_transaction(fun() -> + UserFun(fun(Arg) -> do_merge_schema(Arg) end) + end). do_merge_schema(LockTabs0) -> {_Mod, Tid, Ts} = get_tid_ts_and_lock(schema, write), @@ -2795,11 +3480,23 @@ do_merge_schema(LockTabs0) -> end. fetch_cstructs(Node) -> - rpc:call(Node, mnesia_controller, get_remote_cstructs, []). + Convert = mnesia_monitor:needs_protocol_conversion(Node), + case rpc:call(Node, mnesia_controller, get_remote_cstructs, []) of + {cstructs, Cs0, RemoteRunning1} when Convert -> + {cstructs, [list2cs(cs2list(Cs)) || Cs <- Cs0], RemoteRunning1}; + Result -> + Result + end. -need_old_cstructs() -> false. +need_old_cstructs() -> + need_old_cstructs(val({schema, where_to_write})). -need_old_cstructs(_Nodes) -> false. +need_old_cstructs(Nodes) -> + Filter = fun(Node) -> mnesia_monitor:needs_protocol_conversion(Node) end, + case lists:filter(Filter, Nodes) of + [] -> false; + Ns -> lists:min([element(1, ?catch_val({protocol, Node})) || Node <- Ns]) + end. tab_to_nodes(Tab) when is_atom(Tab) -> Cs = val({Tab, cstruct}), @@ -2952,8 +3649,8 @@ do_make_merge_schema(Node, NeedsConv, RemoteCs = #cstruct{}) -> %% invariants must be enforced in order to allow merge of cstructs. %% %% Returns a new cstruct or issues a fatal error -merge_cstructs(Cs, RemoteCs, Force) -> - verify_cstruct(Cs), +merge_cstructs(Cs0, RemoteCs, Force) -> + Cs = verify_cstruct(Cs0), try do_merge_cstructs(Cs, RemoteCs, Force) of MergedCs when is_record(MergedCs, cstruct) -> MergedCs @@ -2963,15 +3660,15 @@ merge_cstructs(Cs, RemoteCs, Force) -> error:Reason -> exit(Reason) end. -do_merge_cstructs(Cs, RemoteCs, Force) -> - verify_cstruct(RemoteCs), +do_merge_cstructs(Cs, RemoteCs0, Force) -> + RemoteCs = verify_cstruct(RemoteCs0), Ns = mnesia_lib:uniq(mnesia_lib:cs_to_nodes(Cs) ++ mnesia_lib:cs_to_nodes(RemoteCs)), {AnythingNew, MergedCs} = merge_storage_type(Ns, false, Cs, RemoteCs, Force), - MergedCs2 = merge_versions(AnythingNew, MergedCs, RemoteCs, Force), - verify_cstruct(MergedCs2), - MergedCs2. + verify_cstruct( + merge_versions(AnythingNew, MergedCs, RemoteCs, Force)). + merge_storage_type([N | Ns], AnythingNew, Cs, RemoteCs, Force) -> Local = mnesia_lib:cs_to_storage_type(N, Cs), @@ -3115,4 +3812,3 @@ unannounce_im_running([N | Ns]) -> unannounce_im_running(Ns); unannounce_im_running([]) -> ok. - diff --git a/lib/mnesia/src/mnesia_sup.erl b/lib/mnesia/src/mnesia_sup.erl index 5037a17ada..4aece81308 100644 --- a/lib/mnesia/src/mnesia_sup.erl +++ b/lib/mnesia/src/mnesia_sup.erl @@ -60,9 +60,10 @@ init() -> Flags = {one_for_all, 0, 3600}, % Should be rest_for_one policy Event = event_procs(), + Ext = ext_procs(), Kernel = kernel_procs(), - {ok, {Flags, Event ++ Kernel}}. + {ok, {Flags, Event ++ Ext ++ Kernel}}. event_procs() -> KillAfter = timer:seconds(30), @@ -75,6 +76,11 @@ kernel_procs() -> KA = infinity, [{K, {K, start, []}, permanent, KA, supervisor, [K, supervisor]}]. +ext_procs() -> + K = mnesia_ext_sup, + KA = infinity, + [{K, {K, start, []}, permanent, KA, supervisor, [K, supervisor]}]. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% event handler diff --git a/lib/mnesia/src/mnesia_tm.erl b/lib/mnesia/src/mnesia_tm.erl index 6888f61dbd..b116b48312 100644 --- a/lib/mnesia/src/mnesia_tm.erl +++ b/lib/mnesia/src/mnesia_tm.erl @@ -42,7 +42,8 @@ put_activity_id/2, block_tab/1, unblock_tab/1, - fixtable/3 + fixtable/3, + new_cr_format/1 ]). %% sys callback functions @@ -206,10 +207,10 @@ doit_loop(#state{coordinators=Coordinators,participants=Participants,supervisor= {_From, {async_dirty, Tid, Commit, Tab}} -> case lists:member(Tab, State#state.blocked_tabs) of false -> - do_async_dirty(Tid, Commit, Tab), + do_async_dirty(Tid, new_cr_format(Commit), Tab), doit_loop(State); true -> - Item = {async_dirty, Tid, Commit, Tab}, + Item = {async_dirty, Tid, new_cr_format(Commit), Tab}, State2 = State#state{dirty_queue = [Item | State#state.dirty_queue]}, doit_loop(State2) end; @@ -217,10 +218,10 @@ doit_loop(#state{coordinators=Coordinators,participants=Participants,supervisor= {From, {sync_dirty, Tid, Commit, Tab}} -> case lists:member(Tab, State#state.blocked_tabs) of false -> - do_sync_dirty(From, Tid, Commit, Tab), + do_sync_dirty(From, Tid, new_cr_format(Commit), Tab), doit_loop(State); true -> - Item = {sync_dirty, From, Tid, Commit, Tab}, + Item = {sync_dirty, From, Tid, new_cr_format(Commit), Tab}, State2 = State#state{dirty_queue = [Item | State#state.dirty_queue]}, doit_loop(State2) end; @@ -241,10 +242,11 @@ doit_loop(#state{coordinators=Coordinators,participants=Participants,supervisor= reply(From, {error, {system_limit, Msg, Reason}}, State) end; - {From, {ask_commit, Protocol, Tid, Commit, DiscNs, RamNs}} -> + {From, {ask_commit, Protocol, Tid, Commit0, DiscNs, RamNs}} -> ?eval_debug_fun({?MODULE, doit_ask_commit}, [{tid, Tid}, {prot, Protocol}]), mnesia_checkpoint:tm_enter_pending(Tid, DiscNs, RamNs), + Commit = new_cr_format(Commit0), Pid = case Protocol of asym_trans when node(Tid#tid.pid) /= node() -> @@ -1137,14 +1139,14 @@ arrange(Tid, Store, Type) -> reverse([]) -> []; reverse([H=#commit{ram_copies=Ram, disc_copies=DC, - disc_only_copies=DOC,snmp = Snmp} + disc_only_copies=DOC, ext=Ext} |R]) -> [ H#commit{ - ram_copies = lists:reverse(Ram), - disc_copies = lists:reverse(DC), - disc_only_copies = lists:reverse(DOC), - snmp = lists:reverse(Snmp) + ram_copies = lists:reverse(Ram), + disc_copies = lists:reverse(DC), + disc_only_copies = lists:reverse(DOC), + ext = [{Type, lists:reverse(E)} || {Type,E} <- Ext] } | reverse(R)]. @@ -1311,8 +1313,13 @@ pick_node({dirty,_}, Node, [], Done) -> pick_node(_Tid, Node, [], _Done) -> mnesia:abort({bad_commit, {missing_lock, Node}}). -prepare_node(Node, Storage, [Item | Items], Rec, Kind) when Kind == snmp -> - Rec2 = Rec#commit{snmp = [Item | Rec#commit.snmp]}, +prepare_node(Node, Storage, [Item | Items], #commit{ext=Ext0}=Rec, Kind) when Kind == snmp -> + Rec2 = case lists:keytake(snmp, 1, Ext0) of + false -> + Rec#commit{ext = [{snmp,[Item]}|Ext0]}; + {_, {snmp,Snmp},Ext} -> + Rec#commit{ext = [{snmp,[Item|Snmp]}|Ext]} + end, prepare_node(Node, Storage, Items, Rec2, Kind); prepare_node(Node, Storage, [Item | Items], Rec, Kind) when Kind /= schema -> Rec2 = @@ -1323,7 +1330,15 @@ prepare_node(Node, Storage, [Item | Items], Rec, Kind) when Kind /= schema -> Rec#commit{disc_copies = [Item | Rec#commit.disc_copies]}; disc_only_copies -> Rec#commit{disc_only_copies = - [Item | Rec#commit.disc_only_copies]} + [Item | Rec#commit.disc_only_copies]}; + {ext, Alias, Mod} -> + Ext0 = Rec#commit.ext, + case lists:keytake(ext_copies, 1, Ext0) of + false -> + Rec#commit{ext = [{ext_copies, [{{ext,Alias,Mod}, Item}]}|Ext0]}; + {_,{_,EC},Ext} -> + Rec#commit{ext = [{ext_copies, [{{ext,Alias,Mod}, Item}|EC]}|Ext]} + end end, prepare_node(Node, Storage, Items, Rec2, Kind); prepare_node(_Node, _Storage, Items, Rec, Kind) @@ -1750,16 +1765,30 @@ do_commit(Tid, Bin) when is_binary(Bin) -> do_commit(Tid, binary_to_term(Bin)); do_commit(Tid, C) -> do_commit(Tid, C, optional). + do_commit(Tid, Bin, DumperMode) when is_binary(Bin) -> do_commit(Tid, binary_to_term(Bin), DumperMode); do_commit(Tid, C, DumperMode) -> mnesia_dumper:update(Tid, C#commit.schema_ops, DumperMode), - R = do_snmp(Tid, C#commit.snmp), + R = do_snmp(Tid, proplists:get_value(snmp, C#commit.ext, [])), R2 = do_update(Tid, ram_copies, C#commit.ram_copies, R), R3 = do_update(Tid, disc_copies, C#commit.disc_copies, R2), R4 = do_update(Tid, disc_only_copies, C#commit.disc_only_copies, R3), + R5 = do_update_ext(Tid, C#commit.ext, R4), mnesia_subscr:report_activity(Tid), - R4. + R5. + +%% This could/should be optimized +do_update_ext(_Tid, [], OldRes) -> OldRes; +do_update_ext(Tid, Ext, OldRes) -> + case lists:keyfind(ext_copies, 1, Ext) of + false -> OldRes; + {_, Ops} -> + Do = fun({{ext, _,_} = Storage, Op}, R) -> + do_update(Tid, Storage, [Op], R) + end, + lists:foldl(Do, OldRes, Ops) + end. %% Update the items do_update(Tid, Storage, [Op | Ops], OldRes) -> @@ -1782,12 +1811,12 @@ do_update(_Tid, _Storage, [], Res) -> Res. do_update_op(Tid, Storage, {{Tab, K}, Obj, write}) -> - commit_write(?catch_val({Tab, commit_work}), Tid, + commit_write(?catch_val({Tab, commit_work}), Tid, Storage, Tab, K, Obj, undefined), mnesia_lib:db_put(Storage, Tab, Obj); do_update_op(Tid, Storage, {{Tab, K}, Val, delete}) -> - commit_delete(?catch_val({Tab, commit_work}), Tid, Tab, K, Val, undefined), + commit_delete(?catch_val({Tab, commit_work}), Tid, Storage, Tab, K, Val, undefined), mnesia_lib:db_erase(Storage, Tab, K); do_update_op(Tid, Storage, {{Tab, K}, {RecName, Incr}, update_counter}) -> @@ -1805,86 +1834,84 @@ do_update_op(Tid, Storage, {{Tab, K}, {RecName, Incr}, update_counter}) -> mnesia_lib:db_put(Storage, Tab, Zero), {Zero, []} end, - commit_update(?catch_val({Tab, commit_work}), Tid, Tab, + commit_update(?catch_val({Tab, commit_work}), Tid, Storage, Tab, K, NewObj, OldObjs), element(3, NewObj); do_update_op(Tid, Storage, {{Tab, Key}, Obj, delete_object}) -> commit_del_object(?catch_val({Tab, commit_work}), - Tid, Tab, Key, Obj, undefined), + Tid, Storage, Tab, Key, Obj), mnesia_lib:db_match_erase(Storage, Tab, Obj); do_update_op(Tid, Storage, {{Tab, Key}, Obj, clear_table}) -> - commit_clear(?catch_val({Tab, commit_work}), Tid, Tab, Key, Obj), + commit_clear(?catch_val({Tab, commit_work}), Tid, Storage, Tab, Key, Obj), mnesia_lib:db_match_erase(Storage, Tab, Obj). -commit_write([], _, _, _, _, _) -> ok; -commit_write([{checkpoints, CpList}|R], Tid, Tab, K, Obj, Old) -> +commit_write([], _, _, _, _, _, _) -> ok; +commit_write([{checkpoints, CpList}|R], Tid, Storage, Tab, K, Obj, Old) -> mnesia_checkpoint:tm_retain(Tid, Tab, K, write, CpList), - commit_write(R, Tid, Tab, K, Obj, Old); -commit_write([H|R], Tid, Tab, K, Obj, Old) + commit_write(R, Tid, Storage, Tab, K, Obj, Old); +commit_write([H|R], Tid, Storage, Tab, K, Obj, Old) when element(1, H) == subscribers -> mnesia_subscr:report_table_event(H, Tab, Tid, Obj, write, Old), - commit_write(R, Tid, Tab, K, Obj, Old); -commit_write([H|R], Tid, Tab, K, Obj, Old) + commit_write(R, Tid, Storage, Tab, K, Obj, Old); +commit_write([H|R], Tid, Storage, Tab, K, Obj, Old) when element(1, H) == index -> - mnesia_index:add_index(H, Tab, K, Obj, Old), - commit_write(R, Tid, Tab, K, Obj, Old). + mnesia_index:add_index(H, Storage, Tab, K, Obj, Old), + commit_write(R, Tid, Storage, Tab, K, Obj, Old). -commit_update([], _, _, _, _, _) -> ok; -commit_update([{checkpoints, CpList}|R], Tid, Tab, K, Obj, _) -> +commit_update([], _, _, _, _, _, _) -> ok; +commit_update([{checkpoints, CpList}|R], Tid, Storage, Tab, K, Obj, _) -> Old = mnesia_checkpoint:tm_retain(Tid, Tab, K, write, CpList), - commit_update(R, Tid, Tab, K, Obj, Old); -commit_update([H|R], Tid, Tab, K, Obj, Old) + commit_update(R, Tid, Storage, Tab, K, Obj, Old); +commit_update([H|R], Tid, Storage, Tab, K, Obj, Old) when element(1, H) == subscribers -> mnesia_subscr:report_table_event(H, Tab, Tid, Obj, write, Old), - commit_update(R, Tid, Tab, K, Obj, Old); -commit_update([H|R], Tid, Tab, K, Obj, Old) + commit_update(R, Tid, Storage, Tab, K, Obj, Old); +commit_update([H|R], Tid,Storage, Tab, K, Obj, Old) when element(1, H) == index -> - mnesia_index:add_index(H, Tab, K, Obj, Old), - commit_update(R, Tid, Tab, K, Obj, Old). + mnesia_index:add_index(H, Storage, Tab, K, Obj, Old), + commit_update(R, Tid, Storage, Tab, K, Obj, Old). -commit_delete([], _, _, _, _, _) -> ok; -commit_delete([{checkpoints, CpList}|R], Tid, Tab, K, Obj, _) -> +commit_delete([], _, _, _, _, _, _) -> ok; +commit_delete([{checkpoints, CpList}|R], Tid, Storage, Tab, K, Obj, _) -> Old = mnesia_checkpoint:tm_retain(Tid, Tab, K, delete, CpList), - commit_delete(R, Tid, Tab, K, Obj, Old); -commit_delete([H|R], Tid, Tab, K, Obj, Old) + commit_delete(R, Tid, Storage, Tab, K, Obj, Old); +commit_delete([H|R], Tid, Storage, Tab, K, Obj, Old) when element(1, H) == subscribers -> mnesia_subscr:report_table_event(H, Tab, Tid, Obj, delete, Old), - commit_delete(R, Tid, Tab, K, Obj, Old); -commit_delete([H|R], Tid, Tab, K, Obj, Old) + commit_delete(R, Tid, Storage, Tab, K, Obj, Old); +commit_delete([H|R], Tid, Storage, Tab, K, Obj, Old) when element(1, H) == index -> - mnesia_index:delete_index(H, Tab, K), - commit_delete(R, Tid, Tab, K, Obj, Old). + mnesia_index:delete_index(H, Storage, Tab, K), + commit_delete(R, Tid, Storage, Tab, K, Obj, Old). commit_del_object([], _, _, _, _, _) -> ok; -commit_del_object([{checkpoints, CpList}|R], Tid, Tab, K, Obj, _) -> - Old = mnesia_checkpoint:tm_retain(Tid, Tab, K, delete_object, CpList), - commit_del_object(R, Tid, Tab, K, Obj, Old); -commit_del_object([H|R], Tid, Tab, K, Obj, Old) - when element(1, H) == subscribers -> - mnesia_subscr:report_table_event(H, Tab, Tid, Obj, delete_object, Old), - commit_del_object(R, Tid, Tab, K, Obj, Old); -commit_del_object([H|R], Tid, Tab, K, Obj, Old) - when element(1, H) == index -> - mnesia_index:del_object_index(H, Tab, K, Obj, Old), - commit_del_object(R, Tid, Tab, K, Obj, Old). - -commit_clear([], _, _, _, _) -> ok; -commit_clear([{checkpoints, CpList}|R], Tid, Tab, K, Obj) -> +commit_del_object([{checkpoints, CpList}|R], Tid, Storage, Tab, K, Obj) -> + mnesia_checkpoint:tm_retain(Tid, Tab, K, delete_object, CpList), + commit_del_object(R, Tid, Storage, Tab, K, Obj); +commit_del_object([H|R], Tid, Storage, Tab, K, Obj) when element(1, H) == subscribers -> + mnesia_subscr:report_table_event(H, Tab, Tid, Obj, delete_object), + commit_del_object(R, Tid, Storage, Tab, K, Obj); +commit_del_object([H|R], Tid, Storage, Tab, K, Obj) when element(1, H) == index -> + mnesia_index:del_object_index(H, Storage, Tab, K, Obj), + commit_del_object(R, Tid, Storage, Tab, K, Obj). + +commit_clear([], _, _, _, _, _) -> ok; +commit_clear([{checkpoints, CpList}|R], Tid, Storage, Tab, K, Obj) -> mnesia_checkpoint:tm_retain(Tid, Tab, K, clear_table, CpList), - commit_clear(R, Tid, Tab, K, Obj); -commit_clear([H|R], Tid, Tab, K, Obj) + commit_clear(R, Tid, Storage, Tab, K, Obj); +commit_clear([H|R], Tid, Storage, Tab, K, Obj) when element(1, H) == subscribers -> mnesia_subscr:report_table_event(H, Tab, Tid, Obj, clear_table, undefined), - commit_clear(R, Tid, Tab, K, Obj); -commit_clear([H|R], Tid, Tab, K, Obj) + commit_clear(R, Tid, Storage, Tab, K, Obj); +commit_clear([H|R], Tid, Storage, Tab, K, Obj) when element(1, H) == index -> mnesia_index:clear_index(H, Tab, K, Obj), - commit_clear(R, Tid, Tab, K, Obj). + commit_clear(R, Tid, Storage, Tab, K, Obj). do_snmp(_, []) -> ok; -do_snmp(Tid, [Head | Tail]) -> +do_snmp(Tid, [Head|Tail]) -> try mnesia_snmp_hook:update(Head) catch _:Reason -> %% This should only happen when we recently have @@ -1896,31 +1923,34 @@ do_snmp(Tid, [Head | Tail]) -> end, do_snmp(Tid, Tail). -commit_nodes([C | Tail], AccD, AccR) - when C#commit.disc_copies == [], - C#commit.disc_only_copies == [], - C#commit.schema_ops == [] -> - commit_nodes(Tail, AccD, [C#commit.node | AccR]); commit_nodes([C | Tail], AccD, AccR) -> - commit_nodes(Tail, [C#commit.node | AccD], AccR); + case C of + #commit{disc_copies=[], disc_only_copies=[], schema_ops=[], ext=Ext} -> + case lists:keyfind(ext_copies, 1, Ext) of + false -> commit_nodes(Tail, AccD, [C#commit.node | AccR]); + _ -> commit_nodes(Tail, [C#commit.node | AccD], AccR) + end; + _ -> + commit_nodes(Tail, [C#commit.node | AccD], AccR) + end; commit_nodes([], AccD, AccR) -> {AccD, AccR}. commit_decision(D, [C | Tail], AccD, AccR) -> N = C#commit.node, {D2, Tail2} = - case C#commit.schema_ops of - [] when C#commit.disc_copies == [], - C#commit.disc_only_copies == [] -> - commit_decision(D, Tail, AccD, [N | AccR]); - [] -> + case C of + #commit{disc_copies=[], disc_only_copies=[], schema_ops=[], ext=Ext} -> + case lists:keyfind(ext_copies, 1, Ext) of + false -> commit_decision(D, Tail, AccD, [N | AccR]); + _ -> commit_decision(D, Tail, [N | AccD], AccR) + end; + #commit{schema_ops=[]} -> commit_decision(D, Tail, [N | AccD], AccR); - Ops -> + #commit{schema_ops=Ops} -> case ram_only_ops(N, Ops) of - true -> - commit_decision(D, Tail, AccD, [N | AccR]); - false -> - commit_decision(D, Tail, [N | AccD], AccR) + true -> commit_decision(D, Tail, AccD, [N | AccR]); + false -> commit_decision(D, Tail, [N | AccD], AccR) end end, {D2, [C#commit{decision = D2} | Tail2]}; @@ -1948,7 +1978,7 @@ sync_send_dirty(Tid, [Head | Tail], Tab, WaitFor) -> Res = do_dirty(Tid, Head), {WF, Res}; true -> - {?MODULE, Node} ! {self(), {sync_dirty, Tid, Head, Tab}}, + {?MODULE, Node} ! {self(), {sync_dirty, Tid, ext_format(Head), Tab}}, sync_send_dirty(Tid, Tail, Tab, [Node | WaitFor]) end; sync_send_dirty(_Tid, [], _Tab, WaitFor) -> @@ -1967,11 +1997,11 @@ async_send_dirty(Tid, [Head | Tail], Tab, ReadNode, WaitFor, Res) -> NewRes = do_dirty(Tid, Head), async_send_dirty(Tid, Tail, Tab, ReadNode, WaitFor, NewRes); ReadNode == Node -> - {?MODULE, Node} ! {self(), {sync_dirty, Tid, Head, Tab}}, + {?MODULE, Node} ! {self(), {sync_dirty, Tid, ext_format(Head), Tab}}, NewRes = {'EXIT', {aborted, {node_not_running, Node}}}, async_send_dirty(Tid, Tail, Tab, ReadNode, [Node | WaitFor], NewRes); true -> - {?MODULE, Node} ! {self(), {async_dirty, Tid, Head, Tab}}, + {?MODULE, Node} ! {self(), {async_dirty, Tid, ext_format(Head), Tab}}, async_send_dirty(Tid, Tail, Tab, ReadNode, WaitFor, Res) end; async_send_dirty(_Tid, [], _Tab, _ReadNode, WaitFor, Res) -> @@ -2028,23 +2058,29 @@ ask_commit(Protocol, Tid, [Head | Tail], DiscNs, RamNs, WaitFor, Local) -> Node == node() -> ask_commit(Protocol, Tid, Tail, DiscNs, RamNs, WaitFor, Head); true -> - Bin = opt_term_to_binary(Protocol, Head, DiscNs++RamNs), - Msg = {ask_commit, Protocol, Tid, Bin, DiscNs, RamNs}, + CR = ext_format(Head), + Msg = {ask_commit, Protocol, Tid, CR, DiscNs, RamNs}, {?MODULE, Node} ! {self(), Msg}, ask_commit(Protocol, Tid, Tail, DiscNs, RamNs, [Node | WaitFor], Local) end; ask_commit(_Protocol, _Tid, [], _DiscNs, _RamNs, WaitFor, Local) -> {WaitFor, Local}. -%% This used to test protocol conversion between mnesia-nodes -%% but it is really dependent on the emulator version on the -%% two nodes (if funs are sent which they are in transform table op). -%% to be safe we let erts do the translation (many times maybe and thus -%% slower but it works. -% opt_term_to_binary(asym_trans, Head, Nodes) -> -% opt_term_to_binary(Nodes, Head); -opt_term_to_binary(_Protocol, Head, _Nodes) -> - Head. +ext_format(#commit{ext=[]}=CR) -> CR; +ext_format(#commit{node=Node, ext=Ext}=CR) -> + case mnesia_monitor:needs_protocol_conversion(Node) of + true -> + case lists:keyfind(snmp, 1, Ext) of + false -> CR#commit{ext=[]}; + {snmp, List} -> CR#commit{ext=List} + end; + false -> CR + end. + +new_cr_format(#commit{ext=[]}=Cr) -> Cr; +new_cr_format(#commit{ext=[{_,_}|_]}=Cr) -> Cr; +new_cr_format(#commit{ext=Snmp}=Cr) -> + Cr#commit{ext=[{snmp,Snmp}]}. rec_all([Node | Tail], Tid, Res, Pids) -> receive diff --git a/lib/mnesia/test/Makefile b/lib/mnesia/test/Makefile index f0efbe6375..5b61b1af65 100644 --- a/lib/mnesia/test/Makefile +++ b/lib/mnesia/test/Makefile @@ -52,7 +52,8 @@ MODULES= \ mnesia_schema_recovery_test \ mnesia_measure_test \ mnesia_cost \ - mnesia_dbn_meters + mnesia_dbn_meters \ + ext_test DocExamplesDir := ../doc/src/ diff --git a/lib/mnesia/test/ext_test.erl b/lib/mnesia/test/ext_test.erl new file mode 100644 index 0000000000..45ddb148bc --- /dev/null +++ b/lib/mnesia/test/ext_test.erl @@ -0,0 +1,237 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 1996-2014. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +-module(ext_test). + +%% Initializations +-export([init_backend/0, add_aliases/1, remove_aliases/1, + check_definition/4, semantics/2]). + +-export([ + create_table/3, load_table/4, + delete_table/2, close_table/2, sync_close_table/2, + + sender_init/4, + receiver_first_message/4, receive_data/5, receive_done/4, + + index_is_consistent/3, is_index_consistent/2, + + real_suffixes/0, tmp_suffixes/0, + + info/3, + fixtable/3, + validate_key/6, validate_record/6, + + first/2, last/2, next/3, prev/3, slot/3, + + insert/3, update_counter/4, + lookup/3, + delete/3, match_delete/3, + select/1, select/3, select/4, repair_continuation/2 + ]). + +-ifdef(DEBUG). +-define(DBG(DATA), io:format("~p:~p: ~p~n",[?MODULE, ?LINE, DATA])). +-define(DBG(FORMAT, ARGS), io:format("~p:~p: " ++ FORMAT,[?MODULE, ?LINE] ++ ARGS)). +-else. +-define(DBG(DATA), ok). +-define(DBG(FORMAT, ARGS), ok). +-endif. + +%% types() -> +%% [{fs_copies, ?MODULE}, +%% {raw_fs_copies, ?MODULE}]. + +semantics(ext_ets, storage) -> ram_copies; +semantics(ext_ets, types ) -> [set, ordered_set, bag]; +semantics(ext_ets, index_types) -> [ordered]; +semantics(_Alias, _) -> + undefined. + +%% valid_op(_, _) -> +%% true. + +init_backend() -> + ?DBG(init_backend), + ok. + +add_aliases(_As) -> + ?DBG(_As), + ok. + +remove_aliases(_) -> + ok. + + +%% Table operations + +check_definition(ext_ets, _Tab, _Nodes, _Props) -> + ?DBG("~p ~p ~p~n", [_Tab, _Nodes, _Props]), + ok. + +create_table(ext_ets, Tab, Props) when is_atom(Tab) -> + Tid = ets:new(Tab, [public, proplists:get_value(type, Props, set), {keypos, 2}]), + ?DBG("~p Create: ~p(~p) ~p~n", [self(), Tab, Tid, Props]), + mnesia_lib:set({?MODULE, Tab}, Tid), + ok; +create_table(_, Tag={Tab, index, {_Where, Type0}}, _Opts) -> + Type = case Type0 of + ordered -> ordered_set; + _ -> Type0 + end, + Tid = ets:new(Tab, [public, Type]), + ?DBG("~p(~p) ~p~n", [Tab, Tid, Tag]), + mnesia_lib:set({?MODULE, Tag}, Tid), + ok; +create_table(_, Tag={_Tab, retainer, ChkPName}, _Opts) -> + Tid = ets:new(ChkPName, [set, public, {keypos, 2}]), + ?DBG("~p(~p) ~p~n", [_Tab, Tid, Tag]), + mnesia_lib:set({?MODULE, Tag}, Tid), + ok. + +delete_table(ext_ets, Tab) -> + try + ets:delete(mnesia_lib:val({?MODULE,Tab})), + mnesia_lib:unset({?MODULE,Tab}), + ok + catch _:_ -> + ?DBG({double_delete, Tab}), + ok + end. + +load_table(ext_ets, _Tab, init_index, _Cs) -> ok; +load_table(ext_ets, _Tab, _LoadReason, _Cs) -> + ?DBG("Load ~p ~p~n", [_Tab, _LoadReason]), + ok. +%% mnesia_monitor:unsafe_create_external(Tab, ext_ets, ?MODULE, Cs). + +sender_init(Alias, Tab, _RemoteStorage, _Pid) -> + KeysPerTransfer = 100, + {standard, + fun() -> mnesia_lib:db_init_chunk({ext,Alias,?MODULE}, Tab, KeysPerTransfer) end, + fun(Cont) -> mnesia_lib:db_chunk({ext,Alias,?MODULE}, Cont) end}. + +receiver_first_message(Sender, {first, Size}, _Alias, Tab) -> + ?DBG({first,Size}), + {Size, {Tab, Sender}}. + +receive_data(Data, ext_ets, Name, _Sender, {Name, Tab, _Sender}=State) -> + ?DBG({Data,State}), + true = ets:insert(Tab, Data), + {more, State}; +receive_data(Data, Alias, Tab, Sender, {Name, Sender}) -> + receive_data(Data, Alias, Tab, Sender, {Name, mnesia_lib:val({?MODULE,Tab}), Sender}). + +receive_done(_Alias, _Tab, _Sender, _State) -> + ?DBG({done,_State}), + ok. + +close_table(Alias, Tab) -> sync_close_table(Alias, Tab). + +sync_close_table(ext_ets, _Tab) -> + ?DBG(_Tab). + +fixtable(ext_ets, Tab, Bool) -> + ?DBG({Tab,Bool}), + ets:safe_fixtable(mnesia_lib:val({?MODULE,Tab}), Bool). + +info(ext_ets, Tab, Type) -> + ?DBG({Tab,Type}), + Tid = mnesia_lib:val({?MODULE,Tab}), + try ets:info(Tid, Type) of + Val -> Val + catch _:_ -> + undefined + end. + +real_suffixes() -> + [".dat"]. + +tmp_suffixes() -> + []. + +%% Index + +index_is_consistent(_Alias, _Ix, _Bool) -> ok. % Ignore for now +is_index_consistent(_Alias, _Ix) -> false. % Always rebuild + +%% Record operations + +validate_record(_Alias, _Tab, RecName, Arity, Type, _Obj) -> + {RecName, Arity, Type}. + +validate_key(_Alias, _Tab, RecName, Arity, Type, _Key) -> + {RecName, Arity, Type}. + +insert(ext_ets, Tab, Obj) -> + ?DBG({Tab,Obj}), + try + ets:insert(mnesia_lib:val({?MODULE,Tab}), Obj), + ok + catch _:Reason -> + io:format("CRASH ~p ~p~n",[Reason, mnesia_lib:val({?MODULE,Tab})]) + end. + +lookup(ext_ets, Tab, Key) -> + ets:lookup(mnesia_lib:val({?MODULE,Tab}), Key). + +delete(ext_ets, Tab, Key) -> + ets:delete(mnesia_lib:val({?MODULE,Tab}), Key). + +match_delete(ext_ets, Tab, Pat) -> + ets:match_delete(mnesia_lib:val({?MODULE,Tab}), Pat). + +first(ext_ets, Tab) -> + ets:first(mnesia_lib:val({?MODULE,Tab})). + +last(Alias, Tab) -> first(Alias, Tab). + +next(ext_ets, Tab, Key) -> + ets:next(mnesia_lib:val({?MODULE,Tab}), Key). + +prev(Alias, Tab, Key) -> + next(Alias, Tab, Key). + +slot(ext_ets, Tab, Pos) -> + ets:slot(mnesia_lib:val({?MODULE,Tab}), Pos). + +update_counter(ext_ets, Tab, C, Val) -> + ets:update_counter(mnesia_lib:val({?MODULE,Tab}), C, Val). + +select('$end_of_table' = End) -> End; +select({ext_ets, C}) -> ets:select(C). + +select(Alias, Tab, Ms) -> + Res = select(Alias, Tab, Ms, 100000), + select_1(Res). + +select_1('$end_of_table') -> []; +select_1({Acc, C}) -> + case ets:select(C) of + '$end_of_table' -> Acc; + {New, Cont} -> + select_1({New ++ Acc, Cont}) + end. + +select(ext_ets, Tab, Ms, Limit) when is_integer(Limit); Limit =:= infinity -> + ets:select(mnesia_lib:val({?MODULE,Tab}), Ms, Limit). + +repair_continuation({Alias, Cont}, Ms) -> + {Alias, ets:repair_continuation(Cont, Ms)}. diff --git a/lib/mnesia/test/mnesia_config_test.erl b/lib/mnesia/test/mnesia_config_test.erl index 204b8fa394..45da909264 100644 --- a/lib/mnesia/test/mnesia_config_test.erl +++ b/lib/mnesia/test/mnesia_config_test.erl @@ -1100,8 +1100,8 @@ dynamic_basic(Config) when is_list(Config) -> ?match(ok, mnesia:dirty_write({tab1, 1, 1})), ?match(ok, mnesia:dirty_write({tab2, 1, 1})), - - ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes, [N1]}]])), + + ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes, [N1]}, {schema, ?BACKEND}]])), ?match(ok, rpc:call(N2, mnesia, wait_for_tables, [[tab1,tab2],5000])), io:format("Here ~p ~n",[?LINE]), check_storage(N2, N1, [N3]), @@ -1112,7 +1112,7 @@ dynamic_basic(Config) when is_list(Config) -> ?match(ok, mnesia:delete_schema([N3])), io:format("T1 ~p ~n",[rpc:call(N3,?MODULE,c_nodes,[])]), - ?match(ok, rpc:call(N3, mnesia, start, [])), + ?match(ok, rpc:call(N3, mnesia, start, [[{schema, ?BACKEND}]])), io:format("T2 ~p ~n",[rpc:call(N3,?MODULE,c_nodes,[])]), timer:sleep(2000), io:format("T3 ~p ~n",[rpc:call(N3,?MODULE,c_nodes,[])]), @@ -1127,7 +1127,7 @@ dynamic_basic(Config) when is_list(Config) -> ?match([], mnesia_test_lib:kill_mnesia([N3])), ?match(ok, mnesia:delete_schema([N3])), - ?match(ok, rpc:call(N3, mnesia, start, [])), + ?match(ok, rpc:call(N3, mnesia, start, [[{schema, ?BACKEND}]])), ?match({ok, [N3]}, sort(?rpc_connect(N1, [N3]))), ?match(ok, rpc:call(N3, mnesia, wait_for_tables, [[tab1,tab2],5000])), io:format("Here ~p ~n",[?LINE]), @@ -1143,7 +1143,7 @@ dynamic_basic(Config) when is_list(Config) -> % mnesia should come up now. ?match({atomic, ok}, mnesia:add_table_copy(tab1, N2, ram_copies)), - ?match(ok, rpc:call(N2, mnesia, start, [])), + ?match(ok, rpc:call(N2, mnesia, start, [[{schema, ?BACKEND}]])), ?match({ok, _}, sort(?rpc_connect(N2, [N3]))), ?match(SNs, sort(rpc:call(N1, mnesia, system_info, [running_db_nodes]))), @@ -1162,7 +1162,7 @@ dynamic_basic(Config) when is_list(Config) -> ?match([N3,N1], sort(rpc:call(N1, mnesia, system_info, [running_db_nodes]))), ?match([N3,N1], sort(rpc:call(N3, mnesia, system_info, [running_db_nodes]))), - ?match(ok, rpc:call(N2, mnesia, start, [])), + ?match(ok, rpc:call(N2, mnesia, start, [[{schema, ?BACKEND}]])), ?match({ok, _}, sort(?rpc_connect(N3, [N2]))), ?match(SNs, sort(rpc:call(N1, mnesia, system_info, [running_db_nodes]))), @@ -1192,7 +1192,7 @@ dynamic_ext(Config) when is_list(Config) -> mnesia_test_lib:kill_mnesia([N2]), ?match(ok, mnesia:delete_schema([N2])), - ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes, [N1]}]])), + ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes, [N1]}, {schema, ?BACKEND}]])), ?match(SNs, sort(rpc:call(N1, mnesia, system_info, [running_db_nodes]))), ?match(SNs, sort(rpc:call(N2, mnesia, system_info, [running_db_nodes]))), @@ -1213,7 +1213,7 @@ dynamic_ext(Config) when is_list(Config) -> ?match(ok, mnesia:dirty_write({tab3, 42, T})), ?match(stopped, rpc:call(N2, mnesia, stop, [])), - ?match(ok, rpc:call(N2, mnesia, start, [])), + ?match(ok, rpc:call(N2, mnesia, start, [[{schema, ?BACKEND}]])), ?match(SNs, sort(rpc:call(N2, mnesia, system_info, [running_db_nodes]))), ?match(ok, mnesia:wait_for_tables([tab0,tab1,tab2,tab3], 10000)), ?match(ok, rpc:call(N2, mnesia, wait_for_tables, [[tab1,tab2,tab3], 100])), @@ -1226,7 +1226,7 @@ dynamic_ext(Config) when is_list(Config) -> ?match(stopped, rpc:call(N1, mnesia, stop, [])), - ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes,[N1,N2]}]])), + ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes,[N1,N2]}, {schema, ?BACKEND}]])), ?match({timeout,[tab0]}, rpc:call(N2, mnesia, wait_for_tables, [[tab0], 500])), ?match(ok, rpc:call(N1, mnesia, start, [[{extra_db_nodes, [N1,N2]}]])), @@ -1238,7 +1238,7 @@ dynamic_ext(Config) when is_list(Config) -> ?match(stopped, rpc:call(N1, mnesia, stop, [])), mnesia_test_lib:kill_mnesia([N2]), ?match(ok, mnesia:delete_schema([N2])), - ?match(ok, rpc:call(N1, mnesia, start, [[{extra_db_nodes, [N1,N2]}]])), + ?match(ok, rpc:call(N1, mnesia, start, [[{extra_db_nodes, [N1,N2]}, {schema, ?BACKEND}]])), ?match({timeout,[tab0]}, rpc:call(N1, mnesia, wait_for_tables, [[tab0], 500])), ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes,[N1,N2]}]])), diff --git a/lib/mnesia/test/mnesia_dirty_access_test.erl b/lib/mnesia/test/mnesia_dirty_access_test.erl index 88b3a215e1..c9df8ed353 100644 --- a/lib/mnesia/test/mnesia_dirty_access_test.erl +++ b/lib/mnesia/test/mnesia_dirty_access_test.erl @@ -40,43 +40,40 @@ all() -> groups() -> [{dirty_write, [], - [dirty_write_ram, dirty_write_disc, - dirty_write_disc_only]}, + [dirty_write_ram, dirty_write_disc, dirty_write_disc_only, + dirty_write_xets]}, {dirty_read, [], - [dirty_read_ram, dirty_read_disc, - dirty_read_disc_only]}, + [dirty_read_ram, dirty_read_disc, dirty_read_disc_only, dirty_read_xets]}, {dirty_update_counter, [], [dirty_update_counter_ram, dirty_update_counter_disc, - dirty_update_counter_disc_only]}, + dirty_update_counter_disc_only, dirty_update_counter_xets]}, {dirty_delete, [], [dirty_delete_ram, dirty_delete_disc, - dirty_delete_disc_only]}, + dirty_delete_disc_only, dirty_delete_xets]}, {dirty_delete_object, [], [dirty_delete_object_ram, dirty_delete_object_disc, - dirty_delete_object_disc_only]}, + dirty_delete_object_disc_only, dirty_delete_object_xets]}, {dirty_match_object, [], [dirty_match_object_ram, dirty_match_object_disc, - dirty_match_object_disc_only]}, + dirty_match_object_disc_only, dirty_match_object_xets]}, {dirty_index, [], [{group, dirty_index_match_object}, {group, dirty_index_read}, {group, dirty_index_update}]}, {dirty_index_match_object, [], - [dirty_index_match_object_ram, - dirty_index_match_object_disc, - dirty_index_match_object_disc_only]}, + [dirty_index_match_object_ram, dirty_index_match_object_disc, + dirty_index_match_object_disc_only, dirty_index_match_object_xets]}, {dirty_index_read, [], [dirty_index_read_ram, dirty_index_read_disc, - dirty_index_read_disc_only]}, + dirty_index_read_disc_only, dirty_index_read_xets]}, {dirty_index_update, [], - [dirty_index_update_set_ram, - dirty_index_update_set_disc, - dirty_index_update_set_disc_only, + [dirty_index_update_set_ram, dirty_index_update_set_disc, + dirty_index_update_set_disc_only, dirty_index_update_set_xets, dirty_index_update_bag_ram, dirty_index_update_bag_disc, - dirty_index_update_bag_disc_only]}, + dirty_index_update_bag_disc_only, dirty_index_update_bag_xets]}, {dirty_iter, [], - [dirty_iter_ram, dirty_iter_disc, - dirty_iter_disc_only]}, + [dirty_iter_ram, dirty_iter_disc, dirty_iter_disc_only, + dirty_iter_xets]}, {admin_tests, [], [del_table_copy_1, del_table_copy_2, del_table_copy_3, add_table_copy_1, add_table_copy_2, add_table_copy_3, @@ -106,6 +103,9 @@ dirty_write_disc_only(suite) -> []; dirty_write_disc_only(Config) when is_list(Config) -> dirty_write(Config, disc_only_copies). +dirty_write_xets(Config) when is_list(Config) -> + dirty_write(Config, ext_ets). + dirty_write(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_write, @@ -137,6 +137,9 @@ dirty_read_disc_only(suite) -> []; dirty_read_disc_only(Config) when is_list(Config) -> dirty_read(Config, disc_only_copies). +dirty_read_xets(Config) when is_list(Config) -> + dirty_read(Config, ext_ets). + dirty_read(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_read, @@ -180,6 +183,9 @@ dirty_update_counter_disc_only(suite) -> []; dirty_update_counter_disc_only(Config) when is_list(Config) -> dirty_update_counter(Config, disc_only_copies). +dirty_update_counter_xets(Config) when is_list(Config) -> + dirty_update_counter(Config, ext_ets). + dirty_update_counter(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_update_counter, @@ -222,6 +228,9 @@ dirty_delete_disc_only(suite) -> []; dirty_delete_disc_only(Config) when is_list(Config) -> dirty_delete(Config, disc_only_copies). +dirty_delete_xets(Config) when is_list(Config) -> + dirty_delete(Config, ext_ets). + dirty_delete(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_delete, @@ -259,6 +268,9 @@ dirty_delete_object_disc_only(suite) -> []; dirty_delete_object_disc_only(Config) when is_list(Config) -> dirty_delete_object(Config, disc_only_copies). +dirty_delete_object_xets(Config) when is_list(Config) -> + dirty_delete_object(Config, ext_ets). + dirty_delete_object(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_delete_object, @@ -302,6 +314,9 @@ dirty_match_object_disc_only(suite) -> []; dirty_match_object_disc_only(Config) when is_list(Config) -> dirty_match_object(Config, disc_only_copies). +dirty_match_object_xets(Config) when is_list(Config) -> + dirty_match_object(Config, ext_ets). + dirty_match_object(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_match, @@ -326,7 +341,6 @@ dirty_match_object(Config, Storage) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Dirty read matching records by using an index - dirty_index_match_object_ram(suite) -> []; dirty_index_match_object_ram(Config) when is_list(Config) -> dirty_index_match_object(Config, ram_copies). @@ -339,6 +353,9 @@ dirty_index_match_object_disc_only(suite) -> []; dirty_index_match_object_disc_only(Config) when is_list(Config) -> dirty_index_match_object(Config, disc_only_copies). +dirty_index_match_object_xets(Config) when is_list(Config) -> + dirty_index_match_object(Config, ext_ets). + dirty_index_match_object(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_index_match_object, @@ -376,6 +393,9 @@ dirty_index_read_disc_only(suite) -> []; dirty_index_read_disc_only(Config) when is_list(Config) -> dirty_index_read(Config, disc_only_copies). +dirty_index_read_xets(Config) when is_list(Config) -> + dirty_index_read(Config, ext_ets). + dirty_index_read(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_index_read, @@ -437,6 +457,9 @@ dirty_index_update_set_disc_only(suite) -> []; dirty_index_update_set_disc_only(Config) when is_list(Config) -> dirty_index_update_set(Config, disc_only_copies). +dirty_index_update_set_xets(Config) when is_list(Config) -> + dirty_index_update_set(Config, ext_ets). + dirty_index_update_set(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = index_test, @@ -525,6 +548,9 @@ dirty_index_update_bag_disc_only(suite) -> []; dirty_index_update_bag_disc_only(Config)when is_list(Config) -> dirty_index_update_bag(Config, disc_only_copies). +dirty_index_update_bag_xets(Config) when is_list(Config) -> + dirty_index_update_bag(Config, ext_ets). + dirty_index_update_bag(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = index_test, @@ -631,7 +657,6 @@ dirty_index_update_bag(Config, Storage) -> %% Dirty iteration %% dirty_slot, dirty_first, dirty_next - dirty_iter_ram(suite) -> []; dirty_iter_ram(Config) when is_list(Config) -> dirty_iter(Config, ram_copies). @@ -644,6 +669,9 @@ dirty_iter_disc_only(suite) -> []; dirty_iter_disc_only(Config) when is_list(Config) -> dirty_iter(Config, disc_only_copies). +dirty_iter_xets(Config) when is_list(Config) -> + dirty_iter(Config, ext_ets). + dirty_iter(Config, Storage) -> [Node1] = Nodes = ?acquire_nodes(1, Config), Tab = dirty_iter, diff --git a/lib/mnesia/test/mnesia_evil_coverage_test.erl b/lib/mnesia/test/mnesia_evil_coverage_test.erl index 2ed62b1538..6e34040bc4 100644 --- a/lib/mnesia/test/mnesia_evil_coverage_test.erl +++ b/lib/mnesia/test/mnesia_evil_coverage_test.erl @@ -72,7 +72,9 @@ groups() -> {record_name_dirty_access, [], [record_name_dirty_access_ram, record_name_dirty_access_disc, - record_name_dirty_access_disc_only]}]. + record_name_dirty_access_disc_only, + record_name_dirty_access_xets + ]}]. init_per_group(_GroupName, Config) -> Config. @@ -112,6 +114,7 @@ system_info(Config) when is_list(Config) -> ?match(I when is_integer(I), mnesia:system_info(transaction_log_writes)), ?match(I when is_integer(I), mnesia:system_info(send_compressed)), ?match(L when is_list(L), mnesia:system_info(all)), + ?match(L when is_list(L), mnesia:system_info(backend_types)), ?match({'EXIT', {aborted, Reason }} when element(1, Reason) == badarg , mnesia:system_info(ali_baba)), ?verify_mnesia(Nodes, []). @@ -132,11 +135,11 @@ table_info(Config) when is_list(Config) -> Schema = case mnesia_test_lib:diskless(Config) of true -> [{type, Type}, {attributes, Attrs}, {index, [ValPos]}, - {ram_copies, Nodes}]; + {ram_copies, [Node1, Node2]}, {ext_ets, [Node3]}]; false -> [{type, Type}, {attributes, Attrs}, {index, [ValPos]}, - {disc_only_copies, [Node1]}, {ram_copies, [Node2]}, - {disc_copies, [Node3]}] + {disc_only_copies, [Node1]}, {ram_copies, [Node2]}, + {ext_ets, [Node3]}] end, ?match({atomic, ok}, mnesia:create_table(Tab, Schema)), @@ -144,28 +147,22 @@ table_info(Config) when is_list(Config) -> Keys = lists:seq(1, Size), Records = [{Tab, A, 7} || A <- Keys], lists:foreach(fun(Rec) -> ?match(ok, mnesia:dirty_write(Rec)) end, Records), - ?match(Mem when is_integer(Mem), mnesia:table_info(Tab, memory)), - ?match(Size, mnesia:table_info(Tab, size)), - ?match(Type, mnesia:table_info(Tab, type)), case mnesia_test_lib:diskless(Config) of true -> ?match(Nodes, mnesia:table_info(Tab, ram_copies)); false -> - ?match([Node3], mnesia:table_info(Tab, mnesia_test_lib:storage_type(disc_copies, Config))), + ?match([Node3], mnesia:table_info(Tab, ext_ets)), ?match([Node2], mnesia:table_info(Tab, ram_copies)), ?match([Node1], mnesia:table_info(Tab, mnesia_test_lib:storage_type(disc_only_copies, Config))) end, Read = [Node1, Node2, Node3], - ?match(true, lists:member(mnesia:table_info(Tab, where_to_read), Read)), Write = ?sort([Node1, Node2, Node3]), - ?match(Write, ?sort(mnesia:table_info(Tab, where_to_write))), - ?match([ValPos], mnesia:table_info(Tab, index)), - ?match(Arity, mnesia:table_info(Tab, arity)), - ?match(Attrs, mnesia:table_info(Tab, attributes)), - ?match({Tab, '_', '_'}, mnesia:table_info(Tab, wild_pattern)), - ?match({atomic, Attrs}, mnesia:transaction(fun() -> - mnesia:table_info(Tab, attributes) end)), + + {[ok,ok,ok], []} = rpc:multicall(Nodes, ?MODULE, info_check, + [Tab, Read, Write, Size, Type, ValPos, Arity, Attrs]), + + ?match({atomic, Attrs}, mnesia:transaction(fun() -> mnesia:table_info(Tab, attributes) end)), ?match(L when is_list(L), mnesia:table_info(Tab, all)), @@ -179,6 +176,17 @@ table_info(Config) when is_list(Config) -> ?match(0, mnesia:table_info(tab_info, size)), ?verify_mnesia([Node1, Node3], [Node2]). +info_check(Tab, Read, Write, Size, Type, ValPos, Arity, Attrs) -> + ?match(true, lists:member(mnesia:table_info(Tab, where_to_read), Read)), + ?match(Write, ?sort(mnesia:table_info(Tab, where_to_write))), + ?match(Mem when is_integer(Mem), mnesia:table_info(Tab, memory)), + ?match(Size, mnesia:table_info(Tab, size)), + ?match(Type, mnesia:table_info(Tab, type)), + ?match([ValPos], mnesia:table_info(Tab, index)), + ?match(Arity, mnesia:table_info(Tab, arity)), + ?match(Attrs, mnesia:table_info(Tab, attributes)), + ?match({Tab, '_', '_'}, mnesia:table_info(Tab, wild_pattern)), + ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Check the error descriptions @@ -216,11 +224,12 @@ db_node_lifecycle(Config) when is_list(Config) -> [Node1, Node2, Node3] = AllNodes = ?acquire_nodes(3, Config), Tab = db_node_lifecycle, - Who = fun(T) -> + Who = fun(T) -> L1 = mnesia:table_info(T, ram_copies), L2 = mnesia:table_info(T, disc_copies), L3 = mnesia:table_info(T, disc_only_copies), - L1 ++ L2 ++ L3 + L4 = mnesia:table_info(T, ext_ets), + L1 ++ L2 ++ L3 ++ L4 end, SNs = ?sort(AllNodes), @@ -235,6 +244,7 @@ db_node_lifecycle(Config) when is_list(Config) -> ?match({error, _}, mnesia:create_schema([foo@bar])), ?match(ok, mnesia:start()), ?match(false, mnesia:system_info(use_dir)), + ?match([ram_copies, disc_copies, disc_only_copies], mnesia:system_info(backend_types)), ?match({atomic, ok}, mnesia:create_table(Tab, [])), ?match({aborted, {has_no_disc, Node1}}, mnesia:dump_tables([Tab])), ?match({aborted, {has_no_disc, Node1}}, mnesia:change_table_copy_type(Tab, node(), disc_copies)), @@ -242,7 +252,7 @@ db_node_lifecycle(Config) when is_list(Config) -> ?match(stopped, mnesia:stop()), - ?match(ok, mnesia:create_schema(AllNodes)), + ?match(ok, mnesia:create_schema(AllNodes, ?BACKEND)), ?match([], mnesia_test_lib:start_mnesia(AllNodes)), ?match([SNs, SNs, SNs], @@ -271,12 +281,15 @@ db_node_lifecycle(Config) when is_list(Config) -> Tab3 = not_local, Tab4 = local, Tab5 = remote, + Tab6 = ext1, Tabs = [Schema, [{name, Tab2}, {disc_copies, AllNodes}], [{name, Tab3}, {ram_copies, [Node2, Node3]}], [{name, Tab4}, {disc_only_copies, [Node1]}], - [{name, Tab5}, {disc_only_copies, [Node2]}]], + [{name, Tab5}, {disc_only_copies, [Node2]}], + [{name, Tab6}, {ext_ets, [Node1, Node2]}] + ], [?match({atomic, ok}, mnesia:create_table(T)) || T <- Tabs ], @@ -287,31 +300,28 @@ db_node_lifecycle(Config) when is_list(Config) -> ?match({aborted, {node_not_running, Node1}}, mnesia:del_table_copy(schema, Node2)), - ?match([], mnesia_test_lib:start_mnesia([Node1],[Tab2,Tab4])), + ?match([], mnesia_test_lib:start_mnesia([Node1],[Tab2,Tab4,Tab6])), ?match([], mnesia_test_lib:stop_mnesia([Node2])), - ?match({atomic, ok}, - mnesia:del_table_copy(schema, Node2)), + ?match({atomic, ok}, mnesia:del_table_copy(schema, Node2)), - %% Check + %% Check RemNodes = AllNodes -- [Node2], - ?match(RemNodes, mnesia:system_info(db_nodes)), + ?match(RemNodes, mnesia:system_info(db_nodes)), ?match([Node1], Who(Tab)), ?match(RemNodes, Who(Tab2)), ?match([Node3], Who(Tab3)), ?match([Node1], Who(Tab4)), ?match({'EXIT', {aborted, {no_exists, _, _}}}, Who(Tab5)), + ?match([Node1], Who(Tab6)), - ?match({atomic, ok}, - mnesia:change_table_copy_type(Tab2, Node3, ram_copies)), + ?match({atomic, ok}, mnesia:change_table_copy_type(Tab2, Node3, ram_copies)), - ?match({atomic, ok}, - mnesia:change_table_copy_type(schema, Node3, ram_copies)), + ?match({atomic, ok}, mnesia:change_table_copy_type(schema, Node3, ram_copies)), ?match([], mnesia_test_lib:stop_mnesia([Node3])), - ?match({atomic, ok}, - mnesia:del_table_copy(schema, Node3)), - ?match([Node1], mnesia:system_info(db_nodes)), + ?match({atomic, ok}, mnesia:del_table_copy(schema, Node3)), + ?match([Node1], mnesia:system_info(db_nodes)), ?match([Node1], Who(Tab)), ?match([Node1], Who(Tab2)), ?match({'EXIT', {aborted, {no_exists, _, _}}}, Who(Tab3)), @@ -363,7 +373,8 @@ start_and_stop(Config) when is_list(Config) -> checkpoint(suite) -> []; checkpoint(Config) when is_list(Config) -> checkpoint(2, Config), - checkpoint(3, Config). + checkpoint(3, Config), + ok. checkpoint(NodeConfig, Config) -> [Node1 | _] = TabNodes = ?acquire_nodes(NodeConfig, Config), @@ -381,7 +392,7 @@ checkpoint(NodeConfig, Config) -> CreateTab(Type, 3, [lists:last(TabNodes)])] ++ Acc end, - Types = [ram_copies, disc_copies, disc_only_copies], + Types = [ram_copies, disc_copies, disc_only_copies, ext_ets], Tabs = lists:foldl(CreateTabs, [], Types), Recs = ?sort([{T, N, N} || T <- Tabs, N <- lists:seq(1, 10)]), lists:foreach(fun(R) -> ?match(ok, mnesia:dirty_write(R)) end, Recs), @@ -430,25 +441,29 @@ checkpoint(NodeConfig, Config) -> replica_location(suite) -> []; replica_location(Config) when is_list(Config) -> [Node1, Node2, Node3] = Nodes = ?acquire_nodes(3, Config), - Tab = replica_location, %% Create three replicas - Schema = [{name, Tab}, {disc_only_copies, [Node1]}, - {ram_copies, [Node2]}, {disc_copies, [Node3]}], - ?match({atomic, ok}, mnesia:create_table(Schema)), - ?match([], ?vrl(Tab, [Node1], [Node2], [Node3], Nodes)), + Check = fun(Tab, Schema) -> + ?match({atomic, ok}, mnesia:create_table([{name, Tab}|Schema])), + ?match([], ?vrl(Tab, [Node1], [Node2], [Node3], Nodes)), - %% Delete one replica - ?match({atomic, ok}, mnesia:del_table_copy(Tab, Node2)), - ?match([], ?vrl(Tab, [Node1], [], [Node3], Nodes)), + %% Delete one replica + ?match({atomic, ok}, mnesia:del_table_copy(Tab, Node2)), + ?match([], ?vrl(Tab, [Node1], [], [Node3], Nodes)), - %% Move one replica - ?match({atomic, ok}, mnesia:move_table_copy(Tab, Node1, Node2)), - ?match([], ?vrl(Tab, [Node2], [], [Node3], Nodes)), + %% Move one replica + ?match({atomic, ok}, mnesia:move_table_copy(Tab, Node1, Node2)), + ?match([], ?vrl(Tab, [Node2], [], [Node3], Nodes)), + + %% Change replica type + ?match({atomic, ok}, mnesia:change_table_copy_type(Tab, Node2, ram_copies)), + ?match([], ?vrl(Tab, [], [Node2], [Node3], Nodes)) + end, + Check(replica_location, [{disc_only_copies, [Node1]}, + {ram_copies, [Node2]}, {disc_copies, [Node3]}]), - %% Change replica type - ?match({atomic, ok}, mnesia:change_table_copy_type(Tab, Node2, ram_copies)), - ?match([], ?vrl(Tab, [], [Node2], [Node3], Nodes)), + Check(ext_location, [{disc_only_copies, [Node1]}, + {ext_ets, [Node2]}, {disc_copies, [Node3]}]), ?verify_mnesia(Nodes, []). @@ -720,7 +735,7 @@ replica_management(Config) when is_list(Config) -> %% ?match({atomic, ok}, mnesia:create_table([{name, Tab}, {attributes, Attrs}, - {ram_copies, [Node1, Node3]}])), + {ram_copies, [Node1]}, {ext_ets, [Node3]}])), [?match(ok, mnesia:dirty_write({Tab, K, K + 2})) || K <-lists:seq(1, 10)], ?match([], ?vrl(Tab, [], [Node1, Node3], [], Nodes)), %% R - - @@ -757,7 +772,7 @@ replica_management(Config) when is_list(Config) -> ?match([], ?vrl(Tab, [Node2], [], [Node1], Nodes)), ?match([0,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), %% D DO - - ?match({atomic, ok}, mnesia:add_table_copy(Tab, Node3, ram_copies)), + ?match({atomic, ok}, mnesia:add_table_copy(Tab, Node3, ext_ets)), ?match([], ?vrl(Tab, [Node2], [Node3], [Node1], Nodes)), ?match([10,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), %% D DO R @@ -784,7 +799,7 @@ replica_management(Config) when is_list(Config) -> ?match([10,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), %% D DO D0 - ?match({atomic, ok}, mnesia:change_table_copy_type(Tab, Node3, ram_copies)), + ?match({atomic, ok}, mnesia:change_table_copy_type(Tab, Node3, ext_ets)), ?match([], ?vrl(Tab, [Node2], [Node3], [Node1], Nodes)), ?match([10,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), %% D DO R @@ -841,18 +856,31 @@ replica_management(Config) when is_list(Config) -> ?match([], ?vrl(Tab, [Node3], [], [Node2], Nodes)), ?match([0,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), %% - D DO + ?match({atomic, ok}, mnesia:change_table_copy_type(Tab, Node3, ext_ets)), + ?match([], ?vrl(Tab, [], [Node3], [Node2], Nodes)), + ?match([0,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), + %% - D ER + ?match({atomic, ok}, mnesia:move_table_copy(Tab, Node3, Node1)), + ?match([], ?vrl(Tab, [], [Node1], [Node2], Nodes)), + ?match([0,10,10], ?SS(rpc:multicall(Nodes, mnesia, table_info, [Tab, size]))), + %% ER D - + ?match({aborted, _}, mnesia:move_table_copy(Tab, Node1, Node2)), + ?match({aborted, _}, mnesia:move_table_copy(Tab, Node3, Node2)), + ?match({atomic, ok}, mnesia:move_table_copy(Tab, Node1, Node3)), + %% - D ER ?match([], mnesia_test_lib:stop_mnesia([Node3])), ?match({atomic,ok}, mnesia:transaction(fun() -> mnesia:write({Tab, 43, sync_me}) end)), - ?match([], ?vrl(Tab, [Node3], [], [Node2],Nodes -- [Node3])), - %% - D DO + ?match([], ?vrl(Tab, [], [Node3], [Node2],Nodes -- [Node3])), + %% - D ER ?match({aborted,Reason56} when element(1, Reason56) == not_active, mnesia:move_table_copy(Tab, Node3, Node1)), - ?match([], ?vrl(Tab, [Node3], [], [Node2],Nodes -- [Node3])), - %% DO D - + ?match([], ?vrl(Tab, [], [Node3], [Node2],Nodes -- [Node3])), + %% - D ER ?match([], mnesia_test_lib:start_mnesia([Node3])), - ?match([], ?vrl(Tab, [Node3], [], [Node2], Nodes)), - %% DO D - + ?match([], ?vrl(Tab, [], [Node3], [Node2], Nodes)), + %% - D ER + ?match([{Tab,43,sync_me}], mnesia:dirty_read({Tab,43})), %% %% Transformer @@ -990,7 +1018,7 @@ local_content(Config) when is_list(Config) -> ?match([], mnesia_test_lib:stop_mnesia([Node3])), %% Added for OTP-44306 - ?match(ok, rpc:call(Node3, mnesia, start, [])), + ?match(ok, rpc:call(Node3, mnesia, start, [[{schema, ?BACKEND}]])), ?match({ok, _}, mnesia:change_config(extra_db_nodes, [Node3])), mnesia_test_lib:sync_tables([Node3], [Tab1]), @@ -1427,7 +1455,7 @@ unsupp_user_props(Config) when is_list(Config) -> table_info, [silly1, user_properties])), ?match([{prop,propval2}], rpc:call(Node1, mnesia, table_info, [silly2, user_properties])), - ?match([{prop,propval3}], rpc:call(Node1, mnesia, + ?match([_,{prop,propval3}], rpc:call(Node1, mnesia, table_info, [schema, user_properties])), F2 = fun() -> @@ -2262,6 +2290,10 @@ record_name_dirty_access_disc_only(suite) -> record_name_dirty_access_disc_only(Config) when is_list(Config) -> record_name_dirty_access(disc_only_copies, Config). +record_name_dirty_access_xets(Config) when is_list(Config) -> + record_name_dirty_access(ext_ets, Config). + + record_name_dirty_access(Storage, Config) -> [Node1, _Node2] = Nodes = ?acquire_nodes(2, Config), diff --git a/lib/mnesia/test/mnesia_recovery_test.erl b/lib/mnesia/test/mnesia_recovery_test.erl index cd581787fb..2388b595d0 100644 --- a/lib/mnesia/test/mnesia_recovery_test.erl +++ b/lib/mnesia/test/mnesia_recovery_test.erl @@ -576,7 +576,7 @@ delete_during_start(Config) when is_list(Config) -> mnesia_test_lib:kill_mnesia([N2,N3]), %% timer:sleep(500), ?match({[ok,ok],[]}, rpc:multicall([N2,N3], mnesia,start, - [[{extra_db_nodes,[N1]}]])), + [[{extra_db_nodes,[N1]}, {schema, ?BACKEND}]])), [Tab1,Tab2,Tab3|_] = Tabs, ?match({atomic, ok}, mnesia:delete_table(Tab1)), ?match({atomic, ok}, mnesia:delete_table(Tab2)), @@ -1542,7 +1542,7 @@ disc_less(Config) when is_list(Config) -> ?match(ok, rpc:call(Node2, mnesia, start, [])), timer:sleep(500), - ?match(ok, rpc:call(Node3, mnesia, start, [[{extra_db_nodes, [Node1, Node2]}]])), + ?match(ok, rpc:call(Node3, mnesia, start, [[{extra_db_nodes, [Node1, Node2]}, {schema, ?BACKEND}]])), ?match(ok, rpc:call(Node3, mnesia, wait_for_tables, [[Tab1, Tab2, Tab3], 20000])), ?match(ok, rpc:call(Node1, mnesia, wait_for_tables, [[Tab1, Tab2, Tab3], 20000])), diff --git a/lib/mnesia/test/mnesia_test_lib.erl b/lib/mnesia/test/mnesia_test_lib.erl index 616207ed1a..6e84a27ec9 100644 --- a/lib/mnesia/test/mnesia_test_lib.erl +++ b/lib/mnesia/test/mnesia_test_lib.erl @@ -142,6 +142,9 @@ %% included for test server compatibility %% assume that all test cases only takes Config as sole argument init_per_testcase(_Func, Config) -> + Env = application:get_all_env(mnesia), + [application:unset_env(mnesia, Key, [{timeout, infinity}]) || + {Key, _} <- Env, Key /= included_applications], global:register_name(mnesia_global_logger, group_leader()), Config. @@ -668,11 +671,13 @@ do_prepare([delete_schema | Actions], Selected, All, Config, File, Line) -> end, do_prepare(Actions, Selected, All, Config, File, Line); do_prepare([create_schema | Actions], Selected, All, Config, File, Line) -> + Ext = ?BACKEND, case diskless(Config) of true -> + rpc:multicall(Selected, application, set_env, [mnesia, schema, Ext]), skip; _Else -> - case mnesia:create_schema(Selected) of + case mnesia:create_schema(Selected, Ext) of ok -> ignore; BadNodes -> @@ -975,7 +980,6 @@ reload_appls([Appl | Appls], Selected) -> {Ok2temp, Empty} = rpc:multicall(Selected, application, unload, [Appl]), Conv = fun({error,{not_loaded,mnesia}}) -> ok; (Else) -> Else end, Ok2 = {lists:map(Conv, Ok2temp), Empty}, - Ok3 = rpc:multicall(Selected, application, load, [Appl]), if Ok /= Ok2 -> @@ -1040,7 +1044,8 @@ verify_replica_location(Tab, DiscOnly0, Ram0, Disc0, AliveNodes0) -> S1 = ?match(AliveNodes, lists:sort(mnesia:system_info(running_db_nodes))), S2 = ?match(DiscOnly, lists:sort(mnesia:table_info(Tab, disc_only_copies))), - S3 = ?match(Ram, lists:sort(mnesia:table_info(Tab, ram_copies))), + S3 = ?match(Ram, lists:sort(mnesia:table_info(Tab, ram_copies) ++ + mnesia:table_info(Tab, ext_ets))), S4 = ?match(Disc, lists:sort(mnesia:table_info(Tab, disc_copies))), S5 = ?match(Write, lists:sort(mnesia:table_info(Tab, where_to_write))), S6 = case lists:member(This, Read) of diff --git a/lib/mnesia/test/mnesia_test_lib.hrl b/lib/mnesia/test/mnesia_test_lib.hrl index 714ec7a491..ba7eb10ea2 100644 --- a/lib/mnesia/test/mnesia_test_lib.hrl +++ b/lib/mnesia/test/mnesia_test_lib.hrl @@ -150,3 +150,5 @@ -define(verify_mnesia(Ups, Downs), mnesia_test_lib:verify_mnesia(Ups, Downs, ?FILE, ?LINE)). + +-define(BACKEND, [{backend_types, [{ext_ets, ext_test},{ext_dets, ext_test}]}]). diff --git a/lib/mnesia/test/mnesia_trans_access_test.erl b/lib/mnesia/test/mnesia_trans_access_test.erl index 38fb8d6a77..aa50ee4cb1 100644 --- a/lib/mnesia/test/mnesia_trans_access_test.erl +++ b/lib/mnesia/test/mnesia_trans_access_test.erl @@ -931,20 +931,20 @@ index_update_bag(Config)when is_list(Config) -> [IPos] = mnesia_lib:val({Tab,index}), ITab = mnesia_lib:val({index_test,{index, IPos}}), io:format("~n Index ~p @ ~p => ~p ~n~n",[IPos,ITab, ets:tab2list(ITab)]), - ?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), + %?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:write(Rec5) end)), {atomic, R60} = mnesia:transaction(fun() -> mnesia:index_read(Tab, 2, ValPos) end), ?match([Rec1,Rec5,Rec2], lists:sort(R60)), - ?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), + %?match([{2,1},{2,2},{12,1}], lists:keysort(1,ets:tab2list(ITab))), ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:delete_object(Rec3) end)), {atomic, R61} = mnesia:transaction(fun() -> mnesia:index_read(Tab, 2, ValPos) end), ?match([Rec1,Rec5,Rec2], lists:sort(R61)), {atomic, R62} = mnesia:transaction(fun() -> mnesia:index_read(Tab,12, ValPos) end), ?match([], lists:sort(R62)), - ?match([{2,1},{2,2}], lists:keysort(1,ets:tab2list(ITab))), + %% ?match([{2,1},{2,2}], lists:keysort(1,ets:tab2list(ITab))), %% reset for rest of testcase ?match({atomic, ok}, mnesia:transaction(fun() -> mnesia:write(Rec3) end)), @@ -1142,8 +1142,9 @@ create_live_table_index(Config, Storage) -> ?match([{atomic,ok}|_], [Create(N) || N <- lists:seq(1,50)]), ?match([], mnesia_test_lib:stop_mnesia([N2,N3])), - ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes,[N1]}]])), - ?match(ok, rpc:call(N3, mnesia, start, [[{extra_db_nodes,[N1]}]])), + Ext = [{schema, ?BACKEND}], + ?match(ok, rpc:call(N2, mnesia, start, [[{extra_db_nodes,[N1]}|Ext]])), + ?match(ok, rpc:call(N3, mnesia, start, [[{extra_db_nodes,[N1]}|Ext]])), ?match({atomic, ok}, mnesia:add_table_index(Tab, ValPos)), diff --git a/lib/odbc/configure.in b/lib/odbc/configure.in index 2871d91a1c..cb2b23d534 100644 --- a/lib/odbc/configure.in +++ b/lib/odbc/configure.in @@ -73,6 +73,16 @@ AC_CHECK_TOOL(LD, ld, '$(CC)') AC_SUBST(LD) +_search_path=/bin:/usr/bin:/usr/local/bin:$PATH + +AC_PATH_PROG(RM, rm, false, $_search_path) +if test "$ac_cv_path_RM" = false; then + AC_MSG_ERROR([No 'rm' command found]) +fi + +_search_path= + + # Sockets #-------------------------------------------------------------------- # Check for the existence of the -lsocket and -lnsl libraries. @@ -128,7 +138,7 @@ dnl Checks for library functions. AC_CHECK_FUNCS([memset socket]) # ODBC -/bin/rm -f "$ERL_TOP/lib/odbc/SKIP" +$RM -f "$ERL_TOP/lib/odbc/SKIP" LM_CHECK_THR_LIB AC_SUBST(THR_DEFS) diff --git a/lib/os_mon/c_src/cpu_sup.c b/lib/os_mon/c_src/cpu_sup.c index 61bf1b0352..353c7e674e 100644 --- a/lib/os_mon/c_src/cpu_sup.c +++ b/lib/os_mon/c_src/cpu_sup.c @@ -47,6 +47,10 @@ #include <kstat.h> #endif +#if (defined(__APPLE__) && defined(__MACH__)) +#include <mach/mach.h> +#endif + #include <errno.h> #if defined(__sun__) || defined(__linux__) @@ -73,6 +77,10 @@ typedef struct { #endif +#if (defined(__APPLE__) && defined(__MACH__)) +#define CU_OSX_VALUES (5) +#endif + #if defined(__FreeBSD__) #include <sys/resource.h> #include <sys/sysctl.h> @@ -164,7 +172,7 @@ static int processors_online() { } #endif -#if defined(__FreeBSD__) +#if (defined(__APPLE__) && defined(__MACH__)) || defined(__FreeBSD__) void getsysctl(const char *, void *, size_t); #endif @@ -173,7 +181,7 @@ int main(int argc, char** argv) { int rc; int sz; unsigned int *rv; -#if defined(__linux__) || defined(__FreeBSD__) +#if defined(__linux__) || (defined(__APPLE__) && defined(__MACH__)) ||defined(__FreeBSD__) unsigned int no_of_cpus = 0; #endif @@ -190,6 +198,13 @@ int main(int argc, char** argv) { } #endif +#if (defined(__APPLE__) && defined(__MACH__)) + getsysctl("hw.ncpu", &no_of_cpus, sizeof(int)); + if ( (rv = (unsigned int*)malloc(sizeof(unsigned int)*(2 + 2*no_of_cpus*CU_OSX_VALUES))) == NULL) { + error("cpu_sup: malloc error"); + } +#endif + #if defined(__FreeBSD__) getsysctl("hw.ncpu", &no_of_cpus, sizeof(int)); if ( (rv = (unsigned int*)malloc(sizeof(unsigned int)*(2 + 2*no_of_cpus*CU_BSD_VALUES))) == NULL) { @@ -222,7 +237,7 @@ int main(int argc, char** argv) { case AVG5: bsd_loadavg(1); break; case AVG15: bsd_loadavg(2); break; #endif -#if defined(__sun__) || defined(__linux__) || defined(__FreeBSD__) +#if defined(__sun__) || defined(__linux__) || (defined(__APPLE__) && defined(__MACH__)) || defined(__FreeBSD__) case UTIL: util_measure(&rv,&sz); sendv(rv, sz); break; #endif case QUIT: free((void*)rv); return 0; @@ -538,20 +553,60 @@ static void util_measure(unsigned int **result_vec, int *result_sz) { #endif /* ---------------------------- * - * FreeBSD stat functions * + * OSX util functions * * ---------------------------- */ -#if defined(__FreeBSD__) +#if (defined(__APPLE__) && defined(__MACH__)) -#define EXIT_WITH(msg) (rich_error(msg, __FILE__, __LINE__)) -#define RICH_BUFLEN (213) /* left in error(char*) */ +static void util_measure(unsigned int **result_vec, int *result_sz) { + natural_t no_of_cpus; + processor_info_array_t info_array; + mach_msg_type_number_t info_count; + mach_port_t host_port; + kern_return_t error; + processor_cpu_load_info_data_t *cpu_load_info = NULL; + unsigned int *rv = NULL; + int i; -void rich_error(const char *reason, const char *file, const int line) { - char buf[RICH_BUFLEN]; - snprintf(buf, RICH_BUFLEN, "%s (%s:%i)", reason, file, line); - error(buf); + host_port = mach_host_self(); + error = host_processor_info(host_port, PROCESSOR_CPU_LOAD_INFO, + &no_of_cpus, &info_array, &info_count); + if (error != KERN_SUCCESS) { + *result_sz = 0; + return; + } + mach_port_deallocate(mach_task_self(), host_port); + cpu_load_info = (processor_cpu_load_info_data_t *) info_array; + + rv = *result_vec; + rv[0] = no_of_cpus; + rv[1] = CU_OSX_VALUES; + ++rv; /* first value is number of cpus */ + ++rv; /* second value is number of entries */ + + for (i = 0; i < no_of_cpus; ++i) { + rv[0] = CU_CPU_ID; rv[1] = i; + rv[2] = CU_USER; rv[3] = cpu_load_info[i].cpu_ticks[CPU_STATE_USER]; + rv[4] = CU_NICE_USER; rv[5] = cpu_load_info[i].cpu_ticks[CPU_STATE_NICE]; + rv[6] = CU_KERNEL; rv[7] = cpu_load_info[i].cpu_ticks[CPU_STATE_SYSTEM]; + rv[8] = CU_IDLE; rv[9] = cpu_load_info[i].cpu_ticks[CPU_STATE_IDLE]; + rv += CU_OSX_VALUES*2; + } + + *result_sz = 2 + 2*CU_OSX_VALUES * no_of_cpus; + + error = vm_deallocate(mach_task_self(), (vm_address_t)info_array, + info_count * sizeof(int)); + if (error != KERN_SUCCESS) + *result_sz = 0; } -#undef RICH_BUFLEN +#endif + +/* ---------------------------- * + * FreeBSD stat functions * + * ---------------------------- */ + +#if defined(__FreeBSD__) static void util_measure(unsigned int **result_vec, int *result_sz) { int no_of_cpus; @@ -588,6 +643,23 @@ static void util_measure(unsigned int **result_vec, int *result_sz) { *result_sz = 2 + 2*CU_BSD_VALUES * no_of_cpus; } +#endif + +/* ---------------------------- * + * Utils for OSX and FreeBSD * + * ---------------------------- */ + +#if (defined(__APPLE__) && defined(__MACH__)) || defined(__FreeBSD__) + +#define EXIT_WITH(msg) (rich_error(msg, __FILE__, __LINE__)) +#define RICH_BUFLEN (213) /* left in error(char*) */ + +void rich_error(const char *reason, const char *file, const int line) { + char buf[RICH_BUFLEN]; + snprintf(buf, RICH_BUFLEN, "%s (%s:%i)", reason, file, line); + error(buf); +} +#undef RICH_BUFLEN void getsysctl(const char *name, void *ptr, size_t len) { diff --git a/lib/os_mon/src/cpu_sup.erl b/lib/os_mon/src/cpu_sup.erl index 5664615230..e758b63d19 100644 --- a/lib/os_mon/src/cpu_sup.erl +++ b/lib/os_mon/src/cpu_sup.erl @@ -162,7 +162,8 @@ handle_call({?util, D, PC}, {Client, _Tag}, #state{os_type = {unix, Flavor}} = State) when Flavor == sunos; Flavor == linux; - Flavor == freebsd -> + Flavor == freebsd; + Flavor == darwin -> case measurement_server_call(State#state.server, {?util, D, PC, Client}) of {error, Reason} -> { reply, @@ -531,11 +532,11 @@ measurement_server_loop(State) -> measurement_server_loop(State) end; {Pid, Request} -> - try get_uint32_measurement(Request, State) of - Result -> Pid ! {data, Result} - catch - Error -> Pid ! {error, Error} - end, + _ = try get_uint32_measurement(Request, State) of + Result -> Pid ! {data, Result} + catch + Error -> Pid ! {error, Error} + end, measurement_server_loop(State); {'EXIT', OldPid, _n} when State#internal.port == OldPid -> {ok, NewPid} = port_server_start_link(), diff --git a/lib/os_mon/src/disksup.erl b/lib/os_mon/src/disksup.erl index 2b5447cfcb..492e4814da 100644 --- a/lib/os_mon/src/disksup.erl +++ b/lib/os_mon/src/disksup.erl @@ -153,7 +153,7 @@ handle_cast(_Msg, State) -> handle_info(timeout, State) -> NewDiskData = check_disk_space(State#state.os, State#state.port, State#state.threshold), - timer:send_after(State#state.timeout, timeout), + {ok, _Tref} = timer:send_after(State#state.timeout, timeout), {noreply, State#state{diskdata = NewDiskData}}; handle_info({'EXIT', _Port, Reason}, State) -> {stop, {port_died, Reason}, State#state{port=not_used}}; diff --git a/lib/os_mon/src/memsup.erl b/lib/os_mon/src/memsup.erl index 833e1ce6d1..d9d3083540 100644 --- a/lib/os_mon/src/memsup.erl +++ b/lib/os_mon/src/memsup.erl @@ -408,18 +408,18 @@ handle_info({collected_sys, {Alloc,Total}}, State) -> %% Last, if this was a periodic check, start a timer for the next %% one. New timeout = interval-time spent collecting, - case lists:member(reg, State#state.pending) of - true -> - Time = case State2#state.timeout - TimeSpent of - MS when MS<0 -> - 0; - MS -> - MS - end, - erlang:send_after(Time, self(), time_to_collect); - false -> - ignore - end, + _ = case lists:member(reg, State#state.pending) of + true -> + Time = case State2#state.timeout - TimeSpent of + MS when MS<0 -> + 0; + MS -> + MS + end, + erlang:send_after(Time, self(), time_to_collect); + false -> + ignore + end, {noreply, State2#state{wd_timer=undefined, pending=[]}}; handle_info({'EXIT', Pid, normal}, State) when is_pid(Pid) -> %% Temporary pid terminating when job is done @@ -448,17 +448,17 @@ handle_info(reg_collection_timeout, State) -> %% If it is a periodic check which has timed out, start a timer for %% the next one %% New timeout = interval-helper timeout - case lists:member(reg, State#state.pending) of - true -> - Time = - case State#state.timeout-State#state.helper_timeout of - MS when MS<0 -> 0; - MS -> MS - end, - erlang:send_after(Time, self(), time_to_collect); - false -> - ignore - end, + _ = case lists:member(reg, State#state.pending) of + true -> + Time = + case State#state.timeout-State#state.helper_timeout of + MS when MS<0 -> 0; + MS -> MS + end, + erlang:send_after(Time, self(), time_to_collect); + false -> + ignore + end, {noreply, State#state{wd_timer=undefined, pending=[]}}; handle_info({'EXIT', Pid, cancel}, State) when is_pid(Pid) -> %% Temporary pid terminating as ordered @@ -469,7 +469,7 @@ handle_info({collected_ext_sys, SysMemUsage}, State) -> %% Cancel watchdog timer (and as a security mearure, %% also flush any ext_collection_timeout message) - erlang:cancel_timer(State#state.ext_wd_timer), + ok = erlang:cancel_timer(State#state.ext_wd_timer, [{async,true}]), flush(ext_collection_timeout), %% Send the reply to all waiting clients, preserving time order @@ -535,7 +535,7 @@ code_change(Vsn, PrevState, "1.8") -> undefined -> ignore; TimerRef1 -> - erlang:cancel_timer(TimerRef1), + ok = erlang:cancel_timer(TimerRef1, [{async,true}]), SysOnly = PrevState#state.sys_only, MemUsage = dummy_reply(get_memory_data, SysOnly), SysMemUsage1 = dummy_reply(get_system_memory_data), @@ -545,7 +545,7 @@ code_change(Vsn, PrevState, "1.8") -> undefined -> ignore; TimerRef2 -> - erlang:cancel_timer(TimerRef2), + ok = erlang:cancel_timer(TimerRef2, [{async,true}]), SysMemUsage2 = dummy_reply(get_system_memory_data), reply(PrevState#state.pending, undef, SysMemUsage2) end, @@ -589,7 +589,7 @@ code_change(Vsn, PrevState, "1.8") -> undefined -> ignore; TimerRef1 -> - erlang:cancel_timer(TimerRef1), + ok = erlang:cancel_timer(TimerRef1, [{async,true}]), MemUsage = dummy_reply(get_memory_data, SysOnly), Pending2 = lists:map(fun(From) -> {reg,From} end, Pending), @@ -599,7 +599,7 @@ code_change(Vsn, PrevState, "1.8") -> undefined -> ignore; TimerRef2 -> - erlang:cancel_timer(TimerRef2), + ok = erlang:cancel_timer(TimerRef2, [{async,true}]), SysMemUsage = dummy_reply(get_system_memory_data), ExtPending2 = lists:map(fun(From) -> {ext,From} end, ExtPending), diff --git a/lib/os_mon/test/cpu_sup_SUITE.erl b/lib/os_mon/test/cpu_sup_SUITE.erl index 11648d3547..41115ee6e2 100644 --- a/lib/os_mon/test/cpu_sup_SUITE.erl +++ b/lib/os_mon/test/cpu_sup_SUITE.erl @@ -63,6 +63,8 @@ all() -> [load_api, util_api, util_values, port, unavailable]; {unix, freebsd} -> [load_api, util_api, util_values, port, unavailable]; + {unix, darwin} -> + [load_api, util_api, util_values, port, unavailable]; {unix, _OSname} -> [load_api]; _OS -> [unavailable] end. diff --git a/lib/percept/doc/src/egd_ug.xmlsrc b/lib/percept/doc/src/egd_ug.xmlsrc index 563780aa66..85d41ada79 100644 --- a/lib/percept/doc/src/egd_ug.xmlsrc +++ b/lib/percept/doc/src/egd_ug.xmlsrc @@ -51,24 +51,27 @@ </section> <section> <title>File example</title> - <p>Drawing examples:</p> - <codeinclude file="img.erl" tag="" type="none"></codeinclude> - <image file="test1.gif"> - First save. - <icaption>test1.png</icaption> - </image> - <image file="test2.gif"> - Second save. - <icaption>test2.png</icaption> - </image> - <image file="test3.gif"> - Third save. - <icaption>test3.png</icaption> - </image> - <image file="test4.gif"> - Fourth save. - <icaption>test4.png</icaption> - </image> + <p>Drawing examples:</p> + <codeinclude file="img.erl" tag="" type="none"></codeinclude> + <p> First save. </p> + <image file="test1.gif"> + <icaption>test1.png</icaption> + </image> + + <p> Second save. </p> + <image file="test2.gif"> + <icaption>test2.png</icaption> + </image> + + <p> Third save. </p> + <image file="test3.gif"> + <icaption>test3.png</icaption> + </image> + + <p> Fourth save. </p> + <image file="test4.gif"> + <icaption>test4.png</icaption> + </image> </section> <section> <title>ESI example</title> diff --git a/lib/reltool/doc/src/notes.xml b/lib/reltool/doc/src/notes.xml index 082079aa74..0a83954865 100644 --- a/lib/reltool/doc/src/notes.xml +++ b/lib/reltool/doc/src/notes.xml @@ -194,7 +194,6 @@ <section><title>Fixed Bugs and Malfunctions</title> <list> <item> - <p> <list> <item> If <c>incl_cond</c> was set to <c>derived</c> on module level, then reltool_server would crash with a <c>case_clause</c>. This has been corrected. @@ -225,7 +224,7 @@ implemented in reltool. </item> <item> Instead of only looking at the directory name, reltool now first looks for a <c>.app</c> file in order to figure out the name of - an application. </item> </list></p> + an application. </item> </list> <p> Own Id: OTP-10012 Aux Id: kunagi-171 [82] </p> </item> @@ -272,7 +271,6 @@ <section><title>Fixed Bugs and Malfunctions</title> <list> <item> - <p> Miscellaneous corrections: <list> <item> Start of reltool GUI would sometimes crash with a badmatch in reltool_sys_win:do_init. This has been corrected. </item> @@ -311,7 +309,7 @@ and when generating target system. </item> <item> Title of dependecies column in app and mod window is changed from "Modules used by others" to "Modules using this". - </item> </list></p> + </item> </list> <p> Own Id: OTP-9792</p> </item> @@ -338,7 +336,7 @@ </item> <item> <p> - Some bug fixes related to the handling of escripts: + Some bug fixes related to the handling of escripts:</p> <list> <item> Reltool could not handle escripts with inlined applications. This has been corrected. Inlined applications will be visible in the GUI, but not possible @@ -357,7 +355,7 @@ another escript, for which the name sorts before the existing one, would cause reltool to fail saying "Application name clash". This has been corrected. - </item> </list></p> + </item> </list> <p> Own Id: OTP-9968</p> </item> diff --git a/lib/reltool/doc/src/reltool.xml b/lib/reltool/doc/src/reltool.xml index f4effc3f2e..38448e7961 100644 --- a/lib/reltool/doc/src/reltool.xml +++ b/lib/reltool/doc/src/reltool.xml @@ -248,7 +248,7 @@ <p>When starting this release, three things must be specified:</p> <taglist> - <tag><b>Which <c>releases</c> directory to use</b></tag> + <tag><em>Which <c>releases</c> directory to use</em></tag> <item>Tell the release handler to use the <c>releases</c> directory in our target structure instead of <c>$OTP_ROOT/releases</c>. This is done by setting the SASL @@ -257,7 +257,7 @@ <target-dir>/releases</c>) or in <c>sys.config</c>.</item> - <tag><b>Which boot file to use</b></tag> + <tag><em>Which boot file to use</em></tag> <item>The default boot file is <c>$OTP_ROOT/bin/start</c>, but in this case we need to specify a boot file from our target structure, typically @@ -265,7 +265,7 @@ is done with the <c>-boot</c> command line option to <c>erl</c></item> - <tag><b>The location of our applications</b></tag> + <tag><em>The location of our applications</em></tag> <item>The generated .script (and .boot) file uses the environment variable <c>$RELTOOL_EXT_LIB</c> as prefix for the paths to all applications. The <c>-boot_var</c> option @@ -275,8 +275,8 @@ </taglist> <p>Example:</p> - <p><code>erl -sasl releases_dir \"mytarget/releases\" -boot mytarget/releases/1.0/myrel\ - -boot_var RELTOOL_EXT_LIB mytarget/lib</code></p> + <code>erl -sasl releases_dir \"mytarget/releases\" -boot mytarget/releases/1.0/myrel\ + -boot_var RELTOOL_EXT_LIB mytarget/lib</code> </item> <tag><c>incl_sys_filters</c></tag> diff --git a/lib/runtime_tools/c_src/dyntrace.c b/lib/runtime_tools/c_src/dyntrace.c index 0178d95efb..e7a4a73373 100644 --- a/lib/runtime_tools/c_src/dyntrace.c +++ b/lib/runtime_tools/c_src/dyntrace.c @@ -433,7 +433,7 @@ static ERL_NIF_TERM trace_call(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv char class[LTTNG_BUFFER_SZ]; enif_get_tuple(env, argv[4], &arity, &tuple); - erts_snprintf(class, LTTNG_BUFFER_SZ, "%T", tuple[0]); + enif_snprintf(class, LTTNG_BUFFER_SZ, "%T", tuple[0]); if (enif_get_tuple(env, argv[3], &arity, &tuple)) { enif_get_uint(env, tuple[2], &len); @@ -465,7 +465,7 @@ static ERL_NIF_TERM trace_send(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv char msg[LTTNG_BUFFER_SZ]; lttng_pid_to_str(argv[4], to); - erts_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG3(message_send, pid, to, msg); } else if (argv[0] == atom_send_to_non_existing_process) { @@ -473,7 +473,7 @@ static ERL_NIF_TERM trace_send(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv char msg[LTTNG_BUFFER_SZ]; lttng_pid_to_str(argv[4], to); - erts_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); /* mark it as non existing ? */ LTTNG3(message_send, pid, to, msg); @@ -496,7 +496,7 @@ static ERL_NIF_TERM trace_receive(ErlNifEnv* env, int argc, const ERL_NIF_TERM a char msg[LTTNG_BUFFER_SZ]; lttng_pid_to_str(argv[2], pid); - erts_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(msg, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG2(message_receive, pid, msg); } @@ -560,11 +560,11 @@ static ERL_NIF_TERM trace_procs(ErlNifEnv* env, int argc, const ERL_NIF_TERM arg /* register */ } else if (argv[0] == atom_register) { char name[LTTNG_BUFFER_SZ]; - erts_snprintf(name, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(name, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG3(process_register, pid, name, "register"); } else if (argv[0] == atom_unregister) { char name[LTTNG_BUFFER_SZ]; - erts_snprintf(name, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(name, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG3(process_register, pid, name, "unregister"); /* link */ } else if (argv[0] == atom_link) { @@ -582,7 +582,7 @@ static ERL_NIF_TERM trace_procs(ErlNifEnv* env, int argc, const ERL_NIF_TERM arg /* exit */ } else if (argv[0] == atom_exit) { char reason[LTTNG_BUFFER_SZ]; - erts_snprintf(reason, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(reason, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG2(process_exit, pid, reason); } return atom_ok; @@ -622,11 +622,11 @@ static ERL_NIF_TERM trace_ports(ErlNifEnv* env, int argc, const ERL_NIF_TERM arg lttng_decl_procbuf(pid); lttng_pid_to_str(argv[3], pid); - erts_snprintf(driver, LTTNG_BUFFER_SZ, "%T", argv[4]); + enif_snprintf(driver, LTTNG_BUFFER_SZ, "%T", argv[4]); LTTNG3(port_open, pid, driver, port); } else if (argv[0] == atom_closed) { char reason[LTTNG_BUFFER_SZ]; - erts_snprintf(reason, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(reason, LTTNG_BUFFER_SZ, "%T", argv[3]); LTTNG2(port_exit, port, reason); /* link */ @@ -705,7 +705,7 @@ static ERL_NIF_TERM trace_running_ports(ErlNifEnv* env, int argc, const ERL_NIF_ lttng_decl_mfabuf(where); lttng_portid_to_str(argv[2], pid); - erts_snprintf(where, LTTNG_BUFFER_SZ, "%T", argv[3]); + enif_snprintf(where, LTTNG_BUFFER_SZ, "%T", argv[3]); /* running ports */ if (argv[0] == atom_in) { diff --git a/lib/runtime_tools/c_src/dyntrace_lttng.h b/lib/runtime_tools/c_src/dyntrace_lttng.h index 3550a1cab5..2a3224e191 100644 --- a/lib/runtime_tools/c_src/dyntrace_lttng.h +++ b/lib/runtime_tools/c_src/dyntrace_lttng.h @@ -59,10 +59,10 @@ char Name[LTTNG_MFA_BUFFER_SZ] #define lttng_pid_to_str(pid, name) \ - erts_snprintf(name, LTTNG_PROC_BUFFER_SZ, "%T", (pid)) + enif_snprintf(name, LTTNG_PROC_BUFFER_SZ, "%T", (pid)) #define lttng_portid_to_str(pid, name) \ - erts_snprintf(name, LTTNG_PORT_BUFFER_SZ, "%T", (pid)) + enif_snprintf(name, LTTNG_PORT_BUFFER_SZ, "%T", (pid)) #define lttng_proc_to_str(p, name) \ lttng_pid_to_str(((p) ? (p)->common.id : ERTS_INVALID_PID), name) @@ -71,7 +71,7 @@ lttng_portid_to_str(((p) ? (p)->common.id : ERTS_INVALID_PORT), name) #define lttng_mfa_to_str(m,f,a, Name) \ - erts_snprintf(Name, LTTNG_MFA_BUFFER_SZ, "%T:%T/%lu", (Eterm)(m), (Eterm)(f), (Uint)(a)) + enif_snprintf(Name, LTTNG_MFA_BUFFER_SZ, "%T:%T/%lu", (Eterm)(m), (Eterm)(f), (Uint)(a)) /* Process scheduling */ diff --git a/lib/runtime_tools/doc/src/LTTng.xml b/lib/runtime_tools/doc/src/LTTng.xml index 4a87133c57..06152c66d6 100644 --- a/lib/runtime_tools/doc/src/LTTng.xml +++ b/lib/runtime_tools/doc/src/LTTng.xml @@ -76,7 +76,7 @@ $ make </code> <item><c>entry : string</c> :: Code Location. Ex. <c>"lists:sort/1"</c></item> </list> <p>Example:</p> - <p><code type="none">process_spawn: { cpu_id = 3 }, { pid = "<0.131.0>", parent = "<0.130.0>", entry = "erlang:apply/2" }</code></p> + <code type="none">process_spawn: { cpu_id = 3 }, { pid = "<0.131.0>", parent = "<0.130.0>", entry = "erlang:apply/2" }</code> <p><em>process_link</em></p> <list type="bulleted"> @@ -85,7 +85,7 @@ $ make </code> <item><c>type : string</c> :: <c>"link" | "unlink"</c></item> </list> <p>Example:</p> - <p><code type="none">process_link: { cpu_id = 3 }, { from = "<0.130.0>", to = "<0.131.0>", type = "link" }</code></p> + <code type="none">process_link: { cpu_id = 3 }, { from = "<0.130.0>", to = "<0.131.0>", type = "link" }</code> <p><em>process_exit</em></p> @@ -94,7 +94,7 @@ $ make </code> <item><c>reason : string</c> :: Exit reason. Ex. <c>"normal"</c></item> </list> <p>Example:</p> - <p><code type="none">process_exit: { cpu_id = 3 }, { pid = "<0.130.0>", reason = "normal" }</code></p> + <code type="none">process_exit: { cpu_id = 3 }, { pid = "<0.130.0>", reason = "normal" }</code> <p><em>process_register</em></p> <list type="bulleted"> @@ -103,7 +103,7 @@ $ make </code> <item><c>type : string</c> :: <c>"register" | "unregister"</c></item> </list> <p>Example:</p> - <p><code type="none">process_register: { cpu_id = 0 }, { pid = "<0.128.0>", name = "dyntrace_lttng_SUITE" type = "register" }</code></p> + <code type="none">process_register: { cpu_id = 0 }, { pid = "<0.128.0>", name = "dyntrace_lttng_SUITE" type = "register" }</code> <p><em>process_scheduled</em></p> <list type="bulleted"> @@ -113,7 +113,7 @@ $ make </code> </list> <p>Example:</p> - <p><code type="none">process_scheduled: { cpu_id = 0 }, { pid = "<0.136.0>", entry = "erlang:apply/2", type = "in" }</code></p> + <code type="none">process_scheduled: { cpu_id = 0 }, { pid = "<0.136.0>", entry = "erlang:apply/2", type = "in" }</code> <p><em>port_open</em></p> @@ -124,7 +124,7 @@ $ make </code> </list> <p>Example:</p> - <p><code type="none">port_open: { cpu_id = 5 }, { pid = "<0.131.0>", driver = "'/bin/sh -s unix:cmd'", port = "#Port<0.1887>" }</code></p> + <code type="none">port_open: { cpu_id = 5 }, { pid = "<0.131.0>", driver = "'/bin/sh -s unix:cmd'", port = "#Port<0.1887>" }</code> <p><em>port_exit</em></p> <list type="bulleted"> @@ -132,7 +132,7 @@ $ make </code> <item><c>reason : string</c> :: Exit reason. Ex. <c>"normal"</c></item> </list> <p>Example:</p> - <p><code type="none">port_exit: { cpu_id = 5 }, { port = "#Port<0.1887>", reason = "normal" }</code></p> + <code type="none">port_exit: { cpu_id = 5 }, { port = "#Port<0.1887>", reason = "normal" }</code> <p><em>port_link</em></p> <list type="bulleted"> @@ -141,7 +141,7 @@ $ make </code> <item><c>type : string</c> :: <c>"link" | "unlink"</c></item> </list> <p>Example:</p> - <p><code type="none">port_link: { cpu_id = 5 }, { from = "#Port<0.1887>", to = "<0.131.0>", type = "unlink" }</code></p> + <code type="none">port_link: { cpu_id = 5 }, { from = "#Port<0.1887>", to = "<0.131.0>", type = "unlink" }</code> <p><em>port_scheduled</em></p> <list type="bulleted"> @@ -151,7 +151,7 @@ $ make </code> </list> <p>Example:</p> - <p><code type="none">port_scheduled: { cpu_id = 5 }, { pid = "#Port<0.1905>", entry = "close", type = "out" }</code></p> + <code type="none">port_scheduled: { cpu_id = 5 }, { pid = "#Port<0.1905>", entry = "close", type = "out" }</code> <p><em>function_call</em></p> <list type="bulleted"> @@ -160,7 +160,7 @@ $ make </code> <item><c>depth : integer</c> :: Stack depth. Ex. <c>0</c></item> </list> <p>Example:</p> - <p><code type="none">function_call: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:'-t_call/1-fun-1-'/0", depth = 0 }</code></p> + <code type="none">function_call: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:'-t_call/1-fun-1-'/0", depth = 0 }</code> <p><em>function_return</em></p> <list type="bulleted"> @@ -169,7 +169,7 @@ $ make </code> <item><c>depth : integer</c> :: Stack depth. Ex. <c>0</c></item> </list> <p>Example:</p> - <p><code type="none">function_return: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:waiter/0", depth = 0 }</code></p> + <code type="none">function_return: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:waiter/0", depth = 0 }</code> <p><em>function_exception</em></p> <list type="bulleted"> @@ -178,7 +178,7 @@ $ make </code> <item><c>class : string</c> :: Error reason. Ex. <c>"error"</c></item> </list> <p>Example:</p> - <p><code type="none">function_exception: { cpu_id = 5 }, { pid = "<0.144.0>", entry = "t:call_exc/1", class = "error" }</code></p> + <code type="none">function_exception: { cpu_id = 5 }, { pid = "<0.144.0>", entry = "t:call_exc/1", class = "error" }</code> <p><em>message_send</em></p> <list type="bulleted"> @@ -187,7 +187,7 @@ $ make </code> <item><c>message : string</c> :: Message sent. Ex. <c>"{<0.162.0>,ok}"</c></item> </list> <p>Example:</p> - <p><code type="none">message_send: { cpu_id = 3 }, { from = "#Port<0.1938>", to = "<0.160.0>", message = "{#Port<0.1938>,eof}" }</code></p> + <code type="none">message_send: { cpu_id = 3 }, { from = "#Port<0.1938>", to = "<0.160.0>", message = "{#Port<0.1938>,eof}" }</code> <p><em>message_receive</em></p> <list type="bulleted"> @@ -195,7 +195,7 @@ $ make </code> <item><c>message : string</c> :: Message received. Ex. <c>"{<0.162.0>,ok}"</c></item> </list> <p>Example:</p> - <p><code type="none">message_receive: { cpu_id = 7 }, { to = "<0.167.0>", message = "{<0.165.0>,ok}" }</code></p> + <code type="none">message_receive: { cpu_id = 7 }, { to = "<0.167.0>", message = "{<0.165.0>,ok}" }</code> <p><em>gc_minor_start</em></p> <list type="bulleted"> @@ -205,7 +205,7 @@ $ make </code> <item><c>old_heap : integer</c> :: Old heap word size. Ex. <c>233</c></item> </list> <p>Example:</p> - <p><code type="none">gc_minor_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 0, heap = 610, old_heap = 0 }</code></p> + <code type="none">gc_minor_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 0, heap = 610, old_heap = 0 }</code> <p><em>gc_minor_end</em></p> <list type="bulleted"> @@ -215,7 +215,7 @@ $ make </code> <item><c>old_heap : integer</c> :: Old heap word size. Ex. <c>233</c></item> </list> <p>Example:</p> - <p><code type="none">gc_minor_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 120, heap = 1598, old_heap = 1598 }</code></p> + <code type="none">gc_minor_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 120, heap = 1598, old_heap = 1598 }</code> <p><em>gc_major_start</em></p> <list type="bulleted"> @@ -225,7 +225,7 @@ $ make </code> <item><c>old_heap : integer</c> :: Old heap word size. Ex. <c>233</c></item> </list> <p>Example:</p> - <p><code type="none">gc_major_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 8, heap = 2586, old_heap = 1598 }</code></p> + <code type="none">gc_major_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 8, heap = 2586, old_heap = 1598 }</code> <p><em>gc_major_end</em></p> <list type="bulleted"> @@ -235,7 +235,7 @@ $ make </code> <item><c>old_heap : integer</c> :: Old heap word size. Ex. <c>233</c></item> </list> <p>Example:</p> - <p><code type="none">gc_major_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 240, heap = 4185, old_heap = 0 }</code></p> + <code type="none">gc_major_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 240, heap = 4185, old_heap = 0 }</code> </section> @@ -250,7 +250,7 @@ $ make </code> <item><c>runnable : integer</c> :: Runnable. Ex. <c>1</c></item> </list> <p>Example:</p> - <p><code type="none">scheduler_poll: { cpu_id = 4 }, { scheduler = 1, runnable = 1 }</code></p> + <code type="none">scheduler_poll: { cpu_id = 4 }, { scheduler = 1, runnable = 1 }</code> <p><em>driver_init</em></p> <list type="bulleted"> @@ -260,7 +260,7 @@ $ make </code> <item><c>flags : integer</c> :: Flags. Ex. <c>1</c></item> </list> <p>Example:</p> - <p><code type="none">driver_init: { cpu_id = 2 }, { driver = "caller_drv", major = 3, minor = 3, flags = 1 }</code></p> + <code type="none">driver_init: { cpu_id = 2 }, { driver = "caller_drv", major = 3, minor = 3, flags = 1 }</code> <p><em>driver_start</em></p> <list type="bulleted"> @@ -269,7 +269,7 @@ $ make </code> <item><c>port : string</c> :: Port ID. Ex. <c>"#Port<0.1031>"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_start: { cpu_id = 2 }, { pid = "<0.198.0>", driver = "caller_drv", port = "#Port<0.3676>" }</code></p> + <code type="none">driver_start: { cpu_id = 2 }, { pid = "<0.198.0>", driver = "caller_drv", port = "#Port<0.3676>" }</code> <p><em>driver_output</em></p> <list type="bulleted"> @@ -279,7 +279,7 @@ $ make </code> <item><c>bytes : integer</c> :: Size of data returned. Ex. <c>82</c></item> </list> <p>Example:</p> - <p><code type="none">driver_output: { cpu_id = 2 }, { pid = "<0.198.0>", port = "#Port<0.3677>", driver = "/bin/sh -s unix:cmd", bytes = 36 }</code></p> + <code type="none">driver_output: { cpu_id = 2 }, { pid = "<0.198.0>", port = "#Port<0.3677>", driver = "/bin/sh -s unix:cmd", bytes = 36 }</code> <p><em>driver_outputv</em></p> <list type="bulleted"> @@ -289,7 +289,7 @@ $ make </code> <item><c>bytes : integer</c> :: Size of data returned. Ex. <c>82</c></item> </list> <p>Example:</p> - <p><code type="none">driver_outputv: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet", bytes = 3 }</code></p> + <code type="none">driver_outputv: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet", bytes = 3 }</code> <p><em>driver_ready_input</em></p> <list type="bulleted"> @@ -298,7 +298,7 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_ready_input: { cpu_id = 5 }, { pid = "<0.189.0>", port = "#Port<0.3637>", driver = "inet_gethost 4 " }</code></p> + <code type="none">driver_ready_input: { cpu_id = 5 }, { pid = "<0.189.0>", port = "#Port<0.3637>", driver = "inet_gethost 4 " }</code> <p><em>driver_ready_output</em></p> <list type="bulleted"> @@ -307,7 +307,7 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_ready_output: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet" }</code></p> + <code type="none">driver_ready_output: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet" }</code> <p><em>driver_timeout</em></p> <list type="bulleted"> @@ -316,14 +316,14 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_timeout: { cpu_id = 5 }, { pid = "<0.196.0>", port = "#Port<0.3664>", driver = "tcp_inet" }</code></p> + <code type="none">driver_timeout: { cpu_id = 5 }, { pid = "<0.196.0>", port = "#Port<0.3664>", driver = "tcp_inet" }</code> <p><em>driver_stop_select</em></p> <list type="bulleted"> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_stop_select: { cpu_id = 5 }, { driver = "unknown" }</code></p> + <code type="none">driver_stop_select: { cpu_id = 5 }, { driver = "unknown" }</code> <p><em>driver_flush</em></p> <list type="bulleted"> @@ -332,7 +332,7 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_flush: { cpu_id = 7 }, { pid = "<0.204.0>", port = "#Port<0.3686>", driver = "tcp_inet" }</code></p> + <code type="none">driver_flush: { cpu_id = 7 }, { pid = "<0.204.0>", port = "#Port<0.3686>", driver = "tcp_inet" }</code> <p><em>driver_stop</em></p> <list type="bulleted"> @@ -341,7 +341,7 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_stop: { cpu_id = 5 }, { pid = "[]", port = "#Port<0.3673>", driver = "efile" }</code></p> + <code type="none">driver_stop: { cpu_id = 5 }, { pid = "[]", port = "#Port<0.3673>", driver = "efile" }</code> <p><em>driver_process_exit</em></p> <list type="bulleted"> @@ -357,7 +357,7 @@ $ make </code> <item><c>driver : string</c> :: Driver name. Ex. <c>"efile"</c></item> </list> <p>Example:</p> - <p><code type="none">driver_ready_async: { cpu_id = 3 }, { pid = "<0.181.0>", port = "#Port<0.3622>", driver = "efile" }</code></p> + <code type="none">driver_ready_async: { cpu_id = 3 }, { pid = "<0.181.0>", port = "#Port<0.3622>", driver = "efile" }</code> <p><em>driver_call</em></p> <list type="bulleted"> @@ -368,7 +368,7 @@ $ make </code> <item><c>bytes : integer</c> :: Size of data returned. Ex. <c>82</c></item> </list> <p>Example:</p> - <p><code type="none">driver_call: { cpu_id = 2 }, { pid = "<0.202.0>", port = "#Port<0.3676>", driver = "caller_drv", command = 0, bytes = 2 }</code></p> + <code type="none">driver_call: { cpu_id = 2 }, { pid = "<0.202.0>", port = "#Port<0.3676>", driver = "caller_drv", command = 0, bytes = 2 }</code> <p><em>driver_control</em></p> <list type="bulleted"> @@ -379,7 +379,7 @@ $ make </code> <item><c>bytes : integer</c> :: Size of data returned. Ex. <c>82</c></item> </list> <p>Example:</p> - <p><code type="none">driver_control: { cpu_id = 3 }, { pid = "<0.32767.8191>", port = "#Port<0.0>", driver = "forker", command = 83, bytes = 32 }</code></p> + <code type="none">driver_control: { cpu_id = 3 }, { pid = "<0.32767.8191>", port = "#Port<0.0>", driver = "forker", command = 83, bytes = 32 }</code> <p><em>aio_pool_get</em></p> <list type="bulleted"> @@ -387,7 +387,7 @@ $ make </code> <item><c>length : integer</c> :: Async queue length. Ex. <c>0</c></item> </list> <p>Example:</p> - <p><code type="none">aio_pool_get: { cpu_id = 4 }, { port = "#Port<0.3614>", length = 0 }</code></p> + <code type="none">aio_pool_get: { cpu_id = 4 }, { port = "#Port<0.3614>", length = 0 }</code> <p><em>aio_pool_put</em></p> <list type="bulleted"> @@ -396,7 +396,7 @@ $ make </code> </list> <p>Async queue length is not defined for <c>put</c> operations.</p> <p>Example:</p> - <p><code type="none">aio_pool_put: { cpu_id = 3 }, { port = "#Port<0.3614>", length = -1 }</code></p> + <code type="none">aio_pool_put: { cpu_id = 3 }, { port = "#Port<0.3614>", length = -1 }</code> <p><em>carrier_create</em></p> <list type="bulleted"> @@ -414,7 +414,7 @@ $ make </code> </list> <p>Example:</p> - <p><code type="none">carrier_create: { cpu_id = 2 }, { type = "ets_alloc", instance = 7, size = 2097152, mbc_carriers = 4, mbc_carriers_size = 3440640, mbc_blocks = 526, mbc_blocks_size = 1278576, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }</code></p> + <code type="none">carrier_create: { cpu_id = 2 }, { type = "ets_alloc", instance = 7, size = 2097152, mbc_carriers = 4, mbc_carriers_size = 3440640, mbc_blocks = 526, mbc_blocks_size = 1278576, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }</code> <p><em>carrier_destroy</em></p> <list type="bulleted"> @@ -432,7 +432,7 @@ $ make </code> </list> <p>Example:</p> - <p><code type="none">carrier_destroy: { cpu_id = 6 }, { type = "ets_alloc", instance = 7, size = 262144, mbc_carriers = 3, mbc_carriers_size = 3178496, mbc_blocks = 925, mbc_blocks_size = 2305336, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }</code></p> + <code type="none">carrier_destroy: { cpu_id = 6 }, { type = "ets_alloc", instance = 7, size = 262144, mbc_carriers = 3, mbc_carriers_size = 3178496, mbc_blocks = 925, mbc_blocks_size = 2305336, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }</code> <p><em>carrier_pool_put</em></p> <list type="bulleted"> @@ -441,7 +441,7 @@ $ make </code> <item><c>size : integer</c> :: Carrier size. Ex. <c>262144</c></item> </list> <p>Example:</p> - <p><code type="none">carrier_pool_put: { cpu_id = 3 }, { type = "ets_alloc", instance = 5, size = 1048576 }</code></p> + <code type="none">carrier_pool_put: { cpu_id = 3 }, { type = "ets_alloc", instance = 5, size = 1048576 }</code> <p><em>carrier_pool_get</em></p> <list type="bulleted"> @@ -450,11 +450,9 @@ $ make </code> <item><c>size : integer</c> :: Carrier size. Ex. <c>262144</c></item> </list> <p>Example:</p> - <p><code type="none">carrier_pool_get: { cpu_id = 7 }, { type = "ets_alloc", instance = 4, size = 3208 }</code></p> - + <code type="none">carrier_pool_get: { cpu_id = 7 }, { type = "ets_alloc", instance = 4, size = 3208 }</code> </section> - <section> <title>Examples</title> </section> diff --git a/lib/runtime_tools/doc/src/dbg.xml b/lib/runtime_tools/doc/src/dbg.xml index 0128e23a47..103b8b52e9 100644 --- a/lib/runtime_tools/doc/src/dbg.xml +++ b/lib/runtime_tools/doc/src/dbg.xml @@ -36,8 +36,8 @@ <modulesummary>The Text Based Trace Facility</modulesummary> <description> <p>This module implements a text based interface to the - <c><seealso marker="erts:erlang#trace-3">trace/3</seealso></c> and the - <c><seealso marker="erts:erlang#trace_pattern-2">trace_pattern/2</seealso></c> BIFs. It makes it + <seealso marker="erts:erlang#trace-3"><c>trace/3</c></seealso> and the + <seealso marker="erts:erlang#trace_pattern-2"><c>trace_pattern/2</c></seealso> BIFs. It makes it possible to trace functions, processes, ports and messages. </p> <p> @@ -185,7 +185,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <item>The corresponding process or port is traced. The process or port may be a remote process or port (on another Erlang node). The node must be in the list of traced nodes (see <seealso marker="#n-1"><c>n/1</c></seealso> - and <c><seealso marker="#tracer-3">tracer/3</seealso></c>).</item> + and <seealso marker="#tracer-3"><c>tracer/3</c></seealso>).</item> <tag><c>all</c></tag> <item>All processes and ports in the system as well as all processes and ports created hereafter are to be traced.</item> @@ -208,22 +208,23 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <tag><c>atom()</c></tag> <item>The process or port with the corresponding registered name is traced. The process or port may be a remote process (on another Erlang node). The node must be - added with the <c><seealso marker="#n-1">n/1</seealso></c> or - <c><seealso marker="#tracer-3">tracer/3</seealso></c> function.</item> + added with the <seealso marker="#n-1"><c>n/1</c></seealso> or + <seealso marker="#tracer-3"><c>tracer/3</c></seealso> function.</item> <tag><c>integer()</c></tag> <item>The process <c><![CDATA[<0.Item.0>]]></c> is traced.</item> <tag><c>{X, Y, Z}</c></tag> <item>The process <c><![CDATA[<X.Y.Z>]]></c> is traced. </item> - <tag><c>string()</c></tag> - <item>If the <c>Item</c> is a string <![CDATA["<X.Y.Z>"]]> - as returned from <c><seealso marker="erts:erlang#pid_to_list-1">pid_to_list/1</seealso></c>, the process - <c><![CDATA[<X.Y.Z>]]></c> is traced. </item> - </taglist> + <tag><c>string()</c></tag> + <item>If the <c>Item</c> is a string <![CDATA["<X.Y.Z>"]]> + as returned from <seealso marker="erts:erlang#pid_to_list-1"><c>pid_to_list/1</c></seealso>, + the process <c><![CDATA[<X.Y.Z>]]></c> is traced. + </item> + </taglist> <p>When enabling an <c>Item</c> that represents a group of processes, the <c>Item</c> is enabled on all nodes added with the - <c><seealso marker="#n-1">n/1</seealso></c> or - <c><seealso marker="#tracer-3">tracer/3</seealso></c> function.</p> + <seealso marker="#n-1"><c>n/1</c></seealso> or + <seealso marker="#tracer-3"><c>tracer/3</c></seealso> function.</p> <p><c>Flags</c> can be a single atom, or a list of flags. The available flags are: @@ -275,7 +276,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <item> <p>This is the same as <c>sol</c>, but only for the first call to - <c><seealso marker="erts:erlang#link-1">link/1</seealso></c> by the traced process.</p> + <seealso marker="erts:erlang#link-1"><c>link/1</c></seealso> by the traced process.</p> </item> <tag><c>all</c></tag> <item> @@ -288,7 +289,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </item> </taglist> <p>The list can also include any of the flags allowed in - <c><seealso marker="erts:erlang#trace-3">erlang:trace/3</seealso></c></p> + <seealso marker="erts:erlang#trace-3"><c>erlang:trace/3</c></seealso></p> <p>The function returns either an error tuple or a tuple <c>{ok, List}</c>. The <c>List</c> consists of specifications of how many processes and ports that matched (in the @@ -368,11 +369,11 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ please turn to the <em>User's guide</em> part of the online documentation for the runtime system (<em>erts</em>). The - chapter <em><seealso marker="erts:match_spec">Match Specifications in Erlang</seealso></em> + chapter <seealso marker="erts:match_spec"><em>Match Specifications in Erlang</em></seealso> explains the general match specification "language". The most common generic match specifications used can be found as <c>Built-inAlias</c>', see - <c><seealso marker="#ltp-0">ltp/0</seealso></c> below for details. + <seealso marker="#ltp-0"><c>ltp/0</c></seealso> below for details. </p> <p>The Module, Function and/or Arity parts of the tuple may be specified as the atom <c>'_'</c> which is a "wild-card" @@ -380,21 +381,21 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ Module is specified as <c>'_'</c>, the Function and Arity parts have to be specified as '_' too. The same holds for the Functions relation to the Arity.</p> - <p>All nodes added with <c><seealso marker="#n-1">n/1</seealso></c> or - <c><seealso marker="#tracer-3">tracer/3</seealso></c> will + <p>All nodes added with <seealso marker="#n-1"><c>n/1</c></seealso> or + <seealso marker="#tracer-3"><c>tracer/3</c></seealso> will be affected by this call, and if Module is not <c>'_'</c> the module will be loaded on all nodes.</p> <p>The function returns either an error tuple or a tuple <c>{ok, List}</c>. The <c>List</c> consists of specifications of how many functions that matched, in the same way as the processes and ports - are presented in the return value of <c><seealso marker="#p-2">p/2</seealso></c>. </p> + are presented in the return value of <seealso marker="#p-2"><c>p/2</c></seealso>. </p> <p>There may be a tuple <c>{saved, N}</c> in the return value, if the MatchSpec is other than []. The integer <c>N</c> may then be used in subsequent calls to this function and will stand as an "alias" for the given expression. There are also a couple of - built-in aliases for common expressions, see - <c><seealso marker="#ltp-0">ltp/0</seealso></c> below for details.</p> + built-in aliases for common expressions, see + <seealso marker="#ltp-0"><c>ltp/0</c></seealso> below for details.</p> <p>If an error is returned, it can be due to errors in compilation of the match specification. Such errors are presented as a list of tuples <c>{error, string()}</c> where @@ -433,7 +434,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <name>tpl({Module, Function, Arity}, MatchSpec) -> {ok, MatchDesc} | {error, term()}</name> <fsummary>Set pattern for traced local (as well as global) function calls</fsummary> <desc> - <p>This function works as <c><seealso marker="#tp-2">tp/2</seealso></c>, but enables + <p>This function works as <seealso marker="#tp-2"><c>tp/2</c></seealso>, but enables tracing for local calls (and local functions) as well as for global calls (and functions).</p> </desc> @@ -480,10 +481,10 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <p>This function disables call tracing on the specified functions. The semantics of the parameter is the same as for the corresponding function specification in - <c><seealso marker="#tp-2">tp/2</seealso></c> or <c><seealso marker="#tpl-2">tpl/2</seealso></c>. Both local and global call trace + <seealso marker="#tp-2"><c>tp/2</c></seealso> or <seealso marker="#tpl-2"><c>tpl/2</c></seealso>. Both local and global call trace is disabled. </p> <p>The return value reflects how many functions that matched, - and is constructed as described in <c><seealso marker="#tp-2">tp/2</seealso></c>. No tuple + and is constructed as described in <seealso marker="#tp-2"><c>tp/2</c></seealso>. No tuple <c>{saved, N}</c> is however ever returned (for obvious reasons).</p> </desc> </func> @@ -519,8 +520,9 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <name>ctpl({Module, Function, Arity}) -> {ok, MatchDesc} | {error, term()}</name> <fsummary>Clear call trace pattern for the specified functions</fsummary> <desc> - <p>This function works as <c><seealso marker="#ctp-1">ctp/1</seealso></c>, but only disables - tracing set up with <c><seealso marker="#tpl-2">tpl/2</seealso></c> (not with <c><seealso marker="#tp-2">tp/2</seealso></c>).</p> + <p>This function works as <seealso marker="#ctp-1"><c>ctp/1</c></seealso>, but only disables + tracing set up with <seealso marker="#tpl-2"><c>tpl/2</c></seealso> + (not with <seealso marker="#tp-2"><c>tp/2</c></seealso>).</p> </desc> </func> <func> @@ -555,8 +557,9 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <name>ctpg({Module, Function, Arity}) -> {ok, MatchDesc} | {error, term()}</name> <fsummary>Clear call trace pattern for the specified functions</fsummary> <desc> - <p>This function works as <c><seealso marker="#ctp-1">ctp/1</seealso></c>, but only disables - tracing set up with <c><seealso marker="#tp-2">tp/2</seealso></c> (not with <c><seealso marker="#tpl-2">tpl/2</seealso></c>).</p> + <p>This function works as <seealso marker="#ctp-1"><c>ctp/1</c></seealso>, but only disables + tracing set up with <seealso marker="#tp-2"><c>tp/2</c></seealso> + (not with <seealso marker="#tpl-2"><c>tpl/2</c></seealso>).</p> </desc> </func> <func> @@ -565,13 +568,14 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <desc> <p>Use this function to recall all match specifications previously used in the session (i. e. previously saved during calls - to <c><seealso marker="#tp-2">tp/2</seealso></c>, and built-in match specifications. + to <seealso marker="#tp-2"><c>tp/2</c></seealso>, and built-in match specifications. This is very useful, as a complicated match_spec can be quite awkward to write. Note that the - match specifications are lost if <c><seealso marker="#stop-0">stop/0</seealso></c> is called.</p> + match specifications are lost if <seealso marker="#stop-0"><c>stop/0</c></seealso> is called.</p> <p>Match specifications used can be saved in a file (if a read-write file system is present) for use in later - debugging sessions, see <c><seealso marker="#wtp-1">wtp/1</seealso></c> and <c><seealso marker="#rtp-1">rtp/1</seealso></c></p> + debugging sessions, see <seealso marker="#wtp-1"><c>wtp/1</c></seealso> + and <seealso marker="#rtp-1"><c>rtp/1</c></seealso></p> <p>There are three built-in trace patterns: <c>exception_trace</c>, <c>caller_trace</c> and <c>caller_exception_trace</c> (or <c>x</c>, <c>c</c> and @@ -594,10 +598,10 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <fsummary>Delete all saved match specifications.</fsummary> <desc> <p>Use this function to "forget" all match specifications - saved during calls to <c><seealso marker="#tp-2">tp/2</seealso></c>. - This is useful when one wants to restore other match - specifications from a file with <c><seealso marker="#rtp-1">rtp/1</seealso></c>. Use - <c><seealso marker="#dtp-1">dtp/1</seealso></c> to delete specific saved match specifications. </p> + saved during calls to <seealso marker="#tp-2"><c>tp/2</c></seealso>. + This is useful when one wants to restore other match + specifications from a file with <seealso marker="#rtp-1"><c>rtp/1</c></seealso>. Use + <seealso marker="#dtp-1"><c>dtp/1</c></seealso> to delete specific saved match specifications.</p> </desc> </func> <func> @@ -608,7 +612,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </type> <desc> <p>Use this function to "forget" a specific match specification - saved during calls to <c><seealso marker="#tp-2">tp/2</seealso></c>.</p> + saved during calls to <seealso marker="#tp-2"><c>tp/2</c></seealso>.</p> </desc> </func> <func> @@ -620,12 +624,12 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </type> <desc> <p>This function will save all match specifications saved - during the session (during calls to <c><seealso marker="#tp-2">tp/2</seealso></c>) + during the session (during calls to <seealso marker="#tp-2"><c>tp/2</c></seealso>) and built-in match specifications in a text file with the name designated by <c>Name</c>. The format of the file is textual, why it can be edited with an ordinary text editor, and then restored with - <c><seealso marker="#rtp-1">rtp/1</seealso></c>. </p> + <seealso marker="#rtp-1"><c>rtp/1</c></seealso>. </p> <p>Each match spec in the file ends with a full stop (<c>.</c>) and new (syntactically correct) match specifications can be added to the file manually.</p> @@ -643,7 +647,8 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </type> <desc> <p>This function reads match specifications from a file - (possibly) generated by the <c><seealso marker="#wtp-1">wtp/1</seealso></c> function. It checks + (possibly) generated by the <seealso marker="#wtp-1"><c>wtp/1</c></seealso> + function. It checks the syntax of all match specifications and verifies that they are correct. The error handling principle is "all or nothing", i. e. if some of the match specifications are @@ -651,7 +656,8 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ saved match specifications for the running system. </p> <p>The match specifications in the file are <em>merged</em> with the current match specifications, so that no duplicates - are generated. Use <c><seealso marker="#ltp-0">ltp/0</seealso></c> to see what numbers were + are generated. Use <seealso marker="#ltp-0"><c>ltp/0</c></seealso> + to see what numbers were assigned to the specifications from the file.</p> <p>The function will return an error, either due to I/O problems (like a non existing or non readable file) or due @@ -670,9 +676,9 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </type> <desc> <p>The <c>dbg</c> server keeps a list of nodes where tracing - should be performed. Whenever a <c><seealso marker="#tp-2">tp/2</seealso></c> call or a - <c><seealso marker="#p-2">p/2</seealso></c> call is made, it is executed for all nodes in this - list including the local node (except for <c><seealso marker="#p-2">p/2</seealso></c> with a + should be performed. Whenever a <seealso marker="#tp-2"><c>tp/2</c></seealso> call or a + <seealso marker="#p-2"><c>p/2</c></seealso> call is made, it is executed for all nodes in this + list including the local node (except for <seealso marker="#p-2"><c>p/2</c></seealso> with a specific <c>pid()</c> or <c>port()</c> as first argument, in which case the command is executed only on the node where the designated process or port resides). @@ -684,7 +690,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ distribution). If no tracer process is running on the local node, the error reason <c>no_local_tracer</c> is returned. The tracer process on the local node must be started with the - <c><seealso marker="#tracer-2">tracer/0/2</seealso></c> function. + <seealso marker="#tracer-2"><c>tracer/0/2</c></seealso> function. </p> <p>If <c>Nodename</c> is the local node, the error reason <c>cant_add_local_node</c> is returned. @@ -694,7 +700,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ a tracer process. The error reason <c>cant_trace_remote_pid_to_local_port</c> is returned. A trace port can however be started on the remote node with the - <c><seealso marker="#tracer-3">tracer/3</seealso></c> function. + <seealso marker="#tracer-3"><c>tracer/3</c></seealso> function. </p> <p>The function will also return an error if the node <c>Nodename</c> is not reachable.</p> @@ -708,9 +714,10 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ </type> <desc> <p>Clears a node from the list of traced nodes. Subsequent - calls to <c><seealso marker="#tp-2">tp/2</seealso></c> and <c><seealso marker="#p-2">p/2</seealso></c> will not consider that - node, but tracing already activated on the node will continue - to be in effect.</p> + calls to <seealso marker="#tp-2"><c>tp/2</c></seealso> and + <seealso marker="#p-2"><c>p/2</c></seealso> will not consider that + node, but tracing already activated on the node will continue + to be in effect.</p> <p>Returns <c>ok</c>, cannot fail.</p> </desc> </func> @@ -727,14 +734,14 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <desc> <p>This function starts a server on the local node that will be the recipient of all trace messages. All subsequent calls - to <c><seealso marker="#p-2">p/2</seealso></c> will result in messages sent to the newly + to <seealso marker="#p-2"><c>p/2</c></seealso> will result in messages sent to the newly started trace server.</p> <p>A trace server started in this way will simply display the trace messages in a formatted way in the Erlang shell - (i. e. use io:format). See <c><seealso marker="#tracer-2">tracer/2</seealso></c> for a description - of how the trace message handler can be customized. + (i. e. use io:format). See <seealso marker="#tracer-2"><c>tracer/2</c></seealso> + for a description of how the trace message handler can be customized. </p> - <p>To start a similar tracer on a remote node, use <c><seealso marker="#n-1">n/1</seealso></c>.</p> + <p>To start a similar tracer on a remote node, use <seealso marker="#n-1"><c>n/1</c></seealso>.</p> </desc> </func> <func> @@ -758,9 +765,9 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ by a receiving process (<c>process</c>), by a tracer port (<c>port</c>) or by a tracer module (<c>module</c>). For a description about tracer ports see - <c><seealso marker="#trace_port-2">trace_port/2</seealso></c> + <seealso marker="#trace_port-2"><c>trace_port/2</c></seealso> and for a tracer modules see - <c><seealso marker="erts:erl_tracer">erl_tracer</seealso>.</c> + <seealso marker="erts:erl_tracer"><c>erl_tracer</c></seealso>. </p> <p>If <c>Type</c> is <c>process</c>, a message handler function can be specified (<c>HandlerSpec</c>). The handler function, which @@ -776,10 +783,10 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <p>If <c>Type</c> is <c>port</c>, then the second parameter should be a <em>fun</em> which takes no arguments and returns a newly opened trace port when called. Such a <em>fun</em> is - preferably generated by calling <c><seealso marker="#trace_port-2">trace_port/2</seealso></c>. + preferably generated by calling <seealso marker="#trace_port-2"><c>trace_port/2</c></seealso>. </p> <p>if <c>Type</c> is <c>module</c>, then the second parameter should - be either a tuple describing the <c><seealso marker="erts:erl_tracer">erl_tracer</seealso></c> + be either a tuple describing the <seealso marker="erts:erl_tracer"><c>erl_tracer</c></seealso> module to be used for tracing and the state to be used for that tracer module or a fun returning the same tuple.</p> <p>If an error is returned, it can either be due to a tracer @@ -787,7 +794,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ due to the <c>HandlerFun</c> throwing an exception. </p> <p>To start a similar tracer on a remote node, use - <c><seealso marker="#tracer-3">tracer/3</seealso></c>. + <seealso marker="#tracer-3"><c>tracer/3</c></seealso>. </p> </desc> </func> @@ -798,19 +805,19 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <v>Nodename = atom()</v> </type> <desc> - <p>This function is equivalent to <c><seealso marker="#tracer-2">tracer/2</seealso></c>, but acts on + <p>This function is equivalent to <seealso marker="#tracer-2"><c>tracer/2</c></seealso>, but acts on the given node. A tracer is started on the node (<c>Nodename</c>) and the node is added to the list of traced nodes. </p> <note> - <p>This function is not equivalent to <c><seealso marker="#n-1">n/1</seealso></c>. While - <c><seealso marker="#n-1">n/1</seealso></c> starts a process tracer which redirects all trace + <p>This function is not equivalent to <seealso marker="#n-1"><c>n/1</c></seealso>. While + <seealso marker="#n-1"><c>n/1</c></seealso> starts a process tracer which redirects all trace information to a process tracer on the local node (i.e. the - trace control node), <c><seealso marker="#tracer-3">tracer/3</seealso></c> starts a tracer of any + trace control node), <seealso marker="#tracer-3"><c>tracer/3</c></seealso> starts a tracer of any type which is independent of the tracer on the trace control node.</p> </note> - <p>For details, see <c><seealso marker="#tracer-2">tracer/2</seealso></c>.</p> + <p>For details, see <seealso marker="#tracer-2"><c>tracer/2</c></seealso>.</p> </desc> </func> <func> @@ -842,9 +849,9 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <c>file</c> and the <c>ip</c> trace drivers. The file driver sends all trace messages into one or several binary files, from where they later can be fetched and processed with the - <c><seealso marker="#trace_client-2">trace_client/2</seealso></c> function. The ip driver opens a TCP/IP + <seealso marker="#trace_client-2"><c>trace_client/2</c></seealso> function. The ip driver opens a TCP/IP port where it listens for connections. When a client - (preferably started by calling <c><seealso marker="#trace_client-2">trace_client/2</seealso></c> on + (preferably started by calling <seealso marker="#trace_client-2"><c>trace_client/2</c></seealso> on another Erlang node) connects, all trace messages are sent over the IP network for further processing by the remote client. </p> @@ -883,7 +890,8 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ as fast as they are produced by the runtime system, a special message is sent, which indicates how many messages that are dropped. That message will arrive at the handler function - specified in <c><seealso marker="#trace_client-3">trace_client/3</seealso></c> as the tuple <c>{drop, N}</c> where <c>N</c> is the number of consecutive messages + specified in <seealso marker="#trace_client-3"><c>trace_client/3</c></seealso> + as the tuple <c>{drop, N}</c> where <c>N</c> is the number of consecutive messages dropped. In case of heavy tracing, drop's are likely to occur, and they surely occur if no client is reading the trace messages.</p> @@ -960,8 +968,8 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <desc> <p>This function starts a trace client that reads the output created by a trace port driver and handles it in mostly the - same way as a tracer process created by the <c><seealso marker="#tracer-0">tracer/0</seealso></c> - function.</p> + same way as a tracer process created by the + <seealso marker="#tracer-0"><c>tracer/0</c></seealso> function.</p> <p>If <c>Type</c> is <c>file</c>, the client reads all trace messages stored in the file named <c>Filename</c> or specified by <c>WrapFilesSpec</c> (must be the same as used @@ -972,7 +980,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ <p>If <c>Type</c> is <c>follow_file</c>, the client behaves as in the <c>file</c> case, but keeps trying to read (and process) more data - from the file until stopped by <c><seealso marker="#stop_trace_client-1">stop_trace_client/1</seealso></c>. + from the file until stopped by <seealso marker="#stop_trace_client-1"><c>stop_trace_client/1</c></seealso>. <c>WrapFilesSpec</c> is not allowed as second argument for this <c>Type</c>.</p> <p>If <c>Type</c> is <c>ip</c>, the client connects to the @@ -1028,10 +1036,10 @@ hello</pre> <v>InitialData = term()</v> </type> <desc> - <p>This function works exactly as <c><seealso marker="#trace_client-2">trace_client/2</seealso></c>, but - allows you to write your own handler function. The handler + <p>This function works exactly as <seealso marker="#trace_client-2"><c>trace_client/2</c></seealso>, + but allows you to write your own handler function. The handler function works mostly as the one described in - <c><seealso marker="#tracer-2">tracer/2</seealso></c>, but will also have to be prepared to handle + <seealso marker="#tracer-2"><c>tracer/2</c></seealso>, but will also have to be prepared to handle trace messages of the form <c>{drop, N}</c>, where <c>N</c> is the number of dropped messages. This pseudo trace message will only occur if the ip trace driver is used.</p> @@ -1050,7 +1058,8 @@ hello</pre> <desc> <p>This function shuts down a previously started trace client. The <c>Pid</c> argument is the process id returned - from the <c><seealso marker="#trace_client-2">trace_client/2</seealso></c> or <c><seealso marker="#trace_client-3">trace_client/3</seealso></c> call.</p> + from the <seealso marker="#trace_client-2"><c>trace_client/2</c></seealso> + or <seealso marker="#trace_client-3"><c>trace_client/3</c></seealso> call.</p> </desc> </func> <func> @@ -1203,7 +1212,7 @@ SeqTrace [0]: (<0.30.0>) <0.25.0> ! {dbg,{ok,<0.31.0>}} [Serial: {4,5}] of causing a deadlock. This will happen if a group leader process generates a trace message and the tracer process, by calling the trace handler function, sends an IO request to the same group leader. The problem can only occur if the trace handler - prints to tty using an <c>io</c> function such as <c><seealso marker="stdlib:io#format-2">format/2</seealso></c>. + prints to tty using an <c>io</c> function such as <seealso marker="stdlib:io#format-2"><c>format/2</c></seealso>. Note that when <c>dbg:p(all,call)</c> is called, IO processes are also traced. Here's an example:</p> diff --git a/lib/runtime_tools/src/appmon_info.erl b/lib/runtime_tools/src/appmon_info.erl index fead724373..b5500085a3 100644 --- a/lib/runtime_tools/src/appmon_info.erl +++ b/lib/runtime_tools/src/appmon_info.erl @@ -307,9 +307,10 @@ do_work(Key, State) -> {Cmd, Aux, From, _OldRef, Old, Opts} = retrieve(WorkStore, Key), {ok, Result} = do_work2(Cmd, Aux, From, Old, Opts), if - Result==Old -> ok; - true -> - From ! {delivery, self(), Cmd, Aux, Result} + Result==Old -> ok; + true -> + From ! {delivery, self(), Cmd, Aux, Result}, + ok end, case get_opt(timeout, Opts) of at_most_once -> @@ -393,7 +394,7 @@ del_task(Key, WorkStore) -> {_Cmd, _Aux, _From, Ref, _Old, Opts} -> if Ref /= nil -> - timer:cancel(Ref), + {ok,_} = timer:cancel(Ref), receive {do_it, Key} -> Opts diff --git a/lib/runtime_tools/src/dbg.erl b/lib/runtime_tools/src/dbg.erl index d5ff874206..8cdb5a43e3 100644 --- a/lib/runtime_tools/src/dbg.erl +++ b/lib/runtime_tools/src/dbg.erl @@ -20,6 +20,7 @@ -module(dbg). -export([p/1,p/2,c/3,c/4,i/0,start/0,stop/0,stop_clear/0,tracer/0, tracer/2, tracer/3, get_tracer/0, get_tracer/1, tp/2, tp/3, tp/4, + tpe/2, ctpe/1, ctp/0, ctp/1, ctp/2, ctp/3, tpl/2, tpl/3, tpl/4, ctpl/0, ctpl/1, ctpl/2, ctpl/3, ctpg/0, ctpg/1, ctpg/2, ctpg/3, ltp/0, wtp/1, rtp/1, dtp/0, dtp/1, n/1, cn/1, ln/0, h/0, h/1]). @@ -128,7 +129,12 @@ tpl(Module, Pattern) when is_atom(Module) -> do_tp({Module, '_', '_'}, Pattern, [local]); tpl({_Module, _Function, _Arity} = X, Pattern) -> do_tp(X,Pattern,[local]). -do_tp({_Module, _Function, _Arity} = X, Pattern, Flags) + +tpe(Event, Pattern) when Event =:= send; + Event =:= 'receive' -> + do_tp(Event, Pattern, []). + +do_tp(X, Pattern, Flags) when is_integer(Pattern); is_atom(Pattern) -> case ets:lookup(get_pattern_table(), Pattern) of @@ -137,17 +143,16 @@ do_tp({_Module, _Function, _Arity} = X, Pattern, Flags) _ -> {error, unknown_pattern} end; -do_tp({Module, _Function, _Arity} = X, Pattern, Flags) when is_list(Pattern) -> +do_tp(X, Pattern, Flags) when is_list(Pattern) -> Nodes = req(get_nodes), - case Module of - '_' -> - ok; - M when is_atom(M) -> + case X of + {M,_,_} when is_atom(M) -> %% Try to load M on all nodes lists:foreach(fun(Node) -> rpc:call(Node, M, module_info, []) end, - Nodes) + Nodes); + _ -> ok end, case lint_tp(Pattern) of {ok,_} -> @@ -163,9 +168,9 @@ do_tp({Module, _Function, _Arity} = X, Pattern, Flags) when is_list(Pattern) -> end. %% All nodes are handled the same way - also the local node if it is traced -do_tp_on_nodes(Nodes, MFA, P, Flags) -> +do_tp_on_nodes(Nodes, X, P, Flags) -> lists:map(fun(Node) -> - case rpc:call(Node,erlang,trace_pattern,[MFA,P, Flags]) of + case rpc:call(Node,erlang,trace_pattern,[X,P, Flags]) of N when is_integer(N) -> {matched, Node, N}; Else -> @@ -210,13 +215,19 @@ ctpg(Module) when is_atom(Module) -> do_ctp({Module, '_', '_'}, [global]); ctpg({_Module, _Function, _Arity} = X) -> do_ctp(X,[global]). + do_ctp({Module, Function, Arity},[]) -> - do_ctp({Module, Function, Arity},[global]), + {ok,_} = do_ctp({Module, Function, Arity},[global]), do_ctp({Module, Function, Arity},[local]); do_ctp({_Module, _Function, _Arity}=MFA,Flags) -> Nodes = req(get_nodes), {ok,do_tp_on_nodes(Nodes,MFA,false,Flags)}. +ctpe(Event) when Event =:= send; + Event =:= 'receive' -> + Nodes = req(get_nodes), + {ok,do_tp_on_nodes(Nodes,Event,true,[])}. + %% %% ltp() -> ok %% List saved and built-in trace patterns. @@ -260,8 +271,7 @@ wtp(FileName) -> ok end, []), - file:close(File), - ok + ok = file:close(File) end. %% @@ -589,7 +599,7 @@ stop() -> end. stop_clear() -> - ctp(), + {ok, _} = ctp(), stop(). %%% Calling the server. @@ -780,7 +790,8 @@ loop({C,T}=SurviveLinks, Table) -> end. reply(Pid, Reply) -> - Pid ! {dbg,Reply}. + Pid ! {dbg,Reply}, + ok. %%% A process-based tracer. @@ -933,9 +944,11 @@ do_relay(Parent,RelP) -> case RelP of {Type,Data} -> {ok,Tracer} = remote_tracer(Type,Data), - Parent ! {started,Tracer}; + Parent ! {started,Tracer}, + ok; Pid when is_pid(Pid) -> - Parent ! {started,self()} + Parent ! {started,self()}, + ok end, do_relay_1(RelP). @@ -1355,7 +1368,7 @@ mk_reader_wrap([_Hd | Tail] = WrapFiles, File) -> {ok, Term} -> [Term | mk_reader_wrap(WrapFiles, File)]; eof -> - file:close(File), + ok = file:close(File), case Tail of [_|_] -> mk_reader_wrap(Tail); diff --git a/lib/runtime_tools/src/erts_alloc_config.erl b/lib/runtime_tools/src/erts_alloc_config.erl index e94cced911..514530332c 100644 --- a/lib/runtime_tools/src/erts_alloc_config.erl +++ b/lib/runtime_tools/src/erts_alloc_config.erl @@ -128,7 +128,7 @@ make_config(FileName) when is_list(FileName) -> case file:open(FileName, [write]) of {ok, IODev} -> Res = req({make_config, IODev}), - file:close(IODev), + ok = file:close(IODev), Res; Error -> Error @@ -200,9 +200,11 @@ server_loop(State) -> Conf = #conf{segments = ?MBC_MSEG_LIMIT, format_to = IODev}, Res = mk_config(Conf, State#state.alloc), - From ! {response, Ref, Res}; + From ! {response, Ref, Res}, + ok; _ -> - From ! {response, Ref, no_scenario_saved} + From ! {response, Ref, no_scenario_saved}, + ok end, State; {request, From, Ref, stop} -> diff --git a/lib/runtime_tools/src/observer_backend.erl b/lib/runtime_tools/src/observer_backend.erl index 30df3b0b8b..66653c5b7f 100644 --- a/lib/runtime_tools/src/observer_backend.erl +++ b/lib/runtime_tools/src/observer_backend.erl @@ -259,7 +259,8 @@ etop_collect(Collector) -> case SchedulerWallTime of undefined -> - spawn(fun() -> flag_holder_proc(Collector) end); + spawn(fun() -> flag_holder_proc(Collector) end), + ok; _ -> ok end, @@ -334,8 +335,8 @@ ttb_init_node(MetaFile_0,PI,Traci) -> MetaPid ! {metadata,Traci}, case PI of true -> - Proci = pnames(), - MetaPid ! {metadata,Proci}; + MetaPid ! {metadata,pnames()}, + ok; false -> ok end, @@ -354,7 +355,8 @@ ttb_meta_tracer(MetaFile,PI,Parent,SessionData) -> erlang:trace_pattern({erlang,spawn_link,3},ReturnMS,[meta]), erlang:trace_pattern({erlang,spawn_opt,1},ReturnMS,[meta]), erlang:trace_pattern({erlang,register,2},[],[meta]), - erlang:trace_pattern({global,register_name,2},[],[meta]); + erlang:trace_pattern({global,register_name,2},[],[meta]), + ok; false -> ok end, @@ -362,7 +364,8 @@ ttb_meta_tracer(MetaFile,PI,Parent,SessionData) -> case proplists:get_value(overload_check, SessionData) of {Ms, M, F} -> catch M:F(init), - erlang:send_after(Ms, self(), overload_check); + erlang:send_after(Ms, self(), overload_check), + ok; _ -> ok end, @@ -371,10 +374,10 @@ ttb_meta_tracer(MetaFile,PI,Parent,SessionData) -> ttb_meta_tracer_loop(MetaFile,PI,Acc,State) -> receive {trace_ts,_,call,{erlang,register,[Name,Pid]},_} -> - ttb_store_meta({pid,{Pid,Name}},MetaFile), + ok = ttb_store_meta({pid,{Pid,Name}},MetaFile), ttb_meta_tracer_loop(MetaFile,PI,Acc,State); {trace_ts,_,call,{global,register_name,[Name,Pid]},_} -> - ttb_store_meta({pid,{Pid,{global,Name}}},MetaFile), + ok = ttb_store_meta({pid,{Pid,{global,Name}}},MetaFile), ttb_meta_tracer_loop(MetaFile,PI,Acc,State); {trace_ts,CallingPid,call,{erlang,spawn_opt,[{M,F,Args,_}]},_} -> MFA = {M,F,length(Args)}, @@ -390,7 +393,7 @@ ttb_meta_tracer_loop(MetaFile,PI,Acc,State) -> NewAcc = dict:update(CallingPid, fun([H|T]) -> - ttb_store_meta({pid,{NewPid,H}},MetaFile), + ok = ttb_store_meta({pid,{NewPid,H}},MetaFile), T end, Acc), @@ -408,22 +411,22 @@ ttb_meta_tracer_loop(MetaFile,PI,Acc,State) -> NewAcc = dict:update(CallingPid, fun([H|T]) -> - ttb_store_meta({pid,{NewPid,H}},MetaFile), + ok = ttb_store_meta({pid,{NewPid,H}},MetaFile), T end, Acc), ttb_meta_tracer_loop(MetaFile,PI,NewAcc,State); {metadata,Data} when is_list(Data) -> - ttb_store_meta(Data,MetaFile), + ok = ttb_store_meta(Data,MetaFile), ttb_meta_tracer_loop(MetaFile,PI,Acc,State); {metadata,Key,Fun} when is_function(Fun) -> - ttb_store_meta([{Key,Fun()}],MetaFile), + ok = ttb_store_meta([{Key,Fun()}],MetaFile), ttb_meta_tracer_loop(MetaFile,PI,Acc,State); {metadata,Key,What} -> - ttb_store_meta([{Key,What}],MetaFile), + ok = ttb_store_meta([{Key,What}],MetaFile), ttb_meta_tracer_loop(MetaFile,PI,Acc,State); overload_check -> {Ms, M, F} = proplists:get_value(overload_check, State), @@ -439,7 +442,7 @@ ttb_meta_tracer_loop(MetaFile,PI,Acc,State) -> ttb_meta_tracer_loop(MetaFile,PI,Acc, State) end; {'DOWN', _, _, _, _} -> - stop_seq_trace(), + _ = stop_seq_trace(), self() ! stop, ttb_meta_tracer_loop(MetaFile,PI,Acc, State); stop when PI=:=true -> @@ -528,7 +531,7 @@ ttb_store_meta(Data,MetaFile) -> ttb_store_meta([Data],MetaFile). ttb_write_binary(Fd,[H|T]) -> - file:write(Fd,ttb_make_binary(H)), + ok = file:write(Fd,ttb_make_binary(H)), ttb_write_binary(Fd,T); ttb_write_binary(_Fd,[]) -> ok. @@ -585,9 +588,9 @@ ttb_fetch(MetaFile,{Port,Host}) -> send_files({Sock,Host},[File|Files]) -> {ok,Fd} = file:open(File,[raw,read,binary]), - gen_tcp:send(Sock,<<1,(list_to_binary(filename:basename(File)))/binary>>), + ok = gen_tcp:send(Sock,<<1,(list_to_binary(filename:basename(File)))/binary>>), send_chunks(Sock,Fd), - file:delete(File), + ok = file:delete(File), send_files({Sock,Host},Files); send_files({_Sock,_Host},[]) -> done. diff --git a/lib/runtime_tools/src/percept_profile.erl b/lib/runtime_tools/src/percept_profile.erl index ceec4d3b89..1e8e913b80 100644 --- a/lib/runtime_tools/src/percept_profile.erl +++ b/lib/runtime_tools/src/percept_profile.erl @@ -87,7 +87,7 @@ start(Filename, Options) -> start(Filename, {Module, Function, Args}, Options) -> case whereis(percept_port) of undefined -> - profile_to_file(Filename, Options), + {ok, _} = profile_to_file(Filename, Options), erlang:apply(Module, Function, Args), stop(); Port -> @@ -113,7 +113,7 @@ deliver_all_trace() -> -spec stop() -> 'ok' | {'error', 'not_started'}. stop() -> - erlang:system_profile(undefined, [runnable_ports, runnable_procs]), + _ = erlang:system_profile(undefined, [runnable_ports, runnable_procs]), erlang:trace(all, false, [procs, ports, timestamp]), deliver_all_trace(), case whereis(percept_port) of @@ -158,7 +158,8 @@ set_tracer(Port, Opts) -> {TOpts, POpts} = parse_profile_options(Opts), % Setup profiling and tracing erlang:trace(all, true, [{tracer, Port}, timestamp | TOpts]), - erlang:system_profile(Port, POpts). + _ = erlang:system_profile(Port, POpts), + ok. %% parse_profile_options diff --git a/lib/runtime_tools/src/system_information.erl b/lib/runtime_tools/src/system_information.erl index 1add01612d..df25297eb9 100644 --- a/lib/runtime_tools/src/system_information.erl +++ b/lib/runtime_tools/src/system_information.erl @@ -31,6 +31,7 @@ -export([report/0, from_file/1, to_file/1]). + -export([start/0, stop/0, load_report/0, load_report/2, applications/0, applications/1, @@ -64,6 +65,7 @@ start() -> gen_server:start({local, ?SERVER}, ?MODULE, [], []). + stop() -> gen_server:call(?SERVER, stop, infinity). @@ -71,7 +73,7 @@ load_report() -> load_report(data, report()). load_report(file, File) -> load_report(data, from_file(File)); load_report(data, Report) -> - start(), gen_server:call(?SERVER, {load_report, Report}, infinity). + ok = start_internal(), gen_server:call(?SERVER, {load_report, Report}, infinity). report() -> [ {init_arguments, init:get_arguments()}, @@ -219,6 +221,13 @@ code_change(_OldVsn, State, _Extra) -> %% Internal functions %%=================================================================== +start_internal() -> + case start() of + {ok,_} -> ok; + {error, {already_started,_}} -> ok; + Error -> Error + end. + %% handle report values get_value([], Data) -> Data; diff --git a/lib/runtime_tools/test/dbg_SUITE.erl b/lib/runtime_tools/test/dbg_SUITE.erl index 0777503ef7..5a3c885571 100644 --- a/lib/runtime_tools/test/dbg_SUITE.erl +++ b/lib/runtime_tools/test/dbg_SUITE.erl @@ -22,6 +22,7 @@ %% Test functions -export([all/0, suite/0, big/1, tiny/1, simple/1, message/1, distributed/1, port/1, + send/1, recv/1, ip_port/1, file_port/1, file_port2/1, file_port_schedfix/1, ip_port_busy/1, wrap_port/1, wrap_port_time/1, with_seq_trace/1, dead_suspend/1, local_trace/1, @@ -39,6 +40,7 @@ suite() -> all() -> [big, tiny, simple, message, distributed, port, ip_port, + send, recv, file_port, file_port2, file_port_schedfix, ip_port_busy, wrap_port, wrap_port_time, with_seq_trace, dead_suspend, local_trace, saved_patterns, tracer_exit_on_stop, @@ -151,6 +153,226 @@ message(Config) when is_list(Config) -> {trace,S,call,{dbg,ln,[]},S}] = flush(), ok. +send(Config) when is_list(Config) -> + {ok, _} = start(), + Node = start_slave(), + rpc:call(Node, code, add_patha, + [filename:join(proplists:get_value(data_dir, Config), "..")]), + try + Echo = fun F() -> + receive {From, M} -> + From ! M, + F() + end + end, + Rcvr = spawn_link(Echo), + RemoteRcvr = spawn_link(Node, Echo), + + {ok, [{matched, _, 1}]} = dbg:p(Rcvr, send), + + send_test(Rcvr, make_ref(), true), + + %% Test that the test case process is the receiving process + send_test(Rcvr, [{[self(),'_'],[],[]}]), + + %% Test that self() is not the receiving process + {ok, [{matched, _node, 1}, {saved, 2}]} = + dbg:tpe(send, [{['$1','_'],[{'==','$1',{self}}],[]}]), + send_test(Rcvr, make_ref(), false), + + %% Test that self() is the sending process + send_test(Rcvr, [{['_','_'],[{'==',Rcvr,{self}}],[]}]), + + %% Test attaching a message + send_test(Rcvr, [{['_','_'],[{'==',Rcvr,{self}}],[{message, hello}]}], + make_ref(), hello), + + %% Test using a saved trace pattern + send_test(Rcvr, 2, make_ref(), false), + + %% Test clearing of trace pattern + {ok, [{matched, _node, 1}]} = dbg:ctpe(send), + send_test(Rcvr, make_ref(), true), + + %% Test complex message inspection + Ref = make_ref(), + send_test(Rcvr, + [{['_','$2'],[{'==',Ref,{element,1,{element,2,'$2'}}}],[]}], + {test, {Ref}, <<0:(8*1024)>>}, true), + + send_test(Rcvr, {test, {make_ref()}, <<0:(8*1024)>>}, false), + + %% Test send to remote process + remote_send_test(Rcvr, RemoteRcvr, [], make_ref(), true), + + remote_send_test(Rcvr, RemoteRcvr, + [{['$1','_'],[{'==',{node, '$1'},{node}}],[]}], + make_ref(), false), + + remote_send_test(Rcvr, RemoteRcvr, + [{['$1','_'],[{'==',{node, '$1'},Node}],[]}], + make_ref(), true), + + %% Test that distributed dbg works + dbg:tracer(Node, process, {fun myhandler/2, self()}), + Rcvr2 = spawn_link(Echo), + RemoteRcvr2 = spawn_link(Node, Echo), + dbg:p(Rcvr2, [send]), + dbg:p(RemoteRcvr2, [send]), + dbg:tpe(send, [{['_', hej],[],[]}]), + + send_test(Rcvr2, make_ref(), false), + send_test(RemoteRcvr2, make_ref(), false), + send_test(Rcvr2, hej, true), + send_test(RemoteRcvr2, hej, true), + + ok + + after + stop() + end. + +send_test(Pid, Pattern, Msg, TraceEvent) -> + {ok, [{matched, _, _}, _]} = dbg:tpe(send, Pattern), + send_test(Pid, Msg, TraceEvent). +send_test(Pid, Pattern) -> + send_test(Pid, Pattern, make_ref(), true). +send_test(Pid, Msg, TraceEvent) -> + S = self(), + Pid ! {S, Msg}, + receive Msg -> ok end, + send_test_rcv(Pid, Msg, S, TraceEvent). + +remote_send_test(Pid, RPid, Pattern, Msg, TraceEvent) -> + dbg:tpe(send, Pattern), + TMsg = {self(), Msg}, + Pid ! {RPid, TMsg}, + receive Msg -> ok end, + send_test_rcv(Pid, TMsg, RPid, TraceEvent). + +send_test_rcv(Pid, Msg, S, TraceEvent) -> + case flush() of + [] when not TraceEvent -> + ok; + [{trace, Pid, send, Msg, S}] when TraceEvent -> + ok; + [{trace, Pid, send, Msg, S, Message}] when TraceEvent == Message -> + ok; + Else -> + ct:fail({got_unexpected_message, Else}) + end. + +recv(Config) when is_list(Config) -> + {ok, _} = start(), + Node = start_slave(), + rpc:call(Node, code, add_patha, + [filename:join(proplists:get_value(data_dir, Config), "..")]), + try + Echo = fun F() -> + receive {From, M} -> + From ! M, + F() + end + end, + Rcvr = spawn_link(Echo), + RemoteRcvr = spawn_link(Node, Echo), + + {ok, [{matched, _, 1}]} = dbg:p(Rcvr, 'receive'), + + recv_test(Rcvr, make_ref(), true), + + %% Test that the test case process is the sending process + recv_test(Rcvr, [{[node(), self(), '_'],[],[]}]), + + %% Test that self() is the not sending process + {ok, [{matched, _node, 1}, {saved, 2}]} = + dbg:tpe('receive', [{[node(), '$1','_'],[{'==','$1',{self}}],[]}]), + recv_test(Rcvr, make_ref(), false), + + %% Test that self() is the receiving process + recv_test(Rcvr, [{'_',[{'==',Rcvr,{self}}],[]}]), + + %% Test attaching a message + recv_test(Rcvr, [{'_',[{'==',Rcvr,{self}}],[{message, hello}]}], + make_ref(), hello), + + %% Test using a saved trace pattern + recv_test(Rcvr, 2, make_ref(), false), + + %% Test clearing of trace pattern + {ok, [{matched, _node, 1}]} = dbg:ctpe('receive'), + recv_test(Rcvr, make_ref(), true), + + %% Test complex message inspection + Ref = make_ref(), + recv_test(Rcvr, + [{[node(), '_','$2'],[{'==',Ref,{element,1, + {element,2, + {element,2,'$2'}}}}],[]}], + {test, {Ref}, <<0:(8*1024)>>}, true), + + recv_test(Rcvr, {test, {make_ref()}, <<0:(8*1024)>>}, false), + + %% Test recv to remote process + remote_recv_test(RemoteRcvr, Rcvr, [], make_ref(), true), + + remote_recv_test(RemoteRcvr, Rcvr, + [{['$1',undefined,'_'],[{'==','$1',{node}}],[]}], + make_ref(), false), + + remote_recv_test(RemoteRcvr, Rcvr, + [{['$1',undefined,'_'],[{'==','$1',Node}],[]}], + make_ref(), true), + + %% Test that distributed dbg works + dbg:tracer(Node, process, {fun myhandler/2, self()}), + Rcvr2 = spawn_link(Echo), + RemoteRcvr2 = spawn_link(Node, Echo), + dbg:p(Rcvr2, ['receive']), + dbg:p(RemoteRcvr2, ['receive']), + dbg:tpe('receive', [{[node(), '_', '$1'],[{'==',{element,2,'$1'}, hej}],[]}]), + + recv_test(Rcvr2, make_ref(), false), + recv_test(RemoteRcvr2, make_ref(), false), + recv_test(Rcvr2, hej, true), + recv_test(RemoteRcvr2, hej, true), + + ok + + after + stop() + end. + +recv_test(Pid, Pattern, Msg, TraceEvent) -> + {ok, [{matched, _, _}, _]} = dbg:tpe('receive', Pattern), + recv_test(Pid, Msg, TraceEvent). +recv_test(Pid, Pattern) -> + recv_test(Pid, Pattern, make_ref(), true). +recv_test(Pid, Msg, TraceEvent) -> + S = self(), + Pid ! {S, Msg}, + receive Msg -> ok end, + recv_test_rcv(Pid, {S, Msg}, TraceEvent). + +remote_recv_test(RPid, Pid, Pattern, Msg, TraceEvent) -> + dbg:tpe('receive', Pattern), + TMsg = {self(), Msg}, + RPid ! {Pid, TMsg}, + receive Msg -> ok end, + recv_test_rcv(Pid, TMsg, TraceEvent). + +recv_test_rcv(Pid, Msg, TraceEvent) -> + case flush() of + [] when not TraceEvent -> + ok; + [{trace, Pid, 'receive', Msg}] when TraceEvent -> + ok; + [{trace, Pid, 'receive', Msg, Message}] when TraceEvent == Message -> + ok; + Else -> + ct:fail({got_unexpected_message, Else}) + end. + %% Simple test of distributed tracing distributed(Config) when is_list(Config) -> {ok, _} = start(), diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index 946adcf384..e952a333ff 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -1514,39 +1514,35 @@ call(FsmPid, Event, Timeout) -> end. -handle_connection_msg(Msg, StateName, State0 = - #data{starter = User, - connection_state = Connection0, - event_queue = Qev0}) -> +handle_connection_msg(Msg, StateName, D0 = #data{starter = User, + connection_state = Connection0, + event_queue = Qev0}) -> Renegotiation = renegotiation(StateName), Role = role(StateName), - try ssh_connection:handle_msg(Msg, Connection0, Role) of + try ssh_connection:handle_msg(Msg, Connection0, Role) of {{replies, Replies}, Connection} -> - case StateName of - {connected,_} -> - {Repls, State} = send_replies(Replies, - State0#data{connection_state=Connection}), - {keep_state, State, Repls}; - _ -> - {ConnReplies, Replies} = - lists:splitwith(fun not_connected_filter/1, Replies), - {Repls, State} = send_replies(Replies, - State0#data{event_queue = Qev0 ++ ConnReplies}), - {keep_state, State, Repls} - end; + {Repls, D} = + case StateName of + {connected,_} -> + send_replies(Replies, D0#data{connection_state=Connection}); + _ -> + {ConnReplies, NonConnReplies} = lists:splitwith(fun not_connected_filter/1, Replies), + send_replies(NonConnReplies, D0#data{event_queue = Qev0 ++ ConnReplies}) + end, + {keep_state, D, Repls}; {noreply, Connection} -> - {keep_state, State0#data{connection_state = Connection}}; + {keep_state, D0#data{connection_state = Connection}}; {disconnect, Reason0, {{replies, Replies}, Connection}} -> - {Repls,State} = send_replies(Replies, State0#data{connection_state = Connection}), + {Repls, D} = send_replies(Replies, D0#data{connection_state = Connection}), case {Reason0,Role} of {{_, Reason}, client} when ((StateName =/= {connected,client}) and (not Renegotiation)) -> User ! {self(), not_connected, Reason}; _ -> ok end, - {stop_and_reply, {shutdown,normal}, Repls, State#data{connection_state = Connection}} + {stop_and_reply, {shutdown,normal}, Repls, D#data{connection_state = Connection}} catch _:Error -> @@ -1555,8 +1551,8 @@ handle_connection_msg(Msg, StateName, State0 = #ssh_msg_disconnect{code = ?SSH_DISCONNECT_BY_APPLICATION, description = "Internal error"}, Connection0, Role), - {Repls,State} = send_replies(Replies, State0#data{connection_state = Connection}), - {stop_and_reply, {shutdown,Error}, Repls, State#data{connection_state = Connection}} + {Repls, D} = send_replies(Replies, D0#data{connection_state = Connection}), + {stop_and_reply, {shutdown,Error}, Repls, D#data{connection_state = Connection}} end. diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index dbbb25025c..51732b4a59 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -400,24 +400,23 @@ negotiated_next_protocol(Socket) -> end. %%-------------------------------------------------------------------- +-spec cipher_suites() -> [ssl_cipher:erl_cipher_suite()] | [string()]. +%%-------------------------------------------------------------------- +cipher_suites() -> + cipher_suites(erlang). +%%-------------------------------------------------------------------- -spec cipher_suites(erlang | openssl | all) -> [ssl_cipher:erl_cipher_suite()] | [string()]. %% Description: Returns all supported cipher suites. %%-------------------------------------------------------------------- cipher_suites(erlang) -> - Version = tls_record:highest_protocol_version([]), - ssl_cipher:filter_suites([ssl_cipher:erl_suite_definition(S) - || S <- ssl_cipher:suites(Version)]); + [ssl_cipher:erl_suite_definition(Suite) || Suite <- available_suites(default)]; + cipher_suites(openssl) -> - Version = tls_record:highest_protocol_version([]), - [ssl_cipher:openssl_suite_name(S) - || S <- ssl_cipher:filter_suites(ssl_cipher:suites(Version))]; + [ssl_cipher:openssl_suite_name(Suite) || Suite <- available_suites(default)]; + cipher_suites(all) -> - Version = tls_record:highest_protocol_version([]), - ssl_cipher:filter_suites([ssl_cipher:erl_suite_definition(S) - || S <-ssl_cipher:all_suites(Version)]). -cipher_suites() -> - cipher_suites(erlang). + [ssl_cipher:erl_suite_definition(Suite) || Suite <- available_suites(all)]. %%-------------------------------------------------------------------- -spec getopts(#sslsocket{}, [gen_tcp:option_name()]) -> @@ -584,6 +583,16 @@ format_error(Error) -> %%%-------------------------------------------------------------- %%% Internal functions %%%-------------------------------------------------------------------- + +%% Possible filters out suites not supported by crypto +available_suites(default) -> + Version = tls_record:highest_protocol_version([]), + ssl_cipher:filter_suites(ssl_cipher:suites(Version)); + +available_suites(all) -> + Version = tls_record:highest_protocol_version([]), + ssl_cipher:filter_suites(ssl_cipher:all_suites(Version)). + do_connect(Address, Port, #config{transport_info = CbInfo, inet_user = UserOpts, ssl = SslOpts, emulated = EmOpts, inet_ssl = SocketOpts, connection_cb = ConnetionCb}, diff --git a/lib/ssl/src/tls_handshake.erl b/lib/ssl/src/tls_handshake.erl index 102dbba198..f34eebb0e4 100644 --- a/lib/ssl/src/tls_handshake.erl +++ b/lib/ssl/src/tls_handshake.erl @@ -283,7 +283,8 @@ available_signature_algs(undefined, SupportedHashSigns, _, {Major, Minor}) when SupportedHashSigns; available_signature_algs(#hash_sign_algos{hash_sign_algos = ClientHashSigns}, SupportedHashSigns, _, {Major, Minor}) when (Major >= 3) andalso (Minor >= 3) -> - ordsets:intersection(ClientHashSigns, SupportedHashSigns); + sets:to_list(sets:intersection(sets:from_list(ClientHashSigns), + sets:from_list(SupportedHashSigns))); available_signature_algs(_, _, _, _) -> undefined. diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index 38cc3532d8..02fdc4330f 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -386,11 +386,11 @@ cert_options(Config) -> SNIServerBCertFile = filename:join([?config(priv_dir, Config), "b.server", "cert.pem"]), SNIServerBKeyFile = filename:join([?config(priv_dir, Config), "b.server", "key.pem"]), [{client_opts, []}, - {client_verification_opts, [{cacertfile, ClientCaCertFile}, + {client_verification_opts, [{cacertfile, ServerCaCertFile}, {certfile, ClientCertFile}, {keyfile, ClientKeyFile}, {ssl_imp, new}]}, - {client_verification_opts_digital_signature_only, [{cacertfile, ClientCaCertFile}, + {client_verification_opts_digital_signature_only, [{cacertfile, ServerCaCertFile}, {certfile, ClientCertFileDigitalSignatureOnly}, {keyfile, ClientKeyFile}, {ssl_imp, new}]}, @@ -426,7 +426,7 @@ cert_options(Config) -> {user_lookup_fun, {fun user_lookup/3, undefined}}, {ciphers, srp_anon_suites()}]}, {server_verification_opts, [{ssl_imp, new},{reuseaddr, true}, - {cacertfile, ServerCaCertFile}, + {cacertfile, ClientCaCertFile}, {certfile, ServerCertFile}, {keyfile, ServerKeyFile}]}, {client_kc_opts, [{certfile, ClientKeyCertFile}, {ssl_imp, new}]}, {server_kc_opts, [{ssl_imp, new},{reuseaddr, true}, diff --git a/lib/stdlib/doc/src/epp.xml b/lib/stdlib/doc/src/epp.xml index 0802f0e47c..ac87f9c2b6 100644 --- a/lib/stdlib/doc/src/epp.xml +++ b/lib/stdlib/doc/src/epp.xml @@ -102,6 +102,7 @@ <func> <name name="parse_erl_form" arity="1"/> <fsummary>Return the next Erlang form from the opened Erlang source file</fsummary> + <type name="warning_info"/> <desc> <p>Returns the next Erlang form from the opened Erlang source file. The tuple <c>{eof, <anno>Line</anno>}</c> is returned at end-of-file. The first diff --git a/lib/stdlib/doc/src/gen_statem.xml b/lib/stdlib/doc/src/gen_statem.xml index d83e17b177..0e7d6e53e9 100644 --- a/lib/stdlib/doc/src/gen_statem.xml +++ b/lib/stdlib/doc/src/gen_statem.xml @@ -29,39 +29,49 @@ <rev></rev> </header> <module>gen_statem</module> - <modulesummary>Generic State Machine Behaviour</modulesummary> + <modulesummary>Generic state machine behavior.</modulesummary> <description> <p> - A behaviour module for implementing a state machine. Two + This behavior module provides a state machine. Two <seealso marker="#type-callback_mode"><em>callback modes</em></seealso> - are supported. One for finite state machines - (<seealso marker="gen_fsm"><c>gen_fsm</c></seealso> like) - that requires the state to be an atom and uses that state as - the name of the current callback function, - and one without restriction on the state data type - that uses one callback function for all states. - </p> - <p> - This is a new behaviour in OTP-19.0. - It has been thoroughly reviewed, is stable enough - to be used by at least two heavy OTP applications, and is here to stay. - But depending on user feedback, we do not expect - but might find it necessary to make minor - not backwards compatible changes into OTP-20.0, - so its state can be designated as "not quite experimental"... + are supported: </p> + <list type="bulleted"> + <item> + <p>One for finite-state machines + (<seealso marker="gen_fsm"><c>gen_fsm</c></seealso> like), + which requires the state to be an atom and uses that state as + the name of the current callback function + </p> + </item> + <item> + <p>One without restriction on the state data type + that uses one callback function for all states + </p> + </item> + </list> + <note> + <p> + This is a new behavior in Erlang/OTP 19.0. + It has been thoroughly reviewed, is stable enough + to be used by at least two heavy OTP applications, and is here to stay. + Depending on user feedback, we do not expect + but can find it necessary to make minor + not backward compatible changes into Erlang/OTP 20.0. + </p> + </note> <p> - The <c>gen_statem</c> behaviour is intended to replace + The <c>gen_statem</c> behavior is intended to replace <seealso marker="gen_fsm"><c>gen_fsm</c></seealso> for new code. - It has the same features and add some really useful: + It has the same features and adds some really useful: </p> <list type="bulleted"> - <item>State code is gathered</item> - <item>The state can be any term</item> - <item>Events can be postponed</item> - <item>Events can be self generated</item> - <item>A reply can be sent from a later state</item> - <item>There can be multiple sys traceable replies</item> + <item>State code is gathered.</item> + <item>The state can be any term.</item> + <item>Events can be postponed.</item> + <item>Events can be self-generated.</item> + <item>A reply can be sent from a later state.</item> + <item>There can be multiple <c>sys</c> traceable replies.</item> </list> <p> The callback model(s) for <c>gen_statem</c> differs from @@ -71,19 +81,16 @@ </p> <p> A generic state machine process (<c>gen_statem</c>) implemented - using this module will have a standard set of interface functions - and include functionality for tracing and error reporting. - It will also fit into an OTP supervision tree. Refer to - <seealso marker="doc/design_principles:statem"> - OTP Design Principles - </seealso> - for more information. + using this module has a standard set of interface functions + and includes functionality for tracing and error reporting. + It also fits into an OTP supervision tree. For more information, see + <seealso marker="doc/design_principles:statem">OTP Design Principles</seealso>. </p> <p> A <c>gen_statem</c> assumes all specific parts to be located in a - callback module exporting a pre-defined set of functions. - The relationship between the behaviour functions and the callback - functions can be illustrated as follows:</p> + callback module exporting a predefined set of functions. + The relationship between the behavior functions and the callback + functions is as follows:</p> <pre> gen_statem module Callback module ----------------- --------------- @@ -103,59 +110,54 @@ erlang:'!' -----> Module:StateName/3 - -----> Module:code_change/4</pre> <p> Events are of different - <seealso marker="#type-event_type">types</seealso> + <seealso marker="#type-event_type">types</seealso>, so the callback functions can know the origin of an event and how to respond. </p> <p> If a callback function fails or returns a bad value, - the <c>gen_statem</c> will terminate. An exception of class - <seealso marker="erts:erlang#throw/1"><c>throw</c></seealso>, - however, is not regarded as an error but as a valid return. + the <c>gen_statem</c> terminates. However, an exception of class + <seealso marker="erts:erlang#throw/1"><c>throw</c></seealso> + is not regarded as an error but as a valid return. </p> - <marker id="state_function" /> + <marker id="state_function"/> <p> The "<em>state function</em>" for a specific <seealso marker="#type-state">state</seealso> in a <c>gen_statem</c> is the callback function that is called - for all events in this state, and is selected depending on which + for all events in this state. It is selected depending on which <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> - that the implementation specifies when the the server starts. + that the implementation specifies when the server starts. </p> <p> When the <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> - is <c>state_functions</c>, the state has to be an atom and - is used as the state function name. See - <seealso marker="#Module:StateName/3"> - <c>Module:StateName/3</c> - </seealso>. + is <c>state_functions</c>, the state must be an atom and + is used as the state function name; see + <seealso marker="#Module:StateName/3"><c>Module:StateName/3</c></seealso>. This gathers all code for a specific state - in one function and hence dispatches on state first. - Note that in this mode the fact that there is - a mandatory callback function - <seealso marker="#Module:terminate/3"> - <c>Module:terminate/3</c> - </seealso> makes the state name <c>terminate</c> unusable. + in one function as the <c>gen_statem</c> engine + branches depending on state name. + Notice that in this mode the mandatory callback function + <seealso marker="#Module:terminate/3"><c>Module:terminate/3</c></seealso> + makes the state name <c>terminate</c> unusable. </p> <p> When the <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> - is <c>handle_event_function</c> the state can be any term + is <c>handle_event_function</c>, the state can be any term and the state function name is - <seealso marker="#Module:handle_event/4"> - <c>Module:handle_event/4</c> - </seealso>. - This makes it easy to dispatch on state or on event as you desire. + <seealso marker="#Module:handle_event/4"><c>Module:handle_event/4</c></seealso>. + This makes it easy to branch depending on state or event as you desire. Be careful about which events you handle in which - states so you do not accidentally postpone one event + states so that you do not accidentally postpone an event forever creating an infinite busy loop. </p> <p> The <c>gen_statem</c> enqueues incoming events in order of arrival and presents these to the <seealso marker="#state_function">state function</seealso> - in that order. The state function can postpone an event + in that order. The state function can postpone an event so it is not retried in the current state. After a state change the queue restarts with the postponed events. </p> @@ -163,91 +165,85 @@ erlang:'!' -----> Module:StateName/3 The <c>gen_statem</c> event queue model is sufficient to emulate the normal process message queue with selective receive. Postponing an event corresponds to not matching it - in a receive statement and changing states corresponds + in a receive statement, and changing states corresponds to entering a new receive statement. </p> <p> The <seealso marker="#state_function">state function</seealso> can insert events using the - <seealso marker="#type-action"> - <c>action()</c> <c>next_event</c> - </seealso> + <seealso marker="#type-action"><c>action()</c></seealso> + <c>next_event</c> and such an event is inserted as the next to present - to the state function. That is: as if it is - the oldest incoming event. There is a dedicated - <seealso marker="#type-event_type"> - <c>event_type()</c> - </seealso> - <c>internal</c> that can be used for such events making them impossible + to the state function. That is, as if it is + the oldest incoming event. A dedicated + <seealso marker="#type-event_type"><c>event_type()</c></seealso> + <c>internal</c> can be used for such events making them impossible to mistake for external events. </p> <p> Inserting an event replaces the trick of calling your own state handling functions that you often would have to - resort to in for example <seealso marker="gen_fsm"><c>gen_fsm</c></seealso> + resort to in, for example, + <seealso marker="gen_fsm"><c>gen_fsm</c></seealso> to force processing an inserted event before others. - A warning, though: if you in <c>gen_statem</c> for example - postpone an event in one state and then call some other state function of yours, - you have not changed states and hence the postponed event will not be retried, - which is logical but might be confusing. </p> + <note> + <p>If you in <c>gen_statem</c>, for example, postpone + an event in one state and then call another state function + of yours, you have not changed states and hence the postponed event + is not retried, which is logical but can be confusing. + </p> + </note> <p> - See the type - <seealso marker="#type-transition_option"> - <c>transition_option()</c> - </seealso> - for the details of a state transition. + For the details of a state transition, see type + <seealso marker="#type-transition_option"><c>transition_option()</c></seealso>. </p> <p> - A <c>gen_statem</c> handles system messages as documented in + A <c>gen_statem</c> handles system messages as described in <seealso marker="sys"><c>sys</c></seealso>. - The <c>sys</c>module can be used for debugging a <c>gen_statem</c>. + The <c>sys</c> module can be used for debugging a <c>gen_statem</c>. </p> <p> - Note that a <c>gen_statem</c> does not trap exit signals + Notice that a <c>gen_statem</c> does not trap exit signals automatically, this must be explicitly initiated in the callback module (by calling - <seealso marker="erts:erlang#process_flag/2"> - <c>process_flag(trap_exit, true)</c></seealso>. + <seealso marker="erts:erlang#process_flag/2"><c>process_flag(trap_exit, true)</c></seealso>. </p> <p> Unless otherwise stated, all functions in this module fail if the specified <c>gen_statem</c> does not exist or - if bad arguments are given. + if bad arguments are specified. </p> <p> The <c>gen_statem</c> process can go into hibernation; see - <seealso marker="proc_lib#hibernate/3"> - <c>proc_lib:hibernate/3</c>. - </seealso> + <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>. It is done when a <seealso marker="#state_function">state function</seealso> or <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> specifies <c>hibernate</c> in the returned <seealso marker="#type-action"><c>Actions</c></seealso> - list. This feature might be useful to reclaim process heap memory + list. This feature can be useful to reclaim process heap memory while the server is expected to be idle for a long time. - However, use this feature with care - since hibernation can be too costly + However, use this feature with care, + as hibernation can be too costly to use after every event; see - <seealso marker="erts:erlang#hibernate/3"> - <c>erlang:hibernate/3</c>. - </seealso> + <seealso marker="erts:erlang#hibernate/3"><c>erlang:hibernate/3</c></seealso>. </p> </description> <section> - <title>EXAMPLE</title> + <title>Example</title> <p> - This example shows a simple pushbutton model + The following example shows a simple pushbutton model for a toggling pushbutton implemented with <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> <c>state_functions</c>. You can push the button and it replies if it went on or off, and you can ask for a count of how many times it has been - pushed to on. + pushed to switch on. </p> - <p>This is the complete callback module file <c>pushbutton.erl</c>:</p> + <p>The following is the complete callback module file + <c>pushbutton.erl</c>:</p> <code type="erl"> -module(pushbutton). -behaviour(gen_statem). @@ -305,7 +301,7 @@ handle_event(_, _, Data) -> %% Ignore all other events {keep_state,Data}. </code> - <p>And this is a shell session when running it:</p> + <p>The following is a shell session when running it:</p> <pre> 1> pushbutton:start(). {ok,<0.36.0>} @@ -326,12 +322,11 @@ ok in function gen:do_for_proc/2 (gen.erl, line 261) in call from gen_statem:call/3 (gen_statem.erl, line 386) </pre> - <p> - And just to compare styles here is the same example using + To compare styles, here follows the same example using <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> - <c>state_functions</c>, or rather here is code to replace - from the <c>init/1</c> function of the <c>pushbutton.erl</c> + <c>state_functions</c>, or rather the code to replace + from function <c>init/1</c> of the <c>pushbutton.erl</c> example file above: </p> <code type="erl"> @@ -364,107 +359,108 @@ handle_event(_, _, State, Data) -> <datatypes> <datatype> - <name name="server_name" /> + <name name="server_name"/> <desc> <p> Name specification to use when starting - a <c>gen_statem</c> server. See - <seealso marker="#start_link/3"> - <c>start_link/3</c> - </seealso> + a <c>gen_statem</c> server. See + <seealso marker="#start_link/3"><c>start_link/3</c></seealso> and - <seealso marker="#type-server_ref"> - <c>server_ref()</c> - </seealso> below. + <seealso marker="#type-server_ref"><c>server_ref()</c></seealso> + below. </p> </desc> </datatype> <datatype> - <name name="server_ref" /> + <name name="server_ref"/> <desc> <p> Server specification to use when addressing a <c>gen_statem</c> server. See <seealso marker="#call/2"><c>call/2</c></seealso> and - <seealso marker="#type-server_name"> - <c>server_name()</c> - </seealso> + <seealso marker="#type-server_name"><c>server_name()</c></seealso> above. </p> <p>It can be:</p> <taglist> - <tag><c>pid()</c><br /> - <c>LocalName</c></tag> - <item>The <c>gen_statem</c> is locally registered.</item> + <tag><c>pid() | LocalName</c></tag> + <item> + <p> + The <c>gen_statem</c> is locally registered. + </p> + </item> <tag><c>{Name,Node}</c></tag> <item> - The <c>gen_statem</c> is locally registered - on another node. + <p> + The <c>gen_statem</c> is locally registered + on another node. + </p> </item> <tag><c>{global,GlobalName}</c></tag> <item> - The <c>gen_statem</c> is globally registered - in <seealso marker="kernel:global"><c>global</c></seealso>. + <p> + The <c>gen_statem</c> is globally registered in + <seealso marker="kernel:global"><c>kernel:global</c></seealso>. + </p> </item> <tag><c>{via,RegMod,ViaName}</c></tag> <item> - The <c>gen_statem</c> is registered through - an alternative process registry. - The registry callback module <c>RegMod</c> - should export the functions - <c>register_name/2</c>, <c>unregister_name/1</c>, - <c>whereis_name/1</c> and <c>send/2</c>, - which should behave like the corresponding functions - in <seealso marker="kernel:global"><c>global</c></seealso>. - Thus, <c>{via,global,GlobalName}</c> is the same as - <c>{global,GlobalName}</c>. + <p> + The <c>gen_statem</c> is registered in + an alternative process registry. + The registry callback module <c>RegMod</c> + is to export functions + <c>register_name/2</c>, <c>unregister_name/1</c>, + <c>whereis_name/1</c>, and <c>send/2</c>, + which are to behave like the corresponding functions in + <seealso marker="kernel:global"><c>kernel:global</c></seealso>. + Thus, <c>{via,global,GlobalName}</c> is the same as + <c>{global,GlobalName}</c>. + </p> </item> </taglist> </desc> </datatype> <datatype> - <name name="debug_opt" /> + <name name="debug_opt"/> <desc> <p> Debug option that can be used when starting - a <c>gen_statem</c> server through for example + a <c>gen_statem</c> server through, for example, <seealso marker="#enter_loop/5"><c>enter_loop/5</c></seealso>. </p> <p> - For every entry in <c><anno>Dbgs</anno></c> + For every entry in <c><anno>Dbgs</anno></c>, the corresponding function in - <seealso marker="sys"><c>sys</c></seealso> will be called. + <seealso marker="sys"><c>sys</c></seealso> is called. </p> </desc> </datatype> <datatype> - <name name="start_opt" /> + <name name="start_opt"/> <desc> <p> Options that can be used when starting - a <c>gen_statem</c> server through for example + a <c>gen_statem</c> server through, for example, <seealso marker="#start_link/3"><c>start_link/3</c></seealso>. </p> </desc> </datatype> <datatype> - <name name="start_ret" /> + <name name="start_ret"/> <desc> <p> - Return value from the start functions for_example + Return value from the start functions, for example, <seealso marker="#start_link/3"><c>start_link/3</c></seealso>. </p> </desc> </datatype> - <datatype> - <name name="from" /> + <name name="from"/> <desc> <p> - Destination to use when replying through for example the - <seealso marker="#type-action"> - <c>action()</c> - </seealso> + Destination to use when replying through, for example, the + <seealso marker="#type-action"><c>action()</c></seealso> <c>{reply,From,Reply}</c> to a process that has called the <c>gen_statem</c> server using <seealso marker="#call/2"><c>call/2</c></seealso>. @@ -472,231 +468,228 @@ handle_event(_, _, State, Data) -> </desc> </datatype> <datatype> - <name name="state" /> + <name name="state"/> <desc> <p> - After a state change (<c>NextState =/= State</c>) + After a state change (<c>NextState =/= State</c>), all postponed events are retried. </p> </desc> </datatype> <datatype> - <name name="state_name" /> + <name name="state_name"/> <desc> <p> If the <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> is <c>state_functions</c>, - the state has to be of this type. + the state must be of this type. </p> </desc> </datatype> <datatype> - <name name="data" /> + <name name="data"/> <desc> <p> A term in which the state machine implementation - should store any server data it needs. The difference between + is to store any server data it needs. The difference between this and the <seealso marker="#type-state"><c>state()</c></seealso> itself is that a change in this data does not cause - postponed events to be retried. Hence if a change + postponed events to be retried. Hence, if a change in this data would change the set of events that - are handled than that data item should be made + are handled, then that data item is to be made a part of the state. </p> </desc> </datatype> <datatype> - <name name="event_type" /> + <name name="event_type"/> <desc> <p> - External events are of 3 different type: - <c>{call,<anno>From</anno>}</c>, <c>cast</c> or <c>info</c>. + External events are of three types: + <c>{call,<anno>From</anno>}</c>, <c>cast</c>, or <c>info</c>. <seealso marker="#call/2">Calls</seealso> (synchronous) and <seealso marker="#cast/2">casts</seealso> originate from the corresponding API functions. - For calls the event contain whom to reply to. + For calls, the event contains whom to reply to. Type <c>info</c> originates from regular process messages sent - to the <c>gen_statem</c>. It is also possible for the state machine - implementation to generate events of types + to the <c>gen_statem</c>. Also, the state machine + implementation can generate events of types <c>timeout</c> and <c>internal</c> to itself. </p> </desc> </datatype> <datatype> - <name name="callback_mode" /> + <name name="callback_mode"/> <desc> <p> The <em>callback mode</em> is selected when starting the <c>gen_statem</c> using the return value from <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> or when calling - <seealso marker="#enter_loop/5"><c>enter_loop/5-7</c></seealso>, + <seealso marker="#enter_loop/5"><c>enter_loop/5,6,7</c></seealso>, and with the return value from - <seealso marker="#Module:code_change/4"> - <c>Module:code_change/4</c>. - </seealso> + <seealso marker="#Module:code_change/4"><c>Module:code_change/4</c></seealso>. </p> <taglist> <tag><c>state_functions</c></tag> <item> - The state has to be of type - <seealso marker="#type-state_name"><c>state_name()</c></seealso> - and one callback function per state that is - <seealso marker="#Module:StateName/3"> - <c>Module:StateName/3</c> - </seealso> - is used. + <p> + The state must be of type + <seealso marker="#type-state_name"><c>state_name()</c></seealso> + and one callback function per state, that is, + <seealso marker="#Module:StateName/3"><c>Module:StateName/3</c></seealso>, + is used. + </p> </item> <tag><c>handle_event_function</c></tag> <item> - The state can be any term and the callback function - <seealso marker="#Module:handle_event/4"> - <c>Module:handle_event/4</c> - </seealso> - is used for all states. + <p> + The state can be any term and the callback function + <seealso marker="#Module:handle_event/4"><c>Module:handle_event/4</c></seealso> + is used for all states. + </p> </item> </taglist> </desc> </datatype> <datatype> - <name name="transition_option" /> + <name name="transition_option"/> <desc> <p> - Transition options may be set by + Transition options can be set by <seealso marker="#type-action">actions</seealso> - and they modify some details below in how + and they modify the following in how the state transition is done: </p> <list type="ordered"> <item> - All - <seealso marker="#type-action">actions</seealso> - are processed in order of appearance. + <p> + All + <seealso marker="#type-action">actions</seealso> + are processed in order of appearance. + </p> </item> <item> - If - <seealso marker="#type-postpone"> - <c>postpone()</c> - </seealso> - is <c>true</c> - the current event is postponed. + <p> + If + <seealso marker="#type-postpone"><c>postpone()</c></seealso> + is <c>true</c>, + the current event is postponed. + </p> </item> <item> - If the state changes the queue of incoming events - is reset to start with the oldest postponed. + <p> + If the state changes, the queue of incoming events + is reset to start with the oldest postponed. + </p> </item> <item> - All events stored with - <seealso marker="#type-action"> - <c>action()</c> - </seealso> - <c>next_event</c> - are inserted in the queue to be processed before - all other events. + <p> + All events stored with + <seealso marker="#type-action"><c>action()</c></seealso> + <c>next_event</c> + are inserted in the queue to be processed before + all other events. + </p> </item> <item> - If an - <seealso marker="#type-event_timeout"> - <c>event_timeout()</c> - </seealso> - is set through - <seealso marker="#type-action"> - <c>action()</c> - </seealso> - <c>timeout</c> - an event timer may be started or a timeout zero event - may be enqueued. + <p> + If an + <seealso marker="#type-event_timeout"><c>event_timeout()</c></seealso> + is set through + <seealso marker="#type-action"><c>action()</c></seealso> + <c>timeout</c>, + an event timer can be started or a time-out zero event + can be enqueued. + </p> </item> <item> - The (possibly new) - <seealso marker="#state_function">state function</seealso> - is called with the oldest enqueued event if there is any, - otherwise the <c>gen_statem</c> goes into <c>receive</c> - or hibernation - (if - <seealso marker="#type-hibernate"> - <c>hibernate()</c> - </seealso> - is <c>true</c>) - to wait for the next message. In hibernation the next - non-system event awakens the <c>gen_statem</c>, or rather - the next incoming message awakens the <c>gen_statem</c> - but if it is a system event - it goes right back into hibernation. + <p> + The (possibly new) + <seealso marker="#state_function">state function</seealso> + is called with the oldest enqueued event if there is any, + otherwise the <c>gen_statem</c> goes into <c>receive</c> + or hibernation + (if + <seealso marker="#type-hibernate"><c>hibernate()</c></seealso> + is <c>true</c>) + to wait for the next message. In hibernation the next + non-system event awakens the <c>gen_statem</c>, or rather + the next incoming message awakens the <c>gen_statem</c>, + but if it is a system event + it goes right back into hibernation. + </p> </item> </list> </desc> </datatype> <datatype> - <name name="postpone" /> + <name name="postpone"/> <desc> <p> - If <c>true</c> postpone the current event and retry + If <c>true</c>, postpones the current event and retries it when the state changes (<c>NextState =/= State</c>). </p> </desc> </datatype> <datatype> - <name name="hibernate" /> + <name name="hibernate"/> <desc> <p> - If <c>true</c> hibernate the <c>gen_statem</c> + If <c>true</c>, hibernates the <c>gen_statem</c> by calling - <seealso marker="proc_lib#hibernate/3"> - <c>proc_lib:hibernate/3</c> - </seealso> + <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso> before going into <c>receive</c> to wait for a new external event. If there are enqueued events, - to prevent receiving any new event; a - <seealso marker="erts:erlang#garbage_collect/0"> - <c>garbage_collect/0</c> - </seealso> is done instead to simulate + to prevent receiving any new event, an + <seealso marker="erts:erlang#garbage_collect/0"><c>erlang:garbage_collect/0</c></seealso> + is done instead to simulate that the <c>gen_statem</c> entered hibernation and immediately got awakened by the oldest enqueued event. </p> </desc> </datatype> <datatype> - <name name="event_timeout" /> + <name name="event_timeout"/> <desc> <p> - Generate an event of + Generates an event of <seealso marker="#type-event_type"><c>event_type()</c></seealso> <c>timeout</c> - after this time (in milliseconds) unless some other - event arrives in which case this timeout is cancelled. - Note that a retried or inserted event - counts just like a new in this respect. + after this time (in milliseconds) unless another + event arrives in which case this time-out is cancelled. + Notice that a retried or inserted event + counts like a new in this respect. </p> <p> - If the value is <c>infinity</c> no timer is started since - it will never trigger anyway. + If the value is <c>infinity</c>, no timer is started, as + it never triggers anyway. </p> <p> - If the value is <c>0</c> the timeout event is immediately enqueued - unless there already are enqueued events since then the - timeout is immediately cancelled. - This is a feature ensuring that a timeout <c>0</c> event - will be processed before any not yet received external event. + If the value is <c>0</c>, the time-out event is immediately enqueued + unless there already are enqueued events, as the + time-out is then immediately cancelled. + This is a feature ensuring that a time-out <c>0</c> event + is processed before any not yet received external event. </p> <p> - Note that it is not possible nor needed to cancel this timeout - since it is cancelled automatically by any other event. + Notice that it is not possible or needed to cancel this time-out, + as it is cancelled automatically by any other event. </p> </desc> </datatype> <datatype> - <name name="action" /> + <name name="action"/> <desc> <p> - These state transition actions may be invoked by + These state transition actions can be invoked by returning them from the - <seealso marker="#state_function">state function</seealso>, - from <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> + <seealso marker="#state_function">state function</seealso>, from + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> or by giving them to <seealso marker="#enter_loop/6"><c>enter_loop/6,7</c></seealso>. </p> @@ -705,91 +698,81 @@ handle_event(_, _, State, Data) -> </p> <p> Actions that set - <seealso marker="#type-transition_option"> - transition options - </seealso> - overrides any previous of the same type, + <seealso marker="#type-transition_option">transition options</seealso> + override any previous of the same type, so the last in the containing list wins. - For example the last - <seealso marker="#type-event_timeout"> - <c>event_timeout()</c> - </seealso> + For example, the last + <seealso marker="#type-event_timeout"><c>event_timeout()</c></seealso> overrides any other <c>event_timeout()</c> in the list. </p> <taglist> <tag><c>postpone</c></tag> <item> - Set the - <seealso marker="#type-transition_option"> - <c>transition_option()</c> - </seealso> - <seealso marker="#type-postpone"> - <c>postpone()</c> - </seealso> - for this state transition. - This action is ignored when returned from - <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> - or given to - <seealso marker="#enter_loop/5"><c>enter_loop/5,6</c></seealso> - since there is no event to postpone in those cases. + <p> + Sets the + <seealso marker="#type-transition_option"><c>transition_option()</c></seealso> + <seealso marker="#type-postpone"><c>postpone()</c></seealso> + for this state transition. + This action is ignored when returned from + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> + or given to + <seealso marker="#enter_loop/5"><c>enter_loop/5,6</c></seealso>, + as there is no event to postpone in those cases. + </p> </item> <tag><c>hibernate</c></tag> <item> - Set the - <seealso marker="#type-transition_option"> - <c>transition_option()</c> - </seealso> - <seealso marker="#type-hibernate"> - <c>hibernate()</c> - </seealso> - for this state transition. + <p> + Sets the + <seealso marker="#type-transition_option"><c>transition_option()</c></seealso> + <seealso marker="#type-hibernate"><c>hibernate()</c></seealso> + for this state transition. + </p> </item> <tag><c>Timeout</c></tag> <item> - Short for <c>{timeout,Timeout,Timeout}</c> that is - the timeout message is the timeout time. - This form exists to make the - <seealso marker="#state_function">state function</seealso> - return value <c>{next_state,NextState,NewData,Timeout}</c> - allowed like for - <seealso marker="gen_fsm#Module:StateName/2"> - <c>gen_fsm Module:StateName/2</c>. - </seealso> + <p> + Short for <c>{timeout,Timeout,Timeout}</c>, that is, + the time-out message is the time-out time. + This form exists to make the + <seealso marker="#state_function">state function</seealso> + return value <c>{next_state,NextState,NewData,Timeout}</c> + allowed like for <c>gen_fsm</c>'s + <seealso marker="gen_fsm#Module:StateName/2"><c>Module:StateName/2</c></seealso>. + </p> </item> <tag><c>timeout</c></tag> <item> - Set the - <seealso marker="#type-transition_option"> - <c>transition_option()</c> - </seealso> - <seealso marker="#type-event_timeout"> - <c>event_timeout()</c> - </seealso> - to <c><anno>Time</anno></c> with <c><anno>EventContent</anno></c>. + <p> + Sets the + <seealso marker="#type-transition_option"><c>transition_option()</c></seealso> + <seealso marker="#type-event_timeout"><c>event_timeout()</c></seealso> + to <c><anno>Time</anno></c> with <c><anno>EventContent</anno></c>. + </p> </item> <tag><c>reply_action()</c></tag> - <item>Reply to a caller.</item> + <item> + <p> + Replies to a caller. + </p> + </item> <tag><c>next_event</c></tag> <item> - Store the given <c><anno>EventType</anno></c> + <p> + Stores the specified <c><anno>EventType</anno></c> and <c><anno>EventContent</anno></c> for insertion after all actions have been executed. - </item> - <item> + </p> <p> The stored events are inserted in the queue as the next to process - before any already queued events. The order of these stored events - is preserved so the first <c>next_event</c> in the containing list - will become the first to process. + before any already queued events. The order of these stored events + is preserved, so the first <c>next_event</c> in the containing + list becomes the first to process. </p> - </item> - <item> <p> An event of type - <seealso marker="#type-event_type"> - <c>internal</c> - </seealso> - should be used when you want to reliably distinguish + <seealso marker="#type-event_type"><c>internal</c></seealso> + is to be used when you want to reliably distinguish an event inserted this way from any external event. </p> </item> @@ -797,303 +780,126 @@ handle_event(_, _, State, Data) -> </desc> </datatype> <datatype> - <name name="reply_action" /> + <name name="reply_action"/> <desc> <p> - Reply to a caller waiting for a reply in + Replies to a caller waiting for a reply in <seealso marker="#call/2"><c>call/2</c></seealso>. - <c><anno>From</anno></c> must be the term from the - <seealso marker="#type-event_type"> - <c>{call,<anno>From</anno>}</c> - </seealso> - argument to the + <c><anno>From</anno></c> must be the term from argument + <seealso marker="#type-event_type"><c>{call,<anno>From</anno>}</c></seealso> + to the <seealso marker="#state_function">state function</seealso>. </p> </desc> </datatype> <datatype> - <name name="state_function_result" /> + <name name="state_function_result"/> <desc> <taglist> <tag><c>next_state</c></tag> <item> - The <c>gen_statem</c> will do a state transition to - <c><anno>NextStateName</anno></c> - (which may be the same as the current state), - set <c><anno>NewData</anno></c> - and execute all <c><anno>Actions</anno></c> + <p> + The <c>gen_statem</c> does a state transition to + <c><anno>NextStateName</anno></c> + (which can be the same as the current state), + sets <c><anno>NewData</anno></c>, + and executes all <c><anno>Actions</anno></c>. + </p> </item> </taglist> <p> All these terms are tuples or atoms and this property - will hold in any future version of <c>gen_statem</c>, - just in case you need such a promise. + will hold in any future version of <c>gen_statem</c>. </p> </desc> </datatype> <datatype> - <name name="handle_event_result" /> + <name name="handle_event_result"/> <desc> <taglist> <tag><c>next_state</c></tag> <item> - The <c>gen_statem</c> will do a state transition to - <c><anno>NextState</anno></c> - (which may be the same as the current state), - set <c><anno>NewData</anno></c> - and execute all <c><anno>Actions</anno></c> + <p> + The <c>gen_statem</c> does a state transition to + <c><anno>NextState</anno></c> + (which can be the same as the current state), + sets <c><anno>NewData</anno></c>, + and executes all <c><anno>Actions</anno></c>. + </p> </item> </taglist> <p> All these terms are tuples or atoms and this property - will hold in any future version of <c>gen_statem</c>, - just in case you need such a promise. + will hold in any future version of <c>gen_statem</c>. </p> </desc> </datatype> <datatype> - <name name="common_state_callback_result" /> + <name name="common_state_callback_result"/> <desc> <taglist> <tag><c>stop</c></tag> <item> - Terminate the <c>gen_statem</c> by calling - <seealso marker="#Module:terminate/3"> - <c>Module:terminate/3</c> - </seealso> - with <c>Reason</c> and - <c><anno>NewData</anno></c>, if given. + <p> + Terminates the <c>gen_statem</c> by calling + <seealso marker="#Module:terminate/3"><c>Module:terminate/3</c></seealso> + with <c>Reason</c> and + <c><anno>NewData</anno></c>, if specified. + </p> </item> <tag><c>stop_and_reply</c></tag> <item> - Send all <c><anno>Replies</anno></c> - then terminate the <c>gen_statem</c> by calling - <seealso marker="#Module:terminate/3"> - <c>Module:terminate/3</c> - </seealso> - with <c>Reason</c> and - <c><anno>NewData</anno></c>, if given. + <p> + Sends all <c><anno>Replies</anno></c>, + then terminates the <c>gen_statem</c> by calling + <seealso marker="#Module:terminate/3"><c>Module:terminate/3</c></seealso> + with <c>Reason</c> and + <c><anno>NewData</anno></c>, if specified. + </p> </item> <tag><c>keep_state</c></tag> <item> - The <c>gen_statem</c> will keep the current state, or - do a state transition to the current state if you like, - set <c><anno>NewData</anno></c> - and execute all <c><anno>Actions</anno></c>. - This is the same as - <c>{next_state,CurrentState,<anno>NewData</anno>,<anno>Actions</anno>}</c>. + <p> + The <c>gen_statem</c> keeps the current state, or + does a state transition to the current state if you like, + sets <c><anno>NewData</anno></c>, + and executes all <c><anno>Actions</anno></c>. + This is the same as + <c>{next_state,CurrentState,<anno>NewData</anno>,<anno>Actions</anno>}</c>. + </p> </item> <tag><c>keep_state_and_data</c></tag> <item> - The <c>gen_statem</c> will keep the current state or - do a state transition to the current state if you like, - keep the current server data, - and execute all <c><anno>Actions</anno></c>. - This is the same as - <c>{next_state,CurrentState,CurrentData,<anno>Actions</anno>}</c>. + <p> + The <c>gen_statem</c> keeps the current state or + does a state transition to the current state if you like, + keeps the current server data, + and executes all <c><anno>Actions</anno></c>. + This is the same as + <c>{next_state,CurrentState,CurrentData,<anno>Actions</anno>}</c>. + </p> </item> </taglist> <p> All these terms are tuples or atoms and this property - will hold in any future version of <c>gen_statem</c>, - just in case you need such a promise. + will hold in any future version of <c>gen_statem</c>. </p> </desc> </datatype> </datatypes> <funcs> - - <func> - <name name="start_link" arity="3" /> - <name name="start_link" arity="4" /> - <fsummary>Create a linked <c>gen_statem</c> process</fsummary> - <desc> - <p> - Creates a <c>gen_statem</c> process according - to OTP design principles - (using - <seealso marker="proc_lib"><c>proc_lib</c></seealso> - primitives) - that is linked to the calling process. - This is essential when the <c>gen_statem</c> shall be part of - a supervision tree so it gets linked to its supervisor. - </p> - <p> - The <c>gen_statem</c> process calls - <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> - to initialize the server. To ensure a synchronized start-up - procedure, <c>start_link/3,4</c> does not return until - <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> - has returned. - </p> - <p> - <c><anno>ServerName</anno></c> specifies the - <seealso marker="#type-server_name"> - <c>server_name()</c> - </seealso> - to register for the <c>gen_statem</c>. - If the <c>gen_statem</c> is started with <c>start_link/3</c> - no <c><anno>ServerName</anno></c> is provided and - the <c>gen_statem</c> is not registered. - </p> - <p><c><anno>Module</anno></c> is the name of the callback module.</p> - <p> - <c><anno>Args</anno></c> is an arbitrary term which is passed as - the argument to - <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso>. - </p> - <p> - If the option <c>{timeout,Time}</c> is present in - <c><anno>Opts</anno></c>, the <c>gen_statem</c> - is allowed to spend <c>Time</c> milliseconds initializing - or it will be terminated and the start function will return - <seealso marker="#type-start_ret"><c>{error,timeout}</c></seealso>. - </p> - <p> - If the option - <seealso marker="#type-debug_opt"><c>{debug,Dbgs}</c></seealso> - is present in <c><anno>Opts</anno></c>, debugging through - <seealso marker="sys"><c>sys</c></seealso> is activated. - </p> - <p> - If the option <c>{spawn_opt,SpawnOpts}</c> is present in - <c><anno>Opts</anno></c>, <c>SpawnOpts</c> will be passed - as option list to - <seealso marker="erts:erlang#spawn_opt/2"><c>spawn_opt/2</c></seealso> - which is used to spawn the <c>gen_statem</c> process. - </p> - <note> - <p> - Using the spawn option <c>monitor</c> is currently not - allowed, but will cause this function to fail with reason - <c>badarg</c>. - </p> - </note> - <p> - If the <c>gen_statem</c> is successfully created - and initialized this function returns - <seealso marker="#type-start_ret"> - <c>{ok,Pid}</c>, - </seealso> - where <c>Pid</c> is the <c>pid()</c> - of the <c>gen_statem</c>. - If there already exists a process with the specified - <c><anno>ServerName</anno></c> this function returns - <seealso marker="#type-start_ret"><c>{error,{already_started,Pid}}</c></seealso>, - where <c>Pid</c> is the <c>pid()</c> of that process. - </p> - <p> - If <c>Module:init/1</c> fails with <c>Reason</c>, - this function returns - <seealso marker="#type-start_ret"><c>{error,Reason}</c></seealso>. - If <c>Module:init/1</c> returns - <seealso marker="#type-start_ret"> - <c>{stop,Reason}</c> - </seealso> - or - <seealso marker="#type-start_ret"><c>ignore</c></seealso>, - the process is terminated and this function - returns - <seealso marker="#type-start_ret"> - <c>{error,Reason}</c> - </seealso> - or - <seealso marker="#type-start_ret"><c>ignore</c></seealso>, - respectively. - </p> - </desc> - </func> - - - <func> - <name name="start" arity="3" /> - <name name="start" arity="4" /> - <fsummary>Create a stand-alone <c>gen_statem</c> process</fsummary> - <desc> - <p> - Creates a stand-alone <c>gen_statem</c> process according to - OTP design principles (using - <seealso marker="proc_lib"><c>proc_lib</c></seealso> - primitives). - Since it does not get linked to the calling process - this start function can not be used by a supervisor - to start a child. - </p> - <p> - See <seealso marker="#start_link/3"><c>start_link/3,4</c></seealso> - for a description of arguments and return values. - </p> - </desc> - </func> - - <func> - <name name="stop" arity="1" /> - <fsummary>Synchronously stop a generic server</fsummary> - <desc> - <p> - The same as - <seealso marker="#stop/3"> - <c>stop(<anno>ServerRef</anno>, normal, infinity)</c>. - </seealso> - </p> - </desc> - </func> - <func> - <name name="stop" arity="3" /> - <fsummary>Synchronously stop a generic server</fsummary> - <desc> - <p> - Orders the <c>gen_statem</c> - <seealso marker="#type-server_ref"> - <c><anno>ServerRef</anno></c> - </seealso> - to exit with the given <c><anno>Reason</anno></c> - and waits for it to terminate. - The <c>gen_statem</c> will call - <seealso marker="#Module:terminate/3"> - <c>Module:terminate/3</c> - </seealso> - before exiting. - </p> - <p> - This function returns <c>ok</c> if the server terminates - with the expected reason. Any other reason than <c>normal</c>, - <c>shutdown</c>, or <c>{shutdown,Term}</c> will cause an - error report to be issued through - <seealso marker="kernel:error_logger#format/2"> - <c>error_logger:format/2</c>. - </seealso> - The default <c><anno>Reason</anno></c> is <c>normal</c>. - </p> - <p> - <c><anno>Timeout</anno></c> is an integer greater than zero - which specifies how many milliseconds to wait for the server to - terminate, or the atom <c>infinity</c> to wait indefinitely. - The default value is <c>infinity</c>. - If the server has not terminated within the specified time, - a <c>timeout</c> exception is raised. - </p> - <p> - If the process does not exist, a <c>noproc</c> exception - is raised. - </p> - </desc> - </func> - <func> - <name name="call" arity="2" /> - <name name="call" arity="3" /> - <fsummary>Make a synchronous call to a <c>gen_statem</c></fsummary> + <name name="call" arity="2"/> + <name name="call" arity="3"/> + <fsummary>Make a synchronous call to a <c>gen_statem</c>.</fsummary> <desc> <p> Makes a synchronous call to the <c>gen_statem</c> - <seealso marker="#type-server_ref"> - <c><anno>ServerRef</anno></c> - </seealso> + <seealso marker="#type-server_ref"><c><anno>ServerRef</anno></c></seealso> by sending a request and waiting until its reply arrives. - The <c>gen_statem</c> will call the + The <c>gen_statem</c> calls the <seealso marker="#state_function">state function</seealso> with <seealso marker="#type-event_type"><c>event_type()</c></seealso> <c>{call,From}</c> and event content @@ -1108,43 +914,41 @@ handle_event(_, _, State, Data) -> and that <c><anno>Reply</anno></c> becomes the return value of this function. </p> - <p> - <c><anno>Timeout</anno></c> is an integer greater than zero + <p> + <c><anno>Timeout</anno></c> is an integer > 0, which specifies how many milliseconds to wait for a reply, or the atom <c>infinity</c> to wait indefinitely, - which is the default. If no reply is received within + which is the default. If no reply is received within the specified time, the function call fails. </p> <note> <p> To avoid getting a late reply in the caller's - inbox this function spawns a proxy process that - does the call. A late reply gets delivered to the - dead proxy process hence gets discarded. This is + inbox, this function spawns a proxy process that + does the call. A late reply gets delivered to the + dead proxy process, hence gets discarded. This is less efficient than using <c><anno>Timeout</anno> =:= infinity</c>. </p> </note> - <p> - The call may fail for example if the <c>gen_statem</c> dies + <p> + The call can fail, for example, if the <c>gen_statem</c> dies before or during this function call. </p> </desc> </func> <func> - <name name="cast" arity="2" /> - <fsummary>Send an asynchronous event to a <c>gen_statem</c></fsummary> + <name name="cast" arity="2"/> + <fsummary>Send an asynchronous event to a <c>gen_statem</c>.</fsummary> <desc> <p> Sends an asynchronous event to the <c>gen_statem</c> - <seealso marker="#type-server_ref"> - <c><anno>ServerRef</anno></c> - </seealso> + <seealso marker="#type-server_ref"><c><anno>ServerRef</anno></c></seealso> and returns <c>ok</c> immediately, ignoring if the destination node or <c>gen_statem</c> does not exist. - The <c>gen_statem</c> will call the + The <c>gen_statem</c> calls the <seealso marker="#state_function">state function</seealso> with <seealso marker="#type-event_type"><c>event_type()</c></seealso> <c>cast</c> and event content @@ -1154,68 +958,29 @@ handle_event(_, _, State, Data) -> </func> <func> - <name name="reply" arity="1" /> - <name name="reply" arity="2" /> - <fsummary>Reply to a caller</fsummary> - <desc> - <p> - This function can be used by a <c>gen_statem</c> - to explicitly send a reply to a process that waits in - <seealso marker="#call/2"><c>call/2</c></seealso> - when the reply cannot be defined in - the return value of a - <seealso marker="#state_function">state function</seealso>. - </p> - <p> - <c><anno>From</anno></c> must be the term from the - <seealso marker="#type-event_type"> - <c>{call,<anno>From</anno>}</c> - </seealso> - argument to the - <seealso marker="#state_function">state function</seealso>. - <c><anno>From</anno></c> and <c><anno>Reply</anno></c> - can also be specified using a - <seealso marker="#type-reply_action"> - <c>reply_action()</c> - </seealso> - and multiple replies with a list of them. - </p> - <note> - <p> - A reply sent with this function will not be visible - in <seealso marker="sys"><c>sys</c></seealso> debug output. - </p> - </note> - </desc> - </func> - - <func> - <name name="enter_loop" arity="5" /> - <fsummary>Enter the <c>gen_statem</c> receive loop</fsummary> + <name name="enter_loop" arity="5"/> + <fsummary>Enter the <c>gen_statem</c> receive loop.</fsummary> <desc> <p> The same as <seealso marker="#enter_loop/7"><c>enter_loop/7</c></seealso> except that no - <seealso marker="#type-server_name"> - <c>server_name()</c> - </seealso> + <seealso marker="#type-server_name"><c>server_name()</c></seealso> must have been registered. </p> </desc> </func> + <func> - <name name="enter_loop" arity="6" /> - <fsummary>Enter the <c>gen_statem</c> receive loop</fsummary> + <name name="enter_loop" arity="6"/> + <fsummary>Enter the <c>gen_statem</c> receive loop.</fsummary> <desc> <p> - If <c><anno>Server_or_Actions</anno></c> is a <c>list()</c> + If <c><anno>Server_or_Actions</anno></c> is a <c>list()</c>, the same as <seealso marker="#enter_loop/7"><c>enter_loop/7</c></seealso> except that no - <seealso marker="#type-server_name"> - <c>server_name()</c> - </seealso> + <seealso marker="#type-server_name"><c>server_name()</c></seealso> must have been registered and <c>Actions = <anno>Server_or_Actions</anno></c>. </p> @@ -1228,73 +993,350 @@ handle_event(_, _, State, Data) -> </p> </desc> </func> + <func> - <name name="enter_loop" arity="7" /> - <fsummary>Enter the <c>gen_statem</c> receive loop</fsummary> + <name name="enter_loop" arity="7"/> + <fsummary>Enter the <c>gen_statem</c> receive loop.</fsummary> <desc> - <p> - Makes an the calling process become a <c>gen_statem</c>. - Does not return, instead the calling process will enter - the <c>gen_statem</c> receive loop and become + <p> + Makes the calling process become a <c>gen_statem</c>. + Does not return, instead the calling process enters + the <c>gen_statem</c> receive loop and becomes a <c>gen_statem</c> server. The process <em>must</em> have been started using one of the start functions in - <seealso marker="proc_lib"><c>proc_lib</c></seealso>. + <seealso marker="proc_lib"><c>proc_lib</c></seealso>. The user is responsible for any initialization of the process, including registering a name for it. </p> - <p> + <p> This function is useful when a more complex initialization - procedure is needed than - the <c>gen_statem</c> behaviour provides. + procedure is needed than + the <c>gen_statem</c> behavior provides. </p> - <p> - <c><anno>Module</anno></c>, <c><anno>Opts</anno></c> and + <p> + <c><anno>Module</anno></c>, <c><anno>Opts</anno></c>, and <c><anno>Server</anno></c> have the same meanings as when calling - <seealso marker="#start_link/3"> - <c>gen_statem:start[_link]/3,4</c>. - </seealso> - However, the - <seealso marker="#type-server_name"> - <c>server_name()</c> - </seealso> + <seealso marker="#start_link/3"><c>start[_link]/3,4</c></seealso>. + However, the + <seealso marker="#type-server_name"><c>server_name()</c></seealso> name must have been registered accordingly <em>before</em> this function is called.</p> <p> <c><anno>CallbackMode</anno></c>, <c><anno>State</anno></c>, - <c><anno>Data</anno></c> and <c><anno>Actions</anno></c> + <c><anno>Data</anno></c>, and <c><anno>Actions</anno></c> have the same meanings as in the return value of - <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso>. - Also, the callback module <c><anno>Module</anno></c> + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso>. + Also, the callback module <c><anno>Module</anno></c> does not need to export an <c>init/1</c> function. </p> <p> - Failure: If the calling process was not started by a - <seealso marker="proc_lib"><c>proc_lib</c></seealso> + The function fails if the calling process was not started by a + <seealso marker="proc_lib"><c>proc_lib</c></seealso> start function, or if it is not registered - according to + according to <seealso marker="#type-server_name"><c>server_name()</c></seealso>. </p> </desc> </func> - </funcs> + <func> + <name name="reply" arity="1"/> + <name name="reply" arity="2"/> + <fsummary>Reply to a caller.</fsummary> + <desc> + <p> + This function can be used by a <c>gen_statem</c> + to explicitly send a reply to a process that waits in + <seealso marker="#call/2"><c>call/2</c></seealso> + when the reply cannot be defined in + the return value of a + <seealso marker="#state_function">state function</seealso>. + </p> + <p> + <c><anno>From</anno></c> must be the term from argument + <seealso marker="#type-event_type"><c>{call,<anno>From</anno>}</c></seealso> + to the + <seealso marker="#state_function">state function</seealso>. + <c><anno>From</anno></c> and <c><anno>Reply</anno></c> + can also be specified using a + <seealso marker="#type-reply_action"><c>reply_action()</c></seealso> + and multiple replies with a list of them. + </p> + <note> + <p> + A reply sent with this function is not visible + in <seealso marker="sys"><c>sys</c></seealso> debug output. + </p> + </note> + </desc> + </func> + <func> + <name name="start" arity="3"/> + <name name="start" arity="4"/> + <fsummary>Create a standalone <c>gen_statem</c> process.</fsummary> + <desc> + <p> + Creates a standalone <c>gen_statem</c> process according to + OTP design principles (using + <seealso marker="proc_lib"><c>proc_lib</c></seealso> + primitives). + As it does not get linked to the calling process, + this start function cannot be used by a supervisor + to start a child. + </p> + <p> + For a description of arguments and return values, see + <seealso marker="#start_link/3"><c>start_link/3,4</c></seealso>. + </p> + </desc> + </func> + <func> + <name name="start_link" arity="3"/> + <name name="start_link" arity="4"/> + <fsummary>Create a linked <c>gen_statem</c> process.</fsummary> + <desc> + <p> + Creates a <c>gen_statem</c> process according + to OTP design principles + (using + <seealso marker="proc_lib"><c>proc_lib</c></seealso> + primitives) + that is linked to the calling process. + This is essential when the <c>gen_statem</c> must be part of + a supervision tree so it gets linked to its supervisor. + </p> + <p> + The <c>gen_statem</c> process calls + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> + to initialize the server. To ensure a synchronized startup + procedure, <c>start_link/3,4</c> does not return until + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> + has returned. + </p> + <p> + <c><anno>ServerName</anno></c> specifies the + <seealso marker="#type-server_name"><c>server_name()</c></seealso> + to register for the <c>gen_statem</c>. + If the <c>gen_statem</c> is started with <c>start_link/3</c>, + no <c><anno>ServerName</anno></c> is provided and + the <c>gen_statem</c> is not registered. + </p> + <p><c><anno>Module</anno></c> is the name of the callback module.</p> + <p> + <c><anno>Args</anno></c> is an arbitrary term that is passed as + the argument to + <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso>. + </p> + <list type="bulleted"> + <item> + <p> + If option <c>{timeout,Time}</c> is present in + <c><anno>Opts</anno></c>, the <c>gen_statem</c> + is allowed to spend <c>Time</c> milliseconds initializing + or it terminates and the start function returns + <seealso marker="#type-start_ret"><c>{error,timeout}</c></seealso>. + </p> + </item> + <item> + <p> + If option + <seealso marker="#type-debug_opt"><c>{debug,Dbgs}</c></seealso> + is present in <c><anno>Opts</anno></c>, debugging through + <seealso marker="sys"><c>sys</c></seealso> is activated. + </p> + </item> + <item> + <p> + If option <c>{spawn_opt,SpawnOpts}</c> is present in + <c><anno>Opts</anno></c>, <c>SpawnOpts</c> is passed + as option list to + <seealso marker="erts:erlang#spawn_opt/2"><c>erlang:spawn_opt/2</c></seealso>, + which is used to spawn the <c>gen_statem</c> process. + </p> + </item> + </list> + <note> + <p> + Using spawn option <c>monitor</c> is not + allowed, it causes this function to fail with reason + <c>badarg</c>. + </p> + </note> + <p> + If the <c>gen_statem</c> is successfully created + and initialized, this function returns + <seealso marker="#type-start_ret"><c>{ok,Pid}</c></seealso>, + where <c>Pid</c> is the <c>pid()</c> + of the <c>gen_statem</c>. + If a process with the specified <c><anno>ServerName</anno></c> + exists already, this function returns + <seealso marker="#type-start_ret"><c>{error,{already_started,Pid}}</c></seealso>, + where <c>Pid</c> is the <c>pid()</c> of that process. + </p> + <p> + If <c>Module:init/1</c> fails with <c>Reason</c>, + this function returns + <seealso marker="#type-start_ret"><c>{error,Reason}</c></seealso>. + If <c>Module:init/1</c> returns + <seealso marker="#type-start_ret"><c>{stop,Reason}</c></seealso> + or + <seealso marker="#type-start_ret"><c>ignore</c></seealso>, + the process is terminated and this function + returns + <seealso marker="#type-start_ret"><c>{error,Reason}</c></seealso> + or + <seealso marker="#type-start_ret"><c>ignore</c></seealso>, + respectively. + </p> + </desc> + </func> + + <func> + <name name="stop" arity="1"/> + <fsummary>Synchronously stop a generic server.</fsummary> + <desc> + <p> + The same as + <seealso marker="#stop/3"><c>stop(<anno>ServerRef</anno>, normal, infinity)</c></seealso>. + </p> + </desc> + </func> + + <func> + <name name="stop" arity="3"/> + <fsummary>Synchronously stop a generic server.</fsummary> + <desc> + <p> + Orders the <c>gen_statem</c> + <seealso marker="#type-server_ref"><c><anno>ServerRef</anno></c></seealso> + to exit with the specified <c><anno>Reason</anno></c> + and waits for it to terminate. + The <c>gen_statem</c> calls + <seealso marker="#Module:terminate/3"><c>Module:terminate/3</c></seealso> + before exiting. + </p> + <p> + This function returns <c>ok</c> if the server terminates + with the expected reason. Any other reason than <c>normal</c>, + <c>shutdown</c>, or <c>{shutdown,Term}</c> causes an + error report to be issued through + <seealso marker="kernel:error_logger#format/2"><c>error_logger:format/2</c></seealso>. + The default <c><anno>Reason</anno></c> is <c>normal</c>. + </p> + <p> + <c><anno>Timeout</anno></c> is an integer > 0, + which specifies how many milliseconds to wait for the server to + terminate, or the atom <c>infinity</c> to wait indefinitely. + Defaults to <c>infinity</c>. + If the server does not terminate within the specified time, + a <c>timeout</c> exception is raised. + </p> + <p> + If the process does not exist, a <c>noproc</c> exception + is raised. + </p> + </desc> + </func> + </funcs> <section> - <title>CALLBACK FUNCTIONS</title> + <title>Callback Functions</title> <p> - The following functions should be exported from a + The following functions are to be exported from a <c>gen_statem</c> callback module. </p> </section> + <funcs> + <func> + <name>Module:code_change(OldVsn, OldState, OldData, Extra) -> + Result + </name> + <fsummary>Update the internal state during upgrade/downgrade.</fsummary> + <type> + <v>OldVsn = Vsn | {down,Vsn}</v> + <v> Vsn = term()</v> + <v>OldState = NewState = term()</v> + <v>Extra = term()</v> + <v>Result = {NewCallbackMode,NewState,NewData} | Reason</v> + <v> + NewCallbackMode = + <seealso marker="#type-callback_mode">callback_mode()</seealso> + </v> + <v> + OldState = NewState = + <seealso marker="#type-state">state()</seealso> + </v> + <v> + OldData = NewData = + <seealso marker="#type-data">data()</seealso> + </v> + <v>Reason = term()</v> + </type> + <desc> + <p> + This function is called by a <c>gen_statem</c> when it is to + update its internal state during a release upgrade/downgrade, + that is, when the instruction <c>{update,Module,Change,...}</c>, + where <c>Change={advanced,Extra}</c>, is specified in the + <seealso marker="sasl:appup"><c>appup</c></seealso> + file. For more information, see + <seealso marker="doc/design_principles:release_handling#instr">OTP Design Principles</seealso>. + </p> + <p> + For an upgrade, <c>OldVsn</c> is <c>Vsn</c>, and + for a downgrade, <c>OldVsn</c> is + <c>{down,Vsn}</c>. <c>Vsn</c> is defined by the <c>vsn</c> + attribute(s) of the old version of the callback module + <c>Module</c>. If no such attribute is defined, the version + is the checksum of the Beam file. + </p> + <note> + <p> + If you would dare to change + <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> + during release upgrade/downgrade, the upgrade is no problem, + as the new code surely knows what <em>callback mode</em> + it needs. However, for a downgrade this function must + know from argument <c>Extra</c> that comes from the + <seealso marker="sasl:appup"><c>sasl:appup</c></seealso> + file what <em>callback mode</em> the old code did use. + It can also be possible to figure this out + from argument <c>{down,Vsn}</c>, as <c>Vsn</c> + in effect defines the old callback module version. + </p> + </note> + <p> + <c>OldState</c> and <c>OldData</c> is the internal state + of the <c>gen_statem</c>. + </p> + <p> + <c>Extra</c> is passed "as is" from the <c>{advanced,Extra}</c> + part of the update instruction. + </p> + <p> + If successful, the function must return the updated + internal state in an + <c>{NewCallbackMode,NewState,NewData}</c> tuple. + </p> + <p> + If the function returns <c>Reason</c>, the ongoing + upgrade fails and rolls back to the old release.</p> + <p> + This function can use + <seealso marker="erts:erlang#throw/1"><c>erlang:throw/1</c></seealso> + to return <c>Result</c> or <c>Reason</c>. + </p> + </desc> + </func> <func> <name>Module:init(Args) -> Result</name> - <fsummary>Initialize process and internal state</fsummary> + <fsummary>Initialize process and internal state.</fsummary> <type> <v>Args = term()</v> <v>Result = {CallbackMode,State,Data}</v> @@ -1316,25 +1358,25 @@ handle_event(_, _, State, Data) -> <v>Reason = term()</v> </type> <desc> - <marker id="Module:init-1" /> - <p> + <marker id="Module:init-1"/> + <p> Whenever a <c>gen_statem</c> is started using - <seealso marker="#start_link/3"><c>start_link/3,4</c></seealso> + <seealso marker="#start_link/3"><c>start_link/3,4</c></seealso> or - <seealso marker="#start/3"><c>start/3,4</c></seealso>, - this function is called by the new process to initialize + <seealso marker="#start/3"><c>start/3,4</c></seealso>, + this function is called by the new process to initialize the implementation state and server data. </p> - <p> + <p> <c>Args</c> is the <c>Args</c> argument provided to the start - function. + function. </p> - <p> - If the initialization is successful, the function should - return <c>{CallbackMode,State,Data}</c> or + <p> + If the initialization is successful, the function is to + return <c>{CallbackMode,State,Data}</c> or <c>{CallbackMode,State,Data,Actions}</c>. <c>CallbackMode</c> selects the - <seealso marker="#type-callback_mode"><em>callback mode</em></seealso>. + <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> of the <c>gen_statem</c>. <c>State</c> is the initial <seealso marker="#type-state"><c>state()</c></seealso> @@ -1347,28 +1389,127 @@ handle_event(_, _, State, Data) -> <seealso marker="#type-state">state</seealso> just as for a <seealso marker="#state_function">state function</seealso>. </p> - <p> - If something goes wrong during the initialization - the function should return <c>{stop,Reason}</c> - or <c>ignore</c>. See + <p> + If the initialization fails, + the function is to return <c>{stop,Reason}</c> + or <c>ignore</c>; see <seealso marker="#start_link/3"><c>start_link/3,4</c></seealso>. </p> <p> - This function may use - <seealso marker="erts:erlang#throw/1"><c>throw/1</c></seealso> + This function can use + <seealso marker="erts:erlang#throw/1"><c>erlang:throw/1</c></seealso> to return <c>Result</c>. </p> </desc> </func> <func> + <name>Module:format_status(Opt, [PDict,State,Data]) -> + Status + </name> + <fsummary>Optional function for providing a term describing the + current <c>gen_statem</c> status.</fsummary> + <type> + <v>Opt = normal | terminate</v> + <v>PDict = [{Key, Value}]</v> + <v> + State = + <seealso marker="#type-state">state()</seealso> + </v> + <v> + Data = + <seealso marker="#type-data">data()</seealso> + </v> + <v>Key = term()</v> + <v>Value = term()</v> + <v>Status = term()</v> + </type> + <desc> + <note> + <p> + This callback is optional, so a callback module does not need + to export it. The <c>gen_statem</c> module provides a default + implementation of this function that returns + <c>{State,Data}</c>. If this callback fails, the default + function returns <c>{State,Info}</c>, + where <c>Info</c> informs of the crash but no details, + to hide possibly sensitive data. + </p> + </note> + <p>This function is called by a <c>gen_statem</c> process when + any of the following apply:</p> + <list type="bulleted"> + <item> + One of + <seealso marker="sys#get_status/1"><c>sys:get_status/1,2</c></seealso> + is invoked to get the <c>gen_statem</c> status. <c>Opt</c> is set + to the atom <c>normal</c> for this case. + </item> + <item> + The <c>gen_statem</c> terminates abnormally and logs an error. + <c>Opt</c> is set to the atom <c>terminate</c> for this case. + </item> + </list> + <p> + This function is useful for changing the form and + appearance of the <c>gen_statem</c> status for these cases. A + callback module wishing to change the + <seealso marker="sys#get_status/1"><c>sys:get_status/1,2</c></seealso> + return value and how + its status appears in termination error logs exports an + instance of <c>format_status/2</c>, which returns a term + describing the current status of the <c>gen_statem</c>. + </p> + <p> + <c>PDict</c> is the current value of the process dictionary + of the <c>gen_statem</c>. + </p> + <p> + <seealso marker="#type-state"><c>State</c></seealso> + is the internal state of the <c>gen_statem</c>. + </p> + <p> + <seealso marker="#type-data"><c>Data</c></seealso> + is the internal server data of the <c>gen_statem</c>. + </p> + <p> + The function is to return <c>Status</c>, a term that + changes the details of the current state and status of + the <c>gen_statem</c>. There are no restrictions on the + form <c>Status</c> can take, but for the + <seealso marker="sys#get_status/1"><c>sys:get_status/1,2</c></seealso> + case (when <c>Opt</c> + is <c>normal</c>), the recommended form for + the <c>Status</c> value is <c>[{data, [{"State", + Term}]}]</c>, where <c>Term</c> provides relevant details of + the <c>gen_statem</c> state. Following this recommendation is not + required, but it makes the callback module status + consistent with the rest of the + <seealso marker="sys#get_status/1"><c>sys:get_status/1,2</c></seealso> + return value. + </p> + <p> + One use for this function is to return compact alternative + state representations to avoid having large state terms + printed in log files. Another use is to hide sensitive data from + being written to the error log. + </p> + <p> + This function can use + <seealso marker="erts:erlang#throw/1"><c>erlang:throw/1</c></seealso> + to return <c>Status</c>. + </p> + </desc> + </func> + + <func> <name>Module:StateName(EventType, EventContent, Data) -> - StateFunctionResult + StateFunctionResult </name> <name>Module:handle_event(EventType, EventContent, - State, Data) -> HandleEventResult + State, Data) -> HandleEventResult </name> - <fsummary>Handle an event</fsummary> + <fsummary>Handle an event.</fsummary> <type> <v> EventType = @@ -1385,59 +1526,55 @@ handle_event(_, _, State, Data) -> </v> <v> StateFunctionResult = - <seealso marker="#type-state_function_result"> - state_function_result() - </seealso> + <seealso marker="#type-state_function_result">state_function_result()</seealso> </v> <v> HandleEventResult = - <seealso marker="#type-handle_event_result"> - handle_event_result() - </seealso> + <seealso marker="#type-handle_event_result">handle_event_result()</seealso> </v> </type> <desc> - <p> + <p> Whenever a <c>gen_statem</c> receives an event from - <seealso marker="#call/2"><c>call/2</c></seealso>, - <seealso marker="#cast/2"><c>cast/2</c></seealso> or - as a normal process message one of these functions is called. If + <seealso marker="#call/2"><c>call/2</c></seealso>, + <seealso marker="#cast/2"><c>cast/2</c></seealso>, or + as a normal process message, one of these functions is called. If <seealso marker="#type-callback_mode"><em>callback mode</em></seealso> - is <c>state_functions</c> then <c>Module:StateName/3</c> is called, - and if it is <c>handle_event_function</c> - then <c>Module:handle_event/4</c> is called. + is <c>state_functions</c>, <c>Module:StateName/3</c> is called, + and if it is <c>handle_event_function</c>, + <c>Module:handle_event/4</c> is called. </p> <p> If <c>EventType</c> is - <seealso marker="#type-event_type"><c>{call,From}</c></seealso> - the caller is waiting for a reply. The reply can be sent + <seealso marker="#type-event_type"><c>{call,From}</c></seealso>, + the caller waits for a reply. The reply can be sent from this or from any other <seealso marker="#state_function">state function</seealso> by returning with <c>{reply,From,Reply}</c> in <seealso marker="#type-action"><c>Actions</c></seealso>, in - <seealso marker="#type-reply_action"><c>Replies</c></seealso> + <seealso marker="#type-reply_action"><c>Replies</c></seealso>, or by calling <seealso marker="#reply/2"><c>reply(From, Reply)</c></seealso>. </p> <p> If this function returns with a next state that - does not match equal (<c>=/=</c>) to the current state - all postponed events will be retried in the next state. + does not match equal (<c>=/=</c>) to the current state, + all postponed events are retried in the next state. </p> <p> The only difference between <c>StateFunctionResult</c> and <c>HandleEventResult</c> is that for <c>StateFunctionResult</c> - the next state has to be an atom but for <c>HandleEventResult</c> + the next state must be an atom, but for <c>HandleEventResult</c> there is no restriction on the next state. </p> <p> - See <seealso marker="#type-action"><c>action()</c></seealso> - for options that can be set and actions that can be done - by <c>gen_statem</c> after returning from this function. + For options that can be set and actions that can be done + by <c>gen_statem</c> after returning from this function, + see <seealso marker="#type-action"><c>action()</c></seealso>. </p> <p> - These functions may use - <seealso marker="erts:erlang#throw/1"><c>throw/1</c></seealso>, + These functions can use + <seealso marker="erts:erlang#throw/1"><c>erlang:throw/1</c></seealso>, to return the result. </p> </desc> @@ -1445,7 +1582,7 @@ handle_event(_, _, State, Data) -> <func> <name>Module:terminate(Reason, State, Data) -> Ignored</name> - <fsummary>Clean up before termination</fsummary> + <fsummary>Clean up before termination.</fsummary> <type> <v>Reason = normal | shutdown | {shutdown,term()} | term()</v> <v>State = <seealso marker="#type-state">state()</seealso></v> @@ -1453,272 +1590,82 @@ handle_event(_, _, State, Data) -> <v>Ignored = term()</v> </type> <desc> - <p> + <p> This function is called by a <c>gen_statem</c> - when it is about to terminate. It should be the opposite of + when it is about to terminate. It is to be the opposite of <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> - and do any necessary cleaning up. When it returns, - the <c>gen_statem</c> terminates with <c>Reason</c>. The return - value is ignored.</p> - <p> + and do any necessary cleaning up. When it returns, + the <c>gen_statem</c> terminates with <c>Reason</c>. The return + value is ignored.</p> + <p> <c>Reason</c> is a term denoting the stop reason and - <seealso marker="#type-state"><c>State</c></seealso> + <seealso marker="#type-state"><c>State</c></seealso> is the internal state of the <c>gen_statem</c>. </p> - <p> + <p> <c>Reason</c> depends on why the <c>gen_statem</c> is terminating. - If it is because another callback function has returned a - stop tuple <c>{stop,Reason}</c> in + If it is because another callback function has returned, a + stop tuple <c>{stop,Reason}</c> in <seealso marker="#type-action"><c>Actions</c></seealso>, - <c>Reason</c> will have the value specified in that tuple. - If it is due to a failure, <c>Reason</c> is the error reason. + <c>Reason</c> has the value specified in that tuple. + If it is because of a failure, <c>Reason</c> is the error reason. </p> <p> If the <c>gen_statem</c> is part of a supervision tree and is - ordered by its supervisor to terminate, this function will be - called with <c>Reason = shutdown</c> if the following + ordered by its supervisor to terminate, this function is + called with <c>Reason = shutdown</c> if both the following conditions apply:</p> <list type="bulleted"> <item> - the <c>gen_statem</c> has been set - to trap exit signals, and + <p> + The <c>gen_statem</c> has been set + to trap exit signals. + </p> </item> <item> - the shutdown strategy as defined in the supervisor's - child specification is an integer timeout value, not - <c>brutal_kill</c>. + <p> + The shutdown strategy as defined in the supervisor's + child specification is an integer time-out value, not + <c>brutal_kill</c>. + </p> </item> </list> <p> Even if the <c>gen_statem</c> is <em>not</em> - part of a supervision tree, this function will be called + part of a supervision tree, this function is called if it receives an <c>'EXIT'</c> message from its parent. - <c>Reason</c> will be the same as + <c>Reason</c> is the same as in the <c>'EXIT'</c> message. </p> <p> - Otherwise, the <c>gen_statem</c> will be immediately terminated. + Otherwise, the <c>gen_statem</c> is immediately terminated. </p> <p> - Note that for any other reason than <c>normal</c>, - <c>shutdown</c>, or <c>{shutdown,Term}</c> - the <c>gen_statem</c> is assumed to terminate due to an error + Notice that for any other reason than <c>normal</c>, + <c>shutdown</c>, or <c>{shutdown,Term}</c>, + the <c>gen_statem</c> is assumed to terminate because of an error and an error report is issued using - <seealso marker="kernel:error_logger#format/2"> - <c>error_logger:format/2</c>. - </seealso> + <seealso marker="kernel:error_logger#format/2"><c>error_logger:format/2</c></seealso>. </p> <p> - This function may use - <seealso marker="erts:erlang#throw/1"><c>throw/1</c></seealso> + This function can use + <seealso marker="erts:erlang#throw/1"><c>erlang:throw/1</c></seealso> to return <c>Ignored</c>, which is ignored anyway. </p> </desc> </func> - - <func> - <name>Module:code_change(OldVsn, OldState, OldData, Extra) -> - Result - </name> - <fsummary>Update the internal state during upgrade/downgrade</fsummary> - <type> - <v>OldVsn = Vsn | {down,Vsn}</v> - <v> Vsn = term()</v> - <v>OldState = NewState = term()</v> - <v>Extra = term()</v> - <v>Result = {NewCallbackMode,NewState,NewData} | Reason</v> - <v> - NewCallbackMode = - <seealso marker="#type-callback_mode">callback_mode()</seealso> - </v> - <v> - OldState = NewState = - <seealso marker="#type-state">state()</seealso> - </v> - <v> - OldData = NewData = - <seealso marker="#type-data">data()</seealso> - </v> - <v>Reason = term()</v> - </type> - <desc> - <p> - This function is called by a <c>gen_statem</c> when it should - update its internal state during a release upgrade/downgrade, - that is when the instruction <c>{update,Module,Change,...}</c> - where <c>Change={advanced,Extra}</c> is given in the - <seealso marker="sasl:appup"><c>appup</c></seealso> - file. See - <seealso marker="doc/design_principles:release_handling#instr"> - OTP Design Principles - </seealso> - for more information. - </p> - <p> - In the case of an upgrade, <c>OldVsn</c> is <c>Vsn</c>, and - in the case of a downgrade, <c>OldVsn</c> is - <c>{down,Vsn}</c>. <c>Vsn</c> is defined by the <c>vsn</c> - attribute(s) of the old version of the callback module - <c>Module</c>. If no such attribute is defined, the version - is the checksum of the BEAM file. - </p> - <note> - <p> - If you would dare to change - <seealso marker="#type-callback_mode"> - <em>callback mode</em> - </seealso> - during release upgrade/downgrade, the upgrade is no problem - since the new code surely knows what <em>callback mode</em> - it needs, but for a downgrade this function will have to - know from the <c>Extra</c> argument that comes from the - <seealso marker="sasl:appup"><c>appup</c></seealso> - file what <em>callback mode</em> the old code did use. - It may also be possible to figure this out - from the <c>{down,Vsn}</c> argument since <c>Vsn</c> - in effect defines the old callback module version. - </p> - </note> - <p> - <c>OldState</c> and <c>OldData</c> is the internal state - of the <c>gen_statem</c>. - </p> - <p> - <c>Extra</c> is passed as-is from the <c>{advanced,Extra}</c> - part of the update instruction. - </p> - <p> - If successful, the function shall return the updated - internal state in an - <c>{NewCallbackMode,NewState,NewData}</c> tuple. - </p> - <p> - If the function returns <c>Reason</c>, the ongoing - upgrade will fail and roll back to the old release.</p> - <p> - This function may use - <seealso marker="erts:erlang#throw/1"><c>throw/1</c></seealso> - to return <c>Result</c> or <c>Reason</c>. - </p> - </desc> - </func> - - <func> - <name>Module:format_status(Opt, [PDict,State,Data]) -> - Status - </name> - <fsummary>Optional function for providing a term describing the - current <c>gen_statem</c> status</fsummary> - <type> - <v>Opt = normal | terminate</v> - <v>PDict = [{Key, Value}]</v> - <v> - State = - <seealso marker="#type-state">state()</seealso> - </v> - <v> - Data = - <seealso marker="#type-data">data()</seealso> - </v> - <v>Key = term()</v> - <v>Value = term()</v> - <v>Status = term()</v> - </type> - <desc> - <note> - <p> - This callback is optional, so a callback module need not - export it. The <c>gen_statem</c> module provides a default - implementation of this function that returns - <c>{State,Data}</c>. If this callback fails the default - function will return <c>{State,Info}</c> - where <c>Info</c> informs of the crash but no details, - to hide possibly sensitive data. - </p> - </note> - <p>This function is called by a <c>gen_statem</c> process when:</p> - <list type="bulleted"> - <item> - One of - <seealso marker="sys#get_status/1"> - <c>sys:get_status/1,2</c> - </seealso> - is invoked to get the <c>gen_statem</c> status. <c>Opt</c> is set - to the atom <c>normal</c> for this case. - </item> - <item> - The <c>gen_statem</c> terminates abnormally and logs an error. - <c>Opt</c> is set to the atom <c>terminate</c> for this case. - </item> - </list> - <p> - This function is useful for customising the form and - appearance of the <c>gen_statem</c> status for these cases. A - callback module wishing to customise the - <seealso marker="sys#get_status/1"> - <c>sys:get_status/1,2</c> - </seealso> - return value as well as how - its status appears in termination error logs exports an - instance of <c>format_status/2</c> that returns a term - describing the current status of the <c>gen_statem</c>. - </p> - <p> - <c>PDict</c> is the current value of the <c>gen_statem</c>'s - process dictionary. - </p> - <p> - <seealso marker="#type-state"><c>State</c></seealso> - is the internal state of the <c>gen_statem</c>. - </p> - <p> - <seealso marker="#type-data"><c>Data</c></seealso> - is the internal server data of the <c>gen_statem</c>. - </p> - <p> - The function should return <c>Status</c>, a term that - customises the details of the current state and status of - the <c>gen_statem</c>. There are no restrictions on the - form <c>Status</c> can take, but for the - <seealso marker="sys#get_status/1"> - <c>sys:get_status/1,2</c> - </seealso> - case (when <c>Opt</c> - is <c>normal</c>), the recommended form for - the <c>Status</c> value is <c>[{data, [{"State", - Term}]}]</c> where <c>Term</c> provides relevant details of - the <c>gen_statem</c> state. Following this recommendation isn't - required, but doing so will make the callback module status - consistent with the rest of the - <seealso marker="sys#get_status/1"> - <c>sys:get_status/1,2</c> - </seealso> - return value. - </p> - <p> - One use for this function is to return compact alternative - state representations to avoid having large state terms - printed in logfiles. Another is to hide sensitive data from - being written to the error log. - </p> - <p> - This function may use - <seealso marker="erts:erlang#throw/1"><c>throw/1</c></seealso> - to return <c>Status</c>. - </p> - </desc> - </func> - </funcs> <section> - <title>SEE ALSO</title> - <p><seealso marker="gen_event"><c>gen_event(3)</c></seealso>, + <title>See Also</title> + <p> + <seealso marker="gen_event"><c>gen_event(3)</c></seealso>, <seealso marker="gen_fsm"><c>gen_fsm(3)</c></seealso>, <seealso marker="gen_server"><c>gen_server(3)</c></seealso>, - <seealso marker="supervisor"><c>supervisor(3)</c></seealso>, <seealso marker="proc_lib"><c>proc_lib(3)</c></seealso>, - <seealso marker="sys"><c>sys(3)</c></seealso></p> + <seealso marker="supervisor"><c>supervisor(3)</c></seealso>, + <seealso marker="sys"><c>sys(3)</c></seealso>. + </p> </section> </erlref> diff --git a/lib/stdlib/doc/src/proc_lib.xml b/lib/stdlib/doc/src/proc_lib.xml index 245580b1ba..f02b1f0651 100644 --- a/lib/stdlib/doc/src/proc_lib.xml +++ b/lib/stdlib/doc/src/proc_lib.xml @@ -73,6 +73,13 @@ <name name="priority_level"/> </datatype> <datatype> + <name name="max_heap_size"/> + <desc> + <p>See <seealso marker="erts:erlang#process_flag_max_heap_size"> + erlang:process_flag(max_heap_size, MaxHeapSize)</seealso>.</p> + </desc> + </datatype> + <datatype> <name name="dict_or_pid"/> </datatype> </datatypes> diff --git a/lib/stdlib/src/otp_internal.erl b/lib/stdlib/src/otp_internal.erl index 7a59523f06..c3ad261daa 100644 --- a/lib/stdlib/src/otp_internal.erl +++ b/lib/stdlib/src/otp_internal.erl @@ -35,7 +35,7 @@ obsolete(Module, Name, Arity) -> case obsolete_1(Module, Name, Arity) of {deprecated=Tag,{_,_,_}=Replacement} -> - {Tag,Replacement,"in a future release"}; + {Tag,Replacement,"a future release"}; {_,String}=Ret when is_list(String) -> Ret; {_,_,_}=Ret -> diff --git a/lib/stdlib/src/proc_lib.erl b/lib/stdlib/src/proc_lib.erl index fdc2ae4070..4a19603ec2 100644 --- a/lib/stdlib/src/proc_lib.erl +++ b/lib/stdlib/src/proc_lib.erl @@ -43,9 +43,14 @@ %%----------------------------------------------------------------------------- -type priority_level() :: 'high' | 'low' | 'max' | 'normal'. +-type max_heap_size() :: non_neg_integer() | + #{ size => non_neg_integer(), + kill => true, + error_logger => true}. -type spawn_option() :: 'link' | 'monitor' | {'priority', priority_level()} + | {'max_heap_size', max_heap_size()} | {'min_heap_size', non_neg_integer()} | {'min_bin_vheap_size', non_neg_integer()} | {'fullsweep_after', non_neg_integer()} diff --git a/lib/stdlib/test/erl_lint_SUITE.erl b/lib/stdlib/test/erl_lint_SUITE.erl index b0214e5238..d916eb3eef 100644 --- a/lib/stdlib/test/erl_lint_SUITE.erl +++ b/lib/stdlib/test/erl_lint_SUITE.erl @@ -2003,7 +2003,7 @@ otp_5362(Config) when is_list(Config) -> {error, [{5,erl_lint,{call_to_redefined_old_bif,{spawn,1}}}], [{4,erl_lint,{deprecated,{erlang,hash,2},{erlang,phash2,2}, - "in a future release"}}]}}, + "a future release"}}]}}, {otp_5362_5, <<"-compile(nowarn_deprecated_function). @@ -2063,7 +2063,7 @@ otp_5362(Config) when is_list(Config) -> {nowarn_bif_clash,{spawn,1}}]}, % has no effect {warnings, [{5,erl_lint,{deprecated,{erlang,hash,2},{erlang,phash2,2}, - "in a future release"}}]}}, + "a future release"}}]}}, {otp_5362_9, <<"-include_lib(\"stdlib/include/qlc.hrl\"). @@ -2093,7 +2093,7 @@ otp_5362(Config) when is_list(Config) -> [], {warnings, [{1,erl_lint,{deprecated,{erlang,hash,2}, - {erlang,phash2,2},"in a future release"}}]}}, + {erlang,phash2,2},"a future release"}}]}}, {call_removed_function, <<"t(X) -> regexp:match(X).">>, diff --git a/lib/syntax_tools/doc/src/notes.xml b/lib/syntax_tools/doc/src/notes.xml index 406a6c071b..78b2c7c7a4 100644 --- a/lib/syntax_tools/doc/src/notes.xml +++ b/lib/syntax_tools/doc/src/notes.xml @@ -76,11 +76,11 @@ <p> Teach Maps to erl_syntax</p> <p> - Affected functions: <list> + Affected functions:</p> <list> <item>erl_syntax:abstract/1</item> <item>erl_syntax:concrete/1</item> <item>erl_syntax:is_leaf/1</item> - <item>erl_syntax:is_literal/1</item> </list></p> + <item>erl_syntax:is_literal/1</item> </list> <p> Own Id: OTP-12265</p> </item> diff --git a/lib/syntax_tools/vsn.mk b/lib/syntax_tools/vsn.mk index 403e90196e..f09c2a01d0 100644 --- a/lib/syntax_tools/vsn.mk +++ b/lib/syntax_tools/vsn.mk @@ -1 +1 @@ -SYNTAX_TOOLS_VSN = 1.7 +SYNTAX_TOOLS_VSN = 2.0 diff --git a/lib/tools/src/cover.erl b/lib/tools/src/cover.erl index 69331732bf..92c10cc306 100644 --- a/lib/tools/src/cover.erl +++ b/lib/tools/src/cover.erl @@ -573,7 +573,7 @@ call(Request) -> Ref = erlang:monitor(process,?SERVER), receive {'DOWN', Ref, _Type, _Object, noproc} -> erlang:demonitor(Ref), - start(), + {ok,_} = start(), call(Request) after 0 -> ?SERVER ! {self(),Request}, @@ -589,7 +589,9 @@ call(Request) -> end. reply(From, Reply) -> - From ! {?SERVER,Reply}. + From ! {?SERVER,Reply}, + ok. + is_from(From) -> is_pid(From). @@ -615,9 +617,11 @@ remote_call(Node,Request) -> end. remote_reply(Proc,Reply) when is_pid(Proc) -> - Proc ! {?SERVER,Reply}; + Proc ! {?SERVER,Reply}, + ok; remote_reply(MainNode,Reply) -> - {?SERVER,MainNode} ! {?SERVER,Reply}. + {?SERVER,MainNode} ! {?SERVER,Reply}, + ok. %%%---------------------------------------------------------------------- %%% cover_server on main node @@ -627,14 +631,16 @@ init_main(Starter) -> register(?SERVER,self()), %% Having write concurrancy here gives a 40% performance boost %% when collect/1 is called. - ets:new(?COVER_TABLE, [set, public, named_table - ,{write_concurrency, true} - ]), - ets:new(?COVER_CLAUSE_TABLE, [set, public, named_table]), - ets:new(?BINARY_TABLE, [set, public, named_table]), - ets:new(?COLLECTION_TABLE, [set, public, named_table]), - ets:new(?COLLECTION_CLAUSE_TABLE, [set, public, named_table]), - net_kernel:monitor_nodes(true), + ?COVER_TABLE = ets:new(?COVER_TABLE, [set, public, named_table, + {write_concurrency, true}]), + ?COVER_CLAUSE_TABLE = ets:new(?COVER_CLAUSE_TABLE, [set, public, + named_table]), + ?BINARY_TABLE = ets:new(?BINARY_TABLE, [set, public, named_table]), + ?COLLECTION_TABLE = ets:new(?COLLECTION_TABLE, [set, public, + named_table]), + ?COLLECTION_CLAUSE_TABLE = ets:new(?COLLECTION_CLAUSE_TABLE, [set, public, + named_table]), + ok = net_kernel:monitor_nodes(true), Starter ! {?SERVER,started}, main_process_loop(#main_state{}). @@ -672,7 +678,7 @@ main_process_loop(State) -> Imported = do_import_to_table(Fd,File, State#main_state.imported), reply(From, ok), - file:close(Fd), + ok = file:close(Fd), main_process_loop(State#main_state{imported=Imported}); {error,Reason} -> reply(From, {error, {cant_open_file,File,Reason}}), @@ -870,11 +876,12 @@ main_process_loop(State) -> init_remote(Starter,MainNode) -> register(?SERVER,self()), - ets:new(?COVER_TABLE, [set, public, named_table - %% write_concurrency here makes otp_8270 break :( - %,{write_concurrency, true} - ]), - ets:new(?COVER_CLAUSE_TABLE, [set, public, named_table]), + %% write_concurrency here makes otp_8270 break :( + ?COVER_TABLE = ets:new(?COVER_TABLE, [set, public, named_table + %,{write_concurrency, true} + ]), + ?COVER_CLAUSE_TABLE = ets:new(?COVER_CLAUSE_TABLE, [set, public, + named_table]), Starter ! {self(),started}, remote_process_loop(#remote_state{main_node=MainNode}). @@ -907,11 +914,11 @@ remote_process_loop(State) -> '_' -> [M || {M,_} <- State#remote_state.compiled]; _ -> Modules0 end, - spawn(fun() -> - ?SPAWN_DBG(remote_collect, - {Modules, CollectorPid, From}), - do_collect(Modules, CollectorPid, From) - end), + spawn(fun() -> + ?SPAWN_DBG(remote_collect, + {Modules, CollectorPid, From}), + do_collect(Modules, CollectorPid, From) + end), remote_process_loop(State); {remote,stop} -> @@ -952,13 +959,13 @@ remote_process_loop(State) -> end. do_collect(Modules, CollectorPid, From) -> - pmap( - fun(Module) -> - Pattern = {#bump{module=Module, _='_'}, '$1'}, - MatchSpec = [{Pattern,[{'=/=','$1',0}],['$_']}], - Match = ets:select(?COVER_TABLE,MatchSpec,?CHUNK_SIZE), - send_chunks(Match, CollectorPid, []) - end,Modules), + _ = pmap( + fun(Module) -> + Pattern = {#bump{module=Module, _='_'}, '$1'}, + MatchSpec = [{Pattern,[{'=/=','$1',0}],['$_']}], + Match = ets:select(?COVER_TABLE,MatchSpec,?CHUNK_SIZE), + send_chunks(Match, CollectorPid, []) + end,Modules), CollectorPid ! done, remote_reply(From, ok). @@ -994,20 +1001,20 @@ get_downs(Mons) -> end. reload_originals(Compiled) -> - Modules = [M || {M,_} <- Compiled], - pmap(fun do_reload_original/1, Modules). + _ = pmap(fun do_reload_original/1, [M || {M,_} <- Compiled]), + ok. do_reload_original(Module) -> case code:which(Module) of ?TAG -> - code:purge(Module), % remove code marked as 'old' - code:delete(Module), % mark cover compiled code as 'old' + _ = code:purge(Module), % remove code marked as 'old' + _ = code:delete(Module), % mark cover compiled code as 'old' %% Note: original beam code must be loaded before the cover %% compiled code is purged, in order to for references to %% 'fun M:F/A' and %% 'fun F/A' funs to be correct (they %% refer to (M:)F/A in the *latest* version of the module) - code:load_file(Module), % load original code - code:purge(Module); % remove cover compiled code + _ = code:load_file(Module), % load original code + _ = code:purge(Module); % remove cover compiled code _ -> ignore end. @@ -1219,12 +1226,13 @@ remote_reset(Module,Nodes) -> %% Collect data from remote nodes - used for analyse or stop(Node) remote_collect(Modules,Nodes,Stop) -> - pmap(fun(Node) -> - ?SPAWN_DBG(remote_collect, - {Modules, Nodes, Stop}), - do_collection(Node, Modules, Stop) - end, - Nodes). + _ = pmap( + fun(Node) -> + ?SPAWN_DBG(remote_collect, + {Modules, Nodes, Stop}), + do_collection(Node, Modules, Stop) + end, Nodes), + ok. do_collection(Node, Module, Stop) -> CollectorPid = spawn(fun collector_proc/0), @@ -1260,8 +1268,8 @@ insert_in_collection_table([]) -> insert_in_collection_table(Key,Val) -> case ets:member(?COLLECTION_TABLE,Key) of true -> - ets:update_counter(?COLLECTION_TABLE, - Key,Val); + _ = ets:update_counter(?COLLECTION_TABLE, Key,Val), + ok; false -> %% Make sure that there are no race conditions from ets:member case ets:insert_new(?COLLECTION_TABLE,{Key,Val}) of @@ -2421,7 +2429,7 @@ do_analyse_to_file1(Module, OutFile, ErlFile, HTML) -> "<body style='background-color: white;" " color: black'>\n" "<pre>\n"], - file:write(OutFd,Header); + ok = file:write(OutFd,Header); true -> ok end, @@ -2435,7 +2443,7 @@ do_analyse_to_file1(Module, OutFile, ErlFile, HTML) -> string:right(integer_to_list(H), 2, $0), string:right(integer_to_list(Mi), 2, $0), string:right(integer_to_list(S), 2, $0)]), - file:write(OutFd, + ok = file:write(OutFd, ["File generated from ",ErlFile," by COVER ", Timestamp,"\n\n" "**************************************" @@ -2447,14 +2455,13 @@ do_analyse_to_file1(Module, OutFile, ErlFile, HTML) -> CovLines = lists:keysort(1,ets:select(?COLLECTION_TABLE, MS)), print_lines(Module, CovLines, InFd, OutFd, 1, HTML), - if - HTML -> - file:write(OutFd, "</pre>\n</body>\n</html>\n"); + if HTML -> + ok = file:write(OutFd, "</pre>\n</body>\n</html>\n"); true -> ok end, - file:close(OutFd), - file:close(InFd), + ok = file:close(OutFd), + ok = file:close(InFd), {ok, OutFile}; @@ -2472,34 +2479,33 @@ print_lines(Module, CovLines, InFd, OutFd, L, HTML) -> eof -> ignore; {ok,"%"++_=Line} -> %Comment line - not executed. - file:write(OutFd, [tab(),escape_lt_and_gt(Line, HTML)]), + ok = file:write(OutFd, [tab(),escape_lt_and_gt(Line, HTML)]), print_lines(Module, CovLines, InFd, OutFd, L+1, HTML); {ok,RawLine} -> Line = escape_lt_and_gt(RawLine,HTML), case CovLines of [{L,N}|CovLines1] -> %% N = lists:foldl(fun([Ni], Nacc) -> Nacc+Ni end, 0, Ns), - if - N=:=0, HTML=:=true -> - LineNoNL = Line -- "\n", - Str = " 0", - %%Str = string:right("0", 6, 32), - RedLine = ["<font color=red>",Str,fill1(), - LineNoNL,"</font>\n"], - file:write(OutFd, RedLine); - N<1000000 -> - Str = string:right(integer_to_list(N), 6, 32), - file:write(OutFd, [Str,fill1(),Line]); - N<10000000 -> - Str = integer_to_list(N), - file:write(OutFd, [Str,fill2(),Line]); - true -> - Str = integer_to_list(N), - file:write(OutFd, [Str,fill3(),Line]) - end, + if N=:=0, HTML=:=true -> + LineNoNL = Line -- "\n", + Str = " 0", + %%Str = string:right("0", 6, 32), + RedLine = ["<font color=red>",Str,fill1(), + LineNoNL,"</font>\n"], + ok = file:write(OutFd, RedLine); + N < 1000000 -> + Str = string:right(integer_to_list(N), 6, 32), + ok = file:write(OutFd, [Str,fill1(),Line]); + N < 10000000 -> + Str = integer_to_list(N), + ok = file:write(OutFd, [Str,fill2(),Line]); + true -> + Str = integer_to_list(N), + ok = file:write(OutFd, [Str,fill3(),Line]) + end, print_lines(Module, CovLines1, InFd, OutFd, L+1, HTML); _ -> - file:write(OutFd, [tab(),Line]), + ok = file:write(OutFd, [tab(),Line]), print_lines(Module, CovLines, InFd, OutFd, L+1, HTML) end end. @@ -2539,7 +2545,7 @@ do_export(Module, OutFile, From, State) -> {error,{not_cover_compiled,Module}} end end, - file:close(Fd), + ok = file:close(Fd), reply(From, Reply); {error,Reason} -> reply(From, {error, {cant_open_file,OutFile,Reason}}) @@ -2581,10 +2587,9 @@ write(Element,Fd) -> case byte_size(Bin) of Size when Size > 255 -> SizeBin = term_to_binary({'$size',Size}), - file:write(Fd, - <<(byte_size(SizeBin)):8,SizeBin/binary,Bin/binary>>); + ok = file:write(Fd, <<(byte_size(SizeBin)):8,SizeBin/binary,Bin/binary>>); Size -> - file:write(Fd,<<Size:8,Bin/binary>>) + ok = file:write(Fd,<<Size:8,Bin/binary>>) end, ok. diff --git a/lib/tools/src/eprof.erl b/lib/tools/src/eprof.erl index 07e995993b..3ae899a078 100644 --- a/lib/tools/src/eprof.erl +++ b/lib/tools/src/eprof.erl @@ -74,7 +74,6 @@ start() -> gen_server:start({local, ?MODULE}, ?MODULE, [], []). stop() -> gen_server:call(?MODULE, stop, infinity). - analyze() -> analyze(procs). @@ -112,7 +111,7 @@ profile(Rootset, M, F, A, Pattern) when is_list(Rootset), is_atom(M), is_atom(F) %% Returns when M:F/A has terminated profile(Rootset, M, F, A, Pattern, Options) -> - start(), + ok = start_internal(), gen_server:call(?MODULE, {profile_start, Rootset, Pattern, {M,F,A}, Options}, infinity). dump() -> @@ -127,7 +126,7 @@ start_profiling(Rootset) -> start_profiling(Rootset, Pattern) -> start_profiling(Rootset, Pattern, ?default_options). start_profiling(Rootset, Pattern, Options) -> - start(), + ok = start_internal(), gen_server:call(?MODULE, {profile_start, Rootset, Pattern, undefined, Options}, infinity). stop_profiling() -> @@ -251,9 +250,9 @@ handle_call({logfile, File}, _From, #state{ fd = OldFd } = S) -> {ok, Fd} -> case OldFd of undefined -> ok; - OldFd -> file:close(OldFd) + OldFd -> ok = file:close(OldFd) end, - {reply, ok, S#state{ fd = Fd}}; + {reply, ok, S#state{fd = Fd}}; Error -> {reply, Error, S} end; @@ -521,3 +520,10 @@ format(Fd, Format, Strings) -> divide(_,0) -> 0.0; divide(T,N) -> T/N. + +start_internal() -> + case start() of + {ok, _} -> ok; + {error, {already_started,_}} -> ok; + Error -> Error + end. diff --git a/lib/tools/src/fprof.erl b/lib/tools/src/fprof.erl index b21eedc625..8db23dd151 100644 --- a/lib/tools/src/fprof.erl +++ b/lib/tools/src/fprof.erl @@ -1003,7 +1003,7 @@ handle_req(#analyse{dest = Dest, already_open -> ok; ok -> - file:close(DestPid) + ok = file:close(DestPid) end, State end; @@ -1364,7 +1364,7 @@ tracer_loop(Parent, Handler, State) -> Trace when element(1, Trace) =:= trace_ts -> tracer_loop(Parent, Handler, Handler(Trace, State)); {'EXIT', Parent, Reason} -> - handler(end_of_trace, State), + _ = handler(end_of_trace, State), exit(Reason); _ -> tracer_loop(Parent, Handler, State) @@ -1450,12 +1450,10 @@ end_of_trace(Table, TS) -> Procs = get(), put(table, Table), ?dbg(2, "get() -> ~p~n", [Procs]), - lists:map( - fun ({Pid, _}) when is_pid(Pid) -> - trace_exit(Table, Pid, TS) - end, - Procs), - erase(), + _ = lists:map(fun ({Pid, _}) when is_pid(Pid) -> + trace_exit(Table, Pid, TS) + end, Procs), + _ = erase(), ok. @@ -2047,7 +2045,7 @@ trace_exit(Table, Pid, TS) -> [] -> ok; [_ | _] = Stack -> - trace_return_to_int(Table, Pid, undefined, TS, Stack), + _ = trace_return_to_int(Table, Pid, undefined, TS, Stack), ok end, ok. @@ -2173,7 +2171,7 @@ trace_clock(_Table, _Pid, _T, [[{suspend, _}], [{suspend, _}] | _]=_Stack, _Clock) -> ?dbg(9, "trace_clock(Table, ~w, ~w, ~w, ~w)~n", [_Pid, _T, _Stack, _Clock]), - void; + ok; trace_clock(Table, Pid, T, [[{garbage_collect, TS0}], [{suspend, _}]], Clock) -> trace_clock_1(Table, Pid, T, TS0, undefined, garbage_collect, Clock); @@ -2188,7 +2186,7 @@ trace_clock(Table, Pid, T, [[{Func0, TS0}], [{Func1, _} | _] | _], Clock) -> trace_clock(Table, Pid, T, [[{Func0, TS0}]], Clock) -> trace_clock_1(Table, Pid, T, TS0, undefined, Func0, Clock); trace_clock(_, _, _, [], _) -> - void. + ok. trace_clock_1(Table, Pid, _, _, Caller, suspend, #clocks.own) -> clock_add(Table, {Pid, Caller, suspend}, #clocks.own, 0); @@ -2202,7 +2200,7 @@ trace_clock_1(Table, Pid, T, TS, Caller, Func, Clock) -> clock_add(Table, Id, Clock, T) -> ?dbg(1, "clock_add(Table, ~w, ~w, ~w)~n", [Id, Clock, T]), - try ets:update_counter(Table, Id, {Clock, T}) + try ets:update_counter(Table, Id, {Clock, T}), ok catch error:badarg -> ets:insert(Table, #clocks{id = Id}), @@ -2211,7 +2209,7 @@ clock_add(Table, Id, Clock, T) -> true -> ?dbg(0, "Negative counter value ~p ~p ~p ~p~n", [X, Id, Clock, T]) end, - X + ok end. clocks_add(Table, #clocks{id = Id} = Clocks) -> diff --git a/lib/tools/src/lcnt.erl b/lib/tools/src/lcnt.erl index 78407a651a..23d66b084e 100644 --- a/lib/tools/src/lcnt.erl +++ b/lib/tools/src/lcnt.erl @@ -23,67 +23,57 @@ -author("Björn-Egil Dahlberg"). %% gen_server callbacks --export([ - init/1, - handle_call/3, - handle_cast/2, - handle_info/2, - terminate/2, - code_change/3 - ]). +-export([init/1, + handle_call/3, + handle_cast/2, + handle_info/2, + terminate/2, + code_change/3]). %% start/stop --export([ - start/0, - stop/0 - ]). +-export([start/0, + stop/0]). %% erts_debug:lock_counters api --export([ - rt_collect/0, - rt_collect/1, - rt_clear/0, - rt_clear/1, - rt_opt/1, - rt_opt/2 - ]). +-export([rt_collect/0, + rt_collect/1, + rt_clear/0, + rt_clear/1, + rt_opt/1, + rt_opt/2]). %% gen_server call api --export([ - raw/0, - collect/0, - collect/1, - clear/0, - clear/1, - conflicts/0, - conflicts/1, - locations/0, - locations/1, - inspect/1, - inspect/2, - histogram/1, - histogram/2, - information/0, - swap_pid_keys/0, - % set options - set/1, - set/2, - - load/1, - save/1 - ]). +-export([raw/0, + collect/0, + collect/1, + clear/0, + clear/1, + conflicts/0, + conflicts/1, + locations/0, + locations/1, + inspect/1, + inspect/2, + histogram/1, + histogram/2, + information/0, + swap_pid_keys/0, + % set options + set/1, + set/2, + + load/1, + save/1]). %% convenience --export([ - apply/3, - apply/2, - apply/1, - all_conflicts/0, - all_conflicts/1, - pid/2, pid/3, - port/1, port/2 - ]). +-export([apply/3, + apply/2, + apply/1, + all_conflicts/0, + all_conflicts/1, + pid/2, pid/3, + port/1, port/2]). -define(version, "1.0"). @@ -135,6 +125,13 @@ start() -> gen_server:start({local, ?MODULE}, ?MODULE, [], []). stop() -> gen_server:call(?MODULE, stop, infinity). init([]) -> {ok, #state{ locks = [], duration = 0 } }. +start_internal() -> + case start() of + {ok,_} -> ok; + {error, {already_started,_}} -> ok; + Error -> Error + end. + %% -------------------------------------------------------------------- %% %% %% API erts_debug:lock_counters @@ -184,7 +181,7 @@ raw() -> call(raw). set(Option, Value) -> call({set, Option, Value}). set({Option, Value}) -> call({set, Option, Value}). save(Filename) -> call({save, Filename}). -load(Filename) -> start(), call({load, Filename}). +load(Filename) -> ok = start_internal(), call({load, Filename}). call(Msg) -> gen_server:call(?MODULE, Msg, infinity). @@ -195,7 +192,7 @@ call(Msg) -> gen_server:call(?MODULE, Msg, infinity). %% -------------------------------------------------------------------- %% apply(M,F,As) when is_atom(M), is_atom(F), is_list(As) -> - lcnt:start(), + ok = start_internal(), Opt = lcnt:rt_opt({copy_save, true}), lcnt:clear(), Res = erlang:apply(M,F,As), @@ -207,7 +204,7 @@ apply(Fun) when is_function(Fun) -> lcnt:apply(Fun, []). apply(Fun, As) when is_function(Fun) -> - lcnt:start(), + ok = start_internal(), Opt = lcnt:rt_opt({copy_save, true}), lcnt:clear(), Res = erlang:apply(Fun, As), diff --git a/lib/tools/src/tags.erl b/lib/tools/src/tags.erl index 2bc1865503..b833d96c19 100644 --- a/lib/tools/src/tags.erl +++ b/lib/tools/src/tags.erl @@ -101,7 +101,7 @@ files(Files, Options) -> case open_out(Options) of {ok, Os} -> files_loop(Files, Os), - close_out(Os), + ok = close_out(Os), ok; _ -> error @@ -169,7 +169,7 @@ filename(Name, Os) -> case file:open(Name, [read]) of {ok, Desc} -> Acc = module(Desc, [], [], {1, 0}), - file:close(Desc), + ok = file:close(Desc), genout(Os, Name, Acc), ok; _ -> diff --git a/lib/tools/src/xref_base.erl b/lib/tools/src/xref_base.erl index 70eb107781..4322943c59 100644 --- a/lib/tools/src/xref_base.erl +++ b/lib/tools/src/xref_base.erl @@ -696,7 +696,7 @@ do_add_module({Dir, Basename}, AppName, Builtins, Verbose, Warnings, State) -> File = filename:join(Dir, Basename), {ok, M, Bad, NewState} = do_add_module1(Dir, File, AppName, Builtins, Verbose, Warnings, State), - filter(fun({Tag,B}) -> warnings(Warnings, Tag, [[File,B]]) end, Bad), + _ = filter(fun({Tag,B}) -> warnings(Warnings, Tag, [[File,B]]) end, Bad), {ok, M, NewState}. do_add_module1(Dir, File, AppName, Builtins, Verbose, Warnings, State) -> @@ -1727,7 +1727,7 @@ pack(T) -> NT = pack1(T), %% true = T =:= NT, %% io:format("erasing ~p elements...~n", [length(erase())]), - erase(), % wasting heap (and time)... + _ = erase(), % wasting heap (and time)... foreach(fun({K,V}) -> put(K, V) end, PD), NT. @@ -139,7 +139,7 @@ check_erltop () if [ "X$ERL_TOP" = "X" ]; then if [ -f ./otp_build -a -f ./erts/autoconf/config.guess ]; then ERLTOP_FORCED=true - ERL_TOP=`/bin/pwd` + ERL_TOP=`pwd` export ERL_TOP else echo "The environment variable ERL_TOP must be set." >&2 @@ -679,7 +679,7 @@ echo_env_erltop () if [ X"$ERL_TOP" = X"" -o "$ERLTOP_FORCED" = "true" ]; then if [ -f ./otp_build ]; then # Seems to be current directory... - echo_setenv ERL_TOP `/bin/pwd` ';' + echo_setenv ERL_TOP `pwd` ';' else echo "You need to either set ERL_TOP first or stand in the same" \ "directory as this script resides in." >&2 diff --git a/system/doc/design_principles/statem.xml b/system/doc/design_principles/statem.xml index ca0fce55e2..b63327291d 100644 --- a/system/doc/design_principles/statem.xml +++ b/system/doc/design_principles/statem.xml @@ -5,7 +5,7 @@ <header> <copyright> <year>2016</year> - <holder>Ericsson AB. All Rights Reserved.</holder> + <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> Licensed under the Apache License, Version 2.0 (the "License"); @@ -22,7 +22,7 @@ </legalnotice> - <title>gen_statem Behaviour</title> + <title>gen_statem Behavior</title> <prepared></prepared> <docno></docno> <date></date> @@ -33,63 +33,61 @@ <p> This section is to be read with the <seealso marker="stdlib:gen_statem"><c>gen_statem(3)</c></seealso> - manual page in STDLIB, where all interface functions and callback + manual page in <c>STDLIB</c>, where all interface functions and callback functions are described in detail. </p> - <p> - This is a new behaviour in OTP-19.0. - It has been thoroughly reviewed, is stable enough - to be used by at least two heavy OTP applications, and is here to stay. - But depending on user feedback, we do not expect - but might find it necessary to make minor - not backwards compatible changes into OTP-20.0, - so its state can be designated as "not quite experimental"... - </p> + <note> + <p> + This is a new behavior in Erlang/OTP 19.0. + It has been thoroughly reviewed, is stable enough + to be used by at least two heavy OTP applications, and is here to stay. + Depending on user feedback, we do not expect + but can find it necessary to make minor + not backward compatible changes into Erlang/OTP 20.0. + </p> + </note> <!-- =================================================================== --> <section> - <title>Event Driven State Machines</title> + <title>Event-Driven State Machines</title> <p> Established Automata theory does not deal much with how a state transition is triggered, - but in general assumes that the output is a function + but assumes that the output is a function of the input (and the state) and that they are some kind of values. </p> <p> - For an Event Driven State Machine the input is an event + For an Event-Driven State Machine, the input is an event that triggers a state transition and the output is actions executed during the state transition. It can analogously to the mathematical model of a - Finite State Machine be described as - a set of relations of the form: + Finite-State Machine be described as + a set of relations of the following form: </p> <pre> State(S) x Event(E) -> Actions(A), State(S')</pre> - <p>These relations are interpreted as meaning:</p> - <p> - If we are in state <c>S</c> and event <c>E</c> occurs, we + <p>These relations are interpreted as follows: + if we are in state <c>S</c> and event <c>E</c> occurs, we are to perform actions <c>A</c> and make a transition to - state <c>S'</c>. - </p> - <p> - Note that <c>S'</c> may be equal to <c>S</c>. + state <c>S'</c>. Notice that <c>S'</c> can be equal to <c>S</c>. </p> <p> - Since <c>A</c> and <c>S'</c> depend only on - <c>S</c> and <c>E</c> the kind of state machine described - here is a Mealy Machine. - (See for example the corresponding Wikipedia article) + As <c>A</c> and <c>S'</c> depend only on + <c>S</c> and <c>E</c>, the kind of state machine described + here is a Mealy Machine + (see, for example, the corresponding Wikipedia article). </p> <p> - Like most <c>gen_</c> behaviours, <c>gen_statem</c> keeps - a server <c>Data</c> besides the state. This and the fact that + Like most <c>gen_</c> behaviors, <c>gen_statem</c> keeps + a server <c>Data</c> besides the state. Because of this, and as there is no restriction on the number of states - (assuming enough virtual machine memory) - or on the number of distinct input events actually makes - a state machine implemented with this behaviour Turing complete. - But it feels mostly like an Event Driven Mealy Machine. + (assuming that there is enough virtual machine memory) + or on the number of distinct input events, + a state machine implemented with this behavior + is in fact Turing complete. + But it feels mostly like an Event-Driven Mealy Machine. </p> </section> @@ -99,38 +97,39 @@ State(S) x Event(E) -> Actions(A), State(S')</pre> <marker id="callback_modes" /> <title>Callback Modes</title> <p> - The <c>gen_statem</c> behaviour supports two different callback modes. - In the mode - <seealso marker="stdlib:gen_statem#type-callback_mode"> - <c>state_functions</c>, - </seealso> - the state transition rules are written as a number of Erlang - functions, which conform to the following convention: + The <c>gen_statem</c> behavior supports two callback modes: </p> - <pre> + <list type="bulleted"> + <item> + <p> + In mode + <seealso marker="stdlib:gen_statem#type-callback_mode"><c>state_functions</c></seealso>, + the state transition rules are written as some Erlang + functions, which conform to the following convention: + </p> + <pre> StateName(EventType, EventContent, Data) -> .. code for actions here ... {next_state, NewStateName, NewData}.</pre> - <p> - In the mode - <seealso marker="stdlib:gen_statem#type-callback_mode"> - <c>handle_event_function</c> - </seealso> - there is only one - Erlang function that implements all state transition rules: - </p> - <pre> + </item> + <item> + <p> + In mode + <seealso marker="stdlib:gen_statem#type-callback_mode"><c>handle_event_function</c></seealso>, + only one Erlang function provides all state transition rules: + </p> + <pre> handle_event(EventType, EventContent, State, Data) -> .. code for actions here ... - {next_state, State', Data'}</pre> - <p> - Both these modes allow other return tuples - that you can find in the - <seealso marker="stdlib:gen_statem#Module:StateName/3"> - reference manual. - </seealso> - These other return tuples can for example stop the machine, - execute state transition actions on the machine engine itself + {next_state, NewState, NewData}</pre> + </item> + </list> + <p> + Both these modes allow other return tuples; see + <seealso marker="stdlib:gen_statem#Module:StateName/3"><c>Module:StateName/3</c></seealso> + in the <c>gen_statem</c> manual page. + These other return tuples can, for example, stop the machine, + execute state transition actions on the machine engine itself, and send replies. </p> @@ -139,54 +138,56 @@ handle_event(EventType, EventContent, State, Data) -> <p> The two <seealso marker="#callback_modes">callback modes</seealso> - gives different possibilities + give different possibilities and restrictions, but one goal remains: you want to handle all possible combinations of events and states. </p> <p> - You can for example do this by focusing on one state at the time - and for every state ensure that all events are handled, - or the other way around focus on one event at the time - and ensure that it is handled in every state, - or mix these strategies. + This can be done, for example, by focusing on one state at the time + and for every state ensure that all events are handled. + Alternatively, you can focus on one event at the time + and ensure that it is handled in every state. + You can also use a mix of these strategies. </p> <p> - With <c>state_functions</c> you are restricted to use - atom only states, and the <c>gen_statem</c> engine dispatches - on state name for you. This encourages the callback module - to gather the implementation of all event actions particular - to one state in the same place in the code + With <c>state_functions</c>, you are restricted to use + atom-only states, and the <c>gen_statem</c> engine + branches depending on state name for you. + This encourages the callback module to gather + the implementation of all event actions particular + to one state in the same place in the code, hence to focus on one state at the time. </p> <p> - This mode fits well when you have a regular state diagram - like the ones in this chapter that describes all events and actions + This mode fits well when you have a regular state diagram, + like the ones in this chapter, which describes all events and actions belonging to a state visually around that state, and each state has its unique name. </p> <p> - With <c>handle_event_function</c> you are free to mix strategies - as you like because all events and states - are handled in the the same callback function. + With <c>handle_event_function</c>, you are free to mix strategies, + as all events and states are handled in the same callback function. </p> <p> This mode works equally well when you want to focus on - one event at the time or when you want to focus on - one state at the time, but the <c>handle_event/4</c> function - quickly grows too large to handle without introducing dispatching. + one event at the time or on + one state at the time, but function + <seealso marker="stdlib:gen_statem#Module:handle_event/4"><c>Module:handle_event/4</c></seealso> + quickly grows too large to handle without branching to + helper functions. </p> <p> - The mode enables the use of non-atom states for example - complex states or even hiearchical states. + The mode enables the use of non-atom states, for example, + complex states or even hierarchical states. If, for example, a state diagram is largely alike - for the client and for the server side of a protocol; - then you can have a state <c>{StateName,server}</c> or - <c>{StateName,client}</c> and since you do the dispatching - yourself you make <c>StateName</c> decide where in the code + for the client side and the server side of a protocol, + you can have a state <c>{StateName,server}</c> or + <c>{StateName,client}</c>, + and make <c>StateName</c> determine where in the code to handle most events in the state. - The second element of the tuple is then used to select - whether to handle special client side or server side events. + The second element of the tuple is then used to select + whether to handle special client-side or server-side events. </p> </section> </section> @@ -196,31 +197,28 @@ handle_event(EventType, EventContent, State, Data) -> <section> <title>Example</title> <p> - This is an example starting off as equivalent to the the example in the - <seealso marker="fsm"><c>gen_fsm</c> behaviour</seealso> - description. In later chapters additions and tweaks are made + This example starts off as equivalent to the example in section + <seealso marker="fsm"><c>gen_fsm</c> Behavior</seealso>. + In later sections, additions and tweaks are made using features in <c>gen_statem</c> that <c>gen_fsm</c> does not have. - At the end of this section you can find the example again + The end of this chapter provides the example again with all the added features. </p> <p> - A door with a code lock can be viewed as a state machine. - Initially, the door is locked. Anytime someone presses a button, - this generates an event. + A door with a code lock can be seen as a state machine. + Initially, the door is locked. When someone presses a button, + an event is generated. Depending on what buttons have been pressed before, the sequence so far can be correct, incomplete, or wrong. - </p> - <p> - If it is correct, the door is unlocked for 10 seconds (10000 ms). - If it is incomplete, we wait for another button to be pressed. If - it is is wrong, we start all over, - waiting for a new button sequence. + If correct, the door is unlocked for 10 seconds (10,000 milliseconds). + If incomplete, we wait for another button to be pressed. If + wrong, we start all over, waiting for a new button sequence. </p> <image file="../design_principles/code_lock.png"> - <icaption>Code lock state diagram</icaption> + <icaption>Code Lock State Diagram</icaption> </image> <p> - We can implement such a code lock state machine using + This code lock state machine can be implemented using <c>gen_statem</c> with the following callback module: </p> <marker id="ex"></marker> @@ -241,7 +239,6 @@ start_link(Code) -> button(Digit) -> gen_statem:cast(?NAME, {button,Digit}). - init(Code) -> do_lock(), Data = #{code => Code, remaining => Code}, @@ -286,7 +283,7 @@ code_change(_Vsn, State, Data, _Extra) -> <section> <title>Starting gen_statem</title> <p> - In the example in the previous section, the <c>gen_statem</c> is + In the example in the previous section, <c>gen_statem</c> is started by calling <c>code_lock:start_link(Code)</c>: </p> <code type="erl"><![CDATA[ @@ -294,80 +291,71 @@ start_link(Code) -> gen_statem:start_link({local,?NAME}, ?MODULE, Code, []). ]]></code> <p> - <c>start_link</c> calls the function - <seealso marker="stdlib:gen_statem#start_link/4"> - <c>gen_statem:start_link/4</c> - </seealso> - which spawns and links to a new process; a <c>gen_statem</c>. + <c>start_link</c> calls function + <seealso marker="stdlib:gen_statem#start_link/4"><c>gen_statem:start_link/4</c></seealso>, + which spawns and links to a new process, a <c>gen_statem</c>. </p> <list type="bulleted"> <item> <p> - The first argument, <c>{local,?NAME}</c>, specifies - the name. In this case, the <c>gen_statem</c> is locally - registered as <c>code_lock</c> through the macro <c>?NAME</c>. - </p> + The first argument, <c>{local,?NAME}</c>, specifies + the name. In this case, the <c>gen_statem</c> is locally + registered as <c>code_lock</c> through the macro <c>?NAME</c>. + </p> <p> - If the name is omitted, the <c>gen_statem</c> is not registered. - Instead its pid must be used. The name can also be given - as <c>{global,Name}</c>, in which case the <c>gen_statem</c> is - registered using - <seealso marker="kernel:global#register_name/2"> - <c>global:register_name/2</c>. - </seealso> - </p> + If the name is omitted, the <c>gen_statem</c> is not registered. + Instead its pid must be used. The name can also be specified + as <c>{global,Name}</c>, then the <c>gen_statem</c> is + registered using + <seealso marker="kernel:global#register_name/2"><c>global:register_name/2</c></seealso> + in <c>Kernel</c>. + </p> </item> <item> <p> - The second argument, <c>?MODULE</c>, is the name of - the callback module, that is; the module where the callback + The second argument, <c>?MODULE</c>, is the name of + the callback module, that is, the module where the callback functions are located, which is this module. - </p> + </p> <p> - The interface functions (<c>start_link/1</c> and <c>button/1</c>) - are located in the same module as the callback functions - (<c>init/1</c>, <c>locked/3</c>, and <c>open/3</c>). - It is normally good programming practice to have the client - side and the server side code contained in one module. - </p> + The interface functions (<c>start_link/1</c> and <c>button/1</c>) + are located in the same module as the callback functions + (<c>init/1</c>, <c>locked/3</c>, and <c>open/3</c>). + It is normally good programming practice to have the client-side + code and the server-side code contained in one module. + </p> </item> <item> <p> - The third argument, <c>Code</c>, is a list of digits that - is the correct unlock code which is passsed - to the callback function <c>init/1</c>. + The third argument, <c>Code</c>, is a list of digits, which + is the correct unlock code that is passed + to callback function <c>init/1</c>. </p> </item> <item> <p> - The fourth argument, <c>[]</c>, is a list of options. See the - <seealso marker="stdlib:gen_statem#start_link/3"> - <c>gen_statem:start_link/3</c> - </seealso> - manual page for available options. + The fourth argument, <c>[]</c>, is a list of options. + For the available options, see + <seealso marker="stdlib:gen_statem#start_link/3"><c>gen_statem:start_link/3</c></seealso>. </p> </item> </list> <p> If name registration succeeds, the new <c>gen_statem</c> process - calls the callback function <c>code_lock:init(Code)</c>. + calls callback function <c>code_lock:init(Code)</c>. This function is expected to return <c>{CallbackMode,State,Data}</c>, where - <seealso marker="#callback_modes"> - <c>CallbackMode</c> - </seealso> + <seealso marker="#callback_modes"><c>CallbackMode</c></seealso> selects callback module state function mode, in this case - <seealso marker="stdlib:gen_statem#type-callback_mode"> - <c>state_functions</c> - </seealso> - through the macro <c>?CALLBACK_MODE</c> that is; each state + <seealso marker="stdlib:gen_statem#type-callback_mode"><c>state_functions</c></seealso> + through macro <c>?CALLBACK_MODE</c>. That is, each state has got its own handler function. <c>State</c> is the initial state of the <c>gen_statem</c>, - in this case <c>locked</c>; assuming the door is locked to begin with. - <c>Data</c> is the internal server data of the <c>gen_statem</c>. + in this case <c>locked</c>; assuming that the door is locked to begin + with. <c>Data</c> is the internal server data of the <c>gen_statem</c>. Here the server data is a <seealso marker="stdlib:maps">map</seealso> - with the key <c>code</c> that stores - the correct button sequence and the key <c>remaining</c> + with key <c>code</c> that stores + the correct button sequence, and key <c>remaining</c> that stores the remaining correct button sequence (the same as the <c>code</c> to begin with). </p> @@ -377,24 +365,19 @@ init(Code) -> Data = #{code => Code, remaining => Code}, {?CALLBACK_MODE,locked,Data}. ]]></code> - <p> - <seealso marker="stdlib:gen_statem#start_link/3"> - <c>gen_statem:start_link</c> - </seealso> - is synchronous. It does not return until the <c>gen_statem</c> - has been initialized and is ready to receive events. + <p>Function + <seealso marker="stdlib:gen_statem#start_link/3"><c>gen_statem:start_link</c></seealso> + is synchronous. It does not return until the <c>gen_statem</c> + is initialized and is ready to receive events. </p> <p> - <seealso marker="stdlib:gen_statem#start_link/3"> - <c>gen_statem:start_link</c> - </seealso> + Function + <seealso marker="stdlib:gen_statem#start_link/3"><c>gen_statem:start_link</c></seealso> must be used if the <c>gen_statem</c> - is part of a supervision tree, that is; started by a supervisor. - There is another function; - <seealso marker="stdlib:gen_statem#start/3"> - <c>gen_statem:start</c> - </seealso> - to start a standalone <c>gen_statem</c>, that is; + is part of a supervision tree, that is, started by a supervisor. + Another function, + <seealso marker="stdlib:gen_statem#start/3"><c>gen_statem:start</c></seealso> + can be used to start a standalone <c>gen_statem</c>, that is, a <c>gen_statem</c> that is not part of a supervision tree. </p> </section> @@ -402,12 +385,10 @@ init(Code) -> <!-- =================================================================== --> <section> - <title>Events and Handling them</title> + <title>Handling Events</title> <p>The function notifying the code lock about a button event is implemented using - <seealso marker="stdlib:gen_statem#cast/2"> - <c>gen_statem:cast/2</c>: - </seealso> + <seealso marker="stdlib:gen_statem#cast/2"><c>gen_statem:cast/2</c></seealso>: </p> <code type="erl"><![CDATA[ button(Digit) -> @@ -415,9 +396,9 @@ button(Digit) -> ]]></code> <p> The first argument is the name of the <c>gen_statem</c> and must - agree with the name used to start it so therefore we use the + agree with the name used to start it. So, we use the same macro <c>?NAME</c> as when starting. - <c>{button,Digit}</c> is the actual event content. + <c>{button,Digit}</c> is the event content. </p> <p> The event is made into a message and sent to the <c>gen_statem</c>. @@ -452,19 +433,19 @@ open(cast, {button,_}, Data) -> ]]></code> <p> If the door is locked and a button is pressed, the pressed - button is compared with the next correct button and, - depending on the result, the door is either unlocked + button is compared with the next correct button. + Depending on the result, the door is either unlocked and the <c>gen_statem</c> goes to state <c>open</c>, or the door remains in state <c>locked</c>. </p> <p> - If the pressed button is incorrect the server data + If the pressed button is incorrect, the server data restarts from the start of the code sequence. </p> <p> - In state <c>open</c> any button locks the door since - any event cancels the event timer so we will not get - a timeout event after a button event. + In state <c>open</c>, any button locks the door, as + any event cancels the event timer, so no + time-out event occurs after a button event. </p> </section> @@ -478,11 +459,11 @@ open(cast, {button,_}, Data) -> {next_state,open,Data#{remaining := Code},10000}; ]]></code> <p> - 10000 is a time-out value in milliseconds. - After this time, that is; 10 seconds, a time-out occurs. + 10,000 is a time-out value in milliseconds. + After this time (10 seconds), a time-out occurs. Then, <c>StateName(timeout, 10000, Data)</c> is called. The time-out occurs when the door has been in state <c>open</c> - for 10 seconds. After that the door is locked again: + for 10 seconds. After that the door is locked again: </p> <code type="erl"><![CDATA[ open(timeout, _, Data) -> @@ -496,16 +477,16 @@ open(timeout, _, Data) -> <section> <title>All State Events</title> <p> - Sometimes an event can arrive in any state of the <c>gen_statem</c>. + Sometimes events can arrive in any state of the <c>gen_statem</c>. It is convenient to handle these in a common state handler function that all state functions call for events not specific to the state. </p> <p> - Let's introduce a <c>code_length/0</c> function that returns + Consider a <c>code_length/0</c> function that returns the length of the correct code - (that should not be sensitive to reveal...). - We'll dispatch all events that are not state specific - to the common function <c>handle_event/3</c>. + (that should not be sensitive to reveal). + We dispatch all events that are not state-specific + to the common function <c>handle_event/3</c>: </p> <code type="erl"><![CDATA[ ... @@ -530,9 +511,7 @@ handle_event({call,From}, code_length, #{code := Code} = Data) -> ]]></code> <p> This example uses - <seealso marker="stdlib:gen_statem#call/2"> - <c>gen_statem:call/2</c> - </seealso> + <seealso marker="stdlib:gen_statem#call/2"><c>gen_statem:call/2</c></seealso>, which waits for a reply from the server. The reply is sent with a <c>{reply,From,Reply}</c> tuple in an action list in the <c>{keep_state,...}</c> tuple @@ -545,14 +524,16 @@ handle_event({call,From}, code_length, #{code := Code} = Data) -> <section> <title>One Event Handler</title> <p> - If you use the mode <c>handle_event_function</c> - all events are handled in <c>handle_event/4</c> and we - may (but do not have to) use an event-centered approach - where we dispatch on event first and then state: + If mode <c>handle_event_function</c> is used, + all events are handled in + <seealso marker="stdlib:gen_statem#Module:handle_event/4"><c>Module:handle_event/4</c></seealso> + and we can (but do not have to) use an event-centered approach + where we first branch depending on event + and then depending on state: </p> <code type="erl"><![CDATA[ ... --define(CALLBACK_MODE, state_functions). +-define(CALLBACK_MODE, handle_event_function). ... -export([handle_event/4]). @@ -596,19 +577,14 @@ handle_event(timeout, _, open, Data) -> The <c>gen_statem</c> is automatically terminated by its supervisor. Exactly how this is done is defined by a <seealso marker="sup_princ#shutdown">shutdown strategy</seealso> - set in the supervisor. + set in the supervisor. </p> <p> If it is necessary to clean up before termination, the shutdown - strategy must be a time-out value and the <c>gen_statem</c> must - in the <c>init/1</c> function set itself to trap exit signals + strategy must be a time-out value and the <c>gen_statem</c> must + in function <c>init/1</c> set itself to trap exit signals by calling - <seealso marker="erts:erlang#process_flag/2"> - <c>process_flag(trap_exit, true)</c>. - </seealso> - When ordered to shutdown, the <c>gen_statem</c> then calls - the callback function - <c>terminate(shutdown, State, Data)</c>: + <seealso marker="erts:erlang#process_flag/2"><c>process_flag(trap_exit, true)</c></seealso>: </p> <code type="erl"><![CDATA[ init(Args) -> @@ -617,9 +593,13 @@ init(Args) -> ... ]]></code> <p> - In this example we let the <c>terminate/3</c> function - lock the door if it is open so we do not accidentally leave the door - open when the supervision tree terminates. + When ordered to shut down, the <c>gen_statem</c> then calls + callback function <c>terminate(shutdown, State, Data)</c>. + </p> + <p> + In the following example, function <c>terminate/3</c> + locks the door if it is open, so we do not accidentally leave the door + open when the supervision tree terminates: </p> <code type="erl"><![CDATA[ terminate(_Reason, State, _Data) -> @@ -633,9 +613,7 @@ terminate(_Reason, State, _Data) -> <p> If the <c>gen_statem</c> is not part of a supervision tree, it can be stopped using - <seealso marker="stdlib:gen_statem#stop/1"> - <c>gen_statem:stop</c>, - </seealso> + <seealso marker="stdlib:gen_statem#stop/1"><c>gen_statem:stop</c></seealso>, preferably through an API function: </p> <code type="erl"><![CDATA[ @@ -647,8 +625,8 @@ stop() -> gen_statem:stop(?NAME). ]]></code> <p> - This makes the <c>gen_statem</c> call the <c>terminate/3</c> - callback function just like for a supervised server + This makes the <c>gen_statem</c> call callback function + <c>terminate/3</c> just like for a supervised server and waits for the process to terminate. </p> </section> @@ -659,48 +637,41 @@ stop() -> <section> <title>Actions</title> <p> - In the first chapters we mentioned actions as a part of - the general state machine model, and these actions - are implemented with the code the <c>gen_statem</c> - callback module executes in an event handling + In the first sections actions were mentioned as a part of + the general state machine model. These general actions + are implemented with the code that callback module + <c>gen_statem</c> executes in an event-handling callback function before returning to the <c>gen_statem</c> engine. </p> <p> - There are more specific state transition actions + There are more specific state-transition actions that a callback function can order the <c>gen_statem</c> engine to do after the callback function return. These are ordered by returning a list of - <seealso marker="stdlib:gen_statem#type-action"> - actions - </seealso> + <seealso marker="stdlib:gen_statem#type-action">actions</seealso> in the - <seealso marker="stdlib:gen_statem#type-state_function_result"> - return tuple - </seealso> + <seealso marker="stdlib:gen_statem#type-state_function_result">return tuple</seealso> from the - <seealso marker="stdlib:gen_statem#Module:StateName/3"> - callback function. - </seealso> + <seealso marker="stdlib:gen_statem#Module:StateName/3">callback function</seealso>. These state transition actions affect the <c>gen_statem</c> - engine itself. They can: + engine itself and can do the following: </p> <list type="bulleted"> - <item>Postpone the current event.</item> - <item>Hibernate the <c>gen_statem</c>.</item> - <item>Start an event timeout.</item> - <item>Reply to a caller.</item> - <item>Generate the next event to handle.</item> + <item>Postpone the current event</item> + <item>Hibernate the <c>gen_statem</c></item> + <item>Start an event time-out</item> + <item>Reply to a caller</item> + <item>Generate the next event to handle</item> </list> <p> - We have mentioned the event timeout - and replying to a caller in the example above. - An example of event postponing comes in later in this chapter. - See the - <seealso marker="stdlib:gen_statem#type-action"> - reference manual - </seealso> - for details. You can for example actually reply to several callers + In the example earlier was mentioned the event time-out + and replying to a caller. + An example of event postponing is included later in this chapter. + For details, see the + <seealso marker="stdlib:gen_statem#type-action"><c>gen_statem(3)</c></seealso> + manual page. + You can, for example, reply to many callers and generate multiple next events to handle. </p> </section> @@ -710,38 +681,30 @@ stop() -> <section> <title>Event Types</title> <p> - So far we have mentioned a few - <seealso marker="stdlib:gen_statem#type-event_type"> - event types. - </seealso> + The previous sections mentioned a few + <seealso marker="stdlib:gen_statem#type-event_type">event types</seealso>. Events of all types are handled in the same callback function, for a given state, and the function gets <c>EventType</c> and <c>EventContent</c> as arguments. </p> <p> - Here is the complete list of event types and where + The following is a complete list of event types and where they come from: </p> <taglist> <tag><c>cast</c></tag> <item> Generated by - <seealso marker="stdlib:gen_statem#cast/2"> - <c>gen_statem:cast</c>. - </seealso> + <seealso marker="stdlib:gen_statem#cast/2"><c>gen_statem:cast</c></seealso>. </item> <tag><c>{call,From}</c></tag> <item> Generated by - <seealso marker="stdlib:gen_statem#call/2"> - <c>gen_statem:call</c> - </seealso> + <seealso marker="stdlib:gen_statem#call/2"><c>gen_statem:call</c></seealso>, where <c>From</c> is the reply address to use when replying either through the state transition action <c>{reply,From,Msg}</c> or by calling - <seealso marker="stdlib:gen_statem#reply/1"> - <c>gen_statem:reply</c>. - </seealso> + <seealso marker="stdlib:gen_statem#reply/1"><c>gen_statem:reply</c></seealso>. </item> <tag><c>info</c></tag> <item> @@ -750,15 +713,15 @@ stop() -> </item> <tag><c>timeout</c></tag> <item> - Generated by the state transition action + Generated by state transition action <c>{timeout,Time,EventContent}</c> (or its short form <c>Time</c>) timer timing out. </item> <tag><c>internal</c></tag> <item> - Generated by the state transition action + Generated by state transition action <c>{next_event,internal,EventContent}</c>. - In fact all event types above can be generated using + All event types above can also be generated using <c>{next_event,EventType,EventContent}</c>. </item> </taglist> @@ -767,34 +730,32 @@ stop() -> <!-- =================================================================== --> <section> - <title>State Timeouts</title> + <title>State Time-Outs</title> <p> - The timeout event generated by the state transition action - <c>{timeout,Time,EventContent}</c> is an event timeout, - that is; if an event arrives the timer is cancelled. - You get either an event or a timeout but not both. + The time-out event generated by state transition action + <c>{timeout,Time,EventContent}</c> is an event time-out, + that is, if an event arrives the timer is cancelled. + You get either an event or a time-out, but not both. </p> <p> - Often you want a timer to not be cancelled by any event + Often you want a timer not to be cancelled by any event or you want to start a timer in one state and respond - to the timeout in another. This can be accomplished - with a regular erlang timer: - <seealso marker="erts:erlang#start_timer/4"> - <c>erlang:start_timer</c>. - </seealso> + to the time-out in another. This can be accomplished + with a regular Erlang timer: + <seealso marker="erts:erlang#start_timer/4"><c>erlang:start_timer</c></seealso>. </p> <p> - Looking at the example in this chapter so far; using the + For the example so far in this chapter: using the <c>gen_statem</c> event timer has the consequence that if a button event is generated while in the <c>open</c> state, - the timeout is cancelled and the button event is delivered. - Therefore we chose to lock the door if this happended. + the time-out is cancelled and the button event is delivered. + So, we choose to lock the door if this occurred. </p> <p> - Suppose we do not want a button to lock the door, + Suppose that we do not want a button to lock the door, instead we want to ignore button events in the <c>open</c> state. Then we start a timer when entering the <c>open</c> state - and wait for it to expire while ignoring button events: + and waits for it to expire while ignoring button events: </p> <code type="erl"><![CDATA[ ... @@ -816,17 +777,15 @@ open(cast, {button,_}, Data) -> ... ]]></code> <p> - If you need to cancel a timer due to some other event you can use - <seealso marker="erts:erlang#cancel_timer/2"> - <c>erlang:cancel_timer(Tref)</c>. - </seealso> - Note that a timeout message can not arrive after this, + If you need to cancel a timer because of some other event, you can use + <seealso marker="erts:erlang#cancel_timer/2"><c>erlang:cancel_timer(Tref)</c></seealso>. + Notice that a time-out message cannot arrive after this, unless you have postponed it (see the next section) before, - so make sure you do not accidentally postpone such messages. + so ensure that you do not accidentally postpone such messages. </p> <p> - Another way to cancel a timer is to not cancel it, - but instead to ignore it if it arrives in a state + Another way to cancel a timer is not to cancel it, + but to ignore it if it arrives in a state where it is known to be late. </p> </section> @@ -839,19 +798,17 @@ open(cast, {button,_}, Data) -> If you want to ignore a particular event in the current state and handle it in a future state, you can postpone the event. A postponed event is retried after the state has - changed i.e <c>OldState =/= NewState</c>. + changed, that is, <c>OldState =/= NewState</c>. </p> <p> Postponing is ordered by the state transition - <seealso marker="stdlib:gen_statem#type-action"> - action - </seealso> + <seealso marker="stdlib:gen_statem#type-action">action</seealso> <c>postpone</c>. </p> <p> In this example, instead of ignoring button events - while in the <c>open</c> state we can postpone them - and they will be queued and later handled in the <c>locked</c> state: + while in the <c>open</c> state, we can postpone them + and they are queued and later handled in the <c>locked</c> state: </p> <code type="erl"><![CDATA[ ... @@ -861,16 +818,16 @@ open(cast, {button,_}, Data) -> ]]></code> <p> The fact that a postponed event is only retried after a state change - translates into a requirement on the event and state space: - if you have a choice between storing a state data item - in the <c>State</c> or in the <c>Data</c>; - should a change in the item value affect which events that - are handled, then this item ought to be part of the state. + translates into a requirement on the event and state space. + If you have a choice between storing a state data item + in the <c>State</c> or in the <c>Data</c>: + if a change in the item value affects which events that + are handled, then this item is to be part of the state. </p> <p> - What you want to avoid is that you maybe much later decide - to postpone an event in one state and by misfortune it is never retried - because the code only changes the <c>Data</c> but not the <c>State</c>. + You want to avoid that you maybe much later decide + to postpone an event in one state and by misfortune it is never retried, + as the code only changes the <c>Data</c> but not the <c>State</c>. </p> <section> @@ -883,7 +840,7 @@ open(cast, {button,_}, Data) -> or from the context. </p> <p> - Possible actions may be; ignore as in drop the event + Possible actions: ignore as in drop the event (maybe log it) or deal with the event in some other state as in postpone it. </p> @@ -892,10 +849,10 @@ open(cast, {button,_}, Data) -> <section> <title>Selective Receive</title> <p> - Erlang's selective receive statement is often used to - describe simple state machine examples in straightforward - Erlang code. Here is a possible implementation of - the first example: + Erlang's selective receive statement is often used to + describe simple state machine examples in straightforward + Erlang code. The following is a possible implementation of + the first example: </p> <code type="erl"><![CDATA[ -module(code_lock). @@ -937,33 +894,31 @@ do_unlock() -> io:format("Open~n", []). ]]></code> <p> - The selective receive in this case causes <c>open</c> - to implicitly postpone any events to the <c>locked</c> state. + The selective receive in this case causes implicitly <c>open</c> + to postpone any events to the <c>locked</c> state. </p> <p> - A selective receive can not be used from a <c>gen_statem</c> - behaviour just as for any <c>gen_*</c> behavior - since the receive statement is within the <c>gen_*</c> engine itself. - It has to be there because all + A selective receive cannot be used from a <c>gen_statem</c> + behavior as for any <c>gen_*</c> behavior, + as the receive statement is within the <c>gen_*</c> engine itself. + It must be there because all <seealso marker="stdlib:sys"><c>sys</c></seealso> - compatible behaviours must respond to system messages and therefore + compatible behaviors must respond to system messages and therefore do that in their engine receive loop, passing non-system messages to the callback module. </p> <p> The state transition - <seealso marker="stdlib:gen_statem#type-action"> - action - </seealso> - <c>postpone</c> is designed to be able to model - selective receives. A selective receive implicitly postpones + <seealso marker="stdlib:gen_statem#type-action">action</seealso> + <c>postpone</c> is designed to model + selective receives. A selective receive implicitly postpones any not received events, but the <c>postpone</c> state transition action explicitly postpones one received event. </p> <p> - Other than that both mechanisms have got the same theoretical + Both mechanisms have the same theoretical time and memory complexity, while the selective receive - language construct has got smaller constant factors. + language construct has smaller constant factors. </p> </section> </section> @@ -971,43 +926,39 @@ do_unlock() -> <!-- =================================================================== --> <section> - <title>Self Generated Events</title> + <title>Self-Generated Events</title> <p> - It may be beneficial in some cases to be able to generate events + It can sometimes be beneficial to be able to generate events to your own state machine. This can be done with the state transition - <seealso marker="stdlib:gen_statem#type-action"> - action - </seealso> + <seealso marker="stdlib:gen_statem#type-action">action</seealso> <c>{next_event,EventType,EventContent}</c>. </p> <p> You can generate events of any existing - <seealso marker="stdlib:gen_statem#type-action"> - type, - </seealso> - but the <c>internal</c> type can only be generated through the - <c>next_event</c> action and hence can not come from an external source, + <seealso marker="stdlib:gen_statem#type-action">type</seealso>, + but the <c>internal</c> type can only be generated through action + <c>next_event</c>. Hence, it cannot come from an external source, so you can be certain that an <c>internal</c> event is an event from your state machine to itself. </p> <p> - One example of using self generated events may be when you have + One example of using self-generated events can be when you have a state machine specification that uses state entry actions. - That you could code using a dedicated function - to do the state transition. But if you want that code to be - visible besides the other state logic you can insert + You can code that using a dedicated function + to do the state transition. But if you want that code to be + visible besides the other state logic, you can insert an <c>internal</c> event that does the entry actions. This has the same unfortunate consequence as using - state transition functions that everywhere you go to - the state in question you will have to explicitly + state transition functions: everywhere you go to + the state, you must explicitly insert the <c>internal</c> event - or use state transition function. + or use a state transition function. </p> <p> - Here is an implementation of entry actions + The following is an implementation of entry actions using <c>internal</c> events with content <c>enter</c> - utilizing a helper function <c>enter/3</c> for state entry: + using a helper function <c>enter/3</c> for state entry: </p> <code type="erl"><![CDATA[ ... @@ -1051,20 +1002,20 @@ enter(Tag, State, Data) -> <section> <title>Example Revisited</title> <p> - Here is the example after all mentioned modifications - and some more utilizing the entry actions, + This section includes the example after all mentioned modifications + and some more using the entry actions, which deserves a new state diagram: </p> <image file="../design_principles/code_lock_2.png"> - <icaption>Code lock state diagram revisited</icaption> + <icaption>Code Lock State Diagram Revisited</icaption> </image> <p> - Note that this state diagram does not specify how to handle - a button event in the state <c>open</c>, so you will have to - read some other place that is here that unspecified events - shall be ignored as in not consumed but handled in some other state. - Nor does it show that the <c>code_length/0</c> call shall be - handled in every state. + Notice that this state diagram does not specify how to handle + a button event in the state <c>open</c>. So, you need to + read somewhere else that unspecified events + must be ignored as in not consumed but handled in some other state. + Also, the state diagram does not show that the <c>code_length/0</c> + call must be handled in every state. </p> <section> @@ -1147,10 +1098,11 @@ code_change(_Vsn, State, Data, _Extra) -> <section> <title>Callback Mode: handle_event_function</title> <p> - What to change to use one <c>handle_event/4</c> function. - Here a clean first-dispatch-on-event approach - does not work that well due to the generated - entry actions: + This section describes what to change in the example + to use one <c>handle_event/4</c> function. + The previously used approach to first branch depending on event + does not work that well here because of the generated + entry actions, so this example first branches depending on state: </p> <code type="erl"><![CDATA[ ... @@ -1195,7 +1147,7 @@ handle_event({call,From}, code_length, _State, #{code := Code}) -> ]]></code> </section> <p> - Note that postponing buttons from the <c>locked</c> state + Notice that postponing buttons from the <c>locked</c> state to the <c>open</c> state feels like the wrong thing to do for a code lock, but it at least illustrates event postponing. </p> @@ -1206,33 +1158,29 @@ handle_event({call,From}, code_length, _State, #{code := Code}) -> <section> <title>Filter the State</title> <p> - The example servers so far in this chapter will for example - when killed by an exit signal or due to an internal error - print out the full internal state in the error log. + The example servers so far in this chapter + print the full internal state in the error log, for example, + when killed by an exit signal or because of an internal error. This state contains both the code lock code - and which digits that remains to unlock. + and which digits that remain to unlock. </p> <p> This state data can be regarded as sensitive, and maybe not what you want in the error log - because of something unpredictable happening. + because of some unpredictable event. </p> <p> Another reason to filter the state can be - that the state is too big to print out since it fills + that the state is too large to print, as it fills the error log with uninteresting details. </p> <p> - To avoid this you can format the internal state + To avoid this, you can format the internal state that gets in the error log and gets returned from - <seealso marker="stdlib:sys#get_status/1"> - <c>sys:get_status/1,2</c> - </seealso> - by implementing the - <seealso marker="stdlib:gen_statem#Module:format_status/2"> - <c>Module:format_status/2</c> - </seealso> - function, for example like this: + <seealso marker="stdlib:sys#get_status/1"><c>sys:get_status/1,2</c></seealso> + by implementing function + <seealso marker="stdlib:gen_statem#Module:format_status/2"><c>Module:format_status/2</c></seealso>, + for example like this: </p> <code type="erl"><![CDATA[ ... @@ -1257,12 +1205,10 @@ format_status(Opt, [_PDict,State,Data]) -> ]]></code> <p> It is not mandatory to implement a - <seealso marker="stdlib:gen_statem#Module:format_status/2"> - <c>Module:format_status/2</c> - </seealso> - function. If you do not a default implementation is used that + <seealso marker="stdlib:gen_statem#Module:format_status/2"><c>Module:format_status/2</c></seealso> + function. If you do not, a default implementation is used that does the same as this example function without filtering - the <c>Data</c> term that is: <c>StateData = {State,Data}</c>. + the <c>Data</c> term, that is, <c>StateData = {State,Data}</c>. </p> </section> @@ -1272,57 +1218,54 @@ format_status(Opt, [_PDict,State,Data]) -> <title>Complex State</title> <p> The callback mode - <seealso marker="stdlib:gen_statem#type-callback_mode"> - <c>handle_event_function</c> - </seealso> - enables using a non-atom state as described in - <seealso marker="#callback_modes"> - Callback Modes, - </seealso> - for example a complex state term like a tuple. + <seealso marker="stdlib:gen_statem#type-callback_mode"><c>handle_event_function</c></seealso> + enables using a non-atom state as described in section + <seealso marker="#callback_modes">Callback Modes</seealso>, + for example, a complex state term like a tuple. </p> <p> One reason to use this is when you have - a state item that affects the event handling - in particular when combining that with postponing events. - Let us complicate the previous example + a state item that affects the event handling, + in particular in combination with postponing events. + We complicate the previous example by introducing a configurable lock button - (this is the state item in question) - that in the <c>open</c> state immediately locks the door, + (this is the state item in question), + which in the <c>open</c> state immediately locks the door, and an API function <c>set_lock_button/1</c> to set the lock button. </p> <p> Suppose now that we call <c>set_lock_button</c> while the door is open, and have already postponed a button event - that up until now was not the lock button; - the sensible thing might be to say that - the button was pressed too early so it should - not be recognized as the lock button, - but then it might be surprising that a button event + that until now was not the lock button. + The sensible thing can be to say that + the button was pressed too early so it is + not to be recognized as the lock button. + However, then it can be surprising that a button event that now is the lock button event arrives (as retried postponed) immediately after the state transits to <c>locked</c>. </p> <p> - So let us make the <c>button/1</c> function synchronous - by using <c>gen_statem:call</c>, + So we make the <c>button/1</c> function synchronous + by using + <seealso marker="stdlib:gen_statem#call/2"><c>gen_statem:call</c></seealso> and still postpone its events in the <c>open</c> state. Then a call to <c>button/1</c> during the <c>open</c> - state will not return until the state transits to <c>locked</c> - since it is there the event is handled and the reply is sent. + state does not return until the state transits to <c>locked</c>, + as it is there the event is handled and the reply is sent. </p> <p> - If now one process calls <c>set_lock_button/1</c> - to change the lock button while some other process - hangs in <c>button/1</c> with the new lock button - it could be expected that the hanging lock button call + If a process now calls <c>set_lock_button/1</c> + to change the lock button while another process + hangs in <c>button/1</c> with the new lock button, + it can be expected that the hanging lock button call immediately takes effect and locks the lock. - Therefore we make the current lock button a part of the state - so when we change the lock button the state will change - and all postponed events will be retried. + Therefore, we make the current lock button a part of the state, + so that when we change the lock button, the state changes + and all postponed events are retried. </p> <p> - We define the state as <c>{StateName,LockButton}</c> + We define the state as <c>{StateName,LockButton}</c>, where <c>StateName</c> is as before and <c>LockButton</c> is the current lock button: </p> @@ -1441,10 +1384,9 @@ format_status(Opt, [_PDict,State,Data]) -> end. ]]></code> <p> - It may be an ill-fitting model for a physical code lock - that the <c>button/1</c> call might hang until the lock - is locked. But for an API in general it is really not - that strange. + It can be an ill-fitting model for a physical code lock + that the <c>button/1</c> call can hang until the lock + is locked. But for an API in general it is not that strange. </p> </section> @@ -1457,26 +1399,21 @@ format_status(Opt, [_PDict,State,Data]) -> and they have some state(s) in their lifetime in which the servers can be expected to idle for a while, and the amount of heap memory all these servers need - is a problem; then it is possible to minimize - the memory footprint of a server by hibernating it through - <seealso marker="stdlib:proc_lib#hibernate/3"> - <c>proc_lib:hibernate/3</c>. - </seealso> + is a problem, then the memory footprint of a server + can be mimimized by hibernating it through + <seealso marker="stdlib:proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>. </p> <note> <p> - To hibernate a process is rather costly. See - <seealso marker="erts:erlang#hibernate/3"> - <c>erlang:hibernate/3</c>. - </seealso> - It is in general not something you want to do - after every event. + It is rather costly to hibernate a process; see + <seealso marker="erts:erlang#hibernate/3"><c>erlang:hibernate/3</c></seealso>. + It is not something you want to do after every event. </p> </note> <p> - We can in this example hibernate in the <c>{open,_}</c> state - since what normally happens in that state is that - the state timeout after a while + We can in this example hibernate in the <c>{open,_}</c> state, + because what normally occurs in that state is that + the state time-out after a while triggers a transition to <c>{locked,_}</c>: </p> <code type="erl"><![CDATA[ @@ -1493,29 +1430,27 @@ handle_event( ]]></code> <p> The - <seealso marker="stdlib:gen_statem#type-hibernate"> - <c>[hibernate]</c> - </seealso> + <seealso marker="stdlib:gen_statem#type-hibernate"><c>[hibernate]</c></seealso> action list on the last line when entering the <c>{open,_}</c> state is the only change. - If any event arrives in the <c>{open,_},</c> state we - do not bother to re-hibernate, so the server stays + If any event arrives in the <c>{open,_},</c> state, we + do not bother to rehibernate, so the server stays awake after any event. </p> <p> To change that we would need to insert - the <c>hibernate</c> action in more places, - for example for the state independent <c>set_lock_button</c> + action <c>hibernate</c> in more places. + For example, for the state-independent <c>set_lock_button</c> and <c>code_length</c> operations that then would have to be aware of using <c>hibernate</c> while in the - <c>{open,_}</c> state which would clutter the code. + <c>{open,_}</c> state, which would clutter the code. </p> <p> - This server probably does not use an amount of + This server probably does not use heap memory worth hibernating for. - To gain anything from hibernation your server would - have to actually produce some garbage during callback execution, - for which this example server may serve as a bad example. + To gain anything from hibernation, your server would + have to produce some garbage during callback execution, + for which this example server can serve as a bad example. </p> </section> |