diff options
58 files changed, 2068 insertions, 779 deletions
diff --git a/HOWTO/INSTALL.md b/HOWTO/INSTALL.md index 551ecdd224..447a23128f 100644 --- a/HOWTO/INSTALL.md +++ b/HOWTO/INSTALL.md @@ -568,6 +568,10 @@ as before, but the build process will take a much longer time. > automatically when `make` is invoked from `$ERL_TOP` with either the > `clean` target, or the default target. It is also automatically invoked > if `./otp_build remove_prebuilt_files` is invoked. +> +> If you need to verify the bootstrap beam files match the provided +> source files, use `./otp_build update_primary` to create a new commit that +> contains differences, if any exist. #### How to Build a Debug Enabled Erlang RunTime System #### diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index eb1d24cf12..f39b640c7e 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -250,6 +250,11 @@ environment variable <c><![CDATA[DISPLAY]]></c> set to <c><![CDATA[gin:0]]></c>.</p> </item> + <tag><c><![CDATA[-epmd_module Module]]></c> (init flag)</tag> + <item> + <p>Configures the module responsible to communicate to + <seealso marker="epmd">epmd</seealso>. Defaults to <c>erl_epmd</c>.</p> + </item> <tag><c><![CDATA[-eval Expr]]></c> (init flag)</tag> <item> <p>Makes <c><![CDATA[init]]></c> evaluate the expression diff --git a/erts/doc/src/erl_driver.xml b/erts/doc/src/erl_driver.xml index 5191742bc9..d8bf45c523 100644 --- a/erts/doc/src/erl_driver.xml +++ b/erts/doc/src/erl_driver.xml @@ -2039,7 +2039,7 @@ r = driver_async(myPort, &myKey, myData, myFunc); ]]></code> <fsummary>Set and get limits for busy port message queue.</fsummary> <desc> <marker id="erl_drv_busy_msgq_limits"></marker> - <p>Sets and gets limits that will be used for controling the + <p>Sets and gets limits that will be used for controlling the busy state of the port message queue.</p> <p>The port message queue is set into a busy state when the amount of command data queued on the @@ -2112,7 +2112,7 @@ r = driver_async(myPort, &myKey, myData, myFunc); ]]></code> It is used to identify the condition variable in planned future debug functionality.</p> <p>Returns <c>NULL</c> on failure. The driver - creating the condition variable is responsibile for + creating the condition variable is responsible for destroying it before the driver is unloaded.</p> <p>This function is thread-safe.</p> </desc> diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index b0d25389fd..9646953518 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -9529,6 +9529,10 @@ timestamp() -> <type name="trace_info_item_result"/> <type name="trace_info_flag"/> <type name="trace_match_spec"/> + <type name="match_variable"/> + <type_desc name="match_variable"> + Approximation of '$1' | '$2' | '$3' | ... + </type_desc> <desc> <p>Returns trace information about a port, process, function, or event.</p> @@ -9660,6 +9664,10 @@ timestamp() -> </fsummary> <type name="trace_pattern_mfa"/> <type name="trace_match_spec"/> + <type_desc name="match_variable"> + Approximation of '$1' | '$2' | '$3' | ... + </type_desc> + <type name="match_variable"/> <desc> <p>The same as <seealso marker="#trace_pattern/3"> @@ -9672,6 +9680,10 @@ timestamp() -> <name name="trace_pattern" arity="3" clause_i="1"/> <fsummary>Set trace pattern for message sending.</fsummary> <type name="trace_match_spec"/> + <type name="match_variable"/> + <type_desc name="match_variable"> + Approximation of '$1' | '$2' | '$3' | ... + </type_desc> <desc> <p>Sets trace pattern for <em>message sending</em>. Must be combined with @@ -9739,6 +9751,10 @@ timestamp() -> <name name="trace_pattern" arity="3" clause_i="2"/> <fsummary>Set trace pattern for tracing of message receiving.</fsummary> <type name="trace_match_spec"/> + <type name="match_variable"/> + <type_desc name="match_variable"> + Approximation of '$1' | '$2' | '$3' | ... + </type_desc> <desc> <p>Sets trace pattern for <em>message receiving</em>. Must be combined with @@ -9809,6 +9825,10 @@ timestamp() -> <type name="trace_pattern_mfa"/> <type name="trace_match_spec"/> <type name="trace_pattern_flag"/> + <type name="match_variable"/> + <type_desc name="match_variable"> + Approximation of '$1' | '$2' | '$3' | ... + </type_desc> <desc> <p>Enables or disables <em>call tracing</em> for one or more functions. Must be combined with diff --git a/erts/emulator/beam/beam_bif_load.c b/erts/emulator/beam/beam_bif_load.c index 5969197168..8af7703f51 100644 --- a/erts/emulator/beam/beam_bif_load.c +++ b/erts/emulator/beam/beam_bif_load.c @@ -1623,10 +1623,10 @@ erts_purge_state_add_fun(ErlFunEntry *fe) } Export * -erts_suspend_process_on_pending_purge_lambda(Process *c_p) +erts_suspend_process_on_pending_purge_lambda(Process *c_p, ErlFunEntry* fe) { erts_smp_mtx_lock(&purge_state.mtx); - if (is_value(purge_state.module)) { + if (purge_state.module == fe->module) { /* * The process c_p is about to call a fun in the code * that we are trying to purge. Suspend it and call diff --git a/erts/emulator/beam/beam_emu.c b/erts/emulator/beam/beam_emu.c index ef4cdf9d5a..3be5c0d24c 100644 --- a/erts/emulator/beam/beam_emu.c +++ b/erts/emulator/beam/beam_emu.c @@ -6555,7 +6555,7 @@ call_fun(Process* p, /* Current process. */ * and let it try again when the purge operation is * done (may succeed or not). */ - ep = erts_suspend_process_on_pending_purge_lambda(p); + ep = erts_suspend_process_on_pending_purge_lambda(p, fe); ASSERT(ep); } else { diff --git a/erts/emulator/beam/beam_load.h b/erts/emulator/beam/beam_load.h index 1200bb9c6f..9be5e14e40 100644 --- a/erts/emulator/beam/beam_load.h +++ b/erts/emulator/beam/beam_load.h @@ -124,7 +124,8 @@ int erts_is_module_native(BeamCodeHeader* code); void erts_beam_bif_load_init(void); struct erl_fun_entry; void erts_purge_state_add_fun(struct erl_fun_entry *fe); -Export *erts_suspend_process_on_pending_purge_lambda(Process *c_p); +Export *erts_suspend_process_on_pending_purge_lambda(Process *c_p, + struct erl_fun_entry*); /* * Layout of the line table. diff --git a/erts/emulator/beam/erl_fun.c b/erts/emulator/beam/erl_fun.c index c639ba623f..5258d83a18 100644 --- a/erts/emulator/beam/erl_fun.c +++ b/erts/emulator/beam/erl_fun.c @@ -236,7 +236,6 @@ erts_fun_purge_abort_prepare(ErlFunEntry **funs, Uint no) ErlFunEntry *fe = funs[ix]; if (fe->address == unloaded_fun) fe->address = fe->pend_purge_address; - fe->pend_purge_address = NULL; } } diff --git a/erts/emulator/hipe/hipe_bif0.c b/erts/emulator/hipe/hipe_bif0.c index 3336fded7a..dcb6c35bfa 100644 --- a/erts/emulator/hipe/hipe_bif0.c +++ b/erts/emulator/hipe/hipe_bif0.c @@ -1476,6 +1476,10 @@ int hipe_find_mfa_from_ra(const void *ra, Eterm *m, Eterm *f, unsigned int *a) struct hipe_mfa_info **bucket; unsigned int i, nrbuckets; + if (hipe_is_ra_mode_switch(ra)) { + return 0; + } + /* Note about locking: the table is only updated from the loader, which runs with the rest of the system suspended. */ /* XXX: alas not true; see comment at hipe_mfa_info_table.lock */ diff --git a/erts/emulator/hipe/hipe_mode_switch.h b/erts/emulator/hipe/hipe_mode_switch.h index c40077d558..e54b81cf78 100644 --- a/erts/emulator/hipe/hipe_mode_switch.h +++ b/erts/emulator/hipe/hipe_mode_switch.h @@ -64,6 +64,7 @@ Eterm hipe_build_stacktrace(Process *p, struct StackTrace *s); ERTS_GLB_INLINE void hipe_reserve_beam_trap_frame(Process*, Eterm reg[], unsigned arity); ERTS_GLB_INLINE void hipe_unreserve_beam_trap_frame(Process*); +ERTS_GLB_INLINE int hipe_is_ra_mode_switch(const void* ra); extern Uint hipe_beam_pc_return[]; extern Uint hipe_beam_pc_throw[]; @@ -112,6 +113,11 @@ ERTS_GLB_INLINE void hipe_unreserve_beam_trap_frame(Process *p) p->stop += 2; } +ERTS_GLB_INLINE int hipe_is_ra_mode_switch(const void* ra) +{ + return ra == nbif_return; +} + #endif /* ERTS_GLB_INLINE_INCL_FUNC_DEF */ #endif /* ASM */ diff --git a/erts/emulator/test/port_SUITE.erl b/erts/emulator/test/port_SUITE.erl index 4323849465..d4e77d634a 100644 --- a/erts/emulator/test/port_SUITE.erl +++ b/erts/emulator/test/port_SUITE.erl @@ -976,21 +976,21 @@ try_bad_env(Env) -> %% Test that we can handle a very very large environment gracefully. huge_env(Config) when is_list(Config) -> ct:timetrap({minutes, 2}), - Vars = case os:type() of - {win32,_} -> 500; - _ -> - %% We create a huge environment, - %% 20000 variables is about 25MB - %% which seems to be the limit on Linux. - 20000 - end, + {Vars, Cmd} = case os:type() of + {win32,_} -> {500, "cmd /q /c ls"}; + _ -> + %% We create a huge environment, + %% 20000 variables is about 25MB + %% which seems to be the limit on Linux. + {20000, "ls"} + end, Env = [{[$a + I div (25*25*25*25) rem 25, $a + I div (25*25*25) rem 25, $a + I div (25*25) rem 25, $a+I div 25 rem 25, $a+I rem 25], lists:duplicate(100,$a+I rem 25)} || I <- lists:seq(1,Vars)], - try erlang:open_port({spawn,"ls"},[exit_status, {env, Env}]) of + try erlang:open_port({spawn,Cmd},[exit_status, {env, Env}]) of P -> receive {P, {exit_status,N}} = M -> @@ -1009,7 +1009,10 @@ huge_env(Config) when is_list(Config) -> %% Test to spawn program with command payload buffer %% just around pipe capacity (9f779819f6bda734c5953468f7798) pipe_limit_env(Config) when is_list(Config) -> - Cmd = "true", + Cmd = case os:type() of + {win32,_} -> "cmd /q /c true"; + _ -> "true" + end, CmdSize = command_payload_size(Cmd), Limits = [4096, 16384, 65536], % Try a couple of common pipe buffer sizes @@ -1026,7 +1029,7 @@ pipe_limit_env_do(Bytes, Cmd, CmdSize) -> try erlang:open_port({spawn,Cmd},[exit_status, {env, Env}]) of P -> receive - {P, {exit_status,N}} = M -> + {P, {exit_status,N}} -> %% Bug caused exit_status 150 (EINVAL+128) 0 = N end diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam Binary files differindex c68debeabc..fbbfefa462 100644 --- a/erts/preloaded/ebin/erlang.beam +++ b/erts/preloaded/ebin/erlang.beam diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam Binary files differindex 227b62b7d3..2a7bfb8b86 100644 --- a/erts/preloaded/ebin/erts_internal.beam +++ b/erts/preloaded/ebin/erts_internal.beam diff --git a/erts/preloaded/src/erlang.erl b/erts/preloaded/src/erlang.erl index 652a954807..8771089b65 100644 --- a/erts/preloaded/src/erlang.erl +++ b/erts/preloaded/src/erlang.erl @@ -2408,10 +2408,11 @@ term_to_binary(_Term, _Options) -> tl(_List) -> erlang:nif_error(undefined). +-type match_variable() :: atom(). % Approximation of '$1' | '$2' | ... -type trace_pattern_mfa() :: {atom(),atom(),arity() | '_'} | on_load. -type trace_match_spec() :: - [{[term()] | '_' ,[term()],[term()]}]. + [{[term()] | '_' | match_variable() ,[term()],[term()]}]. -spec erlang:trace_pattern(MFA, MatchSpec) -> non_neg_integer() when MFA :: trace_pattern_mfa() | send | 'receive', diff --git a/erts/preloaded/src/erts_internal.erl b/erts/preloaded/src/erts_internal.erl index 6aae5ba38c..f4518c4008 100644 --- a/erts/preloaded/src/erts_internal.erl +++ b/erts/preloaded/src/erts_internal.erl @@ -438,10 +438,11 @@ microstate_accounting(Ref, Threads) -> trace(_PidSpec, _How, _FlagList) -> erlang:nif_error(undefined). +-type match_variable() :: atom(). % Approximation of '$1' | '$2' | ... -type trace_pattern_mfa() :: {atom(),atom(),arity() | '_'} | on_load. -type trace_match_spec() :: - [{[term()] | '_' ,[term()],[term()]}]. + [{[term()] | '_' | match_variable() ,[term()],[term()]}]. -spec trace_pattern(MFA, MatchSpec, FlagList) -> non_neg_integer() when MFA :: trace_pattern_mfa(), diff --git a/lib/compiler/src/sys_core_fold.erl b/lib/compiler/src/sys_core_fold.erl index 5e1602cb5b..4922953407 100644 --- a/lib/compiler/src/sys_core_fold.erl +++ b/lib/compiler/src/sys_core_fold.erl @@ -1130,7 +1130,13 @@ clause_1(#c_clause{guard=G0,body=B0}=Cl, Ps1, Cexpr, Ctxt, Sub1) -> %% %% case A of NewVar when true -> ... %% - sub_set_var(Var, Cexpr, Sub2); + case cerl:is_c_fname(Cexpr) of + false -> + sub_set_var(Var, Cexpr, Sub2); + true -> + %% We must not copy funs, and especially not into guards. + Sub2 + end; _ -> Sub2 end, diff --git a/lib/compiler/test/guard_SUITE.erl b/lib/compiler/test/guard_SUITE.erl index 6302f82f29..429d6b79e0 100644 --- a/lib/compiler/test/guard_SUITE.erl +++ b/lib/compiler/test/guard_SUITE.erl @@ -87,6 +87,7 @@ misc(Config) when is_list(Config) -> {ok,buf,<<>>} = get_data({o,true,0}, 42, buf), {ok,buf,<<>>} = get_data({o,false,0}, 0, buf), error = get_data({o,false,0}, 42, buf), + ok. @@ -343,6 +344,11 @@ complex_semicolon(Config) when is_list(Config) -> ok = csemi7(#{a=>1}, 3, 3), ok = csemi7(#{a=>1, b=>3}, 0, 0), + %% 8: Make sure that funs cannot be copied into guards. + ok = csemi8(true), + error = csemi8(false), + error = csemi8(42), + ok. csemi1(Type, Val) when is_list(Val), Type == float; @@ -457,6 +463,13 @@ csemi6(_, _) -> error. csemi7(A, B, C) when A#{a:=B} > #{a=>1}; abs(C) > 2 -> ok; csemi7(_, _, _) -> error. +csemi8(Together) -> + case fun csemi8/1 of + Typically when Together; Typically, Together -> ok; + _ -> error + end. + + comma(Config) when is_list(Config) -> %% ',' combinations of literal true/false. diff --git a/lib/dialyzer/test/small_SUITE_data/src/ms.erl b/lib/dialyzer/test/small_SUITE_data/src/ms.erl new file mode 100644 index 0000000000..47a5e886cf --- /dev/null +++ b/lib/dialyzer/test/small_SUITE_data/src/ms.erl @@ -0,0 +1,8 @@ +-module(ms). +-export([t/0]). + +-include_lib("stdlib/include/ms_transform.hrl"). + +t() -> + MS = dbg:fun2ms(fun(All) -> message(All) end), + erlang:trace_pattern({m, f, '_'}, MS). diff --git a/lib/eldap/doc/src/eldap.xml b/lib/eldap/doc/src/eldap.xml index 43873e44e2..f2c7889e58 100644 --- a/lib/eldap/doc/src/eldap.xml +++ b/lib/eldap/doc/src/eldap.xml @@ -197,7 +197,7 @@ </type> <desc> <p> Add an entry. The entry must not exist.</p> - <pre> + <code> add(Handle, "cn=Bill Valentine, ou=people, o=Example Org, dc=example, dc=com", [{"objectclass", ["person"]}, @@ -205,7 +205,7 @@ {"sn", ["Valentine"]}, {"telephoneNumber", ["545 555 00"]}] ) - </pre> + </code> </desc> </func> <func> @@ -216,9 +216,9 @@ </type> <desc> <p> Delete an entry.</p> - <pre> + <code> delete(Handle, "cn=Bill Valentine, ou=people, o=Example Org, dc=example, dc=com") - </pre> + </code> </desc> </func> @@ -259,11 +259,11 @@ </type> <desc> <p> Modify an entry.</p> - <pre> + <code> modify(Handle, "cn=Bill Valentine, ou=people, o=Example Org, dc=example, dc=com", [eldap:mod_replace("telephoneNumber", ["555 555 00"]), eldap:mod_add("description", ["LDAP Hacker"]) ]) - </pre> + </code> </desc> </func> <func> @@ -320,10 +320,10 @@ whether the current RDN should be removed from the attribute list after the after operation. <c>NewSupDN</c> is the new parent that the RDN shall be moved to. If the old parent should remain as parent, <c>NewSupDN</c> shall be "".</p> - <pre> + <code> modify_dn(Handle, "cn=Bill Valentine, ou=people, o=Example Org, dc=example, dc=com ", "cn=Bill Jr Valentine", true, "") - </pre> + </code> </desc> </func> <func> @@ -342,10 +342,10 @@ Default values: scope is <c>wholeSubtree()</c>, deref is <c>derefAlways()</c>, types_only is <c>false</c> and timeout is <c>0</c> (meaning infinity). </p> - <pre> + <code> Filter = eldap:substrings("cn", [{any,"V"}]), search(Handle, [{base, "dc=example, dc=com"}, {filter, Filter}, {attributes, ["cn"]}]), - </pre> + </code> <p>The <c>timeout</c> option in the <c>SearchOptions</c> is for the ldap server, while the timeout in <seealso marker="#open/2">eldap:open/2</seealso> is used for each individual request in the search operation. @@ -454,7 +454,7 @@ </type> <desc> <p>Creates an extensible match filter. For example, </p> <code> - eldap:extensibleMatch("Bar", [{type,"sn"}, {matchingRule,"caseExactMatch"}])) + eldap:extensibleMatch("Bar", [{type,"sn"}, {matchingRule,"caseExactMatch"}])) </code> <p>creates a filter which performs a <c>caseExactMatch</c> on the attribute <c>sn</c> and matches with the value <c>"Bar"</c>. The default value of <c>dnAttributes</c> is <c>false</c>.</p> </desc> </func> diff --git a/lib/hipe/icode/hipe_beam_to_icode.erl b/lib/hipe/icode/hipe_beam_to_icode.erl index 224aacd8d7..3386523206 100644 --- a/lib/hipe/icode/hipe_beam_to_icode.erl +++ b/lib/hipe/icode/hipe_beam_to_icode.erl @@ -763,32 +763,10 @@ trans_fun([{test,bs_test_unit,{f,Lbl},[Ms,Unit]}| [MsVar], [], Env, Instructions); trans_fun([{test,bs_match_string,{f,Lbl},[Ms,BitSize,Bin]}| Instructions], Env) -> - True = mk_label(new), - FalseLabName = map_label(Lbl), - TrueLabName = hipe_icode:label_name(True), + %% the current match buffer MsVar = mk_var(Ms), - TmpVar = mk_var(new), - ByteSize = BitSize div 8, - ExtraBits = BitSize rem 8, - WordSize = hipe_rtl_arch:word_size(), - if ExtraBits =:= 0 -> - trans_op_call({hipe_bs_primop,{bs_match_string,Bin,ByteSize}}, Lbl, - [MsVar], [MsVar], Env, Instructions); - BitSize =< ((WordSize * 8) - 5) -> - <<Int:BitSize, _/bits>> = Bin, - {I1,Env1} = trans_one_op_call({hipe_bs_primop,{bs_get_integer,BitSize,0}}, Lbl, - [MsVar], [TmpVar, MsVar], Env), - I2 = hipe_icode:mk_type([TmpVar], {integer,Int}, TrueLabName, FalseLabName), - I1 ++ [I2,True] ++ trans_fun(Instructions, Env1); - true -> - <<RealBin:ByteSize/binary, Int:ExtraBits, _/bits>> = Bin, - {I1,Env1} = trans_one_op_call({hipe_bs_primop,{bs_match_string,RealBin,ByteSize}}, Lbl, - [MsVar], [MsVar], Env), - {I2,Env2} = trans_one_op_call({hipe_bs_primop,{bs_get_integer,ExtraBits,0}}, Lbl, - [MsVar], [TmpVar, MsVar], Env1), - I3 = hipe_icode:mk_type([TmpVar], {integer,Int}, TrueLabName, FalseLabName), - I1 ++ I2 ++ [I3,True] ++ trans_fun(Instructions, Env2) - end; + Primop = {hipe_bs_primop, {bs_match_string, Bin, BitSize}}, + trans_op_call(Primop, Lbl, [MsVar], [MsVar], Env, Instructions); trans_fun([{bs_context_to_binary,Var}|Instructions], Env) -> %% the current match buffer IVars = [trans_arg(Var)], diff --git a/lib/hipe/icode/hipe_icode_primops.erl b/lib/hipe/icode/hipe_icode_primops.erl index cee37b6a57..2a141c514e 100644 --- a/lib/hipe/icode/hipe_icode_primops.erl +++ b/lib/hipe/icode/hipe_icode_primops.erl @@ -287,8 +287,8 @@ pp(Dev, Op) -> io:format(Dev, "bs_start_match<~w>", [Max]); {{bs_start_match, Type}, Max} -> io:format(Dev, "bs_start_match<~w,~w>", [Type,Max]); - {bs_match_string, String, SizeInBytes} -> - io:format(Dev, "bs_match_string<~w, ~w>", [String, SizeInBytes]); + {bs_match_string, String, SizeInBits} -> + io:format(Dev, "bs_match_string<~w, ~w>", [String, SizeInBits]); {bs_get_integer, Size, Flags} -> io:format(Dev, "bs_get_integer<~w, ~w>", [Size, Flags]); {bs_get_float, Size, Flags} -> @@ -596,10 +596,10 @@ type(Primop, Args) -> erl_types:t_subtract(Type, erl_types:t_matchstate()), erl_types:t_matchstate_slot( erl_types:t_inf(Type, erl_types:t_matchstate()), 0)); - {hipe_bs_primop, {bs_match_string,_,Bytes}} -> + {hipe_bs_primop, {bs_match_string,_,Bits}} -> [MatchState] = Args, BinType = erl_types:t_matchstate_present(MatchState), - NewBinType = match_bin(erl_types:t_bitstr(0, Bytes*8), BinType), + NewBinType = match_bin(erl_types:t_bitstr(0, Bits), BinType), erl_types:t_matchstate_update_present(NewBinType, MatchState); {hipe_bs_primop, {bs_test_unit,Unit}} -> [MatchState] = Args, diff --git a/lib/hipe/rtl/hipe_rtl_binary.erl b/lib/hipe/rtl/hipe_rtl_binary.erl index fb9c0c196d..9b400f4c93 100644 --- a/lib/hipe/rtl/hipe_rtl_binary.erl +++ b/lib/hipe/rtl/hipe_rtl_binary.erl @@ -19,7 +19,7 @@ %%% %CopyrightEnd% %%% %%%------------------------------------------------------------------- -%%% File : hipe_rtl_binary_2.erl +%%% File : hipe_rtl_binary.erl %%% Author : Per Gustafsson <[email protected]> %%% Description : %%% diff --git a/lib/hipe/rtl/hipe_rtl_binary_match.erl b/lib/hipe/rtl/hipe_rtl_binary_match.erl index 528672b893..d999cd2743 100644 --- a/lib/hipe/rtl/hipe_rtl_binary_match.erl +++ b/lib/hipe/rtl/hipe_rtl_binary_match.erl @@ -270,24 +270,23 @@ gen_rtl({bs_save, Slot}, [NewMs], [Ms], TrueLblName, _FalseLblName) -> set_field_from_term({matchstate, {saveoffset, Slot}}, Ms, Offset), hipe_rtl:mk_goto(TrueLblName)]; %% ----- bs_match_string ----- -gen_rtl({bs_match_string, String, ByteSize}, Dst, [Ms], +gen_rtl({bs_match_string, String, BitSize}, Dst, [Ms], TrueLblName, FalseLblName) -> {[Offset, BinSize, Base], Instrs} = extract_matchstate_vars([offset, binsize, base], Ms), [SuccessLbl, ALbl, ULbl] = create_lbls(3), [NewOffset, BitOffset] = create_gcsafe_regs(2), - Unit = hipe_rtl_arch:word_size() - 1, - Loops = ByteSize div Unit, - Init = + Unit = (hipe_rtl_arch:word_size() - 1) * ?BYTE_SIZE, + Init = [Instrs, opt_update_ms(Dst, Ms), - check_size(Offset, hipe_rtl:mk_imm(ByteSize*?BYTE_SIZE), BinSize, + check_size(Offset, hipe_rtl:mk_imm(BitSize), BinSize, NewOffset, hipe_rtl:label_name(SuccessLbl), FalseLblName), SuccessLbl], SplitCode = [hipe_rtl:mk_alub(BitOffset, Offset, 'and', hipe_rtl:mk_imm(?LOW_BITS), eq, hipe_rtl:label_name(ALbl), hipe_rtl:label_name(ULbl))], - Loops = ByteSize div Unit, + Loops = BitSize div Unit, SkipSize = Loops * Unit, {ACode1, UCode1} = case Loops of @@ -297,9 +296,9 @@ gen_rtl({bs_match_string, String, ByteSize}, Dst, [Ms], create_loops(Loops, Unit, String, Base, Offset, BitOffset, FalseLblName) end, - <<_:SkipSize/binary, RestString/binary>> = String, + <<_:SkipSize/bits, RestString/bits>> = String, {ACode2, UCode2} = - case ByteSize rem Unit of + case BitSize rem Unit of 0 -> {[], []}; Rem -> @@ -393,12 +392,12 @@ validate_unicode_retract_c_code(Src, Ms, TrueLblName, FalseLblName) -> create_loops(Loops, Unit, String, Base, Offset, BitOffset, FalseLblName) -> [Reg] = create_gcsafe_regs(1), AlignedFun = fun(Value) -> - [get_int_to_reg(Reg, Unit*?BYTE_SIZE, Base, Offset, 'srl', + [get_int_to_reg(Reg, Unit, Base, Offset, 'srl', {unsigned, big}), update_and_test(Reg, Unit, Offset, Value, FalseLblName)] end, UnAlignedFun = fun(Value) -> - [get_unaligned_int_to_reg(Reg, Unit*?BYTE_SIZE, + [get_unaligned_int_to_reg(Reg, Unit, Base, Offset, BitOffset, 'srl', {unsigned, big})| update_and_test(Reg, Unit, Offset, Value, FalseLblName)] @@ -406,31 +405,31 @@ create_loops(Loops, Unit, String, Base, Offset, BitOffset, FalseLblName) -> {create_loops(Loops, Unit, String, AlignedFun), create_loops(Loops, Unit, String, UnAlignedFun)}. -create_rests(Rem, String, Base, Offset, BitOffset, FalseLblName) -> +create_rests(RemBits, String, Base, Offset, BitOffset, FalseLblName) -> [Reg] = create_gcsafe_regs(1), AlignedFun = fun(Value) -> - [get_int_to_reg(Reg, Rem*?BYTE_SIZE, Base, Offset, 'srl', + [get_int_to_reg(Reg, RemBits, Base, Offset, 'srl', {unsigned, big})| just_test(Reg, Value, FalseLblName)] end, UnAlignedFun = fun(Value) -> - [get_unaligned_int_to_reg(Reg, Rem*?BYTE_SIZE, + [get_unaligned_int_to_reg(Reg, RemBits, Base, Offset, BitOffset, 'srl', {unsigned, big})| just_test(Reg, Value, FalseLblName)] end, - {create_loops(1, Rem, String, AlignedFun), - create_loops(1, Rem, String, UnAlignedFun)}. + {create_loops(1, RemBits, String, AlignedFun), + create_loops(1, RemBits, String, UnAlignedFun)}. create_loops(0, _Unit, _String, _IntFun) -> []; create_loops(N, Unit, String, IntFun) -> - {Value, RestString} = get_value(Unit,String), + {Value, RestString} = get_value(Unit, String), [IntFun(Value), create_loops(N-1, Unit, RestString, IntFun)]. update_and_test(Reg, Unit, Offset, Value, FalseLblName) -> - [add_to_offset(Offset, Offset, hipe_rtl:mk_imm(Unit*?BYTE_SIZE), FalseLblName), + [add_to_offset(Offset, Offset, hipe_rtl:mk_imm(Unit), FalseLblName), just_test(Reg, Value, FalseLblName)]. just_test(Reg, Value, FalseLblName) -> @@ -439,8 +438,8 @@ just_test(Reg, Value, FalseLblName) -> hipe_rtl:label_name(ContLbl), FalseLblName), ContLbl]. -get_value(N,String) -> - <<I:N/integer-unit:8, Rest/binary>> = String, +get_value(N, String) -> + <<I:N, Rest/bits>> = String, {I, Rest}. make_int_gc_code(I) when is_integer(I) -> diff --git a/lib/hipe/test/bs_SUITE_data/bs_pmatch_bugs.erl b/lib/hipe/test/bs_SUITE_data/bs_pmatch_bugs.erl index b280705a47..d9f3278b45 100644 --- a/lib/hipe/test/bs_SUITE_data/bs_pmatch_bugs.erl +++ b/lib/hipe/test/bs_SUITE_data/bs_pmatch_bugs.erl @@ -9,6 +9,7 @@ test() -> <<49,50,51>> = lex_digits1(Bin, 1, []), <<49,50,51>> = lex_digits2(Bin, 1, []), ok = var_bind_bug(<<1, 2, 3, 4, 5, 6, 7, 8>>), + ok = bs_match_string_bug(), ok. %%-------------------------------------------------------------------- @@ -65,3 +66,50 @@ var_bind_bug(<<A:1/binary, B:8/integer, _C:B/binary, _Rest/binary>>) -> B -> wrong; _ -> ok end. + +%%-------------------------------------------------------------------- +%% From: Andreas Schultz +%% Date: 2/11/2016 +%% +%% Either HiPE is messing up binary matches in some cases or I'm not +%% seeing the problem. ... <SNIP PROGRAM - CLEANED UP VERSION BELOW> +%% With Erlang 19.1.3 the HiPE compiled version behaves differently +%% than the non-HiPE version: ... <SNIP TEST RUNS> +%% So, do I do something wrong here or is this a legitimate HiPE bug? +%% +%% Yes, this was a legitimate HiPE bug: The BEAM to ICode tranaslation +%% of the bs_match_string instruction, written long ago for binaries +%% (i.e., with byte-sized strings), tried to do a `clever' translation +%% of even bit-sized strings using a HiPE primop that took a `Size' +%% argument expressed in *bytes*. ICode is not really the place to do +%% such a thing, and moreover there is really no reason for the HiPE +%% primop not to take a Size argument expressed in *bits* instead. +%% The bug was fixed by changing the `Size' argument to be in bits, +%% postponing the translation of the bs_match_string primop until RTL +%% and doing a proper translation using bit-sized quantities there. +%%-------------------------------------------------------------------- + +bs_match_string_bug() -> + ok = test0(<<50>>), + Bin = data(), + ok = test1(Bin), + ok = test2(Bin), + ok. + +%% Minimal test case showing the problem matching with strings +test0(<<6:5, 0:1, 0:2>>) -> weird; +test0(<<6:5, _:1, _:2>>) -> ok; +test0(_) -> default. + +data() -> <<50,16,0>>. + +%% This was the problematic test case in HiPE: 'default' was returned +test1(<<1:3, 1:1, _:1, 0:1, 0:1, 0:1, _/binary>>) -> weird; +test1(<<1:3, 1:1, _:1, _:1, _:1, _:1, _/binary>>) -> ok; +test1(_) -> default. + +%% This variation of test1/1 above worked OK, even in HiPE +test2(<<1:3, 1:1, _:1, A:1, B:1, C:1, _/binary>>) + when A =:= 1; B =:= 1; C =:= 1 -> ok; +test2(<<1:3, 1:1, _:1, 0:1, 0:1, 0:1, _/binary>>) -> weird; +test2(_) -> default. diff --git a/lib/kernel/doc/src/file.xml b/lib/kernel/doc/src/file.xml index 09497482cf..b674b3ca93 100644 --- a/lib/kernel/doc/src/file.xml +++ b/lib/kernel/doc/src/file.xml @@ -1477,8 +1477,8 @@ f.txt: {person, "kalle", 25}. <tag><c>16#400</c></tag> <item><p>set group id on execution</p></item> </taglist> - <p>On Unix platforms, the following bits - can also be set:</p> + <p>On Unix platforms, other bits than those listed above + may be set.</p> </item> <tag><c>links = integer() >= 0</c></tag> <item> @@ -2042,8 +2042,8 @@ f.txt: {person, "kalle", 25}. <tag><c>16#400</c></tag> <item><p>Set group id on execution</p></item> </taglist> - <p>On Unix platforms, the following bits - can also be set.</p> + <p>On Unix platforms, other bits than those listed above + may be set.</p> </item> <tag><c>uid = integer() >= 0</c></tag> <item> diff --git a/lib/kernel/src/kernel.app.src b/lib/kernel/src/kernel.app.src index 56d1699656..d184223524 100644 --- a/lib/kernel/src/kernel.app.src +++ b/lib/kernel/src/kernel.app.src @@ -118,6 +118,6 @@ {applications, []}, {env, [{error_logger, tty}]}, {mod, {kernel, []}}, - {runtime_dependencies, ["erts-8.0", "stdlib-3.0", "sasl-3.0"]} + {runtime_dependencies, ["erts-8.1", "stdlib-3.0", "sasl-3.0"]} ] }. diff --git a/lib/mnesia/test/mnesia_test_lib.erl b/lib/mnesia/test/mnesia_test_lib.erl index 6e84a27ec9..0fabdc7929 100644 --- a/lib/mnesia/test/mnesia_test_lib.erl +++ b/lib/mnesia/test/mnesia_test_lib.erl @@ -263,6 +263,7 @@ slave_start_link(Host, Name, Retries) -> Path = code:get_path(), ok = rpc:call(NewNode, file, set_cwd, [Cwd]), true = rpc:call(NewNode, code, set_path, [Path]), + ok = rpc:call(NewNode, error_logger, tty, [false]), spawn_link(NewNode, ?MODULE, slave_sup, []), rpc:multicall([node() | nodes()], global, sync, []), {ok, NewNode}; diff --git a/lib/ssh/src/ssh_channel.erl b/lib/ssh/src/ssh_channel.erl index 426e2f5125..85b31f3669 100644 --- a/lib/ssh/src/ssh_channel.erl +++ b/lib/ssh/src/ssh_channel.erl @@ -261,7 +261,7 @@ handle_info({ssh_cm, _, _} = Msg, #state{cm = ConnectionManager, adjust_window(Msg), {noreply, State#state{channel_state = ChannelState}, Timeout}; {stop, ChannelId, ChannelState} -> - ssh_connection:close(ConnectionManager, ChannelId), + catch ssh_connection:close(ConnectionManager, ChannelId), {stop, normal, State#state{close_sent = true, channel_state = ChannelState}} end; diff --git a/lib/ssh/src/ssh_dbg.erl b/lib/ssh/src/ssh_dbg.erl index bd6bc0335b..ce5596e0f9 100644 --- a/lib/ssh/src/ssh_dbg.erl +++ b/lib/ssh/src/ssh_dbg.erl @@ -113,7 +113,12 @@ setup_tracer(Write, MangleArg) -> ok. %%%---------------------------------------------------------------- -shrink_bin(B) when is_binary(B), size(B)>100 -> {'*** SHRINKED BIN',size(B),element(1,split_binary(B,20)),'***'}; +shrink_bin(B) when is_binary(B), size(B)>100 -> {'*** SHRINKED BIN', + size(B), + element(1,split_binary(B,20)), + '...', + element(2,split_binary(B,size(B)-20)) + }; shrink_bin(L) when is_list(L) -> lists:map(fun shrink_bin/1, L); shrink_bin(T) when is_tuple(T) -> list_to_tuple(shrink_bin(tuple_to_list(T))); shrink_bin(X) -> X. diff --git a/lib/ssh/test/ssh_basic_SUITE.erl b/lib/ssh/test/ssh_basic_SUITE.erl index 51e0d5196b..0a0ab5cdf7 100644 --- a/lib/ssh/test/ssh_basic_SUITE.erl +++ b/lib/ssh/test/ssh_basic_SUITE.erl @@ -315,9 +315,9 @@ init_per_testcase(TC, Config) when TC==shell_no_unicode ; {user_passwords, [{"foo", "bar"}]}]), ct:sleep(500), IO = ssh_test_lib:start_io_server(), - Shell = ssh_test_lib:start_shell(Port, IO, UserDir, - [{silently_accept_hosts, true}, - {user,"foo"},{password,"bar"}]), + Shell = ssh_test_lib:start_shell(Port, IO, [{user_dir,UserDir}, + {silently_accept_hosts, true}, + {user,"foo"},{password,"bar"}]), ct:log("IO=~p, Shell=~p, self()=~p",[IO,Shell,self()]), ct:log("file:native_name_encoding() = ~p,~nio:getopts() = ~p", [file:native_name_encoding(),io:getopts()]), @@ -343,14 +343,15 @@ end_per_testcase(TC, Config) when TC==shell_no_unicode ; TC==shell_unicode_string -> case proplists:get_value(sftpd, Config) of {Pid, _, _} -> - ssh:stop_daemon(Pid), - ssh:stop(); + catch ssh:stop_daemon(Pid); _ -> - ssh:stop() - end; + ok + end, + end_per_testcase(Config); end_per_testcase(_TestCase, Config) -> end_per_testcase(Config). -end_per_testcase(_Config) -> + +end_per_testcase(_Config) -> ssh:stop(), ok. @@ -524,7 +525,7 @@ shell(Config) when is_list(Config) -> ct:sleep(500), IO = ssh_test_lib:start_io_server(), - Shell = ssh_test_lib:start_shell(Port, IO, UserDir), + Shell = ssh_test_lib:start_shell(Port, IO, [{user_dir,UserDir}]), receive {'EXIT', _, _} -> ct:fail(no_ssh_connection); @@ -562,10 +563,10 @@ exec_key_differs(Config, UserPKAlgs) -> ct:sleep(500), IO = ssh_test_lib:start_io_server(), - Shell = ssh_test_lib:start_shell(Port, IO, UserDir, - [{preferred_algorithms,[{public_key,['ssh-rsa']}]}, - {pref_public_key_algs,UserPKAlgs} - ]), + Shell = ssh_test_lib:start_shell(Port, IO, [{user_dir,UserDir}, + {preferred_algorithms,[{public_key,['ssh-rsa']}]}, + {pref_public_key_algs,UserPKAlgs} + ]), receive @@ -596,9 +597,9 @@ exec_key_differs_fail(Config) when is_list(Config) -> ct:sleep(500), IO = ssh_test_lib:start_io_server(), - ssh_test_lib:start_shell(Port, IO, UserDir, - [{preferred_algorithms,[{public_key,['ssh-rsa']}]}, - {pref_public_key_algs,['ssh-dss']}]), + ssh_test_lib:start_shell(Port, IO, [{user_dir,UserDir}, + {preferred_algorithms,[{public_key,['ssh-rsa']}]}, + {pref_public_key_algs,['ssh-dss']}]), receive {'EXIT', _, _} -> ok; diff --git a/lib/ssh/test/ssh_options_SUITE.erl b/lib/ssh/test/ssh_options_SUITE.erl index 61883c0647..4cc12cbcbe 100644 --- a/lib/ssh/test/ssh_options_SUITE.erl +++ b/lib/ssh/test/ssh_options_SUITE.erl @@ -540,10 +540,18 @@ connectfun_disconnectfun_server(Config) -> {disconnect,Ref,R} -> ct:log("Disconnect result: ~p",[R]), ssh:stop_daemon(Pid) - after 2000 -> + after 5000 -> + receive + X -> ct:log("received ~p",[X]) + after 0 -> ok + end, {fail, "No disconnectfun action"} end - after 2000 -> + after 5000 -> + receive + X -> ct:log("received ~p",[X]) + after 0 -> ok + end, {fail, "No connectfun action"} end. @@ -649,7 +657,7 @@ disconnectfun_option_server(Config) -> ct:log("Server detected disconnect: ~p",[Reason]), ssh:stop_daemon(Pid), ok - after 3000 -> + after 5000 -> receive X -> ct:log("received ~p",[X]) after 0 -> ok @@ -974,7 +982,14 @@ ssh_connect_negtimeout(Config, Parallel) -> ct:sleep(round(Factor * NegTimeOut)), case inet:sockname(Socket) of - {ok,_} -> ct:fail("Socket not closed"); + {ok,_} -> + %% Give it another chance... + ct:log("Sleep more...",[]), + ct:sleep(round(Factor * NegTimeOut)), + case inet:sockname(Socket) of + {ok,_} -> ct:fail("Socket not closed"); + {error,_} -> ok + end; {error,_} -> ok end. @@ -1003,7 +1018,7 @@ ssh_connect_nonegtimeout_connected(Config, Parallel) -> ct:sleep(500), IO = ssh_test_lib:start_io_server(), - Shell = ssh_test_lib:start_shell(Port, IO, UserDir), + Shell = ssh_test_lib:start_shell(Port, IO, [{user_dir,UserDir}]), receive Error = {'EXIT', _, _} -> ct:log("~p",[Error]), diff --git a/lib/ssh/test/ssh_renegotiate_SUITE.erl b/lib/ssh/test/ssh_renegotiate_SUITE.erl index b10ec3707f..74bbc291b2 100644 --- a/lib/ssh/test/ssh_renegotiate_SUITE.erl +++ b/lib/ssh/test/ssh_renegotiate_SUITE.erl @@ -92,11 +92,11 @@ rekey(Config) -> ConnectionRef = ssh_test_lib:std_connect(Config, Host, Port, [{rekey_limit, 0}]), - Kex1 = get_kex_init(ConnectionRef), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), receive after ?REKEY_DATA_TMO -> %%By this time rekeying would have been done - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), false = (Kex2 == Kex1), ssh:close(ConnectionRef), ssh:stop_daemon(Pid) @@ -120,31 +120,31 @@ rekey_limit(Config) -> {max_random_length_padding,0}]), {ok, SftpPid} = ssh_sftp:start_channel(ConnectionRef), - Kex1 = get_kex_init(ConnectionRef), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), timer:sleep(?REKEY_DATA_TMO), - Kex1 = get_kex_init(ConnectionRef), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), Data = lists:duplicate(159000,1), ok = ssh_sftp:write_file(SftpPid, DataFile, Data), timer:sleep(?REKEY_DATA_TMO), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), false = (Kex2 == Kex1), timer:sleep(?REKEY_DATA_TMO), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), ok = ssh_sftp:write_file(SftpPid, DataFile, "hi\n"), timer:sleep(?REKEY_DATA_TMO), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), false = (Kex2 == Kex1), timer:sleep(?REKEY_DATA_TMO), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), ssh_sftp:stop_channel(SftpPid), ssh:close(ConnectionRef), @@ -169,7 +169,7 @@ renegotiate1(Config) -> ConnectionRef = ssh_test_lib:std_connect(Config, Host, RPort, [{max_random_length_padding,0}]), {ok, SftpPid} = ssh_sftp:start_channel(ConnectionRef), - Kex1 = get_kex_init(ConnectionRef), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), {ok, Handle} = ssh_sftp:open(SftpPid, DataFile, [write]), @@ -181,7 +181,7 @@ renegotiate1(Config) -> timer:sleep(2000), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), false = (Kex2 == Kex1), @@ -208,7 +208,7 @@ renegotiate2(Config) -> ConnectionRef = ssh_test_lib:std_connect(Config, Host, RPort, [{max_random_length_padding,0}]), {ok, SftpPid} = ssh_sftp:start_channel(ConnectionRef), - Kex1 = get_kex_init(ConnectionRef), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), {ok, Handle} = ssh_sftp:open(SftpPid, DataFile, [write]), @@ -223,7 +223,7 @@ renegotiate2(Config) -> timer:sleep(2000), - Kex2 = get_kex_init(ConnectionRef), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), false = (Kex2 == Kex1), @@ -235,19 +235,3 @@ renegotiate2(Config) -> %%-------------------------------------------------------------------- %% Internal functions ------------------------------------------------ %%-------------------------------------------------------------------- -%% get_kex_init - helper function to get key_exchange_init_msg -get_kex_init(Conn) -> - %% First, validate the key exchange is complete (StateName == connected) - {{connected,_},S} = sys:get_state(Conn), - %% Next, walk through the elements of the #state record looking - %% for the #ssh_msg_kexinit record. This method is robust against - %% changes to either record. The KEXINIT message contains a cookie - %% unique to each invocation of the key exchange procedure (RFC4253) - SL = tuple_to_list(S), - case lists:keyfind(ssh_msg_kexinit, 1, SL) of - false -> - throw(not_found); - KexInit -> - KexInit - end. - diff --git a/lib/ssh/test/ssh_test_lib.erl b/lib/ssh/test/ssh_test_lib.erl index 6fd401d182..f93237f3e7 100644 --- a/lib/ssh/test/ssh_test_lib.erl +++ b/lib/ssh/test/ssh_test_lib.erl @@ -127,24 +127,19 @@ std_simple_exec(Host, Port, Config, Opts) -> ssh:close(ConnectionRef). -start_shell(Port, IOServer, UserDir) -> - start_shell(Port, IOServer, UserDir, []). - -start_shell(Port, IOServer, UserDir, Options) -> - spawn_link(?MODULE, init_shell, [Port, IOServer, [{user_dir, UserDir}|Options]]). - start_shell(Port, IOServer) -> - spawn_link(?MODULE, init_shell, [Port, IOServer, []]). + start_shell(Port, IOServer, []). -init_shell(Port, IOServer, UserDir) -> - Host = hostname(), - Options = [{user_interaction, false}, {silently_accept_hosts, - true}] ++ UserDir, - group_leader(IOServer, self()), - loop_shell(Host, Port, Options). +start_shell(Port, IOServer, ExtraOptions) -> + spawn_link( + fun() -> + Host = hostname(), + Options = [{user_interaction, false}, + {silently_accept_hosts,true} | ExtraOptions], + group_leader(IOServer, self()), + ssh:shell(Host, Port, Options) + end). -loop_shell(Host, Port, Options) -> - ssh:shell(Host, Port, Options). start_io_server() -> spawn_link(?MODULE, init_io_server, [self()]). @@ -802,3 +797,40 @@ busy_wait(Nus, T0) -> end. %%%---------------------------------------------------------------- +%% get_kex_init - helper function to get key_exchange_init_msg + +get_kex_init(Conn) -> + Ref = make_ref(), + {ok,TRef} = timer:send_after(15000, {reneg_timeout,Ref}), + get_kex_init(Conn, Ref, TRef). + +get_kex_init(Conn, Ref, TRef) -> + %% First, validate the key exchange is complete (StateName == connected) + case sys:get_state(Conn) of + {{connected,_}, S} -> + timer:cancel(TRef), + %% Next, walk through the elements of the #state record looking + %% for the #ssh_msg_kexinit record. This method is robust against + %% changes to either record. The KEXINIT message contains a cookie + %% unique to each invocation of the key exchange procedure (RFC4253) + SL = tuple_to_list(S), + case lists:keyfind(ssh_msg_kexinit, 1, SL) of + false -> + throw(not_found); + KexInit -> + KexInit + end; + + {OtherState, S} -> + ct:log("Not in 'connected' state: ~p",[OtherState]), + receive + {reneg_timeout,Ref} -> + ct:log("S = ~p", [S]), + ct:fail(reneg_timeout) + after 0 -> + timer:sleep(100), % If renegotiation is complete we do not + % want to exit on the reneg_timeout + get_kex_init(Conn, Ref, TRef) + end + end. + diff --git a/lib/ssh/test/ssh_to_openssh_SUITE.erl b/lib/ssh/test/ssh_to_openssh_SUITE.erl index f481e9c1ce..2c7fe7898f 100644 --- a/lib/ssh/test/ssh_to_openssh_SUITE.erl +++ b/lib/ssh/test/ssh_to_openssh_SUITE.erl @@ -29,6 +29,7 @@ -define(TIMEOUT, 50000). -define(SSH_DEFAULT_PORT, 22). +-define(REKEY_DATA_TMO, 65000). %%-------------------------------------------------------------------- %% Common Test interface functions ----------------------------------- @@ -55,7 +56,8 @@ groups() -> erlang_client_openssh_server_publickey_rsa, erlang_client_openssh_server_password, erlang_client_openssh_server_kexs, - erlang_client_openssh_server_nonexistent_subsystem + erlang_client_openssh_server_nonexistent_subsystem, + erlang_client_openssh_server_renegotiate ]}, {erlang_server, [], [erlang_server_openssh_client_public_key_dsa, erlang_server_openssh_client_public_key_rsa, @@ -105,6 +107,11 @@ init_per_testcase(erlang_server_openssh_client_public_key_rsa, Config) -> chk_key(sshc, 'ssh-rsa', ".ssh/id_rsa", Config); init_per_testcase(erlang_client_openssh_server_publickey_dsa, Config) -> chk_key(sshd, 'ssh-dss', ".ssh/id_dsa", Config); +init_per_testcase(erlang_server_openssh_client_renegotiate, Config) -> + case os:type() of + {unix,_} -> ssh:start(), Config; + Type -> {skip, io_lib:format("Unsupported test on ~p",[Type])} + end; init_per_testcase(_TestCase, Config) -> ssh:start(), Config. @@ -146,7 +153,7 @@ erlang_shell_client_openssh_server(Config) when is_list(Config) -> IO = ssh_test_lib:start_io_server(), Shell = ssh_test_lib:start_shell(?SSH_DEFAULT_PORT, IO), IO ! {input, self(), "echo Hej\n"}, - receive_hej(), + receive_data("Hej"), IO ! {input, self(), "exit\n"}, receive_logout(), receive_normal_exit(Shell). @@ -393,24 +400,34 @@ erlang_server_openssh_client_renegotiate(Config) -> SystemDir = proplists:get_value(data_dir, Config), PrivDir = proplists:get_value(priv_dir, Config), KnownHosts = filename:join(PrivDir, "known_hosts"), + {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, {public_key_alg, PubKeyAlg}, {failfun, fun ssh_test_lib:failfun/2}]), +%% catch ssh_dbg:messages(fun(String,_D) -> ct:log(String) end), ct:sleep(500), + RenegLimitK = 3, DataFile = filename:join(PrivDir, "renegotiate_openssh_client.data"), - Data = lists:duplicate(32000, $a), + Data = lists:duplicate(trunc(1.1*RenegLimitK*1024), $a), ok = file:write_file(DataFile, Data), Cmd = "ssh -p " ++ integer_to_list(Port) ++ " -o UserKnownHostsFile=" ++ KnownHosts ++ - " -o RekeyLimit=20K" ++ + " -o RekeyLimit=" ++ integer_to_list(RenegLimitK) ++"K" ++ " " ++ Host ++ " < " ++ DataFile, OpenSsh = ssh_test_lib:open_port({spawn, Cmd}), Expect = fun({data,R}) -> - try lists:prefix(binary_to_list(R), Data) + try + NonAlphaChars = [C || C<-lists:seq(1,255), + not lists:member(C,lists:seq($a,$z)), + not lists:member(C,lists:seq($A,$Z)) + ], + Lines = string:tokens(binary_to_list(R), NonAlphaChars), + lists:any(fun(L) -> length(L)>1 andalso lists:prefix(L, Data) end, + Lines) catch _:_ -> false end; @@ -419,9 +436,61 @@ erlang_server_openssh_client_renegotiate(Config) -> end, ssh_test_lib:rcv_expected(Expect, OpenSsh, ?TIMEOUT), + %% Unfortunatly we can't check that there has been a renegotiation, just trust OpenSSH. ssh:stop_daemon(Pid). %%-------------------------------------------------------------------- +erlang_client_openssh_server_renegotiate(_Config) -> + process_flag(trap_exit, true), + + IO = ssh_test_lib:start_io_server(), + Ref = make_ref(), + Parent = self(), + + catch ssh_dbg:messages(fun(X,_) -> ct:log(X) end), + Shell = + spawn_link( + fun() -> + Host = ssh_test_lib:hostname(), + Options = [{user_interaction, false}, + {silently_accept_hosts,true}], + group_leader(IO, self()), + {ok, ConnRef} = ssh:connect(Host, ?SSH_DEFAULT_PORT, Options), + case ssh_connection:session_channel(ConnRef, infinity) of + {ok,ChannelId} -> + success = ssh_connection:ptty_alloc(ConnRef, ChannelId, []), + Args = [{channel_cb, ssh_shell}, + {init_args,[ConnRef, ChannelId]}, + {cm, ConnRef}, {channel_id, ChannelId}], + {ok, State} = ssh_channel:init([Args]), + Parent ! {ok, Ref, ConnRef}, + ssh_channel:enter_loop(State); + Error -> + Parent ! {error, Ref, Error} + end, + receive + nothing -> ok + end + end), + + receive + {error, Ref, Error} -> + ct:fail("Error=~p",[Error]); + {ok, Ref, ConnectionRef} -> + IO ! {input, self(), "echo Hej1\n"}, + receive_data("Hej1"), + Kex1 = ssh_test_lib:get_kex_init(ConnectionRef), + ssh_connection_handler:renegotiate(ConnectionRef), + IO ! {input, self(), "echo Hej2\n"}, + receive_data("Hej2"), + Kex2 = ssh_test_lib:get_kex_init(ConnectionRef), + IO ! {input, self(), "exit\n"}, + receive_logout(), + receive_normal_exit(Shell), + true = (Kex1 =/= Kex2) + end. + +%%-------------------------------------------------------------------- erlang_client_openssh_server_password() -> [{doc, "Test client password option"}]. erlang_client_openssh_server_password(Config) when is_list(Config) -> @@ -476,27 +545,24 @@ erlang_client_openssh_server_nonexistent_subsystem(Config) when is_list(Config) %%-------------------------------------------------------------------- %%% Internal functions ----------------------------------------------- %%-------------------------------------------------------------------- -receive_hej() -> +receive_data(Data) -> receive - <<"Hej", _binary>> = Hej -> - ct:log("Expected result: ~p~n", [Hej]); - <<"Hej\n", _binary>> = Hej -> - ct:log("Expected result: ~p~n", [Hej]); - <<"Hej\r\n", _/binary>> = Hej -> - ct:log("Expected result: ~p~n", [Hej]); - Info -> - Lines = binary:split(Info, [<<"\r\n">>], [global]), - case lists:member(<<"Hej">>, Lines) of + Info when is_binary(Info) -> + Lines = string:tokens(binary_to_list(Info), "\r\n "), + case lists:member(Data, Lines) of true -> ct:log("Expected result found in lines: ~p~n", [Lines]), ok; false -> ct:log("Extra info: ~p~n", [Info]), - receive_hej() - end + receive_data(Data) + end; + Other -> + ct:log("Unexpected: ~p",[Other]), + receive_data(Data) after 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end. + end. receive_logout() -> receive diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index 68f2f97b6e..edc7e0d8b2 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -170,6 +170,14 @@ <tag><c>SNIfun::fun()</c></tag> <item><p><c>= fun(ServerName :: string()) -> [ssl_option()]</c></p></item> + <tag><c>named_curve() =</c></tag> + <item><p><c>sect571r1 | sect571k1 | secp521r1 | brainpoolP512r1 + | sect409k1 | sect409r1 | brainpoolP384r1 | secp384r1 + | sect283k1 | sect283r1 | brainpoolP256r1 | secp256k1 | secp256r1 + | sect239k1 | sect233k1 | sect233r1 | secp224k1 | secp224r1 + | sect193r1 | sect193r2 | secp192k1 | secp192r1 | sect163k1 + | sect163r1 | sect163r2 | secp160k1 | secp160r1 | secp160r2</c></p></item> + </taglist> </section> @@ -217,6 +225,11 @@ Anonymous cipher suites are supported for testing purposes only and are not be used when security matters.</p></item> + <tag><c>{eccs, [named_curve()]}</c></tag> + <item><p> Allows to specify the order of preference for named curves + and to restrict their usage when using a cipher suite supporting them. + </p></item> + <tag><c>{secure_renegotiate, boolean()}</c></tag> <item><p>Specifies if to reject renegotiation attempt that does not live up to @@ -751,6 +764,11 @@ fun(srp, Username :: string(), UserState :: term()) -> (the default), use the client's preference. </item> + <tag><c>{honor_ecc_order, boolean()}</c></tag> + <item>If true, use the server's preference for ECC curve selection. If false + (the default), use the client's preference. + </item> + <tag><c>{signature_algs, [{hash(), ecdsa | rsa | dsa}]}</c></tag> <item><p> The algorithms specified by this option will be the ones accepted by the server in a signature algorithm @@ -804,6 +822,17 @@ fun(srp, Username :: string(), UserState :: term()) -> </func> <func> + <name>eccs() -></name> + <name>eccs(protocol()) -> [named_curve()]</name> + <fsummary>Returns a list of supported ECCs.</fsummary> + + <desc><p>Returns a list of supported ECCs. <c>eccs()</c> + is equivalent to calling <c>eccs(Protocol)</c> with all + supported protocols and then deduplicating the output.</p> + </desc> + </func> + + <func> <name>clear_pem_cache() -> ok </name> <fsummary> Clears the pem cache</fsummary> @@ -898,7 +927,7 @@ fun(srp, Username :: string(), UserState :: term()) -> <fsummary>Returns all the connection information. </fsummary> <type> - <v>Item = protocol | cipher_suite | sni_hostname | atom()</v> + <v>Item = protocol | cipher_suite | sni_hostname | ecc | atom()</v> <d>Meaningful atoms, not specified above, are the ssl option names.</d> <v>Result = [{Item::atom(), Value::term()}]</v> <v>Reason = term()</v> diff --git a/lib/ssl/doc/src/ssl_distribution.xml b/lib/ssl/doc/src/ssl_distribution.xml index 1150043e76..61f88e3860 100644 --- a/lib/ssl/doc/src/ssl_distribution.xml +++ b/lib/ssl/doc/src/ssl_distribution.xml @@ -43,7 +43,7 @@ Erlang node distributed, <c>net_kernel</c> uses this module to set up listen ports and connections.</p> - <p>In the SSL application, an exra distribution + <p>In the SSL application, an extra distribution module, <c>inet_tls_dist</c>, can be used as an alternative. All distribution connections will use SSL and all participating Erlang nodes in a distributed system must use diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index 27b753af2e..aa62ab8865 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -38,7 +38,7 @@ getopts/2, setopts/2, getstat/1, getstat/2 ]). %% SSL/TLS protocol handling --export([cipher_suites/0, cipher_suites/1, +-export([cipher_suites/0, cipher_suites/1, eccs/0, eccs/1, connection_info/1, versions/0, session_info/1, format_error/1, renegotiate/1, prf/5, negotiated_protocol/1, negotiated_next_protocol/1, connection_information/1, connection_information/2]). @@ -420,6 +420,33 @@ cipher_suites(all) -> [ssl_cipher:erl_suite_definition(Suite) || Suite <- available_suites(all)]. %%-------------------------------------------------------------------- +-spec eccs() -> tls_v1:curves(). +%% Description: returns all supported curves across all versions +%%-------------------------------------------------------------------- +eccs() -> + Curves = tls_v1:ecc_curves(all), % only tls_v1 has named curves right now + eccs_filter_supported(Curves). + +%%-------------------------------------------------------------------- +-spec eccs(tls_record:tls_version() | tls_record:tls_atom_version()) -> + tls_v1:curves(). +%% Description: returns the curves supported for a given version of +%% ssl/tls. +%%-------------------------------------------------------------------- +eccs({3,0}) -> + []; +eccs({3,_}) -> + Curves = tls_v1:ecc_curves(all), + eccs_filter_supported(Curves); +eccs(AtomVersion) when is_atom(AtomVersion) -> + eccs(tls_record:protocol_version(AtomVersion)). + +eccs_filter_supported(Curves) -> + CryptoCurves = crypto:ec_curves(), + lists:filter(fun(Curve) -> proplists:get_bool(Curve, CryptoCurves) end, + Curves). + +%%-------------------------------------------------------------------- -spec getopts(#sslsocket{}, [gen_tcp:option_name()]) -> {ok, [gen_tcp:option()]} | {error, reason()}. %% @@ -647,6 +674,8 @@ do_connect(Address, Port, end. %% Handle extra ssl options given to ssl_accept +-spec handle_options([any()], #ssl_options{}) -> #ssl_options{} + ; ([any()], client | server) -> {ok, #config{}}. handle_options(Opts0, #ssl_options{protocol = Protocol, cacerts = CaCerts0, cacertfile = CaCertFile0} = InheritedSslOpts) -> RecordCB = record_cb(Protocol), @@ -725,6 +754,8 @@ handle_options(Opts0, Role) -> srp_identity = handle_option(srp_identity, Opts, undefined), ciphers = handle_cipher_option(proplists:get_value(ciphers, Opts, []), RecordCb:highest_protocol_version(Versions)), + eccs = handle_eccs_option(proplists:get_value(eccs, Opts, eccs()), + RecordCb:highest_protocol_version(Versions)), signature_algs = handle_hashsigns_option(proplists:get_value(signature_algs, Opts, default_option_role(server, tls_v1:default_signature_algs(Versions), Role)), @@ -755,6 +786,9 @@ handle_options(Opts0, Role) -> honor_cipher_order = handle_option(honor_cipher_order, Opts, default_option_role(server, false, Role), server, Role), + honor_ecc_order = handle_option(honor_ecc_order, Opts, + default_option_role(server, false, Role), + server, Role), protocol = proplists:get_value(protocol, Opts, tls), padding_check = proplists:get_value(padding_check, Opts, true), beast_mitigation = handle_option(beast_mitigation, Opts, one_n_minus_one), @@ -780,7 +814,7 @@ handle_options(Opts0, Role) -> alpn_preferred_protocols, next_protocols_advertised, client_preferred_next_protocols, log_alert, server_name_indication, honor_cipher_order, padding_check, crl_check, crl_cache, - fallback, signature_algs, beast_mitigation, v2_hello_compatible], + fallback, signature_algs, eccs, honor_ecc_order, beast_mitigation, v2_hello_compatible], SockOpts = lists:foldl(fun(Key, PropList) -> proplists:delete(Key, PropList) @@ -1010,6 +1044,8 @@ validate_option(sni_fun, Fun) when is_function(Fun) -> Fun; validate_option(honor_cipher_order, Value) when is_boolean(Value) -> Value; +validate_option(honor_ecc_order, Value) when is_boolean(Value) -> + Value; validate_option(padding_check, Value) when is_boolean(Value) -> Value; validate_option(fallback, Value) when is_boolean(Value) -> @@ -1164,6 +1200,14 @@ binary_cipher_suites(Version, Ciphers0) -> Ciphers = [ssl_cipher:openssl_suite(C) || C <- string:tokens(Ciphers0, ":")], binary_cipher_suites(Version, Ciphers). +handle_eccs_option(Value, {_Major, Minor}) when is_list(Value) -> + try tls_v1:ecc_curves(Minor, Value) of + Curves -> #elliptic_curves{elliptic_curve_list = Curves} + catch + exit:_ -> throw({error, {options, {eccs, Value}}}); + error:_ -> throw({error, {options, {eccs, Value}}}) + end. + unexpected_format(Error) -> lists:flatten(io_lib:format("Unexpected error: ~p", [Error])). @@ -1334,6 +1378,14 @@ new_ssl_options([{server_name_indication, Value} | Rest], #ssl_options{} = Opts, new_ssl_options(Rest, Opts#ssl_options{server_name_indication = validate_option(server_name_indication, Value)}, RecordCB); new_ssl_options([{honor_cipher_order, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> new_ssl_options(Rest, Opts#ssl_options{honor_cipher_order = validate_option(honor_cipher_order, Value)}, RecordCB); +new_ssl_options([{honor_ecc_order, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> + new_ssl_options(Rest, Opts#ssl_options{honor_ecc_order = validate_option(honor_ecc_order, Value)}, RecordCB); +new_ssl_options([{eccs, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> + new_ssl_options(Rest, + Opts#ssl_options{eccs = + handle_eccs_option(Value, RecordCB:highest_protocol_version()) + }, + RecordCB); new_ssl_options([{signature_algs, Value} | Rest], #ssl_options{} = Opts, RecordCB) -> new_ssl_options(Rest, Opts#ssl_options{signature_algs = diff --git a/lib/ssl/src/ssl_connection.erl b/lib/ssl/src/ssl_connection.erl index 08fca76123..b6e4d5b433 100644 --- a/lib/ssl/src/ssl_connection.erl +++ b/lib/ssl/src/ssl_connection.erl @@ -1172,14 +1172,23 @@ handle_alert(#alert{level = ?WARNING} = Alert, StateName, %%% Internal functions %%-------------------------------------------------------------------- connection_info(#state{sni_hostname = SNIHostname, - session = #session{cipher_suite = CipherSuite}, + session = #session{cipher_suite = CipherSuite, ecc = ECCCurve}, protocol_cb = Connection, negotiated_version = {_,_} = Version, ssl_options = Opts}) -> RecordCB = record_cb(Connection), + CipherSuiteDef = ssl_cipher:erl_suite_definition(CipherSuite), + IsNamedCurveSuite = lists:member(element(1,CipherSuiteDef), + [ecdh_ecdsa, ecdhe_ecdsa, ecdh_anon]), + CurveInfo = case ECCCurve of + {namedCurve, Curve} when IsNamedCurveSuite -> + [{ecc, {named_curve, pubkey_cert_records:namedCurves(Curve)}}]; + _ -> + [] + end, [{protocol, RecordCB:protocol_version(Version)}, - {cipher_suite, ssl_cipher:erl_suite_definition(CipherSuite)}, - {sni_hostname, SNIHostname}] ++ ssl_options_list(Opts). + {cipher_suite, CipherSuiteDef}, + {sni_hostname, SNIHostname} | CurveInfo] ++ ssl_options_list(Opts). do_server_hello(Type, #hello_extensions{next_protocol_negotiation = NextProtocols} = ServerHelloExt, @@ -1741,12 +1750,13 @@ calculate_secret(#server_dh_params{dh_p = Prime, dh_g = Base, Connection, certify, certify); calculate_secret(#server_ecdh_params{curve = ECCurve, public = ECServerPubKey}, - State, Connection) -> + State=#state{session=Session}, Connection) -> ECDHKeys = public_key:generate_key(ECCurve), PremasterSecret = ssl_handshake:premaster_secret(#'ECPoint'{point = ECServerPubKey}, ECDHKeys), calculate_master_secret(PremasterSecret, - State#state{diffie_hellman_keys = ECDHKeys}, + State#state{diffie_hellman_keys = ECDHKeys, + session = Session#session{ecc = ECCurve}}, Connection, certify, certify); calculate_secret(#server_psk_params{ diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 5b51ac0916..4acc745c5f 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -70,7 +70,7 @@ %% Extensions handling -export([client_hello_extensions/6, handle_client_hello_extensions/9, %% Returns server hello extensions - handle_server_hello_extensions/9, select_curve/2 + handle_server_hello_extensions/9, select_curve/2, select_curve/3 ]). %% MISC @@ -120,11 +120,13 @@ server_hello_done() -> #server_hello_done{}. client_hello_extensions(Host, Version, CipherSuites, - #ssl_options{signature_algs = SupportedHashSigns, versions = AllVersions} = SslOpts, ConnectionStates, Renegotiation) -> + #ssl_options{signature_algs = SupportedHashSigns, + eccs = SupportedECCs, + versions = AllVersions} = SslOpts, ConnectionStates, Renegotiation) -> {EcPointFormats, EllipticCurves} = case advertises_ec_ciphers(lists:map(fun ssl_cipher:suite_definition/1, CipherSuites)) of true -> - client_ecc_extensions(tls_v1, Version); + client_ecc_extensions(SupportedECCs); false -> {undefined, undefined} end, @@ -1169,8 +1171,9 @@ select_session(SuggestedSessionId, CipherSuites, HashSigns, Compressions, Port, {resumed, Resumed} end. -supported_ecc({Major, Minor} = Version) when ((Major == 3) and (Minor >= 1)) orelse (Major > 3) -> - Curves = tls_v1:ecc_curves(Version), +%% Deprecated? +supported_ecc({Major, Minor}) when ((Major == 3) and (Minor >= 1)) orelse (Major > 3) -> + Curves = tls_v1:ecc_curves(Minor), #elliptic_curves{elliptic_curve_list = Curves}; supported_ecc(_) -> #elliptic_curves{elliptic_curve_list = []}. @@ -1454,12 +1457,12 @@ srp_user(#ssl_options{srp_identity = {UserName, _}}) -> srp_user(_) -> undefined. -client_ecc_extensions(Module, Version) -> +client_ecc_extensions(SupportedECCs) -> CryptoSupport = proplists:get_value(public_keys, crypto:supports()), case proplists:get_bool(ecdh, CryptoSupport) of true -> EcPointFormats = #ec_point_formats{ec_point_format_list = [?ECPOINT_UNCOMPRESSED]}, - EllipticCurves = #elliptic_curves{elliptic_curve_list = Module:ecc_curves(Version)}, + EllipticCurves = SupportedECCs, {EcPointFormats, EllipticCurves}; _ -> {undefined, undefined} @@ -1493,22 +1496,34 @@ advertises_ec_ciphers([{ecdh_anon, _,_,_} | _]) -> true; advertises_ec_ciphers([_| Rest]) -> advertises_ec_ciphers(Rest). -select_curve(#elliptic_curves{elliptic_curve_list = ClientCurves}, - #elliptic_curves{elliptic_curve_list = ServerCurves}) -> - select_curve(ClientCurves, ServerCurves); -select_curve(undefined, _) -> + +select_curve(Client, Server) -> + select_curve(Client, Server, false). + +select_curve(#elliptic_curves{elliptic_curve_list = ClientCurves}, + #elliptic_curves{elliptic_curve_list = ServerCurves}, + ServerOrder) -> + case ServerOrder of + false -> + select_shared_curve(ClientCurves, ServerCurves); + true -> + select_shared_curve(ServerCurves, ClientCurves) + end; +select_curve(undefined, _, _) -> %% Client did not send ECC extension use default curve if %% ECC cipher is negotiated - {namedCurve, ?secp256r1}; -select_curve(_, []) -> + {namedCurve, ?secp256r1}. + +select_shared_curve([], _) -> no_curve; -select_curve(Curves, [Curve| Rest]) -> +select_shared_curve([Curve | Rest], Curves) -> case lists:member(Curve, Curves) of true -> {namedCurve, Curve}; false -> - select_curve(Curves, Rest) + select_shared_curve(Rest, Curves) end. + %% RFC 6066, Section 3: Currently, the only server names supported are %% DNS hostnames sni(_, disable) -> diff --git a/lib/ssl/src/ssl_internal.hrl b/lib/ssl/src/ssl_internal.hrl index c19c1787ff..487d1fa096 100644 --- a/lib/ssl/src/ssl_internal.hrl +++ b/lib/ssl/src/ssl_internal.hrl @@ -140,6 +140,8 @@ crl_check :: boolean() | peer | best_effort, crl_cache, signature_algs, + eccs, + honor_ecc_order :: boolean(), v2_hello_compatible :: boolean() }). diff --git a/lib/ssl/src/tls_handshake.erl b/lib/ssl/src/tls_handshake.erl index a2486bf752..2bd103c18a 100644 --- a/lib/ssl/src/tls_handshake.erl +++ b/lib/ssl/src/tls_handshake.erl @@ -160,13 +160,15 @@ handle_client_hello(Version, #client_hello{session_id = SugesstedId, extensions = #hello_extensions{elliptic_curves = Curves, signature_algs = ClientHashSigns} = HelloExt}, #ssl_options{versions = Versions, - signature_algs = SupportedHashSigns} = SslOpts, + signature_algs = SupportedHashSigns, + eccs = SupportedECCs, + honor_ecc_order = ECCOrder} = SslOpts, {Port, Session0, Cache, CacheCb, ConnectionStates0, Cert, _}, Renegotiation) -> case tls_record:is_acceptable_version(Version, Versions) of true -> AvailableHashSigns = ssl_handshake:available_signature_algs( ClientHashSigns, SupportedHashSigns, Cert, Version), - ECCCurve = ssl_handshake:select_curve(Curves, ssl_handshake:supported_ecc(Version)), + ECCCurve = ssl_handshake:select_curve(Curves, SupportedECCs, ECCOrder), {Type, #session{cipher_suite = CipherSuite} = Session1} = ssl_handshake:select_session(SugesstedId, CipherSuites, AvailableHashSigns, Compressions, Port, Session0#session{ecc = ECCCurve}, Version, diff --git a/lib/ssl/src/tls_v1.erl b/lib/ssl/src/tls_v1.erl index 711db77708..7f24ce5192 100644 --- a/lib/ssl/src/tls_v1.erl +++ b/lib/ssl/src/tls_v1.erl @@ -31,9 +31,18 @@ -export([master_secret/4, finished/5, certificate_verify/3, mac_hash/7, setup_keys/8, suites/1, prf/5, - ecc_curves/1, oid_to_enum/1, enum_to_oid/1, + ecc_curves/1, ecc_curves/2, oid_to_enum/1, enum_to_oid/1, default_signature_algs/1, signature_algs/2]). +-type named_curve() :: sect571r1 | sect571k1 | secp521r1 | brainpoolP512r1 | + sect409k1 | sect409r1 | brainpoolP384r1 | secp384r1 | + sect283k1 | sect283r1 | brainpoolP256r1 | secp256k1 | secp256r1 | + sect239k1 | sect233k1 | sect233r1 | secp224k1 | secp224r1 | + sect193r1 | sect193r2 | secp192k1 | secp192r1 | sect163k1 | + sect163r1 | sect163r2 | secp160k1 | secp160r1 | secp160r2. +-type curves() :: [named_curve()]. +-export_type([curves/0, named_curve/0]). + %%==================================================================== %% Internal application API %%==================================================================== @@ -399,13 +408,20 @@ is_pair(Hash, rsa, Hashs) -> lists:member(Hash, AtLeastMd5). %% list ECC curves in prefered order -ecc_curves(_Minor) -> - TLSCurves = [sect571r1,sect571k1,secp521r1,brainpoolP512r1, - sect409k1,sect409r1,brainpoolP384r1,secp384r1, - sect283k1,sect283r1,brainpoolP256r1,secp256k1,secp256r1, - sect239k1,sect233k1,sect233r1,secp224k1,secp224r1, - sect193r1,sect193r2,secp192k1,secp192r1,sect163k1, - sect163r1,sect163r2,secp160k1,secp160r1,secp160r2], +-spec ecc_curves(1..3 | all) -> [named_curve()]. +ecc_curves(all) -> + [sect571r1,sect571k1,secp521r1,brainpoolP512r1, + sect409k1,sect409r1,brainpoolP384r1,secp384r1, + sect283k1,sect283r1,brainpoolP256r1,secp256k1,secp256r1, + sect239k1,sect233k1,sect233r1,secp224k1,secp224r1, + sect193r1,sect193r2,secp192k1,secp192r1,sect163k1, + sect163r1,sect163r2,secp160k1,secp160r1,secp160r2]; +ecc_curves(Minor) -> + TLSCurves = ecc_curves(all), + ecc_curves(Minor, TLSCurves). + +-spec ecc_curves(1..3, [named_curve()]) -> [named_curve()]. +ecc_curves(_Minor, TLSCurves) -> CryptoCurves = crypto:ec_curves(), lists:foldr(fun(Curve, Curves) -> case proplists:get_bool(Curve, CryptoCurves) of @@ -414,6 +430,7 @@ ecc_curves(_Minor) -> end end, [], TLSCurves). + %% ECC curves from draft-ietf-tls-ecc-12.txt (Oct. 17, 2005) oid_to_enum(?sect163k1) -> 1; oid_to_enum(?sect163r1) -> 2; diff --git a/lib/ssl/test/ssl_ECC_SUITE.erl b/lib/ssl/test/ssl_ECC_SUITE.erl index 258922d128..bd0c630d41 100644 --- a/lib/ssl/test/ssl_ECC_SUITE.erl +++ b/lib/ssl/test/ssl_ECC_SUITE.erl @@ -46,7 +46,7 @@ groups() -> {'tlsv1', [], all_versions_groups()}, {'erlang_server', [], key_cert_combinations()}, {'erlang_client', [], key_cert_combinations()}, - {'erlang', [], key_cert_combinations() ++ misc()} + {'erlang', [], key_cert_combinations() ++ misc() ++ ecc_negotiation()} ]. all_versions_groups ()-> @@ -68,6 +68,23 @@ key_cert_combinations() -> misc()-> [client_ecdsa_server_ecdsa_with_raw_key]. +ecc_negotiation() -> + [ecc_default_order, + ecc_default_order_custom_curves, + ecc_client_order, + ecc_client_order_custom_curves, + ecc_unknown_curve, + client_ecdh_server_ecdh_ecc_server_custom, + client_rsa_server_ecdh_ecc_server_custom, + client_ecdh_server_rsa_ecc_server_custom, + client_rsa_server_rsa_ecc_server_custom, + client_ecdsa_server_ecdsa_ecc_server_custom, + client_ecdsa_server_rsa_ecc_server_custom, + client_rsa_server_ecdsa_ecc_server_custom, + client_ecdsa_server_ecdsa_ecc_client_custom, + client_rsa_server_ecdsa_ecc_client_custom + ]. + %%-------------------------------------------------------------------- init_per_suite(Config0) -> end_per_suite(Config0), @@ -218,6 +235,132 @@ client_ecdsa_server_ecdsa_with_raw_key(Config) when is_list(Config) -> check_result(Server, SType, Client, CType), close(Server, Client). +ecc_default_order(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [], + case supported_eccs([{eccs, [sect571r1]}]) of + true -> ecc_test(sect571r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +ecc_default_order_custom_curves(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(sect571r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +ecc_client_order(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{honor_ecc_order, false}], + case supported_eccs([{eccs, [sect571r1]}]) of + true -> ecc_test(sect571r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +ecc_client_order_custom_curves(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{honor_ecc_order, false}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(sect571r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +ecc_unknown_curve(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{eccs, ['123_fake_curve']}], + ecc_test_error(COpts, SOpts, [], ECCOpts, Config). + +%% We can only expect to see a named curve on a conn with +%% a server supporting ecdsa. Otherwise the curve is selected +%% but not used and communicated to the client? +client_ecdh_server_ecdh_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_ecdh_rsa_opts, Config), + SOpts = proplists:get_value(server_ecdh_rsa_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(undefined, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_ecdh_server_rsa_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_ecdh_rsa_opts, Config), + SOpts = proplists:get_value(server_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(undefined, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_rsa_server_ecdh_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_opts, Config), + SOpts = proplists:get_value(server_ecdh_rsa_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(undefined, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_rsa_server_rsa_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_opts, Config), + SOpts = proplists:get_value(server_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(undefined, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_ecdsa_server_ecdsa_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(secp256r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_ecdsa_server_rsa_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(undefined, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_rsa_server_ecdsa_ecc_server_custom(Config) -> + COpts = proplists:get_value(client_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{honor_ecc_order, true}, {eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(secp256r1, COpts, SOpts, [], ECCOpts, Config); + false -> {skip, "unsupported named curves"} + end. + +client_ecdsa_server_ecdsa_ecc_client_custom(Config) -> + COpts = proplists:get_value(client_ecdsa_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(secp256r1, COpts, SOpts, ECCOpts, [], Config); + false -> {skip, "unsupported named curves"} + end. + +client_rsa_server_ecdsa_ecc_client_custom(Config) -> + COpts = proplists:get_value(client_opts, Config), + SOpts = proplists:get_value(server_ecdsa_opts, Config), + ECCOpts = [{eccs, [secp256r1, sect571r1]}], + case supported_eccs(ECCOpts) of + true -> ecc_test(secp256r1, COpts, SOpts, ECCOpts, [], Config); + false -> {skip, "unsupported named curves"} + end. + %%-------------------------------------------------------------------- %% Internal functions ------------------------------------------------ %%-------------------------------------------------------------------- @@ -244,6 +387,30 @@ basic_test(ClientCert, ClientKey, ClientCA, ServerCert, ServerKey, ServerCA, Con check_result(Server, SType, Client, CType), close(Server, Client). +ecc_test(Expect, COpts, SOpts, CECCOpts, SECCOpts, Config) -> + CCA = proplists:get_value(cacertfile, COpts), + CCert = proplists:get_value(certfile, COpts), + CKey = proplists:get_value(keyfile, COpts), + SCA = proplists:get_value(cacertfile, SOpts), + SCert = proplists:get_value(certfile, SOpts), + SKey = proplists:get_value(keyfile, SOpts), + {Server, Port} = start_server_ecc(erlang, CCA, SCA, SCert, SKey, Expect, SECCOpts, Config), + Client = start_client_ecc(erlang, Port, SCA, CCA, CCert, CKey, Expect, CECCOpts, Config), + ssl_test_lib:check_result(Server, ok, Client, ok), + close(Server, Client). + +ecc_test_error(COpts, SOpts, CECCOpts, SECCOpts, Config) -> + CCA = proplists:get_value(cacertfile, COpts), + CCert = proplists:get_value(certfile, COpts), + CKey = proplists:get_value(keyfile, COpts), + SCA = proplists:get_value(cacertfile, SOpts), + SCert = proplists:get_value(certfile, SOpts), + SKey = proplists:get_value(keyfile, SOpts), + {Server, Port} = start_server_ecc_error(erlang, CCA, SCA, SCert, SKey, SECCOpts, Config), + Client = start_client_ecc_error(erlang, Port, SCA, CCA, CCert, CKey, CECCOpts, Config), + Error = {error, {tls_alert, "insufficient security"}}, + ssl_test_lib:check_result(Server, Error, Client, Error). + start_client(openssl, Port, PeerCA, OwnCa, Cert, Key, _Config) -> CA = new_openssl_ca("openssl_client_ca", PeerCA, OwnCa), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), @@ -267,6 +434,31 @@ start_client(erlang, Port, PeerCA, OwnCa, Cert, Key, Config) -> {cacertfile, CA}, {certfile, Cert}, {keyfile, Key}]}]). +start_client_ecc(erlang, Port, PeerCA, OwnCa, Cert, Key, Expect, ECCOpts, Config) -> + CA = new_ca("erlang_client_ca", PeerCA, OwnCa), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, check_ecc, [client, Expect]}}, + {options, + ECCOpts ++ + [{verify, verify_peer}, + {cacertfile, CA}, + {certfile, Cert}, {keyfile, Key}]}]). + +start_client_ecc_error(erlang, Port, PeerCA, OwnCa, Cert, Key, ECCOpts, Config) -> + CA = new_ca("erlang_client_ca", PeerCA, OwnCa), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + ssl_test_lib:start_client_error([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {options, + ECCOpts ++ + [{verify, verify_peer}, + {cacertfile, CA}, + {certfile, Cert}, {keyfile, Key}]}]). + start_server(openssl, PeerCA, OwnCa, Cert, Key, _Config) -> CA = new_openssl_ca("openssl_server_ca", PeerCA, OwnCa), Port = ssl_test_lib:inet_port(node()), @@ -290,6 +482,7 @@ start_server(erlang, PeerCA, OwnCa, Cert, Key, Config) -> [{verify, verify_peer}, {cacertfile, CA}, {certfile, Cert}, {keyfile, Key}]}]), {Server, ssl_test_lib:inet_port(Server)}. + start_server_with_raw_key(erlang, PeerCA, OwnCa, Cert, Key, Config) -> CA = new_ca("erlang_server_ca", PeerCA, OwnCa), {_, ServerNode, _} = ssl_test_lib:run_where(Config), @@ -303,6 +496,29 @@ start_server_with_raw_key(erlang, PeerCA, OwnCa, Cert, Key, Config) -> {certfile, Cert}, {key, Key}]}]), {Server, ssl_test_lib:inet_port(Server)}. +start_server_ecc(erlang, PeerCA, OwnCa, Cert, Key, Expect, ECCOpts, Config) -> + CA = new_ca("erlang_server_ca", PeerCA, OwnCa), + {_, ServerNode, _} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, check_ecc, [server, Expect]}}, + {options, + ECCOpts ++ + [{verify, verify_peer}, {cacertfile, CA}, + {certfile, Cert}, {keyfile, Key}]}]), + {Server, ssl_test_lib:inet_port(Server)}. + +start_server_ecc_error(erlang, PeerCA, OwnCa, Cert, Key, ECCOpts, Config) -> + CA = new_ca("erlang_server_ca", PeerCA, OwnCa), + {_, ServerNode, _} = ssl_test_lib:run_where(Config), + Server = ssl_test_lib:start_server_error([{node, ServerNode}, {port, 0}, + {from, self()}, + {options, + ECCOpts ++ + [{verify, verify_peer}, {cacertfile, CA}, + {certfile, Cert}, {keyfile, Key}]}]), + {Server, ssl_test_lib:inet_port(Server)}. + check_result(Server, erlang, Client, erlang) -> ssl_test_lib:check_result(Server, ok, Client, ok); check_result(Server, erlang, _, _) -> @@ -362,3 +578,17 @@ new_openssl_ca(FileName, CA, OwnCa) -> file:write_file(FileName, Pem) end, FileName. + +supported_eccs(Opts) -> + ToCheck = proplists:get_value(eccs, Opts, []), + Supported = ssl:eccs(), + lists:all(fun(Curve) -> lists:member(Curve, Supported) end, ToCheck). + +check_ecc(SSL, Role, Expect) -> + {ok, Data} = ssl:connection_information(SSL), + case lists:keyfind(ecc, 1, Data) of + {ecc, {named_curve, Expect}} -> ok; + false when Expect =:= undefined -> ok; + Other -> {error, Role, Expect, Other} + end. + diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index 1be43c56c4..f8dea736ae 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -150,6 +150,7 @@ api_tests() -> peercert_with_client_cert, sockname, versions, + eccs, controlling_process, getstat, close_with_timeout, @@ -456,6 +457,15 @@ init_per_testcase(accept_pool, Config) -> init_per_testcase(controller_dies, Config) -> ct:timetrap({seconds, 10}), Config; +init_per_testcase(eccs, Config) -> + case ssl:eccs() of + [] -> + {skip, "named curves not supported"}; + [_|_] -> + ssl_test_lib:ct_log_supported_protocol_versions(Config), + ct:timetrap({seconds, 5}), + Config + end; init_per_testcase(_TestCase, Config) -> ssl_test_lib:ct_log_supported_protocol_versions(Config), ct:timetrap({seconds, 5}), @@ -1504,6 +1514,25 @@ versions(Config) when is_list(Config) -> [_|_] = Versions = ssl:versions(), ct:log("~p~n", [Versions]). + +%%-------------------------------------------------------------------- +eccs() -> + [{doc, "Test API functions eccs/0 and eccs/1"}]. + +eccs(Config) when is_list(Config) -> + [_|_] = All = ssl:eccs(), + [] = SSL3 = ssl:eccs({3,0}), + [_|_] = Tls = ssl:eccs({3,1}), + [_|_] = Tls1 = ssl:eccs({3,2}), + [_|_] = Tls2 = ssl:eccs({3,3}), + [] = SSL3 = ssl:eccs(sslv3), + [_|_] = Tls = ssl:eccs(tlsv1), + [_|_] = Tls1 = ssl:eccs('tlsv1.1'), + [_|_] = Tls2 = ssl:eccs('tlsv1.2'), + %% ordering is currently unverified by the test + true = lists:sort(All) =:= lists:usort(SSL3 ++ Tls ++ Tls1 ++ Tls2), + ok. + %%-------------------------------------------------------------------- send_recv() -> [{doc,""}]. diff --git a/lib/stdlib/doc/src/gen_statem.xml b/lib/stdlib/doc/src/gen_statem.xml index 64267c2af5..fd498ee82e 100644 --- a/lib/stdlib/doc/src/gen_statem.xml +++ b/lib/stdlib/doc/src/gen_statem.xml @@ -533,7 +533,7 @@ handle_event(_, _, State, Data) -> Type <c>info</c> originates from regular process messages sent to the <c>gen_statem</c>. Also, the state machine implementation can generate events of types - <c>timeout</c>, <c>state_timeout</c>, <c>enter</c>, + <c>timeout</c>, <c>state_timeout</c>, and <c>internal</c> to itself. </p> </desc> @@ -639,6 +639,20 @@ handle_event(_, _, State, Data) -> </p> <list type="ordered"> <item> + <p> + If the state changes or is the initial state, and + <seealso marker="#type-state_enter"><em>state enter calls</em></seealso> + are used, the <c>gen_statem</c> calls + the new state callback with arguments + <seealso marker="#type-state_enter">(enter, OldState, Data)</seealso>. + Any + <seealso marker="#type-enter_action"><c>actions</c></seealso> + returned from this call are handled as if they were + appended to the actions + returned by the state callback that changed states. + </p> + </item> + <item> <p> All <seealso marker="#type-action">actions</seealso> @@ -668,36 +682,36 @@ handle_event(_, _, State, Data) -> </p> </item> <item> - <p> - If the state changes or is the initial state, and - <seealso marker="#type-state_enter"><em>state enter calls</em></seealso> - are used, the <c>gen_statem</c> calls - the new state callback with arguments - <seealso marker="#type-state_enter">(enter, OldState, Data)</seealso>. - Any - <seealso marker="#type-enter_action"><c>actions</c></seealso> - returned from this call are handled as if they were - appended to the actions - returned by the state callback that changed states. - </p> - </item> - <item> - <p> - If there are enqueued events the (possibly new) - <seealso marker="#state callback">state callback</seealso> - is called with the oldest enqueued event, - and we start again from the top of this list. - </p> - </item> - <item> <p> Timeout timers <seealso marker="#type-state_timeout"><c>state_timeout()</c></seealso> and <seealso marker="#type-event_timeout"><c>event_timeout()</c></seealso> - are handled. This may lead to a time-out zero event - being generated to the + are handled. Time-outs with zero time are guaranteed to be + delivered to the state machine before any external + not yet received event so if there is such a timeout requested, + the corresponding time-out zero event is enqueued as + the newest event. + </p> + <p> + Any event cancels an + <seealso marker="#type-event_timeout"><c>event_timeout()</c></seealso> + so a zero time event time-out is only generated + if the event queue is empty. + </p> + <p> + A state change cancels a + <seealso marker="#type-state_timeout"><c>state_timeout()</c></seealso> + and any new transition option of this type + belongs to the new state. + </p> + </item> + <item> + <p> + If there are enqueued events the <seealso marker="#state callback">state callback</seealso> + for the possibly new state + is called with the oldest enqueued event, and we start again from the top of this list. </p> </item> @@ -759,8 +773,9 @@ handle_event(_, _, State, Data) -> after this time (in milliseconds) unless another event arrives or has arrived in which case this time-out is cancelled. - Note that a retried, inserted or state time-out zero - events counts as arrived. + Note that a retried or inserted event counts as arrived. + So does a state time-out zero event, if it was generated + before this timer is requested. </p> <p> If the value is <c>infinity</c>, no timer is started, as @@ -802,7 +817,7 @@ handle_event(_, _, State, Data) -> <p> Setting this timer while it is running will restart it with the new time-out value. Therefore it is possible to cancel - this timeout by setting it to <c>infinity</c>. + this time-out by setting it to <c>infinity</c>. </p> </desc> </datatype> @@ -1130,7 +1145,7 @@ handle_event(_, _, State, Data) -> <c><anno>Timeout</anno></c> can also be a tuple <c>{clean_timeout,<anno>T</anno>}</c> or <c>{dirty_timeout,<anno>T</anno>}</c>, where - <c><anno>T</anno></c> is the timeout time. + <c><anno>T</anno></c> is the time-out time. <c>{clean_timeout,<anno>T</anno>}</c> works like just <c>T</c> described in the note above and uses a proxy process for <c>T < infinity</c>, @@ -1773,7 +1788,7 @@ handle_event(_, _, State, Data) -> StateFunctionResult </name> <name>Module:handle_event(enter, OldState, State, Data) -> - StateEnterResult + StateEnterResult(State) </name> <name>Module:handle_event(EventType, EventContent, State, Data) -> HandleEventResult @@ -1802,8 +1817,8 @@ handle_event(_, _, State, Data) -> <seealso marker="#type-event_handler_result">event_handler_result</seealso>(<seealso marker="#type-state_name">state_name()</seealso>) </v> <v> - StateEnterResult = - <seealso marker="#type-state_enter_result">state_enter_result</seealso>(<seealso marker="#type-state">state()</seealso>) + StateEnterResult(State) = + <seealso marker="#type-state_enter_result">state_enter_result(State)</seealso> </v> <v> HandleEventResult = diff --git a/lib/stdlib/doc/src/maps.xml b/lib/stdlib/doc/src/maps.xml index e1edbadcd3..8c7270816b 100644 --- a/lib/stdlib/doc/src/maps.xml +++ b/lib/stdlib/doc/src/maps.xml @@ -160,7 +160,7 @@ val1 <p><em>Example:</em></p> <code type="none"> > Map = #{"42" => value}. -#{"42"> => value} +#{"42" => value} > maps:is_key("42",Map). true > maps:is_key(value,Map). diff --git a/lib/stdlib/src/gen_statem.erl b/lib/stdlib/src/gen_statem.erl index 17d1ebecec..018aca90e6 100644 --- a/lib/stdlib/src/gen_statem.erl +++ b/lib/stdlib/src/gen_statem.erl @@ -85,7 +85,8 @@ -type state_enter() :: 'state_enter'. -type transition_option() :: - postpone() | hibernate() | event_timeout(). + postpone() | hibernate() | + event_timeout() | state_timeout(). -type postpone() :: %% If 'true' postpone the current event %% and retry it when the state changes (=/=) @@ -108,7 +109,7 @@ %% * All action()s are executed in order of apperance. %% * Postponing the current event is performed %% iff 'postpone' is 'true'. - %% * A state timer is started iff 'timeout' is set. + %% * A state timeout is started iff 'timeout' is set. %% * Pending events are handled or if there are %% no pending events the server goes into receive %% or hibernate (iff 'hibernate' is 'true') @@ -154,12 +155,12 @@ -type handle_event_result() :: event_handler_result(state()). %% --type state_enter_result(StateType) :: +-type state_enter_result(State) :: {'next_state', % {next_state,NextState,NewData,[]} - State :: StateType, + State, NewData :: data()} | {'next_state', % State transition, maybe to the same state - State :: StateType, + State, NewData :: data(), Actions :: [enter_action()] | enter_action()} | state_callback_result(enter_action()). @@ -231,9 +232,9 @@ -callback handle_event( 'enter', OldState :: state(), - State :: state(), % Current state + State, % Current state Data :: data()) -> - state_enter_result(state()); + state_enter_result(State); (event_type(), EventContent :: term(), State :: state(), % Current state @@ -596,8 +597,8 @@ enter(Module, Opts, State, Data, Server, Actions, Parent) -> data => Data, postponed => P, %% The rest of the fields are set from to the arguments to - %% loop_event_actions/9 when it finally loops back to loop/3 - %% in loop_events_done/9 + %% loop_event_actions/10 when it finally loops back to loop/3 + %% in loop_events/10 %% %% Marker for initial state, cleared immediately when used init_state => true @@ -605,9 +606,10 @@ enter(Module, Opts, State, Data, Server, Actions, Parent) -> NewDebug = sys_debug(Debug, S, State, {enter,Event,State}), case call_callback_mode(S) of {ok,NewS} -> - StateTimer = undefined, + TimerRefs = #{}, + TimerTypes = #{}, loop_event_actions( - Parent, NewDebug, NewS, StateTimer, + Parent, NewDebug, NewS, TimerRefs, TimerTypes, Events, Event, State, Data, NewActions); {Class,Reason,Stacktrace} -> terminate( @@ -747,6 +749,10 @@ print_event(Dev, {out,Reply,{To,_Tag}}, {Name,State}) -> io:format( Dev, "*DBG* ~p send ~p to ~p from state ~p~n", [Name,Reply,To,State]); +print_event(Dev, {terminate,Reason}, {Name,State}) -> + io:format( + Dev, "*DBG* ~p terminate ~p in state ~p~n", + [Name,Reason,State]); print_event(Dev, {Tag,Event,NextState}, {Name,State}) -> StateString = case NextState of @@ -806,7 +812,7 @@ loop(Parent, Debug, #{hibernate := Hibernate} = S) -> %% Entry point for wakeup_from_hibernate/3 loop_receive( - Parent, Debug, #{timer := Timer, state_timer := StateTimer} = S) -> + Parent, Debug, #{timer_refs := TimerRefs, timer_types := TimerTypes} = S) -> receive Msg -> case Msg of @@ -822,18 +828,23 @@ loop_receive( %% but this will stand out in the crash report... terminate( exit, Reason, ?STACKTRACE(), Debug, S, [EXIT]); - {timeout,Timer,Content} - when Timer =/= undefined -> - loop_receive_result( - Parent, Debug, S, StateTimer, - {timeout,Content}); - {timeout,StateTimer,Content} - when StateTimer =/= undefined -> - loop_receive_result( - Parent, Debug, S, undefined, - {state_timeout,Content}); + {timeout,TimerRef,TimerMsg} -> + case TimerRefs of + #{TimerRef := TimerType} -> + Event = {TimerType,TimerMsg}, + %% Unregister the triggered timeout + loop_receive_result( + Parent, Debug, S, + maps:remove(TimerRef, TimerRefs), + maps:remove(TimerType, TimerTypes), + Event); + _ -> + Event = {info,Msg}, + loop_receive_result( + Parent, Debug, S, + TimerRefs, TimerTypes, Event) + end; _ -> - cancel_timer(Timer), Event = case Msg of {'$gen_call',From,Request} -> @@ -844,12 +855,15 @@ loop_receive( {info,Msg} end, loop_receive_result( - Parent, Debug, S, StateTimer, Event) + Parent, Debug, S, + TimerRefs, TimerTypes, Event) end end. -loop_receive_result(Parent, Debug, #{state := State} = S, StateTimer, Event) -> - %% The fields 'timer', 'state_timer' and 'hibernate' +loop_receive_result( + Parent, Debug, #{state := State} = S, + TimerRefs, TimerTypes, Event) -> + %% The fields 'timer_refs', 'timer_types' and 'hibernate' %% are now invalid in state map S - they will be recalculated %% and restored when we return to loop/3 %% @@ -857,82 +871,196 @@ loop_receive_result(Parent, Debug, #{state := State} = S, StateTimer, Event) -> %% Here the queue of not yet handled events is created Events = [], Hibernate = false, - loop_event(Parent, NewDebug, S, StateTimer, Events, Event, Hibernate). + loop_event( + Parent, NewDebug, S, TimerRefs, TimerTypes, Events, Event, Hibernate). -%% Process the event queue, or if it is empty -%% loop back to loop/3 to receive a new event -loop_events( - Parent, Debug, S, StateTimeout, - [Event|Events], _Timeout, State, Data, P, Hibernate) -> +%% Entry point for handling an event, received or enqueued +loop_event( + Parent, Debug, #{state := State, data := Data} = S, TimerRefs, TimerTypes, + Events, {Type,Content} = Event, Hibernate) -> %% - %% If there was an event timer requested we just ignore that - %% since we have events to handle which cancels the timer - loop_event( - Parent, Debug, S, StateTimeout, - Events, Event, State, Data, P, Hibernate); -loop_events( - Parent, Debug, S, {state_timeout,Time,EventContent}, - [] = Events, Timeout, State, Data, P, Hibernate) -> - if - Time =:= 0 -> - %% Simulate an immediate timeout - %% so we do not get the timeout message - %% after any received event - %% - %% This faked event will cancel - %& any not yet started event timer - Event = {state_timeout,EventContent}, - StateTimer = undefined, - loop_event( - Parent, Debug, S, StateTimer, - Events, Event, State, Data, P, Hibernate); - true -> - StateTimer = erlang:start_timer(Time, self(), EventContent), - loop_events( - Parent, Debug, S, StateTimer, - Events, Timeout, State, Data, P, Hibernate) - end; -loop_events( - Parent, Debug, S, StateTimer, - [] = Events, Timeout, State, Data, P, Hibernate) -> - case Timeout of - {timeout,0,EventContent} -> - %% Simulate an immediate timeout - %% so we do not get the timeout message - %% after any received event - %% - Event = {timeout,EventContent}, - loop_event( - Parent, Debug, S, StateTimer, - Events, Event, State, Data, P, Hibernate); - {timeout,Time,EventContent} -> - Timer = erlang:start_timer(Time, self(), EventContent), - loop_events_done( - Parent, Debug, S, StateTimer, - State, Data, P, Hibernate, Timer); - undefined -> - %% No event timeout has been requested - Timer = undefined, - loop_events_done( - Parent, Debug, S, StateTimer, - State, Data, P, Hibernate, Timer) + %% If Hibernate is true here it can only be + %% because it was set from an event action + %% and we did not go into hibernation since there + %% were events in queue, so we do what the user + %% might rely on i.e collect garbage which + %% would have happened if we actually hibernated + %% and immediately was awakened + Hibernate andalso garbage_collect(), + case call_state_function(S, Type, Content, State, Data) of + {ok,Result,NewS} -> + %% Cancel event timeout + {NewTimerRefs,NewTimerTypes} = + cancel_timer_by_type( + timeout, TimerRefs, TimerTypes), + {NewData,NextState,Actions} = + parse_event_result( + true, Debug, NewS, Result, + Events, Event, State, Data), + loop_event_actions( + Parent, Debug, S, NewTimerRefs, NewTimerTypes, + Events, Event, NextState, NewData, Actions); + {Class,Reason,Stacktrace} -> + terminate( + Class, Reason, Stacktrace, Debug, S, [Event|Events]) end. -%% Back to the top -loop_events_done( - Parent, Debug, S, StateTimer, - State, Data, P, Hibernate, Timer) -> +loop_event_actions( + Parent, Debug, + #{state := State, state_enter := StateEnter} = S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, Actions) -> + case parse_actions(Debug, S, State, Actions) of + {ok,NewDebug,Hibernate,TimeoutsR,Postpone,NextEventsR} -> + if + StateEnter, NextState =/= State -> + loop_event_enter( + Parent, NewDebug, S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR); + StateEnter -> + case maps:is_key(init_state, S) of + true -> + %% Avoid infinite loop in initial state + %% with state entry events + NewS = maps:remove(init_state, S), + loop_event_enter( + Parent, NewDebug, NewS, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR); + false -> + loop_event_result( + Parent, NewDebug, S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR) + end; + true -> + loop_event_result( + Parent, NewDebug, S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR) + end; + {Class,Reason,Stacktrace} -> + terminate( + Class, Reason, Stacktrace, + Debug, S#{data := NewData}, [Event|Events]) + end. + +loop_event_enter( + Parent, Debug, #{state := State} = S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR) -> + case call_state_function(S, enter, State, NextState, NewData) of + {ok,Result,NewS} -> + {NewerData,_,Actions} = + parse_event_result( + false, Debug, NewS, Result, + Events, Event, NextState, NewData), + loop_event_enter_actions( + Parent, Debug, NewS, TimerRefs, TimerTypes, + Events, Event, NextState, NewerData, + Hibernate, TimeoutsR, Postpone, NextEventsR, Actions); + {Class,Reason,Stacktrace} -> + terminate( + Class, Reason, Stacktrace, + Debug, S#{state := NextState, data := NewData}, + [Event|Events]) + end. + +loop_event_enter_actions( + Parent, Debug, S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR, Actions) -> + case + parse_enter_actions( + Debug, S, NextState, Actions, + Hibernate, TimeoutsR) + of + {ok,NewDebug,NewHibernate,NewTimeoutsR,_,_} -> + loop_event_result( + Parent, NewDebug, S, TimerRefs, TimerTypes, + Events, Event, NextState, NewData, + NewHibernate, NewTimeoutsR, Postpone, NextEventsR); + {Class,Reason,Stacktrace} -> + terminate( + Class, Reason, Stacktrace, + Debug, S#{state := NextState, data := NewData}, + [Event|Events]) + end. + +loop_event_result( + Parent, Debug, + #{state := State, postponed := P_0} = S, TimerRefs_0, TimerTypes_0, + Events, Event, NextState, NewData, + Hibernate, TimeoutsR, Postpone, NextEventsR) -> + %% + %% All options have been collected and next_events are buffered. + %% Do the actual state transition. + %% + {NewDebug,P_1} = % Move current event to postponed if Postpone + case Postpone of + true -> + {sys_debug(Debug, S, State, {postpone,Event,State}), + [Event|P_0]}; + false -> + {sys_debug(Debug, S, State, {consume,Event,State}), + P_0} + end, + {Events_1,NewP,{TimerRefs_1,TimerTypes_1}} = + %% Move all postponed events to queue and cancel the + %% state timeout if the state changes + if + NextState =:= State -> + {Events,P_1,{TimerRefs_0,TimerTypes_0}}; + true -> + {lists:reverse(P_1, Events),[], + cancel_timer_by_type( + state_timeout, TimerRefs_0, TimerTypes_0)} + end, + {TimerRefs_2,TimerTypes_2,TimeoutEvents} = + %% Stop and start timers non-event timers + parse_timers(TimerRefs_1, TimerTypes_1, TimeoutsR), + %% Place next events last in reversed queue + Events_2R = lists:reverse(Events_1, NextEventsR), + %% Enqueue immediate timeout events and start event timer + {NewTimerRefs,NewTimerTypes,Events_3R} = + process_timeout_events( + TimerRefs_2, TimerTypes_2, TimeoutEvents, Events_2R), + NewEvents = lists:reverse(Events_3R), + loop_events( + Parent, NewDebug, S, NewTimerRefs, NewTimerTypes, + NewEvents, Hibernate, NextState, NewData, NewP). + +%% Loop until out of enqueued events +%% +loop_events( + Parent, Debug, S, TimerRefs, TimerTypes, + [] = _Events, Hibernate, State, Data, P) -> + %% Update S and loop back to loop/3 to receive a new event NewS = S#{ state := State, data := Data, postponed := P, hibernate => Hibernate, - timer => Timer, - state_timer => StateTimer}, - loop(Parent, Debug, NewS). + timer_refs => TimerRefs, + timer_types => TimerTypes}, + loop(Parent, Debug, NewS); +loop_events( + Parent, Debug, S, TimerRefs, TimerTypes, + [Event|Events], Hibernate, State, Data, P) -> + %% Update S and continue with enqueued events + NewS = + S#{ + state := State, + data := Data, + postponed := P}, + loop_event( + Parent, Debug, NewS, TimerRefs, TimerTypes, Events, Event, Hibernate). + +%%--------------------------------------------------------------------------- +%% Server loop helpers call_callback_mode(#{module := Module} = S) -> try Module:callback_mode() of @@ -996,6 +1124,7 @@ parse_callback_mode([H|T], CBMode, StateEnter) -> parse_callback_mode(_, _CBMode, StateEnter) -> {undefined,StateEnter}. + call_state_function( #{callback_mode := undefined} = S, Type, Content, State, Data) -> @@ -1061,42 +1190,6 @@ call_state_function( {Class,Reason,erlang:get_stacktrace()} end. -%% Update S and continue -loop_event( - Parent, Debug, S, StateTimer, - Events, Event, State, Data, P, Hibernate) -> - NewS = - S#{ - state := State, - data := Data, - postponed := P}, - loop_event(Parent, Debug, NewS, StateTimer, Events, Event, Hibernate). - -loop_event( - Parent, Debug, #{state := State, data := Data} = S, StateTimer, - Events, {Type,Content} = Event, Hibernate) -> - %% - %% If Hibernate is true here it can only be - %% because it was set from an event action - %% and we did not go into hibernation since there - %% were events in queue, so we do what the user - %% might rely on i.e collect garbage which - %% would have happened if we actually hibernated - %% and immediately was awakened - Hibernate andalso garbage_collect(), - case call_state_function(S, Type, Content, State, Data) of - {ok,Result,NewS} -> - {NewData,NextState,Actions} = - parse_event_result( - true, Debug, NewS, Result, - Events, Event, State, Data), - loop_event_actions( - Parent, Debug, S, StateTimer, - Events, Event, NextState, NewData, Actions); - {Class,Reason,Stacktrace} -> - terminate( - Class, Reason, Stacktrace, Debug, S, [Event|Events]) - end. %% Interpret all callback return variants parse_event_result( @@ -1146,32 +1239,32 @@ parse_event_result( Debug, S, [Event|Events]) end. + parse_enter_actions( Debug, S, State, Actions, - Hibernate, Timeout, StateTimeout) -> + Hibernate, TimeoutsR) -> Postpone = forbidden, - NextEvents = forbidden, + NextEventsR = forbidden, parse_actions( Debug, S, State, listify(Actions), - Hibernate, Timeout, StateTimeout, Postpone, NextEvents). + Hibernate, TimeoutsR, Postpone, NextEventsR). parse_actions(Debug, S, State, Actions) -> Hibernate = false, - Timeout = undefined, - StateTimeout = undefined, + TimeoutsR = [], Postpone = false, - NextEvents = [], + NextEventsR = [], parse_actions( Debug, S, State, listify(Actions), - Hibernate, Timeout, StateTimeout, Postpone, NextEvents). + Hibernate, TimeoutsR, Postpone, NextEventsR). %% parse_actions( Debug, _S, _State, [], - Hibernate, Timeout, StateTimeout, Postpone, NextEvents) -> - {ok,Debug,Hibernate,Timeout,StateTimeout,Postpone,NextEvents}; + Hibernate, TimeoutsR, Postpone, NextEventsR) -> + {ok,Debug,Hibernate,TimeoutsR,Postpone,NextEventsR}; parse_actions( Debug, S, State, [Action|Actions], - Hibernate, Timeout, StateTimeout, Postpone, NextEvents) -> + Hibernate, TimeoutsR, Postpone, NextEventsR) -> case Action of %% Actual actions {reply,From,Reply} -> @@ -1180,8 +1273,7 @@ parse_actions( NewDebug = do_reply(Debug, S, State, From, Reply), parse_actions( NewDebug, S, State, Actions, - Hibernate, Timeout, StateTimeout, - Postpone, NextEvents); + Hibernate, TimeoutsR, Postpone, NextEventsR); false -> {error, {bad_action_from_state_function,Action}, @@ -1191,7 +1283,7 @@ parse_actions( {hibernate,NewHibernate} when is_boolean(NewHibernate) -> parse_actions( Debug, S, State, Actions, - NewHibernate, Timeout, StateTimeout, Postpone, NextEvents); + NewHibernate, TimeoutsR, Postpone, NextEventsR); {hibernate,_} -> {error, {bad_action_from_state_function,Action}, @@ -1199,43 +1291,44 @@ parse_actions( hibernate -> parse_actions( Debug, S, State, Actions, - true, Timeout, StateTimeout, Postpone, NextEvents); - {state_timeout,Time,_} = NewStateTimeout + true, TimeoutsR, Postpone, NextEventsR); + {state_timeout,Time,_} = StateTimeout when is_integer(Time), Time >= 0; Time =:= infinity -> parse_actions( Debug, S, State, Actions, - Hibernate, Timeout, NewStateTimeout, Postpone, NextEvents); + Hibernate, [StateTimeout|TimeoutsR], Postpone, NextEventsR); {state_timeout,_,_} -> {error, {bad_action_from_state_function,Action}, ?STACKTRACE()}; - {timeout,infinity,_} -> % Clear timer - it will never trigger + {timeout,infinity,_} -> + %% Ignore - timeout will never happen and already cancelled parse_actions( Debug, S, State, Actions, - Hibernate, undefined, StateTimeout, Postpone, NextEvents); - {timeout,Time,_} = NewTimeout when is_integer(Time), Time >= 0 -> + Hibernate, TimeoutsR, Postpone, NextEventsR); + {timeout,Time,_} = Timeout when is_integer(Time), Time >= 0 -> parse_actions( Debug, S, State, Actions, - Hibernate, NewTimeout, StateTimeout, Postpone, NextEvents); + Hibernate, [Timeout|TimeoutsR], Postpone, NextEventsR); {timeout,_,_} -> {error, {bad_action_from_state_function,Action}, ?STACKTRACE()}; - infinity -> % Clear timer - it will never trigger + infinity -> % Ignore - timeout will never happen parse_actions( Debug, S, State, Actions, - Hibernate, undefined, StateTimeout, Postpone, NextEvents); + Hibernate, TimeoutsR, Postpone, NextEventsR); Time when is_integer(Time), Time >= 0 -> - NewTimeout = {timeout,Time,Time}, + Timeout = {timeout,Time,Time}, parse_actions( Debug, S, State, Actions, - Hibernate, NewTimeout, StateTimeout, Postpone, NextEvents); + Hibernate, [Timeout|TimeoutsR], Postpone, NextEventsR); {postpone,NewPostpone} when is_boolean(NewPostpone), Postpone =/= forbidden -> parse_actions( Debug, S, State, Actions, - Hibernate, Timeout, StateTimeout, NewPostpone, NextEvents); + Hibernate, TimeoutsR, NewPostpone, NextEventsR); {postpone,_} -> {error, {bad_action_from_state_function,Action}, @@ -1243,16 +1336,16 @@ parse_actions( postpone when Postpone =/= forbidden -> parse_actions( Debug, S, State, Actions, - Hibernate, Timeout, StateTimeout, true, NextEvents); + Hibernate, TimeoutsR, true, NextEventsR); {next_event,Type,Content} -> case event_type(Type) of - true when NextEvents =/= forbidden -> + true when NextEventsR =/= forbidden -> NewDebug = sys_debug(Debug, S, State, {in,{Type,Content}}), parse_actions( NewDebug, S, State, Actions, - Hibernate, Timeout, StateTimeout, - Postpone, [{Type,Content}|NextEvents]); + Hibernate, TimeoutsR, Postpone, + [{Type,Content}|NextEventsR]); _ -> {error, {bad_action_from_state_function,Action}, @@ -1264,158 +1357,92 @@ parse_actions( ?STACKTRACE()} end. -loop_event_actions( - Parent, Debug, - #{state := State, state_enter := StateEnter} = S, StateTimer, - Events, Event, NextState, NewData, Actions) -> - case parse_actions(Debug, S, State, Actions) of - {ok,NewDebug,Hibernate,Timeout,StateTimeout,Postpone,NextEvents} -> + +%% Stop and start timers as well as create timeout zero events +%% and pending event timer +%% +%% Stop and start timers non-event timers +parse_timers(TimerRefs, TimerTypes, TimeoutsR) -> + parse_timers(TimerRefs, TimerTypes, TimeoutsR, #{}, []). +%% +parse_timers(TimerRefs, TimerTypes, [], _Seen, TimeoutEvents) -> + {TimerRefs,TimerTypes,TimeoutEvents}; +parse_timers( + TimerRefs, TimerTypes, [Timeout|TimeoutsR], Seen, TimeoutEvents) -> + {TimerType,Time,TimerMsg} = Timeout, + case Seen of + #{TimerType := _} -> + %% Type seen before - ignore + parse_timers( + TimerRefs, TimerTypes, TimeoutsR, Seen, TimeoutEvents); + #{} -> + %% Unseen type - handle + NewSeen = Seen#{TimerType => true}, + %% Cancel any running timer + {NewTimerRefs,NewTimerTypes} = + cancel_timer_by_type(TimerType, TimerRefs, TimerTypes), if - StateEnter, NextState =/= State -> - loop_event_enter( - Parent, NewDebug, S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents); - StateEnter -> - case maps:is_key(init_state, S) of - true -> - %% Avoid infinite loop in initial state - %% with state entry events - NewS = maps:remove(init_state, S), - loop_event_enter( - Parent, NewDebug, NewS, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, - Postpone, NextEvents); - false -> - loop_event_result( - Parent, NewDebug, S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, - Postpone, NextEvents) - end; + Time =:= infinity -> + %% Ignore - timer will never fire + parse_timers( + NewTimerRefs, NewTimerTypes, TimeoutsR, + NewSeen, TimeoutEvents); + TimerType =:= timeout -> + %% Handle event timer later + parse_timers( + NewTimerRefs, NewTimerTypes, TimeoutsR, + NewSeen, [Timeout|TimeoutEvents]); + Time =:= 0 -> + %% Handle zero time timeouts later + TimeoutEvent = {TimerType,TimerMsg}, + parse_timers( + NewTimerRefs, NewTimerTypes, TimeoutsR, + NewSeen, [TimeoutEvent|TimeoutEvents]); true -> - loop_event_result( - Parent, NewDebug, S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents) - end; - {Class,Reason,Stacktrace} -> - terminate( - Class, Reason, Stacktrace, - Debug, S#{data := NewData}, [Event|Events]) + %% Start a new timer + TimerRef = erlang:start_timer(Time, self(), TimerMsg), + parse_timers( + NewTimerRefs#{TimerRef => TimerType}, + NewTimerTypes#{TimerType => TimerRef}, + TimeoutsR, NewSeen, TimeoutEvents) + end end. -loop_event_enter( - Parent, Debug, #{state := State} = S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents) -> - case call_state_function(S, enter, State, NextState, NewData) of - {ok,Result,NewS} -> - {NewerData,_,Actions} = - parse_event_result( - false, Debug, NewS, Result, - Events, Event, NextState, NewData), - loop_event_enter_actions( - Parent, Debug, NewS, StateTimer, - Events, Event, NextState, NewerData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents, Actions); - {Class,Reason,Stacktrace} -> - terminate( - Class, Reason, Stacktrace, - Debug, S#{state := NextState, data := NewData}, - [Event|Events]) - end. +%% Enqueue immediate timeout events and start event timer +process_timeout_events(TimerRefs, TimerTypes, [], EventsR) -> + {TimerRefs, TimerTypes, EventsR}; +process_timeout_events( + TimerRefs, TimerTypes, + [{timeout,0,TimerMsg}|TimeoutEvents], []) -> + %% No enqueued events - insert a timeout zero event + TimeoutEvent = {timeout,TimerMsg}, + process_timeout_events( + TimerRefs, TimerTypes, + TimeoutEvents, [TimeoutEvent]); +process_timeout_events( + TimerRefs, TimerTypes, + [{timeout,Time,TimerMsg}], []) -> + %% No enqueued events - start event timer + TimerRef = erlang:start_timer(Time, self(), TimerMsg), + process_timeout_events( + TimerRefs#{TimerRef => timeout}, TimerTypes#{timeout => TimerRef}, + [], []); +process_timeout_events( + TimerRefs, TimerTypes, + [{timeout,_Time,_TimerMsg}|TimeoutEvents], EventsR) -> + %% There will be some other event so optimize by not starting + %% an event timer to just have to cancel it again + process_timeout_events( + TimerRefs, TimerTypes, + TimeoutEvents, EventsR); +process_timeout_events( + TimerRefs, TimerTypes, + [{_TimeoutType,_TimeoutMsg} = TimeoutEvent|TimeoutEvents], EventsR) -> + process_timeout_events( + TimerRefs, TimerTypes, + TimeoutEvents, [TimeoutEvent|EventsR]). -loop_event_enter_actions( - Parent, Debug, S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents, Actions) -> - case - parse_enter_actions( - Debug, S, NextState, Actions, - Hibernate, Timeout, StateTimeout) - of - {ok,NewDebug,NewHibernate,NewTimeout,NewStateTimeout,_,_} -> - loop_event_result( - Parent, NewDebug, S, StateTimer, - Events, Event, NextState, NewData, - NewHibernate, NewTimeout, NewStateTimeout, Postpone, NextEvents); - {Class,Reason,Stacktrace} -> - terminate( - Class, Reason, Stacktrace, - Debug, S#{state := NextState, data := NewData}, - [Event|Events]) - end. -loop_event_result( - Parent, Debug, - #{state := State, postponed := P_0} = S, StateTimer, - Events, Event, NextState, NewData, - Hibernate, Timeout, StateTimeout, Postpone, NextEvents) -> - %% - %% All options have been collected and next_events are buffered. - %% Do the actual state transition. - %% - NewStateTimeout = - case StateTimeout of - {state_timeout,Time,_} -> - %% New timeout -> cancel timer - case StateTimer of - {state_timeout,_,_} -> - ok; - _ -> - cancel_timer(StateTimer) - end, - case Time of - infinity -> - undefined; - _ -> - StateTimeout - end; - undefined when NextState =/= State -> - %% State change -> cancel timer - case StateTimer of - {state_timeout,_,_} -> - ok; - _ -> - cancel_timer(StateTimer) - end, - undefined; - undefined -> - StateTimer - end, - %% - P_1 = % Move current event to postponed if Postpone - case Postpone of - true -> - [Event|P_0]; - false -> - P_0 - end, - {Events_1,NewP} = % Move all postponed events to queue if state change - if - NextState =:= State -> - {Events,P_1}; - true -> - {lists:reverse(P_1, Events),[]} - end, - %% Place next events first in queue - NewEvents = lists:reverse(NextEvents, Events_1), - %% - NewDebug = - sys_debug( - Debug, S, State, - case Postpone of - true -> - {postpone,Event,State}; - false -> - {consume,Event,State} - end), - %% - loop_events( - Parent, NewDebug, S, NewStateTimeout, - NewEvents, Timeout, NextState, NewData, NewP, Hibernate). %%--------------------------------------------------------------------------- %% Server helpers @@ -1474,16 +1501,20 @@ terminate( sys:print_log(Debug), erlang:raise(C, R, ST) end, - case Reason of - normal -> ok; - shutdown -> ok; - {shutdown,_} -> ok; - _ -> - error_info( - Class, Reason, Stacktrace, S, Q, P, - format_status(terminate, get(), S)), - sys:print_log(Debug) - end, + _ = + case Reason of + normal -> + sys_debug(Debug, S, State, {terminate,Reason}); + shutdown -> + sys_debug(Debug, S, State, {terminate,Reason}); + {shutdown,_} -> + sys_debug(Debug, S, State, {terminate,Reason}); + _ -> + error_info( + Class, Reason, Stacktrace, S, Q, P, + format_status(terminate, get(), S)), + sys:print_log(Debug) + end, case Stacktrace of [] -> erlang:Class(Reason); @@ -1605,8 +1636,19 @@ listify(Item) when is_list(Item) -> listify(Item) -> [Item]. -cancel_timer(undefined) -> - ok; +%% Cancel timer if running, otherwise no op +cancel_timer_by_type(TimerType, TimerRefs, TimerTypes) -> + case TimerTypes of + #{TimerType := TimerRef} -> + cancel_timer(TimerRef), + {maps:remove(TimerRef, TimerRefs), + maps:remove(TimerType, TimerTypes)}; + #{} -> + {TimerRefs,TimerTypes} + end. + +%%cancel_timer(undefined) -> +%% ok; cancel_timer(TRef) -> case erlang:cancel_timer(TRef) of false -> diff --git a/lib/stdlib/test/base64_SUITE.erl b/lib/stdlib/test/base64_SUITE.erl index 9176a3664a..d0abe5c961 100644 --- a/lib/stdlib/test/base64_SUITE.erl +++ b/lib/stdlib/test/base64_SUITE.erl @@ -23,9 +23,7 @@ -include_lib("common_test/include/ct.hrl"). %% Test server specific exports --export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, - init_per_group/2,end_per_group/2, - init_per_testcase/2, end_per_testcase/2]). +-export([all/0, suite/0, groups/0, group/1]). %% Test cases must be exported. -export([base64_encode/1, base64_decode/1, base64_otp_5635/1, @@ -33,41 +31,26 @@ mime_decode_to_string/1, roundtrip_1/1, roundtrip_2/1, roundtrip_3/1, roundtrip_4/1]). -init_per_testcase(_, Config) -> - Config. - -end_per_testcase(_, _Config) -> - ok. - %%------------------------------------------------------------------------- %% Test cases starts here. %%------------------------------------------------------------------------- + suite() -> [{ct_hooks,[ts_install_cth]}, {timetrap,{minutes,4}}]. -all() -> +all() -> [base64_encode, base64_decode, base64_otp_5635, base64_otp_6279, big, illegal, mime_decode, mime_decode_to_string, {group, roundtrip}]. -groups() -> +groups() -> [{roundtrip, [parallel], [roundtrip_1, roundtrip_2, roundtrip_3, roundtrip_4]}]. -init_per_suite(Config) -> - Config. - -end_per_suite(_Config) -> - ok. - -init_per_group(_GroupName, Config) -> - Config. - -end_per_group(_GroupName, Config) -> - Config. - - +group(roundtrip) -> + %% valgrind needs a lot of time + [{timetrap,{minutes,10}}]. %%------------------------------------------------------------------------- %% Test base64:encode/1. @@ -78,9 +61,9 @@ base64_encode(Config) when is_list(Config) -> %% One pad <<"SGVsbG8gV29ybGQ=">> = base64:encode(<<"Hello World">>), %% No pad - "QWxhZGRpbjpvcGVuIHNlc2Ft" = + "QWxhZGRpbjpvcGVuIHNlc2Ft" = base64:encode_to_string("Aladdin:open sesam"), - + "MDEyMzQ1Njc4OSFAIzBeJiooKTs6PD4sLiBbXXt9" = base64:encode_to_string(<<"0123456789!@#0^&*();:<>,. []{}">>), ok. @@ -93,7 +76,7 @@ base64_decode(Config) when is_list(Config) -> %% One pad <<"Hello World">> = base64:decode(<<"SGVsbG8gV29ybGQ=">>), %% No pad - <<"Aladdin:open sesam">> = + <<"Aladdin:open sesam">> = base64:decode("QWxhZGRpbjpvcGVuIHNlc2Ft"), Alphabet = list_to_binary(lists:seq(0, 255)), @@ -208,7 +191,7 @@ mime_decode_to_string(Config) when is_list(Config) -> %% One pad to ignore, followed by more text "Hello World!!" = base64:mime_decode_to_string(<<"SGVsb)(G8gV29ybGQ=h IQ= =">>), %% No pad - "Aladdin:open sesam" = + "Aladdin:open sesam" = base64:mime_decode_to_string("QWxhZGRpbjpvcG¤\")(VuIHNlc2Ft"), %% Encoded base 64 strings may be divided by non base 64 chars. %% In this cases whitespaces. @@ -314,7 +297,7 @@ interleaved_ws_roundtrip_1([], Base64List, Bin, List) -> random_byte_list(0, Acc) -> Acc; -random_byte_list(N, Acc) -> +random_byte_list(N, Acc) -> random_byte_list(N-1, [rand:uniform(255)|Acc]). make_big_binary(N) -> diff --git a/lib/stdlib/test/ets_SUITE.erl b/lib/stdlib/test/ets_SUITE.erl index b02d17bdb6..00e02a06cc 100644 --- a/lib/stdlib/test/ets_SUITE.erl +++ b/lib/stdlib/test/ets_SUITE.erl @@ -19,7 +19,7 @@ %% -module(ets_SUITE). --export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, +-export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_group/2,end_per_group/2]). -export([default/1,setbag/1,badnew/1,verybadnew/1,named/1,keypos2/1, privacy/1,privacy_owner/2]). @@ -31,15 +31,14 @@ -export([match_delete3/1]). -export([firstnext/1,firstnext_concurrent/1]). -export([slot/1]). --export([ match1/1, match2/1, match_object/1, match_object2/1]). --export([ dups/1, misc1/1, safe_fixtable/1, info/1, tab2list/1]). --export([ tab2file/1, tab2file2/1, tabfile_ext1/1, - tabfile_ext2/1, tabfile_ext3/1, tabfile_ext4/1, badfile/1]). --export([ heavy_lookup/1, heavy_lookup_element/1, heavy_concurrent/1]). --export([ lookup_element_mult/1]). --export([]). +-export([match1/1, match2/1, match_object/1, match_object2/1]). +-export([dups/1, misc1/1, safe_fixtable/1, info/1, tab2list/1]). +-export([tab2file/1, tab2file2/1, tabfile_ext1/1, + tabfile_ext2/1, tabfile_ext3/1, tabfile_ext4/1, badfile/1]). +-export([heavy_lookup/1, heavy_lookup_element/1, heavy_concurrent/1]). +-export([lookup_element_mult/1]). -export([foldl_ordered/1, foldr_ordered/1, foldl/1, foldr/1, fold_empty/1]). --export([t_delete_object/1, t_init_table/1, t_whitebox/1, +-export([t_delete_object/1, t_init_table/1, t_whitebox/1, t_delete_all_objects/1, t_insert_list/1, t_test_ms/1, t_select_delete/1,t_ets_dets/1]). @@ -61,8 +60,7 @@ -export([otp_7665/1]). -export([meta_wb/1]). -export([grow_shrink/1, grow_pseudo_deleted/1, shrink_pseudo_deleted/1]). --export([ - meta_lookup_unnamed_read/1, meta_lookup_unnamed_write/1, +-export([meta_lookup_unnamed_read/1, meta_lookup_unnamed_write/1, meta_lookup_named_read/1, meta_lookup_named_write/1, meta_newdel_unnamed/1, meta_newdel_named/1]). -export([smp_insert/1, smp_fixed_delete/1, smp_unfix_fix/1, smp_select_delete/1, @@ -95,7 +93,7 @@ rename_do/1, rename_unnamed_do/1, interface_equality_do/1, ordered_match_do/1, ordered_do/1, privacy_do/1, empty_do/1, badinsert_do/1, time_lookup_do/1, lookup_order_do/1, lookup_element_mult_do/1, delete_tab_do/1, delete_elem_do/1, - match_delete_do/1, match_delete3_do/1, firstnext_do/1, + match_delete_do/1, match_delete3_do/1, firstnext_do/1, slot_do/1, match1_do/1, match2_do/1, match_object_do/1, match_object2_do/1, misc1_do/1, safe_fixtable_do/1, info_do/1, dups_do/1, heavy_lookup_do/1, heavy_lookup_element_do/1, member_do/1, otp_5340_do/1, otp_7665_do/1, meta_wb_do/1, @@ -129,7 +127,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, {timetrap,{minutes,5}}]. -all() -> +all() -> [{group, new}, {group, insert}, {group, lookup}, {group, delete}, firstnext, firstnext_concurrent, slot, {group, match}, t_match_spec_run, @@ -161,7 +159,7 @@ all() -> memory_check_summary]. % MUST BE LAST -groups() -> +groups() -> [{new, [], [default, setbag, badnew, verybadnew, named, keypos2, privacy]}, @@ -249,6 +247,7 @@ t_bucket_disappears_do(Opts) -> %% Check ets:match_spec_run/2. t_match_spec_run(Config) when is_list(Config) -> + ct:timetrap({minutes,30}), %% valgrind needs a lot init_externals(), EtsMem = etsmem(), @@ -709,7 +708,7 @@ adjust_xmem([_T1,_T2,_T3,_T4], {A0,B0,C0,D0} = _Mem0) -> {A0+TabDiff, B0+TabDiff, C0+TabDiff, D0+TabDiff}. %% Misc. whitebox tests -t_whitebox(Config) when is_list(Config) -> +t_whitebox(Config) when is_list(Config) -> EtsMem = etsmem(), repeat_for_opts(whitebox_1), repeat_for_opts(whitebox_1), @@ -1050,6 +1049,7 @@ do_reverse_chunked({L,C},Acc) -> %% Test the ets:select_delete/2 and ets:select_count/2 BIFs. t_select_delete(Config) when is_list(Config) -> + ct:timetrap({minutes,30}), %% valgrind needs a lot EtsMem = etsmem(), Tables = fill_sets_int(10000) ++ fill_sets_int(10000,[{write_concurrency,true}]), lists:foreach @@ -1495,15 +1495,15 @@ update_element(Config) when is_list(Config) -> verify_etsmem(EtsMem). update_element_opts(Opts) -> - TupleCases = [{{key,val}, 1 ,2}, - {{val,key}, 2, 1}, - {{key,val}, 1 ,[2]}, + TupleCases = [{{key,val}, 1 ,2}, + {{val,key}, 2, 1}, + {{key,val}, 1 ,[2]}, {{key,val,val}, 1, [2,3]}, {{val,key,val,val}, 2, [3,4,1]}, {{val,val,key,val}, 3, [1,4,1,2]}, % update pos1 twice {{val,val,val,key}, 4, [2,1,2,3]}],% update pos2 twice - lists:foreach(fun({Tuple,KeyPos,UpdPos}) -> update_element_opts(Tuple,KeyPos,UpdPos,Opts) end, + lists:foreach(fun({Tuple,KeyPos,UpdPos}) -> update_element_opts(Tuple,KeyPos,UpdPos,Opts) end, TupleCases), update_element_neg(Opts). @@ -1519,9 +1519,9 @@ update_element_opts(Tuple,KeyPos,UpdPos,Opts) -> true = ets:delete(OrdSet), ok. -update_element(T,Tuple,KeyPos,UpdPos) -> +update_element(T,Tuple,KeyPos,UpdPos) -> KeyList = [17,"seventeen",<<"seventeen">>,{17},list_to_binary(lists:seq(1,100)),make_ref(), self()], - lists:foreach(fun(Key) -> + lists:foreach(fun(Key) -> TupleWithKey = setelement(KeyPos,Tuple,Key), update_element_do(T,TupleWithKey,Key,UpdPos) end, @@ -1556,29 +1556,29 @@ update_element_do(Tab,Tuple,Key,UpdPos) -> {Pos, element(ToIx+1,Values)} % single {pos,value} arg end, - UpdateF = fun(ToIx,Rand) -> - PosValArg = PosValArgF(ToIx,[],UpdPos,Rand,PosValArgF), - %%io:format("update_element(~p)~n",[PosValArg]), - ArgHash = erlang:phash2({Tab,Key,PosValArg}), - true = ets:update_element(Tab, Key, PosValArg), - ArgHash = erlang:phash2({Tab,Key,PosValArg}), - NewTuple = update_tuple(PosValArg,Tuple), - [NewTuple] = ets:lookup(Tab,Key) + UpdateF = fun(ToIx,Rand) -> + PosValArg = PosValArgF(ToIx,[],UpdPos,Rand,PosValArgF), + %%io:format("update_element(~p)~n",[PosValArg]), + ArgHash = erlang:phash2({Tab,Key,PosValArg}), + true = ets:update_element(Tab, Key, PosValArg), + ArgHash = erlang:phash2({Tab,Key,PosValArg}), + NewTuple = update_tuple(PosValArg,Tuple), + [NewTuple] = ets:lookup(Tab,Key) end, - LoopF = fun(_FromIx, Incr, _Times, Checksum, _MeF) when Incr >= Length -> + LoopF = fun(_FromIx, Incr, _Times, Checksum, _MeF) when Incr >= Length -> Checksum; % done - (FromIx, Incr, 0, Checksum, MeF) -> + (FromIx, Incr, 0, Checksum, MeF) -> MeF(FromIx, Incr+1, Length, Checksum, MeF); - (FromIx, Incr, Times, Checksum, MeF) -> + (FromIx, Incr, Times, Checksum, MeF) -> ToIx = (FromIx + Incr) rem Length, UpdateF(ToIx,Checksum), - if + if Incr =:= 0 -> UpdateF(ToIx,Checksum); % extra update to same value true -> true - end, + end, MeF(ToIx, Incr, Times-1, Checksum+ToIx+1, MeF) end, @@ -1622,7 +1622,7 @@ update_element_neg_do(T) -> Object = {key, 0, "Hej"}, true = ets:insert(T,Object), - UpdateF = fun(Arg3) -> + UpdateF = fun(Arg3) -> ArgHash = erlang:phash2({T,key,Arg3}), {'EXIT',{badarg,_}} = (catch ets:update_element(T,key,Arg3)), ArgHash = erlang:phash2({T,key,Arg3}), @@ -1697,7 +1697,7 @@ update_counter_for(T) -> true = ets:lookup(T, b) =:= [setelement(1, NewObj, b)], ets:delete(T, b), Myself(NewObj,Times-1,Arg3,Myself) - end, + end, LoopF = fun(Obj, Times, Arg3) -> %%io:format("Loop start:\nObj = ~p\nArg3=~p\n",[Obj,Arg3]), @@ -1806,7 +1806,7 @@ uc_mimic(Obj, [Pits|Tail], Acc) -> uc_adder(Init, {_Pos, Add}) -> Init + Add; -uc_adder(Init, {_Pos, Add, Thres, Warp}) -> +uc_adder(Init, {_Pos, Add, Thres, Warp}) -> case Init + Add of X when X > Thres, Add > 0 -> Warp; @@ -1838,7 +1838,7 @@ update_counter_neg_for(T) -> Object = {key,0,false,1}, true = ets:insert(T,Object), - UpdateF = fun(Arg3) -> + UpdateF = fun(Arg3) -> ArgHash = erlang:phash2({T,key,Arg3}), {'EXIT',{badarg,_}} = (catch ets:update_counter(T,key,Arg3)), ArgHash = erlang:phash2({T,key,Arg3}), @@ -1978,15 +1978,16 @@ fixtable_next_do(Opts) -> verify_etsmem(EtsMem). do_fixtable_next(Tab) -> - F = fun(X,T,FF) -> case X of - 0 -> true; - _ -> - ets:insert(T, {X, - integer_to_list(X), - X rem 10}), - FF(X-1,T,FF) - end - end, + F = fun(X,T,FF) -> + case X of + 0 -> true; + _ -> + ets:insert(T, {X, + integer_to_list(X), + X rem 10}), + FF(X-1,T,FF) + end + end, F(100,Tab,F), ets:safe_fixtable(Tab,true), First = ets:first(Tab), @@ -2001,7 +2002,7 @@ do_fixtable_next(Tab) -> %% Check inserts of deleted keys in fixed bags. fixtable_insert(Config) when is_list(Config) -> - Combos = [[Type,{write_concurrency,WC}] || Type<- [bag,duplicate_bag], + Combos = [[Type,{write_concurrency,WC}] || Type<- [bag,duplicate_bag], WC <- [false,true]], lists:foreach(fun(Opts) -> fixtable_insert_do(Opts) end, Combos), @@ -2117,7 +2118,7 @@ heir_do(Opts) -> %% Different types of heir data and link/monitor relations TestFun = fun(Arg) -> {EtsMem,Arg} end, - Combos = [{Data,Mode} || Data<-[foo_data, <<"binary">>, + Combos = [{Data,Mode} || Data<-[foo_data, <<"binary">>, lists:seq(1,10), {17,TestFun,self()}, "The busy heir"], Mode<-[none,link,monitor]], @@ -2157,7 +2158,7 @@ heir_do(Opts) -> Founder4 ! {go, Heir4}, {'DOWN', MrefH4, process, Heir4, normal} = receive_any(), erts_debug:set_internal_state(next_pid, NextPidIx), - DoppelGanger = spawn_monitor_with_pid(Heir4, + DoppelGanger = spawn_monitor_with_pid(Heir4, fun()-> die_please = receive_any() end), Founder4 ! die_please, {'DOWN', MrefF4, process, Founder4, normal} = receive_any(), @@ -2170,12 +2171,12 @@ heir_do(Opts) -> failed -> io:format("Failed to spawn process with pid ~p\n", [Heir4]), true % try again - end + end end), verify_etsmem(EtsMem). -heir_founder(Master, HeirData, Opts) -> +heir_founder(Master, HeirData, Opts) -> {go,Heir} = receive_any(), HeirTpl = case Heir of none -> {heir,none}; @@ -2248,7 +2249,7 @@ heir_1(HeirData,Mode,Opts) -> {'DOWN', Mref, process, Heir, normal} = receive_any(). %% Test ets:give_way/3. -give_away(Config) when is_list(Config) -> +give_away(Config) when is_list(Config) -> repeat_for_opts(give_away_do). give_away_do(Opts) -> @@ -2387,7 +2388,7 @@ bad_table(Config) when is_list(Config) -> ok. bad_table_do(Opts, DummyFile) -> - Parent = self(), + Parent = self(), {Pid,Mref} = my_spawn_opt(fun()-> ets_new(priv,[private,named_table | Opts]), Priv = ets_new(priv,[private | Opts]), ets_new(prot,[protected,named_table | Opts]), @@ -2442,7 +2443,7 @@ bad_table_do(Opts, DummyFile) -> ], Info = {Opts, Priv, Prot}, lists:foreach(fun(Op) -> bad_table_op(Info, Op) end, - OpList), + OpList), Pid ! die_please, {'DOWN', Mref, process, Pid, normal} = receive_any(), ok. @@ -2577,14 +2578,14 @@ interface_equality_do(Opts) -> Set = ets_new(set,[set | Opts]), OrderedSet = ets_new(ordered_set,[ordered_set | Opts]), F = fun(X,T,FF) -> case X of - 0 -> true; - _ -> - ets:insert(T, {X, - integer_to_list(X), - X rem 10}), - FF(X-1,T,FF) - end - end, + 0 -> true; + _ -> + ets:insert(T, {X, + integer_to_list(X), + X rem 10}), + FF(X-1,T,FF) + end + end, F(100,Set,F), F(100,OrderedSet,F), equal_results(ets, insert, Set, OrderedSet, [{a,"a"}]), @@ -2653,20 +2654,20 @@ ordered_match_do(Opts) -> F(3000,T1,F), [[3,3],[3,3],[3,3]] = ets:match(T1, {'_','_','$1','$2',3}), F2 = fun(X,Rem,Res,FF) -> case X of - 0 -> []; - _ -> + 0 -> []; + _ -> case X rem Rem of Res -> FF(X-1,Rem,Res,FF) ++ [{X, - integer_to_list(X), + integer_to_list(X), X rem 10, X rem 100, X rem 1000}]; _ -> FF(X-1,Rem,Res,FF) end - end + end end, OL1 = F2(3000,100,2,F2), OL1 = ets:match_object(T1, {'_','_','_',2,'_'}), @@ -2744,7 +2745,7 @@ pick_all_backwards(T) -> %% Small test case for both set and bag type ets tables. -setbag(Config) when is_list(Config) -> +setbag(Config) when is_list(Config) -> EtsMem = etsmem(), Set = ets_new(set,[set]), Bag = ets_new(bag,[bag]), @@ -2821,7 +2822,7 @@ privacy_do(Opts) -> privacy_check(pub,prot,priv), - Owner ! {shift,1,{pub,prot,priv}}, + Owner ! {shift,1,{pub,prot,priv}}, receive {Pub1,Prot1,Priv1} -> ok = privacy_check(Pub1,Prot1,Priv1), @@ -2960,7 +2961,7 @@ badlookup(Config) when is_list(Config) -> verify_etsmem(EtsMem). %% Test that lookup returns objects in order of insertion for bag and dbag. -lookup_order(Config) when is_list(Config) -> +lookup_order(Config) when is_list(Config) -> EtsMem = etsmem(), repeat_for_opts(lookup_order_do, [write_concurrency,[bag,duplicate_bag]]), verify_etsmem(EtsMem), @@ -2982,7 +2983,7 @@ lookup_order_2(Opts, Fixed) -> case Fixed of true -> ets:safe_fixtable(T,true); false -> ok - end, + end, S10 = {T,[],key}, S20 = check_insert(S10,A), S30 = check_insert(S20,B), @@ -2994,7 +2995,7 @@ lookup_order_2(Opts, Fixed) -> S80 = check_delete(S70,D2b), S90 = check_insert(S80,D2a), SA0 = check_delete(S90,D3a), - SB0 = check_delete(SA0,D3b), + SB0 = check_delete(SA0,D3b), check_insert_new(SB0,D3b), true = ets:delete(T) @@ -3007,7 +3008,7 @@ check_insert({T,List0,Key},Val) -> ets:insert(T,{Key,Val}), List1 = case (ets:info(T,type) =:= bag andalso lists:member({Key,Val},List0)) of - true -> List0; + true -> List0; false -> [{Key,Val} | List0] end, check_check({T,List1,Key}). @@ -3040,8 +3041,6 @@ check_check(S={T,List,Key}) -> Items = length(List), S. - - fill_tab(Tab,Val) -> ets:insert(Tab,{key,Val}), ets:insert(Tab,{{a,144},Val}), @@ -3069,13 +3068,11 @@ lookup_element_mult_do(Opts) -> verify_etsmem(EtsMem). lem_data() -> - [ - {service,'eddie2@boromir',{150,236,14,103},httpd88,self()}, + [{service,'eddie2@boromir',{150,236,14,103},httpd88,self()}, {service,'eddie2@boromir',{150,236,14,103},httpd80,self()}, {service,'eddie3@boromir',{150,236,14,107},httpd88,self()}, {service,'eddie3@boromir',{150,236,14,107},httpd80,self()}, - {service,'eddie4@boromir',{150,236,14,108},httpd88,self()} - ]. + {service,'eddie4@boromir',{150,236,14,108},httpd88,self()}]. lem_crash(T) -> L = ets:lookup_element(T, 'eddie2@boromir', 3), @@ -3126,6 +3123,7 @@ delete_tab_do(Opts) -> %% Check that ets:delete/1 works and that other processes can run. delete_large_tab(Config) when is_list(Config) -> + ct:timetrap({minutes,30}), %% valgrind needs a lot Data = [{erlang:phash2(I, 16#ffffff),I} || I <- lists:seq(1, 200000)], EtsMem = etsmem(), repeat_for_opts(fun(Opts) -> delete_large_tab_do(Opts,Data) end), @@ -3148,7 +3146,7 @@ delete_large_tab_1(Name, Flags, Data, Fix) -> lists:foreach(fun({K,_}) -> ets:delete(Tab, K) end, Data) end, - {priority, Prio} = process_info(self(), priority), + {priority, Prio} = process_info(self(), priority), Deleter = self(), [SchedTracer] = start_loopers(1, @@ -3195,7 +3193,7 @@ delete_large_tab_1(Name, Flags, Data, Fix) -> %% Delete a large name table and try to create a new table with %% the same name in another process. -delete_large_named_table(Config) when is_list(Config) -> +delete_large_named_table(Config) when is_list(Config) -> Data = [{erlang:phash2(I, 16#ffffff),I} || I <- lists:seq(1, 200000)], EtsMem = etsmem(), repeat_for_opts(fun(Opts) -> delete_large_named_table_do(Opts,Data) end), @@ -3566,7 +3564,7 @@ dyn_lookup(T) -> dyn_lookup(T, ets:first(T)). dyn_lookup(_T, '$end_of_table') -> []; dyn_lookup(T, K) -> - NextKey=ets:next(T,K), + NextKey = ets:next(T,K), case ets:next(T,K) of NextKey -> dyn_lookup(T, NextKey); @@ -4085,9 +4083,9 @@ tabfile_ext2_do(Opts,Config) -> Name = make_ref(), [ets:insert(T,{X,integer_to_list(X)}) || X <- L], ok = ets:tab2file(T,FName,[{extended_info,[md5sum]}]), - true = lists:sort(ets:tab2list(T)) =:= + true = lists:sort(ets:tab2list(T)) =:= lists:sort(ets:tab2list(element(2,ets:file2tab(FName)))), - true = lists:sort(ets:tab2list(T)) =:= + true = lists:sort(ets:tab2list(T)) =:= lists:sort(ets:tab2list( element(2,ets:file2tab(FName,[{verify,true}])))), {ok, Name} = disk_log:open([{name,Name},{file,FName}]), @@ -4102,9 +4100,9 @@ tabfile_ext2_do(Opts,Config) -> ets:tab2list( element(2,ets:file2tab(FName2)))), {error,checksum_error} = ets:file2tab(FName2,[{verify,true}]), - {value,{extended_info,[md5sum]}} = + {value,{extended_info,[md5sum]}} = lists:keysearch(extended_info,1,element(2,ets:tabfile_info(FName2))), - {value,{extended_info,[md5sum]}} = + {value,{extended_info,[md5sum]}} = lists:keysearch(extended_info,1,element(2,ets:tabfile_info(FName))), file:delete(FName), file:delete(FName2), @@ -4149,15 +4147,14 @@ tabfile_ext4(Config) when is_list(Config) -> Name2 = make_ref(), [ets:insert(TL,{X,integer_to_list(X)}) || X <- LL], ok = ets:tab2file(TL,FName,[{extended_info,[md5sum]}]), - {ok, Name2} = disk_log:open([{name, Name2}, {file, FName}, + {ok, Name2} = disk_log:open([{name, Name2}, {file, FName}, {mode, read_only}]), {C,[_|_]} = disk_log:chunk(Name2,start), {_,[_|_]} = disk_log:chunk(Name2,C), disk_log:close(Name2), - true = lists:sort(ets:tab2list(TL)) =:= + true = lists:sort(ets:tab2list(TL)) =:= lists:sort(ets:tab2list(element(2,ets:file2tab(FName)))), - Res = [ - begin + Res = [begin {ok,FD} = file:open(FName,[binary,read,write]), {ok, Bin} = file:pread(FD,0,1000), <<B1:N/binary,Ch:8,B2/binary>> = Bin, @@ -4167,7 +4164,7 @@ tabfile_ext4(Config) when is_list(Config) -> ok = file:close(FD), X = case ets:file2tab(FName) of {ok,TL2} -> - true = lists:sort(ets:tab2list(TL)) =/= + true = lists:sort(ets:tab2list(TL)) =/= lists:sort(ets:tab2list(TL2)); _ -> totally_broken @@ -4175,7 +4172,7 @@ tabfile_ext4(Config) when is_list(Config) -> {error,Y} = ets:file2tab(FName,[{verify,true}]), ets:tab2file(TL,FName,[{extended_info,[md5sum]}]), {X,Y} - end || N <- lists:seq(500,600) ], + end || N <- lists:seq(500,600)], io:format("~p~n",[Res]), file:delete(FName), ok. @@ -4404,16 +4401,14 @@ member_do(Opts) -> build_table(L1,L2,Num) -> - T = ets_new(xxx, [ordered_set] - ), + T = ets_new(xxx, [ordered_set]), lists:foreach( fun(X1) -> lists:foreach( fun(X2) -> F = fun(FF,N) -> - ets:insert(T,{{X1,X2,N}, - X1, X2, N}), - case N of + ets:insert(T,{{X1,X2,N}, X1, X2, N}), + case N of 0 -> ok; _ -> @@ -4426,16 +4421,14 @@ build_table(L1,L2,Num) -> T. build_table2(L1,L2,Num) -> - T = ets_new(xxx, [ordered_set] - ), + T = ets_new(xxx, [ordered_set]), lists:foreach( fun(X1) -> lists:foreach( fun(X2) -> F = fun(FF,N) -> - ets:insert(T,{{N,X1,X2}, - N, X1, X2}), - case N of + ets:insert(T,{{N,X1,X2}, N, X1, X2}), + case N of 0 -> ok; _ -> @@ -4726,7 +4719,7 @@ del_one_by_one_dbag_3(T,From,To) -> N = (ets:info(T,size) + 1), Obj2 = {From, integer_to_list(From)}, ets:delete_object(T,Obj2), - N = (ets:info(T,size) + 2) + N = (ets:info(T,size) + 2) end, Next = if From < To -> @@ -4773,14 +4766,14 @@ gen_dets_filename(Config,N) -> filename:join(proplists:get_value(priv_dir,Config), "testdets_" ++ integer_to_list(N) ++ ".dets"). -otp_6842_select_1000(Config) when is_list(Config) -> +otp_6842_select_1000(Config) when is_list(Config) -> Tab = ets_new(xxx,[ordered_set]), [ets:insert(Tab,{X,X}) || X <- lists:seq(1,10000)], AllTrue = lists:duplicate(10,true), AllTrue = [ length( element(1, - ets:select(Tab,[{'_',[],['$_']}],X*1000))) =:= + ets:select(Tab,[{'_',[],['$_']}],X*1000))) =:= X*1000 || X <- lists:seq(1,10) ], Sequences = [[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000], [2000,2000,2000,2000,2000], @@ -4806,7 +4799,13 @@ check_seq(A,B,C) -> false. otp_6338(Config) when is_list(Config) -> - L = binary_to_term(<<131,108,0,0,0,2,104,2,108,0,0,0,2,103,100,0,19,112,112,98,49,95,98,115,49,50,64,98,108,97,100,101,95,48,95,53,0,0,33,50,0,0,0,4,1,98,0,0,23,226,106,100,0,4,101,120,105,116,104,2,108,0,0,0,2,104,2,100,0,3,115,98,109,100,0,19,112,112,98,50,95,98,115,49,50,64,98,108,97,100,101,95,48,95,56,98,0,0,18,231,106,100,0,4,114,101,99,118,106>>), + L = binary_to_term(<<131,108,0,0,0,2,104,2,108,0,0,0,2,103,100,0,19,112,112, + 98,49,95,98,115,49,50,64,98,108,97,100,101,95,48,95,53, + 0,0,33,50,0,0,0,4,1,98,0,0,23,226,106,100,0,4,101,120, + 105,116,104,2,108,0,0,0,2,104,2,100,0,3,115,98,109,100, + 0,19,112,112,98,50,95,98,115,49,50,64,98,108,97,100, + 101,95,48,95,56,98,0,0,18,231,106,100,0,4,114,101,99, + 118,106>>), T = ets_new(xxx,[ordered_set]), lists:foreach(fun(X) -> ets:insert(T,X) end,L), [[4839,recv]] = ets:match(T,{[{sbm,ppb2_bs12@blade_0_8},'$1'],'$2'}), @@ -4825,7 +4824,7 @@ otp_5340_do(Opts) -> ets:delete(T). w(_,0, _) -> ok; -w(T,N, Id) -> +w(T,N, Id) -> ets:insert(T, {N, Id}), w(T,N-1,Id). @@ -4915,7 +4914,7 @@ meta_wb_new(Name, _, Tabs, Opts) -> case (catch ets_new(Name,[named_table|Opts])) of Name -> false = lists:member(Name, Tabs), - [Name | Tabs]; + [Name | Tabs]; {'EXIT',{badarg,_}} -> true = lists:member(Name, Tabs), Tabs @@ -5090,7 +5089,7 @@ meta_lookup_unnamed_read(Config) when is_list(Config) -> Tab end, ExecF = fun(Tab) -> [{key,data}] = ets:lookup(Tab,key), - Tab + Tab end, FiniF = fun(Tab) -> true = ets:delete(Tab) end, @@ -5114,7 +5113,7 @@ meta_lookup_named_read(Config) when is_list(Config) -> Tab end, ExecF = fun(Tab) -> [{key,data}] = ets:lookup(Tab,key), - Tab + Tab end, FiniF = fun(Tab) -> true = ets:delete(Tab) end, @@ -5173,9 +5172,9 @@ smp_fixed_delete_do() -> ets:safe_fixtable(T,true), Buckets = num_of_buckets(T), InitF = fun([ProcN,NumOfProcs|_]) -> {ProcN,NumOfProcs} end, - ExecF = fun({Key,_}) when Key > NumOfObjs -> + ExecF = fun({Key,_}) when Key > NumOfObjs -> [end_of_work]; - ({Key,Increment}) -> + ({Key,Increment}) -> true = ets:delete(T,Key), {Key+Increment,Increment} end, @@ -5204,7 +5203,7 @@ smp_unfix_fix_do() -> T = ets_new(foo,[public,{write_concurrency,true}]), %%Mem = ets:info(T,memory), NumOfObjs = 100000, - Deleted = 50000, + Deleted = 50000, filltabint(T,NumOfObjs), ets:safe_fixtable(T,true), Buckets = num_of_buckets(T), @@ -5217,7 +5216,7 @@ smp_unfix_fix_do() -> true = ets:info(T,fixed), Deleted = get_kept_objects(T), - {Child, Mref} = + {Child, Mref} = my_spawn_opt( fun()-> true = ets:info(T,fixed), @@ -5276,22 +5275,19 @@ otp_8166_do(WC) -> NumOfObjs = 3000, %% Need more than 1000 live objects for match_object to trap one time Deleted = NumOfObjs div 2, filltabint(T,NumOfObjs), - {ReaderPid, ReaderMref} = - my_spawn_opt(fun()-> otp_8166_reader(T,NumOfObjs) end, - [link, monitor, {scheduler,2}]), - {ZombieCrPid, ZombieCrMref} = - my_spawn_opt(fun()-> otp_8166_zombie_creator(T,Deleted) end, - [link, monitor, {scheduler,3}]), + {ReaderPid, ReaderMref} = my_spawn_opt(fun()-> otp_8166_reader(T,NumOfObjs) end, + [link, monitor, {scheduler,2}]), + {ZombieCrPid, ZombieCrMref} = my_spawn_opt(fun()-> otp_8166_zombie_creator(T,Deleted) end, + [link, monitor, {scheduler,3}]), repeat(fun() -> ZombieCrPid ! {loop, self()}, zombies_created = receive_any(), otp_8166_trapper(T, 10, ZombieCrPid) - end, - 100), + end, 100), ReaderPid ! quit, {'DOWN', ReaderMref, process, ReaderPid, normal} = receive_any(), - ZombieCrPid ! quit, + ZombieCrPid ! quit, {'DOWN', ZombieCrMref, process, ZombieCrPid, normal} = receive_any(), false = ets:info(T,fixed), 0 = get_kept_objects(T), @@ -5301,7 +5297,7 @@ otp_8166_do(WC) -> %% Keep reading the table otp_8166_reader(T, NumOfObjs) -> - repeat_while(fun(0) -> + repeat_while(fun(0) -> receive quit -> {false,done} after 0 -> {true,NumOfObjs} end; @@ -5315,14 +5311,14 @@ otp_8166_reader(T, NumOfObjs) -> otp_8166_trapper(T, Try, ZombieCrPid) -> [] = ets:match_object(T,{'_',"Pink Unicorn"}), case {ets:info(T,fixed),Try} of - {true,1} -> + {true,1} -> io:format("failed to provoke unsafe unfix, give up...\n",[]), ZombieCrPid ! unfix; - {true,_} -> + {true,_} -> io:format("trapper too fast, trying again...\n",[]), otp_8166_trapper(T, Try-1, ZombieCrPid); {false,_} -> done - end. + end. %% Fixate table and create some pseudo-deleted objects (zombies) @@ -5342,7 +5338,7 @@ otp_8166_zombie_creator(T,Deleted) -> repeat_while(fun() -> case ets:info(T,safe_fixed_monotonic_time) of {_,[_P1,_P2]} -> false; - _ -> + _ -> receive unfix -> false after 0 -> true end @@ -5399,7 +5395,7 @@ smp_select_delete(Config) when is_list(Config) -> Mod = 17, Zeros = erlang:make_tuple(Mod,0), InitF = fun(_) -> Zeros end, - ExecF = fun(Diffs0) -> + ExecF = fun(Diffs0) -> case rand:uniform(20) of 1 -> Mod = 17, @@ -5421,7 +5417,7 @@ smp_select_delete(Config) when is_list(Config) -> Diffs1; false -> Diffs0 end - end + end end, FiniF = fun(Result) -> Result end, Results = run_workers_do(InitF,ExecF,FiniF,20000), @@ -5432,7 +5428,7 @@ smp_select_delete(Config) when is_list(Config) -> 0, TotCnts), io:format("LeftInTab = ~p\n",[LeftInTab]), LeftInTab = ets:info(T,size), - lists:foldl(fun(Cnt,Eq) -> + lists:foldl(fun(Cnt,Eq) -> WasCnt = ets:select_count(T, [{{'_', '$1'}, [{'=:=', {'rem', '$1', Mod}, Eq}], @@ -5440,7 +5436,7 @@ smp_select_delete(Config) when is_list(Config) -> io:format("~p: ~p =?= ~p\n",[Eq,Cnt,WasCnt]), Cnt = WasCnt, Eq+1 - end, + end, 0, TotCnts), verify_table_load(T), LeftInTab = ets:select_delete(T, [{{'$1','$1'}, [], [true]}]), @@ -5478,8 +5474,8 @@ types_do(Opts) -> %% OTP-9932: Memory overwrite when inserting large integers in compressed bag. %% Will crash with segv on 64-bit opt if not fixed. otp_9932(Config) when is_list(Config) -> - T = ets:new(xxx, [bag, compressed]), - Fun = fun(N) -> + T = ets:new(xxx, [bag, compressed]), + Fun = fun(N) -> Key = {1316110174588445 bsl N,1316110174588583 bsl N}, S = {Key, Key}, true = ets:insert(T, S), @@ -5495,9 +5491,9 @@ otp_9932(Config) when is_list(Config) -> %% write_concurrency table. otp_9423(Config) when is_list(Config) -> InitF = fun(_) -> {0,0} end, - ExecF = fun({S,F}) -> - receive - stop -> + ExecF = fun({S,F}) -> + receive + stop -> io:format("~p got stop\n", [self()]), [end_of_work | {"Succeded=",S,"Failed=",F}] after 0 -> @@ -5593,12 +5589,12 @@ take(Config) when is_list(Config) -> %% Utility functions: %% -add_lists(L1,L2) -> +add_lists(L1,L2) -> add_lists(L1,L2,[]). add_lists([],[],Acc) -> lists:reverse(Acc); add_lists([E1|T1], [E2|T2], Acc) -> - add_lists(T1, T2, [E1+E2 | Acc]). + add_lists(T1, T2, [E1+E2 | Acc]). run_workers(InitF,ExecF,FiniF,Laps) -> run_workers(InitF,ExecF,FiniF,Laps, 0). @@ -5644,9 +5640,9 @@ worker_loop(infinite, ExecF, State) -> worker_loop(N, ExecF, State) -> worker_loop(N-1,ExecF,ExecF(State)). -wait_pids(Pids) -> +wait_pids(Pids) -> wait_pids(Pids,[]). -wait_pids([],Acc) -> +wait_pids([],Acc) -> Acc; wait_pids(Pids, Acc) -> receive @@ -5683,7 +5679,7 @@ etsmem() -> wait_for_memory_deallocations(), AllTabs = lists:map(fun(T) -> {T,ets:info(T,name),ets:info(T,size), - ets:info(T,memory),ets:info(T,type)} + ets:info(T,memory),ets:info(T,type)} end, ets:all()), EtsAllocInfo = erlang:system_info({allocator,ets_alloc}), @@ -5895,7 +5891,7 @@ receive_any() -> receive_any_spinning() -> receive_any_spinning(1000000). receive_any_spinning(Loops) -> - receive_any_spinning(Loops,Loops,1). + receive_any_spinning(Loops,Loops,1). receive_any_spinning(Loops,0,Tries) -> receive M -> io:format("Spinning process ~p got msg ~p after ~p tries\n", [self(),M,Tries]), diff --git a/lib/stdlib/test/gen_statem_SUITE.erl b/lib/stdlib/test/gen_statem_SUITE.erl index 28f9ab81fe..119546be98 100644 --- a/lib/stdlib/test/gen_statem_SUITE.erl +++ b/lib/stdlib/test/gen_statem_SUITE.erl @@ -742,26 +742,40 @@ state_timeout(_Config) -> %% Verify that {state_timeout,0,_} %% comes after next_event and that %% {timeout,0,_} is cancelled by - %% {state_timeout,0,_} + %% pending {state_timeout,0,_} {keep_state, {ok,2,Data}, [{timeout,0,3}]}; - (state_timeout, 2, {ok,2,{Time,From}}) -> - {next_state, state3, 3, + (state_timeout, 2, {ok,2,Data}) -> + %% Verify that timeout 0's are processed + %% in order + {keep_state, {ok,3,Data}, + [{timeout,0,4},{state_timeout,0,5}]}; + (timeout, 4, {ok,3,Data}) -> + %% Verify that timeout 0 is cancelled by + %% enqueued state_timeout 0 and that + %% multiple state_timeout 0 can be enqueued + {keep_state, {ok,4,Data}, + [{state_timeout,0,6},{timeout,0,7}]}; + (state_timeout, 5, {ok,4,Data}) -> + {keep_state, {ok,5,Data}}; + (state_timeout, 6, {ok,5,{Time,From}}) -> + {next_state, state3, 6, [{reply,From,ok}, - {state_timeout,Time,3}]} + {state_timeout,Time,8}]} end, state3 => fun - (info, message_to_self, 3) -> - {keep_state, '3'}; - ({call,From}, check, '3') -> + (info, message_to_self, 6) -> + {keep_state, 7}; + ({call,From}, check, 7) -> {keep_state, From}; - (state_timeout, 3, From) -> + (state_timeout, 8, From) -> {stop_and_reply, normal, {reply,From,ok}} end}, {ok,STM} = gen_statem:start_link(?MODULE, {map_statem,Machine,[]}, []), + sys:trace(STM, true), TRef = erlang:start_timer(1000, self(), kull), ok = gen_statem:call(STM, {go,500}), ok = gen_statem:call(STM, check), diff --git a/lib/stdlib/test/rand_SUITE.erl b/lib/stdlib/test/rand_SUITE.erl index cb778c96d4..02b7cb10c2 100644 --- a/lib/stdlib/test/rand_SUITE.erl +++ b/lib/stdlib/test/rand_SUITE.erl @@ -18,7 +18,11 @@ %% %CopyrightEnd% -module(rand_SUITE). --export([all/0, suite/0,groups/0]). +-compile({nowarn_deprecated_function,[{random,seed,1}, + {random,uniform_s,1}, + {random,uniform_s,2}]}). + +-export([all/0, suite/0, groups/0, group/1]). -export([interval_int/1, interval_float/1, seed/1, api_eq/1, reference/1, @@ -47,18 +51,22 @@ groups() -> [{basic_stats, [parallel], [basic_stats_uniform_1, basic_stats_uniform_2, basic_stats_normal]}]. +group(basic_stats) -> + %% valgrind needs a lot of time + [{timetrap,{minutes,10}}]. + %% A simple helper to test without test_server during dev test() -> Tests = all(), lists:foreach(fun(Test) -> - try - ok = ?MODULE:Test([]), - io:format("~p: ok~n", [Test]) - catch _:Reason -> - io:format("Failed: ~p: ~p ~p~n", - [Test, Reason, erlang:get_stacktrace()]) - end - end, Tests). + try + ok = ?MODULE:Test([]), + io:format("~p: ok~n", [Test]) + catch _:Reason -> + io:format("Failed: ~p: ~p ~p~n", + [Test, Reason, erlang:get_stacktrace()]) + end + end, Tests). algs() -> [exs64, exsplus, exs1024]. diff --git a/lib/tools/emacs/Makefile b/lib/tools/emacs/Makefile index e1b195ef97..35c93ba4ed 100644 --- a/lib/tools/emacs/Makefile +++ b/lib/tools/emacs/Makefile @@ -38,6 +38,7 @@ MAN_FILES= \ tags.3 EMACS_FILES= \ + erldoc \ erlang-skels \ erlang-skels-old \ erlang_appwiz \ diff --git a/lib/tools/emacs/erlang-start.el b/lib/tools/emacs/erlang-start.el index f9a6d24b2c..160057e179 100644 --- a/lib/tools/emacs/erlang-start.el +++ b/lib/tools/emacs/erlang-start.el @@ -78,9 +78,23 @@ (autoload 'erlang-find-tag-other-window "erlang" "Like `find-tag-other-window'. Capable of retreiving Erlang modules.") +;; +;; Declare functions in "erlang-edoc.el". +;; + (autoload 'erlang-edoc-mode "erlang-edoc" "Toggle Erlang-Edoc mode on or off." t) ;; +;; Declare functions in "erldoc.el". +;; + +(autoload 'erldoc-browse "erldoc" "\n\n(fn MFA)" t nil) +(autoload 'erldoc-browse-topic "erldoc" "\n\n(fn TOPIC)" t nil) +(autoload 'erldoc-apropos "erldoc" "\n\n(fn PATTERN)" t nil) +(autoload 'erldoc-eldoc-function "erldoc" "\ +A function suitable for `eldoc-documentation-function'.\n\n(fn)" nil nil) + +;; ;; Associate files extensions ".erl" and ".hrl" with Erlang mode. ;; diff --git a/lib/tools/emacs/erlang.el b/lib/tools/emacs/erlang.el index cc22903e7f..40f0bb7f80 100644 --- a/lib/tools/emacs/erlang.el +++ b/lib/tools/emacs/erlang.el @@ -1440,6 +1440,11 @@ Other commands: (erlang-skel-init) (when (fboundp 'tempo-use-tag-list) (tempo-use-tag-list 'erlang-tempo-tags)) + (when (and (fboundp 'add-function) (fboundp 'erldoc-eldoc-function)) + (or eldoc-documentation-function + (setq-local eldoc-documentation-function #'ignore)) + (add-function :before-until (local 'eldoc-documentation-function) + #'erldoc-eldoc-function)) (run-hooks 'erlang-mode-hook) (if (zerop (buffer-size)) (run-hooks 'erlang-new-file-hook))) diff --git a/lib/tools/emacs/erldoc.el b/lib/tools/emacs/erldoc.el new file mode 100644 index 0000000000..cb355374d9 --- /dev/null +++ b/lib/tools/emacs/erldoc.el @@ -0,0 +1,508 @@ +;;; erldoc.el --- browse Erlang/OTP documentation -*- lexical-binding: t; -*- + +;; %CopyrightBegin% +;; +;; Copyright Ericsson AB 2016. All Rights Reserved. +;; +;; Licensed under the Apache License, Version 2.0 (the "License"); +;; you may not use this file except in compliance with the License. +;; You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; +;; %CopyrightEnd% + +;;; Commentary: + +;; Crawl Erlang/OTP HTML documentation and generate lookup tables. +;; +;; This package depends on `cl-lib', `pcase' and +;; `libxml-parse-html-region'; emacs 24+ compiled with libxml2 should +;; work. On emacs 24.1 and 24.2 do `M-x package-install RET cl-lib +;; RET' to install `cl-lib'. +;; +;; Please customise `erldoc-man-index' to point to your local OTP +;; documentation. +;; +;; To use: +;; +;; (define-key help-map "u" 'erldoc-browse) +;; (define-key help-map "t" 'erldoc-browse-topic) +;; (define-key help-map "a" 'erldoc-apropos) +;; +;; Note: these commands trigger indexing OTP documentation on first +;; run with cache to disk which may take 1-2 minutes. + + +;;; Examples: + +;; 1. `M-x erldoc-browse RET erlang:integer_to_binary/2 RET' opens the +;; `erlang' manual anchored on the entry for `integer_to_binary/2'. +;; +;; 2. `M-x erldoc-apropos RET first RET' list all MFAs matching +;; substring `first'. +;; +;; 3. `M-x erldoc-browse-topic RET efficiency_guide#Introduction RET' +;; opens chapter `Introduction' of the `Efficiency Guide' in the +;; browser. + +;;; History: + +;; Written in December 2013 as a temporary solution to help me browse +;; the rich Erlang/OTP documentation. Three years on I find myself +;; still using it every day. - Leo (2016) + +;;; Code: + +(eval-when-compile (require 'url-parse)) +(require 'cl-lib) +(require 'erlang) + +(eval-and-compile ;for emacs < 24.3 + (or (fboundp 'user-error) (defalias 'user-error 'error))) + +(defgroup erldoc nil + "Browse Erlang document." + :group 'help) + +(defcustom erldoc-man-index "http://www.erlang.org/doc/man_index.html" + "The URL to the man_index.html page. +Note it is advisable to customise this to a local URL for example +`file:///usr/local/19.1/lib/erlang/doc/man_index.html' to speed +up the indexing." + :type 'string + :group 'erldoc) + +(defcustom erldoc-verify-man-path nil + "If non-nil verify man path existence for `file://'." + :type 'boolean + :group 'erldoc) + +(defcustom erldoc-output-file (locate-user-emacs-file "cache/erldoc") + "File to store the parsed results." + :type 'file + :group 'erldoc) + +(defun erldoc-strip-string (s) + (let* ((re "[ \t\n\r\f\v\u00a0]+") + (from (if (string-match (concat "\\`" re) s) (match-end 0) 0)) + (to (and (string-match (concat re "\\'") s) (match-beginning 0)))) + (substring s from (and to (max to from))))) + +;; Note: don't know how to get the BASE-URL to +;; `libxml-parse-html-region' to work. +(defun erldoc-expand-url (url base-url) + (if (url-type (url-generic-parse-url url)) + url + (let* ((base (url-generic-parse-url base-url)) + (dir (directory-file-name (file-name-directory (url-filename base))))) + (setf (url-filename base) (expand-file-name url dir)) + (url-recreate-url base)))) + +(defun erldoc-parse-html (url) + (with-temp-buffer + (url-insert-file-contents url) + (libxml-parse-html-region (point-min) (point-max)))) + +(defalias 'erldoc-dom-text-node-p #'stringp) + +(defun erldoc-dom-attributes (dom) + (and (not (erldoc-dom-text-node-p dom)) (cadr dom))) + +(defun erldoc-dom-get-attribute (dom attrib-name) + (cdr (assq attrib-name (erldoc-dom-attributes dom)))) + +(defun erldoc-dom-children (dom) + (and (not (erldoc-dom-text-node-p dom)) (cddr dom))) + +(defun erldoc-dom-get-text (dom) + (let ((text (car (last (erldoc-dom-children dom))))) + (and (erldoc-dom-text-node-p text) text))) + +(defvar erldoc-dom-walk-parent nil) +(defvar erldoc-dom-walk-siblings nil) + +(defun erldoc-dom-walk (dom k) + (funcall k dom) + (let ((erldoc-dom-walk-parent dom) + (erldoc-dom-walk-siblings (unless (erldoc-dom-text-node-p dom) + (cddr dom)))) + (dolist (child erldoc-dom-walk-siblings) + (erldoc-dom-walk child k)))) + +(defun erldoc-dom-get-element (dom element-name) + (catch 'return + (erldoc-dom-walk dom (lambda (d) + (when (eq (car-safe d) element-name) + (throw 'return d)))))) + +(defun erldoc-dom-get-element-by-id (dom id) + (catch 'return + (erldoc-dom-walk dom (lambda (d) + (when (equal (erldoc-dom-get-attribute d 'id) id) + (throw 'return d)))))) + +(defun erldoc-dom-get-elements-by-id (dom id) + (let (result) + (erldoc-dom-walk dom (lambda (d) + (when (equal (erldoc-dom-get-attribute d 'id) id) + (push d result)))) + (nreverse result))) + +(defun erldoc-fix-path (url) + (if (and erldoc-verify-man-path + ;; Could only verify local files + (equal (url-type (url-generic-parse-url url)) "file")) + (let* ((obj (url-generic-parse-url url)) + (new (car (file-expand-wildcards + (replace-regexp-in-string + "-[0-9]+\\(?:[.][0-9]+\\)*" "*" + (url-filename obj)))))) + (or new (error "File %s does not exist" (url-filename obj))) + (setf (url-filename obj) new) + (url-recreate-url obj)) + url)) + +(defun erldoc-parse-man-index (url) + (let ((table (erldoc-dom-get-element (erldoc-parse-html url) 'table)) + (mans)) + (erldoc-dom-walk + table + (lambda (d) + (when (eq (car-safe d) 'a) + (let ((href (erldoc-dom-get-attribute d 'href))) + (when (and href (not (string-match-p "index\\.html\\'" href))) + (with-demoted-errors "erldoc-parse-man-index: %S" + (push (cons (erldoc-dom-get-text d) + (erldoc-fix-path (erldoc-expand-url href url))) + mans))))))) + (nreverse mans))) + +(defun erldoc-parse-man (man) + (let ((dom (erldoc-parse-html (cdr man))) + (table (make-hash-table :test #'equal))) + (erldoc-dom-walk + (erldoc-dom-get-element-by-id dom "loadscrollpos") + (lambda (d) + (let ((href (erldoc-dom-get-attribute d 'href))) + (when (and href (string-match "#" href)) + (puthash (substring href (match-end 0)) + (list (concat (car man) ":" (erldoc-strip-string + (erldoc-dom-get-text d))) + (erldoc-expand-url href (cdr man))) + table))))) + (let ((span-content + (lambda (span) + (let ((texts)) + (erldoc-dom-walk span + (lambda (d) + (and (erldoc-dom-text-node-p d) + (push (erldoc-strip-string d) texts)))) + (and texts (mapconcat 'identity (nreverse texts) " "))))) + entries) + (erldoc-dom-walk + dom + (lambda (d) + ;; Get the full function signature. + (when (and (eq (car-safe d) 'a) + (gethash (erldoc-dom-get-attribute d 'name) table)) + (push (append (gethash (erldoc-dom-get-attribute d 'name) table) + (list (funcall span-content + (or (erldoc-dom-get-element d 'span) + (cadr (memq d erldoc-dom-walk-siblings)))))) + entries)) + ;; Get data types + (when (and (eq (car-safe d) 'a) + (string-prefix-p "type-" + (or (erldoc-dom-get-attribute d 'name) ""))) + (push (list (concat (car man) ":" (funcall span-content d)) + (concat (cdr man) "#" (erldoc-dom-get-attribute d 'name)) + (funcall span-content erldoc-dom-walk-parent)) + entries)))) + entries))) + +(defun erldoc-parse-all (man-index output &optional json) + (let* ((output (expand-file-name output)) + (table (make-hash-table :size 11503 :test #'equal)) + (mans (erldoc-parse-man-index man-index)) + (progress 1) + (reporter (make-progress-reporter "Parsing Erlang/OTP documentation" + progress (length mans))) + fails all) + (dolist (man mans) + (condition-case err + (push (erldoc-parse-man man) all) + (error (push (error-message-string err) fails))) + (accept-process-output nil 0.01) + (progress-reporter-update reporter (cl-incf progress))) + (when fails + (display-warning 'erldoc-parse-all + (format "\n\n%s" (mapconcat #'identity fails "\n")) + :error)) + (progress-reporter-done reporter) + (mapc (lambda (x) (puthash (car x) (cdr x) table)) + (apply #'nconc (nreverse all))) + (with-temp-buffer + (if (not json) + (pp table (current-buffer)) + (eval-and-compile (require 'json)) + (let ((json-encoding-pretty-print t)) + (insert (json-encode table)))) + (unless (file-directory-p (file-name-directory output)) + (make-directory (file-name-directory output) t)) + (write-region nil nil output nil nil nil 'ask)))) + +(defun erldoc-otp-release () + "Get the otp release version (as string) or nil if not found." + (let ((otp (erldoc-dom-get-text + (erldoc-dom-get-element + (erldoc-parse-html + (erldoc-expand-url "index.html" erldoc-man-index)) + 'title)))) + (and (string-match "[0-9.]+\\'" otp) (match-string 0 otp)))) + +(defvar erldoc-browse-history nil) +(defvar erldoc-lookup-table nil) + +(defun erldoc-lookup-table () + (or erldoc-lookup-table + (progn + (unless (file-exists-p erldoc-output-file) + (let ((of (pcase (erldoc-otp-release) + (`nil erldoc-output-file) + (ver (concat erldoc-output-file "-" ver))))) + (unless (file-exists-p of) + (erldoc-parse-all erldoc-man-index of)) + (unless (string= erldoc-output-file of) + (make-symbolic-link of erldoc-output-file)))) + (setq erldoc-lookup-table + (with-temp-buffer + (insert-file-contents erldoc-output-file) + (read (current-buffer))))))) + +(defun erldoc-best-matches (mfa) + (pcase mfa + ((and `(,m ,f) (let a (erlang-get-function-arity))) + (let ((mfa (format "%s:%s/%s" m f a))) + (cond ((gethash mfa (erldoc-lookup-table)) (list mfa)) + (m (all-completions (concat m ":" f "/") (erldoc-lookup-table))) + (t (let* ((mod (erlang-get-module)) + (mf1 (and mod (concat mod ":" f "/"))) + (mf2 (concat "erlang:" f "/")) + (re (concat ":" (regexp-quote f) "/"))) + (or (and mf1 (all-completions mf1 (erldoc-lookup-table))) + (all-completions mf2 (erldoc-lookup-table)) + (cl-loop for k being the hash-keys of (erldoc-lookup-table) + when (string-match-p re k) + collect k))))))))) + +;;;###autoload +(defun erldoc-browse (mfa) + (interactive + (let ((default + ;; `erlang-mode-syntax-table' is lazily initialised. + (with-syntax-table (or erlang-mode-syntax-table (standard-syntax-table)) + (ignore-errors + (erldoc-best-matches + (or (erlang-get-function-under-point) + (save-excursion + (goto-char (or (cadr (syntax-ppss)) (point))) + (erlang-get-function-under-point)))))))) + (list (completing-read (format (if default "Function {%d %s} (default %s): " + "Function: ") + (length default) + (if (= (length default) 1) "guess" "guesses") + (car default)) + (erldoc-lookup-table) + nil t nil 'erldoc-browse-history default)))) + (or (stringp mfa) + (signal 'wrong-type-argument (list 'string mfa 'mfa))) + (browse-url (or (car (gethash mfa (erldoc-lookup-table))) + (user-error "No documentation for %s" mfa)))) + +;;;###autoload +(defun erldoc-apropos (pattern) + (interactive "sPattern: ") + (with-help-window (help-buffer) + (with-current-buffer standard-output + (princ (concat "Erldoc apropos pattern: " pattern "\n\n")) + (maphash (lambda (k v) + (when (string-match-p pattern k) + (insert-text-button k :type 'help-url + 'help-args (list (car v))) + (insert "\n"))) + (erldoc-lookup-table))))) + +(defun erldoc-tokenize-signature (sig) + ;; Divide SIG into (MF ARGLIST RETTYPE) + (let ((from (if (string-match "\\`.+?(" sig) + (1- (match-end 0)) + 0)) + (to (and (string-match "\\s-*->\\s-*.*?\\'" sig) (match-beginning 0)))) + (list (erldoc-strip-string (substring sig 0 from)) + (erldoc-strip-string (substring sig from (and to (max from to)))) + (and to (erldoc-strip-string (substring sig to)))))) + +(defun erldoc-format-signature (mod fn) + (when (and mod fn (or erldoc-lookup-table + (file-exists-p erldoc-output-file))) + (let ((re (concat "\\`" mod ":" fn "/\\([0-9]+\\)\\'")) + (sigs)) + (maphash (lambda (k v) + (when (string-match re k) + (push (cons (string-to-number (match-string 1 k)) + (cdr (erldoc-tokenize-signature (cadr v)))) + sigs))) + (erldoc-lookup-table)) + (when sigs + ;; Mostly single return type but there are exceptions such as + ;; `beam_lib:chunks/2,3'. + (let ((single-rettype + (cl-reduce (lambda (x1 x2) (and x1 x2 (equal x1 x2) x1)) + sigs :key #'cl-caddr)) + (sigs (sort sigs #'car-less-than-car))) + (if single-rettype + (concat mod ":" fn (mapconcat #'cadr sigs " | ") " " single-rettype) + (mapconcat (lambda (x) (concat mod ":" fn (nth 1 x) " " (nth 2 x))) + sigs "\n"))))))) + +;;;###autoload +(defun erldoc-eldoc-function () + "A function suitable for `eldoc-documentation-function'." + (save-excursion + (pcase (erlang-get-function-under-point) + (`(,_ nil) ) + (`(nil ,fn) (erldoc-format-signature "erlang" fn)) + (`(,mod ,fn) (erldoc-format-signature mod fn))))) + +(defun erldoc-parse-eeps-index () + (let* ((url "http://www.erlang.org/eeps/") + (table (catch 'return + (erldoc-dom-walk (erldoc-parse-html url) + (lambda (d) + (and (eq (car-safe d) 'table) + (equal (erldoc-dom-get-attribute d 'summary) + "Numerical Index of EEPs") + (throw 'return d)))))) + (fix-title (lambda (title) + (replace-regexp-in-string + "`` *" "" (replace-regexp-in-string " *``, *" " by " title)))) + (result)) + (erldoc-dom-walk + table (lambda (d) + (when (eq (car-safe d) 'a) + (push (cons (funcall fix-title (erldoc-dom-get-attribute d 'title)) + (erldoc-expand-url + (erldoc-dom-get-attribute d 'href) + url)) + result)))) + (nreverse result))) + +(defvar erldoc-user-guides nil) + +(defvar erldoc-missing-user-guides + '("compiler" "hipe" "kernel" "os_mon" "parsetools" "typer") + "List of standard Erlang applications with no user guides.") + +;; Search in `code:lib_dir/0' using find LIB_DIR -type f -name +;; '*_app.html'. +(defvar erldoc-app-manuals '("crypto" "diameter" "erl_docgen" + "kernel" "observer" "os_mon" + "runtime_tools" "sasl" "snmp" + "ssl" "test_server" + ("ssh" . "SSH") ("stdlib" . "STDLIB") + ("hipe" . "HiPE") ("typer" . "TypEr")) + "List of applications that come with a manual.") + +(defun erldoc-user-guide-chapters (user-guide) + (pcase-let ((`(,name . ,url) user-guide)) + (unless (member name erldoc-missing-user-guides) + (let ((chaps (erldoc-dom-get-elements-by-id + (erldoc-dom-get-element-by-id (erldoc-parse-html url) "leftnav") + "no"))) + (or chaps (warn "erldoc-user-guide-chapters no chapters found for `%s'" + (cdr user-guide))) + (mapcar (lambda (li) + (cons (concat name "#" (erldoc-dom-get-attribute li 'title)) + (erldoc-expand-url (erldoc-dom-get-attribute + (erldoc-dom-get-element li 'a) 'href) + url))) + chaps))))) + +(defun erldoc-user-guides-1 () + (let ((url (erldoc-expand-url "applications.html" erldoc-man-index)) + app-guides app-mans) + (erldoc-dom-walk + (erldoc-parse-html url) + (lambda (d) + (when (and (eq (car-safe d) 'a) + (not (string-match-p "\\`[0-9.]+\\'" (erldoc-dom-get-text d)))) + (with-demoted-errors "erldoc-user-guides-1: %S" + (let ((name (erldoc-strip-string (erldoc-dom-get-text d))) + (index-page (erldoc-fix-path (erldoc-expand-url + (erldoc-dom-get-attribute d 'href) url)))) + (push (cons name (if (member name erldoc-missing-user-guides) + index-page + (erldoc-expand-url "users_guide.html" index-page))) + app-guides) + ;; Collect application manuals. + (pcase (assoc name (mapcar (lambda (x) (if (consp x) x (cons x x))) + erldoc-app-manuals)) + (`(,_ . ,manual) + (push (cons name + (erldoc-expand-url (format "%s_app.html" manual) + index-page)) + app-mans)))))))) + (list (nreverse app-guides) + (nreverse app-mans)))) + +(defun erldoc-user-guides () + (or erldoc-user-guides + (let ((file (concat erldoc-output-file "-topics"))) + (unless (file-exists-p file) + (unless (file-directory-p (file-name-directory file)) + (make-directory (file-name-directory file) t)) + (with-temp-buffer + (pcase-let ((`(,guides ,mans) (erldoc-user-guides-1))) + (pp (append (cl-mapcan #'erldoc-user-guide-chapters + (append (mapcar + (lambda (dir) + (cons dir (erldoc-expand-url + (concat dir "/users_guide.html") + erldoc-man-index))) + '("design_principles" + "efficiency_guide" + "embedded" + "getting_started" + "installation_guide" + "oam" + "programming_examples" + "reference_manual" + "system_architecture_intro" + "system_principles" + "tutorial")) + guides)) + (mapcar (lambda (man) + (pcase-let ((`(,name . ,url) man)) + (cons (concat name " (App)") url))) + mans) + (erldoc-parse-eeps-index)) + (current-buffer))) + (write-region nil nil file nil nil nil 'ask))) + (setq erldoc-user-guides (with-temp-buffer (insert-file-contents file) + (read (current-buffer))))))) + +;;;###autoload +(defun erldoc-browse-topic (topic) + (interactive + (list (completing-read "User guide: " (erldoc-user-guides) nil t))) + (browse-url (cdr (assoc topic (erldoc-user-guides))))) + +(provide 'erldoc) +;;; erldoc.el ends here diff --git a/lib/tools/src/fprof.erl b/lib/tools/src/fprof.erl index 1291a3e5ec..d1a4624419 100644 --- a/lib/tools/src/fprof.erl +++ b/lib/tools/src/fprof.erl @@ -1702,6 +1702,12 @@ trace_handler({trace_ts, Pid, send, _OtherPid, _Msg, TS} = Trace, dump_stack(Dump, get(Pid), Trace), TS; %% +%% send_to_non_existing_process +trace_handler({trace_ts, Pid, send_to_non_existing_process, _OtherPid, _Msg, TS} = Trace, + _Table, _, Dump) -> + dump_stack(Dump, get(Pid), Trace), + TS; +%% %% 'receive' trace_handler({trace_ts, Pid, 'receive', _Msg, TS} = Trace, _Table, _, Dump) -> diff --git a/system/doc/design_principles/applications.xml b/system/doc/design_principles/applications.xml index 0a1b65ea8e..c673fde07e 100644 --- a/system/doc/design_principles/applications.xml +++ b/system/doc/design_principles/applications.xml @@ -11,7 +11,7 @@ Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - + http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software @@ -19,7 +19,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - + </legalnotice> <title>Applications</title> @@ -172,31 +172,136 @@ ch_app:stop([])</code> </section> <section> - <marker id="app_dir"></marker> - <title>Directory Structure</title> - <p>When packaging code using <c>systools</c>, the code for each - application is placed in a separate directory, - <c>lib/Application-Vsn</c>, where <c>Vsn</c> is the version number.</p> - <p>This can be useful to know, even if <c>systools</c> is not used, - since Erlang/OTP is packaged according to the OTP principles - and thus comes with this directory structure. The code server - (see the <c>code(3)</c> manual page in Kernel) automatically - uses code from - the directory with the highest version number, if more than one - version of an application is present.</p> - <p>The application directory structure can also be used in the - development environment. The version number can then - be omitted from the name.</p> - <p>The application directory has the following sub-directories:</p> - <list type="bulleted"> - <item><c>src</c> - Contains the Erlang source code.</item> - <item><c>ebin</c> - Contains the Erlang object code, the - <c>beam</c> files. The <c>.app</c> file is also placed here.</item> - <item><c>priv</c> - Used for application specific files. For - example, C executables are placed here. The function - <c>code:priv_dir/1</c> is to be used to access this directory.</item> - <item><c>include</c> - Used for include files.</item> - </list> + <marker id="app_dir"></marker> + <title>Directory Structure</title> + <p>When packaging code using <c>systools</c>, the code for each + application is placed in a separate directory, + <c>lib/Application-Vsn</c>, where <c>Vsn</c> is the version number.</p> + <p>This can be useful to know, even if <c>systools</c> is not used, + since Erlang/OTP is packaged according to the OTP principles + and thus comes with a specific directory structure. The code server + (see the <seealso marker="kernel:code"><c>code(3)</c></seealso> manual + page in Kernel) automatically uses code from + the directory with the highest version number, if more than one + version of an application is present.</p> + <section> + <title>Directory Structure guidelines for a Development Environment</title> + <p>Any directory structure for development will suffice as long as the released directory structure + adhere to the <seealso marker="#app_dir_released">description below</seealso>, + but it is encouraged that the same directory structure + also be used in a development environment. The version number should be omitted from the + application directory name since this is an artifact of the release step. + </p> + <p> Some sub-directories are <em>required</em>. Some sub-directories are <em>optional</em>, meaning that it should + only be used if the application itself requires it. Finally, some sub-directories are <em>recommended</em>, + meaning it is encouraged that it is used and used as described here. For example, both documentation + and tests are encouraged to exist in an application for it to be deemed a proper OTP application.</p> +<code type="none"> + ─ ${application} + ├── doc + │ ├── internal + │ ├── examples + │ └── src + ├── include + ├── priv + ├── src + │ └── ${application}.app.src + └── test +</code> + <list type="bulleted"> + <item><c>src</c> - Required. Contains the Erlang source code, the source of the <c>.app</c> file + and internal include files used by the application itself. Additional sub-directories within + <c>src</c> can be used as namespaces to organize source files. These directories should never + be deeper than one level.</item> + <item><c>priv</c> - Optional. Used for application specific files. </item> + <item><c>include</c> - Optional. Used for public include files that must be reachable from + other applications.</item> + <item><c>doc</c> - Recommended. Any source documentation should be placed in sub-directories here.</item> + <item><c>doc/internal</c> - Recommended. Any documentation that describes implementation details about + this application, not intended for publication, should be placed here.</item> + <item><c>doc/examples</c> - Recommended. Source code for examples on how to use this application should + be placed here. It is encouraged that examples are sourced to the public documentation from + this directory.</item> + <item><c>doc/src</c> - Recommended. All source files for documentation, such as Markdown, AsciiDoc or + XML-files, should be placed here.</item> + <item><c>test</c> - Recommended. All files regarding tests, such as test suites and test specifications, + should be placed here. </item> + </list> + + <p>Other directories in the development environment may be needed. If source code from languages other + than Erlang is used, for instance C-code for NIFs, that code should be placed in a separate directory. + By convention it is recommended to prefix such directories with the language name, for example + <c>c_src</c> for C, <c>java_src</c> for Java or <c>go_src</c> for Go. Directories with <c>_src</c> + suffix indicates that it is a part of the application and the compilation step. The final build artifacts + should target the <c>priv/lib</c> or <c>priv/bin</c> directories.</p> + <p>The <c>priv</c> directory holds assets that the application needs during runtime. Executables should + reside in <c>priv/bin</c> and dynamically-linked libraries should reside in <c>priv/lib</c>. Other assets + are free to reside within the <c>priv</c> directory but it is recommended it does so in a structured manner.</p> + <p>Source files from other languages that generate Erlang code, such as ASN.1 or Mibs, should be placed + in directories, at the top level or in <c>src</c>, with the same name as the source language, for example + <c>asn1</c> and <c>mibs</c>. Build artifacts should be placed in their respective language directory, + such as <c>src</c> for Erlang code or <c>java_src</c> for Java code.</p> + <p>The <c>.app</c> file for release may reside in the <c>ebin</c>-directory in a development environment + but it is encouraged that this is an artifact of the build step. By convention a <c>.app.src</c> file + is used, which resides in the <c>src</c> directory. This file is nearly identical as the + <c>.app</c> file but certain fields may be replaced during the build step, such as the application version.</p> + <p>Directory names should not be capitalized.</p> + <p>It is encouraged to omit empty directories.</p> + + </section> + + <section> + <marker id="app_dir_released"></marker> + <title>Directory Structure for a Released System</title> + <p>A released application must follow a certain structure. + </p> +<code type="none"> + ─ ${application}-${version} + ├── bin + ├── doc + │ ├── html + │ ├── man[1-9] + │ ├── pdf + │ ├── internal + │ └── examples + ├── ebin + │ └── ${application}.app + ├── include + ├── priv + │ ├── lib + │ └── bin + └── src +</code> + <list type="bulleted"> + <item><c>src</c> - Optional. Contains the Erlang source code and internal include files + used by the application itself. This directory is no longer required in a released application.</item> + <item><c>ebin</c> - Required. Contains the Erlang object code, the <c>beam</c> files. + The <c>.app</c> file must also be placed here.</item> + <item><c>priv</c> - Optional. Used for application specific files. <c>code:priv_dir/1</c> + is to be used to access this directory.</item> + <item><c>priv/lib</c> - Recommended. Any shared-object files that are used by the application, + such as NIFs or linked-in-drivers, should be placed here.</item> + <item><c>priv/bin</c> - Recommended. Any executable that is used by the application, + such as port-programs, should be placed here.</item> + <item><c>include</c> - Optional. Used for public include files that must be reachable from + other applications.</item> + <item><c>bin</c> - Optional. Any executable that is a product of the application, + such as escripts or shell-scripts, should be placed here.</item> + <item><c>doc</c> - Optional. Any released documentation should be placed in + sub-directories here.</item> + <item><c>doc/man1</c> - Recommended. Man pages for Application executables.</item> + <item><c>doc/man3</c> - Recommended. Man pages for module APIs.</item> + <item><c>doc/man6</c> - Recommended. Man pages for Application overview.</item> + <item><c>doc/html</c> - Optional. HTML pages for the entire Application.</item> + <item><c>doc/pdf</c> - Optional. PDF documentation for the entire Application.</item> + </list> + + <p>The <c>src</c> directory could be useful to release for debugging purposes but is not required. + The <c>include</c> directory should only be released if the applications has public include files.</p> + <p>The only documentation that is recommended to be released in this way are the man pages. HTML and PDF + will normally be distributed in some other manner.</p> + <p>It is encouraged to omit empty directories.</p> + </section> </section> <section> @@ -381,4 +486,3 @@ application:start(Application, Type)</code> <c>shutdown</c>, not <c>normal</c>.</p> </section> </chapter> - diff --git a/system/doc/design_principles/statem.xml b/system/doc/design_principles/statem.xml index bece95e6b8..f627145f9f 100644 --- a/system/doc/design_principles/statem.xml +++ b/system/doc/design_principles/statem.xml @@ -1015,17 +1015,24 @@ open(cast, {button,_}, Data) -> ... ]]></code> <p> - The fact that a postponed event is only retried after a state change - translates into a requirement on the event and state space. - If you have a choice between storing a state data item - in the <c>State</c> or in the <c>Data</c>: - if a change in the item value affects which events that - are handled, then this item is to be part of the state. - </p> - <p> - You want to avoid that you maybe much later decide - to postpone an event in one state and by misfortune it is never retried, - as the code only changes the <c>Data</c> but not the <c>State</c>. + Since a postponed event is only retried after a state change, + you have to think about where to keep a state data item. + You can keep it in the server <c>Data</c> + or in the <c>State</c> itself, + for example by having two more or less identical states + to keep a boolean value, or by using a complex state with + <seealso marker="#Callback Modes">callback mode</seealso> + <seealso marker="stdlib:gen_statem#type-callback_mode"><c>handle_event_function</c></seealso>. + If a change in the value changes the set of events that is handled, + then the value should be kept in the State. + Otherwise no postponed events will be retried + since only the server Data changes. + </p> + <p> + This is not important if you do not postpone events. + But if you later decide to start postponing some events, + then the design flaw of not having separate states + when they should be, might become a hard to find bug. </p> <section> |