diff options
217 files changed, 4679 insertions, 2271 deletions
diff --git a/.gitignore b/.gitignore index 3fc95170aa..71350ca1b9 100644 --- a/.gitignore +++ b/.gitignore @@ -143,6 +143,7 @@ JAVADOC-GENERATED /make/output.mk /make/emd2exml +/make/make_emakefile # Created by "out_build update_primary" /bootstrap/primary_compiler/ diff --git a/OTP_VERSION b/OTP_VERSION index 02f94dcfc1..187bf91fcf 100644 --- a/OTP_VERSION +++ b/OTP_VERSION @@ -1 +1 @@ -19.0.5 +19.1 diff --git a/erts/configure.in b/erts/configure.in index 883ce2db68..46302e8e68 100644 --- a/erts/configure.in +++ b/erts/configure.in @@ -2798,7 +2798,7 @@ if test X${enable_fp_exceptions} != Xyes ; then FPE=unreliable else - AC_MSG_CHECKING([for unreliable floating point execptions]) + AC_MSG_CHECKING([for unreliable floating point exceptions]) AC_TRY_RUN([ diff --git a/erts/doc/src/epmd.xml b/erts/doc/src/epmd.xml index 120ffb6860..311483022d 100644 --- a/erts/doc/src/epmd.xml +++ b/erts/doc/src/epmd.xml @@ -34,24 +34,23 @@ </header> <com>epmd</com> - <comsummary> - <p>Erlang Port Mapper Daemon</p> + <comsummary>Erlang Port Mapper Daemon</comsummary> + + <description> <taglist> <tag><c><![CDATA[epmd [-d|-debug] [DbgExtra...] [-address Addresses] - [-port No] [-daemon] [-relaxed_command_check]]]></c></tag> + [-port No] [-daemon] [-relaxed_command_check]]]></c></tag> <item> <p>Starts the port mapper daemon.</p> </item> <tag><c><![CDATA[epmd [-d|-debug] [-port No] - [-names|-kill|-stop Name]]]></c></tag> + [-names|-kill|-stop Name]]]></c></tag> <item> <p>Communicates with a running port mapper daemon.</p> </item> </taglist> - </comsummary> - <description> <p>This daemon acts as a name server on all hosts involved in distributed Erlang computations. When an Erlang node starts, the node has a name and it obtains an address from the host diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index d0a3a77e43..950a5fe189 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -2689,7 +2689,7 @@ os_prompt%</pre> to the last occurrence is used. Example:</p> <pre> > <input>erlang:make_tuple(5, [], [{2,ignored},{5,zz},{2,aa}]).</input> -{{[],aa,[],[],zz}</pre> +{[],aa,[],[],zz}</pre> </desc> </func> diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index 2a36e5568c..412675fd2b 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -32,6 +32,322 @@ <p>This document describes the changes made to the ERTS application.</p> +<section><title>Erts 8.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix bug for calls from hipe code to BIFs that disable GC + while yielding. Has been causing Dialyzer crashes on ARM + (and presumably all other non-intel platforms).</p> + <p> + Own Id: OTP-13724 Aux Id: PR-1116 </p> + </item> + <item> + <p> + Fix a bug where changing the current working directory of + the VM would not change the current working directory of + programs spawned using <c>erlang:open_port({spawn,""}, + ...)</c>.</p> + <p> + Own Id: OTP-13733 Aux Id: ERL-175 </p> + </item> + <item> + <p> + Fix a bug where disabling tracing from a process that had + return_to tracing enabled and was tracing on + <c>erlang:trace/3</c> would cause a segmentation fault.</p> + <p> + Own Id: OTP-13734</p> + </item> + <item> + <p> + Update all erts documentation to use simpler English, use + consistent terminology and be easier to navigate.</p> + <p> + Own Id: OTP-13740</p> + </item> + <item> + <p> + Add dirty schedulers to the microstate accounting + statistics.</p> + <p> + Own Id: OTP-13744</p> + </item> + <item> + <p> + Fixed dirty scheduler build support on 32-bit windows.</p> + <p> + Own Id: OTP-13759</p> + </item> + <item> + <p> + inet:getstat(Socket) on an SCTP socket returned 0 for + send stats. This bug has now been corrected. Reported by + systra as issue ERL-102 on bugs.erlang.org.</p> + <p> + Own Id: OTP-13773 Aux Id: ERL-102 </p> + </item> + <item> + <p> + AF_UNSPEC and unknown address families were misread by + UDP receive in prim_inet resulting in an exception. This + bug has now been corrected.</p> + <p> + Own Id: OTP-13775</p> + </item> + <item> + <p> + Sweep HiPE stack for literals during code purge.</p> + <p> + Own Id: OTP-13777 Aux Id: PR-1122 </p> + </item> + <item> + <p> + Fix bug in run_erl for OpenBSD that could cause it on + rare occations to exit without starting the program (erl) + at all.</p> + <p> + Own Id: OTP-13795</p> + </item> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + <item> + <p> + Fixed a bug where init:stop could deadlock if a process + with infinite shutdown timeout (e.g. a supervisor) + attempted to load code while terminating.</p> + <p> + Own Id: OTP-13802</p> + </item> + <item> + <p> + Fixed a segmentation fault on sparc CPUs when free'ing a + tracer module's state.</p> + <p> + Own Id: OTP-13803</p> + </item> + <item> + <p> + <c>fun</c>s was not properly handled during purge of a + module. This could cause a crash of the VM after a purge + of a module.</p> + <p> + Own Id: OTP-13809</p> + </item> + <item> + <p> + Fixed a memory leak when the process monitoring a port + crashed.</p> + <p> + Own Id: OTP-13818</p> + </item> + <item> + <p> + Fixed multiple dirty scheduler related tracing bugs.</p> + <p> + Own Id: OTP-13822</p> + </item> + <item> + <p> + Fix error handling in beam code runtime loader for a + number of cases when index and size fields got corrupted + (negative) values.</p> + <p> + Own Id: OTP-13848 Aux Id: ERL-216 </p> + </item> + <item> + <p> + Minor fix of dirty scheduler implementation.</p> + <p> + Own Id: OTP-13852</p> + </item> + <item> + <p> + Calls to <c>erl_drv_send_term()</c> or + <c>erl_drv_output_term()</c> from a non-scheduler thread + while the corresponding port was invalid caused the + emulator to enter an inconsistent state which eventually + caused an emulator crash.</p> + <p> + Own Id: OTP-13866</p> + </item> + <item> + <p> + Fix a rare race condition in <c>erlang:open_port({spawn, + ""}, ...)</c> that would result in the erl_child_setup + program aborting and cause the emulator to exit.</p> + <p> + Own Id: OTP-13868</p> + </item> + <item> + <p>Driver and NIF operations accessing processes or ports + could cause an emulator crash when used from + non-scheduler threads. Those operations are:</p> <list> + <item><c>erl_drv_send_term()</c></item> + <item><c>driver_send_term()</c></item> + <item><c>erl_drv_output_term()</c></item> + <item><c>driver_output_term()</c></item> + <item><c>enif_send()</c></item> + <item><c>enif_port_command()</c></item> </list> + <p> + Own Id: OTP-13869</p> + </item> + <item> + <p> + Fix start scripts generation dependency in Makefile</p> + <p> + Own Id: OTP-13871 Aux Id: ERL-241 </p> + </item> + <item> + <p> + The VM could crash if <c>erlang:get_stacktrace()</c> was + called after a rescheduled NIF had thrown an exception.</p> + <p> + Own Id: OTP-13877</p> + </item> + <item> + <p>Calling <c>code:delete/1</c> before a loading a module + with an on_load function, the old code would be + overwritten, causing a memory or a crash if NIFs were + involved. (Thanks to vans163 for reporting this bug.)</p> + <p> + Own Id: OTP-13893 Aux Id: ERL-240 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Improved accuracy of timeouts on MacOS X. This by setting + premature timeouts followed by a short actual timeout + during scheduler wait.</p> + <p> + Own Id: OTP-13698</p> + </item> + <item> + <p>Added the following symbolic time unit representations + to the <seealso + marker="erlang#type-time_unit"><c>erlang:time_unit()</c></seealso> + type:</p> <list> <item><c>second</c></item> + <item><c>millisecond</c></item> + <item><c>microsecond</c></item> + <item><c>nanosecond</c></item> </list> <p>The following + symbolic time unit representations are now + <em>deprecated</em>, but still part of the + <c>erlang:time_unit()</c> type:</p> <list> + <item><c>seconds</c></item> + <item><c>milli_seconds</c></item> + <item><c>micro_seconds</c></item> + <item><c>nano_seconds</c></item> </list> + <p> + Own Id: OTP-13735</p> + </item> + <item> + <p> + Fix maps hashing entropy of maps with maps keys.</p> + <p> + Own Id: OTP-13763 Aux Id: ERL-199 </p> + </item> + <item> + <p> + Improved dirty scheduler support. A purge of a module can + now be performed without having to wait for completion of + all ongoing dirty NIF calls.</p> + <p> + Note that when enabling support for dirty schedulers, a + new purge strategy will as of ERTS version 8.1 be + enabled. This new strategy is not fully backwards + compatible with the strategy used by default. For more + information see the documentation of <seealso + marker="erts:erlang#check_process_code/3"><c>erlang:check_process_code/3</c></seealso>.</p> + <p> + Own Id: OTP-13808 Aux Id: OTP-13833 </p> + </item> + <item> + <p> + A new purge strategy has been introduced. The new + strategy will by default be disabled during the OTP 19 + release, but will be the only strategy available as of + the OTP 20 release.</p> + <p> + The new strategy is slightly incompatible with the + strategy being used by default in OTP 19. Using the + default strategy, processes holding <c>fun</c>s that + refer to the module being purged either fail a soft + purge, or will be killed during a hard purge. The new + strategy completely ignores <c>fun</c>s. If <c>fun</c>s + referring to the code being purged exist, and are used + after a purge, an exception will be raised upon usage. + That is, the behavior will be exactly the same as the + case when a <c>fun</c> is received by a process after the + purge.</p> + <p> + The new strategy can optionally be enabled when building + OTP during OTP 19, and will automatically be enabled if + the runtime system is built with support for dirty + schedulers.</p> + <p> + For more information see the documentation of <seealso + marker="erts:erlang#check_process_code/3"><c>erlang:check_process_code/3</c></seealso>.</p> + <p> + Own Id: OTP-13833</p> + </item> + <item> + <p> + Fixed unnecessary overestimation of heap size need during + garbage collection.</p> + <p> + Own Id: OTP-13851</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 8.0.5</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed a VM crash that occured in a garbage collection of + a process when it had received binaries. This bug was + introduced in ERTS version 8.0 (OTP 19.0).</p> + <p> + Own Id: OTP-13890</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 8.0.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed a VM crash that occured in garbage collection of a + process when it had received maps over the distribution. + This bug was introduced in ERTS version 8.0 (OTP 19.0).</p> + <p> + Own Id: OTP-13889</p> + </item> + </list> + </section> + +</section> + <section><title>Erts 8.0.3</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/erts/doc/src/run_erl.xml b/erts/doc/src/run_erl.xml index ad7b2c5b85..a9b6a7e2c6 100644 --- a/erts/doc/src/run_erl.xml +++ b/erts/doc/src/run_erl.xml @@ -49,7 +49,7 @@ <funcs> <func> <name>run_erl [-daemon] pipe_dir/ log_dir "exec command - [command_arguments]"</name> + arg1 arg2 ..."</name> <fsummary>Start the Erlang emulator without attached terminal.</fsummary> <desc> <p>Arguments:</p> @@ -92,11 +92,10 @@ </item> </list> </item> - <tag><c>"exec command [command_arguments]"</c></tag> + <tag><c>"exec command arg1 arg2 ..."</c></tag> <item> - <p>In the third argument, <c><![CDATA[command]]></c> is the - executable to execute where everything written to <c>stdin</c> - and <c>stdout</c> is logged to <c><![CDATA[log_dir]]></c>.</p> + <p>A space-separated string specifying the program to be executed. + The second field is typically a command name such as <c>erl</c>.</p> </item> </taglist> </desc> @@ -105,7 +104,7 @@ <section> <title>Notes concerning the Log Files</title> - <p>While running, <c>run_erl</c> (as stated earlier) sends all output, + <p>While running, <c>run_erl</c> sends all output, uninterpreted, to a log file. The file is named <c><![CDATA[erlang.log.N]]></c>, where <c>N</c> is an integer. When the log is "full" (default log size is 100 KB), <c>run_erl</c> starts to log diff --git a/erts/emulator/beam/atom.names b/erts/emulator/beam/atom.names index 9dae67cb2d..2b66bf6f4e 100644 --- a/erts/emulator/beam/atom.names +++ b/erts/emulator/beam/atom.names @@ -505,6 +505,7 @@ atom port_limit atom port_op atom positive atom prepare +atom prepare_on_load atom print atom priority atom private diff --git a/erts/emulator/beam/beam_bif_load.c b/erts/emulator/beam/beam_bif_load.c index ba79068859..5969197168 100644 --- a/erts/emulator/beam/beam_bif_load.c +++ b/erts/emulator/beam/beam_bif_load.c @@ -53,6 +53,7 @@ static struct { ErlFunEntry *def_funs[10]; Uint fe_size; Uint fe_ix; + struct erl_module_instance saved_old; } purge_state; Process *erts_code_purger = NULL; @@ -105,6 +106,8 @@ init_purge_state(void) purge_state.fe_size = sizeof(purge_state.def_funs); purge_state.fe_size /= sizeof(purge_state.def_funs[0]); purge_state.fe_ix = 0; + + purge_state.saved_old.code_hdr = 0; } void @@ -739,10 +742,13 @@ BIF_RETTYPE call_on_load_function_1(BIF_ALIST_1) { Module* modp = erts_get_module(BIF_ARG_1, erts_active_code_ix()); - if (modp && modp->old.code_hdr) { - BIF_TRAP_CODE_PTR_0(BIF_P, modp->old.code_hdr->on_load_function_ptr); + if (!modp || !modp->on_load) { + BIF_ERROR(BIF_P, BADARG); } - else { + if (modp->on_load->code_hdr) { + BIF_TRAP_CODE_PTR_0(BIF_P, + modp->on_load->code_hdr->on_load_function_ptr); + } else { BIF_ERROR(BIF_P, BADARG); } } @@ -765,14 +771,14 @@ BIF_RETTYPE finish_after_on_load_2(BIF_ALIST_2) code_ix = erts_active_code_ix(); modp = erts_get_module(BIF_ARG_1, code_ix); - if (!modp || !modp->old.code_hdr) { + if (!modp || !modp->on_load || !modp->on_load->code_hdr) { error: erts_smp_thr_progress_unblock(); erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_MAIN); erts_release_code_write_permission(); BIF_ERROR(BIF_P, BADARG); } - if (modp->old.code_hdr->on_load_function_ptr == NULL) { + if (modp->on_load->code_hdr->on_load_function_ptr == NULL) { goto error; } if (BIF_ARG_2 != am_false && BIF_ARG_2 != am_true) { @@ -781,14 +787,17 @@ BIF_RETTYPE finish_after_on_load_2(BIF_ALIST_2) if (BIF_ARG_2 == am_true) { int i; - struct erl_module_instance t; /* - * Swap old and new code. + * Make the code with the on_load function current. */ - t = modp->curr; - modp->curr = modp->old; - modp->old = t; + + if (modp->curr.code_hdr) { + modp->old = modp->curr; + } + modp->curr = *modp->on_load; + erts_free(ERTS_ALC_T_PREPARED_CODE, modp->on_load); + modp->on_load = 0; /* * The on_load function succeded. Fix up export entries. @@ -976,8 +985,13 @@ erts_proc_copy_literal_area(Process *c_p, int *redsp, int fcalls, int gc_allowed } if (any_heap_ref_ptrs(c_p->stop, c_p->hend, literals, lit_bsize)) + goto literal_gc; + *redsp += 1; +#ifdef HIPE + if (nstack_any_heap_ref_ptrs(c_p, literals, lit_bsize)) goto literal_gc; *redsp += 1; +#endif if (any_heap_refs(c_p->heap, c_p->htop, literals, lit_bsize)) goto literal_gc; *redsp += 1; @@ -1726,7 +1740,8 @@ BIF_RETTYPE erts_internal_purge_module_2(BIF_ALIST_2) switch (BIF_ARG_2) { - case am_prepare: { + case am_prepare: + case am_prepare_on_load: { /* * Prepare for purge by marking all fun * entries referring to the code to purge @@ -1751,7 +1766,22 @@ BIF_RETTYPE erts_internal_purge_module_2(BIF_ALIST_2) /* * Any code to purge? */ - erts_rlock_old_code(code_ix); + + if (BIF_ARG_2 == am_prepare_on_load) { + erts_rwlock_old_code(code_ix); + } else { + erts_rlock_old_code(code_ix); + } + + if (BIF_ARG_2 == am_prepare_on_load) { + ASSERT(modp->on_load); + ASSERT(modp->on_load->code_hdr); + purge_state.saved_old = modp->old; + modp->old = *modp->on_load; + erts_free(ERTS_ALC_T_PREPARED_CODE, (void *) modp->on_load); + modp->on_load = 0; + } + if (!modp->old.code_hdr) res = am_false; else { @@ -1769,7 +1799,12 @@ BIF_RETTYPE erts_internal_purge_module_2(BIF_ALIST_2) ERTS_SET_COPY_LITERAL_AREA(modp->old.code_hdr->literal_area); #endif } - erts_runlock_old_code(code_ix); + + if (BIF_ARG_2 == am_prepare_on_load) { + erts_rwunlock_old_code(code_ix); + } else { + erts_runlock_old_code(code_ix); + } } #ifndef ERTS_SMP @@ -1906,8 +1941,14 @@ BIF_RETTYPE erts_internal_purge_module_2(BIF_ALIST_2) modp->old.code_length = 0; modp->old.catches = BEAM_CATCHES_NIL; erts_remove_from_ranges(code); + ERTS_BIF_PREP_RET(ret, am_true); } + + if (purge_state.saved_old.code_hdr) { + modp->old = purge_state.saved_old; + purge_state.saved_old.code_hdr = 0; + } erts_rwunlock_old_code(code_ix); } if (is_blocking) { diff --git a/erts/emulator/beam/beam_load.c b/erts/emulator/beam/beam_load.c index b7e802775d..0afdedf6c2 100644 --- a/erts/emulator/beam/beam_load.c +++ b/erts/emulator/beam/beam_load.c @@ -833,11 +833,20 @@ erts_finish_loading(Binary* magic, Process* c_p, size = stp->loaded_size; erts_total_code_size += size; - if (stp->on_load) { - inst_p = &mod_tab_p->old; - } else { + + if (!stp->on_load) { inst_p = &mod_tab_p->curr; + } else { + mod_tab_p->on_load = + (struct erl_module_instance *) + erts_alloc(ERTS_ALC_T_PREPARED_CODE, + sizeof(struct erl_module_instance)); + inst_p = mod_tab_p->on_load; + inst_p->nif = 0; + inst_p->num_breakpoints = 0; + inst_p->num_traced_exports = 0; } + inst_p->code_hdr = stp->hdr; inst_p->code_length = size; @@ -2635,6 +2644,10 @@ load_code(LoaderState* stp) * End of code found. */ case op_int_code_end: + if (function_number != stp->num_functions) { + LoadError2(stp, "too few functions (%u) in module (header said %u)", + function_number, stp->num_functions); + } stp->codev_size = codev_size; stp->ci = ci; stp->function = THE_NON_VALUE; diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index 75f8ebefbd..6f641a1ea7 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -150,6 +150,7 @@ static void move_msgq_to_heap(Process *p); static int reached_max_heap_size(Process *p, Uint total_heap_size, Uint extra_heap_size, Uint extra_old_heap_size); static void init_gc_info(ErtsGCInfo *gcip); +static Uint64 next_vheap_size(Process* p, Uint64 vheap, Uint64 vheap_sz); #ifdef HARDDEBUG static void disallow_heap_frag_ref_in_heap(Process* p); @@ -387,6 +388,11 @@ erts_gc_after_bif_call_lhf(Process* p, ErlHeapFragment *live_hf_end, return result; } + if (!p->mbuf) { + /* Must have GC:d in BIF call... invalidate live_hf_end */ + live_hf_end = ERTS_INVALID_HFRAG_PTR; + } + if (is_non_value(result)) { if (p->freason == TRAP) { #if HIPE @@ -535,9 +541,19 @@ young_gen_usage(Process *p) if (p->flags & F_ON_HEAP_MSGQ) { ErtsMessage *mp; - for (mp = p->msg.first; mp; mp = mp->next) + for (mp = p->msg.first; mp; mp = mp->next) { + /* + * We leave not yet decoded distribution messages + * as they are in the queue since it is not + * possible to determine a maximum size until + * actual decoding. However, we use their estimated + * size when calculating need, and by this making + * it more likely that they will fit on the heap + * when actually decoded. + */ if (mp->data.attached) hsz += erts_msg_attached_data_size(mp); + } } aheap = p->abandoned_heap; @@ -747,6 +763,9 @@ do_major_collection: p->last_old_htop = p->old_htop; #endif + ASSERT(!p->mbuf); + ASSERT(!ERTS_IS_GC_DESIRED(p)); + return reds; } @@ -2251,44 +2270,31 @@ copy_one_frag(Eterm** hpp, ErlOffHeap* off_heap, static void move_msgq_to_heap(Process *p) { + ErtsMessage **mpp = &p->msg.first; + Uint64 pre_oh = MSO(p).overhead; while (*mpp) { ErtsMessage *mp = *mpp; if (mp->data.attached) { ErlHeapFragment *bp; - ErtsHeapFactory factory; - - erts_factory_proc_prealloc_init(&factory, p, - erts_msg_attached_data_size(mp)); - - if (is_non_value(ERL_MESSAGE_TERM(mp))) { - if (mp->data.dist_ext) { - ASSERT(mp->data.dist_ext->heap_size >= 0); - if (is_not_nil(ERL_MESSAGE_TOKEN(mp))) { - bp = erts_dist_ext_trailer(mp->data.dist_ext); - ERL_MESSAGE_TOKEN(mp) = copy_struct(ERL_MESSAGE_TOKEN(mp), - bp->used_size, - &factory.hp, - factory.off_heap); - erts_cleanup_offheap(&bp->off_heap); - } - ERL_MESSAGE_TERM(mp) = erts_decode_dist_ext(&factory, - mp->data.dist_ext); - erts_free_dist_ext_copy(mp->data.dist_ext); - mp->data.dist_ext = NULL; - } - } - else { + + /* + * We leave not yet decoded distribution messages + * as they are in the queue since it is not + * possible to determine a maximum size until + * actual decoding... + */ + if (is_value(ERL_MESSAGE_TERM(mp))) { bp = erts_message_to_heap_frag(mp); if (bp->next) - erts_move_multi_frags(&factory.hp, factory.off_heap, bp, + erts_move_multi_frags(&p->htop, &p->off_heap, bp, mp->m, ERL_MESSAGE_REF_ARRAY_SZ, 0); else - copy_one_frag(&factory.hp, factory.off_heap, bp, + copy_one_frag(&p->htop, &p->off_heap, bp, mp->m, ERL_MESSAGE_REF_ARRAY_SZ); if (mp->data.attached != ERTS_MSG_COMBINED_HFRAG) { @@ -2305,12 +2311,15 @@ move_msgq_to_heap(Process *p) mp = new_mp; } } - - erts_factory_close(&factory); } mpp = &(*mpp)->next; } + + if (pre_oh != MSO(p).overhead) { + /* Got new binaries; update vheap size... */ + BIN_VHEAP_SZ(p) = next_vheap_size(p, MSO(p).overhead, BIN_VHEAP_SZ(p)); + } } static Uint diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index 27392a5996..3a547982da 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -2268,7 +2268,7 @@ int enif_consume_timeslice(ErlNifEnv* env, int percent) * NIF exports need a few more items than the Export struct provides, * including the erl_module_nif* and a NIF function pointer, so the * NifExport below adds those. The Export member must be first in the - * struct. The saved_mfa, exception_thrown, saved_argc, rootset_extra, and + * struct. The saved_current, exception_thrown, saved_argc, rootset_extra, and * rootset members are used to track the MFA, any pending exception, and * arguments of the top NIF in case a chain of one or more * enif_schedule_nif() calls results in an exception, since in that case @@ -2282,7 +2282,7 @@ typedef struct { Export exp; struct erl_module_nif* m; NativeFunPtr fp; - Eterm saved_mfa[3]; + BeamInstr *saved_current; int exception_thrown; int saved_argc; int rootset_extra; @@ -2393,9 +2393,8 @@ init_nif_sched_data(ErlNifEnv* env, NativeFunPtr direct_fp, NativeFunPtr indirec reg[i] = (Eterm) argv[i]; } if (need_save) { - ep->saved_mfa[0] = proc->current[0]; - ep->saved_mfa[1] = proc->current[1]; - ep->saved_mfa[2] = proc->current[2]; + ASSERT(proc->current); + ep->saved_current = proc->current; ep->saved_argc = argc; } proc->i = (BeamInstr*) ep->exp.addressv[0]; @@ -2418,21 +2417,21 @@ static void restore_nif_mfa(Process* proc, NifExport* ep, int exception) { int i; - Eterm* reg = erts_proc_sched_data(proc)->x_reg_array; ERTS_SMP_LC_ASSERT(!(proc->static_flags & ERTS_STC_FLG_SHADOW_PROC)); ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(proc) & ERTS_PROC_LOCK_MAIN); - proc->current[0] = ep->saved_mfa[0]; - proc->current[1] = ep->saved_mfa[1]; - proc->current[2] = ep->saved_mfa[2]; - if (exception) + ASSERT(ep->saved_current != &ep->exp.code[0]); + proc->current = ep->saved_current; + ep->saved_current = NULL; + if (exception) { + Eterm* reg = erts_proc_sched_data(proc)->x_reg_array; for (i = 0; i < ep->saved_argc; i++) reg[i] = ep->rootset[i+1]; + } ep->saved_argc = 0; - ep->saved_mfa[0] = THE_NON_VALUE; } #ifdef ERTS_DIRTY_SCHEDULERS @@ -2507,6 +2506,7 @@ execute_dirty_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) */ ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); ASSERT(ep && fp); + ep->fp = NULL; erts_smp_atomic32_read_band_mb(&proc->state, ~(ERTS_PSFLG_DIRTY_CPU_PROC | ERTS_PSFLG_DIRTY_IO_PROC)); @@ -2591,7 +2591,7 @@ schedule_dirty_nif(ErlNifEnv* env, int flags, int argc, const ERL_NIF_TERM argv[ } ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); - need_save = (ep == NULL || is_non_value(ep->saved_mfa[0])); + need_save = (ep == NULL || !ep->saved_current); result = init_nif_sched_data(env, execute_dirty_nif, fp, need_save, argc, argv); if (scheduler <= 0) erts_smp_proc_unlock(proc, ERTS_PROC_LOCK_MAIN); @@ -2669,7 +2669,7 @@ enif_schedule_nif(ErlNifEnv* env, const char* fun_name, int flags, } ep = (NifExport*) ERTS_PROC_GET_NIF_TRAP_EXPORT(proc); - need_save = (ep == NULL || is_non_value(ep->saved_mfa[0])); + need_save = (ep == NULL || !ep->saved_current); if (flags) { #ifdef ERTS_DIRTY_SCHEDULERS @@ -3203,18 +3203,20 @@ BIF_RETTYPE load_nif_2(BIF_ALIST_2) if (init_func != NULL) handle = init_func; + this_mi = &module_p->curr; + prev_mi = &module_p->old; if (in_area(caller, module_p->old.code_hdr, module_p->old.code_length)) { - if (module_p->old.code_hdr->on_load_function_ptr) { - this_mi = &module_p->old; + ret = load_nif_error(BIF_P, "old_code", "Calling load_nif from old " + "module '%T' not allowed", mod_atom); + goto error; + } else if (module_p->on_load) { + ASSERT(module_p->on_load->code_hdr->on_load_function_ptr); + if (module_p->curr.code_hdr) { prev_mi = &module_p->curr; } else { - ret = load_nif_error(BIF_P, "old_code", "Calling load_nif from old " - "module '%T' not allowed", mod_atom); - goto error; + prev_mi = &module_p->old; } - } else { - this_mi = &module_p->curr; - prev_mi = &module_p->old; + this_mi = module_p->on_load; } if (init_func == NULL && diff --git a/erts/emulator/beam/erl_thr_progress.c b/erts/emulator/beam/erl_thr_progress.c index 26d71f573f..700ed90def 100644 --- a/erts/emulator/beam/erl_thr_progress.c +++ b/erts/emulator/beam/erl_thr_progress.c @@ -700,6 +700,7 @@ leader_update(ErtsThrPrgrData *tpd) tpd->leader_state.chk_next_ix = no_managed; erts_atomic32_set_nob(&intrnl->misc.data.umrefc_ix.current, (erts_aint32_t) new_umrefc_ix); + tpd->leader_state.umrefc_ix.current = new_umrefc_ix; ETHR_MEMBAR(ETHR_StoreLoad); refc = erts_atomic_read_nob(&intrnl->umrefc[umrefc_ix].refc); ASSERT(refc >= 0); diff --git a/erts/emulator/beam/erl_trace.c b/erts/emulator/beam/erl_trace.c index f4d92564c1..8c84303997 100644 --- a/erts/emulator/beam/erl_trace.c +++ b/erts/emulator/beam/erl_trace.c @@ -3157,7 +3157,7 @@ erts_tracer_update(ErtsTracer *tracer, const ErtsTracer new_tracer) Not sure if it is worth it, we save 4 words (sizeof(ErlHeapFragment)) per tracer. */ Eterm *hp = erts_alloc(ERTS_ALC_T_HEAP_FRAG, - 2*sizeof(Eterm) + sizeof(ErtsThrPrgrLaterOp)); + 3*sizeof(Eterm) + sizeof(ErtsThrPrgrLaterOp)); *tracer = CONS(hp, ERTS_TRACER_MODULE(new_tracer), ERTS_TRACER_STATE(new_tracer)); } else { diff --git a/erts/emulator/beam/io.c b/erts/emulator/beam/io.c index cb8792dffa..77dbe92241 100644 --- a/erts/emulator/beam/io.c +++ b/erts/emulator/beam/io.c @@ -6591,16 +6591,20 @@ deliver_term_check_port(ErlDrvTermData port_id, Eterm *connected_p, ErtsThrPrgrDelayHandle dhndl = erts_thr_progress_unmanaged_delay(); #endif erts_aint32_t state; + int res = 1; Port *prt = erts_port_lookup_raw((Eterm) port_id); - if (!prt) - return -1; + if (!prt) { + res = -1; + goto done; + } state = erts_atomic32_read_nob(&prt->state); if (state & (ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP | ERTS_PORT_SFLG_CLOSING)) { if (state & ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP) - return -1; + res = -1; else - return 0; + res = 0; + goto done; } if (connected_p) { #ifdef ERTS_SMP @@ -6609,25 +6613,27 @@ deliver_term_check_port(ErlDrvTermData port_id, Eterm *connected_p, #endif *connected_p = ERTS_PORT_GET_CONNECTED(prt); } + +done: + #ifdef ERTS_SMP if (dhndl != ERTS_THR_PRGR_DHANDLE_MANAGED) { + ERTS_SMP_LC_ASSERT(!prt || !erts_lc_is_port_locked(prt)); erts_thr_progress_unmanaged_continue(dhndl); ETHR_MEMBAR(ETHR_LoadLoad|ETHR_LoadStore); } else #endif - { + if (res == 1) { + ERTS_SMP_LC_ASSERT(erts_lc_is_port_locked(prt)); *trace_prt = prt; } - ERTS_SMP_LC_ASSERT(dhndl == ERTS_THR_PRGR_DHANDLE_MANAGED - ? erts_lc_is_port_locked(prt) - : !erts_lc_is_port_locked(prt)); - return 1; + return res; } int erl_drv_output_term(ErlDrvTermData port_id, ErlDrvTermData* data, int len) { /* May be called from arbitrary thread */ - Eterm connected; + Eterm connected = NIL; /* Shut up faulty warning... */ Port *prt = NULL; int res = deliver_term_check_port(port_id, &connected, &prt); if (res <= 0) diff --git a/erts/emulator/beam/module.c b/erts/emulator/beam/module.c index 3eb11f1693..4f36377450 100644 --- a/erts/emulator/beam/module.c +++ b/erts/emulator/beam/module.c @@ -85,6 +85,7 @@ static Module* module_alloc(Module* tmpl) obj->old.num_breakpoints = 0; obj->curr.num_traced_exports = 0; obj->old.num_traced_exports = 0; + obj->on_load = 0; return obj; } @@ -201,6 +202,7 @@ void module_start_staging(void) dst_mod->curr = src_mod->curr; dst_mod->old = src_mod->old; + dst_mod->on_load = src_mod->on_load; } /* @@ -214,6 +216,7 @@ void module_start_staging(void) dst_mod->curr = src_mod->curr; dst_mod->old = src_mod->old; + dst_mod->on_load = src_mod->on_load; } newsz = index_table_sz(dst); erts_smp_atomic_add_nob(&tot_module_bytes, (newsz - oldsz)); diff --git a/erts/emulator/beam/module.h b/erts/emulator/beam/module.h index 5a60bc90d9..1c1afc8461 100644 --- a/erts/emulator/beam/module.h +++ b/erts/emulator/beam/module.h @@ -39,6 +39,7 @@ typedef struct erl_module { struct erl_module_instance curr; struct erl_module_instance old; /* protected by "old_code" rwlock */ + struct erl_module_instance* on_load; } Module; Module* erts_get_module(Eterm mod, ErtsCodeIndex code_ix); diff --git a/erts/emulator/drivers/common/inet_drv.c b/erts/emulator/drivers/common/inet_drv.c index 254d3baeb1..1885338ce5 100644 --- a/erts/emulator/drivers/common/inet_drv.c +++ b/erts/emulator/drivers/common/inet_drv.c @@ -777,6 +777,7 @@ static int is_nonzero(const char *s, size_t n) #define INET_LOPT_NETNS 38 /* Network namespace pathname */ #define INET_LOPT_TCP_SHOW_ECONNRESET 39 /* tell user about incoming RST */ #define INET_LOPT_LINE_DELIM 40 /* Line delimiting char */ +#define INET_OPT_TCLASS 41 /* IPv6 transport class */ /* SCTP options: a separate range, from 100: */ #define SCTP_OPT_RTOINFO 100 #define SCTP_OPT_ASSOCINFO 101 @@ -1330,6 +1331,7 @@ static ErlDrvTermData am_reuseaddr; static ErlDrvTermData am_dontroute; static ErlDrvTermData am_priority; static ErlDrvTermData am_tos; +static ErlDrvTermData am_tclass; static ErlDrvTermData am_ipv6_v6only; static ErlDrvTermData am_netns; #endif @@ -3720,6 +3722,7 @@ static void inet_init_sctp(void) { INIT_ATOM(dontroute); INIT_ATOM(priority); INIT_ATOM(tos); + INIT_ATOM(tclass); INIT_ATOM(ipv6_v6only); INIT_ATOM(netns); @@ -6228,6 +6231,15 @@ static int inet_set_opts(inet_descriptor* desc, char* ptr, int len) #else continue; #endif +#if defined(IPV6_TCLASS) && defined(SOL_IPV6) + case INET_OPT_TCLASS: + proto = SOL_IPV6; + type = IPV6_TCLASS; + propagate = 1; + DEBUGF(("inet_set_opts(%ld): s=%d, IPV6_TCLASS=%d\r\n", + (long)desc->port, desc->s, ival)); + break; +#endif case TCP_OPT_NODELAY: proto = IPPROTO_TCP; @@ -6661,6 +6673,19 @@ static int sctp_set_opts(inet_descriptor* desc, char* ptr, int len) continue; /* Option not supported -- ignore it */ # endif +# if defined(IPV6_TCLASS) && defined(SOL_IPV6) + case INET_OPT_TCLASS: + { + arg.ival= get_int32 (curr); curr += 4; + proto = SOL_IPV6; + type = IPV6_TCLASS; + arg_ptr = (char*) (&arg.ival); + arg_sz = sizeof ( arg.ival); + break; + } +# endif + + case INET_OPT_IPV6_V6ONLY: # if HAVE_DECL_IPV6_V6ONLY { @@ -7162,6 +7187,15 @@ static ErlDrvSSizeT inet_fill_opts(inet_descriptor* desc, put_int32(0, ptr); continue; #endif + case INET_OPT_TCLASS: +#if defined(IPV6_TCLASS) && defined(SOL_IPV6) + proto = SOL_IPV6; + type = IPV6_TCLASS; + break; +#else + TRUNCATE_TO(0,ptr); + continue; +#endif case INET_OPT_REUSEADDR: type = SO_REUSEADDR; break; @@ -7556,6 +7590,7 @@ static ErlDrvSSizeT sctp_fill_opts(inet_descriptor* desc, case INET_OPT_DONTROUTE: case INET_OPT_PRIORITY : case INET_OPT_TOS : + case INET_OPT_TCLASS : case INET_OPT_IPV6_V6ONLY: case SCTP_OPT_AUTOCLOSE: case SCTP_OPT_MAXSEG : @@ -7629,6 +7664,19 @@ static ErlDrvSSizeT sctp_fill_opts(inet_descriptor* desc, continue; # endif } + case INET_OPT_TCLASS: + { +# if defined(IPV6_TCLASS) && defined(SOL_IPV6) + proto = SOL_IPV6; + type = IPV6_TCLASS; + is_int = 1; + tag = am_tclass; + break; +# else + /* Not supported -- ignore */ + continue; +# endif + } case INET_OPT_IPV6_V6ONLY: # if HAVE_DECL_IPV6_V6ONLY { diff --git a/erts/emulator/sys/unix/erl_child_setup.c b/erts/emulator/sys/unix/erl_child_setup.c index 6b9ddd8da4..69fc6c2879 100644 --- a/erts/emulator/sys/unix/erl_child_setup.c +++ b/erts/emulator/sys/unix/erl_child_setup.c @@ -348,7 +348,7 @@ child_error: * for posterity. */ static void handle_sigchld(int sig) { - int buff[2], res; + int buff[2], res, __preverrno = errno; sys_sigblock(SIGCHLD); @@ -362,6 +362,16 @@ static void handle_sigchld(int sig) { } sys_sigrelease(SIGCHLD); + + /* We save and restore the original errno as otherwise + the thread we are running in may end up with an + unexpected errno. An example of when this happened + was when the select in main had gotten an EINTR but + before the errno was checked the signal handler + was called and set errno to ECHILD from waitpid + which caused erl_child_setup to abort as it does + not expect ECHILD to be set after select */ + errno = __preverrno; } #if defined(__ANDROID__) @@ -423,7 +433,7 @@ main(int argc, char *argv[]) sigemptyset(&sa.sa_mask); sa.sa_flags = SA_RESTART | SA_NOCLDSTOP; if (sigaction(SIGCHLD, &sa, 0) == -1) { - perror(0); + perror(NULL); exit(1); } diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index 6fb86f6dda..089efec3e8 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -715,11 +715,13 @@ static RETSIGTYPE suspend_signal(void) static RETSIGTYPE suspend_signal(int signum) #endif { - int res; - int buf[1]; - do { - res = read(sig_suspend_fds[0], buf, sizeof(int)); - } while (res < 0 && errno == EINTR); + int res, buf[1], __errno = errno; + do { + res = read(sig_suspend_fds[0], buf, sizeof(int)); + } while (res < 0 && errno == EINTR); + + /* restore previous errno in case read changed it */ + errno = __errno; } #endif /* #ifdef ERTS_SYS_SUSPEND_SIGNAL */ diff --git a/erts/emulator/test/driver_SUITE.erl b/erts/emulator/test/driver_SUITE.erl index 1df72193a6..2fbf6eae61 100644 --- a/erts/emulator/test/driver_SUITE.erl +++ b/erts/emulator/test/driver_SUITE.erl @@ -452,11 +452,7 @@ timer_delay(Config) when is_list(Config) -> TimeBefore = erlang:monotonic_time(), Timeout0 = 350, erlang:port_command(Port, <<?DELAY_START_TIMER,Timeout0:32>>), - Timeout = Timeout0 + - case os:type() of - {win32,_} -> 0; %Driver doesn't sleep on Windows. - _ -> 1000 - end, + Timeout = Timeout0 + 1000, receive {Port,{data,[?TIMER]}} -> Elapsed = erl_millisecs() - erl_millisecs(TimeBefore), diff --git a/erts/emulator/test/driver_SUITE_data/timer_drv.c b/erts/emulator/test/driver_SUITE_data/timer_drv.c index 57538e0d57..c3ce3b6e49 100644 --- a/erts/emulator/test/driver_SUITE_data/timer_drv.c +++ b/erts/emulator/test/driver_SUITE_data/timer_drv.c @@ -1,5 +1,13 @@ #include <stdio.h> #include "erl_driver.h" +#ifdef __WIN32__ +# include <windows.h> +#else +# include <sys/time.h> +# include <sys/types.h> +# include <sys/select.h> +# include <unistd.h> +#endif #define get_int32(s) ((((unsigned char*) (s))[0] << 24) | \ (((unsigned char*) (s))[1] << 16) | \ @@ -17,6 +25,7 @@ static ErlDrvData timer_start(ErlDrvPort, char*); static void timer_stop(ErlDrvData); static void timer_read(ErlDrvData, char*, ErlDrvSizeT); static void timer(ErlDrvData); +static void ms_sleep(int ms); static ErlDrvEntry timer_driver_entry = { @@ -75,9 +84,7 @@ static void timer_read(ErlDrvData p, char *buf, ErlDrvSizeT len) reply[0] = CANCELLED; driver_output(port, reply, 1); } else if (buf[0] == DELAY_START_TIMER) { -#ifndef __WIN32__ - sleep(1); -#endif + ms_sleep(1000); driver_set_timer(port, get_int32(buf + 1)); } } @@ -95,3 +102,34 @@ static void timer(ErlDrvData port) reply[0] = TIMER; driver_output((ErlDrvPort)port, reply, 1); } + +static void +ms_sleep(int ms) +{ + /* Important that we do not return too early... */ + ErlDrvTime time, timeout_time; + + time = erl_drv_monotonic_time(ERL_DRV_USEC); + + timeout_time = time + ((ErlDrvTime) ms)*1000; + + while (time < timeout_time) { + ErlDrvTime timeout = timeout_time - time; + +#ifdef __WIN32__ + Sleep((DWORD) (timeout / 1000)); +#else + { + struct timeval tv; + + tv.tv_sec = (long) timeout / (1000*1000); + tv.tv_usec = (long) timeout % (1000*1000); + + select(0, NULL, NULL, NULL, &tv); + } +#endif + + time = erl_drv_monotonic_time(ERL_DRV_USEC); + } + +} diff --git a/erts/emulator/test/hipe_SUITE.erl b/erts/emulator/test/hipe_SUITE.erl index 3e682b8d88..a556b4ddc0 100644 --- a/erts/emulator/test/hipe_SUITE.erl +++ b/erts/emulator/test/hipe_SUITE.erl @@ -51,6 +51,9 @@ t_copy_literals(Config) when is_list(Config) -> true = erlang:delete_module(literals), true = erlang:purge_module(literals), + %% Give the literal collector some time to work... + receive after 2000 -> ok end, + %% check that the ex-literals are ok [a,b,c] = ref_cell:call(PA, get), {a,b,c} = ref_cell:call(PB, get), diff --git a/erts/emulator/test/nif_SUITE.erl b/erts/emulator/test/nif_SUITE.erl index 8df2733fac..9c1694fa8a 100644 --- a/erts/emulator/test/nif_SUITE.erl +++ b/erts/emulator/test/nif_SUITE.erl @@ -29,6 +29,7 @@ -export([all/0, suite/0, init_per_testcase/2, end_per_testcase/2, basic/1, reload/1, upgrade/1, heap_frag/1, + t_on_load/1, types/1, many_args/1, binaries/1, get_string/1, get_atom/1, maps/1, api_macros/1, @@ -68,6 +69,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [basic, reload, upgrade, heap_frag, types, many_args, + t_on_load, binaries, get_string, get_atom, maps, api_macros, from_array, iolist_as_binary, resource, resource_binary, resource_takeover, threading, send, send2, send3, @@ -83,10 +85,19 @@ all() -> nif_port_command, nif_snprintf]. +init_per_testcase(t_on_load, Config) -> + ets:new(nif_SUITE, [named_table]), + Config; init_per_testcase(_Case, Config) -> Config. +end_per_testcase(t_on_load, _Config) -> + ets:delete(nif_SUITE), + testcase_cleanup(); end_per_testcase(_Func, _Config) -> + testcase_cleanup(). + +testcase_cleanup() -> P1 = code:purge(nif_mod), Del = code:delete(nif_mod), P2 = code:purge(nif_mod), @@ -174,11 +185,33 @@ upgrade(Config) when is_list(Config) -> true = erlang:delete_module(nif_mod), [] = nif_mod_call_history(), + %% Repeat upgrade again but from old (deleted) instance + {module,nif_mod} = erlang:load_module(nif_mod,Bin), + undefined = nif_mod:lib_version(), + 1 = call(Pid,lib_version), + [{lib_version,1,9,109}] = nif_mod_call_history(), + + ok = nif_mod:load_nif_lib(Config, 1), + 1 = nif_mod:lib_version(), + [{upgrade,1,10,110},{lib_version,1,11,111}] = nif_mod_call_history(), + + upgraded = call(Pid,upgrade), + false = check_process_code(Pid, nif_mod), + true = erlang:purge_module(nif_mod), + [{unload,1,12,112}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + [{lib_version,1,13,113}] = nif_mod_call_history(), + + true = erlang:delete_module(nif_mod), + [] = nif_mod_call_history(), + + Pid ! die, {'DOWN', MRef, process, Pid, normal} = receive_any(), false = check_process_code(Pid, nif_mod), true = erlang:purge_module(nif_mod), - [{unload,1,9,109}] = nif_mod_call_history(), + [{unload,1,14,114}] = nif_mod_call_history(), %% Module upgrade with different lib version {module,nif_mod} = erlang:load_module(nif_mod,Bin), @@ -215,17 +248,170 @@ upgrade(Config) when is_list(Config) -> true = erlang:delete_module(nif_mod), [] = nif_mod_call_history(), + + %% Reverse upgrade but from old (deleted) instance + {module,nif_mod} = erlang:load_module(nif_mod,Bin), + undefined = nif_mod:lib_version(), + [] = nif_mod_call_history(), + 2 = call(Pid2,lib_version), + [{lib_version,2,4,204}] = nif_mod_call_history(), + + ok = nif_mod:load_nif_lib(Config, 1), + 1 = nif_mod:lib_version(), + [{upgrade,1,1,101},{lib_version,1,2,102}] = nif_mod_call_history(), + + 2 = call(Pid2,lib_version), + [{lib_version,2,5,205}] = nif_mod_call_history(), + + upgraded = call(Pid2,upgrade), + false = check_process_code(Pid2, nif_mod), + true = erlang:purge_module(nif_mod), + [{unload,2,6,206}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + [{lib_version,1,3,103}] = nif_mod_call_history(), + + true = erlang:delete_module(nif_mod), + [] = nif_mod_call_history(), + + Pid2 ! die, {'DOWN', MRef2, process, Pid2, normal} = receive_any(), false= check_process_code(Pid2, nif_mod), true = erlang:purge_module(nif_mod), - [{unload,2,4,204}] = nif_mod_call_history(), + [{unload,1,4,104}] = nif_mod_call_history(), true = lists:member(?MODULE, erlang:system_info(taints)), true = lists:member(nif_mod, erlang:system_info(taints)), verify_tmpmem(TmpMem), ok. +%% Test loading/upgrade in on_load +t_on_load(Config) when is_list(Config) -> + TmpMem = tmpmem(), + ensure_lib_loaded(Config), + + Data = proplists:get_value(data_dir, Config), + File = filename:join(Data, "nif_mod"), + {ok,nif_mod,Bin} = compile:file(File, [binary,return_errors, + {d,'USE_ON_LOAD'}]), + + %% Use ETS to tell nif_mod:on_load what to do + ets:insert(nif_SUITE, {data_dir, Data}), + ets:insert(nif_SUITE, {lib_version, 1}), + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + hold_nif_mod_priv_data(nif_mod:get_priv_data_ptr()), + [{load,1,1,101},{get_priv_data_ptr,1,2,102}] = nif_mod_call_history(), + + {Pid,MRef} = nif_mod:start(), + 1 = call(Pid,lib_version), + [{lib_version,1,3,103}] = nif_mod_call_history(), + + %% Module upgrade with same lib-version + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + 1 = nif_mod:lib_version(), + 1 = call(Pid,lib_version), + [{upgrade,1,4,104},{lib_version,1,5,105},{lib_version,1,6,106}] = nif_mod_call_history(), + + upgraded = call(Pid,upgrade), + false = check_process_code(Pid, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,1,7,107}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + [{lib_version,1,8,108}] = nif_mod_call_history(), + + true = code:delete(nif_mod), + [] = nif_mod_call_history(), + + %% Repeat upgrade again but from old (deleted) instance + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + [{upgrade,1,9,109}] = nif_mod_call_history(), + 1 = nif_mod:lib_version(), + 1 = call(Pid,lib_version), + [{lib_version,1,10,110},{lib_version,1,11,111}] = nif_mod_call_history(), + + upgraded = call(Pid,upgrade), + false = check_process_code(Pid, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,1,12,112}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + [{lib_version,1,13,113}] = nif_mod_call_history(), + + true = code:delete(nif_mod), + [] = nif_mod_call_history(), + + + Pid ! die, + {'DOWN', MRef, process, Pid, normal} = receive_any(), + false = check_process_code(Pid, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,1,14,114}] = nif_mod_call_history(), + + %% Module upgrade with different lib version + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + hold_nif_mod_priv_data(nif_mod:get_priv_data_ptr()), + [{load,1,1,101},{get_priv_data_ptr,1,2,102}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + {Pid2,MRef2} = nif_mod:start(), + 1 = call(Pid2,lib_version), + [{lib_version,1,3,103},{lib_version,1,4,104}] = nif_mod_call_history(), + + true = ets:insert(nif_SUITE,{lib_version,2}), + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + [{upgrade,2,1,201}] = nif_mod_call_history(), + + 2 = nif_mod:lib_version(), + 1 = call(Pid2,lib_version), + [{lib_version,2,2,202},{lib_version,1,5,105}] = nif_mod_call_history(), + + upgraded = call(Pid2,upgrade), + false = check_process_code(Pid2, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,1,6,106}] = nif_mod_call_history(), + + 2 = nif_mod:lib_version(), + 2 = call(Pid2,lib_version), + [{lib_version,2,3,203},{lib_version,2,4,204}] = nif_mod_call_history(), + + true = code:delete(nif_mod), + [] = nif_mod_call_history(), + + %% Reverse upgrade but from old (deleted) instance + ets:insert(nif_SUITE,{lib_version,1}), + {module,nif_mod} = code:load_binary(nif_mod,File,Bin), + [{upgrade,1,1,101}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + 2 = call(Pid2,lib_version), + [{lib_version,1,2,102},{lib_version,2,5,205}] = nif_mod_call_history(), + + upgraded = call(Pid2,upgrade), + false = check_process_code(Pid2, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,2,6,206}] = nif_mod_call_history(), + + 1 = nif_mod:lib_version(), + [{lib_version,1,3,103}] = nif_mod_call_history(), + + true = code:delete(nif_mod), + [] = nif_mod_call_history(), + + + Pid2 ! die, + {'DOWN', MRef2, process, Pid2, normal} = receive_any(), + false= check_process_code(Pid2, nif_mod), + true = code:soft_purge(nif_mod), + [{unload,1,4,104}] = nif_mod_call_history(), + + true = lists:member(?MODULE, erlang:system_info(taints)), + true = lists:member(nif_mod, erlang:system_info(taints)), + verify_tmpmem(TmpMem), + ok. + + %% Test NIF building heap fragments heap_frag(Config) when is_list(Config) -> TmpMem = tmpmem(), diff --git a/erts/emulator/test/nif_SUITE_data/nif_mod.erl b/erts/emulator/test/nif_SUITE_data/nif_mod.erl index eec1bb8858..1fcc33faa4 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_mod.erl +++ b/erts/emulator/test/nif_SUITE_data/nif_mod.erl @@ -29,6 +29,16 @@ -define(nif_stub,nif_stub_error(?LINE)). +-ifdef(USE_ON_LOAD). +-on_load(on_load/0). + +on_load() -> + [{data_dir, Path}] = ets:lookup(nif_SUITE, data_dir), + [{lib_version, Ver}] = ets:lookup(nif_SUITE, lib_version), + erlang:load_nif(filename:join(Path,libname(Ver)), []). + +-endif. + load_nif_lib(Config, Ver) -> load_nif_lib(Config, Ver, []). diff --git a/erts/etc/unix/cerl.src b/erts/etc/unix/cerl.src index c5422ab2ed..30f2d831b5 100644 --- a/erts/etc/unix/cerl.src +++ b/erts/etc/unix/cerl.src @@ -399,19 +399,29 @@ elif [ "x$GDB" = "xdump" ]; then cmdfile="/tmp/.cerlgdb.$$" case "x$core" in x/*) - gdbcmd="$EMU_NAME ${core}" ;; *) dir=`pwd` - gdbcmd="$EMU_NAME ${dir}/${core}" + core="${dir}/${core}" ;; esac - echo "set width 0 + case `uname` in + Darwin) + echo " +thread backtrace all +quit +" > $cmdfile + exec lldb -s $cmdfile -c ${core} $EMU_NAME + ;; + *) + echo "set width 0 set height 0 set verbose off source $ROOTDIR/erts/etc/unix/etp-commands thread apply all bt " > $cmdfile - exec gdb --batch --command=$cmdfile $gdbcmd + exec gdb --batch --command=$cmdfile $EMU_NAME $core + ;; + esac fi diff --git a/erts/etc/unix/format_man_pages b/erts/etc/unix/format_man_pages index 54f2d90c28..0afff13571 100644 --- a/erts/etc/unix/format_man_pages +++ b/erts/etc/unix/format_man_pages @@ -107,7 +107,7 @@ case :"$TARGET" in if [ -f $file ]; then name=`echo $file | sed 's/\.[^.]*$//'` sec=`echo $file | sed 's/.*\.//'` - /usr/bin/groff -Tascii -mandoc $ERL_ROOT/man/man$sec/$file \ + /usr/bin/groff -Tutf8 -mandoc $ERL_ROOT/man/man$sec/$file \ > $ERL_ROOT/man/cat$sec/$file fi done diff --git a/erts/preloaded/ebin/erts_code_purger.beam b/erts/preloaded/ebin/erts_code_purger.beam Binary files differindex a1eb126098..6956eee740 100644 --- a/erts/preloaded/ebin/erts_code_purger.beam +++ b/erts/preloaded/ebin/erts_code_purger.beam diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam Binary files differindex 22817be8f4..227b62b7d3 100644 --- a/erts/preloaded/ebin/erts_internal.beam +++ b/erts/preloaded/ebin/erts_internal.beam diff --git a/erts/preloaded/ebin/prim_inet.beam b/erts/preloaded/ebin/prim_inet.beam Binary files differindex ae4861c336..1a573ce297 100644 --- a/erts/preloaded/ebin/prim_inet.beam +++ b/erts/preloaded/ebin/prim_inet.beam diff --git a/erts/preloaded/src/erts_code_purger.erl b/erts/preloaded/src/erts_code_purger.erl index ee4fcedd2d..28d71fd07e 100644 --- a/erts/preloaded/src/erts_code_purger.erl +++ b/erts/preloaded/src/erts_code_purger.erl @@ -22,7 +22,8 @@ %% Purpose : Implement system process erts_code_purger %% to handle code module purging. --export([start/0, purge/1, soft_purge/1, pending_purge_lambda/3]). +-export([start/0, purge/1, soft_purge/1, pending_purge_lambda/3, + finish_after_on_load/2]). -spec start() -> term(). start() -> @@ -40,6 +41,11 @@ loop() -> Res = do_soft_purge(Mod), From ! {reply, soft_purge, Res, Ref}; + {finish_after_on_load,{Mod,Keep},From,Ref} + when is_atom(Mod), is_pid(From) -> + Res = do_finish_after_on_load(Mod, Keep), + From ! {reply, finish_after_on_load, Res, Ref}; + {test_purge, Mod, From, Type, Ref} when is_atom(Mod), is_pid(From) -> do_test_purge(Mod, From, Type, Ref); @@ -129,6 +135,35 @@ do_soft_purge(Mod) -> end) end. +%% finish_after_on_load(Module, Keep) +%% Finish after running on_load function. If Keep is false, +%% purge the code for the on_load function. + +finish_after_on_load(Mod, Keep) -> + Ref = make_ref(), + erts_code_purger ! {finish_after_on_load, {Mod,Keep}, self(), Ref}, + receive + {reply, finish_after_on_load, Result, Ref} -> + Result + end. + +do_finish_after_on_load(Mod, Keep) -> + erlang:finish_after_on_load(Mod, Keep), + case Keep of + true -> + ok; + false -> + case erts_internal:purge_module(Mod, prepare_on_load) of + false -> + true; + true -> + _ = check_proc_code(erlang:processes(), Mod, true), + true = erts_internal:purge_module(Mod, complete) + end + end. + + + %% %% check_proc_code(Pids, Mod, Hard) - Send asynchronous %% requests to all processes to perform a check_process_code diff --git a/erts/preloaded/src/erts_internal.erl b/erts/preloaded/src/erts_internal.erl index 6229754c8c..6aae5ba38c 100644 --- a/erts/preloaded/src/erts_internal.erl +++ b/erts/preloaded/src/erts_internal.erl @@ -304,7 +304,7 @@ release_literal_area_switch() -> -spec purge_module(Module, Op) -> boolean() when Module :: module(), - Op :: 'prepare' | 'abort' | 'complete'. + Op :: 'prepare' | 'prepare_on_load' | 'abort' | 'complete'. purge_module(_Module, _Op) -> erlang:nif_error(undefined). diff --git a/erts/preloaded/src/prim_inet.erl b/erts/preloaded/src/prim_inet.erl index bcf16402b0..61f727e8a4 100644 --- a/erts/preloaded/src/prim_inet.erl +++ b/erts/preloaded/src/prim_inet.erl @@ -347,7 +347,17 @@ accept_opts(L, S) -> case getopts(L, [active, nodelay, keepalive, delay_send, priority, tos]) of {ok, Opts} -> case setopts(S, Opts) of - ok -> {ok, S}; + ok -> + case getopts(L, [tclass]) of + {ok, []} -> + {ok, S}; + {ok, TClassOpts} -> + case setopts(S, TClassOpts) of + ok -> + {ok, S}; + Error -> close(S), Error + end + end; Error -> close(S), Error end; Error -> @@ -1196,6 +1206,7 @@ enc_opt(sndbuf) -> ?INET_OPT_SNDBUF; enc_opt(recbuf) -> ?INET_OPT_RCVBUF; enc_opt(priority) -> ?INET_OPT_PRIORITY; enc_opt(tos) -> ?INET_OPT_TOS; +enc_opt(tclass) -> ?INET_OPT_TCLASS; enc_opt(nodelay) -> ?TCP_OPT_NODELAY; enc_opt(multicast_if) -> ?UDP_OPT_MULTICAST_IF; enc_opt(multicast_ttl) -> ?UDP_OPT_MULTICAST_TTL; @@ -1255,6 +1266,7 @@ dec_opt(?INET_OPT_SNDBUF) -> sndbuf; dec_opt(?INET_OPT_RCVBUF) -> recbuf; dec_opt(?INET_OPT_PRIORITY) -> priority; dec_opt(?INET_OPT_TOS) -> tos; +dec_opt(?INET_OPT_TCLASS) -> tclass; dec_opt(?TCP_OPT_NODELAY) -> nodelay; dec_opt(?UDP_OPT_MULTICAST_IF) -> multicast_if; dec_opt(?UDP_OPT_MULTICAST_TTL) -> multicast_ttl; @@ -1329,6 +1341,7 @@ type_opt_1(sndbuf) -> int; type_opt_1(recbuf) -> int; type_opt_1(priority) -> int; type_opt_1(tos) -> int; +type_opt_1(tclass) -> int; type_opt_1(nodelay) -> bool; type_opt_1(ipv6_v6only) -> bool; %% multicast diff --git a/erts/start_scripts/Makefile b/erts/start_scripts/Makefile index dfd8153f32..ae2521474e 100644 --- a/erts/start_scripts/Makefile +++ b/erts/start_scripts/Makefile @@ -17,6 +17,9 @@ # # %CopyrightEnd% # + +.NOTPARALLEL: + include $(ERL_TOP)/make/target.mk include $(ERL_TOP)/make/$(TARGET)/otp.mk @@ -69,7 +72,7 @@ debug opt script: rel $(INSTALL_SCRIPTS) $(RELEASES_SRC) rel: $(REL_SCRIPTS) -RELEASES.src: +RELEASES.src: $(SS_ROOT)/start_sasl.rel $(gen_verbose)$(INSTALL_DIR) $(SS_TMP) $(V_at)( cd $(SS_TMP) && \ $(ERL) -noinput +B -eval 'release_handler:create_RELEASES("%ERL_ROOT%", "$(SS_ROOT)", "$(SS_ROOT)/start_sasl.rel", []), halt()') diff --git a/erts/vsn.mk b/erts/vsn.mk index acd4509304..ad9148f61f 100644 --- a/erts/vsn.mk +++ b/erts/vsn.mk @@ -18,7 +18,7 @@ # %CopyrightEnd% # -VSN = 8.0.3 +VSN = 8.1 # Port number 4365 in 4.2 # Port number 4366 in 4.3 diff --git a/lib/asn1/doc/src/notes.xml b/lib/asn1/doc/src/notes.xml index ac3d9c828e..499a7e40c3 100644 --- a/lib/asn1/doc/src/notes.xml +++ b/lib/asn1/doc/src/notes.xml @@ -32,6 +32,22 @@ <p>This document describes the changes made to the asn1 application.</p> +<section><title>Asn1 4.0.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Compiling multiple ASN.1 modules in the same directory + with parallel make (make -j) should now be safe.</p> + <p> + Own Id: OTP-13624</p> + </item> + </list> + </section> + +</section> + <section><title>Asn1 4.0.3</title> <section><title>Improvements and New Features</title> diff --git a/lib/asn1/vsn.mk b/lib/asn1/vsn.mk index 527af05da1..e4bf3e2236 100644 --- a/lib/asn1/vsn.mk +++ b/lib/asn1/vsn.mk @@ -1 +1 @@ -ASN1_VSN = 4.0.3 +ASN1_VSN = 4.0.4 diff --git a/lib/common_test/doc/src/ct.xml b/lib/common_test/doc/src/ct.xml index ffc64cba67..53ef41dd5b 100644 --- a/lib/common_test/doc/src/ct.xml +++ b/lib/common_test/doc/src/ct.xml @@ -620,6 +620,21 @@ </func> <func> + <name>get_verbosity(Category) -> Level | undefined</name> + <fsummary>Read the verbosity level for a logging category.</fsummary> + <type> + <v>Category = default | atom()</v> + <v>Level = integer()</v> + </type> + <desc><marker id="get_verbosity-1"/> + <p>This function returns the verbosity level for the specified logging + category. See the <seealso marker="write_test_chapter#logging"> + User's Guide</seealso> for details. Use the value <c>default</c> to read + the general verbosity level.</p> + </desc> + </func> + + <func> <name>install(Opts) -> ok | {error, Reason}</name> <fsummary>Installs configuration files and event handlers.</fsummary> <type> @@ -1225,6 +1240,21 @@ </func> <func> + <name>set_verbosity(Category, Level) -> ok</name> + <fsummary>Set the verbosity level for a logging category.</fsummary> + <type> + <v>Category = default | atom()</v> + <v>Level = integer()</v> + </type> + <desc><marker id="set_verbosity-2"/> + <p>Use this function to set, or modify, the verbosity level for a logging + category. See the <seealso marker="write_test_chapter#logging"> + User's Guide</seealso> for details. Use the value <c>default</c> to set the + general verbosity level.</p> + </desc> + </func> + + <func> <name>sleep(Time) -> ok</name> <fsummary>This function, similar to timer:sleep/1, suspends the test case for a specified time.</fsummary> diff --git a/lib/common_test/doc/src/ct_telnet.xml b/lib/common_test/doc/src/ct_telnet.xml index eba3c3030d..8e85cccc99 100644 --- a/lib/common_test/doc/src/ct_telnet.xml +++ b/lib/common_test/doc/src/ct_telnet.xml @@ -337,7 +337,7 @@ <c>FullMatch</c> is the string matched by the whole regular expression, and <c>SubMatchN</c> is the string that matched subexpression number <c>N</c>. Subexpressions are denoted with - <c>(' ')</c> in the regular expression.</p> + <c>'(' ')'</c> in the regular expression.</p> <p>If a <c>Tag</c> is speciifed, the returned <c>Match</c> also includes the matched <c>Tag</c>. Otherwise, only <c>RxMatch</c> diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index 32ae699c7a..7653670d30 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -33,6 +33,42 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.12.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + If the telnet server would pause during transmission of a + line of text before terminating the line, the + <c>ct_telnet:expect/3</c> function would print the line + twice in the test case HTML log. This problem has been + fixed.</p> + <p> + Own Id: OTP-13730 Aux Id: seq13135 </p> + </item> + <item> + <p> + The functions <c>ct:set_verbosity/2</c> and + <c>ct:get_verbosity/1</c> have been added in order to + make it possible for test cases, CT Hooks, or test + framework functions, to modify and read verbosity levels + for logging.</p> + <p> + Own Id: OTP-13841</p> + </item> + <item> + <p><c>make</c> (tools) and <c>ct_make</c> (common_test) + would crash if an Erlang source file contained a + <c>-warning()</c> directive.</p> + <p> + Own Id: OTP-13855</p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.12.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index 7bd2ccf588..1d3fbb6f76 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -1031,6 +1031,10 @@ 4. Categorized info, importance = 25 6. Categorized error, importance = 99</pre> + <p>The functions <seealso marker="ct#set_verbosity-2"><c>ct:set_verbosity/2</c></seealso> + and <seealso marker="ct#get_verbosity-1"><c>ct:get_verbosity/1</c></seealso> may be used + to modify and read verbosity levels during test execution.</p> + <p>The arguments <c>Format</c> and <c>FormatArgs</c> in <c>ct:log/print/pal</c> are always passed on to the STDLIB function <c>io:format/3</c> (For details, see the <seealso marker="stdlib:io"><c>io</c></seealso> manual page).</p> diff --git a/lib/common_test/src/ct.erl b/lib/common_test/src/ct.erl index d7ae81a5ce..f9f845e1a9 100644 --- a/lib/common_test/src/ct.erl +++ b/lib/common_test/src/ct.erl @@ -68,6 +68,7 @@ log/1, log/2, log/3, log/4, log/5, print/1, print/2, print/3, print/4, pal/1, pal/2, pal/3, pal/4, + set_verbosity/2, get_verbosity/1, capture_start/0, capture_stop/0, capture_get/0, capture_get/1, fail/1, fail/2, comment/1, comment/2, make_priv_dir/0, testcases/2, userdata/2, userdata/3, @@ -715,6 +716,24 @@ pal(Category,Importance,Format,Args) -> ct_logs:tc_pal(Category,Importance,Format,Args). %%%----------------------------------------------------------------- +%%% @spec set_verbosity(Category, Level) -> ok +%%% Category = default | atom() +%%% Level = integer() +%%% +%%% @doc Set the verbosity level for a category +set_verbosity(Category, Level) -> + ct_util:set_verbosity({Category,Level}). + +%%%----------------------------------------------------------------- +%%% @spec get_verbosity(Category) -> Level | undefined +%%% Category = default | atom() +%%% Level = integer() +%%% +%%% @doc Read the verbosity level for a category +get_verbosity(Category) -> + ct_util:get_verbosity(Category). + +%%%----------------------------------------------------------------- %%% @spec capture_start() -> ok %%% %%% @doc Start capturing all text strings printed to stdout during diff --git a/lib/common_test/test_server/ts_run.erl b/lib/common_test/test_server/ts_run.erl index 66db1ff9a7..82ae44ec06 100644 --- a/lib/common_test/test_server/ts_run.erl +++ b/lib/common_test/test_server/ts_run.erl @@ -258,7 +258,7 @@ make_command(Vars, Spec, State) -> run_batch(Vars, _Spec, State) -> process_flag(trap_exit, true), - Command = State#state.command ++ " -noinput -s erlang halt", + Command = State#state.command ++ " -noinput -eval \"erlang:halt(0,[{flush,false}]).\"", ts_lib:progress(Vars, 1, "Command: ~ts~n", [Command]), io:format(user, "Command: ~ts~n",[Command]), Port = open_port({spawn, Command}, [stream, in, eof, exit_status]), diff --git a/lib/common_test/vsn.mk b/lib/common_test/vsn.mk index c68750886a..ab5cfd7a80 100644 --- a/lib/common_test/vsn.mk +++ b/lib/common_test/vsn.mk @@ -1 +1 @@ -COMMON_TEST_VSN = 1.12.2 +COMMON_TEST_VSN = 1.12.3 diff --git a/lib/compiler/doc/src/notes.xml b/lib/compiler/doc/src/notes.xml index dd6b132a92..6aaf16e9a5 100644 --- a/lib/compiler/doc/src/notes.xml +++ b/lib/compiler/doc/src/notes.xml @@ -32,6 +32,43 @@ <p>This document describes the changes made to the Compiler application.</p> +<section><title>Compiler 7.0.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + If the compiler fails to write the BEAM file, it will now + report the reason of the error for the write operation.</p> + <p> + Own Id: OTP-13701</p> + </item> + <item> + <p> + Fixed an internal compiler error. (Thanks to Svilen + Ivanov for reporting this bug.)</p> + <p> + Own Id: OTP-13780 Aux Id: ERL-202 </p> + </item> + <item> + <p> + The compiler could crash when trying to compile a + complicated expression with multiple catches all on one + line . (Thanks to Thomas Arts for reporting this bug.)</p> + <p> + Own Id: OTP-13804 Aux Id: ERL-209 </p> + </item> + <item> + <p> + Eliminated a few internal compiler failures.</p> + <p> + Own Id: OTP-13863</p> + </item> + </list> + </section> + +</section> + <section><title>Compiler 7.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/compiler/src/beam_validator.erl b/lib/compiler/src/beam_validator.erl index 4c0cb6780a..16dba35adc 100644 --- a/lib/compiler/src/beam_validator.erl +++ b/lib/compiler/src/beam_validator.erl @@ -808,9 +808,11 @@ validate_bs_skip_utf(Fail, Ctx, Live, Vst0) -> %% A possibility for garbage collection must not occur between setelement/3 and %% set_tuple_element/3. %% +%% Note that #vst.current will be 'none' if the instruction is unreachable. +%% val_dsetel({move,_,_}, Vst) -> Vst; -val_dsetel({call_ext,3,{extfunc,erlang,setelement,3}}, #vst{current=St}=Vst) -> +val_dsetel({call_ext,3,{extfunc,erlang,setelement,3}}, #vst{current=#st{}=St}=Vst) -> Vst#vst{current=St#st{setelem=true}}; val_dsetel({set_tuple_element,_,_,_}, #vst{current=#st{setelem=false}}) -> error(illegal_context_for_set_tuple_element); diff --git a/lib/compiler/src/sys_core_fold.erl b/lib/compiler/src/sys_core_fold.erl index e0de50f3ae..08b02101a6 100644 --- a/lib/compiler/src/sys_core_fold.erl +++ b/lib/compiler/src/sys_core_fold.erl @@ -468,7 +468,8 @@ bitstr(#c_bitstr{val=Val,size=Size}=BinSeg, Sub) -> %% Currently, we don't attempt to check binaries because they %% are difficult to check. -is_safe_simple(#c_var{}, _) -> true; +is_safe_simple(#c_var{}=Var, _) -> + not cerl:is_c_fname(Var); is_safe_simple(#c_cons{hd=H,tl=T}, Sub) -> is_safe_simple(H, Sub) andalso is_safe_simple(T, Sub); is_safe_simple(#c_tuple{es=Es}, Sub) -> is_safe_simple_list(Es, Sub); diff --git a/lib/compiler/test/beam_validator_SUITE.erl b/lib/compiler/test/beam_validator_SUITE.erl index 263fd2ca7e..ca85eef688 100644 --- a/lib/compiler/test/beam_validator_SUITE.erl +++ b/lib/compiler/test/beam_validator_SUITE.erl @@ -32,7 +32,8 @@ bad_bin_match/1,bad_dsetel/1, state_after_fault_in_catch/1,no_exception_in_catch/1, undef_label/1,illegal_instruction/1,failing_gc_guard_bif/1, - map_field_lists/1,cover_bin_opt/1]). + map_field_lists/1,cover_bin_opt/1, + val_dsetel/1]). -include_lib("common_test/include/ct.hrl"). @@ -60,7 +61,7 @@ groups() -> freg_state,bad_bin_match,bad_dsetel, state_after_fault_in_catch,no_exception_in_catch, undef_label,illegal_instruction,failing_gc_guard_bif, - map_field_lists,cover_bin_opt]}]. + map_field_lists,cover_bin_opt,val_dsetel]}]. init_per_suite(Config) -> Config. @@ -546,3 +547,23 @@ beam_val(M) -> _ = [io:put_chars(beam_validator:format_error(E)) || E <- Errors], Errors. + +%%%------------------------------------------------------------------------- + +val_dsetel(_Config) -> + self() ! 13, + {'EXIT',{{try_clause,participating},_}} = (catch night(0)), + ok. + +night(Turned) -> + receive + 13 -> + try participating of engine -> 16 after false end + end, + %% The setelement/3 call is unreachable. + Turned(setelement(#{true => Turned}, + participating(Turned, "suit", 40, []), + Turned < Turned)), + ok. + +participating(_, _, _, _) -> ok. diff --git a/lib/compiler/test/core_fold_SUITE.erl b/lib/compiler/test/core_fold_SUITE.erl index 376d2c8e9a..ced0e39d06 100644 --- a/lib/compiler/test/core_fold_SUITE.erl +++ b/lib/compiler/test/core_fold_SUITE.erl @@ -26,7 +26,7 @@ unused_multiple_values_error/1,unused_multiple_values/1, multiple_aliases/1,redundant_boolean_clauses/1, mixed_matching_clauses/1,unnecessary_building/1, - no_no_file/1]). + no_no_file/1,configuration/1]). -export([foo/0,foo/1,foo/2,foo/3]). @@ -45,7 +45,7 @@ groups() -> unused_multiple_values_error,unused_multiple_values, multiple_aliases,redundant_boolean_clauses, mixed_matching_clauses,unnecessary_building, - no_no_file]}]. + no_no_file,configuration]}]. init_per_suite(Config) -> @@ -499,4 +499,16 @@ experiment() -> end, ok. + +%% Make sure we don't try to move a fun into a guard. +configuration(_Config) -> + {'EXIT',_} = (catch configuration()), + ok. + +configuration() -> + [forgotten || Components <- enemy, is_tuple(fun art/0)]. + +art() -> + creating. + id(I) -> I. diff --git a/lib/compiler/vsn.mk b/lib/compiler/vsn.mk index 334784657e..87fde38f2b 100644 --- a/lib/compiler/vsn.mk +++ b/lib/compiler/vsn.mk @@ -1 +1 @@ -COMPILER_VSN = 7.0.1 +COMPILER_VSN = 7.0.2 diff --git a/lib/crypto/c_src/crypto.c b/lib/crypto/c_src/crypto.c index d0044fe723..00fc81c84f 100644 --- a/lib/crypto/c_src/crypto.c +++ b/lib/crypto/c_src/crypto.c @@ -37,7 +37,9 @@ #include <openssl/opensslconf.h> #include <openssl/crypto.h> +#ifndef OPENSSL_NO_DES #include <openssl/des.h> +#endif /* #ifndef OPENSSL_NO_DES */ /* #include <openssl/idea.h> This is not supported on the openssl OTP requires */ #include <openssl/dsa.h> #include <openssl/rsa.h> @@ -458,16 +460,29 @@ struct cipher_type_t { const size_t key_len; /* != 0 to also match on key_len */ }; +#ifdef OPENSSL_NO_DES +#define COND_NO_DES_PTR(Ptr) (NULL) +#else +#define COND_NO_DES_PTR(Ptr) (Ptr) +#endif + struct cipher_type_t cipher_types[] = { {{"rc2_cbc"}, {&EVP_rc2_cbc}}, - {{"des_cbc"}, {&EVP_des_cbc}}, - {{"des_cfb"}, {&EVP_des_cfb8}}, - {{"des_ecb"}, {&EVP_des_ecb}}, - {{"des_ede3_cbc"}, {&EVP_des_ede3_cbc}}, - {{"des_ede3_cbf"}, + {{"des_cbc"}, {COND_NO_DES_PTR(&EVP_des_cbc)}}, + {{"des_cfb"}, {COND_NO_DES_PTR(&EVP_des_cfb8)}}, + {{"des_ecb"}, {COND_NO_DES_PTR(&EVP_des_ecb)}}, + {{"des_ede3_cbc"}, {COND_NO_DES_PTR(&EVP_des_ede3_cbc)}}, + {{"des_ede3_cbf"}, /* Misspelled, retained */ +#ifdef HAVE_DES_ede3_cfb_encrypt + {COND_NO_DES_PTR(&EVP_des_ede3_cfb8)} +#else + {NULL} +#endif + }, + {{"des_ede3_cfb"}, #ifdef HAVE_DES_ede3_cfb_encrypt - {&EVP_des_ede3_cfb8} + {COND_NO_DES_PTR(&EVP_des_ede3_cfb8)} #else {NULL} #endif @@ -749,7 +764,7 @@ static ERL_NIF_TERM algo_hash[8]; /* increase when extending the list */ static int algo_pubkey_cnt; static ERL_NIF_TERM algo_pubkey[7]; /* increase when extending the list */ static int algo_cipher_cnt; -static ERL_NIF_TERM algo_cipher[21]; /* increase when extending the list */ +static ERL_NIF_TERM algo_cipher[23]; /* increase when extending the list */ static void init_algorithms_types(ErlNifEnv* env) { @@ -785,10 +800,13 @@ static void init_algorithms_types(ErlNifEnv* env) algo_pubkey[algo_pubkey_cnt++] = enif_make_atom(env, "srp"); algo_cipher_cnt = 0; +#ifndef OPENSSL_NO_DES algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "des3_cbc"); algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "des_ede3"); #ifdef HAVE_DES_ede3_cfb_encrypt algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "des3_cbf"); + algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "des3_cfb"); +#endif #endif algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "aes_cbc"); algo_cipher[algo_cipher_cnt++] = enif_make_atom(env, "aes_cbc128"); @@ -800,8 +818,11 @@ static void init_algorithms_types(ErlNifEnv* env) #ifdef HAVE_AES_IGE algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"aes_ige256"); #endif +#ifndef OPENSSL_NO_DES algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"des_cbc"); algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"des_cfb"); + algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"des_ecb"); +#endif algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"blowfish_cbc"); algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"blowfish_cfb64"); algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"blowfish_ofb64"); @@ -2141,7 +2162,7 @@ static ERL_NIF_TERM dss_verify_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM DSA *dsa; int i; - if (!argv[0] == atom_sha + if (argv[0] != atom_sha || !enif_inspect_binary(env, argv[1], &digest_bin) || digest_bin.size != SHA_DIGEST_LENGTH || !enif_inspect_binary(env, argv[2], &sign_bin) @@ -2467,7 +2488,7 @@ static ERL_NIF_TERM dss_sign_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar DSA* dsa; int i; - if (!argv[0] == atom_sha + if (argv[0] != atom_sha || !enif_inspect_binary(env, argv[1], &digest_bin) || digest_bin.size != SHA_DIGEST_LENGTH) { return enif_make_badarg(env); diff --git a/lib/crypto/doc/src/crypto.xml b/lib/crypto/doc/src/crypto.xml index 5a5627747c..eda0f7af51 100644 --- a/lib/crypto/doc/src/crypto.xml +++ b/lib/crypto/doc/src/crypto.xml @@ -136,7 +136,7 @@ <code>stream_cipher() = rc4 | aes_ctr </code> <code>block_cipher() = aes_cbc | aes_cfb8 | aes_cfb128 | aes_ige256 | blowfish_cbc | - blowfish_cfb64 | des_cbc | des_cfb | des3_cbc | des3_cbf | des_ede3 | rc2_cbc </code> + blowfish_cfb64 | des_cbc | des_cfb | des3_cbc | des3_cfb | des_ede3 | rc2_cbc </code> <code>aead_cipher() = aes_gcm | chacha20_poly1305 </code> @@ -161,7 +161,7 @@ </p> <code> cipher_algorithms() = aes_cbc | aes_cfb8 | aes_cfb128 | aes_ctr | aes_gcm | aes_ige256 | blowfish_cbc | blowfish_cfb64 | chacha20_poly1305 | des_cbc | des_cfb | - des3_cbc | des3_cbf | des_ede3 | rc2_cbc | rc4 </code> + des3_cbc | des3_cfb | des_ede3 | rc2_cbc | rc4 </code> <code> public_key_algorithms() = rsa |dss | ecdsa | dh | ecdh | ec_gf2m</code> <p>Note that ec_gf2m is not strictly a public key algorithm, but a restriction on what curves are supported with ecdsa and ecdh. diff --git a/lib/crypto/doc/src/notes.xml b/lib/crypto/doc/src/notes.xml index 56e165a1d1..4ae64e059e 100644 --- a/lib/crypto/doc/src/notes.xml +++ b/lib/crypto/doc/src/notes.xml @@ -31,6 +31,32 @@ </header> <p>This document describes the changes made to the Crypto application.</p> +<section><title>Crypto 3.7.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Crypto has been fixed to work against OpenSSL versions + with disabled DES ciphers. Correct spelling of cipher + algorithm 'des3_cfb' has been introduced; the previous + misspeling still works.</p> + <p> + Own Id: OTP-13783 Aux Id: ERL-203 </p> + </item> + <item> + <p> + The size of an internal array in crypto has been fixed to + not segfault when having all possible ciphers. Bug fix by + Duncan Overbruck.</p> + <p> + Own Id: OTP-13789 Aux Id: PR-1140 </p> + </item> + </list> + </section> + +</section> + <section><title>Crypto 3.7</title> <section><title>Improvements and New Features</title> diff --git a/lib/crypto/src/crypto.erl b/lib/crypto/src/crypto.erl index 025d57e9c5..da8626e38a 100644 --- a/lib/crypto/src/crypto.erl +++ b/lib/crypto/src/crypto.erl @@ -274,7 +274,7 @@ hmac_final_n(Context, HashLen) -> %% Ecrypt/decrypt %%% -spec block_encrypt(des_cbc | des_cfb | - des3_cbc | des3_cbf | des_ede3 | + des3_cbc | des3_cbf | des3_cfb | des_ede3 | blowfish_cbc | blowfish_cfb64 | blowfish_ofb64 | aes_cbc128 | aes_cfb8 | aes_cfb128 | aes_cbc256 | aes_ige256 | aes_cbc | @@ -301,6 +301,9 @@ block_encrypt(Type, Key0, Ivec, Data) when Type =:= des3_cbc; block_encrypt(des3_cbf, Key0, Ivec, Data) -> Key = check_des3_key(Key0), block_crypt_nif(des_ede3_cbf, Key, Ivec, Data, true); +block_encrypt(des3_cfb, Key0, Ivec, Data) -> + Key = check_des3_key(Key0), + block_crypt_nif(des_ede3_cfb, Key, Ivec, Data, true); block_encrypt(aes_ige256, Key, Ivec, Data) -> aes_ige_crypt_nif(Key, Ivec, Data, true); block_encrypt(aes_gcm, Key, Ivec, {AAD, Data}) -> @@ -311,7 +314,7 @@ block_encrypt(chacha20_poly1305, Key, Ivec, {AAD, Data}) -> chacha20_poly1305_encrypt(Key, Ivec, AAD, Data). -spec block_decrypt(des_cbc | des_cfb | - des3_cbc | des3_cbf | des_ede3 | + des3_cbc | des3_cbf | des3_cfb | des_ede3 | blowfish_cbc | blowfish_cfb64 | blowfish_ofb64 | aes_cbc128 | aes_cfb8 | aes_cfb128 | aes_cbc256 | aes_ige256 | aes_cbc | @@ -338,6 +341,9 @@ block_decrypt(Type, Key0, Ivec, Data) when Type =:= des3_cbc; block_decrypt(des3_cbf, Key0, Ivec, Data) -> Key = check_des3_key(Key0), block_crypt_nif(des_ede3_cbf, Key, Ivec, Data, false); +block_decrypt(des3_cfb, Key0, Ivec, Data) -> + Key = check_des3_key(Key0), + block_crypt_nif(des_ede3_cfb, Key, Ivec, Data, false); block_decrypt(aes_ige256, Key, Ivec, Data) -> notsup_to_error(aes_ige_crypt_nif(Key, Ivec, Data, false)); block_decrypt(aes_gcm, Key, Ivec, {AAD, Data, Tag}) -> @@ -857,10 +863,10 @@ des_ede3_cbc_decrypt(Key1, Key2, Key3, IVec, Data) -> binary(). des3_cfb_encrypt(Key1, Key2, Key3, IVec, Data) -> - block_encrypt(des3_cbf, [Key1, Key2, Key3], IVec, Data). + block_encrypt(des3_cfb, [Key1, Key2, Key3], IVec, Data). des3_cfb_decrypt(Key1, Key2, Key3, IVec, Data) -> - block_decrypt(des3_cbf, [Key1, Key2, Key3], IVec, Data). + block_decrypt(des3_cfb, [Key1, Key2, Key3], IVec, Data). %% %% Blowfish diff --git a/lib/crypto/test/crypto_SUITE.erl b/lib/crypto/test/crypto_SUITE.erl index 6732f27824..7b07cef33f 100644 --- a/lib/crypto/test/crypto_SUITE.erl +++ b/lib/crypto/test/crypto_SUITE.erl @@ -50,6 +50,7 @@ all() -> {group, des_cfb}, {group, des3_cbc}, {group, des3_cbf}, + {group, des3_cfb}, {group, des_ede3}, {group, blowfish_cbc}, {group, blowfish_ecb}, @@ -94,6 +95,7 @@ groups() -> {des3_cbc,[], [block]}, {des_ede3,[], [block]}, {des3_cbf,[], [block]}, + {des3_cfb,[], [block]}, {rc2_cbc,[], [block]}, {aes_cbc128,[], [block]}, {aes_cfb8,[], [block]}, @@ -381,11 +383,8 @@ block_cipher({Type, Key, IV, PlainText, CipherText}) -> ct:fail({{crypto, block_decrypt, [Type, Key, IV, CipherText]}, {expected, Plain}, {got, Other1}}) end. -block_cipher_increment({Type, Key, IV, PlainTexts}) when Type == des_cbc; - Type == des3_cbc; - Type == aes_cbc; - Type == des_cbf - -> +block_cipher_increment({Type, Key, IV, PlainTexts}) + when Type == des_cbc; Type == aes_cbc; Type == des3_cbc -> block_cipher_increment(Type, Key, IV, IV, PlainTexts, iolist_to_binary(PlainTexts), []); block_cipher_increment({Type, Key, IV, PlainTexts, _CipherText}) when Type == aes_cbc -> Plain = iolist_to_binary(PlainTexts), @@ -582,6 +581,8 @@ do_block_iolistify({des3_cbc = Type, Key, IV, PlainText}) -> {Type, Key, IV, des_iolistify(PlainText)}; do_block_iolistify({des3_cbf = Type, Key, IV, PlainText}) -> {Type, Key, IV, des_iolistify(PlainText)}; +do_block_iolistify({des3_cfb = Type, Key, IV, PlainText}) -> + {Type, Key, IV, des_iolistify(PlainText)}; do_block_iolistify({des_ede3 = Type, Key, IV, PlainText}) -> {Type, Key, IV, des_iolistify(PlainText)}; do_block_iolistify({Type, Key, PlainText}) -> @@ -792,6 +793,9 @@ group_config(des3_cbc, Config) -> group_config(des3_cbf, Config) -> Block = des3_cbf(), [{block, Block} | Config]; +group_config(des3_cfb, Config) -> + Block = des3_cfb(), + [{block, Block} | Config]; group_config(des_ede3, Config) -> Block = des_ede3(), [{block, Block} | Config]; @@ -1193,7 +1197,16 @@ des_ede3() -> des3_cbf() -> [{des3_cbf, - [hexstr2bin("0123456789abcdef"), + [hexstr2bin("0123456789abcdef"), + hexstr2bin("fedcba9876543210"), + hexstr2bin("0f2d4b6987a5c3e1")], + hexstr2bin("1234567890abcdef"), + <<"Now is the time for all ">> + }]. + +des3_cfb() -> + [{des3_cfb, + [hexstr2bin("0123456789abcdef"), hexstr2bin("fedcba9876543210"), hexstr2bin("0f2d4b6987a5c3e1")], hexstr2bin("1234567890abcdef"), diff --git a/lib/crypto/test/old_crypto_SUITE.erl b/lib/crypto/test/old_crypto_SUITE.erl index 0d97290d10..4a6753b2ed 100644 --- a/lib/crypto/test/old_crypto_SUITE.erl +++ b/lib/crypto/test/old_crypto_SUITE.erl @@ -58,6 +58,7 @@ des_cfb_iter/1, des_ecb/1, des3_cbc/1, + des3_cbf/1, des3_cfb/1, rc2_cbc/1, aes_cfb/1, @@ -102,7 +103,7 @@ groups() -> hmac_rfc2202, hmac_rfc4231_sha224, hmac_rfc4231_sha256, hmac_rfc4231_sha384, hmac_rfc4231_sha512, des_cbc, aes_cfb, aes_cbc, - des_cfb, des_cfb_iter, des3_cbc, des3_cfb, rc2_cbc, + des_cfb, des_cfb_iter, des3_cbc, des3_cbf, des3_cfb, rc2_cbc, aes_cbc_iter, aes_ctr, aes_ctr_stream, des_cbc_iter, des_ecb, rand_uniform_test, strong_rand_test, rsa_verify_test, dsa_verify_test, rsa_sign_test, @@ -969,6 +970,9 @@ des_cbc(doc) -> des_cbc(suite) -> []; des_cbc(Config) when is_list(Config) -> + if_supported(des_cbc, fun des_cbc_do/0). + +des_cbc_do() -> ?line Key = hexstr2bin("0123456789abcdef"), ?line IVec = hexstr2bin("1234567890abcdef"), ?line Plain = "Now is the time for all ", @@ -992,6 +996,9 @@ des_cbc_iter(doc) -> des_cbc_iter(suite) -> []; des_cbc_iter(Config) when is_list(Config) -> + if_supported(des_cbc, fun des_cbc_iter_do/0). + +des_cbc_iter_do() -> ?line Key = hexstr2bin("0123456789abcdef"), ?line IVec = hexstr2bin("1234567890abcdef"), ?line Plain1 = "Now is the time ", @@ -1011,6 +1018,9 @@ des_cfb(doc) -> des_cfb(suite) -> []; des_cfb(Config) when is_list(Config) -> + if_supported(des_cfb, fun des_cfb_do/0). + +des_cfb_do() -> ?line Key = hexstr2bin("0123456789abcdef"), ?line IVec = hexstr2bin("1234567890abcdef"), ?line Plain = "Now is the", @@ -1027,6 +1037,9 @@ des_cfb_iter(doc) -> des_cfb_iter(suite) -> []; des_cfb_iter(Config) when is_list(Config) -> + if_supported(des_cfb, fun des_cfb_iter_do/0). + +des_cfb_iter_do() -> ?line Key = hexstr2bin("0123456789abcdef"), ?line IVec = hexstr2bin("1234567890abcdef"), ?line Plain1 = "Now i", @@ -1045,6 +1058,9 @@ des_ecb(doc) -> des_ecb(suite) -> []; des_ecb(Config) when is_list(Config) -> + if_supported(des_ecb, fun des_ecb_do/0). + +des_ecb_do() -> ?line Key = hexstr2bin("0123456789abcdef"), ?line Cipher1 = crypto:des_ecb_encrypt(Key, "Now is t"), ?line m(Cipher1, hexstr2bin("3fa40e8a984d4815")), @@ -1081,6 +1097,9 @@ des3_cbc(doc) -> des3_cbc(suite) -> []; des3_cbc(Config) when is_list(Config) -> + if_supported(des3_cbc, fun des3_cbc_do/0). + +des3_cbc_do() -> ?line Key1 = hexstr2bin("0123456789abcdef"), ?line Key2 = hexstr2bin("fedcba9876543210"), ?line Key3 = hexstr2bin("0f2d4b6987a5c3e1"), @@ -1112,6 +1131,19 @@ des3_cbc(Config) when is_list(Config) -> %% %% +des3_cbf(doc) -> + "Encrypt and decrypt according to CFB 3DES, and check the result."; +des3_cbf(suite) -> + []; +des3_cbf(Config) when is_list(Config) -> + case openssl_version() of + V when V < 16#90705F -> {skipped,"OpenSSL version too old"}; + _ -> + if_supported(des3_cbf, fun des3_cfb_do/0) + end. + +%% +%% des3_cfb(doc) -> "Encrypt and decrypt according to CFB 3DES, and check the result."; des3_cfb(suite) -> @@ -1119,7 +1151,8 @@ des3_cfb(suite) -> des3_cfb(Config) when is_list(Config) -> case openssl_version() of V when V < 16#90705F -> {skipped,"OpenSSL version too old"}; - _ -> des3_cfb_do() + _ -> + if_supported(des3_cfb, fun des3_cfb_do/0) end. des3_cfb_do() -> diff --git a/lib/crypto/vsn.mk b/lib/crypto/vsn.mk index 96466869d1..bbee24554a 100644 --- a/lib/crypto/vsn.mk +++ b/lib/crypto/vsn.mk @@ -1 +1 @@ -CRYPTO_VSN = 3.7 +CRYPTO_VSN = 3.7.1 diff --git a/lib/debugger/doc/src/notes.xml b/lib/debugger/doc/src/notes.xml index 2e0d834269..93bc46ddbe 100644 --- a/lib/debugger/doc/src/notes.xml +++ b/lib/debugger/doc/src/notes.xml @@ -33,6 +33,22 @@ <p>This document describes the changes made to the Debugger application.</p> +<section><title>Debugger 4.2.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + </list> + </section> + +</section> + <section><title>Debugger 4.2</title> <section><title>Improvements and New Features</title> diff --git a/lib/debugger/vsn.mk b/lib/debugger/vsn.mk index dd496013cd..f5440865ef 100644 --- a/lib/debugger/vsn.mk +++ b/lib/debugger/vsn.mk @@ -1 +1 @@ -DEBUGGER_VSN = 4.2 +DEBUGGER_VSN = 4.2.1 diff --git a/lib/dialyzer/doc/src/notes.xml b/lib/dialyzer/doc/src/notes.xml index 6400072b1f..b0f0a9aef0 100644 --- a/lib/dialyzer/doc/src/notes.xml +++ b/lib/dialyzer/doc/src/notes.xml @@ -32,6 +32,29 @@ <p>This document describes the changes made to the Dialyzer application.</p> +<section><title>Dialyzer 3.0.2</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> The translation of forms to types is improved for + opaque types in a few cases. </p> + <p> + Own Id: OTP-13682</p> + </item> + <item> + <p> Add warning suppression to compiler-generated case + statements. Warnings about clauses that cannot match and + are also compiler generated are suppressed unless none of + the clauses return. </p> + <p> + Own Id: OTP-13723 Aux Id: ERL-159, PR-1121 </p> + </item> + </list> + </section> + +</section> + <section><title>Dialyzer 3.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/dialyzer/test/map_SUITE_data/src/opaque_bif.erl b/lib/dialyzer/test/map_SUITE_data/src/opaque_bif.erl new file mode 100644 index 0000000000..40214a1887 --- /dev/null +++ b/lib/dialyzer/test/map_SUITE_data/src/opaque_bif.erl @@ -0,0 +1,13 @@ +-module(opaque_bif). +-export([o1/1]). +-export_type([opaque_any_map/0]). +-opaque opaque_any_map() :: map(). + +%% ERL-249: A bug with opaque arguments to maps:merge/2 +%% Reported by Felipe Ripoll on 6/9/2016 +-spec o1(opaque_any_map()) -> opaque_any_map(). +o1(Map) -> + maps:merge(o1_c(), Map). + +-spec o1_c() -> opaque_any_map(). +o1_c() -> #{}. diff --git a/lib/dialyzer/vsn.mk b/lib/dialyzer/vsn.mk index b9a28afdd9..6723876208 100644 --- a/lib/dialyzer/vsn.mk +++ b/lib/dialyzer/vsn.mk @@ -1 +1 @@ -DIALYZER_VSN = 3.0.1 +DIALYZER_VSN = 3.0.2 diff --git a/lib/diameter/doc/src/diameter.xml b/lib/diameter/doc/src/diameter.xml index d68a78ed6d..72181a42b0 100644 --- a/lib/diameter/doc/src/diameter.xml +++ b/lib/diameter/doc/src/diameter.xml @@ -94,12 +94,12 @@ in this module.</p> <taglist> -<tag><c>Address()</c></tag> -<tag><c>DiameterIdentity()</c></tag> -<tag><c>Grouped()</c></tag> -<tag><c>OctetString()</c></tag> -<tag><c>Time()</c></tag> -<tag><c>Unsigned32()</c></tag> +<tag><c>Address()</c></tag><item/> +<tag><c>DiameterIdentity()</c></tag><item/> +<tag><c>Grouped()</c></tag><item/> +<tag><c>OctetString()</c></tag><item/> +<tag><c>Time()</c></tag><item/> +<tag><c>Unsigned32()</c></tag><item/> <tag><c>UTF8String()</c></tag> <item> <p> @@ -159,8 +159,7 @@ Has one the following types.</p> <p> Unique identifier for the application in the scope of the service. -Defaults to the value of the <c>dictionary</c> option if -unspecified.</p> +Defaults to the value of the <c>dictionary</c> option.</p> </item> <tag><c>{dictionary, atom()}</c></tag> @@ -187,7 +186,7 @@ Initial callback state. The prevailing state is passed to some &man_app; callbacks, which can then return a new state. -Defaults to the value of the <c>alias</c> option if unspecified.</p> +Defaults to the value of the <c>alias</c> option.</p> </item> <tag><c>{call_mutates_state, true|false}</c></tag> @@ -195,7 +194,7 @@ Defaults to the value of the <c>alias</c> option if unspecified.</p> <p> Whether or not the &app_pick_peer; application callback can modify the application state. -Defaults to <c>false</c> if unspecified.</p> +Defaults to <c>false</c>.</p> <warning> <p> @@ -228,7 +227,7 @@ question is as if a callback had taken place and returned <c>{error, failure}</c>.</p> <p> -Defaults to <c>discard</c> if unspecified.</p> +Defaults to <c>discard</c>.</p> </item> <tag><c>{request_errors, answer_3xxx|answer|callback}</c></tag> @@ -249,7 +248,7 @@ place and its return value determines the answer sent to the peer, if any.</p> <p> -Defaults to <c>answer_3xxx</c> if unspecified.</p> +Defaults to <c>answer_3xxx</c>.</p> <note> <p> @@ -339,8 +338,8 @@ Has one of the following types.</p> <taglist> -<tag><c>{'Origin-Host', &dict_DiameterIdentity;}</c></tag> -<tag><c>{'Origin-Realm', &dict_DiameterIdentity;}</c></tag> +<tag><c>{'Origin-Host', &dict_DiameterIdentity;}</c></tag><item/> +<tag><c>{'Origin-Realm', &dict_DiameterIdentity;}</c></tag><item/> <tag><c>{'Host-IP-Address', [&dict_Address;]}</c></tag> <item> <p> @@ -352,8 +351,8 @@ question communicates an address list as described in &man_transport;</p> </item> -<tag><c>{'Vendor-Id', &dict_Unsigned32;}</c></tag> -<tag><c>{'Product-Name', &dict_UTF8String;}</c></tag> +<tag><c>{'Vendor-Id', &dict_Unsigned32;}</c></tag><item/> +<tag><c>{'Product-Name', &dict_UTF8String;}</c></tag><item/> <tag><c>{'Origin-State-Id', &dict_Unsigned32;}</c></tag> <item> <p> @@ -366,8 +365,8 @@ can be used as to retrieve a value that is computed when the diameter application is started.</p> </item> -<tag><c>{'Supported-Vendor-Id', [&dict_Unsigned32;]}</c></tag> -<tag><c>{'Auth-Application-Id', [&dict_Unsigned32;]}</c></tag> +<tag><c>{'Supported-Vendor-Id', [&dict_Unsigned32;]}</c></tag><item/> +<tag><c>{'Auth-Application-Id', [&dict_Unsigned32;]}</c></tag><item/> <tag><c>{'Inband-Security-Id', [&dict_Unsigned32;]}</c></tag> <item> <p> @@ -377,9 +376,9 @@ If 1 (TLS) is specified then TLS is selected if the CER/CEA received from the peer offers it.</p> </item> -<tag><c>{'Acct-Application-Id', [&dict_Unsigned32;]}</c></tag> -<tag><c>{'Vendor-Specific-Application-Id', [&dict_Grouped;]}</c></tag> -<tag><c>{'Firmware-Revision', &dict_Unsigned32;}</c></tag> +<tag><c>{'Acct-Application-Id', [&dict_Unsigned32;]}</c></tag><item/> +<tag><c>{'Vendor-Specific-Application-Id', [&dict_Grouped;]}</c></tag><item/> +<tag><c>{'Firmware-Revision', &dict_Unsigned32;}</c></tag><item/> </taglist> @@ -567,9 +566,8 @@ Can have one of the following types.</p> <taglist> -<tag><c>start</c></tag> +<tag><c>start</c></tag><item/> <tag><c>stop</c></tag> - <item> <p> The service is being started or stopped. @@ -578,8 +576,8 @@ No event follows a <c>stop</c> event, and this event implies the termination of all transport processes.</p> </item> -<tag><c>{up, Ref, Peer, Config, Pkt}</c></tag> -<tag><c>{up, Ref, Peer, Config}</c></tag> +<tag><c>{up, Ref, Peer, Config, Pkt}</c></tag><item/> +<tag><c>{up, Ref, Peer, Config}</c></tag><item/> <tag><c>{down, Ref, Peer, Config}</c></tag> <item> <pre> @@ -788,8 +786,8 @@ be matched by corresponding &capability; configuration, of </item> -<marker id="incoming_maxlen"/> -<tag><c>{incoming_maxlen, 0..16777215}</c></tag> +<tag> +<marker id="incoming_maxlen"/><c>{incoming_maxlen, 0..16777215}</c></tag> <item> <p> Bound on the expected size of incoming Diameter messages. @@ -917,8 +915,8 @@ Options <c>monitor</c> and <c>link</c> are ignored.</p> Defaults to the empty list.</p> </item> -<marker id="strict_mbit"/> -<tag><c>{strict_mbit, boolean()}</c></tag> +<tag> +<marker id="strict_mbit"/><c>{strict_mbit, boolean()}</c></tag> <item> <p> Whether or not to regard an AVP setting the M-bit as erroneous when @@ -935,7 +933,7 @@ Defaults to <c>true</c>.</p> <p> RFC 6733 is unclear about the semantics of the M-bit. One the one hand, the CCF specification in section 3.2 documents AVP -in a command grammar as meaning <b>any</b> arbitrary AVP; on the +in a command grammar as meaning <em>any</em> arbitrary AVP; on the other hand, 1.3.4 states that AVPs setting the M-bit cannot be added to an existing command: the modified command must instead be placed in a new Diameter application.</p> @@ -945,7 +943,7 @@ allowing arbitrary AVPs setting the M-bit in a command makes its interpretation implementation-dependent, since there's no guarantee that all implementations will understand the same set of arbitrary AVPs in the context of a given command. -However, interpreting <c>AVP</c> in a command grammar as <b>any</b> +However, interpreting <c>AVP</c> in a command grammar as any AVP, regardless of M-bit, renders 1.3.4 meaningless, since the receiver can simply ignore any AVP it thinks isn't relevant, regardless of the sender's intent.</p> @@ -960,8 +958,8 @@ occur in the message in question.</p> </item> -<marker id="string_decode"/> -<tag><c>{string_decode, boolean()}</c></tag> +<tag> +<marker id="string_decode"/><c>{string_decode, boolean()}</c></tag> <item> <p> Whether or not to decode AVPs of type &dict_OctetString; and its @@ -1031,8 +1029,9 @@ Option passed to &add_transport;. Has one of the following types.</p> <taglist> -<marker id="applications"/> -<tag><c>{applications, [&application_alias;]}</c></tag> + +<tag> +<marker id="applications"/><c>{applications, [&application_alias;]}</c></tag> <item> <p> Diameter applications to which the transport should be restricted. @@ -1050,8 +1049,8 @@ implies having to set matching *-Application-Id AVPs in a </item> -<marker id="capabilities"/> -<tag><c>{capabilities, [&capability;]}</c></tag> +<tag> +<marker id="capabilities"/><c>{capabilities, [&capability;]}</c></tag> <item> <p> AVPs used to construct outgoing CER/CEA messages. @@ -1064,8 +1063,8 @@ may be particularly appropriate for Inband-Security-Id, in case TLS is desired over TCP as implemented by &man_tcp;.</p> </item> -<marker id="capabilities_cb"/> -<tag><c>{capabilities_cb, &evaluable;}</c></tag> +<tag> +<marker id="capabilities_cb"/><c>{capabilities_cb, &evaluable;}</c></tag> <item> <p> Callback invoked upon reception of CER/CEA during capabilities @@ -1112,8 +1111,8 @@ case the corresponding callbacks are applied until either all return <c>ok</c> or one does not.</p> </item> -<marker id="capx_timeout"/> -<tag><c>{capx_timeout, &dict_Unsigned32;}</c></tag> +<tag> +<marker id="capx_timeout"/><c>{capx_timeout, &dict_Unsigned32;}</c></tag> <item> <p> Number of milliseconds after which a transport process having an @@ -1127,8 +1126,8 @@ For a listening transport, the peer determines the timing.</p> Defaults to 10000.</p> </item> -<marker id="connect_timer"/> -<tag><c>{connect_timer, Tc}</c></tag> +<tag> +<marker id="connect_timer"/><c>{connect_timer, Tc}</c></tag> <item> <pre> Tc = &dict_Unsigned32; @@ -1158,9 +1157,8 @@ Defaults to 30000 for a connecting transport and 60000 for a listening transport.</p> </item> -<marker id="disconnect_cb"/> -<tag><c>{disconnect_cb, &evaluable;}</c></tag> - +<tag> +<marker id="disconnect_cb"/><c>{disconnect_cb, &evaluable;}</c></tag> <item> <p> Callback invoked prior to terminating the transport process of a @@ -1236,8 +1234,8 @@ configured them.</p> Defaults to a single callback returning <c>dpr</c>.</p> </item> -<marker id="dpa_timeout"/> -<tag><c>{dpa_timeout, &dict_Unsigned32;}</c></tag> +<tag> +<marker id="dpa_timeout"/><c>{dpa_timeout, &dict_Unsigned32;}</c></tag> <item> <p> Number of milliseconds after which a transport connection is @@ -1247,8 +1245,8 @@ terminated following an outgoing DPR if DPA is not received.</p> Defaults to 1000.</p> </item> -<marker id="dpr_timeout"/> -<tag><c>{dpr_timeout, &dict_Unsigned32;}</c></tag> +<tag> +<marker id="dpr_timeout"/><c>{dpr_timeout, &dict_Unsigned32;}</c></tag> <item> <p> Number of milliseconds after which a transport connection is @@ -1259,8 +1257,8 @@ connection.</p> Defaults to 5000.</p> </item> -<marker id="length_errors"/> -<tag><c>{length_errors, exit|handle|discard}</c></tag> +<tag> +<marker id="length_errors"/><c>{length_errors, exit|handle|discard}</c></tag> <item> <p> How to deal with errors in the Message Length field of the @@ -1308,8 +1306,8 @@ the same peer.</p> </item> -<marker id="spawn_opt"/> -<tag><c>{spawn_opt, [term()]}</c></tag> +<tag> +<marker id="spawn_opt"/><c>{spawn_opt, [term()]}</c></tag> <item> <p> Options passed to &spawn_opt; when spawning a process for an @@ -1320,15 +1318,15 @@ Options <c>monitor</c> and <c>link</c> are ignored.</p> Defaults to the list configured on the service if not specified.</p> </item> -<marker id="transport_config"/> -<tag><c>{transport_config, term()}</c></tag> +<tag> +<marker id="transport_config"/><c>{transport_config, term()}</c></tag><item/> <tag><c>{transport_config, term(), &dict_Unsigned32; | infinity}</c></tag> <item> <p> Term passed as the third argument to the &transport_start; function of the relevant &transport_module; in order to start a transport process. -Defaults to the empty list if unspecified.</p> +Defaults to the empty list.</p> <p> The 3-tuple form additionally specifies an interval, in milliseconds, @@ -1349,12 +1347,12 @@ request a connection with one peer over SCTP or another To listen on both SCTP and TCP, define one transport for each.</p> </item> -<marker id="transport_module"/> -<tag><c>{transport_module, atom()}</c></tag> +<tag> +<marker id="transport_module"/><c>{transport_module, atom()}</c></tag> <item> <p> Module implementing a transport process as defined in &man_transport;. -Defaults to <c>diameter_tcp</c> if unspecified.</p> +Defaults to <c>diameter_tcp</c>.</p> <p> Multiple <c>transport_module</c> and &transport_config; @@ -1369,8 +1367,9 @@ modules in order until one establishes a connection within the corresponding timeout (see below) or all fail.</p> </item> -<marker id="watchdog_config"/> -<tag><c>{watchdog_config, [{okay|suspect, non_neg_integer()}]}</c></tag> +<tag> +<marker id="watchdog_config"/><c>{watchdog_config, + [{okay|suspect, non_neg_integer()}]}</c></tag> <item> <p> Configuration that alters the behaviour of the watchdog @@ -1393,8 +1392,8 @@ misbehaving nodes during test.</p> </warning> </item> -<marker id="watchdog_timer"/> -<tag><c>{watchdog_timer, TwInit}</c></tag> +<tag> +<marker id="watchdog_timer"/><c>{watchdog_timer, TwInit}</c></tag> <item> <pre> TwInit = &dict_Unsigned32; @@ -1412,7 +1411,7 @@ the callback.</p> <p> An integer value must be at least 6000 as required by RFC 3539. -Defaults to 30000 if unspecified.</p> +Defaults to 30000.</p> </item> </taglist> @@ -1422,10 +1421,10 @@ Unrecognized options are silently ignored but are returned unmodified by &service_info; and can be referred to in predicate functions passed to &remove_transport;.</p> -<marker id="transport_ref"/> </item> -<tag><c>transport_ref() = reference()</c></tag> +<tag> +<marker id="transport_ref"/><c>transport_ref() = reference()</c></tag> <item> <p> Reference returned by &add_transport; that @@ -1682,17 +1681,17 @@ returned.</p> <taglist> -<tag><c>'Origin-Host'</c></tag> -<tag><c>'Origin-Realm'</c></tag> -<tag><c>'Vendor-Id'</c></tag> -<tag><c>'Product-Name'</c></tag> -<tag><c>'Origin-State-Id'</c></tag> -<tag><c>'Host-IP-Address'</c></tag> -<tag><c>'Supported-Vendor'</c></tag> -<tag><c>'Auth-Application-Id'</c></tag> -<tag><c>'Inband-Security-Id'</c></tag> -<tag><c>'Acct-Application-Id'</c></tag> -<tag><c>'Vendor-Specific-Application-Id'</c></tag> +<tag><c>'Origin-Host'</c></tag><item/> +<tag><c>'Origin-Realm'</c></tag><item/> +<tag><c>'Vendor-Id'</c></tag><item/> +<tag><c>'Product-Name'</c></tag><item/> +<tag><c>'Origin-State-Id'</c></tag><item/> +<tag><c>'Host-IP-Address'</c></tag><item/> +<tag><c>'Supported-Vendor'</c></tag><item/> +<tag><c>'Auth-Application-Id'</c></tag><item/> +<tag><c>'Inband-Security-Id'</c></tag><item/> +<tag><c>'Acct-Application-Id'</c></tag><item/> +<tag><c>'Vendor-Specific-Application-Id'</c></tag><item/> <tag><c>'Firmware-Revision'</c></tag> <item> <p> diff --git a/lib/diameter/doc/src/diameter_app.xml b/lib/diameter/doc/src/diameter_app.xml index 973b6eb732..dfcd00975b 100644 --- a/lib/diameter/doc/src/diameter_app.xml +++ b/lib/diameter/doc/src/diameter_app.xml @@ -90,17 +90,14 @@ is called in response to an incoming Diameter request message.</p> </list> -</description> - -<note> <p> -The arities given for the the callback functions here assume no extra -arguments. +The arities for the the callback functions here assume no extra arguments. All functions will also be passed any extra arguments configured with the callback module itself when calling &mod_start_service; and, for the call-specific callbacks, any extra arguments passed to &mod_call;.</p> -</note> + +</description> <!-- ===================================================================== --> <!-- ===================================================================== --> @@ -110,9 +107,8 @@ and, for the call-specific callbacks, any extra arguments passed to <taglist> -<marker id="capabilities"/> - -<tag><c>capabilities() = #diameter_caps{}</c></tag> +<tag> +<marker id="capabilities"/><c>capabilities() = #diameter_caps{}</c></tag> <item> <p> A record containing the identities of @@ -126,9 +122,8 @@ Optional or possibly multiple values are encoded as lists of values, mandatory values as the bare value.</p> </item> -<marker id="message"/> - -<tag><c>message() = &codec_message;</c></tag> +<tag> +<marker id="message"/><c>message() = &codec_message;</c></tag> <item> <p> The representation of a Diameter message as passed to @@ -136,9 +131,8 @@ The representation of a Diameter message as passed to </item> -<marker id="packet"/> - -<tag><c>packet() = &codec_packet;</c></tag> +<tag> +<marker id="packet"/><c>packet() = &codec_packet;</c></tag> <item> <p> A container for incoming and outgoing Diameter messages that's passed @@ -146,25 +140,22 @@ through encode/decode and transport. Fields should not be set in return values except as documented.</p> </item> -<marker id="peer_ref"/> - -<tag><c>peer_ref() = term()</c></tag> +<tag> +<marker id="peer_ref"/><c>peer_ref() = term()</c></tag> <item> <p> A term identifying a transport connection with a Diameter peer.</p> </item> -<marker id="peer"/> - -<tag><c>peer() = {&peer_ref;, &capabilities;}</c></tag> +<tag> +<marker id="peer"/><c>peer() = {&peer_ref;, &capabilities;}</c></tag> <item> <p> A tuple representing a Diameter peer connection.</p> </item> -<marker id="state"/> - -<tag><c>state() = term()</c></tag> +<tag> +<marker id="state"/><c>state() = term()</c></tag> <item> <p> The state maintained by the application callback functions diff --git a/lib/diameter/doc/src/diameter_codec.xml b/lib/diameter/doc/src/diameter_codec.xml index a0313e2877..91e96058dd 100644 --- a/lib/diameter/doc/src/diameter_codec.xml +++ b/lib/diameter/doc/src/diameter_codec.xml @@ -88,10 +88,9 @@ files resulting from dictionary file compilation.</p> <taglist> -<marker id="integers"/> - -<tag><c>uint8() = 0..255</c></tag> -<tag><c>uint24() = 0..16777215</c></tag> +<tag> +<marker id="integers"/><c>uint8() = 0..255</c></tag><item/> +<tag><c>uint24() = 0..16777215</c></tag><item/> <tag><c>uint32() = 0..4294967295</c></tag> <item> <p> @@ -99,9 +98,8 @@ files resulting from dictionary file compilation.</p> headers.</p> </item> -<marker id="avp"/> - -<tag><c>avp() = #diameter_avp{}</c></tag> +<tag> +<marker id="avp"/><c>avp() = #diameter_avp{}</c></tag> <item> <p> The application-neutral representation of an AVP. @@ -116,9 +114,9 @@ Fields have the following types.</p> <taglist> -<tag><c>code = uint32()</c></tag> -<tag><c>is_mandatory = boolean()</c></tag> -<tag><c>need_encryption = boolean()</c></tag> +<tag><c>code = uint32()</c></tag><item/> +<tag><c>is_mandatory = boolean()</c></tag><item/> +<tag><c>need_encryption = boolean()</c></tag><item/> <tag><c>vendor_id = uint32() | undefined</c></tag> <item> <p> @@ -167,9 +165,8 @@ Possible types are <c>undefined</c> and the Diameter types: </item> -<marker id="dictionary"/> - -<tag><c>dictionary() = module()</c></tag> +<tag> +<marker id="dictionary"/><c>dictionary() = module()</c></tag> <item> <p> @@ -179,9 +176,8 @@ The interface provided by a dictionary module is an implementation detail that may change.</p> </item> -<marker id="header"/> - -<tag><c>header() = #diameter_header{}</c></tag> +<tag> +<marker id="header"/><c>header() = #diameter_header{}</c></tag> <item> <p> The record representation of the Diameter header. @@ -204,11 +200,11 @@ Fields have the following types.</p> <taglist> -<tag><c>version = uint8()</c></tag> -<tag><c>length = uint24()</c></tag> -<tag><c>cmd_code = uint24()</c></tag> -<tag><c>application_id = uint32()</c></tag> -<tag><c>hop_by_hop_id = uint32()</c></tag> +<tag><c>version = uint8()</c></tag><item/> +<tag><c>length = uint24()</c></tag><item/> +<tag><c>cmd_code = uint24()</c></tag><item/> +<tag><c>application_id = uint32()</c></tag><item/> +<tag><c>hop_by_hop_id = uint32()</c></tag><item/> <tag><c>end_to_end_id = uint32()</c></tag> <item> <p> @@ -217,9 +213,9 @@ Hop-by-Hop Identifier and End-to-End Identifier fields of the Diameter header.</p> </item> -<tag><c>is_request = boolean()</c></tag> -<tag><c>is_proxiable = boolean()</c></tag> -<tag><c>is_error = boolean()</c></tag> +<tag><c>is_request = boolean()</c></tag><item/> +<tag><c>is_proxiable = boolean()</c></tag><item/> +<tag><c>is_error = boolean()</c></tag><item/> <tag><c>is_retransmitted = boolean()</c></tag> <item> <p> @@ -232,9 +228,8 @@ header.</p> </item> -<marker id="message"/> - -<tag><c>message() = record() | list()</c></tag> +<tag> +<marker id="message"/><c>message() = record() | list()</c></tag> <item> <p> The representation of a Diameter message as passed to @@ -257,9 +252,8 @@ question: messages are sent exactly as specified.</p> </item> -<marker id="packet"/> - -<tag><c>packet() = #diameter_packet{}</c></tag> +<tag> +<marker id="packet"/><c>packet() = #diameter_packet{}</c></tag> <item> <p> A container for incoming and outgoing Diameter messages. @@ -296,7 +290,7 @@ corresponding values.</p> <warning> <p> -A record-valued <c>msg</c> field does <b>not</b> imply an absence of +A record-valued <c>msg</c> field does <em>not</em> imply an absence of decode errors. The <c>errors</c> field should also be examined.</p> </warning> diff --git a/lib/diameter/doc/src/diameter_dict.xml b/lib/diameter/doc/src/diameter_dict.xml index ae40f99aee..9584d682c2 100644 --- a/lib/diameter/doc/src/diameter_dict.xml +++ b/lib/diameter/doc/src/diameter_dict.xml @@ -1,5 +1,5 @@ <?xml version="1.0" encoding="utf-8" ?> -<!DOCTYPE erlref SYSTEM "fileref.dtd" [ +<!DOCTYPE fileref SYSTEM "fileref.dtd" [ <!ENTITY format '<seealso marker="#FILE_FORMAT">FILE FORMAT</seealso>'> <!ENTITY records @@ -121,9 +121,8 @@ The order in which sections are specified is unimportant.</p> <taglist> -<marker id="id"/> - -<tag><c>@id Number</c></tag> +<tag> +<marker id="id"/><c>@id Number</c></tag> <item> <p> Defines the integer Number as the Diameter Application Id of the @@ -146,14 +145,13 @@ Example:</p> </item> -<marker id="name"/> - -<tag><c>@name Mod</c></tag> +<tag> +<marker id="name"/><c>@name Mod</c></tag> <item> <p> Defines the name of the generated dictionary module. Can occur at most once and defaults to the name of the dictionary file -minus any extension if unspecified. +minus any extension. The section has empty content.</p> <p> @@ -169,9 +167,8 @@ Example:</p> </item> -<marker id="prefix"/> - -<tag><c>@prefix Name</c></tag> +<tag> +<marker id="prefix"/><c>@prefix Name</c></tag> <item> <p> Defines Name as the prefix to be added to record and constant names @@ -194,9 +191,8 @@ Example:</p> </item> -<marker id="vendor"/> - -<tag><c>@vendor Number Name</c></tag> +<tag> +<marker id="vendor"/><c>@vendor Number Name</c></tag> <item> <p> Defines the integer Number as the the default Vendor-Id of AVPs for @@ -216,9 +212,8 @@ Example:</p> </item> -<marker id="avp_vendor_id"/> - -<tag><c>@avp_vendor_id Number</c></tag> +<tag> +<marker id="avp_vendor_id"/><c>@avp_vendor_id Number</c></tag> <item> <p> Defines the integer Number as the Vendor-Id of the AVPs listed in the @@ -238,9 +233,8 @@ Region-Set </item> -<marker id="inherits"/> - -<tag><c>@inherits Mod</c></tag> +<tag> +<marker id="inherits"/><c>@inherits Mod</c></tag> <item> <p> Defines the name of a dictionary module containing AVP @@ -274,9 +268,8 @@ Example:</p> </pre> </item> -<marker id="avp_types"/> - -<tag><c>@avp_types</c></tag> +<tag> +<marker id="avp_types"/><c>@avp_types</c></tag> <item> <p> Defines the name, code, type and flags of individual AVPs. @@ -308,9 +301,8 @@ The P flag has been deprecated by &the_rfc;.</p> </item> -<marker id="custom_types"/> - -<tag><c>@custom_types Mod</c></tag> +<tag> +<marker id="custom_types"/><c>@custom_types Mod</c></tag> <item> <p> Specifies AVPs for which module Mod provides encode/decode functions. @@ -331,9 +323,8 @@ Framed-IP-Address </pre> </item> -<marker id="codecs"/> - -<tag><c>@codecs Mod</c></tag> +<tag> +<marker id="codecs"/><c>@codecs Mod</c></tag> <item> <p> Like <c>@custom_types</c> but requires the specified module to export @@ -350,9 +341,8 @@ Framed-IP-Address </pre> </item> -<marker id="messages"/> - -<tag><c>@messages</c></tag> +<tag> +<marker id="messages"/><c>@messages</c></tag> <item> <p> Defines the messages of the application. @@ -397,9 +387,8 @@ RTA ::= < Diameter Header: 287, PXY > </item> -<marker id="grouped"/> - -<tag><c>@grouped</c></tag> +<tag> +<marker id="grouped"/><c>@grouped</c></tag> <item> <p> Defines the contents of the AVPs of the application having type @@ -424,9 +413,8 @@ Specifying a Vendor-Id in the definition of a grouped AVP is equivalent to specifying it with <c>@avp_vendor_id</c>.</p> </item> -<marker id="enum"/> - -<tag><c>@enum Name</c></tag> +<tag> +<marker id="enum"/><c>@enum Name</c></tag> <item> <p> Defines values of AVP Name having type Enumerated. @@ -452,9 +440,8 @@ REMOVE_SIP_SERVER 3 </pre> </item> -<marker id="end"/> - -<tag><c>@end</c></tag> +<tag> +<marker id="end"/><c>@end</c></tag> <item> <p> Causes parsing of the dictionary to terminate: diff --git a/lib/diameter/doc/src/diameter_examples.xml b/lib/diameter/doc/src/diameter_examples.xml index 853ef96bb3..2e1f2b3c03 100644 --- a/lib/diameter/doc/src/diameter_examples.xml +++ b/lib/diameter/doc/src/diameter_examples.xml @@ -42,4 +42,3 @@ Example code can be found in the diameter application's <c>examples</c> subdirectory.</p> </chapter> - diff --git a/lib/diameter/doc/src/diameter_tcp.xml b/lib/diameter/doc/src/diameter_tcp.xml index 00ccc39c15..6ca280c52b 100644 --- a/lib/diameter/doc/src/diameter_tcp.xml +++ b/lib/diameter/doc/src/diameter_tcp.xml @@ -150,7 +150,7 @@ Options <c>binary</c>, <c>packet</c> and <c>active</c> cannot be specified. Also, option <c>port</c> can be specified for a listening transport to specify the local listening port, the default being the standardized -3868 if unspecified. +3868. Note that the option <c>ip</c> specifies the local address.</p> <p> diff --git a/lib/diameter/doc/src/diameter_transport.xml b/lib/diameter/doc/src/diameter_transport.xml index 736d4cbfbd..294e8a8864 100644 --- a/lib/diameter/doc/src/diameter_transport.xml +++ b/lib/diameter/doc/src/diameter_transport.xml @@ -65,9 +65,8 @@ parent).</p> <taglist> -<marker id="message"/> - -<tag><c>message() = binary() | &codec_packet;</c></tag> +<tag> +<marker id="message"/><c>message() = binary() | &codec_packet;</c></tag> <item> <p> A Diameter message as passed over the transport interface.</p> @@ -160,9 +159,9 @@ It should exit if its transport connection with its peer is lost.</p> </funcs> <!-- ===================================================================== --> -<marker id="MESSAGES"/> <section> +<marker id="MESSAGES"/> <title>MESSAGES</title> <p> @@ -234,7 +233,7 @@ established a connection with the peer. Not sent if the transport process has <c>Type=connect</c>.</p> </item> -<tag><c>{diameter, {self(), connected, Remote}}</c></tag> +<tag><c>{diameter, {self(), connected, Remote}}</c></tag><item/> <tag><c>{diameter, {self(), connected, Remote, [LocalAddr]}}</c></tag> <item> <p> diff --git a/lib/diameter/doc/src/diameterc.xml b/lib/diameter/doc/src/diameterc.xml index 5bffe9a771..8f1c660989 100644 --- a/lib/diameter/doc/src/diameterc.xml +++ b/lib/diameter/doc/src/diameterc.xml @@ -11,7 +11,7 @@ <comref> <header> <copyright> -<year>2011</year><year>2013</year> +<year>2011</year><year>2016</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -79,14 +79,14 @@ Write generated source to the specified directory. Defaults to the current working directory.</p> </item> -<tag><![CDATA[-E]]></tag> +<tag><![CDATA[-E]]></tag><item/> <tag><![CDATA[-H]]></tag> <item> <p> Suppress erl and hrl generation, respectively.</p> </item> -<tag><![CDATA[--name <name>]]></tag> +<tag><![CDATA[--name <name>]]></tag><item/> <tag><![CDATA[--prefix <prefix>]]></tag> <item> <p> diff --git a/lib/diameter/doc/src/notes.xml b/lib/diameter/doc/src/notes.xml index b1be7bdcf7..c2bbed2e5a 100644 --- a/lib/diameter/doc/src/notes.xml +++ b/lib/diameter/doc/src/notes.xml @@ -43,6 +43,31 @@ first.</p> <!-- ===================================================================== --> +<section><title>diameter 1.12.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Close diameter_tcp/sctp listening sockets at + diameter:stop_service/1.</p> + <p> + Broken by OTP-13611.</p> + <p> + Own Id: OTP-13787 Aux Id: OTP-13611 </p> + </item> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + </list> + </section> + +</section> + <section><title>diameter 1.12</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/diameter/examples/code/relay.erl b/lib/diameter/examples/code/relay.erl index 3846b1d161..cf4ce8848b 100644 --- a/lib/diameter/examples/code/relay.erl +++ b/lib/diameter/examples/code/relay.erl @@ -53,7 +53,7 @@ {'Auth-Application-Id', [16#FFFFFFFF]}, {string_decode, false}, {application, [{alias, relay}, - {dictionary, diameter_relay}, + {dictionary, diameter_gen_relay}, {module, relay_cb}]}]). %% start/1 diff --git a/lib/diameter/src/base/diameter_lib.erl b/lib/diameter/src/base/diameter_lib.erl index b835e87967..3928769b5e 100644 --- a/lib/diameter/src/base/diameter_lib.erl +++ b/lib/diameter/src/base/diameter_lib.erl @@ -299,8 +299,28 @@ spawn_opts(server, Opts) -> spawn_opts(worker, Opts) -> opts(5000, Opts). -opts(HeapSize, Opts) -> - [{min_heap_size, HeapSize} | lists:keydelete(min_heap_size, 1, Opts)]. +%% These setting are historical rather than useful. In particular, the +%% server setting can bloat many processes unnecessarily. Let them be +%% disabled with -diameter min_heap_size false. + +opts(Def, Opts) -> + Key = min_heap_size, + case getenv(Key, Def) of + N when is_integer(N), 0 =< N -> + [{Key, N} | lists:keydelete(Key, 1, Opts)]; + _ -> + Opts + end. + +%% getenv/1 + +getenv(Key, Def) -> + case application:get_env(Key) of + {ok, T} -> + T; + undefined -> + Def + end. %% --------------------------------------------------------------------------- %% # wait/1 diff --git a/lib/diameter/src/base/diameter_traffic.erl b/lib/diameter/src/base/diameter_traffic.erl index 2112941d5e..d93a3e71e3 100644 --- a/lib/diameter/src/base/diameter_traffic.erl +++ b/lib/diameter/src/base/diameter_traffic.erl @@ -123,8 +123,16 @@ peer_up(TPid) -> %% --------------------------------------------------------------------------- peer_down(TPid) -> - ets:delete(?REQUEST_TABLE, TPid), - failover(TPid). + ets:delete_object(?REQUEST_TABLE, {TPid}), + lists:foreach(fun failover/1, ets:lookup(?REQUEST_TABLE, TPid)). +%% Note that a request process can store its request after failover +%% notifications are sent here: insert_request/2 sends the notification +%% in that case. + +%% failover/1 + +failover({_TPid, {Pid, TRef}}) -> + Pid ! {failover, TRef}. %% --------------------------------------------------------------------------- %% incr/4 @@ -911,7 +919,7 @@ failed(Rec, FailedAvp, Dict) -> {'Failed-AVP', [FailedAvp]} catch error: _ -> - Avps = Dict:'get-'('AVP', Rec), + Avps = Dict:'#get-'('AVP', Rec), A = #diameter_avp{name = 'Failed-AVP', value = FailedAvp}, {'AVP', [A|Avps]} @@ -1452,7 +1460,7 @@ make_request_packet(#diameter_packet{header = Hdr} = Pkt, make_request_packet(Msg, Pkt) -> Pkt#diameter_packet{msg = Msg}. -%% make_retransmit_packet/2 +%% make_retransmit_packet/1 make_retransmit_packet(#diameter_packet{msg = [#diameter_header{} = Hdr | Avps]} @@ -1703,16 +1711,13 @@ send_request(TPid, #diameter_packet{bin = Bin} = Pkt, Req, _SvcName, Timeout) when node() == node(TPid) -> Seqs = diameter_codec:sequence_numbers(Bin), TRef = erlang:start_timer(Timeout, self(), TPid), - Entry = {Seqs, Req, TRef}, + Entry = {Seqs, #request{handler = Pid} = Req, TRef}, - %% Ensure that request table is cleaned even if we receive an exit - %% signal. An alternative would be to simply trap exits, but - %% callbacks are applied in this process, and these could possibly - %% be expecting the prevailing behaviour. - Self = self(), - spawn(fun() -> diameter_lib:wait([Self]), erase_request(Entry) end), + %% Ensure that request table is cleaned even if the process is + %% killed. + spawn(fun() -> diameter_lib:wait([Pid]), delete_request(Entry) end), - store_request(Entry, TPid), + insert_request(Entry), send(TPid, Pkt), TRef; @@ -1774,6 +1779,8 @@ retransmit({TPid, Caps, App} SvcName, Timeout, []). +%% When sending a binary, it's up to prepare_retransmit to modify it +%% accordingly. retransmit({send, Msg}, Transport, @@ -1818,15 +1825,21 @@ resend_request(Pkt0, TRef = send_request(TPid, Pkt, Req, SvcName, Tmo), {TRef, Req}. -%% store_request/2 +%% insert_request/1 -store_request(T, TPid) -> - ets:insert(?REQUEST_TABLE, T), - ets:member(?REQUEST_TABLE, TPid) - orelse begin - {_Seqs, _Req, TRef} = T, - self() ! {failover, TRef} %% failover/1 may have missed - end. +insert_request({_Seqs, #request{transport = TPid}, TRef} = T) -> + ets:insert(?REQUEST_TABLE, [T, {TPid, {self(), TRef}}]), + is_peer_up(TPid) + orelse (self() ! {failover, TRef}). %% failover/1 may have missed + +%% is_peer_up/1 +%% +%% Is the entry written by peer_up/1 and deleted by peer_down/1 still +%% in the request table? + +is_peer_up(TPid) -> + Spec = [{{TPid}, [], ['$_']}], + '$end_of_table' /= ets:select(?REQUEST_TABLE, Spec, 1). %% lookup_request/2 %% @@ -1846,16 +1859,11 @@ lookup_request(Msg, TPid) -> false end. -%% erase_request/1 - -erase_request(T) -> - ets:delete_object(?REQUEST_TABLE, T). - -%% match_requests/1 +%% delete_request/1 -match_requests(TPid) -> - Pat = {'_', #request{transport = TPid, _ = '_'}, '_'}, - ets:select(?REQUEST_TABLE, [{Pat, [], ['$_']}]). +delete_request({_Seqs, #request{handler = Pid, transport = TPid}, TRef} = T) -> + Spec = [{R, [], [true]} || R <- [T, {TPid, {Pid, TRef}}]], + ets:select_delete(?REQUEST_TABLE, Spec). %% have_request/2 @@ -1864,28 +1872,6 @@ have_request(Pkt, TPid) -> Pat = {Seqs, #request{transport = TPid, _ = '_'}, '_'}, '$end_of_table' /= ets:select(?REQUEST_TABLE, [{Pat, [], ['$_']}], 1). -%% --------------------------------------------------------------------------- -%% # failover/1-2 -%% --------------------------------------------------------------------------- - -failover(TPid) - when is_pid(TPid) -> - lists:foreach(fun failover/1, match_requests(TPid)); -%% Note that a request process can store its request after failover -%% notifications are sent here: store_request/2 sends the notification -%% in that case. - -%% Failover as a consequence of peer_down/1: inform the -%% request process. -failover({_, Req, TRef}) -> - #request{handler = Pid, - packet = #diameter_packet{msg = M}} - = Req, - M /= undefined andalso (Pid ! {failover, TRef}). -%% Failover is not performed when msg = binary() since sending -%% pre-encoded binaries is only partially supported. (Mostly for -%% test.) - %% get_destination/2 get_destination(Dict, Msg) -> diff --git a/lib/diameter/src/compiler/diameter_codegen.erl b/lib/diameter/src/compiler/diameter_codegen.erl index cdaa9aa7f9..4007d6b7b1 100644 --- a/lib/diameter/src/compiler/diameter_codegen.erl +++ b/lib/diameter/src/compiler/diameter_codegen.erl @@ -790,20 +790,7 @@ header() -> ("%% -------------------------------------------------------------------\n" "%% This is a generated file.\n" "%% -------------------------------------------------------------------\n" - "\n" - "%%\n" - "%% Copyright (c) Ericsson AB. All rights reserved.\n" - "%%\n" - "%% The information in this document is the property of Ericsson.\n" - "%%\n" - "%% Except as specifically authorized in writing by Ericsson, the\n" - "%% receiver of this document shall keep the information contained\n" - "%% herein confidential and shall protect the same in whole or in\n" - "%% part from disclosure and dissemination to third parties.\n" - "%%\n" - "%% Disclosure and disseminations to the receivers employees shall\n" - "%% only be made on a strict need to know basis.\n" - "%%\n\n"). + "\n"). hrl_header(Name) -> header() ++ "-hrl_name('" ++ ?S(Name) ++ ".hrl').\n". diff --git a/lib/diameter/src/diameter.appup.src b/lib/diameter/src/diameter.appup.src index 618d5a7f10..b1b8e38d39 100644 --- a/lib/diameter/src/diameter.appup.src +++ b/lib/diameter/src/diameter.appup.src @@ -49,7 +49,11 @@ {"1.10", [{restart_application, diameter}]}, %% 18.0 {"1.11", [{restart_application, diameter}]}, %% 18.1 {"1.11.1", [{restart_application, diameter}]}, %% 18.2 - {"1.11.2", [{restart_application, diameter}]} %% 18.3 + {"1.11.2", [{restart_application, diameter}]}, %% 18.3 + {"1.12", [{load_module, diameter_lib}, %% 19.0 + {load_module, diameter_traffic}, + {load_module, diameter_tcp}, + {load_module, diameter_sctp}]} ], [ {"0.9", [{restart_application, diameter}]}, @@ -80,6 +84,10 @@ {"1.10", [{restart_application, diameter}]}, {"1.11", [{restart_application, diameter}]}, {"1.11.1", [{restart_application, diameter}]}, - {"1.11.2", [{restart_application, diameter}]} + {"1.11.2", [{restart_application, diameter}]}, + {"1.12", [{load_module, diameter_sctp}, + {load_module, diameter_tcp}, + {load_module, diameter_traffic}, + {load_module, diameter_lib}]} ] }. diff --git a/lib/diameter/src/transport/diameter_sctp.erl b/lib/diameter/src/transport/diameter_sctp.erl index 4a005b853d..f48e4347ee 100644 --- a/lib/diameter/src/transport/diameter_sctp.erl +++ b/lib/diameter/src/transport/diameter_sctp.erl @@ -98,7 +98,7 @@ -record(listener, {ref :: reference(), socket :: gen_sctp:sctp_socket(), - count = 0 :: uint(), %% attached transport processes + service = false :: false | pid(), %% service process pending = {0, queue:new()}, accept :: [match()]}). %% Field pending implements two queues: the first of transport-to-be @@ -129,11 +129,14 @@ -> {ok, pid(), [inet:ip_address()]} when Ref :: diameter:transport_ref(). -start(T, #diameter_service{capabilities = Caps}, Opts) +start(T, Svc, Opts) when is_list(Opts) -> + #diameter_service{capabilities = Caps, + pid = SPid} + = Svc, diameter_sctp_sup:start(), %% start supervisors on demand Addrs = Caps#diameter_caps.host_ip_address, - s(T, Addrs, lists:map(fun ip/1, Opts)). + s(T, Addrs, SPid, lists:map(fun ip/1, Opts)). ip({ifaddr, A}) -> {ip, A}; @@ -144,18 +147,22 @@ ip(T) -> %% when there is not yet an association to assign it, or at comm_up on %% a new association in which case the call retrieves a transport from %% the pending queue. -s({accept, Ref} = A, Addrs, Opts) -> - {LPid, LAs} = listener(Ref, {Opts, Addrs}), - try gen_server:call(LPid, {A, self()}, infinity) of - {ok, TPid} -> {ok, TPid, LAs} +s({accept, Ref} = A, Addrs, SPid, Opts) -> + {ok, LPid, LAs} = listener(Ref, {Opts, Addrs}), + try gen_server:call(LPid, {A, self(), SPid}, infinity) of + {ok, TPid} -> + {ok, TPid, LAs}; + No -> + {error, No} catch - exit: Reason -> {error, Reason} + exit: Reason -> + {error, Reason} end; %% This implementation is due to there being no accept call in %% gen_sctp in order to be able to accept a new association only %% *after* an accepting transport has been spawned. -s({connect = C, Ref}, Addrs, Opts) -> +s({connect = C, Ref}, Addrs, _SPid, Opts) -> diameter_sctp_sup:start_child({C, self(), Opts, Addrs, Ref}). %% start_link/1 @@ -281,24 +288,23 @@ i({K, Ref}, #transport{mode = {accept, _}} = S) -> %% Accepting processes can be started concurrently: ensure only one %% listener is started. -listener(LRef, T) -> - diameter_sync:call({?MODULE, listener, LRef}, - {?MODULE, listener, [{LRef, T}]}, +listener(Ref, T) -> + diameter_sync:call({?MODULE, listener, Ref}, + {?MODULE, listener, [{Ref, T}]}, infinity, infinity). -listener({LRef, T}) -> - l(diameter_reg:match({?MODULE, listener, {LRef, '_'}}), LRef, T). +listener({Ref, T}) -> + l(diameter_reg:match({?MODULE, listener, {Ref, '_'}}), Ref, T). %% Existing listening process ... l([{{?MODULE, listener, {_, AS}}, LPid}], _, _) -> - {LAs, _Sock} = AS, - {LPid, LAs}; + {LAs, _Sock} = AS, + {ok, LPid, LAs}; %% ... or not. -l([], LRef, T) -> - {ok, LPid, LAs} = diameter_sctp_sup:start_child({listen, LRef, T}), - {LPid, LAs}. +l([], Ref, T) -> + diameter_sctp_sup:start_child({listen, Ref, T}). %% open/3 @@ -364,11 +370,17 @@ type(T) -> %% # handle_call/3 %% --------------------------------------------------------------------------- -handle_call({{accept, Ref}, Pid}, _, #listener{ref = Ref, - count = K} - = S) -> +handle_call({{accept, Ref}, Pid}, _, #listener{ref = Ref} = S) -> {TPid, NewS} = accept(Ref, Pid, S), - {reply, {ok, TPid}, NewS#listener{count = K+1}}; + {reply, {ok, TPid}, NewS}; + +handle_call({{accept, _} = T, Pid, SPid}, From, #listener{service = P} = S) -> + handle_call({T, Pid}, From, if not is_pid(P), is_pid(SPid) -> + monitor(process, SPid), + S#listener{service = SPid}; + true -> + S + end); handle_call(_, _, State) -> {reply, nok, State}. @@ -441,6 +453,13 @@ l({sctp, Sock, _RA, _RP, Data} = T, #listener{socket = Sock, setopts(Sock), NewS; +%% Service process has died. +l({'DOWN', _, process, Pid, _} = T, #listener{service = Pid, + socket = Sock}) -> + gen_sctp:close(Sock), + x(T); + +%% Accepting process has died. l({'DOWN', _MRef, process, TPid, _}, #listener{pending = {_,Q}} = S) -> down(queue:member(TPid, Q), TPid, S); @@ -454,20 +473,17 @@ l({transport, remove, _} = T, #listener{socket = Sock}) -> %% Accepting transport has died. %% One that's waiting for transport start in the pending queue ... -down(true, TPid, #listener{pending = {N,Q}, - count = K} - = S) -> +down(true, TPid, #listener{pending = {N,Q}} = S) -> NQ = queue:filter(fun(P) -> P /= TPid end, Q), if N < 0 -> %% awaiting an association ... - S#listener{count = K-1, - pending = {N+1, NQ}}; + S#listener{pending = {N+1, NQ}}; true -> %% ... or one has been assigned S#listener{pending = {N-1, NQ}} end; %% ... or one that's already attached. -down(false, _TPid, #listener{count = K} = S) -> - S#listener{count = K-1}. +down(false, _TPid, S) -> + S. %% t/2 %% diff --git a/lib/diameter/src/transport/diameter_tcp.erl b/lib/diameter/src/transport/diameter_tcp.erl index 546c2cfa5e..44abc5c3b4 100644 --- a/lib/diameter/src/transport/diameter_tcp.erl +++ b/lib/diameter/src/transport/diameter_tcp.erl @@ -71,11 +71,8 @@ %% a process owning the listening port. %% Listener process state. --record(listener, {socket :: inet:socket(), - count = 1 :: non_neg_integer()}). %% accepting processes -%% The count of accepting processes was previously used to terminate -%% the listening process, but diameter_reg:subscribe/2 is now used for -%% this. Leave the the count for trace purposes. +-record(listener, {socket :: inet:socket(), + service = false :: false | pid()}). %% service process %% Monitor process state. -record(monitor, @@ -138,11 +135,15 @@ | {ok, pid()} when Ref :: diameter:transport_ref(). -start({T, Ref}, #diameter_service{capabilities = Caps}, Opts) -> +start({T, Ref}, Svc, Opts) -> + #diameter_service{capabilities = Caps, + pid = SPid} + = Svc, + diameter_tcp_sup:start(), %% start tcp supervisors on demand {Mod, Rest} = split(Opts), Addrs = Caps#diameter_caps.host_ip_address, - Arg = {T, Ref, Mod, self(), Rest, Addrs}, + Arg = {T, Ref, Mod, self(), Rest, Addrs, SPid}, diameter_tcp_sup:start_child(Arg). split([{module, M} | Opts]) -> @@ -196,7 +197,7 @@ init(T) -> %% i/1 %% A transport process. -i({T, Ref, Mod, Pid, Opts, Addrs}) +i({T, Ref, Mod, Pid, Opts, Addrs, SPid}) when T == accept; T == connect -> monitor(process, Pid), @@ -214,7 +215,7 @@ i({T, Ref, Mod, Pid, Opts, Addrs}) ?DEFAULT_FRAGMENT_TIMEOUT), ?IS_TIMEOUT(Tmo) orelse ?ERROR({fragment_timer, Tmo}), Throttle = proplists:get_value(throttle_cb, OwnOpts, false), - Sock = init(T, Ref, Mod, Pid, SslOpts, Rest, Addrs), + Sock = init(T, Ref, Mod, Pid, SslOpts, Rest, Addrs, SPid), MPid ! {stop, self()}, %% tell the monitor to die M = if SslOpts -> ssl; true -> Mod end, putr(?REF_KEY, Ref), @@ -228,6 +229,11 @@ i({T, Ref, Mod, Pid, Opts, Addrs}) %% Put the reference in the process dictionary since we now use it %% advertise the ssl socket after TLS upgrade. +i({T, _Ref, _Mod, _Pid, _Opts, _Addrs} = Arg) %% from old code + when T == accept; + T == connect -> + i(erlang:append_element(Arg, _SPid = false)); + %% A monitor process to kill the transport if the parent dies. i(#monitor{parent = Pid, transport = TPid} = S) -> proc_lib:init_ack({ok, self()}), @@ -240,16 +246,18 @@ i(#monitor{parent = Pid, transport = TPid} = S) -> %% death. However, a link can be unlinked and this is exactly what %% gen_tcp seems to so. Links should be left to supervisors. -i({listen, LRef, APid, {Mod, Opts, Addrs}}) -> - [_] = diameter_config:subscribe(LRef, transport), %% assert existence +i({listen = L, Ref, _APid, T}) -> %% from old code + i({L, Ref, T}); + +i({listen, Ref, {Mod, Opts, Addrs}}) -> + [_] = diameter_config:subscribe(Ref, transport), %% assert existence {[LA, LP], Rest} = proplists:split(Opts, [ip, port]), LAddrOpt = get_addr(LA, Addrs), LPort = get_port(LP), {ok, LSock} = Mod:listen(LPort, gen_opts(LAddrOpt, Rest)), LAddr = laddr(LAddrOpt, Mod, LSock), - true = diameter_reg:add_new({?MODULE, listener, {LRef, {LAddr, LSock}}}), + true = diameter_reg:add_new({?MODULE, listener, {Ref, {LAddr, LSock}}}), proc_lib:init_ack({ok, self(), {LAddr, LSock}}), - monitor(process, APid), #listener{socket = LSock}. laddr([], Mod, Sock) -> @@ -268,21 +276,22 @@ ssl_opts([{ssl_options, Opts}]) ssl_opts(T) -> ?ERROR({ssl_options, T}). -%% init/7 +%% init/8 %% Establish a TLS connection before capabilities exchange ... -init(Type, Ref, Mod, Pid, true, Opts, Addrs) -> - init(Type, Ref, ssl, Pid, [{cb_info, ?TCP_CB(Mod)} | Opts], Addrs); +init(Type, Ref, Mod, Pid, true, Opts, Addrs, SPid) -> + init(Type, Ref, ssl, Pid, [{cb_info, ?TCP_CB(Mod)} | Opts], Addrs, SPid); %% ... or not. -init(Type, Ref, Mod, Pid, _, Opts, Addrs) -> - init(Type, Ref, Mod, Pid, Opts, Addrs). +init(Type, Ref, Mod, Pid, _, Opts, Addrs, SPid) -> + init(Type, Ref, Mod, Pid, Opts, Addrs, SPid). -%% init/6 +%% init/7 -init(accept = T, Ref, Mod, Pid, Opts, Addrs) -> +init(accept = T, Ref, Mod, Pid, Opts, Addrs, SPid) -> {[Matches], Rest} = proplists:split(Opts, [accept]), - {LAddr, LSock} = listener(Ref, {Mod, Rest, Addrs}), + {ok, LPid, {LAddr, LSock}} = listener(Ref, {Mod, Rest, Addrs}), + ok = gen_server:call(LPid, {accept, SPid}, infinity), proc_lib:init_ack({ok, self(), [LAddr]}), Sock = ok(accept(Mod, LSock)), ok = accept_peer(Mod, Sock, accept(Matches)), @@ -290,7 +299,7 @@ init(accept = T, Ref, Mod, Pid, Opts, Addrs) -> diameter_peer:up(Pid), Sock; -init(connect = T, Ref, Mod, Pid, Opts, Addrs) -> +init(connect = T, Ref, Mod, Pid, Opts, Addrs, _SPid) -> {[LA, RA, RP], Rest} = proplists:split(Opts, [ip, raddr, rport]), LAddrOpt = get_addr(LA, Addrs), RAddr = get_addr(RA), @@ -344,24 +353,26 @@ accept(Opts) -> %% Accepting processes can be started concurrently: ensure only one %% listener is started. -listener(LRef, T) -> - diameter_sync:call({?MODULE, listener, LRef}, - {?MODULE, listener, [{LRef, T, self()}]}, +listener(Ref, T) -> + diameter_sync:call({?MODULE, listener, Ref}, + {?MODULE, listener, [{Ref, T, self()}]}, infinity, infinity). -listener({LRef, T, TPid}) -> - l(diameter_reg:match({?MODULE, listener, {LRef, '_'}}), LRef, T, TPid). +%% listener/1 + +listener({Ref, T, _TPid}) -> + l(diameter_reg:match({?MODULE, listener, {Ref, '_'}}), Ref, T). + +%% l/3 %% Existing listening process ... -l([{{?MODULE, listener, {_, AS}}, LPid}], _, _, TPid) -> - LPid ! {accept, TPid}, - AS; +l([{{?MODULE, listener, {_, AS}}, LPid}], _, _) -> + {ok, LPid, AS}; %% ... or not. -l([], LRef, T, TPid) -> - {ok, _, AS} = diameter_tcp_sup:start_child({listen, LRef, TPid, T}), - AS. +l([], Ref, T) -> + diameter_tcp_sup:start_child({listen, Ref, T}). %% get_addr/1 @@ -440,6 +451,14 @@ portnr(Sock) -> %% # handle_call/3 %% --------------------------------------------------------------------------- +handle_call({accept, SPid}, _From, #listener{service = P} = S) -> + {reply, ok, if not is_pid(P), is_pid(SPid) -> + monitor(process, SPid), + S#listener{service = SPid}; + true -> + S + end}; + handle_call(_, _, State) -> {reply, nok, State}. @@ -507,19 +526,20 @@ m({'DOWN', _, process, Pid, _}, #monitor{parent = Pid, %% %% Transition listener state. -%% An accepting transport is attaching. -l({accept, TPid}, #listener{count = N} = S) -> - monitor(process, TPid), - S#listener{count = N+1}; - -%% Accepting process has died. -l({'DOWN', _, process, _, _}, #listener{count = N} = S) -> - S#listener{count = N-1}; +%% Service process has died. +l({'DOWN', _, process, Pid, _} = T, #listener{service = Pid, + socket = Sock}) -> + gen_tcp:close(Sock), + x(T); %% Transport has been removed. l({transport, remove, _} = T, #listener{socket = Sock}) -> gen_tcp:close(Sock), - x(T). + x(T); + +%% Possibly death of an accepting process monitored in old code. +l(_, S) -> + S. %% t/2 %% diff --git a/lib/diameter/test/diameter_traffic_SUITE.erl b/lib/diameter/test/diameter_traffic_SUITE.erl index 6f3a4801ee..4c82d4dee2 100644 --- a/lib/diameter/test/diameter_traffic_SUITE.erl +++ b/lib/diameter/test/diameter_traffic_SUITE.erl @@ -248,17 +248,14 @@ all() -> groups() -> Ts = tc(), Sctp = ?util:have_sctp(), - [{?util:name([R,D,A,C]), [parallel], Ts} || R <- ?ENCODINGS, - D <- ?RFCS, - A <- ?ENCODINGS, - C <- ?CONTAINERS] + [{B, [P], Ts} || {B,P} <- [{true, shuffle}, {false, parallel}]] ++ [{?util:name([T,R,D,A,C,SD,CD]), [], [start_services, add_transports, result_codes, - {group, ?util:name([R,D,A,C])}, + {group, SD orelse CD}, remove_transports, stop_services]} || T <- ?TRANSPORTS, diff --git a/lib/diameter/test/diameter_util.erl b/lib/diameter/test/diameter_util.erl index 37fcbbc267..cca28dd23c 100644 --- a/lib/diameter/test/diameter_util.erl +++ b/lib/diameter/test/diameter_util.erl @@ -195,13 +195,21 @@ unique_string() -> %% have_sctp/0 have_sctp() -> - case gen_sctp:open() of - {ok, Sock} -> - gen_sctp:close(Sock), - true; - {error, E} when E == eprotonosupport; - E == esocktnosupport -> %% fail on any other reason - false + case erlang:system_info(system_architecture) of + %% We do not support the sctp version present in solaris + %% version "sparc-sun-solaris2.10", that behaves differently + %% from later versions and linux + "sparc-sun-solaris2.10" -> + false; + _-> + case gen_sctp:open() of + {ok, Sock} -> + gen_sctp:close(Sock), + true; + {error, E} when E == eprotonosupport; + E == esocktnosupport -> %% fail on any other reason + false + end end. %% --------------------------------------------------------------------------- diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index cb750c69a3..23219950bb 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -17,5 +17,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 1.12 +DIAMETER_VSN = 1.12.1 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/edoc/doc/src/notes.xml b/lib/edoc/doc/src/notes.xml index ae8147c564..e6ad2c683f 100644 --- a/lib/edoc/doc/src/notes.xml +++ b/lib/edoc/doc/src/notes.xml @@ -32,6 +32,21 @@ <p>This document describes the changes made to the EDoc application.</p> +<section><title>Edoc 0.8</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>Improve types and specs in OTP documentation generated + from Erlang source files. </p> + <p> + Own Id: OTP-13720 Aux Id: ERL-120 </p> + </item> + </list> + </section> + +</section> + <section><title>Edoc 0.7.19</title> <section><title>Improvements and New Features</title> diff --git a/lib/edoc/vsn.mk b/lib/edoc/vsn.mk index f38800b3e0..d3cc732e9c 100644 --- a/lib/edoc/vsn.mk +++ b/lib/edoc/vsn.mk @@ -1 +1 @@ -EDOC_VSN = 0.7.19 +EDOC_VSN = 0.8 diff --git a/lib/erl_docgen/doc/src/notes.xml b/lib/erl_docgen/doc/src/notes.xml index 21dc617d55..cf24161d43 100644 --- a/lib/erl_docgen/doc/src/notes.xml +++ b/lib/erl_docgen/doc/src/notes.xml @@ -31,7 +31,22 @@ </header> <p>This document describes the changes made to the <em>erl_docgen</em> application.</p> - <section><title>Erl_Docgen 0.5</title> + <section><title>Erl_Docgen 0.6</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>Improve types and specs in OTP documentation generated + from Erlang source files. </p> + <p> + Own Id: OTP-13720 Aux Id: ERL-120 </p> + </item> + </list> + </section> + +</section> + +<section><title>Erl_Docgen 0.5</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/erl_docgen/vsn.mk b/lib/erl_docgen/vsn.mk index 98d1c3f7be..6489d26327 100644 --- a/lib/erl_docgen/vsn.mk +++ b/lib/erl_docgen/vsn.mk @@ -1 +1 @@ -ERL_DOCGEN_VSN = 0.5 +ERL_DOCGEN_VSN = 0.6 diff --git a/lib/erl_interface/doc/src/notes.xml b/lib/erl_interface/doc/src/notes.xml index a69c5aac11..4ef5454f44 100644 --- a/lib/erl_interface/doc/src/notes.xml +++ b/lib/erl_interface/doc/src/notes.xml @@ -31,6 +31,23 @@ </header> <p>This document describes the changes made to the Erl_interface application.</p> +<section><title>Erl_Interface 3.9.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Look for .erlang.cookie in windows system directory if + HOMEDRIVE and HOMEPATH is not set. The same behaviour as + the VM.</p> + <p> + Own Id: OTP-13849</p> + </item> + </list> + </section> + +</section> + <section><title>Erl_Interface 3.9</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/erl_interface/vsn.mk b/lib/erl_interface/vsn.mk index 33705d1e8b..82be43b7df 100644 --- a/lib/erl_interface/vsn.mk +++ b/lib/erl_interface/vsn.mk @@ -1,2 +1,2 @@ -EI_VSN = 3.9 +EI_VSN = 3.9.1 ERL_INTERFACE_VSN = $(EI_VSN) diff --git a/lib/eunit/doc/src/notes.xml b/lib/eunit/doc/src/notes.xml index 88602e8222..6ae3c04bc8 100644 --- a/lib/eunit/doc/src/notes.xml +++ b/lib/eunit/doc/src/notes.xml @@ -33,6 +33,23 @@ </header> <p>This document describes the changes made to the EUnit application.</p> +<section><title>Eunit 2.3.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + When asserts were moved out to a separate header file, + the automatic enabling of asserts when testing is enabled + stopped working.</p> + <p> + Own Id: OTP-13892</p> + </item> + </list> + </section> + +</section> + <section><title>Eunit 2.3</title> <section><title>Improvements and New Features</title> diff --git a/lib/eunit/include/eunit.hrl b/lib/eunit/include/eunit.hrl index 7fd6c206a4..8a4cad1e7e 100644 --- a/lib/eunit/include/eunit.hrl +++ b/lib/eunit/include/eunit.hrl @@ -51,7 +51,9 @@ %% note that the main switch used within this file is NOTEST; however, %% both TEST and EUNIT may be used to check whether testing is enabled -ifndef(NOTEST). --undef(NOASSERT). % testing requires that assertions are enabled +-ifndef(ASSERT). +-define(ASSERT, true). % testing requires that assertions are enabled +-endif. -ifndef(TEST). -define(TEST, true). -endif. diff --git a/lib/eunit/vsn.mk b/lib/eunit/vsn.mk index b551ee6eb6..83d826f8b6 100644 --- a/lib/eunit/vsn.mk +++ b/lib/eunit/vsn.mk @@ -1 +1 @@ -EUNIT_VSN = 2.3 +EUNIT_VSN = 2.3.1 diff --git a/lib/gs/doc/src/notes.xml b/lib/gs/doc/src/notes.xml index 20188c75e2..4f3f0645a5 100644 --- a/lib/gs/doc/src/notes.xml +++ b/lib/gs/doc/src/notes.xml @@ -31,7 +31,23 @@ </header> <p>This document describes the changes made to the GS application.</p> - <section><title>GS 1.6.1</title> + <section><title>GS 1.6.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + </list> + </section> + +</section> + +<section><title>GS 1.6.1</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/gs/vsn.mk b/lib/gs/vsn.mk index c762507bab..d2c5d15a96 100644 --- a/lib/gs/vsn.mk +++ b/lib/gs/vsn.mk @@ -1,2 +1,2 @@ -GS_VSN = 1.6.1 +GS_VSN = 1.6.2 diff --git a/lib/hipe/cerl/erl_bif_types.erl b/lib/hipe/cerl/erl_bif_types.erl index 9453ca6c6f..230fce2e68 100644 --- a/lib/hipe/cerl/erl_bif_types.erl +++ b/lib/hipe/cerl/erl_bif_types.erl @@ -124,7 +124,7 @@ t_map_entries/2, t_map_put/3, t_map_update/3, - map_pairwise_merge/3 + t_map_pairwise_merge/4 ]). -ifdef(DO_ERL_BIF_TYPES_TEST). @@ -1689,10 +1689,10 @@ type(maps, merge, 2, Xs, Opaques) -> BDefK = t_map_def_key(MapB, Opaques), ADefV = t_map_def_val(MapA, Opaques), BDefV = t_map_def_val(MapB, Opaques), - t_map(map_pairwise_merge( + t_map(t_map_pairwise_merge( fun(K, _, _, mandatory, V) -> {K, mandatory, V}; (K, MNess, VA, optional, VB) -> {K, MNess, t_sup(VA,VB)} - end, MapA, MapB), + end, MapA, MapB, Opaques), t_sup(ADefK, BDefK), t_sup(ADefV, BDefV)) end, Opaques); type(maps, put, 3, Xs, Opaques) -> diff --git a/lib/hipe/cerl/erl_types.erl b/lib/hipe/cerl/erl_types.erl index c9dd1051f3..15f7b793a1 100644 --- a/lib/hipe/cerl/erl_types.erl +++ b/lib/hipe/cerl/erl_types.erl @@ -159,6 +159,7 @@ t_map_get/2, t_map_get/3, t_map_is_key/2, t_map_is_key/3, t_map_update/2, t_map_update/3, + t_map_pairwise_merge/4, t_map_put/2, t_map_put/3, t_matchstate/0, t_matchstate/2, @@ -219,8 +220,7 @@ is_erl_type/1, atom_to_string/1, var_table__new/0, - cache__new/0, - map_pairwise_merge/3 + cache__new/0 ]). %%-define(DO_ERL_TYPES_TEST, true). @@ -494,9 +494,9 @@ t_contains_opaque(?function(Domain, Range), Opaques) -> t_contains_opaque(Domain, Opaques) orelse t_contains_opaque(Range, Opaques); t_contains_opaque(?identifier(_Types), _Opaques) -> false; -t_contains_opaque(?integer(_Types), _Opaques) -> false; t_contains_opaque(?int_range(_From, _To), _Opaques) -> false; t_contains_opaque(?int_set(_Set), _Opaques) -> false; +t_contains_opaque(?integer(_Types), _Opaques) -> false; t_contains_opaque(?list(Type, Tail, _), Opaques) -> t_contains_opaque(Type, Opaques) orelse t_contains_opaque(Tail, Opaques); t_contains_opaque(?map(_, _, _) = Map, Opaques) -> @@ -1768,13 +1768,26 @@ mapdict_insert(E1={K1,_,_}, [E2={K2,_,_}|T]) when K1 > K2 -> [E2|mapdict_insert(E1, T)]; mapdict_insert(E={_,_,_}, T) -> [E|T]. +-type map_pairwise_merge_fun() :: fun((erl_type(), + t_map_mandatoriness(), erl_type(), + t_map_mandatoriness(), erl_type()) + -> t_map_pair() | false). + +-spec t_map_pairwise_merge(map_pairwise_merge_fun(), erl_type(), erl_type(), + opaques()) -> t_map_dict(). +t_map_pairwise_merge(F, MapA, MapB, Opaques) -> + do_opaque(MapA, Opaques, + fun(UMapA) -> + do_opaque(MapB, Opaques, + fun(UMapB) -> + map_pairwise_merge(F, UMapA, UMapB) + end) + end). + %% Merges the pairs of two maps together. Missing pairs become (?opt, DefV) or %% (?opt, ?none), depending on whether K \in DefK. --spec map_pairwise_merge(fun((erl_type(), - t_map_mandatoriness(), erl_type(), - t_map_mandatoriness(), erl_type()) - -> t_map_pair() | false), - erl_type(), erl_type()) -> t_map_dict(). +-spec map_pairwise_merge(map_pairwise_merge_fun(), erl_type(), erl_type()) + -> t_map_dict(). map_pairwise_merge(F, ?map(APairs, ADefK, ADefV), ?map(BPairs, BDefK, BDefV)) -> map_pairwise_merge(F, APairs, ADefK, ADefV, BPairs, BDefK, BDefV). diff --git a/lib/hipe/doc/src/notes.xml b/lib/hipe/doc/src/notes.xml index e2a1524be6..fc529fba61 100644 --- a/lib/hipe/doc/src/notes.xml +++ b/lib/hipe/doc/src/notes.xml @@ -31,6 +31,46 @@ </header> <p>This document describes the changes made to HiPE.</p> +<section><title>Hipe 3.15.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed various hipe compiler backend bugs affecting mostly + ARM and SPARC.</p> + <p> + Own Id: OTP-13846 Aux Id: PR-1146 </p> + </item> + <item> + <p> + Fixed some Dialyzer warnings and code cleanup for the + Sparc compiler backend.</p> + <p> + Own Id: OTP-13861 Aux Id: PR-1148 </p> + </item> + <item> + <p> Fix erl_bif_types opaque bug. </p> + <p> + Own Id: OTP-13878 Aux Id: PR-1161, ERL-249 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix erl_types opaque match order</p> + <p> + Own Id: OTP-13876</p> + </item> + </list> + </section> + +</section> + <section><title>Hipe 3.15.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/hipe/vsn.mk b/lib/hipe/vsn.mk index e61c1a042c..f00ff0cf2e 100644 --- a/lib/hipe/vsn.mk +++ b/lib/hipe/vsn.mk @@ -1 +1 @@ -HIPE_VSN = 3.15.1 +HIPE_VSN = 3.15.2 diff --git a/lib/ic/doc/src/notes.xml b/lib/ic/doc/src/notes.xml index 08b02bc4a4..ea8bf758cf 100644 --- a/lib/ic/doc/src/notes.xml +++ b/lib/ic/doc/src/notes.xml @@ -31,7 +31,29 @@ <file>notes.xml</file> </header> - <section><title>IC 4.4.1</title> + <section><title>IC 4.4.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> Correct bugs when path to mib or idl spec files + contains UTF-8 characters. </p> + <p> + Own Id: OTP-13718 Aux Id: ERL-179 </p> + </item> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + </list> + </section> + +</section> + +<section><title>IC 4.4.1</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/ic/src/ic_codegen.erl b/lib/ic/src/ic_codegen.erl index adad021da1..a3f141f606 100644 --- a/lib/ic/src/ic_codegen.erl +++ b/lib/ic/src/ic_codegen.erl @@ -245,8 +245,8 @@ emit_stub_head(G, F1, Name, java) -> stub_header(G, Name) -> ["Implementation stub file", "", - io_lib:format("Target: ~s", [Name]), - io_lib:format("Source: ~s", [ic_genobj:idlfile(G)]), + io_lib:format("Target: ~ts", [Name]), + io_lib:format("Source: ~ts", [ic_genobj:idlfile(G)]), io_lib:format("IC vsn: ~s", [?COMPILERVSN]), "", "This file is automatically generated. DO NOT EDIT IT."]. @@ -298,8 +298,8 @@ emit_hrl_head(G, Fd, Name, c_server) -> hrl_header(G, Name) -> ["", - io_lib:format("Target: ~s", [Name]), - io_lib:format("Source: ~s", [ic_genobj:idlfile(G)]), + io_lib:format("Target: ~ts", [Name]), + io_lib:format("Source: ~ts", [ic_genobj:idlfile(G)]), io_lib:format("IC vsn: ~s", [?COMPILERVSN]), "", "This file is automatically generated. DO NOT EDIT IT."]. diff --git a/lib/ic/vsn.mk b/lib/ic/vsn.mk index 7d00ae0170..f0e5e7c266 100644 --- a/lib/ic/vsn.mk +++ b/lib/ic/vsn.mk @@ -1 +1 @@ -IC_VSN = 4.4.1 +IC_VSN = 4.4.2 diff --git a/lib/inets/doc/src/notes.xml b/lib/inets/doc/src/notes.xml index caa5a083a3..0c7604ef65 100644 --- a/lib/inets/doc/src/notes.xml +++ b/lib/inets/doc/src/notes.xml @@ -33,7 +33,24 @@ <file>notes.xml</file> </header> - <section><title>Inets 6.3.2</title> + <section><title>Inets 6.3.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The legacy option 'inet6fb4' for inets had stopped + working. This bug has now been corrected. Fix by Edwin + Fine in bugs.erlang.org ERL-200 and Github PR#1132.</p> + <p> + Own Id: OTP-13776 Aux Id: ERL-200 PR-1132 </p> + </item> + </list> + </section> + +</section> + +<section><title>Inets 6.3.2</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/inets/src/http_client/httpc_response.erl b/lib/inets/src/http_client/httpc_response.erl index 91256fa6a2..d8bdac24e3 100644 --- a/lib/inets/src/http_client/httpc_response.erl +++ b/lib/inets/src/http_client/httpc_response.erl @@ -110,27 +110,30 @@ result(Response = {{_, 300, _}, _, _}, redirect(Response, Request); result(Response = {{_, Code, _}, _, _}, + Request = #request{settings = + #http_options{autoredirect = true}, + method = post}) when (Code =:= 301) orelse + (Code =:= 302) orelse + (Code =:= 303) -> + redirect(Response, Request#request{method = get}); +result(Response = {{_, Code, _}, _, _}, + Request = #request{settings = + #http_options{autoredirect = true}, + method = post}) when (Code =:= 307) -> + redirect(Response, Request); +result(Response = {{_, Code, _}, _, _}, Request = #request{settings = #http_options{autoredirect = true}, - method = head}) when (Code =:= 301) orelse + method = Method}) when (Code =:= 301) orelse (Code =:= 302) orelse (Code =:= 303) orelse (Code =:= 307) -> - redirect(Response, Request); -result(Response = {{_, Code, _}, _, _}, - Request = #request{settings = - #http_options{autoredirect = true}, - method = get}) when (Code =:= 301) orelse - (Code =:= 302) orelse - (Code =:= 303) orelse - (Code =:= 307) -> - redirect(Response, Request); -result(Response = {{_, 303, _}, _, _}, - Request = #request{settings = - #http_options{autoredirect = true}, - method = post}) -> - redirect(Response, Request#request{method = get}); - + case lists:member(Method, [get, head, options, trace]) of + true -> + redirect(Response, Request); + false -> + transparent(Response, Request) + end; result(Response = {{_,503,_}, _, _}, Request) -> status_service_unavailable(Response, Request); diff --git a/lib/inets/src/inets_app/inets.appup.src b/lib/inets/src/inets_app/inets.appup.src index d4d21f6774..3a31daeb20 100644 --- a/lib/inets/src/inets_app/inets.appup.src +++ b/lib/inets/src/inets_app/inets.appup.src @@ -18,16 +18,10 @@ %% %CopyrightEnd% {"%VSN%", [ - {<<"6.3.1">>, - [{load_module, mod_esi, soft_purge, soft_purge, []} - ]}, {<<"6\\..*">>,[{restart_application, inets}]}, {<<"5\\..*">>,[{restart_application, inets}]} ], [ - {<<"6.3.1">>, - [{load_module, mod_esi, soft_purge, soft_purge, []} - ]}, {<<"6\\..*">>,[{restart_application, inets}]}, {<<"5\\..*">>,[{restart_application, inets}]} ] diff --git a/lib/inets/test/httpc_SUITE.erl b/lib/inets/test/httpc_SUITE.erl index 932567ec55..57da82c6ad 100644 --- a/lib/inets/test/httpc_SUITE.erl +++ b/lib/inets/test/httpc_SUITE.erl @@ -500,10 +500,11 @@ redirect_multiple_choises(Config) when is_list(Config) -> httpc:request(get, {URL300, []}, [{autoredirect, false}], []). %%------------------------------------------------------------------------- redirect_moved_permanently() -> - [{doc, "If the 301 status code is received in response to a request other " - "than GET or HEAD, the user agent MUST NOT automatically redirect the request " - "unless it can be confirmed by the user, since this might change " - "the conditions under which the request was issued."}]. + [{doc, "The server SHOULD generate a Location header field in the response " + "containing a preferred URI reference for the new permanent URI. The user " + "agent MAY use the Location field value for automatic redirection. The server's " + "response payload usually contains a short hypertext note with a " + "hyperlink to the new URI(s)."}]. redirect_moved_permanently(Config) when is_list(Config) -> URL301 = url(group_name(Config), "/301.html", Config), @@ -514,15 +515,16 @@ redirect_moved_permanently(Config) when is_list(Config) -> {ok, {{_,200,_}, [_ | _], []}} = httpc:request(head, {URL301, []}, [], []), - {ok, {{_,301,_}, [_ | _], [_|_]}} + {ok, {{_,200,_}, [_ | _], [_|_]}} = httpc:request(post, {URL301, [],"text/plain", "foobar"}, [], []). %%------------------------------------------------------------------------- redirect_found() -> - [{doc," If the 302 status code is received in response to a request other " - "than GET or HEAD, the user agent MUST NOT automatically redirect the " - "request unless it can be confirmed by the user, since this might change " - "the conditions under which the request was issued."}]. + [{doc, "The server SHOULD generate a Location header field in the response " + "containing a URI reference for the different URI. The user agent MAY " + "use the Location field value for automatic redirection. The server's " + "response payload usually contains a short hypertext note with a " + "hyperlink to the different URI(s)."}]. redirect_found(Config) when is_list(Config) -> URL302 = url(group_name(Config), "/302.html", Config), @@ -533,14 +535,14 @@ redirect_found(Config) when is_list(Config) -> {ok, {{_,200,_}, [_ | _], []}} = httpc:request(head, {URL302, []}, [], []), - {ok, {{_,302,_}, [_ | _], [_|_]}} + {ok, {{_,200,_}, [_ | _], [_|_]}} = httpc:request(post, {URL302, [],"text/plain", "foobar"}, [], []). %%------------------------------------------------------------------------- redirect_see_other() -> [{doc, "The different URI SHOULD be given by the Location field in the response. " "Unless the request method was HEAD, the entity of the response SHOULD contain a short " - "hypertext note with a hyperlink to the new URI(s). "}]. + "hypertext note with a hyperlink to the new URI(s)."}]. redirect_see_other(Config) when is_list(Config) -> URL303 = url(group_name(Config), "/303.html", Config), @@ -556,10 +558,11 @@ redirect_see_other(Config) when is_list(Config) -> [], []). %%------------------------------------------------------------------------- redirect_temporary_redirect() -> - [{doc," If the 307 status code is received in response to a request other " - "than GET or HEAD, the user agent MUST NOT automatically redirect the request " - "unless it can be confirmed by the user, since this might change " - "the conditions under which the request was issued."}]. + [{doc, "The server SHOULD generate a Location header field in the response " + "containing a URI reference for the different URI. The user agent MAY " + "use the Location field value for automatic redirection. The server's " + "response payload usually contains a short hypertext note with a " + "hyperlink to the different URI(s)."}]. redirect_temporary_redirect(Config) when is_list(Config) -> URL307 = url(group_name(Config), "/307.html", Config), @@ -570,7 +573,7 @@ redirect_temporary_redirect(Config) when is_list(Config) -> {ok, {{_,200,_}, [_ | _], []}} = httpc:request(head, {URL307, []}, [], []), - {ok, {{_,307,_}, [_ | _], [_|_]}} + {ok, {{_,200,_}, [_ | _], [_|_]}} = httpc:request(post, {URL307, [],"text/plain", "foobar"}, [], []). diff --git a/lib/inets/vsn.mk b/lib/inets/vsn.mk index 747724a86b..f668ef106c 100644 --- a/lib/inets/vsn.mk +++ b/lib/inets/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = inets -INETS_VSN = 6.3.2 +INETS_VSN = 6.3.3 PRE_VSN = APP_VSN = "$(APPLICATION)-$(INETS_VSN)$(PRE_VSN)" diff --git a/lib/jinterface/doc/src/notes.xml b/lib/jinterface/doc/src/notes.xml index 3151fc4b5e..30f607c357 100644 --- a/lib/jinterface/doc/src/notes.xml +++ b/lib/jinterface/doc/src/notes.xml @@ -31,6 +31,22 @@ </header> <p>This document describes the changes made to the Jinterface application.</p> +<section><title>Jinterface 1.7.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Update build scripts to not make assumtions about where + env, cp and perl are located.</p> + <p> + Own Id: OTP-13800</p> + </item> + </list> + </section> + +</section> + <section><title>Jinterface 1.7</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/jinterface/vsn.mk b/lib/jinterface/vsn.mk index 752b34e78c..c29d9df3d7 100644 --- a/lib/jinterface/vsn.mk +++ b/lib/jinterface/vsn.mk @@ -1 +1 @@ -JINTERFACE_VSN = 1.7 +JINTERFACE_VSN = 1.7.1 diff --git a/lib/kernel/doc/src/file.xml b/lib/kernel/doc/src/file.xml index d734ee25b8..09497482cf 100644 --- a/lib/kernel/doc/src/file.xml +++ b/lib/kernel/doc/src/file.xml @@ -79,7 +79,7 @@ <seealso marker="#list_dir_all"><c>list_dir_all/1</c></seealso> and <seealso marker="#read_link_all"><c>read_link_all/1</c></seealso>.</p> - <p>See also section <seealso marker="stdlib:unicode_usage#notes-about-raw-filenames">Notes About Raw Filenames</seealso> in the STDLIB User´s Giude.</p> + <p>See also section <seealso marker="stdlib:unicode_usage#notes-about-raw-filenames">Notes About Raw Filenames</seealso> in the STDLIB User's Guide.</p> </description> diff --git a/lib/kernel/doc/src/inet.xml b/lib/kernel/doc/src/inet.xml index 3f4e3684f4..d907cef7d3 100644 --- a/lib/kernel/doc/src/inet.xml +++ b/lib/kernel/doc/src/inet.xml @@ -1092,6 +1092,15 @@ setcap cap_sys_admin,cap_sys_ptrace,cap_dac_read_search+epi beam.smp</code> The option is ignored on platforms where it is not implemented. Use with caution.</p> </item> + <tag><c>{tclass, Integer}</c></tag> + <item> + <p> + Sets <c>IPV6_TCLASS IP</c> level options on platforms + where this is implemented. The behavior and allowed range + varies between different systems. + The option is ignored on platforms where it is not + implemented. Use with caution.</p> + </item> </taglist> <p>In addition to these options, <em>raw</em> option specifications can be used. The raw options are diff --git a/lib/kernel/doc/src/notes.xml b/lib/kernel/doc/src/notes.xml index 9e9be3f661..5bcc0b7c09 100644 --- a/lib/kernel/doc/src/notes.xml +++ b/lib/kernel/doc/src/notes.xml @@ -31,6 +31,74 @@ </header> <p>This document describes the changes made to the Kernel application.</p> +<section><title>Kernel 5.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix a memory leak when calling + seq_trace:get_system_tracer().</p> + <p> + Own Id: OTP-13742</p> + </item> + <item> + <p> + Fix for the problem that when adding the ebin directory + of an application to the code path, the + <c>code:priv_dir/1</c> function returns an incorrect path + to the priv directory of the same application.</p> + <p> + Own Id: OTP-13758 Aux Id: ERL-195 </p> + </item> + <item> + <p> + Fix code_server crash when adding code paths of two + levels.</p> + <p> + Own Id: OTP-13765 Aux Id: ERL-194 </p> + </item> + <item> + <p> + Respect -proto_dist switch while connection to EPMD</p> + <p> + Own Id: OTP-13770 Aux Id: PR-1129 </p> + </item> + <item> + <p> + Fixed a bug where init:stop could deadlock if a process + with infinite shutdown timeout (e.g. a supervisor) + attempted to load code while terminating.</p> + <p> + Own Id: OTP-13802</p> + </item> + <item> + <p> + Close stdin of commands run in os:cmd. This is a + backwards compatiblity fix that restores the behaviour of + pre 19.0 os:cmd.</p> + <p> + Own Id: OTP-13867 Aux Id: seq13178 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add <c>net_kernel:setopts/2</c> and + <c>net_kernel:getopts/2</c> to control options for + distribution sockets in runtime.</p> + <p> + Own Id: OTP-13564</p> + </item> + </list> + </section> + +</section> + <section><title>Kernel 5.0.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/kernel/src/code_server.erl b/lib/kernel/src/code_server.erl index 48541ec500..59b26176bf 100644 --- a/lib/kernel/src/code_server.erl +++ b/lib/kernel/src/code_server.erl @@ -811,7 +811,13 @@ clear_namedb([], _) -> %% Dir must be a complete pathname (not only a name). insert_dir(Dir, Db) -> Splitted = filename:split(Dir), - Name = get_name_from_splitted(Splitted), + case get_name_from_splitted(Splitted) of + Name when Name /= "ebin", Name /= "." -> + Name; + _ -> + SplittedAbsName = filename:split(absname(Dir)), + Name = get_name_from_splitted(SplittedAbsName) + end, AppDir = filename:join(del_ebin_1(Splitted)), do_insert_name(Name, AppDir, Db). @@ -952,6 +958,10 @@ del_ebin_1([Parent,App,"ebin"]) -> [Archive] end end; +del_ebin_1(Path = [_App,"ebin"]) -> + del_ebin_1(filename:split(absname(filename:join(Path)))); +del_ebin_1(["ebin"]) -> + del_ebin_1(filename:split(absname("ebin"))); del_ebin_1([H|T]) -> [H|del_ebin_1(T)]; del_ebin_1([]) -> @@ -1382,11 +1392,10 @@ finish_on_load(PidRef, OnLoadRes, #state{on_load=OnLoad0}=St0) -> finish_on_load_1(Mod, OnLoadRes, Waiting, St) -> Keep = OnLoadRes =:= ok, - erlang:finish_after_on_load(Mod, Keep), + erts_code_purger:finish_after_on_load(Mod, Keep), Res = case Keep of false -> _ = finish_on_load_report(Mod, OnLoadRes), - _ = erts_code_purger:purge(Mod), {error,on_load_failure}; true -> {module,Mod} diff --git a/lib/kernel/src/inet.erl b/lib/kernel/src/inet.erl index 75dd800c6b..f5c13ecdd7 100644 --- a/lib/kernel/src/inet.erl +++ b/lib/kernel/src/inet.erl @@ -677,7 +677,7 @@ parse_strict_address(Addr) -> %% Return a list of available options options() -> [ - tos, priority, reuseaddr, keepalive, dontroute, linger, + tos, tclass, priority, reuseaddr, keepalive, dontroute, linger, broadcast, sndbuf, recbuf, nodelay, ipv6_v6only, buffer, header, active, packet, deliver, mode, multicast_if, multicast_ttl, multicast_loop, @@ -698,7 +698,7 @@ stats() -> %% Available options for tcp:connect %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% connect_options() -> - [tos, priority, reuseaddr, keepalive, linger, sndbuf, recbuf, nodelay, + [tos, tclass, priority, reuseaddr, keepalive, linger, sndbuf, recbuf, nodelay, header, active, packet, packet_size, buffer, mode, deliver, line_delimiter, exit_on_close, high_watermark, low_watermark, high_msgq_watermark, low_msgq_watermark, send_timeout, send_timeout_close, delay_send, raw, @@ -766,7 +766,7 @@ con_add(Name, Val, #connect_opts{} = R, Opts, AllOpts) -> %% Available options for tcp:listen %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% listen_options() -> - [tos, priority, reuseaddr, keepalive, linger, sndbuf, recbuf, nodelay, + [tos, tclass, priority, reuseaddr, keepalive, linger, sndbuf, recbuf, nodelay, header, active, packet, buffer, mode, deliver, backlog, ipv6_v6only, exit_on_close, high_watermark, low_watermark, high_msgq_watermark, low_msgq_watermark, send_timeout, send_timeout_close, delay_send, @@ -846,7 +846,7 @@ tcp_module_1(Opts, Address) -> %% Available options for udp:open %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% udp_options() -> - [tos, priority, reuseaddr, sndbuf, recbuf, header, active, buffer, mode, + [tos, tclass, priority, reuseaddr, sndbuf, recbuf, header, active, buffer, mode, deliver, ipv6_v6only, broadcast, dontroute, multicast_if, multicast_ttl, multicast_loop, add_membership, drop_membership, read_packets,raw, @@ -917,7 +917,7 @@ udp_module(Opts) -> % (*) passing of open FDs ("fdopen") is not supported. sctp_options() -> [ % The following are generic inet options supported for SCTP sockets: - mode, active, buffer, tos, priority, dontroute, reuseaddr, linger, sndbuf, + mode, active, buffer, tos, tclass, priority, dontroute, reuseaddr, linger, sndbuf, recbuf, ipv6_v6only, high_msgq_watermark, low_msgq_watermark, % Other options are SCTP-specific (though they may be similar to their diff --git a/lib/kernel/src/inet_int.hrl b/lib/kernel/src/inet_int.hrl index c8a8962e78..4e8f59a3b9 100644 --- a/lib/kernel/src/inet_int.hrl +++ b/lib/kernel/src/inet_int.hrl @@ -153,6 +153,7 @@ -define(INET_LOPT_NETNS, 38). -define(INET_LOPT_TCP_SHOW_ECONNRESET, 39). -define(INET_LOPT_LINE_DELIM, 40). +-define(INET_OPT_TCLASS, 41). % Specific SCTP options: separate range: -define(SCTP_OPT_RTOINFO, 100). -define(SCTP_OPT_ASSOCINFO, 101). diff --git a/lib/kernel/src/kernel.appup.src b/lib/kernel/src/kernel.appup.src index d16e200cb3..82cf73cbda 100644 --- a/lib/kernel/src/kernel.appup.src +++ b/lib/kernel/src/kernel.appup.src @@ -18,9 +18,9 @@ %% %CopyrightEnd% {"%VSN%", %% Up from - max one major revision back - [{<<"5\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* + [{<<"5\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* {<<"4\\.[0-2](\\.[0-9]+)*">>,[restart_new_emulator]}], % OTP-18.* %% Down to - max one major revision back - [{<<"5\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* + [{<<"5\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* {<<"4\\.[0-2](\\.[0-9]+)*">>,[restart_new_emulator]}] % OTP-18.* }. diff --git a/lib/kernel/src/os.erl b/lib/kernel/src/os.erl index 05bbf1069e..f8519d3a5e 100644 --- a/lib/kernel/src/os.erl +++ b/lib/kernel/src/os.erl @@ -254,7 +254,19 @@ mk_cmd(_,Cmd) -> {"/bin/sh -s unix:cmd", [out], %% We insert a new line after the command, in case the command %% contains a comment character. - ["(", unicode:characters_to_binary(Cmd), "\n); echo \"\^D\"\n"], + %% + %% The </dev/null closes stdin, which means that programs + %% that use a closed stdin as an termination indicator works. + %% An example of such a program is 'more'. + %% + %% The "echo ^D" is used to indicate that the program has executed + %% and we should return any output we have gotten. We cannot use + %% termination of the child or closing of stdin/stdout as then + %% starting background jobs from os:cmd will block os:cmd. + %% + %% I tried changing this to be "better", but got bombarded with + %% backwards incompatibility bug reports, so leave this as it is. + ["(", unicode:characters_to_binary(Cmd), "\n) </dev/null; echo \"\^D\"\n"], <<$\^D>>}. validate(Atom) when is_atom(Atom) -> diff --git a/lib/kernel/test/code_SUITE.erl b/lib/kernel/test/code_SUITE.erl index 8da67c89f8..c5167efa56 100644 --- a/lib/kernel/test/code_SUITE.erl +++ b/lib/kernel/test/code_SUITE.erl @@ -36,6 +36,7 @@ code_archive/1, code_archive2/1, on_load/1, on_load_binary/1, on_load_embedded/1, on_load_errors/1, on_load_update/1, on_load_purge/1, on_load_self_call/1, on_load_pending/1, + on_load_deleted/1, big_boot_embedded/1, native_early_modules/1, get_mode/1, normalized_paths/1]). @@ -66,6 +67,7 @@ all() -> bad_erl_libs, code_archive, code_archive2, on_load, on_load_binary, on_load_embedded, on_load_errors, on_load_update, on_load_purge, on_load_self_call, on_load_pending, + on_load_deleted, big_boot_embedded, native_early_modules, get_mode, normalized_paths]. groups() -> @@ -1602,6 +1604,98 @@ on_load_pending(_Config) -> ok = Mod:t(), ok. +on_load_deleted(_Config) -> + Mod = ?FUNCTION_NAME, + + R0 = fun() -> + Tree = ?Q(["-module('@Mod@').\n", + "-on_load(f/0).\n", + "f() -> ok.\n"]), + merl:print(Tree), + {ok,Mod,Code} = merl:compile(Tree), + {module,Mod} = code:load_binary(Mod, "", Code) + end, + delete_before_reload(Mod, R0), + delete_before_reload(Mod, R0), + + R1 = fun() -> + Tree = ?Q(["-module('@Mod@').\n", + "-on_load(f/0).\n", + "f() -> fail.\n"]), + merl:print(Tree), + {ok,Mod,Code} = merl:compile(Tree), + {error,on_load_failure} = code:load_binary(Mod, "", Code) + end, + delete_before_reload(Mod, R1), + delete_before_reload(Mod, R1), + + OtherMod = list_to_atom(lists:concat([Mod,"_42"])), + OtherTree = ?Q(["-module('@OtherMod@').\n"]), + merl:print(OtherTree), + {ok,OtherMod,OtherCode} = merl:compile(OtherTree), + + R2 = fun() -> + RegName = 'on_load__registered_name', + Tree = ?Q(["-module('@Mod@').\n", + "-on_load(f/0).\n", + "f() ->\n", + " register('@RegName@', self()),\n", + " receive _ -> ok end.\n"]), + merl:print(Tree), + {ok,Mod,Code} = merl:compile(Tree), + spawn(fun() -> + {module,Mod} = code:load_binary(Mod, "", Code) + end), + receive after 1 -> ok end, + {module,OtherMod} = code:load_binary(OtherMod, "", + OtherCode), + RegName ! stop + end, + delete_before_reload(Mod, R2), + + ok. + +delete_before_reload(Mod, Reload) -> + false = check_old_code(Mod), + + Tree1 = ?Q(["-module('@Mod@').\n", + "-export([f/1]).\n", + "f(Parent) ->\n", + " register('@Mod@', self()),\n", + " Parent ! started,\n", + " receive _ -> ok end.\n"]), + merl:print(Tree1), + {ok,Mod,Code1} = merl:compile(Tree1), + + Self = self(), + spawn(fun() -> + {module,Mod} = code:load_binary(Mod, "", Code1), + Mod:f(Self) + end), + receive started -> ok end, + + true = code:delete(Mod), + true = check_old_code(Mod), + + Reload(), + + %% When loading the the module with the -on_load() function, + %% the reference to the old code would be lost. Make sure that + %% the old code is remembered and is still preventing the + %% purge. + false = code:soft_purge(Mod), + + %% Get rid of the old code. + Mod ! stop, + receive after 1 -> ok end, + true = code:soft_purge(Mod), + + %% Unload the version of the module with the -on_load() function. + true = code:delete(Mod), + true = code:soft_purge(Mod), + + ok. + %% Test that the native code of early loaded modules is loaded. native_early_modules(Config) when is_list(Config) -> diff --git a/lib/kernel/test/file_SUITE.erl b/lib/kernel/test/file_SUITE.erl index 5f049c6f99..c37d114a58 100644 --- a/lib/kernel/test/file_SUITE.erl +++ b/lib/kernel/test/file_SUITE.erl @@ -493,22 +493,13 @@ read_write_file(Config) when is_list(Config) -> %% Try writing and reading back some term SomeTerm = {"This term",{will,be},[written,$t,$o],1,file,[]}, - ok = ?FILE_MODULE:write_file(Name,term_to_binary(SomeTerm)), - {ok,Bin1} = ?FILE_MODULE:read_file(Name), - SomeTerm = binary_to_term(Bin1), + Bin1 = term_to_binary(SomeTerm), + ok = do_read_write_file(Name, Bin1), %% Try a "null" term NullTerm = [], - ok = ?FILE_MODULE:write_file(Name,term_to_binary(NullTerm)), - {ok,Bin2} = ?FILE_MODULE:read_file(Name), - NullTerm = binary_to_term(Bin2), - - %% Try some "complicated" types - BigNum = 123456789012345678901234567890, - ComplTerm = {self(),make_ref(),BigNum,3.14159}, - ok = ?FILE_MODULE:write_file(Name,term_to_binary(ComplTerm)), - {ok,Bin3} = ?FILE_MODULE:read_file(Name), - ComplTerm = binary_to_term(Bin3), + Bin2 = term_to_binary(NullTerm), + ok = do_read_write_file(Name, Bin2), %% Try reading a nonexistent file Name2 = filename:join(RootDir, @@ -519,25 +510,42 @@ read_write_file(Config) when is_list(Config) -> {error, enoent} = ?FILE_MODULE:read_file(''), %% Try writing to a bad filename - {error, enoent} = - ?FILE_MODULE:write_file("",term_to_binary(NullTerm)), + {error, enoent} = do_read_write_file("", Bin2), %% Try writing something else than a binary - {error, badarg} = ?FILE_MODULE:write_file(Name,{1,2,3}), - {error, badarg} = ?FILE_MODULE:write_file(Name,self()), + {error, badarg} = do_read_write_file(Name, {1,2,3}), + {error, badarg} = do_read_write_file(Name, self()), %% Some non-term binaries - ok = ?FILE_MODULE:write_file(Name,[]), - {ok,Bin4} = ?FILE_MODULE:read_file(Name), - 0 = byte_size(Bin4), + ok = do_read_write_file(Name, []), - ok = ?FILE_MODULE:write_file(Name,[Bin1,[],[[Bin2]]]), - {ok,Bin5} = ?FILE_MODULE:read_file(Name), - {Bin1,Bin2} = split_binary(Bin5,byte_size(Bin1)), + %% Write some iolists + ok = do_read_write_file(Name, [Bin1,[],[[Bin2]]]), + ok = do_read_write_file(Name, ["string",<<"binary">>]), + ok = do_read_write_file(Name, "pure string"), [] = flush(), ok. +do_read_write_file(Name, Data) -> + case ?FILE_MODULE:write_file(Name, Data) of + ok -> + BinData = iolist_to_binary(Data), + {ok,BinData} = ?FILE_MODULE:read_file(Name), + + ok = ?FILE_MODULE:write_file(Name, Data, []), + {ok,BinData} = ?FILE_MODULE:read_file(Name), + + ok = ?FILE_MODULE:write_file(Name, Data, [raw]), + {ok,BinData} = ?FILE_MODULE:read_file(Name), + + ok; + {error,_}=Res -> + Res = ?FILE_MODULE:write_file(Name, Data, []), + Res = ?FILE_MODULE:write_file(Name, Data, [raw]), + Res + end. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/lib/kernel/test/gen_tcp_api_SUITE.erl b/lib/kernel/test/gen_tcp_api_SUITE.erl index 77ec89b4f4..92a74465b7 100644 --- a/lib/kernel/test/gen_tcp_api_SUITE.erl +++ b/lib/kernel/test/gen_tcp_api_SUITE.erl @@ -38,7 +38,7 @@ t_local_basic/1, t_local_unbound/1, t_local_fdopen/1, t_local_fdopen_listen/1, t_local_fdopen_listen_unbound/1, t_local_fdopen_connect/1, t_local_fdopen_connect_unbound/1, - t_local_abstract/1]). + t_local_abstract/1, t_accept_inet6_tclass/1]). -export([getsockfd/0,closesockfd/1]). @@ -50,6 +50,7 @@ all() -> [{group, t_accept}, {group, t_connect}, {group, t_recv}, t_shutdown_write, t_shutdown_both, t_shutdown_error, t_shutdown_async, t_fdopen, t_fdconnect, t_implicit_inet6, + t_accept_inet6_tclass, {group, t_local}]. groups() -> @@ -521,6 +522,24 @@ local_handshake(S, SAddr, C, CAddr) -> SData = ok(gen_tcp:recv(C, length(SData))), ok. +t_accept_inet6_tclass(Config) when is_list(Config) -> + TClassOpt = {tclass,8#56 bsl 2}, % Expedited forwarding + case gen_tcp:listen(0, [inet6,TClassOpt]) of + {ok,L} -> + LPort = ok(inet:port(L)), + Loopback = {0,0,0,0,0,0,0,1}, + Sa = ok(gen_tcp:connect(Loopback, LPort, [])), + Sb = ok(gen_tcp:accept(L)), + [TClassOpt] = ok(inet:getopts(Sb, [tclass])), + ok = gen_tcp:close(Sb), + ok = gen_tcp:close(Sa), + ok = gen_tcp:close(L), + ok; + {error,_} -> + {skip,"IPv6 TCLASS not supported"} + end. + + %%% Utilities %% Calls M:F/length(A), which should return a timeout error, and complete diff --git a/lib/kernel/test/os_SUITE.erl b/lib/kernel/test/os_SUITE.erl index 19ab3713a1..e76d6ec482 100644 --- a/lib/kernel/test/os_SUITE.erl +++ b/lib/kernel/test/os_SUITE.erl @@ -25,7 +25,7 @@ -export([space_in_cwd/1, quoting/1, cmd_unicode/1, space_in_name/1, bad_command/1, find_executable/1, unix_comment_in_command/1, deep_list_command/1, large_output_command/1, background_command/0, background_command/1, - message_leak/1, perf_counter_api/1]). + message_leak/1, close_stdin/0, close_stdin/1, perf_counter_api/1]). -include_lib("common_test/include/ct.hrl"). @@ -37,7 +37,7 @@ all() -> [space_in_cwd, quoting, cmd_unicode, space_in_name, bad_command, find_executable, unix_comment_in_command, deep_list_command, large_output_command, background_command, message_leak, - perf_counter_api]. + close_stdin, perf_counter_api]. groups() -> []. @@ -54,7 +54,8 @@ init_per_group(_GroupName, Config) -> end_per_group(_GroupName, Config) -> Config. -init_per_testcase(background_command, Config) -> +init_per_testcase(TC, Config) + when TC =:= background_command; TC =:= close_stdin -> case os:type() of {win32, _} -> {skip,"Should not work on windows"}; @@ -294,7 +295,7 @@ message_leak(_Config) -> case os:type() of {unix, _} -> - os:cmd("while true; do echo hello; done&"), + os:cmd("for i in $(seq 1 100); do echo hello; done&"), [] = receive_all(); _ -> ok % Cannot background on non-unix @@ -302,6 +303,16 @@ message_leak(_Config) -> process_flag(trap_exit, false). +%% Test that os:cmd closes stdin of the program that is executed +close_stdin() -> + [{timetrap, {seconds, 5}}]. +close_stdin(Config) -> + DataDir = proplists:get_value(data_dir, Config), + Fds = filename:join(DataDir, "my_fds"), + + "-1" = os:cmd(Fds). + + %% Test that the os:perf_counter api works as expected perf_counter_api(_Config) -> diff --git a/lib/kernel/test/os_SUITE_data/Makefile.src b/lib/kernel/test/os_SUITE_data/Makefile.src index 912d0cbcb1..f83f781411 100644 --- a/lib/kernel/test/os_SUITE_data/Makefile.src +++ b/lib/kernel/test/os_SUITE_data/Makefile.src @@ -3,7 +3,7 @@ LD = @LD@ CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@ CROSSLDFLAGS = @CROSSLDFLAGS@ -PROGS = my_echo@exe@ +PROGS = my_echo@exe@ my_fds@exe@ all: $(PROGS) @@ -12,3 +12,9 @@ my_echo@exe@: my_echo@obj@ my_echo@obj@: my_echo.c $(CC) -c -o my_echo@obj@ $(CFLAGS) my_echo.c + +my_fds@exe@: my_fds@obj@ + $(LD) $(CROSSLDFLAGS) -o my_fds my_fds@obj@ @LIBS@ + +my_fds@obj@: my_fds.c + $(CC) -c -o my_fds@obj@ $(CFLAGS) my_fds.c diff --git a/lib/kernel/test/os_SUITE_data/my_fds.c b/lib/kernel/test/os_SUITE_data/my_fds.c new file mode 100644 index 0000000000..704a4d1e1d --- /dev/null +++ b/lib/kernel/test/os_SUITE_data/my_fds.c @@ -0,0 +1,9 @@ +#include <stdio.h> + +int +main(int argc, char** argv) +{ + char buff[1]; + int res = read(stdin, buff, 1); + printf("%d", res); +} diff --git a/lib/kernel/vsn.mk b/lib/kernel/vsn.mk index cdd200a234..d3b2d18ae5 100644 --- a/lib/kernel/vsn.mk +++ b/lib/kernel/vsn.mk @@ -1 +1 @@ -KERNEL_VSN = 5.0.2 +KERNEL_VSN = 5.1 diff --git a/lib/mnesia/doc/src/notes.xml b/lib/mnesia/doc/src/notes.xml index 7d8e8d0c44..e621bd593c 100644 --- a/lib/mnesia/doc/src/notes.xml +++ b/lib/mnesia/doc/src/notes.xml @@ -39,7 +39,21 @@ thus constitutes one section in this document. The title of each section is the version number of Mnesia.</p> - <section><title>Mnesia 4.14</title> + <section><title>Mnesia 4.14.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> Correct some minor documentation issues. </p> + <p> + Own Id: OTP-13891</p> + </item> + </list> + </section> + +</section> + +<section><title>Mnesia 4.14</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/mnesia/vsn.mk b/lib/mnesia/vsn.mk index fb4200f62d..f08e364276 100644 --- a/lib/mnesia/vsn.mk +++ b/lib/mnesia/vsn.mk @@ -1 +1 @@ -MNESIA_VSN = 4.14 +MNESIA_VSN = 4.14.1 diff --git a/lib/observer/doc/src/notes.xml b/lib/observer/doc/src/notes.xml index f79f75fead..659eb28292 100644 --- a/lib/observer/doc/src/notes.xml +++ b/lib/observer/doc/src/notes.xml @@ -32,6 +32,22 @@ <p>This document describes the changes made to the Observer application.</p> +<section><title>Observer 2.2.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed error handling in observer when mnesia tables was + requested and not available.</p> + <p> + Own Id: OTP-13845 Aux Id: ERL-237 </p> + </item> + </list> + </section> + +</section> + <section><title>Observer 2.2.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/observer/doc/src/ttb.xml b/lib/observer/doc/src/ttb.xml index 42b0fa1d8a..7cd15e15d3 100644 --- a/lib/observer/doc/src/ttb.xml +++ b/lib/observer/doc/src/ttb.xml @@ -114,7 +114,8 @@ ttb:p(all, call).</input></pre> <v>Opt = {file,Client} | {handler, FormatHandler} | {process_info,PI} | shell | {shell, ShellSpec} | {timer, TimerSpec} | {overload_check, {MSec, Module, Function}} | - {flush, MSec} | resume | {resume, FetchTimeout}</v> + {flush, MSec} | resume | {resume, FetchTimeout} | + {queue_size, QueueSize}</v> <v>TimerSpec = MSec | {MSec, StopOpts}</v> <v>MSec = FetchTimeout = integer()</v> <v>Module = Function = atom() </v> @@ -126,6 +127,7 @@ ttb:p(all, call).</input></pre> <v>FormatHandler = See format/2</v> <v>PI = true | false </v> <v>ShellSpec = true | false | only</v> + <v>QueueSize = non_neg_integer()</v> </type> <desc> <p>Starts a file trace port on all specified nodes @@ -147,6 +149,18 @@ ttb:p(all, call).</input></pre> <c>Client</c> must be <c>{local, File}</c>. All trace information is then sent to the trace control node where it is written to file.</p></item> + <tag><c>queue_size</c></tag> + <item><p>When tracing to shell or <c>{local,File}</c>, an ip + trace driver is used internally. The ip trace driver has a + queue of maximum <c>QueueSize</c> messages waiting to be + delivered. If the driver cannot deliver messages as fast as + they are produced, the queue size might be exceeded and + messages are dropped. This parameter is optional, and is + only useful if many <c>{drop,N}</c> trace messages are + received by the trace handler. It has no meaning if shell + or <c>{local,File}</c> is not used. See + <seealso marker="runtime_tools:dbg#trace_port/2">dbg:trace_port/2</seealso> + for more information about the ip trace driver.</p></item> <tag><c>process_info</c></tag> <item><p>Indicates if process information is to be collected. If <c>PI = true</c> (which is diff --git a/lib/observer/src/ttb.erl b/lib/observer/src/ttb.erl index ac6c4572eb..87a50e046b 100644 --- a/lib/observer/src/ttb.erl +++ b/lib/observer/src/ttb.erl @@ -78,6 +78,11 @@ do_tracer(Nodes0,PI,Client,Traci) -> do_tracer(Clients,PI,Traci) -> Shell = proplists:get_value(shell, Traci, false), + IpPortSpec = + case proplists:get_value(queue_size, Traci) of + undefined -> 0; + QS -> {0,QS} + end, DefShell = fun(Trace) -> dbg:dhandler(Trace, standard_io) end, {ClientSucc,Succ} = lists:foldl( @@ -98,7 +103,7 @@ do_tracer(Clients,PI,Traci) -> [_,H] = string:tokens(atom_to_list(N),"@"), H end, - case catch dbg:tracer(N,port,dbg:trace_port(ip,0)) of + case catch dbg:tracer(N,port,dbg:trace_port(ip,IpPortSpec)) of {ok,N} -> {ok,Port} = dbg:trace_port_control(N,get_listen_port), {ok,T} = dbg:get_tracer(N), @@ -160,6 +165,8 @@ opt([{resume,MSec}|O],{PI,Client,Traci}) -> opt(O,{PI,Client,[{resume, {true, MSec}}|Traci]}); opt([{flush,MSec}|O],{PI,Client,Traci}) -> opt(O,{PI,Client,[{flush, MSec}|Traci]}); +opt([{queue_size,QueueSize}|O],{PI,Client,Traci}) -> + opt(O,{PI,Client,[{queue_size,QueueSize}|Traci]}); opt([],Opt) -> ensure_opt(Opt). diff --git a/lib/observer/vsn.mk b/lib/observer/vsn.mk index 9a7c6a546f..444089151e 100644 --- a/lib/observer/vsn.mk +++ b/lib/observer/vsn.mk @@ -1 +1 @@ -OBSERVER_VSN = 2.2.1 +OBSERVER_VSN = 2.2.2 diff --git a/lib/odbc/doc/src/notes.xml b/lib/odbc/doc/src/notes.xml index 40f9339028..7fb19a072e 100644 --- a/lib/odbc/doc/src/notes.xml +++ b/lib/odbc/doc/src/notes.xml @@ -32,7 +32,23 @@ <p>This document describes the changes made to the odbc application. </p> - <section><title>ODBC 2.11.2</title> + <section><title>ODBC 2.11.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + ODBC build configure has been updated to accept Mac OS X + El Capitan. Fixed by Lee Bannard.</p> + <p> + Own Id: OTP-13781</p> + </item> + </list> + </section> + +</section> + +<section><title>ODBC 2.11.2</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/odbc/vsn.mk b/lib/odbc/vsn.mk index 957c6b42eb..a7c8f8079b 100644 --- a/lib/odbc/vsn.mk +++ b/lib/odbc/vsn.mk @@ -1 +1 @@ -ODBC_VSN = 2.11.2 +ODBC_VSN = 2.11.3 diff --git a/lib/parsetools/doc/src/notes.xml b/lib/parsetools/doc/src/notes.xml index b826b4d03a..30a9374e81 100644 --- a/lib/parsetools/doc/src/notes.xml +++ b/lib/parsetools/doc/src/notes.xml @@ -31,6 +31,22 @@ </header> <p>This document describes the changes made to the Parsetools application.</p> +<section><title>Parsetools 2.1.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> Yecc generates Dialyzer suppressions to avoid + warnings when operator precedence declarations are used. + </p> + <p> + Own Id: OTP-13681</p> + </item> + </list> + </section> + +</section> + <section><title>Parsetools 2.1.2</title> <section><title>Improvements and New Features</title> diff --git a/lib/parsetools/src/yecc.erl b/lib/parsetools/src/yecc.erl index 1b426141a1..f6b80eb1b4 100644 --- a/lib/parsetools/src/yecc.erl +++ b/lib/parsetools/src/yecc.erl @@ -1978,7 +1978,8 @@ output_goto(St, [{_Nonterminal, []} | Go], StateInfo) -> output_goto(St, Go, StateInfo); output_goto(St0, [{Nonterminal, List} | Go], StateInfo) -> F = function_name(yeccgoto, Nonterminal), - St10 = output_goto1(St0, List, F, StateInfo, true), + St05 = fwrite(St0, <<"-dialyzer({nowarn_function, ~w/7}).\n">>, [F]), + St10 = output_goto1(St05, List, F, StateInfo, true), St = output_goto_fini(F, Nonterminal, St10), output_goto(St, Go, StateInfo); output_goto(St, [], _StateInfo) -> diff --git a/lib/parsetools/test/yecc_SUITE.erl b/lib/parsetools/test/yecc_SUITE.erl index 3710569aba..e91ddb11d1 100644 --- a/lib/parsetools/test/yecc_SUITE.erl +++ b/lib/parsetools/test/yecc_SUITE.erl @@ -342,7 +342,7 @@ syntax(Config) when is_list(Config) -> {L2,_,{bad_inline,{yeccpars2_2_,1}}}]}], []} = compile:file(Parserfile1, [basic_validation,return]), ?line L1 = 31 + SzYeccPre, - ?line L2 = 38 + SzYeccPre + ?line L2 = 39 + SzYeccPre end(), %% Bad macro in action. OTP-7224. @@ -360,7 +360,7 @@ syntax(Config) when is_list(Config) -> {L2,_,{bad_inline,{yeccpars2_2_,1}}}]}], []} = compile:file(Parserfile1, [basic_validation,return]), ?line L1 = 31 + SzYeccPre, - ?line L2 = 38 + SzYeccPre + ?line L2 = 39 + SzYeccPre end(), %% Check line numbers. OTP-7224. @@ -1623,7 +1623,7 @@ otp_7292(Config) when is_list(Config) -> [{_,[{16,_,{unused_function,{foo,0}}}]}]} = compile:file(Parserfile1, [basic_validation, return]), L1 = 41 + SzYeccPre, - L2 = 48 + SzYeccPre + L2 = 49 + SzYeccPre end(), YeccPre = filename:join(Dir, "yeccpre.hrl"), @@ -1641,7 +1641,7 @@ otp_7292(Config) when is_list(Config) -> [{_,[{16,_,{unused_function,{foo,0}}}]}]} = compile:file(Parserfile1, [basic_validation, return]), ?line L1 = 40 + SzYeccPre, - ?line L2 = 47 + SzYeccPre + ?line L2 = 48 + SzYeccPre end(), file:delete(YeccPre), diff --git a/lib/parsetools/vsn.mk b/lib/parsetools/vsn.mk index befdd82d6e..07cd959c98 100644 --- a/lib/parsetools/vsn.mk +++ b/lib/parsetools/vsn.mk @@ -1 +1 @@ -PARSETOOLS_VSN = 2.1.2 +PARSETOOLS_VSN = 2.1.3 diff --git a/lib/public_key/doc/src/public_key_app.xml b/lib/public_key/doc/src/public_key_app.xml index 1f87932b6c..923a9f1dfb 100644 --- a/lib/public_key/doc/src/public_key_app.xml +++ b/lib/public_key/doc/src/public_key_app.xml @@ -61,7 +61,7 @@ <section> <title>DEPENDENCIES</title> <p>The <c>public_key</c> application uses the - Crypto application to preform cryptographic operations and the + Crypto application to perform cryptographic operations and the ASN-1 application to handle PKIX-ASN-1 specifications, hence these applications must be loaded for the <c>public_key</c> application to work. In an embedded environment this means they must be started with @@ -72,7 +72,7 @@ <section> <title>ERROR LOGGER AND EVENT HANDLERS</title> <p> The <c>public_key</c> application is a library application - and does not use the error logger. The functions will either sucssed + and does not use the error logger. The functions will either succeed or fail with a runtime error. </p> </section> diff --git a/lib/reltool/doc/src/notes.xml b/lib/reltool/doc/src/notes.xml index 25d983afd3..2365a68feb 100644 --- a/lib/reltool/doc/src/notes.xml +++ b/lib/reltool/doc/src/notes.xml @@ -38,7 +38,38 @@ thus constitutes one section in this document. The title of each section is the version number of Reltool.</p> - <section><title>Reltool 0.7.1</title> + <section><title>Reltool 0.7.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Dependencies specified in .app files would earlier only + be followed for applications that are included in a 'rel' + spec in the reltool config. For other applications, only + xref would decide the dependencies.</p> + <p> + Some dependency chains would even be missed for + applications that are included in a 'rel' spec in the + reltool config. E.g.</p> + <p> + <list> <item>Application x has y as included application, + and y in turn has z as included application. Then z is + not included. </item> <item>Application x has y in its + 'applications' tag in the .app file, and y in turn has z + as included application. Then z is not included.</item> + </list></p> + <p> + These bugs are now corrected.</p> + <p> + Own Id: OTP-11993</p> + </item> + </list> + </section> + +</section> + +<section><title>Reltool 0.7.1</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/reltool/vsn.mk b/lib/reltool/vsn.mk index 76f69fd294..2b23ff6f20 100644 --- a/lib/reltool/vsn.mk +++ b/lib/reltool/vsn.mk @@ -1 +1 @@ -RELTOOL_VSN = 0.7.1 +RELTOOL_VSN = 0.7.2 diff --git a/lib/runtime_tools/doc/src/dbg.xml b/lib/runtime_tools/doc/src/dbg.xml index db04bfdf7b..95f74d4607 100644 --- a/lib/runtime_tools/doc/src/dbg.xml +++ b/lib/runtime_tools/doc/src/dbg.xml @@ -954,7 +954,7 @@ Error: fun containing local erlang function calls ('is_atomm' called in guard)\ as the tuple <c>{drop, N}</c> where <c>N</c> is the number of consecutive messages dropped. In case of heavy tracing, drop's are likely to occur, and they surely occur if no client is reading the trace - messages.</p> + messages. The default value of <c>QueSize</c> is 200.</p> </desc> </func> <func> diff --git a/lib/runtime_tools/doc/src/notes.xml b/lib/runtime_tools/doc/src/notes.xml index b6dfb2dd28..0b830593b4 100644 --- a/lib/runtime_tools/doc/src/notes.xml +++ b/lib/runtime_tools/doc/src/notes.xml @@ -32,6 +32,20 @@ <p>This document describes the changes made to the Runtime_Tools application.</p> +<section><title>Runtime_Tools 1.10.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> Correct some minor documentation issues. </p> + <p> + Own Id: OTP-13891</p> + </item> + </list> + </section> + +</section> + <section><title>Runtime_Tools 1.10</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/runtime_tools/src/dbg.erl b/lib/runtime_tools/src/dbg.erl index c0d4665bda..f17aa528ed 100644 --- a/lib/runtime_tools/src/dbg.erl +++ b/lib/runtime_tools/src/dbg.erl @@ -427,7 +427,7 @@ trace_port(file, Filename) -> trace_port1(file, Filename, nowrap); trace_port(ip, Portno) when is_integer(Portno) -> - trace_port(ip,{Portno,50}); + trace_port(ip,{Portno,200}); trace_port(ip, {Portno, Qsiz}) when is_integer(Portno), is_integer(Qsiz) -> fun() -> diff --git a/lib/runtime_tools/vsn.mk b/lib/runtime_tools/vsn.mk index b33f6f4721..0fc86e42f7 100644 --- a/lib/runtime_tools/vsn.mk +++ b/lib/runtime_tools/vsn.mk @@ -1 +1 @@ -RUNTIME_TOOLS_VSN = 1.10 +RUNTIME_TOOLS_VSN = 1.10.1 diff --git a/lib/sasl/doc/src/notes.xml b/lib/sasl/doc/src/notes.xml index bd713af932..055d433524 100644 --- a/lib/sasl/doc/src/notes.xml +++ b/lib/sasl/doc/src/notes.xml @@ -31,6 +31,59 @@ </header> <p>This document describes the changes made to the SASL application.</p> +<section><title>SASL 3.0.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Improved dirty scheduler support. A purge of a module can + now be performed without having to wait for completion of + all ongoing dirty NIF calls.</p> + <p> + Note that when enabling support for dirty schedulers, a + new purge strategy will as of ERTS version 8.1 be + enabled. This new strategy is not fully backwards + compatible with the strategy used by default. For more + information see the documentation of <seealso + marker="erts:erlang#check_process_code/3"><c>erlang:check_process_code/3</c></seealso>.</p> + <p> + Own Id: OTP-13808 Aux Id: OTP-13833 </p> + </item> + <item> + <p> + A new purge strategy has been introduced. The new + strategy will by default be disabled during the OTP 19 + release, but will be the only strategy available as of + the OTP 20 release.</p> + <p> + The new strategy is slightly incompatible with the + strategy being used by default in OTP 19. Using the + default strategy, processes holding <c>fun</c>s that + refer to the module being purged either fail a soft + purge, or will be killed during a hard purge. The new + strategy completely ignores <c>fun</c>s. If <c>fun</c>s + referring to the code being purged exist, and are used + after a purge, an exception will be raised upon usage. + That is, the behavior will be exactly the same as the + case when a <c>fun</c> is received by a process after the + purge.</p> + <p> + The new strategy can optionally be enabled when building + OTP during OTP 19, and will automatically be enabled if + the runtime system is built with support for dirty + schedulers.</p> + <p> + For more information see the documentation of <seealso + marker="erts:erlang#check_process_code/3"><c>erlang:check_process_code/3</c></seealso>.</p> + <p> + Own Id: OTP-13833</p> + </item> + </list> + </section> + +</section> + <section><title>SASL 3.0</title> <section><title>Improvements and New Features</title> diff --git a/lib/sasl/vsn.mk b/lib/sasl/vsn.mk index fd0fc9b8b5..a7d7c09cde 100644 --- a/lib/sasl/vsn.mk +++ b/lib/sasl/vsn.mk @@ -1 +1 @@ -SASL_VSN = 3.0 +SASL_VSN = 3.0.1 diff --git a/lib/snmp/doc/src/notes.xml b/lib/snmp/doc/src/notes.xml index 0f5c35b300..3323d32878 100644 --- a/lib/snmp/doc/src/notes.xml +++ b/lib/snmp/doc/src/notes.xml @@ -34,7 +34,35 @@ </header> - <section><title>SNMP 5.2.3</title> + <section><title>SNMP 5.2.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> Correct bugs when path to mib or idl spec files + contains UTF-8 characters. </p> + <p> + Own Id: OTP-13718 Aux Id: ERL-179 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Solves snmp config string handling as reported by ERL-164 + and solved by PR-1100</p> + <p> + Own Id: OTP-13706</p> + </item> + </list> + </section> + +</section> + +<section><title>SNMP 5.2.3</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/snmp/src/app/snmp.app.src b/lib/snmp/src/app/snmp.app.src index b593e9ea84..d25f66f44a 100644 --- a/lib/snmp/src/app/snmp.app.src +++ b/lib/snmp/src/app/snmp.app.src @@ -23,12 +23,12 @@ {vsn, "%VSN%"}, {modules, [ %% Compiler modules (not in the runtime part of the app) -% snmpc, -% snmpc_lib, -% snmpc_mib_gram, -% snmpc_mib_to_hrl, -% snmpc_misc, -% snmpc_tok, + snmpc, + snmpc_lib, + snmpc_mib_gram, + snmpc_mib_to_hrl, + snmpc_misc, + snmpc_tok, %% Application modules snmp, diff --git a/lib/snmp/src/compile/snmpc.erl b/lib/snmp/src/compile/snmpc.erl index db1f9ee61b..d86692aaf6 100644 --- a/lib/snmp/src/compile/snmpc.erl +++ b/lib/snmp/src/compile/snmpc.erl @@ -64,7 +64,7 @@ compile(Input, _Output, Options) -> {ok, _} -> ok; {error, Reason} -> - io:format("~p", [Reason]), + io:format("~tp", [Reason]), error end. @@ -126,7 +126,14 @@ compile(FileName) -> %%---------------------------------------------------------------------- compile(FileName, Options) when is_list(FileName) -> - true = snmpc_misc:is_string(FileName), + case snmpc_misc:check_file(FileName) of + true -> + compile_1(FileName, Options); + false -> + {error, {invalid_file, FileName}} + end. + +compile_1(FileName, Options) -> DefOpts = [{deprecated, true}, {group_check, true}, {i, ["./"]}, diff --git a/lib/snmp/src/compile/snmpc_misc.erl b/lib/snmp/src/compile/snmpc_misc.erl index 933d629746..312074f2e7 100644 --- a/lib/snmp/src/compile/snmpc_misc.erl +++ b/lib/snmp/src/compile/snmpc_misc.erl @@ -29,7 +29,7 @@ bits_to_int/2, ensure_trailing_dir_delimiter/1, foreach/3, - is_string/1, + check_file/1, read_mib/1, read_noexit/2, strip_extension_from_filename/2, @@ -86,21 +86,21 @@ to_upper([C|Cs]) -> [C|to_upper(Cs)]; to_upper([]) -> []. -is_string([]) -> true; -is_string([Tkn | Str]) - when is_integer(Tkn) andalso (Tkn >= 0) andalso (Tkn =< 255) -> - is_string(Str); -is_string(_) -> false. - - +check_file(FileName) -> + case filename:extension(FileName) of + ".mib" -> + filelib:is_regular(FileName); + _ -> + filelib:is_regular(FileName ++ ".mib") + end. + + foreach(Function, ExtraArgs, [H | T]) -> apply(Function, [H | ExtraArgs]), foreach(Function, ExtraArgs, T); foreach(_Function, _ExtraArgs, []) -> true. - - %%---------------------------------------------------------------------- %% Returns: {ok, Mib}|{error, Reason} %% The reason for having the function if this module is: diff --git a/lib/snmp/test/snmp_agent_test.erl b/lib/snmp/test/snmp_agent_test.erl index 8ae495bb1b..3c1a6f2afd 100644 --- a/lib/snmp/test/snmp_agent_test.erl +++ b/lib/snmp/test/snmp_agent_test.erl @@ -647,22 +647,22 @@ init_per_group(GroupName, Config) -> snmp_test_lib:init_group_top_dir(GroupName, Config). init_per_group_ipv6(GroupName, Config, Init) -> + {ok, Hostname0} = inet:gethostname(), case ct:require(ipv6_hosts) of ok -> - case gen_udp:open(0, [inet6]) of - {ok, S} -> - ok = gen_udp:close(S), - Init( - snmp_test_lib:init_group_top_dir( - GroupName, - [{ipfamily, inet6}, - {ip, ?LOCALHOST(inet6)} - | lists:keydelete(ip, 1, Config)])); - {error, _} -> - {skip, "Host seems to not support IPv6"} - end; + case lists:member(list_to_atom(Hostname0), ct:get_config(ipv6_hosts)) of + true -> + Init( + snmp_test_lib:init_group_top_dir( + GroupName, + [{ipfamily, inet6}, + {ip, ?LOCALHOST(inet6)} + | lists:keydelete(ip, 1, Config)])); + false -> + {skip, "Host does not support IPV6"} + end; _ -> - {skip, "Host does not support IPV6"} + {skip, "Test config ipv6_hosts is missing"} end. end_per_group(all_tcs, Config) -> diff --git a/lib/snmp/test/snmp_app_test.erl b/lib/snmp/test/snmp_app_test.erl index 6e7e85d3b4..5e69866f9a 100644 --- a/lib/snmp/test/snmp_app_test.erl +++ b/lib/snmp/test/snmp_app_test.erl @@ -23,366 +23,29 @@ %%---------------------------------------------------------------------- -module(snmp_app_test). --export([ - all/0, groups/0, - init_per_group/2, end_per_group/2, - init_per_suite/1, end_per_suite/1, - init_per_testcase/2, end_per_testcase/2, +%% Note: This directive should only be used in test suites. +-compile(export_all). - fields/1, - modules/1, - exportall/1, - app_depend/1, - - start_and_stop_empty/1, - start_and_stop_with_agent/1, - start_and_stop_with_manager/1, - start_and_stop_with_agent_and_manager/1, - start_epmty_and_then_agent_and_manager_and_stop/1, - start_with_agent_and_then_manager_and_stop/1, - start_with_manager_and_then_agent_and_stop/1 - ]). - - --include_lib("kernel/include/file.hrl"). -include_lib("common_test/include/ct.hrl"). --include("snmp_test_lib.hrl"). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%-------------------------------------------------------------------- +%% Common Test interface functions ----------------------------------- +%%-------------------------------------------------------------------- all() -> - Cases = - [ - fields, - modules, - exportall, - app_depend, - {group, start_and_stop} - ], - Cases. - -groups() -> - [{start_and_stop, [], - [start_and_stop_empty, - start_and_stop_with_agent, - start_and_stop_with_manager, - start_and_stop_with_agent_and_manager, - start_epmty_and_then_agent_and_manager_and_stop, - start_with_agent_and_then_manager_and_stop, - start_with_manager_and_then_agent_and_stop]}]. - -init_per_group(_GroupName, Config) -> - Config. - -end_per_group(_GroupName, Config) -> - Config. - - -init_per_suite(Config) when is_list(Config) -> - ?DISPLAY_SUITE_INFO(), - - %% Note that part of this stuff (the suite top dir creation) - %% may already be done (if we run the entire snmp suite). - - PrivDir = ?config(priv_dir, Config), - TopDir = filename:join(PrivDir, app), - case file:make_dir(TopDir) of - ok -> - ok; - {error, eexist} -> - ok; - Error -> - fail({failed_creating_subsuite_top_dir, Error}) - end, - AppFile = - case is_app() of - {ok, File} -> - io:format("File: ~n~p~n", [File]), - snmp:print_version_info(), - File; - {error, Reason} -> - fail(Reason) - end, - [{app_topdir, TopDir}, {app_file, AppFile} | Config]. - - -is_app() -> - is_app(?APPLICATION). - -is_app(App) -> - LibDir = code:lib_dir(App), - File = filename:join([LibDir, "ebin", atom_to_list(App) ++ ".app"]), - case file:consult(File) of - {ok, [{application, App, AppFile}]} -> - {ok, AppFile}; - Error -> - {error, {invalid_format, Error}} - end. - -end_per_suite(suite) -> []; -end_per_suite(doc) -> []; -end_per_suite(Config) when is_list(Config) -> - Config. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -%% Test server callbacks -init_per_testcase(_Case, Config) -> - Config. - -end_per_testcase(_Case, Config) -> - Config. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -fields(suite) -> - []; -fields(doc) -> - []; -fields(Config) when is_list(Config) -> - AppFile = key1search(app_file, Config), - Fields = [vsn, description, modules, registered, applications], - case check_fields(Fields, AppFile, []) of - [] -> - ok; - Missing -> - fail({missing_fields, Missing}) - end. - -check_fields([], _AppFile, Missing) -> - Missing; -check_fields([Field|Fields], AppFile, Missing) -> - check_fields(Fields, AppFile, check_field(Field, AppFile, Missing)). - -check_field(Name, AppFile, Missing) -> - io:format("checking field: ~p~n", [Name]), - case lists:keymember(Name, 1, AppFile) of - true -> - Missing; - false -> - [Name|Missing] - end. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -modules(suite) -> - []; -modules(doc) -> - []; -modules(Config) when is_list(Config) -> - AppFile = key1search(app_file, Config), - Mods = key1search(modules, AppFile), - EbinList = get_ebin_mods(snmp), - case missing_modules(Mods, EbinList, []) of - [] -> - ok; - Missing -> - fail({missing_modules, Missing}) - end, - Allowed = [snmpc, - snmpc_lib, - snmpc_misc, - snmpc_mib_gram, - snmpc_mib_to_hrl, - snmpc_tok], - case extra_modules(Mods, EbinList, Allowed, []) of - [] -> - ok; - Extra -> - fail({extra_modules, Extra}) - end, - {ok, Mods}. - -get_ebin_mods(App) -> - LibDir = code:lib_dir(App), - EbinDir = filename:join([LibDir,"ebin"]), - {ok, Files0} = file:list_dir(EbinDir), - Files1 = [lists:reverse(File) || File <- Files0], - [list_to_atom(lists:reverse(Name)) || [$m,$a,$e,$b,$.|Name] <- Files1]. - - -missing_modules([], _Ebins, Missing) -> - Missing; -missing_modules([Mod|Mods], Ebins, Missing) -> - case lists:member(Mod, Ebins) of - true -> - missing_modules(Mods, Ebins, Missing); - false -> - io:format("missing module: ~p~n", [Mod]), - missing_modules(Mods, Ebins, [Mod|Missing]) - end. - - -extra_modules(_Mods, [], Allowed, Extra) -> - Extra--Allowed; -extra_modules(Mods, [Mod|Ebins], Allowed, Extra) -> - case lists:member(Mod, Mods) of - true -> - extra_modules(Mods, Ebins, Allowed, Extra); - false -> - io:format("superfluous module: ~p~n", [Mod]), - extra_modules(Mods, Ebins, Allowed, [Mod|Extra]) - end. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - - -exportall(suite) -> - []; -exportall(doc) -> - []; -exportall(Config) when is_list(Config) -> - AppFile = key1search(app_file, Config), - Mods = key1search(modules, AppFile), - check_export_all(Mods). - - -check_export_all([]) -> - ok; -check_export_all([Mod|Mods]) -> - case (catch apply(Mod, module_info, [compile])) of - {'EXIT', {undef, _}} -> - check_export_all(Mods); - O -> - case lists:keysearch(options, 1, O) of - false -> - check_export_all(Mods); - {value, {options, List}} -> - case lists:member(export_all, List) of - true -> - fail({export_all, Mod}); - false -> - check_export_all(Mods) - end - end - end. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -app_depend(suite) -> - []; -app_depend(doc) -> - []; -app_depend(Config) when is_list(Config) -> - AppFile = key1search(app_file, Config), - Apps = key1search(applications, AppFile), - check_apps(Apps). - - -check_apps([]) -> - ok; -check_apps([App|Apps]) -> - case is_app(App) of - {ok, _} -> - check_apps(Apps); - Error -> - throw({error, {missing_app, {App, Error}}}) - end. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_and_stop_empty(suite) -> - []; -start_and_stop_empty(doc) -> - ["Start and stop the application empty (no configured components)"]; -start_and_stop_empty(Config) when is_list(Config) -> - ?line false = ?IS_SNMP_RUNNING(), - - ?line ok = snmp:start(), - - ?line true = ?IS_SNMP_RUNNING(), - - ?line ok = snmp:stop(), - - ?line false = ?IS_SNMP_RUNNING(), - - ok. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_and_stop_with_agent(suite) -> - []; -start_and_stop_with_agent(doc) -> - ["Start and stop the application with the agent pre-configured"]; -start_and_stop_with_agent(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_and_stop_with_manager(suite) -> - []; -start_and_stop_with_manager(doc) -> - ["Start and stop the application with the manager pre-configured"]; -start_and_stop_with_manager(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_and_stop_with_agent_and_manager(suite) -> - []; -start_and_stop_with_agent_and_manager(doc) -> - ["Start and stop the application with both the agent " - "and the manager pre-configured"]; -start_and_stop_with_agent_and_manager(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_epmty_and_then_agent_and_manager_and_stop(suite) -> - []; -start_epmty_and_then_agent_and_manager_and_stop(doc) -> - ["Start the application empty, then start the agent and then " - "the manager and then stop the application"]; -start_epmty_and_then_agent_and_manager_and_stop(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_with_agent_and_then_manager_and_stop(suite) -> - []; -start_with_agent_and_then_manager_and_stop(doc) -> - ["Start the application with the agent pre-configured, " - "then start the manager and then stop the application"]; -start_with_agent_and_then_manager_and_stop(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -start_with_manager_and_then_agent_and_stop(suite) -> - []; -start_with_manager_and_then_agent_and_stop(doc) -> - ["Start the application with the manager pre-configured, " - "then start the agent and then stop the application"]; -start_with_manager_and_then_agent_and_stop(Config) when is_list(Config) -> - ?SKIP(not_implemented_yet). - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - - -fail(Reason) -> - exit({suite_failed, Reason}). - -key1search(Key, L) -> - case lists:keysearch(Key, 1, L) of - undefined -> - fail({not_found, Key, L}); - {value, {Key, Value}} -> - Value - end. + [ + app, + appup + ]. + +%%-------------------------------------------------------------------- +%% Test Cases -------------------------------------------------------- +%%-------------------------------------------------------------------- +app() -> + [{doc, "Test that the snmp app file is ok"}]. +app(Config) when is_list(Config) -> + ok = test_server:app_test(snmp). +%%-------------------------------------------------------------------- +appup() -> + [{doc, "Test that the snmp appup file is ok"}]. +appup(Config) when is_list(Config) -> + ok = test_server:appup_test(snmp). diff --git a/lib/snmp/test/snmp_manager_test.erl b/lib/snmp/test/snmp_manager_test.erl index d17882e765..71f4017d8b 100644 --- a/lib/snmp/test/snmp_manager_test.erl +++ b/lib/snmp/test/snmp_manager_test.erl @@ -583,38 +583,38 @@ init_per_group(event_tests_mt = GroupName, Config) -> GroupName, [{manager_net_if_module, snmpm_net_if_mt} | Config]); init_per_group(ipv6_mt = GroupName, Config) -> + {ok, Hostname0} = inet:gethostname(), case ct:require(ipv6_hosts) of ok -> - case gen_udp:open(0, [inet6]) of - {ok, S} -> - ok = gen_udp:close(S), + case lists:member(list_to_atom(Hostname0), ct:get_config(ipv6_hosts)) of + true -> ipv6_init( snmp_test_lib:init_group_top_dir( GroupName, [{manager_net_if_module, snmpm_net_if_mt} | Config])); - {error, _} -> - {skip, "Host seems to not support IPv6"} + false -> + {skip, "Host does not support IPv6"} end; _ -> - {skip, "Host does not support IPV6"} + {skip, "Test config ipv6_hosts is missing"} end; init_per_group(ipv6 = GroupName, Config) -> + {ok, Hostname0} = inet:gethostname(), case ct:require(ipv6_hosts) of ok -> - case gen_udp:open(0, [inet6]) of - {ok, S} -> - ok = gen_udp:close(S), + case lists:member(list_to_atom(Hostname0), ct:get_config(ipv6_hosts)) of + true -> ipv6_init(snmp_test_lib:init_group_top_dir(GroupName, Config)); - {error, _} -> - {skip, "Host seems to not support IPv6"} + false -> + {skip, "Host does not support IPv6"} end; _ -> - {skip, "Host does not support IPV6"} + {skip, "Test config ipv6_hosts is missing"} end; init_per_group(GroupName, Config) -> snmp_test_lib:init_group_top_dir(GroupName, Config). - + end_per_group(_GroupName, Config) -> %% Do we really need to do this? lists:keydelete(snmp_group_top_dir, 1, Config). diff --git a/lib/snmp/test/snmp_to_snmpnet_SUITE.erl b/lib/snmp/test/snmp_to_snmpnet_SUITE.erl index ac9e37bc8b..24c14d86ea 100644 --- a/lib/snmp/test/snmp_to_snmpnet_SUITE.erl +++ b/lib/snmp/test/snmp_to_snmpnet_SUITE.erl @@ -121,14 +121,14 @@ init_per_group(_, Config) -> Config. init_per_group_ipv6(Families, Config) -> + {ok, Hostname0} = inet:gethostname(), case ct:require(ipv6_hosts) of ok -> - case gen_udp:open(0, [inet6]) of - {ok, S} -> - ok = gen_udp:close(S), + case lists:member(list_to_atom(Hostname0), ct:get_config(ipv6_hosts)) of + true -> init_per_group_ip(Families, Config); - {error, _} -> - {skip, "Host seems to not support IPv6"} + false -> + {skip, "Host does not support IPv6"} end; _ -> {skip, "Test config ipv6_hosts is missing"} diff --git a/lib/snmp/vsn.mk b/lib/snmp/vsn.mk index f95b428290..28eba0d0d6 100644 --- a/lib/snmp/vsn.mk +++ b/lib/snmp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = snmp -SNMP_VSN = 5.2.3 +SNMP_VSN = 5.2.4 PRE_VSN = APP_VSN = "$(APPLICATION)-$(SNMP_VSN)$(PRE_VSN)" diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index f9d11b2a60..b990c18e9a 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -30,6 +30,68 @@ <file>notes.xml</file> </header> +<section><title>Ssh 4.3.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Upgrade of an established client connection could crash + because the ssh client supervisors children had wrong + type. This is fixed now.</p> + <p> + Own Id: OTP-13782 Aux Id: seq13158 </p> + </item> + <item> + <p> + Partly checks the public key early in public key + authorization</p> + <p> + Own Id: OTP-13847 Aux Id: + defensics-ssh3.1.0-190243,205277,219318 </p> + </item> + <item> + <p> + Corrected handling of SHA for ECDSA (Elliptic curve + public keys)</p> + <p> + Own Id: OTP-13850 Aux Id: defensics-ssh3.1.0-214168 </p> + </item> + <item> + <p> + Problems found by test suites as well as by + Codenomicon/Defensics fixed: - reduce max random padding + to 15 bytes (Codenomicon/Defensics) - inclomplete pdu + handling (Codenomicon/Defensics) - badmatch in test suite + - non-blocking send fixes deadlock in + ssh_connection_SUITE:interrupted_send</p> + <p> + Own Id: OTP-13854</p> + </item> + <item> + <p> + Caller is now notified when a tcp close is received.</p> + <p> + Own Id: OTP-13859 Aux Id: seq13177 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Use application:ensure_all_started/2 instead of + hard-coding deps</p> + <p> + Own Id: OTP-13843 Aux Id: PR-1147 </p> + </item> + </list> + </section> + +</section> + <section><title>Ssh 4.3.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/ssh/src/ssh_auth.erl b/lib/ssh/src/ssh_auth.erl index 1dcf5d0708..ac35b70209 100644 --- a/lib/ssh/src/ssh_auth.erl +++ b/lib/ssh/src/ssh_auth.erl @@ -140,7 +140,7 @@ publickey_msg([Alg, #ssh{user = User, session_id = SessionId, service = Service, opts = Opts} = Ssh]) -> - Hash = sha, %% Maybe option?! + Hash = ssh_transport:sha(Alg), KeyCb = proplists:get_value(key_cb, Opts, ssh_file), case KeyCb:user_key(Alg, Opts) of {ok, PrivKey} -> @@ -260,43 +260,54 @@ handle_userauth_request(#ssh_msg_userauth_request{user = User, handle_userauth_request(#ssh_msg_userauth_request{user = User, service = "ssh-connection", method = "publickey", - data = Data}, - SessionId, + data = <<?BYTE(?FALSE), + ?UINT32(ALen), BAlg:ALen/binary, + ?UINT32(KLen), KeyBlob:KLen/binary, + _/binary + >> + }, + _SessionId, #ssh{opts = Opts, userauth_supported_methods = Methods} = Ssh) -> - <<?BYTE(HaveSig), - ?UINT32(ALen), BAlg:ALen/binary, - Rest/binary>> = Data, - - {KeyBlob, SigWLen} = - case Rest of - <<?UINT32(KLen0), KeyBlob0:KLen0/binary, SigWLen0/binary>> -> - {KeyBlob0, SigWLen0}; - <<>> -> - {<<>>, <<>>} - end, - - case HaveSig of - ?TRUE -> - case verify_sig(SessionId, User, "ssh-connection", - binary_to_list(BAlg), - KeyBlob, SigWLen, Opts) of - true -> - {authorized, User, - ssh_transport:ssh_packet( - #ssh_msg_userauth_success{}, Ssh)}; - false -> - {not_authorized, {User, undefined}, - ssh_transport:ssh_packet(#ssh_msg_userauth_failure{ - authentications = Methods, - partial_success = false}, Ssh)} - end; - ?FALSE -> + case pre_verify_sig(User, binary_to_list(BAlg), + KeyBlob, Opts) of + true -> {not_authorized, {User, undefined}, ssh_transport:ssh_packet( #ssh_msg_userauth_pk_ok{algorithm_name = binary_to_list(BAlg), - key_blob = KeyBlob}, Ssh)} + key_blob = KeyBlob}, Ssh)}; + false -> + {not_authorized, {User, undefined}, + ssh_transport:ssh_packet(#ssh_msg_userauth_failure{ + authentications = Methods, + partial_success = false}, Ssh)} + end; + +handle_userauth_request(#ssh_msg_userauth_request{user = User, + service = "ssh-connection", + method = "publickey", + data = <<?BYTE(?TRUE), + ?UINT32(ALen), BAlg:ALen/binary, + ?UINT32(KLen), KeyBlob:KLen/binary, + SigWLen/binary>> + }, + SessionId, + #ssh{opts = Opts, + userauth_supported_methods = Methods} = Ssh) -> + + case verify_sig(SessionId, User, "ssh-connection", + binary_to_list(BAlg), + KeyBlob, SigWLen, Opts) of + true -> + {authorized, User, + ssh_transport:ssh_packet( + #ssh_msg_userauth_success{}, Ssh)}; + false -> + {not_authorized, {User, undefined}, + ssh_transport:ssh_packet(#ssh_msg_userauth_failure{ + authentications = Methods, + partial_success = false}, Ssh)} end; handle_userauth_request(#ssh_msg_userauth_request{user = User, @@ -395,10 +406,22 @@ handle_userauth_info_response(#ssh_msg_userauth_info_response{num_responses = 1, kb_tries_left = KbTriesLeft, user = User, userauth_supported_methods = Methods} = Ssh) -> + SendOneEmpty = proplists:get_value(tstflg, Opts) == one_empty, case check_password(User, unicode:characters_to_list(Password), Opts, Ssh) of + {true,Ssh1} when SendOneEmpty==true -> + Msg = #ssh_msg_userauth_info_request{name = "", + instruction = "", + language_tag = "", + num_prompts = 0, + data = <<?BOOLEAN(?FALSE)>> + }, + {authorized_but_one_more, User, + ssh_transport:ssh_packet(Msg, Ssh1)}; + {true,Ssh1} -> {authorized, User, ssh_transport:ssh_packet(#ssh_msg_userauth_success{}, Ssh1)}; + {false,Ssh1} -> {not_authorized, {User, {error,"Bad user or password"}}, ssh_transport:ssh_packet(#ssh_msg_userauth_failure{ @@ -408,6 +431,11 @@ handle_userauth_info_response(#ssh_msg_userauth_info_response{num_responses = 1, )} end; +handle_userauth_info_response({extra,#ssh_msg_userauth_info_response{}}, + #ssh{user = User} = Ssh) -> + {authorized, User, + ssh_transport:ssh_packet(#ssh_msg_userauth_success{}, Ssh)}; + handle_userauth_info_response(#ssh_msg_userauth_info_response{}, _Auth) -> ssh_connection_handler:disconnect( @@ -484,19 +512,34 @@ get_password_option(Opts, User) -> false -> proplists:get_value(password, Opts, false) end. -verify_sig(SessionId, User, Service, Alg, KeyBlob, SigWLen, Opts) -> - {ok, Key} = decode_public_key_v2(KeyBlob, Alg), - KeyCb = proplists:get_value(key_cb, Opts, ssh_file), +pre_verify_sig(User, Alg, KeyBlob, Opts) -> + try + {ok, Key} = decode_public_key_v2(KeyBlob, Alg), + KeyCb = proplists:get_value(key_cb, Opts, ssh_file), + KeyCb:is_auth_key(Key, User, Opts) + catch + _:_ -> + false + end. - case KeyCb:is_auth_key(Key, User, Opts) of - true -> - PlainText = build_sig_data(SessionId, User, - Service, KeyBlob, Alg), - <<?UINT32(AlgSigLen), AlgSig:AlgSigLen/binary>> = SigWLen, - <<?UINT32(AlgLen), _Alg:AlgLen/binary, - ?UINT32(SigLen), Sig:SigLen/binary>> = AlgSig, - ssh_transport:verify(PlainText, sha, Sig, Key); - false -> +verify_sig(SessionId, User, Service, Alg, KeyBlob, SigWLen, Opts) -> + try + {ok, Key} = decode_public_key_v2(KeyBlob, Alg), + KeyCb = proplists:get_value(key_cb, Opts, ssh_file), + + case KeyCb:is_auth_key(Key, User, Opts) of + true -> + PlainText = build_sig_data(SessionId, User, + Service, KeyBlob, Alg), + <<?UINT32(AlgSigLen), AlgSig:AlgSigLen/binary>> = SigWLen, + <<?UINT32(AlgLen), _Alg:AlgLen/binary, + ?UINT32(SigLen), Sig:SigLen/binary>> = AlgSig, + ssh_transport:verify(PlainText, ssh_transport:sha(list_to_atom(Alg)), Sig, Key); + false -> + false + end + catch + _:_ -> false end. diff --git a/lib/ssh/src/ssh_cli.erl b/lib/ssh/src/ssh_cli.erl index 74cd2e081a..8af0ecc5f9 100644 --- a/lib/ssh/src/ssh_cli.erl +++ b/lib/ssh/src/ssh_cli.erl @@ -208,8 +208,15 @@ handle_msg({Group, Req}, #state{group = Group, buf = Buf, pty = Pty, write_chars(ConnectionHandler, ChannelId, Chars), {ok, State#state{buf = NewBuf}}; -handle_msg({'EXIT', Group, _Reason}, #state{group = Group, - channel = ChannelId} = State) -> +handle_msg({'EXIT', Group, Reason}, #state{group = Group, + cm = ConnectionHandler, + channel = ChannelId} = State) -> + Status = case Reason of + normal -> 0; + _ -> -1 + end, + ssh_connection:exit_status(ConnectionHandler, ChannelId, Status), + ssh_connection:send_eof(ConnectionHandler, ChannelId), {stop, ChannelId, State}; handle_msg(_, State) -> diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index 2eb29c9b32..facf6b561a 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -822,9 +822,21 @@ handle_event(_, #ssh_msg_userauth_info_response{} = Msg, {userauth_keyboard_inte {not_authorized, {User, Reason}, {Reply, Ssh}} -> retry_fun(User, Reason, D), send_bytes(Reply, D), - {next_state, {userauth,server}, D#data{ssh_params = Ssh}} + {next_state, {userauth,server}, D#data{ssh_params = Ssh}}; + + {authorized_but_one_more, _User, {Reply, Ssh}} -> + send_bytes(Reply, D), + {next_state, {userauth_keyboard_interactive_extra,server}, D#data{ssh_params = Ssh}} end; +handle_event(_, #ssh_msg_userauth_info_response{} = Msg, {userauth_keyboard_interactive_extra, server}, D) -> + {authorized, User, {Reply, Ssh}} = ssh_auth:handle_userauth_info_response({extra,Msg}, D#data.ssh_params), + send_bytes(Reply, D), + D#data.starter ! ssh_connected, + connected_fun(User, "keyboard-interactive", D), + {next_state, {connected,server}, D#data{auth_user = User, + ssh_params = Ssh#ssh{authenticated = true}}}; + handle_event(_, Msg = #ssh_msg_userauth_failure{}, {userauth_keyboard_interactive, client}, #data{ssh_params = Ssh0} = D0) -> Prefs = [{Method,M,F,A} || {Method,M,F,A} <- Ssh0#ssh.userauth_preference, @@ -1238,9 +1250,12 @@ handle_event(internal, prepare_next_packet, _, D) -> handle_event(info, {CloseTag,Socket}, StateName, D = #data{socket = Socket, transport_close_tag = CloseTag}) -> - disconnect(#ssh_msg_disconnect{code = ?SSH_DISCONNECT_BY_APPLICATION, - description = "Connection closed"}, - StateName, D); + %% Simulate a disconnect from the peer + handle_event(info, + #ssh_msg_disconnect{code = ?SSH_DISCONNECT_BY_APPLICATION, + description = "Connection closed"}, + StateName, + D); handle_event(info, {timeout, {_, From} = Request}, _, #data{connection_state = #connection{requests = Requests} = C0} = D) -> diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index 7cb3b75ac0..15b80de30a 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -46,7 +46,7 @@ handle_kex_ecdh_reply/2, extract_public_key/1, ssh_packet/2, pack/2, - sign/3, verify/4]). + sha/1, sign/3, verify/4]). %%% For test suites -export([pack/3]). @@ -1619,6 +1619,11 @@ kex_h(SSH, Key, Min, NBits, Max, Prime, Gen, E, F, K) -> crypto:hash(sha((SSH#ssh.algorithms)#alg.kex), L). +sha('ssh-rsa') -> sha; +sha('ssh-dss') -> sha; +sha('ecdsa-sha2-nistp256') -> sha(secp256r1); +sha('ecdsa-sha2-nistp384') -> sha(secp384r1); +sha('ecdsa-sha2-nistp521') -> sha(secp521r1); sha(secp256r1) -> sha256; sha(secp384r1) -> sha384; sha(secp521r1) -> sha512; diff --git a/lib/ssh/test/ssh_basic_SUITE.erl b/lib/ssh/test/ssh_basic_SUITE.erl index d52d453007..51e0d5196b 100644 --- a/lib/ssh/test/ssh_basic_SUITE.erl +++ b/lib/ssh/test/ssh_basic_SUITE.erl @@ -67,7 +67,8 @@ shell_unicode_string/1, ssh_info_print/1, key_callback/1, - key_callback_options/1 + key_callback_options/1, + shell_exit_status/1 ]). %%% Common test callbacks @@ -106,7 +107,8 @@ all() -> multi_daemon_opt_fd, packet_size_zero, ssh_info_print, - {group, login_bad_pwd_no_retry} + {group, login_bad_pwd_no_retry}, + shell_exit_status ]. groups() -> @@ -1167,6 +1169,33 @@ login_bad_pwd_no_retry(Config, AuthMethods) -> end end. + +%%---------------------------------------------------------------------------- +%%% Test that when shell REPL exit with reason normal client receives status 0 +shell_exit_status(Config) when is_list(Config) -> + process_flag(trap_exit, true), + SystemDir = proplists:get_value(data_dir, Config), + UserDir = proplists:get_value(priv_dir, Config), + + ShellFun = fun (_User) -> spawn(fun() -> ok end) end, + {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, + {user_dir, UserDir}, + {user_passwords, [{"vego", "morot"}]}, + {shell, ShellFun}, + {failfun, fun ssh_test_lib:failfun/2}]), + ConnectionRef = + ssh_test_lib:connect(Host, Port, [{silently_accept_hosts, true}, + {user_dir, UserDir}, + {user, "vego"}, + {password, "morot"}, + {user_interaction, false}]), + + {ok, ChannelId} = ssh_connection:session_channel(ConnectionRef, infinity), + ok = ssh_connection:shell(ConnectionRef, ChannelId), + ssh_test_lib:receive_exec_end(ConnectionRef, ChannelId), + ssh:stop_daemon(Pid). + + %%-------------------------------------------------------------------- %% Internal functions ------------------------------------------------ %%-------------------------------------------------------------------- diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index 212b99c695..09e707ad07 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,5 +1,5 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 4.3.2 +SSH_VSN = 4.3.3 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index 9d68ee0eee..d02e21a013 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -28,6 +28,45 @@ <p>This document describes the changes made to the SSL application.</p> +<section><title>SSL 8.0.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Correctly formed handshake messages received out of order + will now correctly fail the connection with unexpected + message.</p> + <p> + Own Id: OTP-13853</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + ssl application now behaves gracefully also on partially + incorrect input from peer.</p> + <p> + Own Id: OTP-13834</p> + </item> + <item> + <p> + Add application environment configuration + bypass_pem_cache. This can be used as a workaround for + the current implementation of the PEM-cache that has + proven to be a bottleneck.</p> + <p> + Own Id: OTP-13883</p> + </item> + </list> + </section> + +</section> + <section><title>SSL 8.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/ssl/doc/src/ssl_app.xml b/lib/ssl/doc/src/ssl_app.xml index a66e947bc1..f317dfded4 100644 --- a/lib/ssl/doc/src/ssl_app.xml +++ b/lib/ssl/doc/src/ssl_app.xml @@ -141,6 +141,16 @@ marker="ssl#clear_pem_cache-0">ssl:clear_pem_cache/0</seealso> </item> + + <tag><c><![CDATA[bypass_pem_cache = boolean() <optional>]]></c></tag> + <item> + <p>Introduced in ssl-8.0.2. Disables the PEM-cache. + The PEM cache has proven to be a bottleneck, until the + implementation has been improved this can be used as + a workaround. Defaults to false. + </p> + </item> + <tag><c><![CDATA[alert_timeout = integer() <optional>]]></c></tag> <item> <p> diff --git a/lib/ssl/src/dtls_connection.erl b/lib/ssl/src/dtls_connection.erl index a0d9982aaa..479f68f4bb 100644 --- a/lib/ssl/src/dtls_connection.erl +++ b/lib/ssl/src/dtls_connection.erl @@ -42,22 +42,17 @@ -export([next_record/1, next_event/3]). %% Handshake handling --export([%%renegotiate/2, - send_handshake/2, queue_handshake/2, queue_change_cipher/2]). +-export([renegotiate/2, + reinit_handshake_data/1, + send_handshake/2, queue_handshake/2, queue_change_cipher/2, + select_sni_extension/1]). %% Alert and close handling --export([%%send_alert/2, handle_own_alert/4, handle_close_alert/3, - handle_normal_shutdown/3 %%, close/5 - %%alert_user/6, alert_user/9 - ]). +-export([send_alert/2, close/5]). %% Data handling --export([%%write_application_data/3, - read_application_data/2, - passive_receive/2, next_record_if_active/1%, - %%handle_common_event/4, - %handle_packet/3 +-export([passive_receive/2, next_record_if_active/1, handle_common_event/4 ]). %% gen_statem state functions @@ -102,10 +97,11 @@ send_handshake(Handshake, State) -> send_handshake_flight(queue_handshake(Handshake, State)). queue_flight_buffer(Msg, #state{negotiated_version = Version, - connection_states = #connection_states{ - current_write = - #connection_state{epoch = Epoch}}, + connection_states = ConnectionStates, flight_buffer = Flight} = State) -> + ConnectionState = + ssl_record:current_connection_state(ConnectionStates, write), + Epoch = maps:get(epoch, ConnectionState), State#state{flight_buffer = Flight ++ [{Version, Epoch, Msg}]}. queue_handshake(Handshake, #state{negotiated_version = Version, @@ -139,6 +135,25 @@ send_alert(Alert, #state{negotiated_version = Version, Transport:send(Socket, BinMsg), State0#state{connection_states = ConnectionStates}. +close(downgrade, _,_,_,_) -> + ok; +%% Other +close(_, Socket, Transport, _,_) -> + Transport:close(Socket). + +reinit_handshake_data(#state{protocol_buffers = Buffers} = State) -> + State#state{premaster_secret = undefined, + public_key_info = undefined, + tls_handshake_history = ssl_handshake:init_handshake_history(), + protocol_buffers = + Buffers#protocol_buffers{dtls_fragment_state = + dtls_handshake:dtls_handshake_new_flight(0)}}. + +select_sni_extension(#client_hello{extensions = HelloExtensions}) -> + HelloExtensions#hello_extensions.sni; +select_sni_extension(_) -> + undefined. + %%==================================================================== %% tls_connection_sup API %%==================================================================== @@ -232,7 +247,7 @@ hello(internal, #client_hello{client_version = ClientVersion, case dtls_handshake:hello(Hello, SslOpts, {Port, Session0, Cache, CacheCb, ConnectionStates0, Cert, KeyExAlg}, Renegotiation) of #alert{} = Alert -> - handle_own_alert(Alert, ClientVersion, hello, State); + ssl_connection:handle_own_alert(Alert, ClientVersion, hello, State); {Version, {Type, Session}, ConnectionStates, Protocol0, ServerHelloExt, HashSign} -> Protocol = case Protocol0 of @@ -256,7 +271,7 @@ hello(internal, #server_hello{} = Hello, ssl_options = SslOptions} = State) -> case dtls_handshake:hello(Hello, SslOptions, ConnectionStates0, Renegotiation) of #alert{} = Alert -> - handle_own_alert(Alert, ReqVersion, hello, State); + ssl_connection:handle_own_alert(Alert, ReqVersion, hello, State); {Version, NewId, ConnectionStates, ProtoExt, Protocol} -> ssl_connection:handle_session(Hello, Version, NewId, ConnectionStates, ProtoExt, Protocol, State) @@ -335,7 +350,7 @@ handle_info({Protocol, _, Data}, StateName, {Record, State} -> next_event(StateName, Record, State); #alert{} = Alert -> - handle_normal_shutdown(Alert, StateName, State0), + ssl_connection:handle_normal_shutdown(Alert, StateName, State0), {stop, {shutdown, own_alert}} end; handle_info({CloseTag, Socket}, StateName, @@ -355,7 +370,7 @@ handle_info({CloseTag, Socket}, StateName, %%invalidate_session(Role, Host, Port, Session) ok end, - handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), + ssl_connection:handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), {stop, {shutdown, transport_closed}}; handle_info(Msg, StateName, State) -> ssl_connection:handle_info(Msg, StateName, State). @@ -363,6 +378,51 @@ handle_info(Msg, StateName, State) -> handle_call(Event, From, StateName, State) -> ssl_connection:handle_call(Event, From, StateName, State, ?MODULE). +handle_common_event(internal, #alert{} = Alert, StateName, + #state{negotiated_version = Version} = State) -> + ssl_connection:handle_own_alert(Alert, Version, StateName, State); + +%%% DTLS record protocol level handshake messages +handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE} = Record, + StateName, + #state{protocol_buffers = + #protocol_buffers{dtls_packets = Packets0, + dtls_fragment_state = HsState0} = Buffers, + negotiated_version = Version} = State0) -> + try + {Packets1, HsState} = dtls_handshake:get_dtls_handshake(Record, HsState0), + State = + State0#state{protocol_buffers = + Buffers#protocol_buffers{dtls_fragment_state = HsState}}, + Events = dtls_handshake_events(Packets0 ++ Packets1), + case StateName of + connection -> + ssl_connection:hibernate_after(StateName, State, Events); + _ -> + {next_state, StateName, State, Events} + end + catch throw:#alert{} = Alert -> + ssl_connection:handle_own_alert(Alert, Version, StateName, State0) + end; +%%% DTLS record protocol level application data messages +handle_common_event(internal, #ssl_tls{type = ?APPLICATION_DATA, fragment = Data}, StateName, State) -> + {next_state, StateName, State, [{next_event, internal, {application_data, Data}}]}; +%%% DTLS record protocol level change cipher messages +handle_common_event(internal, #ssl_tls{type = ?CHANGE_CIPHER_SPEC, fragment = Data}, StateName, State) -> + {next_state, StateName, State, [{next_event, internal, #change_cipher_spec{type = Data}}]}; +%%% DTLS record protocol level Alert messages +handle_common_event(internal, #ssl_tls{type = ?ALERT, fragment = EncAlerts}, StateName, + #state{negotiated_version = Version} = State) -> + case decode_alerts(EncAlerts) of + Alerts = [_|_] -> + handle_alerts(Alerts, {next_state, StateName, State}); + #alert{} = Alert -> + ssl_connection:handle_own_alert(Alert, Version, StateName, State) + end; +%% Ignore unknown TLS record level protocol messages +handle_common_event(internal, #ssl_tls{type = _Unknown}, StateName, State) -> + {next_state, StateName, State}. + %%-------------------------------------------------------------------- %% Description:This function is called by a gen_fsm when it is about %% to terminate. It should be the opposite of Module:init/1 and do any @@ -385,10 +445,21 @@ format_status(Type, Data) -> %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- + +dtls_handshake_events([]) -> + throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); +dtls_handshake_events(Packets) -> + lists:map(fun(Packet) -> + {next_event, internal, {handshake, Packet}} + end, Packets). + + encode_handshake(Handshake, Version, ConnectionStates0, Hist0) -> {Seq, ConnectionStates} = sequence(ConnectionStates0), {EncHandshake, Frag} = dtls_handshake:encode_handshake(Handshake, Version, Seq), - Hist = ssl_handshake:update_handshake_history(Hist0, EncHandshake), + %% DTLS does not have an equivalent version to SSLv2. So v2 hello compatibility + %% will always be false + Hist = ssl_handshake:update_handshake_history(Hist0, EncHandshake, false), {Frag, ConnectionStates, Hist}. encode_change_cipher(#change_cipher_spec{}, Version, ConnectionStates) -> @@ -424,12 +495,12 @@ encode_handshake_record(_Version, _Epoch, _Space, _MsgType, _MsgSeq, _Len, <<>>, encode_handshake_record(Version, Epoch, Space, MsgType, MsgSeq, Len, Bin, Offset, MRS, Encoded0, CS0) -> MaxFragmentLen = Space - 25, - case Bin of - <<BinFragment:MaxFragmentLen/bytes, Rest/binary>> -> - ok; + {BinFragment, Rest} = + case Bin of + <<BinFragment0:MaxFragmentLen/bytes, Rest0/binary>> -> + {BinFragment0, Rest0}; _ -> - BinFragment = Bin, - Rest = <<>> + {Bin, <<>>} end, FragLength = byte_size(BinFragment), Frag = [MsgType, ?uint24(Len), ?uint16(MsgSeq), ?uint24(Offset), ?uint24(FragLength), BinFragment], @@ -460,13 +531,13 @@ finish_pack_records({[], Acc}) -> finish_pack_records({Buf, Acc}) -> lists:reverse([lists:reverse(Buf)|Acc]). -%% decode_alerts(Bin) -> -%% ssl_alert:decode(Bin). +decode_alerts(Bin) -> + ssl_alert:decode(Bin). initial_state(Role, Host, Port, Socket, {SSLOptions, SocketOptions}, User, {CbModule, DataTag, CloseTag, ErrorTag}) -> #ssl_options{beast_mitigation = BeastMitigation} = SSLOptions, - ConnectionStates = ssl_record:init_connection_states(Role, BeastMitigation), + ConnectionStates = dtls_record:init_connection_states(Role, BeastMitigation), SessionCacheCb = case application:get_env(ssl, session_cb) of {ok, Cb} when is_atom(Cb) -> @@ -549,7 +620,7 @@ passive_receive(State0 = #state{user_data_buffer = Buffer}, StateName) -> {Record, State} = next_record(State0), next_event(StateName, Record, State); _ -> - {Record, State} = read_application_data(<<>>, State0), + {Record, State} = ssl_connection:read_application_data(<<>>, State0), next_event(StateName, Record, State) end. @@ -561,7 +632,7 @@ next_event(connection = StateName, no_record, State0, Actions) -> {no_record, State} -> ssl_connection:hibernate_after(StateName, State, Actions); {#ssl_tls{} = Record, State} -> - {next_state, StateName, State, [{next_event, internal, {dtls_record, Record}} | Actions]}; + {next_state, StateName, State, [{next_event, internal, {protocol_record, Record}} | Actions]}; {#alert{} = Alert, State} -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end; @@ -570,20 +641,11 @@ next_event(StateName, Record, State, Actions) -> no_record -> {next_state, StateName, State, Actions}; #ssl_tls{} = Record -> - {next_state, StateName, State, [{next_event, internal, {dtls_record, Record}} | Actions]}; + {next_state, StateName, State, [{next_event, internal, {protocol_record, Record}} | Actions]}; #alert{} = Alert -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end. -read_application_data(_,State) -> - {#ssl_tls{fragment = <<"place holder">>}, State}. - -handle_own_alert(_,_,_, State) -> %% Place holder - {stop, {shutdown, own_alert}, State}. - -handle_normal_shutdown(_, _, _State) -> %% Place holder - ok. - %% TODO This generates dialyzer warnings, has to be handled differently. %% handle_packet(Address, Port, Packet) -> %% try dtls_record:get_dtls_records(Packet, <<>>) of @@ -632,5 +694,34 @@ handle_normal_shutdown(_, _, _State) -> %% Place holder %% address_to_bin({A,B,C,D,E,F,G,H}, Port) -> %% <<A:16,B:16,C:16,D:16,E:16,F:16,G:16,H:16,Port:16>>. -sequence(#connection_states{dtls_write_msg_seq = Seq} = CS) -> - {Seq, CS#connection_states{dtls_write_msg_seq = Seq + 1}}. +sequence(#{write_msg_seq := Seq} = ConnectionState) -> + {Seq, ConnectionState#{write_msg_seq => Seq + 1}}. + +renegotiate(#state{role = client} = State, Actions) -> + %% Handle same way as if server requested + %% the renegotiation + Hs0 = ssl_handshake:init_handshake_history(), + {next_state, connection, State#state{tls_handshake_history = Hs0, + protocol_buffers = #protocol_buffers{}}, + [{next_event, internal, #hello_request{}} | Actions]}; + +renegotiate(#state{role = server, + connection_states = CS0} = State0, Actions) -> + HelloRequest = ssl_handshake:hello_request(), + CS = CS0#{write_msg_seq => 0}, + State1 = send_handshake(HelloRequest, + State0#state{connection_states = + CS}), + Hs0 = ssl_handshake:init_handshake_history(), + {Record, State} = next_record(State1#state{tls_handshake_history = Hs0, + protocol_buffers = #protocol_buffers{}}), + next_event(hello, Record, State, Actions). + +handle_alerts([], Result) -> + Result; +handle_alerts(_, {stop,_} = Stop) -> + Stop; +handle_alerts([Alert | Alerts], {next_state, StateName, State}) -> + handle_alerts(Alerts, ssl_connection:handle_alert(Alert, StateName, State)); +handle_alerts([Alert | Alerts], {next_state, StateName, State, _Actions}) -> + handle_alerts(Alerts, ssl_connection:handle_alert(Alert, StateName, State)). diff --git a/lib/ssl/src/dtls_handshake.erl b/lib/ssl/src/dtls_handshake.erl index 5a799cf441..c6535d5928 100644 --- a/lib/ssl/src/dtls_handshake.erl +++ b/lib/ssl/src/dtls_handshake.erl @@ -35,7 +35,7 @@ %% Internal application API %%==================================================================== %%-------------------------------------------------------------------- --spec client_hello(host(), inet:port_number(), #connection_states{}, +-spec client_hello(host(), inet:port_number(), ssl_record:connection_states(), #ssl_options{}, integer(), atom(), boolean(), der_cert()) -> #client_hello{}. %% @@ -48,7 +48,7 @@ client_hello(Host, Port, ConnectionStates, SslOpts, Cache, CacheCb, Renegotiation, OwnCert). %%-------------------------------------------------------------------- --spec client_hello(host(), inet:port_number(), term(), #connection_states{}, +-spec client_hello(host(), inet:port_number(), term(), ssl_record:connection_states(), #ssl_options{}, integer(), atom(), boolean(), der_cert()) -> #client_hello{}. %% @@ -61,7 +61,7 @@ client_hello(Host, Port, Cookie, ConnectionStates, Cache, CacheCb, Renegotiation, OwnCert) -> Version = dtls_record:highest_protocol_version(Versions), Pending = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = Pending#connection_state.security_parameters, + SecParams = maps:get(security_parameters, Pending), CipherSuites = ssl_handshake:available_suites(UserSuites, Version), Extensions = ssl_handshake:client_hello_extensions(Host, dtls_v1:corresponding_tls_version(Version), CipherSuites, @@ -445,29 +445,23 @@ enc_handshake(#client_hello{client_version = {Major, Minor}, cookie = Cookie, cipher_suites = CipherSuites, compression_methods = CompMethods, - extensions = HelloExtensions}, Version) -> + extensions = HelloExtensions}, _Version) -> SIDLength = byte_size(SessionID), - BinCookie = enc_client_hello_cookie(Version, Cookie), + CookieLength = byte_size(Cookie), BinCompMethods = list_to_binary(CompMethods), CmLength = byte_size(BinCompMethods), BinCipherSuites = list_to_binary(CipherSuites), CsLength = byte_size(BinCipherSuites), ExtensionsBin = ssl_handshake:encode_hello_extensions(HelloExtensions), - + {?CLIENT_HELLO, <<?BYTE(Major), ?BYTE(Minor), Random:32/binary, ?BYTE(SIDLength), SessionID/binary, - BinCookie/binary, + ?BYTE(CookieLength), Cookie/binary, ?UINT16(CsLength), BinCipherSuites/binary, ?BYTE(CmLength), BinCompMethods/binary, ExtensionsBin/binary>>}; enc_handshake(HandshakeMsg, Version) -> ssl_handshake:encode_handshake(HandshakeMsg, Version). -enc_client_hello_cookie(_, <<>>) -> - <<>>; -enc_client_hello_cookie(_, Cookie) -> - CookieLength = byte_size(Cookie), - <<?BYTE(CookieLength), Cookie/binary>>. - decode_handshake(_Version, ?CLIENT_HELLO, <<?BYTE(Major), ?BYTE(Minor), Random:32/binary, ?BYTE(SID_length), Session_ID:SID_length/binary, ?BYTE(Cookie_length), Cookie:Cookie_length/binary, diff --git a/lib/ssl/src/dtls_record.erl b/lib/ssl/src/dtls_record.erl index 5387fcafa8..8a6e2d315c 100644 --- a/lib/ssl/src/dtls_record.erl +++ b/lib/ssl/src/dtls_record.erl @@ -30,7 +30,7 @@ -include("ssl_cipher.hrl"). %% Handling of incoming data --export([get_dtls_records/2]). +-export([get_dtls_records/2, init_connection_states/2]). %% Decoding -export([decode_cipher_text/2]). @@ -58,7 +58,26 @@ %%==================================================================== %% Internal application API %%==================================================================== - +%%-------------------------------------------------------------------- +-spec init_connection_states(client | server, one_n_minus_one | zero_n | disabled) -> + ssl_record:connection_states(). +%% % + % +%% Description: Creates a connection_states record with appropriate +%% values for the initial SSL connection setup. +%%-------------------------------------------------------------------- +init_connection_states(Role, BeastMitigation) -> + ConnectionEnd = ssl_record:record_protocol_role(Role), + Current = initial_connection_state(ConnectionEnd, BeastMitigation), + Pending = ssl_record:empty_connection_state(ConnectionEnd, BeastMitigation), + #{write_msg_seq => 0, + prvious_read => undefined, + current_read => Current, + pending_read => Pending, + prvious_write => undefined, + current_write => Current, + pending_write => Pending}. + %%-------------------------------------------------------------------- -spec get_dtls_records(binary(), binary()) -> {[binary()], binary()} | #alert{}. %% @@ -122,63 +141,59 @@ get_dtls_records_aux(Data, Acc) -> end. encode_plain_text(Type, Version, Data, - #connection_states{current_write = - #connection_state{ - epoch = Epoch, - sequence_number = Seq, - compression_state=CompS0, - security_parameters= - #security_parameters{ - cipher_type = ?AEAD, - compression_algorithm=CompAlg} - }= WriteState0} = ConnectionStates) -> + #{current_write := + #{epoch := Epoch, + sequence_number := Seq, + compression_state := CompS0, + security_parameters := + #security_parameters{ + cipher_type = ?AEAD, + compression_algorithm = CompAlg} + }= WriteState0} = ConnectionStates) -> {Comp, CompS1} = ssl_record:compress(CompAlg, Data, CompS0), - WriteState1 = WriteState0#connection_state{compression_state = CompS1}, + WriteState1 = WriteState0#{compression_state => CompS1}, AAD = calc_aad(Type, Version, Epoch, Seq), {CipherFragment, WriteState} = ssl_record:cipher_aead(dtls_v1:corresponding_tls_version(Version), Comp, WriteState1, AAD), CipherText = encode_tls_cipher_text(Type, Version, Epoch, Seq, CipherFragment), - {CipherText, ConnectionStates#connection_states{current_write = - WriteState#connection_state{sequence_number = Seq +1}}}; + {CipherText, ConnectionStates#{current_write => WriteState#{sequence_number => Seq +1}}}; encode_plain_text(Type, Version, Data, - #connection_states{current_write=#connection_state{ - epoch = Epoch, - sequence_number = Seq, - compression_state=CompS0, - security_parameters= - #security_parameters{compression_algorithm=CompAlg} - }= WriteState0} = ConnectionStates) -> + #{current_write := + #{epoch := Epoch, + sequence_number := Seq, + compression_state := CompS0, + security_parameters := + #security_parameters{compression_algorithm = CompAlg} + }= WriteState0} = ConnectionStates) -> {Comp, CompS1} = ssl_record:compress(CompAlg, Data, CompS0), - WriteState1 = WriteState0#connection_state{compression_state = CompS1}, + WriteState1 = WriteState0#{compression_state => CompS1}, MacHash = calc_mac_hash(WriteState1, Type, Version, Epoch, Seq, Comp), {CipherFragment, WriteState} = ssl_record:cipher(dtls_v1:corresponding_tls_version(Version), Comp, WriteState1, MacHash), CipherText = encode_tls_cipher_text(Type, Version, Epoch, Seq, CipherFragment), - {CipherText, ConnectionStates#connection_states{current_write = - WriteState#connection_state{sequence_number = Seq +1}}}. + {CipherText, ConnectionStates#{current_write => WriteState#{sequence_number => Seq +1}}}. decode_cipher_text(#ssl_tls{type = Type, version = Version, epoch = Epoch, sequence_number = Seq, fragment = CipherFragment} = CipherText, - #connection_states{current_read = - #connection_state{ - compression_state = CompressionS0, - security_parameters= - #security_parameters{ - cipher_type = ?AEAD, - compression_algorithm=CompAlg} - } = ReadState0}= ConnnectionStates0) -> + #{current_read := + #{compression_state := CompressionS0, + security_parameters := + #security_parameters{ + cipher_type = ?AEAD, + compression_algorithm = CompAlg} + } = ReadState0} = ConnnectionStates0) -> AAD = calc_aad(Type, Version, Epoch, Seq), case ssl_record:decipher_aead(dtls_v1:corresponding_tls_version(Version), CipherFragment, ReadState0, AAD) of {PlainFragment, ReadState1} -> {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ConnnectionStates = ConnnectionStates0#connection_states{ - current_read = ReadState1#connection_state{ - compression_state = CompressionS1}}, + ConnnectionStates = ConnnectionStates0#{ + current_read => ReadState1#{ + compression_state => CompressionS1}}, {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; #alert{} = Alert -> Alert @@ -188,13 +203,12 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, epoch = Epoch, sequence_number = Seq, fragment = CipherFragment} = CipherText, - #connection_states{current_read = - #connection_state{ - compression_state = CompressionS0, - security_parameters= - #security_parameters{ - compression_algorithm=CompAlg} - } = ReadState0}= ConnnectionStates0) -> + #{current_read := + #{compression_state := CompressionS0, + security_parameters := + #security_parameters{ + compression_algorithm = CompAlg} + } = ReadState0}= ConnnectionStates0) -> {PlainFragment, Mac, ReadState1} = ssl_record:decipher(dtls_v1:corresponding_tls_version(Version), CipherFragment, ReadState0, true), MacHash = calc_mac_hash(ReadState1, Type, Version, Epoch, Seq, PlainFragment), @@ -202,17 +216,17 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, true -> {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ConnnectionStates = ConnnectionStates0#connection_states{ - current_read = ReadState1#connection_state{ - compression_state = CompressionS1}}, + ConnnectionStates = ConnnectionStates0#{ + current_read => ReadState1#{ + compression_state => CompressionS1}}, {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; false -> ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC) end. %%-------------------------------------------------------------------- --spec encode_change_cipher_spec(dtls_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode_change_cipher_spec(dtls_version(), ssl_record:connection_states()) -> + {iolist(), ssl_record:connection_states()}. %% %% Description: Encodes a change_cipher_spec-message to send on the ssl socket. %%-------------------------------------------------------------------- @@ -352,92 +366,87 @@ is_acceptable_version(Version, Versions) -> %%-------------------------------------------------------------------- --spec init_connection_state_seq(dtls_version(), #connection_states{}) -> - #connection_state{}. +-spec init_connection_state_seq(dtls_version(), ssl_record:connection_states()) -> + ssl_record:connection_state(). %% %% Description: Copy the read sequence number to the write sequence number %% This is only valid for DTLS in the first client_hello %%-------------------------------------------------------------------- init_connection_state_seq({254, _}, - #connection_states{ - current_read = Read = #connection_state{epoch = 0}, - current_write = Write = #connection_state{epoch = 0}} = CS0) -> - CS0#connection_states{current_write = - Write#connection_state{ - sequence_number = Read#connection_state.sequence_number}}; + #{current_read := #{epoch := 0} = Read, + current_write := #{epoch := 0} = Write} = CS0) -> + Seq = maps:get(sequence_number, Read), + CS0#{current_write => Write#{sequence_number => Seq}}; init_connection_state_seq(_, CS) -> CS. %%-------------------------------------------------------- --spec current_connection_state_epoch(#connection_states{}, read | write) -> +-spec current_connection_state_epoch(ssl_record:connection_states(), read | write) -> integer(). %% %% Description: Returns the epoch the connection_state record %% that is currently defined as the current conection state. %%-------------------------------------------------------------------- -current_connection_state_epoch(#connection_states{current_read = Current}, +current_connection_state_epoch(#{current_read := Current}, read) -> - Current#connection_state.epoch; -current_connection_state_epoch(#connection_states{current_write = Current}, + maps:get(epoch, Current); +current_connection_state_epoch(#{current_write := Current}, write) -> - Current#connection_state.epoch. + maps:get(epoch, Current). %%-------------------------------------------------------------------- --spec connection_state_by_epoch(#connection_states{}, integer(), read | write) -> - #connection_state{}. +-spec connection_state_by_epoch(ssl_record:connection_states(), integer(), read | write) -> + ssl_record:connection_state(). %% %% Description: Returns the instance of the connection_state record %% that is defined by the Epoch. %%-------------------------------------------------------------------- -connection_state_by_epoch(#connection_states{current_read = CS}, Epoch, read) - when CS#connection_state.epoch == Epoch -> +connection_state_by_epoch(#{current_read := #{epoch := Epoch}} = CS, Epoch, read) -> CS; -connection_state_by_epoch(#connection_states{pending_read = CS}, Epoch, read) - when CS#connection_state.epoch == Epoch -> +connection_state_by_epoch(#{pending_read := #{epoch := Epoch}} = CS, Epoch, read) -> CS; -connection_state_by_epoch(#connection_states{current_write = CS}, Epoch, write) - when CS#connection_state.epoch == Epoch -> +connection_state_by_epoch(#{current_write := #{epoch := Epoch}} = CS, Epoch, write) -> CS; -connection_state_by_epoch(#connection_states{pending_write = CS}, Epoch, write) - when CS#connection_state.epoch == Epoch -> +connection_state_by_epoch(#{pending_write := #{epoch := Epoch}} = CS, Epoch, write) -> CS. %%-------------------------------------------------------------------- --spec set_connection_state_by_epoch(#connection_states{}, - #connection_state{}, read | write) - -> #connection_states{}. +-spec set_connection_state_by_epoch(ssl_record:connection_states(), + ssl_record:connection_state(), read | write) + -> ssl_record:connection_states(). %% %% Description: Returns the instance of the connection_state record %% that is defined by the Epoch. %%-------------------------------------------------------------------- -set_connection_state_by_epoch(ConnectionStates0 = - #connection_states{current_read = CS}, - NewCS = #connection_state{epoch = Epoch}, read) - when CS#connection_state.epoch == Epoch -> - ConnectionStates0#connection_states{current_read = NewCS}; - -set_connection_state_by_epoch(ConnectionStates0 = - #connection_states{pending_read = CS}, - NewCS = #connection_state{epoch = Epoch}, read) - when CS#connection_state.epoch == Epoch -> - ConnectionStates0#connection_states{pending_read = NewCS}; - -set_connection_state_by_epoch(ConnectionStates0 = - #connection_states{current_write = CS}, - NewCS = #connection_state{epoch = Epoch}, write) - when CS#connection_state.epoch == Epoch -> - ConnectionStates0#connection_states{current_write = NewCS}; - -set_connection_state_by_epoch(ConnectionStates0 = - #connection_states{pending_write = CS}, - NewCS = #connection_state{epoch = Epoch}, write) - when CS#connection_state.epoch == Epoch -> - ConnectionStates0#connection_states{pending_write = NewCS}. +set_connection_state_by_epoch(#{current_read := #{epoch := Epoch}} = ConnectionStates0, + NewCS = #{epoch := Epoch}, read) -> + ConnectionStates0#{current_read => NewCS}; +set_connection_state_by_epoch(#{pending_read := #{epoch := Epoch}} = ConnectionStates0, + NewCS = #{epoch := Epoch}, read) -> + ConnectionStates0#{pending_read => NewCS}; +set_connection_state_by_epoch(#{current_write := #{epoch := Epoch}} = ConnectionStates0, + NewCS = #{epoch := Epoch}, write) -> + ConnectionStates0#{current_write => NewCS}; +set_connection_state_by_epoch(#{pending_write := #{epoch := Epoch}} = ConnectionStates0, +NewCS = #{epoch := Epoch}, write) -> + ConnectionStates0#{pending_write => NewCS}. %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- - +initial_connection_state(ConnectionEnd, BeastMitigation) -> + #{security_parameters => + ssl_record:initial_security_params(ConnectionEnd), + epoch => 0, + sequence_number => 1, + beast_mitigation => BeastMitigation, + compression_state => undefined, + cipher_state => undefined, + mac_secret => undefined, + secure_renegotiation => undefined, + client_verify_data => undefined, + server_verify_data => undefined + }. lowest_list_protocol_version(Ver, []) -> Ver; @@ -454,8 +463,8 @@ encode_tls_cipher_text(Type, {MajVer, MinVer}, Epoch, Seq, Fragment) -> [<<?BYTE(Type), ?BYTE(MajVer), ?BYTE(MinVer), ?UINT16(Epoch), ?UINT48(Seq), ?UINT16(Length)>>, Fragment]. -calc_mac_hash(#connection_state{mac_secret = MacSecret, - security_parameters = #security_parameters{mac_algorithm = MacAlg}}, +calc_mac_hash(#{mac_secret := MacSecret, + security_parameters := #security_parameters{mac_algorithm = MacAlg}}, Type, Version, Epoch, SeqNo, Fragment) -> Length = erlang:iolist_size(Fragment), NewSeq = (Epoch bsl 48) + SeqNo, diff --git a/lib/ssl/src/ssl.appup.src b/lib/ssl/src/ssl.appup.src index 22e24af0a8..32252386b4 100644 --- a/lib/ssl/src/ssl.appup.src +++ b/lib/ssl/src/ssl.appup.src @@ -7,5 +7,5 @@ [ {<<"^8[.]0([.][0-9]+)?$">>, [{restart_application, ssl}]}, {<<"^[3-7][.][^.].*">>, [{restart_application, ssl}]} - ] + ] }. diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index d2aeb3258f..27b753af2e 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -43,7 +43,7 @@ renegotiate/1, prf/5, negotiated_protocol/1, negotiated_next_protocol/1, connection_information/1, connection_information/2]). %% Misc --export([handle_options/2]). +-export([handle_options/2, tls_version/1]). -deprecated({negotiated_next_protocol, 1, next_major_release}). -deprecated({connection_info, 1, next_major_release}). @@ -607,6 +607,11 @@ format_error(Error) -> Other end. +tls_version({3, _} = Version) -> + Version; +tls_version({254, _} = Version) -> + dtls_v1:corresponding_tls_version(Version). + %%%-------------------------------------------------------------- %%% Internal functions %%%-------------------------------------------------------------------- diff --git a/lib/ssl/src/ssl_alert.erl b/lib/ssl/src/ssl_alert.erl index db71b16d80..05dfb4c1b3 100644 --- a/lib/ssl/src/ssl_alert.erl +++ b/lib/ssl/src/ssl_alert.erl @@ -39,8 +39,8 @@ %%==================================================================== %%-------------------------------------------------------------------- --spec encode(#alert{}, ssl_record:ssl_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode(#alert{}, ssl_record:ssl_version(), ssl_record:connection_states()) -> + {iolist(), ssl_record:connection_states()}. %% %% Description: Encodes an alert %%-------------------------------------------------------------------- diff --git a/lib/ssl/src/ssl_certificate.erl b/lib/ssl/src/ssl_certificate.erl index 3ec3f50e05..f359655d85 100644 --- a/lib/ssl/src/ssl_certificate.erl +++ b/lib/ssl/src/ssl_certificate.erl @@ -64,7 +64,7 @@ trusted_cert_and_path(CertChain, CertDbHandle, CertDbRef, PartialChainHandler) - {ok, IssuerId} = public_key:pkix_issuer_id(OtpCert, self), {self, IssuerId}; false -> - other_issuer(OtpCert, BinCert, CertDbHandle) + other_issuer(OtpCert, BinCert, CertDbHandle, CertDbRef) end, case SignedAndIssuerID of @@ -200,7 +200,7 @@ certificate_chain(OtpCert, BinCert, CertDbHandle, CertsDbRef, Chain) -> {_, true = SelfSigned} -> certificate_chain(CertDbHandle, CertsDbRef, Chain, ignore, ignore, SelfSigned); {{error, issuer_not_found}, SelfSigned} -> - case find_issuer(OtpCert, BinCert, CertDbHandle) of + case find_issuer(OtpCert, BinCert, CertDbHandle, CertsDbRef) of {ok, {SerialNr, Issuer}} -> certificate_chain(CertDbHandle, CertsDbRef, Chain, SerialNr, Issuer, SelfSigned); @@ -232,7 +232,7 @@ certificate_chain(CertDbHandle, CertsDbRef, Chain, SerialNr, Issuer, _SelfSigned {ok, undefined, lists:reverse(Chain)} end. -find_issuer(OtpCert, BinCert, CertDbHandle) -> +find_issuer(OtpCert, BinCert, CertDbHandle, CertsDbRef) -> IsIssuerFun = fun({_Key, {_Der, #'OTPCertificate'{} = ErlCertCandidate}}, Acc) -> case public_key:pkix_is_issuer(OtpCert, ErlCertCandidate) of @@ -250,12 +250,24 @@ find_issuer(OtpCert, BinCert, CertDbHandle) -> Acc end, - try ssl_pkix_db:foldl(IsIssuerFun, issuer_not_found, CertDbHandle) of - issuer_not_found -> - {error, issuer_not_found} - catch - {ok, _IssuerId} = Return -> - Return + if is_reference(CertsDbRef) -> % actual DB exists + try ssl_pkix_db:foldl(IsIssuerFun, issuer_not_found, CertDbHandle) of + issuer_not_found -> + {error, issuer_not_found} + catch + {ok, _IssuerId} = Return -> + Return + end; + is_tuple(CertsDbRef), element(1,CertsDbRef) =:= extracted -> % cache bypass byproduct + {extracted, CertsData} = CertsDbRef, + DB = [Entry || {decoded, Entry} <- CertsData], + try lists:foldl(IsIssuerFun, issuer_not_found, DB) of + issuer_not_found -> + {error, issuer_not_found} + catch + {ok, _IssuerId} = Return -> + Return + end end. is_valid_extkey_usage(KeyUse, client) -> @@ -281,12 +293,12 @@ public_key(#'OTPSubjectPublicKeyInfo'{algorithm = #'PublicKeyAlgorithm'{algorith subjectPublicKey = Key}) -> {Key, Params}. -other_issuer(OtpCert, BinCert, CertDbHandle) -> +other_issuer(OtpCert, BinCert, CertDbHandle, CertDbRef) -> case public_key:pkix_issuer_id(OtpCert, other) of {ok, IssuerId} -> {other, IssuerId}; {error, issuer_not_found} -> - case find_issuer(OtpCert, BinCert, CertDbHandle) of + case find_issuer(OtpCert, BinCert, CertDbHandle, CertDbRef) of {ok, IssuerId} -> {other, IssuerId}; Other -> diff --git a/lib/ssl/src/ssl_connection.erl b/lib/ssl/src/ssl_connection.erl index 8a990870e8..304d1706f5 100644 --- a/lib/ssl/src/ssl_connection.erl +++ b/lib/ssl/src/ssl_connection.erl @@ -58,6 +58,13 @@ -export([handle_info/3, handle_call/5, handle_session/7, ssl_config/3, prepare_connection/2, hibernate_after/3]). +%% Alert and close handling +-export([handle_own_alert/4,handle_alert/3, + handle_normal_shutdown/3 + ]). + +%% Data handling +-export([write_application_data/3, read_application_data/2]). %%==================================================================== %% Internal application API @@ -264,7 +271,7 @@ prf(ConnectionPid, Secret, Label, Seed, WantedLength) -> %%-------------------------------------------------------------------- -spec handle_session(#server_hello{}, ssl_record:ssl_version(), - binary(), #connection_states{}, _,_, #state{}) -> + binary(), ssl_record:connection_states(), _,_, #state{}) -> gen_statem:state_function_result(). %%-------------------------------------------------------------------- handle_session(#server_hello{cipher_suite = CipherSuite, @@ -272,19 +279,21 @@ handle_session(#server_hello{cipher_suite = CipherSuite, Version, NewId, ConnectionStates, ProtoExt, Protocol0, #state{session = #session{session_id = OldId}, negotiated_version = ReqVersion, - negotiated_protocol = CurrentProtocol} = State0) -> + negotiated_protocol = CurrentProtocol} = State0) -> {KeyAlgorithm, _, _, _} = ssl_cipher:suite_definition(CipherSuite), PremasterSecret = make_premaster_secret(ReqVersion, KeyAlgorithm), - {ExpectNPN, Protocol} = case Protocol0 of - undefined -> {false, CurrentProtocol}; - _ -> {ProtoExt =:= npn, Protocol0} - end, + {ExpectNPN, Protocol} = case Protocol0 of + undefined -> + {false, CurrentProtocol}; + _ -> + {ProtoExt =:= npn, Protocol0} + end, State = State0#state{key_algorithm = KeyAlgorithm, - negotiated_version = Version, + negotiated_version = Version, connection_states = ConnectionStates, premaster_secret = PremasterSecret, expecting_next_protocol_negotiation = ExpectNPN, @@ -382,7 +391,7 @@ abbreviated(internal, #finished{verify_data = Data} = Finished, session = #session{master_secret = MasterSecret}, connection_states = ConnectionStates0} = State0, Connection) -> - case ssl_handshake:verify_connection(Version, Finished, client, + case ssl_handshake:verify_connection(ssl:tls_version(Version), Finished, client, get_current_prf(ConnectionStates0, write), MasterSecret, Handshake) of verified -> @@ -392,7 +401,7 @@ abbreviated(internal, #finished{verify_data = Data} = Finished, expecting_finished = false}, Connection), Connection:next_event(connection, Record, State); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, abbreviated, State0) + handle_own_alert(Alert, Version, abbreviated, State0) end; abbreviated(internal, #finished{verify_data = Data} = Finished, @@ -400,7 +409,7 @@ abbreviated(internal, #finished{verify_data = Data} = Finished, session = #session{master_secret = MasterSecret}, negotiated_version = Version, connection_states = ConnectionStates0} = State0, Connection) -> - case ssl_handshake:verify_connection(Version, Finished, server, + case ssl_handshake:verify_connection(ssl:tls_version(Version), Finished, server, get_pending_prf(ConnectionStates0, write), MasterSecret, Handshake0) of verified -> @@ -412,7 +421,7 @@ abbreviated(internal, #finished{verify_data = Data} = Finished, {Record, State} = prepare_connection(State1#state{expecting_finished = false}, Connection), Connection:next_event(connection, Record, State); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, abbreviated, State0) + handle_own_alert(Alert, Version, abbreviated, State0) end; %% only allowed to send next_protocol message after change cipher spec @@ -452,9 +461,9 @@ certify(internal, #certificate{asn1_certificates = []}, #state{role = server, negotiated_version = Version, ssl_options = #ssl_options{verify = verify_peer, fail_if_no_peer_cert = true}} = - State, Connection) -> + State, _Connection) -> Alert = ?ALERT_REC(?FATAL,?HANDSHAKE_FAILURE), - Connection:handle_own_alert(Alert, Version, certify, State); + handle_own_alert(Alert, Version, certify, State); certify(internal, #certificate{asn1_certificates = []}, #state{role = server, @@ -469,9 +478,9 @@ certify(internal, #certificate{}, #state{role = server, negotiated_version = Version, ssl_options = #ssl_options{verify = verify_none}} = - State, Connection) -> + State, _Connection) -> Alert = ?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE, unrequested_certificate), - Connection:handle_own_alert(Alert, Version, certify, State); + handle_own_alert(Alert, Version, certify, State); certify(internal, #certificate{} = Cert, #state{negotiated_version = Version, @@ -492,7 +501,7 @@ certify(internal, #certificate{} = Cert, handle_peer_cert(Role, PeerCert, PublicKeyInfo, State#state{client_certificate_requested = false}, Connection); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State) + handle_own_alert(Alert, Version, certify, State) end; certify(internal, #server_key_exchange{exchange_keys = Keys}, @@ -506,10 +515,10 @@ certify(internal, #server_key_exchange{exchange_keys = Keys}, Alg == psk; Alg == dhe_psk; Alg == rsa_psk; Alg == srp_dss; Alg == srp_rsa; Alg == srp_anon -> - Params = ssl_handshake:decode_server_key(Keys, Alg, Version), + Params = ssl_handshake:decode_server_key(Keys, Alg, ssl:tls_version(Version)), %% Use negotiated value if TLS-1.2 otherwhise return default - HashSign = negotiated_hashsign(Params#server_key_params.hashsign, Alg, PubKeyInfo, Version), + HashSign = negotiated_hashsign(Params#server_key_params.hashsign, Alg, PubKeyInfo, ssl:tls_version(Version)), case is_anonymous(Alg) of true -> @@ -517,13 +526,13 @@ certify(internal, #server_key_exchange{exchange_keys = Keys}, State#state{hashsign_algorithm = HashSign}, Connection); false -> case ssl_handshake:verify_server_key(Params, HashSign, - ConnectionStates, Version, PubKeyInfo) of + ConnectionStates, ssl:tls_version(Version), PubKeyInfo) of true -> calculate_secret(Params#server_key_params.params, State#state{hashsign_algorithm = HashSign}, Connection); false -> - Connection:handle_own_alert(?ALERT_REC(?FATAL, ?DECRYPT_ERROR), + handle_own_alert(?ALERT_REC(?FATAL, ?DECRYPT_ERROR), Version, certify, State) end end; @@ -533,9 +542,9 @@ certify(internal, #certificate_request{} = CertRequest, role = client, ssl_options = #ssl_options{signature_algs = SupportedHashSigns}, negotiated_version = Version} = State0, Connection) -> - case ssl_handshake:select_hashsign(CertRequest, Cert, SupportedHashSigns, Version) of + case ssl_handshake:select_hashsign(CertRequest, Cert, SupportedHashSigns, ssl:tls_version(Version)) of #alert {} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0); + handle_own_alert(Alert, Version, certify, State0); NegotiatedHashSign -> {Record, State} = Connection:next_record(State0#state{client_certificate_requested = true}), Connection:next_event(certify, Record, @@ -554,7 +563,7 @@ certify(internal, #server_hello_done{}, when Alg == psk -> case ssl_handshake:premaster_secret({Alg, PSKIdentity}, PSKLookup) of #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0); + handle_own_alert(Alert, Version, certify, State0); PremasterSecret -> State = master_secret(PremasterSecret, State0#state{premaster_secret = PremasterSecret}), @@ -575,7 +584,7 @@ certify(internal, #server_hello_done{}, case ssl_handshake:premaster_secret({Alg, PSKIdentity}, PSKLookup, RSAPremasterSecret) of #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0); + handle_own_alert(Alert, Version, certify, State0); PremasterSecret -> State = master_secret(PremasterSecret, State0#state{premaster_secret = RSAPremasterSecret}), @@ -589,13 +598,13 @@ certify(internal, #server_hello_done{}, negotiated_version = Version, premaster_secret = undefined, role = client} = State0, Connection) -> - case ssl_handshake:master_secret(record_cb(Connection), Version, Session, + case ssl_handshake:master_secret(ssl:tls_version(Version), Session, ConnectionStates0, client) of {MasterSecret, ConnectionStates} -> State = State0#state{connection_states = ConnectionStates}, client_certify_and_key_exchange(State, Connection); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0) + handle_own_alert(Alert, Version, certify, State0) end; %% Master secret is calculated from premaster_secret @@ -605,7 +614,7 @@ certify(internal, #server_hello_done{}, negotiated_version = Version, premaster_secret = PremasterSecret, role = client} = State0, Connection) -> - case ssl_handshake:master_secret(record_cb(Connection), Version, PremasterSecret, + case ssl_handshake:master_secret(ssl:tls_version(Version), PremasterSecret, ConnectionStates0, client) of {MasterSecret, ConnectionStates} -> Session = Session0#session{master_secret = MasterSecret}, @@ -613,7 +622,7 @@ certify(internal, #server_hello_done{}, session = Session}, client_certify_and_key_exchange(State, Connection); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0) + handle_own_alert(Alert, Version, certify, State0) end; certify(internal = Type, #client_key_exchange{} = Msg, @@ -627,11 +636,11 @@ certify(internal = Type, #client_key_exchange{} = Msg, certify(internal, #client_key_exchange{exchange_keys = Keys}, State = #state{key_algorithm = KeyAlg, negotiated_version = Version}, Connection) -> try - certify_client_key_exchange(ssl_handshake:decode_client_key(Keys, KeyAlg, Version), + certify_client_key_exchange(ssl_handshake:decode_client_key(Keys, KeyAlg, ssl:tls_version(Version)), State, Connection) catch #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State) + handle_own_alert(Alert, Version, certify, State) end; certify(Type, Msg, State, Connection) -> @@ -662,21 +671,21 @@ cipher(internal, #certificate_verify{signature = Signature, %% Use negotiated value if TLS-1.2 otherwhise return default HashSign = negotiated_hashsign(CertHashSign, KexAlg, PublicKeyInfo, Version), case ssl_handshake:certificate_verify(Signature, PublicKeyInfo, - Version, HashSign, MasterSecret, Handshake) of + ssl:tls_version(Version), HashSign, MasterSecret, Handshake) of valid -> {Record, State} = Connection:next_record(State0), Connection:next_event(cipher, Record, State#state{cert_hashsign_algorithm = HashSign}); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, cipher, State0) + handle_own_alert(Alert, Version, cipher, State0) end; %% client must send a next protocol message if we are expecting it cipher(internal, #finished{}, #state{role = server, expecting_next_protocol_negotiation = true, negotiated_protocol = undefined, negotiated_version = Version} = State0, - Connection) -> - Connection:handle_own_alert(?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE), Version, cipher, State0); + _Connection) -> + handle_own_alert(?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE), Version, cipher, State0); cipher(internal, #finished{verify_data = Data} = Finished, #state{negotiated_version = Version, @@ -688,7 +697,7 @@ cipher(internal, #finished{verify_data = Data} = Finished, = Session0, connection_states = ConnectionStates0, tls_handshake_history = Handshake0} = State, Connection) -> - case ssl_handshake:verify_connection(Version, Finished, + case ssl_handshake:verify_connection(ssl:tls_version(Version), Finished, opposite_role(Role), get_current_prf(ConnectionStates0, read), MasterSecret, Handshake0) of @@ -697,7 +706,7 @@ cipher(internal, #finished{verify_data = Data} = Finished, cipher_role(Role, Data, Session, State#state{expecting_finished = false}, Connection); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, cipher, State) + handle_own_alert(Alert, Version, cipher, State) end; %% only allowed to send next_protocol message after change cipher spec @@ -730,7 +739,7 @@ connection({call, From}, {application_data, Data}, %% parallize send and receive decoding and not block the receiver %% if sending is overloading the socket. try - Connection:write_application_data(Data, From, State) + write_application_data(Data, From, State) catch throw:Error -> hibernate_after(connection, State, [{reply, From, Error}]) end; @@ -801,33 +810,37 @@ handle_common_event(internal, {handshake, {#hello_request{}, _}}, StateName, #st when StateName =/= connection -> {keep_state_and_data}; handle_common_event(internal, {handshake, {Handshake, Raw}}, StateName, - #state{tls_handshake_history = Hs0} = State0, Connection) -> + #state{tls_handshake_history = Hs0, + ssl_options = #ssl_options{v2_hello_compatible = V2HComp}} = State0, + Connection) -> + + PossibleSNI = Connection:select_sni_extension(Handshake), %% This function handles client SNI hello extension when Handshake is %% a client_hello, which needs to be determined by the connection callback. %% In other cases this is a noop - State = Connection:handle_sni_extension(Handshake, State0), - HsHist = ssl_handshake:update_handshake_history(Hs0, Raw), + State = handle_sni_extension(PossibleSNI, State0), + HsHist = ssl_handshake:update_handshake_history(Hs0, Raw, V2HComp), {next_state, StateName, State#state{tls_handshake_history = HsHist}, [{next_event, internal, Handshake}]}; -handle_common_event(internal, {tls_record, TLSRecord}, StateName, State, Connection) -> - Connection:handle_common_event(internal, TLSRecord, StateName, State); +handle_common_event(internal, {protocol_record, TLSorDTLSRecord}, StateName, State, Connection) -> + Connection:handle_common_event(internal, TLSorDTLSRecord, StateName, State); handle_common_event(timeout, hibernate, _, _, _) -> {keep_state_and_data, [hibernate]}; handle_common_event(internal, {application_data, Data}, StateName, State0, Connection) -> - case Connection:read_application_data(Data, State0) of + case read_application_data(Data, State0) of {stop, Reason, State} -> {stop, Reason, State}; {Record, State} -> Connection:next_event(StateName, Record, State) end; handle_common_event(internal, #change_cipher_spec{type = <<1>>}, StateName, - #state{negotiated_version = Version} = State, Connection) -> - Connection:handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE), Version, + #state{negotiated_version = Version} = State, _) -> + handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE), Version, StateName, State); handle_common_event(_Type, Msg, StateName, #state{negotiated_version = Version} = State, - Connection) -> + _) -> Alert = ?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE), - Connection:handle_own_alert(Alert, Version, {StateName, Msg}, State). + handle_own_alert(Alert, Version, {StateName, Msg}, State). handle_call({application_data, _Data}, _, _, _, _) -> %% In renegotiation priorities handshake, send data when handshake is finished @@ -905,9 +918,8 @@ handle_call(renegotiate, From, StateName, _, _) when StateName =/= connection -> handle_call({prf, Secret, Label, Seed, WantedLength}, From, _, #state{connection_states = ConnectionStates, negotiated_version = Version}, _) -> - ConnectionState = + #{security_parameters := SecParams} = ssl_record:current_connection_state(ConnectionStates, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{master_secret = MasterSecret, client_random = ClientRandom, server_random = ServerRandom, @@ -922,7 +934,7 @@ handle_call({prf, Secret, Label, Seed, WantedLength}, From, _, (client_random, Acc) -> [ClientRandom|Acc]; (server_random, Acc) -> [ServerRandom|Acc] end, [], Seed)), - ssl_handshake:prf(Version, PRFAlgorithm, SecretToUse, Label, SeedToUse, WantedLength) + ssl_handshake:prf(ssl:tls_version(Version), PRFAlgorithm, SecretToUse, Label, SeedToUse, WantedLength) catch exit:_ -> {error, badarg}; error:Reason -> {error, Reason} @@ -933,20 +945,19 @@ handle_call(_,_,_,_,_) -> handle_info({ErrorTag, Socket, econnaborted}, StateName, #state{socket = Socket, transport_cb = Transport, - start_or_recv_from = StartFrom, role = Role, protocol_cb = Connection, + start_or_recv_from = StartFrom, role = Role, error_tag = ErrorTag, tracker = Tracker} = State) when StateName =/= connection -> - Connection:alert_user(Transport, Tracker,Socket, - StartFrom, ?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), Role), + alert_user(Transport, Tracker,Socket, + StartFrom, ?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), Role, Connection), {stop, normal, State}; handle_info({ErrorTag, Socket, Reason}, StateName, #state{socket = Socket, - protocol_cb = Connection, error_tag = ErrorTag} = State) -> Report = io_lib:format("SSL: Socket error: ~p ~n", [Reason]), error_logger:info_report(Report), - Connection:handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), + handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), {stop, normal, State}; handle_info({'DOWN', MonitorRef, _, _, _}, _, @@ -1049,13 +1060,124 @@ format_status(terminate, [_, StateName, State]) -> ssl_options = NewOptions, flight_buffer = ?SECRET_PRINTOUT} }}]}]. + +%%-------------------------------------------------------------------- +%%% +%%-------------------------------------------------------------------- +write_application_data(Data0, From, + #state{socket = Socket, + negotiated_version = Version, + protocol_cb = Connection, + transport_cb = Transport, + connection_states = ConnectionStates0, + socket_options = SockOpts, + ssl_options = #ssl_options{renegotiate_at = RenegotiateAt}} = State) -> + Data = encode_packet(Data0, SockOpts), + + case time_to_renegotiate(Data, ConnectionStates0, RenegotiateAt) of + true -> + Connection:renegotiate(State#state{renegotiation = {true, internal}}, + [{next_event, {call, From}, {application_data, Data0}}]); + false -> + {Msgs, ConnectionStates} = ssl_record:encode_data(Data, Version, ConnectionStates0), + Result = Transport:send(Socket, Msgs), + ssl_connection:hibernate_after(connection, State#state{connection_states = ConnectionStates}, + [{reply, From, Result}]) + end. + +read_application_data(Data, #state{user_application = {_Mon, Pid}, + socket = Socket, + protocol_cb = Connection, + transport_cb = Transport, + socket_options = SOpts, + bytes_to_read = BytesToRead, + start_or_recv_from = RecvFrom, + timer = Timer, + user_data_buffer = Buffer0, + tracker = Tracker} = State0) -> + Buffer1 = if + Buffer0 =:= <<>> -> Data; + Data =:= <<>> -> Buffer0; + true -> <<Buffer0/binary, Data/binary>> + end, + case get_data(SOpts, BytesToRead, Buffer1) of + {ok, ClientData, Buffer} -> % Send data + SocketOpt = deliver_app_data(Transport, Socket, SOpts, + ClientData, Pid, RecvFrom, Tracker, Connection), + cancel_timer(Timer), + State = State0#state{user_data_buffer = Buffer, + start_or_recv_from = undefined, + timer = undefined, + bytes_to_read = undefined, + socket_options = SocketOpt + }, + if + SocketOpt#socket_options.active =:= false; Buffer =:= <<>> -> + %% Passive mode, wait for active once or recv + %% Active and empty, get more data + Connection:next_record_if_active(State); + true -> %% We have more data + read_application_data(<<>>, State) + end; + {more, Buffer} -> % no reply, we need more data + Connection:next_record(State0#state{user_data_buffer = Buffer}); + {passive, Buffer} -> + Connection:next_record_if_active(State0#state{user_data_buffer = Buffer}); + {error,_Reason} -> %% Invalid packet in packet mode + deliver_packet_error(Transport, Socket, SOpts, Buffer1, Pid, RecvFrom, Tracker, Connection), + {stop, normal, State0} + end. +%%-------------------------------------------------------------------- +%%% +%%-------------------------------------------------------------------- +handle_alert(#alert{level = ?FATAL} = Alert, StateName, + #state{socket = Socket, transport_cb = Transport, + protocol_cb = Connection, + ssl_options = SslOpts, start_or_recv_from = From, host = Host, + port = Port, session = Session, user_application = {_Mon, Pid}, + role = Role, socket_options = Opts, tracker = Tracker}) -> + invalidate_session(Role, Host, Port, Session), + log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), + alert_user(Transport, Tracker, Socket, StateName, Opts, Pid, From, Alert, Role, Connection), + {stop, normal}; + +handle_alert(#alert{level = ?WARNING, description = ?CLOSE_NOTIFY} = Alert, + StateName, State) -> + handle_normal_shutdown(Alert, StateName, State), + {stop, {shutdown, peer_close}}; + +handle_alert(#alert{level = ?WARNING, description = ?NO_RENEGOTIATION} = Alert, StateName, + #state{ssl_options = SslOpts, renegotiation = {true, internal}} = State) -> + log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), + handle_normal_shutdown(Alert, StateName, State), + {stop, {shutdown, peer_close}}; + +handle_alert(#alert{level = ?WARNING, description = ?NO_RENEGOTIATION} = Alert, StateName, + #state{ssl_options = SslOpts, renegotiation = {true, From}, + protocol_cb = Connection} = State0) -> + log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), + gen_statem:reply(From, {error, renegotiation_rejected}), + {Record, State} = Connection:next_record(State0), + %% Go back to connection! + Connection:next_event(connection, Record, State); + +%% Gracefully log and ignore all other warning alerts +handle_alert(#alert{level = ?WARNING} = Alert, StateName, + #state{ssl_options = SslOpts, protocol_cb = Connection} = State0) -> + log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), + {Record, State} = Connection:next_record(State0), + Connection:next_event(StateName, Record, State). + %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- connection_info(#state{sni_hostname = SNIHostname, session = #session{cipher_suite = CipherSuite}, - negotiated_version = Version, ssl_options = Opts}) -> - [{protocol, tls_record:protocol_version(Version)}, + protocol_cb = Connection, + negotiated_version = {_,_} = Version, + ssl_options = Opts}) -> + RecordCB = record_cb(Connection), + [{protocol, RecordCB:protocol_version(Version)}, {cipher_suite, ssl_cipher:erl_suite_definition(CipherSuite)}, {sni_hostname, SNIHostname}] ++ ssl_options_list(Opts). @@ -1067,7 +1189,7 @@ do_server_hello(Type, #hello_extensions{next_protocol_negotiation = NextProtocol = State0, Connection) when is_atom(Type) -> ServerHello = - ssl_handshake:server_hello(SessId, Version, ConnectionStates0, ServerHelloExt), + ssl_handshake:server_hello(SessId, ssl:tls_version(Version), ConnectionStates0, ServerHelloExt), State = server_hello(ServerHello, State0#state{expecting_next_protocol_negotiation = NextProtocols =/= undefined}, Connection), @@ -1094,14 +1216,14 @@ new_server_hello(#server_hello{cipher_suite = CipherSuite, Connection:next_event(certify, Record, State) catch #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, hello, State0) + handle_own_alert(Alert, Version, hello, State0) end. resumed_server_hello(#state{session = Session, connection_states = ConnectionStates0, negotiated_version = Version} = State0, Connection) -> - case ssl_handshake:master_secret(record_cb(Connection), Version, Session, + case ssl_handshake:master_secret(ssl:tls_version(Version), Session, ConnectionStates0, server) of {_, ConnectionStates1} -> State1 = State0#state{connection_states = ConnectionStates1, @@ -1111,7 +1233,7 @@ resumed_server_hello(#state{session = Session, {Record, State} = Connection:next_record(State2), Connection:next_event(abbreviated, Record, State); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, hello, State0) + handle_own_alert(Alert, Version, hello, State0) end. server_hello(ServerHello, State0, Connection) -> @@ -1177,7 +1299,7 @@ verify_client_cert(#state{client_certificate_requested = true, role = client, tls_handshake_history = Handshake0} = State, Connection) -> case ssl_handshake:client_certificate_verify(OwnCert, MasterSecret, - Version, HashSign, PrivateKey, Handshake0) of + ssl:tls_version(Version), HashSign, PrivateKey, Handshake0) of #certificate_verify{} = Verified -> Connection:queue_handshake(Verified, State); ignore -> @@ -1200,7 +1322,7 @@ client_certify_and_key_exchange(#state{negotiated_version = Version} = Connection:next_event(cipher, Record, State) catch throw:#alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0) + handle_own_alert(Alert, Version, certify, State0) end. do_client_certify_and_key_exchange(State0, Connection) -> @@ -1291,12 +1413,11 @@ key_exchange(#state{role = server, key_algorithm = Algo, Algo == dhe_rsa; Algo == dh_anon -> DHKeys = public_key:generate_key(Params), - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {dh, DHKeys, Params, + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), {dh, DHKeys, Params, HashSignAlgo, ClientRandom, ServerRandom, PrivateKey}), @@ -1316,15 +1437,15 @@ key_exchange(#state{role = server, key_algorithm = Algo, Algo == ecdh_anon -> ECDHKeys = public_key:generate_key(select_curve(State0)), - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {ecdh, ECDHKeys, - HashSignAlgo, ClientRandom, - ServerRandom, - PrivateKey}), + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), + {ecdh, ECDHKeys, + HashSignAlgo, ClientRandom, + ServerRandom, + PrivateKey}), State = Connection:queue_handshake(Msg, State0), State#state{diffie_hellman_keys = ECDHKeys}; @@ -1338,14 +1459,14 @@ key_exchange(#state{role = server, key_algorithm = psk, connection_states = ConnectionStates0, negotiated_version = Version } = State0, Connection) -> - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {psk, PskIdentityHint, - HashSignAlgo, ClientRandom, - ServerRandom, + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), + {psk, PskIdentityHint, + HashSignAlgo, ClientRandom, + ServerRandom, PrivateKey}), Connection:queue_handshake(Msg, State0); @@ -1358,16 +1479,16 @@ key_exchange(#state{role = server, key_algorithm = dhe_psk, negotiated_version = Version } = State0, Connection) -> DHKeys = public_key:generate_key(Params), - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {dhe_psk, - PskIdentityHint, DHKeys, Params, - HashSignAlgo, ClientRandom, - ServerRandom, - PrivateKey}), + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), + {dhe_psk, + PskIdentityHint, DHKeys, Params, + HashSignAlgo, ClientRandom, + ServerRandom, + PrivateKey}), State = Connection:queue_handshake(Msg, State0), State#state{diffie_hellman_keys = DHKeys}; @@ -1381,15 +1502,15 @@ key_exchange(#state{role = server, key_algorithm = rsa_psk, connection_states = ConnectionStates0, negotiated_version = Version } = State0, Connection) -> - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {psk, PskIdentityHint, - HashSignAlgo, ClientRandom, - ServerRandom, - PrivateKey}), + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), + {psk, PskIdentityHint, + HashSignAlgo, ClientRandom, + ServerRandom, + PrivateKey}), Connection:queue_handshake(Msg, State0); key_exchange(#state{role = server, key_algorithm = Algo, @@ -1410,15 +1531,15 @@ key_exchange(#state{role = server, key_algorithm = Algo, Keys0 = {_,_} -> Keys0 end, - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates0, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, - Msg = ssl_handshake:key_exchange(server, Version, {srp, Keys, SrpParams, - HashSignAlgo, ClientRandom, - ServerRandom, - PrivateKey}), + Msg = ssl_handshake:key_exchange(server, ssl:tls_version(Version), + {srp, Keys, SrpParams, + HashSignAlgo, ClientRandom, + ServerRandom, + PrivateKey}), State = Connection:queue_handshake(Msg, State0), State#state{srp_params = SrpParams, srp_keys = Keys}; @@ -1428,7 +1549,7 @@ key_exchange(#state{role = client, public_key_info = PublicKeyInfo, negotiated_version = Version, premaster_secret = PremasterSecret} = State0, Connection) -> - Msg = rsa_key_exchange(Version, PremasterSecret, PublicKeyInfo), + Msg = rsa_key_exchange(ssl:tls_version(Version), PremasterSecret, PublicKeyInfo), Connection:queue_handshake(Msg, State0); key_exchange(#state{role = client, @@ -1439,7 +1560,7 @@ key_exchange(#state{role = client, when Algorithm == dhe_dss; Algorithm == dhe_rsa; Algorithm == dh_anon -> - Msg = ssl_handshake:key_exchange(client, Version, {dh, DhPubKey}), + Msg = ssl_handshake:key_exchange(client, ssl:tls_version(Version), {dh, DhPubKey}), Connection:queue_handshake(Msg, State0); key_exchange(#state{role = client, @@ -1449,14 +1570,14 @@ key_exchange(#state{role = client, when Algorithm == ecdhe_ecdsa; Algorithm == ecdhe_rsa; Algorithm == ecdh_ecdsa; Algorithm == ecdh_rsa; Algorithm == ecdh_anon -> - Msg = ssl_handshake:key_exchange(client, Version, {ecdh, Keys}), + Msg = ssl_handshake:key_exchange(client, ssl:tls_version(Version), {ecdh, Keys}), Connection:queue_handshake(Msg, State0); key_exchange(#state{role = client, ssl_options = SslOpts, key_algorithm = psk, negotiated_version = Version} = State0, Connection) -> - Msg = ssl_handshake:key_exchange(client, Version, + Msg = ssl_handshake:key_exchange(client, ssl:tls_version(Version), {psk, SslOpts#ssl_options.psk_identity}), Connection:queue_handshake(Msg, State0); @@ -1465,7 +1586,7 @@ key_exchange(#state{role = client, key_algorithm = dhe_psk, negotiated_version = Version, diffie_hellman_keys = {DhPubKey, _}} = State0, Connection) -> - Msg = ssl_handshake:key_exchange(client, Version, + Msg = ssl_handshake:key_exchange(client, ssl:tls_version(Version), {dhe_psk, SslOpts#ssl_options.psk_identity, DhPubKey}), Connection:queue_handshake(Msg, State0); @@ -1476,7 +1597,7 @@ key_exchange(#state{role = client, negotiated_version = Version, premaster_secret = PremasterSecret} = State0, Connection) -> - Msg = rsa_psk_key_exchange(Version, SslOpts#ssl_options.psk_identity, + Msg = rsa_psk_key_exchange(ssl:tls_version(Version), SslOpts#ssl_options.psk_identity, PremasterSecret, PublicKeyInfo), Connection:queue_handshake(Msg, State0); @@ -1488,7 +1609,7 @@ key_exchange(#state{role = client, when Algorithm == srp_dss; Algorithm == srp_rsa; Algorithm == srp_anon -> - Msg = ssl_handshake:key_exchange(client, Version, {srp, ClientPubKey}), + Msg = ssl_handshake:key_exchange(client, ssl:tls_version(Version), {srp, ClientPubKey}), Connection:queue_handshake(Msg, State0). rsa_key_exchange(Version, PremasterSecret, PublicKeyInfo = {Algorithm, _, _}) @@ -1501,7 +1622,7 @@ rsa_key_exchange(Version, PremasterSecret, PublicKeyInfo = {Algorithm, _, _}) Algorithm == ?sha384WithRSAEncryption; Algorithm == ?sha512WithRSAEncryption -> - ssl_handshake:key_exchange(client, Version, + ssl_handshake:key_exchange(client, ssl:tls_version(Version), {premaster_secret, PremasterSecret, PublicKeyInfo}); rsa_key_exchange(_, _, _) -> @@ -1518,7 +1639,7 @@ rsa_psk_key_exchange(Version, PskIdentity, PremasterSecret, Algorithm == ?sha384WithRSAEncryption; Algorithm == ?sha512WithRSAEncryption -> - ssl_handshake:key_exchange(client, Version, + ssl_handshake:key_exchange(client, ssl:tls_version(Version), {psk_premaster_secret, PskIdentity, PremasterSecret, PublicKeyInfo}); rsa_psk_key_exchange(_, _, _, _) -> @@ -1530,12 +1651,14 @@ request_client_cert(#state{ssl_options = #ssl_options{verify = verify_peer, cert_db = CertDbHandle, cert_db_ref = CertDbRef, negotiated_version = Version} = State0, Connection) -> - #connection_state{security_parameters = - #security_parameters{cipher_suite = CipherSuite}} = + #{security_parameters := + #security_parameters{cipher_suite = CipherSuite}} = ssl_record:pending_connection_state(ConnectionStates0, read), - HashSigns = ssl_handshake:available_signature_algs(SupportedHashSigns, Version, [Version]), + TLSVersion = ssl:tls_version(Version), + HashSigns = ssl_handshake:available_signature_algs(SupportedHashSigns, + TLSVersion, [TLSVersion]), Msg = ssl_handshake:certificate_request(CipherSuite, CertDbHandle, CertDbRef, - HashSigns, Version), + HashSigns, TLSVersion), State = Connection:queue_handshake(Msg, State0), State#state{client_certificate_requested = true}; @@ -1548,7 +1671,7 @@ calculate_master_secret(PremasterSecret, connection_states = ConnectionStates0, session = Session0} = State0, Connection, _Current, Next) -> - case ssl_handshake:master_secret(record_cb(Connection), Version, PremasterSecret, + case ssl_handshake:master_secret(ssl:tls_version(Version), PremasterSecret, ConnectionStates0, server) of {MasterSecret, ConnectionStates} -> Session = Session0#session{master_secret = MasterSecret}, @@ -1557,7 +1680,7 @@ calculate_master_secret(PremasterSecret, {Record, State} = Connection:next_record(State1), Connection:next_event(Next, Record, State); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, certify, State0) + handle_own_alert(Alert, Version, certify, State0) end. finalize_handshake(State0, StateName, Connection) -> @@ -1590,7 +1713,7 @@ finished(#state{role = Role, negotiated_version = Version, connection_states = ConnectionStates0, tls_handshake_history = Handshake0} = State0, StateName, Connection) -> MasterSecret = Session#session.master_secret, - Finished = ssl_handshake:finished(Version, Role, + Finished = ssl_handshake:finished(ssl:tls_version(Version), Role, get_current_prf(ConnectionStates0, write), MasterSecret, Handshake0), ConnectionStates = save_verify_data(Role, Finished, ConnectionStates0, StateName), @@ -1655,7 +1778,7 @@ master_secret(#alert{} = Alert, _) -> master_secret(PremasterSecret, #state{session = Session, negotiated_version = Version, role = Role, connection_states = ConnectionStates0} = State) -> - case ssl_handshake:master_secret(tls_record, Version, PremasterSecret, + case ssl_handshake:master_secret(ssl:tls_version(Version), PremasterSecret, ConnectionStates0, Role) of {MasterSecret, ConnectionStates} -> State#state{ @@ -1738,11 +1861,11 @@ is_anonymous(_) -> false. get_current_prf(CStates, Direction) -> - CS = ssl_record:current_connection_state(CStates, Direction), - CS#connection_state.security_parameters#security_parameters.prf_algorithm. + #{security_parameters := SecParams} = ssl_record:current_connection_state(CStates, Direction), + SecParams#security_parameters.prf_algorithm. get_pending_prf(CStates, Direction) -> - CS = ssl_record:pending_connection_state(CStates, Direction), - CS#connection_state.security_parameters#security_parameters.prf_algorithm. + #{security_parameters := SecParams} = ssl_record:pending_connection_state(CStates, Direction), + SecParams#security_parameters.prf_algorithm. opposite_role(client) -> server; @@ -1964,7 +2087,7 @@ handle_resumed_session(SessId, #state{connection_states = ConnectionStates0, session_cache = Cache, session_cache_cb = CacheCb} = State0) -> Session = CacheCb:lookup(Cache, {{Host, Port}, SessId}), - case ssl_handshake:master_secret(tls_record, Version, Session, + case ssl_handshake:master_secret(ssl:tls_version(Version), Session, ConnectionStates0, client) of {_, ConnectionStates} -> {Record, State} = @@ -1973,7 +2096,7 @@ handle_resumed_session(SessId, #state{connection_states = ConnectionStates0, session = Session}), Connection:next_event(abbreviated, Record, State); #alert{} = Alert -> - Connection:handle_own_alert(Alert, Version, hello, State0) + handle_own_alert(Alert, Version, hello, State0) end. make_premaster_secret({MajVer, MinVer}, rsa) -> @@ -2040,7 +2163,7 @@ handle_active_option(_, StateName, To, Reply, #state{user_data_buffer = <<>>} = %% user_data_buffer =/= <<>> handle_active_option(_, StateName0, To, Reply, #state{protocol_cb = Connection} = State0) -> - case Connection:read_application_data(<<>>, State0) of + case read_application_data(<<>>, State0) of {stop, Reason, State} -> {stop, Reason, State}; {Record, State1} -> @@ -2054,3 +2177,264 @@ handle_active_option(_, StateName0, To, Reply, #state{protocol_cb = Connection} Stop end end. + +encode_packet(Data, #socket_options{packet=Packet}) -> + case Packet of + 1 -> encode_size_packet(Data, 8, (1 bsl 8) - 1); + 2 -> encode_size_packet(Data, 16, (1 bsl 16) - 1); + 4 -> encode_size_packet(Data, 32, (1 bsl 32) - 1); + _ -> Data + end. + +encode_size_packet(Bin, Size, Max) -> + Len = erlang:byte_size(Bin), + case Len > Max of + true -> throw({error, {badarg, {packet_to_large, Len, Max}}}); + false -> <<Len:Size, Bin/binary>> + end. + +time_to_renegotiate(_Data, + #{current_write := #{sequence_number := Num}}, + RenegotiateAt) -> + + %% We could do test: + %% is_time_to_renegotiate((erlang:byte_size(_Data) div ?MAX_PLAIN_TEXT_LENGTH) + 1, RenegotiateAt), + %% but we chose to have a some what lower renegotiateAt and a much cheaper test + is_time_to_renegotiate(Num, RenegotiateAt). + +is_time_to_renegotiate(N, M) when N < M-> + false; +is_time_to_renegotiate(_,_) -> + true. + + +%% Picks ClientData +get_data(_, _, <<>>) -> + {more, <<>>}; +%% Recv timed out save buffer data until next recv +get_data(#socket_options{active=false}, undefined, Buffer) -> + {passive, Buffer}; +get_data(#socket_options{active=Active, packet=Raw}, BytesToRead, Buffer) + when Raw =:= raw; Raw =:= 0 -> %% Raw Mode + if + Active =/= false orelse BytesToRead =:= 0 -> + %% Active true or once, or passive mode recv(0) + {ok, Buffer, <<>>}; + byte_size(Buffer) >= BytesToRead -> + %% Passive Mode, recv(Bytes) + <<Data:BytesToRead/binary, Rest/binary>> = Buffer, + {ok, Data, Rest}; + true -> + %% Passive Mode not enough data + {more, Buffer} + end; +get_data(#socket_options{packet=Type, packet_size=Size}, _, Buffer) -> + PacketOpts = [{packet_size, Size}], + case decode_packet(Type, Buffer, PacketOpts) of + {more, _} -> + {more, Buffer}; + Decoded -> + Decoded + end. + +decode_packet({http, headers}, Buffer, PacketOpts) -> + decode_packet(httph, Buffer, PacketOpts); +decode_packet({http_bin, headers}, Buffer, PacketOpts) -> + decode_packet(httph_bin, Buffer, PacketOpts); +decode_packet(Type, Buffer, PacketOpts) -> + erlang:decode_packet(Type, Buffer, PacketOpts). + +%% Just like with gen_tcp sockets, an ssl socket that has been configured with +%% {packet, http} (or {packet, http_bin}) will automatically switch to expect +%% HTTP headers after it sees a HTTP Request or HTTP Response line. We +%% represent the current state as follows: +%% #socket_options.packet =:= http: Expect a HTTP Request/Response line +%% #socket_options.packet =:= {http, headers}: Expect HTTP Headers +%% Note that if the user has explicitly configured the socket to expect +%% HTTP headers using the {packet, httph} option, we don't do any automatic +%% switching of states. +deliver_app_data(Transport, Socket, SOpts = #socket_options{active=Active, packet=Type}, + Data, Pid, From, Tracker, Connection) -> + send_or_reply(Active, Pid, From, format_reply(Transport, Socket, SOpts, Data, Tracker, Connection)), + SO = case Data of + {P, _, _, _} when ((P =:= http_request) or (P =:= http_response)), + ((Type =:= http) or (Type =:= http_bin)) -> + SOpts#socket_options{packet={Type, headers}}; + http_eoh when tuple_size(Type) =:= 2 -> + % End of headers - expect another Request/Response line + {Type1, headers} = Type, + SOpts#socket_options{packet=Type1}; + _ -> + SOpts + end, + case Active of + once -> + SO#socket_options{active=false}; + _ -> + SO + end. + +format_reply(_, _,#socket_options{active = false, mode = Mode, packet = Packet, + header = Header}, Data, _, _) -> + {ok, do_format_reply(Mode, Packet, Header, Data)}; +format_reply(Transport, Socket, #socket_options{active = _, mode = Mode, packet = Packet, + header = Header}, Data, Tracker, Connection) -> + {ssl, ssl_socket:socket(self(), Transport, Socket, Connection, Tracker), + do_format_reply(Mode, Packet, Header, Data)}. + +deliver_packet_error(Transport, Socket, SO= #socket_options{active = Active}, Data, Pid, From, Tracker, Connection) -> + send_or_reply(Active, Pid, From, format_packet_error(Transport, Socket, SO, Data, Tracker, Connection)). + +format_packet_error(_, _,#socket_options{active = false, mode = Mode}, Data, _, _) -> + {error, {invalid_packet, do_format_reply(Mode, raw, 0, Data)}}; +format_packet_error(Transport, Socket, #socket_options{active = _, mode = Mode}, Data, Tracker, Connection) -> + {ssl_error, ssl_socket:socket(self(), Transport, Socket, Connection, Tracker), + {invalid_packet, do_format_reply(Mode, raw, 0, Data)}}. + +do_format_reply(binary, _, N, Data) when N > 0 -> % Header mode + header(N, Data); +do_format_reply(binary, _, _, Data) -> + Data; +do_format_reply(list, Packet, _, Data) + when Packet == http; Packet == {http, headers}; + Packet == http_bin; Packet == {http_bin, headers}; + Packet == httph; Packet == httph_bin -> + Data; +do_format_reply(list, _,_, Data) -> + binary_to_list(Data). + +header(0, <<>>) -> + <<>>; +header(_, <<>>) -> + []; +header(0, Binary) -> + Binary; +header(N, Binary) -> + <<?BYTE(ByteN), NewBinary/binary>> = Binary, + [ByteN | header(N-1, NewBinary)]. + +send_or_reply(false, _Pid, From, Data) when From =/= undefined -> + gen_statem:reply(From, Data); +%% Can happen when handling own alert or tcp error/close and there is +%% no outstanding gen_fsm sync events +send_or_reply(false, no_pid, _, _) -> + ok; +send_or_reply(_, Pid, _From, Data) -> + send_user(Pid, Data). + +send_user(Pid, Msg) -> + Pid ! Msg. + +alert_user(Transport, Tracker, Socket, connection, Opts, Pid, From, Alert, Role, Connection) -> + alert_user(Transport, Tracker, Socket, Opts#socket_options.active, Pid, From, Alert, Role, Connection); +alert_user(Transport, Tracker, Socket,_, _, _, From, Alert, Role, Connection) -> + alert_user(Transport, Tracker, Socket, From, Alert, Role, Connection). + +alert_user(Transport, Tracker, Socket, From, Alert, Role, Connection) -> + alert_user(Transport, Tracker, Socket, false, no_pid, From, Alert, Role, Connection). + +alert_user(_, _, _, false = Active, Pid, From, Alert, Role, _) when From =/= undefined -> + %% If there is an outstanding ssl_accept | recv + %% From will be defined and send_or_reply will + %% send the appropriate error message. + ReasonCode = ssl_alert:reason_code(Alert, Role), + send_or_reply(Active, Pid, From, {error, ReasonCode}); +alert_user(Transport, Tracker, Socket, Active, Pid, From, Alert, Role, Connection) -> + case ssl_alert:reason_code(Alert, Role) of + closed -> + send_or_reply(Active, Pid, From, + {ssl_closed, ssl_socket:socket(self(), + Transport, Socket, Connection, Tracker)}); + ReasonCode -> + send_or_reply(Active, Pid, From, + {ssl_error, ssl_socket:socket(self(), + Transport, Socket, Connection, Tracker), ReasonCode}) + end. + +log_alert(true, Info, Alert) -> + Txt = ssl_alert:alert_txt(Alert), + error_logger:format("SSL: ~p: ~s\n", [Info, Txt]); +log_alert(false, _, _) -> + ok. + +handle_own_alert(Alert, Version, StateName, + #state{transport_cb = Transport, + socket = Socket, + connection_states = ConnectionStates, + ssl_options = SslOpts} = State) -> + try %% Try to tell the other side + {BinMsg, _} = + ssl_alert:encode(Alert, Version, ConnectionStates), + Transport:send(Socket, BinMsg) + catch _:_ -> %% Can crash if we are in a uninitialized state + ignore + end, + try %% Try to tell the local user + log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), + handle_normal_shutdown(Alert,StateName, State) + catch _:_ -> + ok + end, + {stop, {shutdown, own_alert}}. + +handle_normal_shutdown(Alert, _, #state{socket = Socket, + transport_cb = Transport, + protocol_cb = Connection, + start_or_recv_from = StartFrom, + tracker = Tracker, + role = Role, renegotiation = {false, first}}) -> + alert_user(Transport, Tracker,Socket, StartFrom, Alert, Role, Connection); + +handle_normal_shutdown(Alert, StateName, #state{socket = Socket, + socket_options = Opts, + transport_cb = Transport, + protocol_cb = Connection, + user_application = {_Mon, Pid}, + tracker = Tracker, + start_or_recv_from = RecvFrom, role = Role}) -> + alert_user(Transport, Tracker, Socket, StateName, Opts, Pid, RecvFrom, Alert, Role, Connection). + +invalidate_session(client, Host, Port, Session) -> + ssl_manager:invalidate_session(Host, Port, Session); +invalidate_session(server, _, Port, Session) -> + ssl_manager:invalidate_session(Port, Session). + +handle_sni_extension(undefined, State) -> + State; +handle_sni_extension(#sni{hostname = Hostname}, State0) -> + NewOptions = update_ssl_options_from_sni(State0#state.ssl_options, Hostname), + case NewOptions of + undefined -> + State0; + _ -> + {ok, Ref, CertDbHandle, FileRefHandle, CacheHandle, CRLDbHandle, OwnCert, Key, DHParams} = + ssl_config:init(NewOptions, State0#state.role), + State0#state{ + session = State0#state.session#session{own_certificate = OwnCert}, + file_ref_db = FileRefHandle, + cert_db_ref = Ref, + cert_db = CertDbHandle, + crl_db = CRLDbHandle, + session_cache = CacheHandle, + private_key = Key, + diffie_hellman_params = DHParams, + ssl_options = NewOptions, + sni_hostname = Hostname + } + end. + +update_ssl_options_from_sni(OrigSSLOptions, SNIHostname) -> + SSLOption = + case OrigSSLOptions#ssl_options.sni_fun of + undefined -> + proplists:get_value(SNIHostname, + OrigSSLOptions#ssl_options.sni_hosts); + SNIFun -> + SNIFun(SNIHostname) + end, + case SSLOption of + undefined -> + undefined; + _ -> + ssl:handle_options(SSLOption, OrigSSLOptions) + end. diff --git a/lib/ssl/src/ssl_connection.hrl b/lib/ssl/src/ssl_connection.hrl index 4b54943ddf..f1e612a41b 100644 --- a/lib/ssl/src/ssl_connection.hrl +++ b/lib/ssl/src/ssl_connection.hrl @@ -46,7 +46,7 @@ socket :: port(), ssl_options :: #ssl_options{}, socket_options :: #socket_options{}, - connection_states :: #connection_states{} | secret_printout(), + connection_states :: ssl_record:connection_states() | secret_printout(), protocol_buffers :: term() | secret_printout() , %% #protocol_buffers{} from tls_record.hrl or dtls_recor.hrl tls_handshake_history :: ssl_handshake:ssl_handshake_history() | secret_printout() | 'undefined', diff --git a/lib/ssl/src/ssl_crl.erl b/lib/ssl/src/ssl_crl.erl index d9f21e04ac..01be1fb9ab 100644 --- a/lib/ssl/src/ssl_crl.erl +++ b/lib/ssl/src/ssl_crl.erl @@ -47,7 +47,7 @@ trusted_cert_and_path(CRL, issuer_not_found, {Db, DbRef} = DbHandle) -> {ok, unknown_crl_ca, []} end. -find_issuer(CRL, {Db,_}) -> +find_issuer(CRL, {Db,DbRef}) -> Issuer = public_key:pkix_normalize_name(public_key:pkix_crl_issuer(CRL)), IsIssuerFun = fun({_Key, {_Der,ErlCertCandidate}}, Acc) -> @@ -55,15 +55,27 @@ find_issuer(CRL, {Db,_}) -> (_, Acc) -> Acc end, - - try ssl_pkix_db:foldl(IsIssuerFun, issuer_not_found, Db) of - issuer_not_found -> - {error, issuer_not_found} - catch - {ok, _} = Result -> - Result + if is_reference(DbRef) -> % actual DB exists + try ssl_pkix_db:foldl(IsIssuerFun, issuer_not_found, Db) of + issuer_not_found -> + {error, issuer_not_found} + catch + {ok, _} = Result -> + Result + end; + is_tuple(DbRef), element(1,DbRef) =:= extracted -> % cache bypass byproduct + {extracted, CertsData} = DbRef, + Certs = [Entry || {decoded, Entry} <- CertsData], + try lists:foldl(IsIssuerFun, issuer_not_found, Certs) of + issuer_not_found -> + {error, issuer_not_found} + catch + {ok, _} = Result -> + Result + end end. + verify_crl_issuer(CRL, ErlCertCandidate, Issuer, NotIssuer) -> TBSCert = ErlCertCandidate#'OTPCertificate'.tbsCertificate, case public_key:pkix_normalize_name(TBSCert#'OTPTBSCertificate'.subject) of diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 081efda768..5b51ac0916 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -51,8 +51,8 @@ %% Handle handshake messages -export([certify/10, client_certificate_verify/6, certificate_verify/6, verify_signature/5, - master_secret/5, server_key_exchange_hash/2, verify_connection/6, - init_handshake_history/0, update_handshake_history/2, verify_server_key/5 + master_secret/4, server_key_exchange_hash/2, verify_connection/6, + init_handshake_history/0, update_handshake_history/3, verify_server_key/5 ]). %% Encode/Decode @@ -94,15 +94,14 @@ hello_request() -> #hello_request{}. %%-------------------------------------------------------------------- --spec server_hello(#session{}, ssl_record:ssl_version(), #connection_states{}, +-spec server_hello(#session{}, ssl_record:ssl_version(), ssl_record:connection_states(), #hello_extensions{}) -> #server_hello{}. %% %% Description: Creates a server hello message. %%-------------------------------------------------------------------- server_hello(SessionId, Version, ConnectionStates, Extensions) -> - Pending = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = Pending#connection_state.security_parameters, - + #{security_parameters := SecParams} = + ssl_record:pending_connection_state(ConnectionStates, read), #server_hello{server_version = Version, cipher_suite = SecParams#security_parameters.cipher_suite, compression_method = @@ -335,9 +334,8 @@ verify_server_key(#server_key_params{params_bin = EncParams, signature = Signature}, HashSign = {HashAlgo, _}, ConnectionStates, Version, PubKeyInfo) -> - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = ConnectionState#connection_state.security_parameters, #security_parameters{client_random = ClientRandom, server_random = ServerRandom} = SecParams, Hash = server_key_exchange_hash(HashAlgo, @@ -447,7 +445,7 @@ init_handshake_history() -> {[], []}. %%-------------------------------------------------------------------- --spec update_handshake_history(ssl_handshake:ssl_handshake_history(), Data ::term()) -> +-spec update_handshake_history(ssl_handshake:ssl_handshake_history(), Data ::term(), boolean()) -> ssl_handshake:ssl_handshake_history(). %% %% Description: Update the handshake history buffer with Data. @@ -457,14 +455,14 @@ update_handshake_history(Handshake, % special-case SSL2 client hello ?UINT16(CSLength), ?UINT16(0), ?UINT16(CDLength), CipherSuites:CSLength/binary, - ChallengeData:CDLength/binary>>) -> + ChallengeData:CDLength/binary>>, true) -> update_handshake_history(Handshake, <<?CLIENT_HELLO, ?BYTE(Major), ?BYTE(Minor), ?UINT16(CSLength), ?UINT16(0), ?UINT16(CDLength), CipherSuites:CSLength/binary, - ChallengeData:CDLength/binary>>); -update_handshake_history({Handshake0, _Prev}, Data) -> + ChallengeData:CDLength/binary>>, true); +update_handshake_history({Handshake0, _Prev}, Data, _) -> {[Data|Handshake0], Handshake0}. %% %%-------------------------------------------------------------------- @@ -696,33 +694,32 @@ select_hashsign_algs(undefined, ?'id-dsa', _) -> %%-------------------------------------------------------------------- --spec master_secret(atom(), ssl_record:ssl_version(), #session{} | binary(), #connection_states{}, - client | server) -> {binary(), #connection_states{}} | #alert{}. +-spec master_secret(ssl_record:ssl_version(), #session{} | binary(), ssl_record:connection_states(), + client | server) -> {binary(), ssl_record:connection_states()} | #alert{}. %% %% Description: Sets or calculates the master secret and calculate keys, %% updating the pending connection states. The Mastersecret and the update %% connection states are returned or an alert if the calculation fails. %%------------------------------------------------------------------- -master_secret(RecordCB, Version, #session{master_secret = Mastersecret}, +master_secret(Version, #session{master_secret = Mastersecret}, ConnectionStates, Role) -> - ConnectionState = + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = ConnectionState#connection_state.security_parameters, - try master_secret(RecordCB, Version, Mastersecret, SecParams, + try master_secret(Version, Mastersecret, SecParams, ConnectionStates, Role) catch exit:_ -> ?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, key_calculation_failure) end; -master_secret(RecordCB, Version, PremasterSecret, ConnectionStates, Role) -> - ConnectionState = +master_secret(Version, PremasterSecret, ConnectionStates, Role) -> + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = ConnectionState#connection_state.security_parameters, + #security_parameters{prf_algorithm = PrfAlgo, client_random = ClientRandom, server_random = ServerRandom} = SecParams, - try master_secret(RecordCB, Version, + try master_secret(Version, calc_master_secret(Version,PrfAlgo,PremasterSecret, ClientRandom, ServerRandom), SecParams, ConnectionStates, Role) @@ -1219,13 +1216,18 @@ certificate_authorities(CertDbHandle, CertDbRef) -> end, list_to_binary([Enc(Cert) || {_, Cert} <- Authorities]). -certificate_authorities_from_db(CertDbHandle, CertDbRef) -> +certificate_authorities_from_db(CertDbHandle, CertDbRef) when is_reference(CertDbRef) -> ConnectionCerts = fun({{Ref, _, _}, Cert}, Acc) when Ref == CertDbRef -> [Cert | Acc]; (_, Acc) -> Acc end, - ssl_pkix_db:foldl(ConnectionCerts, [], CertDbHandle). + ssl_pkix_db:foldl(ConnectionCerts, [], CertDbHandle); +certificate_authorities_from_db(_CertDbHandle, {extracted, CertDbData}) -> + %% Cache disabled, Ref contains data + lists:foldl(fun({decoded, {_Key,Cert}}, Acc) -> [Cert | Acc] end, + [], CertDbData). + %%-------------Extension handling -------------------------------- @@ -1343,29 +1345,29 @@ do_select_version( renegotiation_info(_, client, _, false) -> #renegotiation_info{renegotiated_connection = undefined}; renegotiation_info(_RecordCB, server, ConnectionStates, false) -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - case CS#connection_state.secure_renegotiation of + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + case maps:get(secure_renegotiation, ConnectionState) of true -> #renegotiation_info{renegotiated_connection = ?byte(0)}; false -> #renegotiation_info{renegotiated_connection = undefined} end; renegotiation_info(_RecordCB, client, ConnectionStates, true) -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - case CS#connection_state.secure_renegotiation of + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + case maps:get(secure_renegotiation, ConnectionState) of true -> - Data = CS#connection_state.client_verify_data, + Data = maps:get(client_verify_data, ConnectionState), #renegotiation_info{renegotiated_connection = Data}; false -> #renegotiation_info{renegotiated_connection = undefined} end; renegotiation_info(_RecordCB, server, ConnectionStates, true) -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - case CS#connection_state.secure_renegotiation of + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + case maps:get(secure_renegotiation, ConnectionState) of true -> - CData = CS#connection_state.client_verify_data, - SData =CS#connection_state.server_verify_data, + CData = maps:get(client_verify_data, ConnectionState), + SData = maps:get(server_verify_data, ConnectionState), #renegotiation_info{renegotiated_connection = <<CData/binary, SData/binary>>}; false -> #renegotiation_info{renegotiated_connection = undefined} @@ -1388,9 +1390,9 @@ handle_renegotiation_info(_RecordCB, _, undefined, ConnectionStates, false, _, _ handle_renegotiation_info(_RecordCB, client, #renegotiation_info{renegotiated_connection = ClientServerVerify}, ConnectionStates, true, _, _) -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - CData = CS#connection_state.client_verify_data, - SData = CS#connection_state.server_verify_data, + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + CData = maps:get(client_verify_data, ConnectionState), + SData = maps:get(server_verify_data, ConnectionState), case <<CData/binary, SData/binary>> == ClientServerVerify of true -> {ok, ConnectionStates}; @@ -1404,8 +1406,8 @@ handle_renegotiation_info(_RecordCB, server, #renegotiation_info{renegotiated_co true -> ?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, {server_renegotiation, empty_renegotiation_info_scsv}); false -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - Data = CS#connection_state.client_verify_data, + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + Data = maps:get(client_verify_data, ConnectionState), case Data == ClientVerify of true -> {ok, ConnectionStates}; @@ -1426,8 +1428,8 @@ handle_renegotiation_info(RecordCB, server, undefined, ConnectionStates, true, S end. handle_renegotiation_info(_RecordCB, ConnectionStates, SecureRenegotation) -> - CS = ssl_record:current_connection_state(ConnectionStates, read), - case {SecureRenegotation, CS#connection_state.secure_renegotiation} of + ConnectionState = ssl_record:current_connection_state(ConnectionStates, read), + case {SecureRenegotation, maps:get(secure_renegotiation, ConnectionState)} of {_, true} -> ?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, already_secure); {true, false} -> @@ -1645,7 +1647,7 @@ calc_finished({3, 0}, Role, _PrfAlgo, MasterSecret, Handshake) -> calc_finished({3, N}, Role, PrfAlgo, MasterSecret, Handshake) -> tls_v1:finished(Role, N, PrfAlgo, MasterSecret, lists:reverse(Handshake)). -master_secret(_RecordCB, Version, MasterSecret, +master_secret(Version, MasterSecret, #security_parameters{ bulk_cipher_algorithm = BCA, client_random = ClientRandom, @@ -1728,18 +1730,16 @@ hello_pending_connection_states(_RecordCB, Role, Version, CipherSuite, Random, C NewWriteSecParams, ConnectionStates). -hello_security_parameters(client, Version, ConnectionState, CipherSuite, Random, +hello_security_parameters(client, Version, #{security_parameters := SecParams}, CipherSuite, Random, Compression) -> - SecParams = ConnectionState#connection_state.security_parameters, NewSecParams = ssl_cipher:security_parameters(Version, CipherSuite, SecParams), NewSecParams#security_parameters{ server_random = Random, compression_algorithm = Compression }; -hello_security_parameters(server, Version, ConnectionState, CipherSuite, Random, +hello_security_parameters(server, Version, #{security_parameters := SecParams}, CipherSuite, Random, Compression) -> - SecParams = ConnectionState#connection_state.security_parameters, NewSecParams = ssl_cipher:security_parameters(Version, CipherSuite, SecParams), NewSecParams#security_parameters{ client_random = Random, diff --git a/lib/ssl/src/ssl_manager.erl b/lib/ssl/src/ssl_manager.erl index c7dcbaabe9..5bd9521de7 100644 --- a/lib/ssl/src/ssl_manager.erl +++ b/lib/ssl/src/ssl_manager.erl @@ -115,13 +115,25 @@ start_link_dist(Opts) -> %% Description: Do necessary initializations for a new connection. %%-------------------------------------------------------------------- connection_init({der, _} = Trustedcerts, Role, CRLCache) -> - call({connection_init, Trustedcerts, Role, CRLCache}); + case bypass_pem_cache() of + true -> + {ok, Extracted} = ssl_pkix_db:extract_trusted_certs(Trustedcerts), + call({connection_init, Extracted, Role, CRLCache}); + false -> + call({connection_init, Trustedcerts, Role, CRLCache}) + end; connection_init(<<>> = Trustedcerts, Role, CRLCache) -> call({connection_init, Trustedcerts, Role, CRLCache}); connection_init(Trustedcerts, Role, CRLCache) -> - call({connection_init, Trustedcerts, Role, CRLCache}). + case bypass_pem_cache() of + true -> + {ok, Extracted} = ssl_pkix_db:extract_trusted_certs(Trustedcerts), + call({connection_init, Extracted, Role, CRLCache}); + false -> + call({connection_init, Trustedcerts, Role, CRLCache}) + end. %%-------------------------------------------------------------------- -spec cache_pem_file(binary(), term()) -> {ok, term()} | {error, reason()}. @@ -129,13 +141,18 @@ connection_init(Trustedcerts, Role, CRLCache) -> %% Description: Cache a pem file and return its content. %%-------------------------------------------------------------------- cache_pem_file(File, DbHandle) -> - case ssl_pkix_db:lookup_cached_pem(DbHandle, File) of - [{Content,_}] -> - {ok, Content}; - [Content] -> - {ok, Content}; - undefined -> - call({cache_pem, File}) + case bypass_pem_cache() of + true -> + ssl_pkix_db:decode_pem_file(File); + false -> + case ssl_pkix_db:lookup_cached_pem(DbHandle, File) of + [{Content,_}] -> + {ok, Content}; + [Content] -> + {ok, Content}; + undefined -> + call({cache_pem, File}) + end end. %%-------------------------------------------------------------------- @@ -506,6 +523,14 @@ delay_time() -> ?CLEAN_SESSION_DB end. +bypass_pem_cache() -> + case application:get_env(ssl, bypass_pem_cache) of + {ok, Bool} when is_boolean(Bool) -> + Bool; + _ -> + false + end. + max_session_cache_size(CacheType) -> case application:get_env(ssl, CacheType) of {ok, Size} when is_integer(Size) -> diff --git a/lib/ssl/src/ssl_pkix_db.erl b/lib/ssl/src/ssl_pkix_db.erl index b16903d7c7..0006ce14d9 100644 --- a/lib/ssl/src/ssl_pkix_db.erl +++ b/lib/ssl/src/ssl_pkix_db.erl @@ -29,10 +29,11 @@ -include_lib("kernel/include/file.hrl"). -export([create/0, add_crls/3, remove_crls/2, remove/1, add_trusted_certs/3, + extract_trusted_certs/1, remove_trusted_certs/2, insert/3, remove/2, clear/1, db_size/1, ref_count/3, lookup_trusted_cert/4, foldl/3, select_cert_by_issuer/2, lookup_cached_pem/2, cache_pem_file/2, cache_pem_file/3, - lookup/2]). + decode_pem_file/1, lookup/2]). %%==================================================================== %% Internal application API @@ -82,12 +83,22 @@ remove(Dbs) -> %% <SerialNumber, Issuer>. Ref is used as it is specified %% for each connection which certificates are trusted. %%-------------------------------------------------------------------- -lookup_trusted_cert(DbHandle, Ref, SerialNumber, Issuer) -> +lookup_trusted_cert(DbHandle, Ref, SerialNumber, Issuer) when is_reference(Ref) -> case lookup({Ref, SerialNumber, Issuer}, DbHandle) of undefined -> undefined; [Certs] -> {ok, Certs} + end; +lookup_trusted_cert(_DbHandle, {extracted,Certs}, SerialNumber, Issuer) -> + try + [throw(Cert) + || {decoded, {{_Ref,CertSerial,CertIssuer}, Cert}} <- Certs, + CertSerial =:= SerialNumber, CertIssuer =:= Issuer], + undefined + catch + Cert -> + {ok, Cert} end. lookup_cached_pem([_, _, PemChache | _], File) -> @@ -103,6 +114,9 @@ lookup_cached_pem(PemChache, File) -> %% runtime database. Returns Ref that should be handed to lookup_trusted_cert %% together with the cert serialnumber and issuer. %%-------------------------------------------------------------------- +add_trusted_certs(_Pid, {extracted, _} = Certs, _) -> + {ok, Certs}; + add_trusted_certs(_Pid, {der, DerList}, [CertDb, _,_ | _]) -> NewRef = make_ref(), add_certs_from_der(DerList, NewRef, CertDb), @@ -122,6 +136,21 @@ add_trusted_certs(_Pid, File, [CertsDb, RefDb, PemChache | _] = Db) -> undefined -> new_trusted_cert_entry(File, Db) end. + +extract_trusted_certs({der, DerList}) -> + {ok, {extracted, certs_from_der(DerList)}}; +extract_trusted_certs(File) -> + case file:read_file(File) of + {ok, PemBin} -> + Content = public_key:pem_decode(PemBin), + DerList = [Cert || {'Certificate', Cert, not_encrypted} <- Content], + {ok, {extracted, certs_from_der(DerList)}}; + Error -> + %% Have to simulate a failure happening in a server for + %% external handlers. + {error, {badmatch, Error}} + end. + %%-------------------------------------------------------------------- %% %% Description: Cache file as binary in DB @@ -141,6 +170,18 @@ cache_pem_file(Ref, File, [_CertsDb, _RefDb, PemChache| _]) -> insert(File, {Content, Ref}, PemChache), {ok, Content}. +-spec decode_pem_file(binary()) -> {ok, term()}. +decode_pem_file(File) -> + case file:read_file(File) of + {ok, PemBin} -> + Content = public_key:pem_decode(PemBin), + {ok, Content}; + Error -> + %% Have to simulate a failure happening in a server for + %% external handlers. + {error, {badmatch, Error}} + end. + %%-------------------------------------------------------------------- -spec remove_trusted_certs(reference(), db_handle()) -> ok. %% @@ -203,6 +244,8 @@ select_cert_by_issuer(Cache, Issuer) -> %% %% Description: Updates a reference counter in a <Db>. %%-------------------------------------------------------------------- +ref_count({extracted, _}, _Db, _N) -> + not_cached; ref_count(Key, Db, N) -> ets:update_counter(Db,Key,N). @@ -248,23 +291,39 @@ add_certs_from_der(DerList, Ref, CertsDb) -> [Add(Cert) || Cert <- DerList], ok. +certs_from_der(DerList) -> + Ref = make_ref(), + [Decoded || Cert <- DerList, + Decoded <- [decode_certs(Ref, Cert)], + Decoded =/= undefined]. + add_certs_from_pem(PemEntries, Ref, CertsDb) -> Add = fun(Cert) -> add_certs(Cert, Ref, CertsDb) end, [Add(Cert) || {'Certificate', Cert, not_encrypted} <- PemEntries], ok. add_certs(Cert, Ref, CertsDb) -> + try + {decoded, {Key, Val}} = decode_certs(Ref, Cert), + insert(Key, Val, CertsDb) + catch + error:_ -> + ok + end. + +decode_certs(Ref, Cert) -> try ErlCert = public_key:pkix_decode_cert(Cert, otp), TBSCertificate = ErlCert#'OTPCertificate'.tbsCertificate, SerialNumber = TBSCertificate#'OTPTBSCertificate'.serialNumber, Issuer = public_key:pkix_normalize_name( TBSCertificate#'OTPTBSCertificate'.issuer), - insert({Ref, SerialNumber, Issuer}, {Cert,ErlCert}, CertsDb) + {decoded, {{Ref, SerialNumber, Issuer}, {Cert, ErlCert}}} catch error:_ -> Report = io_lib:format("SSL WARNING: Ignoring a CA cert as " "it could not be correctly decoded.~n", []), - error_logger:info_report(Report) + error_logger:info_report(Report), + undefined end. new_trusted_cert_entry(File, [CertsDb, RefDb, _ | _] = Db) -> diff --git a/lib/ssl/src/ssl_record.erl b/lib/ssl/src/ssl_record.erl index 5bb1c92c2d..71cd0279f3 100644 --- a/lib/ssl/src/ssl_record.erl +++ b/lib/ssl/src/ssl_record.erl @@ -30,8 +30,7 @@ -include("ssl_alert.hrl"). %% Connection state handling --export([init_connection_states/2, - current_connection_state/2, pending_connection_state/2, +-export([initial_security_params/1, current_connection_state/2, pending_connection_state/2, activate_pending_connection_state/2, set_security_params/3, set_mac_secret/4, @@ -39,7 +38,8 @@ set_pending_cipher_state/4, set_renegotiation_flag/2, set_client_verify_data/3, - set_server_verify_data/3]). + set_server_verify_data/3, + empty_connection_state/2, initial_connection_state/2, record_protocol_role/1]). %% Encoding records -export([encode_handshake/3, encode_alert_record/3, @@ -52,122 +52,92 @@ -export([cipher/4, decipher/4, is_correct_mac/2, cipher_aead/4, decipher_aead/4]). --export_type([ssl_version/0, ssl_atom_version/0]). +-export_type([ssl_version/0, ssl_atom_version/0, connection_states/0, connection_state/0]). -type ssl_version() :: {integer(), integer()}. -type ssl_atom_version() :: tls_record:tls_atom_version(). - +-type connection_states() :: term(). %% Map +-type connection_state() :: term(). %% Map %%==================================================================== %% Internal application API %%==================================================================== + %%-------------------------------------------------------------------- --spec init_connection_states(client | server, one_n_minus_one | zero_n | disabled ) -> - #connection_states{}. -%% -%% Description: Creates a connection_states record with appropriate -%% values for the initial SSL connection setup. -%%-------------------------------------------------------------------- -init_connection_states(Role, BeastMitigation) -> - ConnectionEnd = record_protocol_role(Role), - Current = initial_connection_state(ConnectionEnd, BeastMitigation), - Pending = empty_connection_state(ConnectionEnd, BeastMitigation), - #connection_states{dtls_write_msg_seq = 1, % only used by dtls - current_read = Current, - pending_read = Pending, - current_write = Current, - pending_write = Pending - }. - -%%-------------------------------------------------------------------- --spec current_connection_state(#connection_states{}, read | write) -> - #connection_state{}. +-spec current_connection_state(connection_states(), read | write) -> + connection_state(). %% -%% Description: Returns the instance of the connection_state record +%% Description: Returns the instance of the connection_state map %% that is currently defined as the current conection state. %%-------------------------------------------------------------------- -current_connection_state(#connection_states{current_read = Current}, - read) -> - Current; -current_connection_state(#connection_states{current_write = Current}, - write) -> - Current. +current_connection_state(ConnectionStates, read) -> + maps:get(current_read, ConnectionStates); +current_connection_state(ConnectionStates, write) -> + maps:get(current_write, ConnectionStates). %%-------------------------------------------------------------------- --spec pending_connection_state(#connection_states{}, read | write) -> - term(). +-spec pending_connection_state(connection_states(), read | write) -> + connection_state(). %% -%% Description: Returns the instance of the connection_state record -%% that is currently defined as the pending conection state. +%% Description: Returns the instance of the connection_state map +%% that is pendingly defined as the pending conection state. %%-------------------------------------------------------------------- -pending_connection_state(#connection_states{pending_read = Pending}, - read) -> - Pending; -pending_connection_state(#connection_states{pending_write = Pending}, - write) -> - Pending. - +pending_connection_state(ConnectionStates, read) -> + maps:get(pending_read, ConnectionStates); +pending_connection_state(ConnectionStates, write) -> + maps:get(pending_write, ConnectionStates). %%-------------------------------------------------------------------- --spec activate_pending_connection_state(#connection_states{}, read | write) -> - #connection_states{}. +-spec activate_pending_connection_state(connection_states(), read | write) -> + connection_states(). %% %% Description: Creates a new instance of the connection_states record %% where the pending state of <Type> has been activated. %%-------------------------------------------------------------------- -activate_pending_connection_state(States = - #connection_states{current_read = Current, - pending_read = Pending}, +activate_pending_connection_state(#{current_read := Current, + pending_read := Pending} = States, read) -> - NewCurrent = Pending#connection_state{epoch = dtls_next_epoch(Current), - sequence_number = 0}, - BeastMitigation = Pending#connection_state.beast_mitigation, - SecParams = Pending#connection_state.security_parameters, + #{secure_renegotiation := SecureRenegotation} = Current, + #{beast_mitigation := BeastMitigation, + security_parameters := SecParams} = Pending, + NewCurrent = Pending#{sequence_number => 0}, ConnectionEnd = SecParams#security_parameters.connection_end, EmptyPending = empty_connection_state(ConnectionEnd, BeastMitigation), - SecureRenegotation = NewCurrent#connection_state.secure_renegotiation, - NewPending = EmptyPending#connection_state{secure_renegotiation = SecureRenegotation}, - States#connection_states{current_read = NewCurrent, - pending_read = NewPending - }; - -activate_pending_connection_state(States = - #connection_states{current_write = Current, - pending_write = Pending}, + NewPending = EmptyPending#{secure_renegotiation => SecureRenegotation}, + States#{current_read => NewCurrent, + pending_read => NewPending + }; + +activate_pending_connection_state(#{current_write := Current, + pending_write := Pending} = States, write) -> - NewCurrent = Pending#connection_state{epoch = dtls_next_epoch(Current), - sequence_number = 0}, - BeastMitigation = Pending#connection_state.beast_mitigation, - SecParams = Pending#connection_state.security_parameters, + NewCurrent = Pending#{sequence_number => 0}, + #{secure_renegotiation := SecureRenegotation} = Current, + #{beast_mitigation := BeastMitigation, + security_parameters := SecParams} = Pending, ConnectionEnd = SecParams#security_parameters.connection_end, EmptyPending = empty_connection_state(ConnectionEnd, BeastMitigation), - SecureRenegotation = NewCurrent#connection_state.secure_renegotiation, - NewPending = EmptyPending#connection_state{secure_renegotiation = SecureRenegotation}, - States#connection_states{current_write = NewCurrent, - pending_write = NewPending - }. - + NewPending = EmptyPending#{secure_renegotiation => SecureRenegotation}, + States#{current_write => NewCurrent, + pending_write => NewPending + }. %%-------------------------------------------------------------------- -spec set_security_params(#security_parameters{}, #security_parameters{}, - #connection_states{}) -> #connection_states{}. + connection_states()) -> connection_states(). %% %% Description: Creates a new instance of the connection_states record %% where the pending states gets its security parameters updated. %%-------------------------------------------------------------------- -set_security_params(ReadParams, WriteParams, States = - #connection_states{pending_read = Read, - pending_write = Write}) -> - States#connection_states{pending_read = - Read#connection_state{security_parameters = - ReadParams}, - pending_write = - Write#connection_state{security_parameters = - WriteParams} - }. +set_security_params(ReadParams, WriteParams, + #{pending_read := Read, + pending_write := Write} = States) -> + States#{pending_read => Read#{security_parameters => ReadParams}, + pending_write => Write#{security_parameters => WriteParams} + }. %%-------------------------------------------------------------------- -spec set_mac_secret(binary(), binary(), client | server, - #connection_states{}) -> #connection_states{}. + connection_states()) -> connection_states(). %% %% Description: update the mac_secret field in pending connection states %%-------------------------------------------------------------------- @@ -177,152 +147,145 @@ set_mac_secret(ClientWriteMacSecret, ServerWriteMacSecret, server, States) -> set_mac_secret(ClientWriteMacSecret, ServerWriteMacSecret, States). set_mac_secret(ReadMacSecret, WriteMacSecret, - States = #connection_states{pending_read = Read, - pending_write = Write}) -> - States#connection_states{ - pending_read = Read#connection_state{mac_secret = ReadMacSecret}, - pending_write = Write#connection_state{mac_secret = WriteMacSecret} + States = #{pending_read := Read, + pending_write := Write}) -> + States#{pending_read => Read#{mac_secret => ReadMacSecret}, + pending_write => Write#{mac_secret => WriteMacSecret} }. %%-------------------------------------------------------------------- --spec set_master_secret(binary(), #connection_states{}) -> #connection_states{}. +-spec set_master_secret(binary(), connection_states()) -> connection_states(). %% %% Description: Set master_secret in pending connection states %%-------------------------------------------------------------------- set_master_secret(MasterSecret, - States = #connection_states{pending_read = Read, - pending_write = Write}) -> - ReadSecPar = Read#connection_state.security_parameters, - Read1 = Read#connection_state{ - security_parameters = ReadSecPar#security_parameters{ - master_secret = MasterSecret}}, - WriteSecPar = Write#connection_state.security_parameters, - Write1 = Write#connection_state{ - security_parameters = WriteSecPar#security_parameters{ - master_secret = MasterSecret}}, - States#connection_states{pending_read = Read1, pending_write = Write1}. - -%%-------------------------------------------------------------------- --spec set_renegotiation_flag(boolean(), #connection_states{}) -> #connection_states{}. + States = #{pending_read := Read = #{security_parameters := ReadSecPar}, + pending_write := Write = #{security_parameters := WriteSecPar}}) -> + Read1 = Read#{security_parameters => ReadSecPar#security_parameters{ + master_secret = MasterSecret}}, + Write1 = Write#{security_parameters => WriteSecPar#security_parameters{ + master_secret = MasterSecret}}, + States#{pending_read => Read1, pending_write => Write1}. + +%%-------------------------------------------------------------------- +-spec set_renegotiation_flag(boolean(), connection_states()) -> connection_states(). %% %% Description: Set secure_renegotiation in pending connection states %%-------------------------------------------------------------------- -set_renegotiation_flag(Flag, #connection_states{ - current_read = CurrentRead0, - current_write = CurrentWrite0, - pending_read = PendingRead0, - pending_write = PendingWrite0} +set_renegotiation_flag(Flag, #{current_read := CurrentRead0, + current_write := CurrentWrite0, + pending_read := PendingRead0, + pending_write := PendingWrite0} = ConnectionStates) -> - CurrentRead = CurrentRead0#connection_state{secure_renegotiation = Flag}, - CurrentWrite = CurrentWrite0#connection_state{secure_renegotiation = Flag}, - PendingRead = PendingRead0#connection_state{secure_renegotiation = Flag}, - PendingWrite = PendingWrite0#connection_state{secure_renegotiation = Flag}, - ConnectionStates#connection_states{current_read = CurrentRead, - current_write = CurrentWrite, - pending_read = PendingRead, - pending_write = PendingWrite}. + CurrentRead = CurrentRead0#{secure_renegotiation => Flag}, + CurrentWrite = CurrentWrite0#{secure_renegotiation => Flag}, + PendingRead = PendingRead0#{secure_renegotiation => Flag}, + PendingWrite = PendingWrite0#{secure_renegotiation => Flag}, + ConnectionStates#{current_read => CurrentRead, + current_write => CurrentWrite, + pending_read => PendingRead, + pending_write => PendingWrite}. %%-------------------------------------------------------------------- -spec set_client_verify_data(current_read | current_write | current_both, - binary(), #connection_states{})-> - #connection_states{}. + binary(), connection_states())-> + connection_states(). %% %% Description: Set verify data in connection states. %%-------------------------------------------------------------------- set_client_verify_data(current_read, Data, - #connection_states{current_read = CurrentRead0, - pending_write = PendingWrite0} + #{current_read := CurrentRead0, + pending_write := PendingWrite0} = ConnectionStates) -> - CurrentRead = CurrentRead0#connection_state{client_verify_data = Data}, - PendingWrite = PendingWrite0#connection_state{client_verify_data = Data}, - ConnectionStates#connection_states{current_read = CurrentRead, - pending_write = PendingWrite}; + CurrentRead = CurrentRead0#{client_verify_data => Data}, + PendingWrite = PendingWrite0#{client_verify_data => Data}, + ConnectionStates#{current_read => CurrentRead, + pending_write => PendingWrite}; set_client_verify_data(current_write, Data, - #connection_states{pending_read = PendingRead0, - current_write = CurrentWrite0} + #{pending_read := PendingRead0, + current_write := CurrentWrite0} = ConnectionStates) -> - PendingRead = PendingRead0#connection_state{client_verify_data = Data}, - CurrentWrite = CurrentWrite0#connection_state{client_verify_data = Data}, - ConnectionStates#connection_states{pending_read = PendingRead, - current_write = CurrentWrite}; + PendingRead = PendingRead0#{client_verify_data => Data}, + CurrentWrite = CurrentWrite0#{client_verify_data => Data}, + ConnectionStates#{pending_read => PendingRead, + current_write => CurrentWrite}; set_client_verify_data(current_both, Data, - #connection_states{current_read = CurrentRead0, - current_write = CurrentWrite0} + #{current_read := CurrentRead0, + current_write := CurrentWrite0} = ConnectionStates) -> - CurrentRead = CurrentRead0#connection_state{client_verify_data = Data}, - CurrentWrite = CurrentWrite0#connection_state{client_verify_data = Data}, - ConnectionStates#connection_states{current_read = CurrentRead, - current_write = CurrentWrite}. + CurrentRead = CurrentRead0#{client_verify_data => Data}, + CurrentWrite = CurrentWrite0#{client_verify_data => Data}, + ConnectionStates#{current_read => CurrentRead, + current_write => CurrentWrite}. %%-------------------------------------------------------------------- -spec set_server_verify_data(current_read | current_write | current_both, - binary(), #connection_states{})-> - #connection_states{}. + binary(), connection_states())-> + connection_states(). %% %% Description: Set verify data in pending connection states. %%-------------------------------------------------------------------- set_server_verify_data(current_write, Data, - #connection_states{pending_read = PendingRead0, - current_write = CurrentWrite0} + #{pending_read := PendingRead0, + current_write := CurrentWrite0} = ConnectionStates) -> - PendingRead = PendingRead0#connection_state{server_verify_data = Data}, - CurrentWrite = CurrentWrite0#connection_state{server_verify_data = Data}, - ConnectionStates#connection_states{pending_read = PendingRead, - current_write = CurrentWrite}; + PendingRead = PendingRead0#{server_verify_data => Data}, + CurrentWrite = CurrentWrite0#{server_verify_data => Data}, + ConnectionStates#{pending_read => PendingRead, + current_write => CurrentWrite}; set_server_verify_data(current_read, Data, - #connection_states{current_read = CurrentRead0, - pending_write = PendingWrite0} + #{current_read := CurrentRead0, + pending_write := PendingWrite0} = ConnectionStates) -> - CurrentRead = CurrentRead0#connection_state{server_verify_data = Data}, - PendingWrite = PendingWrite0#connection_state{server_verify_data = Data}, - ConnectionStates#connection_states{current_read = CurrentRead, - pending_write = PendingWrite}; + CurrentRead = CurrentRead0#{server_verify_data => Data}, + PendingWrite = PendingWrite0#{server_verify_data => Data}, + ConnectionStates#{current_read => CurrentRead, + pending_write => PendingWrite}; set_server_verify_data(current_both, Data, - #connection_states{current_read = CurrentRead0, - current_write = CurrentWrite0} + #{current_read := CurrentRead0, + current_write := CurrentWrite0} = ConnectionStates) -> - CurrentRead = CurrentRead0#connection_state{server_verify_data = Data}, - CurrentWrite = CurrentWrite0#connection_state{server_verify_data = Data}, - ConnectionStates#connection_states{current_read = CurrentRead, - current_write = CurrentWrite}. + CurrentRead = CurrentRead0#{server_verify_data => Data}, + CurrentWrite = CurrentWrite0#{server_verify_data => Data}, + ConnectionStates#{current_read => CurrentRead, + current_write => CurrentWrite}. %%-------------------------------------------------------------------- --spec set_pending_cipher_state(#connection_states{}, #cipher_state{}, +-spec set_pending_cipher_state(connection_states(), #cipher_state{}, #cipher_state{}, client | server) -> - #connection_states{}. + connection_states(). %% %% Description: Set the cipher state in the specified pending connection state. %%-------------------------------------------------------------------- -set_pending_cipher_state(#connection_states{pending_read = Read, - pending_write = Write} = States, +set_pending_cipher_state(#{pending_read := Read, + pending_write := Write} = States, ClientState, ServerState, server) -> - States#connection_states{ - pending_read = Read#connection_state{cipher_state = ClientState}, - pending_write = Write#connection_state{cipher_state = ServerState}}; + States#{ + pending_read => Read#{cipher_state => ClientState}, + pending_write => Write#{cipher_state => ServerState}}; -set_pending_cipher_state(#connection_states{pending_read = Read, - pending_write = Write} = States, +set_pending_cipher_state(#{pending_read := Read, + pending_write := Write} = States, ClientState, ServerState, client) -> - States#connection_states{ - pending_read = Read#connection_state{cipher_state = ServerState}, - pending_write = Write#connection_state{cipher_state = ClientState}}. + States#{ + pending_read => Read#{cipher_state => ServerState}, + pending_write => Write#{cipher_state => ClientState}}. %%-------------------------------------------------------------------- --spec encode_handshake(iolist(), ssl_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode_handshake(iolist(), ssl_version(), connection_states()) -> + {iolist(), connection_states()}. %% %% Description: Encodes a handshake message to send on the ssl-socket. %%-------------------------------------------------------------------- encode_handshake(Frag, Version, - #connection_states{current_write = - #connection_state{ - beast_mitigation = BeastMitigation, - security_parameters = - #security_parameters{bulk_cipher_algorithm = BCA}}} = + #{current_write := + #{beast_mitigation := BeastMitigation, + security_parameters := + #security_parameters{bulk_cipher_algorithm = BCA}}} = ConnectionStates) -when is_list(Frag) -> + when is_list(Frag) -> case iolist_size(Frag) of N when N > ?MAX_PLAIN_TEXT_LENGTH -> Data = split_bin(iolist_to_binary(Frag), ?MAX_PLAIN_TEXT_LENGTH, Version, BCA, BeastMitigation), @@ -341,8 +304,8 @@ encode_handshake(Frag, Version, ConnectionStates) -> encode_plain_text(?HANDSHAKE, Version, Frag, ConnectionStates). %%-------------------------------------------------------------------- --spec encode_alert_record(#alert{}, ssl_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode_alert_record(#alert{}, ssl_version(), connection_states()) -> + {iolist(), connection_states()}. %% %% Description: Encodes an alert message to send on the ssl-socket. %%-------------------------------------------------------------------- @@ -352,8 +315,8 @@ encode_alert_record(#alert{level = Level, description = Description}, ConnectionStates). %%-------------------------------------------------------------------- --spec encode_change_cipher_spec(ssl_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode_change_cipher_spec(ssl_version(), connection_states()) -> + {iolist(), connection_states()}. %% %% Description: Encodes a change_cipher_spec-message to send on the ssl socket. %%-------------------------------------------------------------------- @@ -361,15 +324,14 @@ encode_change_cipher_spec(Version, ConnectionStates) -> encode_plain_text(?CHANGE_CIPHER_SPEC, Version, <<1:8>>, ConnectionStates). %%-------------------------------------------------------------------- --spec encode_data(binary(), ssl_version(), #connection_states{}) -> - {iolist(), #connection_states{}}. +-spec encode_data(binary(), ssl_version(), connection_states()) -> + {iolist(), connection_states()}. %% %% Description: Encodes data to send on the ssl-socket. %%-------------------------------------------------------------------- encode_data(Frag, Version, - #connection_states{current_write = #connection_state{ - beast_mitigation = BeastMitigation, - security_parameters = + #{current_write := #{beast_mitigation := BeastMitigation, + security_parameters := #security_parameters{bulk_cipher_algorithm = BCA}}} = ConnectionStates) -> Data = split_bin(Frag, ?MAX_PLAIN_TEXT_LENGTH, Version, BCA, BeastMitigation), @@ -390,73 +352,74 @@ compressions() -> [?byte(?NULL)]. %%-------------------------------------------------------------------- --spec cipher(ssl_version(), iodata(), #connection_state{}, MacHash::binary()) -> - {CipherFragment::binary(), #connection_state{}}. +-spec cipher(ssl_version(), iodata(), connection_state(), MacHash::binary()) -> + {CipherFragment::binary(), connection_state()}. %% %% Description: Payload encryption %%-------------------------------------------------------------------- cipher(Version, Fragment, - #connection_state{cipher_state = CipherS0, - security_parameters= - #security_parameters{bulk_cipher_algorithm = - BulkCipherAlgo} - } = WriteState0, MacHash) -> - + #{cipher_state := CipherS0, + security_parameters := + #security_parameters{bulk_cipher_algorithm = + BulkCipherAlgo} + } = WriteState0, MacHash) -> + {CipherFragment, CipherS1} = ssl_cipher:cipher(BulkCipherAlgo, CipherS0, MacHash, Fragment, Version), - {CipherFragment, WriteState0#connection_state{cipher_state = CipherS1}}. + {CipherFragment, WriteState0#{cipher_state => CipherS1}}. %%-------------------------------------------------------------------- --spec cipher_aead(ssl_version(), iodata(), #connection_state{}, MacHash::binary()) -> - {CipherFragment::binary(), #connection_state{}}. +-spec cipher_aead(ssl_version(), iodata(), connection_state(), MacHash::binary()) -> + {CipherFragment::binary(), connection_state()}. %% %% Description: Payload encryption %%-------------------------------------------------------------------- cipher_aead(Version, Fragment, - #connection_state{cipher_state = CipherS0, - sequence_number = SeqNo, - security_parameters= - #security_parameters{bulk_cipher_algorithm = - BulkCipherAlgo} - } = WriteState0, AAD) -> - + #{cipher_state := CipherS0, + sequence_number := SeqNo, + security_parameters := + #security_parameters{bulk_cipher_algorithm = + BulkCipherAlgo} + } = WriteState0, AAD) -> + {CipherFragment, CipherS1} = ssl_cipher:cipher_aead(BulkCipherAlgo, CipherS0, SeqNo, AAD, Fragment, Version), - {CipherFragment, WriteState0#connection_state{cipher_state = CipherS1}}. + {CipherFragment, WriteState0#{cipher_state => CipherS1}}. %%-------------------------------------------------------------------- --spec decipher(ssl_version(), binary(), #connection_state{}, boolean()) -> {binary(), binary(), #connection_state{}} | #alert{}. +-spec decipher(ssl_version(), binary(), connection_state(), boolean()) -> {binary(), binary(), connection_state} | #alert{}. %% %% Description: Payload decryption %%-------------------------------------------------------------------- decipher(Version, CipherFragment, - #connection_state{security_parameters = - #security_parameters{bulk_cipher_algorithm = - BulkCipherAlgo, - hash_size = HashSz}, - cipher_state = CipherS0 - } = ReadState, PaddingCheck) -> + #{security_parameters := + #security_parameters{bulk_cipher_algorithm = + BulkCipherAlgo, + hash_size = HashSz}, + cipher_state := CipherS0 + } = ReadState, PaddingCheck) -> case ssl_cipher:decipher(BulkCipherAlgo, HashSz, CipherS0, CipherFragment, Version, PaddingCheck) of {PlainFragment, Mac, CipherS1} -> - CS1 = ReadState#connection_state{cipher_state = CipherS1}, + CS1 = ReadState#{cipher_state => CipherS1}, {PlainFragment, Mac, CS1}; #alert{} = Alert -> Alert end. %%-------------------------------------------------------------------- --spec decipher_aead(ssl_version(), binary(), #connection_state{}, binary()) -> {binary(), binary(), #connection_state{}} | #alert{}. +-spec decipher_aead(ssl_version(), binary(), connection_state(), binary()) -> + {binary(), binary(), connection_state()} | #alert{}. %% %% Description: Payload decryption %%-------------------------------------------------------------------- decipher_aead(Version, CipherFragment, - #connection_state{sequence_number = SeqNo, - security_parameters = - #security_parameters{bulk_cipher_algorithm = - BulkCipherAlgo}, - cipher_state = CipherS0 - } = ReadState, AAD) -> + #{sequence_number := SeqNo, + security_parameters := + #security_parameters{bulk_cipher_algorithm = + BulkCipherAlgo}, + cipher_state := CipherS0 + } = ReadState, AAD) -> case ssl_cipher:decipher_aead(BulkCipherAlgo, CipherS0, SeqNo, AAD, CipherFragment, Version) of {PlainFragment, CipherS1} -> - CS1 = ReadState#connection_state{cipher_state = CipherS1}, + CS1 = ReadState#{cipher_state => CipherS1}, {PlainFragment, CS1}; #alert{} = Alert -> Alert @@ -466,8 +429,15 @@ decipher_aead(Version, CipherFragment, %%-------------------------------------------------------------------- empty_connection_state(ConnectionEnd, BeastMitigation) -> SecParams = empty_security_params(ConnectionEnd), - #connection_state{security_parameters = SecParams, - beast_mitigation = BeastMitigation}. + #{security_parameters => SecParams, + beast_mitigation => BeastMitigation, + compression_state => undefined, + cipher_state => undefined, + mac_secret => undefined, + secure_renegotiation => undefined, + client_verify_data => undefined, + server_verify_data => undefined + }. empty_security_params(ConnectionEnd = ?CLIENT) -> #security_parameters{connection_end = ConnectionEnd, @@ -481,10 +451,10 @@ random() -> Random_28_bytes = ssl_cipher:random_bytes(28), <<?UINT32(Secs_since_1970), Random_28_bytes/binary>>. -dtls_next_epoch(#connection_state{epoch = undefined}) -> %% SSL/TLS - undefined; -dtls_next_epoch(#connection_state{epoch = Epoch}) -> %% DTLS - Epoch + 1. +%% dtls_next_epoch(#connection_state{epoch = undefined}) -> %% SSL/TLS +%% undefined; +%% dtls_next_epoch(#connection_state{epoch = Epoch}) -> %% DTLS +%% Epoch + 1. is_correct_mac(Mac, Mac) -> true; @@ -497,11 +467,17 @@ record_protocol_role(server) -> ?SERVER. initial_connection_state(ConnectionEnd, BeastMitigation) -> - #connection_state{security_parameters = - initial_security_params(ConnectionEnd), - sequence_number = 0, - beast_mitigation = BeastMitigation - }. + #{security_parameters => + initial_security_params(ConnectionEnd), + sequence_number => 0, + beast_mitigation => BeastMitigation, + compression_state => undefined, + cipher_state => undefined, + mac_secret => undefined, + secure_renegotiation => undefined, + client_verify_data => undefined, + server_verify_data => undefined + }. initial_security_params(ConnectionEnd) -> SecParams = #security_parameters{connection_end = ConnectionEnd, diff --git a/lib/ssl/src/ssl_record.hrl b/lib/ssl/src/ssl_record.hrl index a41264ff9b..ed007f58d7 100644 --- a/lib/ssl/src/ssl_record.hrl +++ b/lib/ssl/src/ssl_record.hrl @@ -30,29 +30,27 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Connection states - RFC 4346 section 6.1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% --record(connection_state, { - security_parameters, - compression_state, - cipher_state, - mac_secret, - epoch, %% Only used by DTLS - sequence_number, - %% RFC 5746 - secure_renegotiation, - client_verify_data, - server_verify_data, - %% How to do BEAST mitigation? - beast_mitigation - }). - --record(connection_states, { - dtls_write_msg_seq, %% Only used by DTLS +%% For documentation purposes are now maps in implementation +%% -record(connection_state, { +%% security_parameters, +%% compression_state, +%% cipher_state, +%% mac_secret, +%% sequence_number, +%% %% RFC 5746 +%% secure_renegotiation, +%% client_verify_data, +%% server_verify_data, +%% %% How to do BEAST mitigation? +%% beast_mitigation +%% }). - current_read, - pending_read, - current_write, - pending_write - }). +%% -record(connection_states, { +%% current_read, +%% pending_read, +%% current_write, +%% pending_write, +%% }). -record(security_parameters, { cipher_suite, diff --git a/lib/ssl/src/ssl_sup.erl b/lib/ssl/src/ssl_sup.erl index 7fa1f7dc9e..ba20f65f44 100644 --- a/lib/ssl/src/ssl_sup.erl +++ b/lib/ssl/src/ssl_sup.erl @@ -47,11 +47,13 @@ init([]) -> SessionCertManager = session_and_cert_manager_child_spec(), TLSConnetionManager = tls_connection_manager_child_spec(), %% Not supported yet - %%DTLSConnetionManager = tls_connection_manager_child_spec(), + %%DTLSConnetionManager = dtls_connection_manager_child_spec(), %% Handles emulated options so that they inherited by the accept socket, even when setopts is performed on %% the listen socket ListenOptionsTracker = listen_options_tracker_child_spec(), - {ok, {{one_for_all, 10, 3600}, [SessionCertManager, TLSConnetionManager, ListenOptionsTracker]}}. + {ok, {{one_for_all, 10, 3600}, [SessionCertManager, TLSConnetionManager, + %%DTLSConnetionManager, + ListenOptionsTracker]}}. manager_opts() -> @@ -93,15 +95,14 @@ tls_connection_manager_child_spec() -> {Name, StartFunc, Restart, Shutdown, Type, Modules}. %% dtls_connection_manager_child_spec() -> -%% Name = dtls_connection, +%% Name = dtls_connection, %% StartFunc = {dtls_connection_sup, start_link, []}, -%% Restart = permanent, +%% Restart = permanent, %% Shutdown = 4000, %% Modules = [dtls_connection, ssl_connection], %% Type = supervisor, %% {Name, StartFunc, Restart, Shutdown, Type, Modules}. - listen_options_tracker_child_spec() -> Name = ssl_socket, StartFunc = {ssl_listen_tracker_sup, start_link, []}, diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl index 8b828f3421..9b9031473a 100644 --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -51,17 +51,13 @@ %% Handshake handling -export([renegotiate/2, send_handshake/2, queue_handshake/2, queue_change_cipher/2, - reinit_handshake_data/1, handle_sni_extension/2]). + reinit_handshake_data/1, select_sni_extension/1]). %% Alert and close handling --export([send_alert/2, handle_own_alert/4, handle_close_alert/3, - handle_normal_shutdown/3, - close/5, alert_user/6, alert_user/9 - ]). +-export([send_alert/2, close/5]). %% Data handling --export([write_application_data/3, read_application_data/2, - passive_receive/2, next_record_if_active/1, handle_common_event/4]). +-export([passive_receive/2, next_record_if_active/1, handle_common_event/4]). %% gen_statem state functions -export([init/3, error/3, downgrade/3, %% Initiation and take down states @@ -107,9 +103,10 @@ send_handshake(Handshake, State) -> queue_handshake(Handshake, #state{negotiated_version = Version, tls_handshake_history = Hist0, flight_buffer = Flight0, + ssl_options = #ssl_options{v2_hello_compatible = V2HComp}, connection_states = ConnectionStates0} = State0) -> {BinHandshake, ConnectionStates, Hist} = - encode_handshake(Handshake, Version, ConnectionStates0, Hist0), + encode_handshake(Handshake, Version, ConnectionStates0, Hist0, V2HComp), State0#state{connection_states = ConnectionStates, tls_handshake_history = Hist, flight_buffer = Flight0 ++ [BinHandshake]}. @@ -147,6 +144,11 @@ reinit_handshake_data(State) -> tls_handshake_history = ssl_handshake:init_handshake_history() }. +select_sni_extension(#client_hello{extensions = HelloExtensions}) -> + HelloExtensions#hello_extensions.sni; +select_sni_extension(_) -> + undefined. + %%==================================================================== %% tls_connection_sup API %%==================================================================== @@ -186,7 +188,7 @@ callback_mode() -> init({call, From}, {start, Timeout}, #state{host = Host, port = Port, role = client, - ssl_options = SslOpts, + ssl_options = #ssl_options{v2_hello_compatible = V2HComp} = SslOpts, session = #session{own_certificate = Cert} = Session0, transport_cb = Transport, socket = Socket, connection_states = ConnectionStates0, @@ -202,7 +204,7 @@ init({call, From}, {start, Timeout}, HelloVersion = tls_record:lowest_protocol_version(SslOpts#ssl_options.versions), Handshake0 = ssl_handshake:init_handshake_history(), {BinMsg, ConnectionStates, Handshake} = - encode_handshake(Hello, HelloVersion, ConnectionStates0, Handshake0), + encode_handshake(Hello, HelloVersion, ConnectionStates0, Handshake0, V2HComp), Transport:send(Socket, BinMsg), State1 = State0#state{connection_states = ConnectionStates, negotiated_version = Version, %% Requested version @@ -250,7 +252,7 @@ hello(internal, #client_hello{client_version = ClientVersion, case tls_handshake:hello(Hello, SslOpts, {Port, Session0, Cache, CacheCb, ConnectionStates0, Cert, KeyExAlg}, Renegotiation) of #alert{} = Alert -> - handle_own_alert(Alert, ClientVersion, hello, State); + ssl_connection:handle_own_alert(Alert, ClientVersion, hello, State); {Version, {Type, Session}, ConnectionStates, Protocol0, ServerHelloExt, HashSign} -> Protocol = case Protocol0 of @@ -274,7 +276,7 @@ hello(internal, #server_hello{} = Hello, ssl_options = SslOptions} = State) -> case tls_handshake:hello(Hello, SslOptions, ConnectionStates0, Renegotiation) of #alert{} = Alert -> - handle_own_alert(Alert, ReqVersion, hello, State); + ssl_connection:handle_own_alert(Alert, ReqVersion, hello, State); {Version, NewId, ConnectionStates, ProtoExt, Protocol} -> ssl_connection:handle_session(Hello, Version, NewId, ConnectionStates, ProtoExt, Protocol, State) @@ -374,7 +376,7 @@ handle_info({Protocol, _, Data}, StateName, {Record, State} -> next_event(StateName, Record, State); #alert{} = Alert -> - handle_normal_shutdown(Alert, StateName, State0), + ssl_connection:handle_normal_shutdown(Alert, StateName, State0), {stop, {shutdown, own_alert}} end; handle_info({CloseTag, Socket}, StateName, @@ -394,14 +396,14 @@ handle_info({CloseTag, Socket}, StateName, %%invalidate_session(Role, Host, Port, Session) ok end, - handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), + ssl_connection:handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), {stop, {shutdown, transport_closed}}; handle_info(Msg, StateName, State) -> ssl_connection:handle_info(Msg, StateName, State). handle_common_event(internal, #alert{} = Alert, StateName, #state{negotiated_version = Version} = State) -> - handle_own_alert(Alert, Version, StateName, State); + ssl_connection:handle_own_alert(Alert, Version, StateName, State); %%% TLS record protocol level handshake messages handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = Data}, @@ -422,7 +424,7 @@ handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = Data}, {next_state, StateName, State, Events} end catch throw:#alert{} = Alert -> - handle_own_alert(Alert, Version, StateName, State0) + ssl_connection:handle_own_alert(Alert, Version, StateName, State0) end; %%% TLS record protocol level application data messages handle_common_event(internal, #ssl_tls{type = ?APPLICATION_DATA, fragment = Data}, StateName, State) -> @@ -437,12 +439,15 @@ handle_common_event(internal, #ssl_tls{type = ?ALERT, fragment = EncAlerts}, Sta Alerts = [_|_] -> handle_alerts(Alerts, {next_state, StateName, State}); [] -> - handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, empty_alert), Version, StateName, State); + ssl_connection:handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, empty_alert), + Version, StateName, State); #alert{} = Alert -> - handle_own_alert(Alert, Version, StateName, State) + ssl_connection:handle_own_alert(Alert, Version, StateName, State) catch _:_ -> - handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, alert_decode_error), Version, StateName, State) + ssl_connection:handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, alert_decode_error), + Version, StateName, State) + end; %% Ignore unknown TLS record level protocol messages handle_common_event(internal, #ssl_tls{type = _Unknown}, StateName, State) -> @@ -470,9 +475,9 @@ code_change(_OldVsn, StateName, State, _) -> %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- -encode_handshake(Handshake, Version, ConnectionStates0, Hist0) -> +encode_handshake(Handshake, Version, ConnectionStates0, Hist0, V2HComp) -> Frag = tls_handshake:encode_handshake(Handshake, Version), - Hist = ssl_handshake:update_handshake_history(Hist0, Frag), + Hist = ssl_handshake:update_handshake_history(Hist0, Frag, V2HComp), {Encoded, ConnectionStates} = ssl_record:encode_handshake(Frag, Version, ConnectionStates0), {Encoded, ConnectionStates, Hist}. @@ -486,7 +491,7 @@ decode_alerts(Bin) -> initial_state(Role, Host, Port, Socket, {SSLOptions, SocketOptions, Tracker}, User, {CbModule, DataTag, CloseTag, ErrorTag}) -> #ssl_options{beast_mitigation = BeastMitigation} = SSLOptions, - ConnectionStates = ssl_record:init_connection_states(Role, BeastMitigation), + ConnectionStates = tls_record:init_connection_states(Role, BeastMitigation), SessionCacheCb = case application:get_env(ssl, session_cb) of {ok, Cb} when is_atom(Cb) -> @@ -521,23 +526,6 @@ initial_state(Role, Host, Port, Socket, {SSLOptions, SocketOptions, Tracker}, Us flight_buffer = [] }. - -update_ssl_options_from_sni(OrigSSLOptions, SNIHostname) -> - SSLOption = - case OrigSSLOptions#ssl_options.sni_fun of - undefined -> - proplists:get_value(SNIHostname, - OrigSSLOptions#ssl_options.sni_hosts); - SNIFun -> - SNIFun(SNIHostname) - end, - case SSLOption of - undefined -> - undefined; - _ -> - ssl:handle_options(SSLOption, OrigSSLOptions) - end. - next_tls_record(Data, #state{protocol_buffers = #protocol_buffers{tls_record_buffer = Buf0, tls_cipher_texts = CT0} = Buffers} = State0) -> case tls_record:get_tls_records(Data, Buf0) of @@ -585,7 +573,7 @@ passive_receive(State0 = #state{user_data_buffer = Buffer}, StateName) -> {Record, State} = next_record(State0), next_event(StateName, Record, State); _ -> - {Record, State} = read_application_data(<<>>, State0), + {Record, State} = ssl_connection:read_application_data(<<>>, State0), next_event(StateName, Record, State) end. @@ -597,7 +585,7 @@ next_event(connection = StateName, no_record, State0, Actions) -> {no_record, State} -> ssl_connection:hibernate_after(StateName, State, Actions); {#ssl_tls{} = Record, State} -> - {next_state, StateName, State, [{next_event, internal, {tls_record, Record}} | Actions]}; + {next_state, StateName, State, [{next_event, internal, {protocol_record, Record}} | Actions]}; {#alert{} = Alert, State} -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end; @@ -606,169 +594,11 @@ next_event(StateName, Record, State, Actions) -> no_record -> {next_state, StateName, State, Actions}; #ssl_tls{} = Record -> - {next_state, StateName, State, [{next_event, internal, {tls_record, Record}} | Actions]}; + {next_state, StateName, State, [{next_event, internal, {protocol_record, Record}} | Actions]}; #alert{} = Alert -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end. -read_application_data(Data, #state{user_application = {_Mon, Pid}, - socket = Socket, - transport_cb = Transport, - socket_options = SOpts, - bytes_to_read = BytesToRead, - start_or_recv_from = RecvFrom, - timer = Timer, - user_data_buffer = Buffer0, - tracker = Tracker} = State0) -> - Buffer1 = if - Buffer0 =:= <<>> -> Data; - Data =:= <<>> -> Buffer0; - true -> <<Buffer0/binary, Data/binary>> - end, - case get_data(SOpts, BytesToRead, Buffer1) of - {ok, ClientData, Buffer} -> % Send data - SocketOpt = deliver_app_data(Transport, Socket, SOpts, ClientData, Pid, RecvFrom, Tracker), - cancel_timer(Timer), - State = State0#state{user_data_buffer = Buffer, - start_or_recv_from = undefined, - timer = undefined, - bytes_to_read = undefined, - socket_options = SocketOpt - }, - if - SocketOpt#socket_options.active =:= false; Buffer =:= <<>> -> - %% Passive mode, wait for active once or recv - %% Active and empty, get more data - next_record_if_active(State); - true -> %% We have more data - read_application_data(<<>>, State) - end; - {more, Buffer} -> % no reply, we need more data - next_record(State0#state{user_data_buffer = Buffer}); - {passive, Buffer} -> - next_record_if_active(State0#state{user_data_buffer = Buffer}); - {error,_Reason} -> %% Invalid packet in packet mode - deliver_packet_error(Transport, Socket, SOpts, Buffer1, Pid, RecvFrom, Tracker), - {stop, normal, State0} - end. - -%% Picks ClientData -get_data(_, _, <<>>) -> - {more, <<>>}; -%% Recv timed out save buffer data until next recv -get_data(#socket_options{active=false}, undefined, Buffer) -> - {passive, Buffer}; -get_data(#socket_options{active=Active, packet=Raw}, BytesToRead, Buffer) - when Raw =:= raw; Raw =:= 0 -> %% Raw Mode - if - Active =/= false orelse BytesToRead =:= 0 -> - %% Active true or once, or passive mode recv(0) - {ok, Buffer, <<>>}; - byte_size(Buffer) >= BytesToRead -> - %% Passive Mode, recv(Bytes) - <<Data:BytesToRead/binary, Rest/binary>> = Buffer, - {ok, Data, Rest}; - true -> - %% Passive Mode not enough data - {more, Buffer} - end; -get_data(#socket_options{packet=Type, packet_size=Size}, _, Buffer) -> - PacketOpts = [{packet_size, Size}], - case decode_packet(Type, Buffer, PacketOpts) of - {more, _} -> - {more, Buffer}; - Decoded -> - Decoded - end. - -decode_packet({http, headers}, Buffer, PacketOpts) -> - decode_packet(httph, Buffer, PacketOpts); -decode_packet({http_bin, headers}, Buffer, PacketOpts) -> - decode_packet(httph_bin, Buffer, PacketOpts); -decode_packet(Type, Buffer, PacketOpts) -> - erlang:decode_packet(Type, Buffer, PacketOpts). - -%% Just like with gen_tcp sockets, an ssl socket that has been configured with -%% {packet, http} (or {packet, http_bin}) will automatically switch to expect -%% HTTP headers after it sees a HTTP Request or HTTP Response line. We -%% represent the current state as follows: -%% #socket_options.packet =:= http: Expect a HTTP Request/Response line -%% #socket_options.packet =:= {http, headers}: Expect HTTP Headers -%% Note that if the user has explicitly configured the socket to expect -%% HTTP headers using the {packet, httph} option, we don't do any automatic -%% switching of states. -deliver_app_data(Transport, Socket, SOpts = #socket_options{active=Active, packet=Type}, - Data, Pid, From, Tracker) -> - send_or_reply(Active, Pid, From, format_reply(Transport, Socket, SOpts, Data, Tracker)), - SO = case Data of - {P, _, _, _} when ((P =:= http_request) or (P =:= http_response)), - ((Type =:= http) or (Type =:= http_bin)) -> - SOpts#socket_options{packet={Type, headers}}; - http_eoh when tuple_size(Type) =:= 2 -> - % End of headers - expect another Request/Response line - {Type1, headers} = Type, - SOpts#socket_options{packet=Type1}; - _ -> - SOpts - end, - case Active of - once -> - SO#socket_options{active=false}; - _ -> - SO - end. - -format_reply(_, _,#socket_options{active = false, mode = Mode, packet = Packet, - header = Header}, Data, _) -> - {ok, do_format_reply(Mode, Packet, Header, Data)}; -format_reply(Transport, Socket, #socket_options{active = _, mode = Mode, packet = Packet, - header = Header}, Data, Tracker) -> - {ssl, ssl_socket:socket(self(), Transport, Socket, ?MODULE, Tracker), - do_format_reply(Mode, Packet, Header, Data)}. - -deliver_packet_error(Transport, Socket, SO= #socket_options{active = Active}, Data, Pid, From, Tracker) -> - send_or_reply(Active, Pid, From, format_packet_error(Transport, Socket, SO, Data, Tracker)). - -format_packet_error(_, _,#socket_options{active = false, mode = Mode}, Data, _) -> - {error, {invalid_packet, do_format_reply(Mode, raw, 0, Data)}}; -format_packet_error(Transport, Socket, #socket_options{active = _, mode = Mode}, Data, Tracker) -> - {ssl_error, ssl_socket:socket(self(), Transport, Socket, ?MODULE, Tracker), - {invalid_packet, do_format_reply(Mode, raw, 0, Data)}}. - -do_format_reply(binary, _, N, Data) when N > 0 -> % Header mode - header(N, Data); -do_format_reply(binary, _, _, Data) -> - Data; -do_format_reply(list, Packet, _, Data) - when Packet == http; Packet == {http, headers}; - Packet == http_bin; Packet == {http_bin, headers}; - Packet == httph; Packet == httph_bin -> - Data; -do_format_reply(list, _,_, Data) -> - binary_to_list(Data). - -header(0, <<>>) -> - <<>>; -header(_, <<>>) -> - []; -header(0, Binary) -> - Binary; -header(N, Binary) -> - <<?BYTE(ByteN), NewBinary/binary>> = Binary, - [ByteN | header(N-1, NewBinary)]. - -send_or_reply(false, _Pid, From, Data) when From =/= undefined -> - gen_statem:reply(From, Data); -%% Can happen when handling own alert or tcp error/close and there is -%% no outstanding gen_fsm sync events -send_or_reply(false, no_pid, _, _) -> - ok; -send_or_reply(_, Pid, _From, Data) -> - send_user(Pid, Data). - -send_user(Pid, Msg) -> - Pid ! Msg. - tls_handshake_events([]) -> throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); tls_handshake_events(Packets) -> @@ -776,55 +606,7 @@ tls_handshake_events(Packets) -> {next_event, internal, {handshake, Packet}} end, Packets). -write_application_data(Data0, From, - #state{socket = Socket, - negotiated_version = Version, - transport_cb = Transport, - connection_states = ConnectionStates0, - socket_options = SockOpts, - ssl_options = #ssl_options{renegotiate_at = RenegotiateAt}} = State) -> - Data = encode_packet(Data0, SockOpts), - - case time_to_renegotiate(Data, ConnectionStates0, RenegotiateAt) of - true -> - renegotiate(State#state{renegotiation = {true, internal}}, - [{next_event, {call, From}, {application_data, Data0}}]); - false -> - {Msgs, ConnectionStates} = ssl_record:encode_data(Data, Version, ConnectionStates0), - Result = Transport:send(Socket, Msgs), - ssl_connection:hibernate_after(connection, State#state{connection_states = ConnectionStates}, - [{reply, From, Result}]) - end. - -encode_packet(Data, #socket_options{packet=Packet}) -> - case Packet of - 1 -> encode_size_packet(Data, 8, (1 bsl 8) - 1); - 2 -> encode_size_packet(Data, 16, (1 bsl 16) - 1); - 4 -> encode_size_packet(Data, 32, (1 bsl 32) - 1); - _ -> Data - end. - -encode_size_packet(Bin, Size, Max) -> - Len = erlang:byte_size(Bin), - case Len > Max of - true -> throw({error, {badarg, {packet_to_large, Len, Max}}}); - false -> <<Len:Size, Bin/binary>> - end. -time_to_renegotiate(_Data, - #connection_states{current_write = - #connection_state{sequence_number = Num}}, - RenegotiateAt) -> - - %% We could do test: - %% is_time_to_renegotiate((erlang:byte_size(_Data) div ?MAX_PLAIN_TEXT_LENGTH) + 1, RenegotiateAt), - %% but we chose to have a some what lower renegotiateAt and a much cheaper test - is_time_to_renegotiate(Num, RenegotiateAt). - -is_time_to_renegotiate(N, M) when N < M-> - false; -is_time_to_renegotiate(_,_) -> - true. renegotiate(#state{role = client} = State, Actions) -> %% Handle same way as if server requested %% the renegotiation @@ -854,131 +636,10 @@ handle_alerts([], Result) -> handle_alerts(_, {stop,_} = Stop) -> Stop; handle_alerts([Alert | Alerts], {next_state, StateName, State}) -> - handle_alerts(Alerts, handle_alert(Alert, StateName, State)); + handle_alerts(Alerts, ssl_connection:handle_alert(Alert, StateName, State)); handle_alerts([Alert | Alerts], {next_state, StateName, State, _Actions}) -> - handle_alerts(Alerts, handle_alert(Alert, StateName, State)). -handle_alert(#alert{level = ?FATAL} = Alert, StateName, - #state{socket = Socket, transport_cb = Transport, - ssl_options = SslOpts, start_or_recv_from = From, host = Host, - port = Port, session = Session, user_application = {_Mon, Pid}, - role = Role, socket_options = Opts, tracker = Tracker}) -> - invalidate_session(Role, Host, Port, Session), - log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), - alert_user(Transport, Tracker, Socket, StateName, Opts, Pid, From, Alert, Role), - {stop, normal}; - -handle_alert(#alert{level = ?WARNING, description = ?CLOSE_NOTIFY} = Alert, - StateName, State) -> - handle_normal_shutdown(Alert, StateName, State), - {stop, {shutdown, peer_close}}; - -handle_alert(#alert{level = ?WARNING, description = ?NO_RENEGOTIATION} = Alert, StateName, - #state{ssl_options = SslOpts, renegotiation = {true, internal}} = State) -> - log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), - handle_normal_shutdown(Alert, StateName, State), - {stop, {shutdown, peer_close}}; - -handle_alert(#alert{level = ?WARNING, description = ?NO_RENEGOTIATION} = Alert, StateName, - #state{ssl_options = SslOpts, renegotiation = {true, From}} = State0) -> - log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), - gen_statem:reply(From, {error, renegotiation_rejected}), - {Record, State} = next_record(State0), - %% Go back to connection! - next_event(connection, Record, State); - -%% Gracefully log and ignore all other warning alerts -handle_alert(#alert{level = ?WARNING} = Alert, StateName, - #state{ssl_options = SslOpts} = State0) -> - log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), - {Record, State} = next_record(State0), - next_event(StateName, Record, State). + handle_alerts(Alerts, ssl_connection:handle_alert(Alert, StateName, State)). -alert_user(Transport, Tracker, Socket, connection, Opts, Pid, From, Alert, Role) -> - alert_user(Transport, Tracker, Socket, Opts#socket_options.active, Pid, From, Alert, Role); -alert_user(Transport, Tracker, Socket,_, _, _, From, Alert, Role) -> - alert_user(Transport, Tracker, Socket, From, Alert, Role). - -alert_user(Transport, Tracker, Socket, From, Alert, Role) -> - alert_user(Transport, Tracker, Socket, false, no_pid, From, Alert, Role). - -alert_user(_, _, _, false = Active, Pid, From, Alert, Role) when From =/= undefined -> - %% If there is an outstanding ssl_accept | recv - %% From will be defined and send_or_reply will - %% send the appropriate error message. - ReasonCode = ssl_alert:reason_code(Alert, Role), - send_or_reply(Active, Pid, From, {error, ReasonCode}); -alert_user(Transport, Tracker, Socket, Active, Pid, From, Alert, Role) -> - case ssl_alert:reason_code(Alert, Role) of - closed -> - send_or_reply(Active, Pid, From, - {ssl_closed, ssl_socket:socket(self(), - Transport, Socket, ?MODULE, Tracker)}); - ReasonCode -> - send_or_reply(Active, Pid, From, - {ssl_error, ssl_socket:socket(self(), - Transport, Socket, ?MODULE, Tracker), ReasonCode}) - end. - -log_alert(true, Info, Alert) -> - Txt = ssl_alert:alert_txt(Alert), - error_logger:format("SSL: ~p: ~s\n", [Info, Txt]); -log_alert(false, _, _) -> - ok. - -handle_own_alert(Alert, Version, StateName, - #state{transport_cb = Transport, - socket = Socket, - connection_states = ConnectionStates, - ssl_options = SslOpts} = State) -> - try %% Try to tell the other side - {BinMsg, _} = - ssl_alert:encode(Alert, Version, ConnectionStates), - Transport:send(Socket, BinMsg) - catch _:_ -> %% Can crash if we are in a uninitialized state - ignore - end, - try %% Try to tell the local user - log_alert(SslOpts#ssl_options.log_alert, StateName, Alert), - handle_normal_shutdown(Alert,StateName, State) - catch _:_ -> - ok - end, - {stop, {shutdown, own_alert}}. - -handle_normal_shutdown(Alert, _, #state{socket = Socket, - transport_cb = Transport, - start_or_recv_from = StartFrom, - tracker = Tracker, - role = Role, renegotiation = {false, first}}) -> - alert_user(Transport, Tracker,Socket, StartFrom, Alert, Role); - -handle_normal_shutdown(Alert, StateName, #state{socket = Socket, - socket_options = Opts, - transport_cb = Transport, - user_application = {_Mon, Pid}, - tracker = Tracker, - start_or_recv_from = RecvFrom, role = Role}) -> - alert_user(Transport, Tracker, Socket, StateName, Opts, Pid, RecvFrom, Alert, Role). - -handle_close_alert(Data, StateName, State0) -> - case next_tls_record(Data, State0) of - {#ssl_tls{type = ?ALERT, fragment = EncAlerts}, State} -> - [Alert|_] = decode_alerts(EncAlerts), - handle_normal_shutdown(Alert, StateName, State); - _ -> - ok - end. - -cancel_timer(undefined) -> - ok; -cancel_timer(Timer) -> - erlang:cancel_timer(Timer), - ok. - -invalidate_session(client, Host, Port, Session) -> - ssl_manager:invalidate_session(Host, Port, Session); -invalidate_session(server, _, Port, Session) -> - ssl_manager:invalidate_session(Port, Session). %% User closes or recursive call! close({close, Timeout}, Socket, Transport = gen_tcp, _,_) -> @@ -1017,42 +678,16 @@ convert_options_partial_chain(Options, up) -> convert_options_partial_chain(Options, down) -> list_to_tuple(proplists:delete(partial_chain, tuple_to_list(Options))). -handle_sni_extension(#client_hello{extensions = HelloExtensions}, State0) -> - case HelloExtensions#hello_extensions.sni of - undefined -> - State0; - #sni{hostname = Hostname} -> - NewOptions = update_ssl_options_from_sni(State0#state.ssl_options, Hostname), - case NewOptions of - undefined -> - State0; - _ -> - {ok, Ref, CertDbHandle, FileRefHandle, CacheHandle, CRLDbHandle, OwnCert, Key, DHParams} = - ssl_config:init(NewOptions, State0#state.role), - State0#state{ - session = State0#state.session#session{own_certificate = OwnCert}, - file_ref_db = FileRefHandle, - cert_db_ref = Ref, - cert_db = CertDbHandle, - crl_db = CRLDbHandle, - session_cache = CacheHandle, - private_key = Key, - diffie_hellman_params = DHParams, - ssl_options = NewOptions, - sni_hostname = Hostname - } - end - end; -handle_sni_extension(_, State) -> - State. - -gen_handshake(GenConnection, StateName, Type, Event, #state{negotiated_version = Version} = State) -> +gen_handshake(GenConnection, StateName, Type, Event, + #state{negotiated_version = Version} = State) -> try GenConnection:StateName(Type, Event, State, ?MODULE) of Result -> Result catch _:_ -> - handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake_data), Version, StateName, State) + ssl_connection:handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, + malformed_handshake_data), + Version, StateName, State) end. gen_info(Event, connection = StateName, #state{negotiated_version = Version} = State) -> @@ -1061,7 +696,9 @@ gen_info(Event, connection = StateName, #state{negotiated_version = Version} = Result catch _:_ -> - handle_own_alert(?ALERT_REC(?FATAL, ?INTERNAL_ERROR, malformed_data), Version, StateName, State) + ssl_connection:handle_own_alert(?ALERT_REC(?FATAL, ?INTERNAL_ERROR, + malformed_data), + Version, StateName, State) end; gen_info(Event, StateName, #state{negotiated_version = Version} = State) -> @@ -1070,6 +707,8 @@ gen_info(Event, StateName, #state{negotiated_version = Version} = State) -> Result catch _:_ -> - handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake_data), Version, StateName, State) + ssl_connection:handle_own_alert(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, + malformed_handshake_data), + Version, StateName, State) end. diff --git a/lib/ssl/src/tls_handshake.erl b/lib/ssl/src/tls_handshake.erl index 6e593950d9..a2486bf752 100644 --- a/lib/ssl/src/tls_handshake.erl +++ b/lib/ssl/src/tls_handshake.erl @@ -41,7 +41,7 @@ %% Internal application API %%==================================================================== %%-------------------------------------------------------------------- --spec client_hello(host(), inet:port_number(), #connection_states{}, +-spec client_hello(host(), inet:port_number(), ssl_record:connection_states(), #ssl_options{}, integer(), atom(), boolean(), der_cert()) -> #client_hello{}. %% @@ -54,8 +54,7 @@ client_hello(Host, Port, ConnectionStates, } = SslOpts, Cache, CacheCb, Renegotiation, OwnCert) -> Version = tls_record:highest_protocol_version(Versions), - Pending = ssl_record:pending_connection_state(ConnectionStates, read), - SecParams = Pending#connection_state.security_parameters, + #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates, read), AvailableCipherSuites = ssl_handshake:available_suites(UserSuites, Version), Extensions = ssl_handshake:client_hello_extensions(Host, Version, AvailableCipherSuites, @@ -78,14 +77,14 @@ client_hello(Host, Port, ConnectionStates, %%-------------------------------------------------------------------- -spec hello(#server_hello{} | #client_hello{}, #ssl_options{}, - #connection_states{} | {inet:port_number(), #session{}, db_handle(), - atom(), #connection_states{}, + ssl_record:connection_states() | {inet:port_number(), #session{}, db_handle(), + atom(), ssl_record:connection_states(), binary() | undefined, ssl_cipher:key_algo()}, boolean()) -> {tls_record:tls_version(), session_id(), - #connection_states{}, alpn | npn, binary() | undefined}| + ssl_record:connection_states(), alpn | npn, binary() | undefined}| {tls_record:tls_version(), {resumed | new, #session{}}, - #connection_states{}, binary() | undefined, + ssl_record:connection_states(), binary() | undefined, #hello_extensions{}, {ssl_cipher:hash(), ssl_cipher:sign_algo()} | undefined} | #alert{}. %% diff --git a/lib/ssl/src/tls_record.erl b/lib/ssl/src/tls_record.erl index 9348c8bbdd..5331dd1303 100644 --- a/lib/ssl/src/tls_record.erl +++ b/lib/ssl/src/tls_record.erl @@ -32,7 +32,7 @@ -include("ssl_cipher.hrl"). %% Handling of incoming data --export([get_tls_records/2]). +-export([get_tls_records/2, init_connection_states/2]). %% Decoding -export([decode_cipher_text/3]). @@ -56,12 +56,28 @@ %%==================================================================== %% Internal application API %%==================================================================== +%%-------------------------------------------------------------------- +-spec init_connection_states(client | server, one_n_minus_one | zero_n | disabled) -> + ssl_record:connection_states(). +%% % + % +%% Description: Creates a connection_states record with appropriate +%% values for the initial SSL connection setup. +%%-------------------------------------------------------------------- +init_connection_states(Role, BeastMitigation) -> + ConnectionEnd = ssl_record:record_protocol_role(Role), + Current = initial_connection_state(ConnectionEnd, BeastMitigation), + Pending = ssl_record:empty_connection_state(ConnectionEnd, BeastMitigation), + #{current_read => Current, + pending_read => Pending, + current_write => Current, + pending_write => Pending}. %%-------------------------------------------------------------------- -spec get_tls_records(binary(), binary()) -> {[binary()], binary()} | #alert{}. %% -%% Description: Given old buffer and new data from TCP, packs up a records %% and returns it as a list of tls_compressed binaries also returns leftover +%% Description: Given old buffer and new data from TCP, packs up a records %% data %%-------------------------------------------------------------------- get_tls_records(Data, <<>>) -> @@ -129,63 +145,61 @@ get_tls_records_aux(Data, Acc) -> end. encode_plain_text(Type, Version, Data, - #connection_states{current_write = - #connection_state{ - sequence_number = Seq, - compression_state=CompS0, - security_parameters= - #security_parameters{ - cipher_type = ?AEAD, - compression_algorithm=CompAlg} - }= WriteState0} = ConnectionStates) -> + #{current_write := + #{sequence_number := Seq, + compression_state := CompS0, + security_parameters := + #security_parameters{ + cipher_type = ?AEAD, + compression_algorithm = CompAlg} + }= WriteState0} = ConnectionStates) -> {Comp, CompS1} = ssl_record:compress(CompAlg, Data, CompS0), - WriteState1 = WriteState0#connection_state{compression_state = CompS1}, + WriteState1 = WriteState0#{compression_state => CompS1}, AAD = calc_aad(Type, Version, WriteState1), {CipherFragment, WriteState} = ssl_record:cipher_aead(Version, Comp, WriteState1, AAD), CipherText = encode_tls_cipher_text(Type, Version, CipherFragment), - {CipherText, ConnectionStates#connection_states{current_write = WriteState#connection_state{sequence_number = Seq +1}}}; + {CipherText, ConnectionStates#{current_write => WriteState#{sequence_number => Seq +1}}}; encode_plain_text(Type, Version, Data, - #connection_states{current_write = - #connection_state{ - sequence_number = Seq, - compression_state=CompS0, - security_parameters= - #security_parameters{compression_algorithm=CompAlg} - }= WriteState0} = ConnectionStates) -> + #{current_write := + #{sequence_number := Seq, + compression_state := CompS0, + security_parameters := + #security_parameters{compression_algorithm = CompAlg} + }= WriteState0} = ConnectionStates) -> {Comp, CompS1} = ssl_record:compress(CompAlg, Data, CompS0), - WriteState1 = WriteState0#connection_state{compression_state = CompS1}, + WriteState1 = WriteState0#{compression_state => CompS1}, MacHash = calc_mac_hash(Type, Version, Comp, WriteState1), {CipherFragment, WriteState} = ssl_record:cipher(Version, Comp, WriteState1, MacHash), CipherText = encode_tls_cipher_text(Type, Version, CipherFragment), - {CipherText, ConnectionStates#connection_states{current_write = WriteState#connection_state{sequence_number = Seq +1}}}. + {CipherText, ConnectionStates#{current_write => WriteState#{sequence_number => Seq +1}}}; +encode_plain_text(_,_,_, CS) -> + exit({cs, CS}). %%-------------------------------------------------------------------- --spec decode_cipher_text(#ssl_tls{}, #connection_states{}, boolean()) -> - {#ssl_tls{}, #connection_states{}}| #alert{}. +-spec decode_cipher_text(#ssl_tls{}, ssl_record:connection_states(), boolean()) -> + {#ssl_tls{}, ssl_record:connection_states()}| #alert{}. %% %% Description: Decode cipher text %%-------------------------------------------------------------------- decode_cipher_text(#ssl_tls{type = Type, version = Version, fragment = CipherFragment} = CipherText, - #connection_states{current_read = - #connection_state{ - compression_state = CompressionS0, - sequence_number = Seq, - security_parameters= - #security_parameters{ - cipher_type = ?AEAD, - compression_algorithm=CompAlg} - } = ReadState0} = ConnnectionStates0, _) -> + #{current_read := + #{compression_state := CompressionS0, + sequence_number := Seq, + security_parameters := + #security_parameters{ + cipher_type = ?AEAD, + compression_algorithm = CompAlg} + } = ReadState0} = ConnnectionStates0, _) -> AAD = calc_aad(Type, Version, ReadState0), case ssl_record:decipher_aead(Version, CipherFragment, ReadState0, AAD) of {PlainFragment, ReadState1} -> {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ConnnectionStates = ConnnectionStates0#connection_states{ - current_read = ReadState1#connection_state{ - sequence_number = Seq + 1, - compression_state = CompressionS1}}, + ConnnectionStates = ConnnectionStates0#{ + current_read => ReadState1#{sequence_number => Seq + 1, + compression_state => CompressionS1}}, {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; #alert{} = Alert -> Alert @@ -193,13 +207,12 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, decode_cipher_text(#ssl_tls{type = Type, version = Version, fragment = CipherFragment} = CipherText, - #connection_states{current_read = - #connection_state{ - compression_state = CompressionS0, - sequence_number = Seq, - security_parameters= - #security_parameters{compression_algorithm=CompAlg} - } = ReadState0} = ConnnectionStates0, PaddingCheck) -> + #{current_read := + #{compression_state := CompressionS0, + sequence_number := Seq, + security_parameters := + #security_parameters{compression_algorithm = CompAlg} + } = ReadState0} = ConnnectionStates0, PaddingCheck) -> case ssl_record:decipher(Version, CipherFragment, ReadState0, PaddingCheck) of {PlainFragment, Mac, ReadState1} -> MacHash = calc_mac_hash(Type, Version, PlainFragment, ReadState1), @@ -207,10 +220,10 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, true -> {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ConnnectionStates = ConnnectionStates0#connection_states{ - current_read = ReadState1#connection_state{ - sequence_number = Seq + 1, - compression_state = CompressionS1}}, + ConnnectionStates = ConnnectionStates0#{ + current_read => ReadState1#{ + sequence_number => Seq + 1, + compression_state => CompressionS1}}, {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; false -> ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC) @@ -375,6 +388,18 @@ is_acceptable_version(_,_) -> %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- +initial_connection_state(ConnectionEnd, BeastMitigation) -> + #{security_parameters => + ssl_record:initial_security_params(ConnectionEnd), + sequence_number => 0, + beast_mitigation => BeastMitigation, + compression_state => undefined, + cipher_state => undefined, + mac_secret => undefined, + secure_renegotiation => undefined, + client_verify_data => undefined, + server_verify_data => undefined + }. lowest_list_protocol_version(Ver, []) -> Ver; @@ -413,15 +438,15 @@ sufficient_tlsv1_2_crypto_support() -> proplists:get_bool(sha256, proplists:get_value(hashs, CryptoSupport)). calc_mac_hash(Type, Version, - PlainFragment, #connection_state{sequence_number = SeqNo, - mac_secret = MacSecret, - security_parameters = - SecPars}) -> + PlainFragment, #{sequence_number := SeqNo, + mac_secret := MacSecret, + security_parameters:= + SecPars}) -> Length = erlang:iolist_size(PlainFragment), mac_hash(Version, SecPars#security_parameters.mac_algorithm, MacSecret, SeqNo, Type, Length, PlainFragment). calc_aad(Type, {MajVer, MinVer}, - #connection_state{sequence_number = SeqNo}) -> + #{sequence_number := SeqNo}) -> <<SeqNo:64/integer, ?BYTE(Type), ?BYTE(MajVer), ?BYTE(MinVer)>>. diff --git a/lib/ssl/test/ssl.spec b/lib/ssl/test/ssl.spec index 86e14c033e..0ad94e22bc 100644 --- a/lib/ssl/test/ssl.spec +++ b/lib/ssl/test/ssl.spec @@ -1,4 +1,5 @@ {suites,"../ssl_test",all}. {skip_cases, "../ssl_test", - ssl_bench_SUITE, [setup_sequential, setup_concurrent, payload_simple], + ssl_bench_SUITE, [setup_sequential, setup_concurrent, payload_simple, + use_pem_cache, bypass_pem_cache], "Benchmarks run separately"}. diff --git a/lib/ssl/test/ssl_ECC_SUITE.erl b/lib/ssl/test/ssl_ECC_SUITE.erl index 69ac9908fa..258922d128 100644 --- a/lib/ssl/test/ssl_ECC_SUITE.erl +++ b/lib/ssl/test/ssl_ECC_SUITE.erl @@ -145,7 +145,7 @@ init_per_testcase(TestCase, Config) -> ssl_test_lib:ct_log_supported_protocol_versions(Config), ct:log("Ciphers: ~p~n ", [ ssl:cipher_suites()]), end_per_testcase(TestCase, Config), - ssl:start(), + ssl_test_lib:clean_start(), ct:timetrap({seconds, 15}), Config. diff --git a/lib/ssl/test/ssl_alpn_handshake_SUITE.erl b/lib/ssl/test/ssl_alpn_handshake_SUITE.erl index da181faf64..9d57e89b9b 100644 --- a/lib/ssl/test/ssl_alpn_handshake_SUITE.erl +++ b/lib/ssl/test/ssl_alpn_handshake_SUITE.erl @@ -71,7 +71,7 @@ init_per_suite(Config) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config), proplists:get_value(priv_dir, Config)), ssl_test_lib:cert_options(Config) diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index 38341f77aa..57963fd44b 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -40,6 +40,7 @@ -define(SLEEP, 500). -define(RENEGOTIATION_DISABLE_TIME, 12000). -define(CLEAN_SESSION_DB, 60000). +-define(SEC_RENEGOTIATION_TIMEOUT, 30). %%-------------------------------------------------------------------- %% Common Test interface functions ----------------------------------- @@ -249,7 +250,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), %% make rsa certs using oppenssl {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), @@ -306,6 +307,7 @@ init_per_testcase(protocol_versions, Config) -> init_per_testcase(reuse_session_expired, Config) -> ssl:stop(), application:load(ssl), + ssl_test_lib:clean_env(), application:set_env(ssl, session_lifetime, ?EXPIRE), application:set_env(ssl, session_delay_cleanup_time, 500), ssl:start(), @@ -315,6 +317,7 @@ init_per_testcase(reuse_session_expired, Config) -> init_per_testcase(empty_protocol_versions, Config) -> ssl:stop(), application:load(ssl), + ssl_test_lib:clean_env(), application:set_env(ssl, protocol_version, []), ssl:start(), ct:timetrap({seconds, 5}), @@ -340,7 +343,7 @@ init_per_testcase(TestCase, Config) when TestCase == client_renegotiate; TestCase == renegotiate_dos_mitigate_passive; TestCase == renegotiate_dos_mitigate_absolute -> ssl_test_lib:ct_log_supported_protocol_versions(Config), - ct:timetrap({seconds, 90}), + ct:timetrap({seconds, ?SEC_RENEGOTIATION_TIMEOUT + 5}), Config; init_per_testcase(TestCase, Config) when TestCase == psk_cipher_suites; @@ -440,7 +443,9 @@ init_per_testcase(accept_pool, Config) -> ssl_test_lib:ct_log_supported_protocol_versions(Config), Config end; - +init_per_testcase(controller_dies, Config) -> + ct:timetrap({seconds, 10}), + Config; init_per_testcase(_TestCase, Config) -> ssl_test_lib:ct_log_supported_protocol_versions(Config), ct:timetrap({seconds, 5}), @@ -451,6 +456,11 @@ end_per_testcase(reuse_session_expired, Config) -> application:unset_env(ssl, session_delay_cleanup_time), end_per_testcase(default_action, Config); +end_per_testcase(Case, Config) when Case == protocol_versions; + Case == empty_protocol_versions-> + application:unset_env(ssl, protocol_versions), + end_per_testcase(default_action, Config); + end_per_testcase(_TestCase, Config) -> Config. @@ -4298,7 +4308,7 @@ erlang_ssl_receive(Socket, Data) -> erlang_ssl_receive(Socket, tl(Data)); Other -> ct:fail({unexpected_message, Other}) - after ?SLEEP * 3 * test_server:timetrap_scale_factor() -> + after timer:seconds(?SEC_RENEGOTIATION_TIMEOUT) * test_server:timetrap_scale_factor() -> ct:fail({did_not_get, Data}) end. diff --git a/lib/ssl/test/ssl_bench_SUITE.erl b/lib/ssl/test/ssl_bench_SUITE.erl index ed439a425f..21989f8d99 100644 --- a/lib/ssl/test/ssl_bench_SUITE.erl +++ b/lib/ssl/test/ssl_bench_SUITE.erl @@ -25,11 +25,12 @@ suite() -> [{ct_hooks,[{ts_install_cth,[{nodenames,2}]}]}]. -all() -> [{group, setup}, {group, payload}]. +all() -> [{group, setup}, {group, payload}, {group, pem_cache}]. groups() -> [{setup, [{repeat, 3}], [setup_sequential, setup_concurrent]}, - {payload, [{repeat, 3}], [payload_simple]} + {payload, [{repeat, 3}], [payload_simple]}, + {pem_cache, [{repeat, 3}], [use_pem_cache, bypass_pem_cache]} ]. init_per_group(_GroupName, Config) -> @@ -49,9 +50,33 @@ init_per_suite(Config) -> end_per_suite(_Config) -> ok. +init_per_testcase(use_pem_cache, Conf) -> + case bypass_pem_cache_supported() of + false -> {skipped, "PEM cache bypass support required"}; + true -> + application:set_env(ssl, bypass_pem_cache, false), + Conf + end; +init_per_testcase(bypass_pem_cache, Conf) -> + case bypass_pem_cache_supported() of + false -> {skipped, "PEM cache bypass support required"}; + true -> + application:set_env(ssl, bypass_pem_cache, true), + Conf + end; init_per_testcase(_Func, Conf) -> Conf. +end_per_testcase(use_pem_cache, _Config) -> + case bypass_pem_cache_supported() of + false -> ok; + true -> application:set_env(ssl, bypass_pem_cache, false) + end; +end_per_testcase(bypass_pem_cache, _Config) -> + case bypass_pem_cache_supported() of + false -> ok; + true -> application:set_env(ssl, bypass_pem_cache, false) + end; end_per_testcase(_Func, _Conf) -> ok. @@ -94,6 +119,18 @@ payload_simple(Config) -> {suite, "ssl"}, {name, "Payload simple"}]}), ok. +use_pem_cache(_Config) -> + {ok, Result} = do_test(ssl, pem_cache, 100, 500, node()), + ct_event:notify(#event{name = benchmark_data, + data=[{value, Result}, + {suite, "ssl"}, {name, "Use PEM cache"}]}). + +bypass_pem_cache(_Config) -> + {ok, Result} = do_test(ssl, pem_cache, 100, 500, node()), + ct_event:notify(#event{name = benchmark_data, + data=[{value, Result}, + {suite, "ssl"}, {name, "Bypass PEM cache"}]}). + ssl() -> test(ssl, ?COUNT, node()). @@ -172,6 +209,18 @@ server_init(ssl, payload, Loop, _, Server) -> ssl:close(TSocket) end, setup_server_connection(Socket, Test); +server_init(ssl, pem_cache, Loop, _, Server) -> + {ok, Socket} = ssl:listen(0, ssl_opts(listen_der)), + {ok, {_Host, Port}} = ssl:sockname(Socket), + {ok, Host} = inet:gethostname(), + Server ! {self(), {init, Host, Port}}, + Test = fun(TSocket) -> + ok = ssl:ssl_accept(TSocket), + Size = byte_size(msg()), + server_echo(TSocket, Size, Loop), + ssl:close(TSocket) + end, + setup_server_connection(Socket, Test); server_init(Type, Tc, _, _, Server) -> io:format("No server init code for ~p ~p~n",[Type, Tc]), @@ -185,6 +234,11 @@ client_init(Master, ssl, payload, Host, Port) -> Master ! {self(), init}, Size = byte_size(msg()), {Sock, Size}; +client_init(Master, ssl, pem_cache, Host, Port) -> + {ok, Sock} = ssl:connect(Host, Port, ssl_opts(connect_der)), + Master ! {self(), init}, + Size = byte_size(msg()), + {Sock, Size}; client_init(_Me, Type, Tc, Host, Port) -> io:format("No client init code for ~p ~p~n",[Type, Tc]), {Host, Port}. @@ -228,6 +282,13 @@ payload(Loop, ssl, D = {Socket, Size}) when Loop > 0 -> payload(_, _, {Socket, _}) -> ssl:close(Socket). +pem_cache(N, ssl, Data = {Socket, Size}) when N > 0 -> + ok = ssl:send(Socket, msg()), + {ok, _} = ssl:recv(Socket, Size), + pem_cache(N-1, ssl, Data); +pem_cache(_, _, {Socket, _}) -> + ssl:close(Socket). + msg() -> <<"Hello", 0:(512*8), @@ -352,16 +413,43 @@ stop_profile(fprof, File) -> ssl_opts(listen) -> [{backlog, 500} | ssl_opts("server")]; ssl_opts(connect) -> - [{verify, verify_peer} - | ssl_opts("client")]; + [{verify, verify_peer} | ssl_opts("client")]; +ssl_opts(listen_der) -> + [{backlog, 500} | ssl_opts("server_der")]; +ssl_opts(connect_der) -> + [{verify, verify_peer} | ssl_opts("client_der")]; ssl_opts(Role) -> - Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), + CertData = cert_data(Role), [{active, false}, {depth, 2}, {reuseaddr, true}, {mode,binary}, {nodelay, true}, - {ciphers, [{dhe_rsa,aes_256_cbc,sha}]}, - {cacertfile, filename:join([Dir, Role, "cacerts.pem"])}, + {ciphers, [{dhe_rsa,aes_256_cbc,sha}]} + |CertData]. + +cert_data(Der) when Der =:= "server_der"; Der =:= "client_der" -> + [Role,_] = string:tokens(Der, "_"), + Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), + {ok, CaCert0} = file:read_file(filename:join([Dir, Role, "cacerts.pem"])), + {ok, Cert0} = file:read_file(filename:join([Dir, Role, "cert.pem"])), + {ok, Key0} = file:read_file(filename:join([Dir, Role, "key.pem"])), + [{_, Cert, _}] = public_key:pem_decode(Cert0), + CaCert1 = public_key:pem_decode(CaCert0), + CaCert = [CCert || {_, CCert, _} <- CaCert1], + [{KeyType, Key, _}] = public_key:pem_decode(Key0), + [{cert, Cert}, + {cacerts, CaCert}, + {key, {KeyType, Key}}]; +cert_data(Role) -> + Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), + [{cacertfile, filename:join([Dir, Role, "cacerts.pem"])}, {certfile, filename:join([Dir, Role, "cert.pem"])}, {keyfile, filename:join([Dir, Role, "key.pem"])}]. + +bypass_pem_cache_supported() -> + %% This function is currently critical to support cache bypass + %% and did not exist in prior versions. + catch ssl_pkix_db:module_info(), % ensure module is loaded + erlang:function_exported(ssl_pkix_db, extract_trusted_certs, 1). + diff --git a/lib/ssl/test/ssl_certificate_verify_SUITE.erl b/lib/ssl/test/ssl_certificate_verify_SUITE.erl index c83c513eb3..4c6f1d7c01 100644 --- a/lib/ssl/test/ssl_certificate_verify_SUITE.erl +++ b/lib/ssl/test/ssl_certificate_verify_SUITE.erl @@ -85,7 +85,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), %% make rsa certs using oppenssl {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), diff --git a/lib/ssl/test/ssl_crl_SUITE.erl b/lib/ssl/test/ssl_crl_SUITE.erl index e37e127440..bc2822f0c4 100644 --- a/lib/ssl/test/ssl_crl_SUITE.erl +++ b/lib/ssl/test/ssl_crl_SUITE.erl @@ -136,7 +136,7 @@ init_per_testcase(Case, Config0) -> true -> end_per_testcase(Case, Config0), inets:start(), - ssl:start(), + ssl_test_lib:clean_start(), ServerRoot = make_dir_path([proplists:get_value(priv_dir, Config0), idp_crl, tmp]), %% start a HTTP server to serve the CRLs {ok, Httpd} = inets:start(httpd, [{ipfamily, proplists:get_value(ipfamily, Config0)}, @@ -155,7 +155,7 @@ init_per_testcase(Case, Config0) -> [{cert_dir, CertDir} | Config]; false -> end_per_testcase(Case, Config0), - ssl:start(), + ssl_test_lib:clean_start(), Config0 end. diff --git a/lib/ssl/test/ssl_handshake_SUITE.erl b/lib/ssl/test/ssl_handshake_SUITE.erl index a671e3e307..51f0651568 100644 --- a/lib/ssl/test/ssl_handshake_SUITE.erl +++ b/lib/ssl/test/ssl_handshake_SUITE.erl @@ -60,7 +60,7 @@ init_per_testcase(ignore_hassign_extension_pre_tls_1_2, Config0) -> ok -> case is_supported(sha512) of true -> - ssl:start(), + ssl_test_lib:clean_start(), %% make rsa certs using oppenssl {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), diff --git a/lib/ssl/test/ssl_npn_handshake_SUITE.erl b/lib/ssl/test/ssl_npn_handshake_SUITE.erl index c55fa73cfb..a02881f1ae 100644 --- a/lib/ssl/test/ssl_npn_handshake_SUITE.erl +++ b/lib/ssl/test/ssl_npn_handshake_SUITE.erl @@ -68,7 +68,7 @@ init_per_suite(Config) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config), proplists:get_value(priv_dir, Config)), ssl_test_lib:cert_options(Config) diff --git a/lib/ssl/test/ssl_npn_hello_SUITE.erl b/lib/ssl/test/ssl_npn_hello_SUITE.erl index 6ae9efe5e9..69aeea10c5 100644 --- a/lib/ssl/test/ssl_npn_hello_SUITE.erl +++ b/lib/ssl/test/ssl_npn_hello_SUITE.erl @@ -135,15 +135,12 @@ create_server_handshake(Npn) -> }, Vsn). create_connection_states() -> - #connection_states{ - pending_read = #connection_state{ - security_parameters = #security_parameters{ + #{pending_read => #{security_parameters => #security_parameters{ server_random = <<1:256>>, compression_algorithm = 1, cipher_suite = ?TLS_DHE_DSS_WITH_DES_CBC_SHA } - }, - current_read = #connection_state { - secure_renegotiation = false - } - }. + }, + current_read => #{secure_renegotiation => false + } + }. diff --git a/lib/ssl/test/ssl_packet_SUITE.erl b/lib/ssl/test/ssl_packet_SUITE.erl index 17237118a0..81a49776e4 100644 --- a/lib/ssl/test/ssl_packet_SUITE.erl +++ b/lib/ssl/test/ssl_packet_SUITE.erl @@ -140,8 +140,7 @@ init_per_suite(Config) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:stop(), - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config), proplists:get_value(priv_dir, Config)), ssl_test_lib:cert_options(Config) @@ -278,6 +277,7 @@ packet_raw_active_once_many_small() -> [{doc,"Test packet option {packet, raw} in active once mode."}]. packet_raw_active_once_many_small(Config) when is_list(Config) -> + ct:timetrap({seconds, ?BASE_TIMEOUT_SECONDS * ?MANY_SCALE}), Data = "Packet option is {packet, raw}", packet(Config, Data, send_raw, active_once_raw, ?MANY, raw, once). @@ -394,6 +394,7 @@ packet_0_active_some_big() -> [{doc,"Test packet option {packet, 0} in active mode."}]. packet_0_active_some_big(Config) when is_list(Config) -> + ct:timetrap({seconds, ?BASE_TIMEOUT_SECONDS * ?SOME_SCALE}), Data = lists:append(lists:duplicate(100, "1234567890")), packet(Config, Data, send, active_raw, ?SOME, 0, true). @@ -429,6 +430,7 @@ packet_2_active_some_big() -> [{doc,"Test packet option {packet, 2} in active mode"}]. packet_2_active_some_big(Config) when is_list(Config) -> + ct:timetrap({seconds, ?BASE_TIMEOUT_SECONDS * ?SOME_SCALE}), Data = lists:append(lists:duplicate(100, "1234567890")), packet(Config, Data, send, active_packet, ?SOME, 2, true). @@ -1902,6 +1904,31 @@ header_decode_two_bytes_one_sent_passive(Config) when is_list(Config) -> %%-------------------------------------------------------------------- %% Internal functions ------------------------------------------------ %%-------------------------------------------------------------------- + +packet(Config, Data, Send, Recv, Quantity, Packet, Active) when Packet == 0; + Packet == raw -> + ClientOpts = ssl_test_lib:ssl_options(client_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + + Server = ssl_test_lib:start_server([{node, ClientNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, Send ,[Data, Quantity]}}, + {options, [{nodelay, true},{packet, Packet} | ServerOpts]}]), + Port = ssl_test_lib:inet_port(Server), + Client = ssl_test_lib:start_client([{node, ServerNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, Recv, [Data, Quantity]}}, + {options, [{active, Active}, {nodelay, true}, + {packet, Packet} | + ClientOpts]}]), + + ssl_test_lib:check_result(Client, ok), + + ssl_test_lib:close(Server), + ssl_test_lib:close(Client); + packet(Config, Data, Send, Recv, Quantity, Packet, Active) -> ClientOpts = ssl_test_lib:ssl_options(client_opts, Config), ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), diff --git a/lib/ssl/test/ssl_payload_SUITE.erl b/lib/ssl/test/ssl_payload_SUITE.erl index c0b762760d..cb1957327a 100644 --- a/lib/ssl/test/ssl_payload_SUITE.erl +++ b/lib/ssl/test/ssl_payload_SUITE.erl @@ -70,7 +70,7 @@ init_per_suite(Config) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config), proplists:get_value(priv_dir, Config)), ssl_test_lib:cert_options(Config) catch _:_ -> diff --git a/lib/ssl/test/ssl_pem_cache_SUITE.erl b/lib/ssl/test/ssl_pem_cache_SUITE.erl index 13b0ce8ed9..02c98fc40f 100644 --- a/lib/ssl/test/ssl_pem_cache_SUITE.erl +++ b/lib/ssl/test/ssl_pem_cache_SUITE.erl @@ -43,7 +43,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), %% make rsa certs using oppenssl {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), @@ -63,14 +63,15 @@ end_per_group(_GroupName, Config) -> Config. init_per_testcase(pem_cleanup = Case, Config) -> - end_per_testcase(Case, Config) , application:load(ssl), + end_per_testcase(Case, Config) , application:set_env(ssl, ssl_pem_cache_clean, ?CLEANUP_INTERVAL), ssl:start(), ct:timetrap({minutes, 1}), Config. end_per_testcase(_TestCase, Config) -> + ssl_test_lib:clean_env(), ssl:stop(), Config. diff --git a/lib/ssl/test/ssl_session_cache_SUITE.erl b/lib/ssl/test/ssl_session_cache_SUITE.erl index b352844ba0..28637fc32d 100644 --- a/lib/ssl/test/ssl_session_cache_SUITE.erl +++ b/lib/ssl/test/ssl_session_cache_SUITE.erl @@ -58,7 +58,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), %% make rsa certs using {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), diff --git a/lib/ssl/test/ssl_sni_SUITE.erl b/lib/ssl/test/ssl_sni_SUITE.erl index 34ef2e6af9..4e916a7f03 100644 --- a/lib/ssl/test/ssl_sni_SUITE.erl +++ b/lib/ssl/test/ssl_sni_SUITE.erl @@ -41,7 +41,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), ssl_test_lib:cert_options(Config0) diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index fd8af5efaa..81f16030f7 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -807,22 +807,24 @@ send_selected_port(_,_,_) -> rsa_suites(CounterPart) -> ECC = is_sane_ecc(CounterPart), FIPS = is_fips(CounterPart), + CryptoSupport = crypto:supports(), + Ciphers = proplists:get_value(ciphers, CryptoSupport), lists:filter(fun({rsa, des_cbc, sha}) when FIPS == true -> false; ({dhe_rsa, des_cbc, sha}) when FIPS == true -> false; - ({rsa, _, _}) -> - true; - ({dhe_rsa, _, _}) -> - true; - ({ecdhe_rsa, _, _}) when ECC == true -> - true; - ({rsa, _, _, _}) -> - true; - ({dhe_rsa, _, _,_}) -> - true; - ({ecdhe_rsa, _, _,_}) when ECC == true -> - true; + ({rsa, Cipher, _}) -> + lists:member(Cipher, Ciphers); + ({dhe_rsa, Cipher, _}) -> + lists:member(Cipher, Ciphers); + ({ecdhe_rsa, Cipher, _}) when ECC == true -> + lists:member(Cipher, Ciphers); + ({rsa, Cipher, _, _}) -> + lists:member(Cipher, Ciphers); + ({dhe_rsa, Cipher, _,_}) -> + lists:member(Cipher, Ciphers); + ({ecdhe_rsa, Cipher, _,_}) when ECC == true -> + lists:member(Cipher, Ciphers); (_) -> false end, @@ -1353,3 +1355,19 @@ ct_log_supported_protocol_versions(Config) -> _ -> ct:log("TLS/SSL version ~p~n ", [tls_record:supported_protocol_versions()]) end. + +clean_env() -> + application:unset_env(ssl, protocol_version), + application:unset_env(ssl, session_lifetime), + application:unset_env(ssl, session_cb), + application:unset_env(ssl, session_cb_init_args), + application:unset_env(ssl, session_cache_client_max), + application:unset_env(ssl, session_cache_server_max), + application:unset_env(ssl, ssl_pem_cache_clean), + application:unset_env(ssl, alert_timeout). + +clean_start() -> + ssl:stop(), + application:load(ssl), + clean_env(), + ssl:start(). diff --git a/lib/ssl/test/ssl_to_openssl_SUITE.erl b/lib/ssl/test/ssl_to_openssl_SUITE.erl index 83a4dae0a1..9ecfe5b0ea 100644 --- a/lib/ssl/test/ssl_to_openssl_SUITE.erl +++ b/lib/ssl/test/ssl_to_openssl_SUITE.erl @@ -55,7 +55,9 @@ groups() -> basic_tests() -> [basic_erlang_client_openssl_server, basic_erlang_server_openssl_client, - expired_session]. + expired_session, + ssl2_erlang_server_openssl_client_comp + ]. all_versions_tests() -> [ @@ -74,7 +76,8 @@ all_versions_tests() -> ciphers_dsa_signed_certs, erlang_client_bad_openssl_server, expired_session, - ssl2_erlang_server_openssl_client]. + ssl2_erlang_server_openssl_client + ]. alpn_tests() -> [erlang_client_alpn_openssl_server_alpn, @@ -116,8 +119,7 @@ init_per_suite(Config0) -> catch crypto:stop(), try crypto:start() of ok -> - ssl:stop(), - ssl:start(), + ssl_test_lib:clean_start(), {ok, _} = make_certs:all(proplists:get_value(data_dir, Config0), proplists:get_value(priv_dir, Config0)), Config1 = ssl_test_lib:make_dsa_cert(Config0), @@ -181,7 +183,8 @@ special_init(TestCase, Config) {ok, Version} = application:get_env(ssl, protocol_version), check_sane_openssl_renegotaite(Config, Version); -special_init(ssl2_erlang_server_openssl_client, Config) -> +special_init(Case, Config) when Case == ssl2_erlang_server_openssl_client; + Case == ssl2_erlang_server_openssl_client_comp -> case ssl_test_lib:supports_ssl_tls_version(sslv2) of true -> Config; @@ -955,8 +958,52 @@ ssl2_erlang_server_openssl_client(Config) when is_list(Config) -> Data = "From openssl to erlang", Server = ssl_test_lib:start_server_error([{node, ServerNode}, {port, 0}, + {from, self()}, + {options, ServerOpts}]), + Port = ssl_test_lib:inet_port(Server), + + Exe = "openssl", + Args = ["s_client", "-connect", "localhost:" ++ integer_to_list(Port), + "-ssl2", "-msg"], + + OpenSslPort = ssl_test_lib:portable_open_port(Exe, Args), + true = port_command(OpenSslPort, Data), + + ct:log("Ports ~p~n", [[erlang:port_info(P) || P <- erlang:ports()]]), + receive + {'EXIT', OpenSslPort, _} = Exit -> + ct:log("Received: ~p ~n", [Exit]), + ok + end, + receive + {'EXIT', _, _} = UnkownExit -> + Msg = lists:flatten(io_lib:format("Received: ~p ~n", [UnkownExit])), + ct:log(Msg), + ct:comment(Msg), + ok + after 0 -> + ok + end, + ssl_test_lib:check_result(Server, {error, {tls_alert, "handshake failure"}}), + process_flag(trap_exit, false). +%%-------------------------------------------------------------------- +ssl2_erlang_server_openssl_client_comp() -> + [{doc,"Test that ssl v2 clients are rejected"}]. + +ssl2_erlang_server_openssl_client_comp(Config) when is_list(Config) -> + process_flag(trap_exit, true), + ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), + V2Compat = proplists:get_value(v2_hello_compatible, Config), + + ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), + + {_, ServerNode, _} = ssl_test_lib:run_where(Config), + + Data = "From openssl to erlang", + + Server = ssl_test_lib:start_server_error([{node, ServerNode}, {port, 0}, {from, self()}, - {options, ServerOpts}]), + {options, [{v2_hello_compatible, V2Compat} | ServerOpts]}]), Port = ssl_test_lib:inet_port(Server), Exe = "openssl", diff --git a/lib/stdlib/doc/src/notes.xml b/lib/stdlib/doc/src/notes.xml index f0347703e7..554150380f 100644 --- a/lib/stdlib/doc/src/notes.xml +++ b/lib/stdlib/doc/src/notes.xml @@ -31,6 +31,66 @@ </header> <p>This document describes the changes made to the STDLIB application.</p> +<section><title>STDLIB 3.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The <c>zip:unzip/1,2</c> and <c>zip:extract/1,2</c> + functions have been updated to handle directory traversal + exploits. Any element in the zip file that contains a + path that points to a directory above the top level + working directory, <c>cwd</c>, will instead be extracted + in <c>cwd</c>. An error message is printed for any such + element in the zip file during the unzip operation. The + <c>keep_old_files</c> option determines if a file will + overwrite a previous file with the same name within the + zip file.</p> + <p> + Own Id: OTP-13633</p> + </item> + <item> + <p> Correct the contracts for + <c>ets:match_object/1,3</c>. </p> + <p> + Own Id: OTP-13721 Aux Id: PR-1113 </p> + </item> + <item> + <p> + Errors in type specification and Emacs template + generation for <c>gen_statem:code_change/4</c> has been + fixed from bugs.erlang.org's Jira cases ERL-172 and + ERL-187.</p> + <p> + Own Id: OTP-13746 Aux Id: ERL-172, ERL-187 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + gen_statem has been changed to set the callback mode for + a server to what Module:callback_mode/0 returns. This + facilitates e.g code downgrade since the callback mode + now becomes a property of the currently active code, not + of the server process.</p> + <p> + Exception handling from Module:init/1 has also been + improved.</p> + <p> + *** POTENTIAL INCOMPATIBILITY ***</p> + <p> + Own Id: OTP-13752</p> + </item> + </list> + </section> + +</section> + <section><title>STDLIB 3.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/stdlib/src/edlin_expand.erl b/lib/stdlib/src/edlin_expand.erl index ec64470461..5f821caef0 100644 --- a/lib/stdlib/src/edlin_expand.erl +++ b/lib/stdlib/src/edlin_expand.erl @@ -118,7 +118,7 @@ format_col([A|T], Width, Len, Acc0) -> {H1, _} -> H1; H2 -> H2 end, - Acc = [io_lib:format("~-*s", [Width,H]) | Acc0], + Acc = [io_lib:format("~-*ts", [Width,H]) | Acc0], format_col(T, Width, Len+Width, Acc); format_col([], _, _, Acc) -> lists:reverse(Acc, "\n"). diff --git a/lib/stdlib/src/stdlib.appup.src b/lib/stdlib/src/stdlib.appup.src index 9877662743..e917b7ea1f 100644 --- a/lib/stdlib/src/stdlib.appup.src +++ b/lib/stdlib/src/stdlib.appup.src @@ -18,9 +18,9 @@ %% %CopyrightEnd% {"%VSN%", %% Up from - max one major revision back - [{<<"3\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* + [{<<"3\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* {<<"2\\.[5-8](\\.[0-9]+)*">>,[restart_new_emulator]}], % OTP-18.* %% Down to - max one major revision back - [{<<"3\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* + [{<<"3\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-19.* {<<"2\\.[5-8](\\.[0-9]+)*">>,[restart_new_emulator]}] % OTP-18.* }. diff --git a/lib/stdlib/src/zip.erl b/lib/stdlib/src/zip.erl index f8ba6f18e9..340cc21390 100644 --- a/lib/stdlib/src/zip.erl +++ b/lib/stdlib/src/zip.erl @@ -279,7 +279,8 @@ do_openzip_get(F, #openzip{files = Files, in = In0, input = Input, case file_name_search(F, Files) of {#zip_file{offset = Offset},_}=ZFile -> In1 = Input({seek, bof, Offset}, In0), - case get_z_file(In1, Z, Input, Output, [], fun silent/1, CWD, ZFile) of + case get_z_file(In1, Z, Input, Output, [], fun silent/1, + CWD, ZFile, fun all/1) of {file, R, _In2} -> {ok, R}; _ -> throw(file_not_found) end; @@ -1403,9 +1404,10 @@ get_z_files([{#zip_file{offset = Offset},_} = ZFile | Rest], Z, In0, true -> In1 = Input({seek, bof, Offset}, In0), {In2, Acc1} = - case get_z_file(In1, Z, Input, Output, OpO, FB, CWD, ZFile) of + case get_z_file(In1, Z, Input, Output, OpO, FB, + CWD, ZFile, Filter) of {file, GZD, Inx} -> {Inx, [GZD | Acc0]}; - {dir, Inx} -> {Inx, Acc0} + {_, Inx} -> {Inx, Acc0} end, get_z_files(Rest, Z, In2, Opts, Acc1); _ -> @@ -1413,7 +1415,8 @@ get_z_files([{#zip_file{offset = Offset},_} = ZFile | Rest], Z, In0, end. %% get a file from the archive, reading chunks -get_z_file(In0, Z, Input, Output, OpO, FB, CWD, {ZipFile,Extra}) -> +get_z_file(In0, Z, Input, Output, OpO, FB, + CWD, {ZipFile,Extra}, Filter) -> case Input({read, ?LOCAL_FILE_HEADER_SZ}, In0) of {eof, In1} -> {eof, In1}; @@ -1433,29 +1436,64 @@ get_z_file(In0, Z, Input, Output, OpO, FB, CWD, {ZipFile,Extra}) -> end, {BFileN, In3} = Input({read, FileNameLen + ExtraLen}, In1), {FileName, _} = get_file_name_extra(FileNameLen, ExtraLen, BFileN), - FileName1 = add_cwd(CWD, FileName), - case lists:last(FileName) of - $/ -> - %% perhaps this should always be done? - Output({ensure_dir,FileName1},[]), - {dir, In3}; - _ -> - %% FileInfo = local_file_header_to_file_info(LH) - %%{Out, In4, CRC, UncompSize} = - {Out, In4, CRC, _UncompSize} = - get_z_data(CompMethod, In3, FileName1, - CompSize, Input, Output, OpO, Z), - In5 = skip_z_data_descriptor(GPFlag, Input, In4), - %% TODO This should be fixed some day: - %% In5 = Input({set_file_info, FileName, FileInfo#file_info{size=UncompSize}}, In4), - FB(FileName), - CRC =:= CRC32 orelse throw({bad_crc, FileName}), - {file, Out, In5} + ReadAndWrite = + case check_valid_location(CWD, FileName) of + {true,FileName1} -> + true; + {false,FileName1} -> + Filter({ZipFile#zip_file{name = FileName1},Extra}) + end, + case ReadAndWrite of + true -> + case lists:last(FileName) of + $/ -> + %% perhaps this should always be done? + Output({ensure_dir,FileName1},[]), + {dir, In3}; + _ -> + %% FileInfo = local_file_header_to_file_info(LH) + %%{Out, In4, CRC, UncompSize} = + {Out, In4, CRC, _UncompSize} = + get_z_data(CompMethod, In3, FileName1, + CompSize, Input, Output, OpO, Z), + In5 = skip_z_data_descriptor(GPFlag, Input, In4), + %% TODO This should be fixed some day: + %% In5 = Input({set_file_info, FileName, + %% FileInfo#file_info{size=UncompSize}}, In4), + FB(FileName), + CRC =:= CRC32 orelse throw({bad_crc, FileName}), + {file, Out, In5} + end; + false -> + {ignore, In3} end; _ -> throw(bad_local_file_header) end. +%% make sure FileName doesn't have relative path that points over CWD +check_valid_location(CWD, FileName) -> + %% check for directory traversal exploit + case check_dir_level(filename:split(FileName), 0) of + {FileOrDir,Level} when Level < 0 -> + CWD1 = if CWD == "" -> "./"; + true -> CWD + end, + error_logger:format("Illegal path: ~ts, extracting in ~ts~n", + [add_cwd(CWD,FileName),CWD1]), + {false,add_cwd(CWD, FileOrDir)}; + _ -> + {true,add_cwd(CWD, FileName)} + end. + +check_dir_level([FileOrDir], Level) -> + {FileOrDir,Level}; +check_dir_level(["." | Parts], Level) -> + check_dir_level(Parts, Level); +check_dir_level([".." | Parts], Level) -> + check_dir_level(Parts, Level-1); +check_dir_level([_Dir | Parts], Level) -> + check_dir_level(Parts, Level+1). get_file_name_extra(FileNameLen, ExtraLen, B) -> case B of diff --git a/lib/stdlib/test/edlin_expand_SUITE.erl b/lib/stdlib/test/edlin_expand_SUITE.erl index ccffa2e244..718d91c6a3 100644 --- a/lib/stdlib/test/edlin_expand_SUITE.erl +++ b/lib/stdlib/test/edlin_expand_SUITE.erl @@ -21,7 +21,7 @@ -export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_testcase/2, end_per_testcase/2, init_per_group/2,end_per_group/2]). --export([normal/1, quoted_fun/1, quoted_module/1, quoted_both/1]). +-export([normal/1, quoted_fun/1, quoted_module/1, quoted_both/1, erl_1152/1]). -include_lib("common_test/include/ct.hrl"). @@ -36,7 +36,7 @@ suite() -> {timetrap,{minutes,1}}]. all() -> - [normal, quoted_fun, quoted_module, quoted_both]. + [normal, quoted_fun, quoted_module, quoted_both, erl_1152]. groups() -> []. @@ -149,5 +149,12 @@ quoted_both(Config) when is_list(Config) -> {yes,"weird-fun-name'()",[]} = do_expand("'ExpandTestCaps1':'#"), ok. +erl_1152(Config) when is_list(Config) -> + "\n"++"foo"++" "++[1089]++_ = do_format(["foo",[1089]]), + ok. + do_expand(String) -> edlin_expand:expand(lists:reverse(String)). + +do_format(StringList) -> + lists:flatten(edlin_expand:format_matches(StringList)). diff --git a/lib/stdlib/test/zip_SUITE.erl b/lib/stdlib/test/zip_SUITE.erl index 2add5a39a2..7d90795c9e 100644 --- a/lib/stdlib/test/zip_SUITE.erl +++ b/lib/stdlib/test/zip_SUITE.erl @@ -25,6 +25,7 @@ zip_to_binary/1, unzip_options/1, zip_options/1, list_dir_options/1, aliases/1, openzip_api/1, zip_api/1, open_leak/1, unzip_jar/1, + unzip_traversal_exploit/1, compress_control/1, foldl/1]). @@ -38,7 +39,8 @@ all() -> [borderline, atomic, bad_zip, unzip_from_binary, unzip_to_binary, zip_to_binary, unzip_options, zip_options, list_dir_options, aliases, openzip_api, - zip_api, open_leak, unzip_jar, compress_control, foldl]. + zip_api, open_leak, unzip_jar, compress_control, foldl, + unzip_traversal_exploit]. groups() -> []. @@ -377,6 +379,52 @@ unzip_options(Config) when is_list(Config) -> 0 = delete_files([Subdir]), ok. +%% Test that unzip handles directory traversal exploit (OTP-13633) +unzip_traversal_exploit(Config) -> + DataDir = proplists:get_value(data_dir, Config), + PrivDir = proplists:get_value(priv_dir, Config), + ZipName = filename:join(DataDir, "exploit.zip"), + + %% $ zipinfo -1 test/zip_SUITE_data/exploit.zip + %% clash.txt + %% ../clash.txt + %% ../above.txt + %% subdir/../in_root_dir.txt + + %% create a temp directory + SubDir = filename:join(PrivDir, "exploit_test"), + ok = file:make_dir(SubDir), + + ClashFile = filename:join(SubDir,"clash.txt"), + AboveFile = filename:join(SubDir,"above.txt"), + RelativePathFile = filename:join(SubDir,"subdir/../in_root_dir.txt"), + + %% unzip in SubDir + {ok, [ClashFile, ClashFile, AboveFile, RelativePathFile]} = + zip:unzip(ZipName, [{cwd,SubDir}]), + + {ok,<<"This file will overwrite other file.\n">>} = + file:read_file(ClashFile), + {ok,_} = file:read_file(AboveFile), + {ok,_} = file:read_file(RelativePathFile), + + %% clean up + delete_files([SubDir]), + + %% create the temp directory again + ok = file:make_dir(SubDir), + + %% unzip in SubDir + {ok, [ClashFile, AboveFile, RelativePathFile]} = + zip:unzip(ZipName, [{cwd,SubDir},keep_old_files]), + + {ok,<<"This is the original file.\n">>} = + file:read_file(ClashFile), + + %% clean up + delete_files([SubDir]), + ok. + %% Test unzip a jar file (OTP-7382). unzip_jar(Config) when is_list(Config) -> DataDir = proplists:get_value(data_dir, Config), diff --git a/lib/stdlib/test/zip_SUITE_data/exploit.zip b/lib/stdlib/test/zip_SUITE_data/exploit.zip Binary files differnew file mode 100644 index 0000000000..afb8dbd192 --- /dev/null +++ b/lib/stdlib/test/zip_SUITE_data/exploit.zip diff --git a/lib/syntax_tools/doc/src/notes.xml b/lib/syntax_tools/doc/src/notes.xml index ef207f7c3d..82c4484d96 100644 --- a/lib/syntax_tools/doc/src/notes.xml +++ b/lib/syntax_tools/doc/src/notes.xml @@ -32,6 +32,40 @@ <p>This document describes the changes made to the Syntax_Tools application.</p> +<section><title>Syntax_Tools 2.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> Fix a bug where <c>erl_tidy</c> crashed on the tilde + character when printing to standard output. </p> + <p> + Own Id: OTP-13725 Aux Id: ERL-151, PR-1071 </p> + </item> + <item> + <p><c>merl_transform</c> could get into an infinite loop + when syntactically incorrect text was passed to a + <c>merl:qquote/2,3</c> call.</p> + <p> + Own Id: OTP-13755</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>Improve types and specs in OTP documentation generated + from Erlang source files. </p> + <p> + Own Id: OTP-13720 Aux Id: ERL-120 </p> + </item> + </list> + </section> + +</section> + <section><title>Syntax_Tools 2.0</title> <section><title>Improvements and New Features</title> diff --git a/lib/syntax_tools/vsn.mk b/lib/syntax_tools/vsn.mk index f09c2a01d0..c0ca083c38 100644 --- a/lib/syntax_tools/vsn.mk +++ b/lib/syntax_tools/vsn.mk @@ -1 +1 @@ -SYNTAX_TOOLS_VSN = 2.0 +SYNTAX_TOOLS_VSN = 2.1 diff --git a/lib/tools/doc/src/notes.xml b/lib/tools/doc/src/notes.xml index a0a817c0f2..2d9bee0dd1 100644 --- a/lib/tools/doc/src/notes.xml +++ b/lib/tools/doc/src/notes.xml @@ -31,6 +31,38 @@ </header> <p>This document describes the changes made to the Tools application.</p> +<section><title>Tools 2.8.6</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Errors in type specification and Emacs template + generation for <c>gen_statem:code_change/4</c> has been + fixed from bugs.erlang.org's Jira cases ERL-172 and + ERL-187.</p> + <p> + Own Id: OTP-13746 Aux Id: ERL-172, ERL-187 </p> + </item> + <item> + <p> + Fix gc_start/gc_end in fprof tags when parsing old trace + logs.</p> + <p> + Own Id: OTP-13778 Aux Id: PR-1136 </p> + </item> + <item> + <p><c>make</c> (tools) and <c>ct_make</c> (common_test) + would crash if an Erlang source file contained a + <c>-warning()</c> directive.</p> + <p> + Own Id: OTP-13855</p> + </item> + </list> + </section> + +</section> + <section><title>Tools 2.8.5</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/wx/doc/src/notes.xml b/lib/wx/doc/src/notes.xml index 4f0e166924..70ff0a92b7 100644 --- a/lib/wx/doc/src/notes.xml +++ b/lib/wx/doc/src/notes.xml @@ -32,6 +32,23 @@ <p>This document describes the changes made to the wxErlang application.</p> +<section><title>Wx 1.7.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Increased the stacksize for the wx thread. The default + stacksize on Windows is 1MB which is not enough if the + user created many nested dialogs.</p> + <p> + Own Id: OTP-13816</p> + </item> + </list> + </section> + +</section> + <section><title>Wx 1.7</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/wx/vsn.mk b/lib/wx/vsn.mk index de4e5e1935..0ce63d9f71 100644 --- a/lib/wx/vsn.mk +++ b/lib/wx/vsn.mk @@ -1 +1 @@ -WX_VSN = 1.7 +WX_VSN = 1.7.1 diff --git a/lib/xmerl/doc/src/notes.xml b/lib/xmerl/doc/src/notes.xml index 0abcb87998..12e64537ed 100644 --- a/lib/xmerl/doc/src/notes.xml +++ b/lib/xmerl/doc/src/notes.xml @@ -32,6 +32,21 @@ <p>This document describes the changes made to the Xmerl application.</p> +<section><title>Xmerl 1.3.12</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> Fix a number of broken links in the xmerl + documentation. </p> + <p> + Own Id: OTP-13880</p> + </item> + </list> + </section> + +</section> + <section><title>Xmerl 1.3.11</title> <section><title>Improvements and New Features</title> diff --git a/lib/xmerl/src/xmerl_eventp.erl b/lib/xmerl/src/xmerl_eventp.erl index 2cb76abc6e..8d7ea25e24 100644 --- a/lib/xmerl/src/xmerl_eventp.erl +++ b/lib/xmerl/src/xmerl_eventp.erl @@ -25,6 +25,90 @@ %% Each contain more elaborate settings of xmerl_scan that makes usage of %% the customization functions. %% +%% @type xmlElement() = #xmlElement{}. +%% +%% @type option_list(). <p>Options allow to customize the behaviour of the +%% scanner. +%% See also <a href="xmerl_examples.html">tutorial</a> on customization +%% functions. +%% </p> +%% <p> +%% Possible options are: +%% </p> +%% <dl> +%% <dt><code>{acc_fun, Fun}</code></dt> +%% <dd>Call back function to accumulate contents of entity.</dd> +%% <dt><code>{continuation_fun, Fun} | +%% {continuation_fun, Fun, ContinuationState}</code></dt> +%% <dd>Call back function to decide what to do if the scanner runs into EOF +%% before the document is complete.</dd> +%% <dt><code>{event_fun, Fun} | +%% {event_fun, Fun, EventState}</code></dt> +%% <dd>Call back function to handle scanner events.</dd> +%% <dt><code>{fetch_fun, Fun} | +%% {fetch_fun, Fun, FetchState}</code></dt> +%% <dd>Call back function to fetch an external resource.</dd> +%% <dt><code>{hook_fun, Fun} | +%% {hook_fun, Fun, HookState}</code></dt> +%% <dd>Call back function to process the document entities once +%% identified.</dd> +%% <dt><code>{close_fun, Fun}</code></dt> +%% <dd>Called when document has been completely parsed.</dd> +%% <dt><code>{rules, ReadFun, WriteFun, RulesState} | +%% {rules, Rules}</code></dt> +%% <dd>Handles storing of scanner information when parsing.</dd> +%% <dt><code>{user_state, UserState}</code></dt> +%% <dd>Global state variable accessible from all customization functions</dd> +%% +%% <dt><code>{fetch_path, PathList}</code></dt> +%% <dd>PathList is a list of +%% directories to search when fetching files. If the file in question +%% is not in the fetch_path, the URI will be used as a file +%% name.</dd> +%% <dt><code>{space, Flag}</code></dt> +%% <dd>'preserve' (default) to preserve spaces, 'normalize' to +%% accumulate consecutive whitespace and replace it with one space.</dd> +%% <dt><code>{line, Line}</code></dt> +%% <dd>To specify starting line for scanning in document which contains +%% fragments of XML.</dd> +%% <dt><code>{namespace_conformant, Flag}</code></dt> +%% <dd>Controls whether to behave as a namespace conformant XML parser, +%% 'false' (default) to not otherwise 'true'.</dd> +%% <dt><code>{validation, Flag}</code></dt> +%% <dd>Controls whether to process as a validating XML parser: +%% 'off' (default) no validation, or validation 'dtd' by DTD or 'schema' +%% by XML Schema. 'false' and 'true' options are obsolete +%% (i.e. they may be removed in a future release), if used 'false' +%% equals 'off' and 'true' equals 'dtd'.</dd> +%% <dt><code>{schemaLocation, [{Namespace,Link}|...]}</code></dt> +%% <dd>Tells explicitly which XML Schema documents to use to validate +%% the XML document. Used together with the +%% <code>{validation,schema}</code> option.</dd> +%% <dt><code>{quiet, Flag}</code></dt> +%% <dd>Set to 'true' if xmerl should behave quietly and not output any +%% information to standard output (default 'false').</dd> +%% <dt><code>{doctype_DTD, DTD}</code></dt> +%% <dd>Allows to specify DTD name when it isn't available in the XML +%% document. This option has effect only together with +%% <code>{validation,'dtd'</code> option.</dd> +%% <dt><code>{xmlbase, Dir}</code></dt> +%% <dd>XML Base directory. If using string/1 default is current directory. +%% If using file/1 default is directory of given file.</dd> +%% <dt><code>{encoding, Enc}</code></dt> +%% <dd>Set default character set used (default UTF-8). +%% This character set is used only if not explicitly given by the XML +%% declaration. </dd> +%% <dt><code>{document, Flag}</code></dt> +%% <dd>Set to 'true' if xmerl should return a complete XML document +%% as an xmlDocument record (default 'false').</dd> +%% <dt><code>{comments, Flag}</code></dt> +%% <dd>Set to 'false' if xmerl should skip comments otherwise they will +%% be returned as xmlComment records (default 'true').</dd> +%% <dt><code>{default_attrs, Flag}</code></dt> +%% <dd>Set to 'true' if xmerl should add to elements missing attributes +%% with a defined default value (default 'false').</dd> +%% </dl> +%% -module(xmerl_eventp). -vsn('0.19'). -date('03-09-17'). diff --git a/lib/xmerl/src/xmerl_scan.erl b/lib/xmerl/src/xmerl_scan.erl index 2147a46a13..5e0459ec21 100644 --- a/lib/xmerl/src/xmerl_scan.erl +++ b/lib/xmerl/src/xmerl_scan.erl @@ -111,13 +111,16 @@ %% <dd>Set to 'true' if xmerl should add to elements missing attributes %% with a defined default value (default 'false').</dd> %% </dl> +%% @type xmlElement() = #xmlElement{}. +%% The record definition is found in xmerl.hrl. +%% @type xmlDocument() = #xmlDocument{}. +%% The record definition is found in xmerl.hrl. %% @type document() = xmlElement() | xmlDocument(). <p> %% The document returned by <tt>xmerl_scan:string/[1,2]</tt> and %% <tt>xmerl_scan:file/[1,2]</tt>. The type of the returned record depends on %% the value of the document option passed to the function. %% </p> - -module(xmerl_scan). -vsn('0.20'). -date('03-09-16'). diff --git a/lib/xmerl/src/xmerl_xpath.erl b/lib/xmerl/src/xmerl_xpath.erl index bbebda1030..6146feba49 100644 --- a/lib/xmerl/src/xmerl_xpath.erl +++ b/lib/xmerl/src/xmerl_xpath.erl @@ -43,13 +43,27 @@ %% </pre> %% %% @type nodeEntity() = -%% xmlElement() -%% | xmlAttribute() -%% | xmlText() -%% | xmlPI() -%% | xmlComment() -%% | xmlNsNode() -%% | xmlDocument() +%% #xmlElement{} +%% | #xmlAttribute{} +%% | #xmlText{} +%% | #xmlPI{} +%% | #xmlComment{} +%% | #xmlNsNode{} +%% | #xmlDocument{} +%% +%% @type docNodes() = #xmlElement{} +%% | #xmlAttribute{} +%% | #xmlText{} +%% | #xmlPI{} +%% | #xmlComment{} +%% | #xmlNsNode{} +%% +%% @type docEntity() = #xmlDocument{} | [docNodes()] +%% +%% @type xPathString() = string() +%% +%% @type parentList() = [{atom(), integer()}] +%% %% @type option_list(). <p>Options allows to customize the behaviour of the %% XPath scanner. %% </p> @@ -115,7 +129,7 @@ string(Str, Doc, Options) -> %% Parents = parentList() %% Doc = nodeEntity() %% Options = option_list() -%% Scalar = xmlObj +%% Scalar = #xmlObj{} %% @doc Extracts the nodes from the parsed XML tree according to XPath. %% xmlObj is a record with fields type and value, %% where type is boolean | number | string diff --git a/lib/xmerl/src/xmerl_xs.erl b/lib/xmerl/src/xmerl_xs.erl index 3e9f6622b8..1ce76cfa41 100644 --- a/lib/xmerl/src/xmerl_xs.erl +++ b/lib/xmerl/src/xmerl_xs.erl @@ -45,7 +45,6 @@ % XSLT package which is written i C++. % See also the <a href="xmerl_xs_examples.html">Tutorial</a>. % </p> - -module(xmerl_xs). -export([xslapply/2, value_of/1, select/2, built_in_rules/2 ]). @@ -71,15 +70,13 @@ %% xslapply(fun template/1, E), %% "</h1>"]; %% </pre> - xslapply(Fun, EList) when is_list(EList) -> - lists:map( Fun, EList); + lists:map(Fun, EList); xslapply(Fun, E = #xmlElement{})-> lists:map( Fun, E#xmlElement.content). - %% @spec value_of(E) -> List -%% E = unknown() +%% E = term() %% %% @doc Concatenates all text nodes within the tree. %% diff --git a/lib/xmerl/src/xmerl_xsd.erl b/lib/xmerl/src/xmerl_xsd.erl index 4b5efae8dd..a89b3159ec 100644 --- a/lib/xmerl/src/xmerl_xsd.erl +++ b/lib/xmerl/src/xmerl_xsd.erl @@ -49,6 +49,7 @@ %% <dd>It is possible by this option to provide a state with process %% information from an earlier validation.</dd> %% </dl> +%% @type filename() = string() %% @end %%%------------------------------------------------------------------- -module(xmerl_xsd). @@ -138,7 +139,7 @@ state2file(S=#xsd_state{schema_name=SN}) -> %% @spec state2file(State,FileName) -> ok | {error,Reason} %% State = global_state() -%% FileName = filename() +%% FileName = string() %% @doc Saves the schema state with all information of the processed %% schema in a file. You can provide the file name for the saved %% state. FileName is saved with the <code>.xss</code> extension @@ -153,7 +154,7 @@ state2file(S,FileName) when is_record(S,xsd_state) -> %% @spec file2state(FileName) -> {ok,State} | {error,Reason} %% State = global_state() -%% FileName = filename() +%% FileName = string() %% @doc Reads the schema state with all information of the processed %% schema from a file created with <code>state2file/[1,2]</code>. The %% format of this file is internal. The state can then be used @@ -202,7 +203,7 @@ xmerl_xsd_vsn_check(S=#xsd_state{vsn=MD5_VSN}) -> process_validate(Schema,Xml) -> process_validate(Schema,Xml,[]). %% @spec process_validate(Schema,Element,Options) -> Result -%% Schema = filename() +%% Schema = string() %% Element = XmlElement %% Options = option_list() %% Result = {ValidXmlElement,State} | {error,Reason} @@ -282,7 +283,7 @@ validate3(_,_,S) -> process_schema(Schema) -> process_schema(Schema,[]). %% @spec process_schema(Schema,Options) -> Result -%% Schema = filename() +%% Schema = string() %% Result = {ok,State} | {error,Reason} %% State = global_state() %% Reason = [ErrorReason] | ErrorReason @@ -324,7 +325,7 @@ process_schema2({SE,_},State,_Schema) -> process_schemas(Schemas) -> process_schemas(Schemas,[]). %% @spec process_schemas(Schemas,Options) -> Result -%% Schemas = [{NameSpace,filename()}|Schemas] | [] +%% Schemas = [{NameSpace,string()}|Schemas] | [] %% Result = {ok,State} | {error,Reason} %% Reason = [ErrorReason] | ErrorReason %% Options = option_list() @@ -5426,7 +5427,7 @@ add_key_once(Key,N,El,L) -> %% {filename:join([[io_lib:format("/~w(~w)",[X,Y])||{X,Y}<-Parents],Type]),Pos}. %% @spec format_error(Errors) -> Result -%% Errors = error_tuple() | [error_tuple()] +%% Errors = tuple() | [tuple()] %% Result = string() | [string()] %% @doc Formats error descriptions to human readable strings. format_error(L) when is_list(L) -> diff --git a/lib/xmerl/vsn.mk b/lib/xmerl/vsn.mk index a78a035a1f..95adaa5bb0 100644 --- a/lib/xmerl/vsn.mk +++ b/lib/xmerl/vsn.mk @@ -1 +1 @@ -XMERL_VSN = 1.3.11 +XMERL_VSN = 1.3.12 diff --git a/otp_versions.table b/otp_versions.table index 9d7d0a78bb..999c2c6d46 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,3 +1,6 @@ +OTP-19.1 : asn1-4.0.4 common_test-1.12.3 compiler-7.0.2 crypto-3.7.1 debugger-4.2.1 dialyzer-3.0.2 diameter-1.12.1 edoc-0.8 erl_docgen-0.6 erl_interface-3.9.1 erts-8.1 eunit-2.3.1 gs-1.6.2 hipe-3.15.2 ic-4.4.2 inets-6.3.3 jinterface-1.7.1 kernel-5.1 mnesia-4.14.1 observer-2.2.2 odbc-2.11.3 parsetools-2.1.3 reltool-0.7.2 runtime_tools-1.10.1 sasl-3.0.1 snmp-5.2.4 ssh-4.3.2 ssl-8.0.2 stdlib-3.1 syntax_tools-2.1 tools-2.8.6 wx-1.7.1 xmerl-1.3.12 # cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 eldap-1.2.2 et-1.6 megaco-3.18.1 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 percept-0.9 public_key-1.2 typer-0.9.11 : +OTP-19.0.7 : erts-8.0.5 # asn1-4.0.3 common_test-1.12.2 compiler-7.0.1 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7 debugger-4.2 dialyzer-3.0.1 diameter-1.12 edoc-0.7.19 eldap-1.2.2 erl_docgen-0.5 erl_interface-3.9 et-1.6 eunit-2.3 gs-1.6.1 hipe-3.15.1 ic-4.4.1 inets-6.3.2 jinterface-1.7 kernel-5.0.2 megaco-3.18.1 mnesia-4.14 observer-2.2.1 odbc-2.11.2 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.2 percept-0.9 public_key-1.2 reltool-0.7.1 runtime_tools-1.10 sasl-3.0 snmp-5.2.3 ssh-4.3.1 ssl-8.0.1 stdlib-3.0.1 syntax_tools-2.0 tools-2.8.5 typer-0.9.11 wx-1.7 xmerl-1.3.11 : +OTP-19.0.6 : erts-8.0.4 # asn1-4.0.3 common_test-1.12.2 compiler-7.0.1 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7 debugger-4.2 dialyzer-3.0.1 diameter-1.12 edoc-0.7.19 eldap-1.2.2 erl_docgen-0.5 erl_interface-3.9 et-1.6 eunit-2.3 gs-1.6.1 hipe-3.15.1 ic-4.4.1 inets-6.3.2 jinterface-1.7 kernel-5.0.2 megaco-3.18.1 mnesia-4.14 observer-2.2.1 odbc-2.11.2 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.2 percept-0.9 public_key-1.2 reltool-0.7.1 runtime_tools-1.10 sasl-3.0 snmp-5.2.3 ssh-4.3.1 ssl-8.0.1 stdlib-3.0.1 syntax_tools-2.0 tools-2.8.5 typer-0.9.11 wx-1.7 xmerl-1.3.11 : OTP-19.0.5 : kernel-5.0.2 # asn1-4.0.3 common_test-1.12.2 compiler-7.0.1 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7 debugger-4.2 dialyzer-3.0.1 diameter-1.12 edoc-0.7.19 eldap-1.2.2 erl_docgen-0.5 erl_interface-3.9 erts-8.0.3 et-1.6 eunit-2.3 gs-1.6.1 hipe-3.15.1 ic-4.4.1 inets-6.3.2 jinterface-1.7 megaco-3.18.1 mnesia-4.14 observer-2.2.1 odbc-2.11.2 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.2 percept-0.9 public_key-1.2 reltool-0.7.1 runtime_tools-1.10 sasl-3.0 snmp-5.2.3 ssh-4.3.1 ssl-8.0.1 stdlib-3.0.1 syntax_tools-2.0 tools-2.8.5 typer-0.9.11 wx-1.7 xmerl-1.3.11 : OTP-19.0.4 : erts-8.0.3 # asn1-4.0.3 common_test-1.12.2 compiler-7.0.1 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7 debugger-4.2 dialyzer-3.0.1 diameter-1.12 edoc-0.7.19 eldap-1.2.2 erl_docgen-0.5 erl_interface-3.9 et-1.6 eunit-2.3 gs-1.6.1 hipe-3.15.1 ic-4.4.1 inets-6.3.2 jinterface-1.7 kernel-5.0.1 megaco-3.18.1 mnesia-4.14 observer-2.2.1 odbc-2.11.2 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.2 percept-0.9 public_key-1.2 reltool-0.7.1 runtime_tools-1.10 sasl-3.0 snmp-5.2.3 ssh-4.3.1 ssl-8.0.1 stdlib-3.0.1 syntax_tools-2.0 tools-2.8.5 typer-0.9.11 wx-1.7 xmerl-1.3.11 : OTP-19.0.3 : inets-6.3.2 kernel-5.0.1 ssl-8.0.1 # asn1-4.0.3 common_test-1.12.2 compiler-7.0.1 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7 debugger-4.2 dialyzer-3.0.1 diameter-1.12 edoc-0.7.19 eldap-1.2.2 erl_docgen-0.5 erl_interface-3.9 erts-8.0.2 et-1.6 eunit-2.3 gs-1.6.1 hipe-3.15.1 ic-4.4.1 jinterface-1.7 megaco-3.18.1 mnesia-4.14 observer-2.2.1 odbc-2.11.2 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.2 percept-0.9 public_key-1.2 reltool-0.7.1 runtime_tools-1.10 sasl-3.0 snmp-5.2.3 ssh-4.3.1 stdlib-3.0.1 syntax_tools-2.0 tools-2.8.5 typer-0.9.11 wx-1.7 xmerl-1.3.11 : diff --git a/system/doc/tutorial/c_port.xmlsrc b/system/doc/tutorial/c_port.xmlsrc index 695f16515d..3c3bc48044 100644 --- a/system/doc/tutorial/c_port.xmlsrc +++ b/system/doc/tutorial/c_port.xmlsrc @@ -98,11 +98,11 @@ loop(Port) -> {call, Caller, Msg} -> Port ! {self(), {command, encode(Msg)}}, receive - {Port, {data, Data}} -> + {Port, {data, Data}} -> Caller ! {complex, decode(Data)} end, loop(Port) - end.</pre> + end.</pre> <p>Assuming that both the arguments and the results from the C functions are less than 256, a simple encoding/decoding scheme is employed. In this scheme, <c>foo</c> is represented by byte |