diff options
122 files changed, 2623 insertions, 2209 deletions
diff --git a/OTP_VERSION b/OTP_VERSION index 07256210c7..ab1e4f6a31 100644 --- a/OTP_VERSION +++ b/OTP_VERSION @@ -1 +1 @@ -20.0-rc1 +20.0-rc2 diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index e1aa5ce76e..638e88ca31 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -379,6 +379,16 @@ <c><![CDATA[Host]]></c> is the fully qualified host name of the current host. For short names, use flag <c><![CDATA[-sname]]></c> instead.</p> + <warning> + <p> + Starting a distributed node without also specifying + <seealso marker="#proto_dist"><c>-proto_dist inet_tls</c></seealso> + will expose the node to attacks that may give the attacker + complete access to the node and in extension the cluster. + When using un-secure distributed nodes, make sure that the + network is configured to keep potential attackers out. + </p> + </warning> </item> <tag><c><![CDATA[-noinput]]></c></tag> <item> @@ -428,12 +438,17 @@ </item> <tag><c><![CDATA[-proto_dist Proto]]></c></tag> <item> + <marker id="proto_dist"/> <p>Specifies a protocol for Erlang distribution:</p> <taglist> <tag><c>inet_tcp</c></tag> <item>TCP over IPv4 (the default)</item> <tag><c>inet_tls</c></tag> - <item>Distribution over TLS/SSL</item> + <item>Distribution over TLS/SSL, See the + <seealso marker="ssl:ssl_distribution"> + Using SSL for Erlang Distribution</seealso> User's Guide + for details on how to setup a secure distributed node. + </item> <tag><c>inet6_tcp</c></tag> <item>TCP over IPv6</item> </taglist> @@ -497,6 +512,16 @@ exist between nodes running with flag <c><![CDATA[-sname]]></c> and those running with flag <c><![CDATA[-name]]></c>, as node names must be unique in distributed Erlang systems.</p> + <warning> + <p> + Starting a distributed node without also specifying + <seealso marker="#proto_dist"><c>-proto_dist inet_tls</c></seealso> + will expose the node to attacks that may give the attacker + complete access to the node and in extension the cluster. + When using un-secure distributed nodes, make sure that the + network is configured to keep potential attackers out. + </p> + </warning> </item> <tag><marker id="start_epmd"/><c>-start_epmd true | false</c></tag> <item> @@ -1528,32 +1553,27 @@ <item> <p><em>Unix systems</em>: This variable gives the number of seconds that the emulator is allowed to spend writing a crash dump. When the - given number of seconds have elapsed, the emulator is terminated by a - <c>SIGALRM</c> signal.</p> - <p>If the variable is <em>not</em> set or set to <c>0</c> seconds - (<c><![CDATA[ERL_CRASH_DUMP_SECONDS=0]]></c>), the runtime system does - not even attempt to write the crash dump file. It only terminates.</p> - <p>If the variable is set to negative value, such as - <c><![CDATA[ERL_CRASH_DUMP_SECONDS=-1]]></c>, the runtime system - waits indefinitely for the crash dump file to be written.</p> - <p>This variable is used with <seealso marker="kernel:heart"> - <c>heart(3)</c></seealso> if <c>heart</c> is running:</p> + given number of seconds have elapsed, the emulator is terminated.</p> <taglist> <tag><c><![CDATA[ERL_CRASH_DUMP_SECONDS=0]]></c></tag> - <item>Suppresses the writing a crash dump file entirely, thus - rebooting the runtime system immediately. This is the same as not - setting the environment variable. + <item>If the variable is set to <c>0</c> seconds, the runtime system does + not even attempt to write the crash dump file. It only terminates. + This is the default if option <c>-heart</c> is passed to <c>erl</c> + and <c>ERL_CRASH_DUMP_SECONDS</c> is not set. + </item> + <tag><c><![CDATA[ERL_CRASH_DUMP_SECONDS=S]]></c></tag> + <item>If the variable is set to a positive value <c>S</c>, + wait for <c>S</c> seconds to complete the crash dump file and + then terminates the runtime system with a <c>SIGALRM</c> signal. </item> <tag><c><![CDATA[ERL_CRASH_DUMP_SECONDS=-1]]></c></tag> - <item>Setting the environment variable to a negative value causes the - termination of the runtime system to wait until the crash dump file - has been completly written. - </item> - <tag><c><![CDATA[ERL_CRASH_DUMP_SECONDS=S]]></c></tag> - <item>Waits for <c>S</c> seconds to complete the crash dump file and - then terminates the runtime system. + <item>A negative value causes the termination of the runtime system + to wait indefinitely until the crash dump file has been completly + written. This is the default if option <c>-heart</c> is <em>not</em> + passed to <c>erl</c> and <c>ERL_CRASH_DUMP_SECONDS</c> is not set. </item> </taglist> + <p>See also <seealso marker="kernel:heart"><c>heart(3)</c></seealso>.</p> </item> <tag><c><![CDATA[ERL_CRASH_DUMP_BYTES]]></c></tag> <item> diff --git a/erts/doc/src/erl_dist_protocol.xml b/erts/doc/src/erl_dist_protocol.xml index ee74983730..8391408a2e 100644 --- a/erts/doc/src/erl_dist_protocol.xml +++ b/erts/doc/src/erl_dist_protocol.xml @@ -70,6 +70,17 @@ <p>The integers in all multibyte fields are in big-endian order.</p> + <warning> + <p> + The Erlang Distribution protocol is not by itself secure and does not + aim to be so. In order to get secure distribution the distributed nodes + should be configured to use distribution over tls. + See the <seealso marker="ssl:ssl_distribution"> + Using SSL for Erlang Distribution</seealso> User's Guide + for details on how to setup a secure distributed node. + </p> + </warning> + <section> <title>EPMD Protocol</title> <p>The requests served by the EPMD are summarized in the following diff --git a/erts/doc/src/erl_tracer.xml b/erts/doc/src/erl_tracer.xml index 63feebb0b5..fd3c17f337 100644 --- a/erts/doc/src/erl_tracer.xml +++ b/erts/doc/src/erl_tracer.xml @@ -653,7 +653,7 @@ ok <0.37.0> 3> erlang:trace(new, true, [send,{tracer, erl_msg_tracer, Tracer}]). 0 -{<0.39.0>,<0.27.0>} +{trace,<0.39.0>,<0.27.0>} 4> {ok, D} = file:open("/tmp/tmp.data",[write]). {trace,#Port<0.486>,<0.40.0>} {trace,<0.40.0>,<0.21.0>} @@ -758,18 +758,21 @@ static ERL_NIF_TERM enabled(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) /* * argv[0]: TraceTag, should only be 'send' - * argv[1]: TracerState, process to send {argv[2], argv[4]} to + * argv[1]: TracerState, process to send {Tracee, Recipient} to * argv[2]: Tracee - * argv[3]: Recipient - * argv[4]: Options, ignored + * argv[3]: Message + * argv[4]: Options, map containing Recipient */ static ERL_NIF_TERM trace(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { ErlNifPid to_pid; + ERL_NIF_TERM recipient, msg; if (enif_get_local_pid(env, argv[1], &to_pid)) { - ERL_NIF_TERM msg = enif_make_tuple3(env, enif_make_atom(env, "trace"), argv[2], argv[4]); + if (enif_get_map_value(env, argv[4], enif_make_atom(env, "extra"), &recipient)) { + msg = enif_make_tuple3(env, enif_make_atom(env, "trace"), argv[2], recipient); enif_send(env, &to_pid, NULL, msg); + } } return enif_make_atom(env, "ok"); diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index 08f6732036..e61114c504 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -31,6 +31,54 @@ </header> <p>This document describes the changes made to the ERTS application.</p> +<section><title>Erts 8.3.5</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Active-mode TCP sockets are now cleaned up properly on + send/shutdown errors.</p> + <p> + Own Id: OTP-14441 Aux Id: ERL-430 </p> + </item> + <item> + <p> + A code purge operation could under certain circumstances + expand the size of hibernated processes.</p> + <p> + Own Id: OTP-14444 Aux Id: ERIERL-24 </p> + </item> + <item> + <p> + Fix so that the ERL_ZZ_SIGTERM_KILL introduced in + erts-8.3.4 works.</p> + <p> + Own Id: OTP-14451</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 8.3.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Add option to make SIGTERM trigger the OS default + behaviour instead of doing a gracefull shutdown. To + activate this bahviour the environment variable + ERL_ZZ_SIGTERM_KILL should be set to "true". This option + only works in OTP 19 as OTP 20 will have a different way + to deal with SIGTERM.</p> + <p> + Own Id: OTP-14418 Aux Id: ERIERL-15 </p> + </item> + </list> + </section> + +</section> <section><title>Erts 8.3.3</title> @@ -513,6 +561,22 @@ </section> +<section><title>Erts 8.1.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + A code purge operation could under certain circumstances + expand the size of hibernated processes.</p> + <p> + Own Id: OTP-14444 Aux Id: ERIERL-24 </p> + </item> + </list> + </section> + +</section> + <section><title>Erts 8.1.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index 254e59f2c7..61c1e14741 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -149,6 +149,8 @@ endif LIBS += $(TYPE_LIBS) +ORIG_LIBS:= $(LIBS) + comma:=, space:= space+= @@ -965,7 +967,7 @@ $(OBJDIR)/%.o: hipe/%.c $(V_CC) $(subst O2,O3, $(CFLAGS)) $(INCLUDES) -c $< -o $@ $(BINDIR)/hipe_mkliterals$(TF_MARKER): $(OBJDIR)/hipe_mkliterals.o - $(ld_verbose)$(CC) $(LDFLAGS) -o $@ $< $(TYPE_LIBS) + $(ld_verbose)$(CC) $(LDFLAGS) -o $@ $< $(ORIG_LIBS) $(OBJDIR)/hipe_mkliterals.o: $(HIPE_ASM) $(TTF_DIR)/erl_alloc_types.h $(DTRACE_HEADERS) \ $(TTF_DIR)/OPCODES-GENERATED $(TTF_DIR)/TABLES-GENERATED diff --git a/erts/emulator/beam/beam_bif_load.c b/erts/emulator/beam/beam_bif_load.c index 5aceae8ffe..007bf99b6e 100644 --- a/erts/emulator/beam/beam_bif_load.c +++ b/erts/emulator/beam/beam_bif_load.c @@ -1068,10 +1068,10 @@ return_ok: literal_gc: if (!gc_allowed) - return am_need_gc; + return am_need_gc; if (c_p->flags & F_DISABLE_GC) - return THE_NON_VALUE; + return THE_NON_VALUE; *redsp += erts_garbage_collect_literals(c_p, (Eterm *) literals, lit_bsize, oh, fcalls); diff --git a/erts/emulator/beam/bif.c b/erts/emulator/beam/bif.c index 7134d2da56..40dd4129d2 100644 --- a/erts/emulator/beam/bif.c +++ b/erts/emulator/beam/bif.c @@ -2047,8 +2047,6 @@ ebif_bang_2(BIF_ALIST_2) #define SEND_YIELD_CONTINUE (-8) -Sint do_send(Process *p, Eterm to, Eterm msg, Eterm *refp, ErtsSendContext*); - static Sint remote_send(Process *p, DistEntry *dep, Eterm to, Eterm full_to, Eterm msg, ErtsSendContext* ctx) @@ -2102,8 +2100,8 @@ static Sint remote_send(Process *p, DistEntry *dep, return res; } -Sint -do_send(Process *p, Eterm to, Eterm msg, Eterm *refp, ErtsSendContext* ctx) +static Sint +do_send(Process *p, Eterm to, Eterm msg, Eterm *refp, ErtsSendContext *ctx) { Eterm portid; Port *pt; diff --git a/erts/emulator/beam/erl_alloc_util.c b/erts/emulator/beam/erl_alloc_util.c index 4889fac923..6fddba4b34 100644 --- a/erts/emulator/beam/erl_alloc_util.c +++ b/erts/emulator/beam/erl_alloc_util.c @@ -821,7 +821,7 @@ static void clear_literal_range(void* start, Uint size) #if HAVE_ERTS_MSEG -void* +static void* erts_alcu_mseg_alloc(Allctr_t *allctr, Uint *size_p, Uint flags) { void *res; @@ -832,7 +832,7 @@ erts_alcu_mseg_alloc(Allctr_t *allctr, Uint *size_p, Uint flags) return res; } -void* +static void* erts_alcu_mseg_realloc(Allctr_t *allctr, void *seg, Uint old_size, Uint *new_size_p) { @@ -845,7 +845,7 @@ erts_alcu_mseg_realloc(Allctr_t *allctr, void *seg, return res; } -void +static void erts_alcu_mseg_dealloc(Allctr_t *allctr, void *seg, Uint size, Uint flags) { erts_mseg_dealloc_opt(allctr->alloc_no, seg, (UWord) size, flags, &allctr->mseg_opt); @@ -996,7 +996,7 @@ erts_alcu_exec_mseg_dealloc(Allctr_t *allctr, void *seg, Uint size, Uint flags) #endif /* HAVE_ERTS_MSEG */ -void* +static void* erts_alcu_sys_alloc(Allctr_t *allctr, Uint* size_p, int superalign) { void *res; @@ -1013,7 +1013,7 @@ erts_alcu_sys_alloc(Allctr_t *allctr, Uint* size_p, int superalign) return res; } -void* +static void* erts_alcu_sys_realloc(Allctr_t *allctr, void *ptr, Uint *size_p, Uint old_size, int superalign) { void *res; @@ -1035,7 +1035,7 @@ erts_alcu_sys_realloc(Allctr_t *allctr, void *ptr, Uint *size_p, Uint old_size, return res; } -void +static void erts_alcu_sys_dealloc(Allctr_t *allctr, void *ptr, Uint size, int superalign) { #if ERTS_SA_MB_CARRIERS && ERTS_HAVE_ERTS_SYS_ALIGNED_ALLOC diff --git a/erts/emulator/beam/erl_alloc_util.h b/erts/emulator/beam/erl_alloc_util.h index f570703f25..7a96bd0319 100644 --- a/erts/emulator/beam/erl_alloc_util.h +++ b/erts/emulator/beam/erl_alloc_util.h @@ -195,10 +195,6 @@ extern UWord erts_literal_vspace_map[]; # define ERTS_VSPACE_WORD_BITS (sizeof(UWord)*8) #endif -void* erts_alcu_mseg_alloc(Allctr_t*, Uint *size_p, Uint flags); -void* erts_alcu_mseg_realloc(Allctr_t*, void *seg, Uint old_size, Uint *new_size_p); -void erts_alcu_mseg_dealloc(Allctr_t*, void *seg, Uint size, Uint flags); - #if HAVE_ERTS_MSEG # if defined(ARCH_32) void* erts_alcu_literal_32_mseg_alloc(Allctr_t*, Uint *size_p, Uint flags); @@ -218,9 +214,6 @@ void erts_alcu_exec_mseg_dealloc(Allctr_t*, void *seg, Uint size, Uint flags); # endif #endif /* HAVE_ERTS_MSEG */ -void* erts_alcu_sys_alloc(Allctr_t*, Uint *size_p, int superalign); -void* erts_alcu_sys_realloc(Allctr_t*, void *ptr, Uint *size_p, Uint old_size, int superalign); -void erts_alcu_sys_dealloc(Allctr_t*, void *ptr, Uint size, int superalign); #ifdef ARCH_32 void* erts_alcu_literal_32_sys_alloc(Allctr_t*, Uint *size_p, int superalign); void* erts_alcu_literal_32_sys_realloc(Allctr_t*, void *ptr, Uint *size_p, Uint old_size, int superalign); diff --git a/erts/emulator/beam/erl_bits.c b/erts/emulator/beam/erl_bits.c index eea55b3239..71c64997c1 100644 --- a/erts/emulator/beam/erl_bits.c +++ b/erts/emulator/beam/erl_bits.c @@ -1636,7 +1636,7 @@ erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb) return LSB[0] | (LSB[1]<<8) | (LSB[2]<<16) | (LSB[3]<<24); } -void +static void erts_align_utf8_bytes(ErlBinMatchBuffer* mb, byte* buf) { Uint bits = mb->size - mb->offset; diff --git a/erts/emulator/beam/erl_bits.h b/erts/emulator/beam/erl_bits.h index 632855255e..5da2b28a89 100644 --- a/erts/emulator/beam/erl_bits.h +++ b/erts/emulator/beam/erl_bits.h @@ -202,7 +202,6 @@ void erts_new_bs_put_string(ERL_BITS_PROTO_2(byte* iptr, Uint num_bytes)); Uint erts_bits_bufs_size(void); Uint32 erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb); -void erts_align_utf8_bytes(ErlBinMatchBuffer* mb, byte* buf); Eterm erts_bs_get_utf8(ErlBinMatchBuffer* mb); Eterm erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags); Eterm erts_bs_append(Process* p, Eterm* reg, Uint live, Eterm build_size_term, diff --git a/erts/emulator/beam/erl_cpu_topology.c b/erts/emulator/beam/erl_cpu_topology.c index 4347f9f2b7..d0fd763798 100644 --- a/erts/emulator/beam/erl_cpu_topology.c +++ b/erts/emulator/beam/erl_cpu_topology.c @@ -2243,45 +2243,6 @@ add_cpu_groups(int groups, return cgm; } -static void -remove_cpu_groups(erts_cpu_groups_callback_t callback, void *arg) -{ - erts_cpu_groups_map_t *prev_cgm, *cgm; - erts_cpu_groups_callback_list_t *prev_cgcl, *cgcl; - - ERTS_SMP_LC_ASSERT(erts_lc_rwmtx_is_rwlocked(&cpuinfo_rwmtx)); - - no_cpu_groups_callbacks--; - - prev_cgm = NULL; - for (cgm = cpu_groups_maps; cgm; cgm = cgm->next) { - prev_cgcl = NULL; - for (cgcl = cgm->callback_list; cgcl; cgcl = cgcl->next) { - if (cgcl->callback == callback && cgcl->arg == arg) { - if (prev_cgcl) - prev_cgcl->next = cgcl->next; - else - cgm->callback_list = cgcl->next; - erts_free(ERTS_ALC_T_CPU_GRPS_MAP, cgcl); - if (!cgm->callback_list) { - if (prev_cgm) - prev_cgm->next = cgm->next; - else - cpu_groups_maps = cgm->next; - if (cgm->array) - erts_free(ERTS_ALC_T_CPU_GRPS_MAP, cgm->array); - erts_free(ERTS_ALC_T_CPU_GRPS_MAP, cgm); - } - return; - } - prev_cgcl = cgcl; - } - prev_cgm = cgm; - } - - erts_exit(ERTS_ABORT_EXIT, "Cpu groups not found\n"); -} - static int cpu_groups_lookup(erts_cpu_groups_map_t *map, ErtsSchedulerData *esdp) @@ -2321,21 +2282,3 @@ update_cpu_groups_maps(void) for (cgm = cpu_groups_maps; cgm; cgm = cgm->next) make_cpu_groups_map(cgm, 0); } - -void -erts_add_cpu_groups(int groups, - erts_cpu_groups_callback_t callback, - void *arg) -{ - erts_smp_rwmtx_rwlock(&cpuinfo_rwmtx); - add_cpu_groups(groups, callback, arg); - erts_smp_rwmtx_rwunlock(&cpuinfo_rwmtx); -} - -void erts_remove_cpu_groups(erts_cpu_groups_callback_t callback, - void *arg) -{ - erts_smp_rwmtx_rwlock(&cpuinfo_rwmtx); - remove_cpu_groups(callback, arg); - erts_smp_rwmtx_rwunlock(&cpuinfo_rwmtx); -} diff --git a/erts/emulator/beam/erl_db_hash.c b/erts/emulator/beam/erl_db_hash.c index 08a0f0e83b..0addfaa3c7 100644 --- a/erts/emulator/beam/erl_db_hash.c +++ b/erts/emulator/beam/erl_db_hash.c @@ -676,7 +676,7 @@ int db_create_hash(Process *p, DbTable *tbl) sizeof(DbTableHashFineLocks)); for (i=0; i<DB_HASH_LOCK_CNT; ++i) { erts_smp_rwmtx_init_opt_x(&tb->locks->lck_vec[i].lck, &rwmtx_opt, - "db_hash_slot", make_small(i)); + "db_hash_slot", tb->common.the_name); } /* This important property is needed to guarantee the two buckets * involved in a grow/shrink operation it protected by the same lock: diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index 2ff49c97b3..3c8bdaa62e 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -834,7 +834,7 @@ do_major_collection: esdp->gc_info.reclaimed += reclaimed_now; } - FLAGS(p) &= ~F_FORCE_GC; + FLAGS(p) &= ~(F_FORCE_GC|F_HIBERNATED); p->live_hf_end = ERTS_INVALID_HFRAG_PTR; ERTS_MSACC_POP_STATE_M(); @@ -891,8 +891,8 @@ erts_garbage_collect(Process* p, int need, Eterm* objv, int nobj) * Place all living data on a the new heap; deallocate any old heap. * Meant to be used by hibernate/3. */ -void -erts_garbage_collect_hibernate(Process* p) +static int +garbage_collect_hibernate(Process* p, int check_long_gc) { Uint heap_size; Eterm* heap; @@ -909,13 +909,13 @@ erts_garbage_collect_hibernate(Process* p) #ifdef ERTS_DIRTY_SCHEDULERS if (ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(p))) p->flags &= ~(F_DIRTY_GC_HIBERNATE|F_DIRTY_MAJOR_GC|F_DIRTY_MINOR_GC); - else { + else if (check_long_gc) { Uint flags = p->flags; p->flags |= F_NEED_FULLSWEEP; check_for_possibly_long_gc(p, (p->htop - p->heap) + p->mbuf_sz); if (p->flags & (F_DIRTY_MAJOR_GC|F_DIRTY_MINOR_GC)) { p->flags = flags|F_DIRTY_GC_HIBERNATE; - return; + return 1; } p->flags = flags; } @@ -1012,12 +1012,20 @@ erts_garbage_collect_hibernate(Process* p) ErtsGcQuickSanityCheck(p); + p->flags |= F_HIBERNATED; + erts_smp_atomic32_read_band_nob(&p->state, ~ERTS_PSFLG_GC); reds = gc_cost(actual_size, actual_size); - BUMP_REDS(p, reds); + return reds; } +void +erts_garbage_collect_hibernate(Process* p) +{ + int reds = garbage_collect_hibernate(p, 1); + BUMP_REDS(p, reds); +} /* * HiPE native code stack scanning procedures: @@ -1090,6 +1098,7 @@ erts_garbage_collect_literals(Process* p, Eterm* literals, Uint ygen_usage = 0; struct erl_off_heap_header** prev = NULL; Sint64 reds; + int hibernated = !!(p->flags & F_HIBERNATED); if (p->flags & (F_DISABLE_GC|F_DELAY_GC)) ERTS_INTERNAL_ERROR("GC disabled"); @@ -1104,10 +1113,13 @@ erts_garbage_collect_literals(Process* p, Eterm* literals, if (ERTS_SCHEDULER_IS_DIRTY(erts_proc_sched_data(p))) p->flags &= ~F_DIRTY_CLA; else { + Uint size = byte_lit_size/sizeof(Uint); ygen_usage = young_gen_usage(p); - check_for_possibly_long_gc(p, - (byte_lit_size/sizeof(Uint) - + 2*ygen_usage)); + if (hibernated) + size = size*2 + 3*ygen_usage; + else + size = size + 2*ygen_usage; + check_for_possibly_long_gc(p, size); if (p->flags & F_DIRTY_MAJOR_GC) { p->flags |= F_DIRTY_CLA; return 10; @@ -1274,6 +1286,12 @@ erts_garbage_collect_literals(Process* p, Eterm* literals, erts_smp_atomic32_read_band_nob(&p->state, ~ERTS_PSFLG_GC); reds += (Sint64) gc_cost((p->htop - p->heap) + byte_lit_size/sizeof(Uint), 0); + + if (hibernated) { + /* Restore the process into hibernated state... */ + reds += garbage_collect_hibernate(p, 0); + } + if (reds > INT_MAX) return INT_MAX; return (int) reds; diff --git a/erts/emulator/beam/erl_init.c b/erts/emulator/beam/erl_init.c index ac99f043e6..6172595552 100644 --- a/erts/emulator/beam/erl_init.c +++ b/erts/emulator/beam/erl_init.c @@ -431,7 +431,7 @@ erl_first_process_otp(char* modname, void* code, unsigned size, int argc, char** } static Eterm -erl_system_process_otp(Eterm parent_pid, char* modname) +erl_system_process_otp(Eterm parent_pid, char* modname, int off_heap_msgq) { Eterm start_mod; Process* parent; @@ -447,6 +447,8 @@ erl_system_process_otp(Eterm parent_pid, char* modname) parent = erts_pid2proc(NULL, 0, parent_pid, ERTS_PROC_LOCK_MAIN); so.flags = erts_default_spo_flags|SPO_SYSTEM_PROC; + if (off_heap_msgq) + so.flags |= SPO_OFF_HEAP_MSGQ; res = erl_create_process(parent, start_mod, am_start, NIL, &so); erts_smp_proc_unlock(parent, ERTS_PROC_LOCK_MAIN); return res; @@ -2326,14 +2328,14 @@ erl_start(int argc, char **argv) */ Eterm pid; - pid = erl_system_process_otp(otp_ring0_pid, "erts_code_purger"); + pid = erl_system_process_otp(otp_ring0_pid, "erts_code_purger", !0); erts_code_purger = (Process *) erts_ptab_pix2intptr_ddrb(&erts_proc, internal_pid_index(pid)); ASSERT(erts_code_purger && erts_code_purger->common.id == pid); erts_proc_inc_refc(erts_code_purger); - pid = erl_system_process_otp(otp_ring0_pid, "erts_literal_area_collector"); + pid = erl_system_process_otp(otp_ring0_pid, "erts_literal_area_collector", !0); erts_literal_area_collector = (Process *) erts_ptab_pix2intptr_ddrb(&erts_proc, internal_pid_index(pid)); @@ -2342,7 +2344,7 @@ erl_start(int argc, char **argv) erts_proc_inc_refc(erts_literal_area_collector); #ifdef ERTS_DIRTY_SCHEDULERS - pid = erl_system_process_otp(otp_ring0_pid, "erts_dirty_process_code_checker"); + pid = erl_system_process_otp(otp_ring0_pid, "erts_dirty_process_code_checker", !0); erts_dirty_process_code_checker = (Process *) erts_ptab_pix2intptr_ddrb(&erts_proc, internal_pid_index(pid)); diff --git a/erts/emulator/beam/erl_process.c b/erts/emulator/beam/erl_process.c index 7952e3031d..fc2b34e70f 100644 --- a/erts/emulator/beam/erl_process.c +++ b/erts/emulator/beam/erl_process.c @@ -11220,8 +11220,9 @@ execute_sys_tasks(Process *c_p, erts_aint32_t *statep, int in_reds) reds--; } else { - if (!minor_gc - || (!major_gc && type == ERTS_PSTT_GC_MAJOR)) { + if ((!minor_gc + || (!major_gc && type == ERTS_PSTT_GC_MAJOR)) + && !(c_p->flags & F_HIBERNATED)) { if (type == ERTS_PSTT_GC_MAJOR) { FLAGS(c_p) |= F_NEED_FULLSWEEP; } @@ -11331,7 +11332,9 @@ cleanup_sys_tasks(Process *c_p, erts_aint32_t in_state, int in_reds) erts_aint32_t state = in_state; int max_reds = in_reds; int reds = 0; - int qmask = 0; + int qmask = 1; /* Set to 1 to force looping as long as there + * are dirty tasks. + */ ERTS_SMP_LC_ASSERT(erts_proc_lc_my_proc_locks(c_p) == ERTS_PROC_LOCK_MAIN); @@ -13930,11 +13933,25 @@ erts_continue_exit_process(Process *p) erts_set_gc_state(p, 1); state = erts_smp_atomic32_read_acqb(&p->state); - if (state & ERTS_PSFLG_ACTIVE_SYS) { + if (state & ERTS_PSFLG_ACTIVE_SYS +#ifdef ERTS_DIRTY_SCHEDULERS + || p->dirty_sys_tasks +#endif + ) { if (cleanup_sys_tasks(p, state, CONTEXT_REDS) >= CONTEXT_REDS/2) goto yield; } +#ifdef DEBUG + erts_smp_proc_lock(p, ERTS_PROC_LOCK_STATUS); + ASSERT(p->sys_task_qs == NULL); + ASSERT(ERTS_PROC_GET_DELAYED_GC_TASK_QS(p) == NULL); +#ifdef ERTS_DIRTY_SCHEDULERS + ASSERT(p->dirty_sys_tasks == NULL); +#endif + erts_smp_proc_unlock(p, ERTS_PROC_LOCK_STATUS); +#endif + if (p->flags & F_USING_DDLL) { erts_ddll_proc_dead(p, ERTS_PROC_LOCK_MAIN); p->flags &= ~F_USING_DDLL; diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index d44e8c252d..9d7ba27c50 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -1422,6 +1422,7 @@ extern int erts_system_profile_ts_type; #define F_DIRTY_GC_HIBERNATE (1 << 22) /* Dirty GC hibernate scheduled */ #define F_DIRTY_MAJOR_GC (1 << 23) /* Dirty major GC scheduled */ #define F_DIRTY_MINOR_GC (1 << 24) /* Dirty minor GC scheduled */ +#define F_HIBERNATED (1 << 25) /* Hibernated */ /* * F_DISABLE_GC and F_DELAY_GC are similar. Both will prevent diff --git a/erts/emulator/drivers/common/inet_drv.c b/erts/emulator/drivers/common/inet_drv.c index bfebff5706..13ee935e45 100644 --- a/erts/emulator/drivers/common/inet_drv.c +++ b/erts/emulator/drivers/common/inet_drv.c @@ -10490,6 +10490,9 @@ static int tcp_send_or_shutdown_error(tcp_descriptor* desc, int err) set_busy_port(desc->inet.port, 0); } + tcp_clear_output(desc); + tcp_clear_input(desc); + /* * We used to handle "expected errors" differently from unexpected ones. * Now we handle all errors in the same way (unless the show_econnreset @@ -10512,8 +10515,6 @@ static int tcp_send_or_shutdown_error(tcp_descriptor* desc, int err) else desc_close(INETP(desc)); } else { - tcp_clear_output(desc); - tcp_clear_input(desc); tcp_close_check(desc); erl_inet_close(INETP(desc)); diff --git a/erts/emulator/drivers/unix/unix_efile.c b/erts/emulator/drivers/unix/unix_efile.c index 0acc2432a7..f8341f788a 100644 --- a/erts/emulator/drivers/unix/unix_efile.c +++ b/erts/emulator/drivers/unix/unix_efile.c @@ -969,17 +969,21 @@ efile_sendfile(Efile_error* errInfo, int in_fd, int out_fd, fdrec.sfv_len = SENDFILE_CHUNK_SIZE; else fdrec.sfv_len = *nbytes; + retval = sendfilev(out_fd, &fdrec, 1, &len); - /* Sometimes sendfilev can return -1 and still send data. - When that happens we just pretend that no error happend. */ - if (retval != -1 || errno == EAGAIN || errno == EINTR || - len != 0) { + if (retval == -1 && errno == EINVAL) { + /* On some solaris versions (I've seen it on SunOS 5.10), + using a sfv_len larger then a filesize will result in + a -1 && errno == EINVAL return. We translate this so + a successful send of the data.*/ + retval = len; + } + + if (retval != -1 || errno == EAGAIN || errno == EINTR) { *offset += len; *nbytes -= len; written += len; - if (errno != EAGAIN && errno != EINTR && len != 0) - retval = len; } } while (len == SENDFILE_CHUNK_SIZE); #elif defined(__DARWIN__) diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index b1bea3a960..5cf0a49972 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -1456,12 +1456,12 @@ erts_sys_main_thread(void) erts_thread_disable_fpe(); #ifdef __DARWIN__ initialize_darwin_main_thread_pipes(); -#endif +#else /* Become signal receiver thread... */ #ifdef ERTS_ENABLE_LOCK_CHECK erts_lc_set_thread_name("signal_receiver"); #endif - +#endif smp_sig_notify(0); /* Notify initialized */ /* Wait for a signal to arrive... */ diff --git a/erts/emulator/test/Makefile b/erts/emulator/test/Makefile index 2479ccc01f..fcd7244ae9 100644 --- a/erts/emulator/test/Makefile +++ b/erts/emulator/test/Makefile @@ -117,7 +117,6 @@ MODULES= \ tracer_SUITE \ tracer_test \ scheduler_SUITE \ - old_scheduler_SUITE \ port_trace_SUITE \ unique_SUITE \ z_SUITE \ diff --git a/erts/emulator/test/binary_SUITE.erl b/erts/emulator/test/binary_SUITE.erl index 355be7a36d..4d17276e5c 100644 --- a/erts/emulator/test/binary_SUITE.erl +++ b/erts/emulator/test/binary_SUITE.erl @@ -1358,17 +1358,19 @@ do_trapping(N, Bif, ArgFun) -> io:format("N=~p: Do ~p ~s gc.\n", [N, Bif, case N rem 2 of 0 -> "with"; 1 -> "without" end]), Pid = spawn(?MODULE,trapping_loop,[Bif, ArgFun, 1000, self()]), receive ok -> ok end, - receive after 100 -> ok end, Ref = make_ref(), case N rem 2 of - 0 -> erlang:garbage_collect(Pid, [{async,Ref}]), - receive after 100 -> ok end; + 0 -> + erlang:garbage_collect(Pid, [{async,Ref}]), + receive after 1 -> ok end; 1 -> void end, - exit(Pid,kill), + exit(Pid, kill), case N rem 2 of - 0 -> receive {garbage_collect, Ref, _} -> ok end; - 1 -> void + 0 -> + receive {garbage_collect, Ref, _} -> ok end; + 1 -> + void end, receive after 1 -> ok end, do_trapping(N-1, Bif, ArgFun). diff --git a/erts/emulator/test/bs_construct_SUITE.erl b/erts/emulator/test/bs_construct_SUITE.erl index 95042ac802..779d81daa5 100644 --- a/erts/emulator/test/bs_construct_SUITE.erl +++ b/erts/emulator/test/bs_construct_SUITE.erl @@ -35,7 +35,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [test1, test2, test3, test4, test5, testf, not_used, @@ -537,6 +537,8 @@ huge_binary(Config) when is_list(Config) -> ct:timetrap({seconds, 60}), 16777216 = size(<<0:(id(1 bsl 26)),(-1):(id(1 bsl 26))>>), garbage_collect(), + FreeMem = free_mem(), + io:format("Free memory (Mb): ~p\n", [FreeMem]), {Shift,Return} = case free_mem() of undefined -> %% This test has to be inlined inside the case to @@ -552,10 +554,14 @@ huge_binary(Config) when is_list(Config) -> garbage_collect(), id(<<0:((1 bsl 31)-1)>>), {31,"Limit huge binaries to 256 Mb"}; - _ -> + Mb when Mb > 200 -> garbage_collect(), id(<<0:((1 bsl 30)-1)>>), - {30,"Limit huge binary to 128 Mb"} + {30,"Limit huge binary to 128 Mb"}; + _ -> + garbage_collect(), + id(<<0:((1 bsl 29)-1)>>), + {29,"Limit huge binary to 64 Mb"} end, garbage_collect(), id(<<0:((1 bsl Shift)-1)>>), @@ -567,13 +573,14 @@ huge_binary(Config) when is_list(Config) -> Comment -> {comment, Comment} end. +%% Return the amount of free memory in Mb. free_mem() -> {ok,Apps} = application:ensure_all_started(os_mon), Mem = memsup:get_system_memory_data(), [ok = application:stop(App)||App <- Apps], case proplists:get_value(free_memory,Mem) of undefined -> undefined; - Val -> Val div 1024 + Val -> Val div (1024*1024) end. system_limit(Config) when is_list(Config) -> diff --git a/erts/emulator/test/call_trace_SUITE.erl b/erts/emulator/test/call_trace_SUITE.erl index 95171d04ce..1251d644ae 100644 --- a/erts/emulator/test/call_trace_SUITE.erl +++ b/erts/emulator/test/call_trace_SUITE.erl @@ -43,7 +43,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 30}}]. + {timetrap, {minutes, 2}}]. all() -> Common = [errors, on_load], diff --git a/erts/emulator/test/code_SUITE.erl b/erts/emulator/test/code_SUITE.erl index 35f7baf2cf..ab0fc0d42c 100644 --- a/erts/emulator/test/code_SUITE.erl +++ b/erts/emulator/test/code_SUITE.erl @@ -358,9 +358,40 @@ constant_pools(Config) when is_list(Config) -> erlang:purge_module(literals), OldHeap ! done, receive - {'EXIT',OldHeap,{A,B,C,[1,2,3|_]=Seq}} when length(Seq) =:= 16 -> - ok - end. + {'EXIT',OldHeap,{A,B,C,[1,2,3|_]=Seq}} when length(Seq) =:= 16 -> + ok + end, + + {module,literals} = erlang:load_module(literals, Code), + %% Have a hibernated process that references the literals + %% in the 'literals' module. + {Hib, Mon} = spawn_monitor(fun() -> hibernated(Self) end), + receive go -> ok end, + [{heap_size,OldHeapSz}, + {total_heap_size,OldTotHeapSz}] = process_info(Hib, [heap_size, + total_heap_size]), + OldHeapSz = OldTotHeapSz, + io:format("OldHeapSz=~p OldTotHeapSz=~p~n", [OldHeapSz, OldTotHeapSz]), + true = erlang:delete_module(literals), + false = erlang:check_process_code(Hib, literals), + erlang:check_process_code(self(), literals), + erlang:purge_module(literals), + receive after 1000 -> ok end, + [{heap_size,HeapSz}, + {total_heap_size,TotHeapSz}] = process_info(Hib, [heap_size, + total_heap_size]), + io:format("HeapSz=~p TotHeapSz=~p~n", [HeapSz, TotHeapSz]), + Hib ! hej, + receive + {'DOWN', Mon, process, Hib, Reason} -> + {undef, [{no_module, + no_function, + [{A,B,C,[1,2,3|_]=Seq}], _}]} = Reason, + 16 = length(Seq) + end, + HeapSz = TotHeapSz, %% Ensure restored to hibernated state... + true = HeapSz > OldHeapSz, + ok. no_old_heap(Parent) -> A = literals:a(), @@ -383,6 +414,13 @@ old_heap(Parent) -> exit(Res) end. +hibernated(Parent) -> + A = literals:a(), + B = literals:b(), + Res = {A,B,literals:huge_bignum(),lists:seq(1, 16)}, + Parent ! go, + erlang:hibernate(no_module, no_function, [Res]). + create_old_heap() -> case process_info(self(), [heap_size,total_heap_size]) of [{heap_size,Sz},{total_heap_size,Total}] when Sz < Total -> diff --git a/erts/emulator/test/ddll_SUITE.erl b/erts/emulator/test/ddll_SUITE.erl index 0b9f76a892..031b05790d 100644 --- a/erts/emulator/test/ddll_SUITE.erl +++ b/erts/emulator/test/ddll_SUITE.erl @@ -55,7 +55,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [ddll_test, errors, reference_count, kill_port, diff --git a/erts/emulator/test/distribution_SUITE.erl b/erts/emulator/test/distribution_SUITE.erl index 434e729310..dc7afa381b 100644 --- a/erts/emulator/test/distribution_SUITE.erl +++ b/erts/emulator/test/distribution_SUITE.erl @@ -62,8 +62,7 @@ -export([sender/3, receiver2/2, dummy_waiter/0, dead_process/0, roundtrip/1, bounce/1, do_dist_auto_connect/1, inet_rpc_server/1, dist_parallel_sender/3, dist_parallel_receiver/0, - dist_evil_parallel_receiver/0, - sendersender/4, sendersender2/4]). + dist_evil_parallel_receiver/0]). %% epmd_module exports -export([start_link/0, register_node/2, register_node/3, port_please/2]). @@ -129,52 +128,53 @@ bulk_send_small(Config) when is_list(Config) -> bulk_send_big(Config) when is_list(Config) -> bulk_send(32, 64). -bulk_send_bigbig(Config) when is_list(Config) -> - bulk_sendsend(32*5, 4). - bulk_send(Terms, BinSize) -> ct:timetrap({seconds, 30}), io:format("Sending ~w binaries, each of size ~w K", [Terms, BinSize]), {ok, Node} = start_node(bulk_receiver), Recv = spawn(Node, erlang, apply, [fun receiver/2, [0, 0]]), - Bin = list_to_binary(lists:duplicate(BinSize*1024, 253)), + Bin = binary:copy(<<253>>, BinSize*1024), Size = Terms*size(Bin), {Elapsed, {Terms, Size}} = test_server:timecall(?MODULE, sender, [Recv, Bin, Terms]), stop_node(Node), - {comment, integer_to_list(trunc(Size/1024/max(1,Elapsed)+0.5)) ++ " K/s"}. + {comment, integer_to_list(round(Size/1024/max(1,Elapsed))) ++ " K/s"}. -bulk_sendsend(Terms, BinSize) -> +sender(To, _Bin, 0) -> + To ! {done, self()}, + receive + Any -> + Any + end; +sender(To, Bin, Left) -> + To ! {term, Bin}, + sender(To, Bin, Left-1). + +bulk_send_bigbig(Config) when is_list(Config) -> + Terms = 32*5, + BinSize = 4, {Rate1, MonitorCount1} = bulk_sendsend2(Terms, BinSize, 5), {Rate2, MonitorCount2} = bulk_sendsend2(Terms, BinSize, 995), Ratio = if MonitorCount2 == 0 -> MonitorCount1 / 1.0; true -> MonitorCount1 / MonitorCount2 end, - Comment = integer_to_list(Rate1) ++ " K/s, " ++ - integer_to_list(Rate2) ++ " K/s, " ++ - integer_to_list(MonitorCount1) ++ " monitor msgs, " ++ - integer_to_list(MonitorCount2) ++ " monitor msgs, " ++ - float_to_list(Ratio) ++ " monitor ratio", - if - %% A somewhat arbitrary ratio, but hopefully one that will - %% accommodate a wide range of CPU speeds. - Ratio > 8.0 -> - {comment,Comment}; - true -> - io:put_chars(Comment), - ct:fail(ratio_too_low) - end. + Comment0 = io_lib:format("~p K/s, ~p K/s, " + "~p monitor msgs, ~p monitor msgs, " + "~.1f monitor ratio", + [Rate1,Rate2,MonitorCount1, + MonitorCount2,Ratio]), + Comment = lists:flatten(Comment0), + {comment,Comment}. bulk_sendsend2(Terms, BinSize, BusyBufSize) -> ct:timetrap({seconds, 30}), - io:format("Sending ~w binaries, each of size ~w K", + io:format("\nSending ~w binaries, each of size ~w K", [Terms, BinSize]), {ok, NodeRecv} = start_node(bulk_receiver), Recv = spawn(NodeRecv, erlang, apply, [fun receiver/2, [0, 0]]), - Bin = list_to_binary(lists:duplicate(BinSize*1024, 253)), - %%Size = Terms*size(Bin), + Bin = binary:copy(<<253>>, BinSize*1024), %% SLF LEFT OFF HERE. %% When the caller uses small hunks, like 4k via @@ -185,74 +185,62 @@ bulk_sendsend2(Terms, BinSize, BusyBufSize) -> %% default busy size and "+zdbbl 5", and if the 5 case gets %% "many many more" monitor messages, then we know we're working. - {ok, NodeSend} = start_node(bulk_sender, "+zdbbl " ++ integer_to_list(BusyBufSize)), - _Send = spawn(NodeSend, erlang, apply, [fun sendersender/4, [self(), Recv, Bin, Terms]]), + {ok, NodeSend} = start_node(bulk_sender, "+zdbbl " ++ + integer_to_list(BusyBufSize)), + _Send = spawn(NodeSend, erlang, apply, + [fun sendersender/4, [self(), Recv, Bin, Terms]]), {Elapsed, {_TermsN, SizeN}, MonitorCount} = - receive - %% On some platforms (windows), the time taken is 0 so we - %% simulate that some little time has passed. - {sendersender, {0.0,T,MC}} -> - {0.0015, T, MC}; - {sendersender, BigRes} -> - BigRes - end, + receive + %% On some platforms (Windows), the time taken is 0 so we + %% simulate that some little time has passed. + {sendersender, {0.0,T,MC}} -> + {0.0015, T, MC}; + {sendersender, BigRes} -> + BigRes + end, stop_node(NodeRecv), stop_node(NodeSend), - {trunc(SizeN/1024/Elapsed+0.5), MonitorCount}. - -sender(To, _Bin, 0) -> - To ! {done, self()}, - receive - Any -> - Any - end; -sender(To, Bin, Left) -> - To ! {term, Bin}, - sender(To, Bin, Left-1). + {round(SizeN/1024/Elapsed), MonitorCount}. %% Sender process to be run on a slave node sendersender(Parent, To, Bin, Left) -> erlang:system_monitor(self(), [busy_dist_port]), - [spawn(fun() -> sendersender2(To, Bin, Left, false) end) || - _ <- lists:seq(1,1)], + _ = spawn(fun() -> + sendersender_send(To, Bin, Left), + exit(normal) + end), {USec, {Res, MonitorCount}} = - timer:tc(?MODULE, sendersender2, [To, Bin, Left, true]), + timer:tc(fun() -> + sendersender_send(To, Bin, Left), + To ! {done, self()}, + count_monitors(0) + end), Parent ! {sendersender, {USec/1000000, Res, MonitorCount}}. -sendersender2(To, Bin, Left, SendDone) -> - sendersender3(To, Bin, Left, SendDone, 0). +sendersender_send(_To, _Bin, 0) -> + ok; +sendersender_send(To, Bin, Left) -> + To ! {term, Bin}, + sendersender_send(To, Bin, Left-1). -sendersender3(To, _Bin, 0, SendDone, MonitorCount) -> - if SendDone -> - To ! {done, self()}; - true -> - ok - end, +count_monitors(MonitorCount) -> receive {monitor, _Pid, _Type, _Info} -> - sendersender3(To, _Bin, 0, SendDone, MonitorCount + 1) + count_monitors(MonitorCount + 1) after 0 -> - if SendDone -> - receive - Any when is_tuple(Any), size(Any) == 2 -> - {Any, MonitorCount} - end; - true -> - exit(normal) - end - end; -sendersender3(To, Bin, Left, SendDone, MonitorCount) -> - To ! {term, Bin}, - %%timer:sleep(50), - sendersender3(To, Bin, Left-1, SendDone, MonitorCount). + receive + {_,_}=Any -> + {Any,MonitorCount} + end + end. %% Receiver process to be run on a slave node. receiver(Terms, Size) -> receive {term, Bin} -> - receiver(Terms+1, Size+size(Bin)); + receiver(Terms+1, Size+byte_size(Bin)); {done, ReplyTo} -> ReplyTo ! {Terms, Size} end. diff --git a/erts/emulator/test/evil_SUITE.erl b/erts/emulator/test/evil_SUITE.erl index 9416ac7a02..fb1954ce37 100644 --- a/erts/emulator/test/evil_SUITE.erl +++ b/erts/emulator/test/evil_SUITE.erl @@ -34,7 +34,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 30}}]. + {timetrap, {minutes, 1}}]. all() -> [heap_frag, encode_decode_ext, decode_integer_ext, diff --git a/erts/emulator/test/exception_SUITE.erl b/erts/emulator/test/exception_SUITE.erl index 76e3556bc4..ad6d8c890f 100644 --- a/erts/emulator/test/exception_SUITE.erl +++ b/erts/emulator/test/exception_SUITE.erl @@ -33,7 +33,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [badmatch, pending_errors, nil_arith, stacktrace, diff --git a/erts/emulator/test/gc_SUITE.erl b/erts/emulator/test/gc_SUITE.erl index 35dd147550..2c2cb9c32d 100644 --- a/erts/emulator/test/gc_SUITE.erl +++ b/erts/emulator/test/gc_SUITE.erl @@ -203,89 +203,90 @@ long_receive() -> end. minor_major_gc_option_self(_Config) -> - Endless = fun Endless() -> - receive - {gc, Type} -> erlang:garbage_collect(self(), [{type, Type}]) - after 100 -> ok end, - Endless() - end, - - %% Try as major, a test process will self-trigger GC - P1 = spawn(Endless), - erlang:garbage_collect(P1, []), - erlang:trace(P1, true, [garbage_collection]), - P1 ! {gc, major}, - expect_trace_messages(P1, [gc_major_start, gc_major_end]), - erlang:trace(P1, false, [garbage_collection]), - erlang:exit(P1, kill), - - %% Try as minor, a test process will self-trigger GC - P2 = spawn(Endless), - erlang:garbage_collect(P2, []), - erlang:trace(P2, true, [garbage_collection]), - P2 ! {gc, minor}, - expect_trace_messages(P2, [gc_minor_start, gc_minor_end]), - erlang:trace(P2, false, [garbage_collection]), - erlang:exit(P2, kill). + %% Try as major, the test process will self-trigger GC + check_gc_tracing_around( + fun(Pid, Ref) -> + Pid ! {gc, Ref, major} + end, [gc_major_start, gc_major_end]), + + %% Try as minor, the test process will self-trigger GC + check_gc_tracing_around( + fun(Pid, Ref) -> + Pid ! {gc, Ref, minor} + end, [gc_minor_start, gc_minor_end]). minor_major_gc_option_async(_Config) -> - Endless = fun Endless() -> - receive after 100 -> ok end, - Endless() - end, - - %% Try with default option, must be major gc - P1 = spawn(Endless), - erlang:garbage_collect(P1, []), - erlang:trace(P1, true, [garbage_collection]), - erlang:garbage_collect(P1, []), - expect_trace_messages(P1, [gc_major_start, gc_major_end]), - erlang:trace(P1, false, [garbage_collection]), - erlang:exit(P1, kill), + %% Try with default option, must be major GC + check_gc_tracing_around( + fun(Pid, _Ref) -> + erlang:garbage_collect(Pid, []) + end, [gc_major_start, gc_major_end]), %% Try with the 'major' type - P2 = spawn(Endless), - erlang:garbage_collect(P2, []), - erlang:trace(P2, true, [garbage_collection]), - erlang:garbage_collect(P2, [{type, major}]), - expect_trace_messages(P2, [gc_major_start, gc_major_end]), - erlang:trace(P2, false, [garbage_collection]), - erlang:exit(P2, kill), + check_gc_tracing_around( + fun(Pid, _Ref) -> + erlang:garbage_collect(Pid, [{type, major}]) + end, [gc_major_start, gc_major_end]), %% Try with 'minor' option, once - P3 = spawn(Endless), - erlang:garbage_collect(P3, []), - erlang:trace(P3, true, [garbage_collection]), - erlang:garbage_collect(P3, [{type, minor}]), - expect_trace_messages(P3, [gc_minor_start, gc_minor_end]), - erlang:trace(P3, false, [garbage_collection]), - erlang:exit(P3, kill), + check_gc_tracing_around( + fun(Pid, _Ref) -> + erlang:garbage_collect(Pid, [{type, minor}]) + end, [gc_minor_start, gc_minor_end]), %% Try with 'minor' option, once, async - P4 = spawn(Endless), + check_gc_tracing_around( + fun(Pid, Ref) -> + ?assertEqual(async, + erlang:garbage_collect(Pid, [{type, minor}, {async, Ref}])), + + receive + {garbage_collect, Ref, true} -> + ok + after 10000 -> + ct:fail("Did not receive a completion notification on async GC") + end + end, [gc_minor_start, gc_minor_end]). + +%% Traces garbage collection around the given operation, and fails the test if +%% it results in any unexpected messages or if the expected trace tags are not +%% received. +check_gc_tracing_around(Fun, ExpectedTraceTags) -> Ref = erlang:make_ref(), - erlang:garbage_collect(P4, []), - erlang:trace(P4, true, [garbage_collection]), - ?assertEqual(async, - erlang:garbage_collect(P4, [{type, minor}, {async, Ref}])), - expect_trace_messages(P4, [gc_minor_start, gc_minor_end]), - erlang:trace(P4, false, [garbage_collection]), - receive {garbage_collect, Ref, true} -> ok; - Other4 -> ct:pal("Unexpected message: ~p~n" - ++ "while waiting for async gc result", [Other4]) - after 2000 -> ?assert(false) - end, - erlang:exit(P4, kill). - -%% Given a list of atoms, trace tags - receives messages and checks if they are -%% trace events, and if the tag matches. Else will crash failing the test. -expect_trace_messages(_Pid, []) -> ok; + Pid = spawn( + fun Endless() -> + receive + {gc, Ref, Type} -> + erlang:garbage_collect(self(), [{type, Type}]) + after 100 -> + ok + end, + Endless() + end), + erlang:garbage_collect(Pid, []), + erlang:trace(Pid, true, [garbage_collection]), + Fun(Pid, Ref), + expect_trace_messages(Pid, ExpectedTraceTags), + erlang:trace(Pid, false, [garbage_collection]), + erlang:exit(Pid, kill), + check_no_unexpected_messages(). + +%% Ensures that trace messages with the provided tags have all been received +%% within a reasonable timeframe. +expect_trace_messages(_Pid, []) -> + ok; expect_trace_messages(Pid, [Tag | TraceTags]) -> receive - {trace, Pid, Tag, _Data} -> ok; - AnythingElse -> - ct:pal("Unexpected message: ~p~nWhile expected {trace, _, ~p, _}", - [AnythingElse, Tag]), - ?assert(false) - end, - expect_trace_messages(Pid, TraceTags). + {trace, Pid, Tag, _Data} -> + expect_trace_messages(Pid, TraceTags) + after 4000 -> + ct:fail("Didn't receive tag ~p within 4000ms", [Tag]) + end. + +check_no_unexpected_messages() -> + receive + Anything -> + ct:fail("Unexpected message: ~p", [Anything]) + after 0 -> + ok + end. diff --git a/erts/emulator/test/lttng_SUITE.erl b/erts/emulator/test/lttng_SUITE.erl index c12f63706a..1c1952f912 100644 --- a/erts/emulator/test/lttng_SUITE.erl +++ b/erts/emulator/test/lttng_SUITE.erl @@ -40,7 +40,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [t_lttng_list, diff --git a/erts/emulator/test/match_spec_SUITE.erl b/erts/emulator/test/match_spec_SUITE.erl index eb189c2c33..92ddc23592 100644 --- a/erts/emulator/test/match_spec_SUITE.erl +++ b/erts/emulator/test/match_spec_SUITE.erl @@ -42,7 +42,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 30}}]. + {timetrap, {minutes, 1}}]. all() -> case test_server:is_native(match_spec_SUITE) of diff --git a/erts/emulator/test/nested_SUITE.erl b/erts/emulator/test/nested_SUITE.erl index 7af2873ce2..5059317172 100644 --- a/erts/emulator/test/nested_SUITE.erl +++ b/erts/emulator/test/nested_SUITE.erl @@ -27,7 +27,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [case_in_case, case_in_after, catch_in_catch, diff --git a/erts/emulator/test/nif_SUITE.erl b/erts/emulator/test/nif_SUITE.erl index c8bd9f81f8..05c250125d 100644 --- a/erts/emulator/test/nif_SUITE.erl +++ b/erts/emulator/test/nif_SUITE.erl @@ -496,7 +496,7 @@ select(Config) when is_list(Config) -> %% Wait for read eagain = read_nif(R, 3), 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,null,Ref), - [] = flush(), + [] = flush(0), ok = write_nif(W, <<"hej">>), [{select, R, Ref, ready_input}] = flush(), 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,self(),Ref2), @@ -513,7 +513,7 @@ select(Config) when is_list(Config) -> %% Wait for write Written = write_full(W, $a), 0 = select_nif(W,?ERL_NIF_SELECT_WRITE,W,self(),Ref), - [] = flush(), + [] = flush(0), Written = read_nif(R,byte_size(Written)), [{select, W, Ref, ready_output}] = flush(), @@ -523,7 +523,7 @@ select(Config) when is_list(Config) -> [{fd_resource_stop, W_ptr, _}] = flush(), {1, {W_ptr,_}} = last_fd_stop_call(), true = is_closed_nif(W), - [] = flush(), + [] = flush(0), 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,self(),Ref), [{select, R, Ref, ready_input}] = flush(), eof = read_nif(R,1), @@ -548,7 +548,7 @@ select_2(Config) -> 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,null,Ref1), 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,self(),Ref2), - [] = flush(), + [] = flush(0), ok = write_nif(W, <<"hej">>), [{select, R, Ref2, ready_input}] = flush(), <<"hej">> = read_nif(R, 3), @@ -559,7 +559,7 @@ select_2(Config) -> Papa = self(), spawn_link(fun() -> 0 = select_nif(R,?ERL_NIF_SELECT_READ,R,null,Ref1), - [] = flush(), + [] = flush(0), Papa ! sync, [{select, R, Ref1, ready_input}] = flush(), <<"hej">> = read_nif(R, 3), @@ -568,7 +568,7 @@ select_2(Config) -> sync = receive_any(), ok = write_nif(W, <<"hej">>), done = receive_any(), - [] = flush(), + [] = flush(0), check_stop_ret(select_nif(R,?ERL_NIF_SELECT_STOP,R,null,Ref1)), [{fd_resource_stop, R_ptr, _}] = flush(), @@ -637,6 +637,15 @@ monitor_process_a(Config) -> monitor_process_b(Config) -> ensure_lib_loaded(Config), + monitor_process_b_do(false), + case erlang:system_info(threads) of + true -> monitor_process_b_do(true); + false -> ok + end, + ok. + + +monitor_process_b_do(FromThread) -> Pid = spawn_link(fun() -> receive return -> ok @@ -645,8 +654,11 @@ monitor_process_b(Config) -> R_ptr = alloc_monitor_resource_nif(), {0,_} = monitor_process_nif(R_ptr, Pid, true, self()), [R_ptr] = monitored_by(Pid), - ok = release_resource(R_ptr), - [] = flush(), + case FromThread of + false -> ok = release_resource(R_ptr); + true -> ok = release_resource_from_thread(R_ptr) + end, + [] = flush(0), {R_ptr, _, 1} = last_resource_dtor_call(), [] = monitored_by(Pid), Pid ! return, @@ -668,7 +680,7 @@ monitor_process_c(Config) -> exit end), [{Pid, done, R_ptr, Mon1}, - {monitor_resource_down, R_ptr, Pid, Mon2}] = flush(), + {monitor_resource_down, R_ptr, Pid, Mon2}] = flush(2), compare_monitors_nif(Mon1, Mon2), {R_ptr, _, 1} = last_resource_dtor_call(), ok. @@ -716,7 +728,7 @@ demonitor_process(Config) -> 1 = demonitor_process_nif(R_ptr, MonBin2), ok = release_resource(R_ptr), - [] = flush(), + [] = flush(0), {R_ptr, _, 1} = last_resource_dtor_call(), [] = monitored_by(Pid), Pid ! return, @@ -2315,10 +2327,16 @@ receive_any(Timeout) -> after Timeout -> timeout end. flush() -> - flush(10). -flush(Timeout) -> + flush(1). + +flush(0) -> + flush(0, 10); % don't waste too much time waiting for nothing +flush(N) -> + flush(N, 1000). + +flush(N, Timeout) -> receive M -> - [M | flush(Timeout)] + [M | flush(N-1)] after Timeout -> [] end. @@ -2627,9 +2645,9 @@ nif_snprintf(Config) -> nif_internal_hash(Config) -> ensure_lib_loaded(Config), HashValueBitSize = nif_hash_result_bitsize(internal), - Terms = unique([random_term() || _ <- lists:seq(1, 5000)]), + Terms = unique([random_term() || _ <- lists:seq(1, 500)]), HashValues = [hash_nif(internal, Term, 0) || Term <- Terms], - test_bit_distribution_fitness(HashValues, HashValueBitSize, 0.05). + test_bit_distribution_fitness(HashValues, HashValueBitSize). nif_internal_hash_salted(Config) -> ensure_lib_loaded(Config), @@ -2638,7 +2656,7 @@ nif_internal_hash_salted(Config) -> nif_phash2(Config) -> ensure_lib_loaded(Config), HashValueBitSize = nif_hash_result_bitsize(phash2), - Terms = unique([random_term() || _ <- lists:seq(1, 5000)]), + Terms = unique([random_term() || _ <- lists:seq(1, 500)]), HashValues = lists:map( fun (Term) -> @@ -2651,12 +2669,12 @@ nif_phash2(Config) -> HashValue end, Terms), - test_bit_distribution_fitness(HashValues, HashValueBitSize, 0.05). + test_bit_distribution_fitness(HashValues, HashValueBitSize). test_salted_nif_hash(HashType) -> HashValueBitSize = nif_hash_result_bitsize(HashType), - Terms = unique([random_term() || _ <- lists:seq(1, 5000)]), - Salts = unique([random_uint32() || _ <- lists:seq(1, 100)]), + Terms = unique([random_term() || _ <- lists:seq(1, 500)]), + Salts = unique([random_uint32() || _ <- lists:seq(1, 50)]), {HashValuesPerSalt, HashValuesPerTerm} = lists:mapfoldl( fun (Salt, Acc) -> @@ -2677,22 +2695,20 @@ test_salted_nif_hash(HashType) -> % Test per-salt hash distribution of different terms lists:foreach( fun ({_Salt, HashValues}) -> - test_bit_distribution_fitness(HashValues, HashValueBitSize, 0.05) + test_bit_distribution_fitness(HashValues, HashValueBitSize) end, HashValuesPerSalt), % Test per-term hash distribution of different salts dict:fold( fun (_Term, HashValues, Acc) -> - % Be more tolerant of relative deviation, - % as there's fewer hash values here. - test_bit_distribution_fitness(HashValues, HashValueBitSize, 0.30), + test_bit_distribution_fitness(HashValues, HashValueBitSize), Acc end, ok, HashValuesPerTerm). -test_bit_distribution_fitness(Integers, BitSize, MaxRelativeDeviation) -> +test_bit_distribution_fitness(Integers, BitSize) -> MaxInteger = (1 bsl BitSize) - 1, OnesPerBit = lists:foldl( @@ -2708,19 +2724,29 @@ test_bit_distribution_fitness(Integers, BitSize, MaxRelativeDeviation) -> orddict:new(), Integers), - ExpectedNrOfOnes = length(Integers) div 2, + N = length(Integers), + ExpectedNrOfOnes = N div 2, + %% ExpectedNrOfOnes should have a binomial distribution + %% with a standard deviation as: + ExpectedStdDev = math:sqrt(N) / 2, + %% which can be approximated as a normal distribution + %% where we allow a deviation of 6 std.devs + %% for a fail probability of 0.000000002: + MaxStdDevs = 6, + FailureText = orddict:fold( fun (BitIndex, NrOfOnes, Acc) -> - RelativeDeviation = abs(NrOfOnes - ExpectedNrOfOnes) / length(Integers), - case RelativeDeviation >= MaxRelativeDeviation of - false -> Acc; + Deviation = abs(NrOfOnes - ExpectedNrOfOnes) / ExpectedStdDev, + case Deviation >= MaxStdDevs of + false -> + Acc; true -> [Acc, io_lib:format( "Unreasonable deviation on number of set bits (i=~p): " - "expected ~p, got ~p (relative dev. ~.3f)~n", - [BitIndex, ExpectedNrOfOnes, NrOfOnes, RelativeDeviation])] + "expected ~p, got ~p (# std.dev ~.3f > ~p)~n", + [BitIndex, ExpectedNrOfOnes, NrOfOnes, Deviation, MaxStdDevs])] end end, [], @@ -2952,6 +2978,7 @@ alloc_resource(_,_) -> ?nif_stub. make_resource(_) -> ?nif_stub. get_resource(_,_) -> ?nif_stub. release_resource(_) -> ?nif_stub. +release_resource_from_thread(_) -> ?nif_stub. last_resource_dtor_call() -> ?nif_stub. make_new_resource(_,_) -> ?nif_stub. check_is(_,_,_,_,_,_,_,_,_,_,_) -> ?nif_stub. diff --git a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c index 003a0f2929..307d1c390f 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c +++ b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c @@ -997,6 +997,30 @@ static ERL_NIF_TERM release_resource(ErlNifEnv* env, int argc, const ERL_NIF_TER return enif_make_atom(env,"ok"); } +static void* threaded_release_resource(void* resource) +{ + enif_release_resource(resource); +} + +static ERL_NIF_TERM release_resource_from_thread(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + void* resource; + ErlNifTid tid; + int err; + + if (!get_pointer(env, argv[0], &resource)) { + return enif_make_badarg(env); + } + if (enif_thread_create("nif_SUITE:release_resource_from_thread", &tid, + threaded_release_resource, resource, NULL) != 0) { + return enif_make_badarg(env); + } + err = enif_thread_join(tid, NULL); + assert(err == 0); + return atom_ok; +} + + /* * argv[0] an atom * argv[1] a binary @@ -2793,6 +2817,7 @@ static ERL_NIF_TERM monitor_frenzy_nif(ErlNifEnv* env, int argc, const ERL_NIF_T static unsigned long spawn_cnt = 0; static unsigned long kill_cnt = 0; static unsigned long proc_histogram[FRENZY_PROCS_MAX]; + static int initialized = 0; static const unsigned int primes[] = {7, 13, 17, 19}; @@ -2812,7 +2837,7 @@ static ERL_NIF_TERM monitor_frenzy_nif(ErlNifEnv* env, int argc, const ERL_NIF_T if (enif_is_atom(env, Op)) { if (Op == atom_init) { - if (procs_lock || !enif_get_uint(env, Rnd, &frenzy_rand_bits_max)) + if (initialized || !enif_get_uint(env, Rnd, &frenzy_rand_bits_max)) return enif_make_badarg(env); procs_lock = enif_mutex_create("nif_SUITE:monitor_frenzy.procs"); @@ -2839,6 +2864,7 @@ static ERL_NIF_TERM monitor_frenzy_nif(ErlNifEnv* env, int argc, const ERL_NIF_T spawn_cnt = 1; kill_cnt = 0; + initialized = 1; return enif_make_uint(env, 0); /* SelfPix */ } else if (Op == atom_stats) { @@ -2869,7 +2895,7 @@ static ERL_NIF_TERM monitor_frenzy_nif(ErlNifEnv* env, int argc, const ERL_NIF_T enif_make_ulong(env, res_dtor_cnt))); } - else if (Op == atom_stop && procs_lock) { /* stop all */ + else if (Op == atom_stop && initialized) { /* stop all */ /* Release all resources */ for (rix = 0; rix < FRENZY_RESOURCES_MAX; rix++) { @@ -3159,6 +3185,7 @@ static ErlNifFunc nif_funcs[] = {"make_resource", 1, make_resource}, {"get_resource", 2, get_resource}, {"release_resource", 1, release_resource}, + {"release_resource_from_thread", 1, release_resource_from_thread}, {"last_resource_dtor_call", 0, last_resource_dtor_call}, {"make_new_resource", 2, make_new_resource}, {"check_is", 11, check_is}, diff --git a/erts/emulator/test/old_scheduler_SUITE.erl b/erts/emulator/test/old_scheduler_SUITE.erl deleted file mode 100644 index 8515a87df8..0000000000 --- a/erts/emulator/test/old_scheduler_SUITE.erl +++ /dev/null @@ -1,384 +0,0 @@ -%% -%% %CopyrightBegin% -%% -%% Copyright Ericsson AB 2004-2016. All Rights Reserved. -%% -%% Licensed under the Apache License, Version 2.0 (the "License"); -%% you may not use this file except in compliance with the License. -%% You may obtain a copy of the License at -%% -%% http://www.apache.org/licenses/LICENSE-2.0 -%% -%% Unless required by applicable law or agreed to in writing, software -%% distributed under the License is distributed on an "AS IS" BASIS, -%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -%% See the License for the specific language governing permissions and -%% limitations under the License. -%% -%% %CopyrightEnd% -%% - --module(old_scheduler_SUITE). - --include_lib("common_test/include/ct.hrl"). - --export([all/0, suite/0, - init_per_testcase/2, end_per_testcase/2]). --export([equal/1, many_low/1, few_low/1, max/1, high/1]). - -suite() -> - [{ct_hooks,[ts_install_cth]}, - {timetrap, {minutes, 11}}]. - -all() -> - case catch erlang:system_info(modified_timing_level) of - Level when is_integer(Level) -> - {skipped, - "Modified timing (level " ++ - integer_to_list(Level) ++ - ") is enabled. Testcases gets messed " - "up by modfied timing."}; - _ -> [equal, many_low, few_low, max, high] - end. - - -%%----------------------------------------------------------------------------------- -%% TEST SUITE DESCRIPTION -%% -%% The test case function spawns two controlling processes: Starter and Receiver. -%% Starter spawns a number of prio A and a number of prio B test processes. Each -%% test process loops for a number of times, sends a report to the Receiver, then -%% loops again. For each report, the Receiver increases a counter that corresponds -%% to the priority of the sender. After a certain amount of time, the Receiver -%% sends the collected data to the main test process and waits for the test case -%% to terminate. From this data, it's possible to calculate the average run time -%% relationship between the prio A and B test processes. -%% -%% Note that in order to be able to run tests with high or max prio test processes, -%% the main test process and the Receiver needs to run at max prio, or they will -%% be starved by the test processes. The controlling processes must not wait for -%% messages from a normal (or low) prio process while max or high prio test processes -%% are running (which happens e.g. if an io function is called). -%%----------------------------------------------------------------------------------- - -init_per_testcase(_Case, Config) -> - %% main test process needs max prio - Prio = process_flag(priority, max), - MS = erlang:system_flag(multi_scheduling, block_normal), - [{prio,Prio},{multi_scheduling, MS}|Config]. - -end_per_testcase(_Case, Config) -> - erlang:system_flag(multi_scheduling, unblock_normal), - Prio=proplists:get_value(prio, Config), - process_flag(priority, Prio), - ok. - -ok(Config) when is_list(Config) -> - case proplists:get_value(multi_scheduling, Config) of - blocked -> - {comment, - "Multi-scheduling blocked during test. This testcase was not " - "written to work with multiple schedulers."}; - _ -> ok - end. - -%% Run equal number of low and normal prio processes. - -equal(Config) when is_list(Config) -> - Self = self(), - - %% specify number of test processes to run - Normal = {normal,500}, - Low = {low,500}, - - %% specify time of test (in seconds) - Time = 30, - - %% start controllers - Receiver = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Normal, Low) end), - Starter = - spawn(fun() -> starter(Normal, Low, Receiver) end), - - %% receive test data from Receiver - {NRs,NAvg,LRs,LAvg,Ratio} = - receive - {Receiver,Res} -> Res - end, - - %% stop controllers and test processes - exit(Starter, kill), - exit(Receiver, kill), - - io:format("Reports: ~w normal (~w/proc), ~w low (~w/proc). Ratio: ~w~n", - [NRs,NAvg,LRs,LAvg,Ratio]), - - %% runtime ratio between normal and low should be ~8 - if Ratio < 7.5 ; Ratio > 8.5 -> - ct:fail({bad_ratio,Ratio}); - true -> - ok(Config) - end. - - -%% Run many low and few normal prio processes. - -many_low(Config) when is_list(Config) -> - Self = self(), - Normal = {normal,1}, - Low = {low,1000}, - - %% specify time of test (in seconds) - Time = 30, - - Receiver = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Normal, Low) end), - Starter = - spawn(fun() -> starter(Normal, Low, Receiver) end), - {NRs,NAvg,LRs,LAvg,Ratio} = - receive - {Receiver,Res} -> Res - end, - exit(Starter, kill), - exit(Receiver, kill), - io:format("Reports: ~w normal (~w/proc), ~w low (~w/proc). Ratio: ~w~n", - [NRs,NAvg,LRs,LAvg,Ratio]), - if Ratio < 7.5 ; Ratio > 8.5 -> - ct:fail({bad_ratio,Ratio}); - true -> - ok(Config) - end. - - -%% Run few low and many normal prio processes. - -few_low(Config) when is_list(Config) -> - Self = self(), - Normal = {normal,1000}, - Low = {low,1}, - - %% specify time of test (in seconds) - Time = 30, - - Receiver = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Normal, Low) end), - Starter = - spawn(fun() -> starter(Normal, Low, Receiver) end), - {NRs,NAvg,LRs,LAvg,Ratio} = - receive - {Receiver,Res} -> Res - end, - exit(Starter, kill), - exit(Receiver, kill), - io:format("Reports: ~w normal (~w/proc), ~w low (~w/proc). Ratio: ~w~n", - [NRs,NAvg,LRs,LAvg,Ratio]), - if Ratio < 7.0 ; Ratio > 8.5 -> - ct:fail({bad_ratio,Ratio}); - true -> - ok(Config) - end. - - -%% Run max prio processes and verify they get at least as much -%% runtime as high, normal and low. - -max(Config) when is_list(Config) -> - max = process_flag(priority, max), % should already be max (init_per_tc) - Self = self(), - Max = {max,2}, - High = {high,2}, - Normal = {normal,100}, - Low = {low,100}, - - %% specify time of test (in seconds) - Time = 30, - - Receiver1 = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Max, High) end), - Starter1 = - spawn(fun() -> starter(Max, High, Receiver1) end), - {M1Rs,M1Avg,HRs,HAvg,Ratio1} = - receive - {Receiver1,Res1} -> Res1 - end, - exit(Starter1, kill), - exit(Receiver1, kill), - io:format("Reports: ~w max (~w/proc), ~w high (~w/proc). Ratio: ~w~n", - [M1Rs,M1Avg,HRs,HAvg,Ratio1]), - if Ratio1 < 1.0 -> - ct:fail({bad_ratio,Ratio1}); - true -> - ok(Config) - end, - - Receiver2 = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Max, Normal) end), - Starter2 = - spawn(fun() -> starter(Max, Normal, Receiver2) end), - {M2Rs,M2Avg,NRs,NAvg,Ratio2} = - receive - {Receiver2,Res2} -> Res2 - end, - exit(Starter2, kill), - exit(Receiver2, kill), - io:format("Reports: ~w max (~w/proc), ~w normal (~w/proc). Ratio: ~w~n", - [M2Rs,M2Avg,NRs,NAvg,Ratio2]), - if Ratio2 < 1.0 -> - ct:fail({bad_ratio,Ratio2}); - true -> - ok - end, - - Receiver3 = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, Max, Low) end), - Starter3 = - spawn(fun() -> starter(Max, Low, Receiver3) end), - {M3Rs,M3Avg,LRs,LAvg,Ratio3} = - receive - {Receiver3,Res3} -> Res3 - end, - exit(Starter3, kill), - exit(Receiver3, kill), - io:format("Reports: ~w max (~w/proc), ~w low (~w/proc). Ratio: ~w~n", - [M3Rs,M3Avg,LRs,LAvg,Ratio3]), - if Ratio3 < 1.0 -> - ct:fail({bad_ratio,Ratio3}); - true -> - ok(Config) - end. - - -%% Run high prio processes and verify they get at least as much -%% runtime as normal and low. - -high(Config) when is_list(Config) -> - max = process_flag(priority, max), % should already be max (init_per_tc) - Self = self(), - High = {high,2}, - Normal = {normal,100}, - Low = {low,100}, - - %% specify time of test (in seconds) - Time = 30, - - Receiver1 = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, High, Normal) end), - Starter1 = - spawn(fun() -> starter(High, Normal, Receiver1) end), - {H1Rs,H1Avg,NRs,NAvg,Ratio1} = - receive - {Receiver1,Res1} -> Res1 - end, - exit(Starter1, kill), - exit(Receiver1, kill), - io:format("Reports: ~w high (~w/proc), ~w normal (~w/proc). Ratio: ~w~n", - [H1Rs,H1Avg,NRs,NAvg,Ratio1]), - if Ratio1 < 1.0 -> - ct:fail({bad_ratio,Ratio1}); - true -> - ok - end, - - Receiver2 = - spawn(fun() -> receiver(erlang:monotonic_time(), Time, Self, High, Low) end), - Starter2 = - spawn(fun() -> starter(High, Low, Receiver2) end), - {H2Rs,H2Avg,LRs,LAvg,Ratio2} = - receive - {Receiver2,Res2} -> Res2 - end, - exit(Starter2, kill), - exit(Receiver2, kill), - io:format("Reports: ~w high (~w/proc), ~w low (~w/proc). Ratio: ~w~n", - [H2Rs,H2Avg,LRs,LAvg,Ratio2]), - if Ratio2 < 1.0 -> - ct:fail({bad_ratio,Ratio2}); - true -> - ok(Config) - end. - - -%%----------------------------------------------------------------------------------- -%% Controller processes and help functions -%%----------------------------------------------------------------------------------- - -receiver(T0, TimeSec, Main, {P1,P1N}, {P2,P2N}) -> - %% prio should be max so that mailbox doesn't overflow - process_flag(priority, max), - receiver(T0, TimeSec*1000, Main, P1,P1N,0, P2,P2N,0, 100000). - -%% uncomment lines below to get life sign (debug) -receiver(T0, Time, Main, P1,P1N,P1Rs, P2,P2N,P2Rs, 0) -> - % T = erlang:convert_time_unit(erlang:monotonic_time() - T0, native, millisecond), - % erlang:display({round(T/1000),P1Rs,P2Rs}), - receiver(T0, Time, Main, P1,P1N,P1Rs, P2,P2N,P2Rs, 100000); - -receiver(T0, Time, Main, P1,P1N,P1Rs, P2,P2N,P2Rs, C) -> - Remain = Time - erlang:convert_time_unit(erlang:monotonic_time() - T0, - native, millisecond), % test time remaining - Remain1 = if Remain < 0 -> - 0; - true -> - Remain - end, - {P1Rs1,P2Rs1} = - receive - {_Pid,P1} -> % report from a P1 process - {P1Rs+1,P2Rs}; - {_Pid,P2} -> % report from a P2 process - {P1Rs,P2Rs+1} - after Remain1 -> - {P1Rs,P2Rs} - end, - if Remain > 0 -> % keep going - receiver(T0, Time, Main, P1,P1N,P1Rs1, P2,P2N,P2Rs1, C-1); - true -> % finish - %% calculate results and send to main test process - P1Avg = P1Rs1/P1N, - P2Avg = P2Rs1/P2N, - Ratio = if P2Avg < 1.0 -> P1Avg; - true -> P1Avg/P2Avg - end, - Main ! {self(),{P1Rs1,round(P1Avg),P2Rs1,round(P2Avg),Ratio}}, - flush_loop() - end. - -starter({P1,P1N}, {P2,P2N}, Receiver) -> - %% start N1 processes with prio P1 - start_p(P1, P1N, Receiver), - %% start N2 processes with prio P2 - start_p(P2, P2N, Receiver), - erlang:display({started,P1N+P2N}), - flush_loop(). - -start_p(_, 0, _) -> - ok; -start_p(Prio, N, Receiver) -> - spawn_link(fun() -> p(Prio, Receiver) end), - start_p(Prio, N-1, Receiver). - -p(Prio, Receiver) -> - %% set process priority - process_flag(priority, Prio), - p_loop(0, Prio, Receiver). - -p_loop(100, Prio, Receiver) -> - receive after 0 -> ok end, - %% if Receiver gone, we're done - case is_process_alive(Receiver) of - false -> exit(bye); - true -> ok - end, - %% send report - Receiver ! {self(),Prio}, - p_loop(0, Prio, Receiver); - -p_loop(N, Prio, Receiver) -> - p_loop(N+1, Prio, Receiver). - - -flush_loop() -> - receive _ -> - ok - end, - flush_loop(). diff --git a/erts/emulator/test/port_SUITE.erl b/erts/emulator/test/port_SUITE.erl index 94ee9851dd..fccbaf13ee 100644 --- a/erts/emulator/test/port_SUITE.erl +++ b/erts/emulator/test/port_SUITE.erl @@ -153,7 +153,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 10}}]. + {timetrap, {minutes, 1}}]. all() -> [otp_6224, {group, stream}, basic_ping, slow_writes, diff --git a/erts/emulator/test/port_trace_SUITE.erl b/erts/emulator/test/port_trace_SUITE.erl index 03efdc15db..bfc3c8cb51 100644 --- a/erts/emulator/test/port_trace_SUITE.erl +++ b/erts/emulator/test/port_trace_SUITE.erl @@ -52,7 +52,7 @@ -define(ECHO_DRV_REMOTE_SEND_TERM, 15). suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 30}}]. + {timetrap, {minutes, 2}}]. all() -> [port_specs, ports, open_close, diff --git a/erts/emulator/test/process_SUITE.erl b/erts/emulator/test/process_SUITE.erl index e14185e881..4204d12eb3 100644 --- a/erts/emulator/test/process_SUITE.erl +++ b/erts/emulator/test/process_SUITE.erl @@ -134,6 +134,11 @@ init_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> [{testcase, Func}|Config]. end_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> + %% Restore max_heap_size to default value. + erlang:system_flag(max_heap_size, + #{size => 0, + kill => true, + error_logger => true}), ok. fun_spawn(Fun) -> @@ -1024,36 +1029,48 @@ bump_big(Prev, Limit) -> %% Priority 'low' should be mixed with 'normal' using a factor of %% about 8. (OTP-2644) low_prio(Config) when is_list(Config) -> - case erlang:system_info(schedulers_online) of - 1 -> - ok = low_prio_test(Config); - _ -> - erlang:system_flag(multi_scheduling, block_normal), - ok = low_prio_test(Config), - erlang:system_flag(multi_scheduling, unblock_normal), - {comment, - "Test not written for SMP runtime system. " - "Multi scheduling blocked during test."} - end. + erlang:system_flag(multi_scheduling, block_normal), + Prop = low_prio_test(Config), + erlang:system_flag(multi_scheduling, unblock_normal), + Str = lists:flatten(io_lib:format("Low/high proportion is ~.3f", + [Prop])), + {comment,Str}. low_prio_test(Config) when is_list(Config) -> process_flag(trap_exit, true), - S = spawn_link(?MODULE, prio_server, [0, 0]), + + %% Spawn the server running with high priority. The server must + %% not run at normal priority as that would skew the results for + %% two reasons: + %% + %% 1. There would be one more normal-priority processes than + %% low-priority processes. + %% + %% 2. The receive queue would grow faster than the server process + %% could process it. That would in turn trigger the reduction + %% punishment for the clients. + S = spawn_opt(?MODULE, prio_server, [0, 0], [link,{priority,high}]), + + %% Spawn the clients and let them run for a while. PCs = spawn_prio_clients(S, erlang:system_info(schedulers_online)), - ct:sleep({seconds,3}), + ct:sleep({seconds,2}), lists:foreach(fun (P) -> exit(P, kill) end, PCs), + + %% Stop the server and retrieve the result. S ! exit, - receive {'EXIT', S, {A, B}} -> check_prio(A, B) end, - ok. + receive + {'EXIT', S, {A, B}} -> + check_prio(A, B) + end. check_prio(A, B) -> Prop = A/B, ok = io:format("Low=~p, High=~p, Prop=~p\n", [A, B, Prop]), - %% It isn't 1/8, it's more like 0.3, but let's check that - %% the low-prio processes get some little chance to run at all. - true = (Prop < 1.0), - true = (Prop > 1/32). + %% Prop is expected to be appr. 1/8. Allow a reasonable margin. + true = Prop < 1/4, + true = Prop > 1/16, + Prop. prio_server(A, B) -> receive @@ -2057,6 +2074,7 @@ max_heap_size_test(Option, Size, Kill, ErrorLogger) -> end, if ErrorLogger -> receive + %% There must be at least one error message. {error, _, {emulator, _, [Pid|_]}} -> ok end; @@ -2069,22 +2087,33 @@ max_heap_size_test(Option, Size, Kill, ErrorLogger) -> {'DOWN', Ref, process, Pid, die} -> ok end, - flush(); + %% If the process was not killed, the limit may have + %% been reached more than once and there may be + %% more {error, ...} messages left. + receive_error_messages(Pid); true -> ok end, + + %% Make sure that there are no unexpected messages. + receive_unexpected(). + +receive_error_messages(Pid) -> receive - M -> - ct:fail({unexpected_message, M}) - after 10 -> + {error, _, {emulator, _, [Pid|_]}} -> + receive_error_messages(Pid) + after 1000 -> ok end. -flush() -> +receive_unexpected() -> receive - _M -> - flush() - after 1000 -> + {info_report, _, _} -> + %% May be an alarm message from os_mon. Ignore. + receive_unexpected(); + M -> + ct:fail({unexpected_message, M}) + after 10 -> ok end. diff --git a/erts/emulator/test/receive_SUITE.erl b/erts/emulator/test/receive_SUITE.erl index 83653a7a36..c7d5a3f5a0 100644 --- a/erts/emulator/test/receive_SUITE.erl +++ b/erts/emulator/test/receive_SUITE.erl @@ -39,25 +39,43 @@ groups() -> call_with_huge_message_queue(Config) when is_list(Config) -> Pid = spawn_link(fun echo_loop/0), - - {Time,ok} = tc(fun() -> calls(10, Pid) end), - - [self() ! {msg,N} || N <- lists:seq(1, 500000)], + _WarmUpTime = time_calls(Pid), + Time = time_calls(Pid), + _ = [self() ! {msg,N} || N <- lists:seq(1, 500000)], + io:format("Time for empty message queue: ~p", [Time]), erlang:garbage_collect(), - {NewTime1,ok} = tc(fun() -> calls(10, Pid) end), - {NewTime2,ok} = tc(fun() -> calls(10, Pid) end), + call_with_huge_message_queue_1(Pid, Time, 5). + +call_with_huge_message_queue_1(_Pid, _Time, 0) -> + ct:fail(bad_ratio); +call_with_huge_message_queue_1(Pid, Time, NumTries) -> + HugeTime = time_calls(Pid), + io:format("Time for huge message queue: ~p", [HugeTime]), + + case (HugeTime+1) / (Time+1) of + Q when Q < 10 -> + ok; + Q -> + io:format("Too high ratio: ~p\n", [Q]), + call_with_huge_message_queue_1(Pid, Time, NumTries-1) + end. - io:format("Time for empty message queue: ~p", [Time]), - io:format("Time1 for huge message queue: ~p", [NewTime1]), - io:format("Time2 for huge message queue: ~p", [NewTime2]), - - case hd(lists:sort([(NewTime1+1) / (Time+1), (NewTime2+1) / (Time+1)])) of - Q when Q < 10 -> - ok; - Q -> - ct:fail("Best Q = ~p", [Q]) - end, - ok. +%% Time a number calls. Try to avoid returning a zero time. +time_calls(Pid) -> + time_calls(Pid, 10). + +time_calls(_Pid, 0) -> + 0; +time_calls(Pid, NumTries) -> + case timer:tc(fun() -> calls(Pid) end) of + {0,ok} -> + time_calls(Pid, NumTries-1); + {Time,ok} -> + Time + end. + +calls(Pid) -> + calls(100, Pid). calls(0, _) -> ok; calls(N, Pid) -> @@ -108,6 +126,3 @@ echo_loop() -> Pid ! {Ref,Msg}, echo_loop() end. - -tc(Fun) -> - timer:tc(erlang, apply, [Fun,[]]). diff --git a/erts/emulator/test/timer_bif_SUITE.erl b/erts/emulator/test/timer_bif_SUITE.erl index 7cbd93a0f3..a977eb41c4 100644 --- a/erts/emulator/test/timer_bif_SUITE.erl +++ b/erts/emulator/test/timer_bif_SUITE.erl @@ -488,24 +488,40 @@ registered_process(Config) when is_list(Config) -> same_time_yielding(Config) when is_list(Config) -> Mem = mem(), + Ref = make_ref(), SchdlrsOnln = erlang:system_info(schedulers_online), Tmo = erlang:monotonic_time(millisecond) + 3000, Tmrs = lists:map(fun (I) -> process_flag(scheduler, (I rem SchdlrsOnln) + 1), - erlang:start_timer(Tmo, self(), hej, [{abs, true}]) + erlang:start_timer(Tmo, self(), Ref, [{abs, true}]) end, lists:seq(1, (?TIMEOUT_YIELD_LIMIT*3+1)*SchdlrsOnln)), true = mem_larger_than(Mem), - lists:foreach(fun (Tmr) -> receive {timeout, Tmr, hej} -> ok end end, Tmrs), + receive_all_timeouts(length(Tmrs), Ref), Done = erlang:monotonic_time(millisecond), true = Done >= Tmo, + MsAfterTmo = Done - Tmo, + io:format("Done ~p ms after Tmo\n", [MsAfterTmo]), case erlang:system_info(build_type) of - opt -> true = Done < Tmo + 200; - _ -> true = Done < Tmo + 1000 + opt -> + true = MsAfterTmo < 200; + _ -> + true = MsAfterTmo < 1000 end, Mem = mem(), ok. +%% Read out all timeouts in receive queue order. This is efficient +%% even if there are very many messages. + +receive_all_timeouts(0, _Ref) -> + ok; +receive_all_timeouts(N, Ref) -> + receive + {timeout, _Tmr, Ref} -> + receive_all_timeouts(N-1, Ref) + end. + same_time_yielding_with_cancel(Config) when is_list(Config) -> same_time_yielding_with_cancel_test(false, false). diff --git a/erts/emulator/test/trace_SUITE.erl b/erts/emulator/test/trace_SUITE.erl index 643c2e0472..bd0ea22de9 100644 --- a/erts/emulator/test/trace_SUITE.erl +++ b/erts/emulator/test/trace_SUITE.erl @@ -46,7 +46,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 5}}]. + {timetrap, {minutes, 1}}]. all() -> [cpu_timestamp, receive_trace, link_receive_call_correlation, @@ -184,10 +184,10 @@ receive_trace(Config) when is_list(Config) -> {'EXIT', Intruder, {badarg, _}} = receive_first(), %% Untrace the process; we should not receive anything. - ?line 1 = erlang:trace(Receiver, false, ['receive']), - ?line Receiver ! {hello, there}, - ?line Receiver ! any_garbage, - ?line receive_nothing(), + 1 = erlang:trace(Receiver, false, ['receive']), + Receiver ! {hello, there}, + Receiver ! any_garbage, + receive_nothing(), %% Verify restrictions in matchspec for 'receive' F3 = fun (Pat) -> {'EXIT', {badarg,_}} = (catch erlang:trace_pattern('receive', Pat, [])) end, diff --git a/erts/emulator/test/trace_port_SUITE.erl b/erts/emulator/test/trace_port_SUITE.erl index e4db368ea1..5eb27a7b68 100644 --- a/erts/emulator/test/trace_port_SUITE.erl +++ b/erts/emulator/test/trace_port_SUITE.erl @@ -37,7 +37,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}, - {timetrap, {seconds, 30}}]. + {timetrap, {minutes, 2}}]. all() -> [call_trace, return_trace, send, receive_trace, diff --git a/erts/etc/common/erlexec.c b/erts/etc/common/erlexec.c index 70520eea15..51ed2d0dff 100644 --- a/erts/etc/common/erlexec.c +++ b/erts/etc/common/erlexec.c @@ -555,7 +555,7 @@ int main(int argc, char **argv) if(s) { add_Eargs(s); /* argv[0] = scriptname*/ } else { - add_Eargs(progname); /* argv[0] = erl or cerl */ + add_Eargs(emu); /* argv[0] = erl or cerl */ } /* * Add the bindir to the path (unless it is there already). diff --git a/erts/etc/common/escript.c b/erts/etc/common/escript.c index 630e241882..7f0af77a4c 100644 --- a/erts/etc/common/escript.c +++ b/erts/etc/common/escript.c @@ -433,7 +433,7 @@ main(int argc, char** argv) char* emulator; char* env; char* basename; - char* absname; + char* def_emu_lookup_path; char scriptname[PMAX]; char** last_opt; char** first_opt; @@ -480,6 +480,7 @@ main(int argc, char** argv) #else if (strcmp(basename, "escript") == 0) { #endif + def_emu_lookup_path = argv[0]; /* * Locate all options before the script name. */ @@ -498,27 +499,24 @@ main(int argc, char** argv) argc--; argv++; } else { + char *absname = find_prog(argv[0]); #ifdef __WIN32__ - int len; -#endif - absname = find_prog(argv[0]); -#ifdef __WIN32__ - len = strlen(absname); + int len = strlen(absname); if (len >= 4 && _stricmp(absname+len-4, ".exe") == 0) { absname[len-4] = '\0'; } #endif - erts_snprintf(scriptname, sizeof(scriptname), "%s.escript", absname); - efree(absname); + efree(absname); + def_emu_lookup_path = scriptname; } /* Determine path to emulator */ emulator = env = get_env("ESCRIPT_EMULATOR"); if (emulator == NULL) { - emulator = get_default_emulator(scriptname); + emulator = get_default_emulator(def_emu_lookup_path); } if (strlen(emulator) >= PMAX) @@ -528,7 +526,7 @@ main(int argc, char** argv) * Push initial arguments. */ - push_words(emulator); + PUSH(emulator); free_env_val(env); PUSH("+B"); diff --git a/erts/etc/unix/etp-commands.in b/erts/etc/unix/etp-commands.in index fc7b614c21..8f70f879d5 100644 --- a/erts/etc/unix/etp-commands.in +++ b/erts/etc/unix/etp-commands.in @@ -1,3 +1,4 @@ +# -*- gdb-script -*- # # %CopyrightBegin% # @@ -2153,13 +2154,22 @@ define etp-processes printf "No processes, since system isn't initialized!\n" else set $proc_ix = 0 - while $proc_ix < erts_proc.r.o.max - set $proc = (Process *) *((UWord *) &erts_proc.r.o.tab[$proc_ix]) - if ($proc != ((Process *) 0) && $proc != &erts_invalid_process) + set $proc_max_ix = erts_proc.r.o.max + set $proc_tab = erts_proc.r.o.tab + set $invalid_proc = &erts_invalid_process + set $proc_decentile = $proc_max_ix / 10 + set $proc_printile = $proc_decentile + while $proc_ix < $proc_max_ix + set $proc = (Process *) *((UWord *) ($proc_tab + $proc_ix)) + if ($proc != ((Process *) 0) && $proc != $invalid_proc) printf "---\n" printf " Pix: %d\n", $proc_ix etp-process-info $proc end + if $proc_ix == $proc_printile + printf "--- %d%% (%d / %d) searched\n", $proc_printile / $proc_decentile * 10, $proc_ix, $proc_max_ix + set $proc_printile += $proc_decentile + end set $proc_ix++ end printf "---\n", @@ -2479,15 +2489,19 @@ document etp-port-info %--------------------------------------------------------------------------- end - define etp-ports if (!erts_initialized) printf "No ports, since system isn't initialized!\n" else set $port_ix = 0 - while $port_ix < erts_port.r.o.max - set $port = (Port *) *((UWord *) &erts_port.r.o.tab[$port_ix]) - if ($port != ((Port *) 0) && $port != &erts_invalid_port) + set $port_max_ix = erts_port.r.o.max + set $port_tab = erts_port.r.o.tab + set $invalid_port = &erts_invalid_port + set $port_decentile = $port_max_ix / 10 + set $port_printile = $port_decentile + while $port_ix < $port_max_ix + set $port = (Port *) *((UWord *) ($port_tab + $port_ix)) + if ($port != ((Port *) 0) && $port != $invalid_port) if (*(((Uint32 *) &(((Port *) $port)->state))) & 0x100) == 0 # I.e, not free printf "---\n" @@ -2495,6 +2509,10 @@ define etp-ports etp-port-info $port end end + if $port_ix == $port_printile + printf "--- %d%% (%d / %d) searched\n", $port_printile / $port_decentile * 10, $port_ix, $port_max_ix + set $port_printile += $port_decentile + end set $port_ix++ end printf "---\n", diff --git a/lib/compiler/src/compile.erl b/lib/compiler/src/compile.erl index b3c8c42af7..c6e61d543e 100644 --- a/lib/compiler/src/compile.erl +++ b/lib/compiler/src/compile.erl @@ -467,8 +467,10 @@ mpf(Ms) -> passes(Type, Opts) -> {Ext,Passes0} = passes_1(Opts), Passes1 = case Type of - file -> Passes0; - forms -> tl(Passes0) + file -> + Passes0; + forms -> + fix_first_pass(Passes0) end, Passes = select_passes(Passes1, Opts), @@ -505,6 +507,22 @@ pass(from_beam) -> {".beam",[?pass(read_beam_file)|binary_passes()]}; pass(_) -> none. +%% For compilation from forms, replace the first pass with a pass +%% that retrieves the module name. The module name is needed for +%% proper diagnostics and for compilation to native code. + +fix_first_pass([{parse_core,_}|Passes]) -> + [?pass(get_module_name_from_core)|Passes]; +fix_first_pass([{beam_consult_asm,_}|Passes]) -> + [?pass(get_module_name_from_asm)|Passes]; +fix_first_pass([{read_beam_file,_}|Passes]) -> + [?pass(get_module_name_from_beam)|Passes]; +fix_first_pass([_|Passes]) -> + %% When compiling from abstract code, the module name + %% will be set after running the v3_core pass. + Passes. + + %% select_passes([Command], Opts) -> [{Name,Function}] %% Interpret the lists of commands to return a pure list of passes. %% @@ -836,6 +854,12 @@ beam_consult_asm(_Code, St) -> {error,St#compile{errors=St#compile.errors ++ Es}} end. +get_module_name_from_asm({Mod,_,_,_,_}=Asm, St) -> + {ok,Asm,St#compile{module=Mod}}; +get_module_name_from_asm(Asm, St) -> + %% Invalid Beam assembly code. Let it crash in a later pass. + {ok,Asm,St}. + read_beam_file(_Code, St) -> case file:read_file(St#compile.ifile) of {ok,Beam} -> @@ -853,6 +877,16 @@ read_beam_file(_Code, St) -> {error,St#compile{errors=St#compile.errors ++ Es}} end. +get_module_name_from_beam(Beam, St) -> + case beam_lib:info(Beam) of + {error,beam_lib,Error} -> + Es = [{"((forms))",[{none,beam_lib,Error}]}], + {error,St#compile{errors=St#compile.errors ++ Es}}; + Info -> + {module,Mod} = keyfind(module, 1, Info), + {ok,Beam,St#compile{module=Mod}} + end. + no_native_compilation(BeamFile, #compile{options=Opts0}) -> case beam_lib:chunks(BeamFile, ["CInf"]) of {ok,{_,[{"CInf",Term0}]}} -> @@ -940,6 +974,16 @@ parse_core(_Code, St) -> {error,St#compile{errors=St#compile.errors ++ Es}} end. +get_module_name_from_core(Core, St) -> + try + Mod = cerl:concrete(cerl:module_name(Core)), + {ok,Core,St#compile{module=Mod}} + catch + _:_ -> + %% Invalid Core Erlang code. Let it crash in a later pass. + {ok,Core,St} + end. + compile_options([{attribute,_L,compile,C}|Fs]) when is_list(C) -> C ++ compile_options(Fs); compile_options([{attribute,_L,compile,C}|Fs]) -> diff --git a/lib/compiler/test/beam_validator_SUITE.erl b/lib/compiler/test/beam_validator_SUITE.erl index ca85eef688..75bfbf68cc 100644 --- a/lib/compiler/test/beam_validator_SUITE.erl +++ b/lib/compiler/test/beam_validator_SUITE.erl @@ -446,7 +446,7 @@ do_bin_opt(Mod, Asm) -> do_bin_opt(Transform, Mod, Asm0) -> Asm = Transform(Asm0), case compile:forms(Asm, [from_asm,no_postopt,return]) of - {ok,[],Code,_Warnings} when is_binary(Code) -> + {ok,Mod,Code,_Warnings} when is_binary(Code) -> ok; {error,Errors0,_} -> %% beam_validator must return errors, not simply crash, diff --git a/lib/compiler/test/compile_SUITE.erl b/lib/compiler/test/compile_SUITE.erl index f5e904a50a..4e2753ba5f 100644 --- a/lib/compiler/test/compile_SUITE.erl +++ b/lib/compiler/test/compile_SUITE.erl @@ -163,6 +163,24 @@ forms_2(Config) when is_list(Config) -> ok end, + {ok,simple,Core} = compile:forms(SimpleCode, [to_core0,binary]), + forms_compile_and_load(Core, [from_core]), + forms_compile_and_load(Core, [from_core,native]), + + {ok,simple,Asm} = compile:forms(SimpleCode, [to_asm,binary]), + forms_compile_and_load(Asm, [from_asm]), + forms_compile_and_load(Asm, [from_asm,native]), + + {ok,simple,Beam} = compile:forms(SimpleCode, []), + forms_compile_and_load(Beam, [from_beam]), + forms_compile_and_load(Beam, [from_beam,native]), + + %% Cover the error handling code. + error = compile:forms(bad_core, [from_core,report]), + error = compile:forms(bad_asm, [from_asm,report]), + error = compile:forms(<<"bad_beam">>, [from_beam,report]), + error = compile:forms(<<"bad_beam">>, [from_beam,native,report]), + ok. @@ -180,6 +198,14 @@ forms_load_code(Mod, Src, Bin) -> SourceOption. +forms_compile_and_load(Code, Opts) -> + Mod = simple, + {ok,Mod,Bin} = compile:forms(Code, Opts), + {module,Mod} = code:load_binary(Mod, "ignore", Bin), + _ = Mod:module_info(), + true = code:delete(simple), + false = code:purge(simple), + ok. module_mismatch(Config) when is_list(Config) -> DataDir = proplists:get_value(data_dir, Config), @@ -835,7 +861,7 @@ do_core_pp_1(M, A, Outdir) -> ok = file:delete(CoreFile), %% Compile as usual (including optimizations). - compile_forms(Core, [clint,from_core,binary]), + compile_forms(M, Core, [clint,from_core,binary]), %% Don't optimize to test that we are not dependent %% on the Core Erlang optmimization passes. @@ -844,13 +870,13 @@ do_core_pp_1(M, A, Outdir) -> %% records; if sys_core_fold was run it would fix %% that; if sys_core_fold was not run v3_kernel would %% crash.) - compile_forms(Core, [clint,from_core,no_copt,binary]), + compile_forms(M, Core, [clint,from_core,no_copt,binary]), ok. -compile_forms(Forms, Opts) -> +compile_forms(Mod, Forms, Opts) -> case compile:forms(Forms, [report_errors|Opts]) of - {ok,[],_} -> ok; + {ok,Mod,_} -> ok; Other -> throw({error,Other}) end. diff --git a/lib/crypto/c_src/crypto.c b/lib/crypto/c_src/crypto.c index 793cff166c..688ec339aa 100644 --- a/lib/crypto/c_src/crypto.c +++ b/lib/crypto/c_src/crypto.c @@ -609,7 +609,7 @@ struct digest_type_t { }md; }; -struct digest_type_t digest_types[] = +static struct digest_type_t digest_types[] = { {{"md4"}, {&EVP_md4}}, {{"md5"}, {&EVP_md5}}, @@ -666,7 +666,7 @@ struct cipher_type_t { #define COND_NO_DES_PTR(Ptr) (Ptr) #endif -struct cipher_type_t cipher_types[] = +static struct cipher_type_t cipher_types[] = { {{"rc2_cbc"}, #ifndef OPENSSL_NO_RC2 @@ -1845,7 +1845,7 @@ static ERL_NIF_TERM aes_cfb_128_crypt_nif(ErlNifEnv* env, int argc, const ERL_NI AES_cfb128_encrypt((unsigned char *) text.data, enif_make_new_binary(env, text.size, &ret), text.size, &aes_key, ivec_clone, &new_ivlen, - (argv[3] != atom_true)); + (argv[3] == atom_true)); CONSUME_REDS(env,text); return ret; } diff --git a/lib/crypto/test/Makefile b/lib/crypto/test/Makefile index 5a81c84558..138081d386 100644 --- a/lib/crypto/test/Makefile +++ b/lib/crypto/test/Makefile @@ -77,6 +77,7 @@ release_spec: release_tests_spec: $(TEST_TARGET) $(INSTALL_DIR) "$(RELSYSDIR)" $(INSTALL_DATA) crypto.spec crypto.cover $(RELTEST_FILES) "$(RELSYSDIR)" + @tar cfh - crypto_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) chmod -R u+w "$(RELSYSDIR)" release_docs_spec: diff --git a/lib/crypto/test/crypto_SUITE.erl b/lib/crypto/test/crypto_SUITE.erl index 8cb08cade0..30ee782fe9 100644 --- a/lib/crypto/test/crypto_SUITE.erl +++ b/lib/crypto/test/crypto_SUITE.erl @@ -169,6 +169,12 @@ groups() -> %%------------------------------------------------------------------- init_per_suite(Config) -> + file:set_cwd(datadir(Config)), + {ok, _} = zip:unzip("KAT_AES.zip"), + {ok, _} = zip:unzip("aesmmt.zip"), + {ok, _} = zip:unzip("cmactestvectors.zip"), + {ok, _} = zip:unzip("gcmtestvectors.zip"), + try crypto:start() of ok -> try crypto:strong_rand_bytes(1) of @@ -330,7 +336,7 @@ no_hmac(Config) when is_list(Config) -> cmac() -> [{doc, "Test all different cmac functions"}]. cmac(Config) when is_list(Config) -> - Pairs = proplists:get_value(cmac, Config), + Pairs = lazy_eval(proplists:get_value(cmac, Config)), lists:foreach(fun cmac_check/1, Pairs), lists:foreach(fun cmac_check/1, cmac_iolistify(Pairs)). %%-------------------------------------------------------------------- @@ -350,7 +356,7 @@ block(Config) when is_list(Config) -> ok end, - Blocks = proplists:get_value(block, Config), + Blocks = lazy_eval(proplists:get_value(block, Config)), lists:foreach(fun block_cipher/1, Blocks), lists:foreach(fun block_cipher/1, block_iolistify(Blocks)), lists:foreach(fun block_cipher_increment/1, block_iolistify(Blocks)). @@ -359,7 +365,7 @@ block(Config) when is_list(Config) -> no_block() -> [{doc, "Test disabled block ciphers"}]. no_block(Config) when is_list(Config) -> - Blocks = proplists:get_value(block, Config), + Blocks = lazy_eval(proplists:get_value(block, Config)), Args = case Blocks of [{_Type, _Key, _PlainText} = A | _] -> tuple_to_list(A); @@ -376,7 +382,7 @@ no_aead() -> [{doc, "Test disabled aead ciphers"}]. no_aead(Config) when is_list(Config) -> [{Type, Key, PlainText, Nonce, AAD, CipherText, CipherTag} | _] = - proplists:get_value(aead, Config), + lazy_eval(proplists:get_value(aead, Config)), EncryptArgs = [Type, Key, Nonce, {AAD, PlainText}], DecryptArgs = [Type, Key, Nonce, {AAD, CipherText, CipherTag}], notsup(fun crypto:block_encrypt/4, EncryptArgs), @@ -628,10 +634,15 @@ block_cipher({Type, Key, IV, PlainText, CipherText}) -> block_cipher_increment({Type, Key, IV, PlainTexts}) when Type == des_cbc; Type == aes_cbc; Type == des3_cbc -> block_cipher_increment(Type, Key, IV, IV, PlainTexts, iolist_to_binary(PlainTexts), []); +block_cipher_increment({Type, Key, IV, PlainTexts, CipherText}) + when Type == des_cbc; Type == des3_cbc -> + block_cipher_increment(Type, Key, IV, IV, PlainTexts, iolist_to_binary(PlainTexts), CipherText, []); block_cipher_increment({Type, Key, IV, PlainTexts, _CipherText}) when Type == aes_cbc -> Plain = iolist_to_binary(PlainTexts), Blocks = [iolistify(Block) || << Block:128/bitstring >> <= Plain], block_cipher_increment(Type, Key, IV, IV, Blocks, Plain, []); +block_cipher_increment({_Type, _, _, _, _}) -> + ok; block_cipher_increment({_Type, _, _, _}) -> ok; block_cipher_increment({_,_,_}) -> @@ -648,6 +659,17 @@ block_cipher_increment(Type, Key, IV0, IV, [PlainText | PlainTexts], Plain, Acc) CipherText = crypto:block_encrypt(Type, Key, IV, PlainText), NextIV = crypto:next_iv(Type, CipherText), block_cipher_increment(Type, Key, IV0, NextIV, PlainTexts, Plain, [CipherText | Acc]). +block_cipher_increment(Type, Key, IV0, _IV, [], _Plain, CipherText, Acc) -> + case iolist_to_binary(lists:reverse(Acc)) of + CipherText -> + ok; + Other -> + ct:fail({{crypto, block_decrypt, [Type, Key, IV0, CipherText]}, {expected, CipherText}, {got, Other}}) + end; +block_cipher_increment(Type, Key, IV0, IV, [PlainText | PlainTexts], Plain, CipherText, Acc) -> + CT = crypto:block_encrypt(Type, Key, IV, PlainText), + NextIV = crypto:next_iv(Type, CT), + block_cipher_increment(Type, Key, IV0, NextIV, PlainTexts, Plain, CipherText, [CT | Acc]). stream_cipher({Type, Key, PlainText}) -> Plain = iolist_to_binary(PlainText), @@ -812,6 +834,8 @@ notsup(Fun, Args) -> hexstr2point(X, Y) -> <<4:8, (hexstr2bin(X))/binary, (hexstr2bin(Y))/binary>>. +hexstr2bin(S) when is_binary(S) -> + list_to_binary(hexstr2list(binary_to_list(S))); hexstr2bin(S) -> list_to_binary(hexstr2list(S)). @@ -1181,24 +1205,24 @@ group_config(rc2_cbc, Config) -> Block = rc2_cbc(), [{block, Block} | Config]; group_config(aes_cbc128 = Type, Config) -> - Block = aes_cbc128(), - Pairs = cmac_nist(Type), + Block = fun() -> aes_cbc128(Config) end, + Pairs = fun() -> cmac_nist(Config, Type) end, [{block, Block}, {cmac, Pairs} | Config]; group_config(aes_cbc256 = Type, Config) -> - Block = aes_cbc256(), - Pairs = cmac_nist(Type), + Block = fun() -> aes_cbc256(Config) end, + Pairs = fun() -> cmac_nist(Config, Type) end, [{block, Block}, {cmac, Pairs} | Config]; group_config(aes_ecb, Config) -> - Block = aes_ecb(), - [{block, Block} | Config]; + Block = fun() -> aes_ecb(Config) end, + [{block, Block} | Config]; group_config(aes_ige256, Config) -> Block = aes_ige256(), [{block, Block} | Config]; group_config(aes_cfb8, Config) -> - Block = aes_cfb8(), + Block = fun() -> aes_cfb8(Config) end, [{block, Block} | Config]; group_config(aes_cfb128, Config) -> - Block = aes_cfb128(), + Block = fun() -> aes_cfb128(Config) end, [{block, Block} | Config]; group_config(blowfish_cbc, Config) -> Block = blowfish_cbc(), @@ -1219,13 +1243,13 @@ group_config(aes_ctr, Config) -> Stream = aes_ctr(), [{stream, Stream} | Config]; group_config(aes_gcm, Config) -> - AEAD = aes_gcm(), + AEAD = fun() -> aes_gcm(Config) end, [{aead, AEAD} | Config]; group_config(chacha20_poly1305, Config) -> AEAD = chacha20_poly1305(), [{aead, AEAD} | Config]; group_config(aes_cbc, Config) -> - Block = aes_cbc(), + Block = aes_cbc(Config), [{block, Block} | Config]; group_config(_, Config) -> Config. @@ -1311,9 +1335,10 @@ rfc_4634_sha512_digests() -> long_msg() -> fun() -> lists:duplicate(1000000, $a) end. -%% Building huge terms (like long_msg/0) in init_per_group seems to cause -%% test_server crash with 'no_answer_from_tc_supervisor' sometimes on some -%% machines. Therefore lazy evaluation when test case has started. +%% Passing huge terms (like long_msg/0) through config causes excessive memory +%% consumption and long runtimes in the test server. This results in test_server +%% crash with 'no_answer_from_tc_supervisor' sometimes on some machines. +%% Therefore lazy evaluation when test case has started. lazy_eval(F) when is_function(F) -> F(); lazy_eval(Lst) when is_list(Lst) -> lists:map(fun lazy_eval/1, Lst); lazy_eval(Tpl) when is_tuple(Tpl) -> list_to_tuple(lists:map(fun lazy_eval/1, tuple_to_list(Tpl))); @@ -1601,209 +1626,30 @@ rc2_cbc() -> }]. %% AES CBC test vectors from http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf -aes_cbc() -> - [ - %% F.2.1 CBC-AES128.Encrypt, F.2.2 CBC-AES128.Decrypt - {aes_cbc, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), %% Key - hexstr2bin("000102030405060708090a0b0c0d0e0f"), %% IV - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" %% PlainText - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710"), - hexstr2bin("7649abac8119b246cee98e9b12e9197d" %% CipherText - "5086cb9b507219ee95db113a917678b2" - "73bed6b8e3c1743b7116e69e22229516" - "3ff1caa1681fac09120eca307586e1a7")}, - %% F.2.3 CBC-AES192.Encrypt, F.2.4 CBC-AES192.Decrypt - {aes_cbc, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e5" %% Key - "62f8ead2522c6b7b"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), %% IV - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" %% PlainText - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710"), - hexstr2bin("4f021db243bc633d7178183a9fa071e8" %% CipherText - "b4d9ada9ad7dedf4e5e738763f69145a" - "571b242012fb7ae07fa9baac3df102e0" - "08b0e27988598881d920a9e64f5615cd")}, - %% F.2.5 CBC-AES256.Encrypt, F.2.6 CBC-AES256.Decrypt - {aes_cbc, - hexstr2bin("603deb1015ca71be2b73aef0857d7781" %% Key - "1f352c073b6108d72d9810a30914dff4"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), %% IV - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" %% PlainText - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710"), - hexstr2bin("f58c4c04d6e5f1ba779eabfb5f7bfbd6" %% CipherText - "9cfc4e967edb808d679f777bc6702c7d" - "39f23369a9d9bacfa530e26304231461" - "b2eb05e2c39be9fcda6c19078c6a9d1b")} - ]. - -aes_cbc128() -> - [{aes_cbc128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cbc128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("7649ABAC8119B246CEE98E9B12E9197D"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cbc128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("5086CB9B507219EE95DB113A917678B2"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cbc128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("73BED6B8E3C1743B7116E69E22229516"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")} - ]. - -aes_cbc256() -> - [{aes_cbc256, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("000102030405060708090A0B0C0D0E0F"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cbc256, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("F58C4C04D6E5F1BA779EABFB5F7BFBD6"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cbc256, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("9CFC4E967EDB808D679F777BC6702C7D"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cbc256, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("39F23369A9D9BACFA530E26304231461"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")} - ]. - -aes_ecb() -> - [ - {aes_ecb, - <<"YELLOW SUBMARINE">>, - <<"YELLOW SUBMARINE">>}, - {aes_ecb, - <<"0000000000000000">>, - <<"0000000000000000">>}, - {aes_ecb, - <<"FFFFFFFFFFFFFFFF">>, - <<"FFFFFFFFFFFFFFFF">>}, - {aes_ecb, - <<"3000000000000000">>, - <<"1000000000000001">>}, - {aes_ecb, - <<"1111111111111111">>, - <<"1111111111111111">>}, - {aes_ecb, - <<"0123456789ABCDEF">>, - <<"1111111111111111">>}, - {aes_ecb, - <<"0000000000000000">>, - <<"0000000000000000">>}, - {aes_ecb, - <<"FEDCBA9876543210">>, - <<"0123456789ABCDEF">>}, - {aes_ecb, - <<"7CA110454A1A6E57">>, - <<"01A1D6D039776742">>}, - {aes_ecb, - <<"0131D9619DC1376E">>, - <<"5CD54CA83DEF57DA">>}, - {aes_ecb, - <<"07A1133E4A0B2686">>, - <<"0248D43806F67172">>}, - {aes_ecb, - <<"3849674C2602319E">>, - <<"51454B582DDF440A">>}, - {aes_ecb, - <<"04B915BA43FEB5B6">>, - <<"42FD443059577FA2">>}, - {aes_ecb, - <<"0113B970FD34F2CE">>, - <<"059B5E0851CF143A">>}, - {aes_ecb, - <<"0170F175468FB5E6">>, - <<"0756D8E0774761D2">>}, - {aes_ecb, - <<"43297FAD38E373FE">>, - <<"762514B829BF486A">>}, - {aes_ecb, - <<"07A7137045DA2A16">>, - <<"3BDD119049372802">>}, - {aes_ecb, - <<"04689104C2FD3B2F">>, - <<"26955F6835AF609A">>}, - {aes_ecb, - <<"37D06BB516CB7546">>, - <<"164D5E404F275232">>}, - {aes_ecb, - <<"1F08260D1AC2465E">>, - <<"6B056E18759F5CCA">>}, - {aes_ecb, - <<"584023641ABA6176">>, - <<"004BD6EF09176062">>}, - {aes_ecb, - <<"025816164629B007">>, - <<"480D39006EE762F2">>}, - {aes_ecb, - <<"49793EBC79B3258F">>, - <<"437540C8698F3CFA">>}, - {aes_ecb, - <<"018310DC409B26D6">>, - <<"1D9D5C5018F728C2">>}, - {aes_ecb, - <<"1C587F1C13924FEF">>, - <<"305532286D6F295A">>}, - {aes_ecb, - <<"0101010101010101">>, - <<"0123456789ABCDEF">>}, - {aes_ecb, - <<"1F1F1F1F0E0E0E0E">>, - <<"0123456789ABCDEF">>}, - {aes_ecb, - <<"E0FEE0FEF1FEF1FE">>, - <<"0123456789ABCDEF">>}, - {aes_ecb, - <<"0000000000000000">>, - <<"FFFFFFFFFFFFFFFF">>}, - {aes_ecb, - <<"FFFFFFFFFFFFFFFF">>, - <<"0000000000000000">>}, - {aes_ecb, - <<"0123456789ABCDEF">>, - <<"0000000000000000">>}, - {aes_ecb, - <<"FEDCBA9876543210">>, - <<"FFFFFFFFFFFFFFFF">>}, - %% AES ECB test vectors from http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf - %% F.1.1 ECB-AES128.Encrypt, F.1.2 ECB-AES128.Decrypt - {aes_ecb, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710")}, - %% F.1.3 ECB-AES192.Encrypt, F.1.4 ECB-AES192.Decrypt - {aes_ecb, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e5" - "62f8ead2522c6b7b"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710")}, - %% F.1.5 ECB-AES256.Encrypt, F.1.6 ECB-AES256.Decrypt - {aes_ecb, - hexstr2bin("603deb1015ca71be2b73aef0857d7781" - "1f352c073b6108d72d9810a30914dff4"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710")} - ]. +aes_cbc(Config) -> + read_rsp(Config, aes_cbc, + ["CBCVarTxt128.rsp", "CBCVarKey128.rsp", "CBCGFSbox128.rsp", "CBCKeySbox128.rsp", + "CBCVarTxt192.rsp", "CBCVarKey192.rsp", "CBCGFSbox192.rsp", "CBCKeySbox192.rsp", + "CBCVarTxt256.rsp", "CBCVarKey256.rsp", "CBCGFSbox256.rsp", "CBCKeySbox256.rsp", + "CBCMMT128.rsp", "CBCMMT192.rsp", "CBCMMT256.rsp" + ]). + +aes_cbc128(Config) -> + read_rsp(Config, aes_cbc128, + ["CBCVarTxt128.rsp", "CBCVarKey128.rsp", "CBCGFSbox128.rsp", "CBCKeySbox128.rsp", + "CBCMMT128.rsp"]). + +aes_cbc256(Config) -> + read_rsp(Config, aes_cbc256, + ["CBCVarTxt256.rsp", "CBCVarKey256.rsp", "CBCGFSbox256.rsp", "CBCKeySbox256.rsp", + "CBCMMT256.rsp"]). + +aes_ecb(Config) -> + read_rsp(Config, aes_ecb, + ["ECBVarTxt128.rsp", "ECBVarKey128.rsp", "ECBGFSbox128.rsp", "ECBKeySbox128.rsp", + "ECBVarTxt192.rsp", "ECBVarKey192.rsp", "ECBGFSbox192.rsp", "ECBKeySbox192.rsp", + "ECBVarTxt256.rsp", "ECBVarKey256.rsp", "ECBGFSbox256.rsp", "ECBKeySbox256.rsp", + "ECBMMT128.rsp", "ECBMMT192.rsp", "ECBMMT256.rsp"]). aes_ige256() -> [{aes_ige256, @@ -1824,107 +1670,19 @@ aes_ige256() -> hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")} ]. -aes_cfb8() -> - [{aes_cfb8, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb8, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("3B3FD92EB72DAD20333449F8E83CFB4A"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb8, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("C8A64537A0B3A93FCDE3CDAD9F1CE58B"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb8, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("26751F67A3CBB140B1808CF187A4F4DF"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")}, - {aes_cfb8, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb8, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("cdc80d6fddf18cab34c25909c99a4174"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb8, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("67ce7f7f81173621961a2b70171d3d7a"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb8, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("2e1e8a1dd59b88b1c8e60fed1efac4c9"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")}, - {aes_cfb8, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb8, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("dc7e84bfda79164b7ecd8486985d3860"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb8, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("39ffed143b28b1c832113c6331e5407b"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb8, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("df10132415e54b92a13ed0a8267ae2f9"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")} - ]. +aes_cfb8(Config) -> + read_rsp(Config, aes_cfb8, + ["CFB8VarTxt128.rsp", "CFB8VarKey128.rsp", "CFB8GFSbox128.rsp", "CFB8KeySbox128.rsp", + "CFB8VarTxt192.rsp", "CFB8VarKey192.rsp", "CFB8GFSbox192.rsp", "CFB8KeySbox192.rsp", + "CFB8VarTxt256.rsp", "CFB8VarKey256.rsp", "CFB8GFSbox256.rsp", "CFB8KeySbox256.rsp", + "CFB8MMT128.rsp", "CFB8MMT192.rsp", "CFB8MMT256.rsp"]). -aes_cfb128() -> - [{aes_cfb128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("3B3FD92EB72DAD20333449F8E83CFB4A"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("C8A64537A0B3A93FCDE3CDAD9F1CE58B"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb128, - hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - hexstr2bin("26751F67A3CBB140B1808CF187A4F4DF"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")}, - {aes_cfb128, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb128, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("cdc80d6fddf18cab34c25909c99a4174"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb128, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("67ce7f7f81173621961a2b70171d3d7a"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb128, - hexstr2bin("8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b"), - hexstr2bin("2e1e8a1dd59b88b1c8e60fed1efac4c9"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")}, - {aes_cfb128, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("000102030405060708090a0b0c0d0e0f"), - hexstr2bin("6bc1bee22e409f96e93d7e117393172a")}, - {aes_cfb128, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("dc7e84bfda79164b7ecd8486985d3860"), - hexstr2bin("ae2d8a571e03ac9c9eb76fac45af8e51")}, - {aes_cfb128, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("39ffed143b28b1c832113c6331e5407b"), - hexstr2bin("30c81c46a35ce411e5fbc1191a0a52ef")}, - {aes_cfb128, - hexstr2bin("603deb1015ca71be2b73aef0857d77811f352c073b6108d72d9810a30914dff4"), - hexstr2bin("df10132415e54b92a13ed0a8267ae2f9"), - hexstr2bin("f69f2445df4f9b17ad2b417be66c3710")} - ]. +aes_cfb128(Config) -> + read_rsp(Config, aes_cfb128, + ["CFB128VarTxt128.rsp", "CFB128VarKey128.rsp", "CFB128GFSbox128.rsp", "CFB128KeySbox128.rsp", + "CFB128VarTxt192.rsp", "CFB128VarKey192.rsp", "CFB128GFSbox192.rsp", "CFB128KeySbox192.rsp", + "CFB128VarTxt256.rsp", "CFB128VarKey256.rsp", "CFB128GFSbox256.rsp", "CFB128KeySbox256.rsp", + "CFB128MMT128.rsp", "CFB128MMT192.rsp", "CFB128MMT256.rsp"]). blowfish_cbc() -> [{blowfish_cbc, @@ -2098,284 +1856,14 @@ aes_ctr() -> ]. -%% AES GCM test vectors from http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/gcm/gcm-spec.pdf -aes_gcm() -> - [ - %% Test Case 1 - {aes_gcm, hexstr2bin("00000000000000000000000000000000"), %% Key - hexstr2bin(""), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin(""), %% CipherText - hexstr2bin("58e2fccefa7e3061367f1d57a4e7455a")}, %% CipherTag - - %% Test Case 2 - {aes_gcm, hexstr2bin("00000000000000000000000000000000"), %% Key - hexstr2bin("00000000000000000000000000000000"), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin("0388dace60b6a392f328c2b971b2fe78"), %% CipherText - hexstr2bin("ab6e47d42cec13bdf53a67b21257bddf")}, %% CipherTag - - %% Test Case 3 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308"), %% Key - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b391aafd255"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin("42831ec2217774244b7221b784d0d49c" %% CipherText - "e3aa212f2c02a4e035c17e2329aca12e" - "21d514b25466931c7d8f6a5aac84aa05" - "1ba30b396a0aac973d58e091473f5985"), - hexstr2bin("4d5c2af327cd64a62cf35abd2ba6fab4")}, %% CipherTag - - %% Test Case 4 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308"), %% Key - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("42831ec2217774244b7221b784d0d49c" %% CipherText - "e3aa212f2c02a4e035c17e2329aca12e" - "21d514b25466931c7d8f6a5aac84aa05" - "1ba30b396a0aac973d58e091"), - hexstr2bin("5bc94fbc3221a5db94fae95ae7121a47")}, %% CipherTag - - %% Test Case 5 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308"), %% Key - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbad"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("61353b4c2806934a777ff51fa22a4755" %% CipherText - "699b2a714fcdc6f83766e5f97b6c7423" - "73806900e49f24b22b097544d4896b42" - "4989b5e1ebac0f07c23f4598"), - hexstr2bin("3612d2e79e3b0785561be14aaca2fccb")}, %% CipherTag - - %% Test Case 6" - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308"), %% Key - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("9313225df88406e555909c5aff5269aa" %% IV - "6a7a9538534f7da1e4c303d2a318a728" - "c3c0c95156809539fcf0e2429a6b5254" - "16aedbf5a0de6a57a637b39b"), - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("8ce24998625615b603a033aca13fb894" %% CipherText - "be9112a5c3a211a8ba262a3cca7e2ca7" - "01e4a9a4fba43c90ccdcb281d48c7c6f" - "d62875d2aca417034c34aee5"), - hexstr2bin("619cc5aefffe0bfa462af43c1699d050")}, %% CipherTag - - %% Test Case 7 - {aes_gcm, hexstr2bin("00000000000000000000000000000000" %% Key - "0000000000000000"), - hexstr2bin(""), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin(""), %% CipherText - hexstr2bin("cd33b28ac773f74ba00ed1f312572435")}, %% CipherTag - - %% Test Case 8 - {aes_gcm, hexstr2bin("00000000000000000000000000000000" %% Key - "0000000000000000"), - hexstr2bin("00000000000000000000000000000000"), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin("98e7247c07f0fe411c267e4384b0f600"), %% CipherText - hexstr2bin("2ff58d80033927ab8ef4d4587514f0fb")}, %% CipherTag - - %% Test Case 9 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b391aafd255"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin(""), %% ADD - hexstr2bin("3980ca0b3c00e841eb06fac4872a2757" %% CipherText - "859e1ceaa6efd984628593b40ca1e19c" - "7d773d00c144c525ac619d18c84a3f47" - "18e2448b2fe324d9ccda2710acade256"), - hexstr2bin("9924a7c8587336bfb118024db8674a14")}, %% CipherTag - - %% Test Case 10 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("3980ca0b3c00e841eb06fac4872a2757" %% CipherText - "859e1ceaa6efd984628593b40ca1e19c" - "7d773d00c144c525ac619d18c84a3f47" - "18e2448b2fe324d9ccda2710"), - hexstr2bin("2519498e80f1478f37ba55bd6d27618c")}, %% CipherTag - - %% Test Case 11 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbad"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("0f10f599ae14a154ed24b36e25324db8" %% CipherText - "c566632ef2bbb34f8347280fc4507057" - "fddc29df9a471f75c66541d4d4dad1c9" - "e93a19a58e8b473fa0f062f7"), - hexstr2bin("65dcc57fcf623a24094fcca40d3533f8")}, %% CipherTag - - %% Test Case 12 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("9313225df88406e555909c5aff5269aa" %% IV - "6a7a9538534f7da1e4c303d2a318a728" - "c3c0c95156809539fcf0e2429a6b5254" - "16aedbf5a0de6a57a637b39b"), - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("d27e88681ce3243c4830165a8fdcf9ff" %% CipherText - "1de9a1d8e6b447ef6ef7b79828666e45" - "81e79012af34ddd9e2f037589b292db3" - "e67c036745fa22e7e9b7373b"), - hexstr2bin("dcf566ff291c25bbb8568fc3d376a6d9")}, %% CipherTag - - %% Test Case 13 - {aes_gcm, hexstr2bin("00000000000000000000000000000000" %% Key - "00000000000000000000000000000000"), - hexstr2bin(""), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin(""), %% CipherText - hexstr2bin("530f8afbc74536b9a963b4f1c4cb738b")}, %% CipherTag - - %% Test Case 14 - {aes_gcm, hexstr2bin("00000000000000000000000000000000" %% Key - "00000000000000000000000000000000"), - hexstr2bin("00000000000000000000000000000000"), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin("cea7403d4d606b6e074ec5d3baf39d18"), %% CipherText - hexstr2bin("d0d1c8a799996bf0265b98b5d48ab919")}, %% CipherTag - - %% Test Case 15 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c6d6a8f9467308308"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b391aafd255"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin("522dc1f099567d07f47f37a32a84427d" %% CipherText - "643a8cdcbfe5c0c97598a2bd2555d1aa" - "8cb08e48590dbb3da7b08b1056828838" - "c5f61e6393ba7a0abcc9f662898015ad"), - hexstr2bin("b094dac5d93471bdec1a502270e3cc6c")}, %% CipherTag - - %% Test Case 16 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c6d6a8f9467308308"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbaddecaf888"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("522dc1f099567d07f47f37a32a84427d" %% CipherText - "643a8cdcbfe5c0c97598a2bd2555d1aa" - "8cb08e48590dbb3da7b08b1056828838" - "c5f61e6393ba7a0abcc9f662"), - hexstr2bin("76fc6ece0f4e1768cddf8853bb2d551b")}, %% CipherTag - - %% Test Case 17 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c6d6a8f9467308308"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("cafebabefacedbad"), %% IV - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("c3762df1ca787d32ae47c13bf19844cb" %% CipherText - "af1ae14d0b976afac52ff7d79bba9de0" - "feb582d33934a4f0954cc2363bc73f78" - "62ac430e64abe499f47c9b1f"), - hexstr2bin("3a337dbf46a792c45e454913fe2ea8f2")}, %% CipherTag - - %% Test Case 18 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c6d6a8f9467308308"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("9313225df88406e555909c5aff5269aa" %% IV - "6a7a9538534f7da1e4c303d2a318a728" - "c3c0c95156809539fcf0e2429a6b5254" - "16aedbf5a0de6a57a637b39b"), - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("5a8def2f0c9e53f1f75d7853659e2a20" %% CipherText - "eeb2b22aafde6419a058ab4f6f746bf4" - "0fc0c3b780f244452da3ebf1c5d82cde" - "a2418997200ef82e44ae7e3f"), - hexstr2bin("a44a8266ee1c8eb0c8b5d4cf5ae9f19a")}, %% CipherTag - - %% Test Case 0 for TagLength = 1 - {aes_gcm, hexstr2bin("00000000000000000000000000000000"), %% Key - hexstr2bin(""), %% PlainText - hexstr2bin("000000000000000000000000"), %% IV - hexstr2bin(""), %% AAD - hexstr2bin(""), %% CipherText - hexstr2bin("58"), %% CipherTag - 1}, %% TagLength - - %% Test Case 18 for TagLength = 1 - {aes_gcm, hexstr2bin("feffe9928665731c6d6a8f9467308308" %% Key - "feffe9928665731c6d6a8f9467308308"), - hexstr2bin("d9313225f88406e5a55909c5aff5269a" %% PlainText - "86a7a9531534f7da2e4c303d8a318a72" - "1c3c0c95956809532fcf0e2449a6b525" - "b16aedf5aa0de657ba637b39"), - hexstr2bin("9313225df88406e555909c5aff5269aa" %% IV - "6a7a9538534f7da1e4c303d2a318a728" - "c3c0c95156809539fcf0e2429a6b5254" - "16aedbf5a0de6a57a637b39b"), - hexstr2bin("feedfacedeadbeeffeedfacedeadbeef" %% AAD - "abaddad2"), - hexstr2bin("5a8def2f0c9e53f1f75d7853659e2a20" %% CipherText - "eeb2b22aafde6419a058ab4f6f746bf4" - "0fc0c3b780f244452da3ebf1c5d82cde" - "a2418997200ef82e44ae7e3f"), - hexstr2bin("a4"), %% CipherTag - 1} %% TagLength - ]. +aes_gcm(Config) -> + read_rsp(Config, aes_gcm, + ["gcmDecrypt128.rsp", + "gcmDecrypt192.rsp", + "gcmDecrypt256.rsp", + "gcmEncryptExtIV128.rsp", + "gcmEncryptExtIV192.rsp", + "gcmEncryptExtIV256.rsp"]). %% https://tools.ietf.org/html/rfc7539#appendix-A.5 chacha20_poly1305() -> @@ -2750,49 +2238,13 @@ ecc() -> end, TestCases). -%% Test data from Appendix D of NIST Special Publication 800-38B -%% http://csrc.nist.gov/publications/nistpubs/800-38B/Updated_CMAC_Examples.pdf -%% The same AES128 test data are also in the RFC 4493 -%% https://tools.ietf.org/html/rfc4493 -cmac_nist(aes_cbc128 = Type) -> - Key = hexstr2bin("2b7e151628aed2a6abf7158809cf4f3c"), - [{Type, Key, <<"">>, - hexstr2bin("bb1d6929e95937287fa37d129b756746")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a"), - hexstr2bin("070a16b46b4d4144f79bdd9dd04a287c")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411"), - hexstr2bin("dfa66747de9ae63030ca32611497c827")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710"), - hexstr2bin("51f0bebf7e3b9d92fc49741779363cfe")}, - % truncation - {Type, Key, <<"">>, 4, - hexstr2bin("bb1d6929")}]; - -cmac_nist(aes_cbc256 = Type) -> - Key = hexstr2bin("603deb1015ca71be2b73aef0857d7781" - "1f352c073b6108d72d9810a30914dff4"), - [{Type, Key, <<"">>, - hexstr2bin("028962f61b7bf89efc6b551f4667d983")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a"), - hexstr2bin("28a7023f452e8f82bd4bf28d8c37c35c")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411"), - hexstr2bin("aaf3d8f1de5640c232f5b169b9c911e6")}, - {Type, Key, hexstr2bin("6bc1bee22e409f96e93d7e117393172a" - "ae2d8a571e03ac9c9eb76fac45af8e51" - "30c81c46a35ce411e5fbc1191a0a52ef" - "f69f2445df4f9b17ad2b417be66c3710"), - hexstr2bin("e1992190549f6ed5696a2c056c315410")}, - % truncation - {Type, Key, <<"">>, 4, - hexstr2bin("028962f6")}]. +cmac_nist(Config, aes_cbc128 = Type) -> + read_rsp(Config, Type, + ["CMACGenAES128.rsp", "CMACVerAES128.rsp"]); +cmac_nist(Config, aes_cbc256 = Type) -> + read_rsp(Config, Type, + ["CMACGenAES256.rsp", "CMACVerAES256.rsp"]). no_padding() -> Public = [_, Mod] = rsa_public_stronger(), @@ -2813,3 +2265,123 @@ int_to_bin_neg(-1, Ds=[MSB|_]) when MSB >= 16#80 -> list_to_binary(Ds); int_to_bin_neg(X,Ds) -> int_to_bin_neg(X bsr 8, [(X band 255)|Ds]). + +datadir(Config) -> + proplists:get_value(data_dir, Config). + +-define(KiB, 1024). +-define(MiB, (1024 * 1024)). +-define(GiB, (1024 * 1024 * 1024)). + +fmt_words(Words) -> + BSize = Words * erlang:system_info(wordsize), + if BSize < ?KiB -> + integer_to_list(BSize); + BSize < ?MiB -> + io_lib:format("~8.2fKiB (~8w)", [BSize / ?KiB, BSize]); + BSize < ?GiB -> + io_lib:format("~8.2fMiB (~8w)", [BSize / ?MiB, BSize]); + true -> + io_lib:format("~8.2fGiB (~8w)", [BSize / ?GiB, BSize]) + end. + +log_rsp_size(Label, Term) -> + S = erts_debug:size(Term), + ct:pal("~s: ~w test(s), Memory used: ~s", + [Label, length(Term), fmt_words(S)]). + +read_rsp(Config, Type, Files) -> + Tests = + lists:foldl( + fun(FileName, Acc) -> + read_rsp_file(filename:join(datadir(Config), FileName), + Type, Acc) + end, [], Files), + log_rsp_size(Type, Tests), + Tests. + +read_rsp_file(FileName, Type, Acc) -> + {ok, Raw} = file:read_file(FileName), + Split = binary:split(Raw, [<<"\r">>, <<"\n">>], [global, trim_all]), + parse_rsp(Type, Split, Acc). + +parse_rsp(_Type, [], Acc) -> + Acc; +parse_rsp(_Type, [<<"DECRYPT">>|_], Acc) -> + Acc; +%% AES format +parse_rsp(Type, [<<"COUNT = ", _/binary>>, + <<"KEY = ", Key/binary>>, + <<"IV = ", IV/binary>>, + <<"PLAINTEXT = ", PlainText/binary>>, + <<"CIPHERTEXT = ", CipherText/binary>>|Next], Acc) -> + parse_rsp(Type, Next, [{Type, hexstr2bin(Key), hexstr2bin(IV), + hexstr2bin(PlainText), hexstr2bin(CipherText)}|Acc]); +%% CMAC format +parse_rsp(Type, [<<"Count = ", _/binary>>, + <<"Klen = ", _/binary>>, + <<"Mlen = ", Mlen/binary>>, + <<"Tlen = ", Tlen/binary>>, + <<"Key = ", Key/binary>>, + <<"Msg = ", Msg/binary>>, + <<"Mac = ", MAC/binary>>|Rest], Acc) -> + case Rest of + [<<"Result = P">>|Next] -> + parse_rsp_cmac(Type, Key, Msg, Mlen, Tlen, MAC, Next, Acc); + [<<"Result = ", _/binary>>|Next] -> + parse_rsp(Type, Next, Acc); + _ -> + parse_rsp_cmac(Type, Key, Msg, Mlen, Tlen, MAC, Rest, Acc) + end; +%% GCM format decode format +parse_rsp(Type, [<<"Count = ", _/binary>>, + <<"Key = ", Key/binary>>, + <<"IV = ", IV/binary>>, + <<"CT = ", CipherText/binary>>, + <<"AAD = ", AAD/binary>>, + <<"Tag = ", CipherTag0/binary>>, + <<"PT = ", PlainText/binary>>|Next], Acc) -> + CipherTag = hexstr2bin(CipherTag0), + TestCase = {Type, + hexstr2bin(Key), + hexstr2bin(PlainText), + hexstr2bin(IV), + hexstr2bin(AAD), + hexstr2bin(CipherText), + CipherTag, + size(CipherTag)}, + parse_rsp(Type, Next, [TestCase|Acc]); +%% GCM format encode format +parse_rsp(Type, [<<"Count = ", _/binary>>, + <<"Key = ", Key/binary>>, + <<"IV = ", IV/binary>>, + <<"PT = ", PlainText/binary>>, + <<"AAD = ", AAD/binary>>, + <<"CT = ", CipherText/binary>>, + <<"Tag = ", CipherTag0/binary>>|Next], Acc) -> + CipherTag = hexstr2bin(CipherTag0), + TestCase = {Type, + hexstr2bin(Key), + hexstr2bin(PlainText), + hexstr2bin(IV), + hexstr2bin(AAD), + hexstr2bin(CipherText), + CipherTag, + size(CipherTag)}, + parse_rsp(Type, Next, [TestCase|Acc]); + +parse_rsp(Type, [_|Next], Acc) -> + parse_rsp(Type, Next, Acc). + +parse_rsp_cmac(Type, Key0, Msg0, Mlen0, Tlen, MAC0, Next, Acc) -> + Key = hexstr2bin(Key0), + Mlen = binary_to_integer(Mlen0), + <<Msg:Mlen/bytes, _/binary>> = hexstr2bin(Msg0), + MAC = hexstr2bin(MAC0), + + case binary_to_integer(Tlen) of + 0 -> + parse_rsp(Type, Next, [{Type, Key, Msg, MAC}|Acc]); + I -> + parse_rsp(Type, Next, [{Type, Key, Msg, I, MAC}|Acc]) + end. diff --git a/lib/crypto/test/crypto_SUITE_data/KAT_AES.zip b/lib/crypto/test/crypto_SUITE_data/KAT_AES.zip Binary files differnew file mode 100644 index 0000000000..128a74c52e --- /dev/null +++ b/lib/crypto/test/crypto_SUITE_data/KAT_AES.zip diff --git a/lib/crypto/test/crypto_SUITE_data/aesmmt.zip b/lib/crypto/test/crypto_SUITE_data/aesmmt.zip Binary files differnew file mode 100644 index 0000000000..5024de1d06 --- /dev/null +++ b/lib/crypto/test/crypto_SUITE_data/aesmmt.zip diff --git a/lib/crypto/test/crypto_SUITE_data/cmactestvectors.zip b/lib/crypto/test/crypto_SUITE_data/cmactestvectors.zip Binary files differnew file mode 100644 index 0000000000..0d52444e57 --- /dev/null +++ b/lib/crypto/test/crypto_SUITE_data/cmactestvectors.zip diff --git a/lib/crypto/test/crypto_SUITE_data/gcmtestvectors.zip b/lib/crypto/test/crypto_SUITE_data/gcmtestvectors.zip Binary files differnew file mode 100644 index 0000000000..81eaa6c2f0 --- /dev/null +++ b/lib/crypto/test/crypto_SUITE_data/gcmtestvectors.zip diff --git a/lib/eldap/test/eldap_basic_SUITE.erl b/lib/eldap/test/eldap_basic_SUITE.erl index ac3447cfe6..4bfb0dd291 100644 --- a/lib/eldap/test/eldap_basic_SUITE.erl +++ b/lib/eldap/test/eldap_basic_SUITE.erl @@ -119,7 +119,10 @@ init_per_suite(Config) -> {ldaps_server, LDAPS_server} | Config]. end_per_suite(_Config) -> - ssl:stop(). + try ssl:stop() + catch + _:_ -> ok + end. init_per_group(return_values, Config) -> diff --git a/lib/erl_docgen/priv/xsl/db_html.xsl b/lib/erl_docgen/priv/xsl/db_html.xsl index 8a1a70448e..d863c056e9 100644 --- a/lib/erl_docgen/priv/xsl/db_html.xsl +++ b/lib/erl_docgen/priv/xsl/db_html.xsl @@ -95,7 +95,7 @@ <xsl:value-of select="$elem/@name"/> </xsl:when> <xsl:otherwise> - <xsl:value-of select="$elem"/> + <xsl:value-of select="substring-before($elem, '(')"/> </xsl:otherwise> </xsl:choose> </xsl:otherwise> diff --git a/lib/inets/src/http_client/httpc_handler.erl b/lib/inets/src/http_client/httpc_handler.erl index c99200777b..4b2bcc7242 100644 --- a/lib/inets/src/http_client/httpc_handler.erl +++ b/lib/inets/src/http_client/httpc_handler.erl @@ -1175,17 +1175,20 @@ handle_empty_queue(Session, ProfileName, TimeOut, State) -> %% If a pipline | keep_alive session has been idle for some time is not %% closed by the server, the client may want to close it. NewState = activate_queue_timeout(TimeOut, State), - update_session(ProfileName, Session, #session.queue_length, 0), - %% Note mfa will be initialized when a new request - %% arrives. - {noreply, - NewState#state{request = undefined, - mfa = undefined, - status_line = undefined, - headers = undefined, - body = undefined - } - }. + case update_session(ProfileName, Session, #session.queue_length, 0) of + {stop, Reason} -> + {stop, {shutdown, Reason}, State}; + _ -> + %% Note mfa will be initialized when a new request + %% arrives. + {noreply, + NewState#state{request = undefined, + mfa = undefined, + status_line = undefined, + headers = undefined, + body = undefined + }} + end. receive_response(Request, Session, Data, State) -> NewState = init_wait_for_response_state(Request, State), @@ -1224,7 +1227,7 @@ close_socket(#session{socket = Socket, socket_type = SocketType}) -> http_transport:close(SocketType, Socket). activate_request_timeout( - #state{request = #request{timer = undefined} = Request} = State) -> + #state{request = #request{timer = OldRef} = Request} = State) -> Timeout = (Request#request.settings)#http_options.timeout, case Timeout of infinity -> @@ -1232,17 +1235,21 @@ activate_request_timeout( _ -> ReqId = Request#request.id, Msg = {timeout, ReqId}, + case OldRef of + undefined -> + ok; + _ -> + %% Timer is already running! This is the case for a redirect or retry + %% We need to restart the timer because the handler pid has changed + cancel_timer(OldRef, Msg) + end, Ref = erlang:send_after(Timeout, self(), Msg), Request2 = Request#request{timer = Ref}, ReqTimers = [{Request#request.id, Ref} | (State#state.timers)#timers.request_timers], Timers = #timers{request_timers = ReqTimers}, State#state{request = Request2, timers = Timers} - end; - -%% Timer is already running! This is the case for a redirect or retry -activate_request_timeout(State) -> - State. + end. activate_queue_timeout(infinity, State) -> State; @@ -1673,7 +1680,7 @@ update_session(ProfileName, #session{id = SessionId} = Session, Pos, Value) -> Session2 = erlang:setelement(Pos, Session, Value), insert_session(Session2, ProfileName); error:badarg -> - exit(normal); %% Manager has been shutdown + {stop, normal}; T:E -> %% Unexpected this must be an error! Stacktrace = erlang:get_stacktrace(), @@ -1693,14 +1700,14 @@ update_session(ProfileName, #session{id = SessionId} = Session, Pos, Value) -> Session, (catch httpc_manager:lookup_session(SessionId, ProfileName)), T, E]), - exit({failed_updating_session, - [{profile, ProfileName}, - {session_id, SessionId}, - {pos, Pos}, - {value, Value}, - {etype, T}, - {error, E}, - {stacktrace, Stacktrace}]}) + {stop, {failed_updating_session, + [{profile, ProfileName}, + {session_id, SessionId}, + {pos, Pos}, + {value, Value}, + {etype, T}, + {error, E}, + {stacktrace, Stacktrace}]}} end. diff --git a/lib/inets/test/httpc_SUITE.erl b/lib/inets/test/httpc_SUITE.erl index fc7f01245b..e6dcd2285f 100644 --- a/lib/inets/test/httpc_SUITE.erl +++ b/lib/inets/test/httpc_SUITE.erl @@ -108,6 +108,7 @@ only_simulated() -> tolerate_missing_CR, userinfo, bad_response, + timeout_redirect, internal_server_error, invalid_http, invalid_chunk_size, @@ -785,6 +786,14 @@ bad_response(Config) when is_list(Config) -> ct:print("Wrong Statusline: ~p~n", [Reason]). %%------------------------------------------------------------------------- +timeout_redirect() -> + [{doc, "Test that timeout works for redirects, check ERL-420."}]. +timeout_redirect(Config) when is_list(Config) -> + URL = url(group_name(Config), "/redirect_to_missing_crlf.html", Config), + {error, timeout} = httpc:request(get, {URL, []}, [{timeout, 400}], []). + +%%------------------------------------------------------------------------- + internal_server_error(doc) -> ["Test 50X codes"]; internal_server_error(Config) when is_list(Config) -> @@ -1915,6 +1924,16 @@ handle_uri(_,"/missing_crlf.html",_,_,_,_) -> "Content-Length:32\r\n" ++ "<HTML><BODY>foobar</BODY></HTML>"; +handle_uri(_,"/redirect_to_missing_crlf.html",Port,_,Socket,_) -> + NewUri = url_start(Socket) ++ + integer_to_list(Port) ++ "/missing_crlf.html", + Body = "<HTML><BODY><a href=" ++ NewUri ++ + ">New place</a></BODY></HTML>", + "HTTP/1.1 303 See Other \r\n" ++ + "Location:" ++ NewUri ++ "\r\n" ++ + "Content-Length:" ++ integer_to_list(length(Body)) + ++ "\r\n\r\n" ++ Body; + handle_uri(_,"/wrong_statusline.html",_,_,_,_) -> "ok 200 HTTP/1.1\r\n\r\n" ++ "Content-Length:32\r\n\r\n" ++ diff --git a/lib/kernel/doc/src/inet.xml b/lib/kernel/doc/src/inet.xml index b7c904ff45..b71e8a1e5d 100644 --- a/lib/kernel/doc/src/inet.xml +++ b/lib/kernel/doc/src/inet.xml @@ -75,8 +75,8 @@ Address ip_address() ------- ------------ ::1 {0,0,0,0,0,0,0,1} ::192.168.42.2 {0,0,0,0,0,0,(192 bsl 8) bor 168,(42 bsl 8) bor 2} -FFFF::192.168.42.2 - {16#FFFF,0,0,0,0,0,(192 bsl 8) bor 168,(42 bsl 8) bor 2} +::FFFF:192.168.42.2 + {0,0,0,0,0,16#FFFF,(192 bsl 8) bor 168,(42 bsl 8) bor 2} 3ffe:b80:1f8d:2:204:acff:fe17:bf38 {16#3ffe,16#b80,16#1f8d,16#2,16#204,16#acff,16#fe17,16#bf38} fe80::204:acff:fe17:bf38 @@ -87,8 +87,8 @@ fe80::204:acff:fe17:bf38 <pre> 1> <input>inet:parse_address("192.168.42.2").</input> {ok,{192,168,42,2}} -2> <input>inet:parse_address("FFFF::192.168.42.2").</input> -{ok,{65535,0,0,0,0,0,49320,10754}}</pre> +2> <input>inet:parse_address("::FFFF:192.168.42.2").</input> +{ok,{0,0,0,0,0,65535,49320,10754}}</pre> </description> <datatypes> diff --git a/lib/kernel/doc/src/kernel_app.xml b/lib/kernel/doc/src/kernel_app.xml index 9fccb4c7ac..75e1e18d86 100644 --- a/lib/kernel/doc/src/kernel_app.xml +++ b/lib/kernel/doc/src/kernel_app.xml @@ -58,6 +58,7 @@ </section> <section> + <marker id="erl_signal_server"/> <title>OS Signal Event Handler</title> <p>Asynchronous OS signals may be subscribed to via the Kernel applications event manager (see <seealso marker="doc/design_principles:des_princ">OTP Design Principles</seealso> and diff --git a/lib/kernel/doc/src/net_kernel.xml b/lib/kernel/doc/src/net_kernel.xml index 4e2b0c69db..7ddb849824 100644 --- a/lib/kernel/doc/src/net_kernel.xml +++ b/lib/kernel/doc/src/net_kernel.xml @@ -64,6 +64,19 @@ $ <input>erl -sname foobar</input></pre> by the magic cookie system, see section <seealso marker="doc/reference_manual:distributed">Distributed Erlang</seealso> in the Erlang Reference Manual.</p> + <warning> + <p> + Starting a distributed node without also specifying + <seealso marker="erts:erl#proto_dist"><c>-proto_dist inet_tls</c></seealso> + will expose the node to attacks that may give the attacker + complete access to the node and in extension the cluster. + When using un-secure distributed nodes, make sure that the + network is configured to keep potential attackers out. + See the <seealso marker="ssl:ssl_distribution"> + Using SSL for Erlang Distribution</seealso> User's Guide + for details on how to setup a secure distributed node. + </p> + </warning> </description> <funcs> diff --git a/lib/kernel/doc/src/os.xml b/lib/kernel/doc/src/os.xml index 64c5cbe571..0e9add4161 100644 --- a/lib/kernel/doc/src/os.xml +++ b/lib/kernel/doc/src/os.xml @@ -174,8 +174,9 @@ DirOut = os:cmd("dir"), % on Win32 platform</code> <tag><c>handle</c></tag> <item> - This signal will notify <c>erl_signal_server</c> when it is received by - the Erlang runtime system. + This signal will notify + <seealso marker="kernel_app#erl_signal_server"><c>erl_signal_server</c></seealso> + when it is received by the Erlang runtime system. </item> </taglist> </desc> diff --git a/lib/kernel/src/dist_util.erl b/lib/kernel/src/dist_util.erl index 1c326afca8..b3507e5d13 100644 --- a/lib/kernel/src/dist_util.erl +++ b/lib/kernel/src/dist_util.erl @@ -572,12 +572,25 @@ recv_name(#hs_data{socket = Socket, f_recv = Recv}) -> ?shutdown(no_node) end. -get_name([$n,VersionA, VersionB, Flag1, Flag2, Flag3, Flag4 | OtherNode]) -> - {?u32(Flag1, Flag2, Flag3, Flag4), list_to_atom(OtherNode), - ?u16(VersionA,VersionB)}; +get_name([$n,VersionA, VersionB, Flag1, Flag2, Flag3, Flag4 | OtherNode] = Data) -> + case is_valid_name(OtherNode) of + true -> + {?u32(Flag1, Flag2, Flag3, Flag4), list_to_atom(OtherNode), + ?u16(VersionA,VersionB)}; + false -> + ?shutdown(Data) + end; get_name(Data) -> ?shutdown(Data). +is_valid_name(OtherNodeName) -> + case string:lexemes(OtherNodeName,"@") of + [_OtherNodeName,_OtherNodeHost] -> + true; + _else -> + false + end. + publish_type(Flags) -> case Flags band ?DFLAG_PUBLISHED of 0 -> diff --git a/lib/kernel/test/code_SUITE.erl b/lib/kernel/test/code_SUITE.erl index afc32283ba..6f8e949aac 100644 --- a/lib/kernel/test/code_SUITE.erl +++ b/lib/kernel/test/code_SUITE.erl @@ -1791,6 +1791,19 @@ do_normalized_paths([]) -> %% Test that module_status/1 behaves as expected module_status(_Config) -> + case test_server:is_cover() of + true -> + module_status(); + false -> + %% Make sure that we terminate the cover server. + try + module_status() + after + cover:stop() + end + end. + +module_status() -> %% basics not_loaded = code:module_status(fubar), % nonexisting {file, preloaded} = code:is_loaded(erlang), diff --git a/lib/kernel/test/erl_distribution_SUITE.erl b/lib/kernel/test/erl_distribution_SUITE.erl index d7a9ac39a3..bbfaa9d147 100644 --- a/lib/kernel/test/erl_distribution_SUITE.erl +++ b/lib/kernel/test/erl_distribution_SUITE.erl @@ -230,7 +230,7 @@ legal(Name) -> end. illegal(Name) -> - case test_node(Name) of + case test_node(Name, true) of not_started -> ok; started -> @@ -238,12 +238,20 @@ illegal(Name) -> end. test_node(Name) -> + test_node(Name, false). +test_node(Name, Illigal) -> ProgName = atom_to_list(lib:progname()), Command = ProgName ++ " -noinput " ++ long_or_short() ++ Name ++ - " -eval \"net_adm:ping('" ++ atom_to_list(node()) ++ "')\"", + " -eval \"net_adm:ping('" ++ atom_to_list(node()) ++ "')\"" ++ + case Illigal of + true -> + " -eval \"timer:sleep(10000),init:stop().\""; + false -> + "" + end, net_kernel:monitor_nodes(true), BinCommand = unicode:characters_to_binary(Command, utf8), - open_port({spawn, BinCommand}, [stream]), + Prt = open_port({spawn, BinCommand}, [stream]), Node = list_to_atom(Name), receive {nodeup, Node} -> diff --git a/lib/kernel/test/gen_tcp_api_SUITE.erl b/lib/kernel/test/gen_tcp_api_SUITE.erl index 92a74465b7..3f11e25b93 100644 --- a/lib/kernel/test/gen_tcp_api_SUITE.erl +++ b/lib/kernel/test/gen_tcp_api_SUITE.erl @@ -302,9 +302,9 @@ t_implicit_inet6(Config) when is_list(Config) -> end. t_implicit_inet6(Host, Addr) -> - case gen_tcp:listen(0, [inet6]) of + Loopback = {0,0,0,0,0,0,0,1}, + case gen_tcp:listen(0, [inet6, {ip,Loopback}]) of {ok,S1} -> - Loopback = {0,0,0,0,0,0,0,1}, io:format("~s ~p~n", ["::1",Loopback]), implicit_inet6(S1, Loopback), ok = gen_tcp:close(S1), @@ -524,10 +524,10 @@ local_handshake(S, SAddr, C, CAddr) -> t_accept_inet6_tclass(Config) when is_list(Config) -> TClassOpt = {tclass,8#56 bsl 2}, % Expedited forwarding - case gen_tcp:listen(0, [inet6,TClassOpt]) of + Loopback = {0,0,0,0,0,0,0,1}, + case gen_tcp:listen(0, [inet6, {ip, Loopback}, TClassOpt]) of {ok,L} -> LPort = ok(inet:port(L)), - Loopback = {0,0,0,0,0,0,0,1}, Sa = ok(gen_tcp:connect(Loopback, LPort, [])), Sb = ok(gen_tcp:accept(L)), [TClassOpt] = ok(inet:getopts(Sb, [tclass])), diff --git a/lib/kernel/test/gen_udp_SUITE.erl b/lib/kernel/test/gen_udp_SUITE.erl index 1029d7ef0a..836e0c5a05 100644 --- a/lib/kernel/test/gen_udp_SUITE.erl +++ b/lib/kernel/test/gen_udp_SUITE.erl @@ -717,9 +717,9 @@ implicit_inet6(Config) when is_list(Config) -> implicit_inet6(Host, Addr) -> Active = {active,false}, - case gen_udp:open(0, [inet6,Active]) of + Loopback = {0,0,0,0,0,0,0,1}, + case gen_udp:open(0, [inet6,Active,{ip, Loopback}]) of {ok,S1} -> - Loopback = {0,0,0,0,0,0,0,1}, io:format("~s ~p~n", ["::1",Loopback]), implicit_inet6(S1, Active, Loopback), ok = gen_udp:close(S1), diff --git a/lib/kernel/test/inet_sockopt_SUITE.erl b/lib/kernel/test/inet_sockopt_SUITE.erl index 322b9f30fe..9413cbd976 100644 --- a/lib/kernel/test/inet_sockopt_SUITE.erl +++ b/lib/kernel/test/inet_sockopt_SUITE.erl @@ -620,7 +620,7 @@ ipv6_v6only_close(Module, Socket) -> %% Test using socket option ipv6_v6only for UDP. use_ipv6_v6only_udp(Config) when is_list(Config) -> - case gen_udp:open(0, [inet6,{ipv6_v6only,true}]) of + case gen_udp:open(0, [inet6,{ip,{0,0,0,0,0,0,0,1}}, {ipv6_v6only,true}]) of {ok,S6} -> case inet:getopts(S6, [ipv6_v6only]) of {ok,[{ipv6_v6only,true}]} -> diff --git a/lib/kernel/test/os_SUITE.erl b/lib/kernel/test/os_SUITE.erl index 3cbb75a633..1233e362f2 100644 --- a/lib/kernel/test/os_SUITE.erl +++ b/lib/kernel/test/os_SUITE.erl @@ -319,31 +319,38 @@ perf_counter_api(_Config) -> true = is_integer(os:perf_counter()), true = os:perf_counter() > 0, - T1 = os:perf_counter(), + Conv = fun(T1, T2) -> + erlang:convert_time_unit(T2 - T1, perf_counter, nanosecond) + end, + + do_perf_counter_test([], Conv, 120000000, 80000000), + do_perf_counter_test([1000], fun(T1, T2) -> T2 - T1 end, 120, 80). + +do_perf_counter_test(CntArgs, Conv, Upper, Lower) -> + %% We run the test multiple times to try to get a somewhat + %% stable value... what does this test? That the + %% calculate_perf_counter_unit in sys_time.c works somewhat ok. + do_perf_counter_test(CntArgs, Conv, Upper, Lower, 10). + +do_perf_counter_test(CntArgs, _Conv, Upper, Lower, 0) -> + ct:fail("perf_counter_test ~p ~p ~p",[CntArgs, Upper, Lower]); +do_perf_counter_test(CntArgs, Conv, Upper, Lower, Iters) -> + + T1 = apply(os, perf_counter, CntArgs), timer:sleep(100), - T2 = os:perf_counter(), - TsDiff = erlang:convert_time_unit(T2 - T1, perf_counter, nanosecond), - ct:pal("T1: ~p~n" + T2 = apply(os, perf_counter, CntArgs), + TsDiff = Conv(T1, T2), + ct:log("T1: ~p~n" "T2: ~p~n" "TsDiff: ~p~n", [T1,T2,TsDiff]), - %% We allow a 15% diff - true = TsDiff < 115000000, - true = TsDiff > 85000000, - - T1Ms = os:perf_counter(1000), - timer:sleep(100), - T2Ms = os:perf_counter(1000), - MsDiff = T2Ms - T1Ms, - ct:pal("T1Ms: ~p~n" - "T2Ms: ~p~n" - "MsDiff: ~p~n", - [T1Ms,T2Ms,MsDiff]), - - %% We allow a 15% diff - true = MsDiff < 115, - true = MsDiff > 85. + if + TsDiff < Upper, TsDiff > Lower -> + ok; + true -> + do_perf_counter_test(CntArgs, Conv, Upper, Lower, Iters-1) + end. %% Util functions diff --git a/lib/kernel/test/sendfile_SUITE.erl b/lib/kernel/test/sendfile_SUITE.erl index 2673c38494..e839959623 100644 --- a/lib/kernel/test/sendfile_SUITE.erl +++ b/lib/kernel/test/sendfile_SUITE.erl @@ -100,13 +100,13 @@ init_per_testcase(TC,Config) when TC == t_sendfile_recvduring; %% Check if sendfile is supported on this platform case catch sendfile_send(Send) of ok -> - Config; + init_per_testcase(t_sendfile, Config); Error -> ct:log("Error: ~p",[Error]), {skip,"Not supported"} end; init_per_testcase(_Tc,Config) -> - Config. + Config ++ [{sendfile_opts,[{use_threads,false}]}]. t_sendfile_small(Config) when is_list(Config) -> diff --git a/lib/mnesia/src/mnesia_controller.erl b/lib/mnesia/src/mnesia_controller.erl index 17b47c059e..93a7e21f4e 100644 --- a/lib/mnesia/src/mnesia_controller.erl +++ b/lib/mnesia/src/mnesia_controller.erl @@ -1888,7 +1888,7 @@ info_format(Tab, Size, Mem, Media) -> StrT = mnesia_lib:pad_name(atom_to_list(Tab), 15, []), StrS = mnesia_lib:pad_name(integer_to_list(Size), 8, []), StrM = mnesia_lib:pad_name(integer_to_list(Mem), 8, []), - io:format("~s: with ~s records occupying ~s ~s~n", + io:format("~ts: with ~s records occupying ~s ~s~n", [StrT, StrS, StrM, Media]). %% Handle early arrived messages diff --git a/lib/observer/src/observer_lib.erl b/lib/observer/src/observer_lib.erl index 7352af936c..c7ee294719 100644 --- a/lib/observer/src/observer_lib.erl +++ b/lib/observer/src/observer_lib.erl @@ -636,12 +636,12 @@ user_term_multiline(Parent, Title, Default) -> parse_string(Str) -> try - Tokens = case erl_scan:string(Str) of + Tokens = case erl_scan:string(Str, 1, [text]) of {ok, Ts, _} -> Ts; {error, {_SLine, SMod, SError}, _} -> throw(io_lib:format("~s", [SMod:format_error(SError)])) end, - case erl_parse:parse_term(Tokens) of + case lib:extended_parse_term(Tokens) of {error, {_PLine, PMod, PError}} -> throw(io_lib:format("~s", [PMod:format_error(PError)])); Res -> Res diff --git a/lib/orber/test/multi_ORB_SUITE.erl b/lib/orber/test/multi_ORB_SUITE.erl index d739e47cc1..8becc11d6a 100644 --- a/lib/orber/test/multi_ORB_SUITE.erl +++ b/lib/orber/test/multi_ORB_SUITE.erl @@ -135,13 +135,12 @@ cases() -> setup_multi_connection_timeout_api, setup_multi_connection_timeout_attempts_api, setup_multi_connection_timeout_random_api, - ssl_1_multi_orber_api, ssl_1_multi_orber_generation_3_api, - ssl_2_multi_orber_api, ssl_2_multi_orber_generation_3_api, - ssl_reconfigure_api, ssl_reconfigure_generation_3_api]. +% ssl_1_multi_orber_api,ssl_2_multi_orber_api,ssl_reconfigure_api, + %%----------------------------------------------------------------- %% Init and cleanup functions. %%----------------------------------------------------------------- diff --git a/lib/orber/test/orber_test_lib.erl b/lib/orber/test/orber_test_lib.erl index 95ab26cd30..4a247ce492 100644 --- a/lib/orber/test/orber_test_lib.erl +++ b/lib/orber/test/orber_test_lib.erl @@ -356,11 +356,17 @@ get_options(ssl, Role, Level) -> get_options(ssl, Role, 2, Options) -> Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), - [{depth, 2}, - {verify, 2}, - {keyfile, filename:join([Dir, Role, "key.pem"])}, - {cacertfile, filename:join([Dir, Role, "cacerts.pem"])}, - {certfile, filename:join([Dir, Role, "cert.pem"])} |Options]; + Options1 = [{depth, 2}, + {verify, 2}, + {keyfile, filename:join([Dir, Role, "key.pem"])}, + {cacertfile, filename:join([Dir, Role, "cacerts.pem"])}, + {certfile, filename:join([Dir, Role, "cert.pem"])} |Options], + case Role of + client -> + [{server_name_indication, disable} |Options1]; + server -> + Options1 + end; get_options(iiop_ssl, _Role, 2, Options) -> Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), [{ssl_server_options, [{depth, 2}, @@ -369,10 +375,11 @@ get_options(iiop_ssl, _Role, 2, Options) -> {cacertfile, filename:join([Dir, "server", "cacerts.pem"])}, {keyfile, filename:join([Dir, "server", "key.pem"])}]}, {ssl_client_options, [{depth, 2}, - {verify, 2}, - {certfile, filename:join([Dir, "client", "cert.pem"])}, - {cacertfile, filename:join([Dir, "client", "cacerts.pem"])}, - {keyfile, filename:join([Dir, "client", "key.pem"])}]}, + {verify, 2}, + {server_name_indication, disable}, + {certfile, filename:join([Dir, "client", "cert.pem"])}, + {cacertfile, filename:join([Dir, "client", "cacerts.pem"])}, + {keyfile, filename:join([Dir, "client", "key.pem"])}]}, {secure, ssl} |Options]; get_options(iiop_ssl, _Role, 1, Options) -> Dir = filename:join([code:lib_dir(ssl), "examples", "certs", "etc"]), @@ -382,10 +389,11 @@ get_options(iiop_ssl, _Role, 1, Options) -> {cacertfile, filename:join([Dir, "server", "cacerts.pem"])}, {keyfile, filename:join([Dir, "server", "key.pem"])}]}, {ssl_client_options, [{depth, 1}, - {verify, 0}, - {certfile, filename:join([Dir, "client", "cert.pem"])}, - {cacertfile, filename:join([Dir, "client", "cacerts.pem"])}, - {keyfile, filename:join([Dir, "client", "key.pem"])}]}, + {verify, 0}, + {server_name_indication, disable}, + {certfile, filename:join([Dir, "client", "cert.pem"])}, + {cacertfile, filename:join([Dir, "client", "cacerts.pem"])}, + {keyfile, filename:join([Dir, "client", "key.pem"])}]}, {secure, ssl} |Options]. create_paths() -> diff --git a/lib/reltool/doc/src/reltool_examples.xml b/lib/reltool/doc/src/reltool_examples.xml index 83eee6017a..49bde56964 100644 --- a/lib/reltool/doc/src/reltool_examples.xml +++ b/lib/reltool/doc/src/reltool_examples.xml @@ -41,30 +41,31 @@ <p>The main process in Reltool is the server. It can be used as it is or be used via the GUI frontend process. When the GUI is started, a server process will automatically be started. The GUI - process is started with <c>reltool:start/0</c>, - <c>reltool:start/1</c> or <c>reltool:start_link/1</c>. The pid of - its server can be obtained with <c>reltool:get_server/1</c></p> + process is started with + <seealso marker="reltool#start-0"><c>reltool:start/0</c></seealso>, + <seealso marker="reltool#start-1"><c>reltool:start/1</c></seealso> or + <seealso marker="reltool#start_link-1"><c>reltool:start_link/1</c></seealso>. + The pid of its server can be obtained with + <seealso marker="reltool#start_link-1"><c>reltool:get_server/1</c></seealso> + </p> <pre> -Erlang R13B02 (erts-5.7.3) [source] [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] - -Eshell V5.7.3 (abort with ^G) +Erlang/OTP 20 [erts-9.0] [source-c13b302] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] +[hipe] [kernel-poll:false] +Eshell V9.0 (abort with ^G) +1> 1> {ok, Win} = reltool:start([]). {ok,<0.36.01>} -2> {ok, Server} = reltool:get_server([]). +2> {ok, Server} = reltool:get_server(Win). {ok,<0.37.01>} 3> reltool:get_config(Server). {ok,{sys,[]}} -4> reltool:stop(Win). -ok - - -5> {ok, Server2} = reltool:start_server([]). +4> +4> {ok, Server2} = reltool:start_server([]). {ok,<0.6535.01>} -6> reltool:get_config(Server2). +5> reltool:get_config(Server2). {ok,{sys,[]}} -7> reltool:stop(Server2). +6> reltool:stop(Server2). ok </pre> @@ -74,13 +75,11 @@ ok <title>Inspecting the configuration</title> <pre> -Erlang R13B02 (erts-5.7.3) [source] [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] - -Eshell V5.7.3 (abort with ^G) -1> Config = {sys, [{escript, - "examples/display_args", - [{incl_cond, include}]}, +Erlang/OTP 20 [erts-9.0] [source-c13b302] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] +[hipe] [kernel-poll:false] +Eshell V9.0 (abort with ^G) +1> +1> Config = {sys, [{escript, "examples/display_args", [{incl_cond, include}]}, {app, inets, [{incl_cond, include}]}, {app, mnesia, [{incl_cond, exclude}]}, {app, ssl, [{incl_cond, exclude}]}, @@ -92,88 +91,105 @@ Eshell V5.7.3 (abort with ^G) {app,ssl,[{incl_cond,exclude}]}, {app,runtime_tools,[{incl_cond,exclude}]}, {app,syntax_tools,[{incl_cond,exclude}]}]} - - - +2> 2> {ok, Server} = reltool:start_server([Config]). -{ok,<0.35.0>} +{ok,<0.66.0>} +3> 3> reltool:get_config(Server). -{ok,{sys,[{escript,"/clearcase/otp/tools/reltool/examples/display_args", - [{incl_cond,include}]}]}} +{ok,{sys,[{escript,"/usr/local/lib/erlang/lib/reltool-0.7.3/examples/display_args", + [{incl_cond,include}]}, + {app,inets,[{incl_cond,include}]}, + {app,mnesia,[{incl_cond,exclude}]}, + {app,runtime_tools,[{incl_cond,exclude}]}, + {app,ssl,[{incl_cond,exclude}]}, + {app,syntax_tools,[{incl_cond,exclude}]}]}} +4> 4> reltool:get_config(Server, false, false). -{ok,{sys,[{escript,"/clearcase/otp/tools/reltool/examples/display_args", - [{incl_cond,include}]}]}} - - - +{ok,{sys,[{escript,"/usr/local/lib/erlang/lib/reltool-0.7.3/examples/display_args", + [{incl_cond,include}]}, + {app,inets,[{incl_cond,include}]}, + {app,mnesia,[{incl_cond,exclude}]}, + {app,runtime_tools,[{incl_cond,exclude}]}, + {app,ssl,[{incl_cond,exclude}]}, + {app,syntax_tools,[{incl_cond,exclude}]}]}} +5> 5> reltool:get_config(Server, true, false). -{ok,{sys,[{root_dir,"/ldisk/hakan/otp_test"}, +{ok,{sys,[{root_dir,"/usr/local/lib/erlang"}, {lib_dirs,[]}, - {escript,"/clearcase/otp/tools/reltool/examples/display_args", + {escript,"/usr/local/lib/erlang/lib/reltool-0.7.3/examples/display_args", [{incl_cond,include}]}, {mod_cond,all}, {incl_cond,derived}, + {app,inets, + [{incl_cond,include},{vsn,undefined},{lib_dir,undefined}]}, + {app,mnesia,[{incl_cond,exclude}]}, + {app,runtime_tools,[{incl_cond,exclude}]}, + {app,ssl,[{incl_cond,exclude}]}, + {app,syntax_tools,[{incl_cond,exclude}]}, {boot_rel,"start_clean"}, + {rel,"start_clean","1.0",[]}, + {rel,"start_sasl","1.0",[sasl]}, {emu_name,"beam"}, {relocatable,true}, {profile,development}, - {incl_sys_files,[".*"]}, - {excl_sys_files,[]}, - {incl_app_files,[".*"]}, - {excl_app_files,[]}, - {incl_archive_dirs,[".*"]}, - {excl_archive_dirs,["^include$","^priv$"]}, + {incl_sys_filters,[".*"]}, + {excl_sys_filters,[]}, + {incl_app_filters,[".*"]}, + {excl_app_filters,[]}, + {incl_archive_filters,[".*"]}, + {excl_archive_filters,[[...]|...]}, {archive_opts,[]}, - {app_type,permanent}, - {app_file,keep}, - {debug_info,keep}]}} - - - + {rel_app_type,...}, + {...}|...]}} +6> 6> reltool:get_config(Server, true, true). -{ok,{sys,[{root_dir,"/ldisk/hakan/otp_test"}, +{ok,{sys,[{root_dir,"/usr/local/lib/erlang"}, {lib_dirs,[]}, - {escript,"/clearcase/otp/tools/reltool/examples/display_args", + {escript,"/usr/local/lib/erlang/lib/reltool-0.7.3/examples/display_args", [{incl_cond,include}]}, {mod_cond,all}, {incl_cond,derived}, - {erts,[{vsn,"5.7.3"}, - {mod,erl_prim_loader,[]}, - {mod,erlang,[]}, - {mod,init,[]}, - {mod,otp_ring0,[]}, - {mod,prim_file,[]}, - {mod,prim_inet,[]}, - {mod,prim_zip,[]}, - {mod,zlib,[]}]}, + {erts,[{app,erts, + [{vsn,"9.0"}, + {lib_dir,"/usr/local/lib/erlang/lib/erts-9.0"}, + {mod,erl_prim_loader,[]}, + {mod,erl_tracer,[]}, + {mod,erlang,[]}, + {mod,erts_code_purger,[]}, + {mod,erts_dirty_process_code_checker,[]}, + {mod,erts_internal,[]}, + {mod,erts_literal_area_collector,[]}, + {mod,init,[]}, + {mod,otp_ring0,...}, + {mod,...}, + {...}|...]}]}, {app,compiler, - [{vsn,"4.6.3"}, + [{vsn,"7.0.4"}, + {lib_dir,"/usr/local/lib/erlang/lib/compiler-7.0.4"}, + {mod,beam_a,[]}, {mod,beam_asm,[]}, {mod,beam_block,[]}, - {mod,beam_bool,[]}, + {mod,beam_bs,[]}, {mod,beam_bsm,[]}, {mod,beam_clean,[]}, {mod,beam_dead,[]}, {mod,beam_dict,[]}, {mod,beam_disasm,[]}, - {mod,beam_flatten,[]}, - {mod,beam_jump,[]}, - {mod,beam_listing,[]}, - {mod,beam_opcodes,...}, + {mod,beam_except,[]}, + {mod,beam_flatten,...}, {mod,...}, {...}|...]}, {app,crypto, - [{vsn,"1.6.1"}, + [{vsn,"3.7.4"}, + {lib_dir,"/usr/local/lib/erlang/lib/crypto-3.7.4"}, {mod,crypto,[]}, - {mod,crypto_app,[]}, - {mod,crypto_server,[]}, - {mod,crypto_sup,[]}]}, + {mod,crypto_ec_curves,[]}]}, {app,hipe, - [{vsn,"3.7.3"}, + [{vsn,"3.15.4"}, + {lib_dir,"/usr/local/lib/erlang/lib/hipe-3.15.4"}, {mod,cerl_cconv,[]}, {mod,cerl_closurean,[]}, {mod,cerl_hipeify,[]}, - {mod,cerl_hybrid_transform,[]}, {mod,cerl_lib,[]}, {mod,cerl_messagean,[]}, {mod,cerl_pmatch,[]}, @@ -182,65 +198,110 @@ Eshell V5.7.3 (abort with ^G) {mod,cerl_typean,...}, {mod,...}, {...}|...]}, + {app,inets, + [{incl_cond,include}, + {vsn,"6.3.9"}, + {lib_dir,"/usr/local/lib/erlang/lib/inets-6.3.9"}, + {mod,ftp,[]}, + {mod,ftp_progress,[]}, + {mod,ftp_response,[]}, + {mod,ftp_sup,[]}, + {mod,http_chunk,[]}, + {mod,http_request,[]}, + {mod,http_response,...}, + {mod,...}, + {...}|...]}, {app,kernel, - [{vsn,"2.13.3"}, + [{vsn,"5.2"}, + {lib_dir,"/usr/local/lib/erlang/lib/kernel-5.2"}, {mod,application,[]}, {mod,application_controller,[]}, {mod,application_master,[]}, {mod,application_starter,[]}, {mod,auth,[]}, {mod,code,[]}, - {mod,code_server,[]}, - {mod,disk_log,[]}, - {mod,disk_log_1,...}, + {mod,code_server,...}, + {mod,...}, + {...}|...]}, + {app,mnesia,[{incl_cond,exclude}]}, + {app,runtime_tools,[{incl_cond,exclude}]}, + {app,sasl, + [{vsn,"3.0.3"}, + {lib_dir,"/usr/local/lib/erlang/lib/sasl-3.0.3"}, + {mod,alarm_handler,[]}, + {mod,erlsrv,[]}, + {mod,format_lib_supp,[]}, + {mod,misc_supp,...}, {mod,...}, {...}|...]}, + {app,ssl,[{incl_cond,exclude}]}, {app,stdlib, - [{vsn,"1.16.3"}, + [{vsn,"3.3"}, + {lib_dir,"/usr/local/lib/erlang/lib/stdlib-3.3"}, {mod,array,[]}, - {mod,base64,[]}, - {mod,beam_lib,[]}, - {mod,c,[]}, - {mod,calendar,[]}, - {mod,dets,[]}, - {mod,dets_server,[]}, - {mod,dets_sup,...}, + {mod,base64,...}, {mod,...}, {...}|...]}, + {app,syntax_tools,[{incl_cond,exclude}]}, + {app,tools, + [{vsn,"2.9.1"},{lib_dir,[...]},{mod,...},{...}|...]}, {boot_rel,"start_clean"}, + {rel,"start_clean","1.0",[]}, + {rel,"start_sasl","1.0",[...]}, {emu_name,"beam"}, {relocatable,true}, - {profile,development}, - {incl_sys_files,[".*"]}, - {excl_sys_files,[]}, - {incl_app_files,[".*"]}, - {excl_app_files,[]}, - {incl_archive_dirs,[".*"]}, - {excl_archive_dirs,["^include$",[...]]}, - {archive_opts,[]}, - {app_type,permanent}, - {app_file,...}, - {...}]}} - - - -7> reltool:get_config([{sys,[{profile, embedded}]}]). -{ok,{sys,[{profile,embedded}, + {profile,...}, + {...}|...]}} +7> +7> reltool:get_config([{sys, [{profile, embedded}]}], true, false). +{ok,{sys,[{root_dir,"/usr/local/lib/erlang"}, + {lib_dirs,[]}, + {mod_cond,all}, + {incl_cond,derived}, + {boot_rel,"start_clean"}, + {rel,"start_clean","1.0",[]}, + {rel,"start_sasl","1.0",[sasl]}, + {emu_name,"beam"}, + {relocatable,true}, + {profile,embedded}, {incl_sys_filters,["^bin","^erts","^lib","^releases"]}, {excl_sys_filters,["^bin/(erlc|dialyzer|typer)(|\\.exe)$", "^erts.*/bin/(erlc|dialyzer|typer)(|\\.exe)$", "^erts.*/bin/.*(debug|pdb)"]}, - {incl_app_filters,["^ebin","^include","^priv"]}]}} -8> reltool:get_config([{sys,[{profile, standalone}]}]). -{ok,{sys,[{profile,standalone}, + {incl_app_filters,["^ebin","^include","^priv"]}, + {excl_app_filters,[]}, + {incl_archive_filters,[".*"]}, + {excl_archive_filters,["^include$","^priv$"]}, + {archive_opts,[]}, + {rel_app_type,permanent}, + {embedded_app_type,load}, + {app_file,keep}, + {debug_info,keep}]}} +8> +8> reltool:get_config([{sys, [{profile, standalone}]}], true, false). +{ok,{sys,[{root_dir,"/usr/local/lib/erlang"}, + {lib_dirs,[]}, + {mod_cond,all}, + {incl_cond,derived}, + {boot_rel,"start_clean"}, + {rel,"start_clean","1.0",[]}, + {rel,"start_sasl","1.0",[sasl]}, + {emu_name,"beam"}, + {relocatable,true}, + {profile,standalone}, {incl_sys_filters,["^bin/(erl|epmd)(|\\.exe|\\.ini)$", "^bin/start(|_clean).boot$","^erts.*/bin","^lib$"]}, {excl_sys_filters,["^erts.*/bin/(erlc|dialyzer|typer)(|\\.exe)$", "^erts.*/bin/(start|escript|to_erl|run_erl)(|\\.exe)$", "^erts.*/bin/.*(debug|pdb)"]}, {incl_app_filters,["^ebin","^priv"]}, - {excl_app_filters,["^ebin/.*\\.appup$"]}]}} - + {excl_app_filters,["^ebin/.*\\.appup$"]}, + {incl_archive_filters,[".*"]}, + {excl_archive_filters,["^include$","^priv$"]}, + {archive_opts,[]}, + {rel_app_type,permanent}, + {app_file,keep}, + {debug_info,keep}]}} </pre> </section> @@ -248,43 +309,51 @@ Eshell V5.7.3 (abort with ^G) <section> <title>Generate release and script files</title> <pre> -5> {ok, Server} = reltool:start_server([{config, {sys, [{boot_rel, "NAME"}, +Erlang/OTP 20 [erts-9.0] [source-c13b302] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] +[hipe] [kernel-poll:false] +Eshell V9.0 (abort with ^G) +1> +1> {ok, Server} = reltool:start_server([{config, {sys, [{boot_rel, "NAME"}, {rel, "NAME", "VSN", [sasl]}]}}]). {ok,<0.1288.0>} -6> reltool:get_config(Server). +2> +2> reltool:get_config(Server). {ok,{sys,[{boot_rel,"NAME"}, {rel,"NAME","VSN",[sasl]}]}} -7> reltool:get_rel(Server, "NAME"). +3> +3> reltool:get_rel(Server, "NAME"). {ok,{release,{"NAME","VSN"}, - {erts,"5.7"}, - [{kernel,"2.13"},{stdlib,"1.16"},{sasl,"2.1.6"}]}} -8> reltool:get_script(Server, "NAME"). + {erts,"9.0"}, + [{kernel,"5.2"},{stdlib,"3.3"},{sasl,"3.0.3"}]}} +4> +4> reltool:get_script(Server, "NAME"). {ok,{script,{"NAME","VSN"}, - [{preLoaded,[erl_prim_loader,erlang,init,otp_ring0, - prim_eval,prim_file,prim_inet,prim_zip, - zlib]}, + [{preLoaded,[erl_prim_loader,erl_tracer,erlang, + erts_code_purger,erts_dirty_process_code_checker, + erts_internal,erts_literal_area_collector,init,otp_ring0, + prim_eval,prim_file,prim_inet,prim_zip,zlib]}, {progress,preloaded}, - {path,["$ROOT/lib/kernel-2.13/ebin", - "$ROOT/lib/stdlib-1.16/ebin"]}, + {path,["$ROOT/lib/kernel-5.2/ebin", + "$ROOT/lib/stdlib-3.3/ebin"]}, {primLoad,[error_handler]}, {kernel_load_completed}, {progress,kernel_load_completed}, - {path,["$ROOT/lib/kernel-2.13/ebin"]}, + {path,["$ROOT/lib/kernel-5.2/ebin"]}, {primLoad,[application,application_controller, application_master,application_starter,auth,code, code_server,disk_log,disk_log_1,disk_log_server, disk_log_sup,dist_ac,dist_util,erl_boot_server|...]}, - {path,["$ROOT/lib/stdlib-1.16/ebin"]}, - {primLoad,[array,base64,beam_lib,c,calendar,dets, + {path,["$ROOT/lib/stdlib-3.3/ebin"]}, + {primLoad,[array,base64,beam_lib,binary,c,calendar,dets, dets_server,dets_sup,dets_utils,dets_v9,dict|...]}, - {path,["$ROOT/lib/sasl-2.1.6/ebin"]}, + {path,["$ROOT/lib/sasl-3.0.3/ebin"]}, {primLoad,[alarm_handler,erlsrv,format_lib_supp,misc_supp, - overload,rb,rb_format_supp,release_handler, - release_handler_1,sasl|...]}, + rb,rb_format_supp,release_handler,release_handler_1,sasl, + sasl_report|...]}, {progress,modules_loaded}, - {path,["$ROOT/lib/kernel-2.13/ebin", - "$ROOT/lib/stdlib-1.16/ebin","$ROOT/lib/sasl-2.1.6/ebin"]}, + {path,["$ROOT/lib/kernel-5.2/ebin", + "$ROOT/lib/stdlib-3.3/ebin","$ROOT/lib/sasl-3.0.3/ebin"]}, {kernelProcess,heart,{heart,start,[]}}, {kernelProcess,error_logger,{error_logger,start_link,[]}}, {kernelProcess,application_controller, @@ -296,7 +365,8 @@ Eshell V5.7.3 (abort with ^G) {apply,{...}}, {apply,...}, {...}|...]}} -9> reltool:stop(Server). +5> +5> reltool:stop(Server). ok </pre> </section> @@ -304,13 +374,11 @@ ok <section> <title>Create a target system</title> <pre> -Erlang R13B02 (erts-5.7.3) [source] [64-bit] [smp:4:4] [rq:4] - [async-threads:0] [kernel-poll:false] - -Eshell V5.7.3 (abort with ^G) -1> Config = {sys, [{escript, - "examples/display_args", - [{incl_cond, include}]}, +Erlang/OTP 20 [erts-9.0] [source-c13b302] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] +[hipe] [kernel-poll:false] +Eshell V9.0 (abort with ^G) +1> +1> Config = {sys, [{escript, "examples/display_args", [{incl_cond, include}]}, {app, inets, [{incl_cond, include}]}, {app, mnesia, [{incl_cond, exclude}]}, {app, ssl, [{incl_cond, exclude}]}, @@ -322,156 +390,183 @@ Eshell V5.7.3 (abort with ^G) {app,ssl,[{incl_cond,exclude}]}, {app,runtime_tools,[{incl_cond,exclude}]}, {app,syntax_tools,[{incl_cond,exclude}]}]} - - - +2> 2> {ok, Spec} = reltool:get_target_spec([Config]). {ok,[{create_dir,"releases", - [{write_file,"start_erl.data","5.7.3 1.0"}, - {create_dir,"1.0", - [{write_file,"start_clean.rel", - [37,37,32,114,101,108,32,103,101,110,101|...]}, - {write_file,"start_clean.script", - [37,37,32,115,99,114,105,112,116,32|...]}, - {write_file,"start_clean.boot", - <<131,104,3,100,0,6,115,99,114,...>>}, - {write_file,"start_sasl.rel", - [37,37,32,114,101,108,32,103,101,110,101|...]}, - {write_file,"start_sasl.script", - [37,37,32,115,99,114,105,112,116,32|...]}, - {write_file,"start_sasl.boot", - <<131,104,3,100,0,6,115,99,114,...>>}]}]}, + [{write_file,"start_erl.data","9.0 1.0\n"}, + {create_dir,"1.0", + [{write_file,"start_clean.rel", + [37,37,32,114,101,108,32,103,101,110,101,114,97,116|...]}, + {write_file,"start_clean.script", + [37,37,32,115,99,114,105,112,116,32,103,101,110|...]}, + {write_file,"start_clean.boot", + <<131,104,3,119,6,115,99,114,105,112,116,104,...>>}, + {write_file,"start_sasl.rel", + [37,37,32,114,101,108,32,103,101,110,101|...]}, + {write_file,"start_sasl.script", + [37,37,32,115,99,114,105,112,116,32|...]}, + {write_file,"start_sasl.boot", + <<131,104,3,119,6,115,99,114,105,...>>}]}]}, {create_dir,"bin", - [{copy_file,"display_args.escript", - "/clearcase/otp/tools/reltool/examples/display_args"}, - {copy_file,"display_args","erts-5.7.3/bin/escript"}, - {copy_file,"start","erts-5.7.3/bin/start"}, - {copy_file,"erl","erts-5.7.3/bin/dyn_erl"}, - {copy_file,"epmd","erts-5.7.3/bin/epmd"}, - {copy_file,"to_erl","erts-5.7.3/bin/to_erl"}, - {copy_file,"run_erl","erts-5.7.3/bin/run_erl"}, - {copy_file,"escript","erts-5.7.3/bin/escript"}, - {copy_file,"erlc","erts-5.7.3/bin/erlc"}, - {copy_file,"dialyzer","erts-5.7.3/bin/dialyzer"}, - {copy_file,"typer","erts-5.7.3/bin/typer"}, - {write_file,"start_clean.boot", - <<131,104,3,100,0,6,115,...>>}, - {write_file,"start_sasl.boot",<<131,104,3,100,0,6,...>>}, - {write_file,"start.boot",<<131,104,3,100,0,...>>}]}, - {create_dir,"misc", - [{copy_file,"makewhatis"},{copy_file,"format_man_pages"}]}, + [{copy_file,"display_args.escript", + "/usr/local/lib/erlang/lib/reltool-0.7.3/examples/display_args"}, + {copy_file,"display_args","erts-9.0/bin/escript"}, + {copy_file,"start","erts-9.0/bin/start"}, + {copy_file,"ct_run","erts-9.0/bin/ct_run"}, + {copy_file,"dialyzer","erts-9.0/bin/dialyzer"}, + {copy_file,"run_erl","erts-9.0/bin/run_erl"}, + {copy_file,"erl","erts-9.0/bin/dyn_erl"}, + {copy_file,"to_erl","erts-9.0/bin/to_erl"}, + {copy_file,"epmd","erts-9.0/bin/epmd"}, + {copy_file,"erlc","erts-9.0/bin/erlc"}, + {copy_file,"typer","erts-9.0/bin/typer"}, + {copy_file,"escript","erts-9.0/bin/escript"}, + {write_file,"start_clean.boot",<<131,104,3,119,6,115,...>>}, + {write_file,"start_sasl.boot",<<131,104,3,119,6,...>>}, + {write_file,"start.boot",<<131,104,3,119,...>>}]}, {copy_file,"Install"}, + {create_dir,"misc", + [{copy_file,"makewhatis"},{copy_file,"format_man_pages"}]}, {create_dir,"usr", - [{create_dir,"lib", - [{copy_file,"liberts_r.a"},{copy_file,"liberts.a"}]}, - {create_dir,"include", - [{copy_file,"erl_fixed_size_int_types.h"}, - {copy_file,"erl_int_sizes_config.h"}, - {copy_file,"erl_memory_trace_parser.h"}, - {create_dir,"obsolete",[{copy_file,"driver.h"}]}, - {copy_file,"driver_int.h"}, - {copy_file,"erl_driver.h"}]}]}, - {create_dir,"erts-5.7.3", - [{create_dir,"lib", - [{create_dir,"internal", - [{copy_file,"liberts_internal_r.a"}, - {copy_file,"liberts_internal.a"}, - {copy_file,"libethread.a"}, - {copy_file,"README"}]}, - {copy_file,"liberts_r.a"}, - {copy_file,"liberts.a"}]}, - {create_dir,"bin", - [{copy_file,"start"}, - {copy_file,"erl","erts-5.7.3/bin/dyn_erl"}, - {copy_file,"epmd"}, - {copy_file,"to_erl"}, - {copy_file,"run_erl"}, - {copy_file,"escript"}, - {copy_file,"erlc"}, - {copy_file,"dialyzer"}, - {copy_file,"typer"}, - {copy_file,"erlexec"}, - {copy_file,[...]}, - {copy_file,...}, - {...}|...]}, - {create_dir,"doc",[]}, - {create_dir,"man",[]}, - {create_dir,"include", - [{create_dir,"internal", - [{create_dir,"tile",[{copy_file,...},{...}]}, - {create_dir,"sparc64",[{...}]}, - {create_dir,"sparc32",[...]}, - {create_dir,[...],...}, - {create_dir,...}, - {...}|...]}, - {copy_file,"erl_fixed_size_int_types.h"}, - {copy_file,"erl_int_sizes_config.h"}, - {copy_file,"erl_memory_trace_parser.h"}, - {copy_file,"driver_int.h"}, - {copy_file,"erl_driver.h"}]}, - {create_dir,"src",[{copy_file,"setuid_socket_wrap.c"}]}]}, + [{create_dir,"lib", + [{copy_file,"liberts.a"}, + {copy_file,"liberl_interface_st.a"}, + {copy_file,"liberts_r.a"}, + {copy_file,"libic.a"}, + {copy_file,"liberl_interface.a"}, + {copy_file,"libei_st.a"}, + {copy_file,"libei.a"}]}, + {create_dir,"include", + [{copy_file,"erl_memory_trace_parser.h"}, + {copy_file,"driver_int.h"}, + {copy_file,"ei_connect.h"}, + {copy_file,"ei.h"}, + {copy_file,"erl_nif_api_funcs.h"}, + {copy_file,"erl_fixed_size_int_types.h"}, + {copy_file,"erl_int_sizes_config.h"}, + {copy_file,"erl_interface.h"}, + {copy_file,"eicode.h"}, + {copy_file,"erl_driver.h"}, + {copy_file,"erlang.idl"}, + {copy_file,[...]}, + {copy_file,...}, + {...}]}]}, + {create_dir,"erts-9.0", + [{create_dir,"bin", + [{copy_file,"start"}, + {copy_file,"ct_run"}, + {copy_file,"erlexec"}, + {copy_file,"dialyzer"}, + {copy_file,"beam.smp"}, + {copy_file,"run_erl"}, + {copy_file,"erl","erts-9.0/bin/dyn_erl"}, + {copy_file,"to_erl"}, + {copy_file,"epmd"}, + {copy_file,"erl_child_setup"}, + {copy_file,"heart"}, + {copy_file,[...]}, + {copy_file,...}, + {...}|...]}, + {create_dir,"lib", + [{create_dir,"internal", + [{copy_file,"liberts_internal.a"}, + {copy_file,"liberts_internal_r.a"}, + {copy_file,"libethread.a"}, + {copy_file,"README"}]}, + {copy_file,"liberts.a"}, + {copy_file,"liberts_r.a"}]}, + {create_dir,"src",[{copy_file,"setuid_socket_wrap.c"}]}, + {create_dir,"doc",[]}, + {create_dir,"man",[]}, + {create_dir,"include", + [{create_dir,"internal", + [{create_dir,"i386",[{...}|...]}, + {copy_file,"erl_errno.h"}, + {copy_file,[...]}, + {copy_file,...}, + {...}|...]}, + {copy_file,"erl_memory_trace_parser.h"}, + {copy_file,"driver_int.h"}, + {copy_file,"erl_nif_api_funcs.h"}, + {copy_file,"erl_fixed_size_int_types.h"}, + {copy_file,"erl_int_sizes_config.h"}, + {copy_file,[...]}, + {copy_file,...}, + {...}]}]}, {create_dir,"lib", - [{archive,"compiler-4.6.3.ez",[], - [{create_dir,"compiler-4.6.3", - [{create_dir,"ebin", - [{copy_file,"compiler.appup"}, - {copy_file,[...]}, - {copy_file,...}, - {...}|...]}, - {create_dir,"src", - [{copy_file,[...]}, - {copy_file,...},{...}|...]}]}]}, - {archive,"crypto-1.6.1.ez",[], - [{create_dir,"crypto-1.6.1", - [{create_dir,"ebin", - [{copy_file,[...]}, - {copy_file,...},{...}|...]}, - {create_dir,"src",[{copy_file,...},{...}|...]}]}]}, - {create_dir,"crypto-1.6.1", - [{create_dir,"priv", - [{create_dir,"lib",[{copy_file,[...]}]}, - {create_dir,"obj",[{copy_file,...},{...}]}]}]}, - {archive,"erts-5.7.3.ez",[], - [{create_dir,"erts-5.7.3", - [{create_dir,"ebin",[{...}|...]}, - {create_dir,"src",[...]}]}]}, - {archive,"hipe-3.7.3.ez",[], - [{create_dir,"hipe-3.7.3", - [{create_dir,"util",[...]}, - {create_dir,[...],...}, - {create_dir,...}, - {...}|...]}]}, - {archive,"kernel-2.13.3.ez",[], - [{create_dir,"kernel-2.13.3", - [{create_dir,[...],...},{create_dir,...},{...}]}]}, - {create_dir,"kernel-2.13.3", - [{create_dir,"include", - [{copy_file,[...]},{copy_file,...},{...}]}]}, - {archive,"stdlib-1.16.3.ez",[], - [{create_dir,"stdlib-1.16.3",[{...}|...]}]}, - {create_dir,"stdlib-1.16.3", - [{create_dir,"include",[{...}|...]}]}]}]} - - - -3> TargetDir = "my_target_dir". -"my_target_dir" + [{archive,"compiler-7.0.4.ez",[], + [{create_dir,"compiler-7.0.4", + [{create_dir,"src", + [{copy_file,"beam_flatten.erl"}, + {copy_file,[...]}, + {copy_file,...}, + {...}|...]}, + {create_dir,"ebin", + [{copy_file,[...]},{copy_file,...},{...}|...]}]}]}, + {archive,"crypto-3.7.4.ez",[], + [{create_dir,"crypto-3.7.4", + [{create_dir,"src",[{copy_file,[...]},{copy_file,...}]}, + {create_dir,"ebin",[{copy_file,...},{...}|...]}]}]}, + {create_dir,"crypto-3.7.4", + [{create_dir,"priv", + [{create_dir,"lib",[{copy_file,[...]},{copy_file,...}]}, + {create_dir,"obj",[{copy_file,...},{...}|...]}]}]}, + {archive,"erts-9.0.ez",[], + [{create_dir,"erts-9.0", + [{create_dir,"src",[{...}|...]}, + {create_dir,"ebin",[...]}]}]}, + {archive,"hipe-3.15.4.ez",[], + [{create_dir,"hipe-3.15.4", + [{create_dir,"flow",[...]}, + {copy_file,[...]}, + {create_dir,...}, + {...}|...]}]}, + {archive,"inets-6.3.9.ez",[], + [{create_dir,"inets-6.3.9", + [{create_dir,[...],...},{create_dir,...},{...}]}]}, + {create_dir,"inets-6.3.9", + [{create_dir,"priv",[{create_dir,[...],...}]}, + {create_dir,"include",[{copy_file,...},{...}]}]}, + {archive,"kernel-5.2.ez",[], + [{create_dir,"kernel-5.2",[{...}|...]}]}, + {create_dir,"kernel-5.2", + [{create_dir,"include",[{...}|...]}]}, + {archive,"sasl-3.0.3.ez",[],[{create_dir,[...],...}]}, + {archive,"stdlib-3.3.ez",[],[{create_dir,...}]}, + {create_dir,"stdlib-3.3",[{create_dir,...}]}, + {archive,"tools-2.9.1.ez",[],[...]}]}]} +3> +3> TargetDir = "/tmp/my_target_dir". +"/tmp/my_target_dir" +4> 4> reltool:eval_target_spec(Spec, code:root_dir(), TargetDir). -{error,"/clearcase/otp/tools/reltool/my_target_dir: no such file or directory"} -5> file:make_dir("my_target_dir"). +{error,"/tmp/my_target_dir: no such file or directory"} +5> +5> file:make_dir(TargetDir). ok +6> 6> reltool:eval_target_spec(Spec, code:root_dir(), TargetDir). ok +7> 7> file:list_dir(TargetDir). -{ok,["lib","erts-5.7.3","usr","Install","misc","bin","releases"]} +{ok,["bin","Install","lib","misc","usr","erts-9.0", + "releases"]} +8> 8> file:list_dir(filename:join([TargetDir,"lib"])). -{ok,["stdlib-1.16.3","stdlib-1.16.3.ez","kernel-2.13.3", - "kernel-2.13.3.ez","hipe-3.7.3.ez","erts-5.7.3.ez", - "crypto-1.6.1","crypto-1.6.1.ez","compiler-4.6.3.ez"]} -9> file:make_dir("yet_another_target_dir"). +{ok,["tools-2.9.1.ez","kernel-5.2.ez","inets-6.3.9.ez", + "kernel-5.2","sasl-3.0.3.ez","hipe-3.15.4.ez","inets-6.3.9", + "crypto-3.7.4","crypto-3.7.4.ez","stdlib-3.3.ez", + "erts-9.0.ez","stdlib-3.3","compiler-7.0.4.ez"]} +9> +9> file:make_dir("/tmp/yet_another_target_dir"). ok -10> reltool:create_target(Config, "yet_another_target_dir"). +10> +10> reltool:create_target([Config], "/tmp/yet_another_target_dir"). ok +11> +11> file:list_dir("/tmp/yet_another_target_dir"). +{ok,["bin","Install","lib","misc","usr","erts-9.0", + "releases"]} </pre> </section> diff --git a/lib/ssh/src/ssh_auth.erl b/lib/ssh/src/ssh_auth.erl index 6cf659f830..ac64a7bf14 100644 --- a/lib/ssh/src/ssh_auth.erl +++ b/lib/ssh/src/ssh_auth.erl @@ -28,7 +28,8 @@ -include("ssh_auth.hrl"). -include("ssh_transport.hrl"). --export([publickey_msg/1, password_msg/1, keyboard_interactive_msg/1, +-export([get_public_key/2, + publickey_msg/1, password_msg/1, keyboard_interactive_msg/1, service_request_msg/1, init_userauth_request_msg/1, userauth_request_msg/1, handle_userauth_request/3, handle_userauth_info_request/2, handle_userauth_info_response/2 @@ -136,41 +137,49 @@ keyboard_interactive_msg([#ssh{user = User, Ssh) end. -publickey_msg([SigAlg, #ssh{user = User, - session_id = SessionId, - service = Service, - opts = Opts} = Ssh]) -> - Hash = ssh_transport:sha(SigAlg), + +get_public_key(SigAlg, #ssh{opts = Opts}) -> KeyAlg = key_alg(SigAlg), {KeyCb,KeyCbOpts} = ?GET_OPT(key_cb, Opts), UserOpts = ?GET_OPT(user_options, Opts), case KeyCb:user_key(KeyAlg, [{key_cb_private,KeyCbOpts}|UserOpts]) of - {ok, PrivKey} -> - SigAlgStr = atom_to_list(SigAlg), + {ok, PrivKey} -> try Key = ssh_transport:extract_public_key(PrivKey), public_key:ssh_encode(Key, ssh2_pubkey) of - PubKeyBlob -> - SigData = build_sig_data(SessionId, User, Service, - PubKeyBlob, SigAlgStr), - Sig = ssh_transport:sign(SigData, Hash, PrivKey), - SigBlob = list_to_binary([?string(SigAlgStr), - ?binary(Sig)]), - ssh_transport:ssh_packet( - #ssh_msg_userauth_request{user = User, - service = Service, - method = "publickey", - data = [?TRUE, - ?string(SigAlgStr), - ?binary(PubKeyBlob), - ?binary(SigBlob)]}, - Ssh) + PubKeyBlob -> {ok,{PrivKey,PubKeyBlob}} catch _:_ -> - {not_ok, Ssh} + not_ok end; - _Error -> + _Error -> + not_ok + end. + + +publickey_msg([SigAlg, #ssh{user = User, + session_id = SessionId, + service = Service} = Ssh]) -> + case get_public_key(SigAlg, Ssh) of + {ok, {PrivKey,PubKeyBlob}} -> + SigAlgStr = atom_to_list(SigAlg), + SigData = build_sig_data(SessionId, User, Service, + PubKeyBlob, SigAlgStr), + Hash = ssh_transport:sha(SigAlg), + Sig = ssh_transport:sign(SigData, Hash, PrivKey), + SigBlob = list_to_binary([?string(SigAlgStr), + ?binary(Sig)]), + ssh_transport:ssh_packet( + #ssh_msg_userauth_request{user = User, + service = Service, + method = "publickey", + data = [?TRUE, + ?string(SigAlgStr), + ?binary(PubKeyBlob), + ?binary(SigBlob)]}, + Ssh); + _ -> {not_ok, Ssh} end. diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index f1ce337947..8d3ddb09a4 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -453,16 +453,20 @@ init_ssh_record(Role, _Socket, PeerAddr, Opts) -> PeerName0 when is_list(PeerName0) -> PeerName0 end, - S0#ssh{c_vsn = Vsn, - c_version = Version, - io_cb = case ?GET_OPT(user_interaction, Opts) of - true -> ssh_io; - false -> ssh_no_io - end, - userauth_pubkeys = ?GET_OPT(pref_public_key_algs, Opts), - userauth_quiet_mode = ?GET_OPT(quiet_mode, Opts), - peer = {PeerName, PeerAddr} - }; + S1 = + S0#ssh{c_vsn = Vsn, + c_version = Version, + io_cb = case ?GET_OPT(user_interaction, Opts) of + true -> ssh_io; + false -> ssh_no_io + end, + userauth_quiet_mode = ?GET_OPT(quiet_mode, Opts), + peer = {PeerName, PeerAddr} + }, + S1#ssh{userauth_pubkeys = [K || K <- ?GET_OPT(pref_public_key_algs, Opts), + is_usable_user_pubkey(K, S1) + ] + }; server -> S0#ssh{s_vsn = Vsn, @@ -1700,29 +1704,49 @@ handle_ssh_msg_ext_info(#ssh_msg_ext_info{data=Data}, D0) -> lists:foldl(fun ext_info/2, D0, Data). -ext_info({"server-sig-algs",SigAlgs}, D0 = #data{ssh_params=#ssh{role=client, - userauth_pubkeys=ClientSigAlgs}=Ssh0}) -> - %% Make strings to eliminate risk of beeing bombed with odd strings that fills the atom table: - SupportedAlgs = lists:map(fun erlang:atom_to_list/1, ssh_transport:supported_algorithms(public_key)), - ServerSigAlgs = [list_to_atom(SigAlg) || SigAlg <- string:tokens(SigAlgs,","), - %% length of SigAlg is implicitly checked by the comparison - %% in member/2: - lists:member(SigAlg, SupportedAlgs) - ], - CommonAlgs = [Alg || Alg <- ServerSigAlgs, - lists:member(Alg, ClientSigAlgs)], - SelectedAlgs = - case CommonAlgs of - [] -> ClientSigAlgs; % server-sig-algs value is just an advice - _ -> CommonAlgs - end, - D0#data{ssh_params = Ssh0#ssh{userauth_pubkeys = SelectedAlgs} }; +ext_info({"server-sig-algs",SigAlgsStr}, + D0 = #data{ssh_params=#ssh{role=client, + userauth_pubkeys=ClientSigAlgs}=Ssh0}) -> + %% ClientSigAlgs are the pub_key algortithms that: + %% 1) is usable, that is, the user has such a public key and + %% 2) is either the default list or set by the caller + %% with the client option 'pref_public_key_algs' + %% + %% The list is already checked for duplicates. + + SigAlgs = [A || Astr <- string:tokens(SigAlgsStr, ","), + A <- try [list_to_existing_atom(Astr)] + %% list_to_existing_atom will fail for unknown algorithms + catch _:_ -> [] + end], + + CommonAlgs = [A || A <- SigAlgs, + lists:member(A, ClientSigAlgs)], + + %% Re-arrange the client supported public-key algorithms so that the server + %% preferred ones are tried first. + %% Trying algorithms not mentioned by the server is ok, since the server can't know + %% if the client supports 'server-sig-algs' or not. + + D0#data{ + ssh_params = + Ssh0#ssh{ + userauth_pubkeys = + CommonAlgs ++ (ClientSigAlgs -- CommonAlgs) + }}; ext_info(_, D0) -> %% Not implemented D0. %%%---------------------------------------------------------------- +is_usable_user_pubkey(A, Ssh) -> + case ssh_auth:get_public_key(A, Ssh) of + {ok,_} -> true; + _ -> false + end. + +%%%---------------------------------------------------------------- handle_request(ChannelPid, ChannelId, Type, Data, WantReply, From, D) -> case ssh_channel:cache_lookup(cache(D), ChannelId) of #channel{remote_id = Id} = Channel -> diff --git a/lib/ssh/src/ssh_dbg.erl b/lib/ssh/src/ssh_dbg.erl index 7dfbfc3b4b..3f742ad9b6 100644 --- a/lib/ssh/src/ssh_dbg.erl +++ b/lib/ssh/src/ssh_dbg.erl @@ -22,9 +22,8 @@ -module(ssh_dbg). --export([messages/0, - messages/1, - messages/2, +-export([messages/0, messages/1, messages/2, messages/3, + auth/0, auth/1, auth/2, auth/3, stop/0 ]). @@ -36,98 +35,193 @@ -include("ssh_connect.hrl"). -include("ssh_auth.hrl"). --record(data, { - writer, - acc = []}). %%%================================================================ -messages() -> - messages(fun(String,_D) -> io:format(String) end). +messages() -> start(msg). +messages(F) -> start(msg,F). +messages(F,X) -> start(msg,F,X). +messages(F,M,I) -> start(msg,F,M,I). -messages(Write) when is_function(Write,2) -> - messages(Write, fun(X) -> X end). +auth() -> start(auth). +auth(F) -> start(auth,F). +auth(F,X) -> start(auth,F,X). +auth(F,M,I) -> start(auth,F,M,I). -messages(Write, MangleArg) when is_function(Write,2), - is_function(MangleArg,1) -> - catch dbg:start(), - setup_tracer(Write, MangleArg), - dbg:p(new,[c,timestamp]), - dbg_ssh_messages(). +stop() -> dbg:stop(). -dbg_ssh_messages() -> +%%%---------------------------------------------------------------- +start(Type) -> start(Type, fun io:format/2). + +start(Type, F) when is_function(F,2) -> start(Type, fmt_fun(F)); +start(Type, F) when is_function(F,3) -> start(Type, F, id_fun()). + +start(Type, WriteFun, MangleArgFun) when is_function(WriteFun, 3), + is_function(MangleArgFun, 1) -> + start(Type, WriteFun, MangleArgFun, []); +start(Type, WriteFun, InitValue) -> + start(Type, WriteFun, id_fun(), InitValue). + +start(Type, WriteFun, MangleArgFun, InitValue) when is_function(WriteFun, 3), + is_function(MangleArgFun, 1) -> + cond_start(Type, WriteFun, MangleArgFun, InitValue), + dbg_ssh(Type). + +%%%---------------------------------------------------------------- +fmt_fun(F) -> fun(Fmt,Args,Data) -> F(Fmt,Args), Data end. + +id_fun() -> fun(X) -> X end. + +%%%---------------------------------------------------------------- +dbg_ssh(msg) -> + dbg_ssh(auth), dbg:tp(ssh_message,encode,1, x), dbg:tp(ssh_message,decode,1, x), dbg:tpl(ssh_transport,select_algorithm,4, x), dbg:tp(ssh_transport,hello_version_msg,1, x), - dbg:tp(ssh_transport,handle_hello_version,1, x). + dbg:tp(ssh_transport,handle_hello_version,1, x), + dbg:tpl(ssh_connection_handler,ext_info,2, x); + +dbg_ssh(auth) -> + dbg:tp(ssh_transport,hello_version_msg,1, x), + dbg:tp(ssh_transport,handle_hello_version,1, x), + dbg:tp(ssh_message,encode,1, x), + dbg:tpl(ssh_transport,select_algorithm,4, x), + dbg:tpl(ssh_connection_handler,ext_info,2, x), + lists:foreach(fun(F) -> dbg:tp(ssh_auth, F, x) end, + [publickey_msg, password_msg, keyboard_interactive_msg]). -%%%---------------------------------------------------------------- -stop() -> - dbg:stop(). - %%%================================================================ -msg_formater({trace_ts,Pid,call,{ssh_message,encode,[Msg]},TS}, D) -> +cond_start(Type, WriteFun, MangleArgFun, Init) -> + try + dbg:start(), + setup_tracer(Type, WriteFun, MangleArgFun, Init), + dbg:p(new,[c,timestamp]) + catch + _:_ -> ok + end. + + +msg_formater(msg, {trace_ts,Pid,call,{ssh_message,encode,[Msg]},TS}, D) -> fmt("~n~s SEND ~p ~s~n", [ts(TS),Pid,wr_record(shrink_bin(Msg))], D); -msg_formater({trace_ts,_Pid,return_from,{ssh_message,encode,1},_Res,_TS}, D) -> +msg_formater(msg, {trace_ts,_Pid,return_from,{ssh_message,encode,1},_Res,_TS}, D) -> D; -msg_formater({trace_ts,_Pid,call,{ssh_message,decode,_},_TS}, D) -> +msg_formater(msg, {trace_ts,_Pid,call,{ssh_message,decode,_},_TS}, D) -> D; -msg_formater({trace_ts,Pid,return_from,{ssh_message,decode,1},Msg,TS}, D) -> +msg_formater(msg, {trace_ts,Pid,return_from,{ssh_message,decode,1},Msg,TS}, D) -> fmt("~n~s ~p RECV ~s~n", [ts(TS),Pid,wr_record(shrink_bin(Msg))], D); + +msg_formater(auth, {trace_ts,Pid,return_from,{ssh_message,decode,1},#ssh_msg_userauth_failure{authentications=As},TS}, D) -> + fmt("~n~s ~p Client login FAILURE. Try ~s~n", [ts(TS),Pid,As], D); -msg_formater({trace_ts,_Pid,call,{ssh_transport,select_algorithm,_},_TS}, D) -> +msg_formater(auth, {trace_ts,Pid,return_from,{ssh_message,decode,1},#ssh_msg_userauth_success{},TS}, D) -> + fmt("~n~s ~p Client login SUCCESS~n", [ts(TS),Pid], D); + + +msg_formater(_, {trace_ts,_Pid,call,{ssh_transport,select_algorithm,_},_TS}, D) -> D; -msg_formater({trace_ts,Pid,return_from,{ssh_transport,select_algorithm,_},{ok,Alg},TS}, D) -> +msg_formater(_, {trace_ts,Pid,return_from,{ssh_transport,select_algorithm,_},{ok,Alg},TS}, D) -> fmt("~n~s ~p ALGORITHMS~n~s~n", [ts(TS),Pid, wr_record(Alg)], D); -msg_formater({trace_ts,_Pid,call,{ssh_transport,hello_version_msg,_},_TS}, D) -> +msg_formater(_, {trace_ts,_Pid,call,{ssh_transport,hello_version_msg,_},_TS}, D) -> D; -msg_formater({trace_ts,Pid,return_from,{ssh_transport,hello_version_msg,1},Hello,TS}, D) -> +msg_formater(_, {trace_ts,Pid,return_from,{ssh_transport,hello_version_msg,1},Hello,TS}, D) -> fmt("~n~s ~p TCP SEND HELLO~n ~p~n", [ts(TS),Pid,lists:flatten(Hello)], D); -msg_formater({trace_ts,Pid,call,{ssh_transport,handle_hello_version,[Hello]},TS}, D) -> +msg_formater(_, {trace_ts,Pid,call,{ssh_transport,handle_hello_version,[Hello]},TS}, D) -> fmt("~n~s ~p RECV HELLO~n ~p~n", [ts(TS),Pid,lists:flatten(Hello)], D); -msg_formater({trace_ts,_Pid,return_from,{ssh_transport,handle_hello_version,1},_,_TS}, D) -> +msg_formater(_, {trace_ts,_Pid,return_from,{ssh_transport,handle_hello_version,1},_,_TS}, D) -> D; -msg_formater({trace_ts,Pid,send,{tcp,Sock,Bytes},Pid,TS}, D) -> +msg_formater(_, {trace_ts,Pid,call,{ssh_connection_handler,ext_info,[{"server-sig-algs",SigAlgs},State]},TS}, D) -> + try lists:keyfind(ssh, 1, tuple_to_list(State)) of + false -> + D; + #ssh{userauth_pubkeys = PKs} -> + fmt("~n~s ~p Client got suggestion to use user public key sig-algs~n ~p~n and can use~n ~p~n", + [ts(TS),Pid,string:tokens(SigAlgs,","),PKs], D) + catch + _:_ -> + D + end; + +msg_formater(_, {trace_ts,Pid,return_from,{ssh_connection_handler,ext_info,2},State,TS}, D) -> + try lists:keyfind(ssh, 1, tuple_to_list(State)) of + false -> + D; + #ssh{userauth_pubkeys = PKs} -> + fmt("~n~s ~p Client will try user public key sig-algs~n ~p~n", [ts(TS),Pid,PKs], D) + catch + _:_ -> + D + end; + +msg_formater(_, {trace_ts,Pid,call,{ssh_auth,publickey_msg,[[SigAlg,#ssh{user=User}]]},TS}, D) -> + fmt("~n~s ~p Client will try to login user ~p with public key algorithm ~p~n", [ts(TS),Pid,User,SigAlg], D); +msg_formater(_, {trace_ts,Pid,return_from,{ssh_auth,publickey_msg,1},{not_ok,#ssh{user=User}},TS}, D) -> + fmt("~s ~p User ~p can't login with that kind of public key~n", [ts(TS),Pid,User], D); + +msg_formater(_, {trace_ts,Pid,call,{ssh_auth,password_msg,[[#ssh{user=User}]]},TS}, D) -> + fmt("~n~s ~p Client will try to login user ~p with password~n", [ts(TS),Pid,User], D); +msg_formater(_, {trace_ts,Pid,return_from,{ssh_auth,password_msg,1},{not_ok,#ssh{user=User}},TS}, D) -> + fmt("~s ~p User ~p can't login with password~n", [ts(TS),Pid,User], D); + +msg_formater(_, {trace_ts,Pid,call,{ssh_auth,keyboard_interactive_msg,[[#ssh{user=User}]]},TS}, D) -> + fmt("~n~s ~p Client will try to login user ~p with password~n", [ts(TS),Pid,User], D); +msg_formater(_, {trace_ts,Pid,return_from,{ssh_auth,keyboard_interactive_msg,1},{not_ok,#ssh{user=User}},TS}, D) -> + fmt("~s ~p User ~p can't login with keyboard_interactive password~n", [ts(TS),Pid,User], D); + +msg_formater(msg, {trace_ts,Pid,send,{tcp,Sock,Bytes},Pid,TS}, D) -> fmt("~n~s ~p TCP SEND on ~p~n ~p~n", [ts(TS),Pid,Sock, shrink_bin(Bytes)], D); -msg_formater({trace_ts,Pid,send,{tcp,Sock,Bytes},Dest,TS}, D) -> +msg_formater(msg, {trace_ts,Pid,send,{tcp,Sock,Bytes},Dest,TS}, D) -> fmt("~n~s ~p TCP SEND from ~p TO ~p~n ~p~n", [ts(TS),Pid,Sock,Dest, shrink_bin(Bytes)], D); -msg_formater({trace_ts,Pid,send,ErlangMsg,Dest,TS}, D) -> +msg_formater(msg, {trace_ts,Pid,send,ErlangMsg,Dest,TS}, D) -> fmt("~n~s ~p ERL MSG SEND TO ~p~n ~p~n", [ts(TS),Pid,Dest, shrink_bin(ErlangMsg)], D); -msg_formater({trace_ts,Pid,'receive',{tcp,Sock,Bytes},TS}, D) -> +msg_formater(msg, {trace_ts,Pid,'receive',{tcp,Sock,Bytes},TS}, D) -> fmt("~n~s ~p TCP RECEIVE on ~p~n ~p~n", [ts(TS),Pid,Sock,shrink_bin(Bytes)], D); -msg_formater({trace_ts,Pid,'receive',ErlangMsg,TS}, D) -> +msg_formater(msg, {trace_ts,Pid,'receive',ErlangMsg,TS}, D) -> fmt("~n~s ~p ERL MSG RECEIVE~n ~p~n", [ts(TS),Pid,shrink_bin(ErlangMsg)], D); -msg_formater(M, D) -> - fmt("~nDBG ~n~p~n", [shrink_bin(M)], D). +%% msg_formater(_, {trace_ts,_Pid,return_from,MFA,_Ret,_TS}=M, D) -> +%% case lists:member(MFA, [{ssh_auth,keyboard_interactive_msg,1}, +%% {ssh_auth,password_msg,1}, +%% {ssh_auth,publickey_msg,1}]) of +%% true -> +%% D; +%% false -> +%% fmt("~nDBG ~n~p~n", [shrink_bin(M)], D) +%% end; + +%% msg_formater(_, M, D) -> +%% fmt("~nDBG ~n~p~n", [shrink_bin(M)], D). -%% msg_formater(_, D) -> -%% D. +msg_formater(_, _, D) -> + D. +%%%---------------------------------------------------------------- +-record(data, {writer, + acc}). -fmt(Fmt, Args, D=#data{writer=Write,acc=Acc}) -> - D#data{acc = Write(io_lib:format(Fmt, Args), Acc)}. +fmt(Fmt, Args, D=#data{writer=Write, acc=Acc}) -> + D#data{acc = Write(Fmt,Args,Acc)}. ts({_,_,Usec}=Now) -> {_Date,{HH,MM,SS}} = calendar:now_to_local_time(Now), io_lib:format("~.2.0w:~.2.0w:~.2.0w.~.6.0w",[HH,MM,SS,Usec]); ts(_) -> "-". -%%%---------------------------------------------------------------- -setup_tracer(Write, MangleArg) -> + +setup_tracer(Type, WriteFun, MangleArgFun, Init) -> Handler = fun(Arg, D) -> - msg_formater(MangleArg(Arg), D) + msg_formater(Type, MangleArgFun(Arg), D) end, - InitialData = #data{writer = Write}, + InitialData = #data{writer = WriteFun, + acc = Init}, {ok,_} = dbg:tracer(process, {Handler, InitialData}), ok. diff --git a/lib/ssh/src/ssh_message.erl b/lib/ssh/src/ssh_message.erl index 4f2eeca026..b1fc05ae33 100644 --- a/lib/ssh/src/ssh_message.erl +++ b/lib/ssh/src/ssh_message.erl @@ -611,13 +611,3 @@ encode_signature({{#'ECPoint'{}, {namedCurve,OID}},_}, Signature) -> CurveName = public_key:oid2ssh_curvename(OID), <<?Ebinary(<<"ecdsa-sha2-",CurveName/binary>>), ?Ebinary(Signature)>>. -%% encode_signature(#'RSAPublicKey'{}, Signature) -> -%% SignName = <<"ssh-rsa">>, -%% <<?Ebinary(SignName), ?Ebinary(Signature)>>; -%% encode_signature({_, #'Dss-Parms'{}}, Signature) -> -%% <<?Ebinary(<<"ssh-dss">>), ?Ebinary(Signature)>>; -%% encode_signature({#'ECPoint'{}, {namedCurve,OID}}, Signature) -> -%% CurveName = public_key:oid2ssh_curvename(OID), -%% <<?Ebinary(<<"ecdsa-sha2-",CurveName/binary>>), ?Ebinary(Signature)>>. - - diff --git a/lib/ssh/src/ssh_options.erl b/lib/ssh/src/ssh_options.erl index aebb5a7062..7eeed70739 100644 --- a/lib/ssh/src/ssh_options.erl +++ b/lib/ssh/src/ssh_options.erl @@ -674,7 +674,11 @@ check_pref_public_key_algs(V) -> PKs = ssh_transport:supported_algorithms(public_key), CHK = fun(A, Ack) -> case lists:member(A, PKs) of - true -> [A|Ack]; + true -> + case lists:member(A,Ack) of + false -> [A|Ack]; + true -> Ack % Remove duplicates + end; false -> error_in_check(A, "Not supported public key") end end, diff --git a/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl b/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl index 0995182623..165274241c 100644 --- a/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl +++ b/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl @@ -280,21 +280,21 @@ msg_code(Num) -> Name -include_lib("ssh/src/ssh_transport.hrl"). %%% Encoding and decodeing is asymetric so out=binary in=string. Sometimes. :( +-define(fix_asym_Xdh_reply(S), + fix_asym(#S{public_host_key = Key, h_sig = {Alg,Sig}} = M) -> + M#S{public_host_key = {Key, list_to_atom(Alg)}, h_sig = Sig} +). + + fix_asym(#ssh_msg_global_request{name=N} = M) -> M#ssh_msg_global_request{name = binary_to_list(N)}; fix_asym(#ssh_msg_debug{message=D,language=L} = M) -> M#ssh_msg_debug{message = binary_to_list(D), language = binary_to_list(L)}; fix_asym(#ssh_msg_kexinit{cookie=C} = M) -> M#ssh_msg_kexinit{cookie = <<C:128>>}; - -fix_asym(#ssh_msg_kexdh_reply{public_host_key = Key} = M) -> M#ssh_msg_kexdh_reply{public_host_key = key_sigalg(Key)}; -fix_asym(#ssh_msg_kex_dh_gex_reply{public_host_key = Key} = M) -> M#ssh_msg_kex_dh_gex_reply{public_host_key = key_sigalg(Key)}; -fix_asym(#ssh_msg_kex_ecdh_reply{public_host_key = Key} = M) -> M#ssh_msg_kex_ecdh_reply{public_host_key = key_sigalg(Key)}; - +?fix_asym_Xdh_reply(ssh_msg_kexdh_reply); +?fix_asym_Xdh_reply(ssh_msg_kex_dh_gex_reply); +?fix_asym_Xdh_reply(ssh_msg_kex_ecdh_reply); fix_asym(M) -> M. -%%% Keys now contains an sig-algorithm name -key_sigalg(#'RSAPublicKey'{} = Key) -> {Key,'ssh-rsa'}; -key_sigalg({_, #'Dss-Parms'{}} = Key) -> {Key,'ssh-dss'}; -key_sigalg({#'ECPoint'{}, {namedCurve,OID}} = Key) -> {Key,"ecdsa-sha2-256"}. %%% Message codes 30 and 31 are overloaded depending on kex family so arrange the decoder %%% input as the test object does diff --git a/lib/ssh/test/ssh.spec b/lib/ssh/test/ssh.spec index 68268cb20d..b4e3d36072 100644 --- a/lib/ssh/test/ssh.spec +++ b/lib/ssh/test/ssh.spec @@ -1,6 +1,7 @@ {suites,"../ssh_test",all}. -{skip_suites, "../ssh_test", [ssh_bench_SUITE +{skip_suites, "../ssh_test", [ssh_bench_SUITE, + ssh_upgrade_SUITE ], "Benchmarks run separately"}. diff --git a/lib/ssh/test/ssh_algorithms_SUITE.erl b/lib/ssh/test/ssh_algorithms_SUITE.erl index 736461624d..98964a2c8a 100644 --- a/lib/ssh/test/ssh_algorithms_SUITE.erl +++ b/lib/ssh/test/ssh_algorithms_SUITE.erl @@ -131,9 +131,14 @@ init_per_group(public_key=Tag, Alg, Config) -> ct:log("Init tests for public_key ~p",[Alg]), PrefAlgs = {preferred_algorithms,[{Tag,[Alg]}]}, %% Daemon started later in init_per_testcase - [{pref_algs,PrefAlgs}, - {tag_alg,{Tag,Alg}} - | Config]; + try + setup_pubkey(Alg, + [{pref_algs,PrefAlgs}, + {tag_alg,{Tag,Alg}} + | Config]) + catch + _:_ -> {skip, io_lib:format("Unsupported: ~p",[Alg])} + end; init_per_group(Tag, Alg, Config) -> PA = @@ -167,17 +172,24 @@ init_per_testcase(TC, Config) -> init_per_testcase(TC, proplists:get_value(tag_alg,Config), Config). -init_per_testcase(_, {public_key,Alg}, Config) -> - Opts = pubkey_opts(Config), +init_per_testcase(TC, {public_key,Alg}, Config) -> + ExtraOpts = case TC of + simple_connect -> + [{user_dir, proplists:get_value(priv_dir,Config)}]; + _ -> + [] + end, + Opts = pubkey_opts(Config) ++ ExtraOpts, case {ssh_file:user_key(Alg,Opts), ssh_file:host_key(Alg,Opts)} of {{ok,_}, {ok,_}} -> - start_pubkey_daemon([proplists:get_value(pref_algs,Config)], + start_pubkey_daemon([proplists:get_value(pref_algs,Config) + | ExtraOpts], [{extra_daemon,true}|Config]); - {{ok,_}, _} -> - {skip, "No host key"}; + {{ok,_}, {error,Err}} -> + {skip, io_lib:format("No host key: ~p",[Err])}; - {_, {ok,_}} -> - {skip, "No user key"}; + {{error,Err}, {ok,_}} -> + {skip, io_lib:format("No user key: ~p",[Err])}; _ -> {skip, "Neither host nor user key"} @@ -221,6 +233,19 @@ simple_exec(Config) -> ssh_test_lib:std_simple_exec(Host, Port, Config). %%-------------------------------------------------------------------- +%% A simple exec call +simple_connect(Config) -> + {Host,Port} = proplists:get_value(srvr_addr, Config), + Opts = + case proplists:get_value(tag_alg, Config) of + {public_key,Alg} -> [{pref_public_key_algs,[Alg]}]; + _ -> [] + end, + ConnectionRef = ssh_test_lib:std_connect(Config, Host, Port, Opts), + ct:log("~p:~p connected! ~p",[?MODULE,?LINE,ConnectionRef]), + ssh:close(ConnectionRef). + +%%-------------------------------------------------------------------- %% Testing if no group matches simple_exec_groups_no_match_too_small(Config) -> try_exec_simple_group({400,500,600}, Config). @@ -302,9 +327,15 @@ sshc_simple_exec_os_cmd(Config) -> %%-------------------------------------------------------------------- %% Connect to the ssh server of the OS sshd_simple_exec(Config) -> + ClientPubKeyOpts = + case proplists:get_value(tag_alg,Config) of + {public_key,Alg} -> [{pref_public_key_algs,[Alg]}]; + _ -> [] + end, ConnectionRef = ssh_test_lib:connect(22, [{silently_accept_hosts, true}, proplists:get_value(pref_algs,Config), - {user_interaction, false}]), + {user_interaction, false} + | ClientPubKeyOpts]), {ok, ChannelId0} = ssh_connection:session_channel(ConnectionRef, infinity), success = ssh_connection:exec(ConnectionRef, ChannelId0, "echo testing", infinity), @@ -361,8 +392,8 @@ split(Alg) -> ssh_test_lib:to_atoms(string:tokens(atom_to_list(Alg), " + ")). specific_test_cases(Tag, Alg, SshcAlgos, SshdAlgos, TypeSSH) -> case Tag of - public_key -> []; - _ -> [simple_exec, simple_sftp] + public_key -> [simple_connect]; + _ -> [simple_connect, simple_exec, simple_sftp] end ++ case supports(Tag, Alg, SshcAlgos) of true when TypeSSH == openSSH -> @@ -437,10 +468,26 @@ setup_pubkey(Config) -> Keys = [ssh_test_lib:setup_dsa(DataDir, UserDir), ssh_test_lib:setup_rsa(DataDir, UserDir), - ssh_test_lib:setup_ecdsa("256", DataDir, UserDir)], + ssh_test_lib:setup_ecdsa("256", DataDir, UserDir) + ], ssh_test_lib:write_auth_keys(Keys, UserDir), % 'authorized_keys' shall contain ALL pub keys Config. +setup_pubkey(Alg, Config) -> + DataDir = proplists:get_value(data_dir, Config), + UserDir = proplists:get_value(priv_dir, Config), + ct:log("Setup keys for ~p",[Alg]), + case Alg of + 'ssh-dss' -> ssh_test_lib:setup_dsa(DataDir, UserDir); + 'ssh-rsa' -> ssh_test_lib:setup_rsa(DataDir, UserDir); + 'rsa-sha2-256' -> ssh_test_lib:setup_rsa(DataDir, UserDir); + 'rsa-sha2-512' -> ssh_test_lib:setup_rsa(DataDir, UserDir); + 'ecdsa-sha2-nistp256' -> ssh_test_lib:setup_ecdsa("256", DataDir, UserDir); + 'ecdsa-sha2-nistp384' -> ssh_test_lib:setup_ecdsa("384", DataDir, UserDir); + 'ecdsa-sha2-nistp521' -> ssh_test_lib:setup_ecdsa("521", DataDir, UserDir) + end, + Config. + simple_exec_group(I, Config) when is_integer(I) -> simple_exec_group({I,I,I}, Config); diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384 b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384 new file mode 100644 index 0000000000..4c39e916e9 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384 @@ -0,0 +1,6 @@ +-----BEGIN EC PRIVATE KEY----- +MIGkAgEBBDAughXu55DNyhxe6x+MNjv4oZKWUDh7bhi4CqjvxhCp9KMpsybltcq+ +lsuKTarzTdKgBwYFK4EEACKhZANiAASu1vvDL0SQoXGtzlltaPHPyDfEVMG/sKLA +pqv8vfRN5Wcs7+yaRKw92nYEKGXfZLbhVX8ArFPMtXPWHcRHCntvL1Acn2kJQ8Gc +7iL4NAr8JhTIUBv4YMhHDa9Pv/CH2zk= +-----END EC PRIVATE KEY----- diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384.pub b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384.pub new file mode 100644 index 0000000000..caa9604c84 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa384.pub @@ -0,0 +1 @@ +ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBK7W+8MvRJChca3OWW1o8c/IN8RUwb+wosCmq/y99E3lZyzv7JpErD3adgQoZd9ktuFVfwCsU8y1c9YdxEcKe28vUByfaQlDwZzuIvg0CvwmFMhQG/hgyEcNr0+/8IfbOQ== uabhnil@elxadlj3q32 diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521 b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521 new file mode 100644 index 0000000000..1e16fcbd57 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521 @@ -0,0 +1,7 @@ +-----BEGIN EC PRIVATE KEY----- +MIHbAgEBBEEWXGoVLiNwQVUwAGZWxOu6uxtU8ntxyZNlcWU4Z8pze9kq3eK7a9XH +l/wxL75Vk1QdOiR/rE3s/L/zOuChp44o1aAHBgUrgQQAI6GBiQOBhgAEAfCrtwjO +kQYKr4/F3uanS7Eby1+SYDdRl1ABuDFhNC3CivVBFt4CnRneV+Mf0viDAxD+HEpd +/GaE2CdsFoVpglN5AVG+fEePY2PiCLHmjc4/pBuR+tWhErzcWAd0KLBCBuc4OAvl +aLLYV1NAJI6COnnfGTCVvYYE5nKMG4LLX0zaWtWl +-----END EC PRIVATE KEY----- diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521.pub b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521.pub new file mode 100644 index 0000000000..069683eba7 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/id_ecdsa521.pub @@ -0,0 +1 @@ +ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHwq7cIzpEGCq+Pxd7mp0uxG8tfkmA3UZdQAbgxYTQtwor1QRbeAp0Z3lfjH9L4gwMQ/hxKXfxmhNgnbBaFaYJTeQFRvnxHj2Nj4gix5o3OP6QbkfrVoRK83FgHdCiwQgbnODgL5Wiy2FdTQCSOgjp53xkwlb2GBOZyjBuCy19M2lrVpQ== uabhnil@elxadlj3q32 diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384 b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384 new file mode 100644 index 0000000000..5835bcd74c --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384 @@ -0,0 +1,6 @@ +-----BEGIN EC PRIVATE KEY----- +MIGkAgEBBDB+l0+SMLYgQ3ZRzg2Pn5u+1ZwKbEnJzXsTKTJM9QSJbKkbA7uCnjdS +CvEW+66CoHqgBwYFK4EEACKhZANiAAT6awCCIrcCr9H4wq0bJ/rQou3tpLHyyf33 +c8D6FPn48/hNqinpx7b0le/0D+Rrhdl9edIplAf6oki7yoFFGl4yuzWtv7rag9jB +vv6w1508ChOmyQ094rFt/xj4KVBhEHI= +-----END EC PRIVATE KEY----- diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384.pub b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384.pub new file mode 100644 index 0000000000..714fc4eb89 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key384.pub @@ -0,0 +1 @@ +ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBPprAIIitwKv0fjCrRsn+tCi7e2ksfLJ/fdzwPoU+fjz+E2qKenHtvSV7/QP5GuF2X150imUB/qiSLvKgUUaXjK7Na2/utqD2MG+/rDXnTwKE6bJDT3isW3/GPgpUGEQcg== uabhnil@elxadlj3q32 diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521 b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521 new file mode 100644 index 0000000000..81aa8df39f --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521 @@ -0,0 +1,7 @@ +-----BEGIN EC PRIVATE KEY----- +MIHbAgEBBEHHxgYEfDclsu5bW+pZfg+bkaqWpgEpXtuzLVm++FFPjhAPhMkurSRj +WQ+CuI2TxgYkBbYFNjn9JqgdMF7FzaiojKAHBgUrgQQAI6GBiQOBhgAEAFTM8TKG +xexxmfAGuyl/Tpk4wytB/OyuVfkF+Q3H1v17HLcpMacA5xUFr80+D5XnjxGttBsS ++X0uexR7QbPbhhPqADgQzFqvTsB1mUNAZnJBD6QNCZkfWwRRwFYQWSmisb43H6G3 +iUTKqiCXMXO8drKLA+Wi+L7VyfoI1CvatBBlDHbV +-----END EC PRIVATE KEY----- diff --git a/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521.pub b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521.pub new file mode 100644 index 0000000000..17b9a1d834 --- /dev/null +++ b/lib/ssh/test/ssh_algorithms_SUITE_data/ssh_host_ecdsa_key521.pub @@ -0,0 +1 @@ +ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBABUzPEyhsXscZnwBrspf06ZOMMrQfzsrlX5BfkNx9b9exy3KTGnAOcVBa/NPg+V548RrbQbEvl9LnsUe0Gz24YT6gA4EMxar07AdZlDQGZyQQ+kDQmZH1sEUcBWEFkporG+Nx+ht4lEyqoglzFzvHayiwPlovi+1cn6CNQr2rQQZQx21Q== uabhnil@elxadlj3q32 diff --git a/lib/ssl/src/dtls_connection.erl b/lib/ssl/src/dtls_connection.erl index f078b87bce..f338471829 100644 --- a/lib/ssl/src/dtls_connection.erl +++ b/lib/ssl/src/dtls_connection.erl @@ -115,7 +115,7 @@ send_handshake_flight(#state{socket = Socket, {Encoded, ConnectionStates} = encode_handshake_flight(lists:reverse(Flight), Version, 1400, Epoch, ConnectionStates0), send(Transport, Socket, Encoded), - start_flight(State0#state{connection_states = ConnectionStates}); + {State0#state{connection_states = ConnectionStates}, []}; send_handshake_flight(#state{socket = Socket, transport_cb = Transport, @@ -129,7 +129,7 @@ send_handshake_flight(#state{socket = Socket, {EncChangeCipher, ConnectionStates} = encode_change_cipher(ChangeCipher, Version, Epoch, ConnectionStates1), send(Transport, Socket, [HsBefore, EncChangeCipher]), - start_flight(State0#state{connection_states = ConnectionStates}); + {State0#state{connection_states = ConnectionStates}, []}; send_handshake_flight(#state{socket = Socket, transport_cb = Transport, @@ -145,7 +145,7 @@ send_handshake_flight(#state{socket = Socket, {HsAfter, ConnectionStates} = encode_handshake_flight(lists:reverse(Flight1), Version, 1400, Epoch, ConnectionStates2), send(Transport, Socket, [HsBefore, EncChangeCipher, HsAfter]), - start_flight(State0#state{connection_states = ConnectionStates}); + {State0#state{connection_states = ConnectionStates}, []}; send_handshake_flight(#state{socket = Socket, transport_cb = Transport, @@ -159,7 +159,7 @@ send_handshake_flight(#state{socket = Socket, {HsAfter, ConnectionStates} = encode_handshake_flight(lists:reverse(Flight1), Version, 1400, Epoch, ConnectionStates1), send(Transport, Socket, [EncChangeCipher, HsAfter]), - start_flight(State0#state{connection_states = ConnectionStates}). + {State0#state{connection_states = ConnectionStates}, []}. queue_change_cipher(ChangeCipher, #state{flight_buffer = Flight, connection_states = ConnectionStates0} = State) -> @@ -235,12 +235,14 @@ init([Role, Host, Port, Socket, Options, User, CbInfo]) -> end. callback_mode() -> - state_functions. + [state_functions, state_enter]. %%-------------------------------------------------------------------- %% State functions %%-------------------------------------------------------------------- +init(enter, _, State) -> + {keep_state, State}; init({call, From}, {start, Timeout}, #state{host = Host, port = Port, role = client, ssl_options = SslOpts, @@ -282,6 +284,8 @@ init({call, _} = Type, Event, #state{role = server} = State) -> init(Type, Event, State) -> ssl_connection:init(Type, Event, State, ?MODULE). +error(enter, _, State) -> + {keep_state, State}; error({call, From}, {start, _Timeout}, {Error, State}) -> {stop_and_reply, normal, {reply, From, {error, Error}}, State}; error({call, From}, Msg, State) -> @@ -295,6 +299,11 @@ error(_, _, _) -> #state{}) -> gen_statem:state_function_result(). %%-------------------------------------------------------------------- +hello(enter, _, #state{role = server} = State) -> + {keep_state, State}; +hello(enter, _, #state{role = client} = State0) -> + {State, Actions} = handle_flight_timer(State0), + {keep_state, State, Actions}; hello(internal, #client_hello{cookie = <<>>, client_version = Version} = Hello, #state{role = server, transport_cb = Transport, @@ -374,6 +383,9 @@ hello(state_timeout, Event, State) -> hello(Type, Event, State) -> ssl_connection:hello(Type, Event, State, ?MODULE). +abbreviated(enter, _, State0) -> + {State, Actions} = handle_flight_timer(State0), + {keep_state, State, Actions}; abbreviated(info, Event, State) -> handle_info(Event, abbreviated, State); abbreviated(internal = Type, @@ -391,6 +403,9 @@ abbreviated(state_timeout, Event, State) -> abbreviated(Type, Event, State) -> ssl_connection:abbreviated(Type, Event, State, ?MODULE). +certify(enter, _, State0) -> + {State, Actions} = handle_flight_timer(State0), + {keep_state, State, Actions}; certify(info, Event, State) -> handle_info(Event, certify, State); certify(internal = Type, #server_hello_done{} = Event, State) -> @@ -400,6 +415,9 @@ certify(state_timeout, Event, State) -> certify(Type, Event, State) -> ssl_connection:certify(Type, Event, State, ?MODULE). +cipher(enter, _, State0) -> + {State, Actions} = handle_flight_timer(State0), + {keep_state, State, Actions}; cipher(info, Event, State) -> handle_info(Event, cipher, State); cipher(internal = Type, #change_cipher_spec{type = <<1>>} = Event, @@ -417,6 +435,8 @@ cipher(state_timeout, Event, State) -> cipher(Type, Event, State) -> ssl_connection:cipher(Type, Event, State, ?MODULE). +connection(enter, _, State) -> + {keep_state, State}; connection(info, Event, State) -> handle_info(Event, connection, State); connection(internal, #hello_request{}, #state{host = Host, port = Port, @@ -449,6 +469,9 @@ connection(internal, #client_hello{}, #state{role = server, allow_renegotiate = connection(Type, Event, State) -> ssl_connection:connection(Type, Event, State, ?MODULE). +%%TODO does this make sense for DTLS ? +downgrade(enter, _, State) -> + {keep_state, State}; downgrade(Type, Event, State) -> ssl_connection:downgrade(Type, Event, State, ?MODULE). @@ -750,31 +773,62 @@ next_event(connection = StateName, no_record, {#ssl_tls{epoch = Epoch, type = ?HANDSHAKE, version = _Version}, State1} = _Record when Epoch == CurrentEpoch-1 -> - {State, MoreActions} = send_handshake_flight(State1, Epoch), - {next_state, StateName, State, Actions ++ MoreActions}; + {State2, MoreActions} = send_handshake_flight(State1, CurrentEpoch), + {NextRecord, State} = next_record(State2), + next_event(StateName, NextRecord, State, Actions ++ MoreActions); + %% From FLIGHT perspective CHANGE_CIPHER_SPEC is treated as a handshake + {#ssl_tls{epoch = Epoch, + type = ?CHANGE_CIPHER_SPEC, + version = _Version}, State1} = _Record when Epoch == CurrentEpoch-1 -> + {State2, MoreActions} = send_handshake_flight(State1, CurrentEpoch), + {NextRecord, State} = next_record(State2), + next_event(StateName, NextRecord, State, Actions ++ MoreActions); {#ssl_tls{epoch = _Epoch, - version = _Version}, State} -> + version = _Version}, State1} -> %% TODO maybe buffer later epoch - {next_state, StateName, State, Actions}; + {Record, State} = next_record(State1), + next_event(StateName, Record, State, Actions); {#alert{} = Alert, State} -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end; +next_event(connection = StateName, Record, + #state{connection_states = #{current_read := #{epoch := CurrentEpoch}}} = State0, Actions) -> + case Record of + #ssl_tls{epoch = CurrentEpoch} -> + {next_state, StateName, State0, [{next_event, internal, {protocol_record, Record}} | Actions]}; + #ssl_tls{epoch = Epoch, + type = ?HANDSHAKE, + version = _Version} when Epoch == CurrentEpoch-1 -> + {State1, MoreActions} = send_handshake_flight(State0, CurrentEpoch), + {NextRecord, State} = next_record(State1), + next_event(StateName, NextRecord, State, Actions ++ MoreActions); + %% From FLIGHT perspective CHANGE_CIPHER_SPEC is treated as a handshake + #ssl_tls{epoch = Epoch, + type = ?CHANGE_CIPHER_SPEC, + version = _Version} when Epoch == CurrentEpoch-1 -> + {State1, MoreActions} = send_handshake_flight(State0, CurrentEpoch), + {NextRecord, State} = next_record(State1), + next_event(StateName, NextRecord, State, Actions ++ MoreActions); + _ -> + next_event(StateName, no_record, State0, Actions) + end; next_event(StateName, Record, - #state{connection_states = #{current_read := #{epoch := CurrentEpoch}}} = State, Actions) -> + #state{connection_states = #{current_read := #{epoch := CurrentEpoch}}} = State0, Actions) -> case Record of no_record -> - {next_state, StateName, State, Actions}; + {next_state, StateName, State0, Actions}; #ssl_tls{epoch = CurrentEpoch, version = Version} = Record -> {next_state, StateName, - dtls_version(StateName, Version, State), + dtls_version(StateName, Version, State0), [{next_event, internal, {protocol_record, Record}} | Actions]}; #ssl_tls{epoch = _Epoch, version = _Version} = _Record -> %% TODO maybe buffer later epoch - {next_state, StateName, State, Actions}; + {Record, State} = next_record(State0), + next_event(StateName, Record, State, Actions); #alert{} = Alert -> - {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} + {next_state, StateName, State0, [{next_event, internal, Alert} | Actions]} end. decode_cipher_text(#state{protocol_buffers = #protocol_buffers{dtls_cipher_texts = [ CT | Rest]} = Buffers, @@ -814,13 +868,13 @@ next_flight(Flight) -> change_cipher_spec => undefined, handshakes_after_change_cipher_spec => []}. -start_flight(#state{transport_cb = gen_udp, - flight_state = {retransmit, Timeout}} = State) -> +handle_flight_timer(#state{transport_cb = gen_udp, + flight_state = {retransmit, Timeout}} = State) -> start_retransmision_timer(Timeout, State); -start_flight(#state{transport_cb = gen_udp, +handle_flight_timer(#state{transport_cb = gen_udp, flight_state = connection} = State) -> {State, []}; -start_flight(State) -> +handle_flight_timer(State) -> %% No retransmision needed i.e DTLS over SCTP {State#state{flight_state = reliable}, []}. diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl index 96c3ab86e9..352874c77d 100644 --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -600,8 +600,12 @@ next_record(#state{protocol_buffers = next_record(#state{protocol_buffers = #protocol_buffers{tls_packets = [], tls_cipher_texts = []}, socket = Socket, transport_cb = Transport} = State) -> - tls_socket:setopts(Transport, Socket, [{active,once}]), - {no_record, State}; + case tls_socket:setopts(Transport, Socket, [{active,once}]) of + ok -> + {no_record, State}; + _ -> + {socket_closed, State} + end; next_record(State) -> {no_record, State}. @@ -626,10 +630,15 @@ passive_receive(State0 = #state{user_data_buffer = Buffer}, StateName) -> next_event(StateName, Record, State) -> next_event(StateName, Record, State, []). +next_event(StateName, socket_closed, State, _) -> + ssl_connection:handle_normal_shutdown(?ALERT_REC(?FATAL, ?CLOSE_NOTIFY), StateName, State), + {stop, {shutdown, transport_closed}, State}; next_event(connection = StateName, no_record, State0, Actions) -> case next_record_if_active(State0) of {no_record, State} -> ssl_connection:hibernate_after(StateName, State, Actions); + {socket_closed, State} -> + next_event(StateName, socket_closed, State, Actions); {#ssl_tls{} = Record, State} -> {next_state, StateName, State, [{next_event, internal, {protocol_record, Record}} | Actions]}; {#alert{} = Alert, State} -> diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index 58870a3419..d13b1b3f2a 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -240,6 +240,7 @@ error_handling_tests()-> error_handling_tests_tls()-> [controller_dies, tls_client_closes_socket, + tls_closed_in_active_once, tls_tcp_error_propagation_in_active_mode, tls_tcp_connect, tls_tcp_connect_big, @@ -430,6 +431,7 @@ init_per_testcase(prf, Config) -> init_per_testcase(TestCase, Config) when TestCase == tls_ssl_accept_timeout; TestCase == tls_client_closes_socket; + TestCase == tls_closed_in_active_once; TestCase == tls_downgrade -> ssl_test_lib:ct_log_supported_protocol_versions(Config), ct:timetrap({seconds, 15}), @@ -961,6 +963,48 @@ tls_client_closes_socket(Config) when is_list(Config) -> ssl_test_lib:check_result(Server, {error,closed}). %%-------------------------------------------------------------------- +tls_closed_in_active_once() -> + [{doc, "Test that ssl_closed is delivered in active once with non-empty buffer, check ERL-420."}]. + +tls_closed_in_active_once(Config) when is_list(Config) -> + ClientOpts = ssl_test_lib:ssl_options(client_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), + {_ClientNode, _ServerNode, Hostname} = ssl_test_lib:run_where(Config), + TcpOpts = [binary, {reuseaddr, true}], + Port = ssl_test_lib:inet_port(node()), + Server = fun() -> + {ok, Listen} = gen_tcp:listen(Port, TcpOpts), + {ok, TcpServerSocket} = gen_tcp:accept(Listen), + {ok, ServerSocket} = ssl:ssl_accept(TcpServerSocket, ServerOpts), + lists:foreach( + fun(_) -> + ssl:send(ServerSocket, "some random message\r\n") + end, lists:seq(1, 20)), + %% Close TCP instead of SSL socket to trigger the bug: + gen_tcp:close(TcpServerSocket), + gen_tcp:close(Listen) + end, + spawn_link(Server), + {ok, Socket} = ssl:connect(Hostname, Port, [{active, false} | ClientOpts]), + Result = tls_closed_in_active_once_loop(Socket), + ssl:close(Socket), + case Result of + ok -> ok; + _ -> ct:fail(Result) + end. + +tls_closed_in_active_once_loop(Socket) -> + ssl:setopts(Socket, [{active, once}]), + receive + {ssl, Socket, _} -> + tls_closed_in_active_once_loop(Socket); + {ssl_closed, Socket} -> + ok + after 5000 -> + no_ssl_closed_received + end. + +%%-------------------------------------------------------------------- connect_dist() -> [{doc,"Test a simple connect as is used by distribution"}]. diff --git a/lib/stdlib/doc/src/gen_statem.xml b/lib/stdlib/doc/src/gen_statem.xml index ad7f2f2e95..a7caa71dcb 100644 --- a/lib/stdlib/doc/src/gen_statem.xml +++ b/lib/stdlib/doc/src/gen_statem.xml @@ -1098,7 +1098,7 @@ handle_event(_, _, State, Data) -> <seealso marker="#Module:init/1"><c>Module:init/1</c></seealso> or <seealso marker="#enter_loop/5"><c>enter_loop/5,6</c></seealso> - would be weird on the border of whichcraft + would be weird on the border of witchcraft since there has been no earlier call to a <seealso marker="#state callback">state callback</seealso> in this server. diff --git a/lib/stdlib/doc/src/string.xml b/lib/stdlib/doc/src/string.xml index 343904a49a..9d5edd9ecf 100644 --- a/lib/stdlib/doc/src/string.xml +++ b/lib/stdlib/doc/src/string.xml @@ -311,7 +311,9 @@ true</pre> <desc> <p> Returns the first codepoint in <c><anno>String</anno></c> - and the rest of <c><anno>String</anno></c> in the tail. + and the rest of <c><anno>String</anno></c> in the tail. Returns + an empty list if <c><anno>String</anno></c> is empty or an + <c>{error, String}</c> tuple if the next byte is invalid. </p> <p><em>Example:</em></p> <pre> @@ -326,7 +328,9 @@ true</pre> <desc> <p> Returns the first grapheme cluster in <c><anno>String</anno></c> - and the rest of <c><anno>String</anno></c> in the tail. + and the rest of <c><anno>String</anno></c> in the tail. Returns + an empty list if <c><anno>String</anno></c> is empty or an + <c>{error, String}</c> tuple if the next byte is invalid. </p> <p><em>Example:</em></p> <pre> diff --git a/lib/stdlib/src/erl_tar.erl b/lib/stdlib/src/erl_tar.erl index 168ea4002c..76f0b38108 100644 --- a/lib/stdlib/src/erl_tar.erl +++ b/lib/stdlib/src/erl_tar.erl @@ -176,7 +176,7 @@ check_extract(Name, #read_opts{files=Files}) -> -type tar_entry() :: {filename(), typeflag(), non_neg_integer(), - calendar:datetime(), + tar_time(), mode(), uid(), gid()}. @@ -274,8 +274,13 @@ mode_to_string(Mode, [_|T], Acc) -> mode_to_string(_, [], Acc) -> Acc. -%% Converts a datetime tuple to a readable string -time_to_string({{Y, Mon, Day}, {H, Min, _}}) -> +%% Converts a tar_time() (POSIX time) to a readable string +time_to_string(Secs0) -> + Epoch = calendar:datetime_to_gregorian_seconds(?EPOCH), + Secs = Epoch + Secs0, + DateTime0 = calendar:gregorian_seconds_to_datetime(Secs), + DateTime = calendar:universal_time_to_local_time(DateTime0), + {{Y, Mon, Day}, {H, Min, _}} = DateTime, io_lib:format("~s ~2w ~s:~s ~w", [month(Mon), Day, two_d(H), two_d(Min), Y]). two_d(N) -> @@ -452,7 +457,8 @@ add(Reader, NameOrBin, NameInArchive, Options) do_add(#reader{access=write}=Reader, Name, NameInArchive, Options) when is_list(NameInArchive), is_list(Options) -> - Opts = #add_opts{read_info=fun(F) -> file:read_link_info(F) end}, + RF = fun(F) -> file:read_link_info(F, [{time, posix}]) end, + Opts = #add_opts{read_info=RF}, add1(Reader, Name, NameInArchive, add_opts(Options, Opts)); do_add(#reader{access=read},_,_,_) -> {error, eacces}; @@ -460,7 +466,8 @@ do_add(Reader,_,_,_) -> {error, {badarg, Reader}}. add_opts([dereference|T], Opts) -> - add_opts(T, Opts#add_opts{read_info=fun(F) -> file:read_file_info(F) end}); + RF = fun(F) -> file:read_file_info(F, [{time, posix}]) end, + add_opts(T, Opts#add_opts{read_info=RF}); add_opts([verbose|T], Opts) -> add_opts(T, Opts#add_opts{verbose=true}); add_opts([{chunks,N}|T], Opts) -> @@ -503,7 +510,7 @@ add1(#reader{}=Reader, Name, NameInArchive, #add_opts{read_info=ReadInfo}=Opts) end; add1(Reader, Bin, NameInArchive, Opts) when is_binary(Bin) -> add_verbose(Opts, "a ~ts~n", [NameInArchive]), - Now = calendar:now_to_local_time(erlang:timestamp()), + Now = os:system_time(seconds), Header = #tar_header{ name = NameInArchive, size = byte_size(Bin), @@ -612,7 +619,7 @@ build_header(#tar_header{}=Header, Opts) -> devmajor=Devmaj, devminor=Devmin } = Header, - Mtime = datetime_to_posix(Header#tar_header.mtime), + Mtime = Header#tar_header.mtime, Block0 = ?ZERO_BLOCK, {Block1, Pax0} = write_string(Block0, ?V7_NAME, ?V7_NAME_LEN, Name, ?PAX_PATH, #{}), @@ -770,14 +777,6 @@ join_split_ustar_path([Part|Rest], {ok, Name, nil}) -> join_split_ustar_path([Part|Rest], {ok, Name, Acc}) -> join_split_ustar_path(Rest, {ok, Name, <<Acc/binary,$/,Part/binary>>}). -datetime_to_posix(DateTime) -> - Epoch = calendar:datetime_to_gregorian_seconds(?EPOCH), - Secs = calendar:datetime_to_gregorian_seconds(DateTime), - case Secs - Epoch of - N when N < 0 -> 0; - N -> N - end. - write_octal(Block, Pos, Size, X) -> Octal = zero_pad(format_octal(X), Size-1), if byte_size(Octal) < Size -> @@ -984,7 +983,7 @@ do_get_format(#header_v7{}=V7, Bin) unpack_format(Format, #header_v7{}=V7, Bin, Reader) when is_binary(Bin), byte_size(Bin) =:= ?BLOCK_SIZE -> - Mtime = posix_to_erlang_time(parse_numeric(V7#header_v7.mtime)), + Mtime = parse_numeric(V7#header_v7.mtime), Header0 = #tar_header{ name=parse_string(V7#header_v7.name), mode=parse_numeric(V7#header_v7.mode), @@ -1051,9 +1050,9 @@ unpack_modern(Format, #header_v7{}=V7, Bin, #tar_header{}=Header0) Star = to_star(V7, Bin), Prefix0 = parse_string(Star#header_star.prefix), Atime0 = Star#header_star.atime, - Atime = posix_to_erlang_time(parse_numeric(Atime0)), + Atime = parse_numeric(Atime0), Ctime0 = Star#header_star.ctime, - Ctime = posix_to_erlang_time(parse_numeric(Ctime0)), + Ctime = parse_numeric(Ctime0), {Prefix0, H1#tar_header{ atime=Atime, ctime=Ctime @@ -1313,11 +1312,6 @@ is_header_only_type(?TYPE_LINK) -> true; is_header_only_type(?TYPE_DIR) -> true; is_header_only_type(_) -> false. -posix_to_erlang_time(Sec) -> - OneMillion = 1000000, - Time = calendar:now_to_datetime({Sec div OneMillion, Sec rem OneMillion, 0}), - erlang:universaltime_to_localtime(Time). - foldl_read(#reader{access=read}=Reader, Fun, Accu, #read_opts{}=Opts) when is_function(Fun,4) -> case foldl_read0(Reader, Fun, Accu, Opts) of @@ -1423,7 +1417,7 @@ do_merge_pax(Header, [_Ignore|Rest]) -> do_merge_pax(Header, Rest). %% Returns the time since UNIX epoch as a datetime --spec parse_pax_time(binary()) -> calendar:datetime(). +-spec parse_pax_time(binary()) -> tar_time(). parse_pax_time(Bin) when is_binary(Bin) -> TotalNano = case binary:split(Bin, [<<$.>>]) of [SecondsStr, NanoStr0] -> @@ -1450,8 +1444,7 @@ parse_pax_time(Bin) when is_binary(Bin) -> Micro = TotalNano div 1000, Mega = Micro div 1000000000000, Secs = Micro div 1000000 - (Mega*1000000), - Micro2 = Micro rem 1000000, - calendar:now_to_datetime({Mega, Secs, Micro2}). + Secs. %% Given a regular file reader, reads the whole file and %% parses all extended attributes it contains. @@ -1671,7 +1664,7 @@ set_extracted_file_info(Name, #tar_header{typeflag = ?TYPE_BLOCK}=Header) -> set_device_info(Name, Header); set_extracted_file_info(Name, #tar_header{mtime=Mtime,mode=Mode}) -> Info = #file_info{mode=Mode, mtime=Mtime}, - file:write_file_info(Name, Info). + file:write_file_info(Name, Info, [{time, posix}]). set_device_info(Name, #tar_header{}=Header) -> Mtime = Header#tar_header.mtime, diff --git a/lib/stdlib/src/erl_tar.hrl b/lib/stdlib/src/erl_tar.hrl index d646d02989..cff0c2f500 100644 --- a/lib/stdlib/src/erl_tar.hrl +++ b/lib/stdlib/src/erl_tar.hrl @@ -55,6 +55,8 @@ {string(), binary()} | {string(), file:filename()}]. +-type tar_time() :: non_neg_integer(). + %% The tar header, once fully parsed. -record(tar_header, { name = "" :: string(), %% name of header file entry @@ -62,15 +64,15 @@ uid = 0 :: non_neg_integer(), %% user id of owner gid = 0 :: non_neg_integer(), %% group id of owner size = 0 :: non_neg_integer(), %% length in bytes - mtime :: calendar:datetime(), %% modified time + mtime :: tar_time(), %% modified time typeflag :: char(), %% type of header entry linkname = "" :: string(), %% target name of link uname = "" :: string(), %% user name of owner gname = "" :: string(), %% group name of owner devmajor = 0 :: non_neg_integer(), %% major number of character or block device devminor = 0 :: non_neg_integer(), %% minor number of character or block device - atime :: calendar:datetime(), %% access time - ctime :: calendar:datetime() %% status change time + atime :: tar_time(), %% access time + ctime :: tar_time() %% status change time }). -type tar_header() :: #tar_header{}. diff --git a/lib/stdlib/src/string.erl b/lib/stdlib/src/string.erl index 17135dd64a..6f7009b5d9 100644 --- a/lib/stdlib/src/string.erl +++ b/lib/stdlib/src/string.erl @@ -486,12 +486,14 @@ find(String, SearchPattern, trailing) -> %% Fetch first codepoint and return rest in tail -spec next_grapheme(String::unicode:chardata()) -> - maybe_improper_list(grapheme_cluster(),unicode:chardata()). + maybe_improper_list(grapheme_cluster(),unicode:chardata()) | + {error,unicode:chardata()}. next_grapheme(CD) -> unicode_util:gc(CD). %% Fetch first grapheme cluster and return rest in tail -spec next_codepoint(String::unicode:chardata()) -> - maybe_improper_list(char(),unicode:chardata()). + maybe_improper_list(char(),unicode:chardata()) | + {error,unicode:chardata()}. next_codepoint(CD) -> unicode_util:cp(CD). %% Internals @@ -508,7 +510,7 @@ equal_1(A0,B0) -> case {unicode_util:cp(A0), unicode_util:cp(B0)} of {[CP|A],[CP|B]} -> equal_1(A,B); {[], []} -> true; - _ -> false + {L1,L2} when is_list(L1), is_list(L2) -> false end. equal_nocase(A, A) -> true; @@ -517,7 +519,7 @@ equal_nocase(A0, B0) -> unicode_util:cp(unicode_util:casefold(B0))} of {[CP|A],[CP|B]} -> equal_nocase(A,B); {[], []} -> true; - _ -> false + {L1,L2} when is_list(L1), is_list(L2) -> false end. equal_norm(A, A, _Norm) -> true; @@ -526,7 +528,7 @@ equal_norm(A0, B0, Norm) -> unicode_util:cp(unicode_util:Norm(B0))} of {[CP|A],[CP|B]} -> equal_norm(A,B, Norm); {[], []} -> true; - _ -> false + {L1,L2} when is_list(L1), is_list(L2) -> false end. equal_norm_nocase(A, A, _Norm) -> true; @@ -535,7 +537,7 @@ equal_norm_nocase(A0, B0, Norm) -> unicode_util:cp(unicode_util:casefold(unicode_util:Norm(B0)))} of {[CP|A],[CP|B]} -> equal_norm_nocase(A,B, Norm); {[], []} -> true; - _ -> false + {L1,L2} when is_list(L1), is_list(L2) -> false end. reverse_1(CD, Acc) -> diff --git a/lib/stdlib/src/unicode.erl b/lib/stdlib/src/unicode.erl index aa1da400ce..fbe8a94074 100644 --- a/lib/stdlib/src/unicode.erl +++ b/lib/stdlib/src/unicode.erl @@ -250,89 +250,110 @@ encoding_to_bom(latin1) -> -define(GC_N, 200). %% arbitrary number %% Canonical decompose string to list of chars --spec characters_to_nfd_list(chardata()) -> [char()]. +-spec characters_to_nfd_list(chardata()) -> [char()] | {error, [char()], chardata()}. characters_to_nfd_list(CD) -> + characters_to_nfd_list(CD, []). +characters_to_nfd_list(CD, Acc) -> case unicode_util:nfd(CD) of - [GC|Str] when is_list(GC) -> GC++characters_to_nfd_list(Str); - [CP|Str] -> [CP|characters_to_nfd_list(Str)]; - [] -> [] + [GC|Str] when is_list(GC) -> characters_to_nfd_list(Str, lists:reverse(GC, Acc)); + [CP|Str] -> characters_to_nfd_list(Str, [CP | Acc]); + [] -> lists:reverse(Acc); + {error,Error} -> {error, lists:reverse(Acc), Error} end. --spec characters_to_nfd_binary(chardata()) -> unicode_binary(). +-spec characters_to_nfd_binary(chardata()) -> unicode_binary() | {error, unicode_binary(), chardata()}. characters_to_nfd_binary(CD) -> - list_to_binary(characters_to_nfd_binary(CD, ?GC_N, [])). + characters_to_nfd_binary(CD, ?GC_N, [], []). -characters_to_nfd_binary(CD, N, Row) when N > 0 -> +characters_to_nfd_binary(CD, N, Row, Acc) when N > 0 -> case unicode_util:nfd(CD) of - [GC|Str] -> characters_to_nfd_binary(Str, N-1, [GC|Row]); - [] -> [characters_to_binary(lists:reverse(Row))] + [GC|Str] -> characters_to_nfd_binary(Str, N-1, [GC|Row], Acc); + [] -> acc_to_binary(prepend_row_to_acc(Row, Acc)); + {error, Error} -> {error, acc_to_binary(prepend_row_to_acc(Row, Acc)), Error} end; -characters_to_nfd_binary(CD, _, Row) -> - [characters_to_binary(lists:reverse(Row))|characters_to_nfd_binary(CD,?GC_N,[])]. +characters_to_nfd_binary(CD, _, Row, Acc) -> + characters_to_nfd_binary(CD, ?GC_N, [], prepend_row_to_acc(Row, Acc)). %% Compability Canonical decompose string to list of chars. --spec characters_to_nfkd_list(chardata()) -> [char()]. +-spec characters_to_nfkd_list(chardata()) -> [char()] | {error, [char()], chardata()}. characters_to_nfkd_list(CD) -> + characters_to_nfkd_list(CD, []). +characters_to_nfkd_list(CD, Acc) -> case unicode_util:nfkd(CD) of - [GC|Str] when is_list(GC) -> GC++characters_to_nfkd_list(Str); - [CP|Str] -> [CP|characters_to_nfkd_list(Str)]; - [] -> [] + [GC|Str] when is_list(GC) -> characters_to_nfkd_list(Str, lists:reverse(GC, Acc)); + [CP|Str] -> characters_to_nfkd_list(Str, [CP | Acc]); + [] -> lists:reverse(Acc); + {error,Error} -> {error, lists:reverse(Acc), Error} end. --spec characters_to_nfkd_binary(chardata()) -> unicode_binary(). +-spec characters_to_nfkd_binary(chardata()) -> unicode_binary() | {error, unicode_binary(), chardata()}. characters_to_nfkd_binary(CD) -> - list_to_binary(characters_to_nfkd_binary(CD, ?GC_N, [])). + characters_to_nfkd_binary(CD, ?GC_N, [], []). -characters_to_nfkd_binary(CD, N, Row) when N > 0 -> +characters_to_nfkd_binary(CD, N, Row, Acc) when N > 0 -> case unicode_util:nfkd(CD) of - [GC|Str] -> characters_to_nfkd_binary(Str, N-1, [GC|Row]); - [] -> [characters_to_binary(lists:reverse(Row))] + [GC|Str] -> characters_to_nfkd_binary(Str, N-1, [GC|Row], Acc); + [] -> acc_to_binary(prepend_row_to_acc(Row, Acc)); + {error, Error} -> {error, acc_to_binary(prepend_row_to_acc(Row, Acc)), Error} end; -characters_to_nfkd_binary(CD, _, Row) -> - [characters_to_binary(lists:reverse(Row))|characters_to_nfkd_binary(CD,?GC_N,[])]. +characters_to_nfkd_binary(CD, _, Row, Acc) -> + characters_to_nfkd_binary(CD, ?GC_N, [], prepend_row_to_acc(Row, Acc)). %% Canonical compose string to list of chars --spec characters_to_nfc_list(chardata()) -> [char()]. +-spec characters_to_nfc_list(chardata()) -> [char()] | {error, [char()], chardata()}. characters_to_nfc_list(CD) -> + characters_to_nfc_list(CD, []). +characters_to_nfc_list(CD, Acc) -> case unicode_util:nfc(CD) of - [CPs|Str] when is_list(CPs) -> CPs ++ characters_to_nfc_list(Str); - [CP|Str] -> [CP|characters_to_nfc_list(Str)]; - [] -> [] + [GC|Str] when is_list(GC) -> characters_to_nfc_list(Str, lists:reverse(GC, Acc)); + [CP|Str] -> characters_to_nfc_list(Str, [CP | Acc]); + [] -> lists:reverse(Acc); + {error,Error} -> {error, lists:reverse(Acc), Error} end. --spec characters_to_nfc_binary(chardata()) -> unicode_binary(). +-spec characters_to_nfc_binary(chardata()) -> unicode_binary() | {error, unicode_binary(), chardata()}. characters_to_nfc_binary(CD) -> - list_to_binary(characters_to_nfc_binary(CD, ?GC_N, [])). + characters_to_nfc_binary(CD, ?GC_N, [], []). -characters_to_nfc_binary(CD, N, Row) when N > 0 -> +characters_to_nfc_binary(CD, N, Row, Acc) when N > 0 -> case unicode_util:nfc(CD) of - [GC|Str] -> characters_to_nfc_binary(Str, N-1, [GC|Row]); - [] -> [characters_to_binary(lists:reverse(Row))] + [GC|Str] -> characters_to_nfc_binary(Str, N-1, [GC|Row], Acc); + [] -> acc_to_binary(prepend_row_to_acc(Row, Acc)); + {error, Error} -> {error, acc_to_binary(prepend_row_to_acc(Row, Acc)), Error} end; -characters_to_nfc_binary(CD, _, Row) -> - [characters_to_binary(lists:reverse(Row))|characters_to_nfc_binary(CD,?GC_N,[])]. +characters_to_nfc_binary(CD, _, Row, Acc) -> + characters_to_nfc_binary(CD, ?GC_N, [], prepend_row_to_acc(Row, Acc)). %% Compability Canonical compose string to list of chars --spec characters_to_nfkc_list(chardata()) -> [char()]. +-spec characters_to_nfkc_list(chardata()) -> [char()] | {error, [char()], chardata()}. characters_to_nfkc_list(CD) -> + characters_to_nfkc_list(CD, []). +characters_to_nfkc_list(CD, Acc) -> case unicode_util:nfkc(CD) of - [CPs|Str] when is_list(CPs) -> CPs ++ characters_to_nfkc_list(Str); - [CP|Str] -> [CP|characters_to_nfkc_list(Str)]; - [] -> [] + [GC|Str] when is_list(GC) -> characters_to_nfkc_list(Str, lists:reverse(GC, Acc)); + [CP|Str] -> characters_to_nfkc_list(Str, [CP | Acc]); + [] -> lists:reverse(Acc); + {error,Error} -> {error, lists:reverse(Acc), Error} end. --spec characters_to_nfkc_binary(chardata()) -> unicode_binary(). +-spec characters_to_nfkc_binary(chardata()) -> unicode_binary() | {error, unicode_binary(), chardata()}. characters_to_nfkc_binary(CD) -> - list_to_binary(characters_to_nfkc_binary(CD, ?GC_N, [])). + characters_to_nfkc_binary(CD, ?GC_N, [], []). -characters_to_nfkc_binary(CD, N, Row) when N > 0 -> +characters_to_nfkc_binary(CD, N, Row, Acc) when N > 0 -> case unicode_util:nfkc(CD) of - [GC|Str] -> characters_to_nfkc_binary(Str, N-1, [GC|Row]); - [] -> [characters_to_binary(lists:reverse(Row))] + [GC|Str] -> characters_to_nfkc_binary(Str, N-1, [GC|Row], Acc); + [] -> acc_to_binary(prepend_row_to_acc(Row, Acc)); + {error, Error} -> {error, acc_to_binary(prepend_row_to_acc(Row, Acc)), Error} end; -characters_to_nfkc_binary(CD, _, Row) -> - [characters_to_binary(lists:reverse(Row))|characters_to_nfkc_binary(CD,?GC_N,[])]. +characters_to_nfkc_binary(CD, _, Row, Acc) -> + characters_to_nfkc_binary(CD, ?GC_N, [], prepend_row_to_acc(Row, Acc)). + +acc_to_binary(Acc) -> + list_to_binary(lists:reverse(Acc)). +prepend_row_to_acc(Row, Acc) -> + [characters_to_binary(lists:reverse(Row))|Acc]. %% internals diff --git a/lib/stdlib/test/string_SUITE.erl b/lib/stdlib/test/string_SUITE.erl index 4320b735ac..90f980c0e5 100644 --- a/lib/stdlib/test/string_SUITE.erl +++ b/lib/stdlib/test/string_SUITE.erl @@ -582,6 +582,8 @@ cd_gc(_) -> [$e,778] = string:next_codepoint([$e,778]), [$e|<<204,138>>] = string:next_codepoint(<<$e,778/utf8>>), [778|_] = string:next_codepoint(tl(string:next_codepoint(<<$e,778/utf8>>))), + [0|<<128,1>>] = string:next_codepoint(<<0,128,1>>), + {error,<<128,1>>} = string:next_codepoint(<<128,1>>), [] = string:next_grapheme(""), [] = string:next_grapheme(<<>>), @@ -589,6 +591,8 @@ cd_gc(_) -> "abcd" = string:next_grapheme("abcd"), [[$e,778]] = string:next_grapheme([$e,778]), [[$e,778]] = string:next_grapheme(<<$e,778/utf8>>), + [0|<<128,1>>] = string:next_grapheme(<<0,128,1>>), + {error,<<128,1>>} = string:next_grapheme(<<128,1>>), ok. diff --git a/lib/stdlib/test/tar_SUITE.erl b/lib/stdlib/test/tar_SUITE.erl index e9ab12e061..4061008812 100644 --- a/lib/stdlib/test/tar_SUITE.erl +++ b/lib/stdlib/test/tar_SUITE.erl @@ -27,7 +27,8 @@ extract_from_binary_compressed/1, extract_filtered/1, extract_from_open_file/1, symlinks/1, open_add_close/1, cooked_compressed/1, memory/1,unicode/1,read_other_implementations/1, - sparse/1, init/1, leading_slash/1, dotdot/1]). + sparse/1, init/1, leading_slash/1, dotdot/1, + roundtrip_metadata/1]). -include_lib("common_test/include/ct.hrl"). -include_lib("kernel/include/file.hrl"). @@ -41,7 +42,7 @@ all() -> extract_filtered, symlinks, open_add_close, cooked_compressed, memory, unicode, read_other_implementations, - sparse,init,leading_slash,dotdot]. + sparse,init,leading_slash,dotdot,roundtrip_metadata]. groups() -> []. @@ -953,6 +954,42 @@ dotdot(Config) -> ok. +roundtrip_metadata(Config) -> + PrivDir = proplists:get_value(priv_dir, Config), + Dir = filename:join(PrivDir, ?FUNCTION_NAME), + ok = file:make_dir(Dir), + + do_roundtrip_metadata(Dir, "name-does-not-matter"), + ok. + +do_roundtrip_metadata(Dir, File) -> + Tar = filename:join(Dir, atom_to_list(?FUNCTION_NAME)++".tar"), + BeamFile = code:which(compile), + {ok,Fd} = erl_tar:open(Tar, [write]), + ok = erl_tar:add(Fd, BeamFile, File, []), + ok = erl_tar:close(Fd), + + ok = erl_tar:extract(Tar, [{cwd,Dir}]), + + %% Make sure that size and modification times are the same + %% on all platforms. + {ok,OrigInfo} = file:read_file_info(BeamFile), + ExtractedFile = filename:join(Dir, File), + {ok,ExtractedInfo} = file:read_file_info(ExtractedFile), + #file_info{size=Size,mtime=Mtime,type=regular} = OrigInfo, + #file_info{size=Size,mtime=Mtime,type=regular} = ExtractedInfo, + + %% On Unix platforms more fields are expected to be the same. + case os:type() of + {unix,_} -> + #file_info{access=Access,mode=Mode} = OrigInfo, + #file_info{access=Access,mode=Mode} = ExtractedInfo, + ok; + _ -> + ok + end. + + %% Delete the given list of files. delete_files([]) -> ok; delete_files([Item|Rest]) -> diff --git a/lib/stdlib/test/unicode_SUITE.erl b/lib/stdlib/test/unicode_SUITE.erl index 3d97ab93f1..e01ba3fbb0 100644 --- a/lib/stdlib/test/unicode_SUITE.erl +++ b/lib/stdlib/test/unicode_SUITE.erl @@ -998,6 +998,30 @@ normalize(_) -> true = unicode:characters_to_nfkc_list("ホンダ") =:= unicode:characters_to_nfkc_list("ﾎﾝﾀﾞ"), true = unicode:characters_to_nfkd_list("32") =:= unicode:characters_to_nfkd_list("３２"), + + {error, [0], <<128>>} = unicode:characters_to_nfc_list(<<0, 128>>), + {error, [0], <<128>>} = unicode:characters_to_nfkc_list(<<0, 128>>), + {error, [0], <<128>>} = unicode:characters_to_nfd_list(<<0, 128>>), + {error, [0], <<128>>} = unicode:characters_to_nfkd_list(<<0, 128>>), + + {error, <<0>>, <<128>>} = unicode:characters_to_nfc_binary(<<0, 128>>), + {error, <<0>>, <<128>>} = unicode:characters_to_nfkc_binary(<<0, 128>>), + {error, <<0>>, <<128>>} = unicode:characters_to_nfd_binary(<<0, 128>>), + {error, <<0>>, <<128>>} = unicode:characters_to_nfkd_binary(<<0, 128>>), + + LargeBin = binary:copy(<<"abcde">>, 50), + LargeList = binary_to_list(LargeBin), + + {error, LargeList, <<128>>} = unicode:characters_to_nfc_list(<<LargeBin/binary, 128>>), + {error, LargeList, <<128>>} = unicode:characters_to_nfkc_list(<<LargeBin/binary, 128>>), + {error, LargeList, <<128>>} = unicode:characters_to_nfd_list(<<LargeBin/binary, 128>>), + {error, LargeList, <<128>>} = unicode:characters_to_nfkd_list(<<LargeBin/binary, 128>>), + + {error, LargeBin, <<128>>} = unicode:characters_to_nfc_binary(<<LargeBin/binary, 128>>), + {error, LargeBin, <<128>>} = unicode:characters_to_nfkc_binary(<<LargeBin/binary, 128>>), + {error, LargeBin, <<128>>} = unicode:characters_to_nfd_binary(<<LargeBin/binary, 128>>), + {error, LargeBin, <<128>>} = unicode:characters_to_nfkd_binary(<<LargeBin/binary, 128>>), + ok. diff --git a/lib/stdlib/test/unicode_util_SUITE.erl b/lib/stdlib/test/unicode_util_SUITE.erl index e9b3d7f98d..03c24c7027 100644 --- a/lib/stdlib/test/unicode_util_SUITE.erl +++ b/lib/stdlib/test/unicode_util_SUITE.erl @@ -97,6 +97,8 @@ cp(_) -> "hejsan" = fetch(<<"hejsan">>, Get), "hejsan" = fetch(["hej",<<"san">>], Get), "hejsan" = fetch(["hej"|<<"san">>], Get), + {error, <<128>>} = Get(<<128>>), + {error, [<<128>>, 0]} = Get([<<128>>, 0]), ok. gc(Config) -> @@ -106,6 +108,8 @@ gc(Config) -> "hejsan" = fetch(<<"hejsan">>, Get), "hejsan" = fetch(["hej",<<"san">>], Get), "hejsan" = fetch(["hej"|<<"san">>], Get), + {error, <<128>>} = Get(<<128>>), + {error, [<<128>>, 0]} = Get([<<128>>, 0]), 0 = fold(fun verify_gc/3, 0, DataDir ++ "/GraphemeBreakTest.txt"), ok. diff --git a/lib/stdlib/uc_spec/gen_unicode_mod.escript b/lib/stdlib/uc_spec/gen_unicode_mod.escript index c8b815e435..fefd7d3b70 100755 --- a/lib/stdlib/uc_spec/gen_unicode_mod.escript +++ b/lib/stdlib/uc_spec/gen_unicode_mod.escript @@ -170,7 +170,7 @@ gen_header(Fd) -> io:put_chars(Fd, "-export([spec_version/0, lookup/1, get_case/1]).\n"), io:put_chars(Fd, "-inline([class/1]).\n"), io:put_chars(Fd, "-compile(nowarn_unused_vars).\n"), - io:put_chars(Fd, "-dialyzer({no_improper_lists, cp/1}).\n"), + io:put_chars(Fd, "-dialyzer({no_improper_lists, [cp/1, gc_prepend/2, gc_e_cont/2]}).\n"), io:put_chars(Fd, "-type gc() :: char()|[char()].\n\n\n"), ok. @@ -237,39 +237,43 @@ gen_static(Fd) -> gen_norm(Fd) -> io:put_chars(Fd, - "-spec nfd(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()).\n" + "-spec nfd(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()) | {error, unicode:chardata()}.\n" "nfd(Str0) ->\n" " case gc(Str0) of\n" " [GC|R] when GC < 127 -> [GC|R];\n" " [GC|Str] -> [decompose(GC)|Str];\n" - " [] -> []\n end.\n\n" + " [] -> [];\n" + " {error,_}=Error -> Error\n end.\n\n" ), io:put_chars(Fd, - "-spec nfkd(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()).\n" + "-spec nfkd(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()) | {error, unicode:chardata()}.\n" "nfkd(Str0) ->\n" " case gc(Str0) of\n" " [GC|R] when GC < 127 -> [GC|R];\n" " [GC|Str] -> [decompose_compat(GC)|Str];\n" - " [] -> []\n end.\n\n" + " [] -> [];\n" + " {error,_}=Error -> Error\n end.\n\n" ), io:put_chars(Fd, - "-spec nfc(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()).\n" + "-spec nfc(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()) | {error, unicode:chardata()}.\n" "nfc(Str0) ->\n" " case gc(Str0) of\n" " [GC|R] when GC < 255 -> [GC|R];\n" " [GC|Str] -> [compose(decompose(GC))|Str];\n" - " [] -> []\n end.\n\n" + " [] -> [];\n" + " {error,_}=Error -> Error\n end.\n\n" ), io:put_chars(Fd, - "-spec nfkc(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()).\n" + "-spec nfkc(unicode:chardata()) -> maybe_improper_list(gc(),unicode:chardata()) | {error, unicode:chardata()}.\n" "nfkc(Str0) ->\n" " case gc(Str0) of\n" " [GC|R] when GC < 127 -> [GC|R];\n" " [GC|Str] -> [compose_compat_0(decompose_compat(GC))|Str];\n" - " [] -> []\n end.\n\n" + " [] -> [];\n" + " {error,_}=Error -> Error\n end.\n\n" ), io:put_chars(Fd, @@ -448,18 +452,20 @@ gen_ws(Fd, Props) -> gen_cp(Fd) -> io:put_chars(Fd, "-spec cp(String::unicode:chardata()) ->" - " maybe_improper_list().\n"), + " maybe_improper_list() | {error, unicode:chardata()}.\n"), io:put_chars(Fd, "cp([C|_]=L) when is_integer(C) -> L;\n"), io:put_chars(Fd, "cp([List]) -> cp(List);\n"), io:put_chars(Fd, "cp([List|R]) ->\n"), io:put_chars(Fd, " case cp(List) of\n"), io:put_chars(Fd, " [] -> cp(R);\n"), io:put_chars(Fd, " [CP] -> [CP|R];\n"), - io:put_chars(Fd, " [C|R0] -> [C|[R0|R]]\n"), + io:put_chars(Fd, " [C|R0] -> [C|[R0|R]];\n"), + io:put_chars(Fd, " {error,Error} -> {error,[Error|R]}\n"), io:put_chars(Fd, " end;\n"), io:put_chars(Fd, "cp([]) -> [];\n"), io:put_chars(Fd, "cp(<<C/utf8, R/binary>>) -> [C|R];\n"), - io:put_chars(Fd, "cp(<<>>) -> [].\n\n"), + io:put_chars(Fd, "cp(<<>>) -> [];\n"), + io:put_chars(Fd, "cp(<<R/binary>>) -> {error,R}.\n\n"), ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -468,7 +474,7 @@ gen_gc(Fd, GBP) -> %% see http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundary_Rules io:put_chars(Fd, "-spec gc(String::unicode:chardata()) ->" - " maybe_improper_list().\n"), + " maybe_improper_list() | {error, unicode:chardata()}.\n"), io:put_chars(Fd, "gc(Str) ->\n" " gc_1(cp(Str)).\n\n" @@ -521,7 +527,8 @@ gen_gc(Fd, GBP) -> [GenEBG(CP) || CP <- merge_ranges(maps:get(e_base_gaz,GBP))], io:put_chars(Fd, "gc_1([CP|R]) -> gc_extend(R, CP);\n"), - io:put_chars(Fd, "gc_1([]) -> [].\n\n"), + io:put_chars(Fd, "gc_1([]) -> [];\n"), + io:put_chars(Fd, "gc_1({error,_}=Error) -> Error.\n\n"), io:put_chars(Fd, "%% Handle Prepend\n"), io:put_chars(Fd, @@ -536,7 +543,8 @@ gen_gc(Fd, GBP) -> " [GC|R1] -> [[CP0|GC]|R1]\n" " end\n" " end;\n" - " [] -> [CP0]\n" + " [] -> [CP0];\n" + " {error,R} -> [CP0|R]\n" " end.\n\n"), IsCtrl = fun(Range) -> io:format(Fd, "is_control~s true;\n", [gen_single_clause(Range)]) end, @@ -574,7 +582,10 @@ gen_gc(Fd, GBP) -> " [_]=Acc -> Acc;\n" " [_|_]=Acc -> [lists:reverse(Acc)];\n" " Acc -> [Acc]\n" - " end.\n\n"), + " end;\n" + "gc_extend({error,R}, T, Acc0) ->\n" + " gc_extend([], T, Acc0) ++ [R].\n\n" + ), [ZWJ] = maps:get(zwj, GBP), GenExtend = fun(R) when R =:= ZWJ -> io:format(Fd, "is_extend~s zwj;\n", [gen_single_clause(ZWJ)]); (Range) -> io:format(Fd, "is_extend~s true;\n", [gen_single_clause(Range)]) @@ -604,6 +615,11 @@ gen_gc(Fd, GBP) -> " case Acc of\n" " [A] -> [A];\n" " _ -> [lists:reverse(Acc)]\n" + " end;\n" + " {error,R} ->\n" + " case Acc of\n" + " [A] -> [A|R];\n" + " _ -> [lists:reverse(Acc)|R]\n" " end\n" " end.\n\n"), @@ -660,6 +676,7 @@ gen_gc(Fd, GBP) -> [GenHangulT_1(CP) || CP <- merge_ranges(maps:get(t,GBP))], io:put_chars(Fd, " R1 -> gc_extend(R1, R0, Acc)\n end.\n\n"), + io:put_chars(Fd, "gc_h_lv_lvt({error,_}=Error, Acc) -> gc_extend(Error, [], Acc);\n"), io:put_chars(Fd, "%% Handle Hangul LV\n"), GenHangulLV = fun(Range) -> io:format(Fd, "gc_h_lv_lvt~s gc_h_V(R1,[CP|Acc]);\n", [gen_clause2(Range)]) end, diff --git a/lib/wx/c_src/wxe_impl.cpp b/lib/wx/c_src/wxe_impl.cpp index 7817e7fa8a..1510866f09 100644 --- a/lib/wx/c_src/wxe_impl.cpp +++ b/lib/wx/c_src/wxe_impl.cpp @@ -267,7 +267,7 @@ int WxeApp::dispatch_cmds() return more; } -#define BREAK_BATCH 200 +#define BREAK_BATCH 10000 int WxeApp::dispatch(wxeFifo * batch) { @@ -284,7 +284,7 @@ int WxeApp::dispatch(wxeFifo * batch) if(blevel>0) { blevel--; if(blevel==0) - wait += BREAK_BATCH*100; + wait += BREAK_BATCH/4; } break; case WXE_BATCH_BEGIN: @@ -317,7 +317,7 @@ int WxeApp::dispatch(wxeFifo * batch) erl_drv_mutex_lock(wxe_batch_locker_m); batch->Cleanup(); } - if(blevel <= 0 || wait > BREAK_BATCH) { + if(blevel <= 0 || wait >= BREAK_BATCH) { erl_drv_mutex_unlock(wxe_batch_locker_m); if(blevel > 0) { return 1; // We are still in a batch but we can let wx check for events diff --git a/lib/xmerl/doc/src/notes.xml b/lib/xmerl/doc/src/notes.xml index ef4831e6ec..56856d026e 100644 --- a/lib/xmerl/doc/src/notes.xml +++ b/lib/xmerl/doc/src/notes.xml @@ -32,6 +32,27 @@ <p>This document describes the changes made to the Xmerl application.</p> +<section><title>Xmerl 1.3.14</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> A couple of bugs are fixed in the sax parser + (xmerl_sax_parser). </p> <list> <item>The continuation + function was not called correctly when the XML directive + was fragmented. </item> <item>When the event callback + modules (xmerl_sax_old_dom and xmerl_sax_simple) got an + endDocument event at certain conditions the parser + crashed.</item> <item>Replaced internal ets table with + map to avoid table leakage.</item> </list> + <p> + Own Id: OTP-14430</p> + </item> + </list> + </section> + +</section> + <section><title>Xmerl 1.3.13</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/xmerl/src/xmerl_sax_old_dom.erl b/lib/xmerl/src/xmerl_sax_old_dom.erl index fefcf03fce..411121370f 100644 --- a/lib/xmerl/src/xmerl_sax_old_dom.erl +++ b/lib/xmerl/src/xmerl_sax_old_dom.erl @@ -127,9 +127,10 @@ build_dom(endDocument, State#xmerl_sax_old_dom_state{dom=[Decl, Current#xmlElement{ content=lists:reverse(C) }]}; - _ -> - %%?dbg("~p\n", [D]), - ?error("we're not at end the document when endDocument event is encountered.") + _ -> + %% endDocument is also sent by the parser when a fault occur to tell + %% the event receiver that no more input will be sent + State end; %% Element diff --git a/lib/xmerl/src/xmerl_sax_parser.erl b/lib/xmerl/src/xmerl_sax_parser.erl index 1aef6c58c4..e383c4c349 100644 --- a/lib/xmerl/src/xmerl_sax_parser.erl +++ b/lib/xmerl/src/xmerl_sax_parser.erl @@ -64,7 +64,7 @@ %% Description: Parse file containing an XML document. %%---------------------------------------------------------------------- file(Name,Options) -> - case file:open(Name, [raw, read,binary]) of + case file:open(Name, [raw, read_ahead, read,binary]) of {error, Reason} -> {error,{Name, file:format_error(Reason)}}; {ok, FD} -> @@ -120,21 +120,22 @@ stream(Xml, Options, InputType) when is_binary(Xml), is_list(Options) -> normal -> parse end, - case detect_charset(Xml, State) of - {error, Reason} -> {fatal_error, - { - State#xmerl_sax_parser_state.current_location, - State#xmerl_sax_parser_state.entity, - 1 - }, - Reason, - [], - State#xmerl_sax_parser_state.event_state}; - {Xml1, State1} -> - parse_binary(Xml1, - State1#xmerl_sax_parser_state{input_type = InputType}, - ParseFunction) - end + try + {Xml1, State1} = detect_charset(Xml, State), + parse_binary(Xml1, + State1#xmerl_sax_parser_state{input_type = InputType}, + ParseFunction) + catch + throw:{fatal_error, {State2, Reason}} -> + {fatal_error, + { + State2#xmerl_sax_parser_state.current_location, + State2#xmerl_sax_parser_state.entity, + 1 + }, + Reason, [], + State2#xmerl_sax_parser_state.event_state} + end end. %%---------------------------------------------------------------------- @@ -156,8 +157,8 @@ parse_binary(Xml, #xmerl_sax_parser_state{encoding={utf16,big}}=State, F) -> xmerl_sax_parser_utf16be:F(Xml, State); parse_binary(Xml, #xmerl_sax_parser_state{encoding=latin1}=State, F) -> xmerl_sax_parser_latin1:F(Xml, State); -parse_binary(_, #xmerl_sax_parser_state{encoding=Enc}, _) -> - {error, lists:flatten(io_lib:format("Charcter set ~p not supported", [Enc]))}. +parse_binary(_, #xmerl_sax_parser_state{encoding=Enc}, State) -> + ?fatal_error(State, lists:flatten(io_lib:format("Charcter set ~p not supported", [Enc]))). %%---------------------------------------------------------------------- %% Function: initial_state/0 @@ -211,8 +212,7 @@ parse_options([{entity, Entity} |Options], State) -> parse_options([skip_external_dtd |Options], State) -> parse_options(Options, State#xmerl_sax_parser_state{skip_external_dtd = true}); parse_options([O |_], _State) -> - {error, - lists:flatten(io_lib:format("Option: ~p not supported", [O]))}. + {error, lists:flatten(io_lib:format("Option: ~p not supported", [O]))}. check_encoding_option(E) when E==utf8; E=={utf16,little}; E=={utf16,big}; @@ -230,16 +230,10 @@ check_encoding_option(E) -> %% Output: {utf8|utf16le|utf16be|iso8859, Xml, State} %% Description: Detects which character set is used in a binary stream. %%---------------------------------------------------------------------- -detect_charset(<<>>, #xmerl_sax_parser_state{continuation_fun = undefined} = _) -> - {error, "Can't detect character encoding due to no indata"}; -detect_charset(<<>>, #xmerl_sax_parser_state{continuation_fun = CFun, - continuation_state = CState} = State) -> - case CFun(CState) of - {<<>>, _} -> - {error, "Can't detect character encoding due to lack of indata"}; - {NewBytes, NewContState} -> - detect_charset(NewBytes, State#xmerl_sax_parser_state{continuation_state = NewContState}) - end; +detect_charset(<<>>, #xmerl_sax_parser_state{continuation_fun = undefined} = State) -> + ?fatal_error(State, "Can't detect character encoding due to lack of indata"); +detect_charset(<<>>, State) -> + cf(<<>>, State, fun detect_charset/2); detect_charset(Bytes, State) -> case unicode:bom_to_encoding(Bytes) of {latin1, 0} -> @@ -249,25 +243,47 @@ detect_charset(Bytes, State) -> {RealBytes, State#xmerl_sax_parser_state{encoding=Enc}} end. +detect_charset_1(<<16#00>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#00, 16#3C>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#00, 16#3C, 16#00>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); detect_charset_1(<<16#00, 16#3C, 16#00, 16#3F, _/binary>> = Xml, State) -> {Xml, State#xmerl_sax_parser_state{encoding={utf16, big}}}; +detect_charset_1(<<16#3C>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#00>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#00, 16#3F>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); detect_charset_1(<<16#3C, 16#00, 16#3F, 16#00, _/binary>> = Xml, State) -> {Xml, State#xmerl_sax_parser_state{encoding={utf16, little}}}; -detect_charset_1(<<16#3C, 16#3F, 16#78, 16#6D, 16#6C, Xml2/binary>> = Xml, State) -> - case parse_xml_directive(Xml2) of +detect_charset_1(<<16#3C>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#3F>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#3F, 16#78>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#3F, 16#78, 16#6D>> = Xml, State) -> + cf(Xml, State, fun detect_charset_1/2); +detect_charset_1(<<16#3C, 16#3F, 16#78, 16#6D, 16#6C, Xml2/binary>>, State) -> + {Xml3, State1} = read_until_end_of_xml_directive(Xml2, State), + case parse_xml_directive(Xml3) of {error, Reason} -> - {error, Reason}; + ?fatal_error(State, Reason); AttrList -> case lists:keysearch("encoding", 1, AttrList) of {value, {_, E}} -> case convert_encoding(E) of {error, Reason} -> - {error, Reason}; + ?fatal_error(State, Reason); Enc -> - {Xml, State#xmerl_sax_parser_state{encoding=Enc}} + {<<16#3C, 16#3F, 16#78, 16#6D, 16#6C, Xml3/binary>>, + State1#xmerl_sax_parser_state{encoding=Enc}} end; _ -> - {Xml, State} + {<<16#3C, 16#3F, 16#78, 16#6D, 16#6C, Xml3/binary>>, State1} end end; detect_charset_1(Xml, State) -> @@ -377,7 +393,7 @@ parse_value_1(<<C, Rest/binary>>, Stop, Acc) -> parse_value_1(Rest, Stop, [C |Acc]). %%====================================================================== -%%Default functions +%% Default functions %%====================================================================== %%---------------------------------------------------------------------- %% Function: default_event_cb(Event, LineNo, State) -> Result @@ -393,7 +409,7 @@ default_event_cb(_Event, _LineNo, State) -> %%---------------------------------------------------------------------- %% Function: default_continuation_cb(IoDevice) -> Result %% IoDevice = iodevice() -%% Output: Result = {[char()], State} +%% Output: Result = {binary(), IoDevice} %% Description: Default continuation callback reading blocks. %%---------------------------------------------------------------------- default_continuation_cb(IoDevice) -> @@ -403,3 +419,82 @@ default_continuation_cb(IoDevice) -> {ok, FileBin} -> {FileBin, IoDevice} end. + +%%---------------------------------------------------------------------- +%% Function: read_until_end_of_xml_directive(Rest, State) -> Result +%% Rest = binary() +%% Output: Result = {binary(), State} +%% Description: Reads a utf8 or latin1 until it finds '?>' +%%---------------------------------------------------------------------- +read_until_end_of_xml_directive(Rest, State) -> + case binary:match(Rest, <<"?>">>) of + nomatch -> + case cf(Rest, State) of + {<<>>, _} -> + ?fatal_error(State, "Can't detect character encoding due to lack of indata"); + {NewBytes, NewState} -> + read_until_end_of_xml_directive(NewBytes, NewState) + end; + _ -> + {Rest, State} + end. + + +%%---------------------------------------------------------------------- +%% Function : cf(Rest, State) -> Result +%% Parameters: Rest = binary() +%% State = #xmerl_sax_parser_state{} +%% NextCall = fun() +%% Result : {Rest, State} +%% Description: Function that uses provided fun to read another chunk from +%% input stream and calls the fun in NextCall. +%%---------------------------------------------------------------------- +cf(_Rest, #xmerl_sax_parser_state{continuation_fun = undefined} = State) -> + ?fatal_error(State, "Continuation function undefined"); +cf(Rest, #xmerl_sax_parser_state{continuation_fun = CFun, continuation_state = CState} = State) -> + Result = + try + CFun(CState) + catch + throw:ErrorTerm -> + ?fatal_error(State, ErrorTerm); + exit:Reason -> + ?fatal_error(State, {'EXIT', Reason}) + end, + case Result of + {<<>>, _} -> + ?fatal_error(State, "Can't detect character encoding due to lack of indata"); + {NewBytes, NewContState} -> + {<<Rest/binary, NewBytes/binary>>, + State#xmerl_sax_parser_state{continuation_state = NewContState}} + end. + +%%---------------------------------------------------------------------- +%% Function : cf(Rest, State, NextCall) -> Result +%% Parameters: Rest = binary() +%% State = #xmerl_sax_parser_state{} +%% NextCall = fun() +%% Result : {Rest, State} +%% Description: Function that uses provided fun to read another chunk from +%% input stream and calls the fun in NextCall. +%%---------------------------------------------------------------------- +cf(_Rest, #xmerl_sax_parser_state{continuation_fun = undefined} = State, _) -> + ?fatal_error(State, "Continuation function undefined"); +cf(Rest, #xmerl_sax_parser_state{continuation_fun = CFun, continuation_state = CState} = State, + NextCall) -> + Result = + try + CFun(CState) + catch + throw:ErrorTerm -> + ?fatal_error(State, ErrorTerm); + exit:Reason -> + ?fatal_error(State, {'EXIT', Reason}) + end, + case Result of + {<<>>, _} -> + ?fatal_error(State, "Can't detect character encoding due to lack of indata"); + {NewBytes, NewContState} -> + NextCall(<<Rest/binary, NewBytes/binary>>, + State#xmerl_sax_parser_state{continuation_state = NewContState}) + end. diff --git a/lib/xmerl/src/xmerl_sax_parser_base.erlsrc b/lib/xmerl/src/xmerl_sax_parser_base.erlsrc index f3470b2809..1dca9608cb 100644 --- a/lib/xmerl/src/xmerl_sax_parser_base.erlsrc +++ b/lib/xmerl/src/xmerl_sax_parser_base.erlsrc @@ -64,38 +64,42 @@ %% Description: Parsing XML from input stream. %%---------------------------------------------------------------------- parse(Xml, State) -> - RefTable = ets:new(xmerl_sax_entity_refs, [private]), - - State1 = event_callback(startDocument, State), - - case catch parse_document(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}) of - {ok, Rest, State2} -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - case check_if_rest_ok(State3#xmerl_sax_parser_state.input_type, Rest) of - true -> - {ok, State3#xmerl_sax_parser_state.event_state, Rest}; - false -> - format_error(fatal_error, State3, "Input found after legal document") - end; - {fatal_error, {State2, Reason}} -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - format_error(fatal_error, State3, Reason); - {event_receiver_error, State2, {Tag, Reason}} -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - format_error(Tag, State3, Reason); - {endDocument, Rest, State2} -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - {ok, State3#xmerl_sax_parser_state.event_state, Rest}; - Other -> - _State2 = event_callback(endDocument, State1), - ets:delete(RefTable), - {fatal_error, Other} + RefTable = maps:new(), + + try + State1 = event_callback(startDocument, State), + Result = parse_document(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}), + handle_end_document(Result) + catch + throw:Exception -> + handle_end_document(Exception); + _:OtherError -> + handle_end_document({other, OtherError, State}) end. + % case catch parse_document(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}) of + % {ok, Rest, State2} -> + % State3 = event_callback(endDocument, State2), + % case check_if_rest_ok(State3#xmerl_sax_parser_state.input_type, Rest) of + % true -> + % {ok, State3#xmerl_sax_parser_state.event_state, Rest}; + % false -> + % format_error(fatal_error, State3, "Input found after legal document") + % end; + % {fatal_error, {State2, Reason}} -> + % State3 = event_callback(endDocument, State2), + % format_error(fatal_error, State3, Reason); + % {event_receiver_error, State2, {Tag, Reason}} -> + % State3 = event_callback(endDocument, State2), + % format_error(Tag, State3, Reason); + % {endDocument, Rest, State2} -> + % State3 = event_callback(endDocument, State2), + % {ok, State3#xmerl_sax_parser_state.event_state, Rest}; + % Other -> + % _State2 = event_callback(endDocument, State1), + % {fatal_error, Other} + % end. + %%---------------------------------------------------------------------- %% Function: parse_dtd(Xml, State) -> Result %% Input: Xml = string() | binary() @@ -105,38 +109,120 @@ parse(Xml, State) -> %% Description: Parsing XML DTD from input stream. %%---------------------------------------------------------------------- parse_dtd(Xml, State) -> - RefTable = ets:new(xmerl_sax_entity_refs, [private]), - - State1 = event_callback(startDocument, State), - - case catch parse_external_entity_1(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}) of - {fatal_error, {State2, Reason}} -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - format_error(fatal_error, State3, Reason); - {event_receiver_error, State2, {Tag, Reason}} -> - State3 = event_callback(endDocument, State2), - format_error(Tag, State3, Reason); - {Rest, State2} when is_record(State2, xmerl_sax_parser_state) -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - {ok, State3#xmerl_sax_parser_state.event_state, Rest}; - {endDocument, Rest, State2} when is_record(State2, xmerl_sax_parser_state) -> - State3 = event_callback(endDocument, State2), - ets:delete(RefTable), - {ok, State3#xmerl_sax_parser_state.event_state, Rest}; - Other -> - _State2 = event_callback(endDocument, State1), - ets:delete(RefTable), - {fatal_error, Other} + RefTable = maps:new(), + + try + State1 = event_callback(startDocument, State), + Result = parse_external_entity_1(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}), + handle_end_document(Result) + catch + throw:Exception -> + handle_end_document(Exception); + _:OtherError -> + handle_end_document({other, OtherError, State}) end. + % case catch parse_external_entity_1(Xml, State1#xmerl_sax_parser_state{ref_table=RefTable}) of + % {fatal_error, {State2, Reason}} -> + % State3 = event_callback(endDocument, State2), + % format_error(fatal_error, State3, Reason); + % {event_receiver_error, State2, {Tag, Reason}} -> + % State3 = event_callback(endDocument, State2), + % format_error(Tag, State3, Reason); + % {Rest, State2} when is_record(State2, xmerl_sax_parser_state) -> + % State3 = event_callback(endDocument, State2), + % {ok, State3#xmerl_sax_parser_state.event_state, Rest}; + % {endDocument, Rest, State2} when is_record(State2, xmerl_sax_parser_state) -> + % State3 = event_callback(endDocument, State2), + % {ok, State3#xmerl_sax_parser_state.event_state, Rest}; + % Other -> + % _State2 = event_callback(endDocument, State1), + % {fatal_error, Other} + % end. + + %%====================================================================== %% Internal functions %%====================================================================== %%---------------------------------------------------------------------- +%% Function: handle_end_document(ParserResult) -> Result +%% Input: ParseResult = term() +%% Output: Result = {ok, Rest, EventState} | +%% EventState = term() +%% Description: Ends the parsing and formats output +%%---------------------------------------------------------------------- +handle_end_document({ok, Rest, State}) -> + %%ok case from parse + try + State1 = event_callback(endDocument, State), + case check_if_rest_ok(State1#xmerl_sax_parser_state.input_type, Rest) of + true -> + {ok, State1#xmerl_sax_parser_state.event_state, Rest}; + false -> + format_error(fatal_error, State1, "Input found after legal document") + end + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end; +handle_end_document({endDocument, Rest, State}) -> + %% ok case from parse and parse_dtd + try + State1 = event_callback(endDocument, State), + {ok, State1#xmerl_sax_parser_state.event_state, Rest} + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end; +handle_end_document({fatal_error, {State, Reason}}) -> + try + State1 = event_callback(endDocument, State), + format_error(fatal_error, State1, Reason) + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end; +handle_end_document({event_receiver_error, State, {Tag, Reason}}) -> + try + State1 = event_callback(endDocument, State), + format_error(Tag, State1, Reason) + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end; +handle_end_document({Rest, State}) when is_record(State, xmerl_sax_parser_state) -> + %%ok case from parse_dtd + try + State1 = event_callback(endDocument, State), + {ok, State1#xmerl_sax_parser_state.event_state, Rest} + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end; +handle_end_document({other, Error, State}) -> + try + _State1 = event_callback(endDocument, State), + {fatal_error, Error} + catch + throw:{event_receiver_error, State2, {Tag, Reason}} -> + format_error(Tag, State2, Reason); + _:Other -> + {fatal_error, Other} + end. + +%%---------------------------------------------------------------------- %% Function: parse_document(Rest, State) -> Result %% Input: Rest = string() | binary() %% State = #xmerl_sax_parser_state{} @@ -439,6 +525,7 @@ check_if_rest_ok(stream, _) -> check_if_rest_ok(_, _) -> false. + %%---------------------------------------------------------------------- %% Function: parse_pi_1(Rest, State) -> Result %% Input: Rest = string() | binary() @@ -883,11 +970,11 @@ send_end_prefix_mapping_event([{Prefix, _Uri} |Ns], State) -> parse_eq(?STRING_EMPTY, State) -> cf(?STRING_EMPTY, State, fun parse_eq/2); parse_eq(?STRING_REST("=", Rest), State) -> - {Rest, State}; + {Rest, State}; parse_eq(?STRING_UNBOUND_REST(C, _) = Bytes, State) when ?is_whitespace(C) -> - {_WS, Rest, State1} = - whitespace(Bytes, State, []), - parse_eq(Rest, State1); + {_WS, Rest, State1} = + whitespace(Bytes, State, []), + parse_eq(Rest, State1); parse_eq(Bytes, State) -> unicode_incomplete_check([Bytes, State, fun parse_eq/2], "expecting = or whitespace"). @@ -905,11 +992,11 @@ parse_eq(Bytes, State) -> parse_att_value(?STRING_EMPTY, State) -> cf(?STRING_EMPTY, State, fun parse_att_value/2); parse_att_value(?STRING_UNBOUND_REST(C, Rest), State) when C == $'; C == $" -> - parse_att_value(Rest, State, C, []); + parse_att_value(Rest, State, C, []); parse_att_value(?STRING_UNBOUND_REST(C, _) = Bytes, State) when ?is_whitespace(C) -> - {_WS, Rest, State1} = - whitespace(Bytes, State, []), - parse_att_value(Rest, State1); + {_WS, Rest, State1} = + whitespace(Bytes, State, []), + parse_att_value(Rest, State1); parse_att_value(Bytes, State) -> unicode_incomplete_check([Bytes, State, fun parse_att_value/2], "\', \" or whitespace expected"). @@ -1360,17 +1447,20 @@ parse_pe_reference_1(Bytes, State, Name) -> %%---------------------------------------------------------------------- -%% Function: insert_reference(Reference, State) -> Result -%% Parameters: Reference = string() +%% Function: insert_reference(Name, Ref, State) -> Result +%% Parameters: Name = string() +%% Ref = {Type, Value} +%% Type = atom() +%% Value = term() %% State = #xmerl_sax_parser_state{} %% Result : %%---------------------------------------------------------------------- -insert_reference({Name, Type, Value}, Table) -> - case ets:lookup(Table, Name) of - [{Name, _, _}] -> - ok; +insert_reference(Name, Value, #xmerl_sax_parser_state{ref_table = Map} = State) -> + case maps:find(Name, Map) of + error -> + State#xmerl_sax_parser_state{ref_table = maps:put(Name, Value, Map)}; _ -> - ets:insert(Table, {Name, Type, Value}) + State end. @@ -1391,8 +1481,8 @@ look_up_reference("apos", _, _) -> look_up_reference("quot", _, _) -> {internal_general, "quot", "\""}; look_up_reference(Name, HaveToExist, State) -> - case ets:lookup(State#xmerl_sax_parser_state.ref_table, Name) of - [{Name, Type, Value}] -> + case maps:find(Name, State#xmerl_sax_parser_state.ref_table) of + {ok, {Type, Value}} -> {Type, Name, Value}; _ -> case HaveToExist of @@ -1474,7 +1564,7 @@ parse_system_litteral(?STRING_EMPTY, State, Stop, Acc) -> parse_system_litteral(?STRING_UNBOUND_REST(Stop, Rest), State, Stop, Acc) -> {lists:reverse(Acc), Rest, State}; parse_system_litteral(?STRING_UNBOUND_REST(C, Rest), State, Stop, Acc) -> - parse_system_litteral(Rest, State, Stop, [C |Acc]); + parse_system_litteral(Rest, State, Stop, [C |Acc]); parse_system_litteral(Bytes, State, Stop, Acc) -> unicode_incomplete_check([Bytes, State, Stop, Acc, fun parse_system_litteral/4], undefined). @@ -1646,9 +1736,11 @@ parse_external_entity(State, _PubId, SysId) -> end_tags = []}, - EventState = handle_external_entity(ExtRef, State1), + {EventState, RefTable} = handle_external_entity(ExtRef, State1), - NewState = event_callback({endEntity, SysId}, SaveState#xmerl_sax_parser_state{event_state=EventState}), + NewState = event_callback({endEntity, SysId}, + SaveState#xmerl_sax_parser_state{event_state=EventState, + ref_table=RefTable}), NewState#xmerl_sax_parser_state{file_type=normal}. @@ -1675,7 +1767,8 @@ handle_external_entity({file, FileToOpen}, State) -> entity=filename:basename(FileToOpen), input_type=file}), ok = file:close(FD), - EntityState#xmerl_sax_parser_state.event_state + {EntityState#xmerl_sax_parser_state.event_state, + EntityState#xmerl_sax_parser_state.ref_table} end; handle_external_entity({http, Url}, State) -> @@ -1695,7 +1788,9 @@ handle_external_entity({http, Url}, State) -> input_type=file}), ok = file:close(FD), ok = file:delete(TmpFile), - EntityState#xmerl_sax_parser_state.event_state + {EntityState#xmerl_sax_parser_state.event_state, + EntityState#xmerl_sax_parser_state.ref_table} + end catch throw:{error, Error} -> @@ -1716,7 +1811,7 @@ handle_external_entity({Tag, _Url}, State) -> parse_external_entity_1(?STRING_EMPTY, #xmerl_sax_parser_state{file_type=Type} = State) -> case catch cf(?STRING_EMPTY, State, fun parse_external_entity_1/2) of {Rest, State1} when is_record(State1, xmerl_sax_parser_state) -> - {Rest, State}; + {Rest, State1}; {fatal_error, {State1, "No more bytes"}} when Type == dtd; Type == entity -> {?STRING_EMPTY, State1}; Other -> @@ -2442,24 +2537,24 @@ parse_entity_def(?STRING_EMPTY, State, Name) -> cf(?STRING_EMPTY, State, Name, fun parse_entity_def/3); parse_entity_def(?STRING_UNBOUND_REST(C, Rest), State, Name) when C == $'; C == $" -> {Value, Rest1, State1} = parse_entity_value(Rest, State, C, []), - insert_reference({Name, internal_general, Value}, State1#xmerl_sax_parser_state.ref_table), - State2 = event_callback({internalEntityDecl, Name, Value}, State1), - {_WS, Rest2, State3} = whitespace(Rest1, State2, []), - parse_def_end(Rest2, State3); + State2 = insert_reference(Name, {internal_general, Value}, State1), + State3 = event_callback({internalEntityDecl, Name, Value}, State2), + {_WS, Rest2, State4} = whitespace(Rest1, State3, []), + parse_def_end(Rest2, State4); parse_entity_def(?STRING_UNBOUND_REST(C, _) = Rest, State, Name) when C == $S; C == $P -> {PubId, SysId, Rest1, State1} = parse_external_id(Rest, State, false), {Ndata, Rest2, State2} = parse_ndata(Rest1, State1), case Ndata of undefined -> - insert_reference({Name, external_general, {PubId, SysId}}, - State2#xmerl_sax_parser_state.ref_table), - State3 = event_callback({externalEntityDecl, Name, PubId, SysId}, State2), - {Rest2, State3}; + State3 = insert_reference(Name, {external_general, {PubId, SysId}}, + State2), + State4 = event_callback({externalEntityDecl, Name, PubId, SysId}, State3), + {Rest2, State4}; _ -> - insert_reference({Name, unparsed, {PubId, SysId, Ndata}}, - State2#xmerl_sax_parser_state.ref_table), - State3 = event_callback({unparsedEntityDecl, Name, PubId, SysId, Ndata}, State2), - {Rest2, State3} + State3 = insert_reference(Name, {unparsed, {PubId, SysId, Ndata}}, + State2), + State4 = event_callback({unparsedEntityDecl, Name, PubId, SysId, Ndata}, State3), + {Rest2, State4} end; parse_entity_def(Bytes, State, Name) -> unicode_incomplete_check([Bytes, State, Name, fun parse_entity_def/3], @@ -2636,19 +2731,19 @@ parse_pe_def(?STRING_EMPTY, State, Name) -> parse_pe_def(?STRING_UNBOUND_REST(C, Rest), State, Name) when C == $'; C == $" -> {Value, Rest1, State1} = parse_entity_value(Rest, State, C, []), Name1 = "%" ++ Name, - insert_reference({Name1, internal_parameter, Value}, - State1#xmerl_sax_parser_state.ref_table), - State2 = event_callback({internalEntityDecl, Name1, Value}, State1), - {_WS, Rest2, State3} = whitespace(Rest1, State2, []), - parse_def_end(Rest2, State3); + State2 = insert_reference(Name1, {internal_parameter, Value}, + State1), + State3 = event_callback({internalEntityDecl, Name1, Value}, State2), + {_WS, Rest2, State4} = whitespace(Rest1, State3, []), + parse_def_end(Rest2, State4); parse_pe_def(?STRING_UNBOUND_REST(C, _) = Bytes, State, Name) when C == $S; C == $P -> {PubId, SysId, Rest1, State1} = parse_external_id(Bytes, State, false), Name1 = "%" ++ Name, - insert_reference({Name1, external_parameter, {PubId, SysId}}, - State1#xmerl_sax_parser_state.ref_table), - State2 = event_callback({externalEntityDecl, Name1, PubId, SysId}, State1), - {_WS, Rest2, State3} = whitespace(Rest1, State2, []), - parse_def_end(Rest2, State3); + State2 = insert_reference(Name1, {external_parameter, {PubId, SysId}}, + State1), + State3 = event_callback({externalEntityDecl, Name1, PubId, SysId}, State2), + {_WS, Rest2, State4} = whitespace(Rest1, State3, []), + parse_def_end(Rest2, State4); parse_pe_def(Bytes, State, Name) -> unicode_incomplete_check([Bytes, State, Name, fun parse_pe_def/3], "\", \', SYSTEM or PUBLIC expected"). diff --git a/lib/xmerl/src/xmerl_sax_simple_dom.erl b/lib/xmerl/src/xmerl_sax_simple_dom.erl index 7eb3afd499..d842bd982b 100644 --- a/lib/xmerl/src/xmerl_sax_simple_dom.erl +++ b/lib/xmerl/src/xmerl_sax_simple_dom.erl @@ -129,8 +129,9 @@ build_dom(endDocument, State#xmerl_sax_simple_dom_state{dom=[Decl, {Tag, Attributes, lists:reverse(Content)}]}; _ -> - ?dbg("~p\n", [D]), - ?error("we're not at end the document when endDocument event is encountered.") + %% endDocument is also sent by the parser when a fault occur to tell + %% the event receiver that no more input will be sent + State end; %% Element diff --git a/lib/xmerl/test/Makefile b/lib/xmerl/test/Makefile index e5c89f84b9..3204f081ba 100644 --- a/lib/xmerl/test/Makefile +++ b/lib/xmerl/test/Makefile @@ -126,5 +126,6 @@ release_tests_spec: opt @tar cfh - xmerl_xsd_MS2002-01-16_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) @tar cfh - xmerl_xsd_NIST2002-01-16_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) @tar cfh - xmerl_xsd_Sun2002-01-16_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) + @tar cfh - xmerl_sax_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) @tar cfh - xmerl_sax_stream_SUITE_data | (cd "$(RELSYSDIR)"; tar xf -) chmod -R u+w "$(RELSYSDIR)" diff --git a/lib/xmerl/test/xmerl_sax_SUITE.erl b/lib/xmerl/test/xmerl_sax_SUITE.erl index eb9cefe0df..68b9bcc4a2 100644 --- a/lib/xmerl/test/xmerl_sax_SUITE.erl +++ b/lib/xmerl/test/xmerl_sax_SUITE.erl @@ -33,7 +33,6 @@ %%====================================================================== %% External functions %%====================================================================== - %%---------------------------------------------------------------------- %% Initializations %%---------------------------------------------------------------------- @@ -42,12 +41,15 @@ all() -> [{group, bugs}]. groups() -> - [{bugs, [], [ticket_8213, ticket_8214, ticket_11551]}]. + [{bugs, [], [ticket_8213, ticket_8214, ticket_11551, + fragmented_xml_directive, + old_dom_event_fun_endDocument_bug, + event_fun_endDocument_error_test, + event_fun_startDocument_error_test]}]. -%%---------------------------------------------------------------------- +%%====================================================================== %% Tests -%%---------------------------------------------------------------------- - +%%====================================================================== %%---------------------------------------------------------------------- %% Test Case %% ID: ticket_8213 @@ -56,7 +58,6 @@ ticket_8213(_Config) -> {ok,ok,[]} = xmerl_sax_parser:stream("<elem/>", [{event_fun, fun (_E,_,_) -> ok end}]), ok. - %%---------------------------------------------------------------------- %% Test Case %% ID: ticket_8214 @@ -99,3 +100,66 @@ ticket_11551(_Config) -> <a>hej</a>">>, {ok, undefined, <<"\n\n<?xml", _/binary>>} = xmerl_sax_parser:stream(Stream3, []), ok. + +%%---------------------------------------------------------------------- +%% Test Case +%% ID: fragmented_xml_directive +%% Test of fragmented xml directive by reading one byte per continuation ca +fragmented_xml_directive(Config) -> + DataDir = proplists:get_value(data_dir, Config), + Name = filename:join(DataDir, "test_data_1.xml"), + {ok, Fd} = file:open(Name, [raw, read,binary]), + Cf = fun cf_fragmented_xml_directive/1, + {ok, undefined, _} = xmerl_sax_parser:stream(<<>>, + [{continuation_fun, Cf}, + {continuation_state, Fd}]), + ok. + +%%---------------------------------------------------------------------- +%% Test Case +%% ID: old_dom_event_fun_endDocument_bug +%% The old_dom backend previous generateded an uncatched exception +%% instead of the correct fatal_error from the parser. +old_dom_event_fun_endDocument_bug(_Config) -> + %% Stream contains bad characters, + {fatal_error, _, _, _, _} = + xmerl_sax_parser:stream([60,63,120,109,108,32,118,101,114,115,105,111,110,61,39,49,46,48,39,32,101,110,99,111,100,105,110,103,61,39,117,116,102,45,56,39,63,62,60, + 99,111,109,109,97,110,100,62,60,104,101,97,100,101,114,62,60,116,114,97,110,115,97,99,116,105,111,110,73,100,62,49,60,47,116,114,97,110, + 115,97,99,116,105,111,110,73,100,62,60,47,104,101,97,100,101,114,62,60,98,111,100,121,62,95,226,130,172,59,60,60,47,98,111,100,121,62,60, + 47,99,111,109,109,97,110,100,62,60,47,120,49,95,49,62], + [{event_fun,fun xmerl_sax_old_dom:event/3}, + {event_state,xmerl_sax_old_dom:initial_state()}]), + ok. + +%%---------------------------------------------------------------------- +%% Test Case +%% ID: event_fun_endDocument_error_test +event_fun_endDocument_error_test(_Config) -> + Stream = <<"<?xml version=\"1.0\" encoding=\"utf-8\"?><a>hej</a>">>, + Ef = fun(endDocument, _ , _) -> throw({event_error, "endDocument error"}); + (_, _, S) -> S + end, + {event_error, _, _, _, _} = xmerl_sax_parser:stream(Stream, [{event_fun, Ef}]), + ok. + +%%---------------------------------------------------------------------- +%% Test Case +%% ID: event_fun_startDocument_error_test +event_fun_startDocument_error_test(_Config) -> + Stream = <<"<?xml version=\"1.0\" encoding=\"utf-8\"?><a>hej</a>">>, + Ef = fun(startDocument, _ , _) -> throw({event_error, "endDocument error"}); + (_, _, S) -> S + end, + {event_error, _, _, _, _} = xmerl_sax_parser:stream(Stream, [{event_fun, Ef}]), + ok. + +%%====================================================================== +%% Internal functions +%%====================================================================== +cf_fragmented_xml_directive(IoDevice) -> + case file:read(IoDevice, 1) of + eof -> + {<<>>, IoDevice}; + {ok, FileBin} -> + {FileBin, IoDevice} + end. diff --git a/lib/xmerl/test/xmerl_sax_SUITE_data/test_data_1.xml b/lib/xmerl/test/xmerl_sax_SUITE_data/test_data_1.xml new file mode 100644 index 0000000000..efbaee6b81 --- /dev/null +++ b/lib/xmerl/test/xmerl_sax_SUITE_data/test_data_1.xml @@ -0,0 +1,4 @@ +<?xml version="1.0" encoding="utf-8"?> +<a> +Hej +</a> diff --git a/lib/xmerl/test/xmerl_sax_stream_SUITE.erl b/lib/xmerl/test/xmerl_sax_stream_SUITE.erl index a306eb66a2..7315f67374 100644 --- a/lib/xmerl/test/xmerl_sax_stream_SUITE.erl +++ b/lib/xmerl/test/xmerl_sax_stream_SUITE.erl @@ -40,7 +40,8 @@ all() -> [ one_document, two_documents, - one_document_and_junk + one_document_and_junk, + end_of_stream ]. %%---------------------------------------------------------------------- @@ -62,6 +63,9 @@ end_per_testcase(_Func, _Config) -> %%---------------------------------------------------------------------- %% Tests %%---------------------------------------------------------------------- + +%%---------------------------------------------------------------------- +%% Send One doc over stream one_document(Config) -> Port = 11111, @@ -107,6 +111,8 @@ one_document(Config) -> end, ok. +%%---------------------------------------------------------------------- +%% Send Two doc over stream two_documents(Config) -> Port = 11111, @@ -152,6 +158,8 @@ two_documents(Config) -> end, ok. +%%---------------------------------------------------------------------- +%% Send one doc and then junk on stream one_document_and_junk(Config) -> Port = 11111, @@ -196,6 +204,13 @@ one_document_and_junk(Config) -> ct:fail("Timeout") end, ok. + +%%---------------------------------------------------------------------- +%% Test of continuation when end of stream +end_of_stream(Config) -> + Stream = <<"<?xml version=\"1.0\" encoding=\"utf-8\"?><a>hej</a>">>, + {ok, undefined, <<>>} = xmerl_sax_parser:stream(Stream, []), + ok. %%---------------------------------------------------------------------- %% Utility functions diff --git a/lib/xmerl/vsn.mk b/lib/xmerl/vsn.mk index 1515a4e37d..4e741d59a8 100644 --- a/lib/xmerl/vsn.mk +++ b/lib/xmerl/vsn.mk @@ -1 +1 @@ -XMERL_VSN = 1.3.13 +XMERL_VSN = 1.3.14 diff --git a/otp_versions.table b/otp_versions.table index 62eaa62fed..cb52cbd51e 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,3 +1,5 @@ +OTP-19.3.6 : erts-8.3.5 # asn1-4.0.4 common_test-1.14 compiler-7.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.4 debugger-4.2.1 dialyzer-3.1.1 diameter-1.12.2 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.3 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.4 ic-4.4.2 inets-6.3.9 jinterface-1.7.1 kernel-5.2 megaco-3.18.1 mnesia-4.14.3 observer-2.3.1 odbc-2.12 orber-3.8.2 os_mon-2.4.2 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.4 reltool-0.7.3 runtime_tools-1.11.1 sasl-3.0.3 snmp-5.2.5 ssh-4.4.2 ssl-8.1.3 stdlib-3.3 syntax_tools-2.1.1 tools-2.9.1 typer-0.9.12 wx-1.8 xmerl-1.3.14 : +OTP-19.3.5 : erts-8.3.4 xmerl-1.3.14 # asn1-4.0.4 common_test-1.14 compiler-7.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.4 debugger-4.2.1 dialyzer-3.1.1 diameter-1.12.2 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.3 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.4 ic-4.4.2 inets-6.3.9 jinterface-1.7.1 kernel-5.2 megaco-3.18.1 mnesia-4.14.3 observer-2.3.1 odbc-2.12 orber-3.8.2 os_mon-2.4.2 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.4 reltool-0.7.3 runtime_tools-1.11.1 sasl-3.0.3 snmp-5.2.5 ssh-4.4.2 ssl-8.1.3 stdlib-3.3 syntax_tools-2.1.1 tools-2.9.1 typer-0.9.12 wx-1.8 : OTP-19.3.4 : inets-6.3.9 ssl-8.1.3 # asn1-4.0.4 common_test-1.14 compiler-7.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.4 debugger-4.2.1 dialyzer-3.1.1 diameter-1.12.2 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.3 erts-8.3.3 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.4 ic-4.4.2 jinterface-1.7.1 kernel-5.2 megaco-3.18.1 mnesia-4.14.3 observer-2.3.1 odbc-2.12 orber-3.8.2 os_mon-2.4.2 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.4 reltool-0.7.3 runtime_tools-1.11.1 sasl-3.0.3 snmp-5.2.5 ssh-4.4.2 stdlib-3.3 syntax_tools-2.1.1 tools-2.9.1 typer-0.9.12 wx-1.8 xmerl-1.3.13 : OTP-19.3.3 : dialyzer-3.1.1 erts-8.3.3 inets-6.3.8 # asn1-4.0.4 common_test-1.14 compiler-7.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.4 debugger-4.2.1 diameter-1.12.2 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.3 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.4 ic-4.4.2 jinterface-1.7.1 kernel-5.2 megaco-3.18.1 mnesia-4.14.3 observer-2.3.1 odbc-2.12 orber-3.8.2 os_mon-2.4.2 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.4 reltool-0.7.3 runtime_tools-1.11.1 sasl-3.0.3 snmp-5.2.5 ssh-4.4.2 ssl-8.1.2 stdlib-3.3 syntax_tools-2.1.1 tools-2.9.1 typer-0.9.12 wx-1.8 xmerl-1.3.13 : OTP-19.3.2 : erts-8.3.2 # asn1-4.0.4 common_test-1.14 compiler-7.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.4 debugger-4.2.1 dialyzer-3.1 diameter-1.12.2 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.3 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.4 ic-4.4.2 inets-6.3.7 jinterface-1.7.1 kernel-5.2 megaco-3.18.1 mnesia-4.14.3 observer-2.3.1 odbc-2.12 orber-3.8.2 os_mon-2.4.2 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.4 reltool-0.7.3 runtime_tools-1.11.1 sasl-3.0.3 snmp-5.2.5 ssh-4.4.2 ssl-8.1.2 stdlib-3.3 syntax_tools-2.1.1 tools-2.9.1 typer-0.9.12 wx-1.8 xmerl-1.3.13 : @@ -7,6 +9,7 @@ OTP-19.2.3 : erts-8.2.2 inets-6.3.5 # asn1-4.0.4 common_test-1.13 compiler-7.0.3 OTP-19.2.2 : mnesia-4.14.3 # asn1-4.0.4 common_test-1.13 compiler-7.0.3 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.2 debugger-4.2.1 dialyzer-3.0.3 diameter-1.12.1 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.2 erts-8.2.1 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.3 ic-4.4.2 inets-6.3.4 jinterface-1.7.1 kernel-5.1.1 megaco-3.18.1 observer-2.3 odbc-2.12 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.3 reltool-0.7.2 runtime_tools-1.11 sasl-3.0.2 snmp-5.2.4 ssh-4.4 ssl-8.1 stdlib-3.2 syntax_tools-2.1.1 tools-2.9 typer-0.9.11 wx-1.8 xmerl-1.3.12 : OTP-19.2.1 : erts-8.2.1 # asn1-4.0.4 common_test-1.13 compiler-7.0.3 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.2 debugger-4.2.1 dialyzer-3.0.3 diameter-1.12.1 edoc-0.8.1 eldap-1.2.2 erl_docgen-0.6.1 erl_interface-3.9.2 et-1.6 eunit-2.3.2 gs-1.6.2 hipe-3.15.3 ic-4.4.2 inets-6.3.4 jinterface-1.7.1 kernel-5.1.1 megaco-3.18.1 mnesia-4.14.2 observer-2.3 odbc-2.12 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.4 percept-0.9 public_key-1.3 reltool-0.7.2 runtime_tools-1.11 sasl-3.0.2 snmp-5.2.4 ssh-4.4 ssl-8.1 stdlib-3.2 syntax_tools-2.1.1 tools-2.9 typer-0.9.11 wx-1.8 xmerl-1.3.12 : OTP-19.2 : common_test-1.13 compiler-7.0.3 crypto-3.7.2 dialyzer-3.0.3 edoc-0.8.1 erl_docgen-0.6.1 erl_interface-3.9.2 erts-8.2 eunit-2.3.2 hipe-3.15.3 inets-6.3.4 kernel-5.1.1 mnesia-4.14.2 observer-2.3 odbc-2.12 parsetools-2.1.4 public_key-1.3 runtime_tools-1.11 sasl-3.0.2 ssh-4.4 ssl-8.1 stdlib-3.2 syntax_tools-2.1.1 tools-2.9 wx-1.8 # asn1-4.0.4 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 debugger-4.2.1 diameter-1.12.1 eldap-1.2.2 et-1.6 gs-1.6.2 ic-4.4.2 jinterface-1.7.1 megaco-3.18.1 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 percept-0.9 reltool-0.7.2 snmp-5.2.4 typer-0.9.11 xmerl-1.3.12 : +OTP-19.1.6.1 : erts-8.1.1.1 # asn1-4.0.4 common_test-1.12.3 compiler-7.0.2 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.1 debugger-4.2.1 dialyzer-3.0.2 diameter-1.12.1 edoc-0.8 eldap-1.2.2 erl_docgen-0.6 erl_interface-3.9.1 et-1.6 eunit-2.3.1 gs-1.6.2 hipe-3.15.2 ic-4.4.2 inets-6.3.3 jinterface-1.7.1 kernel-5.1 megaco-3.18.1 mnesia-4.14.1 observer-2.2.2 odbc-2.11.3 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.3 percept-0.9 public_key-1.2 reltool-0.7.2 runtime_tools-1.10.1 sasl-3.0.1 snmp-5.2.4 ssh-4.3.6 ssl-8.0.3 stdlib-3.1 syntax_tools-2.1 tools-2.8.6 typer-0.9.11 wx-1.7.1 xmerl-1.3.12 : OTP-19.1.6 : erts-8.1.1 # asn1-4.0.4 common_test-1.12.3 compiler-7.0.2 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.1 debugger-4.2.1 dialyzer-3.0.2 diameter-1.12.1 edoc-0.8 eldap-1.2.2 erl_docgen-0.6 erl_interface-3.9.1 et-1.6 eunit-2.3.1 gs-1.6.2 hipe-3.15.2 ic-4.4.2 inets-6.3.3 jinterface-1.7.1 kernel-5.1 megaco-3.18.1 mnesia-4.14.1 observer-2.2.2 odbc-2.11.3 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.3 percept-0.9 public_key-1.2 reltool-0.7.2 runtime_tools-1.10.1 sasl-3.0.1 snmp-5.2.4 ssh-4.3.6 ssl-8.0.3 stdlib-3.1 syntax_tools-2.1 tools-2.8.6 typer-0.9.11 wx-1.7.1 xmerl-1.3.12 : OTP-19.1.5 : ssh-4.3.6 # asn1-4.0.4 common_test-1.12.3 compiler-7.0.2 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.1 debugger-4.2.1 dialyzer-3.0.2 diameter-1.12.1 edoc-0.8 eldap-1.2.2 erl_docgen-0.6 erl_interface-3.9.1 erts-8.1 et-1.6 eunit-2.3.1 gs-1.6.2 hipe-3.15.2 ic-4.4.2 inets-6.3.3 jinterface-1.7.1 kernel-5.1 megaco-3.18.1 mnesia-4.14.1 observer-2.2.2 odbc-2.11.3 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.3 percept-0.9 public_key-1.2 reltool-0.7.2 runtime_tools-1.10.1 sasl-3.0.1 snmp-5.2.4 ssl-8.0.3 stdlib-3.1 syntax_tools-2.1 tools-2.8.6 typer-0.9.11 wx-1.7.1 xmerl-1.3.12 : OTP-19.1.4 : ssh-4.3.5 # asn1-4.0.4 common_test-1.12.3 compiler-7.0.2 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.1 cosTime-1.2.2 cosTransactions-1.3.2 crypto-3.7.1 debugger-4.2.1 dialyzer-3.0.2 diameter-1.12.1 edoc-0.8 eldap-1.2.2 erl_docgen-0.6 erl_interface-3.9.1 erts-8.1 et-1.6 eunit-2.3.1 gs-1.6.2 hipe-3.15.2 ic-4.4.2 inets-6.3.3 jinterface-1.7.1 kernel-5.1 megaco-3.18.1 mnesia-4.14.1 observer-2.2.2 odbc-2.11.3 orber-3.8.2 os_mon-2.4.1 otp_mibs-1.1.1 parsetools-2.1.3 percept-0.9 public_key-1.2 reltool-0.7.2 runtime_tools-1.10.1 sasl-3.0.1 snmp-5.2.4 ssl-8.0.3 stdlib-3.1 syntax_tools-2.1 tools-2.8.6 typer-0.9.11 wx-1.7.1 xmerl-1.3.12 : diff --git a/system/doc/getting_started/conc_prog.xml b/system/doc/getting_started/conc_prog.xml index f3136898ad..4b19095d95 100644 --- a/system/doc/getting_started/conc_prog.xml +++ b/system/doc/getting_started/conc_prog.xml @@ -355,8 +355,8 @@ pong ! {ping, self()},</code> <p>Let us rewrite the ping pong program with "ping" and "pong" on different computers. First a few things are needed to set up to get this to work. The distributed Erlang - implementation provides a basic security mechanism to prevent - unauthorized access to an Erlang system on another computer. + implementation provides a very basic authentication mechanism to prevent + unintentional access to an Erlang system on another computer. Erlang systems which talk to each other must have the same <em>magic cookie</em>. The easiest way to achieve this is by having a file called <c>.erlang.cookie</c> in your home diff --git a/system/doc/reference_manual/distributed.xml b/system/doc/reference_manual/distributed.xml index 0a4a323fe9..01d78436c5 100644 --- a/system/doc/reference_manual/distributed.xml +++ b/system/doc/reference_manual/distributed.xml @@ -42,6 +42,19 @@ <p>The distribution mechanism is implemented using TCP/IP sockets. How to implement an alternative carrier is described in the <seealso marker="erts:alt_dist">ERTS User's Guide</seealso>.</p> + <warning> + <p> + Starting a distributed node without also specifying + <seealso marker="erts:erl#proto_dist"><c>-proto_dist inet_tls</c></seealso> + will expose the node to attacks that may give the attacker + complete access to the node and in extension the cluster. + When using un-secure distributed nodes, make sure that the + network is configured to keep potential attackers out. + See the <seealso marker="ssl:ssl_distribution"> + Using SSL for Erlang Distribution</seealso> User's Guide + for details on how to setup a secure distributed node. + </p> + </warning> </section> <section> |